Philippe

Farah

Paul

Matthis, Charles Agathe Florian

Thomas

Keywords: Robotique, vision par ordinateur, traitement du langage naturel, transformer i ii Robotics, computer vision, natural language processing, transformer iii iv

Les progrès réalisés dans le domaine de l'apprentissage automatique ont permis des percées importantes, notamment en vision par ordinateur, en traitement du langage naturel et en robotique. Peut-on aller plus loin et combiner ces domaines de recherche ? Cela développera de nouvelles applications, comme la robotique guidée par le langage, où un robot doit suivre les instructions fournies par un opérateur.

Alors que les humains apprennent à suivre des instructions dès leur enfance, la même tâche est difficile pour des robots, et cela pour plusieurs raisons: (i) le manque de données d'entraînements, (ii) les raisonnements faits sur multiples niveaux d'abstraction, et (iii) l'espace d'actions ayant une haute dimension.

L'objectif de cette thèse est d'améliorer la robotique guidée par le langage en relevant ces défis. Nous décomposons la difficulté de la robotique guidée par le langage en considérant deux types de tâches : (i) un robot mobile doit se rendre à un endroit cible décrit par des instructions ; (ii) les instructions décrivent une séquence d'actions qu'un bras robotique doit opérer sur des objets placés sur une table.

Nos contributions sont les suivantes : (i) pour résoudre le manque de données d'entraînement, nous avons développé une procédure efficace de pré-entraînement basé sur le nouveau jeu de données BnB, (ii) nous avons construit de nouvelles architectures neuronales basées sur une approche hiérarchique pour encoder plusieurs niveaux d'abstractions, et (iii) nous avons proposé une nouvelle méthode pour prédire des actions continus et en sur plusieurs dimensions pour résoudre un grand nombre de tâches.

Chapter 1 Introduction

This thesis addresses problems at the intersection of computer vision, natural language understanding, and robotics. Our main contributions concern language-guided visual navigation and manipulation. In this chapter, we first outline the goal and motivation of our work in Sections 1.1-1.2 and then summarize the challenges and contributions of our work in Sections 1.3-1.4. Section 1.5 provides a list of publications, datasets, and awards obtained in the course of this thesis.

Goal: language-guided robots

Artificial intelligence has a great potential to help people with a variety of tasks including healthcare, transportation, medicine, and education. Over the last decades, machine learning-based algorithms have surpassed people in multiple tasks which have been associated exclusively with human cognitive abilities. Examples of such tasks include playing the game of go [START_REF] Risi | From chess and atari to starcraft and beyond: How game ai is driving the world of ai[END_REF], the detection of tumoural cells [START_REF] Esteva | Deep learningenabled medical computer vision[END_REF], reading comprehension [START_REF] Therasa | Survey of machine reading comprehension models and its evaluation metrics[END_REF], or protein folding prediction [221].

Despite its success in solving particular problems, artificial intelligence (AI) is still far from human intelligence in general. The success of AI also builds on the huge amount of data and computing power that may not scale well to the large variety of existing problems. Thanks to large computational resources, models can be trained on larger datasets than humans could study over their lifetime. Scaling AI to a variety of problems with limited data, hence, requires new methods with a higher degree of generalization.

Animals and humans exhibit learning abilities and understandings of the world that are far beyond the capabilities of current AI [176]. For instance, if a child learns how to speak a language with limited exposure, a teenager can similarly learn to drive a car in about 20 hours of practice. People and other animals can understand how their environment works and interact with it while using a limited amount of observations and trials. Moreover, they demonstrate a stronger level of robustness than current AI systems [START_REF] Creswell | Generative adversarial networks: An overview[END_REF].

Current AI systems mostly have indirect knowledge about the physical world and 1 arguably lack embodied intelligence. To reduce the gap between artificial and human intelligence, AI systems may need to acquire knowledge through direct interaction with physical environments or their simulations. Towards this goal, AI systems can be equipped with bodies that (a) obey physical laws such as gravity and friction, (b) are able to perceive the environment through visual, touch, and other sensors, and (c) can change the state of the environment through actuators. In the rest of this thesis, we refer to such systems as embodied agents or robots. In particular, we consider scenarios where robots must follow instructions either for navigating the environment or for manipulating objects as illustrated in Figure 1.1. While a 5-year-old child would be able to solve these tasks, we will see that they remain challenging for robots. Solving navigation and manipulation tasks requires understanding visual sensor data, and natural language while acting in an unknown environment. While much progress has been done on these tasks separately, language-guided visual navigation and manipulation require a joint approach. The goal of this thesis is to present such a joint approach that can leverage limited annotated resources using weakly-supervised learning, reinforcement learning, and new architectures.

The first part of this thesis is focused on navigation tasks, where a mobile robot is supposed to navigate to a target location, inside an indoor environment that it has never seen before. As described in Figure 1.2, instructions provided to the robot consist either of step-by-step indications, such as in "walk straight then turn right and walk down the hallway until you get near the large painting of a man", or of a description of the target location, as in "Go to the dining room and pet the dog". In particular, we investigate how the robot can efficiently cope with unseen objects and unseen referring expressions. We approach the problem from two directions by (a) exploring additional readily-available data in combination with auxiliary losses and (b) new neural architectures based on recent Transformer models.

In the second part of the thesis, we turn to a related and less addressed problem of language-guided manipulation. Compared to navigation, object manipulation is more challenging as it requires appropriate contact between the object and the robot end-effector. As illustrated in Figure 1.3, we consider here object manipulation tasks

The robot navigates in an unseen environment by following the instruction.

At each step, it receives a panoramic view .

Walk past the large picture and chair. Walk past the dining room table turn right into the hallway and stop. The robot manipulates household objects. At each step, it receives a multi-view image with depth.

Press the yellow button, then press the black button and end up with the white button.

RealSense d435i

Robot UR5 using a robotics arm. We show that with a single architecture, we are able to learn several dozens of robotics tasks. We also demonstrate promising generalizations to human-written natural language instructions and to unseen variations of a given task.

Motivation

Robots can reduce our workload, improve our productivity and safety, and extend capabilities beyond what the human body and mind can do on their own. Therefore, robots that can follow instructions in unknown environments are beneficial for a myriad of tasks and can open up new possibilities for applications in an increasing number of domains, such as healthcare, education, search and rescue, and logistics.

Autonomous navigation. Search and rescue are key applications for robots that can navigate unknown environments. Autonomous robots could be deployed in hazardous locations, such as collapsed buildings, landmines, or contaminated areas. The benefits could be huge when it comes to saving lives in these scenarios, and the number of potential applications is growing. For example, fire departments are already using robots to find survivors in dangerous buildings [START_REF] Bogue | The role of robots in firefighting[END_REF], and robots able to communicate with those survivors would help them by offering emotional and psychological support, or warn them about dangers, such as gas leaks or unstable structures.

In addition to search and rescue, these robots could also be applied to urban exploration. Autonomous robots can be used to gather data in challenging and seemingly inaccessible areas, such as underground subway systems and sewers. Robots are already being used in many cities to inspect pipes, sewer lines, and other vital infrastructure [START_REF] Yu | Inspection robots in oil and gas industry: a review of current solutions and future trends[END_REF]. This type of data gathering combined with social skills can provide invaluable insights into urban development, allowing city planners to make better decisions about citywide infrastructure.

Another exciting place where robots can be used is in the area of logistics. Autonomous robots are already used to more efficiently move goods around a warehouse [START_REF] Jagtap | Food logistics 4.0: Opportunities and challenges[END_REF], but they require challenging and expensive programming, slowing down their deployment. Robots able to follow instructions could more easily be deployed in existing factories [START_REF] Wen | Swarm robotics control and communications: Imminent challenges for next generation smart logistics[END_REF]. This would drastically reduce the costs associated with manual labor and increase efficiency in various sectors.

Healthcare. In healthcare, robots have been proven to be able to fulfill therapeutic roles, like helping children with learning disabilities [324], motivating teens with diabetes to exercise, and keeping elderly Alzheimer's patients company [START_REF] Marlena H Shin | Improving a mobile telepresence robot for people with alzheimer disease and related dementias: Semistructured interviews with stakeholders[END_REF].

Education. Robots able to communicate with people have also the potential to be useful in education because they can provide personalized and interactive learning experiences for students. For example, robots could be used to provide one-on-one tutoring sessions, where the robot can answer questions, provide feedback, and reinforce learning objectives [START_REF] Mubin | A review of the applicability of robots in education[END_REF]. Additionally, social robots can serve as a bridge between physical and digital learning environments, allowing students to access resources and content in a more engaging and efficient manner. As such, social robots can be used to supplement traditional teaching methods, providing students with an enhanced learning experience that is both effective and enjoyable. Robots have already been tested and Belpaeme et al. [START_REF] Belpaeme | Social robots for education: A review[END_REF] shows that on restricted tasks, robots used as tutors or peer learners can achieve a similar impact that human tutoring. Understanding of our brain. This thesis builds on the Transformer neural network architecture. Exploration of neural network architectures may give us hints towards an understanding of the human brain [START_REF] James Cr Whittington | Relating transformers to models and neural representations of the hippocampal formation[END_REF]. This understanding can help us to better model language [START_REF] Caucheteux | Brains and algorithms partially converge in natural language processing[END_REF], which can lead to improved natural language processing applications, such as voice assistants and machine translation. Additionally, a better understanding of how our brains process information can help us to develop more effective artificial intelligence algorithms [START_REF] George | A hierarchical bayesian model of invariant pattern recognition in the visual cortex[END_REF]. Indeed, the neocortex has also highly uniform architecture, although it is involved in numerous tasks, such as sensory perception, language, and generation of motor commands. Enhancing further research. First of all, studies at the interface between computer vision, natural language processing, and robotics could strengthen research on each field independently, notably when consolidating them into a unique formulation. Consolidation is a major milestone for any scientific field, a valuable recognition of its maturity. Secondly, a ubiquitous architecture also helps to learn from the enormous volume of data available on the Internet (more than 500 hours of videos are uploaded to YouTube every minute [342] and almost 800 billion webpages have been recorded in the Wayback Machine [START_REF] Archive | Search the history of over 739 billion web pages on the internet[END_REF]). Several research papers [START_REF] Miech | HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips[END_REF][START_REF] Yang | Tubedetr: Spatio-temporal video grounding with transformers[END_REF] showed that leveraging such datasets can boost performance for downstream tasks. Similarly, in Chapter 3, we will see how we can leverage samples from a rental marketplace to pre-train a language-guided robot to navigate in unseen environments.

Challenges and contributions

By encompassing research topics related to natural language processing, robotics, and computer vision, the idea of controlling robots using instructions is cumulating their difficulties. In this Section, we go through some of the most challenging ones.

Diversity of the real world

Modeling our reality is a fundamental challenge due to its complexity and diversity. In this perspective, the Matterport3D simulator, used in Chapters 3 and 4, provides reconstructions of environments using RGB-D photos. But, collecting these photos is costly, and it can hardly be done at scale, whereas our algorithms are trained on a portion of them, and then tested on the others. For example, an unseen environment contains a Christmas tree, but how can we expect the model to "stop in front of the Christmas tree" when such an object does not appear in any of the training environments? Human-written instructions. The large diversity does not only encompass objects, but also our language. Indeed, not only vocabularies are extensive, with each word having a large number of synonyms, and our syntax is flexible, but also our utterances must be considered in a specific context. For example, if we ask a robot to bring us water while lying on a sofa, we are implicitly asking the robot to use a glass of water and not a bottle. Since we have access only to a limited amount of instructions during training, our models again suffer from a gap between training and testing. In this thesis, we will consider several strategies: in Chapter 3, we study how we can create new instructions using other datasets; in Chapter 5, our model is trained on synthetic instructions and then tested on human-written instructions thanks to a powerful CLIP's embeddings [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF]. Robots must learn with limited demonstrations. Having a limited number of instructions also means that we have a limited number of demonstrations, and this is particularly challenging for supervised learning algorithms, such as behavioral cloning, which are known to easily overfit due to their exposure bias [START_REF] Ross | A reduction of imitation learning and structured prediction to no-regret online learning[END_REF][START_REF] Torabi | Behavioral cloning from observation[END_REF]. As a solution, we will study how self-supervised learning and reinforcement learning can help us in Chapter 4. The scarcity of demonstrations is all the more penalizing that the action space is large. Indeed the larger the action space, the smaller it has been explored during training with a limited set of demonstrations. In Chapter 3, we will consider training our models on a large-scale dataset collected in a rental marketplace. In Chapters 3 and 4, we partially circumvent this problem by considering environments discretized into a graph smaller than one hundred nodes. In Chapter 5, the problem must be directly addressed, because our action space is made of 7 dimensions. We will see that our offered model can solve this issue.

Partially-observable environments

Throughout this thesis, we are interested in tasks, that require solving a sequence of steps in a specific order. For example, when a robot is asked to "clean the sink of the bathroom located on the second floor", the robot must first find the stairways, walk up the stairs, find the bathroom, and finally clean the sink. Sequential tasks raise a series of challenges: to avoid climbing several floors, the robot must remember that it has already climbed one floor, and hence its previous actions. Moreover, the robot might not find the bathroom on its first attempt, but it could enter several rooms first: how can it remember what has been seen before? Long-term planning is hence required, but it is also challenging because of the exponentially increasing number of possible combinations of actions that can be taken as the time horizon increases. This results in large search spaces and increases the complexity of the problem, making it more difficult to find optimal solutions. Additionally, there may be hidden patterns or dependencies between different points in the sequence that a model may not be able to detect. Long sequences also usually require data from previous timesteps to make decisions at later times, which makes handling temporal data a challenge. In Chapters 4 and 5, we tackle this issue by including previous steps when inferring the next action. High dimensional observations. Including the previous steps is not an easy task, since inputs have a high dimension. High dimensional data are known to be challenging to deal with due to the curse of dimensionality [162]. In the context of this thesis, we will use the transformer architecture, which was demonstrated to support high dimension data [START_REF] Jaegle | Perceiver io: A general architecture for structured inputs & outputs[END_REF]. However, this performance comes at a price, since the transformer suffers from a O(N 2 ) algorithmic complexity, where N is the number of tokens provided to the transformer. A token is a small piece of data, such as a block of pixels or a segment of a word. Through this thesis, N is particularly high: its value is between a few hundred to a thousand dimensions. Indeed, our transformer-based architectures are fed with instructions and with multi-views (or even panoramic views). Moreover, we show the positive impact of reasoning not only from the instant when the robot generates the next command but also from all previous ones. Multiple levels of abstraction. A solution to face this problem, as well as the scarcity of resources is to limit what needs to be learned by decomposing our problem into several layers of abstraction: (i) visual and textual representations extract meaning from raw data, (ii) motion planning computes a sequence of valid configurations to move a robot, (iii) and task planning breaks down a task into a sequence of steps executable by the motion planner. It is noticeable that this layout of layers is naturally conducted by humans. For example, when we are driving, our eyes observe the scene at a high frame rate (we are able to detect any change in the environment very quickly). But, reacting to a pedestrian appearing suddenly takes roughly a second, because we need to update our short-term plan. Moreover, in case we miss an exit on the road, we would need several seconds to come up with a new long-term plan. This thesis focuses on learning a task planner while using pre-trained models for extracting visual and textual representations and out-of-the-self inverse kinematics as a motion planner.

Real-world experiments

Hardware limitations. While RGB cameras provide higher-resolution images than human eyes, most of the other robotic sensors remain noisy and inaccurate. In particular, depth sensors, used in Chapter 5, suffer from many artifacts such as ghost points and important Gaussian noise. Robotic hardware is also limited by grippers. While precise grippers exist [START_REF] Akkaya | Solving rubik's cube with a robot hand[END_REF], their costs and their fragility are prohibitive for public research laboratories. Grippers have generally between two to three fingers with only a single degree of freedom, meaning that a robot can only close or open its pawn. Moreover, grippers have rarely haptic sensors able to provide feedback signals. Moreover, robots are slow: even pressing a button takes roughly 5 seconds. Running several millions of episodes (a complete trial of the robot) as we did during training is impossible to do. Another limitation to running experiments in the real world is the fragility of the hardware. For example, during this thesis, a bug in the motion planner severely damaged the gripper. As a consequence, interactions of a robot with its environment remain limited. While a human baby can spend several years playing with toys to discover object affordances, and learn manipulation skills, learning robotics policies from yearslong real-interaction data is impractical. Instead, we preferred to focus on simplified versions of robots running on a simulator [START_REF] Pfeifer | How the body shapes the way we think: a new view of intelligence[END_REF]. Simulated embodied agents follow physical rules, such as gravity or collisions. However, they benefit from defectless sensors and accurate actioners. Simulation-to-reality gap. Simulated embodied agents have several advantages: (i) obviously, any damage caused on a simulator has no impact; (ii) simulators provide a high-frequency rendering, allowing to run episodes faster than in the real world ; (iii) simulators can be run in parallel, providing a fleet of robots at a considerably lower cost than an equivalent real one. However, teaching skills to an embodied agent does not mean we are able to execute such skills in the real world. First, because they are visually unrealistic [234] (with the exception of photo-realistic renderings [43]), providing a large domain gap between reality and simulation. Second, contacts are typically hard to simulate: it happens frequently that a gripper is going through an object. As a consequence, grasping is dramatically simplified: if the embodied agent closes its gripper while an object is located close to the gripper, the object is "magically" grasped, even if it is physically impossible. In Chapter 5, we address this challenge, by finetuning a model pre-trained on a simulator with only one hundred demonstrations collected on a real-world robot.

Outline

This thesis consists of six chapters including this introduction. Our main technical contributions are separated into two parts: the first one is focused on language-guided navigation tasks, whereas the second addresses language-guided manipulation.

Literature survey. Chapter 2 reviews the related works in the literature on robotics controlled by instructions with a particular focus on (i) tasks based on navigation, dubbed vision-and-language navigation (VLN), and (ii) tasks based on manipulation, called similarly VLM for vision-and-language manipulation.

Self-supervised pre-training for sequential tasks. To improve generalization to unseen environments, we study strategies to pre-train a transformer architecture in Chapter 3. The generic formulation of the transformer deprived this architecture of prior knowledge, which allows this architecture to work well with different types of modalities. Although previous methods tried to pre-train a transformer on generic image-caption datasets, their approaches provided limited improvements in results. We introduce BnB, a large-scale and diverse in-domain VLN dataset collected from hundreds of thousands of listings from an online rental marketplace. As such, BnB is closer to the VLN domain than generic image-caption datasets. We mitigate even further the domain gap between image-caption pairs and the VLN task by rephrasing descriptions into navigation-like instructions and by adding contextual views to a single image. Hence, BnB contains (i) more than a million image-caption pairs, and (ii) automatically generated millions of VLN path-instruction pairs. Furthermore, a shuffling loss is proposed to enhance temporal order reasoning inside path-instructions pairs. First, we use BnB to pre-train our Airbert model in a discriminative setting. In this setting, Airbert must decide which trajectory fits the most the instructions given a pool of several candidate trajectories. Airbert needs hence an internal representation of the environment, as visiting each candidate trajectory in the real world is not realistic. To circumvent this limitation, Airbert is also extended to a generative setting, where it is predicting its next step given previously visited locations. Airbert is here trained to fit a ground truth demonstration with supervised learning. It induces an exposure bias as the training set is not explored enough. We fix this bias using reinforcement learning which increases the exploration of training environments. Overall, we show that Airbert outperforms state-of-the-art for R2R and REVERIE benchmarks. Moreover, our in-domain pretraining significantly increases performance on a challenging few-shot VLN evaluation, where we train the model only on VLN instructions from a few houses.

A hierarchical approach for the transformer. In the generative setting previously presented, the previously visited locations are encoded through a state vector. Similarly to recurrent neural networks, the state vector is propagated from step to step. It is suboptimal because the state vector encodes only a summary of the history. We would prefer to infer the next actions of a robot given all images from all previous steps. However, the robot observes its environment through multiple views, providing a input that are too large to be processed directly with a Transformer architecture. To mitigate this constraint, we employ a hierarchical approach in Chapter 4: each panoramic view is encoded into a single token, whereas, for the current time step, each image of the panoramic grid is encoded into a token. which first encodes individual images, then models spatial relation between images in a panoramic observation, and finally takes into account temporal relation between panoramas in history. Our obtained model dubbed a History Aware Multimodal Transformer (HAMT), is hence combining text, history, and current observation to predict the next action.

Manipulation-based tasks in the real world. So far, we consider scenarios in which robots must find a target location given some human-written instructions. In Chapter 5, We consider another scenario: a robotic arm with a gripper lying on a table has to accomplish various manipulation-based tasks given simple natural language instructions. Yet, some of these tasks can be highly challenging as it requires fine-grained motor control (e.g. playing Jenga), and long-term memory (e.g. when rearranging groceries in a cupboard) well as a generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state-of-the-art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.

Conclusion.

We conclude in Chapter 6 with a summary of contributions, a discussion of open problems, and future work.

Publications and Awards

In the following, we list the publications, awards, as well as software and dataset releases that were obtained during this thesis.

List of Publications

Contributions of this thesis have led to the following publications: In addition, three other papers were published during this Ph.D., extending the works presented in this thesis on language-guided navigation tasks, and in language referring tasks in 3D environments.

List of Awards

Our method [START_REF] Chen | Think global, act local: Dual-scale graph transformer for visionand-language navigation[END_REF] was ranked first at the REVERIE and SOON Challenges organized in conjunction with the ICCV 2021 Workshop on Human Interaction for Robotic Navigation. Our follow-up submission has obtained the second place at the REVERIE Challenge @ CSIG 2022 on two tracks.

List of Datasets

BnB. We collect 1.4 million images (half of them have a caption) over 144 thousand listings in a rental marketplace. This is several orders of magnitude higher than the number of images in Matterport3DSimulator [START_REF] Anderson | Bottom-up and top-down attention for image captioning and visual question answering[END_REF]. Airbnb has a total of 6 million listings, so using the provided code source might provide 50 times more images. R2R-Back. In Chapter 4, we assess the efficiency of our history module. That is why we offer a new variant of R2R [START_REF] Anderson | On evaluation of embodied navigation agents[END_REF] in which the robot must reach the target location and be able to go back to its initial location.

R2R-Last. Simularly, to judge the spatial reasoning of our approach, we create another variant, where only the last sentence of the instructions is provided. Auto-VLN. In Vil3DRel [START_REF] Chen | Language conditioned spatial relation reasoning for 3d object grounding[END_REF], we offer a new dataset made of pseudo-labels on 900 3D environments. This dataset has been successfully employed to pretrain a model and obtains new state-of-the-art results. RLBench testing instructions. In Chapter 5, we collect 162 human-written instructions to test our approach on two tasks: one where the robot presses up to three colored buttons, and another where the robot stacks colored blocks. In both cases, the sequence of colors is provided by the instructions.

Chapter 2

Literature review

This Chapter surveys the literature on vision, language, and robotics. We first review recent progress in computer vision and robotics in Sections 2.1 and 2.2 respectively. Related work on the intersection of natural language understanding, computer vision and robotics is then presented in Section 2.3 with a particular focus on language-guided manipulation in Section 2.3.3 and language-guided navigation in Section 2.3.4.

Computer vision

In recent years, the field of computer vision has seen enormous progress due to the emergence of deep learning algorithms. Major factors in this progress have been the emergence of new large-scale annotated datasets [START_REF] Bayoudh | A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets[END_REF] and the significant increase in computing power, which together enabled successful performance of convolutional neural networks (CNNs) [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] for a large range of problems such as image and video classification [START_REF] Tapaswi | Movieqa: Understanding stories in movies through question-answering[END_REF], object detection [START_REF] Liu | Deep learning for generic object detection: A survey[END_REF] and image segmentation [START_REF] Ronneberger | UNet: Convolutional networks for biomedical image segmentation[END_REF].

CNNs are composed of layers of neurons, where each neuron is connected to a certain number of neighboring neurons in the previous layer, as well as a certain number of weights associated with each of those connections. CNNs have been used extensively in the field of image recognition and classification. Notably, LeCun et al. [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] used CNNs to classify handwritten digits from the MNIST dataset in 1998, achieving a classification accuracy of 98%. Since then, a variety of other CNN architectures have been developed for image recognition and classification, such as AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], VGGNet [280], GoogLeNet [START_REF] Szegedy | Going deeper with convolutions[END_REF], and ResNet [START_REF] He | Deep residual learning for image recognition[END_REF]. These architectures have been used to achieve state-of-the-art results on a variety of tasks, such as image classification [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF][START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF], object detection, and semantic segmentation [START_REF] Lin | Microsoft coco: Common objects in context[END_REF][START_REF] Geiger | Vision meets robotics: The kitti dataset[END_REF][START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF]. The success of CNNs on this task led to more challenging applications, such as optical flow estimation [START_REF] Tian | Unsupervised learning of optical flow with cnn-based non-local filtering[END_REF], human pose estimation [START_REF] Varol | Bodynet: Volumetric inference of 3d human body shapes[END_REF], or shape reconstruction [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF]. The predominance of CNNs in computer vision applications can be explained by their efficient convolutional structure, but this structure may also impose disadvantages. For example, CNNs have a limited receptive field, which can limit the amount of context they can use in the decision-making process [START_REF] Szegedy | Going deeper with convolutions[END_REF].

Transformer [START_REF] Vaswani | Attention is all you need[END_REF] is an architecture based on a self-attention mechanism, which allows the network to learn long-term dependencies between input and output. Transformers have been originally developed for natural language processing, but they were recently found to outperform CNNs [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF], notably for image classification [START_REF] Arnab | Vivit: A video vision transformer[END_REF], image segmentation [START_REF] Strudel | Segmenter: Transformer for semantic segmentation[END_REF], and image generation [START_REF] Chang | Maskgit: Masked generative image transformer[END_REF]. This success can be impeded by the fact they are better suited to capturing long-range correlations between input features [START_REF] Zhu | Long-short transformer: Efficient transformers for language and vision[END_REF], and they have less bias in their formulation [START_REF] Xu | Vitae: Vision transformer advanced by exploring intrinsic inductive bias[END_REF]. Furthermore, Transformers enable convenient joint modeling of multiple modalities such as text, images, video, and sound [START_REF] Jaegle | Perceiver io: A general architecture for structured inputs & outputs[END_REF][START_REF] Tan | Lxmert: Learning cross-modality encoder representations from transformers[END_REF][START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF]. However, their adaptation to computer vision remains an open challenge, partially because they require additional resources such as memory and computing to be able to extract features from the data [START_REF] Lan | Albert: A lite bert for self-supervised learning of language representations[END_REF]. Moreover, their lack of inductive biases makes them hard to generalize to small-scale datasets [START_REF] Arnab | Vivit: A video vision transformer[END_REF].

In this work, we aim to address the above limitations of the Transformer architecture, by collecting additional datasets (Ch. 3), reformulating this architecture with a hierarchical approach (Ch. 4), or by combining it with CNNs (Ch. 5), allowing us to capture long-range correlations using a very limited corpus of demonstrations.

Robotics

Control is an essential component of robotics, as it allows robots to interact with the environment and carry out their intended tasks. Real-time control of autonomous robots requires an efficient and robust control architecture, which can rapidly respond to environmental changes. This includes the ability to detect and avoid obstacles, as well as accurately track and follow a given path. To improve the accuracy and reliability of robotic motion control, model predictive control [START_REF] Eduardo | Model predictive control[END_REF] uses a model of the system to predict the future state of the system and then optimizes a cost function to determine a control action. Furthermore, control architectures suitable for autonomous robots need to be robust enough to handle dynamic environments, and for this purpose, adaptive control [START_REF] Karl | Adaptive control[END_REF] uses feedback to adjust the parameters of a controller in response to changes in the environment or system. In the context of this thesis, we assume the environment is controlled and fixed, allowing to use of more basic controllers, such as inverse kinematics [START_REF] Kucuk | Robot kinematics: Forward and inverse kinematics[END_REF], a technique that solves the problem of finding the joint angles necessary to achieve a desired end effector position.

Motion planning. Robotics research has made great strides in motion planning [START_REF] Yang | Survey of optimal motion planning[END_REF], to generate collision-free paths that can be followed by the robot. Much of the existing work assumes a known structure of the environment. Notably, visual servoing [START_REF] Chaumette | Visual servo control. i. basic approaches[END_REF] is a process by which a robot is able to navigate through a given environment by relying on visual feedback from its sensors, and it has been widely used in applications such as robotic navigation and robotic manipulation. Furthermore, rapidly exploring random trees (RRT) [START_REF] Karaman | Anytime motion planning using the rrt[END_REF] have been applied to motion planning. RRT generates a tree-like structure to efficiently search for the optimal path from a given start to a given goal. RRT is often used for visual servoing applications since RRT is able to generate paths that are close to the desired goal and can be easily followed by the robot. However, having an access to a known structure of the environment is not realistic for unknown and complex environments [START_REF] Strudel | Learning obstacle representations for neural motion planning[END_REF]. Simultaneous Localization and Mapping [START_REF] Thrun | Probabilistic robotics[END_REF] (SLAM) has been introduced to overcome this limitation. It enables robots to create a map of their environment and localize themselves within the map. SLAM algorithms use a combination of computer vision, odometry, and sensor fusion to create accurate maps and enable robots to autonomously navigate in unknown environments. Techniques for SLAM include Monte-Carlo localization [START_REF] Behzadian | Monte carlo localization in hand-drawn maps[END_REF], extended Kalman filter [START_REF] Horaţiu | Extended kalman filter (ekf)-based local slam in dynamic environments: A framework[END_REF], Rao-Blackwellized particle filter [START_REF] Grisetti | Fast and accurate slam with rao-blackwellized particle filters[END_REF], visual odometry [START_REF] Yousif | An overview to visual odometry and visual slam: Applications to mobile robotics[END_REF] and optic flow [START_REF] Zhang | Flowfusion: Dynamic dense rgb-d slam based on optical flow[END_REF]. Policy learning is an alternative line of work that directly learns sensorimotor policies from raw sensor inputs. For example, imitation learning methods [START_REF] Hussein | Imitation learning: A survey of learning methods[END_REF] allow robots to learn from demonstrations of desired behaviors, while reinforcement learning (RL) [START_REF] Richard | Reinforcement learning: An introduction[END_REF] methods enable robots to learn from a reward signal. These methods have been successful for a variety of robotic tasks, such as robot navigation, object manipulation, and locomotion. Nevertheless, learning-based methods are often data-hungry and require extensive training in simulation before being able to transfer to the real world. This is known as the sim-to-real problem [START_REF] Zhao | Sim-to-real transfer in deep reinforcement learning for robotics: a survey[END_REF] and is an area of active research. Recent work has attempted to address this issue by combining imitation and RL [START_REF] Strudel | Learning to combine primitive skills: A step towards versatile robotic manipulation[END_REF], using domain randomization to make models more robust to domain shifts [START_REF] Labbé | Monte-carlo tree search for efficient visually guided rearrangement planning[END_REF], and using meta-learning to enable an agent to rapidly adapt to new environments [START_REF] Finn | One-shot visual imitation learning via meta-learning[END_REF]. In our work, we present a method for learning visuomotor controllers for unknown and complex environments. Our method is based on a combination of imitation learning and RL and is capable of generalizing to unseen environments. We also tested the method developed in Chapter 5 on a real-world robot.

Language-guided agents

Does language influence the way we think? Linguists [263] have been passionate about this question for a century. Whorf [START_REF] Lee | Language, thought, and reality: Selected writings of Benjamin Lee Whorf[END_REF] suggests the linguistic relativity hypothesis saying that language determines thought and hence cognition abilities. Later, the role of language has experimentally been studied in the acquisition of counter-factual analysis [START_REF] Alfred | The impact of chinese linguistic structure on cognitive style[END_REF][START_REF] Au | Counterfactuals: In reply to alfred bloom[END_REF], memory [START_REF] Frode | Memory models and language comprehension[END_REF], navigation [START_REF] Stephen C Levinson | Studying spatial conceptualization across cultures: Anthropology and cognitive science[END_REF], and color [108,[START_REF] John | Linguistic relativity[END_REF][START_REF] Saunders | Revisiting basic color terms[END_REF].

Does it also hold for AI systems? In other words, could AI systems benefit from embodiment and language acquisition? According to Bisk et al. [START_REF] Bisk | Experience grounds language[END_REF], "in addition to learning basic physical properties of the world from interaction, [embodiment] also allows the agent to construct rich pre-linguistic representations from which to generalize."

Connecting language to embodied agents

Translating natural language to formal language. Winograd [329] offered seminal works in this direction: using exhaustive symbolic rules to translate natural language into a set of executable instructions, his program called SHRDLU is able to assemble geometric shapes given an order provided by an operator. Further works improved this idea by translating natural language instructions into a set of pre-defined actions [START_REF] Tellex | Understanding natural language commands for robotic navigation and mobile manipulation[END_REF], lambda calculus expressions [START_REF] John | Learning to parse database queries using inductive logic programming[END_REF][START_REF] Zettlemoyer | Online learning of relaxed ccg grammars for parsing to logical form[END_REF], or reward functions [START_REF] Squire | Grounding english commands to reward functions[END_REF]. Those works have been using probabilistic graphical models [START_REF] Kollar | Toward understanding natural language directions[END_REF][START_REF] Thomas M Howard | A natural language planner interface for mobile manipulators[END_REF], weakly supervised learning [START_REF] Artzi | Weakly supervised learning of semantic parsers for mapping instructions to actions[END_REF], or sequence-to-sequence recurrent networks [START_REF] Nakul Gopalan | Sequence-to-sequence language grounding of non-markovian task specifications[END_REF][START_REF] Arumugam | Grounding natural language instructions to semantic goal representations for abstraction and generalization[END_REF].

Using formal language as an intermediary connection between language and sensorimotor processing has a significant consequence: the environment does not help to learn the language. In other words, the learning process is unilateral, contrary to previous examples from embodied language processing. To address this limitation, can we bring closer language and sensorimotor processing?

Connecting language to vision. A fundamental task is semantic segmentation, which consists in assigning each pixel of an image to a semantic label. First neural architectures [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] emerged for this purpose. Further improvements were suggested, such as regional CNNs [START_REF] Girshick | Fast R-CNN[END_REF][START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF], feature pyramids [START_REF] Lin | Feature pyramid networks for object detection[END_REF], recurrent CNNs [START_REF] Pinheiro | Recurrent convolutional neural networks for scene labeling[END_REF], or encoder-decoder [START_REF] Md | Gated feedback refinement network for dense image labeling[END_REF]. Those solutions have thereafter been employed for visual question answering [START_REF] Antol | VQA: Visual Question Answering[END_REF][START_REF] Drew | GQA: A new dataset for realworld visual reasoning and compositional question answering[END_REF] and video question answering [START_REF] Tapaswi | Movieqa: Understanding stories in movies through question-answering[END_REF][START_REF] Lei | Tvqa: Localized, compositional video question answering[END_REF], offering innovative approaches based on LSTMs [146], and hierarchical learning [START_REF] Zhao | Video question answering via hierarchical dual-level attention network learning[END_REF].

Connecting language to spatial localization. Going one step further, several vision and language tasks assess the agent's abilities to ground spatial referring expressions [START_REF] Qiao | Referring expression comprehension: A survey of methods and datasets[END_REF].

In ReferIt [START_REF] Kazemzadeh | Refer-ItGame: Referring to objects in photographs of natural scenes[END_REF], the agent receives a referring expression describing the localization of an item in an image, and the agent must localize such objects. The tasks were popularized with the 2D ReferIt Game [START_REF] Kazemzadeh | Refer-ItGame: Referring to objects in photographs of natural scenes[END_REF] and it was developed in further datasets [START_REF] Irmak Dogan | Learning to generate unambiguous spatial referring expressions for real-world environments[END_REF][START_REF] Kong | What are you talking about? text-to-image coreference[END_REF][START_REF] Krishna | Visual genome: Connecting language and vision using crowdsourced dense image annotations[END_REF]. A similar task [START_REF] Lu | Visual relationship detection with language priors[END_REF][START_REF] Krishna | Visual Genome: Connecting language and vision using crowdsourced dense image annotations[END_REF] consists in detecting visual relationships in images.

Connecting language to temporal localization is another attempt to bring closer language understanding with an agent's cognition processing. The temporal activity localization via language query [START_REF] Gao | Tall: Temporal activity localization via language query[END_REF][START_REF] Chen | Temporally grounding natural sentence in video[END_REF][START_REF] Lisa | Localizing moments in video with natural language[END_REF] formulated as a clip retrieval task based on the description of an activity. Methods are based on auxiliary losses [START_REF] Xu | Multilevel language and vision integration for text-to-clip retrieval[END_REF], contrastive learning [START_REF] Ma | X-clip: End-to-end multi-grained contrastive learning for video-text retrieval[END_REF], and transformers [START_REF] Luo | Clip4clip: An empirical study of clip for end to end video clip retrieval[END_REF].

These aforementioned tasks and methods are restricted to 2D representations of an environment, failing to capture the true 3D nature of our world. Chen et al. [START_REF] Zhenyu | ScanRefer: 3d object localization in rgb-d scans using natural language[END_REF] mentioned notably that the "first giraffe" is particularly challenging to recognize on a 2D image of two interlaced giraffes. In the general case, an image alone might not provide precise position cues, which are essential for robotic tasks such as grasping. and then they are match to instructions [START_REF] Achlioptas | ReferIt3D: Neural listeners for fine-grained 3d object identification in real-world scenes[END_REF][START_REF] Zhenyu | ScanRefer: 3d object localization in rgb-d scans using natural language[END_REF][START_REF] Roh | Languagerefer: Spatiallanguage model for 3d visual grounding[END_REF], notably by using a graphical representation of the scene [START_REF] Yuan | Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring[END_REF]129] to infer spatial relations. Latest methods [START_REF] Yang | Sat: 2d semantics assisted training for 3d visual grounding[END_REF][START_REF] He | Transrefer3d: Entity-and-relation aware transformer for fine-grained 3d visual grounding[END_REF][START_REF] Roh | Languagerefer: Spatiallanguage model for 3d visual grounding[END_REF][START_REF] Huang | Multi-view transformer for 3d visual grounding[END_REF][START_REF] Luo | 3d-sps: Single-stage 3d visual grounding via referred point progressive selection[END_REF][START_REF] Zhao | 3dvg-transformer: Relation modeling for visual grounding on point clouds[END_REF]] have adopted the transformer architecture [START_REF] Vaswani | Attention is all you need[END_REF], allowing to detect and match simultaneously [START_REF] Jain | Looking outside the box to ground language in 3d scenes[END_REF][START_REF] Luo | 3d-sps: Single-stage 3d visual grounding via referred point progressive selection[END_REF]. ReCLIP [START_REF] Subramanian | Reclip: A strong zero-shot baseline for referring expression comprehension[END_REF] is using precomputed embeddings on the instructions.

Language-guided manipulation

Interest for robotics in research communities was boosted with international competitions, notably RoboCup [START_REF] Asada | Robocup: Today and tomorrow-what we have learned[END_REF], and Amazon Robotics Challenge [START_REF] Eppner | Lessons from the amazon picking challenge: Four aspects of building robotic systems[END_REF], but also easilyreproducible benchmarks such as ARCV [START_REF] Leitner | The acrv picking benchmark: A robotic shelf picking benchmark to foster reproducible research[END_REF], OpenGrasp [START_REF] León | Opengrasp: a toolkit for robot grasping simulation[END_REF], and VisGrab [START_REF] Kootstra | Visgrab: A benchmark for vision-based grasping[END_REF]. Although the robot remains fixed, manipulation tasks are particularly challenging as they require dealing with a continuous action space across several dimensions. Simulators and datasets. Simulations of physically-realistic robots require the development of a specific range of simulators. For example, ManipulaTHOR [START_REF] Ehsani | ManipulaTHOR: A framework for visual object manipulation[END_REF] introduced realistic interaction with objects using a mobile manipulator with seven degrees of freedom.

Based on the simulator CoppeliaSim [START_REF] S Rooban | Coppeliasim: Adaptable modular robot and its different locomotions simulation framework[END_REF] (formerly V-REP), RLBench [START_REF] James | RL-Bench: The robot learning benchmark & learning environment[END_REF] offers more than 100 tasks with automatically generated demonstrations. Moreover, RLBench is composed of synthetic-generated instructions. Built as an extension of RLBench, VLMBench [START_REF] Zheng | Vlmbench: A compositional benchmark for vision-and-language manipulation[END_REF] is focused on language-guided embodied agents. It is composed of eight tasks but thousands of variations of these tasks.

CALVIN [START_REF] Mees | CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks[END_REF] is another simulated benchmark to learn long-horizon tasks specified with human language. Compared to VLMBench, CALVIN tasks are more complex in terms of sequence length and language. Moreover, it can evaluate embodied agents in unseen environments. Drawing inspirations from Learning from Play [START_REF] Lynch | Learning latent plans from play[END_REF]211], CALVIN provides annotations with 20k language directives of 24 hours of teleoperated unstructured play data. Imitation learning. Assuming the existence of demonstrations, an embodied agent can learn to reproduce ground truth trajectories conditioned on instructions. Behavioral cloning-based approaches [START_REF] Torabi | Behavioral cloning from observation[END_REF][START_REF] Strudel | Learning to combine primitive skills: A step towards versatile robotic manipulation[END_REF] rely on supervised learning to learn the trajectories.

Imitation learning has intensively been employed in robotics manipulation, notably to train recurrent neural networks [START_REF] Lynch | Learning latent plans from play[END_REF], U-Net structures [START_REF] Liu | Auto-Lambda: Disentangling dynamic task relationships[END_REF], or with active learning [START_REF] Ross | A reduction of imitation learning and structured prediction to no-regret online learning[END_REF]. Particularly impressive results were achieved for generalization to new skills [START_REF] Florence | Implicit behavioral cloning[END_REF][START_REF] Jang | BC-Z: Zero-shot task generalization with robotic imitation learning[END_REF][START_REF] Dasari | Robonet: Large-scale multi-robot learning[END_REF].

Imitation learning is the main paradigm for training neural networks for solving language-guided manipulation tasks, as the continuous action space and the limited amount of data are particularly challenging for reinforcement learning-based algorithms. The transporter network was adapted to this class of tasks in CLIPort [START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF] and was extended to a seven-dimensional action space in VLMBench [START_REF] Zheng | Vlmbench: A compositional benchmark for vision-and-language manipulation[END_REF]. Hristov et al. [START_REF] Hristov | Disentangled relational representations for explaining and learning from demonstration[END_REF], Lynch et al.

[211] and CALVIN's baseline [START_REF] Mees | CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks[END_REF] are based on a sequence-to-sequence conditional variational auto-encoder network. Applications to real-world. Tellex et al. [START_REF] Tellex | Understanding natural language commands for robotic navigation and mobile manipulation[END_REF] and Misra et al. [START_REF] Dipendra K Misra | Tell me dave: Context-sensitive grounding of natural language to manipulation instructions[END_REF] succeeded in applying algorithms on a real-world robot. However, using a translation of the instructions into a formal language, the set of executable actions remained limited in their work. Among the aforementioned methods that predict low-level commands, only a few have been tested on a real-world robot [START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF][START_REF] Hristov | Disentangled relational representations for explaining and learning from demonstration[END_REF]. However, CLIPort [START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF] is using a simplified action space, whereas Hristov et al. [START_REF] Hristov | Disentangled relational representations for explaining and learning from demonstration[END_REF] is using simplified instructions. In Chapter 5, we will see that our methods can be applied in a real-world robot with seven degrees of freedom and human-written instructions.

Language-guided navigation

While the aforementioned tasks have popularized language-guided embodied agents, the latter have visual access to most of the environment though some parts are visually occluded. This is not the case for mobile agents that must navigate to a different room of their environment to fulfill their tasks. Simulators. To handle embodied agents, more simulators have been offered to the research community. They differ by their scale [START_REF] Johnson | The malmo platform for artificial intelligence experimentation[END_REF][START_REF] Savva | Habitat: A platform for embodied ai research[END_REF][START_REF] Savva | Minos: Multimodal indoor simulator for navigation in complex environments[END_REF], their degree of customization [27,[START_REF] Srivastava | Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments[END_REF], their latency [154], their interactivity [START_REF] Brodeur | Home: A household multimodal environment[END_REF][START_REF] Wu | Building generalizable agents with a realistic and rich 3d environment[END_REF][START_REF] Kolve | AI2-THOR: An interactive 3d environment for visual ai[END_REF][START_REF] Shen | igibson 1.0: a simulation environment for interactive tasks in large realistic scenes[END_REF][START_REF] Xiang | Sapien: A simulated part-based interactive environment[END_REF], or their visual realism [43].

A new generation of simulators is appearing, combining most of these advantages, notably with Gibson 2.0 [START_REF] Li | igibson 2.0: Object-centric simulation for robot learning of everyday household tasks[END_REF], BEHAVIOR-1K [START_REF] Li | Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation[END_REF], ThreeDWorld [START_REF] Gan | Threedworld: A platform for interactive multi-modal physical simulation[END_REF]. The emergence of these simulators led to new challenges as described below. Embodied Question Answering. In embodied question answering [START_REF] Das | Embodied question answering[END_REF], a question is asked to an embodied agent. But contrary to visual question answering [START_REF] Antol | VQA: Visual Question Answering[END_REF], the agent must navigate in its environment to find the answer. For example, when asked "what is the color of the car in the garage", the agent must first navigate to the garage. Das et al. [START_REF] Das | Embodied question answering[END_REF]'s seminal work was developed in the House3D environment [START_REF] Wu | Building generalizable agents with a realistic and rich 3d environment[END_REF] was been extended to multi-agents [START_REF] Tan | Multi-agent embodied question answering in interactive environments[END_REF], multi-targets [START_REF] Yu | Multi-target embodied question answering[END_REF], and photo-realistic environments [START_REF] Wijmans | Embodied Question Answering in Photorealistic Environments with Point Cloud Perception[END_REF]. VideoNavQA [START_REF] Cangea | Videonavqa: Bridging the gap between visual and embodied question answering[END_REF] extended the dataset to 28 types of questions to improve the diagnostics of an agent's reasoning abilities. Vision-and-language navigation. Because of the versatile nature of such embodied agents, different tasks have been offered. Touchdown [START_REF] Chen | Touchdown: Natural language navigation and spatial reasoning in visual street environments[END_REF], R2R [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Anderson | On evaluation of embodied navigation agents[END_REF], R4R [START_REF] Jain | Stay on the path: Instruction fidelity in vision-and-language navigation[END_REF], and RxR [START_REF] Ku | Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding[END_REF], Landmark-RxR [113], Talk2Nav [START_REF] Balajee Vasudevan | Talk2nav: Longrange vision-and-language navigation with dual attention and spatial memory[END_REF], StreetLearn [START_REF] Mirowski | The streetlearn environment and dataset[END_REF] provide fine-grained navigation instructions to the agent that must find a target location. In REVERIE [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF] and SOON [START_REF] Zhu | Soon: Scenario oriented object navigation with graph-based exploration[END_REF], the task is similar, but the agent receives only a coarse-grained instruction, describing the target location and not the path towards this location. Other datasets have encompassed the presence of dialogs [START_REF] Thomason | Visionand-dialog navigation[END_REF][START_REF] Nguyen | Help, Anna! Visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning[END_REF][START_REF] Nguyen | Vision-based navigation with language-based assistance via imitation learning with indirect intervention[END_REF]. Efforts were also put forward to increase interactions, embodied agents are also asked to interact with objects in CHAI [START_REF] Misra | Mapping instructions and visual observations to actions with reinforcement learning[END_REF] and ALFRED [START_REF] Shridhar | ALFRED: A benchmark for interpreting grounded instructions for everyday tasks[END_REF] or fine-grained action space with VLN-CE [START_REF] Krantz | Beyond the nav-graph: Vision-and-language navigation in continuous environments[END_REF] and RoboVLN [START_REF] Zubair Irshad | Hierarchical crossmodal agent for robotics vision-and-language navigation[END_REF]. Finally, dialog and interaction have been combined in DialFRED [START_REF] Gao | Dialfred: Dialogue-enabled agents for embodied instruction following[END_REF] and TEACh [237]. Evaluation of vision-and-language navigation. Metrics are generally of the success of the goal by measuring the agent's proximity to the goal. For example, in datasets based on Matterport3D [START_REF] Chang | Matterport3D: Learning from RGB-D data in indoor environments[END_REF], an agent is successfully completing a task when it stops on the target location with a tolerance of 3 meters. The success rate [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Chen | Touchdown: Natural language navigation and spatial reasoning in visual street environments[END_REF] then how frequently the agent has completed tasks on the testing set. However, this metric does not take into account the path taken by the agent, whereas taking the shortest path is reasonably harder than exploring first the environment. For this reason, the success rate is often weighted by the path length [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF]. Some other metrics are penalizing long trajectories, such as the path fidelity score [START_REF] Jain | Stay on the path: Instruction fidelity in vision-and-language navigation[END_REF] or the normalized dynamic time warping [START_REF] Ilharco | General evaluation for instruction conditioned navigation using dynamic time warping[END_REF]. As the numbers of datasets grow, the methods for addressing them have also increased. Here, we summarize the main directions in the literature.

Pre-training. The multimodality nature of those tasks allowed methods to study different types of pre-training: on text [START_REF] Li | Robust navigation with language pretraining and stochastic sampling[END_REF], on semantic segmentation [START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF], on other vision and language datasets [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF]. Several articles [START_REF] Shen | How much can CLIP benefit vision-andlanguage tasks?[END_REF][START_REF] Khandelwal | Simple but effective: CLIP embeddings for embodied ai[END_REF][START_REF] Li | CLEAR: Improving vision-language navigation with cross-lingual, environment-agnostic representations[END_REF] show the advantages of using CLIP embeddings [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF], achieving notably remarkable performance in zeroshot learning [START_REF] Shah | Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action[END_REF]. In Chapter 3, we study how we can benefit from an in-domain pretraining dataset for pretraining a transformer-based architecture.

Data-Augmentation. Similarly, several works suggest data-augmentation methods, notably by generating novel instructions using a speaker model [START_REF] Fried | Speaker-Follower models for vision-and-language navigation[END_REF][START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF][START_REF] Dou | Foam: A follower-aware speaker model for visionand-language navigation[END_REF], editing environments [START_REF] Li | Envedit: Environment editing for vision-andlanguage navigation[END_REF][START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF], or mixing several existing environments [START_REF] Liu | Vision-language navigation with random environmental mixup[END_REF]. Several methods are also generating subgoal instructions [START_REF] Lin | Adapt: Vision-language navigation with modality-aligned action prompts[END_REF][START_REF] Hong | Sub-instruction aware vision-and-language navigation[END_REF][START_REF] Shah | Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action[END_REF].

Memory.. Since the task is a sequential process, encoding a memory mechanism is crucial to remember previous steps achieved by the agent. First algorithms employed recurrent neural networks [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Fried | Speaker-Follower models for vision-and-language navigation[END_REF]212,[START_REF] Li | Robust navigation with language pretraining and stochastic sampling[END_REF][START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF][START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF], but it remains difficult to remember long sequences of steps. Another solution consists in building a separate memory model dedicated to the summarization of previous steps [START_REF] Zhu | Babywalk: Going farther in vision-and-language navigation by taking baby steps[END_REF][START_REF] Lin | Multimodal transformer with variable-length memory for vision-andlanguage navigation[END_REF][START_REF] Nguyen | Robot object retrieval with contextual natural language queries[END_REF]. Recent works [START_REF] Pashevich | Episodic transformer for vision-and-language navigation[END_REF][START_REF] Moudgil | Soat: A scene-and object-aware transformer for vision-and-language navigation[END_REF] are encoding the full history through a multi-modal transformer architecture. This solution is further explored in Chapter 4.

Auxiliary tasks have been shown to be beneficial to agents [START_REF] Zhu | Vision-language navigation with self-supervised auxiliary reasoning tasks[END_REF], either acting as regularizers [212], or improving the alignment between several modalities [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF]. In Chapter 3, we provide an auxiliary task enhancing the agent's capacity to reason over time, and we combine this task as an additional loss term. In Chapter 4, four different auxiliary tasks are shown to significantly improve our architecture performance.

Map representation. When optimizing the SPL metrics, we can not assume that the robot has access to a previously scan of the environment, contrary to Chapter 3. In this case, we need some mechanisms to track previously seen locations [START_REF] Ma | The Regretful Agent: Heuristic-aided navigation through progress estimation[END_REF], and notably a map incorporating structured information from observations and eventually from instructions. An example of map representation in language-guided embodied agents can be found in Blukis et al. [START_REF] Blukis | Mapping navigation instructions to continuous control actions with position-visitation prediction[END_REF] using a 2D representation of the trajectory of a quadcopter. In the context of the aforementioned datasets, Anderson et al. [START_REF] Anderson | Chasing ghosts: Instruction following as bayesian state tracking[END_REF] employs metrics maps based on SLAM [START_REF] Thrun | Probabilistic robotics[END_REF][START_REF] Singh Chaplot | Neural topological SLAM for visual navigation[END_REF] tasks. But, to alleviate the difficulties of constructing such maps and then encoding them, a topological graph structure is proposed in several works using pre-exploration [START_REF] Chen | Topological planning with transformers for vision-and-language navigation[END_REF] or back-tracking [START_REF] Ma | The Regretful Agent: Heuristic-aided navigation through progress estimation[END_REF], at the risk of increasing the path length. Topological maps have been encoded in separate modules without language grounding and using recurrent neural networks [START_REF] Zubair Irshad | Hierarchical crossmodal agent for robotics vision-and-language navigation[END_REF][START_REF] Wang | Structured scene memory for vision-language navigation[END_REF][START_REF] Deng | Evolving graphical planner: Contextual global planning for vision-and-language navigation[END_REF]. We have provided a solution addressing these issues in DUET [START_REF] Chen | Think global, act local: Dual-scale graph transformer for visionand-language navigation[END_REF].

Reinforcement learning enables embodied agents to learn how to act from a reward provided by the environment [START_REF] Richard | Reinforcement learning: An introduction[END_REF]. This framework formulates a sequential decision as an optimization process, maximizing the expected discounted cumulative return, which is the sum of rewards received during an episode. Reinforcement learning has been used to solve tasks conditioned on language [START_REF] Luketina | A survey of reinforcement learning informed by natural language[END_REF] and it has been shown efficient to improve the environment exploration [START_REF] Wang | Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation[END_REF][START_REF] Wang | Soft expert reward learning for visionand-language navigation[END_REF][START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF]. However, it suffers from the credit assignment problem [START_REF] Richard | Reinforcement learning: An introduction[END_REF], since the success signal is provided only at the end of the task, making it difficult to know which steps to penalize.

Chapter 3

In-domain Pretraining for Vision and Language Navigation

This Chapter presents our first contribution to vision-and-language navigation (VLN), which focuses on pre-training a transformer on a large-scale and diverse dataset collected on the web. This work is addressing the scarcity of domain-specific training data and the high diversity of image and language inputs, making the generalization of VLN agents to unseen environments particularly challenging.

Recent methods explore pretraining to improve generalization, however, the use of generic image-caption datasets or existing small-scale VLN environments is suboptimal and results in limited improvements. In this chapter, we introduce BnB1 , a large-scale and diverse in-domain VLN dataset. We first collect image-caption (IC) pairs from hundreds of thousands of listings from online rental marketplaces. Using IC pairs we next propose automatic strategies to generate millions of VLN path-instruction (PI) pairs. We further propose a shuffling loss that improves the learning of temporal order inside PI pairs. We use BnB to pretrain our Airbert2 model. For two years, our method using a discriminative setting has been ranked first on the leaderboard of the Room-to-Room (R2R) navigation benchmark [START_REF] Eval | Leadearboard evalai -room-to-room dataset[END_REF], which is a rare feat in a competitive research field.

This model is assuming that the action space has been discretized: to navigate from the initial location to the target location, the agent jumps from node to node, assuming that a lower-level motion planner can effectively control the robot between the two nodes, notably by avoiding collisions. At first, we also assume that the model has access to an overview scan of the environmnent; but, we will relax this hypothesis afterwards.

Introduction

In vision-and-language navigation (VLN), an agent is asked to navigate in home environments following natural language instructions [START_REF] Anderson | On evaluation of embodied navigation agents[END_REF][START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF]. This task is attractive to many real-world applications such as domestic robotics and personal assistants. However, given the high diversity of VLN data across environments and the difficulty of the manual collection and annotation of VLN training data at scale, the performance of current methods remains limited, especially for previously unseen environments [START_REF] Zhang | Diagnosing the environment bias in vision-and-language navigation[END_REF].

This chapter is motivated by significant improvements in vision and language pretraining [START_REF] Alberti | Fusion of detected objects in text for visual question answering[END_REF][START_REF] Chen | Uniter: Universal image-text representation learning[END_REF][START_REF] Li | Oscar: Object-semantics aligned pre-training for vision-language tasks[END_REF][START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Lu | 12-in-1: Multi-task vision and language representation learning[END_REF][START_REF] Su | Vl-bert: Pre-training of generic visual-linguistic representations[END_REF], where deep transformer models [START_REF] Vaswani | Attention is all you need[END_REF] are trained via self-supervised proxy tasks [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] using large-scale, automatically harvested imagetext datasets [START_REF] Ordonez | Im2Text: Describing images using 1 million captioned photographs[END_REF][START_REF] Sharma | Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[END_REF]. Such pretraining enables learning transferable multi-modal representations achieving state-of-the-art performance in various vision and language tasks. Similarly, with the goal of learning an embodied agent that generalizes, recent works [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF][START_REF] Huang | Transferable representation learning in visionand-language navigation[END_REF][START_REF] Li | Robust navigation with language pretraining and stochastic sampling[END_REF][START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] have explored different pretraining approaches for VLN tasks.

In [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF][START_REF] Huang | Transferable representation learning in visionand-language navigation[END_REF], annotated path-instruction pairs are augmented with a speaker model that generates instructions for random unseen paths. However, as these paths originate from a small set of 61 houses used during training, they are limited in visual diversity. The limited pretraining environments do not equip agents with visual understanding abilities that enable generalization to unseen houses, see Fig. 3.1. To address this problem, VLN-BERT [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] proposes to pretrain the agent on generic image-caption datasets that are abundant and cover diverse visio-linguistic knowledge. However, these image-caption pairs are quite different from the dynamic visual stream (path) and navigable instructions observed by a VLN agent. Such out-of-domain pretraining, although promising, only brings limited gains to the navigation performance. Besides the above limitations, existing pretraining methods do not place much emphasis on temporal reasoning abilities in their proxy tasks such as one-step action prediction [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF] and path-instruction pairing [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF], while such reasoning is important to a sequential decision making task like VLN. As a result, even if performance in downstream tasks is improved, the pretrained models may still be brittle. For example, a simple corruption of instructions by swapping noun phrases within the instruction, or replacing them with other nouns, leads to significant confusion as models are unable to pick the correct original pair.

In this chapter, we explore a different data source and proxy tasks to address the above limitations in pretraining a generic VLN agent. Though navigation instructions are rarely found on the Internet, image-caption pairs from home environments are abundant in online marketplaces (e.g. Airbnb), which include images and descriptions of rental listings. We collect BnB, a new large-scale dataset with 1.4M indoor images and 0.7M captions. First, we show that in-domain image-caption pairs bring additional benefits for downstream VLN tasks when applied with generic web data [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF]. In order to further reduce the domain gap between the BnB pretraining and the VLN task, we present an approach to transform static image-caption pairs into visual paths and navigationlike instructions (Fig. 3.1 bottom), leading to large additional performance gains. We also propose a shuffling loss that improves the model's temporal reasoning abilities by learning a temporal alignment between a path and the corresponding instruction.

Our pretrained model, Airbert, is a generic transformer backbone that can be readily integrated in both discriminative VLN tasks such as path-instruction compatibility prediction [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] and generative VLN tasks [START_REF] Hong | A recurrent vision-and-language BERT for navigation[END_REF] in R2R navigation [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF] and REVERIE remote referring expression [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF]. We achieve state-of-the-art performance on these VLN tasks with our pretrained model. Beyond the standard evaluation, our in-domain pretraining opens an exciting new direction of one/few-shot VLN where the agent is trained on examples only from one/few environment(s) and expected to generalize to other unseen environments.

In summary, the contributions of this chapter are three-fold. [START_REF] Achlioptas | ReferIt3D: Neural listeners for fine-grained 3d object identification in real-world scenes[END_REF] We collect a new large-scale in-domain dataset, BnB, to promote pretraining for vision-and-language navigation tasks. [START_REF] Ahmed | CausalWorld: A robotic manipulation benchmark for causal structure and transfer learning[END_REF] We curate the dataset in different ways to reduce the distribution shift between pretraining and VLN and also propose the shuffling loss to improve temporal reasoning abilities. (3) Our pretrained Airbert can be plugged into generative or discriminative architectures and achieves state-of-the-art performance on R2R and REVERIE datasets. Moreover, our model generalizes well under a challenging one/fewshot VLN evaluation, truly highlighting the capabilities of our learning paradigm.

Related work

Vision-and-language navigation. VLN [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF] has received significant attention with a large number of followup tasks introduced in recent years [START_REF] Anderson | On evaluation of embodied navigation agents[END_REF][START_REF] Chen | Touchdown: Natural language navigation and spatial reasoning in visual street environments[END_REF]165,[START_REF] Ku | Room-Across-Room: Multilingual vision-and-language navigation with dense spatiotemporal grounding[END_REF][START_REF] Nguyen | Help, Anna! Visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning[END_REF][START_REF] Nguyen | Vision-based navigation with language-based assistance via imitation learning with indirect intervention[END_REF][START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF][START_REF] Shridhar | ALFRED: A benchmark for interpreting grounded instructions for everyday tasks[END_REF][START_REF] Thomason | Visionand-dialog navigation[END_REF]. Early days of VLN saw the use of sequence-to-sequence LSTMs to predict low-level actions [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF] or high-level directions in a panoramic action space [START_REF] Fried | Speaker-Follower models for vision-and-language navigation[END_REF].

Different attention mechanisms [212,[START_REF] Qi | Object-and-action aware model for visual language navigation[END_REF] are proposed to improve cross-modal alignment. Various reinforcement learning based training algorithms [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF][START_REF] Wang | Soft expert reward learning for vision-andlanguage navigation[END_REF][START_REF] Wang | Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation[END_REF][START_REF] Wang | Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation[END_REF] and searching algorithms in inference [START_REF] Fried | Speaker-Follower models for vision-and-language navigation[END_REF]212,[START_REF] Ma | The Regretful Agent: Heuristic-aided navigation through progress estimation[END_REF] have also been explored to improve the VLN performance.

To improve an agent's generalization to unseen environments, data augmentation is performed by using a speaker model [START_REF] Fried | Speaker-Follower models for vision-and-language navigation[END_REF] that generates instructions for random paths in seen environments, and environment dropout [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF] is used to mimic new environments. While pretraining LSTMs for transferable representations is adopted by [START_REF] Huang | Transferable representation learning in visionand-language navigation[END_REF], recently, there has been a shift towards transformer models [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF] to learn generic multimodal representations. This is further extended to a recurrent model that significantly improves sequential action prediction [START_REF] Hong | A recurrent vision-and-language BERT for navigation[END_REF]. However, the limited environments in pretraining [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF][START_REF] Huang | Transferable representation learning in visionand-language navigation[END_REF] constrain the generalization ability to unseen scenarios. The most related work, VLN-BERT [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] transfers knowledge from abundant, but out-of-domain image-text data to improve path-instruction matching. In this chapter, we not only create a large-scale, in-domain BnB dataset, but also propose effective pretraining strategies to mitigate the domain-shift between webly crawled image-text pairs and VLN data. Large-scale visio-linguistic pretraining. Thanks to large-scale image-caption pairs automatically collected from the web [START_REF] Miech | HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips[END_REF][START_REF] Ordonez | Im2Text: Describing images using 1 million captioned photographs[END_REF][START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF][START_REF] Sharma | Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[END_REF], visio-linguistic pretraining (VLP) has made great breakthroughs in recent years. Several VLP models [START_REF] Chen | Uniter: Universal image-text representation learning[END_REF][START_REF] Li | Oscar: Object-semantics aligned pre-training for vision-language tasks[END_REF][START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Tan | Lxmert: Learning cross-modality encoder representations from transformers[END_REF] have been proposed based on the transformer architecture [START_REF] Vaswani | Attention is all you need[END_REF]. These models are often pretrained with self-supervised objectives akin to those in BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]: masked language modeling, masked region modeling and vision-text pairing. Finetuning them on downstream datasets achieves state-of-the-art performance on various VL tasks [START_REF] Antol | VQA: Visual Question Answering[END_REF][START_REF] Kazemzadeh | Refer-ItGame: Referring to objects in photographs of natural scenes[END_REF][START_REF] Wang | Learning deep structure-preserving image-text embeddings[END_REF][START_REF] Vinyals | Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge[END_REF]. While such pretraining focuses on learning correlations between vision and text, it is not designed for sequential decision making as required in embodied VLN. The goal of this chapter is not to improve VLP architectures but to present in-domain training strategies that lead to performance improvements for VLN tasks.

BnB Dataset

Hosts that rent places on online marketplaces often upload attractive and unique photos along with descriptions. One such marketplace, Airbnb, has 5.6M listings from over 100K cities all around the world [START_REF] Airbnb | Airbnb fast facts[END_REF]. We propose to use this abundant and curated data for large-scale in-domain VLN pretraining. In this section, we first describe how we collect image-caption pairs from Airbnb. Then, we propose methods to transform images and captions into VLN-like path-instruction pairs to reduce the domain gap between webly crawled image-caption pairs and VLN tasks (see Fig. 3.3).

Collecting BnB Image-Caption Pairs

Collection process. We restrict our dataset to listings from the US (about 10% of Airbnb) to ensure high quality English captions and visual similarity with Matterport environments [START_REF] Chang | Matterport3D: Learning from RGB-D data in indoor environments[END_REF]. The data collection proceeds as follows: (1) obtain a list of locations Statistics. We downloaded almost 150k listings and their metadata (1/4 of the listings in the US) in step 3, leading to over 3M images and 1M captions. After data cleaning with steps 4 and 5, we obtain 713K image-caption pairs and 676K images without captions. Table 3.1 compares our BnB dataset to other datasets used in previous works for VLN (pre-)training. It is larger than R2R [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF], REVERIE [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF] and includes a large diversity of rooms and objects, which is not the case for Conceptual Captions [START_REF] Sharma | Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[END_REF]. We posit that such in-domain data is crucial to deal with the data scarcity challenge in VLN environments as illustrated in Fig. 3.1. We use 95% of our BnB dataset for training and the remaining 5% for validation.

Apart from images and captions, our collected listings contain structured data including a list of amenities, a general description, reviews, location, and rental price, which may offer additional applications in the future.

Filtering image-caption pairs: Outdoor images

Images of outdoor scenes are almost never seen in the environments used in downstream VLN tasks. In fact, not only are the images out-of-domain (such images are rarely seen in the VLN environments), their captions are often irrelevant to a VLN task. In order to alleviate the impact of such noisy images and captions, we discard outdoor images from the pretraining process. Figure 3.2 illustrates several examples of misleading outdoor image-caption pairs. Captions as written by the host are presented in the label below the image. The caption for the image in Figure 3.2a refers to a "bedroom", however, the image does not show a bedroom. Similarly, the image-caption pair in the Figure 3.2b talks about activities or festivals that take place in the neighborhood of the listing, however, they are not relevant for solving indoor navigation tasks. Finally, Figure 3.2c shows an outdoor scene with several birds along with a noisy caption that is not directly related to the image content, but the emotion that the image may evoke. [START_REF] Chang | Matterport3D: Learning from RGB-D data in indoor environments[END_REF] refers to the #panoramas. The speaker model [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF] generates instructions for randomly selected trajectories, but is limited to panoramas from 60 training environments. Note that the data from Conceptual Captions (ConCaps) may feature some houses, but it is not the main category.

Dataset

Creating BnB Path-Instruction Pairs

BnB image-caption (IC) pairs are complementary to Conceptual Captions (ConCaps) as they capture diverse VLN environments. However, they still have large differences from path-instruction (PI) pairs in VLN tasks. For example, during navigation, an agent observes a sequence of panoramic views rather than a single image, and the instruction may contain multiple sentences. To mitigate this domain gap, we propose strategies to automatically craft path-instruction pairs starting from BnB-IC pairs.

Concatenating Images and Texts in a BnB Listing

Images in a BnB listing usually depict different locations in a house, mimicking the sequential visual observations an agent makes while navigating in the house. To create a VLN-like path-instruction pair, we randomly select and concatenate K 4 image-caption pairs from the listing. In between each caption, we randomly add a word from "and", "then", "." or nothing to make the concatenated instruction more fluent and diverse.

Augmenting Paths with Visual Contexts

In the above concatenated path, each location only contains one BnB image, and perhaps with a limited view angle as hosts may focus on objects or amenities they wish to highlight. Therefore, it lacks the panoramic visual context at each location that the agent receives in real navigation paths. Moreover, each location in the concatenated instruction is described by a unique sentence, while adjacent locations are often expressed together in one sentence in VLN instructions [119]. To address the above issues with concatenation, we propose two approaches to compose paths that have more visual context and can also leverage the abundant images without captions (denoted as captionless images).

1. Image merging extends the panoramic context of a location by grouping images from similar room categories (see Fig. 3.3). For example, if the image depicts a kitchen sink, it is natural to expect images of other objects such as forks and knives nearby. Specifically, we first cluster images of similar categories (e.g. kitchen) using room labels predicted by a pretrained Places365 model [START_REF] Zhou | Places: A 10 million image database for scene recognition[END_REF]. Then, we extract multiple regions from this merged set of images, and use them as an approximation to the panoramic visual representation.

2. Captionless image insertion. Table 1 shows that half of the BnB images are captionless. Using them allows to increase the size of the dataset. When creating a path-instruction pair from the concatenation approach, a captionless image is inserted as if its caption was an empty string. The BnB PI pairs hence better approximate the distribution of the R2R path-instructions: (1) some images in the path are not described and (2) instructions have similar number of noun phrases.

Crafting Instructions with Fluent Transitions

The concatenated captions mainly describe rooms or objects at different locations, but do not contain any of the actionable verbs as in navigation instructions, e.g. "turn left at the door" or "walk straight down the corridor". We suggest two strategies to create fake instructions that have fluent transitions between sentences.

1. Instruction rephrasing. We use a fill-in-the-blanks approach to replace noun-phrases in human annotated navigation instructions [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF] by those in BnB captions (see Fig. 3.3). Concretely, we create more than 10K instruction templates containing 2-7 blanks, and fill the blanks with noun-phrases extracted from BnB captions. The noun-phrases matched to object categories from the Visual Genome [START_REF] Krishna | Visual Genome: Connecting language and vision using crowdsourced dense image annotations[END_REF] dataset are preferred during selection. This allows us to create VLN-like instructions with actionable verbs interspersed with room and object references for visual cues that are part of the BnB path (see Fig. 3.3).

2. Instruction generation is a video captioning like model that takes in a sequence of images and generates an instruction corresponding to an agent's path through an environment. To train this model, we adopt ViLBERT and train it to generate captions for single BnB image-caption pairs. Further, this model is fine-tuned on trajectories of the R2R dataset to generate instructions. Finally, we use this model to generate BnB PI pairs by producing an instruction for a concatenated image sequence from BnB (the path).

Dataset details and Statistics

BnB image-caption pairs. We collect BnB IC pairs from 150K listings on Airbnb resulting in 713K image-caption pairs and 676K images without captions. In Figure 3.4, we present some key statistics about this data. Figure 3.4a presents a histogram of the number of images found in each listing. While most listings have less than 20 images, this is still a sufficiently large and diverse in-domain distribution. In Figure 3.4b, we summarize the rooms depicted in the images through predicted category labels obtained using a CNN trained on the Places365 dataset [START_REF] Zhou | Places: A 10 million image database for scene recognition[END_REF]. These category labels are used as part of our proposed extensions such as image merging.

Creating BnB path-instruction pretraining samples. We create the BnB PI pairs on-the-fly during training to mimic the agent's visual trajectory and a corresponding instruction through an environment. Each sample in a batch is created by randomly sampling a listing without replacement during an epoch (one epoch consists of one PI pair from each listing). Then, the number of IC pairs K that form the PI pair are chosen (as an integer) from a uniform distribution, K ∼ U [START_REF] Akkaya | Solving rubik's cube with a robot hand[END_REF][START_REF] Bou Ammar | Online multi-task learning for policy gradient methods[END_REF]. We sample N ∼ U[2, K] IC pairs that have a non-empty caption and the remainder K -N images are chosen from the set of captionless images. Any image in the path may include additional visual context (from the same room) via the image merging strategy. Similarly, the instruction rephrasing strategy may be employed by using existing R2R instruction templates and filling them with noun phrases extracted from the image captions.

The above procedure results in creating one correctly aligned (positive) PI pair, X + . To employ the shuffling loss for each sample, we create 9 additional negatives, X - n , by shuffling either the sequence of images or captions, ensuring that the post-shuffling order does not align with the positive pair. Statistics for BnB PI pairs. Due to the large number of possible combinations, we can (theoretically) create 200 billion path-instruction pairs, using the simple concatenation strategy. This number grows to over 300 quadrillion when considering additional visual context augmentations and fluent instructions.

For instruction rephrasing, we create 11,626 fill-in-the-blank templates from the R2R training set. Figure 3.4c shows the distribution of the number of blanks in the templates -most instruction templates have 2-7 blanks into which we insert noun phrases from the BnB captions. While we are unable to generate the entire BnB PI dataset for computing statistics, we generate 50K PI pairs as a representative sample. Figure 3.4d presents the distribution of instruction lengths (number of words) for different datasets. We see that the captions in BnB IC pairs are much shorter than typical instructions in R2R and REVERIE, while our automatically created instructions in BnB PI pairs exhibit a high level of similarity in terms of their length. Among the methods to create an instruction, simple concatenation lacks action verbs between sentences for fluent transition leading to a domain shift from real instructions. Instruction rephrasing selects noun phrases from BnB image descriptions and inserts them into real instruction templates, providing a natural feel to the created instruction. Finally, while the learning approach of instruction generation (recall, this is learned on downstream VLN dataset) produces fluent sentences, it is unable to leverage the diverse captions of BnB images due to the limited vocabulary stemming from the downstream VLN dataset. For example, the generated instruction in Figure 3.5c does not contain noun phrases related to images in the path. Better caption generation models such as Pointer network [START_REF] Vinyals | Pointer networks[END_REF] may help avoid such problems, however are left for future work.

Examples of BnB PI Pairs

Among augmentations for path generation, we can see that image merging helps to expand relevant visual context from single images to semi-panoramic views, see the bedroom in Figure 3.5a or the kitchen in Figure 3.5b. Captionless image insertion also improves the path diversity by mimicking unmentioned viewpoints in the instruction (indicated by images with a dotted border).

Airbert: A Pretrained VLN Model

In this section, we present Airbert, our multi-modal transformer pretrained on the BnB dataset with masking and shuffling losses. We first introduce the architecture of Airbert, and then describe datasets and pretext tasks in pretraining. Finally, we show

Embed-Vis Co-TRM-Vis TRM-Vis Embed-Lang Co-TRM-Lang TRM-Lang [IMG], 𝑉 ! , ⋯ , [IMG], 𝑉 " [CLS], 𝐶, [SEP] TRM-Lang 𝐿 ! layers alignment score 𝐿 # layers (a)
Adapting Airbert to a discriminative setting to predict path-instruction alignment score, similar to [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF].

Embed-Vis

Co-TRM-Vis TRM-Vis Embed-Lang TRM-Lang how Airbert can be adapted to downstream VLN tasks.

𝑠 $%! , 𝑉 $ [CLS], 𝐶, [SEP] TRM-Lang initialization 𝑠 & , 𝐻 navigation 𝑠 $ , 𝑝 $ 𝐿 # layers 𝐿 # layers 𝐿 ! layers

ViLBERT-like Architecture

ViLBERT [START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF] is a multi-modal transformer extended from BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] to learn joint visio-linguistic representations from image-caption pairs, as illustrated in Fig. 3.7.

Given an image-caption pair (V,C), the model encodes the image as region features [v 1 , . . . , v V ] via a pretrained Faster R-CNN [START_REF] Anderson | Bottom-up and top-down attention for image captioning and visual question answering[END_REF], and embeds the text as a series of tokens:

[[CLS], w 1 , . . . , w T , [SEP]]
, where [CLS]and [SEP]are special tokens added to the text. ViLBERT contains two separate transformers that encode V and C and it learns cross-modal interactions via co-attention [START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF].

We follow a similar strategy to encode path-instruction pairs (created in Sec. 3.3.3) that contain multiple images and captions

{(V k ,C k )} K k=1 .
Here, each V k is represented as visual regions v k i and C k as word tokens w k t . Respectively, the visual and text inputs to Airbert are:

X V = [[IMG], v 1 1 , . . . , v 1 V 1 , . . . , [IMG], v K 1 , . . . , v K V K ], (3.1)
X C = [[CLS], w 1 1 , . . . , w 1 T 1 , . . . , w K 1 , . . . , w K T K , [SEP]], (3.2)
where the [IMG] token is used to separate image region features taken at different locations.

Note that while our approach is not limited to a ViLBERT-like architecture, we choose ViLBERT for a fair comparison with previous work [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF]. We fine-tune Airbert on downstream VLN tasks using both discriminative or generative models (panel 3).

Datasets and Pretext Tasks for Pretraining

We use Conceptual Captions (ConCaps) [START_REF] Sharma | Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[END_REF] and BnB-PI in subsequent pretraining steps (see Fig. 3.7) to reduce the domain gap for downstream VLN tasks. Previous multi-modal pretraining efforts [START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF][START_REF] Huang | Transferable representation learning in visionand-language navigation[END_REF] commonly use two selfsupervised losses given image-caption (IC) pairs or path-instruction (PI) pairs: (1) Masking loss: An input image region or word is randomly replaced by a [MASK] token. The output feature of this masked token is trained to predict the region label or the word given its multi-modal context. (2) Pairing loss: Given the output features of [IMG]and [CLS] tokens, a binary classifier is trained to predict whether the image (path) and caption (instruction) are paired.

The above two pretext tasks mainly focus on learning object-word associations instead of reasoning about the temporal order of paths and instructions. For example, if an image V i appears before V j , then words from its caption C i should appear before C j . In order to promote such a temporal reasoning ability, we propose an additional shuffling loss to enforce alignment between PI pairs. Given an aligned PI pair

X + = {(V k ,C k )} K k=1 , we generate N negative pairs X - n = {(V k ,C l )}, k ̸ = l,
by shuffling the composed images or the captions. We train our model to choose the aligned PI pair as compared to the shuffled negatives by minimizing the cross-entropy loss:

L = -log exp( f (X + )) exp( f (X + )) + ∑ n exp( f (X - n )) , (3.3)
where f (X) denotes the similarity score (logit) computed via Airbert for the PI pair X.

Adaptations for Downstream VLN tasks

We consider two VLN tasks: goal-oriented navigation (R2R [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF]) and object-oriented navigation (REVERIE [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF]). Airbert can be readily integrated in discriminative and generative models for the above VLN tasks. Discriminative Model: Navigation as Path-Selection [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF]. The navigation problem on the R2R dataset is formulated as a path selection task in [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF]. Several candidate paths are generated via beam search from a navigation agent such as [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF], and a discriminative model is trained to choose the best path among them. We fine-tune Airbert on the R2R dataset for path selection. A two-stage fine-tuning process is adopted: in the first phase, we use masking and shuffling losses on the PI pairs of the target VLN dataset in a manner similar to BnB PI pairs; in the second phase, we choose a positive candidate path as one that arrives within 3m of the goal, and contrast it against 3 negative candidate paths. We also compare multiple strategies to mine additional negative pairs (other than the 3 negative candidates), and in fact, empirically show that negatives created using shuffling outperform other options.

Generative Model: Recurrent VLN-BERT [START_REF] Hong | A recurrent vision-and-language BERT for navigation[END_REF]. The Recurrent VLN-BERT model adds recurrence to a state in the transformer to sequentially predict actions, achieving state-of-the-art performance on R2R and REVERIE tasks. We use our Airbert architecture as its backbone and apply it to the two tasks as follows. First, the language transformer encodes the instruction via self-attention. Then, the embedded [CLS] token in the instruction is used to track history and concatenated with visual tokens (observable navigable views or objects) in each action step. Self-attention and cross-attention on embedded instructions are employed to update the state and visual tokens and the attention score from the state token to visual tokens is used to decide the action at each step. We fine-tune the Recurrent VLN-BERT model with Airbert as the backbone in the same way as [START_REF] Hong | A recurrent vision-and-language BERT for navigation[END_REF].

Experimental Results

We first perform ablation studies evaluating alternative ways to pretrain Airbert in Sec. 3.5.4. Then, we compare Airbert with state-of-the-art methods on R2R and REVERIE tasks in Sec. 3.5.5. Finally, in Sec. 3.5.8, we evaluate models in a more challenging setup: VLN few-shot learning where an agent is trained on examples taken from one/few houses.

Implementation details

We present the implementation details for learning Airbert via pretraining using BnB, and subsequent fine-tuning in both discriminative or generative settings.

Airbert Pretraining

Airbert's architecture is the same as VLN-BERT (see Figure 3.6a where the number of layers L 1 = L 2 = 6). The feature vector v k i (corresponding to ith image region of the kth image) is composed of three terms: the first term is the visual feature extracted by the Bottom-Up Top-Down attention model [START_REF] Anderson | Bottom-up and top-down attention for image captioning and visual question answering[END_REF]; the second term encodes the location of the region in the image as MLP(l k i ), where l k i is the 5-dim location vector of the given image region defined as the top corner (x, y), the width, height and area; and the last term Emb(k) encodes the position, where Emb is an embedding layer for the image order.

We use 8 V100 SXM2 GPUs (32 GB each) for pretraining Airbert. The model is trained for 15 epochs with a batch size of 64 and learning rate of 4 × 10 -5 . Each epoch consists of one randomly sampled PI pair from 95% of the listings, while the remaining 5% are used for validation and preventing overfitting.

Fine-tuning in Discriminative Setting

In the discriminative setting, R2R navigation is formulated as a path selection problem given the instruction. The pretrained Airbert model can be directly fine-tuned without any modifications to the architecture to predict the path-instruction alignment (or compatibility) score as shown in Figure 3.6a.

We follow the same fine-tuning setup as VLN-BERT [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] to allow for a fair comparison. We use the Adam optimizer with a learning rate of 4 × 10 -5 . The optimizer is controlled by a learning rate scheduler with a linear warmup and cooldown. We fine-tune Airbert for 30 epochs with a batch size of 64. Samples from the R2R training set are used for fine-tuning and the model checkpoint with the highest success rate on the unseen validation set (val unseen) is selected for the test set and leaderboard submission.

Fine-tuning in Generative Setting

In the generative setting, an agent is required to predict navigable actions step by step. We adopt the state-of-the-art generative model Recurrent VLN-BERT [START_REF] Hong | A recurrent vision-and-language BERT for navigation[END_REF] for R2R and REVERIE tasks. The model uses a pretrained multimodal transformer as a backbone and adds recurrence to a state token to keep track of history for sequential action prediction. Although the original Recurrent VLN-BERT model only implements an LXMERT-like [START_REF] Tan | Lxmert: Learning cross-modality encoder representations from transformers[END_REF] architecture PREVALENT [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF], and one-stream BERT-like architecture OSCAR [START_REF] Li | Oscar: Object-semantics aligned pre-training for vision-language tasks[END_REF], it is easy to plug our two-stream ViLBERT architecture as the backbone.

The adapted model is shown in Figure 3.6b. For initialization, the language stream is used to encode the instruction C into an instruction representation H. As no visual inputs are used during the initialization, the co-attention modules in the original language stream of ViLBERT are removed, and the output feature of the [CLS] token is used as the agent's initial state s 0 . For navigation at each step k, the visual stream takes the previous state s k-1 , visual observations V k at step k and the encoded language features H to generate a new state s k and action decision p k .

When fine-tuning on the R2R dataset, we use scene features with a ResNet-152 pretrained on Places365 [START_REF] Zhou | Places: A 10 million image database for scene recognition[END_REF] and augment the training data with generated pathinstruction pairs from [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF]. We train the model via imitation learning and A2C reinforcement learning for 300,000 iterations with a batch size of 16 and learning rate of 10 -5 . When fine-tuning on the REVERIE dataset, object features encoded by a Bottom-Up Top-Down attention model [START_REF] Anderson | Bottom-up and top-down attention for image captioning and visual question answering[END_REF] are used along with the scene features. The model is trained for 200,000 iterations with a batch size of 8. All the experimental setups for fine-tuning are the same as [START_REF] Hong | A recurrent vision-and-language BERT for navigation[END_REF] for a fair comparison.

Datasets and metrics

R2R Setup. Most of our experiments are conducted on the R2R dataset [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF], where we adopt standard splits and metrics defined by the task. We focus on success rate (SR), which is the ratio of predicted paths that stop within 3m of the goal. Please refer to [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] for a more detailed explanation of the metrics. In particular, as the discriminative model uses path selection for R2R, we follow the pre-explored environment setting adopted by VLN-BERT [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF]. REVERIE Setup. We also adopt standard splits and metrics on the REVERIE task [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF]. Here, the success rate (SR) is the ratio of paths for which the agent stops at a viewpoint where the target object is visible. Remote Grounding Success Rate (RGS) measures accuracy of localizing the target object in the stopped viewpoint, and RGS per path length (RGSPL) is a path length weighted version.

Pretraining with BnB

We perform ablation studies on the impact of various methods for creating pathinstruction pairs. We also present ablation studies that highlight the impact of using the shuffling loss during Airbert's pretraining as well as fine-tuning stages. Throughout this section, our primary focus is on the SR on the unseen validation set and we compare our results against VLN-BERT [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF], which achieves a SR of 59.26%.

1. Impact of creating path-instruction pairs. Table 3.2 presents the performance of multiple ways of using the BnB dataset after ConCaps pretraining as illustrated in Fig. 3 Naive concatenation (row 2) does only slightly better than using the IC pairs (row 1) as there are still domain shifts with respect to fluency of transitions and lack of visual context. Rows 3-6 show that each method mitigates domain-shift to some extent. Instruction rephrasing (row 3) performs better at improving instructions than instruction generation (row 4), possibly since the generator is unable to use the diverse vocabulary of the BnB captions. Inserting captionless images at random locations (row 6) reduces the domain-shift significantly and achieves the highest individual performance. Finally, a combination of instruction rephrasing, image merging and captionless insertion provides an overall 3.8% improvement over concatenation, and a large 7.2% improvement over VLN-BERT.

2. Shuffling loss applied during pretraining. Table 3.3 demonstrates that shuffling is an effective strategy to train the model to reason about temporal order, and enforce alignment between PI pairs. Rows 3-5 show that shuffling is beneficial both during pretraining with BnB-PI data, or during fine-tuning with R2R data, and results in 2.3% and 0.4% improvements respectively. In combination with the Speaker dataset (paths from seen houses with generated instruction yielding 178K additional PI pairs [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF]), we see that the shuffling loss provides 3.1% overall improvement (row 6 vs. 7). The BnB-PI data brings more improvements than the Speaker dataset (row 2 vs. 5). Putting together the BnB-PI data, Speaker dataset and shuffling, we achieve 68.67% SR on the R2R dataset with a single model. R2R involves fine-tuning to rank multiple candidate paths that form the path selection task. We compare various approaches to improve this fine-tuning procedure (results in Table 3.4). (1) In row 2, we explore the impact of using additional negative paths. Unsurprisingly, this does not improve performance. (2) Inspired by [START_REF] Gupta | Contrastive learning for weakly supervised phrase grounding[END_REF], we highlight keywords in the instruction using a part-of-speech tagger [START_REF] Joshi | Extending a parser to distant domains using a few dozen partially annotated examples[END_REF], and include an extra loss term that encourages the model to pay attention to their similarity scores (row 3).

Shuffling

(3) Another alternative suggested by [START_REF] Gupta | Contrastive learning for weakly supervised phrase grounding[END_REF] involves masking keywords in the instruction and using VLP models to suggest replacements, resulting in hard negatives (row 4).

Hard negatives and highlighting keywords improve performance by 2.1-2.3%, but at the cost of extra parsers or VLP models. In contrast, shuffling visual paths to create two additional negatives results in highest improvement (row 5, +2.7% on val unseen) and appears to be a strong strategy to enforce temporal order reasoning, that neither requires external parsers nor additional VLP models.

4. Error analysis. We study the areas in which Airbert brings major improvements by analyzing scores for aligned PI pairs and simple corruptions that involve replacing noun phrases (e.g. bedroom by sofa), swapping noun phrases appearing within the instruction, or switching left and right directions (e.g. turn left/right or leftmost/rightmost chair). In particular, for every ground-truth aligned PI pair, we create 10 additional negatives by corrupting the instruction, and measure the accuracy of the model selecting the correct pair. Table 3.5 shows that Airbert with in-domain training and the shuffling loss achieves large improvements (> 8%) for corruptions involving replacement or swapping of noun phrases. On the other hand, distinguishing directions continues to be a challenging problem; but here as well we see Airbert outperform VLN-BERT by 4.5%.

Model

Comparison against state-of-the-art

R2R. We first evaluate the discriminative model for the R2R task. Similar to VLN-BERT, we evaluate Airbert as an ensemble model created by a linear combination (chosen through grid search) of multiple model outputs (see Table 3.6). First, we see that Airbert alone (row 2) outperforms VLN-BERT (row 1) by 9.4% on the unseen environments and a strong ensemble of speaker and follower models [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF] (row 3) by 0.7%. Ensembling Airbert results in a gain of 1.4% over the VLN-BERT ensemble (row 4 vs. 5).

We also obtain results on the test set by submitting our best method to the R2R leaderboard 5 . As seen from Table 3.8, our method of ensembling Airbert, speaker, and follower (similar to VLN-BERT with speaker and follower [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]) achieves the highest success rate at 77% and is ranked first as of the submission deadline. Both VLN-BERT and Airbert use 30 candidate trajectories sampled by beam search with EnvDrop [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF], inducing the same path length (PL) for the three methods. As the SPL metric on the leaderboard takes into account the total path length over the 30 trajectories, the SPL is very low and similar across the approaches. Airbert also benefits generative models for the R2R task. REVERIE. Table 3.7 presents results for the REVERIE dataset. The last four rows in the table use Recurrent VLN-BERT [START_REF] Hong | A recurrent vision-and-language BERT for navigation[END_REF] with different backbones or parameter initialization. The OSCAR and ViLBERT backbones are pretrained on out-of-domain image-caption pairs. As compared to OSCAR, we observe slight improvements using the ViLBERT backbone for the REVERIE task. VLN-BERT shares the same architecture as ViLBERT, but is pretrained on the R2R dataset, resulting in performance improvement on the unseen environments. Our pretrained Airbert achieves significantly better performance than VLN-BERT, with over 2.4% gain on navigation SR and 1.8% gain on RGS in unseen environments (val unseen). Without any special adaptation, we see that Airbert brings benefits from pretraining on the BnB dataset. We also achieve the state-of-the-art performance on the REVERIE test set by the time of submission, surpassing previous works by a large margin.

Results on R2R with Generative Models

Table 3.10 shows the performance of different generative models on the R2R dataset. The OSCAR and ViLBERT backbones for Recurrent VLN-BERT [START_REF] Hong | A recurrent vision-and-language BERT for navigation[END_REF] (Rec) are all pretrained on large-scale out-of-domain image-caption pairs with object features and similar self-supervised tasks. On the other hand, the PREVALENT [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF] backbone is pretrained on in-domain R2R dataset with scene features and fine-tuned with an additional action prediction task. We suspect that this is the reason for PREVALENT's higher performance as compared to using OSCAR or VLN-BERT as backbones. Note that our Airbert backbone is not fine-tuned further on downstream tasks after pretraining.

Replacing OSCAR's single BERT-like architecture with the ViLBERT architecture slightly improves the performance (similar to our results on the REVERIE dataset presented in the Table 3.7). The VLN-BERT model further fine-tunes ViLBERT on the R2R dataset (with the masking loss). This is beneficial to the navigation performance on the unseen environments validation set 6 . Our Airbert initialization achieves substantial performance improvement as compared to the OSCAR and VLN-BERT backbones on unseen environments, while achieving comparable performance with the PREVALENT initialization.

Qualitative results

We visualize the predicted paths from VLN-BERT and Airbert models. In the following figures, • is the starting viewpoint of the agent, ■ denotes viewpoints in the ground-truth path, ■ for VLN-BERT and ■ for Airbert. Arrows indicate the navigation direction. New houses. In Figure 3.8, we compare predicted paths from VLN-BERT and Airbert in new houses beyond the training environments. Benefiting from BnB dataset that provides diverse visual environments in pretraining, our Airbert model generalizes better to recognize different room types in new houses (see Figure 3.8a-3.8d), and performs better on significantly different environments such as a church (Figure 3.8e) or castle (Figure 3.8f). New objects. Airbert also improves the understanding of new objects in home environments, e.g. through noun phrases related to household objects. As shown in Figure 3.9, it is successful at following instructions containing noun phrases that rarely occur or are even unseen on the training set, while the VLN-BERT model that is trained on a large image-caption corpus not pertaining to houses fails. Similar environments and instructions. Figure 3.10 displays examples where the environments and the instructions are similar to those on the training set, with the aim to show that the shuffling loss in pretraining also benefits learning. For example, in Figure 3.10a, the VLN-BERT agent ■ focuses on the stairs (in the last step) and

Methods

Validation goes upstairs incorrectly, whereas Airbert learns to consider intermediate steps such as "lounge chairs" and "cabinet" besides the last step by learning from the shuffling task. Similarly, in Figure 3.10c, we see that the VLN-BERT agent stops at the wrong stairs, while Airbert considers intermediate steps such as "hallway" and "wooden doors", and ends within the acceptable range of 3m from the goal. Failure cases. Figure 3.11 presents some failure cases for both VLN-BERT and Airbert. It reveals that current models still struggle to deal with relationships such as "between" (Figure 3.11a), or directional instructions such as "on the left" (Figure 3.11b). Similar failures are also highlighted by Table 3.5 where we show that models fail to choose the correct instruction when a direction keyword (left/right) is switched.

Training a navigation agent on few houses

We hypothesize that in-domain pretraining, especially one that leverages proposed PI pair generation methods, can achieve superior performance while requiring less training data. To evaluate this, we propose a novel few shot evaluation paradigm for VLN: models are allowed to fine-tune on samples (PI pairs) from one (or few) environments. Few-shot learning for VLN is particularly interesting as visual appearance of houses may differ vastly across geographies, and while training data is hard to obtain, pretraining data like BnB may be readily available.

One/few shot tasks. We considered two types of setups: (1) learning from a single environment, which we refer as one-shot learning; and (2) learning from 6 environments (representing 10% of the total training size). For both cases, we randomly sample 5 sets of environments, and report average results. As the number of paths in an environment may have a large impact on performance, we exclude 17 of 61 environments with less than 80 paths.

Results. We adopt VLN-BERT, pretrained on ConCaps, as a baseline for few-shot tasks. Recall that fine-tuning VLN-BERT and Airbert on R2R relies on candidate paths drawn from an existing model (EnvDrop [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF]). However, as this would lead to unfair comparisons (EnvDrop is trained on the full dataset), candidate paths are sampled as the shortest path between two random positions. Table 3.11 shows that Airbert largely outperforms VLN-BERT on the unseen validation set: 27.6% with 1 house and 22% with 6 houses. Airbert fine-tuned on 6 houses is almost as good as VLN-BERT on the entire training set. The last two rows of the table shows that using random paths does not lead to a large performance drop for both models and is a testament to the power of pretrained networks.

Conclusion

We introduced BnB, a large-scale, in-domain, image-text dataset from houses listed on online rental marketplaces and showed how domain gaps between BnB image-caption pairs and VLN tasks can be mitigated through the creation of path-instruction pairs. We also proposed shuffling, as a means to improve an agent's reasoning about temporal order. Our pretrained model Airbert, achieved state-of-the-art on R2R through the discriminative path-selection setting, and REVERIE through a generative setting. We also demonstrated large performance improvements when applying our model to a challenging one/few-shot VLN setup, highlighting the impact of good pretraining in VLN tasks.

History Aware Multimodal Transformer for Vision-and-Language Navigation

Vision-and-language navigation aims to build autonomous visual agents that follow instructions and navigate in real scenes. To remember previously visited locations and actions taken, most approaches to VLN implement memory using recurrent states, as in Chapter 3.

In this Chapter, we introduce a History Aware Multimodal Transformer (HAMT) to incorporate a long-horizon history into multimodal decision making. HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history. It, then, jointly combines text, history and current observation to predict the next action. We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction, and then use reinforcement learning to further improve the navigation policy.

HAMT achieves new state of the art on a broad range of VLN tasks, including VLN with fine-grained instructions (R2R, RxR), high-level instructions (R2R-Last, REVERIE), dialogs (CVDN) as well as long-horizon VLN (R4R, R2R-Back). We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories.

Introduction

Vision-and-language navigation (VLN) has recently received growing attention [START_REF] Anderson | On evaluation of embodied navigation agents[END_REF][START_REF] Chen | Touchdown: Natural language navigation and spatial reasoning in visual street environments[END_REF][START_REF] Jain | Stay on the path: Instruction fidelity in vision-and-language navigation[END_REF][START_REF] Zhang | Diagnosing the environment bias in vision-and-language navigation[END_REF][START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF]. VLN requires an agent to understand natural language instructions, perceive the visual world, and perform navigation actions to arrive at a target location. A number of datasets have been proposed to support various VLN tasks such as indoor

Q K V V K Q Q K V Q K V Cross-modal Transformer Layer & 4 [, ' 4 ; . ' 4 ]
Hierarchical Vision Transformer Vision Transformer Look-up Table and outdoor navigation with fine-grained instructions [START_REF] Chen | Touchdown: Natural language navigation and spatial reasoning in visual street environments[END_REF][START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Ku | Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding[END_REF], language-driven remote object finding [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF] and navigation in dialogs [START_REF] Thomason | Visionand-dialog navigation[END_REF].

! ./0 # !! " " # !#$! " " # !! % " # !& % " &′ = () ./0 4 , ) # 4 , ⋯ , ) " 4 ) , ' 4 = (ℎ ./0 4 , ℎ # 4 , ⋯ , ℎ '(# 4 ) . ' 4 = (/ # 4 , ⋯ , / - 4 , / 0123
VLN agents are faced with several challenges. First, as opposed to static vision-text grounding [START_REF] Yu | Mattnet: Modular attention network for referring expression comprehension[END_REF], the agent continuously receives new visual observations and should align them with instructions. Most of existing works adopt recurrent neural networks (RNNs) [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Fried | Speaker-Follower models for vision-and-language navigation[END_REF][START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF]212,[START_REF] Wang | Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation[END_REF][START_REF] Hong | Language and visual entity relationship graph for agent navigation[END_REF][START_REF] Lin | Scene-intuitive agent for remote embodied visual grounding[END_REF] to encode historical observations and actions within a fixed-size state vector to predict the next action. Such condensed states might be sub-optimal for capturing essential information in extended trajectories [START_REF] Fang | Scene memory transformer for embodied agents in long-horizon tasks[END_REF]. For instance, "bring the spoon to me" requires the agent to remember its start location after navigating to the "spoon", while early memories are prone to fade in the recurrent state. Few endeavors [START_REF] Deng | Evolving graphical planner: Contextual global planning for vision-and-language navigation[END_REF][START_REF] Wang | Structured scene memory for vision-language navigation[END_REF] construct external map-like memories for received observations. Nevertheless, these approaches still rely on RNNs to track the navigation state. As the history plays an important role in environment understanding and instruction grounding, we propose to explicitly encode the history as a sequence of previous actions and observations instead of using recurrent states.

Another VLN challenge concerns the generalizations of agents to new environments that have not been observed during training [START_REF] Zhang | Diagnosing the environment bias in vision-and-language navigation[END_REF]. One direction is to learn more generic text-image representations. The PRESS model [START_REF] Li | Robust navigation with language pretraining and stochastic sampling[END_REF] improves language representation with a pretrained BERT encoder [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], and PREVALENT [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF] uses pairs of instruction and single-step observations to pretrain a multimodal transformer. Though achieved promising results, these works do not optimize visual representation for the target navigation task. Moreover, lack of history in training [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF] makes it hard to learn cross-modal alignment and increases the risk of overfitting to training environments. Another direction towards better generalization is to overcome exposure bias [256] due to discrepancy between training and inference. Different methods have been adopted for VLN including DAgger [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Ross | A reduction of imitation learning and structured prediction to no-regret online learning[END_REF] and scheduled sampling [START_REF] Li | Robust navigation with language pretraining and stochastic sampling[END_REF][START_REF] Bengio | Scheduled sampling for sequence prediction with recurrent neural networks[END_REF]. Reinforcement Learning (RL) [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF][START_REF] Richard | Reinforcement learning: An introduction[END_REF] is one of the most effective approach among them, but it is considered unstable to directly train large-scale transformers via RL [START_REF] Parisotto | Stabilizing transformers for reinforcement learning[END_REF].

To address the above challenges, we propose the History Aware Multimodal Transformer (HAMT), a fully transformer-based architecture for multimodal decision making in VLN tasks. As illustrated in Figure 4.1, HAMT consists of unimodal transformers for text, history and observation encoding, and a cross-modal transformer to capture long-range dependencies of the history sequence, current observation and instruction. Since our history contains a sequence of all previous observations, its encoding is computationally expensive. To resolve complexity issues, we propose a hierarchical vision transformer as shown in Figure 4.2, which progressively learns representations for a single view, spatial relationships among views within a panorama and, finally, the temporal dynamics across panoramas of the history. In order to learn better visual representations, we propose auxiliary proxy tasks for end-to-end training. Such tasks include single-step action prediction based on imitation learning, self-supervised spatial relationship reasoning, masked language and image predictions and instruction-trajectory matching. We empirically show that our training facilitates the subsequent fine-tuning of our model with RL [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. We carry out extensive experiments on various VLN tasks, including VLN with fine-grained instructions (R2R [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF] and RxR [START_REF] Ku | Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding[END_REF]), high-level instructions (REVERIE [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF] and our proposed R2R-Last), dialogs [START_REF] Thomason | Visionand-dialog navigation[END_REF] as well as long-horizon VLN (R4R [START_REF] Jain | Stay on the path: Instruction fidelity in vision-and-language navigation[END_REF] and our proposed R2R-Back which requires the agent to return back after arriving at the target location). HAMT outperforms state of the art on both seen and unseen environments in all the tasks.

We summarize our contributions as follows: [START_REF] Achlioptas | ReferIt3D: Neural listeners for fine-grained 3d object identification in real-world scenes[END_REF] We introduce HAMT to efficiently model long-horizon history of observed panoramas and actions via hierarchical vision transformer; (2) We train HAMT with auxiliary proxy tasks in an end-to-end fashion and use RL to improve the navigation policy; (3) We validate our method and outperform state of the art in a diverse range of VLN tasks, while demonstrating larger gains for long-horizon navigation.

Related work

Vision-and-language navigation. Training instruction-following navigation agents has attracted increasing research attention [START_REF] Anderson | On evaluation of embodied navigation agents[END_REF][START_REF] Chen | Touchdown: Natural language navigation and spatial reasoning in visual street environments[END_REF][START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Ku | Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding[END_REF][START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF][START_REF] Shridhar | ALFRED: A benchmark for interpreting grounded instructions for everyday tasks[END_REF]. Anderson et al. [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF] propose a sequence-to-sequence LSTM baseline for the VLN task. Fried et al.

[85] extend it with panoramic action space and synthesized instructions. To improve cross-modal alignment, the self-monitoring agent [212] proposes co-grounding and progress estimation, and RelGraph [START_REF] Hong | Language and visual entity relationship graph for agent navigation[END_REF] uses graphs to model relationships across scene, objects and directions. Reinforcement learning (RL) is typically used to improve navigation policy. The EnvDrop model [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF] mixes imitation learning and A3C [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. The RCM [START_REF] Wang | Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation[END_REF] utilizes intrinsic reward of cross-modal matching in REINFORCE algorithm. Wang et al. [START_REF] Wang | Soft expert reward learning for visionand-language navigation[END_REF] propose to learn rewards via soft expert distillation. Due to the success of transformer [START_REF] Vaswani | Attention is all you need[END_REF], recent works explore transformer architectures in VLN. PRESS [START_REF] Li | Robust navigation with language pretraining and stochastic sampling[END_REF] replaces LSTM instruction encoder with pretrained BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]. SIA [START_REF] Lin | Scene-intuitive agent for remote embodied visual grounding[END_REF] uses transformer for single-step multimodal fusion and LSTM for sequential action prediction. PTA [START_REF] Landi | Perceive, transform, and act: Multi-modal attention networks for vision-and-language navigation[END_REF] is a transformer VLN model using CNNs to extract visual features [START_REF] He | Deep residual learning for image recognition[END_REF]. Here we propose the first full transformer architecture for VLN and train it end-to-end. Memory-based policy for navigation. LSTMs [START_REF] Hochreiter | Long short-term memory[END_REF] have been the dominant approach to encode memories for navigation [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Fried | Speaker-Follower models for vision-and-language navigation[END_REF][START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF][START_REF] Wang | Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation[END_REF]. Condensing all history into one feature vector, however, is prone to the loss of information. Alternative approaches include topological map memory structures [START_REF] Gupta | Cognitive mapping and planning for visual navigation[END_REF][START_REF] Savinov | Semi-parametric topological memory for navigation[END_REF]. Deng et al. [START_REF] Deng | Evolving graphical planner: Contextual global planning for vision-and-language navigation[END_REF] use graphs to capture environment layout and enable long-term planing. A similar graph is adopted in [START_REF] Wang | Structured scene memory for vision-language navigation[END_REF] with frontier-exploration based decision making. But these works still utilize LSTMs for state tracking. To exploit long-term spatio-temporal dependencies, Fang et al. [START_REF] Fang | Scene memory transformer for embodied agents in long-horizon tasks[END_REF] store histories in a sequence encoded with transformer. Recurrent VLN-BERT [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF] injects a recurrent unit to encode histories in transformer for VLN. The most similar work to ours is Episodic Transformer (E.T.) [START_REF] Pashevich | Episodic transformer for vision-and-language navigation[END_REF]. Differently from [START_REF] Pashevich | Episodic transformer for vision-and-language navigation[END_REF], we propose a hierarchical encoding of the panoramic observation history and optimize the whole model in end-to-end training.

Multimodal pretraining with transformers. Recent works show significant progress in vision and language tasks using multimodal pretraining. In particular, transformer architectures such as one-stream [START_REF] Chen | Uniter: Universal image-text representation learning[END_REF][START_REF] Li | Oscar: Object-semantics aligned pre-training for vision-language tasks[END_REF] and dual-stream [START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Tan | Lxmert: Learning cross-modality encoder representations from transformers[END_REF] achieve state of the art for a number of downstream tasks including visual question answering, image-text retrieval and image captioning. While most previous methods rely on CNN to extract image representations, ViLT [START_REF] Kim | Vilt: Vision-and-language transformer without convolution or region supervision[END_REF] adopts Vision Transformer (ViT) [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] and trains it with associated texts in an end-to-end manner thanks to the efficiency of ViT. A few endeavors [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF][START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] explore multimodal pretraining for VLN. PREVALENT [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF] pretrains a transformer using instructions and single-step observations without referring to trajectory history. VLN-BERT [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] measures the compatibility between an instruction and images in a path but does not support action prediction. This chapter presents the first end-to-end trainable VLN transformer that jointly encodes text, history and observation, and is able to sequentially predict actions.

Method

Problem definition The VLN problem [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF] is formulated as a partially observable Markov decision process, where future observations are independent of the past conditioning on current state s t . Given an instruction W containing a sequence of L words (w 1 , w 2 , • • • , w L ), an agent should follow the instruction to move in a connectivity graph to reach the goal location. At each step t, the agent receives an observation O t , a panorama of its surrounding environment. The O t consists of K single view images split from the panorama

O t ≜ ([v o 1 ; a o 1 ], • • • , [v o K ; a o K ])
, where v o i is the visual feature of the i-th view and a o i denotes the relative angle to face the view (subscript t is omitted for simplicity). There are n navigable viewpoints among all the K views1 , denoted as

O c t ≜ ([v c 1 ; a c 1 ], • • • , [v c n ; a c n ]
). We follow the setup in [START_REF] Fried | Speaker-Follower models for vision-and-language navigation[END_REF] and use O c t as the decision space, so the agent only needs to select a candidate in O c t at each step. All observations O i and performed actions a h i before step t form the history

H t ≜ ([O 1 ; a h 1 ], • • • , [O t-1 ; a h t-1 ]
), where a h i denotes the turned angles at step i. The goal is to learn a policy π parametrized by Θ to predict the next action based on the instruction, history and the current observation, which is π(a t |W , H t , O t , O c t ; Θ). Unlike dominant recurrent approaches to condense H t into a fixed-size vector, in this section, we present the History Aware Multimodal Transformer (HAMT) that jointly encodes text, long-horizon history, and observation for sequential action prediction. The model architecture is described in Section 4.3.1. We propose end-to-end training for HAMT in Section 4.3.2 to learn unimodal and multimodal representations, and then use RL to fine-tune the navigation policy in Section 4.3.3. Text Encoding. For each token i in the instruction W , we embed it as the summation of its word embedding w i , position embedding E P i and type embedding of text E T 0 . Then we employ a transformer with N L layers to obtain contextual representation x i following the standard BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] = (sin θ i , cos θ i , sin φ i , cos φ i ) where θ i and φ i are the relative heading and elevation angle to the agent's orientation. Then the observation embedding o i is as follows: [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] and E.T. [START_REF] Pashevich | Episodic transformer for vision-and-language navigation[END_REF] respectively. The flattened approach treats each view image in O i as a token, so the history sequence contains tK tokens. Though it enables to learn relationships among all image views, the computation cost quadratically increases with the sequence length, making it inefficient for long-horizon tasks. In the temporal-only approach, only the oriented view of the agent in each O i is taken as inputs instead of the whole panorama, so only t temporal tokens are encoded. However, this approach can lose critical information in past observations. For example, in the instruction "with the windows on your left, walk through the large room past the sitting areas", the object "window" does not appear in the oriented view of the agent. Therefore, the encoded history is insufficient to tell whether the agent passed the window or not, making the model confused to take the next action.

HAMT: History Aware Multimodal Transformer

o i = LN(W o v v o i ) + LN(W o a E A a o i ) + E N o i + E T 1 ( 4
In order to balance computational efficiency and information integrity, we propose a hierarchical history encoding approach as illustrated in Figure 4.2a. It hierarchically encodes view images within each panorama and then temporal relationships across panoramas, similar to the factorized spatial-temporal video transformer [START_REF] Arnab | Vivit: A video vision transformer[END_REF]. For each O i , its constituent view images are first embeded via ViT and Eq (4.1), and then encoded via a panoramic transformer with N h layers to learn spatial relationships within the panorama. We apply average pooling to obtain panorama embedding, and add it with the oriented view image feature in residual connection. The parameters in ViT and panoramic transformer are shared for different steps. In this way, each historical observation O i is represented as v h i , and the final temporal token h i is computed as:

h i = LN(W h v v h i ) + LN(W h a E A a h i ) + E S i + E T 2 (4.2)
where E S i denotes the i-th step embedding, E T 2 is the type embedding of history. The computational cost is O(tK 2 + t 2 ), which significantly reduces from O(t 2 K 2 ) in the flattened approach. To be noted, we add a special token [cls] to the start of the history sequence to obtain a global representation. The embedding of [cls] is a parameter to learn, which is initialized from a zero vector. Cross-modal Encoding. We concatenate history and observation as the vision modality, and use cross-modal transformer with N x layers to fuse features from text, history and observation as shown in the right of Figure 4.1. The reason of using such dual-stream architecture rather than one-stream is that the length of different modalities can be highly imbalanced, and the dual-stream architecture can balance the importance of intra-and inter-modal relationships by model design [START_REF] Cao | Behind the scene: Revealing the secrets of pre-trained vision-and-language models[END_REF]. In each cross-modal layer, a vision-text cross-attention is firstly performed for vision modality to attend relevant text information and vice versa for text modality. Then each modality uses self-attention to learn intra-modal relationship such as interaction between observation and history, followed by a fully-connected neural network. Finally, the HAMT model outputs

✓ ✓ ✓ VLN-BERT [216] ✓ ✓ ✓ ✓ ✓ HAMT (Ours) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ embeddings X ′ = (x ′ cls , x ′ 1 , • • • , x ′ L ), H ′ t = (h ′ cls , h ′ 1 , • • • , h ′ t-1 ), O ′ t = (o ′ 1 , • • • , o ′ K , o ′ stop )
for tokens in text, history and observation respectively.

End-to-end training with proxy tasks

As it is difficult to train large-scale transformers with RL due to sparse supervision [START_REF] Parisotto | Stabilizing transformers for reinforcement learning[END_REF], we propose to first end-to-end train HAMT via several proxy tasks to learn unimodal and multimodal representation.

Table 4.1 compares our HAMT with previous VLN transformers PREVALENT [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF] and VLN-BERT [START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] in inputs and proxy tasks. As neither PREVALENT nor VLN-BERT jointly encodes text, history and observation, a limited choice of proxy tasks can be applied in training. Our model instead can take advantage of various proxy tasks to learn cross-modal alignment, spatial and temporal reasoning, and history-aware action prediction. Given the input pair (W , H T ) where T is the length of full trajectory, we can apply common proxy tasks as in vision-and-language pretraining [START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF], including Masked Language Modeling (MLM), Masked Region Modeling (MRM) and Instruction Trajectory Matching (ITM). In the following, we introduce new proxy tasks given the triplet input (W , H t , O t ) specifically for VLN tasks. Single-step Action Prediction/Regression (SAP/SAR). The task deploys imitation learning to predict the next action based on instruction, history from expert demonstration and the current observation. We formulate it as a classification and a regression task respectively. In the SAP classification task, we predict action probability for each navigable view in O c t which is

p t (o ′ i ) = exp( f SAP (o ′ i ⊙x ′ cls )) ∑ j exp( f SAP (o ′ j ⊙x ′ cls ))
, where f SAP is a two-layer fully-connected network, ⊙ is element-wise multiplication and x ′ cls is output embedding of special text token [cls]. The objective is to minimize negative log probability of the target view action o ′ * : L SAP = -log p t (o ′ * ). In SAR regression task, we directly predict the action heading and elevation angles based on the text token

[cls] which is θt , φt = f SAR (x ′ cls ). The loss function is L SAR = ( θt -θ t ) 2 + ( φt -φ t ) 2 .
The two proxy tasks enable the model to learn how to make action decision conditioning on instruction and contextual history. Spatial Relationship Prediction (SPREL). Expressions of egocentric and allocentric spatial relations are frequent in navigational instructions, such as "walk into the room on your left" and "enter the bedroom next to the stairs". In order to learn spatial relation aware representations, we propose the SPREL self-supervised task to predict relative spatial position of two views in a panorama based on only visual feature, angle feature or both. with probability of 0.3. Their encoded representations are o ′ i and o ′ j , and their relative heading and elevation angles are θ i j , φ i j . We then predict θi j ,

Q K V V K Q Q K V Q K V Cross-modal Transformer Layer & 4 [, ' 4 ; . ' 4 ] & [, ' ; . ' ] Self-attn Self-attn FFN FFN Cross-attn V K Q Q K V Q K V Encoder-Decoder Variant & 4 [, ' 4 ; . ' 4 ]
φi j = f SPREL ([o ′ i ; o ′ j ]
) where [; ] denotes vector concatenation and optimize L SPREL = ( θi jθ i j ) 2 + ( φi jφ i j ) 2 . The task helps for spatial relationship reasoning in the observation. Training Strategy. Instead of directly training the whole HAMT model at once, we propose to progressively train HAMT in two stages. In the first stage, we freeze ViT pretrained on ImageNet [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF] and train the rest of the modules which are randomly initialized. This aims to avoid catastrophic forgetting of the pretrained weights in ViT. Then we unfreeze ViT and train the whole model end-to-end. The learning rate for ViT is set to be higher than for others modules to avoid vanishing gradients and to speedup convergence.

Fine-tuning for sequential action prediction

Structure Variants. We present two variants of HAMT for action prediction in the following. 1) MLP action head: we directly reuse the action prediction network f SAP in the SAP task to predict navigable views. We use it as default for VLN tasks. 2) MLP action head based on encoder-decoder structure: the original HAMT model applies cross-modal attention for both vision-to-text and text-to-vision, which is computationally expensive when instructions are long. Therefore, we remove the cross-modal attention from text to vision. In this way, we separate the cross-modal transformer into an encoder which only takes instruction as input, and a decoder that inputs history and observation as query and attends over encoded text tokens.

We present the encoder-decoder variant of HAMT in fine-tuning on the right of Figure 4.3. Compared to the original cross-modal transformer on the left, the variant removes text-to-vision cross-modal attention. The encoder encodes the texts to obtain textual embeddings. Then the decoder reuses the same text embeddings in vision-to-text attention layer at each navigation step. In this way, the variant is more efficient when instructions are long e.g. in R4R and RxR datasets. here µ is the learning rate, a * t is the expert action at step t of the expert trajectory of length T * . RL+IL Objective. We combine Reinforcement Learning (RL) and Imitation Learning (IL) to fine-tune HAMT for sequential action prediction. The IL relies on the SAP loss defined in Section 4.3.2 and follows the expert action at each step while RL samples actions according to the policy π. Specifically, we use the Asynchronous Advantage Actor-Critic (A3C) RL algorithm [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. At each step t, the agent samples an action based on policy π:

âh t ∼ π(a t |W , H t , O t , O c t )
and receives an immediate reward r t . For non-stop actions, we set r t as the reduced distance of taking the action to the target and the increased alignment score [START_REF] Jain | Stay on the path: Instruction fidelity in vision-and-language navigation[END_REF] compared to expert demonstration as defined in [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF]; for the stop action, r t = 2 if the agent successfully arrives at the target otherwise -2. A critic network is trained to estimate the value of each state s t , which is R t = ∑ T -t k=0 γ k r t+k where γ is discount factor. We implement it as

V t = f critic (x ′ cls ⊙ h ′ cls ).
As the reward signal favors shortest distance, we empirically find it benefits to combine A3C RL with IL weighted by λ , which is:

Θ ← Θ + µ 1 T T ∑ t=1 ∇ Θ log π( âh t ; Θ)(R t -V t )
Reinforcement Learning (RL)

+ λ µ 1 T * T * ∑ t=1 ∇ Θ log π(a * t ; Θ) Imitation Learning (IL) (4.3)

Proxy tasks in training

We employ five proxy tasks to train HAMT and introduced SAP/SAR and SPREL in Section 4.3.2. In the following, we present the other three proxy tasks, which are all based on the input pair (W , H T ), where W is the textual instruction and H T is the full trajectory with length T . Masked Language Modeling (MLM). The task predicts masked words based on contextual words and the full trajectory. We randomly mask out tokens in W with the probability of 15% with a special token [mask] as in BERT, and predict the word distribution p(w i |W \i , H T ) = f MLM (x ′ i ) where W \i is the masked instruction, x ′ i is the output embedding of the masked word w i and f MLM is a two-layer fully-connected network. The objective is to minimize the negative log-likelihood of original words:

L MLM = -log p(w i |W \i , H T ).
The task is beneficial to learn grounded language representations and cross-modal alignment. Masked Region Modeling (MRM). The task aims to predict semantic labels of masked observations in the trajectory given an instruction and neighboring observations. We zero out observations in H T 15% of the time. The target of a masked O i is the class probability predicted by an image classification model pretrained on ImageNet. We use ViT-B/16 [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] in this chapter. Suppose P i ∈ R 1000 is the target class probability for a masked O i , we predict Pi = f MRM (o ′ i ) where o ′ i is the output embedding of masked O i , and minimize the KL divergence between the two probability distributions: L MRM = -∑ 1000 j=1 P i, j log Pi, j . In order to solve the task, o ′ i should capture temporal continuity in the history sequence and align with relevant instructions.

Instruction Trajectory Matching (ITM).

The task predicts whether a pair of instruction and trajectory is aligned. We predict the alignment score as s(W ,

H T ) = f ITM (x ′ cls ⊙ h ′ cls )
, where ⊙ is element-wise multiplication and x ′ cls , h ′ cls are output embeddings for the text [cls] token and the history [cls] token respectively. We sample 4 negative trajectories for each positive instruction-trajectory pair during training, in which two negative trajectories are randomly selected from other positive pairs in the mini-batch, two are obtained by temporally shuffling the positive trajectory. The objective is the Noisy Contrastive Estimation loss [START_REF] Gutmann | Noise-contrastive estimation: A new estimation principle for unnormalized statistical models[END_REF]:

L ITM = -log exp(s(W ,H T )) exp(s(W ,H T ))+∑ 4 k=1 exp(s(W ,H neg T,k ))
. The model is supposed to learn cross-modal alignment and be sensitive to temporal orders of history to solve the task.

Experiments

Experimental setup

Datasets. We evaluate our method on four VLN tasks (seven datasets): VLN with fine-grained instructions (R2R [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF], RxR [START_REF] Ku | Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding[END_REF]); VLN with high-level instructions (REVERIE [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF], R2R-Last); vision-and-dialogue navigation (CVDN [START_REF] Thomason | Visionand-dialog navigation[END_REF]); and long-horizon VLN (R4R [START_REF] Jain | Stay on the path: Instruction fidelity in vision-and-language navigation[END_REF], R2R-Back).

• R2R [START_REF] Anderson | On evaluation of embodied navigation agents[END_REF] to return to its start location after arriving at the destination. The agent needs to remember its navigation histories to solve the task. We add a return command at the end of each instruction in R2R and a reverse path from the end to start locations as expert demonstration. • CVDN [START_REF] Thomason | Visionand-dialog navigation[END_REF] defines a navigation from dialog history task, which requires an agent to arrive at goal regions based on multi-turn question-answering dialogs. Such types of instructions are often ambiguous and under-specified. The lengths of instructions and paths are also long. • REVERIE [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF] replaces step-by-step instructions in R2R with high-level instructions, which mainly describe the target location and object. The agent, hence, is required to navigate to the goal without detailed guidance and depends on its past experiences. • R2R-Last is our proposed VLN setup similar to REVERIE. It only uses the last sentence from the original R2R instructions describing the final destination. Table 4.2 summarizes details of the dataset split. The proposed R2R-Back and R2R-Last setups consider exactly the same splits as the R2R dataset. We present details to construct R2R-Back and R2R-Last in the following. R2R-Back. We append a returning command at the end of annotated instructions in R2R to create new instructions for R2R-Back. The returning command is randomly sampled from the following sentences: "walk back to the start", "return by the way you came", "double back to where you start", "backtrack to the start", "back the way you came", "return to the starting point". The original target location is viewed as a middle stop point. The groundtruth trajectory in R2R-Back is the concatenation of the original and its inverse trajectory. R2R-Last. We use spacy toolkit2 to split sentences for instructions in R2R. We only select the last sentence in each instruction as the new high-level instruction. It mainly describes where the goal location is e.g. "stop in front of the vent", requiring the agent to explore houses without step-by-step textual guidance. The groundtruth trajectory is the same as R2R. Evaluation metrics. We adopt standard metrics [START_REF] Anderson | On evaluation of embodied navigation agents[END_REF], including (1) Trajectory Length (TL): the agent's navigated path in meters; (2) Navigation Error (NE): the average distance in meters between the agent's final position and the target; (3) Success Rate (SR): the ratio of trajectories reaching the destination with a maximum error of 3 meters to the target; and (4) Success Rate normalized by the ratio between the length of the shortest path and the predicted path (SPL). SPL is more relevant than SR as it balances the navigation accuracy and efficiency. For long-horizon VLN task (R4R and R2R-Back), we further employ three metrics to measure the path fidelity between the predicted path and target path, including (5) Coverage weighted by Length Score (CLS) [START_REF] Jain | Stay on the path: Instruction fidelity in vision-and-language navigation[END_REF]; [START_REF] Md | Gated feedback refinement network for dense image labeling[END_REF] the normalized Dynamic Time Warping (nDTW) [START_REF] Ilharco | General evaluation for instruction conditioned navigation using dynamic time warping[END_REF]; and [START_REF] Bou Ammar | Online multi-task learning for policy gradient methods[END_REF] the Success weighted by nDTW (SDTW).

In R2R, RxR, R4R and R2R-Last datasets, a predicted trajectory is considered to be successful if the agent arrives 3 meters near to the final destination. However, such definition would make a motionless agent achieve 100% success rate (SR) on R2R-Back dataset as the final destination is the same as the starting location. Therefore, in R2R-Back evaluation, we define the success as that an agent firstly arrives 3 meters near to the original destination and then returns 3 meters near to its starting location. The groundtruth length in the SPL metric is also modified as the total traversed distance in groundtruth trajectory rather than the shortest distance between start and target location. As the REVERIE task aims for remote object grounding, the success on REVERIE is defined as arriving at a viewpoint where the target object is visible. Implementation details. For the HAMT model, we set N L = 9 for language transformer, N h = 2 for panoramic transformer in hierarchical history encoding, and N x = 4 for crossmodal transformer. There are K = 36 view images in each panoramic observation. We use ViT-B/16 [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] for image encoding if not otherwise specified. In training with proxy tasks, we randomly select proxy tasks for each mini-batch with predefined ratio. We train HAMT for 200k iterations with fixed ViT using learning rate of 5e-5 and batch size of 64 on 4 NVIDIA Tesla P100 GPUs (∼1 day). The whole HAMT model is trained end-to-end for 20k iterations on 20 NVIDIA V100 GPUs with learning rate of 5e-5 for ViT and 1e-5 for the others (∼20 hours). We use R2R training set and augmented pairs from [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF] for training unless otherwise noted. In fine-tuning with RL+IL, we set λ = 0.2 in Eq (4.3) and γ = 0.9. The model is fine-tuned for 100k iterations with learning rate of 1e-5 and batch size of 8 on a single GPU. Unimodal encoders are fixed by default. The best model is selected according to performance on val unseen split. We use the same augmented data as [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF] for R2R for fair comparison, while no augmented data is used for other datasets. Greedy search is applied in inference following the single-run setting.

Training with proxy tasks. We sample proxy tasks for each mini-batch to train the HAMT model. The sampling ratio is MLM:MRM:ITM:SAP:SAR:SPREL=5:2:2:1:1:1. The optimizer is AdamW [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF]. In the end-to-end training stage, we use image augmentation and regularization techniques to avoid overfitting of the ViT model, including RandAugment [62] and stochastic depth [START_REF] Huang | Deep networks with stochastic depth[END_REF]. Fine-tuning for sequential action prediction. Due to different goals in various VLN tasks, we design different rewards in reinforcement learning for each downstream VLN dataset. In R2R, RxR and R4R datasets, the reward is introduced in Section 4.3.3 to take both goal distance and path fidelity into account. In R2R-Last, REVERIE and CVDN datasets where the instruction may not describe detailed navigation path, we only use the reduced distance to the goal viewpoints as rewards. We normalize the reduced distance in the same way as in the R2R dataset. In R2R-Back dataset, we use a different fine-tune strategy to avoid trivial motionless solutions. We require the agent to predict stop actions twice for the original destination (midpoint) and its starting point (final destination) respectively. Before arriving at the midpoint, the RL reward is computed based on distances to the midpoint. If the agent predicts a wrong location to stop for the midpoint, the episode is stopped; otherwise the agent continues its task while receiving rewards based on the distance to the final destination for fine-tuning. We run each experiment twice for ablation study and use the best result on the validation unseen split for the state-of-the-art comparison.

Ablation studies

In this section, we evaluate each component in the HAMT model, including: hierarchical history encoding, end-to-end training with proxy tasks, and fine-tuning objectives.

How important is the history encoding for VLN? For fair comparison with the state-of-the-art recurrent architecture RecBERT [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF], we use the same Resnet152 visual features and train all the models from scratch with RL+IL objectives to avoid the influence of different weight initialization. The models are optimized for 300k iterations end-to-end except for the visual feature. Table 4.3 compares different history encoding approaches on R2R dataset. Our recurrent model slightly differs from RecBERT (no init. OSCAR) [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF] in transformer architecture as shown in Figure 4.1. It achieves slightly better performance on val unseen split. The temporal-only model uses transformer to encode agent's oriented visual observations in history sequence, and outperforms the recurrent method by relative gains of 1.9% on SR and 2.1% on SPL for val unseen split. Adding panoramic observations in a hierarchical way results in 4.2% (SR) and 3.6% (SLP) relative improvements on the val unseen split compared to the recurrent method.

Even larger improvements are achieved on val seen split as the hierarchical model has a larger capacity to fit the seen environments. This evaluation demonstrates the advantage of our hierarchical history representation compared to the recurrent and temporal-only history representation.

How much does training with proxy tasks help?

We next evaluate the advantage of training HAMT end-to-end with proxy tasks. In Table 4.4a, the first row uses RL+IL objectives to train HAMT from scratch, while the second row uses proxy tasks for training prior to RL+IL fine-tuning. We can see that it significantly boosts the performance to first train with proxy tasks. It improves on val unseen split with 16.7% and 18.0% relative gains on SR and SPL respectively, indicating that training with auxiliary proxy tasks enables better generalization. In the third row, we replace the visual feature from Resnet152 to ViT. The ViT feature improves the performance on both val seen and val unseen splits, showing that more powerful visual representations matter. Finally, training ViT end-to-end obtains 2.1% gains on SPL on val unseen split. This is the first time to show that optimizing visual representations end-to-end is beneficial for VLN tasks. In Table 4.4b, we evaluate the benefit of the two new proxy tasks for frozen ViT features using the other proxy tasks by default. The SAP(R) uses imitation learning to predict actions, which directly influences the navigation policy and improves the performance by a large margin. The SPREL is a self-supervised proxy task that forces the model to learn spatial relationships in panorama and helps generalization in unseen environments.

What is the impact of the fine-tuning objectives? Table 4.5 presents results using different objectives in fine-tuning. The first row directly applies HAMT trained by proxy tasks, which achieves lower performance than that after IL fine-tuning, because we mainly use augmented data in proxy task training to increase visual diversity, but such noisy data deteriorates action prediction performance. Previous work [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF] has shown that RL alone performs poorly. However, training with proxy tasks stabilizes the followup RL fine-tuning. HAMT optimized by RL achieves much better performance than that when fine-tuning with IL on the SR metric. It indicates that RL is able to learn better exploration strategy on unseen environments. However, as the reward for RL focuses more on shortest paths rather than path fidelity with instructions, the improvement on SPL metric is relatively small compared to SR metric. Moreover, the fluctuation of the pure RL objective is larger than IL. Therefore, mixing the RL and IL achieves the best performance. Computation Efficiency. To assess the influence of history encoding on the inference time, we compare HAMT with RecBERT [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF]. The HAMT and RecBERT use the same number of layers in the language transformer and cross-modal transformer. The main difference of two models is in the history encoding and the attended length of history for action prediction. We run each model on the R2R val unseen split (2349 instructions) and report inference times averaged over two runs using a single Tesla P100 GPU. For our method we compare variants with and without Text-to-Vision Attention, denoted here as HAMT and HAMT noT2V respectively. We can see that HAMT and its noT2V variant are only 1.5x and 1.1x slower compared to RecBERT, suggesting that attending to the whole history does not increase the inference time significantly. Moreover, while HAMT noT2V is only 10% slower compared to [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF], it still outperforms [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF] in SR and SPL on val unseen split.

Comparison to state of the art

VLN with fine-grained instructions: R2R and RxR. Table 4.7 compares HAMT with previous VLN methods on the R2R benchmark. Our model outperforms state-ofthe-art results of RecBERT [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF] by relative 5.9% and 7.0% improvements in SPL on val seen and unseen splits respectively. We achieve state-of-the-art performance under the single-run setting on the unseen testing split of the leaderboard Long-horizon VLN: R4R and R2R-Back. Table 4.8 shows navigation results on R4R dataset. As R4R contains longer instructions and trajectories compared to R2R, we use the encoder-decoder variant of HAMT for better efficiency. Our method outperforms previous approaches in all metrics and shows particularly large improvements for the path fidelity related metrics. Compared to RecBERT, HAMT achives 8.2% and 9.5% relative improvement in CLS and nDTW respectively. The large improvements on these path fidelity related metrics indicate that HAMT is better to follow the designated path of the fine-grained instruction. Figure 4.4 evaluates the performance of HAMT and RecBERT with respect to instruction length measured by words. Though the nDTW decreases for longer instructions, the relative improvement of HAMT increases with the instruction length.

The navigation performance on R2R-Back dataset is presented in Table 4.9. We compare with two state-of-the-art recurrent models EnvDrop [START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF] and RecBERT [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF] based on LSTM and transformer respectively (both models are trained on R2R-Back for fair comparison). The improvements are more significant on this task as it requires the agent to remember the way it came to the target to successfully return back. The recurrent state is insufficient to capture such history and leads to inferior performance compared to the HAMT model. Vision-and-Dialog Navigation: CVDN. The CVDN dataset contains dialogs as instructions and use Goal Progress (GP) in meters as the primary evaluation metric. GP measures the difference between completed distance and left distance to the goal, so the higher the better. There are two types of demonstrations in the dataset. One is shortest-path trajectory and the other is player's navigation trajectory. We mix the two types of demonstrations as supervision in training which has shown to be the most effective in previous works [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF][START_REF] Shrivastava | Visitron: Visual semanticsaligned interactively trained object-navigator[END_REF][START_REF] Xin | Environment-agnostic multitask learning for natural language grounded navigation[END_REF]. As navigation paths in CVDN dataset are much longer than R2R dataset, we adopt the encoder-decoder variant of HAMT. As shown in Table 4.10, HAMT outperforms existing recurrent approaches on both seen and unseen environments, and achieves the top position in the leaderboard 4 . It demonstrates that our HAMT model is generalizable to different types of instructions in new VLN tasks.

VLN with high-level instructions: R2R-Last and REVERIE. Table 4.11 shows results on the R2R-Last dataset that specifies the goal location and contains no stepby-step instructions. The HAMT model with the hierarchical history encoding is able to better accumulate the knowledge of the environment and achieves 9.8% and 10.5% relative gains on SPL metric on seen and unseen splits respectively compared to RecBERT [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF]. The REVERIE dataset also contains high-level instructions but requires object grounding at the target location besides navigation. Our HAMT achieves 4 https://eval.ai/web/challenges/challenge-page/463/leaderboard/1292 (25/10/2021) † We run the original implementation of methods released by the authors.

RxR dataset

As shown in Table 4.2, RxR dataset contains much more instructions than R2R dataset. Therefore, we directly use RxR in training proxy tasks rather than R2R with augmented data. As there are three different languages in RxR, we take advantage of pretrained multilingual BERT [START_REF] Conneau | Unsupervised cross-lingual representation learning at scale[END_REF] to initialize the unimodal language encoder, so we are able to deal with multilingual instructions using the same HAMT model. We employ the encoder-decoder variant of HAMT for computational efficiency. For fair comparison with other approaches in RxR testing leaderboard 5 which adopt pretrained CLIP [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] features, we use the same visual features without end-to-end optimization. Table 4.12 presents navigation performances on RxR test split. Our multilingual HAMT model achieves 12.83% and 6.25% gains on SR and nDTW respectively than the second place. Nevertheless, there is still a large gap compared to the human performance. We further present results on val seen and val unseen splits in Table 4.13.

REVERIE dataset

The remote object localization task in REVERIE dataset requires both navigation and object grounding. To support the two subtasks in HAMT, we concatenate object features with original view image features for each viewpoint, and add an object grounding head to predict the target object given output embeddings of all object tokens. We fine-tune HAMT that is end-to-end pretrained on R2R dataset, and use the optimized ViT to extract object features given groundtruth object bounding boxes in REVERIE dataset. As shown in Table 4.14, HAMT achieves better navigation performance (SR and SPL), but the object grounding performance (RGS and RGSPL) on test split is worse than state of the art. Since HAMT can more effectively encode observed visual scenes and actions in the history sequence, it is able to better understand house environments and navigate to target viewpoints more efficiently as shown in the much higher SPL score. However, as we use ViT optimized on R2R dataset to extract object features, the object representation might not be as generalizable as object features used in previous works which are pretrained on large-scale object detection datasets.

Additional ablations History in training with proxy tasks

We show that the history input plays a critical role for training with proxy tasks. We compare HAMT with history input and PREVALENT [START_REF] Hao | Towards learning a generic agent for vision-and-language navigation via pre-training[END_REF] without history. For fair comparison, we re-implement PREVALENT which only takes instruction W and singlestep observation O t as input and the other architectures are set the same as HAMT.

We train PREVALENT with all proxy tasks except the ITM task because there is no trajectory input in PREVALENT for instruction-trajectory matching. ViT features pretrained on ImageNet are used in this experiment. In Figure 4.5, we present the single-step action prediction (SAP) accuracy of HAMT and PREVALENT during the training. The SAP accuracies on val seen split are similar for the two models, however, PREVALENT performs much worse on the val unseen split than HAMT. Due to the capacity of large-scale transformer, PREVALENT is likely to memorize the map structure of seen houses, and thus achieves comparable performance to HAMT. However, such knowledge cannot be transferred to unseen houses because the structure and visual observations are distinct for seen and unseen houses. Feeding history as inputs avoids the model simply cramming the structure of seen houses, and enables it to align the history with an instruction to predict actions for better generalization. After fine-tuning the two models on R2R dataset, we obtain SPL 57.5 on val unseen split for HAMT, while 52.7 for PREVALENT without history input. As the same proxy tasks are used in training, the large gains of our HAMT model contribute to the history encoding. Therefore, the proposed history encoding can largely improve the navigation performance on top of training proxy tasks. 4.15, end-to-end feature optimization improves SPL by 2.1% on val unseen split but decreases SPL by 0.8% on val seen split. Note that we follow previous VLN works [START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF][START_REF] Hao Tan | Learning to navigate unseen environments: Back translation with environmental dropout[END_REF] to select the best model based on val unseen and use the same model for val seen split. We observe that the performance on val seen split can be improved with longer training time. After optimizing visual representations, HAMT converges faster on val unseen split and achieves the best performance at earlier iterations. Therefore, the performance on val seen split is slightly worse than no end-to-end optimization. If training longer, the performance with optimized ViT features on val seen split can be higher.

Visual features in training with proxy tasks

Different proxy tasks in end-to-end training

In Table 4.4b, we fix ViT features to ablate contributions of different proxy tasks in training. We further present the ablation results in a fully end-to-end training setup in Table 4. [START_REF] Antol | VQA: Visual Question Answering[END_REF], where different proxy tasks are used to train HAMT including the ViT features. The results show the same trend as Table 4.4b, where our proposed two new proxy tasks (SAP/R and SPREL) are beneficial. Moreover, we can see that the end-to-end ViT features are superior to fixed ViT features in Table 4.4b on val unseen split for all the three proxy task combinations.

History encoding in long-horizon VLN task

We compare different history encoding approaches on the R2R-Back dataset to show that the history information is more beneficial for the long-horizon VLN task. Table 4.17 presents navigation results. All the models are initialized from weights after training with proxy tasks. In order to successfully return back, the agent should remember the way it comes to the targets. The recurrent state is insufficient to capture all the information and achieves the worst navigation performance. Encoding agent's oriented view at each step in temporal-only model improves over the recurrent approach. However, as the oriented view of the agent in backward trajectory is different from the view in forward trajectory, temporal-only model does not take advantage of the full memory in previous exploration and performs inferior to our hierarchical history encoding model. It demonstrates the effectiveness of our proposed method in long-horizon VLN task that requires long-term dependency. We also show that using the end-to-end trained ViT features further benefits the navigation performance.

Structure variants in fine-tuning

Our model reuses the f SAP (o ′ i ⊙ x ′ cls ) in training proxy tasks to sequentially predict action in fine-tuning. In Table 4.18, we compare using different input tokens for the action prediction in f SAP , including different combinations of the observation token o ′ i , global history token h ′ cls and special text token x ′ cls . We can see that the performance varies little on the val unseen split, which indicates that the cross-modal transformer in our model is able to effectively fuse different modalities so that the performance is influenced little by tokens used in prediction.

Conclusion

This chapter presents the first end-to-end transformer for vision-and-language navigation, denoted as History Aware Multimodal Transformer (HAMT). Our method efficiently encodes long-horizon history and combines it with instructions and observations to derive multimodal action prediction. The HAMT is first trained with proxy tasks in an end-to-end manner, and is then fine-tuned with RL to improve the navigation policy.

We achieve state-of-the-art navigation performance on a diverse range of challenging VLN tasks, demonstrating improved accuracy and generalization of our approach compared to the dominant recurrent methods. Future work could extend our history-aware transformer to VLN with continuous actions [START_REF] Krantz | Beyond the nav-graph: Vision-and-language navigation in continuous environments[END_REF] and could benefit from pretraining on larger navigation datasets. Chapter 5

Instruction-driven history-aware policies for robotic manipulations

In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. In Chapters 3 and 4, the action space was simplified, as we predicted the next node location of the robot, and then use a lower-level controller to move the robot from node to node. This simplification does not hold anymore when tackling manipulation-based tasks, as it requires to control fine-grained motors. Furthermore, robotic manipulation requires as well long-term memory as well as a generalization to previously unseen tasks and environments.

To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views.

We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state-of-the-art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.

Introduction

People can naturally follow language instructions and manipulate objects to accomplish a wide range of tasks from cooking to assembly and repair. It is also easy to generalize to new tasks by building upon skills learned from previously seen tasks. Hence, one of the long-term goals for robotics is to create generic instruction-following agents that can generalize to multiple tasks and environments. Thanks to significant advances in learning generic representations for vision and language [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF], recent work has made great progress towards this goal [211,[START_REF] Mees | Composing pick-and-place tasks by grounding language[END_REF][START_REF] Mees | CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks[END_REF][START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF][START_REF] Jang | BC-Z: Zero-shot task generalization with robotic imitation learning[END_REF]. For example, CLIPort [START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF] exploits CLIP models [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] to encode single-step visual observations and language instructions and to learn a single policy for 10 simulated tasks. BC-Z [START_REF] Jang | BC-Z: Zero-shot task generalization with robotic imitation learning[END_REF] uses a pre-trained sentence encoder [START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF] to generalize to multiple manipulation tasks. However, several challenges remain underexplored. One important challenge is that sequential tasks require to track object states that may be hidden from current observations, or to remember previously executed actions. This behaviour is hard to model with recent methods that mainly rely on current observations [START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF][START_REF] Jang | BC-Z: Zero-shot task generalization with robotic imitation learning[END_REF].

Another challenge concerns manipulation tasks that require precise control of the robot end-effector to reach target locations. Such tasks can be difficult to solve with single-view approaches [142], especially in situations with visual occlusions and objects of different sizes, e.g. see put money in safe Figure 5.1 (left). While several recent approaches combine views from multiple cameras by converting multi-view images into a unified 2D/3D space [START_REF] Zeng | Transporter networks: Rearranging the visual world for robotic manipulation[END_REF][START_REF] James | Coarseto-fine Q-Attention: Efficient learning for visual robotic manipulation via discretisation[END_REF] or through a late fusion of multi-view predictions [START_REF] Liu | Auto-Lambda: Disentangling dynamic task relationships[END_REF], learning representations for multiple camera views is an open research problem. Furthermore, cross-modal alignment between vision, action, and text is challenging, in particular when training and test tasks differ in terms of objects and the order of actions, see Figure 5.1 (right). Most of existing methods [START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF][START_REF] Jang | BC-Z: Zero-shot task generalization with robotic imitation learning[END_REF][START_REF] Shao | Concept2robot: Learning manipulation concepts from instructions and human demonstrations[END_REF][START_REF] Nair | Learning language-conditioned robot behavior from offline data and crowdsourced annotation[END_REF] condense instructions into a global vector to condition policies [START_REF] Perez | FiLM: Visual reasoning with a general conditioning layer[END_REF] and are prone to lose fine-grained information about different objects.

To address the above challenges, we introduce Hiveformer -a History-aware instruction-conditioned multi-view transformer. It converts instructions into language tokens given a pre-trained language encoder [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF], and combines visual tokens for both past and current visual observations and proprioception. These tokens are concatenated and fed into a multimodal transformer which jointly models dependencies between the current and past observations, spatial relations among views from multiple cameras, as well as fine-grained cross-modal alignment between vision and instruction. Based on the output representations from our multimodal transformer, we predict 7-DoF actions, i.e. , position, rotation and state of the gripper, with a UNet [START_REF] Ronneberger | UNet: Convolutional networks for biomedical image segmentation[END_REF] decoder.

We carry out extensive experiments on RLBench [START_REF] James | RL-Bench: The robot learning benchmark & learning environment[END_REF] in three setups: single-task learning, multi-task learning, and multi-variation generalization1 . Our Hiveformer significantly outperforms state-of-the-art models for all three settings, demonstrating the effectiveness of encoding instruction, history and views from multiple cameras with the proposed transformer. Moreover, we evaluate our model on 74 tasks of RLBench, which goes beyond the 10 tasks used by Liu et al. [START_REF] Liu | Auto-Lambda: Disentangling dynamic task relationships[END_REF]. We manually group all the tasks into 9 categories according to their main challenges and analyze results per category for a better understanding. Hiveformer not only excels in the multiple task setting with seen instructions in training, but also enables generalization to new instructions that represent different variations of the task, even with human-written language instructions. Finally, we evaluate our model deployed on a real robot and show excellent performance.

Interestingly, pretraining the model in the RLBench simulator results in significant performance gains when only a small number of real robot demonstration is available.

To summarize, our contributions are three-fold: • We introduce a new model Hiveformer to solve various challenges in robotics tasks. It jointly models instruction, multiple views, and history via a multimodal transformer for action prediction in robotic manipulation. • We perform extensive ablations of our model on RLBench with 74 tasks grouped into 9 distinct categories. The history improves long-term tasks and the multi-view setting is most helpful for tasks requiring high precision or in the presence of visual occlusions. • We demonstrate that Hiveformer outperforms the state of the art in three RLBench setups, namely single-task, multi-task and multi-variation. A single Hiveformer trained with synthetic instructions is able to solve multiple tasks and task variations, can generalize to unseen human-written instructions and shows excellent performance on a real robot after finetuning. Our code, pre-trained models and additional results are available from the project webpage [START_REF] Guhur | Hiveformer webpage[END_REF].

Related work

Vision-based robotic manipulation. While earlier methods for solving robotics tasks such as visual servoing [START_REF] Hill | Real time control of a robot with a mobile camera[END_REF][START_REF] Chaumette | Visual servo control. i. basic approaches[END_REF] were designed manually, the need to cope with large variations of objects and environments led to the emergence of learning-based neural approaches [START_REF] Strudel | Learning to combine primitive skills: A step towards versatile robotic manipulation[END_REF][START_REF] Lee | Learning visual servoing with deep features and fitted q-iteration[END_REF][START_REF] Bateux | Training deep neural networks for visual servoing[END_REF][START_REF] Florence | Implicit behavioral cloning[END_REF]. Deep neural networks [START_REF] Artúr István Károly | Deep learning in robotics: Survey on model structures and training strategies[END_REF][START_REF] Labbé | Cosypose: Consistent multi-view multi-object 6d pose estimation[END_REF] have achieved impressive results in manipulation for single tasks [START_REF] Akkaya | Solving rubik's cube with a robot hand[END_REF], and recently led to more challenging setups such as multi-task learning [START_REF] Caruana | Learning many related tasks at the same time with backpropagation[END_REF][START_REF] Bou Ammar | Online multi-task learning for policy gradient methods[END_REF][START_REF] Devin | Learning modular neural network policies for multi-task and multi-robot transfer[END_REF][START_REF] Sodhani | Multi-task reinforcement learning with context-based representations[END_REF]. Different multi-task approaches are explored by discovering which tasks should be trained together [START_REF] Liu | Auto-Lambda: Disentangling dynamic task relationships[END_REF][START_REF] Standley | Which tasks should be learned together in multi-task learning?[END_REF], determining shared features across tasks [START_REF] Thrun | Discovering structure in multiple learning tasks: The TC algorithm[END_REF][START_REF] Chen | Robust policies via mid-level visual representations: An experimental study in manipulation and navigation[END_REF], meta-learning [START_REF] Finn | Deep visual foresight for planning robot motion[END_REF][START_REF] Yu | One-shot imitation from observing humans via domain-adaptive meta-learning[END_REF][START_REF] Yu | Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning[END_REF], goal-conditioned learning [START_REF] Finn | One-shot visual imitation learning via meta-learning[END_REF][START_REF] Kalashnikov | MT-Opt: Continuous multi-task robotic reinforcement learning at scale[END_REF], or inverse reinforcement learning [START_REF] Annie S Chen | Learning generalizable robotic reward functions from "in-the-wild" human videos[END_REF]. These approaches can be generally split in two categories according to the training algorithm: reinforcement learning (RL) methods [START_REF] Richard | Reinforcement learning: An introduction[END_REF][START_REF] Haarnoja | Soft actor-critic algorithms and applications[END_REF][START_REF] Peters | Reinforcement learning for humanoid robotics[END_REF][START_REF] Pashevich | Modulated policy hierarchies[END_REF] which learn policies from rewards provided by environments and behavioral cloning methods [START_REF] Torabi | Behavioral cloning from observation[END_REF][START_REF] Ho | Generative adversarial imitation learning[END_REF][START_REF] Lynch | Learning latent plans from play[END_REF] that learn from demon-strations using supervised learning. Demonstrations can be obtained from humans [START_REF] Petrík | Learning object manipulation skills via approximate state estimation from real videos[END_REF], robots [START_REF] Strudel | Learning to combine primitive skills: A step towards versatile robotic manipulation[END_REF][START_REF] Hristov | Disentangled relational representations for explaining and learning from demonstration[END_REF] or play interactions [START_REF] Lynch | Learning latent plans from play[END_REF]. The emergence of robotic simulators, such as Gym [START_REF] Brockman | OpenAI Gym[END_REF], manipulaTHOR [START_REF] Ehsani | ManipulaTHOR: A framework for visual object manipulation[END_REF], dm_control [START_REF] Tunyasuvunakool | dm_control: Software and tasks for continuous control[END_REF], Sapien [START_REF] Xiang | Sapien: A simulated part-based interactive environment[END_REF], CausalWorld [START_REF] Ahmed | CausalWorld: A robotic manipulation benchmark for causal structure and transfer learning[END_REF], and RLBench [START_REF] James | RL-Bench: The robot learning benchmark & learning environment[END_REF], also greatly accelerated the development of manipulation methods. In this chapter, we use behavioral cloning to train policies given scripted demonstrations from RLBench [START_REF] James | RL-Bench: The robot learning benchmark & learning environment[END_REF] which covers many challenging manipulation tasks.

Instruction-driven vision-based robotic manipulation has received growing attention for manipulations in 2D planar [START_REF] Misra | Mapping instructions and visual observations to actions with reinforcement learning[END_REF][START_REF] Stengel-Eskin | Guiding multi-step rearrangement tasks with natural language instructions[END_REF] or recent 3D environments [START_REF] Mees | CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks[END_REF][START_REF] Moritz Hermann | Grounded language learning in a simulated 3D world[END_REF]299], and has been transferred to the real world [START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF][START_REF] Jang | BC-Z: Zero-shot task generalization with robotic imitation learning[END_REF]. As grounding the language in visual scenes is important, existing works have focused on challenges in object grounding, such as localizing objects based on referring expressions [START_REF] Paul | Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators[END_REF][START_REF] Nguyen | Robot object retrieval with contextual natural language queries[END_REF][START_REF] Goodwin | Semantically grounded object matching for robust robotic scene rearrangement[END_REF] and grounding spatial relationships [START_REF] Mees | Composing pick-and-place tasks by grounding language[END_REF][START_REF] Tellex | Understanding natural language commands for robotic navigation and mobile manipulation[END_REF][START_REF] Liu | StructFormer: Learning spatial structure for language-guided semantic rearrangement of novel objects[END_REF]. Since language describes high-level actions, several works [299, [START_REF] Huang | Language models as zero-shot planners: Extracting actionable knowledge for embodied agents[END_REF][START_REF] Garg | LISA: Learning interpretable skill abstractions from language[END_REF] consider a hierarchical approach to decompose a task into sub-goals. Because natural language is rich and diverse, while training resources are limited, further works learn from collected offline data with instructions [START_REF] Jang | BC-Z: Zero-shot task generalization with robotic imitation learning[END_REF][START_REF] Nair | Learning language-conditioned robot behavior from offline data and crowdsourced annotation[END_REF] or leverage pre-trained vision-language models [START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] for action prediction [START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF][START_REF] Khandelwal | Simple but effective: Clip embeddings for embodied ai[END_REF]. To further improve the precision of manipulation skills, Mees et al. [START_REF] Mees | CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks[END_REF] align instructions with multiple cameras by fusing input images with known camera parameters. Most of these works [211,[START_REF] Mees | CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks[END_REF][START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF][START_REF] Jang | BC-Z: Zero-shot task generalization with robotic imitation learning[END_REF] are stateless, since they only employ current observations to predict next actions. Instead, this chapter proposes to jointly model language instructions, history, and multi-view observations.

Transformers [START_REF] Vaswani | Attention is all you need[END_REF] have led to significant gains in natural language processing [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], computer vision [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] and related fields [START_REF] Lu | ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF][START_REF] Tan | Lxmert: Learning cross-modality encoder representations from transformers[END_REF]. They have also been used in the context of supervised reinforcement learning, such as Decision Transformer [START_REF] Chen | Decision transformer: Reinforcement learning via sequence modeling[END_REF] or Trajectory Transformer [START_REF] Louie Putterman | Pretraining for language conditioned imitation with transformers[END_REF]. Recent works in Vision-and-Language Navigation (VLN) [START_REF] Chen | History aware multimodal transformer for vision-and-language navigation[END_REF][START_REF] Pashevich | Episodic transformer for vision-and-language navigation[END_REF]100] further demonstrate that the transformer allows to better leverage previous observations to improve multi-modal action prediction. Transformers are also used to build a multi-modal, multi-task, multi-embodiment generalist agent, GATO [START_REF] Reed | A generalist agent[END_REF]. Inspired by the success of transformers, we explore the transformer architecture for instruction-driven and history-aware robotic manipulation.

Problem Definition

Our goal is to train a policy π a t+1 |{x l } n l=1 , {o i } t i=1 , {a i } t i=1 for robotic manipulation conditioned on a natural language instruction {x l } n l=1 , visual observations {o i } t i=1 , and previous actions {a i } t i=1 where n is the number of words in the instruction and t is the current step. For robotic control, we use macro steps [142] -key turning points in action trajectories where the gripper changes its state (open/close) or velocities of joints are close to zero. We employ an inverse-kinematics based controller to find a trajectory between macro-steps. In this way, the sequence length of an episode is significantly reduced from hundreds of small steps to typically less than 10 macro steps.

The observation o t at step t consists of RGB images I t and point clouds P t aligned with the RGB images. I t is composed of {I k t } K k=1 RGB images from K cameras, with each I k t being of size H ×W × 3 (height, width, 3 channels). Following [START_REF] Liu | Auto-Lambda: Disentangling dynamic task relationships[END_REF], we use

Text Text

Step 1

Step 1

Step t

Step t t ∈ R H×W ×3 is obtained by projecting a single channel depth image H ×W from the k-th camera in world coordinates using known camera intrinsics and extrinsics. Each point in P k t has thus 3D coordinates and is aligned with a pixel in I k t . The action space a t consists of the gripper pose and its state following the standard setup in RLBench [142]. The gripper pose is composed of the Cartesian coordinates p t = (x t , y t , z t ) and its rotation described by a quaternion q t = (q 0 t , q 1 t , q 2 t , q 3 t ) relative to the base frame. The gripper's state c t is boolean and indicates whether the gripper is open or closed. An object is grasped when it is located in between the gripper's two fingers and the gripper is closing its grasp. The execution of an action is achieved by a motion planner in RLBench.

Our Model: Hiveformer

We propose a unified architecture for robotic tasks called Hiveformer (History-aware instruction-conditioned multi-view transformer), see Figure 5.2 for an overview. It consists of three modules: feature encoding, multimodal transformer and action prediction. The feature encoding module (Sec. 5.4.1) generates token embeddings for instructions {x l } n l=1 , visual observations {o i } t i=1 and previous actions {a i } t i=1 . Then, the multimodal transformer (Sec. 5.4.2) learns relationships between the instruction, current multi-camera observations and history. Finally, the action prediction module (Sec. 5.4.3) utilizes a convolutional network (CNN) to predict the next rotation q t+1 and gripper state c t+1 , and adopts a UNet decoder [START_REF] Ronneberger | UNet: Convolutional networks for biomedical image segmentation[END_REF] to predict the next position p t+1 .

Feature Encoding

We encode the instruction, visual observations, and actions as a sequence of tokens.

Instructions. We employ a pre-trained language encoder to tokenize and encode the sentence instruction. Specifically, we use the language encoder in the CLIP model [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF].

Thanks to its vision-and-language pre-training, it is better at differentiating visionrelated semantics such as colors compared to pure language-only pre-trained models like BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], see Table 5.5.

We freeze the pre-trained language encoder and use a linear layer on top of it to obtain embeddings xl ∈ R d for each word token:

xl = LN(W x xl ) + E x T , (5.1)
with xl the l-th embedding output by the language encoder, LN layer normalization [START_REF] Lei Ba | Layer normalization[END_REF], W x a projection matrix, and E x T a type embedding which differentiates instructions from visual observations.

Observations and

F k t ∈ R H v ×W v ×(d v +3
) is computed as follows:

F k t = CNN([I k t ; A k t ]); MeanPool(P k t ) . (5.2)
UNet encoder for image encoding. The CNN in Eq 5.2 is composed of 6 convolutional layers. The first two layers use kernels of size 3x3, strides of size 1, and output channels of sizes 8 and 16 respectively with the LeakyReLU activation function. The remaining four layers use 3x3 kernels, strides of size 2, and output channels of size 16 followed by group normalization and LeakyReLU activation function. Therefore, an image of size H ×W × 3 is encoded by a feature map of size H 16 × W 16 × 16. UNet decoder for position prediction. The decoder uses a sequence of convolutional and upsampling layers to generate a heatmap on the point clouds. Specifically, the convolutional layer is fed with the output from the previous layer and the residual connection from the corresponding layer in the UNet encoder. Its output channel size is 16, kernel size is 3, and stride size is 1. The upsampling layer uses a scale factor of 2 and bilinear sampling. We stack 4 blocks of the layers to recover the original image size H ×W .

Patches. We use patches

f k t,h,w ∈ F k t , h ∈ [1, H v ], w ∈ [1,W v ]
as separate visual tokens. We further encode f k t,h,w using embeddings of the camera id E k C , of the step id E t S , and of the patch location E h,w L as well as an embedding to indicate the visual nature of the tokens E v T as follows:

g k t,h,w = LN(W f f k t,h,w ) + E k C + E t S + E h,w L + E v T . (5.3)
The encoded visual tokens of the k-th camera at step t are denoted as

G k t = {g k t,h,w } H v ,W v h=1,w=1 ∈ R H v ×W v ×d .
We concatenate the encoded tokens for all cameras as

G t = (G 1 t , • • • , G K t ).

Multimodal transformer

Given the encoded tokens at the current macro step t, the multimodal transformer aims to obtain a contextualized representation for G t conditioned on the encoded instruction { xl } n l=1 and history {G i } t-1 i=1 . This enables learning relationships among views from multiple cameras, the current observations and instructions, and between the current observations and history for action prediction.

We use the transformer's attention mechanism [START_REF] Vaswani | Attention is all you need[END_REF] to learn such relationships:

Attn(Q, K,V ) = Softmax W Q Q(W K K) T √ d W V V, (5.4)
where W Q ,W K ,W V are learnable parameters. Unlike previous work [START_REF] Pashevich | Episodic transformer for vision-and-language navigation[END_REF] that uses self-attention layers to capture all relationships, we employ different attention layers to capture different types of relationships, in order to reinforce the importance of the context. First, we use a cross-attention layer to learn the inter-modal relationships between G t and its conditioned contexts C t consisting of tokens in the instruction { xl } n l=1 and history {G i } t-1 i=1 , which is:

Gt = CA(G t ,C t ) = Attn (G t ,C t ,C t ) . (5.5)
Then we learn the intra-modal relationships among patch tokens obtained from the views from multiple cameras through a self-attention layer, i.e. SA( Gt ) = Attn( Gt , Gt , Gt ). Finally, a feed-forward network consisting of two linear layers W 1 and W 2 is applied as follows: Ĝt = LN W 2 GeLU W 1 SA( Gt ) .

(5.6)

Action Prediction

We concatenate the output embeddings of the transformer Ĝt in Eq (5.6) and the original encoded visual representations Ft in Sec. 5.4.1 in the channel dimension and reshape the flattened sequence into a feature map H t ∈ R K×H v ×W v ×(d+d v ) to predict the next action a t+1 = [p t+1 ; q t+1 ; c t+1 ]. As some RLBench tasks require accurate fine-grained positioning, different from the rotation q t+1 and gripper state c t+1 , the position p t+1 is predicted through a separate module that uses point clouds P t .

Rotation and gripper's state. We transform H t into R H v ×W v ×K(d+d v ) and feed it into a CNN decoder. We then apply average pooling across spatial dimensions and employ a linear layer to regress a 5-dimension vector [q t+1 ; c t+1 ].

Position. The prediction of the gripper position p t+1 is decomposed into an expected point on point clouds p e t+1 and an offset p o t+1 , i.e. p t+1 = p e t+1 + p o t+1 . The offset allows us to predict a virtual point outside the convex hull of the point cloud, e.g. when a robotic arm reaches first above the object and then touch the object. For each camera k, a CNN with an upsampling layer predicts an attention map B k t ∈ R H×W over the point clouds P k t . Each value B k t,h,w ∈ B k t corresponds to the probability of reaching the point P k t,h,w ∈ P k t . Therefore, we compute p e t+1 as the expected position over all cameras:

p e t+1 = ∑ k,h,w B k t,h,w • P k t,h,w . (5.7)
The offset p o t+1 is computed from the instruction and the current step id. Let E O ∈ R N τ ×T ×3 be a learnable embedding, where N τ is the number of tasks and T is the maximum length of episodes. We predict the task id from the instruction: Pr(m) = Softmax W m

Training and Inference

Losses. We use behavioral cloning to train the models. In RLBench, we generate D, a collection of N successful demonstrations for each task. Each demonstration δ ∈ D is composed of a sequence of (maximum) T macro-steps with observations {o δ i } T i=1 , actions {a * i } T i=1 , task m * and instruction {x l } n l=1 . We minimize a loss function L over a batch of demonstrations B = {δ j } |B| j=1 ⊂ D. The loss function is the sum of two losses: a mean-square error (MSE) on the gripper's action and a cross-entropy (CE) over the task classification:

L = 1 |B| ∑ δ ∈B ∑ t≤T MSE (a t , a * t ) + CE (Pr(m), m * ) . (5.8)
Masking current observation. To ensure that the model uses past information

{o i } t-1 i=1 , {a i } t-1 i=1
instead of only relying on the current observation o t , we randomly mask the current observation with a probability of 0.1. The masking zeros out randomly selected patch features in the current observation. Therefore, even if the unmasked current observations contain sufficient information, the model still requires to complete the masked observations from the history for action prediction.

Experiments

In this section we present experiments on RLBench [START_REF] James | RL-Bench: The robot learning benchmark & learning environment[END_REF] tasks to demonstrate the effectiveness of our Hiveformer model in three settings: single-task, multi-task, and multi-variation. In the single-task setup, a separate model is trained and tested for each task with no variations of the task. Multi-task refers to a setting where one model is trained for multiple tasks (but each task has a unique variation). In the multi-variation case we train a single model to solve multiple variations of a single task and test it on new variations of the task unseen during training.

Experimental Setup

Dataset setups. RLBench [START_REF] James | RL-Bench: The robot learning benchmark & learning environment[END_REF] is a benchmark of robotic tasks. To compare our method with previous work [START_REF] Liu | Auto-Lambda: Disentangling dynamic task relationships[END_REF], we use the same 10 tasks with 100 demonstrations for training unless stated otherwise. We further evaluate our model on 74 tasks for which RLBench provides successful demonstrations. Although RLBench 2 currently contains 106 supported tasks, we had difficulties to produce demonstrations for 32 of them due to issues with the scripts and the motion planner. To analyze the performance of our model applied to different types of tasks, we manually group 74 tasks into 9 categories according to their key challenges. The 9 task groups are defined as follows:

• The Planning group contains tasks with multiple sub-goals (e.g. picking a basket ball and then throwing the ball). The included tasks are: basketball in hoop, put rubbish in bin, meat off grill, meat on grill, change channel, tv on, tower3, push buttons, stack wine.

• The Tools group is a special case of planning where a robot must grasp an object to interact with the target object. The included tasks are: slide block to target, reach and drag, take frame off hanger, water plants, hang frame on hanger, scoop with spatula, place hanger on rack, move hanger, sweep to dustpan, take plate off colored dish rack, screw nail.

• The Long term group requires more than 10 macro-steps to be completed. The included tasks are: wipe desk, stack blocks, take shoes out of box, slide cabinet open and place cups.

• The Rotation-invariant group can be solved without changes in the gripper rotation. The included tasks are: reach target, push button, lamp on, lamp off, push buttons, pick and lift, take lid off saucepan. • The Multimodal group can have multiple possible trajectories to solve a task due to a large affordance area of the target object (e.g. the edge of a cup). The included tasks are: pick up cup, turn tap, lift numbered block, beat the buzz, stack cups.

• The Precision group involves precise object manipulation. The included tasks are: take usb out of computer, play jenga, insert onto square peg, take umbrella out of umbrella stand, insert usb in computer, straighten rope, pick and lift small, put knife on chopping board, place shape in shape sorter, take toilet roll off stand, put umbrella in umbrella stand, setup checkers.

× × × × × × × 73.2 ± 3.0 R2 Channel × × ✓ × Self × 74.3 ± 4.5 R3 Channel ✓ × ✓ × Self × 76.7 ± 5.7 R4 Channel ✓ ✓ ✓ × Self × 77.3 ± 5.6 R5 Channel ✓ ✓ ✓ ✓ Self × 81.8 ± 5.2 R6 Channel ✓ ✓ ✓ ✓ Self ✓ 82.3 ± 6.3 R7 Patch ✓ ✓ ✓ ✓ Self ✓ 84.5 ± 6.4 R8 Patch ✓ ✓ ✓ ✓ Cross ✓ 88.3 ± 5.1
We evaluate models by measuring the per task success rate for 500 unseen episodes.

Implementation details. We use the Adam optimizer with a learning rate of 5 × 10 -5 . Each batch consists of 32 demonstrations. Models were trained for 100,000 iterations. We apply data augmentation in training including jitter over RGB images I k t , and a random crop of I k t , P k t , and A k t while keeping them aligned. Models are trained on one NVIDIA Tesla V100 SXM2 GPU using a Singularity container with headless rendering. Auto-λ [START_REF] Liu | Auto-Lambda: Disentangling dynamic task relationships[END_REF] uses a UNet network and applies late fusion to predictions from multiple views.

Collection of human-written instructions. In addition to synthetic instructions used for training, we collect human-written natural language instructions for testing. We collect 162 human-written instructions and measure the success rate for each instruction on 10 episodes with random object locations. 8 native English speakers participated in the dataset collection, leading to 63 instructions of 51 testing variations for the push buttons task, and 99 instructions for 99 testing variations for the tower task. Human-written instructions are more varied than the synthetic ones. They contain unseen verbs (e.g. "Tap on the green button, then the grey button and end up pressing the pink button"), unseen formulations (e.g. "Press the green, cyan and pink buttons in that order"), longer sentences (e.g. "Press the white button and then you go to green button and press it and finally press the black button") or unseen color references (e.g. "Press the darker blue button, then the gray one and finally the lighter blue button.").

Motion Planner. We modified the default motion planner in RLBench, as it sometimes fails to reach a target pose even though there exist successful trajectories in the 3D space. To reduce the impact of the imperfect motion planner, we run the motion planner with different seeds up to 10 times until it finds a trajectory to the target.

Ablations

To demonstrate the effectiveness of the proposed model architecture, we ablate the impact of its components in Table 5.1. The model in R1 (row 1) is a UNet architecture similar to Auto-λ [START_REF] Liu | Auto-Lambda: Disentangling dynamic task relationships[END_REF] except that it is conditioned on instructions rather than task ids. This baseline only uses visual observations at the current step and already achieves promising results with a success rate of 73.2%. On top of R1's architecture, a multimodal transformer with self-attention is added in R2 to improve the modeling of multi-view images. Visual tokens {G i t } K i=1 are different channels in the feature map instead of spatial patches used in our final model. In R3 and R4, we further add point clouds P t and gripper position A t in the feature encoding, which leads to 3% improvement in total. The impact of history, i.e. the use of observations from previous steps, ({G i j } K,t-1 i=1, j=1 ) is studied in R5 and R6. The history information brings 4.5% absolute gains and the masking of observations during training further improves the performance by 0.5%. In R7, we replace the tokenization of feature maps from channels g k t,c to patches g k t,h,w , and obtain another 2.2% gain. This improvement can be attributed to patch tokens that help encode fine-grained spatial information. Finally, we use cross-attention instead of self-attention (Eq. 5.5) to condition on the instruction and history context. It further boosts the performance with a 3.8% gain.

Comparison with State of the Art

Single-task evaluation. The upper block in Table 5.2 presents results of different models on 10 single tasks in RLBench. We compare our model with ARM [142] and Auto-λ [START_REF] Liu | Auto-Lambda: Disentangling dynamic task relationships[END_REF], two state-of-the-art methods on RLBench and observe a consistent improvement for all tasks.

Extending tasks in a single-task evaluation setup. In Table 5.3, we further compare Auto-λ and Hiveformer's variants across 74 RLBench tasks grouped into 9 categories. The variant without history removes the history tokens in Hiveformer, while the variant with one view only uses one camera at each step (we take the best among the 3 cameras for each task). The full Hiveformer achieves consistently better performance compared to Auto-λ [START_REF] Liu | Auto-Lambda: Disentangling dynamic task relationships[END_REF] on all types of tasks. Among them, the Long-term, Tools and Planning task groups assess the use of history, where our model brings improves significantly over the variant without history. Compared to the one view variant, our full model performs significantly better on tasks requiring fine-grained control or with large occlusions such as Screw, Precision and Visual Occlusion categories. Yet, our method performs relatively poorly for Long-term tasks with more than 10 steps, such as "take shoes out of box". As Long-term tasks have an average number of steps 2-4 times higher than others, they are more prone to distribution shift issues and accumulated errors. Hierarchical modeling or better training algorithms such as reinforcement learning and dagger could be helpful, but are left as future work.

Comparison with Additional State-of-the-Art Approach

LanCon-Learn [START_REF] Silva | Lancon-learn: Learning with language to enable generalization in multi-task manipulation[END_REF] is a recent instruction-conditioned multi-task approach. It takes as input the gripper state and the object state, namely the ground truth pose of each object in the scene, instead of raw visual observations as ours. It encodes instructions with GloVe embeddings and a bi-directional LSTM. It predicts the next pose of the gripper based on a modular architecture conditioned on encoded text features. We run experiments on RLBench with the code provided by the authors. Since some RLBench tasks require identifying the colors of an object, we modify their object state to include a RGB reference of each object. Moreover, we complete their method with a history mechanism, where the gripper state is concatenated with the gripper state from the previous step. As described in Table 5.4, we obtained an average success rate of 63.9% with their original method (vs. 72.9% with history vs. 88.3% for our approach) in our single-task setting and 47.8% (vs. 57.4% with history vs. 83.8% for our approach) in our multi-task setting.

Multi-task evaluation. The lower half in Table 5.2 shows the results in a multitask setting. Notably, Auto-λ uses a training algorithm that dynamically adjusts the weights of different tasks, while our model simply treats all tasks with equal weights. Nevertheless, our model outperforms Auto-λ by 14%, demonstrating the improvements due to our architecture. We further compare our model with a variant without instructions in the input sequence (since p o t+1 is predicted from instructions, we modify the model such as it is predicted from H k t ). The results show that instructions are important in the multi-task setting. Moreover, the performance of our single model trained for all tasks is only slightly worse than the performance of individual models for each task.

Task Setup in Multi-variation Setting. For the multi-variation setting we choose tasks with as many variations as possible. We hence select the push buttons and tower tasks, for which we can easily construct new variations, as illustrated in Figure 5.3. For each of these tasks we use 100 variations for training and 100 different variations for testing.

• The Push Buttons task has three buttons with unique colors in the scene. The robot should press some or all of the buttons according to the order in an instruction. Variations of the task are defined by the different order and different colors of buttons. RLBench provides three sentence templates to generate synthetic instructions with changing button colors such as "push the red button, and then push the cyan one". • The Tower task is inspired by the "stack block" task. The robot must stack some of the three colored cubes at a target location following the color order provided by the instruction. We generate synthetic instructions for each variation, such as "Stack the red, blue, green blocks", or "Stack the yellow block. Stack the purple block on top of it, then add the cyan cube".

Additional Ablations on Multi-variation Setting

In the multi-variation setting in Table 5.5, the gap between LanCon-Learn and our approach is more significant than in Table 5.4, since GloVe embeddings differentiate poorly colors: for the "push buttons" task on unseen variations and synthetic instructions, the performance reaches only 1.7% (vs. 86.3% with our approach). This also happens when replacing CLIP embeddings with BERT in our model (40.2%).

The important role of history for long-term planning tasks such as "pushing buttons" is confirmed when comparing LanCon-Learn or our model with and without history in the Table 5.5. The model without history can only use its current observation to predict the next action. Therefore, it is hard to infer which buttons have been pressed and which button is the next target, leading to poor performance on the task.

We found that removing E x T decreases the performance only by 3.1%, but removing both E x

T and E v T decreases the performance by 21.1%. Moreover, replacing the instructions with one-hot encoding of the variation index increases the performance for seen variations (by 19.4% on the tower task), but prevents the model from generalizing to unseen variations.

We performed ablations with corrupted instructions on unseen variations. Corrupted instructions were created from synthetic instructions by replacing color references with synonyms that have not been seen during training. For example, the color "azure" is replaced with "light blue", and the color "maroon" with "dark red". Baselines using CLIP as a language encoder have a much smaller drop of performance than any other encoder.

We also test our model with a global language embedding (average over word tokens) as in [START_REF] Shridhar | CLIPort: What and where pathways for robotic manipulation[END_REF] and observe a significant drop in performance. The main reason is that the averaged embeddings do not represent well different action orders, e.g. we have obtained the average cosine similarity of 0.97 for instructions corresponding to same actions in different orders.

Generalization to multi-variations. Table 5.6 shows results of Hiveformer trained on different variations of the two tasks Tower and Push Buttons. The Tower (resp. Push Buttons) task requires the robot to sequentially stack colored cubes (resp. push colored buttons) using the order specified in the instruction, see Figure 5.1 (right). We use 100 variations in training and test models for both the 100 seen variations and 100 unseen variations. In this setting, instructions are necessary to generalize to unseen variations (e.g. it is impossible to distinguish the order of pushing buttons red-greenblue vs. blue-red-green by only looking at the scene). We compare the models trained with different numbers of demonstrations per variation. Even in the most challenging case where only 10 demonstrations are available per variation, Hiveformer achieves a success rate of 71.1% for the push buttons task and 49.8% for the tower task in unseen variations. Furthermore, besides tests on synthetic instructions (Synt), we also test the generalization to real instructions. Despite being only trained on synthetic instructions with limited vocabulary and diversity, our model performs well on instructions generated by humans (Real). Finetuning Hiveformer on human instructions [START_REF] Pashevich | Episodic transformer for vision-and-language navigation[END_REF] is expected to result in further improvements.

Real-robot Experiments

Setup. We conduct real-robot experiments for the push buttons task on a 6-DoF UR5 robotic arm equipped with a 2-finger Robotiq RG2 gripper and two cameras on each side of the scene. As there exists a large difference between simulated and real environments, we finetune the simulator-trained policy on real-robot demonstrations. We use 10 variations of the task and 10 real-robot demonstrations per variation. The cameras are Intel RealSense RGB-D cameras mounted on a fixed support as illustrated in Figure 5.4. We adapt our model to use K = 2 cameras. The resolution of the captured images is at a resolution of 1280 × 720, we apply center crop and downsampling to obtain images of size 128 × 128, which is the input to our model. We use nearest approximation to downsample depth images and bilinear approximation for RGB images. We use intrinsic parameters provided by Intel, and perform extrinsic calibration between the camera and the robot base-frame using an AprilTag marker [START_REF] Olson | Apriltag: A robust and flexible visual fiducial system[END_REF]. We built 10 buttons using white cellulose foams: we manually cut them into 5 × 5 cm squares and attached to each square a painted rounded foam. The button bases and buttons have an average size of 4.95 ± 0.1 cm and 3.16 ± 0.22 cm respectively.

To collect demonstrations with the real robot, we design a script that automatically solves the task provided ground truth locations of buttons and the correct sequence of actions. In each demonstration objects are placed at random locations on the workspace and actions are executed at 10 Hz. We finetune the model for 8k iterations using the same training setup as that in the simulator.

Qualitative Results. Figure 5.5 shows a successful example from our real robot experiments. The attention maps reveal that the robot correctly attends to the next buttons. Thanks to the history of previous observations and actions, the model is confident to not press a button that has already been pressed before (for example the cyan in the fourth column).

In Figure 5.6, we analyze the robustness of our model for to unseen variations in more challenging situations. The instruction of the variation is written by human: "Press the yellow button and then press the black button and finish with the white button". Our model successfully pressed the buttons in the correct order for all situations in Figure 5.6.

• Figure 5.6a: Since the foam buttons have low friction with the table, the gripper has accidentally flipped the black button, providing two buttons looking white. However, thanks to its history component, the robot is able to successfully press the right white button instead of the flipped button.

• Figure 5.6b: Two white buttons are present in the scene. This is a multi-modal example, in which the robot might predict a mean position between the two white buttons, whereas our robot can cope with this challenge.

• Figure 5.6c: We use a ruler to move the location the white button in the scene after the robot pushed the yellow button. Although such perturbations have never been used in training sequences, the robot remains robust to this dynamic environment.

• Figure 5.6d: We change the shape of the button by increasing the height of the yellow button.

• Figure 5.6e: We add occlusion to the gripper.

• Figure 5.6f: We change the appearance of the table. Our model is robust to the above visual modifications.

More details and video demonstrations of our real-robot experiments are available from the project webpage [START_REF] Guhur | Hiveformer webpage[END_REF].

Conclusion

We introduced Hiveformer, a multimodal transformer that jointly models instructions, views from multiple cameras, and history for instruction-driven robotics manipulation. We evaluated the model on RLBench in three settings: single-task learning, multi-task learning, and multi-variation generalization, and we demonstrated its effectiveness outperforming state-of-the-art. We deployed our model on a real robot that is able to generalize to unseen variations and human-written instructions.

Limitations. The computational cost quadratically increases with the input sequence length due to the transformer. Furthermore, our model is trained with behavioral cloning, which may suffer from exposure bias. Future works could improve the efficiency of long-term tasks with hierarchical models and also incorporate reinforcement learning. Moreover, our model is trained on only synthetic instructions and performs worse on human-written instructions. Training on human-written automatically generated instructions could help improve performance. Chapter 6

Perspectives

In this Chapter, we summarize the main contributions of this thesis in Section 6.1, and we discuss open research problems and directions for future work in Section 6.2.

Contributions

This thesis has studied how robots can follow instructions. This research field, languageguided robotics, has its foundations in computer vision, natural language processing, and task planning. We have addressed two different tasks of language-guided robotics. The first task is called vision-and-language navigation. It consists in asking a robot to find a target location, described by provided instructions. We performed experiments in the Matterport3D simulator [START_REF] Chang | Matterport3D: Learning from RGB-D data in indoor environments[END_REF] and considered several benchmarks: some instructions are fine-grained [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Jain | Stay on the path: Instruction fidelity in vision-and-language navigation[END_REF][START_REF] Ku | Room-Across-Room: Multilingual vision-and-language navigation with dense spatiotemporal grounding[END_REF], meaning that step-by-step guidance is provided, but some other instructions are coarse-grained [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF][START_REF] Zhu | Soon: Scenario oriented object navigation with graph-based exploration[END_REF], in which case only a description of the target location is provided. We have also tested our approaches on recorded dialogs [START_REF] Thomason | Visionand-dialog navigation[END_REF].

Since collecting instructions and environments is particularly costly, models suffer from data scarcity. In Chapter 3, we have addressed this issue by collecting BnB, which consists of more than 1 million images and captions on a rental marketplace. We have suggested several directions to efficiently pre-train discriminative and generative models. In addition, we have proposed a new training loss, dubbed as the shuffle loss, to enhance the learning of temporal causality in the transformer. Airbert, our model trained with the BnB dataset and the shuffling loss, obtained best results in the REVERIE [START_REF] Qi | REVERIE: Remote embodied visual referring expression in real indoor environments[END_REF] and in R2R [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF], which is still ranked first two years later [START_REF] Eval | Leadearboard evalai -room-to-room dataset[END_REF]. Furthermore, this work has suggested a new challenging setup where the training dataset is reduced to a single environment. This work has inspired other researchers: in particular, Qiao et al. [START_REF] Qiao | Hop: History-and-order aware pre-training for vision-and-language navigation[END_REF] uses our BnB dataset and an improvement of our shuffle loss to obtain state-of-the-art results over 4 datasets; Hahn and Rehg [START_REF] Hahn | Which way is 'right'?: Uncovering limitations of vision-and-language navigation models[END_REF] demonstrates the significant improvement brought by our shuffle loss.

While Airbert is using pre-computed visual features, in Chapter 4 we proposed an end-to-end approach dubbed HAMT. In this approach, a transformer-based architecture is aligning instructions to the panoramic-view observations obtained for all previous steps. The panoramic views are reshaped into thousands of tokens, which is problematic for Transformer models, whose algorithmic complexity is polynomial with respect to the number of tokens. That is why, we considered a hierarchical approach: a first Transformer architecture encodes visual features, while a second architecture learns the alignment. We have outperformed state-of-the-art over four datasets, and our model has been used as a backbone of multiple follow-up works, notably CLEAR [START_REF] Li | CLEAR: Improving vision-language navigation with cross-lingual, environment-agnostic representations[END_REF], EnvEdit [START_REF] Li | Envedit: Environment editing for vision-andlanguage navigation[END_REF], An et al. [START_REF] An | 1st place solutions for rxr-habitat vision-and-language navigation competition (cvpr 2022)[END_REF], Iterative VLN [START_REF] Krantz | Iterative vision-and-language navigation[END_REF], ULN [START_REF] Feng | Uln: Towards underspecified vision-and-language navigation[END_REF], CSAP [START_REF] Wu | Cross-modal semantic alignment pre-training for vision-and-language navigation[END_REF], and Kamath et al. [START_REF] Kamath | A new path: Scaling vision-and-language navigation with synthetic instructions and imitation learning[END_REF].

The second axis of this thesis considers tasks based on manipulation. In Chapter 5, we have provided instructions to a robotics arm, requesting it to manipulate objects. Manipulation is a different task, because (i) the environment is entirely observable, but (ii) the action space is continuous. We have offered a new variant of the transformer to deal with 74 tasks on RLBench [START_REF] James | RL-Bench: The robot learning benchmark & learning environment[END_REF], outperforming existing methods on single-task and multi-task settings. We have also suggested a new challenging setting, referred to as multi-variation, where the robot learns several variations on a single task and must generalize to unseen variations. We tested this challenging setting on a real-world robot. Finally, we tested that our robot can generalize to human-written instructions while being trained on synthetic instructions.

Future research directions

While this thesis achieves new state-of-the-art results over a dozen of benchmarks, our proposed methods suffer from several limitations. In particular, this thesis deals separately with navigation and manipulation, and one could wonder how to combine them. Below we suggest several directions to address current limitations.

Pre-training. In Chapter 3, we show that pre-training on the BnB dataset can significantly improve performance. Nevertheless, it still tends to overfit training environments and would benefit from additional training data. While large-scale task-specific data collection is prohibitive, alternative approaches exist. Notably, in Auto-VLN [START_REF] Chen | Learning from unlabeled 3d environments for vision-and-language navigation[END_REF], embodied agents are pre-trained on 1000 unlabelled 3D environments, and we show that their performance would even benefit from a higher number of pre-trained environments. But how to obtain a much higher number of environments? For example, one could use the images of the BnB dataset to generate environments, as in Hosseini and Furukawa [START_REF] Hosseini | Extreme floorplan reconstruction by structure-hallucinating transformer cascades[END_REF]. Even though the quality of such environments might be lowered, they could enhance any embodied agents. Pre-training could also benefit from videos. Numerous YouTube videos present walking tours, whereas the large-scale Something-Something dataset [START_REF] Goyal | The" something something" video database for learning and evaluating visual common sense[END_REF] contains 220k short videos of human manipulation. Those videos have successfully been employed to learn robotics tasks [START_REF] Petrík | Learning object manipulation skills via approximate state estimation from real videos[END_REF], and could potentially enhance language-guided robots.

Exposure bias. Models trained to reproduce demonstrations have not been exposed to any new or unseen data points and may not be able to accurately predict outcomes for novel inputs. This issue is a typical limitation of behavioral cloning, used in Hiveformer 5. In Chapters 3 and 4, reinforcement learning has successfully mitigated this issue, because it enhances the exploration of environments during training. But in the context of continuous actions, reinforcement learning is sample inefficient [START_REF] Luketina | A survey of reinforcement learning informed by natural language[END_REF][START_REF] Murray | Following natural language instructions for household tasks with landmark guided search and reinforced pose adjustment[END_REF], and future works could investigate other strategies to reduce the exposure bias of Hiveformer.

Motion-planning. In this thesis, we have assumed the existence of a lower-level controller, a motion planner, which enabled us to focus on a higher-level controller, a task planner. In the case of navigation, we supposed that such a controller would allow an agent to move from one location to another, whereas in the case of manipulation, we employed inverse kinematics to predict the trajectory of the gripper. However, James et al. [141] showed that learning-based controllers could outperform inverse-kinematics in RLBench, as it allows a robot to learn the affordance of objects. For navigation, recent works [START_REF] Krantz | Beyond the nav-graph: Vision-and-language navigation in continuous environments[END_REF][START_REF] Zubair Irshad | Hierarchical crossmodal agent for robotics vision-and-language navigation[END_REF] studied how motion planners and task planners can be learned at the same time. Further works could extend it to the context of manipulation.

Robots able to understand natural language utterances help the evaluation of their cognitive abilities and can be used by anthropologists to gain more insights about robotics. In particular, it has led to the development of innovative methods, such as "integrative social robotics" [START_REF] Seibt | Five principles of integrative social robotics[END_REF].

Environmental concerns. Robots pose also an environmental risk [START_REF] Amparo | Environmental robotics for a sustainable future in circular economies[END_REF]. Sustainability is affected by several factors. First, raw materials to build a robot have a significant environmental cost [START_REF] Wyatt | Life cycle analysis of double-arm type robotic tools for lcd panel handling[END_REF], while the vast majority of manufacturers do not conceive and build in a circular economy paradigm. Second, the greenhouse gas emissions of the scope 1 and 2 (direct emissions) of a robot are significant, mostly due to greenhouse gases emitted during the fabrication of a robot [START_REF] Li | Life cycle greenhouse gas emissions for last-mile parcel delivery by automated vehicles and robots[END_REF], and emissions used by cloud computing [START_REF] Mytton | Hiding greenhouse gas emissions in the cloud[END_REF]. Last but not least, the scope 3 (indirect emissions) [START_REF] Papadopoulou | Environmental sustainability in robotic and laparoscopic surgery: Systematic review[END_REF] is likely to become the main concern of robotics since new applications emerging with robots are expected to increase the demand for goods and services [START_REF] Shukla | Climate change 2022: Climate change 2022: Mitigation of climate change[END_REF].

More than 200 000 V100 GPU hours have been used during this thesis, consuming roughly 100 MWh. Given the carbon intensity of France, the used electricity has produced around 10 tCO2eq. However, the environmental impact of this thesis is difficult to estimate. If the environmental impact can be estimated with the life cycle assessment, other phases such as the fabrication, transport, and end-of-life of the computer cluster, must be taken into account and they may represent around 50% of the carbon footprint [START_REF] Berthoud | Estimation de l'empreinte carbone d'une heure[END_REF]. But the attributional framework of life cycle assessment might hide real impacts impeded to the consequence of this thesis [191]: the emergence of robots solving household tasks could significantly increase the amount of energy and materials required to build such robots. Being able to improve the impacts of our research should be made a priority to analyze the costs and benefits of our research.

  (a) Navigation: a mobile robot follows instructions to find a target location in an unseen photo-realistic environment, on Matterport [43](b) Manipulation: a robotic arm follows instructions to execute household tasks in RL-Bench[START_REF] James | RL-Bench: The robot learning benchmark & learning environment[END_REF]

Figure 1 . 1 :

 11 Figure 1.1: An overview of the two robotics tasks considered in this thesis.

Figure 1 . 2 :

 12 Figure 1.2: An example of a language-guided visual navigation task. The mobile robot receives instructions as well as a panoramic view (obtained from a 360 degrees rotation) at its current location. It must predict the sequence of locations to reach a target location.

Figure 1 . 3 :

 13 Figure 1.3: An example of a language-guided manipulation task. The robotics arm receives instructions as well as multiple views of the countertop.

  Build a large-scale dataset with navigation instructions from BnB listings Fully equipped kitchen Living room space with ample seating How to follow navigation instructions in environments with new objects? Problem Solution Walk around the kitchen. Walk into the living room. Turn right and stop next to the fireplace.

Figure 3 . 1 :

 31 Figure 3.1: VLN tasks are evaluated on unseen environments at test time. Top: None of the training houses contain a Christmas theme making this test environment particularly challenging. Bottom: We build a large-scale, visually diverse, and in-domain dataset by creating path-instruction pairs close to a VLN-like setup and show the benefits of self-supervised pretraining.

( a )Figure 3 . 2 :

 a32 Figure 3.2: Examples of outdoor images with their corresponding captions.

4 typically 4 -Figure 3 . 3 :

 433 Figure 3.3: We explore several strategies to automatically create navigation-like instructions from image-caption pairs.

  Distribution of the number of images per listing. Fill-in-the-blanks templates built using the R2R training set.

  Distribution of the instruction lengths.

Figure 3 . 4 :

 34 Figure 3.4: Statistics of BnB Dataset.

Figure 3 .

 3 Figure 3.5 presents generated BnB PI pairs using various strategies proposed in this chapter, including naive concatenation, instruction rephrasing, instruction generation, image merging and captionless image insertion.Among the methods to create an instruction, simple concatenation lacks action verbs between sentences for fluent transition leading to a domain shift from real instructions. Instruction rephrasing selects noun phrases from BnB image descriptions and inserts them into real instruction templates, providing a natural feel to the created instruction. Finally, while the learning approach of instruction generation (recall, this is learned on downstream VLN dataset) produces fluent sentences, it is unable to leverage the diverse

Concatenation:

  extra guest room with comfy full bed on top floor of house and top floor shared bathroom for both guest rooms then adjoining modern private bath with stall shower bath and beach towels provided then granny's treasures add a homey touch Instruction rephrasing: exit extra guest room and turn left. pass top floor shared bathroom then turn right. walk toward a homey touch and wait there. Instruction generation: walk to the other side of the bathroom and stop next to the last corner on the wall with the candles.

(a) Example 1 (c) Example 3 Figure 3 . 5 :

 1335 Figure 3.5: Examples of path-instruction pairs created by different strategies. The images with dotted borders are images chosen from the captionless image insertion strategy, and the clustered images are from the image merging strategy.

  (b) Adapting Airbert to a generative setting based on the Recurrent VLN-BERT[START_REF] Hong | A recurrent vision-and-language BERT for navigation[END_REF].

Figure 3 . 6 :

 36 Figure 3.6: The adapted Airbert model in both discriminative and generative settings for downstream VLN tasks.

Figure 3 . 7 :

 37 Figure3.7: Overview of our pretraining approach. Instead of the usual VL pretraining (panel 1), we adopt in-domain data and use the path-instruction pairs to train Airbert with the masking and shuffling losses (panel 2). We fine-tune Airbert on downstream VLN tasks using both discriminative or generative models (panel 3).

  (a) R2R ✔: Walk over the kitchen counter, turn left, walk ahead till wall, turn right, walk to the closet room, wait at front. (b) REVERIE ✔: Walk past the kitchen and enter the hallway. Turn right at the artwork and wait by the closet. (c) R2R ✔: Walk forward to the sitting area to the right of the stairs. Walk to the wall of windows and take a right into the recreation room and stop before you reach the pool table.

  (d) R2R ✔: Go between the counters, turn left, turn right, and stop before the display and dining room.

  (e) R2R ✔: Turn right and head towards the end. Once you reach the end make a right and stop. (f) R2R ✔: Walk straight out the door in front of you and follow the red carpet. Keep going through the room with the ropes and stop when you enter the next room with ropes.

Figure 3 . 8 :

 38 Figure 3.8: When navigating in new houses, our Airbert model not only successfully recognizes the closet room in (a) and (b), pool table (c), living room (d), but also generalizes better to challenging environments, such as the church (e) and castle (f).

Figure 3 . 9 :

 39 Figure 3.9: The Airbert model outperforms VLN-BERT to recognize rare or even unseen objects in training set. (a) Rare object "night stand"; (b) unseen object "antique furniture"; (c) rare object "pool table"; and (d) unseen object "freezer".

  (a) R2R ✔: Walk from dining room to living room turning slightly right before lounge chairs, walk straight following cabinet. Turn slight right and stop at stairs. (b) R2R ✔: Walk on into the kitchen and turn to the right. Walk past the staircase, behind the chairs. Walk to the right of the pillar. Stop and wait by the footstool. (c) R2R ✔: Walk out of the hallway and turn left. Walk down the steps and through the wooden doors. Walk down the steps and stop. (d) R2R ✔: Go straight passed the coffee table turn left and go through the left door to the stairs. Stop in front of the stairs.

Figure 3 . 10 :

 310 Figure 3.10: Examples in similar environments and instructions to the training set. The improvements of Airbert model can be contributed to the shuffling loss in pretraining.

  (a) R2R ✘: Walk between the two kitchen islands and then turn right. Pass through the stone archway and stop just after you pass through it. Wait there. (b) R2R ✘: Exit the bathroom and go down the stairs. Enter the last doorway on the left and stop just before stepping on the rug. (c) REVERIE ✘: go to level 3 bathroom in the first bedroom left of the stairs and grab the mirror on the wall (d) REVERIE ✘: Go to the lounge on this level and polish the black leather armchair in the corner

Figure 3 . 11 :

 311 Figure 3.11: Failure cases for both VLN-BERT and Airbert models.

  go straight behind the blue couch. turn left when in front of brick wall. turn left and wait. [sep] ) ./0 , ) # , ⋯ , ) " ) , ' = (ℎ ./0 , ℎ # , ⋯ , ℎ '(# ) . ' = (/ # , ⋯ , / -, / 0123 ) & [, ' ; . ' ]

4 )Figure 4 . 1 :

 441 Figure 4.1: The architecture of History Aware Multimodal Tranformer (HAMT). HAMT jointly encodes textual instruction, full history of previous observations and actions, and current observation to predict the next action.

  Hierarchical history encoding. It first encodes individual view images with ViT, then models the spatial relation between images in each panorama, and finally captures the temporal relation between panoramas in the history. Hierarchical vision transformer for history encoding. It first encodes individual view images with ViT, then models the spatial relation between images in each panorama, and finally capture the temporal relation between panoramas in the history. (b) Flattend history encoding. It jointly encode spatial and temporal relations of all images in the history. Transformer (c) Temporal-only history encoding. It only considers temporal relation of agent's oriented images in the history. Hierarchical vision transformer for history encoding. It first encodes individual view images with ViT, then models the spatial relation between images in each panorama, and finally capture the temporal relation between panoramas in the history. (b) Flattend history encoding. It jointly encode spatial and temporal relations of all images in the history. Transformer (c) Temporal-only history encoding. It only considers temporal relation of agent's oriented images in the history. Temporal-only history encoding. It only considers temporal relation of oriented views.

Figure 4 . 2 :

 42 Figure 4.2: A comparison of history encoding methods. Circle nodes in different colors denote view images of panorama at different steps. Darker circle nodes are the oriented view of the agent.

Figure 4 .

 4 Figure 4.1 illustrates the model architecture of HAMT. The inputs text W , history H t and observation O t are first encoded via the corresponding unimodal transformers respectively, and then fed into the cross-modal transformer encoder to capture multimodal relationships.Text Encoding. For each token i in the instruction W , we embed it as the summation of its word embedding w i , position embedding E P i and type embedding of text E T 0 . Then we employ a transformer with N L layers to obtain contextual representation x i following the standard BERT[START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]. Observation Encoding. For each view [v o i ; a o i ] in the panoramic observation O t , we first represent the relative angle a o i as E A

Figure 4 . 3 :

 43 Figure 4.3: Comparison of the original cross-modal transformer layer (left) and the encoderdecoder based variant (right).

Figure 4 . 4 :

 44 Figure 4.4: nDTW with respect to instruction length on R4R val unseen split.

Figure 4 . 5 :

 45 Figure 4.5: SAP accuracy of PREVALENT (w/o history) and HAMT (w/ history) on R2R dataset.

  Two-stage end-to-end (e2e) training strategy We compare our two-stage e2e training strategy with a single-stage e2e training of HAMT. However, single-stage e2e training achieves inferior performance to the twostage training or even no e2e training. When trained for 25k iterations and evaluated on the val unseen split, the single-stage e2e training of HAMT results in SPL 53.5 while no e2e training achieves SPL 56.5. We hypothesize that the single-stage e2e training is less effective for VLN given (a) the limited training data available for the VLN task and (b) the higher complexity of VLN compared to common vision and language tasks.

( a )

 a Predicted trajectory by RecBERT [122] (failed). (b) Predicted trajectory by HAMT (succeed).

Figure 4 . 6 :

 46 Figure 4.6: Examples in R2R val unseen split. Navigation steps inside red box are incorrect. The instruction is "Walk to the right of the stairs. Continue past and to the right of the stairs that go down. Turn right and stop in the doorway of the double glass doors." (id: 697_0). The RecBERT misunderstands the instruction and goes down the stairs instead of turning right. Our HAMT is better to understand the instruction and spatial relation related to the stairs to turn to the right of the stairs.

Figures 4 .

 4 Figures 4.6-4.9 illustrate trajectories obtained by our HAMT model and compare them to results of the state-of-the-art RecBERT [122] model. We can see that HAMT enables to better interpret instructions (Figure 4.6), recognize the scene (Figure 4.7), follow the correct direction (Figure 4.8), and align the current observation with the instruction (Figure 4.9). We also provide some failure cases in Figures 4.10-4.11, where the HAMT model still needs improvements on scene and object recognition.

- 4 .

 4 Figures 4.6-4.9 illustrate trajectories obtained by our HAMT model and compare them to results of the state-of-the-art RecBERT [122] model. We can see that HAMT enables to better interpret instructions (Figure 4.6), recognize the scene (Figure 4.7), follow the correct direction (Figure 4.8), and align the current observation with the instruction (Figure 4.9). We also provide some failure cases in Figures 4.10-4.11, where the HAMT model still needs improvements on scene and object recognition.

Figure 4 . 7 :

 47 Figure 4.7: Examples in R2R val unseen split. Navigation steps inside red box are incorrect. The instruction is "Walk into the kitchen area. Walk by the sink and oven. Walk straight into the hallway. Turn right into the little room. Turn left and walk into the bedroom. Stop by the corner of the bed." (id: 155_0). The RecBERT fails to recognize the kitchen area and navigates back and forth in wrong locations. Our HAMT correctly recognizes the kitchen and follows the instruction.

( a )

 a Predicted trajectory by RecBERT [122] (failed). (b) Predicted trajectory by HAMT (succeed).

Figure 4 . 8 :

 48 Figure 4.8: Examples in R2R val unseen split. Navigation steps inside red box are incorrect. The instruction is "Walk straight until you get to a room that has a black table on the left with flowers on it. Wait there." (id: 4182_2). The RecBERT takes the wrong direction at the first step, while our HAMT follows the instruction and successfully stops.

( a )

 a Predicted trajectory by RecBERT [122] (failed). (b) Predicted trajectory by HAMT (succeed).

Figure 4 . 9 :

 49 Figure 4.9: Examples in R2R val unseen split. Navigation steps inside red box are incorrect. The instruction is "Walk out of the bathroom and turn right. Turn left and walk down the hallway.Turn right and stop by the end table." (id: 5153_0). The RecBERT correctly performs the first two turns but fails to track the third turn right action and stops incorrectly. Our HAMT is better to align the current state with the instruction to correctly perform the third turn right action.

  Predicted trajectory by HAMT (failed).

Figure 4 . 10 :

 410 Figure 4.10: Failure cases in R2R val unseen split. The instruction is "Go stand underneath the stairs, next to the liquor shelf. " (id: 36968_2). Though HAMT correctly goes towards the direction, it fails to recognize the liquor shelf and results in exploring further the room until reaching the maximum number of navigation steps.

( a )

 a Groundtruth trajectory. (b) Predicted trajectory by HAMT (failed).

Figure 4 . 11 :

 411 Figure 4.11: Failure cases in R2R val unseen split. The instruction is "With the low stone or concrete barrier behind you, walk parallel to the board covering the floor and turn left before reaching the end. Move forward to leave the wooden flooring and when on the stone flooring, turn right and stand in front of the doors leading out of the room." (id: 5873_1). As the scene is unusual, HAMT fails to locate itself in the correct direction at the first step.

Figure 5 . 1 :

 51 Figure 5.1: Left: Hiveformer can adapt to perform 74 tasks from RLBench [143] given language instructions. Right: Multiple variations of the push buttons task.

Figure 5 . 2 :

 52 Figure 5.2: Hiveformer jointly models instructions, views from multiple cameras, and past actions and observations with a multimodal transformer for robotic manipulation.

1 n

 1 ∑ n l=1 xl , where Pr(m) ∈ [0, 1] N τ , and we obtain the offset as: p o t+1 = ∑ m Pr(m) • E O (m,t, :).

Figure 5 . 3 :

 53 Figure 5.3: The tasks used in multi-variation setting. Left: push buttons task. Right: tower task.

Figure 5 . 4 :

 54 Figure 5.4: The robot scene with two RGB-D cameras and a UR5 robotics arm with an RG2 gripper.

Figure 5 . 5 :

 55 Figure 5.5: The instruction is "Press the cyan button, and then press the rose one, and then press the purple one". Top row: sequence of observations from one of the two side cameras in the robot scene. Bottom row: sequence of predicted attention maps by our model that indicate the gripper's position for the next step.

( a )

 a Flipped button. (b) Multimodal. (c) Dynamic Environment. (d) Unseen Button. (e) Unseen Gripper. (f) Unseen Table.

Figure 5 . 6 :

 56 Figure 5.6: Robustness of the learned policy on an unseen variation: "Press the yellow button and then press the black button and finish with the white button".

  1. Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid.

	In Proceedings of the IEEE/CVF International Conference on Computer Vision.
	2021.
	2. Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. "History
	aware multimodal transformer for vision-and-language navigation." Advances in
	Neural Information Processing Systems. 2021.
	3. Pierre-Louis Guhur, Shizhe Chen, Ricardo Pinel, Makarand Tapaswi, Ivan Laptev,
	and Cordelia Schmid. "Instruction-driven history-aware policies for robotic
	manipulations". In Proceedings of the Conference on Robotics Learning (Oral).
	2022.

"Airbert: In-domain pretraining for vision-and-language navigation."

Table 3 . 1 :

 31 Comparing BnB to other existing VLN datasets. The #images from Matterport environments

Table 3 . 2 :

 32 Comparison between various BnB PI pair creation strategies for pretraining. The first row denotes the use of image-caption pairs. All methods from the second row use masking and shuffling during pretraining.

		BnB		Speaker	R2R		SR on Val
		Mask Shuf. Rank Shuf. Rank Shuf. Seen Unseen
	1	-	-	-	-	✓	-	70.20	59.26
	2	-	-	✓	✓	✓	✓	73.12	65.50
	3	✓	-	-	-	✓	-	73.24	64.21
	4	✓	✓	-	-	✓	-	73.57	66.52
	5	✓	✓	-	-	✓	✓	74.69	66.90
	6	✓	-	✓	-	✓	-	70.21	65.52
	7	✓	✓	✓	✓	✓	✓	73.83	68.67

Cat: naive concatenation; Rep: instruction rephrasing; Gen: instruction generation; Merge: image merging; and Insert: captionless image insertion.

Table 3 .

 3

3: Impact of shuffling during pretraining and fine-tuning. While additional data helps, we see that using the shuffling loss (abbreviated as Shuf.) consistently improves model performance. Row 1 corresponds to VLN-BERT

[START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF]

.

Table 3 . 4 :

 34 Comparison between different strategies for fine-tuning a ViLBERT model on the R2R task. VLN-BERT[START_REF] Majumdar | Improving vision-and-language navigation with image-text pairs from the web[END_REF] fine-tunes ViLBERT with a masking and ranking loss. Each row (described in the text) is an independent data augmentation and can be compared directly against the baseline (row 1).

	Model	Replace-Nouns Seen Unseen Seen Unseen Seen Unseen Swap-Nouns Directions
	VLN-BERT 60.3	58.7	53.4	52.3	46.2	45.3
	Airbert	68.3	66.6	66.6	61.1	47.3	49.8

Table 3 .

 3

5: Accuracy of models attempting to pick the correct PI pair from a pool of correct + 10 negatives created by simple corruptions such as replacing or swapping noun phrases and switching directions (left with right). Random performance is

1

11 or 9.1%.

Table 3 . 6 :

 36 loss applied during fine-tuning. The final stage of model training on Performance of single models and the impact of ensembling VLN-BERT or Airbert with the speaker and follower. MATTN [251] 50.53 55.17 45.50 16.35 31.97 29.66 14.40 28.20 7.19 45.28 7.84 4.67 19.88 30.63 11.61 39.05 11.28 6.08 Rec (OSCAR) [121] 39.85 41.32 35.86 12.85 24.46 22.28 25.53 27.66 21.06 14.35 14.20 12.00 24.62 26.67 19.48 14.88 12.65 10.00 Rec (ViLBERT) 43.64 45.61 37.86 15.75 31.69 27.58 24.57 29.91 19.81 17.83 15.14 12.15 22.17 25.51 17.28 18.22 12.87 10.00 Rec (VLN-BERT) 41.11 42.87 35.55 15.62 28.39 24.99 25.53 29.42 20.51 16.94 16.42 13.29 23.57 26.83 18.73 17.63 14.24 11.63 Rec (Airbert) 47.01 48.98 42.34 15.16 32.75 30.01 27.89 34.51 21.88 18.71 18.23 14.18 30.28 34.20 23.61 17.91 16.83 13.28

	Airbert	VLN-BERT Speaker Follower [215] [295] [295]	PL	Val Seen NE SPL OSR	SR	PL	Val Unseen NE SPL OSR	SR
	1	-	✓	-		-	10.28 3.73 0.66 76.47 70.20 9.60 4.10 0.55 69.22 59.26
	2	✓	-	-		-	10.59 3.21 0.69 80.71 73.85 10.03 3.24 0.63 78.45 68.67
	3	-	-	✓		✓	10.69 2.72 0.70 82.94 74.22 10.10 3.32 0.63 76.63 67.90
	4	-	✓	✓		✓	10.61 2.35 0.78 86.57 81.86 10.00 2.76 0.68 81.91 73.61
	5	✓	-	✓		✓	10.63 2.13 0.77 87.17 81.40 9.99 2.69 0.70 82.89 75.01
				Validation Seen				Validation Unseen			Test Unseen
	Methods	SR	Navigation OSR SPL	TL	RGS RGSPL	SR	Navigation OSR SPL	TL	RGS RGSPL	SR	Navigation OSR SPL	TL	RGS RGSPL
	Seq2Seq-SF [13]	29.59 35.70 24.01 12.88 18.97 14.96	4.20 8.07 2.84 11.07 2.16	1.63	3.99 6.88 3.09 10.89 2.00	1.58
	RCM [321]	23.33 29.44 21.82 10.70 16.23 15.36	9.29 14.23 6.97 11.98 4.89	3.89	7.84 11.68 6.67 10.60 3.67	3.14
	SMNA [212]	41.25 43.29 39.61 7.54 30.07 28.98	8.15 11.28 6.44 9.07 4.54	3.61	5.80 8.39 4.53 9.23 3.10	2.39
	FAST-

Table 3 .

 3

[START_REF] Bou Ammar | Online multi-task learning for policy gradient methods[END_REF]

: Navigation and object localization performance on the REVERIE dataset, including results on the unseen test set (leaderboard).

Table 3 . 8 :

 38 Navigation performance on the R2R unseen test set as indicated on the benchmark leaderboard.

			Test Unseen
		PL	NE SPL OSR SR
	Speaker-Follower [85] 1,257 4.87 0.01	96	53
	PreSS [188]	10.5 24.5 0.63	57	53
	PREVALENT [106]	10.21 4.52 0.56	64	59
	Self-Monitoring [212]	373	4.48 0.02	97	61
	Reinforced CM [321]	358	4.03 0.02	96	63
	EnvDrop [13]	687	3.26 0.01	99	69
	AuxRN [362]	41	3.24 0.21	81	71
	VLN-BERT [215]	687	3.09 0.01	99	73
	Airbert (ours)	687	2.69 0.01	99	77

Table 3 . 9 :

 39 Navigation performance of different generative models on the R2R dataset for the validation sets.

	Methods	TL	Test Unseen NE SR SPL
	Seq2Seq-SF [13]	8.13 7.85 20	18
	Speaker-Follower [85]	14.82 6.62 35	28
	PRESS [188]	10.77 5.49 49	45
	EnvDrop [295]	11.66 5.23 51	47
	PREVALENT [106]	10.51 5.30 54	51
	Rec (no init. OSCAR) [121] 11.15 5.45 51	47
	Rec (OSCAR) [121]	12.34 4.59 57	53
	Rec (PREVALENT) [121]	12.35 4.09 63	57
	Rec (Airbert)	12.41 4.13 62	57

Table 3 .

 3 10: Navigation performance of different generative models on the R2R dataset for the testing set.

	# Env. Traj.	PL	Val Seen SR NE SPL OSR	SR	PL	Val Unseen SR NE SPL OSR	SR
	1	Rand 10.97 5.36 0.44 63.74 47.87 ±0.03 10.84 4.86 0.51 68.46 54.48 ±0.04
	6	Rand 9.84 5.49 0.47 65.93 50.00 ±0.02 9.55 4.55 0.55 70.89 57.97 ±0.01
	61	Rand 10.91 4.87 0.60 76.23	64.24	9.50 3.70 0.62 76.24	65.60
	61	[295] 10.59 3.21 0.69 80.71	73.85	10.03 3.24 0.63 78.45	68.67

Table 3 .

 3 11: Performance of Airbert on R2R few-shot evaluation. During training, only a subset of the Matterport [44] environments are accessible.

  .

	Observation Encoding. For each view [v o i ; a o i ] in the panoramic observation O t , we first represent the relative angle a o i as E A a o i

  As H t consists of all the past panoramic observations O i and performed actions a h i before step t, it is important to encode H t efficiently as context. Figures 4.2b-4.2c depict the flattened and temporal-only history encoding approaches used in VLN-BERT

	and E N 2 for stop view (we append a stop token in observation to support stop action). The E T 1 is the type embedding of observation. We omit bias terms for simplicity. The LN denotes layer normalization [23]. Because a o i has much lower feature dimensions than v o i , we apply LN to balance the encoded a o i and v o i .
	Hierarchical History Encoding.
	.1)
	where W o v ,W o a are learnable weights. The E N o i denotes the navigable embedding to differentiate types of views, with E N 0 for non-navigable view, E N 1 for navigable view

Table 4 . 1 :

 41 Comparison of HAMT and previous VLN transformers.

	Models	Inputs Text History Observation MLM MRM ITM SAP/SAR SPREL Proxy Tasks
	PREVALENT [107] ✓

  builds upon Matterport3D[43] and includes 90 photo-realistic houses with 10,567 panoramas. It contains 7,189 shortest-path trajectories, each associated with 3 instructions. The dataset is split into train, val seen, val unseen and test unseen sets with 61, 56, 11 and 18 houses respectively. Houses in val seen split are the same as training, while houses in val unseen and test splits are different from training.• RxR[START_REF] Ku | Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding[END_REF] is a large multilingual VLN dataset based on Matterport 3D. The instructions are in three different languages (English, Hindi and Telugu). The dataset emphasizes the role of language in VLN by addressing biases in paths and describing more visible entities than R2R. • R4R[START_REF] Jain | Stay on the path: Instruction fidelity in vision-and-language navigation[END_REF] extends R2R dataset by concatenating two adjacent tail-to-head trajectories in R2R. Therefore, it has longer instructions and trajectories. The trajectories are also less biased as they are not necessarily the shortest-path from start to end location. • R2R-Back is a new VLN setup proposed in this chapter. The agent is required

Table 4 . 2 :

 42 Dataset statistics. #traj, #instr denote the number of trajectories and instructions respectively.

	Dataset	Train #traj #instr #traj #instr #traj #instr #traj #instr Val Seen Val Unseen Test Unseen
	R2R [14]	4,675 14,039 340 1,021 783 2,349 1,391 4,173
	RxR [169]	11,077 79,467 1,244 8,813 1,517 13,652	-	11,888
	R4R [139]	25,921 233,532 115 1,035 5,026 45,234	-	-
	R2R-Back	4,675 14,039 340 1,021 783 2,349	-	-
	CVDN [303]	4,742 4,742	382 382 907	907 1,384 1,384
	R2R-Last	4,675 14,039 340 1,021 783 2,349	-	-
	REVERIE [251] 4,150 10,466 515 1,423 1,328 3,521 2,304 6,292

Table 4 . 3 :

 43 R2R navigation results for alternative methods of history encoding. All methods use Resnet152 visual features and are trained from scratch on R2R dataset. ±1.0 56.6 ±1.1 52.2 ±0.7 47.0 ±0.5 Temporal-only 61.5 ±0.8 57.7 ±0.7 53.2 ±0.1 48.0 ±0.4 Hierarchical 65.5 ±1.2 61.3 ±1.4 54.4 ±0.4 48.7 ±0.4

	History	Val Seen	Val Unseen
	Encoding	SR↑	SPL↑	SR↑	SPL↑
	RecBERT [122]	62	59	50	46
	Recurrent	60.9

Table 4 . 4 :

 44 Ablations for end-to-end HAMT training on R2R dataset using proposed proxy tasks. ±1.2 61.3 ±1.4 54.4 ±0.4 48.7 ±0.4 ✓ × 69.3 ±1.0 64.8 ±1.2 63.5 ±0.5 57.5 ±0.5 ViT ✓ × 75.7 ±1.0 72.5 ±1.0 64.4 ±0.3 58.8 ±0.0 ✓ ✓ 75.0 ±0.9 71.7 ±0.7 65.7 ±0.7 60.9 ±0.7 ±2.3 67.2 ±2.0 62.8 ±1.3 57.7 ±1.0 ✓ × 74.7 ±0.6 71.1 ±0.9 63.6 ±0.1 58.1 ±0.4 ✓ ✓ 75.7 ±1.0 72.5 ±1.0 64.4 ±0.3 58.8 ±0.0

	(a) Comparison of visual features and end-to-end train-ing. The "PT" stands for proxy tasks in training; "e2e" for optimizing the visual representation. feature PT e2e Val Seen Val Unseen SR↑ SPL↑ SR↑ SPL↑ Resnet × × 65.5 (b) Comparison of different proxy tasks. The "SAP(R)" denotes the single step action prediction and regression task, and "SPREL" is the spatial rela-tionship prediction task. SAP SP Val Seen Val Unseen (R) REL SR↑ SPL↑ SR↑ SPL↑
	152	×	× 71.2

Table 4 . 5 :

 45 Ablations for fine-tuning objectives of sequential action prediction on R2R dataset. ±2.1 61.7 ±2.2 57.2 ±0.1 54.7 ±0.3 × ✓ 70.5 ±2.9 65.6 ±2.8 63.5 ±1.4 57.5 ±1.1 ✓ ✓ 75.0 ±0.9 71.7 ±0.7 65.7 ±0.7 60.9 ±0.7

	IL RL	Val Seen SR↑ SPL↑	Val Unseen SR↑ SPL↑
	× ×	57.9	54.8	51.8	48.9
	✓ × 63.7

Table 4 . 6 :

 46 Computation time in inference on R2R val unseen split.

		Inference Time (s)	SR SPL
	RecBERT [122]	69	63	57
	HAMT	104	66	61
	HAMT noT2V	76	65	60

Table 4 . 7 :

 47 Comparison with state-of-the-art methods on R2R dataset.

	Methods	Validation Seen TL NE↓ SR↑ SPL↑ TL NE↓ SR↑ SPL↑ TL NE↓ SR↑ SPL↑ Validation Unseen Test Unseen
	Seq2Seq [14]	11.33 6.01 39	-	8.39 7.81 22	-	8.13 7.85 20	18
	SF [85]	-	3.36 66	-	-	6.62 35	-	14.82 6.62 35	28
	PRESS [189]	10.57 4.39 58	55 10.36 5.28 49	45 10.77 5.49 49	45
	EnvDrop [296]	11.00 3.99 62	59 10.70 5.22 52	48 11.66 5.23 51	47
	AuxRN [363]	-	3.33 70	67	-	5.28 55	50	-	5.15 55	51
	PREVALENT [107] 10.32 3.67 69	65 10.19 4.71 58	53 10.51 5.30 54	51
	RelGraph [118]	10.13 3.47 67	65	9.99 4.73 57	53 10.29 4.75 55	52
	RecBERT [122]	11.13 2.90 72	68 12.01 3.93 63	57 12.35 4.09 63	57
	HAMT (Ours)	11.15 2.51 76	72 11.46 2.29 66	61 12.27 3.93 65	60

Table 4 . 8 :

 48 3 . It demonstrates the effectiveness and generalization of our model. Comparison on R4R val unseen split.

	Methods	NE↓ SR↑ CLS↑ nDTW↑ SDTW↑
	SF [85]	8.47	24	30	-	-
	RCM [321]	-	29	35	30	13
	PTA [175]	8.25	24	37	32	10
	EGP [67]	8.0 30.2 44.4	37.4	17.5
	RelGraph [118]	7.43	36	41	47	34
	RecBERT † [122] 6.67 43.6 51.4	45.1	29.9
	HAMT (Ours)	6.09 44.6 57.7	50.3	31.8

Table 4 . 9 :

 49 Comparison of methods on the R2R-Back dataset.

	Methods	Val Seen TL SR↑ SPL↑ nDTW↑ SDTW↑ TL SR↑ SPL↑ nDTW↑ SDTW↑ Val Unseen
	EnvDrop † [296] 23.83 44.1 42.0	61.3	39.4	24.57 32.4 30.2	51.1	28.0
	RecBERT † [122] 22.33 51.4 48.4	67.3	45.7	23.35 41.1 37.7	58.2	35.6
	HAMT (Ours)	22.76 64.8 61.8	73.7	58.9	23.78 57.2 53.1	65.1	49.5

Table 4 .

 4 10: Navigation performance on CVDN dataset.

		Val Seen Val Unseen Test Unseen
	PREVALENT [107]	-	3.15	2.44
	VISITRON [277]	5.11	3.25	3.11
	MT-RCM+EnvAg [323]	5.07	4.65	3.91
	HAMT (Ours)	6.91	5.13	5.58

Table 4 .

 4 11: Comparison on the R2R-Last dataset.

	Methods	Val Seen SR↑ SPL↑ SR↑ SPL↑ Val Unseen
	EnvDrop † [296]	42.8 38.4 34.3 28.3
	RecBERT † [122] 50.2 45.8 41.6 37.3
	HAMT (Ours)	53.3 50.3 45.2 41.2

Table 4 .

 4 12: Navigation performance on RxR test split.

		PL	SR↑ SPL↑ nDTW↑ SDTW↑
	Multilingual Baseline [169] 16.88 20.98 18.55	41.05	20.59
	Monolingual Baseline [169] 17.05 25.40 22.59	41.05	20.59
	CLIP-ViL	15.43 38.34 35.17	51.10	32.42
	CLEAR-CLIP	16.46 40.29 36.57	53.69	34.86
	Multilingual HAMT	19.77 53.12 46.62	59.94	45.19
	Human	20.78 93.92 74.13	79.48	76.90

SPL 30.20 and 26.67 on val unseen and test splits respectively, outperforming the state of the art navigation performance

[START_REF] Hong | Vln bert: A recurrent vision-and-language bert for navigation[END_REF]

by 5.3% and 2.7%.

Table 4 .

 4 [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF]: Navigation performances on RxR val seen and val unseen splits.

				Val Seen		Val Unseen
		SR↑ SPL↑ nDTW↑ SDTW↑ SR↑ SPL↑ nDTW↑ SDTW↑
	Multilingual Baseline [169] 25.2	-	42.2	20.7 22.8	-	38.9	18.2
	Monolingual Baseline [169] 28.8	-	46.8	23.8 28.5	-	44.5	23.1
	Multilingual HAMT	59.4 58.9	65.3	50.9 56.5 56.0	63.1	48.3

Table 4 .

 4 14: Navigation and object grounding performances on REVERIE val unseen and test splits. MATTN [251] 45.28 14.40 28.20 7.19 7.84 4.67 39.05 19.88 30.63 11.6 11.28 6.08 SIA [196] 41.53 31.53 44.67 16.28 22.41 11.56 48.61 30.80 44.56 14.85 19.02 9.20 RecBERT [122] 16.78 30.67 35.02 24.90 18.77 15.27 15.86 29.61 32.91 23.99 16.50 13.51 HAMT 14.08 32.95 36.84 30.20 18.92 17.28 13.62 30.40 33.41 26.67 14.88 13.08

		Validation Unseen		Test Unseen
	Methods	Navigation	Grounding	Navigation	Grounding
		TL SR↑ OSR↑ SPL↑ RGS↑ RGSPL↑ TL SR↑ OSR↑ SPL↑ RGS↑ RGSPL↑
	Seq2Seq [14]	11.07 4.20 8.07 2.84 2.16	1.63 10.89 3.99 6.88 3.09 2.00	1.58
	RCM [321]	11.98 9.29 14.23 6.97 4.89	3.89 10.60 7.84 11.68 6.67 3.67	3.14
	SMNA [212]	9.07 8.15 11.28 6.44 4.54	3.61	9.23 5.80 8.39 4.53 3.10	2.39
	FAST-

Table 4 .

 4 15 provides an additional experiment in the third row compared to Table 4.4a. It demonstrates that ViT features outperform ResNet152 features with and without training proxy tasks. Comparing the last two rows in Table

Table 4 .

 4 15: Comparison of features (same notations as Table 4.4a).

		Val Seen Val Unseen
	Features PT e2e SR SPL SR SPL
	Resnet	× × 65.5 61.3 54.4 48.7
	152	✓ × 69.3 64.8 63.5 57.5
		× × 68.8 66.1 56.3 52.5
	ViT	✓ × 75.7 72.5 64.4 58.8
		✓ ✓ 75.0 71.7 65.7 60.9

Table 4 .

 4 [START_REF] Antol | VQA: Visual Question Answering[END_REF]: Comparison of different proxy tasks in end-to-end optimization.

			Val Seen	Val Unseen
	SAP(R) SPREL SR SPL SR SPL
	×	×	70.1 65.9 63.3 57.7
	✓	×	72.5 69.2 64.5 59.4
	✓	✓	75.0 71.7 65.7 60.9

Table 4 .

 4 17: Navigation results for R2R-Back dataset.

	History Encoding	e2e	Val Seen TL SR↑ SPL↑ nDTW↑ SDTW↑ TL SR↑ SPL↑ nDTW↑ SDTW↑ Val Unseen
	Recurrent	× 22.33 51.4 48.4	67.3	45.7	23.35 41.1 37.7	58.2	35.6
	Temporal-only × 22.70 51.6 49.6	67.8	46.7	22.93 45.1 42.9	62.7	40.2
	Hierarchical	× 23.52 66.8 63.5	73.8	60.4	24.58 56.5 51.7	63.6	48.4
	Hierarchical ✓ 22.76 64.8 61.8	73.7	58.9	23.78 57.2 53.1	65.1	49.5

Table 4 . 18 :

 418 Comparison of using different tokens in f SAP in fine-tuning.

	Action Prediction Token	Val Seen Val Unseen
	obs txt hist SR↑ SPL↑ SR↑ SPL↑
	✓ ×	×	76.1 72.8 66.0 60.3
	✓ ✓	×	75.0 71.7 65.7 60.9
	✓ ×	✓	78.0 75.9 65.5 60.2
	✓ ✓	✓	76.3 73.4 65.5 60.9

  Proprioception. We encode the RGB image I k t , point clouds P k t , and proprioception A k t for each camera k separately. A k t ∈ {0, 1} H×W is a binary attention map used to encode the position of the gripper p t . It takes value one at the location of the gripper center and zeroes elsewhere. We concatenate I k t and A k t in the channel dimension and use a UNet encoder to obtain a feature map Fk t ∈ R H v ×W v ×d v , where H v ,W v , d v are the height, width, and the number of channels of the feature map. Next, we concatenate Fk t with point cloud representations in the channel dimension to indicate the spatial location of each patch in the feature map. To match the size of P k t and Fk t , we apply mean-pooling to P k t . The final encoded feature map

  • The Screw group requires screwing an object. The included tasks are: turn oven on, change clock, open window, open wine bottle. • The Visual Occlusion group involves tasks with large objects and thus there are occlusions from certain views. The included tasks are: close microwave, close fridge, close grill, open grill, unplug charger, press switch, take money out safe, open microwave, put money in safe, open door, close door, open fridge, open oven, plug charger in power supply.

Table 5 . 1 :

 51 Success rate on the single-task setting. Variance is obtained with demonstrations generated with 5 random seeds.

		Inputs			Transformer		Training
	Visual Tokens	Point Clouds	Gripper Position	Multi-View	History	Attn	Mask Obs	SR
	R1

Table 5 . 2 :

 52 Comparison with state-of-the-art methods on 10 tasks. We report success rate (%).

		Pick & Lift	Pick-Up Cup	Push Button	Put Knife	Put Money	Reach Target	Slide Block	Stack Wine	Take Money	Take Umbr.	Avg.
	Single-task learning
	ARM [142]	70	80	-	-	-	100	-	70	-	70	-
	Auto-λ [200]	82	72	95	36	31	100	36	23	38	37	55.0
	Ours	93.8	82.7	99.5 69.9	96.3 100.0 95.3 82.1 82.3	90.7 88.3
	Multi-task learning
	Auto-λ [200]	87	78	95	31	62	100	77	19	64	80	69.3
	Ours (w/o inst) 84.0	13.8	97.6 41.8	54.2	98.8 36.0 68.8 74.6	72.6 64.6
	Ours	88.8	93.0	100.0 75.0	58.0 100.0 79.8 70.4 79.0	89.2 83.3

Table 5 . 3 :

 53 Comparison with state-of-the-art on 74 RLBench tasks grouped into 9 categories. We report success rate (%) for the single-task setting. * The performance of Auto-λ is obtained by running their code.

		Planning Tools	Long Term	Rot. Invar.	Motion Planning	Screw	Multi Modal	Precision	Visual Occlusion	Avg
	Num. of tasks	9	11	4	7	9	4	5	11	14	74
	Auto-λ [200] *	58.9	20.0 2.3 73.1	66.7	48.2	47.6	34.6	40.6	44.0
	Ours (w/o hist)	78.9	46.7 10.0 84.6	73.3	72.6	60.0	63.8	57.9	60.9
	Ours (one view)	57.7	23.2 12.3 57.8	63.2	35.6	40.7	33.7	37.1	40.1
	Ours	81.6	53.0 16.9 84.2	72.7	80.9	67.1	64.7	60.2	65.4

Table 5 . 4 :

 54 Comparison with LanCon-Learn [279] on 10 tasks.

		Hist.	Pick & Lift	Pick-Up Cup	Push Button	Put Knife	Put Money	Reach Target	Slide Block	Stack Wine	Take Money	Take Umbr.	Avg.
	Single-task learning
	LanCon-Learn -	20.2	25.2	96.2 57.8 91.4 99.6 60.2 57.0 58.4 73.0 63.9
	LanCon-Learn ✓ 64.8	56.8	96.4 59.4 90.6 98.7 63.4 56.6 67.8 74.8 72.9
	Ours	✓ 93.8	82.7	99.5 69.9 96.3 100.0 95.3 82.1 82.3 90.7 88.3
	Multi-task learning
	LanCon-Learn -	18.2	23.2	80.2 28.8 59.6 100.0 38.8 25.2 58.2 45.6 47.8
	LanCon-Learn ✓ 52.6	44.2	81.5 32.2 75.6 100.0 42.2 24.6 70.2 50.8 57.4
	Ours	✓ 88.8	93.0 100.0 75.0 58.0 100.0 79.8 70.4 79.0 89.2 83.3

Table 5 . 5 :

 55 Comparison with LanCon-Learn[START_REF] Silva | Lancon-learn: Learning with language to enable generalization in multi-task manipulation[END_REF] and ablation of the instruction encoding in the multi-variation setting for seen or unseen variations and synthetic, corrupted or real instructions.

				Instructions		Visual		Push buttons	Tower
	Method	Hist.	Format Encoder	Emb. E x T	Emb. E v T	Seen Synt. Synt. Corr. Real Synt. Synt. Corr. Real Unseen Seen Unseen
	LanCon-Learn No	Cat.	GloVe	-	-	25.6 12.1 0.3 0.1 21.3 9.1	0.1 0.0
	LanCon-Learn Yes	Cat.	GloVe	-	-	37.8 16.7 1.6 0.9 34.9 14.2 1.2 0.8
	Ours	No	Seq.	CLIP	✓	✓	8.6	3.6	0.3 0.1 7.1	4.5	0.2 0.0
	Ours	Yes Avg.	CLIP	✓	✓	9.1	1.1	0.0 0.0 5.3	0.2	0.0 0.0
	Ours	Yes	Seq.	CLIP	-	✓	100 83.2 81.1 71.3 77.1 53.2 51.3 21.3
	Ours	Yes	Seq.	CLIP	-	-	86.2 65.2 56.4 49.8 54.9 34.8 29.8 24.7
	Ours	Yes	Seq.	BERT	✓	✓	54.6 40.2 15.6 21.8 42.9 28.9 8.2 10.2
	Ours	Yes	Seq.	OHE	✓	✓	100 3.1	0.1 0.0 96.8 3.8	0.4 0.0
	Ours	Yes	Seq.	CLIP	✓	✓	100 86.3 85.6 74.2 77.4 56.2 53.6 24.1

Table 5 . 6 :

 56 Success rate (%) in the multi-variation setting for seen or unseen variations and synthetic or real instructions.

	# Demos	Push Buttons			Tower
	Per	Seen	Unseen	Seen	Unseen
	Variation	Synt.	Synt.	Real	Synt.	Synt.	Real
	10	96.4	71.1	65.7	71.6	49.8	19.4
	50	99.4	83.1	70.9	74.3	52.1	20.6
	100	100	86.3	74.2	77.4	56.2	24.1

Table 5 . 7 :

 57 Success rate of push buttons task on real robots.

	Pretrain Seen Vars Unseen Vars
	-	86.7	13.3
	✓	92.2	85.7

Bed and Breakfast

Airbert is an Old Irish word meaning practice, here referring to model pretraining on pretext tasks similar to VLN.[START_REF] Asada | Robocup: Today and tomorrow-what we have learned[END_REF]

While outdoor images may contain interesting features (e.g. a patio), we observe that removing them increases performance.

https://eval.ai/web/challenges/challenge-page/97/overview also shows performance for ensembling Airbert, VLN-BERT, speaker and follower at a unseen test set SR of 78%.

The performance of VLN-BERT on the seen validation set is lower because the model checkpoint is selected to maximize performance on validation unseen set which happens to be at an earlier iteration.

A navigable view can lead to one or multiple viewpoints. We follow[122,

296] to use different features for these viewpoints. The viewpoints share the same visual features but differ in angle features.

https://spacy.io/

We report the published results on the testing unseen split as shown in https://eval.ai/web/ challenges/challenge-page/97/leaderboard/270 (25/10/2021).

https://ai.google.com/research/rxr/competition?active_tab= leaderboard (25/10/2021).

We follow definitions in RLBench[START_REF] James | RL-Bench: The robot learning benchmark & learning environment[END_REF] for tasks and variations. A task can be composed of multiple variations that share the same skills but differ in objects, attributes or order as shown in Figure5.1 (right).

https://github.com/stepjam/RLBench/tree/master/rlbench/tasks

Acknowledgments

Acknowledgments v

MOTS CLÉS

Robotique, vision par ordinateur, traitement du langage naturel, transformer RÉSUMÉ Les progrès réalisés dans le domaine de l'apprentissage automatique ont permis des percées importantes, notamment en vision par ordinateur, en traitement du langage naturel et en robotique. Peut-on aller plus loin et combiner ces domaines de recherche ? Cela développera de nouvelles applications, comme la robotique guidée par le langage, où un robot doit suivre les instructions fournies par un opérateur. Alors que les humains apprennent à suivre des instructions dès leur enfance, la même tâche est difficile pour des robots, et cela pour plusieurs raisons: (i) le manque de données d'entraînements, (ii) les raisonnements faits sur multiples niveaux d'abstraction, et (iii) l'espace d'actions ayant une haute dimension. L'objectif de cette thèse est d'améliorer la robotique guidée par le langage en relevant ces défis. Nous décomposons la difficulté de la robotique guidée par le langage en considérant deux types de tâches : (i) un robot mobile doit se rendre à un endroit cible décrit par des instructions ; (ii) les instructions décrivent une séquence d'actions qu'un bras robotique doit opérer sur des objets placés sur une table. Nos contributions sont les suivantes : (i) pour résoudre le manque de données d'entraînement, nous avons développé une procédure efficace de pré-entraînement basé sur le nouveau jeu de données BnB, (ii) nous avons construit de nouvelles architectures neuronales basées sur une approche hiérarchique pour encoder plusieurs niveaux d'abstractions, et (iii) nous avons proposé une nouvelle méthode pour prédire des actions continus et en sur plusieurs dimensions pour résoudre un grand nombre de tâches.

ABSTRACT

Recent progress in machine learning has enabled groundbreaking improvements notably in computer vision, natural language processing, and robotics. Can we go one step further and combine these research fields? This would allow new applications, such as language-guided robotics, where a robot must follow instructions provided by an operator. While people learn to follow natural language instructions from their childhood, the same task is difficult for robots. Current challenges include (i) the limited amount of training data, (ii) the multiple levels of reasoning, and (iii) the multi-dimensional continuous action space. The goal of this thesis is to improve language-guided robotics by addressing these challenges. We break down the difficulty of language-guided robotics by considering two types of tasks: (i) vision-and-language navigation, where a mobile robot must go to a target location, and (ii) vision-and-language manipulation, where a robotic arm should manipulate objects on a tabletop. Our contributions are the following: (i) we address the scarcity of training data and develop an efficient pre-training procedure based on the new BnB dataset, (ii) we propose a hierarchical approach based on the Transformer architecture to encode several layers of abstractions, and (iii) we propose a new method predicting continuous and multi-dimensional actions for solving a large number of robotics tasks on a tabletop. Methods developed in this thesis have been tested in photo-realistic simulators and on a real-world robot. They have outperformed the state-of-the-art performance on a dozen of benchmarks.

KEYWORDS

Robotics, computer vision, natural language processing, transformer