Edward Beeching

Tom Gillooly

Arthur Aubret

Assem Sadek

Pierre Marza

Steeven Janny

Quentin Possamaï

Échanger

Céline Hudelot

David Auber D'

Liming Chen

REMERCIEMENTS

Premièrement, j'ai une grande reconnaissance envers mes encadrants de thèse Christian Wolf et Romain Vuillemot qui m'ont fait confiance il y a de ça 5 ans pour un stage de fin d'études qui m'a permis d'arriver ou j'en suis aujourd'hui. Merci pour votre enthousiasme, encouragements, et conseils qui ont rythmé notre temps passé ensembles, ces restaurants célébrant l'acceptation d'un manuscrit, et plus simplement nos échanges joviaux autour d'une bière. Vous avez sus avec aise aller au-delà de vos domaines respectifs afin d'en enrichir leurs intersections. Je n'aurais pas pu avoir de meilleurs encadrants de thèse.

Je tiens à remercier toutes les personnes avec qui j'ai l'occasion de collaborer tout au long de ma thèse ; Corentin Kervadec, Moez baccouche, Grigory Antipov, Guillaume Bono, Edward Beeching, et Sylvain Lesage. Travailler avec vous fut un plaisir qui m'a permis d'apporter de nouvelles perspectives sur ma façon de travailler.

A B S T R A C T

In the last couple of years, Artificial Intelligence (AI) and Machine Learning have evolved, from research domains addressed in laboratories far from the public eye, to technology deployed on industrial scale widely impacting our daily lives. This trend has started to raise legitimate concerns, as it is also used to address critical problems like finance and autonomous driving, in which decisions can have a life-threatening impact. Since a large part of the underlying complexity of the decision process is learned from massive amounts of data, it remains unknown to both the builders of those models and to the people impacted by them how models take decisions. This led to the new field of eXplainable AI (XAI) and the problem of analyzing the behavior of trained models to shed their reasoning modes and the underlying biases they are subject to. This thesis we contributed to this emerging field with the design of visual analytics systems tailored to the study and improvement of interpretability of Deep Neural Networks. Our goal was to empower experts with tools helping them to better interpret the decisions of their models. We also contributed with explorable applications designed to introduce Deep Learning methods to non-expert audiences. Our focus was on the under-explored challenge of interpreting and improving models for different applications such as robotics, where important decisions must be taken from high-dimensional and low-level inputs such as images. All tools designed during this thesis were published as open-source projects, and when possible, our visualizations have been made available online as prototypes, as they are highly interactive and thus can help to foster further research by the research community.

In the first part of this thesis, we addressed how a robot may answer natural questions (Visual Question Answering (VQA)) with state-of-the-art transformer models based on self-attention. Our focus was on the interpretation of their attention maps which we analyzed with a visual analytics system we designed. We showed how our system can be used by model builders to detect when their model may exploit biases, i.e. undesired shortcuts in learning by exploiting spurious correlations in data, instead of resorting to the requested reasoning process.

In a second part, we addressed the interpretability of robot navigation, i.e. how a robot may explore its environment (e.g. an office). In our case, such a step was addressed by neural models which include a recurrent memory, which is at the core of their decisions. We focused on how this memory (and its impact on decisions) can be interpreted. We designed a new visual analytics system to account for the challenge of analyzing this type of neural model.

iii abstract The third part of this thesis focused on the interpretability of robotics systems trained in simulation and deployed to real physical environments, which has now quickly become a standard mode of operations. We focused on the tasks of robot ego-localization and the interpretations of its decisions. We provided an interpretation of potential sources of gaps with the help of a new visual analytics interface, with which we identified biases and gaps that affect models' decisions.

Finally, we concluded this thesis with a reflection across these parts to highlight the challenges of building an end-to-end model to robotic task of answering natural language questions using navigation in a real-world environment. This conclusion also introduce novel and still open research directions to provide better interpretability of such models' decisions and mitigate their bias.

R É S U M É

Au cours des dernières années, l'IA et l'apprentissage automatique ont évolué, passant de domaines de recherche reclus dans des laboratoires éloignés du public à des technologies déployées à l'échelle industrielle ayant un impact considérable sur notre vie quotidienne. Étant donné que ces technologies sont également utilisées pour résoudre des problèmes critiques tels que la finance et la conduite de voitures autonomes, dans lesquels leurs décisions peuvent mettre des personnes en danger, cette tendance a commencé à susciter des inquiétudes légitimes. Puisqu'une grande partie de la complexité sous-jacente du processus de décision est apprise à partir de quantités massives de données, la manière dont ces modèles prennent des décisions reste inconnue tant pour leurs créateurs que pour les personnes impactés par de telles décisions. Cela a conduit au nouveau domaine de l'eXplainable AI (XAI) et au problème de l'analyse du comportement des modèles entraînés, pour mettre en lumière leurs capacité raisonnement et les biais auxquels ils sont soumis. Dans cette thèse, nous avons contribué à ce domaine émergent avec la conception de systèmes d'analyse visuelle destinés à l'étude et à l'amélioration de l'interprétabilité des réseaux de neurones profonds. Notre objectif était de permettre aux experts de disposer d'outils les aidant à mieux interpréter les décisions de leurs modèles et éventuellement des les améliorer. Nous avons également proposé des applications explorables conçues pour présenter des méthodes d'apprentissage profond à un public non-expert. Durant cette thèse, nous nous sommes concentrés sur le défi sous-exploré de l'interprétation et de l'amélioration des modèles pour différentes applications de la robotique, où des décisions importantes doivent être prises à partir de données d'entrées de haut niveau et de haute dimension telles que des images. Tous les outils conçus au cours de cette thèse ont été publiés en tant que projets open-source, et lorsque cela était possible, nos visualisations ont été mises à disposition en ligne en tant que prototypes. Ces outils sont hautement interactifs et peuvent contribuer à encourager la poursuite de l'interprétation de décisions de modèles par la communauté de recherche.

Dans la première partie de cette thèse, nous avons abordé la manière dont un robot peut répondre à des questions naturelles (VQA) avec des modèles état de l'art transformer basés sur attention. Nous nous sommes concentrés sur l'interprétation de leurs cartes d'attention que nous avons analysé avec un système d'analyse visuelle que nous avons conçu. Nous avons montré comment notre système peut être utilisé par les constructeurs de modèles pour détecter lorsque ceux-ci sont suceptibles exploiter des biais. C'est-à-dire des raccourcis indésirables dans l'apprentissage en exploitant des corrélations trompeuses dans les données, au lieu de recourir au processus de raisonnement demandé. v vi résumé Dans une deuxième partie, nous avons abordé l'interprétabilité de la navigation des robots, c'est-à-dire la manière dont un robot peut explorer son environnement (par exemple un bureau). Dans notre cas, une telle étape a été traitée par des modèles neuronaux qui incluent une mémoire récurrente, qui est au coeur de leurs décisions. Nous nous sommes concentrés sur la façon dont cette mémoire et son impact sur les décisions peut être interprétés. Nous avons conçu un nouveau système d'analyse visuelle pour tenir compte du défi que représente l'analyse de ce type de modèle récurrent.

La troisième partie de cette thèse s'est centrée sur l'interprétabilité des systèmes robotiques entraînés en simulation et déployés dans des environnements physiques réels, ce qui est rapidement devenu un mode d'opération standard. Nous nous sommes concentrés sur la tâche d'ego-localisation du robot et les interprétations de ses décisions. Nous avons fourni une interprétation des sources potentielles de différence entre simulation et réalité à l'aide d'une nouvelle interface d'analyse visuelle, avec laquelle nous avons également identifié des biais qui affectent les décisions de ces modèles.

Enfin, nous avons conclu cette thèse par une réflexion sur l'ensemble de ces parties pour mettre en évidence les défis de la construction d'un modèle de bout en bout pour la tâche robotique de réponse aux questions en langage naturel via une navigation dans un environnement réel. Cette conclusion présente également les nouvelles directions de recherche encore ouvertes pour fournir une meilleure interprétabilité des décisions de ces modèles et atténuer leur biais. Examples of failures of Deep Learning (DL) models:

C O N T E N T S

x Pulse [START_REF] Menon | Pulse: Self-supervised photo upsampling via latent space exploration of generative models[END_REF], with an input image of the former US president Barack Obama, the model tasked to reproduce it, outputs the image of a Caucasian man. y the CLIP model [START_REF] Radford | Learning Transferable Visual Models From Natural Language Supervision[END_REF], can be induced to output wrong classification by simply taping tags to objects, e.g. as illustrated here, the apple is predicted as "library" [START_REF] Goh | Multimodal Neurons in Artificial Neural Networks[END_REF]. z, traffic signs detection models can interpret stop signs as speed limit ones only with only a bit of duct tape on it [START_REF] Eykholt | Robust Physical-World Attacks on Deep Learning Visual Classification[END_REF]. . . . . . . . . . . . . . . . . . . .

4 Figure 1.2
To be able to answer mundane questions such as "where are my keys", robots are required to master three reasoning skills: first x, the ability to understand natural language questions to grasp what is asked, and analyze its vision to answer. Then, robots need the ability to navigate in an environment to look for those keys as fast as possible, and then, the ability to self-localize to both avoid going to already searched rooms, and, when found, to communicate the position of the keys. . . . . . . . . . . . . . . . . . . . . 5

Chapter 2: related work

9 Figure 2.1
Illustration of the structure of a Deep Neuron Network. With the input image X i , the neurons of the model, arranged in inter-connected layers, yield the output Y i "dog". This output is the outcome of a multitude of neuron computations. As depicted on the right of this figure, each neuron (e.g. here N 2 1 ) relies on the result of neurons from the previous layer. A more formal representation of this computation is depicted in Equation 2.1. . . . . . . . . . . . 10 On the left x, two examples of an image containing a digit which have been correctly evaluated as "two" by a Deep Neural Networks (DNN). On the right y, two other images which according to the MNIST dataset also contain handwritten "two". However, the same DNN fails and labels those images as "six". Such mistakes may be more acceptable to Humans, as those ambiguous images may even induce us to estimate that they contain sixes. . . . . . . . . . . . . . . 14

Figure 2.4

Building blocks for Deep Learning interpretability are designed to provide an understanding of a model's decision or behavior. Here illustrated using an Multi-Layer Perceptron (MLP), they can be divided in four categories. For a given input, a model produces intermediate results (Activations), to reach an output which can be used either by derivation to produce Gradients, or with output to produce Representation. Finally inputs and outputs can be combined to yield Model-agnostic interpretations. . . . . . 17 Comparison of activations from a simple CNN model (Lenet5 [START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF]) trained to identify the hand-written digit in its input image. We observe that for the input image of a "2" x, the activations (the whiter the higher) of the first layer are sensitive to the overall image, and shape of the digit y, whereas activations from the last layer are more abstract shapes harder to analyze z. In this figure, images were resized for the sake of readability. Initially, both the image input and first layer activations were 28 × 28, while layer second layer activations were 5 × 5 pixels. . . . . . . . . . . . . . . . . . . 18

Figure 2.6

Left x, the Top-k image with the highest activations per neuron. Each row is a neuron, and each column from left to right corresponds the top-10 images. We can, for example, observe that the neuron corresponding to the first row might be sensitive to circular patterns such as dog eyes and snouts. Credits for this sub-figure go to Springenberg et al. [START_REF] Tobias Springenberg | Striving for Simplicity: The All Convolutional Net[END_REF]. Right y, examples of overlap between a manually annotated segmentation dataset, and a model's activation from 4 neurons. We can observe that the neurons corresponding to the two rows on the left are sensitive to houses. Hence indicating that some neurons may function as "object detectors". Credits for this subfigure are due to Bau et al. [START_REF] Bau | Network Dissection: Quantifying Interpretability of Deep Visual Representations[END_REF] Visualization of activations of an image classification CNN using DeConv [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF]. This figure represents a manually selected sub-set of neurons which are displayed in a 3 × 3 grid of their most active images. We can observe that the first layer x contains patterns such lines and color gradients, the second layer y seems to seek for textures or patterns, while the layer 5 z seems to responsive to more complex image feature such as faces. In {, we combined with the display of the top-k images one make sens of abstract DeConv activations, e.g. the grass in the background.

Activations in this figure were sampled from [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF]. . . . . 20

Figure 2.8

Left x visualization of a manually selected activation of a hidden state of a recurrent model. Such activation is displayed over its input text ranging from blue (negatives values) to red (positive values). It can be observed that this activation seems correlated with the level of text indentation (credits to [START_REF] Karpathy | Visualizing and Understanding Recurrent Networks[END_REF]). Right y, visualization of a complete hidden state as a grid. It can be observed that only a handful of activation are high at the same time, and thus that they may be used as "functions" representing different elements of the inputs. Sampled from [START_REF] Carter | Experiments in Handwriting with a Neural Network[END_REF]. . . . . . . . . . 21

Figure 2.9

Examples of visualization with gradients-based building blocks. From left to right, x with an input image, y guided back-propagation, class [START_REF] Tobias Springenberg | Striving for Simplicity: The All Convolutional Net[END_REF], z optimization of input [START_REF] Mehta | Active domain randomization[END_REF] targeting the class "dog", and { Grad-Cam [START_REF] Ramprasaath R Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF] highlight pixel assimilated to the class "dog". . . . . . . . . . . . . . 23

Figure 2.10

Examples of visualization with the representation building blocks. From left to right, x t-sne representation of embedings on the MNSIT dataset, each dot is an input, and its color represents it class. y Grand-tour of the MNIST-Fashion [START_REF] Li | Visualizing Neural Networks with the Grand Tour[END_REF] with as many dimensions as the number of classes displayed on the most right of the figure. . . . . . . 26

Figure 2.11

Example of insight gained using lime [START_REF] Tulio Ribeiro | Why should i trust you?: Explaining the predictions of any classifier[END_REF], when given the image of a husky x, the analyzed model fails and predicts "wolf". By looking at lime's explanation, with the gray areas corresponding to the "super-pixels" removed, we can see that the analyzed model relies on the presence of snow in the background rather than the dog to yield an output, hence the mistake. This example and images were sampled from [START_REF] Tulio Ribeiro | Why should i trust you?: Explaining the predictions of any classifier[END_REF]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

.12 Overview of CNNVis [START_REF] Liu | Towards better analysis of deep convolutional neural networks[END_REF], a visual analytics system leveraging activation building blocks to display to what neurons may be sensitive to. This system follows, the model's architecture to display with the help of clustering methods, among others, the top-k images of neurons. This view was sampled from an online prototype [START_REF]CNNVIs online prototype[END_REF]. . . . . . . . . . . .

Chapter 3: visual question answering This is chapter is dedicated to providing to robots the ability to answer natural-language questions about images. For example here, when asked the mundane question "Where are my keys?" x, the robot needs to understand what we are looking for, and then search for it within a given image to provide an answer e.g. here "On the desk!". . . . . . . . .

Figure 3.2

In Visual Question Answering tasks, we provide to a model a question in a textual form along with an image. In our case, we expect the model to analyze the image to answer the given question with a single word. As an example here, when asked "What is the color of the bananas?", the model should output "Green". . . . . . . . . . . . . . . . . . . . . .

Figure 3.3

An Illustration of the VL-Transformer architecture we rely on. Question and image are first tokenized and then encoded using vision (in green) and language (in blue) only transformers [START_REF] Vaswani | Attention is all you need[END_REF], followed by (bi-directional) intermodality transformers [START_REF] Hao | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF]. The answer is predicted from the "CLS" token. Yellow and orange rectangles represent, respectively, inter-and intra-modality attention heads. i and j are the layers and head indices used for naming attention heads through the manuscript. . . . . . . . . . . .

Figure 3.4

When asked "Is the knife in the top part of the photo" x the tiny-LXMERT model, with the image of a knife at the bottom y, incorrectly outputs "yes" z with more than 95% confidence. While an exploitation of bias can be considered, we can observe that the answer "yes" represents only 33% of answers of similar questions over the complete dataset. Thus in-depth analysis of the attention of the model may be required to grasp what led to such a mistake. . . . . . .

Figure 3.5

Visualization of a selected vision-to-language head and attention map for two different models. x the noisy model associates the "knife" word with a large number of different objects, including fruit. y the oracle model learns a perfect association between the word "knife" and the "knife" object; z the oracle transfer model associates the word "knife" with two different bounding boxes of type knife handle, whose embeddings are sufficiently close for correct reasoning. Head selections are not comparable between models, we therefore checked for permutations. . . . . . . . . . . . . . 50

Figure 3.6

Opening the black box of neural models for vision and language reasoning: given an open-ended question and an image , VisQA enables to investigate whether a trained model resorts to reasoning or to bias exploitation to provide its answer. This can be achieved by exploring the behavior of a set of attention heads , each producing an attention map , which manages how different items of the problem relate to each other. Heads can be selected , for instance, based on color-coded activity statistics. Their semantics can be linked to language functions derived from datasetlevel statistics , filtered and compared between different models. This tool is available online at: https://visqa.liris.cnrs.fr. . . . . . . . . . . . . . . . . 53 When asked "Is the person wearing shorts?", the oracle transfer model successfully answers "yes". It can be observed in its first Language-to-Vision attention maps, that the word "shorts" (column) is strongly associated with the object "shorts" (row). The same phenomenon is also observed for the word "person', strongly associated with objects labeled as "woman" among others. . . . . . . . . . . . . . . . . . . . 60 Figure 3.9 When asked "Are there both knives and pizzas in this image?", the oracle transfer model fails and answers "yes". By filtering heads associated with a selected word, we can observe that language self-attention heads are more responsive to the word "both" x, as opposed to the word "and" y. . . . . . 61 Figure 3.10 Without any"hair dryer" provided by the object detector, the oracle transfer associates in its vision-to-language x the object "hand" with the words {"[CLS]","is","?","[SEP]"}.

While vision-to-vision focuses on a "knob" object y. . . . .

Figure 3.11

Despite different performances, we can observe that both tiny-LXMERT and large-LXMERT have the same behavior as the frequency of question/answers increases. Hence, we can estimate that they may be exploiting similar shortcuts defined in GQA-OOD [START_REF] Kervadec | Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?[END_REF]. . . . . . . . . . . . . . . . . . .

Figure 3.12

Difference between the attentions of Head LV_1_0, when asked "is the the train blue?", and "is the the train red?". We can observe that in this head, the attention focuses on different objects (row) depending on the color asked (column).

Chapter 4: navigation Our navigation problem consists in solving a visual task (e.g. fetch, interact, or recognize items) while avoiding obstacles in an environment. Deep Reinforcement Learning can be used to solve this problem by using an image as input x at time t. Features are then extracted from this image y, and combined with the previous memory vector t -1 z. Using this memory vector, the agent decides to move forward or turn left, for instance {. . . . . . . . . . .

Figure 4.3

Memory construction process: at a current time-step t, the agent updates its memory by producing a new memory vector. Each dimension of this vector (represented as a column) is appended to the previous ones chronologically (from left to right). As a result, each row of the appended vectors represent the actions of a single memory element. . DRLViz displays a trained agent memory, which is a large temporal vector, as a horizontal heat-map . Analysts can browse this memory following its temporal construction; filter according to movements of the agent and derived metrics we calculated (e.g. when an item is in the field of view ); and select the memory to filter elements and compare them . . . . . . . . . . . . . . . . . . . . . . . . . DRLViz allows to compare selected time intervals x. For instance to compare when agents face dead-ends y and when they face health-packs z. One can observe that more elements are active while the agent is facing Health Packs than while facing a dead-end. Perhaps those elements are encoding information concerning Health Packs. When facing a dead-end, both the orientation variation and decision ambiguity are high which can be interpreted as the agent hesitating on which action to choose. . . . . . . . . . . . . 84 Summary of the insights gained by the experts. Expert #1 noticed two intervals during which the agent only turned right, by using both trajectory x and stacked area chart of actions y views. Once he replayed those sequences, he stated that the agent came twice in the same dead-end z. Expert #3 observed a hidden state dimension which is blue when the agent sees the red armor before the green armor, and then remained orange until when he saw the green armor {. Expert #2 probed a dimension that is active as the agent first saw the Health Pack, and remained active until it gathered it. Expert #1 also identified two hidden state elements that change as the agent gathered the health pack and then kept their values until the end of the episode }.

Using saliency maps ~, Expert #2 observed that the agent ignored the soul-sphere until it gathered the first three items . Finally, Expert #3 identified clusters in the t-SNE projection which corresponds to the agent's objectives e.g. gathering the green armor . . . . . . . . . . . . . . . . . . 85 When answering questions such as "Where are my keys?", robots need to be able to communicate their position. To do so, we trained models to yield coordinates and orientation angle from any given image z. Such a task is also used as a proxy to evaluate gaps between simulation and reality. . 96 In the studied robot ego-localization task, an RGB-D image x, is given to a trained model y, which uses it to regress the location (x, y), and orientation angle (α) in the environment from which this image was taken from z. As illustrated above, images taken from the same coordinates in simulation and real-world x may lead to different predictions due to differences, such as here, among others, the additional presence of a bike in the scene. We are interested in reducing the gap between the SIM and REAL predictions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Figure 5.3

The capture of real-world images are carried by an embedded RGB-D camera on a "Locobot" [START_REF] Murali | PyRobot: An Opensource Robotics Framework for Research and Benchmarking[END_REF] x. On the right y such a robot is displayed with respect to proportions in its environment. . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Figure

5.4

Using Sim2RealViz, the Sim2Real gap of a Data Augmentation model can be compared against other models (e.g. Vanilla or Fine-tuned) and displayed on the real-world environment map along with its performance metrics. In particular, Sim2RealViz shows x this model is particularly effective in simulation but we identified errors in the environment, such as the model failing to regress its position because of a closed-door that was opened in training. Such an error can then be selected by instance on the map y to identify key features extracted by the model either as superimposed on the heat-map z or as a first-person view {.104 To tackle over-plotting in geo-map while preserving the insights users can grasp with this view, Sim2RealViz provides three way to encode SIM / REAL predictions along with their ground-truth. From x which displays a pin per domain and link them to their ground truth, to y a glyph in which portion size encodes the distance to their color encoded domain, while their position encodes their direction. Or z, only display ground-truth pins with their color encoding the distance between simulation and real-world. 106 Artificial Intelligence (AI), is a vast domain with concrete applications now in our daily lives. It has however a long history of both definitions and applications 1 . In its early days, AI's popularity emerged in science-fiction from movies and literature, often in the shape of autonomous and sometimes sentient robots, such as in "Metropolis" (1926), the "Wizard of Oz" (1939), or "Tobor the great" (1954). Such popularity anchored high expectations for AI, to the point that in 1968 A. Clarke, and S. Kubrick estimated, in their movie entitled "2001: a space odyssey", that by 2001 machines would have at least matched Humans' intelligence. Such popularity, lead to the first wave of AI in academia, with government agencies such as Defense Advanced Research Projects Agency (DARPA) funding research. This was motivated by the advocacy of researchers who argued, among others, that the ability to interpret and translate languages was within reach in the next couple of years. While climbing the hill of the first advents of AI research, came in 1974 the disillusion that many mountains remained to be escalated to meet such high expectations. One of the major setbacks, at this time, was the lack of computational power to match the theoretical groundwork. This led to a downfall of funding, which is now referred to as "winters".

This winter has been "re-heated" thanks to endeavors, and discoveries that lead, among others, to the birth of expert systems in the '80s. This was also encouraged by the ever-increasing performance of computers-in terms of memory, storage, and computation-which improved their affordability. Expert systems consist of a set of rules established by an expert of a domain, e.g. a doctor, that the AI could rely on when yielding a diagnostic, for example. Despite their requirement for an exhaustive set of rules, those systems surged in industry and were applied to many domains such as health care [START_REF] William Van Melle | EMYCIN: A knowledge engineer's tool for constructing rule-based expert systems[END_REF]. However, in the '90s those systems began to be disregarded because the more complex they became, the harder they were to maintain and took time and effort to set up, without, once again, meeting the same high expectations inherent to AI. For instance, the myth of "sentient" computer programs able to sustain a conversation with a human. Surprisingly, during this new winter, AI reached new milestones. One of the most memorable at this time was in 1997 by beating for the first time in history the then world chess champion Gary Kasparov. This event reignited the spark of the so-called era of "heat" of AI, which kept increasing to its highest ever nowadays. Once again the non-stopping technological advances took a major role in this, by leading computers to be powerful enough to train artificial Neural Network (NN). Those NNs are composed of inter-connected artificial neurons, which design began in the '40s with the first mathematical model of a biological neuron [START_REF] Mcculloch | A logical calculus of the ideas immanent in nervous activity[END_REF]. At the core of the success of NNs is their ability to learn to identify patterns from data they are fed to. Since the capacity of computers grows, we are able to build bigger models, with neurons inter-connected in multiple layers known as Deep Neural Networks (DNN). This helped models to formulate more elaborated rules, and address more challenging tasks. Hence, as a repercussion, the size of models kept increasing. In 1998, LeNet-5 [START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF] one of the first DNN that gained popularity, had around 60 thousand of trainable parameters. Fourteen years later, AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] had over 60 million trainable parameters.

Nowadays, with models reaching hundreds of billions of trainable parameters [START_REF] Tom | Language Models are Few-Shot Learners[END_REF], researchers across many domains reached new horizons and breakthroughs in problems previously seen as unfathomable. Among others, there is notable examples in biology by unfolding proteins [START_REF] Jumper | Highly accurate protein structure prediction with Al-phaFold[END_REF], in healthcare by identifying cancerous cells [START_REF] Coudray | Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning[END_REF][START_REF] Ullah Khan | A novel deep learning based framework for the detection and classification of breast cancer using transfer learning[END_REF], beating humans at long-term strategies board [START_REF] Silver | Mastering the game of Go without human knowledge[END_REF] and video games [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF][START_REF] Vinyals | Grandmaster level in StarCraft II using multi-agent reinforcement learning[END_REF], autonomously driving cars from cameras [START_REF] Grigorescu | A survey of deep learning techniques for autonomous driving[END_REF], and generating creative art designs such as music [START_REF] Engel | GANSynth: Adversarial Neural Audio Synthesis[END_REF], and images [START_REF] Bau | Semantic Photo Manipulation with a Generative Image Prior[END_REF][START_REF] Gatys | A Neural Algorithm of Artistic Style[END_REF]. With such endeavors, the high expectations of sentient robots from the past century have never seemed so close! These successes soared thanks to the DNN's ability to learn to associate inputs and outputs through a mapping driven by rules they discover buried in the data they consume. Prior to that, developers had to manually write those rules themselves in their code. For instance, with rules, one may establish that an animal with two pointy ears and whiskers should be classified as a cat. Nonetheless, such a classification may not exclude the possibility that this animal may be a tiger, or that a cat may be missing an ear. Thus, due to mistakes that may occur, and the unrealistic amount of time it would require to account for every possibility, developers could not address problems with an exhaustive set of rules as a machine would. However, the side effect of letting any algorithm learn its own system, is that the underlying process that leads to an outcome remains unknown to both developers and end-users.

Performances that may exceed Humans raise nonetheless trust issues as they do not ensure the fairness of decisions that might be reached thanks to rules grasped from a biased design or data. This concern is emphasized in critical situations where even small errors of judgment can lead to dire consequences. For instance, when a driverless car may decide to turn on the road with no apparent reason for the humans inside [START_REF] Tesla | Inquiry After Series of Crashes -The New York Times[END_REF]. Understanding models' decisions, even in non-life-threatening situations, is primordial as nowadays they have a broad impact on our everyday lives. For instance, whenever we query information using keywords on search engines, when we receive recommendations of related topics or videos, when paths and directions are presented according to traffic predictions in navigation mobile applications, and sometimes even when candidates are selected for job interviews. Hence the rise of ethical and fairness questions of who decides how a model should behave, and on what ground a model should reach a decision. Such interests are also accentuated by legislation at country or continental level. The GDPR [START_REF] Parliament | of the European Union. General data protection regulation[END_REF] is a notable example to protect Europeans with is a set of regulations applicable since May 2018, stating that: "any AI system that is integrated into people's lives must be capable of contest, account, and redress to citizens and representatives of the public interest" [START_REF] Sartor | The impact of the General Data Protection Regulation (GDPR) on artificial intelligence | Think Tank[END_REF].

Numerous examples of failure cases of AI that impacted Humans' lives have been revealed and investigated. As an example, and illustrated in Figure 1.1 x, when using Pulse [START_REF] Menon | Pulse: Self-supervised photo upsampling via latent space exploration of generative models[END_REF], an up-sampling model with the image the most left, the model is trained to yield images with better quality. However, We can observe in the reconstructed image (x bottom) that the model fails to convey the individual's features replaced them with Caucasian ones. In addition, models designed to treat language by predicting the next words of sentences, have been shown to leverage discriminating samples towards individuals and ethnicities from their training dataset [START_REF] Tom | Language Models are Few-Shot Learners[END_REF]. As a result, when following words of certain religions or minorities, the model predicted only negative words and racial slurs. Sadly, such models have also been shown to convey gender bias [START_REF] Marcelo | Assessing gender bias in machine translation: a case study with Google Translate[END_REF][START_REF] Bolukbasi | Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings[END_REF]. For instance, by translating occupations where men are over-represented as male e.g."he is a doctor", while other occupations are translated as female e.g."she is a nurse". In order to learn to predict words, those models have been fed with a very large amount of data directly sampled from the internet, hence it may not have been properly curated. This led models to leak personal information such as home addresses [START_REF] Carlini | Extracting Training Data from Large Language Models[END_REF] simply by prompting the name of a person and letting a model autocomplete the rest of the sentence. As those models are the underlying program that runs widely in platforms we use daily, a very large audience is facing those biases. Many problems have been raised, but only a few solutions emerged to tackle them. For instance, Google Photos' model for image classification identified some humans as "gorillas", and in three years span, the only solution found was to remove image categories related to primates such as "gorilla", "chimpanzee", and "monkey" from their image labeling models. This solution cannot be reliable as it requires taking into account every bias from datasets and mistakes made by models as many of them are hidden, it would require an unfathomable amount of time to only obtain an incomplete model. In addition, models tend to behave x Pulse [START_REF] Menon | Pulse: Self-supervised photo upsampling via latent space exploration of generative models[END_REF], with an input image of the former US president Barack Obama, the model tasked to reproduce it, outputs the image of a Caucasian man. y the CLIP model [START_REF] Radford | Learning Transferable Visual Models From Natural Language Supervision[END_REF], can be induced to output wrong classification by simply taping tags to objects, e.g. as illustrated here, the apple is predicted as "library" [START_REF] Goh | Multimodal Neurons in Artificial Neural Networks[END_REF]. z, traffic signs detection models can interpret stop signs as speed limit ones only with only a bit of duct tape on it [START_REF] Eykholt | Robust Physical-World Attacks on Deep Learning Visual Classification[END_REF].

unexpectedly [START_REF] Lehman | The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities[END_REF] for example discovering and exploiting unknown glitches in video games [START_REF] Ecoffet | Go-Explore: a New Approach for Hard-Exploration Problems[END_REF]. This is because Deep Learning (DL) models tend to be "lazy", in the sense that they often learn to rely on shortcuts in their process to yield outputs. As an example, a model designed to predict what object is represented in an image can be easily be fooled by taping a tag with the name of another object on it. As illustrated in Figure 1.1 y, CLIP [START_REF] Radford | Learning Transferable Visual Models From Natural Language Supervision[END_REF] one of the most popular DL models of 2021 matching image and text, can be fooled and classify the image of an apple with a tag labeled as "library" as a library [START_REF] Goh | Multimodal Neurons in Artificial Neural Networks[END_REF].

Another issue with models deployed in production is their ability to be tricked by malicious users. Small disturbances on images that may even be imperceptible to humans [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] can alter a model's decision to its opposite. As depicted in Figure 1.1 z, stop traffic signs can be attacked by small patches of tape to fool models' and make them recognize a speed limit sign [START_REF] Eykholt | Robust Physical-World Attacks on Deep Learning Visual Classification[END_REF]. Such an attack can easily be overlooked by humans as the patches of tape may not alter their ability to identify stop signs. Those problems may only be the tip of the iceberg, and yet they hinder any deployment of the achievements of DL algorithms to real-life applications that may have an influence on one's life, and be life-threatening in critical applications (e.g. driverless cars).

Among the many ways to prevent biases in DL algorithms, the field of interpretabilty aims at empowering developers with a set of tools that helps them to identify and understand the hidden rules that those models may exploit to yield a decision. Interpretabilty also aims at benefiting end-users with increased fairness, 1.1 robotics task: "where are my keys" and ethics of their algorithms which are currently difficult to formulate in a way that those algorithms could leverage while learning [START_REF] Doshi | Towards A Rigorous Science of Interpretable Machine Learning[END_REF]. In addition, being able to inquire why a model reaches a certain output will foster a broader social acceptance [START_REF] Carvalho | Machine learning interpretability: A survey on methods and metrics[END_REF] which can lead to a more genuine trust of their decisions [START_REF] Tulio Ribeiro | Why should i trust you?: Explaining the predictions of any classifier[END_REF][START_REF] Anca D Dragan | Legibility and predictability of robot motion[END_REF]. But what does it means to interpret a model? How can one dig in the billions of hidden connections within models to extract hidden rules that lead to decisions? How can one leverage insights gained to further improve models and prevent the exploitation of bias? This manuscript is dedicated to the design of visual analytics systems conceived to address these challenges. Our aim is to develop interpretability methods for robotic tasks, to help robot reach their history of high expectations while mitigating their biases.

Where are my keys?

On the Desk! 2 -To be able to answer mundane questions such as "where are my keys", robots are required to master three reasoning skills: first x, the ability to understand natural language questions to grasp what is asked, and analyze its vision to answer. Then, robots need the ability to navigate in an environment to look for those keys as fast as possible, and then, the ability to self-localize to both avoid going to already searched rooms, and, when found, to communicate the position of the keys.

Robotics Task: "Where are my keys"

Robotics is among the most promising application of artificial intelligence, and probably the most represented one in movies and literature. It has applications, from robots doing our chores such as groceries and house-works, to social robots assisting humans with handicaps, and even autonomous cars. But despite the advances in academia and the soaring of DL, we haven't encountered them yet in a fully automated and reliable way in our daily life. This is due to the fact that deploying robots into our world requires addressing a combination of challenges yet to be successfully tackled enough to ensure their safety. As illustrated in Figure 1.2, to be able to answer an everyday question such as "Where are my keys?" a robot needs to tackle multiple sub-problems. First, natural language knowledge to grasp what needs to be found from the question, and a Visual recognition ability to identify the keys among hundreds of other items. This is known as Visual Question Answering (VQA). Second, the robot needs navigation to efficiently search the keys without being a danger to itself and others (e.g. bumping into humans, or falling off stairs). Finally, ego-pose localization enables the robot to communicate the position of the keys, and avoid searching the same area twice. The slightest error in those sub-problems can lead to a radical drop in performance and thus reliability on them. If the robot takes too much time to move, or often confuse items to identify, it would be pointless to rely on it. In addition, a robot also needs to cope with the ever-changing difficulties generated by real-world conditions such as opened/closed doors, moving objects (e.g. humans), or luminosity throughout the day that may prevent it to accomplish its task.

Visual Analytics for Model Interpretability

Visualizations, and especially Visual Analytics, are well suited to ease access to relevant information leading to a model's decision while conveying intuitive interpretability-i.e. the "I know it when I see it" statement [START_REF] Offert | I know it when I see it". Visualization and Intuitive Interpretability[END_REF][START_REF] Doshi | Towards A Rigorous Science of Interpretable Machine Learning[END_REF]. This is due to the fact that there is not a stereotypical explanation for a model's decision [START_REF] Sokol | One Explanation Does Not Fit All[END_REF], and thus in many cases, experts need to rely on multiple blocks to build their hypotheses on how a model reached a decision in order to interpret it. Visual Analytics enables humans to explore large data spaces, such as models parameters or outputs. This is particularly useful during early exploration stages where problems are unknown and thus the research space is vast. Visual Analytics is potentially efficient for model builders to decide on what data it will train on, what functions the model will need to optimize, and thus ultimately what constitutes a good behavior for the model. Therefore, model builders are in the front line of the efforts for bias reduction and model improvements. However, to understand their models and their decisions, experts need to go through millions (and sometimes billions) of parameters per decision which is not feasible in a reasonable amount of time. The exploration of any hypotheses they draw on a model's behavior can then be carried through Human-computer interactions to associate parts of the model as rules leading to a decision. Finally, any insight gained needs to be 1.3 thesis overview extended to a complete dataset by gathering statistical evidence, in order to grasp more global tendencies in models' behaviors beyond single decisions.

One aspect of visual analytics systems is that they are usually designed for specific tasks and users. For instance, experts require in-depth analysis of models' decisions which can lead them to compare different hypotheses through models' parameters. This can be too overwhelming for end-users who are more drawn towards one main explanation, often obtained through guided visualizations design to confirm a model's decision rather than explore its source in its inner workings. While it is capital that end-users understand and are able to contest decisions of models, those models require nonetheless experts' insights to be improved and then deployed to real-life situations. Therefore as a first mission to reach interpretability, this thesis focuses on designing interactive visual analytics systems for experts analyzing their models on specific tasks and datasets. Chapter 6 discuss how those systems may be generalized to other models and tasks. While most of the contributions in this thesis are dedicated to experts, we also provide complementary works dedicated to the introduction to Deep Learning methods to non-expert audiences through interactive web pages-referred to as explainables or explorables [5,[START_REF]Explorable explanation -Wikipedia[END_REF].

Thesis Overview

The rest of the manuscript is organized as follows: first, Chapter 2 provides a background of the visual analytics and interpretability works related to this thesis. Then, as depicted in Figure 1.2, the following chapters are ordered around the robotic challenges of designing models able to answer open-ended humanreadable questions such as: "where are my keys?". Chapter 3 covers step x, i.e. the ability for models to answer natural language questions on given images, using computer vision. This chapter includes the design of VisQA, a visual analytics system in which users can browse the inner structure of transformer models to identify shortcut-biases, and leverage such insights to improve the reasoning of their models. Then, Chapter 4 addresses step y, the challenge of using Deep Reinforcement learning approaches to tackle automatic navigation, along with examples on how visualization systems we designed help experts interpret a model's decisions. The ego-pose localization step (step z) is covered in Chapter 5, in which we also explore how those models may face performance gaps when deployed to real-world's ever-changing conditions. Finally, Chapter 6 presents the current challenges and perspectives for future works of this thesis. This manuscript relies on the following published material: Chapter This thesis is related to designing interactive visual analytics systems to assist experts in gaining knowledge on their trained deep neural networks by interpreting their decisions. This can help them get a better understanding of their neural networks, and ultimately leverage this knowledge to improve them. This chapter first provides in Section 2.1 a broad overview and definition of interpretability, along with other main concepts. Then, Section 2.2 depicts the building blocks of visually interpretable trained systems, and finally, Section 2.3 introduces how these blocks are combined to form elaborated visual analytics tools. Additional related work will be provided in the next chapters, to cover specific Deep Neural Networks (DNN)s and application domains.

Definitions

The following sections present the definitions of the key terms visual analytics, interpretability, and explainability, often used interchangeably or assimilated as interpretability, with the lens of different communities and backgrounds within the scope of Machine Learning. Along with those terms, follows a broader approach of such a discipline as perceived by other domains (e.g. biology) known as explanation, and how it may benefit the Machine Learning (ML) community.

Visual Analytics

Visual analytics studies decision-making processes and analytical reasoning, and how they can be assisted by the design of interactive visualization systems [START_REF] Cook | Illuminating the Path: The Research and Development Agenda for Visual Analytics[END_REF]. Initially, visual analytics emerged from information visualization and diverged to focus on the complete analysis of decision processes by combining visualization, human factors, and data analysis. Visual analytics heavily relies on humans for tasks that cannot be automated, e.g. to decide what piece of information may be relevant [START_REF] Keim | Visual Analytics: Definition, Process, and Challenges[END_REF], and formulate hypotheses on its influence on a decision. The popularity of visual analytics systems, lead in 2007 to the creation of IEEE Visual Analytics Science and Technology (VAST), a Symposium of the IEEE 2006, which was held jointly with the IEEE Vis conference, one of the main conferences of visualization. Since 2021, VAST, along with infoVis and sciVis merged to IEEE VIS. As stated by Hohman et al. [START_REF] Matthew Hohman | Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers[END_REF], thanks to early endeavors of visualization and ML communities, the popularity of visual analytics soared when it was leveraged for the interpretation of deep neural networks. While Visual Analytics systems can provide insightful information on how inspected models behave this domain is still in its early stage, with challenges to overcome [START_REF] Keim | Visual Analytics: Definition, Process, and Challenges[END_REF][START_REF] Matthew Hohman | Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers[END_REF].

max(0, ) + N 1 2 1 N 1 0 N 1 Dog Input Xi Output Yi Layer Layer Layer 0 1 2 0 N 0 N 0 1 N 0 2 N 0 3 N 0 4 1 N 2 N 1 0 N 1 2 . + W 1 0 W 1 2 N 1 1 . W 1 1 .

Neuron Computation

Deep Neural Networks

DNNs are often considered as black-boxes with the unique purpose to convert inputs X into outputs Y. Formally this can be represented as the following function Y i = f (Θ, X i ) in which Θ corresponds to a set of parameters, and i is an element of the sets X and Y. In practice, a DNN is composed of a multitude of simple linear functions-referred to as neurons-generally connected with non-linear activations. As illustrated in Figure 2.1, those neurons represented as circles, are arranged in interconnected layers. Each connection between two neurons of successive layers is weighted by a parameter W l i where l indicates the layer of the incoming neuron, and i is the index of this neuron within its layer. Formally, in i's simplest form as a fully connected layer, a neuron (e.g. N l+1 0 ) and its incoming weighted connections can be represented as it follows:

N l+1 0 = i ∑ i=0 W l i N l i + b l i (2.1)
Where N corresponds to the ensemble of neurons in the model, W the weights, and b a bias added to provide to models the ability to formulate more complex functions (i.e. avoid passing by (0, 0) by adding a term). However, despite a multitude of interconnected neurons, a model can only formulate complex linear functions. Thus, to provide non-linearity, in DNNs, for each neuron, after the computation of Equation 2.1, the yielded value is then used in a non-linear function (e.g. sigmoid, tanh, or ReLU [START_REF] Fred | Deep Learning using Rectified Linear Units (ReLU)[END_REF]). The ReLu function, widely used in the Deep Learning community, can be represented as follows: ReLU(λ) = max(0, λ). Those functions, referred to as activation functions, are key for models to be able to approximate elaborated functions, such as, for instance, image classification. And thus, the result of this operation is called activation, i.e. an intermediate result of the model computation towards an output which can be used to try to grasp the decision process of the model. Since the neurons of a model are interconnected by layers, all activations of a selected layer represent how the model considers the given input X i . Such a representation is called an embedding, and, as illustrated in Section 2.2.3, embeddings can be used to capture how the model may construct an inner presentation of a given dataset. Now that we have interconnected neurons capable of formulating non-linear functions, the next step is to be able to tune this Neural Network for a specific task, i.e. what we call training. Aside from adjusting a model's structure, i.e. adding more or fewer layers, or neurons, and swapping activation functions, the only element that can be changed in a model is Θ: its weights (W). In biology, Θ can be seen as the neuro-transmitters we have in our brain, dampening or increasing a signal from another connected neuron. Hence, in computer science, those weights (or parameters) are values that need to be adjusted in order to fit the desired task. To do so, models are first initialized with random weights, and then start working-i.e. convert inputs X into outputs Y. Obviously, with random parameters, the model is doomed to yield errors which can be evaluated with a loss function, for example, the Mean Squared Error (MSE) L = ||y -y|| 2 , the most widely used in Deep Learning. Such a loss indicates how wrong the model is for a set of outputs y compared with the ground-truth optimal results y. With such information, the next step is to determine how one should adjust the model's weights to reduce L. This is done using gradients computed from the partial derivative of the loss with respect to the parameters as follows:

∇ Θi = ∂L ∂Θ i (2.2)
Here, i indicates the batch, i.e. a computational step after which the model yielded enough outputs to have a loss computed, and its weights updated. The resulting gradient ∇ Θ describes towards which direction the values of weights should evolve to reach a better result. However, if such changes were followed blindly, after many iterations the weights of the model could oscillate around its optimal configuration without ever reaching it. Thus, to limit how much the weights of a model may change, and help it find its optimal solution, the gradient is multiplied by the hyper-parameter σ (i.e. a parameter manually set by a human), a single value, called the learning rate. Typical values for a learning rate range over orders of magnitude from 1 × 10 -2 to 1 × 10 -6 . The smaller the value, the longer the model will need to be trained, but the more precise it will be. Hence, usually, there is a trade-off to be found between performances and training time 1 .

In the previous paragraphs, DNNs were introduced with the lens of Multi-Layer Perceptron (MLP)s, the most straightforward structure of models. However, there exist multiple types usually preferred for different types of inputs. Usually, for image processing, CNNs are preferred over MLPs as they are able to preserve local spatial coherence, and hence have fewer weights than an MLP would have for the same input. Because convolutions are shift equi-variant operations and share weights spatially, their usage lead to a significant reduction of the number of parameters needed. In this case, as illustrated in Figure 2.2, we refer to the weight of a model as filters which are convolved with the inputs for the first layers or with the previous activation. Similar to MLP the result of a neuron, despite being a matrix, is processed by the same activation functions. In many cases within the Deep Learning (DL) community, and in this thesis, CNN models often include some Fully Connected (FC) layers-i.e. MLP usually considered as the core of those models' decisions. This thesis also uses both Recurrent Neural Network (RNN)s and transformer models, whose design and structures particularities will be 1. Initially large values also provide a regularization effect that may help to avoid some local minima. further introduced within concerned chapters, namely Chapter 4 for RNNs, and Chapter 3 for transformers.

Input Xi

Interpretability

Historically, the concern of understanding models emerged in the '80s by studying expert systems [START_REF] William Van Melle | EMYCIN: A knowledge engineer's tool for constructing rule-based expert systems[END_REF], Bayesian networks [START_REF] Lacave | A review of explanation methods for Bayesian networks[END_REF], and more lately in the '90s, by studying neural networks [START_REF] Andrews | Survey and critique of techniques for extracting rules from trained artificial neural networks[END_REF]. However, the domain only gained popularity during the past decade along with the soaring performances and complexity of Deep Networks. Despite such a history, and a large amount of literature of different approaches and definitions to what may be seen as understanding a model, a broadly accepted definition is yet to be discovered. While a vast majority of those definitions centers around humans (e.g. "the ability to explain or to present in understandable terms to a human" by Doshi-Velez and Kim [START_REF] Doshi | Towards A Rigorous Science of Interpretable Machine Learning[END_REF]), they are torn on what it should focus on. They either focus on:

• The decision (i.e. output) of a trained model, analyzed after-the-fact (referred to as post-hoc.

• The inner workings and designs of models, describing, for instance, how many layers should be used-sometimes referred to as transparency.

• The mapping of data, which addresses how models can tie together a dataset of inputs to outputs-also called representation [START_REF] Zachary C Lipton | The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery[END_REF].

"six" "two" On the right y, two other images which according to the MNIST dataset also contain hand-written "two". However, the same DNN fails and labels those images as "six". Such mistakes may be more acceptable to Humans, as those ambiguous images may even induce us to estimate that they contain sixes.

This difference may affect the design and purpose of approaches to interpretability. For instance, to focus on a single decision of a trained model, one needs to design a system capable of analyzing this decision along with its input, otherwise it may be irrelevant. To illustrate this, let's consider the analysis of a decision of a Deep Network trained to detect hand-written digits in a given image. With the images shown in Figure 2.3 x, the model predicts that they correspond to "two"s. By looking at the image, one may indeed understand that those digits correspond to twos, and hence that the model is well trained. In contrast, with images of ambiguous digits, where even for humans it is hard to recognize the corresponding digit, (Figure 2.3 y), the model yielding "six" instead of "two" is rather understandable. Hence such a mistake may be acceptable to users who may gain more trust in it, as the model made a mistake that a human also would [START_REF] Tulio Ribeiro | Why should i trust you?: Explaining the predictions of any classifier[END_REF].

In contrast, to focus on the inner workings, one may either target the structure of models [START_REF] Wongsuphasawat | Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow[END_REF], or the learned parameters [START_REF] Tzeng | Opening the black box-data driven visualization of neural networks[END_REF][START_REF] Voss | Visualizing Weights[END_REF]. Both of those targets do not rely on the analysis of inputs or outputs. Hence, when analyzing the same model that outputs the hand-written digit in images, with the lens of inner workings, one only relies on the model itself. As an example, this can be done by collecting the parameters of the model, and browsing through them to try to grasp the shapes and colors patterns it may be sensitive to (e.g. white curves which may be seen in twos). This, however, is inherently much more difficult for Deep Networks (due to the number of parameters, and abstract data they manipulate) than, for example, for rule-based expert systems.

Finally, interpretability, within the scope of mapping of data, arises from the assessment that models are or should be considered as black-boxes where only their inputs and outputs may be understandable to humans. And hence that interpretability endeavors should not focus on shedding light on the inner workings of a model, which cannot be understood, but rather on how it processes data. As an example with the same hand-written digits recognition model as earlier, such a model would be fed with a plethora of images and the conducted analysis may consist in observing what the model may consider as similar images, or images with which the model is undecided.

Interpretability Vs. Explainability

Despite being used interchangeably by the community, and carrying a similar objective (i.e. to grasp an understanding of Artificial Intelligence (AI)s' behaviors), the terms interpretability and explainability may convey different meanings. For instance, according to Rudin [START_REF] Rudin | Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead[END_REF], interpretability comes through the design of machine learning models (i.e. is intrinsic) either by using Deep Networks trained to communicate their decision process (e.g. in natural language), or with simpler models such as rule-based ones in which a path resulting in a classification can be followed by Humans. In contrast, to him, explainability comes once a model is trained, either with the usage of a simpler model, e.g. rule-based, trained to "mimic" the behavior of a more complex Deep Network. To him, explainability can also be reached by using methods after-the-fact (i.e. post-hoc) to analyze a decision of a neural network. Roscher et al. [START_REF] Roscher | Explainable machine learning for scientific insights and discoveries[END_REF], indicate that interpretability refers to elements that inform humans about a models' decision process, and that the explainability of models, is the human-centric approach of using those interpretable features to grasp the causes that lead to a decision.

In this manuscript, we suggest the burden of analysis as a distinctive feature between those two concepts. In interpretability, the burden of exploration and analysis of a model's behavior is tackled by the user, while in explainability, information that may help understand a model's behavior is presented to the user. While the distinction between those terms is thin, we argue that this reinforces the distinction between targeted audiences, and thus the design of solutions. As an example, experimented users may be more inclined to analyze models themselves, while end-users may seek more accessible information. Such a distinction between those terms will be carried through the manuscript, which focuses on a humancentered approach to assist domain experts, often builders of models, to interpret the decisions of their models by exploring their inner workings visually, and through human-computer interaction.

Explanations

The quest for interpretability is a path also explored outside of Machine Learning and Computer Science, in domains such as law, biology, social or cognitive sciences. However, according to Mittelstadt et al. [START_REF] Mittelstadt | Explaining explanations in AI[END_REF], in ML, the most recent discipline, interpretability endeavors do not rest on the shoulders of giants to leverage already existing methods and definitions from those domains, hence the emergence of a gap between them. For Miller [START_REF] Miller | Explanation in artificial intelligence: Insights from the social sciences[END_REF], closing such a gap would benefit the ML community, along with focusing on everyday explanations. To him, everyday explanations target local "why-questions", e.g. "why did the model provide this output?", as opposed to scientific explanations which may address more global approaches e.g. "What did the model learn?". The benefit of everyday explanation is that it may encourage users to build trust as they may be able to more easily identify causes for a decision. This is due to the fact that everyday explanations are expected to convey key causes leading to a decision, as a human would, rather than providing the complete chain. Such explanations are also encouraged by the fact that human cognitive functions tend to be overwhelmed when there are too many elements to focus on, and hence, humans tend to be more reserved towards complete explanations with many parameters to account for.

While there is no consensus on definitions in machine learning interpretability, the term "explanation", is often described as addressing a model's outcome individually by using both the input and the models' perception of outputs (e.g. class discriminative features) [START_REF] Lisa | Generating Visual Explanations[END_REF][START_REF] Montavon | Methods for Interpreting and Understanding Deep Neural Networks[END_REF]. This more broadly falls in post-hoc interpretability area, with the exception that explanations focus on one instance at a time. Following these works, throughout the manuscript, we will refer to an explanation as an answer to "why" questions. For instance, " Why did the autonomous car decide to hit the breaks?". An answer could be because a traffic light turned red, forcing by law any car to stop there. Another explanation could be that a vehicle ahead was stopping, thus breaking was mandatory to prevent any collision. We refer to post-hoc interpretability as the collection of methods that may shed the light on this ambiguity, for example, by addressing questions designed to verify potential causes for a decision such as "has the car seen the traffic lights?", thus potentially leading to the explanation to "Why did the autonomous car decide to hit the breaks?". An explanation may rely on multiple interpretations which might co-exist in order to grasp the behavior of a model. In addition, explanations can be erroneous, biased, or victim of human over-interpretation, hence the need for multiple explanations and tools to analyze them. Here illustrated using an MLP, they can be divided in four categories. For a given input, a model produces intermediate results (Activations), to reach an output which can be used either by derivation to produce Gradients, or with output to produce Representation. Finally inputs and outputs can be combined to yield Modelagnostic interpretations.

Dog Activation

Building Blocks of DNN Interpretability

Interpreting Deep Neural Networks (DNN) is often tackled with respect to their application domain (e.g. image, or text), and their type such as, for example, MLP, CNN, or RNN. The following presents some of the building blocks of visualization for Deep Learning interpretability, i.e. individual approaches designed to provide an understanding of a model's decision or behavior. As illustrated in Figure 2.4, those building blocks are here presented in four categories, namely: Activations which focus on the intermediate results extracted from the inner operations a model does to reach an output, Gradients produced when deriving an output with respect to either the input, or activations, Representation which addresses tendencies and global behavior of models through their embeddings, and finally Model-agnostic methods leveraging inputs and outputs of models to yield insights on their behaviors. Those building blocks of visualization which mostly emerged from ML communities, and are presented in ML, are detailed in the following sections. [START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF]) trained to identify the hand-written digit in its input image. We observe that for the input image of a "2" x, the activations (the whiter the higher) of the first layer are sensitive to the overall image, and shape of the digit y, whereas activations from the last layer are more abstract shapes harder to analyze z. In this figure, images were resized for the sake of readability. Initially, both the image input and first layer activations were 28 × 28, while layer second layer activations were 5 × 5 pixels.

Activations of Neurons

As introduced in Section 2.1.2, and illustrated in Figure 2.1, during their execution Deep Neural Networks (DNNs) provide intermediate results referred to as activations. Those activations can be visualized for local explanations of what regions of the input a model may be sensitive to. Hence the reason why to analyze them. In computer vision with CNNs and images as input, activations are often visualized as images describing how a neuron might be sensitive to the shapes or colors of the input image. Usually, the most targeted neuron activations to visualize are those within the first layers. This is due to fact that these activations are directly tied to the image input, and thus the activations can be interpreted in pixel space, i.e. be compared with the input image. Since layers in CNNs are sequential, i.e. the layer n -1 provides inputs for the layer n, the deeper in the model activations are sampled from, the more abstract and smaller they become, making chances to interpret them grow thinner. As illustrated in Figure 2.5 y, with a CNN model designed to classify digits within an image, the activations (the whiter the more intense) from the first layer can be assimilated close to the input image x, whereas activations for the last CNN layer are harder to analyze as they are. Those activations, however, remain a vast majority within DNNs, hence the raising desire to understand what information they may convey throughout the model. We can, for example, observe that the neuron corresponding to the first row might be sensitive to circular patterns such as dog eyes and snouts. Credits for this sub-figure go to Springenberg et al. [START_REF] Tobias Springenberg | Striving for Simplicity: The All Convolutional Net[END_REF]. Right y, examples of overlap between a manually annotated segmentation dataset, and a model's activation from 4 neurons. We can observe that the neurons corresponding to the two rows on the left are sensitive to houses. Hence indicating that some neurons may function as "object detectors". Credits for this sub-figure are due to Bau et al. [START_REF] Bau | Network Dissection: Quantifying Interpretability of Deep Visual Representations[END_REF].

The visualization of features from deeper layers can be addressed with other approaches, however, many of them revolve around the challenge of contextualizing them with respect to their input, e.g. display them over input image. For example, as depicted in Figure 2.6, to grasp to what shape or color a neuron may be sensitive to, one can do a forward pass on each dataset sample and can collect the activations of this neuron. Then, the next step is to sort those activations by their intensity, e.g. in the case of CNNs with a matrix as activation noted a, with n rows and m columns the following sum:

n ∑ i=0 m ∑ j=0 a ij (2.3)
The goal is to only preserve the top-k activations and their corresponding input image. Then, by visualizing those images one may grasp patterns and trends in the top-k images such as objects (e.g. cars) or shapes (e.g. circles) which may convey what the neuron may be sensitive to. Thanks to this, as illustrated in Figure 2.6, one may observe that some neurons may be sensitive to circular patterns, while others to text, or lines. Using this approach, Zhou et al. [START_REF] Zhou | Object Detectors Emerge in Deep Scene CNNs[END_REF] have studied some neurons of their analyzed CNN model were used as "object detector", i.e., where only highly activated when a certain object (e.g. a lamp) was within the input image. This can also be tackled in the opposite way, i.e. by selecting an activation map from a neuron, and evaluating its upsampled Intersection over Union (IoU), with an annotated segmentation dataset of objects [START_REF] Bau | Network Dissection: Quantifying Interpretability of Deep Visual Representations[END_REF]. That way, if the activation map completely overlaps the regions of the image in the dataset corresponding to an object, one may conclude that the corresponding neuron might be sensitive to it. As depicted in Figure 2.6 we can observe a strong correlation between activations of neurons, and the presence of "house" or "dog" within the inputs. [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF]. This figure represents a manually selected sub-set of neurons which are displayed in a 3 × 3 grid of their most active images. We can observe that the first layer x contains patterns such lines and color gradients, the second layer y seems to seek for textures or patterns, while the layer 5 z seems to responsive to more complex image feature such as faces. In {, we combined with the display of the top-k images one make sens of abstract DeConv activations, e.g. the grass in the background. Activations in this figure were sampled from [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF].

While those methods yielded promising results, they remain subject to humans' interpretations to either grasp patterns within the top-k images which may not be always straightforward or limit the range of activations to only visualize those that match previously annotated objects. An alternative, introduced by Zeiler et al. [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF], is to convert those activations from deep layers back to the input space, i.e. with the size of the image, and its highlighted regions corresponding to what a selected activation might be sensitive to. To do so, such an approach, named Deconv [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF], implements another inverted model, in parallel to a traditional CNN model, with the purpose to convert any activation back to the input. This inverted model samples activations from a selected layer, and nullifies (i.e. set their values to 0) all of them except the one analyzed. As illustrated in Figure 2 It can be observed that this activation seems correlated with the level of text indentation (credits to [START_REF] Karpathy | Visualizing and Understanding Recurrent Networks[END_REF]). Right y, visualization of a complete hidden state as a grid. It can be observed that only a handful of activation are high at the same time, and thus that they may be used as "functions" representing different elements of the inputs. Sampled from [START_REF] Carter | Experiments in Handwriting with a Neural Network[END_REF].

this results in an image, the same size as the input with shapes highlighting what triggered the selected activation. It can be observed, as mentioned earlier, that the activations of the first layer of the model tend to be responsive to lines and color hue x. However, as the activation are sample deeper within the model, they become complex, with shapes such as dog faces, or certain textures in layer#5 (Figure 2.7 z). As illustrated by Zeiler et al. [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF], those visualizations reach their full potential when combined with the top-k images maximizing an activation method. This can help humans to make sense of otherwise hard to grasp patterns resulting from Deconv, by helping humans grasp the common factor within top-k images those activations may correspond to. For example, in Figure 2.7 { using the combination of those methods, it has been concluded that this neuron from layer#5 might be sensitive the presence of grass in the background.

With Natural Language Processing (NLP) tasks addressing textual input (e.g. translation), one of the main challenges for models is being able to retain information seen from previous words of a sentence which may still have an influence on the current input. To do so, we rely on RNNs, another kind of model which maintains a recurrent inner state (called hidden state) that updates after an input. Formally, this can be represented as it follows: h t = Φ(h t-1 , Θ Φ , a n ) where h t is the current inner state, h t-1 the one from the previous input, and a n represents all activations from the layer n, to which the recurrent layer is connected. Hence the hidden state is a time-varying vector which is often represented as a heatmap that can be displayed over inputs (e.g. highlighting words and characters in sentences) [START_REF] Karpathy | Visualizing and Understanding Recurrent Networks[END_REF]. As illustrated in Figure 2.8 x, thanks to such a visualization, it has been discovered that the hidden states of those models are able to model hierarchy in texts such as indentation, may be able to count characters in sentences without being asked to, enabling them to convey information from inputs outside of is expected from their design. Heatmap visualization of hidden states can also be applied to trajectories in which some activations activations are sensitive, among other, to trajectories' direction [START_REF] Carter | Experiments in Handwriting with a Neural Network[END_REF] (Figure 2.8 y). Thanks to this it has been discovered that their model was, for example, sensitive to trajectories going up or down. More recently, NLP tasks are addressed with attention models [START_REF] Vaswani | Attention is all you need[END_REF], whose attention, i.e. activations, can be visualized as bipartite graphs [START_REF] Olah | Attention and Augmented Recurrent Neural Networks[END_REF], and heatmaps [START_REF] Alexander M Rush | A Neural Attention Model for Sentence Summarization[END_REF][START_REF] Bahdanau | Neural Machine Translation by Jointly Learning to Align and Translate[END_REF] to grasp how a model may associate concepts and words together while providing an output. Chapter 4 and Chapter 3 are respectively to the visualization and analysis of activation in RNNs, and transformers. Furthermore, as depicted in the following section, methods to visualize activations can be further improved when used jointly with back-propagated gradients.

Visualization with Gradients

As introduced in Section 2.1.2, gradients are at the center of neural algorithms' learning, they are used to update model weights with respect to their output, thus it is only natural to visualize them. However, a multitude of gradients can be computed from DNN models, and the gradient computed with Equation 2.2, may not be the most suitable in the scope of interpretability. This is due to the fact that it only indicates how the model's weights may need change to better bring closer its outputs y and ground-truth labels ŷ, and for large models there may be billions of them! Thus a human may have a hard time understanding the meaning of those changes. More suitable information, would be to understand what pixels of an input image x i are the most influential for a model to yield its output y i . This can be done with the following gradient:

∇ xi = ∂y i ∂x i (2.4)
In other words, how a change of pixels values of the input, may affect the output. A key feature of this gradient is its ability to transform this information into a medium that a human can make sense of, i.e. the input image. This is referred to as saliency maps [START_REF] Simonyan | Deep inside convolutional networks: Visualising image classification models and saliency maps[END_REF], in which the gradient ∇ xi can be converted into an image and displayed over the input to display relevant parts of the image for a selected output. The downside of such an approach is that in many cases a majority of the image is highlighted, making it difficult to assess what particular points drew the model's attention. To tackle this issue, guided-backpropagation [START_REF] Tobias Springenberg | Striving for Simplicity: The All Convolutional Net[END_REF] (illustrated in Figure 2.9 y) proposes to clip to zero parts of the gradient's contributions to other classes, i.e. those that may be negative near the output, but positive throughout [START_REF] Tobias Springenberg | Striving for Simplicity: The All Convolutional Net[END_REF], z optimization of input [START_REF] Mehta | Active domain randomization[END_REF] targeting the class "dog", and { Grad-Cam [START_REF] Ramprasaath R Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF] highlight pixel assimilated to the class "dog".

the model. This results in the highlight of only key pixels to classify images, such as, for example, the snout of a dog. By doing so, in contrast to activation-based methods, one can assess the pixels of an image that may be associated with a class not even targetted by the model. Integrated gradients [START_REF] Sundararajan | Axiomatic Attribution for Deep Networks[END_REF], propose to tackle the saturation of gradients by using as input the integral interpolation of an image and a baseline (usually a black image). This approach yields visualizations in which features of images such, for instance, a dog's snout are more reflected. This is due to the fact that in many cases, models can successfully classify an image indistinguishable by humans (e.g. with pixels having less than 10% of their original value) [START_REF] Sturmfels | Visualizing the Impact of Feature Attribution Baselines[END_REF], hence the saturation of gradients.

The gradient ∇ xi extracted from Equation 2.4, can also be used to update the input image with respect to the model's output. The intuition behind this is that, by modifying the pixels responsible for the model's output, one may ultimately grasp what patterns the model may seek in an image to yield such an output. By doing this iteratively to alter an image (usually filled random noise) one can maximize an output to provide 100% confidence on a class [START_REF] Mordvintsev | Inceptionism: Going deeper into neural networks[END_REF]. As depicted in Figure 2.9 z, this results in the creation of abstract artifacts of key class's features all over the input image such as, for instance, snouts for dogs [START_REF] Olah | Feature Visualization[END_REF]. This indicates that for the model to output the class "dog" for an image, the model needs, among others, to distinguish a snout. When used in combination with neuron activations, input optimization can provide insights on the role of each unit of a model, i.e. what each neuron is looking for in an image. To do so, instead of optimizing the input image to maximize the output, one can maximize the activation of a selected unit [START_REF] Bengio | Visualizing Higher-Layer Features of a Deep Network Visualizing Higher-Layer Features of a Deep Network Département d[END_REF] as it follows:

∇ xi = ∂a l n ∂x i (2.5)
Where a l n is the n th activation of the layer l. Such an approach has been improved through regularization using an image prior [START_REF] Mehta | Active domain randomization[END_REF], and generative models [START_REF] Nguyen | Synthesizing the preferred inputs for neurons in neural networks via deep generator networks[END_REF] to provide more human-readable images. Such an approach can also be applied to NLP, in which words of sentences can be optimized in order to grasp how a unit may associate words together [START_REF] Bäuerle | What does BERT dream of?[END_REF].

To date, the most popular method to visualize gradients of CNN models with images as input is grad-CAM [START_REF] Ramprasaath R Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF], which relies on the activations of the last convolutional layer to produce a heatmap over the image input highlighting the regions related to a particular class. Such a heatmap indicates the attention of the model, i.e. the regions of the input the model looks at in order to provide the output. Thus, humans can assess if the model focuses on the correct objects within images corresponding to the outputted class. This approach weights the activations of the last convolutional layer (a l ) using global average pooling of gradients issued from the derivation of the output y i with respect to each activation of the last layer l indexed by n as it follows:

α y ln = 1 Z w ∑ j=0 h ∑ k=0 ∂y i ∂a ln jk (2.6)
Where w, h are the width and height of the activation map. Here, α y ln is the weights of the neuron n of layer l extracted from the output y. Such a weight is then combined with activations, and passed to a ReLU function to only preserve positive values, i.e. those contributing to the output as it follows:

m y = ReLU ∑ n α y ln a l n (2.7)
The resulting m y corresponds to a heatmap, the size of activations of the last CNN layer, describing which regions of the input image are responsible for the model's output. As illustrated in Figure 2.9 {, we can observe with the heatmapthe redder, the more intense-that according to the model the dog's face is a key feature for the model to output the class "dog". However, in order to be relevant, grad-cam needs for the activations to have a sufficient size, otherwise the resulting heatmap, which needs to be upsampled to match the input's size, may cover too much of the input, hindering any analysis. The size of activations is dictated by the model's architecture, and thus needs to be considered before using grad-cam heatmaps. Nonetheless, thanks to such an approach experts were able to grasp bias such as, for instance, how a model exploits humans' faces to predict their job, instead of their surroundings and tools they manipulate [START_REF] Ramprasaath R Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF].

Inner Representation of Data

As introduced in Section 2.1.2, like a funnel, as the dataflow reaches deeper layers of the model, the amount of parameters grows thinner. Because of this, the model needs to build an inner representation of inputs in its own abstract space. Thus, the complete set of activations with one layer, noted A l , describes how the model perceives the given image, i.e. what information it deemed important to convey. This is referred to as an embedding and its core to the analysis of models with the lens of representation. In such an analysis, one may seek to understand how the models behave globally, i.e. how the model may perceive the data that it is fed to, and how the model may associate data-points together. To do so, such methods often focus on activations in the last layer. This is due to the fact that it is argued that such a layer is the most suited to be analyzed since it should convey high-level semantics [START_REF] Ramprasaath R Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF] which are then the premise of decisions made by fully-connected layers. Hence, the first step of any study of a model's inner representation of data is to pass a complete dataset through it and collect, for each data-point its corresponding embedding.

However, due to the nature and size of models, those embeddings are highdimensional elements which makes comparison or interpretation by humans as they are, nearly impossible. Thus, to do so, the dimensionality of the embeddings is frequently reduced by techniques such as pca, umap [START_REF] Mcinnes | Umap: Uniform manifold approximation and projection for dimension reduction[END_REF], or t-sne [START_REF] Van | Visualizing Data using t-SNE[END_REF]. The following describes the execution of t-sne, the most popular method. First, it computes the probability for each pair of points to be neighbors, using the euclidean distance between them. This is formally represented as it follows:

p j|i = exp(-||x i -x j || 2 /2σ 2 i ) ∑ k =i exp(-||x i -x k || 2 /2σ 2 i ) (2.8)
Where p j|i is the probability that the embedding x i picks x j as neighbor, when embeddings follow the probability density under a Gaussian centered at xi of variance σ 2 i . Then this needs to be converted into a low dimensional space (the visualization space, usually 2D or 3D) using the following approach:

q j|i = exp(-||y i -y j || 2 ) ∑ k =i exp(-||y i -y k || 2 ) (2.9)
in which y i and y j represents the data points x i and x j in this low dimensional space, and q j|i the probability that they are neighbors. Hence, the objective of t-sne is to minimize the difference between p j|i , and q j|i to 0 which is a synonym of a perfect mapping between those two spaces with respect to the distance between x i and x j . Such minimization is computed using the sum of Kullback-Leibler divergences on embeddings as follows:

C = ∑ i KL(P i ||Q i ) = ∑ i ∑ j p j|i log p j|i q j|i (2.10)
This yields C, the cost function of all probabilities P|Q, which is then used to update data-points in low-dimensional space following a gradient-descent approach ∂C ∂y i . Intuitively, this can be seen as data-points attracting and repelling each other in order to minimize their overall distance when their p j|i , and q j|i are too different. t-sne is a stochastic and non-parametric approach, which means that it must be re-executed in order to add new data-points to the representation, and doing so may yield different results. From left to right, x t-sne representation of embedings on the MNSIT dataset, each dot is an input, and its color represents it class. y Grand-tour of the MNIST-Fashion [START_REF] Li | Visualizing Neural Networks with the Grand Tour[END_REF] with as many dimensions as the number of classes displayed on the most right of the figure.

As illustrated in Figure 2.10 x, with such a dimensionality reduction method, DNN embeddings can be projected into two [START_REF] Bostock | Let's Try t-SNE![END_REF] dimensions. In such a visualization, each dot is an embedding, at its color encoding, for instance, its ground-truth class. In any case, the goal is to regroup similar embeddings in clusters, and eventually compare a selected instance to the rest of the dataset-referred to as explanation by example by Lipton [START_REF] Zachary C Lipton | The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery[END_REF]. The more a model is able to successfully divide a dataset into clusters of the same class, the more it has a good representation, hence indicating successful learning of classification. In contrast, in those clusters, we can also observe sources of confusion for the model by identifying inputs that are between two clusters, or within a wrong cluster. This can be complemented by sampling embeddings from inputs and outputs of an intermediate layer, instead of the last layer. By doing so, we can compare the scatter plot before and after going through the layer, and thus observe the influence as the model produces outputs. Such an influence can also be observed using grand-tour [START_REF] Asimov | The Grand Tour: A Tool for Viewing Multidimensional Data[END_REF], a projection of embeddings in n-dimensions. For instance, as seen in Figure 2.10 y n equals to the number of classes, the model as to identify in its training. However, visualizing n-dimensions on a 2D screen is a challenge by itself. Grand-tour tackle this by relying on animation to rotate such space and emphasize a few dimensions at the time, and on interactions to enable users to manually arrange dimensions as they want, to pick which dimensions to visualize more easily at the expense of others. Using this, it has been observed how the separation of different concepts such as classes corresponding to shoes, and those corresponding to clothes may emerge from early layers [START_REF] Li | Visualizing Neural Networks with the Grand Tour[END_REF]. The impact of training on models' inner representation can also be monitored using this approach by animating representation scatter plots after each training step to, hopefully, highlight the convergence of the model. By monitoring the coordinates of centroids from each cluster, after each training epoch, one can use such an approach to grasp an overview of a model's training [START_REF] Rauber | Visualizing the hidden activity of artificial neural networks[END_REF]. While the study of inner representations through the help of dimensionality reduction techniques may yield interpretable results, due to the stochasticity of algorithms such as t-sne and fine-tuning of hyper-parameters that may be required to optimize the final layout, we argue such visualization should serve as an entry-point to draw hypotheses from rather than a way to interpret a model's decisions.

Model-Agnostic Methods

As illustrated in Figure 2.4, another approach to reach interpretability of Deep Neural Networks is to consider them as black-boxes from which only inputs and outputs may be understood by humans. The advantage of such a building block is that it is model-agnostic, hence it can be applied regardless of a model's architecture, and in some cases even of the data they manipulate. A common method is to distill a trained model's knowledge into a smaller, and thus easier to interpret, model [START_REF] Tan | Interpretable Approaches to Detect Bias in Black-Box Models[END_REF]. This relies on the argument that simpler models such as decision trees can reach sufficient performances to be deployed in real-world situations such as in healthcare [START_REF] Stiglic | Interpretability of machine learning-based prediction models in healthcare[END_REF] while providing to health professionals the ability to inspect their decision process. To do so, we can train a DNN, as we normally would, e.g. a large model with a lot of training data. Such a model may then reach high performances on its task, and learn to model patterns in its training data, key in distillation. Finally, the next step is to use a smaller model, e.g. a small DNN, or a rule-based model, and train it to mimic the bigger model's outputs on the same inputs. For instance, this can be done by altering the training loss L as introduced in Section 2.1.2, to ||y -y|| 2 , where y is the output of the smaller model, and y the output of the larger one. As a result, the smaller model not only learns to be accurate, as it would have using ground-"Wolf" Explanation 1 2

Figure 2.11 -Example of insight gained using lime [START_REF] Tulio Ribeiro | Why should i trust you?: Explaining the predictions of any classifier[END_REF], when given the image of a husky x, the analyzed model fails and predicts "wolf". By looking at lime's explanation, with the gray areas corresponding to the "super-pixels" removed, we can see that the analyzed model relies on the presence of snow in the background rather than the dog to yield an output, hence the mistake. This example and images were sampled from [START_REF] Tulio Ribeiro | Why should i trust you?: Explaining the predictions of any classifier[END_REF].

truth labels but also how to behave, i.e. how to handle unexpected data, as the bigger would. In particular, the smaller model learns how the bigger model distributes errors over non-ground-truth classes, which contains a large amount of knowledge on its decision process [START_REF] Hinton | Distilling the Knowledge in a Neural Network[END_REF]. That way, the decision process of a complex DNN can be summarized as rules to follow to grasp why, for example, a patient may have been classified as potentially carrying a disease (e.g. under 30 years old, and fever). Since we rely on smaller models, such a process may yield performance drops, their accuracy, hence distillation thrives when applied to situations where decisions' accuracy is of the utmost importance (e.g. healthcare). Thus, in many cases, any explanation provided by a model is required to be analyzed by a domain expert (e.g. a doctor) who evaluates, and sometimes fixes models' decisions. However, despite promising results, and distillation being an active research domain for interpretability, there is an equilibrium to be found between a distilled model's accuracy, and its complexity that may make it inherently difficult to interpret. To illustrate, while each rule in a rule-based model might interpretable, the process that leads the creation of those rules, and paths to decisions remain unclear. This is emphasized as the number of rules increases, and hence when it becomes too laborious to follow the rules applied to a particular dataset [START_REF] Zachary C Lipton | The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery[END_REF]. Nonetheless, when applied locally i.e. for a single data-point, distillation can provide useful insights on how a model behaves.

Another way to analyze DNNs with a Model-Agnostic lens (i.e. using only inputs and outputs), is to apply disruptions over an input, and observe how they may affect the output. This is the case for lime [START_REF] Tulio Ribeiro | Why should i trust you?: Explaining the predictions of any classifier[END_REF] one of the most popular model-agnostic approaches for interpretability in the industry. In lime, an input is altered multiple times. For instance, when applied to an image, it consists in dividing it into clusters of pixels using a segmentation algorithm, e.g. growing regions. Then, lime computes the complete combination of this image in which those clusters may be replaced by their average value, gray or dark. Each variation is forwarded to the analyzed model, in order to collect its impact on the output (e.g. less confidence on the prediction of a class). This is then used to train a simple linear model to predict the analyzed model's output with respect to its altered input. By doing so, the linear model learns which super-pixels are key in order for the model to yield its output. Thus, it can be used to grasp the minimal number of super-pixels required for the model to reach an output similar to what it did with its unaltered input. As illustrated in Figure 2.11, using lime, Ribeiro et al. understood how a neural network exploits the snow in the background of images to distinguish between huskies and wolves. Similarly, occlusion patches can be applied to images in order to visualize which regions of the image may increase or decrease the model's confidence for an output [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF]. While being less precise than lime, the advantage of such an approach is that it does not require a regression by a linear model, and hence can be faster as there are fewer combinations of inputs occlusion to evaluate.

Leveraging Building Blocks in Visual Analytics

While the aforementioned building blocks of model interpretability are crucial for the analysis of single elements (e.g. a neuron, or a class), it can be quite time-consuming to go through a large number of individual blocks to produce explanations [START_REF] Lapuschkin | Unmasking Clever Hans predictors and assessing what machines really learn[END_REF]. However, in order for experts to obtain a broad understanding of the decision process of models, and how it may convey biases, they need to analyze different decisions which themselves can have multiple explanations [START_REF] Sokol | One Explanation Does Not Fit All[END_REF]. Thus, raising the need for interactive interfaces to quickly probe building blocks. To this end, the community relies on Visual Analytics systems which often combine those blocks in multi-coordinated views to study decisions through the lens of different blocks at once [START_REF] Matthew Hohman | Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers[END_REF]. Those systems which mainly revolve around activations, often include a representation block, such as a dimensionality reduction of embeddings. When targeting an expert audience, the purpose of those tools is often to draw a hypothesis of a model's behavior which then may be confirmed or denied through an in-depth analysis and statistical evidence external to the tool. Figure 2.12 -Overview of CNNVis [START_REF] Liu | Towards better analysis of deep convolutional neural networks[END_REF], a visual analytics system leveraging activation building blocks to display to what neurons may be sensitive to. This system follows, the model's architecture to display with the help of clustering methods, among others, the top-k images of neurons. This view was sampled from an online prototype [START_REF]CNNVIs online prototype[END_REF].

Interactive Activations

By itself, the visualization of a single activation only provides cues on what triggers the inspected neuron. However, a model often relies on a large combination of neurons to produce an output, hence the need to understand how neurons interact with each other. To this end, early works provided an overview, in which users could browse the layers of the model, each depicted as a matrix with cells corresponding to the activation of a neuron for a given input (a picture sampled from a webcam in [START_REF] Yosinski | Understanding Neural Networks Through Deep Visualization[END_REF]). Similarly, in CNNVis [START_REF] Liu | Towards better analysis of deep convolutional neural networks[END_REF] an activation overview display activations in a directed node-link diagram. Such a diagram follows the design of a parallel plot, in which, from left (inputs), to the right (outputs), each layer is a vertical axis, composed of activations of neurons within it, and interconnected to the previous layer and the next one. As illustrated in Figure 2.12, neurons are represented by the top-k images yielding the highest activation for the complete dataset. Such an overview helped experts grasp insights on how, for example, the model's gradients vanished during training. The interactions as designed in those systems, i.e. the ability to browse layers, and select neurons for further information such as alternative visualization, is a key feature for experts to actively debug their model. This is due to the fact that causes for a model to fail can be multiple, and occur at different levels, e.g. gradients throughout the model, or a neuron confusing similar features of classes. CNNVis targets datasets with a limited amount of classes to output, thus raising scaling issues when deployed to larger datasets with thousand of classes such as ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF].

However, displaying each neuron of a model can be overwhelming, especially with the ever-increasing size of state-of-the-art models. Thus, more recent endeavors reduce the number of activations for experts to assess, and scale up those systems to larger models and datasets. In Activis [START_REF] Kahng | ACTIVIS: Visual Exploration of Industry-Scale Deep Neural Network Models[END_REF], a layer can be selected by clicking on it in a model's structure view, similar to tensorflow's [START_REF] Wongsuphasawat | Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow[END_REF], and then, each neuron is displayed as a column in a table with instances as rows. That way, each cell encodes through a single hue-colored rectangle the intensity of an activation. Thanks to this, Activis provides traditional table interactions such as filtering and re-ordering to quickly probe interesting activations. In Activation Atlas [START_REF] Carter | Activation Atlas[END_REF], feature attributions, i.e. activation + gradients, of each neuron are displayed in a dimensionality reduction layout (representation block) to bring together similar activations of the same layer. This helped users grasp insights on how the model may tend to focus on the water in the background to separate "fireboat" and "streetcar" classes from ImageNet. One limitation of such a system, apart from its computational cost, is that each neuron is analyzed individually, while it has been argued that DNNs may learn a hierarchical representation of the data, and hence that they rely on a combination of neuron across layers to identify the class of an image [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF]. To get a hierarchical sense of data flowing through convolutional layers and units, openAI's Microscope [158], arranged model activations in a node-link diagram following the model's structure. Such a tool can also provide details on-demand switching from layers from which only a summary of each layer is visible, to a unit view with every unit from a layer displayed, to finally a detailed view of a selected unit, with among others, the top-k images. Despite those endeavors, this system suffers from scalability issues as the size of models to analyze increases, going through all layers and units can be an overwhelming task. In order to tackle such an issue, summit [START_REF] Hohman | Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations[END_REF] proposes to build a semantic graph that highlights the influential connections between neurons for a given image, along with an interactive representation block aside to provide an overview on how the model associates classes. This tool helped experts grasp, for instance, how a model distinguishes between the similar brown and black bears classes using neurons sensitive to the respected colors of furs and faces of bears.

Sequential Activations and Gradients

In domains the manipulated data is sequential. For instance, in NLP, with sentiment analysis in which words of a sentence need to be evaluated jointly. This raises new challenges as models may need to handle time-dependent concepts to yield accurate outputs, e.g. the position of words in a sentence to generate text. Such a sequential constraint on the data they manipulate impacts their design to the point where those models incorporate time-varying activations which can, for example, convey information from previous inputs. Such a behavior hinders a straightforward comprehension of their decisions. Hence a clear understanding of which previous inputs weigh in when a model predicts a decision being at the stem of interpreting them.

In a vast majority of visual analytics systems addressing the challenge of interpreting models handling time-varying data, the focus is on those activations and how their intensity patterns may be correlated with inputs. For example, in LSTMVis [START_REF] Strobelt | LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks[END_REF], which design focuses on the study of the activations of recurrent models applied to the character and word prediction, activations are represented as lines with inputs as the horizontal axis, and intensity as a vertical axis. In this system, users formulate hypotheses as queries filtering activations matching a selected interval of inputs. Such queries can be guided by complementary information such as a Part-Of-Speech (POS) tagging of words, top-k predictions and, coordinated complementary heatmaps of data such as word count. Unlike most instance-based visual analytic systems, LSTMVis offers a comparison with similar examples (i.e. sentences) from a dataset, key to grasping patterns in sequential activations. In RNNVis [START_REF] Yao | Understanding Hidden Memories of Recurrent Neural Networks[END_REF], those activations are groups of clusters with similar intensity and represented as heatmaps linked to POS-colored word cloud visualization of inputs over the complete dataset. A key feature of this system is its glyph encoding (i.e. a combination of graphical objects to represent multiple attributes of a data-point ) which summarizes the evolution of activations with respect to words in a sentence. This enables users to quickly compare the increase or decrease of aggregated activations intensity through bar-charts as inputs go through the model. In opposition, RNNBow [START_REF] Cashman | RNNbow: Visualizing Learning via Backpropagation Gradients in Recurrent Neural Networks[END_REF] focuses on the evolution of gradients of a character prediction RNN model during training represented as bar-charts (one bar per training batch). Thanks to this, users can overview how the model may learn, and upon selection of a batch, understand how the model may be sensitive to previous outputs. RNNBow has been shown to be particularly useful to detect vanishing gradients i.e. the limit of how far in the past a model may be able to seek information.

More recently, models with attention [START_REF] Bahdanau | Neural Machine Translation by Jointly Learning to Align and Translate[END_REF] increasingly gained popularity due to their improvement of state-of-the-art performance, and their attention mechanism is considered more interpretable than recurrent models. Such attention is a form of activation which associates inputs together. Commonly, attention is represented in instance-based visualization as graphs with bipartite connections from which each side is an input sequence (e.g. a sentence), and thickness of connections between elements of this sequence (e.g. words) encodes the intensity of the association. In seq2seq-vis [START_REF] Strobelt | Seq2seq-vis: A visual debugging tool for sequence-to-sequence models[END_REF], such a view is complemented with a display of the top-k prediction for each input, which can at anytime replace an input with an interaction, and compared to the initial sequence of inputs. This system also provides complementary information such as a representation view of neighbor inputs, along with a list of similar sequences containing a selected input.

Nowadays, attention models such as transformers [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] exploit attention in modules used in each layer, as opposed to recurrent models with a single activation layer addressing the sequential aspect of its data. Thus, along with those models arose the challenge to visualize each of those attention maps-which can spread over hundreds of modules. To do so, BertViz [START_REF] Vig | A multiscale visualization of attention in the transformer model[END_REF] relies on interaction to enable users to quickly switch between modules, and within one module, to display on hover the connections of one input (words) to the others, while filtering those from other inputs. In order to guide users' interaction, such a system also provides an overview that displays each attention graph in a matrix in which rows are layers, and columns attention modules within a layer. Thanks to this, users can grasp patterns that may require further inspection. In EXBERT [START_REF] Hoover | exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models[END_REF], such an overview is represented as a heatmap, in which each module is a column, and each input is a row. Similar to seq2seq-vis, this system provides, upon selection of an input, a complementary representation view of neighbors with respect to corpus meta-data such as POS. Language interpretability tool [START_REF] Tenney | The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models[END_REF] sought to provide a modular, and easy-to-use system for domain experts. In such a tool, the attention graph must be selected through a drop-down list of each module available. Indeed, this tool provides a vast collection of interpretability building blocks such as, among others, a model-agnostic one with LIME, a representation with a projection of embeddings, and metrics such as confusion matrices. Those blocks may help users understand which attention module might be relevant, and thus limit the amount of exploration required by users. Finally, Attention Flows [START_REF] Joseph F Derose | Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models[END_REF] provides an overview of attention modules as heatmaps regrouped per input in a radial layout. Such a tool addresses the influence of BERT pre-training on model predictions by comparing two transformers, and two input sequences at once. While those visual analytics systems designed to address attention in models yielded great results, it is worth noting that according to [START_REF] Jain | Attention is not Explanation[END_REF], they might be taken carefully following the well-known precept "correlation is not causation". This is due to the fact that according to them, in order for attention to be explanations, other approaches such as gradient-based saliency should yield overlapping results. Additionally, if we were to force a particular attention map on the model (e.g. to focus on an input), the model's output should evolve accordingly. Since then, those claims have been argued and challenged by Wiegreffe et al [START_REF] Wiegreffe | Attention is not not explanation[END_REF]. According to them, their experiment to evaluate the overlap of attention and gradient-based saliency is insufficient to prove that they are indeed unaligned. In addition, concerning the second point, this might be due to a disagreement on the definition of explanation, in a sense that, following Wiegreffe et al, attention modules are not the complete explanation to a model's decision but rather an explanation among others (as detailed in Section 2.1.5). Nonetheless, according to [START_REF] Brunner | On Identifiability in Transformers[END_REF][START_REF] Hoover | exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models[END_REF] in order to prevent the pitfall of miss-leading interpretations, attention should be analyzed with the lens of complementary views such as for instance, embeddings and corpus-wide meta information.

For further information on the design of attention models, and visual-analytics systems addressing the interpretability of models with attention such as transformers [START_REF] Vaswani | Attention is all you need[END_REF] applied to Visual Question Answering (VQA), please refer to Chapter 3.

Interacting with Models for Non-experts

A common use of building blocks is to provide a visual introduction to a Deep Neural Network or method to non-expert users. In order to be successful, such an approach heavily relies on trial-error using exploratory interactions. For example, drawing an input to probe a model's behavior on ambiguous handwritten digits [208,[START_REF] Preusse | Interactive Convolutional Neural Network[END_REF], hence the need for tools to be resilient to potential mistakes. TensorFlow playground [START_REF] Smilkov | Direct-Manipulation Visualization of Deep Networks[END_REF], and GAN Lab [START_REF] Ullah Khan | A novel deep learning based framework for the detection and classification of breast cancer using transfer learning[END_REF], took the decision to use toy datasets, with low dimensionality, and small models. Along with encouraging users to get an understanding of the intuitions behind the inner workings of deep learning, this also enables such a tool to train those models within the interface, while monitoring the evolution of its inner representation of data. Such a representation is at the core of those two systems in which users are invited to explore hyper-parameters such as the number of layers, or units within a layer to fit an expected output distribution. In Adversarial-Playground [START_REF] Norton | Adversarial-Playground: A visualization suite showing how adversarial examples fool deep learning[END_REF], the available hyper-parameters are adversarial methods that can be used to alter a model's inputs and observe the new predictions. Similar to model-agnostic blocks, such a system focuses on the inputs and outputs of a model rather than its inner workings. In CNN Explainer [START_REF] Zijie | CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization[END_REF], the light is shed on the inner operations of CNNs, such as those leading to the creation of activations. Similarl to TensorFlow playground, those activations are combined in a node-link diagram directed from the left for the input, to the right for the output. On users selection, any intermediate result can be decomposed down to the mathematical operations connecting a selected result to the corresponding one from the previous layer, such as, for instance, the pixel per pixel max function of the ReLU activation method.

Outside of traditional academic media, visual interpretability for education, flourished online as interactive blogs, referred to as explorables. A key feature of explorables, in opposition to other visual analytics tools, is how information is presented to users through scrollable storytelling. This is done to progressively provide to users the intuitions needed to understand the core method presented. In most explorables, available interaction often offers alternative scenarios, on a few selected datapoints, rather than a broad exploration of a dataset. Pearce et al [START_REF] Pearce | Hidden Bias[END_REF][START_REF] Pearce | Measuring Fairness[END_REF] provided a collection of explorables addressing, among others, how biases may emerge from models asked to classify data. Such a collection has the particularity to enable users, through interactions, to propose their own approach to classification, with the conclusion that a perfect solution, even outside of computers may be nearly impossible to reach. To do so, those explorables rely on model-agnostics blocks of metrics and input-output of simplistic classifier models. In order to introduce users to potential biases and productiveness of model decisions, users can be invited, through interaction, to explore relevant features of image faces [START_REF] Wojcik | How Does a Computer "See" Gender?[END_REF]. During such an experiment, users should grasp that contrary to popular belief, those models may not follow human logic, and rely upon unexpected face attributes to output a gender, e.g. one's left eyebrow.

Interpretable Visual Analytics Throughout this Manuscript

As it can be observed in a vast majority of visual analytics systems introduced in the previous sections, activations of models are a core element to analyze. This might be due to fact that each of them may be visualized individually and eventually interpreted, hence one of the main hindrances to interpreting a complete decision process of a DNN, is their number. Nonetheless, those systems share common features such as the presence of an interactive representation viewusually on the left side of the system and are often instance-based, i.e. focusing on a single input at the time, while providing interactions to manually switch inputs. Each of those systems is intertwined with the model they are dedicated to, and its task. Hence, in many cases, they cannot be extended to another model, task, or dataset without requiring a large refactoring, and reshaping them. However, a vast majority of those systems are either applied to CNN models for image classification, or RNNs and attention models applied to text processing (e.g. translation, or completion). This raises gaps of interpretability between those popular models and their tasks, and the ones less addressed such as robotics. Those systems are, however, particularly useful for DL experts to design models, ultimately increasing the general audience's trust in their models [START_REF] Anca D Dragan | Legibility and predictability of robot motion[END_REF].

The next chapters of this thesis are dedicated to the design of visual analytics systems relying on DNN activations as their core element. However, in contrast to the previously introduced systems, in this manuscript, we tackle the under-explored challenge of analyzing, and improving models for robotics. As introduced in Section 1.1, our focus is on three steps of robotics, namely: VQA, navigation, and sim2real ego-pose localization. Each of them is addressed with a different kind of model, along with different challenges. We tackle VQA (Chapter 3) tasks with transformer models, which by their size, and by the fact that they manipulate both texts, and images, and some of their activations working as attention, require particular attention in their design to prevent users to feel overwhelmed by the quantity of data to analyze. Navigation (Chapter 4), is addressed by Deep Reinforcement Learning (DRL) algorithms which, in our case combines both CNNs layers, and RNNs ones which raise the challenge of how to combine visualization of those two kinds of layers. As such a model uses images as input, the visualization of recurrent activation with respect to inputs needs includes multiple images at once without occluding other visualizations on one's screen. In addition, by nature, DRL algorithms are trained by themselves, i.e. without ground-truth, thus when analyzed, they require for humans to evaluate themselves whether, for example, if a trajectory follows a coherent path. Finally, Sim2Real ego-pose (Chapter 5) is addressed using a regression CNN model, in such application, the challenge lies in how one may assess the gaps between simulation and real-world as perceived by the model. Hence, one may need a visual analytics system designed to visualize and compare activations from related sim/real inputs to grasp what may be difficult to transfer for the model, and then contextualize that information within its environment. We expect those contributions to pave the way to broaden the design horizons of visual analytics systems for DNN interpretability outside of the popular image classification and text-processing domains. We also expect that they contribute to raising concerns about the biases of models' decisions (e.g. the learning of shortcuts), along with helping DNN builders to further study and evaluate the reasoning capabilities of their models. As stated in Chapter 1, and in illustrated in Fig. Figure 3.1, this chapter addresses the first step x of our robotic goal, i.e. the ability to understand questions asked in natural language and answer them by analyzing images. Such a domain is known as Visual Question Answering (VQA). This is a testbed for learning high-level reasoning with primary use in Human-Computer Interaction (HCI), for instance, by providing audio description for the visually impaired. However, recent research has shown that state-of-the-art models tend to produce answers exploiting biases and shortcuts in the training data, and sometimes overlook Figure 3.1 -This is chapter is dedicated to providing to robots the ability to answer natural-language questions about images. For example here, when asked the mundane question "Where are my keys?" x, the robot needs to understand what we are looking for, and then search for it within a given image to provide an answer e.g. here "On the desk!". input images, instead of performing the required reasoning steps. This chapter addresses this by introducing VisQA, a visual analytics tool that explores the question of reasoning vs. bias exploitation. It exposes the key element of stateof-the-art neural models -attention maps in transformers [START_REF] Hao | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF]. Our working hypothesis is that reasoning steps leading to model predictions are observable from attention distributions, which are particularly useful for visualization.

The work presented here is a result of a collaboration of three fields, machine learning, vision and language reasoning, and data analytics, which lead to two contributions: one in a Computer Vision (CV) venue, CVPR 2021, and another in Visualization, IEEE VIS 2021 as follows:

• Théo Jaunet, Corentin Kervadec The design process of VisQA was motivated by well-known bias examples from the fields of deep learning and Vision-Language (VL) reasoning and evaluated in two ways. We believe that this work led to a better understanding of bias exploitation of neural models for VQA, which eventually resulted in an impact on its design and training through the proposition of a method for the transfer of reasoning patterns from an oracle model. Secondly, the design of VisQA, and a goal-oriented evaluation of VisQA targeting the analysis of a model decision process from multiple experts, provides evidence that it makes the inner workings of models accessible to users, and that those models may be improved.

VisQA is available online as an interactive prototype https://visqa.liris.cnrs. fr, and code source and data are available as an open-source project: https: //github.com/Theo-Jaunet/VisQA .

A special thanks goes to Corentin Kervadec, Grigory Antipov, and Moez Baccouche without whom the works presented in this chapter would have not been possible. Figure 3.2 -In Visual Question Answering tasks, we provide to a model a question in a textual form along with an image. In our case, we expect the model to analyze the image to answer the given question with a single word. As an example here, when asked "What is the color of the bananas?", the model should output "Green".

Introduction

Visual Question Answering (VQA) systems [START_REF] Antol | Vqa: Visual question answering[END_REF] attempt to answer questions provided as input in a textual form together with a corresponding image. As an example, asking the question "What is the color of the bananas?" to a model, with the input image shown in Figure 3.2 should yield the answer "Green".

Direct applications of such systems are support for the visually impaired, semiautonomous robot navigation through language instructions, and, more generally, Artificial Intelligence (AI) tools covering a broad spectrum of tasks guided through language input. In particular, VQA serves as a testbed for learning high-level reasoning from data, as the performance of targeted models and methods relies on advances in CV, Natural Language Processing (NLP), and Machine Learning (ML). The task deals with large varieties, and solving an instance can involve visual recognition, logic, arithmetic, spatial reasoning, intuitive physics, causality, and multi-hop reasoning. It also requires combining two modalities of different nature: images and language.

Recent VQA models are based on a powerful type of deep neural network called transformers [START_REF] Vaswani | Attention is all you need[END_REF]. Originally developed for NLP tasks, these models have been extensively applied to VQA [START_REF] Lu | Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Hao | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF][START_REF] Yu | Deep Modular Co-Attention Networks for Visual Question Answering[END_REF] and recently even on pixel-level in pure image-based problems [START_REF] Dosovitskiy | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[END_REF][START_REF] Duke | SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation[END_REF][START_REF] Girdhar | Video Action Transformer Network[END_REF][START_REF] Wang | Non-local Neural Networks[END_REF][START_REF] Zhao | Exploring Self-attention for Image Recognition[END_REF]. Transformers are conceptually simple models, which, however, can learn very complex relationships between the items of un-ordered sets, each of which is represented as a (learned) embedding in a vector space. Making sense of a learned neural model and verifying its inner workings is a difficult problem, which we address in this work.

In this chapter, we focus on a typical and important problem arising with trained neural models, and in particular models for vision and language reasoning: as they are trained with supervision to provide correct answers, they often tend to find shortcuts in learning and learn to exploit spurious biases in training data instead of the desired reasoning a human would apply in a similar situation [3,[START_REF] Goyal | Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering[END_REF][START_REF] Kervadec | Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?[END_REF][START_REF] Manjunatha | Explicit bias discovery in visual question answering models[END_REF]]. To provide an example, if a model is asked "What is the color of the bananas"? with he input image shown in Figure 3.2, it might learn to answer "yellow" despite the real color being "green". This is due to fact that "yellow" is the correct answer for a large majority of input images including bananas. Hence, learning to output "yellow" when bananas are involved is easier than solving the correct reasoning problem for only a small minority of cases. An exact definition of the term "correct reasoning" is difficult, we refer to [START_REF] Bottou | From machine learning to machine reasoning[END_REF][START_REF] Kervadec | Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?[END_REF] and define it as algebraically manipulating words and visual objects to answer a new question. In particular, we interpret reasoning as the opposite of exploiting spurious biases in training data.

Existing work on bias reduction and the evaluation of bias origins tends to focus on statistical techniques, whose power lies in quantitative evaluation and visualizations on dataset-level, showing full or marginal distributions of inputs, features, and outputs, and resort to dimensionality reduction. While these techniques are very useful, their power is limited when we search for insights into detailed inner workings of neural models, for which an investigation per sample is more helpful. Only for a single instance, it is possible to observe the origins for lack of reasoning, which can include, aside from errors in the trained reasoning module itself, also problems in the input pipeline (the object detection module) and wrong annotations of ground truth data.

We introduce VisQA, an instance-based visual analytics tool designed to help domain experts, referred to as model builders [START_REF] Matthew Hohman | Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers[END_REF]. Using VisQA, we investigate how information flows in a neural model and how the model relates different items of interest to each other in vision and language reasoning. Attention maps are at the heart of transformer-based deep networks, and as such are the primary object studied by VisQA. It allows an expert to browse through image and question pairs sorted by an automatic estimate of the amount of reasoning that went into answering each sample. Once a pair is selected, users can explore the different attention maps represented as heatmaps. The exploration is guided by their position in the model, but also by color codes that convey the intensity of each head, i.e. whether they focus attention narrowly on specific items, or broadly over the full input set. Complementary dataset-wide statistics are provided for each selected attention head, either globally, or with respect to specific reasoning modes of language functions, e.g. "What is", "Where is", "What color" etc. While the tool is post-hoc, it is also interactive and allows certain modifications to the internal structure of the model. At any time, attention maps can be pruned to observe their impact on the output answer.

VisQA is the result of a collaboration between experts in visual analytics, and experts in Visual Question Answering systems and Machine Learning. As will be detailed in Section 3.4, this collaboration, and data gathered using VisQA, led to improvements of the reasoning capabilities of transformer-based models by introducing new methodological contributions in machine learning and computer vision, which we also recently reported in a different associated publication [START_REF] Kervadec | How Transferable are Reasoning Patterns in VQA?[END_REF]. The usability of VisQA has been evaluated by different experts in deep learning, who were not involved in the project nor its design. We report experiments with qualitative interviews and results in Section 3.8.

In this work, we contribute to a better understanding of bias in VQA models as follows:

• VisQA an interactive visual analytics tool which helps experts to explore the inner workings of transformers models for VQA by displaying models' attention heads in an instance-based fashion.

• A set of visualizations to address bias in VQA systems designed to explore models' performances in real-time with altered attention, and/or by asking free-text questions.

• Insights on the emergence bias in transformers for VQA gained by experts through an in-depth analysis using VisQA, along with an evaluation of its usability to estimate models' predictions and eventually bias exploitation.

visual question answering

Background

We first introduce some background on understanding neural networks in vision and language reasoning, the context of this work, and we provide a short and concise introduction into transformers, the type of neural networks which currently dominates academic and industrial research in language reasoning, and in vision-based language problems.

Transformers and Attention

Following the introduction and success of transformers applied to natural language processing tasks [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF], transformer-based models were also proposed for VQA [START_REF] Gao | Dynamic fusion with intra-and intermodality attention flow for visual question answering[END_REF][START_REF] Yu | Deep Modular Co-Attention Networks for Visual Question Answering[END_REF]. Their key strength is the ability to contextualize input representations, i.e. to take input items like words and objects, each one represented in a vectorial form called "embedding", and to enrich them, adding information on relationships. This is achieved by a series of transformations of the input vectors, which effectively encodes the reasoning process, and the underlying key mechanism is attention (self-attention). We start with a brief overview of how a typical language transformer works by applying it to encode the question "What is the name of the clothing item that is white?".

Step 1: Preparation: Sentence Tokenization -We split the question into elementary language items (called "tokens") with the WordPiece tokenizer [START_REF] Wu | Google's neural machine translation system: Bridging the gap between human and machine translation[END_REF]. Two special tokens are added at the beginning and at the end of the sentence (respectively 'cls' and 'sep'). While the latter encodes the end of the sentence, the former is of the uttermost importance; it is transformed by the model as are the other tokens, with the difference that the 'cls' token is transformed to encode the task-specific information, and the answer. In the given example, at the end of the transformation, it is expected to contain the information required to predict the name of the white clothing. Each token (including special tokens) is then projected into a high-dimensional vector space through a learned dictionary, resulting in a sequence of N token embeddings: L = [l CLS , l 1 , . . . , l i , . . . , l N-2 , l SEP ], l i ∈ R n , n being the (chosen) embedding dimension.

Step 2: Attention Maps -Transformers progressively contextualize each of the N input embeddings by a sequence of self-attention operations (layers), with the objective of making each token embedding "aware" of the neighboring embeddings. In our example, it might be helpful to combine the information from the embeddings of tokens 'item', 'clothing' and 'white' into one "enriched" embedding describing the referred object, as the three words are semantically related. More generally, the so-called "self-attention operations" (layers) are the key elements Question and image are first tokenized and then encoded using vision (in green) and language (in blue) only transformers [START_REF] Vaswani | Attention is all you need[END_REF], followed by (bi-directional) intermodality transformers [START_REF] Hao | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF]. The answer is predicted from the "CLS" token. Yellow and orange rectangles represent, respectively, inter-and intra-modality attention heads. i and j are the layers and head indices used for naming attention heads through the manuscript.

of this type of model. They are implemented via the calculation of attention maps A = {α ij }, which reflect the N 2 pairwise interactions between the tokens, each α ij being the similarity between token embeddings i and j. The similarity function, here, the scaled dot-product, is calculated between a trainable projection of the embedding i, called query, and a trainable projection of the embedding j, called key. The per-token attention energy is normalized into a probability distribution using a row-wise softmax. In our example, the row vector A 7 = {α 7j } j∈{0,...,N-1} encodes the N interactions between 'clothing' and the other words.

Step 3: Token Updates -Each token embedding is updated as a linear combination of a trained function of the input embeddings, called values, weighted by the attention map A. Hence, the model transforms each token by learning a strategy for looking at specific other words.

Multi-headed Attention -Like most classical neural networks, transformers are organized into a sequence of layers. Each of these layers bundles together multiple attention heads working in parallel, which allows the model to learn different cooperating strategies. Different heads might learn different syntactic or semantic language functions, as shown in [START_REF] Voita | Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned[END_REF] for language models, and as we will show in the experimental section for vision and language reasoning. The outputs of the attention heads are combined with standard neural network blocks.

Vision-Language (VL)-Transformers

Transformers have been extended to reasoning on multiple modalities, in particular vision and language, through different types of layers: Language-only and vision-only layers, referred to as intra-modal layers, and language-vision layers, referred to as the inter-modal ones. Figure 3.3 depicts the transformer architecture designed for VQA which we call "VL-Transformer". Each layer is named as X_i_j, where X denotes the layer type (e.g. vision-only intra-modal layer, vision-language inter-modal layer, etc.) and i and j index, respectively, layer and head.

Intra-modality -Both modalities are first processed in two independent streams (cf. Figure 3.3): heads lang_i_j encode question words, and vis_i_j heads encode visual objects detected in the image by an off-the-shelf object detector, a mainstream approach in VQA [START_REF] Anderson | Bottom-up and top-down attention for image captioning and visual question answering[END_REF][START_REF] Hao | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF]. Visual embeddings are the concatenation of 2048-dimensional object appearance embeddings and 4-dimensional bounding box coordinates.

Inter-modality -Subsequent layers combine information between both modalities, c.f. Figure 3.3, in a bidirectional way: from question words to visual objects in lv_i_j, and vice-versa in vl_i_j (lv means 'language to vision' while the opposite vl means 'vision to language'). This requires a minor, but essential, modification of the attention mechanism. Intuitively, and a bit simplified, in vision-to-language heads, the language (word) embeddings are transformed by taking each word and checking its similarity to the full set of visual input objects, and vice-versa for language-to-vision heads. More precisely, query vectors are taken from the modality to be contextualized, and the key and value vectors from the other one. Thereby, in the case of the vision to language heads, vl_i_j, attention maps A V->L are computed as the outer product between the query projections L q of the language embeddings and the key projections V k of the visual ones:

(3.1)
The softmax is applied row-wisely, such that each attention map's row sum to 1. Then, the language embeddings L are updated with the value projections V val of visual tokens:

(3.2)
Where + = represents a residual connection and FFN is a trainable feed-forward layer. For the sake of clarity, we omit the multi-head mechanism in Equation 3.1 and Equation 3.2. Nevertheless, it is important to notice that the inter-modality transformers are multi-headed, similarly to the intra-modal ones. As shown in Figure 3.3 of the manuscript, each lv or vl attention head is immediately followed by an intra-modal attention head called, respectively, vv or ll.

Predicting the answer -The answer is produced by decoding the final representation of the "CLS" token using a 2-layered neural network. It predicts a probability vector over the most frequent answers found in the training set, the answer with the highest score is then chosen.

Training details -For the experiments in Section 3.8, we set the embedding size to d=128 and the number of heads per-layers to h=4. Following [START_REF] Hao | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF], our model is composed of 9 language only and 5 vision only intra-modality transformers layers, and 5 language → vision and vision → language layers. In addition to the VQA objective, we train the model parameters also on MS-COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] and Visual-Genome [START_REF] Krishna | Visual genome: Connecting language and vision using crowdsourced dense image annotations[END_REF] images following the semi-supervised BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF]-like strategy introduced in [START_REF] Hao | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF]. In particular, we trained the model to perform simple tasks such as recognizing masked words and visual objects, or predicting if a given sentence matches the question. After pre-training on these auxiliary tasks, the model is fine-tuned on the GQA [START_REF] Drew | Gqa: A new dataset for real-world visual reasoning and compositional question answering[END_REF] dataset with the VQA objective. Our VL-Transformer is a variant of the LXMERT model [START_REF] Hao | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF], in line with the many works adapting BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF]-like pre-training to vision and language tasks [START_REF] Su | Vl-bert: Pre-training of generic visual-linguistic representations[END_REF][START_REF] Chen | Uniter: Universal image-text representation learning[END_REF][START_REF] Li | Oscar: Objectsemantics aligned pre-training for vision-language tasks[END_REF][START_REF] Gan | Large-Scale Adversarial Training for Vision-and-Language Representation Learning[END_REF][START_REF] Lu | Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF].

Discussion -In this work, we focus on the interpretation of the attention maps, as they contain crucial cues on the internal reasoning in transformers. These maps highlight to what extent a given token has been contextualized by which neighbors, high attention α ij indicating strong interaction between tokens i and j. We argue, that attention maps provide strong insights on how our the model handles interactions between the question the image.

visual question answering

Related Work

Our work is related to building visual analytics tools for interpretability of Deep Learning. In contrast to Chapter 2, this section focuses on how visualizations can be applied to the attention maps from transformer models specialized for VQA. Our design targets in particular the study of attention maps from transformers models to grasp insights on their potential exploitation of bias. This section reviews previous work on the visual analysis of deep learning models, and a review of previous work from machine learning communities to tackle bias in VQA systems.

Interpretability of VQA

The work the most related to VisQA is Attention Flows [START_REF] Joseph F Derose | Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models[END_REF] which addresses the influence of BERT pre-training on model predictions by comparing two transformers models. Like VisQA, such a tool displays an overview of each attention head with a color encoding their activity. While Attention Flows is designed to address the comparison of attention maps of models trained for NLP tasks, in this work, we tackle the challenges provided by the bi-modality of vision and language reasoning, and expand the interpretability of VQA systems which can rely on visual cues or dataset biases. Current practices of VQA visualization include attention heatmaps of selected VL heads based on their activation [START_REF] Harold | What Does BERT with Vision Look At?[END_REF] to highlight word/key-object associations, global overview heatmaps of attention heatmaps towards a specific token [START_REF] Cao | Behind the scene: Revealing the secrets of pre-trained vision-andlanguage models[END_REF], and guided backpropagation [START_REF] Goyal | Towards transparent ai systems: Interpreting visual question answering models[END_REF] to highlight the most relevant words in questions. Following those works, VisQA provides a visualization of every head's attention heatmaps and word/object associations, along with an overview of their activations.

Here, our focus is on post-hoc interpretability [START_REF] Zachary C Lipton | The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery[END_REF], i.e. the analysis of a trained model's decision policy after-the-fact. Our approach relies on instance-based analysis, which displays inner model parameters with respect to the current input. Such analysis is often combined with direct manipulation mechanisms designed to let users experiment with desired input conditions, like drawing the input [START_REF] Carter | Experiments in Handwriting with a Neural Network[END_REF]. Our working hypothesis is that a transformer-based model's mode of operation, i.e. whether it is reasoning or exploiting dataset biases, is observable from its trained parameters, and in particular from attention maps, an intermediate representation dependent on parameters.

Bias Reduction in VQA

Bias reduction has been addressed on the data side through cleaning and balancing. In particular, GQA [START_REF] Drew | Gqa: A new dataset for real-world visual reasoning and compositional question answering[END_REF] focuses on semantics with the help of human-annotated scene graphs and automatically generated questions. On the contrary, VQAv2 [START_REF] Goyal | Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering[END_REF] dataset is crowdsourced, which leads to more natural questions, but includes cognitive and/or social biases [START_REF] Eickhoff | Cognitive biases in crowdsourcing[END_REF] as well as annotation mistakes. In addition, evaluation benchmarks have been specifically designed to identify the presence of bias dependencies. VQA-CP [4] proposes to evaluate models bias dependency by introducing distribution shifts between training and testing sets. However, this approach has limitations to evaluate the decisions and biases exploitation of models as it emphasizes the diversity in answers rather than the reasoning itself. As a result, random predictions can also contribute to improving a model's score on this benchmark [START_REF] Teney | On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law[END_REF][START_REF] Shrestha | A negative case analysis of visual grounding methods for VQA[END_REF]. On the other side, GQA-OOD [START_REF] Kervadec | Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?[END_REF] undertakes a similar strategy while keeping the training set distribution untouched. Instead, the authors propose to evaluate the VQA performances on question-answer pairs regarding their frequency in the dataset. This makes it possible to evaluate both in-and out-of-distribution performances and, at the same time, estimate the reasoning ability of the system. Indeed, if a model's prediction is correct while the question-answer pair is rare, there are chances that the model has not used statistical biases. These datasets and evaluation benchmarks have led to the conception of diverse bias-reduction methods. While an exhaustive survey of these methods is out of our scope, one can mention families of approaches such as training regularization [START_REF] Shrestha | A negative case analysis of visual grounding methods for VQA[END_REF], or counterfactual learning [START_REF] Abbasnejad | Counterfactual vision and language learning[END_REF][START_REF] Gokhale | MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering[END_REF].

In this work, we define bias as the way how a model learns regularities or shortcuts from its training dataset, which may push it to provide answers which it frequently encountered as correct during training, without even considering information from the input image. Technically, we evaluate bias as introduced in [START_REF] Kervadec | Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?[END_REF]. Hence, every question from the dataset is grouped using their topic (e.g. "furniture") and function (i.e. task extracted from the semantic of the question). Both the semantic and topic metadata are provided by the GQA dataset. Then for each kind of question, the frequency of their answers is computed. We estimate that the model exploits bias when it incorrectly predicts an answer which is among the 20% most frequent answers for the given question, while its ground-truth answer is among the 20% least frequent. While our proposed tool allows us to partially open the black-box of transformer-based neural vision and language models in a very general sense, making it possible to inspect how they handle relationships between data items, we nevertheless specifically focus on the important problem of bias reduction.

Motivating Case Study

This work was primarily motivated by growing concerns in the field over bias exploitation of models trained in large-scale settings, in particular when trained on very broad problems like vision and language reasoning [3,[START_REF] Goyal | Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering[END_REF][START_REF] Manjunatha | Explicit bias discovery in visual question answering models[END_REF]. Contributions x the tiny-LXMERT model, with the image of a knife at the bottom y, incorrectly outputs "yes" z with more than 95% confidence. While an exploitation of bias can be considered, we can observe that the answer "yes" represents only 33% of answers of similar questions over the complete dataset. Thus in-depth analysis of the attention of the model may be required to grasp what led to such a mistake. of our group in this area include new benchmarks [START_REF] Kervadec | Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?[END_REF] and additional supervision and regularization for neural models [START_REF] Kervadec | Weak Supervision helps Emergence of Word-Object Alignment and improves Vision-Language Tasks[END_REF]. Here we extend these previous efforts by providing a tool for instance-level visualizations and performing visual analytics on a single sample. This choice was ultimately taken when comparisons of different trained models through statistics failed to provide concrete answers on the sources of confusion and errors. Statistical models enabled us to drill down examinations to a certain minimal level of aggregation, for instance, linguistic function groups, but this kind of analysis was not fine-grained enough.

We illustrate the advantages and the power of instance-level visualizations with our contribution, VisQA on the following case study. It is based on the exploration of a tiny version of the state-of-the-art neural model LXMERT [START_REF] Hao | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF] as described in Section 3.2.2. We provide it with the following input instance, i.e. the image given in Figure 3.4 y, and associated question "Is the knife in the top part of this photo?" x. The correct ground truth answer is of course "No", but the baseline tiny-LXMERT model incorrectly answers "Yes" z. We see the frequency of the different possible answers provided in the interface, and observe that the wrong answer "Yes" is not the most frequent one for this kind of question as "No" is the correct answer 67% of the time, which does not provide evidence for bias exploitation. The objective is to use VisQA to dive deeper into the inner workings of the model.

A first step is to analyze whether the model is provided with all necessary information. While the input image itself does contain all clear pictures of the answer, the neural transformer model reasons over a list of objects detected by a first object detection and recognition module (Faster R-CNN [START_REF] Shaoqing Ren | Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[END_REF]) which may output errors. In the rest of this manuscript, we will refer to this tiny-LXMERT as the noisy model.

Is the knife detected by the vision module?

VisQA provides access to the bounding boxes of the objects detected by the input pipeline. Each bounding box can be displayed superimposed over the input image along with the corresponding object label predicted by the object recognition module. We can observe that the key object "knife" lacks a suitable bounding box or class label, it has not been detected. Since this object is required to answer the question for this image, the model cannot predict a coherent answer. However, the question remains why the wrong answer is "yes", corresponding to the presence of a knife.

Can attention maps provide cues for reasoning modes?

VisQA focuses on attention maps, which are a key feature of transformer-based neural models, as they fully determine relationships between input items. Users can select different heads and explore the corresponding attention maps. For the example case, we are interested in checking the correspondence between the question word "knife" and the set of bounding boxes. This should provide us with evidence of whether the model was capable of associating the concept with the visual object in the scene, which is, of course, not sufficient for a correct answer, but a necessary step. This verification is non-trivial, however, since the model is free to perform this operation in any of the inter-modality layers and heads. VisQA allows to select the different heads, and we could observe that none of the heads provides a correct association. As an example, we can see the behavior of a head in Figure 3.5 x, which associates the word "knife" to various objects, mostly fruits. No other head is found indicating a more promising relationship.

Is computer vision the bottleneck? From the example above, as well as similar observations in other instances, we conjecture that the computer vision input pipeline (notably, the imperfect object detector) is one of the main bottlenecks in preventing correct reasoning. To validate this hypothesis, we explored training an Oracle model with perfect sight, which thus takes as input the ground truth objects provided by human annotation instead of the noisy object detections by a trained neural model. This improves the performance of the model considerably, reaching ∼80% accuracy on the difficult questions with rare ground-truth answers, compared to ∼20% for the standard model reasoning on noisy input. This particularly high difference in performance for questions with rare answers suggests a higher performance in correct reasoning of the oracle model. By loading this model into VisQA, we observe in Figure 3.5 y, that there exists an attention map which associates the word "knife" to a visual object "knife", which, we recall, is an x the noisy model associates the "knife" word with a large number of different objects, including fruit. y the oracle model learns a perfect association between the word "knife" and the "knife" object; z the oracle transfer model associates the word "knife" with two different bounding boxes of type knife handle, whose embeddings are sufficiently close for correct reasoning. Head selections are not comparable between models, we therefore checked for permutations.

object indicated through human annotation. This correct association is reassuring, but by itself does not yet guarantee correct reasoning -further exploration is possible, but we will now concentrate on this problem of finding correspondences between words and visual objects and explore this question further.

Transferring reasoning modes -Given these observations, we conjecture that a transformer model with perfect sight can learn modes of reasoning which are less biased than models trained on real but noisy input data. This is an insight we gathered from using VisQA as a tool for visual analytics. Oracle models, on the downside, are not deployable to real-life problems, as they work on human annotations only, per definition. We explore a solution to transfer reasoning modes from oracle models to noisy input data, which can be done using knowledge transfer by parameter initialization [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF]. In more detail, we pre-train the Oracle model on ground-truth data, and once it reaches convergence, we use its weights as initialization for the model using noisy inputs. We load this model, which we call Oracle Transfer, into VisQA, with the objective of exploring whether reasoning modes have been transferred into a deployable model capable of providing answers given real input. Figure 3.5 shows the visionto-language attention maps from the same layer as the ones explored above 1 . We can observe that the model's attention is drawn towards "handle" visual objects, which are parts of the only knife in the picture. While the object classes "knife" and "handle" are logically different, we can conjecture that their vectorial feature embeddings are different but sufficiently similar to allow reasoning. More importantly, we can deduce that the model adapted to the absence of the knife object by relying on what is available, i.e. knife handles. This leads the model to correctly answer "No", as did the Oracle model on ground truth data, and in contrast to the baseline noisy model without transfer -see also the illustration in Figure 3.6. We further describe this Oracle knowledge transfer, and improvements it provides to transformer-based VQA in a different associated publication [START_REF] Kervadec | How Transferable are Reasoning Patterns in VQA?[END_REF]. This model will be used in evaluations performed by deep learning experts in Section 3.8. All models, noisy, oracle, and oracle transfer, can be tested and explored online in our prototype.

Design Goals

Prior to the design of VisQA, when working on an associated publication [START_REF] Kervadec | How Transferable are Reasoning Patterns in VQA?[END_REF], we conducted discussions with experts (co-authors of the papers associated with this chapter), about their workflow to analyze bias in VQA systems. From those discussions, and literature review introduced in Section 3.3, we distill their needs in the following four main themes of design goals for instance-based analysis.

G1 Examine the performances of each instance for a given model.

To investigate bias in VQA systems as introduced in Section 3.3.2, it is first important to examine the model predictions, along with its confidence score with respect to its inputs. In order to be useful, those predictions need to be combined with the ground-truth, to estimate if the model is wrong, and how frequent is the ground-truth answer and predictions. Inputs need to be inspected as well as they may convey ambiguities that may be at the beginning of an explanation for a mistake. Finally, due to a large amount of data available for inspection, experts 1. Each layer contains several attention heads, and these heads are not ordered. Reasoning associated with a given head in one model can correspond to the reasoning mode of a different head in another model. We cope with these possible permutations in our experiments by searching over all possible heads of a layer may prioritize inputs the more likely to be biased, i.e. those with infrequent ground-truth answers and frequent predictions.

G2 Browse the attention of the model for an instance. Analyzing the attention maps of LXMERT models is crucial for understanding what factors influenced its decision, and eventually whether or not the model attends to both language and vision. While visualizing individually each attention map is feasible, we aim at improving such an exploration, by contextualizing each attention head with their neighborhood (i.e. other attention heads directly connected to it), and position within the model. This is relevant as attention heads get closer to the output, they both encapsulate previous attention, and may be more influential on models' decisions. In addition, experts may need to prioritize heads conveying salient attention, thus those heads need to be summarized and/or emphasized. Finally, for in-depth analysis, experts need to visualize the complete attention map and link each of their elements to human-understandable information, i.e. words of the question and bounding boxes within the input image.

G3 Link attention to language tasks. Once a relevant attention head is observed by an expert, the user should be able to contextualize it with the rest of the dataset. In particular, experts are interested in evaluating whether or not this head is responsive to certain tasks provided by the semantics of questions (e.g. find a color), or rather if the head is responsive to certain topics (e.g. clothing). Such information can be provided in the VQA training dataset, considered as categories.

G4 Explore alternative scenarios. Once cues on how the model uses its attention to output a decision are gathered, the next step, is to test this knowledge by querying the model on altered input or parameters. Ultimately the experts desire to answer questions such as: "would the model have a similar attention if the question were on another object of the image?", or "is this head or group of heads relevant for the final decision?". This can be regrouped into two categories first, the possibility to ask free-from questions, and second the possibility to modify the model's attention. In order to be usable, and due to the number of queries an expert may need to execute, both those manipulations require to interact with the model in a reasonable amount of time, e.g. less than a couple of seconds.

Design of VisQA

We designed VisQA, an visual analytics tool designed to facilitate in-depth analysis of the internal structure of transformers models applied to visual question answering. VisQA implements attention heads and interactive heatmaps visualizations, and other interactions such as free-form question and pruning to provide its answer. This can be achieved by exploring the behavior of a set of attention heads , each producing an attention map , which manages how different items of the problem relate to each other. Heads can be selected , for instance, based on color-coded activity statistics. Their semantics can be linked to language functions derived from dataset-level statistics , filtered and compared between different models. This tool is available online at: https://visqa.liris.cnrs.fr. mechanisms, to assess whether a model resorts to reasoning or bias exploitation when answering questions.

Workflow

Through iterative design resulting from frequent meetings with VQA experts co-authors of this work, we extracted the following workflow of use of VisQA, based on their experience analyzing VQA systems, and the mantra overview first, zoom and filter, then details on demand [START_REF] Shneiderman | The eyes have it: A task by data type taxonomy for information visualizations[END_REF]:

1.
the user picks and loads into the model an instance, informed by the likeliness of the question to be answered with bias;

2.

the result of the model is presented both as attention heads (internal structure of the model), as well as the top-5 predicted answers;

3.

the user then may interact with the model internal structure (e.g. heads intensity, attention maps, etc.) which triggers updates to the statistical views;

4.

bounding boxes are displayed on the input image to reflect based on attention heat-map selections, showing how the model associates words and visual objects.

VisQA can also be used beyond this typical workflow for in real-time query of the model by asking user-defined questions, or pruning attention heads, as further detailed in Section 3.6.4.

Visualization of Instances

At its core, VisQA is organized following an end-to-end approach, from model input to output. This is done to contextualize attention heads with their neighborhood and position with the model (G4). We added input selection to guide users in their exploration, and a visual summary of the attention of the model to keep it visually compact.

Image ranking-by-feature (Figure 3.6 x) -In order to ease user exploration over the complete dataset, VisQA displays images in the top bar from left to right based on the likelihood of their questions to be answered using statistical biases. To do so, we classify questions using ground truth answers, as proposed in [START_REF] Kervadec | Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?[END_REF]: top 20% of the most frequent answer will be classified as Head, as opposed to Tail which describes questions with the least frequent answers. We attribute a score to each image based on their Head-questions/Tail-questions ratio. The more an image has Tail-questions over Head-questions, the higher its score is. The underlying hypothesis is that frequent answers will be chosen more likely when a model tends to exploit biases (e.g. "Yellow bananas"). Also, frequent questions are harder to analyze since if any bias is exploited by the model, it will answer correctly. Answers and their frequencies are displayed in (Figure 3.6 y right) following design goal (G4).

Instance view (Figure 3.6 y) -This view, inspired by VL-Transformer representations illustrated in Figure 3.3, is the root of any analysis of attention maps using VisQA (G4). It matches the internal data flow through the internal structure of the model, from left to right: the input image/question pair, layers and heads with intra-modality layers first, and finally the answer output distribution (encoded as horizontal bars). A particular design decision was to display all attention maps at once, using a single-colored rectangle encoding the attention intensity as k-number [START_REF] Ramsauer | Hopfield networks is all you need[END_REF] (see next paragraph for details). In the Instance view, attention heads can be selected with a mouse-over interaction as illustrated in Figure 3.6 z in order to provide details on demand, by displaying the corresponding attention heat-map Figure 3.6 { and head statistics.

Visual summary -Our model uses 136 attention maps with dimensions varying with respect to the number of words in the question and bounding boxes provided. As displaying all of those matrices would prevent experts to analyze them in a reasonable amount of time, we rely on summarizing each of them to a single scalar. Such a scalar, referred to as k-number [START_REF] Ramsauer | Hopfield networks is all you need[END_REF], represents the normalized amount of tokens per row summed up to reach a threshold of 90% of energy. A k-number close to 0 indicates that the corresponding row has peaky attention focusing on only one column (as seen in Figure 3.5 x), and a high k-number encodes a uniform attention (as in Figure 3.5 y). Then we combine each of those k-number together using either min, median, or max functions. Such functions can be selected in VisQA by users, depending on the attention maps intensity they want to investigate. VisQA provides this interaction because for a head to have a low k-number, the majority of its rows needs to be highly activated. This can shadow attention maps with less than half of their rows with peaky attention. In VisQA, the k-number is discretized and color encoded in 4 categories as it follows:

. The decision to use a logarithmic discretization and color encoding, as introduced in [START_REF] Ramsauer | Hopfield networks is all you need[END_REF], was done to emphasize peaky attention maps (k < 12) that need to be prioritized for analysis. In addition, this discretization, used throughout VisQA, is particularly useful to regroup instances in head-stat view Figure 3.6 { and select them by clicking on the corresponding square (legend on the right of y in Figure 3.6) for Head pruning further detailed in Section 3.6.4. Finally, this reduces the learning curve of VisQA, as experts are familiar with this discretization of k-numbers.

Visualization of Selected Heads

VisQA provides details for a selected head in the Instance view using the attention map and head statistics.

Attention Maps (Figure 3.6 |) -are represented as heat-maps, with cell colors encoding the attention intensity over a sequential, single hue scale from no attention ( i.e. 0) in beige, to full attention (i.e. 1) in brown. This matrix-based approach contrasts with bipartite connections representations found in Seq2Seq-Vis [START_REF] Strobelt | Seq2seq-vis: A visual debugging tool for sequence-to-sequence models[END_REF] and BertViz [START_REF] Vig | A multiscale visualization of attention in the transformer model[END_REF]. We use matrices as they provide a clutter-free representation of attention and they can display multiple heads at once by aggregating connections. However, it may suffer from visual clutter issues as the number of words grows. Seq2Seq-Vis tackles this issue by using interactions to hide graph lowest connections (i.e. lowest attention). In our case, we argue that visualizing the attention of one head at a time can lead to a better understanding of its function in the model. Such an exploration of head functions is further detailed in Section 3.8. 3.6 {) -are represented using three charts. A vertical area chart (leftmost chart) represents the distribution of k-numbers of the selected head over the complete validation dataset (around 1500 image/question pairs). The vertical axis encodes the values of k-numbers, while the horizontal axis encodes the density of the corresponding k-number. The current k-number, for the selected head, which corresponds to the image/question pair loaded in VisQA, is represented as a horizontal red bar positioned on the vertical axis. This area-chart provides insights such as the detection of useless heads with constant high k-number which can be reduced to calculation on average overall items instead of selecting specific items. In contrast, heads with constant low k-number can be interpreted as conveying key information. More specialized heads, with bi-modal k-number distributions, can also be observed. Two stacked bar-charts represent the k-numbers of the selected head grouped by question operations (G3). Question operations are ground truth information provided by the GQA dataset [START_REF] Drew | Gqa: A new dataset for real-world visual reasoning and compositional question answering[END_REF] describing the semantic reasoning operation of asked questions (e.g. select, query..). Each bar is associated with an operation, and its length encodes how many questions in the dataset are using the corresponding operation.

Head Statistics (Figure

Interacting with Models

VisQA also includes features, complementary to the usual workflow introduced in Section 3.6.4, designed to investigate hypotheses on reasoning.

Free-form questions -By default, VisQA loads the GQA dataset [START_REF] Drew | Gqa: A new dataset for real-world visual reasoning and compositional question answering[END_REF] to provide images and questions. But at any time, users can type and ask free-form openended questions (G4). Such an interaction allows investigating the model's bias exploitation. For instance, when asked the following question from the GQA dataset "Is this a mirror or a sofa", the model correctly outputs "mirror". However, when asked the following user-inputted question "Is there a mirror in this image?", the model fails and outputs "no". This suggests that the model might have exploited biases when it answered the first question, which is supported by the fact that in the GQA dataset, "mirror" is the correct answer to the question "Is this a mirror or a sofa" in 85% of all cases.

Head filtering (Figure 3.7) -As shown in Figure 3.7 x, attention heat-maps feature two interactions. First, hovering with the cursor a row, a column, a cell, which corresponds to an object in the image, automatically displays its corresponding bounding box along with its label over the input (Figure 3.7 y). The user can also, by clicking on a row, column, or cell, filter attention heads to only keep the ones in which the corresponding clicked element has attention above a threshold. For rows and columns which contain multiple attention values, such a filtering process will merge those values by using one of three functions min, median, and max depending on a user selection. The result of such a filtering, as displayed in Figure 3.7 z, occurs in the instance view in which the size and opacity heads that do not match with the user's query are reduced while the others are preserved. Such interactions facilitate seeking for heads in which a specific association is expected e.g. a word in the question with an object of the image required to answer.

Head pruning -Users can select attention heads by clicking on them in the instance view, or by their k-number category. Such a selection can then be used to prune the corresponding head for the next forward of the model. Pruning here means that the attention head does not perform any focused attention, but uniformly distributes attention over the full set of items (objects or words). Each row of a pruned attention map is thus the equivalent of an average calculation. At any time, users can request a new forward pass of the model by clicking on the top left button "ask" (G4), which allows to see the effect of the configured pruning on the model's predictions. This can be used in order to test hypotheses on attention head interpretations as explored in Section 3.8.2.

Implementation

The GQA [START_REF] Drew | Gqa: A new dataset for real-world visual reasoning and compositional question answering[END_REF] standard dataset provides question/image pairs along with their answers, the ground truth of bounding boxes, and semantic descriptions of questions. By default, VisQA provides around 1500 question/image pairs, but as images are loaded progressively when users request it, such a quantity can be increased without affecting performances. The models have been trained on a significantly larger amount of training data (about 9M image/sentence pairs), which is different from the validation data on which the performance is evaluated. The user interface of VisQA is implemented using D3 [START_REF] Bostock | D3: Data-Driven Documents[END_REF], and directly interacts with transformer models implemented in Pytorch [START_REF] Paszke | PyTorch: An Imperative Style, High-Performance Deep Learning Library[END_REF], using JSON files through a python Flask server. As the possibility to ask free-form questions offers an infinity of possible combinations, each question/image pair is forwarded through the model in a plug-in fashion, i.e. without altering the model and its performances.

Evaluation with Domain Experts

We conducted a user study with 6 experts with experience in building deep neural networks, who were not involved in the project or the design process of VisQA. We report on their feedback using VisQA to evaluate the decision process of the Oracle transfer model, with 57.8% accuracy on GQA, introduced in Section 3.4 on several provided problem instances, as well as insights they received from this experience. Hypotheses drawn from single instances cannot be confirmed or denied, but as illustrated in the following sections, such a finegrained analysis aims to provide cues (often unexpected) that can later be explored through statistical evidence outside of VisQA.

Evaluation Protocol

For each expert, we conducted an interview session lasting on average two hours. Sessions were organized remotely and began with a training on VisQA, showing step-by-step how to analyze attention maps. During this presentation, experts were able to ask questions. The study then began with questions on 6 problem instances, i.e. image/question pairs loaded into VisQA in a browser window on participants' workstations. Those instances were balanced between the prediction failures and successes, head or tail distributions of question rarity as described in Section 3.3.2, as well as our estimation on whether the model resorts to bias for this instance grasped using VisQA. VisQA, configured as conditioned during evaluations is accessible online at: (https://theo-jaunet.github.io/visqEval/). The model outputs were hidden and the experts were asked to use VisQA to provide an estimate for two different questions: (Q1) will the model predict a correct answer, (Q2) what will it be?, and (Q3) does it exploit biases for its prediction, or does it reason correctly? During this part of the interview, experts were asked to explain out loud what lead them to each decision. Once those questions were completed, post-study questions were asked on the usability of VisQA, such as "Which part of VisQA is the least useful?", and "What was the hardest part to understand?". Results -The ability of users to predict failures and specific answers of VQA systems has already been addressed through evaluation [START_REF] Chandrasekaran | Do explanations make VQA models more predictable to a human?[END_REF] under different conditions. The experiment closest to ours is question+image attention [START_REF] Lu | Hierarchical question-image co-attention for visual question answering[END_REF] with instant feedback -similarly to ours, users were asked to estimate whether a model will predict a correct answer when provided with attention visualizations of the model, and reaching a similar score of ∼75% accuracy. The difference is that in [START_REF] Lu | Hierarchical question-image co-attention for visual question answering[END_REF] attention is overlaid over the visual input, whereas our attention maps allow to inspect reasoning in a more detailed and fine-grained manner, and not necessarily tied to the visual aspects. The similarity in results changes when users are asked to provide the specific answer predicted by the model: this accuracy drops to 61% in our case, and to 51% in [START_REF] Chandrasekaran | Do explanations make VQA models more predictable to a human?[END_REF]. While our results are promising, they cannot be directly compared to their results due to the different pool and amount of users. Future work will address studies on a larger number of human experts. It can be observed in its first Language-to-Vision attention maps, that the word "shorts" (column) is strongly associated with the object "shorts" (row). The same phenomenon is also observed for the word "person', strongly associated with objects labeled as "woman" among others.

More importantly, our work focuses on qualitative results of bias estimation in which experts obtained a precision of 75% on whether the model exploited any bias. We extracted the ground truth estimate by comparing the rarity of the question, following the Head/Tail attribution from GQA-OOD [START_REF] Kervadec | Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?[END_REF]. These results are encouraging, as they provide a first indication that the reasoning behavior of VL models can be examined and estimated by human users with VisQA. While 75% of performance reasoning vs. bias is not a perfect score, it is also far away from the random performance of 50%, which is important given the large capacity of these models, which contain millions of trainable parameters. Details on individual performance with respect to questions can be observed in Table 3.1.

Object Detection and Attention

To provide an answer, a model must first grasp which objects from the image are requested and thus are essential to focus on. Such an association needs to occur early in the model as those objects are needed for further reasoning. The experts widely observed high intensity in the first Language-to-Vision (LV) layer. As illustrated in Figure 3.8, when asked "Is the person wearing shorts?", the attention map LV_0_1 has peaky activations in the columns "person" and "shorts". This can be interpreted as the model correctly identifying with its self-attention for language that those two words are essential to answer the given question. In GT: "no" KNIVES Pizza Pizza KNIVES Q: "Are there both knives and pizzas in this image?" "both" 1 2 "and" Figure 3.9 -When asked "Are there both knives and pizzas in this image?", the oracle transfer model fails and answers "yes". By filtering heads associated with a selected word, we can observe that language self-attention heads are more responsive to the word "both" x, as opposed to the word "and" y.

Figure 3.8, the word "person" is associated with the bounding boxes labeled as "woman","shirt","shoe","leg", while the word "shorts" is associated with the "shorts" bounding box. Based on this observation, all experts concluded that the model correctly sees the required objects, and more broadly over the evaluation instances, that the first LV layer might be responsible for the recognition of objects with respect to the question. One of the experts mentioned that therefore, "if we don't see a good word/bounding-box association here, the model can hardly cope with such a mistake and might exploit dataset biases". In order to verify such a statement, we pruned the four heads in this LV layer, to observe how the model would behave with no association in them. From such pruning, we observe that the following vision-to-language layers have lower attention distributions, close uniform in some cases. In addition, after pruning, the model's prediction wrongly switched from "Yes", a rare answer (in Tail), to "No", the most frequent one.

Questions with Logical Operators

During the evaluation, experts were shown two instances with questions containing the word "and". Such instances are interesting because, as one of the experts mentioned, "this word has a lot of importance is this question". To answer correctly, the model needs to grasp that it must analyze the image over two different aspects. With the image, illustrated in Figure 3.9, and asked "Are there both knives and pizzas in this image?", the model fails and answer "yes", the most frequent answer despite having no knife in the picture nor provided boundingboxes. However, when asked "Are there knives in this image?" the model correctly answers "no". This suggests that the model failed to grasp the meaning of the keyword "and", and thus that the self-attention language heads might associate Vision-to-language Vision-to-Vision Figure 3.10 -Without any"hair dryer" provided by the object detector, the oracle transfer associates in its vision-to-language x the object "hand" with the words {"[CLS]","is","?","[SEP]"}. While vision-to-vision focuses on a "knob" object y.

wrong words. Also, swapping the terms "knives" and "pizzas" in the question, yields the correct answer, i.e. "no". This indicates that the model ignores the first term when questions contain the operator "and". Using the head-filtering interaction, we can observe that in self-attention heads, the word "and" has little to no attention. Instead, the word "both" has peaky attention scattered across most of self-language layers, and some language-to-language heads. Pruning those 19 heads makes the model correctly yield "no", regardless of the order the words "knives" and "pizzas" are in the question. Such a behavior can be observed over our evaluation dataset, in which 34 questions have the keyword "and". On those questions the model, without pruning, can provide a correct answer 62% of cases, up to 64% with the two words around "and" are swapped. In opposition, while having the 19 attention-heads with peaky attention for the word "both" pruned, the model reached an accuracy of 76%, down to 74% with words around "and" swapped. Thus, in the worst case, this pruning of the 19 attention heads illustrated in Figure 3.9 is responsible for an improvement of 10% on question containing the operator "and".

Vision to Vision Contextualization

When asked "What is the woman holding?", with the image in Figure 3.10, the Oracle transfer model fails and outputs "remote-control", a frequent answer, instead of "hair dryer". This can be interpreted as the model exploiting a statistical bias from its training dataset. However, in such a dataset, "remote-control" is not among the 10 most common answers to this question. This raises the question of what leads the model to output such an answer. During evaluation on this instance, experts noticed that the object detector failed to provide a "hair dryer" object. Similar to the use case given in Section 3.4, such a mistake forces the Oracle transfer to draw its attention towards other bounding boxes related to the missing object. In this case, as observed by experts, a majority of the vision-to-language reached their highest association between the word "holding" and bounding boxes labeled as "hands". Such an association is expected as held objects are directly related to hands, and no "hair dryer" bounding box is provided. Among those bounding boxes, we can observe the presence of one labeled as "television", and another as "knob" which are associated to "holding" and "woman" in both visionto-vision_2_2 and early vision-to-language layers. This suggests that those heads might have influenced the model's predictions towards "remote-control" instead of the most common dataset bias. This can be confirmed by pruning those heads which yields a more frequent answer: "cell phone". In addition, one of the experts also highlighted that those attention heads had a high association with the tokens "[CLS]", "is", "?", and "[SEP]". Which the expert interpreted as "the model correctly transferred the context of the question".

Discussions, Limitations and Future Work

Usability of VisQA -Overall, VisQA was positively received by experts, during discussions at the end of interviews, they expressed that VisQA is "well designed", "complete", and "particularly useful as VQA transformers are hard to interpret". Two out of the six experts confessed that they felt "overwhelmed at first", but gradually "grasped where to look", and getting "used to interactions". Four out of six experts stated that the head-statistics Figure 3.6 { is the least useful feature implemented in VisQA, as it felt "hard to understand". One expert mentioned that such a view is "useful in theory, but less usable in practice". However, the rest of the experts mentioned that the head-statistics view helped them while analyzing attention to "see if heads acted out of their distribution". Experts were unanimous, the attention heat-map, and in particular, its interactions, is the most useful feature of VisQA, as it "provides a lot of information', and head filtering "speeds up the analysis". One of the experts used this interaction on the first language to vision layer as an entry-point to grasp if the model had every information required to correctly answer the given question.

Expert Suggestions -As expressed by experts during evaluation, the main limitation of VisQA is how instances can be selected by users. Currently, such selections are handled by a bar at the top of the tool (Figure 3.6 x), displaying images ordered from left to right based on the likelihood of their questions to be answered using statistical biases. However, during interviews, experts mentioned their need to quickly switch between similar instances in order to grasp if a behavior can be seen across different cases. To do so, experts suggested that such similarity could be measured at three levels: switch to an instance with the exact same question, switch to a similar image, and switch to an instance in which a selected head has similar attention distribution. In addition, currently in VisQA, it is difficult to evaluate how influential a head is over the model's output. To tackle such an issue, experts proposed to encode this information in the representation size of the instance-view attention heads. The impact of each head over the model's input could be retrieved through back-propagation, in particular by calculating the gradient of the model outputs with respect to a statistic of the attention head, for instance, its k-number. In addition to addressing those limits and experts' suggestions, we plan as future works to adapt the usage of VisQA on other problems involving transformers, such as machine translation, and to further investigate the role of each attention heads as done in [START_REF] Voita | Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned[END_REF]. Scalability -To date, the largest model loaded in VisQA is LXMERT with 12 attention heads per block and 768 hidden dimensions. As illustrated in Figure 3.11, such a model (in red), and tiny-LXMERT (in blue) experiments the same performance drops as question/answer frequencies decreases, for average accuracy of 57.8% on GQA, compared to 59.6% for the large-LXMERT. It is also worth noting that tiny-LXMERT can outperform the standard LXMERT with auxiliary losses [START_REF] Kervadec | Supervising the Transfer of Reasoning Patterns in VQA[END_REF]. In addition, LXMERT is less accurate than the tiny-oracle model we used for evaluation, on questions with rare answers-i.e. those that may be the most susceptible to convey biases. While inspecting this model with VisQA, we noted that the more the number of heads increases, the more summary and filtering of heads became relevant for faster analysis. However, our visual encoding of heads may become tedious to use as the number of heads increases, and the number of pixels allocated per head decreases. A workaround can be to increase the desig-nated space of the model in instance-view, but ultimately, VisQA will be limited by screen real-estate. Thus, larger models (e.g. UNITER-large [START_REF] Chen | Uniter: Universal image-text representation learning[END_REF]), may need their heads to be filtered (e.g. by k-numbers) to avoid being overwhelming, before being displayed in VisQA. Similarly, heatmaps of attention may suffer from the same limitation. However, we decided to use them for attention as the maximum visual object (36 in most VQA systems using a Faster R-CNN), and the average of words per question is known beforehand. For larger sets of inputs, alternative designs such as bipartite graphs combined with aggregation methods to hide elements under a threshold of attention may be more space-efficient. However, we argue that this is a trade-off as displaying every bounding boxes and their attention may be relevant to evaluate a prediction (as depicted in Section 3.4).

Generalization -This work focuses on detector-based VL transformers such as LXMERT, however other VQA systems may include single stream models such as [START_REF] Chen | Uniter: Universal image-text representation learning[END_REF][START_REF] Su | Vl-bert: Pre-training of generic visual-linguistic representations[END_REF]. While VisQA is applicable to both single and 2-stream models, we decided to focus on the latest during evaluation. Those models are considered by experts as more interpretable because self-attention layers for language and vision can be observed separately, and there are no significant differences in performance between both architecture [START_REF] Bugliarello | Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs[END_REF]. VQA systems' state-of-the-art often alternate between detector-based approaches (e.g. VinVL [START_REF] Zhang | Vinvl: Revisiting visual representations in vision-language models[END_REF]) and pixel models [START_REF] Huang | Pixel-bert: Aligning image pixels with text by deep multi-modal transformers[END_REF]. However, at its current state, VisQA is not applicable to the latter type, as it would only require adapting to the higher number of visual tokens (depending on the granularity of the method) which raises the challenge of scalability of heatmaps discussed above. Future works can address this through aggregation methods to reduce the number of tokens displayed (e.g. by dividing input images into regions). Finally, our work mainly focuses on insights on reasoning skills grasped from models on the GQA dataset. This decision was taken because it has been argued [START_REF] Drew | Gqa: A new dataset for real-world visual reasoning and compositional question answering[END_REF] that it involves a larger variety of reasoning skills (spatial, logical, relational, and comparative) than in a dataset with where questions were annotated by humans directly (e.g. VQA2 [START_REF] Goyal | Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering[END_REF]) which contains questions targeting difficult external knowledge (e.g. a name of a baseball team). Despite not being presented in this work, VisQA can directly be used on the VQA2 dataset. 3.12) -Comparison between models, and instances can also yield interesting insights on how a model behaves [START_REF] Joseph F Derose | Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models[END_REF]. Currently, to this end, VisQA memorizes inputs and all intermediate and final results including k-numbers and attention maps. This state can be saved, and then used at any time and compared to a new current instance through the compare button at the top of the instance-view. The comparison itself is obtained by computing the difference of k-numbers, and complete attention maps. As a result, head representations in instance-view now encode, with a single hue color scale, the difference between the two attention maps. Besides, as shown in Figure 3.12, .12 -Difference between the attentions of Head LV_1_0, when asked "is the the train blue?", and "is the the train red?". We can observe that in this head, the attention focuses on different objects (row) depending on the color asked (column).

Comparison of instances (Figure

attention heat-maps also encodes such a difference using a diverging color scale from dark blue for values close to -1, i.e. a cell with high attention in the previous instance but not in the current one, to brown for values close to 1, the opposite. Comparisons are particularly useful when combined with the possibility to ask free-form questions. As it can be observed in Figure 3.12, the manually asked questions "Is the train blue?" and "Is the train red?" are responsible for different attention modes between the word representing the color and bounding boxes of the image. By browsing those bounding boxes, it can be observed that the selected head LV_1_00 associates them, in this case, to the word color only if their color matches. As an example, in Figure 3.12 (right), the third row, which corresponds to the bounding box of a blue train, is associated through attention with the word "blue" but not the word "red". In order for such a comparison to be relevant, shifts between two instances must be on a few words of the question, as otherwise, attention maps rows and columns would not align between the instances, and thus any difference would occur for wrong reasons. We plan for future work to enhance this functionality of VisQA through better visual encoding, more intuitive interactions, and an evaluation.

Conclusion

We introduced VisQA, an interactive visual analytics tool designed to perform an instance-based in-depth analysis of the reasoning behavior transformer neural networks for vision and language reasoning, in particular visual question answering. VisQA allows users to select display VQA instances based on the likelihood of bias exploitation; to display attention head intensities; to inspect attention distributions; to prune attention heads; and to directly interact with the model by asking free-form questions. Our quantitative evaluations are encouraging, providing first evidence that human users can obtain indications on the reasoning behavior of a neural network using VisQA, i.e. estimates on whether it correctly predicts an answer, and whether it exploits biases. VisQA received positive feedback from these experts, who also provided additional qualitative feedback on the nature of the information they extracted on the behavior of different neural models.

Thanks to VQA as introduced in Chapter 3, we have robots able to understand what we are looking for, and seek within an image for it. The next step to find our keys, is for robots to successfully explore an environment (e.g. our apartment) as depicted in Figure 4.1. The following chapter addresses step y, referred to as navigation, and how visual analytics systems can be designed to interpret models trained with Deep Reinforcement Learning (DRL) and ultimately improve them. In this chapter, we present DRLViz, a visual analytics interface to interpret the internal memory of an agent (e.g. a robot) trained using DRL. This memory is composed of large temporal vectors updated when the agent moves in an environment and is not trivial to understand due to the number of dimensions, dependencies to past vectors, spatial/temporal correlations, and co-correlation between dimensions. It is often referred to as a black box as only inputs (images) and outputs (actions) are intelligible for humans. Using DRLViz, experts are assisted to interpret to investigate the role of parts of the memory when errors have been made (e.g. a wrong direction). In this chapter, DRLViz is applied in the context of video games simulators (ViZDoom) for a navigation scenario with item gathering tasks in order to speed up the training process. The challenge of transferring the knowledge acquired in simulation to robots in the real-world will be addressed in Chapter 5. DRLViz has been evaluated by experts.

Finally, we conducted the experimentation of pruning the memory of DRL agents down to core identified element, and evaluated the impact on their performances. This was inspired by Macdonald et al. [START_REF] Macdonald | A rate-distortion framework for explaining neural network decisions[END_REF]'s definition of interpreting Deep Neural Networks (DNN)s, to only consider a set of features relevant if a model's output distribution is preserved despite randomizing the remaining features. We also designed MemRed, an online explorable of such an experiment, tailored for non-expert users, which is detailed in Section 4.7.1 of this chapter.

This chapter is related to the following published materials: Both code (https://github.com/sical/drlviz), and an limited interactive prototype (https://sical.github.io/drlviz) of DRLViz are available online. Similarly, the MemRed (https://theo-jaunet.github.io/MemoryReduction/) and its code (https://github.com/Theo-Jaunet/MemoryReduction), are also accessible.

Introduction

Automatic navigation is among the most challenging problems in Computer Science with a wide range of applications, from finding the shortest paths between pairs of points, to efficiently exploring and covering unknown environments, up to complex semantic visual problems ("Where are my keys?"). Addressing such problems is important for modern applications such as autonomous vehicles to improve urban mobility, social robots, and assisting elderly people. Historically, navigation was often solved with discrete optimization algorithms such as Dijkstra [START_REF] Dijkstra | A note on two problems in connexion with graphs[END_REF], A-Star [START_REF] Hart | A formal basis for the heuristic determination of minimum cost paths[END_REF], Front-propagation [START_REF] Yamauchi | A frontier-based approach for autonomous exploration[END_REF] etc., applied in settings where spatial maps are constructed simultaneously with solving the navigation problem. These algorithms are well understood, but are restricted to simple waypoint navigation. Recently, techniques from Machine/Deep Learning have shown spectacular progress on more complex tasks involving visual recognition, especially in settings where the agent is required to discover the problem statement itself from data. In particular, Reinforcement Learning (RL) and the underlying Markov Decision Process (MDP) provide a mathematically founded framework for a class of problems focusing on interactions between an agent and its environment [START_REF] Richard | Reinforcement learning: An introduction[END_REF]. In combination with deep networks as function approximators, this kind of model was very successful in problems like game playing [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF][START_REF] Silver | Mastering the game of Go without human knowledge[END_REF], navigation in simulated environments [START_REF] Das | Embodied Question Answering[END_REF][START_REF] Gordon | IQA: Visual Question Answering in Interactive Environments[END_REF][START_REF] Parisotto | Neural Map: Structured Memory for Deep Reinforcement Learning[END_REF], and work in Human-Computer Interaction (HCI) emerging [START_REF] Debard | Learning 3D Navigation Protocols on Touch Interfaces with Cooperative Multi-Agent Reinforcement Learning[END_REF].

The goal of Deep Reinforcement Learning (DRL) is to train agents which interact with an environment. The agent sequentially takes decisions a t , where t is a time instant, and receives a scalar reward R t , as well as a new observation o t . The reward encodes the success of the agent's behavior, but a reward R t at time t does not necessarily reflect the quality of the agent's action at time t. As an example, if an agent is to steer an autonomous vehicle, receiving a (very) negative reward at some instant because the car is crashed into a wall, this reflects a sequence of actions taken earlier than the last action right before the crash, which is known as the credit assignment problem. The reinforcement learning process aims at learning an optimal policy of actions which optimizes the expected accumulated future reward V t = ∑ t+τ t =t R t over a horizon τ. If agents trained with DRL were deployed to real-world scenarios, failures and unexpected behaviors [START_REF] Lehman | The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities[END_REF] could lead to severe consequences. This raises new concerns in understanding on what ground models' decisions (e.g. brake) are based [START_REF] Tulio Ribeiro | Why should i trust you?: Explaining the predictions of any classifier[END_REF]. To assess the decision of a trained model, developers [START_REF] Matthew Hohman | Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers[END_REF] must explore its context (e.g. a pedestrian on the road, speed, previous decisions) and associate it with millions of deep networks parameters which is not feasible manually. Analyzing a decision after-the-fact, referred to as post-hoc interpretability [START_REF] Zachary C Lipton | The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery[END_REF], has been a common approach in visualization. It consists in collecting any relevant information such as inputs and inner-representations produced while the model outputs decision. With such an approach, DRL experts explore their models without having to modify them and face the trade-off between interpretability and performances. Visual analytics for post-hoc interpretability [START_REF] Matthew Hohman | Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers[END_REF] yields promising results on tasks such as image classification [START_REF] Olah | The Building Blocks of Interpretability[END_REF], or text prediction [START_REF] Strobelt | LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks[END_REF]; however, it remains an under-explored challenge for DRL with memory.

We built DRLViz, a novel Visual Analytics interface aimed at making Deep Reinforcement Learning models with memory more interpretable for experts. DRLViz exposes a trained agent's memory using a set of interactive visualizations the user can overview and filter, to identify sub-sets of the memory involved in the agent's decision. DRLViz targets expert users in DRL, who are used to work with such models (referred to as developers in [START_REF] Matthew Hohman | Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers[END_REF]). Typically, those experts have already trained agents and want to investigate their decision-making process. We validated DRLViz usability with three experts and report on their findings that informed us on future improvements such as applicability to other scenarios, and novel interactions to reduce the memory of an agent and better find patterns within it.

Context and Background

The context of our work is related to building deep neural network models to train robots to achieve human assistance tasks in real-world environments.

As the sample efficiency of current RL algorithms is limited, training requires a massive amount of interactions of the agent with the environment -typically in the order of a billion. Simulators can provide this amount of interactions in a reasonable time frame, and enable to work with a constantly controlled world, that will generate less noise (e.g. a shade) in the agent's latent representation. We will discuss in the perspectives section, and in Chapter 5 the extension of our work beyond simulators and the knowledge transfer from simulation to real-world scenarios, where variability (e.g. lighting, clouds, shades, etc.) and non-deterministic behaviors (e.g. robots may turn more or less depending on its battery charge) occur.

Navigation Problem Definitions

Our focus is on navigation problems, where an agent (e.g. robot, human) moves within a 2D space we call environment (Figure 4.2). An environment contains obstacles (e.g. walls), items the agent may want to gather or avoid, and is usually bounded (e.g. a room). The goal of the agent can vary according to the problem variation, but typically is to reach a particular location (e.g. gather items, find a particular spot). Importantly, the goal itself needs to be discovered by the agent through feedback in the form of a scalar reward signal the environment provides: for instance, hitting a wall may provide a negative reward, finding a certain item may result in a positive reward. To discover and achieve the goal, the agent must explore its environment using actions. In our case, those actions are discrete and elements of the following alphabet: a∈A, with A ={forward, forward+right, right, left, forward+left}. The navigation task ends when the agent reaches its goal, or when it fails (e.g. dies, timeout).

As the agent explores its environment, it produces a trajectory. A trajectory is a series of positions p (x,y) in a space S bounded by the environment. Those positions are ordered by time-step t ∈ T, where t 0 < t 1 < t n , and the interval between t n and t n+1 is the time the agent takes to act. In addition to positions, trajectories contain complementary attributes b, which may vary depending on the agent goal (e.g. number of gathered items, velocity, etc.). We call step t the combination of both the agent position p and its attributes b, at a given time-step t. Thus step t can be represented as follows <p t , b t >. The transition between steps occurs as the agent makes a decision. An episode groups all iterations from the first step at t 0 , until the agent wins or looses at t n .

Navigation using the ViZDoom Simulation

The simulation environment we use to train agents to navigate is ViZDoom [START_REF] Kempka | ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning[END_REF] which provides instances of the navigation problem based on Doom, a very Reinforcement Learning can be used to solve this problem by using an image as input x at time t. Features are then extracted from this image y, and combined with the previous memory vector t -1 z. Using this memory vector, the agent decides to move forward or turn left, for instance {. popular video game in the '90s. ViZDoom is a 3D world and as such is a proxy problem to mobile service robotics. It supplies different scenarios focusing on various goals (e.g. survive, reach a location, gather items, avoid enemies, etc.). For expert evaluation, and Figure 4.2 we used the k-items scenario from [START_REF] Beeching | Deep Reinforcement Learning on a Budget: 3D Control and Reasoning Without a Supercomputer[END_REF] with k = 4. In this scenario, the agent, walls, and items are randomly placed in an environment at the beginning of each episode. Then the agent needs to explore the environment until it succeeds, fails or reaches a timeout of 525 steps. To succeed the agent must first gather a green armor, then a red armor, followed by a Health Pack, and finally a soul-sphere (blue circle). Gathering the items in another order instantly kills the agent and ends the episode (fail). Gathering an item in the right order grants a +0.5 reward r, while failing grants a reward of -0.25. Additionally, the agent receives a reward of -0.0001 at each step. This scenario has been designed to provide sufficient learning signals to the agent, pushing it towards learning the positions of objects in maps. This is not the case for simpler tasks requiring to search an object like "ObjectNav" [START_REF] Batra | ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects[END_REF], as in this case collecting an object only requires to reach it immediately upon its detection.

Despite ViZDoom being a 3D world, the agent positions p are within a bounded continuous 2D plane corresponding to the bird's eye view of the environment. We summarize a time-step t as follows: <p t , (r t )>.

This task is challenging as the agent is required to take a decision on the next action based on partial information of the environment, i.e. the task is partially observable. The observed image represents the agent's Field of View (FoV) (i.e. what is in front of it), in a 90-degree range and unlimited depth. The agent is required to recall previously seen observations in some way as it doesn't have access to a global view of the map. These views are stored in its latent memory, the representation studied in this work. The agent should also use its memory to encode information on the items it gathered, and the positions of items or walls encountered in order to quickly complete this task.

Deep Reinforcement Learning and Memory

As expressed in the taxonomy [START_REF] Arulkumaran | Deep Reinforcement Learning: A Brief Survey[END_REF], DRL reached state-of-the-art performance in tasks such as robot control [START_REF] Levine | End-to-End Training of Deep Visuomotor Policies[END_REF] and board games [START_REF] Silver | Mastering the game of Go without human knowledge[END_REF][START_REF] Justesen | Deep learning for video game playing[END_REF] where it even surpasses humans in some cases. Recent Deep Reinforcement learning models, such as Deep Q-Networks (DQN)s [START_REF] Mnih | Playing Atari with Deep Reinforcement Learning[END_REF][START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], and Asynchronous Advantage Actor-Critic (A3C) [START_REF] Mnih | Asynchronous Methods for Deep Reinforcement Learning[END_REF], learned to play video games with human-level control using only images as input. As a result, they achieved human-level performances on Atari 2600 games [START_REF] Marc G Bellemare | The Arcade Learning Environment: An Evaluation Platform for General Agents[END_REF] such as breakout. Those models rely on the hypothesis that the optimal action can be decided based on a single frame.

However, these approaches operate on environments that can be totally observed (like chess or GO board game), and not partially observable, with a field of view that is smaller than the environment. To address this, an internal latent memory can be introduced [START_REF] Hausknecht | Deep Recurrent q-learning for Partially Observable mdps[END_REF] to provide a space the model can use to store an approximation of the history of previous inputs and solve navigation problems [START_REF] Mirowski | Learning to Navigate in Complex Environments[END_REF][START_REF] Zhu | Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning[END_REF][START_REF] Oh | Control of Memory, Active Perception, and Action in Minecraft[END_REF], allowing learning in simulated environments such as Matterport3D [START_REF] Chang | Matterport3D: Learning from RGB-D Data in Indoor Environments[END_REF], ViZDoom [START_REF] Kempka | ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning[END_REF][START_REF] Beeching | Deep Reinforcement Learning on a Budget: 3D Control and Reasoning Without a Supercomputer[END_REF].

Visual Analytics and Deep Reinforcement Learning

Thanks to analyses conducted with the help of visual analytics systems addressing sequential models (introduced in Chapter 2, Section 2.3.2), it has been observed that a decision at a time-step t can be affected by an input seen at tn. In our case, such inputs are images and experts must first assess what the model did grasp from them before exploring what is stored in the memory. In addition, the rewards from navigation tasks are often sparse which results in a lack of supervision over actions, known as the credit assignment problem inherent to RL problems (the reward provided at a given time step can correspond to decisions taken at arbitrary time steps in the past). The model interacts with an environment it only sees partially, therefore, its performances can be altered by factors outside its inputs. This forces experts to visualize multiple time-steps in order to analyze a single decision which makes them more difficult to analyze with existing tools.

To our knowledge, DRL visualizations are under-represented in the literature compared to other methods of visualizing deep learning. LSTM activations from an A3C agent [START_REF] Mirowski | Learning to Navigate in Complex Environments[END_REF] have been displayed using a t-SNE [START_REF] Van | Visualizing Data using t-SNE[END_REF] projection. Despite being effective for an overview of the agent's memory, it offers limited information on the role of the memory. T-SNE projections have also been applied to memoryless DRL agents on 2D Atari 2600 games, in the seminal DQN paper [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], and in [START_REF] Zahavy | Graying the black box: Understanding dqns[END_REF]. DQNViz [START_REF] Wang | Dqnviz: A visual analytics approach to understand deep q-networks[END_REF] displays the training of memory-less models under 4 perspectives. First an overview of the complete training, action distribution of one epoch, a trajectory replay combined with metrics such as rewards, and whether an action was random. DQNViz also includes a detailed view to explore Convolutional Neural Network (CNN) parameters. Such a tool demonstrates the effectiveness of visual analytics solutions applied to DRL. However, DQNViz focuses on the training of the model, and how random decisions through training can affect it. In addition, the model of DQNViz is limited to fully observable 2D environments in which the only movements available are left or right and thus cannot be applied to navigation tasks. Finally, DQNViz is not designed to display or analyze any memory.

In this chapter, we address the under-explored challenge of visualizing a trained DRL model's memory in a 3D partially observed environment. We contextualize this memory with output decisions, inputs, and derived metrics. We also provide interaction to overview, filter, and select parts of such memory based on this context to provide clues on agents' decision reasoning and potentially identify how the model uses its memory elements.

Model and Design Goals

This section presents the model we used to design and implement DRLViz. We describe the inner workings of those models and data characteristics. One key aspect is how the memory of DRL is created and updated by the agent, over space and time. Note that those data will be generated and then visualized with DRLViz after the training phase.

DRL Model

The DRL model we relied on only receives pixels from an RGB image as input, from which it decides the action the agent should perform with the Advantage Actor-Critic (A2C) [START_REF] Mnih | Asynchronous Methods for Deep Reinforcement Learning[END_REF] algorithm. The model is composed of 3 convolutional layers followed by a layer of Gated Recurrent Unit (GRU) [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF], and Fully Connected (FC) layers to match the actions set A. This model is inspired by LSTM A3C as presented in [START_REF] Mirowski | Learning to Navigate in Complex Environments[END_REF] with A3C instead of A2C, and a LSTM [START_REF] Greff | LSTM: A Search Space Odyssey[END_REF] instead of GRU. Those changes reduce the agent's training time while preserving its performance. The underlying structure that allows our model to associate raw pixels to an action is illustrated on Figure 4.2 and described as follows:

Stage x: Environment → Image. First, the agent's field of view is captured as image x t , i.e. a matrix with dimensions of 112 × 64 with 3 RGB color channels.

Stage y: Image → Feature vector. The image x t is then analyzed by 3 convolu- tional layers designed to extract features f t , resulting in a tensor of 32 features shaped as a 10 × 4 matrices. These features are then flattened and further processed with a FC layer. Formally, the full stack of convolutional and FC layers is denoted as function f t = Φ(x t , θ Φ ) with trainable parameters θ Φ taking x t as input and given features f t as output.

Stage z: (Features + previous memory) → New memory. The model maintains and updates a latent representation of its inputs using a GRU [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF] layer, a gated variant of Recurrent Neural Network (RNN). This representation, called hidden state h t , is a time varying vector of 128 dimensions, which is updated at each timestep t with a trainable function Ψ taking as input the current observation, encoded in features f t , and the previous hidden state h t-1 , as follows:

h t = Ψ(h t-1 , f t , θ Ψ ).
Stage {: Memory vector → Action. The model maps the current hidden state h t to a probability distribution over actions A using a fully connected layer followed by a softmax activation function, denoted as the following trainable function: a t = ξ(h t , θ ξ ) with trainable parameters θ ξ . The highest probability corresponds to the action a t which the agent estimated as optimal for the current step t.

The full set of parameters θ = {θ Φ , θ Ψ , θ ξ } is trained end-to-end. We used 16 parallel agents and updated the model every 128 steps in the environments. The gamma factor was 0.99, and we used the RMSProp [START_REF] Tieleman | Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF] optimizer with a learning rate of 7e -4. We trained the agent over 5 million frames, with a frame-skip of 4.

During training, the agent does not necessarily pick the action with the highest probability, as it needs to explore its environment, and eventually find better solutions. However, once the agent is trained, it always chooses the action with the highest probability.

Constructing the Memory of DRL

In the partially observed navigation problem we focus on, the agent only sees the current observation, i.e. what is in its field of view at the time-step t. However, past observations are also relevant for decision making (e.g. to gather previously seen items). Therefore the agent needs to build a representation of relevant information extracted from the history of observations. This information is encoded in h t , a high dimensional (128 in our case) time-varying vector.

Figure 4.3 represents the construction process of the hidden states matrix, which consists of the hidden states h t over the time of an episode -the central visualization in DRLViz (Figure 4.4). Each hidden state is vertically aligned per time-step t at which they are produced. Therefore, the accumulation of hidden states forms a large 2D matrix, where the horizontal axis is time (h t-1 < h t < h t+1 ) and the rows are elements. A row of this 2D matrix represents the evolution and activity of a hidden state element through time and space as the agent moves. The activity of a hidden state element is characterized by its value. In our case, each element of the hidden states is a quantity within the range [-1, 1]. A value close to 0 represents low activity, whereas a value close to any extremity represents high activity. As it can be seen in Figure 4.3, hidden states can drastically change their values between two time-steps. Such value changes can be widely observed across hidden states elements during episodes. However, it remains unclear which elements, correspond to which representations, and thus, are responsible for decisions.

Norms of latent activations are an informative way of visualizing influences [START_REF] Cadene | Murel: Multimodal relational reasoning for visual question answering[END_REF][START_REF] Zhou | Learning deep features for discriminative localization[END_REF]. With modern training methods such as weight decay, dropout, and batch normalization, it is highly improbable that a high activation can occur for unused features. An alternative to hidden state activations would be to analyze gradients of action probabilities with respect to hidden states [START_REF] Ramprasaath R Selvaraju | Taking a hint: Leveraging explanations to make vision and language models more grounded[END_REF][START_REF] Cashman | RNNbow: Visualizing Learning via Backpropagation Gradients in Recurrent Neural Networks[END_REF]. Such an approach can provide information on how a chosen action is directly tied to the current state of the memory, and which dimension influences the more this decision. However, in DRLViz we focus on actions through the episode as a sequence rather than small sub-sequences. When visualizing activations of an LSTM on text, Karpathy et al. [START_REF] Karpathy | Visualizing and understanding recurrent networks[END_REF] discovered a pattern occurring outside back-propagation limitations of the gradient signal. A solution would be to display both activations and gradients, however preserving the usability and interpretability of a tool conveying such information is a challenge yet to be tackled.

Design of DRLViz

We built DRLViz as a visual analytics interface to understand the connections between the latent memory representation (as depicted in Figure 4.3) and decisions of an agent trained using Deep Reinforcement Learning. DRLViz primarily exposes the internal memory (Figure 4.4) which is interactive and provides overviewing, filtering and reduction both for exploration and knowledge generation [START_REF] Keim | Visual Analytics: Definition, Process, and Challenges[END_REF]. DR-LViz is designed towards experts in DRL to identify elements responsible for both low-level decisions (e.g. move towards a spotted Health Pack) and eventually higher-level strategies (e.g. optimizing a path).

Design Motivation and Goals

We iteratively built DRLViz with frequent meetings from colleagues experts in Deep Learning and DRL (a total of 12 meetings with three experts over 7 months). We first identified their current process to analyze trained agents, e.g. recording videos to playback agents episodes (from its point of view) and decisions (actions probability) to get a sense of the strategy. We also observed experts do a system print of the models' inner values, sometimes add conditions to those prints (e.g. when an item is in the field of view of the agent), and manually look for unusual values. Our approach was to re-build a similar interface in DRLViz with input/output views and facilitate playback, but 1) in an interactive way, and 2) by adding advanced, coordinated views to support advanced models visualization aimed at models developers [START_REF] Matthew Hohman | Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers[END_REF] (e.g. view on the agent's memory).

Based on a review of the current practices of researchers from our focus group, and related work, we identified the following design goals (G) to be addressed to understand the behavior of a trained agent using a learning model for navigation problems:

G1 Show an agent's decisions over (a) space and (b) time, especially input and outputs of the model. G2 Expose the memory's internal structure, i.e. the temporal vector built over time (Figure 4.3). G3 Link memory over (a) time and (b) decisions with multiple endpoints, e.g.

starting from any time, location, memory or trajectory point. G4 Filter a sub-set of the memory (a sub-space) tied to a specific agent behavior or strategy. Those interactions are carried out using a vertical thumb similar to a slider to explore time-steps t and select intervals. Such a selection is propagated to allow the views on the interface, whose main ones are image (perception) and probabilities (of actions) which provide context on the agent's decisions (G1 (b)). The input image can be animated as a video feed with the playback controls, and a saliency map overlay can be activated [START_REF] Tobias Springenberg | Striving for Simplicity: The All Convolutional Net[END_REF][START_REF] Greydanus | Visualizing and understanding atari agents[END_REF] representing the segmentation of the image by the agent. The trajectories view (Figure 4.4) displays the sequence of agent positions p t-1 > p t > p t+1 on a 2D map (G1 (a)). This view also displays the items in the agent's field of view as colored circles matching the ones on the timeline. The position p t , and orientation of the agent are represented as an animated triangle. The user can brush the 2D map to select time-steps, which filters the memory view with corresponding time-steps for further analysis (G3 (a)). DRLViz, also includes a t-SNE [START_REF] Van | Visualizing Data using t-SNE[END_REF] view of time-steps t using a two-dimensional projection (Figure 4.4 bottom left). T-SNE is a dimensionality reduction technique, which shows similar items nearby, and in this view, each dot represents a hidden state h occurring in a time-step t. The dot corresponding to the current time-step t is filled in red, while the others are blue. The user can select using a lasso interaction clusters of hidden states to filter the memory with the corresponding time steps. Dimensions among the selected hidden states can then be re-ordered with any criteria listed in Table 4.1, and brushed vertically (Figure 4.4 {).

Overview and Workflow of DRLViz

The result of such an exploratory process is the subset of elements of the memory (rows) that are linked to an agent's decision (Figure 4.4 {). This addresses the design goal G4. Such subset can be seen as a memory reduction that can be used as a substitute to the whole memory (we will discuss it in the perspective sections). This subset can also be used in other episodes listed as clickable squares at the bottom left corner of DRLViz.

Memory Timeline View

The memory timeline exposes the memory's internal structure (G2), which is vector (vertical column) of 128 dimensions over 525 time-steps as a heat-map (Figure 4.4 y) from which an interval can be brushed for further analysis. Each cell (square) encodes a quantitative value, whose construction is illustrated in

∑ t=1 |h ti -h t+1i | Maximal change. stable change -1 Minimal change. similar | 1 n n-1 ∑ t=1 h ti -1 k k-1 ∑ t=1 h tj |
Elements in average different during an interval of k timesteps than outside it.

Figure 4.3, using a bi-variate color scale from [START_REF] Light | The end of the rainbow? Color schemes for improved data graphics[END_REF] with blue for negative values and orange for positive values. Preserving the values as they were initially produced by the model is pertinent as some memory elements (rows) can have both positive and negative values, which may not have the same signification for the model and thus cause different decisions. This will be further explored in Section 4.6.4.

By default DRLViz displays the vector as it is produced by the model, hence the order of elements has no particular semantic. The memory can be re-ordered using a drop-down menu according to comparison criteria listed in Table 4.1. With the activation criteria, a user can observe elements that may be most involved in decisions, while with change, elements that may be the most used by the model are emphasized, with similar, a user can see elements with constant activations during selected intervals. In addition to those criteria, we provided the possibility to re-order the memory as presented in [START_REF] Carter | Experiments in Handwriting with a Neural Network[END_REF] i.e. a one-dimensional t-SNE projection of the absolute values. The re-ordering can either be applied to the whole memory or a selected interval. An order is preserved across memory filters and episodes until the user changes it.

Derived Metrics View

The derived metrics timeline addresses the design goals G3 and G4. It represents metrics calculated from ground truth information provided by the simulator. Those metrics aim at supporting the user in finding interesting agent behaviors such as What does a trained agent do when it has no health pack in its field of view?. The view encodes measures of both the inputs (e.g. health pack is spotted) simulator (e.g. reward) and outputs (e.g. actions). Finally, DRLViz features a stacked area chart of actions probabilities encoding probabilities per action represented by The derived metrics and stacked area chart are below the memory timeline and share the vertical thumb from the memory slider (G3 (a)) to facilitate comparisons between the memory and the behavior of the agent (G3 (b)) as depicted in Figure 4.5. The derived metrics can be dragged vertically by the user as an overlay of the memory to compare metrics with activation values, and identify memory elements related to them (G4). A constant activation of an element during the same intervals of a metric such as Health Packs in FoV, while being different otherwise; may hint that they are related. We provide a full list of metrics in Table 4.2. Two metrics are particularly complex and described as follows:

variation describes how the the agent's orientation (i.e. its FoV) changes over three consecutive time-steps. High variations indicate hesitation in directions and intervals during which the agent turns around, whereas low variations indicate an agent aligned with where it wants to go. However, in some cases (e.g. the agent stuck into a wall), actions may have no effect on the agent's orientation which leads the variation to remain low. ambiguity of a decision is computed using the variance V of action probabilities. The variance describes how uniform actions probabilities are with respect to the mean. A variance V = 0 indicates that there is no difference between actions probabilities, and hence that the agent is uncertain of its decision. On the other way, a high variance represents strong differences in the actions probabilities and the agent's high confidence in its decision. Since the sum of all actions probabilities equals to 1, the variance is bounded within the range [0, 1]. To ease the readability, the variance is inverted as follows: ambiguity = 1 -V. An ambiguity close to 1 represents incertitude in the agent's decision.

Implementation

To explore the memory of a trained agent, one needs to create instances of exploration scenarios. For experts evaluations (Section 4.6) we used a trained Once he replayed those sequences, he stated that the agent came twice in the same dead-end z. Expert #3 observed a hidden state dimension which is blue when the agent sees the red armor before the green armor, and then remained orange until when he saw the green armor {. Expert #2 probed a dimension that is active as the agent first saw the Health Pack, and remained active until it gathered it. Expert #1 also identified two hidden state elements that change as the agent gathered the health pack and then kept their values until the end of the episode }. Using saliency maps ~, Expert #2 observed that the agent ignored the soul-sphere until it gathered the first three items . Finally, Expert #3 identified clusters in the t-SNE projection which corresponds to the agent's objectives e.g. gathering the green armor .

Evaluation by Experts

We conducted a user study with three DRL experts who are experienced researchers building DRL models and match the target profile for DRLViz [START_REF] Matthew Hohman | Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers[END_REF]. We report on their use of DRLViz, as well as insights they gathered. Those results provide hints that can be leveraged to formulate hypotheses that can then be studied outside DRLViz e.g. through statistical evidence.

Protocol and Navigation Problem

We recruited three DRL experts (Expert #1, Expert #2, Expert #3) from two different academic laboratories to evaluate DRLViz. They were shown a 10 minutes demonstration of DRLViz on a simple ViZDoom scenario: health gathering supreme. The evaluation started with DRLViz loaded with data extracted from a model developed by Expert #1, and ended after 35 minutes, during which experts could explore the displayed data. While using DRLViz experts were told to explain their thoughts and what they wanted to see. Then, experts were asked to fill a post-study questionnaire to collect their first impressions with open questions such as "Which part of DRLViz was the least useful?". The evaluation ended with a discussion guided by the post-study questionnaire on their experience using DRLViz and how it can be improved. The complete evaluation lasted on average 1 hour depending on the length of the discussion. The model used was an A2C [START_REF] Mnih | Asynchronous Methods for Deep Reinforcement Learning[END_REF] with 3 convolutional layers and GRU layer with 128 dimensions as memory.

Feedback from Expert #1

Expert #1 is the most experienced expert for this evaluation as he designed both the model and the navigation task [START_REF] Beeching | Deep Reinforcement Learning on a Budget: 3D Control and Reasoning Without a Supercomputer[END_REF] and created animations of agents' behaviors. Expert #1 was our primary collaborator to design and build DRLViz.

Figure 4.6 shows DRLViz loaded with the k-item scenario. Expert #1 first selected an interval corresponding to the agent searching and gathering the last item. This interval started one step after the agent gathered the Health Pack (third item), and ended as the agent gathered the soul-sphere (last item). Expert #1, then used the change criteria to re-order the interval. While replaying it, he noticed two elements with similar activations (Figure 4.6 }). Those elements remained blue during the interval, however, they were inactivated (gray) during the rest of the episode. With further investigation, Expert #1 noticed that those elements were active 4 steps before the agent gathered the Health Pack. Expert #1 described those elements as flags i.e. elements that encodes binary information. Expert #1's intuition was that the agent learned to complete the navigation problem by focusing on one item at a time. And only used its memory to encode information on items it already gathered, and hence which item it should currently gather. Expert #1 concluded that the two elements may be the agent's representation that it gathered the Health Pack, and hence that it should now focus on gathering the soul-sphere.

Then using the action probabilities temporal stacked area chart (Figure 4.6 y), Expert #1 noticed a specific time interval during which the agent repeated the same action for almost 15 steps. Intrigued by such behavior, Expert #1 replayed this interval and noticed that the agent was within a dead-end (Figure 4.6 z) and repeated the action right until it changed its orientation up to 180 degrees. Expert #1 commented that observing such interval is interesting because as the agent converges towards an optimal policy, it may have fewer chances to encounter dead-ends, and thus forget how to escape them. Expert #1 also observed a similar interval with only right actions in which the agent escaped the same dead-end.

Expert #1 concluded that the dead-end was not encoded in the agent's memory, and hence the agent returned to it while searching for items.

Feedback from Expert #2

Our second expert, Expert #2, started by re-ordering the memory using the stable criteria. He noticed a hidden state element, and zoomed (vertical brush) on it. This element had continuous activations starting as the agent first saw the Health Pack and remained active until the agent gathered both the red armor and the Health Pack. Because such element is active regardless of the items the agent gathered yet, Expert #2 interpreted this element as a flag encoding if the agent has seen the health pack or not.

Then Expert #2 focused on the saliency maps combined with episode playback. He noticed that in one episode, the agent encountered the soul-sphere (last item) before it gathered the red armor (second item). During those time-steps, the saliency maps are not activated towards the soul-sphere despite being the agent's FoV (Figure 4.6 ~), and the memory had no visible changes. Expert #2 intuition was that the agent did not perceive the item. In the final steps of the episode, once the agent gathered the firsts 3 items and then re-encountered the soul-sphere, the saliency maps were activated towards it (Figure 4.6 ) and the memory activations changed. Expert #2 expressed that "It is interesting because as soon as it sees it [the soul-sphere] its behavior changes". Expert #2 concluded that the agent intentionally ignored the soul-sphere before it gathered previous items, and as Expert #1 mentioned, the agent learned to solve this navigation problem by focusing on one item at a time.

Feedback from Expert #3

Expert #3 began his exploration with the t-SNE 2D projection as an entry point to identify clusters of hidden states. Expert #3 selected groups of states using the lasso selection (Figure 4.6 ) to filter the memory timeline. The selected cluster represented consecutive steps, forming a continuous time interval. After replaying this interval, Expert #3 observed that it started at the beginning of the episode and ended when the green armor (first item) entered the agent's FoV. Expert #3 interpreted this cluster as corresponding to an agent objective, in this case gathering the first item.

Following up on the previously identified cluster, Expert #3 re-ordered it with the stable criteria. Expert #3 noticed one particular hidden state dimension that was activated in blue until the green armor entered the agent's FoV, and then was activated in orange for the rest of the episode. Expert #3 interpreted such element activation as a flag encoding if the agent has seen the green armor. However, after observing this element activations across episodes, Expert #3 noted that it was inactivated (grayish) at the start of an episode. After re-playing this episode he observed that the agent had no armor in its FoV, as opposed to the first episode analyzed where the agent started with the red armor in its FoV. In another episode, where the agent has the green armor in its FoV since the start, the element was constantly activated in orange. Expert #3 concluded that this element encoded if the agent saw an armor rather than just the green armor. However, once the agent gathered the green armor, the element remained orange despite still having the red armor in the agent's FoV. Expert #3 added that this element also encodes if the agent gathered the green armor.

Memory Reduction

The following depicts how memory elements may be non-essential in order for DRL agents to preserve their performances and a coherent behavior.

Evaluation of Reductions with DRLViz

As experts noticed during interviews, the agents' memory can often be sparse (e.g. Figure 4.4) or hold redundancy (e.g. Figure 4.6 }). Thus we hypothesize that some elements may either never be activated, or there might be redundant activations at the same time. In order to explore such a hypothesis, we conducted an experiment to assess that some sub-set of the memory is sufficient to solve a navigation problem, and the rest may be discarded. We used the health gathering supreme scenario in which the agent must collect Health Packs to survive, hence it is easier to solve than k-item. During the experiment, with a memory of 512 dimensions, we "removed" hidden state elements by multiplying them by 0 before going any further in the model's decision process. And thus, while still within the model, those nullified elements are conveying no information, and hence should have little to no impact on any decision outside of their absence of activation. We then re-run such a model with its reduced memory and generate new episodes to be evaluated. For the experiment's sake, we decided to only preserve the half of most activated elements of the memory, sampled from DRLViz, with the activation re-order metric on the health gathering supreme scenario and a large memory of 512 dimensions.

The results of this experiment are presented in Table 4.3 which shows similar performances between agents with full and half memory over the same 100 episodes lasting on average around 500 steps. The most notable difference is on the agents' ability to avoid poison vials, with the reduced model taking on average slightly more of them. This might be explained by replaying episodes generated of this model, in which we can observe that the agents' actions seem a bit more erratic, and thus, miss-steps more often. One hypothesis to draw is that the agent has at least 256 non-essential elements. Efficient selection of those elements remains a challenge, as it must account for complex temporal dependencies. This might be due to the fact that the size of the memory (i.e. the number of elements), is a hyper-parameter manually set by experts. And, when deciding on a size, experts often tend to opt for a large memory to ensure that the memory is not a bottleneck preventing the agent from learning complex behaviors. With such memory reduction, i.e. masking undesired elements, they remain within the model and take part in the computation. However, having smaller models would be useful as they may be more interpretable, with fewer memory elements to analyze, but also require less computing power and have a lower energy consumption footprint. A solution would be to train a model with large memory, through our reduction method grasp how many memory elements would be optimal, and then re-train a new model with such a memory size.

MemRed, an Online Explorable

To address the challenge of optimizing the memory of DRL agents, and introduce it to non-experts, we designed MemRed, an online explorable visualization (illustrated in Figure 4.7). This explorable leverage the views of DRLViz to deliver to non-expert more accessible insights on a simplified agent's memory and how it can be reduced. In this explorable, users can only interact with a small memory for easier interpretation. Users can experiment with different strategies to reduce the memory of a simplified model with only 32 dimensions in the k-item scenario, and asses how different their corresponding trajectories are. Here, the focus is on a single episode, and hence our goal for models with reduced memory to complete it with the same number of steps as with full memory. In addition, in order to avoid being too overwhelming, the activation of memory elements is encoded in a single color hue from 0 white to 1 brownish. Traditionally those elements' activations range from -1 to 1, hence we only display their absolute value, representing how "used" is an element rather than what information it conveys.

First, as an introduction, we provide random reductions, which yields agents failing at different degrees from turning on the opposite direction of an item to collect, to turning in circles, skipping items, or succeeding. It is observed that such degrees of failure are not correlated with the number of elements removed, but rather which one. The next strategy is, similarly to the previous section, to re-order the memory using metrics, and only preserve the top half or top-quarter. With this, the best result, with the least number of elements is achieved with to top 8 most changing elements, with the agent almost reaching the last items, but taking too many steps to be there, ending the episode. Finally, the overall best results are obtained by using those top 8 elements and manually adding 5 of them that were active at the end when there was no reduction. This yields an agent with only 40% of its memory able to successfully complete this selected episode. While this paves the way for direct filtering by elements of the memory heat-map in a future version of DRLViz, those results need to be analyzed through the lens of multiple episodes and trajectories required, in order to ensure that the agent is not lacking any knowledge to complete its task in harder situations.

Discussion

In this section, we discuss the collected feedback from experts, as well as the limits of the current version of DRLViz.

Summary of Experts Feedback

Experts filled a post-study questionnaire relative to DRLViz usefulness and usability. Overall DRLViz was positively received by all of them: both Expert #1 and Expert #2 stated that DRLViz is "interesting to explain the behavior of the agent" and "easy to use". However, Expert #3 stated that he felt "overwhelmed at first, but soon got used to navigation". All 3 experts evaluated the 2D t-SNE projection as the most useful view because it can provide insights on the agent's memory and strategies. They used this view as an entry point on at least one episode. They commented that the re-ordering was effective to observe desired hidden states dimensions. Both Expert #2 and Expert #3 used the stable criteria because it highlights elements that are different from the rest and should correspond to the selected interval. On the other hand, Expert #1 preferred the change reordering criteria because those elements have information concerning the interval. Expert #3 also noted that "it s handy being able to drag it up [derived metrics timeline] and overlay it on the hidden states" (G3). The experts concluded that the agent learned to solve this task sequentially, i.e. by focusing on gathering one item at a time. And thus that the agent only stored information corresponding to which items it has gathered rather than the positions of every seen item at any time-steps.

All three experts evaluated the memory reduction interaction that filters the memory view (zoom) as not intuitive and hard to use without losing visual contact with the hidden state dimensions they wanted to focus on. This partially validates our memory reduction goal (G4). On this matter, Expert #1 commented that since this agent's memory has 128 dimensions the zoom is not as useful as it could on larger memories. Expert #2 also commented on the use of the different re-ordering criteria, and that their specific functioning was hard to understand, especially the projection. Expert #3 also mentioned that he "doesn't fully understand how the projections re-ordering methods are helpful". To tackle those issues, Expert #3 suggested using the derived timeline to re-order the memory, i.e. observe hidden states activations when a feature enters the FoV. Expert #3 also commented that a horizontal zoom could be useful to focus on one particular time interval, and reduce the number of steps to observe. Expert #1 mentioned that brushing the memory while keeping activation areas as squares, i.e. both horizontally and vertically could be a better way to implement a more consistent zooming interaction.

Limits

Generalization and scalability are the main limits of the current version of DRLViz. Regarding generalization, specific calculations need to be made such as for the derived metrics timeline that is generated from the simulator i.e. the items in the agent's FoV. So the current metrics are tied to ViZDoom but a minor adaptation of the tool to specific environments will be needed, but requiring technical knowledge. In the next section, we will explain how the interaction techniques in DRLViz can be used beyond the tool for better timeline comparisons. Scalability is always a concern with visualization techniques. DRLViz supports long episodes and variable memory size. However, if those are larger than the screen real estate (e.g. beyond on average 1200 steps and more than 1000 memory dimensions) each memory cell would be smaller than one pixel, and thus difficult to investigate. To tackle such an issue, LSTMVis [START_REF] Strobelt | LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks[END_REF] introduced a parallel coordinate plot with each line encoding a memory element. However, with DRLViz we sought to support trend detection and thus encode the memory overview using colored stripes [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF] which complies with our data density challenge and requirement to align the memory with the derived metrics below. We then rely on zoom interactions for details for both time and elements.

We plan in the future to support aggregation strategies [START_REF] Wongsuphasawat | Life-Flow: Visualizing an Overview of Event Sequences[END_REF] to provide a more compact representation of the timelines. Alignment by an event of interest [START_REF] Taowei | Aligning Temporal Data by Sentinel Events: Discovering Patterns in Electronic Health Records[END_REF] (e.g. gathering an item) may also provide more compact representations of metrics, and also better support comparison of behavior before and after this event. A concern raised by experts was the communication of insights gained during the exploration process. We plan to explore summarizing techniques for those insights, such as state charts [START_REF] Salomón | Human Activity Recognition through Weighted Finite Automata[END_REF] in which each state corresponds to a local strategy e.g. reach an item.

Perspectives

We present and discuss three works in progress that may be potential improvements of DRLViz, based on experts feedback, that primarily expand its exploration power and generalization.

Guiding Exploration with Extended Timelines

During our interviews, experts suggested to better support the memory analysis process, as the current version of DRLViz relies on visual exploration by the user, with no specific guidance. We identified two areas of improvement for a future version of DRLViz: adding more metrics, and advanced filtering. Regarding the metrics, Table 4.2 introduced derived indicators from the agent decision. illustrates that more metrics can be added using variations of their parameters (e.g. changing variability thresholds or the distance to consider an enemy is in the FoV or not) which support more questions a user may want to investigate. Such metrics are represented in a compact way, and easy to activate by scrolling down, while remaining focused on the memory. Regarding the comparison, the current version only implements juxtaposition and overlay; while explicit encoding [START_REF] Gleicher | Visual comparison for information visualization[END_REF] is a third way to compare timelines and memory. We applied this third way by adding a boolean queries builder [START_REF] Lex | UpSet: visualization of intersecting sets[END_REF] using AND or OR to filter timelines. Those boolean operators are also applicable to all views of DRLViz, such as 2D-map, t-sne, or a brush on the memory. This helps users to combine multiple views and answer questions such as Where are the areas of memory with the agent has high health, in this part of the environment, with an enemy and an explosive barrel in FoV? This results in intervals in which the agent is susceptible to shoot on barrels to kill enemies.

In order to summarize the input images and ease their comparison with derived metrics, we developed slit square interaction based on slit tears [START_REF] Tang | Exploring video streams using slit-tear visualizations[END_REF]. With a slit square, a user can brush a square on the inputs. Those squares are then compacted to the width shared by all time-aligned elements in DRLViz.

Generalization to other Scenarios and Simulations

Finally, we started investigating using DRLViz as a general-purpose tool for any trained agents with a memory and spatio-temporal information. Figure 4.8 illustrates DRLViz loaded with a different scenario where the agent shoots towards enemies on the death-match [START_REF] Lample | Playing FPS games with deep reinforcement learning[END_REF] with the Arnold model. In general, DRLViz can be used beyond ViZDoom (e.g. referred in [START_REF] Beeching | Deep Reinforcement Learning on a Budget: 3D Control and Reasoning Without a Supercomputer[END_REF]), such as Atari games [START_REF] Wang | Dqnviz: A visual analytics approach to understand deep q-networks[END_REF] without any major change. Using pixel-based representations [START_REF] Daniel | Designing Pixel-Oriented Visualization Techniques: Theory and Applications[END_REF] and zooming [START_REF] Kerpedjiev | HiGlass: webbased visual exploration and analysis of genome interaction maps[END_REF] would assure scalability of the timeline representations with scenarios requiring more time steps. We plan to conduct further research to identify other metrics and extend DRLViz to other simulators mentioned by our experts, such as Matter-port3D [START_REF] Chang | Matterport3D: Learning from RGB-D Data in Indoor Environments[END_REF] and Habitat-AI [START_REF] Savva | Habitat: A Platform for Embodied AI Research[END_REF] for real-world scenarios, and competitions such as Animal-AI [START_REF] Crosby | The Animal-AI Olympics[END_REF].

Conclusion

In this work, we introduced DRLViz, a visual analytics interface leveraging activation and representation interpretability building blocks. This interface empowers users to overview, filter and select the memory of Deep Reinforcement Learning. Analysts using DRLViz were able to explain parts of the memory of agents trained to solve navigation problems of the ViZDoom game simulator, in particular local decisions and higher-level strategies. DRLViz received positive feedback from experts familiar with DRL models, who managed to browse an agent memory and form hypotheses on it. DRLViz paves the way for tools to better support memory reductions of such models that tend to be large and mostly inactive.

The final step, to find our keys, now that we are able to understand what we are looking for (Chapter 3), and able to explore unknown environments (Chapter 4), is to communicate their location (coordinates and orientation angle) from a given image (as illustrated in Figure 5.1). In addition, in the previous chapters, our robots were trained in simulation which bolsters thousandfold the number of experiments possible in the same amount of time. However, despite the increasing accuracy of realistic 3D simulators for large-scale training of robots, those robots, once deployed to the real-world tend to fail. This is known as the Sim2Real gap which emerges from gaps between simulation and real physical environment (e.g. approximated physics engine), as well as dynamic changes in the real world such as luminosity and lights through days, and objects displacements.

This chapter is dedicated to the analysis of the Sim2Real gap applied to the task of ego-pose estimation, i.e. the estimation of a robot's position using trained models. To do so, we introduce Sim2RealViz, a visual analytics tool to assist experts in understanding and reducing this gap for robot ego-pose estimation To do so, we trained models to yield coordinates and orientation angle from any given image z. Such a task is also used as a proxy to evaluate gaps between simulation and reality.

tasks. Sim2RealViz displays details of a given model and the performance of its instances in both simulation and real-world. Experts can identify environment differences that impact model predictions at a given location and explore through direct interactions with the model hypothesis to fix it. In this chapter, we detail the design of the tool, and case studies related to the exploit of the regression to the mean bias and how it can be addressed, and how models are perturbed by the vanish of landmarks such as bikes. Along with a new visual analytics system, we designed Théo Guesser, an online explorable in which non-expert users are introduced to the ego-pose localization task by playing against a model on it. In addition, users are progressively introduced to the challenge that arises with using real-world data and a model trained in a simulator, along with basic domain randomization and domain adaptation methods to tackle it. Both this challenge and solution are demonstrated using interpretability building blocks we designed. This explorable is available online: https://theo-jaunet.github. io/theo-guesser, and so is its code source: https://github.com/Theo-Jaunet/ theo-guesser.

The works presented here relies on the following published materials:

• Théo Jaunet, Romain Vuillemot, and Christian Wolf. "Théo Guesser". IEEE Workshop on Visualization for AI Explainability at IEEE VIS (VISxAI), 2020.

• Théo Jaunet, Guillaume Bono, Romain Vuillemot, and Christian Wolf. "Sim2RealViz: Visualizing the Sim2Real Gap in Robot Ego-Pose Estimation". NeurIPS XAI Workshop on eXplainable AI approaches for debugging and diagnosis, 2021.

A special thanks goes to Guillaume Bono who handled the robot to collect real-world data, and helped fine-tune the simulator-end of Sim2RealViz.

Introduction

Visual navigation is at the core of most autonomous robotic applications such as self-driving cars or service robotics. One of the main challenges for the robot is to efficiently explore the environment, to robustly identify navigational space, and eventually be able to find the shortest paths in complex environments with obstacles. The Robotics and Deep Learning communities have introduced models trained with Reinforcement Learning (RL), Inverse RL, or Imitation Learning, targeting complex scenarios requiring visual reasoning beyond waypoint navigation and novel ways to interact with robots, e.g. combining vision, robotics, and natural language processing through queries like "Where are my keys?". Current learning algorithms are not sampled efficiently enough, this kind of capability requires an extremely large amount of data. In the case of RL, this is in the hundreds of millions or in the billions of interactions -this simply cannot be addressed in a reasonable amount of time using a physical robot in a real environment, which also may damage itself in the process.

To tackle this issue, the field heavily relies on simulation, where training can proceed significantly faster than in physical (wall clock) time on fast modern hardware, easily distributing multiple simulated environments over a large number of cores and machines. However, neural networks trained in simulated environments often perform poorly when deployed on real-world robots and environments, mainly due to the"Sim2Real gap", -i.e. the lack of accuracy and fidelity in simulating real-world environment conditions such as, among others, image acquisition conditions, sensors noise, but also furniture changes and other moving objects. The exact nature of the gap is often difficult to pinpoint. It is well known that adversarial examples, were only a few pixel shifts occur, considered as small artifacts by humans, or which might even be undetectable by humans, can directly alter the decisions of trained models [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF][START_REF] Moosavi-Dezfooli | Deepfool: a simple and accurate method to fool deep neural networks[END_REF][START_REF] Liu | Analyzing the noise robustness of deep neural networks[END_REF].

The Sim2Real gap is currently addressed by various methods, including domain randomization, where the physical reality is considered to be a single parametrization of a large variety of simulations [START_REF] Tobin | Domain randomization for transferring deep neural networks from simulation to the real world[END_REF][START_REF] Mehta | Active domain randomization[END_REF], and Domain Adaptation, i.e. explicitly adapting a model trained in simulation to the real-world [START_REF] Tzeng | Adversarial discriminative domain adaptation[END_REF][START_REF] Chadha | Improved techniques for adversarial discriminative domain adaptation[END_REF]. However, identifying the sources of the sim2real gap would help experts in designing and optimizing transfer methods by directly targeting simulators and design choices of the agents themselves. To this end, we propose Sim2RealViz, a As illustrated above, images taken from the same coordinates in simulation and real-world x may lead to different predictions due to differences, such as here, among others, the additional presence of a bike in the scene. We are interested in reducing the gap between the SIM and REAL predictions.

visual analytics interface aiming to understand the gap between a simulator and a real-world environment. We claim that this tool is helpful to gather insights on the studied agents' behavior by comparing decisions made in simulation and in the real-world physical environment. Sim2RealViz exposes different trajectories, and their divergences, in which a user can dive deeply for further analysis. In addition to behavior analysis, it provides features designed to explore and study the models' inner representations, and thus grasp differences between the simulated environment and the real-world as perceived by agents. Experts can rely on multiple-coordinated views, which can be used to compare model performances estimated with different metrics such as a distance, orientation, or a UMAP [START_REF] Mcinnes | Umap: Uniform manifold approximation and projection for dimension reduction[END_REF] projection of latent representations. In addition, experts dispose of three different approaches to highlight the estimated sim2real gap overlaid over either 3D projective inputs or over a bird's eye view ("Geo-map") of the environment.

Sim2RealViz targets domain experts, referred to as model builders and trainers [START_REF] Strobelt | LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks[END_REF][START_REF] Matthew Hohman | Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers[END_REF]. The goal is assistance during real-world deployment, pinpointing the root causes of decisions. Once a model is trained in simulation, those experts are often required to adapt it to real-world conditions through transfer learning and similar procedures. In Section 5.6, we report on insights gained through experiments using Sim2RealViz, on how a selection of pre-trained neural models exploits specific sensor data, hints on their internal reasoning, and sensibility to sim2real gaps.

Context and problem definition

We study trained models for Ego-Localization of mobile robots in navigation scenarios, which regress the coordinates (x, y) and camera angle α from observed RGB and depth images. Physical robots take these inputs from a depth-cam, whereas in the simulation they are rendered using computer graphics software from a 3D scan of the environment.

Figure 5.2 provides a concrete example, where two images are taken at the same spatial coordinates x, one from simulation and the other from a physical robot. As our goal is to estimate the sim2real gap, we do not focus on generalization to unseen environments. Instead, our simulated environment corresponds to a 3D scanned version of the same physical environment in which the robot navigates, which allows precise estimation of the difference in localization performance, as the gap leads to differences in predicted positions. The full extent of the gap, and how it may affect models is hard to understand by humans, which makes it difficult to take design choices and optimize decisions for sim2real transfer.

Simulation -we use the Habitat [START_REF] Savva | Habitat: A Platform for Embodied AI Research[END_REF] simulator and load a high fidelity 3D scan of a modern office building1 created with the Matterport3D software [START_REF] Chang | Matterport3D: Learning from RGB-D Data in Indoor Environments[END_REF] from individual 360-degree camera images taken at multiple viewpoints. The environment is of size 22 × 22 meters and can potentially contain differences to the physical place due to estimation errors of geometry, texture, lighting, alignment of the individual views, as well as changes done after the acquisition, such as moved furniture, or opened/closed doors. The Habitat simulator handles environment rendering and agent physics, starting with its shape and size (e.g. a cylindrical with diameter 0.2m and height 1.5m), its action space (e.g. turn left, right or move forward), and sensors -a simulated e.g. RGB-D camera. Depending on the hardware, the simulator can produce up to 10.000 frames per second, allowing to train agents on billions of interactions in a matter of days.

Real-world -As illustrated in Figure 5.3, our physical robot is a "Locobot" [START_REF] Murali | PyRobot: An Opensource Robotics Framework for Research and Benchmarking[END_REF] featuring a RGB-D camera and an additional lidar sensor which we installed. We use the lidar and the ROS NavStack to collect ground truth information on the robot's position (x * , y * ) and angle α * , used as a reference to evaluate ego-pose localization performances on the real-world. To increase precision, we do not build the map online with SLAM, but instead, export a global map from the 3D scan described above and align this map with the lidar scan using the ROS NavStack.

Ego-pose estimation: the trained agent -Traditionally, ego-pose estimation [START_REF] Murali | PyRobot: An Opensource Robotics Framework for Research and Benchmarking[END_REF] x. On the right y such a robot is displayed with respect to proportions in its environment.

of robots is performed from various inputs such as lidar, odometry, or visual input. Localization from RGB data is classically performed from keypoint-based approaches and matching/alignment [START_REF] Brachmann | On the Limits of Pseudo Ground Truth in Visual Camera Re-localisation[END_REF]. More recently, this task has been addressed using end-to-end training of deep networks.

We opted for the latter, and, inspired by poseNet [START_REF] Kendall | Posenet: A convolutional network for real-time 6-dof camera relocalization[END_REF], trained a deep convolutional network to take a stacked RGB-D image X i of shape (256×256×4) and directly output a vector Y i = (x i , y i , α i ) of three values: the coordinates x i , y i and the orientation angle α i . The model is trained on 60.000 images sampled from the simulator with varying positions and orientations while assuring a minimal distance of 0.3 meters between data points. We optimize the following loss function between predictions Y i = (x i , y i , α i ) and Ground-truth (GT)

Y * i = (x * i , y * i , α * i ) over training samples i: L = (1 -γ) ∑ i x i y i - x * i y * i 2 2 + γ ∑ i |α i -α * i |mod 2π (5.1)
where ||.|| 2 is the L 2 norm and γ is a weighting hyper-parameter set to 0.3 in our experiments.

Related work

Closing the sim2real gap and transfer learning -Addressing the Sim2Real gap relies on methods for knowledge transfer, which usually combine a large number of samples from simulation and/or interactions obtained with a simulator simulation with a significantly smaller number of samples collected from the realworld. Although machine learning is a primary way of addressing the transfer, it remains important to assess and analyze the main sources of discrepancies between simulation and real environments. A common strategy is to introduce noise to the agent state based on statistics collected in the real-world [START_REF] Savva | Habitat: A Platform for Embodied AI Research[END_REF]. Additionally, tweaking the collision detection algorithm to prevent wall sliding has been shown to improve the performance in the real-world of navigation policies trained in simulation [START_REF] Kadian | Sim2Real predictivity: Does evaluation in simulation predict real-world performance?[END_REF], which tend to exploit inaccurate physics simulation. Another approach is to uniformly alter simulation parameters through domain randomization, e.g. modifying lighting and object textures, to encourage models to learn invariant features during training [START_REF] Tobin | Domain randomization for transferring deep neural networks from simulation to the real world[END_REF][START_REF] Tremblay | Training Deep Networks With Synthetic Data: Bridging the Reality Gap by Domain Randomization[END_REF]. This line of work highly benefits from domain expert knowledge on the targeted environment, which can provide randomizations closer to reality [START_REF] Prakash | Structured domain randomization: Bridging the reality gap by context-aware synthetic data[END_REF][START_REF] James | Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks[END_REF].

A different family of methods addresses the Sim2Real gap through Domain Adaption, which focuses on modifying trained models' and their features learned from simulation to match those needed for high performance in real environments. This has been explored by different statistical methods from the machine learning toolbox, including discriminative adversarial losses [START_REF] Tzeng | Adversarial discriminative domain adaptation[END_REF][START_REF] Chadha | Improved techniques for adversarial discriminative domain adaptation[END_REF]. Feature-wise adaptation has also been addressed by extensive use of loss [START_REF] Fang | Multi-task domain adaptation for deep learning of instance grasping from simulation[END_REF][START_REF] Tzeng | Adapting deep visuomotor representations with weak pairwise constraints[END_REF], and through fine-tuning [START_REF] Rusu | Sim-to-real robot learning from pixels with progressive nets[END_REF]. Instead of creating invariant features, other approaches perform Domain Adaption at pixel level [START_REF] Bousmalis | Unsupervised pixel-level domain adaptation with generative adversarial networks[END_REF][START_REF] Bousmalis | Using simulation and domain adaptation to improve efficiency of deep robotic grasping[END_REF]. Despite great results, Domain Adaptation suffers from the need for real-world data, which is often hard to come by. We argue that there is a need for the assistance of domain experts and model builders to understand the main sources of sim2real gaps, which can then be leveraged for targeted adapted domain transfer, e.g. through specific types of representations or custom losses.

Interpretable robotics -This classical line of work remains an under-explored challenge when applied to regression tasks such as robot ego-localization, our targeted application, in which attributions may be harder to interpret. To our knowledge, visualization of transfer learning, and especially targeting sim2real is an under-explored area, in particular for navigation tasks, where experiments with real physical robots are harder to perform compared to, say, grasping problems. In [START_REF] Szabó | Visualizing Transfer Learning[END_REF], the evolution of features is explored before and after transfer through pair-wise alignment. Systems such as [START_REF] Ma | A Visual Analytics Framework for Explaining and Diagnosing Transfer Learning Processes[END_REF] address transfer gaps through multi-coordinated views and in-depth analysis for models weights and features w.r.t. domains. Finally, common visualizations consist of heatmaps designed to illustrate results from papers in Machine Learning (ML) communities such as [START_REF] Tzeng | Adversarial discriminative domain adaptation[END_REF][START_REF] Zhu | Sim-real joint reinforcement transfer for 3d indoor navigation[END_REF]. Despite providing insights on how models adapt to different domains, and in contrast to our work, those methods have not been designed to directly target what parts of the environment, or which sensor settings may produce sim2real gaps, which we consider as relevant information for domain experts.

Design Motivation

Prior to the design of Sim2RealViz, we conducted interviews with 3 experts in Robotics and discussed their workflow, with the objective being to address and identify sim2real gaps. Two of those experts, co-authors of this work, then took part in the design of Sim2RealViz. The workflow of interrogated experts consisted in identifying failure cases through statistics or video replaying a robot's trajectory, and then manually finding equivalent images in simulation to compare to.

Tasks analysis

From those discussions with experts, and literature review introduced in Section 5.3, we distill the process of analyzing Sim2Real gap transfer in the following three families of tasks.

T1. Fine-grained assessment of model performance gap between SIM and REAL -What is the best performing sim2real transfer method (e.g. fine-tuning, domain randomization etc.)? What are the optimal hyper-parameters? Answering those questions requires experts to study a large number of predictions in SIM and REAL from a large number of observed images and evaluate performance distribution over different environment conditions and factors of variation.

T2. Identification of the source of the performance gap -what are the factors of variation in the environment, agent, or trained model, which are responsible for the performance gap? This is inherently difficult, as the sources may be due to the global environment (differences in e.g., lightening, 3D scanning performance), the agent (e.g. differences in camera focal length or height) or changes due to the time span between scanning and physical deployment (e.g. furniture changes). In addition, some gaps may also be beyond human comprehension such as adversarial noise. For a human designer, it may not immediately be clear, which differences will have the largest impact on prediction performance.

T3. Closing the sim2real gaps -successful knowledge transfer requires the simulator to be as close as possible to the real-world scenario with respect to the factors of variation identified in T2. The goal is to close the loop and increase prediction performance using the insights gained from using Sim2RealViz.

Design goals

Based on the identified kind of tasks (T.) introduced in Section 5.4.1, literature review, and interviews with domain experts, we distill the following key design goals (G.) for Sim2RealViz.

G1. Summary of models' performances w.r.t. their domain. Following challenge (T1.), in most cases the entry-point for any analysis of sim2real transfer starts with experts trying the assess the performances of their models. Therefore, to give users an overview of their model performances, we aim to create a visual summary of each model, along with their performance distribution, while ensuring a layout that eases comparisons between sim and real overall performances, but also inter-domain comparisons of models to answer questions such as:"Which is the most successful transfer knowledge method on real-world images?" G2. Gather instances based on their performances. To ease the analysis of sim2real performance gaps, users need to quickly identify instances sharing the same performance gaps (T2.). For example, every image that contains a bike in simulation, which is absent in the real-world, hence causing trouble in predictions. That way, the analysis of sim2real gaps in any image of the set of instances, should be useful for the rest of the set. In order to bolster its efficiency, This selection of instances should rely on different metrics such as, among others, the sim or real performance, the position of ground-truth, the position of sim/real prediction, and instances invariant towards transfer knowledge approaches tried.

G3. Per instance Sim and Real comparison.

To identify the source of performance gap (T2.) on a designated instance, users need to inspect humanunderstandable data -i.e. both RGB and Depth images from simulation and real-world used as input. Thus, those images need to be as close as possible in order to quickly overview the most obvious changes such as furniture. Inputs also need to be inspected as perceived by the model, as some sim2real gaps that may be salient to humans, can be irrelevant for it. Finally, the performance of models is usually computed by the distance of their prediction to the corresponding ground-truth. However, if the distance between sim and real is close to 0, and the distance to ground-truth high, perhaps the corresponding instance is not influenced by any sim2real gap.

G4. Adjust Sim and Real images.

In order to ease the sim2real knowledge transfer, users need for the simulation and real-world to be as close as possible (T3.). However, aside from the camera settings such as height and resolution, other shifts need to be tackled by manually adjusting images parameters such as brightness or dynamic range from depth images. The task of finding the correct configuration which makes real-world predictions more accurate can be fastidious. Thus we aim at helping users with ways to manipulate images settings and simulation settings of sensors to help experts quickly address some sim2real gaps issued from inaccurate configurations.

Sim2RealViz: A visual analytics tool to explore the sim2real gap

We introduce Sim2RealViz, an interactive visual analytics tool designed to assist domain experts in conducting in-depth analyses of the performance gaps between simulation and real environments of models whose primary task is ego-localization. The tool is implemented in JavaScript and the D3 [START_REF] Bostock | D3: Data-Driven Documents[END_REF] library to run in modern browsers and directly interacts with models implemented in Pytorch [START_REF] Paszke | PyTorch: An Imperative Style, High-Performance Deep Learning Library[END_REF]. The source code is available as an open-source project at: https://github.com/Theo-Jaunet/sim2realViz . tuned) and displayed on the real-world environment map along with its performance metrics. In particular, Sim2RealViz shows x this model is particularly effective in simulation but we identified errors in the environment, such as the model failing to regress its position because of a closed-door that was opened in training. Such an error can then be selected by instance on the map y to identify key features extracted by the model either as superimposed on the heat-map z or as a first-person view {.

Design rationale

Our design is centered around the comparison of simulation instances and real-world ones. As we deal with complex objects, and because sim2real gaps can emerge from various sources, we implemented several views with different comparison strategies [START_REF] Gleicher | Visual comparison for information visualization[END_REF]. As illustrated in Figure 5.4, Sim2RealViz follows the overview+detail interface scheme [START_REF] Cockburn | A review of overview+ detail, zooming, and focus+ context interfaces[END_REF] with a range from the most global views (left), to the most specific ones (right). To ease the comparison, simulation and real-world data are displayed next to each other within each view, with, if possible, simulation on the left side and real-world on the right side. The objective of the Statistics view (Figure 5.4 x) is to help in quickly identifying the performance of a model and to grasp global behavior with simple visualizations. The Geomap (Figure 5.4 y), is key in providing context on the instance predictions, and for users to grasp what factors of variation may cause sim2real gaps. Finally, the Instance view (Figure 5.4 z), displays how models may perceive sim2real gaps under different scopes. To encode the main information related to the gap we used three colors corresponding to either SIM , REAL , or GT , across the visualizations. We also used color to encode the distance between two sets of coordinates or the intensity of the models' attention towards parts of input images using a continuous turbo [START_REF] Mikhailov | An Improved Rainbow Colormap for Visualization[END_REF] color scale, commonly used by experts, to emphasize the most critical instances, i.e. those with high values.

Main-stream workflow

We now provide a typical workflow of use of Sim2RealViz:

1. Models are pre-loaded and their overall performances on both SIM and REAL are displayed on the top left of Sim2RealViz (Figure 5.4 x).

2. After model selection, and following G1., users can start a fine-grained performance analysis of sim and real models by observing global statistics views such as a UMAP [START_REF] Mcinnes | Umap: Uniform manifold approximation and projection for dimension reduction[END_REF] projection of embeddings in which each dot is an instance, and its color encodes how far it is to its counterpart (sim or real). Followed by a radial bar chart of predicted or ground-truth orientation, and finally, a distribution of positions in which each room is a bar, and their height corresponds to how many predictions there are. In any of those views, users can select a set of instances to be inspected, for example, a cluster in UMAP G2.).

3. Any of those selection updates a geo-map (Figure 5.4 y), i.e. a "geometric" bird's eye view, in which users can inspect the predictions in a finer scale. Users can adapt the geo-map to either color-mode which only displays ground-truth positions with their colors indicating how far sim and real predictions are, or full-mode which displays sim predictions, ground-truth positions, and real predictions. Instances can be selected for further inspection by mouse-hovering them.

4. An instance selection updates the instance view (Figure 5.4 z) and displays heatmaps, which highlights the portions of images on which the model most focuses on, or which it perceives as different (G3.). Such a heatmap is also back-projected over the geo-map to highlight portions of the environment, which most likely carry sim2real gaps (Figure 5.4 {).

The views in Sim2RealViz are multi-coordinated, i.e. any of them, including the geo-map, can be used as an entry point to formulate complex queries such as "what are the instances which perform poorly in simulation, but good in real-world while being in a selected region of the environment?". Concretely, those combinations of selection can be done using sets operations ({union, intersection, and complementary}), which can be selected through interactions with the corresponding views. This is further emphasized by the fact that the performance differences between simulation and real-world are also color-encoded on the geo-map.

Geo-Map and Encoding of Positions

Predictions need to be contextualized within their environment in order to have a grasp on which portions of the environment may be susceptible to convey sim2real gaps. This is emphasized by the fact that not every miss-prediction, despite a similar distance, has the same significance. For example, we should distinguish between predictions within the same room, and predictions in different rooms or unreachable locations. As depicted in Figure 5.5 x, the ground-truth coordinates (x, y) and view-point orientation angle (α), along with their sim and real predictions are represented as pins on a bird's eye top-down view of the environment, with their color encoding the domain they belong to. However, when several instances are displayed over the geo-map, pins of a single instance need to be connected to avoid any confusion. With such an amount of marks displayed per instance, this view can easily suffer from occlusion and overplotting (Figure 5.5 x).

To address the challenge of the triplet of coordinates visualizations, Sim2RealViz provides three levels of aggregation of triplets that users can select at any time. To limit occlusion, while preserving the information displayed, instances can be encoded as glyphs (Figure 5.5 y) positioned at ground-truth locations, and with two wedges to represent SIM and REAL information. The radius of each wedge encodes the distance of its color-encoded prediction to ground-truth, while the centroid of the wedge points towards the position of this prediction. Users can then easily compare sim and real predictions for selected instances. If those wedges overlap, their intersection is color encoded in green, indicating that both simulation and real-world predictions may be similar. Finally, as illustrated in Figure 5.5 z, by default instances are displayed on the geo-map using only ground-truth pins, with their color encoding the distance between simulation and real-world per instance. To ease the readability, the background map of the environment can be clipped to only highlight the surroundings of selected instances.

Heatmaps

To facilitate the inspection of the sim2real gap through image comparisons, Sim2RealViz provides heatmaps superimposed over images, from a selected instance, to draw user attention towards key portions of inputs extracted by the trained model (Figure 5.4 z). Feature-wise visualizations are essential, as visual differences between simulated and real-world images perceived by humans may not correspond to differences in features with a high impact on model decisions. Figure 5.4 z illustrates the result of three approaches to generate those heatmaps, as follows (from top to bottom):

Regression activation mapping -Inspired by grad-CAM [START_REF] Ramprasaath R Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF] and RAM [START_REF] Wang | Diabetic retinopathy detection via deep convolutional networks for discriminative localization and visual explanation[END_REF], we design heatmaps to highlight regions in the input, which have a high impact on model prediction. For each forward-pass of a model, we collect feature maps from the last Convolutional Neural Network (CNN) layer and multiply them by the weights of the last Fully Connected (FC) layer, obtaining an overlay of the size of the feature map, which is then re-scaled to fit the input image and normalized to fit a turbo color scale (Section 5.5.1). The similarity of activation maps between two similar sim and real images suggests a similarity of the two input images from the model's reasoning perspective.

Sim/Real occlusion -Occlusion sensitivity [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF] is a common method to visualize how neural networks in computer vision rely on some portions of their input images. It consists in applying gray patches over an image, forwarding it to a model, and observing its impact on the model's prediction. By sampling a set of patches, we can then overlay the input with this information, blue color indicating that the occluded prediction is closer to the original ground truth, and red otherwise.

In our case, the intuition and solution are slightly different from the standard case. We are interested in areas of the input image, where the model performance is improved when the real-world observation is replaced by simulated information, indicating a strong sim2real gap. We, therefore, occlude input REAL images with RGB or Depth patches from the corresponding simulated image. Thus, a further advantage of this approach is the possibility to discriminate between gaps in RGB or Depth input. The size of the patches is governed by a Heisenberg-like uncertainty trade-off between localization performance and measurement power. After experimenting with patch sizes ranging from 2 × 2 pixels to 128 × 128, we concluded that patches of 40 × 40 pixels, i.e. a total of 6 × 6 patches per image, are the more suitable to analyze images on our computer as we estimated that response time for such an interaction should be less than one second. This is due to the fact that this is displayed on mouse-over, hence multiple instances can quickly be probed by a user, and a longer interaction time dampens the usability and user experience of Sim2RealViz.

Feature map distance -Another approach implemented in Sim2RealViz is to gather the feature map of the last CNN layer during a forward pass on both the simulation and its corresponding real-world image, and then compute a distance between them. The result is a matrix with the size of the feature map which is then overlaid like the activation mapping. After some iterations, we opted for the product of the cosine distance which favors small changes, and L1 which is more inclined to produce small spots. Such a product offers a trade-off between highlighting every change and facing over-plotting while focusing only on one specific spot with the risk of losing relevant changes. To optimize their computation, geo-maps are discretized into squares larger than a pixel, as a trade-off between the accuracy of projections, and the user to the feedback.

Contextualization on the global geo-map

As illustrated in Figure 5.4 { and in Figure 5.6, information from the individual first-person 3D projective input images, including heatmaps, can be projected into the global bird's eye view, and thus overlaid over the geo-map. This is possible thanks to ground truth information, i.e. coordinates, and orientation of the instance, combined with information of the calibrated onboard cameras (simulated and real) themselves such as its field-of-view, position on the robot, resolution, and the range of the depth sensor. To do so, the environment is discretized in 264 × 264 blocks initially filled with zeroes, and images are downsampled to 128×128. Each cell is converted into (x, y) coordinates, and its average value from a heatmap is summed with the closest environment block to (x, y) coordinates. Finally, the values of environment blocks are normalized to fit the turbo color scale and then displayed as an overlay on the geo-map. This process can also be applied to the complete dataset available at once to provide an overview of sim2real gaps of the environment as perceived by a model. Figure 5.6 shows the conversion of heatmaps from the complete real-world dataset to a geo-map overlay using different aggregation strategies. This overlay can be displayed using the button make-heat from the geo-map view (Figure 5.4 y).

Exploration of input configurations

Following the design goal G4., to check the impact of sim2real gaps due to global imaging parameters, Sim2RealViz provides ways to adjust real-world images through filters such as brightness, contrast, temperature, and dynamic range of depth. As illustrated in Figure 5.7, those filters can be generated with sliders on the right of instance view (Figure 5.4 {). Any adjustment on a selected instance updates the corresponding prediction in real-time. Once a set of adjustments is validated by the user, it can be saved, applied to the whole real-world dataset, and treated as a new model in the model gaps overview x for further analysis. The configuration of inputs can also be used on images from simulation, to analyze the performance of the model under specific Domain Randomization configurations, or simulation settings such as, for example, the height of the camera, or its Field of View (FoV). Of course, for the simulation to have an impact on how models perceive real-world images, they need to be retrained outside of Sim2RealViz. To do so, one must first generate a new dataset with the modified settings, in our case 60k images, which can take around half an hour as we also need to enforce diversity of sample images (coordinates and orientation). Then, we train from scratch our model takes around 3 to 4 hours on our single NVIDIA Quadro P4000 GPU. Despite such a delay, adjusting simulation images in Sim2RealViz can be useful to help manually extract parameters of the real-world camera, and hence assist in the configuration of the simulator. Producing images, by configuring the simulator with direct feedback, should reduce the workload usually required to configure simulators and real-world robots.

Case studies

We report on illustrative case studies we conducted to demonstrate how Sim2RealViz can be used to provide insights on how different neural mod- However, when we slightly increase the brightness of each real-world image, we can observe that instances are more evenly distributed over the environment y. els may be influenced by sim2real gaps. During these experiments, Sim2RealViz is loaded with the following methods for sim2real transfer: vanilla (i.e. no transfer, deployment as-is), dataAug (i.e. with Domain Randomization over brightness, contrast, dynamic range, hue), fine-tuning on real-world images, and perlin, a hand-crafted noise on depth images designed to be similar to real-world noise. We use visual data extracted from two different trajectories of the physical Locobot agent in the real environment performed with several months between them and at different times of the day, which provides a diverse range of sim2real gaps and optimizes generalization. Those models and data are available in our GitHub repository at: https://github.com/Theo-Jaunet/sim2realViz .

Insights on sim2real gaps grasped using Sim2RealViz can be leveraged from two different perspectives echoing current sim2real transfer approaches. First, similar to Domain Adaptation, we can provide global modifications of the REAL images (e.g. brightness), which can be placed as filters and used in Sim2RealViz. Second, related to Domain Randomization, by modifying the simulator settings (e.g. adding or removing objects in the environment), and then by training a new model on it. In what follows, we describe several types of sim2real gaps, which have been identified and partially addressed in our experiments.

Unveiling biases in predictions -Once loaded, users can observe how models perform on simulated and real-world data provided by different models trained and transferred with different methods, as shown in Figure 5.4 x. We report that best real-world performances are reached using dataAug, with an average of 84% accuracy, rather than Fine-tuning, with an average accuracy of 80%. This performance is evaluated on traj#1, whereas traj#2 had been used for fine-tuning on real-world data, ensuring generalization over experimental conditions in the environment. In what follows we will focus on the dataAug model, which a user can further analyze by clicking on its corresponding diamond (Figure 5.4 x). This updates every other view to display data extracted from this model. To assess what the worst real-world prediction is, users can use the min filter on the top of Figure 5.4 x. This removes from each view of Sim2RealViz instances whose real-world performances are not among the bottom 15%. In our case, the remaining data displayed corresponds to instances sampled from the right corridor regardless of the model used. We conjecture, that corridors are among the most difficult instances as they are quite large and lack discriminative landmarks. However, in opposition, by using the max filter, we can also observe that the left-side corridor is among the most successful predictions. By hovering those corridor instances with a successful transfer, we can inspect activation heatmaps and observe that model attention is driven towards the limit between a wooden path (unique to the left corridor) and a wall. Thus, the model seems to have learned to discriminate between corridors, which suggests that the confusion between them may be due to other reasons. By switching the encoding on the geo-map to full using the slider on the middle top of Sim2RealViz, the geo-map updates to display SIM , REAL , and GT positions (Figure 5.8 x). With this, we can observe that the vanilla model, incorrectly predicts real-world positions from the half-right of the environment in the half-left. Since those instances are correctly predicted in simulation, this indicates a very strong bias from most of the half-right real-world instances. A similar phenomenon is also observed for the dataAug model with instances on the right corridor creating predictions pointing to the middle of the environment, which is also an unreachable area.

Closing the loop with filters -We verify the hypothesis of regression to the mean, which is often an "easy" short-cut solution for a model in absence of regularities in data, or when regularities are not learned. The following focuses on the vanilla model, as it is the one with the most real-world predictions on the half left of the environment. We perform global adjustments of the imaging parameters of the real-world images as described in Section 5.5.6, in particular setting both RGB and depth input to zero (i.e. uniform black images), leading to the same constant predictions in the middle of the environment, corroborating the hypothesis.

While adjusting the brightness filter, we noticed that making images from the right corridor darker, yielded real-world predictions to be even more to the half left of the environment. In opposition, by making those images 15% brighter, yielded real-world predictions, more accurately, in the half right of the environment leading to a slight improvement of the overall performance of 1.5% (Figure 5.8 y).

Simulator

Edited Simulator Sim2real gaps due to layout changes -Trained models deployed to real environments need to be resilient to dynamic layout changes such as opened/closed doors, the presence of humans or moved furniture and other objects. In Sim2RealViz, this can be investigated using the global heatmap with feat-dist, displayable with the make heat button on the middle top of Sim2RealViz as seen in Figure 5.4. In such geo-map overlay, some areas of the environment noticeably stand out (red in Figure 5.9). By hovering over instances on the geo-map nearby those areas, and browsing their corresponding images (as in Figure 5.4 {), we can observe that those areas are triggered by different factors. For instance, in Figure 5.9 y, the highlighted area corresponds to the presence of a bike in the simulated data, which was not present when the real-world data had been captured. Other areas correspond to changed furniture, and imperfections of the simulation when ren-dering, for instance, a fire-extinguisher (Figure 5.9 x). Such behavior, which can be observed across models, may benefit from specific attention while addressing sim2real transfer.

Editing the simulator -In order to test such a hypothesis, we manually edited the simulator and removed objects corresponding to two red areas using blender, a 3D modeling software. This new simulation is then used to generate 60k new images. Using these data, we trained a new dataAug model and loaded its predictions in Sim2RealViz to evaluate the influence of those changes on realworld performance. Figure 5.9 shows the global feat-dist heatmap on trajectory#1 created with the new model, taking into account the changes. We can see that the areas with the most significant differences are more uniformly distributed over the environment. Since global heatmaps are normalized over the complete dataset, this indicates that, to the model, those areas are now closer in feature space.

The experiments described above have led to a better understanding of the sim2real gap of our baseline agent, and we reported more robust localization performance once these insights were leveraged to modify the simulator or by learning filters for the existing model. We hope that Sim2RealViz will be adopted and facilitate the design and training of trained robotic agents.

Limitations and Perspectives

Evaluation -While the case studies detailed in Section 5.6 showcase how Sim2RealViz can be useful to identify potential sources of sim2real gaps, it would be beneficial to conduct a thorough expert evaluation-similar to the other visual analytics systems introduced in thesis. Such perspective should address both the usability of Sim2RealViz for Deep Learning (DL) experts, and their ability to identify sim2real gaps with our visual analytics system. To this end, we plan to evaluate Sim2RealViz in two ways. First, experts can be asked to identify the most adequate model and instances the most susceptible to convey sim2real gaps as quickly as possible. Time would be a relevant performance measure for this task, along with comparing the instances picked by experts. This should provide a better understanding of how reliable Sim2RealViz can be when used by experts. The second phase of the evaluation will be follow-up interviews with the same experts to collect qualitative insight from this experience. Thirdly, in order to evaluate Sim2RealViz's ability to identify the source of sim2real gaps, we propose to manually select instances, and balance their distribution on their potential presence or not of gaps, and rooms from which they are sampled. Then, these instances will be loaded in Sim2RealViz while hiding their real-world predictions, and ask experts to answer the following questions: 1) "Where do you think the real-world prediction is?", 2) "Do think such an instance may be sensitive to a sim2real gap?", and 3) "If yes, what can it be?". Answers will be collected and analyzed quantitatively in order to assess Sim2RealViz's ability to empower experts to conduct an in-depth analysis of models' decisions.

Comparison of models and robots -Currently, in Sim2RealViz models can only be compared through the parallel plot visualization as illustrated in Figure 5.4 x. However, such a comparison could also be beneficial if it was propagated to other views (e.g. the geo-map or the instance view). That way, we expect experts could to be more confident to evaluate if a particular model has overcome a sim2real gap that the others did not. On the geo-map, such a comparison could also enable experts to analyze if certain models are more adapted to certain areas of the environment, or rather, as illustrated in Section 5.6, follow the same regression to the mean bias. Furthermore, aside from comparing models, it could be useful to adapt Sim2RealViz to account for different robots. Indeed, many sim2real gaps may directly emerge from the robot itself. For example, a different focal lens of the camera, vibrations that blur images, or more basically a different colorimetry. In the future, being able to distinguish from which robot an instance is sampled, may empower experts to more quickly identify what are the differences and hence the potential gaps that may occur.

In addition, the only way to leverage any insight on how a model behaves is to train a new model on, for instance, a new simulation. However, such a process is time-consuming as it requires for a model to be completely trained for hours, and manually added into Sim2RealViz. This is doomed to be repeated if the analysis of the new model is not conclusive. Hence hindering the deployment of our system into any expert's workflow. This may be tackled by including the analysis of models in earlier stages of their training into Sim2RealViz. That way, experts could compare models without having to wait for their training, and hence more quickly assess if any of their updates is effective. While such a solution is limited as experts still need to manually adapt their models (e.g. add or remove layers, add a new loss function) outside of Sim2RealViz, we argue that we cannot anticipate every changes an expert may potentially execute on their model, and hence that at some point they will be forced to go back to their own code.

Future visualizations -Finally, another perspective is to extend the views of Sim2RealViz to include a deeper level of available information on the environment. For instance, the bird's eye geo-map could be combined with a 3D view of the simulation. Such a view can exploit the same first-person to coordinates projection to produce a heatmap on textures (e.g. a red tint on a wall could indicate a presence of a gap). This would result in a more immersive and precise analysis of sim2real gaps, while the bird's view provides a more global view of the environment's heatmap. Additionally, the encoding of positions on the geo-map may suffer from overplotting issues as the number of instances to be displayed increases. Our glyph encoding only partially tackles this problem, occlusion issues can still be encountered as the density of positions increases. Future works could address this issue by relying on aggregation methods to limit the number of instances to be displayed. A representation for such an aggregation could be a new glyph that emphasizes the number of instances it encodes along with the overall discrepancies (e.g. average distance between them) between simulation, ground-truth, and real-world predictions.

Finally, Sim2RealViz heavily relies on real-world images with ground-truth labels, and hence their simulation equivalent. Here, such an alignment for pairs of images is done manually with the help of heuristics and a lidar. Hence, such a process is laborious and may have approximations. As a result, the amount of real-world data available for analysis is limited. Future works could loosen this requirement by empowering users with the ability to manually adjust groundtruth labels to fix any potential approximations. Furthermore, the requirement for the lidar to generate ground-truth labels of real-world images could be avoided. This can be done by giving real-world images (without any label) to the model and collecting their predictions. That way, we could rely on those potentially false predictions to generate to which simulation image they correspond to. By comparing both images, we may be able to assess if the model yielded correct coordinates and orientation angle for the real-world image. Such a comparison can either be done manually or with the help of metrics-(e.g. with the Mean Squared Error (MSE)). While this process does not provide the same sim/real alignment, it has the benefit to be able to process a large amount of data, and sort instances by how likely they are to be influenced by sim2real gaps. As a result, this approach could potentially help experts browse through more discrepancies, and eventually n adjust false predictions for more in-depth analysis which requires an alignment.

Conclusion

We introduced Sim2RealViz, an interactive visual analytics tool designed to perform an in-depth analysis of the emergence of sim2real gaps from neural networks applied to robot ego-localization. Sim2RealViz supports both overview and comparison of the performances of different neural models, which instances can be browsed based on metrics such as performance or distribution. Those metrics can be combined using set operations to formulate more elaborated queries. We also reported scenarios of use of Sim2RealViz to investigate how models are inclined to exploit biases, such as regression to the mean, and are easily disturbed by layout changes, such as moved objects.

Summary of Contributions

This thesis provides contributions to multiple fields, including interactive data visualization, machine learning, and how to build bridges between those two worlds. Those contributions can be summarized as follows:

Assessing reasoning bias in Visual Question Answering (VQA) -Preventing models, especially large ones such as transformers, to learn to blindly yield outputs known by heart instead of learning a proper reasoning process, is key for a future of Artificial Intelligence (AI) that impacts our everyday life. In Chapter 3, we identified, thanks to interviews and participatory design with model builders, their workflow to analyze bias in the attention of their models, and design goals for a visual analytics system to assist them. We designed VisQA, an interactive visual analytics system, with a new set of visualizations to analyze the attention mechanism of bi-modal transformer models. This visualization is the first to display attention maps in a layout following the studied model's structure enabling a fast identification, and selection of these maps while mitigating the learning curve required by such a tool. VisQA is available as an online prototype (https:// visqa.liris.cnrs.fr), and is open-source (https://github.com/Theo-Jaunet/VisQA). This system is the first to address the bi-modality attention of VQA transformer models, and is the first to provide interactions to alter models' attention in realtime. As a result, experts can explore models' predictions and draw hypotheses on the role of a selected set of attention maps. Through an evaluation protocol we designed, we showed first evidence that human users can obtain indications on the reasoning behavior of a neural network using our system. During this evaluation, domain experts, builders of Deep Neural Networks (DNN) models, were able to estimate whether the model correctly predicted an answer and whether it exploited biases. These insights lead to improvements of the state-of-the-art LXMERT model.

Understanding the Memory of Deep Reinforcement Learning (DRL) -

The memory of robots trained with DRL is at the root of their decisions, thus analyzing it, and being able to grasp how it is used by a model may provide a glimpse into the black-box that is their decision process. In Chapter 4, we addressed the under-explored challenge of visualizing such a memory and went further by contextualizing it within the environment, previous decisions, and other metrics. After discussions with domain experts about their workflow of training DRL agents on navigation tasks, we designed a new interactive visual analytics system to help them study the memory of their trained models. This prototype is available online (https://sical.github.io/drlviz), and is open-source (https: //github.com/sical/drlviz). In this system, the contextualization of the memory is carried by a surrounding set of multiple-coordinated views which can be used to filter and query any subset of memory. In particular, we provide a timeline of events dissecting a robot's action which can be used as an overlay of the memory through an interaction we designed. This tool has been evaluated by domain experts during qualitative interviews during which they noticed, among other things, how some memory elements may function as "flags" encoding whether or not the model has collected an item. It can be observed that in some cases the agent may miss an item and lose vision of it, inducing those memory elements to behave as they would if the agent had successfully gathered such an item. This led to the design of memory reduction, a new interaction directly tied to the model. Which such an interaction, a user can select a subset of the memory, and observe how the model would behave without it. This was combined with the intuition that the size of memories is manually picked by the model's builder, hence resulting in an often too large memory. Using this, we discovered that in some cases the robot can preserve a similar performance with only a key subset of less than half of the memory elements. We extended such interaction and addressed it to a non-expert audience in an online explorable (https://theo-jaunet.github. io/MemoryReduction), in which they were invited to explore different strategies to reduce a robot's memory, along with an explanation of how it works.

Identifying the Root of Gaps Between Simulation and Real-world -Despite great progress in DRL, robots are yet to be seen in our everyday life. This can be explained by the fact that in many cases robots trained in simulation tend to have unexpected behavior when deployed into real-world conditions. However, we need to rely on simulation to train them as they require millions, and sometimes billions, of interactions with their environment to learn, which is infeasible in real-time. In Chapter 5, we introduced non-expert users to this challenge, by designing Théo Guesser, and online explorable (https://theo-jaunet.github.io/ theo-guesser/) in which users are invited to explore how a model trained in simulation might be sensible to noises on input images, and one may tackle such an issue. In addition, we addressed the challenge of transferring the knowledge of a DNN model trained for ego-pose localization on simulation to the realworld. To do so, we designed and trained a model to yield its position within its environment from an image input. Along with such a model, we designed an interactive visual analytics system to visually compare how the model may differently perceive a simulation and a real-world environment. This open-source system (https://github.com/Theo-Jaunet/sim2realViz) provides an overview in multiple-coordinated views on how the model performs in simulation and realworld, which can be used to provide a bird's eye view of the environment, and only display data-points that may be altered by real-world conditions (e.g. a moving object). In such a system, we provide visualization methods, such as the ability to convert gradient heatmaps such as grad-cam (as introduced in Section 2.2.2) from the robot's first-person point of view, into a 2D bird's eye view of the environment. By doing so, one can quickly analyze which are the most troublesome areas of the environment and hence act in consequence (e.g. edit the simulator). Additionally, in this chapter we managed for the simulation and realworld images to be aligned-i.e. have a simulation and real-world image sampled from the same coordinates and camera orientation. Inspired by occlusion [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF], we designed a new technique named domain occlusion which applies patches over an input image to investigate if the output is further or closer than the ground-truth. However, instead of applying gray patches, we switch from one domain to another, e.g. replace a region of an image from simulation with the region from real-world. This allows to assess how the model may exploit biases such as a regression to the mean, and whether it is sensitive to moving objects such as bikes, or wrong texture mapping in simulation. Thanks to the system's ability to directly manipulate the simulator settings, and apply filters (e.g. increase brightness) to real-world images, we explored how those biases and layout-related issues may be tackled.

Perspectives for Future Works

The contributions of this thesis, along with the recent advances from literature, spurred new opportunities, and a wide range of perspectives for future work. In this section, we distilled some possible directions for the challenges we tackled in this thesis, along with the rise of more global research directions for the community.

Invade and Conquer Model Builders' Workflow

Despite the plethora of visual analytics systems, and building blocks for DNNs' interpretability designed by the community, interpretability is yet to be a common practice in model builders' workflow. Nonetheless, during evaluations, it has been shown that with the help of those systems, model builders were able to identify issues among their models, and eventually leverage such information to improve them. We can therefore raise the question: Why are such tools not included in the design process of models yet? A light can be shed onto this question using the following lenses:

The trenches of training and optimization-A vast majority of visual analytics systems for interpretability, including those introduced in this manuscript, are designed and used to analyze trained DNNs, which is often done after the hyperparameter optimization phase. As a result, most of this process, which is the most time-consuming step for model builders, is excluded from these visual analytics systems, hence forcing them to rely on alternative tools and methodology to study their models in their early stages. For example, a common approach is to either directly monitor a model's loss evolution on a terminal console, or rather to use TensorBoard [START_REF] Wongsuphasawat | Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow[END_REF]207]. TensorBoard is one of the most popular toolkits, which, among other features, enables users to plot metrics as line charts. We argue that easing the burden of optimizing a model under construction and including this step of model-building into the design of our visual analytics systems could encourage model builders to share such a time-span, usually allocated to optimization, with interpretability. Thus reinforcing the message that a model's accuracy on a dataset does not describe its ability to exploit, or convey biases [START_REF] Doshi | Towards A Rigorous Science of Interpretable Machine Learning[END_REF]. Currently, such a step may be incorporated manually in visual analytics systems by using snapshots of a model. However, in addition to being too laborious to be incorporated into builders' workflow, such a solution, and our current visual analytics systems also lack the capability to compare models. Such a comparison is, however, core to the optimization phase, and perhaps could also be crucial for its interpretability. For instance, comparing activations of two different models using the same input may reveal which one is relying on the most interpretable information to convey an output.

Vanquish the learning curve-By itself, understanding the decision process arising from a single neuron is straightforward. It is their overpowering number within a DNN, and the time it would require for a Human to analyze them all, that makes a model a "black-box". Similarly, our visual analytics systems, through their large number of multi-coordinated visualizations, interactions available, and data displayed tend to be overwhelming for users. This results in a steep learning curve, which in many cases discourages users and may even prevent them to understand the core functionalities of a visual analytics system without any assistance. This is emphasized by the fact that many visual analytics systems are instance-based, meaning that they are dedicated to the analyzing of a single data-point and output at a time. While this provides a fine-grained analysis, this also raises the challenging task of selecting which data-point may be susceptible to portrait a model's faults, and hence worth analyzing. As a result, those visual analytics systems tend to convey a feeling of "Where to begin?" even to experimented users. Currently, in the community, such an issue is tackled through the use of an interactive representation block (e.g. Chapter 4), in which one may click on any dot, updating other views with the corresponding input. Another approach is to rely on a metric describing the model performance to re-order inputs, and draw users' attention to the worst ones (e.g. Chapter 3). However, this approach depends on the metric's ability to describe bias exploitation from a model, and the users' comprehension of such a metric. Finally, some works in the shape of prototypes, choose to only offer the possibility to switch between a handful of inputs, manually selected for their ability to portrait the studied model's biases, hence avoiding for their users the task of finding relevant inputs to analyze. With the ever-growing size of state-of-the-art models and the amount of data they digest, future studies should focus on understanding how experts use visual analytics systems, and provide different levels of information, in order to ease their experience and screen-space real-estate taken by visualizations, and hence limit the amount of time required to efficiently analyze a single data-point. Along with such a study, understanding how to more efficiently identify inputs that should interest the user, e.g. with data-mining methods, and more global visualization of models' behavior is a core research direction for better interpretability of DNNs.

Blitz deployment of systems-Designing a visual analytics system to interpret a model's decision requires time, and lasting collaboration between model builders, and visual analytics designers. This is emphasized by the fact that such systems often require a lot of programming and engineering, shaped as the collection of data from models (e.g. activations), the construction of systems, and the infrastructure to bridge models often on python, and in-browser visualizations on JavaScript, along with knowledge from both worlds. As a result, in most cases, the emergence of tools to interpret a new model comes with a delay. Thus many published models are not evaluated with respect to their interpretability, and visual analytic systems fail to cope with the ever-increasing fast-paced creation of new models. This is because those systems are designed for one specific implementation of a model within a class of models (e.g. the LXMERT among transformers in Chapter 3), a specific dataset (e.g. GQA), a unique task (e.g. VQA).

Therefore, the deployment of a visual analytics system to another configuration requires for such an engineering time needs to be repeated, hence hindering model builders from easily interpreting variations of a model. However, as experimented in this manuscript despite different models, datasets, and tasks addressed, our visual analytics systems and those from the community share common features. Each of those systems needs to obtain information from models, and for example, share common visualizations such as the visualization of attention and memory as heatmaps, the presence of interactive representation view issued from a dimensionality reduction algorithm, or even the softmax distribution of top classes for an output. Hence, in the future, we should inquire how can we rest on the shoulders of giants, and ease the burden of future engineering requirements. This can be done by adapting a system to another related model, by studying and designing more modular and generic systems. Along with that comes the need for future works to tackle the unexplored challenge of adapting those visual interfaces to the ever-increasing size of models and dataset, dampening their ability to have a reasonable response time for their interactions, and increasing overall performances required to even load those systems in a web browser.

Mitigating Human Biases

When analyzing and interpreting models' decision processes one should also be on alert for one's own biases. For instance, as humans, we tend to expect models to have explanations for each of their decisions, because we mistakenly believe that each of our decisions has rational explanations, instead of anchored priors [START_REF] Kahneman | Thinking, Fast and Slow[END_REF]. Such biases may blind the shortcomings of our interpretations and should be taken into account when analyzing a model's decision. Future works could address this with the following research directions: Evaluate interpretations -When interpreting a decision, we, as humans, are looking for patterns and clues that help us seek a reasoning process similar to our own, thanks to DNN's data (e.g. activations). This is known as confirmation bias, and because of this, we may interpret an explanation as in favor of our intuition concerning a decision, and consider it as an artifact any explanation that points toward something different. As an example, a heatmap around a cat's whiskers from an input image may be interpreted as the model correctly assimilating that indeed cats have whiskers, when in reality, it may be the model mistaking the background in-between whiskers as a discriminatory feature for cats. Furthermore, deciding when one has enough information to interpret a model's decision, and not missing any piece of information that could shatter one's conclusion, is a challenge even for model builders. This may result in a trade-off between jumping to conclusions in the interest of time and eventually missing crucial information, and being too meticulous, hence taking too much time on a single input and eventually skipping inputs that could yield valuable insight.

The study of human biases, and their effect on decision-making through visualization [START_REF] Wall | Left, Right, and Gender: Exploring Interaction Traces to Mitigate Human Biases[END_REF], and visual analytics [START_REF] Wall | Warning, Bias May Occur: A Proposed Approach to Detecting Cognitive Bias in Interactive Visual Analytics[END_REF], is an on-growing area of research that provided metrics describing biases to a user [START_REF] Wall | Warning, Bias May Occur: A Proposed Approach to Detecting Cognitive Bias in Interactive Visual Analytics[END_REF], and a design space to mitigate cognitive biases [START_REF] Wall | Toward a Design Space for Mitigating Cognitive Bias in Vis[END_REF] which may be hidden when analyzing a visualization. In addition to those contributions, we should also consider studying those biases with respect to DNN interpretability. Hence a rising need for stronger evaluation protocols of how users may interpret decisions thanks to visual analytics systems. Currently, a vast majority of research reports introducing new visual analytics systems for DNN interpretability provide illustrative examples and case studies. While this may showcase how a new system can be used and what kind of explanation can be found with it, we argue that a more thorough evaluation of how users may adapt to this new system, and evaluation of the interpretations they can grasp may be beneficial. This trend may be explained by the fact that interpreting a model's decision is a challenging task, and thus such an operation is often addressed in lengthy qualitative interviews, with model builders experimented with the studied model. Such conditions limit the pool of available experts able to participate in an experiment, resulting in only a handful of experts. In a few cases, these experts contributed to the design of the visual analytics system being evaluated, which hinders the evaluation of a new user's adaptation to the system. Despite the emergence of strategies and protocols to evaluate an interpretation [START_REF] Doshi | Towards A Rigorous Science of Interpretable Machine Learning[END_REF][START_REF] Schlegel | Towards a rigorous evaluation of XAI methods on time series[END_REF], the quantification of what is a "good" interpretation, and when a model is "good enough" remains an open challenge. Thus, their applications to visual analytics remain elusive. As an example, for VisQA (Chapter 3), such an application relies on a more practical approach consisting in hiding the model's output and asking users to estimate, based on their analysis, what it should be. That way, we quantitatively compare the results between users, and between an answer and the model's prediction.

Include end-users into the loop -When building a model, we tend to follow our intuition and biases based on our experience. For example, with statements such as "Traditionally this amount of parameters are enough", or "Usually such a loss yields great results". However, it has been shown that such fine-grained changes may have a direct impact on how a model behaves [START_REF] Ritter | Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study[END_REF]. A solution would be to formulate an exhaustive exploration of every combination of hyper-parameters per model, which is not feasible in a timely manner and would sky-rocket the energy consumption of model training. This illustrates that breaking away from Human biases is nearly impossible, especially if a model builder analyzing a model has no knowledge of what may constitute an inequality of opportunity [START_REF] Hardt | Equality of Opportunity in Supervised Learning[END_REF]. Thus, a key research direction is to empower end-users with the ability to experiment not only how a model may treat one's data, but also how other cases are handled.

However, this leads to the seemingly insurmountable challenge of adapting visual analytics systems to accommodate non-expert users to prevent them from being overwhelmed. Or rather build on existing non-expert works for interpretability, which are often relying on simplified models, and toy datasets. Those works then need an adaptation to fit the desired model and dataset. Such an adaptation may, however, bring back the same feeling of being overwhelmed, despite having a limited exploration space and hence a chance to miss potential explanations. Moreover, even with the consideration that end-users are able to analyze the decision of models, how they may leverage any insight to improve them, remains an open problem.

Finally Finding those Keys!

Throughout this manuscript, we designed and analyzed DNN models, each addressing a step required for a robot to answer a question such as: "Where are my keys?". The next logical step is to combine these models into a single stack to finally have a robot able to understand what we are looking for from a natural language question, and through its environment (e.g. an apartment) finding and communicating its location. To do so, the VQA model, as introduced in Chapter 3, should both communicate to the model dedicated to the robot's navigation what it should look for, and when it should stop (e.g. when the keys are visible). Technically, this would require for the navigation model to be retrained to support a slightly different task, in a more realistic simulator. Instead of reaching several items in an order, as instructed by a reward function, the model must focus on exploring efficiently its environment (e.g. avoid coming back into previously scanned rooms), and thus yield the robot's next actions until the VQA model detects the presence of "keys". This approach could be improved as it would result in a robot aimlessly roaming its environment, instead of being guided by intuitive biases. For instance, if the navigation model knew that it is looking for keys, it could exploit prior knowledge such as that the keys are often left on a desk. Hence a better solution would be to adapt the VQA model to share with the navigation model what item it must detect. Such information may take the form of an embedding of the question or rather be issued from a modification of the VQA model with an auxiliary branch trained to output the name of the object from a question (e.g. "keys") which can then be used as input to the navigation model (using a representation of this class (e.g. one-hot vector, embedding, or a word2vec representation). That way, such a model, may learn the intuition that the keys are often on the same desk, directly head for it without exploring the complete environment. And once the VQA model detects those keys, it can provide the ego-pose localization model with the image from the robot's perspective to yield the position of the keys. In that sense, one could consider for the navigation model, and the ego-pose localization one the share the same Convolutional layers and weights, since both of their goals require an inner representation of the environment's layout. This would have the benefit of easing endeavors to interpret the decisions of those models as they would have fewer parameters to analyze. This has also the benefit of preserving task-specialized visualizations on their respected fully-connected layers.

Similarly, the task of using a robot to answer VQA questions, known as Embodied Question Answering (EQA) [START_REF] Das | Embodied Question Answering[END_REF], with a hierarchical approach gained popularity [START_REF] Gordon | IQA: Visual Question Answering in Interactive Environments[END_REF][START_REF] Das | Neural Modular Control for Embodied Question Answering[END_REF]. This approach combines low-level controllers executing actions (similar to our navigation model), along with higher-level modules either designed to process the natural language questions, or designed as a planner that yields a sequence of action to execute. Alternatively, EQA have been tackled by a single model. For instance, in target-driven navigation [START_REF] Zhu | Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning[END_REF], such a model is asked to reach a destination by using as input the image corresponding to the point of view of the robot, along is a target image corresponding to the view from the robot's perspective of its destination. More recently, Suglia et al. [START_REF] Suglia | Embodied bert: A transformer model for embodied, language-guided visual task completion[END_REF] relied on a single transformer model using jointly the input image of the robot's point of view, and is asked to yield the next action the robot should do. While there is no consensus in the community on whether a single model, or a multiple-stacked one, may have the best performances, we argue that relying on multiple models may be more interpretable. One argument is that each model may be interpreted and evaluated with respect to its own goal-e.g. the ability to identify the presence of keys in an image. Therefore, model builders may be able to act upon the observation of bias exploitation on a specific task (e.g. re-train a model with new loss), without dampening the performances of other models.

Nonetheless, in contrast to the works introduced in this manuscript (i.e. the study individual models), in large industries, the deployment of multiple stacked models is frequent e.g. Google home [START_REF] Li | Acoustic Modeling for Google Home[END_REF], or Amazon's Alexa [210], and by selfdriving cars by Waymo [START_REF] Bansal | ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst[END_REF][START_REF] Sun | Scalability in Perception for Autonomous Driving: Waymo Open Dataset[END_REF] or Tesla [START_REF] Karpathy | CVPR'21 WAD] Keynote[END_REF][START_REF][END_REF]. Those solutions, and the future works for EQA stacked models, carry the yet-to-tackle challenge of designing visual analytics systems able to help domain experts explore the overwhelming amount of data those collection models may represent, along with how they depend on each other. For example, raising questions such as: "Is a mistake occurring because the last model needs to be re-trained, or rather because it received faulty inputs?". This is emphasized by the fact that in many cases models within a stack are trained individually, and hence the model builder may not have access to the upstream, i.e. to inner workings of models it depends on, nor to the downstream ones affected by it. Thus, a challenge for designing more interpretable models is not only to mitigate their bias exploitation, but also to understand how their decisions may induce errors in downstream models of a stack. As an example, the LXMERT model introduced in Chapter 3, first receives inputs from a RCNN model, whose noisy predictions have a direct impact on the downstream transformer model's ability to learn a proper reasoning. This influence may not be straightforward to grasp when analyzing the RCNN by itself. Such findings pave the road for new research opportunities, especially for the interpretability and understanding of biases that may be buried within decisions and the design of models in a stack. Au cours des dernières années, l'IA et l'apprentissage automatique ont évolué, passant de domaines de recherche reclus dans des laboratoires éloignés du public à des technologies déployées à l'échelle industrielle ayant un impact considérable sur notre vie quotidienne. Étant donné que ces technologies sont également utilisées pour résoudre des problèmes critiques tels que la finance et la conduite de voitures autonomes, dans lesquels leurs décisions peuvent mettre des personnes en danger, cette tendance a commencé à susciter des inquiétudes légitimes. Puisqu'une grande partie de la complexité sous-jacente du processus de décision est apprise à partir de quantités massives de données, la manière dont ces modèles prennent des décisions reste inconnue tant pour leurs créateurs que pour les personnes impactés par de telles décisions. Cela a conduit au nouveau domaine de l'eXplainable AI (XAI) et au problème de l'analyse du comportement des modèles entraînés, pour mettre en lumière leurs capacité raisonnement et les biais auxquels ils sont soumis. Dans cette thèse, nous avons contribué à ce domaine émergent avec la conception de systèmes d'analyse visuelle destinés à l'étude et à l'amélioration de l'interprétabilité des réseaux de neurones profonds. Notre objectif était de permettre aux experts de disposer d'outils les aidant à mieux interpréter les décisions de leurs modèles et éventuellement des les améliorer. Nous avons également proposé des applications explorables conçues pour présenter des méthodes d'apprentissage profond à un public non-expert. Durant cette thèse, nous nous sommes concentrés sur le défi sous-exploré de l'interprétation et de l'amélioration des modèles pour différentes applications de la robotique, où des décisions importantes doivent être prises à partir de données d'entrées de haut niveau et de haute dimension telles que des images. Tous les outils conçus au cours de cette thèse ont été publiés en tant que projets open-source, et lorsque cela était possible, nos visualisations ont été mises à disposition en ligne en tant que prototypes. Ces outils sont hautement interactifs et peuvent contribuer à encourager la poursuite de l'interprétation de décisions de modèles par la communauté de recherche

35 3 visual question answering 37 3 . 1 2 . 1 3 . 1

 312131 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.Transformers and Attention . . . . . . . . . . . . . . . . . . . 42 3.2.2 Vision-Language (Vision-Language (VL))-Transformers . . . 44 3.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.Interpretability of VQA . . . . . . . . . . . . . . . . . . . . . . 46 3.3.2 Bias Reduction in VQA . . . . . . . . . . . . . . . . . . . . . . 46 3.4 Motivating Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.5 Design Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Figure 2 . 2

 22 Figure 2.2 Illustration of the convolution operation within the first layer of a Convolutional Neural Network (CNN) model. Given an image x i , the model applies a filter W 0 n of learnable weights over the complete input, which yields the values that the neuron N 0 n will convey to an activation function, and the next layer. . . . . . . . . . . . . . . . . . . . . . . . . 13

Figure 2 .

 2 Figure 2.3On the left x, two examples of an image containing a digit which have been correctly evaluated as "two" by a Deep Neural Networks (DNN). On the right y, two other images which according to the MNIST dataset also contain handwritten "two". However, the same DNN fails and labels those images as "six". Such mistakes may be more acceptable to Humans, as those ambiguous images may even induce us to estimate that they contain sixes. . . . . . . . . . . . . . . 14

3

 3 Figure 2.3On the left x, two examples of an image containing a digit which have been correctly evaluated as "two" by a Deep Neural Networks (DNN). On the right y, two other images which according to the MNIST dataset also contain handwritten "two". However, the same DNN fails and labels those images as "six". Such mistakes may be more acceptable to Humans, as those ambiguous images may even induce us to estimate that they contain sixes. . . . . . . . . . . . . . . 14

Figure 2 . 5

 25 Figure 2.5Comparison of activations from a simple CNN model (Lenet5[START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF]) trained to identify the hand-written digit in its input image. We observe that for the input image of a "2" x, the activations (the whiter the higher) of the first layer are sensitive to the overall image, and shape of the digit y, whereas activations from the last layer are more abstract shapes harder to analyze z. In this figure, images were resized for the sake of readability. Initially, both the image input and first layer activations were 28 × 28, while layer second layer activations were 5 × 5 pixels. . . . . . . . . . . . . . . . . . . 18

Figure 3 . 1

 31 Figure 3.1This is chapter is dedicated to providing to robots the ability to answer natural-language questions about images. For example here, when asked the mundane question "Where are my keys?" x, the robot needs to understand what we are looking for, and then search for it within a given image to provide an answer e.g. here "On the desk!". . . . . . . . .

Figure 3 . 7 57 Figure 3 . 8

 375738 Figure 3.7 Hovering the mouse over a cell of the attention maps x filters the corresponding object bounding box in the input image y. While clicking on this cell filters attention heads in instance-view to display those within which the selected cell is highly activated. . . . . . . . . . . . . . . . . . . . . . 57 Figure 3.8When asked "Is the person wearing shorts?", the oracle transfer model successfully answers "yes". It can be observed in its first Language-to-Vision attention maps, that the word "shorts" (column) is strongly associated with the object "shorts" (row). The same phenomenon is also observed for the word "person', strongly associated with objects labeled as "woman" among others. . . . . . . . . . . . . . . . . . . . 60

Figure 4 . 1

 41 Figure 4.1 This chapter is dedicated to step y of finding my keys, i.e. the ability for robots to explore unknown environments. To do so, such a robot relies on images sampled from an onboard camera (most right) to decide what should be its next direction. . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 2

 42 Figure 4.2Our navigation problem consists in solving a visual task (e.g. fetch, interact, or recognize items) while avoiding obstacles in an environment. Deep Reinforcement Learning can be used to solve this problem by using an image as input x at time t. Features are then extracted from this image y, and combined with the previous memory vector t -1 z. Using this memory vector, the agent decides to move forward or turn left, for instance {. . . . . . . . . . .

Figure 4 . 4

 44 Figure 4.4DRLViz displays a trained agent memory, which is a large temporal vector, as a horizontal heat-map . Analysts can browse this memory following its temporal construction; filter according to movements of the agent and derived metrics we calculated (e.g. when an item is in the field of view ); and select the memory to filter elements and compare them . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 5

 45 Figure 4.5DRLViz allows to compare selected time intervals x. For instance to compare when agents face dead-ends y and when they face health-packs z. One can observe that more elements are active while the agent is facing Health Packs than while facing a dead-end. Perhaps those elements are encoding information concerning Health Packs. When facing a dead-end, both the orientation variation and decision ambiguity are high which can be interpreted as the agent hesitating on which action to choose. . . . . . . . . . . . . 84

Figure 4 . 6

 46 Figure 4.6Summary of the insights gained by the experts. Expert #1 noticed two intervals during which the agent only turned right, by using both trajectory x and stacked area chart of actions y views. Once he replayed those sequences, he stated that the agent came twice in the same dead-end z. Expert #3 observed a hidden state dimension which is blue when the agent sees the red armor before the green armor, and then remained orange until when he saw the green armor {. Expert #2 probed a dimension that is active as the agent first saw the Health Pack, and remained active until it gathered it. Expert #1 also identified two hidden state elements that change as the agent gathered the health pack and then kept their values until the end of the episode }.Using saliency maps ~, Expert #2 observed that the agent ignored the soul-sphere until it gathered the first three items . Finally, Expert #3 identified clusters in the t-SNE projection which corresponds to the agent's objectives e.g. gathering the green armor . . . . . . . . . . . . . . . . . . 85

Figure 4 . 7 90 Figure 4 . 8

 479048 Figure 4.7 Overview of MemRed, an online explorable in which users are invited to try different strategies to reduce the memory of a DRL agent and observe how it affects its behavior. This explorable earned the distinction of "best paper" at the VISxAI workshop 2019. . . . . . . . . . . . . . . . . . . . . . 90

Figure 5 . 1

 51 Figure 5.1When answering questions such as "Where are my keys?", robots need to be able to communicate their position. To do so, we trained models to yield coordinates and orientation angle from any given image z. Such a task is also used as a proxy to evaluate gaps between simulation and reality. . 96

Figure 5 . 2

 52 Figure 5.2In the studied robot ego-localization task, an RGB-D image x, is given to a trained model y, which uses it to regress the location (x, y), and orientation angle (α) in the environment from which this image was taken from z. As illustrated above, images taken from the same coordinates in simulation and real-world x may lead to different predictions due to differences, such as here, among others, the additional presence of a bike in the scene. We are interested in reducing the gap between the SIM and REAL predictions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Figure 5 . 5

 55 Figure 5.5To tackle over-plotting in geo-map while preserving the insights users can grasp with this view, Sim2RealViz provides three way to encode SIM / REAL predictions along with their ground-truth. From x which displays a pin per domain and link them to their ground truth, to y a glyph in which portion size encodes the distance to their color encoded domain, while their position encodes their direction. Or z, only display ground-truth pins with their color encoding the distance between simulation and real-world. 106

Figure 5 . 6 109 Figure 5 . 7 111 Figure 5 . 9 113 Chapter 6 :

 5610957111591136 Figure 5.6 Conversion from pixels on a first-person point of view image to coordinates on a bird's eye geo-map (left) using inverse projection given a calibrated camera. Such a process, used in Sim2RealViz to display global heatmaps (right) on the geo-map, relies on ground-truth, image, and camera information. To optimize their computation, geomaps are discretized into squares larger than a pixel, as a trade-off between the accuracy of projections, and the user to the feedback. . . . . . . . . . . . . . . . . . . . . . . . . . 109 Figure 5.7 By clicking on the adjust button (on the top-right of Figure 5.4), users can display sliders on the right of instance view Figure 5.4 {) that can be used to fine-tuning realworld images with filters and observe how it affect models' prediction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Figure 5.8By using the full encoding, we can observe that most realworld predictions are located in the half left of the environment x. Hence, instances sampled from the half-right of the environment provide the worst predictions. However, when we slightly increase the brightness of each real-world image, we can observe that instances are more evenly distributed over the environment y. . . . . . . . . . . . . . . . 111 Figure5.9With global heatmaps of feature-distance, we can observe (in red) areas of the environment that may be affected by a sim2real gap. Those areas correspond to changes in objects present in the simulation, for instance as illustrated here, a fire-extinguisher. By removing such objects in simulation and retraining a model on them, we can observe that they disappeared from most highlighted areas. . . . . . . . . . . 113Chapter 6: conclusion and future directions

Chapter 5 : 1 . 1 5 1. 2 6 1. 3

 5115263 from simulation to reality Chapter 6: conclusion and future directions xxiii Robotics Task: "Where are my keys" . . . . . . . . . . . . . . . . . Visual Analytics for Model Interpretability . . . . . . . . . . . . . Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Figure 1 . 1 -

 11 Figure 1.1 -Examples of failures of DL models:x Pulse[START_REF] Menon | Pulse: Self-supervised photo upsampling via latent space exploration of generative models[END_REF], with an input image of the former US president Barack Obama, the model tasked to reproduce it, outputs the image of a Caucasian man. y the CLIP model[START_REF] Radford | Learning Transferable Visual Models From Natural Language Supervision[END_REF], can be induced to output wrong classification by simply taping tags to objects, e.g. as illustrated here, the apple is predicted as "library"[START_REF] Goh | Multimodal Neurons in Artificial Neural Networks[END_REF]. z, traffic signs detection models can interpret stop signs as speed limit ones only with only a bit of duct tape on it[START_REF] Eykholt | Robust Physical-World Attacks on Deep Learning Visual Classification[END_REF].

Figure 1 .

 1 Figure1.2 -To be able to answer mundane questions such as "where are my keys", robots are required to master three reasoning skills: first x, the ability to understand natural language questions to grasp what is asked, and analyze its vision to answer. Then, robots need the ability to navigate in an environment to look for those keys as fast as possible, and then, the ability to self-localize to both avoid going to already searched rooms, and, when found, to communicate the position of the keys.

Figure 2 . 1 -

 21 Figure 2.1 -Illustration of the structure of a Deep Neuron Network. With the input image X i , the neurons of the model, arranged in inter-connected layers, yield the output Y i "dog". This output is the outcome of a multitude of neuron computations. As depicted on the right of this figure, each neuron (e.g. here N 2 1 ) relies on the result of neurons from the previous layer. A more formal representation of this computation is depicted in Equation 2.1.

Figure 2 . 2 -

 22 Figure 2.2 -Illustration of the convolution operation within the first layer of a Convolutional Neural Network (CNN) model. Given an image x i , the model applies a filter W 0 n of learnable weights over the complete input, which yields the values that the neuron N 0 n will convey to an activation function, and the next layer.

1 2 Figure 2 . 3 -

 1223 Figure 2.3 -On the left x, two examples of an image containing a digit which have been correctly evaluated as "two" by a DNN.On the right y, two other images which according to the MNIST dataset also contain hand-written "two". However, the same DNN fails and labels those images as "six". Such mistakes may be more acceptable to Humans, as those ambiguous images may even induce us to estimate that they contain sixes.

Figure 2 . 4 -

 24 Figure 2.4 -Building blocks for Deep Learning interpretability are designed to provide an understanding of a model's decision or behavior. Here illustrated using an MLP, they can be divided in four categories. For a given input, a model produces intermediate results (Activations), to reach an output which can be used either by derivation to produce Gradients, or with output to produce Representation. Finally inputs and outputs can be combined to yield Modelagnostic interpretations.

Figure 2 . 5 -

 25 Figure 2.5 -Comparison of activations from a simple CNN model (Lenet5[START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF]) trained to identify the hand-written digit in its input image. We observe that for the input image of a "2" x, the activations (the whiter the higher) of the first layer are sensitive to the overall image, and shape of the digit y, whereas activations from the last layer are more abstract shapes harder to analyze z. In this figure, images were resized for the sake of readability. Initially, both the image input and first layer activations were 28 × 28, while layer second layer activations were 5 × 5 pixels.

Figure 2 . 6 -

 26 Figure 2.6 -Left x, the Top-k image with the highest activations per neuron. Each row is a neuron, and each column from left to right corresponds the top-10 images.We can, for example, observe that the neuron corresponding to the first row might be sensitive to circular patterns such as dog eyes and snouts. Credits for this sub-figure go to Springenberg et al.[START_REF] Tobias Springenberg | Striving for Simplicity: The All Convolutional Net[END_REF]. Right y, examples of overlap between a manually annotated segmentation dataset, and a model's activation from 4 neurons. We can observe that the neurons corresponding to the two rows on the left are sensitive to houses. Hence indicating that some neurons may function as "object detectors". Credits for this sub-figure are due to Bau et al.[START_REF] Bau | Network Dissection: Quantifying Interpretability of Deep Visual Representations[END_REF].

Figure 2 . 7 -

 27 Figure 2.7 -Visualization of activations of an image classification CNN using DeConv[START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF]. This figure represents a manually selected sub-set of neurons which are displayed in a 3 × 3 grid of their most active images. We can observe that the first layer x contains patterns such lines and color gradients, the second layer y seems to seek for textures or patterns, while the layer 5 z seems to responsive to more complex image feature such as faces. In {, we combined with the display of the top-k images one make sens of abstract DeConv activations, e.g. the grass in the background. Activations in this figure were sampled from[START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF].

. 7 ,

 7 Activations at step 224

Figure 2 . 8 -

 28 Figure 2.8 -Left x visualization of a manually selected activation of a hidden state of a recurrent model. Such activation is displayed over its input text ranging from blue (negatives values) to red (positive values).It can be observed that this activation seems correlated with the level of text indentation (credits to[START_REF] Karpathy | Visualizing and Understanding Recurrent Networks[END_REF]). Right y, visualization of a complete hidden state as a grid. It can be observed that only a handful of activation are high at the same time, and thus that they may be used as "functions" representing different elements of the inputs. Sampled from[START_REF] Carter | Experiments in Handwriting with a Neural Network[END_REF].

Figure 2 . 9 -

 29 Figure 2.9 -Examples of visualization with gradients-based building blocks. From left to right, x with an input image, y guided back-propagation, class[START_REF] Tobias Springenberg | Striving for Simplicity: The All Convolutional Net[END_REF], z optimization of input[START_REF] Mehta | Active domain randomization[END_REF] targeting the class "dog", and { Grad-Cam[START_REF] Ramprasaath R Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF] highlight pixel assimilated to the class "dog".

1 2 Figure 2 . 10 -

 12210 Figure 2.10 -Examples of visualization with the representation building blocks.From left to right, x t-sne representation of embedings on the MNSIT dataset, each dot is an input, and its color represents it class. y Grand-tour of the MNIST-Fashion[START_REF] Li | Visualizing Neural Networks with the Grand Tour[END_REF] with as many dimensions as the number of classes displayed on the most right of the figure.

C h a p t e r 3 V 51 3. 6

 3516 I S U A L Q U E S T I O N A N S W E R I N G Contents 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.2.1 Transformers and Attention . . . . . . . . . . . . . . . . . 42 3.2.2 Vision-Language (Vision-Language (VL))-Transformers . 44 3.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.3.1 Interpretability of VQA . . . . . . . . . . . . . . . . . . . . 46 3.3.2 Bias Reduction in VQA . . . . . . . . . . . . . . . . . . . . 46 3.4 Motivating Case Study . . . . . . . . . . . . . . . . . . . . . . . . 47 3.5 Design Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Design of VisQA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.6.1 Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.6.2 Visualization of Instances . . . . . . . . . . . . . . . . . . 54 3.6.3 Visualization of Selected Heads . . . . . . . . . . . . . . . 55 3.6.4 Interacting with Models . . . . . . . . . . . . . . . . . . . 56 3.7 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.8 Evaluation with Domain Experts . . . . . . . . . . . . . . . . . . . 58 3.8.1 Evaluation Protocol . . . . . . . . . . . . . . . . . . . . . . 58 3.8.2 Object Detection and Attention . . . . . . . . . . . . . . . 60 3.8.3 Questions with Logical Operators . . . . . . . . . . . . . 61 3.8.4 Vision to Vision Contextualization . . . . . . . . . . . . . 62 3.9 Discussions, Limitations and Future Work . . . . . . . . . . . . . 63 3.10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Figure 3 . 3 -

 33 Figure 3.3 -An Illustration of the VL-Transformer architecture we rely on.Question and image are first tokenized and then encoded using vision (in green) and language (in blue) only transformers[START_REF] Vaswani | Attention is all you need[END_REF], followed by (bi-directional) intermodality transformers[START_REF] Hao | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF]. The answer is predicted from the "CLS" token. Yellow and orange rectangles represent, respectively, inter-and intra-modality attention heads. i and j are the layers and head indices used for naming attention heads through the manuscript.

Figure 3 . 4 -

 34 Figure 3.4 -When asked "Is the knife in the top part of the photo"x the tiny-LXMERT model, with the image of a knife at the bottom y, incorrectly outputs "yes" z with more than 95% confidence. While an exploitation of bias can be considered, we can observe that the answer "yes" represents only 33% of answers of similar questions over the complete dataset. Thus in-depth analysis of the attention of the model may be required to grasp what led to such a mistake.

  Is the knife in the top part of the photo?"

Figure 3 . 5 -

 35 Figure3.5 -Visualization of a selected vision-to-language head and attention map for two different models. x the noisy model associates the "knife" word with a large number of different objects, including fruit. y the oracle model learns a perfect association between the word "knife" and the "knife" object; z the oracle transfer model associates the word "knife" with two different bounding boxes of type knife handle, whose embeddings are sufficiently close for correct reasoning. Head selections are not comparable between models, we therefore checked for permutations.

Figure 3 . 6 -

 36 Figure 3.6 -Opening the black box of neural models for vision and language reasoning: given an open-ended question and an image , VisQA enables to investigate whether a trained model resorts to reasoning or to bias exploitationto provide its answer. This can be achieved by exploring the behavior of a set of attention heads , each producing an attention map , which manages how different items of the problem relate to each other. Heads can be selected , for instance, based on color-coded activity statistics. Their semantics can be linked to language functions derived from dataset-level statistics , filtered and compared between different models. This tool is available online at: https://visqa.liris.cnrs.fr.

Figure 3 . 7 -

 37 Figure 3.7 -Hovering the mouse over a cell of the attention maps x filters the corresponding object bounding box in the input image y. While clicking on this cell filters attention heads in instance-view to display those within which the selected cell is highly activated.

Figure 3 . 8 -

 38 Figure3.8 -When asked "Is the person wearing shorts?", the oracle transfer model successfully answers "yes". It can be observed in its first Language-to-Vision attention maps, that the word "shorts" (column) is strongly associated with the object "shorts" (row). The same phenomenon is also observed for the word "person', strongly associated with objects labeled as "woman" among others.

Figure 3 . 11 -

 311 Figure3.11 -Despite different performances, we can observe that both tiny-LXMERT and large-LXMERT have the same behavior as the frequency of question/answers increases. Hence, we can estimate that they may be exploiting similar shortcuts defined in GQA-OOD[START_REF] Kervadec | Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?[END_REF].

Figure 3

 3 Figure3.12 -Difference between the attentions of Head LV_1_0, when asked "is the the train blue?", and "is the the train red?". We can observe that in this head, the attention focuses on different objects (row) depending on the color asked (column).

Figure 4 . 1 -

 41 Figure 4.1 -This chapter is dedicated to step y of finding my keys, i.e. the ability for robots to explore unknown environments. To do so, such a robot relies on images sampled from an onboard camera (most right) to decide what should be its next direction.

Figure 4 . 2 -

 42 Figure 4.2 -Our navigation problem consists in solving a visual task (e.g. fetch, interact, or recognize items) while avoiding obstacles in an environment. DeepReinforcement Learning can be used to solve this problem by using an image as input x at time t. Features are then extracted from this image y, and combined with the previous memory vector t -1 z. Using this memory vector, the agent decides to move forward or turn left, for instance {.

Figure 4 . 3 -

 43 Figure 4.3 -Memory construction process: at a current time-step t, the agent updates its memory by producing a new memory vector. Each dimension of this vector (represented as a column) is appended to the previous ones chronologically (from left to right). As a result, each row of the appended vectors represent the actions of a single memory element.

Figure 4 . 4 -

 44 Figure4.4 -DRLViz displays a trained agent memory, which is a large temporal vector, as a horizontal heat-map . Analysts can browse this memory following its temporal construction; filter according to movements of the agent and derived metrics we calculated (e.g. when an item is in the field of view ); and select the memory to filter elements and compare them .

Figure 4 . 4 1 . 2 . 3 .

 44123 Figure 4.4 shows an overview of DRLViz where the most prominent visualization is the memory timeline of a trained agent (G2). The primary interaction is

Figure 4 . 6 -

 46 Figure 4.6 -Summary of the insights gained by the experts. Expert #1 noticed two intervals during which the agent only turned right, by using both trajectory x and stacked area chart of actions y views.Once he replayed those sequences, he stated that the agent came twice in the same dead-end z. Expert #3 observed a hidden state dimension which is blue when the agent sees the red armor before the green armor, and then remained orange until when he saw the green armor {. Expert #2 probed a dimension that is active as the agent first saw the Health Pack, and remained active until it gathered it. Expert #1 also identified two hidden state elements that change as the agent gathered the health pack and then kept their values until the end of the episode }. Using saliency maps ~, Expert #2 observed that the agent ignored the soul-sphere until it gathered the first three items . Finally, Expert #3 identified clusters in the t-SNE projection which corresponds to the agent's objectives e.g. gathering the green armor .

Figure 4 . 7 -

 47 Figure 4.7 -Overview of MemRed, an online explorable in which users are invited to try different strategies to reduce the memory of a DRL agent and observe how it affects its behavior. This explorable earned the distinction of "best paper" at the VISxAI workshop 2019.

Figure 4 . 8

 48

Figure 4 . 8 -

 48 Figure 4.8 -Extended version of DRLViz loaded on with death-match data. From a slit square selection x outputs a timeline that summarizes the agent's point of view y. And the additional metrics and operators z.

Figure 5 . 1 -

 51 Figure5.1 -When answering questions such as "Where are my keys?", robots need to be able to communicate their position. To do so, we trained models to yield coordinates and orientation angle from any given image z. Such a task is also used as a proxy to evaluate gaps between simulation and reality.

Figure 5 . 2 -

 52 Figure 5.2 -In the studied robot ego-localization task, an RGB-D image x, is given to a trained model y, which uses it to regress the location (x, y), and orientation angle (α) in the environment from which this image was taken from z.As illustrated above, images taken from the same coordinates in simulation and real-world x may lead to different predictions due to differences, such as here, among others, the additional presence of a bike in the scene. We are interested in reducing the gap between the SIM and REAL predictions.

1 2Figure 5 . 3 -

 153 Figure 5.3 -The capture of real-world images are carried by an embedded RGB-D camera on a "Locobot"[START_REF] Murali | PyRobot: An Opensource Robotics Framework for Research and Benchmarking[END_REF] x. On the right y such a robot is displayed with respect to proportions in its environment.

Figure 5 . 4 -

 54 Figure5.4 -Using Sim2RealViz, the Sim2Real gap of a Data Augmentation model can be compared against other models (e.g. Vanilla or Fine-tuned) and displayed on the real-world environment map along with its performance metrics. In particular, Sim2RealViz shows x this model is particularly effective in simulation but we identified errors in the environment, such as the model failing to regress its position because of a closed-door that was opened in training. Such an error can then be selected by instance on the map y to identify key features extracted by the model either as superimposed on the heat-map z or as a first-person view {.

2 1 3 Figure 5 . 5 -

 355 Figure5.5 -To tackle over-plotting in geo-map while preserving the insights users can grasp with this view, Sim2RealViz provides three way to encode SIM / REAL predictions along with their ground-truth. From x which displays a pin per domain and link them to their ground truth, to y a glyph in which portion size encodes the distance to their color encoded domain, while their position encodes their direction. Or z, only display ground-truth pins with their color encoding the distance between simulation and real-world.

Figure 5 . 6 -

 56 Figure 5.6 -Conversion from pixels on a first-person point of view image to coordinates on a bird's eye geo-map (left) using inverse projection given a calibrated camera. Such a process, used in Sim2RealViz to display global heatmaps (right) on the geo-map, relies on ground-truth, image, and camera information.To optimize their computation, geo-maps are discretized into squares larger than a pixel, as a trade-off between the accuracy of projections, and the user to the feedback.

Figure 5 . 7 -

 57 Figure 5.7 -By clicking on the adjust button (on the top-right of Figure 5.4), users can display sliders on the right of instance view Figure 5.4 {) that can be used to fine-tuning real-world images with filters and observe how it affect models' prediction.

2 1 Figure 5 . 8 -

 2158 Figure5.8 -By using the full encoding, we can observe that most real-world predictions are located in the half left of the environment x. Hence, instances sampled from the half-right of the environment provide the worst predictions. However, when we slightly increase the brightness of each real-world image, we can observe that instances are more evenly distributed over the environment y.

1 2Figure 5 . 9 -

 159 Figure 5.9 -With global heatmaps of feature-distance, we can observe (in red) areas of the environment that may be affected by a sim2real gap. Those areas correspond to changes in objects present in the simulation, for instance as illustrated here, a fire-extinguisher. By removing such objects in simulation and retraining a model on them, we can observe that they disappeared from most highlighted areas.

C h a p t e r 6 C

 6 O N C L U S I O N A N D F U T U R E D I R E C T I O N S Contents

  FOLIO ADMINISTRATIF THESE DE L'UNIVERSITE DE LYON OPEREE AU SEIN DE L'INSA LYON NOM : Jaunet DATE de SOUTENANCE : (approx. 16/05/2022) Prénoms : Théo, Antoine TITRE : Interprétabilité de l'Apprentissage Profond via Analyse Visuelle : Exploration de Raisonnements et de l'Exploitation de Biais NATURE : Doctorat Numéro d'ordre : 2022LYSEI041 Ecole doctorale : EDA 512 : Informatique et Mathématiques de Lyon Spécialité : Informatique RESUME :

  Robotics Task: "Where are my keys" . . . . . . . . . . . . . . . . . . 5 1.2 Visual Analytics for Model Interpretability . . . . . . . . . . . . . . 6 1.3 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.1.1 Visual Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1.2 Deep Neural Networks . . . . . . . . . . . . . . . . . . . . . . 11 2.1.3 Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.1.4 Interpretability Vs. Explainability . . . . . . . . . . . . . . . 15 2.1.5 Explanations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.2 Building Blocks of DNN Interpretability . . . . . . . . . . . . . . . . 17 Interacting with Models for Non-experts . . . . . . . . . . . 34 2.3.4 Interpretable Visual Analytics Throughout this Manuscript

	abstract	iii
	résumé	v
	remerciements	vii
	contents	ix
	list of figures	xiii
	list of tables	xxiii
	acronyms	xxv
	1 introduction	1
	1.1 7
	2 related work	9
	2.1

2.2.1 Activations of Neurons . . . . . . . . . . . . . . . . . . . . . 18 2.2.2 Visualization with Gradients . . . . . . . . . . . . . . . . . . 22 2.2.3 Inner Representation of Data . . . . . . . . . . . . . . . . . . 25 2.2.4 Model-Agnostic Methods . . . . . . . . . . . . . . . . . . . . 27 2.3 Leveraging Building Blocks in Visual Analytics . . . . . . . . . . . . 29 2.3.1 Interactive Activations . . . . . . . . . . . . . . . . . . . . . . 30 2.3.2 Sequential Activations and Gradients . . . . . . . . . . . . . 31 2.3.3

Table 3 . 1

 31 Experts performances averaged by instances while evaluating VisQA. We can observe that Expert#3 reached the best accuracy on every questions, while Expert#5 reached the lowest results. Overall, the average performance for Q1 and Q3 is 75% of accuracy, and 61% for Q2. . . . . . . . . . . .

Chapter 4: navigation

Table 4 .

 4

1 List of re-ordering criteria as they appear in DRLViz. t is the current time-step, n the number of steps (525 at most), and i the element. . . . . . . . . . . . . . . . . . . . . . . . . Table 4.2 List of derived metrics (from top to bottom on Figure 4.4 z) Table 4.3 Performances of agents with different memory reduction strategies (each averaged over 100 episodes). Best result of each column is bold. . . . . . . . . . . . . . . . . . . . . . .

  Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.1.1 Visual Analytics . . . . . . . . . . . . . . . . . . . . . . . . Building Blocks of DNN Interpretability . . . . . . . . . . . . . .

	3 • Théo Jaunet, Corentin Kervadec, Romain Vuillemot, Grigory Antipov, Moez C h a p t e r 2
	Baccouche, and Christian Wolf. "VisQA: X-raying Vision and Language Rea-
	soning in Transformers". In: IEEE Transactions on Visualization and Computer R E L AT E D W O R K
	Graphics (TVCG), 2021.
	• Corentin Kervadec, Théo Jaunet, Grigory Antipov, Moez Baccouche, Romain Vuillemot, and Christian Wolf. "How Transferable are Reasoning Patterns Contents
	in VQA?". In: IEEE Conference on Computer Vision and Pattern Recognition 2.1
	(CVPR), 2021.
	Chapter 4

• Théo Jaunet, Romain Vuillemot, and Christian Wolf. "DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning". Computer Graphics Forum (Proceedings of EuroVis 2020), 2020. • Théo Jaunet, Romain Vuillemot, and Christian Wolf. "What if we Reduce the Memory of an Artificial Doom Player?". IEEE Workshop on Visualization for AI Explainability at IEEE VIS (VISxAI), 2019. This work was awarded "best-paper". Chapter 5 • Théo Jaunet, Romain Vuillemot, and Christian Wolf. "Théo Guesser". IEEE Workshop on Visualization for AI Explainability at IEEE VIS (VISxAI), 2020. • Théo Jaunet, Guillaume Bono, Romain Vuillemot, and Christian Wolf. "Sim2RealViz: Visualizing the Sim2Real Gap in Robot Ego-Pose Estimation". NeurIPS XAI Workshop on eXplainable AI approaches for debugging and diagnosis, 2021. This work will be extended in a follow-up journal. 2.1.2 Deep Neural Networks . . . . . . . . . . . . . . . . . . . . 2.1.3 Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.4 Interpretability Vs. Explainability . . . . . . . . . . . . . . 2.1.5 Explanations . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 2.2.1 Activations of Neurons . . . . . . . . . . . . . . . . . . . 2.2.2 Visualization with Gradients . . . . . . . . . . . . . . . . 2.2.3 Inner Representation of Data . . . . . . . . . . . . . . . . 2.2.4 Model-Agnostic Methods . . . . . . . . . . . . . . . . . . 2.3 Leveraging Building Blocks in Visual Analytics . . . . . . . . . . 2.3.1 Interactive Activations . . . . . . . . . . . . . . . . . . . . 2.3.2 Sequential Activations and Gradients . . . . . . . . . . . 2.3.3 Interacting with Models for Non-experts . . . . . . . . . 2.3.4 Interpretable Visual Analytics Throughout this Manuscript

Table 3 . 1 -

 31 Experts performances averaged by instances while evaluating VisQA. We can observe that Expert#3 reached the best accuracy on every questions, while Expert#5 reached the lowest results. Overall, the average performance for Q1 and Q3 is 75% of accuracy, and 61% for Q2.

	Experts	Q1	Q2	Q3
	Expert#1	0.6	0.5	0.67
	Expert#2	0.83	0.67	0.83
	Expert#3	1	0.83	1
	Expert#4	0.83	0.67	0.5
	Expert#5	0.5	0.33	0.67
	Expert#6	0.67	0.67	0.83
	Total	0.75	0.61	0.75

  • Théo Jaunet, Romain Vuillemot, and Christian Wolf. "DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning". Computer Graphics Forum (Proceedings of EuroVis 2020), 2020. • Théo Jaunet, Romain Vuillemot, and Christian Wolf. "What if we Reduce the Memory of an Artificial Doom Player?". IEEE Workshop on Visualization for AI Explainability at IEEE VIS (VISxAI), 2019.

Table 4 . 1 -

 41 List of re-ordering criteria as they appear in DRLViz. t is the current time-step, n the number of steps (525 at most), and i the element.

	Criteria	Formula	Description
	activation	n ∑ t=1	|h ti |	Elements most involved in deci-sions.
		n-1
	change

Table 4 . 2 -

 42 List of derived metrics (from top to bottom on Figure 4.4 z)

	Metric	Data Type Values
	Health of the agent	Quantitative death [0,100] full
	Event (item gathered)	Flag	(1) gathered
	Item in FoV	Binary	no item (0, 1) item
	Orientation to items	Degree	left [-45,45] right
	Variation of orientation Quantitative stable [0,30] change
	Decision ambiguity	Ratio	sure [0,1] unsure
	colors corresponding to the ones on the action distribution graph in 4.4. With this
	visualization, users can observe similar sequences of decisions.

Table 4 . 3 -

 43 Performances of agents with different memory reduction strategies (each averaged over 100 episodes). Best result of each column is bold.

	Type of reduction Steps	Health Pack Poison	Health
		survived gathered	gathered
	Full-memory	503.98	37.56	4.28	81.47
	Half-memory	493.92	37.88	4.66	81.61

  Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . 117 6.2 Perspectives for Future Works . . . . . . . . . . . . . . . . . . . . 119 6.2.1 Invade and Conquer Model Builders' Workflow . . . . . 120 6.2.2 Mitigating Human Biases . . . . . . . . . . . . . . . . . . 122 6.2.3 Finally Finding those Keys! . . . . . . . . . . . . . . . . . 124

	6.1

  MOTS-CLÉS : Apprentissage Profond, Visualisation de données, Robotique, Explicabilité de l'IA Laboratoire (s) de recherche : LIRIS Directeur de thèse: Christian Wolf, Maître de conférences HDR, INSA-Lyon Co-directeur de thèse : Romain Vuillemot, Maître de conférences, École Centrale Lyon Président de jury : Liming Chen, Professeur des Universités, École Centrale Lyon Composition du jury : Céline Hudelot, Professeure des Universités, CentraleSupélec David Auber, Professeur des Universités, Université Bordeaux Shixia Liu, Professeure des Universités, Tsinghua University Hendrik Strobelt, Chargé de Recherche, IBM Research

Cette thèse est accessible à l'adresse : https://theses.insa-lyon.fr/publication/2022LYSEI041/these.pdf © [T. Jaunet], [2022], INSA Lyon, tous droits réservés

The second floor of the CITI laboratory at INSA-Lyon. Cette thèse est accessible à l'adresse : https://theses.insa-lyon.fr/publication/2022LYSEI041/these.pdf © [T. Jaunet], [2022], INSA Lyon, tous droits réservés

agent to explore the environment 20 times with different configurations (i.e. positions of items, start position of the agent, its orientation). During those episodes, we collected information from the agent at each time-step such as its FoV image, action probabilities, memory vector, and information from the environment such as the items in the agent's FoV, the position of the agent, the agent's orientation and its health. The collected data is formatted as a JSON file which groups data elements per episode and then per step with an average of 30Mo per episode. Those data are generated using DRL models implemented in Pytorch [START_REF] Paszke | Automatic differentiation in PyTorch[END_REF], and formatted in Python 3. The user interface of DRLViz loads data using JavaScript and D3 [START_REF] Bostock | D3: Data-Driven Documents[END_REF]. The interactions between the model and the front-end are handled by a Flask Python server. The data, separated per episode is generated in a plug-in fashion i.e. without altering the model or the simulator.