La voie de la recheche est pour moi un projet de longue date. Je voulais déjà à l'âge de 14 ans devenir chercheuse en chimie. Je n'avais alors aucune idée de ce qu'était la vision par ordinateur ou l'intelligence artificielle, et il est assez fou de réaliser à quel point ces domaines ont évolué rapidement depuis cette époque. Je regarde aujourd'hui mes trois ans de thèse avec une grande fierté, ascenseur émotionnel permanent entre doutes et détermination, exaspération et passion, échecs et succès pour ne finalement garder que le meilleur. J'emprunte aujourd'hui une voie différente de celle que je m'étais fixée il y a des années, laissant de côté la recherche académique pour le monde industriel. Toutefois je garderai une vision éclairée sur la recherche, l'innovation, la technologie, et la transition entre l'académique et l'industrie qui ne manquera pas de me servir à l'avenir. Il est maintenant temps de remercier en bien moins de lignes qu'il n'en faudrait ceux qui ont, de manière délibérée ou non, directement ou indirectement, contribué à ce travail et à cette tranche de vie qui restera probablement l'une des meilleures que j'ai vécue.

Dans un premier temps, je voudrais remercier David qui a été un directeur de thèse parfait. Très à l'écoute, avec une vision claire et précise des problèmes, apte à répondre sur la plupart des questions techniques, et de proposer de très bonnes idées tout en laissant une grande liberté d'orientation. Il restera pour moi un modèle dans la manière d'encadrer des projets de recherche. Je sais qu'avoir un encadrant aussi impliqué est loin d'être courant et je mesure à quel point ma thèse aurait été différente sans lui.

Je remercie également Jean-François qui a représenté le pendant industriel de mon encadrement. Je n'ai pas forcément donné à mon travail toute la dimension bio-mimétique qu'il aurait souhaité, mais espère que mes recherches apporteront tout de même leur modeste contribution à l'édifice de ce très ambitieux projet. Par ailleurs je ne peux qu'être mille fois reconnaissante pour son accompagnement dans la découverte du métier d'ingénieur recherche chez Thales. Je lui dois à la fois mon recrutement en tant que doctorante et celui en tant que docteur, et il a su apporter à ma thèse la dimension entreprise qui m'a été et me sera précieuse pour la suite de ma carrière.

Je remercie mon jury de thèse, dont j'ai beaucoup apprécié la composition, car chacun des membres a eu une influence directe sur mes travaux. Bien entendu, Pierre-Yves Oudeyer qui a suivi depuis Bordeaux mes travaux, et dont les discussions, relectures et conseils m'ont beaucoup aidées. Il reste également pour moi un modèle pour la qualité technique et l'originalité de ses travaux de recherche, mais aussi dans la manière de les valoriser. Simone Frintrop que je n'ai eu le plaisir de rencontrer que durant ma soutenance, mais dont les publications scientifiques ont largement influencées, inspirées et guidées mes recherches. Sans cet apport, mes travaux auraient sans doute pris une autre direction. Olivier le Meur dont j'avais suivi une présentation lors du GdR Isis au tout début de ma thèse, et dont la qualité du travail m'avait marqué et également inspiré et éclairé. Enfin, Sergio Rodriguez, qui m'avait proposé lors de mes recherches de thèses un sujet très intéressant que j'ai bien failli accepter. J'ai finalement opté pour la proposition de Thales, mais notre collaboration n'est peut-être que partie remise.

Il est maintenant temps de remercier un peu plus précisément les deux entités dans lesquelles j'ai passé le plus clair de mon temps. Je ne remercierai évidemment pas l'ANRT, dont l'efficacité administrative m'a coûté sept mois de chômage dans un flou total avant de pouvoir commencer ma thèse. La suite fut heureusement bien plus réjouissante.

Dans un premier temps, je voudrais remercier les doctorants, post-docs, ingénieurs, chercheurs, et stagiaires de l'U2IS. Bien entendu, une pensée toute particulière pour Antoine avec qui j'ai partagé avec une constance presque rituelle de nombreuses « pauses café» à refaire le monde et débattre sur tout et n'importe quoi. À quoi aurait ressemblé ma thèse sans Yuxin, mon voisin de bureau excentrique, dont les habitudes n'ont pas manquées de m'interloquer, m'amuser, m'exaspérer, me toucher tout au long de ces trois ans. Amir, le workoholic acharné, probablement voué à une grande carrière scientifique. Adrien, le théoricien adepte de la méta-discussion. De manière plus générale, merci à ceux avec qui j'ai pu partager des soirées jeux ou encore des repas de groupe. Pauline et Gennaro, Clément P et Clément M ou Louise pour ne citer qu'eux. Je remercie également les membres de l'équipe Flowers que j'ai pu rencontrer à Bordeaux ou à l'autre bout du monde en conférence. Enfin ceux qui ont partagé mon bureau pour des périodes plus ou moins longues au cours de ces trois ans : Alexandre, Mathieu ou encore Timothée. Sur un plan plus pratique, je remercie Louis-Charles dont les travaux de segmentation m'ont beaucoup aidé pour ma propre recherche, Natalia dont le manuscrit m'a fortement inspiré en début de thèse, Panos, avec qui nous parviendrons peut être un jour à co-écrire un article, et enfin Mathieu Lefort, qui, malgré lui, m'a par deux fois aidé à voir et aller plus loin dans ma thèse. Sans lui, RL-IAC n'aurait probablement pas vu le jour.

Je tiens à présent à remercier mes collègues de Thales, qu'ils soient toujours présents à Palaiseau ou déjà partis vers de nouveaux horizons. Tout d'abord Philippe Mouttou puis Thierry Lamarque, ainsi que Catherine Simon puis Alain Marcuzzi pour avoir rendu ma thèse possible au sein de Thales. Merci à Louise Naud également pour avoir proposé ma candidature de thèse au sein du vision lab. Merci maintenant à ceux qui m'ont entouré au cours de ces trois années dans une ambiance très chaleureuse et amicale. Autour d'un café, d'un écran de PC, d'un robot, d'un escape game, d'une ou deux (ou plus !) bières, d'une démo qui marche bien, d'un pain au chocolat pour fêter quelque chose (et il y en a souvent !), devant, derrière ou sous une caméra, au fond d'un train 1 ou dans un vieux grenier 2 . Que chacun comprenant le sens de cette liste le prenne personnellement comme étant parti intégrante de ces remerciements. Un merci tout particulier pour Michael (celui avec un h) avec qui j'ai eu l'occasion de travailler davantage et avec grand plaisir pour monter les démos autour de BioVision. Enfin un remerciement particulièrement ému pour Luigi que j'ai encadré avec beaucoup d'enthousiasme, et dont les dernières semaines de stage ont été particulièrement captivantes.

Pour conclure, je tiens finalement à remercier mes amis. Clément et Louise pour les soirées jeux quasi-hebdomadaires et la pratique assidue du time's up, Kévin, Aurélien, Etienne et JA, mes amis d'enfance que j'ai pris plaisir à retrouver après mon exil brestois et canadien, les « pipos » de Télécom Bretagne avec un souvenir très ému du mariage inattendu de Perez, Julie, Alexis et mes parents, mais aussi ma belle-famille. Bien entendu, une grande partie de ces remerciements vont à Simon qui me suit (à moins que ce ne soit moi ?), m'accompagne, me soutient, me supporte depuis plus de sept ans. Je n'aurais pas pu imaginer meilleur retour à Paris après mes années d'étude au Canada. Malgré le climat défaitiste qui règne en France depuis quelques années, c'est en partant loin qu'on se rend compte que l'herbe n'est pas forcément plus verte dans le jardin du voisin, et qu'on peut y vivre plutôt bien ! Chapter 1

Introduction 1.1 Context

The last few years have shown a strong renewed interest toward artificial intelligence, robotics and perceptual systems. The fast and impressive progresses in these fields suggest that exciting breakthroughs are to be announced in the years to come. Moreover, the biological inspiration is at the heart of this process, providing a rich pool of ideas and examples to truly understand and create intelligence.

This work is part of this framework, and investigates more precisely the problem of designing perceptual systems capable of autonomous behaviors, in a bioinspired context.

Autonomous perceptual systems

An autonomous perceptual system is a system equipped with one or several sensors, able to behave or execute tasks with a high level of autonomy. This involves on the one hand using the available sensor data to make decisions, and, on the other hand avoiding as much as possible human assistance or intervention. Robots are a good examples of perceptual systems, but they are neither necessarily perceptive, nor autonomous. For example, factory robots designed to accomplish pre-defined sequences of actions are autonomous systems, but they are not perceptual one. On the contrary, drones equipped with cameras are perceptual systems, but they are most of the time remotely controlled by users; therefore, they are not autonomous. Building autonomous perceptual systems is one of the major challenges of this century. In the past decades, considerable advances have been achieved in that regard, but the conception of truly autonomous systems is still several decades ahead. This section presents a few present and future examples of autonomous perceptual systems, as well as the main challenges related to those technologies.

Present and future

Autonomous perceptual systems are already ubiquitous (See Figure 1.1. Among popular examples, one can think of the autonomous vacuum cleaner Roomba, that detects dust, avoids obstacles and autonomously finds trajectories to clean rooms. Intelligent personal assistants such as Siri, Google Now, or Cortana also lie in this category, using a microphone to analyze speech and answer, when possible, the user's request. The third generation "fire-and-forget" of missiles uses radars, inertial sensor and satellite guidance to autonomously adjust their trajectory towards a target. Lastly, Mars exploration rover Curiosity has used a large set of sensors to get a 3D mapping of the surface and navigate autonomously and safely on Mars' ground in spite of delayed and sometimes broken communications with earth. The future of autonomous systems involves a large variety of applications (see Figure 1.2) and will likely change our lifestyle by strongly enhancing human assistance or by accomplishing annoying or dangerous tasks. Systems of the next generation will operate in more complex environments, handle interactions with humans while coping with safety issues, and accomplish more diverse and more complex tasks. Those systems are heavily studied, and several promising prototypes have already shown good reliability on constrained environments. There is no doubt that many of those technologies will be part of our life in the next few decades.

Driverless cars is a first example of such systems and already has number of prototypes around the world. The most famous example is the Google Car, that has already shown high efficiency by driving millions of kilometers with a very small number of minor accidents. Autonomous UAVs also show promising potential applications, ranging from search and rescue missions in dangerous environment to mail delivery. Autonomous localization and navigation by UAVs raises a large number of challenges that are still far from being solved. In the context of urban surveillance, PTZ (pan-tilt-zoom) cameras could become fully autonomous so as to avoid human monitoring. This would for instance involve automatic detection and tracking of pedestrians or cars. Lastly, domestic robotics is also a promising field for helping or replacing humans in their everyday life chores. Today's domestic robots are specialized in a single task, but strong efforts are put to make them versatile and adapted to the user's need. Nevertheless, regarding the complexity of some tasks and human safety aspects, this area of research is still at its earliest stages.

Both academy and industries are supporting efforts to push forward advances 1.1. Context in this field, for example with challenges such as the DARPA Robotics Challenge1 , or the annual Imagenet for Large Scale Visual Recognition Challenge2 .

Challenges

Today's perceptual systems are often designed to perform a very specific task. However, to conceive more versatile, autonomous and interactive systems, significant challenges must be tackled. The ultimate goal of an artificial general intelligence (AGI) able to solve the Turing test [START_REF] Turing | Computing machinery and intelligence[END_REF] (try to fool a human while having a conversation with him) or the Coffee test [START_REF] Goertzel | The architecture of human-like general intelligence[END_REF] (go into an average American house and figure out how to make coffee) is still far from being achieved. Those systems should efficiently operate in complex environments. Indoor environments are typically composed with many types of objects of various sizes and purposes. Strong object recognition and localization capacities are then required to efficiently interact with them, even when those objects are not stowed or piled up. The presence of humans in this environment is another major issue: understanding and predicting their behavior to avoid harming them, or just bothering them is of paramount importance. Outdoor environment are also challenging, as perception must be robust to various lighting, temperature and weather conditions. Dealing with rough or muggy ground surfaces also makes the problem harder to solve.

Another challenge consists in processing the raw collected data to make it usable for analysis and understanding of the environment. This processing should be both accurate and stable enough to handle an infinite variety of inputs. Major improvements in that regard have emerged with the use of deep neural networks on many types of input signal. Today's deep networks perform remarkably well on object or speech recognition, on extremely challenging types of data [START_REF] Sak | Fast and accurate recurrent neural network acoustic models for speech recognition[END_REF], [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. However, integrating novel, unusual, or modified data to strengthen knowledge on the fly, is still far from being solved.

The process of actions and decisions is a critical aspect as well. Asimov's laws enunciated in the 40's described a set of rules a robot should respect. More recently, Google announced five key problems towards robot safety [START_REF] Amodei | Concrete problems in ai safety[END_REF]. Among the main difficulties, decisions should be such that it enable a smart and safe displacement within the environment (for example, to avoid missing important information in the case of surveillance), consider contextual information and take decisions accordingly (for example, being quiet at night, consider moods and emotions of humans before interacting with them), and most importantly, avoid harmful situations, either for human beings or for itself.

Bio-inspired systems

Living-beings have been seen for centuries as a wide source of inspiration for human technology. Although biology is not always a good example to produce artificial mechanisms (such as wheel or plane for instance), it contains a wide variety of concrete examples for thoughts and ideas (would we have planes if birds did not exist?). Unlike bio-mimetics, bio-inspiration takes biology as a pool of interesting problems rather than interesting solutions. In robotics, nature and especially humans represent a good baseline to achieve complex functions such as environment perception or learning.

Human vision

The human vision system, ranging from the retinal photo-receptors to the deepest layers of the visual cortex able to identify objects, is an excellent example of biological system that has inspired number of applications related with computer vision -including the digital camera itself. So far, the performance of human vision as regards detection, recognition and contextual information integration remains outstanding and unequaled by machines. It is therefore very likely that a deeper understanding of its core mechanisms could push advances in computer vision forward.

A first interesting component of the human vision is the retina, which is the light-sensitive layer of tissue at the back of the eye. The retina is composed with photo-receptors (cones and rods) as well as ganglion cells that apply pre-processing on contrasts, illuminations or motions before transferring information to the visual cortex. An interesting feature of the retina lies in the density distribution of photoreceptors: cones are significantly (more that 30 times) denser at the center of the retina than at the periphery. The area having the denser population of cones is called the fovea and only covers 2 • of the visual field. It is particularly important in the vision process, as the acuity is much higher in this region, and about 25% of the visual information projected to the brain comes from there. As opposed to the fovea, the periphery covers a much larger field of view with a much more modest resolution (See Figure 1.3).

The fovea is strongly involved in the process of recognition. Conversely, the From Gilbert and Li [START_REF] Gilbert | Top-down influences on visual processing[END_REF] periphery monitors with a very rough and fast process the potentially interesting (or salient) elements of the visual field. As a result, the information from the periphery of the retina is processed by the brain as a pre-attentive analysis of the visual field, to detect targets where fovea should be directed to, for more attentive processes (such as recognition). Such mechanisms have been widely imitated in robotics, typically by using pairs of cameras with different focal distances (See Figure 1.4). The pre-attentive step is processed on the cameras with a wide field of view and produces saliency maps to suggest interesting targets. Cameras with a sharp field of view are then oriented toward those targets to obtain images at a much higher resolution.

The major processing of the visual inputs takes place in the visual cortex. This one is composed with interconnected neurons, and can be divided into areas (See Figure 1.5). Those areas are also connected, and process the information at several layers of abstraction, starting from basic edges and orientations (V1) to simple shapes (V4) and finally objects (IT). This architecture of interconnected neurons has strongly inspired artificial systems able to learn representations from raw inputs (HTM [START_REF] Hawkins | Hierarchical temporal memory: Concepts, theory and terminology[END_REF], convolutional neural networks [START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF]).

Developmental learning

A natural idea when designing fully autonomous systems and robots is to make them gain knowledge as they evolve in their environment instead of just programming them for the task they are meant to. This approach should enable a better behavior and understanding of their environment. This approach also makes it possible to integrate new skills depending on the needs of the user, and to remain flexible to any change in the environment. Designing robots able to learn by themselves is not a new idea. In 1950, Turing [START_REF] Turing | Computing machinery and intelligence[END_REF] already suggested to create robots able to learn like children, hoping that using such learning mechanisms would be easier than reproducing an adult human intelligence from scratch.

Developmental robotics takes inspiration from learning and development of humans and animals, to conceive robots able to adapt to their environment and acquire skills of increasing complexity over time [START_REF] Weng | Autonomous mental development by robots and animals[END_REF]. Examples of developmental robotics setups are displayed in Figure 1.6. Of particular interest are the constraints that make learning skills or knowledge easier. Indeed, the extremely high number of things that can be learned during a lifetime makes it impossible to explore in a fully open-ended manner, without any constraint. Considering vision, the infant's visual system in the few days following their birth is surprisingly poor: the child only perceives very close objects or faces. As infants grow, their recognition capacity extends to farther objects as the visual system gains maturity. More than a limitation due to the immature visual system, this development also makes learning constrained at the beginning, before more complex skills such as visual attention can emerge.

A major area of study in developmental robotics is the one of intrinsic motivation [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF]. This mechanism makes an autonomous agent select actions or activities where learning is possible, without any short-term goal. More specifically, some approaches implement curiosity mechanisms based on learning progress to learn how to perform tasks [START_REF] Oudeyer | Intrinsically motivated learning of real-world sensorimotor skills with developmental constraints[END_REF]. The main idea is to make robots focus on tasks where learning progress is maximal at a given time. This strategy allows a robot to learn continuously, by focusing first on simple things, before investigating more complex tasks.

Goals and contributions 1.2.1 Problem definition and areas of investigation

To better understand the mechanisms lying behind autonomous perceptual systems, let us consider Figure 1.7. Autonomous perceptual systems are composed with a physical and a mental layer. The physical (hardware) layer is used as an interface between mental (software) layer of the system and the environment: The actuator is controlled by a decision and performs an action that influences the environment. In addition, the state of the environment is reflected by perceptual data accessible by sensors. It is then turned into appropriate representations to construct a mental set of knowledge and skills.

As a general problem, we would like to investigate how to design the mental components of the system. This raises two major questions: • How to convert perceptual data to usable skills and knowledge?

• What kind of exploration strategies can be applied to methodically explore the environment in order to acquire new skills and knowledge?

In this work, the problem is presented in a more restrictive scenario by considering the following elements:

• autonomous perceptual systems are limited to active vision systems, able to autonomously control the viewpoint of their camera sensors;

• the skill that should be learned is a visual attention mechanism able to localize salient objects in the surrounding environment;

• we use a developmental learning approach based on intrinsic motivation to provide a methodical and lifelong exploration and learning strategy.

Figure 1.8 represents our case study. To solve this problem, we have identified three main areas of research described in the following sections.

Object localization

To efficiently localize objects in their environment, we propose to develop an approach able to generate saliency maps given visual inputs.

Visual saliency was defined by Laurent Itti as [START_REF] Itti | Visual salience[END_REF] "the distinct subjective perceptual quality which makes some items in the world stand out from their neighbors and immediately grab our attention.". Salient items can either be objects, shapes or any visual stimuli. The only common point is that our fovea is naturally attracted by these items rather than the neighborhood.

A saliency map could be simply defined as a mapping of the input image, where the intensity of each pixel represent the saliency of the input pixel (See example in Figure 1.9). They are typically obtained by applying and combining filters on the input image. This filtering approach is consistent with the biological pre-attentive process in the periphery of the visual field selecting salient items before eye saccades.

Active vision system

Learning state FIGURE 1.8: The problem in our case FIGURE 1.9: An example of RGB input and associated saliency map (image from [START_REF] Bylinskii | Mit saliency benchmark[END_REF] with GBVS saliency map [START_REF] Harel | Graph-based visual saliency[END_REF])

The link between saliency maps and object localization is not immediate though. First, salient items are not always objects. Then, some objects may not be salient in their context. In this work, we are interested in a type of saliency oriented for a specific task. Instead of being attracted by naturally salient items (bottom-up saliency), we would like to enhance areas that are relevant for a task, in a specific environment (top-down saliency). Based on this definition, our saliency maps are specialized for objects having specific constraints (for instance, being on the floor or on tables, belonging to a database), what we produce is an intensity map representing the likelihood of a pixel to belong to these objects. Another aspect is the one of grouping salient pixels to determines the boundaries of the object, that is not taken into account by the saliency map itself.

Online learning

The core mechanism of this work lies in a way to lean saliency models on-the-fly, directly during the exploration of an environment. Instead of providing the system with a model consistent with all types of environments and not able to integrate additional knowledge, we would like the system to construct its own representations, that would be adapted to both the environment and tasks, and be flexible enough to continuously integrate additional knowledge.

Learning and updating models online raises a certain number of constraints. First, the system has to learn by itself without receiving any feedback from users.

Second, the model update process should be fast enough and should not increase too much in time, so that the system can continuously add knowledge to its representations and rapidly make inferences based on this update. Lastly, the model should remain stable and robust in time.

To create a saliency model for object detection, we suggest to use restrictive object detectors, working for examples on objects that are close enough, clearly separated from the background, or that require a sufficiently high resolution. Those restrictive detectors are then used as a teaching signal to feed a saliency model, that should progressively be able to generalize this knowledge to predict the position of those objects when they are farther away, partially occluded, or in cluttered environments.

Intrinsically motivated exploration

In our approach, we take advantage of the action capacities of the active vision system to improve its own representation of the world. As online learning is used to integrate additional knowledge on-the-fly, it does not directly provide a strategy to drive the system and select the best action to take to improve the model. Without any exploration strategy, the system cannot learn autonomously, and without an efficient exploration strategy, there is no warranty that the integrated knowledge will not destroy the present representation, or that learning can be done in a reasonable time.

We then propose to drive the exploration by intrinsic motivation in order to learn autonomously, by actively seeking for missing information given the current state of knowledge. This approach is used to create saliency models adapted to the environment and to the capacity of the system, while making sure that the model is able to adapt to any evolution in the environment. In addition, intrinsic motivation provides a way to learn in an organized manner, adapted to a lifelong learning scenario. This approach ensures efficiency by avoiding catastrophic forgetting, and provides a faster learning than a pure random exploration of the environment.

Contributions

Figure 1.8, divides the mental states in two components that are salient objects localization and intrinsic motivation. Although intertwined in the final system, the two problems are orthogonal and can be treated and applied separately. Our main contributions are then related with those two aspects and could be summarized by the following points.

An incremental approach for learning visual saliency

As a first contribution, we propose a mechanism able to learn saliency models in an incremental way. The literature in the field of visual attention already contains a large number of techniques to generate saliency maps, but these approaches are usually designed to be used as black boxes for other tasks and are not learned (although sometimes refined) during the robot's exploration.

Our system is working in two modes. On the one hand, a learning mode, integrating on the fly the visual inputs perceived by the robot into a model of visual saliency. On the other hand, an inference mode, using the learned model to generate saliency maps. The produced saliency maps are then adapted to the environment of the robot, and remain flexible to future model updates. In addition, our implementation is modular and flexible to the integration of new components.

Aside from this main mechanism, we have developed and improved an existing technique for segmenting objects on planar surfaces [START_REF] Caron | Neural network fusion of color, depth and location for object instance recognition on a mobile robot[END_REF]. This technique is not essential in our work, but is used in the process of saliency learning.

An intrinsically motivated environment exploration with IAC and RL-IAC

Our second contribution is the adaptation of the IAC algorithm [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] in the case of saliency learning in general, and saliency learning on a mobile robot. The IAC (for Intelligent Adaptive Curiosity) mechanism is a drive able to guide the robot in the exploration of the environment in a lifelong learning scenario. This is one of the most famous implementation of an intrinsically motivated exploration drive. In addition, the method is generic enough to be applicable in many scenarios.

We then adapted for the first time IAC to the task of saliency learning. This involved a number of design choices and modifications to make the algorithm consistent and efficient in that regard.

Moreover, we found that IAC was not integrating the time for processing action in the decision scheme. For a mobile robot, this factor was too important to be omitted. We therefore created a new algorithm, RL-IAC (for Reinforcement Learning-Intelligent Adaptive Curiosity) to overcome this limitation and ensure an efficient exploration strategy for mobile robots.

A generic approach tested on several platforms and scenarios

Our methods aims to be generic and applicable to a large number of active vision systems. We therefore conducted our experiments on two types of robotics platforms, as well as on publicly available datasets.

The first robotic platform, called BioVision, was developed at Thales. This platform consists in an anthropomorphic head constituted with foveal and contextual cameras. During the experiments, BioVision is placed on a table, but has the ability to control the orientation of the foveal and contextual cameras to explore its environment.

The second platform is a mobile robot equipped with a Kinect camera and a laser range finder. The laser range finder provides the robot with a localization capability, and is used in the construction of the mapping of the environment surrounding the robot. This robot is then able to explore the environment by displacements across a room or a building, while keeping a track of its trajectory and actual position.

Our last contribution is then the construction, recording and preparation of several datasets to qualitatively and quantitatively evaluate the performance of our systems. In addition, to be more easily compared with other techniques, we also used publicly available datasets.

Publications and patents

This work has led to several publications and one patent that are listed below: National conferences: [START_REF] Craye | Apprentissage incrémental de la saillance visuelle pour des applications robotique[END_REF] Craye, C., Filliat, D. & Goudou, J.F. (2015). Apprentissage incrémental de la saillance visuelle pour des applications robotique. Accepted for oral presentation.

International conferences:

[20] Craye, C., Filliat, D. & Goudou, J.F. (2015).Exploration Strategies for Incremental Learning of Object-Based Visual Saliency. ICDL-EPIROB-The fifth joint IEEE International Conference on Development and Learning and on Epigenetic Robotics, At Providence, RI, USA. Accepted for oral presentation (±25% acceptance rate).

[21] Craye, C., Filliat, D. & Goudou, J.F. (2016).Environment Exploration for Object-Based Visual Saliency Learning. Robotics and Automaton (ICRA), 2016 IEEE International Conference on, At Stockholm, Sweden. Accepted for interactive session (33% acceptance rate) [START_REF]Rl-iac: An exploration policy for online saliency learning on an autonomous mobile robot[END_REF] Craye, C., Filliat, D. & Goudou, J.F. (2016). RL-IAC: An autonomous exploration policy for online saliency learning on a mobile robot. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Accepted for oral presentation (8% acceptance rate) [

Overview of the thesis

This manuscript is divided in two parts, tackling on the one hand the incremental learning of visual saliency, and, on the other hand, the intrinsically motivated exploration of the environment. Each part of the manuscript contains a literature review, a technical description of the proposed approach, and an experimental evaluation. Following this introduction, a chapter presents the proposed experimental setups that are to be used in the two parts of the manuscript. Lastly, a concluding chapters provides a discussion and some perspectives to this work.

Chapter 2 -Experimental setup This chapter details the two robotics platforms used to carry out our experiments. We provide technical details on both platforms as well as the required software to run our implementations. Additional information on the publicly available datasets used in this work are also reported. Lastly, we explain the datasets recording, construction and annotations procedures used for evaluation.

Part I -Incremental learning of visual saliency

Chapter 3 -Visual attention, object localization and learning This bibliographic chapter covers the state-of-the-art of the fields related to our proposed approach. We more specifically study the literature on visual attention, object localization in computer vision and machine learning techniques. Although very different, those approaches are intertwined and necessary to fully explain the background of our technique.

Chapter 4 -Proposed approach

We propose in this chapter a detailed description of the algorithms and software components developed to learn a model of saliency on-the-fly during exploration. With this approach, the robot is able to generate saliency maps specialized for the environment the robot is exploring, for the task the robot is asked to accomplish, while remaining flexible to any change or novelty in the environment.

Chapter 5 -Experimental results

We propose in this chapter to evaluate the performance, behaviors and limitations of the incremental saliency learning approach. To this end, we rely on four datasets, and conduct a set of experiment to evaluate different aspects of the method. In a first part, we study the incremental and self-supervised aspect of the method. Then, we study the performance of our approach once a stable model of saliency has been learned. Our evaluations suggests that we outperform state-ofthe-art methods when evaluated in the environment the model was trained for.

Part II -Intrinsically motivated exploration Chapter 6 -Environment exploration on an autonomous mobile robot

This chapter provides an overview of the exploration problem in autonomous robotics under two different fields of robotics that are mobile robotics and developmental robotics. In mobile robotics, exploration is typically seen as a problem of path planning across the environment by maximizing different costs, while in developmental robotics, the stress is put on the learning and development of skills and knowledge based on motivations. In spite of highlighting two different ways for tackling the problem of exploration, we also demonstrate a strong overlap between those fields.

Chapter 7 -Proposed approach

In the scope of an autonomous, life-long exploration and learning, the robot must be equipped with an exploration strategy able to drive the robot's displacements. This one should allow the robot to collect interesting examples, while avoiding problems such as catastrophic forgetting. The IAC algorithm is a possible option for obtaining such a behavior. In the meanwhile, exploration should be organized and efficient, so that the time spent traveling across the environment to reach goals is acceptable. The proposed algorithm, RL-IAC, is a possible way of tackling this issue. We then describe in this chapter the technical details of the implementation of IAC and RL-IAC for the task of saliency learning.

Chapter 8 -Experimental results

We propose in this chapter a numerical and qualitative evaluation of the approach proposed in the previous chapter. We then exhibit the behavior, advantages and drawbacks of both IAC and RL-IAC for the task of saliency learning. For IAC, the stress is put on the behavior of the robot during exploration, and its ability to methodically explore the environment. For RL-IAC, we emphasize the influence of some parameters of the systems and demonstrate the efficiency of the approach versus other exploration techniques.

Chapter 9 -Discussion and perspectives This concluding chapter wraps up the work presented in this manuscript and provides a discussion on the main components of this work. We then propose future work directions and perspectives that would be worth investigating in this context.

Chapter 2

Experimental setup

Introduction

This chapter details the methodology applied to conduct our experiments. We first present the available robotic platforms and additional software components. Then, we describe the procedure used to produce annotated datasets and to simulate the displacement of the robot in semi-simulated environments.

Indeed, our methods aims to be generic and applicable to a large number of active vision systems, as long as it is possible to control the viewpoint of the visual input. We essentially conducted our experiments on two types of robotics platforms, one relying on the foveal vision principle, the other one being a mobile robot. To further enhance and validate our results, some publicly available datasets were used to make our methods more easily compared with state-of-the-art.

We also relied on a semi-simulated framework to deeply investigate the performance of exploration strategies available to the robot. With this simulation procedure, we could produce thorough and repeatable evaluations.

Robotics platforms and software

We present in this section a technical description of the three robotics platforms used in the scope of this work. We also mention required libraries and software to make our work usable on those platforms.

Robots

BioVision v1

The BioVision platform was developed in the scope of a bio-mimetics project at Thales, aiming to study and exhibit some aspects of the human vision. BioVision consists in a bio-mimetic head imitating the human vision system to develop new strategies for learning, recognition, or tracking. The platform first takes into account the properties of a human eye by relying on both foveal and contextual fields of view. Then, the visual cortex is modeled by deep convolutional neural networks performing object recognition.

The first version of the platform is an anthropomorphic robotic head presented in Figure 2.1. This head is composed with a Smartek vision camera1 having a large field of view of 54 • (the peripheral camera or contextual camera), and two Tokina cameras2 with a very narrow field of view of 2 of those cameras offers a binocular vision on the narrow fields as well as a foveallike vision by having a much higher resolution at the center of the field of view.

The vergence of the two foveal cameras is done automatically based on a telemeter measuring the distance from the object at the center of the visual field. The head is also composed with pan and tilt rotation axes 3 to pilot the orientation of the cameras. In this setup, the three cameras move along the same pan-tilt angles and are oriented the same way depending on the pan-tilt position. The object recognition system described in Section 2.2.2 was trained from images recorded on this setup.

BioVision v2

The second version of the head aims to be lighter and faster, so that it can be mounted and operate on a mobile robot. The BioVision2 platform is presented in Figure 2.2. The foveal cameras are replaced by a single EXG50 Baumer camera 4with a narrow field of view of 5 • . An Optotune EL-10-30TC liquid lens5 is used on this camera to adjust the focus within a few milliseconds. An RGB-D Asus Xtion Pro live camera 6 plays the role of the peripheral camera with a field of view of 57 • horizontally. The pan-tilt motors are now piezoelectric ones 7 , and the setup is around 2kg so that it can be easily mounted and powered on a mobile robot.

The results presented in this manuscript are obtained on the second version, without using the TurtleBot motion capacities. In our experimental setups, the pantilt and camera components of the robot are simply placed on a table. Actions performed by the robot are then eye saccades, used to direct the foveal camera to relevant portions of the visual field, in order to get a higher resolution and a better representation.

Heat sink

Liquid lens Foveal camera (Baumer) Peripheral camera (Xtion) Pan-tilt motors Arduino

Mobile robot

The second platform is a mobile robot Pioneer 3-DX8 as presented in Figure 2.3. It can move at a speed of 1.6 meters per second. Moreover, the robot is equipped with a Kinect RGB-D camera 9 . The Kinect is mounted on the robot about 1 meter from the ground. It faces the front of the robot and is slightly tilted downward. The depth component of the Kinect does not provide reliable measurements if an obstacle is closer than 0.8 meters or farther than 4 meters. To overcome the limitations of the Kinect in obstacle detection, an Hokuyo UTM-30LX laser range finder 10 , having an accuracy of a few centimeters at 10 meters is also mounted on the robot. The wide field of view of 270 • makes it well-suited to precisely scan the surroundings of the robot.

In this setup, the actions performed by the robot are displacements through the building. The displacements modify the perception of the robot by showing areas of the environment at different points of view.

Software

Object recognition

For the experiments carried out on the BioVision platform, we relied on an object classification system applied to images of the foveal camera.

The object classifier is based on convolutional neural networks, and more precisely, an Alexnet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] architecture pre-trained on the ImageNet dataset [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. We replaced the last layer of the network to identify 8 different categories of objects, and a last class called "other", containing any other types of elements. To train the network, we constituted a dataset of objects captured from the foveal camera of BioVision1. In total, 8 different objects were placed on a white table and recorded at thousands of different points of view (see Figure 2.4). The additional class "other" was collected similarly with various kinds of background surrounding the objects. In total, around 10 000 images were collected this way.

The network was finally fine-tuned with the collected dataset, and an accuracy of 99 % was measured on the validation set. Note that the dataset construction and training was done months before the experiments presented in this work, recorded on another platform than the one used later on (BioVision2). Therefore, this object classifier is not seen as a part of our system but as an additional component.

Simultaneous localization and mapping

Our work strongly relies on the position of the robot within the environment. For BioVision, the robot remains on a table and the only useful position is the orientation of the pan and tilt axes. However, an efficient algorithm is required to determine the position and orientation of the mobile robots within the environment.

To this end, we rely on a Simultaneous Localization And Mapping (SLAM) algorithm called Hector mapping [START_REF] Kohlbrecher | A flexible and scalable slam system with full 3d motion estimation[END_REF]. Hector mapping is part of the ROS library and is easily interfaced with the Hokuyo laser range finder and odometry measurement of our Pioneer robot. The algorithm is able to fuse those two input cues to provide a 2D pose estimation of the robot, while building a map of the surrounding obstacles. The implementation provides a reliable estimation of the position in most indoor scenarios and is widely used by the mobile robotics community.

Operating system and additional libraries

Our implementation is written in C++ and has been tested on Ubuntu 14.04 with an Intel Core i3-3240, CPU at 3.4GHz quadcore processor. The implementation requires OpenCV [START_REF] Bradski | Learning opencv: Computer vision with the opencv library[END_REF], PCL [START_REF] Rusu | 3d is here: Point cloud library (pcl)[END_REF] and Caffe [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF]. Wrapper nodes have also been implemented to make our work compatible with the ROS framework [START_REF] Quigley | Ros: An open-source robot operating system[END_REF].

Biovision2 requires ROS Indigo for pan-tilt controls and input acquisition. Similarly, the Pioneer robot is fully interfaced with the ROS Hydro framework for controls and acquisitions.

Datasets and simulations

To efficiently evaluate our systems, we have used publicly available datasets, and created our own from the available robotics platforms. We describe in this section the four datasets used in our evaluation, including data collection and annotation. Table 2.1 summarizes the main components of each dataset.

BioVision dataset

Our first dataset, denoted as the BioVision dataset, is recorded on the BioVision2 robot.

Dataset recording

During the sequence acquisition, BioVision was put at the extremity of a two meterslength table, looking mainly at objects at the other end. The table and adjacent wall are all white, so that objects on the table are likely to be naturally salient in this local context. However, the background of the room is largely visible by BioVision, and contains a lot of distractors.

To make the experiment consistent with the object classifier described in Section 2.2.2, we put the same list of objects on the table, sometimes with additional distractors (such as pens).

Our dataset is composed with 10 sequences recorded at different times of the day. For each sequence, we placed an arbitrary configuration of objects on the table, either known by the object classifier or not. We then let BioVision observe the environment for about three minutes, meaning that the robot was randomly selecting a pan-tilt position, reaching this position, acquiring both contextual and foveal images at this position, and selecting a new one. In total, around 2000 observations (an observation is defined here as a pair of foveal and contextual views) were collected this way. Figure 2.5 presents a few samples of contextual and foveal images of the dataset.

Annotations

To evaluate our results from this dataset, we constituted a training and a testing set. The training set is composed with the observations taken from 7 of the sequences, and the testing set is composed with the 3 last sequences. In those three sequences, we randomly selected 150 samples that we annotated. To do so, we manually segmented the contextual images of the selected samples, to create ground truth masks of salient and non salient elements. Figure 2.6 shows an example of contextual images and associated ground truth masks.

ENSTA dataset

The second dataset, denoted as the ENSTA dataset, was collected from the pioneer 3DX mobile robot. The robot is able to record RGB-D streams from the Kinect camera mounted on the robot.

Dataset recording

To build the dataset, we manually controlled the robot in the ENSTA building in order to visit corridors, a laboratory, a hall and offices. The sequence was recorded so as to avoid as much as possible the presence of humans in the images. This way, we recorded a 15 minutes-length video sequence at 20Hz in which the robot was moving at the average speed of 0.5m/s. During the sequence, the robot recorded a large variety of views and lightning conditions across the building and different rooms. Figure 2.7 presents an overview of the robot trajectory during the sequence as well as a few recorded viewpoints in each room.

We also collected at different times of the year 6 smaller sequences of the same area to make sure that illumination and object position in the rooms were not the same. These additional sequences were used for evaluation only.

Annotations

We separated again our dataset into a training and an evaluation set. The training set was not annotated, but we selected a subset of 100 images to constitute the evaluation set.

We annotated the frames with a similar procedure than for the BioVision dataset, except that the ground truth masks were drawn so that chairs, desks or tables constituted the salient elements.

Lastly, we annotate the same 100 images in terms of bounding boxes around salient objects. Figure 2.8 shows a subset of images annotated with ground truth masks and bounding boxes annotated this way.

Lab

Washington RGB-D scenes dataset

The third dataset is constructed from a publicly available dataset called the RGB-D scenes dataset, that is part of the Washington dataset [START_REF] Lai | A large-scale hierarchical multi-view rgb-d object dataset[END_REF].

Original dataset

This dataset is composed with 8 video sequences of indoor scenes containing everydaylife objects on tabletop surfaces. Sequences are recorded with an RGB-D sensor and contain the RGB and depth map separately. Figure 2.9 provides a sample RGB-D input for each sequence and the associated annotations.

The recording sensor is held by a human user during the sequence. Therefore, the point of view is concentrated on the tabletop containing objects. Each sequence is roughly one minute length, and the point of view at which objects are observed is changing as the user moves in the room. The overall dataset contains around 1500 samples.

Each sequence is annotated by bounding boxes around objects registered in the database (cereal boxes, bowls, caps, etc...). However, some objects of the sequence are distractors (staplers, computers ...). They are objects that would be salient according to our definition, but they are not annotated by bounding boxes.

Annotations

The annotations provided by the original dataset are not exactly compatible with the way we would like to exploit them, so that we had to create our own once again.

Similar to the ENSTA dataset, we construct our own set of annotations by selecting a subset of 100 images from all of these scenes and manually construct ground truth masks and bounding boxes around objects on the tabletops (including the ones considered as distractors in the original dataset). These images were removed from the training set and kept for evaluation only. Figure 2.10 provides a subset of such annotations.

Ciptadi dataset

Our last dataset is again a publicly available one, called the Ciptadi dataset [START_REF] Ciptadi | An in depth view of saliency[END_REF]. This one was design to evaluate RGB-D saliency algorithms in indoor environments. We found this dataset particularly interesting as perfectly well-suited for our experimentation and evaluation, but also because this is one of the rare dataset designed for evaluating RGB-D-based saliency. This dataset shows some similarities with the RGB-D scenes dataset, while containing objects, rooms and viewpoints that RGB-D scenes does not. The Ciptadi dataset is composed with 80 images of indoor scenes, recorded in the Georgia Tech Aware Home by a Kinect camera mounted on a Willow Garage PR2 robot. Each image contains a ground truth mask that was manually annotated by a human user (see sample images in Figure 2.11). With this dataset, we then have an external and objective set of annotations to evaluate our method.

Simulated environments

Motivations

A strong limitation when using robots, especially in scenarios involving active learning, is the reproducibility of the experiments. If a single experiment requires the robot to explore a building for hours, the total number of possible trials is rather limited, and the efficiency of a method may be hard to analyze. For that reason, we also rely on simulations that can be run in parallel, without any particular user monitoring. This way, a large number of experimental results can be obtained, and a better evaluation and analysis is carried out.

Methodology

When required by the experiments, our method is then evaluated by constructing semi-simulated environments in which the robot can virtually navigate. We use the term semi-simulated, as real images taken from the aforementioned datasets are used, but actions taken by the robot are simulated.

More precisely, to make the robot virtually move in its environment, we add in our datasets a localization information, that associates a position with each available observation of the dataset (either an RGB-D frame, or foveal-contextual pair). These positions typically correspond to the positions estimated by a SLAM algorithm, or the pan and tilt orientations of the BioVision head. Then, by taking virtual actions, the robot reaches one of the available positions, and we estimate the time that should have been taken by the robot to reach this position in the real environment it has been recorded. Suppose for example that the robot is in position p 1 and virtually reaches position p 2 . The estimated time t p 1 ,p 2 for performing a displacement between p 1 and p 2 would be obtained by

t p 1 ,p 2 = V N i=1 (p 2 (i) -p 1 (i)) 2 (2.1)
N being the number of dimensions defining the robot's position and V being the average speed of the robot measured during the sequence recording. As a result, when the robot takes an action in our simulated environment, we just determine and keep track of the time to reach this position. We then consider that the robot has successfully reached the target, and process the associated observation as if the robot had really moved to that position.

Chapter 8 explains in deeper details the procedure for simulating displacement time in the scope of the robot's exploration.

Conclusion

In conclusion, we presented in this chapter the robotics platforms available to carry out our experiments, and the required hardware and software to construct and evaluate our techniques.

Four datasets that were used in our experimentation, either constructed from robot sequences recorded in our laboratories, or publicly available. We give a description of each dataset as well as their available annotations. For more clarity, the annotation process is to be further discussed in Chapter 5, after the incremental saliency learning method has been presented.

Lastly, we discussed the need to construct simulated environments to efficiently evaluate our approach, and provide an overview of the way we simulated the displacements of the robot. This simulation aspect is to be further explained in Chapter 8 after the intrinsically motivated exploration has been presented.

Part I

Incremental learning of visual saliency Chapter 3

Visual attention, object localization and learning

Introduction

This chapter is a literature review on the background fields our saliency learning technique is based upon. Although covering very different aspects of the problem, we aim to present the literature so as to make links between them.

The main background of this work is the one of visual attention from biological, computer vision and robotics points of view. We intend to describe the main aspects of visual attention as well as the biological inspiration in the artificial systems. We then precisely describe the literature regarding saliency maps and evaluation means. Lastly, we provide a list of applications and robotics platforms relying on visual attention.

Another field, not directly related to visual attention, but extremely active in computer vision is the one of object localization in natural images. We go through the literature on this field for two reasons. First, our system is built on tools developed in this field, such as superpixels, bounding box proposals, or binary segmentation. Second, the concept of saliency maps and object localization are largely intertwined, especially in the setup where we carry out experiments. Lastly, as the goal of our approach is mostly to learn a model able to generate saliency maps, we review approaches that learn models of visual attention, and we describe the main features and challenges of our learning constraints. This approach then provides the context of our work under a machine learning perspective.

Visual attention

Because our retina is composed with photo-receptors that are not homogeneously distributed, our brain needs to use a mechanism of selection aiming to prioritize input stimuli, and to examine them in a selective and sequential way. This attentive selection mechanism is probably due to evolution, in order to handle the high amount of information constantly surrounding us.

Generalities

Several literature review on the topic of visual attention define the main keywords and concepts. The work of Itti and Koch [START_REF] Itti | Computational modelling of visual attention[END_REF] is a pioneer in the field of computational visual attention by defining the main ideas of a visual attention system, while insisting on the importance of eye saccades. Henderson's review [START_REF] Henderson | Human gaze control during real-world scene perception[END_REF] is more focused on the gaze control at a computational and cortical level. Frintrop et al. [START_REF] Frintrop | Computational visual attention systems and their cognitive foundations: A survey[END_REF] have produced a literature review on visual attention by comparing computational and cognitive points of view. Lastly, the recent survey of Borji et al. [START_REF] Borji | State-of-the-art in visual attention modeling[END_REF] covers pretty exhaustively a large panel of methods and aspects of computational visual attention. In this section, we define a few important concepts and recalls regarding visual attention.

Attention, saliency and gaze

First, let us make a distinction between the three terms visual attention, visual saliency and gaze control that are sometimes mismatched. Visual attention is the most general term and represents the whole mechanism responsible for deciding where to look at.

Saliency is more related to the scene and to visual stimuli that are likely to catch the attention. In computer vision, saliency is usually represented by a saliency map. A saliency map is a grayscale (or heatmap) image where the intensity (or heat) depends on the interest of each pixel of the original image. In this document, we may represent saliency maps in both ways depending on the case. They have exactly the same meaning though.

Gaze is directly related to eye motion and represents the orientation of the eyes. Saliency is then related to external stimuli whereas gaze is intrinsic. The visual attention systems then coordinates both aspects so that gaze is directed towards salient area of the visual space.

Overt and covert attention

Directing attention towards a region of interest is usually associated with an eye movement, also called a saccade (represented as black lines in Figure 3.1). Once attention target has been reached, the eye stays focused for some time to analyze the corresponding visual input. This is called a fixation (represented as yellow dots in Figure 3.1). Lastly, the sequence of fixations and saccades necessary to examine a visual scene is called the scanpath.

Overt attention is then defined as the visual attention process that directs attention towards areas of interest for a deeper examination. This was first studied by Yarbus [START_REF] Yarbus | Eye movements during perception of complex objects[END_REF] in the 60's. However, we are still able to examine elements at the periphery without moving the eyes. We call this phenomenon covert attention. Covert attention usually comes before overt attention as it makes a pre-selection of interesting targets and potentially moves the gaze toward these items. Sometimes, covert attention is enough and does not require a visual focus of attention, for example for simple manipulation tasks. Conversely, it may happen that eye movements are not related to spontaneous covert attention, typically in the case of learned saccades, where the frequency of saccades is much higher than the cognitive demand for covert attention.

Bottom-up and top-down saliency

A key concept in visual attention is the difference between bottom-up and top-down visual attention. Bottom-up visual attention is related to visual stimuli of the environment. The gaze is directed towards the item that produces the highest stimuli at the retinal or visual cortex level. This occurs when observing a scene with no specific goal: our attention is then simply guided by surrounding elements. On the contrary, top-down attention depends on a goal and has been highlighted by Yarbus experiments [START_REF] Yarbus | Eye movements during perception of complex objects[END_REF] (see Figure 3.2). In this experiment, several subjects were asked to look at an image to answer a specific question (estimate the age of the people, remember their position in the room, etc...). Yarbus discovered that the subjects' gaze was drastically different depending on the question, suggesting that attention was depending on the task of the user. Top-down attention is both more demanding and more specialized than bottom-up attention, as it is used in a more conscious and voluntary perspective. It then requires higher cognitive loads.

Stimuli, space, and object-based saliency

The question of whether our attention is directed towards regions of the space, visual features, or directly objects is an open debate. Most studies on visual attention at neurobiologic level consider visual attention as being region-based, but other works have also shown evidences of the other types of attention. Most research today agree that those theories are not incompatible, and that visual attention can, depending on the context, alternate between those models.

Computational modeling of visual attention

Dozens to hundreds of visual attention methods have been reported in the literature, for a large variety of applications. Borji and Itti [START_REF] Borji | State-of-the-art in visual attention modeling[END_REF] have identified more than 70 of them, without pretending to be fully exhaustive. However, most of these models use a similar core architecture, presented in Figure 3.3. Visual features are first extracted from the input image, then combined to produce a saliency map. We will describe some of these approaches in the next section. The global maximum of this map is then determined and visual attention is finally directed towards this area. The focus of attention can be expressed in many different ways depending on the system, either by cropping the area directly in the image, or by a saccade centering the gaze at the focus of attention.

In the case of a robotics application, the difference between visual attention and saliency is clear, as a new focus of attention usually results in a physical action (possibly a saccade). However, the distinction is much thinner when still images are considered.

Saliency maps in computer vision

In this section, we examine the different approaches for producing and evaluating saliency maps in computer vision.

Many definitions of saliency

Among the large number of methods for generating saliency maps, there is no single consensus to define what a salient item is and what should be enhanced in the input image. To make the classification easier, Borji et al. [START_REF] Borji | State-of-the-art in visual attention modeling[END_REF] proposed a taxonomy of the saliency models.

Taxonomy: The cognitive models [START_REF] Itti | A model of saliency-based visual attention for rapid scene analysis[END_REF], [START_REF] Le Meur | A coherent computational approach to model bottom-up visual attention[END_REF] try to define saliency based on our knowledge of the human visual system. Models are then developed by following the experimental findings on the human neurological responses to visual stimuli.

Other methods define saliency areas as being "unusual" and use mathematical properties of the image to enhance them. In this category, the spectral models rely for example on spectral residuals [START_REF] Hou | Saliency detection: A spectral residual approach[END_REF] or image signature [START_REF] Hou | Image signature: Highlighting sparse salient regions[END_REF], while graphical models such as the one of Harel et al. [START_REF] Harel | Graph-based visual saliency[END_REF] have defined a saliency approach (GBVS) based on markov random fields and a definition of unusual pixels in their neighborhood. Other approaches defined as information theoretic models rather defined saliency as being "informative". For instance, Bruce et al. [START_REF] Bruce | Attention based on information maximization[END_REF] have used a criterion based on Shannon self information to define saliency. Lastly the pattern classification models [START_REF] Kruthiventi | Deepfix: A fully convolutional neural network for predicting human eye fixations[END_REF], [START_REF] Borji | Boosting bottom-up and top-down visual features for saliency estimation[END_REF] rely on machine learning technique to predict the position of salient items. Our own method of saliency could be classified in this last category.

Top-down and bottom-up saliency: Similar to visual attention, saliency can also be defined from a bottom-up or top-down point of view. Depending on the task and application, saliency maps may enhance items that are naturally salient in their context, or important for a given task. There is no single definition of what needs to be enhanced in a top-down context as the task is different for each application. However, in a robotics context and with no further precision, top-down saliency usually mean "object-oriented". We follow such a definition in our work, by defining salient elements as surrounding objects the robot can potentially interact with. The pure top-down approaches are not as common as bottom-up ones, and a few of them suggest to focus on combining bottom-up and top-down cues [START_REF] Rasolzadeh | An active vision system for detecting, fixating and manipulating objects in the real world[END_REF], [START_REF] Frintrop | Vocus: A visual attention system for object detection and goal-directed search[END_REF].

Stimuli: Saliency maps are also classified in terms of stimuli. Depending on the purpose they are used for, they can enhance positions where subjects are likely to make a fixation [START_REF] Itti | Computational modelling of visual attention[END_REF], or enhance a whole object or a whole area [START_REF] Orabona | A proto-object based visual attention model[END_REF]. The notion of scale is also important depending on what is supposed to be salient. For example, depending on how big a subject is in an image, saliency can either be on the whole subject [START_REF] Frintrop | Traditional saliency reloaded: A good old model in new shape[END_REF], or on areas constituting the subject [START_REF] Borji | State-of-the-art in visual attention modeling[END_REF], [START_REF] Paletta | Q-learning of sequential attention for visual object recognition from informative local descriptors[END_REF]. Our work considers saliency at the scale of an objects and enhances the whole area of the image containing those objects.

Input: One could also think of defining saliency based on the nature of the input stream. Although a large number of saliency approach consider only the RGB component of still images, a natural extension is to integrate temporal or spatiotemporal components. In that regard, the optical flow has been used by Le Meur et al. [START_REF] Le Meur | Predicting visual fixations on video based on lowlevel visual features[END_REF]. When using an RGB-D camera, the depth integration is also a natural extension, either using an early [START_REF] Peng | Rgbd salient object detection: A benchmark and algorithms[END_REF] or late fusion approach [START_REF] García | Saliency-based object discovery on rgb-d data with a late-fusion approach[END_REF] between RGB and depth components. In the RGB-D approaches, the geometrical nature of depth signal is better suited to detect salient objects rather than salient stimuli. Lastly, saliency can also be detected with event-driven cameras, at a much higher frequency than any RGB-based algorithm [START_REF] Rea | Event-driven visual attention for the humanoid robot icub[END_REF]. Our saliency maps aim to be generated from RGB images only, but we also rely on geometrical information from depth maps to learn our model of saliency.

Saccadic models: Another family of techniques proposes to study saccadic models rather than just saliency maps. Instead of considering the only spatial component of visual attention, saccadic models also study the temporal aspect of exploration by predicting plausible visual scanpaths rather than only salient locations [START_REF] Najemnik | Optimal eye movement strategies in visual search[END_REF], [START_REF] Wang | Simulating human saccadic scanpaths on natural images[END_REF]. These kind of approaches both rely on saliency models to predict fixations, but are also able to produce saliency maps based on the simulated scanpaths [START_REF] Tavakoli | Stochastic bottom-up fixation prediction and saccade generation[END_REF]. The scanpath-based saliency maps are potentially more accurate, as additional stochastic components related to fixation transitions are integrated in the model (for exaple, vertical saccades are much more likely than oblique ones) [START_REF] Meur | Saccadic model of eye movements for free-viewing condition[END_REF], [START_REF] Meur | Introducing context-dependent and spatially-variant viewing biases in saccadic models[END_REF].

Computing a saliency map

The most famous model of saliency is probably the one of Itti and Koch [START_REF] Itti | A model of saliency-based visual attention for rapid scene analysis[END_REF] developed at the end of the 90's. Although many updates and improvements have been added to this method (for example, by adapting the scales of receptive fields [START_REF] Frintrop | Vocus: A visual attention system for object detection and goal-directed search[END_REF], or by integrating face detectors and weights related to human fixations [START_REF] Zhao | Learning a saliency map using fixated locations in natural scenes[END_REF]), this one has the advantage of being fast, simple, and based upon biologically plausible mechanisms inspired by retinal and cortical processing of the visual information. Figure 3.4 presents the general approach for this method. Here, visual features are color, contrast and orientation-based. They are combined together by centersurround differences at different scales. This model is inspired by the receptive fields in the retina and visual cortex. A total of 42 maps are then generated and combined to create the saliency map. The basic idea of feature map extraction followed by a feature combination has inspired a large number of approaches in the literature.

Feature extraction: Saliency maps are usually obtained from simple visual features, applied on the whole input image. Most of these features are designed to be applied on RGB images, but some are also based on motion in the case of videos streams [START_REF] Le Meur | Predicting visual fixations on video based on lowlevel visual features[END_REF], [START_REF] Vijayakumar | Overt visual attention for a humanoid robot[END_REF], or exploit the depth component of RGB-D sensors [START_REF] Ciptadi | An in depth view of saliency[END_REF], [START_REF] Peng | Rgbd salient object detection: A benchmark and algorithms[END_REF]. Features may be based upon center-surround differences [START_REF] Itti | A model of saliency-based visual attention for rapid scene analysis[END_REF], [START_REF] Frintrop | Vocus: A visual attention system for object detection and goal-directed search[END_REF], integrating sometimes spatio-temporal components such as optical flow [START_REF] Harel | Graph-based visual saliency[END_REF], [START_REF] Le Meur | Predicting visual fixations on video based on lowlevel visual features[END_REF]. Classical computer vision filters have also been proposed to produce saliency maps, such as HOGs1 [START_REF] Xie | Unifying visual saliency with hog feature learning for traffic sign detection[END_REF] , wavelets [START_REF] Imamoglu | A saliency detection model using low-level features based on wavelet transform[END_REF], SIFT 2 [START_REF] Gao | Attention model based sift keypoints filtration for image retrieval[END_REF] or gist [START_REF] Torralba | Modeling global scene factors in attention[END_REF]. Features can also be based on thresholds in the color space [START_REF] Cheng | Global contrast based salient region detection[END_REF], [START_REF] Zhang | Saliency detection: A boolean map approach[END_REF], statistical descriptors such as covariance [START_REF] Erdem | Visual saliency estimation by nonlinearly integrating features using region covariances[END_REF] or Shannon information [START_REF] Bruce | Attention based on information maximization[END_REF]. Similarly, spectral methods use transformation spaces of the image such as DFT 3 , DCT 4 to obtain visual features [START_REF] Hou | Saliency detection: A spectral residual approach[END_REF], [START_REF] Hou | Image signature: Highlighting sparse salient regions[END_REF]. In the last few years, a new process of feature extraction has been proposed based on deep convolutional neural network filers. These kind of features have also been successfully applied to the generation of saliency maps [START_REF] Zhou | Learning Deep Features for Discriminative Localization[END_REF], [START_REF] Li | Visual saliency based on multiscale deep features[END_REF]. We propose in our work three types of visual features that are center surround-based for the first two techniques, and deep feature-based for the third one.

Feature combination: Once extracted, the visual features are combined to generate a saliency map. By default, a simple summation or max-pooling approach can be enough [START_REF] Itti | A model of saliency-based visual attention for rapid scene analysis[END_REF], but smarter strategies are commonly used to improve the performances. Some approaches use statistical approaches to build meta descriptors in order to get a better feature combination scheme. Thus, Erdem et al. [START_REF] Erdem | Visual saliency estimation by nonlinearly integrating features using region covariances[END_REF] have used a set of descriptors based on region covariance from basic features in order to improve saliency prediction. Bayesian strategies that aim to integrate prior and posterior knowledge to improve the feature maps combinations have also been proposed in the literature. For instance, Klein et al. [START_REF] Klein | Salient pattern detection using w 2 on multivariate normal distributions[END_REF] have used Itti and Koch center-surround saliency maps to compute saliency estimators from local statistics. In the perspective of optimizing saliency to get results as close as possible to human saliency, Zhao and Koch [START_REF] Zhao | Learning a saliency map using fixated locations in natural scenes[END_REF] have attributed optimal weights to each feature based on statistics obtained from human fixations on a large number of images. Li et al. [START_REF] Li | Visual saliency with statistical priors[END_REF] have proposed an improvement for basic saliency maps based on missed salient areas retrieval and false distractors removal using co-occurrence patterns. Lastly, pattern recognition models combine the extracted features by the mean of a classifier, thus creating a non-linear combination scheme [START_REF] Borji | Boosting bottom-up and top-down visual features for saliency estimation[END_REF], [START_REF] Judd | Learning to predict where humans look[END_REF], [START_REF] Kavak | Visual saliency estimation by integrating features using multiple kernel learning[END_REF]. This classifier approach is the one used in our technique.

Top-down modulation: Feature combination is also used to differentiate bottomup and top-down approaches. As bottom-up approaches are used for an openended exploration of the visual environment, they do not require a combination approach able to bias features against each other. When using top-down visual attention to specialize the attention towards a task of a certain type of stimuli, the features have this time to be combined accordingly. The dedicated term for this kind of approach is the top-down modulation. Such modulation can be obtained by exciting visual features related to the objects of interest [START_REF] Hamker | The emergence of attention by population-based inference and its role in distributed processing and cognitive control of vision[END_REF], by inhibiting irrelevant regions [START_REF] Choi | Biologically motivated visual attention system using bottom-up saliency map and top-down inhibition[END_REF], [START_REF] Tsotsos | Modeling visual attention via selective tuning[END_REF], or by mixing those two approaches [START_REF] Rasolzadeh | An active vision system for detecting, fixating and manipulating objects in the real world[END_REF], [START_REF] Zhao | Learning a saliency map using fixated locations in natural scenes[END_REF] with learning.

Top-down cues: To further improve the performance of the saliency, many approaches integrate additional cues that are nothing more than object detectors. This integration may make approaches better responsive to human shapes, by integrating for example detectors for face [START_REF] Zhao | Learning a saliency map using fixated locations in natural scenes[END_REF], [START_REF] Cerf | Predicting human gaze using low-level saliency combined with face detection[END_REF], skin [START_REF] Li | Saliency model-based face segmentation and tracking in head-andshoulder video sequences[END_REF] or cars [START_REF] Goferman | Context-aware saliency detection[END_REF]. Although not being in the philosophy of pure bottom-up approaches, integrating these cues is still consistent with the idea that visual saliency is partly learned from what we commonly see interact with in our everyday life.

Post-processing: The notion of stimuli-base and object-based visual saliency also influences the way saliency is obtained. Usually, a post-processing stage is enough to turn a stimuli-based saliency map into an object-based one. For stimulibased, Gaussian blur are often applied to better fit the statistical distribution of the fixations [START_REF] Bruce | Attention based on information maximization[END_REF], [START_REF] Erdem | Visual saliency estimation by nonlinearly integrating features using region covariances[END_REF]. Object-based saliency is rather obtained by looking at homogeneous colors in contrast with their environment [START_REF] Cheng | Global contrast based salient region detection[END_REF], [START_REF] Zhang | Saliency detection: A boolean map approach[END_REF], or combining saliency with segmentation techniques such as superpixels [START_REF] Zhao | Saliency detection by multi-context deep learning[END_REF] or mean-shift [START_REF] Frintrop | Traditional saliency reloaded: A good old model in new shape[END_REF]. Figure 3.5 shows a comparison of the same saliency method using the stimuli-based and object-based post-processing. Note that object-based saliency is tightly bounded with the notion of object detection or object segmentation, as the output saliency provides a rough estimation of object boundaries. In that regard, our method lies in the category of object-based saliency maps, and we propose a post-processing step to recover the shape of the objects when the produced saliency map is not clear enough.

Connectionist methods: An alternative to the filter-combination scheme is the fully-connectionist approach. Saliency is learned in this context using neural networks techniques and annotated datasets. The connectionist approach proposes to perform filtering and feature combination of an image within the same network, thus following a more biologically plausible scheme. Nevertheless, early approaches of connectionist models [START_REF] Humphreys | Search via recursive rejection (serr): A connectionist model of visual search[END_REF] were found to be less efficient and harder to design than the filter-based ones. This trend has been inverted with the recent breakthroughs in deep learning, who paved the way for a new paradigm about how to generate saliency maps. Similar to many other fields of computer vision, deep convolutional networks (or CNNs) have reported the best accuracy on many saliency benchmarks [START_REF] Kruthiventi | Deepfix: A fully convolutional neural network for predicting human eye fixations[END_REF], [START_REF] Kümmerer | Deep gaze i: Boosting saliency prediction with feature maps trained on imagenet[END_REF]. Two types of supervising signals can be used to train such classifiers: first, on saliency datasets, annotated either in terms of fixations [START_REF] Kruthiventi | Deepfix: A fully convolutional neural network for predicting human eye fixations[END_REF], [START_REF] Kümmerer | Deep gaze i: Boosting saliency prediction with feature maps trained on imagenet[END_REF], or segmented objects [START_REF] Zhao | Saliency detection by multi-context deep learning[END_REF]. Second, a simple set of object detectors can suffice to train a descent saliency enhancer [START_REF] Zhou | Learning Deep Features for Discriminative Localization[END_REF], [START_REF] Oquab | Is object localization for free?-weakly-supervised learning with convolutional neural networks[END_REF]. This approach is particularly interesting, as they can be learned on a weakly supervised basis, using as a training signal only picture of salient objects. The localization aspect is then learned for free. However, this approach has two counterparts: first, the model is specialized for objects in the dataset rather than learning a pure bottom-up saliency. Second, the method does not detect the object, but rather what make the object discriminant from other elements. The limit between saliency and object localization is very thin in this type of approach. We do not propose in our work a fully connectionnist approach, but the influence of this new paradigm is clearly present in the way we design our technique.

Evaluation and datasets

Given the abundance of existing saliency approaches, dedicated datasets have been created for benchmark and comparison. The proposed datasets usually take human annotation and behaviors as ground truth. Saliency maps are then computed from those datasets and compared with ground truth based on metrics. Although most of these datasets propose an evaluation on RGB still images only, some are also based on video sequences [START_REF] Le Meur | A coherent computational approach to model bottom-up visual attention[END_REF], [START_REF] Gitman | Semiautomatic visual-attention modeling and its application to video compression[END_REF], or RGB-D inputs [START_REF] Ciptadi | An in depth view of saliency[END_REF], [START_REF] Peng | Rgbd salient object detection: A benchmark and algorithms[END_REF]. Figure 3.6 shows an overview of the kind of images and ground truth found in the saliency benchmarks. [START_REF] Borji | Cat2000: A large scale fixation dataset for boosting saliency research[END_REF]. In the middle, manually segmented image [START_REF] Liu | Learning to detect a salient object[END_REF]. On the right, dataset avoiding the center bias [START_REF] Ciptadi | An in depth view of saliency[END_REF].

In general one should distinguish approaches trying to mimic human saliency models, and models for detecting salient objects. The former aims to determine where humans are going to look at whereas the latter seeks to extract the most salient objects, which are defined by well-defined boundaries. The goal of these models and the way we evaluate their performance are dramatically different.

Human fixations-based datasets: A first category of benchmark proposes to predict human fixations. The ground truth is then constructed with gaze trackers: several human subjects are asked to observe the same visual input for a certain time. The eye trajectory of each subject is then recorded and statistics are computed in order to determine the most frequent fixations and scanpaths. Popular datasets of this category include the MIT300 [START_REF] Judd | A benchmark of computational models of saliency to predict human fixations[END_REF], the CAT2000 [START_REF] Borji | Cat2000: A large scale fixation dataset for boosting saliency research[END_REF], or the VIU dataset [START_REF] Koehler | What do saliency models predict?[END_REF].

Manually segmented datasets: A second category uses manual annotations from a few users to crop or draw boundaries around important. Among popular datasets of this type, the Achanta's salient object dataset [START_REF] Liu | Learning to detect a salient object[END_REF], the MSRA [START_REF] Cheng | Global contrast based salient region detection[END_REF], or the PASCAL-S [START_REF] Li | The secrets of salient object segmentation[END_REF]. Depending on the benchmark, a single or several regions are isolated this way. This type of datasets may also be used for segmentation tasks, or for object detection. We are mainly interesting in this type of annotations, more consistent with our definition of saliency.

Center bias: Most datasets are biased by the position of salient objects. As images constituting the datasets are most of the time taken by humans, the most salient element tends to be at the center of the image (which corresponds to the center of attention of the viewer). In that regard, considering a simple Gaussian distribution at the center of the image is a good saliency baseline that may outperform much more complex saliency techniques [START_REF] Tseng | Quantifying center bias of observers in free viewing of dynamic natural scenes[END_REF]. To limit this bias, some datasets are constituted from free view-points inputs, where the most salient element does not have to be at the center, or may not even be on the image at all [START_REF] Ciptadi | An in depth view of saliency[END_REF], [START_REF] Bruce | Attention based on information maximization[END_REF]. The datasets used in our experiments are mostly taken from a robotic platform, and are not biased by the center bias.

Top-down datasets: Most of the proposed benchmarks are conceived and labeled for a bottom-up saliency evaluation, probably because a better consensus exists on the definition of bottom-up saliency. Nevertheless, a top-down evaluation is still possible on datasets specialized on a particular type of stimuli, such as the FiFA dataset containing lots of faces [START_REF] Cerf | Predicting human gaze using low-level saliency combined with face detection[END_REF], but to evaluate saliency on a specific task (which is our case), a dedicated dataset often needs to be created or adapted.

Common evaluation metrics: To compare an estimated saliency map with a ground truth, several metrics can be used (for more details about their exact computation, please refer to Riche et al. [START_REF] Riche | Saliency and human fixations: State-of-the-art and study of comparison metrics[END_REF] or Borji et al. [START_REF] Borji | Quantitative analysis of human-model agreement in visual saliency modeling: A comparative study[END_REF]). If the saliency map is seen as a probability distribution of eye fixations, metrics such as Kullback-Leibler divergence (DL), Histogram Intersection (HI) or the Earth Mover's Distance (EMD) are adequate, as they measure the distance between two distributions. When comparing the statistical relationship between estimate and ground truth, the linear correlation coefficients (CC) or the normalized scanpath saliency (NSS) are advised. Lastly, the comparison can be seen as a binary classification problem (salient and not salient) and is then evaluated in terms of false positive and false negative rates. The most common measure in this case is the area under curve (AUC) that measures the area under the ROC (Receiver Operating Characteristic) curve.

The AUC is probably the best-suited metrics for our evaluation and is the one used in our experiments. Further details on the computation of this metrics are provided in Chapter 5. The AUC has the nice property of being invariant by any monotonically increasing transformation. However, the result is represented in terms of true and false positives, which does not penalize false negatives in the estimated saliency.

Visual attention in robotics

Unlike classical computer vision, where the point of view is often chosen by a human operator and most likely centered at the object of interest, the viewpoint in robotics depends on the robot's position and movements. Is is therefore not -at least at first -centered at the interesting elements of the environment. Providing the robot with a good visual attention strategy is therefore critical for a good analysis of the environment. Aloimonos and Westelius' reference books constitute a good introductions to this field [START_REF] Aloimonos | Active vision[END_REF], [START_REF] Westelius | Focus of attention and gaze control for robot vision[END_REF].

In Section 3.2.2, we presented in details how to obtain a saliency map. We explain here how to exploit them in a visual attention setup. As presented in Figure 3.3, the saliency map is typically used to determine the most salient regions. The basic visual attention scheme follows a winner-takes-all strategy by determining the global maximum of this map, and focusing attention on this particular region. Then, in case of a static scene, the system should keep exploring and avoid as much as possible looking always in the same direction. For this reason, an inhibition of return is usually associated with the system. The inhibition of return has biological foundations [START_REF] Klein | Inhibition of return[END_REF] and simply consists in inhibiting the salient region that has just been the focus of attention for a certain period of time. This way, attention is naturally disengaged after examination, and the next most salient region is necessarily a different one.

Applications using visual attention

In most cases, when a robotics system is equipped with a visual sensor, a strategy of visual attention must be applied. We describe in this section the main applications relying on visual attention. For a more extensive review, please refer to Frintrop et al. [START_REF] Frintrop | Computational visual attention systems and their cognitive foundations: A survey[END_REF] or Bengum et al. [START_REF] Begum | Visual attention for robotic cognition: A survey[END_REF].

Finding areas of interest: First, visual attention and especially saliency have a wide variety of application when a need to localize regions of interest is required. Indeed, instead of examining the input stream with the same attention everywhere, a pre-selection of good candidates can significantly reduce the computational load. This principle is, for example, used for image segmentation, when segmentation seeds are required [START_REF] Ouerhani | Cue normalization schemes in saliency-based visual attention models[END_REF]. One may also think of image matching, when trying to find correspondences to associate images of the same area at different time (for example in SLAM 5 or loop closure detection systems [START_REF] Ho | Loop closure detection in slam by combining visual and spatial appearance[END_REF], [START_REF] Kim | Real-time visual slam for autonomous underwater hull inspection using visual saliency[END_REF]). In these approaches, salient regions are associated with descriptors or landmarks, able to provide a meaningful and exploitable representation of the region while limiting the area of research. Similarly, saliency is widely used in the tasks of navigation and semantic mapping, where the goal is to update a 3D or 2D grids containing objects of the environment [START_REF] Caron | Neural network fusion of color, depth and location for object instance recognition on a mobile robot[END_REF], [START_REF] Meger | Curious george: An attentive semantic robot[END_REF]- [START_REF] Gould | Peripheral-foveal vision for real-time object recognition and tracking in video[END_REF].

Object recognition: Object discovery or object recognition, especially for indoor robotics is also a very natural and useful application of visual attention. Early biologically inspired object recognition systems such as HMAX [START_REF] Serre | Robust object recognition with cortex-like mechanisms[END_REF] already integrated visual attention in the object recognition process, in order to select, modulate or suppress the most significant cues. Approaches combining both attention and recognition have known a renewed interest with CNNs able to simultaneously localize and identify object [START_REF] Oquab | Is object localization for free?-weakly-supervised learning with convolutional neural networks[END_REF], [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF]. A more sophisticated approach based on recurrent neural networks also proposed a sequential attention scheme to learn to localize, count and classify objects of a scene [START_REF] Eslami | Attend, infer, repeat: Fast scene understanding with generative models[END_REF]. More generally, the integration of a visual attention scheme for object recognition may be used at two levels: first, for isolating object candidates in the environment, followed by a displacement of the robot to improve the perception of the target candidate [START_REF] Rasolzadeh | An active vision system for detecting, fixating and manipulating objects in the real world[END_REF], [START_REF] Björkman | Vision in the real world: Finding, attending and recognizing objects[END_REF]. Second, for isolating areas within the object that may be relevant for its identification [START_REF] Paletta | Q-learning of sequential attention for visual object recognition from informative local descriptors[END_REF], [START_REF] Borji | Learning sequential visual attention control through dynamic state space discretization[END_REF].

Guiding the robot's actions: Lastly, attention systems may be used to guide the robot's action. The selective aspect of attention is particularly well-suited to perform a single task at a time. The natural action associated with visual attention is the displacement of the vision sensor towards the focus of interest. This action can either be a saccade [START_REF] Björkman | Vision in the real world: Finding, attending and recognizing objects[END_REF], [START_REF] Minut | A reinforcement learning model of selective visual attention[END_REF] or a displacement [START_REF] Cañas | An overt visual attention mechanism based on saliency dynamics[END_REF], [START_REF] Gould | Peripheral-foveal vision for real-time object recognition and tracking in video[END_REF], [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF]. In the scope of visual tracking, active vision system can perform smooth pursuit to systematically keep the target at the center of the field of view [START_REF] Mian | Realtime face detection and tracking using a single pan, tilt, zoom camera[END_REF], [START_REF] Frintrop | Most salient region tracking[END_REF]. Rather than moving the sensor to modify the perception of the robot, the attention may also be used as target to reach for robot grasping [START_REF] Kragic | Strategies for object manipulation using foveal and peripheral vision[END_REF], [START_REF] Metta | Early integration of vision and manipulation[END_REF] or in robot navigation [START_REF] Davison | Mobile robot localisation using active vision[END_REF], [START_REF] Lauri | Stochastic control for maximizing mutual information in active sensing[END_REF]. Finally, the field of human-robots interaction is also needing attention to create a context and common objects of interest at which both humans and robots should focus on [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF], [START_REF] Heidemann | Integrating context-free and contextdependent attentional mechanisms for gestural object reference[END_REF].

In our research, visual attention is not associated with a concrete application, but we exploit the mechanisms of saccade and mobile robot displacements to improve our model of saliency.

Indoor visual attention platforms

In this section, we mainly consider indoor robotics systems as it is the most related to our research.

Among indoor robotics platforms using visual attention, a distinction can be made between mobile robots able to move across their environment [START_REF] Cañas | An overt visual attention mechanism based on saliency dynamics[END_REF], [START_REF] Gould | Peripheral-foveal vision for real-time object recognition and tracking in video[END_REF], and static robots able to change their point of view with pan and tilt rotation axis of the camera. In this case, PTZ cameras [START_REF] Minut | A reinforcement learning model of selective visual attention[END_REF] or a set of cameras [START_REF] Rasolzadeh | An active vision system for detecting, fixating and manipulating objects in the real world[END_REF], [START_REF] Björkman | Vision in the real world: Finding, attending and recognizing objects[END_REF] are typically used to be able to modify the perception of some elements of the environment depending on their interest. Humanoid robots equipped with neck motion capacities, but placed in a human-robot interaction setup also lie in this category [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF], [START_REF] Holzbach | A fast and scalable system for visual attention, object based attention and object recognition for humanoid robots[END_REF]. Figure 3.7 presents a few of these setups. Foveated system: In foveated systems, peripheral cameras are integrated to localize objects of interest, and foveal cameras are used to get a high resolution representation of the target. Our first robotic platform BioVision, is typically classified among this type of systems. Bjorkman, Kragik et al. [START_REF] Bjorkman | Combination of foveal and peripheral vision for object recognition and pose estimation[END_REF], [START_REF] Rasolzadeh | An active vision system for detecting, fixating and manipulating objects in the real world[END_REF], [START_REF] Björkman | Vision in the real world: Finding, attending and recognizing objects[END_REF], [START_REF] Kragic | Strategies for object manipulation using foveal and peripheral vision[END_REF], [START_REF] Kragic | Vision for robotic object manipulation in domestic settings[END_REF] have worked on a system of four cameras (two foveal, two peripheral). Not only were the pairs of cameras alternating between foveal and peripheral visions, but also estimating the depth of the environment from stereoscopy. Other teams have simply used PTZ cameras to have an easier setup based on a single camera, while alternative foveal and peripheral view points. Minut et al. [START_REF] Minut | A reinforcement learning model of selective visual attention[END_REF] have used this kind of setup to jointly learn to localize and identify an object, while Kragic et al. [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF] have used a descriptor to adjust the zoom level and enable a better recognition. Gould et al. [START_REF] Gould | Peripheral-foveal vision for real-time object recognition and tracking in video[END_REF] as well as Canas et al. [START_REF] Cañas | An overt visual attention mechanism based on saliency dynamics[END_REF] have mounted a PTZ camera on a mobile platform able to explore their environment by moving in a room and identifying simple objects.

Mobile robots: When the robot moves in its environment, mechanisms similar to saccades and fixations are used. The shift of attention is associated with a displacement of the robot, either to put the object at the center of the visual frame [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF], or to simply get closer and have a better representation [START_REF] Cañas | An overt visual attention mechanism based on saliency dynamics[END_REF], [START_REF] Gould | Peripheral-foveal vision for real-time object recognition and tracking in video[END_REF]. However, two main differences have to be noted as compared to a non-mobile robot. First, the displacement modifies both the resolution of the object and the orientation at which the object is seen. This way, the object can be represented and identified based on a set of images rather than on a single one [START_REF] Thomas | Towards multiview object class detection[END_REF], [START_REF] Welke | Autonomous acquisition of visual multi-view object representations for object recognition on a humanoid robot[END_REF]. Second, the time for moving the robot to a new position may have to be considered, as the distance to the target can be significantly different, and the required time to reach it is roughly proportional to this distance [START_REF]Rl-iac: An exploration policy for online saliency learning on an autonomous mobile robot[END_REF], [START_REF] Oßwald | Speeding-up robot exploration by exploiting background information[END_REF]. Both of these problems are related to our research when considering experiments on our mobile robot platform.

Interactive perception: A last category could be the one of robots able to modify their perception of objects by manipulation and interaction. The field of interactive perception then proposes to use the robot's actions to directly interact with the object. When using a robotic arm, interactions such as pushing can help object segmentation given the concept of "what-moves-together-belongs-togeter" [START_REF] Kootstra | Exploring objects for recognition in the real word[END_REF]. More complex interactions may be used to learn object appearance, either with robotics arms able to grasp and manipulate objects [START_REF] Kragic | Strategies for object manipulation using foveal and peripheral vision[END_REF], [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF], [START_REF] Browatzki | Active object recognition on a humanoid robot[END_REF], possibly with additional tactile sensor [START_REF] Natale | From sensorimotor development to object perception[END_REF]. The presence of a human user assisting in the learning task is common in this case, either to place objects in the robot's hand [START_REF] Browatzki | Active object recognition on a humanoid robot[END_REF], [START_REF] Natale | From sensorimotor development to object perception[END_REF], or to interact with objects instead of the robot [START_REF] Lyubova | Developmental approach for interactive object discovery[END_REF], [START_REF] Ivaldi | Perception and human interaction for developmental learning of objects and affordances[END_REF]. The advantages of interactive learning versus simple robot displacements is the ability to add proprioceptive inputs in the perception of the objects. The robot then not only learns the visual appearance of objects, but also their physical properties.

Saliency in robotics

Among all saliency methods, only part of them are well-suited for robotics setups. First, methods taking advantage of the center-bias are no longer efficient when the viewpoint is chosen by a robot. Then, the saliency map should be light and fast enough to compute to be reactive to the robot's displacement while dealing with limited computational resources. Lastly, it should be adapted to the sensor and potentially exploit all the components they can offer (for example, exploiting depth if an RGB-D sensor is available).

Objectness and top-down saliency: In robotics, we have shown that saliency is often tightly bounded with objects, and therefore the concept of objectness and object localization. In that regard, pure bottom-up attention is not always wellsuited for robotics applications. Indeed, objects are not always intrinsically salient, and conversely, some elements of the environment can show salient property while being irrelevant for the robot's task. Top-down modulation is then a good approach to ensure the relevance of salient targets. This can be done by exploiting the context in which the object is supposed to be [START_REF] Minut | A reinforcement learning model of selective visual attention[END_REF], [START_REF] Kollar | Utilizing object-object and object-scene context when planning to find things[END_REF], [START_REF] Ali | Contextual object category recognition for rgb-d scene labeling[END_REF], or by enhancing features relative to the object [START_REF] Frintrop | Vocus: A visual attention system for object detection and goal-directed search[END_REF], [START_REF] Hamker | The emergence of attention by population-based inference and its role in distributed processing and cognitive control of vision[END_REF], inhibiting irrelevant ones [START_REF] Choi | Biologically motivated visual attention system using bottom-up saliency map and top-down inhibition[END_REF], [START_REF] Tsotsos | Modeling visual attention via selective tuning[END_REF], or both [START_REF] Rasolzadeh | An active vision system for detecting, fixating and manipulating objects in the real world[END_REF], [START_REF] Zhao | Learning a saliency map using fixated locations in natural scenes[END_REF] Proto-objects: Related to the idea of visual attention targeted at objects, many robotics systems rely on saliency maps able to provide an estimation of the boundaries of the object [START_REF] Orabona | A proto-object based visual attention model[END_REF], [START_REF] Walther | Modeling attention to salient proto-objects[END_REF], [START_REF] Wischnewski | Where to look next? combining static and dynamic proto-objects in a tva-based model of visual attention[END_REF], sometimes referenced as proto-objects. As mentioned in Section 3.2.2, some saliency maps directly provide this feature. It is therefore easier to focus the attention at the center of the object, using an appropriate scale.

Exploration-exploitation trade-off: Although top-town attention is better-suited to localize and identify objects, it is sometimes useful to alternate between bottomup and top-down in order to keep a good general exploration of the environment [START_REF] Rutishauser | Is bottom-up attention useful for object recognition?[END_REF], [START_REF] Zhu | Unsupervised object class discovery via saliency-guided multiple class learning[END_REF]. The alternance between the two may be obtained by a static weighted fusion [START_REF] Frintrop | A bimodal laser-based attention system[END_REF], more complex schemes driven by temporal differential equations [START_REF] Rasolzadeh | An active vision system for detecting, fixating and manipulating objects in the real world[END_REF] or just an alternation between exploration and exploitation modes [START_REF] Xu | Autonomous switching of top-down and bottom-up attention selection for vision guided mobile robots[END_REF].

Localizing objects in an image

We demonstrated that finding object candidates and top-down saliency were often tightly coupled in a robotics context, and that object detection can be obtained with saliency maps. We see object localization in this section at a wider point of view, and we investigate the alternative approaches to detect and localize objects in their environment from a still input image. An interesting review mentioning recent deep learning approaches in visual understanding problems has been written by Guo et al. [START_REF] Guo | Deep learning for visual understanding: A review[END_REF]

Segmentation-based approaches

Image segmentation can be define by the process of splitting an image into several segments, or groups of pixels. Most of the time, the segments are grouped in a meaningful way and should help for further processing or for image understanding. Figure 3.8 illustrates the three types of segmentation that are further described in this section: over-segmentation, background-foreground segmentation, and semantic segmentation.

Over-segmentation

In over-segmentation, the image is separated into an arbitrary number of segments. These segments do not have a particular semantic meaning. The only criterion to gather pixels are based on image properties such as color, texture, or orientation. Pixels within the same over-segmented region should also be gathered in terms of position, so that regions are spatially consistent. Over-segmentation is usually useful as a pre-processing step to form meaningful clusters, that should be combined later on. These approaches are usually highly tunable depending on the user's requirement (size of superpixels, edge smoothness, etc...).

Most popular over-segmentation approaches consists in dividing the image into superpixels. A first type of superpixel uses seeds and grows regions around them. The SLIC superpixels [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF] for example produces a set of segments from a regular distribution of seeds in the image. These seeds are used as an initialization for a K-means clustering considering both pixel position and color. Similarly, the SEEDs superpixels [START_REF] Van Den Bergh | Seeds: Superpixels extracted via energy-driven sampling[END_REF] are obtained from a regular grid, by incrementally exchanging pixels between regions to adjust boundaries. This approach is particularly interesting as it is both fast and efficient. A second family of superpixels is based on graphs [START_REF] Liu | Entropy rate superpixel segmentation[END_REF], [START_REF] Zhang | Superpixels via pseudo-boolean optimization[END_REF], that are incrementally cut or clip the graph vertices (typically the pixels) based on objective functions. Graph-based approaches usually propose regions of more variable size and shape.

Over-segmentation is also applicable to point cloud or RGB-D images. The geometrical curvatures, normal orientations, convexity and symmetries of the point cloud are then extracted to generate super-voxels [START_REF] Papon | Voxel cloud connectivity segmentationsupervoxels for point clouds[END_REF], [START_REF] Xu | Flattening supervoxel hierarchies by the uniform entropy slice[END_REF]. 3D segmentation is also a complementary information that can be added to traditional RGB oversegmentation, either following an early [START_REF] Weikersdorfer | Depth-adaptive superpixels[END_REF] or late [START_REF] García | Saliency-based object discovery on rgb-d data with a late-fusion approach[END_REF] fusion approach. Superpixels are used in our work for two different reasons. First, for a feature extraction process, second, to recover the shape of the salient objects when the saliency map has a weak resolution. We use in both cases the SLIC superpixels [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF] Foreground-background segmentation When doing foreground-background segmentation, the goal is to separate each pixel of the image into background and foreground. The problem can then be seen as a binary classification problem. The definition of background and foreground is unclear and depends on the context it is used for. If the task is to segment objects in general, we can consider foreground as being objects and background as being everything else. This notion of foreground and background has a lot in common with the definition of saliency, foreground items being most of the time considered as relevant elements.

When considering video, moving objects may be seen as forground elements. Traditional background learning and subtraction [START_REF] Elgammal | Non-parametric model for background subtraction[END_REF] may suffice if the sensor is set at a fixed position. Otherwise, other approaches exploit motion and tracking to find the boundaries of the target [START_REF] Badenas | Motion-based segmentation and region tracking in image sequences[END_REF], [START_REF] Mahsa Ghafarianzadeh | Efficient, dense, object-based segmentation from rgbd video[END_REF].

Similar to superpixels, traditional approaches were using seeds and Markov random fields to optimally segregate foreground and background based on color, texture or contours [START_REF] Felzenszwalb | Efficient graph-based image segmentation[END_REF], [START_REF] Rother | Grabcut: Interactive foreground extraction using iterated graph cuts[END_REF]. Those methods are sometimes trained on a set of representative images to learn the optimal parameter values [START_REF] Dubois | A comparison of geometric and energy-based point cloud semantic segmentation methods[END_REF]. Some other approaches also exploit the prior knowledge that the object of interest is surrounded by the background and rely on a logpolar transform as the key of the segmentation [START_REF] Mishra | Active visual segmentation[END_REF], [START_REF] Mishra | Active segmentation with fixation[END_REF].

Other methods [START_REF] Stein | Object partitioning using local convexity[END_REF], [START_REF] Karpathy | Object discovery in 3d scenes via shape analysis[END_REF] start with an over-segmentation approach and progressively group superpixels based on compactness, convexity, curvature or symmetries to reconstruct foreground objects. The notion of symmetry to detect objects is particularly exploited with RGB-D sensors and tend to be efficient even in case of partial occlusions [START_REF] Potapova | Attention-driven object detection and segmentation of cluttered table scenes using 2.5 d symmetry[END_REF], [START_REF] Ecins | Cluttered scene segmentation using the symmetry constraint[END_REF].

For indoor environments, and when a depth cue is available, a popular separation technique exploits the fact that objects of interest are generally lying on planar surfaces [START_REF] Caron | Neural network fusion of color, depth and location for object instance recognition on a mobile robot[END_REF], [START_REF] García | Saliency-based object discovery on rgb-d data with a late-fusion approach[END_REF], [START_REF] Ali | Contextual object category recognition for rgb-d scene labeling[END_REF], [START_REF] Potapova | Attention-driven object detection and segmentation of cluttered table scenes using 2.5 d symmetry[END_REF]. Planar surfaces are either tabletops or floor, or any other horizontal support. The main idea is to first detect this plane (typically using a RANSAC algorithm [START_REF] Fischler | Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography[END_REF]) to discard the points belonging to this plane, and to use geometrical constraints to detect objects lying on it. A common approach is to only keep elements contained in the enclosing convex hull of the detected plane [START_REF] Ali | Contextual object category recognition for rgb-d scene labeling[END_REF]. When the major plane is the floor, other important structures such as walls and ceiling must be detected and removed as well, as proposed by Caron et al. [START_REF] Caron | Neural network fusion of color, depth and location for object instance recognition on a mobile robot[END_REF]. Many RGB-D datasets are composed with objects on tabletops, either well separated [START_REF] Lai | A large-scale hierarchical multi-view rgb-d object dataset[END_REF] or in a more cluttered scenario [START_REF] Ecins | Cluttered scene segmentation using the symmetry constraint[END_REF]. These types of approaches are performing well on these datasets, as long as a large portion of the main plane is visible on the image.

In our work, the produced saliency maps could amount to a sort of soft backgroundforeground segmentation, where foreground elements would be the objects. Depthbased segmentation is also at the heart of our saliency learning procedure as we use a modified version of the work of Caron et al. [START_REF] Caron | Neural network fusion of color, depth and location for object instance recognition on a mobile robot[END_REF] to generate a learning signal.

Semantic segmentation

Semantic Segmentation aims to label each pixel with a class of objects (such as car, pedestrians, dogs, ...) and non-objects (for example, water, sky, or road). It is therefore a multi-class classification problem. In recent years, challenging datasets have been proposed to evaluate semantic segmentation accuracy. The most famous one is probably the PASCAL VOC 2007 dataset for RGB [START_REF] Everingham | The pascal visual object classes challenge: A retrospective[END_REF] and the NYU dataset for RGB-D [START_REF] Silberman | Indoor segmentation and support inference from rgbd images[END_REF].

Before deep learning approaches, the best performing techniques were based on hand designed features applied on a set of pixels to generate patches. Those patches were locally describing the image and were sent to a classifier, typically a random forest or a boosted cascade, to predict the class probability of the center of the patch. These types of approaches were providing pretty noisy probabilities and were typically refined with conditional random fields [START_REF] Ladick | What, where and how many? combining object detectors and crfs[END_REF]. Other techniques tried to predict every pixels of the patch rather than just the center [START_REF] Zhang | Semantic segmentation of urban scenes using dense depth maps[END_REF], or used an over-segmentation technique to group pixels [START_REF] Tighe | Superparsing: Scalable nonparametric image parsing with superpixels[END_REF]. Classification refinement was typically able to improve the classification in large areas like sky, but did not really improve small clusters. Lastly, Gupta et al. [START_REF] Gupta | Perceptual organization and recognition of indoor scenes from rgb-d images[END_REF] achieved good performance by detecting boundaries and applying hierarchical clustering as a pre-processing step to segmentation.

Similar to many other fields of computer vision, a major breakthrough occurred in the field of semantic segmentation through CNN-based techniques. Early deep learning approaches were based on blocks identification, but provided a segmentation mask at a poor resolution [START_REF] Grangier | Deep convolutional networks for scene parsing[END_REF]. More recent approaches such as SegNet [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling[END_REF] are following the auto-encoders philosophy to design fully convolutional networks able to retrieve a segmentation mask at the original scale. This approach is sometimes coupled with a post processing using recurrent neural networks [START_REF] Lin | Efficient piecewise training of deep structured models for semantic segmentation[END_REF].

An interesting advantage when using CNN for semantic segmentation is the possibility to train a network in a weakly supervised basis. Papandreou et al. [START_REF] Papandreou | Weakly-and semi-supervised learning of a dcnn for semantic image segmentation[END_REF] typically demonstrated that state-of-the-art performance could be obtained from a small number of fully supervised data and a large number of weakly supervised data such as bounding boxes or raw image with a single label.

Semantic segmentation is not directly used in our work, but the methods for retrieving the original resolution and the weak supervision aspects are interesting items that have inspired some of our choices.

Object recognition and bounding box proposals

Another way to consider object localization is to think in terms of instances rather than at the pixel level. When doing object localization in this case, the output of the system is typically one or several bounding boxes, providing information about the number of instances, and a rough localization and size of them in the image. The boundaries are not precisely detected, but the information is easier to interpret at a semantic level. Before going any further, a reminder on the terminology in this field is presented. Figure 3.9 provides an illustration of the three discussed categories of object localization that are class-agnostic bounding box proposals, object recognition, and object detection.

Sliding-window approaches

Until recently, the best performing approaches for object detection and recognition were based on sliding windows, where a light classifier was applied multiple times on a portion of the image (the window) at regular intervals all along the image. This approach then required a significant amount of inferences, considering about 10 4 windows per images, and potentially up to 10 7 if the task also required to detect the scale of the object. When considering an object recognition problem, each window had to be evaluated for each detector, thus multiplying the number of operation by the number of output classes. As a result, a very light object classifier was applied in these types of methods to provide a result in a reasonable time. Incidentally, the efficiency of the classifier was pretty limited. In addition and in spite of the large number of windows, they were not necessarily adapted to the size of the object to detect and then provided a very limited efficiency.

Among the most popular and best performing approaches relying on a sliding window, one can think of the Viola and Jones object detector [START_REF] Viola | Robust real-time object detection[END_REF]. The basic idea was to train a binary classifier to discriminate between the object to detect and distractors. The original image was first decomposed into wavelets, and classification was obtained though a boosted cascade classifier. A more recent and successful approach was the Deformable Part Model [START_REF] Felzenszwalb | A discriminatively trained, multiscale, deformable part model[END_REF], where discriminative portions of objects were first detected and combined based on their relative positioning with each other.

Class-agnostic bounding box proposals

In recent years, other approaches have proposed to directly generate bounding boxes around potential objects of interest to limit the high amount of inference required by the sliding window approach. The class-agnostic bounding box proposals then aims to take advantage of the image shapes and colors to produce bounding boxes that may contain objects. They are not restricted to a single class but rather exploit the general properties of objects (colors, context, boundaries) to generate bounding boxes. Again, these properties are strongly related to the notion of saliency. Among available techniques, two categories of approaches have emerged (See the review of Hosang et al. [START_REF] Hosang | What makes for effective detection proposals?[END_REF] for an extensive review on this field.

The first category is referred as the grouping methods. The most popular one probably being the Selective Search [START_REF] Van De Sande | Segmentation as selective search for object recognition[END_REF]. In these approaches, the goal is to generate segments that are likely to represent objects. The most naive approach would be to generate a bounding box out of each segment provided by a segmentation technique. Most of the time, these methods first apply an over-segmentation technique, and combine the segments based on shapes, convexity, appearance of boundaries. Several combination trials are performed, and related bounding boxes are produced accordingly. One of the advantages of these approaches is that they provide a segmentation proposal on top of bounding boxes.

The second one is the window scoring proposal method. In this category, Objectness [START_REF] Alexe | Measuring the objectness of image windows[END_REF] and EdgeBoxes [START_REF] Zitnick | Edge boxes: Locating object proposals from edges[END_REF] are the most famous techniques. In these approaches, a set of randomly sampled bounding boxes are generated, and a score is attributed to each of them based on shapes, contours, superpixel straddling and so on. Although faster, these approaches are known to be less accurate in localization.

Class-agnostic bounding boxes are now typically coupled with deep learning classifiers, faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] probably being the most famous example. The first reason is that they significantly reduce the number of proposals over the sliding window, thus making efficient the use of a slower classifier. The second one is that proposals are most of the time more accurately centered and scaled than a sliding window approach would. This way the recognition is likely to be more efficient.

Several methods have been proposed to evaluate the efficiency of a box proposal. The standard one is probably the recall vs number of proposals, relying on an intersection over union measure (IoU) that defines how well a box matches the ground truth. As this method relies on an arbitrary threshold to determine the match, Hosang et al. [START_REF] Hosang | What makes for effective detection proposals?[END_REF] proposed to use the average Recall (AR) that uses an integral over a set of threshold and demonstrated that this approach was more objective and stable over datasets. Another approach is to use the detector responses around objects.

In our work, we propose a way to use saliency maps to provide relevant bounding boxes candidates around salient objects. Our technique is based on a posterior bias on the scores produced by the EdgeBoxes algorithm.

Multi-class object detection

Object detection and object recognition are two different classes of problems, usually solved in a similar way. Object recognition consists in the detection, localization and labeling of all objects of an image (or a restricted number of classes) with a labeled bounding box. In object detection, the purpose is similar except that only one class of object is considered. For example in pedestrian detection, the goal is to detect all pedestrians (and only pedestrians) with their bounding boxes. We describe in this section methods able to solve the more general problem of multi-class object detection.

Multi-class object detection is close to the problem of semantic segmentation. The techniques used in both fields sometimes even are the same and simply differ in the post-processing step. Common datasets such as PASCAL VOC [START_REF] Everingham | The pascal visual object classes challenge: A retrospective[END_REF] or Microsoft COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] propose both annotations alternatives.

Today state-of-the-art's methods all use deep learning to identify objects, but two major axes are then used to locate them.

The first one is based on bounding box proposals. Each proposal is cropped and send to a classifier able to identify the class of the whole image, finally deciding if the proposal contains one of the objects to detect or not. The most representative work in this type of approaches is R-CNN or faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF]. As a critical aspect of these methods is the quality of the bounding box, approaches such as LocNet [START_REF] Gidaris | Locnet: Improving localization accuracy for object detection[END_REF] try to refine the proposal to obtain a better identification.

The second one uses a single convolution pass on the whole image to simultaneously localize and identify the object. These approach, such as YOLO [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF], are generally much faster and have more in common with semantic segmentation by designing fully convolutional networks able to solve both tasks [178]. Another interesting approach is to use labeled images as a weak supervision signal to turn an image classifier into an object detector [START_REF] Hoffman | Lsda: Large scale detection through adaptation[END_REF].

Like semantic segmentation, we do not consider the problem of multi-class object detection in our method, but it could be a direct application of our saliency maps. In addition, the state-of-the art in this field is a source of inspiration in terms of methodology.

Learning visual saliency on a robot

This section describes the main requirements to learn visual saliency directly on a robot. After presenting existing methods related to visual attention and learning, we focus on three main components of the approach described in the next chapter. This section does not aim to be an extensive review in the fields of machine learning, but rather a spotlight on the mechanisms underlying our approach.

Robotics systems that learn visual attention

Learning is used in robotics for many types of applications. Specifically for saliency, we mentioned in Section 3.2.2 the pattern classification and the fully connexionist approaches. These approaches are however designed to be learned offline and not modified during the robot's exploration. Learning or refining visual attention directly during exploration is not so common. Nevertheless, we identified two cases in which this aspect is used.

Top-down refinement

The first one consists in refining the top-down modulation by finding appropriate weights to combine the extracted features. In VOCUS, Frintrop [START_REF] Frintrop | Vocus: A visual attention system for object detection and goal-directed search[END_REF], [START_REF] Mitri | Robust object detection at regions of interest with an application in ball recognition[END_REF] has proposed to learn weights to discriminate a specified target based on training images. Learning was done through a Region Of Interest (ROI) found in a manually specified rectangle, and each weight was determined based on the mean activation ratio with and without the ROI.

Rasolzadeh et al. [START_REF] Rasolzadeh | An active vision system for detecting, fixating and manipulating objects in the real world[END_REF] based their approach on the same idea of ROI, but proposed to find the weights by solving an optimization problem with the Levenberg-Marquardt algorithm. They further improved the method by integrating the context in the optimization. As the problem was not convex anymore, the weights were found by training a neural network. An additional idea in this work was to learn the top-down weights online by alternating between bottom-up and top-down attentions. Bottom-up attention served as an open exploration to select targets and discover objects. Once an object was discovered, a foveated segmentation was used to create the ROI, and the top-down weights were optimized. To combine bottomup and top-down attentions, a system of temporal differential equations was used to weight the importance of each component.

In the context of learning how to combine features, Borji et al. [START_REF] Borji | Boosting bottom-up and top-down visual features for saliency estimation[END_REF] have investigated the use of several classifiers to combine both bottom-up and top-down (in this case, object detectors) cues. They concluded that the AdaBoost classifier had the best prediction accuracy for this task. Nevertheless, this approach was trained offline, in a fully-supervised manner, based on eye fixation datasets.

Visual attention control policies

The second type of methods aims to learn eye movements displacements in order to better perceive the objects of interest. These approaches typically rely on reinforcement learning techniques.

Minut et al. [START_REF] Minut | A reinforcement learning model of selective visual attention[END_REF] have presented a system based on Q-learning to determine the area the most likely to contain the object to identify. They then used the camera zoom and small saccades to precisely localize and identify the object.

Saccades can also be directed towards areas of interest within an object to improve its identification. Paletta et al. [START_REF] Paletta | Q-learning of sequential attention for visual object recognition from informative local descriptors[END_REF] proposed a framework based on Q-learning to learn the best scanpath within an object to ensure correct identification.

Lastly, the work of Borji et al. [START_REF] Borji | Online learning of task-driven object-based visual attention control[END_REF], inspired from McCallum's research [START_REF] Mccallum | Reinforcement learning with selective perception and hidden state[END_REF], proposed to learn a top-down attentional model able to simultaneously discriminate objects in an environment (for example traffic signs, buildings) from their visual appearance, and learn the best sequence of actions to reach a goal (for examples, how to move the robot across the streets to reach a house).

Learning constraints of our setups

Learning visual signals on a robot can be done in many different ways, but we here focus on two aspects related to our setups. First, our robot starts exploration with no prior knowledge about the environment, collects data through observations and learn a representation on-the-fly as exploration goes. Therefore, exploration needs to be coupled with an online learning mechanism. Second, our approach does not include a human in the loop, so that learning has to be done without supervision. We talk here of self-supervised learning rather than unsupervised, that has a different meaning in the machine learning literature.

Online learning

Online learning, or incremental learning refers to a machine learning approach where data arrive in a sequential order, and where the model integrating those data is updated simultaneously with the reception of new data. Online learning is opposed to offline learning, or batch learning, where data is available from the beginning, and where the model is build in a single shot. Online learning is common in robotics to refine predictions and representations as new perceptual data is acquired by the robot.

Statistical methods: Methods doing predictions based on a statistical distribution of the data are well-suited for being updated incrementally. In particular, classification based on a k-nearest neighbor [START_REF] Larose | K-nearest neighbor algorithm[END_REF] comparison is trivially updated by adding the new sample to the representation dataset, although the search method may also require a KD-tree update to be faster. A major drawback of this approach is that memory increases as new data arrives. Voting methods based on bags of visual words [START_REF] Angeli | Fast and incremental method for loopclosure detection using bags of visual words[END_REF] also have a natural incremental version where statistics (counts) of visual words are updated on the fly.

Online random forests: Most offline machine learning algorithms also have their incremental version, and it is the case for random forests we have used in our work. To make a random forest incremental, the general approach is to update the splits of the tree by maintaining quality scores at every leaves, and to grow the tree incrementally [START_REF] Saffari | On-line random forests[END_REF], [START_REF] Denil | Consistency of online random forests[END_REF]. However, most online random forests approaches are memory consuming (the whole dataset is stored and re-used for updates), slow for a large dataset, and do not perform as well as in offline mode. A recent promising approach is the Mondrian forests [START_REF] Lakshminarayanan | Mondrian forests: Efficient online random forests[END_REF] that determines new tree splits based on mondrian processes. We have developed in our work an alternative technique to update random forests faster than what offline training would.

Neural networks: Neural networks are from essence online classifiers. Although often used in batch mode and trained by feeding the network with the whole dataset several times (a whole dataset pass is called an epoch), samples are provided one by one to update the network. A very desirable property of these methods is that there is no need to keep the whole dataset for updates. Once presented to the network, the model keeps an internal memory of the sample. A major issue though is that the order in which examples are presented strongly influences the performance of the system. The earliest samples tend to be forgotten and the model is more accurate for recent examples. Moreover, presenting too many similar examples in a row may have catastrophic consequences in the learning quality (this phenomenon is called catastrophic forgetting). Online learning with a neural network must then be considered carefully to lead to successful results. In CNNs, the problem is the same. Although little work has been presented yet in that regard, the field is actively investigated as well [START_REF] Pasquale | Object identification from few examples by improving the invariance of a deep convolutional neural network[END_REF].

Self-supervised learning

In most supervised learning techniques, a human contributes in the task of annotating data, which means provides a trustful signal that the algorithm should follow to build its predictive model. In the case where a robot does not use any human assistance, other approaches should be considered.

Alternatives to supervised learning: A first family of learning without human supervision is unsupervised learning, where classes or clusters are determined only by considering properties of the dataset distribution. Another approach is to learn from a partially or roughly annotated dataset, referred respectively as semisupervised and weakly supervised learning. When the signal used for learning is a reward from the environment, following an action, the problem is called reinforcement learning.

Automatic or self-supervised annotation: Sometimes, there is a way to automatically annotate data, or to generate annotated samples without any supervision. We call this process self-supervised learning. In computer vision, a typical example of automatic annotation is the colorization of black and white films, since there is an unlimited amount of color films and that conversion from color to grayscale is trivial. Another example is the work proposed in Make3D [START_REF] Saxena | Learning depth from single monocular images[END_REF] that aims to learn to predict the depth map of an RGB image. From RGB-D sensors or stereo cameras, a fully supervised correspondence is available between these two components. In robotics, sensory-motor correspondences may also be automatically obtained by performing an action from a motor command, and waiting for the sensor feedback that plays the role of the annotation [START_REF] Oudeyer | Intrinsically motivated learning of real-world sensorimotor skills with developmental constraints[END_REF], [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF]. Our system is based on this principle of automatic annotation of the saliency, based on a correspondence between a depth-based segmentation and RGB images.

Transfer learning: Transfer learning, or inductive transfer [START_REF] Pan | A survey on transfer learning[END_REF] aims to transfer knowledge between a source and a target. The problem is typically formalized by a source/target data, a source/target model and a source/target label. The goal is then to take advantage of the source model to improve the target one (see Figure 3.10). This field gathers a lot of different methods, but two of them are of particular interest for our work.

The first one is the domain adaptation, or fine-tuning: in some cases, a large, fully annotated dataset is available for a source problem, and a limited amount of data (annotated or not) is available for target problem, having a lot in common with the source (in terms of data and/or labels). It is then possible to take advantage of the large dataset to train a robust model, and use the limited dataset just to optimize the learned model to the target task. Domain adaptation has been widely exploited in speech recognition to adjust a natural language trained on thousands of speaker to a single user [START_REF] Margolis | A literature review of domain adaptation with unlabeled data[END_REF], or in CNN for a specific task on which little annotation is available [START_REF] Sharif Razavian | Cnn features off-the-shelf: An astounding baseline for recognition[END_REF].

The second one is the multi-task learning, where problems A and B are solved at the same time by exploiting common properties between the tasks. Our learning method follows this type of approach, but solves the two tasks sequentially: In our case, a first object recognition task is learned offline, and the corresponding model is used to automatically annotate data for the task of saliency learning. This kind of approach is called a knowledge transfer, and is also used in deep learning framework [START_REF] Donahue | Decaf: A deep convolutional activation feature for generic visual recognition[END_REF].

Weakly-supervised learning: In a weakly supervised context, the data is poorly annotated, and the model is able to generalize over this signal to learn more than what was originally provided (see example of Figure 3.11 where the goal is to localize cats in images with cropped images of cat as the only learning signal). The core characteristics of the annotated data are then naturally extracted from the partial annotations. With such learning, it is then possible to generalize the learned model, and to apply it to a slightly more general task. Several examples of weak supervision have been presented in Sections 3.3.1 and 3.3.2 for the task of semantic segmentation or object recognition. We also rely on such weak supervision to generalize saliency over a partial learning signal.

Conclusion

In this chapter, we have presented an overview of the background related to the approach to be proposed in the next chapter. Visual attention, object localization and machine learning are separate fields, but they all contributed to the construction of our methods. Moreover, we have shown that those fields were sometimes largely intertwined, and that our system precisely lies at their junction. We aim to design a system able to learn visual saliency on a robot. Visual saliency is at the heart of many visual attention systems, and we have re-used a very classical scheme to construct our own technique. Then, as our robotics platform are designed for indoor environments, our saliency must be oriented for the task of localizing objects in their surrounding. We therefore investigate this field that is more related to computer vision. Lastly, as our approach involves learning on a robot, we have studied machine learning techniques able to satisfy the constraints of our setup.

The next chapter is dedicated to the technical description of our proposed approach and its connections with each of the aforementioned fields.

Chapter 4

Proposed Approach

Introduction

For a robot to efficiently identify the interesting elements of its environment, we base our approach on saliency maps. More specifically, we would like to learn these saliency maps on-the-fly during exploration. Thus, these maps will be specialized for the environment the robot is exploring, for the task the robot is asked to accomplish, while remaining flexible to any change or novelty in the environment.

This chapter describes the algorithms and software components developed to learn such a saliency model. We first position our approach versus the state-of-theart provided in Chapter 3. We also provide a general overview of the approach before going in details through each component of the architecture.

The incremental saliency learning is made possible by three major items that are feature extraction, object detection and online classification. We provide for each of them a description of several possible methods, and discuss their advantages and drawbacks.

Lastly, we describe the pipeline to produce saliency maps having the desired properties, and we suggest a measure to exploit these maps as a mean to produce bounding box proposals around salient objects.

Overview of the method 4.2.1 Positioning versus state-of-the-art

Regarding visual attention

Our approach has a strong background on the field of visual attention. We intend to follow the traditional visual attention pipeline described in Section 3.2.1 consisting in feature extraction and feature combination. Moreover, one of the setups used for evaluation is a foveated platform, very similar to many existing systems (See Section 3.2.3).

However, our approach differs from most visual attention techniques for the following reasons:

• Instead of using saliency maps as a black box, plugged as is in a visual attention system, we propose two operating modes: a learning mode, and an exploitation mode.

• In the learning mode, we focus on how to build an efficient feature combination online, during the robot's exploration. This combination should be consistent with the robot's future tasks.

• In the learning mode, the focus of attention is not directed towards the most salient elements. We do not discuss it here, but Part II of the manuscript suggests alternative methods to learn saliency more efficiently.

• In the exploitation mode, the combination model that has been learned is exploited to produce saliency maps. From there, it is possible to follow a classical visual attention pipeline.

Our method is designed to learn top-down types of saliency maps, that are dedicated to object detection tasks. In addition, our saliency models are specialized for the environment they were trained in, an are very likely to fail if exploited in a different context.

Regarding object localization

As discussed in Chapter 3, the multiple similarities between object-oriented saliency and object localization positions our approach at the junction of those two fields. Although our goal is to produce saliency maps, we also rely on object localization techniques for several reasons

• In a first setup, we rely on a background-foreground, depth-based, objectoriented segmentation technique to learn the saliency.

• In a second setup, we use an object classifier as a weak supervision signal to learn saliency maps, similar to recent approaches presented in Section 3.3.2

• We show that our saliency maps can be used to bias the output of existing agnostic bounding box proposals methods. The outcome is a set of bounding boxes that are much more likely to contain salient objects.

Regarding learning

Our method has a lot in common with the mechanism proposed in Frintrop [START_REF] Frintrop | Vocus: A visual attention system for object detection and goal-directed search[END_REF] or in Rasolzadeh's work [START_REF] Rasolzadeh | An active vision system for detecting, fixating and manipulating objects in the real world[END_REF], where an ROI of the visual field was exploited to modulate top-down saliency. In Rasolzadeh's work, this modulation was done online, and an efficient trade-off was found between open exploration and samples seeking for top-down refinement. Our approach differs from these work for the following reasons:

• Top-down modulation is not obtained by a simple linear combination of the feature maps, but by a non linear classifier, able to capture more complex feature combinations.

• The method is self-supervised which means that salient elements of the visual fields are detected (when possible) and integrated to the model without any user supervision.

Many approaches propose to learn visual saliency offline, from a large database, the best performing results being obtained with CNN architectures [START_REF] Kruthiventi | Deepfix: A fully convolutional neural network for predicting human eye fixations[END_REF]. However, learning saliency online with a classifier, without any initial database is far less common. We then investigate the online aspect of learning, while taking advantage of the CNN capacities.

In our approach, self supervision is made possible by a multi-task transfer learning run sequentially (called knowledge transfer). In our case, we learn offline, before robot's exploration, an easy and restrictive task (for example, object recognition) that produces a reliable estimate (the class label). Then, we use the predicted output as a learning signal for a more complex tasks (for example, low-resolution object localization in a cluttered environment). Lastly, the self-supervision provides a learning signal that is restrictive, thus making the problem weakly supervised. Our approach then aims to generalize over this weak learning signal.

General mechanisms

Our approach is designed so that the robot moves within the environment and receives image streams. Saliency is then learned progressively as image are acquired and processed.

Our approach is composed with three interconnected modules for learning, and works in a learning mode and exploitation mode. We here provide an overview of these module and the way they interact with each other (see Figure 4.1 to visualize the general architecture).

Saliency learning

During the learning phase, the robot explores the environment and progressively receives visual information that is processed and integrated to the model. The visual information is composed with two visual streams denoted as the main visual stream and additional visual stream. We also define as an observation a set of two synchronized frames from the visual and additional streams.

The core idea of the saliency learning is to continuously provide a set of features and labels (essentially two classes denoted here as "salient" and "not salient") that are send to an online classifier. The classifier then learns a saliency model that can be exploited to infer saliency maps. The classifier chosen in this study is a random forest designed to be updated incrementally (see Section 4.5.1).

Features are extracted from the main visual stream (RGB in our case) from which saliency is to be estimated. In Section 4.3, we propose thee feature extraction techniques to this end.

Contextual input

Object segmentation RGB Feature maps

Online classifier

Saliency model

Features Labels

Foveal input Depth input

Object segmentation RGB Feature maps

Online classifier

Saliency model

Features Labels However, a more complex issue is to provide a reliable supervision signal without any human assistance. To this end, we rely on an additional visual stream from which a reliable saliency signal is obtained. This signal is then turned into labels and sent to the classifier. This saliency estimation must be trustful enough to be used as a learning signal, but can also be restrictive. The role of the classifier is then twofold: first, to transfer the saliency knowledge from the additional visual stream to the main visual stream. Second, to generalize the restrictive signal to produce saliency maps on the whole main visual stream.

RGB input

(a) (b)

Saliency exploitation

Once a saliency model is available, this one can be exploited to produce saliency maps in the specific environment it was trained for. In this configuration, we do not rely on the additional visual stream anymore. Features are extracted from the main visual stream, the same way it was done for learning, and these features are sent to the classifier to infer saliency according to the model (see Section 4.5.2).

Additionally, in Section 4.5.3, we show that the saliency map constructed this way can be used to produce bounding boxes around salient objects.

Two application scenarios

We describe in Section 4.4 two scenarios for which an additional visual stream can produce a restrictive saliency as a learning signal. These scenarios are represented in Figure 4.2.

The first scenario (Section 4.4.1) requires both a contextual camera (also called peripheral camera) and a foveal camera. The contextual camera plays the role of the main visual stream, while the foveal is the additional one. Saliency is then detected based on an object recognizer trained for some specific objects on the foveal stream. Saliency is then determined by whether an object was identified in the fovea or not. Here, the saliency estimate is partial as salient objects are detected only when visible in the fovea.

The second scenario (Section 4.4.2) works mainly for indoor environments and considers salient objects as being on planar surfaces. The additional visual stream here is a depth map produced by a Kinect or similar sensors. The depth map is processed and produces a segmentation of the salient objects when detection was possible (when object was seen from an appropriate point of view). Saliency is then partial as salient objects are not always detected.

To make data formatting easier, the information produced by the additional streams are turned to what we call segmentation masks. A segmentation mask provides a pixel-wise mapping between the main visual stream and the saliency found in the additional stream.

Feature extraction

We want to use visual features based on the RGB components to construct saliency maps. Those features must be light enough to be computed in real time on the whole field of view, and representative enough to discriminate salient and non salient elements. Lastly, the features must be stable enough to avoid saliency misdetections, especially in areas of strong or sharp color variation.

We present here three different types of features that we used and evaluated. We systematically discuss the choice for such features, as well as their advantages and drawbacks.

Itti and Koch feature extractor Motivation

A first natural idea to obtain relevant features for saliency is to look at the one commonly used in the literature. In that regard, many visual attention system rely on a saliency model created by Itti and Koch [START_REF] Itti | A model of saliency-based visual attention for rapid scene analysis[END_REF]. As it is flexible and modular, our first attempt was to use the filtering procedure proposed by the method as a feature extractor. Frintrop [START_REF] Frintrop | Vocus: A visual attention system for object detection and goal-directed search[END_REF] has used a similar type of filtering and obtained better results. We did not investigate the performance of these, but expect similar performances.

Description

Itti and Koch have proposed a method for visual attention systems, including a saliency map computation and a way to consider this saliency map in the allocation procedure of the attention. We are more specifically interested here in the batch of filters applied on the input image to produce a set of feature maps, as presented in Figure 4.3.

These filters are based on Gabor filters [START_REF] Gabor | Theory of communication. part 1: The analysis of information[END_REF] of various orientations at several scales. A Gabor filter is defined by a scale σ and an orientation θ as:

       G(σ, θ) = exp(-1 2 x 2 θ +y 2 θ σ 2 )cos(2πx θ ) x θ = x × cos(θ) + y × sin(θ) y θ = -x × sin(θ) + y × cos(θ) (4.1)
More precisely, each filter is obtained by the following equation:

O(σ f , σ c , θ) = |G(σ c , θ) -G(σ f , θ)| (4.2)
where θ ∈ {0, π 4 , π 2 , 3π 4 } is the orientation, σ f ∈ {2, 3, 4} is the scale of the central stimuli and σ c ∈ {σ f + 1, σ f + 2} the scale of the peripheral stimuli. In total, 24 filters are therefore calculated. Another type of filters represents the difference of intensity between the center and the periphery, obtained by equation 4.3:

I(c, f ) = |I(c) -I(f )| (4.3)
where I is a Gaussian pyramid created from a grayscale image. f ∈ {2, 3, 4} and c ∈ {f + 1, f + 2} represent the scales of this pyramid. In total, 6 filters of this type are computed. Lastly, we obtain the center-surround color opposition, from equations 4.4 et 4.5:

RG(c, f ) = |(R(f ) -G(f )) -(G(c) -R(c))| (4.4) BY (c, f ) = |(B(f ) -Y (f )) -(Y (c) -B(c))| (4.5)
where R G B Y stand for the Gaussian pyramids obtained from the red, green, blue and yellow components of the image, at the same scales f and c used for the grayscale filters. In total, 12 filters are computed this way.

Pros and cons

These features are very common, many implentations are publicly available. They are also fast and easy to compute. However, the center-surround averages provide a poor discrimination of the saliency at the borders of an object.

Make3D-based feature extractor Motivation

The problem considered in Make3D [START_REF] Saxena | Learning depth from single monocular images[END_REF] is the one of predicting depth from a single RGB image. To this end, features are extracted from the RGB image for each pixel, and sent to a regressive classifier to predict the depth of each pixel independently. This classification phase is followed by a refinement stage, enhancing the overall depth map consistency with Markov Random Fields. As explained in Section 3.4.2 of the previous chapter, the saliency learning approach is a sort of transfer learning from depth to RGB (at least for one of the two segmentation techniques), and saliency maps are obtained from features extracted for each pixels of the original image. Make3D then shares strong similarities with our technique and the type of signals we aim to manipulate. We then build our second feature extractor on the ideas proposed by Make3D.

Description

In Make3D, absolute and relative features are considered and extracted from the RGB image. Absolute features are sent to a classifier to estimate the absolute depth of a pixel, while relative ones are used to refine the global depth map. We base our technique on the absolute features. For each pixel, texture variation, gradients and color statistics are obtained from a patch surrounding the pixel (19 types of features in total). Then, to better describe the geometry of the surrounding, these features are also computed for neighbor patches (up, down, left, and right) at three different scales. In more recent work, Saxena et al. [START_REF] Saxena | Make3d: Learning 3d scene structure from a single still image[END_REF] also compute superpixels on the image and calculated a single descriptor per superpixel.

To extract our features, the first step consists in a superpixel extraction. We compute the SEEDS superpixels [START_REF] Van Den Bergh | Seeds: Superpixels extracted via energy-driven sampling[END_REF] on the whole image. The number of SEEDS to produce depends on the size of the input image, and on the average size of the objects on the image. Superpixels should then be small enough to oversegment the salient objects. In our experiments, we have used between 150 and 350 superpixels for images of size 640 × 480. Once extracted, we compute the mean RGB value in each of these superpixels.

Then, for a given pixel of the image, we associate a feature vector based on the mean superpixel value containing this pixel, and the mean superpixel values containing neighbor pixels at three different scales. More precisely, we obtain three features for the central pixel (red, green and blue components), 3 × 4 features for the four (up, down, left, right) superpixels located at 25 pixels of the central one, 3 × 4 for superpixels at 50 pixels, and finally 3 × 4 for superpixels at 100 pixels. This process is applied on each pixel of the image to produce in total 39 feature maps. To obtain features at the border of the frame, a padding is applied to the image, where the border pixel is simply replicated. Using the superpixel mean value instead of the pixel value itself has several advantages: first, it makes the feature vector more robust to noise and sharp variations in the image. Second, it solves the problem of local averaging at the border of the objects: as pixels are grouped based on their consistency and oversegment objects, the color average is done within each segment. Thus, pixels at the border of an object will have the same representation as pixels at the center, as long as they belong to the same superpixel. Lastly, as features are representative of superpixels rather than a pixel, the descriptor is more robust to scale variations. We then base our saliency estimation on this set of features, pretty reduced compared to the one proposed in Make3D. However, we did not find much improvement in using the whole batch of features proposed by the method (using standard deviation types of features even decreased the performance), and the computation time was much higher.

Pros and cons

These features are conceptually very simple, easy to implement, and their extraction are pretty fast. Unlike Itti and Koch features, superpixels make an excellent distinction between objects and background, even at their periphery. As a drawback, they rely on RGB colors, thus making them very sensitive to illumination changes. In addition, the number of superpixels required by the algorithm depends on the average size of the objects of the scene, and has an impact on the feature quality.

Deep features

Motivations

Deep learning based on convolutional neural networks are now applied everywhere to solve computer vision problems. As discussed in Sections 3.3.1 and 3.3.2 of the previous chapter, saliency and object localization now have a bunch of techniques based on deep learning. A very nice property of such networks is their modularity, and their capacity to be re-used in many other types of computer vision problems by fine-tuning.

An end-to-end neural network architecture may be used for saliency learning, starting with convolutional layers and ending up with fully connected one. However, meta-parameters (learning rate, minibatch size, etc...) are hard to configure to allow an efficient incremental learning. A bad configuration could significantly deteriorate weights that were correctly learned for another problem. We then consider another approach to exploit deep neural networks: the use of the first layers of a well-trained network as a feature extractor. This way, we avoid instability problems, and the module is easily plugged in our architecture.

Description

In convolutional neural networks, the input image is send across the network and is filtered by a large number of convolutional kernels, whose parameters are determined by a feed-forward back-propagation procedure, similar to multi-layer perceptrons. The convolutional layers of a CNN can then be seen as a set of filters whose values have been optimized by the dataset it was trained for. Similar to Gabor filtering in Section 4.3.1, we here produce feature maps by a succession of convolutions that aim to discriminate salient features.

Several popular CNN-based architectures have been proposed to solve computer vision problems, and are publicly available with weights trained on very large datasets (ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], Coco [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], etc...). Among them, Alexnet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], GoogLeNet [START_REF] Szegedy | Going deeper with convolutions[END_REF] or more recently SegNet [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling[END_REF] are the most widely used networks. Most computer vision approaches use these networks with the proposed weights and remove the last layers of the architecture to fit their own needs. They next retrain the network with their own dataset (potentially much more modest than the initial ones), taking the weights trained for the other problem as a good initialization. This process is called fine-tuning (described in Section 3.4.2).

We base our feature extraction upon this idea of fine tuning by following the ideas of Zhou et al. [START_REF] Zhou | Learning Deep Features for Discriminative Localization[END_REF]. In their article, a GoogLeNet architecture is used and fine tuned to perform object localization. The end of the network is replaced by a global average pooling layer, followed by fully connected layers providing strong localization capacities and working on weakly supervised training data. This type or architecture is then able to produce, in some sense, class specific saliency maps. In addition, the weights of this network are publicly available.

We then use this available trained model and do not consider the layers after the global average pooling one. The feature extraction is done at the level of the To get a set of feature maps, we then feed the network with the original RGB input image, without resizing it, and extract the 1024 maps of the CAM layer (See Figure 4.6).

Because of striding and pooling in the network, the output feature maps have a resolution that is 16 times lower than the input image. To overcome this loss of resolution, we present in Section 4.5.2 a method to reconstruct saliency maps at the original scale.

Pros and cons

Deep features are very generic and robust features. They may have a better generalization capacity than the other proposed features. However, to be applied in real time on our images, a GPU is mandatory. Moreover, the successive convolutional layers produce a strong image downsampling, thus making small details hard to retrieve when computing the saliency map.

Salient objects discovery

To learn a model of saliency, we base our methods upon object detectors and classifiers that will constitute our learning signals. These learning signal have the properties to be reliable, but are only partial. The goal is then to generalize this signal to produce saliency maps in a more general case. Two different object detectors have been used in our two application scenarios.

Foveal stream

Contextual stream

CNN-based classifier

Recognition label CAR

Foveal label SALIENT Output mask

Context label UNDETERMINED

Foveated objects discovery

This approach has been designed for the BioVision platform and works on the principle of the foveated vision. The idea is to provide a saliency estimation in the fovea, and to transfer this information to the contextual view, as a segmentation mask integrating the foveal location. The method can then be applied on setups having both a foveal and contextual camera.

Figure 4.7 illustrates the general approach to obtain a segmentation mask in this context.

Object recognition in the fovea

The stream provided by the foveal camera is sent to an object classifier that has been described in Section 2.2.2. As a reminder, this classifier is based on a deep convolutional neural network and is able to identify 8 different categories of objects, plus a last class called "other", containing any other types of elements.

We take this classifier already trained, and simply use it to infer in real time the class of the object observed by the fovea. The classifier is considered robust enough to provide a reliable class estimation, that is to be used as our learning signal. In our case, we consider the eight objects as being salient (annotated with the "salient" label), and any other visual item as being "not salient".

Segmentation mask reconstruction

In this setup, the contextual view the main input stream, on which the saliency map is to be estimated. To make it compatible with the saliency learning algorithm, we then need to turn the object class estimation on the foveal image into a segmentation mask in the contextual view.

First, the label provided by the object recognizer of the fovea is tuned into a label compatible with our saliency learning framework. If the output label is among the nine objects, we transform it to a "salient" label. If the estimated class is "other" the new label is "not salient".

Second, the extrinsic calibration of the setup provides a correspondence between pixels of the fovea and pixels of the contextual view. Based on this information, we associate each pixel of the contextual view that are visible in the fovea with the label ("salient" or "not salient") found earlier. Everywhere else in the contextual image, we assign the label "undetermined").

Note that in this configuration, learning is weakly supervised at two different levels. First, a saliency label is available only within the fovea, so that the learner should use several points of view to be able to generalize the estimation out of the fovea. Second, labels "salient" or "not salient" are assigned on the whole fovea, even if the salient element only represent a fraction of the foveal image. To learn saliency, the classifier must learn by itself to discriminate between what is really salient within the fovea and what is not.

3D segmentation for object detection

This type of object detector has been designed for RGB-D cameras such as Kinect or Asus Xtion, and mainly works for indoor environments. The method is a modified version of the depth-based object segmenter of Caron et al. [START_REF] Caron | Neural network fusion of color, depth and location for object instance recognition on a mobile robot[END_REF] and is used to produce a partial but accurate estimation of the saliency. The depth map is the only component considered in the segmentation process.

General idea

The object segmenter detects objects lying on planar surfaces (typically tables or floor), with a diagonal size between 10 and 180 centimeters. As a first step, the depth map is turned into a point cloud. Depending on processing step, we either consider the 3D points of the cloud, or their corresponding pixels in the depth map. We then follow the procedure below, further detailed in the next paragraphs

• The major plane (in our case the floor or a tabletop) is detected and tracked during the whole sequence to make sure that the same surface is used during the whole experiment. For simplicity, we call this major plane the floor in these explanations.

• Given the floor position, potential walls are detected. Floor and wall are then filtered out, and the remaining points of the cloud are likely to be part of salient objects, but could as well be undetected portions of walls, poles, or any other artifact that is irrelevant for a robot.

• Remaining points are grouped in blobs to form object candidates. We then remove candidates that are either too small, too large, or too far from the floor.

• To avoid false detections from large objects cut by the border of the frame, only candidates having no contact with the border of the depth field of view are kept and considered as salient objects.

• Lastly, the segmentation procedure is converted into a 2D segmentation mask, further used to produce "salient" and "not salient" labels.

The main steps of the segmentation are summarized in Figure 4.8. We also synthesize in Table 4.1 the parameters of the method as well as their numerical values.

Plane tracking

The segmentation algorithm is using a plane detection as a pre-processing step. To detect the major plane of an image, the RANSAC algorithm [START_REF] Fischler | Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography[END_REF] applied on the whole point cloud of the frame. The detected major plane is then considered as the floor or tabletop. However, this is far from always being the case. Walls are very often detected this way and may conduct the algorithm to fail. In addition, computing a RANSAC from scratch on the whole image can be pretty slow, especially if the environment is complex. In Caron et al. [START_REF] Caron | Neural network fusion of color, depth and location for object instance recognition on a mobile robot[END_REF], the RANSAC computation is avoided by determining the floor equation prior to the experiment, and supposing that orientation is the same during the whole experiment.

To overcome these limitation, we develop a method of plane tracking that returns after each frame an equation of this plane. Tracking is initialized by detecting the main plane in the whole image, and re-initialized the same way each time the tracking is lost. After initialization, the pixel position of the inliers (the points belonging to the plane) are turned into a binary mask (main floor/not main floor). Then, given this binary mask, the next frame is processed by applying a RANSAC to points of the cloud whose pixels of the mask are positive. Given the asumption that two successive frames are similar enough, this masking procedure provides an excellent initialization for the RANSAC. This way, the algorithm converges much faster and is almost sure to find the right plane, even when the major plane of the image has changed. An even better mask is obtained by filtering out pixels whose color value between the two successive frames are too far from each other. After each plane estimation, the points belonging to this plane our of the mask are retrieved, and a new mask is determined for the next frame. The main steps of this algorithm are illustrated in Figure 4.9.

Unreliable data filtering

As a first processing step, we remove points of the cloud that are likely to be hard to process, or for which data is not fully accessible.

Local normals are first estimated for each point of the cloud. However, pixels with too strong variations in their neighborhood (typically at the border of an object) cannot be assigned a normal and are therefore removed from the point cloud. They are represented in green in Figure 4.8. Second, points that were not assigned a numerical value, due to reflective surfaces, shadows (red color in the figure), or projection of the depth map on the rgb frame (denoted "Out of FoV", in blue in the

Floor and wall removal

The plane equation provided by the plane tracker is a set of four values a, b, c and d so that each point (x, y, z) of the planes verifies the equation

ax + by + cz + d = 0 a 2 + b 2 + c 2 = 1 (4.6)
Points belonging to the floor are first removed. We identify the points of the cloud that verify the conditions

|ax + by + cz + d| < |n x a + n y b + n z c| > τ (4.7)
with ≈ 0, > 0 and τ ≈ 1, τ < 1 two threshold values, and (n x , n y , n z ) the unit normal vector associated with (x, y, z). Those constraints forces both the point (x, y, z) to be close enough to the theoretical plane, and to have a normal (n x , n y , n z ) close enough to the plane normal (dot product close to 1).

Then, walls are identified and filtered out. We define walls as being large planes orthogonal to the floor plane, and having a contact with the border of the image. To detect them we run the RANSAC algorithm on the remaining point cloud. Whenever a plane is detected this way, the orthogonality condition is verified by the dot product between the plane normal and the floor normal:

aa + bb + cc < δ (4.8)
with δ ≈ 0, δ > 0 a threshold value. If the plane satisfies the equation, and if a portion of this plane is close enough to the border of the depth field of view, it is considered as a wall and removed from the point cloud. Otherwise, those points are removed from the next RANSAC computation, and another plane is detected on the remaining point cloud. When the planes found have a too small number of pixels, the procedure is stopped.

Blobs segmentation

The remaining point cloud (recall that it contains points close enough to the kinect, being neither part of the floor, nor of the wall, and whose normal could be estimated) is turned into a set of blobs based on image processing. For this step, we consider the pixels corresponding to each 3D point of the cloud rather than the 3D points themselves. This way, we can apply simple image processing segmentation.

Recall that points at the border of an object are not considered because of their sharp depth variation in their neighborhood. This way, filtering them out of the point cloud makes a natural delimitation between objects (colored in red in Figure 4.8, section blobs segmentation). The blob segmentation step them simply consists in finding the connected components of the remaining pixels of the image. The set of 3D points associated with a connected component is now called a blob (or a cluster), and is used in the next processing steps.

These blobs are also used to generate the SegBoxes (see Section 4.5.3), considering their bounding boxes in the depth frame rather than their pixels.

Blobs filtering

A set of filters are then applied to the remaining blobs:

• Blobs whose centroids are too close to the floor or to a wall are filtered out as they are likely to be false RANSAC outliers.

• Blobs having a too large diagonal, or having a bottom too far from the floor are removed.

• Blobs having a contact with the border of the depth field of view are filtered out.

An optional final step is applied to clip together blobs belonging to the same object that would have been divided by the image-based segmentation (this is, for example, often the case for desk chairs like in Figure 4.8, section blobs filtering). For this step, we project all the remaining blobs to the floor, and we use a KD-Treebased clustering to group pixels that are close enough or overlapping each other.

Object segmentation mask reconstruction

To convert this segmentation procedure into a 2-D mask compatible with saliency learning, we project the whole point cloud back to the image frame and attribute a label value to each corresponding pixel. In these examples, pixels having no corresponding point cloud (because of reflectance, shadows or because visible points are too far from the Kinect) are labeled "unavailable" (dark gray in Figure 4.8, section mask construction). Points of the cloud that either belong to the major plane or to a wall are labeled "not salient" (colored in black in the figure). Points of the cloud belonging to a cluster detected as a salient object is labeled "salient" (colored in white). Lastly, all remaining points are categorized "undetermined" (light gray), as the algorithm was not able to determine their actual state of saliency. This includes candidates that are too close to a border and could be either walls or objects, or candidates bigger or smaller than the tolerance threshold, that could either be several

Saliency learning and saliency exploitation

In this section, we describe how feature extraction and saliency discovery are integrated into a learning framework to produce a saliency model. Moreover, we explain how saliency maps are retrieved based on this model, and how they can be exploited to produce bounding boxes around salient objects.

Incremental learning

Given, on the one hand, features continuously extracted from the main visual stream and, on the other hand, segmentation masks providing a partial estimation of the saliency, we integrate these cues into an online classifier to build a saliency model.

Data formatting

Learning is done at the pixel level, by turning the incoming feature maps and segmentation masks into a set of labels and features. Figure 4.10 illustrates this procedure.

First of all, the segmentation mask is resized at the same size as the feature maps. This way, each pixel of the feature maps and of the mask correspond to the same area of the visual field. Then, for a given pixel, the n features (n being the number of feature maps) are collected and turned into a feature vector, and the corresponding mask pixel value (which stands for a label) is retrieved. This way, a pair of features-label is associated with this pixel. We call it a sample in the following explanations.

The data send to the classifier is then a matrix of features and labels representing the samples collected at different pixel locations. However, creating a sample at each pixel would produce a very large amount of very redundant data to send to the classifier. For this reason, we downsample the number of extracted samples, by considering only the central pixels of the cells of a regular grid. In addition, pixels classified "unavailable" or "undetermined" cannot constitute relevant samples for saliency learning. Therefore, only samples labeled "salient" or "not salient" in the segmentation mask are selected to feed the saliency model.

Classifier

After each new observation, the classifier is updated based on the new incoming samples. We therefore need this update to be fast enough and stable in time. In spite of a few attempts with neural networks, we decided to use a random forest classifier that provided much better results while being decently fast to re-train. Random forests have been introduced by Breiman et al. [START_REF] Breiman | Random forests[END_REF] and consist in a bagging approach based on an ensemble of decision trees. Unlike decision trees, random forests have the property of not overfitting their training set. In the case of classification with numerical attributes (called features in this text), a decision tree is a binary tree in which samples of the dataset are propagated to fall in a leaf providing an estimated label. Figure 4.11 is an illustration support for the following explanations.

Each node d of the tree is associated with a feature F d and a numerical threshold th d . To determine the label of a given sample V, the numerical value F 11 (V) of the feature of the root node is extracted, and compared with the root threshold th 11 . Depending whether F 11 (V) > th 11 or not, V ends up in the right of left node of the next level. In this node, the next feature F 2d (V) is considered and compared with threshold th 2d , and so on until V reaches a leaf node. This leaf contains statistics about the labels of the training samples that have reached this node. Thus, the probability of a sample falling in a leaf L to be of class C is given by

P r L (Cl = C) = 1 |T L | s∈T L Cl(s) = C (4.9)
where index [START_REF] Gini | Concentration and dependency ratios[END_REF]. The complexity of such algorithm is O(mn), n being the number of samples in the training set and m the number of features.

A random forest is constituted with a set of decision trees, each of them being trained with a fraction of the training set. This way, each tree is unique and has its own statistics about class probabilities. To retrieve the probability of a sample V to belong to a class C, the following formula is used

P r(Cl = C|V) = P r(Cl = C) × 1 N N i=1 P r L i (V) (Cl = C) (4.10)
N being the number of trees in the forest, P r(Cl = C) being a prior on the class distribution, and L i (V) being the leaf node of tree i in which V is falling. The major drawback of random forests in our system is that they are not designed for online training. No available version [START_REF] Saffari | On-line random forests[END_REF], [START_REF] Lakshminarayanan | Mondrian forests: Efficient online random forests[END_REF] of online random forest was satisfying in terms of speed and performance, so that we adapted the offline version to make re-training fast enough.

To re-train random forests incrementally, we therefore use the following procedure: first, the whole dataset of extracted samples is kept in memory. To avoid memory issues or too slow updates, we forget the oldest samples when the training set exceeds a certain limit. Then, we update the classifier by only re-training a small fraction of the forest at a time: we randomly select K trees among the N in the forest, and we retrain those trees with a random fraction of the dataset available in memory. As a result, after each update, the classifier is able to estimate the saliency of an input based on the model trained with the previous observations.

Parameters

Our learning system depends on a certain number of parameters that must be carefully chosen. Most of them were empirically determined to allow a good trade-off between computation speed and classifier accuracy. We summarize in Table 4.2 the main parameters and their numerical values used in our implementation. Aside from these parameters, a statistical prior was added on the classes. In practice, the fraction of "salient" samples is around 5%, thus making the dataset highly unbalanced. We believe this is one the reason why neural networks failed at estimating saliency. To handle unbalanced dataset, the a prior probability of 0.95 was attributed to the "salient" class.

In Table 4.2, we define a parameter that restricts the max number of samples M at each observation. This parameter defines the cell length of the grids used to downsample the pixels from which we extract features and labels ( W ×H C 2 = M , C being the cell length and W, H the image width and height). In practice however, this max number of samples is never reached, as a large portion of the image is labeled "unavailable" or "undetermined". The number of samples actually sent to the classifier after an observation is more realistically around 200.

Saliency inference and map reconstruction Inference and saliency map reconstruction

The saliency maps are constructed by applying the saliency model to RGB images of the main visual stream. More precisely, the saliency map is obtained as follows: we first generate feature maps from the input RGB image, and associate each pixel of these maps with a feature vector, similar to the procedure of Section 4.5.1. For a pixel P (x, y) with associated feature vector V (x, y), we evaluate the probability of the pixel to be salient based on the classifier output. The random forest produces a probability for each class. As our problem is a binary one, we then obtain the probability of P (x, y) to be salient by

S(x, y) = P r(Salient|V(x,y)) = 1 N N i=1 P r i (Salient|V(x,y)) (4.11)
we then generate the saliency map by gathering the S(x, y) results in an image of the same size as the feature maps. Unlike the partial estimation of the segmentation mask, we here obtain an estimation of the saliency for each pixel of the image. Figure 4.12 illustrates the construction of the saliency map.

Superpixel-based saliency recovery

The produced saliency map has the same resolution as the feature maps. For two of the three presented features, this has no influence on the output, but for feature maps produced with CNNs (see details in Section 4.3.3, recall that the deep feature extraction downsamples the image by a factor 16), the output saliency has a very low resolution. To rescale this low-resolution saliency map to the original input size, we generate SEEDS superpixels [START_REF] Van Den Bergh | Seeds: Superpixels extracted via energy-driven sampling[END_REF] from the original image (350 superpixels for 640×480 images in our experiments). Our goal is now to associate a saliency value to each of these superpixels. For that, we resize the saliency map at the original scale using a nearest neighbor interpolation approach, and we calculate the average saliency of this map within each superpixel. Mathematically, for a superpixel P such that A P = {(x, y) ∈ P } is the set of pixels contained in P , the superpixel saliency map S sp is such that

Superpixel average reconstruction

S sp (A P ) = 1 |A P | x,y∈P S(x, y) (4.12)
Using this approach, the produced saliency map looks nicer, as saliency at the border of the objects is preserved. Figure 4.13 illustrate the procedure described in this Section.

Bounding box proposals

The saliency map provides an indication of the interestingness of a given pixel. In order to localize objects in an image, an additional step is necessary to group salient pixels into object candidates. To this end, we use two types of bounding box proposals, and we associate them a score that is strongly related to the obtained saliency map. This score is then used as a mean to determine boxes that are the most likely to contain a objects, or to reject boxes having a very low score.

EdgeBoxes

The first bounding boxes are obtained by the EdgeBoxes [START_REF] Zitnick | Edge boxes: Locating object proposals from edges[END_REF] algorithm. The edge-Boxes are among the existing techniques of agnostic bounding boxes, described in Section 3.3.2. They rely on a simple criterion of contours to produce these boxes, and are very fast to compute compared with most existing techniques. In addition, they are among the most efficient techniques of this field, and produce a confidence score.

To exploit this algorithm in our implementation, we compute for a given RGB input the 100 most likely EdgeBoxes, and store their associated edge confidence score, called h in b in the reference paper [START_REF] Zitnick | Edge boxes: Locating object proposals from edges[END_REF]. This score is already a good measure

SegBoxes

The second type of bounding boxes can only be obtained from RGB-D data, as they are obtained with the 3D object segmentation described in Section 4.4.2. To make explanations more clear, we call the bounding boxes obtained this way the SegBoxes.

During segmentation, pixels of the depth map are grouped in blobs in order to create object candidates. Among object candidates, some are labeled "salient", some should be salient objects but are labeled "undetermined" because they touch a border or are poorly segmented, and some are just artifacts. We define the Segboxes as the 2D bounding boxes around all blobs (labeled "salient" or "undetermined") of the segmentation mask. Additionally, blobs that were clipped together when projected to the floor plane are also added to the SegBoxes list.

Saliency score and re-ranking procedure

For both EdgeBoxes and SegBoxes, we associate each box B with a score related to saliency (called here the saliency consistency score, or SCscore), representing the ratio of salient pixels in the box:

SCscore(B) = 1 w B × h B i,j∈B S(i, j) (4.13)
where S(i, j) is the saliency of the pixel at (i, j) , obtained from the saliency map and w B and h B are the width and height of B. The highest the score is for a given box, the most likely it is to contain a salient object.

For the EdgeBoxes, the SCscore is multiplied by h in b . This way, small boxes found within a salient object might be rejected if the h in b score is low enough. Lastly we filter out Segboxes and Edgeboxes whose final score is below a certain threshold. In our dataset, we found SCscore = 0.2 for SegBoxes and SCscore×h in b = 0.01 to be good trade-off thresholds between false alarms and false rejections.

Conclusion

In this chapter, we have presented the technical details of a method for learning a model of saliency incrementally. This approach is designed to be run on a robot exploring its environment. The saliency model is then improved as the robot moves and receives new observations.

The method is based on three modules. A feature extraction module processes an RGB stream and extracts relevant patterns to determine the state of saliency at a pixel level. A saliency discovery module provides, when possible, and estimation of the saliency that is reliable and converted to labels. Lastly, an online classifiers processes both labels and features to learn a model of saliency.

Once a model of saliency is available, the robot can now infer from the whole RGB image a state of saliency that is specialized for the explored environment. Lastly, the estimated saliency maps can also be used to produce bounding box proposals very likely to contain salient objects.

To demonstrate the modularity and the generalization of our approach, we have proposed different types of feature extraction, each having advantages and drawbacks. We also developed for two different setups, two strategies for discovering salient elements in the environment.

However, another question is raised to fully exploit the capacities of our technique: which exploration strategy should the robot adopt to increase and speed up the accuracy of the saliency model ? The problem of selecting relevant targets that the robot should explore constitutes the main problems of Part II of this manuscript.

Chapter 5

Experimental results

Introduction

The previous chapter was presenting the technical details of the proposed method to incrementally learn visual saliency. We propose in this chapter to evaluate the performance, behaviors and limitations of this approach. To this end, we rely on the four datasets described in Chapter 2, and conduct a set of experiment to evaluate different aspects of the method.

First, we study the incremental and self-supervised aspect of the method. We measure the execution time of each component and provide qualitative results on the evolution of the saliency model in time. In addition, we discuss the selfsupervision process by quantitatively evaluating its reliability. We also demonstrate the ability of the learning process to generalize over the learning signal to create a reliable saliency model even when the supervision signal was partly missing, weak, or even wrong.

The second aspect of our evaluation is related to the quality of the saliency itself, once the model has been learned. We then evaluate the quality of each feature extraction technique, and the performance of our method versus state-of-the-art. Lastly, we show that our saliency approach can be used to produce bounding box proposals around salient object more efficiently than agnostic bounding box proposals methods would.

Setups and datasets

Our saliency learning technique is evaluated on the four datasets described in Chapter 2. We here provide a few recalls and explanations on the annotation procedure in lights of the method proposed in the previous chapter. Indeed, defining annotations that are relevant and compatible with our method is not completely straightforward.

Requirements for evaluation

Nature of the input images

Our method for learning saliency was developed for setups having at least two synchronized streams, called the main and addition visual streams in the previous chapter. Therefore, to train our system, we need datasets containing synchronized images from those two streams.

We described in our work two possible cases. The first one is the use of an RGB-D stream, where the RGB component is used as the main visual stream, and the depth as the additional one. The second one relies on a contextual camera playing the role of the main visual stream, and a foveal camera for the additional stream.

Our datasets are designed for those two configurations: The ENSTA dataset, RGB-D scenes dataset and Ciptadi dataset contain RGB-D images, while the BioVision dataset contains both foveal and contextual images.

Self-supervised training and objective evaluation

Our method is based on a self-supervised learning technique. This means that training is done without prior available annotation. We can then train our method from a dataset having only input images, and do not need annotations for the whole dataset.

Nevertheless, we also want an objective evaluation of our system, and cannot consider the self-annotations as a valid ground truth. Indeed, the self-annotations are only partial, and are not necessarily perfect (for example, the object classifier may misclassify some objects, the depth segmentation may fail sometimes, etc...). Therefore, we also need a portion of the available dataset to be associated with objective annotations to compare the produced saliency maps (and potentially our self-annotations) with a ground truth.

We therefore divided the BioVision, ENSTA, and RGB-D scene datasets into a training and a testing set. The training set constitutes the large majority of the dataset and is not associated with any annotation. Conversely, the testing set is composed with a small number of frames (100 to 150), all having an objective ground truth annotation.

Lastly, we wanted to test the generalization of our saliency models on environments they were not trained for. To this end, we relied on the Ciptadi dataset, composed of 80 annotated frames. In this case, training is done independently on another dataset, and the dataset is fully dedicated for evaluation.

Relevant annotations

Our last constraint was to define relevant annotations, compatible with our saliency models. Recall that our saliency technique produces top-down saliency maps focused on object in indoor environments. As mentioned in Chapter 3, these kind of datasets are not extremely frequent, and the teams evaluating top-down models of saliency usually build their own datasets to fit their own needs.

But there is more: the saliency maps produced with our technique are directly based on the supervision signal used to train the model. The annotations used for evaluation must then be compatible with our definition of saliency, represented by our learning signal. We then provide in the next section the annotation procedure used in each of our datasets.

Annotation procedure BioVision dataset

In the BioVision dataset, the learning signal is produced from an object classifier, trained on eight different objects and a last class containing background elements. We define salient element as being exactly the eight objects recognized by the classifier. Thus, any distractor such as pens or notebooks are not supposed to be learned by our model as salient elements.

Therefore, we constructed our ground truth by manually drawing segmentation masks around those eight objects. Please refer to Section 2.

ENSTA dataset

In the ENSTA dataset, a robot moves across a building and visualizes objects on the floor. The learning signal is obtained from a depth-based segmentation technique (described in Section 4.4.2) that isolates any element of a certain size (between 10 and 180 cm) on the floor. This includes for example desks, chairs or boxes. However, objects hanged on the walls (heaters, sprinkler, posters) are not considered as being salient.

Similar to the BioVision dataset, we manually annotated the ENSTA dataset by creating ground truth masks around those objects. We also needed ground truth bounding boxes around objects to evaluate our bounding box proposals method. Figure 5.1 provides sample frames of this dataset along with their ground truth masks and bounding boxes. We also display the result of the depth-based segmentation to get an idea of the learning signal on which our saliency technique is learning (recall that gray areas of this segmentation mask are either "unavailable" or "unknown" pixels, that are not integrated in the saliency process).

RGB-D scenes dataset

In the RGB-D scenes dataset, the learning signal is obtained from the same depthbased segmentation algorithm. However, we here look at objects lying on tabletops, so that detected objects are not the tables, but the objects on them (caps, cereal boxes, etc...). Therefore, chairs or objects out of the tabletop are not considered as salient elements.

The RGB-D scene is a publicly available dataset that already contains annotations. However, these annotations are not perfectly compatible with our definition of saliency. Indeed, only a few categories of objects are annotated, while other are considered as distractors. As our segmentation technique detects all objects lying on tabletops, a lot of them would have been classified salient by our algorithm and not salient by the dataset. We then had to manually re-annotate them. We followed the exact same procedure as for the ENSTA dataset to this end. Figure 5.2 illustrates a few examples of the original annotations, the result of the segmentation mask, and our own annotations.

To use the original annotations, an alternative solution would have consisted in using a specialized object detector for the objects originally annotated. The design would then have been very close to the BioVision setup. However, we did not exploited this idea in this work.

Ciptadi dataset

To make sure that our annotation system is not biased too much and that our saliency technique still provides relevant results in a general case, we used a last dataset on which annotations were already provided: the Ciptadi dataset. In this case, the ground truth of the dataset does not perfectly match our definition of saliency.

Incremental and self-supervised learning

To get a good insight of the way saliency is learned and how much the classifier can generalize the state of saliency given a partial learning signal, we provide qualitative and quantitative results.

Self-supervised learning Evaluation of the segmentation process

A first set of experiments were conducted on the three datasets, BioVision, ENSTA and RGB-D scenes, to evaluate the performance of the self-supervision process, corresponding to the creation of a segmentation mask. To this end, we used the manually annotated ground truth mask of each dataset that we compared with the automatically generated segmentation masks. The segmentation technique used for Bio-Vision was the one based on object classification (described in Section 4.4.1), while the depth-based technique of Section 4.4.2 was used for the two other datasets.

Experimental results are reported in Table 5.1. The first measure (available data) is the percentage of pixels labeled either "salient" or "not salient" over the total number of pixels. The BioVision dataset has a significantly small (0.5%) portion of the image available for labeling, while the other datasets are around 45%. Second, we measured the percentage of salient pixels among pixels for which data was available. This time, ENSTA and RGB-D scenes have a very small fraction (around 5%) of the pixels labeled salient. This confirms that the training dataset of the online classifier is highly unbalanced.

Lastly, we used the ground truth masks to measure precision and recall on a pixelwise basis. We here make a distinction between relative and absolute measures.

The relative precision and relative recall are obtained by considering only pixels for which segmentation provided a "salient" or "not salient" label. This measure is important as it reflects the quality of our learning signal: the precision is very high in both ENSTA and RGB-D scenes datasets, thus showing that the segmentation process is reliable. The relatively low recall is explained by two factors: on the one hand, segmentation is such that the bottom of the segmented objects are often considered as part of the floor. For small objects such as the one contained in RGB-D scenes, this may represent a non negligible portion of the object. On the other hand, big and flat objects (such as computer screens or shelves) are sometimes detected as walls, thus making objects of the ENSTA dataset potentially misdetected. Another interesting point is the low precision rate for the BioVision dataset, as the whole fovea is labeled with a single value. For example, if a salient object is contained in only 20% of the fovea, and if the provided label is "salient", the precision measure will be 20% whereas the label was correctly provided. A more relevant measure for BioVision is then to consider the accuracy of the object detector, which is of 94%. Nevertheless, this behavior does not prevent the classifier to have a good generalization capacity, as we discuss in the following sections.

The overall precision and overall recall are measured by comparing the ground truth with the whole segmentation mask. This measures reflects more the salient detection capacity of the segmentation process. As a large portion of the segmentation mask is not available, the overall recall are much lower (the precision is the same in both cases as the same number of false positives are obtained). Nevertheless, this poor overall performance does not influence the learning quality of our models.

Generalization capacities

We now demonstrate the capacity of the saliency model to generalize what was learned from the segmentation mask, and to produce a reliable estimation of the saliency on the whole image. This generalization is made possible by two factors: first salient objects often show common visual properties. Generalization from depth segmentation: Figure 5.3, second row, shows a set of segmentation results. Recall that for a segmentation mask, black and white pixels represent "not salient" and "salient" portions of the image. Grey pixels of the segmentation mask represent areas where the algorithm could not clearly determine the state of saliency. Except cases where the segmentation fails (sample 6) because of a bad plane estimation, the segmentation mask produces a pretty reliable saliency segmentation. However, this segmentation is only partial because of the many undetermined areas, thus making the incremental learning of saliency (third row) useful to recover missing data. For example, the segmentation algorithm is such that nothing salient can be detected further than 4 meters away (samples 4, 5), but saliency estimation is applied on the whole image and the generalization capability of the classifier makes it possible to detect salient objects at more than four meters. Second, reflective surfaces are often hard to detect by the Kinect sensor (computer screen on sample 3, black plastic on sample 2). However, the aspect of salient reflective objects can be partially learned and fully retrieved based on the RGB data only. Third, the segmentation algorithm is very restrictive and is often not able to detect salient elements if they are in contact with a border of the image (samples 5, 1, 2) or badly clustered by the segmentation (sample 1). Conversely, the saliency algorithm provides an estimate of saliency even if the object is partially cut, occluded, or captured with a poor image quality.

Generalization from foveal observation:

To demonstrate the generalization capability from foveal observations, we train and evaluate the model on the same sequence. To provide a better visual representation of the way saliency is mapped with objects of the room, we represent saliency map as a heatmap for which red are the most salient areas and blue are the least salient ones. In Figure 5.4, we display a sample frame (cropped around salient elements of the scene) and some foveal observations used to feed the saliency model. All of these observations (and therefore, each pixels within the fovea) were labeled "salient" by the object recognizer, while it is clear that a large fraction of the pixels are not part of salient objects. Nevertheless, when looking at the resulting saliency map, the classifier was able to discriminate areas that were part of the objects, and areas from the background (walls behind or table). Of course, the discrimination process is not perfect: elements such as plastic bottle were not clearly learned as salient, and portions of the wall behind objects were partially classified salient, but the weak supervision process seems to be working reasonably well. In figure 5.5, we point out two additional features of the saliency model. First, distractors such as the pen (sample 1) that would be naturally salient are correctly classified not salient. Second, the classifier is able to retrieve similarities in areas that have not been observed by the fovea. In sample 2, a computer keyboard and monitor are detected at the other side of the room.

Temporal aspect of the saliency learning

Our method has been designed to run online during the robot's exploration. It is therefore of paramount importance that the algorithm is executed in a reasonable amount of time, and to make sure that the saliency quality gets better as more examples are provided to the system. We cover these aspects in this section.

Execution time

The runtime performance was measured on each components of the method and is summarized in Table 5.2. To this end, we ran different sequences from the BioVision dataset on Ubuntu 14.4 with an Intel Core i3-3240, CPU at 3.4GHz quadcore processor. The code was implemented in C++ and adapted to be used under the ROS framework.

Feature extraction first depends on the type of feature considered, but also on the type of device used. For this device, the deep features are too long to process to be used in a real time scenario. However, the processing time is much more reasonable on a low-cost GPU (Quadro K600) and is particularly low on a powerful GPU (GTX Titan X). The segmentation mask is obtained either from a geometrical approach based on the depth map, or from an object classifier based on CNN. The geometrical approach has an execution time that depends on the size of the major plane, the number of objects in the scene, or the difficulty to find walls. When the tracking is lost, additional time is also required to detect the major plane again. Segmentation based on the foveal observation mainly depends on the time required by the CNN to identify the class of the foveal image. The network is such that foveal images are resized to 240×240, thus making inference much faster than on 640×480 contextual images.

As explained in the technical Section 4.5.1, the classifier update mainly depends on the number of input features and samples. As random forest are not incremental classifiers, trees are re-trained from scratch with the whole dataset that is constantly increasing. Figure 5.6 represents the time measured for a classifier update depending on the number of samples in the training set. We evaluate the three types of features this way (recall that Make3D produces 39 feature vectors, Itti and Koch 42 and the deep features 1024). For 100 000 samples (above this number of samples, we randomly remove old examples), the classifier update is around one second for both Make3D and Itti and Koch features, while it is around 13 seconds for the deep features. We also clearly identify the linear relation between the execution time and number of samples. To enable a continuous sample acquisition and saliency evaluation during the classifier update, the classifier is trained on a separate thread (and a dedicated ROS node).

The time required to extract features is not considered in our measure of saliency estimation time. Saliency inference with a random forest is linear with the depth of the trees, it is then very fast. However, to construct the saliency map, this inference is done on each pixel of the image. For deep features, only 14 ms are required to generate the whole saliency map, as the resolution of the feature map is very low (40 × 30). For Itti and Make3D, we construct the saliency map by inferring only one pixel out of four. Processing time is then around 40 ms in this case. Last when using superpixels to retrieve the original resolution, the superpixels extraction requires an additional 60 ms.

Saliency quality evolution

We now look at the evolution of the saliency quality during the incremental learning. To get an idea of the way saliency is evolving as new observations are taken After a single observation, the classifier is usually not able to produce a saliency map. After 5 observation, the produced saliency is very noisy but already able to detect portions of salient objects (the back of the chair in sample 304 for example). The model then makes fast progresses, and in less than 100 observations, the main salient elements of the scene are identified by the model, even before they are discovered by the robot. This property is due to the fact that salient object show common visual properties that can be distinguished from non salient elements. Between 100 and 400 observations, the saliency model is slowly refined. This time, the model makes more fine-grain improvements, by discovering each object separately and learning their visual properties in deeper details. A good example of this behavior is illustrated by frame 335: the mobile container at the back of the room is not naturally salient (almost white with a white wall behind). It is therefore not strongly identified by the saliency model after 100 observations. However, after 400 observations, the robot has reached a similar point of view, and the segmentation process has detected the container. The saliency model has then integrated the visual properties of the box, which is now recognized as salient. Conversely, the extinguisher of frame 260 is red and shows some salient properties. In observation 100, its state of saliency is not clearly defined by the model. After 400 observations, the robot has visited a similar point of view where the segmentation process identified the extinguisher as a part of the wall, and the model integrated the fact that this element is not salient. The model keeps evolving after 400 observations, but the visual evolution is not significant anymore.

Two videos representing the evolution of the saliency are available to better visualize this learning process. The first video1 exactly represent the aforementioned experiment. Note that this video is not a live video. Training was done offline by updating the classifier after each frame. The second video2 is a live one and represents a similar experiment with the BioVision setup and the Make3D features. Chapter 8 is dedicated to the numerical evaluation of the saliency evolution, and considers several exploration strategies for that. We therefore limit this section to qualitative evaluation, and refer to this chapter for a more quantified one.

Comparison with state-of-the-art

To evaluate the saliency model versus existing state-of-the-art approaches, we analyze the final performance reached by the classifier when all samples of the training set are used. When not specified, all experiments conducted in this section were obtained with the deep feature extraction process (detailed in Section 4.3.3) and the superpixel refinement (detailed in Section 4.5.2). We also denote in this section our incremental saliency learning approach and produced saliency maps as ISL.

Saliency maps evaluation

Experiments and metrics

We measure the saliency performance based on the ROC curve evaluation. The ROC (receiver operating characteristic) curve is among the most common techniques to evaluate saliency [START_REF] Riche | Saliency and human fixations: State-of-the-art and study of comparison metrics[END_REF]. The idea is to construct a curve representing the true positive rate versus the false positive rate of the method. Theoretically, a method having a higher area under the ROC curve should be more efficient.

To obtain such a ROC curve, we rely on the ground truth masks of each dataset, that we compare with a saliency map. A saliency map is composed with values between 0 and 1. The procedure then consists in applying thresholds to the saliency maps at regular intervals (10 thresholds in our case). Each threshold value provides a binary image that can be compared with the ground truth to produce a measure of true positive rate (TPR) and false positive rate (FPR) 3 by a pixelwise comparison.

The same procedure is applied for each threshold of an image. As a threshold produces a specific binary image, the batch of thresholds produces a set of TPR-FPR points that are used to draw the ROC curve of the image. To obtain the ROC curve of a dataset, the FPR values are discretized in bins. TPRs are collected in each bins and the average TPR value is calculated within each bin. The ROC curve is then plot such that the abscissa represents the FPR bins, and ordinate the corresponding average TPR.

Features performance

We first evaluate our three different features on the three datasets by producing the associated ROC curves. These results are reported in Figure 5.8. The three types of features are denoted as Deep features, Make3D and Itti and Koch. To simplify the comparison with results of the next section, we also report the performance of the method called CAM [START_REF] Zhou | Learning Deep Features for Discriminative Localization[END_REF]. In each dataset, the deep features are always the best performing ones. The performance of the Make3D features are varying a lot depending on the datasets. This is most likely because these features have a moderate generalization capability, while deep features are known to be very good at it.

We also provide visual examples in Figure 5.9 to better understand the performances. On BioVision (sample 2), the deep features are the only ones able to correctly estimate saliency and to properly generalize. This is particularly visible in this dataset as the fovea provides a very localized and restricted learning signal. On the two other datasets, the deep features seems to be visually worse than the other. This is explained by the low resolution of the classifier unable to retrieve fine details (for example, the chair leg of sample 3). However, the ROC score is computed at the global image scale and does not really penalize fine details. Again, to demonstrate the good generalization capability of the deep features, we provide in sample 4 the very complex case of a chessboard, where salient elements (the pawns) are of the same color than the ground (black and white squares). In this case, both Itti and Make3D features fail at discriminating salient and non salient elements, while the deep features can.

State-of-the-art performance

To demonstrate the accuracy of our saliency model, we use the four datasets for which we compute ROC curves. For BioVision, ENSTA and RGB-D scenes, the results were obtained by training ISL on their associated training set, and evaluating on their evaluation set. For the Ciptadi dataset, we trained our model on RGB-D scenes, which is the dataset having the most similar points of view, and we compute the ROC curve based on Ciptadi's images and ground truth masks.

For comparison, we select three publicly available saliency algorithms. First, BMS [START_REF] Zhang | Saliency detection: A boolean map approach[END_REF] saliency benchmark [START_REF] Bylinskii | Mit saliency benchmark[END_REF]. We use BMS with the configuration that highlight salient objects rather than salient fixations. Second, we use the new version of the VOCUS2 algorithm [START_REF] Frintrop | Traditional saliency reloaded: A good old model in new shape[END_REF] along with the configuration file dedicated to the task of object detection in cluttered scenes (top-down saliency). Lastly, we compare our method with saliency maps produced with the CAM [START_REF] Zhou | Learning Deep Features for Discriminative Localization[END_REF] model. This model is trained to detect objects among the 1000 classes of ILSVRC. For a fair comparison, we disabled classes that were not present in the images of our datasets (i.e. their output score were systematically set to 0), so that the produced saliency maps were responsive to relevant objects only. In addition, the maps produced by the CAM approach have the same low resolution than our model. We therefore apply the superpixels refinement technique presented in Section 4.5.2 to increase the resolution of these maps.

For the ENSTA and RGB-D scenes datasets, we also provide the performance of the segmentation. Recall that the segmentation mask provides only a partial estimation of the saliency. For that reason, we replace pixels that were neither labeled "salient" nor "not salient" by the procedure by a random value. Lastly, Ciptadi's dataset also comes with associated saliency maps extracted by the method described in their article [START_REF] Ciptadi | An in depth view of saliency[END_REF]. As their source code is not available, we only report their results on their dataset. This method has the particularity of being obtained from RGB-D images rather than RGB only.

The results of the ROC-based evaluation are reported in Figure 5.10 and suggest that ISL significantly outperforms the evaluated bottom-up and top-down techniques on all datasets. This result was expected for at least the three first datasets, because our model is trained for a dedicated environment, from a learning signal that is close to the ground truth. However, the good performance on Ciptadi's dataset shows that the method is still usable in environments in which it was not trained. We also observe that all of the evaluated techniques outperform the depth segmentation. This is because the ROC curve is estimated on the whole image, whereas pixels without saliency estimation in the segmentation mask (more than 50% of them) are replaced by noise.

In figure 5.11, a visual comparison between the methods is presented. Samples 1 and 2 are from the Ciptadi dataset. For these samples, the results provided by ISL do not look as neat as in the other datasets. This is because ISL is used in an environment it was not trained for. Sample 3 is taken from the BioVision dataset and represents an image with a highly textured background. In this case, ISL significantly outperforms other approaches as it is the only one for which these elements are part of the model and clearly defined as not salient items. Regarding the performance of ISL for the other samples, the superpixel reconstruction approach makes it possible to retrieve shapes of salient objects (in spite of the low-resolution feature maps produced by the output of the CNN). When applied on the CAM saliency map, the superpixel reconstruction does not provide such good results. This might be because the produced saliency is much more diffuse (as a comparison, the CAM algorithm is displayed samples 1, 2 and 3 without superpixel reconstruction). Second, ISL is learned from a segmentation derived from a depth map. This way, salient and not salient elements are determined from geometrical criteria rather than from RGB textures. As a results, ISL avoids the detection of distractors such as windows or trees outside (sample 6), or red power outlet (samples 5,7), that are visually salient but irrelevant for an indoor mobile robot. Lastly, it enhances elements that are not naturally salient (mobile container on sample 7) but consistent with our definition.

Bounding box proposals

Experiments

We now demonstrate that ISL can be used to produce relevant bounding boxes around objects. To this end, we run the EdgeBoxes [START_REF] Zitnick | Edge boxes: Locating object proposals from edges[END_REF] algorithm for each frame of the ENSTA and RGB-D scenes evaluation sets and keep the 100 best ranked bounding boxes along with their h in b scores. These boxes are used as a reference to evaluate our method. Then, these EdgeBoxes are re-ranked based on the saliency map to make boxes containing salient pixels better ranked than others. For that, we rank each of these boxes according to the SCscore defined in Section 4.5.3:

SCscore(B) = 1 w B × h B i,j∈B S(i, j) (5.1)
where S(i, j) is the saliency of the pixel at (i, j) , obtained from the saliency map and w B and h B are the width and height of B. To demonstrate the ability of our saliency map to produce relevant box proposals, we calculate an SCscore based on ISL saliency maps, and another one based on BMS saliency maps (denoted as EB+ISL and EB+BMS in Figure 5.12). Lastly, we generate the SegBoxes from the depth segmentation process, as described in Section 4.5.3. We also produce an SCscore for each of them. We filter out Segboxes having a low SCscore (below 0.2 in our case). Those SegBoxes, obtained from depth segmentation are complementary to the RGB-based EdgeBoxes and allow the detection of additional relevant boxes. The remaining SegBoxes are reported as SegBoxes in Figure 5.12. In practice, a small number of SegBoxes are detected on each frame (between 0 and 7 on average). Lastly, we combine the re-ranked EdgeBoxes and the SegBoxes to produce a better set of box proposals. This approach is presented in Figure 5.12 and Figure 5.13 as EB+ISL+SB.

Metrics

The chosen evaluation metric is the detection rate versus the number of proposals, based on the intersection over union (IoU) measure to count the number of detections. This measure is used by Zitnick et al. [START_REF] Zitnick | Edge boxes: Locating object proposals from edges[END_REF] to evaluate their performance over stateof-the-art approaches. This measure first needs a ranking of the proposed bounding boxes. We then use the ranking obtained by the SCscore for both the EdgeBoxes and Segboxes. When combining EdgeBoxes and SegBoxes, we arbitrarily rank the Seg-Boxes higher than the EdgeBoxes, as they are much more likely to actually contain objects. The detection rate for a set of calculated bounding boxes is obtained by comparing these proposed boxes with the ground truth (recall that ENSTA and RGB-D scenes dataset were annotated with bounding boxes around objects). The computation procedure is then as follows: for a given image, each bounding box GT i of the ground truth is compared versus each proposed bounding boxes Bbox j . For that, we produce for each pair of boxes an IoU score

IoU (GT i , Bbox j ) = A GT i ∩ A Bbox j A GT i ∪ A Bbox j (5.2)
A B being the area delimited by the bounding box. We consider GT i and Bbox j to be matching if their IoU score is above 0.5. Therefore, the number of true positives T P k in an image k is the number of ground truth bounding boxes matched with at least one of the proposed bounding boxes. The detection rate (or recall) for this N GT being the number of ground truth bounding boxes in this image. To draw the detection rate versus number of bounding boxes, we take for each N ∈ [1..100] the N best ranked boxes for each image of the dataset, and we compute the average detection rate based on these boxes. The curve then keeps increasing with the number of proposed boxes.

Results

Numerical results are reported in Figure 5.12 for both ENSTA and RGB-D scenes datasets. As expected, the use of ISL maps to improve the EdgeBoxes ranking allows a much better detection rate on both datasets. Moreover, using a bottom-up saliency map such as BMS instead of our saliency does not show significant improvements on both datasets. The SegBoxes usually propose relevant candidates, possibly not detected by the EdgeBoxes. Because they are complementary to the EdgeBoxes, combining the two approaches significantly improve the detection rate on both datasets. However, the number of proposals is low (between 0 and 7 most of the time), and they do not cover the entire image as they are produced from the depth segmentation. Figure 5.13 shows sample results of the top 5 EdgeBoxes (row EdgeBoxes), top 5 EbdgeBoxes re-ranked by the SCscore with ISL (displayed in row EB+ISL+SB, blue boxes), and Segboxes (yellow boxes). The SegBoxes almost always provide relevant boxes, but many objects are also missed this way, either because they are too far to be segmented (sample 5), or because segmentation failed (sample 6). In this case, the remaining objects locations are recovered by the EdgeBoxes. Again, the use of ISL to rank the EdgeBoxes favors boxes that surround salient elements while removing distractors such as windows (sample 4, 5). Lastly, it is possible to cope with frames that do not contain any salient object (sample 3) by filtering boxes with an SCscore below a certain threshold (0.01 in our case).

Conclusion

In this chapter, we considered different aspects of our saliency learning technique and evaluated them on four different datasets. As a summary and concluding remarks, our experimental evaluation led us to the following findings:

First, the self-supervision signal on all datasets is such that the saliency model is able to generalize well. Provided that supervision is partial, strongly unbalanced and, in the case of BioVision, weakly supervised, this generalization capability was not straightforward. Aside from this result, we evaluated the supervision signal and found it reliable enough in terms of precision and recall to ensure a descent learning quality.

Second, we have shown that the execution time depends on many factors and still suffers from limitations. First, the deep features require a GPU to work at a reasonable framerate. Second, the random forest are such that the training time is increasing linearly with the dataset size. We partially solved the problem by training the model on a separate thread, but using a truly incremental classifier may be more appropriate.

Third, we illustrated the saliency evolution process, and explained that this model was evolving with several steps. After a few iterations required to get a relatively stable saliency model, naturally salient elements were first enhanced by the common properties they shared with already discovered object. The model is then progressively refined after each object is discovered separately.

Fourth, we demonstrated that the deep features were the features having the best generalization capability. This was especially visible on the BioVision dataset were the supervision signal was extremely weak. However, the poor resolution of the produced map cannot retrieve very fine details and could probably be improved in a future work.

Fifth, we outperform state-of-the-art techniques not only on our datasets, but also on an external one with a model not trained on it. Unlike other tested saliency techniques, our approach is really good at removing naturally salient elements that are not salient in our context, and conversely, enhancing salient elements in our context that are not naturally salient.

Lastly, our saliency technique can be used to produce relevant bounding boxes around objects by re-ranking EdgeBoxes according to our saliency model. In addition, using boxes from object segmentation (the SegBoxes) on top of the EdgeBoxes further improves the results.

Part II

Intrinsically motivated exploration Chapter 6

Environment exploration on an autonomous robot

Introduction

This chapter provides an overview of the exploration problem in autonomous robotics when considered by three different fields of robotics that are mobile robotics, visionbased robotics (or active vision) and developmental robotics.

In mobile robotics, exploration is typically seen as a problem of path planning across the environment by maximizing different costs (time, energy, information, etc...). We focus more precisely on the problems of localization, mapping and planning for mobile robots.

We then concentrate our review around the exploration problem in a more general scope. We study in deeper details exploration techniques of mobile robotics and active vision systems.

In developmental robotics, a very specific type of exploration is considered, where the stress is put on the learning and development of skills and knowledge based on motivations. The problem is seen under a bio-inspired point of view, and many parallels are made with neuro-science and psychology. We then present an overview of the field of developmental robotics, and focus on the motivations as a drive for exploration. Lastly, we describe in details robotics architectures driven by a particular type of intrinsic motivation: the learning progress.

We show that for mobile, vision and developmental robotics, the philosophy and formalism may be different in the way to tackle exploration. However, they share an essential common idea: the acquisition of knowledge through actions impacting the environment.

Autonomous navigation on a mobile robot

A mobile robot is defined as a machine capable of locomotion. They have the capacity to move in their environment and are not limited to a simple physical location. Foveated systems that are not embedded in a platform able to move are then not considered as mobile robots. Mobile robots are then able to interact with their environment, and must therefore be careful of the geometry and behavior of the surrounding world.

To autonomously move in their environment, mobile robots must be equipped with an efficient navigation strategy to accomplish a specific mission. As depicted in Figure 6.1, the most general navigation scheme is to consider a sensorimotor loop in which the robot perceives data from the environment and exploits this data for self-localization. The robot is also equipped with a mission plan composed with successive goals or tasks that should be achieved. The localization is then combined with the mission plan to generate motion control and displace the robot. Most autonomous mobile robots use a mental representation of their environment as a map in which they can be localized. To accomplish a mission plan, a set of goals must be reached, and a trajectory must be found in that regard. We then describe and discuss in this section the three aspects of navigation that are mapping, localization and planning. The next two sections are widely inspired by Filliat's course on mobile robotics [START_REF] Filliat | Robotique mobile[END_REF].

Mapping and localization

The approach used to determine a trajectory strongly depends on the way the environment is represented. Depending on the sensor used and available information, the map of the environment can either be topological or metric (See Figure 6.2 for examples).

Topological maps

In topological maps, the environment is represented as a graph (also called a navigation graph) where nodes are places that the robot can reach, and edges represent the available connections between nodes. In general the information stored in edges is related to the way to reach nodes of the graph, whereas nodes store identification information (for example, a set of images [START_REF] Bazeille | Incremental topo-metric slam using vision and robot odometry[END_REF]) to localize the robot from sensory input. In topological maps, nodes may represent significant or meaningful places (such as doors, intersections or room centroids), that are either manually designed [START_REF]Rl-iac: An exploration policy for online saliency learning on an autonomous mobile robot[END_REF], [START_REF] Dedeoglu | Incremental online topological map building with a mobile robot[END_REF] or automatically added when discovered by the robot [START_REF] Kortenkamp | Topological mapping for mobile robots using a combination of sonar and vision sensing[END_REF]. Some approaches rather define nodes based on the robot exploration without any semantic meaning. Criteria for splitting nodes in this case can be the strong variation in the perceived inputs [START_REF] Gaussier | The visual homing problem: An example of robotics/biology cross fertilization[END_REF], or just euclidean distance boundaries [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF]. Edges are defined either by adjacent relations (when the robot has the ability to directly attain a node from neighbor [START_REF]Rl-iac: An exploration policy for online saliency learning on an autonomous mobile robot[END_REF], [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF], [START_REF] Oßwald | Speeding-up robot exploration by exploiting background information[END_REF], potentially storing path commands for displacement), or by metric information containing relative positions of the nodes [START_REF] Bazeille | Incremental topo-metric slam using vision and robot odometry[END_REF]. Sometimes, maps even contain absolute positions of the nodes and are referred as topometric maps [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF], [START_REF] Oßwald | Speeding-up robot exploration by exploiting background information[END_REF], [START_REF] Bazeille | Incremental topo-metric slam using vision and robot odometry[END_REF]. They are particularly well-suited for vision. The advantages of topological maps is their simplicity to store and update information. Moreover, grouping information among distinct places make the representation of those places better-suited for a representation at a sensor's level. Lastly, discretizing the environment is a good representation for planning, where paths are directly found from the graph. A major disadvantage occurs when data need to be interpreted or integrated from places that have not been visited yet. Topological graphs usually require a very throughout exploration of the environment to avoid these type of cases. Moreover, in case of noisy data or dynamic environment, the localization may become a problem as identification is likely to fail.

Our exploration technique is based on a topometric map, where edges contain distances between adjacent nodes, and nodes contain information about the local model quality. As several approach rely on an already existing topometric map to guide their exploration [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF], [START_REF] Oßwald | Speeding-up robot exploration by exploiting background information[END_REF], other focus on the construction of this map online [START_REF] Bazeille | Incremental topo-metric slam using vision and robot odometry[END_REF], [START_REF] Krajník | Longterm topological localisation for service robots in dynamic environments using spectral maps[END_REF]. We also propose in this work a simple technique to incrementally construct a topometric map adapted to our needs and considering the localization problem as solved and reliable.

Metric maps

In metric maps the environment is represented by a set of objects associated with positions in a metric space (typically a 2D map), where the position of the robot is estimated. Metric maps can be represented in terms of "objects" or occupancy grids. In the first case, "objects" are detected by sensors, localized from the robot's position and integrated to the map. Objects are either points [START_REF] Feder | Adaptive mobile robot navigation and mapping[END_REF] or oriented points [START_REF] Smith | Estimating uncertain spatial relationships in robotics[END_REF] (also called anchors), or frontiers (also called features) composed of primitive shapes such as plans, lines or ellipses [START_REF] Ayache | Maintaining representations of the environment of a mobile robot[END_REF]. These objects are often associated with a measure of uncertainty to avoid navigation mistakes [START_REF] Feder | Adaptive mobile robot navigation and mapping[END_REF]- [START_REF] Ayache | Maintaining representations of the environment of a mobile robot[END_REF]. In the second case, occupancy grids [START_REF] Moravec | High resolution maps from wide angle sonar[END_REF] are based on fine grids segmenting the environment, on which each cell contains an occupancy probability. Although more memory consuming, occupancy grids do not require any object extractor. Unlike topological maps, metric maps combine localization and sensory data at the same level. With metric maps, the whole environment is represented, and the robot can be localized more precisely. This kind of maps is also more easily usable by other types of robots, using different sensors, as the environment is not represented in terms of sensory data. It is also easier for a human user to interpret and follow the robot trajectory. However, metric map usually require a more efficient localization method. Planning strategies from these maps are also less straightforward, as the environment representation is continuous.

The navigation graph we rely on in our work is actually based on an occupancy grid produced by a laser range finder (similar to the example of Figure 6.3). Therefore, we rely on both metric and topological maps.

Simultaneous localization and mapping

Except for cases where the map is known beforehand and static during the whole experiment [START_REF] Oßwald | Speeding-up robot exploration by exploiting background information[END_REF], [START_REF] Borji | Simultaneous learning of spatial visual attention and physical actions[END_REF], localization and mapping are strongly intertwined and should be jointly processed. The map of the environment is then incrementally constructed or updated, and localization within the map is inferred from the same observations. In the literature, this joint problem is referred as Simultaneous Localization And Mapping (or SLAM) [START_REF] Leonard | Simultaneous map building and localization for an autonomous mobile robot[END_REF], [START_REF] Chatila | Position referencing and consistent world modeling for mobile robots[END_REF]. Localization may be obtained by proprioceptive sensors (wheel speed, inertial sensors) but is often subject to drift in the measurement. Therefore, exteroceptive sensors are required to perceive the environment of the robot and add a correction feedback to the displacement measures. SLAM based on laser range finders for example detect obstacles and perform a scan correlation with the previous measurements to estimate the displacement of the robot [START_REF] Kohlbrecher | A flexible and scalable slam system with full 3d motion estimation[END_REF]. They typically generate occupancy grids. Another family of SLAM techniques are visual-based, and use the similarities of successive frames to estimate the displacement of the robot [START_REF] Engel | Lsd-slam: Large-scale direct monocular slam[END_REF]. Of course, the combination of visual and geometrical cues, when possible, provides even more accurate results. The development of RGB-D SLAM techniques have then become popular in the last few years [START_REF] Endres | 3-d mapping with an rgb-d camera[END_REF]. Visual and RGB-D SLAM are usually better-suited to construct "object maps".

Our exploration system relies on the hector mapping algorithm [START_REF] Kohlbrecher | A flexible and scalable slam system with full 3d motion estimation[END_REF], based on a laser range finder and odometry combination. We consider this SLAM algorithm reliable enough for our needs.

Planning and path finding

During a mobile robot's mission, a sequence of tasks involving the displacement of the robot must be achieved. Tasks can then be divided into a set of target positions to reach that will move the robot to a certain location. In this section, we investigate the problem of planning a strategy to reach a goal, once mapping and localization are known. We also focus on planning approaches determined in two-dimensional maps. To be efficient, the chosen path should optimize a cost function (based on distance, time or energy), and be safe to the robot (collision-free).

Continuous and discrete environments

Finding an optimal trajectory with multiple obstacles in a continuous space is an NP-hard problem, but several approach tend to solve the problem locally, or by heuristic methods. Early approaches relied on potential fields [START_REF] Murphy | Introduction to ai robotics[END_REF], while more recent techniques such as improved version of RRT [START_REF] Gammell | Informed rrt*: Optimal sampling-based path planning focused via direct sampling of an admissible ellipsoidal heuristic[END_REF] rely on sampling a very large number of trajectories to approximate to the optimal solution. Another class of approaches use a "teach and repeat" strategy to make the robot learn its own path (not necessarily optimal) from experience and adapt them in new situations [START_REF] Jetchev | Trajectory prediction: Learning to map situations to robot trajectories[END_REF], [START_REF] Jiang | Learning humanoid reaching tasks in dynamic environments[END_REF].

To simplify the problem, many planning approaches are using techniques of path finding in a graph. If using a metric map, the space must then be turned into a topometric graph by discretization. A first approach to discretize maps is to cut them in segments, either by following the obstacle shapes [START_REF] Kraetzschmar | Probabilistic quadtrees for variable-resolution mapping of large environments[END_REF], or according to a regular grid [START_REF] Hengbo | Pose graph optimization with hierarchical conditionally independent graph partitioning[END_REF]. Nodes are then defined as centroids or corners of each segment, and adjacent segments are linked by edges if no obstacles are found in between. Another approach consists in pre-defining paths between points of the environment, either by visibility graphs [START_REF] Kuipers | A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations[END_REF], Voronoi diagrams to define path as far as possible from obstacles [START_REF] Oßwald | Speeding-up robot exploration by exploiting background information[END_REF], [START_REF] Neira | Path planning in graph slam using expected uncertainty[END_REF], or generated by sampling approaches [START_REF] Lavalle | Rapidly-exploring random trees: A new tool for path planning[END_REF]. Points are then used as nodes and pre-defined path are links.

Our system uses a method of path finding in a graph, and therefore turns the occupancy grid obtained from a SLAM algorithm into a discrete navigation graph from regular cells.

Path finding in a graph

When trying to find an optimal path in a graph, the problem is typically solved by finding an optimal sequence of actions to follow. To do so, methods based on direct path finding or policies can be used (see Figure 6.4).

Direct path finding may be obtained by exact resolution methods using graph search, such as the Dijkstra [START_REF] Dijkstra | A note on two problems in connexion with graphs[END_REF] algorithm. If the number of nodes is too high for an extensive search, heuristics such as A [START_REF] Hart | A formal basis for the heuristic determination of minimum cost paths[END_REF] (or D in dynamic environments) are able to find an acceptable solution in a significantly reduced time. Alternative heuristic such as the particle swarm optimization [START_REF] Qin | Path planning for mobile robot using the particle swarm optimization with mutation operator[END_REF] or the ant colony optimization [START_REF] Guan-Zheng | Ant colony system algorithm for real-time globally optimal path planning of mobile robots[END_REF] are also commonly used. In these types of approaches, the problem is supposed to be fully deterministic, and problems occur if the robot cannot reach the goal or deviates from its course. In this case, a re-planning process is required. The second category of planning is based on policies. A policy associates for each node (whether the robot is in it or not) the best action to take from this node. To reach the goal with a policy, the robot will perform a sequence of node identification and action dictated by the policy at the current node. Although slower than a simple path finding, a policy does not have to be re-computed if the estimated position of the robot is wrong, and is better-suited if several specific goals are concurrent in the environment (even potentially no goal at all). Instead of considering a discrete set of actions for each state, a possible approach to determine a policy is to use a potential field that depends on the cost to reach the goal. The action is then calculated by following a gradient descent. To find such potential, Dijkstra algorithm can be used as well [START_REF] Dijkstra | A note on two problems in connexion with graphs[END_REF], or a value iteration procedure [START_REF] Sutton | Reinforcement learning: An introduction[END_REF].

Our planning strategy is based on a path finding found from a policy, and does not depends on a single goal, but rather on attractive areas.

Path finding under uncertainty

In case of uncertainty (noisy observations, dynamic environment ...), the problem should not be modeled with a deterministic graph but rather as a Markov Decision Process. A probability is then associated with each state-action pair of the policy. If the environment is only partially known by the robot, the model is a partially observable decision process, or POMDP. The field of path finding in a POMDP context is very active, the main goal being to find the best action to take for the current state of belief [START_REF] Lauri | Stochastic control for maximizing mutual information in active sensing[END_REF], [START_REF] Ross | Online planning algorithms for pomdps[END_REF], [START_REF] Somani | Despot: Online pomdp planning with regularization[END_REF] and to update this belief on the fly. POMDP is also introduced the context of reinforcement learning techniques that are further described in Section 6.3.2.

Exploration in mobile robotics and active vision

In robotics, the exploration problem is the one of maximizing knowledge over a working environment by means of a single or several robots. In many practical problems, the robot has a specific mission (such as delivering mail, monitoring a known area, driving a user to a specific address). Goals are specified by the mission, and path plans are generated toward these goals. When the mission is related to exploration, an exploration strategy is provided by the user, and the robot should autonomously

Art gallery problem

Travelling salesman problem Chinese postman problem generate goals that are likely to increase its knowledge. Exploration is then seen as a mean to reach self-defined goals in the general purpose of maximizing knowledge. It may happen that the robot performs a simultaneous exploration (for knowledge updates or initial discovery) and task. Several approaches raise the problem of trade-off between exploration and exploitation [START_REF] Audibert | Exploration-exploitation tradeoff using variance estimates in multi-armed bandits[END_REF], [START_REF] Santos | Spatio-temporal exploration strategies for long-term autonomy of mobile robots[END_REF], but these aspects are beyond the scope of this review.

We here present three possible types of exploration that could be used in most robotics systems. We however concentrate our explanations and examples on mobile robotics and active vision systems. A first purpose for exploration is the one of finding concrete elements of the environment, and memorizing knowledge about them. We call it memory-oriented exploration. Another purpose for exploring is the one probing the environment for learning. By learning, the robot acquires skills or predictive knowledge that are not directly representative of the environment, but useful for future tasks. We divide learning-based exploration between reinforcement learning, and model-based exploration.

Memory-oriented exploration

In a memory-oriented exploration, the goal is to have a good coverage of the environment towards elements of interest. The robot then keeps an instance of these elements in memory, varying from state variables, maps, lists of objects and events and their attributes.

In mobile robotics

In mobile robotics, building a map of the environment may be the most straightforward exploration application. In this case, exploration is related to the coverage whole environment rather than isolated targets of interest [START_REF] Oßwald | Speeding-up robot exploration by exploiting background information[END_REF], [START_REF] Bazeille | Incremental topo-metric slam using vision and robot odometry[END_REF], [START_REF] Jebari | Combined vision and frontier-based exploration strategies for semantic mapping[END_REF]. Autonomous SLAM approaches [START_REF] Tovar | Planning exploration strategies for simultaneous localization and mapping[END_REF], for example, need exploration techniques to efficiently cover the environment and spend more time on more challenging areas.

Localizing and identifying objects in the environment is another typical purpose for exploration [START_REF] Cañas | An overt visual attention mechanism based on saliency dynamics[END_REF], [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF], [START_REF] Borji | Online learning of task-driven object-based visual attention control[END_REF]. When no prior about the objects are given, the problem is called object discovery, and exploration should be such that areas likely to contain objects are examined. Lastly, in dynamic environments [START_REF] Santos | Spatio-temporal exploration strategies for long-term autonomy of mobile robots[END_REF], [START_REF] Levihn | Planning with movable obstacles in continuous environments with uncertain dynamics[END_REF], exploration should be such that places are constantly re-visited and memory updated accordingly.

When the robot is provided with a static map at the beginning of the exploration, the user can design pre-defined path plans prior to the experiment [START_REF] Meger | Curious george: An attentive semantic robot[END_REF]. Otherwise, navigation path plans can be found by solving specific mathematical problems from graphs or continuous environments. Figure 6.5 presents a few examples of common problems to be solved for environment exploration. The most famous one is probably the traveling salesman problem [START_REF] Dorigo | Ant colony system: A cooperative learning approach to the traveling salesman problem[END_REF], when an optimal path is found to join a set of areas of the map while minimizing the traveled distance. If the problem is to find an appropriate set of views to cover the whole environment (called view planning problems), the art gallery problem [START_REF] González-Baños | A randomized art-gallery algorithm for sensor placement[END_REF] is a good approach to consider. Otherwise, when the goal is to navigate to visit all edges of a graph, the chinese postman problem is the approach to considered [START_REF] Eiselt | Arc routing problems, part i: The chinese postman problem[END_REF]. As these problems are often NP-hard, solving them directly on a robot is not feasible. They are then either solved offline [START_REF] Oßwald | Speeding-up robot exploration by exploiting background information[END_REF], require heuristics [START_REF] Jebari | Combined vision and frontier-based exploration strategies for semantic mapping[END_REF], or are replaced by more naive greedy algorithms [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF]. Lastly, modern approaches usually combine these optimal path with sensory-based refinement to ensure that exploration is consistent with the true environment [START_REF] Oßwald | Speeding-up robot exploration by exploiting background information[END_REF], [START_REF] Jebari | Combined vision and frontier-based exploration strategies for semantic mapping[END_REF].

When the map of the environment is not known beforehand, the robot can only rely on sensory data to drive exploration. A very common technique is the frontierbased exploration [START_REF] Jebari | Combined vision and frontier-based exploration strategies for semantic mapping[END_REF], [START_REF] Yamauchi | A frontier-based approach for autonomous exploration[END_REF], where sensory inputs are used to detect the boundaries (or frontiers) between known and unknown areas. Exploration is then done to visit these frontiers. Entropy reduction or information gain maximization [START_REF] Stachniss | Information gain-based exploration using raoblackwellized particle filters[END_REF] can also drive the robot to unexplored or highly informative regions from sensory inputs. The time spent in displacement can also be taken into account to speed up exploration [START_REF] Stachniss | Exploring unknown environments with mobile robots using coverage maps[END_REF], and planning to re-visit locations is sometimes considered to increase the knowledge quality [START_REF] Fentanes | A new method for efficient three-dimensional reconstruction of outdoor environments using mobile robots[END_REF]. In dynamic environments, temporal statistical approaches are used to determine how frequently exploration should be performed in a particular area [START_REF] Santos | Spatio-temporal exploration strategies for long-term autonomy of mobile robots[END_REF].

In some cases, the problem of gathering information is considered jointly with the one of minimizing displacements. This is for example the case in Wang et al.'s work [START_REF] Wang | View planning problem with combined view and traveling cost[END_REF], where a combination of the traveling salesman and art gallery problems are sequentially solved to optimize both aspects of exploration. This idea of joint optimization is also raised by Tovar et al. [START_REF] Tovar | Planning exploration strategies for simultaneous localization and mapping[END_REF] in the context of simultaneous localization and mapping. In their work, a utility function was proposed to measure the interestingness of potential location and balancing the potential information gain with the displacement cost. Jointly combining information and displacement cost is at the heart of the exploration technique proposed in the next chapter.

In active vision systems

Exploration can be guided by visual inputs as well, and is tightly linked with the mechanism of visual attention in Chapter 3 by targeting salient regions of the field of view. If the environment is static, the inhibition of return plays a major role in exploration, as it avoids the system to get stuck in the most salient location. When the robot is equipped with PTZ cameras (pan-tilt-zoom) [START_REF] Cañas | An overt visual attention mechanism based on saliency dynamics[END_REF], [START_REF] Minut | A reinforcement learning model of selective visual attention[END_REF], [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF] or with a combination of contextual (wide field of view) and foveal (sharp field of view) cameras [START_REF] Vijayakumar | Overt visual attention for a humanoid robot[END_REF], [START_REF] Björkman | Active 3d scene segmentation and detection of unknown objects[END_REF], the actions performed for exploration are typically saccades, so that the zoomed camera is oriented to informative regions. If the robot can move across its environment, actions are displacements in order to get a closer or better point of view of areas of interest [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF], [START_REF] Lauri | Stochastic control for maximizing mutual information in active sensing[END_REF], [START_REF] Borji | Online learning of task-driven object-based visual attention control[END_REF], [START_REF] Massios | A best next view selection algorithm incorporating a quality criterion[END_REF].

Reinforcement learning-driven exploration

When the goal of the robot is to learn behaviors (for example, following an optimal trajectory towards a given area of the environment, performing saccades to identify an object, ...), a possible approach is to learn it by trying successive alternatives until the desired behavior is reached. Reinforcement learning exploits this principle of learning by successive trials, either leading to success or failure. This is made possible by receiving a feedback from the environment based on the quality of the attempt. This feedback, called a reward, will guide the robot to find which actions are good based on past experiments.

Applications

Reinforcement learning is a vast field of research and is used for many different purposes in robotics (see for example Kober and Kormushev reviews [START_REF] Kober | Reinforcement learning in robotics: A survey[END_REF], [START_REF] Kormushev | Reinforcement learning in robotics: Applications and real-world challenges[END_REF]). In the context of visual attention, such techniques have been exploited to automatically guide the gaze toward salient areas [START_REF] Minut | A reinforcement learning model of selective visual attention[END_REF], or to learn the optimal saccades for object recognition [START_REF] Paletta | Q-learning of sequential attention for visual object recognition from informative local descriptors[END_REF].

In mobile robotics, reinforcement learning is often used to find an optimal displacement policy in a given environment [START_REF] Jodogne | Closed-loop learning of visual control policies[END_REF], [START_REF] Chentanez | Intrinsically motivated reinforcement learning[END_REF] and for a specific task. In this context, reinforcement learning is equivalent to a planning tasks minimizing an objective function, except that this objective function is expressed in terms of rewards. In that regard, reinforcement learning is a possible technique to find optimal path plans by means of policies, as described in Section 6.2.2. In our approach, we use reinforcement learning in this context.

Lastly, some hybrid approaches combine saccades and robot locomotion in a reinforcement learning framework to jointly identify salient objects, and use them as visual anchors to navigate in the environment for reaching a target location. Borji et al.have proposed a set of publications in that regard [START_REF] Borji | Learning sequential visual attention control through dynamic state space discretization[END_REF], [START_REF] Borji | Simultaneous learning of spatial visual attention and physical actions[END_REF], inspired by the work of Mc Callum [START_REF] Mccallum | Learning visual routines with reinforcement learning[END_REF].

A large panel of reinforcement learning techniques have been proposed in the literature to solve problems in very different configurations (continuous or discrete states and actions, model-based and model-free techniques). A description of basic reinforcement learning problems has been proposed by Sutton and Barto [START_REF] Barto | Reinforcement learning: An introduction[END_REF].

Q-learning

To keep explanations easier, we describe in this section the Q-learning approach in a discrete state-action space only, that is further used in Chapter 7.

Reinforcement learning problems are modeled by an autonomous agent confronted to a given environment and having to select actions depending on its current state. To illustrate our explanations, let us consider the simple case of a robot able to move on a grid and whose goal is to reach a target G while avoiding a wall (See Figure 6.6). After making an action, the agent arrives in a new state and receives a reward (positive or negative) related to this state. This reward is the only signal that will guide the agent's learning, to progressively find the best action to

G -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 100 -1 -1 -1 -1 -1 -1 -1
States and actions Reward table Optimal policy FIGURE 6.6: Environment, rewards and policy take in each state of the environment. This correspondence between states and actions, that describes the optimal behavior to adopt is called the optimal policy (often called Π ).

Problem definition: Formally, the problem can be defined as a set of states S (the reachable squares of the grid), a set of actions A ("up", "down", "left" "right"), and a set of rewards R = r(s, a). Typically, a high reward is associated with the goal state, and low reward are distributed everywhere else. Rewards are collected by the agent after taking an action while being in a given state. The agent starts in an initial state s 0 , chooses an action a 1 and arrives in a new state s 1 . If considering that reaching state s t+1 from s t only depends on s t and on action a t+1 , the problem can be modeled as a Markov Decision Process. An MDP is defined by a system (S, A, P, R), where P = P (s |s, a) are the transition probabilities between states given an action. Lastly, the policy Π : S → A is a function that associates an action to follow for each state of the environment. The goal is to iteratively modify the policy, and test its efficiency to progressively converge to an optimal one (Π ).

Action-value function:

To converge to an optimal policy, a large number of experiments (or episodes) are conducted, in which the robot takes actions suggested by the current policy, receives rewards, and evaluates the benefits of these actions for the policy. This benefit is evaluated by considering an action-value function Q π (s, a) (which is a matrix in our case) that is progressively updated with experiments. This action value function represents the average cumulated reward during experiments.

Q π (s, a) = E[R|s, a, Π] (6.1)
where R is the cumulated reward received after taking action a in state s, and following policy Π thereafter. Mathematically, R is defined as

R = N t=1 γ t r t+1 (6.2)
where N is the number of successive actions in the sequence, r t+1 is the reward received at time step t + 1, and γ is the discount factor between 0 and 1. γ is used to make the agent reason about the long term consequences of its actions. It is a parameter that controls how much the value function should take future rewards into account. If γ = 0, then only immediate reward matters. If γ = 1, reward received in future steps all matter the same. The discount factor is an important parameter of our system, and its influence on the performance of our algorithm is discussed in Chapter 8. The action-value function associated with the optimal policy Π must then be such that Q π , or Q is maximum for each state-action pair.

Exploration strategy: The Q-learning algorithm provides a way to guide the robot's exploration and to update the action-value function so that it naturally converges to Q . The robot should, most of the time, follow the policy that is currently suggested by the current policy, and occasionally select a random action (this is called the -greedy strategy). Formally, the exploration strategy will select the next action a t among A from state s t to take by following the rule

a t      Argmax a ∈A (Q(s t , a )) if v > Rand a ∈A (a ) otherwise (6.3)
Update rule: Lastly, after each action a t is performed and a reward r t+1 is received, the Q-matrix is updated according to the following rule:

Q(s t , a t ) ⇐ (1 -α)Q(s t , a t ) + α[r t+1 + γ(max a Q(s t+1 , a ))] (6.4)
where α is a learning rate (between 0 and 1) that controls how much the current action should be considered to update the model. Finally, to exploit the optimal policy to reach the goal, the agent in a state s should take the action a for which Q * (s, .) is maximum.

Improving reinforcement-learning exploration

We now examine the possible exploration strategies to make reinforcement learning techniques converge rapidly. The worse case is the one of a pure random exploration. In this case, the value function converges to the optimal one, but very slowly. The -greedy technique takes advantage of the current value function to favor exploration in areas that have been the most rewarding. is then an arbitrary threshold to balance exploration (by random selection) and exploitation (by value function selection).

Several improvements have been proposed to further speed-up exploration while ensuring the convergence of the value function. A very basic, yet efficient approach is the "optimism towards uncertainty", also known as R-max algorithm [START_REF] Brafman | R-max-a general polynomial time algorithm for nearoptimal reinforcement learning[END_REF]. This technique simply favors areas that have not been visited enough, thus ensuring a rapid coverage of the whole exploration space. Other techniques such as the bayesian exploration bonuses [START_REF] Kolter | Near-bayesian exploration in polynomial time[END_REF] follow a similar approach by rewarding stateaction pairs inversely with their visitation counts.

The concept of intrinsic motivation has been proposed to speed-up reinforcement learning problems. Chentanez et al. [START_REF] Chentanez | Intrinsically motivated reinforcement learning[END_REF] have proposed a technique where the learning agent is attracted by "salient events", while Lopes et al. [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF] have use a strategy following learning progress. In these approach, an intrinsic reward is added to the exploration scheme and biases the value function to make the agent focus more on areas that really matter. This intrinsic reward decreases during exploration, thus avoiding any bias in the final policy. The field of intrinsic motivation is actively studied in a developmental robotics context and is further discussed in Section 6.4.2.

In our work, we only use reinforcement learning as a mean to solve a planning task. Therefore speeding up RL convergence is not our first priority. Nevertheless, we also use a type of -greedy displacement rule, and our problem initialization strongly resembles the R-max approach.

Model-oriented exploration

In model-oriented exploration, the goal is to learn something from the environment by building predictive models. The approach in this case is to collect sensory information, turn it into exploitable data, and apply machine learning techniques to make inferences about the world. The exploration strategy is then oriented towards finding samples, able to build a good representation. The question of speeding up exploration to learn faster is another aspect considered when designing exploration strategies in that regard. This kind of exploration may be considered as a type of active learning problem, also known as the optimal experiment design in statistics [START_REF] Emery | Optimal experiment design[END_REF].

Applications

Model-oriented exploration is used in number of applications. In mobile robotics, one can for example think of learning the visual aspects of the environment to predict loop closure [START_REF] Ho | Loop closure detection in slam by combining visual and spatial appearance[END_REF], or to predict what unexplored areas may look like based on a limited number of evidences [START_REF] Ström | Predictive exploration considering previously mapped environments[END_REF]. Learning object representations in an openended manner [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF], [START_REF] Chen | An experimental comparison between nmf and lda for active cross-situational object-word learning[END_REF], or trying to predict the localization of objects based on contextual input data are other possibilities [START_REF] Pandey | Scene recognition and weakly supervised object localization with deformable part-based models[END_REF]. Those models could be learned without any particular exploration strategies (by randomly sampling the environment for example), but very slowly and potentially poorly in complex environments or high dimensional spaces. Exploration strategies in these types of setups are of two kinds.

Exploration following extrinsic criteria

The first category centers exploration towards elements of interest, relevant for what needs to be learned. As an example, for the task of learning object representations, saliency maps are a good mean to isolate objects and guide exploration [START_REF] Rutishauser | Is bottom-up attention useful for object recognition?[END_REF]. Very often approaches used for memory-based exploration also reveal efficient for model learning [START_REF] Ström | Predictive exploration considering previously mapped environments[END_REF]. This is mainly because elements of interest are often the same in both cases, the only difference being the way they are analyzed by the system and integrated to the internal representation.

Exploration following intrinsic criteria

The second category of exploration chooses actions based on the internal state of the model. For example, if two distinct objects have a very close representation in the internal model of the robot, a good approach would be to sample additional data from these objects and increase the discrimination capacity of the model [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF]. An early implementation of this idea was developed by Schmidhuber [START_REF] Schmidhuber | Curious model-building control systems[END_REF] in the scope of artificial curiosity, who proposed to guide the robot's exploration by looking at the prediction error of the neural network trying to learn a model. This kind of behavior cannot be obtained from the first type of exploration, unless of a handdesigned strategy dedicated to this goal. As this type of exploration is based on the internal representation of the knowledge, it is much closer to the way biological exploration is conducted. This exploration technique is actually at the heart of developmental robotics, relying on intrinsic motivational systems. These aspects are more deeply described and discussed in the next section.

Exploration in developmental robotics

This section presents autonomous exploration from a developmental robotics point of view. We first provide an overview of the field, main goals, challenges and areas of research. We then focus on the problem of motivations to guide the exploration, and more specifically the use of learning progress as an intrinsic reward.

An overview of developmental robotics

Background

Developmental robotics, also called epigenetic robotics, aims to study the mechanisms that would be able to provide a robot with a lifelong and autonomous learning skills or knowledge. Pioneering thoughts on developmental robotics were formulated by Alan Turing himself [START_REF] Turing | Computing machinery and intelligence[END_REF]: "instead of trying to produce a program to simulate the adult mind, why not rather try to produce one which simulates a child's? If this were then subjected to an appropriate course of education one would obtain the adult brain" Developmental robotics then suggests to find inspiration from human and animals, and from the way they gain knowledge of increasing complexity. Many works, either in biology or psychology, study the development and learning of humans [START_REF] Chaiklin | The zone of proximal development in vygotsky's analysis of learning and instruction[END_REF]. Developmental robotics builds on those approaches, ranging from infant behavior to cortical structures involved in learning.

This is not a one way inspiration, as developmental robotics also aims to validate theories about development and learning. Applying a developmental psychology theory on a robot usually requires a deep reflection and understanding to make operational a weakly formalized theory. Implementation on a robot then allows a deeper study on original aspects. Although Turing's ideas date back to the 50's, no concrete investigation was done in that regard for almost 40 years. Schmidhuber proposed in the 90's preliminary ideas on the way a robot should explore based on artificial curiosity [START_REF] Schmidhuber | Curious model-building control systems[END_REF]. However, the field of developmental robotics has really emerged in the late 90's with Weng's work on autonomous mental development [START_REF] Weng | Autonomous mental development by robots and animals[END_REF].

Today, developmental robotics strongly builds upon the fields of artificial intelligence or machine learning, and widely overlaps cognitive robotics and computational neuroscience. It has emerged at the crossroad of those fields following the idea that development and learning is the result of interactions between the brain, the body, and the environment.

Goals and research in developmental robotics

Goals and challenges: Developmental robots should be designed so that a certain number of behaviors can emerge. A few robotics platforms have been proposed as developmental robotics frameworks in that regard, such as the iCub robot [START_REF] Tikhanoff | An open-source simulator for cognitive robotics research: The prototype of the icub humanoid robot simulator[END_REF], Erato's baby robot [START_REF] Minato | A baby robot platform for cognitive developmental robotics[END_REF], or the opensource Poppy robot [START_REF] Lapeyre | Poppy: Plate-forme robotique open source, imprimée en 3d et totalement modulaire pour l'experimentation scientifique, artistique et pédagogique[END_REF] (See Figure 6.7). More specifically, engineers are interested in the following components: • Building a task-independent learning mechanism that could be applied to any type of tasks, even the one that are unknown to the robot's designer.

• Build a system able to tackle lifelong learning in an open-ended manner. This implies that robots should be able to acquire new skills continuously during their whole operation time, in an unconstrained world.

• Acquire skills of increasing complexity, and use the previously learned knowledge to acquire them.

One of the main difficulty is to equip robots with a developmental plan that is not manually designed, and to let the robots decide by themselves the type of activity they should follow. This is a very challenging issue, as programmers should accept to be removed from the learning process and rather focus on how robot can select relevant tasks and learn from them. On the other hand, letting robots decide on their own also raises ethics and safety issues that should be considered by both programmers and users.

Lastly, a consequence of making a robot learn in a developmental way is the emergence of subjectivity [START_REF] Brooks | Achieving artificial intelligence through building robots[END_REF]. What robots actually learns directly depends on what they have been experiencing. As a results, two robots with the same architecture may not have the same world representation in the end. This may be a problem in terms of performance evaluation, as there is no way to warranty that the system learns a model that is well-suited for the tasks it was trained for.

Research directions: As one of the main challenges of developmental learning is to learn a growing number of skills in unconstrained environments and within a limited lifetime, many research investigate how to constrain exploration to make learning possible and efficient. A first approach is the use of motivations able to drive the robot's exploration towards relevant areas of the space [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF]. Motivations are defined by means of rewards, intrinsic or extrinsic and are further described in Section 6.4.2. An emblematic research in this field is the Intelligent Adaptive Curiosity (or IAC) proposed by Oudeyer et al. [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] and has been evaluated on a AIBO robot discovering how to interact with its environment by playing on a baby playground (the setup was called the playground experiment, see Figure 6.8). IAC was re-used and improved by many research work [START_REF]Rl-iac: An exploration policy for online saliency learning on an autonomous mobile robot[END_REF], [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF], [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF], [START_REF] Baranès | R-iac: Robust intrinsically motivated exploration and active learning[END_REF], [START_REF] Forestier | Overlapping waves in tool use development: A curiositydriven computational model[END_REF] and is the fundamental mechanism of the work described in Chapter 7. A second way of guiding the robot's exploration and learning is to study social guidance. In this context, a human user [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF], or a peer [START_REF] Moulin-Frier | Exploration strategies in developmental robotics: A unified probabilistic framework[END_REF] has the role of a teacher that guides the robot for learning desired skills.

A third area of research is related to the way skills and knowledge should be represented and should evolve in time [START_REF] Kompella | Autonomous learning of abstractions using curiosity-driven modular incremental slow feature analysis[END_REF]. The question of how to create highlevel abstractions of skills to be used in more complex applications is another critical point [START_REF] Matricon | An iterative algorithm for forward-parameterized skill discovery[END_REF].

Lastly, maturational constraints is also a way to efficiently guide exploration accordingly with the robot's skills. Instead of providing the robot with a fully operational body and brain, it is possible to progressively enable higher degrees of freedom, or better representation capacities to help the robot focus on simple tasks first [START_REF] Baranes | The interaction of maturational constraints and intrinsic motivations in active motor development[END_REF].

Motivation and rewards to guide exploration

A challenge of developmental robotics is to make robots learn skills by exploring a complex environment, with many degrees of freedom for exploration. To be able to learn in a reasonable time, exploration should be guided by mechanisms able to engage and disengage attention towards areas of the space where learning is possible, and keep trying until a given skill is acquired. This process of engaging the implication for a given task is called motivation.

Intrinsic and extrinsic motivation

Motivation is guiding our everyday life's action, whether we need to learn something, or just accomplish a task. Ryan and Decy [START_REF] Ryan | Intrinsic and extrinsic motivations: Classic definitions and new directions[END_REF] have distinguished two types of motivations, namely intrinsic motivation and extrinsic motivation. Considering Ryan and Decy's work:

"Intrinsic motivation is defined as the doing of an activity for its inherent satisfaction rather than for some separable consequence. When intrinsically motivated, a person is moved to act for the fun or challenge entailed rather than because of external products." Children are perfect example of intrinsically motivated living beings. They have the faculty to engage in an activity for no particular purpose. Indeed, young children are constantly trying to grasp, shoot, bite or squash any new object of their environment. They may be deeply involved in interacting with an object, and, for no external reason, suddenly drop it and find another exciting focus.

Intrinsic motivation is opposed to extrinsic motivation, defined as "Extrinsic motivation is a construct that pertains whenever an activity is done in order to attain some separable outcome. Extrinsic motivation thus contrasts with intrinsic motivation, which refers to doing an activity simply for the enjoyment of the activity itself, rather than its instrumental value" Extrinsic motivation is then associated with any activity where an external goal must be achieved. In robotics, systems relying on reinforcement learning with a simple reward function are typically extrinsically motivated.

Motivation, whether intrinsic or extrinsic, is strongly related to the notion of reward. Schlutz et al. [START_REF] Schultz | A neural substrate of prediction and reward[END_REF] have revealed that a strong correlation exists between the reward function used in reinforcement learning and the dopamine emission in the midbrain (See Figure 6.9). Although this was demonstrated for extrinsic kind of reward, additional studies suggested that dopamine could also be involved in intrinsic motivation [START_REF] Dayan | Reward, motivation, and reinforcement learning[END_REF].

Lastly, there is a major difference in the role of intrinsic and extrinsic reward in the learning process. In reinforcement learning (extrinsic reward), the reward is the only learning signal provided by the environment, and this learning signal is entirely integrated into the acquired knowledge (for example, the value function providing a policy to follow). Intrinsic rewards are rather used as a guide for exploration and are not part of the model in the end.

Exploration strategies based on intrinsic reward

The degree of motivation towards an activity in the process of learning can be highlighted by the theory of flow. Figure 6.10 represents the mental state associated with a skill level and a challenge level. To reach the full state of concentration required for efficient learning conditions, the activity should be both challenging and just

Many types of intrinsic motivation:

The list of intrinsic motivation strategies used in the literature is huge. Among them, the traditional novelty [START_REF] Brafman | R-max-a general polynomial time algorithm for nearoptimal reinforcement learning[END_REF], [START_REF] Huang | Novelty and reinforcement learning in the value system of developmental robots[END_REF], surprise [START_REF] Chentanez | Intrinsically motivated reinforcement learning[END_REF], [START_REF] Boehnke | Visual adaptation and novelty responses in the superior colliculus[END_REF], or uncertainty [START_REF] Brafman | R-max-a general polynomial time algorithm for nearoptimal reinforcement learning[END_REF], [START_REF] Rothkopf | Credit assignment in multiple goal embodied visuomotor behavior[END_REF] are widely used. However, more exotic rewards may be attributed to information gain [START_REF] Ryan | Intrinsic and extrinsic motivations: Classic definitions and new directions[END_REF], confidence [START_REF] Ngo | Confidence-based progress-driven selfgenerated goals for skill acquisition in developmental robots[END_REF], progress [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF], curiosity [START_REF] Schmidhuber | Curious model-building control systems[END_REF], interest [START_REF] Saunders | Curious agents and situated design evaluations[END_REF], achievement [START_REF] Merrick | A computational model of achievement motivation for artificial agents[END_REF], saliency [START_REF] Chentanez | Intrinsically motivated reinforcement learning[END_REF] (not to be confused with visual saliency of Part I), diversity [START_REF] Benureau | Diversity-driven selection of exploration strategies in multiarmed bandits[END_REF], or familiarity [START_REF] Kaplan | Motivational principles for visual know-how development[END_REF]. Many of them have been reviewed and compared in recent surveys [START_REF] Oudeyer | How can we define intrinsic motivation?[END_REF], [START_REF] Merrick | Value systems for developmental cognitive robotics: A survey[END_REF]. However, these denominations should be used very carefully as they are sometimes used by different teams under very different mathematical definitions. To our knowledge, there is no generally accepted definition (mathematically speaking) for each of these terms.

Many implementations of those strategies are in fact heuristics aiming at biasing the exploration towards areas that are potentially interesting. For example, novelty would reward states that have not been recently visited [START_REF] Brafman | R-max-a general polynomial time algorithm for nearoptimal reinforcement learning[END_REF], surprise (or contextual novelty) targets attention towards salient events [START_REF] Boehnke | Visual adaptation and novelty responses in the superior colliculus[END_REF], and uncertainty is related to the high variance or entropy in states or actions [START_REF] Rothkopf | Credit assignment in multiple goal embodied visuomotor behavior[END_REF]. This kind of approaches are efficient in the sense that they may speed up exploration by focusing on interesting areas only, but there is no warranty that learning may be possible. Regarding the theory of flow, they are focused on finding challenging areas, and do not really consider the skill level of the agent. They are most of the time similar to exploration techniques described in Section 6.3.1, except that they are examined under a neuroscience point of view (the term "reward" designating in fact the criterion to maximize in Section 6.3.1).

Prediction error and learning progress: To be consistent with the theory of flow, an evaluation of the skill level is required. For that, a possible measure is the socalled prediction error [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] (sometimes also called novelty [START_REF] Huang | Novelty and reinforcement learning in the value system of developmental robots[END_REF] or even surprise [START_REF] Chentanez | Intrinsically motivated reinforcement learning[END_REF]), evaluating the difference between what the robot is predicting, and what is actually measured by sensors. Though, a strong limitation of the prediction error is that the robot may be stuck in unlearnable situations, especially in the presence of noise. Focusing only on areas with a high level of prediction error may then have disastrous consequences on learning.

Another approach, early suggested by Schmidhuber [START_REF] Schmidhuber | Curious model-building control systems[END_REF] and implemented by Kaplan and Oudeyer [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF], [START_REF] Kaplan | Motivational principles for visual know-how development[END_REF] in a developmental learning context suggests to rely on the evolution of the prediction error rather than current state of this error. This evolution can be seen as a sort of derivative and is called the learning progress.

Progress is particularly interesting as it avoids both trivial (low error) and unlearnable (high error) areas of the space, while focusing on tasks that are neither too easy, nor too hard. Progress also has interesting properties on the order tasks should be accomplished, starting with easy ones (where error decreases quickly) and progressively increasing their complexities (having a slower error decrease), thus being consistent with a lifelong learning scenario. Figure 6.11 illustrates this behavior by considering the time spent in 4 activities based on their prediction errors. Activities 1 and 4 are trivial and unlearnable, so that the robot only spends time on them at the beginning of the experiment. Then, activity 3 is prioritized as learning progress is the highest. When the competence associated with activity 3 is acquired, learning progress becomes low, and focus is concentrated on activity 2, whose progresses where lower.

Learning progress is then a particularly interesting intrinsic motivation. In IAC, a measure of learning progress is proposed to drive the robot's exploration. We then rely on this measure in our approach.

Visual attention and intrinsic motivation

Visual attention is an excellent study case for intrinsic motivation. In a general case, eye saccades and fixations can be seen as a way to actively sample information. This sampling is generally oriented to reduce uncertainty in future states of observation, which can be seen as a form of intrinsically motivated exploration strategy [START_REF] Gottlieb | Information-seeking, curiosity, and attention: Computational and neural mechanisms[END_REF], [START_REF] Baranes | Eye movements reveal epistemic curiosity in human observers[END_REF].

In neuroscience, several studies have highlighted the fact that eye movement patterns are not the same when trying to learn a skill, and once this skill is acquired [START_REF] Sailer | Eye-hand coordination during learning of a novel visuomotor task[END_REF]. This suggests that saccades are used as a mean of exploration. Moreover, studies on monkeys have shown that the brain is weighting visual information based on the collected reward they convey [START_REF] Gottlieb | Information-seeking, curiosity, and attention: Computational and neural mechanisms[END_REF]. This type of weighting approach strongly resembles saliency map computation approaches.

In developmental robotics, visual attention is also considered as a common study case, most of the time used as a way to develop proprioceptive skills (predict the position of the hand in a field of view [START_REF] Baranès | R-iac: Robust intrinsically motivated exploration and active learning[END_REF], head orientation toward a light [START_REF] Kaplan | Motivational principles for visual know-how development[END_REF]) or visual servoing skills (learning options from visual inputs [START_REF] Kompella | Explore to see, learn to perceive, get the actions for free: Skillability[END_REF],

smooth pursuit [START_REF] Zhang | Intrinsically motivated learning of visual motion perception and smooth pursuit[END_REF], simultaneous gaze control and reaching [START_REF] Hulse | Integration of active vision and reaching from a developmental robotics perspective[END_REF]). However, these methods mainly focus on developing motor skills rather than visual skills. For example, the discovery and learning of what should be salient in a environment has not been examined so far in a developmental robotics framework, although clear evidences show that such saliency differs from age groups [START_REF] Trukenbrod | Icat: A computational model for the adaptive control of fixation durations[END_REF], [START_REF] Meur | Computational model for predicting visual fixations from childhood to adulthood[END_REF] and culture [START_REF] Boduroglu | Cultural differences in allocation of attention in visual information processing[END_REF]. It is therefore, at least partially, learned.

Implementations of progress-based exploration

In this section, we highlight the main components of existing implementations of learning progress-based exploration strategies. For an alternative classification and formalism of the problem, one could refer to the strategic student problem formulation [START_REF] Lopes | The strategic student approach for life-long exploration and learning[END_REF].

Architectures

Similar to any robotics system trying to learn from the environment, systems relying on progress for exploration need at least inputs sensory data to produce, when learning is finished, an outcome that is to be used in future tasks. Another common element of those systems is that they learn to make predictions about the external world. This prediction process is performed by a learner, an internal learning machine (classifier, regression system, POMDP, etc...) predicting output states Y given input signals X.

A general internal architecture of intrinsically motivated systems based on learning progress has been described by Oudeyer et al. [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] and is represented in Figure 6.12. The learner, is the central element of the architecture, learning to make predictions. The learner is iteratively updated by the input signals, and produces from each input signal an output state prediction ( Ỹ in the figure). The prediction error E is obtained by measuring the distance with the learning signal (Y in the figure). Second, a meta-learner learns to predict the error in prediction of the learner. This time, the meta-learner receives successive prediction errors E as a learning signal and tries to predict how this error will evolve in a near future. Lastly, a knowledge gain assessor receives the estimated future prediction error E and considers past E and future E to compute the progress measure.

Lastly, the progress measure is transferred to a controller to derive a policy and guide the future robot's actions.

Sensory and proprioceptive signals:

To make its own world representation, the robot perceives and analyzes both sensory and proprioceptive signals. All or part of them will constitute the input signals (X) of the learner. In vision-based systems, these predictions are made from raw [START_REF]Rl-iac: An exploration policy for online saliency learning on an autonomous mobile robot[END_REF], [START_REF] Lyubova | Developmental approach for interactive object discovery[END_REF], [START_REF] Kompella | Explore to see, learn to perceive, get the actions for free: Skillability[END_REF], [START_REF] Luciw | Artificial curiosity with planning for autonomous perceptual and cognitive development[END_REF] or transformed [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF], [START_REF] Kaplan | Motivational principles for visual know-how development[END_REF] visual signals providing a higher level information (for example, the position of a spotlight in the field of view). When learning sensorimotor skills [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF], [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF], proprioceptive signals (such as motor commands) are more likely to be used as input signals. Lastly, those signals are sometimes discretized before being processed by the learner, to simplify their representation [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF].

Learning signal: Then, for the learner to learn something, a learning signal is required. This learning signal should be seen as a ground truth to be compared with the output state Y the learner is predicting, and should be removed from the prediction process once learning is finished. These learning signals may be obtained by other sensory data [START_REF]Rl-iac: An exploration policy for online saliency learning on an autonomous mobile robot[END_REF], temporal consistency of successive observations [START_REF] Lyubova | Developmental approach for interactive object discovery[END_REF] FIGURE 6.12: Global architecture of progress-based motivation systems, from [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] (two successive views contain the same object), sensory measurements after a particular event occurred (for example, after performing an action [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF], [START_REF] Kompella | Explore to see, learn to perceive, get the actions for free: Skillability[END_REF], [START_REF] Oudeyer | The playground experiment: Taskindependent development of a curious robot[END_REF] or after discovering a salient event [START_REF] Chentanez | Intrinsically motivated reinforcement learning[END_REF]), or based on an external assistance from an oracle [START_REF] Lyubova | Developmental approach for interactive object discovery[END_REF]. Note that this learning signal may be different than what the system should learn in the end. For example, if an intrinsically motivated system is using extrinsic reward to learn a policy, the learning signal associated with the learner would just be a hint to guide exploration, not what the system should learn in the end [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF], [START_REF] Kompella | Autonomous learning of abstractions using curiosity-driven modular incremental slow feature analysis[END_REF].

System outcome: The expected outcome of the system should be the final product of what it was trained for. A sensorimotor mapping in the case of forward and backward models [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF], [START_REF] Oudeyer | The playground experiment: Taskindependent development of a curious robot[END_REF], classifiers in the case of recognition systems [START_REF]Rl-iac: An exploration policy for online saliency learning on an autonomous mobile robot[END_REF], [START_REF] Lyubova | Developmental approach for interactive object discovery[END_REF], abstraction libraries of sensorimotor primitives [START_REF] Kompella | Explore to see, learn to perceive, get the actions for free: Skillability[END_REF], or optimal policies for problems of goal reaching [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF], [START_REF] Luciw | Artificial curiosity with planning for autonomous perceptual and cognitive development[END_REF]. Most of the time, the outcome of such systems is the learner, but it sometimes happens that the learner is just here to guide exploration for a better outcome [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF], [START_REF] Kompella | Explore to see, learn to perceive, get the actions for free: Skillability[END_REF].

Learning progress implementation A local evaluation:

The idea behind progress is to guide exploration towards areas where the prediction error decreases fast. However, simply measuring the prediction error (and consequently, learning progress) at a global scale does not provide any information about areas that should be explored. It is therefore extremely important to provide a progress estimation at a local scale. A critical point is then to find a way to estimate the progress by grouping similar examples and providing a local measure for each group. Exploration will then favor examples in a group having a high measure of learning progress. These groups are called regions in Oudeyer's work [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF]. Finding the space where those regions should be defined has a strong influence on the efficiency of the method. In a discrete world representation with a small number of states, this may be done by considering the progress in each unit (state-action for example) of the environment [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF], [START_REF] Kompella | Autonomous learning of abstractions using curiosity-driven modular incremental slow feature analysis[END_REF]. In continuous environment, regions may be defined in the motor or sensorimotor space [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF]. Local estimation may also be part of the learner internal representation, considering for example the activation units of a self-organizing map [START_REF] Lefort | Active learning of local predictable representations with artificial curiosity[END_REF]. For learning sensorimotor skills, Baranes et al. [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF] have shown that defining regions in the tasks space, or goal space was much more efficient than considering regions in the motor space. This finding was consistent with the goal babbling approach proposed by Rolf [START_REF] Rolf | Online goal babbling for rapid bootstrapping of inverse models in high dimensions[END_REF]. Another way to consider regions is to see what the robot should learn as a set of problems (identifying several objects [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF], using several tools [START_REF] Forestier | Overlapping waves in tool use development: A curiositydriven computational model[END_REF]). A single region is then associated for each of these problems, and the capacity of the robot to solve each of them is used to estimate learning progress.

Prediction error measurement: The prediction error measure is easily defined as a distance (euclidean, for example) between the prediction and the learning signal when the learner is making a regression [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF], [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF], [START_REF] Lefort | Active learning of local predictable representations with artificial curiosity[END_REF]. In classification mode, distances may represent probabilities of the correct class [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF] or be based on a set of classification output to produce an average accuracy (in our system, we use a more exotic measures, the F1 score, that is also a classification score). If the goal is to build a set of representative features, a distance to the nearest representant is appropriate [START_REF] Kompella | Autonomous learning of abstractions using curiosity-driven modular incremental slow feature analysis[END_REF]. Lastly, in [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF], a cross-validation measure was used to compare an estimated and a measured distributions.

Progress measurement: Lastly, once prediction error is evaluated, an estimation of the learning progress must be produced. The general idea is to build a predictor that estimates the next prediction error, and to consider progress as the difference between E(t), the actual error and E (t + 1), the estimated error at next step. Early approaches were using a meta-learner with complex prediction techniques (for example with neural networks [START_REF] Schmidhuber | Curious model-building control systems[END_REF]) to estimate E (t + 1). However, the progress can be seen and defined in a more natural and simple way. Kalplan [START_REF] Kaplan | Motivational principles for visual know-how development[END_REF] suggested to consider, for guiding exploration, the evolution of the error measure rather than the error itself. Mathematically, the evolution corresponds to a variation, a sort of derivative of the error. If the error is quickly decreasing, progresses are high, and a definition of the progress may be the inverse derivative of the error - dE(t) dt . Based on this idea, progress is then usually defined from a sort of linear regression over the last few local examples [START_REF]Rl-iac: An exploration policy for online saliency learning on an autonomous mobile robot[END_REF], [START_REF] Kompella | Explore to see, learn to perceive, get the actions for free: Skillability[END_REF]. Most of the time, progress is simply obtained by [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF], [START_REF] Oudeyer | The playground experiment: Taskindependent development of a curious robot[END_REF], [START_REF] Lefort | Active learning of local predictable representations with artificial curiosity[END_REF]

progress(t) = t-τ 2 k=t-τ E(k) - t k=t-τ 2 E(k) (6.5)
where τ is a time scale used to segregate recent and older events. An average over a few samples is preferred to a single example, as there is often a high variance in successive error measures.

Exploration policy

Last but not least, we raise the question of how to integrate progress estimation in the exploration strategy. To better understand the different proposed approaches, let us recall the basic exploration sequence of intrinsically motivated systems.

1. Initially, the robot is in a given state and a given region (that may or may not be the same object) The policy then depends on the estimated progress, but above all, on the type of actions the robot can take.

In the case where any state can be reached from any other state by a single action, and if all of these actions have a roughly similar execution time, there is no cost associated with reaching a given state. In this context, the general idea is to focus directly on the most progressing region by using an -greedy policy [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF], or probabilistic rules based on the proportion of learning progress [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF], [START_REF] Baranès | R-iac: Robust intrinsically motivated exploration and active learning[END_REF], [START_REF] Lefort | Active learning of local predictable representations with artificial curiosity[END_REF]. These approaches were shown to provide a set of example that is quasi-optimal, under the condition that learning functions are submodular [START_REF] Lopes | The strategic student approach for life-long exploration and learning[END_REF].

When reaching a particular state requires a sequence of actions, the problem must be seen as a Markov decision process. Reinforcement learning may then be used to determine the policy to follow. In this case, the learning progress represents another source of reward and is integrated in the policy calculation as it would be done for extrinsic reward. Q-learning [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF], [START_REF] Schmidhuber | Curious model-building control systems[END_REF], or model-based least square policy iteration [START_REF] Kompella | Autonomous learning of abstractions using curiosity-driven modular incremental slow feature analysis[END_REF], [START_REF] Luciw | Artificial curiosity with planning for autonomous perceptual and cognitive development[END_REF] are typical algorithms used to find these policies. In this context, robots are able to explore their environment without any extrinsic reward.

In our system, we use two types of configurations to determine the policy. The first one follows an -greedy policy, and is not well-suited for all of our setups. We then build a second policy based on reinforcement learning.

Summary and conclusion

In this chapter, we have examined the problem of exploration in both mobile robotics and developmental robotics. Although having different constraints and goals, and using a different formalism, many overlaps can be found between those fields.

For mobile robots, exploration is usually constrained by the robot's displacements that need to be safe and efficient. The navigation problem is typically formalized on a map in which the robot is localized. Path finding techniques are then considered to determine a good trajectory fitting the environment constraints. Exploration usually tries to optimize trajectories, either for map coverage, goal reaching, or models learning.

Exploration in developmental robotics is focused on the learning and skill acquisition process in a lifelong perspective. The field of intrinsic motivations to drive the robot's decisions and constrain exploration is very active.

The next chapter is dedicated to the description of exploration strategies improving the quality of our models of saliency. Our approach is based on the IAC algorithm that we applied on a foveated system (BioVision) and on a mobile robot.

IAC has not been applied before on a mobile platform, and is not perfectly adapted to mobile robotics constraints. The junction of these two field then conducted us to the creation of a new algorithm, RL-IAC, that is one of the main contributions of our work.

Chapter 7

Proposed approach 7.1 Introduction

In the scope of an autonomous, life-long exploration and learning, the robot must be equipped with an exploration strategy able to drive the robot's displacements. This one should allow the robot to collect interesting examples, while avoiding problems such as catastrophic forgetting. In the meanwhile, exploration should be organized and efficient, so that the time spent traveling across the environment to reach goals is minimized.

We studied in Part I the capacity of a robot to learn a model of visual saliency on the fly during exploration. This approach has proved to be efficient when a sufficient number of samples are provided to feed the model. However, nothing is specified in this approach about the way samples should be collected and how representative they should be of the environment. We investigate here the strategy the robot should apply to visit relevant places to learn saliency efficiently. The previous part presented an open-loop framework for learning such a model, and this part is a closed-loop framework to refine and improve the learning capacity of the system.

We design our method based on the Intelligent Adaptive Curiosity (IAC), an algorithm able to guide the robot's actions depending on its current state of knowledge, to make it learn better and faster. We first explain how to apply such a system to the task of online saliency learning. Then, we show the limits of this algorithm when applied to a mobile robot exploring a building, and propose an extension of this approach. This extension, called Reinforcement Learning-Intelligent Adaptive Curiosity (RL-IAC) is now able to find good trade-offs between the time spent taking actions, and the time spent for learning and improving the internal knowledge of the robot.

Overview of the method 7.2.1 Position versus state-of-the-art

In this section, we make a link with Chapter 6 by positioning our work versus state-of-the-art. For more clarity, this positioning is sometimes discussed directly within the corresponding sections. In this case, we simply make references to these sections.

Regarding visual attention

A major difference between our approach and traditional visual attention systems is the way we consider salient regions. Our purpose here is to consider saliency as something to learn rather that something to direct attention towards. We then direct our attention towards areas able to improve learning rather than towards salient elements themselves. As discussed all along this chapter, our focus of attention is focused at highly progressing areas.

Second, some visual attention techniques rely on a reinforcement learning setup to learn a sequence of saccades and displacements [START_REF] Paletta | Q-learning of sequential attention for visual object recognition from informative local descriptors[END_REF], [START_REF] Borji | Online learning of task-driven object-based visual attention control[END_REF]. We here learn some visual aspects of our environment in a reinforcement learning context, but the model learned in the end is completely independent from any action (saccade or displacement) of the robot.

Regarding mobile robotics

When considering experiments with a mobile robot, we rely on a SLAM algorithm [START_REF] Kohlbrecher | A flexible and scalable slam system with full 3d motion estimation[END_REF] that we consider reliable enough to position the robot. This SLAM techniques builds metric maps and localizes the robot in terms of position and orientation on this map.

In Section 7.4.3, we rely on topometric maps, and provide a possible approach to build these maps incrementally. We also make in Section 7.4.2 a positioning of RL-IAC with respect to classical path finding approaches. More precisely, RL-IAC relies on path finding techniques that are not directed towards external goals, but towards intrinsic ones. Goals are then chosen to optimize learning rather than area coverage. Our path finding is also based on a Q-learning approach, able to find a good policy given some learning quality measurements.

Regarding intrinsically motivated approaches

Lastly, our method relies on intrinsic motivation and is an implementation of the IAC algorithm. The main differences with other IAC implementations are discussed in Sections 7.3.1, 7.3.2, 7.3.2 and 7.4.2. We also propose two main contributions in that regard.

The first is a measurement of the learning progress for the case where the learner is shared among different regions, that we call the backward progress estimation. More importantly, the second is an extension of IAC, containing a more general action policy that the one originally proposed. Based on this extension, IAC is now usable on mobile robots exploring large environments and spending a lot of time in displacements across the building. We call this extension RL-IAC.

Mechanisms and contributions

General framework

IAC can be seen as a way to guide the robot's actions when exploration is dedicated to the task of learning something. Unlike exploration strategies whose aim is to have an extensive coverage of the exploration space, IAC focuses on particular areas of the space so that learning is done efficiently. The essential component of this technique is a local measure of the learning progress, that catches the attention of the robot until knowledge has been acquired. The general architecture and different components of such approaches were described in Section 6.4.3. We here focus on describing the mechanism in our particular framework.

IAC is then constituted of four main components:

• a learner that learns a particular model;

• a way to divide the exploration space into regions;

• a meta-learner that monitors the learning evolution in each region and estimates progresses1 ;

• a policy that determines the next action to take given the state of progress in each region.

In our implementation, the learner is the online classifier used in the saliency learning technique described in Chapter 4. The goal is to be able to predict, given an RGB image, the portions of the image containing salient objects. To this end, the saliency technique uses a supervision signal (also called learning signal) that provides, when possible, the state of saliency of a pixel of the image.

The exploration space strongly depends on the problem and varies a lot in the different implementations of IAC. The only constraint is that regions should be reachable by the different actions the robot can take (meaning that taking an action leads the robot to a given region), and wisely divide the space to locally estimate learning progress. In our case, regions are defined in the low dimensional space of positions the robot can reach. Unlike early approaches proposed by Oudeyer et al. [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] where regions are incrementally divided to refine learning progress estimation in relevant areas, regions are, in our case, separated along a regular grid. They are progressively discovered and updated during the experiment.

The meta-learner is the module that estimates the learning progress in each discovered region. To evaluate learning progress, the learning error (the difference between the supervision signal and the model prediction) is evaluated. In our approach, the supervision signal is the segmentation mask for pixels that have been segmented "salient" or "not salient", and the model prediction is simply the estimated saliency map. To obtain the learning progress, we consider the history of the error in each region as a function of time, and we define the opposite slope of this function as a progress measure.

Lastly, based on the learning progress estimation, the decision of the next action to take is determined. In our setup, an action is a displacement of the robot that leads to a new point of view, and, as a result, a new observation. The basic idea of IAC is to prioritize actions towards the most progressing regions. To this end, these techniques typically use a greedy approach in the regions with highest learning progress. We first follow a similar approach and highlight its potential inefficiency when applied to a mobile robot. We then propose an extension of the IAC architecture, that we call RL-IAC (for Reinforcement Learning-Intelligent Adaptive Curiosity) containing an additional module able to follow learning progress while being still efficient when a greedy policy fails.

Figure 7.1 summarizes the main components of the architecture and the way they interact with each other. We here provide a brief recall on the basic procedure for using it. At time t, the robot receives observations from the main and additional visual streams, as described in Section 7.3.1. They are, on the one hand, turned into a segmentation mask S, and, on the other hand, sent to the online classifier to estimate the saliency map S. Then, the learner and meta-learner are both updated: S is first turned into labels and sent with the extracted features F to the online classifier (learner) for a model update. Second, as this observation was taken at a particular point of view, in a region R, the prediction error S -S is added to the error history of R, and the progress in region R is re-evaluated. Lastly, the next action of the robot is calculated from a policy depending on the learning progress. The robot then takes the action, ends up in a new position and receives an new visual input that is to be processed, and so on.

New visual observation

Regions definition

Two application scenarios

The implementation of IAC in the scope of saliency learning has been tested on two different types of robots. Although based on the same mechanism, the exploration does not have the same constraints, and the process of action selection to guide exploration is different. The first robot is BioVision, described in Section 2.2.1. In this case, the robot is an anthropomorphic head put on a table, watching by fixations and saccades at a static scene. The only actions the robot can take are motor commands modifying the pan and tilt orientations of the head.

The second robot is a mobile robot described in Section 2.2.1. This time, the robot is able to move across the rooms of the experimental environment. Actions taken in this case are based on reachable positions on a 2D map representing the environment, and the orientation of the robot within the environment.

In both scenarios, exploration is made possible by displacing the robot's field of view to a new position. Thus, the salient elements to be observed are not the same, and can be integrated to the model. Action then consists in sending commands to reach a particular point of view. In both scenarios, the learner is the online classifier, and the meta-learner calculates learning progress in a given region the same way. Section 7.3 is dedicated to the description of the components of IAC that apply to both setups.

The major difference between the two scenarios is related to the nature of the actions. For BioVision, reaching a position of the pan-tilt space is relatively fast, whatever the current position of the robot. Actions can be considered negligible with respect to the processing time required for segmentation, saliency computation and model update. For the mobile robot, the size of the environment does not depends on the architecture of the robot, but on the choice of the experimenter. In small environments (such as rooms) where different points of view are reachable by many rotations and little displacements, the same kind of approximation can apply.

However, if the space to consider is a building, the displacement required to reach a position at the other end of the environment may be time consuming and much longer than the processing step. The question of the time for performing actions is one of the main problems of Section 7.4. We there propose RL-IAC, an extension of the traditional IAC implementations in which the time required to displace the robot to a given position is considered. This way, a good trade-off is found between the time spent for updating the saliency model, and the time spent to get a new observation.

IAC and saliency learning

We provide in this section the implementation details for applying IAC to the case of saliency learning.

Learner

The online classifier described in Chapter 4 that learns the saliency model plays the role of the learner. As most implementation details were provided in this chapter, we only give a brief overview on the way it is used in the IAC framework.

Saliency learning

Our learner tries to construct a prediction function I → S, able to estimate the saliency S of the visual field given an input RGB image I. This process is done by moving in the environment and collecting observations after each displacement.

Each observation is characterized by inputs from the main and additional visual streams. Features F I are extracted from the main visual stream, and a segmentation mask S is obtained from the additional visual stream to estimate the saliency label of each pixel of the image. Samples are obtained on a pixel-wise basis by associating pairs of features and labels.

These features and labels are used in two different modes, either to update the saliency model (rather called the learner's model in the context of IAC), or to infer saliency maps from the current model. Figure 7.2 presents the saliency learning mechanism and these two modes used in the IAC framework.

After an observation at time t + 1, a set of samples are extracted from the main and additional streams. The features are first sent to the learner to infer the saliency map S given the model updated at time t. S is used to evaluate the learning quality, further discussed in Section 7.3.3. Note that the focus here is not the quality of the saliency model itself, but the way this quality evolves with new observations. Second, both the features and labels are sent to the learner in learning mode to update the model at time t + 1. Therefore, a new model is available and ready to be used for inference once the update is finished.

Comparison with other IAC implementations

In our context, the learner is designed for a single activity, the one of saliency learning. Additional tasks could be easily integrated by adding classes to the online classifier, but they are not considered here.

Our approach then learns a mapping from a high dimensional space (the number of dimensions being the number of extracted feature maps) to a simple binary Unlike most techniques where each action provides a single sample, each observation provides hundreds of samples at the same time. The learning progress must then be updated by considering the integration of this large number of samples at the same time. In addition our self-supervision technique provides samples that are only partially supervised to evaluate the progress. This partial observation biases the learner as a large portion of the environment cannot be labeled and integrated to the model. This bias may have consequences on the learning progress estimation, as it is not representative of the whole environment.

Regions definition

Finding appropriate regions to estimate learning progress is an essential element. For each implementation of IAC, regions can be defined in several different spaces. Moreover, regions should be bounded by parameters that are defined prior to the experiment (size of the region, maximum number of samples, splitting criteria, and so on). We also detail in this section the reflection that conducted us to use this kind of regions in our implementation.

Proposed definition

To define appropriate regions, we considered subdivisions of a space containing all reachable visual points of view. Actions are defined in our scenarios by a displacement of the robot from a given point of view to a new reachable one. These points of view are more simply defined by positions in a proprioceptive or exteroceptive frame.

BioVision: For BioVision, the robotic head is placed on a table. It is therefore not moving within the environment as a mobile robot would. The only possible way to modify the point of view is to send a pan or tilt motor command so as to change the inclination of the cameras.

Positions of the reachable points of view are then defined as a pan and tilt position (p, t). Regions are defined by cutting the pan and tilt spaces in regular intervals. The robot is then in region R if the pan tilt position falls within the boundaries of this region. Figure 7.3 illustrates the regions boundaries and makes the link between the pan-tilt axes and the foveal point of view.

Of course, pan and tilt positions are not so easily convertible to a foveal point of view. However, we make the assumption that objects are far enough to consider the optical center of the foveal camera to be at the interserction of the pan and tilt axes. Pan and tilt values are easy to obtain and actions are simple pan and tilt motor commands. Our second assumption is to consider each displacement command to fall in the right position which, given the precision of the motor, is realistic.

In practice, we divide the pan positions (ranging from -60 • to 60 • ) into 4 intervals and the tilt (ranging from -30 • to 30 • ) into 4 intervals as well. We then have the space divided into 16 regions.

Mobile robot: A mobile robot has the ability to modify the point of view of its cameras by moving within the environment. We here consider the 2D map space, that is a projection of the environment to the floor. In this space, the robot's position is represented by a triplet (x, y, θ) where (x, y) are the coordinates of the robot on the map, and θ is the orientation. This triplet is enough to define a unique point of view, as the camera mounted on the robot is static. Again, we define regions in this space by cutting each dimension in regular intervals. Figure 7.4 presents the trajectory of a robot in a room, and a possible way to define regions in the map space. Here, the translation dimensions are split in 2 × 2 intervals, and the orientation in 4 intervals. In total, the space is then divided in 16 regions.

An action in this configuration is given by a position (x , y , θ ) to reach from the current position (x, y, θ) of the robot. In our work, we do not consider the conversion of this position into a trajectory avoiding obstacles, and the conversion of this trajectory to motor commands. We suppose these steps as being processed by a reliable plug-and-play module. Similarly, position and mapping obtained by a SLAM algorithm is considered as providing a perfect localization.

As the map of the environment is not evolving so much in time (at least for walls), regions may be defined manually before the experiment from a reliable and static map of the environment. However, in section 7.4.3, we present a way to obtain these regions incrementally directly during the robot's exploration. This is particularly interesting when the robot has to explore a large environment (for example, a building).

Remarks: This choice of regions was motivated by the low dimensionality of the represented space (two dimensions for BioVision, three for the mobile robot). Because dimensionality is small, cutting the space along a regular grid provides a very simple and safe way to generate regions, while providing consistent groups of samples within regions. In addition, regions boundaries are chosen to be small enough to group similar samples within a region, and large enough to limit the number of regions to explore.

In recent implementations of IAC [START_REF] Nguyen | Learning to recognize objects through curiosity-driven manipulation with the icub humanoid robot[END_REF], [START_REF] Forestier | Overlapping waves in tool use development: A curiositydriven computational model[END_REF], the algorithm defines regions in a space of problems (or activities) that are accessible to the robot. We may consider our regions definition as a set of activities as well: In this case, a problem would be the one of learning saliency within a bounded set of reachable points of view. The robot then has to switch between activities by performing displacement actions.

Discussion 1: role of the actions for saliency learning

In many IAC implementations, where the goal is to anticipate the sensor state after performing an action, the learner's model integrates the different actions taken by the robot. For example, if the goal is to predict the position of the hand Y of the robot in the visual field given a motor command of the arm X, then the learner constructs a function X → Y based on any action X taken by the robot. In this case, the action space is used both by the learner and the meta-learner. On the one hand, the meta-learner uses learning progress to guide actions. These actions are performed to provide samples to the learner. On the other hand, actions are used directly as inputs for the learner to predict future sensor states.

In our implementation, the robot's actions are just used as a way to modify the point of view. These actions are essential to provide a good panel of visual inputs and construct a representative saliency model, but they are independent from the learner in the inference stage as the learner does not integrate any data related to actions or motor commands. Recall that we try to learn a mapping I → S between images and saliency maps. Instead of using a sensorimotor loop to learn a sensorimotor mapping, we learn a sort of sensori-sensori mapping, where I would be a low-level sensor (pixels are RGB colors without any meaning) and S a highlevel sensor (in the segmentation mask, each pixel intensity represents the interest of the entity -floor, walls or objects -they belong to).

A consequence is that, as visual signals (the input space of the learner) and actions (output space of the meta-learner) are independent, our version of the algorithm is somehow compatible with dynamic environments. If a salient object is moved somewhere else in the environment, it will still look roughly the same in the new position, and is therefore likely to be estimated salient by the learner.

Discussion 2: on defining regions in the learner spaces

The first versions of IAC [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] were using a motor babbling approach by defining regions in the motor space, that was the input space of the learner. Other implementations [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF] preferred a goal babbling approach by defining regions in the goal space, the output space of the learner. It is then naturally interesting to consider the possibility of using the input or output space of our learner to define regions for our problem.

The input space of our learner is related to the RGB image of the main visual stream. A first approach would then split the image space into regions. Given the very high dimensionality of this space, finding relevant clusters is potentially complex and not so appropriate. More realistically, we could consider the input of the classifier that receives feature vectors. The dimensionality is here much lower (39 to 1024 depending on the method), but finding an appropriate distance to determine clusters in this space remains difficult. The few attempts we conducted in that regard did not show any promising result.

But more importantly, selecting regions in the feature space raises a problem for reaching samples. An implicit requirement for defining a region is that the robot should know the action (or sequence of actions) required to get a sample in this region. If not, the policy of action selection based on learning progress does not make any sense. For features describing pixels of an image, there is no direct way to associate a set of features with a set of positions in the exploration space, unless the space has been fully explored and analyzed beforehand. Thus, the image feature space is not well-suited for regions definition in the IAC framework.

As the classifier is binary, the output space only has two states ("salient" and "not salient"). A goal babbling approach would then select samples that are salient or not salient depending on the progress in each of these categories. However, the same problem of reachability occurs and cannot provide a good action policy.

Meta-learner

The meta-learner aims to monitor the local error made by the learner, and derive an estimate of the learning progress. The local estimation is made possible by grouping and making statistics on samples collected within the same region. Recall that the robot is in region R i at time t if its current position falls within R i 's boundaries.

We provide in this section two methods for estimating learning progress. The first one is the most natural one, and the most frequently used in general in IAC implementations. We call it the forward evaluation. We then demonstrate that this kind of evaluation has some limitations when the learner is shared among regions, and we propose a backward evaluation able to overcome this issue.

Forward evaluation of the learning progress

The error made by the learner at time t is computed by comparing the segmentation mask S(t) (obtained from the additional visual stream and standing for the ground truth in this case) with the corresponding saliency map S(t) (the estimated signal with the current available model). More precisely, from the RGB main visual stream I(t), we extract features F I (t). In parallel and from the additional visual stream, we compute a segmentation mask S(t). We then consider the image label set L(t) by keeping only pixels labeled "salient" or "not salient" in S(t). We estimate and binarize the saliency response S(t) for each of these pixels to obtain the image estimation set E(t) 2 . Lastly, we compute the estimated learning error Err(t) of a particular frame based on Equation 7.1:

Err(t) = 1 -F 1 (L(t), E(t)) (7.1)
where F 1 (., .) is the F1 score:

F 1 = 2 • precision • recall precision + recall = 2tp 2tp + f p + f n (7.2)
with tp, f p and f n the true positives, false positives and false negatives 3 . We call this error evaluation is called "forward", as the samples used to estimate the error

are not yet integrated to the model. The evaluation then considers the prediction of saliency for samples that have never been seen before.

To evaluate learning progress, the meta-learner stores a history of the prediction error for each region. Suppose now that at time t, the robot is in region R i and makes an observation in this region. Suppose that in this region, n -1 observations were already recorded and added to the history from the beginning of the experiment. The observation at time t is then the nth of region R i , and the learning error Err i (n) associated with this observation is then added to the history of R i .

The estimation of the learning progress in R i , is obtained by exploiting the error history sequence. For that, we apply a linear regression of the error history over the last τ samples:

    Err i (n -τ ) . . . Err i (n)     = β i (n) ×     n -τ . . . n     +     (n -τ ) . . . (n)     (7.3)
with (n) the residual error and β i (n) the regression coefficient. β i (n) then represents the derivative of the learning error after n observations. The learning progress being defined as the derivative of the learning curve (opposite of the prediction error), we obtain the progress LP i (n) in region R i by Equation 7.4:

LP i (n) = 2 π |atan(-β i (n))| (7.4)
We transform the slope β i with an arc tangent to have the learning progress normalized between -1 and 1, and we consider the absolute value of the arc tangent to force the robot to explore regions where learning is decreasing as well. Figure 7.5 illustrates this procedure.

Shared learner and backward evaluation

In some implementations of IAC [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF], [START_REF] Baranès | R-iac: Robust intrinsically motivated exploration and active learning[END_REF], a dedicated learner is associated with each region, so that the model learned in a given region is not influenced by any sample collected in another region. The forward evaluation is well-suited for this case, as the learner is not influenced by observations collected in other regions. Conversely, saliency shows common properties in different areas of the environment (for example, walls are often uniform and white, whatever the point of view at which you observe them). It is then more interesting to have a common learner shared by different regions. The drawback is that taking samples in a region may influence the performance of the learner in other regions, and therefore, their learning progress.

With a forward evaluation, the consequence of adding samples from outside a region R i are measured only when new samples are observed in R i , thus making the learning progress not evaluated as frequently as it should. Moreover, this measurement is clearly biased, as the difference between Err i (n -1) and Err i (n) does not measure how much the nth sample helped improving the model, but rather how much the whole observations (in any region) taken between sample n -1 and and n did. To avoid poor estimation of the learning evolution in the case where a region is not visited frequently enough, we define a way to monitor learning progress based on previously observed examples, that we call the backward progress estimation.

The proposed backward evaluation relies on a set of previously observed feature maps and segmentation masks in each region. Error estimation is then obtained from previously observed data in each region, making progress evaluation possible even when the robot is taking observations outside of the region. In deeper details, let us consider that at time t, the learner is associated with a model M t constructed with previous observations taken from all regions of the environment. At that time, let us consider that the robot has already taken n observations in region R i from the beginning of the experiment (the robot may or may not be in that region at time t, it does not matter).

We then define a regional label set L i (n) = {L i (1), L i (2), ..., L i (n)} for region R i , where L i (k) is the image label set obtained from the kth observation (See Section 7.3.3 for the detailed procedure related to the image label set, denoted L(t) in the explanations). This regional label set is updated each time a new observation is taken in R i .

The regional feature set F i (n) = {F i (1), F i (2), ...F i (n)} is obtained in a similar manner by considering the image feature set F i (k) of the kth observation. Similar to the image estimation set of the forward estimation, F i (k) only contains features for which the pixels were stored to create the image label set. The regional label and feature sets are then stored for each regions and updated each time a new observation is collected in this region.

To obtain the backward evaluation of the error Err back i (t) in R i at time t, we now need to generate a set of saliency estimates based on the current model M t . To this end, we simply use each feature vector of F i (n), binarize each produced estimate, and constitute this way the regional estimation set

E Mt i (n) = {E Mt i (1), E Mt i (2), ...E Mt i (n)}.
The backward evaluation is now estimated after each observation in each region (whatever the position of the robot), by considering the regional label and estimation sets:

Err back i (t) = 1 -F 1 (L i (n), E Mt i (n)) (7.5)
The error history is also updated at the same time, and the learning progress reevaluated as well.

The procedure to obtain the backward progress estimation is very similar to forward version. The main difference is that the history is represented as a function of time rather than as a function of observations in the region. Taking the same notations as Equation 7.3, the backward learning progress is obtained by

    Err back i (t -τ ) . . . Err back i (t)     = β i (t) ×     t -τ . . . t     +     (t -τ ) . . . (t)     (7.6)
and

LP back i (t) = 2 π |atan(-β i (t))| (7.7)
For practical reasons, we also limit the size of L i and F i by keeping at most 10 frames in memory from R i , and by randomly replacing them when new inputs are available. The backward evaluation procedure is summarized in Figure 7.6.

Displacement policy

Now that learning progress is evaluated in each region, the measure is sent to a module able to decide the next region to visit, and more precisely, the next action to take in that regard. LPi back (t) = -β i Progress estimation

1 0 0 … 1 1 0 1 … 0 1 1 0 … 1 0 0 0 … 1 L i (1) L i (2) L i (n) E i (1) E i (2) E i (n)
Previous masks taken in R i

A greedy approach

In most IAC implementations, learning progress is used directly as an intrinsic reward the robot should follow. More precisely, many implementations select the region having the highest learning progress and randomly choose an action among all possible actions leading to that region. As the next action is selected based on immediate learning progress, without any long term planning consideration, we call this behavior "greedy". We also follow an -greedy procedure to select the next action. However, we do not directly select the most progressing region, but select the region with a probability proportional to the learning progress. This idea was already proposed by Baranes et al. [START_REF] Baranès | R-iac: Robust intrinsically motivated exploration and active learning[END_REF], and suggests that the probability of the next region to visit r being region R i is given by equation 7.8

P r(r = R i )              LP i (t) N j=1 LP j (t) if v > 1 N otherwise (7.8)
where LP i (t) is the learning progress (forward or backward) described in Section 7.3.3, v is a uniform random variable, and = 0.25 is used to select a random region 25% of the time.

Once the next region is selected, a position within this region is randomly determined as the target to reach. The corresponding action to reach this position is then used to displace the robot.

Limitations

In the approach we described in this section, we never mentioned the time required to take an action, and this time is never integrated in the model to decide the next action. The IAC actually considers each action to be costless. This hypothesis holds when the time required for learning is longer than the time required to perform an action, or when all actions take the same time.

For a mobile robot moving in a large environment (e.g., a building), the displacements between two regions can be extremely time consuming, thus making the proposed greedy policy inefficient. We therefore propose to extend the IAC policy with a module able to estimate the best trade-off policy between progress and displacement. This is the topic of the next section.

A last point worth mentioning is related to learning progress evaluation when a shared learner is used. Our definition of learning error actually measures how good the model is performing in a given region, and learning progress measures how much it should progress in this region. However, the true quantity the learning progress should measure is how much an observation in a given region should help improving the overall quality of the model. For a learner dedicated for any region, the forward evaluation provides this quantity (supposing that every region has the same weight), but this is not the case for a shared learner. We did not investigate here how to measure such a quantity in the case of a shared learner, but we believe this problem is worth exploring in a future work.

Learning on a mobile robot: the RL-IAC approach

We propose in this section an extension of IAC able to cope with problems where actions have a different cost. We here consider the case of a mobile robot exploring a large environment (in our case, a building), in which reaching position B from position A can be extremely time consuming.

In this section, we first describe our problem as an extension of the strategic student problem, and we describe a new algorithm called Reinforcement Learning-Intelligent Adaptive Curiosity (or RL-IAC) that is based on IAC, but proposes two additional modules to integrate the time required for taking actions in the decision process.

An extension of the strategic student problem

Initial problem

The strategic student problem (SSP) was introduced by Lopes and Oudeyer [START_REF] Lopes | The strategic student approach for life-long exploration and learning[END_REF] as a theoretical and general framework for lifelong learning problems. The SSP considers the case where a student has to study K topics within a time N before an exam, and to optimize his final grade by allocating a limited amount of time in each topic. The problem is modeled by considering the learning function q i , i ∈ [1..K] such that spending time n i studying topic i makes the student end up with grade q i (n i ). Each grade may be weighted by a factor C i to compute the global final grade. The optimization problem is then formulated as follows:

       argmax n 1 ,...,n K K i=1 C i q i (n i ) K i=1 n i = N (7.9)
The strategic student problem can be solved by simply looking at each time step the progresses made in each topic (considering the derivatives of the q i s), and allocating the next slot of time on the topic where learning progress is the highest. Experimental results demonstrated that this approach is optimal. This idea is consistent with the IAC approach of greedily following learning progress for exploration. As a bonus, Lopes and Oudeyer demonstrated that this approach was quasi-optimal if the learning curves are monotonic sub-modular 4 .

Analogy with a learning mobile robot

We now consider the problem of a mobile robot in a building, moving across K rooms and corridors to learn a model of saliency in each of them. Each room has its own learning function, and the strategic student problem can easily be applied to this case. However, a major difference is that the displacement of the robot between rooms has a non negligible time during which the robot does not learn. In the SSP, the transition between topics actually not considered. However, the problem could be made more complex by considering that the student learns a given topic by consulting books on site in different specialized libraries in town. In this configuration, switching between topics is pretty time consuming, and the student must optimize the time spent in displacements on top of the time allocated for learning. The problem would then extent the SSP as follows:

         argmax n 1 ,...,n K K i=1 C i q i (n i ) K i=1 n i + i,j T (i, j)ψ(i, j) = N (7.10)
where T (i, j) is the time required to move from library for topic i to library for topic j, and ψ(i, j) is the number of time this displacement has been done. Thus, available time N is divided between the time to learn different topics, and the time to switch between topics. The problem is now about jointly optimizing learning and displacement. If ψ(., .) is constant, the problem to solve is a simple SSP. However, switching between topics modifies ψ(., .) and the cumulated displacement time, which is equivalent to solve a Traveling Salesman Problem (TSP).

This problem cannot be solved in practice for several reasons. First, the learning functions are not known beforehand, so that only short-term estimates can be used to define the time allocation in the near future. A planning from time 0 to time N is not feasible. Second, the TSP is known to be NP-hard and is solvable in reasonable time only though heuristics. The problem is even more complex here as constisting in a joint optimization. Previous works already tried to jointly solve a TSP with the art gallery problem [START_REF] Wang | View planning problem with combined view and traveling cost[END_REF], but they solve both problems sequentially and with heuristics.

In RL-IAC, we then use an empirical evaluation of the learning progress that we update as soon as possible, and we rely on path planning techniques to estimate, given the current configuration of the environment and learning progress, the best path to follow. This path is re-estimated jointly with progress estimation update.

RL-IAC: General idea

RL-IAC works on the same principle than IAC in the sense that the exploration space is divided in regions, that a learner tries to make predictions about the environment, and that a meta-learner monitors the error made in each region and derives a learning progress. However, two additional components are considered in RL-IAC, and are meant to provide an action policy able to handle the time spent in performing actions. 1, representing the general mechanisms of IAC, the only differences are about the region module and the action policy modules. Provided as an input for the meta-learner in the case of IAC, the regions here also help in the action policy process. In addition, they are updated by the robot displacement by an incremental procedure.

IAC vs RL-IAC

First, in the region module, we now rely on a graph describing the connectivity between regions and encoding the average time required for displacements between regions. Given this representation, all displacements are not possible from a given region. To reach an arbitrary position from another position, a sequence of displacements may be required. In addition, the distance between regions is integrated in the action policy module given this graph representation. In mobile robotics, this kind of representation is called a navigation graph.

Second, the action policy module is a more complex one (called the reinforcementlearning module), that integrates the navigation graph and the learning progress to calculate a displacement policy that optimizes the amount of collected learning progress. To this end, the environment is modeled in a reinforcement learning context, where learning progress is a reward to be collected, and the navigation graph models the states and actions of the environment. The RL module is re-trained at each step to provide the robot with a displacement policy.

Aside from these two aspects, the learner, the region space definitions and the meta-learner work exactly the same way as in IAC. Please refer to the corresponding paragraphs of Section 7.3 for more details.

Link with path planning approaches on a mobile robot

The problem of action policy is turned in our approach as a problem of path planning where the robot has to move in the environment to collect some rewarding learning progress. The displacement of the robot is seen as a sequence of actions performed to collect this reward, that can be solved by techniques of path finding.

As mentioned in Section 6.2.2, path finding can be solved by many different approaches. Dijsktra [START_REF] Dijkstra | A note on two problems in connexion with graphs[END_REF] or LSPI [START_REF] Lagoudakis | Least-squares policy iteration[END_REF] 5 algorithms may have been used instead of Q-learning and would probably have lead to similar results given the inputs of the problem. Q-learning however has the advantage of being model-free and then potentially applicable to more complex scenarios.

Unlike most path finding problems, our robot does not have to reach a specific goal. In addition, the learning progress is, by nature, progressively vanishing during the experiment. We then consider our problem in a reinforcement learning context, without any external goal, and with a non stationary intrinsic reward. This kind of configuration is not very common in the literature.

Link with artificial curiosity

A few research works have tackled the problem of mixing learning progress or intrinsic motivation with reinforcement learning [START_REF] Chentanez | Intrinsically motivated reinforcement learning[END_REF], [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF], [START_REF] Schmidhuber | Curious model-building control systems[END_REF], [START_REF] Kompella | Explore to see, learn to perceive, get the actions for free: Skillability[END_REF], [START_REF] Luciw | Artificial curiosity with planning for autonomous perceptual and cognitive development[END_REF] to provide the robot with an intrinsically motivated exploration policy. In these approaches, there is no warranty on the optimality of the final policy. Indeed, the proofs of convergence towards an optimal policy only holds for a constant reward. Luciw reported this aspect in a previous work [START_REF] Luciw | Artificial curiosity with planning for autonomous perceptual and cognitive development[END_REF], and Lopes et al. provided a mathematical proof of the exploration efficiency of their approach [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF], without claiming optimality. So far, the benefit of using learning progress with reinforcement learning has only be empirically demonstrated.

The main difference between these approaches and RL-IAC is the way reinforcement learning is considered for action selection. In the aforementioned work, reinforcement learning is part of the experiment and is what makes the robot select a new action. As actions are taken and reward is collected, the policy is refined, and this new policy guides the future robot's actions. In our case, a new reinforcement learning problem is initialized and solved after each model update. The optimal policy is obtained by internal simulations, and the next action to take is decided based on the optimal policy. In other words, we follow a philosophy of optimal control [START_REF] Bryson | Optimal control-1950 to 1985[END_REF], where Q-learning is used as a tool to find an optimal policy given available data at time t, and use this policy to determine a single action.

Regions and navigation graph

The traditional region module is extended by a navigation graph representing the spatial relationship between regions. We discuss in this section the construction of such a navigation graph, and propose a method to build it incrementally as the robot explores the environment.

Regions and navigation graph formatting

Regions in RL-IAC still represent a set of reachable points of view in an appropriate space. We mainly reason here in the 2D map space of a mobile robot, as the method was designed in that purpose. However, the navigation graph definition still holds in other kind of spaces.

In Section 7.3.2, we defined regions in the (x, y, θ) space of the robot's position, divided into regular intervals. Here, we use a slightly different definition of regions, although simple considerations could make the former definition compatible with the description of this section.

Here, regions are defined as an area of the 2D map represented by a set of (x, y) positions that the robot can reach. The θ component representing the orientation of the robot is then not considered. This means that the region space is not formally the one of all reachable positions of the robot. However, when selecting an action to reach a given region, we add a θ component that fully describes the future state of the robot. In additions, regions are not necessarily obtained by regular subdivisions of the space.

To define a navigation graph from these regions, we consider the centroid of each area describing a region. These centroids stand for the nodes of the navigation graph. Regions are connected with each other if they share a common frontier. This is represented in the navigation graph by connecting the nodes by an edge. Each edge is associated with a weight that represents the euclidean distance between the two centroids. The proposed navigation graph then represents the environment as a topometric map. Figure 7.9 presents an example of regions defined by areas on a 2D map and the associated navigation graph. In this representation, nodes of the navigation graph are represented at the physical centroid position of the region, and weights of the edges are represented by their length.

The navigation graph is then a representation of the displacement cost, as time for displacement is roughly proportional to the distance. It also represents the set of reachable positions from a current position by considering the edges associated with the current region. This representation has the advantage of being compact and simple to obtain. However, this is only a rough approximation of the displacement cost, as only distances between centroids are considered. Moreover, the sequence of actions proposed to join two points of the map is not necessarily the shortest path.

Lastly, the way to obtain regions and navigation graph is flexible. If the environment map is known beforehand, a manual definition of the regions may be provided in order to define areas that are likely to be consistent (for example, a region could be defined as a room of a building). However, nothing in RL-IAC forces the regions to be static, so that we developed a technique to determine regions and navigation graph on-the-fly during the robot's exploration.

Incremental navigation graph building for a mobile robot

The method described in this section is able to jointly define regions and build a navigation graph as the robot moves in the environment. Both are then updated when new areas of the environment are discovered, or when known areas have changed (opening of a closed door for example). Note that this method has been designed for a mobile robot equipped with an RGB-D camera. More precisely, the method relies on an occupancy grid (the 2-D map) of the environment and the visible field of view of the RGB-D sensor. Figure 7.10 presents an example of regions and navigation graph obtained this way. On this figure, visible areas are colored depending on the region they belong to. Walls and obstacles are represented by gray pixels. Lastly, circles with region index correspond to the region centroid, and edges represent the available displacements of the robot. Each region has at most 4 neighbor regions in adjacent proto-regions Initially, the occupancy grid has a pre-allocated size, where each pixel state is "unexplored". This map is first divided in proto-regions based on a regular grid of arbitrary size. This way, each position of the occupancy grid is then associated with a proto-region. The regions determined in our algorithm are subdivisions of those proto-regions.

As soon as a new robot observation is available, we update the occupancy grid based on the RGB-D field of view: we transpose the point cloud obtained from the depth map in the occupancy grid frame. Points belonging to the floor are marked as "free" on the occupancy grid, and other visible points are marked as "occupied". Let us denote V the list of visible points marked as "free" on the map. Let us consider {P i } i=1..N the N proto-regions having an overlap with V . Each P i is then the list of all pixels contained in the square delimiting the proto-region. For a given P i , let us define {R j } j=1..M the pixels of the M regions contained in P i . Regions are now updated using the following procedure for each P i : • If V ∩ P i ∩ {R j } j=1..M = ∅, create a new region R M +1 constituted with all pixels of V ∩ P i ;

• If V ∩ P i ∩ {R j } j=1..M = ∅ and V is overlapping a single region R j , update R j with V . R j ←-R j ∪ V ∩ P i • If V ∩ P i ∩ {R j } j=1..M = ∅ and V is overlapping a set of L ≥ 2 regions {R k } k∈1.
.L merge all those regions by creating a region R M +1 = V ∩ P i ∪ {R k } k∈1..M . Then, empty all R k .

Figure 7.11 provides an illustration of each aforementioned cases. In this configuration, the robot is in proto-region P 2 but the field of view V covers proto-regions P 1 to P 4 . In P 2 , no regions were previously defined, so a new region is created (region 6). In P 3 , region 4 already existed but has no overlap with V : a new region is created (region 7). Region 1 in P 1 overlaps V , so the region is extended. Lastly, V overlaps two regions in P 4 , so that regions 2 and 3 are merged to create region 5. Last configuration, when a wall is splitting a proto-region, two separate regions are naturally created on each side (as they are not connected components). Thus, region 0 and 1 of P 1 are separated by a wall and cannot be merged together.

The nodes of the navigation graph correspond to the positions of the regions centroids (if not empty). Edges are determined based on the connexity between regions and their neighbours. In other words, if a region has a common border with another region, they are connected by an edge. Note that regions within the same proto-region do not have any connexity, otherwise they would have been merged. As a results, edges are necessarily between a region of proto-region P i , and a region of the upper, lower, left or right proto-region P j . We also attribute to each edge a weight representing the distance between the two centroids.

P2 P3 P4 V V

Reinforcement learning module

To determine an action policy able to take into account the displacement cost, we use the navigation graph, as it encodes the relative distances between regions and the possible displacements to reach them. We use this navigation graph to model states and actions of a Q-learning problem [START_REF] Watkins | Q-learning[END_REF]. The reward is represented by the learning progress in each region, and a batch of simulations is run on this setup to determine the policy that optimizes progress while minimizing displacements. After simulations, the robot moves to another region by following this policy. We suppose in this work that the positions of the robot are perfectly known and that a target position is perfectly reached. As Q-learning is designed for setups where the transitions between states are probabilistic, the integration of noise in the position measurement could be considered in the algorithm as a straightforward extension.

To describe how the module works, let us consider that the robot is in a region R i at time t. As the navigation graph and learning progress are constantly evolving, the procedure is repeated before each robot's displacement.

States, actions and reward definition

Q-learning problems are typically defined by states, actions and rewards. The theoretical background about the Q-learning algorithm is recalled in Section 6.3.2. In our setup, the states are the node of the navigation graph. They then represent the regions, and are defined by the region centroids.

Actions are represented by displacements on the graph. Each region is connected by edges with neighbor regions in one of the four (above, below, left and right) adjacent proto-regions. As a result, the robot selects one action among "up", "down", "left" or "right", depending on the graph edges, and moves to the corresponding neighbor region.

A reward r is associated with each region R j . We take for each region the last calculated learning progress. Thus, at time t, we define reward as:

r(R j ) = LP j (t) (7.11)
LP j being the forward or backward learning progress defined by Equations 7.6 and 7.7.

State-Action policy computation

To determine a state-action policy, we simulate a batch of 1000 episodes where a virtual robot moves in the navigation graph. For each episode, the initial state is R i (the actual state of the robot). During the episode, reward is collected right after taking an action and arriving in a given region. The episode stops when the travelled distance exceeds N = 1km 6 .

During the batch of episodes, the action-value function Q (also called the Q matrix for simplicity) is updated according to the following rule

Q(s k , a k ) = r(s k ) + γMax a ∈A Q(s k+1 , a ) (7.12)
with γ the discount factor, s k the region where the robot is after k (virtual) actions, a k the action to take next, A the set of all possible actions, and s k+1 the region after taking action a k . During the whole simulation, the virtual robot follows an -greedy policy ( = 0.1) to take the next action.

Robot displacement

Once all the episodes have been simulated, we use the Q-matrix to select the next (not virtual) region to visit. For that, we define the next action a t taken by the robot by Equation 7.13:

a t = Argmax a ∈A (Q(R i , a )) (7.13)
Here again, an -greedy policy is used and 10% of the time, the policy is not followed and the action is selected randomly among all available actions. We now consider the region R j connected to R i and accessible by executing action a t . To perform the action in reality, a position (x j , y j ) in R j is randomly selected and constitutes the next target to reach. Additionally, an orientation θ is also randomly determined to fully describe the next position of the robot. The robot then moves to this position, updates the navigation graph, grabs a new RGB-D input, updates the learner and meta-learner, and launches a new batch of simulation to determine the next position to reach.

Note that each Q-learning policy is obtained by considering the navigation graph and the reward as constant during the whole simulation. This assumption is not representative of the real world, as each displacement influences both the regions and the learning progress (that would eventually decrease to 0 when the learner cannot get any better). However, the assumption is accurate enough to provide a reasonable displacement. As the Q matrix is re-estimated before each new action, this approximation does not introduce a significant bias. Moreover, to force the robot to quickly get a first estimation of the progress in each region, we force the progress in a given region to be very high as long as less than three samples are collected in there. This additional constraint has the same effect as the R-max [START_REF] Brafman | R-max-a general polynomial time algorithm for nearoptimal reinforcement learning[END_REF] exploration policy in reinforcement learning.

Remarks

To conclude this section, we point out two features of RL-IAC that could be further investigated in a future work.

IAC as a trivial case of RL-IAC

The action policy module proposed in RL-IAC is noting more than an extension of the greedy action policy proposed by IAC. The greedy behavior of IAC could be obtained in RL-IAC by providing the action policy module with a navigation graph fully connected between regions, and having a displacement cost of 0 between each region. In this kind of graph, the best policy is such that the action to take from any region makes the robot end up in the most progressing region, which is exactly the behavior proposed by IAC.

Of course, using Q-learning on a fully connected graph with a large number of regions is not very efficient. The direct selection of the most progressing regions without running the whole Q-learning simulations would lead to the same result much faster, but this is just a matter of implementation.

Optimality and possible improvements

In RL-IAC, we calculate a policy at each step, apply this policy and recalculate a new one, so that the robot follows in the end a sequence of displacement that is not associated with a unique policy.

The policy is actually not the right tool for defining an optimal displacement sequence of the robot, as learning progress in the different regions are evolving.

In a future work, the computation of the next states to reach should integrate the diminution of the learning progress as new observations are taken in a given region, and a different representation of the states should be considered for calculating the optimal sequence of actions to take.

Conclusion

In this chapter, we proposed an exploration strategy dedicated for the task of learning saliency. More precisely, we have adapted and implemented the IAC algorithm for our particular problem and setup. Equipped with this exploration strategy, the robot is intrinsically motivated, and can efficiently explore and learn in a lifelong learning perspective.

This approach was developed for two types of setups: first, a fixed but orientable head equipped with a set of cameras. Second, and a mobile robot able to move across a building. We then detailed and discussed the implementation choices we took, as well as their innovation towards previous implementations.

A first contribution was the definition of the backward evaluation of the learning progress, which provides a better error evaluation in problems having a learner shared between regions. The second contribution (probably the major contribution of our work) is the creation of RL-IAC as an extension of the IAC algorithm, able to take into account long and costly displacements across the environment. RL-IAC is still, however, an open problem with many possible improvements.

Chapter 8

Experimental results

Introduction

The previous chapter was presenting the technical details of the proposed method to efficiently and autonomously explore the environment. We propose in this chapter a quantitative and qualitative evaluation of this approach. We then exhibit the behavior, advantages and drawbacks of both IAC and RL-IAC for the task of saliency learning.

First, we propose to evaluate IAC applied in the context of saliency learning based on several datasets and robotics platforms. We particularly exhibit the need for a methodical exploration of the environment to rapidly obtain a reliable model. We compare the efficiency of our approach with other exploration strategies to emphasize this finding. Aside from these results, We also examine the behavior of IAC in terms of time allocation in each region. We also study how and to which extend the method can adapt to a significant change in the environment.

Second, we study in more details the RL-IAC approach in the case where the displacement time is non negligible. We present in this section three environments in which the robot can jointly navigate and learn saliency. Based on experimental results on these environments, we first exhibit the need for using RL-IAC instead of IAC in this context. We then examine the influence of some parameters of the system on the exploration efficiency. Lastly, we demonstrate that RL-IAC outperforms other exploration techniques, and that learning progress as an intrinsic motivation factor is efficient in spite of estimation biases.

IAC

In this section, we provide a set of experiments aiming to evaluate the use of the Intelligent Adaptive Curiosity in the scope of saliency learning. To this end, we produce results from three different setups, and we evaluate both the performance of the algorithm and its behavior in this problem.

Experimental setups and evaluation procedure

Construction of the experiments

Our method is evaluated by conducting exploration in semi-simulated environments. This is done by using the recorded sequences presented in Chapter 2. We call them semi-simulated as saliency is learned from real images taken from these sequences, but actions taken by the robot are simulated.

More precisely, to make the robot reach a position in our setup, we simply select one of the frames recorded in the sequence. This process has in practice no physical Here, 16 regions are also defined before the experiment. This time the space is divided along the x and y axes of the map (two subdivisions for each), and the θ angle (four subdivisions) representing the orientation of the robot.

Here again, each observation recorded during the sequence acquisition is associated with a position lying within a region interval. Each region then contains a list of observations that the robot can reach by random selection. Here again, the time for displacement and rotation is not considered.

RGB-D scenes

The last evaluation dataset is constructed from the RGB-D scenes dataset, and more precisely the table_small_2 sequence. This time, we do not consider any information about the position of the camera, and simply consider the sequence as a temporal succession of observations. In this case, the space where to define regions is simply the temporal axis. We then define five different regions by spliting the sequence into 5 sub-sequences of equal length. Some of these regions were more challenging than other because of the local points of view. In this configuration, the robot reaches an observation by simply selecting a frame index in a given region. Figure 8.3 shows sample images for each region of the sequence.

Evaluation procedure

In general, the performance of IAC is hard to evaluate [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] as intrinsic motivation systems are designed to develop skills autonomously and independently of any task. The traditional approach that consists in evaluating the performance level of a task does not make any sense in this case. As alternative evaluations, Oudeyer et al. [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] have proposed a set of measures based on the robot's point of view, external point of view and the observation of successive stages. In our case, our system uses a developmental approach for the single task of learning a saliency model. An objective evaluation based on the saliency map quality is then possible.

The measure of the saliency quality is evaluated using the evolution of the overall error rate of the system: based on the reference frames of the datasets for which a ground truth is available, we produce an error measure of the estimated saliency maps. To this end, we produce after each learner update the saliency maps for each reference frame, and we compare them with the available ground truth. We use for that the F1 score and a very similar formula than the one proposed for the backward evaluation in Section 7.3.3:

Err ovl (t) = 1 -F 1 (GT , E Mt ) (8.1)
where GT is the ground truth label set, produced in a similar fashion than the regional label set of Section 7.3.3, with the set of available ground truth masks. E Mt is the estimation set produced with the available model M t at time t in a similar way than the regional estimation set of the same section.

Given this measure, we can monitor the evolution of the error rate all along the experiment and objectively compare the different exploration approaches. Note that the overall error rate has a different signification than the regional error rate used to compute learning progress. Indeed, the regional error rate is an intrinsic metrics, whose evaluation is based on segmentation rather than on an external ground truth. Second, we use the F1 score, rather than the ROC curve for evaluation to be consistent with our evaluation of the learning progress.

Behavior of IAC in the task of saliency learning

Learning curves in different regions

For IAC to be efficient and useful, the learning curves should be different in each region. If not, the learning progress would be the same everywhere, and a uniform exploration of the environment should lead to similar results much more easily.

We therefore estimate the error rate evolution on the table_small_2 sequence for each region taken separately. To this end, we run five experiments (one for each region) in which the robot is allowed to explore only in a single region. The corresponding curve then represents the evolution of the error rate in that particular region. Results are reported in Figure 8 We notice that some regions converge very quickly (R2 for example), while other are much more challenging (R4 and R5). These results were somehow expected, given the average point of view in each of these regions. In R2, the objects and the table represent a large fraction of the input images. The table is easy to find by the segmentation process, and objects on it are easily segmented. Conversely, regions 4 and 5 show the table and objects on it at a smaller scale. The background is much more present, and significant illumination changes (due to the auto white balance) make some frames harder to classify.

As a result, the variability in the error evolution among the different regions suggests that an intrinsically motivated exploration of the environment should lead to better and faster learning.

Time allocation in regions

We now demonstrate that the IAC approach provides such a methodical exploration of the environment, and that the exploration procedure taken by the robot follows some stages of development. To this end, we use again the table_small_2 sequence and compare the exploration using IAC and the random exploration.

Figure 8.5 represents the proportion of time spend in each region during the exploration. To build these graphs, we monitored during the whole sequence the regions associated with each observation of the robot. Then, we measured the local number of observations in each region by applying a sliding window of 50 observation all along the sequence. Each curve then represents the percentage of time spent in a given region.

As opposed to random selection, where the time spend in each region is almost constant, IAC shows a clear methodological exploration, consistent with the error rate presented in Figure 8.4. For example, region R2 progresses very fast in the beginning, then becomes flat after 50 observations. For that reason, the time spent in R2 with IAC exploration is slightly higher at first, then significantly drops to a very low rate. Region R1 and R5 have a similar behavior. Then, as R3 and R4 have a much slower learning rate (and, as a result, a lower progress), much more energy is spent in these regions to improve the model. In addition, a switch is clearly visible between regions R3 and R4 precisely when R4 starts making fewer progress than R3.

As a result, in IAC, the robots organized its exploration to visit regions of increasing complexity. To better visualize this process, a video is also available 1 , showing the evolution of the error in each region and the switch between regions.

Adaptation to a change in the environment

Most of our experiments use a static environment. We now want to examine the evolution of the exploration in the case where the environment is changing. Suppose now that BioVision is learning the saliency of the environment, when someone comes and suddenly moves the positions of the objects on the table. The learning curve in the regions containing objects should be drastically modified, and the exploration strategy should exhibit different patterns.

To produce such behavior, we consider three sequences of the BioVision dataset: we make BioVision start learning with a certain sequence, and we switch to a new one during the experiment. We use for that a first sequence with a moderate number of objects, and switch with two different sequences: first, a sequence in which all objects were removed from the table (called "no objects" in the explanations). Second, a sequence where additional objects were put on the table("new objects" in the explanations). Figure 8.6 illustrates the switch principle to simulate the change in the environment. To further analyze the behavior of IAC in this configuration, we run several experiments by changing the time at which the switch operates: after 100 or 400 observations, or never.

We first present in Figure 8.7 the evolution of the learning curve when operating a switch in the environment after 100 and 400 observations (curves 100 and 400 in the figure). We also display the resulting error rate without any change (curve inf). In the experiments, the error rate is evaluated based on the ground truth of the sequences constituting each experiment. As a result, the no objects and new objects experiments are not evaluated on the same set of ground truth and do not have the same error rate curves. As expected, the change in the environment (indicated by arrows in the plots) strongly modifies the learning curve quality: when the environment switches to a table without any object, the scene is simplified, and the robot cannot get additional samples to learn the visual aspect of salient objects. As a result, the later the switch occurs, the better the learning. This is verified by the first plot of the figure, where the switch after 100 observations produces a very bad model. Conversely, when the new configuration of the environment is a more complex one, the robot is able to learn the aspect of additional objects, which is likely to improve the learning quality. This time, the switch is supposed to make the model better. This is verified by the second plot of the figure, where the switch at 100 observations enables a fast decrease of the error rate. However, when the switch never occurs (represented by the inf curve), the error rate cannot reach a descent performance. Lastly, the curve corresponding to a switch at 400 observations seems to have the best final performance (as compared to the switch at 100). This results would need to be confirmed with additional experiments, but this may be due to the greater number of samples from the first configuration that produces a better balanced model.

Second, we represent the amount of time spent in each region by following a similar procedure than in Section 8.2.2. To better analyze the results, we group the 16 regions in four areas: first, regions containing salient objects at the beginning of the experiment. Second, the one containing a background easy to learn (white walls and table ). Third, the one containing a complex background. Lastly, the one for which the more complex environment contains salient objects and the initial one does not (See Figure 8.6 to visualize the different environments, and Figure 8.8 to visualize the groups of regions called areas in these explanations). We now display in Figure 8.8 the evolution of the proportion of time in each of these groups, for all of the experiments.

Regarding the curves obtained for both the no objects and new objects environments, the following behaviors can be observed.

• Between 0 and 200 observations, the exploration time drops in each area, except the one containing salient objects. This area is the most progressing one.

• After 900 observations, the area containing salient object decreases as well for the 'inf' curve. This is because the model does not make much more progress in this area.

When looking in more details at the first row of the figure, obtained when switching to the no objects environment:

• For the curve '100', after 200 observations, exploration decreases in the salient objects area and increases everywhere else. This is because this area does not contain any salient object anymore.

• After 200 observations, the complex background area now seems to be the most observed one.

• The same behavior is observed for curve '400', after 600 observations. Lastly, when examining the second row, obtained when switching to the new objects environment, the following comments can be made:

• For the curve '100', after 200 observations, exploration decreases in the salient objects area and increases in the new salient objects area. This is because this area now contains much more salient objects and is now worth exploring more.

• The same behavior is observed for curve '400', after 450 observations.

As a last general comment, we observe a certain delay (between 50 and 200 observations) between the actual switch of the environment, and the exploration behavior of the robot. This may be explained by the time required by the system to correctly assess the change in the learning progress. Therefore, the system is reactive to changes in the environment, but with a certain inertia.

IAC versus other exploration strategies

To demonstrate the benefits of exploring the environment using IAC, we compare the performance of the system with three other types of exploration strategies. For each dataset, results are reported in Figure 8.9 for the following explorations:

• Chronological: Frames of the sequence are integrated to the learner in chronological order;

• IAC: The next region to visit is determined with IAC and the next position to reach in it is selected randomly;

• Random: The next region to visit is determined randomly and the next position to reach in it is selected randomly as well.

Each of these exploration techniques were tested 10 times on each dataset, and the mean and standard deviation were considered and displayed in the result curves.

As a reference, we also display the error rate obtained by training the classifier offline with all the dataset (Offline in the figure), and the error rate of the bottomup saliency method BMS [START_REF] Zhang | Saliency detection: A boolean map approach[END_REF] already used in Chapter 5 (BMS in the figure).

As expected, the offline model is a lower bound of the error rate, and the system tends to reach this limit after a certain number of observations, whatever the exploration strategy. Our method rapidly outperforms BMS when enough observations are obtained. The chronological exploration is slower to converge than random exploration at the beginning. Lastly, IAC seems to be the exploration strategy for which learning is the fastest. Note that the difference between random and IACbased exploration is less significant on the robot sequence, but the error variance of IAC is much lower than the one for random exploration.

RL-IAC

In chapter 7, we described the limitations of IAC, and the need for a new algorithm in the case of a mobile robot moving across a large environment. This section aims to provide quantitative evidences on this need, and evaluates the performance of RL-IAC as compared to IAC and other exploration strategies.

Experimental setup

To evaluate our results, we also rely on different experimental setups to simulate the displacement of the robot across the environment.

ENSTA

The first dataset was recorded in the ENSTA building and presented in Chapter 2. This time, we take the longest sequence recorded in the whole building (See Section 2.3.2 to get an overview of the environment and the trajectory taken by the robot). This sequence contains both the RGB-D images and the positions and orientations of the robot associated with these images.

From this sequence, we build a navigation graph based on the technique described in Section 7.4.3. Since this method constructs a navigation graph incrementally as the robot is getting new RGB-D inputs from its environment, it could work in parallel with the saliency learning model and the RL-IAC exploration strategy. However, to simplify the experimentation, we use the sequence once to construct a navigation graph, and we start the RL-IAC exploration with this navigation graph as a static input. The navigation graph used in all our experiments is depicted in Figure 8.10. Proto-regions were arbitrarily defined to be of 5 meters length.

Similar to the exploration procedure for IAC, we consider the positions of each observation recorded during the sequence, and we associate each of them to a region. When selecting a position in a given region, we then select one among all associated frames, we consider the position of this frame, and we simulate the displacement of the robot to reach it. Once attained, we use this observation to update our model. To get an overview of the incremental map building and the simulated robot displacement, a video is available 2 .

Note that some of the regions here did not contain any frame. In these regions, when the robot attempted to take an observation, no update was done in the model, so that the prediction error was always high, and the region not attractive in terms of learning progress.

RGB-D scenes

The two other datasets consist in artificial buildings constructed from the RGB-D scene dataset [START_REF] Lai | A large-scale hierarchical multi-view rgb-d object dataset[END_REF]. Each of the eight video sequences of the dataset is recorded in a single particular room (kitchen, office, meeting room), so that our artificial building contains rooms (one for each sequence) divided into 5 to 6 regions, with some regions connected to other rooms (as if there were doors and corridors between rooms).

We created two different building configurations, illustrated by Figure 8.11. In these maps, the navigation graph is superimposed on the map of the building, where regions are represented by a circle, and allowed transitions between regions are edges of the graph. The first artificial building, denoted as the short corridor, is composed with five of the video sequences, and contains a short corridor of three regions to switch between rooms. Some artificial doors between rooms also enable a pathway between them without using the corridor. The second one, denoted as the long corridor, is composed with the eight video sequences and contains three long corridors. The regions constituting the corridor represent almost half the total number of regions in the building.

To construct the navigation graph in each room, we first tried to apply a SLAM algorithm to the video sequences and use the (x, y) positions of the camera to define regions. However, the displacements of the camera were much smaller than the variation of the camera orientation, thus making the (x, y) positions not well-suited to define regions boundaries. To simplify the region definitions, we then cut each of the sequence into five of six sub-sequences of equal length (similar to what was done in Section 8.2.1), and we created an arbitrary trajectory to travel across the sequence. The navigation graph between regions is also arbitrarily designed. The main constraints were to have equally-spaced regions in the corridor, and a building with a plausible size. Lastly, we limited the number of connections per region to four. This corresponds to the four possible actions the robot can take, namely "up", "down", "left" and "right".

In both datasets, the observations that the robot can take in the corridors are such that nothing salient can be detected. The only observation to be taken in the corridor is a black RGB frame and a segmentation mask of only non salient pixels. This way, it is very uninformative for the robot to travel across the corridor. The large proportion of corridor in the second datasets should then emphasize the behavior of IAC.

Processing

Displacement simulation procedure

To simulate the displacements of the robot, we considered the following sequence of steps:

1. The robot takes an observation, and updates the meta-learner;

2. the robot determines the next region to visit given learning progress and Qlearning training;

3. the robot determines the next position to reach in this region;

4. the robot moves to this position; 5. while moving, the robot updates the learner based on the previous observation;

6. before taking the next observation, the robot waits for the displacement and the learner update to be both finished.

In our experiments, this sequence was repeated 3000 times. Each estimated error rate was timestamped with the simulated time starting at the beginning of the experiment. This timestamps was then used to plot our results.

To obtain the simulated time, we measure for each iteration the time spent by the system to compute steps 1 to 3 (not simulated), and we add the longest step between steps 4 and 5, as they are supposed to be run in parallel and wait for the other to be finished. Table 8.1 provides additional measurements to get a better overview of the execution time for each step.

The meta-learner update is much longer for backward than forward evaluation, because saliency evaluation is done on a single frame in one case, and on a set of samples in the other case. This step contains the time required for feature extraction (deep features on GPU), segmentation, and learning progress estimation (which is much faster than the first two steps). The Q-learning training is relatively constant in time, and the learner update processing time was detailed in Chapter 5.

For the robot displacement, we consider an average speed of 0.5 m/s, and we measure the time for a robot to reach a certain point by considering the euclidean distance to this point and a constant speed of 0.5 m/s. The maximum displacement time is then bounded by the distance between two points of two adjacent regions. Of course, this measure is an approximation, and more considerations should be taken into account to produce more accurate simulations (path planning for obstacle avoidance, non constant speed, rotation time ...)

To get a rough idea of the simulated time for a single experiment, the five steps of an iteration take on average 10 seconds. Given the 3000 successive iterations, an experiment then lasts for 8 hours.

The evaluation procedure is very similar to the one used in Section 8.2.1. The overall error rate is once again our main evaluation metrics. However, this time, the overall error rate is displayed versus time rather than versus the number of observations. Again, we run 10 times each experiment and consider the average and standard deviation to plot the result curves.

RL-IAC parameters influence

To evaluate and analyze the behavior of RL-IAC, we first examine the influence of some parameters of the system. Because many parameters could be investigated and fine tuned, we only select the most interesting ones and provide a few insight on the way they influence the learning quality.

Robot's displacement speed

To validate the need to consider displacement time in the exploration strategy, we first examine the error rate evolution when the robot is moving at 0.5m/s, and the corresponding evolution in the scenario where the robot can teleport across the building and instantaneously reach a given position.

To simply observe such a behavior, we run experiments on short corridor building with four exploration strategies, namely

• RL-IAC : the RL-IAC approach described in Chapter 7

• Random pos. : after each observation, the robot selects a random position in the building, determines the best sequence of actions (from the navigation graph) to reach it, and moves to that position.

• Random act. : after each observation, the robot selects a random action to reach a neighbor region in the navigation graph. It then selects a random position in that region and moves to that position.

• IAC : after each observation, the robot selects the most progressing region and a random position in that region. It then determines the best sequence of actions to reach it, and moves to that position.

Note that in the case of IAC and Random pos., the robot does not learn in all visited regions, but only the one targeted, once the displacement across the building is finished.

We then display the results of these experiments in two different ways. First, we plot the error rate evolution versus simulated time, based on the considerations described in the previous section. This plot correspond to the behavior of the robot with a 0.5m/s speed. Second, we plot the error rate versus the number of observations, without considering the displacement of the robot. This representation is the one used in the IAC evaluation part and roughly corresponds to the evolution if the robot could teleport across the building.

Based on results in Figure 8.12, RL-IAC is the most efficient approach when considering time, and IAC is the best when considering observations. As RL-IAC is designed to find a compromise between learning and displacements, the progress between two consecutive observations is not as good as the one obtained with IAC. matters in the calculation of the cumulated one. Thus, a small discount factor favors an immediate reward, while a large discount factors favor all rewards equally.

To measure the influence of the discount factor, we run RL-IAC on the three datasets by varying the value of γ (successively 0.1, 0.3, 0.5, 0.7 and 0.9). We report our experimental results in Figure 8. [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF].

The clearest result is on the long corridor dataset: the larger the discount factor, the better. This is probably due to the large amount of uninformative regions the robot has to come across. With an immediate reward the robot will probably never tend to visit rooms that are far away in the building. This behaviour is also visible in the ENSTA dataset, where a discount factor of 0.9 seems to provide the best exploration, while a discount factor of 0.1 provides the worst. However, the results are not as clear as they are in the long corridor dataset. Lastly, for the short corridor dataset, the discount factor does not have a strong influence in the exploration performance. Based on these results, we favor a high discount factor of 0.9 for the remaining experiments.

Forward versus backward evaluation

Third, we compare the performance of RL-IAC using a forward and a backward progress evaluation. We suggested in Chapter 7 that the backward evaluation was better suited to accurately measure the learning progress in the case of a learner shared among regions. To verify this hypothesis, we run again RL-IAC on the three datasets.

We compare the performance obtained with the forward and backward evaluation in Figure 8.14. At least for the ENSTA and the long corridor datasets, the backward evaluation provides a much better exploration strategy. The results are not as convincing for the short corridor dataset. This is probably because of the configuration of this building and the number of regions to visit. Indeed, the restricted number of regions makes the update of the forward estimation much more frequent, and the monitoring of the learning quality in this case more realistic.

Performance evaluation

As a last set of evaluations, we would like to compare the performance of RL-IAC versus other types of exploration strategies and other types of intrinsic motivation. To run the experiments in this section, we took a discount factor of 0.9 and considered the backward evaluation only.

Versus other exploration strategies

To demonstrate the benefits of exploring the environment using RL-IAC, we now compare the evolution of the saliency with different exploration strategies on the three datasets.

In mobile robotics navigation, the goal is generally to have a good coverage of the environment to explore so as to get an accurate mapping. This coverage approach may be solved by heuristics of NP hard problems (traveling salesman, art gallery problem ...). For example, Osswald et al. [START_REF] Oßwald | Speeding-up robot exploration by exploiting background information[END_REF] have used a pre-defined map and navigation graph, and solved a traveling salesman problem before exploration to obtain an efficient exploration route. This route was refined during exploration based on the sensory inputs of the robot. Our goal is not to make a mapping of the environment, but using an exploration based on an extensive and efficient coverage of the environment is a good baseline to compare with. For this reason, our first exploration strategy consists in determining an exploration pattern covering the whole regions of the environment, and repeating this pattern until the end of the experiment. In Osswald et al., the global path was found with the Concorde software [START_REF] Applegate | Concorde tsp solver[END_REF] that solves the traveling salesman problem with various heuristics. We used the same software with our regions configurations, and slightly adapted the resulting route to be consistent with the allowed displacements defined by our navigation graphs.

A few approaches rely on learning progress to guide exploration in a reinforcement learning context. In particular, Schmidhuber [START_REF] Schmidhuber | Curious model-building control systems[END_REF] or Lopes et al. [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF] have used Q-learning to guide the robot's displacements in a context where the only reward is the learning progress. This kind of approach is very similar to RL-IAC, but differs at a critical point: while a single Q-learning is run during the experiment and directly decides the next action of the robot in Schmidhuber's approach, we define and solve a new problem with Q-learning after each robot's displacements. We then use the entire problem to find the next best displacement rather than following a policy from a partially trained Q-matrix. Our second approach to compare with is then following Schmidhuber's approach: instead of running virtual displacement simulations to train our Q-matrix, we simply update it based on Bellman's equation after collecting reward r(s k ) in the current region (s k )

Q(s k , a k ) = r(s k ) + γMax a ∈A Q(s k+1 , a ) (8.3)
We also use an -greedy approach (50% random) to decide the next action to take. Figure 8.15 shows the evolution of the overall error rate in time on both environments, for the 4 exploration strategies:

• RL-IAC: As described in Chapter 7. Selects the next region from the Q-matrix, and the next position to reach in that region randomly.

• Uniform: We drive the exploration by a uniform coverage of the environment, from the sequence of regions determined with the TSP heuristic. This pattern is repeated until the end of the experiment. The next region to visit is determined from the sequence, and the next position to reach in that region is taken randomly.

• Schmidhuber: Similar to RL-IAC, except that the Q-matrix is updated after each observation rather than running a batch of simulations.

• Random act.: To get a worse case scenario, we select a random action to reach a region, and random position in that region.

On all datasets, RL-IAC is the method with the fastest decreasing error. The uniform exploration has a reasonable performance, even similar to RL-IAC in the short-corridor dataset. This can be explained by the fact that this setup only has a very small number of uninformative regions. Schmidhuber has a varying performance depending on the dataset. We actually found this approach very sensitive to the parameters of the experiment, and performing well with very different parameters than RL-IAC. For example, to converge rapidly enough, a large percentage of random actions were necessary (typically 50%), while RL-IAC only works with 10% of random actions. Lastly, and as expected, the random action is providing the worse performance, sometimes close to Schmidhuber's approach. as defined in Section 7.3.3 and used as the reward in the Q-learning training by the following terms:

• Error: Err back i (t) (see Equation 7.1). The agent is rewarded by exploring regions with high prediction error.

• Confidence: Conf i (t) = 1 t i , t i the number of frames observed in region R i . The agent is rewarded by exploring regions in which confidence is low (having the fewest number of observations).

• Uncertainty: U nc i (t) = t i j=1 -|E Lt j -0.5|, where E Lt j are the samples of the regional estimation set E Lt t i , described in Section 7.3.3. These samples are values between 0 and 1 and represent the probability of a sample to be salient. As a result, the closer to 0.5, the most "uncertain" the learner is about the sample. This measure was already used and detailed in Craye et al. [START_REF]Exploration strategies for incremental learning of object-based visual saliency[END_REF]. In this configuration, the robot is then rewarded by exploring regions producing fuzzy saliency maps.

We compare the performance of RL-IAC with the aforementioned intrinsic motivations on the long corridor and short corridor datasets. The comparison is displayed in Figure 8. [START_REF] Bylinskii | Mit saliency benchmark[END_REF]. In both configurations, progress seems to be the best source of intrinsic motivation, but the difference is much more significant in the long corridor configuration. In this configuration, the error fails as the absence of positive samples keeps the error rate very high in the whole corridor 3 . Confidence spends as much time in the corridor as in the rooms, thus making learning much slower. Uncertainty is the only one able to avoid the corridor, as, in the absence of positive samples in the model, saliency maps are set to '0' by default everywhere, thus making output saliency maps not fuzzy at all.

Conclusion

In this chapter, we separately evaluated IAC and RL-IAC for the task of saliency learning. To demonstrate the generalization capability of our approach, we evaluated them on several datasets. As a summary and concluding remarks, our experimental evaluation led us to the following findings:

First, IAC shows a clear behavior of methodical exploration of the environment. When driven by IAC, the robot first focuses at regions where learning progress is high. Then, as learning progress decreases in these regions, the robot tends to explore more complex ones and spends more time in them to acquire knowledge.

Second, when a significant change occurs in the environment (for example, if all salient objects are removed), IAC makes the robot still able to adapt to the most progressing regions. Nevertheless, this adaptation is not immediate because of a delayed evaluation of the learning progress.

Third, when displacement time is not negligible, IAC can reveal very inefficient. Indeed, focusing on the most progressing region forces the robot to perform long displacements that could be used to learn in other regions nearby. As a result, the use of RL-IAC makes sense in large environments such as buildings. Nevertheless, when measuring the learning evolution in terms of number of observations rather than in time, IAC still performs better than RL-IAC.

Fourth, the parameters of the system have a strong influence on the performance. We demonstrated that the backward evaluation was, in general, better than the forward one, and that a high discount factor was preferable in a large environment. Nevertheless, IAC and RL-IAC's parameters are complex to optimize, and there is no rule of thumb to determine them before the experiment.

Lastly, both IAC and RL-IAC have been found to be efficient exploration strategies as compared with more extensive and uniform kinds of exploration. In addition, learning progress has been found to be an appropriate source of intrinsic motivation, as compared with other sources unable to detect unlearnable areas of the environment. Nevertheless, those findings are much more significant on large environments where the proportion of uninformative regions is high. Indeed, geometrical segmentation techniques are often unable to provide a result on the entire image frame, because of the quality of the sensor, or the reliability of the segmentation algorithm.

Intrinsically motivated exploration

Summary

In the second part of the manuscript, we investigated the methods to drive the robot exploration in order to improve the quality of the saliency model. We directed our research towards developmental robotics and the area of intrinsic motivations. We got particularly interested in the Intelligent Adaptive Curiosity algorithm, and designed an adapted version for the task of saliency learning.

IAC is a mechanism able to drive a robot's actions, based on the current state of knowledge, towards areas of the space likely to help improving this knowledge. This method conveys a methodical and efficient exploration strategy, by choosing actions that are adapted to the current skills of the robot: not to complex, not too trivial.

IAC relies on a measure of the local learning progress, estimated in regions of the exploration space. The actions taken by the robot are then selected so as to favor regions having the highest learning progress. This means that the robot focuses the attention on the most progressing region and tries to gain more knowledge in there. As knowledge increases, the learning progress vanishes, until a more progressing region catches the robot's attention.

Contribution

Our contribution here is the adaptation of the IAC algorithm for the task of saliency learning in general, and saliency learning on a mobile robot. This adaptation involved a number of design choices such as regions definition, learning error measurement, progress estimation, or action policy. This required a deep understanding of the algorithm to make IAC consistent and efficient for our application. In particular, we proposed a new way of estimating learning error in cases where the model to learn by IAC was shared among the regions. We called this error estimation the backward error estimation.

Moreover, we found that IAC was not integrating the time for processing action in the decision scheme. For a mobile robot, this factor was too important to be omitted. We therefore created an extension of IAC, called RL-IAC (for Reinforcement Learning-Intelligent Adaptive Curiosity) to overcome this limitation and ensure an efficient exploration strategy for mobile robots. RL-IAC then relies on reinforcement learning to find a good trade off between learning progress attraction and displacement time.

Aside from the improvements of the algorithm itself, this work was also the occasion to propose an original exploration strategy to the fields of foveated visual attention, and the one of mobile robotics. In visual attention setups, exploration is typically guided by saliency maps, and the focus is targeted at the most salient regions. We took in this work the problem the other way around by learning saliency with intrinsic motivation. In addition, the use of intrinsic motivation is not very common for mobile robotics. We then proposed an exploration approach based on knowledge optimization rather than the amount of information collected in the environment.

Experimental setups

As a last contribution, our work was tested and evaluated on several experimental setups. In particular, we constructed two different datasets from two robotics platforms.

The first platform is a bio-inspired head placed on a table and able to observe its environment by performing saccades and fixations. Equipped with both foveal and contextual cameras, this robot is an excellent platform for demonstrating bioinspired visual attention mechanisms.

The second one is a mobile robot equipped with an RGB-D sensor and a laser range finder. This robot is able to move across indoor environments such as rooms or entire buildings while building a map and localizing itself in this environment.

Our contribution then consisted in creating sequence datasets on those two platforms so as to be able to efficiently evaluate our work. We also processed all the recorded data, formatted them and annotated them to perform a numerical evaluation.

To evaluate the performance of IAC and RL-IAC, we also adapted our datasets to create a simulated environment in which the robot was able to move and collect images, while avoiding the constraints of a robot navigating across a real building. Moreover, these simulated environments allowed us to repeat the experiments several times, to precisely analyze the behavior of our algorithms, and to compare our approach with other exploration strategies.

Discussions and current limitations

This work is still an ongoing research that could be largely improved and extended. We discuss in this section the strength and limitations of our approach.

Incremental learning of visual saliency

Strengths of the method

The incremental learning of visual saliency was evaluated on four different datasets and has shown a very similar behavior in each of them. We demonstrated that the method was able to generalize saliency over a weak supervision signal, and that the produced saliency maps, when evaluated on a similar environment than the one the model was trained in, were more accurate than state-of-the-art techniques. Lastly, we evaluated the saliency quality on a dataset with a model of saliency trained on another environment. The results were still acceptable and outperforming state-ofthe-art. As a results, the similar behavior in each dataset suggests that the method is robust and should work in many types of indoor environments.

In addition, the modular architecture of our system makes it easy to replace components with more efficient ones. We already demonstrated this ability by testing several types of feature extractors and two types of segmentation masks. Using a new classifier, or even using the system for another related problem (joint object detection and recognition for example) should not require a strong re-implementation effort.

Current limitations

A major limitation of our approach is the strong dependence on the learning signal (the product of the object segmentation process). For saliency to be correctly learned, the segmentation needs to be reliable and adapted to the environment. If the robots explores an environment where so many objects are piled-up that the floor cannot be detected, the whole process will fail. As a result, the design of a reliable segmentation process is a potentially complex procedure that must be carefully treated prior to any experiment. This is the cost for having a self-supervised learning rather than a supervised one.

Another potential problem is that the algorithm is not completely incremental. As random forests are not designed to be incremental, the training set and the training time necessarily increases as exploration progresses. This aspect is limited by the process of forgetting oldest samples, but this approach is not viable in a lifelong learning scenario.

Lastly, the generalization capacity of the model is hard to predict. When testing the saliency algorithm in a new environment, there is absolutely no warranty about the performance.

Intrinsically motivated exploration

Strengths of the method

The implementation of IAC for the task of saliency learning has demonstrated a set of interesting behavioral properties and good performance on several datasets.

We demonstrated that exploration with this principle was methodical, but also logical. When monitoring the learning progress in each region, the exploration priority followed by the robot was intuitive and understandable. Equipped with this kind of exploration drive, our experiments have shown that the robot was behaving in a developmental way, thus reinforcing the findings of Oudeyer et al.in previous experiments on other problems [START_REF] Oudeyer | Intrinsically motivated learning of real-world sensorimotor skills with developmental constraints[END_REF], [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF], [START_REF] Forestier | Overlapping waves in tool use development: A curiositydriven computational model[END_REF].

The integration of reinforcement learning in the process of action selection was also a successful solution to the problem of long and varying displacement times to get new observations. We demonstrated with several experiments that exploration with RL-IAC was still efficient in this type of scenarios.

We also demonstrated the ability of IAC to be flexible to changes in the environment, by adapting the exploration strategy and refocus after some time on the areas of high learning progress. This property is extremely important in the scope of a realistic lifelong learning experiment.

Current limitations

In spite of promising and interesting results, IAC and RL-IAC still suffer from a few limitations. First, this kind of exploration strategies only show a strong advantage over other techniques when the environment contains a large fraction or trivial or unlearnable areas. Otherwise, a regular exploration pattern is likely to show the same performance.

In addition, IAC depends on many factors, and uses learning progress measurement at the heart the process. By definition, learning progress is obtained by approximating the derivative of an approximation of the learning error. This succession of approximation is likely to make the measure of progress unstable and unreliable. Our own experience feedback is that IAC is globally hard to set up, and required a lot of trials to get a good insight on the influence of the parameters.

Moreover, the action policy proposed by RL-IAC is not optimal and only provides a reasonable solution given the current configuration. We believe that further theoretical and experimental investigations should be conducted to create a more satisfying model.

Lastly, we highlighted the adaptation behavior in the case of a change in the environment of the robot, but the environment still needs to be globally static. In a very dynamic environment, the measure of progress will most likely not be reliable.

Perspectives

To conclude this manuscript, we suggest a few short and long term perspectives that would be worth investigating in a future work.

Short-term perspectives

Coupling saliency with object recognition Saliency maps are not an end in themselves. We provided in this work a way to learn them, but they are meaningless if not used in another context. A first possible plan would be to integrate them into an object recognition framework. We already suggested a way to use them to improve the proposal of agnostic bounding boxes around objects, but this could be largely improved to provide a framework able to both localize and identify objects within their environments.

Make RL-IAC fully incremental

In this work, we presented separately a method for building incremental maps and navigation graphs of the environment, and a method using a navigation graph to explore the environment. Those two are used one after the other, but everything has been done to build the navigation graph and autonomously explore at the same time. A few implementation efforts should be sufficient to make RL-IAC fully incremental.

Using more biologically plausible models

For practical reasons, the regions defined in RL-IAC are very arbitrarily splited and could be largely improved with biologically plausible components. For example, considering gist aspects to make regions more visually and biologically consistent is a possible approach. In addition, the use of an -greedy approach to guide exploration could be replaced by more biologically plausible stochastic transition models. Thus, the displacement policy could rely on the way human deploy their visual gaze in a scene which is not steered by a greedy process. In the BioVision scenario, fixation and saccades could then be driven by curiosity, but also the distribution of saccade amplitudes, orientations, the fixation durations and so on measured on human populations. To go one step further, and for the future research, it would make sense to consider some aspects used by saccadic models of the literature.

Long-term perspectives

Integrating the method in an end-to-end deep architecture In this work, we have shown the promising performance of deep features and highlighted the limitation of random forests as a non incremental classifier. The next algorithmic step would be to integrate the saliency learning technique into an endto-end deep learning architecture. The biggest challenge would then probably be the one of making the network both incremental and stable.

Because of the flexibility of neural networks, this architecture would have very interesting properties. First, the network could not only learn saliency, but also bounding boxes detection, and even eventually recognition. Ideally, the system should perform similar tasks than an end-to-end system like YOLO [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF], but incrementally learn this task for a dedicated environment.

Additionally, coupling neural network and intrinsic motivation (and especially learning progress) would be a very exciting area of investigation. Research in this area are at their early stages [START_REF] Kulkarni | Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation[END_REF], but there are for sure many interesting ways to explore in that direction.

Testing RL-IAC in a real environment

Our experiments have been so far limited to the recording of video sequences on a robot, and to the simulation of the robot displacements on virtual environments. Now that our method seems to be efficient in this context, it would be interesting to verify its performance in a real life scenario.

Given the estimations provided in the last experimental chapter, we suggested that the experiment should last for about eight hours for 3000 observation. This is a considerably long experiment for a mobile robot that will very likely lack batteries before the end, and needs a very reliable hardware and software to run for so long.

To make this possible, the robot would also need an efficient obstacle detection and avoidance mechanism, able to determine a realistic and reliable path between two locations, while avoiding potential obstacle on the way. The navigation graph should also be constantly updated based on the evolution of the environment (doors closed then opened, presence of humans, etc...).

The problem of discovering the exploration space at the first time is also a question to consider. Should we define boundaries of the experiment to make sure the robot does not explore dangerous or unexpected places (stairs for example).

To simplify, but also to enhance the experiment, the presence of a human assistant or teacher, helping the robot would be an interesting alternative.

A potential application of IAC for active image annotations

A last interesting future work could be related to the integration of IAC in an interactive manual annotation software.

Using active learning to simplify the task of image annotation is not a new idea [START_REF] Wang | Active learning in multimedia annotation and retrieval: A survey[END_REF]. Nevertheless, IAC would be perfectly suited to this task, and the work done in the scope of this thesis would be a good starting point to develop this idea.

Suppose that a video sequence needs to be manually annotated by providing bounding boxes around objects of interest. IAC would then be applied in the following manner: the sequence could be divided into subsequence of arbitrary length to define the regions. IAC would propose to the user selected frames and ask for a manual annotation. This annotation would serve as a learning signal, and would replace the segmentation mask of our current approach. Thus the user would not have to annotate the entire dataset, but only a set of frames selected by the algorithm to improve its own model. IAC should naturally avoid frames that are trivial (containing no annotation for example), and progressively focus on the most complex cases. Lastly, once enough confident in the model, the remaining frames could be annotated automatically. Résumé : La conception de systèmes de perception autonomes, tels que des robots capables d'accomplir un ensemble de tâches de manière sûre et sans assistance humaine, est l'un des grands défis de notre siècle. Pour ce faire, la robotique développementale propose de concevoir des robots qui, comme des enfants, auraient la faculté d'apprendre directement par interaction avec leur environnement. Nous avons dans cette thèse exploré de telles possibilités en se limitant à l'apprentissage de la localisation des objets d'intérêt (ou objets saillants) dans l'environnement du robot. Pour ce faire, nous présentons dans ces travaux un mécanisme capable d'apprendre la saillance visuelle directement sur un robot, puis d'utiliser le modèle appris de la sorte pour localiser des objets saillants dans son environnement. Cette méthode a l'avantage de permettre la création de modèles spécialisés pour l'environnement du robot et les tâches qu'il doit accomplir, tout en restant flexible à d'éventuelles nouveautés ou modifications de l'environnement. De plus, afin de permettre un apprentissage efficace et de qualité, nous avons développé des stratégies d'explorations basées sur les motivations intrinsèques, très utilisées en robotique développementale. Nous avons notamment adapté l'algorithme IAC à l'apprentissage de la saillance visuelle, et en avons conçu une extension, RL-IAC, pour permettre une exploration efficace sur un robot mobile. Afin de vérifier et d'analyser les performances de nos algorithmes, nous avons réalisé des évaluations sur plusieurs plateformes robotiques dont une plateforme fovéale et un robot mobile, ainsi que sur des bases de données publiques Title : Intrinsic motivation mechanisms for incremental learning of visual saliency Keywords : Visual saliency, mobile robotics, cognitive robotics, bio-inspiration, intrinsic motivation, object localization Abstract : Conceiving autonomous perceptual systems, such as robots able to accomplish a set of tasks in a safe way, without any human assistance, is one of the biggest challenge of the century. To this end, the developmental robotics suggests to conceive robots able to learn by interacting directly with their environment, just like children would. This thesis is exploring such possibility while restricting the problem to the one of localizing objects of interest (or salient objects) within the robot's environment.

Index

For that, we present in this work a mechanism able to learn visual saliency directly on a robot, then to use the learned model so as to localize salient objects within their environment. The advantage of this method is the creation of models dedicated to the robot's environment and tasks it should be asked to accomplish, while remaining flexible to any change or novelty in the environment. Furthermore, we have developed exploration strategies based on intrinsic motivations, widely used in developmental robotics, to enable efficient learning of good quality. In particular, we adapted the IAC algorithm to visual saliency leanring, and proposed an extension, RL-IAC to allow an efficient exploration on mobile robots. In order to verify and analyze the performance of our algorithms, we have carried out various experiments on several robotics platforms, including a foveated system and a mobile robot, as well as publicly available datasets.

Université Paris-Saclay

Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France

FIGURE 1 . 1 :FIGURE 1 . 2 :

 1112 FIGURE 1.1: Today's autonomous perceptual systems: (a) Roomba vacuum cleaner (b) Intelligent Personal Assistants (c) Fire and forget missile (d) Mars rover

FIGURE 1 . 3 :FIGURE 1 . 4 :

 1314 FIGURE 1.3: Cones and rods distribution in the retina

FIGURE 1 . 5 :

 15 FIGURE 1.5: Main areas and interconnections of the visual cortex.From Gilbert and Li[START_REF] Gilbert | Top-down influences on visual processing[END_REF]

FIGURE 1 . 6 :

 16 FIGURE 1.6: (a) A typical study case in developmental robotics: the infant developmental milestones. (b) A developmental robot: the ICub [12]

FIGURE 1 . 7 :

 17 FIGURE 1.7: General problem

  23] Craye, C., Filliat, D. & Goudou, J.F. (2016). On the Use of Intrisic Motivation for Visual Saliency Learning. ICDL-EPIROB-The sixth joint IEEE International Conference on Development and Learning and on Epigenetic Robotics, At Cergy Pontoise, France. Accepted for oral presentation ( 25% acceptance rate). Journals: Craye, C., Filliat, D. & Goudou, J.F. (2017). Exploring to learn visual saliency: The RL-IAC approach. To be submitted in the International Journal of Robotics Research Patents: Goudou, J.F., Craye, C. & Fagno, M. (2016). Procédé et système de reconnaissance d'objets par analyse de signaux d'image numérique d'une scène. THALES SA (THALES). EP3070643 -2016-09-21.

FIGURE 2 . 2 :

 22 FIGURE 2.2: The BioVision2 platform mounted on a Turtlebot

FIGURE 2 . 3 :

 23 FIGURE 2.3: The ENSTA mobile robot platform

FIGURE 2 . 5 :

 25 FIGURE 2.5: Sample images from the BioVision dataset

FIGURE 2 . 6 :

 26 FIGURE 2.6: Contextual images (top) and associated mask (bottom)

FIGURE 2 . 7 :FIGURE 2 . 8 :FIGURE 2 . 9 :

 272829 FIGURE 2.7: Map of the trajectory and views recorded by the robot in the ENSTA dataset.

FIGURE 2 . 10 :FIGURE 2 . 11 :

 210211 FIGURE 2.10: Ground truth masks and bounding boxes used in our experiments

FIGURE 3 . 1 :FIGURE 3 . 2 :

 3132 FIGURE 3.1: An example of saccades and fixation sequences (from [37])

FIGURE 3 . 3 :

 33 FIGURE 3.3: General architecture of most computational visual attention systems

FIGURE 3 . 4 :

 34 FIGURE 3.4: General architecture of Itti and Koch visual attention model (image from [38])

FIGURE 3 . 5 :

 35 FIGURE 3.5: Comparison of stimuli-based and object-based postprocessing on the same saliency method (images from Frintrop et al. [48])

FIGURE 3 . 6 :

 36 FIGURE 3.6: Sample image and ground truth from datasets. On the left, ground truth based on human fixations[START_REF] Borji | Cat2000: A large scale fixation dataset for boosting saliency research[END_REF]. In the middle, manually segmented image[START_REF] Liu | Learning to detect a salient object[END_REF]. On the right, dataset avoiding the center bias[START_REF] Ciptadi | An in depth view of saliency[END_REF].

FIGURE 3 . 7 :

 37 FIGURE 3.7: Visual attention setups (a) foveal vision [107] (b)PTZ[START_REF] Gould | Peripheral-foveal vision for real-time object recognition and tracking in video[END_REF] (c) mobile robot[START_REF] Cañas | An overt visual attention mechanism based on saliency dynamics[END_REF] (d) humanoid ICub[START_REF] Holzbach | A fast and scalable system for visual attention, object based attention and object recognition for humanoid robots[END_REF]

FIGURE 3 . 8 :

 38 FIGURE 3.8: Examples of segmentation approaches (a) over segmentation by SLIC superpixels [138]. (b) foreground-background segmentation. (c) semantic segmentation [139]

FIGURE 3 . 9 :

 39 FIGURE 3.9: (a) Object (here, pedestrians) detection. (b) Object recognition (from ImageNet challenge [24]). (c) Example of class-agnostic bounding boxes (with EdgeBoxes [169])

FIGURE 3 . 10 :

 310 FIGURE 3.10: Two categories of transfer learning: domain adaptation and multi-task learning

FIGURE 3 . 11 :

 311 FIGURE 3.11: Fully versus weakly supervised learning

FIGURE 4 . 1 :

 41 FIGURE 4.1: General architecture of our system

FIGURE 4 . 2 :

 42 FIGURE 4.2: Saliency learning applied to two kinds of additional streams: (a) an RGB foveal input and (b) a depth input.

FIGURE 4 . 3 :

 43 FIGURE 4.3: Feature extraction procedure from Itti and Koch[START_REF] Itti | Computational modelling of visual attention[END_REF]

  Figure 4.4 shows an example of the feature extraction process for one pixel.

FIGURE 4 . 4 :

 44 FIGURE 4.4: RGB-based feature extraction based on Make3D

FIGURE 4 . 5 :

 45 FIGURE 4.5: CAM mapping of the top 5 responding object. Image from [68]

FIGURE 4 . 6 :

 46 FIGURE 4.6: Example of deep feature maps extraction from an input RGB image

FIGURE 4 . 7 :

 47 FIGURE 4.7: Segmentation mask from foveal object recognition

  equation ax + by + cz + d = 0

FIGURE 4 . 9 :

 49 FIGURE 4.9: Main components of the plane tracker

FIGURE 4 .

 4 FIGURE 4.10: Pixel-wise data formatting

VF 11 (F

 11 V)>th 11 F 22 (V)>th 22 F 21 (V)<th 21 = 𝐶 𝑽) = 𝑃𝑟 𝐿11 (𝐶𝑙 = 𝐶) 𝑃𝑟 𝑁 𝐶𝑙 = 𝐶 𝑽) = 𝑃𝑟 𝐿𝑁7 (𝐶𝑙 = 𝐶)Pr 𝑪𝒍 = 𝑪 𝑽)

FIGURE 4 . 11 :

 411 FIGURE 4.11: Class probability extraction from random forests

  Saliency estimate S(x,y) = Pr(Saliency|V(x,y))Pixel (x,y)

FIGURE 4 . 12 :

 412 FIGURE 4.12: Saliency map reconstruction

FIGURE 4 . 13 :

 413 FIGURE 4.13: Illustration of the superpixel reconstruction technique

FIGURE 4 . 14 :

 414 FIGURE 4.14: Bounding box proposals biased by our saliency map

  3.1 to visualize examples of the produced segmentation masks.

FIGURE 5 . 1 :

 51 FIGURE 5.1: Samples of the ENSTA dataset and their associated ground truth. (a) RGB image, (b) manually annotated ground truth mask, (c) automatic segmentation mask, (d) manually annotated bounding boxes

FIGURE 5 . 2 :

 52 FIGURE 5.2: Ground truth masks and bounding boxes used in our experiments

FIGURE 5 . 3 :

 53 FIGURE 5.3: Generalization capabilities of the classifier. First column: the RGB input. Second column: segmentation mask. Third column: the estimated saliency map.

FIGURE 5 . 4 :

 54 FIGURE 5.4: Generalization capabilities from a weakly supervised signal (the foveal image)

FIGURE 5 . 6 :

 56 FIGURE 5.6: Execution time of the classifier update as number of samples increases

FIGURE 5 . 7 :

 57 FIGURE 5.7: Evolution of the saliency model by generating saliency maps of fixed frames during a sequence.

FIGURE 5 . 9 :

 59 FIGURE 5.9: Sample results of saliency maps obtained with the three types of features

6 FIGURE 5 . 13 :

 6513 FIGURE 5.13: Sample results of bounding box proposals versus EdgeBoxes

FIGURE 6 . 1 :

 61 FIGURE 6.1: Basic architecture for autonomous navigation on a mobile robot. Localization, mapping and path planning are the three essential components of the control.

FIGURE 6 . 2 :

 62 FIGURE 6.2: (a) Real building map (b) Topological map (c) Metric map

FIGURE 6 . 3 :

 63 FIGURE 6.3: Occupancy grid obtained by a SLAM algorithm and possible associated navigation graph modeling rooms, halls and corridors.

FIGURE 6 . 4 :

 64 FIGURE 6.4: Direct path finding with the A* algorithm (source Sebastian Lague[START_REF] Lague | Github seblague path finding project[END_REF]) and potential field policy (from[START_REF] Schwager | Multi-robot systems lab projects[END_REF])

FIGURE 6 . 5 :

 65 FIGURE 6.5: Example of exploration problem.

FIGURE 6 . 7 :

 67 FIGURE 6.7: Example of developmental robotics platforms. (a) iCub, (b) baby robot, (c) Poppy

FIGURE 6 . 8 :

 68 FIGURE 6.8: The playground experiment (image from [13])

FIGURE 6 . 9 :

 69 FIGURE 6.9: TD learning prediction error reward and dopamine emission. (image from [274])

FIGURE 6 . 10 :

 610 FIGURE 6.10: Stages of the theory of flow

FIGURE 6 . 11 :

 611 FIGURE 6.11: Prediction error of 4 different activities, and time dedicated to each of them by following the progress (image from [286])

2 . 3 .

 23 The robot takes an action and arrives in a new state (and potentially a new region) From this state, the robot gets sensory (or sensorimotor) data and learning signals 4. This data is used to update the learner 5. The update leads to a new local progress estimation in the region 6. Based on progress estimation and environment constraints, the robot computes a new policy 7. The robot follows this policy to select the next action to take.

FIGURE 7 . 1 :

 71 FIGURE 7.1: General architecture of the IAC algorithm for saliency learning.

FeatureFIGURE 7 . 2 :

 72 FIGURE 7.2: Integration of the saliency learning module in the IAC framework

FIGURE 7 . 3 :

 73 FIGURE 7.3: Regions definition in the pan-tilt space

FIGURE 7 . 4 :

 74 FIGURE 7.4: Regions definition for a mobile robot. On the left, representation of the position of the robot by (x, y, θ). On the right, subdivision of each space for an experiment in a room. Each arrow represents a position of the robot trajectory.

FIGURE 7 . 5 :

 75 FIGURE 7.5: Forward evaluation of the learning progress

FIGURE 7 . 6 :

 76 FIGURE 7.6: Backward evaluation of the learning progress in the region R i

Figure 7 .

 7 7 presents an example of the next position selection given a current position.

FIGURE 7 . 7 :

 77 FIGURE 7.7: Next position selection example with a greedy policy

Figure 7 .

 7 Figure 7.8 presents the general architecture of RL-IAC. As compared with Figure 7.1, representing the general mechanisms of IAC, the only differences are about the region module and the action policy modules. Provided as an input for the meta-learner in the case of IAC, the regions here also help in the action policy process. In addition, they are updated by the robot displacement by an incremental procedure.First, in the region module, we now rely on a graph describing the connectivity between regions and encoding the average time required for displacements between regions. Given this representation, all displacements are not possible from a given region. To reach an arbitrary position from another position, a sequence of displacements may be required. In addition, the distance between regions is integrated in the action policy module given this graph representation. In mobile robotics, this kind of representation is called a navigation graph.Second, the action policy module is a more complex one (called the reinforcementlearning module), that integrates the navigation graph and the learning progress to calculate a displacement policy that optimizes the amount of collected learning progress. To this end, the environment is modeled in a reinforcement learning context, where learning progress is a reward to be collected, and the navigation graph models the states and actions of the environment. The RL module is re-trained at each step to provide the robot with a displacement policy.Aside from these two aspects, the learner, the region space definitions and the meta-learner work exactly the same way as in IAC. Please refer to the corresponding paragraphs of Section 7.3 for more details.

FIGURE 7 . 8 :

 78 FIGURE 7.8: General architecture of RL-IAC

FIGURE 7 . 9 :

 79 FIGURE 7.9: Example of arbitrarily defined regions (on the left) and associated navigation graph (on the right)

FIGURE 7 . 10 :

 710 FIGURE 7.10: Example of regions and navigation graph after exploration.

FIGURE 8 . 2 :

 82 FIGURE 8.2: The 16 regions of the ENSTA laboratory sequence, and sample observations for a few regions

FIGURE 8 . 3 :

 83 FIGURE 8.3: The 5 regions cut along the temporal axis of the ta-ble_small_2 sequence

FIGURE 8 . 10 :

 810 FIGURE 8.10: Navigation graph of the ENSTA building

FIGURE 8 . 13 :FIGURE 8 . 14 :

 813814 FIGURE 8.13: Influence of the discount factor γ in the action selection process

Titre:

  Apprentissage incrémental de la saillance visuelle par des mécanismes de motivation intrinsèque Mots clefs : Saillance visuelle, robotique mobile, robotique cognitive, bio-inspiration, motivation intrinsèque, localisation d'objets

TABLE 2 .

 2

1: Dataset features summary

is generally Unreliable data filtering Wall and floor removal Blobs segmentation Blobs filtering Mask construction

	Parameter	Numerical value	processing step
	Max dist. to kinect	4 m	Unreliable data filtering
		5 cm	Wall and floor removal
	τ	0.95	Wall and floor removal
	δ Max object diag. size	0.05 180 cm	Too far Wall and floor removal Out of FoV Blobs filtering
	Min object diag. size	10 cm	Reflective	Blobs filtering
	Blobs too high thresh.	150 cm	surface	Blobs filtering
	Plane artifact thresh.	5 cm	No normal Blobs filtering
			estimate
				WALL1
				WALL2
				FLOOR
	Depth FoV border
			Too high
			Too big
			Clipped blobs
			Close to FoV
	Wall artifact	Too small
			Unavailable
			Undetermined
			Salient
			Not salient
	FIGURE 4.8: Main steps of the 3D segmentation

TABLE 4 .

 4 1: Main parameters of the depth segmentation algorithm

TABLE 4 .

 4

	Parameter	Numerical value	Comment
	number of trees N	50	proportional to inference time
	number of trees updated K	4	proportional to training time
	number of features by tree	30	proportional to training time
	max depth of the tree	10	proportional to inference time
	max number of samples in memory	100 000	proportional to training time
	max number of samples at each observation	500
	Fraction of the dataset used in each tree	70%	commonly used in random forests

T L is the set of training samples that have reached L, and CL(s) is the label associated with training sample s. A classical way to train a decision tree, is to grow from an iterative process where nodes are splitted based on their purity Ginny 2: Main parameters for saliency learning

  The classifier is able find

	Measure (in %)	BioVision	RGB-D scenes	ENSTA
	available data	0.52	46.2	45.5
	salient pixels	17.4	7.1	4.1
	relative precision	49.2	96.1	97.2
	relative recall	83.3	78.2	75.8
	overall precision	49.2	96.1	97.2
	overall recall	2.53	62.4	58.8
	object recognition accuracy	94.2%	n/a	n/a

TABLE 5 .

 5 1: Features and performance measures of the segmentation process on the three datasets those properties and is therefore able, to a certain extent, to find salient objects that were not detected by the object detector. Second, the datasets are such that the same objects are visible at different points of view. This way, if the object detector fails at identifying an object for a given frame, this object may be detected under other points of view. The classifier then extrapolates those points of view and is able to retrieve undetected salient objects. The segmentation samples of this section were obtained with the deep features, by learning a model withe the entire available training dataset.

Table 5 .

 5 [START_REF] Sak | Fast and accurate recurrent neural network acoustic models for speech recognition[END_REF] provides the numerical results associated with our experiments in that regard. We first tested each of the three features on a CPU.

	Features	Device	Average time
	Make3D	CPU i3-3240	120 ms
	Itti and Koch	CPU i3-3240	60 ms
	Deep features	CPU i3-3240	3 300 ms
	Deep features	GPU Quadro K600	240 ms
	Deep features	GPU GTX Titan X	26 ms
	TABLE 5.3: Processing time of the feature extraction on different de-
		vices.

  is among the most accurate RGB saliency methods according to the MIT

		1			Features evaluation -BioVision
	positive rate	0.4 0.6 0.8								ISL -Make3D ISL -Itti and Koch ISL -Deep features CAM
	True	0.2
		0	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
							False positive rate
		0.9				Features evaluation -ENSTA
		0.8
	positive rate	0.4 0.6								ISL-Deep features ISL -Itti and Koch ISL -Make3D CAM
	True	0.2
		0	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
							False positive rate
		1			Features evaluation -RGB-D scenes
	positive rate	0.4 0.6 0.8								ISL -Deep features ISL -Itti and Koch ISL -Make3D CAM
	True
		0.2
		0	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
							False positive rate
		FIGURE 5.8: ROC curves comparing the different features perfor-
						mance on the three datasets

IAC exploration -Time spent in regions

  .4.

	rate	0.6 0.8 1		Per region error rate evolution -table_small_2	R1 R2 R3 R4 R5
	Error	0.4
		0.2
		0	0	50	nb. of observations 100 150 200 250 300 350 400 450 500 550
		FIGURE 8.4: Error rate in each of the five regions of the table_small_2
					sequence
		75 100	Random exploration -Time spent in regions	R1 R2 R3 R4 R5
	time	50
	%
		25
		0
			0	50	nb. of observations 100 150 200 250 300 350 400 450 500 550
		75 100		R1 R2 R3 R4 R5
	time	50
	%
		25
		0
			0	50	nb. of observations 100 150 200 250 300 350 400 450 500 550
		FIGURE 8.5: Time spent in each region for random exploration and
					IAC-based exploration

TABLE 8 .

 8 1: Processing time for the main steps of RL-IAC

		Min time	Max time
	Meta-learner update	100 ms	300 ms
	Q-learning training	250 ms	300 ms
	Learner update	23 ms	13.5 s
	Robot displacement	0 ms	22 s

Exploration strategies -Short corridor

  FIGURE 8.15: Error rate evolution for several exploration strategies procedure. More precisely, we replaced the backward learning progress LP back

									i	(t)
				LP back i	(t) =	2 π	|atan(-β i (t))|	(8.4)
		1		Exploration strategies -ENSTA
									Schmidhuber
									Uniform
		0.8							Random act. RL-IAC
	Error rate	0.6
		0.4
		0.2	5000	10000	15000	20000	25000	30000	35000
					time (in s)
		1	Exploration strategies -Long corridor
									Schmidhuber
									Uniform
		0.8							Random act. RL-IAC
	Error rate	0.4 0.6
		0.2
			5000	10000		15000	20000	25000
					time (in s)
		1
									Schmidhuber
									Uniform
		0.8							Random act. RL-IAC
	Error rate	0.4 0.6
		0.2
			2500	5000	7500	10000 12500 15000 17500 20000
					time (in s)

Comprenez le wagon bien entendu

Le labo du fond, bien entendu v

http://www.darpa.mil/program/darpa-robotics-challenge

http://image-net.org/challenges/LSVRC

https://smartek.vision/products-services/cameras/29x29-usb3-vision

www.tokinalens.com/tokina/products

Flir PTU E-46, http://www.flir.com/mcs/view/?id=63554

http://www.baumer.com/us-en/products/identification-image-processing/industrialcameras/

[START_REF] Amodei | Concrete problems in ai safety[END_REF] http://www.optotune.com/technology/focus-tunable-

lenses 6 https://www.asus.com/3D-Sensor/

Xtion_PRO_LIVE/[START_REF] Bjorkman | Combination of foveal and peripheral vision for object recognition and pose estimation[END_REF] Pan: ServoCity DDP125. Tilt: ServoCity DDP500

http://www.mobilerobots.com/ResearchRobots/PioneerP3DX.aspx

http://msdn.microsoft.com/en-us/library/jj131033.aspx

https://www.hokuyo-aut.jp/02sensor/07scanner/utm_30lx.html

Histograms of oriented gradients

Discrete Fourier Transform

Discrete Cosine Transform

Simultaneous Localization And Mapping

https://www.youtube.com/edit?o=U&video_id=jNovc5jIIhw

https://www.youtube.com/watch?v=bCdsGLT2UQw

Recall that T P R = tp tp+f n and F P R = f p f p+tn with tp the number of true positive pixels, tn the true negative, and f n the false negatives.

For more simplicity, we consider the meta-learner and the knowledge gain assessor as a unique module, both monitoring error and deriving progress.

Recall that each pixel of S(t) is a probability of being salient. We binarize these probabilities by a threshold at 0.5

We use the F1 score as our error metrics, because "not salient" pixels are representing more than 90% of the samples, making accuracy inappropriate for error estimation.

A function f : X → R is a submodular function if the following property is verified: ∀Y, Z ⊂ X, f (Y ∪ Z) + f (Y ∩ Z) ≤ f (Y ) + f (Z)

Least Square Policy Iteration

This distance is obtained by cumulating the edge weights visited by the robot during the simulation.

https://www.youtube.com/edit?o=U&video_id=w-nGugA5SS4

https://www.youtube.com/edit?o=U&video_id=BZgcF-iIBDY

When no positive samples are observed in a region, the F1 score is either undefined or equals 0 (So that error equals 1). If undefined, we force the F1 score to be 0

EdgeBoxes, 44, 71 ENSTA dataset, 19, 76 epoch, 47 estimation set, 150 expected outcome, 118 exploration problem, 104 exteroceptive sensors, 102 extrinsic motivation, 113 eye saccades, 14 F1 score, 132 false positive rate, 87 familiarity, 115 features, 53, 68 fine-tuning, 48, 59 fixation, 28 forward evaluation, 131 foveal camera, 13, 54 foveated systems, 38 gaze control, 28 goal babbling, 131 graphical models, 31 ground truth label set, 150 IAC, 112 image estimation set, 132, 134 image feature set, 134 image label set, 132, 134 Image segmentation, 40 incremental learning, 46 inductive transfer, 48 information gain, 115 information theoretic models, 31 inhibition of return, 36 input signals, 117 Intelligent Adaptive Curiosity, 10, 112, 123 interest, 115 intersection over union, 44, 92 intrinsic motivation, 113 intrinsic reward, 109 knowledge gain assessor, 117 knowledge transfer, 48, 52 labels, 53 learner, 117, 124 learning progress, 116 learning rate, 109 learning signal, 53, 117, 125 lifelong learning, 112 localization, 100 long corridor, 159, 160 main visual stream, 53

partially observable decision process, 104 pattern classification models, 31 peripheral camera, 13, 54 planning, 100 playground experiment, 112 policy, 103, 104, 125 POMDP, 104 potential field, 104 prediction error, 115, 117, 119 progress, 115 proprioceptive sensors, 102 proto-objects, 39 regional estimation set, 134, 150, 170 regional feature set, 134 regional label set, 134, 150 regions, 118, 124 reinforcement learning, 47, 105, 107 Reinforcement Learning-Intelligent Adaptive Curiosity, 10, 123, 125, 136, 174 reward, 107, 114 RGB-D scenes dataset, 21, 76 RL-IAC, 124 saccade, 28 saliency, 115 saliency consistency score, 72 saliency map, 5, 7, 28 salient, 5 sample, 67 scanpath, 28 SegBoxes, 66 segmentation masks, 55 Selective Search, 44 self-supervised learning, 46, 47 semantic segmentation, 40 semi-supervised, 47 short corridor, 159, 160 Simultaneous Localization And Mapping, 17, 102 SLAM, 102 space of problems, 130 spectral models, 31 strategic student problem, 137 supervision signal, 125 surprise, 115 theory of flow, 114 top-down, 28 top-down modulation, 33 topological maps, 100 topometric maps, 101 Transfer learning, 48 traveling salesman problem, 106 true positive rate, 87 uncertainty, 115 unsupervised learning, 47 view planning problems, 106 visual attention, 27, 28 visual saliency, 28 weakly supervised learning, 47 winner-takes-all, 36

Acknowledgements

cost and do not take into account the time required to take this action in real life.

Once selected, we just consider that the robot has reached this position and can start processing the associated observation.

BioVision

The first evaluation setup uses the BioVision platform. Considering the dataset described in Section 2.3.1, we only work with one of the recorded sequences. This sequence contains both RGB contextual and foveal images to estimate saliency and update the model. In addition, we also use for the experiments the pan-tilt positions associated with each observation. Regions are defined before the experiment by dividing the reachable pan-tilt positions into 4 pan intervals and 4 tilt intervals (for a total of 16 regions). The size of each region is defined so that regions are not equally interesting. Some of them only contain white wall, without any salient object, while other are mainly composed with them. Figure 8.1 illustrate a sample of foveal images reachable in each region.

For the action selection process, we attach to each region the list of observations falling within the region's boundaries. When the robot selects a random action within this region, one of these frames is randomly selected.

ENSTA

To exploit the ENSTA dataset described in Section 2.3.2 in this context, we consider a sequence recorded in a single room of the building. The room considered is the laboratory, represented in Figure 8.2.

During the sequences recorded in the ENSTA building, the robot was able to estimate its position in the environment from a SLAM algorithm. We then consider for each recorded observation the associated position and orientation of the robot.

No objects experiment

New objects experiment

Area names

No objects

Nb. of observations

New objects

Nb. of observations Therefore, when displacement time is not represented, RL-IAC is not doing much better than random exploration. Lastly, the difference in the performance of Random pos. and Random act. mainly comes from the fact that Random pos. makes long displacements across the building without learning in between, thus having a shape similar to IAC. However, Random pos. enables a better sampling of the overall environment, as Random act. only chooses a displacement among the immediate neighbor regions, thus explaining the better performance when considering the number of observations rather than displacement time.

Discount factor

The second parameter of our discussion is the discount factor used in the reinforcement learning process. Recall that the goal of reinforcement learning algorithms is to maximize the amount of cumulated reward R such that

r being the immediate reward received at a certain time, and γ being the discount factor. γ is a value between 0 and 1, and represents how much the long term reward

Versus other kinds of intrinsic motivation

So far, we only considered learning progress as our intrinsic motivation factor. The theoretical reasons for such a choice have been widely discussed in the literature [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF], [START_REF] Lopes | The strategic student approach for life-long exploration and learning[END_REF], but learning progress estimation suffers from several biases in practice. First, a sufficient number of samples is required to make the evaluation of the error statistically significant. Second, at time t, the progress is estimated based on the evolution of the error over the last N samples. The progress estimated this way then represents the progress at time t -N 2 , so that progress estimation is delayed in time. Therefore, the larger N is, the more accurate, but also delayed in time the progress estimation is. On the other hand, using learning progress makes the robot focus on areas where learning is actually possible and avoids loosing time in unlearnable or trivial ones. This makes learning progress even more efficient when learnable areas represent a small portion of the environment. As a result, a bigger gap should be observed between learning progress and other strategies in environments that are essentially unlearnable.

We are then interested in comparing exploration based on learning progress with other sources of intrinsic motivation. We then propose four factors, namely Error, Confidence and Uncertainty) to replace the learning progress in the RL-IAC Chapter 9

Discussions and perspectives

In this works we have presented several contributions, both in the field of visual attention and intrinsically motivated exploration. We summarize in this concluding chapter the proposed approach and highlight the main contributions of this work. We then discuss the strength and limitations of our system under a theoretical and experimental points of view. Lastly, we open the discussion by suggesting future work directions and possible applications of our method.

Contributions of this work

Incremental learning of visual saliency

Summary

The first part of the manuscript was dedicated to the description and evaluation of a method for learning a model of visual saliency directly on a robot. In our experiments, the saliency is specialized in the detection of objects in indoor environments, but could be generalized to other applications.

To do so, we have used a modular block architecture of three main components. First, a feature extractor able to encode the characteristics of an image at the pixel level. Second, a self-annotation process able to provide a partial but accurate state of the saliency on the image. Lastly, a classifier combining features and partial saliency annotation to learn a model and generalize over the partial signal.

Our system then works in two modes. On the one hand, a learning mode, integrating on the fly the visual inputs perceived by the robot into a model of visual saliency. On the other hand, an inference mode, using the learned model to generate saliency maps.

Contributions

Unlike most saliency methods proposed by the state-of-the-art, our approach is incremental. This means that the current available model of saliency can be learned directly on the robot, within its operating environment. The advantages of such approach are threefold: first, it does not require a long dataset preparation and learning period. Unlike saliency approaches based on machine learning techniques, ours does not require any prior model before the experiment, and gets better as exploration goes. Second, the produced saliency maps are adapted to the environment and the task the robot should accomplish (this is a top-down kind of saliency). Lastly, the saliency model remains flexible to any change in the environment by simply updating the model continuously.

Unlike object detectors based on geometrical segmentation, our saliency model is able to provide a probability of a pixel to belong to an object on the whole image.