Loïs Vanhée
email: lois.vanhee@unicaen.fr.

Laurent Jeanpierre

Abdel-Illah Mouaddib

Augmenting Markov Decision Processes with Advising

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Introduction

Markov Decision Processes (MDPs) are one of the most classic model for planning under uncertainty. However, MDPs are difficult to deploy for many real-world problems (e.g. robots in public space, military operations, disaster-recovery). These problems introduce irregularities that are difficult to foresee at design time (e.g. pathfinding hardened by wind, water, glass, acid, armed threats). These irregularities cause discrepancies between the planning model and the real-world, thus entailing the generation of irrelevant plans and failures.

Mixed-Initiative Planning Systems (MIPS) aim to overcome these issues by providing means for human operators to improve plan generation through the validation of actions and plans, and by providing information that the system cannot access or infer other-Finally, this paper demonstrates the relevance of Advice-MDPs for solving real-world problems, by deploying them for a professional-class application. Our industry partner seeks to deploy mobile-robots in unexpected and complex environments, for contexts such as supporting the evacuation of civilians in harmful situations, disaster recovery [START_REF] Ramchurn | HAC-ER: A Disaster Response System Based on Human-Agent Collectives[END_REF], and security missions (e.g. terrorism, patrol in sensitive sites).

The system, which we call the "Anywhere Deployment" Robot System (ADRS), should be extremely flexible i.e. capable of adapting to unexpected irregularities that can affect robot orientation (e.g. wind, water, sand, acidity, glass, unstable ground, threats), which go beyond the possibilities of classic fully-autonomous systems. Furthermore, operator availability is highly limited and unpredictable: while operators can generally watch and support the system for a few minutes after deployment, they have little time for actively training it and, suddenly, they may become overloaded due to arising threats. Illustrations of the implementation of the ADRS are presented in Figure 1 and Figure 2.

Related Research

Mixed-Initiative Planning Systems (MIPSs): (Zilberstein 2015) define MIPSs as automated systems that collaboratively build plans with their operators. MIPS are acknowledged to be particularly useful when the planning model of the automated system is partial and when human judgment is needed to evaluate the plans generated based on the model (Zilberstein 2015).

Classic approaches for implementing MIPSs consists of an iterative loop where the system generates a plan and operators validate or refuse it, with a mechanism for justifying the rejection. If accepted, the plan is executed by the system. If refused, the system generates another plan, taking into consideration the reasons for the rejection of the first plan (e.g. the MAPGEN system, for planning Mars rover activities [START_REF] Bresina | Mixed-initiative activity planning for Mars rovers[END_REF]). The design of these systems is generally coupled with the design of interfaces for enabling operators to evaluate plans and argue on their rejection.

Advanced (MDP-based) MIPS approaches aim to reduce OW costs by reasoning about the operators' workload and capabilities and cost for requesting support from them. The HHP-MDP, HOP-POMDP, and OPOMDP formalisms integrate operators as a manageable (costly) resource of the planning domain, which can be used for acquiring observations and setting subgoals (Armstrong-Crews and Veloso 2007; [START_REF] Côté | Humans-Robots Sliding Collaboration Control in Complex Environments with Adjustable Autonomy[END_REF][START_REF] Rosenthal | Modeling humans as observation providers using pomdps[END_REF]. However, these approaches require from the system to be capable of evaluating the relevance of asking operator support, something which can be difficult to achieve. MIPSs suffer from the limitations 1) and 2) presented in Section Introduction: OW costs are proportional to system use, as plans must be validated each time they are issued. Second, little OW flexibility is possible, as plans must be validated after goals are issued and before plans are executed. Operator support cannot be done a priori and delayed support generally entails neglect costs or even system failures [START_REF] Crandall | Validating human-robot interaction schemes in multitasking environments[END_REF].

Advice-Sensitive Reinforcement Learning (ASRL): Advising has been applied for speeding up the learning process of Reinforcement Learning [START_REF] Abel | Agent-Agnostic Humanin-the-Loop Reinforcement Learning[END_REF]Ng, Harada, and Russell 1999;[START_REF] Torrey | Using advice to transfer knowledge acquired in one reinforcement learning task to another[END_REF]. In ASRL, operators can advise learning agents by providing feedback on the desirability of (intended) agent actions, or by relating the task to similar ones.

ASRL faces the three limitations presented in Section Introduction (high OW costs, low OW management flexibility, expensive training phase). The training phase is particularly expensive in terms of OW costs, as operators are constantly involved during this period. Furthermore, reinforcement learning may be infeasible due to real-world constraints (e.g. the delay for being rewarded by a human is incompatible with a robot affected by world-dynamics such as sliding on ice). In addition, this form of advising only tweaks reward and transition functions, disregarding finer-grained advising that can introduce a richer semantic and behavioral indications for the system (e.g. a "bystanders may be encountered there" advice).

Background on Markov Decision Processes

Classic Markov Decision Processes: Markov Decision Processes (MDPs) are stochastic processes that are (partially) controlled by an agent [START_REF] Sigaud | Markov Decision Processes in Artificial Intelligence[END_REF]. Formally, a MDP is represented by a tuple (S, A, T, R), where S is a set of states; A is a set of actions; T : S × A × S → [0, 1] is a transition function, where T (s, a, s ) is the probability of reaching s when playing a from s; and R : S → R is the reward function, where R(s) is the expected reward for reaching s.

Optimizing MDPs within a bounded horizon h ∈ N consists in finding a policy that maximizes the expected reward in using h actions. A policy π : S → A defines an action to be played for each state and Π is the set of all policies. The value of a policy π within a horizon h is V π h : S → R, where:

V π k (s) = R(s) + s ∈S T (s, π(s), s ).V π k-1 (s ) if k > 0, else V π 0 (s) = R(s).
The set of optimal policies for the horizon h, represented by Π * h ⊆ Π is the set of policies π * ∈ Π * h such that for all s ∈ S, V π * h (s) = max π∈Π V π h (s). Multi-Objective MDPs: In Multi-Objective MDPs (MOMDPs) (Roijers, Vamplew, and Whiteson 2013), the agent controls a process that is related to not one, but a set of reward criteria (e.g. money, satisfaction, environment quality). The reward function of MOMDPs is represented by R : S → R k . Optimizing MOMDPs is influenced by this multiplicity of criteria: the set of optimal policies is a Pareto-front of policies along R k (e.g. two policies, one generating less money, but offering higher satisfaction, belong to the same Pareto-front). Often, MOMDPs are turned into MDPs using a formula for linearizing the MOMDP rewards (e.g. R(s) = 0.5 × R mon (s) + 0.7 × R sat (s) + 2.3 × R env (s)). However, doing so reduces the capability for expressing more elaborated reward profiles and constraints (e.g. the value of environment is minimal, as long as ten points are scored).

Formalizing Advice-MDPs

Advice-MDPs aim to enable operators to augment the planning model with advice, towards fitting the irregularities of the environment (like ASRL) with prescriptive a priori advising (like MIPSs). This way, Advice-MDPs cut down the OW costs of MIPS, by requiring operators to augment the planning model for including real-world irregularities once for all instead of once per use. Advice-MDPs offer higher OW management flexibility, by enabling advising a priori (once the environment is known, without needing goals to be set) instead of last-minute advising for MIPSs; Advice-MDPs require no training phase and advising can be achieved without executing plans, unlike ASRL approaches. Advice-MDPs enable for a large variety of advice types, which can introduce a helpful semantic context for advanced decision-making, unlike ASRL approaches that only enable to tweak mono-dimensional rewards.

Advice-MDPs expand classic MDPs, replacing reward function with advising. This paper considers the following five advice types: {G, S f , S d , S u , C π }, where G ⊆ S are goal states, to be reached by the system, S f ⊆ S are forbidden states, S d ⊆ S are desired states, S u ⊆ S are undesired states, and C π ∈ f : S → 2 A are forbidden actions to be avoided in certain states. In our ADRS application, S d are safe and efficient zones, S u are slow or inconvenient zones (e.g. sand, rock), and S f are dangerous zones that should be avoided, unless inevitable (e.g. very poor floor, very close to enemy location), C π are contextually forbidden actions (e.g. activating a visible sensor). The relative importance given to the various advice types follows a lexicographic order: first, match C π ; then, best satisfy the tradeoff G, S f ; finally, best satisfy the trade-off S d , S u . Formally, policies matching C π are represented by:

Π C π = Π \ {π|∃s ∈ S, π(s) ∈ C π (s)} Policies that best satisfy the tradeoff G, S f belong to the Pareto-front of policies evaluated by V G,S f ,π = (V G,π , -V S f ,π ), where: V S ,π (s) = N vis (S |s, π), for which N vis (S |s, π)
is the expected number of times S is expected to be visited from s following π. Policies that best satisfy the tradeoff S d , S u belong to the Pareto-front of policies evaluated by V S d ,Su,π = (V S d ,π , -V Su,π ). Note that this representation for advising is selected for its simplicity, genericity, and ease to connect to user interfaces. However, more advice types can be integrated in the model.

Efficiently Solving Advice-MDPs

We resolve Advice-MDPs by transforming them into MOMDPs. As a transformation, actions belonging to C π are removed from the base T , and each advicetype is mapped to a criterion, which is rewarded when the system reaches an accordingly-advised state (e.g.

R S d (s) is 1 (else 0) if s ∈ S d ).
Rewards are positive for desirable or goal states and negative otherwise. Since rewards are summed, the value of criterion c when following π is the number of expected visitations N π vis,c (e.g. "V(s)=(2,0,5,-7)" when following π for h steps means: "from s, following π is expected to lead to the reach of 2 goals, 0 forbidden, 5 desirable, 7 undesirable states). Optimal solutions for this MOMDP are Pareto-optimal tradeoffs that maximize the number of visited goals and desired states, and minimize the number of forbidden and undesired states.

This transformation into MOMDPs (vs. a monodimentional MDP) offers multiple benefits, by avoiding boiling down the whole reward function in a single scale. First, MOMDPs enable for non-linear combination of factors (e.g. losing 3 wheels is much worse Algorithm 1: Fast Advice-MDP policy computer

Input : S, A, T , G, S f , S d , Su, C π , H Output : MDP Value Function V T ← T \ {(s, a, s )|∀s, s , a ∈ C π (s)} V π,0 ← 0 ∀s ∈ S, R (s) ←      (1, 0, 0, 0) if s ∈ G (0, -1, 0, 0) if s ∈ S f (0, 0, 1, 0) if s ∈ S d (0, 0, 0, -1) if s ∈ Su for t = 0 . . . H -1 do foreach s ∈ S do foreach criterion c ∈ {G, S f , S d , Su} do V π * c ,t+1 c (s) ← Rc(s)+ maxa∈A s ∈S T (s, a, s ).V π * c ,t c (s ) ideal c ← maxa Rc(s) + s ∈S T (s, a, s ).V π,t c (s ) antiIdeal c ← mina Rc(s) + s ∈S T (s, a, s ).V π,t c (s ) foreach a ∈ A do Rgt t c (s, a, π) ← ||[Rc(s)+ s ∈S T (s,a,s ).V π,t c (s )]-V π * c ,t+1 c (s)|| ||ideal t,π c (s)-antiIdeal t,π c (s)|| optA ← argmin a lexdiff (C π , Rgt G (s, a, π) + Rgt S f (s, a, π), Rgt S d (s, a, π) + Rgt Su (s, a, π)) V π,t+1 (s) ← R(s) + s ∈S T (s, optA, s ).V π,t (s ) return V
than losing one; crossing the first undesirable is very bad, but if inevitable, a rational balance should be found). Such combination can be achieved in monodimensional MDPs, by adding variables to the statespace, thus causing a combinatorial explosion. Else, if mono-dimensional reward functions introduce linearization artifacts, such as "trading three undesirable for one desirable", which are often not suitable for applications. Second, the MOMDP representation is simpler for operators, which lowers risks of errors. MOMDPs introduce a semantic context for applying advising and for easing system management (e.g. a slider for setting the balance between "following desirable" and "avoiding undesirable", a slider for balancing the importance between achieving goals and the tradeoff desirable/undesirable). This operation is less intuitive with linear MDPs, as it implies to reason about fiddling reward functions. Third, MOMDPs simplifies the introduction of a wider range of advice-types (e.g. a "can be pushed", "can annoy bystanders"), without having to re-design the reward function as a whole.

Our fast approximate algorithm for solving Advice-MDPs (Algorithm 1) is inspired by Algorithm 10.1 from (Sigaud and Buffet 2010, p.325). Exact MOMDPresolution algorithms (Wakuta 2001) were left out due to their exponential complexity and our need for responsiveness. At each iteration, our algorithm backpropagates for each state the set of most promising expected multidimensional value for each action (and ideal and anti-ideal values for computing the relative value of each state, see next paragraph). Then, a comparator function selects the action leading to the most promising reward tuple, given the immediate choice (e.g. "(2,0,5,-7)" is better than "(2,-1,5,0)").

The MOMDP comparator is based on the Ordered Weighted Regret (OWR) function (Roijers, Vamplew, and Whiteson 2013), based on the LexDiff order [START_REF] Sigaud | Markov Decision Processes in Artificial Intelligence[END_REF], which enables maintaining the priority order of advice types, while introducing properties such as "undesirable states are to be avoided, but if crossing them is inevitable, then they should be best balanced with crossing desirable states". The OWR "calculates the regret of each [criterion] with respect to an estimated ideal reference point, sorts these into descending order, and calculates a weighted sum in which the weights are also in descending order" (Roijers, Vamplew, and Whiteson 2013). The regret for criterion c when playing action a in state s given a policy π for a horizon h is:

Rgt h c (s, a, π) = ||V π,h c (s) -V π * c ,h c (s)|| ||ideal h,π c (s) -antiIdeal h,π c (s)|| where V π,t c
is expected total reward for c from following π for t steps, π * c is the "pure" policy that singlemindedly maximizes c, while discarding other criteria, ideal t,π(s) c (respectively antiIdeal t,π c (s)) is the highest (respectively lowest) expected reward for c that can be acquired from s for all actions and then following π. ||ideal t,π c (s) -antiIdeal t,π c (s)|| is the Chebyshev norm, which makes the regret relative to the relative grasp that the system has on c. If ideal t,π c (s) = antiIdeal t,π c (s), then Rgt c (s, a, π) = 0 (no regret when no grasp). The LexDiff strictly orders sets of criteria (in our case, {C π } > {G, S f } > {S d , S u }), lower-rank criteria are only considered when higher-rank criteria are equally satisfied. Altogether, LexDiff minimizes the regret of the most important advice types first and then the regret of lower-rank advice types. Note that this comparator function can be changed for altering the reaction of the system with regards to advising (e.g. allowing to trade the crossing of one forbidden state for avoiding many undesirable states).

In terms of solution quality, in our experiments, our algorithm always finds optimal MDP policies when undesirable and forbidden states can be avoided. When being forced to cross undesirable and forbidden states, our algorithm generates very satisfactory trajectories (avoiding as much as possible forbidden states, then a relatively conservative balance between avoiding undesirable states and reaching desirable states).

Professional-Grade Application

Application Context

Our industry partners aim to deploy highly flexible mobile robots for supporting operators for contexts such as disaster-recovery [START_REF] Ramchurn | HAC-ER: A Disaster Response System Based on Human-Agent Collectives[END_REF], scouting Robots are given missions such as live observations (camera feed, establishing maps, monitoring threats) and situational actions (e.g. open a door, disarm traps). The robot we work with are NERVA robots illustrated in Figure 1, which are actually already being deployed by our partners, exclusively controlled through teleoperation. NERVAs are very robust (can be thrown over obstacles) and fast (up to 15km/h). They are equipped with four cameras and a wide array of specific sensors can be plugged in for capturing situational measurements (e.g. sound, gas, explosives, radioactivity).

As an application for intersecting both industrial and academic goals, we aim to make the robot capable of reaching as autonomously as possible set of locations that is required for completing sensing tasks. This goal revisits the very classical pathfinding problem, except that the many irregularities of varied, difficult, and unexpected real-world situations are added to the equation. These irregularities introduce two challenges making fully-autonomy currently unfeasible: first, available models fail to incorporate all these irregularities and the design of such an all-encompassing model would be very difficult and likely to be computationally too expensive; second, robot sensory capabilities are sometimes insufficient for recognizing such irregularity (e.g. the robot, only equipped with cameras, faces a puddle in a chemical plant: can the puddle be recognized? Can the puddle be crossed? Is it deep? Is it a chemical that will destroy the robot? If so, how fast?). We called our system the Anywhere Deployment Robot System (ADRS). An overview of the system loop presented in Figure 3. The ADRS generates a map of the surroundings via a Simultaneous Localization and Mapping (SLAM) using a laser. This map is then displayed with a Graphical User Interface (GUI), presented in Figure 2, on a tablet that operators wear on their wrist. This GUI enables operators to advise on goals, desirable, undesirable and forbidden areas, by simply clicking on an "advice-type" button and drawing on the map (points, lines, shapes). Then, the system generates an advice-compliant waypoint-based trajectory that minimizes the total distance crossed for reaching all goals. This trajectory is displayed back to the operator, who decides whether the robot should apply it (a "rush mode" can be turned on for skipping this check). At any time, before and during plan execution, the operator can update the advising, which then updates the waypoint trajectory. Likewise, robot trajectories are updated if the SLAM discovers new obstacles that jeopardize the current plan. Towards achieving high responsiveness and smooth interactions with operators, our fast Advice-MDP resolution algorithm enables completing a whole sense-reason-act loop within two seconds1 , which is satisfactory for operators. Special cases can be improved through engineering (e.g. having dodge reactive behaviors during replanning, so the robots does not become an easy target).

As an overall idea of the system behavior, and thus of the advising semantics, desirable areas (in green) mark zones that are interesting to be crossed by robots (safe ground, concealed from opposing threats), while still generally moving towards the goal. Undesirable areas (in orange), mark zones that are best avoided if possible, but that can be crossed for the sake of reaching the goal, or many more desirable areas (e.g. populated nonhostile areas, unsteady ground that slows the robot, slow acid or irradiated areas that will destroy the robot after the mission). Red areas mark zones that may harm the mission, to be avoided unless they there is no other means for reaching the goal (e.g. risk of being spotted, slippery ground that may cause crashes, unidentified puddle). These categories remain flexible and can change due to situational context (e.g. slow acid areas may be set to forbidden if losing the robot after its mission becomes an issue for follow-up actions).

Technically, the generated map is a 4096 × 4096 pixel map. Each pixel captures a 4cm 2 square. For keeping high responsiveness, this map is abstracted into a 400 × 400-tiles hexagonal grid (any tile containing at least one obstacle pixel is considered as an obstacle tile). This grid is then transformed into a canonical transition function for moving on grids (e.g. playing "East" on state/tile "(4,12)" leads to state/tile "(5,12)"). The optimal policy, π * , provides a pathway, i.e. a sequence of sets of optimal tiles per iteration. As a straight transformation of this pathway creates highly inefficient stepby-step "stop-and-turn" move, a smoothing phase fastens robot moves by greedily searching for the waypoint path with the least number of steps that is totally covered by the optimal pathway. This step is illustrated in Figure 4. Note that, for descriptive simplicity, this implementation relies on a deterministic MDP. However, Advice-MDPs can be applied on non-deterministic MDPs (e.g. uncertainty on the tile to be reached when playing move actions).

Empirical Evaluation

Varying The Degree of Autonomy

We compared Advice-MDPs against Fully-Autonomous Systems (FAS, based on classic advice-less MDPs) and Non-Autonomous systems (i.e. teleoperation), along the following criteria, partly drawn from [START_REF] Hart | Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research[END_REF][START_REF] Steinfeld | Common metrics for human-robot interaction[END_REF]): 1) flexibility (range of situations that can be handled by the system); 2) efficiency (time for completing the mission); 3) Operator Workload (OW) (time required from operators for having the task completed, i.e. for teleoperation and advising); 4) effectiveness (percentage of the mission that is autonomously completed); 5) OW flexibility (flexibility left for the operator to decide when to operate), 6) responsiveness (time from task-allocation to action).

As a setup, we compared the performance of FAS, Advice-MDPs, and teleoperation in two scenarios of differing degrees of flexibility: in the corridor scenario, the robot is tasked to move to the other side of a building (Figure 5); in the hole scenario, a (virtual) hole in the ground is to be avoided, else the robot risks breaking (Figure 6). Each experiment was repeated 20 times.

Each experimenter was trained to teleoperate and advise the robot for at least ten minutes.

Experimental results (Table 1) detail the compromises between efficiency, flexibility, and OW costs. In terms of flexibility, FASs only succeed in the corridor scenario, as they risk getting stuck in the hole scenario; the Advice-MDPs systems handle both the corridor and hole missions; teleoperation handles all situations. Teleoperation remains more flexible than Advice-MDPs, as humans can make situational assumptions that fall out of the planning model (e.g. seeing empty crates as pushable objects rather than impassable obstacles).

In terms of OW costs, FASs are obviously the most effective, followed by Advice-MDPs, which only require one fast intervention for augmenting the planning model, and teleoperation last, which is much more expensive. Note that, once augmented, Advice-MDPs can operate with no subsequent OW costs, unless major changes in the environment (e.g. a zone catches fire).

In terms of efficiency, teleoperation is the most efficient; Advice-MDPs and FASs are equivalent (teleoperation allows for higher speed and avoiding ineffective stop-and-turn actions at waypoints).

In terms of effectiveness, Advice-MDPs achieve a score of 81.3% when augmenting the planning model is needed, which is very reasonable. In terms of OW flexibility, Advice-MDPs offers the highest OW-flexibility as advising can be done a priori, while teleoperation offers no OW-flexibility (must be done last minute, any delay lowers the efficiency of the system). Last, in terms of responsiveness, the software responds within two seconds for FASs and Advice-MDPs. Comparing Against Related Approaches

This section compares Advice-MDPs against the two related approaches: MIPS and ASRL approaches. The experimental results are detailed in Table 1.

For the MIPS approaches, the operator is in charge of validating or rejecting the trajectories generated by the system. When rejecting a trajectory, operators indicate at which point the trajectory is incorrect. Then, the system removes the indicated state, replans and proposes an alternative trajectory.

In terms of performance, Advice-MDPs achieve slightly better efficiency for lower OW costs for the corridor experiment, as operators are asked to perform an unnecessary check for the MIPS (although, the cost is relatively low). For the hole experiment, Advice-MDPs achieve much better performance than MIPS, as rejecting plans until one becomes acceptable incurs low efficiency and high OW costs. Furthermore, after one planning phase, the Advice-MDPs system is ready to be operated without subsequent OW costs, while the whole process is to be repeated for the MIPS approach.

For the ASRL approach, a module has been implemented using the most relevant technique from [START_REF] Abel | Agent-Agnostic Humanin-the-Loop Reinforcement Learning[END_REF] for this context: action pruning. During the training phase, the system asks before playing an action whether this action is the most relevant. If not, the system associates it with a low reward and proceeds with the new best action. Results highlight the that the Advice-MDP approach overcomes the ASRL approach. First, Advice-MDP avoids a very slow and OW-cost expensive learning phase, totally in 98 operator checks for the corridor scenario and 123 operator checks for the hole scenario. Second, the Advice-MDP approach enables for an effective prescription of all undesired states, while the ASRL approach only learns what is related to this specific trajectory. While some knowledge is re-used, the system is to be trained again for moving back to the initial position. Furthermore, during this learning phase, the robot can only rely on brute MDP-based plans, which imply operating on the level of "stop and turn" low-level actions, which dramatically slow down robot operations.

Conclusions and Perspectives

This paper introduces Advice-MDPs, a new formalism for designing systems capable of integrating operator support for augmenting planning models towards efficiently adapting the behavior of the system to realworld unexpected irregularities. Advice-MDPs expand the Markov Decision Process (MDP) formalism with the possibility to advise on the desirability, undesirability, or prohibition of states and actions. Furthermore, this paper introduces a fast algorithm for generating near-optimal advice-compliant policies, making possible to create reactive interaction-loops with operators for smoothing the task of supporting the robot.

We empirically demonstrate the benefits of Advice-MDPs against fully autonomous planning systems (Advice-MDPs trade reasonable operator workload costs for greater system flexibility), fully teleoperated systems (Advice-MDPs can dramatically decrease the operator workload costs), classic mixed-initiative planning systems (Advice-MDPs lower operators workload costs, the invested time from operator is better re-used over multiple planning operations, operators can better manage when to advise the system), and reinforcementlearning based planning systems (Advice-MDPs skip a long learning phase, thus decreasing operator workload costs and making the system much faster to set up). Conceptually, the core novelty and benefits offered by Advice-MDPs compared to classic approaches lies in enabling operators to prescriptively augment the worldmodel. This augmentation can then be used for multiple subsequent planning operations, be given early on (before planning is required), and does not require system training. Though, the benefits of advising depend on external factors, such as the advising quality (e.g. mistakes from operators) and unexpected environmental changes (e.g. new areas becoming forbidden or desirable), which can require additional efforts from operators (e.g. watching the environment and the robot, updating advising) and introduce risks of failure.

We demonstrate the applicability of these theoretical findings, by deploying Advice-MDPs in the design of a professional-grade robotic system, which we called the Anywhere Deployment Robot System (ADRS). The ADRS has been tested by its intended final users (our industry partners), who were highly satisfied by the flexibility features offered by Advice-MDPs and the relief the ADRS brings on the cognitive strain required for operating the system. The ADRS is being integrated within the robotic platform of our industry partners.

Regarding future work, we are expanding Advice-MDPs for integrating more advice types (e.g. a "passable" advice type for handling e.g. box-pushing behavior). Additionally, we are expanding our application to a multi-robot setting, considering new challenging issues in term of multi-agent planning and multi-robot coordination. Furthermore, we are investigating the combination between Advice-MDP and HOP-POMDP for combining the benefits of both approaches [START_REF] Rosenthal | Modeling humans as observation providers using pomdps[END_REF].

Figure 1 :

 1 Figure 1: Two NERVA robots we experimented with, in the realistic arena set by our industry partners.

Figure 2 :

 2 Figure 2: Operator interface: map, video stream. Lines on the map represent operator advising: desired (green), undesired (orange), and forbidden (red) advising. The target shape is the goal of the robot. Numbers are navigation waypoints.

Figure 3 :

 3 Figure 3: System loop: operator and sensor input are merged and displayed by the GUI. When goals are set, this input is turned into Advice-MDPs, given to the planner for generating MDP policies, turned into waypoints, smoothed, and executed by the robot.

Figure 4 :

 4 Figure 4: Optimal pathway towards the goal (blue tiles), considering obstacles (grey tiles). Red lines are the base waypoint path generated from the base MDP policy (many stop and turn operations). Green lines represented the smoothed path (minimizing the amount of waypoint, exploiting diagonals).

Figure 5 :

 5 Figure 5: Advice-MDP waypoint trajectories for the corridor scenario. Lines are former robot paths.

Figure 6 :

 6 Figure 6: Advice-MDP trajectories for the hole scenario. Red marks are "forbidden" advice, drawn by the operator.

Table 1 :

 1 Average autonomy effectiveness, efficiency, and operator workload costs, over 12 repetitions for the corridor and hole scenarios, depending on the system type. Lines marked by an asterisk note the OW cost during a training phase.

	Scenario	System	Autonomy Effectiveness	Efficiency (s)	OW (s)
	Corridor	Advice-MDPs	100%	33.7	0
	Corridor	Teleop	0%	22.3	22.3
	Corridor	Base MDPs	100%	34.5	0
	Corridor	Action Pruning (ASRL)	NA	269.9	348.8 *
	Corridor	Plan Revision (MIPS)	95.1%	35.0	1.7
	Hole	Advice-MDPs	81.3%	43.4	8.1
	Hole	Teleop	0%	27.2	27.2
	Hole	Base MDPs	100%	Falls	0
	Hole	Action Pruning (ASRL)	NA	316.7	472.7 *
	Hole	Plan Revision (MIPS)	54.6%	75.2	34.9

Demonstration videos are available at https://www.youtube.com/playlist?list= PLcULZgrd6hEgBYO6zJ74S7tH3DvX3Sf7g

Acknowledgements

The authors wish to thank Melania Borit for her support improving the text and the reviewers for their constructive feedback.

Funded by the Agence Nationale de la Recherche and the Direction Générale de l'Armement (ANR-14-ASTR-0018). Copyright c 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org).


Antoine Deleforge
email: antoine.deleforge@inria.fr

Alexander Schmidt

Walter Kellermann

Audio-Motor Integration for Robot Audition

Keywords: audio-motor integration, robot audition, sound localization, ego-noise reduction, dictionary learning, single-channel, multi-channel

published or not. The documents may come

INTRODUCTION

The most natural way for humans to communicate is speech. For this reason, building robots which can interact with humans via speech is an important goal in robotics which has received growing research interest over the past 20 years [START_REF] Huang | Building ears for robots: sound localization and separation[END_REF][START_REF] Nakadai | Active audition for humanoid[END_REF][START_REF] Nakadai | Real-time sound source localization and separation for robot audition[END_REF][START_REF] Prasad | Robots that can hear, understand and talk[END_REF][START_REF] Nakadai | Design and implementation of robot audition system HARK-open source software for listening to three simultaneous speakers[END_REF][START_REF] Heinrich W Löllmann | Challenges in acoustic signal enhancement for humanrobot communication[END_REF][START_REF] Argentieri | A survey on sound source localization in robotics: From binaural to array processing methods[END_REF][START_REF] Rascon | Localization of sound sources in robotics: A review[END_REF][START_REF] Heinrich W Löllmann | Microphone array signal processing for robot audition[END_REF]. Examples of desired high-level features for such communicating robots are the ability to look towards the person they interact with [START_REF] Sanchez-Riera | Online multimodal speaker detection for humanoid robots[END_REF], speech recognition in noisy, multi-source and reverberant environments [START_REF] Valin | Robust recognition of simultaneous speech by a mobile robot[END_REF] or speech diarization i.e., the identification of who talks to whom and when [START_REF] Otsuka | A realtime multimodal system for analyzing group meetings by combining face pose tracking and speaker diarization[END_REF]. These high-level abilities can be associated to lower level audio signal processing tasks such as sound source localization and tracking [START_REF] Argentieri | A survey on sound source localization in robotics: From binaural to array processing methods[END_REF][START_REF] Rascon | Localization of sound sources in robotics: A review[END_REF], sound source separation and speech enhancement [START_REF] Gannot | A consolidated perspective on multimicrophone speech enhancement and source separation[END_REF] or dereverberation [START_REF] Naylor | Speech dereverberation[END_REF]. None of these tasks is specific to robot audition and they have been extensively studied over the past decades in contexts as varied as hearing aids, voice-controlled assistants in cars, smartphones or smart homes, audio signal restoration, or live music recording. But what makes robot audition fundamentally different from these fields?

A distinctive feature is that, by definition of a robot, the microphones are mounted on a system equipped with actuators, i.e., motors. These actuators may impact received auditory signals in two different ways: introduces a single-channel sound source localization method based on head movements. Section 2.4 presents a general ego-noise reduction framework exploiting proprioceptors and dictionary learning. Finally, a conclusion and future perspectives on audio-motor integration for robot audition is presented in Section 2.5. An illustration of the different components of a typical auditory-motor system, their associated literature, and the parts addressed in this chapter is showed in Fig. 2.1.

AUDIO-MOTOR INTEGRATION IN PSYCHOPHYSICS AND ROBOTICS

The majority of studies on audio-motor integration, both in psychophysics and robotics, focus on sound source localization. From a psychophysical point of view, it is known since the experiments of Lord Rayleigh in 1907 that humans use binaural cues in order to estimate the direction of a sound source [START_REF] Strutt | On the perception of the direction of sound[END_REF]. Two types of binaural cues seem to play an essential role, namely interaural level differences (ILD) and interaural time differences (ITD), the frequency domain counterpart of the latter being referred to as interaural phase differences (IPD). Both ILD and IPD are known to be subject-dependent and frequency-dependent cues. This is captured by the so-called head related transfer functions (HRTFs) determined by the shape

Audio signal processing

Motor data processing Audio-motor integration [START_REF] Braasch | A binaural model that analyses acoustic spaces and stereophonic reproduction systems by utilizing head rotations[END_REF][START_REF] Bustamante | Towards information-based feedback control for binaural active localization[END_REF][START_REF] Furukawa | Noise correlation matrix estimation for improving sound source localization by multirotor uav[END_REF][START_REF] Ince | Ego noise suppression of a robot using template subtraction[END_REF][START_REF] Ito | Internal noise suppression for speech recognition by small robots[END_REF][START_REF] Kneip | Binaural model for artificial spatial sound localization based on interaural time delays and movements of the interaural axis[END_REF][START_REF] Kreković | Echoslam: Simultaneous localization and mapping with acoustic echoes[END_REF][START_REF] Ma | Exploiting deep neural networks and head movements for robust binaural localization of multiple sources in reverberant environments[END_REF][START_REF] May | Robust localisation of multiple speakers exploiting head movements and multi-conditional training of binaural cues[END_REF][START_REF] Quan V Nguyen | Long-term robot motion planning for active sound source localization with monte carlo tree search[END_REF][START_REF] Portello | Acoustic models and kalman filtering strategies for active binaural sound localization[END_REF][START_REF] Schmidt | Ego-noise reduction using a motor data-guided multichannel dictionary[END_REF][START_REF] Schmidt | A novel egonoise suppression algorithm for acoustic signal enhancement in autonomous systems[END_REF] Active audition [START_REF] Barfuss | An adaptive microphone array topology for target signal extraction with humanoid robots[END_REF][START_REF] Bernard | Phonotaxis behavior in the artificial rat psikharpax[END_REF][START_REF] Ma | A machine-hearing system exploiting head movements for binaural sound localisation in reverberant conditions[END_REF][START_REF] Magassouba | Sound-based control with two microphones[END_REF][START_REF] Nakadai | Robot recognizes three simultaneous speech by active audition[END_REF][START_REF] Tourbabin | Enhanced robot audition by dynamic acoustic sensing in moving humanoids[END_REF]

FIGURE 2.1 Auditory-Motor System Components

Illustration of an auditory-motor system whose goal is to enhance a speech source and to turn the head towards it in the presence of ego-noise.

of the head, pinna and torso. HRTFs filter the sound propagating from a source to eardrums and depend on the source direction. It is known that spatial information provided by interaural-difference cues within a restricted band of frequency is spatially ambiguous, particularly along vertical and front/back axis [START_REF] John | Sound localization by human listeners[END_REF]. This suggests that humans make use of full spectral information for 2D sound source localization [START_REF] Hofman | Spectro-temporal factors in twodimensional human sound localization[END_REF]. This is confirmed by biological models of the auditory system hypothesizing the existence of neurons dedicated to the computation of interaural cues in specific frequency bands [START_REF] Wang | Computational Auditory Scene Analysis: Principles, Algorithms and Applications[END_REF]. A large number of computational models were developed for robust sound localization and tracking based on ITD, ILD, IPD and HRTFs in the context of robot audition, see for example the recent reviews [START_REF] Argentieri | A survey on sound source localization in robotics: From binaural to array processing methods[END_REF][START_REF] Rascon | Localization of sound sources in robotics: A review[END_REF].

While the features mentioned above are generally used assuming static sensors, sound localization features could also be extracted using sensor motions, although this idea has received much less attention. Early psychophysical experiments suggested that head motions are useful for disambiguating potential confusions generated by the human pinna's filter [START_REF] Wallach | The role of head movements and vestibular and visual cues in sound localization[END_REF], notably to estimate the elevation of low frequency sounds [START_REF] Perrett | The effect of head rotations on vertical plane sound localization[END_REF]. Other experiments [START_REF] Willard R Thurlow | Head movements during sound localization[END_REF][START_REF] Frederic | Resolution of front-back ambiguity in spatial hearing by listener and source movement[END_REF] further support the idea that head movements are useful for localization, although less significantly so than ITD and ILD [START_REF] John | Sound localization by human listeners[END_REF]. Interestingly, [START_REF] Kato | The effect of head motion on the accuracy of sound localization[END_REF] suggests that taking into account head motions is useful to improve the impression of 3D sound in virtual reality. They show that when listening through different HRTFs than his/her own, a listener often complains that auditory events are spatially diffuse, and makes incorrect judgements about the source locations. The experimental study of [START_REF] Kato | The effect of head motion on the accuracy of sound localization[END_REF] on human subjects demonstrates that head motion can overcome HRTF mismatches, increasing perceived location accuracy. Despite these psychophysical evidences, very few computational sound localization studies incorporate head motion. In [START_REF] Kneip | Binaural model for artificial spatial sound localization based on interaural time delays and movements of the interaural axis[END_REF], several ITD values obtained from different motor states with a two-microphone device with two degrees of freedom (translation, rotation) are used for 2D sound source direction and distance estimation. The authors of [START_REF] May | Robust localisation of multiple speakers exploiting head movements and multi-conditional training of binaural cues[END_REF][START_REF] Ma | Exploiting deep neural networks and head movements for robust binaural localization of multiple sources in reverberant environments[END_REF] propose to map binaural cues to the azimuths of multiple sound sources using Gaussian mixture models [START_REF] May | Robust localisation of multiple speakers exploiting head movements and multi-conditional training of binaural cues[END_REF] or deep neural networks [START_REF] Ma | Exploiting deep neural networks and head movements for robust binaural localization of multiple sources in reverberant environments[END_REF] and two static head positions are used to resolve front-back ambiguities. In [START_REF] Portello | Acoustic models and kalman filtering strategies for active binaural sound localization[END_REF][START_REF] Bustamante | Towards information-based feedback control for binaural active localization[END_REF][START_REF] Quan V Nguyen | Long-term robot motion planning for active sound source localization with monte carlo tree search[END_REF] mobile robots are used to actively collect several viewpoints of one or several emitting sound sources in order to accurately estimate their azimuths and distances. More broadly, studies tackling the problem of audio-based simultaneous localization and mapping (SLAM) have recently emerged [START_REF] Evers | Acoustic simultaneous localization and mapping (a-slam) of a moving microphone array and its surrounding speakers[END_REF][START_REF] Kreković | Echoslam: Simultaneous localization and mapping with acoustic echoes[END_REF]. Interestingly, [START_REF] Kreković | Echoslam: Simultaneous localization and mapping with acoustic echoes[END_REF] does it with a single microphone at multiple viewpoints thanks to acoustic echoes. Note that the above mentioned studies rely on static localization techniques applied from different viewpoints. In contrast, [START_REF] Braasch | A binaural model that analyses acoustic spaces and stereophonic reproduction systems by utilizing head rotations[END_REF] computes the average binaural cross-correlation of a continuously rotating head to estimate a source's azimuth in an anechoic scenario. To the best of the authors' knowledge, exploiting continuous sensor movement for 2D sound sound direction estimation with a single microphone, as showed in Section 2.3 of this chapter, has not been proposed before.

Complementarily to sound source localization, [START_REF] Nakadai | Robot recognizes three simultaneous speech by active audition[END_REF] introduced the idea of active robot audition (see Fig. 2.1) where the head of a robot is directed towards estimated source locations in order to enhance the signal of interest via, e.g. beamforming. This idea was recently further exploited in [START_REF] Barfuss | An adaptive microphone array topology for target signal extraction with humanoid robots[END_REF][START_REF] Tourbabin | Enhanced robot audition by dynamic acoustic sensing in moving humanoids[END_REF] using a so-called robomorphic array, where microphones are placed on a robot's limbs. In [START_REF] Ma | A machine-hearing system exploiting head movements for binaural sound localisation in reverberant conditions[END_REF], it is showed that turning the head of a binaural robotic system towards an estimated source direction enhances sound localization performance. From a psychophysical viewpoint, it is well known that humans use active audition to improve sound perception notably to address the so-called cocktail party problem [START_REF] Haykin | The cocktail party problem[END_REF]. Other approaches focus on the goal of orienting a system towards a sound source, without necessarily exploiting this for enhancement. This ability is referred to as phonotaxis, and was implemented on a rat-inspired robot equipped with mobile ears in [START_REF] Bernard | Phonotaxis behavior in the artificial rat psikharpax[END_REF]. More recently, sound has been used in feedback-control loops, i.e., audio-servoing. In [START_REF] Magassouba | Sound-based control with two microphones[END_REF] motor commands are used to align perceived ITDs to a target value, resulting in the robot placing itself on a line with constant ITD value.

Another category of methods consists in using the motor capabilities of a robot to help it learn sound source localization techniques [START_REF] Berglund | Sound source localisation through active audition[END_REF][START_REF] Hornstein | Sound localization for humanoid robots-building audio-motor maps based on the hrtf[END_REF][START_REF] Talmon | Supervised source localization using diffusion kernels[END_REF][START_REF] Deleforge | The cocktail party robot: Sound source separation and localisation with an active binaural head[END_REF][START_REF] Bernard | Sensorimotor learning of sound localization from an auditory evoked behavior[END_REF][START_REF] Deleforge | Acoustic space learning for sound-source separation and localization on binaural manifolds[END_REF], much like humans and mammals do during early stages of development. Several psychophysical studies suggest that the link between auditory features and source locations would not be hard-coded in the brain but rather learned [START_REF] Wright | A review of learning with normal and altered sound-localization cues in human adults[END_REF][START_REF] Aytekin | A sensorimotor approach to sound localization[END_REF] or re-learned [START_REF] Paul M Hofman | Relearning sound localization with new ears[END_REF] from experience. One example of such learning processes is the sensori-motor theory of perception, originally laid by Poincaré [START_REF] Poincaré | The foundations of science; Science and hypothesis, the value of science, science and method[END_REF] and more recently investigated in [START_REF] J Kevin | A sensorimotor account of vision and visual consciousness[END_REF]. This theory suggests that experiencing the sensory consequences of voluntary motor actions is necessary for an organism to learn the perception of space. In [START_REF] Aytekin | A sensorimotor approach to sound localization[END_REF] a psychophysical sensorimotor model of sound source localization using head-related transfer function (HRTF) datasets of bats and humans was proposed. Similar ideas were implemented on robots using reinforcement learning [START_REF] Berglund | Sound source localisation through active audition[END_REF], linear regression [START_REF] Hornstein | Sound localization for humanoid robots-building audio-motor maps based on the hrtf[END_REF], locally-linear regression [START_REF] Deleforge | Acoustic space learning for sound-source separation and localization on binaural manifolds[END_REF], look-up tables [START_REF] Deleforge | The cocktail party robot: Sound source separation and localisation with an active binaural head[END_REF] or manifold learning [START_REF] Talmon | Supervised source localization using diffusion kernels[END_REF][START_REF] Bernard | Sensorimotor learning of sound localization from an auditory evoked behavior[END_REF]. Interestingly, [START_REF] Talmon | Supervised source localization using diffusion kernels[END_REF] uses multiple view points and diffusion kernels to learn the estimation of the azimuth angle of a white-noise source using a single microphone. Finally, a few robotic studies use audio-motor integration for ego-noise reduction. The idea is to map the current motor-state of a robot to some estimates of the corresponding acoustic noise produced by the motors. For instance, the ego-noise power-spectral density can be predicted this way using an artificial neural network [START_REF] Ito | Internal noise suppression for speech recognition by small robots[END_REF] or a nearest neighbour search [START_REF] Ince | Ego noise suppression of a robot using template subtraction[END_REF] based on a training dataset. In [START_REF] Furukawa | Noise correlation matrix estimation for improving sound source localization by multirotor uav[END_REF] the same idea is applied to predict multichannel ego-noise covariance matrices using Gaussian process regression. These statistics can be used to cancel the noise via Wiener filtering, under a local stationarity assumption. In [START_REF] Schmidt | Ego-noise reduction using a motor data-guided multichannel dictionary[END_REF][START_REF] Schmidt | A novel egonoise suppression algorithm for acoustic signal enhancement in autonomous systems[END_REF], a motor data-guided dictionary-learning framework is presented, allowing to model noise which is nonstationary both spatially and spectrally. This framework and exemplary results on real data are presented in Section 2.4 of this chapter.

SINGLE-MICROPHONE SOUND LOCALIZATION USING HEAD MOVEMENTS

In this section, we present a method that uses motor movements to extract auditory features enabling 2D sound source localization with a single microphone placed on an acoustic dummy head 1 . The mapping from features to direction is learned from a training dataset, building on the audio-motor, learning-based sound source localization framework of [START_REF] Deleforge | Acoustic space learning for sound-source separation and localization on binaural manifolds[END_REF].

HRTF MODEL AND DYNAMIC CUES

Let s( f, t) ∈ C denote the signal emitted by a static sound source and y( f, t) ∈ C denote the signal recorded by a microphone placed on an acoustic dummy head in the short-time Fourier domain, with f and t denoting frequency and time indexes, respectively. We assume that the emitted signal is stationary over the considered time interval. The acoustic head acts as a rigid body causing reflections and shadowing of the source signal. The corresponding linear filter, referred to as the head related transfer function (HRTF), is denoted a( f, φ, ψ) and depends on the frequency f and on the relative azimuth φ and elevation ψ of the source in the microphones' frame 2 . Note that in the case of microphone movements, φ and ψ will depend on time as well. We have:

y( f, t) = a f, φ(t), ψ(t) s( f, t). (2.1)
We also define the received log-power spectrogram by

p( f, t) = log |y( f, t)| 2 = log |a f, φ(t), ψ(t) | 2 + log |s( f, t)| 2 . (2.2)
Since the emitted signal s( f, t) is assumed stationary, its power spectral density E t {|s( f, t)| 2 } does not depend on time, and hence its instantaneous estimate |s( f, t)| 2 should be independent of t on average.

Let us now consider that the microphone is mounted on a robotic head with 2 degrees of freedom: pan (azimuth rotation) and tilt (elevation rotation), such as the one showed in Fig. 2.2. Any motor command ξ will change the relative direction (φ, ψ) of the source in the microphone's frame. A command ξ can be identified with a function associating a time t to a motor state ξ(t). This motor state can be accessed through the proprioceptors of the robot. We define a dynamic cue τ

(ξ) = {τ( f, ξ)} F f =1 ∈ R F by the expected temporal derivatives of p( f, t) at all frequencies FIGURE 2.

Active Binaural System

The POPEYE setup used in the CAMIL dataset [START_REF] Deleforge | Learning the direction of a sound source using head motions and spectral features[END_REF] and in our experiments.

A binaural dummy head is mounted on a motor system with two degrees of freedom: pan (left-right) and tilt (up-down).

while the robot performs command ξ:

τ( f, ξ) = E t ∂p( f, t) ∂t , f = 1 . . . F, (2.3)
where F is the number of frequency bands considered. As mentioned, the stationarity assumption on s( f, t) implies that log |s( f, t)| 2 is time-independent on average and hence its expected derivative is zero. On the other hand, the motor command ξ significantly changes the relative source directions φ and ψ, implying that log |a f, φ(t), ψ(t) | 2 has a non-negligible expected derivative. Hence, τ( f, ξ) is a spatial cue that does not depend on the emitted signal and only depends on the source direction (φ, ψ).

In practice, τ(ξ) can be approximated at each frequency by the slope of the leastsquare linear regression between discrete time indexes and the received log-energies {p( f, t)} T t=1 , where T is the number of time frames considered. The validity of this approximation relies on the fact that typical HRTFs are approximately locally-linear with respect to source directions, as demonstrated in [START_REF] Deleforge | Acoustic space learning for sound-source separation and localization on binaural manifolds[END_REF]. Hence, using motor commands with constant angular velocities, p( f, t) should vary approximately linearly in time, justifying the use of linear regression. The slope of the least-square linear regression is then a least mean square estimator of the expected temporal derivative, assuming zero-mean perturbations. This is illustrated in Figure 2.3 using the CAMIL dataset version 0.1 3 [14]. This dataset was recorded with the setup of Fig.

FIGURE 2.3 Illustration of Dynamic Cues

These figures represent the received log-power (2.2) by a single acoustic dummy-head microphone as a function of time (or head-angle) at different frequencies. The least-square linear regression of each curve is represented by a solid line. The emitted signal is an 800ms random mixture of 600 sine waves with frequencies ranging from 50Hz to 6000Hz. Left: the head is static. Middle: the head performs a pan movement at constant 9 • /sec velocity. Right: the head performs a tilt movement at constant 9 • /sec velocity.

2.2.

As can be seen, the stationarity of the emitted signal implies that the received log-power is roughly constant over time when the head is static (left). On the other hand, head movements induce near linear variations of the log-power. The expected derivative of these variations can be approximated by the slope of their least-square linear regression (solid lines). The sampling frequency of the signal was 48 kHz and the sliding short-time Fourier Hamming window was set to 200 ms with 95% overlap. This resulted in 8193 discrete positive frequencies between 0 and 24 kHz and T = 101 time frames per second of signal. In practice, only cues corresponding to frequencies between 1500 and 6000 Hz were kept, as they showed to be the most useful for localization. This resulted in F = 1537 frequency bands used in the proposed dynamic cues.

LEARNING-BASED SOUND LOCALIZATION

Once dynamic cues τ(ξ) are computed, they need to be mapped to a corresponding source direction. While computationally demanding physical models exist to generate HRTF filters based on source directions and accurate 3D head models [START_REF] Xiao | Finite difference computation of head-related transfer function for human hearing[END_REF], we are interested here in the other way around mapping, i.e., from dynamic cues to source directions. This cannot be easily obtained in practice, in particular when head movements are considered. Due to the infeasibility of fully modelling the physics of sound propagation in realistic settings, an alternative approach has recently emerged and is referred to as supervised or learning-based sound source localization [START_REF] Talmon | Supervised source localization using diffusion kernels[END_REF][START_REF] Deleforge | Acoustic space learning for sound-source separation and localization on binaural manifolds[END_REF][START_REF] Deleforge | Colocalization of audio sources in images using binaural features and locallylinear regression[END_REF]. These methods bypass the use of an explicit, approximate physical model by directly learning a mapping from audio features to spatial properties using an appropriate training dataset.

The CAMIL dataset version 0.1 [START_REF] Deleforge | Learning the direction of a sound source using head motions and spectral features[END_REF] consists of recordings made with a binaural dummy head (Fig. 2.2) in the presence of a sound source (loudspeaker) placed at 16, 200 annotated relative directions in the microphones' frame. The source is placed 2.7 meters from the receiver in all recordings, and the reverberation time of the room is around 400ms. For each direction, 3 recordings of 1 second each are available: (i) no head movement, (ii) a 9 • pan movement rightwards at constant speed and (iii) a 9 • tilt movement downwards at constant speed. The emitted source signals are designed to be approximately stationary and correspond to random linear combinations of 600 sine waves with frequencies ranging from 50Hz to 6000Hz and random phase offsets. During motor commands, the annotated relative source position corresponds to the one half-way through the movement. For the single microphone experiments, only the left microphone channel is used.

The training dataset is composed of N pairs {τ n (ξ), z n } N n=1 ⊆ R F × R 2 where z n = (φ n , ψ n ) is the n-th source direction
and ξ is a fixed command throughout the dataset. Three motor commands are considered: pan, tilt or the successive combination of both, each at constant angular velocity (9 • /sec). Only source directions corresponding to azimuth angles between -90 • and +90 • and elevation angles between -45 • and +45 • are kept, resulting in 3812 directions out of which N = 2859 were kept for training and the 953 others for testing.

This training set must be used so that given a new test observation τ(ξ), an associated source direction z can be estimated. To achieve this, we use the high-to low-dimensional regression method Gaussian locally-linear mapping (GLLiM4 [START_REF] Deleforge | Phase-optimized k-svd for signal extraction from underdetermined multichannel sparse mixtures[END_REF]) proposed in [START_REF] Deleforge | High-dimensional regression with gaussian mixtures and partially-latent response variables[END_REF]. GLLiM is a probabilistic method that estimates Q local affine transformations from a low-dimensional space (here, the space of source directions) to a high-dimensional space (here, the space of dynamic cues) using a Gaussian mixture model. This mapping is then reversed through Bayesian inversion, yielding an efficient estimator of z given τ(ξ). GLLiM was notably successfully applied to supervised binaural sound source localization using either real [START_REF] Deleforge | Colocalization of audio sources in images using binaural features and locallylinear regression[END_REF][START_REF] Deleforge | Acoustic space learning for sound-source separation and localization on binaural manifolds[END_REF] or simulated [START_REF] Gaultier | Vast: The virtual acoustic space traveler dataset[END_REF] training sets. Here, a fixed value Q = 50, diagonal and equal noise covariance matrices and equal mixture weights are used in all experiments (see [START_REF] Deleforge | High-dimensional regression with gaussian mixtures and partially-latent response variables[END_REF] for details on the GLLiM method).

RESULTS

The proposed method was trained on a random subset of N = 2859 cue-to-direction pairs and tested on the remaining 953 dynamic cues. This was done so that the emitted training sounds are all distinct from the emitted test sounds and the training source directions are distinct from the test source directions. Results are shown in Table 2.1. As can be seen, audio-motor integration and small head movements enable the localization of a sound source in 2D with high precision (< 4 • with combined commands) using a single microphone. This is impossible to achieve with any of the existing sound source localization methods in the literature, to the best of the authors' knowledge. For comparison, results obtained with a static head and dynamic cues (1 microphone) or binaural cues (ILD and IPD, 2 microphones, as in [START_REF] Deleforge | Acoustic space learning for sound-source separation and localization on binaural manifolds[END_REF]) are showed in Table 2.2. Unsurprisingly, using dynamic cues with a static head yields localization results similar to randomness, with over 50% of outliers. This is because the absence of movement removes spatial information from τ(ξ), bringing (2.3) close to 0. On the other hand, using traditional binaural cues with two static microphones, as done in e.g., [START_REF] Deleforge | Acoustic space learning for sound-source separation and localization on binaural manifolds[END_REF], yields comparable results to the ones obtained with the proposed dynamic cues and a single moving microphone. This validates the feasibility of single-microphone sound source localization by audio-motor integration.

An important limit of this approach is that it requires a carefully annotated training dataset, which is likely to be room-and system-dependent. Besides, the method strongly relies on the assumption that emitted signals are approximately stationary during the emission period (≈ 0.8s in our experiments). Robust extensions of this method that discard time-frequency points with low-energy would be an interesting route to investigate. Besides engineering applicability, these results are the first ones to corroborate psycho-physical evidence suggesting that small continuous head movements may help localizing sounds [START_REF] Wallach | The role of head movements and vestibular and visual cues in sound localization[END_REF][START_REF] Willard R Thurlow | Head movements during sound localization[END_REF][START_REF] Perrett | The effect of head rotations on vertical plane sound localization[END_REF][START_REF] Frederic | Resolution of front-back ambiguity in spatial hearing by listener and source movement[END_REF], using an artificial system.

TABLE 2.1 Single-microphone 2D dynamic sound localization results

Azimuth and elevation estimation errors on the testing set using a single microphone and different motor commands. Results are presented in the form Avg±Std (Out %) where Avg±Std denote the mean and standard deviations of inlying absolute angular errors and Out denotes the percentage of outliers. Outlying errors are defined as those larger than 30 • .

Command: Pan Tilt Pan + Tilt Azimuth ( • ) 3.00 ± 3.3 (0.5%) 4.02 ± 3.5 (0.1%) 3.26 ± 3.2 (0.0%) Elevation ( • ) 2.50 ± 2.8 (0.3%) 1.62 ± 1.3 (0.2%) 1.71 ± 1.6 (0.0%)

TABLE 2.2 2D static sound localization results

Azimuth and elevation estimation errors on the testing set using one static microphone and dynamic cues or two static microphones and binaural (ILPD) cues. Results are presented in the same form as Table 2.1.

Method: Dynamic cues (1 mic.) ILD + IPD [START_REF] Deleforge | Acoustic space learning for sound-source separation and localization on binaural manifolds[END_REF] (2 mic.) Azimuth ( • ) 14.0 ± 8.7 (65%) 2.04 ± 1.6 (0.0%) Elevation ( • ) 13.5 ± 8.5 (40%) 1.26 ± 1.0 (0.0%)

EGO-NOISE REDUCTION USING PROPRIOCEPTORS

This section presents a general audio-motor integration framework for ego-noise reduction, initially presented in [START_REF] Schmidt | Ego-noise reduction using a motor data-guided multichannel dictionary[END_REF] and extended in [START_REF] Schmidt | A novel egonoise suppression algorithm for acoustic signal enhancement in autonomous systems[END_REF]. This frameworks enables the modelling of noise signals that are non-stationary both spatially and spectrally by building on a phase-optimized dictionary-learning method5 [START_REF] Deleforge | Phase-optimized k-svd for signal extraction from underdetermined multichannel sparse mixtures[END_REF] which is also outlined here for completeness.

EGO-NOISE: CHALLENGES AND OPPORTUNITIES

When a robot is moving, its rotating joints as well as the moving parts of its body cause significant noise which is referred to as ego-noise. This corrupts recordings and therefore degrades performance of, e.g., a speech recognizer. To relieve this problem, a suitable noise reduction mechanism is required. This task is particularly challenging because the noise involved is often louder than the signals of interest. Moreover, it is highly non-stationary as the robot performs different movements with varying speeds and accelerations. Furthermore, ego-noise cannot be modelled as a single static point interferer as the joints are located all over the body of the robot.

All this discourages the use of traditional statistical noise reduction techniques such as (multichannel) Wiener filtering or beamforming [START_REF] Heinrich W Löllmann | Challenges in acoustic signal enhancement for humanrobot communication[END_REF][START_REF] Gannot | A consolidated perspective on multimicrophone speech enhancement and source separation[END_REF]. On the bright side, however, two robot-specific opportunities may be exploited. First, ego-noise will usually be strongly structured both spatially and spectrally (e.g., Fig. 2.4, bottomright) because it is produced by an automated system restricted to a limited number of degrees of freedom. Second, important extra information may be exploited in addition to the audio signals, namely, the instantaneous motor state of the robot, e.g., the joints' angles and angular velocities collected by proprioceptors (e.g., Fig. 2.4, top-right).

PROPRIOCEPTOR-GUIDED DICTIONARY LEARNING

The existence of a strong spectral structure in ego-noise motivates the use of dictionary learning methods. The idea is to express structure in terms of sparsity in a particular basis. More precisely, if Y = [y 1 , . . . , y T ] ∈ C P×T represents T examples of P-dimensional signals, there must exist a set of K atoms or a dictionary D = [d 1 , . . . , d K ] ∈ C P×K such that each signal is a linear combination of only a few atoms, i.e., Y ≈ DX where X has sparse columns. Estimating D and X from Y is a sparse instance of matrix factorization. In audio signal processing, it is natural to seek such a factorization in the non-negative power spectral density (PSD) domain, since the magnitude spectra of natural sounds such as speech often feature redundancy and sparsity. This approach gave rise to a large number of methods for audio signal representation and extraction within the framework of non-negative matrix factorization (NMF) [START_REF] Smaragdis | Non-negative matrix factorization for polyphonic music transcription[END_REF][START_REF] Virtanen | Monaural sound source separation by nonnegative matrix factorization with temporal continuity and sparseness criteria[END_REF]. Extensions of NMF to complex-valued multichannel spectrograms have later been proposed [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF][START_REF] Sawada | Multichannel extensions of non-negative matrix factorization with complex-valued data[END_REF]. Most of these extensions assume a simple spatial structure: the modelled signal consists in a mixture of a few fixed, point sources, i.e., with constant steering vectors. Such models are not appropriate for ego-noise which features complex spatial structures. In [START_REF] Deleforge | Phase-optimized k-svd for signal extraction from underdetermined multichannel sparse mixtures[END_REF], a model including both complex spectral and spatial structures was proposed, via phase-optimized dictionary learning. This model is summarized in Section 2.4.3 of this chapter.

The general concept of dictionary-based noise reduction comprises the following steps. First, a noise dictionary D noise is learned from a set of noise-only examples Y noise via the factorization Y noise ≈ D noise X noise with X noise being maximally sparse. Then, given a test observation ỹ ∈ C P containing both noise and target signals, its noisy part is estimated by finding x ∈ C K such that ỹ ≈ D noise x. The enhanced target signal is then given by ỹ -D noise x.

Besides methods purely based on the structure of audio signals, another approach stipulates to exploit available proprioceptor, i.e., motor data, and map them to a noise model. In [START_REF] Ito | Internal noise suppression for speech recognition by small robots[END_REF], the time-varying noise power spectral density (PSD) is estimated by a deep neural network (DNN). The DNN is fed by motor data, which incorporates not only current, but also past sensor values. In [START_REF] Ince | Ego noise suppression of a robot using template subtraction[END_REF], PSD noise templates are used for spectral subtraction. In the learning step, only ego-noise is present and each point in the motor data space is associated with a certain spectral noise template. In the testing (or working) phase, the approach uses a nearest-neighbour criterion to find the best matching template, which is then subtracted from the magnitude spectrum of the recording. In [START_REF] Furukawa | Noise correlation matrix estimation for improving sound source localization by multirotor uav[END_REF], multichannel noise covariance matrices are predicted from motor states via Gaussian process regression.

In [START_REF] Schmidt | Ego-noise reduction using a motor data-guided multichannel dictionary[END_REF], we introduced an alternative and generic audio-motor integration framework to fuse the information brought by an advanced learned structured audio model on the one hand, and instantaneous motor data on the other hand. The key idea is to replace the computationally costly search in the dictionary, which is untractable for large dictionaries (NP-hard [START_REF] Davis | Adaptive greedy approximations[END_REF]), by a classification procedure guided by current proprioceptor data α. These data are fed into support vector machines (SVMs) [START_REF] Schölkopf | Estimating the support of a high-dimensional distribution[END_REF] to efficiently find suitable entries in the ego-noise dictionary D noise . We showed that this approach reduces computational complexity while simultaneously improving performance. This approach and some results are outlined in Section 2.4.4.

PHASE-OPTIMIZED DICTIONARY LEARNING

Although ego-noise is highly non-stationary, it has distinctive spectral and spatial characteristics. The basic idea of a dictionary representation is to capture such characteristics by a collection of prototype signals, called atoms, collected in a dictionary. In our case, the structured ego-noise signal should be represented by a linear combination of a few atoms at each time frame. If these atoms are specifically designed to represent signals sharing spectral and spatial characteristic of ego-noise only, subtracting these atoms should remove the noise while preserving the residual signal of interest such as speech. We briefly summarize here the recent approach [START_REF] Deleforge | Phase-optimized k-svd for signal extraction from underdetermined multichannel sparse mixtures[END_REF] that automatically learns a multichannel dictionary capturing both spatial and spectral characteristics of a training signal. In the following we represent a multichannel signal in the spectral domain by concatenation of the M channels per frequency bin, giving a signal vector of dimension P = MF, where F represents the number of frequency bins per channel. Then, we denote the dictionary by D = [d 1 , . . . , d K ] ∈ C P×K containing K atoms d k ∈ C P . Moreover, the dictionary is corrected by a time-varying phase matrix Φ t ∈ C F×T at each time frame t, where each element has unit complex modulus. The phase-corrected dictionary is then given by

D{Φ t } :=                  d 1,1 . . . d 1,K d 2,1 . . . . . . . . . . . . . . . d F,1 . . . d F,K                                   φ 1,1,t . . . φ 1,K,t φ 2,1,t . . . . . . . . . . . . . . . φ F,1,t . . . φ F,K,t                  (2.4)
where each element d f,k ∈ C M captures the spectral value of atom k at frequency bin f as well as the relative phases and gains between the M channels. Here, denotes a modified Hadamard product, where each vector

d f,k ∈ C M in matrix D
x t 0 ≤ S max , x kt ≥ 0 and |φ f t,k | 2 = 1 ∀ f, k, t. (2.5)
Here, • 2 and • 0 denote the 2 -and 0 -norm, respectively. The latter counts the number of nonzero elements in x t . The minimization (2.5) is done with respect to (w.r.t.) different arguments, depending on which stage of the algorithm is considered.

The training and testing stages are outlined in the following (see [START_REF] Deleforge | Phase-optimized k-svd for signal extraction from underdetermined multichannel sparse mixtures[END_REF] for details).

• Training: (2.5) is minimized w.r.t. D, x t and Φ t . In the training stage, D noise should be learned using a set of training examples Y noise . For this, [START_REF] Deleforge | Phase-optimized k-svd for signal extraction from underdetermined multichannel sparse mixtures[END_REF] proposes the phase-optimized K-SVD (PO-KSVD) algorithm. It can be viewed as a phaseoptimized complex extension of the popular dictionary learning method K-SVD [START_REF] Aharon | rmk-svd: An algorithm for designing overcomplete dictionaries for sparse representation[END_REF]. It alternates between a sparse coding step and a dictionary update step.

• Testing: (2.5) is minimized w.r.t. x t and Φ t for each new test observation ỹt while the pre-trained dictionary D noise is fixed. The best fitting entries from dictionary D noise are searched and subtracted from ỹt , which may contain ego-noise and speech, for example. The NP-hard problem of finding the best combination of atoms is done using an extension of orthogonal matching pursuit (OMP, after [START_REF] Joel | Signal recovery from random measurements via orthogonal matching pursuit[END_REF]) due to its empirically good performance. The extension is called PO-OMP (for phase-optimized OMP).

AUDIO-MOTOR INTEGRATION VIA SUPPORT VECTOR MACHINES

Method description

While the knowledge of the robot's instantaneous motor state should intuitively be beneficial for ego-noise reduction, a crucial question is at which stage motor data should be included. As described in the previous section, one of the main bottlenecks of the multichannel dictionary method in [START_REF] Deleforge | Phase-optimized k-svd for signal extraction from underdetermined multichannel sparse mixtures[END_REF] is the testing phase, where an NP-hard sparse coding problem is approximately solved by the costly iterative PO-OMP procedure. Hence, we propose to replace the entire testing stage by a novel and more efficient motor-guided atom selection method while keeping the dictionary learning stage of [START_REF] Deleforge | Phase-optimized k-svd for signal extraction from underdetermined multichannel sparse mixtures[END_REF], for which computational time is less of an issue.

The physical state of a 1-dimensional robot joint can be described by its position in terms of an angle α t at a given time stamp τ t . Furthermore, from successive angle stamps we can calculate a discrete-time approximation of its first angle derivatives, i.e., angle speed

αt = α t -α t-1 τ t -τ t-1 .
For each joint, we collect the recorded and calculated angle data in a feature vector α t = (α t , αt ), which is, a bit loosely, referred to as motor data in the following. Note that using the joints' acceleration is also an option, but is disregarded here as it did not prove useful in our experiments. Assume a spectrum y t is observed at a time frame τ t in which ego-noise alone is present. Additionally, the motor data α t for this frame is available. The proposed concept stipulates to associate this motor data point with those atoms which PO-OMP would select in a pre-trained ego-noise dictionary to represent spectrum y t . Note that the order in which atoms are chosen is unimportant. For example, if at time step t PO-OMP selects atoms 8, 5 and 7 in the dictionary and at time step t + 1 the output is 7, 8 and 5, both motor data samples α t and α t+1 are associated with the set of atom {8, 7, 5}. The curly brackets {•} emphasize that the order of selected atoms is not considered. The number of possible atom combinations for any given atom set is then

N C = K S max . (2.6)
In the following, the N C possible atom sets are denoted as Dq ∈

C P×S max , q = 1 • • • N C ,
where the choice of notation emphasizes that each Dq contains a selection of atoms from D. Our plan in the following is to decide on a set Dq based on a motor data sample α t and use all atoms in Dq for ego-noise suppression. Fig. 2.5 shows some motor data points in the (α, α)-plane which are associated to a certain set of atoms as an example. They appear to form clusters. The non-linearity i i "ch-Deleforge/figures/tikz-externalize/Book-figure1" -2018/4/27 -14:

-1 1 -1 1 α → ↑ α α t , α t → {8, 4, 3}, α t → {8, 5, 3} FIGURE 2.

Clustering of auditory atoms in the motor-state space

Motor data points α t = (α t , αt ) for a right shoulder pitch movement. Points highlighted in blue and red are those associated with a certain set of selected atoms in the dictionary. They appear to form clusters.

of the clusters' contours motivates the use of a kernel method. It is reasonable to classify these points using a classifier C q (α) ∈ {-1, +1} deciding if a new incoming motor data point α falls into the clustering area of atom set q. If yes, C q (α) = 1, if not C q (α) = -1 holds. All in all, N C such classifiers must be trained. For notational convenience, they are represented in vector form

C(α) = [C 1 (α), . . . , C N C (α)]. (2.7)
We propose to model the data points to cluster as following an unknown probability density function (pdf). By estimating its support, the above described clustering problem is solved. To do so, we use a method from the broad range of support vector machines (SVM). The 1-Class-SVM [START_REF] Schölkopf | Estimating the support of a high-dimensional distribution[END_REF] is a method based on Gaussian kernels that estimates a classifier C(•) whose decision boundaries can be shown to be the support of a pdf that generated the training data with high probability. The reader is referred to [START_REF] Schölkopf | Estimating the support of a high-dimensional distribution[END_REF] for more details on this classification method.

Note that the decision regions of trained classifiers can partly overlap. Formally, an ambiguity is given if for an input motor data vector α t more than two classifiers return +1. To handle this, all N C classifiers are associated with a weighting factor w 1 , . . . , w N C , being identical to the number of involved data points in each of the N C trainings. By this, a decision region gets a larger weight when it contains more data points. Each of the K entries of the dictionary D gets a counter, initialized with zero. It is then iterated over the K atoms: the counter is increased by w q if C q (α t ) = 1 and has the currently investigated atom in its recommendation. The final decision is then given by choosing those atoms that have the S max largest weights.

FIGURE 2.7 Audio-motor ego-noise reduction

Illustration of the test-phase of the proposed audio-motor integration framework for ego-noise reduction. Atoms Dq are selected from D on the basis of motor data α t only. The incoming audio data sample y t is a mixture of Ego-noise and a target signal.

Method Summary

The proposed proprioceptor-guided multichannel dictionary methodology uses the following steps:

• Training: The input consists in spectrogram frame samples y t , t = 1, . . . , T containing ego-noise only. Each sample is associated with a motor data vector α t , t = 1, . . . , T . After D is learned using PO-KSVD [START_REF] Deleforge | Phase-optimized k-svd for signal extraction from underdetermined multichannel sparse mixtures[END_REF] (recall that we do not use motor data for this), PO-OMP is performed with the same samples y t as input.

The selected atoms per sample and their associated motor vector are then processed in the second training step which learns the 1-Class SVMs. This gives N C classifiers, as defined by (2.7). Each of the classifier is associated to one specific set of atoms from D. Fig. 2.6 gives a schematic overview of the training phase. • Testing: The input consists in a new incoming noisy observation y t contain-ing both ego-noise and a target signal to denoise, and a corresponding motor data sample α t . The latter is used to decide immediately on a set of atoms Dq , q = 1, . . . , N C using the trained classifiers. The iterative search in the dictionary is unnecessary, so that the proposed algorithm can be expected to be of significantly lower complexity than PO-OMP without motor data. What remains is only the calculation of the gains for all entries in Dq , collected in vector form xt and the phase optimization, resulting in the phase matrix Φt . Determining those unknowns corresponds to the very last step of PO-OMP [START_REF] Deleforge | Phase-optimized k-svd for signal extraction from underdetermined multichannel sparse mixtures[END_REF], when all atoms have been selected. Fig. 2.7 gives a schematic overview of the testing phase.

RESULTS

We present an experiment performed with the robot NAO [START_REF] Gouaillier | Mechatronic design of nao humanoid[END_REF] (See Fig. 2.4, left). NAO has four microphones which are all located in the head. Furthermore, the robot has 26 joints, 2 in the head, 12 in the arms, 12 in the legs. We perform exclusively movements of the right arm that involve 6 joints. This gives a feature vector of dimension 12, α ∈ R 12 . The sampling frequency for all recordings is F s = 16 kHz, the short-time Fourier transform (STFT) domain uses a Hamming window of length 64 ms and an overlap of 50%. NAO performed its movements in a room with moderate reverberation (T 60 = 200 ms). Each dictionary used in the following was trained with 30 s of recording. The stationary noise from a cooling fan was removed before the training started. For this, we employed a speech distortion weighted multichannel Wiener filter (MWF) [START_REF] Heinrich W Löllmann | Challenges in acoustic signal enhancement for humanrobot communication[END_REF][START_REF] Gannot | A consolidated perspective on multimicrophone speech enhancement and source separation[END_REF]. It needs the power spectral density matrix of pure fan noise as input, which can be easily estimated for constant rotation speed of the fan when the robot is not moving. For testing, 200 utterances from the GRID corpus [START_REF] Cooke | An audio-visual corpus for speech perception and automatic speech recognition[END_REF] were recorded with the fan switched off. The loudspeaker was positioned at a 1 m distance of NAO, at a height of 1.5 m. The recorded utterances were added to out-oftraining movement noise. These mixtures were then used to evaluate the ego-noise suppression algorithms described above after applying the MWF to suppress the fan noise. The classifiers were trained on 2800 motor data samples in total. To find the best parameter ν and γ, we started a sweep over different settings for both variables. Each setting was cross-checked on a set of data points which was excluded from the training. The overall performance of the ego-noise suppression is measured in terms of Signal-to-Inference-Ratio (SIR in dB) and Signal-to-Distortion-Ratio (SDR in dB), as defined in [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF]. While SIR measures the overall noise cancellation, SDR also incorporates information about how much speech is distorted by the suppression algorithm. Additionally, we measure the keyword speech recognition rate (RR), using pocketsphinx [START_REF] Huggins-Daines | Pocketsphinx: A free, real-time continuous speech recognition system for hand-held devices[END_REF] in the GRID corpus [START_REF] Cooke | An audio-visual corpus for speech perception and automatic speech recognition[END_REF], as defined by the CHiME challenge [START_REF] Vincent | The second chimespeech separation and recognition challenge: An overview of challenge systems and outcomes[END_REF].

We tested different parameter constellations for K and S max . The best results were obtained for a dictionary size of K = 20 with sparsity level S max = 3 for this scenario. We parametrized the SVM with a sparsity regularizer ν = 2 2 and a Gaussian kernel width and γ = 2 -2 (see [START_REF] Schölkopf | Estimating the support of a high-dimensional distribution[END_REF] for details on these parameters). 2.3 compares the results obtained with the proposed audio-motor integration method compared to the results obtained with PO-OMP (no proprioceptor data is used). Both the audio-motor and PO-OMP approach clearly outperform the unprocessed recordings in all metrics used. For comparison, we also give suppression results of one-channel NMF [START_REF] Li | Versatile sparse matrix factorization and its applications in high-dimensional biological data analysis[END_REF]. Although NMF brings an improvement, best results are obtained using PO-OMP and the audio-motor approach. The latter clearly reproduces results of PO-OMP and slightly even outperforms it. This can be explained by the fact that PO-OMP sometimes wrongly estimates atoms due to the presence of speech (recall that PO-OMP uses audio data only). As expected, the needed calculation time in Matlab for the classifiers approach is approximately 30% below that of PO-OMP as the search in the dictionary is unnecessary. Note that the theoretical number of possible atom sequences and therefore classifiers is given by (2.6), i.e., 20 3 = 1140 in our case. Interestingly, only 252 classifiers had to be trained in the given case as only 252 atom sets appeared. Therefore, (2.6) is indeed only an upper bound. Nevertheless, we noted that the computational bottleneck of this approach remains the non-convex estimation of the phase-corrections matrix Φ t . This difficult and general problem in audio, referred to as phase unmixing is the subject of current research [START_REF] Deleforge | Phase unmixing: Multichannel source separation with magnitude constraints[END_REF].

CONCLUSION AND PERSPECTIVES

This chapter attempted to bring a unified perspective on audio-motor integration with the key question in mind: How can audio and motor modalities be combined to enhance robot audition? After summarizing the challenges and opportunities specific to robot audition, we surveyed the literature on audio-motor integration, from both psychophysics and robotics viewpoints. We then presented two recent examples of learning-based audio-motor integration framework for robotics. In the first one, addressing the fundamental problem of acoustic source localization, the expected derivative of received audio signal log-power in one microphone with respect to motor-state during a motor command is used to derive spatial features. We showed how these features enable 2D sound direction estimation with a single-microphone, a task impossible to achieve without audio-motor integration. In the second example, addressing the generic problem of ego-noise reduction for acoustic signal enhancement, we presented a general audio-motor integration framework to fuse the information brought by a dictionary-based structured audio model on the one hand, and instantaneous motor data on the other hand. We showed that using motor data reduced the computational complexity while improving performance.

Despite psychophysical evidences pointing out the usefulness of audio-motor integration with humans, only few computational and engineering studies exploit this opportunity. However, with the increasing prominence of robots in our daily lives, and with their fast-developing capability to naturally interact with humans via voice, it is likely that research in computational audio-motor integration will keep gaining momentum. A number of crucial questions will need to be answered along this development:

• How to step out of the classical static-source, static-microphone signal processing framework and, in particular, how to model fast and complex motions? • What is the optimum auditory representation that can be predicted from motor states? Which motor-space features are best suited for audio prediction? • How to close the sensori-motor loop by deriving long-and short-term optimal actions that ease the enhancement of recorded audio signals? • Which information may be extracted from dynamic audio inputs for simultaneous localization and mapping, and how to extract them?

We hope that this chapter helps to direct further research interest towards these exciting challenges with great practical relevance on the horizon.

FIGURE 2 . 4

 24 FIGURE 2.4 Ego-noise of the humanoid robot NAO waving the arm Left: the robot NAO. Bottom-right: Ego-noise Spectrogram of Nao waving the left arm, showing distinctive spectral structures. Note that stationary fan noise components have been removed by a multi-channel Wiener filter [36]. Topright: corresponding motor data, i.e., the angle α and first derivative α of the left shoulder pitch joint involved in the movement (both normalized).

  is multiplied by a global phase term φ f,k,t ∈ C in Φ t . A given multichannel spectrogram frame y t should then be approximated by y t ≈ D{Φ t }x t , where the vector x t ∈ C K picks the atoms from the dictionary. Since only a few atoms should be used, x t is constrained to be sparse, i.e., it should contain at most S max nonzero elements, where S max is referred to as the sparsity level. The overall problem can then be written as

	argmin D,Φ,X

T t=1 y t -D{Φ t }x t 2 2 subject to:

  Illustration of the training phase of the proposed audio-motor integration framework for ego-noise reduction, using training samples y t , t = 1, . . . , T , and associated motor data α t , t = 1, . . . , T .

		i								i
	i	α t , ∀t	"ch-Deleforge/figures/tikz-externalize/Book-figure2" -2018/4/27 -14:38 -page 15 -#1 Dictionary Learning Classifiers Learning	i
		y t , ∀t
				PO-KSVD	D		PO-OMP		Dq , ∀q	1-Class SVM	C(α t )
										D
		FIGURE 2.6 Audio-motor ego-noise training phase
		i								i
	i	D	"ch-Deleforge/figures/tikz-externalize/Book-figure3" -2018/4/27 -14:38 -page 15 -#1	i
				e.g.
		α t	Evaluate Classifiers C(α t )	{1, 5, 9}, {2, 5, 6}	Ambiguity Handling	e.g. {1, 5, 6}	Atom Selection	Dq	Phase Φt , Optimization Gain xt	Dq { Φq } xt
		y t								-	r t
	i									i
		i								i

TABLE 2 .3 Ego-noise reduction results

 2 Comparison of the proposed audio-motor integrated ego-noise reduction method with two baselines and unprocessed signals, using different metrics.

	Proposed [60] PO-OMP [17] NMF [35] Unprocessed	SIR [dB] SDR [dB] RR [%] 14.71 2.64 73.0 14.46 2.57 71.8 2.51 0.8 45.2 -5.48 -8.15 36.1
	Table

This is a preprint version of Chapter

-Audio-Motor Integration for Robot Audition in Multimodal Behavior Analysis in the Wild, Academic Press, 2019, Pages 27-51.

Data available athttps://team.inria.fr/perception/the-camil-dataset/.

The code for this method is available at https://team.inria.fr/perception/gllim_ toolbox/

The code for this method is available at https://robot-ears.eu/po_ksvd/