Clèment Elvira

Adrien Llave

; Cassio

master of explaining game rules. . . fast, but still safe; Fraga Dantas

Laureline Perotin

Katharina Boudgoust

Valentin Gilot

Corentin Louboutin

Scalchi's Doppelganger

Younes, Roilhi, Kilian, Hakim Helena Marin

Mohammed, Corentin, Ewen, Romain, Stephanie, Frederic Axel Nathan

Anaïs Firas

Nicolas

Sunnit Manuel Nicholas

Guillaume ( Oh

Hi Mark

David

Gal Daniel

Shlomi A Nili

Marta Franco

Candy ; Stefania

Keywords: Acoustic echoes, acoustic echo retrieval, room impulse response estimation, audio scene analysis, room acoustics, Blind Channel Identi cation, Super Resolution, Sparsity, Acoustic Impulse Response. Resources:

, based on

A nonna Leti, "Lavati le man e va a lavorare!"; a nonna Carla, "Uuuuuurrrrrcaaaaaaa, che brao ció!"; Résumé en français L'analyse de scène audio vise à récupérer des informations utiles à partir Mots-clés : Echos acoustiques, récupération des échos acoustiques, estimation des réponses impulsionnelles d'une pièce ; analyse de scène sonore, acoustique des salles ; séparation de sources audio, estimation de la géométrie d'une pièce, ltrage spatial, localisation de sources sonores ; apprentissage profond, dictionnaires continus.

d'enregistrements microphoniques. La séparation et la localisation de sources sonores sont des exemples de ces problèmes, dans lesquels on s'intéresse à l'estimation du contenu et des positions de multiples sources de son dans un environnement. En tant qu'humains, nous e ectuons ces tâches sans e ort. Cependant, pour les ordinateurs et les robots, ces tâches restent des dé s à relever. L'une des principales limites est que la plupart des technologies disponibles résolvent les problèmes d'analyse de scènes sonores en tenant compte soit de la sémantique du son, soit des informations spatiales. Le thème central de cette thèse est celui des échos acoustiques : les éléments de la propagation du son faisant le pont entre les informations sémantiques et spatiales des sources sonores. En e et, en tant que répétitions d'un signal source, leurs contributions sémantiques peuvent être aggrégées pour améliorer ce signal. De plus, comme ils sont issus d'une interaction avec l'environnement, leurs cheminements peuvent être retracés et utilisés pour estimer la géométrie de la scène sonore. Sur la base de ces observations, des méthodes récentes en traitement du signal audio tenant compte des échos ont été proposées. Deux questions principales se posent : comment estimer les échos acoustiques et comment exploiter leur connaissance ? Ce travail de thèse vise à améliorer l'état de l'art actuel en traitement du signal audio dans des salles selon ces deux axes. Il fournit également de nouvelles méthodologies et données pour traiter les échos acoustiques et dépasser les limites des approches actuelles. À ces ns, nous présentons tout d'abord deux approches : Une nouvelle méthode basée sur le cadre des dictionnaires continus qui ne nécessite pas de réglages de paramètres ni de données d'apprentissage, et un modèle d'apprentissage profond estimant le décalage temporel de l'arrivée des premiers échos à l'aide de régularisateurs physiquement motivés. En outre, nous présentons un nouvel ensemble de données entièrement annotées, spécialement conçu pour l'estimation d'échos acoustiques et les applications prenant en compte les échos, ouvrant la voie à de futures recherches dans ce nouveau domaine. La deuxième partie de cette thèse concerne l'extension de méthodes existantes d'analyse de scènes audio dans leur forme adaptée aux échos. Le cadre de la NMF multicanale pour la séparation de sources audio, la méthode de localisation SRP-PHAT et la technique de formation de voies (beamforming) MVDR pour l'amélioration de la parole étendues à des versions "echo-aware". Ces applications montrent comment un simple modèle d'écho peut conduire à une amélioration des performances.

C

souligne la di culté d'exploiter les échos acoustiques pour améliorer le traitement audio à dans des salles. Elle constitue une première tentative de jeter des bases analytiques et méthodologiques uni ées pour résoudre ces problèmes et nous espérons qu'elle serve de point de départ à des de nouvelles recherches prometteuses dans ce domaine. vii

Résumé étendu en Français

C

présente en français un aperçu des travaux abordés dans cette thèse. Le thème de l'analyse de scènes audio couvre de nombreuses tâches di érentes qui visent à récupérer des informations utiles à partir d'enregistrements microphoniques. Des exemples de ces problèmes sont la séparation et la localisation de sources sonores, où l'on s'intéresse à l'estimation de la parole et de la position d'un orateur. En tant qu'humains, nous e ectuons ces tâches sans e ort : imaginez que quelqu'un nous appelle de l'autre côté d'une pièce. Votre réaction typique sera probablement de tourner votre attention vers cette personne ou même d'aller vers elle. Cependant, pour les ordinateurs et les robots, ce problème de traitement du signal audio pour ces tâches reste un dé à relever.

Les sons transmettent des informations sémantiques (ce votre ami a dit), temporelles (quand il l'a dit) et spatiales (où ils l'ont dit). Nous pouvons modéliser ces contributions à l'aide de signaux décrivant le contenu du son, et à l'aide de la "réponse impulsionnelle" de la pièce, capturant la propagation du son dans l'espace. Certaines méthodes de traitement audio se concentrent sur ces premiers, ignorant ou décrivant grossièrement cette dernière en raison de la di culté de l'estimer. Les réponses impulsionnelles de pièces intègrent tous les éléments de la propagation du son, tels que les échos, la ré exion di use et la réverbération.

Le thème central de cette thèse sont les échos acoustiques. Ces éléments de propagation du son créent un pont entre les informations sémantiques et spatiales des sources sonores. Comme ce sont des répétitions et des copies du signal source, on peu améliorer le son cible en intégrant par rapport aux autres sources de bruit. Comme ces ré exions sont issues de l'interaction du signal source avec l'environnement, de part leurs temps d'arrivée, on peut remonter leur parcours et ainsi reconstruire la géométrie d'une scène sonore. Sur la base de ces observations, des méthodes de traitement du signal audio ont commencé à prendre en compte ces éléments de propagation du son dans l'analyse de scèness audio. Deux questions principales se posent : comment estimer les échos acoustiques, et comment exploiter leur connaissance ? Ce travail de thèse vise à améliorer l'état de l'art actuel en traitement du signal audio d'intérieur de salles selon ces deux axes. En particulier, il fournit de nouvelles méthodologies et données pour traiter les échos acoustiques et dépasser les limites des approches actuelles. De plus, il prolonge les méthodes classiques d'analyse de scènes audio des formes adaptées aux échos. Ces deux contributions sont développées dans les deux parties principales de la thèse, qui viennent après une introduction, comme résumé ci-dessous. ix x P I, L' Tout d'abord, nous donnons quelques dé nitions préliminaires en traitement du signal audio et énumérons quelques problèmes fondamentaux qui seront abordés tout au long de la thèse, à savoir l'estimation d'échos acoustiques, la séparation de sources audio, la localisation de sources sonores et l'estimation de la géométrie d'une pièce.

• Le chapitre 2 construira un premier pont important : de l'acoustique au traitement du signal audio. Il dé nit d'abord le son, sa propagation dans l'environnement et puis les échos et leurs origines.

• Dans le chapitre 3, nous passons de la physique au traitement du signal où les échos sont modélisés comme des éléments de ltres, appelés Réponses Impulsionnelles de la de Salle, opérant sur le signal source.

Comme le traitement dans le domaine temporel natif est di cile, nous introduisons la représentation de Fourier, qui facilite à la fois l'exposition des méthodes et la mise en oeuvre des algorithmes.

Ce chapitre clôt la première partie introductive.

P II E ' Dans cette deuxième partie de la thèse, nous nous intéressons à l'estimation des premiers échos acoustiques à partir d'enregistrements microphoniques. Basée sur les modèles et dé nitions décrits dans la première partie, cette partie comprend d'abord un aperçu général des méthodes d'estimation d'échos, suivi de la présentation de deux travaux publiés lors de conférences internationales et d'un ensemble de données sur le point d'être publié.

• Tout d'abord, dans le chapitre 4, nous fournissons au lecteur une revue de l'état de l'art en estimation déchos acoustiques, c'est à dire, sur l'estimation de leurs propriétés. Après avoir présenté le problème, nous passons en revue la littérature. A n de fournir un aperçu complet de l'estimation, certains ensembles de données et mesures d'évaluation fréquemment utilisés dans la littérature et ainsi que dans les chapitres suivant sont présentés. Les trois chapitres suivants présentent trois travaux que nous avons menés sur l'estimation des échos acoustiques.

• Le chapitre 5 présente une nouvelle approche pour estimer les échos d'un enregistrement stéréophonique d'une source sonore inconnue telle que de la parole. Contrairement aux méthodes existantes, celle-ci s'appuie sur le cadre récent des dictionnaires continus et ne repose pas sur des réglages de paramètres. La précision et la robustesse de la méthode sont évaluées sur des con gurations simulées di ciles, avec des niveaux de bruit, et de réverbération variables et sont comparées à deux méthodes de l'état de l'art. L'évaluation expérimentale sur des données synthétiques montre que des taux de récupération comparables ou légèrement inférieurs sont observés pour la récupération de sept échos ou plus. En revanche, de meilleurs résultats sont obtenus pour un nombre d'échos inférieurs, et la nature "o -grid" de l'approche donne généralement des erreurs d'estimation plus faibles. C'est prometteur, xi puisque l'avantage pratique de connaître les temps d'arrivée de quelques échos par canal sera démontré dans la dernière partie de la thèse.

• Dans le chapitre 6, nous proposons des techniques d'apprentissage profond pour estimer les propriétés des échos acoustiques. À notre connaissance, il s'agit des premières méthodes de ce type pour ce problème, même si elles présentent des points communs avec des techniques d'apprentissage profond pour la localisation de sources sonores. Nous présenterons trois architectures di érentes qui abordent le problème de l'estimation des échos acoustiques avec un ordre de complexité croissant : l'estimation du temps d'arrivée du champs direct et des premiers échos proéminents ; puis l'exécution de cette estimation de manière plus robuste.

• En n, pour conclure cette deuxième partie, dans le chapitre 7, nous décrivons un ensemble de données que nous avons recueillies et spéci quement conçu pour l'estimation des échos acoustiques. Cet ensemble de données comprend des mesures de réponses impulsionnelles multicanales, accompagnées des annotations des premiers échos et des positions 3D des microphones et des sources réelles et des images sous di érentes con gurations de murs dans une pièce rectangulaire. Ces données fournissent un nouvel outil pour l'évaluation comparative des méthodes récentes en traitement du signal audio "echo-aware" et des outils logiciels permettant d'accéder, de manipuler et de visualiser facilement les données.

P

III A E La troisième partie de la thèse concerne les applications de traitement audio où la connaissance des premiers échos peut améliorer les performances par rapport aux méthodes standards. Pour l'occasion, nous supposons que les propriétés des échos sont disponibles a priori. La structure de cette partie suit le format de la précédente.

• Un chapitre d'introduction (chapitre 8) rassemble les dé nitions standards et présente l'état de l'art actuel en traitement audio d'intérieur sous une même enseigne. Nous considérons quatre problèmes fondamentaux : la séparation de sources audio, la localisation de sources sonores, le ltrage spatial et l'estimation de la géométrie de la pièce. Ces problèmes sont présentés tour à tour avec la revue de la littérature correspondante, en mettant en évidence les dé s actuels. Ces problèmes particuliers seront les protagonistes des trois chapitres suivants.

• Au chapitre 9, les échos sont utilisés pour améliorer les performances de méthodes classiques de séparation de sources audio. C'est le résultat d'une collaboration avec d'autres collègues, publiée lors d'une conférence internationale. Nous proposons notamment une interprétation physique des échos, à savoir les "microphones-images", qui permet de mieux comprendre comment les algorithmes peuvent tirer parti de leur connaissance. Notre étude porte sur deux variantes de la séparation de sources par factorisation en matrices non-négatives multicanale : l'une utilise uniquement les amplitudes des fonctions de transfert et l'autre utilise également les xii phases. Les résultats montrent que l'extension proposées bat l'originale en n'utilisant que quelques échos et que les échos permettent parfois la séparation dans des cas où elle était jugée inabordable.

• Le chapitre 10 aborde le problème de la localisation de sources audio dans le contexte de forts échos acoustiques. En utilisant le modèle de microphones-images présenté dans le chapitre précédent, nous montrons que ces contributions parasites peuvent être utilisées pour modi er la manière classique dont la localisation de sources est e ectuée. En particulier, nous montrons que dans un scénario simple impliquant deux microphones proches d'une surface ré échissante et d'une source, l'approche proposée est capable d'estimer à la fois les angles d'azimut et d'élévation, tâche impossible en supposant une propagation idéale, comme le font les approches classiques. Ces résultats ont fait l'objet d'une publication pour une conférence internationale.

• Le chapitre 11 présente deux applications sensibles aux échos pouvant être utilisées sur l'ensemble de données dEchorate, présenté dans le chapitre 7. Nous illustrons l'utilisation de ces données en considérant deux problèmes possibles d'analyse de scènes audio : le ltrage spatial par échos et l'estimation de la géométrie d'une pièce. A n de valider les données et de montrer leur potentiel, des algorithmes connus de l'état de l'art sont utilisés. Ainsi, pour chacune des applications, les méthodes envisagées sont contextualisées et résumées. Les résultats numériques con rment la valeur potentielle de cet ensemble de données pour la communauté du traitement du signal audio. L'ensemble des données et ces méthodes seront rendus publics pour que des contributeurs externes puissent les a n de pour développer des méthodes de traitement audio plus robustes.

L IV comprend le dernier chapitre (Chapitre 12), qui récapitule les principaux résultats présentés dans ce manuscrit et les perspectives liées à ce travail. Parmi ceux-ci, nous montrons comment peu d'échos acoustiques peuvent être estimés à partir de la seule observation d'enregistrements microphoniques comportant de la parole réverbérante en utilisant un modèle dérivé de la physique de la propagation du son ou des modèles d'apprentissage profond sur simulateur acoustique. De plus, nous démontrons les avantages d'inclure la connaissance des échos acoustiques dans les méthodes de traitement du son. Pour ce qui est de l'évaluation sur données réelles, nous observons que des ensembles de données de référence disponibles librement manquent actuellement dans la littérature. Par conséquent, dans l'esprit de la recherche ouverte, nous construisons un nouvel ensemble de données qui sera bientôt publié. Ces données sont accompagnées d'annotations précises et d'outils algorithmiques pour la recherche "echo-aware", couvrant une grande partie des applications en analyse de scènes audio.

En n, nous voulons souligner la di culté de la tâche d'estimation et d'exploitation des échos acoustiques pour l'amélioration du traitement audio. Cette thèse ne constitue donc qu'une première tentative d'attaquer ces problèmes et pose xiii des bases analytiques sur la façon de les modéliser. Comme toutes premières investigations, beaucoup de choses peuvent être améliorées, et nous espérons que celle-ci pourra servir de point de départ à de futures recherches dans ce nouveau domaine prometteur.

G

x i 3D position of the microphone i recording x i (t) s j 3D position of the source j emitting s j (t) d ii distance between microphone i and i q ij distance between microphone i and source j r j distance of source j w. r. t. to a reference frame θ j azimuth of source j w. r. t. to a reference frame ϕ j elevation of source j w. r. t. to a reference frame ϑ j angle of arrival of source j w. r. . We live immersed in a complex acoustical world, where every concrete thing can sound, resound, and echo. For humans, it is di cult to internalize what is sound, its constituents, and its generation. It is processed by our auditory system and brain so e ciently that our attention is detached from the physical laws governing it. Therefore, when listening to something, we typically focus directly on its semantic content. Evolution lead us to conduct this process without any e orts, despite the presence of a high level of background noise, for instance during a concert. This outstanding capability is not limited to humans and is common to many of the creatures we are sharing the physical world with.

Nonetheless, we process many other information of the complex acoustic scenes we are immersed into. In addition to the semantic content, a sound also conveys temporal and spatial information. For instance, the ticking of a metronome9 or clock provides units of time and when hearing someone shouting, we unconsciously know where to turn our attention. However, this latter information is determined by how sound propagates in space and not by the source itself.

Before reaching the ears, sound propagates in all directions and a portion of its energy arrives at us directly, and another indirectly, after being re ected around. This process leads to the creation of echoes and reverberation. Typical examples are the echoes produced by large rocky mountains or walls in monumental buildings, such as the Panthéon in Rome or the Pont de Neuilly in Paris. 10 In common language, echoes refer to distinctive re ected sounds which can 10 "Écho. Citer ceux du Panthéon et du pont de Neuilly." -Gustave Flaubert, Dictionnaire des idées reçues.

be heard, thus, characterized by a speci c time of arrival and attenuation. In smaller environments, echoes are still present but are typically less perceived as they arrive more quickly and densely. What is perceived here is the so-called reverberation, for which large empty rooms or churches are great examples. Some animals evolved to "see" thanks to echoes. Two of the most striking examples are bats and whales which use them for navigation and hunting. By emitting sound patterns and listening to their re ections returned from the environment, these animals scan the surrounding space, identifying and locating objects. Here, the echoes are voluntarily produced and this is referred to as active echo-location or (bio) sonar. In contrast, in passive echo-location, 1 external, unknown sound sources are used instead. "Locating it" means es-This technique is developed instinctively by some blind people as well. By tapping their canes or clicking their tongues, they are able for instance to avoid obstacles when walking. In the 18th century, French philosopher Denis Diderot recorded the this incredible ability, which was labeled as "echo-location" only 300 years later by Donald Gri n. [START_REF] Kolarik | Sparse reconstruction of incomplete relative transfer function: Discrete and continuous time domain[END_REF]. How I use sonar to navigate the world timating its delay with respect to the direct sound. These delays are then processed as distances in the brain, in the same way our grandparents taught us to localize a storm by counting the time between a lightning and its thunder. That is how bats and whales nd preys, see obstacles, and orientate themselves in dark caves or deep seas. However, the term "echo-location" here could be misleading as it may solely refer to the only problem of locating objects. As we will discuss later, the application of echoes goes beyond simple localization. Therefore, in this thesis, we will prefer the terminology of echo estimation.

Remarkable examples of passive echo estimation in nature are not very well known. Sand scorpions use the propagation of vibrations in the sand to follow the movement of other insects in the dark night. By using their 8 legs as a radar, they perform passive (seismic) echo-location with inevitable consequences for the prey. This technique is common to spiders who sense reverberation in their complex web 11 . They are not only able to localize the preys fast, but also 11 According to some recent studies, spiders appear to o oad cognitive tasks to their webs. The web may then act then as a complex system processing and ltering the information, which is then returned to their owner. [START_REF] Sokol | The thoughts of a spiderweb[END_REF] to identify them, and disambiguate them from simple objects moved by the wind or malicious visitors. In this case, instead of emitting sounds, evolution taught them to use complex structures (for scorpions their legs, for spiders webs) in order to feed and survive.

Echoes do not only serve for computing distances or localizing preys. For instance, they make speech more intelligible, improve our sense of orientation and balancing [Huggett 1953, Chapter 5.4], and provide music with "dimensionality" [START_REF] Sacks | Musico lia[END_REF]]. This phenomenon is a branch of study in room acoustics, pyschoacoustics and sound design. In particular, the former study acoustic echoes for designing theatres, auditoriums and meeting rooms, with the aim of a good listening quality.

The problems addressed in this thesis are indicated in the thesis title: Echoaware signal processing for audio scene analysis. There are three parts in the sentence that deserve an explanation: echo-aware, signal processing and audio scene analysis. In order, we will rst elaborate the last two as they contextualize this thesis, and after, we will explain why and how echoes help.

A

Signal processing is the process of analyzing and modifying a signal, which are mathematical representations of quantities carrying information about a phenomenon. In audio signal processing, this phenomenon is sound. Typical signals are speech or music, and various mathematical and algorithmic techniques have been developed to process them. There are multiple reasons to Audio is a used as a technical term, referring to sound coming from a recording and transmission. Acoustic, instead, refer to the physical aspect of sound.

do this, such as producing new signals with higher quality or and retrieving high-level information that the signal carry. To this end, complex systems are built which can be represented as a collection of simpler subsystems, with well-de ned tasks, interacting with each other. In (audio) signal processing, these subsystems into categories, for instance, representation, enhancement, estimation, etc. Many related problems can be then decomposed into blocks, which may include one or more of the following steps.

R

. The signals can be represented in many di erent ways, so that the information they contain becomes more easily accessible for speci c tasks. It is generally implemented through change of domain or feature extraction. In audio, the most widely used representation is the Fourier basis, which changes the signal domain from time to frequencies.

E

. Observed signals may be a ected by distortions, which corrupts and hides the relevant information. Examples of these are measurement and quantization noise, noisy sources, reverberation, clipping, etc. Therefore, signal enhancement, namely, reducing any form of distortion, is typically a necessary step. Practical examples of enhancement are removing background noise from a mobile phone recording, isolating instrument tracks from in a song, etc.

E

. Often, we wish to estimate some key properties of the target signal, which may be used as inputs to a di erent algorithm. For instance, we may be interested in estimating a speaker's position in a recording, the time of arrival of an echo, or the pitch of a sound.

Enhancement and estimation can be conducted in adaptive fashion, namely with algorithms that are able to adapt themselves to the data. They typically process the data on the y and are controlled by variable parameters resulting from previous estimation blocks. They usually rely on the optimization of an objective function designed to meet speci c requirements. Examples of these algorithms are present in noise-canceling headphones or echo cancellation modules implemented in video conference call systems.

A

Pay attention to what you are hearing right now: there might be music, someone talking to you, footsteps echoing in the other room, background noise due to cars, a heating system, maybe rain or wind, the sound of your movements, and many others. Everything you hear now as well as its location in space is what is called the audio scene 12 . Therefore, audio scene analysis is 12 The correct terminology for it is auditory scene, which relates to human perception. Psychologist Albert Bregman in [START_REF] Bregman | Auditory scene analysis[END_REF]] coined it. However, we will use this terminology since we extend this concept to audio signal processing, and as it is commonly accepted in the literature.

trivially the analysis of it. More speci cally, the extraction and organization

From the ancient greek, analysis means dismantling into constituent elements. It allows then to reach information otherwise obfuscated by the big picture. It is opposed to synthesis, which instead combines parts into a whole. of all the information contained in the sound associated with an audio scene.

In audio signal processing, this process involves using algorithmic and mathematical tools to retrieve and organize such information. After recording the audio scene with microphones, complex systems, as described above, are used to access the information. Accessing di erent types of information at di erent levels of complexity leads to the de nition of di erent problems. These problems focus on well-de ned tasks and some are referred to with established names. Table 1.1 lists some selected audio scene analysis problems that will be considered later in this thesis.

Without getting too philosophical, it is possible to re-cast these problems to some (simple) human interrogations: • What? Answered by Audio Source Separation and Enhancement, Automatic Speech Recognition, and Source Identi cation, operating on the source signals' semantic content;

• Where? Answered by Sound Source Localization, Microphone Calibration, and Room Geometry Estimation, by elaborating the spatial information of the sound propagation;

• When? Answered by Speech Diarization, or leveraging the sound temporal information;

Our brain and the auditory system can instantly and often e ortlessly solve "Everything is connected" -Douglas Adams, Dirk Gently's Holistic Detective Agency

these problems, such that they may seem like trivial tasks. However, they hide many di cult challenges when it comes to designing e cient and robust algorithms. Moreover, most of these problems may exhibit strong interconnections, and the solution of one of them depends on the solution of others.

For instance, knowing when someone is speaking and its location in the room, sound source separation can be achieved more easily. This should not be surprising since it bears a strong parallelism with our everyday experience.

Similarly, echoes may help audio signal processing.

T

As proven by natural behaviors, acoustic echoes are important for humans and animals to analyze the audio scene: As repetitions of a sound, they convey information about that sound. As characterized by temporal instants and attenuation related to distances, they convey spatial information about the audio scene. As modi ed by the frequency description of the object that generates them, they convey acoustic information about it. This observation motivated many researchers to include echoes in signal processing applications, not only limited to audio 13 . However, it was not always 13 The idea of integrating re ection in models is also studied in other elds of engineering.

In telecommunication and networking, for instance, where these phenomena are referred to as multipath propagation.

the case. Many audio scene analysis methods make limiting assumptions on the sound propagation to derive e cient algorithms. One of the most common ones is the so-called anechoic or free-eld scenario, assuming neither echoes nor reverberation is present in the audio scene. Even if this assumption can be seen as reasonable in some scenarios, it is easy to understand its underlying limitations when applied to real-world recordings. Furthermore, in some cases, echoes are considered a source of noise and interference and then modeled as something to cancel out.

Instead, some researchers proposed to explicitly include acoustic echoes in leading to what we will refer to as echo-aware methods. Some of the earliest examples in this direction are the works of [START_REF] Flanagan | Spatially selective sound capture for speech and audio processing[END_REF][START_REF] Jan | Matched-lter processing of microphone array for spatial volume selectivity[END_REF][START_REF] Jan | Sound capture from spatial volumes: Matched-lter processing of microphone arrays having randomly-distributed sensors[END_REF] in source enhancement. However, only recently, these methods have regained interest in audio processing as manifested by the European project SCENIC [START_REF] Annibale | The SCENIC project: Environment-aware sound sensing and rendering[END_REF] and the UK research S 3 A project. In some recent studies, echoes are used to boost performances of typical audio scene analysis problems, e. g., speech enhancement [Dokmanić et al. 2015;[START_REF] Kowalczyk | Raking early re ection signals for late reverberation and noise reduction[END_REF], sound source localization [Ribeiro et al. 2010a], and source separation [Scheibler et al. 2018c;Leglaive et al. 2016;[START_REF] Remaggi | Modeling the Comb Filter E ect and Interaural Coherence for Binaural Source Separation[END_REF], and room geometry estimation from sound [START_REF] Remaggi | Acoustic re ector localization: novel image source reversion and direct localization methods[END_REF][START_REF] Dokmanić | Acoustic echoes reveal room shape[END_REF][START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF].

All these methods show the importance and the bene ts of modeling acoustic re ections. However, prior to all of them is the Acoustic Echo Retrieval (AER) problem. This step, which is most often given for granted in the above applications, is extremely challenging, as shown throughout this entire thesis work.

1.4 T

The goal of this thesis is to improve the current state-of-the-art for indoor "Sometimes a scream is better than a thesis." -Ralph Waldo Emerson audio signal processing along two axes:

1. Provide new methodologies and data to process and estimate acoustic echoes and surpass the limits of current approaches.

2. Extend previous classical methods for audio scene analysis by incorporating the knowledge of echo properties in sound propagation models.

These two goals are elaborated in the two main parts of the thesis, which follow after an introductory one, as summarized below. However the parts are largely interconnected, as shown in Figure 1.1.

P I R A S P

Chapter 2 This chapter builds a rst important bridge: from acoustics to audio signal processing. It rst de nes sound and how it propagates in the environment § 2.1, presenting the fundamental concepts of this thesis: echoes in § 2.2 and Room Impulse Responses (RIRs) in § 2.3. By assuming some approximations, RIRs will be described by parts in relation to methods to compute them. Finally, in § 2.4, how the human auditory system perceives reverberation will be reported. processing such as signals, mixtures, lters and noise in the time domain. In § 3.2, we will present a fundamental signal representation that we will use throughout the entire thesis: the Short Time Fourier Transform (STFT). Finally, in § 3.3, some essential models of the RIR are described.

P

II A E E This part focuses on how to estimate echoes from the only observation of microphone recordings.

Chapter 4 This chapter aims to provide the reader with knowledge in the state-of-the-art of Acoustic Echo Retrieval (AER). After presenting the AER problem in § 4.1, the chapter is divided into three main sections: § 4.2 de nes the categories of methods according to which the literature can be clustered and analyzed in details later in § 4.3. Finally, in § 4.4 some datasets and evaluation metrics for AER are presented.

Chapter 6 In this chapter, we use virtually supervised deep learning models to learn the mapping from microphone recordings to the echoes' timings.

After presenting a quick overview of deep learning techniques ( § 6.2), we will present a rst simple model to estimate the echo coming from a close surface ( § 6.3). This case is motivated by an application in Sound Source Localization (SSL), which will be discussed in details in Chapter 10. Finally, we will present a possible way to achieve a more robust estimation ( § 6.4) and discuss a possible way to scale this approach to multiple echoes ( § 6.5).

Chapter 5 This chapter proposes a novel approach for o -grid AER from a stereophonic recording of an unknown sound source such as speech.

In order to address some limitation of existing methods, we propose a new approach, named Blind And Sparse Technique for Echo Retrieval (BLASTER). It builds on the recent framework of Continous Dictionary (CD), and it does not rely on parameter tuning nor peak picking techniques by working directly in the parameter space of interest. The method's accuracy and robustness are assessed on challenging simulated setups with varying noise and reverberation levels and are compared to two state-of-the-art methods. While comparable or slightly worse recovery rates are observed for recovering seven echoes or more, better results are obtained for fewer echoes, and the o -grid nature of the approach yields generally smaller estimation errors.

Chapter 7 This chapter presents dEchorate: a new database of measured multichannel room impulse response (RIRs) including annotations of early echoes and 3D positions of microphones, real and image sources under di erent wall con gurations in a cuboid room. These data provide a tool for benchmarking recent methods in echo-aware speech enhancement, room geometry estimation, RIR estimation, acoustic echo retrieval, microphone calibration, echo labeling, and re ectors estimation. The database is accompanied by software utilities to easily access, manipulate, and visualize the data and baseline methods for echo-related tasks.

P III E

A S A In this part, we present some audio scene analysis problems that will be later discussed in their echo-aware extension.

Chapter 8 In this chapter, we present a selection of algorithms and methodologies which we identi ed as potential bene ciaries of echo-aware additions. At rst, we present a typical scenario that highlights some cardinal problems. Then, state-of-the-art approaches to address these problems are listed and commented in dedicated sections, highlighting the relationship with some acoustic propagation models, respectively. The content presented here serves as a basis for a deeper investigation conducted in each of the following chapters.

Chapter 9 In this chapter, echoes are used for improving performance of classical Multichannel Audio Source Separation (MASS) methods. Existing methods typically either ignore the acoustic propagation or attempt to estimate it fully. Instead, this work investigates whether sound separation can bene t from the knowledge of early acoustic echoes. These echo properties are derived from the known locations of a few image microphones. The improvements are shown for two variants of a method based on non-negative matrix factorization based on the work of [Ozerov and Févotte 2010]: one that uses only magnitudes of the transfer functions and one that uses the phases as well. The experimental part shows that the proposed approach beats its vanilla variant by using only a few echoes and that with magnitude information only, echoes enable separation where it was previously impossible.

Chapter 10 In this chapter, we show that early-echo characteristics can, in fact, bene t SSL. To this end, we introduce the concept of Microphone Augmentation with Echoes (MIRAGE), based on the image microphone model. In particular, we show that in a simple scenario involving two microphones close to a re ective surface and one source, the proposed approach can estimate both azimuthal and elevation angles, an impossible task assuming an anechoic propagation.

Chapter 11 This chapter presents two echo-aware applications that can benet from the dataset dEchorate. In particular, we exemplify the utilization of these data considering two possible use-cases: echo-aware speech enhancement ( § 11.1) and room geometry estimation ( § 11.2). This investigation is conducted using state-of-the-art algorithms described and contextualized in the corresponding sections. In the nal section ( § 11.3), the main results are summarized, and future perspectives are presented.

F

, the dissertation concludes with Chapter 12, which summarizes the contributions and raises several additional research questions.

L

This dissertation draws heavily on the work and writing of the following papers, written jointly with several collaborators:

• Di Carlo, Diego, Pinchas Tandeitnik, Sharon Gannot, Antoine Deleforge, and Nancy Bertin (2021). "dEchorate: a calibrated Room Impulse Response database for acoustic echo retrieval". In: Workin progess The reader will have already noticed that a large margin is left free on each manuscript page. We will use it to insert personal comments, historical notes, additional insights, and gures and tables to complete each subject. This graphic template is inspired by the work of Tufte and Graves-Morris [Tufte and Graves-Morris 1983] 14 . We emphasize the presence of clickable links by 14 The colophon of the thesis reports more information on the template.

the logo. Code libraries are written in typewriter font, e. g. dEchorate.

1.6.1 Quick vademecum

For the readers:

• orange is used for clickable internal reference, such as for sections § 1.1 and acronyms FFT;

• grey and is used for clickable external link, such as my website ;

• reference sidenotes on the margin are used as footnotes, providing additional insights;

• italic sidenotes and gures without proper reference numbers on the margin are meant to provide optional information and can be read in a second moment;

• should capture the reader attention towards the important points;

• ? indicates a question-and-answer paragraph;

• indicates the presence of de nitions by dichotomy;

• the following icon denotes the end of the chapter.

The golden ratio of the thesis

This thesis has been written following personal stylistic rules:

• after the colon punctuation mark, ":", small caps. In case of list, items are delimited by semicolon punctuation mark ";";

• sidenotes number referring to a word superscripts the word itself, sidenotes referring to a whole sentence superscripts the corresponding full stop;

• N'ilazo color palette: black, grey and orange;

• at most three levels of sub-headings: section, subsection, and Tufte's new-thought [Tufte and Graves-Morris 1983] to capture attention;

• the usage of dichotomies is privileged when presenting concepts and de nitions, thus they are emphasized;

• each section should be introduced brie y at the end of the previous one to promote reading ow;

S

This chapter builds a rst important bridge: from acoustics to audio signal processing. It rst de nes sound and how it propagates in the environment § 2.1, presenting the fundamental concepts of this thesis: echoes in § 2.2 and Room Impulse Responses (RIRs) in § 2.3. By assuming some approximations, RIRs will be described by parts in relation to methods to compute them. Finally, in § 2.4, how the human auditory system perceives reverberation will be reported.

The material on waves and acoustic re ection is digested from classical texts on room acoustics [START_REF] Kuttru | Room acoustics[END_REF][START_REF] Pierce | Acoustics: an introduction to its physical principles and applications[END_REF]] and on partial di erential equations [START_REF] Du | Green's functions with applications[END_REF].

S

Imagine a calm pond. The surface is at and smooth. Drop a rock into it. Kerploop! The surface is now disturbed. The disturbances spread and propagate, as waves. The medium here is the water surface.

According to common dictionaries and encyclopedias, sound is the sensation caused by the vibration of air and perceived by the ear. This de nition highlights two aspects of sound: a physical one, characterized by the air particles vibration; and a perceptual one, involving the auditory system. Focusing on the former phenomenon, when vibrating objects excites air, surrounding air molecules starts oscillating, producing zones with di erent air densities leading to a compressions-rarefactions phenomenon. Such a vibration of molecules takes place in the direction of the excitement, with the next layer of molecules excited by the previous one. Pushing layer by layer forward, a longitudinal mechanical wave15 is generated. Notice that therefore sound needs a medium to travel: it cannot travel through a vacuum and no sound is present in outer space. Thus, sound propagates through a medium, which can be solid, liquid or gaseous. The propagation happens at a certain speed which depends on the physical properties of the medium, such as its density. The medium assumed throughout the entire thesis is air, although extensions of the developed methods to other media could be envisioned. Under the fair assumption of air being homogeneous and steady, the speed of sound can be approximated as follows:

c air = 331.4 + 0.6T + 0.0124H [ m /s],
(2.1)

where T is the air temperature [ • C] and H is the relative air humidity [%].

13

The air pressure variations at one point in space can be represented by a waveform, which is a graphical representation of a sound, as shown in Figure 2.2. We can think of this process in the light of the classical source-medium-receiver model of communication theory: the source is anything that emits waves 16 , the 16 Example of sources are vibrating solids (e. g. loudspeakers membrane), rapid compression or expansion (e. g. explosions or implosions) or air vortices (e. g. ute and whistles).

medium carries the waves from one point to another, and the receiver receives them.

The acoustic wave equation

The acoustic wave equation is a second-order partial di erential equation 17 17 In 1746, d'Alembert discovered the onedimensional wave equation for music strings, and within ten years Euler discovered the three-dimensional wave equation for uids.

which describes the evolution of acoustic pressure p as a function of the position x and time t

∇ 2 p(x, t) - 1 c 2 ∂ 2 p(x, t) ∂t 2 = 0, (2.2)
where

∇ 2 = ∂ 2 ∂x 2 + ∂ 2 ∂y 2 + ∂ 2
∂z 2 stands for the 3-dimensional Laplacian operator. The constant c is the sound velocity in the medium and has dimension m s . Bundary condition and free eld de nition here. The wave equation is linear, which implies the followings:

• the pressure eld at any time is the sum of the pressure elds resulting from each source at that time;

• the pressure eld emitted at a given position propagates over space and time according to a linear operation.

Assuming the propagation of the wave in a homogeneous medium, one can obtain the equation above by combining three fundamental physical laws:

• the conservation of momentum,

• the conservation of mass, and

• the polytropic process relation, meaning that the medium is an ideal gas undergoing a reversible adiabatic process.

However, media are not uniform and feature inhomogeneities of two types: scalar inhomogeneities, e. g. due to temperature variation, and vector inhomogeneities, e. g. due to presence of fans or air conditioning. Although these a ect the underlying assumption of the model, the e ects are small in typical application of speech and audio signal processing. Therefore they are commonly ignored.

T H ' Equation 2.2 is expressed in the space-time domain (x, t). By applying the temporal Fourier transform, we obtain the Helmholtz equation:

∇ 2 P (x, f ) + k 2 P (x, f ) = 0, (2.3)
where k = 2πf c is known as wave number and relates the frequency f to the propagation velocity c. Both the wave (2.2) and the Helmholtz's equation (2.3) are source-independent, namely no source is present in the medium. Therefore they are said to be homogeneous as the right-hand term is zero. Normally the sound eld is a complex eld generated by acoustic sources. As consequence, the two equations become inhomogeneous as some non-zero terms needs to be added to the right-hand sides.

In the presence of a sound source producing waves with source function s(x, t), the wave equation can be written

∇ 2 p(x, t) - 1 c 2 ∂ 2 p(x, t) ∂t 2 = s(x, t).
(2.4) Thus, the corresponding Helmholtz's equation writes

∇ 2 P (x, f ) -k 2 P (x, f ) = S(x, f ). (2.5)
For instance one can assume an in nitesimally small sphere locate at s emitting a sound at frequency f , i. e. S(x) = δ(xs). At the receiver position x = s, the Helmholtz's equation writes

∇ 2 H(x, f | s) -k 2 H(x, f | s) = δ(x -s), (2.6)
The function H(x, f | s) satisfying Eq. (2.6) is called the Green's function and is associated to Eq. (2.3), for which it is also a solution. Green's Functions are mathematical tools for solving linear di erential equations with speci ed initial-and boundary-conditions [START_REF] Du | Green's functions with applications[END_REF]. They have been used to solve many fundamental equations, among which Eqs. (2.2) and (2.3) for both free and bounded propagation. They can be seen as a concept analogous to impulse responses 18 in signal processing. Under this light, the According to Green's method, the equations above can be solved in the frequency domain for arbitrary source as follows:

P (x, f ) = Vs H(x, f | s)S(s, f ) ds, (2.7)
where V s denotes the source volume, and ds = dx s dy s dz s the di erential volume element at position s. If one ignores the space integral, one can see the close relation with a transfer function. Finally, the requested sound pressure p(x, t) can be computed by taking the frequency-directional inverse Fourier transform of Eq. (2.7).

It can be shown [START_REF] Kuttru | Room acoustics[END_REF]] that the Green's function for Eqs. (2.3) and (2.6) writes form to the result above, we can write the time-domain Green's function as

H(x, f | s) = 1 4π x -s e -i2πf x-s c (2.8)
h(x, t | s) = 1 4π x -s δ t - x -s c (2.9)
where δ(•) is the time-directional Dirac delta function.

As consequence, the free eld, that is open air without any obstacle, the sound propagation incurs a delay q/c and an attention 1/(4πq) as function of the distance q = xs from the source to the microphone.

According to Eq. (2.9), the sound propagates away from a point source with a spherical pattern. When the receiver is far enough from the source, the curvature of the wavefront may be ignored. The waves can be approximated as plane waves orthogonal to the propagation direction. This scenario depicted in Figure 2.3 is known as far-eld. In contrast, when the distance between F 2.3: Visualization of the sound propagation. Since the sensor (i.e. a microphone) is drawn in the far eld, the incoming waves can be approximated as plane waves.

the source and the receiver is small, the scenario is called near eld. These distances can be quanti ed exactly based on the sound wavelength,

λ = 2π k = c f [m],
(2.10)

where f is the frequency of the sound wave. As depicted in Figure 2.2, λ measures the spatial distance between two points around which the medium has the same value of pressure. In practical acoustic applications, the boundary between near-and far-eld can be empirically set at 2λ. For instance, for a sound source emitting a pure sinusoidal tone at 1 kHz, the the receiver sensor can be considered in far-eld if placed ∼69 cm away from the source.

A

The equations derived so far assumed unbounded medium, i. e. free space: a rare scenario in everyday applications. Real mediums are typically bounded, at least partially. For instance in a room, the air (propagation medium) is bounded by walls, ceiling, and oor. When sound travels outdoor, the ground acts as a boundary for one of the propagation directions. Therefore, the sound wave does not just stop when it reaches the end of the medium or when it encounters an obstacle in its path. Rather, a sound wave will undergo certain behaviors depending on the obstacles' acoustics and geometrical properties, including

• re ection o the obstacle,

• di raction around the obstacle, and

• transmission into the obstacle, causing refraction through it, and

dissipation of the energy.

Re ections typically arise when a sound wave hits a large surface, like a room wall. When the sound meets a wall edge or a slit, the wave di racts, namely it bends around the corners of an obstacle. The point of di raction e ectively becomes a secondary source which may interact with the rst one. The part of energy transmitted to the object may be absorbed and refracted. Objects are characterized by a proper acoustic resistance, called acoustic impedance (see § 2.2.1), which describes their acoustic inertia as well as the energy dissipation. The remaining contribution may continue to propagate resulting in the refraction phenomenon.

When sound re ects on an solid surface, two types of acoustic re ections can occur: part of the sound energy • is re ected specularly, i. e., the angle of incidence equals the angle of re ection; and

• is re ected di usely -or scattered, i. e., scatter in every direction).

All the phenomena occur with di erent proportions depending on the acoustics and geometrical properties of surfaces and the frequency content of the wave.

In acoustics, it is common to de ne the operating points and di erent regimes, e. g. for instance near-vs. far-eld, according the wavelength λ. Therefore, we can identify the following three responses of objects (irregularities) of size d to a plane-wave, as depicted in Figure 2.6

• λ d, the irregularities are negligible and the sound wave re ection is of specular type;

• λ ≈ d, the irregularities break the sound wave which is re ected towards every direction;

• λ d, each irregularities is a surface re ecting specularly the sound waves. This presented behavior can be described with the wave equation by imposing adequate boundary conditions. A simpli ed yet e ective approach -just as in optics -is to model incoming sound waves as acoustic rays [START_REF] Davis | Sound pulse photography as applied to the study of architectural acoustics[END_REF][START_REF] Krokstad | Calculating the acoustical room response by the use of a ray tracing technique[END_REF]. A ray has well-de ned direction and velocity of propagation perpendicular to the wavefront, and conveys a total wave energy which remains constant. This simpli ed description undergoes with the name of Geometrical (room) acoustics (GA) [START_REF] Savioja | Overview of geometrical room acoustic modeling techniques[END_REF], and share many fundamentals with geometrical optics. This model will be convenient to Schlieren photographs showing successive stages in the progress of a sound pulse in a section of a Debating Chamber. Image courtesy of [START_REF] Davis | Sound pulse photography as applied to the study of architectural acoustics[END_REF]. Sabine used Schlieren photography to investigate sound propagation paths in the early 1900s. Their impressive visualizations show wavefronts that are augmented with rays that are perpendicular to the wavefronts describe and visualize the re ection behavior hereafter.

Large smooth surfaces, absorption and echoes

Specular re ections are generated by surfaces which can be modelled as in nite, at, smooth and rigid. As mentioned above, this assumption is valid as long as the surface has dimension much larger than the sound wavelength. Here the acoustic ray is re ected according to the law of re ection, stating that (i) the re ected ray remains in the plane identi ed by the incident ray and the normal to the surface, and (ii) the angles of the incident and re ected rays with the normal are equal. If the surface S is not perfectly rigid or impenetrable, its behavior is described by the acoustic impedance, Z S (f ) ∈ C. Analytically, it is de ned as a relation between sound pressure and particle velocity at the boundary. It consists of a real and imaginary part, called respectively acoustic resistance and reactance. The former can be seen as the part of the energy which is lost, and the latter as the part which is stored.

T

β can be derived from the acoustic impedance for plane waves, i. e. under assuming a far-eld regime between source, receiver and surface. It measures the portion of the incident wave energy absorbed by the surface. Analytically, it is de ned as [START_REF] Kuttru | Room acoustics[END_REF][START_REF] Pierce | Acoustics: an introduction to its physical principles and applications[END_REF] .11) where Z S (f ) are the frequency-dependent impedance of the surface, ξ air (f ) is a parameter of the propagation medium (air) sometimes called as intrinsic impedance, and θ is the angle of incidence.

β(f, θ) = Z S (f ) cos θ -ξ air (f ) Z S (f ) cos θ + ξ air (f ) , ( 2
T is typically used instead in the context of GA and audio signal processing. It comes from the following approximations [START_REF] Savioja | Overview of geometrical room acoustic modeling techniques[END_REF]: (i) the energy or intensity of the plane wave 19 is consid- 19 Since it is the square magnitude of the acoustic pressure, the phase information is lost.

ered instead of the acoustic pressure; (ii) dependency on the angle of incidence is relaxed in favor of the averaged quantities; (iii) local dependency on frequencies is relaxed in favor of a frequency-independent scalar or at most a description per octave-band. These assumptions are motivated by the diculty of measuring the acoustic impedance and the possibility to compute an equivalent coe cient a posteriori. Therefore, it is customary to use the absorption coe cient, de ned as

α(f ) = 1 -β(f ) 2 , (2.12)
where β is the re ection coe cient averaged over the angles θ.

E which distinguish themselves in terms

The word echo derives from the Greek echos, literally "sound". In the folk stories of Greece, Echo is a mountain nymph whose ability to speak was cursed: she was only able to repeat the last words anyone spoke to her.

of gain and timing. Originally this term is used to refer to sound re ections which are subjectively noticeable as a separated repetition of the original sound signal. These can be heard consciously in outdoor, for instance in mountains. However, they are less noticeable to the listener in close rooms. In § 2.3.1 a proper de nition of echoes will be given with respect to the temporal distribution of the acoustic re ections.

Di usion, scattering and di raction of sound

Real-world surfaces are not ideally at and smooth; they are rough and uneven.

Examples of such surfaces are co ered ceilings, faceted walls, raw brick walls as well as the entire audience area of a concert hall. When such irregularities are in the same order as the sound wavelength, di use re ections is observed.

In the context of GA, the acoustic ray associated to a plane-wave can be though of as a bundle of rays traveling in parallel. When it strikes such a surface, each individual rays are bounced o irregularly, creating scattering: a number of new rays are created, uniformly distributed in the original half-space. The energy carried by each of the outgoing ray is angle dependent and it is well modeled thought the Lambert's cosine law, originally used to describe optical di use re ection. The total amount of energy of this re ection may be computed a-priori knowing the scattering coe cient of the surface material. Alternatively, it can be derived a-posteriori with the di usion coe cient, namely the ratio between the specularly re ected energy over the total re ected energy.

Di raction waves occur when the sound confronts the edge of a nite surface, for instance around corners or through door openings. This e ect is shown in Figure 2.8 At rst the sound wave propagates spherically from the source. F 2.8: Schematic representation of sound di raction. This e ect allows to hear "behind walls".

Once it reaches the re ector's apertures, the wave is di racted, i. e. bended, behind it. It is interesting to note that the di raction waves produced by the semi-in nite re ector edge allow the area that is "behind" the re ector to be reached by the propagating sound. This physical e ect is exploited naturally by the human auditory system to localize sound sources.

R

Room acoustics studies acoustic waves propagating in enclosed volume delimited by surfaces (walls, oors, etc.), with which an incident wave interacts as described in § 2.2. In this context, F 2.9: Schematic illustration of the shape of an RIR and the rst 100 ms of a measured one. Image taken from [START_REF] Vincent | Audio source separation and speech enhancement[END_REF] a room is a physical enclosure containing a propagation medium with boundaries limiting the sound propagation.

Mathematically, the sound propagation is described by the wave equation (2.2). By solving it, the Acoustic Impulse Response (AIR) 20 from a source to a mi- 20 The Acoustic Transfer Function (ATF) is the Fourier transform of the AIR crophone can be obtained. In the context of room acoustics, it is commonly referred to as the Room Impulse Response (RIR), usually stressing the geometric relation between re ections and the geometry of the scene. In this thesis the two terms will be used indistinctly.

The room impulse response

Room Impulse Response (RIR) is where physical room acoustics and indoor audio signal processing meet and from now on, we will adopt a signal processing perspective. Therefore, a RIR is a causal time-domain lter that accounts for the whole indoor sound propagation from a source to a receiver.

Figure 2.9 provides a schematic illustration of the shape of a RIR compared to a measured one. The RIRs usually exhibit common structures. Based on the consideration of § 2.2, they are commonly divided into three partially overlapping components:

h(t) = h d (t) + h e (t) + h l (t), (2.13)
where

• the direct path h d (t) is the line-of-sight contribution of the sound wave. This term coincides with the "pure delay" modeled by the free-eld propagation model (2.9).

• the acoustics echoes or early re ections are included in h e (t) comprising few disjoint re ections coming typically from room surfaces. They are usually characterized by sparsity in the time domain and amplitude prominence greater than the later re ections. These rst re ections are typically specular and are well modeled in general by the Image Source Method (ISM) explained in § 2.3.3.

• the late reverberation, or simply reverberation, h l (t) collects many re ections occurring simultaneously. This part is characterized by a di use sound eld with exponentially decreasing energy.

These three components are not only "visible" when plotting the RIR against time, but are characterized by di erent perceptual features, as explained in § 2.4.

To conclude, let s and h be the continuous-time source signal and RIR, respectively. Then, assuming that the RIR is time-invariant, the received sound writes

x(t) = ( h s)(t) def = h(u)s(t -u) du, (2.14)
where the symbol is the continuous-time convolution operator.

Apart for certain simple scenarios, computing RIRs in closed forms is a cumbersome task. Therefore numerical solvers or approximate models are used instead.

Simulating room acoustics

The docoumentation of the Wayverb acoustic simulator o ers a complete overview of the State of the Art (SOTA) in acoustic simulator methods [START_REF] Thomas | Wayverb: A Graphical Tool for Hybrid Room Acoustics Simulation[END_REF]].

Most available simulators falls in three main categories:

• Wave-based simulators aims at solving the wave equation numerically;

• Geometric simulators make some simplifying assumption about the wave propagation. They typically ignore the wave physic, instead they adopt much lighter models such as rays or particles;

• Hybrid simulators combining both approaches.

W M are iterative methods that divide the 3D bounded enclosure into a grid of interconnected nodes 21 . For instance, the Finite Element 21 i. e. mechanical unit with simple degrees of freedoms, like mass-spring system or onesample-delay unit Method (FEM) divides the space into small volume elements smaller than the sound wavelengths, while the Boundary Element Method (BEM) divides only the boundaries of the space into surface elements. These nodes interact with each other according to the math of the wave equation. Unfortunately, simulating higher frequencies requires denser interconnection, so the computational complexity increases. The Finite-Di erence-Time-Domain (FDTD) method replaces the derivatives with their discrete approximation, i. e. nite di erences. The space is divided into a regular grid, where the changes of a quantity (air pressure or velocity) is computed over time at each grid point. Digital Waveguide Mesh (DWM) methods are a subclass of FDTD often used in acoustics problem.

Example of a mass-spring linear mesh used to simulate a 1D transversal wave.

The main drawback of these methods is discretisation: less dense grids may simplify too much the simulation, while denser grids increase the computational load and other intrinsic limitations. 22 Moreover, they require delicate 22 For more details on this, the reader can refers to the Courant-Friedrichs-Lewy conditions and the documentation of the Wayverb .

de nitions of the boundary condition at the physical level, like knowing complex impedances, which are rarely available in practice. On the other hand these methods inherently account for many e ects such as occlusions, re ections, di usion, di ractions and interferences. In particular, by simulating F 2.11: Simulation of Sound propagation at four consecutive timestamps using the DWM technique. A short, sharp, impulsive sound red into the larger of two rooms causes a circular wavefront to spread out from the sound source. The wave is re ected from the walls and part of it passes through a gap into the smaller room. In the larger room, interference e ects are clearly visible; in the smaller room, the sound wave has spread out into an arc, demonstrating the e ects of di raction. A short while after the initial event, the sound energy has spread out in a much more random and complex fashion. Image taken form University of York's AudioLab website.

accurately low-frequency components of the RIR, they are able to well characterize the room modes 23 , namely, collections of resonances that exist in a room 23 Room modes have the e ect of amplifying and attenuating speci c frequencies in the RIR, and produce much of the subjective sonic "colour" of a room. Their analysis and synthesis is of vital importance for evaluating acoustic of rooms, such as concert halls and recording studios or when producing musically pleasing reverbs. and characterize it. As stated in [START_REF] Välimäki | More than 50 years of arti cial reverberation[END_REF], among the wave-based methods, the DWMs are usually preferred: they run directly in the time domain, requiring typically an easier implementation, and they exhibit a high level of parallelism.

G can be sub-grouped into stochastic and deterministic

For a detailed discussion about geometric acoustic methods, please refer to [START_REF] Savioja | Overview of geometrical room acoustic modeling techniques[END_REF].

approaches. They typically compute the re ection path(s) between the source and the receivers, assuming that the wave behaves like a particle or a ray carrying the acoustic energy around the scene.

S are approximate by nature. They are based on statistical modeling of the RIRs or Monte Carlo simulation methods. The former ones write statistical signal processing models based on prior knowledge, such as probability distribution of the RIR in regions of the time-frequency domain [START_REF] Badeau | Common mathematical framework for stochastic reverberation models[END_REF]]. Rather than the detailed room geometry, these methods generally use high-level descriptors 24 to synthesize RIRs and in some applica-24 such as the amount of reverberation or source-to-receiver distance.

tion are preferable. The latter ones randomly and repeatedly subsample the problem space, e. g. tracing the path of random re ections, recording samples which ful l some correctness criteria, and discarding the rest. By combining the results from multiple samples, the probability of an incorrect result is reduced, and the accuracy is increased. Typically the trade-o between quality and speed of these approaches is based on the number of samples and the quality of the prior knowledge modeled.

Ray-tracing [START_REF] Kulowski | Algorithmic representation of the ray tracing technique[END_REF]] is one the most common methods that fall in this category and is very popular in the eld of computer graphics for light simulation. The basic idea is to collect "valid" paths of discrete rays traced around the room. Many technique have been proposed to reduce the computational load, among which the di use rain algorithm [START_REF] Schröder | A fast reverberation estimator for virtual environments[END_REF][START_REF] Heinz | Binaural room simulation based on an image source model with addition of statistical methods to include the di use sound scattering of walls and to predict the reverberant tail[END_REF]] is commonly used in many acoustic simulators. Each ray trajectory is re ected in a random direction every time it hits a wall and its energy is scaled according to the wall absorption. The process of tracing a ray is continued until the ray's energy falls below a prede ned threshold. At each re ection time and for each frequency (bin or band), the ray's energy and angle of arrival are recorded in histograms, namely a directional-time-frequency energy map of the room's di use sound eld for a given receiver location (see Figure 2.12). This F 2.12: Directional-time-frequency energy map resulting form the di use rain algorithm [START_REF] Schröder | A fast reverberation estimator for virtual environments[END_REF]]. For each direction, that is receiver's spherical bin, a timefrequency histogram collects the energy of incoming rays. Image taken from [START_REF] Schimmel | A fast and accurate "shoebox" room acoustics simulator[END_REF] map is then used as a prior distribution for drawing random sets of impulses which are used to form the RIR. While lacking a detailed description of early re ections and room modes, these methods are good to capture and simulate the statistical behavior of the di use sound eld at a low computational cost. F 2.13: Visualization of ray-tracing method. From top to bottom: rst the method will eventually nd specular re ection; then di use re ections can be modeled either by splitting a ray into several new rays or a single random one. In the di use rain technique a shadow-ray is cast from each di use re ection point to the receiver to speed-up convergence of the simulation. Image courtesy of [START_REF] Savioja | Overview of geometrical room acoustic modeling techniques[END_REF] D are good to simulate early re ections instead: they accurately trace the exact direction and the timing of the main re ections' paths. The most popular method is the Image Source Method (ISM), proposed by Allen and Berkley in [START_REF] Allen | Image method for e ciently simulating small-room acoustics[END_REF]. Even if the basic idea is rather simple, the model is able to produce the exact solution to the wave equation for a 3D shoebox with rigid walls. It models only specular re ections, ignoring di use and di racted components. It only approximates arbitrary enclosures and the late di use re ections. The implementation re ects the sound source against all surfaces in the scene, resulting in a set of image sources. Then, each of these image sources is itself re ected against all surfaces and so on, up to a prede ned order. There are two main limitations of this method. First, in a shoebox the complexity of the algorithm is cubic in the order of re ections. Therefore when a high order is required, the algorithm becomes impractical. Second, it models only the specular re ection, neglecting the di use sound eld. For these reasons, the image-source method is generally combined with a stochastic method to model the full impulse response.

H

combines the best of these two approaches. As discussed above, the image-source method is accurate for early re ections, but slow and not accurate for longer responses. The ray tracing method is by nature an approximation, but produces acceptable responses for di use elds. And in general geometric methods fail to properly model lower frequencies and room modes. The waveguide method models physical phenomena better than geometric methods, but is expensive at high frequencies. All these limitations correspond to three regions in the Time-Frequency (TF) representation of the RIR. As depicted in Figure 2.14,

• in the time domain, a transition can be identi ed between the early vs. late re ection, corresponding to the validity of the deterministic vs. stochastic models; and

• in the frequency domain, between geometric vs. wave-based modeling.

Providing these time-and frequency-domain crossover points [START_REF] Badeau | Common mathematical framework for stochastic reverberation models[END_REF]] it is possible to combine image-source, ray-tracing and wave models. identi es the moment after which re ections are so frequent that they form a continuum. After it, the sound level decays exponentially over time, as it is partially absorbed by the room surfaces at every re ection. This point can be then used to set a cross-fade between the deterministic and the stochastic process. The crossover point in the frequency domain is called Schroeder's frequency. It splits the spectrum of the RIR into a region with a few isolated modes and one denser, called respectively the resonant and even behaviors. Therefore, this point de nes the cross-fade between the geometrical and wavebased model.

Each simulator available has its own way to compute and implement this crossover points as well as mixing the results of the three methods.

The method of images and the image source model

The Method of Images is a mathematical tool for solving a certain class of di erential equations subjected to boundary conditions. By assuming the presence of a "mirrored" source, certain boundary conditions are veri ed facilitating the solution of the original problem. This methods is widely used in many elds of physics, and interestingly with speci c applications to Green's functions. Its application to acoustics was originally proposed by Allen and Berkley in [START_REF] Allen | Image method for e ciently simulating small-room acoustics[END_REF] and it is know as the Image Source Method (ISM). Now ISM is probably the most used technique for deterministic RIR simulation due to its conceptual simplicity and its exibility.

The ISM is based on purely specular re ection and it assumes that the sound energy travels around a scene in "rays". In the appendix of [START_REF] Allen | Image method for e ciently simulating small-room acoustics[END_REF], the authors also proved that this method produces a solution of the Helmholtz's equation for cuboid enclosures with rigid boundaries. The image source de nes the interaction of the propagating sound and the surface. It is based on the observation that when a ray is re ected, it spawns a secondary source "behind" the boundary surface. As show in Figure 2.16, this additional source is located on a line perpendicular to the wall, at the same distance from it as the original source, as if the original source had been "mirrored" in the surface. In this way, each wavefront that arrives to the receiver from each re ection o the walls corresponds to the direct path received from an equivalent (or image) source. The ISM makes use of the following assumptions:

• sound source and receiver are points in a cuboid enclosure;

• purely specular re ection paths between a source and a receiver;

• this process is simpli ed by assuming that sound propagates only along straight lines or rays;

• and rays are perfectly re ected at boundaries.

Finally, the RIR is found by summing the contribution from each (image) source, delayed and attenuated appropriately depending on their distance from the receiver. Therefore, in the time domain, the RIR associated to the source at position s and the receiver at x reads

h ISM (x, t | s) = R r=0 ᾱr 4π x -s r δ t - x -s r c (2.15)
where s r is the r-th image of the source and ᾱr is the total frequency-independent 25 25 Which is equivalent to consider perfectly rigid and re ective walls damping coe cient related to the r-th image. Such coe cient accounts for all the dissipation e ects encountered in the re ection path, e. g. absorption, air attention and scattering. In this thesis, we will refer to Eq. (2.15) as the time-domain echo model.

The reader should notice that the summation in the echo models of Eq. (2.15) induce an "order" among re ections indexed by r. Re ections are usually sorted for increasing Time of Arrival (TOA), τ r = xs r /c, or decreasing amplitudes, ᾱr /(4π xs r ). Alternatively, one can sort them according to their "image" generation, e. g. direct path, rst-, second-order images etc. This would require an arbitrary order within the same generation, based typically on arbitrary wall sequence. This translates into non trivial de nition of evaluation metrics for the task of estimating echoes.

In the original formulation of the ISM, ᾱ0 = 1 is assumed for the direct propagation, while for the rst order images, it coincides with the frequencyindependent surface absorption coe cient of the surface. For the subsequent orders of images, the product of all the coe cients of the surfaces encounters in the re ection path is considered.

In order to easily incorporate frequency-dependent damping e ects, the Fourier transform of Eq. (2.15) is considered instead, where each re ection term is appropriately scaled

H ISM (x, f | s) = R r=0 α r (f ) 4π x -s r exp -i2πf x -s r c , (2.16)
where now the r-th damping coe cient α r is frequency dependent. In this thesis, we will refer to Eq. (2.16) as the frequency-domain echo model.

Notice that now the damping coe cients correspond to lters, requiring Eq. (2.15) to be written as sum of (delayed) lters. This has a strong implication when modeling and estimating the RIRs as stream of Dirac impulses. Ideally they consists of scaled Diracs with well de ned time locations. The probability that two or more Diracs arrive at the same time is then very small. However, if we now assume that each re ection has a non-at frequency response, lters are observed in the time domain. Such lters have arbitrary long time-domain description and now the probability that two or more overlap is much higher.

? C ? Yes, in certain cases re ections maybe carry energy comparable or stronger than the direct contribution. This happens for instance when directional sources are directed towards reectors. Typical scenarios are when a speaker is giving a talk while facing the slides projected on a wall giving the shoulders to the microphones. Moreover, when multiple re ections arrive within a very short time di erences, their energy may be aggregated together. This e ect will be studied in detail is Chapter 5. Nevertheless, aggregating echoes is something that the human auditory system do to enhance speech intelligibility, as discussed in the next section.

2.4 P So far we have analyzed reverberation from a purely physical point of view. However in many applications it is important to correlate it to human perception. This will be important in order to de ne some acoustic parameter which will be used later in this thesis to characterize evaluation scenarios.

The perception of the RIR's elements

It is commonly accepted that the RIR components de ned in § 2.3.1 play rather separate roles in the perception of sound propagation.

T

is the delayed and attenuated version of source signal itself. It coincides with the free-eld sound propagation and, as we will see in Chapter 10, it reveals the direction of the source.

T

are re ections which are by nature highly correlated with the direct sound. They convey a sense of geometry which modi es the general perception of the sound:

• The precedence e ect occurs when two correlated sounds are perceived as a single auditory event [START_REF] Wallach | The precedence e ect in sound localization (tutorial reprint)[END_REF]]. This happens usually when they reach the listener with a delay within 5 ms to 40 ms. However, the perceived spatial location carried by the rst-arriving sound suppressing the perceived location of the lagging sound. This allows human to accurately localize the direction of the main source, even in presence of its strong re ections.

• The comb lter e ect indicates the change in timbre of the perceived sound, named coloration. This happens when multiple re ections arrive with periodic patterns and some constructive or destructive interferences arise. Such phenomena can be well modeled with a comb lter [START_REF] Barron | The subjective e ects of rst re ections in concert halls-the need for lateral re ections[END_REF]].

• Apparent source width is the audible impression of a spatially extended sound source [START_REF] Griesinger | The psychoacoustics of apparent source width, spaciousness and envelopment in performance spaces[END_REF]]. By the presence of early re ections, the perceived energy increases, providing the impression that a source is larger than its true size.

• Distance and depth perception provides the listener with cues about the source 3D location. "Distance" refers to only the range and applies to outdoor scenarios, as opposed to "depth" which relates to the indoor case.

A fundamental cue for distance perception is the Direct-to-Reverberant Ratio (DRR), i. e. the ratio between the direct path ratio and the remaining portion of the RIR (see § 2.4.4). Regarding the depth perception, early re ections are the main responsible. In the context of virtual reality, correctly modeling these quantities is essential in order to maintain a coherent depth impression [START_REF] Kearney | Distance perception in interactive virtual acoustic environments using rst and higher order ambisonic sound elds[END_REF].

T is indicative of the size of the environment and the materials within [START_REF] Välimäki | More than 50 years of arti cial reverberation[END_REF]. It produces the so-called the listener envelopment, i. e. the degree of immersion in the sound eld [START_REF] Griesinger | The psychoacoustics of apparent source width, spaciousness and envelopment in performance spaces[END_REF]]. This portion of the RIR is mainly characterized by sound di usion, which depends on the surfaces roughness.

Mixing time

Perceptually, the mixing time could be regarded as the moment when the di use tail cannot be distinguished from that of any other position or listener's orientation in the room [START_REF] Lindau | Perceptual evaluation of model-and signal-based predictors of the mixing time in binaural room impulse responses[END_REF]]. Analytically, it can be identi ed as the point dividing the early re ections from the late reverberation in a RIR. The previously cited work shows the equivalence between the two de nitions. Due to this, it is an important parameter also in the context of RIRs synthesis as it de nes cross-over point for room acoustics simulator using hybrid methods [START_REF] Savioja | Overview of geometrical room acoustic modeling techniques[END_REF](see § 2.3.2).

Reverberation time

The reverberation time measures the time that takes the sound to "fade away" after the source has ceased to emit. In order to quantify it, acoustics and audio signal processing use the Reverberation Time at 60 dB (RT 60 ), the time after which the sound energy dropped by 60 dB. It depends on the size and absorption level of the room (including obstacles), but not on the speci c positions of the source and the receiver. Real measurements of RIRs are a ected by noise. As a consequence, it is not always possible to consider a dynamic range of 60 dB, i. e. the energy gap between the direct path and the ground noise level. In this case, the RT 60 value must be approximated with other methods. By knowing the room geometry and the surfaces acoustics pro les, it is possible to use the empirical Sabine's formula:

RT 60 ≈ 0.161 V TOT l α l S l [s],
(2.17)

where V TOT is the total volume of the room [m 3 ] and α l and S l are the absorption coe cient and the area [m 2 ] of the l-th surface.

Direct-to-Reverberant ratio and the critical distance

The Direct-to-Reverberant Ratio (DRR) quanti es the power of direct against indirect sound [START_REF] Zahorik | Direct-to-reverberant energy ratio sensitivity[END_REF]]. It varies with the size and the absorption of the room, but also with the distance between the source and the receiver according to the curves depicted in Figure 2.17. The distance beyond which the power of F 2.17: DRR as a function of the RT60 and the source distance rij based on Eyring's formula (Gustafsson et al., 2003). These curves assume that there is no obstacle between the source and the microphone, so that the direct path exists. The room dimensions are the same as in Figure 3.1.

indirect sound becomes larger than that of direct sound is called the critical distance. These two quantities represent an important parameter to assert the robustness of audio signal processing methods, since they basically measure the validity of the free-eld assumption.

C

This chapter presented notions of room acoustics, de ned echoes and RIRs, which are the main protagonist of this thesis. In the following chapter, we will study these quantities under the light of audio signal processing.

Elements of Audio Signal Processing

"Everything in its right place There are two colours in my head" -Radiohead Everything In Its Right Place

S

We now move from physics to digital signal processing. At rst, in § 3.1, this chapter formalizes fundamental concepts of audio signal processing such as signals, mixtures, lters and noise in the time domain. In § 3.2, we will present a fundamental signal representation that we will use throughout the entire thesis: the Short Time Fourier Transform (STFT). Finally, in § 3.3, some essential models of the RIR are described.

Unless speci ed, the notation and de nitions presented in this chapter for the audio signal model are excerpted from Vincent et al.'s book Audio source separation and speech enhancement. The material used for illustrating concepts of digital signal processing are taken from standard books on the topics.

S

In the previous chapter we formalized the physics that rule the sound propagation from the source to the microphone. A raw audio signal encodes the variation of pressure over time on the microphone membrane. Mathematically it is denoted as the function that the sampling process involves two steps: rst, the impinging signal undergoes an ideal low-pass lter φLP with frequency support in ] -Fs /2, Fs /2]; 26 then it time-support is regularly discretized, t = n/F s for n ∈ Z. This is 26 The ideal low-pass lter is φLP(t) = sinc(t) = sin(πFst)/(πFst). The term sinc stands for sinus cardinal and was introduced in [START_REF] Woodward | Information theory and inverse probability in telecommunication[END_REF] in 1952, in which he said that the function "occurs so often in Fourier analysis and its applications that it does seem to merit some notation of its own" expressed by

x : R → R t → x(t), ( 3
x[n] = φLP x n F s ∈ R, (3.2)
where is the continuous-time convolution operator. This will restrict the frequency support of signal to satisfy the Nyquist-Shannon sampling theorem and avoid aliasing e ect.

Finally, at the end of the discretisation process, the x(t) is represented as the nite time series or a vector, Point sources are single and well-de ned points in the space emitting single-channel signal. In certain application, human speakers or the sound emitted by a loudspeaker can be reasonably modeled as in this way. Di use sources refers for instance to wind, tra c noise, or large musical instruments, which emit sound in a large region of space. Their sound cannot be associate to a punctual source, but rather a distributed collection of them.

x N ∈ R N , ( 3

D

. O . An omnidirectional source (resp. receiver) will in principle emit (resp. record) sound equally from all directions, both in time and in frequency. Although this greatly simpli es the models, it is not true in real scenario, where the physical properties of sources (resp. receivers) leads to frequency-dependent directivity patterns. In this thesis we will assume omnidirectional sources and receivers.

The mixing process

Let us assume the observed signal has I channels indexed by i ∈ {1, . . . , I}. Let us assume that there are J sources indexed by j ∈ {1, . . . , J}. Each microphone i and each source j have a well de ned position in the space, x i , s j , respectively. The mixing process describes then the nature of the mixtures. In order to better formalized it, in source separation (e. g. in [START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF]) it is common to use an intermediate representation called source spatial images: cij (t) describes the contribution of source j to microphone i. Consequently, the mixture xj is the combination of images associated to the source j. This is illustrated in Figure 3.2. Depending on the "contribution" the image describes, the following type of mixture can be de ned:

N

. A M . The former refers to microphone mixtures recorded simultaneously the same auditory scene, e. g. teleconferencing systems or hands-free devices. By contrast, the latters are created by mixing together di erent individual, possibly processed, recordings. This are the typical mixtures used professional music production where the usage of long-chain of audio e ects typically "hide", willingly or not, the recording environment of the sound sources.

I

. C M . In the rst case, the mixing process boils down to a simple linear combination of the source signals, namely the mixing lters are just scalar factors. This is the typical scenario when sources are mixed using a mixing console. Convolutive mixtures, instead,

instantaneous cij = α ij s(t) anechoic cij = α ij sj (t -τ ij ) convolutive cij = ( hij sj )(t)
T 3.1: Taxonomy of linear mixing models for a mixture channel x i , sources s j , impulse response hij (t), scaling factor α ij and delay τ ij . Notice that the anechoic cases is a particular case of the convolutive one, where

hij (t) = α ij δ(t -τ ij )
denote the more general case where the each mixture is the sum of ltered signals. In between are the anechoic mixtures involving the sum of scaled and delayed source signals. Natural mixtures are convolutive by nature and free-far-eld natural recording are well approximated by anechoic mixtures.

In this thesis, we will particularly focus on natural mixture. The microphones listen to the propagation of sound in the room and this process is linear (cf. § 2.1) and time invariant provided a static scenario. Therefore, the resulting mixture is the simple summation of the sound images, which are the collections of convolution between the RIRs and source signals:

cij (t) = hij sj (t) (3.4) cj (t) = [c 1j (t), . . . , cIj (t)] x(t) = J j=1 cj (t). (3.5)
Considering the time domain description of the RIR derived (and approximated) in the previous chapter, the time-domain mixing lters hij (t) will be modeled as follows:

hij (t) = R r=0 α (r) ij 4πcτ (r) ij δ t -τ (r) ij + εij (t) (3.6)
where α r ij ∈ R and τ r ij ∈ R are the attenuation coe cient and the time delay of the re ection r. The noise term εij (t) collects later echoes (r > R) and the tail of the reverberation. We do not assume εij (t) to be known.

Noise, interferer and errors

In Eq. (3.5) no noise is included: all the sources are treated in the same way, including target, interfering and noise sources. While the de nition of target sound source is quite self-explanatory (and will denoted by default as the rst source, that is j = 1), the term interfer and noise depends on the speci c use case, problem, application, and research eld. Notice that in Eq. (3.6) a noise term is added to gather unknown quantities.

Noise is a general term for unwanted (and, in general, unknown) modi cations that a signal may su er during capture, storage, transmission, processing, or conversion [START_REF] Tuzlukov | Signal processing noise[END_REF]].

Therefore, we will de ne and use the following type of noises:

s1(t) s2(t) n(t) x(t) ε(t)

I

identi es undesired sources with properties similar to the target source. For instance, a concurrent speech source for speech application or concurrent music instrument in case of music. Later, in this thesis the interfer sources will be denoted as additional sources indexed by j > 1.

N collects all the remaining e ects, typically nonspeech sources. Moreover we will make a further distinction between the followings.

• Di use Noise Field describes the background di use sources present in the auditory scene, e. g. car noise, indistinct talking or winds. It can be recorded or approximated as Additive White Gaussian Noise (AWGN) with a speci c spatial description as described in [Habets and Gannot 2007].

• Measurement and Model Noise accounts for general residual mis-modeling error. In this manuscript, it will denoted as εij (t) and will be modeled as AWGN. Examples of this term are the error due to approximation of the RIR with the Image Source Method (ISM) or sensor noise, respectively.

By making the noisy terms explicit, the mixing model in Eqs. (3.4) and (3.5) writes:

cij (t) = hij sj (t) + εij (t) (3.7) cj (t) = [c 1j (t), . . . , cIj (t)] x(t) = J j=1 cj (t) + ñ(t) (3.8)
Note that the noise term εij of Eq. (3.7) does not correspond to the one of Eq. (3.6), but includes it. Thus, as not to weigh down the notation, we will use the same notation for both.

S

The frequency analysis was introduced by Joseph Fourier in his work on the heat equation [START_REF] Fourier | Théorie analytique de la chaleur[END_REF]]. His mathematical tool, named later Fourier Decomposition, aims at approximating any signal by a sum of sine and cosine waves.

The frequency, or spectral, representation is probably the most famous signal representation used in signal processing: Speech and music signals naturally exhibit pseudo-periodic behaviors (on well chosen time scale). In this domain, they are described as combination of sinusoids as function of their frequencies. This operation is achieved by the operator Fourier Transform (FT), F , which projects a continuous-time-domain square-integrable signal x onto a space spanned by continuous-frequency complex exponentials (see Figure 3.5):

X(f ) = (F x)(f ) = +∞ -∞ x(t)e -i2πf t dt ∈ C, (3.9)
where f ∈ R are the natural frequency in Hz and i is the imaginary unit.

Time Frequency F 3.5: A signal resolved into its Fourier series: a linear combination of sines and cosines represented as peaks in the frequency domain. Adapted from the online tikz example.

Apart from providing a space where an audio signal reveals its harmonic structure, the Fourier transform bene ts from two fundamental properties: it is linear and it converts time-domain convolution into element-wise product. First, linearity allows to write Eq. (3.5) simply as:

x(t) = J j=1 cj (t) F → X(f ) = J j=1 Cj (f ) (3.10)
Secondly, by the convolution theorem, the source spatial images in Eq. (3.4) writes as:

cij (t) = hij sj (t) F → Cij (f ) = Hij (f ) Sj (f ). (3.11)
As discussed in Chapter 2, assuming a pure echo model, the FT of RIR, a. k. a. the Room Transfer Function (RTF), can be computed exactly in closed-form as

Hij (f ) = R r=0 α (r) ij 4πcτ (r) ij e -i2πf τ (r) ij .
(3.12)

In practice, the lters hij are neither available in the continuous time domain nor in the continuous frequency domain directly. They must be estimated from the observation of the discrete-time mixtures x i [n], therefore, after the convolution with a source and the measurement process. In practice, we don't have access to continuous signals, neither in time nor in frequency domain.

Every signal the microphones capture and the related spectra are represented by nite-and discrete-time signals for which the properties (3.11) are valid only with some precautions.

Discrete time and frequency domains

In case of discrete-time signal, x[n] with n ∈ Z and frequency support in [ -Fs /2, -Fs /2], its spectral representation is given by the (forward) Discrete-Time Fourier Transform (DTFT), F Fs :

XFs (f ) = (F Fs x)(f ) = +∞ n=-∞ x[n]e -i2πf n/F s , (3.13)
which is a continuous function of f with period F s . Notice that the term "discrete-time" refers to the fact that the transform operates on discrete signal.

In case of uniformly spaced samples, it produces a function of continuous frequency that is a periodic summation of the continuous Fourier transform of the original continuous function. Under certain theoretical conditions, described by the sampling theorem, both the original continuous signal x and its sampled version x can be recovered perfectly from the DTFT. The analogous convolution theorem for discrete-time domain signals s

[n] and h[n] is c[n] = (h s)[n] FF s → CFs (f ) = HFs (f ) SFs (f ), (3.14)
where the dependency on the source and microphone index have been omitted for clarity.

The DTFT itself is a continuous function of frequency which require in nite discrete value to be computed. For these two reason, it is not accessible in practice or computed in the digital domain. Therefore the following representation is used instead.

For a discrete-and nite-time signal x N , the spectral representation is given by its (forward) Discrete Fourier Transform (DFT)27 , F :

X F [k] = (F x N )[k] = N -1 n=0 x N [n]e -i2πkn/F . (3.15)
where k ∈ [0, F -1] is the discrete frequency bin and F is the total number of bins. Again we use the subscript F and the brackets [k] to stress the nite and discrete frequency support of the DFT. Note that now the sampling frequency term disappeared from the de nition of the operators and, in general, operates directly on "sequences". Therefore its link with the continuous FT for continuous-time domain signal is not trivial and it will be discussed in the following section. Similarly to the FT and the DTFT, the convolution theorem can be de ned for the DFT as well. However, it comes with following important modi cation. Let be two nite-and discrete-time domain periodic signals ... s N and ... hN of period N . Here ...

• is used to denote periodicity. Let be S F and H F their DFT, respectively, then the convolution theorem for the DFT writes

... c N [n] = ( ... hN ... s N )[n] F → C F [k] = H F [k]S F [k], (3.16)
where denotes the circular convolution 28 . Note that now the results of the 28 The nite-time circular convolution for two vectors u

N , v N ∈ R N is the ... q N = (u N v N )[n] = N -1 m=0 u N [m] ... v N [n-m], where ... v N is the periodic version of v N and ... q N is periodic. convolution, ... c N [n]
, is periodic with period N .

3.2.2

The DFT as approximation of the FT An important application of the DFT is to approximate numerically the FT. As mentioned at the beginning of the chapter, with the discretisation process the continuous signal is periodically sampled, low-passed and nally truncated. It can be proved that sampling in the time domain corresponds to periodizing the signal spectrum with period equal to the sampling frequency. In order to avoid aliasing artifact, it is common to limit the signal bandwidth before the sampling (and using the lowpass lter in Eq. (3.2)). By assuming sampling at rate F s , in the continuous-frequency domain the spectrum X(f ) is repeated every intervals of size F s Hz . By further assuming that the signal undergoes an ideal low-pass lter, no spectral leakage is presents between each repetition.

So far, the sampled time domain signal, x[n], is mapped to the continuous frequency domain X(f ). This particular case of the FT is called Discrete-Time Fourier Transform (DTFT) and it is denote with XFs

[k]. X(f ) = +∞ -∞ x(t)e -i2πf t dt → XFs (f ) = ∞ n=-∞ x[n]e -i2πf n Fs .
(3.17) Here the continuous integral the FT is approximated by Riemann sum over the discrete points n ∈ Z: To be more rigorous, when computing a Riemann sum approximation, the length of the discretisation interval multiply the summation. In our application, this quantity always set to F s and for readability reason such term is dropped. The quality of this approximation w. r. t. the original continuous spectrum is regulated by the choice of F s : the higher F s , the better the approximation. The lower bound to the possible value F s is the results known as the Nyquist-Shannon's sampling theorem.

Furthermore, we consider only the nite sequence x N consisting of N samples. This would reduce the summation ranges the right part of Eq. (3.17). Instead, we can keep the in nite summation by multiplying the sampled signal by a discrete-time window function w selecting the non-zero porting of x,

x N [n] = w[n]x[n]
. By the convolution theorem, the multiplication in the time domain translates in a convolution between the corresponding spectra. As a consequence, the spectrum of the truncated signal is distorted by the spectrum of the window function. In math,

XFs (f ) = ∞ n=-∞ x[n]w[n]e -i2πf n Fs ↔ XN (f ) = N -1 n=0 x N [n]e -i2πf n Fs .
(3.18) By the convolution theorem, we have that

x N [n] = x[n]w[n] ↔ XN (f ) = ( XFs WFs )(f ) (3.19)
where WFs is the DTFT of the sampled window function w [n].

Assuming the window function to be an ideal rectangular function, its DTFT is a ideal low-pass lter, which acts on the original spectrum as a smoothing function. As a consequence, the quality of this approximation is then based on the spectral leakage of the chosen window function, w[n]. As a rule of thumb, here the longer the segment, the better the approximation29

Finally, we cannot access the DTFT directly because that involves an in nite number of frequencies f ∈ R. Therefore, taking F uniformly-spaced frequency f k ∈ R as in Eq. (3.21), we nally obtain the DFT as in Eq. (3.15), that is

XN (f k ) = N -1 n=0 x N [n]e -i2πf n Fs ↔ X F [k] = N -1 n=0 x N [n]e -i2πkn/F .
(3.20) The natural frequency f k in Hz corresponding to the k-th frequency bin can be then computed as

f k = k F F s . (3.21)
Notice that F s term disappeared in the right part of the equation above as it cancels out when using Eq. (3.21). By increasing F , we can sample more densely X F [k] which leads to a better approximation to XN . However this does not eliminate the distortion of the previous steps, due to WFs .

Again, we sampled a domain. Thus, according to the de ned sampling process, this involve using a ideal low-pass lter. This lter acts now on the discrete spectrum, smoothing it and limiting the support of its transformation in the dual domain. Therefore, the inverse DFT of

X F [k] is not properly x N [n],
but its periodic version repeated every F samples. In fact, sampling in one of the two domain is equivalent to a periodization in the other domain while truncating lead to convolving with a window function. Moreover, the chain of operation (sampling in time and truncation in time and sampling in frequencies) are valid in both way. Thus one can arbitrarily rst sample and truncate frequency domain and nally sample in time. The only di erence is in the interpretation of the windowing function, which in one case smooth the spectrum and in the other smooth the signal. All this relation and approximation that connects the FT to the DFT are well explained in this explanatory material Poisson Summation Formula, Revisited or in many classic books of digital signal processing, such as [START_REF] Oppenheim | Signals and Systems: An Introduction to Analog and Digital Signal Processing[END_REF]].

Signal models in the discrete Fourier domain

Conscious of the above approximations, we can now rewrite our signal model for the discrete case. Hereafter we will always consider nite-length sequences and the index N will be dropped to lighten the notation.

The DFT is linear, so the discrete version of Eq. (3.10) becomes

x[n] = J j=1 c j [n] F → X[k] = J j=1 C j [k] (3.22)
Secondly, by using naïvely the discrete convolution theorem, one could translate Eq. (3.4) as

c ij [n] = (h ij * s)[n] F → C ij [k] = H ij [k]S[k], (3.23)
where * is the nite-time linear convolution operator 30 . In general, this relation 30 The nite-time linear convolution for two

vectors u ∈ R L and v ∈ R D is (u * v)[n] = L-1 l=0 u[l]v[L -1 + n -l] for n = 0, • • • , D -L .
is wrong and the following paragraph will provide further insight on why is used in practice.

The lter H ij [k] is the DFT of the room impulse response. As mentioned in the § 3.2.2, this just approximates the true RTF de ned in the previous chapter. We recall that two approximation are involved: one concern that we model only a nite number of specular re ections; one concern sampling its FT. Nevertheless, we can use the closed-form frequency-domain echo model Eq. (2.16) to write the RTF's DFT in closed-form by taking equally spaced frequencies, that is

H ij [k] = R r=0 α r ij 4πcτ r ij e -i2πkFsτ r ij /F . (3.24)
A , the model (3.23) makes use of other approximations that are worth presenting. In particular, the work by [START_REF] Tukuljac | MULAN: a blind and o -grid method for multichannel echo retrieval[END_REF] properly discuss them in the context of the echo estimation problem. The paper mention three approximations, which are depicted in the following diagram. The diagram shows a chain of operators (sampling and transforms)

c(t) = h s (t) C(f ) = H(f ) S(f ) Prop. in (i) truncation (ii) c N [n] = (h N * s N )[n] CF s (f ) = HF s (f ) SF s (f ) ĉ[n] = φLP h s n Fs C F [k] = H F [k]S F [k] c[n] ≈ (h * s)[n]
F Fs sampling CF (f k ) = HF s (f k ) SF s (f k ) Continuous Discrete c N [n] ≈ (h N s N )[n] h N * s N ≈ h N s N (ii) F closed-form echo model (iii)
Eq. (3.23)

with provable and approximated equivalences that lead to Eq. (3.23) used in practice. In order, (i) In [van denBoomgaard and van derWeij 2001, Proposition 2] 31 the au-thors show that if the signal s(t) is band-limited by F s , then sampling the continuous convolution is exactly equivalent to linearly convolving the in nite discrete signal s[n] and the discrete and low-passed version of the lter. However, while the source signal is band-limited by nature, the true RIR h * (t) is not (in fact the RIR is modeled as a nite summation of spikes, which has spectrum with in nite support). Thus, the rst approximation

(i) considers h[n] ≈ ( φLP * h * )[n]
, in words we assume that the lter is band-limited by ± Fs /2.

(ii) As introduced in § 3.2.1, the discrete-time convolution theorem applies to the circular convolution involving periodic signals. This can be approximated by the linear convolution, that is

(h ŝ)[n] ≈ (h * ŝ)[n]
under some assumption. In particular such an approximation is reasonably good when many samples are available and when one of the two signals is quasi-periodic, which are typical cases for audio signals.

(iii) The third approximation regards the closed-form of the RTF, H * (f ).

Here we make two strong assumptions: rst, the RTF follows the echo model discussed in Eq. (2.16); second, we used its closed-from DTFT (Eq. (3.24)) at some frequency f k as its DFT. In order to be computed exactly, this would require in nitely many samples and unlimited frequency support, which are not possible in practice.

Nevertheless, it is important to notice that approximations (ii) and (iii) become arbitrarily precise as the number of samples N grows to in nity. First, the DFT approaches the DTFT as N → ∞ due to their de nition (see § 3.2.1). Second, while the convolution theorem is exact for the DTFT, for the DFT it is only for considering the circular convolution on a nite-time signal (or, equivalently, considering periodic signals). Therefore, it is easy to notice that that when the period becomes in nite, the circular convolution and the linear convolution become arbitrarily close. In the approximation (i), limiting the RIR's spectrum with an ideal lowpass lter introduces smearing in the time-domain. In particular, the true peaks in the RIR do not necessarily correspond to the true echoes as shown in [START_REF] Tukuljac | MULAN: a blind and o -grid method for multichannel echo retrieval[END_REF] an important observation in this regard: even if in nite number of samples are available, after the measurement process, the discrete-time lter h[n] consists of in nite-length decimated combinations of sinc functions. In the context of this thesis, this observation tell us that even in ideal conditions, that is without noise, possibly knowing the transmitted signal, and processing in nitely many samples, the exact estimation of the echo properties of the RIR is a challenging task itself. This is a fundamental di erence between RIR estimation and estimating the time of arrivals of the early echoes. Note, for instance, that we wrote the echo model only in the continuous-time domain or with its closed-form form discrete frequencies. The discrete-time domain was avoided on purpose since the echoes' arrival time are naturally o the sampling grid, namely they are not integer multiples of F s . This fundamental limitation is subject of research in the eld of super-resolution and it will discussed in Chapter 5.

While the raw audio signal encodes the amplitude of a sound as a function of time, its spectrum represents it as a function of frequency. In order to jointly account for both temporal and spectral characteristic, joint time-frequency representations are used.

Time-Frequency domain representation

Time-Frequency (TF) representations aim to jointly describe the signal in the time and frequency domains. Instead of considering the entire signal, the main idea is to consider only a small section of the signal. To this end, one xes a so-called window function, w N [n], which is nonzero for only a period of time L win shorter than the entire signal length, L win N . This function iteratively shifts and multiplies the original signal, producing consecutive frames. Finally, the frequency information are extracted independently from each frame. The choice of a window function w[n] depends on the application since its contribution re ects in the TF representation together with the one of the signal.

The discrete STFT is the most commonly used TF-representation in audio

The STFT was introduced by Dennis Gabor in 1946, the person behind Holography.

signal processing. This representation encodes the time-varying spectra into a matrix X[k, l] ∈ C F,T with frequency index k and time frame index l. More formally, the process to compute the complex STFT coe cients is given by

X[k, l] = Lwin-1 n=0 w[n]x[n + lL hop ]e -i2πkn/F ∈ C (3.25)
where L win is the window length and L hop is the hop size which speci es how much the window needs to be shifted across the signal. Equivalently, Eq. (3.25) can be expressed as DFTs of windowed frames, For more mathematical detailed description on DFT and STFT can be found in [START_REF] Oppenheim | Signals and Systems: An Introduction to Analog and Digital Signal Processing[END_REF]]. For a audio-processing-oriented and music-processing-oriented explanation please refer to Chapter 2 of [START_REF] Vincent | Audio source separation and speech enhancement[END_REF]] (Chap-ter2) and Chapter 2 of [Müller 2015], respectively.

X[k, l] = F x[n, l] where x[n, l] = x[n + lL hop ]w[n]. Since each STFT coe cient X[k, l]
whose axes represent time frames and frequency bins. In this image, the (log) value |X[k, l]| 2 is represented by the intensity or color in the image at the coordinate [k, l]. Throughout this work processing will be typically conducted in the STFT domain, unless speci ed. This is a common approach in the audio signal processing community, but it is not the only one: many algorithm are designed directly in the time domain or in alternatives TF representation, e. g., Mel-Scale, Filter-Banks.

As discussed [START_REF] Vincent | Audio source separation and speech enhancement[END_REF], the STFT has the following useful properties for audio processing:

• the frequency f k is a linear function of the frequency bin k;

• the resulting matrix allows the computation of the phase ∠ X[k, l], the magnitude |X[k, l]| and the power |X[k, l]| 2 ;

• the DFT can be e cienciently computed with the Fast Fourier Transform (FFT) algorithm;

• the process to invert the STFT is relatively easy (using the inverse of the DFT, overlap-add method with proper window functions);

• the STFT inherits the linearity and convolution property of the DFT under some condition about the length of the signals.

3.2.5

The nal model

The model (3.23) shows how in practice the RIRs are treated in the frequencydomain. However this does not generalize straightforwardly to the timefrequency domain: it depends on the length of the lter w. r. t. to the length of the analysis window on of the STFT. To circumvent this issue, the convolutional STFT 32 for arbitrary window functions have been proposed [Gilloire and 32 It translates the time-domain convolution into inter-frame and inter-band convolutions, rather than pointwise multiplication of Fourier transforms. [START_REF] Gilloire | Adaptive ltering in sub-bands with critical sampling: analysis, experiments, and application to acoustic echo cancellation[END_REF]]. Although it leads to better approximation than the one of product, it is computationally and memory intensive. The exact e ect of the time-domain convolution in the STFT would be modeled by cross-band lters [START_REF] Avargel | System identi cation in the short-time Fourier transform domain with crossband ltering[END_REF]] performing a double convolution along time and frequency axes. 33 However, these models are not investigated in this thesis work.

In this thesis, we will consider moderated reverberant scenario. This re ects in long room impulse responses, which would require to long analysis window. Instead, as we are interested in only the early echoes, we can consider lters shorter than typical window length used in speech processing (e. g. 64 ms). In the literature, this is known as the narrowband approximation, namely the time-domain ltering can be approximated by complex-valued multiplication in each time-frequency bin [l, k]:

C j [l, k] ≈ H[k]S j [l, k], (3.26)
where the

H j [k] = [h 1j [k], • • • , h Ij [k]
] is the I × 1 vector of the RTFs for source j. While this is a resonable approximation for short lters, it gets worsens as the lter length grows.

It is sometimes practical to concatenate all these vectors into an

I × J matrix H[k] = [H 1 (f ), • • • , H J (f )] called mixing matrix.
With the above notation and considerations, mixing process including noise terms can be written in the STFT domain compactly as:

X[l, k] = H[k]S[l, k] + U [l, k] (3.27)
where

U [l, k] = N [l, k] + ε(l, k)
includes the contribution of both di use noise sources, modeling and measurement errors.

( ) 41

O ( )

RIRs are complicated quantities to model, compute and estimate. The representations of the RIR discussed so far explicitly models early echoes and reverberation. Besides, alternative models are common in the audio processing literature.

Steering vector model

In the absence of echoes and reverberation, namely assuming free-eld propagation, the RIRs simplify to steering vectors, namely the DFT of Eq. (2.9):

D j [k] = 1 4πq 1j e -i2πf k q1j /c , • • • , 1 4πq Ij e -i2πf k q Ij /c , (3.28)
Furthermore, assuming far-eld regimes, the microphone-to-source distance q ij is larger than the inter-microphone distance d ii making the attenuation factors 1 /4πqij approximately equal, hence ignored.

Relative transfer function and interchannel models

Let us consider now only two channels and only one source signal in the model Eq. (3.27). Dropping the dependency on j for readability and taking the rst channel as reference, the Relative Transfer Function (ReTF) 34 associated 34 In the literature of spatial ltering, the Relative Transfer Function is typically denoted as RTF, which collides with the notation proposed in the thesis. We reminds here that we use the following notation: Room Transfer Function (RTF) and Relative Transfer Function (ReTF).

to the i-th channel is de ned as the element-wise ratio of the (D)FTs of the two lters [START_REF] Gannot | Signal enhancement using beamforming and nonstationarity with applications to speech[END_REF]]

G i [k] = H i [k] H 1 [k] . (3.29)
The continuous-time domain counterpart is called as Relative Impulse Response (ReIR) and can be interpreted as the lter "transforming" the impulse response of the reference channel into the i-th one. Considering the observation xi and x1 of a single noisy and reverberant source, their signals can be re-written in term of gi as follows

   x1 = h1 s + ũ1 xi = hi s + ũi →    x1 = h1 s + ũ1 xi = gi h1 s + ũi . (3.30)
Notice that hi = gi h1 corresponds to Eq. (3.29) in the frequency domain.

Moreover although the real-world RIRs h 1 and h i are causal, their ReTF needs not be so.

The ReTF bene ts of several interesting properties that will be of fundamental importance for this thesis. In particular:

• the ReTF associated to the reference channel (i = 1) is equal to 1 for each frequency bin k;

• the problem of estimating the ReTF can be considered "easier" than RIRs estimation; In fact, in the noiseless case, it holds that xi = gi x1 . Since xi and x1 are both available, the ReTF estimation boils down to transfer function estimation with known input signal;

• in the literature, many e cient and robust methods have been proposed to estimate ReTF [START_REF] Gannot | Signal enhancement using beamforming and nonstationarity with applications to speech[END_REF][START_REF] Doclo | GSVD-based optimal ltering for single and multimicrophone speech enhancement[END_REF][START_REF] Markovich | Multichannel eigenspace beamforming in a reverberant noisy environment with multiple interfering speech signals[END_REF]Koldovsky and Tichavsky 2015;[START_REF] Kodrasi | EVD-based multi-channel dereverberation of a moving speaker using di erent RETF estimation methods[END_REF][START_REF] Tammen | Iterative Alternating Least-Squares Approach to Jointly Estimate the RETFs and the Di use PSD[END_REF]];

• a RIR can be seen as a special case of ReTF where the non-reference microphone is a virtual one whose output is the original (non-spatial) source signal s. In fact, if h 1 = δ then g i = h i ; 35 35 In practice this virtual microphone is sometimes substituted by a microphone that is very close to the source.

• as discussed below, ReTF simplify to special steering vectors in free-and far-eld conditions, which have interesting geometrical properties;

• the ReTF encodes acoustics properties of the related RIRs and is independent of the source signal. They can be used as features for the estimation of parameters of the acoustic propagation. Examples of this are many works in the Sound Source Localization domain (e. g., [START_REF] Laufer | Relative transfer function modeling for supervised source localization[END_REF]Deleforge et al. 2015a;Di Carlo et al. 2019].

Under the narrowband approximation, the ReTF can be approximated by the ratio of the DFTs of the channel signal

X i [k] and X 1 [k]. G i [k] ≈ X i [k] X 1 [k] ≈ H i [k]S[k] H 1 [k]S[k] = H i [k] H 1 [k] . (3.31)
It can be noticed that ideally, the ReTF removes the dependency from the source signal. However, this is true only in theory. In practice, the presence of external noise and the spectral sparsity of particular source signals (e. g., speech, and music) make the ratio diverging from the real ReTF. We will refer to the ReTFs computed with this vanilla approach as instantaneous ReTF. A way to improve the estimation of the ReTF is to perform the above operation in the STFT and then consider the average over multiple time frames. In this way, the nonstationarity and the spectral sparsity of the speech signals are alleviated. This approach is typically performed for computing the acoustic features known as interchannel cues, described below. More advanced estimators for the ReTF have been proposed in the literature and will be discussed in § 4.3.3. The reader can refer also to ?? for common and state of the art methods for ReTF estimation.

In the general case of multiple microphone arrays (I > 2) and multiple sources, the vector of ReTF

G j [k] = [G 1j [k], • • • , G Ij [k]] for the j-th source is de ned as G j [k] = 1 H 1j [k] H j [k]. (3.32)
T are obtained by combining Eqs. (3.28) and (3.29) as

D j [k] = 1, e -i2πf k (q2j -q i j )/c , . . . , e -i2πf k (q Ij -q i j )/c (3.33)
where (q ijq 1j )/c is the Time Di erence of Arrival (TDOA) between the i-th and the reference microphones. The TDOAs will be the protagonists of Chapter 10 as they are fundamental quantities for sound source localization. [dB] 0 1000 2000 3000 4000 5000 6000 7000 8000

Frequency [Hz] 2 0 2 IPD2[k]
[rad] F 3.8: (Top) Synthetic RIR for di erent absorption conditions for the reference channel 1. (Below) ILD and IPD related the too channel 2 computed with respect to channel 1. Orange lines denote the theoretical far-and free-eld ILD and IPD as de ned by the relative steering vectors of Eq. (3.33). As the source is placed far from the microphones, the attenuation is similar on both the channels, whereas the delay is not.

I [START_REF] Bregman | Auditory scene analysis[END_REF]] and Computational Auditory Scene Analysis (CASA) [START_REF] Brown | Separation of speech by computational auditory scene analysis[END_REF], the ReTF is related to the interchannel cues 36 . In fact, the ReTF encodes the 36 sometimes refers to as interaural cues when a stress is put on the fact that the two ears are considered as receivers so-called Interchannel Level Di erence (ILD) and the Interchannel Phase Difference (IPD)

ILD ij [k] = 20 log 10 |G[k]| [dB] IPD ij [k] = ∠ G[k] [rad] (3.34)
As shown in Figure 3.8, the ILD and the IPD cluster around the direct path, associated to the direct path component. However early echoes and reverberation make them signi cantly diverge.

C

In this chapter, we presented the mathematical notions, notation, and tools to model audio scene analysis problems in signal processing. We particularly focused on the crucial approximations and assumptions that arise from the merging of physical and signal processing models. As we conclude the part of the thesis dedicated to background knowledge, we will now focus on estimating acoustic echoes in Part II, and how to use them for solving audio scene analysis problems in Part III.

S

This chapter aims to provide the reader with knowledge in the state-of-the-art of Acoustic Echo Retrieval (AER). After presenting the AER problem in § 4.1, the chapter is divided into three main sections: § 4.2 de nes the categories of methods according to which the literature can be clustered and analyzed in details later in § 4.3. Finally, in § 4.4 some datasets and evaluation metrics for AER are presented.

P

As discussed in § 3.1.1, the continuous-time multi-channel signal model for one source signal s(t) and I channels writes, for each channel i,

xi (t) = ( hi s)(t) + ñi (t) hi (t) = R r=0 α (r) i δ(t -τ (r) i ) + εi (t), (4.1)
where hi (t) is the echo model for the RIR between the source and the i-th microphone. The sum comprises the line-of-sight propagation (direct path) and the earliest R echoes we want to account for, while the error term εi (t) collects later echoes and the reverberation tail. This model can be generalized to frequency dependent attention, but it will be not addressed here.

T A E R (AER) in estimating the echoes' timings {τ (r) i } i,r and attenuations (or gains) {α (r) i } i,r from multichannel observations xi of Eq. (4.1). Depending on the eld of research, the echoes' timings are also known as time delays, Time of Arrival TOA, or locations. The term AER is not an established name for this problem and, depending on the eld of research and the prior knowledge available, it can be referred to with di erent names. In fact AER can be seen as a generalization of TOAs estimation, or an instance of acoustic channel estimation (or shaping), spike retrieval and onset detection (see following § 4.3.1). As opposed to AER, the task of TOAs estimation is only focused in estimating the echoes' timings {τ (r) i } i,r . The only knowledge of TOAs is su cient for typical applications related to Sound Source Localization (SSL) [Ribeiro et al. 2010a] and Room Geometry Estimation (RooGE) [START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF]]. Moreover, knowing {τ (r) i } i,r , the attenuations {α (r) i } i,r can be estimated in closed-form as showed in [Condat and Hirabayashi 2015]. 49 TOAs estimation is sometimes called time delays estimation, when the origin of time is taken w. r. t. the rst TOA and not to the time of emission. Hereafter we will make explicit the distinction between the two. AER may be confused with the acoustic echo cancellation (AEC) problem in telecommunication and telephony which refers to the problem of estimating and suppressing feedbacks due to loudspeakers being too close to microphones. 37 37 AEC is a problem of very di erent nature, amounting to suppress from a mixture a known signal distorted by an unknown lter. This will not be further discussed in this thesis 4.2 T A .

. Active methods assume active scenarios, namely, they use one or more loudspeakers to probe the environment and one or more microphones to record the propagated probe sound. Therefore, they assume that the source reference signal is known. They falls into the broader category of deconvolution problems since a "clean" reference signal is used to deconvolve the observed one. The main advantages of these approaches are twofold. First, provided a proper probe signal, a good estimation of the RIR can be achieved. Second, these methods can be used on single-channel recordings. Instead, passive approaches use sets of passive sensors to record the sound eld. To decouple the environment from the source, they rely either on prior knowledge about the source signal or by comparing the signals received at two (or more) spatially-separated microphones. When no prior information is available, they can be seen as a blind problem and are far more challenging. In the literature, the term blind is used as opposed to informed, accounting the amount of prior information available. With this taxonomy, the above active methods are informed by the exact knowledge of the reference signal. On the contrary, passive methods need to rely on other general type of information such as sparsity, nonnegativity, etc. In general the labels "active vs. passive" apply to the scenario, while "informed vs. blind" to the problem. Passive scenarios are more common in real applications and are sometimes preferable to active ones as they are non-intrusive since only already existing sounds are used in the estimation. Therefore, a great deal of e orts has been devoted to these problems and research is still active in topic.

RIR

. RIR . RIR-based methods estimate the echoes' properties after estimating the (full or partial) RIR(s). By modeling the early part of the RIR as in Eq. (4.1), solving the AER problem can be seen as solving two sequential tasks: RIR estimation followed by echo extraction. The former can be seen as an instance of channel estimation (a. k. a. system identi cation) problem, while the latter as a spike retrieval, peak picking or onset detection problem. Other methods estimate the RIRs partially, using assumptions derived by the application. It is the case of impulse response shaping or shortening. In the context of room acoustics, the aim is to reduce the late reverberation to enhance a few early re ections which are perceptually useful.

RIR-agnostic methods, instead, try to overcome the challenging tasks of estimating the full acoustic channels and tuning peak-picking methods. They attempt to estimate echo properties directly in the parameter space of echos' TOAs and amplitudes. Given the above categories, we can now review the AER methods presented in the literature. is typically modeled as a deconvolution problem whose performance depends on the type of transmitted signal. When the transmitted signal can be chosen, several methods were developed to measure real RIRs. Since the RIR corresponds to the room response to a perfect impulse, one can measure it by producing an impulse sound, e. g. a clap, piercing a balloon, or a gun shot. Even though these methods are commonly used, they show clear limitations in terms of reproducibility and safety. Instead, modern computational techniques are used, involving the computation of a deconvolution or cross-correlation38 between the known emitted signal and the recorded output. The Minimum Length Sequence (MLS) technique was rst proposed in [START_REF] Schroeder | Integrated-impulse method measuring sound decay without using impulses[END_REF]] and it is based on the excitation of the acoustical space by a periodic pseudo-random signal, called MLS. The RIR is then calculated by circular correlation between the measured output and the original MLS signal. This method was further improved in order to achieve better RIR estimation in [START_REF] Aoshima | Computer-generated pulse signal applied for sound measurement[END_REF][START_REF] Dunn | Distortion immunity of MLS-derived impulse response measurements[END_REF]. Unfortunately this technique introduces several artifacts which yield spurious peaks in the estimation. Moreover, it is sensible to harmonic distortions generally introduced by playback devices, e. g., the loudspeakers. To overcome these issues, the Exponential Sine Sweep (ESS) technique was introduced by Farina [START_REF] Farina | Simultaneous measurement of impulse response and distortion with a swept-sine technique[END_REF][START_REF] Farina | Simultaneous measurement of impulse response and distortion with a swept-sine technique[END_REF]]. The probe signal is the ESS signal, a. k. a. chirp signal, which enjoys the following properties: the signal spans a user-de ned frequency range; it compresses into a narrow impulse during autocorrelation 39 ; and its Fourier inverse is available in closed form. 39 Ideally, if an in nite range of frequencies is used, then it compresses into the ideal Dirac function.

hi (t) xi (t) xi (t) τ (r) i , α (r) i r τ (r) i , α (r) i r
The reader can nd a review of the presented techniques applied to RIR measurements in [START_REF] Szöke | Building and evaluation of a real room impulse response dataset[END_REF]]. Sometimes the reference signal is known, but none of the above techniques can be used. For instance, when the source signal is narrowband or when the acoustic conditions are challenging. Therefore the RIR estimation problem needs to be addressed as a more general deconvolution problem, typically solved through optimization methods. This approach has been well studied in the literature and can be solved using standard Linear Least Squares Error methods with closed-form solutions [START_REF] Krim | Two decades of array signal processing research: the parametric approach[END_REF][START_REF] Glentis | E cient least squares adaptive algorithms for FIR transversal ltering[END_REF] However, in the case of signals with sparse frequency content (e. g. speech or music) or low SNR, it becomes ill-conditioned and prior knowledge about the RIR is used to improve the estimation, as it will discussed later. E R RIR. As discussed in Part I, acoustic echoes can be identi ed as peaks in the early part of the RIR. In general, due to the measurement process, such peaks are not necessarily positive. Thus, to better visualize them, the echogram [START_REF] Kuttru | Room acoustics[END_REF]], | h(t)|, or the energy envelope 40 [Schroeder 1979] 40 The energy envelope of a signal is computed as the magnitude of its analytic representation, computed with the Hilbert transform. For the signal x, its envelope is computed as

e[n] = x 2 [n] + H{x} 2 [n], (4.2)
where H{•} denotes Hilbert transform.

are used instead. Provided a good estimation of the RIRs, the echoes' locations and amplitudes could be extracted manually by experts. However, even in an ideal scenario, the automation of this process and the correct identi cation of such quantities are not straightforward tasks. As showed in [START_REF] Tukuljac | MULAN: a blind and o -grid method for multichannel echo retrieval[END_REF], since the TOAs are not necessarily multiples of the sampling grid, their true locations (and amplitudes) are blurred by spurious side peaks. This issue is referred to as basis mismatch in the compressed sensing literature. Although it can be alleviated by increasing the sampling frequency, it is bound to occur in practice. Moreover, the harmonic distortion due to the non-ideal source-receiver coupling may introduce other spurious peaks as well. Furthermore, as noticed in [Defrance et al. 2008b], even small errors in echo' timing estimation may lead to signi cant di erences for echo-based applications.

Existing methods for extracting echoes from RIRs can be further dichotomized into two broad categories: on-grid and o -grid approaches. The methods belonging to the former group are the most used in practice, and advanced technique are used to cope with the presence of spurious peaks [START_REF] Kuster | Reliability of estimating the room volume from a single room impulse response[END_REF][START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF][START_REF] Remaggi | Acoustic re ector localization: novel image source reversion and direct localization methods[END_REF]Defrance et al. 2008a;[START_REF] Bello | A tutorial on onset detection in music signals[END_REF] The most straightforward approach is to deploy iterative and adaptive thresholding algorithms on the RIR, followed by robust and manually-tuned peak nders [START_REF] Kuster | Reliability of estimating the room volume from a single room impulse response[END_REF][START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF]]. To better inform the peak-picking, several strategies have been proposed. In the work of [START_REF] Remaggi | Acoustic re ector localization: novel image source reversion and direct localization methods[END_REF]], based on a algorithm presented in [START_REF] Naylor | Estimation of glottal closure instants in voiced speech using the DYPSA algorithm[END_REF], peaks are clustered according to changes in the phase slope of the RIR spectrum. Other works apply onset detection techniques used in music information retrieval based on edge-detection wavelet lters [START_REF] Bello | A tutorial on onset detection in music signals[END_REF]], non-negative matrix factorization [START_REF] Cheng | An attack/decay model for piano transcription[END_REF], or considering the RIR's Kurtosis [START_REF] Usher | An improved method to determine the onset timings of re ections in an acoustic impulse response[END_REF]. By noticing that the re ections in the RIRs exhibit a similar shape to the direct path, the author of [Defrance et al. 2008a] rst proposed the use of Matching Pursuit (and improvements) to identify such shapes. Here, the direct sound part was used as a pattern (or atom) to be retrieved across the RIR. Unfortunately, in its pure form, this approach is unsuitable for real RIRs because of the frequency-dependent characteristic of the room absorption material. In order to improve the detection, in [START_REF] Kelly | Detecting arrivals in room impulse responses with dynamic time warping[END_REF] this approach is extended to Dynamic Time Warping to account for the non-uniform compression, dilation and concurrency of echoes. Nevertheless, the idea of exploiting the direct path component to isolate the source-receiver coupling and thus identify rst prominent re ections through deconvolution was used in [START_REF] Annibale | Geometric inference of the room geometry under temperature variations[END_REF]]. This technique is also referred to as matching lter or direct-path compensation.

Alternative approaches detect the echo timings in another transformed domain.

In [START_REF] Vesa | Segmentation and analysis of early re ections from a binaural room impulse response[END_REF] the echoes are localized in the Time-Frequency (TF) domain using cross-wavelet transform based on previous works [START_REF] Guillemain | Characterization of acoustic signals through continuous linear time-frequency representations[END_REF][START_REF] Loutridis | Decomposition of impulse responses using complex wavelets[END_REF]]. The works [START_REF] Ristić | Detection of early re ections using multifractals[END_REF][START_REF] Pavlović | Multifractal analysis of visualized room impulse response for detecting early re ections[END_REF]] use (multi-)fractal analysis to detect echoes in the Time-Frequency (TF) domain. Alternatively, the authors of [START_REF] Ferguson | Improved multipath time delay estimation using cepstrum subtraction[END_REF][START_REF] Jia | Extraction of echo characteristics of underwater target based on cepstrum method[END_REF] propose to identify echoes properties in the cepstral domain. The cepstrum is usually de ned as the inverse Fourier transform of the logarithmic magnitude spectrum. 41 One interesting property of the cepstrum is to separate 41 There are several concurrent de nitions of cepsturm. For a given time-domain signal x it is usually de ned as |F -1 log |F x| 2 | 2 , where F is the FT operator.

the spectral envelop form the line-spectra components in di erent "quefrency" ranges (i. e., the nal independent variable of the cepstrum domain). Therefore, it is used to detect periodicity in the spectrum of the observed signal, with many applications in speech processing as well as in hydraulic and mechanic applications. This approach seems promising since time-domain peaks are mapped to complex sinusoids in the frequency domain. However, this representation is highly sensible to external and sampling noise and the accuracy is limited by the approximation of the DFT operator.

All the above mentioned active and RIR-based works aim at detecting echoes on the sampling grid. In order to face the inherent limitations of these approaches, o -grid frameworks have been proposed, e. g. [START_REF] Condat | Robust spike train recovery from noisy data by structured low rank approximation[END_REF]. This approach can be related to classical Maximum Likelihood (ML) estimation approaches, which consist in selecting the model which is most likely to explain the observed noisy data. In this category fall classical spectral estimation techniques, e. g. Multiple Signal Classi cation (MUSIC) [START_REF] Loutridis | Decomposition of impulse responses using complex wavelets[END_REF]], Estimation of Signal Parameters via Rational Invariance Techniques (ESPRIT) [START_REF] Roy | ESPRIT-A subspace rotation approach to estimation of parameters of cisoids in noise[END_REF]], which are fast but statistically suboptimal. The method presented in [START_REF] Condat | Robust spike train recovery from noisy data by structured low rank approximation[END_REF] focuses on the general problem of estimating a nite stream of Dirac's impulses from uniform, noisy and lowpass-ltered samples. The authors showed that this particular problem can be reformulated as matrix denoising, from which the echoes' locations and amplitudes can be retrieved in closed-form. Although this method reaches the statistical optimality in the ML sense, the exact knowledge of the number of Diracs needs to be known in advance. If this number is unknown or approximated, errors in the estimation are observed. This results in a drawback since the exact number of echoes is di cult to know a priori and false-positive peaks are present even in clean RIRs due to the source's harmonic distortions or limited bandwidth.

That being said, AER is far from trivial even when clean RIR estimates are provided. It is important to note that, for every TOA estimator, a practical trade o exists between the number of missed TOAs (false negative) and the number of spurious TOAs (false positive). This trade-o is only partially dependent on the noise level since many factors can provide spurious peaks. For instance, side lobes due to nite signal bandwidth, echo distortions due to frequency dependent attenuations and coalescing peaks due to close TOAs can a ect peak estimation. This fact is often a source of unavoidable outliers that make the robustness of subsequent steps in echo-aware applications a delicate and very important issue. A way to overcome this trade-o is to allows spurious TOAs estimates and prune them using echo labeling afterwards. [START_REF] Scheuing | Disambiguation of TDOA estimates in multi-path multi-source environments (DATEMM)[END_REF][START_REF] Zannini | Improved TDOA disambiguation techniques for sound source localization in reverberant environments[END_REF]], microphone calibration [START_REF] Parhizkar | Single-channel indoor microphone localization[END_REF][START_REF] Salvati | Sound source and microphone localization from acoustic impulse responses[END_REF] and RooGE [START_REF] Antonacci | Geometric reconstruction of the environment from its response to multiple acoustic emissions[END_REF][START_REF] Filos | Robust inference of room geometry from acoustic measurements using the Hough transform[END_REF][START_REF] Venkateswaran | Localizing multiple events using times of arrival: a parallelized, hierarchical approach to the association problem[END_REF][START_REF] Antonacci | Inference of room geometry from acoustic impulse responses[END_REF][START_REF] Dokmanić | Acoustic echoes reveal room shape[END_REF][START_REF] Crocco | Towards fully uncalibrated room reconstruction with sound[END_REF]Jager et al. 2016;El Baba et al. 2017]. A brief review of these methods is provided in [START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF]]. In the context of SSL, the disambiguation is typically performed in the TDOA space [START_REF] Scheuing | Disambiguation of TDOA estimates in multi-path multi-source environments (DATEMM)[END_REF][START_REF] Zannini | Improved TDOA disambiguation techniques for sound source localization in reverberant environments[END_REF]. Moreover these works focus on actively localizing (the direction of arrival of) multiple sources while discarding re ections, rather than localizing the actual image sources. Other disambiguation schemes are typically used for RooGE. In [Venkateswaran wall i s x s F 4.4: Illustration of the ellipsoid method: the length of the re ected path from the image source s to the microphone at x constrain the re ector line to be tangent to an ellipse. and Madhow 2012] the pruning of the combinatorial candidate-image-source search is done through Bayesian inference. A similar approach can be found in [START_REF] Dokmanić | Acoustic echoes reveal room shape[END_REF][START_REF] Parhizkar | Single-channel indoor microphone localization[END_REF] where the validity check is based on a particular structured matrix called Euclidean Distance Matrix and further improved using compatibility graphs in [Jager et al. 2016]. These methods rely on a combinatorial search with a potentially high number of candidates, which leads to intractable computational complexity when multiple re ections are considered. Moreover these methods require that all the distances between microphones are known with precision, which may not be true in practice. In the works of [START_REF] Antonacci | Geometric reconstruction of the environment from its response to multiple acoustic emissions[END_REF][START_REF] Filos | Robust inference of room geometry from acoustic measurements using the Hough transform[END_REF][START_REF] Antonacci | Inference of room geometry from acoustic impulse responses[END_REF], the re ectors are modelled as planes tangent to ellipsoids with foci given by each pair of microphone/source, as shown in Figure 4.4.By solving a nonconvex optimization problem based on geometrical reasoning and the Hough transform 42 , they are able to disambiguate TOAs and reconstruct the re ectors' positions and orientations. However, they all require a very speci c acquisition setup and the optimized non-convex cost functions are sensible to local minima.

In general, none of the above methods come with speci c strategies to cope with missing or spurious echo estimates. These may be due to malfunctioning of the peak nder or by erroneous selection of peaks corresponding to higher re ection order. A way to solve this issue is to exploit particular prior knowledge. For instance, the approach presented in [El Baba et al. 2017] exploits the shapes of linear and compact arrays of loudspeakers, which provides a natural ordering among the loudspeakers. By stacking side-by-side the measured RIRs in a matrix, they can be visualized as an image. Here the wavefront of each re ection draws speci c patterns which can be identi ed easily and robustly even in the presence of (a few) spurious and missing peaks. Moreover, this approach avoids the combinatorial search. However, it requires a very speci c setup for recordings. In the work [START_REF] Crocco | Towards fully uncalibrated room reconstruction with sound[END_REF]] an iterative strategy is used. First, the direct path arrivals are used to estimate a rst guess of the microphones and source positions. Then, the whole set of extracted peaks are used to estimate the planar re ectors' positions which are then used to re ne the microphones and source localization. Alternating between the geometrical space of microphones and source coordinates and the signal space of the echoes' TOAs, the ambiguous peaks are pruned during the optimization.

Active and RIR-agnostic method

This class of methods uses the signal at the microphones to directly estimate the echoes re ections, rather than estimating the RIRs. Here two di erent approaches can be identi ed: optimization-based approaches [START_REF] Jensen | An EM method for multichannel TOA and DOA estimation of acoustic echoes[END_REF][START_REF] Saqib | Estimation of acoustic echoes using expectationmaximization methods[END_REF]] and cross-correlation-based approaches [START_REF] Crocco | Towards fully uncalibrated room reconstruction with sound[END_REF][START_REF] Al-Karawi | Early re ection detection using autocorrelation to improve robustness of speaker veri cation in reverberant conditions[END_REF].

The former approaches exploit the strong relation between the TOA of an echo with its Direction of Arrival (DOA). When multiple microphones are used and their geometry is known, the relation between these two quantities can be expressed in closed-form and used in an ML-based frameworks. By modeling the DOAs, such approaches are able to implicitly reduce the ambiguity of the estimated echoes. This idea is rooted in existing methods used in multipath communication systems, denoted as Joint Angle and Delay Estimation (JADE) [START_REF] Vanderveen | Joint angle and delay estimation (JADE) for multipath signals arriving at an antenna array[END_REF][START_REF] Verhaevert | Direction of arrival (DOA) parameter estimation with the SAGE algorithm[END_REF]].

Alternatively, the echo contributions can be extracted from the cross-correlation between the observed and the reference signals. Cross-correlation analysis is a mathematical tool for the identi cation of repeated patterns in a signal as a function of a certain time lag. Due to indoor sound propagation, the received signal consists in repeated copies of the emitted signal. Therefore, the received signal may correlate with the emitted one for certain time lags. Hence, peaks in the cross-correlation function can be observed. By the extraction of these peaks, echoes' TOAs and relative amplitudes can be identi ed. This approach was used in [START_REF] Tervo | Direction of arrival estimation of re ections from room impulse responses using a spherical microphone array[END_REF][START_REF] Crocco | Towards fully uncalibrated room reconstruction with sound[END_REF][START_REF] Al-Karawi | Early re ection detection using autocorrelation to improve robustness of speaker veri cation in reverberant conditions[END_REF].

When the array geometry is known, the time lag axes of cross-correlation functions between channels can be mapped to possible 2D directions of arrivals (elevation and azimuth), namely from TOAs to 2D-DOAs. The identi cation of strong re ections can then be performed in the so-called angular spectrum domain [START_REF] Dibiase | Robust localization in reverberant rooms[END_REF]. With a proper clustering approach, the re ections can be inspected, disambiguated and their TOAs deduced. This approach is used in [O' Donovan et al. 2008;[START_REF] O'donovan | Automatic matched lter recovery via the audio camera[END_REF][START_REF] Tervo | Direction of arrival estimation of re ections from room impulse responses using a spherical microphone array[END_REF] and can be generalized by spatial ltering methods, such as steered-response power-based beamforming. In [START_REF] O'donovan | Imaging concert hall acoustics using visual and audio cameras[END_REF], it was referred to as acoustic camera since it bene ts from the following visual interpretation: as shown in Figure 4.5, the 2D-polar Coordinates can be mapped into cartesian ones so that the angular spectrum can be superimposed to a panoramic picture of the audio scene taken by the barycenter of the recording arrays.

Passive and RIR-based method

Passive approaches rely on external, unknown sound sources in the environ-

The innovative idea of passive Blind Channel Estimation (BCE) can be traced back to [START_REF] Sato | A method of self-recovering equalization for multilevel amplitude-modulation systems[END_REF]. A review of the the evolution of Single Input Multiple Output (SIMO) BCE can be found in [START_REF] Huang | A class of frequency-domain adaptive approaches to blind multichannel identi cation[END_REF].

ment to conduct the estimation. In the literature, this problem belongs to the broad and deeply studied category of Blind Channel Estimation (BCE) or Blind System Identi cation (BSI) problems. In the particular case of a single source, it is referred to as SIMO BCE. Common to all these methods is the assumption that RIRs are discrete Finite Impulse Response (FIR) lters de ned on the sampling grid, namely, vectors in the Euclidean space. In the general setting of arbitrary signals and lters, rigorous theoretical ambiguities under which the problem is ill-posed have been identi ed [START_REF] Xu | A least-squares approach to blind channel identi cation[END_REF]. Some well-known limitations of these approaches are their sensitivity to the chosen length of the lters, and their intractability when the lters are too large. FIR SIMO BCE can be broadly dichotomized into the class of statistical methods and the class of source-agnostic methods. Although it makes the exposition clearer, there is a lot of overlap between these two methods nowadays. It could also be said that the accuracy of certain source-agnostic methods can be improved by the availability of source statistics.

S

exploit knowledge about the emitted signal. Since the nature of the source signal is often non-deterministic and it statistics can be modeled based on the signal category, e. g. speech or music. Two main approaches can be identi ed [START_REF] Tong | Multichannel blind identi cation: From subspace to maximum likelihood methods[END_REF]]:

• Second Order Moments approaches derive closed-form solutions for which the knowledge of the source auto-correlation function is required.

• Maximum Likelihood approaches require instead the source probability function. Even though they are optimal in the ML sense, they optimize non-convex cost functions, typically via Expectation Maximization (EM).

In this category one may include the methods developed for MASS [Ozerov and Févotte 2010; [START_REF] Duong | Under-determined reverberant audio source separation using a full-rank spatial covariance model[END_REF][START_REF] Sawada | Multichannel extensions of non-negative matrix factorization with complex-valued data[END_REF]Leglaive et al. 2016;Leglaive et al. 2018;Scheibler et al. 2018c]. These methods are built on the well-studied framework of Multichannel Nonnegative Matrix Factorization (NMF) [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF][START_REF] Sawada | Multichannel extensions of non-negative matrix factorization with complex-valued data[END_REF]] which lends itself to account for various types of side information.

Here the source signals are typically modeled as Gaussian distributions centred at zero and with unknown variances. In general, they aim at estimating both the acoustic channels and the source contribution. In particular, the work of [START_REF] Duong | Under-determined reverberant audio source separation using a full-rank spatial covariance model[END_REF]] extends this framework to reverberant recordings using physics-based models for the late reverberation, while the works of [Leglaive et al. 2016] considers explicitly the contribution of early echoes, further improved in [Leglaive et al. 2018].

Even if statistical methods have reported a considerable success in the eld of sound source separation, they play a minor role in RIR estimation. This is due to the di culty of estimating reliable statistics of the emitted signals or a good initialization point required by the EM algorithm. Moreover, although the nal estimated RIRs may match the real ones in the statistical sense, they lack a su cient level of details, indispensable for AER.

S

comprises two main groups: subspace methods [START_REF] Abed-Meraim | A subspace algorithm for certain blind identi cation problems[END_REF]] and cross-relation methods [START_REF] Tong | Blind identi cation and equalization based on secondorder statistics: A time domain approach[END_REF][START_REF] Xu | A least-squares approach to blind channel identi cation[END_REF][START_REF] Lin | Blind sparse-nonnegative (BSN) channel identi cation for acoustic time-di erence-of-arrival estimation[END_REF]Lin et al. 2008;[START_REF] Kowalczyk | Blind system identi cation using sparse learning for TDOA estimation of room re ections[END_REF][START_REF] Crocco | Room impulse response estimation by iterative weighted l 1-norm[END_REF]Crocco and Del Bue 2016]. The formers are based on the key idea that the channel (or part of it ) vector spans a one-dimensional subspace of noiseless observations. These methods have the attractive property that the channel estimates can often be obtained in closed-form by optimizing a quadratic cost function. However that they are typically computationally expensive and sensible to noise. The second family of methods rely on the clever observation that in the noiseless case, for every pair of microphone (i, i ), it holds that

(x i hi )(t) = (x i hi )(t) = (( hi hi ) s)(t), (4.3)
by commutativity and associativity of the convolution operator. This identity is called the cross-relation and it was rst introduced by [START_REF] Tong | Blind identi cation and equalization based on secondorder statistics: A time domain approach[END_REF]. In this work, the RIR are estimated by solving a Least Square minimization of the sum of square cross-relation errors. In [START_REF] Xu | A least-squares approach to blind channel identi cation[END_REF][START_REF] Tong | Multichannel blind identi cation: From subspace to maximum likelihood methods[END_REF], su cient and necessary conditions for channel identi cation are discussed. This approach has received signi cant attention as it does not require any assumption about the source signal. Later, the accuracy of estimated RIRs has been subsequently improved using prior knowledge on the lters: in particular, the authors of [START_REF] Lin | Blind sparse-nonnegative (BSN) channel identi cation for acoustic time-di erence-of-arrival estimation[END_REF]] have proposed to use sparsity penalty and non-negativity constraints to increase robustness to noise as well as Bayesian-learning methods to automatically infer the value of the hyperparameters in [Lin et al. 2008]. Even if sparsity and non-negativity could be seen as a strong assumptions, works in speech enhancement [Ribeiro et al. 2010a;Dokmanić et al. 2015] and room geometry estimation [START_REF] Antonacci | Inference of room geometry from acoustic impulse responses[END_REF][START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF]] have proven the e ectiveness of this approach. On a similar scheme, in [START_REF] Kowalczyk | Blind system identi cation using sparse learning for TDOA estimation of room re ections[END_REF], the problem in Eq. ( 4.3) is solved using an adaptive time-frequency-domain approach while [Aissa- [START_REF] Aissa-El-Bey | Blind SIMO channel identi cation using a sparsity criterion[END_REF] proposes to use the p -norm instead of the 1 -norm. A successful approach has been presented by [START_REF] Crocco | Room impulse response estimation by iterative weighted l 1-norm[END_REF]Crocco and Del Bue 2016], where the so-called anchor constraint is replaced by an iterative weighted 1 equality constraint to better balance sparsity penalty and the model constraints. 43 Finally, the very recent work [Qi et al. 43 These approaches will be further formalized and detailed in Chapter 5.

2019] extends cross-relation approaches under the umbrella of the Kalman lter which was previously used for echo-cancellation applications.

An alternative approach is used in [Čmejla et al. 2019], where the ReIR estimation problem is treated as a special case of ReIR estimation. As mentioned in § 3.3.2, in the noiseless case, the ReIR identi es to the ReIR when the reference microphone is placed very close to the source. ReIR estimation found its root in the eld of Speech Enhancement (SE) [START_REF] Gannot | Signal enhancement using beamforming and nonstationarity with applications to speech[END_REF]] and many techniques have been proposed since then [START_REF] Gannot | Signal enhancement using beamforming and nonstationarity with applications to speech[END_REF][START_REF] Koldovskỳ | Spatial source subtraction based on incomplete measurements of relative transfer function[END_REF]; Koldovsky and Tichavsky 2015; [START_REF] Kodrasi | EVD-based multi-channel dereverberation of a moving speaker using di erent RETF estimation methods[END_REF]. By its de nition, in the noiseless case, the ReIR describes the relative lter between two observations and not directly their RIRs. The main limitation of this approach is that it is possible only in controlled scenarios, where the user has the possibility to place the microphone arbitrarily in the room and in the presence of high Signal-to-Noise-Ratio (SNR) levels. Nevertheless, in this context, this particular setup is found to be useful not only for RTF estimation, but also for microphone calibration, since it allows to solve geometrical ambiguities, yielding a closed-form solution, as done in [START_REF] Crocco | A closed form solution to the microphone position self-calibration problem[END_REF].

In general, the main drawbacks of FIR SIMO BCE works is that they rely on on-grid estimation, sparsity-enforcing regularizers and peak-picking which need to be tuned manually. As described in § 3.2.3, due to the sampling process involving an ideal low-pass kernel, the lters are non-sparse and non-negative. This general bottleneck has been referred to as basis mismatch and was notably studied in the compressed sensing community [START_REF] Chi | Sensitivity to basis mismatch in compressed sensing[END_REF]]. In particular, the true peaks in the RIR do not necessarily correspond to the true echoes as discussed in the previous chapter. Since these methods are fundamentally on-grid, the estimated echo locations are integer multiples of the sampling period 1/F s . This prevents subsample resolution, which may be important in applications such as RooGE [START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF] or acoustic parameter estimation [Defrance et al. 2008b]. Moreover, these methods strongly rely on the knowledge of the length of the lters. When this parameter is underestimated or overestimated, identi ability and computational issues may arise, a ecting the estimation. Nevertheless, despite this slight mismatch between theoretical assumptions and real data, for some scenarios, the position of the estimated peaks by the methods [Crocco and Del Bue 2016] reproduces the positions of the groundtruth peaks with remarkable precision.

Passive and RIR-agnostic methods

Methods in this category bypass the onerous task of estimating the (full or partial) acoustic channel and, to the best of our knowledge, only a few methods have been identi ed. As for the active and RIR-agnostic case, the audio camera based on the cross-correlation function can be used in passive settings.

Exploiting the geometrical knowledge of the microphone array, TDOAs extracted from robust correlation function can be mapped to DOAs [START_REF] Dibiase | Robust localization in reverberant rooms[END_REF][START_REF] O'donovan | Imaging concert hall acoustics using visual and audio cameras[END_REF][START_REF] O'donovan | Automatic matched lter recovery via the audio camera[END_REF]. Assuming a single source scenario, distinct DOAs can be disambiguated using geometrical prior knowledge and can be associated to image sources, hence re ectors. These methods typically ignore the echoes amplitudes and in general only consider angles on the unit sphere, ignoring the distance from the source. Without proper prior knowledge, their application to AER is far from trivial, as RooGE and re ector estimation methods need to be used to convert DOAs back to echo timing.

Recently, a fully blind, passive, o -grid and RIR-agnostic method was proposed by the authors of [START_REF] Tukuljac | MULAN: a blind and o -grid method for multichannel echo retrieval[END_REF] for stereophonic recordings (i. e., I = 2). They proposed a method, called Multichannel Annihilation (MULAN), based on the properties of the annihilation lter 44 , [Condat and Hirabayashi 44 For a sequence of Fourier coe cients a ∈

C N (describing a signal or a lter), its annihilation lter b L ∈ C L is such that the linear convolution between the sequence and the lter coe cients is identically zero:

L-1 l=0 b[l]a[n -l] = 0 ∀ n = -N + L, . . . , N .
2013] and the theory of Finite Rate of Innovation (FRI) [START_REF] Vetterli | Sampling signals with nite rate of innovation[END_REF]. If the source signal is known, starting from the cross-relation identity, the AER problem translates into nding the annihilation lter for the RTFs, which can be recasted into an eigenvalue problem. In the fully blind case, the problem is solved with non-convex optimization, iterating between the estimation of the two lters and the signal until convergence. The method was later extended to the multichannel case in [START_REF] Tukuljac | Sparse and Parametric Modeling with Applications to Acoustics and Audio[END_REF]] using a generalization of the Cadzow denoising framework [Condat and Hirabayashi 2015]. This method is shown to outperform conventional approaches by several orders of magnitude in precision in the noiseless case, with synthetic data and when the correct number of echoes is known a priori. However its e ectiveness was not tested on challenging real scenarios featuring external noise and partial knowledge on the number of echoes. In this direction, we contributed with two works: a knowledge-driven approach [Di Carlo et al. 2020], presented in Chapter 5, and a learning-based approach [Di Carlo et al. 2019], discussed in Chapter 6.

D

AER is a relatively recent problem which is typically addressed in the context of much broader applications, e. g. SE, RooGE, SSL. Therefore the literature lacks standard datasets for it as well as standard evaluation frameworks.

Datasets

As listed in [START_REF] Szöke | Building and evaluation of a real room impulse response dataset[END_REF] and in [START_REF] Genovese | Blind room volume estimation from single-channel noisy speech[END_REF]], a number of recorded RIR corpora are freely available online, each of them meeting the demands of certain applications, usually SE and Automatic Speech Recognition (ASR). However, even if these datasets feature reverberation and strong early re ections, they lack of proper annotations, making them di cult to use for testing AER methods. For this reason, to bypass the complexity of recording real annotated RIR datasets, acoustic simulators typically based on the ISM are extensively used instead. While simulated datasets are more versatile, simple and quicker to obtain, they fail to fully capture the complexity and the richness of real acoustic environments. Due to these intrinsic issues, methods trained or validated on them may fail to generalize to real conditions, as will be shown in Chapter 7. For instance in [START_REF] Dokmanić | Acoustic echoes reveal room shape[END_REF] and in [START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF] echoes are computed after the whole RooGE processing from some manually-selected peaks in the RIRs. The exact timing is not available. The reader is invited to refer to [START_REF] Szöke | Building and evaluation of a real room impulse response dataset[END_REF][START_REF] Genovese | Blind room volume estimation from single-channel noisy speech[END_REF]] for a more complete list of existing RIR datasets.

† The dataset in [START_REF] Remaggi | Acoustic re ector localization: novel image source reversion and direct localization methods[END_REF]] is originally intended for RooGE and further extended for (binaural) SE in [START_REF] Remaggi | Acoustic re ector localization: novel image source reversion and direct localization methods[END_REF]] with a similar setup.

A good dataset for AER should include a variety of environments (room geometries and surface materials), of microphone placements (close to or away from re ectors, scattered or forming ad-hoc arrays) and, most importantly, precise annotations of the scene's geometry and echo parameters. Moreover, in order to be versatile and used in echo-aware applications, the provided annotations should match the ISM, i.e., TOAs should be expressed in terms of image sources and vice-versa. Such data are di cult to collect since they require precise measurements of the positions and orientations of all the acoustic emitters, receivers and re ective surfaces inside the environment with dedicated planimetric equipment. We identi ed here two main classes of related RIR datasets in the literature: SE/ASR-oriented datasets, e.g. [START_REF] Szöke | Building and evaluation of a real room impulse response dataset[END_REF][START_REF] Bertin | VoiceHome-2, an extended corpus for multichannel speech processing in real homes[END_REF]Čmejla et al. 2019], and RooGE-oriented datasets, e.g. [START_REF] Dokmanić | Acoustic echoes reveal room shape[END_REF][START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF][START_REF] Remaggi | Acoustic re ector localization: novel image source reversion and direct localization methods[END_REF]]. The formers include acoustic echoes as highly correlated interfering sources coming from close re ectors, (e. g. a desk in meeting rooms or a nearby wall), however annotations are not provided. The latter group deals with sets of distributed, synchronized microphones and loudspeakers in a room. These setups are not exactly suitable for SE methods, which typically involve compact or ad hoc arrays. To bridge this gap, we recorded a new dataset, dubbed Dataset dechorated by echoes (DECHORATE), that will be described in Chapter 7 and used for echo-aware application in Chapter 11. Table 4.1 summarizes some existing datasets that could be used in the context of echo-aware processing.

Metrics

The metrics used in AER depend on the application and the methods used to estimate the echoes. When addressed as a FIR SIMO BCE problem, the groundtruth acoustic channels are considered as a discrete vector h ∈ R L , and similarly to their estimates, that is, ĥ ∈ R L . To assess the quality of the estimated discrete lters, the following metrics have been proposed in the literature:

• The Root Mean Square Error (RMSE) measures the distance between points in the Euclidean space, de ned by vector coordinates:

RMSE( ĥ, h) def = 1 L L-1 n=0 ĥ[n] -h[n]
2

[seconds (or, samples)], (4.4)

where |•| denotes the absolute value. This metrics is known to be highly sensitive to scaling and translations.

• The Normalized Projection Misaligment (NPM) was originally proposed in [START_REF] Morgan | On the evaluation of estimated impulse responses[END_REF]] to solve the limitation of the RMSE. In the formulation provided in [START_REF] Huang | A class of frequency-domain adaptive approaches to blind multichannel identi cation[END_REF][START_REF] Ahmad | Proportionate frequency domain adaptive algorithms for blind channel identi cation[END_REF]], it writes as NPM( ĥ, h) def = 20 log 10 1 h 2 h -h ĥ ĥ ĥ ĥ 2

[dB], (4.5)

where • 2 denotes the Euclidean norm. By projecting ĥ onto h and de ning a projection error, only the intrinsic misalignment of the channel estimate is considered, disregarding an arbitrary gain factor of both vectors. However it is not translation invariant.

• The Hermitian angle is similar to NPM and was used in the context of RTF estimation in [START_REF] Varzandeh | An iterative multichannel subspace-based covariance subtraction method for relative transfer function estimation[END_REF][START_REF] Tammen | Iterative Alternating Least-Squares Approach to Jointly Estimate the RETFs and the Di use PSD[END_REF] ∆Θ( ĥ, h) = arccos h Hĥ h 2 ĥ 2 .

(4.6)

As for NPM, this metrics is invariant to possible scaling factors between the groundtruth and the estimated vectors.

In the context of RooGE, SSL and microphone calibration, echo timings are typically mapped to re ectors or image source positions, either in cartesian or polar coordinates. Therefore, the models for AER are evaluated in the geometrical space, rather than in the space of echoes' parameters. For instance, for the task of re ectors localization, the accuracy is measured in terms of plane-to-plane distance between estimated and groundtruth surfaces and the angular error between their normals. In the case of SSL and microphone calibration, the Euclidean distance between the 3D coordinates is typically computed as RMSE between groundtruth and estimated DOAs. This metrics considers only the echoes' TOA, ignoring their amplitudes.

To our best knowledge, the literature lacks of metrics properly de ned for AER. As for the application mentioned above, echoes' amplitudes in a single RIR or between them, are typically ignored or considered for peak picking only. More attention is paid on the echo timings which are evaluated using regression/classi cation metrics of information retrieval and machine learning.

Let be τ = {τ r } R r=0 and τ = {τ r } R r=0 the sets of estimated and reference echoes' TOAs. The following metrics are used:

• the RMSE is de ned as

RMSE(τ , τ ) def = 1 R + 1 R r=0 |τ r -τ r |
2 [seconds (or, samples)], (4.7)

This metric describes the mean error between estimated and reference echo TOAs. Unfortunately, the RMSE is proportional to the size of the squared error, thus is sensitive to outliers. In the context of AER, the RMSE is computed only on the matched TOAs.

• the Precision, Recall, and F-measure are standard metrics used in information retrieval for evaluating classi cation problems, e. g. in onset detection [START_REF] Böck | Evaluating the Online Capabilities of Onset Detection Methods[END_REF]]. Here the real valued estimates and groundtruth need to be converted into binary values indicating a match. Typically,

Given the number of true positive cases in the estimation TP, the false negative FN and the false positive PF, precision, recall and Fmeasure as:

Precision = TP/(TP + FP), Recall = TP/(TP + FN), Fmeasure = 2 • Precision • Recall Precision + Recall .
hard thresholding is used to assess whether estimated TOAs match the reference one. In the context of AER, precision expresses the fraction of matching TOAs among all the estimated ones, while recall measure the fraction of matching TOAs that are correctly estimated. Finally, the F-measure, de ned as the harmonic mean of precision and recall, is used to summarize precision and recall in one value. Depending on the application, precision and recall can have di erent impact. RooGE methods are more sensible to missing TOAs than to their misalignment which can be rede ned with geometrical reasoning. Thus they are more incline to privilege recall over precision and allow for some false-positive which can be pruned using echo labelling methods. Instead, echo-aware SE methods prefer to accurately select the relevant echoes, thus favoring higher precision.

Since these metrics rely on decision thresholds, their usage is not straightforward. In fact, in order to compare echoes, both estimated and reference echoes need to be labeled, pruned and matched rst. As discussed at the end of § 2.3.3, echoes can be sorted di erently according to their amplitudes, their TOAs or re ection order. AER tends to return echo parameters sorted by the echoes' amplitudes which can be distorted by the measurement process and modelling errors. This matching and labeling process introduces strong biases in the evaluation process which is currently unsolved without a proper echo labeling step.

P

So far, we presented a view of current methods for solving the AER problem. In the following two chapters, we will explore two novel approaches which follow two paradigms having taken hold in the recent years of signal processing:

K take advantage of prior information which may have deterministic (e. g. physical equation) or asymptotic behaviors (e. g. statistical models). In this context, AER exploits prior information about the sources, the mixing process and the physics laws of acoustic propagation, along with the audio. This knowledge is typically translated into mathematical models which lead to solutions computed through closed-forms or optimization-based algorithm to estimate latent variables. All the literature presented in this chapter follows this approach. In general, the advantages and the disadvantages of these approaches depend on the nature of available knowledge on the context in which it is used. Regarding our contributions, Chapter 5 proposes a new knowledge-driven method for solving AER based on the theory of Continous Dictionarys (CDs).

D

, instead, are based on machine learning algorithms where information is automatically "learned from data" in supervised ways, therefore the knowledge of the physical laws is not need. Provided comprehensive and exhaustively annotated training datasets, such methods can learn functions that maps an input to an output based on example input-output pairs. Due to its recent success, machine learning, and in particular deep learning, has been applied in many signal processing tasks. Alongside the great bene t of having black-box models that are able to learn on their own, this paradigm hides a few limitations. First, these models rely on the information encoded in training data which are sometimes not representative enough of the real-world end-application. In order to overcome this drawback, many strategies have been proposed, e. g. using data augmentations techniques or knowledge-driven generating models, based on simulators. It this case, prior knowledge is used to generate synthetic observations (i. e., from latent variables to the observations) and and the task of inverting the mapping is left to the learning method (i. e., from observation to latent variables). This leads to a second limitation of these approaches, that is over tting to the data obeying the generative model. Finally, machine learning models learn black-box functions. Although they can reach incredible performance, it is di cult to predict their behavior when facing new type of data. Despite these issues, data-driven methods have shown to achieve impressive performance. Therefore they are intensively studied and interlaced with knowledge-driven approaches. In this direction, we propose our contribution in Chapter 6, as a new data-driven method for solving AER based on virtually supervised learning.

• Paper

• Code

• Open-access paper with supplemen- tary material

• Slides

• Presentation stereophonic recording of an unknown sound source such as speech. In order to address some limitation of existing methods, we propose a new approach, named BLASTER. It builds on the recent framework of Continous Dictionary (CD), and it does not rely on parameter tuning nor peak picking techniques by working directly in the parameter space of interest. The method's accuracy and robustness are assessed on challenging simulated setups with varying noise and reverberation levels and are compared to two state-of-the-art methods. While comparable or slightly worse recovery rates are observed for recovering seven echoes or more, better results are obtained for fewer echoes, and the o -grid nature of the approach yields generally smaller estimation errors.

The material presented in this chapter was previously published in [Di Carlo et al. 2020] and results from a collaboration with my colleague Clement Elvira Di Carlo et al., "Blaster: An O -Grid Method for Blind and Regularized Acoustic Echoes Retrieval" whose domain of expertise is in the Continous Dictionary (CD) framework, while my expertise lies in the audio and acoustic signal processing and modeling aspects. The section dedicated on the presentation of the CD framework applied to AER is extracted from the related publication and it is written with the help of my colleague. Here we brie y commented and expanded it and some attention is paid in highlighting the motivation behind it. Finally, this chapter recalls the main ndings of the paper, while bringing additional insights in the existing models for AER.

I

Let us recall from Chapter 4 some knowledge-based methods addressing AER. Some existing methods rely on a known source signal [START_REF] Park | Compressive time delay estimation o the grid[END_REF][START_REF] Jensen | An EM method for multichannel TOA and DOA estimation of acoustic echoes[END_REF]. In contrast, when multiple receivers attend an unknown single source, AER can be seen as an instance of Single Input Multiple Output (SIMO) Blind Channel Estimation (BCE) problem, i. e. estimating the lters entailing an unknown input and an observed output of a system. A common approach for solving AER in the context of SIMO-BCE is to rst blindly estimate a discrete version of the acoustic channels using the so-called cross-relation identity [START_REF] Xu | A least-squares approach to blind channel identi cation[END_REF]Crocco and Del Bue 2016]. The location of the echoes are then chosen among the strongest peaks with ad-hoc peak-picking techniques. Such 65 methods are generally on-grid in the sense that the estimation relies on a xed grid of time samples and a priori chosen lter lengths. However, in practice, the true timings of echoes rarely match the sampling grid, thus leading to pathological issues called basis-mismatch in the eld of compressed sensing. To circumvent this issue, the authors of [START_REF] Tukuljac | MULAN: a blind and o -grid method for multichannel echo retrieval[END_REF] proposed to leverage the framework of nite-rate-of-innovation sampling to make one step towards o -grid approaches. Despite promising results in the absence of noise and with synthetic data, the quality of the estimation highly relies on the choice of a good initialization point.

Of particular interest in the proposed approach is the recently proposed framework of Continous Dictionary (CD) [START_REF] Candès | Towards a mathematical theory of super-resolution[END_REF]. By formulating an inverse problem as the recovery of a discrete measure over some parameter space, CD has allowed to overcome imaging device limitations in many applications such as super-resolution [START_REF] Candès | Towards a mathematical theory of super-resolution[END_REF] or PALM/STORM imaging [START_REF] Denoyelle | The sliding Frank-Wolfe algorithm and its application to super-resolution microscopy[END_REF]. In this work, we formulate the problem of AER for stereophonic mixtures, i. e. using only one microphone pair, within the framework of continuous dictionaries. The resulting optimization problem is convex and thus not prone to spurious minimizers. The proposed method is coined Blind And Sparse Technique for Echo Retrieval (BLASTER) and requires no parameter tuning. The method is compared to state-of-the art on-grid approaches under various noise and reverberation levels using simulated data.

S

We consider here the common setup of stereophonic mixtures, that is 2channel microphone recordings. Using the notation presented Chapter 3, the continuous-time signal recorded at microphone i ∈ {1, 2} reads

xi (t) = (s h i )(t) + ñi (t) (5.1)
where denotes the (continuous) convolution operator, ñi models some additive noise in the measurement process and h i denotes the Room Impulse Response (RIR). In the remainder of this chapter, the superscript refers to the requested RIR. Assuming the echo model, the RIRs read

h i (t) = Ri r=0 α (r) i δ(t -τ (r) i ) + εi (t) (5.2)
where R i is the (unknown) number of echoes.

In the noiseless case, that is when ñi = 0 for i ∈ {1, 2}, we have the crossrelation identity

x1 h 2 = x2 h 1 (5.3)
by commutativity of the convolution operator. However, in practice, only sampled versions of the two recorded signals are available. More precisely, we consider the measurement model introduced in Chapter 3: the incoming signal undergoes a (ideal) low-pass lter φLP with frequency support [ -Fs /2, Fs /2] before being regularly sampled at the rate F s .

We denote x 1 , x 2 ∈ R 2N the two vectors of 2N (consecutive) samples and i ∈ {1, 2} by

x i [n] = φLP * x n F s ∀n ∈ {0, . . . , 2N -1}.
(5.4)

B

We now select and elaborate more on some of the methods mentioned in § 4.3.3 concerning optimization approaches for FIR-SIMO-BCE. Therefore, the following methods operate on discrete-time signal, which are denoted without the tilde according to the notation presented in Chapter 3. Starting from the identity Eq. ( 5.3), the common SIMO-BCE cross-relation framework aims to compute h1 , h2 solving the following problem in the discrete-time domain [START_REF] Lin | Blind sparse-nonnegative (BSN) channel identi cation for acoustic time-di erence-of-arrival estimation[END_REF]:

h 1 , h 2 = arg min h1,h2 1 2 T(x 1 )h 2 -T(x 2 )h 1 2 2 + λ h 1 (5.5)
where

•
x i and h i are the discrete, sampled version of xi , hi respectively,

• T(x i ) is the (2N +L-1)×L Toeplitz matrix (built as shown in Figure 5.1)

x i T(x i ) 2N L 2N + L -1 F 5.1: Graphical representation of the construction of T(x i ) from x i
associated to a convolution where 2N and L respectively denote the microphone and lter signal lengths,

• h = [h 1 , h 2 ]
is the concatenation of the two vectorized discrete lters,

• and the 1 regularization term is used to enforce sparsity in the estimation, which is consistent with the "train of impulses" model for the early part of RIRs (but not for the tail).

This type of problem can be seen as an instance of Least Absolute Shrinkage and Selection Operator (LASSO) problem [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]], written in the form:

arg min u 1 2 v -Au 2 2 + λ u 1 . (5.6)
This type of well-known optimization problem are convex and, despite the non-di erentiability of the 1 -norm, they can be easily tackled by standard optimization tool. Later, in this section, we show how to express Eq. (5.5) as a standard LASSO problem.

The accuracy of estimated RIRs has been subsequently improved using a priori knowledge on the lters. In particular, the authors of [START_REF] Lin | Blind sparse-nonnegative (BSN) channel identi cation for acoustic time-di erence-of-arrival estimation[END_REF]] have proposed to use non-negativity constraints to increase robustness to noise and avoid trivial solution. Therefore, let us de ne a more general formulation for Eq. (5.5), as follows

h 1 , h 2 = arg min h1,h2 J(h 1 , h 2 ) + P(h 1 , h 2 ) s.t. C(h 1 , h 2 ) (5.7)
where

J(h 1 , h 2 ) = 1 2 T(x 1 )h 2 -T(x 2 )h 1 2
2 is the cost function to optimize. P(h 1 , h 2 ) and C(h 1 , h 2 ) are respectively a regularization term used to promote sparse solutions and a constrained set. Thanks to this formulation, current state-of-the-art approaches can be summarized as in the Table 5.1. [START_REF] Tong | Blind identi cation and equalization based on secondorder statistics: A time domain approach[END_REF] h 2 2 = 1 Equivalent to a smallest-eigenvalue problem. [START_REF] Kowalczyk | Blind system identi cation using sparse learning for TDOA estimation of room re ections[END_REF] λ h 1 h 2 2 = 1 Non-convex due to the quadratic constraint. [Lin et al. 2008] λ h

1 |h1[0]| = 1
With Bayesian learning for optimal λ. [START_REF] Lin | Blind sparse-nonnegative (BSN) channel identi cation for acoustic time-di erence-of-arrival estimation[END_REF] λ h 1 h1[0] = 1, h ≥ 0 Non-negativity and anchor constraints. The constraint h i [0] = 1 is called an anchor constraint and it is used to replace the 2 -norm while keeping the problem convex. The non-negativity h ≥ 0 constraint may not be satis ed due to e ects such as measurement process, the ltering in the propagation media or the imperfect frequency response of a microphone. However, when those e ects are common to both channels, they can be viewed as distortions to a common source. Therefore, the non-negativity assumption seems reasonable for real acoustic environments. Nevertheless, applications concerning RooGE require just the recovery of lower order re ections, i.e. the sparse portion of the RIR. Likewise works in speech enhancement have proven to work under such assumption, thus proving the e ectiveness of this approach, such as in [Lin et al. 2008;[START_REF] Yu | Multi-Channel l_{1} Regularized Convex Speech Enhancement Model and Fast Computation by the Split Bregman Method[END_REF]].

On a similar scheme, in [START_REF] Kowalczyk | Blind system identi cation using sparse learning for TDOA estimation of room re ections[END_REF]], Eq. (5.5) is solved using an adaptive time-frequency-domain approach while [Aissa- [START_REF] Aissa-El-Bey | Blind SIMO channel identi cation using a sparsity criterion[END_REF] proposes to use the p -norm instead of the 1 -norm. Choosing p < 1, sparsity is enforced, however the problem become non-convex and ad-hoc optimization technique was proposed. A successful approach has been presented recently by the authors of [Crocco and Del Bue 2016], where the anchor constraint is replaced by an iterative weighted 1 equality constraint, i. e., such that at each iteration z, p (z) h (z) = 1. 45 In particular, the method is 45 Note that when p (z) = 1, the constraint returns to the 1 penalty.

initialized using the solution of [START_REF] Lin | Blind sparse-nonnegative (BSN) channel identi cation for acoustic time-di erence-of-arrival estimation[END_REF]] and iterated enforcing sparsity using the solution of the previous problem, that is p (z) = ĥ(z-1) . The reader can nd a comprehensive review of these methods in [Crocco and Del Bue 2015; Crocco and Del Bue 2016].

T described above are the followings:

• Basis mismatch: As explained in § 4.3.3, the lter are not sparse in F 5.2: Schematics of the bodyguard effect a ecting on-grid approaches. This is true only if h is non-negative.

practice due to the basis mismatch. This implies that the peaks of the lter do not necessarily correspond to the true echoes and lead to followings drawbacks. As these methods operates fundamentally on-grid, they return echoes' timings which are integer multiples of the inverse of F s .

• Bodyguard e ect. In addition to a ecting the AER performance, ongrid methods may converge slowly to suboptimal solutions. In fact, as shown in Figure 5.2, instead of estimating the positive peaks at its true location, two smaller "bodyguard" peaks are estimated around it instead. The estimation procedure may stop at this point returning two wrong peaks. Having smaller coe cients, these peaks may not be selected by the subsequent peak picking technique. Alternatively, the optimization procedure may continue, alternating tuning the weights of the two "bodyguards", without converging to a solution.

• Computational bottleneck. A way to cope with the above limitations is to increase the F s . However this results into a memory and computational bottleneck as several huge (Toeplitz) matrices need to be built, one for each pair of microphones. In addition, this leads to the risk that the optimization problem becomes ill-conditioned.

In the following section we will present a novel approach that aims at addressing the above limitations. It is based on a framework proposed for solving LASSO problems for continuous variables, hence the name CD. Before, let us show how to express the on-grid BCE problem proposed by [Lin et al. 2008] (See Table 5.1) as a standard LASSO problem.

From cross-relation to LASSO

Integrating the sparse penalty and the constraints proposed in Eq. (5.7), the Blind Sparse Nonnegative Channel Identi cation (BSN) problem proposed reads

h 1 , h 2 = arg min h1,h2 1 2 T(x 1 )h 2 -T(x 2 )h 1 2 2 + λ h 1 s.t.    h ≥ 0 h 1 [0] = 1 .
(5.8) This cross-relation based optimization problem can be rewritten in the LASSO formulation of Eq. (5.6) as

u = arg min u 1 2 v -Bu 2 2 + λ u 1 s.t. u ≥ 0, where v = T 2 e 1 , u = h 1 [1 :] h 2 , A = -T 2 [:, 1 :] T 1 ,
where

T i = T(x i ).
Here we used the light, yet common, Python notation for indexing the matrices and vectors. The matrix A is typically called dictionary.

P

The cross-relation identity Eq. ( 5.3) ensures that the relation

φLP * x1 * h 2 = φLP * x2 * h 1 (5.9)
holds even during the introduced measurement process, hence

F( φLP * x1 ) • F h 2 = F( φLP * x2 ) • F h 1 (5.10)
where F denotes the Fourier Transform (FT) described in § 3.2.

In contrast with SIMO-BCE methods that operates in the time domain, here we propose to use Eq. (5.10) in a penalized least-square problem. Such a formulation in the Fourier domain may even be considered as more convenient since the convolution operator is no longer involved. While the FT of h i can be expressed in closed-form (see Eq. (5.13) below), the FT of φLP * x i is not available due to the measurement process. To circumvent this issue, we consider the Discrete Fourier Transform (DFT) of xi : 11) for all integers k ∈ {0, . . . , N }, where

F( φLP * xi ) k 2N F s = X i [k] (5.
X i [k] = 2N -1 n=0 x i [n]e -i2π kn

2N

(5.12)

is the DFT of the real vector xi as de ned in Eq. (3.15) for positive frequencies only.

Let us de ne ∆ τ the following parametric vector of complex exponential (5.13) where we consider only the N positive frequencies due to the Hermitian symmetry of the signal spectra in this application. Then, the Fourier-domain cross-relation of Eq. (5.10

∆ τ def = e -i2π k 2N Fsτ 0≤k≤N ∈ C N +1 ,
) evaluated at f = k 2N F s where k ∈ {0, . . . , N } reads R2-1 r=0 α (r) 2 X 1 ∆ τ (r) 2 = R1-1 r=0 α (r) 1 X 2 ∆ τ (r) 1 (5.14)
where denotes the component-wise Hadamard product.

With the above notation, in the following subsection we will present the CD framework for AER. This section is written with the help of the colleague Clement Elvira, co-author of a publication based on this work.

Echo localization with continuous dictionaries

By interpreting the FT of a Dirac as a parametric atom, we propose to cast the problem of RIR estimation into the framework of CD. To this aim, let the so-called parameter set be

Θ def = [0, T ] × {1, 2}, (5.15)
where T is the length (in time) of the lter. Then, the two desired lters h 1 , h 2 given by Eq. (5.2) can be uniquely represented by the following discrete measure over

Θ µ = 2 i=1 Ri-1 r=0 α (r) i δ (τ (r) ,i) .
(5.16)

where δ (τ (r) ,i) denotes the Dirac measure which is di erent from the Dirac function used when modeling the RIRs. The need of de ning a measure over the parameter set Θ makes easier the parametrization of the problem in the context of CD. For instance, it is possible to better de ne operations which are used in the algorithms and in the literature to solve this type of problems. Moreover, the rationale behind Eq. (5.15) and Eq. (5.16) is as follows. A couple of lters is now represented by a single stream of Diracs, where we have considered an augmented variable i indicating to which lter the spike belongs.

For instance, a Dirac measure at (τ, 1) indicates that the lter 1 contains a Dirac at τ .

The set M + (Θ) of all unsigned and discrete Radon measures over Θ (i. e., the set of all couples of lters) is equipped with the total-variation norm (TV-norm) µ TV 46 . We now de ne the linear observation operator

A : M + (Θ) → C N +1 ,
46 See [START_REF] Rudin | Real and complex analysis (mcgraw-hill international editions: Mathematics series)[END_REF]] for a rigorous construction of measures set and the TV-norm.

which is such that

Aδ (τ,i) =    -X 1 ∆ τ if i = 1 +X 2 ∆ τ if i = 2. ∀(τ, i) ∈ Θ.
(5.17)

Therefore, by the linearity of the observation operator A, the relation Eq. (5.14) can be rewritten as (5.18) where 0 N +1 is a N + 1-length vector of zeros. Before continuing our exposition, we note that the anchor constraint can be written in a more convenient way. Indeed, the constraint µ({(0, 1)}) = 1 ensures the existence of a Dirac at 0 in the lter 1. Then, the targeted lter reads µ = δ (0,1) + μ (5.21-P 0 TV )

Aµ = 0 N +1 ,
Note that (5.21-P 0 TV ) has to be interpreted as a natural extension of the wellknown basis pursuit problem to the continuous setting. Indeed, for any nite discrete measure µ = R-1 r=0 α (r) δ (τ (r) ,i) , the TV-norm of µ returns to the 1 -norm of the coe cients, i. e., µ TV = R-1 r=0 α (r) . Finally, Eq. (5.20) can be exploited to take into account noise during the measurement process (i. e., n i = 0 in Eq. (5.1)), as well as approximation errors (see Eq. (5.11)-Eq. (5.14)). In that case, the rst equality constraint in (5.21-P 0 TV ) is relaxed, leading to the so-called Beurling-LASSO (BLASSO) problem

µ = arg min µ∈M+(Θ) 1 2 y -Aµ 2 2 + λ µ TV s.t. µ({(0, 1)}) = 0. (5.22-P λ TV )
We emphasize that although continuous Radon measures may potentially be admissible, the minimizers of Eq. (5.22-P λ TV ) are guaranteed to be streams of Diracs [Bredies and Carioni 2020, Theorem 4.2]. In addition, although problem Eq. (5.22-P λ TV ) seems to depend on some regularization parameter λ, we describe in § 5.4.4 a procedure to automatically tune it to recover a desired number of spikes. Finally, note that the problem in Eq. (5.22-P λ TV ) is convex with linear constraints over the parameter set Θ. Therefore, theoretically, the problem can be solved exactly. However, optimizing over the in nite-dimensional space of measures is not possible, and hence non-convex optimization with respect to the measures parameters is performed instead.

From LASSO to BLASSO

In order to better understand the proposed approach based on the BLASSO algorithm, we can present it in light of the LASSO formulation.

arg min u 1 2 v -Au 2 2 + λ u 1 s.t. u ≥ 0 ↓ arg min u 1 2 y -Aµ 2 2 + λ µ TV s.t. µ ∈ M + (Θ)
Now, some parallelism can be envisioned:

• From dictionary to operator: The matrix A is typically referred to as dictionaries. Then selecting the l-th column of the dictionaries, i. e. Ae l , means selecting an echo at location l-th w. r. t. the vector u = h[1 :]. In the context of CD, the dictionary is translated into the operator A thanks to the closed-form of the atom based on the Fourier theory. Therefore, A(δ τ ) can be seen as the selection of an echo at location τ ∈ [0, T ] ms.

• Solution: The LASSO-like approach promotes a solution u = h[1 :] which is sparse and non-negative vector. In the BLASSO, this is translated assuming the channels being spare non-negative measures, i. e., µ = r α (r) δ(tτ (r) ).

• Sparsity: while in the initial case, the sparsity is enforced by the 1 -norm, in the second case it is pursued with the TV-norm.

• Solver: the former optimization problem can be solved with standard LASSO solvers, while for the latter a gradient-descent algorithm is used.

The resulting algorithm

The algorithm used to solve Eq. (5.22-P λ TV ) is an of instance the sliding Frank-Wolfe algorithm proposed in [START_REF] Denoyelle | The sliding Frank-Wolfe algorithm and its application to super-resolution microscopy[END_REF]] to solve Eq. (5.22-P λ TV ). Detailed descriptions of the steps of the algorithm are given in [Di Carlo et al. 2020, Supplementary Material] and in § 12.2. In a nutshell, the algorithm iterates over the following steps until a condition on the cost function is met.

1. Anchor constraint. At rst the anchor constraint in added arbitrarily on one of the two lters. This is used to initialize the two lters.

2. Local cost based on Cross-relation. For both the lters, a local costfunction derived from the cross-relation for both the lters is computed. At this step either the initialization or a previously found solution is used.

3. Find the maximizer. A new candidate echo's location is found as maximizer among the two local cost functions of the previous step.

4. Update the amplitudes. By solving a non-negative LASSO problem, all the echo's amplitude coe cients estimated until this point are updated.

5. Joint re nement. The position and the coe cient of the current solution are jointly re ned to ease numeric resolution using the original cost function.

6. Current solution and repeat. The algorithm stops as soon as an iterate satis es the rst order optimality condition associated to the convex problem. Otherwise, the algorithm iterates from step 2. using the current solution as input.

These steps are illustrated in Figure 5.3.

Homotopic path for λ estimation

Existing works, as well as the proposed one, rely on of the regularization parameter λ to weight the sparsity penalty. However, this becomes an hyperparameter that needs to be carefully tuned according to the input data. Instead, we propose to compute a path of solutions to automatically estimate it in Eq. (5.22-P λ TV ). In the context of sparse optimization this technique is also referred to as homotopic path. More precisely, let λ max be the smallest value of λ such that the null measure is the solution to Eq. (5.22-P λ TV ). It can be shown that λ max is upper bounded by max θ∈Θ |y Aδ θ |. Starting from z = 1 and the empty lter, we consider a sequential implementation where the solution of Eq. (5.22-P λ TV ) is computed for λ (z) = 10 -0.05z λ max until the desired number of spikes is found in each channel when incrementing z. For each λ (z) , we search for a solution of Eq. (5.22-P λ TV ) with the solution obtained for λ (z-1) as a warm start.

E

The proposed method (BLASTER) is compared against the non-negative 1norm method (BSN) of [START_REF] Lin | Blind sparse-nonnegative (BSN) channel identi cation for acoustic time-di erence-of-arrival estimation[END_REF]] and the iterative 1 -norm approach (IL1C) described in [Crocco and Del Bue 2016]. 47 The problem is formulated 47 Reference implementations for IL1C and BSN were kindly provided by the authors of [Crocco and Del Bue 2016].

as estimating the time locations of the rst R = 7 strongest components of the RIRs for 2 microphones listening to a single sound source in a shoebox room. It corresponds to the challenging task of estimating rst-order early re ections. The robustness of the methods is tested against di erent levels of noise (SNR) and reverberation time (RT 60 ).

The quality of AER estimation is assessed in terms of precision 48 in percentage 48 Since only K time locations are considered in both the ground truth and the estimation, precision and recall are equal.

as in the literature of onset detection [START_REF] Böck | Evaluating the Online Capabilities of Onset Detection Methods[END_REF]] and the RMSE in samples. Both metrics evaluate only the matched peaks, where a match is de ned as being within a small window τ max of a reference delay. These two metrics are similar to the ones used in [START_REF] Crocco | Room impulse response estimation by iterative weighted l 1-norm[END_REF].

For this purpose we created three synthetic datasets of 1000 observations each, which are summarized in Table 5 D (valid) is used for tuning the hyperparameter λ and the peak-picking parameters for IL1C and BSN using RT 60 and SNR randomly drawn from U [0, 1] (sec) and U [0, 20] (dB) respectively; D SNR features SNR value uniformly sampled in [0, 6, 14, 20, ∞] while the RT 60 is kept xed to 400 ms; akin the D RT60 is built sampling RT 60 value uniformly in [200,400,600,800,1000] ms keeping SNR to 20 dB. Moreover, while for D (valid) broadband signals (white noise) are used as the source, for D SNR and D RT60 speech utterances from the TIMIT dataset [START_REF] Garofolo | TIMIT acoustic-phonetic continuous speech corpus[END_REF]] are also included. The signal duration is kept xed to 1 s with sampling frequency F s = 16 kHz. For a given RT 60 value and room with random dimensions, a unique absorption coe cient is assigned to all surfaces based on Sabine's formula (Eq. (2.17)). Then, the two microphones and the source are randomly positioned inside the room. The parameters of the audio scene are then passed as input to the pyroomacoustics simulator [Scheibler et al. 2018b], which returns the corresponding RIRs as well as the o -grid echo delays and attenuation coe cients computed with the Image Source Method (ISM) [START_REF] Allen | Image method for e ciently simulating small-room acoustics[END_REF]. Note that when generating the data, no samples have been pruned to match any minimal separation condition. To generate the microphone signals, an over-sampled version of the source signal is convolved with ideal RIRs at high frequency (F s = 1024 kHz) made up of on-grid Diracs. The results are later resampled to meet the original F s and Gaussian white noise is added to meet the given SNR value.

Finally, as described throughout Chapter 4, IL1C and BSN uses tuned peak picking step to identity the echoes. Here the same peak picking technique provided with reference implementation of these methods was used and tuned on a small validation set. are reported in Figure 5.4, Figure 5.5 and Table 5.3. Here, for both RMSE and precision and for both broadband and speech signal, the metrics are displayed against the dataset parameters. We observe that BSN performs worst in all tested conditions, possibly due to its strong reliance on the peak picking step. For R = 7 or higher, BLASTER yields similar or slightly worse performance than IL1C for the considered noise and reverberation levels, with decreasing performance for both as these levels increase. Using speech rather than broadband signals also yields worse results for all methods. However, the echo timing RMSE is signi cantly smaller using BLASTER due to its o -grid advantage. We also note that BLASTER signi cantly outperforms IL1C on the task of recovering R = 2 echoes. As showed in Tab. 5.3, in mild conditions (RT 60 = 200 ms, SNR = 20 dB), up to 68% of echoes can be retrieved by BLASTER with errors lower than a quarter of sample in that case. This is promising since the practical advantage of knowing the timings of two echoes per channel has been demonstrated in [Di Carlo et al. 2019] (See Chapter 10), and in [Scheibler et al. 2018c

C

In this chapter we presented a novel knowledge-driven blind, o -grid echo retrieval method, based on the framework of continuous dictionaries. The particular "knowledge" we used is the echo model for the early part of the RIRs. The main motivation behind this approach is to overcome the pathological limitation of classical methods for BCE, discussed in the previous chapter. Despite an heavy mathematical formulation, it can seen as as the continuous extension of a LASSO problem used for addressing BCE. Comparisons with state-of-the-art approaches on various noise and reverberation conditions show that this method performs best when the number of echoes to retrieve is small. Future works will include many exciting directions, such as:

• extending the framework to multichannel recordings using the Multichannel cross-relation, as already envisioned by the related works [START_REF] Crocco | Room impulse response estimation by iterative weighted l 1-norm[END_REF]Lin et al. 2008];

• compare this approach with other o -grid AER methods [Tukuljac 2020, Chapter 6];

• adapt this approach to deal with ReTF which allow for source-independent acoustic features (See Chapter 11);

• use deep learning approaches to estimate the level of sparsity (a. k. a. number of the most relevant echoes) in the RIRs;

• validate the approach on real-world recordings, such as the one provided by the DECHORATE dataset (See Chapter 7).

Lantern: Data-driven Acoustic Echo Retrieval

• ICASSP2018 Paper • Code • ICASSP2018 Poster
learn the mapping from microphone recordings to the echoes' timings. After presenting a quick overview of deep learning techniques ( § 6.2), we will present a rst simple model to estimate the echo coming from a close surface ( § 6.3). This case is motivated by an application in Sound Source Localization (SSL), which will be discussed in details in Chapter 10. Finally, we will present a possible way to achieve a more robust estimation ( § 6.4) and discuss a possible way to scale this approach to multiple echoes ( § 6.5).

Deep learning notions come from readings of materials available in standard machine learning textbooks, such as [START_REF] Bishop | Mixture density networks[END_REF][START_REF] Goodfellow | Deep learning[END_REF], and the recent comprehensive work of [START_REF] Purwins | Deep learning for audio signal processing[END_REF]] oriented towards audio signal processing. In the remaining sections, we will present part of the previously published work [Di Carlo et al. 2019] and ongoing research.

Di Carlo et al., "Mirage: 2D source localization using microphone pair augmentation with echoes"

It is essential to say that the approaches presented here are part of a rst investigation in echo-aware SSL. Therefore they are profoundly interconnected with Chapter 10, which can be considered as a follow-up on the earlier work [START_REF] Gaultier | VAST: The virtual acoustic space traveler dataset[END_REF]] authored by colleagues. With that being said, we will mainly focus on a simple yet common scenario where we estimate only the rst and strongest echo per channel. The generalization to multiple echoes will only be discussed as future work.

I

Recently, Deep Neural Networks (DNNs) have captured the attention of many research elds due to recent advances that allow for faster implementation and training, as well as for achieving considerable performance improvement. Therefore, DNNs have been widely used in many domains. They belong to the class of supervised and data-driven models, where the training step is performed on labeled data, namely input-output pairs. The inclusion of these models in audio signal processing tasks has led to a considerable improvement in performance, in particular in audio and music source separation, speech enhancement, and source localization (see related sections in Chapter 8). Moreover, with respect to traditional machine learning methods, the use of deep learning models presents several advantages. They are exible and adaptable across tasks; for instance, Convolutional Neural Network (CNN)based models from Computer Vision can be adapted to audio source separation (e. g., [START_REF] Xiao | Deep beamforming networks for multi-channel speech recognition[END_REF][START_REF] Sainath | Multichannel signal processing with deep neural networks for automatic speech recognition[END_REF][START_REF] Perotin | CRNN-based joint azimuth and elevation localization with the Ambisonics intensity vector[END_REF]), and to source localization (e. g., [START_REF] Chakrabarty | Broadband DOA estimation using convolutional neural networks trained with noise signals[END_REF][START_REF] Vesperini | Localizing speakers in multiple rooms by using deep neural networks[END_REF][START_REF] Nguyen | Autonomous sensorimotor learning for sound source localization by a humanoid robot[END_REF]Salvati et al. 2018]). Furthermore, DNNs reduce -even sometimes remove -the step of designing hand-crafted features, by including feature learning as part of the learning process.

Other supervised learning methods have been proposed in the audio processing

At rst, the Gaussian Locally-Linear Mapping (GLLiM) framework based on Gaussian Mixture Model (GMM) was considered to address the AER problem. Then, we oriented our attention towards DNN models, which gave satisfactory results. It seemed that the local-linearity assumption of GLLiM was in contrast with the highly non-linear nature of the mapping between echoes and observations. Nevertheless, we recognize other bene ts in using the GLLiM framework, which cannot be found in the DNNbased approach, such as, the ability to generalize to missing data at the input or including prior statistical knowledge in the model.

literature, such as GMM-based frameworks [Deleforge et al. 2015a;[START_REF] Gaultier | VAST: The virtual acoustic space traveler dataset[END_REF]]. This framework allows to build partially informed, invertible regression models based on probabilistic prior knowledge and are competing with DNN models in case of a small training dataset. However, they require stronger assumptions on the nature of the mapping between input-output pairs, such as (local-)linearity or independence between variables.

The use of supervised learning models (not only the DNNs), presents some disadvantages. One of the most obvious ones is their dependence on annotated data. It results in a signi cant bottleneck in audio signal processing tasks because collecting and annotating comprehensive real-world acoustic data is not trivial, since it requires proper tools, expertise, and time. To overcome this issue, a standard approach nowadays is to use training data generated by simulators. We will refer to this approach as virtually supervised learning and it provides great versatility, for instance:

• many di erent acoustic conditions can be included in the training data, e. g., di erent Reverberation Time (RT 60 ) and Signal-to-Noise-Ratio (SNR) levels;

• a precise (up to computer precision) annotation of the data is directly available, e. g. spatial or temporal information on the audio scene events;

• and, the data can be potentially re-used for di erent applications, e. g., localization, separation, diarization, etc.;

Besides, the solutions obtained in a data-driven fashion su er from bias depending on the dataset used and may not generalize to real-world scenarios.

For instance, authors of [Deleforge et al. 2015b] showed, in a SSL context, that training on real data collected in one part of a room and testing in another part of the same room does not generalize well. Interestingly, the works in [Deleforge et al. 2015a;[START_REF] Nguyen | Autonomous sensorimotor learning for sound source localization by a humanoid robot[END_REF] propose an automatic approach for collecting and annotating real data automatically as a pre-calibration step. However, this approach is possible only with speci c technologies and equipments (in both works, the authors programmed a humanoid robot equipped with a binaural hearing system that automatically builds a training set). The following section will give a quick overview of relevant models used in our work.

6.2 D

Multi-layer perceptrons

Multi-layer Perceptrons (MLPs) are simple and basic modules of DNNs and are also known as fully-connected or dense layers. They consist of a sequence of layers, each of which de ned by an a ne vector transformation followed by a an entry-wise non-linear function:

y = f (Wx + b),
where

• x ∈ R Din is the input,

• y ∈ R Dout is the output,

• and b ∈ R Dout and W ∈ R Dout×Din are the bias vector and the weight matrix, respectively.

The function f (•) is a non-linear activation function, which allows the model to learn non-linear structures. Models built with these layers are typically used to map an input to a representation space where problems (classi cation and regression) can be addressed more easily. The main drawback of these simple models is that they are not invariant to scaled or shifted inputs. Although scaling can be compensated by data normalization, they are often not suitable to capture temporal-and frequency-variations of audio signals. Nevertheless, MLPs are used in combination with other type of layers, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs).

N constitute one of the key features of DNNs. Thanks to these, the model can achieve more expressive power with respect to linear models [START_REF] Goodfellow | Deep learning[END_REF]. Without these functions, it can be shown that the composition of a ne transformations is equivalent to a single a ne transformation. This means that, the activation functions make the model capable of accounting for more complex relationships between the input and the output than an a ne transformation. Typical examples of these functions are the hyperbolic tangent, the Sigmoid function, and the Recti ed Linear Unit (ReLU).

Convolutional neural networks

The CNNs consist of convolutional layers that have been introduced to overcome some limitations of the simple linear ones. In a nutshell, they consist of learnable kernel functions that are convolved with their input. Therefore, they are characterized by

• shift-invariance properties, thanks to the convolution operation;

• reduced model complexity, since the same kernel can be used at di erent input location;

• the ability to detect local structures at a di erent levels of abstraction in the network.

Using a Python-like notation, a 2D-convolutional layer is de ned by the following equation:

Y[:, :, k] = f ( I-1 i=0 W[:, :, i, k] * X[:, :, i] + b[:, :, k]),
where

• X[:, :, i] ∈ R F ×T ×I and Y[:, :, k] ∈ R A×B×K are input and output tensors, respectively.

• i ∈ [0, I] denotes the channel dimensions; 49 49 In the deep learning community, input dimensions are often referred to as channels Based on the application, such channels do not necessarily correspond to the channels of microphone recordings.

• k ∈ [0, K] denotes the channel dimension in the output;

• the tensors W and b denotes the weight and bias tensors, respectively;

• f is an activation function;

• * denotes discrete convolution;

A CNN can be designed with either 1D, 2D or 3D convolutions (e. g., along time, frequency and channel dimension), or a combination of them. In general, in audio, 1D convolution layers are used to process the time-domain or frequency-domain input, whereas 2D convolutional layers are used for time-frequency representations, such as spectrograms. The output of a convolutional layer consists of a collection of convolved versions of the input signal, which are typically referred to as feature maps. It is common to use CNN architectures combining multiple convolutional layers with pooling layers in between. Pooling layers are used to down-sample the features maps. In addition, to reduce the model complexity (i. e., number of free-parameters), pooling layers subsequently reduce the size of the data as the model goes "deeper". In this way, deeper layers can integrate larger extents of the data and extract spatial (in the sense of the input dimensions) features at a larger scale. Typical pooling operators are max-pooling and mean-pooling, which samples non-overlapping patches by keeping the largest and the average value in the region, respectively.

Hybrid architectures

As mentioned earlier, modern DNN architectures consist of a combination of di erent types of layers. For instance, CNNs are used to overcome the lack of shift and scale invariance that MLPs su er from, and allow to the extract of spatial features. In contrast, MLPs o er a simple and generic mappings from high-dimensional spaces to lower ones, suitable for classi cation and regression problems. Therefore, hybrid architectures are now the standard for deep learning models.

Learning and optimization

A DNN model consists of thousands of parameters θ. In order to learn a speci c task, such as regression or classi cation, they need to be optimized. To optimize the parameters θ, a variant of the gradient descent algorithm is usually implemented. A loss function L θ ( Ŷ, Y) measure the di erence between the predicted values Ŷ and the desired ones Y. The optimization process then iteratively updates the parameters θ so the loss function is minimized. In the case of vanilla gradient descent this writes:

θ ← θ -η ∇ θ L θ ( Ŷ, Y),
where η is referred to as the learning rate and controls the length of the gradient step at each iteration. Due to the composite structure of the DNNs, the gradient ∇ θ L θ ( Ŷ, Y) is computed via the chain rule, using the back propagation algorithm. Most gradient descent variants used today adapt the learning rate along iterations, such as the popular Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF].

The Stochastic Gradient Descent is a variant of the gradient descent algorithm which has become the standard for training DNNs today. It was introduced to improve convergence and to solve computational and memory load issues occurring when using training sets. It approximates the gradient at each step on a mini-bach of data samples. Finally, we would like to mention the following common techniques used to avoid over tting and speed up the convergence.

• early stopping allows to stop the training upon some conditions on the loss function (or other metrics), for instance, when it is not improving after certain amount of iterations. This criteria is typically computed on a dedicated data set known as validation set to avoid over tting;

• batch normalization consists of scaling the data by estimating their statistics during training, which usually leads to better performance and faster convergence;

• dropout is a regularization technique that reduces over tting by randomly omitting some network units during training.

In this chapter, we will propose a framework based on the following hybrid approach: we will use DNN to estimate echoes' properties, and later, in Chapter 10, we will use such quantities to address SSL problem. As a rst step, we will use DNNs to estimate the timings of the rst and strongest echo in each channel from stereophonic recordings. This setup is related to "table-top" scenarios, which are commonly encountered in typical home smart audio devices and it is motivated by an echo-aware Sound Source Localization (SSL) application that will be discussed in Chapter 10. To this end, we consider the following simple setup: two microphones placed close to a surface, as illustrated in Figure 6.1. The reason why we consider only one microphone pair is that this approach can be generalized to microphone arrays using data-fusion-like techniques. In fact, by considering all the pairs, it is possible to aggregate their estimation in a second stage. 50 In the reminder 50 An example of such techniques is the Steered Response Power with Phase Transform (SRP-PHAT) [START_REF] Dibiase | Robust localization in reverberant rooms[END_REF]] algorithm used for SSL, which uses the knowledge of the microphone array geometry to aggregate the contribution of each microphone pair.

of this chapter, we will discuss how to achieve robust rst-echo estimation.

Proposed approach

Our approach is to train a DNN on a dataset simulating the considered closesurface scenario. Under this assumption and the Short Time Fourier Transform (STFT) signal model presented in Eq. (3.25), we can consider the following simpli ed model for the Room Transfer Functions (RTFs),

H i [k] = R=1 r=0 α (r) i e -i2πf k τ (r) i + ε i [k],
(6.1) where τ (r) i and α

(r)

i are the echoes' times of arrival and amplitudes, respectively. Here, f k = kF s /F denotes k-th of F frequency bins and the error term ε i [k] collects later echoes, the reverberation tail, di usion, and noise. As we consider only the rst echo per channel, we have R = 1. Given the close-surface scenario, the rst echo is the earliest and strongest.

We model the AER problem as a multi-target regression problem, namely, the outputs of the model are multiple real-valued parameters. Following the approaches suggested in [Deleforge et al. 2015a;[START_REF] Gaultier | VAST: The virtual acoustic space traveler dataset[END_REF], we consider the instantaneous Interchannel Level Di erence (ILD) and the Interchannel Phase Di erence (IPD) as input features. As discussed in § 3.3.1, ILD and IPD can be consider as a particular case of Relative Transfer Function (ReTF) in a single source scenario. They can be estimated from the STFT of the microphone signals as follows

   ILD[k] = 20 log 10 1 T T l=1 X2[k,l] X1[k,l] , IPD[k] = 1 T T l=1 X2[k,l]/|X2[k,l]| X1[k,l]/|X1[k,l]| , (6.2)
where X i [k, l] is the STFT of the i-th microphone. In practice we do not consider the IPD in radiants, but rather its complex exponential, denoted as

IPD.

More precisely, the input of the network is ξ = ILD, Re IPD , Im IPD , namely, the concatenation of the above features in all frequencies into a single vector.. Here Re{•} and Im{•} denote real and imaginary part operators, respectively. Finally, the entries of ξ corresponding to IPD[0] are discarded because always equal 1. Initial investigations showed that learning the echoes amplitudes yielded large estimation errors, probably due to the complexity of the task. Thus, we considered echo timings with respect to the direct-path TOA in a reference microphone for the following three main reasons: rst, the origin of time cannot be known in the blind setting; second, motivated by an application to passive SSL discussed in Chapter 10, Time Di erences of Arrival (TDOA) can be easily converted to angles. Then, the timings of the rst echoes in the two channels can be fully parameterized by the following three quantities:

             τ TDOA = 1 c x 2 -s -1 c x 1 -s = τ (0) 2 -τ (0) 1 [s], τ iTDOA = 1 c x2 -s -1 c x1 -s = τ (1) 2 -τ (1) 1 [s], τ TDOE1 = 1 c x1 -s -1 c x 1 -s = τ (1) 1 -τ (0) 1 [s], τ TDOE2 = 1 c x2 -s -1 c x 2 -s = τ (1) 2 -τ (0) 2 [s], (6.3)
where xi denotes the position of the image of the microphone at position x i with respect to the re ector. Note that, since τ TDOE,2 = τ iTDOA + τ TDOE,1τ TDOA , only the rst three will be considered. These three quantities are directly connected to the early parts of the RIRs, as illustrated in Figure 6 V = [τ TDOA , τ iTDOA , τ TDOE1 ] ∈ R 3 be the vector of parameters of interest. Since it will be useful later, let de ne V = {TDOA, iTDOA, TDOE 1 } the set of three indexes denoted by TDOA, Image TDOA (iTDOA), Time Di erence of Echoes (TDOE).

Since we use ReTF-based features to estimate echoes, the proposed approach is dubbed LeArNing echo regression from room TransfER functioNs (LANTERN).

While, given the early part of the RTF, the V is uniquely de ned, the opposite is not. Therefore, in general, di erent values of V can yield the same ReTF and hence the same ILD/IPD features. In particular, this happens when τ

(1) 1 - τ (0) 1 = τ (1) 2 -τ (0)
2 for r = {0, 1}. That is, TDOEs are equal, or equivalently, the TDOA is equal to the iTDOA. Geometrically, this correspond to the case where real and image source have the same angle of arrival, which occur in very speci c source / array / re ector con gurations. Since this degenerate case is fundamentally unsolvable in the blind setting, we preventively pruned, all the entries with |τ TDOAτ iTDOA | < 10 -6 from all the dataset. In particular, they are removed from the training set so that the training is not a ected by these degenerate cases. Moreover, these are degenerate situations are known to be non-solvable and are not included in the evaluation to not bias the metrics.

Here we use the simple MLP architecture described in § 6.2.1. This model consists of a D-dimensional input layer, a 3-dimensional output layer, and 3 fully connected hidden layers with respective input sizes 500, 300, and 50. ReLU activation functions are used except at the output layer, and each hidden layer has a dropout probability p do = 0.3 to prevent over tting. We use the Mean Squared Error (MSE) loss function for training, that is,

L θ (V, V ) = 1 |V| ξ∈Ξ v∈V |τ v,ξ -τv,ξ | 2 (6.4)
where v indexes one of the TDOAs in V, ξ denotes a sample index in the mini-batch Ξ, and θ contains the model parameters.

The normalized Root Mean Squared Error (nRMSE) 51 is taken as validation 51 The nRMSE takes values between 0 (perfect t) and ∞ (bad t). If it is equal to 1, then the prediction is no better than a constant.

metric to assess the quality of the estimation V . Here we consider the vectors τ v , τ v ∈ R N containing the reference and estimated value for all the N samples in the set. The nRMSE is de ne as

nRMSE(τ v , τ v ) = τ v -τ v 2 τ v -avg ξ (τ v ) 2 , (6.5)
where • 2 denotes the 2 norm and avg ξ (•) is the average operation over the sample ξ in the set. The network is manually tuned on a validation set to nd the best combination of number of hidden layers, layer sizes and p do .

Data, implementation and experimental results

For training and validation of the MLP we generate many random, shoe-box room con gurations using the software presented in [START_REF] Schimmel | A fast and accurate "shoebox" room acoustics simulator[END_REF]]. This software implements both the Image Source Method (ISM) for simulating re ections and a ray-tracing algorithm for di usion § 2.3.2. Room lengths and widths are uniformly drawn at random in [3, 9] m, and heights in [2, 4] m. Random source and microphone positions are used, respecting the closesurface scenario. 52 In particular, the microphones are at most 30 cm from the 52 A rejection-sampling strategy was used to approximate uniform distributions.

close-surface, placed 10 cm from each other. The (frequency-independent) absorption coe cients of the close-surface is sampled uniformly in ]0, 0.5[, while the ones of the other walls' are sampled in ]0.5, 1[. The same realistic di usion pro le [START_REF] Gaultier | VAST: The virtual acoustic space traveler dataset[END_REF] is used for all surfaces. Around 90, 000 RIRs are generated this way, with a RT 60 between 20 ms and 250 ms.

The RIRs are convolved with 1 sec of white-noise (wn) with no additional noise. All signals and RIRs are sampled at 16 kHz. The STFT is performed on 1024 points with 50% overlap. Finally the features are computed as in (6.2), that is taking the average over the entire duration of the sequence and yielding a vector of size D = 1534 for each observation.

While we validate the MLP on a portion of the dataset in a holdout fashion, the test is conducted on 200 new RIRs convolved with both wn and speech (sp) utterances. This set is generated similarly to the training and validation sets. Moreover, the test recordings are perturbed by external white noise at 10 dB Signal-to-Noise-Ratio (SNR) (wn+n, sp+n). The speech signals are speech utterances of various lengths (from 1 s to 6 s), randomly selected from the TIMIT corpus [START_REF] Garofolo | TIMIT acoustic-phonetic continuous speech corpus[END_REF], normalized with respected to their energy.

E

To check the validity of TDOA estimation with the proposed MLP model, we compare it to a baseline algorithm Generalized Cross Correlation with Phase Transform (GCC-PHAT) (see § 10.3.1). GCC-PHAT is a popular method for estimating the TDOA between two microphone recordings. It is known to achieve good performance in the case of broadband source signals. However, it was shown that as soon as the acoustic conditions become challenging due to strong early echoes, high reverberation level, and noise, the method performances decrease [START_REF] Chen | Time delay estimation in room acoustic environments: an overview[END_REF]].

TDOA estimation errors using the proposed approach and GCC-PHAT are presented in nRMSE for clean white noise and speech signals. However, the estimation of iTDOA and TDOE seems to be a more challenging task for such a simple MLP model. For both methods, the performance are generally worse on speech signals than on white noise signals, which is due to the spectral sparsity of speech signals. However, in the noiseless scenario, it is interesting to see that even a simple architecture trained on ReTF-based features and broadband data is somewhat able to generalize to speech. Unfortunately, when some external noise is added, the performances of the methods degrade in all the cases. This is a well-known and expected behavior for GCC-PHAT, and for the MLP it suggests that noise should also be considered in the training phase, as already investigated for instance in [Deleforge et al. 2015a;[START_REF] Chakrabarty | Broadband DOA estimation using convolutional neural networks trained with noise signals[END_REF]. Nevertheless, our results con rm the possibility of retrieving the strongest echoes from only two-microphone recordings, in the absence of noise.

R

The above study was presented in our published work [Di Carlo et al. 2019]. To overcome the limitations of of our simple MLP-based method, we introduced the following three modi cations:

• use a more elaborate architecture inspired by a state-of-the-art SSL method;

• include corrupting noise in the training data;

• use a robust loss function which is able to return a level of con dence on the estimations.

Motivated by their success, we consider a CNN architecture (see § 6.2.2). To this end, we inspire from the work of [START_REF] Chakrabarty | Broadband DOA estimation using convolutional neural networks trained with noise signals[END_REF][START_REF] Nguyen | Autonomous sensorimotor learning for sound source localization by a humanoid robot[END_REF] in SSL. As shown in Figure 6.3, the architecture consists of two convolutional modules made of a one-dimensional convolutional layer (Conv1D) followed by max-pooling along frequencies, followed by ReLU activation function and batch-normalization. The second part consists of a cascade of fully connected feed-forward (FF) layers (basically the MLP model discussed above).

In order to perform 1D-convolution in the correct dimension, we re-arranged the input so that the each of the features {ILD, Re{IPD}, Im{IPD}} is considered as a channel for the Conv1D. After each layer, a dropout probability p do = 0.3 is applied to prevent over tting. In the MLP model presented in the previous section, the output consisted only in the TDOAs V . As mentioned earlier, our nal goal is to generalize this approach to microphone arrays in a data-fusion-like fashion. To this end, we need to provide a con dence measure on the estimate. In our work [Di Carlo et al. 2019], we proposed to use predicted values and the errors on the validation set as means and variances to build a Gaussian priors (see Chapter 10 for details on this). Instead, here, we explicitly modify the CNN model to estimated these parameters. This idea recalls the Mixture Density Networks (MDNs) as presented in in [START_REF] Bishop | Mixture density networks[END_REF]], where the output of a neural network parametrizes a Gaussian mixture model. It is also the foundation of Deep Generative Networks (or generative DNNs) for which the two most striking examples are Variational Auto-Encoders (VAEs) [START_REF] Kingma | Auto-encoding variational Bayes[END_REF][START_REF] Rezende | Turning enemies into friends: Using re ections to improve sound source localization[END_REF], Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF]. The rationale behind this design choice is to allow the learning model to assess its prediction quality. In order to achieve this, both the loss functions and the network outputs need to be modi ed accordingly.

Conv1D (3 × 1) ReLU MaxPool (3 × 1) Conv1D (3 × 1) ReLU MaxPool (3 × 1) dropout ILD ReIPD ImIPD FF (6048 × 1024) ReLU FF (1024 × 512) ReLU FF (512 × 256) ReLU FF (256 × 128) ReLU dropout Reshape FF (128 × 3) τv FF (128 × 3) λ 2 v FF (128 × 3) νv FF (128 × 3) µv FF (128 × 3) νv FF (128 × 3) µv CN N V T CN N V N CN N V M LP (
T G is derived as follows. First, we modify the

The general form of the Gaussian probability density function is

N x; µ, σ 2 = 1 √ 2πσ 2 e -1 2 x-µ σ 2 .
network to return means and variances, namely

V N = µ v (ξ; θ), σ 2 v (ξ; θ) v for v = {TDOA, iTDOA, TDOE 1 }.
Moreover, we assume that the posterior probability of observing τ v for v ∈ {TDOA, iTDOA, TDOE 1 }, given the observation ξ in the mini-batch Ξ, follows a Gaussian distribution, namely

p(τ v | ξ; θ) ∼ N τ v ; µ v (ξ; θ), σ 2 v (ξ; θ) . (6.6)
From now on, we omit the dependency on (ξ; θ) for the sake of clarity. Finally, the corresponding training loss function is the negative the negative log-posterior probability over the batch Ξ, that is,

L N θ (V, V ) = -log ξ∈Ξ v∈V p(τ v |ξ; θ) c = 1 |V| ξ∈Ξ v∈V log σ 2 v + |τ v -µ v | 2 σ 2 v , (6.7)
where c = denotes equivalence up to a constant. Hence, for a given input ξ, the network attempts to nd mean and variance parameters maximizing the a posteriori probability of TDOAs according to the considered Gaussian model. The means are then used as TDOA estimates, while variances can be seen as a measure of uncertainty from the model. Hereafter we denote by VN the set of 6 network outputs. This approach can then be generalized to other this of distributions. For instance, we can consider a Student's t-distribution, which is more robust to outliers, due to its heavy-tailed nature. Using this distribution has already been proposed in the context of binaural SSL, e. g., in [START_REF] Zohny | Modelling interaural level and phase cues with Student's tdistribution for robust clustering in MESSL[END_REF][START_REF] Deleforge | Recti ed binaural ratio: A complex T-distributed feature for robust sound localization[END_REF].

T S ' can be derived similarly as the Gaus-

The general form of the Student's t probability density function is

T (x; µ, λ, ν) = Γ( ν+1 2 ) Γ( ν 2 ) √ πνλ 2 1 + (x -µ) 2 νλ 2 -ν+1 2 ,
where Γ(n) = (n -1)! is the Gamma function, de ned for positive integers.

sian case assuming a t-distribution posterior on the prediction. The corresponding negative log-posterior probability over the batch Ξ now writes

L T θ (V, V ) = ξ∈Ξ v∈V 1 2 log(ν v π v ) + 1 2 log(λ 2 v ) -log Γ ν v + 1 2 + log Γ ν v 2 + ν v + 1 2 log 1 + |µ v -τ v )| 2 ν v λ 2 v (6.8)
where Γ is the Gamma function. Similarly to the Gaussian case, for a given input, the network returns the 3 parameters of 3 t-distributions (µ v , ν v , λ v ), one for each index v ∈ V. The set of 9 network outputs is denoted by VT .

Experimental results

In order to validate the proposed approach, it is compared to the vanilla MLP model proposed above in the previous section. ReTF estimator or DNN handling missing data. Finally, the network is manually tuned on a validation set in order to nd the best combination of number of hidden layers and their sizes, which led to the model in Figure 6.3. Figure 6.4 shows the performances of the proposed models CNN V , CNN V N and CNN V T with respect to MLP V and the baseline GCC -PHAT for the task of estimating TDOA, iTDOA and TDOE. Again we report the performances in terms of the nRMSE computed on test samples. It can be seen that the proposed modi cations (CNN architecture, robust loss function, and considering noise in training) yield considerable improvement in performances, compared to

G C C P H A T M L P C N N C N N C N N Approach 0.0 0.2 0.4 0.6 0.8 nRMSE v = TDOA G C C P H A T M L P C N N C N N C N N Approach v = iTDOA G C C P H A T M L P C N N C N N C N N Approach v = TDOE1
F 6.4: Normalize root mean squared error (the lower, the better) for TDOA estimation using the proposed architectures on white noise source signals. the vanilla and baseline methods To give a sense to the values, a TDOA's nRMSE lower than 0.2 corresponds to an angular RMSE lower that 4 degree (more details in Chapter 10), resulting in a excellent estimation. In particular, the new methods are more robust to noise, as indicated by error bars. While both CNN V N and CNN V T outperform the CNN V for TDOA and iTDOA estimation, there is no signi cant 53 di erence between the three of them for TDOE 53 Signi cance assessed with a t-test for p < 0.05.

estimation.

Deeper insights about this can be gathered from Figure 6.5, where performances are reported for the di erent SNR levels considered in the test. These results suggest that using a simple MSE-based loss function leads to errors more sensitive to external noise. Moreover, they shows that TDOE estimation is a more di cult task than TDOA estimation, yielding bigger estimation errors as well as a smaller di erence between the three methods. The former task is probably made more challenging. In fact, the ranges that the TDOEs span are much larger values which highly depends not only on the source and microphone relative position, but also on the surface position and orientation. In general, there does not seem to be a signi cant di erence between using a Gaussian distribution or a Student's t-distribution at the network's output in our experiments. Nevertheless, the choice of modeling the output of the network according to di erent distributions may lead to di erences at the data-fusion step.

C

This chapter presented an attempt at using deep learning models to learn the complex mapping from stereophonic recordings to echoes' timings. As part of a rst investigation, we considered a simple, yet common scenario where two microphones are placed close to a re ective surface and attend a to a single sound source. The problem is then restricted to estimating the the relative delays between the direct path and the rst echo timings within the two channels. Moreover, this simpli cation is motivated by further application to echo-aware SSL, which will be proposed and tested in Chapter 10. We studied two standard DNN architectures, which are trained in a virtually supervised fashion on data generated by an acoustic simulator. Finally, we discuss three ways to improving such performances, i. e., training on noisy data, using a CNN-based architecture, and using robust loss functions. The investigation so far was limited to a microphone pair. Taking inspiration from data-fusion approaches, in Chapter 10, we will propose to aggregate the contributions of real and image microphone pairs to preform 2D SSL with only two microphones.

A natural follow-up of this work could consider the following directions.

• The rst one would be a generalization to more re ections, which is far from being trivial. Informal investigations spanned di erent training paradigms, such as, curriculum learning [START_REF] Bengio | Curriculum learning[END_REF]], teacherstudent learning [START_REF] Tu | Speech Enhancement Based on Teacher-Student Deep Learning Using Improved Speech Presence Probability for Noise-Robust Speech Recognition[END_REF], where the network is iterative trained to estimate more and more echoes. Another idea would be to use a collection of DNNs, each pre-trained for single echo estimation.

• Due to severe drops in performance observed using speech signals, we could consider more robust audio features (e. g., ReTF using stateof-the-art ReTF estimators) or DNN architectures designed to handle missing entries in the input feature. This latter direction was investigated in [Deleforge et al. 2015a] to make a GMM-based model able to generalize to speech data. However the best author knowledge, it is still an open problem which could be connected to the so-called (audio) "inpainting" inverse problem [START_REF] Bilen | Joint audio inpainting and source separation[END_REF]].

• Finally, another interesting direction would be to use physics-driven neural networks as explored in [START_REF] Karpatne | Physics-guided neural networks (pgnn): An application in lake temperature modeling[END_REF][START_REF] Nabian | Physics-driven regularization of deep neural networks for enhanced engineering design and analysis[END_REF][START_REF] Rao | Physics-informed deep learning for incompressible laminar ows[END_REF]Jin et al. 2020], In these work physically-motivated regularizers or entire loss-functions obeying physical laws are used to ensure the correctness of the solution or restrict the search space. It can be done by adding simple formulas, such as the temperature-density relation for uids as in [START_REF] Karpatne | Physics-guided neural networks (pgnn): An application in lake temperature modeling[END_REF] • Code repository

• Dataset data tichannel room impulse response (RIRs) including annotations of early echoes and 3D positions of microphones, real and image sources under di erent wall con gurations in a cuboid room. These data provide a tool for benchmarking recent methods in echo-aware speech enhancement, room geometry estimation, RIR estimation, acoustic echo retrieval, microphone calibration, echo labeling, and re ectors estimation. The database is accompanied by software utilities to easily access, manipulate, and visualize the data and baseline methods for echo-related tasks.

The material presented in the chapter is the result of a work done while visiting prof. Sharon Gannot and ing. Pinchas Tandeitnik at the Bar'Ilan University, Israel. The work described here, together with its continuation described in Chapter 11 will be submitted as a journal article to the EURASIP special edition Data-driven Audio Signal Processing: Methods and Apps.

I

As discussed § 4.4.1, many RIRs datasets are available online. However, most of them are speci cally designed for applications either to Speech Enhancement (SE) or to Room Geometry Estimation (RooGE). The main common drawback of these datasets in that they can not be easily used for other tasks than the one which they were designed for. In particular, SE-oriented datasets lack of a proper annotation of echoes in the RIRs or the absolute position of objects inside the room. Conversely, datasets for RooGE typically features design choices which are not suitable for many SE application, involving the positioning of microphone(s) and source(s). dEchorate was designed to ll this gap: a fully calibrated multichannel RIR database with accurate annotation of the geometry and echoes in di erent con gurations of a cuboid rooms with varying wall acoustic pro les. The database currently features 1800 annotated RIRs obtained from 6 arrays of 5 microphones each, 6 sound sources in 10 di erent acoustic conditions. All the measurements were realized in the acoustic lab at Bar-Ilan university following a consolidated protocol previously established for the realization of two other multichannel RIRs databases: the BIU's Impulse Response Database [Hadad et al. 2014] gathering RIRs of 91 di erent reverberation levels sensed by uniform linear arrays (ULAs); and

MIRaGE [Čmejla et al. 2019] providing a set of measurements of a source placed on a dense position grid. dEchorate is designed for Acoustic Echo Retrieval (AER) with linear arrays, and is more generally aimed at analyzing and benchmarking RooGE and echo-aware signal processing methods on real data. In particular, it can be used to assess robustness against the number of re ectors, the reverberation time, additive spatially-di use noise and non-ideal frequency and directive characteristics of microphone-source pairs and surfaces in a controlled way. Due to the amount of data and recording conditions, it could also be used to train machine learning models or as a reference to improve RIR simulators. The database is accompanied with a Python toolbox that can be used to process and visualize the data, to perform analysis or to annotate new datasets.

7.2 D

Recording setup

The recording setup is situated in a cuboid room with dimension 6 m × 6 m × 2.4 m. The 6 facets of the room (walls, ceiling, oor) are covered by acoustic panels allowing controllable reverberation time (RT 60 ). We placed 4 directional loudspeakers (direct sources) facing the center of the room and 30 microphones mounted on 6 non-uniform linear arrays (nULA) of 5 sensors each. An additional channel is used for the loop-back signal, which serves to compute the time of emission and detect errors. Each loudspeaker and each array was positioned close to one of the walls in such a way that the nature of the strongest echo can be easily identi ed. Moreover, their positioning was chosen to cover a wide distribution of source-to-receiver distances, hence, a wide range of direct-to-reverberant ratios (DRR). Further, 2 more loudspeakers were positioned pointing towards the walls (indirect sources). This was done to study the case of early re ections being stronger than the direct-path. Each linear microphone array consists of 5 microphones with non-uniform intermicrophone spacings of [4, 5, 7.5, 10] cm 54 . Each array is steered towards a 54 i. e. [-12.25, -8.25, -3.25, 3.25, 13.25

Measurements

The main feature of this room is the capability to change the acoustic pro le of each of its facet by ipping double-sided panels with one re ective and one absorbing face. This allows to achieve precise values of RT 60 that range from 0.1 to almost 1 second. In this dataset the panels of the oor were kept always absorbent.

Two types of sessions were considered, namely, one-hot and incremental. For the rst type, a single facet was placed in re ective mode while all the others were kept absorbent. For the second type, starting from fully-absorbent mode, facets were progressively switched to re ective mode one after the other until all but the oor were re ective, as shown in Table 7.2.

The dataset features an extra recording session. For this session, o ce furnitures were positioned in the room to simulate a typical meeting room with chairs and tables (See Figure 7.1). Theses recordings will be used in future works for asserting the robustness of echo-aware methods in a more realistic scenario?.

For each room con guration and loudspeaker, three di erent excitation signals were played and recorded in sequence: chirps, white noise and speech utterances. The former consists in a repetition of 3 Exponential Sine Sweep (ESS) signals of duration 10 seconds and frequency range from 100 Hz to 14 kHz interspersed with 2 seconds of silence. Such frequency range was chosen to match the characteristics of the loudspeakers. To prevent rapid phase changes and "popping" e ects, the signals were linearly faded in and out over 0.2 seconds with a Tuckey taper window 55 . Secondly, 10 seconds bursts of white 55 The code to generate the reference signals and to process them is available togheter with the data. Such code is based on the pyrirtool Python library.

noise and 3 anechoic speech utterances from the Wall Street Journal (WSJ) dataset [START_REF] Paul | The design for the Wall Street Journal-based CSR corpus[END_REF] were reproduced in the room. Through all the recordings, at least 40 dB of sound dynamic range was asserted and a room temperature of 24°± 0.5°and humidity of 80% were registered. Moreover, 1 minute of room tone (silence) and 4 minutes of di use babble noise were recorded for each session. The latter was simulated by transmitting di erent chunks of the same single-channel babble noise recording from additional loudspeakers facing the four corners of the room.

All the microphone signals were synchronously acquired and digitally converted to 48 kHz with 32 bits/sample using the equipment listed in Table 7.1. The polarity of each microphone was registered by clapping a book in the middle of the room.

D

RIRs are estimated with the ESS technique [START_REF] Farina | Simultaneous measurement of impulse response and distortion with a swept-sine technique[END_REF]] at 48 kHz: the signal of a microphone recording an ESS source is deconvolved by division in the frequency domain. Notice that the FT of the ESS signal is available in closed form and we used its DFT approximation.

RIRs annotation

The objective of this database is to feature o -grid annotations in the "geometrical space", namely microphone, wall and source positions, fully consistent with annotations in the "signal space", namely the echo timings within the RIRs. This results is achieved as follows:

(i) First, the ground-truth position of array and source centres are acquired via a Beacon indoor positioning system (bIPS). This system consists in 4 stationary bases positioned at the corners of the ceiling and a movable probe used for measurements which can be located within errors of ±2 cm. The elements of this system are shown in Figure 7.4. (iv) By employing a multilateration algorithm [START_REF] Beck | Exact and approximate solutions of source localization problems[END_REF], where the positions of one microphone per array serve as anchors and the TOAs are converted into distances, it was possible to localize image sources along side with the real sources. This step will be further discussed in Chapter 11.

Knowing the geometry of the recording room, we were able to manually label the echoes by iterating through steps (ii), (iii) and (iv).

T was chosen as a compromise between the bIPS measurements and the MDS output. While the former ones are noisy but consistent with the scene's geometry, the latter ones matches the TOAs but not necessarily the physical world. In particular, geometrical ambiguities such as global rotation, translation and up-down ips were observed. Instead of manually correcting this error, we modi ed the original problem from using only the direct path distances (dMDS) to considering the image sources' TOA of the ceiling as well in the cost function (dcMDS). Table 7.3 shows numerically the mismatch (in cm) between the geometric space (de ned by the bIPS measurements) and the signal space (the one de ned by the echo timings, converted in cm). To better quantify it, we introduce here a goodness of match (GoM) metric: it measures the fraction of ( rst-order) echo timings annotated in the RIRs matching the annotation produced by the geometry within a threshold. Including the ceiling information, dcMDS produces a geometrical con guration which has a small mismatch (0.41 cm T 7.3: Mismatch between geometric measurements and signal measurements in terms of maximum (Max.), average (Avg.) and standard deviation (Std) of absolute mismatch in centimeters. The goodness of match (GoM) between the signal and geometrical measurements is reported as fraction of matching echo timing for di erent threshold in milliseconds.

in average) in both the signal and geometric spaces with 98.1% of matching rst order echoes within a 1 ms threshold. Nevertheless, it is interesting to see that the bIPS measurements produce a good but less precise annotation.

Other tools for RIRs annotation

Finally, we want to mention that the following tools and techniques were found helpful in annotating the echoes.

T skyline consists in presenting multiple RIRs as an image, such that the wavefronts corresponding to echoes can be highlighted [START_REF] Baba | 3D room geometry inference based on room impulse response stacks[END_REF]. More precisely, it is the visualization of the L × N matrix H created by stacking column-wise N normalized echograms 57 , that is H l,n = 57 The echogram is de ned either as the absolute value or as the squared value of the RIR. 2012; [START_REF] Eaton | Time of arrival disambiguation using the linear Radon transform[END_REF]. In particular, the direct path of the RIR was manually isolated and used as an equalization lter to enhance early re ections from their superimposition and from background noise before proceed with peak picking. Each RIR was equalized with its relative direct path. As depicted in Figure 7.5, in some situation this process was necessary for correctly identifying the underlying TOAs' peaks.

D

for the same geometry in uenced the peaks' predominance in the RIR, hence facilitating its echo annotation. An example of RIRs corresponding to 2 di erent surface con gurations is shown in Figure 7.5: the reader can notice how the peak predominance changes for the di erent con gurations.

A 58 was used on equalized echograms 58 In this work, peaks are found using the Python peakutils library.

ηn (l) to provide an initial guess on the peak positions.

Limitations of current annotation

As stated in [Defrance et al. 2008b], we want to emphasize that annotating the correct TOAs of echoes and even the direct path in "clean" real RIRs is far from straightforward. The peaks can be blurred out by the loudspeaker characteristics or the concurrency of multiple re ections. However as showed in Figure 7.6, the proposed annotation was found to be su ciently consistent both in the geometric and in the echo space. Thus, no further re nement was done. This database can be used as a rst basis to develop better AER methods which could be used to iteratively improve the annotation, for instance including 2 nd order re ections.

T dEchorate

The dataset comes with both data and code to parse and process it. The data are presented in 2 modalities: the raw data, that is, the collection of recorded wave les, are organized in folders and can be retrieved by querying a simple database table; the processed data, which comprise the estimated RIRs and the geometrical and signal annotations, are organized in tensors directly importable in Matlab or Python (e.g. all the RIRs are stored in a tensor of dimension L × I × J × D, respectively corresponding to the RIR length in F 7.7: Sample view of the database table to retrieve the raw wave le and its attributes.

samples, the number of microphones, of sources and of room con gurations). Together with the data a Python package is available on the same website. This includes wrappers, GUI, examples as well as the code to reproduce this study.

In particular, all the scripts used for estimating the RIRs and annotating them are available and can be used to further improve and enrich the annotation or as baselines for future works.

* T introduced a new database of Room Impulse Response (RIR) featuring accurate annotation of early echoes and microphone positions. These data can be used to test methods in the room geometry estimation pipeline and in echo-aware audio signal processing. We will show some application in SE and RooGE in Chapter 11. Future works will explore di erent directions. By such as making this dataset freely available to the audio signal processing community, we hope to foster research in AER and echo-aware to improve the performance of existing methods on real data. Moreover, the dataset could be updated by including more robust annotations derived from more advanced algorithms for calibration and AER.

Part III ECHO-AWARE APPLICATION

Audio Scene Analysis meets Signal Processing

S

In this chapter, we present a selection of algorithms and methodologies which we identi ed as potential bene ciaries of echo-aware additions. At rst, we present a typical scenario that highlights some cardinal problems. Then, state-of-the-art approaches to address these problems are listed and commented in dedicated sections, highlighting the relationship with some acoustic propagation models, respectively. The content presented here serves as a basis for a deeper investigation conducted in each of the following chapters.

Here, we present some audio scene analysis problems that will be later discussed in their echo-aware extension. Following the last part's structure, this introduction gathers the common knowledge shared across the following chapters. Here we make a strong transition: we assume the echo properties are known a priori, so that our focus is only on the bene t of their knowledge. The literature for each of them is reviewed, but since it is vast and spans diverse scienti c research decades, we do not aim to cover it entirely. Moreover, since the following chapters are dedicated to these problems under the echo-aware perspective, this speci c literature is not considered here. The material presented here results from the personal synthesis of concepts and references available in the literature. Furthermore, some de nitions are digested from classical textbooks already used for this thesis, such as [START_REF] Vincent | Audio source separation and speech enhancement[END_REF]].

A S A P

As mentioned in the rst chapter, audio scene analysis aims to extract relevant information in the audio scene. Di erent types of information are estimated or inferred by solving speci c problems. Despite their diversity, most of these problems can be de ned with a common model.

Common scenario and model

Let there be a meeting room with well-de ned geometry. In it, J sound sources are located at determined positions, such as some speakers chatting while standing in the room, as in Figure 8.1. As an indoor scenario, all the elements of reverberation (in particular echoes) are present. Di use background noise is present as well, for instance, due to the air conditioner or car tra c outside. This whole audio scene is recorded by a device featuring a microphone array of I sensors. Furthermore we assume a static far eld scenario and we model each source and microphone as well-de ned points with coordinate s and x, respectively. This is a reasonable assumption in the context of table-top devices, such as smart home devices. Recalling the (discrete) time-domain signal model already discussed in § 3.1.1, the signal recorded at the i-th microphones reads

x i [n] = J j=1 h ij (x i |s j ) * s j [n] + n i [n], (8.1)
or alternatively, using the source spatial image signals,

c ij [n] = h ij (x i |s j ) * s j [n]x i [n] = J j=1 c ij [n] + n i [n]. (8.2)
Note that the lter h ij (x i |s j ) denotes the RIR where we intentionally highlight the dependencies on geometry, namely, accounting for the whole sound propagation for the source position s j to the microphone position x i . In fact, as discussed throughout Part I, we can decouple the information of indoor microphone natural recordings into two complementary sub-problems: the estimation of the RIRs (thus the mixing matrix) that account for only the sound propagation, and of the source signals accounting only the semantic content. The former problem can be seen as an instance a more general the Blind Channel Estimation (BCE) problem where the (acoustic) channel is the RIR (see the review in Chapter 4).

Problem formulation

The Audio Scene Analysis Problems presented already in the introductory chapter (See § 1.2) can now be extended and rewritten in terms of the above notation. Furthermore, we will consider here the only ones directly addressed in this thesis: room impulse response estimation, audio source separation, spatial ltering, sound source localization, and room geometry estimation.

As introduced in § 4.1, these problems can be said either informed or blind and the related scenario active or passive. These two dichotomies emphasize the amount of prior knowledge available for solving them. As opposed to the active scenario, where the source signal is known, transmitted, and available, the passive one considers only the microphone measurements. For instance, when addressing the active echo estimation problem or RIR measurement, the exact time of emission of the source signal is known, as well as the source signal itself.

The second dichotomy refers to the possibility of exploiting prior knowledge to solve the problem more easily. This information may derive from annotations and meta-data. In the community of audio source separation, the following de nitions were proposed in [START_REF] Vincent | From blind to guided audio source separation: How models and side information can improve the separation of sound[END_REF]]: as opposed to informed problems, for solving the blind ones, absolutely no information is given about the source signal or the mixing process. In between, there are semi-blind or guided problems: here general information is available, such as on the nature of the source signal (speech, music, environmental sounds), microphone position, recording scenario (indoor, outdoor, professional music), mixing process, etc.

In some books and works other categories of problems are de ned, such as weakly-guided, strongly-guided. Here we do not consider these distinctions.

In the considered echo-aware applications, the echoes properties build our prior knowledge on the problem. Therefore, according to the above taxonomy, the addressed problems are necessarily guided. In general and unless speci ed, this is the only knowledge we will assume to have. Based on this, we will now review some classical works for solving the above problems. I we will present the general overview of the literature related to the problems considered in this thesis: multichannel audio source separation, spatial ltering, and sound source localization. We will limit the discussion to the most relevant techniques adopted nowadays with respect to the acoustic propagation modeling. Later in the thesis, dedicated sections on echo-aware method to address these problems will be provided in each related chapter. Since room geometry estimation is mainly based on echo estimation and labeling discussed in § 4.3.1, its description is postponed to § 11.2.

O M A S S

Multichannel Audio Source Separation (MASS) refers to the process of extracting acoustic signals from multichannel mixtures featuring targets, interfering, and noisy sounds. In psychoacoustics, this problem is known as the cocktail party problem [START_REF] Cherry | Cocktail party problem[END_REF]], referring to the human ability to focus on a particular stimulus in a audio scene. This problem has interested mainly in two research elds in the audio signal processing community: speech and music processing. The scienti c literature on audio source separation is vast, still active, and spans decades. The problem covers a huge number of scenarios and use-cases, such as, number of microphones (single-vs. multi-channel recordings), the number of sources with respect to the number of channels (under-vs. over-vs. determined), the type of observed signals (speech vs. music), type of recordings (arti cial vs. microphone recordings) etc., and, of course, combination of them. Both share many methods, which are accordingly modi ed, taking into account scenarios and applications. In the context

Many publications have addressed this issue, including books [START_REF] Vincent | Audio source separation and speech enhancement[END_REF][START_REF] Makino | Audio Source Separation[END_REF], and overview papers [START_REF] Cardoso | Blind signal separation: statistical principles[END_REF][START_REF] O'grady | Survey of sparse and non-sparse methods in source separation[END_REF][START_REF] Gannot | A consolidated perspective on multimicrophone speech enhancement and source separation[END_REF][START_REF] Girin | Audio source separation into the wild[END_REF]].

of multichannel speech recordings, some of the most successful and popular methods used nowadays include [START_REF] Vincent | Probabilistic modeling paradigms for audio source separation[END_REF]]:

• spatial ltering, which uses spatial information only;

• TF masking, which uses spectral information only;

• Multichannel Wiener Filter (MWF) which combines the above two;

• and end-to-end regression which use powerful end-to-end (i. e., from waveforms to waveform) learning models (e. g., DNN).

In this thesis, we deliberately distinguish between spatial ltering, which will be discussed in the following subsection, and TF masking. TF masking relies on TF diversity of the sources and processes each mixture channel separately. In a nutshell, it involves computing the STFTs of the mixture channels, multiplying them by masks containing gains between 0 and 1. One of the most popular masking rules is Wiener ltering. For each time-frequency bin, the STFTs of the estimated source spatial images C ij of the j-th source at the i microphone, writes

Ĉij = |C ij | 2 J j=0 |C ij | 2 Wiener lter X i (8.3)
where X i is the STFT of the microphone channel. Here the dependency on the TF bin is omitted for clarity. In order to be computed, the Wiener lter requires estimating source spatial images statistics, in particular an estimate of the power in each TF bin (and discarding the phase). In these thesis we stress the di erence between source separation and spatial ltering. In the former, source signals and mixing lters statistics are needed to weigh each of the TF bins of the microphone channel's STFT. As opposed to, in the latter problem, the "mask" of one of the sources, that is the spatial lters, are estimate only based the mixing lters and noise statistics.

In multichannel recordings, a clear overlap exist between the two problems, and some techniques can be used reciprocally, e. g. in the case of MWF [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]. Furthermore the related research trends are now converging under the same umbrella of the so-called speech enhancement. The work [START_REF] Gannot | A consolidated perspective on multimicrophone speech enhancement and source separation[END_REF]] provide an uni ed framework merging source separation and spatial ltering model. Nevertheless, we will treat them as separated problems.

Moreover, the bene t of the TF masking approach is that the masks can be estimated in various ways. For instance, clustering and classi cation techniques [START_REF] Rickard | The DUET blind source separation algorithm[END_REF]] can be used to assign each TF-bin to each of the sources. Recently learning-based methods have been used in this sense on the same task [START_REF] Hershey | Deep clustering: Discriminative embeddings for segmentation and separation[END_REF][START_REF] Wang | Multi-channel deep clustering: Discriminative spectral and spatial embeddings for speaker-independent speech separation[END_REF]]. Alternatively, deep learning techniques can used to directly estimated the sources' TF, as done in one of the reference implementation [Stöter et al. 2019]. The work of [START_REF] Nugraha | Multichannel audio source separation with deep neural networks[END_REF], instead, uses a deep learning model build by unfolding the the popular Multichannel NMF source separation framework of [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF][START_REF] Sawada | Multichannel extensions of non-negative matrix factorization with complex-valued data[END_REF]].

The Multichannel NMF [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]] is one of the most successful framework for source separation using MWF, that is, combining spatial ltering and TF masking. It uses the NMF model for modeling the source Power Spectral Density (PSD) within a convolutive mixing model (i. e., spatial model, as discussed in § 3.2.5). It then deploys optimization-based framework for estimating both the mixing matrix and an estimate of the source PSD based on NMF. Once these PSD and lter parameters are estimated, separation can be achieved with MWF. One of the main advantage of this approach is that it allows to easily incorporated prior knowledge on the problem. Thanks to the narrowband approximation (Eq. (3.26)), estimation of spatial and semantic content are two complementary sub-problems that can be solved independently. This opens to many possibilities, such as, using pre-trained dictionaries to model source content [START_REF] Schmidt | Single-channel speech separation using sparse non-negative matrix factorization[END_REF][START_REF] Smaragdis | A sparse non-parametric approach for single channel separation of known sounds[END_REF], or using proper model for the mixing lters, such as RIRs, steering vectors, ReTFs, etc.

Here, we will focus on multichannel source separation based on Nonnegative Matrix Factorization (NMF). NMF refers to a set of technique to model spectra of complex sounds by a sum of basic components. Modeling sound structure is bene cial for source separation, since it make separation possible in many challenging scenario. Moreover, this approach allows to easily incorporate side information on both the sources and the acoustic propagation as will be shown later in the chapter.

However, the work in [START_REF] Luo | Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF] showed that even with oracle TF masks, the estimation is still a ected by artifacts. This limitation a ects all the approaches operating in the TF domain. To overcome this, end-to-deep deep learning models [START_REF] Luo | Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF][START_REF] Défossez | Music Source Separation in the Waveform Domain[END_REF]Tzinis et al. 2020] were developed and now hold the record in source separation. These models work directly in the time domain: both input and output are time-domain waveforms. This approach has prove to reach good separation quality, especially in terms of perceived sounds at the listener. Nevertheless, all deep learning methods rely on trained black-box models for which is hard to inject prior knowledge. Instead, Multichannel NMF-based frameworks accounts for this freedom.

"Classic" (pre-deep learning) MASS spans almost 25 years and hundreds of MASS techniques have been proposed. 59 With extreme simpli cation, one can be broadly grouped such techniques according to how they model sound propagation of the mixing process:

• (no propagation) those that simply ignore it, e. g. [START_REF] Kokkinis | A Wiener lter approach to microphone leakage reduction in close-microphone applications[END_REF][START_REF] Roux | Deep NMF for speech separation[END_REF][START_REF] Carlo | Gaussian framework for interference reduction in live recordings[END_REF]];

• (free eld propagation) those that assume a single anechoic path, e. g. [START_REF] Rickard | The DUET blind source separation algorithm[END_REF][START_REF] Nesta | Convolutive underdetermined source separation through weighted interleaved ICA and spatio-temporal source correlation[END_REF][START_REF] Higuchi | Underdetermined blind separation and tracking of moving sources based ONDOA-HMM[END_REF]] ;

• (reverberant propagation) those that model the sound propagation path entirely, e. g. [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF][START_REF] Duong | Under-determined reverberant audio source separation using a full-rank spatial covariance model[END_REF]Li et al. 2019a];

• (reverberant propagation) and those that attempt to separately estimate the contribution of the early echoes and the contribution of the late tail, e. g. [START_REF] Leglaive | Multichannel audio source separation with probabilistic reverberation modeling[END_REF]].

Therefore, these existing approaches either ignore sound propagation or aim at estimating it fully, which a ect the quality of the separation. In the rst case, strong echoes and reverberant constitute a low bound in the separation capability. In fact, these elements of the sound propagation blur and spread the energy of the source source over multiple TF bins, for which the assignation is harder. When computing the TF masking operation, these bins may introduce strong artifacts. In the second case, the algorithm need to estimated more parameters with consequences in complexity and estimation accuracy.

E have been introduced as a possible solution to overcome some of these limitations,. More details will be given in Chapter 9, where a new method for speech source separation based on the Multichannel NMF framework and echoes is described.

O

For a comprehensive review on spatial ltering methods, the reader can refer to the book [START_REF] Van Trees | Optimum array processing: Part IV of detection, estimation, and modulation theory[END_REF].

Spatial ltering aim at the enhancement of a desired signal while suppressing the background noise and/or interfering signals. It is a large and active research eld that has interested the signal processing and telecommunication communities since several decades. It has produced a vast literature including several reference books dedicated to the topic. For more details in this direction, the reader can refer to, e.g., the book [START_REF] Van Trees | Optimum array processing: Part IV of detection, estimation, and modulation theory[END_REF]. In audio, this topic has been extensively reviewed in the context of speech enhancement in a recent publication [START_REF] Gannot | A consolidated perspective on multimicrophone speech enhancement and source separation[END_REF] 60 . 60 The content of this work has been extended in the book [START_REF] Vincent | Audio source separation and speech enhancement[END_REF]].

In spatial ltering, the RIRs (and related models, e. g., RTFs, steering vectors or ReTFs) play a central role. Intuitively, giving the mixing model in Eq. (8.1), the enhancement of a target source can be achieved by merely denoising the recordings and ltering by the inverting RIRs. However, this is not always possible as the inversion of the RIRs is not a straightforward operation. The work in [START_REF] Neely | Invertibility of a room impulse response[END_REF]] discuss explicitly the issues of inverting RIRs. Later, several techniques were investigated, which are sometimes referred to as Room Response Equalization [START_REF] Cecchi | Room response equalization-A review[END_REF]].

Beamforming

Beamforming is one of the most famous techniques used in spatial ltering, and the two term are almost equivalent. One of the most famous beamformer is the Delay-and-Sum (DS) beamformer. The intuitive idea behind it is to sum the microphone channels constructively by compensating the time delays between the sound source and the spatially separated microphones. Thus, the target source signal is enhanced, while noise, interferences, and reverberation being suppressed. Figure 8.2 illustrate this ideas. Later, this idea has been extended to Frequency and TF processing with direct modeling of the noise sources. More formally, beamformers design mathematical optimization criterion, namely objective function, de ning the desired shape of the estimated signal and return a lter to be applied to the microphone recordings. For instance, one may want to keep a unit gain towards the desired sound source's direction while minimizing the sounds from all the other directions. The literature on beamformers spans in two directions: di erent optimization criteria and how to estimate the parameters required by their computation. We will now elaborate the two directions in turn.

M

have been proposed. Among all, some of the most famous are the DS, the Minimum-Variance-Distortionless-Response (MVDR) [START_REF] Capon | High-resolution frequency-wavenumber spectrum analysis[END_REF]], the Maximum SNR (MaxSNR) [START_REF] Cox | Robust adaptive beamforming[END_REF], the Maximum SINR (MaxSINR) [START_REF] Van Veen | Beamforming: A versatile approach to spatial ltering[END_REF], and the Linearly-Constrained-Minimum-Variance (LCMV) [START_REF] Frost | An algorithm for linearly constrained adaptive array processing[END_REF]]. These criteria are designed to satisfy di erent constraints and model prior knowledge, as discussed in § 11.1.3. The reader can also refer to the above-suggested book for more details.

P

is a crucial step for beamformers. We can identify two main categories of parameters: the one related to the RIRs and the one related to the source and noise statistics. In the former case fall all the methods that model the acoustic propagation of sound. Therefore, similarly to the methods for separation, we can group existing methods in the following groups:

• (free and far eld propagation) methods based on relative steering vectors build on DOA [START_REF] Takao | An adaptive antenna array under directional constraint[END_REF][START_REF] Applebaum | Adaptive arrays with main beam constraints[END_REF][START_REF] Cox | Robust adaptive beamforming[END_REF][START_REF] Van Veen | Beamforming: A versatile approach to spatial ltering[END_REF]];

• (multipath propagation) methods based on rake receiver [START_REF] Flanagan | Spatially selective sound capture for speech and audio processing[END_REF][START_REF] Jan | Matched-lter processing of microphone array for spatial volume selectivity[END_REF]Dokmanić et al. 2015;[START_REF] Peled | Linearly-constrained minimum-variance method for spherical microphone arrays based on plane-wave decomposition of the sound eld[END_REF][START_REF] Scheibler | Raking echoes in the time domain[END_REF][START_REF] Kowalczyk | Raking early re ection signals for late reverberation and noise reduction[END_REF]];

• (reverberant propagation) methods based on BCE considering the RIR as the acoustic channel (see Chapter 4);

• (reverberant propagation) methods based on DOAs and the statistical modeling of the di use sound eld [START_REF] Thiergart | An informed LCMV lter based on multiple instantaneous direction-of-arrival estimates[END_REF][START_REF] Schwartz | Multi-microphone speech dereverberation and noise reduction using relative early transfer functions[END_REF]];

• (reverberant propagation) methods based on ReTF [START_REF] Gannot | Signal enhancement using beamforming and nonstationarity with applications to speech[END_REF][START_REF] Doclo | GSVD-based optimal ltering for single and multimicrophone speech enhancement[END_REF][START_REF] Cohen | Relative transfer function identi cation using speech signals[END_REF][START_REF] Markovich | Multichannel eigenspace beamforming in a reverberant noisy environment with multiple interfering speech signals[END_REF]];

• (reverberant propagation) methods based on (deep) learning [Li et al. 2016a;[START_REF] Xiao | Deep beamforming networks for multi-channel speech recognition[END_REF][START_REF] Sainath | Multichannel signal processing with deep neural networks for automatic speech recognition[END_REF][START_REF] Ernst | Speech dereverberation using fully convolutional networks[END_REF]];

The DOAs-based methods exploit the closed-form mapping between DOAs and the steering vectors in far-eld scenarios. Thus, good performances are possible only upon a reliable estimation of the DOAs (see next section), a challenging problem in noisy and reverberant environments. The steering vectors' computation depends on the array geometry, which is unknown in some practical cases. Alternatively, one can estimate the full acoustic channels, which is a cumbersome task by itself. The ReTF-based approaches have been introduced to overcome these two limitations. They automatically encode the RIRs, the geometrical information, and are "easier" to estimate than the RIRs. The main limitation of these methods is that they return spatial source image at the reference microphone, rather than the "dry" source signal. Therefore, when reverberation is detrimentally a ecting the speech signal's intelligibility, post-processing is necessary [Schwartz et al. 2016].

Recently, DNNs have been proposed for solving this task, either to estimate the beamformer lter [Li et al. 2016a;[START_REF] Xiao | Deep beamforming networks for multi-channel speech recognition[END_REF][START_REF] Sainath | Multichannel signal processing with deep neural networks for automatic speech recognition[END_REF] or in an end-to-end task [START_REF] Ernst | Speech dereverberation using fully convolutional networks[END_REF]. Moreover, DNNs have been used to estimate some of the parameters, such as the DOAs [Salvati et al. 2018;[START_REF] Chazan | Multimicrophone speaker separation based on deep DOA estimation[END_REF] or speech activity detection for ReTF estimation [START_REF] Chazan | DNN-based concurrent speakers detector and its application to speaker extraction with LCMV beamforming[END_REF]].

E are neither considered nor modeled as noise terms in the literature discussed thus far. This direction is taken by the echo-aware methods accounting speci cally for the multipath propagation. We will discuss these methods in more detail in chapter Chapter 11 together with their implementation.

O S S L

The reader can nd more details is SSL in the recent review articles [START_REF] Rascon | Localization of sound sources in robotics: A review[END_REF][START_REF] Argentieri | A survey on sound source localization in robotics: From binaural to array processing methods[END_REF] as well as in [Vincent et al. 2018, Chapter 4].

Most of Sound Source Localization (SSL) methods consist in determining the distance and direction of sound sources from microphone (array) in the 3D space, typically in a passive scenario. As discussed above, the information on the sources' and microphones' position in the room is encoded in the RIRs. Therefore, assuming the uniqueness of the mapping between locations to a RIR, it is theoretically possible to retrieve the absolute position of microphones and sources, as show in [Ribeiro et al. 2010a;Crocco and Del Bue 2016]. However, this is yet a very challenging task, which typically involves the solution of several sub-problems. Therefore, it is more common to relax the SSL problem as follows. First, rather than operating in the 3D cartesian coordinate system, most of the existing methods aim at estimating 2D-Direction of Arrival (DOA), namely the angles for on the unit sphere with the center in a reference point. This reference point is usually the barycenter of the microphone array. These angles are called azimuth and elevation as shown is Figure 8.3. When only a single microphone pair is considered, the problem is simpli ed to 1D-SSL, estimation the 1D-DOA, named angle of arrival, with respect the microphone axis. Second, they assume far-eld scenarios. The main reasons for adopting 8.3: Geometrical illustration of the Sound Source Localization, showing the azimuth θ j , the elevation ϕ j and angle of arrival α j for the source j at s j and two microphones at x 1 , x 2 respectively. such simpli cations are the followings:

• estimating the distance is known to be a much more challenging task than estimating the DOAs [START_REF] Vesa | Binaural sound source distance learning in rooms[END_REF]];

• the far-eld scenario is a reasonable assumption when using a compact array recording distant sounds.

• the problem can be independent to room geometry, whose estimation is an ambitious task;

Finally, in far-eld settings, sometimes DOAs are su cient to achieve reasonable speech enhancement performance as demonstrated by the literature in spatial ltering.

Despite these approximations, the SSL problem still challenges today's com-

As an example, consider the TDOA-based SSL methods: the feature extraction step consist into extract time delays from audio recordings, then these quantities are mapped to direction of arrivals, thanks to the acoustic propagation model and the array geometry.

putational methods. Popular approaches consists in two main components:

feature extraction and mapping. First, the audio data are represented as features, as independent as possible from the source's content while preserving spatial information. Second, the features are mapped to the source position. Two lines of research have been investigated to obtain such mappings: knowledge-driven and data-driven approaches.

Knowledge-based approaches

Knowledge-based approaches rely on a physical model for sound propagation. These models rely on closed-form mapping from the sound's direct path Time Di erences of Arrival (TDOAs) at the microphone pair and the source's azimuth angle in this pair. If multiple microphone pairs are available and form a non-linear array, their TDOAs can be aggregated to obtain 2D-DOAs [START_REF] Dibiase | Robust localization in reverberant rooms[END_REF]. Furthermore, what di erentiates the approaches in this categories lies in their ability to localize either single sources or multiple ones, their robustness to noise and reverberation, and the particular methods they used. We can identify the following approaches based on:

• subspace methods, such as MUSIC [START_REF] Dmochowski | Broadband MUSIC: Opportunities and challenges for multiple source localization[END_REF]];

• TDOA-based techniques, which uses Generalized Cross Correlation (GCC) functions [START_REF] Knapp | The generalized correlation method for estimation of time delay[END_REF][START_REF] Blandin | Multi-source TDOA estimation in reverberant audio using angular spectra and clustering[END_REF][START_REF] Lebarbenchon | Evaluation of an open-source implementation of the SRP-PHAT algorithm within the 2018 LOCATA challenge[END_REF]] to estimate TDOA and then compute the most reliable DOA from them; These methods are related to beamformingbased techniques, such as SRP-PHAT [START_REF] Dibiase | Robust localization in reverberant rooms[END_REF]], which search the direction that maximizes the power of the output of a beamformer.

• methods based of RIRs estimation and blind system identi cation [START_REF] Chen | Time delay estimation in room acoustic environments: an overview[END_REF]],

• methods based on probabilistic framework solved with Maximum Likelihood optimization [START_REF] Stoica | Maximum likelihood methods for direction-of-arrival estimation[END_REF]Li et al. 2016b].

The main limitations of these approaches arise from the approximation considered in the models. In particular, common to all of them is the assumption that sound propagation being typically free-eld. Thus, they intensely su er in environments it is violated, e. g., in the presence of strong acoustic echoes and reverberation as discussed in [START_REF] Chen | Time delay estimation in room acoustic environments: an overview[END_REF]]. Lot of recent work has been dedicated to make SSL methods more robust to noise and reverberation even in challenging scenario of multiple moving speakers, such as [Li et al. 2019b;[START_REF] Kitic | TRAMP: Tracking by a realtime Ambisonic-based particle lter[END_REF]. The interested reader is invited to see the results of the recent LOCATA challenge [START_REF] Evers | The LOCATA challenge: Acoustic source localization and tracking[END_REF]].

Data-driven approaches

Data-driven approaches have been proposed to overcome the challenging task of modeling sound propagation. This is done using a supervised-learning framework, that is, using annotated training dataset to implicitly learn the mapping from audio features to source positions Such data can be obtained from annotated real recordings [Deleforge et al. 2015a;[START_REF] Nguyen | Autonomous sensorimotor learning for sound source localization by a humanoid robot[END_REF]] or using acoustic simulators [START_REF] Laufer | Relative transfer function modeling for supervised source localization[END_REF][START_REF] Vesperini | Localizing speakers in multiple rooms by using deep neural networks[END_REF][START_REF] Adavanne | Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network[END_REF][START_REF] Chakrabarty | Broadband DOA estimation using convolutional neural networks trained with noise signals[END_REF][START_REF] Perotin | CRNN-based joint azimuth and elevation localization with the Ambisonics intensity vector[END_REF][START_REF] Gaultier | VAST: The virtual acoustic space traveler dataset[END_REF].

In comparison to knowledge-driven methods, these methods have the advantage that they can be adapted to di erent acoustic conditions by including challenging scenarios in the training dataset. Therefore, these methods were showed to overcome some limitations of the free-eld model. Under this perspective, in the data-driven literature two main trends can envisioned: end-to-end learning models and two-step models. In the former case, all the SSL pipeline is encapsulated into a single robust learning framework, taking as input the microphone recordings and returning the source(s) DOAs. Examples of these approaches are the works in [START_REF] Chakrabarty | Broadband DOA estimation using convolutional neural networks trained with noise signals[END_REF][START_REF] Perotin | CRNN-based joint azimuth and elevation localization with the Ambisonics intensity vector[END_REF][START_REF] Adavanne | Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network[END_REF], where the task is performed with DNNs models. In the latter, learning models are used as a substitute for either feature extraction or the mapping. For instance, in [START_REF] Laufer | Relative transfer function modeling for supervised source localization[END_REF]Deleforge et al. 2015a;[START_REF] Gaultier | VAST: The virtual acoustic space traveler dataset[END_REF], GMMs-based models were used to learn the mapping from features derived from the ReTF of pair of microphones. The work in [START_REF] Nguyen | Autonomous sensorimotor learning for sound source localization by a humanoid robot[END_REF]] compare GMM-and DNN-based approach for SSL for the interesting case of a robot gathering training examples. In [START_REF] Vesperini | Localizing speakers in multiple rooms by using deep neural networks[END_REF], the author proposes to use DNN models to estimate source location using features computed through GCC-PHAT. Despite the considerable bene t of data-driven approaches in learning com-"If you dont pay attention, your learing methods risk to learn a model from model" -Zybnek Koldowsky plex functions, their main limitation lies in the training data. First, these data are typically tuned for speci c microphone arrays and fail whenever test conditions strongly mismatch training conditions. Moreover, due to the cumbersome task of collecting annotated datasets that cover as many possible scenarios as possible, synthetic data generated by simulators are used. However, these data may be too arti cial or simplistic, with the consequence that the models may not be able to generalize to real-world conditions.

, , are typically treated as nuisance in methods developed for SSL and DOAs estimation. Moreover, while knowledge-based approaches operate under strong assumptions which are typically violated in multipath scenario, for data-driven methods it is not trivial to inject prior information about sound propagation. Based on these limitations, our contribution propose to combines the best of the two worlds to e ectively exploits the echo knowledge: More speci cally, to deploy a DNN to estimate few echoes Chapter 6 and use well-understood knowledge-based method to map them to source DOAs Chapter 10.

* C

This chapter presented some fundamental audio signal processing problems and an overview of related approaches to address them. These problems will be considered in their echo-aware settings in the following chapters, in particular:

• Multichannel NMF-based Audio Source Separation in Chapter 9,

• SRP-PHAT-based Sound Source Localization in Chapter 10 using only two microphones,

• and in Chapter 11, we will discuss some applications of the dEchorate dataset (Chapter 7) for some spatial ltering methods ( § 11.1) and room geometry estimation ( § 11.2).

Separake: Echo-aware Sound Source Separation [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]: one that uses only magnitudes of the transfer functions and one that uses the phases as well. The experimental part shows that the proposed approach beats its vanilla variant by using only a few echoes and that with magnitude information only, echoes enable separation where it was previously impossible.

The material presented in this chapter results from a collaboration with Robin Scheibler and Ivan Dokmanić and was published in [Scheibler et al. 2018c]. This Scheibler et al., "Separake: Source separation with a little help from echoes" chapter recalls the main ndings of the paper and brings additional insights on the literature and on the proposed model, which has been re-written using this thesis' notations. The personal contribution to this collaboration, done in the early months of the Ph. D., was adapting and implementing in Python of the proposed NMF method accounting for echoes and using pre-trained dictionaries.

L M A S S

The scienti c literature on MASS is vast, still active, and spans decades. For a brief introduction, the reader can refer to § 8.2 and to recent books [START_REF] Vincent | Audio source separation and speech enhancement[END_REF][START_REF] Makino | Audio Source Separation[END_REF]. In this chapter we will consider only the case of multichannel convolutive recordings featuring reverberant speech data in overdetermined settings. Even selecting this narrow case, the literature remain vast and we will not review it in the context of this thesis.

Here, we will focus on Nonnegative Matrix Factorization (NMF)-based MASS, which allows to easily incorporate side information on both the sources and the acoustic propagation. This is a key feature, as opposed to the end-to-end Virtual microphones F 9.1: Typical setup with two speakers recorded by two microphones. The illustration shows the virtual microphone model (grey microphones) with direct sound path (dotted lines) and resulting rst-order echoes (colored lines). Image taken from our paper [Scheibler et al. 2018c].

Deep Neural Network (DNN)-model. These latter models are very popular nowadays and are able to reach impressive performances in the general case. However, when it comes include side information or adapt to speci c tasks, it is far from trivial.

To the best author knowledge, in the literature only few works can be found that incorporate the knowledge of echoes into sound source separation. The work in [START_REF] Asaei | Structured sparsity models for reverberant speech separation[END_REF] proposes geometry-based approach embedded in a sparse optimization framework. First, by localizing the image sources and estimating the room geometry, the echoes' timings are estimated. Then, after computing the echoes' coe cients in a convex optimization framework, the individual speech signals are separated with either inverse-ltering or sparse recovery. The performance of this approach relies on the RIR and geometry estimation steps, which are very sensitive to the challenging acoustic condition, e. g. low SNR or high RT 60 . Instead, the work in [START_REF] Leglaive | Multichannel audio source separation with probabilistic reverberation modeling[END_REF] proposes to tackle the convolutive model by imposing a probabilistic prior on the early part of the RIRs, namely, modeled as an autoregressive process in the frequency domain. Later, the same authors extended this work in [Leglaive et al. 2016] accounting for both early and late part of the mixing lters.

T is yet di erent from those presented above. First, rather than tting the echo model as in [START_REF] Leglaive | Multichannel audio source separation with probabilistic reverberation modeling[END_REF]Leglaive et al. 2016], or estimating the mixing lters as in [START_REF] Asaei | Structured sparsity models for reverberant speech separation[END_REF], we aim to show that separation in the presence of known echoes is better than separation without echoes. Second, we conduct this investigation in the context of source separation with non-negative source models with a deterministic model for the mixing lters. Third, we propose to solve the problem from the point of view of image microphones, already used in [START_REF] Bergamo | Collaborative sensor networking towards real-time acoustical beamforming in free-space and limited reverberance[END_REF][START_REF] Korhonen | Acoustic localization using reverberation with virtual microphones[END_REF]. The image microphone model is equivalent to the Image Source Method (ISM) [START_REF] Allen | Image method for e ciently simulating small-room acoustics[END_REF], where virtual receivers are placed outside of the room (See Figure 9.1). Even if the ISM is more common and implemented in practice in acoustic simulators, the two models are strictly equivalent. This approach is based on the acoustic rake receivers 61 previously proposed 61 Rake receivers are a particular type of beamformers accounting for the e ect of multipath propagation (see § 11.1) in [Dokmanić et al. 2015] and is thus dubbed Separake. This model allows for a simple and intuitive way to structure the mixing matrix so that less parameters need to be estimated. The considered setup is illustrated in Figure 9.1. We assume that the array is placed close to a wall or a corner. This is useful for the following reasons: rst, it makes echoes from the nearby walls signi cantly stronger than all other echoes; second, it ensures that the resulting image array (real and image microphones) is compact, allowing to assume the far eld regime. This assumption is motivated by typical echo-aware signal processing applications, such as for smart-home devices, typically placed on a table or close to a wall.

T

provides an interesting geometrical interpretation in light of beamforming theory. Real and virtual microphones form dipoles with diverse frequency-dependent directivity patterns. By integrating more and more virtual microphones, the directivity patterns change and higher spatial selectivity can be achieved [Dokmanić et al. 2015]. This e ect is shown in Figure 9.2. Therefore, the goal of this work is to design audio source separation algorithms which bene t from this known spatial diversity. Note that the signal virtual microphones are not accessible directly, as the observations will remain the mixture of convolutive sound at the real microphones. However, such formulation rather o er a useful interpretation and a way to model incoming reverberant signals.

M

Recalling the echo model for the RIRs, and assuming R echoes per source are known, the approximate Room Transfer Function (RTF) from source j to microphone i writes

Hij (f ) = R r=0 α (r) ij e -i2πf τ (r) ij . (9.1)
Absolute TOAs relate to the source's distance which is not assumed to be known here. Instead, we will assume that only the TDOAs are known, by arbitrarily xing the delay of the direct path of a reference microphone to zero.

In addition, we assume all walls to be spectrally at in the frequency range of interest and that α (r) ij are known up to a scaling (i.e. α (0) ij = 1). In this work all these echoes properties are assumed to be known.

Assuming the narrowband approximation, the mixing process can be modeled as in § 3.2.5. That is, the Short Time Fourier Transform (STFT) of the i-th microphone signal reads

X i [k, l] = J j=1 H ij [k]S j [k, l] + N i [k, l] (9.2)
with k ∈ [0, . . . , F ] and l ∈ [0, . . . , T ] being the frequency and frame index,

H ij [k]
is the DFT approximating the RTF of (9.1), X j [k, l] the STFT of the j-th source signal, and N i [k, l] a term including noise and model mismatch. It is convenient to group the microphone observations in vector-matrix form,

X[k, l] = H[k]S[k, l] + N [k, l], (9.3)
where

X[k, l], N [k, l] ∈ C I×1 , S[k, l] ∈ C J×1 and H[k, l] ∈ C I×J .
Let the squared magnitude of the spectrogram of the j-th source be P j = |S j | 2 kl ∈ R F ×T . As depicted in Figure 9.3, the spectrogram can be modeled as the product of 2 non-negative matrices: F 9.3: Spectrogram of a sound source signal decomposed into dictionary and activation. Image taken from the slides accompanying the paper of this work [Scheibler et al. 2018c]. (9.4) where D j is the non-negative dictionary whose columns are called atoms and can be interpreted as interpreted as spectral templates of the source, while the latent variables Z j , called activations, indicating when and how this templates are activated.

P j = D j Z j ,
9.3 M A S S NMF NMF-based Multichannel Audio Source Separation (MASS) can then be cast as an inference problem in which we minimize the error between the observed X over all possible non-negative factorizations (9.4). This normally involves learning the channels, namely the frequency-domain mixing matrices H. Instead of learning them, we build the channels based on the prior knowledge of the earliest few echoes. As introduced in § 8.2, the NMF-based MASS consists in two steps: the estimation of source's Power Spectral Density (PSD) via NMF and lters and the actual separation via Wiener Filter.

Here we consider two classical, well-understood multi-channel source separation algorithms which, by default, estimate the channels together with sources' dictionaries and activations. The rst algorithm is Nonnegative Matrix Factorization (NMF) via Multiplicative Updates (MU) and consider only the magnitudes of the transfer functions. The second one is the multichannel NMF via Expectation Maximization (EM), which instead explicitly models the phases of the mixing lters. In this work, we considered only the (over)determined case (J ≤ I). In the following we brie y describe the idea behind the two algorithms. The reader is invite to refer to the work of [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF] for further details. 9.3.1 NMF using Multiplicative Updates (MU-NMF)

MU for NMF involves the observed magnitude spectrograms only and the updates rules guarantee non-negativity as long as the initialization is nonnegative. This model has been originally proposed by in [START_REF] Lee | Algorithms for Non-negative Matrix Factorization[END_REF], however we will consider its formulation as it appear in [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]. The observed multi-channel squared magnitude spectra are denoted

V i = |X i [k, l]| 2
kl and their non-negative factorizations (9.5) where

V i = J j=1 diag(Q ij )D j Z j , i = 1, . . . , I
Q ij = |H ij [k]| 2
k is the vector of squared magnitudes of the approx-F 9.4: Schematics of the signal model used for MU-NMF. Image taken from the slides accompanying the paper of this work [Scheibler et al. 2018c].

imate RTF between microphone i and source j.

T MU the divergence 62 between the observed spectrogram 62 The divergence is an asimmetrical "distance" measure. The Itakura-Saito diverge was proposed to account for perceptum (dis)similarity between spectra. Other type of divergence are used in Audio Source Separation, such as the Kullback-Leibler divergence.

V i [k, l] and the model V i [k, l]. In case of Itakura-Saito divergence [START_REF] Févotte | Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis[END_REF]], the cost function writes (9.6) where

C MU (Θ MU ) = jkl D IS (V i [k, l]| V i [k, l]) + γ j Z j 1 ,
D IS (v|v) = v v -log v v -1 and Θ MU = {Q ij , {D j , Z j } j } ij
is the set of parameters. We add an 1 -penalty term to promote sparsity in the activations due to the potentially large size of the dictionary [START_REF] Sun | Universal speech models for speaker independent single channel source separation[END_REF].

T MU

for each scalar parameter of interest θ is obtained by multiplying its value at previous iteration by the ratio of the negative and positive parts of the derivative of the criterion w. r. t. this parameter, namely,

θ ← θ [∇ θ C MU (Θ MU )] - [∇ θ C MU (Θ MU )] + where C MU (Θ MU ) = [∇ θ C MU (Θ MU )] + -[∇ θ C MU (Θ MU )]
-and the summands are both nonnegative. Following the MU rule derivations as in Ozerov and Févotte, we obtain:

Q ij ← Q ij V -2 j V j (Z j D j ) 1 1×T V -1 j (Z j D j ) 1 1×T (9.7) Z j ← Z j i (diag(Q ij )D j ) V j V -2 j i (diag(Q ij )D j ) V -1 j + γ , (9.8) D j ← D j i diag(Q ij ) V j V -2 j Z j i diag(Q ij ) V -1 j Z j , (9.9)
where multiplication , power, and division are element-wise and 1 1×T is a vector of T ones. 9.3.2 NMF using Expectation Maximization (EM-NMF)

Unlike the MU algorithm that independently maximizes the log-likelihood of mixture spectral magnitudes over individual channels, the EM-NMF maximizes the joint log-likelihood over all complex-valued channels [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]. Hence, the model takes explicitly into account observed phases. In this approach, each source j is modeled as a complex Gaussian in the form of S j [k, l] ∼ N c (0, (D j Z j ) kl ), (9.10) and the magnitude spectrum P j of (9.4) can be understood as the variance of source j. This underlying statistical model for the source signal can be used (with little changes) in the MU-NMF approach. However, the MU-based approach can be applied deterministically, without any statistical assumption on the latent variables. Under this model, and assuming uncorrelated noise, the microphone signals also follow a complex Gaussian distribution with covariance matrix 9.11) where Σ S and Σ N are the covariance matrices of the sources and noise, respectively.

Σ X [k, l] = H[k]Σ S [k, l]H H [k] + Σ N [k, l], (
T EM corresponds to the negative log-likelihood of the observed signal, that is,

C EM (Θ EM ) = kl trace X[k, l]X[k, l] H Σ -1 X [k, l] +log det Σ X [k, l]. (9.12)
where the Θ EM = H, {D j , Z j } j , Σ N is the set of parameters.

T EM estimates all the parameters Θ by alternating between the so-called E-step and M-step. In a nutshell, one iteration of the E-step consists in computing the conditional expectation of the the "complete" log likelihood 63 with respect to the current parameter estimates, and the M-step 63 The complete data log-likelihood includes both observed variables X and latent variables S re-estimates the parameters by maximizing the conditional expectation of the complete log-likelihood. This quantity can be e ciently minimized using the EM algorithm proposed in [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]. Since adding sparsity priors is not straightforward in the EM framework, it was not included in the proposed method. 9.3.3 Supervised NMF with pre-trained Dictionaries Pre-trained dictionaries are a typical way to informing the NMF algorithm, which is sometimes referred to as supervised NMF. The idea is to run NMF on training sets containing examples from desired sound classes and collect the atoms of the estimated non-negative matrices [START_REF] Schmidt | Single-channel speech separation using sparse non-negative matrix factorization[END_REF]. At test phase, these atoms are used as basis vectors for the dictionary matrix (i. e., D) and can be used as a good initialization point or kept xed in the algorithm 64 . This can be seen as an instance of the problem of dictionary 64 In the context of NMF-based music transcription applied to piano music, the dictionary can be a collection of spectral templates, each of which is associated to a piano note [Müller 2015] learning which also exists in many other research elds. For audio source separation, this idea has been studied extensively since promising results were obtained, even in single channel scenarios [START_REF] Smaragdis | A sparse non-parametric approach for single channel separation of known sounds[END_REF]]. As discussed later in § 9.5.2, in this work we will use two di erent dictionaries: one universal, and the other speaker-speci c.

E A S S

To evaluate the usefulness of echoes in source separation, we modi ed the the multi-channel NMF framework of Ozerov and Févotte [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]. The knowledge of the echoes is embedded in the model by approximating the entries of mixing matrices with (9.1), that is, (9.13) where f k = kF s /F are the discretized frequencies in Hz corresponding to the k-th bin in the DFT. Futhermore, the early-echo channel model is kept xed throughout the iterations. Moreover, instead of updating both sources' dictionaries and activations, we adapted pre-trained dictionaries to better guide the source separation.

H ij [k] = R r=0 α (r) ij e -i2πf k τ (r) ij , H[k] = [H ij [k]] ij ,
Neglecting the reverberation (or working in the far-eld anechoic regime) leads to a constant and equal Q ij for all j and i. A consequence is that, if we also assume a unique, universal dictionary, namely, D = D j ∀j, then MU-NMF framework can be simpli ed. Indeed, (9.5) becomes the same for all i,

V i = j DZ j = D j Z j ,
so even with the correct atoms identi ed, we can assign them to any source without changing the value of the cost function. Therefore, anechoic multichannel separation with a universal dictionary cannot work well. This intuitive reasoning is corroborated by numerical experiments in Section 9.5.4. The problem is overcome by the EM-NMF algorithm which keeps the channel phase and is thus able to exploit the phase diversity across the array. Of course, as showed in this work, it is also overcome by using echoes.

N

We test our hypotheses through computer simulations. In the following, we describe the simulation setup, dictionary learning protocols, and we discuss the results. The sound propagation between sources and microphones is simulated using the image source model implemented in the pyroomacoustics Python package [Scheibler et al. 2018a]. The wall absorption factor is set to 0.4, leading to a RT 60 of approximately 100 ms. 65 The sampling frequency is set to 16 kHz, 65 The resulting RT 60 is short. Realistic values for o ce and house rooms are typically around 500 ms. At the time of the publication we did not investigate di erent reverberation levels as the focus was on the early echoes.

STFT frame size to 2048 samples with 50% overlap between frames, and we use a cosine window for analysis and synthesis. Partial RTFs are then built from the R nearest image microphones. The global delay is discarded, and only the relative amplitudes between echoes are kept.

With this setup, we perform three di erent experiments. In the rst one, we evaluate MU-NMF with a universal dictionary. In the other two, we evaluate the performance of MU-NMF and EM-NMF with speaker-speci c dictionaries. We vary R from 1 to 6 and use the following three baseline scenarios:

1. anechoic: anechoic conditions, thus, without echoes in the mixture.

2. learn: the RTFs are learned from the data together with the activations as originally proposed [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF] and the full reverberation is present in the observed data.

3. no echoes: Reverberation is present but ignored (i.e. R = 0).

With the universal dictionary, the large number of latent variables warrants the introduction of sparsity-inducing regularization. The value of the regularization parameter γ was chosen by a grid search on a holdout set with the signal-to-distortion ratio (SDR) as the gure of merit [START_REF] Vincent | First stereo audio source separation evaluation campaign: data, algorithms and results[END_REF]] (See Table 9.1).

Dictionary Training, Test Set

First, we introduce a dictionary learned from available training data. We explore both speaker-speci c and universal dictionaries [START_REF] Sun | Universal speech models for speaker independent single channel source separation[END_REF]. Speaker-speci c dictionaries can be bene cial when speakers are known in advance. Universal dictionary is more versatile but gives a weaker regularization prior. All dictionaries were trained on samples from the TIMIT corpus [START_REF] Garofolo | TIMIT acoustic-phonetic continuous speech corpus[END_REF] (b) Speaker-speci c dictionary F 9.6: Schematic representation of the dictionary protocol used in this work. Image taken from the slides accompanying the paper of this work [Scheibler et al. 2018c].

S

S D : Two dictionaries were trained on one male and one female speaker. One utterance per speaker was excluded to be used for testing. The number of latent variables per speaker was set to 20.

Implementation:

Authors of [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]] provide a Matlab implementation 66 of 66 Multichannel nonnegative matrix factorization toolbox (in Matlab) MU-NMF and EM-NMF methods for stereo separation. We ported their code to Python and extended it to arbitrary number of input channels. However this software features some ad-hoc decisions which do not t our scenario. Thus, we provide a Python adaptation with the following modi cations:

• rst the original code was restricted to the 2-channel case, i.e. I = 2, thus, in order to embrace the speci cities of our scenario and for the sake of generalization, we extend it to the multi-channel case, that is ∀I ≥ 1;

• the MU-NMF was modi ed to handle sparsity constraint as described in 9.3.1;

• since the EM method degenerates when zero-valued entries are present in the dictionary matrix, D, all these entries are initially set to a small constant value of 10 -6 ;

• separation is achieved via MWF as described in [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]].

• the code was further modi ed to deal with xed dictionary and channel model matrices, which are normalized in order to avoid indeterminacy issues [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF].

Finally, no "simulated annealing" strategy [Ozerov and Févotte 2010, Section III.A.6] was used in the nal experiments. In fact in some preliminary and informal investigations we noticed that this yielded better results than using annealing. In the experiments, the number of iterations for MU-NMF (resp. EM-NMF) was set to 200 (resp. 300).

Results

We evaluate performance in terms of Signal to Distortion Ratio (SDR) and Signal-to-Interference Ratio (SIR) as de ned in [START_REF] Vincent | First stereo audio source separation evaluation campaign: data, algorithms and results[END_REF]]. We compute these metrics using the mir _ eval toolbox [START_REF] El | mir_eval: A transparent implementation of common MIR metrics[END_REF]. The distributions of SDR and SIR for separation using MU-NMF and a universal dictionary are shown in Figure 9.7a, with a summary in Figure 9.8. We use the median performance to compare the results from di erent algorithms. First, we con rm that separation fails for at RTFs (anechoic and R = 0) with SIR at around 0 dB. Learning the RTFs performs somewhat better in terms of SIR than in terms of SDR, though both are low. Introducing approximate RTFs signi cantly improves performance: the proposed approach outperforms the learned approach even with a single echo. With up to six echoes, gains are +2 dB SDR and +5 dB SIR. Interestingly, with more than one echo, nonnegativity and echo priors are already su cient for achieving good separation, overtaking the 1 regularization.

Separation with speaker-dependent dictionaries is less challenging since we have a stronger prior. Accordingly, as shown in Figures 9.7b and 9.8, MU-NMF now achieves a certain degree of separation even without the channel information. The gains from using echoes are smaller, though one echo is still su cient to match the median performance of learned RTFs. Using an echo, however, results in a smaller variance, while adding more echoes further improves SDR (SIR) by up to +2 dB (+3 dB). In the same scenario, EM-NMF (Figure 9.7c) has near-perfect performance on anechoic signals which is expected as the problem is overdetermined. For MU, a single echo su ces to reach the performance of learned RTFs and further improve it. Moreover, echoes signi cantly improve separation quality as illustrated by up to 3 dB improvement over learn. It is interesting to note that in all experiments the rst three echoes nearly saturate the metrics. This is good news since higher order echoes are hard to estimate.

C

In this work, we investigated the potential bene t of early echo knowledge for the problem of sound source separation. Unlike earlier work, instead of tting an echo model or trying to estimate blindly the acoustic channels, we investigate the potential of including the properties of known echoes in well established NMF-based source separation algorithms. In particular, we modi ed the MU approach (which considers only spectral magnitudes) and the EM (which accounts for complex spectra) by integrating a simple echo model. Despite its simplicity, such echo model lends itself to a interesting interpretation by revising the ISM model: to each echo corresponds an image microphone (instead of image source as in ISM). It follows that real and image microphones can be considered as microphones arrays with speci c directivity pattern. Numerical results shows that echoes seem to play an essential role in magnitudeonly algorithms, like the MU-NMF. In general, they show that using knowledge of a few echoes signi cantly improve results with respect to an anechoic model. This improvement is measured by the standard metrics even when compared to approaches that learn the transfer functions. Finally, this work con rms the potential of including echoes in sound source separation framework.

F

on echo-aware source separation could include:

• integrating the blind estimation of the echoes properties, e. g. using the algorithm blaster, proposed in Chapter 5;

• including the late reverberation part in the model for the RTFs;

• experiment with more microphones, more room con gurations, more sources on real data, e. g., using the ones o ered by the dEchorate dataset, described in Chapter 7; In fact, such dataset was designed and recorded also for echo-aware application. In this sense, the method discussed in this chapter can be perfectly tested on such kind of dataset. • Paper

• Code

• Poster

• HARU Robot presentation bene t SSL. To this end, we introduce the concept of Microphone Augmentation with Echoes (MIRAGE), based on the image microphone model. In particular, we show that in a simple scenario involving two microphones close to a re ective surface and one source, the proposed approach can estimate both azimuthal and elevation angles, an impossible task assuming an anechoic propagation.

Together with Chapter 6, this chapter describes methods and results published in [Di Carlo et al. 2019], which considers only stereophonic recordings. In this sense, this chapter provides an application for LANTERN, the learning-based echo estimation method presented in the above mentioned chapter.

10.1 L S S L

Common to most sound source localization approaches reviewed in § 8.4 is the challenge posed by reverberation. It is typical to observe that Direction of Arrival (DOA) estimation degrades with increasing acoustic re ections [START_REF] Chen | Time delay estimation in room acoustic environments: an overview[END_REF]]. For these reasons, most sound source localization methods regard reverberation, and in particular acoustic echoes, as a nuisance. Some prior works, e. g., [START_REF] Rui | Time delay estimation in the presence of correlated noise and reverberation[END_REF][START_REF] Chen | Time delay estimation in room acoustic environments: an overview[END_REF][START_REF] Zhang | Maximum likelihood sound source localization for multiple directional microphones[END_REF]], modeled the reverberation as a noise term. However, the commonly used dereverberation methods do not reduce strong early echoes. Alternatively, approaches in [START_REF] Weinstein | Iterative and sequential algorithms for multisensor signal enhancement[END_REF][START_REF] Taghizadeh | Spatial sound localization via multipath euclidean distance matrix recovery[END_REF][START_REF] Salvati | Sound source and microphone localization from acoustic impulse responses[END_REF] attempt to solve SSL by estimating the full RIRs (or ReTFs [Li et al. 2016b]), which is known to be a very di cult task.

Instead, echo-aware sound source localization methods take another direction: they exploit the closed-form relation between echo timings and the audio scene geometry as expressed in the Image Source Method (ISM). Early works such as [START_REF] Korhonen | Acoustic localization using reverberation with virtual microphones[END_REF]Ribeiro et al. 2010a;Ribeiro et al. 2010b;[START_REF] Svaizer | Use of re ected wavefronts for acoustic source localization with a line array[END_REF]] use knowledge form the room geometry to estimate the position of the sound source with respect to the array. This idea was subsequently extended in later works, reducing the amount of prior knowledge required or addressing di erent applications. The authors of [START_REF] Nakashima | A localization method for multiple sound sources by using coherence function[END_REF] study the SSL problem in binaural recordings. To improve localization, they propose to used ad-hoc re ectors as arti cial pinnae and a simple re ection model. In the work [START_REF] Kreković | EchoSLAM: Simultaneous localization and mapping with acoustic echoes[END_REF], the authors address the problem of Acoustic Source Localization and Mapping (SLAM) 67 using echoes. The 67 Acoustic SLAM enables the estimation of a moving robot's position in relation to a number of external acoustic sources [START_REF] Evers | Acoustic slam[END_REF]].

authors of [An et al. 2018] leverage on cameras, depth sensors, and laser sensors to identify re ectors and build a corresponding acoustic model that is used later for localizing sources. Finally, in a very recent work, the well-known MUSIC framework is modi ed to account for an echo model in the spherical harmonic representation [START_REF] Birnie | Re ection Assisted Sound Source Localization Through a Harmonic Domain MUSIC Framework[END_REF] All the above mentioned echo-aware methods are explicitly knowledge-driven, namely, they use closed-form solutions based on physics, acoustics, and signal processing models. The bene t of manipulating "simple" models is however paid by strong assumptions, simpli cations and a need for hand-crafted features. In order to overcome these limitations, data-driven methods have been proposed to address SSL (see § 8.4). This approach leverages training datasets to implicitly learn the mapping from audio features to the source position. Such data can be obtained from real recordings or using physics-based simulators. These methods were showed to overcome some limitations of physics-based model, especially when some assumptions are violated. However, they are typically trained for speci c applications and use-cases (e. g., speci c array geometries, acoustic conditions, etc.) and fail whenever test conditions strongly mismatch training conditions.

10.2 P

In this work, we propose to combine the best of the two worlds:

• to use a data-driven model to estimate sound propagation parameters;

• to use a knowledge-driven model to map such parameters to the source's DOA.

Let us rst introduced a simple yet common scenario: two microphones, one

s 1 m 1 m 2 F 10
.1: Typical setup with one sound source recorded by two microphones. The illustration shows the direct sound path (blue lines) and the resulting rst-order echoes (orange lines). source, and a nearby re ective surface, as illustrated in Figure 10.1. This may occur when the sensors are placed on a table or next to a wall. Striking examples of this scenario are smart table-top devices, such as Amazon Echo, Google Home, etc. The re ective surface is assumed to be the most re ective and closest one to the microphones in the environment, generating the strongest and earliest echo in each microphone. Under this stereophonic close-surface model, we ask the following questions. Note that answering this question is an impossible task in free-and far-eld stereophonic conditions, due to the well known spatial ambiguity of a single TDOA, referred to as the cone of confusion [START_REF] Bregman | Auditory scene analysis[END_REF]]., that is a cone- shaped region where a source provide the same TDOA and (see Figure 10.2). For instance, this makes it impossible for humans and device with only 2 microphone to determine the elevation and the front-back position of a sound source using binaural cues alone.

B SSL

In this section, we brie y review some necessary background in microphone array SSL. Let us assume a microphone array of I sensors is placed inside a room and records the sound emitted by one static point sound source (J = 1).

Recalling the signal model presented in Eq. (4.1), the relationship between the signal x i recorded by the i-th sensor placed at xed position x i and the signal s emitted by the source at xed position s writes

xi (t) = ( hi s)(t) + ñi (t) hi (t) = R r=0 α (r) i δ(t -τ (r) i ) + εi (t), (10.1)
where ñi denotes possible measurement noise and εi collects later echoes, the reverberation tail, di usion, and undermodeling noise. In this work, we will consider only the rst strongest echo, therefore R = 1. Note that for r = 0 denotes the direct propagation path, where τ (0) i is the direct path's time of arrival from the source to the i-th microphone, and α (0) i captures both the sound wave attenuation and air absorption e ects. In the remainder of this chapter, we make the approximation of α

= τ (0) 2 -τ (0) 1 [second],
as a proxy for the estimation of the Angle of Arrival (AOA), ϑ, since:

ϑ ≈ arccos(c τ TDOA / d) [rad], (10.2)
where c is the speed of sound and d is the inter-microphone distance, considered small compared to the source's distance. This relation is illustrated in Figure 10.3. Then, SSL reduces to estimating the TDOA, which can be done by a Cross Correlation (CC)-based methods, e. g., the Generalized Cross Correlation with Phase Transform (GCC-PHAT) method [START_REF] Knapp | The generalized correlation method for estimation of time delay[END_REF][START_REF] Blandin | Multi-source TDOA estimation in reverberant audio using angular spectra and clustering[END_REF] The "generalized" cross-correlation methods add weighting functions (e. g. the Phase Transform (PHAT), or the smoothed coherence transform (SCOT)) to the frequency-domain CC.

Their purpose is to improve the estimation of the time delay depending on speci c characteristics of the signal and noise. See [START_REF] Chen | Time delay estimation in room acoustic environments: an overview[END_REF]] for an overview. 2012]. Given the STFT X 1 and X 2 of the two microphones signals, the CC and GCC-PHAT angular spectra are de ned as:

Ψ CC (τ ) = k,l X 1 [k, l]X * 2 [k, l]e -i2πf k τ , (10.3) Ψ PHAT (τ ) = k,l X 1 [k, l]X * 2 [k, l] |X 1 [k, l]X * 2 [k, l]| e -i2πf k τ , (10.4)
where * denotes the complex conjugate, [k, l] indexes a TF bin of the STFT and f k is the k-th natural frequency in the STFT.

The weighting function 1/|X 1 [k, l]X * 2 [k, l]| is called Phase Transform (PHAT). This function was introduced to suppress the source's autocorrelation component from the angular spectrum. Then, the TDOA estimate is given by τTDOA = arg max τ Ψ(τ ), with Ψ being either Ψ CC (τ ) or Ψ PHAT (τ ). Note that these functions can also be expressed directly as a function of the AOA using (10.2), hence the term angular spectrum. Despite the theoretical limitation of CC-based methods presented in [START_REF] Chen | Time delay estimation in room acoustic environments: an overview[END_REF], they are known to work well in practice under moderate reverberation and noise. Moreover, it was showed to be state-of-the-art for SSL in a large benchmark study [START_REF] Blandin | Multi-source TDOA estimation in reverberant audio using angular spectra and clustering[END_REF].

Multichannel 2D-SSL

When more microphones are available and the microphone array is compact and not linear 68 , 2D-SSL can be envisioned. A possible approach is to use 1D- 68 In the case where the microphones are complanar, the angle can be estimated up to an "up-down" ambigity.

SSL on all pairs and combine their results, a principle which was successfully applied in the Steered Response Power with Phase Transform (SRP-PHAT) method [START_REF] Dibiase | Robust localization in reverberant rooms[END_REF]].

The SRP-PHAT method returns the source's DOA, namely azimuth-elevation pair (θ, φ), by estimating TDOAs in all microphone pairs. In order to achieve this, it requires the geometry of the microphone array to be known. In a nutshell, this algorithm aims to estimate a global angular spectrum Ψ SRP (θ, φ) in the polar coordinates system with respect to a reference frame in the array, typically centered at its barycenter. This function will exhibit a local maximum in the direction of the active source. The algorithm can be decomposed into in the following steps:

See MBSSLocate for a free MATLAB implementation and comprehensive documentation of this algorithm.

1. a global grid of DOAs candidates is de ned according to a desired resolution and computational load; 5. the DOA(s) maximizing Ψ SRP is (are) used as estimate(s).

M A A E

We now introduce the proposed concept of Microphone Augmentation with Echoes (MIRAGE). In the well-known Image Source Method (ISM) all the re ections are treated as mirror images of the true source with respect to re ective surfaces, emitting the same signal. We will employ here a less common but equivalent interpretation to ISM, namely, the image-microphone model (see § 9.1). As illustrated in Figure 10.5, image microphones are mirror images of the true microphones with respect to re ective surfaces. In this view, the echoic signal received at a true microphone is the sum of the anechoic signals received at this microphone and its images. If we consider the virtual array consisting of both true and image microphones, multiple microphone pairs are now available. For each of them, it is then possible to de ne a corresponding time di erence of arrival. Among them, we will refer to the one de ned in § 6.3, that is,

τ TDOA = 1 c x 2 -s -1 c x 1 -s = τ (0) 2 -τ (0) 1 , τ iTDOA = 1 c x2 -s -1 c x1 -s = τ (1) 2 -τ (1) 1 , τ TDOE1 = 1 c x1 -s -1 c x 1 -s = τ (1) 1 -τ (0) 1 τ TDOE2 = 1 c x2 -s -1 c x 2 -s = τ (1) 2 -τ (0) 2 (10.5)
where xi denotes the position of the image of x i with respect to the re ector.

It is easy to see that τ TDOE2 = τ TDOE1 + τ iTDOAτ TDOA . Therefore, we will consider only the set V = [τ TDOA , τ iTDOA , τ TDOE1 ] ∈ R 3 . These three quantities are directly connected to RIRs, as illustrated in Figure 10.6. To learn the parameters V , we consider the LANTERN data-driven approaches presented in Chapter 6. It consists in a class of DNN architectures trained to perform multi-target regression from the input audio features Interchannel Level Di erence (ILD) and Interchannel Phase Di erence (IPD) to the parameters V ∈ R 3 . In fact, those models were proposed exactly to address the task of estimating the rst and strongest echo per channel in a close-surface scenario. Note that here the network provides directly a (unique) set of TDOA estimates, not "continuous" CC values.

Following the SRP-PHAT-like 2D-SSL scheme described in § 10.3.2 and given the virtual microphone-array geometry (which depends on the relative position of microphones to the surface), the TDOA estimates in V can in principle be used to estimate the 2D direction of arrival of the source if provide as continues CC values. To this end, the 3 real-valued estimates in V are interpolated to create virtual local angular spectra (see § 10.3.2). In the rst investigation of [Di Carlo et al. 2019], we proposed to use Gaussians functions centered at those estimates with variances equal to the prediction errors made by the DNN on the validation set. Later, we modi ed the networks in order to estimated both these means and the variances, as explained in § 6.4.

Experimental Results

In this section we will report the experimental results for the proposed approach. At rst, we will consider the continuation of the investigation started in § 6.3, namely, the analysis is limited to stereophonic recordings in a singlesource close-surface scenario.

Here we compare the MIRAGE approach using the LANTERN MLP (MIRAGE-MLP) learning model. Note that in § 6.4 we demonstrated that MLP is outperformed by more recent DNN architecture. However, here we are reporting the results obtained in the work [Di Carlo et al. 2019], antecedent to the experimental results reported in § 6. the observation with AWGN noise at 10 dB SNR (wn+n, sp+n, respectively). In this stereophonic experiment, the baseline method GCC-PHAT is not considered: it can access only the true microphone signals, thus, it is unable to perform 2D-SSL. The dataset, the implementation, and the training procedure are the same as in § 6.3.3. In particular, the test set features both white noise (wn), and speech (sp) signals convolved with RIRs generated by the acoustic simulator [START_REF] Schimmel | A fast and accurate "shoebox" room acoustics simulator[END_REF]]. Moreover, some observations feature background AWGN at 10 dB SNR level (wn+n, sp+,n respectively). N R Table 10.1 reports the performance of the full MIRAGE-MLP 2D-SSL pipeline. Within a tolerance of 20°, the MIRAGE-MLP model allows estimation of 2D-DOA with an accuracy of 79% for azimuth and 88% for elevation for a source emitting white noise in noiseless conditions. It is interesting to notice that our model trained and validated with white noise sources somewhat generalizes to speech sources. In fact, for noiseless speech signals the performances are just slightly lower than the corresponding broadband case. As already observed in § 6.3, external noise severely degrades performances.

With this work, we demonstrated how a simple echo model could allow 2D SSL with only two microphones, using simulated data. Motivated by these results, we put lot of e ort in collecting real-world data in order to test the whole MIRAGE pipeline, including LANTERN. One direction was o ered by a collaboration with Randy Gomez from the Honda Research Institute. The idea was to test the proposed method on the microphone array of the autonomous robot platform called HARU [START_REF] Ackerman | HARU: An Experimental Social Robot From Honda Research[END_REF][START_REF] Gomez | HARU: Hardware design of an experimental tabletop robot assistant[END_REF]]. We collected real-world data, but unfortunately due to several problems identi ed in the resulting dataset, these recordings were discarded.

C

This chapter demonstrated how a simple echo model could boost an SSL algorithm in strongly echoic conditions when microphones are placed close to a re ector. We proposed to use a successful algorithm for multichannel SSL on the virtual array created by image microphones. In order to leverage such an array, the echoes' parameters need to be estimated. To this end, we used the learning-based acoustic echo retrieval methods proposed in Chapter 6. Preliminary results on synthetic data for stereophonic recordings prove the e ectiveness of the proposed approach. However, the task is still very challenging in the presence of noise for both the proposed and baseline methods.

Considering the current knowledge, this is the rst time an echo-aware method combines both knowledge-driven and data-driven approaches to SSL. The learning approach could still be signi cantly improved by considering other acoustic features (such as advance ReTF methods), other architectures, and other improvements. For instance, handling the missing frequencies of the speech while training on a broadband signal such as in [START_REF] Gaultier | VAST: The virtual acoustic space traveler dataset[END_REF]], using physics-driven regularizers such as in [START_REF] Nabian | Physics-driven regularization of deep neural networks for enhanced engineering design and analysis[END_REF] and making the learning phase independent of the array geometry. Secondly, the approach needs to be tested on real-world data, for which the dEchorate could be used as a valuable testing dataset in the future. • dEchorate

Echo-aware Applications of dEchorate

• Risotto

• Brioche from the dataset dEchorate. In particular, we exemplify the utilization of these data considering two possible use-cases: echo-aware speech enhancement ( § 11.1) and room geometry estimation ( § 11.2). This investigation is conducted using state-of-the-art algorithms described and contextualized in the corresponding sections. In the nal section ( § 11.3), the main results are summarized, and future perspectives are presented.

This chapter is the continuation of the work presented in Chapter 7. Therefore, it is the results of the collaboration with prof. Sharon Gannot and ing. Pinchas Tandeitnik at the Bar'Ilan University, Israel. The algorithms presented here are straightforward extension of the ones available in the literature. Nevertheless, they are presented according to the thesis notation. In addition, they are gathered and implemented in the following Python library available online:

dEchorate related to the DECHORATE dataset, Risotto for RIR estimation and Brioche for echo-aware beamforming.

E S

In the previous chapters, we showed how to integrate echoes for sound source separation (Chapter 9) and sound source localization (Chapter 10). In this section, we investigate this in the context of spatial ltering. To this end, we compare two types of spatial lters: echo-agnostic and echo-aware beamformers.

In order to study their empirical potential, we will evaluate their performances on both synthetic and measured data, as available in the dEchorate dataset (Chapter 7). For all the methods presented in this part of the thesis, we assume that echoes are known. To this end, we used the annotations that come with the considered dataset.

Literature review

The following paragraphs provide a broad overview of existing beamforming methods, with a speci c focus on how they handle echoes. Spatial ltering methods exist in many forms, one of the most popular of which being beamforming.

135 E do not need any echo-estimation step: they either ignore their contributions, such as the direct-path beamformers [START_REF] Van Trees | Optimum array processing: Part IV of detection, estimation, and modulation theory[END_REF], or they consider coupling lters between pairs of microphones, using so-called Relative Impulse Response (ReIR) [START_REF] Gannot | Signal enhancement using beamforming and nonstationarity with applications to speech[END_REF]. In their vanilla form, neither approaches compute the full acoustic channels explicitly. In case of direct-path Delay-and-Sum (DS) beamformers, only the DOA of the target source is used to build the so called (relative) steering vector. Then, in order to cope with distortions due to reverberation, external noise or interfering speakers, the statistical description of such forms of noise can be included in extended beamformer design, such as the MVDR beamformer [START_REF] Van Trees | Optimum array processing: Part IV of detection, estimation, and modulation theory[END_REF]. The ReIRs (and their frequency counterparts, ReTFs) have been introduced with the explicit purpose of avoiding the computation of the acoustic channel related to each microphone [START_REF] Gannot | Signal enhancement using beamforming and nonstationarity with applications to speech[END_REF]. ReIRs-based beamformers instead of returning the dry source signal, return the reverberant source spatial image as it is recorded at a reference microphone. Compared to the di cult task of estimating the acoustic channels and of relying on bad channel estimates, this is typically su cient for achieving good enhancement performances in many practical scenarios. 71 Since then, ReIRs have been incorporated in powerful beamforming algorithms, used for both dereverberation and noise reduction (e. g. [START_REF] Schwartz | Multi-microphone speech dereverberation and noise reduction using relative early transfer functions[END_REF][START_REF] Kodrasi | EVD-based multi-channel dereverberation of a moving speaker using di erent RETF estimation methods[END_REF]).

E

explicitly model multipath sound propagation instead. They can be seen as rake receivers, borrowing the idea from telecommunication where an antenna rakes (i.e. combines) coherent signals arriving from di erent propagation paths. In particular, they consider "extended" steering vectors, whose formulation uses known echo delays and attenuations [START_REF] Flanagan | Spatially selective sound capture for speech and audio processing[END_REF][START_REF] Jan | Matched-lter processing of microphone array for spatial volume selectivity[END_REF]. The underlying motivation is two-fold: on the one hand, they better describe the acoustic propagation of the source signal; on the other hand, the early echoes' energy is included in the estimated signal and not considered as noise to be removed. Later, this approach has been extended to enhance desired signals in the context of the cocktail party problem in [Dokmanić et al. 2015] and for noise and reverberation reduction [START_REF] Peled | Linearly-constrained minimum-variance method for spherical microphone arrays based on plane-wave decomposition of the sound eld[END_REF][START_REF] Javed | Spherical microphone array acoustic rake receivers[END_REF][START_REF] Kowalczyk | Raking early re ection signals for late reverberation and noise reduction[END_REF]. In particular, the work [START_REF] Peled | Linearly-constrained minimum-variance method for spherical microphone arrays based on plane-wave decomposition of the sound eld[END_REF] estimates early echoes in the spherical harmonic domain and uses their DOAs to build ReIR. However, this approach is not generalizable to all microphone array con gurations as it requires the deployment of (3D) spherical arrays. Alternatively, the authors of [Dokmanić et al. 2015] (with its extension to the time-domain [START_REF] Scheibler | Raking echoes in the time domain[END_REF]) propose to modify the original formulation of the DS and MVDR beamformer designs to include the knowledge of the echoes as image sources. While that study covers the case of interferer and noise reduction, the late reverberation reduction is not considered, which was instead addressed by the work in [START_REF] Kowalczyk | Raking early re ection signals for late reverberation and noise reduction[END_REF]].

In this section, we compare the beamformer designs proposed in [START_REF] Kowalczyk | Raking early re ection signals for late reverberation and noise reduction[END_REF]] for both noise and late reverberation reduction. Besides, we take a di erent perspective: we investigate the bene t of echo knowledge when using either synthetic or measured impulse responses.

Background in spatial ltering

Given the narrowband STFT signal model presented is § 3.2.5, the signals captured by I microphones listening to a single sound source (J = 1) in a noisy reverberant room reads:

X[k, l] = H[k]S[k, l] + N [k, l], (11.1)
where

X[k, l], H[k], N [k, n] ∈ C I and S[k, l] ∈ C.
Note that since only one sound source is considered (J = 1), for a given TF bin, the mixing matrix reduces to the vector H[k] and the source contribution reduces to the complex scalar S[k, l]. Hereafter, we omit the dependency on the discrete time-frequency bin [k, l] for the sake of clarity.

The lter vectors can be decomposed in order to highlight the sound propagation components, that is,

H = H dp i + H ee i + H lr i i (11.2)
where the summands correspond to the direct-path (dp), early echoes (ee) and late reverberation (lr), respectively. It is sometimes common to consider the entire early contributions, H ec comprising both direct and echo paths. We can now model the such part of the RIR associated to the i-th channel according the echo model, that is,

H ec i = H dp i + H ee i = R r=0 α (r) i e -i2πf k τ (r) i , (11.3)
where r = 0 is the index of the direct propagation. Given such a decomposition of the RIRs, the vector X can be expressed accordingly as: X = X ec + X lr + N . (11.4) In the context of echo-aware spatial ltering, the source signal of interest includes both the direct path and the R early re ections, as done in [Dokmanić et al. 2015;[START_REF] Kowalczyk | Raking early re ection signals for late reverberation and noise reduction[END_REF]]. This assumption is motivated by psychoacoustics studies as discussed in § 2.4: the rst early echoes are shown to contribute to increasing speech intelligibility, as they are fully integrated into the direct sound, increasing its perceived intensity [START_REF] Bradley | On the importance of early re ections for speech in rooms[END_REF]]. Based on this, we de ne as the signal of interest the following

X ec = X dp + X ee = H ec S. (11.5)
The noise and the late reverberation are typically described as random processes for which priors are typically available. Therefore, it is common to express the microphone signal model of Eq. ( 11.4) from a statistical point of view. Under the assumption of source and noise signals being statistically independent, we can de ne the covariance matrix of the microphone signals, 11.6) where E[•] denotes the expectation operator. Here Σ lr x and Σ n denote the PSD matrices of the late reverberation and noise, respectively. We will describe each term in turn.

Σ x = E XX H ∈ C I×I , as Σ x = H ec Σ s H ecH + Σ lr s + Σ n , (
T covariance matrix Σ s is assumed here to be diagonal, since we assume that all the spatial information is expressed by the lters H ec and the source signals are independent to each other, that is, (11.7) where I is the identity matrix of dimension I × I and σ 2 s = E |s| 2 .

Σ s = σ 2 s I,
T covariance matrix can be estimated using the timeinvariant spatial coherent matrix model proposed in [Kuster 2012], based a di use sound eld model [START_REF] Kuttru | Room acoustics[END_REF]].

Σ lr

x = σ 2 lr Γ, (11.8) where σ 2 lr denotes the power of the late reverberation and Γ is the I ×I spatial coherence matrix, which is available in closed-form. 72 This approach has been 72 Given the distance d ii between to microphone i and i , the interchannel coherence function in the continuous-frequency domain writes

γii [k] = sin(2πf k d ii /c) 2πf k d ii /c .
Then, the matrix Γ = γii is built for all the pairs of channel on a discrete set of frequencies.

found successful in many dereverberation application [START_REF] Naylor | Dereverberation, Speech[END_REF][START_REF] Cauchi | Joint dereverberation and noise reduction using beamforming and a single-channel speech enhancement scheme[END_REF][START_REF] Tammen | Iterative Alternating Least-Squares Approach to Jointly Estimate the RETFs and the Di use PSD[END_REF].

T term N is assumed to be stationary over the recording. Therefore its covariance matrix can be estimated from the observation using non-speech segments. In a blind setting, this would require the usage of a voice activity detector. Alternatively, the noise component can be modeled as a stationary random process whose spatial covariance matrix can be estimated with advanced techniques exploiting speech non-stationarity [START_REF] Gannot | Signal enhancement using beamforming and nonstationarity with applications to speech[END_REF].

Elements of Beamforming

In the STFT domain, a beamformer forms a linear combination of the microphone channels to yield the desired output Y ∈ C:

Y = W H X = W H HS + W H N ,
where the vector W ∈ C I contains the beamformer weights. These weights can be computed by optimizing di erent design criteria which will be described below.

T D S is the simplest and often a quite e ective beamformer. In its vanilla version, the DS is designed to only compensate the propagation delay from the source to the microphones along to the ideal propagation path. Assuming the far eld scenario and i = 0 to be the reference microphone, this is typically achieved using the direct-path relative steering vector D de ned in Eq. (3.33), that is, 11.9) where f k is the k-th frequency bin in Hz, τ (0) i is the TOA of the direct-path for channel i, and c is the speed of sound. Therefore, the beamformer weights reads 11.10) where • denotes the Euclidean norm.

D = 1, e -i2πf k τ (0) i /c , . . . , e -i2πf k τ (0) I /c (
W dp = D D , (
T MVDR beamformer optimizes the following criterion 73 73 The MVDR design is equivalent to the Minimum Power Distortionless Response (MPDR) beamformer which minimize the following

W MPDR = arg min W W H ΣxW s.t. W H H = 1.
However, it exhibit higher sensitivity to misalignment errors than the MVDR beamformer [Gannot et al. 2017, Section V.A]. (11.11) It aims at minimizing the total output energy (i. e., minimum variance) while simultaneously keeping the unit gain of the array on the desired signal xed (i. e. distortionless response). Therefore, the reduction of the output energy suppresses any external noise modeled by Σ n . The Least-Square minimization through the Lagrangian multiplier method yields the following closed-form optimal solutions .12) Note that these techniques do not require any reference signal, only the knowledge of the source's lter H and an estimate of the observed signal's covariance matrix. This criterion design can be easily extended to work with any type of noise and acoustic transfer functions modeled by Σ n and H, respectively.

W MVDR = arg min W W H Σ n W s.t. W H H = 1.
W MVDR = Σ -1 n H H H Σ -1 n H . ( 11
11.1.4 Noise, steering vectors, rake lters, and relative transfer functions

The vectors H between the source and the I microphones account for the RTFs.

To overcome the complexity of estimating them, three main directions have been pursued: steering vectors, rake receivers and relative transfer functions.

S are the RTFs of the ideal propagation path component, namely, the two are equivalent in anechoic scenarios. They can be built on the knowledge of the TOAs of the source signal and their integration in the MVDR criterion is straightforward: the H simply identi es with the corresponding relative steering vector D, that is, .13) In distant-talking scenarios, relative time delays, or TDOAs, with respect to a reference microphone, are used. Moreover, knowing the microphone array geometry, the TDOAs may be derived from the source's DOA 74 , thus, the 74 The mapping between TDOAs and DOAs is not always unique. It depends on the microphone array geometry, such as its compactness, its shape, and the number of sensors.

H DP [k] = D[k] = e -i2πf k τ (r) i i . ( 11
steering vectors can be easily computed. However, DOAs need to be estimate aside, using Sound Source Localization (SSL) methods (See § 8.4).

T are beamformers that explicitly deal with the multipath propagation model in Eq. ( 11.3). To this end, the MVDR design is modi ed by integrating the spatial information of R re ections into the distortionless constraint. This turns out to be equivalent to extending the de nition of (relative) steering vectors as follows: .14) As before, both the echoes' delays and amplitudes are considered relatively to the reference microphone.

H RAKE [k] = R r=0 α (r) i e -i2πf k τ (r) i i . ( 11
T R T F (R TF) was originally proposed in the work of [START_REF] Gannot | Signal enhancement using beamforming and nonstationarity with applications to speech[END_REF] to overcome the limitation of accessing the full RTF for each channel. Given the RTF of the i-th channel H i , its ReTFs with respect to the rst channel is given by 11.15) In a reverberant environment, it contains both direct-path information and information representing early and late reverberations. More details are given in § 3.3.2.

H ReTF [k] = H[k] H 1 [k] (
T is taken into account through the covariance matrix Σ n . This term could potentially include every source of "noise", such as interfering sources, measurement noise as well as di use background noise. As long as the power spectral density of the modelled noise source is available, its suppression can be achieved by including it in Σ n and using it in Eq. ( 11.12). 75 75 This type of criterion design is more properly known as MaxSINR or MaxSNR beamformers. In general, they aim at optimizing directly the SNR or the Signal-to-Interferenceplus-Noise-Ratio (SINR) metrics. Nevertheless, it can be shown that the any MaxSINR (or MaxSNR) can be identi ed with an MVDR if the distortionless contraint is satis ed [START_REF] Gannot | A consolidated perspective on multimicrophone speech enhancement and source separation[END_REF].

For instance, 11.16) where the summands are the covariance matrix of an interfering source q and of the an independent background noise n.

Σ n = Σ sq + Σ n , (
Assuming stationarity of the noise sources, a naïve approach to estimate their contribution is to use voice activity detection or speaker diarization 76 tools. If 76 Speaker diarization is the problem of partitioning an audio signal into segments according to the source identities. In other words, it addresses the problem of "who is speaking when".

these excerpts are long enough, the whole covariance matrix Σ n , including both noise and interferer, is well approximated by its sampled version whose calculation is straightforward. Alternatively, other designs can be used, for instance, in the LCMV beamformer, interference reduction is achieved using steering vectors for all interfering sources of the interfering source instead of their PSD matrices.

T L R contribution Σ lr s is not directly available in isolated audio segments as it is "glued" to the target signal. A common way to achieve dereverberation in a beamformer design [START_REF] Schwartz | Multi-microphone speech dereverberation and noise reduction using relative early transfer functions[END_REF][START_REF] Thiergart | An informed parametric spatial lter based on instantaneous direction-of-arrival estimates[END_REF][START_REF] Kowalczyk | Raking early re ection signals for late reverberation and noise reduction[END_REF] is by adding the PSD of the late di use part of Eq. (11.8) to the noise covariance matrix, that is,

Σ noise-late = Σ u + Σ lr s = Σ u + σ 2 lr Γ. ( 11
.17)

Considered beamformers

In this work we evaluate the performance of echo-agnostic and echo-aware beamformers for noise and late reverberation suppression. Table 11.1 summarizes the considered beamformers designs.

In this work, the elements used to build the beamformers are estimated as follows:

• the noise covariance matrix, Σ u , is estimated from 0.5 second excerpt of di use noise only audio segments (this is equivalent to having access to an ideal voice activity detector);

W = (HDP H Σ -1 u HDP) -1 Σ -1 u HDP MVDR-ReTF min W H ΣuW s.t. W H HReTF = 1 W = (HReTF H Σ -1 u HReTF) -1 Σ -1 u HReTF MVDR-Rake* min W H ΣuW s.t. W H HRAKE = 1 W = (HRAKE H Σ -1 u HRAKE) -1 Σ -1 u HRAKE MVDR-DP-Late min W H (Σu + Σlr)W s.t. W H HDP = 1 W = (HDP H (Σu + Σlr) -1 HDP) -1 (Σu + Σlr) -1 HDP MVDR-ReTF-Late min W H (Σu + Σlr)W s.t. W H HReTF = 1 W = (HReTF H (Σu + Σlr) -1 HReTF) -1 (Σu + Σlr) -1 HReTF MVDR-Rake-Late* min W H (Σu + Σlr)W s.t. W H HRAKE = 1 W = (HRAKE H (Σu + Σlr) -1 HRAKE) -1 (Σu + Σlr) -1 HRAKE T 11
.1: Summary of the considered beamformers. (*) denotes echo-aware beamformers.

• the ReTFs, H ReTF , is estimated from the observed signal using the Generalized Eigenvalue Decomposition (GEVD) method described in [Doclo and Moonen 2003];

• H DP , H RAKE are computed using known relative delays and amplitudes available in the DECHORATE dataset;

• the late reverberation power, σ 2 lr , is estimated in closed-form knowing the lters H and the noise PSD, as suggested in [Schwartz et al. 2016]. Recently, in [START_REF] Tammen | Iterative Alternating Least-Squares Approach to Jointly Estimate the RETFs and the Di use PSD[END_REF]] it was proposed an iterative Least-Square approach to estimate the late PSD together with the Relative Early Transfer Functions (ReETFs). This work is based on [START_REF] Kodrasi | EVD-based multi-channel dereverberation of a moving speaker using di erent RETF estimation methods[END_REF], where the e ectiveness of such approach was shown.

Experimental evaluation

The performances of the di erent designs are compared on the task of enhancing a target speech signal in a 5-channel mixture using a linear array from the dEchorate dataset. In particular, they are tested in scenarios featuring high reverberation and di use babble noise, appropriately scaled to given pre-de ned signal-to-noise ratio (SNR ∈ {0, 10, 20}). Using the dEchorate data, we considered the room con guration 011111 (RT 60 ≈ 600 ms) and all the possible combinations of target/array's positions. Both real and corresponding synthetic RIRs are used, which are then convolved with anechoic utterances from the WSJ corpus and corrupted by recorded di use noise. The correspondence between real and synthetic RIRs is a property of the dEchorate dataset and is described in Chapter 7. The synthetic RIRs are computed with the pyroomacoustics Python library, based purely on the Image Source Method (ISM).

The evaluation is conducted similarly to the one in [Dokmanić et al. 2015;[START_REF] Kodrasi | EVD-based multi-channel dereverberation of a moving speaker using di erent RETF estimation methods[END_REF][START_REF] Kowalczyk | Raking early re ection signals for late reverberation and noise reduction[END_REF]]. Here we consider the rst microphone as the reference one and the clean target signal as the clean signal convolved with the early part of the RIR (up to the R-th echo), namely, X ec in Eq. (11.5). On one hand, this choice numerically penalizes both direct-path-based and ReTF-based beamformers which aim at extract the direct-path signal only and the reverberant signal at the reference microphone, respectively. On the other hand, considering only the direct path or the full reverberant signal would be equally unfair for the other beamformers. Moreover, choosing the early contribution as target signal is perceptually motivated by the fact that early re ection contributes to speech intelligibility [START_REF] Bradley | On the importance of early re ections for speech in rooms[END_REF]]. For evaluating the performances, we consider the following metrics: T S N R I (∆SNRR) in [dB] is the di erence between the input SNRR i at the reference microphone and the SNRR o at the lter output. Denoting with X 1 the signal at the reference microphone and X ec 1 the target speech at the same microphone (See Eq. (11.5)), these quantities are de ned as follows:

SNRR i = 10 log 10 σ 2 X ec 1 σ 2 X1 -σ 2 X ec 1 [dB] SNRR o = 10 log 10 σ 2 W H X ec σ 2 W H X -σ 2 W H X ec [dB] ∆SNRR = SNRR o -SNRR i [dB]
(11.18)

T S R M R (SRMR) I (∆SRMR) is an adimensional and absolute (i. e., it does not require the reference signal) measure of dereverberation and was initially proposed in [START_REF] Falk | A non-intrusive quality and intelligibility measure of reverberant and dereverberated speech[END_REF]. It is based on the modulation spectrum, which allows for reliable characterization of the speech envelope smearing due to reverberation [START_REF] Santos | Updating the SRMR-CI metric for improved intelligibility prediction for cochlear implant users[END_REF]]. 77 Later it has been applied in several works 77 The modulation spectral domain quanties the envelop uctuations that can be attributed to perceptive room acoustics characteristics, such as late reverberation levels and coloration (See § 2.4).

addressing dereverberation as well as in some challenges on speech processing, such as the ACE challenge [START_REF] Eaton | The ACE challenge-Corpus description and performance evaluation[END_REF]. It is computed as follows:

• the processed speech signals ltered by a 23-channel Gammatone lterbank to emulate the processing performed by the cochlea.

• the envelope of each output of the lter is computed though Hilbert transform in order to extract the temporal dynamics information (See Eq. (4.2)).

• the modulation spectrum is computed as the 8-bands STFT on a selected frequency range (up to 8 Hz) of each envelope.

• the SRMR is obtained as the ratio of the average modulation energy content available in the rst four modulation bands, to the average modulation energy content available in the higher frequency modulation bands.

An implementation of these metrics is available at the speechmetrics Python library.

T P E S (PESQ) is a relative adimensional metrics presented in [START_REF] Rix | Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs[END_REF]] and outputs a score ranging from 1 (bad) to 5 (excellent). This measure, assumed to cover several speech degradation and distortion, was promoted as a standard in the ITU-T recommendation P.862. It was originally used for telecommunication and telephony and it is now considered one of the most reliable metrics to predict the overall speech quality. Practically, after applying an auditory model to the reference and distorted (i. e., the estimated) signals (based on a Bark frequency scale) the loudness spectra are estimated. From the loudness spectra di erences, a Mean Opinion Score (MOS) 78 is inferred using a linear regression model. 78 The MOS is a measure used in the domain of Quality of Experience and telecommunications engineering, representing overall quality of a stimulus or system. MOS were originally obtained from perceptual tests involving human subjects.

An implementation of this metrics is available at the speechmetrics Python library.

N

are reported in Figure 11.1. When using synthetic data, the known echo timings perfectly match the components in the simulated RIRs, and, likewise, the echo model matches the RIRs' early part. Here, one can see that more information is used, the better performances are. ReTFs-and Rakebeamformers outperform the simple designs based on the direct path, and including the late reverberation statistics considerably boosts performance in all cases. Interestingly, ReTFs have a slight edge over Rake-versions in terms of mean Signal-to-Noise-plus-Reverberation-Ratio (SNRR). This can be explained by the fact that GEVD methods tend to robustly consider the stronger and more stable components of the ReTFs, which in reverberant and noisy static scenarios may identify with the earlier portion of the RIRs. Moreover, since it is not constrained by a x echo model, the ReTFs can capture more information which, in turn, yields to sightly better enhancement. Nevertheless, the PESQ metrics suggest that for this ideal scenario both echo-aware (Rake) and echoagnostic (ReTFs) design are comparable. When it comes to measured RIRs, the trends are di erent. Here, the errors in echo estimation, due to calibration mismatch and the richness of the acoustic propagation, lead to a drop in the performances for echo-aware methods, both in terms of means and variances. This is even clearer when considering the ∆PESQ metric, as it also accounts for artifacts. Here, the echo-agnostic beamformer considering late reverberation MVDR ReTF+Late outperforms the other methods, maintaining the trend exhibited on simulated data. In general, it looks like the MVDR Rake+Late has more variance than MVDR ReTF+Late , suggesting that in some situations, it performs better, while in other it performs less well. This is probably due to the tiny annotation mismatch and the complexity of the RIRs and future work will be devoted in a deeper in understanding of the underlying factors.

R G E

In this section, we shortly present another application of the dEchorate dataset: Room Geometry Estimation (RooGE), namely, the task of estimating the shape of a room knowing the positions of rst-order image sources. This problem is typically addressed by solving multiple instances of re ector localization, aiming at estimating the position of a single surface (e. g. wall, oor, etc.). Several methods have been proposed which take into account di erent levels of prior information and noise. They were brie y discussed in the context of echo labeling in § 4.3.1. In general, these methods can be decomposed into three successive steps:

1. echo labeling, in order to associate the echoes to image sources using one of the methods mentioned in § 4.3.1;

2. estimation of the image source position either through multilateration [START_REF] Dokmanić | Acoustic echoes reveal room shape[END_REF]Dokmanić et al. 2015] (see below), Maximum Likelihood (ML) [START_REF] Tervo | Localization and tracing of early acoustic re ections[END_REF]] or convex optimization [START_REF] Crocco | A closed form solution to the microphone position self-calibration problem[END_REF];

3. and nally, the image-source-reversion, in order to localize the re ector, based on the geometrical assumption of the Image Source Method (ISM).

More advance techniques have been proposed in the literature of re ector localization for di erent setups and scenarios. A comprehensive review can be found in [START_REF] Remaggi | Acoustic re ector localization: novel image source reversion and direct localization methods[END_REF][START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF]]. Nonetheless, when the echoes' TOAs and their labeling are known for 4 spatially-separated non-coplanar microphones, one can perform this task using closed-form multilateration algorithms.

Room Geometry Estimation through multilateration

Multilateration is the problem of recovering the position of a point in the space from multiple distances between the point and known spatially-separated locations. It is the 3D extension of the trilateration problem, namely, determining an unknown position based on the distance to two other known vertices of a triangle. In the context of RooGE, the distances from the source to the microphones can be obtained by converting the TOAs [seconds] into distances [meters]. Then, the 3D coordinates of each image source can be retrieved, solving a convex problem as described in [START_REF] Beck | Exact and approximate solutions of source localization problems[END_REF]. Ideally, this problem can be solved in closed-form. However, due to measurement error (e. g. errors in estimating the image's TOAs), it may be ill-conditioned. To overcome this, the algorithm proposed in [START_REF] Beck | Exact and approximate solutions of source localization problems[END_REF]] relies on a robust iterative approach yielding accurate solutions. Finally, the position and orientation of each wall can be easily derived from the ISM as the plane bisecting the line joining the real source position and the position of its corresponding image (see Figure 11.2).

Using the dEchorate dataset for RooGE

In dEchorate, the annotation of all the rst-order images of sound sources is available. We used the Beck et al.'s multilateration method (available in the Python library dEchorate) to estimate the image source position of each of the direct source using 6 non-coplanar microphones (on for each of the 6 arrays). Then, room facets are estimated using each of the sources as a probe. Table 11.2 shows the results of the estimation of the wall positions in terms of distance error (in centimeters) and surface orientation error (in degrees). These metrics were previously used in the literature of re ector estimation, such as in [START_REF] Annibale | Geometric inference of the room geometry under temperature variations[END_REF][START_REF] Crocco | Uncalibrated 3D room geometry estimation from sound impulse responses[END_REF]. Figure 11.2b 11.2: Distance errors (DE) in centimeters and angular errors (AE) in degrees between ground truth and estimated room sides using each of the sound source (#1 to #4) as a probe. For each facet, bold font is used in correspondence to the source yielding the best DE and AE; while, italic font highlights outliers.

Despite a few outliers, the majority of the facets are estimated correctly in terms of their placement and orientation with respect to the coordinate system. For instance, for source #4, all 6 surfaces were localized within less than 6 cm and 2.5°errors. Small errors are due to concurrency of multiple factors, such as tiny o sets in the annotation and the ideal shoebox approximation 79 . It 79 In the real recording room, some gaps were present between revolving panels in the walls is also possible that for some source-receiver pairs, the far-eld assumption is not veri ed, causing the inaccuracy of reverting the ISM. Finally, the 2 outliers for source #3 are due to a wrong annotation caused by the source directivity and misclassi cation. In particular, when a wall is "behind" the source, the energy of the related re ection is very small and might not appear in the RIRs. This happened for the eastern wall, and a second-order image was taken instead. Secondly, the contribution of multiple re ections arriving at the same time can be merged into signal spikes in estimated RIRs. This e ect is particularly ampli ed when the microphones and loudspeakers exhibit long impulse responses. As a consequence, some spikes were probably missclassi ed. This can be noticed for the southern-wall were again a second-order image was taken instead. Nevertheless, this second type of error can be manually corrected, and the annotations updated.

C

In this chapter, we presented two applications of the dEchorate dataset described in Chapter 7: echo-aware spatial ltering and room geometry estimation. The rst one deals with the possibility of using early echoes to enhance a target speech signal corrupted by di use noise and a high level of reverberation. To this end, two types of state-of-the-art spatial ltering criteria are considered: echo-agnostic and echo-aware beamformers. Experimental results on real and synthetic data, both available in the proposed dataset, led to the following ndings. The synthetic data were computed using ISM-based simulation; thus, the early parts of RIRs match the early echo model. Therefore, replacing the acoustic vectors with few know echoes gives signi cant enhancement performance gains compared with baseline methods, which consider only the direct ideal propagation. In this scenario, both echo-aware and state-of-the-art ReTFs-based echo-agnostic perform similarly, suggesting the e ectiveness of echo-aware approaches. However, when using the corresponding real data available in the dataset, performances drop in terms of perceptual quality as predicted by the PESQ score. This may be due to the small mismatches between real and annotated echoes and the richness of the acoustic eld, which impact the echo-aware methods. The best-performing method is the echo-agnostic one based on ReTFs, which does not su er from any echo mismatch and can include other information about the acoustic propagation. The relatively lower performance of rake-based lters on real scenarios than simulated ones emphasizes the importance of having precise enough AER algorithms and encourages further studies on these echo-aware methods. Moreover, the knowledge of the very same echoes is limited to spatial ltering and can be used to retrieve the entire room geometry, as demonstrated in the second section of this chapter. By using standard approaches based on geometrical reasoning and robust multilateration algorithms, it is possible to revert the ISM and map echoes' TOAs to source and image-source position. Here, we showed this on the dEchoratedata both as an application and as a way to validate the dataset. Although the results highlight that some echo TOAs have not been correctly classi ed, the overall annotation is consistent with the actual room planimetry. Finally, we would like to mention that the best of our knowledge, this is the very rst dataset of this kind, that is, featuring real data with full echo/image-source annotations.

Future works will explore several directions. Regarding the applications, the echo-aware methods presented above could be validated over more challenging scenarios than the one presented. Such scenarios, e. g. the presence of interfering sound sources and challenging levels of SNR and RT 60 , are already included in the dataset, but not used in the evaluation above. By the amount of the data collected, they can be used to train learning-based signal processing methods or for data augmentation. In a more long-term perspective, the data analysis conducted in this chapter brings the attention to exploring the impact of mismatch between simulated and real RIR on audio signal processing methods. Moreover, by using the pairs of simulated vs. real RIRs available in the dataset, it could be possible to develop techniques to convert one to the other, using style transfer and domain adaptation techniques.

Part IV EPILOGUE in echo-related tasks. Moreover, by evaluating state-of-the-art methods on these data and comparing their results with corresponding RIRs generated by acoustic simulators, we showed that current methodologies did not always generalize well to real acoustic conditions.

• An E A S S M , dubbed SEPARAKE. It is based on the popular Multichannel NMF framework, which allows simple yet e ective integration of the echoes' properties. Assuming their knowledge, we can reformulate such a framework in terms of image microphones and virtual arrays. Results show how this leads to enough spatial diversity to get a performance boost over the vanilla version of two classical NMF-based algorithms.

• An E S S L M , dubbed MIRAGE. By regarding echoes as image microphones, this method allows for source's azimuth and elevation estimation from passive stereophonic recordings. Therefore, the strong echo coming from a close re ective table can be used to create a virtual array on which powerful array processing techniques can be applied. This method allows for simple extensions to multi-channel recordings as long as the geometry of the array is available.

• The following for echo-aware processing:

dEchorate -code for DECHORATE, Room Impulse Response es-dEchorate timation and annotation.

Joint work with Pinchas Tandeitnik.

L A

The work presented in this dissertation took a few steps towards echo-aware audio signal processing. Nevertheless, there are many issues and many potential areas of improvement in the methods presented here. Besides, it is clear that we have only scratched the surface of many problems related to echo processing. In the following points, we elaborate on some short and long term research possibilities that arise as natural follow-ups to the topics discussed thus far.

E

This thesis' work highlighted the di culty of estimating echoes accurately in arbitrary scenarios. Using prior information, the problem can be substantially relaxed, for instance, knowing the microphone array's geometry. The impact of the microphone array (and antenna) geometry has been well studied in various branches of signal processing, such as telecommunication. Therefore, the knowledge of array geometry could extend the echo model to closed-form steering vectors, depending on both the echo timing and direction of arrival. This might simplify the parameter space where the echoes are searched for.

Even if this approach was used for the considered echo-aware applications, it was not for the proposed AER methods, which are currently restricted to the stereophonic case. Recent works in [START_REF] Jensen | An EM method for multichannel TOA and DOA estimation of acoustic echoes[END_REF]] (resp. in [START_REF] Kowalczyk | Raking early re ection signals for late reverberation and noise reduction[END_REF]) demonstrated the bene t of using this kind of steering vectors in AER (resp. echo-aware methods).

Besides this, we also noticed that a critical parameter for AER is the number of early echoes within a limited time window. Therefore the echo counting problem appears to be a reasonable direction to investigate. Estimating early re ections knowing their number can be recast as a sparse coding problem. Unfortunately, problems in this class are not necessarily easier to solve, but it was shown that they produce more accurate solutions [START_REF] Bourguignon | Exact sparse approximation problems via mixed-integer programming: Formulations and computational performance[END_REF][START_REF] Nadisic | Exact Sparse Nonnegative Least Squares[END_REF].

Another important line of study which we did not pursue here is to consider robust acoustic features instead of the simple magnitude and phase of the instantaneous ReTF. On the same direction, one could directly enhance the recorded sound with respect to noise and late reverberation. The most promising direction in this regard is to use recent results on relative early transfer functions estimation [Schwartz et al. 2016;[START_REF] Kodrasi | EVD-based multi-channel dereverberation of a moving speaker using di erent RETF estimation methods[END_REF][START_REF] Tammen | Iterative Alternating Least-Squares Approach to Jointly Estimate the RETFs and the Di use PSD[END_REF]]. Relative early transfer functions are transfer functions accounting for the early part of the ReTF only and are used for speech enhancement and dereverberation.

Alternatively, statistical models for reverberation that are parametrized by the RT 60 or the Direct-to-Reverberant Ratio (DRR) can be used. Particularly appealing are the works of [START_REF] Leglaive | Multichannel audio source separation with probabilistic reverberation modeling[END_REF]] and [START_REF] Badeau | Common mathematical framework for stochastic reverberation models[END_REF]], which de ne a framework to deal with early re ections within a statistical model. This exploratory direction is also motivated by recent results in estimating these parameters blindly from microphone recordings [START_REF] Looney | Joint Estimation Of Acoustic Parameters From Single-Microphone Speech Observations[END_REF][START_REF] Bryan | Impulse Response Data Augmentation and Deep Neural Networks for Blind Room Acoustic Parameter Estimation[END_REF]. As these parameters depend on the spatial characterization of the audio scene, using them as priors may seem natural. Finally, statistical models have the potential to solve issues due to the strong approximation used in the image source model, such as at walls, frequency-independent absorption coe cients and specular-only re ections.

Learning-based approaches and especially Deep Neural Networks (DNNs) are potent tools. Unfortunately, their potential was not fully exploited in this thesis work, but many exciting directions can be pursued. Of particular interest are the recent developments in physics-driven neural networks [START_REF] Nabian | Physics-driven regularization of deep neural networks for enhanced engineering design and analysis[END_REF][START_REF] Rao | Physics-informed deep learning for incompressible laminar ows[END_REF]Jin et al. 2020]. In these work physicallymotivated regularizers or entire loss-functions obeying physical laws are used to ensure the correctness of the solution or restrict the search space. It can be done by adding simple physical relations, such as the temperature-density relation for uids as in [START_REF] Karpatne | Physics-guided neural networks (pgnn): An application in lake temperature modeling[END_REF], or entire PDEs as in the other cited works.

U

In this thesis, we presented a few selected applications of acoustic echoes to audio scene analysis problems. This could be extended to other related problems such as binaural hearing, source tracking and acoustic measurements.

Humans have two ears and complex auditory systems that provide a natural ltering of incoming sounds. Moreover, re ections from the the torso and shoulders are integrated into the processing to provide localization cues [START_REF] Rascon | Localization of sound sources in robotics: A review[END_REF]. Therefore, one could imagine studying these re ections and settings in echo-aware processing to design better hearing aids, smart noise-canceling headphones, and auditory models. The latter ones are used to conduct experimental studies on sound perception [START_REF] Barumerli | Localization in Elevation with Non-Individual Head-Related Transfer Functions: Comparing Predictions of Two Auditory Models[END_REF], and to build perceptual evaluation metrics.

The MIRAGE framework presented in Chapter 10 can be extended in the long term to sound source localization and tracking, as well as to acoustic feedback cancellation (better know as acoustic echo cancellation) or to microphone array self-calibration. This has a great potential for industrial applications, as illustrated by vocal assistant devices such as Google Home or Amazon Echo. These smart speakers could bene t from echo-aware processing, not only in recording sounds but also in sound production. For instance, echo processing could provide dynamic sweet-spots that adapt to the user and device positions. The device could then produce an immersive sound eld for music listening or better directivity in hand-free phone-calls.

Finally, the frameworks proposed in the thesis could be expanded beyond the eld of indoor audio signal processing. Other research elds putting relevant focus on re ections include submarine navigation [START_REF] Kleeman | Sonar sensing[END_REF], in seismology [Sato et al. 2012], in ultrasound imaging [START_REF] Achim | Compressive sensing for ultrasound RF echoes using a-stable distributions[END_REF]] and in radioastronomy [START_REF] Pan | Towards generalized FRI sampling with an application to source resolution in radioastronomy[END_REF]].

U dEchorate

The main idea behind this dataset is to foster research in AER and echo-aware processing on real data. A natural step is to use these data for the applications discussed in this thesis and particularly for AER, which was the dataset's initial goal. Moreover, new lines of research using deep neural networks and optimal transport, such as style transfer and domain adaptation, could be envisioned. For instance, by using the pairs of simulated vs. real RIRs available in the dataset, one could develop techniques to convert one to the other. Ideally, this approach could be at the basis of a new type of learning-based acoustic simulators.

C

Ultimately, the two parts of estimation and application in this dissertation should be combined together. So far we only showed how from audio features it was possible to estimate echoes and how from echoes it was possible to estimate audio scene analysis information, e. g. source content and location. These problems have an innate uroboric nature: where, what, when and how are connected -the knowledge of one helps the estimation of the others, in a virtuous circle. Therefore it should be possible to build iterative schemes linking echo-estimation and echo-applications.

So long and thanks for all the echoes.

Algorithm 1: Sliding Frank-Wolfe algorithm for solving (5.22-P λ TV ). Input: Observation operator A, positive scalar λ, precision ε Output: Channels represented as a measure µ // Initialization 1 y ← -Aδ (0,1) // observation vector 2 µ (0) = 0 M // estimated filters 3 E (0) = ∅ // estimated echoes 4 x max = (2λ) In particular, using the real part in the implementation allows to remove the imaginary part that may appear due to the imprecision. P . Unfortunately, condition (12.1) cannot be met due to the machine precision, i.e., the solution of (5.22-P λ TV ) is computed up to some prescribed accuracy. In this paper, we say that the algorithm stops as soon as (12.4)

To solve this optimization problem, we rst nd a maximizer on a thin grid made of 20000 points. We then proceed to a local re nement using the scipy optimization library 80 .

80 https://docs.scipy.org/doc/scipy/ reference/generated/scipy.optimize. minimize.html.

N L (L 14). The nonnegative Lasso is solved using a custom implementation of a proximal gradient algorithm. In particular, the procedure stops as soon as a stopping criterion in terms of duality gap is reached (10 -6 ).

J

(L 15). In order to ease the numerical resolution, we show that given a positive integer R, the solution of (12.7) Indeed, let us denote θ , c the minimizers of (12.5). For any θ ∈ Θ R , the couple θ, 0 R is admissible for (12.5) so we have by de nition x max .

(12.9)

Finally, the joint update of the coe cients and parameters is performed using the Sequential Least SQuares Programming (SLSQP) implemented in the scipy optimization library, see Sidenote 80.

(Incomplete) Recommended Listenings Abstract: Most of audio signal processing methods regard reverberation and in particular acoustic echoes as a nuisance. However, they convey important spatial and semantic information about sound sources and, based on this, recent echoaware methods have been proposed. In this work we focus on two directions. First, we study the how to estimate acoustic echoes blindly from microphone recordings. Two approaches are proposed, one leveraging on continuous dictionaries, one using recent deep learning techniques. Then, we focus on extending existing methods in audio scene analysis to their echo-aware forms. The Multichannel NMF framework for audio source separation, the SRP-PHAT localization method, and the MVDR beamformer for speech enhancement are all extended to their echo-aware versions.

  signal processing . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Audio scene analysis . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 The echo-aware approach . . . . . . . . . . . . . . . . . . . . . 4 1.4 Thesis outline and main contributions . . . . . . . . . . . . . . . 5 1.5 List of contributions . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.6 Don't panic! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 I ROOM ACOUSTICS MEET SIGNAL PROCESSING 11 13 2.1 Sound wave propagation . . . . . . . . . . . . . . . . . . . . . . 13 2.2 Acoustic re ections . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.3 Room acoustics and room impulse responses . . . . . . . . . . . 19 2.4 Perception and some acoustic parameters . . . . . . . . . . . . . 27 2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 29 3.1 Signal model in the time domain . . . . . . . . . . . . . . . . . . 29 3.2 Signal model in the spectral domain . . . . . . . . . . . . . . . . 33 3.3 Other (room) impulse response spectral models . . . . . . . . . 41 3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 II ACOUSTIC ECHO RETRIEVAL 45 49 4.1 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . 49 4.2 Taxonomy of acoustic echo retrieval methods . . . . . . . . . . 50 4.3 Literature review . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.4 Data and evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.5 Proposed approaches . . . . . . . . . . . . . . . . . . . . . . . . 62 blaster: 65 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 5.2 Signal model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 xvii xviii 5.3 Background on on-grid blind channel estimation . . . . . . . . . 67 5.4 Proposed approach . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 6 lantern: 77 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 6.2 Deep learning models . . . . . . . . . . . . . . . . . . . . . . . . 79 6.3 Learning the rst echo . . . . . . . . . . . . . . . . . . . . . . . 81 6.4 Robustly learning the rst echo . . . . . . . . . . . . . . . . . . 85 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 7 dechorate: 91 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 7.2 Database realization . . . . . . . . . . . . . . . . . . . . . . . . . 92 7.3 Dataset annotation . . . . . . . . . . . . . . . . . . . . . . . . . 94 7.4 The dEchorate package . . . . . . . . . . . . . . . . . . . . . . . 97 III ECHO-AWARE APPLICATION 99 8 103 8.1 Audio Scene Analysis Problems . . . . . . . . . . . . . . . . . 103 8.2 Overview on Multichannel Audio Source Separation . . . . . . 105 8.3 Overview on spatial ltering . . . . . . . . . . . . . . . . . . . 108 8.4 Overview on Sound Source Localization . . . . . . . . . . . . . 110 9 separake: 115 9.1 Literature review in echo-aware Multichannel Audio Source Separation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 9.2 Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 9.3 Multichannel Audio Source Separation by NMF . . . . . . . . 118 9.4 Echo-aware Audio Source Separation . . . . . . . . . . . . . . 121 9.5 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . 121 9.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 10 mirage: 127 10.1 Literature review in echo-aware Sound Source Localization . . 127 10.2 Proposed approach . . . . . . . . . . . . . . . . . . . . . . . . 128 10.3 Background in microphone array SSL . . . . . . . . . . . . . . 129 10.4 Microphone Array Augmentation with Echoes . . . . . . . . . 131 10.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . -aware Spatial ltering . . . . . . . . . . . . . . . . . . . 135 11.2 Room Geometry Estimation . . . . . . . . . . . . . . . . . . . 143 11.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . Back . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 12.2 Looking Ahead . . . . . . . . . . . . . . . . . . . . . . . . . . . (Hermitian) of the vector x Re[x] real part scalar (vector) x (x) Im[x] imaginary part scalar (vector) x (x) i imaginary unit N set of natural numbers R set of real numbers R + set of real positive numbers C set of complex numbers C i microphone or channel index in {0, . . . , I -1} j source index in {0, . . . , J -1} r re ection (echo) index in {0, . . . , R -1} t continuous time index n discrete sample index in {0, . . . , N -1} f continuous frequency index in {0, . . . , T -1} k discrete frequency index in {0, . . . , F -1} l discrete time-frame index in {0, . . . , L -1} τ continuous tap index

  Illustration of the molecules under sound compression and rarefaction due to longitudinal sound wave and its waveform representation.

  A re ector having irregularities on its surface with width d much smaller than the sound wavelength λ. Image courtesy of[START_REF] Kuttru | Room acoustics[END_REF]].

F 2 .

 2 15: 3-D representation of the Image Source Method (ISM) and its propagation paths for selected echoes.

F 2 .

 2 16: From left to right, path involving the direct path, one re ection obtained using rst-order image, and two re ections obtained using two images. Image inspired from[START_REF] Guillemain | Characterization of acoustic signals through continuous linear time-frequency representations[END_REF]].

F 3. 2 :

 2 General mixing process, illustrated in the case of J = 3 sources, including three point sources and one di use source, and I = 2 channels.

  Graphical representation of the mixing model 3.5 for 2 sources and 2 microphones. The text is overlapping on purpose.

F 3. 4 :

 4 Graphical representation of the mixing model (3.5): s 2 (t) is the interferer, n(t) contributes to the di use noise eld, and ε(t) model acquisition and modeling errors. With proper prior knowledge, it can be understood.

  Figure 3.6. Tukuljac et al. made F 3.6: (a) Continuos-time stream of Diracs h(t), (b) sinc kernel φLP(t), (c) smoothed stream of Diracs, (d) original stream of Dirac h(t) (red) and its sampled (i. e., smoothed and discrete) version (blue). Image courtesy of

  lives in the complex space C, the squared magnitude of the STFT, |X[k, l]| 2 is commonly used for visualization and for processing. The resulting two-dimensional representation is called (log)

F 3. 7 :

 7 STFT spectrogram of an example speech signal. Higher energies are illustrated with darker colors. power spectrogram. It can be visualized by means of a two-dimensional image,

F 4. 2 :

 2 Schematics of RIR-based and RIRagnostic approaches.

  4.3 L 4.3.1 Active and RIR-based method T RIR

  Illustration of di erent orders of arrival of wall re ections echoes to di erent image sources or re ectors, as shown in Figure 4.3. Many methods have been proposed in the context of SSL

F 4. 5 :

 5 Visualization of the audio camera:The angular spectrum is overlapped to the corresponding images. Also shown are the impulse responses for 6 microphones. The numbered boxes indicate events shown in the audio camera. Images from[START_REF] Tervo | Localization and tracing of early acoustic re ections[END_REF]].

  F 4.6: Classi cation of the State of the Art in channel estimation.

[

  [START_REF] Crocco | Room impulse response estimation by iterative weighted l 1-norm[END_REF] λ h 1 p (z) h = 1, h ≥ 0 Iterative weighted 1 constraint.T 5.1: Some state-of-the-art penalties and constraints used in Eq. (5.7).

  a ( nite) discrete measure verifying μ ({(0, 1)}) = 0. Denoting y def = -Aδ (0,1) ∈ C N +1 , the relation Eq. (5.18) becomes Aμ = y.(5.20)For the sake of clarity, we use these conventions hereafter and omit the bar over µ. Now, following[START_REF] Castro | Exact reconstruction using Beurling minimal extrapolation[END_REF][START_REF] Candès | Towards a mathematical theory of super-resolution[END_REF], one can expect to recover the desired lter µ by solving µ = arg minM+(Θ)µ TV s.t.

  Illustration of the sliding Frank-Wolfe algorithm proposed in [Denoyelle et al. 2019] in BLASTER.

  versus number of echoes R to be retrieved for broadband (left) and speech (right) signals with RT 60 = 400 ms and SNR = 20 dB. Error bands denotes 95% con dence intervals.

  setup with one sound source recorded by two microphones. The illustration shows the direct sound path (blue lines) and the resulting rst-order echoes (orange lines).

  Architecture of the proposed deep neural network. Input dimensions for each stage are reported. The rst dimension is the batch size |Ξ| = 200 6.4.1 Gaussian-and Student's T-based CNNs

  Normalize root mean squared error (the lower, the better) for TDOA estimation using the proposed architectures for di erent SNR levels on white noise source signals.

F

  Broad-view picture of the acoustic lab at Bar-Ilan university.

  Picture of the acoustic lab. From left to right: the overall setup, one microphone array, the setup with revolved panels. Surface coding in the dataset: each binary digit indicates if the surface is absorbent (0, ) or re ective (1, ).

F 7. 4 :

 4 Picture of the Beacon indoor positioning system used for measuring array and loudspeaker 3D position. (ii) The estimated RIRs are superimposed on synthetic RIRs computed with the Image Source Method (ISM) from the geometry obtained in the previous step. A Python GUI 56 (showed in Figure 7.5), was used to 56 This GUI is available in the dataset package. manually tune a peak nder and label the echoes corresponding to found peaks, that is, annotate their positions and their corresponding image source position and wall label.. (iii) By solving a simple Multi-Dimensional Scaling (MDS) problem [Dokmanić et al. 2015; Crocco and Del Bue 2016; Plinge et al. 2016], re ned microphone and source positions were computed. The non-convexity of the problem was alleviated by using a good initialization (obtained at the previous step), by the high SNR of the measurements and, later, by including the additional image sources in the formulation. The prior information about the arrays' structures reduced the number of variables of the problem, leaving the 3D positions of the sources and of the arrays' barycenters in addition to the the arrays' tilt on the azimuthal plane.

  ηn (l) = |h n (l)|/ max |h n (l)|, where l = 0, . . . , L -1 is the sample index and n is an arbitrary indexing of all the microphones for a xed room con guration. 4 RIR skylines for 4 directional sources for the full re ective scenario are shown in Figure 7.6, stacked horizontally, preserving the order of microphones within the arrays. The reader can notice several clusters of 5 adjacent bins of similar color (intensity) corresponding to the arrivals at the array's sensors. Thanks to the usage of linear arrays, this visualization allowed us to identify both TOAs and their labeling. Measured RIRs (selected room confs) Direct Path Deconvolution (selected room confs) Peak Finder (current room conf) RIR skyline (current array) Detail of the GUI used to manually annotate the RIRs. For a given source and microphone, a) and b) shows 2 RIRs for 2 di erent room walls con guration (blue and orange) before and after the direct path deconvolution respectively. c) shows the results of the peak nder for one of the deconvolved RIRs, and d) is a zoom on the RIR skyline (See Figure 7.6).D / was used to compensate the frequency response of the source loudspeaker and microphone[Antonacci et al.

  RIR Skyline annotated with observed peaks (×) together with their geometrically-expected position (•) computed with the Pyroomacoustic simulator. As speci ed in the legend, di erent colors are used to indicate the room facets responsible for the re ection: direct path (d), ceiling (c), oor (f), west wall (w), . . . , north wall (n).

8. 1

 1 Audio Scene Analysis Problems . . . . . . . . . . . . . . . . . 8.1.1 Common scenario and model . . . . . . . . . . . . . . 8.1.2 Problem formulation . . . . . . . . . . . . . . . . . . . 8.2 Overview on Multichannel Audio Source Separation . . . . . . 8.3 Overview on spatial ltering . . . . . . . . . . . . . . . . . . . 8.3.1 Beamforming . . . . . . . . . . . . . . . . . . . . . . . 8.4 Overview on Sound Source Localization . . . . . . . . . . . . . 8.4.1 Knowledge-based approaches . . . . . . . . . . . . . . 8.4.2 Data-driven approaches . . . . . . . . . . . . . . . . . 9 separake: 9.1 Literature review in echo-aware Multichannel Audio Source Separation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Multichannel Audio Source Separation by NMF . . . . . . . . 9.3.1 NMF using Multiplicative Updates (MU-NMF) . . . . . 9.3.2 NMF using Expectation Maximization (EM-NMF) . . . 9.3.3 Supervised NMF with pre-trained Dictionaries . . . . 9.4 Echo-aware Audio Source Separation . . . . . . . . . . . . . . 9.5 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . 9.5.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.2 Dictionary Training, Test Set . . . . . . . . . . . . . . 9.5.3 Implementation: . . . . . . . . . . . . . . . . . . . . . 9.5.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 mirage: 10.1 Literature review in echo-aware Sound Source Localization . . 10.2 Proposed approach . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Background in microphone array SSL . . . . . . . . . . . . . . 10.3.1 2-channel 1D-SSL . . . . . . . . . . . . . . . . . . . . . 10.3.2 Multichannel 2D-SSL . . . . . . . . . . . . . . . . . . . 10.4 Microphone Array Augmentation with Echoes . . . . . . . . . 10.4.1 Experimental Results . . . . . . . . . . . . . . . . . . . 10.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Illustration of an indoor audio scene recorded by three microphones. The microphone signals capture the reverberant mixture of several sound sources such as speech, music (guitar) and background diffuse noise (an air conditioner).

F 8. 2 :

 2 Illustration of the DS Beamforming in the time-domain applied to a pulse source signal. The delays of the recorded signals at each microphone are compensated. By summing the shifted signals, the source signal is enhanced. Image courtesy of Robin Scheibler, author of[START_REF] Scheibler | Raking echoes in the time domain[END_REF]].

F

9.5. 1

 1 SetupAn array of three microphones arranged on the corners of an equilateral triangle with edge length 0.3 m is placed in the corner of a 3D room with 7 walls. We select 40 sources at random locations at a distance ranging from 2.5 m to 4 m from the microphone array. Pairs of sources are chosen so that they are at least 1 m apart. The oor plan and the locations of microphones are depicted in Figure9.5. The scenario is repeated for every two active sources The simulated scenario. Image taken from the paper of this work[Scheibler et al. 2018c] out of the 780 possible pairs. Value of the regularization parameter γ used with the universal dictionary.

  using the NMF solver in scikit-learn Python package [Pedregosa et al. 2011]. Figure 9.6 illustrates the two di erent dictionaries used in this work. U D : Following the methodology of [Sun and Mysore 2013] we select 25 male and 25 female speakers and use all available training sentences to form the universal dictionary D= [D M 1 • • • D M 25 D F 1 • • • D F 25 ]. The test signals were selected from speakers and utterances outside the training set. The number of latent variables per speaker is 10 so that with STFT frame size of 2048 we have D ∈ R 1025×500 .

  Distribution of SDR and SIR for male and female speakers as a function of the number of echoes included in modeling, and comparison with the three baselines.

  of the median SDR and SIR for the di erent algorithms evaluated.

  this question, we propose to use the class of deep learning models LANTERN, presented in Chapter 6. These models are trained on a close-surface dataset to estimate the rst early echoes per channel from audio features. ? C ? To answer the second question, we propose the Microphone Augmentation with Echoes (MIRAGE) framework. It exploits echo times of arrival by expressing them as TDOAs in a virtual 4-microphone array formed by the true microphone pair and its image with respect to the re ective surface. This model is based on the image microphones model (See § 9.1).

F 10. 2 :

 2 Illustration of the cone of confusion.

i

  being frequency-independent. 10.3.1 2-channel 1D-SSL Let us rst consider the case of stereophonic recordings (I = 2). Under fareld assumption, traditional SSL methods use the Time Di erence of Arrival (TDOA), τ TDOA def

  Illustration of the relation between DOA and TDOA with ones source and two microphones. Knowing the distance d between the two microphones, simple trigonometry yields the AOA ϑ according to Eq. (10.2).

  Illustration of the di erent DOAs at each microphone pairs attending one sound source. Knowing the position of the microphones, the angle with respect to a reference frame can be deduced in closedform.

2

  . for each pair of microphones, a local set of AOAs (hence, TDOAs) is de ned based on the above chosen DOAs and the input geometry; 3. a TDOA-based algorithm (e. g. GCC-PHAT) is used to compute the associated local angular spectrum; 4. all the local contributions (a collection of local Ψ GCC (τ )) are geometrical aggregated and interpolated back to the global DOA grid to form Ψ SRP (θ, φ);

  of the images m1 and m2 of microphones m 1 and m 2 in the presence of a re ective surface and a source. Blue lines correspond to direct paths, orange lines correspond to echo paths.

  4. A free and open-source Matlab implementation of SRP-PHAT 69 is used to aggregate local angular spectra obtained from the 69 MBSSLocate model output. 70 External noises in the recordings are simulated by perturbing 70 For SRP-PHAT we used a sphere sampling with 0.5°resolution and coordinates θ ∈ [-179, 180] and φ ∈ [0, 90] are used for the DOA search.

  Comparison of echo-aware beamforming for the room con guration 011111 (RT 60 ≈ 600 ms) on measured and synthetic data for all combinations of source-array positions in the dEchorate dataset.

  Images source estimation (right) and corresponding re ector estimation (left) for one of the sound sources in the DECHORATE dataset.

  Re( Aδ θ , y -Aµ ) ≤ 1 + ε (12.3)where ε is a positive scalar set to ε = 10 -3 . The new parameter is found by solving arg max θ∈Θ Re( Aδ θ , y -A µ ).

  List of selected audio scene analysis problems. The one above the line are considered in this thesis.

	Problems		From the recordings, can we...
	Audio Source Separation	... estimate the audio signal of sound sources?
	Audio Source Enhancement	... estimate the audio signal of a target sound source?
	Sound Source Localization	... estimate the positions of sound sources?
	Microphone Calibration	... estimate the positions (and gains) of the microphones?
	Room Geometry Estimation	... estimate the shape of the room?
	Acoustic Echo Estimation	... estimate the echoes' properties?
	Acoustic measurement	... estimate physical properties related to sound propagation?
	Source Identi cation	... estimate the type of source signal?
	Speech Diarization	... who is speaking and when ?
	Source Counting	... count the number of speakers?
	Automatic Speech Recognition	... estimated the content of the speech ?
	T	1.1:

  Sound wave propagation . . . . . . . . . . . . . . . . . . . . . . 2.1.1 The acoustic wave equation . . . . . . . . . . . . . . . . 2.1.2 ... and its Green solution . . . . . . . . . . . . . . . . . . 2.2 Acoustic re ections . . . . . . . . . . . . . . . . . . . . . . . . . Room acoustics and room impulse responses . . . . . . . . . . . 2.3.1 The room impulse response . . . . . . . . . . . . . . . . 2.3.2 Simulating room acoustics . . . . . . . . . . . . . . . . . 2.3.3 The method of images and the image source model . . . 2.4 Perception and some acoustic parameters . . . . . . . . . . . . .

	2
	2.1 2.2.1 Large smooth surfaces, absorption and echoes . . . . . .
	2.2.2 Di usion, scattering and di raction of sound . . . . . .
	2.3
	Part I
	ROOM ACOUSTICS MEET SIGNAL PROCESSING

• no indentation, but well-separated text blocks; 2.4.1 The perception of the RIR's elements . . . . . . . . . . . 2.4.2 Mixing time . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Reverberation time . . . . . . . . . . . . . . . . . . . . . 2.4.4 Direct-to-Reverberant ratio and the critical distance . . 2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.1 Signal model in the time domain . . . . . . . . . . . . . . . . . . 3.1.1 The mixing process . . . . . . . . . . . . . . . . . . . . . 3.1.2 Noise, interferer and errors . . . . . . . . . . . . . . . . 3.2 Signal model in the spectral domain . . . . . . . . . . . . . . . . 3.2.1 Discrete time and frequency domains . . . . . . . . . . . 3.2.2 The DFT as approximation of the FT . . . . . . . . . . . 3.2.3 Signal models in the discrete Fourier domain . . . . . . 3.2.4 Time-Frequency domain representation . . . . . . . . . 3.2.5 The nal model . . . . . . . . . . . . . . . . . . . . . . . 3.3 Other (room) impulse response spectral models . . . . . . . . . 3.3.1 Steering vector model . . . . . . . . . . . . . . . . . . . 3.3.2 Relative transfer function and interchannel models . . . 3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Elements of Room Acoustics "Sound, a certain movement of air." -Aristotele, De Anima II.8 420b12

  where • denotes the Euclidean norm. By applying the inverse Fourier trans-

	Eqs. (2.8) and (2.9) are respectively the free-
	eld transfer function and the impulse re-
	sponse.

  The time-domain cross-over point is called transition time, or mixing time. It

					f	f	deterministic model	stochastic model
		Room frequency Response (RFR)	Schroeder's frequency	resonant modes	even modes		Direct sound image source	Early echoes waveguide raytracing Late reverberation	t
								t
						Transition time
							Room impulse Response (RIR)
	F	2.14: Time-Frequency regions of the RIR associated to the method that better simulate them. Image adapted from [Thomas 2017;
	Badeau 2019].

  This corresponds to nite and discrete-time signal x N obtained by periodically sampling the continuous-time signal x at rate F s[Hz], truncate it to N samples. As common to most measurement models, we assume

				Original continuous-time signal
		1
		0
		1
		0.0	0.2	0.4	0.6	0.8	1.0	1.2
				Sampled signals at Fs = 20
	.1)	0 1
		1
	continuous both in time t ∈ R and amplitudes.	0.0	0.2	0.4	0.6 Time (seconds)	0.8	1.0	1.2
		F	3.1: Continuous-time signal and its
	Today signals are typically processed, stored and analyzed by computers as	sampled version.
	digital audio signals. Strickly speaking, the digital representation
		of a continuous signal involves sampling and
		quantization. In this thesis we assume the sam-
		pled signals are real-valued, ignoring the quan-
		tization process.

  4.1 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Taxonomy of acoustic echo retrieval methods . . . . . . . . . . 4.3 Literature review . . . . . . . . . . . . . . . . . . . . . . . . . . Data and evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Proposed approaches . . . . . . . . . . . . . . . . . . . . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 7.2 Database realization . . . . . . . . . . . . . . . . . . . . . . . . . 92 7.2.1 Recording setup . . . . . . . . . . . . . . . . . . . . . . . 92 7.2.2 Measurements . . . . . . . . . . . . . . . . . . . . . . . 93 7.3 Dataset annotation . . . . . . . . . . . . . . . . . . . . . . . . . 94 7.3.1 RIRs annotation . . . . . . . . . . . . . . . . . . . . . . . 94 7.3.2 Other tools for RIRs annotation . . . . . . . . . . . . . . 96 7.3.3 Limitations of current annotation . . . . . . . . . . . . . 97 7.4 The dEchorate package . . . . . . . . . . . . . . . . . . . . . . . 97

	7 dechorate:
	4.3.1 Active and RIR-based method . . . . . . . . . . . . . . . 4.3.2 Active and RIR-agnostic method . . . . . . . . . . . . . 4.3.3 Passive and RIR-based method . . . . . . . . . . . . . . 4.3.4 Passive and RIR-agnostic methods . . . . . . . . . . . . 7.1 "[. . .] dicebat Bernardus Carnotensis nos esse quasi nanos gigantium humeris insidentes." 4.4 5 blaster: -Giovanni of Salisbury, Metalogicon (III, 4)
	6.3.1 Scenario and motivations . . . . . . . . . . . . . . . . .
	6.3.2 Proposed approach . . . . . . . . . . . . . . . . . . . . .
	6.3.3 Data, implementation and experimental results . . . . .
	6.4 Robustly learning the rst echo . . . . . . . . . . . . . . . . . .
	6.4.1 Gaussian-and Student's T-based CNNs . . . . . . . . . .

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Signal model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Background on on-grid blind channel estimation . . . . . . . . . 5.3.1 From cross-relation to LASSO . . . . . . . . . . . . . . . 5.4 Proposed approach . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Echo localization with continuous dictionaries . . . . . 5.4.2 From LASSO to BLASSO . . . . . . . . . . . . . . . . . . 5.4.3 The resulting algorithm . . . . . . . . . . . . . . . . . . 5.4.4 Homotopic path for λ estimation . . . . . . . . . . . . . 5.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6 lantern: 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Deep learning models . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Multi-layer perceptrons . . . . . . . . . . . . . . . . . . 6.2.2 Convolutional neural networks . . . . . . . . . . . . . . 6.2.3 Hybrid architectures . . . . . . . . . . . . . . . . . . . . 6.2.4 Learning and optimization . . . . . . . . . . . . . . . . . 6.3 Learning the rst echo . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 Experimental results . . . . . . . . . . . . . . . . . . . . 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Comparison of some existing RIR databases which account for early acoustic re ections. Receiver positions are indicated in terms of number of microphones per array times number of di erent positions of the array (∼ stands for partially available information.

		Database Name	Pos.	Annotated Echoes	Rooms	RIRs	Number of Rooms Mic×Pos.	Src	Key characteristics	Purpose
		[Dokmanić et al. 2013]		∼	∼	15	3	5	1	Non shoebox room	RooGE
		[Crocco et al. 2017]		∼		204	1	17	12	Accurate 3D calibration Many mic and src pos.	RooGE
		[Remaggi et al. 2016]		∼		∼1.5k	4	48×2	4-24	Circular dense array Circular placement of src.	RooGE SE †
		[Remaggi et al. 2019]		∼		∼1.6k	4	48×2 +2×2	3-24	Circular dense array Binaural Recordings	RooGE † SE
		BUT Reverb [Szöke et al. 2019]			∼	∼1.3k	8	(2-10)×6	3-11	Accurate metadata various rooms di erent device/arrays	SE
		VoiceHome [Bertin et al. 2019]				188	12	8×2	7-9	Various rooms, real homes	SE
		dEchorate (Ch. 7)				∼1.8k	1	30	6	Accurate annotation Di erent Echo-energy	RooGE SE
	T	4.1:

  .2.

	Dataset D (valid)	Signals broadband noise	SNR [dB]	RT60 [s]	T evaluation. 5.2: Summary of the dataset used for and stands for randomly sampled from a continuous and discrete set
	D SNR	broadband noise, speech		400 ms	of values, respectively, with uniform law.
	D RT 60	broadband noise, speech	20 dB

  ] (See Chapter 9). Mean error (left) and precision (right) versus SNR level (top) and RT 60 level (bottom) using broadband and speech signals for the task of recovering R = 7 echoes. A threshold of τmax = 2 samples is used to compute the precision. Error bands denotes 95% con dence intervals.

				Broadband signal				Speech signal		1.0	Broadband signal	Speech signal
	Error [samples]	0.5 1.0									Precision [%]	0.2 0.4 0.6 0.8
			0	5 SNR [dB] using D SNR 10 15	20	0	5 SNR [dB] using D SNR 10 15	20	0	5 SNR [dB] using D SNR 10 15	20	0	5 SNR [dB] using D SNR 10 15	20
												1.0
	Error [samples]	0.5 1.0									Precision [%]	0.2 0.4 0.6 0.8	IL1C BLASTER BSN
			0.2	0.4 RT 60 using D RT60 0.6 0.8	1.0	0.2		0.4 RT 60 using D RT60 0.6 0.8	1.0	0.2	0.4 RT 60 using D RT60 0.6 0.8	1.0	0.2	0.4 RT 60 using D RT60 0.6 0.8	1.0
	F τmax BSN IL1C	5.4: Precision [%] R = 2 echoes 0.5 1 2 3 10 0.5 8 9 27 46 62 5 51 55 55 56 58 42	R = 7 echoes 2 38 53 1 8 55	3 54 56	10 73 58	T τmax in samples for the recovery of R = 2 5.3: Precision for di erent threshold and 7 echoes, RT 60 = 200 ms and SNR = 20 dB.
	BLASTER	68	73	74		75	75	46	53	56	57	61

Table 6

 6

				nRMSE
		Input	TDOA	iTDOA	TDOE
	MIRAGE-MPL	wn	0.18	0.28	0.25
	MIRAGE-MPL MIRAGE-MPL MIRAGE-MPL	wn+n sp sp+n	0.68 0.31 0.99	0.69 0.34 0.98	0.89 0.56 1.48	T Errors (nRMSEs) for TDOA estimation using 6.1: normalized Root Mean Squared the MLP architecture.
	GCC-PHAT	wn	0.21	-	-
	GCC-PHAT	wn+n	0.68	-	-
	GCC-PHAT	sp	0.32	-	-
	GCC-PHAT	sp+n	1.38	-	-

.1. From these results, one can see that training a MLP to estimate TDOAs brings similar performances as GCC-PHAT in terms of

  Rather than training on noiseless observation, we consider microphone recordings featuring background noise. To this end, we perturb the observed microphone signals with Additive White Gaussian Noise (AWGN), leading to SNR values uniformly distributed in [0, 30] dB. Meanwhile, the test set includes observations featuring SNR levels of 0, 10, or 20 dB. The protocol to generate the observations and the feature extraction step are the same ( § 6.3.3), but the investigation is restricted to broadband source signals (white noise) only. Informal test on speech data yielded to 3.3 times larger errors respect to noise signals and close to random. One of the main reason is due to the missing frequencies in speech spectrum. Such spectral holes lead values close to zero at both the numerator and denominator of the observation used to calculate ILD and IPD (see § 3.3.2). Alternative technique are currently investigates to overcome this limitation, e. g., robust

  Audio scene analysis problems from the mixtures {xi} i , can we estimate... List of audio scene analysis problems considered in this thesis accompanied by their mathematical description.

				Chapter
	Audio Source Separation	the source signals {sj} j , or the spatial images {cij} ij ?	Chapter 9
	Spatial ltering	the source signals {sj} j , knowing the lters hij(x i |s j ) ij ?	Chapter 11 § 11.1
	Sound Source Localization	the source positions s j j ?	Chapter 10
	Room Geometry Estimation	the shape of the room?	Chapter 11 § 11.2
	T	8.1:

  Slidescal Multichannel Audio Source Separation (MASS) methods. Existing methods typically either ignore the acoustic propagation or attempt to estimate it fully. Instead, this work investigates whether sound separation can bene t from the knowledge of early acoustic echoes. These echo properties are derived from the known locations of a few image microphones. The improvements are shown for two variants of a method based on non-negative matrix factorization based on the work of

	S	In this chapter, echoes are used for improving performance of classi-	"Please could you stop the noise, I'm trying to get some rest
		From all the unborn chicken voices in my
			head."
			-Radiohead, Paranoid Android
		Keywords: Blind Channel Identi cation, Su-
		per Resolution, Sparsity, Acoustic Impulse
		Response.
		Resources:
		• Paper
		• Code
		•

  The frequency-dependent directivity pattern for a virtual array built with a real and one (left) and two (right) image microphones. Image taken from the presentation of the paper[Scheibler et al. 2018c].

		1000 Hz	2000 Hz	1000 Hz	2000 Hz
		3000 Hz	4000 Hz	3000 Hz	4000 Hz
		Virtual Dipoles	Virtual Tripoles
	F	9.2:

  depicts an example of re ector estimation using the dEchorate data.

	source id	1		2		3		4
	wall	DE	AE	DE	AE	DE	AE	DE	AE
	west	0.74	8.99°4.59	8.32°5.89	5.75°0.05 2.40°e
	ast	0.81	0.08°0.9	0.50°69 .51 55 .70 •	0.31	0.21°s
	outh	3.94	16 .08 • 0.18 1.77°14 .37 18 .55 •	0.82	1.65°n
	orth	1.34	0.76°1.40	8.94°0.63	0.17°2.08	1.38°
	oor	5.19	1.76°7.27	2.66°7.11	2.02°5.22	1.90°c
	eiling	1.16	0.28°0.67	0.76°0.24	1.16°0.48 0.26°T

Iteration index // 1. Add new element to the support 7

  Find θ new ∈ arg max θ∈Θ Re Aδ θ , y -Aµ (t-1) ; Re Aδ θ new , y -Aµ (t-1) ;

	-1 y	2 2 ;
	// Starting algorithm
	5 repeat
	c∈R R (t) +	1 2	y -	θ∈E (t-½)	c θ Aδ θ	2 2 + λ c 1
	approximated using a proximal gradient algorithm ;
	arg min θ∈Θ R (t) ,c∈[0,xmax] R (t)	2 1	y -	R (t)

6 t ← t + 1 // 8 η (t) ← λ -1 9 if η (t) ≤ 1 + ε then Stop and return µ = µ (t-1) is a solution ; end E (t-½) ← E (t-½) ∪ {θ new } ; R (t) ← card(E (t-½) ) //

Number of detected echoes // 2. Lasso update of the coefficients c (t-½) ← arg min // 3. Joint update for a given number of spikes E (t) , c (t) ← r=1 c r Aδ θr 2 2 + λ c 1 approximated using a non-convex solver initialized with (E (t-½) , c (t-½) ) ; // 4.

Eventually remove zero amplitude Dirac masses

  Re( Aδ θ , y -Aµ ) ≤ 1.(12.2)

	negative) scalars, (12.1) can be rewritten as
		sup	λ -1
		θ∈Θ
	E (t) ← θ (t) r ∈ E (t) | c (t) r = 0 ; c (t) ← c (t) (t) r = 0 ; r | c card(E (t) )
	µ (t) ←	r=1	c (t) r δ θ (t) r	;
	until until convergence;

  An incomplete selection of the most played music that fueled this Ph.D. work (according to Deezer).DiCarlo, Diego, Pinchas Tandeitnik, Sharon Gannot, Antoine Deleforge, and Nancy Bertin (2021). "dEchorate: a calibrated Room Impulse Response database for acoustic echo retrieval". In: Workin progess (cit. on p. 9). van denBoomgaard, Rein and Rik van derWeij(2001). "Gaussian convolutions numerical approximations based on interpolation". In: Scale-Space and Morphology in Computer Vision. Springer, pp. 205-214 (cit. on p. 37). Traitement du signal avec des echos acoustiques pour l'analyse des scènes audio Mot clés : Traitement des signaux audio, échos acoustiques, estimation des canaux aveugles, séparation de sources sonores, localisation de sources sonores, estimation de la géométrie d'une salle Résumé : La plupart des méthodes de traitement du signal audio considèrent la réverbération et en particulier les échos acoustiques comme une nuisance. Cependant, ceux-ci transmettent des informations spatiales et sémantiques importantes sur les sources sonores et des méthodes essayant de les prendre en compte ont donc récemment émergé.. Dans ce travail, nous nous concentrons sur deux directions. Tout d'abord, nous étudions la manière d'estimer les échos acoustiques à l'aveugle à partir d'enregistrements microphoniques. Deux approches sont proposées, l'une s'appuyant sur le cadre des dictionnaires continus, l'autre sur des techniques récentes d'apprentissage profond. Ensuite, nous nous concentrons sur l'extension de méthodes existantes d'analyse de scènes audio à leurs formes sensibles à l'écho. Le cadre NMF multicanal pour la séparation de sources audio, la méthode de localisation SRP-PHAT et le formateur de voies MVDR pour l'amélioration de la parole sont tous étendus pour prendre en compte les échos. Ces applications montrent comment un simple modèle d'écho peut conduire à une amélioration des performances.

	First Ph.D. Year	Second Ph.D. Year	Third Ph.D. Year
	Titre :	Gojira -L'Enfant Savage	Beyond Creation -The Aura	Author & Punisher -Beastland
				Polo & Pan
				-Caravelle
		Xploding Plastix -Amateur Girlfriends Go Proskirt Agents	Venetian Snares -Traditional Synthesizer Music	The Claypool Lennon Delirium -South of Reality
				Dark Funeral
			Plaid	-Diabolis Interium
		Clark -Totems Flare	-The Digging Remedy	Zeal & Ardor -Devil Is Fine
			Hannes Grossmann	Ruby My Dear
			-The Crypts of Sleep	-Stranger in Paradise
		Aufgang -Aufgang	Thylacine	Tool -Fear Inoculum
			-Transsiberian	Giobia
				-The Magni er
		The Knife -Silent Shout	Baroness -Purple	Edoardo Maggiolo -Mappe.
				Coldplay
		General Elektriks -To Be a Stranger	Princess Nokia -A Girl Cried Red	-X&Y Childish Gambino -"Awaken, My Love!"
		Twenty One Pilot	Ruby My Dear	Thesis redaction
		-Blurryface	-Brame	Death
				-Individual Thought Patterns
		There's A Light -A Long Lost Silence	Devin Townsend -Empath	The Faceless -In Becoming a Ghost
				Foehn Trio
			Tatran	-Highlines
		Corpo-Mente -Corpo-Mente	-Shvat	Mandela -The Sound of Grass
			Sunn O))) -White2	Moderat -Modelar
		Igorrr -Sinusoid Savage		Rings of Saturn -Ultu Ulla
			The Gaslamp Killer -Instrumentalepathy	Obscura -Cosmogenesis
		John Frusciante -A Sphere In The Heart Of Silence	Rïcïnn	Mort Garson -Mother Earth's Plantasia
			-Lïan	Caroline Lavelle
				-Spirit
		Sister Iodine -Flame Desastre	Öxxö XööX -Nämïdäë	Wintergatan -Wintergatan
				Three Trapped Tigers
				-Numbers: 1-13
		Metronomy -The English Riviera	Rosalía -El Mal Querer	Caroline Lavelle -Spirit

Title: Echo-aware signal processing for audio scene analysis Keywords: Audio Signal Processing, Acoustic Echoes, Blind Channel Estimation, Sound Source Separation, Sound Source Localization, Room Geometry Estimation

For his experiments, Galileo Galilei was measuring time using the sound of a metronome.

As opposed to mechanical vibrations in a string or (drum) membrane, acoustic vibrations are longitudinal rather than transversal, i. e. the air particles are displaced in the same direction of the wave propagation.

Impulse responses in time domain, transfer functions in the frequency domain.physic so-far can be rewritten using the vocabulary of the communication theory, namely input, lter and output.

This can be intrepreded as the projection onto the space spanned by a nite number of complex exponentials.

When short excerpt are considered instead (e. g. in case of the Short Time Fourier Transform (STFT)), particular types of window function are used but their analysis are out of the scope of this thesis.

Even if this is a classic result and it is discussed is classical book, however the provided reference concisely discusses it.

The interested reader can refer to[START_REF] Girin | Audio source separation into the wild[END_REF]] for a review on the topic applied to audio source separation

Cross-correlation and convolution operations are very similar, an mathematically they di er just by the inversion of the reference signal. While, the former measures the similarity between two signals as function of a translation, the latter measure the e ect of one signal on the other signal. See § 4.3.2 for further explanation.

A mathematical operator that maps points into curves in a 2-D space. If a set of points belongs to the same line, the corresponding curves will intersect in a single point. In computer vision, this tranform is typically used as feature extractor to detect lines and edges in pictures.

The reader can refer to the review papers[START_REF] Gannot | A consolidated perspective on multimicrophone speech enhancement and source separation[END_REF][START_REF] Girin | Audio source separation into the wild[END_REF] for more details.)

Note that, as opposed to channel estimation, estimating the ReTFs is a non-blind problem (See § 3.3)

Acknowledgements

"There is so much drama in the PhD". Excuse me, Snoop for crippling your words, but I could not nd anything better. . . or maybe "one does not simply defend a phd thesis" (Excuse me, Boromir). This thesis work is just the emerged part of an iceberg, oating of an ocean of people (sorry for the cheesy metaphor). Without their technical and emotional support, this thesis would probably not exist. Fortunately, I was followed and accompanied by a committee of people, each entering the stage just at the right time.

Echo-aware Re ective Re ection "Some say the end is near. Some say we'll see Armageddon soon.

Certainly hope we will. I sure could use a vacation from this [...]" -Tool, AEnema I , we studied acoustic echoes for audio scene analysis and signal processing. The two main lines of work can be brie y summarized as follows:

A. We investigated new methodologies for acoustic echo retrieval (AER) in the case of passive stereophonic recordings.

B. We revisited some fundamental audio scene analysis problems and methodologies under an echo-aware perspective.

12.1 L B

After reviewing some useful acoustic notions and presenting signal precessing modeling in Part I, the contributions of this thesis were presented in Parts II and III, developing the two directions above. The pursuit of these two goals took the form of the following methods and outcomes.

• A K M AER dubbed BLASTER. This approach enables direct and o -grid estimation of the echoes' properties in stereophonic passive recordings. In particular, the "knowledge" we used is the echo model for the early part of the RIRs. Due to its o -grid nature, the method overcomes some theoretical limitation of on-grid approaches. Although it is currently not outperforming the state-of-theart when retrieving more than 2 echoes per channel, this investigation is motivated by theoretical guarantees.

• A D M AER based on deep learning, dubbed LANTERN. Thanks to the availability of powerful acoustic simulators, the properties of the rst echoes are estimated using state-of-the-art architectures which are trained in a virtually-supervised fashion. The proposed model combines recent deep learning methodologies to reduce estimation error with a loss functions able to measure uncertainty. Having a measure for con dence allows to aggregate multiple observations of the same scenario in a data-fusion-like approach.

• An E D designed for both AER and echo-aware applications, dubbed DECHORATE. These annotated data should ll a gap between existing datasets and it is designed for validating future echo-aware research. The dataset is accompanied by software utilities to easily access, manipulate, and visualize the data and baseline methods 151

Sliding Frank-Wolfe algorithm & Non-negative BLASSO

S

This appendix brie y describes the Sliding Franck-Wolfe Algorithm

Resources:

• Code

• Open-access paper with supplemen- tary material used to solve the Non-negative BLASSO presented is Chapter 5. The material reported here are extracted from the supplementary material accompanying the work in [Di Carlo et al. 2020]. The main author of this work is Clement Elvira, to whom I extend my greatest thanks, and I will report it for sake of completeness.

Among all the methods that address the resolution of (5.22-P λ TV ), a signi cant number of them are based on variations of the well-known Frank-Wolfe iterative algorithm, see, e.g., [START_REF] Bredies | Sparsity of solutions for variational inverse problems with nite-dimensional data[END_REF][START_REF] Denoyelle | The sliding Frank-Wolfe algorithm and its application to super-resolution microscopy[END_REF]. In this paper, we particularize the sliding Frank-Wolfe (SFW) algorithm proposed in [START_REF] Denoyelle | The sliding Frank-Wolfe algorithm and its application to super-resolution microscopy[END_REF]].

Starting from an initial guess (e. g., the null measure), SFW repeats the four following steps until convergence:

1. add a parameter (position of echo) to the support of the solution, 2. update all the coe cients solving a ( nite dimensional) Lasso, 3. update jointly the position of the echoes and the coe cients, 4. eventually remove echoes associated to coe cients equal to zero.

Finally, SFW stops as soon as an iterate satis es the rst order optimality condition associated to the convex problem (5.22-P λ TV ). More particularly, denoting µ (t) the estimated lters at iteration t, SFW stops as soon as µ (t) satis es [Bredies and Carioni 2020, Proposition 3.6]

(12.1)

The complete SFW method for echo estimation is described by Algorithm 1. We now provide additional details about the implementation of each step.

N B To take into account the non-negative constraint on the coe cients, the authors of [START_REF] Denoyelle | The sliding Frank-Wolfe algorithm and its application to super-resolution microscopy[END_REF]] have proposed to slightly modify the SFW algorithm by i) removing the absolute value in (12.1) and ii) adding the nonnegativity constraints at step 2 and 3 (see lines 14 and 15 of Algorithm 1). The reader is referred to [Denoyelle et al. 2019, remark 8 in Section 4.1] for more details.

R

(12.1). We have shown earlier that SFW stops as soon as an iterate µ (t) satis es (12.1) at some iteration t. Since the estimated coe cients c