Keywords: Music source separation, Auditory attention decoding, Electroencephalography, Multimodal processing; Matrix factorisation, Deep learning, One-shot Domain Adaptation Séparation des sources musicales, Décodage de l'attention auditive, Electroencéphalographie, Traitement multimodal, Factorisation matricielle, Apprentissage profond Auditory attention, Polyphonic music, EEG. Resources Auditory attention decoding, Polyphonic music, EEG, Stimulus reconstruction model. Resources:

The body text is set to 10/14pt (long primer) on a 26pc measure. The margin text is set to 8/9pt (brevier) on a 12pc measure.

qui sont mélangés dans un enregistrement de musique. Cette tâche est particulièrement complexe, car même les modèles les plus performants restent peu e caces sur des données nouvelles ou très di érentes de des données utilisées pour l'apprentissage. Néanmoins, des informations supplémentaires sur les sources individuelles peuvent être utilisées pour mieux adapter un modèle de séparation de sources générique au signal observé. Des exemples de telles informations sont : la partition de la musique, les paroles des chansons, les vidéos de peroformance musicale ou le feedback de l'utilisateur. Au-delà de ces métadonnées et annotations manuelles, la réaction de notre corps aux stimuli auditifs se manifeste par de nombreux phénomènes physiologiques observables (par exemple, la variabilité du rythme cardiaque, les mouvements du corps, l'activité neuronale). Parmi ceux-ci, nous nous sommes concentrés sur la réponse neuronale et le concept d'attention auditive sélective, qui permet aux humains de traiter des sons simultanés et d'isoler ceux qui les intéressent. Le codage neuronal de la source à laquelle on porte son attention semble être sensiblement plus fort que celui des autres sources, ce qui permet de décoder la source sonore sur laquelle une personne se " concentre ". Cette tâche est connue sous le nom de décodage de l'attention auditive (AAD) et a été étudiée principalement pour ce qui concerne la perception des sources vocales dans des environnements bruyants ou à plusieurs voix.

Dans cette thèse, nous avons investigué comment l'activité neuronale re ète des informations sur l'instrument de musique auquel l'auditeur porte son attention et comment nous pouvons l'utiliser pour informer un système de séparation de sources et l'adapter au stimulus correspondant. Nous nous sommes concentrés sur les signaux électroencéphalographiques (EEG), qui permettent une acquisition non invasive de l'activité neuronale avec une haute résolution temporelle. Tout d'abord, nous avons étudié le problème du décodage par l'EEG de l'attention auditive d'un instrument spéci que dans une pièce musicale polyphonique, en montrant que l'EEG suit les caractéristiques musicales pertinentes qui sont fortement corrélées avec la représentation temps-fréquence de la source à laquelle on porte son attention et seulement faiblement corrélées avec les autres. Ensuite, nous avons exploité ce "contraste" pour informer un modèle de séparation de sources non supervisé basé sur une nouvelle variante de factorisation en matrices positives (NMF : non-negative matrix factorization), appelée contrastive-NMF (C-NMF) et séparer automatiquement la source à laquelle on porte son attention. Nous avons e ectué une évaluation approfondie du système proposé sur le jeu de données MAD-EEG qui a été spéci quement collecté pour cette étude. Nous avons analysé l'impact de multiples aspects des stimuli musicaux, tels que le nombre et le type d'instruments dans le mélange, le rendu spatial et le genre musical, obtenant des résultats encourageants, en particulier dans les cas di ciles où les modèles non informés sont défaillants. vii viii La NMF non supervisée représente une approche e cace dans de telles applications ne disposant pas ou peu de données d'apprentissage, comme c'est le cas dans des scénarios nécessitant des enregistrements EEG. En e et, les jeux de données EEG liés à la musique disponibles sont encore coûteux et longs à acquérir, ce qui exclut la possibilité d'aborder le problème par des approches d'apprentissage profond entièrement supervisées. Ainsi, dans la dernière partie de la thèse, nous avons exploré des stratégies d'apprentissage alternatives. Plus précisément, nous avons étudié la possibilité d'informer un modèle de séparation de sources basé sur l'apprentissage profond en utilisant les activations temporelles de sources fournies manuellement par l'utilisateur ou dérivées de sa réponse EEG disponible au moment du test. Cette approche peut être considérée comme étant " à adaptation unitaire" (one-shot), car l'adaptation agit uniquement sur une instance de chanson. Bien que préliminaires, les résultats obtenus sont encourageants et indiquent des directions de recherche prometteuses.

• Isabelle, Alexandre, Shihab, Alain, Elaine, Blair. It was an honour and challenge to have you on my thesis committee, and I would like to thank you for the time you dedicated to reading these pages.

• Colleagues from Télécom Paris and the ADASP group. You made me feel welcome, and I consider you more as friends than colleagues.

• Ondřej, Kilian, Karim and Javier for the memorable time travelling around the world and living this beautiful experience together.

• Marie Skłodowska-Curie Actions and the MIP-Frontiers network for making my PhD possible. 1 Special thanks to Alvaro, who supported us 1 This work was supported by the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 765 068 (MIP-Frontiers).

students technically and emotionally as only a sensitive person can.

• Carmen, Laurence, Delphine, Isabelle and Françoise for supporting me with the scary French language and administrative duties with kindness.

• My friends in France, Italy and around the world. Living far away from home has been hard, but thanks to your friendship, kindness, and help, I never felt lonely.

• Cecilia, Camille and Basile, for always making me feel at home and part of their wonderful family.

• My family & animals for their love and support, even if I risked to lose my manuscript because of Mina walking on my Glenny.

• My love Diego, who takes care of me with great courage and lightheartedness even in extreme situations such as the thesis writing.

• Giovanni De Poli, for making me and Diego curious about music and computer science and secretly being the start of all this story. It's been almost 10 years, and you still follow us with great a ection.

xi xii Last but not least, I would like to give credit to all the open-source projects for scienti c computing that made possible or, at least, greatly facilitated the work behind this thesis. Python libraries for scienti c computing, data analysis and visualisations that I would like to mention are: NumPy [START_REF] Harris | Array programming with NumPy[END_REF], SciPy [START_REF] Virtanen | SciPy 1.0: fundamental algorithms for scienti c computing in Python[END_REF]], Scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF], PyTorch [START_REF] Paszke | PyTorch: An Imperative Style, High-Performance Deep Learning Library[END_REF]], PyTorch-Lightening [START_REF] Falcon | Pytorch lightning[END_REF], Matplotlib [START_REF] Hunter | Matplotlib: A 2D graphics environment[END_REF]], Seaborn [START_REF] Waskom | seaborn: statistical data visualization[END_REF]], Pandas [START_REF] Mckinney | Data Structures for Statistical Computing in Python[END_REF]] and statsannot [START_REF] Weber | statannot[END_REF]]. A special mention for open-source projects speci c for audio and EEG processing which were fundamental for carrying out this thesis:

Librosa [START_REF] Mcfee | librosa: Audio and music signal analysis in python[END_REF], museval [START_REF] Stöter | The 2018 signal separation evaluation campaign[END_REF], Asteroid [START_REF] Pariente | Asteroid: the PyTorch-based audio source separation toolkit for researchers[END_REF]], MNE-Python [START_REF] Gramfort | MEG and EEG Data Analysis with MNE-Python[END_REF], Braindecode [START_REF] Schaefer | Deep learning with convolutional neural networks for EEG decoding and visualization[END_REF]] and pymtrf [START_REF] Steinkamp | pymtrf[END_REF]]. Over the past decades, the availability of services and tools for music creation, recording, production and distribution has increased exponentially. These have become accessible to a broader public thanks to many factors, such as the increasing accessibility of music technologies and the connectivity to the Internet, profoundly changing the music landscape and culture where home music production is no more an exception.

F 1.1: U U : user1's skills that are used to get user2's experience allowing user1 to make a model for user2's interface. Image courtesy of xkcd, number 2141.

These changes have undoubtedly a ected the music demography and culture [START_REF] Walzer | Independent music production: how individuality, technology and creative entrepreneurship in uence contemporary music industry practices[END_REF]], leading to a more democratic music landscape where more musicians can create and share their music with people from all around the world without intermediaries. Nevertheless, while these services and tools have become a ordable and reliable, the technical skills and expertise required for using them may still represent an entry barrier for most users. Even professionals and experts are slowed down in their work ow by the complexity of some interfaces, which are not exible and often limit the user creativity by sets of hardly-interpretable functionalities and parameters. There is an unavoidable learning curve that the user has to face to learn and adapt to an interface that is not a neutral intermediary between the user and the desired output. These intrinsic limitations can be overcome only by radically rethinking the way we interact with machines and by fully considering the user from the very beginning in the design of music technologies whose interfaces should be e ortless and friction-free.

In parallel, the incredible growth of Human-Computer Interfaces (HCIs) led to a new way to interact with technology so that the interfaces are maximally simpli ed and adaptive to the user. Among those, Brain-Computer Interfaces (BCIs) are paving the way for a direct communication between humans and their devices by directly decoding the user's brain activity [START_REF] Wolpaw | A Comparison of Temporal Response Function Estimation Methods for Auditory Attention Decoding[END_REF]]. The applications of BCIs are countless and span nowadays from clinical to home entertainment such as neurogaming and VR/AR [START_REF] Kawala-Sterniuk | Summary of over Fifty Years with Brain-Computer Interfaces-A Review[END_REF]. In Music Information Research (MIR), the research eld behind many music technologies, BCIs are still far from complementing the classical interfaces and being considered an integral part of the various applications except for music making and performance, which represent an exciting avant-garde mostly of musicians. 1 BCIs could help narrowing the intention 1 This avant-garde dates from 1965 Alvin Lucier's piece Music for Solo Performer and has evolved during the past decades with the more disparate interpretation of BCI for music making and performance. The reader can refer to [START_REF] Williams | BCI for music making: then, now, and next[END_REF]] by Williams and Miranda for a nice review of brain-computer music interfaces. gap which is a common experience of the user when dealing with complex concepts. For instance, in a music recommendation system, the mental image of the desired music is often much clearer than the queries that the user needs to type in the interface to retrieve it. In music production, an audio e ect can be described better by our mental idea of it than by the tuning of a set of hardly-interpretable parameters. BCIs may also signi cantly speed up and help the work ow of professional sound engineers and musicians, potentially uncover new understanding about the underlying creative process, or even discover new techniques for approaching it. In other cases, instead, the mental guidance can just replace the classical interfaces such as a mouse or a keyboard. However, the BCI, or, more generally, the HCI, is not only bene cial for the user but also for the underlying MIR algorithms that can leverage signi cant human expertise and knowledge to improve their performances.

In this thesis work, we make a rst attempt of addressing the challenge of integrating BCI and music technologies on the speci c MIR application of Music Source Separation (MSS), which is the task of isolating individual sound sources that are mixed in the audio recording of a musical piece (see Figure 1.2).

A MSS system can be either directly exploited by the end-user (e.g., a musician or a sound engineer) or be an intermediate step that signi cantly helps other downstream tasks such as automatic music transcription, instrument classi cation, score following, lyrics alignment, and many more others. This problem has been investigated for decades in the MIR community, but never considering BCI as a possible way to guide and inform MSS systems. This type of guidance can give the user an improved listening experience and boost many MIR downstream tasks making them interactive. The potential applications could target both the general audience and expert users such as sound engineers, video designers, and musicians. 2 Speci cally, we explored how to 2 People could thus enhance the instrument of interest during a concert by only "focusing" on it. Musicians could better study during live performances: imagine a student attending a concert who can enhance di erent instruments by switching their attention. Sound engineers could improve their work ow through intelligent neurosteered headphones while remastering songs or soundtracks or video designers editing a video/movie. perform a multimodal MSS exploiting previously not considered modalities, for instance the user's selective auditory attention to a source characterized in terms of his/her neural activity. Among the signals that can characterize the brain response, we consider the Electroencephalography, which is privileged when monitoring the brain activity for BCI because it allows for non-invasive acquisition with high temporal resolution and a reasonable cost.

The rest of the Chapter will introduce the reader to both music source separation and selective auditory attention. Finally, our contributions will be listed and the structure of the thesis will be exposed.

1.2 B M S S aims to isolate individual sources, such as singing voice, guitar, drums, cello, etc., mixed in an audio recording of a musical piece. More precisely, such individual voices can be referred to as stems, i.e., recordings of individual instruments that are arranged together and mastered into the nal audio mix.

Considering the case of single-channel recordings, one can assume that the mixture signal x(t) at sample t is a linear mixture 3 of J sources s j (t) such as: 3 Usually, the nal audio recording is not a linear sum of its stems due to the mixing and mastering steps, which includes multiple non-linear transformations and audio e ects. This signal model still holds if we consider that the non-linear e ects are applied to the individual stems, which are later summed to obtain the mixture.

x(t) = J j=1 s j (t).

(1.1)

Given only x(t), the goal of a general Audio Source Separation (AuSS) system is to recover one or more sources s j (t), where j ∈ {1, ..., J}.

Nowadays, most state-of-the-art MSS systems are based on supervised Deep Learning (DL) systems [START_REF] Stoller | Wave-u-net: A multi-scale neural network for end-to-end audio source separation[END_REF][START_REF] Défossez | Music source separation in the waveform domain[END_REF][START_REF] Stöter | Open-unmix-a reference implementation for music source separation[END_REF][START_REF] Hennequin | Spleeter: a fast and e cient music source separation tool with pre-trained models[END_REF], where an extensive collection of mixtures and corresponding isolated sources are needed during a training phase. Despite the release of dedicated datasets for this task [START_REF] Ra | The MUSDB18 corpus for music separation[END_REF][START_REF] Bittner | Medleydb: A multitrack dataset for annotation-intensive mir research[END_REF], it is still hard for those models to generalize to unseen test data with signi cant timbral variation compared to training, and high-quality MSS remains an open problem for most instruments and music genres.

To mitigate this issue, one can inform the separation process with any prior knowledge one may have about the sources and the mixing process along with the audio signal. In this case, the approach is referred to as informed MSS and was often shown to enhance the separation result, especially for complex music mixtures, if compared to purely data-driven methods [START_REF] Liutkus | An overview of informed audio source separation[END_REF]]. When the additional information comes from another modality than the audio itself, one can refer to it as multimodal MSS and this is the case depicted in Figure 1.3. Examples of such side information include the score [START_REF] Ewert | Structured dropout for weak label and multi-instance learning and its application to score-informed source separation[END_REF]Ewert and Sandler 2017], the pitch contour [START_REF] Virtanen | Combining pitch-based inference and non-negative spectrogram factorization in separating vocals from polyphonic music[END_REF], the lyrics [START_REF] Schulze-Forster | Weakly informed audio source separation[END_REF], the motion of the sound sources and spatial cues [START_REF] Parekh | Guiding audio source separation by video object information[END_REF]].

Source Separation

Music mixture Isolated instrument

Side information F 1.3: Informed source enhancement process. The aim is to separate one target source from the mixture exploiting any prior information we may have about the source.

One of the most underrated and powerful additional modalities is the user feedback which may leverage signi cant human expertise.

Particularly proli c was the use of time annotations provided by the user to learn an AuSS systems based on Nonnegative Matrix Factorization (NMF) or Nonnegative Tensor Factorization (NTF) [START_REF] Bui | NMF/NTF-based methods applied for user-guided audio source separation: An overview[END_REF][START_REF] Laurberg | Structured non-negative matrix factorization with sparsity patterns[END_REF][START_REF] Ozerov | Multichannel nonnegative tensor factorization with structured constraints for user-guided audio source separation[END_REF]Duong et al. 2014a]. Some of them rely on dedicated graphical user interfaces, while others are interactive, where the user can iteratively improve and correct the separation [START_REF] Bryan | An e cient posterior regularized latent variable model for interactive sound source separation[END_REF]Duong et al. 2014b]. Time annotations were also extended to more general Time-Frequency (TF) annotations [START_REF] Lefevre | Semi-supervised NMF with time-frequency annotations for single-channel source separation[END_REF][START_REF] Lefèvre | A convex formulation for informed source separation in the single channel setting[END_REF][START_REF] Jeong | Informed source separation from monaural music with limited binary time-frequency annotation[END_REF][START_REF] Ra | A simple user interface system for recovering patterns repeating in time and frequency in mixtures of sounds[END_REF]] but those require much more expertise and e ort from the user (and a more complicated user interface).

In DL-based systems, time activations have already been used in multi-task learning paradigms where the AuSS and the instrument activity detection tasks are jointly optimized [Stoller et al. 2018a;[START_REF] Hung | Multitask learning for instrument activation aware music source separation[END_REF]. Often, the time activations are relaxed to weak class labels, indicating a given instrument in a speci c time interval, and are used as an input conditioning for the separation system [START_REF] Swaminathan | Improving singing voice separation using attributeaware deep network[END_REF][START_REF] Slizovskaia | End-to-end sound source separation conditioned on instrument labels[END_REF][START_REF] Seetharaman | Class-conditional embeddings for music source separation[END_REF][START_REF] Karamatlı | Audio source separation using variational autoencoders and weak class supervision[END_REF].

There are also some interesting works where the user can hum [START_REF] Smaragdis | Separation by "humming": User-guided sound extraction from monophonic mixtures[END_REF], sing or play [START_REF] Fitzgerald | User assisted separation using tensor factorisations[END_REF]] the source he/she wants to enhance as an example to the separation system. In the work from El Badawy et al., the user may listen to an audio mixture and type some keywords (e.g., "dog barking", "wind") describing the sound sources to be separated [START_REF] Badawy | On-the-y audio source separation[END_REF]]. These keywords are then used as text queries to search for audio examples from the internet to guide the separation process. The user can also provide the fundamental frequency or manually correct it [START_REF] Durrieu | Musical audio source separation based on user-selected F0 track[END_REF][START_REF] Müller | Interactive deep singingvoice separation based on human-in-the-loop adaptation[END_REF] or associate each instrument to a microphone in a multi-channel recording [START_REF] Carlo | Gaussian framework for interference reduction in live recordings[END_REF].

Beyond manual annotations, our body's reaction to auditory stimuli manifests itself through many observable physiological phenomena. Reaction to music can be seen in the heartbeat variability [START_REF] Chew | Cardiac Response to Live Music Performance: Computing Techniques for Feature Extraction and Analysis[END_REF][START_REF] Chew | On making music with heartbeats[END_REF], in the body movements [START_REF] Müller | Interactive deep singingvoice separation based on human-in-the-loop adaptation[END_REF]], as well as in the neural activity [START_REF] Sturm | Analyzing the perception of natural music with EEG and ECoG[END_REF]] to mention a few. Such kind of information would help the separation process and make it also interactive, allowing for a number of futuristic applications where the human-machine interaction is simpli ed and natural. Among those physiological responses to music stimuli, we are interested in the neural response, focusing on the concept of selective auditory attention.

S

A A refers to a multitude of behavioural and cognitive mechanisms that allow humans to process concurrent sounds in a complex auditory scene to isolate the ones of interest [START_REF] Kawala-Sterniuk | Summary of over Fifty Years with Brain-Computer Interfaces-A Review[END_REF].

A Trumpet

The resulting perceptual e ect is known as the "Cocktail party e ect" and was rst described by Cherry in [START_REF] Cherry | Some experiments on the recognition of speech, with one and with two ears[END_REF]] in relation to the perception of speech sources in noisy or multi-speaker settings. In practice, we can follow a single conversation while ltering out competing speakers, other sounds and noise. Therefore, one can de ne attention as "the set of processes that allow the cognitive system to select the relevant information in a given context" [START_REF] Turatto | notes and lessons[END_REF]]. The need for attention comes from the fact that the cognitive system and the substrate on which it is based, i.e., the brain, is not able to analyze all incoming information at the same level of detail [START_REF] Turatto | notes and lessons[END_REF]]. In other words, it is not possible to be aware of everything at the same time.

A C are di erent processes with di erent functions. Selective attention lters what can be elaborated in depth, so that it has access to the subject's consciousness. Attention is therefore a complex process of information selection that takes place in similar ways in the di erent sensory modalities: visual, tactile, auditory, etc. Consciousness, instead, summarize all information which are previously selected in order to perform deeper elaboration like decision making, language, rational thought, and so on [START_REF] Koch | Attention and consciousness: two distinct brain processes[END_REF].

Attention can be triggered via bottom-up mechanisms or by top-down factors [START_REF] Kawala-Sterniuk | Summary of over Fifty Years with Brain-Computer Interfaces-A Review[END_REF]. In the rst case, our attention is involuntarily attracted by sounds like a phone ringing, an alarm, a baby crying, which significantly di er from the ones of the background/neighbourhood, making them salient in that context [START_REF] Koch | Attention and consciousness: two distinct brain processes[END_REF]. What rst makes a stimulus salient is its sudden appearance (onset), but there are other aspects which are mostly studied for visual stimuli such as chromatic or shape characteristics [START_REF] Jonides | Capturing attention[END_REF]. In the second case, our brain is voluntarily paying attention to a circumscribed region in space (focal attention), a particular feature (feature-based attention) or an object (object-based attention) in order to achieve a given task [START_REF] Koch | Attention and consciousness: two distinct brain processes[END_REF]. In this case, the condition is that the subject knows the characteristic that de nes the target [START_REF] Blankertz | Single-trial analysis and classi cation of ERP components -a tutorial[END_REF]. It is the case of our brain focusing on a single conversation at a cocktail party or a particular musical instrument during a concert.

Since Cherry rst published his work, the interest in auditory attention has increased substantially, producing a vast literature about it. 4 As many other cognitive processes, auditory attention had been studied in earlier times through psychoacoustic experiments. The turning point happened with the advent of ner techniques for measuring the neural activity which can give a deeper insight into how, where and when selective attention manifests in our brain.

M

represent a crucial choice for designing experiments and BCIs based on selective attention. The most common techniques to measure neural activity as a response of a given stimulus are:

• Electroencephalography (EEG);

• Magnetoencephalography (MEG);

• Electrocorticography (ECoG);

• Local Field Potentials (LFP);

• functional Magnetic Resonance Imaging (fMRI).

Each technique has advantages and disadvantages in terms of invasiveness degree, spatial and temporal resolution. EEG and MEG [START_REF] Cohen | Magnetoencephalography: evidence of magnetic elds produced by alpha-rhythm currents[END_REF]], are noninvasive techniques with a high temporal resolution (order of milliseconds), making them appealing for tracking dynamic changes in the brain. They respectively measure the electric and magnetic elds on the scalp which comes with a low spatial resolution (order of centimeters) and low Signal-to-Noise-Ratio (SNR). To improve the SNR and spatial resolution, the electrodes can be implanted on the cortical surface, below the skull, using ECoG [START_REF] Jasper | Electrocorticograms in man: e ect of voluntary movement upon the electrical activity of the precentral gyrus[END_REF], or directly inside the brain to record the LFP [START_REF] Einevoll | Local eld potentials[END_REF]. As those techniques are highly invasive, micro-electrodes are only implanted to monitor epileptic patients making this data rare and private.

Moreover, the limited brain coverage gives information only regarding a restricted view of the auditory processing hierarchy [O'sullivan et al. 2014].

fMRI [START_REF] Ogawa | Hierarchical organization of human auditory cortex: evidence from acoustic invariance in the response to intelligible speech[END_REF]] represents a complementary technique. It has a limited temporal resolution (order of seconds) associated with a much higher spatial resolution (order of millimeters) which allows for precisely localizing the anatomical areas involved in a certain cognitive process. While this technique is largely used in cognitive research, its temporal resolution does not enable the dynamic tracking of attention which occurs within the milliseconds range, but only its allocation in space [START_REF] Wang | Decoding dynamic auditory attention during naturalistic experience[END_REF]].

As we have seen, the choice of one of those techniques is strictly connected with the goal one wants to achieve. In our speci c case, we need a high enough temporal resolution to dynamically track the auditory attention to a sound source. Secondly, we need to exclude invasive techniques which are not appropriate for a use in real-life scenarios as a part of BCIs. Thus the most natural choice for us is to consider EEG signals, which represent the most portable method by which the neural activity can be recorded and from now on we will restrict our focus to them.

T

characterized in terms of EEG signals can be analyzed and understood using diverse approaches. As mentioned in the previous paragraph, EEG signals exhibit a very low SNR and therefore, it is hard to study a single phenomenon of interest. A typical work-ow is to repeat the stimulus several times and then average the EEG responses in order to keep only the stimuli-relevant information and attenuate noise. This approach relies on speci cally selected or designed short stimuli, which are only appropriate to study speci c attributes of music or the reaction to isolated sounds. Short stimuli generate a well-de ned response in the EEG signals, called Event-related Potentials (ERPs).

ERPs exhibit a characteristic morphology: peaks are observed at a speci c time-latency in the average EEG responses, which in the literature are referred to, for instance, as N100, P300, etc. ERPs are actually generated either from short stimuli or stimuli with high contrast with the background. The last characteristics can be re-created in experimental settings through the so-called oddball paradigm, where the subject is stimulated with a rare deviant event occurring among more frequent standard events [START_REF] Treder | Decoding auditory attention to instruments in polyphonic music using single-trial EEG classi cation[END_REF]. In practice, less expected musical events produce stronger neural responses. This process is usually associated with bottom-up mechanisms and it is then di cult to distinguish e ects due to the perceptual novelty of the stimulus from the ones due to the stimulus signi cance.

ERP are typically considered to study attention to particular musical structures such as note onsets, rhythm and pitch patterns or, at least unattended musical deviants among standard and attended events [START_REF] Treder | Decoding auditory attention to instruments in polyphonic music using single-trial EEG classi cation[END_REF]. Some studies aim at understanding how the brain processes basic structural components of music such as pitch [START_REF] Hyde | Evidence for the role of the right auditory cortex in ne pitch resolution[END_REF][START_REF] Kumar | Predictive coding and pitch processing in the auditory cortex[END_REF][START_REF] Nan | Di erential roles of right temporal cortex and Broca's area in pitch processing: evidence from music and Mandarin[END_REF][START_REF] Plack | Pitch coding and pitch processing in the human brain[END_REF], timbre [START_REF] Deike | Auditory stream segregation relying on timbre involves left auditory cortex[END_REF]Goydke et al. 2004;[START_REF] Caclin | EEG-Based Decoding of Auditory Attention to a Target Instrument in Polyphonic Music[END_REF] as well as sensory dissonance, high-level melodic characteristics (e.g., melodic contour, key, mode, scale) and music-syntactic congruity [START_REF] Koelsch | Processing of hierarchical syntactic structure in music[END_REF][START_REF] Sturm | Analyzing the perception of natural music with EEG and ECoG[END_REF]]. The most studied components in the perception of music appear to be note onsets, beats, rhythm and meter [START_REF] Thaut | Rhythm, human temporality, and brain function[END_REF][START_REF] Cirelli | Beat-induced uctuations in auditory cortical beta-band activity: using EEG to measure age-related changes[END_REF][START_REF] Sturm | Analyzing the perception of natural music with EEG and ECoG[END_REF][START_REF] Stober | Brain Beats: Tempo Extraction from EEG Data[END_REF]. Music presents strong timing mechanisms that have been recognized to engage human behavior and brain function in multiple ways [START_REF] Thaut | Rhythm, human temporality, and brain function[END_REF]]. In particular, low-level structural elements of music, such as note onsets, can be considered distinct auditory events which allow the perception of more complex entities such as beat, rhythm, and meter [START_REF] Sturm | Analyzing the perception of natural music with EEG and ECoG[END_REF]].

Even though ERPs can give insights into how these musical attributes individually relate to neural processes, a di erent approach is needed to study the continuous brain response to a complex naturalistic stimulus such as a musical piece in its entirety. A few attempts have been made to track the dynamics of naturalistic music stimuli in the EEG signal. Cong et al. found evidence that the sound time-domain envelope is consistently re ected in the EEG [START_REF] Cong | Analysis of ongoing EEG elicited by natural music stimuli using nonnegative tensor factorization[END_REF]]. Ofner and Stober reconstructed the spectrograms of both perceived and imagined music from the EEG [START_REF] Ofner | Shared generative representation of auditory concepts and EEG to reconstruct perceived and imagined music[END_REF], while Sturm et al. did the same for the note onset sequence [Sturm et al. 2015a;Sturm et al. 2015b]. Kaneshiro et al. investigated how musical engagement is re ected in the EEG-response and to what extent this is related to the temporal organization of acoustical events, their novelty and repetition [Kaneshiro et al. 2016b;Kaneshiro 2016;[START_REF] Kaneshiro | Natural music evokes correlated EEG responses re ecting temporal structure and beat[END_REF][START_REF] Kaneshiro | Inter-subject EEG correlation re ects time-varying engagement with natural music[END_REF]].

Di Liberto et al. showed that the cortical tracking of the music envelope is signi cantly modulated by cognitive factors such as attention and expectation, which strongly depend on the listener's musical culture and expertise during both the listening [Di Liberto et al. 2020b;Di Liberto et al. 2020a] and imagery tasks [START_REF] Marion | The Music of Silence: Part I: Responses to Musical Imagery Encode Melodic Expectations and Acoustics[END_REF][START_REF] Liberto | The Music of Silence: Part II: Music Listening Induces Imagery Responses[END_REF]]. 5 Nevertheless, these works focused on stimulus reconstruction and not on decoding the attended instrument. Two attempts have been made for classifying the attended [START_REF] Schaefer | Deep learning with convolutional neural networks for EEG decoding and visualization[END_REF] or imagined [START_REF] Marion | The Music of Silence: Part I: Responses to Musical Imagery Encode Melodic Expectations and Acoustics[END_REF]] music and the attended instrument [START_REF] Treder | Decoding auditory attention to instruments in polyphonic music using single-trial EEG classi cation[END_REF]] but both of them focused on the elicited ERPs.

A

A D Selective attention has been observed to modulate the neural activity in several di erent ways. E ects can be seen as an enhancement of neural activity [START_REF] Hillyard | Electrical signs of selective attention in the human brain[END_REF][START_REF] Woldor | Modulation of early auditory processing during selective listening to rapidly presented tones[END_REF][START_REF] Woldor | Modulation of early sensory processing in human auditory cortex during auditory selective attention[END_REF][START_REF] Jäncke | Attention modulates activity in the primary and the secondary auditory cortex: a functional magnetic resonance imaging study in human subjects[END_REF], connectivity [START_REF] Lipschutz | Attentiondependent changes of activation and connectivity in dichotic listening[END_REF][START_REF] Tóth | Attention and speech-processing related functional brain networks activated in a multi-speaker environment[END_REF] and synchronization [START_REF] Doesburg | Theta modulation of inter-regional gamma synchronization during auditory attention control[END_REF] or as a more robust encoding of the attended source compared to the unattended ones [START_REF] Mesgarani | In uence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex[END_REF]]. The latter makes it possible to decode the auditory attention, i.e., determining which sound source a person is "focusing on", by just observing at the listener's brain response. This task is known as Auditory Attention Decoding (AAD), and typical applications are intelligent hearing aids where a neuro-steered enhancement of the attended speaker is desired [START_REF] Han | Speakerindependent auditory attention decoding without access to clean speech sources[END_REF]Das et al. 2020b;[START_REF] Aroudi | Improving auditory attention decoding performance of linear and non-linear methods using state-space model[END_REF].

Previous AAD studies based on continuous MEG [START_REF] Ding | Emergence of neural encoding of auditory objects while listening to competing speakers[END_REF][START_REF] Akram | A state-space model for decoding auditory attentional modulation from MEG in a competing-speaker environment[END_REF][START_REF] Brodbeck | Neural source dynamics of brain responses to continuous stimuli: speech processing from acoustics to comprehension[END_REF], ECoG [START_REF] Mesgarani | In uence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex[END_REF][START_REF] Pasley | Reconstructing speech from human auditory cortex[END_REF][START_REF] Mesgarani | Selective cortical representation of attended speaker in multi-talker speech perception[END_REF] and EEG [O'sullivan et al. 2014;[START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF]] signals have shown that the neural activity tracks dynamic changes in the audio stimulus and can be successfully used to decode selective attention in a complex auditory scene.

In a number of works [START_REF] Mesgarani | In uence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex[END_REF][START_REF] Pasley | Reconstructing speech from human auditory cortex[END_REF][START_REF] Mesgarani | Selective cortical representation of attended speaker in multi-talker speech perception[END_REF]O'sullivan et al. 2014;[START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF], a feature representation of the stimulus is reconstructed from the multi-channel neural recordings through a multi-channel Wiener-lter which is learned by solving a linear regression problem [START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF]]. Mesgarani and Chang were among the rst to show that such reconstructed feature representations (in this case spectrograms) were highly correlated with the salient time-frequency features of the attended speaker's voice, and were only weakly correlated with the unattended speaker ones [START_REF] Mesgarani | Selective cortical representation of attended speaker in multi-talker speech perception[END_REF].

These works all focused on reconstructing a speci c category of stimuli, i.e., speech. Much less developed is AAD research applied to other types of naturalistic stimuli such as music. In the latter case, one can recast the problem as one of decoding the attention to the "voice" of a particular musical instrument playing in an ensemble. 6 However, this transposition is not straightforward 6 Zuk et al. also showed that EEG responses are stronger to speech and music than to other natural sounds [START_REF] Zuk | EEG-based classi cation of natural sounds reveals specialized responses to speech and music[END_REF]], but, among the two, speech evokes larger responses and unique e ects at low frequencies, leading to better reconstructions of the speech envelope than for music [START_REF] Zuk | Envelope reconstruction of speech and music highlights stronger tracking of speech at low frequencies[END_REF].

as, unlike in the cocktail party problem where there is one source of interest to separate from unrelated background noise or speakers, music consists of multiple voices playing together in a coordinated way. Thus the sources are generally highly correlated, making the decoding problem even more di cult [START_REF] Treder | Decoding auditory attention to instruments in polyphonic music using single-trial EEG classi cation[END_REF][START_REF] An | Exploring combinations of auditory and visual stimuli for gaze-independent brain-computer interfaces[END_REF].

N

The main limitation of most AAD approaches is their use of the separate "clean" audio sources. In fact, the feature representations extracted from the isolated sources are correlated with the ones predicted with the neural data to determine the attended source [START_REF] Mesgarani | Selective cortical representation of attended speaker in multi-talker speech perception[END_REF]O'sullivan et al. 2014]. However, the isolated sources are not available in realistic scenarios (e.g., hearing aids) where only the mixture of the sound scene recorded by their microphones is available and an AuSS step, where single audio sources are extracted from their mixture, is needed. This limitation is strongly intertwined with a specular aspect of AuSS, whose process can be informed by prior knowledge one has about the sources.

Few works have been proposed in the last years that relate speech source separation with AAD, but most of the time, the two tasks are tackled independently. Either the separated sources are used as clean sources to decode attention, or the EEG is used to decode which source needs to be enhanced. This has been implemented in multi-channel scenarios using beamforming [START_REF] Van Eyndhoven | EEG-Informed Attended Speaker Extraction From Recorded Speech Mixtures With Application in Neuro-Steered Hearing Prostheses[END_REF][START_REF] Aroudi | EEG-based auditory attention decoding using steerable binaural superdirective beamformer[END_REF][START_REF] Aroudi | Cognitive-driven binaural LCMV beamformer using EEG-based Auditory Attention Decoding[END_REF][START_REF] Aroudi | Improving auditory attention decoding performance of linear and non-linear methods using state-space model[END_REF] and in single-channel scenarios using Deep Neural Networks (DNNs) [START_REF] O'sullivan | Neural decoding of attentional selection in multi-speaker environments without access to separated sources[END_REF][START_REF] Das | EEG-based attention-driven speech enhancement for noisy speech mixtures using N-fold multi-channel Wiener lters[END_REF][START_REF] Han | Speakerindependent auditory attention decoding without access to clean speech sources[END_REF]Das et al. 2020b].

Only a few works proposed in parallel to our own use directly the neural activity of the listener to inform a source separation model, but they all focused on speech sources [START_REF] Pu | A joint auditory attention decoding and adaptive binaural beamforming algorithm for hearing devices[END_REF][START_REF] Ceolini | Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception[END_REF]. In [START_REF] Pu | A joint auditory attention decoding and adaptive binaural beamforming algorithm for hearing devices[END_REF], the authors propose an adaptive beamformer that reduces noise and interference but, at the same time, maximises the Pearson correlation between the envelope of its output and the decoded EEG. In [START_REF] Ceolini | Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception[END_REF], instead, a speech separation DNN is informed with the decoded attended speech envelope. Nevertheless, none of these works considers music audio signals.

C

Within this PhD project, we have investigated how MSS methods can be guided interactively by the user's brain activity. We focused on the concept of selective auditory attention, which allows humans to process concurrent sounds and isolate the ones of interest. Our work investigates how to leverage this phenomenon to guide a MSS system and automatically separate the attended source. Such a formulation would also allow reformulating the AAD problem without the need for the ground truth sources.

Among the signals that can characterize the brain response, we consider the EEG, which allow for non-invasive neural activity acquisition with high temporal resolution, making it an ideal candidate for developing BCIs.

The proposed approach is summarized in Figure 1.4 and can be divided into two main tasks, which can be tackled jointly:

• Decoding auditory attention to a target instrument in polyphonic music mixtures;

• Neuro-steered source separation of the target instrument from a polyphonic music mixture.

Chapters summary

We brie y describe here the contents of each Chapter, emphasizing the contributions and listing the associated publications. While Part I introduces the motivation and objective behind this work, the rest of the thesis is divided in two main parts:

P

II A This part focuses on the rst task, i.e., decoding auditory attention to a target instrument in polyphonic music mixtures.

Chapter 2 This Chapter describes in detail the music-related EEG dataset we have assembled for the thesis, namely MAD-EEG, which allows for studying the problems of single-trial EEG-based Auditory Attention Decoding and EEG-guided Music Source Separation. It is crucial for the reader to have a clear understanding of the recording protocol and how the stimuli were built to easily follow the rest of the thesis, and how the proposed algorithms are applied to this data. MAD-EEG, represents our rst main contribution and is available to the research community as a free resource. The dataset was acquired by my colleague Gabriel Trégoat during his internship at Télécom Paris and nalised by me, leading to the following conference publication: • Cantisani, Giorgia, Gabriel Trégoat, Slim Essid, and Gaël Richard (2019b). "MAD-EEG: an EEG dataset for decoding auditory attention to a target instrument in polyphonic music". In: Proc.

Workshop on Speech, Music and Mind (SMM19), pp. 51-55

Chapter 3 This Chapter describes the second main contribution of the thesis, relating to the problem of decoding the auditory attention to a target instrument in polyphonic music which was extensively investigated on the MAD-EEG dataset. The primary outcome of this study is that the EEG tracks musically relevant features highly correlated with the attended source and weakly correlated with the unattended one making it possible to decode the auditory attention towards a speci c instrument in the mixture. This study is particularly important within the thesis, as the proposed neuro-steered Music Source Separation approaches are built upon the results of this Chapter. This work led to the following conference publication: F , the dissertation concludes with Chapter 6, which discusses the principal ndings of the current investigation, together with discussion on future perspectives and research directions.

I

, the reader will nd a chapter about a science dissemination project I have coordinated which led to the release of a short video explaining in simple terms what Music Information Research (MIR) is all about. This part is not strictly related to the research topic of the thesis but the more general problem of science communication and dissemination.

1.4 L • Cantisani, Giorgia, Gabriel Trégoat, Slim Essid, and Gaël Richard (2019b).

"MAD-EEG: an EEG dataset for decoding auditory attention to a target instrument in polyphonic music". In: Proc. Workshop on Speech, Music and Mind (SMM19), pp. 51-55

• Cantisani, Giorgia, Slim Essid, and Gaël Richard (2019a) 1.5 V

The reader will have already noticed that a large margin is left free on each manuscript page. We will use it to insert additional insights, notes and gures to complete each subject. This graphic template is inspired by the work of Tufte and Graves-Morris [START_REF] Tufte | The visual display of quantitative information[END_REF] 7 and exhibits some 7 The colophon of the thesis reports more information on the template. peculiarities:

• at most three levels of sub-headings: section, subsection, and Tufte's new-thought [START_REF] Tufte | The visual display of quantitative information[END_REF] and to capture attention;

• reference sidenotes on the margin are used as footnotes, providing additional insights;

• italic sidenotes and gures without proper reference numbers on the margin are meant to provide optional information and can be read in a second time;

• orange is used for clickable internal reference, such as for sections § 1.2 and acronyms AAD;

• grey and is used for clickable external link, such as my website .

Part II DECODING OF AUDITORY ATTENTION TO MUSIC , Giorgia, Gabriel Trégoat, Slim Essid, and Gaël Richard (2019b). "MAD-EEG: an EEG dataset for decoding auditory attention to a target instrument in polyphonic music". In: Proc. Workshop on Speech, Music and Mind (SMM19), pp. 51-55 F 2.1: Our research shows that, compared to the overall population, people who agree to participate in my scienti c studies are signi cantly more likely to ask me to participate in their studies. Image courtesy of xkcd, number 1999.

I

MAD-EEG is a new, freely available dataset for studying EEG-based AAD considering the challenging case of subjects attending to a target instrument in polyphonic music. The dataset represents the rst music-related EEG dataset of its kind, enabling, in particular, studies on single-trial EEG-based AAD while also opening the path for research on other EEG-based music analysis tasks such as neuro-steered MSS.

MAD-EEG has so far collected 20-channel EEG signals recorded from 8 subjects listening to solo, duo and trio music excerpts and attending to one pre-speci ed instrument. The stimuli were designed considering variations in the number and type of instruments in the mixture, spatial rendering, music genre and melody, which allow testing the in uence of certain factors on the AAD and neuro-steered MSS performance.

It is worth noting that the setting is entirely di erent from the ones previously proposed. The experimental protocol usually applied for AAD data acquisi-tion like the ones of [O'sullivan et al. 2014;[START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF][START_REF] Treder | Decoding auditory attention to instruments in polyphonic music using single-trial EEG classi cation[END_REF], considers two monaural sources each played to a di erent ear through headphones. Instead, in our recording sessions, the stimuli were reproduced using speakers, and the audio was rendered in varying spatial con gurations.

R

A few publicly available music-related EEG datasets exist. Stanford University researchers have assembled a number of such datasets: the Naturalistic Music EEG Dataset-Hindi (NMED-H) [Kaneshiro et al. 2016a], the Naturalistic Music EEG Dataset-Tempo (NMED-T) [START_REF] Losorelli | NMED-T: A tempo-focused dataset of cortical and behavioral responses to naturalistic music[END_REF], the Naturalistic Music EEG Dataset-Rhythm Pilot (NMED-RP) [START_REF] Appaji | Neural tracking of simple and complex rhythms: pilot study and dataset[END_REF], the Naturalistic Music EEG Dataset-Elgar (NMED-E) [Kaneshiro et al. 2021a] and the Naturalistic Music EEG Dataset -Minimalism (NMED-M) [START_REF] Dauer | Naturalistic Music EEG Dataset -Minimalism (NMED-M)[END_REF]. Each of these contains EEG and behavioural responses to di erent kinds of naturalistic music stimuli. Other music-related EEG datasets are the OpenMIIR dataset [START_REF] Stober | Towards Music Imagery Information Retrieval: Introducing the OpenMIIR Dataset of EEG Recordings from Music Perception and Imagination[END_REF]] acquired for studying music perception and imagination and the DEAP [START_REF] Koelstra | Deap: A database for emotion analysis; using physiological signals[END_REF]] and the BCMI [START_REF] Daly | Neural and physiological data from participants listening to a ective music[END_REF] databases, acquired for studying a ective responses to music. Recently, it was released a dataset that allows to compare brain responses to the music of individuals with an intellectual and developmental disorder and typically developing ones [START_REF] Sareen | EEG dataset of individuals with intellectual and developmental disorder and healthy controls under rest and music stimuli[END_REF]]. In such datasets, the user focused on the entire stimulus and not on a particular instrument. Thus they are not relevant for the study of the AAD problem. The only publicly available music-related EEG dataset where participants were asked to attend to a target instrument in the music mixture is the music BCI dataset collected by [START_REF] Treder | Decoding auditory attention to instruments in polyphonic music using single-trial EEG classi cation[END_REF]]. The dataset was explicitly designed for studying ERP-based AAD using a multi-streamed oddball paradigm, where a repetitive musical pattern is interspersed with a randomly occurring deviant pattern that yields clean P300 ERPs. 2 However, 2 P300: is an ERP across the parietal-central area of the skull that occurs around 300 ms after stimuli presentation [START_REF] Fabiani | De nition, identi cation, and reliability of measurement of the P300 component of the event-related brain potential[END_REF]]. Its wave is larger after the target stimulus and only occurs if the subject actively engages in detecting the targets. Its amplitude varies with the target improbability, while its latency varies with the di culty of discriminating the target from the standard stimuli.

the oddball paradigm's assumption does not often hold in real-world music compositions as we have seen in § 1.2.

The situation is di erent when considering datasets for AAD in speech. In this case, several datasets and methods were designed to study this problem using a single-trial approach. Nevertheless, only a few of them are accessible [START_REF] Fuglsang | Noise-robust cortical tracking of attended speech in real-world acoustic scenes[END_REF]Das et al. 2020a] Taking inspiration from the speech-related EEG datasets, we assembled our EEG dataset from subjects listening to realistic polyphonic music and attending to a particular instrument in the mixture. Our dataset represents the rst EEG dataset designed explicitly for studying AAD applied to realistic polyphonic music using single-trial techniques.

D

Surface EEG signals were recorded from 8 subjects while listening to polyphonic music stimuli. For each audio stimulus consisting of a mixture containing from two to three instruments, the subjects were asked to attend to a particular instrument.

Each subject listened to a total of 78 stimuli presented in a random order, each one consisting of 4 repetitions of the same roughly 6-second-long music excerpt, leading to a total of approximately 30-32 minutes of 20-channel EEG recordings per subject. Each subject listened to 14 solos, 40 duets and 24 trios, except one subject who only listened to 7 solos, 29 duets and 17 trios.

Participants

Eight volunteers (7 males and one female, all but one right-handed, aged between 23 and 54 years, mean age 28) took part in the study. All of them were healthy and reported no history of hearing impairments or neurological disorders. All participants signed a consent that informed them about the experiment's modalities and purposes. All the data was anonymized.

The study conforms with the Declaration of Helsinki [World Medical Association 2013]. The data were collected preventing all possible health risks for the participants. In particular, they were not exposed to sound pressure levels that can impair their hearing, be painful or lead to other adverse e ects; they were taught carefully about how to behave correctly and safely during the experiment and with the EEG acquisition equipment.

The participants were hired within our laboratory, and took part in the experiments as volunteers. In particular, 2 were PhD students, 5 Master students of our lab, and one a sound engineer of the school. They were all non-professional musicians with varying years of musical experience (from 7 to 30 years, mean 13.5), as can be seen in Figure 2.2. However, they all de ned themselves as beginners. Five out of them play the guitar, one the bass, one the drums, and one is a multi-instrumentalist playing the drums, guitar and bass. They all practised regularly with their instrument (from 2 to 14 hours per week, mean 6.25). Figure 2.3 presents the number of hours per week that each subject usually spends listening to music (blue) and practices his/her instrument (red).

All of them were familiar with the modern instruments in the dataset (drums, guitar, bass and singing voice), while for speci c classical instruments (bassoon, French horn and oboe) not all of them were equally con dent as can be seen in Figure 2.4. Thus, they were trained to recognize them before the experiment 2.4: The number of subjects familiar with each instrument that appears in the dataset. The naming convention for the instruments is: Fl for Flute, Ob for Oboe, Fh for French Horn, Bo for Bassoon, Co for Cello, Vx for Voice, Gt for Guitar, Bs for Bass and Dr for Drum. using excerpts not used as stimuli.

Stimuli

The stimuli consist of realistic polyphonic music mixtures containing two to three instruments played concurrently in an ensemble. The chosen mixtures reproduce a realistic setting. In particular, real music compositions for which we had access to the isolated instrument tracks were chosen for pop pieces. For Classical music pieces, instead, a selection of excerpts played by single instruments were linearly mixed as follows:

x(t) = J j=1 g j s j (t), (2.1)
where s j (t) is the mono-channel audio track of the single instrument j, g j is the corresponding gain, T its number of samples, and J is the number F 2.5: On the left, an illustration of the recording session for one subject. A recording session is divided into sections. Each section is associated with a given musical piece and consists of a training and a test phase, where a series of stimuli sequences is played. Each stimulus sequence consists of 4 trials where the same stimulus is listened to repetitively. On the right, details about the mixtures and how they are spatially rendered.

of instruments in the mixture. Finally, the sound volume was normalized to avoid bias due to the loudness of the audio.

In order to test the in uence of certain factors on the attention decoding performance, we considered di erent con gurations in the choice of the musical stimuli (see Figure 2.5 for a map of the variants):

• Two musical genres: pop and Classical music. Pop excerpts were chosen with sharp rhythmical and harmonic patterns to contrast with the Classical music ones, mostly melodic.

• Two musical pieces per genre and two themes per musical piece. That is, for the same piece, two di erent excerpts corresponding to di erent parts of the score.

• Two ensemble types: duets and trios.

• Two spatial rendering con gurations: monophonic and stereo. The loudspeakers were situated ±45 o along the azimuth direction relative to the listener (see Figure 2.5). The stereo spatial rendering was implemented by merely using conventional stereo panning where one has one instrument mostly on the right and the other one mostly on the left for duets, while for trios the third instrument is in the centre. The target instrument is never in the same position across di erent sequences.

• Musical instruments present in the mixture: di erent combinations of ute, oboe, French horn, bassoon and cello for Classical pieces, along with singing voice, guitar, bass and drums for pop excerpts.

Recording protocol

Each stimulus duration had to be long enough to allow the study of AAD on a single-trial basis while targeting realistic music excerpts. On the other hand, the experiment's duration had to remain reasonably short to control the subject's cognitive load and avoid an unsatisfactory concentration level throughout the session. Consequently, we limited the duration of a stimulus to around 6 seconds. Then, during the experiment, each stimulus was heard by the subject four consecutive times, referred to as trials, corresponding to around 24 seconds of EEG recordings, which is long enough for studying singletrial methods while still making it possible to consider EEG-signal averaging techniques. Since each subject listened to 78 stimuli, this corresponds to approximately 30-32 minutes of recordings per subject.

For each subject the recording session was divided in sections as can be seen in Figure 2.5. In each section a series of stimuli sequences is played. Each section is actually composed of a training and a test phase. During the training phase, single instrument tracks of a given piece are played separately as solos, in a random order. Then, during the test phase, all the corresponding duo and trio variants of the same piece are played in a random order, but with a potentially di erent spatial rendering and considering a di erent theme of the same musical piece. For each instrument solo of a given piece, between 2 and 6 mixtures where the same instrument is attended to are available, but the theme and spatial rendering may di er. This is meant to allow studies on the generalization ability of an AAD system when the pitch contour varies between training and testing.

A section is presented to the user through a slide-show video showing instructions, displayed as white text on a black background, asking the participant to attend to a particular instrument and visually x a cross at the centre of the screen. A "beep" precedes each stimulus launch.

A

Right after each section, the subjects were asked to self-assess the level of attention they paid to each stimulus on a discrete scale ranging from 1 to 5. The level of attention was generally high, except for only a few stimuli (see Figure 2.6) which can be used to evaluate how 6XEMHFWV $WWHQWLRQOHYHO F 2.6: Statistics of the level of attention reported by each subject at the end of the sections for each stimulus proposed in the section. The scale goes from 1 to 5, where 0 represents no attention and 5 the maximum level. the performance of an attention decoding system changes with the attention self-assessed by the subjects. At the end of the session, the participants were also asked to indicate the level of fatigue and stress experienced (low, medium or high) and if they had comments or remarks on the whole process. In general, the fatigue/stress experienced was reasonable as can be seen in Figure 2.7 and Figure 2.8, ensuring that the quality of the collected data is good since the subjects were not overloaded.

EEG

Blinking, heartbeat, and other involuntary movements signi cantly modify the EEG recordings while being independent of the stimuli and, therefore, can bias the recorded signals' interpretation. Thus, subjects were instructed to maintain, for the duration of each trial, visual xation on a cross at the centre of the screen and minimize eye blinking and other motor activities. Moreover, during breaks at the beginning, middle and end of the experiment, a series of instruction videos were used to ask the participants to perform di erent gestures (shake their cheeks, blink their eyes), each of which has a particular in uence on the EEG. This portion of EEG signals are also available within the dataset and can be used by those interested in studying artefact removal techniques, possibly using them on the music-related portions of this EEG dataset.

Data Acquisition and Preprocessing

A B-Alert X24 1 headset was used to record the surface EEG, Electrooculography EOG electrodes were placed above and below each eye diagonally, while ECG ones were placed in the middle and left side of the last rib. EMG electrodes were placed in such a way that they record the activity of the big zygomatic (whose activity can be recorded at a position situated at mid-distance between the top of the ear and the eye of the mouth), and the inferior palpebral orbicularis, which can be contracted simultaneously only involuntarily.

S

A custom software interface automatizes the whole acquisition process and save the necessary information to synchronize the stimuli and the EEG responses. An External Sync Unit (ESU) receives data from the EEG headset via Bluetooth and passes it over to the acquisition software along with timestamps associated with each EEG signal sample as can be seen in Figure 2.9. This ESU can also receive the custom experimenter's auxiliary data and record it along with the EEG data. We use this feature of the ESU to record stimulus playback start times accurately. Thus, the beginning of the stimulus playback is detected in real-time through a Python script monitoring the playback PC sound card output. These playback start-events are then sent through the PC's serial port to the ESU so they can be marked as timestamps for the stimuli. This is done to detect the exact time instant when each stimulus starts within a 10-ms tolerance window. The EEG and the stimuli timestamps are thus saved by the EEG recording software and can be subsequently used o ine for synchronization.

A

Music stimuli were presented using a Hi-Fi audio playback system (JMlab chorus lcr700 speakers and Yamaha DSP-AX2 Natural Sound AV ampli er). The listener was seated at the centre of the room, 2 meters from a full HD TV (165 cm) screen and 2.8 meters from the two speakers. The speakers were positioned ±45 o along the azimuth direction relative to the listener as depicted in Figure 2.11. The spatial rendering was implemented by merely using conventional stereo panning. This means, that for each instrument in the mixture the left and right channels are obtained as follows:

L R = α 1 -α s j (t), (2.2)
where α ∈ [0, 1] and s j (t) ∈ R 1×T is the mono-channel audio track of the single instrument j. The volume was set to be comfortable and was kept constant across all sessions.

D

Firstly, the EEG data were visually inspected to detect anomalies and keep only valid recording takes (e.g., subject 5 has EEG responses to 53 stimuli instead of 78). Then, the acquired EEG data was synchronized with each stimulus, the 50 Hz power-line interference was removed using a notch lter, and EOG/ECG artefacts were detected and removed using Independent Component Analysis (ICA). The frequencies below 1 Hz were ltered out using a Butterworth zero-phase lter of order 2. Each channel was normalized to ensure zero mean and unit variance. All the data was anonymized.

C

MAD-EEG is a novel, free dataset that enables studies on the problem of EEGbased Auditory Attention Decoding to a target instrument in realistic polyphonic music and EEG-guided Music Source Separation. The numerous variants in the stimuli and the behavioural data allow for investigating how such factors impact on the AAD and neuro-steered MSS performance.

It represents the rst dataset of its kind for music stimuli and can be di erentiated also from those commonly used for studying AAD for speech stimuli. In fact, the proposed experimental setting di ers from the ones previously considered as the stimuli are polyphonic and are played to the subject using speakers instead of headphones.

It is a common experience that acquiring such a type of dataset is timeconsuming and expensive. It requires speci c equipment and experience but as well a long phase of preparation and experimental design. It takes much time to recruit participants who can participate in the experiment, and each of those is available for a limited amount of time and cannot be overloaded with too long recording sessions. Therefore, the dataset is limited in terms of recording hours and the number of participants but still allows for studying those problems if speci c strategies are adopted to avoid over tting.

MAD-EEG represents the rst main contribution of the thesis and is made available to the research community as a free resource. The primary outcome of this study is that the EEG tracks musically relevant features highly correlated with the attended source and weakly correlated with the unattended one making it possible to decode the auditory attention towards a speci c instrument in the mixture. This study is particularly important within the thesis, as the proposed neuro-steered Music Source Separation approaches are built upon the results of this Chapter. This work led to the following conference publication:

I

In this Chapter, we address the problem of EEG-based decoding of auditory attention to a target instrument in realistic polyphonic music. To this end, we exploit the so-called backward model, which was proven to decode the attention successfully to speech in multi-speaker environments [O'sullivan et al. 2014;[START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF]]. To our knowledge, this model was never applied before to musical stimuli for AAD and we extensively evaluated it on the MAD-EEG dataset. The task we consider here is quite complex compared to the classical one for speech stimuli which considers two monaural sources each played to a di erent ear through headphones. Here, the music stimuli are polyphonic, including duets and trios, and the mixtures are reproduced using loudspeakers in varying spatial con gurations. We consider the decoding of three di erent audio representations and investigate the in uence on the decoding performance of multiple variants of musical stimuli, such as the number and type of instruments in the mixture, the spatial rendering, the 29 music genre and the melody/rhythmical pattern that is played. We obtain promising results comparable to those obtained on speech data in previous works and con rm that it is thus possible to correlate the human brain's activity with musically relevant features of the attended source.

3.2 R EEG-based Auditory Attention Decoding aims at determining which sound source a person is "focusing on" by analysing the listener's brain response. Most of the literature in the eld focuses on decoding auditory attention to naturalistic speech in multi-speaker or noisy scenarios from the brain's electric activity measured on the scalp [O'sullivan et al. 2014;[START_REF] O'sullivan | Improved decoding of attentional selection in a cocktail party environment with EEG via automatic selection of relevant independent components[END_REF]]. Indeed, the topic is raising more and more interest thanks to the multitude of promising applications, especially concerning hearing aids and cochlear implants [START_REF] Van Eyndhoven | EEG-Informed Attended Speaker Extraction From Recorded Speech Mixtures With Application in Neuro-Steered Hearing Prostheses[END_REF][START_REF] Aroudi | Improving auditory attention decoding performance of linear and non-linear methods using state-space model[END_REF][START_REF] Han | Speakerindependent auditory attention decoding without access to clean speech sources[END_REF]Das et al. 2020b;[START_REF] Pu | A joint auditory attention decoding and adaptive binaural beamforming algorithm for hearing devices[END_REF][START_REF] Ceolini | Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception[END_REF].

First studies on AAD based on continuous ECoG [START_REF] Mesgarani | In uence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex[END_REF][START_REF] Mesgarani | Selective cortical representation of attended speaker in multi-talker speech perception[END_REF][START_REF] Pasley | Reconstructing speech from human auditory cortex[END_REF] and EEG [O'sullivan et al. 2014;[START_REF] O'sullivan | Improved decoding of attentional selection in a cocktail party environment with EEG via automatic selection of relevant independent components[END_REF][START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF] responses have shown that changes in the audio stimulus can be tracked in the neural activity. They evidenced how the attended source's neural encoding is substantially stronger than the one of the other sources left in the mixture, allowing for a successful decoding of selective attention to a speaker. Similarly to Treder et al. [START_REF] Treder | Decoding auditory attention to instruments in polyphonic music using single-trial EEG classi cation[END_REF], we recast the AAD problem in the music domain as one of decoding attention to a speci c musical instrument playing in a musical ensemble.

The decoding procedure is usually two-fold [O'sullivan et al. 2014]: rstly, a feature representation of the attended audio source is reconstructed from the neural response. Secondly, the reconstruction is correlated with the ground truth sources to determine the attended source. The stimulus reconstruction is referred to as the backward problem, as one goes from the brain response back to the stimulus. The mapping is usually done using linear models: a Multichannel Wiener Filter (MWF) maps the neural activity back to a stimulus feature representation [START_REF] Lalor | Resolving precise temporal processing properties of the auditory system using continuous stimuli[END_REF][START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF]. Such a lter is known in the eld as backward model and is estimated on a training set using a Minimum Mean Squared Error (MMSE) criterion [START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF]. Therefore, assuming the system to be linear and time-invariant, the relation between stimulus and neural response can be described as a convolution where the impulse response is represented by the backward model [START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF]].

The majority of works studying auditory attention represented the speech by its broadband temporal envelope [START_REF] Lalor | Neural responses to uninterrupted natural speech can be extracted with precise temporal resolution[END_REF][START_REF] Fuglsang | Noise-robust cortical tracking of attended speech in real-world acoustic scenes[END_REF]O'sullivan et al. 2014;[START_REF] O'sullivan | Improved decoding of attentional selection in a cocktail party environment with EEG via automatic selection of relevant independent components[END_REF]. Others obtained promising results with speech spectrograms [START_REF] Mesgarani | In uence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex[END_REF][START_REF] Mesgarani | Selective cortical representation of attended speaker in multi-talker speech perception[END_REF][START_REF] Pasley | Reconstructing speech from human auditory cortex[END_REF], phonemes [START_REF] Liberto | Low-frequency cortical entrainment to speech re ects phoneme-level processing[END_REF], or semantic features [START_REF] Broderick | Electrophysiological correlates of semantic dissimilarity re ect the comprehension of natural, narrative speech[END_REF].

The choice of the speech representation is critical as di erent features are supposed to map onto di erent hierarchical levels of brain processing [START_REF] Liberto | Low-frequency cortical entrainment to speech re ects phoneme-level processing[END_REF][START_REF] Liberto | Cortical measures of phoneme-level speech encoding correlate with the perceived clarity of natural speech[END_REF]. Many studies suggest that speech perception results from a hierarchical auditory system that processes attributes of the audio stimulus with an increasing level of complexity: earlier areas of the auditory system respond to low-level spectrotemporal and acoustic dynamics, while later areas to semantic and phonetic features of the stimulus [START_REF] Ogawa | Hierarchical organization of human auditory cortex: evidence from acoustic invariance in the response to intelligible speech[END_REF][START_REF] Peelle | Hierarchical processing for speech in human auditory cortex and beyond[END_REF][START_REF] Chang | Categorical speech representation in human superior temporal gyrus[END_REF]. This hierarchical encoding of speech ensures that low-level descriptors of the audio stimulus, such as spectrotemporal and acoustic dynamics and high-level ones such as phonetic or semantic features, are re ected di erently in the EEG. Something similar can be said for music, where low-level acoustic features such as the acoustic energy or the fundamental pitch are theorized to be encoded di erently than high-level ones such as the musical structure [Di Liberto et al. 2020a].

However, when the aim is to perform AAD and not to conduct a neuroscienti c study, the focus stays on spectrotemporal/acoustic descriptors which have been proven to be robust for that task. Here we compare multiple acoustic representations of the music stimulus, namely the broadband Amplitude Envelope (AE), the Magnitude Spectrogram (MAG) and the Mel Spectrogram (MEL).

M

The goal is to determine the attended instrument in a single-trial fashion based on 24-second long EEG excerpts aligned to corresponding audio stimuli (of the same length). Our approach can be summarized in two steps and is similar to the one commonly used for decoding the attention to speech [START_REF] Mesgarani | In uence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex[END_REF][START_REF] Pasley | Reconstructing speech from human auditory cortex[END_REF][START_REF] Mesgarani | Selective cortical representation of attended speaker in multi-talker speech perception[END_REF]O'sullivan et al. 2014;[START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF]] and it is shown in Figure 3.2. First, an audio representation of the attended instrument is reconstructed from the single-trial EEG response of the subject exploiting a decoder previously trained on solos of that instrument. Second, given the isolated instrumental tracks, the attended instrument is recognized as the one that has the highest correlation with the reconstructed stimulus in terms of Pearson correlation coe cient (PCC). A subject-speci c model is learned for each instrument from its solo and the EEG response collected while listening to it. Then, the same model is used to predict a representation of the attended instrument from the EEG response to a mixture containing that instrument. The attended instrument is the one that is mostly correlated with the reconstructed stimulus in terms of Pearson's correlation coe cient.

Audio Feature Extraction

Choosing the audio representation is a crucial point of AAD, as this choice includes a hypothesis about the neural coding of the stimulus and can signicantly impact the reconstruction quality and the decoding performance. We studied three di erent audio representations, one in the time domain and two in the time-frequency (TF) domain: the time domain Amplitude Envelope (AE) computed using the Hilbert transform, the Magnitude Spectrogram (MAG), and the Mel Spectrogram (MEL), a perceptually-scaled representation commonly used for music analysis.

The AE is one of the most used audio descriptor for AAD with speech stimuli as the EEG was shown to track slowly varying changes in the audio stimulus [START_REF] Golumbic | Mechanisms underlying selective neuronal tracking of attended speech at a "cocktail party[END_REF]O'sullivan et al. 2014]. The assumption is that the EEG is linearly related to the broadband energy envelope of the stimulus. However, frequency modulations, i.e., envelope uctuations at speci c frequencies, can give a more complete view of the audio signal. In fact, the spectrogram envelope of natural sounds uctuates across both frequency and time, and this was shown, for instance, to be important for the intelligibility of speech [START_REF] Pasley | Reconstructing speech from human auditory cortex[END_REF]. TF audio representations have already shown good performance for speech stimulus-reconstruction tasks [START_REF] Mesgarani | In uence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex[END_REF][START_REF] Pasley | Reconstructing speech from human auditory cortex[END_REF][START_REF] Mesgarani | Selective cortical representation of attended speaker in multi-talker speech perception[END_REF]. A recent work explored auditory spectrograms modelling the peripheral auditory system [START_REF] Akbari | Dynamic estimation of the auditory temporal response function from MEG in competing-speaker environments[END_REF] as they may better model how the attended source is re ected in the EEG.

The same can be said for music, where the modulations' complexity is much higher than in speech. In practice, the spectrogram can be seen as a timevarying representation of the amplitude envelope at each frequency bin [START_REF] Pasley | Reconstructing speech from human auditory cortex[END_REF]]. Thus, we will assume that the neural responses are linearly related to the spectrogram channels, seen as subband temporal envelopes.

Temporal Response Function

A feature representation of the attended source Ŝ ∈ R K×N where K is the number of features coe cients and T is the number of time samples, is reconstructed from the EEG using the backward model commonly used in the AAD framework [START_REF] Crosse | The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[END_REF]. This lter can be seen as a spatiotemporal decoder which linearly maps the neural activity back to the audio feature representation, as a weighted sum of activity at each electrode in a given temporal context, as follows:

Ŝ = g T R, ( 3
R =                              r 1 (1) r 1 (2) r 1 (3) . . . . . . . . . 0 . . . . . . . . . r C (1) . . . r C (N -L)                              (3.3)
In practice, each k-th feature coe cient of Ŝ is reconstructed independently from the others using a multi-channel Wiener lter g k , which is learned through an MMSE criterion on a training set of solos of the same instrument. Each lter is estimated independently as the normalized reverse correlation:

g k = C -1 RR C RS k , (3.4)
where

C RR = RR T (3.5)
is the estimated auto-correlation of the EEG data and

C RS k = RS T k (3.6)
is the estimated cross-correlation of the stimulus and EEG data across all electrodes and time-lags for the k-th feature coe cient.

Regularization

Since EEG signals are high-dimensional, autocorrelated, noisy data with high trial-to-trial variability, the estimate of the covariance matrices can be imprecise and subject to over tting due to the high number of parameters to estimate [START_REF] Blankertz | Single-trial analysis and classi cation of ERP components -a tutorial[END_REF]]. Several methods have been proposed in the literature for regularization and prevent over tting. Wong et al. nicely compared those techniques on a benchmark dataset for attention decoding to a target speaker [START_REF] Cheveigné | Decoding the auditory brain with canonical component analysis[END_REF]].

We choose to use a shrinkage regularization to constrain the model coecients by smoothly penalizing extreme eigenvalues [START_REF] Blankertz | Single-trial analysis and classi cation of ERP components -a tutorial[END_REF]]: the diagonal of the autocovariance matrix C RR is then weighted as follows:

C RR = (1 -λ)C RR + λνI, (3.7)
where I is the identity matrix, ν is the average eigenvalue trace of C RR , and λ ∈ [0, 1] is the smoothing parameter.

E

Evaluation

We evaluate the reconstruction capabilities through the Pearson correlation coe cient (PCC) of the reconstructed stimulus representation with the attended instrument r attended , the unattended instrument r unattended and the mixture r mixture .

Besides the reconstruction capabilities, we also evaluate the decoding performance in terms of accuracy on the AAD task. Their statistical signi cance was assessed using an adaptation of the computationally-intensive randomization test [START_REF] Noreen | Computer-intensive methods for testing hypotheses[END_REF]], a non-parametric hypothesis test, comparing to chance, which does not make any assumption on the score distribution [Yeh 2000]. The considered signi cance levels are 5%, 1%, 0.1% and 0.01%, and the tests were performed over 10 4 iterations. For further explainations and details about the test, please refer to § 6.2.

Experimental Results

All audio representations were time aligned to the EEG responses acquired at 256Hz. Through a grid search over a set of reasonable values for each parameter (λ ∈ [0.1, 1], τ max ∈ [250, 500] ms, number of Mel bands ∈ [12, 60]), we found the best value for the shrinkage parameter to be λ = 0.1, for τ max to be 250ms post stimulus, and for the number of Mel bands to be 24, using the following train/test splits: for each subject train on 14 solos, test on 40 duets and 24 trios.

D

In Table 3.1 one can see the decoding accuracy with respect to the three audio descriptors and number of instruments in the mixture. All the scores are signi cantly above the chance level, which is 50% for duets, around 33% for trios, and around 44% for all the test mixtures together. TF representations are clearly bene cial for the decoding indicating that envelope uctuations at speci c frequencies can give a complete view of the music audio signal. The two spectrograms, especially the MEL, also proved to be more robust to the mixture's number of instruments. Nevertheless, even if the accuracy scores obtained with the AE are drastically below those obtained with the other two descriptors, they are still statistically signi cant.

C

In Figure 3.5 one can see the PCCs of the reconstructed stimulus with the attended source (blue), the unattended one (pink) and the mixture (orange) for the three audio descriptors. In Figure 3.6 one can see the PCC coe cients of the reconstructed stimulus with the attended source contrasted with the one of respectively the mixture and the unattended source (only for duets). The correlation scores are very low, indicating that the reconstructions are highly deteriorated. Nevertheless, the "contrast" between r attended and r unattended is evident, especially for the two TF descriptors, conrming the decoding results of Table 3.1. Thus, the decoding seems to clearly bene t from the use of a ner audio representation, highlighting amplitude modulations in di erent frequency bands. The lowest r attended Pearson's coe cients are those related to the AE (median r = 0.049) but are still comparable to those obtained by O'Sullivan et al. in [O'sullivan et al. 2014] for speech with the same audio descriptor (median r = 0.054). However, since the contrast between r attended and r unattended is only marginal, the decoding accuracy is much lower than the one obtained by the same authors. The broadband envelope is probably enough for discriminating between attended and unattended speakers but is not enough when dealing with music. Music present complex modulations both in time and frequency, for which the energy envelope is not enough representative.

Here the model is likely to account for e ects more related to the whole mixtures than individual instruments, causing r attended and r mixture to be similar. Moreover, we have to consider that in our case the stimuli were polyphonic and rendered through loudspeakers while in [O'sullivan et al. 2014] two concurrent speech stimuli were presented as monaural sources using headphones playing a di erent source to each ear. In general, when both r attended and r unattended are low and similar, the quality of the reconstructed stimulus is highly deteriorated making it hard to correctly decode the attended instrument. Even so, we can state that the r attended and r unattended distributions are statistically di erent (p = 0.0042 using a Wilcoxon test). Also in the case of the linear spectrogram, the obtained correlations are comparable in terms of magnitude order to the ones obtained previously by [START_REF] Mesgarani | Selective cortical representation of attended speaker in multi-talker speech perception[END_REF] for speech in a di erent setting. From the same plot, we can observe that the correlations obtained with the MAG spectrogram are marginally higher than the ones obtained with the MEL one (median r = 0.215 for MAG, median r = 0.119 for MEL). However, the "contrast" between r attended and r unattended is higher for MEL, which is re ected in the decoding accuracy. The MEL spectrogram is a perceptually scaled and compact version of the linear spectrogram (MAG). A non-linear transformation of the frequency scale based on the perception of pitches (Mel scale) is applied to the linear spectrogram so that two pairs of frequencies that are equidistant in the Mel scale are perceived as being equidistant by humans. We observed that a lower number of features K, or MEL bands, is bene cial for the performance during the experiments. In particular, we tested values ∈ [12, 60], and the results we show are relative to 24 Mel bands. Probably, the MAG representa- all for all the test mixtures, duets and trios for those containing respectively 2 or 3 instruments, sm and dm for those which exhibit respectively the same or a di erent melody/rhythmical pattern as the solo used to train the model, mono and stereo for those rendered respectively in mono or stereo. "**" denotes high (p < 0.001), "*" good (p < 0.01), and "n. tion has a too high number of features K, as it corresponds to the number of frequency bins (in our experiments 512), which might be too complex for the AAD task. Also for the TF descriptors, many misclassi cations happen when the reconstructed stimulus quality is low, i.e., when both r attended and r unattended coe cients are very low (r < 0.2) and close (see Figure 3.8). Here the model is accounting for e ects which are probably more related to the whole mixtures than individual instruments. When r attended is high, usually the corresponding r unattended is low, meaning that the model is discriminating the two instruments.

N

As expected, the number of instruments in the mixtures seems to a ect the performances, which are better for duets than trios as can be seen in Table 3.2. Some previous works on AAD applied to speech [START_REF] Fuglsang | Noise-robust cortical tracking of attended speech in real-world acoustic scenes[END_REF][START_REF] Das | EEG-based auditory attention detection: boundary conditions for background noise and speaker positions[END_REF] showed that the attention task is more challenging for the listener with an increasing number of sources and noise levels. In practice, high noise levels can impact the listener's ability to segregate the source of interest leading to poor decoding quality. We can assimilate a multi-instrumental musical piece to a particularly complex multispeaker environment. The more instruments we have, the more di cult is the attention task. In music, this problem can also be related to how much the attended instrument is in the foreground, i.e., to its predominance.

Nevertheless, also the results for trios are still statistically better than chance (considering that the chance levels are 50% for duets, 33% for trios). The MEL descriptor is particularly robust to this variant, both for the F1 scores and r attended (duets median r = 0.12, trios median r = 0.11 vs the one of the MAG which is r = 0.22, trios median r = 0.17).

S

The stimuli were played to the subjects with two possible spatial renderings: one where both instruments are in the centre denoted as mono modality, and one where the instruments are spatialized, denoted as stereo. Intuitively, the stereo setting should help the subject in focusing on the target instrument as it makes it easier to localize it, leading to a better reconstruction of its features and nally giving a better decoding performance. However, it seems that the spatial rendering does not signi cantly a ect neither the decoding performance nor the correlation values, with the di erences not being statistically signi cant (p > 0.05, non-parametric Wilcoxon test). More data and experiments are needed to verify this hypothesis.

M

The genre, instead, is highly in uencing the performances. Both the TF descriptors behave much better for the Classical music mixtures compared to Pop ones as can be seen in Table 3.2 and Figure 3.7. This probably happens because the nature of the Pop excerpts used as stimuli is mostly repetitive musical patterns, which are essentially rhythmical. In our dataset, this is particularly true in mixtures with the drums and the bass, which usually have to guide the rhythm. The Classical mixtures used are inherently di erent: they exhibit long melodic lines which can be translated in well-de ned varying pitch contours. Thus, the very good performances on the classical pieces can be explained by the fact that our model is tracking well the pitch/harmonic contour of the attended instrument. Usually, when one attends to an instrument one focuses on following the melody line or rhythm played.

That is why we tested if our models are invariant to the melody/rhythmical pattern that is played. In fact, the performance clearly changes when we test the models on di erent musical pieces from those which were used for training, and is better when the melody/rhythmical pattern remains the same. It is worth clarifying that even in this case, though the same solo excerpt is used during training and testing, during the latter, that solo excerpt is played as part of a mixture (duet or trio) and the EEG response is obviously completely di erent from that of the training with the solo-only stimulus. This performance degradation observed when the pitch contours vary between training and testing is coherent with the explanation we gave before for the di erence of performance among the genres. However, this also means that the generalization ability of the considered models is limited. Even if the models are not invariant to the changing pitch contour, the performance still remains signi cantly better than chance for the two TF representations. In this case, the linear spectrogram seems to be more robust than the Mel one. The lower performance on the Pop excerpts can be explained also by the fact that the drums are always misclassi ed as bass. Our tentative explanation is that when the subject is listening to the drums and the bass, the brain's activity is mostly tracking the rhythm. More experiments using recordings with clearer distinction between melody and rhythm will be needed to con rm these initial ndings.

C

In this Chapter, we investigated for the rst time the problem of AAD to a target instrument in polyphonic music based on the continuous EEG response. This study is critical within the thesis, as the proposed neuro-steered Music Source Separation approaches are built upon the results of this Chapter.

We conducted an extensive evaluation on the MAD-EEG dataset analysing the in uence on the performance of multiple variants of musical stimuli, such as the number and type of instruments in the mixture, the spatial rendering, the music genre and the melody/rhythmical pattern that is played. We considered three di erent acoustic representations: the amplitude envelope, the magnitude and the MEL spectrograms.

Stimulus reconstruction based on a simple linear regression model yields promising results for decoding the attended instrument. Through experimental evaluation, we have shown that the EEG tracks musically relevant features which are highly correlated with the TF representation of the attended source and only weakly correlated with the unattended one making it possible to decode the auditory attention towards a speci c instrument in the mixture. This contrast is particularly signi cant when using TF audio representations, highlighting amplitude modulations in di erent frequency bands. Among the two TF representations, the more compact and perceptually scaled representation given by the MEL spectrograms appears to be more robust to highlight the contrast. We have shown that we are tracking attention since these features are related to the attended source and not the mixture as a whole. However, it seems that the models are mostly tracking the instrument's pitch contour, which reduces its generalisation capabilities.

The main limitation is that this approach employs the separate "clean" sources of each instrument present in the mixture (to correlate their feature representation to the one predicted with the EEG data). This condition is never met in realistic music listening scenarios where only the mixtures are available. Moreover, the linear model is not tracking all the non-linearity of the EEG signals.

C-NMF

I

In this Chapter, we propose an unsupervised Nonnegative Matrix Factorization (NMF) variant, named Contrastive-NMF (C-NMF), that separates a target instrument, guided by the user's selective auditory attention to that instrument, which is tracked in his/her EEG response to music. Speci cally, we exploited the "contrast" among the sources that can be extracted from the neural response using a decoding model.

From the experiments presented in Chapter 3, we know that the reconstruction of the audio modulations we can get from the EEG is more correlated with 43 those of the attended instrument than with those of the unattended one. We observed that this reconstruction is highly deteriorated but still "good enough" to discriminate between the attended and unattended sources. These two facts can be naturally exploited in an informed NMF-based sound source separation system, where the sources are decomposed into spectral patterns and corresponding activations.

Our proposal is then to reconstruct the attended source's activations from the EEG using the backward model introduced in Chapter 3. Indeed, the NMF activations can be seen as modulations across time of speci c spectral patterns found by the factorisation. Thus, they will represent a rough approximation of the TF representations used in the experiments in Chapter 3.

One advantage over other source separation models is that NMF allows to incorporate additional information about the sources directly in its optimisation cost without requiring a data-intensive training phase. The additional information at our disposal is represented by the attended source's temporal activations for a given set of spectral patterns representing that source reconstructed from the EEG. Since those reconstructed activations are signi cantly deteriorated, it is hard to use them directly. Nevertheless, these reconstructions are good enough to discriminate the attended instrument from the unattended one. In the proposed C-NMF, this "contrast" is used to guide the separation. The factorisation and the decoding are learnt jointly, allowing for adapting both models to the speci c test mixture and leading to encouraging results.

The main advantage of the C-NMF formulation is that it allows us to reformulate the AAD problem without access to the ground truth sources, paving the way for real-life applications. The attended instrument is the one that is automatically separated by the separation system thanks to the contrast.

We conduct an extensive evaluation of the proposed system on the MAD-EEG dataset analysing the impact of multiple aspects of the musical stimuli, such as the number and type of instruments in the mixture, the spatial rendering and the music genre, obtaining encouraging results, especially in di cult cases where non-informed models struggle.

R

The AAD task is naturally related to audio source separation. As previously explained in Chapter 3, the decoding paradigm requires access to the ground truth sources, to correlate them to the neural data. However, this situation is never met in realistic scenarios such as hearing aids and cochlear implants, where only the mixture of the sound scene recorded by their microphones is available. In such scenarios, an additional audio source separation step is needed to extract the reference sources needed for the decoding. Typically the separation and the decoding tasks are tackled sequentially: a separation system provides the reference sources for the decoding, and the decoding system selects the source which needs to be enhanced.

Most of the studies that relate speech source enhancement and AAD have worked in this direction. Many of them focused on the multi-channel audio scenario using beamforming [START_REF] Aroudi | EEG-based auditory attention decoding using steerable binaural superdirective beamformer[END_REF][START_REF] Aroudi | Cognitive-driven binaural LCMV beamformer using EEG-based Auditory Attention Decoding[END_REF][START_REF] Aroudi | Improving auditory attention decoding performance of linear and non-linear methods using state-space model[END_REF] and multi-channel Wiener ltering [START_REF] Van Eyndhoven | EEG-Informed Attended Speaker Extraction From Recorded Speech Mixtures With Application in Neuro-Steered Hearing Prostheses[END_REF][START_REF] Das | EEG-based attention-driven speech enhancement for noisy speech mixtures using N-fold multi-channel Wiener lters[END_REF]Das et al. 2020b] as hearing aids can be equipped with a microphone array. Both techniques estimate spatial lters that return the target speech when applied to the mixture while suppressing the background noise and interfering sources. These approaches use spatial information such as the directions of arrival and the target activity to compute the second-order statistics of the noise and interferers. One of the main limitations lies in estimating the spatial location and the voice activity, which may be di cult in challenging scenarios (e.g., overlapping speakers in space, time or frequency, high reverberation, moving speakers).

Other works focus on the single-channel scenario using DL-based approaches. O'Sullivan et al. were the rst along this line [START_REF] O'sullivan | Neural decoding of attentional selection in multi-speaker environments without access to separated sources[END_REF]]. However, their model requires prior training on the target speakers, which is a substantial limitation in real scenarios. The problem is tackled by Han et al. with a speakerindependent source separation system able to generalize to unseen speakers [START_REF] Han | Speakerindependent auditory attention decoding without access to clean speech sources[END_REF]]. Such a system relies on a deep attractor network, which projects the mixture's time-frequency representation in a high-dimensional space where the speakers are separable [START_REF] Chen | Deep attractor network for single-microphone speaker separation[END_REF][START_REF] Luo | Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF]]. The main di erence with the deep clustering (DC) approach [START_REF] Luo | Deep clustering and conventional networks for music separation: Stronger together[END_REF], is that the DNN is trained end-to-end to estimate a mask, while in DC a postclustering step on the embedding is required, giving an advantage in terms of separation performance. Ceolini et al., instead, informed a speech separation neural network with the decoded attended speech envelope, leading to the extraction of the attended source [START_REF] Ceolini | Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception[END_REF]]. However, the training of the source separation model and that of the AAD model are still decoupled, due to the lack of large datasets collected for AAD.

In general, performing the source separation and AAD steps independently is sub-optimal. In their work Pu et al. propose a uni ed model for joint AAD and binaural beamforming [START_REF] Pu | A joint auditory attention decoding and adaptive binaural beamforming algorithm for hearing devices[END_REF]]. An adaptive beamformer is learned thanks to an objective which minimizes noise and interference but, at the same time, controls the target speaker distortion and maximizes the Pearson correlation coe cient (PCC) between the envelope of the beamformer output and the decoded EEG. In a later work [START_REF] Pu | Evaluation of Joint Auditory Attention Decoding and Adaptive Binaural Beamforming Approach for Hearing Devices with Attention Switching[END_REF]], the same authors showed that their algorithm is robust to attention switching, which can be tracked in real-time thanks to the joint approach.

In this work, we pursued the joint approach and propose to adapt an NMFbased source separation model to a speci c mixture using a weak signal decoded from the EEG using an AAD model. The AAD model is not xed and is as well updated during the optimization. Our work di ers from those by Pu at al. [START_REF] Pu | A joint auditory attention decoding and adaptive binaural beamforming algorithm for hearing devices[END_REF]] as our aim is not to maximize the PCC between the envelope of the beamformer output and the decoded EEG. Since the decoded output can be signi cantly deteriorated (see Chapter 3), we leverage instead the fact that the attended instrument's neural encoding is substantially stronger than the one of the unattended sources left in the mixture. This "contrast" is maximized when solving our separation model estimation problem.

M

The goal is to separate a target instrument from a given music mixture. Along with the audio signal, we have access to the EEG recorded while the subject was listening to the given mixture and attending to the target instrument. From this signal we can reconstruct the attended source's activations from the EEG using a backward model. Those reconstructed activations are signi cantly deteriorated but create a "contrast" that allows to discriminate the attended instrument from the unattended one.

In the proposed C-NMF, this "contrast" is used to guide the separation. The factorisation and the decoding are learnt jointly. The target instrument's activations are reconstructed from the multi-channel EEG at rst using a pre-trained backward model. Then they are used to guide the mixture's factorisation and cluster the components into the respective sources. At the same time, the decoding model is updated every certain number of NMF iterations to adapt to the observed signal. A good initialisation of the decoder can be learned from a small training set of solos and corresponding EEG recordings from the same subject.

NMF-based audio source separation

The proposed Contrastive-NMF (C-NMF) is a novel variant of Nonnegative Matrix Factorization, a technique for data decomposition which has been very popular in many audio inverse problems such as source separation, enhancement or transcription as it is able to unmix superimposed spectral components [START_REF] Févotte | Single-channel audio source separation with NMF: divergences, constraints and algorithms[END_REF]]. Among other factorization techniques (e.g., Principal Component Analysis (PCA), Independent Component Analysis (ICA)), NMF distinguishes itself through its nonnegativity constraints which lead to a partbased representation of the data that is interpretable [START_REF] Lee | Learning the parts of objects by non-negative matrix factorization[END_REF]. In Figure 4.2 one can see an example of NMF decomposition of a single source audio signal.

In the case of single-channel audio source separation, one can assume that an audio signal x(t) at time sample t is given by the linear mixture of J sources s j (t):

x(t) = J j=1 s j (t). (4.1)
Observing the mixture x(t), a source separation system aims to recover one or more sources s j (t) of interest. Such a mixture can be represented in matrix form through its magnitude spectrogram X ∈ R M ×N + , where M represents the number of frequency bins and N the number of Short Time Fourier Transform (STFT) frames.

X can be factorized into two unknown matrices W and H such that X ≈ WH, where the columns of W ∈ R M ×K + are interpreted as non-negative audio spectral patterns, expected to correspond to di erent sources and the rows of H ∈ R K×N + as their temporal activations. Usually, one refers to W as the dictionary and to H as the activation matrix. When K, namely the rank of the factorization, is much smaller than M , WH represents a low-rank The time-frequency matrix X is approximated as a product of two non-negative matrices W and H having a much smaller rank. W can be seen as a dictionary of spectral components representing elementary sound units (notes, chords, percussive sounds) and H as their activations.

approximation of the data matrix X [START_REF] Févotte | Single-channel audio source separation with NMF: divergences, constraints and algorithms[END_REF]]. The factorisation can be achieved by minimizing a cost function as the following:

     C(W, H) = D(X|WH) audio factorization + µ H 1 + β W 1 sparsity W, H ≥ 0. (4.2)
Usually, for the mixture reconstruction β-divergences are used, which have been very popular for audio inverse problems. It is also common to impose a sparsity constrain on both W and H using an 1 regularization controlled by the hyperparameters µ and β, respectively, to improve the source modelling. In fact, music is often given by a repetition of a few audio patterns, thus we can easily assume that the activations are sparse [START_REF] Vincent | Audio source separation and speech enhancement[END_REF]. The same can be assumed for the spectral patterns as there is only a low probability that two given sources are highly activated in the same set of frequency bins [START_REF] Yilmaz | Blind separation of speech mixtures via time-frequency masking[END_REF].

At this point, the separation problem reduces to the assignment of each NMF component to the corresponding source j. Then, the complex-valued spectrogram S j of each source can be estimated by Wiener ltering as [START_REF] Févotte | Single-channel audio source separation with NMF: divergences, constraints and algorithms[END_REF]:

Sj = W j H j WH ⊗ X, (4.3)
where the element-wise division (W j H j )/(WH) is the soft mask associated to source j and X is the complex spectrogram of the mixture. ⊗ denotes an element-wise multiplication. What we have described so far is the so-called unsupervised NMF, i.e., a blind signal decomposition where both the dictionary and the activations are estimated from the mixture [START_REF] Févotte | Single-channel audio source separation with NMF: divergences, constraints and algorithms[END_REF]]. However, in real music compositions a source plays several notes with di erent pitches and it might be hard to represent it with a single component. Moreover, two sources may be represented by similar components as they might overlap and be highly correlated. Therefore, the component assignment might be hard and requires speci c classi cation or clustering techniques to group together components associated to the same source. In such a complex situation, the factorization needs to be "guided" by incorporating prior information about the sources to return a meaningful representation [START_REF] Vincent | From blind to guided audio source separation: How models and side information can improve the separation of sound[END_REF].

Starting from the unsupervised formulation, one can incorporate prior knowledge directly in the optimisation cost, e.g., through hard or soft constraints, speci c regularizers, pretrained dictionaries, or forcing the elements of W and/or H to follow a given distribution [START_REF] Vincent | Audio source separation and speech enhancement[END_REF]. For example, in the case of music, it is possible to impose properties like harmonicity of the spectral patterns or smoothness and sparsity to the activations [START_REF] Vincent | Audio source separation and speech enhancement[END_REF]]. Particularly interesting is the multimodal scenario, where one has access to multiple views of the same phenomenon (e.g., video, motion capture data, score) which are synchronized with the audio. Seichepine et al. [START_REF] Seichepine | Soft Nonnegative Matrix Co-Factorization[END_REF], for instance, propose to impose the equality (hard constraint) or the similarity (soft constraint) of the source activations in the two modalities. This is not applicable in our case as the time activations we can reconstruct from the EEG are very deteriorated, making it hard to use them directly. Nevertheless, these reconstructions are "good enough" to discriminate the attended instrument from the unattended one, leading to a "contrast" that can guide the separation.

A novel NMF variant: Contrastive-NMF (C-NMF)

The general idea of discriminating sources according to some criterion for NMF-based audio source separation was already explored in the past but most of the proposals refer to fully supervised or semi-supervised scenarios, where the basis functions are learned in a training phase. Weninger et al. and Kitamura et al. propose to learn basis matrices that are as much discriminative as possible to have unique spectral templates for each source [START_REF] Weninger | Discriminative NMF and its application to single-channel source separation[END_REF][START_REF] Kitamura | Discriminative and reconstructive basis training for audio source separation with semi-supervised nonnegative matrix factorization[END_REF]. Grais and Erdogan propose to minimize the cross-coherence between dictionaries belonging to di erent sources [Grais and Erdogan 2013], while Chung et al. to learn a factorization so that each basis is classi ed into one source [START_REF] Chung | Discriminative training of NMF model based on class probabilities for speech enhancement[END_REF]]. Kumar et al., in a di erent application setting, propose a max-margin framework, where the projections are learned to maximize an Support Vector Machine (SVM) classi er's discriminative ability [START_REF] Kumar | Max-margin non-negative matrix factorization[END_REF].

Within this work, instead, the projections are learned by an unsupervised NMF to maximize the discrimination ability of a decoding model. Speci cally, the proposed cost aims at decomposing the audio spectrogram while maximizing the similarity of the EEG-derived activations with the audio-derived ones for the target source and minimizing it for the interference sources. Thanks to this formulation, the components resulting from the decomposition should already be clustered into the target and interference sources.

Let us analyze the novel cost function. Considering a mixture x(t) given by the linear mixing of the attended source s a (t) and some interferers s u (t), let W a ∈ R M ×Ka + be a sub-dictionary of W containing a set of basis vectors representing source s a (t) and H a ∈ R Ka×N + be their activations. H a can be roughly approximated by S a ∈ R Ka×N + reconstructed from the time-lagged EEG response R, 1 the assumption being that it is likely to be more correlated 1 There is no constraint on the backward model such that the reconstructed activations Sa are non-negative because the assumption behind a linear regression model is that the output variable follows a Gaussian distribution. To ensure that the reconstruction is non-negative, a generalized regression model should be used, where the output variable is constrained to follow an inverse-Gaussian or Gamma distribution. In our case, we observed that the negative values in the reconstructions had small amplitude and were similar to noise, thus we set them directly to zero.

with the NMF-derived activations of the attended source H a than with the ones of the interferers H u . This contrast can be integrated in the unsupervised NMF cost function as follows:

                     C(W, H) = D KL (X|WH) audio factorization + µ H 1 + β W 1 sparsity + -δ( H a S T a 2 F -H u S T a 2 F ) contrast W, H, S a ≥ 0 h k: 2 = 1, s k: 2 = 1. (4.4)
where D KL (•|•) is the Kullback-Leibler divergence, µ and β are regularization parameters and δ is a parameter weighting the contrast term. h k: and s k: represent the rows of H and S a respectively and are normalized to have unit 2 norm in order to minimize the e ect of a scale mismatch between the modalities.

We derived the update rules for H and W using the Multiplicative Updates (MUs) heuristic, which is based on gradient descent [START_REF] Févotte | Algorithms for nonnegative matrix factorization with the β-divergence[END_REF]. A solution is searched by moving in the direction opposite to the gradient's: W and H are updated alternately according to a scheme called block-coordinate descent: each variable is updated assuming the other to be constant. The learning rate is adaptively chosen so as to have multiplicative updates, which cannot generate negative elements when starting from positive values [START_REF] Lee | Learning the parts of objects by non-negative matrix factorization[END_REF]. The same algorithm can be derived using the heuristic proposed Févotte and Idier, which consists in computing the gradient of the cost ∇C(θ), splitting it into its negative and positive parts, i.e., writing ∇C(θ) = ∇ θ + C(θ) -∇ θ -C(θ), and building the rules as follows [START_REF] Févotte | Algorithms for nonnegative matrix factorization with the β-divergence[END_REF]:

θ ← -θ ⊗ ∇ θ -C(θ) ∇ θ + C(θ) (4.5)
With θ = {W, H}, the update rules can be computed as:

W ← -W ⊗ ∇ W -C(W, H) ∇ W + C(W, H) (4.6) H ← -H ⊗ ∇ H -C(W, H) ∇ H + C(W, H) (4.7)
The cost function in Eq. ( 4.4) is completely separable, therefore, one can compute the gradient for the Kullback-Leibler divergence, the sparsity con-straints and the contrast term separately. The derivation of the update rule for W is trivial because it does not involve the contrast term: 2 2 For the detailed derivation, please refer to § 6.2 in the appendices.

W ← -W ⊗ (Λ -1 ⊗ X)H T 1H T + β (4.8)
where ⊗, divisions and exponents denote element-wise operations, 1 is a matrix of ones whose size is given by context and Λ = WH. The update rule for H requires more attention. In fact, in the contrast term we have the two matrices H a and H u which are respectively the activations of the attended and interference sources. Thus, the gradient with respect to H of the contrast term, will be equal to the gradient computed with respect to H a for the rst K a rows and equal to the gradient computed with respect to H u for the remaining rows:

H (-δ( H a S T a 2 F -H u S T a 2 F )) = -2δH a S T a S a , if 1 < k < K a +2δH u S T a S a , if K a + 1 < k < K (4
.9) leading to the following update rule:

H ← -H ⊗ W T (X ⊗ Λ -1 ) + δP - W T 1 + µ + δP + (4.10)
where P -, P + ∈ R K×N + are auxiliary matrices de ned as:

P -= H a S T a S a , if 1 < k < K a 0, if K a + 1 < k < K (4.11) P + = 0, if 1 < k < K a H u S T a S a , if K a + 1 < k < K.
(4.12)

The derived update rules for H and W are given in lines ( 10) and ( 12) of Algorithm 1 respectively. This pseudo-code provides all the details of the algorithm, including also the update of the decodi model g. In fact, the factorization and the decoding are learnt jointly to improve the source modelling for both the source separation and AAD tasks. Speci cally, the target instrument's activations S a are rst reconstructed from the time-lagged EEG data matrix R using a pre-trained backward model g. Then those activations are used to guide the mixture's factorization and cluster the components into the respective sources obtaining two submatrices W a and H a associated with the attended source. At the same time, the decoding model R is updated every certain number of NMF iterations to adapt to the observed signal using W a as a new feature extractor. After convergence, the dictionary and the activations related to the attended source are used to obtain the Wiener lter mask. The complete pipeline is depicted in 4.4: Proposed scheme: the target instrument's activations are reconstructed from the listener's multi-channel EEG using a pre-trained backward model. They are then used to guide the mixture's factorisation and cluster the components into the respective sources (C-NMF). At the same time, the decoding model is updated every certain number of C-NMF iterations to adapt to the observed signal. After convergence, the dictionary and the activations related to the attended source are used to obtain the WF soft-mask.

Algorithm 1: The experiments are designed to evaluate if the EEG information helps the separation process. However, to verify that the improvement is due to the EEG and not to the cost function's discriminative capacity, it was not enough to have the blind NMF as the only baseline. Therefore, we built a second baseline which consists of the C-NMF to which meaningless side information is given.

Contrastive NMF pseudo-code input : X, R, µ ≥ 0, β ≥ 0, δ ≥ 0, γ ∈ [0, 1] output : W a , H a 1 W, H, g initialization 2 H ← diag( h 1: -1 , ..., h K: -1 )H normalization 3 W ← W diag( h 1: , ..., h K: ) re-scaling 4 Λ = WH 5 repeat 6 S a ← g T R 7 S a ← diag( s 1: -1 , ..., s K: -1 )S a 8 repeat 9 P ← [-H a S T a S a , H u S T a S a ] T 10 H ← H ⊗ W T (X⊗Λ -1 )+δP - W T 1+µ+δP + 11 H ← diag( h 1: -1 , ..., h K: -1 )H 12 W ← W diag( h 1: , ..., h K: ) 13 Λ = WH 14 W ← W ⊗ (Λ -1 ⊗X)H T 1H T +β 15 Λ =
The meaningless side information consists of random activations sampled from a Gaussian distribution. To summarise, we tested three models:

1. Blind NMF (NMF); 2. Contrastive NMF + Random side activations (C-NMF-r);

Contrastive NMF + EEG-derived activations (C-NMF-e).

As the models are entirely unsupervised, the factorised components need to be assigned to each source before applying the multi-channel Wiener lter. In the two baselines, the components are clustered according to their Melfrequency cepstral coe cients (MFCCs) similarity. The di erent pipelines are depicted in Figure 4.5.

In the case of the C-NMF-e, the EEG information automatically identi es and gathers the target instrument components. Thanks to this we can reformulate the AAD problem exposed in Chapter 3, where we had access to the ground truth sources, di erently. This time, the instrument which is predicted as being the attended one is the one that is automatically separated by the proposed source separation system. Speci cally for our formulation, the attended instrument is the one represented by the W a dictionary and H a activations. For each method, NMF is run for 400 iterations while the backward model is updated every 100 iterations of the C-NMF-e. For each method, the initialization of W and H is obtained by applying a blind NMF to the mixture for 200 iterations. For a given mixture, the initialization of the three models is the same to guarantee a fair comparison. As a reconstruction cost, we chose the Kullback-Leibler divergence. We learned a good initialization of the backward model from a training set of solos (di erent from the ones used in the test mixtures) and corresponding EEG recordings for each subject and instrument. The Ridge parameter is set to be γ = 0.1 and the considered temporal context is [0, 250]ms post-stimulus as done in the experiments of Chapter 3.

Evaluation

The models are evaluated using a standard metric in music source separation, i.e., the Signal to Distortion Ratio (SDR) expressed in decibel (dB) and computed using BSSEval v4 [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF][START_REF] Stöter | The 2018 signal separation evaluation campaign[END_REF] 3 . Sometimes, we 3 https://github.com/sigsep/bsseval will refer to the Signal to Distortion Ratio Improvement (SDRi) over the input, as some mixtures may be more or less di cult than others. The metric is computed over the whole length of each music excerpt (around 24 seconds). In the tables below are reported median values. To assert the statistical signi cance of our model's improvement over the baselines, we opted for a non-parametric Wilcoxon test on the metrics' linear values. The considered signi cance levels are 5%, 1%, 0.1% and 0.01%.

Beside the separation quality, we also evaluate the decoding performance in terms of accuracy on the AAD task as done in Chapter 3. However, here the AAD problem is formulated di erently: the instrument that is automatically separated by the proposed source separation system, i.e., the one represented by the W a and H a , is predicted as being the attended one. The statistical signi cance was assessed using an adaptation of the computationally-intensive randomization test [START_REF] Noreen | Computer-intensive methods for testing hypotheses[END_REF]] already introduced in Chapter 3. The considered signi cance levels are 5%, 1%, 0.1% and 0.01%, and the tests were performed over 10 4 iterations. For details about the test, please refer to § 6.2.

It is worth noting that given the user-driven nature of the EEG-driven separation system, the performance, both in terms of separation quality and decoding performance, not only depends on the algorithm but also on the subject's ability to properly attend to the target instrument. Similarly, the decoding performance now depends not only on the decoding model and the subject's ability and attention as in Chapter 3 but also on the separation model and the di culty of the mixture to be separated.

Experimental results

S

In Table 4.1, one can see the median SDR values for di erent methods, instruments and, in the last two rows, di erent spatial renderings. As far as spatial rendering is concerned, it is important to keep in mind that the audio signal processed by the source separation system is always mono (i.e., the task is single-channel audio source separation). The "mono" and "stereo" results relate to the way the stimuli were played to the subjects (which we will refer to as spatial rendering) which di erently a ects Looking at Table 4.1, it is immediate to see that the contrast derived from the EEG can improve the separation quality for all the pop instruments, especially when separated from duets. Particularly signi cant is the improvement over the blind baseline (NMF) for the drums (more than 7 dB).

It is also clear that the proposed model needs to be fed with meaningful side information and that the activations reconstructed with the backward model are indeed meaningful. In fact, the same model informed with the random side information (C-NMF-r) performs signi cantly worse than the one fed with the EEG-derived contrast (drums and bass p < 0.0001, guitar p < 0.01, singing voice p < 0.05, Wilcoxon test). In general, the C-NMF-r model introduces lots of artefacts, even without removing the interferers. Moreover, the random side information can even fool the factorization leading to a degradation of the performance w.r.t. the blind NMF. Only in some rare cases (e.g., vocals, drums, and cello), even with the random information, the proposed approach "guides" the separation indirectly by imposing that the H a and H u activations are di erent, leading to a little improvement over the blind NMF.

The situation is di erent for Classical music instruments, where the improvement over the baselines is statistically signi cant only for the oboe's separation from duets and the bassoon's separation from trios. However, this is not in contrast with the results obtained in Chapter 3 where the decoding performances were better for Classical music instruments than pop ones because here we observe also the e ect of the separation system and of the di culty of separating certain mixtures which are dominant factors. The blind NMF is already obtaining a good separation (see NMF results for Classical music instruments in Table 4.1), as the Classical music mixtures of the MAD-EEG dataset can be too easy to separate (e.g., high/low pass lter), and the EEG information helps especially in di cult cases, where the baselines su er from the task's complexity. An explicative example of such cases is represented, for instance, by the separation of the drums where the proposed method is signi cantly better than the baselines. For easy mixtures, instead, it is hard to see the bene cial e ects of the additional information w.r.t. the baseline. In any case, the baseline's results are overall not signi cantly better than the one of the proposed model (p > 0.05, Wilcoxon test).

We remark that the results in 4.6: Signal to Distortion Ratio (SDR) expressed in dB for di erent instruments in the dataset. In di erent colors are underlined di erent methods. To assert the statistical signi cance of the proposed model with respect to the baselines we opted for a non-parametric Wilcoxon test. The hypothesis test was performed on the linear SDR. "***" denotes high (p < 0.001), "**" good (p < 0.01), "*" marginal (p < 0.05) and "n.s. " no (p > 0.05) statistical signi cance.

10, and δ = 10 4 , set of values which was found to give good overall results. However, we observed that speci c instruments and mixtures would need a speci c hyperparameter tuning to maximize the performance. To give an example, by only reducing the value of µ from 10 to 1 when separating the oboe from trios, one can improve the SDR by more than 4 dB. This data-dependent behaviour of NMF scheme's hyperparameters was previously observed [START_REF] Parekh | Guiding audio source separation by video object information[END_REF] and can be mitigated by allowing a user of the system to adjust the hyperparameter values typically through a knob/slider.

S

The stimuli were played to the subjects with two possible spatial renderings: one where both instruments are in the centre denoted as mono modality, and one where the instruments are spatialized, denoted as stereo. The last two rows of Table 4.1 show the results for these two di erent cases for all the instruments in the dataset. The results are di erentiated w.r.t. the number of instruments in the mixture, and all values are medians over the test set. Intuitively, the stereo setting should help the subject in focusing on the target instrument as it makes it easier to localize it, leading to a better reconstruction of its activations and nally giving a better separation. However, as in Chapter 3, we did not observe statistically signi cant di erences between the two conditions except for a few pop instruments when listened to in duets (guitar p < 0.01, singing voice p < 0.001, drums and bass p < 0.05, Wilcoxon test). In all the other cases, we cannot make any statement (p > 0.05, Wilcoxon test).

Guitar

I

Part of the high variance in the SDR performances is because di erent mixtures in the dataset can be more or less di cult for the separation system. However, most of the variance comes from the very high inter and intra-subject variability. The attention task may be more or less di cult for di erent subjects (inter-subject variability), which may depend on factors such as musical training and attention capacity [Di Liberto et al. 2020b]. Simultaneously, one single subject may perform di erently throughout the experiment (intra-subject variability), maybe due to stress and fatigue that a ect the attention level. These e ects are evident in Figure 4.7, where the SDR results for duets are di erentiated according to the participants involved in the experiment and the target instrument. Looking at Figure 4.7, one can realise that for a given instrument di erent subjects may behave very di erently while for other ones they behave similarly. Moreover, for single instruments, subject's performance may span a wide SDR range. For example, regarding Classical instruments, one can observe that the intra-subject variability is generally lower while sometimes there is a clear inter-subject variability. This may be due to the subjects' unfamiliarity with some instruments like the French horn and the bassoon (see Figure 2.4 in Chapter 2). Another factor is that some instruments can be more di cult than others to follow. For instance, instruments like the bass and the drums, which usually guide the rhythm and tempo, are notably more di cult to track, especially for non-professional musicians and this is re ected in the very high inter and intra-subject variability.

A

Even if the SDR improvement is not systematic for all the instruments, the main advantage of the C-NMF-e model is that it gives an automatic clustering of the components and automatically enhances the attended source. Therefore, the instrument that is automatically separated by the proposed source separation system, i.e., the one represented by the W a and H a , is predicted as being the attended one. It is an asset w.r.t. the baselines, which need an additional step to cluster the components and cannot automatically identify the target source.

In duets and the trios. The accuracy is satisfactory and statistically above chance for four instruments: guitar (duo p < 0.01, trio p < 0.0001, randomization test), drums (duo and trio p < 0.0001, randomization test), French horn (trio p < 0.05, randomization test), and cello (duo p < 0.05, trio p < 0.01, randomization test). For some other instruments (singing voice, bass, bassoon, and oboe), the accuracy is much below chance indicating that the contrastive term is always forcing them not to be represented by W a and H a . The reason for this behaviour lies in a non-customized tuning of the δ parameter. We observed, for instance, that δ = 10 4 causes a drop of the performances for the singing voice and the bassoon, which instead were much above chance with δ < 10 4 . As we said previously, this can be easily solved by a customized netuning of the hyperparameters by the user. In the following section we will further analyze the e ect of the hyperparametes on the system performance.

E

We rst analyze the number of NMF components necessary to describe each instrument testing 4 values ({4, 8, 16, 32}). We observe that an increasing number of components improves the separation performance as it allows a more accurate description of the sources. As for the impact of the sparsity constraints imposed on H and W by µ and β, respectively, which in our experiments are set to be equal, we tested 4 values ({0, 0.1, 1, 10}), observing that higher µ and β improve the separation quality as it allows a better source modelling.

Lastly, we tested four reasonable values for δ ({10 1 , 10 2 , 10 3 , 10 4 }), which weights the contrastive term in the C-NMF cost function. We observed that increasing values of δ lead to signi cantly higher SDR for all the tested instruments except for the French horn, for which there is no signi cant di erence (p > 0.05, Wilcoxon test). However, one has to be careful not to chose a too high value of δ, which may push to a trivial solution where the activations of the interferers H u are set to zero and all the sources in the mixture are represented by the W a and H a . This e ect is re ected in the AAD accuracy reported in Figure 4.9, where the performance drops for δ = 10 4 for the vocals and the bassoon. However, this e ect is strictly instrument-dependent as for other instruments like the cello, the decoding accuracy becomes statistically better than chance only with δ = 10 4 (p < 0.0001, randomization test).

Guitar Voice Drums

C

This Chapter describes a novel neuro-steered music source separation framework and conducts an extensive evaluation of the proposed system on the MAD-EEG dataset. The results support the thesis that the EEG can guide and help a source separation system, especially in di cult cases where non-informed models struggle. Our ablation study, where the proposed model is informed with random side information, shows that the C-NMF formulation is not enough by itself but needs to be informed with meaningful side information and that the activations reconstructed with the decoding model may indeed be meaningful.

We could reformulate the AAD problem without needing access to the "clean" audio sources, which are absent in real-life scenarios. In fact, thanks to the C-NMF formulation and the EEG-guidance, the NMF components are clustered into the target and interference sources and the attended instrument is the one that is automatically separated by the separation system.

The EEG-driven C-NMF system has the intrinsic limitation of the subjectrelated variability: if the level of attention of the subject is not su cient, this will inevitably impact the performance. Another factor that needs to be considered is musical expertise and training, which may help the subject while attending to an instrument.

We believe that this NMF variant is advantageous for neuro-steered music source separation. Indeed the available music-related EEG datasets are still costly and time-expensive to acquire, precluding the possibility to tackle the problem with data-driven approaches. Unsupervised NMF represents a powerful approach in such applications where there is no or a limited amount of training data. Moreover, additional information can be easily incorporated into the model cost function directly at test time. However, even if the C-NMF is unsupervised, we need to keep in mind that we still need pairs of EEG and music data for training the backward model.

Moreover, the proposed algorithm can be generalised and used with temporal activations derived from other modalities than the EEG (e.g., video, score, motion capture data) or from a manual annotation provided by the user (e.g., a sound engineer annotating when the source of interest is active).

UGOSA: User-guided one-shot deep model adaptation for music source separation

I

Deep Learning (DL) has profoundly changed the Music Source Separation (MSS) scene of the last years thanks to the appearance of large datasets where the isolated tracks of a set of instruments, usually the most common ones, are available along with the mixture [START_REF] Ra | The MUSDB18 corpus for music separation[END_REF][START_REF] Bittner | Medleydb: A multitrack dataset for annotation-intensive mir research[END_REF]]. As a consequence, most state-of-the-art MSS systems consist nowadays of Deep Neural Networks (DNNs) trained in a fully supervised fashion [START_REF] Stöter | The 2018 signal separation evaluation campaign[END_REF][START_REF] Takahashi | Mmdenselstm: An e cient combination of convolutional and recurrent neural networks for audio source separation[END_REF][START_REF] Stöter | Open-unmix-a reference implementation for music source separation[END_REF][START_REF] Luo | Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF][START_REF] Défossez | Music source separation in the waveform domain[END_REF][START_REF] Hennequin | Spleeter: a fast and e cient music source separation tool with pre-trained models[END_REF][START_REF] Samuel | Meta-learning extractors for music source separation[END_REF][START_REF] Takahashi | D3net: Densely connected multidilated densenet for music source separation[END_REF][START_REF] Choi | LaSAFT: Latent Source Attentive Frequency Transformation for Conditioned Source Separation[END_REF][START_REF] Li | Sams-net: A sliced attention-based neural network for music source separation[END_REF][START_REF] Sawata | All for One and One for All: Improving Music Separation by Bridging Networks[END_REF]. Those models have proven to be extremely powerful, but only when the training data is enough for learning the enormous amount of parameters they have.

However, the availability of a large datasets is not always realized, especially when working on informed MSS where the annotation of the side information is needed and costly to obtain. When the side information is the listener's 61 EEG response, this is particularly true as the acquisition process of such data can be very long and expensive as described in Chapter 2.

As seen in Chapter 4, unsupervised techniques like NMF are ideal in such cases as it is easy to incorporate additional information about the sources directly in the optimization cost without requiring a data-intensive training phase. However, NMF-based MSS has its limitations in terms of separation performances and it is desirable to nd alternative strategies to inform DLbased systems with side information that is available only at test time.

Usually in DL-based informed MSS, the model is learned using both the side information and the audio material (mixtures) to be separated. One may want, instead, to choose a powerful deep model which was trained in a fully supervised fashion for the MSS task only and adapt it to a speci c mixture using the additional information available only at test time. Speci cally, we investigated if it is possible to inform a MSS model based on DL using the time activations of the sources provided by the user at test time.

We propose a User-guided one-shot deep model adaptation for music source separation (UGOSA), where the time activations of the sources provided by the user are used to ne-tune a pre-trained deep MSS model to the speci c test mixture he/she is listening to as in Figure 5.2. The adaptation is made possible thanks to a proposed loss function which aims to minimize the energy of the silent sources while at the same time forcing the perfect reconstruction of the mixture. We underline that the adaptation is one-shot, as it acts on the target song instance only and not on a new dataset as most ne-tuning strategies do.

The activations of the sources can be manually annotated by the user through an interface or, in a more challenging scenario, be directly derived from his/her neural activity using decoding models like the ones presented in Chapter 3. The EEG-guided variant, namely EGOSA, can be seen as a particularly challenging case of UGOSA, where the interface for annotating the data is replaced by a BCI. This approach also allows us to reformulate the AAD problem of Chapter 3 using the separation model estimates instead of the ground truth sources. Even if immature, the results are encouraging and point at promising research directions. While with "ideal" manually annotated activations the preliminary experiments show signi cant improvements with the adaptation, the EEGinformed case is clearly more challenging and needs further re nements, mainly because the MAD-EEG dataset was not ideal for this study.

R

The idea of using time annotations provided by the user to inform a source separation system was already explored in many previous works, mainly based on NMF or NTF [START_REF] Laurberg | Structured non-negative matrix factorization with sparsity patterns[END_REF][START_REF] Ozerov | Multichannel nonnegative tensor factorization with structured constraints for user-guided audio source separation[END_REF]Duong et al. 2014a]. 1

1 The literature review focuses on userguided methods as it is the primary focus of this investigation. In particular, we considered works where the user is in the loop and actively provides additional information to the source separation system to improve its performance via adapting it to a speci c mixture.

Some of them rely on dedicated graphical user interfaces, while others are interactive, where the user can iteratively improve and correct the separation [START_REF] Bryan | An e cient posterior regularized latent variable model for interactive sound source separation[END_REF]Duong et al. 2014b]. Time annotations were also extended to more general TF annotations [START_REF] Lefevre | Semi-supervised NMF with time-frequency annotations for single-channel source separation[END_REF][START_REF] Lefèvre | A convex formulation for informed source separation in the single channel setting[END_REF][START_REF] Jeong | Informed source separation from monaural music with limited binary time-frequency annotation[END_REF][START_REF] Ra | A simple user interface system for recovering patterns repeating in time and frequency in mixtures of sounds[END_REF]] but those require much more expertise and e ort from the user (and a more complicated user interface). There are also some interesting works where the user can hum [START_REF] Smaragdis | Separation by "humming": User-guided sound extraction from monophonic mixtures[END_REF], sing or play [START_REF] Fitzgerald | User assisted separation using tensor factorisations[END_REF]] the source he/she wants to enhance as an example to the source separation system. The user can also provide the fundamental frequency or manually correct it [START_REF] Durrieu | Musical audio source separation based on user-selected F0 track[END_REF][START_REF] Müller | Interactive deep singingvoice separation based on human-in-the-loop adaptation[END_REF] or associate each instrument to a microphone in a multi-channel recording [START_REF] Carlo | Gaussian framework for interference reduction in live recordings[END_REF].

Only a few works use directly the neural activity of the listener to inform a speech separation model [START_REF] Pu | A joint auditory attention decoding and adaptive binaural beamforming algorithm for hearing devices[END_REF][START_REF] Ceolini | Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception[END_REF]. In [START_REF] Pu | A joint auditory attention decoding and adaptive binaural beamforming algorithm for hearing devices[END_REF], the authors propose an adaptive beamformer that reduces noise and interference but, at the same time, maximizes the Pearson correlation between the envelope of its output and the decoded EEG. In [START_REF] Ceolini | Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception[END_REF], instead, a speech separation neural network is informed with the decoded attended speech envelope. Ceolini et al. hacked the problem of having a large EEG dataset for training the network using what they called a "noise training scheme". The model is trained using the ground truth envelopes to which Gaussian noise is increasingly added across the epochs to simulate the deteriorated speech envelopes that will be reconstructed at test time from the EEG [START_REF] Ceolini | Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception[END_REF].

Within this work, we explore if adaptation is bene cial for DL-based MSS models, as nowadays, most state-of-the-art models are based on a fully datadriven approach without adaptation [START_REF] Défossez | Music source separation in the waveform domain[END_REF][START_REF] Luo | Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF][START_REF] Stöter | Open-unmix-a reference implementation for music source separation[END_REF][START_REF] Stöter | The 2018 signal separation evaluation campaign[END_REF].

Considering the case of user-guided audio source separation based on DL, we observed that the additional information provided by the user is needed in some form also at training time, constraining the architecture and precluding the use of DNNs already pre-trained on other tasks and/or datasets. In the work of Nakano et al., the model was initially trained for both singing voice separation and fundamental frequency estimation and was then adapted using the F0 loss only [START_REF] Müller | Interactive deep singingvoice separation based on human-in-the-loop adaptation[END_REF]]. In the work of Ceolini et al. the network is built to take as input the amplitude envelope of the sources along with the mixture [START_REF] Ceolini | Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception[END_REF]].

In our case, instead, we are interested in a more general framework, where the DNN is trained on the source separation task only, and the activations are used solely for the adaptation. This approach is general since it allows for adapting any DL-based source separation model, using the activations of the target song instance only.

M

The scope of this work was to investigate if it is possible to adapt a pre-trained DNN for MSS to a particular music piece using the time annotations provided manually by the user, or, in a more challenging scenario, derived from his/her neural activity, no matter which model is used. To this aim, we choose a state-of-the-art MSS model working in the time-domain whose pre-trained weights were made available, and we study ne-tuning strategies using a new loss function we propose which makes use of those time annotations.

Proposed adaptation loss

In supervised training of a MSS model working in the time-domain, the mixture is provided as input; the model outputs the estimated sources which are then compared to the original sources used to create the mixture. The di erence between the estimated and the original sources is used to update the model parameters during training. Typically, an 1 or 2 loss is adopted, which respectively represents the average absolute error or average mean squared error between waveforms.

In our case, during adaptation, we do not have access to the isolated sources anymore but only to their binary temporal activations. To adapt the weights of the model to the test mixture, we introduce a new loss function based on the binary activations h j (t) (active: h j (t) = 1 / non-active: h j (t) = 0) of each instrument j at sample t.

We consider X t the set of instruments that are present in the mixture x at time frame t. When one instrument is absent, the loss minimizes the 1 -norm of its estimate while at the same time, it forces the perfect reconstruction of the mixture. 2 To improve the readability, the time information will be considered 2 Often, it is desirable to relax the time activations to weak class labels, indicating a given instrument in a speci c time interval. In such a case, it is straightforward to modify the formulation and de ne X as the set of instruments that are present in the mixture segment x, obtaining:

L = 1 T T t=1 [| j∈X ŝj,t -xt| + j / ∈X |ŝ j,t |]
in the subscript.

L = 1 T T t=1   | j∈Xt ŝj,t -x t | + j / ∈Xt |ŝ j,t |   (5.1)
Given the binary activations h j,t of each instrument j at time frame t, this formulation can be implemented as follows:

L = 1 T T t=1       | J j=1 (h j,t • ŝj,t ) -x t | reconstruction loss + λ J j=1 |(1 -h j,t ) • ŝj,t | activations loss       (5.2)
where the total cost is composed by two terms: the rst one concerns the perfect reconstruction of the mixture while the second one the energy minimization of the silent sources. If the instrument is active in a given frame t, then h j,t = 1 and the energy minimization term is 0.

On the contrary, if h j,t = 0, then the energy of ŝj,t is minimized. Only if the instrument is active, it will concur to the mixture reconstruction loss. λ is a hyper-parameter that weights the contribution of the energy minimization term in the total loss.

Model

The source separation model chosen for our experiment is ConvTasnet. This architecture was proposed for single-channel speech separation by Luo and Mesgarani [START_REF] Luo | Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF]] and extended to multi-channel music separation by Défossez et al. [START_REF] Défossez | Music source separation in the waveform domain[END_REF]]. It achieves state-of-theart results in both tasks, and its implementation and weights are publicly available, reasons why this model was a good candidate for our experiments. Nevertheless, we underline that the proposed approach is general and can be applied to di erent deep model architectures working in the waveform domain. ConvTasnet is structured as three main blocks:

• The encoder (E in Figure 5.3) transforms a mixture's segments into a non-negative representation in an intermediate feature space;

• this representation is then used to estimate a mask for each source at each time step in the separation subnetwork (1-M in Figure 5.3);

• the isolated waveforms are nally reconstructed transforming the masked encoder features using the decoder (D in Figure 5.3).

Further details about the model can be found in the original paper [START_REF] Luo | Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF], while for details about its multi-channel implementation for music, please refer to [START_REF] Défossez | Music source separation in the waveform domain[END_REF].

giorgiacantisani/ugosa

A

When adapting a DL model for a new task, it is often useless and counterproductive to ne-tune all the network parameters as, for example, the rst layers extract some general features which might be useful also for the new task. In our case, the adaptation is not performed over a new task but over a speci c instance of the test set. Thus, the task remains the same as the one for which the network was trained. Moreover, the data on which to perform the adaptation is extremely limited (just one mixture), increasing the risk of over tting. Those factors make the choice of parameters to ne-tune critical and will largely in uence the performance.

Let "P" stand for proposed while "B" stand for baseline. "Lx:y" indicates the layers that are ne-tuned (e.g., P-L2:D means that the network is ne-tuned from the second block to the last one using the proposed loss). Please refer to Figure 5.3 for the layer's names. We consider as the main baseline the original

ConvTanset trained on the MUSDB18 training set (B0). Moreover, for each of the proposed ne-tuning strategies, we obtain a speci c baseline B-Lx:y where the model is adapted in an unsupervised manner using the mixture reconstruction loss only and ignoring the activations.

E

The models are evaluated using standard metrics in MSS , i.e., Signal to Distortion Ratio (SDR), Signal-to-Interference Ratio (SIR), and Signal to Artifacts Ratio (SAR) expressed in decibel (dB) and computed using the BSSEval v4 [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF] as in Chapter 4. As the SDR is not de ned for silent frames, the evaluation is done only where the sources are non-silent. 7 7 The SDR is not de ned on silent frames. Thus, we evaluated the system using BSSeval only on active segments, while when the reference source was silent, we evaluated the predicted energy at silence (PES) proposed by the authors of [START_REF] Schulze-Forster | Weakly informed audio source separation[END_REF]]. However, by construction, the system is very good at predicting silence, thus the PES metric was not informative and we ended up considering only the SDR for non-silent frames.

Each tested con guration is evaluated in terms of the median over all tracks of the median SDR, SIR, and SAR over each track, as done in the SiSEC Mus evaluation campaign [START_REF] Stöter | The 2018 signal separation evaluation campaign[END_REF]].

To assert the statistical signi cance of our model's improvement over the baselines and to compare di erent hyper-parameters tuning, we opted for a Wilcoxon test on the linear values of the metrics as in Chapter 4.

Beside the separation quality, we also evaluated the accuracy on the AAD task. The statistical signi cance was assessed using an adaptation of the randomization test [START_REF] Noreen | Computer-intensive methods for testing hypotheses[END_REF]] explained in § 6.2. 5.4: Amount of silence of each instrument throughout the test songs. The red line represents the 25% of silence we manually set for each source. Without any action, the instruments would be almost always activated, making it hard to evaluate the proposed loss function.

Experiment with manually annotated activations

We rst validated the proposed approach in the most straightforward scenario, the one where we assume the user to manually annotate the time activations of each source in the mixture. In order to simulate this situation and work in a controlled setting, the time activations were computed synthetically from the ground truth sources. A rst evaluation was performed on the MUSDB18 dataset [START_REF] Ra | The MUSDB18 corpus for music separation[END_REF], that is, the same dataset on which the model was pre-trained. In particular, we use the rst ten songs of the test set together with the binary temporal activations of each instrument computed in a controlled way to have a clear understanding of how the proposed loss function works and what its weaknesses are.

S

The procedure to obtain the activations is two-fold. First, we manually set to zero each source composing a mixture for one-quarter of the song so as to have at least 25% of silence for each instrument. This step is necessary because otherwise the MUSDB18 test mixtures we wanted to evaluate did not have enough silent parts as can be clearly seen in Figure 5.4. Indeed, what makes the proposed loss di erent from a simple mixture reconstruction loss is the energy minimisation of the silent sources (second term of Eq. ( 5.2)). If there are no silences in the mixture, the activations will provide no additional information, and the adaptation would be completely unsupervised.

This procedure belongs to a data preparation step before computing the framewise activations. For each test mixture, the procedure is as follows:

1. segment the mixture into four segments of equal length, 2. assign each segment to one source, 3. set each source to zero in the assigned segment.

The source to segment assignment (see step 2. above) is performed randomly to avoid systematic bias. The sources are set to zero in the Short Time Fourier Transform (STFT) domain, so as to have smooth transitions in time between silent and non-silent segments thanks to the STFT windowing.

Then, the time annotations were obtained using the same procedure and hyperparameters used to annotate the MedleyDB dataset [START_REF] Bittner | Medleydb: A multitrack dataset for annotation-intensive mir research[END_REF], a music dataset which provides the temporal activations of each instrument. The amplitude envelopes were generated for each source s j,t using a standard envelope following technique, consisting of half-wave recti cation, compression, smoothing, and down-sampling. The resulting envelope a j,t is then normalized to account for overall signal energy and the total number of sources in the mixture. Finally, the con dence c j,t of the activations a j,t of instrument j at time frame t can be approximated via a logistic function:

c j,t = 1 - 1 1 + e γ(aj,t-θ) , (5.3)
where γ = 20 controls the slope of the function, and θ = 0.15 controls the threshold of activation. If c j,t ≥ 0.5, then instrument j is considered active (h j,t = 1) at time frame t. Otherwise, if c j,t < 0.5, it is considered silent (h j,t = 0). No manual corrections were performed on the annotations. An example of the computed envelopes and activations is depicted in Figure 5.5. F 5.5: Detail of the time activations computed synthetically from the ground-truth source of the drums. The audio waveform s j is represented in black, the amplitude envelope a j in red and the binary time activations h j in green.

H

We veri ed the in uence of the hyper-parameter λ on the performances by testing nine di erent values of λ ranging from 10 -4 to 10 4 with a logarithmic step. Those results were obtained on the P-L3:M con guration using a window length of 10 seconds. λ expresses the weight of the term that minimizes the energy of the absent sources in the total cost function. In Figure 5.7 ( rst row) we can see the in uence of the hyperparameter λ on the performances. Only the vocals performances are pretty stable with respect to this parameter with no statistically signi cant di erence in the SDR, SAR and SIR across di erent values of λ (p > 0.05, Wilcoxon test). For the other classes, a higher λ leads to a higher SIR, meaning that the suppression of the interferes is more aggressive. This e ect is particularly evident for the bass, where one has 5 dB of SIR increment, which, however, is not statistically signi cant (p > 0.05, Wilcoxon test). A more aggressive separation is often counterbalanced by a signi cant deterioration of the SAR F 5.6: Example of time activations computed synthetically from the ground-truth sources of the drums (upper plot), and the bass (lower plot) of one song of MAD-EEG. The audio waveform s j is represented in black, the amplitude envelope a j in red and the binary time activations h j in green.

(p < 0.0001, Wilcoxon test), meaning more artifacts, and of the overall SDR (other p < 0.001, drums p < 0.01 and bass p < 0.05, Wilcoxon test).

The performances are not sensitive, instead, to the length of the input segments. The results in the bottom row of Figure 5.7 were obtained on the P-L3:M con guration with λ = 1 for di erent lengths of the input segments. We tested ve di erent lengths from 2 to 10 seconds obtaining no statistically signi cant di erences in the SDR and SAR performances (p > 0.05, Wilcoxon test) except for the class "other", where, with a window below 4 seconds, the SDR and the SAR marginally decreases (p < 0.05, Wilcoxon test). This parameter does not signi cantly in uence the SIR (p > 0.05, Wilcoxon test) except for the vocals, where it signi cantly decreases below 4 seconds (p < 0.01, Wilcoxon test). For all the instruments except the vocals, a longer context seems to be bene cial to reduce the artifacts however the improvement is not statistically signi cant (p < 0.05, Wilcoxon test) except for the other class where the improvement over the 2 seconds case is signi cant (p < 0.0001 with respects to 4 and 6 s and p < 0.01 with respects to 8 and 10 s, Wilcoxon test).

S

In Figure 5.8 one can see the SDR expressed in dB for di erent ne-tuning strategies and instruments in the dataset. Blue bars correspond to models ne-tuned with the proposed loss while orange ones correspond to models ne-tuned using the mixture reconstruction loss only. The red line represents the B0 baseline, i.e., the original ConvTanset trained on the MUSDB18 training set and not adapted at all. We can see how the SDR changes with respect to the block from which we start ne-tuning the network. It is necessary to ne-tune at least from the third block to obtain a signi cant improvement over the baseline B0. We have to keep in mind that ne-tuning starting from a deeper block corresponds to millions more parameters to ne-tune. If the number of such parameters is high, it requires a proportional amount of training data, which in our case is not possible, as the "adaptation" data comes from only one mixture.

The improvement over the baseline is particularly pronounced for the category "other", for which the original baseline B0 was struggling the most. As we said before, this category in the MUSDB18 dataset does not represent a speci c 15.4 5.9 6.3 18.9 6.7 T 5.1: SDR, SIR, SAR expressed in dB: median over frames, median over tracks for di erent ne-tuning strategies and di erent instruments in the dataset. #TP stands for the number of trainable parameters which are ne-tuned during adaptation.

instrument. So, it has much more variability than the other classes which are homogeneous in terms of type of instruments, and the network struggles to nd a common representation for those sounds. Adaptation is then particularly useful in this situation, where we need to adapt to a speci c instrument which may be di erent from the ones seen in the training phase. The vocals are the only instrument where we do not improve over the baseline, indicating that probably this class was already well represented in the training data, leaving small room for improvement.

In general, the deeper we ne-tune, the higher the improvement of the proposed model over the corresponding unsupervised baseline, showing that the activations play an active role in the adaptation and that the improvement over B0 cannot be achieved easily in a completely unsupervised fashion.

Looking at Table 5.1, we can have an insight into the evolution of all the metrics. The SDR improvement is mostly due to a SAR improvement, while at the same time, the SIR drops. This means that there are fewer artefacts than before the adaptation, but at the same time, the interferences are not entirely removed. The only instrument which shows a di erent trend is the bass, for which the SIR and SDR increase and the SAR drops. The bass is the only instrument for which the SIR improves over B0. Separating the bass often corresponds to a low-pass lter and probably ne-tuning allows for better adapting the lter to the register played by the bass in the given piece of music.

Motivated by the observation that the decoder has the general function of going back from the feature to the waveform domain, two other ne-tuning strategies were experimented: one where the decoder weights are frozen during netuning (P-Lx:M) and one where both the decoder and masking blocks are frozen (P-Lx:3). We experimented those variants for all the ne-tuning depths and compared them to the corresponding variants where the network is netuned until the last layer (P-Lx:D). The three variants' performances are not signi cantly di erent, indicating that there is no need to ne-tune the decoder or the masking blocks and giving us an insight into the network functionality.

Experiment with EEG-derived activations

Now that we have validated our approach on the manually annotated time activations, we can move to the experiments that use the EEG-derived ones, which are more deteriorated and imprecise.

The EGOSA approach is depicted in Figure 5.9: the amplitude envelopes of each source are reconstructed from the multichannel EEG using a decoding model like the ones presented in Chapter 3 and then binarised to obtain the binary temporal activations necessary for the adaptation. This approach also allows us to reformulate the AAD problem exposed in Chapter 3 using the separation model estimates instead of the ground truth sources.

We performed this second evaluation on the MAD-EEG dataset, which was extensively presented in Chapter 2. We considered only pop mixtures, as, by construction, the network separates the four classes of instruments "bass", "drums", "vocals" and "others". Note that the category "other" in the case of the MAD-EEG dataset coincides with a speci c instrument, and precisely the guitar.

EEG

of each instrument in the mixture are obtained in two steps. First, the amplitude envelope of each source is reconstructed from the multichannel EEG exploiting the stimulus reconstruction approach explained in Chapter 3. In particular, we estimated subject-speci c reconstruction lters for each instrument in the mixture by training a backward model on EEG response of solos with their amplitude envelopes (computed as in 5.9: EGOSA: the time activations of the sources derived from the user's neural activity are used to ne-tune a pre-trained deep source separation model to the speci c test mixture he/she is listening to. Speci cally, we rst reconstruct the amplitude envelope of each source from the multichannel EEG using a decoding model. Secondly, those amplitude envelopes are binarised according to a threshold to obtain the binary temporal activations. F 5.10: Example of envelopes and binary activations computed synthetically from the ground-truth sources (in green) or derived from the EEG (in this case, of subject 2) using a decoding model (pink) for (from the upper to the lower plot) the drums, bass, vocals and guitar of one song of MAD-EEG. The audio waveform s j is represented in black, the amplitude envelopes a j with a continuous curve and the binary time activations h j with a colored region. § 5.4.1) as targets for the regression. The Ridge parameter is set to be γ = 0.1 and the considered temporal context is [0, 250]ms post-stimulus as done in the experiments of Chapter 3 and Chapter 4. Secondly, the reconstructed amplitude envelopes are binarized following the same process used in § 5.4.1. An example of how those activations look like and relate to the ones computed synthetically from the ground-truth sources can be seen in Figure 5.10. One can immediately see that the EEG-derived activations are quite imprecise with respect to the synthetic ones. However, the main problem seems to be that the sources are always activated, leaving small room for action to the proposed loss function. In fact, in the MAD-EEG dataset, the sources in the mixtures are almost always activated. As we have seen in the experiments in § 5.4.1, we need enough silent portions in the mixture to bene t from the adaptation as we proposed it. Therefore, we do not expect a clear improvement over the 5.11: Median over all tracks of the median SDR (expressed in dB) over each track for di erent ne-tuning strategies and di erent instruments in the dataset. Blue bars correspond to models adapted with the proposed loss using the EEG-derived activations, while Orange ones correspond to models ne-tuned using the mixture reconstruction loss only. Green bars correspond to models ne-tuned with the proposed loss using the synthetic activations. The horizontal red line represents the B0 baseline, i.e., the original ConvTanset before adaptation.

non-adapted model because the data itself is not ideal for this study. Nevertheless, it is still interesting to perform the experiments and analyze the results to better understand the problematic.

S

In Figure 5.11 one can see the SDR expressed in dB for di erent ne-tuning strategies and instruments in the dataset. On the x-axis, one can see how the SDR changes with respect to the block from which we start ne-tuning the network. The red line represents the B0 baseline, i.e., the original ConvTanset trained on the MUSDB18 training set and not adapted at all. Blue bars correspond to models ne-tuned with the proposed loss function using the EEG-derived activations. In contrast, orange ones correspond to models ne-tuned using the mixture reconstruction loss only. As an additional control, we ne-tuned the model with the proposed loss function using the synthetic activations computed from the ground-truth sources as described in § 5.4.1 (green bars). This additional experiment aims at distinguishing two di erent e ects on the performance, the ones related to the audio data and the ones related to the EEG. In Table 5.2, the reader can have an insight into the evolution of all the metrics for the baseline B0 and the model adapted using the EEG-derived activations for di erent ne-tuning depths.

The rst observation is that the baseline B0 performs very well on the MAD-EEG dataset, achieving excellent performances for all the instrument classes. The reader can compare the results of B0 presented in Table 5.2 with those of Table 5.1 and immediately see that the MAD-EEG mixtures seem much easier to separate for the ConvTasnet than those of the MUSDB19 dataset, especially for the class "other" and the drums, where the SDR almost doubles. While those results are excellent from the more general point of view of source separation, this also indicates that the MAD-EEG dataset is probably too easy for a state-of-the-art model as ConvTasnet to verify if the EEG information can help the separation. The better the original model's performance and the easier the mixtures to separate, the harder it will be to see an improvement.

A second observation can be made by observing the performances of the model adapted with the mixture reconstruction loss only (orange bars) and the one adapted with the "ideal" synthetic activations computed from the ground-truth sources (green bars) in Figure 5.11 or no di erence among the two cases indicating that the sources are almost always activated and con rming our worries about the MAD-EEG dataset. When the sources are always active, the proposed adaptation loss reduces to the only mixture reconstruction term, leading to an unsupervised adaptation where no activations are needed. In such a case, it is simply not possible to assert the in uence of the EEG information.

Nevertheless, we can look at the results obtained using the EEG-derived activations as we can still get some interesting insights for future works. In this case, it is not necessary to ne-tune in-depth as for the previous experiment on the MUSDB18 data. On the contrary, this deteriorates the results, in some cases proportionally to the ne-tuning depth (see the guitar and the bass in Figure 5.11). This probably happens for two main reasons. Firstly, ne-tuning starting from a deeper block corresponds to millions of more parameters to ne-tune. If the number of such parameters is high, this requires a proportional amount of training data, which in our case is not possible, as the "adaptation" data comes from only one mixture. In the MAD-EEG dataset, the mixtures are only about 24-second long, while the MUSDB18 mixtures are full-length tracks lasting some minutes. Secondly, the fact that the model was already performing very well, combined with the fact that the activations are highly deteriorated and imprecise, leads to a degradation of the performances w.r.t. B0. The more parameters are ne-tuned, the more one can degrade the performances.

Two emblematic cases of this situation are the guitar and the bass. In the rst case, we know that the guitar is almost always active throughout most of the mixtures of MAD-EEG and the big di erence between green and blue bars indicates that the EEG-derived activations are highly imprecise (e.g., the case of the lower plot of Figure 5.10). In the second case, instead, we know that the bass is not always active. Therefore, the main problem lies in the EEG-derived activations. In Table 5.1 we can see that the SDR degradation for the bass is primarily due to a degradation of the SAR, which measures the artefacts as a consequence of more aggressive removal of the interferer. The explanation is that the EEG-derived activations are zeros where the bass is instead active (see, for instance, the second row of Figure 5.10). This error forces the separation model to output zeros where the source was instead active, removing the other sources better and increasing the artefacts.

On the contrary, the SDR improvement of the proposed approach over B0 is 8.8 7.9 7.5 8.1 5.9 3.2 5.9 11.4 10.1 11.9 10.1 8.1 10.5 T 5.3: SDR for the best con guration (P-L5:D) di erentiated to whether the separated instrument was also the attended one.

clear for the vocals and the drums, which are the less activated sound source in the dataset. The vocals present an SDR improvement which is mainly due to better removal of the interferers, which does not increase the artefacts. For the drums, instead, the separation is simply less aggressive therefore reducing the artefacts. However, we must underline that those improvements over B0 are not statistically signi cant (p > 0.05, Wilcoxon test).

In Figure 5.10, one can see that the activations computed from the ground truth source of these two sources indicate that they are always activated. This happens because, in MAD-EEG, the ground truth sources present some crosstalk between microphones (the instruments were not recorded separately). Therefore, even if the source is silent, it is possible to track the energy of the residual sources. The EEG-derived activations, instead, even if imprecisely, indicate much more silence.

A In Table 5.3 we untangled the SDR results for one of the best con gurations (P-L5:D) according to whether the separated instrument was also the attended one. Intuitively, we should get a higher SDR if the separated instrument is also the attended one as we should get a better reconstruction of its activations from the EEG. However, the di erence between these attended and unattended is never statistically signi cant (p > 0.05, Mann-Whitney test). As we have seen previously, it seems that the dominant factor in the results is the fact that the sources are always activated and that the mixtures are very easy to separate, precluding the possibility to evaluate the in uence of the EEG information.

In the left plot of Figure 5.12, the reader can see the PCC computed between the AE reconstructed from the EEG and the ones derived from the estimates of the separation system. The di erence between the distribution of PCCs computed with the estimates of the attended source and the ones with the unattended sources is only signi cant for the bass. This fact should not come as a surprise because, as we have seen in Chapter 3, the AE is a poor descriptor of the music signal, and to have a stronger contrast, we would need to use a TF representation as target.

I

AAD TF Therefore, we tested a second audio descriptor, the MEL spectrogram, which was proven to be the more robust for the AAD task for music (see Chapter 3). The adaptation procedure does not change because the MEL is used only as a target for the decoding. For the adaptation, the binary activations are derived from the MEL spectrogram as follows: we consider MEL bands as narrow-band amplitude envelopes at speci c frequencies. The broadband amplitude envelope can then be reconstructed as the average narrow-band amplitude envelopes across the frequency In the right plot of Figure 5.12, the reader can see the PCC computed between the MEL spectrograms reconstructed from the EEG and the ones derived from the estimates of the separation system. For the MEL the contrast between the attended and the unattended source is much more remarkable than for the AE, similarly to what previously veri ed Chapter 3. This fact is also re ected in the decoding performances in Table 5.4. Note that here the AAD problem is tackled without access to the ground truth sources as in Chapter 3 but using the separation system estimates. Speci cally, we computed the PCC between the audio representation reconstructed from the EEG with the ones computed from the separation model estimates, and the attended instrument is recognised as the one that has the highest correlation. The chance level is 50% for duets, around 33% for trios, and approximately 44% for all the test mixtures together. The MEL descriptor indeed achieves better performances than the AE for all the instruments except for the bass, for which they perform similarly. Indeed the bass was the only instrument for which the di erence among the two distributions of PCCs was statistically signi cant also for the AE descriptor. It is also more robust for trios, as already veri ed in Chapter 3.

Except for a slight drop in the performances, the results are consistent with the ones obtained in Chapter 3 where we were using the ground truth sources. This fact indicates that the estimates of the source separation system are good enough to replace the ground truth sources.

Even so, the separation performances do not improve signi cantly with the new audio descriptor. Note that the reconstructed MEL is not used directly in the adaptation, but it is used to derive the energy envelope of the signal and then its binary activations. Therefore, the proposed loss function was not adapted to deal with TF representations, and we reserve it for future works.

Accuracy(%)

All 5.4: Decoding accuracy for the best con guration (P-L5:D) and the AE and MEL audio descriptors. "****" denotes very high (p < 0.0001), "***" high (p < 0.001), "**" good (p < 0.01), "*" marginal (p < 0.05) and "n.s. " no (p > 0.05) statistical signi cance compared to chance level for a non-parametric randomization test.

C

In this chapter, we proposed a User-guided one-shot deep model adaptation for music source separation, where the temporal segmentation annotated by the user is used to adapt a pre-trained deep source separation model to one speci c test mixture. The adaptation is possible thanks to a newly proposed loss function that aims to minimize the energy of the silent instruments while at the same time forcing the perfect reconstruction of the mixture. We emphasise that the proposed approach is general and can be applied to other types of audio sources (speech, natural sounds) or di erent deep model architectures.

We experimented with two variants: one where the user manually annotates the activations of the sources and a more challenging one where the activations are reconstructed from the neural activity.

The results show that for improving the separation quality, we need at least a weak guiding signal (time activations) in a semi-supervised setting and that an utterly unsupervised adaptation is not enough (mixture reconstruction loss only). The results obtained with "ideal" manually annotated activations in the experiments on the MUSDB19 dataset are promising. They show that a state-of-the-art MSS model like ConvTasnet may be signi cantly improved via adaptation with a few epochs to the speci c test mixture, especially in complex cases. However, the improvement is not systematic when it comes to using the EEG-derived time-activations on our data, mainly because MAD-EEG was not ideal for this study. Firstly, in this dataset, the sources tend to be constantly activated, making it hard to see an in uence of an adaptation based on time activations. Secondly, the mixtures to be separated are too easy for a state-of-the-art model such as ConvTasnet, making it hard to see the in uence of the EEG-derived information.

Even if the separation quality does not improve systematically, thanks to the proposed approach, it is possible to reformulate the AAD problem exposed in Chapter 3 using the separation model estimates instead of the ground truth sources. The results obtained with the Mel spectrogram as audio descriptor for the decoding are satisfactory, with only a marginal drop in the performances, if compared with the ones obtained in Chapter 3 where we were using the ground truth sources. This fact indicates that the source separation system estimates are good enough to replace the ground truth sources Part IV

EPILOGUE

• MAAD We investigated for the rst time the problem of AAD to a target instrument in polyphonic music based on the continuous EEG response.

To this end, we exploited the so-called backward model, which was proven to successfully decode the attention to speech in multi-speaker environments. To our knowledge, this model was never applied before to musical stimuli for AAD, and we extensively evaluated it on MAD-EEG.

The primary outcome of this study is that the EEG tracks musically relevant features highly correlated with the attended source and weakly correlated with the unattended one making it possible to decode the auditory attention towards a speci c instrument in the mixture.

P III N

• C-NMF We proposed a neuro-steered MSS framework where we leverage the fact that the attended instrument's neural encoding is substantially stronger than the one of the unattended sources left in the mixture to inform a source separation model based on NMF and automatically separate the attended source. Thanks to the C-NMF formulation, we could reformulate the AAD problem di erently, without needing access to the "clean" audio sources, which are absent in real-life scenarios. We extensively evaluated the proposed system on MAD-EEG, obtaining encouraging results, especially in di cult cases where non-informed models struggle.

• UGOSA We investigated whether it is possible to inform a MSS model based on DL using the time activations of the sources manually annotated by the user or derived from his/her EEG response available at test time. Indeed, the scarcity of music-related EEG data precludes the possibility of using fully supervised DL approaches, which, however, represent the state-of-the-art in MSS. This approach can be referred to as one-shot, as the adaptation acts on the target song instance only. Thanks to the proposed approach, we could reformulate the AAD problem using the separation model estimates instead of the ground truth sources. Even if immature, the results are encouraging and point at promising research directions.

Taken together, we hope that these contributions make a step forward towards the direction of integrating BCI and MSS. Nevertheless, much remains to be done, and many research questions arise from the conducted investigation.

F

In the previous section, we have summarized the main ndings and contributions of the thesis. Nevertheless, we have only scratched the surface of many problems related to EEG-based AAD and neuro-steered MSS. Besides, there are many limitations and much room for improvement in the methods proposed here. This section elaborates on short and long-term research directions that arise as natural follow-ups to the topics discussed so far.

L

The lack of freely available music-related EEG datasets has been a strong hindering factor for the research in this eld. It is a common experience that acquiring such a type of dataset is time-consuming and expensive. It requires speci c equipment and experience and a long phase of experimental design, preparation and participants recruitment. Participants are available for a limited time and cannot be overloaded with too long recording sessions. Therefore, those datasets are often limited in terms of recording hours and the number of participants. These factors represent a signi cant obstacle to the research in the eld. With MAD-EEG we hope to help researchers pursuing research in the eld, especially for those working in MIR who usually do not have the equipment and expertise of a cognitive or neuroscience oriented laboratory. We are aware of the intrinsic limitations of the dataset, and in future works, we plan to extend the dataset in terms of the number of EEG recordings and stimuli variants and behavioural data. Although, the dataset will still have a size that does not allow studying DL models unless speci c learning strategies are adopted. In such low-labelled data regimes, it is necessary to focus on alternative strategies to exploit the large amount of unlabelled data that is often available from close and similar domains.

S

Throughout the thesis, the decoder was always trained in a subject-speci c fashion, which means that only data recorded from the same subject under test are used for training the decoder. Subject-speci c models represent standard practice in BCI because the EEG temporal and spatial characteristics vary signi cantly between subjects. However, this can be a substantial limitation, especially in the optic of real-life applications, as this approach requires a time consuming and inconvenient calibration phase. A subject-independent model would allow avoiding such a calibration phase and also incredibly enlarge the amount of training data of the already existing and available datasets. Indeed, a subject-independent model still requires labelled data, but this can also come from subjects di erent from the one under test. This fact would allow pre-training such models and make them much more practical for realistic applications. However, subject-independent BCIs have generally shown poor performances in the literature if compared to subject-speci c ones [Ghane et al. 2021] and this applies also to AAD models [O'sullivan et al. 2014] mainly due to the high inter-subject variability in the EEG data. Therefore, speci c research needs to be conducted in this direction. Transfer Learning (TL) techniques to adapt a pre-trained decoder to unseen subjects as proposed by Geirnaert et al. might be the way to go [Geirnaert et al. 2021a].

A

Another under-considered aspect in AAD research is that the decoding models are not adaptive to the new test data leading to suboptimal results. Adaptation could help in all situations with changing environmental conditions, audio sources, and brain activity which is nonstationary. The EEG temporal and spatial characteristics vary signi cantly between subjects (di erent scalps, electrodes placement/impedances) but as well among the data of the same subject (electrode displacements, change of electrode-skin contact impedance, di erent recording sessions). There have been a few e orts in this direction [START_REF] Akbari | Dynamic estimation of the auditory temporal response function from MEG in competing-speaker environments[END_REF][START_REF] Miran | Real-time tracking of selective auditory attention from M/EEG: A bayesian ltering approach[END_REF][START_REF] Aroudi | Improving auditory attention decoding performance of linear and non-linear methods using state-space model[END_REF]Geirnaert et al. 2021a]. Akram et al. employ state-space models to compute a dynamic estimate of the decoder over time [START_REF] Akbari | Dynamic estimation of the auditory temporal response function from MEG in competing-speaker environments[END_REF] while Miran et al. extended this work by making it able to operate near real-time [START_REF] Miran | Real-time tracking of selective auditory attention from M/EEG: A bayesian ltering approach[END_REF]]. In both cases, the decoder is estimated for each new data segment in an unsupervised fashion and then applied again to that same data segment to determine the attended source. Hence, the model is adaptive with respect to the new incoming data. Geirnaert et al. instead proposed to adapt subject-speci c models to new subjects in an unsupervised manner. The autocorrelation matrix is updated on the new batch of data, and the subject-speci c model is thus updated. After prediction, one can also compute the cross-correlation matrix and re-update the decoder iteratively. All those approaches are de ned as unsupervised because they do not require the attended/unattended labels but still require the isolated sources to update the cross-correlation. 1 1 In Chapter 3, the decoder adaptation happens di erently: every certain number of NMF iterations, the dictionary associated with the attended source is used as an updated feature extractor for the decoder training data (solos of the attended instrument). The autocorrelation data is not updated, and only the cross-correlation matrix changes. It would be now interesting to explore another variant, where the solos of the attended instrument are used only to obtain a good initialization of the decoding model, which is then updated in an unsupervised fashion using the test data only. The adaptation, in this case, can act on both the autocorrelation and cross-correlation matrices as proposed by Geirnaert et al. [Geirnaert et al. 2021a], obtaining an adaptation that acts on both the neural (non-stationarity, di erent subject) and audio data (di erent environmental conditions, di erent source type).

D

The scarcity of music-related EEG data involving attention precludes the possibility of tackling the problem of neurosteered MSS with fully supervised DL approaches. As explained in Chapter 4, unsupervised techniques such as NMF are ideal in such cases as it is easy to incorporate additional information about the sources directly in the optimization cost without requiring a data-intensive training phase. However, MSS systems based on NMF have their limitation in terms of separation performances if compared to DL approaches. It is desirable then to use DL models and possibly nd alternative learning strategies to alleviate the problem of the lack of music-related EEG data involving attention. In Chapter 5 we proposed a one-shot adaptation of a pre-trained DNN for MSS to a speci c mixture using the user's EEG response, which is available only at test time. It was a rst, straightforward attempt to work around the lack of signi cant training data. However, it has its limitations, and there are many other directions we would like to explore in future works. For instance, only the MSS is DLbased, while the EEG decoding part still relies on a linear regression model. Generally speaking, it is not yet clear if DL has signi cant advantages compared to traditional approaches for a variety of di erent BCI and monitoring applications [START_REF] Roy | Deep learning-based electroencephalography analysis: a systematic review[END_REF]] and the same can be said for AAD [Geirnaert et al. 2021b]. Certainly, such data-hungry models struggle in low-labelled data regimes, which are the standard when working with EEG [START_REF] Roy | Deep learning-based electroencephalography analysis: a systematic review[END_REF]]. We indeed lack data related to attention, but there exist many other small music-related EEG datasets where the subjects were not attending any particular source in the mixture. It would then make sense to aggregate all these heterogeneous auxiliary datasets to scale the training data and use the attention-related dataset only in a second phase as a target dataset thanks to some TL techniques.

S

One possible strategy is to use Self Supervised Learning (SSL) to learn from the auxiliary datasets a feature representation for both the audio and neural data optimal for joint tasks. The idea is then to use those representations to solve multi-modal problems such as AAD and neurosteered MSS on the target dataset. SSL allows learning representations from unlabeled data by exploiting the intrinsic structure of the data in a pretext task [START_REF] Jing | Self-supervised visual feature learning with deep neural networks: A survey[END_REF]Banville et al. 2021a]. These representations can then be used in a downstream task for which there are limited or no annotated data. Speci cally, the unsupervised problem is then reformulated as a supervised one by automatically generating the annotations from the data with the condition that the pretext and downstream tasks must be su ciently related. Despite its potential, only a few works have used SSL to improve EEG-related tasks over standard approaches [START_REF] Yuan | Wave2vec: Learning deep representations for biosignals[END_REF]Banville et al. 2021a;[START_REF] Kostas | BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data[END_REF]], but never in multimodal scenarios.

In our case, a good pretext task can be the one of relative positioning, i.e., determining whether a pair of representations, one for the audio and one for the EEG are synchronized as proposed by [START_REF] Banville | Robust learning from corrupted EEG with dynamic spatial ltering[END_REF]]. 2 One can 2 A similar approach, called match/mismatch paradigm based on [START_REF] Cheveigné | Decoding the auditory brain with canonical component analysis[END_REF] was proposed by Cheveigné et al. to compare performance of di erent linear stimulusresponse models [START_REF] Cheveigné | Auditory stimulus-response modeling with a match-mismatch task[END_REF]]. The paradigm was investigated by Accou et al. in the non-linear case [START_REF] Accou | Modeling the relationship between acoustic stimulus and EEG with a dilated convolutional neural network[END_REF]]. However, in both cases, the match/mismatch paradigm is not used as pre-text task for pretraining a DL-based model for AAD (downstream task).

arti cially generate millions of positive and negative pairs of EEG and audio data, which are respectively located within a local positive window or outside a long-range negative window and train the system in a contrastive fashion. One can also consider using triplets, where the anchor is represented by an EEG segment, and the positive and negative examples are audio segments respectively located within the positive window and outside the negative one. Here, the non-stationarity of the EEG data is not a drawback but a necessary assumption. Moreover, the smoothness assumption (neighbouring representations have the same label) is fair because the EEG tracks slowvarying features of the audio, such as the amplitude envelope.

However, in the experiments of Banville et al., the time windows employed are pretty long (order of minutes), while in our case, we are interested in features on a smaller time scale (order of seconds). Assuming that representations within the positive window are similarly labelled, it might not be easy to expand to time scales closer to that of one of our trials. Additionally, in our case, mining negative pairs hard enough for learning might be tricky. The negative examples need to be di cult enough to prevent the network from learning other data features we do not want it to learn (e.g., subject, song, trial). 3 Adversarial training can be a helpful coupling strategy to make the 3 We do not have, in fact, long recordings as the ones for studying sleep staging [Banville et al. 2021a], but shorter recordings of subjects listening to di erent musical pieces. For instance, if the negative pair is chosen on a di erent audio stimulus, the network might learn to classify the song. If we choose a negative window for another subject or recording session, the network might learn to classify di erent trials or subjects. If we choose the negative pair within the same subject and same audio stimulus, we have to be careful with the repetitions that often occur in music, and that can elicit a similar EEG response.

representations learned with self-supervision independent and robust of those features of the data we are not interested in. We are currently conducting experiments in this direction.

I

Another signi cant challenge that immediately emerges is the fact that the auxiliary datasets are often heterogeneous and inconsistent. It means, for instance, that they were collected using di erent protocols and headsets, resulting in varying channel ordering, numbers, and often di erent signal references. It is also common to have noisy or even missing channels. Consequently, available music-related EEG datasets are heterogeneous, and all of them are very small. Scaling EEG training data seems, therefore, only possible by aggregating them according to some strategy. Moreover, transferring trained systems across datasets, for instance, from the auxiliary to the target one, exhibits the same di culty. These factors require speci c strategies for training a system with heterogeneous datasets and enable TL across the auxiliary and the target datasets. Wu et al. propose to use the common subset of channels shared between headsets [START_REF] Wu | Switching EEG headsets made easy: Reducing o ine calibration e ort using active weighted adaptation regularization[END_REF].

Other more elaborate approaches propose to use attention mechanisms that recombine the input channels into a xed number of virtual channels [Nasiri and Cli ord 2020; [START_REF] Guillot | RobustSleepNet: Transfer learning for automated sleep staging at scale[END_REF][START_REF] Saeed | Learning from heterogeneous eeg signals with di erentiable channel reordering[END_REF]], which, however, are not easily transferable when going from the pretext to the downstream task. In contrast, the dynamic spatial ltering proposed by Banville et al. allows for re-using the same lter learned in the pretext task as it is in the downstream task, allowing the transferability among the two tasks [START_REF] Banville | Robust learning from corrupted EEG with dynamic spatial ltering[END_REF]]. The latter approaches are based on di erentiable preprocessing. It means that the network will learn the best recombination of channels according to their predictive power. In our current investigation, we are interpreting this challenge as a data augmentation problem, where di erent electrodes con gurations represent di erent and augmented views of the same data. To preserve the spatial consistency across training examples, one can select a subset of electrodes as xed centroids and, for each training example, sample an electrode in its neighbourhood. This way, one can potentially generate multiple and augmented versions of the same training example simulating electrodes displacements and di erent head shapes, making the system robust to such variations.

Part V

APPENDICES

Statistical testing

S

Throughout the thesis, we evaluate the statistical signi cance of

Keywords:

Hypothesis testing, nonparametric tests, randomization test.

Resources:

Code randomization test the presented results using hypothesis testing. In this chapter, the reader can nd further explanations and details about each hypothesis test used in the manuscript.

L

The choice of the statistical test depends rst of all on the research question we want to answer. In our case, we want to nd di erences: is there a signi cant di erence among the PCCs in the two conditions attended and unattended instrument? Are the SDR scores of the proposed method better than those of the baseline? Are we decoding attention better than a random classi er?

If we look at di erences, one must understand if the samples are independent (unpaired) or related (paired). When we compare the SDR of the baseline with the SDR of the proposed method, the samples are paired: one music mixture is separated by both the methods obtaining two SDRs which are related. The samples would be unpaired if we tested the baseline and the proposed method on a di erent set of mixtures, but this is not the case when comparing source separation systems. The same consideration can be made when evaluating the decoding performances: since each mixture contains a di erent combination of instruments, the probability of randomly choosing one instrument as the attended one will vary between mixtures.

The second aspect to consider when choosing a statistical test is the distribution of the data. Parametric tests such as the t-test assume that the data are randomly sampled from a population whose distribution of scores is characterized by a xed number of parameters, e.g., a normal distribution parametrized by the mean and standard deviation. If we want the outcome of a statistical test to be valid, assumptions on the data distribution must be met. Otherwise, non-parametric or randomization-based tests should be used. In the case of a small sample size, it might be pretty hard to assess whether an assumption is met or not, and it is better to opt for non-parametric or randomization-based tests which do not make any assumption on the distribution of the data.

C

In our evaluation, to compare two distributions of scores (e.g., PCCs, linear SDR) we opted for two non-parametric tests:

• Wilcoxon signed rank test in the case of paired samples [START_REF] Conover | Practical nonparametric statistics[END_REF]];

• Mann & Whitney U-test in the case of unpaired samples [START_REF] Mann | On a test of whether one of two random variables is stochastically larger than the other[END_REF].

C

Instead, in the cases where we needed to evaluate the decoding performances, the statistical signi cance of classi cation results (e.g., accuracy, F1 score) compared to chance level was assessed using an adaptation of the computationallyintensive randomization test [START_REF] Noreen | Computer-intensive methods for testing hypotheses[END_REF]], a non-parametric hypothesis test, which does not make any assumption on the score distribution and can be used also for complex non-linear measures such as F1 score [Yeh 2000]. In our speci c case, the test is built by implementing the following procedure: rst, we considered a random classi er that, given a test mixture, chooses the attended instrument randomly among the instruments in the given mixture. Then, the performances were computed over the random predictions on the complete test set. This procedure was repeated 10000 times, which resulted in a distribution of the performances. This empirical distribution was then approximated with a theoretical distribution which could be a normal or a t-distribution (the one that ts better). Then we evaluated how likely our model's actual performances were to be produced by this arti cial distribution of performances obtaining the P-value. Our implementation of the hypothesis test can be found at. 4Detailed Derivation of the Multiplicative Update Rules for the Contrastive-NMF

S

This Chapter provides the detailed derivation of the Multiplicative Keywords: Contrastive-NMF, Nonnegative matrix factorisation, Multiplicative updates.

Resources:

• Paper

• Code

• Demo Update (MU) rules for the Contrastive-NMF (C-NMF) which was presented in Chapter 4. The material reported here is extracted from the supplementary material accompanying the work in [Cantisani et al. 2021b]. Let us consider a mixture x(t) given by the linear mixing of the attended source s a (t) and some interferers s u (t). Let W a ∈ R M ×Ka + be a sub-dictionary of W containing a set of basis vectors representing source s a (t) and H a ∈ R Ka×N + be their activations. Let H u ∈ R (K-Ka)×N + be the activations of the interference sources. H a can be roughly approximated by S a reconstructed from the time-lagged EEG response R, the assumption being that it is likely to be more correlated with the NMF-derived activations of the attended source H a than with the ones of the interferers H u . The rows of H and S a (h k: and s k: respectively) are normalized in order to minimize the e ect of a scale mismatch between the modalities.

Multiplicative Update Rules

To derive the MU rules, one can compute the gradient of the cost function ∇C(θ), split it into is negative and positive parts, i.e., ∇C(θ) = ∇ θ + C(θ) -∇ θ -C(θ), (6.2)

and build the rules as follows [START_REF] Lee | Learning the parts of objects by non-negative matrix factorization[END_REF][START_REF] Févotte | Algorithms for nonnegative matrix factorization with the β-divergence[END_REF]:

θ ← -θ ⊗ ∇ θ -C(θ) ∇ θ + C(θ) . (6.3)
Since the variables are θ = {W, H}, the MU rules will be: .5) w 96

H ← -H ⊗ ∇ H -C(W, H) ∇ H + C(W, H) , (6.4) W ← -W ⊗ ∇ W -C(W, H) ∇ W + C(W, H) . ( 6

U W

Since the cost function is completely separable, we can compute the gradient for the KL divergence and for the sparsity constraint separately.

KL Divergence

• D KL (p, q) = p log p q -p + q

• WH| mn = k w mk h kn

• matrix product derivative: Update rule

∂ ∂w ij WH| mn =    h jn if m = i 0 if m = i • Λ = WH ∂D KL (X|WH) ∂w ij = ∂ ∂w ij
W ← -W ⊗ ∇ W -C(W, H) ∇ W + C(W, H) = W ⊗ (Λ -1 ⊗ X)H T 1H T + β (6.8)
where ⊗, divisions and exponents denote element-wise operations, 1 is a matrix of ones whose size is given by context and Λ = WH.

U H

As for W, we can compute the gradient for the KL divergence, the sparsity constraint and for the margin term separately.

KL divergence

• D KL (p, q) = p log p q -p + q

• WH| mn = k w mk h kn

• matrix product derivative:

∂ ∂h ij WH| mn =    w mi if n = j 0 if n = j • Λ = WH ∂D KL (X|WH) ∂h ij = ∂ ∂h ij

Contrast term

Recall that the Frobenius norm can be rewritten as:

X F = M i=1 N j=1
x 2 ij = Tr(X T X) (6.11)

Since H a S T a and H u S T a are square matrices, we have: The gradient with respect to H, will be equal to the gradient computed with respect to H a for the rst K a rows of H and equal to the gradient computed with respect to H u for the remaining rows: Thus, we have:

∇ H (-δ( H a S T a 2 F -H u S T a 2 F )) = -2δH a S T a S a , if 1 < k < K a +2δH u S T
a S a , if K a + 1 < k < K (6.17)

Update Rule

H ← -H ⊗ ∇ H -C(W, H) ∇ H + C(W, H) = H ⊗ W T (X ⊗ Λ -1 ) + δP - W T 1 + µ + δP + (6.18)
where ⊗, divisions and exponents denote element-wise operations, 1 is a matrix of ones whose size is given by context and Λ = WH. P -, P + ∈ R K×N + are auxiliary matrices de ned as:

P -=
H a S T a S a , if 1 < k < K a 0, if K a + 1 < k < K (6.19) There are several reasons why we all should aim to disseminate our research beyond our universities and scienti c communities. For instance, it might be essential to explain your research to a general audience because you are doing it thanks to some public funding. In such a case, it is a social duty to inform the citizens about your ndings and make your research comprehensible. It's a virtuous circle that produces culture and participation, and in return, can pay for new investments in research.

P + = 0, if 1 < k < K a H u S T a S a , if K a + 1 < k < K (
Another reason is to attract the next generation towards science and your speci c research eld. This is an aspect that is often underrated because it hasn't an immediate economic and/or social recognition return, but that is critical in the long term. Undergraduate students can orient their education choices and be our future colleagues and enlarge our research community. It's vital then to let them know that your research exists and might be interesting for them. This would also bene t and increase diversity in the community and reach all those students for whom computer science is not among the options because of societal, demographic, or socioeconomic factors.

In this context, it is still tough for scientists to involve the uninitiated on very speci c topics that seem to have almost no connection with their everyday lives. However, many di erent techniques, tools, and languages have been studied and gradually re ned over time. With the increasing amount of information available online, it is becoming more and more important to be concise and attract the audience's attention from the very beginning.

V have become more and more popular over the last decade as they are a low-barrier medium to communicate ideas e ciently and e ectively. Short videos from 3 to 5 minutes are ideal because they are long enough to explain a concept and su ciently short for viewers to decide if they are interested. We all have learned about the advantages and disadvantages of this medium during the last year of the pandemic. The format of the conferences has changed, and video abstracts are now a standard. However, video abstracts are intended for peers and not for a broader audience. When disseminating science, complex concepts should be made accessible for the largest audience possible. In such a case, motion graphics and animated storytelling can be a possible solution. Through the process of abstraction in an animated representation, we can e ectively simplify the concept we want to transmit. The style, colour palette, transitions, aesthetic and functional choices can all concur to convey the main message. T is not easy. It takes time, many iterations over the script and many drafts before coming up with something good. You have to learn to work with visual designers who do not know anything about your research. We experienced this when working on the MIP-frontiers video communication project, meant to attract young researchers in our research eld. It's very hard to simplify and abstract things you work on every day. It feels like sacri cing many details which are essential to you for the sake of simplicity. Because of that, you have to always keep in mind who's your target audience. In the speci c case of this video, there was an additional problem: we needed to cover the most possible areas in Music Information Once the voiceover, the animation and the music are done, it is not trivial to create the nal video anyway. In fact, in addition to a temporal synchronization of events, automation on the volume of the various instruments and the voice are necessary. This operation is always necessary for video production, and the role of a sound engineer is essential for an optimal result. Especially in this work, where music and its evolving parts are the protagonists, this professional gure had a particularly central role in glueing all the components.

S T

We really thank Mandela (music), Scienseed (animation) and Alberto Di Carlo (sound engineer) for their great work! Mandela is an Italian instrumental jazz band from Vicenza. The sound of the band is characterized by a fusion of jazz idioms, rock, world music, psychedelic, and funk. Over the years, the band has performed in several festivals and venues and released 3 full-length albums. These recordings are all available on the major streaming service. Their last release was presented at the festival Rimusicazioni (Bolzano, Italy) and consists of an original soundtrack for "Grass:

A Nation's Battle for Life" -one of the earliest documentaries ever produced (1925). For this video, the track Simple from the album Mandela s.t. was used.

The song was remixed and remastered by Alberto Di Carlo.

Scienseed is a multifunctional agency for the dissemination of scienti c ndings. Its founding goal is to promote public engagement in science through all available tools in the Era of IT. We are specialized in the translation of scienti c data into di erent accessible products and activities, aimed at either the scienti c community (peers) or the general public (society). We provide support to academic laboratories, research institutes, universities and private institutions to raise public awareness and increase the repercussion of their contribution to science. La NMF non supervis ée est une approche efficace dans de telles applications ne disposant pas ou peu de donn ées d'apprentissage, comme c'est le cas dans des sc énarios n écessitant des enregistrements EEG. En effet, les jeux de donn ées EEG li és à la musique disponibles sont co ûteux et longs à acqu érir, ce qui exclut la possibilit é d'aborder le probl ème par des approches d'apprentissage profond enti èrement supervis ées. Dans la derni ère partie de la th èse, nous avons explor é des strat égies d'apprentissage alternatives. Plus pr écis ément, nous avons étudi é la possibilit é d'adapter un mod èle de s éparation de sources de l' état de l'art à un m élange sp écifique en utilisant les activations temporelles de sources d ériv ées de l'activit é neuronale de l'utilisateur au moment du test. Cette approche peut être consid ér ée comme étant " à adaptation unitaire" (one-shot), car l'adaptation agit uniquement sur une instance de chanson. Nous avons évalu é les approches propos ées sur les jeu de donn ées MAD-EEG qui a ét é sp écifiquement assembl é pour cette étude, obtenant des r ésultats encourageants, en particulier dans les cas difficiles o ù les mod èles non inform és sont mis à mal. Title : Neuro-steered music source separation Keywords : Music source separation, Auditory attention decoding, EEG, Multimodal processing Abstract : In this PhD thesis, we address the challenge of integrating Brain-Computer Interfaces (BCI) and music technologies on the specific application of music source separation, which is the task of isolating individual sound sources that are mixed in the audio recording of a musical piece. This problem has been investigated for decades, but never considering BCI as a possible way to guide and inform separation systems. Specifically, we explored how the neural activity characterized by electroencephalographic signals (EEG) reflects information about the attended instrument and how we can use it to inform a source separation system. First, we studied the problem of EEG-based auditory attention decoding of a target instrument in polyphonic music, showing that the EEG tracks musically relevant features which are highly correlated with the timefrequency representation of the attended source and only weakly correlated with the unattended one. Second, we leveraged this "contrast" to inform an unsupervised source separation model based on a novel non-negative matrix factorisation (NMF) variant, named contrastive-NMF (C-NMF) and automatically separate the attended source. Unsupervised NMF represents a powerful approach in such applications with no or limited amounts of training data as when neural recording is involved. Indeed, the available music-related EEG datasets are still costly and time-consuming to acquire, precluding the possibility of tackling the problem with fully supervised deep learning approaches. Thus, in the last part of the thesis, we explored alternative learning strategies to alleviate this problem. Specifically, we propose to adapt a stateof-the-art music source separation model to a specific mixture using the time activations of the sources derived from the user's neural activity. This paradigm can be referred to as one-shot adaptation, as it acts on the target song instance only. We conducted an extensive evaluation of both the proposed system on the MAD-EEG dataset which was specifically assembled for this study obtaining encouraging results, especially in difficult cases where noninformed models struggle.

Institut Polytechnique de Paris 91120 Palaiseau, France

4 :

 4 Schematics of the proposed framework: the source separation algorithm is guided by the user's selective auditory attention to that instrument, which is tracked in his/her neural response to music.

F 2. 2 :

 2 Years of musical instrument playing experience for each subject.

F 2. 3 :

 3 Hours per week dedicated by each subject respectively to listening to music (blue) and practising their instrument (red).

F

F 2. 7 :

 7 Number of subjects that experiences respectively a low, medium or high level of fatigue and stress.

F 2. 8 :

 8 Fatigue and stress experienced by each subject: low (1), medium (2) or high (3).

  Block diagram of the acquisition system: surface EEG, EOG, EMG and ECG are acquired through a set of electrodes connected to the headset. An ESU receives the EEG data and timestamps from the headset via Bluetooth and transmits them to the acquisition software. The beginning of the stimulus playback is detected in real-time through a Python script monitoring the playback PC sound card output and then transmitted to the ESU for synchronization.

1

  https://www. advancedbrainmonitoring.com/xseries/ x24/(EOG), Electromiography (EMG) and Electrcardiography (ECG) signals of the participants, as well as their head motion acceleration, thanks to an integrated inertial measurement unit, all at a sampling frequency f s = 256Hz. The EEG headset consists of a wireless digital acquisition unit connected to an electrode strip as the one in Figure2.10. The strip features electrodes F1, F2, F3, F4, Fz, C1, C2, C3, C4, Cz, CPz, P1, P2, P3, P4, Pz, POz, O1, O2 and Oz, placed according to the 10-20 montage system. Active electrodes were referenced to the left mastoid in a unipolar setting.

F 2 .

 2 10: A wireless B-Alert X24 headset was used to record the EEG data. In the gure, one can see the EEG electrodes strip (the left part corresponds to the frontal electrodes).

F 2 .

 2 11: The speakers were positioned ±45 o along the azimuth direction relative to the listener.

Paper

  Code randomization test relating to the problem of decoding the auditory attention to a target instrument in polyphonic music which was extensively investigated on the MAD-EEG dataset.

•

  Cantisani, Giorgia, Slim Essid, and Gaël Richard (2019a). "EEG-Based Decoding of Auditory Attention to a Target Instrument in Polyphonic Music". In: IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) F 3.1: We found that subjects performing a simple attention task show brain activity associated with sticky hair, deadlines, PhD duties and worthwhile rewards. Image courtesy of xkcd, number 1453.

  .1) where g = [flatten(g 1 ), ..., flatten(g K )] ∈ R CL×K (Visualization of a the spatiotemporal decoder reshaped as a tensor whose shape is given by the number of EEG channels C, the number of time lags L and the number of feature coe cients K of the audio representation we need to reconstruct. If the audio representation is the magnitude or Mel spectrogram, then the features will coincide with the frequency bins as displayed in the gure. For the broadband amplitude envelope, K = 1 and thus the tensor becomes a matrix.

  is a matrix composed by the column-wise concatenation of K multichannel Wiener lters g k ∈ R C×L which are reshaped in row-major order into vectors of length CL. C represents the number of EEG channels and L the number of time lags, i.e., the temporal context where we assume to see the EEG response to the stimulus. The time lags range between τ min and τ max and build the temporal context where we assume to see the EEG response to the stimulus as shown in Figure 3.4. An example of decoder for the MEL spectrogram is shown in Figure 3.3. Similarly, R ∈ R CL×N is obtained as the row-wise concatenation of C lagged L × T time series matrices of the neural response recorded at electrode i. Such matrices are only padded with zeros on the left to ensure causality [O'sullivan et al. 2014].

4 :

 4 Schematic of the temporal context used by the backward model to reconstruct one frame of stimulus from the EEG data. For the sake of simplicity, we represented only one EEG channel.

Accuracy

  ** 78 **** 69**** MEL 75 **** 76 **** 74 **** T 3.1:Decoding accuracy for di erent subsets of the test set. "****" denotes very high (p < 0.0001), "***" high (p < 0.001), "**" good (p < 0.01), "*" marginal (p < 0.05) and "n.s." no (p > 0.05) statistical signi cance for a non-parametric randomization test. PCC of the reconstructed stimulus with the attended source (blue), the unattended one (pink) and the mixture (orange) for the three audio descriptors.

  In the rst row, r attended is plotted against r mixture while in the second row, r attended is plotted against r unattended (only duets) for all the audio descriptors. Data points below the red line r attended = r unattended , are classi ed correctly. Data points in the bottom-right corner are classi ed correctly with a large margin.

  s. " no (p > 0.05) statistical signi cance of the results.duets trios same diff monostereo pop classic variant 0r attended and r unattended scores for the all the stimuli variants. Only MAG and MEL descriptors are considered. r attended and r unattended distributions are signi cantly di erent for all the variants (p < 0.001, non-parametric Wilcoxon test).

  r attended is plotted against r unattended for each duet in the test set(only MAG and MEL descriptors). Data points below the red line r attended = r unattended , are classi ed correctly. Data points in the bottom-right corner are classi ed correctly with a large margin. The instruments are marked with di erent colors.

  : Neuro-steered music source separation with EEG-based auditory attention decoding and contrastive-NMF S This Chapter introduces the central contribution of the thesis, a Keywords: Audio source separation, Polyphonic music, EEG, Matrix factorisation, Multimodal processing. music source separation framework built upon the results of the previous Chapter and conducts an extensive evaluation of the proposed system on the MAD-EEG dataset. Speci cally, we leverage the fact that the attended instrument's neural encoding is substantially stronger than the one of the unattended sources left in the mixture to inform a source separation model based on a new variant of NMF named Contrastive-NMF and automatically separate the attended source. This unsupervised NMF variant is particularly advantageous as it allows us to incorporate additional information in a principled optimisation fashion and does not need training data, which is particularly di cult to acquire for applications involving EEG recording. This work led to the following conference publication and preprint: • Cantisani, Giorgia, Slim Essid, and Gaël Richard (2021b). "Neuro-steered music source separation with EEG-based auditory attention decoding and contrastive-NMF". in: IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP) • Cantisani, Giorgia, Slim Essid, and Gaël Richard (2021a). "EEG-based Decoding of Auditory Attention to a Target Instrument for Neuro-steered Music Source Separation". In: journal in preparation

F 4 . 1 :

 41 WHo doesn't? Image courtesy of xkcd, number 2506.

F 4 .

 4 2: Example of NMF decomposition of the magnitude spectrogram of a single-source audio recording.

  Through an Inverse Short Time Fourier Transform (iSTFT) one can recover the corresponding audio signal in the time domain. For a schematic of the NMF-based separation pipeline, the reader can refer to Figure4.3.

F 4 .

 4 3: NMF-based source separation pipeline: rst the magnitude or power spectrogram is decomposed in meaningful spectral components and corresponding time activations. At this point, the separation problem reduces to the assignment of each component to the corresponding source which can then be recovered through a Wiener Filter (WF) where the NMF representation is used as source variance model.

Figure 4

 4

F

F 4 . 5 :

 45 The proposed EEG-informed source separation algorithm (C-NMF-e) and the two baselines: the blind NMF (NMF) and the randomly-informed C-NMF.

T 4 .

 4 1: SDR separation results for di erent models, ensemble types and instruments. The metrics are shown in dB and all values are medians over the corresponding subset of the test set. In the last two rows, the SDR results of the proposed method C-NMF-e are split for stereo and mono listening tests. their EEG response. For a deeper insight, in Figure4.6 one can see the same results visualized with boxplots.

F

  Figure 4.8, we report the AAD accuracy values for di erent instruments and ensemble types. The blue and the red lines represent the chance level for the Guitar Voice Drums Bass Oboe Flute Horn Cello Bassoon 0Decoding accuracy for di erent instruments and ensemble types compared with the chance level for duets and trios respectively.

  Decoding accuracy and Signal to Distortion Ratio (SDR) expressed in dB for di erent instruments and values of the hyperparameter δ that weights the contrastive term. "****" denotes very high (p < 0.0001), "***" denotes high (p < 0.001), "**" good (p < 0.01), "*" marginal (p < 0.05) and "n.s. " no (p > 0.05) statistical signi cance for a non-parametric Wilcoxon test on the linear SDR.

S

  The scarcity of music-related EEG data precludes the possibility Keywords: Music Source Separation, Userguided, One-shot Domain Adaptation, EEG. problem of neuro-steered music source separation with fully supervised deep learning approaches. In this chapter, we explored alternative learning strategies to alleviate this problem. Speci cally, we propose to adapt a state-of-the-art music source separation model to a speci c mixture using the time activations of the sources provided manually by the user or derived from his/her neural activity which are available only at test time. This paradigm can be referred to as one-shot adaptation, as it acts on the target song instance only. A large part of the material presented in the chapter is the result of a work conducted during my internship at InterDigital R&D France under the supervision of Alexey Ozerov and led to the following conference publication:• Cantisani, Giorgia, AlexeyOzerov, Slim Essid, and Gaël Richard (2021c)."User-guided one-shot deep model adaptation for music source separation". In: IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) F 5.1: Is such a pile big enough to hide the corpse of the user? I'm afraid we don't have enough data. Image courtesy of xkcd, number 1838.

  Time activations of the sources annotated by the user are used to adapt a pre-trained deep MSS model to one speci c test mixture.

F 5 . 3 :

 53 ConvTasnet architecture.

F

F 5 .

 5 7: SDR, SAR and SIR expressed in dB: median over frames, median over tracks for di erent values of λ and window length.

F 5 . 8 :

 58 Median over all tracks of the median SDR (expressed in dB) over each track for di erent ne-tuning strategies and di erent instruments in the dataset. Blue bars correspond to models adapted with the proposed loss while Orange ones correspond to models adapted using a reconstruction loss only. The horizontal red line represents the B0 baseline, i.e., the original ConvTanset before adaptation.

F

F

F 5 .

 5 12: PCC of the attended and unattended sources for di erent instruments and audio descriptors. "****" denotes very high (p < 0.0001), "***" high (p < 0.001), "**" good (p < 0.01), "*" marginal (p < 0.05) and "n.s. " no (p > 0.05) statistical signi cance of the di erence among the two conditions for a non-parametric Wilcoxon test. bands and binarized following the same process used in § 5.4.1.

F 1 :

 1 If all else fails, use "signi cant at a p > 0.05 level" and hope no one notices. Image courtesy of xkcd, number 1478.

F 2 :

 2 "I am Tom, the solution is Jerry: before I catch Jerry, tons of heavy tools fell on me, and it hurt". Image courtesy of xkcd, number 1364.The cost function of the Contrastive-NMF is formulated as: , S a ≥ 0h k: 2 = 1, s k: 2 = 1. (6.1) where X ∈ R M ×N +is the magnitude spectrogram of the mixture, the columns of W ∈ R M ×K + are interpreted as non-negative audio spectral patterns expected to correspond to di erent sources and the rows of H ∈ R K×N + as their activations. M represents the number of frequency bins, N the number STFT frames and K the number of spectral patterns.

  x mn (log x mn -log WH| mn ) +

H a S T a 2 FF

 2 = Tr[(H a S T a ) T (H a S T a )] = Tr[S a H T a H a S T a ] = Tr[(H u S T a ) T (H u S T a )] = Tr[S a H T u H u S T a ](6.13)

∇ 2 F 2 F 2 F 2 F 2 F 2 F 2 F 2 F 2 F 2 F 2 F 2 F

 222222222222 H (-δ( H a S T a -H u S T a )) = -δ∇ Ha ( H a S T a -H u S T a ), if 1 < k < K a -δ∇ Hu ( H a S T a -H u S T a ), if K a + 1 < k < K (6.14) • Tr(ABC) = Tr(BCA) = Tr(CAB) • ∇ X Tr(XAX T ) = X(A T +A) • (X T Y) T = Y T X ∇ Ha ( H a S T a -H u S T a )) = ∇ Ha H a S T a = ∇ Ha Tr[S a H T a H a S T a ] = = H a (S T a S a ) + H a (S T a S a ) T = = 2H a S T a S a (6.15) ∇ Hu ( H a S T a -H u S T a )) = -∇ Hu H u S T a = -∇ Hu Tr[S a H T u H u S T a ] = = -(H u (S T a S a ) + H u (S T a S a ) T ) = = -2H u S Ta S a (6.16)

  coordinated, which led to the release of a short video explaining in simple terms what Music Information Research (MIR) is all about. This part is not strictly related to the research topic of the thesis but the more general problem of science communication and dissemination.

F 3 :

 3 Curious phenomenon: when you are in charge, it comes the time when equally annoying solutions looks easier and funnier. Image courtesy of xkcd, number 2119. S Sharing your research can be very challenging. Sometimes you may need to target a broader audience than simply the colleagues in your particular research eld. Colleagues in other communities or disciplines are already less likely to read about your work. When it comes to sharing your research with the general public, things become even more di cult.

F 4 :

 4 Examples of scienti c dissemination projects. Image courtesy of Scienseed 101

F 5 :

 5 Number of views per day (log) x video length (min). Plot courtesy of Velho et al. [Velho et al. 2020].

F 6 :

 6 Extract from the MIP-frontiers dissemination video F 8: Extract from the MIP-frontiers dissemination video complex concepts. In the end, we reached a compromise trying to represent as extensively as possible some MIP applications.

Titre:

  S éparation de sources musicales neuroguid éeMots cl és : S éparation de sources audio, d écodage de l'attention auditive, EEG, traitement multimodal R ésum é : Dans cette th èse, nous abordons le d éfi de l'utilisation d'interfaces cerveau-machine (ICM) sur l'application sp écifique de la s éparation de sources musicales qui vise à isoler les instruments individuels qui sont m élang és dans un enregistrement de musique. Ce probl ème a ét é étudi é pendant des d écennies, mais sans jamais consid érer les ICM comme un moyen possible de guider et d'informer les syst èmes de s éparation. Plus pr écis ément, nous avons étudi é comment l'activit é neuronale caract éris ée par des signaux électroenc éphalographiques (EEG) refl ète des informations sur la source à laquelle on porte son attention et comment nous pouvons l'utiliser pour informer un syst ème de s éparation de sources. Tout d'abord, nous avons étudi é le probl ème du d écodage par l'EEG de l'attention auditive d'un instrument sp écifique dans une pi èce musicale polyphonique, en montrant que l'EEG suit les caract éristiques musicales pertinentes qui sont fortement corr él ées avec la repr ésentation temps-fr équence de la source à laquelle on porte l'attention et seulement faiblement corr él ées avec les autres. Ensuite, nous avons exploit é ce "contraste" pour informer un mod èle de s éparation de sources non supervis é bas é sur une nouvelle variante de factorisation en matrices positives (NMF), ap-pel ée contrastive-NMF (C-NMF) et s éparer automatiquement la source à laquelle on porte l'attention.

  Motivation and objective . . . . . . . . . . . . . . . . . . . . . . 1.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Contributions and thesis outline . . . . . . . . . . . . . . . . . . 1.3.1 Chapters summary . . . . . . . . . . . . . . . . . . . . . 1.4 List of publications . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Vademecum . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	Contents Notations 1
	x G x x X x i 1.1 Introduction estimated value of x scalars vector matrix i-th entry of x	v vii xi xvi xix
	X T x p 1.1 M	Transpose of matrix X p norm of a vector x.	xxiii
	x 2 F	squared Frobenius norm of a vector x.
	I INTRODUCTION I Identity matrix R set of real numbers	Part I
	R +	set of real nonnegative numbers
		INTRODUCTION
	S
	x(t)	time domain mixture signal
	X(f, n) magnitude spectrogram of x(t)
	X(f, n) complex spectrogram of x(t)
	s j (t)	time domain signal of source j
	s a (t)	time domain signal of the attended source
	s u (t)	time domain signal of the unattended source
	r i (t)	time domain signal of the i-th EEG channel
			xxiii

1 3 1.1 Motivation and objective . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Contributions and thesis outline . . . . . . . . . . . . . . . . . . 11 1.4 List of publications . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.5 Vademecum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 II DECODING OF AUDITORY ATTENTION TO MUSIC I t discrete time index in {1, . . . , T } n frame index in {1, . . . , N } f discrete frequency index in {1, . . . , M } i EEG channel index in {1, . . . , C} j audio source index in {1, . . . , J} k feature index in {1, . . . , K} τ time lag index in {1, . . . , L}

2 mad-eeg: 2

  .1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Related works . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	2.3 Dataset creation . . . . . . . . . . . . . . . . . . . . . . . . . . .
	2.3.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . .
	2.3.2 Stimuli . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	2.3.3 Recording protocol . . . . . . . . . . . . . . . . . . . . .
	2.3.4 Data Acquisition and Preprocessing . . . . . . . . . . .
	2.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	3 maad:
	3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	3.2 Related works . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	3.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	3.3.1 Audio Feature Extraction . . . . . . . . . . . . . . . . . .
	3.3.2 Temporal Response Function . . . . . . . . . . . . . . .
	3.3.3 Regularization . . . . . . . . . . . . . . . . . . . . . . . .
	3.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	3.4.1 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . .
	3.4.2 Experimental Results . . . . . . . . . . . . . . . . . . . .
	3.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

• Cantisani

Table 4

 4 .1 were obtained with K = 16, µ = β =

	SDR [dB]	5 10 15 20	* n.s.	n.s. n.s. ensemble = duo n.s. n.s.	n.s. n.s.	n.s. n.s.	n.s. n.s.	n.s. ensemble = trio n.s. n.s. n.s.	n.s. n.s.	*	*	method NMF C-NMF-r C-NMF-e
		0
		5	Oboe	Flute	Horn	Cello	Bassoon	Oboe	Flute	Horn	Cello	Bassoon
	SDR [dB]	0 10 20	** n.s.	* n.s. ensemble = duo **** ****	**** n.s.	n.s. *	n.s. ensemble = trio ** n.s. n.s.	**** **	method NMF C-NMF-r C-NMF-e
		10
			Guitar	Voice	Drums	Bass	Guitar	Voice	Drums	Bass

  Inter and intra-subject variability in duets: the SDR results are expressed in dB and di erent nuances of pink indicate di erent subjects.

		10 15					1 2	Subject 3 4	5 7	8 9
	SDR [dB]	0 5
		5
		Voice	Drums	Bass	Oboe	Flute	Horn	Cello	Bassoon
	F	4.7:

  . One can see that there is little SDR, SIR, SAR expressed in dB: median over frames, median over tracks for di erent ne-tuning strategies and di erent instruments in the dataset. #TP stands for the number of trainable parameters which are ne-tuned during adaptation.

			other			bass			drums	vocals
		#TP	SDR SIR	SAR SDR SIR	SAR SDR SIR	SAR SDR SIR	SAR
	P-L1:D 8.2M	7.1	12.9 9.4	3.2	15.6 1.1	10.7 18.3 13.8 9.8	13.7 12.6
	P-L2:D 5.6M	7.7	13.0 9.6	4.1	16.8 2.2	10.9 18.7 13.9 9.9	13.7 12.7
	P-L3:D 2.9M	7.9	13.1 9.8	6.0	15.6 5.8	11.3 19.8 14.0 9.9	13.8 12.7
	P-L4:D 0.4M	7.9	13.9 10.0 6.0	15.2 5.8	11.4 20.4 14.2 10.0 14.1 12.5
	P-L5:D 0.01M 7.9	14.3 10.0 6.0	15.3 5.8	11.5 20.5 14.2 10.1 14.2 12.5
	B0	-	8.1	14.4 9.9	6.3	15.3 7.0	10.7 21.4 12.3 8.4	12.5 12.5
	T	5.2:

The reader can refer to the paper review of Kaya and Elhilali [Kaya and Elhilali 2017] for a nice and concise review.

Many studies had already shown that violations of music expectations, for instance, outof-key notes embedded in chords[START_REF] Koelsch | Brain indices of music processing:"nonmusicians" are musical[END_REF]], unlikely chords[START_REF] Koelsch | Untangling syntactic and sensory processing: An ERP study of music perception[END_REF], elicit consistent ERPs. However, to elicit ERPs one requires substantial violations, which the listener can consider as a musician's mistakes, while even the valid sequential events in a given musical culture do not have the same probability of occurring[START_REF] Pearce | The construction and evaluation of statistical models of melodic structure in music perception and composition[END_REF][START_REF] Pearce | Auditory expectation: the information dynamics of music perception and cognition[END_REF][START_REF] Temperley | A probabilistic model of melody perception[END_REF][START_REF] Rohrmeier | Statistical properties of tonal harmony in Bach's chorales[END_REF][START_REF] Temperley | Statistical analysis of harmony and melody in rock music[END_REF]. Thus, the associated expectation can vary accordingly in the full expectation strength range, and this could only be studied using continuous naturalistic musical stimuli[Di Liberto et al. 2020a].

r 1 (N ) 0 r 1 (1) r 1 (2) . . . . . . . . . r 1 (N -1) . . . . . . . . . . . . . . . . . . . . . 0 . . . . . . . . . r 1 (1) . . . r 1 (N -L) . . . . . . . . . . . . . . . . . . . . . r C (1) r C (2) r C (3) . . . . . . . . . r C (N ) 0 r C (1) r C (2) . . . . . . . . . r C (N -1) . . . . . . . . . . . . . . . . . . . . .

https://github.com/

https://github.com/ giorgiacantisani/randomization-test

Acknowledgements

I would like to thank all the people that believed in me and were co-protagonists of this adventure. Your technical and emotional support was fundamental, and I would like to take this occasion to express my gratitude to:

Part III NEURO-STEERED MUSIC SOURCE SEPARATION

The dataset was acquired by my colleague Gabriel Trégoat during

https://github.com/ webermarcolivier/statannot Drawings and schemes are made with Excalidraw 9 and draw.io. 10 Icons are 9 https://github.com/excalidraw 10 https://www.diagrams.net/ downloaded from the www.flaticon.com/ The bibliography was processed by Biblatex.

E

In this work, we considered the implementation of ConvTasnet for multichannel music separation provided by [START_REF] Défossez | Music source separation in the waveform domain[END_REF]. The weights of the model pre-trained on the MUSDB18 dataset 3 were downloaded from the 3 The MUSDB18 dataset [START_REF] Ra | The MUSDB18 corpus for music separation[END_REF]] consists of 150 full-length music stereo tracks of various genres sampled at 44.1 kHz. For each track, it provides a linear mixture along with the isolated tracks for the four categories: drums, bass, vocals, and others. The "others" category contains all other sources in the mix that are not the drums, bass, or vocals. author's Github page 4 , where the reader can nd further details about the 4 https://github.com/ facebookresearch/demucs model implementation. The model is built to separate the mixture into four tracks associated with the categories drums, bass, vocals, and others.

To adapt the network to each test mixture we ne-tuned it for 10 epochs on 4-second-long segments extracted from the mixture. The initial learning rate was set to 10 -5 , batch size to 1 and Ranger was used as the optimizer. Speci cally, Ranger combines RAdam [START_REF] Liu | On the variance of the adaptive learning rate and beyond[END_REF] and LookAhead [START_REF] Zhang | Lookahead optimizer: k steps forward, 1 step back[END_REF]] optimiser together. Our source code is publicly available. 6 W , we explored how to inform and guide a Music Source Separation system exploiting previously not considered modalities such as the user's selective auditory attention to a source characterized in terms of his/her neural activity. Speci cally, we investigated two main problems which are intrinsically intertwined with each other: In this chapter, we will summarize the principal ndings of the current investigation, and present a discussion on future perspectives and research directions. We hope the results and ideas investigated in this dissertation will stimulate and encourage novel works in this fascinating research direction.

S

After introducing the motivation and objective behind this work, in Part I, the contributions of this thesis were presented in Part II and Part III, elaborating on the two problems above. The pursuit of these goals led into the following contributions and outcomes: P II A

• MAD-EEG We assembled a music-related EEG dataset which allows for studying the problems of single-trial EEG-based AAD and EEG-guided MSS for realistic polyphonic music. It represents the rst dataset of its kind for music stimuli and can also be di erentiated from those commonly used for studying AAD for speech stimuli. The proposed experimental setting di ers from the ones previously considered as the stimuli are polyphonic and are played to the subject using speakers instead of headphones. MAD-EEG represents our rst main contribution and is available to the research community as a free resource.

Processing (MIP), which was quite hard. The trick we found was to trace back the history of a song that an imaginary inhabitant of the future is listening to. We managed to derive a circular story following the song from composition to recording and from distribution to the user experience. Therefore, the music is the backbone of the video, and its choice was crucial.

M

When preparing a motion graphic, you need to provide to the visual designers a script (description of the scenes), the voiceover (text that an actor needs to read and which describes the scene), and the background music. With those three elements, the visual designers built an animation on which you can then give feedback and adapt the voiceover and the music again. This process is reiterated repeatedly until convergence and everyone is happy with the result.

In our case, an additional di culty was that the music wasn't just some "background" music. It was, on the contrary, the absolute protagonist that mainly contributes to conveying the main message. The music evolves throughout the video and changes according to the MIR application we wanted to illustrate. All of this needs a not negligible e ort of synchronization and composition. In Figure 7, you can see an extract of the script I prepared with the musicians.

Regarding the voiceover, we quickly realized how few words can t a 3minutes-long video. More importantly, we learned how hard it can be to summarize the vast diversity of research in our community. Moreover, there are synchronization constraints that impose a xed number of words to express Can you imagine how listening to music might be in the future?

A guy sitting on a space chair in his ultra-modern living room inside the spaceship is listening to a piece of music. Maybe a cat sleeps on his legs (e.g. lofi girl Ahsoka / Cyberpunk )?

And what about the process of creating it?

Zoom out of the spaceship and zoom in another galaxy-earth-continent-country-city-dirty rehearsal room where a band is creating a song 2. Creation : (musicians, MIR avatar) Jamming 00:12 -00:36

Nowadays music results from a creative process that starts with an original idea and culminates in releasing a song.

The truth is: creating music can be very hard.

Luckily, science can support musicians in such a process.

The band is jamming in the rehearsal room (e.g. garage): a song is sketched, but a lot of errors and noises occur (e.