Samuele Cornell

Michel Olvera
email: michel.olvera@inria.fr

Manuel Pariente
email: manuel.pariente@inria.fr

Giovanni Pepe
email: g.pepe@pm.univpm.it@l.gabrielli

Emanuele Principi
email: e.principi@l.gabrielli

Leonardo Gabrielli

Stefano Squartini
email: s.squartini@univpm.it

Domain-Adversarial Training and Trainable Parallel Front-end for the DCASE 2020 Task 4 Sound Event Detection Challenge

Keywords: Sound Event Detection, Domain Adversarial Training, Per-Channel Energy Normalization, Semi-Supervised Learning

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Sound Event Detection (SED) is the task of recognizing the set of active sound events, as well as their onset and offset time, in a given audio recording. Such task is useful for a wide variety of applications such as acoustic monitoring, human-computer interaction, and meeting room transcription [START_REF] Virtanen | Computational analysis of sound scenes and events[END_REF]. Current state-of-the-art methods [START_REF] Piczak | Environmental sound classification with convolutional neural networks[END_REF] for tackling SED are largely based on supervised-learning approaches with Deep-Neural-Networks (DNNs) in which a DNNbased classifier is trained using a strongly-labelled dataset of, possibly co-occurring, audio events. However, acquiring and creating such amount of strongly-labelled data is a costly and tedious procedure as it requires human annotation of onset and offset times for each audio event. On the other hand, weakly-labelled data, for which only the set of active sound events is given, can be acquired much more easily. For this reason, there is a growing interest towards using, less costly, weakly-labelled data and unlabelled data *Equal contributions Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).

to train SED systems and thus decrease the reliance on stronglyannotated data via Semi-Supervised Learning.

In the wake of this trend, several works have recently appeared in the literature that propose semi-supervised methods for SED with weak labels. As a further confirmation of the trend, recent DCASE editions have focused more and more on using weakly and unlabeled data for SED. The majority of the works proposed in the literature for this task are based on Convolutional Neural Networks (CNNs) [START_REF] Mcfee | Adaptive pooling operators for weakly labeled sound event detection[END_REF] or Convolutional Recurrent Neural Networks (CRNNs) [START_REF] Parascandolo | Convolutional recurrent neural networks for polyphonic sound event detection[END_REF][START_REF] Wei | A-crnn: A domain adaptation model for sound event detection[END_REF][START_REF] Yan | Task-aware mean teacher method for large scale weakly labeled semi-supervised sound event detection[END_REF] coupled with a Multiple Instance Learning pooling method [START_REF] Wang | A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling[END_REF] for dealing with weak labels and a consistency loss for exploiting unlabeled data [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semisupervised deep learning results[END_REF]. The winner of the 2019 edition of the DCASE Task 4 challenge used guided learning [START_REF] Lin | Guided learning for weakly-labeled semi-supervised sound event detection[END_REF] where two networks are updated simultaneously and the teacher network is larger than the student network. The consistency loss in this case is represented by the cross-entropy between the student outputs and the teacher pseudo-labels.

In this work, we outline our systems and techniques for SED in the context of the DCASE 2020 Task 4 challenge. As classifier, we used the the baseline framework composed of a CRNN coupled with Mean Teacher [START_REF] Delphin-Poulat | Mean teacher with data agumentation for dcase 2019 task 4[END_REF]. Here, however, we explored the possibility of using Domain Adversarial Training [START_REF] Ganin | Domainadversarial training of neural networks[END_REF] for dealing with the different data domains composing the training set and we employed an online augmentation strategy to increase the amount of acoustic diversity of the data by dynamic mixing of synthetic examples, artificial reverberation and various time-frequency transformations. Moreover, we propose a parallel Per-Channel Energy Normalization (PCEN) [START_REF] Wang | Trainable frontend for robust and far-field keyword spotting[END_REF] layer applied to the input features as a trainable dynamic compression strategy. Finally, we replaced the common median filter used to smooth the outputs of the classifier with a Hidden Markov Model-based smoothing.

This paper is organized as follows: Section 2 gives an overview of the DCASE 2020 Task 4 challenge. In Section 3 we outline our proposed techniques to tackle the Sound Event Detection problem (DCASE 2020 Task 4 Scenario 1). We present an in-depth explanation of such proposed techniques in Sub-Sections 3.1 to 3.3 and then in Section 4 we perform an in depth ablation study and show the results obtained. Finally, in Section 5 we draw conclusions and trace possible future research directions.

THE DCASE 2020 TASK 4 CHALLENGE

In this work, we conducted experiments and tried new techniques in the context of The DCASE 2020 Task 4 Sound Event Detection Challenge. The DCASE 2020 Task 4 challenge offers the opportunity to tackle Sound Event Detection (SED) in domestic environments facing real-world issues such as weakly-annotated data, unlabeled data and only a very small corpus of strongly annotated, synthetic data. More specifically, the ultimate goal of the challenge is to develop a SED system being able to tag onset and offset of 10 different sound events classes: Speech, Dog, Cat, Alarm bell/ringing, Dishes, Frying, Blender, Running water, Vacuum cleaner, Electric shaver/toothbrush. The datasets made available are unbalanced and diverse. Specifically, the DESED [START_REF] Turpault | Sound event detection in domestic environments with weakly labeled data and soundscape synthesis[END_REF] dataset offers, for training, weakly labelled and unlabelled real soundscapes, and isolated synthetic events with strong labels. The SINS [START_REF] Dekkers | The sins database for detection of daily activities in a home environment using an acoustic sensor network[END_REF] and TUT Acoustic scenes 2017 [START_REF] Mesaros | TUT database for acoustic scene classification and sound event detection[END_REF] datasets offer background noise. The FUSS [START_REF] Wisdom | What's all the fuss about free universal sound separation data?[END_REF] source separation dataset offers isolated events but no annotations. Moreover it does come from a completely different domain as it is aimed at arbitrary sound separation [START_REF] Kavalerov | Universal sound separation[END_REF].

PROPOSED TECHNIQUES

In this section we illustrate the techniques we employed in our submitted systems for DCASE 2020 Task 4 Sound Event Detection challenge. We started from the baseline [START_REF] Delphin-Poulat | Mean teacher with data agumentation for dcase 2019 task 4[END_REF] and kept the CRNNbased architecture as well as the mean-teacher training scheme with same network and optimization hyper-parameters 1 . As said, our main contributions are in the training procedure, feature preprocessing and in prediction post-processing and smoothing.

Domain-Adversarial Training

The training dataset includes data from different domains, including real and synthetic ones. Moreover, strong annotations are provided only for synthetic data and only a portion of real recording is provided with weak annotations. Instead, test and development set include only data from real-world recordings. This is problematic, as the training and the target domains are different and the SED system must generalize well enough by learning from the synthetic out of domain examples. Domain Adversarial Training [START_REF] Ganin | Domainadversarial training of neural networks[END_REF] (DAT) provides a solution to this by enforcing a model to learn features that are invariant to the change of domains. This is achieved by embedding the domain adaptation process into the training procedure by adding, to the original architecture, a branch with a gradient reversal layer followed by a domain classifier. The added branch is only used at training time and then dropped at test-time, so there is no computational overhead at run-time. During training, both the network and the added domain classifier are jointly optimized. The gradient reversal layer encourages the feature extraction stage of the original architecture to work adversarially to the added domain classifier by extracting features that are domain-invariant and thus maximize the loss of the domain classification task. We thus employ DAT to enforce learning of features invariant between the synthetic examples domain and real-world recordings domain, reducing in this way the chance of overfitting the strong-labelled synthetic examples.

For the adversarial branch, we employed a modified version of Conv-TasNet [START_REF] Luo | Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF] separator network available in the Asteroid source separation toolkit [START_REF] Pariente | Asteroid: the pytorch-based audio source separation toolkit for researchers[END_REF]. In our modified version, instead of outputting a mask for each transformed-domain feature bin, the separator network outputs a probability on the whole input example by using mean pooling. In fact, the network must classify whether the 1 Code for replicating this work is publicly available at https:// github.com/popcornell/UNIVPM-INRIA-DCASE2020. The CRNN and adversarial branches, are then updated in two different steps adversarially. We denote with Lmain the loss for the CRNN training, comprised of strongly labeled loss, weak labeled loss and consistency loss between teacher and student. Thus for the CRNN the update rule for its parameters θc is:

θc ← -θc -α λ ∂Lmain ∂θc -(1 -λ) ∂L adv ∂θc , (1)
where L adv is the binary cross-entropy loss for the adversarial network, λ is an hyper-parameter which controls the relative magnitude of the two losses and α is the learning rate. Differently, for the adversarial network with parameters θa the update rule is:

θa ← -θa -α(1 -λ) ∂L adv ∂θa . (2)
In our implementation we did not use the gradient reversal layer proposed by [START_REF] Ganin | Domainadversarial training of neural networks[END_REF], but a two step optimizing procedure like the one used in Generative Adversarial Networks [START_REF] Goodfellow | Generative adversarial nets[END_REF]. We found this twostep approach to give better results than the gradient reversal layer approach, as it leads to more stable gradients during training. We tuned λ on the development set and found that a value of 0.1 gave the best results.

Dynamic Mixing and Augmentation

Because of the limited amount of acoustic diversity in DESED synthetic examples, we also employed an online augmentation strategy. Each synthetic training example is constructed at training time by randomly sampling from one to five random foregrounds and one background file from SINS. We apply reverberation to each source independently by using FUSS [START_REF] Wisdom | What's all the fuss about free universal sound separation data?[END_REF] Room Impulse Responses (RIRs). Then we apply a random time-domain augmentation chain with different effects to each source, with a maximum of two random cascaded effects: additive noise bursts, additive sine bursts, time-varying comb filters, compression, pitch shifting, lowpass and high-pass filtering.

Finally we mix the foregrounds and background. The level for each foreground is randomly sampled between -35 dB and 0 dB while the background is constrained to be at max 5 dB over the foreground which has minimum level. On the feature domain, we add Gaussian noise with SNR between -30 dB and 10 dB and we employ SpecAugment [START_REF] Park | Specaugment: A simple data augmentation method for automatic speech recognition[END_REF]. This procedure ensures a virtually infinite amount of different strongly labelled data.

For weak and unlabeled data, we use a slightly different augmentation scheme as the foregrounds and backgrounds are not available. We only randomly add an additional background from SINS to the original mixture with 50% probability and employ only the aforementioned feature domain augmentations. In fact, we found that using time-domain augmentations on this data worsened the performance as the network failed to generalize to development set when weak and unlabeled data was strongly augmented.

Parallel Per-Channel Energy Normalization

We experimented with Per-Channel Energy Normalization (PCEN) [START_REF] Wang | Trainable frontend for robust and far-field keyword spotting[END_REF] as a trainable dynamic compression strategy. This technique is able to enhance transient audio events while transforming many soundscape noise patterns into additive white Gaussian noise, improving the robustness of audio classification algorithms in presence of background noise [START_REF] Lostanlen | Per-channel energy normalization: Why and how[END_REF] with minimal computational overhead. It is defined as

PCEN(t, f ) = E(t, f ) ( + M (t, f )) α + δ r -δ r , (3)
where t and f denote time and Mel frequency band index, α, , r and δ are positive constants and E(t, f ) denotes filter bank energy used as feature representation. M (t, f ) is a smoothed version of E(t, f ), which is computed using a first-order infinite impulse response filter (IIR) as M (t, f ) = (1 -s)M (t -1, f ) + sE(t, f ), with s the smoothing coefficient. While the PCEN operation can be helpful to enhance some sound events in domestic environments, the filtering operation involved in (3) can have a negative impact in sound classes with slowly varying spectro-temporal characteristics, for instance, vacuum cleaner or blender events [START_REF] Olvera | Foregroundbackground ambient sound scene separation[END_REF]. Therefore, instead of learning the parameters of a single transformation that finds a trade-off between standing out fast transition sounds and not degrading the quality of stationary-like sounds, we propose to learn several PCEN transformations in parallel, using a Parallel-PCEN trainable frontend (PPCEN). We hypothesize that each individual layer can focus on the specifics of certain groups of sounds. The output of each layer is given as feature channels to the CRNN model and jointly optimize the parameters of such PPCEN front-end layers using backpropagation.

In trainable PCEN we jointly optimize parameter α, δ and r, but instead of learning the smoothing coefficient s, we predetermine two smoothing coefficients s1 = 0.015 and s2 = 0.25 and learn a combination of the smoother outputs.

In Figure 2 we can see the output of our proposed 2-layers PP-CEN front-end when it is fed an example with speech and vacuum cleaner events. The vacuum cleaner is present in the whole audio clip, while speech appears in three separate sound events. It can be seen that the first PCEN layer captures also more slow-varying events. In fact, the background noise and the vacuum cleaner harmonics can be clearly distinguished and are enhanced with respect to the original log-Mel features. On the other hand, the second PCEN layer focuses only on events with faster onset such as speech.

Hidden-Markov-Model Smoothing

A popular approach used to reduce the number of spurious detections is to apply median filtering to the outputs of the classifier. Here, instead, final predictions are obtained by using a Hidden Markov Model (HMM) with two states for each class. The silence self-loop transition probability was tied to be the same for all HMMs. We tuned the self-loop transition probabilities for every class and silence on the development set using a 50% split by using Random Forest and with the objective of maximizing the eventbased F1 macro-average score of the trained SED model. Once found the optimal parameters for the HMMs transition probabilities, inference is performed by running Viterbi decoding on the CRNNobtained emission probabilities for each class. This approach is different to what has been proposed in [START_REF] Hayashi | Bidirectional lstm-hmm hybrid system for polyphonic sound event detection[END_REF] as we employ only two states for each sound class HMM and, instead of fixing transition probabilities and learning corresponding emission probabilities, as in a full fledged hybrid DNN-HMM system, we tune the HMMs transition probabilities with fixed emission probabilities obtained by the pre-trained SED classifier.

RESULTS AND DISCUSSION

Hereafter we report our results obtained on DCASE2020 Task 4 SED for our proposed techniques and systems. Each system we have submitted is comprised of combinations of aforementioned techniques applied to the baseline system.

Performance of Different Acoustic Front-Ends

In Table 1 we report performance results on the development set according to different acoustic front-ends. As PCEN strictly requires non-negative inputs, we perform global instance min-max scaling instead of z-score normalization over the whole dataset as in the baseline system and no data augmentation was used. The same configuration was used for log-Mel energies. Fixed PCEN 1 uses parameters = 10 -6 , α = 0.98, δ = 2 and T = 400ms. This setup degrades sound classes intrinsically stationary yielding a sub-optimal performance. Fixed PCEN 2 uses same parameters except for α, which is set to 0.8. This setup no longer suppresses slower transients and performance is similar to log-Mel. Fixed PCEN 3 also uses α = 0.8, but the time constant is set to 60 ms. This configuration seems to favor all sound classes in the task and brings a modest improvement over log-Mel.

By jointly optimizing the PCEN parameters with the cost objective (Trainable PCEN), a slightly better score than Fixed PCEN 3 is achieved. Training a second layer on parallel (PPCEN on Mel) brings a little more than 1% improvement over a single PCEN layer. This setup enhances sounds with faster modulations of foreground in one layer and sounds with slower modulations in the second one. Finally, a similar setup with log-Mel energies (PPCEN on log-Mel) achieves the best performance over the common log-Mel front-end by 3.41%. We found that two layers were sufficient to give significant performance improvement and that adding more layers did not bring additional appreciable improvement.

Ablation Study

In Table 2 we compare results obtained on development set by the challenge baseline system and results obtained by adding the proposed techniques to the challenge baseline.

We can see that PPCEN alone is able to bring substantial improvement in performance. Instead, online mixing and augmentation (Augm) brings modest performance improvement on its own. We suspect this is due to the fact that the online generated examples are only partially representative of true target-domain sound events and thus the network fails to generalize to real-world data. It however brings significant benefits when it is coupled with DAT. When PPCEN is combined with data-augmentation we observe a drop in performance. We hypothesize this is due to the fact that strong augmented synthetic examples have a significant different spectro-temporal characteristics and, by fitting the PPCEN also on this data, generalization to real-world data can be hampered.

On the other hand, HMM smoothing alone is able to constantly give at least two points performance improvement on all systems.

Finally adding all proposed techniques to the baseline system yields a significant improvement of more than 10 % points over the baseline model, without significant additional computational overhead. It is worth noting that adding PPCEN on top of DAT leads only to a modest improvement while adding PPCEN on top of the baseline significantly improves the results. This is due to the fact that, PPCEN seems to be affected negatively by data-augmentation and DAT is only partially able to overcome this by forcing PCEN layers produce same output distribution for augmented synthetic and real-world examples.

Challenge Results

Hereafter, in Table 3, we report results obtained by our submitted systems on the development and evaluation sets. We submitted a total of four different SED systems: two single systems and two ensemble systems.

Regarding single-systems, we submitted one system with PP-CEN and HMM (PPCEN+HMM), and another one with DAT and HMM (DAT+HMM). Both improve substantially over the baseline system, with DAT+HMM achieving the highest score both for development but not for test. In fact the PPCEN+HMM system appears to generalize slightly better to evaluation set. We did not submit the combination of PPCEN+DAT+HMM reported in Table 3 and discussed in previous Section 4.2 due to lack of time.

We also submitted two ensemble systems derived from a combination of PPCEN and DAT systems. For these two submissions we used the same models. The only difference is in different HMM transition probabilities. We used an ensemble of three different single systems: PPCEN and two DAT models from two different training runs. To obtain emission probabilities we simply averaged the outputs of the different models. It can be seen that the second ensemble model (DAT+PPCEN+HMM 2) achieves higher score on development but has worse performance on test. This shows that HMM transition probabilities tuning can have a substantial impact on the final system performance and can be prone to overfitting.

CONCLUSIONS

In this work we outlined our proposed techniques for tackling the 2020 DCASE Task 4 challenge. Instead of focusing on exploring new neural architectures we experimented directly with the baseline SED system. We showed that, the addition of PPCEN frontend feature pre-processing, Domain Adversarial Training and online data augmentation and mixing can bring substantial benefits with minimal computational overhead at inference time. As another, parallel, contribution we also showed that HMM smoothing alone can greatly improve performance of the systems by refining network predictions. Future work could explore how the proposed techniques fare when coupled with a stronger baseline system.

Figure 1 :

 1 Figure 1: Domain adversarial training scheme. input example belongs to synthetic examples or to weak/unlabeled examples. The fully convolutional architecture of Conv-Tasnet along with skip connections helped gradient propagation from discriminator to the main network. The adversarial branch was placed in parallel to the RNN block after the CNN layers in the CRNN architecture. The adopted training scheme is illustrated in Figure 1.The CRNN and adversarial branches, are then updated in two different steps adversarially. We denote with Lmain the loss for the CRNN training, comprised of strongly labeled loss, weak labeled loss and consistency loss between teacher and student. Thus for the CRNN the update rule for its parameters θc is:

Figure 2 :

 2 Figure 2: Output of the PPCEN layer: (a) original mixture log-Mels, (b) first PCEN layer, (c) second PCEN layer. The two parallel layers capture different spectro-temporal dynamics.

Table 1 :

 1 Performance comparison per acoustic front-end.

	Acoustic front-end	Event macro F 1 score
	Log-Mel	35.93
	Fixed PCEN 1	28.71
	Fixed PCEN 2	35.42
	Fixed PCEN 3	36.84
	Trainable PCEN	36.92
	PPCEN on Mel	38.20
	PPCEN on log-Mel	39.34

Table 2 :

 2 Ablation study for proposed techniques (development set).

	Method	Event macro F 1 score
	Baseline	34.8
	+HMM	37.13
	+PPCEN	39.93
	+PPCEN+HMM	43.69
	+Augm	37.31
	+DAT+Augm	40.91
	+PPCEN+Augm	33.20
	+DAT+Augm+HMM	45.20
	+PPCEN+DAT+Augm+HMM	46.30

Table 3 :

 3 Performance on development and evaluation sets.

	Method	Event macro F 1 score	PSDS
		dev	eval	dev
	Baseline	34.8	34.9	0.61
	PPCEN+HMM	43.69	42.6	0.63
	DAT+HMM	45.20	42.0	0.68
	Ensemble DAT+PPCEN+HMM	46.17	44.4	0.69
	Ensemble DAT+PPCEN+HMM 2	47.44	43.2	0.69


Samuele Cornell

Michel Olvera
email: michel.olvera@inria.fr

Manuel Pariente
email: manuel.pariente@inria.fr

Giovanni Pepe
email: g.pepe@pm.univpm.it@l.gabrielli

Emanuele Principi
email: e.principi@l.gabrielli

Leonardo Gabrielli

Stefano Squartini
email: s.squartini@univpm.it

Task-Aware Separation for the DCASE 2020 Task 4 Sound Event Detection and Separation Challenge

Keywords: Source Separation, End-to-End, Sound Event Detection, Time Domain, Joint Training

published or not. The documents may come

INTRODUCTION

Source separation aims at extracting from an acoustic mixture its underlying acoustic components. In recent years, Deep-Neural-Networks (DNN) based methods have made significant progress towards this goal especially in the speech processing field [START_REF] Hershey | Deep clustering: Discriminative embeddings for segmentation and separation[END_REF][START_REF] Luo | Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF] but also regarding the more general problem of separating arbitrary sounds [START_REF] Kavalerov | Universal sound separation[END_REF]. In the speech processing field, source separation has a wide variety of applications such as hearing aid devices [START_REF] Reindl | Speech enhancement for binaural hearing aids based on blind source separation[END_REF], Automatic Speech Recognition [START_REF] Boeddeker | Front-end processing for the chime-5 dinner party scenario[END_REF][START_REF] Menne | Analysis of deep clustering as preprocessing for automatic speech recognition of sparsely overlapping speech[END_REF][START_REF] Neumann | End-to-end training of time domain audio separation and recognition[END_REF] (ASR) and diarization [START_REF] Neumann | All-neural online source separation, counting, and diarization for meeting analysis[END_REF]. At the same time, arbitrary sound separation also has potential applications in hearing devices and in creative applications such as video editing.

A recent work [START_REF] Tzinis | Improving universal sound separation using sound classification[END_REF] has demonstrated that sound classification can be used to improve the performance of an arbitrary sound separation algorithm but the inverse problem of whether a source separation algorithm can be used to improve sound classification and *Equal contributions Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). more in general Sound Event Detection (SED) is still a matter of research. In fact, answering such a question is one of the goals of the 2020 Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 Task 4 Sound Event Detection (SED) and Separation challenge. To motivate participants to explore this direction and develop a combined source separation and SED system the challenge organizers have provided a baseline sound separation model and a dataset, the Free Universal Sound Separation (FUSS) [START_REF] Wisdom | What's all the fuss about free universal sound separation data?[END_REF], suitable for training such arbitrary sound separation algorithm.

In this paper, we propose a new technique for combined source separation and SED and show it can improve the back-end SED performance on DCASE 2020 Task 4 development set. In our approach, instead of optimizing separately the separation system with a loss function focused only on source separation, we leverage the pre-trained back-end SED system to optimize the separation system also with the SED objective. In this way, we assure that separation will be helpful to the back-end SED system and will not introduce a mismatch which can deteriorate the performance. This approach, is inspired by previous techniques aimed at improving ASR performance with Source Separation [START_REF] Narayanan | Improving robustness of deep neural network acoustic models via speech separation and joint adaptive training[END_REF][START_REF] Settle | End-to-end multi-speaker speech recognition[END_REF][START_REF] Neumann | End-to-end training of time domain audio separation and recognition[END_REF] and Speech Enhancement [START_REF] Watanabe | Studentteacher network learning with enhanced features[END_REF][START_REF] Bagchi | Spectral feature mapping with mimic loss for robust speech recognition[END_REF] by using either joint training or ASR related losses in order to ensure that separation or enhancement will be beneficial to the ASR task. We compare our proposed method with the aforementioned DCASE 2020 Task 4 combined separation and SED baseline and we show that the proposed approach outperforms it with significantly less parameters.

This work is organized as follows: in Section 2 we briefly describe the datasets available in the context of the DCASE2020 Task 4 Sound Event Detection and Separation challenge; in Section 3 we describe the challenge combined source separation and sound event detection baseline and, following, our proposed approach. In Section 4 we show the results obtained, perform an in-depth ablation study to assess the validity of our work, and discuss the strengths and weaknesses of the proposed method. Finally, in Section 5 we draw conclusions and outline possible future research directions.

DCASE 2020 TASK 4 CHALLENGE DATASETS

The DCASE 2020 Task 4 challenge is focused primarily on Sound Event Detection in real-world domestic environments with weaklyannotated data, unlabeled data, and only a very small corpus of strongly annotated, out of domain, synthetic data.

Three datasets are available for training the SED systems: DESED [START_REF] Turpault | Sound event detection in domestic environments with weakly labeled data and soundscape synthesis[END_REF], SINS [START_REF] Dekkers | The sins database for detection of daily activities in a home environment using an acoustic sensor network[END_REF] and TUT [START_REF] Mesaros | TUT database for acoustic scene classification and sound event detection[END_REF]. The SINS and TUT Acous-tic scenes 2017 datasets offer only background noise while the DESED [START_REF] Reindl | Speech enhancement for binaural hearing aids based on blind source separation[END_REF] dataset offers, for training, weakly labeled and unlabeled real soundscapes, and isolated synthetic events with strong labels. It is a dataset primarily made for SED systems training and evaluation. The synthetic portion has background noise derived from SINS and because oracle foregrounds events are available it could also be used for training a source separation algorithm. Regarding validation and evaluation, instead, DESED has only real soundscapes with strong labels in order to assess algorithms generalization to unseen, real-world scenarios. It features a total of 10 different sounds events classes for training a SED system: Speech, Dog, Cat, Alarm bell/ringing, Dishes, Frying, Blender, Running water, Vacuum cleaner, Electric shaver/toothbrush.

As the challenge is also focused on Source Separation, an additional, out of domain dataset is made available for training Source Separation systems: the Free Universal Sound Separation (FUSS) Dataset [START_REF] Wisdom | What's all the fuss about free universal sound separation data?[END_REF]. Being originally aimed at arbitrary sound separation [START_REF] Kavalerov | Universal sound separation[END_REF], it offers isolated events and noise backgrounds but no SED annotations. Moreover, it does come from a completely different domain than DESED as it does not include the same classes but also arbitrary audio events which the SED system should ideally ignore.

COMBINED SOURCE SEPARATION AND SOUND EVENT DETECTION

Hereafter we present and discuss our proposed technique for tackling SED using Source Separation as a pre-processing step in the context of the DCASE 2020 Task 4 Sound Event Detection and Separation Challenge.

In our experiments for combined separation and SED we used the released pre-trained SED baseline system together with our separation system 1 . This allows to directly compare with results reported by the baseline combined source separation and SED system. The baseline SED system is derived from [START_REF] Delphin-Poulat | Mean teacher with data agumentation for dcase 2019 task 4[END_REF] and is based on a hybrid Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) architecture trained using both weakly, strong and unlabeled data by using a Mean-Teacher strategy [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semisupervised deep learning results[END_REF] for Semi-Supervised training. It uses 128 log-Mel features in input taken with a 2048 samples window and 128 hop size. The network predictions are smoothed with a median filter with a 7 frames length.

DCASE 2020 Combined Source Separation and Sound Event Detection Baseline System

The most straightforward approach to perform combined SED and source separation is to train the Source Separation system and SED separately using different objectives. This is also the approach adopted by the baseline source separation system.

The official source separation baseline is derived from [START_REF] Wisdom | What's all the fuss about free universal sound separation data?[END_REF] and is trained on a synthetic dataset comprised of FUSS as well as synthetic examples from DESED. This baseline model is optimized with an End-to-End (E2E) waveform-based Scale-Invariant Signalto-Distortion-Ratio (SI-SDR) [START_REF] Le Roux | Sdrhalf-baked or well done?[END_REF] objective to remove the background noise from the mixtures. Thus it performs denoising rather than full foregrounds separation from the mixtures. The whole approach is illustrated in Figure 1. The network architecture is based on TDCN++ [START_REF] Kavalerov | Universal sound separation[END_REF][START_REF] Tzinis | Improving universal sound separation using sound classification[END_REF] and follows the analysis/masking/synthesis scheme where a DNN is used to estimate a mask in a transformed domain for each source. The masking is performed in Short-Time-Fourier-Transform (STFT) magnitude spectra domain. A potential drawback is that this approach does not guarantee that the denoised mixtures will be more suitable for SED. In fact, the denoising process could lead to mixtures whose distribution is significantly different from the one of noisy mixtures. Because the baseline SED model is trained on noisy mixtures directly, the denoising process can potentially introduce a mismatch. This issue is common in speech processing applications where, for example, Speech-Enhancement denoising is used as a pre-processing step for ASR [START_REF] Watanabe | Studentteacher network learning with enhanced features[END_REF][START_REF] Bagchi | Spectral feature mapping with mimic loss for robust speech recognition[END_REF][START_REF] Wang | Bridging the gap between monaural speech enhancement and recognition with distortionindependent acoustic modeling[END_REF]. This can explain the modest performance improvement given by the baseline separation model.

Moreover, this modest improvement is only reached using ensembling: the SED model is fed both the denoised and noisy audio features and the predictions are averaged. This allows for improvement over the baseline SED-only system but at the expense of raising considerably the computational requirements.

Task-Aware Separation

On the contrary, we depart significantly from this common procedure and propose what we call Task-Aware separation training to address the aforementioned domain mismatch problem that arises when separation/denoising is performed on a system trained on noisy mixtures.

For this reason, it is often preferable to jointly train from scratch the separation system and SED system in an E2E fashion or jointly fine-tune the two pre-trained systems. In this way, the separation system is guaranteed to help the back-end SED system task as it is trained with the SED objective. This method has been shown to be highly effective for joint source separation and ASR [START_REF] Narayanan | Improving robustness of deep neural network acoustic models via speech separation and joint adaptive training[END_REF][START_REF] Settle | End-to-end multi-speaker speech recognition[END_REF][START_REF] Neumann | End-to-end training of time domain audio separation and recognition[END_REF] even surpassing monolithic, multi-talker, Permutation Invariant Training ASR techniques such as [START_REF] Seki | A purely end-to-end system for multi-speaker speech recognition[END_REF][START_REF] Chang | End-to-end monaural multi-speaker asr system without pretraining[END_REF].

Another, more common, procedure to guarantee that the backend SED system is matched to the separated/denoised pre-processed data is to re-train or fine-tune the SED system on this data. However, this is a rather expensive procedure as it requires separate training of separation system and then separate fine-tuning or retraining of the SED algorithm.

Hereafter we present another approach which we call Task-Aware separation, which allows to train a separation system using a pre-trained SED back-end with the SED objective, thus avoiding the domain mismatch problem. A significant advantage of our procedure over joint training is that potentially a robust back-end, pre-trained on a vast amount of data, for which oracle targets for separation are not available, can be directly used.

Our approach is illustrated in Figure 2. As the combined system ultimate goal is SED, we perform Deep Neural Network (DNN) mask-based separation directly on Mel-spectrograms. The separated features are then fed to the pre-trained SED system after applying logarithm and scaling. We then use both the predictions of the SED as well as its internal activations to train the maskestimation DNN network. In the whole process, the back-end SED We then apply reverberation to each source independently by using FUSS Room Impulse Responses (RIRs) and mix the foregrounds and background. The level for each foreground is randomly sampled between -35 dB and 0 dB while the background is constrained to be at max 5 dB over the foreground which has minimum level. FUSS was not employed for training. In fact, we found that adding FUSS did bring a slight performance loss due to the fact that it features a completely different domain than DESED. Moreover, we found that some DESED foreground classes were considered as backgrounds in FUSS.

Permutation Invariant Training (PIT) [START_REF] Yu | Permutation invariant training of deep models for speaker-independent multi-talker speech separation[END_REF][START_REF] Kolbaek | Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks[END_REF] and Mean Teacher [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semisupervised deep learning results[END_REF] are used to train the mask-estimation DNN. Because the sound class are well defined, it can be argued that the use of PIT is not necessary. However, in our preliminary experiments, we found that the separation stack trained with a non-PIT loss led to overfitting the weakly and synthetic examples very quickly. By using PIT this problem is mitigated, as the separation stack is not anymore required to explicitly recognize the sound classes. In addition, the use of PIT opens to future developments for different applications such as speaker separation.

We denote with f = [fj(t)] t=1...N j=1...J and f = [ fj(t)] t=1...N j=1...J the matrices of true and estimated targets, where J and N are respectively the maximum number of sources and the length of the estimated and true targets. The PIT-equivalent loss of a generic loss function L can be written as:

LPIT = min σ∈F J L( fσ, f ), (1)
where fσ = [ f σ(j) (t)] t=1...N j=1...J is a permutation of f by σ ∈ FJ , with FJ the set of permutations of [1, ..., J]. The whole procedure consists in computing the loss L for all possible permutations of the targets and finding the permutation σ for which the loss is minimized. We used the implementation of PIT available in Asteroid Source Separation Toolkit [START_REF] Pariente | Asteroid: the pytorch-based audio source separation toolkit for researchers[END_REF].

Several different losses depending on what labels are available for the current example are used to train the mask-estimation DNN:

• Strongly Labelled: for DESED synthetic data, for which fore-grounds features f = [fj(t)] t=1...N j=1...J are available we compute PIT Mean-Squared Error loss LMSE and find the optimal permutation σopt for the estimated separated features f = [ fj(t)] t=1...N j=1...J :

LMSE = min σ∈F J M SE( fσ, f ). (2)
The estimated foregrounds features are then re-ordered according to σopt and fed to the SED model for computing Deep Feature Loss (DFL) [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF][START_REF] Germain | Speech denoising with deep feature losses[END_REF]. DFL LDF L is computed between each SED internal activations obtained with re-ordered estimated foregrounds SED( fσ opt ) and those obtained with oracle foregrounds SED(f ):

LDFL = M m=1 SED( fσ opt ) m -SED(f ) m 1 , (3)
where the sum is taken over all M layers of the SED back-end and SED(f ) m denotes the activations of the m-th layer when the SED model is fed the feature matrix f . As in [START_REF] Germain | Speech denoising with deep feature losses[END_REF], L1 norm is used for computing DFL. The total loss for strongly labelled examples is the sum of the two terms:

Lstrong = LDF L + LMSE (4)
• Weakly Labelled: for weakly labelled data no oracle foregrounds are available, thus we train the separation model as in E2E Neural Diarization [START_REF] Fujita | End-to-End Neural Speaker Diarization with Permutation-Free Objectives[END_REF] to minimize PIT binary cross entropy (BCE) between weak predictions of SED model when it is fed the estimated foregrounds features wσ = SED( fσ) weak and the weak labels w weak :

Lweak = min σ∈W J BCE( wσ, w weak ). (5)
• Mean-teacher consistency: to leverage also unlabeled data, we use the Mean Teacher Semi-Supervised loss for the maskestimation network and enforce SED weak and strong predictions consistency between the values obtained with a student separation model S(f ; θs) and an exponential moving average mean teacher separation model T(f ; θt) using permutation invariant MSE loss L teach between the separated features of the two models:

Lteach = min σ∈F J M SE(SED(S(f σ )), SED(T(f ))). (6)
The total loss Ltot used to train the mask-estimation DNN is then the sum of aforementioned strong, weak, and mean-teacher losses.

RESULTS AND DISCUSSION

In our experiments, mask-estimation was accomplished with a reduced version of the separator network from Conv-TasNet [START_REF] Luo | Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF], as implemented in Asteroid, with 5 blocks (X = 5) and 3 repeats (R = 3), 64 bottleneck channels, 128 depth-wise convolution channels and sigmoid mask for a total of 3M trainable parameters. The whole system was trained to separate a maximum of 5 different sound event classes. We used a batch size of 32 examples with 12 synthetic examples, 12 weakly labeled examples, and 8 unlabeled examples, respectively. Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] was used for optimization. We used a warm-up exponential learning rate schedule for the first 10 epochs with the learning rate increasing from 0 to 0.001 as in [START_REF] Delphin-Poulat | Mean teacher with data agumentation for dcase 2019 task 4[END_REF].

In the following, we report results results obtained on DCASE 2020 Task 4 development set. We report results only in terms of SED event-based macro F1 score as oracle sources required for calculating Source Separation metrics such as SI-SDR are only available for the training set.

DCASE 2020 Task 4 Results

In Table 1 we report the results of the proposed separation system trained with the Task-Aware separation objective (Proposed). We also report the performance attained by the SED challenge baseline back-end system on its own without pre-processing (SEDonly Baseline), the performance of the combined separation and sound event detection baseline with (SEP+SED Baseline) and without averaging of predictions between noisy and denoised mixtures (SEP+SED Baseline no avg). In addition, we report the total number of parameters for each model with the back-end SED model included.

As can be seen, the combined separation and SED baseline system fails to improve the SED back-end performance when no ensembling is performed, while ensembling produces only a moderate performance improvement. On the other hand, the proposed method offers more than 2% improvement over the plain SED-only baseline with no ensembling and a significantly smaller separation model. In Figure 3 we show the output of the proposed separation system (in Mel domain) for a mixture where three different sound events are present belonging to two distinct classes: Frying and Cat. It can be seen that the events are effectively separated by the system, but that also distortion is introduced. However, E2E training guarantees that this distortion does not harm SED performance.

Ablation Study

In Table 2 we report results for our separation system when different loss functions are used. As a baseline, we report the system trained with only the PIT MSE loss LMSE (Equation 2) computed with the synthetic examples (strong-PIT), thus with only a source-separation based objective. We report again the performance of the SED backend alone and of the combined separation and SED baseline.

It can be seen that the system trained with only signal-based separation loss slightly degrades the performance compared to the SED-only system. This is due to the separation system quickly overfitting the synthetic data, leading to poor generalization to realworld examples. If the weak examples loss (+weak) Lweak (Equation 5) is added, the separation system is able to improve over the SED-only back-end system. The addition of Deep Feature Loss (Equation 3) brings another substantial improvement while the addition of the semi-supervised Mean-Teacher loss (Equation 6) adds only marginal improvement.

CONCLUSIONS

In this paper, we have proposed a novel training scheme for combined Source Separation and Sound Event Detection. In our method, a source separation system is trained in an End-to-End fashion with a pre-trained SED system that is not updated. For this purpose, we devised a combination of permutation-invariant training objectives, both signal-based and SED-based. We call such an approach Task-Aware separation as the separation system is optimized directly with the back-end task objective. We compared our approach with the state-of-the-art combined separation and Sound Event Detection DCASE 2020 Task 4 baseline and we showed that such a method is able to reach superior performance on DCASE 2020 Task 4 development set while having significantly less parameters. In future works, we will expand this End-to-End approach to other back-end tasks such as Automatic Speech Recognition.

Figure 1 :

 1 Figure 1: DCASE 2020 Task 4 baseline separation system approach.

Figure 2 :

 2 Figure 2: Task-aware Separation for Sound Event Detection.

Figure 3 :

 3 Figure 3: Output of separation stack: (a) input mixture, (b) first estimated source (c) second estimated source.

Table 1 :

 1 Performance of combined separation and SED systems on DCASE 2020 Task 4 development set.

	Method	Event macro F 1 score	Parameters
	SED-only Baseline	34.8	1M
	SEP+SED Baseline	35.6	10M
	SEP+SED Baseline no avg	33.4	10M
	Proposed	37.0	4M

Table 2 :

 2 Ablation study for proposed technique (development set).

	Method	Event macro F 1 score
	SED-only	34.8
	SEP+SED Baseline	35.6
	strong-MSE	34.5
	+weak	35.4
	+weak+DFL	36.7
	+weak+DFL+teach	37

Available at github.com/turpaultn/dcase20_task4.


Manuel Pariente

Samuele Cornell

Joris Cosentino

Sunit Sivasankaran

Efthymios Tzinis

Jens Heitkaemper

Michel Olvera

Fabian-Robert Stöter

Mathieu Hu

Juan M Martín-Doñas

David Ditter

Ariel Frank

Antoine Deleforge

Emmanuel Vincent

Asteroid: the PyTorch-based audio source separation toolkit for researchers

Keywords: source separation, speech enhancement, opensource software, end-to-end

This paper describes Asteroid, the PyTorch-based audio source separation toolkit for researchers. Inspired by the most successful neural source separation systems, it provides all neural building blocks required to build such a system. To improve reproducibility, Kaldi-style recipes on common audio source separation datasets are also provided. This paper describes the software architecture of Asteroid and its most important features. By showing experimental results obtained with Asteroid's recipes, we show that our implementations are at least on par with most results reported in reference papers. The toolkit is publicly available at github.com/mpariente/asteroid.

Introduction

Audio source separation, which aims to separate a mixture signal into individual source signals, is essential to robust speech processing in real-world acoustic environments [1]. Classical open-source toolkits such as FASST [2], HARK [START_REF] Nakadai | An open source software system for robot audition HARK and its evaluation[END_REF], ManyEars [START_REF] Grondin | The ManyEars open framework[END_REF] and openBliSSART [START_REF] Schuller | Blind enhancement of the rhythmic and harmonic sections by nmf: Does it help?[END_REF] which are based on probabilistic modelling, non-negative matrix factorization, sound source localization and/or beamforming have been successful in the past decade. However, they are now largely outperformed by deep learning-based approaches, at least on the task of single-channel source separation [START_REF] Hershey | Deep clustering: discriminative embeddings for segmentation and separation[END_REF][START_REF] Yu | Permutation invariant training of deep models for speaker-independent multi-talker speech separation[END_REF][START_REF] Luo | TasNet: Time-domain audio separation network for real-time, single-channel speech separation[END_REF][START_REF]Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF][START_REF] Zeghidour | Wavesplit: End-to-end speech separation by speaker clustering[END_REF].

Several open-source toolkits have emerged for deep learning-based source separation. These include nussl (Northwestern University Source Separation Library) [START_REF] Manilow | The Northwestern University Source Separation Library[END_REF], ONSSEN (An Open-source Speech Separation and Enhancement Library) [START_REF] Ni | Onssen: an open-source speech separation and enhancement library[END_REF], Open-Unmix [START_REF] Stöter | Open-Unmix -a reference implementation for music source separation[END_REF], and countless isolated implementations replicating some important papers 1 .

We would like to thank Hervé Bredin for fruitful discussions about software design and Kaituo Xu for open-sourcing his Conv-Tasnet implementation.

Experiments presented in this paper were partially carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).

High Performance Computing resources were partially provided by the EXPLOR centre hosted by the University de Lorraine 1 kaituoxu/TasNet, kaituoxu/Conv-TasNet, yluo42/TAC, JusperLee/Conv-TasNet, JusperLee/Dual-Path-RNN-Pytorch, tky1117/DNN-based source separation ShiZiqiang/dual-path-RNNs-DPRNNs-based-speech-separation Both nussl and ONSSEN are written in PyTorch [START_REF] Paszke | Py-Torch: An imperative style, high-performance deep learning library[END_REF] and provide training and evaluation scripts for several state-of-the art methods. However, data preparation steps are not provided and experiments are not easily configurable from the command line. Open-Unmix does provide a complete pipeline from data preparation until evaluation, but only for the Open-Unmix model on the music source separation task. Regarding the isolated implementations, some of them only contain the model, while others provide training scripts but assume that training data has been generated. Finally, very few provide the complete pipeline. Among the ones providing evaluation scripts, differences can often be found, e.g., discarding short utterances or splitting utterances in chunks and discarding the last one. This paper describes Asteroid (Audio source separation on Steroids), a new open-source toolkit for deep learning-based audio source separation and speech enhancement, designed for researchers and practitioners. Based on PyTorch, one of the most widely used dynamic neural network toolkits, Asteroid is meant to be user-friendly, easily extensible, to promote reproducible research, and to enable easy experimentation. As such, it supports a wide range of datasets and architectures, and comes with recipes reproducing some important papers. Asteroid is built on the following principles:

1. Abstract only where necessary, i.e., use as much native

PyTorch code as possible. 2. Allow importing third-party code with minimal changes.

3. Provide all steps from data preparation to evaluation. 4. Enable recipes to be configurable from the command line.

We present the audio source separation framework in Section 2. We describe Asteroid's main features in Section 3 and their implementation in Section 4. We provide example experimental results in Section 5 and conclude in Section 6.

General framework

While Asteroid is not limited to a single task, single-channel source separation is currently its main focus. Hence, we will only consider this task in the rest of the paper. Let x be a single channel recording of J sources in noise:

x(t) = J j=1 sj(t) + n(t), (1)
where {sj}j=1..J are the source signals and n is an additive noise signal. The goal of source separation is to obtain source estimates { sj}j=1..J given x.

Most state-of-the-art neural source separation systems follow the encoder-masker-decoder approach depicted in Fig. 1 [START_REF] Luo | TasNet: Time-domain audio separation network for real-time, single-channel speech separation[END_REF][START_REF]Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF][START_REF] Tzinis | Two-step sound source separation: Training on learned latent targets[END_REF][START_REF] Luo | Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation[END_REF]. The encoder computes a short-time Fourier transform (STFT)-like representation X by convolving the timedomain signal x with an analysis filterbank. The representation X is fed to the masker network that estimates a mask for each source. The masks are then multiplied entrywise with X to obtain sources estimates { Sj}j=1..J in the STFT-like domain. The time-domain source estimates { sj}j=1..J are finally obtained by applying transposed convolutions to { Sj}j=1..J with a synthesis filterbank. The three networks are jointly trained using a loss function computed on the masks or their embeddings [START_REF] Hershey | Deep clustering: discriminative embeddings for segmentation and separation[END_REF][START_REF] Isik | Single-channel multi-speaker separation using deep clustering[END_REF][START_REF] Chen | Deep attractor network for single-microphone speaker separation[END_REF], on the STFT-like domain estimates [START_REF] Yu | Permutation invariant training of deep models for speaker-independent multi-talker speech separation[END_REF][START_REF] Tzinis | Two-step sound source separation: Training on learned latent targets[END_REF][START_REF] Heitkaemper | Demystifying TasNet: A dissecting approach[END_REF], or directly on the time-domain estimates [START_REF] Luo | Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation[END_REF][START_REF] Bahmaninezhad | A comprehensive study of speech separation: Spectrogram vs waveform separation[END_REF].

Mixture waveform Separated waveforms

Encoder Decoder STFT-like rep.

Masked rep. Masker

Functionality

Asteroid follows the encoder-masker-decoder approach, and provides various choices of filterbanks, masker networks, and loss functions. It also provides training and evaluation tools and recipes for several datasets. We detail each of these below.

Analysis and synthesis filterbanks

As shown in [START_REF] Bahmaninezhad | A comprehensive study of speech separation: Spectrogram vs waveform separation[END_REF][START_REF] Kavalerov | Universal sound separation[END_REF][START_REF] Pariente | Filterbank design for end-to-end speech separation[END_REF][START_REF] Ditter | A multi-phase gammatone filterbank for speech separation via TasNet[END_REF], various filterbanks can be used to train end-to-end source separation systems. A natural abstraction is to separate the filterbank object from the encoder and decoder objects. This is what we do in Asteroid. All filterbanks inherit from the Filterbank class. Each Filterbank can be combined with an Encoder or a Decoder, which respectively follow the nn.Conv1d and nn.ConvTranspose1d interfaces from PyTorch for consistency and ease of use. Notably, the STFTFB filterbank computes the STFT using simple convolutions, and the default filterbank matrix is orthogonal.

Asteroid supports free filters [START_REF] Luo | TasNet: Time-domain audio separation network for real-time, single-channel speech separation[END_REF][START_REF]Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF], discrete Fourier transform (DFT) filters [START_REF] Heitkaemper | Demystifying TasNet: A dissecting approach[END_REF][START_REF] Kavalerov | Universal sound separation[END_REF], analytic free filters [START_REF] Pariente | Filterbank design for end-to-end speech separation[END_REF], improved parameterized sinc filters [START_REF] Pariente | Filterbank design for end-to-end speech separation[END_REF][START_REF] Ravanelli | Speaker recognition from raw waveform with SincNet[END_REF] and the multi-phase Gammatone filterbank [START_REF] Ditter | A multi-phase gammatone filterbank for speech separation via TasNet[END_REF]. Automatic pseudo-inverse computation and dynamic filters (computed at runtime) are also supported. Because some of the filterbanks are complex-valued, we provide functions to compute magnitude and phase, and apply magnitude or complex-valued masks. We also provide interfaces to NumPy [START_REF] Van Der Walt | The NumPy array: A structure for efficient numerical computation[END_REF] and torchaudio 2 . Additionally, Griffin-2 github.com/pytorch/audio Lim [START_REF] Griffin | Signal estimation from modified shorttime Fourier transform[END_REF][START_REF] Perraudin | A fast Griffin-Lim algorithm[END_REF] and multi-input spectrogram inversion (MISI) [START_REF] Gunawan | Iterative phase estimation for the synthesis of separated sources from single-channel mixtures[END_REF] algorithms are provided.

Masker network

Asteroid provides implementations of widely used masker networks: TasNet's stacked long short-term memory (LSTM) network [START_REF] Luo | TasNet: Time-domain audio separation network for real-time, single-channel speech separation[END_REF], Conv-Tasnet's temporal convolutional network (with or without skip connections) [START_REF]Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF], and the dual-path recurrent neural network (DPRNN) in [START_REF] Luo | Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation[END_REF]. Open-Unmix [START_REF] Stöter | Open-Unmix -a reference implementation for music source separation[END_REF] is also supported for music source separation.

Loss functions -Permutation invariance

Asteroid supports several loss functions: mean squared error, scale-invariant signal-to-distortion ratio (SI-SDR) [START_REF]Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation[END_REF][START_REF] Le Roux | SDRhalf-baked or well done?[END_REF], scale-dependent SDR [START_REF] Le Roux | SDRhalf-baked or well done?[END_REF], signal-to-noise ratio (SNR), perceptual evaluation of speech quality (PESQ) [START_REF] Martín-Doñas | A deep learning loss function based on the perceptual evaluation of the speech quality[END_REF], and affinity loss for deep clustering [START_REF] Hershey | Deep clustering: discriminative embeddings for segmentation and separation[END_REF].

Whenever the sources are of the same nature, a permutation-invariant (PIT) loss shall be used [START_REF] Yu | Permutation invariant training of deep models for speaker-independent multi-talker speech separation[END_REF][START_REF] Kolbaek | Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks[END_REF]. Asteroid provides an optimized, versatile implementation of PIT losses. Let s = [sj(t)] t=0...T j=1...J and s = [ sj(t)] t=0...T j=1...J be the matrices of true and estimated source signals, respectively. We denote as sσ = [ s σ(j) (t)] t=0...T j=1...J a permutation of s by σ ∈ SJ , where SJ is the set of permutations of [1, ..., J]. A PIT loss LPIT is defined as

LPIT(θ) = min σ∈S J L( sσ, s), (2)
where L is a classical (permutation-dependent) loss function, which depends on the network's parameters θ through sσ.

We assume that, for a given permutation hypothesis σ, the loss L( sσ, s) can be written as L( sσ, s) = G F( s σ(1) , s1), ..., F( s σ(J) , sJ )

where sj = [sj(0), . . . , sj(T )], sj = [ sj(0), . . . , sj(T )], F computes the pairwise loss between a single true source and its hypothesized estimate, and G is the reduce function, usually a simple mean operation. Denoting by F the J × J pairwise loss matrix with entries F( si, sj), we can rewrite (2) as

LPIT(θ) = min σ∈S J G F σ(1)1 , ..., F σ(J)J (4)
and reduce the computational complexity from J! to J 2 by precomputing F's terms. Taking advantage of this, Asteroid provides PITLossWrapper, a simple yet powerful class that can efficiently turn any pairwise loss F or permutation-dependent loss L into a PIT loss.

Datasets

Asteroid provides baseline recipes for the following datasets: wsj0-2mix and wsj0-3mix [START_REF] Hershey | Deep clustering: discriminative embeddings for segmentation and separation[END_REF], WHAM [START_REF] Wichern | WHAM!: extending speech separation to noisy environments[END_REF], WHAMR [START_REF] Maciejewski | WHAMR!: Noisy and reverberant single-channel speech separation[END_REF], LibriMix [START_REF] Cosentino | LibriMix: An open-source dataset for generalizable speech separation[END_REF] FUSS [START_REF] Wisdom | What's all the fuss about free universal sound separation data?[END_REF], Microsoft's Deep Noise Suppression challenge dataset (DNS) [START_REF] Reddy | The Interspeech 2020 deep noise suppression challenge: Datasets, subjective speech quality and testing framework[END_REF], SMS-WSJ [START_REF] Drude | SMS-WSJ: Database, performance measures, and baseline recipe for multi-channel source separation and recognition[END_REF], Kinect-WSJ [START_REF] Sivasankaran | Analyzing the impact of speaker localization errors on speech separation for automatic speech recognition[END_REF], and MUSDB18 [START_REF] Rafii | The MUSDB18 corpus for music separation[END_REF]. Their characteristics are summarized and compared in

Training

For training source separation systems, Asteroid offers a thin wrapper around PyTorch-Lightning [START_REF] Falcon | Pytorch lightning[END_REF] that seamlessly enables distributed training, experiment logging and more, without sacrificing flexibility. Regarding the optimizers, we also rely on native PyTorch and torch-optimizer 3 .

PyTorch provides basic optimizers such as SGD and Adam and torch-optimizer provides state-of-the art optimizers such as RAdam, Ranger or Yogi.

Evaluation

Evaluation is performed using pb bss eval 4 , a sub-toolkit of pb bss 5 [START_REF] Drude | Tight integration of spatial and spectral features for BSS with deep clustering embeddings[END_REF] written for evaluation. It natively supports most metrics used in source separation: SDR, signal-to-interference ratio (SIR), signal-to-artifacts ratio (SAR) [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF], SI-SDR [START_REF] Le Roux | SDRhalf-baked or well done?[END_REF], PESQ [START_REF] Rix | Perceptual evaluation of speech quality (PESQ) -a new method for speech quality assessment of telephone networks and codecs[END_REF], and short-time objective intelligibility (STOI) [START_REF] Taal | An algorithm for intelligibility prediction of time-frequency weighted noisy speech[END_REF].

Implementation

Asteroid follows Kaldi-style recipes [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF], which involve several stages as depicted in Fig. 2. These recipes implement the entire pipeline from data download and preparation to model training and evaluation. We show the typical organization of a recipe's directory in Fig. 3. The entry point of a recipe is the run.sh script which will execute the following stages:

• Stage 0: Download data that is needed for the recipe. In the first stage, necessary data is downloaded (if available) into a storage directory specified by the user. We use the official scripts provided by the dataset's authors to generate the data, and optionally perform data augmentation. All the information required by the dataset's DataLoader such as filenames and paths, utterance lengths, speaker IDs, etc., is then gathered into text files under data/. The training stage is finally followed by the evaluation stage. Throughout the recipe, log files are saved under logs/ and generated data is saved under exp/.

3 github.com/jettify/pytorch-optimizer 4 pypi.org/project/pb bss eval 5 github.com/fgnt/pb bss As can be seen in Fig. 4, the model class, which is a direct subclass of PyTorch's nn.Module, is defined in model.py. It is imported in both training and evaluation scripts. Instead of defining constants in model.py and train.py, most of them are gathered in a YAML configuration file conf.yml. An argument parser is created from this configuration file to allow modification of these values from the command line, with run.sh passing arguments to train.py. The resulting modified configuration is saved in exp/ to enable future reuse. Other arguments such as the experiment name, the number of GPUs, etc., are directly passed to run.sh.

Example results

To illustrate the potential of Asteroid, we compare the performance of state-of-the-art methods as reported in the corresponding papers with our implementation. We do so on two common source separation datasets: wsj0-2mix [START_REF] Hershey | Deep clustering: discriminative embeddings for segmentation and separation[END_REF] and WHAMR [START_REF] Maciejewski | WHAMR!: Noisy and reverberant single-channel speech separation[END_REF]. wsj0-2mix consists of a 30 h training set, a 10 h validation set, and a 5 h test set of single-channel two-speaker mixtures without noise and reverberation. Utterances taken from the Wall Street Journal (WSJ) dataset are mixed together at random SNRs between -5 dB and 5 dB. Speakers in the test set are different from those in the training and validation sets. WHAMR Table 2 reports SI-SDR improvements (SI-SDRi) on the test set of wsj0-2mix for several well-known source separation systems. In Table 3, we reproduce Table 2 from [START_REF] Maciejewski | WHAMR!: Noisy and reverberant single-channel speech separation[END_REF] which reports the performance of an improved TasNet architecture (more recurrent units, overlap-add for synthesis) on the four main tasks of WHAMR: anechoic separation, noisy anechoic separation, reverberant separation, and noisy reverberant separation. On all four tasks, Asteroid's recipes achieved better results than originally reported, by up to 2.6 dB.

Reported Using Asteroid

Deep Clustering [START_REF] Wang | Alternative objective functions for deep clustering[END_REF] 9 3: SI-SDRi (dB) on the four WHAMR tasks using the improved TasNet architecture in [START_REF] Maciejewski | WHAMR!: Noisy and reverberant single-channel speech separation[END_REF].

In both Tables 2 and3, we can see that our implementations outperform the original ones in most cases. Most often, the aforementioned architectures are trained on 4-second segments. For the architectures requiring a large amount of memory (e.g., Conv-TasNet and DPRNN), we reduce the length of the training segments in order to increase the batch size and stabilize gradients. This, as well as using a weight decay of 10 -5 for recurrent architectures increased the final performance of our systems.

Asteroid was designed such that writing new code is very simple and results can be quickly obtained. For instance, starting from stage 2, writing the TasNet recipe used in Table 3 took less than a day and the results were simply generated with the command in Fig. 5, where the GPU ID is specified with the --id argument.

Conclusion

In this paper, we have introduced Asteroid, a new open-source audio source separation toolkit designed for researchers and practitioners. Comparative experiments show that results obtained with Asteroid are competitive on several datasets and for several architectures. The toolkit was designed such that it can quickly be extended with new network architectures or new benchmark datasets. In the near future, pre-trained models will be made available and we intend to interface with ESPNet to enable end-to-end multi-speaker speech recognition.

[1] E. Vincent, T. Virtanen, and S. Gannot, Audio Source Separation and Speech Enhancement, 1st ed. Wiley, 2018.

[2] Y. Salaün, E. Vincent, N. Bertin, N. Souviraà-Labastie, X. Jaureguiberry, D. T. Tran, and F. Bimbot, "The Flexible Audio Source Separation Toolbox Version 2.0," ICASSP Show & Tell, 2014.

Figure 1 :

 1 Figure 1: Typical encoder-masker-decoder architecture.

Figure 2 :

 2 Figure 2: Typical recipe flow in Asteroid.

Figure 3 :

 3 Figure 3: Typical directory structure of a recipe.

Figure 4 :

 4 Figure 4: Simplified code example.

  n=0 for task in clean noisy reverb reverb_noisy do ./run.sh --stage 3 --task $task --id $n n=$(($n+1)) done

Figure 5 :

 5 Figure 5: Example command line usage.

Table 1

 1

	. wsj0-2mix and MUSDB18

Table 1 :

 1 Datasets currently supported by Asteroid.

* White sensor noise. ** Background environmental scenes.

•

  Stage 1: Generate mixtures with the official scripts, optionally perform data augmentation. • Stage 2: Gather data information into text files expected by the corresponding DataLoader. • Stage 3: Train the source separation system. • Stage 4: Separate test mixtures and evaluate.

Table 2 :

 2 SI-SDRi (dB) on the wsj0-2mix test set for several architectures. ks stands for for kernel size, i.e., the length of the encoder and decoder filters.

		.6	9.8
	TasNet [8]	10.8	15.0
	Conv-TasNet [9]	15.2	16.2
	TwoStep [15]	16.1	15.2
	DPRNN (ks = 16) [16]	16.0	17.7
	DPRNN (ks = 2) [16]	18.8	19.3
	Wavesplit [10]	20.4	-
	Reported Using Asteroid
	Noise Reverb	[33]
		14.2	16.8
		12.0	13.7
		8.9	10.6
		9.2	11.0
	Table