Maxime Baelde
email: maxime.baelde@a-volute.com

Christophe Biernacki
email: christophe.biernacki@math.univ-lille1.fr

Raphaël Greff
email: raphael.greff@a-volute.com

Real-Time Monophonic and Polyphonic Audio Classification from Power Spectra

Keywords: real-time, audio classification, machine learning, monophonic, polyphonic, generative model, nonparametric estimation

This work addresses the recurring challenge of real-time monophonic and polyphonic audio source classification. The whole normalized power spectrum (NPS) is directly involved in the proposed process, avoiding complex and hazardous traditional feature extraction. It is also a natural candidate for polyphonic events thanks to its additive property in such cases. The classification task is performed through a nonparametric kernel-based generative modeling of the power spectrum. Advantage of this model is twofold: it is almost hypothesis free and it allows to straightforwardly obtain the maximum a posteriori classification rule of online signals. Moreover it makes use of the monophonic dataset to build the polyphonic one. Then, to reach the real-time target, the complexity of the method can be tuned by using a standard hierarchical clustering preprocessing of the prototypes, revealing a particularly efficient computation time and classification accuracy trade-off. The proposed method, called RARE (for Real-time Audio Recognition Engine) reveals encouraging results both in monophonic and polyphonic classification tasks on benchmark and owned datasets, including also the targeted real-time situation. In particular, this method benefits from several advantages compared to the state-of-the-art methods including a reduced training time, no feature extraction, the ability to control the computation -accuracy trade-off and no training on already mixed sounds for polyphonic classification.

Introduction

Audio source classification has been a challenging research subject for the past thirty years, beginning with speech recognition [START_REF] Rabiner | A Tutorial on hidden Markov Models and Selected Applications in Speech Recognition[END_REF], and currently known as the vast field of sound event detection (SED). The latter consists in detecting and classifying audio sources present in monophonic (one source active at a time) and polyphonic (several sources at a time) audio streams. Many methodologies and algorithms were created to solve SED, and can be distributed in three main topics. The most prominent topic is automatic speech recognition (ASR) whose goal is to identify speech (in particular phonemes) in audio recording [START_REF] Qian | Very Deep Convolutional Neural Networks for Noise Robust Speech Recognition[END_REF][START_REF] Liu | Attention guided deep audio-face fusion for efficient speaker naming[END_REF]. The next topic is music information retrieval aiming at analyzing musics and extracting relevant information such as the musical genre [START_REF] Kour | Music Genre Classification using MFCC, SVM and BPNN[END_REF] (rock, classical, etc.) or the different instruments [START_REF] Joder | Temporal Integration for Audio Classification With Application to Musical Instrument Classification[END_REF]. The last topic is environmental sound recognition which aims at recognizing sounds such as airplanes, dog coughs, trains, gunshots, etc. [START_REF] Piczak | Environmental sound classification with convolutional neural networks[END_REF][START_REF] Hu | Latent topic model for audio retrieval[END_REF][START_REF] Beltrán | Scalable identification of mixed environmental sounds, recorded from heterogeneous sources[END_REF]. SED can be performed offline -using the whole signal -or online -audio data come on the fly as time frames. Online or real-time processing relates to two criteria [START_REF] Flocon-Cholet | Classification audio sous contrainte de faible latence[END_REF]: speed and latency. First, speed is the time to make the decision and is related to how many time frames the system uses (past and future). Second, latency is related to the computation time. For instance, if a time frame lasts 50ms, the decision has to be made within these 50ms, otherwise the result will not be used in the process. The latter is illustrated on Figure 1.

State-of-the-art SED methods typically involve two steps: feature extraction and supervised learning. Feature extraction -a universal stage in Machine Learning -summarizes the available information with a set of (expected) discriminant features. The usual audio features are known as audio descriptors [START_REF] Peeters | The Timbre Toolbox: extracting audio descriptors from musical signals[END_REF], distributed in three groups. Temporal features use the raw audio signal (as a function of time) and consist in the energy, the autocorrelation coefficients and the zero crossing rate (i.e. how many times the signal crosses zero) for instance. Spectral features are extracted using the Fourier Transform (FT) and are mainly the spectral moments (centroid, spread, skewness, kurtosis). Cepstral and perceptual features are computed using the inverse FT of the log-magnitude FT on Mel-scale (for the Mel Frequency Cepstral Coefficient (MFCC)), and a harmonic decomposition (for the fundamental frequency, the inharmonicity, etc.), respectively. It should be worth noting that temporal features can also be extracted by considering the temporal evolution of the spectral and cepstral features [START_REF] Joder | Temporal Integration for Audio Classification With Application to Musical Instrument Classification[END_REF]. The relevance of the features depends on the context: for instance, the MFCC are good features for glass break, but not for gun shot recognition [START_REF] Mesaros | Detection and classification of acoustic scenes and events, Detection and Classification of Acoustic Scenes and Events[END_REF].

Several supervised learning methods have been applied to monophonic SED using the previous features. What is called "monophonic" in the SED field of research corresponds to a multi-class single-label classification task. First, Gaussian Mixture Models (GMM) are generative models assuming that the distribution generating the data given a class is a mixture of Gaussian distributions. The decision is computed using the maximum a posteriori (MAP), which is the maximum posterior probability of the classes. This modeling has been

Time

Get a frame Get a frame Get a frame Get a frame Get a frame High c.t.

Low c.t.

Get the decision

Get the decision

Get the decision

Get the decision

Too late: the decision cannot be used Too late: the decision cannot be used Figure 1: Illustration of the latency of a real-time audio classification system. A real-time process gets the frames at specific time clocks (black bottom arrows). If the computation time (c.t.) is low (Low c.t., blue filled rectangles), the decision will be used by the system because it will be available before the next frame. If the computation time is high (High c.t., red filled rectangles), the decision is discarded because it cannot be used by the system.

applied with MFCC for gunshot detection [START_REF] Clavel | Events Detection for an Audio-Based Surveillance System[END_REF][START_REF] Radhakrishnan | Audio analysis for surveillance applications[END_REF] and real-time voice detection in medical application [START_REF] Istrate | Real time sound analysis for medical remote monitoring[END_REF][START_REF] Choi | On using acoustic environment classification for statistical model-based speech enhancement[END_REF]. Second, Hidden Markov Models (HMM) are used to model the temporal continuity of audio signals, alone or coupled with GMM. Bietti et al. modeled the normalized audio spectra using HMM in an online setting [START_REF] Bietti | An online em algorithm in hidden (semi-)Markov models for audio segmentation and clustering[END_REF], whereas Heittola et al. fed a HMM with histograms of MFCC [START_REF] Heittola | Audio context recognition using audio event histograms[END_REF]. Third, Support Vector Machines (SVM) are binary classifiers that map the features into a high dimensional space using the kernel trick and perform the classification in this space. The SVM can be extended to multi-class classification by considering learning strategies like One-Versus-One or One-Versus-All. SVM have been used with the energy and the signal spectrogram in [START_REF] Lecomte | Abnormal events detection using unsupervised One-Class SVM -Application to audio surveillance and evaluation[END_REF] and [START_REF] Sameh | On the Use of Time--Frequency Reassignment and SVM-Based Classifier for Audio Surveillance Applications[END_REF] for surveillance application, and also in [START_REF] Temko | Classification of acoustic events using SVM-based clustering schemes[END_REF] for meeting room [23] and Piczak [START_REF] Piczak | Environmental sound classification with convolutional neural networks[END_REF] considered directly the raw audio signal and the spectrogram, respectively, using a deep architecture (DCNN). Other authors considered also Random Forest [START_REF] Xia | Random forest classification based acoustic event detection utilizing contextual-information and bottleneck features[END_REF] using more contextual information.

Other algorithms are used in case of polyphonic SED. What is called "polyphonic"1 in the SED field of research corresponds to a multi-class multi-label classification task. It is often cast to a multi-class single-label classification task but the output of the system is thresholded to predict the active classes. C ¸akır et al. [START_REF] Parascandolo | Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection[END_REF] developed a Convolutional Recurrent Neural Network (CRNN), taking advantages of the two structures (convolutional and recurrent). However, NN-based algorithms require a huge amount of labeled data to train the network, which is not always available (no public dataset or time-consuming data recording and labeling). Apart from NN, two main modeling algorithms are considered: PLCA (Probabilistic Latent Component Analysis) [START_REF] Smaragdis | A probabilistic latent variable model for acoustic modeling[END_REF] and NMF (Nonnegative Matrix Factorization) [START_REF] Paatero | Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values[END_REF]. Benetos et al. [START_REF] Benetos | Detection of overlapping acoustic events using a temporally-constrained probabilistic model[END_REF] proposed a probabilistic modeling of ERB (Equivalent Rectangular Bandwidth) spectra using PLCA coupled with HMM. NMF has been used as a task-driven modeling of MFCC spectra [START_REF] Bisot | Overlapping sound event detection with supervised Nonnegative Matrix Factorization[END_REF] or as coupled training of sound spectra and class annotation [START_REF] Mesaros | Sound event detection in real life recordings using coupled matrix factorization of spectral representations and class activity annotations[END_REF]. Heittola et al. [START_REF] Heittola | Supervised model training for overlapping sound events based on unsupervised source separation[END_REF] constructed a two-step method based on unsupervised source separation (with NMF) and classification of the separated sources (using a GMM-HMM).

The previous review arouses three unresolved problems. First, the methods rely on context-based (which are not always relevant) features. Second, the algorithms are not often designed for real-time processing: either they need lots of frames (low speed) or the computations are too heavy (high latency). Third, typical methods for polyphonic SED suffer from three drawbacks: the number of active sources is assumed to be known, the output of the system is thresholded, and a dataset of sound mixtures is needed. Even for general multi-label learning task, usual methods include Binary Relevance (BR, learn a classifier for each label individually: simple but does not learn correlations between labels) or Label Powerset (LP, consider multi-label output as a new single-label: complex and combinatorial) [START_REF] Alazaidah | A multi-label classification approach based on correlations among labels[END_REF], which are far from optimal solutions. More advanced multi-label methods also exist such as RAKEL [START_REF] Tsoumakas | Random k-labelsets: An ensemble method for multilabel classification[END_REF] (combination of LP classifiers) or Classifier Chains [START_REF] Read | Classifier chains for multi-label classification[END_REF] (chain of BR classifiers) and their ensemble versions. To overcome the previous problems, a novel method is proposed in this paper that can perform both monophonic and polyphonic real-time SED without assuming the number of active sources to be known and by using the audio spectrum itself (and not audio descriptors) for the classification.

The method developed in this paper, called RARE (for Real-time Audio Recognition Engine) is based on a generative model of the whole normalized power spectrum (NPS), releasing the need of (sometimes perilous) feature extraction. In particular, the use of the power spectrum instead of standard magnitude spectrum is useful for the polyphonic modeling task thanks to the additive property of uncorrelated signals. Consequently, a suitable decomposition of the polyphonic spectra using monophonic ones allows to dispense with learning mixture of sounds, which is not possible with classical predictive modeling. In addition, the generative model has two advantages. Firstly, it can be considered as a very low assumption situation since it is related to a kernel density estimation using multinomial kernels (nonparametric framework). Secondly, it allows to straightforwardly derive a time-varying MAP for the online classification. However, using the model as is, the real-time target is not reached because of the involved computational load. A model preprocessing using hierarchical clustering is thus developed to reduce this computational load, leading finally to an efficient trade-off between accuracy and computation time. The proposed method is a worthwhile extension of the one presented in our two previous conference papers [START_REF] Baelde | A mixture model-based real-time audio sources classification method[END_REF] and [START_REF] Baelde | Classification de signaux audio en tempsréel par un modèle de mélanges d'histogrammes[END_REF], in a more formalized way and with added extensive experiments.

The contribution of the present article can be summed up by the combination of the following three points:

1. Features: The use of the whole power spectrum instead of usual features extracted from the audio signals, which is moreover essential for polyphonic event; 2. Modeling: A very general generative modeling of the power spectra designed for real-time audio classification, that uses monophonic models to build the polyphonic models; 3. Real-time: A model reduction technique based on hierarchical clustering preprocessing of the sound models.

The paper is organized as follows. The monophonic modeling is disclosed in Section 2 and is extended to polyphonic cases in Section 3. The reduction of the complexity is detailed in Section 4. The experiments to assess the performance of the method are presented in Section 5. Finally, Section 6 concludes the paper.

Monophonic modeling of the classes

The monophonic modeling is a mandatory step toward the polyphonic modeling, but the main goal of this work is to perform polyphonic classification. Indeed, the polyphonic modeling will be build upon the monophonic one in a straighforward manner (see Section 3).

Problem statement

The purpose of this paper is to provide a real-time sound classification method. Consider the case where the objective is to classify sounds coming from video games (case study from the company A-Volute2 ). The classification uses only the audio mix coming from the video game, without any additional knowledgemeta-data from the game for instance. The sound is assumed to contain events coming from some classes of sounds -for instance a gunshot or an airplanewhich have to be inferred (see Figure 2(a)). The objective of classifying at time t a sound can be written as follows:

z = argmax z p z f x time [t-∆T,t] , t , (1)
where z is an estimate of the label z ∈ {1, ..., K} representing the class of sound at time t (for instance, the class z = 1 is composed of airplane sounds, z = 2 is composed of gunshot sounds, etc.), p z f x time [t-∆T,t] , t is the probability of the class z given the features extracted from the sound x time [t-∆T,t] . The class label z is assumed to be the same over the time period [t -∆T, t] (continuity of z and one active sound at most): this is the definition of a monophonic frame. However the overall sound x time can contain instances of difference classes.

x
Several constraints are considered for real-time classification. The first is the time related to the data acquisition which constrains t to be a multiple of the shift length δt, that is t ∈ {0, δt, 2δt, ...}. This shift length is important because the classification has to be done at every multiple of δt, therefore the speed of the recognition system has to be less than this shift length. The second constraint is related to ∆T which is the period of observation of the sound which has to be set according to the processing time (see Section 2.2). The problem formulation as an argmax of a distribution is suitable because it is mathematically well-posed and it is a classical framework in probabilistic modeling. From the objective in Eq. ( 1), f (•) and p (•) have to be chosen carefully, and are disclosed in the following sections.

Remark. In practice, all the processing are done in discrete time, due to the processing on computers. As a result, the sound x time [t-∆T,t] will be a real vector after sampling on the computer at a rate F (see the value F in Section 5).

Feature design

The function f (•) is used to compute relevant features from x time [t-∆T,t] to perform the classification. Indeed, time domain signals are not well suited for audio classification since they are not discriminant enough features: they convey the phase information and also the volume, from which we want the method to be invariant. The sound is converted into the frequency domain: more particularly the normalized power spectrum is considered. Therefore by using normalized power spectrum we reach the invariance property. Moreover this transformation will be particularly relevant for polyphonic spectrum since it preserves the additivity of uncorrelated signals (see Section 3). Consequently,

a possible definition of f x time [t-∆T,t] can be f norm • f FT x time [t-∆T,t]
, where f FT is the function that computes the Fourier Transform and f norm is the function that normalizes the complex spectrum to get the normalized power spectrum (defined later in Eq. ( 5)).

However, since the classification has to be done in real-time, such a large amount of data contained in x time [t-∆T,t] cannot be used. This is why this piece of sound is split first of all into time frames:

(x time 1t , ..., x time N t ) = f (∆t) frame x time [t-∆T,t] , (2)
where N = (∆T -∆t)/δt is the number of frames that can be effectively computed within the time interval [t -∆T, t] and each frame x time τ t ∈ R ∆t is defined by:

x time τ t = x time [t-∆T +τ δt,t-∆T +τ δt+∆t] , τ = 1, ..., N. (3)
The frame x time τ t is thus a portion of x time [t-∆T,t] of size ∆t, overlapping the previous frames by a duration τ δt (see Figure 2(b)). The frame length ∆t is set to a small value to allow fast processing for the Fourier Transform: as a counterpart, the frame will contain less information than a large frame (like previously). As a result, a collection of several normalized power spectra is computed instead of a single large spectrum. The framed normalized power spectra are denoted by: (x 1t , ...,

x N t ) = f x time [t-∆T,t] = f norm • f FT • f (∆t) frame x time [t-∆T,t] , (4)
where f FT and f norm operate frame-wise. Each normalized power spectrum is computed as follows:

x τ t = x freq τ t 2 ||x freq τ t || 2 , (5)
where

x freq τ t ∈ C B is the Fourier Transform of x time τ t (only B ≤ ∆t frequency bins are kept), | • | is the element-wise modulus and || • || is the 2 -norm (see Figure 2(c))
. The choice of ∆t results in a trade-off between the computation time and the quality of the approximation: the larger is ∆t the better is the approximation but the larger is the computation time.

Model design

The question now is to define the distribution p(•). We consider a generative model because a similar generative process will be used in the polyphonic case that simplifies the classification task. Indeed it considers only the monophonic dataset to construct the polyphonic one. This is a significant advantage compared to standard predictive modeling or usual multi-label learning. The process for generating a sequence of N spectra is detailed through the following generative model:

• Random: Draw a class label: z ∼ Mult K (1, p),
• Random: Draw a sound of length T from class z:

x time ∼ p sound (•|z), • Random: Draw a time index: t ∼ U ([0, T ]),
• Deterministic: Compute the features: (x 1t , ...,

x N t ) = f x time [t-∆T,t] ,
where Mult K (1, p) is a multinomial distribution over K categories and 1 draw

with probabilities p = (p k ) K k=1 p k > 0, K k=1 p k = 1 which means that the probability of z = k is p k , and U ([a, b]) is the uniform distribution over the interval [a, b].
The generative modeling we adopt has the main advantage of being hypothesis free: the distribution that generates the class is the very general multinomial distribution and the distribution that generates the sounds is assumed to be a general distribution over real-valued processes of size T denoted by p sound (•|z). The whole generative process is completely defined up to the distribution p sound (•|z) which will be treated further in the conditionaling modeling.

Recall the objective in Eq. ( 1), we can decompose this objective using the Bayes theorem as follows:

p (z|x 1t , ..., x N t , t) ∝ p (x 1t , ..., x N t |z, t) p(z|t). (6)
The generative process models the joint distribution p (x 1t , ..., x N t , t, x time , z), however only the conditional p x 1t , ..., x N t z, t is useful for the monophonic classification task. The full signal x time is redundant since the frames are extracted from it so that it will not be used in the remainder. Moreover, as a consequence of the generative model, we have that p(z|t) = p(z).

Several hypotheses are made to simplify the distribution. The first hypothesis is an independence hypothesis. Assuming that the shift length δt is large enough, two consecutive frames can be considered independent. In practice for δt = ∆t/2 or δt = ∆t/4 (our future chosen values) there is approximately independence between two consecutive frames. The distribution of a sequence of spectra can thus be approximated using the following conditional independence assumption:

p x 1t , ..., x N t z, t = N τ =1 p x τ t z, t . (7)
Eq. ( 7) can be viewed either as an aggregation (as in [START_REF] Baelde | A mixture model-based real-time audio sources classification method[END_REF]) or as an independence assumption. It should worth noting that this assumption can be challenging since in practice acoustic signals are temporally correlated: however for computational purpose this hypothesis is quite convenient and used in some references like [START_REF] Clavel | Events Detection for an Audio-Based Surveillance System[END_REF]. The second hypothesis is an hypothesis of stationarity in time, so that the distribution of spectrum x τ t given the class z does not depend on the time t:

p x τ t z, t = p x τ t z, t . (8)
The third hypothesis is an hypothesis of stationarity over the frames, so that the distribution of a spectrum does not depend on the shift index τ (since the class is the same in the time interval [t -∆T, t]):

p x τ t z = p x τ t z . (9)
Given Eq. ( 8) and Eq. ( 9), the indexes τ, t can be removed so that x τ t = x, and finally the distribution can be written:

p x τ t z, t = p x z . (10)

Estimation of the model

We suppose that a learning set x (i) , z (i) n i=1 is available using the generative process introduced in the previous section. The conditional distribution of Eq. ( 10) is estimated using a kernel density estimation of the form:

p (x|z) = 1 n z i z (i) =z K x, x (i) , (11)
where n z is the number of samples in the class z and K •, x (i) is a kernel that has to be chosen. From the definition of x, several kernels could be used: the Dirichlet kernel -though it is not flexible enough for our purpose -, a mixture of Dirichlet kernel -which is more flexible but in a real-time context the involved computational load is to high -and finally a very classical Gaussian mixture models like in [START_REF] Baelde | A mixture model-based real-time audio sources classification method[END_REF]. We have tried these three kernels (results no reported here) but the best results were obtained with the very simple multinomial kernel that we present now. The multinomial kernel quantizes the spectrum so that it becomes a vector of integers that sums to a quantization factor q ∈ N. Indeed this kernel gathers the advantages to be easy to evaluate, and reaches an efficient computation time -accuracy trade-off as described later in Section 4, because only dot products are required to compute the distribution (the normalization constant is the same for each kernel). Consider x

(i) ∈ N B (where B is the number of frequency bins) the closest integer vector to qx (i) which sums to q defined by:

x (q) (i) = argmin x∈N B qx (i) -x . (12)
We define the so-called approximated kernel by:

K (q) •, x (q)
(i) = Mult B •; q, p (q)
(i) , (13)
where the parameter p (q) (i) is defined by:

p (q) (i) = 1 q x (q) (i) . (14)
The approximated kernel

K (q) •, x (q) (i)
converges to K •, x (i) as q goes to infinity (see Appendix A for a proof). For large enough q this approximation will be correct. In practice, values of q close to B are good enough (see Section 5). Finally, the probability of the classes are estimated by maximum likelihood:

p z = n z n • (15)
Remark. In practice, the learning set is built using a slightly different generative process for practical reasons. A set of sounds already sampled from p sound (•|z) is supposed to be available. We first draw a class label z (i) and then a sound. The step of drawing a time index is different but mimick the generative process: every time index is considered and creates several time frames. The remaining of the process is the same: the time frames are transformed into the normalized power spectrum. The framing implies that in a given frame the sound Figure 3: We illustrate the following phenomenon. In real-time, the sound (blue filled curve) neither begins at the beginning of the frame (top left) nor ends at the ending (top right). This is why we add (bottom left and right) Gaussian white noise (red rectangle) so as to fill the "silence" (green line).

neither necessarily begins at the beginning of this frame nor ends at the ending. Rather than adding silence -which contains some information -Gaussian white noise (GWN) is added -which conveys no statistical information -to fill this "blank" (see Figure 3).

Polyphonic modeling of the classes

Motivation and polyphonic features

The polyphonic classification relies on the monophonic dataset, built in Section 2.4 and uses a similar framework than the monophonic one. Using a suitable decomposition of the polyphonic spectrum, the method uses only the monophonic spectra and does not have to learn mixture of sounds: it is a clear advantage of the proposal. The case for a mixture of two sources is considered but it can be straighforwardly extended to a greated number of sources: this means that a frame has two simultaneous labels, denoted by

z = (z 1 , z 2 ) ∈ {1, ..., K} 2 , z 1 = z 2 .
Recalling the objective in Section 2, the polyphonic decision rule is thus written as:

z = argmax z p z f x time [t-∆T,t] , t , (16)
where x time [t-∆T,t] is the observed polyphonic sound, defined by the sum of the monophonic sounds x time 1[t-∆T,t] and x time 2[t-∆T,t] from the corresponding classes z 1 and z 2 , as:

x time [t-∆T,t] = x time 1[t-∆T,t] + x time 2[t-∆T,t] . (17)
The same features are used since the power spectrum keeps the additivity of the spectra -it was designed to this purpose. Then, the polyphonic feature can be expressed as a combination of the underlying monophonic features. Indeed, some calculi (see Appendix B) on the features lead to the following result: where x • y is the element-wise multiplication between some vectors x and y.

f x time [t-∆T,t] = φ t • f x time 1[t-∆T,t] + (1 -φ t ) • f x time 2[t-∆T,t] , (18)
f f f x 1 x 2 x
By using normalized power spectra the modeling induces proportions φ t = (φ 1t , ..., φ N t ) which represent the ratio between the power of one source and the power of the two sources (see Figure 4). At a given time shift τ the proportion is defined by:

φ τ t = ||x freq 1τ t || 2 ||x freq 1τ t || 2 + ||x freq 2τ t || 2 , (19)
where x freq 1τ t is defined as in the monophonic feature design section 2.2. A detailed explanation is available in Appendix B. As already said, this framework can thus easily be extended to a mixture of more than two sources.

Remark. We assume that the time at which the sounds are observed is the same, but the individual sounds may have two different starting times: here, for the simplicity of the presentation, the underlying synchronization of the sounds is not presented but the proposed method actually takes it into account.

Model design

Based on the monophonic generative model, the following generative model of polyphonic spectra (for two classes) is defined by:

• Random: Draw independently without replacement two different class labels z 1 , z 2 ∼ Mult K (1, p),
• Random: Draw independently two different sounds:

x time 1 ∼ p sound (•|z 1 ) and x time 2 ∼ p sound (•|z 2 ) • Random: Draw a time index: t ∼ U ([0, T ])
• Deterministic: Compute the features: (x 1t , ...,

x N t ) = φ t •f x time 1[t-∆T,t] + (1 -φ t ) • f x time 2[t-∆T,t] .
This polyphonic generative process models the joint distribution p(x 1t , ..., x N t , t, x time 1 , x time 2 , z 1 , z 2 ), and again only the conditional distribution is used here. The Bayes theorem allows to write:

p (z 1 , z 2 |x 1t , ..., x N t , t) ≈ p (x 1t , ..., x N t |z 1 , z 2 , t) p (z 1 , z 2 |t) . ( 20
)
From the generative model we have that p (z 1 , z 2 |t) = p (z 1 , z 2 ). The frame independence assumption of Eq. ( 7) is still valid so it leads to the following approximation:

p (x 1t , ..., x N t |z 1 , z 2 , t) ≈ N τ =1 p (x τ t |z 1 , z 2 , t) . (21)
Moreover the stationary hypotheses are also valid in this context (stationarity in time as in Eq. ( 8) and stationarity over the frames as in Eq. ( 9)). As φ τ t depends on x τ t , the proportion will inherit from the stationarity hypotheses.

As a consequence, the indexes are removed: x τ t = x and φ τ t = φ, and the polyphonic conditional distribution can be expressed as:

p (x τ t |z 1 , z 2 , t) = p (x|z 1 , z 2 ) . (22)

Polyphonic dataset building and Estimation of the model

The main advantage of the previous modeling is that it allows to build a new dataset for polyphonic sounds using only monophonic ones: this releases the need to record and label manually polyphonic events which can be hard and time consuming. The procedure is rather simple: based on the polyphonic generative model, we first draw two class labels from which two sounds are drawn. Then each sound is split in time frames and converted into normalized power spectra, and the energy of each frequency domain frame is computed.

Finally pairwise normalized spectra are computed by weighting two normalized monophonic spectra by the corresponding proportion φ (i) (defined in Eq. [START_REF] Sameh | On the Use of Time--Frequency Reassignment and SVM-Based Classifier for Audio Surveillance Applications[END_REF], with the form:

x (i) = φ (i) x 1(i) + 1 -φ (i) x 2(i) . (23)
This procedure leads to a learning set

x (i) , z (i) n i=1
, where z (i) = z 1(i) , z 2(i) . The previous distribution is estimated using a kernel density estimation of the form:

p (x|z) = 1 n z1 n z2 i z (i) =z K x, x (i) . (24)
As for the monophonic estimation, the kernel will be approximated using a multinomial kernel with the same convergence property. By construction, this method can theoretically only recover polyphonic sounds with the same proportions φ (i) as in the learning set, but we will see in Section 5 that the method performs well even with random proportions between sounds.

Reducing the computational load

The main objective of this paper is to provide a real-time audio classification method for both monophonic and polyphonic sounds. The previous two sections defined such a method from a real-time point of view (split sounds in short frames, use a kernel density estimate with multinomial kernels fast to compute). However, the computational load for computing the distribution is currently too high to be used in practice. This is the point of this section, that is to reduce the computational load.

Evaluating the complexity of the classification task

Consider the following writing of the monophonic distribution in Eq. ( 11):

p (x|z) ∝ i z (i) =z exp x (q) log p (q) (i) , ( 25
)
where x is the transpose x. The complexity of the algorithm is roughly O (n) since it consists in computing dot products between the unknown spectrum and all the p (q) (i) . Even for small datasets, the number of models can be very large (typically 100k prototypes), and therefore the computation time can be larger than the duration of a frame. As the previous equations are derived from the generative model (and cannot be changed), we can essentially reduce the complexity by reducing the number of prototypes n.

Hierarchical clustering of the monophonic models

A model reduction technique was already considered in [START_REF] Mesaros | Sound event detection in real life recordings using coupled matrix factorization of spectral representations and class activity annotations[END_REF] for NMF dictionaries: the authors used a k-means clustering technique and kept the centroid of the clusters as their reduced models. Their results suggested that the accuracy of the resulting system was not monotonic with the considered number of clusters. Contrary to these authors, the proposed model reduction algorithm of the present work allows to control the complexity and leads to an efficient computation time -accuracy trade-off. The idea is to perform class-wise hierarchical clustering of the prototypes parameterized by the p (q) (i) for i|z (i) = z, and use the resulting tree to create mixtures of these parameters according to the clusters. Hierarchical clustering requires a distance between the elements and a linkage criterion. We consider the standard Hellinger distance [START_REF] Hellinger | Neue Begründung der Theorie quadratischer Formen von unendlichvielen Veränderlichen[END_REF] since the elements to cluster are (discrete) probability distributions. The elements are linked using the Ward criterion [START_REF] Ward | Hierarchical Grouping to Optimize and Objective Function[END_REF] a usual linkage criterion.

In class z, a hierarchical clustering is performed as described previously using a class-wise reduction factor r z (or a global reduction r). This factor is defined as the initial number of prototypes n z over the number of prototypes after reduction n z : r z = n z /n z . For each cluster C i ∈ {C 1 , ..., C n z } resulting from the hierarchical clustering,

I i = i : p (q) (i) ∈ C i denotes the set of the p (q) (1) p (q) (2) p (q) (3) p (q) (4) n z C 1 C 2 p (q) (1) = 1 2 p (q) (1) + p (q) (2) p (q) (2) = 1 2 p (q) (3) + p (q) (4)
Figure 5: Illustration of the complexity reduction using hierarchical clustering. Consider that in a given class there are 4 models parameterized by p (q)

(1) , ..., p

(4) . The hierarchical clustering results in a tree representing how the models are structured. Choose a number of models after reduction n (here n = 2) and "cut" the tree so as to get the clusters C 1 , C 2 . Finally merge the models in each cluster by taking the mean mixture of these models, to get p (q)

(1) and p

(q) (2) .
indexes such that p (q) (i) belongs to the cluster C i (i = 1, ..., n z ). A mixture p (q) (i ) of the parameters p (q) (i) in cluster C i is defined as:

p (q) (i ) = 1 card (I i ) i∈I i p (q) (i) . (26)
The label associated to p (q) (i ) is noted z (i ) . An illustration of this process is displayed in Figure 5. It is shown in Section 5 that the reduction method improves the results up to a certain reduction value and suggests that the procedure limits the influence of overfitting.

Controlling the trade-off using a threshold procedure

A heuristic algorithm to reduce the model in a more efficient way was developed based on the previous algorithm. Instead of reducing all the classes with the same factor r z , the classes are reduced independently by choosing the reduction factor so that the class-wise test accuracy reaches a given threshold t accuracy . Consider an initial reduction factor for the classes r (0) z . The models are reduced using this factor and the class-wise test accuracy is computed: if it is above the threshold t accuracy the procedure stops, else the factor is reduced so as to increase the accuracy. This procedure is iterated until the threshold is reached. This procedure can be used to reach a given computation time threshold instead of a class-wise test accuracy threshold.

Remark: Polyphonic modeling reduction. Experiments in Section 5 show that a very reduced model can be used to perform monophonic classification without loosing too much accuracy. Therefore polyphonic classification will be performed using the reduced models parameterized by the p (q) (i ) described in Remark: We emphasize on the hierarchical clustering as a means to reduce the computation load, however when this particular clustering is not suited (for large datasets for instance), a more usual clustering algorithm such as k-means can be used (as we will see in the experiments).

Numerical experiments on real-world datasets

Databases and baseline systems

Databases. Two datasets are considered for monophonic classification. The first is the A-Volute dataset, composed of 704 video game sounds divided into 5 classes (alarm, detonation, step, vehicle, voice). The second is the ESC-10 dataset [START_REF] Piczak | ESC: Dataset for Environmental Sound Classification[END_REF], composed of 400 sounds divided into 10 classes. For polyphonic classification, the mixtures from the A-Volute dataset (created using the protocole of Section 3.3) and audio recordings from the video game Battlefield 1 are considered, the latter containing events of 3 different classes (detonation, step and voice), alone and by mixture of 2 and 3 classes. The TUT-SED 2017 [START_REF] Mesaros | TUT database for acoustic scene classification and sound event detection[END_REF] dataset is also considered, which contains real-life recordings in a street context. A summary of the datasets is available in Table 1.

Parameters tuning. All the sounds are resampled to F = 44.1kHz and centered. The considered frame size is ∆t = 2048 samples (46.4ms) and the time shift is fixed to δt = 512 samples (11.6ms). Other values of ∆t were tested to select the best (the results are not reported). The number of frequency bins B = ∆t/2 + 1 (the highest half of the spectrum is discarded according to the Nyquist theorem). The quantization of the spectra is set to q = B: in practice, this seems to be an optimal number, but values above B are also good. The learning sets are divided using the v-fold scheme to perform cross-validation (with v = 5): precisely the split is performed on the sounds. Frame sequences length is set to N = 40: this corresponds to approximately 500ms of audio data to make the decision.

Evaluation metrics. For monophonic classification, the evaluation metric is the classification accuracy in cross-validation, that is the number of correctly classified frames over the total number of frames. For polyphonic classification, the evaluation metric is the segment-based error rate (E.R.) and segment-based F1 score (F1) (see [START_REF] Mesaros | Metrics for Polyphonic Sound Event Detection[END_REF] for a detailed explanation of these metrics): a large F1 and a low E.R. mean that the system has very good performances. The F1 score is an extension of the accuracy since in a monophonic setting the F1 score becomes the accuracy (and the error rate becomes the misclassification rate).

Monophonic baseline systems. The proposed method, denoted by RARE, for monophonic classification is compared with several baseline systems. The first is a GMM with 20 and 30 components per classes using 20 MFCC and their first and second derivatives, inspired by [START_REF] Clavel | Events Detection for an Audio-Based Surveillance System[END_REF]. The second is also a GMM with 20 and 30 components per classes but using the proposed normalized power spectrum (NPS) as a feature. The third is a multi-class Support Vector Machine trained using error-correcting ouput code with one-versus-one coding design. The fourth is a Deep Convolutional Neural Network (DCNN) using log-mel spectrograms inspired by [START_REF] Piczak | Environmental sound classification with convolutional neural networks[END_REF], trained on 50 epochs. Human listening results were gathered for the two monophonic datasets: the results for ESC-10 are available in [START_REF] Piczak | ESC: Dataset for Environmental Sound Classification[END_REF]. For the A-Volute dataset an experiment was carried out where 27 subjects classified 50 0.5s-sounds randomly chosen in the 5 classes. Polyphonic baseline systems. The polyphonic classification method RARE was also compared with a neural network based method, called CRNN (Convolutional Recurrent Neural Network) inspired by [START_REF] Parascandolo | Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection[END_REF], trained on 100 epochs. It uses log-mel spectrogram to feed a neural network that consists in convolutional layers (for feature extraction) followed by recurrent layers (for temporal detection) and ended by a dense layer that performs classification. As this method usually uses dataset with already mixed sounds, we created artificial mixtures on the A-Volute dataset to train the network. Once the network is trained, a threshold th is used to predict the labels based on the output scores: the default value is 0.5 but we also tested other values (from 0.1 to 0.5).

Remark: polyphonic dataset creation. The polyphonic training data was created by considering a reduced monophonic training dataset. Indeed, the number of possible mixtures from the original dataset was too large to fit a standard computer memory, so that the mixtures are created using a reduced dataset. We define r mono the reduction factor of the monophonic prototypes and r poly the reduction factor of the polyphonic prototypes. For r mono = 100, the monophonic dataset is reduced by a factor 100 and mixtures were created using this reduced dataset. Then the mixtures were classified using the reduced monophonic prototypes and the polyphonic ones. The CRNN has to be trained using a dataset containing mixed sounds, this is why artificial mixtures were created on the A-Volute dataset. The parameter N mixt controls the number of frames from each class mixed together: if N mixt = 200, there were 200 frames from the first class mixed with 200 frames from the second class.

Results on the monophonic classification

Proposed method without reduction. Tables 2 and3 summarize the results for the A-Volute and ESC-10 datasets respectively for: the different methods (RARE, GMM, SVM and DCNN), the different features (NPS or MFCC) and the corresponding tuning parameter (if any). The results are the accuracy (in %), the training time (in seconds) and the testing time (in milliseconds). In the case of the GMM, the hyperparameter to tune is the number of components in the mixture. The reduction factor r in the proposed method is not a truly hyperparameter since it controls the trade-off between accuracy and the testing time (see the next paragraph). In term of accuracy, the best method is the GMM with a relatively few number of components but the proposed method is the second one and achieves good results compared to the other methods, in particular the DCNN and the SVM. We can see that on the A-Volute database, the GMM with NPS performs quite well compared to the GMM with MFCC, which indicates that the NPS is an effective space to describe sounds. The results on the ESC-10 dataset for the proposed method decrease because one class (over the 10) is not well recognized, but overall the results are similar than for the A-Volute dataset.

Proposed method with reduction. The model reduction algorithm seems to be very efficient because the number of prototypes can be reduced by a factor 100 without loosing too much accuracy, but it decreases the testing time dramatically (see last column of Tables 2 and3). For a reduction factor of r = 10 the proposed method can be used in real-time because the testing time is lower than 50ms (which corresponds to the frame length). Moreover, for reduction factors above to 10, the results decrease: this may mean that the method overfits when there is no reduction and this reduction tends to improve the results. However for a large reduction factor there are less prototypes to compute the classification rule so that the results decrease.

Pros and Cons. The proposed method has several advantages compared to the state-of-the-art methods. One advantage is that there is no feature extraction. The monophonic proposed method is not the best method in term of accuracy: a fine tuned GMM can have better results. However, the GMM needs to have a number of components set beforehand or optimized using a model selection criterion (such as Bayesian Information Criterion or the classification accuracy), which increases the training time. The DCNN has a complex network architecture which is hard to fine tune and which is time consuming. The proposed method has this second advantage that the training time is small compared to the other methods. Finally, with the model preprocessing, the proposed method can be used in real-time. To summarize, the proposed method is the second best model (just before the fine tune GMM) and has the ability to control the trade-off between computation time and accuracy.

Results on the polyphonic classification

Proposed method without reduction. Table 4 summarizes the results on the A-Volute dataset for the polyphonic classification task in the same way as in the monophonic classification task. This table shows that the proposed method outperforms the CRNN in term of scores and testing time. When the reduction r mono decreases, the scores are better since more prototypes are used to construct the polyphonic dataset, but the testing time increases dramatically.

We also see that the threshold th of the CRNN controls a trade-off between the F1 and E.R.: the default value 0.5 seems to be the optimal trade-off. However the results are far worst than our method.

Tables 5 and6 summarize the results for the Battlefield and TUT-SED dataset like previously. The main difference between these datasets and the A-Volute dataset is that mixtures are already present in the datasets. This is why the proposed method uses only a reduction factor r poly for the polyphonic classes. For the Battlefield dataset, the proposed method performs better both in term of scores (F1 and error rate) and times (training and testing). For the TUT-SED dataset, with no reduction, the proposed method does not reach the real-time objective and performs badly: since this database contains 6 classes with at most 4 classes mixed together at the same time, there are equivalently around 50 classes which are effectively present so that a completely random decision would have an accuracy of approximately 2%. It is then now interesting to express the following reduction technique to reach the real-time target.

Proposed method with reduction. For a smaller r mono , the proposed method for the A-Volute dataset must use a polyphonic reduction step to work in real-time. Like the monophonic reduction, the polyphonic reduction increases the results so that without reduction there may be overfitting of the proposed method. The Battlefield dataset shows a similar behaviour: better scores for a reduction of a factor 10 and then a decrease for a large reduction factor. The TUT-SED dataset was reduced using a k-means algorithm since the hierarchical clustering was not able to reduce a large amount of prototypes (more than 100k): the results are better with the reduction, which shows again that the reduction brings accuracy benefits in our case.

Pros and cons. All the previous methods used for monophonic classification are not designed to work on polyphonic classification using a monophonic dataset, contrary to our proposed method (RARE), which is a major advantage. Indeed the proposed method does not have to learn the mixtures of sounds but only the individual sounds. Moreover, the reduction of the complexity has a regularizing effect which is very effective on the TUT-SED database for instance. Even with multiple reduction of the complexity, the training time of our proposed method is still lower than the CRNN, which is trained using iterative algorithms such as stochastic gradient descent.

Conclusion

This work dealt with a new method for monophonic and polyphonic real-time audio sources classification. The method used the whole power spectrum instead of predefined audio descriptors, which is also useful for polyphonic events thanks to the additivity of uncorrelated spectra. The classification was based on a generative model of power spectra, which has the main advantage of being hypothesis free and allows to derive a temporal MAP to make the decision. Contrary to other methods like neural networks, this technique modeled both monophonic and polyphonic sources in a single framework.

As shown in the experiments, the polyphonic classification performed quite well on owned and benchmark datasets and outperforms the CRNN. Thanks to the reduction of the complexity, the method has a low computation time and can be used in real-time. This polyphonic classsification is built on monophonic sounds and does not have to learn from already mixed sounds, which is a major advantage compared to other methods like neural networks or Gaussian mixture models for instance. As mentionned in the experiments, the reduction of the complexity has a regularizing effect in addition to an efficient computation time -accuracy trade-off.

Since the NPS feature was essentially designed to handle the polyphonic classification, it is not surprising that the monophonic classification does not fully compete with usual monophonic classification algorithms such as a fine tuned GMM. Moreover, there may be overfitting of our method because of the nonparametric estimation using kernels. As the proposed method is flexible regarding the choice of the kernel, we also tested the Dirichlet kernel and the bGMM kernel (which is the GMM for binned data [START_REF] Baelde | A mixture model-based real-time audio sources classification method[END_REF]). The Dirichlet kernel is constructed by considering that the NPS is the mode of the Dirichlet kernel. This kernel does not improve the results as the accuracy is 48.30% for a real-time application (after a reduction by 20). The bGMM is learned using the quantized NPS and by selecting the number of components using ICL [START_REF] Biernacki | Assessing a Mixture Model for Clustering withthe Integrated Classification Likelihood[END_REF] (Integrated Classification Likelihood). This kernel does not improve the results either since the accuracy is 82.7%. A future area of research would be to change the estimation of the condition distribution of Eq. 11 and 24, using a semiparametric estimation for instance. However it is worth noting that the considered nonparametric estimation using multinomial kernels allows to derive an efficient computation -accuracy trade-off.

The normalized power spectrum associated to x freq τ t is x τ t : where x 1τ t and x 2τ t are defined as in Eq. ( 5). Define the proportion φ τ t as:

x τ t =
φ τ t = P 1τ t P 1τ t + P 2τ t • (B.8)
The previous result becomes:

x τ t = φ τ t x 1τ t + (1 -φ τ t )x 2τ t . (B.9)

  sounds. Finally, Neural Networks (NN) -and their deep variants such as Deep Convolutional/Recurrent Neural Networks (DC/RNN) -have recently attracted the attention of researchers. Biondi et al. used a normalized spectrum as input for a standard NN [21] and Dadula et al. considered the MFCC [22]. Palaz et al.

Figure 2 :

 2 Figure 2: (a) Example with a real sound x time containing a voice. This sound is split into time frames x time τ (red rectangle) of size ∆t and shifted by τ δt (b), and each time frame is converted into the normalized power spectrum xτ (c).

Figure 4 :

 4 Figure 4: Consider that two power spectra x 1 (red, left bottom curve) and x 2 (left top blue curve) are mixed. The resulting spectrum x = φx 1 + (1 -φ)x 2 (thick black right curve) will depend on the relative powers of the two spectra, represented by φ.

Table 1 :

 1 Summary of the datasets used for the experiments. The polyphonic complexity is the maximum number of classes mixed simultaneously: when this complexity equals 2 that means there are at most 2 different classes are mixed simultaneously.Section 4.2. The goal is to construct a new learning set containing relevant mixtures models. The method consists in computing pairwise mixture models from the different classes and reduces this collection of models using a hierarchical clustering like in Section 4.2: the reduction factor is set so that the computation time does not exceed the objective.

	Dataset name	Type	Number of classes Polyphonic complexity
	A-Volute	Monophonic	5	1
	ESC-10	Monophonic	10	1
	A-Volute	Polyphonic	5	2
	Battlefield	Polyphonic	3	3
	TUT-SED	Polyphonic	6	4

Table 2 :

 2 Monophonic classification task (∆t = 2048). Summary of the results for the A-Volute dataset in term of accuracy, training time (in seconds) and testing time per frame (in milliseconds) for different methods. RARE stands for our proposed method and NPS is the normalized power spectrum.

	Method	Features Param.	Accuracy (%) Training t. Testing t.
	RARE	NPS	-	82.5 (6.0)	7.6 × 10 1	2.4 × 10 2
	RARE	NPS	r = 10 r = 100	83.1 (5.6) 79.9 (5.7)	1.0 × 10 3	2.5 × 10 1 2.7 × 10 0
	GMM [12]	MFCC	20 comp. 30 comp.	86.6 (8.5) 84.5 (7.1)	1.3 × 10 2 1.9 × 10 2	3.0 × 10 0 4.1 × 10 0
	GMM [12]	NPS	20 comp. 30 comp.	77.2 (7.3) 88.0 (7.5)	7.3 × 10 2 1.3 × 10 2	2.8 × 10 3 4.0 × 10 3
	SVM	NPS	-	36.9 (9.2)	1.2 × 10 4	2.9 × 10 1
	DCNN [6]	MFCC	-	61.6 (16.0)	1.4 × 10 3	3.1 × 10 0
	CRNN [25]	MFCC	-	19.2 (1.6)	4.0 × 10 2	2.9 × 10 1
	Human	-	-	91.7	-	-

Table 3 :

 3 Monophonic classification task (∆t = 2048). Summary of the results for the ESC-10 dataset in term of accuracy, training time (in seconds) and testing time per frame (in milliseconds) for different methods. RARE stands for our proposed method and NPS is the normalized power spectrum.

	Method	Features Param.	Accuracy (%) Training t. Testing t.
	RARE	NPS	-	64.7 (3.2)	8.2 × 10 1	3.7 × 10 2
	RARE	NPS	r = 10 r = 100	67.5 (4.4) 67.1 (5.1)	7.9 × 10 2	3.7 × 10 1 4.0 × 10 0
	GMM [12]	MFCC	20 comp. 30 comp.	78.1 (4.0) 78.3 (3.5)	2.2 × 10 2 3.0 × 10 2	3.0 × 10 0 3.5 × 10 0
	GMM [12]	NPS	20 comp. 30 comp.	56.7 (3.5) 56.1 (4.2)	1.2 × 10 3 2.0 × 10 3	5.7 × 10 3 8.0 × 10 3
	SVM	NPS	-	46.3 (4.4)	2.2 × 10 4	4.7 × 10 1
	DCNN [6]	MFCC	-	40.7 (6.0)	2.1 × 10 3	1.3 × 10 0
	CRNN [25]	MFCC	-	10.0 (0.0)	6.0 × 10 2	2.6 × 10 1
	Human	-	-	95.7	-	-

Table 4 :

 4 Polyphonic classification task (∆t = 2048). Summary of the results for the A-Volute dataset in term of F1 score (F1), error rate (E.R.), training time (in seconds) and testing time per frame (in milliseconds) for different methods. rmono is the reduction factor of the monophonic prototypes and r poly is the reduction factor of the polyphonic prototypes. th is the threshold of the CRNN output. A large F1 and low E.R. means good performances. RARE stands for our proposed method.

	Method	Tuning param.		F1	E.R.	Training t.	Testing t.
	RARE	rmono = 400	-	69.4 (3.2)	46.2 (3.2)	7.7 × 10 1	6.4 × 10 0
	RARE	rmono = 100	-r poly = 20	72.3 (4.8) 71.4 (3.8)	43.1 (5.4) 41.3 (4.5)	1.9 × 10 4	7.9 × 10 2 4.2 × 10 1
	RARE	rmono = 50	-	74.5 (4.7)	40.6 (5.1)	7.7 × 10 1	3.0 × 10 3
			th = 0.5	39.8 (26.5)	70.3 (20.0)
	CRNN [25]	Nmixt = 200	th = 0.2	56.4 (3.8)	114.4 (6.0)	4.8 × 10 3	2.8 × 10 1
			th = 0.1	55.9 (2.1)	141.7 (15.0)
			th = 0.5	53.1 (0.6)	60.3 (0.6)
	CRNN [25]	Nmixt = 300	th = 0.2	56.2 (1.9)	141.1 (14.4)	1.3 × 10 4	3.1 × 10 1
			th = 0.1	57.1 (0.1)	149.1 (1.9)

Table 5 :

 5 Polyphonic classification task (∆t = 2048). Summary of the results for the Battlefield dataset in term of F1 score (F1), error rate (E.R.), training time (in seconds) and testing time per frame (in milliseconds) for different methods. r poly is the reduction factor of the polyphonic prototypes. th is the threshold of the CRNN output. A large F1 and low E.R. means good performances. RARE stands for our proposed method.

	Method	Tuning param.	F1	E.R.	Training t. Testing t.
	RARE	-	66.0 (4.3)	44.5 (6.3)	5.1 × 10 1	9.4 × 10 1
	RARE	r poly = 10 r poly = 50	69.2 (2.8) 67.9 (1.8)	40.9 (2.6) 42.3 (2.2)	1.2 × 10 2	1.0 × 10 1 2.1 × 10 0
		th = 0.5	61.8 (3.4)	54.4 (4.4)
	CRNN [25]	th = 0.3 th = 0.2	63.5 (4.0) 77.9 (17.3) 65.2 (2.8) 85.1 (7.0)	1.8 × 10 2	2.6 × 10 1
		th = 0.1	56.1 (1.5) 156.7 (9.4)

Table 6 :

 6 Polyphonic classification task (∆t = 2048). Summary of the results for the TUT-SED dataset in term of F1 score (F1), error rate (E.R.), training time (in seconds) and testing time per frame (in milliseconds) for different methods. r poly is the reduction factor of the polyphonic prototypes. th is the threshold of the CRNN output. A large F1 and low E.R. means good performances. RARE stands for our proposed method.

	Method	Tuning param.	F1	E.R.	Training t. Testing t.
	RARE	-	30.1 (4.1)	85.4 (9.8)	5.4 × 10 2	1.3 × 10 3
	RARE	r poly = 100 r poly = 1000	40.2 (9.8) 47.9 (11.9)	60.1 (9.7) 59.1 (11.3)	8.8 × 10 2 8.8 × 10 2	1.6 × 10 1 2.5 × 10 0
		th = 0.5	0.0 (0.0)	100.0 (0.0)
		th = 0.4	7.5 (15.0)	100.0 (0.0)
	CRNN [25]	th = 0.3	25.2 (16.9)	99.1 (2.5)	2.3 × 10 3	2.9 × 10 1
		th = 0.2	39.1 (7.7)	137.4 (57.9)
		th = 0.1	34.5 (5.2)	308.4 (122.1)

  Define P 1τ t = ||x freq 1τ t || 2 and P 2τ t = ||x freq 2τ t || 2 the powers of the two sources. Some calculi lead to the following result: x τ t = x 1τ t P 1τ t P 1τ t + P 2τ t + x 2τ t P 2τ t P 1τ t + P 2τ t , (B.7)

	|x freq 1τ t | ||x freq 1τ t || 2 + |x freq 2τ t | 2 2 + ||x freq 2τ t || 2 •	(B.6)

Polyphonic SED must be distinguished from polyphonic music, the latter referring to playing different melodies simultaneously.

A software editor company specialized in

3D sound.

Appendix A. Point-wise convergence of the approximated kernel Consider X (i) ∈ R B a random vector that sums to 1 related to the spectrum x (i) in the learning set. X (q) (i) is defined as the closest integer random vector to qX (i) by:

We have that 1 q X (q) (i) converges in distribution to X (i) as q goes to infinity. As a result the kernel defined by:

with parameter p

Appendix B. Derivation of the polyphonic spectrum decomposition Eq. ( 18) is derived using the following arguments. A polyphonic sound is the sum in the time domain of several sound sources:

The framing operation is linear so that:

For a given frame:

By the linearity of the Fourier transform, if two signals are summed in the time domain they will be summed in the frequency domain. Denoting by x freq 1τ t and

x freq 2τ t the complex spectra, the sum of these spectra x freq τ t is:

The modeling disclosed in Section 2.4 requires to deal with a normalized version composed of the elements |x freq τ t | 2 as in Eq. ( 5). Two sources from different classes are assumed to be uncorrelated signals (a common assumption in signal separation [START_REF] Grais | Deep neural networks for single channel source separation[END_REF]), meaning that the power spectrum of the sum is approximately the sum of the power spectra: