Gustavo Sena Mafra
email: gsenamafra@gmail.com

Ngoc Q K Duong

Alexey Ozerov

Patrick Pérez

Acoustic scene classification: An evaluation of an extremely compact feature representation

Keywords: Acoustic scene classification, Audio features, Multilayer Perceptron, Convolutional Neural Network, Support Vector Machine

come   L'archive ouverte pluridisciplinaire

INTRODUCTION

Acoustic scene classification (ASC), a particular form of audio classification, consists in using acoustic information (audio signals) to infer the context of the recorded environment [START_REF] Barchiesi | Acoustic scene classification: Classifying environments from the sounds they produce[END_REF]. Examples of such environments are bus, office, street, etc... It offers a wide range of applications in connected home, e.g. expensive video cameras can be replaced by cheap microphones for monitoring daily activity, and for smartphones, e.g. they could automatically switch to silence mode during a meeting or automatically increase the sound volume in a noisy environment. However, real-life ASC is not a trivial task as recognising a greater variety of sounds in both indoor and outdoor environments would require a new set of strategies and adjustments of existing machine learning techniques to make the most out of the available data.

While speaker identification [START_REF] Reynolds | Robust text-independent speaker identification using gaussian mixture speaker models[END_REF], speech recognition [START_REF] Rabiner | Fundamentals of speech recognition[END_REF], and some audio classification tasks in music information retrieval such as music genre recognition [START_REF] Li | A comparative study on contentbased music genre classification[END_REF][START_REF] Kostek | Report of the ISMIR 2011 contest: music information retrieval[END_REF] or music instrument recognition [START_REF] Kostek | Report of the ISMIR 2011 contest: music information retrieval[END_REF] have been studied for a long time, the real-life ASC task has become active quite recently in the research community. While the classification task itself has been studied since at least 2002 [START_REF] Peltonen | Computational auditory scene recognition[END_REF], it was only recently that efforts were made to provide a benchmark for the task, with the new initiative of the DCASE challenges in 2013 and 2016. Various techniques have been proposed to tackle the problem with the use of different acoustic features (e.g. cochleogram representation, wavelets, auditory-motivated representation, features learned by neural networks) and different classifiers (e.g. Support Vector Machine (SVM), Gaussian Mixture Model (GMM), Hidden Markov Model (HMM)) [START_REF] Stowell | Detection and classification of acoustic scenes and events[END_REF]. One of the most popular approaches, known as bag-of-frames (BOF) approach [START_REF] Aucouturier | The bag-offrames approach to audio pattern recognition: A sufficient model for urban soundscapes but not for polyphonic music[END_REF][START_REF] Lagrange | The bag-of-frames approach: a not so sufficient model for urban soundscapes[END_REF] is used as a baseline in the DCASE challenge, and exploits the long-term statistical distribution (by GMM) of the short-term MFCCs.

Besides the DCASE challenge, nonnegative matrix factorization (NMF) was recently exploited for sound event detection in real life recordings [START_REF] Mesaros | Sound event detection in real life recordings using coupled matrix factorization of spectral representations and class activity annotations[END_REF]; recurrent neural networks (RNN) were investigated for polyphonic sound event detection in real life recordings [START_REF] Parascandolo | Recurrent neural networks for polyphonic sound event detection in real life recordings[END_REF]; and deep neural networks (DNN) have been developed for sensing acoustic environment [START_REF] Lane | Deepear: Robust smartphone audio sensing in unconstrained acoustic environments using deep learning[END_REF]. It would be interesting to note that while DNNs [START_REF] Hinton | Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups[END_REF][START_REF] Lecun | Deep learning[END_REF] were recently applied with great success to many different audio, visual and multimedia tasks, it was less investigated within the DCASE 2013 challenge and one of the reasons would be the lack of a substantial amount of labeled data for training.

This paper aims to study the use of well-established lowlevel acoustic feature representations and different machine learning techniques, including DNN-based methods and SVM, for the ASC task. While most existing approaches extract an acoustic feature vector for each short-term audio frame, then perform a frame-based classification based either on BOF over GMMs [START_REF] Aucouturier | The bag-offrames approach to audio pattern recognition: A sufficient model for urban soundscapes but not for polyphonic music[END_REF][START_REF] Lagrange | The bag-of-frames approach: a not so sufficient model for urban soundscapes[END_REF] or simple majority voting [START_REF] Geiger | Recognising acoustic scenes with large-scale audio feature extraction and SVM[END_REF][START_REF] Li | Auditory scene classification using machine learning techniques[END_REF][START_REF] Stowell | Detection and classification of acoustic scenes and events[END_REF], we investigate the use of an another feature representation, i.e. a single vector for a whole audio scene, aiming an extremely compact representation that greatly reduces the computational cost for the whole ASC system, since the number of examples to be used to train the classifier is drastically reduced. We evaluate the use of this compact feature with SVM and MLP on both DCASE 2013 and DCASE 2016 datasets and the performance are more or less equivalent to a frame-based approach with majority voting strategy. Furthermore, it results in classification accuracy comparable to the best systems participating in the DCASE 2013 challenge.

The rest of the paper is organized as follows. In Section 2 we present the general framework which involves different approaches for feature extraction and classification. Experiment results on DCASE dataset obtained by our approaches and some state-of-theart methods are discussed in Section 3. We finally conclude in Section 4.

ACOUSTIC SCENE CLASSIFICATION FRAMEWORK

The general workflow of an ASC system is usually divided into two major steps as shown in Fig. 1. In the first, the feature extraction step, various types of hand-crafted representations have been considered in the literature such as chroma, pitch, spectrograms, zero-crossing rate, and linear predictive coding coefficients [START_REF] Barchiesi | Acoustic scene classification: Classifying environments from the sounds they produce[END_REF]. Among them, features based on Mel-frequency Cepstrum Coefficients (MFCCs) computed for each short-time frame are arguably the most common one, as it can be seen by the DCASE 2013 Challenge, where out of 12 systems submitted, at least 7 involved MFCCs [START_REF] Barchiesi | Acoustic scene classification: Classifying environments from the sounds they produce[END_REF]. More recent DNN-based approaches usually attempt to learn higher level features from these low-level signal representations [START_REF] Lane | Deepear: Robust smartphone audio sensing in unconstrained acoustic environments using deep learning[END_REF][START_REF] Bengio | Representation learning: A review and new perspectives[END_REF]. In the classification step, popular classifiers include SVM and GMM [START_REF] Stowell | Detection and classification of acoustic scenes and events[END_REF]. In the following, we will first describe the standard Mel-log-spectrogram, as the low-level feature used in this work, and the proposed compact representation from it in Section 2.1. We then briefly present some exploited classification approaches in Section 2.2. The choice of hyperparameters for both feature extraction and classifiers is discussed in Section 2.3.

Feature extraction

The time domain audio signal x(n) is first transformed into the frequency domain by means of the short-term Fourier transform (STFT) as

STFT{x}(m, ω) = +∞ n=-∞ x(n)w(n -mL)e -jωn (1)
where w(n) is a window function (which is Hanning window in our implementation), m denotes frame index and L the frame shift. The spectrogram is then defined as

S(m, ω) = | STFT{x}(m, ω)| 2 (2)
In our DNN-based system, we use spectrogram with a logarithmic amplitude scale (named log-spectrogram) as the frame input feature which is computed as

FLog-spec(m, ω) = log(S(m, ω)). (3)
In our other systems, we first map the spectrogram S(m, ω) into the auditory-motivated Mel frequency scale -denoted by MS(m, ω), then transform it into logarithmic scale as

F Mel-log-spec (m, ω) = log(MS(m, ω)). (4)
Note that with the CNN-based system, we use the raw logspectrogram as the input feature in order to give flexibility for the CNN to learn a higher level feature representation optimized for the ASC task. For SVM-based systems we have tested four different features: spectrogram, log-spectrogram, Mel-log-spectrogram, and MFCC, and found that the two last ones result in a very similar ASC performance that outperforms the two first ones. As the Mellog-spectrogram is simpler to compute than the MFCC, we use it as the main acoustic feature in this paper. Finally, we propose to average the feature vectors for all frames so as to present a whole audio example by an extremely compact feature vector whose entries are computed as

f Avg-mel-log-spec (ω) = 1 M M m=1 F Mel-log-spec (m, ω). (5)
This type of averaging of features is very straight-forward and has already been used in past works [START_REF] Piczak | Esc: Dataset for environmental sound classification[END_REF] [START_REF] Salamon | A dataset and taxonomy for urban sound research[END_REF] in ASC. However, no submitted systems in the DCASE 2013 Challenge made use of it, misleadingly pointing to a lack of efficiency of this method.

Classification approaches

Support vector machine

SVM has been known as one of the most popular classifiers for many different tasks. It was also widely used in the DCASE 2013 challenge [START_REF] Stowell | Detection and classification of acoustic scenes and events[END_REF]. In our work, we used SVM as a benchmark classifier to evaluate the effectiveness of different features, as mentioned in Section 2.1, as well as to obtain the optimal choice of hyperparameters (e.g. the window size and the number of Mel-frequency coefficients) for the considered task.

In our implementation, we train SVMs using a coordinate descent algorithm [START_REF] Hsieh | A dual coordinate descent method for large-scale linear SVM[END_REF] and following a one-vs-the-rest scheme to perform classification of multiple classes [START_REF] Fan | LIBLINEAR: A library for large linear classification[END_REF]. We have tested SVM with linear kernel and Gaussian radial basis function (RBF) kernel and found that the linear kernel works slightly better than RBF kernel for the DCASE 2013 dataset.

Multilayer Perceptron

Multilayer Perceptron (MLP) is a fully connected feedforward artificial neural network architecture that maps sets of input data onto a set of appropriate outputs. It can be seen as a logistic regression classifier where the input is first transformed using a non-linear transformation [START_REF] Hornik | Multilayer feedforward networks are universal approximators[END_REF][START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF]. A typical set of equations for an MLP is the following. Layer k computes an output vector h k using the output h k-1 of the previous layer, starting with the input x = h 0 ,

h k = f (W k h k-1 + b k ) (6)
where b k denotes a vector of offsets (or biases) and W k a matrix of weights. The function f is called the activation function and it is applied element-wise. Common options for it are sigmoid function, hyperbolic tangent, and rectified linear unit (ReLU). The latter, i.e. f (x) = max(0, x), was used to obtain the results reported in this document.

The top layer output is used for making a prediction and is combined with the groundtruth label into a loss function. We use softmax as the classification layer and the log-likelihood loss function regularized with 1 and 2 penalties. This cost function is then optimized using mini-batch stochastic gradient descent (SGD) with an adaptive learning rate [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF] and dropout is performed between the hidden layers [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF].

Convolutional Neural Network

Convolutional Neural Network (CNN) is a type of neural network designed to exploit the redundancy and correlation between neighbour units. It has gained great success in different fields such as image and video recognition, natural language processing, speech recognition, etc., [START_REF] Lecun | Deep learning[END_REF]. This motivates us to investigate the use of CNN for the ASC task in this work.

We trained CNNs over the log-spectrogram of the signals with a structure of vertical filters, i.e. the frequency bins can also be interpreted as a CNN channel (as in RGB channels for images) instead of a dimension and the convolution is ran over the time axis. This type of structure was proposed for music recommendation in Spotify 1 and is justifiable by the fact that an audio "pattern" detected in a high-frequency region is usually different from that same pattern in a low-frequency region. Thus it is desirable to model the vertical filters to extract more meaningful information from the spectral representation. More details about the implemented CNN architecture can be found in Section 3.1.

Hyperparameter optimization

The choice of hyperparameters in each step of the ASC system or in any machine learning task can significantly affect the final classification result. Such hyperparameters are e.g. the window length and hop length in the STFT computation for feature extraction, the regularization parameter for SVM, the number of hidden units in an MLP, and the step size for the SGD algorithm in DNN based methods. The conventional strategy of tuning these parameters manually would not be feasible as it requires a great number of trials so that all parameters can be optimized together. Thus, in this work we incorporate a Bayesian optimization [START_REF] Snoek | Practical bayesian optimization of machine learning algorithms[END_REF] method to find these parameters altogether. The algorithm models the generalization accuracy of a classifier as a function of the corresponding parameters, and finds the optimal parameters that maximize the expected accuracy given the observed dataset.

In our implementation, we use Hyperopt [START_REF] Bergstra | Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms[END_REF], a Python library for optimizing hyperparameters in machine learning algorithms, with the Tree of Parzen Estimators (TPE) [START_REF] Bergstra | Algorithms for hyper-parameter optimization[END_REF], an algorithm that falls into the class of sequential model-based optimization (SMBO) [START_REF] Hutter | Sequential model-based optimization for general algorithm configuration[END_REF] algorithms. The TPE algorithm performs cross-validation with the development datasets of DCASE 2013 and DCASE 2016 and finds an optimal set of hyperparameter values. It is interesting to note that the optimal window size for STFT computation found by the TPE algorithm is quite long, i.e. about half of a second. This can be explained by the fact that the acoustic events are more spread in time compared to e.g. speech which is very localized so as the window length used for STFT is usually much smaller.

EXPERIMENTS

We evaluate the ASC performance of our four implementing systems with the benchmark DCASE 2013 dataset, which allows to compare with the state-of-the-art approaches participating in the challenge, in Section 3.1. We then present the result with DCASE 2016 dataset in Section 3.2. Our first system (named Proposed SVM-A) uses an extremely compact feature as the Mel-logspectrogram coefficients averaged for all frames, and SVM with a linear kernel as classifier. The second system (named Proposed SVM-V) performs frame classification by SVM with a linear kernel, 1 http://benanne.github.io/2014/08/05/spotify-cnns.html then majority voting in the end. The third system (named Proposed MLP) takes the compact averaged Mel-log-spectrogram as input, learns an intermediate feature representation by MLP, then classifies by softmax as the last layer of the MLP. The fourth system (named Proposed CNN) takes log-spectrogram as low-level input feature, learns higher feature representations by CNN layers, then classifies by softmax.

Results with the DCASE 2013 dataset

The DCASE 2013 dataset consists of 30-second audio segments belonging to 10 classes. Each class has 10 segments in the development set and 10 other examples in the test set [START_REF] Giannoulis | Detection and classification of acoustic scenes and events: An IEEE AASP challenge[END_REF].

The ASC performance was evaluated in terms of the classification accuracy, averaged over all classes, and shown in Table 2. Note that as we did not have access to the groundtruth labels of the test set at the time of these experiments, we evaluated our systems averaging with the standard 5-fold cross-validation on the development set only, while results for most other approaches in the table are obtained with the test set [START_REF] Stowell | Detection and classification of acoustic scenes and events[END_REF]. Some hyperparameters for each systems were found by the Bayesian optimization method presented in Section 2.3. More detailed settings for each system are as follows. The window length for the STFT was set by 0.57 seconds and 0.41 seconds for the SVM-A and SVM-V system, respectively, the number of Mel-frequency coefficients is about 1900, the regularization parameter C in SVM for SVM-A and SVM-V were 0.98 and 0.62, respectively. MLP had one hidden layer with 677 units, dropout rate and learning rate for parameter training was set by 0.08 and 0.011, respectively. CNN had 3 convolutional layers, the number of filters for each layer are 50, 29, and 19, respectively, and the max-pooling ratios between layers are 3, 4, and 3.

As it can be seen, the two systems based on SVMs outperform the ones based on NNs. This can be explained by the fact that the dataset may be not large enough for training DNNs directly. Three of our proposed systems (SVM-A, SVM-V, and MLP) achieve comparable performance with some of the best performing approaches in the DCASE 2013 Challenge -as we suppose that there is not much difference between development set and the test set. Moreover, we achieve higher accuracy than Li et al. [START_REF] Li | Auditory scene classification using machine learning techniques[END_REF] in the same development set. Finally, we note that the proposed feature, which is extremely compact so as to represent a whole 30-second audio segment by just a single vector, can be sufficient for the classification as the SVM-A and MLP obtained 75% and 72% accuracy, respectively.

Results with the DCASE 2016 dataset

The DCASE 2016 dataset is structured in a similar way as the DCASE 2013 dataset. However the number of acoustic classes is extended to 15, and the number of examples for each class is significantly enlarged to 78 for the development set and 26 for the test set.

The results for development set obtained by our four systems are shown in Table 2, where the best performance of 80% is achieved by the SVM-A system with a window length of 0.42 seconds and a hop size of 0.14 seconds for the STFT computation. This result confirms again the benefit of using the proposed compact feature representation and the use of a long window for the spectral transformation. The MLP, which obtains similar performance as the baseline, had two hidden layers with 66 and 199 units, respectively, SGD was used for parameter training with learning rate of

Method

Acoustic feature Classifier Accuracy Baseline MFCC "bag-of-frames" GMM 55% Geiger et al. [START_REF] Geiger | Recognising acoustic scenes with large-scale audio feature extraction and SVM[END_REF] Diverse features SVM + majority voting 69% Roma et al. [START_REF] Roma | Recurrence quantification analysis features for auditory scene classification[END_REF] MFCC Train Tram 47 0 3 4 5 4 1 0 1 0 3 4 2 0 2 2 61 0 4 1 0 0 0 0 2 0 2 2 3 1 4 0 46 0 0 2 19 4 0 0 0 2 0 0 1 0 2 0 71 0 0 0 0 0 0 0 0 1 0 4 0 0 0 0 75 0 0 0 0 2 0 0 1 0 0 1 0 0 0 0 75 0 0 0 0 0 0 2 0 0 0 0 0 0 2 0 76 0 0 0 0 0 0 0 0 6 0 1 0 0 13 1 30 7 0 3 6 9 1 1 0 1 0 0 0 0 0 0 72 1 0 0 0 4 0 0 0 0 0 0 0 0 0 0 76 0 2 0 0 0 0 0 0 0 0 0 0 8 1 0 69 0 0 0 0 9 0 5 0 2 1 0 1 0 0 0 49 11 0 0 5 0 0 0 9 7 0 0 2 0 0 19 35 0 1 17 3 0 0 2 0 0 0 1 0 0 1 0 49 5 2 0 0 0 0 0 0 0 0 2 0 0 0 0 74 0.003 and batch size of 100, weights for 1 and 2 penalties were 10 -5 and 10 -4 , respectively. The CNN, with the same configuration used for the DCASE 2013 dataset, still resulted in the lowest performance. These four systems will also be tested with the test set for participating in the DCASE 2016 challenge.

The confusion matrix for SVM-A is shown in Fig. 2, where rows are groundtruth, columns are the inferred class label, and values are number of the classified acoustic scene. As it can be seen, some environments containing a specific type of noise (such as car, metro station, forest path) are quite easy to recognize, while some others (such as home, residential area, park) are quite confusing.

CONCLUSION

In this article we present several approaches for the ASC task, targeting on fast systems working with very compact feature representations so that ASC can be implemented e.g. in smartphones. We investigate the use of Bayesian optimization for hyperparameter optimization and find its benefit in e.g. choosing the optimal window length for STFT or setting DNN parameters. By evaluating on benchmark DCASE datasets, we find that (1) a long window size for spectral transformation is more relevant for the environmental acoustic scenes, (2) a very compact feature representation by longterm temporal averaging of Mel-log-spectrogam coefficients would be sufficient for the task compared to more complicated approaches, and (3) similar accuracies are found for the two datasets, possibly owing to the similarities between these datasets or to the robustness of the proposed systems. Finally, it is worth noting that DNN approaches have not reached the same performance of the more classical SVM based systems so far. Thus future work would be devoted to investigate transfer learning strategies for DNN based systems where part of the DNN can be initially learned by a large amount of external audio data.

Figure 1 :

 1 Figure 1: General workflow of the acoustic scene classification framework.

Figure 2 :

 2 Figure 2: Confusion matrix of the SVM-A method in the development set of the DCASE 2016 database after a 4-fold cross-validation over 78 samples of each class.

Table 1 :

 1 Acoustic scene classification results with DCASE 2013 test dataset (for state-of-the-art approaches) and development dataset (for our proposed approaches). Note that other submitting systems resulting in less classification accuracy are not mentioned in the table.

			with Recurrence Quantification Analysis SVM	76%
	Proposed SVM-A Averaged Mel-log-spectrogram	Linear SVM	75%
	Proposed SVM-V Frame Mel-log-spectrogram	Linear SVM + majority voting	78%
	Proposed MLP	Averaged Mel-log-spectrogram	MLP with softmax as classification layer	72%
	Proposed CNN	Log-spectrogram	CNN with softmax as classification layer	62%
	Method		Acoustic feature	Classifier	Accuracy
	Baseline		MFCC	"bag-of-frames" GMM	75%
	Proposed SVM-A Averaged Mel-log-spectrogram Linear SVM	80%
	Proposed SVM-V Frame Mel-log-spectrogram	Linear SVM + majority voting	78%
	Proposed MLP	Averaged Mel-log-spectrogram	MLP with softmax as classification layer 75%
	Proposed CNN	Log-spectrogram	CNN with softmax as classification layer 59%

Table 2 :

 2 Acoustic scene classification results with DCASE 2016 development dataset.

	Beach	Bus	Cafe/restaurant	Car	City center	Forest path	Grocery store	Home	Library	Metro station	Office	Park	Residential area
	Beach
	Bus
	Cafe/restaurant
	Car
	City center
	Forest path
	Grocery store
	Home
	Library
	Metro station
	Office
	Park
	Residential area
	Train
	Tram


de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.


Dalia El Badawy
email: dalia.elbadawy@epfl.ch

Ngoc Q K Duong

Alexey Ozerov
email: alexey.ozerov@technicolor.com

On-the-fly audio source

published or not. The documents may come   L'archive ouverte pluridisciplinaire

On-the-fly audio source separation -a novel user-friendly framework Dalia El Badawy, Ngoc Q. K. Duong, Member, IEEE and Alexey Ozerov, Member, IEEE Abstract-This article addresses the challenging problem of single-channel audio source separation. We introduce a novel user-guided framework where source models that govern the separation process are learned on-the-fly from audio examples retrieved online. The user only provides the search keywords that describe the sources in the mixture. In this framework, the generic spectral characteristics of each source are modeled by a universal sound class model learned from the retrieved examples via non-negative matrix factorization. We propose several group sparsity-inducing constraints in order to efficiently exploit a relevant subset of the universal model adapted to the mixture to be separated. We then derive the corresponding multiplicative update rules for parameter estimation. Separation results obtained from automated and user tests on mixtures containing various types of sounds confirm the effectiveness of the proposed framework.

Index Terms-On-the-fly audio source separation, user-guided, non-negative matrix factorization, group sparsity, universal sound class model.

I. INTRODUCTION

A UDIO source separation is a desired processing step within many real-world applications such as sound postproduction, robotics, and audio enhancement [START_REF] Makino | Blind Speech Separation[END_REF]. However, it has remained a challenging task especially when the input is a single-channel mixture. Indeed, in this case the problem is highly ill-posed and, in contrast to multichannel mixing case, additional spatial information about the sources is not available. Earlier approaches usually assume that the sources are sparse in the short-time Fourier transform (STFT) domain and estimate the predominant source's STFT coefficients via e.g. binary masking [START_REF] Yılmaz | Blind separation of speech mixtures via time-frequency masking[END_REF] or 1 -minimization [START_REF] Winter | MAP-based underdetermined blind source separation of convolutive mixtures by hierarchical clustering and 1 -norm minimization[END_REF], [START_REF] Bofill | Underdetermined blind source separation using sparse representations[END_REF]. The separation performance achievable by these techniques is very limited in reverberant environments [START_REF] Vincent | The Signal Separation Campaign (2007-2010): Achievements and remaining challenges[END_REF], [START_REF] Ono | The 2015 signal separation evaluation campaign[END_REF] where the sources' STFT coefficients are quite overlapped. A more recent class of algorithms known as informed source separation [START_REF] Ewert | Score-informed source separation for musical audio recordings: An overview[END_REF], [START_REF] Magoarou | Text-informed audio source separation. example-based approach using non-negative matrix partial co-factorization[END_REF] utilizes prior information about the sources to guide the separation process, and was shown to be successful in many contexts using different types of prior information. For instance, such information may include musical scores of the corresponding music sources [START_REF] Ewert | Score-informed source separation for musical audio recordings: An overview[END_REF], [START_REF] Ganseman | Source separation by score synthesis[END_REF], [START_REF] Fritsch | Score informed audio source separation using constrained nonnegative matrix factorization and score synthesis[END_REF] or text of the corresponding speech sources [START_REF] Magoarou | Text-informed audio source separation. example-based approach using non-negative matrix partial co-factorization[END_REF]. In some approaches this symbolic information is then converted to audio using a MIDI synthesizer for musical scores [START_REF] Ganseman | Source separation by score synthesis[END_REF], [START_REF] Fritsch | Score informed audio source separation using constrained nonnegative matrix factorization and score synthesis[END_REF] or a speech synthesizer for text [START_REF] Magoarou | Text-informed audio source separation. example-based approach using non-negative matrix partial co-factorization[END_REF]. These synthesized signals (that may also include cover tracks as in [START_REF] Souviraá-Labastie | Music separation guided by cover tracks: designing the joint nmf model[END_REF]) called deformed references in [START_REF] Souviraá-Labastie | Multichannel audio source separation using multiple deformed references[END_REF] can be used to roughly learn the spectral and temporal characteristics of one or more sources in the mixtures so as to guide the separation process [START_REF] Magoarou | Text-informed audio source separation. example-based approach using non-negative matrix partial co-factorization[END_REF]- [START_REF] Fritsch | Score informed audio source separation using constrained nonnegative matrix factorization and score synthesis[END_REF], [START_REF] Souviraá-Labastie | Multichannel audio source separation using multiple deformed references[END_REF]. A subclass of informed source separation approaches is user-guided separation methods where the prior information is provided by a user. Such information can be e.g., user-"hummed" sounds that mimic the sources in the mixture [START_REF] Smaragdis | Separation by humming: User-guided sound extraction from monophonic mixtures[END_REF] or source activity annotation along time [START_REF] Duong | Temporal annotationbased audio source separation using weighted nonnegative matrix factorization[END_REF] or in a time-frequency plane [START_REF] Lefèvre | Semi-supervised NMF with timefrequency annotations for single-channel source separation[END_REF]; the annotation information is then used, instead of training data, to guide the separation process. Furthermore, recent publications disclose an interactive strategy [START_REF] Bryan | Interactive refinement of supervised and semi-supervised sound source separation estimates[END_REF], [START_REF] Duong | An interactive audio source separation framework based on nonnegative matrix factorization[END_REF] where the user can perform annotations on the spectrogram of intermediate separation results to gradually correct the remaining errors. Note however that most of the existing approaches need to use prior information which may not be easy to acquire in advance (e.g., musical score, text transcript), is difficult to produce (e.g., user-hummed examples), or simply requires very experienced users while being very time consuming (e.g., time-frequency annotations).

The main motivation of this work is to introduce a simple framework that enables everyone to easily perform source separation. We hence present the new concept of on-the-fly source separation inspired by on-the-fly visual search methods [START_REF] Parkhi | On-the-fly specific person retrieval[END_REF], [START_REF] Chatfield | Visor: Towards on-the-fly large-scale object category retrieval[END_REF] from the computer vision research community. More specifically, the proposed framework only requires the user to listen to the mixture and type some keywords that describe the sources to be separated; in other words, the user interaction is now carried out at a higher semantic level. For instance, a user would request to separate the "wind noise" (source 1 description) from the "bird song" (source 2 description). The given descriptions or keywords are then used to search the internet for similar audio examples that will be employed to govern the separation process. For this purpose, supervised approaches based on e.g., nonnegative matrix factorization (NMF) [START_REF] Févotte | Non-negative matrix factorization with the Itakura-Saito divergence. with application to music analysis[END_REF], [START_REF] Lefèvre | Itakura-Saito non-negative matrix factorization with group sparsity[END_REF] or its probabilistic formulation known as Probabilistic Latent Component Analysis (PLCA) [START_REF] Smaragdis | Separation by humming: User-guided sound extraction from monophonic mixtures[END_REF], [START_REF] Smaragdis | Supervised and semisupervised separation of sounds from single-channel mixtures[END_REF],

where retrieved examples can be used to pre-learn the spectral dictionaries of the corresponding sources, are of great interest. Other methods in the prior art that couple the decomposition of the reference signals together with the mixture could also be considered [START_REF] Magoarou | Text-informed audio source separation. example-based approach using non-negative matrix partial co-factorization[END_REF], [START_REF] Souviraá-Labastie | Music separation guided by cover tracks: designing the joint nmf model[END_REF], [START_REF] Souviraá-Labastie | Multichannel audio source separation using multiple deformed references[END_REF], [START_REF] Gerber | Audio source separation using multiple deformed references[END_REF]. Regardless of the approach, several challenges, as detailed in Section II, arise in this on-the-fly framework due to (i) the unknown quality of the retrieved examples and (ii) possible lack of source descriptions (i.e. some sources may not be described by the user). In our preliminary work [START_REF] Badawy | On-the-fly audio source separation[END_REF], we investigated several strategies to handle issues with the quality of the retrieved examples and found that the one using a universal sound class model (USCM) 1 learned from examples via NMF with a group sparsity constraint is generally more efficient than the others. Note that since the USCM is actually an over-complete dictionary, a sparsity constraint is needed to help fit the most relevant spectral patterns to the sources in the mixture. This article extends our preliminary work [START_REF] Badawy | On-the-fly audio source separation[END_REF], [START_REF] Badawy | Relative group sparsity for non-negative matrix factorization with application to on-the-fly audio source separation[END_REF] by providing the algorithms along with their mathematical derivations in addition to new results from a user test. Altogether, the main contributions of our proposed on-the-fly paradigm work are four-fold:

• We introduce a general framework for on-the-fly audio source separation which greatly simplifies the user interaction.

• We propose a novel so called relative group sparsity constraint and show its benefit in the semi-supervised case

where training examples for some sources are missing. • We derive several algorithms for parameter estimation when different group sparsity-inducing penalties and relative group sparsity-inducing penalties are used. • We perform a range of evaluations, including both supervised and semi-supervised scenarios, and a user-test to validate the benefit of the proposed framework. The remainder of this paper is organized as follows. Section II gives an overview of the on-the-fly framework and the related challenges. In Section III, we recall some background on supervised source separation based on NMF. We then propose several algorithms for parameter estimation with the use of different sparsity-inducing constraints in Section IV. Evaluation results with a user-test are presented in Section V. Finally, we conclude in Section VI.

II. ON-THE-FLY FRAMEWORK AND CHALLENGES

A. Overview and challenges

The proposed framework only requires minimal user input enabling inexperienced users to apply source separation to essentially any mixture.It is applicable as well when relevant training examples for some sources are either not readily available offline or not representative enough, which is likely the case for uncommon sounds such as animal or environmental sounds. The general workflow is shown in Fig. 1. The user inputs a few keywords specifying the sources in the mixture (e.g., "dog barking", "wind", etc.,), then a search engine retrieves relevant source examples accordingly. The source spectral models are then learned on-the-fly and used for supervising the separation. This approach is actually analogous to on-the-fly methods in visual search where a user types a persons name (e.g., "Clint Eastwood") [START_REF] Parkhi | On-the-fly specific person retrieval[END_REF] or an objects description (e.g., "car") [START_REF] Chatfield | Visor: Towards on-the-fly large-scale object category retrieval[END_REF] and a classifier is trained using example images retrieved via Google Image Search.

Although the on-the-fly approach simplifies the user interaction and eliminates the need for offline training samples, there are several challenges that need to be addressed as follows:

(C the user describes only the sources of interest and ignores the remaining sources or when the search engines do not return results for some of the provided keywords. We refer to this challenge as the semisupervised case where all non-described sources that possibly appear in the mixture should be grouped as one background source.

In fact in our previous work [START_REF] Badawy | On-the-fly audio source separation[END_REF] to handle the first challenge, we investigated the use of a simple example preselection scheme based on the spectral similarity between the examples and the mixture to discard irrelevant examples. Thus, one can imagine having additional user interaction after specifying the keywords. For instance, the user may screen the list of retrieved examples and subjectively select a more relevant subset for training. This is the "Examples Refinement" step in Fig. 1.

B. Graphical User Interface

We implemented the system along with a graphical user interface (GUI) as shown in Fig. 2 and employed it for our user-tests. It features the ability to listen to a mixture and input one or more keywords describing the different sources. Then, per source, an online search for audio is performed. Next, the user can listen to the list of retrieved audio examples as well as view their waveforms or spectrograms (useful for the more advanced users). The optional example selection is then done by ticking the corresponding checkboxes. USCMs are then learned on-the-fly to guide the separation. The last step is to output the separated sources. A video showing a demo is available online at http://youtu.be/mBmJW7cy710/. On the practical side, the data transferred between the user and the server consists of the keywords and the mixture file as well as the separated sources which are sent back to the user. On the server, each example file requires computing the STFT followed by NMF; the examples are independent and these operations can thus be done in parallel. Then once the USCMs have all been constructed, the separation step is faster as the multiplicative updates are performed only one time. The overall complexity thus mostly depends on the number of training examples and the size of USCMs. Thus on an average PC, it would take from 30 seconds to a few minutes to get the separation results back.

III. BACKGROUND ON NMF-BASED SOURCE SEPARATION

A. Conventional supervised approach

We discuss in this section a standard supervised source separation approach. We base our framework on NMF since it is one of the most popular and well-suited models in the state of the art on audio source separation. As per e.g. [START_REF] Smaragdis | Supervised and semisupervised separation of sounds from single-channel mixtures[END_REF], [START_REF] Sun | Universal speech models for speaker independent single channel source separation[END_REF], first source spectral models are learned on-the-fly from training data retrieved online. Then these models are used to supervise the separation.

Assuming J sources, let X ∈ C F ×N and S j ∈ C F ×N be the STFT coefficients of the single channel mixture signal and the j-th source signal, respectively, where F is the number of frequency bins and N the number of time frames. Usual additive mixing is assumed as

X = J j=1 S j . (1)
Let V = |X| .2 be the power spectrogram of the mixture, where X .p is the matrix with entries [X] p il , .p denotes an element-wise operation. Then, NMF algorithms construct two non-negative matrices W ∈ R F ×K and H ∈ R K×N such that V ≈ V = WH. The factorization is usually done by solving the following optimization problem [START_REF] Févotte | Non-negative matrix factorization with the Itakura-Saito divergence. with application to music analysis[END_REF], [START_REF] Lee | Algorithms for non-negative matrix factorization[END_REF] W * , H * = arg min

H≥0,W≥0 D(V WH), (2)
where

D(V V) = F,N f,n=1 d(V f n Vfn ) (3)
and d(• •) is a scalar divergence measure. We use the Itakura-Saito (IS) divergence defined as

d IS (x y) = x y -log x y -1 (4)
which is appropriate for audio signals due to its scale invariance [START_REF] Févotte | Non-negative matrix factorization with the Itakura-Saito divergence. with application to music analysis[END_REF]. The parameters θ = {W, H} are usually initialized with random non-negative values and are iteratively updated via multiplicative update (MU) rules [START_REF] Févotte | Non-negative matrix factorization with the Itakura-Saito divergence. with application to music analysis[END_REF], [START_REF] Lee | Algorithms for non-negative matrix factorization[END_REF].

In the training step of the supervised setting, a spectral model for each source j, denoted by W (j) , is first learned from the corresponding training examples concatenated together by optimizing criterion [START_REF] Yılmaz | Blind separation of speech mixtures via time-frequency masking[END_REF]. Then the spectral model for all sources W is obtained by concatenating the source models as:

W = [W (1) , . . . , W (J) ]. (5)
Then in the separation step, the time activation matrix H is estimated via the MU rules optimizing (2) [START_REF] Févotte | Non-negative matrix factorization with the Itakura-Saito divergence. with application to music analysis[END_REF], while W is kept fixed. Note that the activation matrix is also partitioned into horizontal blocks as

H = [H T (1) , . . . , H T (J) ] T , (6)
where H (j) denotes the block characterizing the time activations for the j-th source.

Once the parameters θ = {W, H} are obtained, Wiener filtering is applied to compute the source STFT coefficients as

Ŝj = W (j) H (j) WH X, (7)
where denotes the element-wise Hadamard product and the division is also element-wise. Finally, the inverse STFT is computed to produce the time domain source estimates.

B. USCM-based approach

The conventional supervised approach as described in Section III-A assumes using all retrieved (or user-selected) examples for a given source to learn the source spectral model. This may not be suitable in the current framework due to the challenges mentioned in Section II where the noisy examples may lead to a poor spectral model. Thus, in this section we propose an efficient and flexible approach to better utilize the examples, when available, for guiding the separation, while also handling the case of missing examples. In the following, the training examples refer to either the full list of retrieved examples or the user-selected examples in case of user intervention. We employ a so-called universal sound class model, learned in advance from training examples, with sparsity constraints on the activation matrix H in order to enforce the selection of only few representative spectral patterns during the model fitting. In the following, we first present the USCM construction, and then the optimization criterion for model fitting.

1) USCM construction: Assuming that the j-th source is described by the user and some examples are retrieved for it, we denote by V jp the spectrogram of the p-th example corresponding to the j-th source. First, V jp is used to learn the NMF spectral model, denoted by W jp , by optimizing the criterion (similar to (2)):

H * jp , W * jp = arg min Hjp≥0,Wjp≥0 D(V jp W jp H jp ), (8)
where H jp is the corresponding time activation matrix. Given W jp for all examples, the USCM for the j-th source is constructed as

W (j) = [W j1 , . . . , W jPj ] (9)
where P j is the number of retrieved examples for the j-th source.

2) Model fitting for supervised source separation: In the supervised setting, we assume having source models for all the sources in the mixture, that is to say that for every source, the user gave its description and examples were successfully retrieved. It can be seen that the USCM W (j) constructed in (9) becomes a large matrix when the number of examples increases, and it is often redundant since different examples may share similar spectral patterns. Therefore, in the NMF decomposition of the mixture, the need for a sparsity constraint arises to fit only a subset of each W (j) to the source in the mixture. In other words, the mixture is decomposed in a supervised manner, given W constructed from W (j) as in [START_REF] Vincent | The Signal Separation Campaign (2007-2010): Achievements and remaining challenges[END_REF] and fixed, by solving the following optimization problem

H * = arg min H≥0 D(V WH) + Ψ(H) (10)
where Ψ(H) denotes a penalty function imposing sparsity on the activation matrix H. Different penalties can be chosen, as will be discussed in Section IV, resulting in a sparse matrix H as visualized in Fig. 3b and Fig. 3c.

3) Model fitting for semi-supervised source separation: We describe in this section a so-called semi-supervised setting where not all of the source models can be learned in advance [START_REF] Sun | Universal speech models for speaker independent single channel source separation[END_REF]. This occurs either when the user only describes the sources of interest and not all of them or when the search engine fails to retrieve examples for a given query.

We propose to model all the "missing" sources as one background source whose spectrogram can be approximately factorized as W b H b , where W b and H b are the corresponding spectral model and activation matrix, respectively. The parameter θ b = {W b , H b } can be randomly initialized with a small number of components (i.e. number of columns in W b ) K b . All the other sources, for which some examples are available, are modeled as in the supervised case by θ = {W, H} (see Fig. 4e and Fig. 4f). The parameters are estimated altogether by optimizing the following criterion [START_REF] Souviraá-Labastie | Music separation guided by cover tracks: designing the joint nmf model[END_REF] We see that in contrast to criterion (10) W b is updated as well and there is no group sparsity-inducing penalty on H b . The reason is that, as opposed to W, W b is neither an overcomplete dictionary nor has an underlying structure that can be exploited for regularization.

H * , W * b , H * b = arg min H≥0,W b ≥0,H b ≥0 D(V WH + W b H b ) + Ψ(H).

IV. SPARSITY CONSTRAINTS AND ALGORITHMS

In this section we consider two classes of sparsity constraints, namely group sparsity and a newly proposed relative group sparsity for the optimization problem [START_REF] Fritsch | Score informed audio source separation using constrained nonnegative matrix factorization and score synthesis[END_REF] and [START_REF] Souviraá-Labastie | Music separation guided by cover tracks: designing the joint nmf model[END_REF]. In each case, two variations are considered: a block sparsityinducing penalty and a component sparsity-inducing penalty. For every constraint, we give the corresponding algorithm for estimating the parameters.

A. Group sparsity constraints and parameter estimation algorithm

We consider a group sparsity-inducing penalty defined as

Ψ gr (H) = J j=1 λ j Gj g=1 log + H (j,g) 1 , (12)
where H (j,g) (g = 1, . . . , G j ) are the groups within the activation sub-matrix H (j) corresponding to the j-th USCM (see equation [START_REF] Ono | The 2015 signal separation evaluation campaign[END_REF] for the definition of H (j) ), G j the total number of groups for the j-th source, • 1 denotes the 1 matrix norm, > 0 and λ j ≥ 0 are trade-off parameters determining the contribution of the penalty for each source. Note that in the remainder of the paper, H (j,g) should not be confused with H jp in [START_REF] Magoarou | Text-informed audio source separation. example-based approach using non-negative matrix partial co-factorization[END_REF]. We introduce two options for defining the groups H (j,g) and derive the corresponding MU rules for the parameter estimation as follows. 1) Block sparsity-inducing penalty: As in [START_REF] Sun | Universal speech models for speaker independent single channel source separation[END_REF], we restrict the groups to be sub-matrices of H (j) corresponding to the spectral models W (j,p) trained using the p-th example (see [START_REF] Magoarou | Text-informed audio source separation. example-based approach using non-negative matrix partial co-factorization[END_REF] for the definition of W (j,p) ). In that case the indices g and p coincide and G j = P j . This so-called block sparsity-inducing strategy allows filtering out irrelevant spectral models W (j,l) , thus dealing with irrelevant retrieved examples (challenge C 1 in Section II). An illustration for the estimated activation matrix H for that case is shown in Fig. 3b where blocks corresponding to irrelevant examples for each source are set to zero.

2) Component sparsity-inducing penalty: As an alternative solution to fitting the universal model, we restrict the groups to be rows of H (j) corresponding to different spectral components (in that case the number of groups G j simply equals to the number of rows in H (j) ). This so-called component sparsity-inducing strategy allows filtering out irrelevant spectral components, thus dealing with noisy retrieved examples (challenge C 2 in Section II). Fig. 3c shows an estimated activation matrix H where rows corresponding to irrelevant spectral components for each source are set to zero.

3) MU rules for parameter estimation: MU rules for the optimization of criterion (10) (supervised case) and (11) (semisupervised case) are summarized in Algorithms 1 and 2, respectively, where η > 0 is a constant parameter, P (j,g) is a matrix of the same size as H (j,g) whose entries have the same value, and P is a matrix concatenating all P (j,g) . This algorithm is almost identical to the one proposed in [START_REF] Lefèvre | Itakura-Saito non-negative matrix factorization with group sparsity[END_REF], except that the groups are defined differently and W is not updated. It is proven in [START_REF] Lefèvre | Itakura-Saito non-negative matrix factorization with group sparsity[END_REF] using a majorizationminimization [START_REF] Hunter | A tutorial on MM algorithms[END_REF] formulation that these updates with η = 1/2 are monotonic, i.e., the cost function is non-increasing after each iteration.

Algorithm 1 MU rules for NMF with group sparsity in supervised case Input: V, W, λ Output: H Initialize H randomly V = WH repeat for j = 1, . . . , J, g = 1, . . . , G j do P (j,g) ← λj + H (j,g) 1 end for P = [P T

(1,1) , . . . , P T (1,G1) , . . . , P T (J,1) , . . . , (1,1) , . . . , P T (1,G1) , . . . , P T (J,1) , . . . ,

P T (J,G J ) ] T H ← H W T (V V.-2 ) W T V.-1 +P .η V ← WH until convergence
P T (J,G J ) ] T H ← H W T (V V.-2 ) W T ( V.-1 )+P .η H b ← H b W T b (V V.-2 ) W T b V.-1 .η W b ← W b (V V.-2 )H T b V.-1 H T b .η
Normalize W b and H b component-wise (see, e.g., [START_REF] Févotte | Non-negative matrix factorization with the Itakura-Saito divergence. with application to music analysis[END_REF])

V ← WH + W b H b until convergence
Note that the updates of H are identical in both the supervised and semi-supervised cases. Additionally, in the semisupervised case, since the derivations of [START_REF] Souviraá-Labastie | Music separation guided by cover tracks: designing the joint nmf model[END_REF] with respect to W b and H b are not affected by the sparsity constraint Ψ(H), the updates of W b and H b are straightforwardly derived as in [START_REF] Févotte | Algorithms for nonnegative matrix factorization with the beta-divergence[END_REF].

B. Relative group sparsity constraints and parameter estimation algorithm

For the separation to be feasible, we require that every learned source model has a corresponding non-zero activation; however, this constraint is not enforced by the group sparsity penalty in [START_REF] Souviraá-Labastie | Multichannel audio source separation using multiple deformed references[END_REF] where it can happen that a group of different sources are fit together using the same source model, instead of separately using their designated models, rendering their separation impossible. We observed this "source vanishing" phenomenon in practice as illustrated in Fig. 4a (in case of

H (1,1)

Group sparsity

Relative group sparsity

(a) (c) (e) (b) (d)
(f)

H (1,2)
H (2,1)

H (2,2)
H

H

H (1,1) using the block sparsity-inducing penalty) and Fig. 4c (in case of using the component sparsity-inducing penalty) in the supervised case. Moreover, the problem worsens in the semisupervised case, depicted in Fig. 4e for the block sparsity case, where the entire mixture is fit by the estimated background model only (the same effect occurs for the component sparsity case). This is due to the fact that W b and H b are now fully unconstrained in [START_REF] Souviraá-Labastie | Music separation guided by cover tracks: designing the joint nmf model[END_REF], whereas W is fixed and H is constrained by the group sparsity-inducing penalty. It can also be seen that increasing the trade-off parameters λ j in the penalty [START_REF] Souviraá-Labastie | Multichannel audio source separation using multiple deformed references[END_REF] (thus decreasing the number of active groups) increases the chances of source vanishing in both the supervised and semisupervised cases. In this section, we present a novel sparsityinducing penalty which helps prevent this problem completely. 1) Relative group sparsity-inducing penalties: This observation motivates us to introduce a general solution based on a new notion, namely relative group sparsity. While we present it here within the context of NMF, the idea extends to other dictionary decomposition schemes. We assume that the groups are organized into so-called supergroups (i.e. H (j) corresponding to a USCM is considered as a supergroup), and we characterize the relative group sparsity constraint Ψ(H) by the following properties

H (1,2) H (1,3) H b
• It induces the sparsity of the groups (as in group sparsity), and at the same time • It induces the anti-sparsity of the supergroups (i.e. prevents them from vanishing entirely). In other words, the group sparsity property is now considered relative to the corresponding supergroup H (j) and not within the full set of coefficients in H. It is formulated as [START_REF] Badawy | Relative group sparsity for non-negative matrix factorization with application to on-the-fly audio source separation[END_REF]

Ψ rel (H) = J j=1 λ j Gj g=1 log + H (j,g) 1 H (j) γj 1 , (13)
where γ j are some non-negative constants. Penalty ( 13) can be also rewritten as

Ψ rel (H) = Ψ gr (H) - J j=1 λ j γ j G j log H (j) 1 , (14)
and one can easily see that, while the new penalty keeps the group sparsity property thanks to Ψ gr (H) defined in [START_REF] Souviraá-Labastie | Multichannel audio source separation using multiple deformed references[END_REF], it prevents (when γ j > 0) the supergroups from vanishing since if H (j) 1 tends to zero, then -log H (j) 1 tends to +∞. This formulation generalizes the group sparsity constraint in the sense that (13) reduces to [START_REF] Souviraá-Labastie | Multichannel audio source separation using multiple deformed references[END_REF] for γ j = 0. So while we only require that γ j is non-zero for the relative group sparsity to be active, in our experiments we show results for γ j = 1 and γ j = 1 Gj . The latter was chosen to act as a normalization such that the effect of the penalty is even across the USCMs regardless of their size.

Note also that one can introduce either the relative block sparsity-inducing penalty or the relative component sparsityinducing penalty by defining a group H (j,g) to be either a block or a row in H similar to what has been presented in Section IV-A.

2) MU rules for parameter estimation: MU rules for parameter estimation when using the new penalty Ψ rel (H) are derived in the same way as the rules for group sparsity in Section IV-A. The resulting algorithms for both supervised and semi-supervised cases are summarized in Algorithm 3 and Algorithm 4, respectively. Details on the derivation of Algorithm 3 are given in the Appendix, and that of Algorithm 4 is very similar. Note that P (j,g) and Q (j,g) are matrices of the same size as H (j,g) whose entries have the same value, and P and Q are concatenations of P (j,g) and Q (j,g) , respectively.

V. EXPERIMENTS

We start by describing the data set, parameter settings, and evaluation metrics in Section V-A. We then evaluate the performance of the proposed supervised and semi-supervised on-the-fly audio source separation algorithms in Section V-B. The sensitivity of the different algorithms with respect to the choice of the trade-off parameter λ j which determines the contribution of the sparsity penalty is presented in Section V-C. We finally present user-test results in Section V-D.

A. Data, parameter settings, and evaluation metrics

We evaluated the performance of the proposed on-the-fly algorithms on a data set of 15 single-channel mixtures of Algorithm 3 MU rules for NMF with relative group sparsity in the supervised case Input: V, W, λ Output: H Initialize H randomly V = WH repeat for j = 1, . . . , J, g = 1, . . . , G j do P (j,g) ← λj + H (j,g) 1

Q (j,g) ← λj Gj γj H (j) 1
end for P = [P T

(1,1) , . . . , P T (1,G1) , . . . , P T (J,1) , . . . ,

P T (J,G J ) ] T Q = [Q T (1,1) , . . . , Q T (1,G1) , . . . , Q T (J,1) , . . . , Q T (J,G J ) ] T H ← H W T (V V.-2 )+Q W T ( V.-1 )+P .η V ← WH until convergence
Algorithm 4 MU rules for NMF with relative group sparsity in the semi-supervised case Input: V, W, λ Output: H Initialize H, H b , and

W b randomly V ← WH + W b H b repeat for j = 1, . . . , J, g = 1, . . . , G j do P (j,g) ← λj + H (j,g) 1 Q (j,g) ← λj Gj γj H (j) 1
end for P = [P T

(1,1) , . . . , P T (1,G1) , . . . , P T (J,1) , . . . ,

P T (J,G J ) ] T Q = [Q T (1,1) , . . . , Q T (1,G1) , . . . , Q T (J,1) , . . . , Q T (J,G J ) ] T H ← H W T (V V.-2 )+Q W T ( V.-1 )+P .η H b ← H b W T b (V V.-2 ) W T b V.-1 .η W b ← W b (V V.-2 )H T b V.-1 H T b .η
Normalize W b and H b component-wise (see, e.g., [START_REF] Févotte | Non-negative matrix factorization with the Itakura-Saito divergence. with application to music analysis[END_REF])

V ← WH + W b H b until convergence
two sources artificially mixed at 0 dB signal to noise ratio (SNR). Note that during the mixing, we made sure that two sources had more or less the same duration so that in all the mixtures both sources appear most of the time. The mixtures were sampled at either 16000 Hz or 11025 Hz and their duration varies between 1 and 13 seconds. The sources in the mixtures were selected as follows: (female speech, traffic), (female speech, cafe), (male speech, bells), (male speech, car), (woman singing, restaurant), (drums, guitar), (applause, electric guitar), (piano, ringtone), (violin, cough), (bat, owl), (chirps, rooster), (chirps, river), (siren, dog), (cat, dog), and (ocean, cricket). The speech samples (female speech, male speech) were obtained from the "American English" ITU-T P.5012 dataset. The following sources cafe, car, and restaurant were obtained from DEMAND3 from one channel out of the 16 channels. The music instruments (drums, electric guitar, guitar, piano, violin, woman singing) were obtained from QUASI 4 . The remainder were from various websites, mostly www.grsites.com/archive/sounds/ (bells, cat, chirps, dog, rooster, river, traffic), but also www.sounddogs.com (bat, owl) and www.wavlist.com (cricket), among others. The diversity in the types of sources should demonstrate the advantage of the proposed on-the-fly strategy since, as opposed to speech where pre-trained models are fairly common, having a pretrained model for every possible sound class is not viable. In the implementation of the framework, sound examples for training were retrieved from www.findsounds.com, a search engine for audio, as well as from www.freesound.org, a database of user-uploaded sounds. Note that these two websites are different from the ones used to get the sources in the mixtures; thus the possibility that the training set contains a source from the mixtures is very small. The retrieved files were restricted to those with sampling rates at least as high as that of the mixture, and the ones with higher sampling rates were down-sampled accordingly. For retrieval in our experiments, we differentiate between two types of search keywords: i) reference keywords given by an expert (the first author) who prepared the dataset and thus had also listened to the separate sources and not only the mixtures and ii) user keywords given by non-expert users in our user test. It is important to note that the reference keywords are not the only "correct" keywords since other synonyms can be used. Table I lists the reference keywords and the corresponding user keywords along with the number of times a keyword was given by the users. Note that some reference keywords like "male speech" or "female speech" are repeated in more than one mixture, thus the count of their corresponding user keywords is more than the number of users.

Other parameters were set as follows. The STFT was calculated using a sine window and a frame length of 47 ms with 50% overlap, the number of iterations for MU updates was 200 for learning the USCM W (j) and 100 for separating the mixture, and the number of NMF components for each spectral model learned from one example W (j,p) was set to 32. In the semi-supervised case, the number of NMF components for the background source was K b = 10 to avoid overfitting since W b is unconstrained. Additionally, since the number of training examples P j per source was different depending on the availability of the data (search results), the trade-off parameter λ j determining the contribution of the sparsityinducing penalty was set to λ 0 F N P j (where λ 0 is a constant) so that λ j is greater when more examples are available. The intuition here is that the smaller the USCM W (j) is, which happens when few examples are available, the lower the level of sparsity that should be imposed in the decomposition. We found these settings to generally result in a good separation performance.

woman singing brasilian woman singing, brazilian song, girl singing (3), singing woman, woman singing (

The source separation performance was evaluated in terms of the normalized signal-to-distortion ratio (NSDR), which measures the overall signal distortion, and the normalized signal-to-interference ratio (NSIR) which measures the leakage of the other sources [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF], [START_REF] Ozerov | One microphone singing voice separation using source-adapted models[END_REF]. Recall that the normalized values are computed by subtracting the SDR and SIR of the original mixture signal from those of the separated sources [START_REF] Ozerov | One microphone singing voice separation using source-adapted models[END_REF]. The normalization serves to show the gain of using the proposed source separation system as opposed to a naive method that simply assigns the mixture as a source estimate. These metrics are measured in dB and are averaged over all sources and all mixtures for the different algorithms.

B. Separation results using reference keywords

In this experiment, we use the reference keywords given by the expert for retrieval. The goal is to evaluate and compare the performance of the different algorithms. For the supervised case (i.e., Algorithm 1 and Algorithm 3), two keywords were used to retrieve examples for both sources in the mixture, while only one keyword was used for the semi-supervised case (i.e., Algorithm 2 and Algorithm 4). Note that, in the semi-supervised scenario, we tested two cases as follows (i) a keyword was provided for source 1 only and (ii) a keyword was provided for source 2 only; we then averaged the obtained separation results.

We compare the average separation performance obtained using the four sparsity-inducing penalties presented in the paper: block sparsity as the baseline [START_REF] Sun | Universal speech models for speaker independent single channel source separation[END_REF], the proposed component sparsity, relative block sparsity, and relative component sparsity. Results for the supervised case (i.e., Algorithm 1 for block sparsity and component sparsity, and Algorithm 3 for relative block sparsity, and relative component sparsity) are shown in Table II, while those for the semi-supervised case (i.e., Algorithm 2 and Algorithm 4) are shown in Table III. In each case, we run the algorithms with different values of the trade-off parameter λ 0 , and the value resulting in the highest average NDSR is chosen and shown in the tables along with the corresponding NSDR and NSIR. Note that the result shown in Table II is 1.8 dB NDSR higher than that reported in our previous work [START_REF] Badawy | On-the-fly audio source separation[END_REF]. The reason is that: (1) the dataset (training and testing set) is enlarged by the size and the variation of the sound sources; and (2) the parameter λ j is here adapted per mixture and not constant for the whole dataset. Also, for easier reading here, we do not compare again the separation performance with the standard supervised NMF setting without using USCM model nor with some other baselines as it has been investigated in our previous study [START_REF] Badawy | On-the-fly audio source separation[END_REF]. For the relative sparsity cases, we tested two values for the hyper-parameter γ j : a fixed γ j = 1 as a natural choice, and γ j = 1 Gj such that the denominator term H (j) γj 1 in the penalty ( 13) is adaptively normalized with respect to the size of the group G j .

First, as expected, the results obtained in the supervised case are much better than those achieved in the semi-supervised case where examples for one source are missing. Second, using an adaptive γ j = 1 Gj in the supervised case improved the NSDR for the relative block sparsity by 0.25 dB but had no significant effect on the relative component sparsity; in contrast it negatively affected the performance in the semisupervised case. Third, we note that the proposed component sparsity-inducing penalty achieves a better separation performance than the block sparsity-inducing penalty which was exploited in [START_REF] Sun | Universal speech models for speaker independent single channel source separation[END_REF], in both supervised and semi-supervised cases. A possible explanation is that the former offers more flexibility by exploiting the most representative spectral patterns from different spectral models that match the mixture. Last, it is worth noting that the proposed relative component sparsityinducing penalty performs the best in both supervised and semi-supervised cases in terms of both NSDR and NSIR, the advantage being more significant in the semi-supervised case likely because the source vanishing problem is more severe. We note that the corresponding average signal-to-artifact ratio (SAR) for the different algorithms was on the order of 11 dB. In particular, the SAR corresponding to the relative component sparsity-inducing penalty was 10.98 dB and 11.35 dB for the adaptive γ j .

In general, the methods would fail if the retrieved examples are quite dissimilar from the actual sources in the mixture. As an example, a mixture of of an electric guitar and applause (cheers and whistles) had low NSDR for both source estimates (0.54 dB and -0.49 dB respectively). In this case, we observed that the retrieved training files for the applause contained mostly just clapping sounds and as such the learned USCM did not capture the cheers; similarly for the guitar where most retrieved examples were not close to the chords in the mixture.

C. Separation results with different choices of λ j

One of the most important parameters in the presented algorithms in the on-the-fly framework is the trade-off parameter λ j determining the contribution of the sparsity-inducing penalty. We propose to set λ j = λ 0 F N P j so that it is normalized with respect to the size of the USCM and is larger when more examples are used. In this experiment, we varied λ 0 and assessed the sensitivity of the different algorithms described in Section V-B to this choice in the semi-supervised scenario. The dataset and other parameter settings are the same as described before. The results are shown in Fig. 5 where λ 0 = {10 -5 , 5 × 10 -5 , 10 -4 , 5 × 10 -4 , 10 -3 } for the block/relative block sparsity algorithms and λ 0 = {10 -7 , 5 × 10 -7 , 10 -6 , 5 × 10 -6 , 10 -5 } for the component/relative component sparsity algorithms. Note that the range of λ 0 is different for the (relative) block and component sparsity algorithms as they are different types of penalties so their optimal range is different.

As can be seen, the relative block sparsity and relative component sparsity algorithms are generally more stable than the block sparsity and component sparsity algorithms over a large range of λ 0 where the results obtained by the former algorithms drop sharply for the last point. Within a good range, i.e. the first four points, the relative block sparsity with γ j = 1 Gj is the most stable one as its NSDR varies at most 0.2 dB. The relative component sparsity algorithm, which offers the highest performance in general, is not very sensitive to the considered parameter though it has more than 1 dB NSDR difference within the considered range.

D. Separation results for the user test

In the second experiment, our goal was to evaluate the performance of the proposed on-the-fly framework when practically used by non-expert users. We also test the effect of the examples refinement step on learning the USCM. The algorithms based on the proposed relative block/component sparsity-inducing penalties, which perform better than those using the block/component sparsity-inducing penalties as shown in Section V-B, were tested using the input from 9 different users who were of different age groups, technical backgrounds, and were all not native English speakers. The best parameter settings as determined from Section V-B were used. Using the GUI described in Section II-B, the users were asked to process each of the 15 mixtures as follows. First, they were asked to listen to the mixture and accordingly type keywords describing the two sources. They were instructed to change the keywords in case the search engine did not return results. Then, they were required to listen to the retrieved examples and select those that sound more similar to the sources in the mixture; at least one example was required to be selected. Given the recorded user input (keywords and selected examples), we examine two possibilities of using the examples in guiding the separation process as follows:

• All retrieved examples are used (All).

• Only the subset of examples selected by the user is used (Subset). The source separation performance, averaged over all 9 users and over all mixtures per method, is shown in Table IV and Table V for the supervised and semi-supervised cases, respectively. We note that the results for the average user are mostly lower than those for the expert in the supervised case due to the following issues. As can be seen from the keywords in Table I, some sounds like bat and owl were sometimes not recognized by the users and were confused with other sounds (e.g., bird sounds). Also, some spelling mistakes can be found (e.g, caugh instead of cough). This may have negatively affected the results. Additionally, one of the mixtures included a popular ringtone composed of marimba notes; however, the retrieved examples mainly included classical telephone rings, perhaps "marimba" would have been a better choice for searching. In the semi-supervised case, the expert results are not better than the average user. The reason is likely that the guidance is reduced in this setting and the overall performance is quite lower compared to the supervised case.

Nevertheless, the performance globally follows the same trend as presented in Section V-B: relative component sparsity generally outperforms relative block sparsity, especially in the semi-supervised case, with the SAR on the order of 9 dB. It is interesting to observe the effect of selecting a subset of examples. As can be seen in Table IV, using a subset of examples selected by the users only improves the performance in the supervised case. However, in the semi-supervised case, such a pre-selection even negatively affects the results as can be observed in Table V. This is likely due to the fact that having few selected examples (only one in the extreme case) leads to having fewer components in the learned spectral model for which a sparse decomposition is not optimal. Thus, it seems to be better to keep all retrieved examples for the known source and let the relative component sparsity penalty induce the appropriate selection.

VI. CONCLUSION

In this paper, we presented the novel concept of on-thefly audio source separation and described several algorithms to implement it. Specifically, we proposed using a universal sound class model learned by NMF from retrieved examples and imposing group sparsity-inducing constraints to efficiently handle the selection of the most representative spectral patterns. Additionally, we introduced the notion of relative group sparsity to overcome a so-called source vanishing problem that occurs in the considered on-the-fly paradigm. In contrast to other state-of-the-art user-guided approaches, the considered framework greatly simplifies the user interaction with the system such that everyone, not necessarily an expert, can do source separation by just typing keywords describing the audio sources in the mixture. Experiments on mixtures containing various types of sounds confirm the potential of the proposed framework as well as the corresponding algorithms. Future work may be devoted to running real-world experiments, studying the use of a different group sparsity model that induces dynamic relationships between atoms or groups [START_REF] Hurmalainen | Similarity induced group sparsity for non-negative matrix factorisation[END_REF], as well as extending the framework to multichannel mixtures where spatial source models (e.g. those from [START_REF] Duong | Under-determined reverberant audio source separation using a full-rank spatial covariance model[END_REF] or [START_REF] Nikunen | Direction of arrival based spatial covariance model for blind sound source separation[END_REF]) may also be learned. Additionally, investigating the optimal USCM model size for different types of sound sources would be an interesting direction.

APPENDIX DERIVATION OF MU RULES IN ALGORITHM 3

Let C(H) denote the right part of criterion [START_REF] Fritsch | Score informed audio source separation using constrained nonnegative matrix factorization and score synthesis[END_REF] with relative group sparsity penalty Ψ(H) = Ψ rel (H) defined as in [START_REF] Smaragdis | Separation by humming: User-guided sound extraction from monophonic mixtures[END_REF] and D(• •) being IS divergence specified as in equations ( 3) and (4). The partial derivative of C(H) with respect to h kn writes

∇ h kn C(H) = F f =1 w f k 1 [WH] f n - v f n [WH] 2 f n + λ j + H (j,g) 1 - λ j G j γ j H (j) 1 (15)
Following a standard approach for MU rules derivation (see e.g., [START_REF] Févotte | Non-negative matrix factorization with the Itakura-Saito divergence. with application to music analysis[END_REF], [START_REF] Lee | Algorithms for non-negative matrix factorization[END_REF]), we represent ∇ h kn C(H) as

∇ h kn C(H) = ∇ + h kn C(H) -∇ - h kn C(H) (16)
with ∇ + h kn C(H), ∇ - h kn C(H) ≥ 0 defined as

∇ + h kn C(H) F f =1 w f k 1 [WH] f n + λ j + H (j,g) 1 , (17)
∇ - h kn C(H) F f =1 w f k v f n [WH] 2 f n + λ j G j γ j H (j) 1 , (18)
and we update each parameter h kn as

h kn ← h kn ∇ - h kn C(H) ∇ + h kn C(H) .η , (19)
where η = 0.5 following the derivation in [START_REF] Lefèvre | Itakura-Saito non-negative matrix factorization with group sparsity[END_REF]. Rewritten in a matrix form, we obtain the updates of the activation matrix H in Algorithm 3.

Fig. 1 .Fig. 2 .

 12 Fig. 1. General workflow of the proposed on-the-fly framework. A user listens to the mixture and types some keywords describing the sources. These keywords are then used to retrieve examples to learn spectral models for the described sources. Optionally, the user may listen to the retrieved examples and discard irrelevant ones.

Time

Fig. 3 .

 3 Fig. 3. Estimated activation matrix H for two sources in a mixture containing a rooster and bird chirps where two retrieved examples for each source were used for training the USCMs: (a) without a sparsity constraint, (b) with a block sparsity-inducing penalty (blocks corresponding to poorly fitting models are zero), and (c) with a component sparsity-inducing penalty (rows corresponding to poorly fitting spectral components from different models are zero).

Algorithm 2

 2 MU rules for NMF with group sparsity in semisupervised case Input: V, W, λ Output: H, H b , W b Initialize H, H b , and W b randomly V = WH + W b H b repeat for j = 1, . . . , J, g = 1, . . . , G j do P (j,g) ← λj + H (j,g) 1 end for P = [P T

Fig. 4 .

 4 Fig. 4. Examples of estimated activation matrices H for two sources in a mixture containing a rooster and bird chirps where two retrieved examples for each source were used for training the USCMs. Left column: (a) block sparsity in the supervised case, (c) component sparsity in the supervised case, and (e) block sparsity in the semi-supervised case. Right column: same settings as in the left column, but for the proposed relative block/component sparsity.

Fig. 5 .

 5 Fig. 5. Separation performance of the different algorithms, in terms of NSDR (a) and NSIR (b), as a function of 0 .

TABLE I REFERENCE

 I KEYWORDS AND THE CORRESPONDING USER KEYWORDS.

	Reference keywords	User keywords
	applause	background noise, cheers, concert, concert crowd, crowd, crowd cheering (2),
		crowd concert, people cheering
	bat	bird (2), bird cackling, bird chirping, birds sound, monkey (2), jungle, night animal
	bells	bell tone, bells, bells church, church bell (3), church bells (3)
	cafe	chattering, crowd, crowd speech, crowd talking (2), many people talking, party, people, people talking
	car	aeroplane noise, ambient noise, boat motor, calm noise, car, drive, nothing, thunder storm, wind
	cat	cat (7), cat meow, cat meowing
	chirps	bird (4), bird chirping (6), birds (2), birds chirping (2), birds sing (2),
		night creatures, sparrows
	cough	caugh, cough (2), coughing (5), man caughing
	cricket	bird, birds, birds sing, cricket (2), night, night animal, night creatures, tweet-tweet
	dog	dog (5), dog bark, dog barking (2), dogs
	drums	bass drum, drum (3), drum beats, drums, percussion, rythmic, tap beats
	electric guitar	electric guitar, guitar (2), guitar concert, music (3), music playing, riff guitar
	female speech	female speech (3), female voice (2), female voice english, girl read, girl talking (5), woman, woman read,
		woman speak, woman speaking, woman speech, woman talking (2), woman voice
	guitar	acoustic guitar, electronic organ, guitar (7)
	male speech	male english speech (3), male speech english, man reading, man speak, man speaking (2),
		man speech, man talking (4), man voice, men speech, poetry recitation, read (2)
	ocean	car, driving a car, road traffic, sea waves, storm, street, traffic noice, waterfall, waves
	owl	dog, dog moaning, owl (4), owl hooting, pigeon, woodpecker
	piano	pianist, piano (4), piano music (3), soft piano strings
	restaurant	chattering ambiance, crowd (2), crowd noise with photo clicks, crowd speech, crowd talking,
		people noise, people talking (2)
	ringtone	jingle phone ringing, mobile ringtone (2), phone ringing, phone ringtone, ring,
		ringing, ringtone, smartphone ring
	river	motor engine noise, river, river flowing, sea, stream, water (2), water boiling, water flowing
	rooster	cock (2), cock cluck, cock-a-doodle-do, hen (2), rooster (3)
	siren	ambulance, police, police car, police siren (5), siren
	traffic	car, car passing, car running, road traffic, road with cars, street traffic, traffic noice,
		traffic noise, traffic sound
	violin	cello strings orchestra, music album, music

TABLE II SUPERVISED

 II CASE: AVERAGE SOURCE SEPARATION PERFORMANCE.

	Method			NSDR	NSIR
	Block sparsity [25] (λ 0 = 1 × 10 -4 )	5.14	9.80
	Component sparsity (λ 0 = 1 × 10 -6 )	5.91	10.67
	Rel. block sparsity (γ j = 1, λ 0 = 1 × 10 -4 )	4.78	9.27
	Rel. component sparsity (γ j = 1, λ 0 = 1 × 10 -6 )	6.15	10.70
	Rel. block sparsity (γ j = 1 G j	, λ 0 = 1 × 10 -4 )	5.03	9.44
	Rel. component sparsity (γ j = 1 G j	, λ 0 = 1 × 10 -6 )	5.96	10.72

TABLE V USER

 V TEST IN THE SEMI-SUPERVISED CASE: AVERAGE SOURCE SEPARATION PERFORMANCE.

	Method	NSDR	NSIR
	Relative block sparsity (All)	1.88	7.50
	Relative block sparsity (Subset)	1.24	7.34
	Relative component sparsity (All)	2.78	8.04
	Relative component sparsity (Subset)	1.53	7.60

The term "universal speech model" was introduced in[START_REF] Sun | Universal speech models for speaker independent single channel source separation[END_REF] for the separation of speech and noise, and was inspired by the term "universal background model" used for speaker verification[START_REF] Reynolds | Speaker verification using adapted Gaussian mixture models[END_REF]. We here extend it to "universal sound class model", since our framework deals with the separation of sources belonging to any sound class.

http://www.itu.int/rec/T-REC-P.501

http://parole.loria.fr/DEMAND/

http://www.tsi.telecom-paristech.fr/aao/en/2012/03/12/quasi/

ACKNOWLEDGMENT

The authors would like to thank all the colleagues at Technicolor's Rennes research center who participated in the experiments.


Stefan Mehedinteanu
email: s.mehedinteanu@yahoo.com

The Quantization of the Gravitational Charges At the Origin of the Gravitational Waves From a Binary Black Hole Merger (LIGO)

The Micro-black-holes particles produced at the horizon entry by quantum fluctuation at Reheating stay at the base of the origin and evolution of Universe. These adjust theirs dimensions by photons radiation but kipping someone else, to become electrons, others leptons (quarks) and uncharged particles (neutrinos). During a merging process of two Black Holes (BH) it results a disruptive part as a new BH that it could be supposed to contain inside the quantized gravitational charges from and which in the following these were released as gravitons that deform the spacetime till the LIGO site. Since, the first observation of gravitational waves 1 February 2016, when the signal was named GW150914, and the event happened at the time m ly billion t 25 16 24 16 m s    after the Big Bang, known as Reionization and 21 centimeter radiation epoch, for that we have considered the two primordial BHs as the nuclei of such galaxies and not originated from stars collapsing. Only by using this model were obtained the parameters of gravitational wave which arrives at LIGO site.

Introduction

Recently [START_REF] Abbott | Observation of Gravitational Waves from a Binary Black Hole Merger[END_REF], it was the first direct detection of gravitational waves and the first observation of a frequency from 35 to 250 Hz with a peak gravitational-wave strain of . During the final 20 milliseconds of the merger, the power of the radiated gravitational waves peaked at about 3.6×10 49 watts -50 times greater than the combined power of all light radiated by all the stars in the observable universe.

In the following will be shown how that one obtain these parameters only if we adopt the gravity quantization in matter particles inside the black holes.

The situation before BH formation-the quantization of gravity in matter particles

The electron

The average magnitude of the electric field inside the model electron is as from [2a], [START_REF] Williamson | Is the electron a photon with toroidal topology?[END_REF]:

4 0 6   c E     , it results ] [ 10 7 . 5 17 C N E     , for electrons of ] [ 10 7 . 3 13 m c m e C       The graviton In [2a], [2b],
we have introduced the concept of primordial Micro-black-holes ( BH  ) which later decay into photons in P t as the Hawking radiation where, the gravitational field is so strong that it causes the spontaneous production (absorbtion) of photon pairs (with black body energy distribution) and even of particle pairs. Sure, only few free photons of photons dimension could remain embedded in ex-like-Micro-black-hole which adjusts its dimensions to electrons, other leptons, noncharged particles (neutrinos, Higgs) in order to equilibrate the gravitational charge P m G as it was explained following electron model ref. [START_REF] Williamson | Is the electron a photon with toroidal topology?[END_REF] . Very early in 1960 L. Motz ref. [3a] ; [3b] has elaborated a new theory of the structure of fundamental particles, which introduces gravitational field into the interior of particles such as electrons to account for their stability. Thus,

P m G
is the gravitational charge defined by L. Motz as resulting from:

2 2 1 r m G m G F G 
, where it can say that the gravitational charge

1 m G is the source of gravitational field 2 1 r m G
at the distance r , and that this field is coupled to the gravitational charge 2 m G at the position r (relative to the source of the field ) via the product of the field strength and the charge. L. Motz derived the quantization condition on gravitational charge in a similar manner by noting that moving particle with velocity V  and with gravitational charge

P m G
is coupled not only to the Newtonian gravitational fields of all other particles (the gravielectric field ) in the usual way, but also to the Coriolis inertial force of field

G c   2
by means of the cross product

    G c c v m G P    2 
, see below the possible Motz's derivation (my guess) , where   is an appropriate angular velocity. This cross product term give rice to an angular momentum ( v rm L P  ;

 r v  ) component in the motion of the particle which is of the order of

 2 2 r m L P 
and is parallel to the field   .

Therefore, it is obtained the quantization condition of spin 2 (graviton) as:

 2 2 2    r m L P .
Since it was interested in the quantization of fundamental gravitational charge, it is found a value for r and  that must be associated with such charge. To do this, L.Motz considered the Universe to two such charges in gravitational equilibrium and resolving about each other in the first Bohr orbit. The radius of this orbit is just

3 2 P Gm r   , c v  ; P m m m   2 1 .

Such a coupling of the charge to gravimagnetic field

P m G L is achieved, G r c v m G P 2 2 
, that becomes just the angular momentum which contains a double moment:

 2 2 r m L P  . This will be so only if c r v   
is of the order of c since the centrifugal force (being

charge times field) is just     G r m G 2 2 
. Introducing these two relationships into the above quantization equation, is obtained

 2 2   cr m L P
, or, substituting for r as Bohr orbit when the electron is held in a circular orbit by electrostatic attraction.

; G k e   4 4 1 0   ; P m m  ,  n mvr  , m e k r e 2 2   , m Gm m e k r e 2 2 2 2     ; when P m G e  ,    3 2 2 P P Gm c m , or   c Gm P 2
In the case of an electron, the centrifugal force of gravity charge is equal with the the Lorentz force of the magnetic flux as been induced by the solenoidal current of the apparent charge which arises from the electric field of the confined photons (as above):

condensate G F F    e E r Gm P    2 2 ; ] [ 10 7 . 5 17 C N E     ; Or, ] [ 091 . 0 15 . 0 N  .
If the square of quantum charge is, indeed c  , we can account for the stable electrically charged particles such as electrons by balancing the explosive positive electro-static energy (as due of the confined photons) with the negative binding energy of the gravitationally charge. It is clear that a gravitational charge of magnitude c  will contribute negative gravitational binding energy of the order of 2 mc if this charge is distributed over a region whose dimensions are equal to the Compton wave length of the particle.

The spacetime inflation induced by the two merging BH

In our case, the resulting BH is supposed to contain inside these gravitational charges as a heritage of the collapsing star matter where these were embedded during matter creation. In the following during the merging process of two black-holes these are released as gravitons, the expression of the gravitational wave G-wave (as in case of photons and the light). Therefore, it seems that only the merged part of the merged black hole is released as gravitons which wraps the space-time till today. of e -folds of slow-roll space-time inflation that occur after the epoch of horizon exit. Indeed, since

 dN k d  
as of eq.( 46) from [START_REF] Mehedinteanu | The Timeline of Universe Based On the Gravitons Release Due of the Merging of the Primordial Micro-black Holes[END_REF], and therefore ) ( ) ln(

 N k k end  , or, ] [m ke k N end 
where end k is the scale leaving the horizon at the end of slow-roll inflation, or usually ] [

1 1 m k k end    , the correct equation being ] [ 1   m e k k N end . When the wavelength ]) [ ( 1 m k  is large compared to the Hubble length ]) [ ( 1 m H 
, the distance that G-wave can travel in a Hubble time becomes small compared to the wavelength, and hence all motion is very slow and the pattern is essentially frozen in, but it will not be the case, as in the following. Since, the FLRW metric of the universe must be of the form ds is a three-dimensional metric that must be one of (a) flat space, (b) a sphere of constant positive curvature or (c) a hyperbolic space with constant negative curvature, or for small commoving time

aHc dt 1 
, we can consider the distance as

end a a ds L    .
Here, the Hubble constant is defined as a we obtain for outside the object (BH, Universe, planets, stars etc.)

3 3 8 2 2 c G a a            
] [ ] [ 2 2 3 4 3 4 3 2 3 2 2 2 2           m n a c G a c GE a c G a GM s H a a g g   ; (1)
, where:

g g E n   ;   g P g merging at n n     ;   3 4 3 a M  ; 2 Mc E  ; BH BH m n M    ; 1   H R .
Or, for inside of these objects: to match the iterations cycle: , all correspond to the situation at cosmic microwave background radiation (CMBR) time [2b]. Since the all-sky map (pattern) of the CMB contains also voids, that represents in fact the radiation emitted during BHs  merging (clustering) into further galaxies nuclei, and the rest is due of matter as a thermal radiation generated at Recombination of the photon decoupling.

] [ 3 4 3 4 ] [ 3 4 2 2 2 3 2 2 3 3 2 2        m c l n M G c G s a a G a a C merging at U       (2)
N a H R T k m end end B g        1    . 3 10 9   T , ] [ 10 2 . 2 7 _ m BH C     , Hz k c end 8 1 10    

The formation of the disruptive BH due of the binary merging

The number of gravitons that has been released following BHs  merging into the primordial resulting disruptive BH of the BHs binary (here, we have considered these as the two primordial galaxy nuclei as in section 4.1, and not as resulting from some stars collapse as in previously version of this work!), as being only ~2% of the cumulative masses [START_REF] Abbott | Observation of Gravitational Waves from a Binary Black Hole Merger[END_REF] to match the iterations cycle: , with this value the eq. ( 1) gives the curvature radius R as being equally with Schwarzschild radius. The strain at LIGO site Now, based on eq. ( 1) we can derive for the G-wave effect in the deformation (strain) of the space-time between the disruptive BH and LIGO site by using the gravitational pressure due of gravity charges on the area of Schwarzschild radius Schw r of BH, respectively with the values calculated in this section, we have: given in [START_REF] Abbott | Observation of Gravitational Waves from a Binary Black Hole Merger[END_REF]. Therefore, the present numerically analysis proves the concept of gravity charges which escape from the BHs during disruptive merging, as the corpuscle part of the G-wave. Therefore, at LIGO is obtained not only the deformation of space-time, but also the graviton properties, the inner structure of BH as gravitational charge, and are verified the FLWR equations with these quantized masses. To mention that only the above datasets as being obtained by iterations its proves the model.

N a H R T k m end end B g        1    . K T 65  , ] [ 10 3 .

The gravitons release during binary

  waves. Also in[START_REF] Abbott | Observation of Gravitational Waves from a Binary Black Hole Merger[END_REF], the gravitational waves (GW) observations constrain the Compton wavelength of the graviton to be ]



  To estimate the horizon entry we use some derivations done in [2b]. In Inflation models [2a, 3b], the scale leaving the horizon at a given epoch is directly related to the number ) ( N

  , the Friedmann equations are equivalent to this pair of equations:

  available volume beingTo mention that only this data set match the model.4. The BH internal structure at the beginning of mergingThus, the field for a gravity charge at time of quantum fluctuation (before BH) is as in section 3. factor at the origin as it results from the inflation model [2a]. of the merged binary blackholes calculated by Numerical Relativity complex methods given in[START_REF] Abbott | Observation of Gravitational Waves from a Binary Black Hole Merger[END_REF]; the energy being J base of BH's nuclei of galaxies production An important contribution to further Universe expansion is given by the BH's nuclei of galaxies which result from another fraction ( 75 . 0 ) of the initially BHs  merging [2b].

  in both cases at (LIGO laser site) the strain is around near equally with the G-wave measured strain 21 10  

  H -the Hubble length is slowly varying, we

	have	d	ln	k		d	(ln(	aH	))		d	ln	a		a dt a 		Hdt	. From the definition Eq. (38) of [2b] this
	gives	ln				(	)

  During Universe evolution at Electroweak epoch or Reheating due of the quantum fluctuations [2a] a huge number of the micro-black holes as Planck particles P n are generated,

												BH  m				P m	; the graviton energy being at horizon leave ;
		g	a c end  		2	26  10 	J		; when	a	leave		k	leave	H	leave		1	,	M U		53 10 2  2 .	kg	;
	m c BH    C 	c	2		;	BH n 			1	H	3 1 		available V		n	P		60 10	, or
	n	P		E U		P		70 10	J		9 10	J	with		BH 		10 11	a ee end 		10 11	GeV		4 . 17	J	;	a	end	ee		92 . 0	,
	it results with	k	1 leave 		H	1 leave 		10 27 	] m [	by iteration for	N		16	.	2	; in eq. (2);
	k end		2 . 9	10 19 	[	m	1 	]		;			H	1  	10 20 	[	] m	;	t		3 . 3	29  10 	s	;	R		5 . 6		] 20 m [  10	;
	l	C		l	P		7 . 1	] 27 m [  10 	;	a	EW		92 . 0	; where	n	g		 	g P		n	P		96 10	as a "fix" number of
	gravitons escaped (an inverse process of a black-hole ) from micro-blacks holes created
	in Universe as the Planck particles during theirs totally decaying, and which deforms the
	spacetime.
	The number of gravitons that has been released following BHs 	merging is only
																															96
																								26		  17 . 4	26		7 . 5	68 10

BHs disruptive merging as a gravitational wave

  Therefore, the new horizon leave as gravitational wave (GW) is just the new disruptive BH escape at its hypothetical Schwarzschild radius:

																						a		k	H		1	, or
																						leave	leave	leave
	k	1 			1   H			r				7 . 8		10 3	] m [	.
	leave					leave					Schw
	Now, the new horizon-entry is when the wave length	k		k	e		N	;
																						end	leave
	k					6	10  8 	[ m	1 	]	,	k	1 		6 . 1	10 7 	] m [	; and the scale arrives factor at	a 	k	H
	end																			end	end	end	end
	, the Hubble length with Compton length	C 	GW 			m	g	_	GW	c		10 13  1 . 2	] m [	; with
	C 		BH		2 . 7	7  10 	m	as in section (4.1) with eq. (2) result	H	end 1 		6	10 24 	] m [	; and
	a	end	_	GW			17 10 6  3 .	,	t	end	end 1   H	c		16 10 3  35 .	s	, we found	N		5 . 7	to match the
	iterations cycle:		m		 		k	T				R		H	1		a		N	.
																					g	B	end	end
	The energy of the graviton becomes at LIGO with the above value at merging
		g	GW			a	GW gBH end _ 			17 22 10 10    3 . 6 9		4 . 1	10  	39	[	J	]		9 . 8	10  	30	GeV	at BHs merging, and the
	frequency is			c		k	end 1  	Hz 18	, the mass is	m	g	_	GW		6 . 1	56  10 	kg	; the number of
	particles released as the gravitational wave (like the photons of the light wave) remains
	equally with	n	g			M	BH	m	g	0		73 10 5  3 .	, the total energy initially released is
						E									n		M	c	2		3 . 5	10 47 	[	J	]	, and the curvature radius it results from
		GW					mBH						g		g	BH
	eq.(1) with the integral (which the starts at the outside of the disrupted BH) graviton
	energy				g		2	26  10 	a	end	_	BH		26  10	J	, where	a	end	_	BH		7 . 1	(when this is firstly born as
	from the above section (4.) ), with	g n , and with	a	end		a	end	_	GW	, as been
	R					7	10 24 	[	] m	. To note, the dark particles viewed as pure gravity charges have near
		LIGO
	the same gravitational charge, see [2a].