Dogac Basaran
email: dogac.basaran@ircam.fr

Slim Essid

Geoffroy Peeters

MAIN MELODY EXTRACTION WITH SOURCE-FILTER NMF AND CRNN

Estimating the main melody of a polyphonic audio recording remains a challenging task. We approach the task from a classification perspective and adopt a convolutional recurrent neural network (CRNN) architecture that relies on a particular form of pretraining by source-filter nonnegative matrix factorisation (NMF). The source-filter NMF decomposition is chosen for its ability to capture the pitch and timbre content of the leading voice/instrument, providing a better initial pitch salience than standard timefrequency representations. Starting from such a musically motivated representation, we propose to further enhance the NMF-based salience representations with CNN layers, then to model the temporal structure by an RNN network and to estimate the dominant melody with a final classification layer. The results show that such a system achieves state-of-the-art performance on the MedleyDB dataset without any augmentation methods or large training sets.

INTRODUCTION

Automatic dominant melody estimation (AME) is a popular and rather challenging task in Music Information Retrieval (MIR). In general, AME can be defined as the estimation of fundamental frequencies that represent the pitch values of the dominant melody [START_REF] Salamon | Melody extraction from polyphonic music signals: Approaches, applications, and challenges[END_REF]. The source of the dominant melody could be a leading singing voice or an instrument. The difficulty is that there is usually a polyphonic accompaniment to the lead vocal/instrument, and that this accompaniment follows the melody rhythmically and harmonically, in the sense that chord progressions will naturally contain the dominant F 0 and/or its harmonics. As a consequence, it is not trivial to obtain a representation that discriminates the main melody from the background music. Hence, one of the main research directions in AME remains finding a salience representation that enhances the fundamental frequency of the dominant melody against the possibly polyphonic background.

One of the most popular and rather simple salience representations is the Harmonic Sum Spectrum (HSS) [START_REF] Klapuri | Multiple fundamental frequency estimation by summing harmonic amplitudes[END_REF] that consists of mapping the energy among harmonically related F 0 s. This has been used effectively in a popular melody extraction algorithm, jbcorso-called Melodia [START_REF] Salamon | Melody extraction from polyphonic music signals using pitch contour characteristics[END_REF]. Durrieu et. al. [START_REF] Durrieu | A musically motivated mid-level representation for pitch estimation and musical audio source separation[END_REF][START_REF] Durrieu | Source/filter model for unsupervised main melody extraction from polyphonic audio signals[END_REF] proposed a salience function where the dominant melody (singing voice or instrument) is modeled with a Source-Filter Nonnegative Matrix Factorization (SF-NMF). This method was later combined with HSS in [START_REF] Bosch | A comparison of melody extraction methods based on source-filter modelling[END_REF] in order to obtain an enhanced salience representation. There also exist other methods that utilize a simple time-frequency representation, e.g., the Short Time Fourier Transform (STFT) or Constant Q-Transform (CQT), as a low-level representation of salience [START_REF] Fuentes | Probabilistic model for main melody extraction using constant-q transform[END_REF][START_REF] Vincent | Adaptive harmonic spectral decomposition for multiple pitch estimation[END_REF].

Recently, Bittner et. al. [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF] proposed a Convolutional Neural Network (CNN) system to learn salience representations based on harmonic CQT. The rationale for this approach is to learn harmonic relationships implicitly and to obtain a salience representation similar to (or better than) HSS.

Salience-based melody estimation methods usually use pitch tracking methods on top of salience representations to exploit the temporal relationships between dominant F 0 s. In [START_REF] Durrieu | Source/filter model for unsupervised main melody extraction from polyphonic audio signals[END_REF], a Hidden Markov model (HMM) was adopted where the states represent the bins of the source activations, i.e. F 0 s. Then a threshold-based voicing estimation (melody/non-melody estimation) was applied. Another very popular pitch tracking method was proposed by Salamon et. al. [START_REF] Salamon | Melody extraction from polyphonic music signals using pitch contour characteristics[END_REF] where the algorithm creates and characterizes pitch contours on top of HSS. Characteristics of these contours have proven very effective in voicing estimation [START_REF] Bosch | A comparison of melody extraction methods based on source-filter modelling[END_REF][START_REF] Salamon | Melody extraction from polyphonic music signals using pitch contour characteristics[END_REF].

Recently, Deep Neural Networks (DNNs) have become very popular in MIR applications such as sound event detection [START_REF] Sharath Adavanne | Stacked convolutional and recurrent neural networks for bird audio detection[END_REF][START_REF] Bisot | Leveraging deep neural networks with nonnegative representations for improved environmental sound classification[END_REF] and chord estimation [START_REF] Mcfee | Structured training for largevocabulary chord recognition[END_REF]. The ability of DNNs to approximate any function with linear weights and non-linear activations, given enough data, makes such systems attractive for MIR tasks. That said, comparatively few attempts have been made to estimate dominant melody using neural networks. In [START_REF] Leglaive | Singing voice detection with deep recurrent neural networks[END_REF][START_REF] Rigaud | Singing voice melody transcription using deep neural networks[END_REF], bidirectional Long Short-Term Memory (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF], a special kind of Recurrent Neural Network (RNN), are used for singing voice separation. Such networks are mostly used in modeling the temporal information in time sequences. Recently, in [START_REF] Balke | Data-driven solo voice enhancement for jazz music retrieval[END_REF], a hierarchical CNN structure similar to a stacked denoising autoencoder (SDA) [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF] is used to learn a mapping between an STFT representation and a transcription similar to a piano roll. A tutorial on deep learning techniques for MIR tasks can be found in [START_REF] Choi | A tutorial on deep learning for music information retrieval[END_REF].

Although most of these DNNs perform end-to-end training, it has proven effective to use a more structured input data, such as harmonic-CQT [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF]. Recently, [START_REF] Bisot | Leveraging deep neural networks with nonnegative representations for improved environmental sound classification[END_REF] achieved state-of-the-art results in sound classification by using NMF activations as input as a form of pretraining.

Contributions. Inspired by these works, we propose a Convolutional-Recurrent Neural Network (CRNN) model whose pretraining is based on the SF-NMF model proposed in [START_REF] Durrieu | Source/filter model for unsupervised main melody extraction from polyphonic audio signals[END_REF]. We show that with NMF-based pretraining, we can achieve state-of-the-art results without requiring large training datasets or data augmentation methods, and using relatively simpler networks in terms of training parameters. Our results clearly demonstrate the usefulness of a good input salience representation to the network, suggesting that performance would climb even higher if the SF-NMF model were improved. Our results are obtained on MedleyDB [START_REF] Bittner | Medleydb: A multitrack dataset for annotation-intensive mir research[END_REF], which is a challenging dataset due to inclusion of singing voice and instrument melodies in a diverse set of music genres.

The rest of the paper is organized as follows: the proposed CRNN system and pretraining with SF-NMF are detailed in Section 2. Section 3 discusses the dominant melody estimation results obtained on the MedleyDB dataset, and also gives an analysis of SF-NMF-based salience and the comparison between different CRNN variants. Finally, some conclusions and future directions are given in Section 4.

SYSTEM OVERVIEW

The block diagram of the CRNN system we propose is given in Figure 1. In the first stage (Pretraining), we estimate an initial salience representation using the SF-NMF model. Then this salience is fed into a CNN (CNN stage), where the salience representation is further enhanced by learning local features. The CNN output activations are then fed into an RNN to exploit the long-term relationships between fundamental frequencies (RNN stage). Then in the final Classification stage, we classify the representations as melody/non-melody and give an estimate for F 0 at each time-frame where each class represents a semitone fundamental frequency. Note that the same procedure is applied in both the training and testing of the system.

In the design of the CRNN system, we are inspired by a similar CRNN proposed in [START_REF] Mcfee | Structured training for largevocabulary chord recognition[END_REF] for chord recognition, where the network is interpreted as an encoder-decoder scheme. In the CRNN structure we propose, the CNN and RNN stages can also be treated together as an encoding stage (input sequence to mid-level salience representation) where the output is an enhanced salience representation that captures both spatial and temporal features. Then the classification stage acts as a decoding stage (mid-level representation to output sequence) where the salience is

Pretraining with SF-NMF

In [START_REF] Durrieu | Source/filter model for unsupervised main melody extraction from polyphonic audio signals[END_REF], the dominant melody (voice/instrument) is modeled using a source-filter model. Assuming the mixing of the dominant melody and the accompaniment (background) is instantaneous, the source, filter and accompaniment parts are modeled with the SF-NMF model as follows:

V ≈ V = V F0 ⊙ V Φ + V B = W F0 H F0 ⊙ W Φ H Φ + W B H B (1) = W F0 H F0 ⊙ W Γ H Γ H Φ + W B H B ( 2
)
where V represents the power spectrogram of the signal, i.e., V = |X| 2 (where X is the STFT of the audio signal to be analyzed); F 0 , Φ and B represents the source, filter and background respectively; W and H represent the basis and activation matrices; and ⊙ denotes the Hadamard product. The filter basis W Φ is further modeled with yet another NMF representation, as in [START_REF] Durrieu | A musically motivated mid-level representation for pitch estimation and musical audio source separation[END_REF]:

W Φ = W Γ H Γ .
In this model, the source, V F0 = W F0 H F0 , is assumed to have a harmonic structure. To ensure such a structure, the basis W F0 is pre-constructed (not estimated) such that each column represents the harmonic structure for one F 0 . Represented F 0 s start from a minimum frequency, i.e., F 0 = 55Hz, and they are logarithmically spaced, i.e., the ratio between consecutive F 0 values would be 2 (1/60) for a resolution of 5 bins per semitone. Such a construction enforces the corresponding row in the activation matrix H F0 to represent the activation of that specific F 0 , similar to a saliency representation. That is the rationale behind using H F0 as a saliency representation as in [START_REF] Bosch | A comparison of melody extraction methods based on source-filter modelling[END_REF][START_REF] Durrieu | A musically motivated mid-level representation for pitch estimation and musical audio source separation[END_REF][START_REF] Durrieu | Source/filter model for unsupervised main melody extraction from polyphonic audio signals[END_REF].

The main assumption with the filter, V Φ , is to have a smooth structure. One way to ensure such smoothness is to construct a basis W Φ from smooth filters in advance, similar to enforcing harmonic structure in the source V F0 . However it is not possible to directly construct W Φ with smooth basis filter structures since it depends on the dominant melody. In [START_REF] Durrieu | A musically motivated mid-level representation for pitch estimation and musical audio source separation[END_REF], it is proposed to represent W Φ with another NMF model, W Γ H Γ , where the columns of W Γ are constructed (not estimated) as simple and smooth band pass filters that are linearly spaced and overlapping. This structure forces W Φ to be smooth, thus ensuring that V Φ will be smooth as expected.

The accompaniment/background, V B = W B H B , is also represented with a standard NMF model where there are no constraints on the basis such as smoothness or being harmonic. In summary, the source basis W F0 and smooth filter basis W Γ are pre-constructed and the rest of the parameters H F0 , H Γ ,H Φ ,W B and H B are estimated using the standard alternating scheme and heuristic multiplicative updates.

In this work, for the SF-NMF model, we follow the parametrization given in [START_REF] Bosch | A comparison of melody extraction methods based on source-filter modelling[END_REF] where the minimum and maximum frequencies represented in H F0 are chosen as 55Hz and 1760Hz respectively. We choose the resolution of the F 0 s as 5 bins per semitone which results in 60 bins per octave (bpo) and 301 bins in total per frame.

Note that due to the logarithmic spacing of the F 0 s where the consecutive frequencies have a ratio of 2 1/60 , one can tune the represented F 0 s with proper choice of the minimum frequency F 0,min . As an example, if F 0,min = 55Hz, the notes will be tuned to A4 = 440Hz whereas if F 0,min = 55.25Hz, they will be tuned to A4 = 442Hz. This choice of tuning might depend on the target dataset. Here, we choose the tuning A4 = 440Hz assuming that such tuning is more widely used. It is important to mention that this construction of F 0 s in W F0 cannot be generalized to all music genres, e.g., traditional Turkish music with makams. Hence the methods based on SF-NMF, as well as the proposed scheme, are limited in that sense.

Although we aim to classify the fundamental frequencies at semitone resolution, we initially choose a higher resolution for the F 0 s in W F0 . In practice, it is highly probable that a dominant voice or instrument will be slightly out-of-tune, and hence will not fit any of the represented F 0 s. In such cases, a high resolution representation of F 0 s might better describe these out-of-tune notes.

CNN stage

In order to enhance the H F0 -salience, we propose two different CNN architectures, which we denote as CNN1 and CNN2. In CNN architecture 1 (CNN1), we first decrease the F 0 resolution to semitones, then we train CNN layers to learn local structures, i.e., the confusions between semitones. In the second approach (CNN2), we follow the network proposed in [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF]. Here, the network learns the features in the original resolution and within a semitone interval with one additional layer that learns the octave patterns.

Note that since each CNN architecture only applies 2D linear filters and non-linear activations, the input structure is not lost through the layers of the network. This provides an advantage of interpretable hidden layer activations and leads to a new form of salience as output where each row still represents the activation of a fundamental frequency.

In both architectures, rectified linear units (ReLus) are used as non-linear activations and are applied to each CNN layer output. Batch normalization is applied before each intermediate CNN layer input, as it has proven effective in the convergence of the network by reducing the internal covariance shift [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF]. The columns of H F0 are normalized with l 1 norm before being fed into the CNN network. Such a normalization is possible since the task at hand is the estimation of the melody; that is, only the position of the fundamental frequency is needed, not the exact energy.

CNN Architecture 1 (CNN1)

There are 5 layers in the CNN1 architecture. The first layer gathers the energy around each semitone by applying focused filters centered around each semitone frequency. In this layer, there are 64 (5x1) filters each with a stride [START_REF] Bittner | Medleydb: A multitrack dataset for annotation-intensive mir research[END_REF][START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF]. This way, not only is the energy focused on the semitones, but also the frequency resolution is decreased to the semitone scale from 5 bins per semitone (time resolution remains the same). The rationale behind the first layer is two-fold: First, the number of parameters is severely decreased by lowering the frequency resolution, i.e., it takes 5 times less filter parameters in order to learn features. Second, out-of-tune notes would already be represented in the vicinity of the corresponding semitone in the H F0 representation. Focused filters on semitones would gather the energy on the semitone that is a way of retuning the melody on the represented semitone fundamental frequencies.

In the following layers, zero padding is applied to convolutions to keep the dimensions unchanged. The second layer has 64 (5 x 3) filters that cover ±2 semitone interval and roughly 30ms in time. Then the third layer has 64 (3 x 3) filters that cover ±1 semitone and 30ms in time.

The fourth layer has 16 (15 x 3) filters to learn note confusions in one octave. Filters cover ±7 semitone interval and again 30ms in time. Then enhanced salience representation is obtained as the output of the final CNN layer that has only one (1x1) filter as in [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF] but with a rectified linear unit instead of a sigmoid. The overall structure of CNN architecture 1 is shown in Figure 2.

CNN Architecture 2 (CNN2)

CNN2 is based on the network proposed in [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF]. In this network, the resolution of the input remains the same throughout the layers of the CNN, i.e., no pooling is applied. Note that the input to CNN2 is H F0 ; therefore, the first layer of the network contains only a single channel instead of six.

As mentioned before, the overall system targets semitone resolution for the output fundamental frequencies. This requires a reduction in resolution somewhere in the system. In this architecture, we left the dimensionality reduction to the final classification layer.

RNN stage

Recurrent neural networks are mostly used in MIR and audio analysis tasks to model the dynamics of the observations, typically for chord recognition [START_REF] Mcfee | Structured training for largevocabulary chord recognition[END_REF] and speech recognition [START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF]. Here, we use a single bidirectional Gated Recurrent Unit (BiGRU) layer to capture temporal relationships between F 0 s. A GRU is a special kind of RNN [START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF] where the units are able to model long-term temporal relationships whilst using a gate structure. It has the advantage of not suffering from the vanishing gradient problem of standard RNN and has proven to be easier to train compared to the LSTM alternative.

The number of units in a BiGRU layer should be chosen higher or equal to the output dimension of the preceding CNN network. In the BiGRU structure, actually two GRU layers are trained with the same input but in reverse directions to model the F 0 relationships from both directions. Later, the two layers are merged to have a single output.

Classification

The final layer of the system is a classifier where one class represents the non-melody and the rest of the 61 classes represent semitone fundamental frequencies between A1 and A6 (inclusive). The multiclass classification output is obtained with a single dense layer and softmax activations.

The overall system is trained minimizing the cross entropy loss between the softmax activations and true probabilities. A frame is classified as a non-melody frame only if the probability of non-melody class is higher than the rest. Regardless of this decision, F 0 is estimated for each frame by simply picking the most probable F 0 class among the 61 note classes. Note that even if the non-melody class has the highest probability, the second-highest probability gives a good estimation of the pitch.

An example output of the classification layer that is obtained from a CNN1 + RNN + Classification architecture is shown in Figure 3. In this example, H F0 input (top-left) gives a very good initial salience. Then the CNN1 output activations (top-right) further enhance the dominant part against the harmonic background. It is observed that the dominant F 0 classes mostly have the highest probabilities against the rest of the class probabilities (bottom-left).

EXPERIMENTS

In this section, we evaluate the proposed NMF-based CRNN system using the MedleyDB dataset [START_REF] Bittner | Medleydb: A multitrack dataset for annotation-intensive mir research[END_REF]. For the annotations, we use the "Melody2" definition in Med-leyDB that is the F 0 of the dominant melody at each time step, drawn from multiple sources. With this definition of melody, it is possible to have separate instruments or We randomly split the MedleyDB set into train, validation and test sets such that the tracks from the same artist do not belong to different sets following the artist conditional random splitting as in [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF][START_REF] Bosch | A comparison of melody extraction methods based on source-filter modelling[END_REF]. There are 27 full-length tracks in the test set, 67 full-length tracks in the training set and 14 full-length tracks in the validation set. Note that we used the same test split with [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF] in the MedleyDB in the rest of the experiments to be able compare the results.

We use the five standard evaluation metrics given in [START_REF] Salamon | Melody extraction from polyphonic music signals: Approaches, applications, and challenges[END_REF], namely: Raw Pitch Accuracy (RPA), Raw Chroma Accuracy (RCA), Overall Accuracy (OA), Voicing False Alarm (VFA) and Voicing Recall (VR). All the codes are written in Python and available online1 . CQT implementation is based on the librosa python package [START_REF] Mcfee | [END_REF].

Network training

We trained three different networks with the following combinations of the architectures given in Section 2:

CRNN-1: CNN1 + 1 layer BiGRU (128 Units) + Classification layer; CRNN-2: CNN2 + 1 layer BiGRU (160 Units) + Classification layer; C-NN: CNN2 + Classification layer.

We further denote the network variants by prepending a label indicating the input to the network: "SF" for H F0 input and "CQT" for CQT input. Note that the CQT parameters are chosen such that the representation of a signal via H F0 or CQT would have the same dimensions2 .

CRNN-1 CRNN-2 Baseline # of Param. 307,199 854,319 406,253

Table 1: The number of trainable parameters for CRNN-1, CRNN-2 and the baseline CNN network [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF] In the proposed CRNN structure, the purpose of the CNN stage is to learn local features, whereas the purpose of the RNN stage is to account for long term temporal relationships. This requires selecting relatively small patch lengths for the CNN layers but longer patch lengths for the RNN layer. For this purpose, we used different patch lengths for the CNN and RNN parts while jointly training them.

In all the models, the CNN layers are trained on either 0.29-second (25-frame) or 0.58-second (50-frame) patches and the RNN layer is trained on 5.8-second (500-frame) patches. The training is performed using mini-batches of 16 patches per batch. We use the ADAM optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF], and reduce the learning rate if there is no improvement in validation loss after 20 epochs. The early stopping strategy is used if the validation loss is not decreased after 20 epochs. The maximum possible number of epochs is set to 200. All models were implemented with Keras 2.0 [START_REF] Chollet | keras[END_REF] and Tensorflow 1.0 [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF] and tested using NVIDIA-Tesla K80 GPUs. The number of parameters for each network model is given in Table 1.

Note that, in the training, we do not benefit from any data augmentation method or from other larger datasets.

Results

We compare the outputs of all three models to a CNNbased melody tracking system [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF], considered as a baseline, which proved to significantly outperform the previous state-of-the-art methods in [START_REF] Bosch | A comparison of melody extraction methods based on source-filter modelling[END_REF][START_REF] Salamon | Melody extraction from polyphonic music signals using pitch contour characteristics[END_REF]. The evaluation results of [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF] are available online. 3 By choosing the same test split from the MedleyDB, we are able to compare these published results to ours without any re-evaluation. The evaluation results for all network variants (SF-CRNN-1, SF-CRNN-2, CQT-CRNN-2, SF-C-NN) and for the baseline are given in Figure 4. We use McNemar's test on the classification results and provide p-values as a measure of significance whenever relevant 4 .

CQT vs. H F0 as salience

We explore the usefulness of pretrained input by comparing the evaluation results of the CRNN-2 model when the input is CQT or H F0 -i.e., comparing CQT-CRNN-2 and SF-CRNN-2. The results show that CRNN-2 model performs significantly better in OA (p=0.0015) and RCA (p=0.0003) scores when the input to the network is H F0 . On average, results for SF-CRNN-2 are 6, 9 and 7 percentage points higher for OA, RPA and RCA, respectively.

The reason the CRNN-2 model performs better with pretrained input is that H F0 provides a better initial 3 github.com/rabitt/ismir2017-deepsalience 4 Mcnemar test is based on statsmodel package in python.

H F0

CQT RPA 0.538 ± 0.141 0.210 ± 0.16 RCA 0.648 ± 0.127 0.411 ± 0.15 salience representation than the CQT. Ideally, a salience representation of melody should be discriminative for each target fundamental frequency against the polyphonic background music. We can analyze both H F0 and CQT representations to see how well they fit this definition of "ideal" salience by performing a simple peak-picking strategy as in [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF]. Specifically, the frequency with maximum amplitude/salience for each time frame point is chosen as the estimate of the fundamental frequency. We can compute the RPA and RCA scores using those estimates to see their performances as salience. The results obtained on the full MedleyDB dataset are given in Table 2. It can be seen that H F0 performs nearly twice as well as the CQT representation in both RPA and RCA scores, showing that H F0 provides a better initial salience to the CRNN networks.

SF-CRNN-2 model vs. Baseline CNN Network

The SF-CRNN-2 model uses the CNN-2 architecture in the CNN stage, the same CNN as the baseline. When we compare the evaluation results given in Figure 4, we observe that the SF-CRNN-2 model outperforms the baseline in the RPA (p = 0.0015) and VR (p=0.052) scores. The model has slightly higher OA and RCA scores on average than the baseline. On the other hand, SF-CRNN-2 has a higher number of network parameters (854, 319) than the baseline CNN (406,253). This is due to the additional RNN layer that exists in SF-CRNN-2.

Comparison between variants SF-CRNN-1, SF-CRNN-2 and SF-C-NN On average, SF-CRNN-1 performs slightly better than all other models in all metrics aside from VFA. Comparing SF-CRNN-1 and SF-CRNN-2, we observe that a similar or OA RPA RCA VR VFA SF-CRNN-1 0.444 0.595 0.677 0.556 0.423 Baseline 0.580 0.756 0.725 0.590 0.219 Table 3: Evaluation results for the track "MatthewEn-twistle_TheFlaxenField" where the worst OA performance occurs against the baseline [START_REF] Bittner | Deep salience representations for f 0 estimation in polyphonic music[END_REF].

higher performance can be achieved by the low resolution CNN1 architecture and with far fewer training parameters (see Table 1). VR rates for SF-CRNN-1 and SF-CRNN-2 are significantly higher than the SF-C-NN; however, VFA rates are higher as well. This behavior could be due to the activations of the RNN layer that should force some sort of temporal smoothing on the salience representation.

On the other hand, the significantly better OA, RPA and RCA scores of SF-CRNN-2 relative to SF-C-NN suggest that the temporal tracking with RNN effectively improves the performance of the melody estimation.

Comparing the best performing network variant SF-CRNN-1 to the baseline, we observe that it outperforms the baseline on the OA (p=0.052), RPA (p=0.0003) and VR (p=0.0015) scores, and achieve those results with a less complex network in terms of network parameters (see Table 1). A track-level comparison by computing the overall accuracy differences for each track shows that SF-CRNN-1 performs better on 19 tracks out of 27.

The worst OA of SF-CRNN-1 occurs against the baseline with the "MatthewEntwistle_TheFlaxenField" track where the dominant melody consists only of instruments including Piano. The evaluation results for this track are given in Table 3. It is observed that both SF-CRNN-1 and baseline have relatively high VFA; however, the effect of this is minimal since the track mostly contains voiced frames. On the other hand, the OA score would be highly affected by the combination of high RPA and VR scores. For this track, although the baseline and SF-CRNN-1 have comparable VR rates, the RPA score of the baseline is better, which explains the difference in OA performance.

Singing voice vs. Instrument

Among the test set in MedleyDB, 16 tracks contain only instrumental dominant melody, 3 tracks contain only dominant singing voice melody and 8 tracks contain both 5 . Evaluation results in Table 4 show that SF-CRNN-1 performs better for singing voice melodies than instrument melodies. SF-CRNN-1 outperforms the baseline in overall accuracy for singing voice melodies and instrument melodies.

CONCLUSIONS AND FUTURE WORK

In this work, we have introduced a novel audio-based dominant melody estimation architecture using source-filter NMF as pretraining for a new variant of deep network for SF-CRNN-1 Baseline S.V.

Ins. S.V. Ins. OA 0.638 0.466 0.598 0.424 RPA 0.791 0.647 0.784 0.619 RCA 0.804 0.726 0.823 0.717 Table 4: OA, RPA and RCA scores for singing voice (S.V.) main melody and Instrument (Ins.) main melody for SF-CRNN-1 and baseline. this task, namely a CNN-BiGRU scheme. We have shown that the proposed system achieves state-of-the-art performance on standard evaluation metrics, even significantly improving on it while maintaining a lower system complexity.

Analysis of H F0 as a salience representation shows that it provides a good initial salience in general with high RPA and RCA, even when performing melody estimation using frame-based salience peak-picking. The evaluation results clearly show the usefulness of SF-NMF-based pretraining in many aspects. We observe that when provided with a good initial salience input to the CRNN structure, the system performs considerably better without requiring any augmentation or additional training data. This encourages the idea of improving the pretraining part to obtain even more discriminative salience representations which will surely increase the melody estimation performance. For such improvements, SF-NMF is a good candidate since many other variants with various constraints such as smoothness or sparsity exist in the literature.

We observe that in the proposed CRNN structure, the CNN stage helps to improve the quality of the salience representation against H F0 . In addition, exploiting temporal information with the RNN significantly improves OA, RPA, RCA and VR. These two stages act similarly to an encoder scheme and the classification layer acts as the decoder. Therefore one can interpret the proposed CRNN as an encoder-decoder network where the encoder is used to obtain an enhanced salience representation and the decoder produces a frame-based transcription.

From a melody classification viewpoint, the MedleyDB dataset is quite challenging due to its diverse range of instrumentation and music genres. Also, there is an imbalance between the note classes and the non-melody class in the dataset. The CRNN network has proven effective in handling such imbalance when pretrained with an SF-NMF model.

A clear future direction to pursue is training the SF-NMF and CRNN jointly, learning the H F0 representation while minimizing the classification error.

ACKNOWLEDGEMENT

This project is partly funded by the DigThatLick project. We'd like to thank Rachel Bittner, Dr. Umut Simsekli and Dr. Jordan Smith for their valuable technical support.

c⃝

  Dogac Basaran, Slim Essid, Geoffroy Peeters. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: Dogac Basaran, Slim Essid, Geoffroy Peeters. "Main Melody Extraction with source-filter NMF and CRNN", 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.

Figure 1 :

 1 Figure 1: Block diagram of the proposed CRNN system with pretraining

Figure 2 :

 2 Figure 2: CNN Architecture 1 (CNN1).

Figure 3 :

 3 Figure 3: (Top-left) H F0 representation of a small audio excerpt as input to CRNN, (Top-right) CNN1 activations, (Bottom-left) Classifier activations of CRNN, (Bottom right) Ground-truth annotations.

  feature and CQT feature by simple peak-picking method.

Figure 4 :

 4 Figure 4: Evaluation metrics for SF-CRNN-1, SF-CRNN-2, CQT-CRNN-2, SF-C-NN and the baseline [6].

Table 2 :

 2 The comparison of RPA and RCA scores for H F0

Proceedings of the 19th ISMIR Conference, Paris, France, September[START_REF] Salamon | Melody extraction from polyphonic music signals using pitch contour characteristics[END_REF][START_REF] Salamon | Melody extraction from polyphonic music signals: Approaches, applications, and challenges[END_REF][START_REF] Vincent | Adaptive harmonic spectral decomposition for multiple pitch estimation[END_REF][START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF][27] 2018

github.com/dogacbasaran/ismir2018_dominant_melody_estimation

CQT parameters: Minimum F 0 =55Hz, # of octaves = 5, bpo = 60 Proceedings of the 19th ISMIR Conference, Paris, France, September 23-27, 2018

The ratio of the dominant singing voice melody frames and the dominant instrumental melody frames among all voiced frames is 0.238 and 0.762, respectively.

Proceedings of the 19th ISMIR Conference, Paris, France, September[START_REF] Salamon | Melody extraction from polyphonic music signals using pitch contour characteristics[END_REF][START_REF] Salamon | Melody extraction from polyphonic music signals: Approaches, applications, and challenges[END_REF][START_REF] Vincent | Adaptive harmonic spectral decomposition for multiple pitch estimation[END_REF][START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF][27] 2018


Gabriel Meseguer-Brocal
email: gabriel.meseguerbrocal@ircam.fr

Alice Cohen-Hadria
email: alice.cohenhadria@ircam.fr

Geoffroy Peeters
email: geoffroy.peeters@ircam.fr

DALI: A LARGE DATASET OF SYNCHRONIZED AUDIO, LYRICS AND NOTES, AUTOMATICALLY CREATED USING TEACHER-STUDENT MACHINE LEARNING PARADIGM

The goal of this paper is twofold. First, we introduce DALI, a large and rich multimodal dataset containing 5358 audio tracks with their time-aligned vocal melody notes and lyrics at four levels of granularity.

The second goal is to explain our methodology where dataset creation and learning models interact using a teacher-student machine learning paradigm that benefits each other. We start with a set of manual annotations of draft time-aligned lyrics and notes made by non-expert users of Karaoke games. This set comes without audio. Therefore, we need to find the corresponding audio and adapt the annotations to it. To that end, we retrieve audio candidates from the Web. Each candidate is then turned into a singing-voice probability over time using a teacher, a deep convolutional neural network singing-voice detection system (SVD), trained on cleaned data. Comparing the time-aligned lyrics and the singing-voice probability, we detect matches and update the time-alignment lyrics accordingly. From this, we obtain new audio sets. They are then used to train new SVD students used to perform again the above comparison. The process could be repeated iteratively. We show that this allows to progressively improve the performances of our SVD and get better audiomatching and alignment.

INTRODUCTION

Singing voice is one of the most important elements in popular music. It combines its two main dimensions: melody and lyrics. Together, they tell stories and convey emotions improving our listening experience. Singing voice is usually the central element around which songs are composed. It adds a linguistic dimension that complements the abstraction of the musical instruments. The relationship between lyrics and music is both global (lyrics topics are usually highly related to music genre) and local (it con-nects specific musical parts with a concrete lexical meaning, and also defines the structure of a song).

Despite its importance, singing voice has not received much attention from the MIR community. It has only been introduced a few years ago as a standalone topic [START_REF] Goto | Singing information processing[END_REF][START_REF] Mesaros | Singing voice identification and lyrics transcription for music information retrieval invited paper[END_REF]. One of the most important factors that prevents its development is the absence of large and good quality reference datasets. This problem also exists in other MIR fields, nevertheless several solutions have been proposed [START_REF] Benzi | FMA: A dataset for music analysis[END_REF][START_REF] Fonseca | Freesound datasets: A platform for the creation of open audio datasets[END_REF]. Currently, researchers working in singing voice use small designed dataset following different methodology [START_REF] Fujihara | Lyrics-to-Audio Alignment and its Application[END_REF]. Large datasets as the one used in [START_REF] Humphrey | Mining labeled data from web-scale collections for vocal activity detection in music[END_REF] are private and not accessible to the community.

The goal of this paper is to propose such a dataset and to describe the methodology followed to construct it.

Proposal

We present the DALI dataset: a large Dataset of synchronised Audio, LyrIcs and notes that aims to stand as a reference for the singing voice community. It contains 5358 songs (real music) each with -its audio in full-duration, -its time-aligned lyrics and -its time-aligned notes (of the vocal melody). Lyrics are described according to four levels of granularity: notes (and textual information underlying a given note), words, lines and paragraphs. For each song, we also provide additional multimodal information such as genre, language, musician, album covers or links to video clips. The rest of this paper focuses on our methodology for creating DALI. In Figure 1, we illustrate the input and output of our dataset creation system. See Section 4 for more details about the dataset itself.

The DALI dataset has been created automatically. Our approach consists in a constant interaction between dataset creation and learning models where they benefit from each other. We developed a system that acquires lyrics and notes aligned in time and finds the corresponding audio tracks. The time-aligned lyrics and notes come from Karaoke resources (see Section 3.1 for more deatils). Here, nonexpert users manually describe the lyrics of a song as a sequence of annotations: time aligned notes with their associated textual information. While this information is powerful it has two major problems: 1) there is no information about the exact audio used for the annotation process (only the song title and artist name which may lead to many different audio versions), 2) even if the right audio is found, Figure 1: [Left part] The inputs of our dataset creation system are karaoke-user annotations presented as a triple of {time (start + duration), musical-notes, text}. [Right part] Our dataset creation system automatically finds the corresponding full-audio track and aligned the vocal melody and the lyrics to it. In this example, we illustrate the alignment for a small excerpts. We only represent two levels of lyrics granularity: notes and lines.

annotations may need to be adjusted to fit the audio perfectly. In Section 3.2, we define how we retrieve from the Web the possible audio candidates for each song. In Section 3.3, we describe how we select the right audio among all the possible candidates and how we automatically adapt the annotated time-alignment lyrics to this audio. In order to do this, we propose a distance that measures the correspondence between an audio track and a sequence of manual annotations. This distance is also used to perform the necessary adaptations on the annotations to be perfectly aligned with the audio. Our distance requires the audio to be described as a singing voice probability sequence. This is computed using a singing voice detection (SVD) system based on deep convolutional neural network (Con-vNet). The performance of our system highly depends on the precision of the SVD. Our first version is trained on few but accurately-labeled ground truths. While this system is sufficient to select the right audio it is not to get the best alignment. To improve the SVD, in Section 3.4 we propose to use a teacher-student paradigm. Thanks to the first SVD system (the teacher) we selected a first set of audio tracks and their corresponding annotations. Using them, we train new SVD systems (the students). We show in Section 3.4.1 that new SVD systems (the students) are better than the initial one (the teacher). With this new version, we increase the quality and size of the DALI dataset. Finally, we discuss our research in Section 5.

RELATED WORKS

We review previous works related to our work: singing voice detection methods and the teacher-student paradigm.

Singing Voice detection. Most approaches share a common architecture. Short-time observations are used to train a classifier that discriminates observations (per frame) in vocal or non-vocal classes. The final stream of predictions is then post-processed to reduce artifacts.

Early works explore classification techniques such as Support Vector Machines (SVMs) [START_REF] Mauch | Timbre and melody features for the recognition of vocal activity and instrumental solos in polyphonic music[END_REF][START_REF] Ramona | Vocal detection in music with support vector machines[END_REF], Gaussian mixture model (GMM) [START_REF] Fujihara | Lyricsynchronizer: Automatic synchronization system between musical audio signals and lyrics[END_REF] or multi-layer perceptron (MLP) [START_REF] Berenzweig | Using voice segments to improve artist classification of music[END_REF]. Other approaches also tried to use specific vocal traits such as vibrato and tremolo [START_REF] Regnier | Singing Voice Detection in Music Tracks using Direct Voice Vibrato Detection[END_REF] or to adapt speech recognition systems for the particularities of singing voice [START_REF] Berenzweig | Locating singing voice segments within music signals[END_REF].

Over the past few years, most works focus on the use of deep learning techniques. For example, [START_REF] Schlüter | Learning to pinpoint singing voice from weakly labeled examples[END_REF][START_REF] Schlüter | Exploring Data Augmentation for Improved Singing Voice Detection with Neural Networks[END_REF] propose the use of ConvNet combined with data augmentation techniques (to increase the size of the training set) or trained on weakly labeled data (the data are only labeled at the file level, not at the segment level). [START_REF] Humphrey | Mining labeled data from web-scale collections for vocal activity detection in music[END_REF] also proposes the use of CNN but with a Constant-Q input and a training on a very large private datasets mined from Spotify resources. Some researchers suggest the use of Recurrent Neural Networks (RNN) [START_REF] Lehner | A low-latency, real-time-capable singing voice detection method with lstm recurrent neural networks[END_REF] or Long Short-Term Memory (LSTM) [START_REF] Leglaive | Singing voice detection with deep recurrent neural networks[END_REF]. One advantage of these models is that they directly model the decisions sequence over time and no post-processing is needed. Other singing voice detection systems are developed to be used as a pre-processingstep: for lyrics transcription [START_REF] Mesaros | Singing voice identification and lyrics transcription for music information retrieval invited paper[END_REF] or for source separation [START_REF] Simpson | Deep karaoke: Extracting vocals from musical mixtures using a convolutional deep neural network[END_REF] trained then to obtain ideal binary masks.

Teacher-student paradigm. Teacher-student learning paradigm [START_REF] Ashok | N2N learning: Network to network compression via policy gradient reinforcement learning[END_REF][START_REF] Wu | Automatic drum transcription using the student-teacher learning paradigm with unlabeled music data[END_REF] has appeared as a solution to overcome the problem of insufficient labeled training data in MIR. Since manual labeling is a time-consuming tasks, the teacherstudent paradigm explores the use of unlabeled data for supervised problems. The two main agents of this paradigm are: the teacher and the student. The teacher is trained with labels of well known ground truths datasets (often manually annotated). It is then used to automatically label unlabeled data on a (usually) larger dataset. These new labels (the one given by the teacher) are the ones used for training the student(s). Student(s) indirectly acquire(s) the desired knowledge by mimicking the "teacher behaviour". This model has achived great results for tasks in speech recognition [START_REF] Watanabe | Student-teacher network learning with enhanced features[END_REF] and multilingual models [START_REF] Cui | Knowledge distillation across ensembles of multilingual models for lowresource languages[END_REF]. It has also been proved that student(s) can achieve superior performances than the teacher [START_REF] Cui | Knowledge distillation across ensembles of multilingual models for lowresource languages[END_REF][START_REF] Wu | Automatic drum transcription using the student-teacher learning paradigm with unlabeled music data[END_REF].

SINGING VOICE DATASET: CREATION

Karaoke resources

Outside the MIR community there are rich sources of information that can be explored. One of these sources is Karaoke video games that fit exactly our requirements. In these games, users have to sing along with the music to win points according to their singing accuracy. To measure their accuracy, the user melody is compared with a Table 1: Terms overview: definition of each term used in this paper.

Term Definition

Annotation basic alignment unit as a triple of time (start + duration wrt F r), musical-notes (with 0 = C3) and text.

A file with annotations group of annotations that define the alignment of a particular song.

Offset time as O it indicates the start of the annotations, its modifications moves all bock to the right or left.

Frame rate as F r it controls the annotation grid size stretching or compressing its basic unit.

Annotation voice sequence as avs(t) 2 {0, 1}

singing voice (SV) sequence extracted from karaoke-users annotations.

Predictions as p(t) 2 [0, 1]
SV probability sequence provided by our singing voice detection.

Labels labels sequence of well known ground truths datasets checked by the MIR community.

Teacher first SV detection (SVD) system trained on Labels.

Student

new SVD system trained on the avs(t) for the subset of track for which NCC(ô, fr) Tcorr.

reference timing note (that has fine time and frequency). Hence, large datasets of time-aligned note and lyrics exist. Such datasets can be found as open-source. Nowadays, there are several active and big karaoke open-source communities. In those, non-expert users exchange text files containing lyrics and melody annotations. However there is no further professional revision. Each file contains all the necessary information to describe a song:

• the sequence of triplets {time, musical-notes, text},

• the offset time (start of the sequence) and frame rate (annotation time-grid),

• the song title and the artist name.

We refer to Table 1 for the definition of all the terms we use. These annotations can be transformed to get the time and note frequencies as seen Figure 1.

We were able to retrieve 13339 karaoke annotation files. Although this information is outstanding for our community, it presents several problems that have to be solved:

Global. When performing the annotation, users can choose the audio file they want. The problem is that only the song title and artist name are provided. This combination might refer to different audio versions (studio, radio edit, live, remix, etc.). Consequently, we do not know which audio version has been used. Annotations made for a version do not work for another. Besides, even if the correct audio is known, annotations may not perfectly fit it. As a result annotations must be adapted. This is done by modifying the provided offset time and frame rate. These issues are not problematic for karaoke-users but critical To the automatic creation of a large dataset for MIR research.

Local. It refers to errors due to fact the that users are nonprofessionals. It covers local alignment problems of particular lyric blocks, text misspellings or note mistakes.

In this paper we only focus on global problems leaving the local ones for future works.

WASABI is a semantic database of song knowledge gathering metadata collected from various music databases on the Web [START_REF] Meseguer-Brocal | WASABI: a Two Million Song Database Project with Audio and Cultural Metadata plus WebAudio enhanced Client Applications[END_REF]. In order to benefit from the richness of this database, we first linked each annotation file to Wasabi. To that end, we connected a specific song title and artist name with all possible corresponding audio versions (studio, radio, edit, live, remix, etc.). The WASABI also provides lyrics in a text only annotations (grouped by lines and paragraphs). Using the two lyrics representations (note-based annotations and text only annotations), we created four levels of granularity: notes, words, lines and paragraphs. Finally, WASABI also provides extra multimodal information such as cover images, links to video clips, metadata, biography, expert notes, etc.

Retrieving audio candidates

Our input is an annotation file connected to the WASABI database. This database provides us with the different existing versions (studio, radio, edit, live, remix, etc.) for a song title and artist name combination. Knowing the possible versions, we then automatically query YouTube1 to get a set of audio candidates. We now need to select among the set of audio candidates the one corresponding to the annotation file.

Selecting the right audio from the candidate and adapting annotation to it

Each audio candidate is compared to the reference annotation file. We do this by measuring a distance between both and keeping the one with the largest value.

Audio and annotations live in two different representation spaces that cannot be directly compared. In order to find a proper distance, we need to transform them to a common representation space. Two directions were studied: Annotations as audio. We have explored lyrics synchronization techniques [START_REF] Fujihara | Lyrics-to-Audio Alignment and its Application[END_REF] but their complexity and phonetic model limitations prevent us to use them. As annotations can be transformed into musical notes, score alignment approaches [START_REF] Cont | Evaluation of Real-Time Audio-to-Score Alignment[END_REF][START_REF] Soulez | Improving polyphonic and poly-instrumental music to score alignment[END_REF] seem a natural choice. However, due to missing information in the corresponding score (we only have the score of the vocal melody) these systems failed. We then tried to reduce the audio to the vocal melody (using Melodia [START_REF] Salamon | Melody extraction from polyphonic music signals using pitch contour characteristics[END_REF]) and then align it to the vocal melody score but this also failed. Consequently, we did not persist in this direction.

Audio as annotations. The idea we develop in the remainder is the following. We convert the audio track to a singing-voice probability p(t) over time t. This sequence has value p(t) ! 1 when voice is present at time t and p(t) ! 0 otherwise. This probability is computed from the audio signal using a Singing Voice Detection (SVD)

Figure 2: Architecture of our Singing Voice Detection system using a ConvNet. system described below. We name this function predictions. Similarly, the sequence of annotated triplets {time, musical-notes, text} can be mapped to the same space: avs(t) = 1 when a vocal note exists at t and avs(t) = 0 otherwise. We name this function annotation voice sequence.

Singing Voice Detection system. Our system is based on the deep Convolutionnal Neural Network proposed by [START_REF] Schlüter | Exploring Data Augmentation for Improved Singing Voice Detection with Neural Networks[END_REF]. The audio signal is first converted to a sequence of patches of 80 Log-Mel bands over 115 time frames. Figure 2 shows the architecture of the network. The output of the system represents the singing voice probability for the center time-frame of the patch. The network is trained on binary target using cross-entropy loss-function, ADAMAX optimizer, mini-batch of 128, and 10 epochs.

Cross-correlation. To compare audio and annotation, we simply compare the functions p(t) and avs(t). As explained before, the annotation files also come with a proposed offset time and frame rate. We denote them by O and F r in the following. The alignment between p(t) and avs(t) depends on the correctness of O and F r values. We will search around O and F r to find the best possible alignment. We denote by o the correction to be applied to O and by fr the best F r. Our goals are to: The NCC provides us directly with the best ô value. This value directly provides the necessary correction to be applied to O to best align both sequences.

To find the best value of fr we compress or stretch annotation by changing the grid size. This warp is constant and respect the annotation structure. We denote it as avs fr (t). The optimal fr value is computed using a brute force approach, testing the values of fr around the original F r in an interval controlled by ↵ (we use ↵ = F r ⇤ 0.05):

fr, ô = arg max fr2[F r ↵,F r+↵],o NCC(o, f r)
Our final score is given by NCC(ô, fr).

The audio is considered as good match the annotation if NCC(ô, fr)

T corr . The value of T corr has been found empirically to be T corr = 0.8. For a specific annotation, if several audio candidate tracks have a value NCC T corr , we only keep the one with the largest value. T corr = 0.8 is quite restrictive but even if we may loose good pairs we ensure that those we keep are well aligned. When an audio match is found, the annotations are adapted to it using fr and ô.

Necessity to improve the Singing Voice Detection system. The score NCC proposed above strongly depends on the quality of p(t) (the prediction provided by the Singing Voice Detection (SVD) system). Small differences in predictions lead to similar NCC(ô, fr) values but very different alignments. While the predictions of the baseline SVD system are good enough to select the correct audio candidates (although there are still quite a few false negatives), it is not good enough to correctly estimate fr and ô. As improving the SVD system the number of false negatives will be reduced and we will also find better alignments. We hence need to improve our SVD system.

The idea we propose below is to re-train the SVD system using the set of candidates audio that match the annotations. This is a much larger training set (around 2000) than the one used to train the baseline system (around 100). We do this using a teacher-student paradigm.

Teacher-Student

Our goal is to improve our Singing Voice Detection (SVD) system. If it becomes better, it will find better matches and align more precisely audio and annotations. Consequently, we will obtain a better DALI dataset. This larger dataset can then be used to train a new SVD system which again, can be used to find more and better matches improving and increasing the DALI dataset. This can be repeated iteratively. After our first iteration and using our best SVD system, we reach 5358 songs in the DALI dataset.

We formulate this procedure as a Teacher-Student paradigm. The processing steps of the whole Singing Voice Dataset creation is summarized in Figure 3.

Upper left box. We start from Karaoke resources that provide our set of annotation files. Each annotation file defines a sequence of triplets {time, note, next} that we con- vert to an annotation voice sequence avs(t). For each annotation file, we retrieve a set of audio candidates.

Upper right box. We independently trained a first version of the SVD system (based on ConvNet) using the training set of a ground truth labeled dataset as provided by the Jamendo [START_REF] Ramona | Vocal detection in music with support vector machines[END_REF] or MedleyDB [START_REF] Bittner | Medleydb: A multitrack dataset for annotation-intensive mir research[END_REF] datasets. We call this first version the teacher.

Upper middle part. This teacher is then applied on each audio candidate to predict p(t).

Lower left box. We measure the distance between avs(t) and p(t) using our cross-correlation method. It allows us to find the best audio candidate for an annotation file and the best alignment parameters fr, ô.

Lower middle box. We select the audio annotation pairs for which NCC(ô, fr) T corr = 0.8. The set of selected audio tracks forms a new training set.

Lower right box. This new set is then used to train a new SVD systems based on the same CNN architecture. This new version is called the student. To this end, we need to define the target p to be minimized in the loss L(p, p).

There are three choices: a) we use as target p the predicted value p given right by the teacher (usual teacher-student paradigm).

b) we use as target p the value avs corresponding to the annotations after aligning them using fr and ô.

c) a combination of both, keeping only these frames for which p(t) = avs(t).

Up to now and since the avs have been found more precise than the p we only investigated option b).

We compare in the following part the results obtained using different teachers and students.

Validating the teacher-student pardigm

In this part, we demonstrate that the students trained on the new training-set actually perform better than the teacher trained on the ground-truth label dataset.

Ground-truth datasets: We use two ground-truth labels datasets: Jamendo [START_REF] Ramona | Vocal detection in music with support vector machines[END_REF] and MedleyDB [START_REF] Bittner | Medleydb: A multitrack dataset for annotation-intensive mir research[END_REF]. We created a third dataset by merging Jamendo and MedleyDB named as J+M. Each dataset is split into a train and a test part using an artist filter (the same artist cannot appear in both).

Teachers: With each ground-truth datasets we trained a teacher using only the training part. Once trained, each teacher is used to select the audio matches as described in Section 3.3. As a result, we produce three new training sets. They contains 2440, 2673 and 1596 items for the teacher J+M, Jamendo and MedleyDB respectively. The intersection of the three sets (not presented here) indicates that 89.8 % of the tracks selected using the Med-leyDB teacher are also present within the tracks selected using the J+M teacher or the Jamendo teacher. Also, 91.4 % of the tracks selected using the Jamendo teacher are within the tracks selected using the J+M teacher. It means that the three teachers agree most of the time on selecting the audio candidates.

Students: We train three students using the audio and the avs value of the new training sets. Even if there is a large audio files overlap within the training sets, their alignment (and therefore the avs value) is different. The reason to this is that each teacher gets a different p which results in different fr, ô values.

Results

We evaluate the performances of the various teachers and students SVD systems using the test parts of Jamendo (J test) and MedleyDB (M test). We measure the quality of each SVD system using the frame accuracy i.e. average value over all the tracks of the test set.

Results are indicated in Table 2. In this table, e.g. "Student (Teacher J train) (2673)" refers to the student trained on the 2673 audio candidates and the avs values computed with the Teacher trained on Jamendo train set. Indeed, in this scenario the results are quite different. Applying Teacher J train on M test the results decreases down to 82% (a 5% drop). Similarly when applying Teacher M train on J test the results decreases down to 76% (a 9% drop). Consequently, we can say that the teachers do not generalize very well.

Lastly, the Teacher J+M train trained on J+M train actually performs worse on both J test (82%) and M test (82%) than their non-joined teacher (87% and 85%). These results are surprising and remain unexplained.

Performance of the students. We now test the students. It is important to note that students are always evaluated in a cross-dataset scenario since the DALI dataset (on which they have been trained) does not contain any track from Jamendo or MedleyDB. Hence, there is no possible overfitting for those. Our hypothesis is that students achieve better the results than the teachers because they have seen more data. Especially, we assume that their generalization to unseen data will be better. This is true for the performances obtained with the student based on Teacher M train . When applied to J test, it reaches 80% which is higher than the performances of the Teacher M train directly (76%). This is also true for the performances computed with the student based on Teacher J+M train. When applied either to J test or M test, it reaches 86.5% (86% on Jamendo and 86% on MedleyDB) which is above the Teacher J+M train (82%). Also, 86.5% is similar or above the results obtained with Teacher J train on J test (87%) and Teacher M train on M test (85%). This is a very interesting result that demonstrates the generalization of the student system whichever data-set it is applied to. The student based on Teacher J+M train is the one used for defining the final 5358 songs of the DALI dataset.

However, the performances obtained with the student based on Teacher J train applied to M test (82%) do not improve over the direct use of the Teacher J train (82%).

On alignment. Not explained in this paper is the fact that the fr and ô values computed with the students are much better (almost perfect) than the ones obtained with the teacher. However, we cannot measure it precisely since DALI dataset does not have ground-truth label annotations to that end. Indeed, the goal of this paper is exactly to obtain such annotations automatically.

SINGING VOICE DATASET: ACCESS

The DALI dataset can be downloaded at https://github. com/gabolsgabs/DALI. There, we provide the detailed desciption of the dataset as well as all the necessary information for using it. DALI is presented under the recommendation made by [START_REF] Peeters | Towards a (better) Definition of Annotated MIR Corpora[END_REF] for the description of MIR corpora. The current version of DALI is 1.0. Future updates will be detailed in the website.

CONCLUSION AND FUTURE WORKS

In this paper we introduced DALI, a large and rich multimodal dataset containing 5358 audio tracks with their time-aligned vocal melody notes and lyrics at four levels of granularity. We explained our methodology where dataset creation and learning models interact using a teacher-student paradigm benefiting one-another. From manual karaoke user annotations of time-aligned lyrics and notes, we found a set of matching audio candidates from the Web. To select and align the best candidate, we compare the annotated vocal sequence (corresponding to the lyrics) to the singing voice probability (obtained with a ConvNet). To improve the latter (and therefore obtain a better selection and alignment) we applied a teacher-student paradigm.

Through an experiment, we proved that the students outperform the teachers notably in a cross-dataset scenario, when train-set and test-set are from different datasets.

It is important to note that the results of the students are higher than the teacher ones, even if they have been training on imperfect data. In our case, we showed that, in the context of deep learning, it is better to have imperfect but large dataset rather than small and perfect ones. However, other works went in the opposite direction [START_REF] Amatriain | in machine learning, is more data always better than better algorithms?[END_REF].

Future work. We have only performed the teacherstudent iteration once. In next works will use the results of the first student generations to train a second student generations. This will define a new DALI dataset. We plan to quantitative measure the quality of ô, fr and to continue exploring the alignments between note annotations and the audio. Currently, we trained our student using as target p = avs, which do not transfer directly the knowledge of the teacher. We will explore other possibilities of knowledge transfer using other targets (points a) and c) in Section 3.4) as well as the local problems describe at Section 3.1.

Acknowledgement. This research has received funding from the French National Research Agency under the contract ANR-16-CE23-0017-01 (WASABI project).

Figure 3 :

 3 Figure 3: Singing Voice Dataset creation using a teacher-student paradigm.

Table 2 :

 2 Performances of the teachers and students using the various datasets. Number of tracks in brackets.Performance of the teachers. We first test the teachers.Teacher J train obtains the best results on J test (87%).Teacher M train obtains the best results on M test (85%).In both cases, since training and testing are performed on two parts of the same dataset, they share similar audio characteristics. These results are artificially high. To best demonstrate the generalization of the trained SVD systems, we need to test them in a cross-dataset scenario, namely train and test in different datasets.

	SVD system	Test set	J test (16) M test (36)
	Teacher J train (61)	87%	82%
	Student (Teacher J train) (2673)	82%	82%
	Teacher M train (98)	76%	85%
	Student (Teacher M train) (1596)	80%	84%
	Teacher J+M train (159)	82%	82%
	Student (teacher J+M train) (2440)	86%	87%

We use https://github.com/rg3/youtube-dl Proceedings of the 19th ISMIR Conference, Paris, France, September

[START_REF] Schlüter | Learning to pinpoint singing voice from weakly labeled examples[END_REF][START_REF] Schlüter | Exploring Data Augmentation for Improved Singing Voice Detection with Neural Networks[END_REF][START_REF] Simpson | Deep karaoke: Extracting vocals from musical mixtures using a convolutional deep neural network[END_REF][START_REF] Soulez | Improving polyphonic and poly-instrumental music to score alignment[END_REF][START_REF] Watanabe | Student-teacher network learning with enhanced features[END_REF] 2018

Matches between p(t) and avs(t) can also be found using Dynamic Time Warping (DTW). However, we found its application not successfull for our purpose. Indeed, DTW computes local warps that does not respect the global structure of the user annotations. In addition, its score is not normalized preventing its use for matches selection.

Proceedings of the 19th ISMIR Conference, Paris, France, September[START_REF] Schlüter | Learning to pinpoint singing voice from weakly labeled examples[END_REF][START_REF] Schlüter | Exploring Data Augmentation for Improved Singing Voice Detection with Neural Networks[END_REF][START_REF] Simpson | Deep karaoke: Extracting vocals from musical mixtures using a convolutional deep neural network[END_REF][START_REF] Soulez | Improving polyphonic and poly-instrumental music to score alignment[END_REF][START_REF] Watanabe | Student-teacher network learning with enhanced features[END_REF] 2018

Proceedings of the 19th ISMIR Conference, Paris, France, September[START_REF] Schlüter | Learning to pinpoint singing voice from weakly labeled examples[END_REF][START_REF] Schlüter | Exploring Data Augmentation for Improved Singing Voice Detection with Neural Networks[END_REF][START_REF] Simpson | Deep karaoke: Extracting vocals from musical mixtures using a convolutional deep neural network[END_REF][START_REF] Soulez | Improving polyphonic and poly-instrumental music to score alignment[END_REF][START_REF] Watanabe | Student-teacher network learning with enhanced features[END_REF] 2018


Electrodeposition of TiO

We report on the electrodeposition of TiO 2 upon transparent conducting oxide (ITO) by using a milder electrochemical bath based on room-temperature ionic liquids (1-ethyl-3-methyl imidazolium-bis(trifluoromethylsulfonyl)imide)) and titanium butoxide as a soluble precursor. By taking advantage of oxygen reduction which proceeds about 500 mV before titanium +IV reduction, well crystallized thin films of anatase Ti0 2 are tailored under potentiostatic control at -1.5 V (vs. AgCI/Ag) / I00'C followed by a rapid post-annealing treatment at 500 °C in air.

[DOi: IO,ll49/2,010406eel] The tailoring of nanostructured metal oxide films is at the basis of thorough investigations for both fondamental and applied purposes. The control from the meso-down to the microstructure, the film 's tex turation or the ability to endorse preferentially a given crystal facet; all of these parameters are pooling the challenges to grasp utmost levels of performances. The relationship between the fi lm's character istics and their functional properties has been scrutinized since a long date in many fields; especially in (photo)-electrochemical devices 1 for (photo)-catalysis 2 or opto-electronic applications. 3 Various meth ods of fabrication have been considered, frorn physical high-vacuum deposition to soften low-temperature chemical methodologies. The preference.between ail these methods is firstly governed by the chem ical complexity of the material to grow. Chimie douce methods are often well-adapted to tailor "simpler" binary/ternary oxide systems where the film's nanostructuration impacts the ensuing material prop erties. The utilization of low temperature processes offers evident advantages to finely control the particles or the film's mesostructure while also being fully compatible with industrial standards for mass production. Ail these advantages are accessible by the electrodepo sition technique with pros and cons. It is a methodology rendering a surface/electrode functional, so far associated with aqueous elec trochemical bath. However, the low boiling points of water together with its narrow electrochemical window are two constraints restricting the practical versatility of electrodeposition. Recent researches were prompted toward less conventional media, eg. room-temperature ionic liquids (RT-ILs) which hold unprecedented specifi cations in terms of ionic conductivity, high chemical, electrochemical and thermal stabil ity ( > 300 °C). They were pushed among the rnost studied media for the last years des pite a strong lack of understanding regarding the nature of the chemical/electrochemical active species formed in these liquids. A large bran ch of materials have been synthesized from noble metals, 4 • 5 metals and interrelated alloys 6 -8 to semi-conductors. 9 -12 The family of imidazolium and pyrolidinium displays a narrow oxygen, sulfur and iodine solubility paving the way to binary systems. 8 • 13 -19 Research on the electrodeposition of anatase TiO2 remains relatively scarce owing to the stability of the Ti 3+ only maintained to highly acidic aqueous solutions and its high sensitiveness to air. 20 In this work, we report for the first time a more adapted and sirnpler/safer methodology for the electrodeposition of well-crystallized mesoporous films of anatase TiO 2 by using titanium butoxide in conjunction with the 1,3 ethyl methylimidazolium bis(trifluoromethylsulfonyl)imide (EMLTFSI),

Experimental

Ti tanium precursors were purchased from Sigma-Aldrich, 99.9% EMI.TFSI from Solvionic and further purified at 75 °C in vacuum for 2 days (H2O content <6 ppm). Prior and during the electrodepo sition, the electrochemical bath is saturated with ultra-pure oxygen dehydrated through a P 2 0 5 column (Alphagaz), zE-mail: frederic.sauvage@u-picardie.fr A three-electrode setup is composed of Asahi ITO glass as working-electrode (WE), Pt gauss as a counter-electrode (CE) and a homemade AgCI/Ag quasi-reference electrode which the potential is -0.09 V vs. Ag + /Ag. The electrochemical measurements were car ried out using an Autolab PGSTAT30 potentiostat/galvanostat (Eco Chemie BV). The morphologies. were investigated using a scanning electron microscope FEI Quanta200 equipped with EDAX analysis or by transmission electron microscopy using a FEI TECNAI F20S TWIN 200 kV. A Bruker D8 diffractometer with CuK ll radiation (À= 1.54056 Â) was used for filrn's structural characterization.

Results and Discussion

The first prerequisite to succeed in the electrodeposition of TiO 2 concems the evaluation of the electrochemical reduction potential of Ti 4+ with respect to 0 2 -which can get reduced at once. Indeed, con versely to aqueous chemical bath, in this case the two-electron oxygen reduction will onset the precipitation of the metal oxide at the elec trode's surface. It is therefore required to gather two distinct features: (i) a soluble titanium precursor and (ii) the titanium (+IV) reduction should proceed at higher cathodic potentials than solubilized oxygen. Different type of soluble titanium precursors have been investigated (eg. chloride, alkoxyde, oxosulfate .. . ). The tetrachloride is the most soluble. The ligand influences drastically the Ti( +IV) reduction as showed the first cycle voltarnperogram in EMI.TFSI containing ei ther 0.12M oftitaniurn butoxide (TiOBu) purged or not with oxygen or 0.35M of titanium tetrachloride (concentrations corresponding to the solubility limit) (Fig. la). The solubility of oxygen is quantita tively rather comparable to the titanium precursor. Oxygen reduction proceeds according to two steps on ITO, the formation of the stable superoxide anion (0 2 -.) at -1.0 V followed by the less stable perox ide anion (0/-) at -1.4 V (Figure 1 b). Whereas the first electron is reported to be a well-reversible process even though the superoxide anion bas about 30 times lower diffusion coefficient than molecular oxygen, 20 our results underline the irreversible character of the whole two-electron reduction which accounts for the instable character of the strong nucleophilic peroxide anion electrogenerated. Comparing the reduction wave of oxygen with (Fig. 1 a) and without TiOBu (Fig. lb), the two-step reduction of oxygen is not as well separated in presence of TiOBu. Interestingly, the introduction ofTiOBu increases remarkably thé oxygen solubility in EMI.TFSI since four times higher current is generated at the electrode Ocath = 815 µ.A/cm 2 ). The integra tion of this voltamperogram, assuming a strict two-electron process, gives an initial oxygen concentration in the medium of ca. 33 mmol/L (at 100 °C) by contrast to the 3.9 rnmol/L determined by Buzzeo et al. in pure EMI.TFSI. 21 Based on our multiple experiences, we ar gue that TiOBu acts as an oxygen carrier in EMI.TFSI, as sirnilarly to the role played by the artificial perfluorinated compounds developed for abiotic blood substitutes.to replace hemoglobin. 22 The second main result concems the role played by the ligand on the reduction potential of the Ti 4+ . In the case of TiC1 4 , the reduction of titanium (+IV) proceeds in two quasi-reversible steps at +0.05 V and -1.65 V, each accounting for two electrons 23 leading to titanium metal. 9 By contrast, TiOBu reduction gives only one sharp reduction peak at ca. -2.0 V ending up to titanium metal as also suggest the characteristic (J-V) loop corresponding to metal electrodeposition/dissolution. The length of the alkoxyde chain has low effect on the redox potential whereas the greater solubility is obtained with the longer lipophilic alkyl butoxide chain. The lowest cathodic potential of the alkoxydes suggest that the central titanium cation has a stronger covalent character than for TiCl 4 .

The electrodeposition of TiO 2 can thus be achieved using titanium alkoxyde since its reduction proceeds 500 mV below oxygen reduction (Fig. 1a). For the electrodeposition process, we privileged potentiostatic deposition over galvanostatic method to preserve selective reduction of oxygen upon deposition time. The chronoamperogram in Fig. 1c features the typical evolution observed at -1.5 V/100 • C for 3 hours. It corresponds to the optimal conditions to reach thin and transparent well-covering film without any trace of IL degradation. Three distinct domains are classically observed: (i) a rapid drop of current, from 2000 to 500 μA.cm -2 during the first minute, (ii) followed by a slight current increase to ca. 700 μA.cm -2 until ca. 2400s and (iii) a continuous current decrease to ca. 350 μA.cm -2 . The chronoamperogram integration leads to a charge density of 5.32 C.cm -2 corresponding to a mass deposited of 2.2 mg.cm -2 when assuming the peroxide anion to react with Ti 4+ or twice more in case the superoxide anion is involved. Tulodziecki et al. demonstrated that in fact both anions are responsible for ZnO electrodeposition. This would lead to an experimental mass lying between 2.2 and 4.4 mg.cm -2 . 15 The film's morphology is composed of regular spherical particles of 1μm diameter (Fig. 2a). The topology appears rough including large macropores which can be found along the film depth. EDX analysis confirm the presence of titanium without residues of carbon and fluorine which could have originated from IL degradation or poorly grafted units. Both XRD and TEM analysis reveals that the as-deposited films are amorphous in nature regardless of the applied potential (-1.3 V < E app < -1.6 V) and deposition temperature (<150 • C) (Fig. 2b-3). For applied potential equal or below -1.6 V and for temperatures above 120 • C, EMI.TFSI degrades yielding on the film's surface globular particles and excessive amount of fluorine and carbon in the film. Since the as-deposited film is X-ray amorphous (Fig. 3), this latter can turn easily crystallized by a rapid post-annealing treatment at 500 • C for 5 hours in air. The resulting XRD pattern shows then the growing of two sharp diffraction peaks attributed to the (101) and (200) reflections of the anatase polymorph while exhibiting the characteristic 3.2 eV bandgap energy. The preferential orientation along the [101] direction is not surprising since it corresponds to the low energy facet of the anatase structure.

To conclude, this work reports the first example of anatase TiO 2 electrodeposition using RT-ILs. This was realized throughout a twostep process: potentiostatic electrodeposition at -1.5 V/100 • C followed by a rapid post-annealing process entailing the anatase crystallization. This methodology is compatible with a versatile choice of substrates, particularly adequate to non-noble metals and transparent conducting oxide (TCO) glasses as opposed to the aggressive TiCl 3(aq.) solution proposed so far. Finally, the results herein presented also highlight one supplementary role of the titanium butoxide which acts as an oxygen carrier, increasing oxygen solubility in the ionic liquid to 33 mmol/L. This preliminary work will pave the way to more in-depth study to control more finely the film morphology, thickness and mesoporosity with the final aim to introduce successfully TiO 2 electrodeposited films for photonic applications.

Figure 1 .

 1 Figure 1. Cyclic voltamperogram recorded at 100 • C (v = 50 mV.s -1 ) of (a) 0.12M TiOBu in EMI.TFSI (black curve) with electrochemical bath purged of O 2 (red curve) and 0.35M TiCl 4 (blue curve). (b) EMI.TFSI purged with O 2 . (c) Typical chronoamperogram recorded at -1.5 V in 0.12M TiOBu in EMI.TFSI at 100 • C purged with O 2 .

Figure 2 .

 2 Figure 2. (a) SEM and (b) TEM image of the as-prepared electrodeposited film on ITO obtained by potentiostatic deposition at -1.5 V for 3 hours.

Figure 3 .

 3 Figure 3. Comparison of the XRD diffractogram before and after air postannealing at 500 • C/5 hours.

2 Using Ionie Liquids Snehangshu Patra,a, h Stéphanie Bruyère,a Pierre�Louis Tabema,h , c and Frédéric Sauvage ", c,z ciwboratoire de Réactivité et Chimie des Solides, Université de Picardie Jules Verne, CNRS UMR7314, 33 rue Saint

  -Leu, 80039 Amiens, France b Réseau sur le Stockage Electrochimique de !'Energie (RS2E), FR CNRS3459, France

c Université Paul Sabatier, Toulouse l/1, CIRIMAT CNRS UMR5085, France