Many people listen to recorded music as part of their everyday lives, for example from radio or TV programmes, CDs, downloads or increasingly from online streaming services. Sometimes we might want to remix the balance within the music, perhaps to make the vocals louder or to suppress an unwanted sound, or we might want to upmix a 2-channel stereo recording to a 5.1channel surround sound system. We might also want to change the spatial location of a musical instrument within the mix. All of these applications are relatively straightforward, provided we have access to separate sound channels (stems) for each musical audio object.

However, if we only have access to the final recording mix, which is usually the case, this is much more challenging. To estimate the original musical sources, which would allow us to remix, suppress or upmix the sources, we need to perform musical source separation (MSS).

In the general source separation problem, we are given one or more mixture signals that contain different mixtures of some original source signals. This is illustrated in Figure 1 where four sources, namely vocals, drums, bass and guitar, are all present in the mixture. The task is to recover one or more of the source signals given the mixtures. In some cases, this is relatively straightforward, for example, if there are at least as many mixtures as there are sources, and if the mixing process is fixed, with no delays, filters or non-linear mastering [START_REF]Independent Component Analysis[END_REF].

However, MSS is normally more challenging. Typically, there may be many musical instruments and voices in a 2-channel recording, and the sources have often been processed with the addition of filters and reverberation (sometimes nonlinear) in the recording and mixing process. In some cases, the sources may move, or the production parameters may change, meaning that the mixture is time-varying. All of these issues make MSS a very challenging problem.

Nevertheless, musical sound sources have particular properties and structures that can help us. For example, musical source signals often have a regular harmonic structure of frequencies at regular intervals, and can have frequency contours characteristic of each musical instrument.

They may also repeat in particular temporal patterns based on the musical structure.

In this paper we will explore the MSS problem and introduce approaches to tackle it. We will begin by introducing characteristics of music signals, we will then give an introduction to MSS, and finally consider a range of MSS models. We will also discuss how to evaluate MSS approaches, and discuss limitations and directions for future research 1 .

II. CHARACTERISTICS OF MUSIC SIGNALS

Music signals have distinct characteristics that clearly differentiate them from other types of audio signals such as speech or environmental sounds. These unique properties are often exploited when designing MSS methods, and so an understanding of these characteristics is crucial.

All music separation problems start with the definition of the desired musical source to be separated, often referred to as the target source. In principle, a musical source refers to a particular musical instrument, such as a saxophone or a guitar, that we wish to extract from the audio 1 A dedicated website with complementary information about MSS including sound examples, an extended bibliography, dataset information, and accompanying code can be reached on the following link: https://soundseparation. songquito.com/ mixture. In practice, the definition of musical source is often more relaxed, and can refer to a group of musical instruments with similar characteristics that we want to separate. This is the case, for example, in singing voice separation, where the goal often includes the separation of both the main and background vocals from the mixture. In some cases, the definition of musical source can be even looser, as is the case in harmonic-percussion separation, where the aim is to separate the pitched instruments from the percussive ones.

In a general sense, musical sources are often categorized as either predominantly harmonic, predominantly percussive, or as singing voice. Harmonic music sources mainly contain tonal components, and as such, they are characterized by the pitch or pitches they produce over time. Harmonic signals exhibit a clear structure composed of a fundamental frequency F0 and a harmonic series. For most instruments, the harmonics appear at multiple integers of the fundamental frequency: for a given F0 at 300 Hz, a harmonic component can be expected close to 600 Hz, the next harmonic around 900 Hz, and so on. Nonetheless, a certain degree of inharmonicity, i.e., deviation of harmonics from multiple integers of F0, should be expected and accounted for. Harmonic sources exhibit a relatively stable behavior over time, and can typically be identified in the spectrogram as horizontal components. This can be observed in Figure 2 where a series of notes played by an acoustic guitar are displayed. Additionally, the trajectories in time of the F0 and the harmonics are usually very similar, a phenomenon referred to as common fate [START_REF] Bregman | Auditory Scene Analysis: The Perceptual Organization of Sound[END_REF]. This can be clearly seen in the spectrogram of the vocals in Figure 2, where it can be observed that the vocal harmonics have common trajectories over time. The relative strengths of the harmonics, and the way that the harmonics evolve over time, contribute to the characteristic sound, or timbre, which allows the listener to differentiate one instrument from another. Furthermore, and particularly in Western music, the sources often play in harmony, where the ratios of the F0s of the notes are close to integer ratios. While harmony can result in homogeneous and pleasing sounds, it most often also implies a large degree of overlap in the frequency content of the sources, which makes separation more challenging.

In contrast to harmonic signals, which contain only a selected number of harmonic components, percussive signals contain energy in a wide range of frequencies. Percussive signals exhibit a much flatter spectrum, and are highly localized in time with transient-like characteristics. This can be observed in the drums spectrogram in Figure 2, where clear vertical structures produced by the drums signal can be observed. Percussive instruments play a very important role of conveying rhythmic information in music, giving a sense of speed or tempo in a musical piece.

In reality, most music signals contain both harmonic and percussive elements. For example, a note produced by a piano is considered predominantly harmonic, but it also contains a percussive attack produced by the hammer hitting the strings. Similarly, singing voice is an intricate combination of harmonic (voiced) components, produced by the vibrations of the vocal chords, and percussive-like (plosive) components including consonant sounds such as 'k' or 'p' where no vocal fold vibration occurs. These components are in turn filtered by the vocal cavity, with different formant frequencies created by changing the shape of the vocal cavity. As seen in Figure 2, singing voice typically exhibits a higher rate of pitch fluctuation compared to other musical instruments.

A notable property of musical sources is that they are typically sparse in the sense that for the majority of points in time and frequency, the sources have very little energy present. This is commonly exploited in MSS, and can be clearly seen for each of the sources in Figure 2.

Another characteristic of music signals is the fact that music signals typically exhibit repeating structures over different time-scales, for example, a repeating percussion pattern over a couple of seconds, to larger structures such as the verse-chorus structures found in pop songs. As will be explained in Section IV-B, these repetitions can be leveraged when performing MSS.

The bulk of research on MSS to-date has focused on Western pop music as the main genre to be separated. It should be noted that other types of music present their own unique problems for MSS, for example, many instruments playing in unison in some types of traditional/folk music. These cases are typically not covered by existing MSS techniques.

Once the target source has been defined, the characteristics of the audio mixture should be carefully considered when developing MSS methods. Modern music production techniques offer innumerable possibilities for transforming and shaping an audio mix. Most music signals nowadays are created using audio content from a great diversity of origins, usually combined and mixed using a Digital Audio Workstation (DAW), a software system for transforming and manipulating audio tracks. As depicted in Figure 3 for the trumpet, some musical sources can be recorded in a traditional manner using a microphone or a set of microphones. Very often, hardware devices that shape and color the sound are introduced into the signal chain; these may include, for example, guitar tube amplifiers or distortion pedals which can impart very particular sound qualities to the signal. In other cases, musical sources are not captured using a microphone.

Instead, the digital signal they produce is directly fed into the DAW, or alternatively created within the system itself, using, for example, a keyboard as an interface as shown in Figure 3 Most frequently, an audio interface is used to facilitate the process of capturing input signals from different origins, and delivering them to the DAW. The DAW itself offers many additional possibilities to further enhance and modify the signal. Once all the audio content is in the DAW, the final step is the creation of the audio mixture. Most commercial music today is in stereo format (2 channels). Other multi-channel formats such as 5.1 (5 main channels plus a low frequency channel) are available but less common. The number of channels available to perform music separation is a key factor that can be exploited when designing MSS models. Multichannel mixtures allow spatial positioning of the music sources in the sound field. This means that a certain source can be perceived as coming from the left, the center, the right or somewhere in between. The spatial positioning of the sources is usually achieved with a pan pot that regulates the contribution of each musical source in each of the available channels. This artificial creation of spatial positioning ignores delay as a spatial cue, and so inter-channel delay is much less important in MSS than in speech source separation. In contrast, monophonic (single channel) recordings offer no information about spatial positioning, and are often the most challenging separation problem.

A final but fundamental aspect to be considered when designing MSS systems is the fact that music quality, and audio quality in general, is inherently defined and measured by human perception. This sets an additional challenge to MSS methods: regardless of the mathematical soundness of the model, all systems are expected to result in perceptually pleasing music signals.

Aside from the task of truthfully capturing the target source, we must also minimize the impact on perceptual quality of the distortions introduced in the separation process.

III. A TYPICAL MUSICAL SOURCE SEPARATION WORK FLOW

A high-level overview of the steps involved in most MSS systems is illustrated in be explained in Section IV, source models and spatial models are the most common approaches used for MSS. In Figure 4, an example is presented where starting with the mixture X, estimates of the magnitude spectrograms of the sources Ŝj are obtained.

Filtering: The goal at this stage is to estimate the separated music source signals given the source models. This is typically done using a soft-masking approach, the most common form of which is the Generalized Wiener Filter (GWF) [START_REF] Duong | Under-determined reverberant audio source separation using a full-rank spatial covariance model[END_REF], though other soft-masking approaches have been used [START_REF] Rafii | REpeating Pattern Extraction Technique (REPET): A simple method for music / voice separation[END_REF]. Given X and Ŝj , this allows recovery of the separated sources provided their characteristics are well estimated. The STFT of source j = 1 can be estimated elementwise using the GWF as

Ŷ1 (k, n) = X(k, n) Ŝ1 (k, n)/ j Ŝj (k, n).
The same procedure is applied for all sources in the mix. Essentially, each time-frequency point in the original mixture is weighted with the ratio of the source magnitude to the sum of the magnitudes of all sources. This can be understood as a multi-band equalizer of hundreds of bands, changed dynamically every few milliseconds to attenuate or let pass the required frequencies for the desired source. A special case of the Generalized Wiener Filter is the process of binary masking, where it is assumed that only one source has energy at a given time-frequency bin, so that masks values are either 0 or 1.

Estimating source parameters from the mixture is not trivial, and it can be difficult to obtain good source parameters to enable successful filtering at the first try. For this reason, it is common to proceed iteratively: First, separation is achieved with the current estimated source models.

These models are then updated from the separated signals, and the process repeated as necessary.

This approach is illustrated in Figure 4 and rooted in the Expectation-Maximization algorithm.

Most of the models presented in Section IV may be used in this iterative methodology.

Inverse Transform: The final stage in the MSS work flow is to obtain the time domain source waveforms y j using the inverse transform of the chosen time frequency representation.

IV. MUSICAL SOURCE SEPARATION MODELS

Having described the necessary steps for MSS, we now focus on how the unique characteristics of musical signals are used to perform MSS. While numerous categorizations of MSS algorithms are possible, such as categorization by source type, here we take the approach of dividing the algorithms into two broad categories: algorithms which model the musical sources, and those which model the position of the sources in multichannel or stereo audio signals. The key distinction between these two categories is that algorithms in the first category model aspects of the mixture intrinsic to the sources, while those in the second category model aspects intrinsic to the recording/mixing process. Models in the two categories exploit distinct but complementary information which can readily be combined to yield more powerful MSS models.

A. Musical Source Position Models

In the case of multichannel music signals, the spatial position of the sources has often been exploited to perform music source separation. In this section, we assume that we are dealing with a stereo (2-channel) mixture signal, and that the spatial positioning of source j has been achieved using a constant power panning law, defined by a single parameter, the panning angle

φ j ∈ [0, π/2].
For a given source q j , its stereo representation (or stereo image) is given by y 1j = q j cosθ j and y 2j = q j sinθ j , with the subscripts 1 and 2 explicitly denoting the first and second channels, respectively. Figure 5 illustrates the spatial positioning of three sources. The singing voice, for example, is positioned in the center and hence, its stereo image is obtained with an angle of π/4. Component Analysis (ICA) [START_REF]Independent Component Analysis[END_REF], which estimates an unmixing matrix for the mixture signals based on statistical independence of the sources. However, ICA requires mixtures that contain the same number of channels as musical sources in the mix. This is not typically the case for music signals, where there are usually more sources than channels.

As a result of the shortcomings of ICA, algorithms which worked when there were more sources than channels were developed. Several techniques utilizing spatial position for separation, such as DUET [START_REF] Rickard | The DUET blind source separation algorithm[END_REF], ADRess [START_REF] Barry | Real-time sound source separation using azimuth discrimination and resynthesis[END_REF], and PROJET [START_REF] Fitzgerald | Projection-based demixing of spatial audio[END_REF], assume that the time-frequency representations of the sources have very little overlap. This assumption, which holds to a surprising degree for speech, does not hold entirely for music, where the use of harmony and percussion instruments ensures there is overlap. Nonetheless, this assumption has proved to be useful in many circumstances, and often results in fast algorithms capable of real-time separation. The degree of overlap between sources can be seen, for example, in the spectrogram shown in Figure 1 which only shows the dominant musical source in each time-frequency bin of the mixture.

To illustrate the usefulness of assuming very little overlap between sources, consider the element-wise ratio of the individual mixture channels in the time-frequency domain PROJET estimates individual source position histograms, the superposition of which results in the mixture spatial position histogram (as shown in Figure 5). Further, PROJET utilizes the GWF for masking. It should also be noted that both DUET and PROJET can also incorporate and deal with inter-channel delays, extending their range of applicability.

R(k, n) = |X 1 (k, n)/X 2 (k, n)|,
The aforementioned separation methods directly model sound engineering techniques such as panning and delay for creating multichannel mixtures. Another line of research models the spatial configuration of a source directly through inter-channel correlations: At each frequency k, the correlation between the STFT coefficients of the different channels is calculated and encoded in a matrix called the spatial covariance matrix. The core idea of such methods is to leverage the correlations between channels to design better filters than those obtained by considering each channel in isolation. This approach is termed Local Gaussian Modeling (LGM) [START_REF] Duong | Under-determined reverberant audio source separation using a full-rank spatial covariance model[END_REF], [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF]. It can give good separation whenever the spatial covariance matrices are well estimated for all sources.

It is also able to handle highly reverberated signals, for which no single direction of arrival may be identifiable. This strength of covariance-based methods in dealing with reverberated signals comes at the price of difficult parameter inference.

LGM algorithms are often very sensitive to initialization, and the estimated spatial covariance matrices alone are often not sufficient to allow separation. Successful LGM methods need to incorporate musical source models to further guide the separation to obtain acceptable solutions. Musical source models are discussed in the following section.

B. Musical Source Models

While spatial positioning can give good results if each source occupies a unique stereo position, it is common for multiple sources to be located in the same stereo position, or for the mixture signal to consist of a single channel. In these cases, model-based approaches that attempt to capture the spectral characteristics of the target source can be used. In the following sections, a range of MSS model-based approaches are described.

Kernel Models: Similarly to the idea that the definition of the target sources can be relatively loose in a MSS scenario (see Section II), source models can also incorporate different degrees of specificity when it comes to describing the sources in the mix. Consider for example the harmonicpercussive separation task: harmonic sources are characterized as horizontal components in the spectrogram, while percussive sources are characterized as time-localized vertical ones. These particularities of the sources can also be understood as harmonic sources exhibiting continuity over time, and percussive sources exhibiting continuity over frequency. Music separation models such as Kernel Additive Models (KAM) particularly exploit local features observable in music spectrograms such as continuity, repetition (at both short time scales and longer scales such as repeating verses and choruses) and common fate [START_REF] Liutkus | Kernel additive models for source separation[END_REF]. In order to estimate a music source at a given time-frequency point, KAM involves selecting a set of time-frequency bins, which, given the nature of the target source, e.g., percussive, harmonic or vocals, are likely to be similar in value.

This set of time-frequency bins is termed a proximity kernel. An example of a vertical proximity kernel used to extract percussive sounds is shown in Figure 6a. Here, a set of adjacent frequency bins are chosen since percussion instruments tend to exhibit stable ridges across frequency. The vertical kernel is also positioned on a percussive hit seen in the spectrogram of the mixture in Figure 6, where the middle time-frequency bin (k, n) is the one to be estimated. In the case of sustained pitched sounds which tend to have similar values in adjacent time frames, a suitable proximity kernel consists of adjacent time-frames across a single frequency (see the horizontal kernel in Figure 6a). KAM approaches assume that interference due to other sources than the target source is sparse, so time-frequency bins with interference are regarded as outliers. To remove these outliers and to obtain an estimate of the target source, the median amplitude of the bins in the proximity kernel is taken as an estimate of the target source at a given timefrequency bin. The median acts as a statistical estimator robust to outliers in energy. Once the proximity kernels have been chosen for each of the sources to be separated, separation proceeds iteratively in the manner described in Section III. KAM is a generalization of previous work on vocal separation and harmonic-percussive separation [START_REF] Rafii | REpeating Pattern Extraction Technique (REPET): A simple method for music / voice separation[END_REF], [START_REF] Fitzgerald | Harmonic/percusssive separation using median filtering[END_REF], and has demonstrated considerable utility for these purposes [START_REF] Liutkus | Kernel additive models for source separation[END_REF].

Spectrogram Factorization Models: Another group of musical source models are based on spectrogram factorization techniques. The most common of those is Non-Negative Matrix Factorization (NMF). NMF attempts to factorize a given non-negative matrix into two nonnegative matrices [START_REF] Lee | Algorithms for non-negative matrix factorization[END_REF]. For the purpose of musical source separation, NMF can be applied to the non-negative magnitude spectrogram of the mix M . The goal is to factorize M into a product M ≈ W H of a matrix of basis vectors W , which is a dictionary of spectral templates modeling spectral characteristics of the sources, and a matrix of time activations H. Figure 6b shows an example of a series of spectral templates and their corresponding time activations. One of the spectral templates W 1 , and its corresponding time activations H 1 , are also displayed next to the mixture spectrogram. Peaks in the time activations represent the time instances where a given spectral template has been identified within the mix. Note, for example, that the peaks in the time activation H 1 coincide with the percussive hits (vertical structures) in the spectrogram. The factorization task is solved as an optimization problem where the divergence (or reconstruction error) between M and W H is minimized using common divergence measures D such Kullback-Leibler, or Itakura-Saito: min W,H≥0 D(M W H). Many variants of NMF algorithms have been proposed for the purpose of musical source separation, including methods with temporal continuity constraints [START_REF] Virtanen | Monaural sound source separation by nonnegative matrix factorization with temporal continuity and sparseness criteria[END_REF], multichannel separation models [START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF], score-informed separation [START_REF] Ewert | Score-informed source separation for musical audio recordings: An overview[END_REF], among others [START_REF] Smaragdis | Static and dynamic source separation using nonnegative factorizations: A unified view[END_REF].

The NMF-based methods presented above typically assume that the spectrogram for all sources may be well approximated through low-rank assumptions, i.e. as the combination of only a few spectral templates. While this assumption is often sufficient for instrumental sounds, it generally falls short on modeling vocals, which typically exhibit great complexity and require more sophisticated models. In this respect, an NMF variant that has been particularly successful in separating the singing voice uses a source-filter model to represent the target vocals [START_REF] Durrieu | Source/filter model for unsupervised main melody extraction from polyphonic audio signals[END_REF]. The idea behind such models is that the voice signal is produced by an excitation that depends on a fundamental frequency (the source), while the excitation is then filtered by the vocal tract or by spectral shapes related to timbre (the filter). A dictionary of source spectral templates and a matrix of filter spectral shapes are used in this model within an Expectation-Maximization framework.

Some models for the singing voice are based on the observation that spectrograms of vocals are usually sparse, composed of strong and well-defined harmonics, and mostly zero elsewhere (as seen in Figure 2). In this setting, the observed mixture is assumed to be equal to the accompaniment for a large portion of the mixture spectrogram entries. This is the case of Robust Principal Component Analysis (RPCA) [START_REF] Huang | Singing-voice separation from monaural recordings using robust principal component analysis[END_REF] which does not rely on over-constraining low-rank assumptions on the vocals, and in turn, uses the factorization only for the accompaniment, leaving the vocals unconstrained. Many elaborations on this technique have been proposed. For instance, [START_REF] Chan | Vocal activity informed singing voice separation with the ikala dataset[END_REF] incorporates voice activity detection in the separation process, allowing the vocals to be inactive in segments of the signal and thus, strongly boosting performance.

Sinusoidal Models: Another strand of research for MSS models focuses on sinusoidal modeling. This method works under the premise that any music signal can be approximated by a number of sinusoids with time-varying frequencies and amplitudes [START_REF] Serra | Musical Sound Modeling with Sinusoids plus Noise[END_REF]. Intuitively, sinusoidal modeling offers a clear representation of music signals, which in most cases is composed of a set of fundamental frequencies and their associated harmonic series. If the pitches present in the target source, as well as the spectral characteristics of the associated harmonics of each pitch, are known or can be estimated, sinusoidal modeling techniques can be very effective for separation of harmonic sources. However, given the complexity of the model and the very detailed knowledge of the target source required to successfully create a realistic representation, the use of sinusoidal modeling techniques for MSS has been limited. Sinusoidal modeling techniques have been proposed for harmonic sound separation [START_REF] Virtanen | Separation of harmonic sound sources using sinusoidal modeling[END_REF] and harmonic-percussive separation [START_REF] Cano | Phase-based harmonic/percussive separation[END_REF].

Deep Neural Network Models: Historically, MSS research has focused heavily on the use of model-based estimation that enforced desired properties on the source spectrograms. However, if the properties required by the models are not present, separation quality can rapidly degrade.

More recently, the use of Deep Neural Networks (DNNs) in MSS has rapidly increased.

In contrast to the above approaches which require explicit models of the source for processing, methods based on DNNs take advantage of optimization techniques to train source models in a supervised manner [START_REF] Huang | Joint optimization of masks and deep recurrent neural networks for monaural source separation[END_REF], [START_REF] Uhlich | Deep neural network based instrument extraction from music[END_REF], [START_REF] Nugraha | Multichannel audio source separation with deep neural networks[END_REF], i.e. using datasets where both the mix and the isolated sources are available. As depicted in Figure 7, most supervised DNN-based methods take magnitude spectrograms of the audio mix as inputs, optionally also incorporating some further context cues.

The targets are set either as the magnitude spectrograms S j of the desired sources (also shown in Figure 7), or as their separating masks (either soft masks or binary masks as described in of the musical sources: they are directly inferred by the network. Secondly, the DNN topology is of great significance, both for the training capabilities of the network and as a means of incorporating prior knowledge in the system. The earliest DNN-based approaches for MSS consisted of taking a given frame of the spectrogram, as well as additional context frames as input, and outputting the corresponding frame for each of the targets. These systems mostly consisted of fully-connected networks (FCN).

However, given the large size of music spectrograms, the resulting FCNs contained a large number of parameters. This restricted the use of temporal context in such networks to less than one second [START_REF] Uhlich | Deep neural network based instrument extraction from music[END_REF]. Therefore, these networks were typically applied on sliding windows of the mixture spectrogram. To overcome this limitation, both recurrent neural networks (RNNs) and convolutional neural networks were investigated as they offered a principled way to learn dynamic models. RNNs are similar to FCNs, except that they apply their weights recursively over an input sequence, and can process sequential data of arbitrary length. This makes such dynamical models ideally suited for the processing of spectrograms of different tracks with different durations, while still modeling long term temporal dependencies. The most commonly used setting for DNN-based separation today is to fix the number and the nature of the sources to separate beforehand. For example, we may learn a network able to separate drums, vocals and bass from a mixture. However, having MSS systems which can dynamically detect and separate an arbitrary number of sources is an open challenge, and deep clustering [START_REF] Hershey | Deep clustering: Discriminative embeddings for segmentation and separation[END_REF] methods represent a possible approach to designing such systems.

A limitation of many DNN models is that they often use Mean Squared Error (MSE) as a cost function. While MSE results in a well-behaved stochastic gradient optimization problem, it also poorly correlates with perceived audio quality. This is the reason why the design of more appropriate cost functions is also an active research topic in MSS [START_REF] Nugraha | Multichannel audio source separation with deep neural networks[END_REF].

Finally, a crucial limitation of current research for DNN-based MSS is the need for large amounts of training data. The largest MSS multi-track public dataset today is MUSDB 2 , which comprises 10 h of data 3 . However, this is still small compared to existing speech corpora comprising hundreds or thousands of hours. The main difficulty in creating realistic multi-track data sets for MSS, comes from the fact that individual recordings for each instrument in a mixture 2 MUSDB: https://doi.org/10.5281/zenodo.1117372 3 Refer to the accompanying website for more information about available data sets: https://soundseparation.songquito. com/evaluationOf.htm are rarely available. Conversely, if individual recordings of instruments are available, the process of creating a realistic music mixture with them is time-consuming and very costly (as outlined in Section II). As a result, designing DNN architectures for MSS still requires fundamental knowledge about the sources to be separated, their input and output representations, as well as the use of suitable signal processing techniques and post-processing operations to further improve the recovered sources.

V. EVALUATION OF MUSICAL SOURCE SEPARATION MODELS Once an MSS system has been developed, its performance needs to be evaluated. All MSS approaches invariably introduce unwanted artifacts in the separated sources. These artifacts can be caused by mismatches between the models and the sources, musical noise that appears whenever rapid phase or spectral changes are introduced in the estimates (for example, when using binary masking), reconstruction errors in re-synthesis, as well as phase errors in the estimates. Quality evaluation in MSS systems is however non-trivial. As musical signals are intended to be heard by human listeners, it is therefore reasonable that evaluation should be based on subjective listening tests such as the Multiple Stimulus with Hidden Reference and Anchors (MUSHRA) test [START_REF]Recommendation BS.1534-3: Method for the subjective assessment of indermediate quality levels of coding systems[END_REF]. However, listening tests are time-consuming and costly, requiring human volunteers to undertake the tests, and need certain expertise to be conducted properly. This has made them an infrequent choice for separation quality evaluation.

In an attempt to reduce the efforts of evaluating MSS systems, objective quality metrics have been proposed in the literature. These include the Blind Source Separation Evaluation (BSS Eval) Toolbox [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF], based on non-perceptual energy ratios, and the Perceptual Evaluation methods for Audio Source Separation (PEASS) toolkit [START_REF] Emiya | Subjective and Objective Quality Assessment of Audio Source Separation[END_REF], which aimed to map results obtained via listening tests to create metrics. However, the validity of these metrics has been questioned in recent years, as the values obtained with them do not seem to correlate with perceptual measures obtained via listening tests [START_REF] Cano | Evaluation of quality of sound source separation algorithms: Human perception vs quantitative metrics[END_REF].

As of today, given the lack of an unified and perceptually valid strategy for MSS quality evaluation, most algorithm development is still conducted using BSS Eval; however, it is highly recommended to conduct a final listening test to verify the validity of separation results. As a final note, the source separation community runs a regular Signal Separation Evaluation Campaign (SiSEC) [START_REF] Liutkus | The 2016 signal separation evaluation campaign[END_REF] including musical audio source separation tasks. SiSEC raises the visibility and importance of evaluation, and acts as a focus for discussions on evaluation methodologies.

MusicalFigure 1 :

 1 Figure 1: Representation of a music mixture in the time-frequency domain. The dominant musical source in each time-frequency bin is displayed with a different color.

Figure 2 :

 2 Figure 2: Magnitude spectrogram of four example music signals: vocals (left), drums (mid-left), bass (mid-right) and acoustic guitar (right). The horizontal axis represents time in seconds, and the vertical axis frequency in Hz.

Figure 3 :

 3 Figure 3: Common music recording and mixing setup. In most cases, musical content is combined and mixed into a stereo signal using a Digital Audio Workstation (DAW).

Figure 4 .Figure 4 :

 44 Figure 4: Common MSS work flow: source models are obtained from the spectrogram of the audio mix. This is often done in an iterative manner where iter represents the total number of iterations and i the iteration index.

Figure 5 :

 5 Figure 5: Illustration of standard Pan Law. The position of the source j = 1 (keyboard), source j = 2 (voice), and source j = 3 (guitar) is defined by the angle φ j which is always measured with respect to the first channel. Also shown are individual (colored) source position histograms and the mixture spatial histogram (black)

  where the subscript indicates channel number. If there is little time-frequency overlap between the sources, then a single source j will contribute most of the energy at a single point in the time-frequency representations, and so R(k, n) ≈ cos φ j / sin φ j . Given that R(k, n) only depends on φ j under this assumption, it can therefore be used to estimate a panning angle for each time-frequency point. By plotting an energy-weighted histogram of these angle estimates, a mixture spatial histogram as the one shown in Figure5can be obtained. A peak in this histogram then provides an estimate of the panning angle φ j of a given source. All time-frequency points with an angle close to that of the jth peak are assigned to source j. Then, recovery of an estimate of source j can be done by means of binary masking. DUET, ADRess and PROJET all estimate histograms of energy vs angle. The main difference between the techniques lies in how these histograms are generated and used, and in the type of masking used. Both DUET and ADRess require peak picking from a mixture spatial histogram and use binary masking.

Figure 6 :

 6 Figure 6: Example of different models used in MSS: (a) Proximity kernels used for harmonic-percussive separation within a Kernel Additive Modeling (KAM) approach, (b) Spectral templates W and time activations H within a Non-Negative Matrix Factorization approach.

  Section III). Regardless of the inputs and targets used, DNN methods work by training the parameters of non-linear functions to minimize the reconstruction error of the chosen outputs (spectrograms or masks) based on the inputs (audio mixes).The models obtained from a neural network depend on two core aspects. Firstly, the type and quantity of the data used for training is of primary importance. To a large extent, representative training data overcomes the need for explicitly modeling the underlying spectral characteristics

Figure 7 :

 7 Figure 7: DNN architecture for MSS: Mixture magnitude spectrograms are set as inputs, and source magnitude spectrogram of the desired source S j are set as the targets.

from previous approaches with data-driven DNN approaches will allow future researchers to overcome current limitations and challenges in musical source separation. VII. ACKNOWLEDGMENTS Mark D. Plumbley is partly supported by grants EP/L027119/2 and EP/N014111/1 from the UK Engineering and Physical Sciences Research Council, and European Commission H2020 "AudioCommons" research and innovation grant 688382. A. Liutkus and F. Stöter are partly supported by the research programme "KAMoulox" (ANR-15-CE38-0003-01) funded by ANR, the French State agency for research.

VI. FUTURE RESEARCH DIRECTIONS

Musical Source Separation is a challenging research area with numerous real-world applications. Due to both the nature of the musical sources and the very particular processes used to create music recordings, MSS has many unique features and problems which make it distinct from other types of source separation problems. This is further complicated by the need to achieve separations which sound good in a perceptual sense.

While the quality of MSS has greatly improved in the last decade, several critical challenges remain. Firstly, audible artifacts are still produced by most algorithms. Possible research directions to reduce artifacts include the use of phase retrieval techniques to estimate the phase of the target source, the use of feature representations that better match human perception, allowing models to concentrate on the parts of the sounds that are most relevant for human listeners, and the exploration of MSS systems that model the signal directly in time domain as waveforms.

Many remaining issues in MSS come from the fact that systems are often not flexible enough to deal with the richness of musical data. For example, it is typically assumed that the actual number of musical sources in a given recording is known. However, this assumption can lead to problems when the number of sources changes over the course of the training procedure. Another issue comes with the separation of sources from the same or similar instrument families, such as the separation of multiple singing voices or violin ensembles.

As previously mentioned, a unified, robust and perceptually valid MSS quality evaluation procedure does not yet exist. Even while new alternatives for evaluation have been explored in recent years [START_REF] Ward | BSS Eval or PEASS? Predicting the perception of singing-voice separation[END_REF], listening tests remain the only reliable quality evaluation method to date. The design of new MSS quality evaluation procedures that are applicable for a wide range of algorithms and musical content, will require large research efforts including large-scale listening experiments, common datasets, and the availability of a wide range of MSS algorithms for use in development.

Additionally, a better understanding of how DNN-based techniques can be exploited for music separation is still needed. In particular, we need better training schemes to avoid over-fitting, and architectures suitable for music separation. The inclusion of perceptually-based optimization schemes and availability of training data are also current challenges in the field.

Recent developments in the area of DNNs have introduced a paradigm shift in MSS research, with an increasing focus on data-driven models. Nonetheless, previous techniques have achieved considerable success in tackling MSS problems. We believe that combining the insights gained


Dominic Ward

Russell D Mason
email: r.mason@surrey.ac.uk

Chungeun Kim

Fabian-Robert Stöter

Antoine Liutkus

Mark D Plumbley

Cvssp

SISEC 2018: STATE OF THE ART IN MUSICAL AUDIO SOURCE SEPARATION -SUBJECTIVE SELECTION OF THE BEST ALGORITHM

The Signal Separation Evaluation Campaign (SiSEC) is a large-scale regular event aimed at evaluating current progress in source separation through a systematic and reproducible comparison of the participants' algorithms, providing the source separation community with an invaluable glimpse of recent achievements and open challenges. This paper focuses on the music separation task from SiSEC 2018, which compares algorithms aimed at recovering instrument stems from a stereo mix. In this context, we conducted a subjective evaluation whereby 34 listeners picked which of six competing algorithms, with high objective performance scores, best separated the singing-voice stem from 13 professionally mixed songs. The subjective results reveal strong differences between the algorithms, and highlight the presence of song-dependent performance for state-of-the-art systems. Correlations between the subjective results and the scores of two popular performance metrics are also presented.

INTRODUCTION

High-quality separation of the singing voice from accompanying instruments is an important yet difficult task serving many applications, including soloing, karaoke, remixing and upmixing music [START_REF] Roma | Music remixing and upmixing using source separation[END_REF], and even for studying music production techniques [START_REF] Ward | Estimating the loudness balance of musical mixtures using audio source separation[END_REF]. Despite the plethora of musical applications that audio source separation affords, separation quality is often far from perfect, introducing distortions and artefacts that degrade the perceived quality of the extracted source. Another issue is interference, whereby the unwanted sources remain present to some extent. Nevertheless, notable improvements in signal separation have been made in recent years, particularly due to the surge of deep-learning techniques which have proved to outperform traditional signal processing methods, as documented by the Signal Separation Evaluation Campaign (SiSEC) [START_REF] Stöter | The 2018 signal separation evaluation campaign[END_REF].

SiSEC was initiated to compare the performance of separation systems on a voluntary and community basis by preparing, organising, deploying and evaluating signal separation challenges that target different domains [START_REF] Stöter | The 2018 signal separation evaluation campaign[END_REF]. The MUS(ic) task of SiSEC is designed to assess systems that extract musical instruments from popular music mixes. It has gained rapid interest from algorithm proponents over recent years, due to the development of high-quality and freely-available audio datasets, as well as its unified platform for training, testing, and submitting source separation algorithms for fair comparison. Separation quality is typically evaluated using objective metrics that quantify distortions between the target and estimated sources; the most common being the blind source separation evaluation (BSS Eval) toolkit [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF], followed by the Perceptual Evaluation methods for Audio Source Separation (PEASS) toolkit [START_REF] Emiya | Subjective and objective quality assessment of audio source separation[END_REF]. Although these metrics show some predictive ability for singing-voice separation quality [START_REF] Ward | BSS EVAL or PEASS? Predicting the Perception of Singing-Voice Separation[END_REF], generalisation to other sources and algorithms has been questioned [START_REF] Cano | Evaluation of quality of sound source separation algorithms: Human perception vs quantitative metrics[END_REF], indicating that further refinement is warranted in order for them to be used for general purpose performance measurement. However, for most applications, subjective quality of the separation result is key; therefore it is imperative that auditory assessments are conducted to measure quality as judged by human listeners.

Accordingly, we have conducted a perceptual experiment using the most recent set of songs and separation algorithms in the SiSEC 2018 MUS task dataset. This paper presents the results of the experiment, with the goal of identifying the separation algorithms that can most successfully extract the singing-voice from a range of song mixtures.

SUBJECTIVE ASSESSMENT

MUSDB18

The MUSDB18 dataset [START_REF] Rafii | The MUSDB18 corpus for music separation[END_REF] is a freely-available set of 150 professionally mixed songs from a range of genres, each comprising four stereo sources (bass, drums, vocals and a group of other instruments) that sum to produce typical song mixes. For the MUS task, MUSDB18 was divided into 100 songs for training and 50 songs for testing, and 30 systems were evaluated using BSS Eval metrics applied to each of the estimated sources [START_REF] Stöter | The 2018 signal separation evaluation campaign[END_REF].

Selecting the stimuli

For the listening test, we limited the selection of test stimuli to the singing-voice (vocals) from the test portion of MUSDB18. As a starting point, we first selected one algorithm from participants with multiple submissions, and then selected the six algorithms (see Table 1) with the highest median source-to-distortion ratios (SDR), a global performance metric provided by the BSS Eval v4 toolkit [START_REF] Stöter | The 2018 signal separation evaluation campaign[END_REF]. The songs were selected with the goal of obtaining content of varying separability. Among the 50 test songs, 10 were selected to yield a wide range of SDR values; a further three were selected from an informal pilot experiment as containing atypical artefacts. The original vocals were included as reference material for listeners to compare against. Stimuli were segmented into seven-second portions where all instruments were most active [START_REF] Bittner | MedleyDB: A multitrack dataset for annotation-intensive MIR research[END_REF], and then loudness normalised according to ITU-R BS.1770 [START_REF] Bs | Algorithms to measure audio programme loudness and true-peak audio level[END_REF]. Recurrent inference algorithm with masker and denoiser architecture [START_REF] Mimilakis | Monaural singing voice separation with skip-filtering connections and recurrent inference of timefrequency mask[END_REF] Table 1: Overview of the six deep-learning based musical audio source separation algorithms evaluated, in descending order of median SDR for the vocals of the MUSDB18 test set.

Procedure

Participants of the listening assessment were delegates of the 14th LVA ICA conference 1 , and the assessments were undertaken in edit rooms with good acoustic isolation and minimal visual distractions. The stimuli were reproduced over professional-grade studio headphones. Forty-two listeners undertook at least part of the experiment at the conference, and two listeners completed the full experiment online, giving a total of 44 participants. Of these, 34 completed the full experiment.

For each of the 13 songs, participants were asked to select one of six extracted vocals (each corresponding to one of the algorithms listed in Table 1) that they perceived to be most similar to the original vocals. A short training stage involving audio examples informed listeners about the music source separation task and explained what audible deviations should be taken into account as promoting dissimilarity from the original vocals, e.g. the presence of other instruments and artificial noise. The presentation order of both songs (on separate pages) and algorithms (within each page) was randomised for each listener. Participants were able to end the session at any time. 2

RESULTS

The analysis was undertaken using the data from the 34 participants that completed the full experiment. The count data, obtained by summing the number of times each stimulus was selected, were subject to a Chi-Square analysis to test the null-hypothesis that all algorithms are equally likely to be selected for a given song (of 13). If the probability of obtaining a Chi-Square statistic at least as extreme as the one calculated 1 http://cvssp.org/events/lva-ica-2018/ 2 The test is available at https://cvssp.github.io/sisec-mus-18-pick-the-best/ was less than 0.05/13, the null-hypothesis was rejected. The standardised residuals (difference between observed count and expected count under the null-hypothesis, divided by square-root of the expected count) were then used to identify algorithms with counts that systematically differed from the expected value, based on a z-score threshold of 2.64 (corresponding to α = 0.05/6). The results are shown in Figure 1. The blue triangles highlight the algorithms with higher counts than expected if the counts were uniform within each song, suggesting that listeners perceived specific separated vocals as being more similar to the reference vocals. Of the 34 participants, none selected MDL1 in any of the songs, and STL1 received a highest count of only 3. TAU1 was selected most often for seven songs, and joint-top in two. TAK2 and UHL3 were selected most often for two and three songs, respectively, both joint-top with TAU1 for one song. Interestingly, the leading algorithms UHL3, TAK2 and TAU1 were rarely selected for the Sunshine Garcia Band track, where JY3 was chosen by 16 of the 34 listeners (flagged as being systematically higher than the expected count).

This difference in the results for some songs is highlighted by the Spearman correlation ρ (in green; Figure 1) between the results across separation algorithms for each song with the sum of the results across songs. The Sunshine Garcia Band track has a correlation of only 0.49, with the Ben Carrigan, Buitraker, and James Elder & Mark M Thompson tracks also showing a weaker correlation with the overall results.

Comparison with objective measures

The subjective results were compared with the SDR metric from BSS Eval v3 [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF] and with the Overall Perceptual Score (OPS) from PEASS [START_REF] Emiya | Subjective and objective quality assessment of audio source separation[END_REF]: both are intended to reflect the over-all separation quality. The loudness-normalised stimuli were input to both metrics, using the vocal accompaniment as the true interfering source. Figure 2 shows the (song-wise) Spearman rank correlation between the observed frequency counts of the test stimuli, and the SDR and OPS measures. The plot suggests that stimuli with higher SDR are more likely to be selected; for seven songs the most frequently selected algorithm (or joint-top) had the highest SDR. OPS is less correlated with the subjective results, including some moderate negative correlations; the algorithm with the highest OPS was most frequently selected in only two songs.

CONCLUSIONS

A listening experiment was performed to identify source separation algorithms that can most successfully extract the singing-voice from a range of song mixtures (defined as the separated vocals perceived as most similar to the original vocals), and whether the success is accurately predicted by the objective measures.

The results show that for many of the songs the TAU1 algorithm was most successful, but there was a certain amount of dependence on song, suggesting that specific algorithms may need further enhancement to generalise across genres. For seven songs, the algorithms with notably high counts also had the highest SDR value (BSS Eval toolkit). Spearman correlations between subjective results and SDR were consistently higher than those with OPS (PEASS toolkit), which included some moderate negative correlations. These results indicate that some refinement of objective measures, primarily OPS, is required to accurately reflect the similarity judgements of human listeners.

Figure 1 :

 1 Figure 1: Subjective results indicating the number of times each algorithm (x-axis) was selected for each song (y-axis), shown in greyscale. Algorithms in ascending order, from left to right, of average count. Blue triangles identify stimuli selected more frequently than expected under the null-hypothesis. Values of ρ indicate the Spearman rank correlation between a given song and the combined results.

Figure 2 :

 2 Figure 2: Spearman correlation between the observed frequency counts of the test stimuli, and the corresponding OPS/SDR values, separated by song.

Proceedings of the 4 th Workshop on Intelligent Music Production, Huddersfield, UK, 14 September 2018