Sylvain Marchand
email: sylvain.marchand@univ-brest.fr

Roland Badeau

Cléo Baras

Laurent Daudet

Dominique Fourer

Laurent Girin

Stanislaw Gorlow

Antoine Liutkus

Jonathan Pinel

Gaël Richard

Nicolas Sturmel

Shuhua Zang

DReaM: a novel system for joint source separation and multi-track coding

Active listening consists in interacting with the music playing, has numerous applications from pedagogy to gaming, and involves advanced remixing processes such as generalized karaoke or respatialization. To get this new freedom, one might use the individual tracks that compose the mix. While multi-track formats loose backward compatibility with popular stereo formats and increase the file size, classic source separation from the stereo mix is not of sufficient quality. We propose a coder/decoder scheme for informed source separation. The coder determines the information necessary to recover the tracks and embeds it inaudibly in the mix, which is stereo and has a size comparable to the original. The decoder enhances the source separation with this information, enabling active listening.

INTRODUCTION

Active listening of music is both an artistic and technological topic of growing interest, that consists in giving to the music consumer the possibility to interact in real time with the music, e.g. to modify the elements, the sound characteristics, and the structure of the music while it is played. This involves advanced remixing processes such as general-ized karaoke (muting any musical element, not only the lead vocal track), adding effects on selected instruments, respatialization and upmixing. The applications are numerous, from learning/teaching of music to gaming, through new creative processes (disc jockeys, live performers, etc.).

To get this new freedom, a simple solution would be to give access to the individual tracks that compose the mix [START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF], by storing them into some multi-track format. This approach has two main drawbacks: First, it leads to larger multi-track files. Second, it yields files that are not compatible with the prevailing stereo standards.

Another solution is to perform blind separation of the sources from the stereo mix. The problem is that even with state-of-the-art blind source separation techniques the quality is usually insufficient and the computation is heavy [START_REF] Comon | Handbook of blind source separation -Independent component analysis and applications[END_REF][START_REF] Ozerov | Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation[END_REF].

In the DReaM project, we propose a system designed to perform source separation and accurately recover the separated tracks from the stereo mix. The system consists of a coder and a decoder.

The coder is used at the mixing stage, where the separated tracks are known. It determines the information necessary to recover the tracks from the mix and embeds it in the mix. In the case of PCM, this information is inaudibly hidden in the mix by a watermarking technique [START_REF] Pinel | A high-capacity watermarking technique for audio signals based on MDCT-domain quantization[END_REF]. In the case of compressed audio formats, it can be embedded in a dedicated data channel or directly in the audio bitstream. With a legacy system, the coded stereo mix can be played and sounds just like the original, although some information is now included in it. Apart from this backward compatibility with legacy systems, an important point is the fact that the file size stays comparable to the one of the original mix, since the additional information sent to the decoder is rather negligible. This decoder performs source separation of the mix with parameters given by the additional information. This Informed Source Separation (ISS) approach [START_REF] Knuth | Informed source separation: a Bayesian tutorial[END_REF] permits to produce good separated tracks, thus enabling active listening applications.

The paper is organized as follows. Section 2 presents the DReaM project: its fundamentals and target applications. Section 3 introduces the mixing models we are considering, Section 4 describes the separation/unmixing methods we have developed so far in the project, and Section 5 illustrates the working prototypes available for demonstration purposes. Finally, Section 6 draws some conclusions and opens new perspectives.

THE DREAM PROJECT

DReaM1 is a French acronym for "le Disque Repensé pour l'écoute active de la Musique", which means "the disc thought over for active listening of music". This is the name of an academic project with industrial finality, funded by the French National Research Agency (ANR). The project members are academics (LaBRI -University of Bordeaux, GIPSA-Lab -Grenoble INP, LTCI -Telecom ParisTech, ESPCI -Institut Langevin) together with iKlax Media, a company for interactive music that contributed to the Interactive Music Application Format (IMAF) standard [START_REF]Information technology -Multimedia application format (MPEG-A) -Part 12: Interactive Music Application Format[END_REF]. The Lab-STICC -University of Brest will join the consortium, as the new affiliation of the first author and coordinator of the project. The Grenoble Innovation Alpes (GRAVIT) structure leads the technology transfer aspects of the project.

The origin of the project comes from the observation of artistic practices. More precisely, composers of acousmatic music conduct different stages through the composition process, from sound recording (generally stereophonic) to diffusion (multiphonic). During live interpretation, they interfere decisively on spatialization and coloration of pre-recorded sonorities. For this purpose, the musicians generally use a(n un)mixing console to upmix the musical piece being played from an audio CD. This requires some skills, and imposes musical constraints on the piece. Ideally, the individual tracks should remain separated. However, this multi-track approach is hardly feasible with a classic (stereophonic) audio CD.

Nowadays, the public is more eager to interact with the musical sound. Indeed, more and more commercial CDs come with several versions of the same musical piece. Some are instrumental versions (e.g. for karaoke), other are remixes. The karaoke phenomenon gets generalized from voice to instruments, in musical video games such as Rock Band 2 . But in this case, to get the interaction the user has to buy the video game, which includes the multi-track recording.

Yet, the music industry seems to be reluctant to releasing the multi-track versions of big-selling hits. The only thing the user can get is a standard CD, thus a stereo mix, or its digital version available for download or streaming.

Project Goals and Objectives

Generally speaking, the project aims at solving an inverse problem, to some quality extent, at the expense of additional information. In particular, an example of such an inverse problem can be source separation: recovering the individual source tracks from the observed mix.

On the one hand coding the solution (e.g., the individual tracks and the way how to combine them) can bring high quality, but with a potentially large file size, and a format not compatible with existing stereo formats.

On the other hand the blind approach (without information) can produce some results, but of insufficient quality for demanding applications (explained below). Indeed, the mixture signals should be realistic music pieces, ideally of professional quality, and the separation should be processed in real-time with reasonable computation costs, so that real-time sound manipulation and remixing can follow. The blind approach can be regarded as an estimation without information, while coding can be regarded as using information (from each source) without any estimation (from the mix).

The informed approach we propose is just in between these two extremes: getting musically acceptable results with a reasonable amount of additional information. The problem is now to identify and encode efficiently this additional information [START_REF] Ozerov | Informed source separation: source coding meets source separation[END_REF]. Remarkably, ISS can thus be seen both as a multi-track audio coding scheme using source separation, or as a source separation system helped by audio coding. This approach addresses the source separation problem in a coder/decoder configuration. At the coder (see Fig. 1), the extra information is estimated from the original source signals before the mixing process and is inaudibly embedded into the final mix. At the decoder (see Fig. 2), this information is extracted from the mix and used to assist the separation process. The residuals can be coded as well, even if joint coding is more efficient (not on the figures for the sake of simplicity, see Section 4 instead).

So, a solution can be found to any problem, thanks to the additional information embedded in the mix. The idea was to inaudibly embed (using a highcapacity watermarking technique, see [START_REF] Pinel | A high-capacity watermarking technique for audio signals based on MDCT-domain quantization[END_REF]) in the audio track some information enabling to some extent the musical decomposition, that is the inversion of the music production chain: dynamics decompression, source separation (unmixing), deconvolution, etc.

With a standard CD player, one would listen to the fixed mix. With an active player however, one could modify the elements and the structure of the audio signal while listening to the music piece.

. . . Towards Enhanced Compressed Mix

Now that the music is getting all digital, the consumer gets access to audio files instead of physical media. Although the previous strategy also applies to the (PCM) files extracted from the audio CD, most audio files are distributed in lossy compressed formats (e.g. ACC, MP3, or OGG).

We are currently working on the extension of the proposed techniques to compressed mixes, based on encouraging preliminary results [START_REF] Girin | Informed audio source separation from compressed linear stereo mixtures[END_REF]. The extra information can then either be included in some ancillary data, or be embedded (almost) inaudibly in the audio bitstream itself. The latter option is much more complicated, since lossy but perceptually lossless coding aims at removing inaudible information. Both coders (perceptual and informed) have then to be merged, to maintain a certain information tradeoff.

Applications

Active listening [START_REF] Lepain | Recherche et applications en informatique musicale, chapter Écoute interactive des documents musicaux numériques[END_REF] consists in performing various operations that modify the elements and structure of the music signal during the playback of a piece.

This process, often simplistically called remixing, includes generalized karaoke, respatialization, or application of individual audio effects (e.g., adding some distortion to an acoustic guitar).

The goal is to enable the listener to enjoy freedom and personalizing of the musical piece through various reorchestration techniques.

Alternatively, active listening solutions intrinsically provide simple frameworks to the artists to produce different versions of a given piece of music. Moreover, it is an interesting framework for music learning/teaching applications.

Respatialization

The original application was to let the public experience the freedom of composers of electroacoustic music during their live performances: moving the sound sources in the acoustic space. Although changing the acoustical scene by means of respatialization is a classic feature of contemporary art (electroacoustic music), and efforts have been made in computer music to bring this practice to a broader audience [START_REF] Pachet | A constraint-based temporal music spatializer[END_REF], the public seems just unaware of this possibility and rather considered as passive consumers by the music industry. However, during the public demonstrations of the DReaM project, we felt that the public was very reactive to this new way of interacting with music, to personalize it, and was ready to adopt active listening, mostly through musical games.

Generalized Karaoke

Games, or "serious" games, can be very useful for music learning/teaching applications. The generalized karaoke application is the ability to suppress any audio source, either the voice (classic karaoke) or any instrument ("music minus one"). The user can then practice singing or playing an instrument while being integrated in the original mix and not a cover song.

Note that these two applications (respatialization and generalized karaoke) are related, since moving a source far away from the listener will result in its muting, and reciprocally the ability to mute sources can lead to the monophonic case (the spatial image of a single source isolated) where respatialization is much easier (possible to some extent even without recovering the audio object from this spatial image).

ISS vs. SAOC

The DReaM project turns out to be close in spirit to the Remix system of Faller et al. [START_REF] Faller | Enhancing stereo audio with remix capability[END_REF]. We are also conscious that leaving artistic applications on uncompressed signals to more commercial applications on compressed formats now places the DReaM project next to MPEG Spatial Audio Object Coding (SAOC) [START_REF] Engdegård | MPEG spatial audio object coding -the ISO/MPEG standard for efficient coding of interactive audio scenes[END_REF], derived from the Spatial Audio Coding (SAC) approach of MPEG Surround (MPS) [START_REF] Herre | MPEG surround -the ISO/MPEG standard for efficient and compatible multichannel audio coding[END_REF] and pioneering works on parametric multi-channel joint audio coding [START_REF] Faller | Parametric joint-coding of audio sources[END_REF].

In MPS [START_REF] Herre | MPEG surround -the ISO/MPEG standard for efficient and compatible multichannel audio coding[END_REF], perceptually relevant spatialization parameters such as interchannel loudness differences (ILD), interchannel time differences (ITD), and interchannel cross-correlations (ICC) are extracted from the multi-channel signal at the encoder. These parameters are transmitted to the decoder in addition to a mono/stereo downmix of the multi-channel signal. At the decoder, those parameters are used to respatialize the multi-channel audio scene from the downmix signal.

This approach has been extended later in SAOC [START_REF] Engdegård | MPEG spatial audio object coding -the ISO/MPEG standard for efficient coding of interactive audio scenes[END_REF] from the audio channels of the spatial image (acoustic scene) to audio objects (sound sources), opening new perspectives for active listening of music.

However, it must be noted that in contrast to SAC/SAOC, the goal of the ISS methods we propose (see Section 4 below) is from the beginning to completely separate the source signals and not only to resynthesize/respatialize the audio scene.

In particular, the spatialization parameters in SAC/SAOC are used to "redistribute" the content of spectral subbands of the downmix signal across the different output channels, but they cannot separate the contribution of two different sources that are present within the same subband (hence the sources are "respatialized together" and not clearly separated; e.g. see [START_REF] Faller | Parametric joint-coding of audio sources[END_REF]). In contrast, the separation of two overlapping sources is precisely one of the original goals of our ISS methods. Note that some aspects of SAOC, notably the Enhanced SAOC option [START_REF] Falch | Spatial audio object coding with enhanced audio object separation[END_REF], tend to fill this gap by encoding additional information that achieves a (much) better separation of the audio objects. But this is done through separately encoding the residuals, which may be shown to be sub-optimal in terms of bitrate [START_REF] Ozerov | Informed source separation: source coding meets source separation[END_REF][START_REF] Liutkus | Spatial coding-based informed source separation[END_REF], compared to a joint coding.

Finally, the connections between SAOC and DReaM might be stated this way: SAOC started from multichannel coding and met source separation (using coding), whereas DReaM started from source separation and met coding.

THE MIXING MODELS

We present here the underlying model of all the methods we will consider in Section 4, as well as some generalizations.

We assume that the audio objects signals (or sources) are defined as M regularly sampled times series s m of same length N . An audio object is thus understood in the following as a mono signal. Furthermore, we suppose that a mixing process produces a K-channel mixture {y k } k=1,••• ,K from the audio objects.

Linear Instantaneous Model

We first consider linear and time-invariant mixing systems. Formally, we suppose that each audio object s m is mixed into each channel k through the use of some mixing coefficient a km , thus:

y k (t) = M m=1 y km (t) (1)
where

y km = a km • s m , (2)
{y km } k=1,••• ,K being the (multi-channel) spatial image of the (mono) audio object s m . In the stereo case where K = 2, we call this mono-to-stereo mixing.

We suppose that the mixing filters are all constant over time, thus leading to a time-invariant mixing system. We say that the mixing is linear instantaneous.

Convolutive Case

If the mixing coefficients a km are replaced by filters, and the product in Eq. ( 2) is replaced by the convolution, we say that the mixing is convolutive. We can easily handle this case (see [START_REF] Liutkus | Informed audio source separation: a comparative study[END_REF]) with the Short-Time Fourier Transform (STFT) representation if the length of the mixing filters is sufficiently short compared to the window length of the STFT, as:

Y km (t, ω) ≈ A km (ω)S m (t, ω) (3)
where A km (ω) is understood as the frequency response of filter a km at frequency ω. When the mixing process is linear instantaneous and time invariant, A km is constant and the K×M matrix A is called the mixing matrix. When it is convolutive, this mixing matrix A(ω) is a function of ω. The mixing model can hence be written in the STFT representation as:

Y (t, ω) ≈ A(ω)S(t, ω) (4)
where

Y = [Y 1 , • • • , Y K ] and S = [S 1 , • • • , S M ]
are column vectors respectively gathering all mixtures and sources at the time-frequency (TF) point (t, ω).

Non-linear Case

Of course, in real musical productions, non-linear effects such as dynamics compression are present in the mixing process. We have shown in [START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF] that it is possible to revert to the previous -linear -case by "moving" all the effects before the sum operation of the mixing model. The problem with this approach is that it might lead to "altered" sound objects -i.e. "contaminated" by the effects -and thus harder to use for some active listening scenarios without noticeable artifacts. Another approach is to invert the effects in order to revert to the linear case. This is clearly out of the scope of this paper, where we rather focus on the inversion of the sum operation of the mixing model, in order to estimate the original sources. However, the methods presented in the next section have proved to be quite resistant to non-linearities of the mixing process.

Image-Based Model

In real-world conditions, the mixing process may be much harder to model [START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF]. Take for instance the stereo sub-mix of a multi-channel captured drum set, or the stereo MS recording of a grand piano. Then the solution is to not consider audio objects anymore but rather directly their spatial images. Source separation consists then in inverting the sum of Eq. ( 1), to recover the M separate images {y km } k=1,••• ,K from the mixture {y k } k=1,••• ,K . Each image has then the exact number of channels as the mix (K = 2 for a stereo mix). Such model will be referred to as stereo-to-stereo mixing. In such case, audio objects are not separated, but the modification of the separated images can still allow a substantial amount of active listening scenarios, including remixing and generalized karaoke. Respatialization, however, can be more difficult.

INFORMED SEPARATION METHODS

The objective of informed source separation is hence to compute some additional information that allows to recover estimates of the sources given the mixture {y k } k=1,••• ,K . Depending on the method, these sources can be either the audio objects s m or their spatial images {y km } k=1,••• ,K (K = 2 for stereo).

For the computation of the additional information, we assume that s m and A are all available at the coder stage. Of course, the main challenge is to develop techniques that produce good estimates with an additional information significantly smaller than the one needed to directly transmit s m .

Over the past years, we already proposed several informed source separation methods. More precisely, this section presents the similarities, differences, strengths, and weaknesses of four of them. A detailed technical description or comparison is out of the scope of this paper. The detailed descriptions of the methods can rather be found in [START_REF] Parvaix | Informed source separation of linear instantaneous underdetermined audio mixtures by source index embedding[END_REF], [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF], [START_REF] Gorlow | Informed audio source separation using linearly constrained spatial filters[END_REF], and [START_REF] Sturmel | Informed source separation using iterative reconstruction[END_REF], while their comparison is done in [START_REF] Liutkus | Informed audio source separation: a comparative study[END_REF].

Time-Frequency Decomposition

All the methods we propose are based on some timefrequency (TF) decomposition, either the MDCT or the STFT, the former providing critical sampling and the latter being preferred for the mixing model (see Section 3) and for filtering thanks to the convolution theorem.

Then, for each TF point, we determine the contribution of each source using several approaches and some additional information.

Additional Information

In the following, we assume that the encoder is provided with the knowledge of the mixing matrix A. However, this matrix may be estimated as demonstrated in [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF]. This information may be used either directly or by deriving the spatial distribution of the sources. Then, our different methods have specific requirements in terms of additional information.

Source Indices

The first information we used was the indices of the two most prominent sources, that is the two sources with the highest energy at the considered TF point. As explained below, this information can be used to solve the interference of the sources at this point. This information can efficiently be coded with log(M (M -1)/2) bits per TF point.

Source Energies

The information about the power spectrum of each source turned out to be extremely useful and more general. Indeed, if we know the power of all the sources, we can determine the two predominant sources. We can also derive activity patterns for all the sources. This information can efficiently be coded using for example the Equivalent Rectangular Bandwidth (ERB) and decibel (dB) scales, closer to the perception, together with entropy coding [START_REF] Gorlow | Informed audio source separation using linearly constrained spatial filters[END_REF],

or alternatively with Non-negative Tensor Factorization (NTF) techniques, as demonstrated in [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF][START_REF] Liutkus | Spatial coding-based informed source separation[END_REF].

Several Approaches

The majority of our ISS methods aims at extracting the contribution of each source from each TF point of the mix, at least in terms of magnitude, and of phase too for most of the methods.

Our first method performs a local inversion [START_REF] Parvaix | Informed source separation of linear instantaneous underdetermined audio mixtures by source index embedding[END_REF] of the mix for each TF point, using the information of the two predominant sources in this point (as well as the knowledge of the mixing matrix). More precisely, at each TF point two sources can be reconstructed from the two (stereo) channels, by a local two-by-two inversion of the mixing matrix. This way, we get estimates of the magnitude and phase of the prominent sources. As discussed below, this method gives the best results with the Signal-to-Distortion Ratio (SDR) objective measure of BSS-Eval [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF]. But the problem is that the remaining M -2 sources exhibit a spectral hole (no estimated signal), which is perceived as quite annoying in subjective listening tests [START_REF] Gorlow | Informed audio source separation using linearly constrained spatial filters[END_REF]. Also, this method requires the mixing matrix A to be of rank M .

Our second method performs Minimum Mean-Square Error (MMSE) filtering [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF] using Wiener filters driven by the information about the power of the sources (as well as the mixing matrix), the corresponding spectrograms being transmitted using either NTF or image compression techniques. Although this method produces results with a lower SDR, the perceived quality is higher, which matters to the listener. In contrast to the local inversion method, MMSE does not constrain as much the mixing matrix A and is therefore more flexible towards the mixing configurations. The separation quality, however, is much better when A is of rank M .

Our third method performs linearly constrained spatial filtering [START_REF] Gorlow | Informed audio source separation using linearly constrained spatial filters[END_REF] using a Power-Constraining Minimum-Variance (PCMV) beamformer, also driven by the information about the power of the sources (and their spatial distribution) and ensuring that the output of the beamformer matches the power of the sources (additional information transmitted in ERB/dB scales). In the stereo case (K = 2), if only two predominant sources are detected, the beamformer is steered such that one signal component is preserved while the other is canceled out. Applying this principle for both signal components results in inverting the mixing matrix (first method). Moreover, dropping the power constraint will turn the PCMV beamformer into an MMSE beamformer (second method). Otherwise, the PCMV beamformer takes advantage of the spatial distribution of the sources to produce best estimates than the early MMSE approach, at least with the PEMO-Q [START_REF] Huber | PEMO-Q -a new method for objective audio quality assessment using a model of auditory perception[END_REF] measure, closer to the perception.

Our fourth method performs iterative phase reconstruction and is called IRISS (Iterative Reconstruction for Informed Source Separation) [START_REF] Sturmel | Informed source separation using iterative reconstruction[END_REF]. It also uses the magnitude of the sources (transmitted in ERB/dB scales) as well as a binary activity map as an additional information to the mix. The main point of the method is to constrain the iterative reconstruction of all the sources so that Eq. ( 3) is satisfied at each iteration very much like the Multiple Input Spectrogram Inversion (MISI) method [START_REF] Gunawan | Iterative phase estimation for the synthesis of separated sources from single-channel mixtures[END_REF]. Contrary to MISI, both amplitude and phase of the STFT are reconstructed in IRISS, therefore the remix error should be carefully distributed. In order to do such a distribution, an activity mask derived from the Wiener filters is used. The sources are reconstructed at the decoder with an initialization conditioned at the coding stage. It is noticeable that this technique is specifically designed for mono mixtures (K = 1), where it gives the best results, and does not yet benefit from the case K > 1.

The main remaining issue with the aforementioned methods is that their performance is bounded. Other methods recently proposed [START_REF] Ozerov | Informed source separation: source coding meets source separation[END_REF][START_REF] Liutkus | Spatial coding-based informed source separation[END_REF] are based on source coding principles in the posterior distribution of the sources given the mixtures and should permit to reach arbitrary quality provided that the bitrate of the additional information is sufficient.

Performances

The quality performance of the system now reaches the needs of many real-life applications (e.g. industrial prototypes, see Section 5 below) with ongoing technology transfers and patents. The comparison of the current implementation of our four methods can be found in [START_REF] Liutkus | Informed audio source separation: a comparative study[END_REF], for the linear instantaneous and convolutive cases (see Section 3), using either the objective SDR criterion of BSSEval [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF] or the PEMO-Q measure [START_REF] Huber | PEMO-Q -a new method for objective audio quality assessment using a model of auditory perception[END_REF], closer to perception. It turns out that the first method (local inversion) exhibits the best SDR (objective) results, while the third method (constrained spatial filtering) exhibits the best PEMO-Q (more subjective) scores; this was also verified in a formal listening test [START_REF] Gorlow | Informed audio source separation using linearly constrained spatial filters[END_REF]. It is important to note that the complexity of these methods is low, enabling active listening in real time. Moreover, as shown in [START_REF] Liutkus | Informed audio source separation: a comparative study[END_REF], the typical bitrates for the additional information are approximately 5-10kbps per mixture and audio object, which is quite reasonable.

PROTOTYPES

Multiple versions of the DReaM system allow applications to uncompressed (PCM) and compressed (AAC/MP3/OGG) mixdown with mono-to-mono, mono-to-stereo, and stereo-to-stereo mixtures including artistic effects on the stereo mix [START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF].

DReaM-RetroSpat

We have presented in [START_REF] Marchand | Interactive music with active audio CDs[END_REF] a real-time system for musical interaction from stereo files, fully backwardcompatible with standard audio CDs (see Fig. 3). This system manages the mono-to-stereo case and consists of a source separator based on the first DReaM method of Section 4 (local inversion) and a spatializer, RetroSpat [START_REF] Mouba | RetroSpat: a perception-based system for semi-automatic diffusion of acousmatic music[END_REF], based on a simplified model of the Head-Related Transfer Functions (HRTF), generalized to any multi-loudspeaker configuration using a transaural technique for the best pair of loudspeakers for each sound source. Although this quite simple technique does not compete with the 3D accuracy of Ambisonics or holophony (Wave Field Synthesis -WFS), it is very flexible (no specific loudspeaker configuration) and suitable for a large audience (no hot-spot effect) with sufficient quality.

The resulting software system is able to separate 5-source stereo mixtures (read from audio CDs or 16-bit PCM files) in real time and it enables the user to remix the piece of music during playback with basic functions such as volume and spatialization control. The system has been demonstrated in several countries with excellent feedback from the users/listeners, with a clear potential in terms of musical creativity, pedagogy, and entertainment.

DReaM-AudioActivity

The DReaM-AudioActivity prototype (see Fig. 4) targets consumer/prosumer applications of the ISS technologies issued of DReaM. The software is written in such as way that each separation method can be included as a separate C++ subclass, but at the time of writing of this article, only the MMSE filter This prototype addresses the issue of studio music production, that is the stereo-to-stereo case. In some cases, the mix may not even be the exact sum of the stereo sources: dynamics processing can be applied and estimated a posteriori [START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF]. The coder performs, in almost real time, high-capacity watermarking of the separation information from the separated stereo tracks into the artistic mix coded in 16-bit PCM. The decoder performs offline reading of this watermark and performs the separation and re-mixing in real time. The number of tracks than can be included in the mix is only limited by the capacity of the watermark. Vector optimization of the audio processing core gives very low CPU usage during live separation and remixing. The end-user can then modify the volume and stereo panning of each source in real time during playback. Automation of global and per track volume and panoramic is also possible. As always, the coded mix is backward compatible with standard 16-bit PCM playback software programs with little to no audio quality impact.

CONCLUSION

In this paper, we have presented the DReaM project. Originally thought as a way to interact with the music signal through its real-time decomposition/manipulation/recomposition, the emphasis has been laid on the mixing stage, leading to source separation/unmixing techniques using additional information to improve the quality of the results. DReaM can also be regarded as a multi-track coding system based on source separation. Some of our techniques have been implemented in software prototypes, for demonstration purposes. These prototypes enable the user to perform, for instance, generalized karaoke and respatialization. We are currently extending our methods to compressed audio formats. We propose to compare our approach to e.g. MPEG SAOC in the near future, and envisage generalizing this informed approach to other problems than source separation, e.g. to the inversion of audio effects.

"Fig. 1 :Fig. 2 :

 12 Fig. 1: General architecture of an ISS coder.

Fig. 3 :

 3 Fig.3: From the stereo mix, the DReaM-RetroSpat player permits the listener (center) to manipulate 5 sources in the acoustic space (and to visualize the sound propagation).

Fig. 4 :

 4 Fig. 4: Manipulation of a 8-source mix by the DReaM-AudioActivity player.

see URL: http://dream.labri.fr

see URL: http://www.rockband.com

ACKNOWLEDGMENTS

This research was partly supported by the French ANR (Agence Nationale de la Recherche), within the scope of the DReaM project (ANR-09-CORD-006).


Antoine Liutkus

Stanislaw Gorlow

Nicolas Sturmel

Shuhua Zhang

Laurent Girin

Roland Badeau

Laurent Daudet

Sylvain Marchand

Gael Richard

Gaël Richard

Informed Audio Source Separation: A Comparative Study

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Active listening, which is the possibility to independently manipulate, mute or equalize audio objects within their mixture [START_REF] Avendano | Frequency-domain source identification and manipulation in stereo mixes for enhancement, suppression and re-panning applications[END_REF][START_REF] Oh | Enhancing Stereo Audio with Remix Capability[END_REF] has recently aroused much interest. In the context of music signal processing, an audio object can be identified with a sound source such as an instrument or a singing voice. All audio objects are mixed together into a mixture, which is possibly multichannel as in the common stereophonic case. From a practical point of view, active listening scenarios require the availability not only of the mixture, but also of the audio objects. Whereas some studies concentrate on a coding strategy [START_REF] Engdegård | Spatial audio object coding (SAOC) -The upcoming MPEG standard on parametric object based audio coding[END_REF][START_REF] Ozerov | Informed source separation: source coding meets source separation[END_REF] to convey the audio objects, others focus on a source separation approach [START_REF] Parvaix | A watermarking-based method for informed source separation of audio signals with a single sensor[END_REF][START_REF] Parvaix | Informed source separation of linear instantaneous under-determined audio mixtures by source index embedding[END_REF][START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF][START_REF] Gorlow | Informed source separation: Underdetermined source signal recovery from an instantaneous stereo mixture[END_REF]. The common idea of all these techniques is to make use of the mixture in order to reduce the required bitrate to transmit the audio objects. Indeed, a straightforward solution would be to separately encode the audio objects using audio coders. This solution is This work is partly funded by the French National Research Agency (ANR) as a part of the DReaM project (ANR-09-CORD-006-03) however not optimal in terms of bitrate because it amounts to encode several times the same information: once within the mixture and once alone for the transmission of the audio objects. Indeed, transmission of the mixture is considered mandatory in most cases.

Source separation consists in separating different signals from their mixture. Unfortunately, the performance of blind source separation, i.e. given the mixtures only, is still very dependent on the signals considered and does not systematically provide separated signals of sufficient quality for active listening applications. In informed source separation (ISS), source separation is improved by transmitting to the separation module some additional side information that strongly enhances the quality of the estimates. The general block diagram of ISS is presented in Fig. 1. Given the (mono) audio objects and the multichannel mixture, a small side information Θ is computed in the encoding stage and transmitted with the mixture. In the decoding stage, it allows to recover the objects given the mixture only. Many strategies can fit in this general framework, which are mainly distinguished by the assumptions made on the signals and by the corresponding separation algorithms. Indeed, it can be seen from Fig. 1 that the general ISS structure is independent from the separation method used at the decoder. Several approaches can hence be found in the literature [START_REF] Parvaix | Informed source separation of linear instantaneous under-determined audio mixtures by source index embedding[END_REF][START_REF] Ozerov | Informed source separation: source coding meets source separation[END_REF][START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF][START_REF] Gorlow | Informed source separation: Underdetermined source signal recovery from an instantaneous stereo mixture[END_REF] and their common objective is to maximize the perceptual quality of the estimates while minimizing the size of the side information. Typical bitrates of the methods under study lie around 5-15kbps per object, which is very competitive compared to bitrates achieved by usual perceptual coders.

The purpose of this study is to present the general ideas of informed source separation and to briefly present four different techniques from the state of the art as well as to give the corresponding references. Then, all those techniques are evaluated on the same corpus in an extensive comparative evaluation. It is structured as follows. First, we present the framework considered and some notations in Section 2. Then, an overview of the different techniques under study is presented During the encoding stage, both the objects and the mixture are known and a small side information is computed. At the decoding stage, this information is used to perform robust source separation in Section 3. Finally, an experimental comparison of all these techniques is found in Section 4, which allows to grasp some of the specificities of each method.

FRAMEWORK AND NOTATION

In the following, we assume that the M audio objects are defined as M regularly sampled times series s m of the same length N n . Furthermore, we suppose that a mixing process produces a K-channel mixture {y k } k=1•••K from the audio objects. The Short-Term Fourier Transform (STFT) is written using capital letters, e.g. S m (ω, and its squared modulus using bold capital letters, e.g. S m (ω, t). (ω, t) is called a Time-Frequency (TF) bin, where ω and t respectively stand for the frequency and frame index. N ω and N t are, respectively, the number of frequency bins and the number of frames of each transformed signal.

We consider linear and time-invariant mixing processes, since they are handled by all the techniques we review for ISS, contrarily to more complex models such as non-punctual or non-linear mixing [START_REF] Duong | Underdetermined reverberant audio source separation using a full-rank spatial covariance model. Audio, Speech, and Language Processing[END_REF][START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF]. Thus, each audio object m is supposed to be mixed into each channel k through the use of a constant mixing filter a km . When a km reduces to a single gain coefficient, the mixing is called instantaneous. We assume that a km are sufficiently short compared to the length of each frame so that the mixing process can be approximated in the STFT domain as:

Y (ω, t) ≈ A (ω) S (ω, t) (1)
where

S (ω, t) = [S 1 (ω, t) • • • S M (ω, t)] Y (ω, t) = [Y 1 (ω, t) • • • Y K (ω, t)]
are M × 1 and K × 1 vectors gathering respectively all audio objects and all channels of the mixture at TF bin (ω, t). A km (ω) is the frequency response of filter a km at frequency bin ω. The K × M matrix A (ω) is called the mixing matrix.

The objective of ISS is to compute some side information Θ that allows to recover good estimates ŝm of the audio objects given the mixture {y k } k=1•••K . For the computation of Θ, we assume that y k , s m and A are all available.

FOUR SEPARATION METHODS

In this section, we briefly present the four different techniques for ISS that we compare in the following. Because of the space limitation, we cannot give the details concerning the models, which can be found in the corresponding references.

Local inversion

The first technique considered [START_REF] Parvaix | Informed source separation of linear instantaneous under-determined audio mixtures by source index embedding[END_REF] is called "Local Inversion" in the following. It relies on the assumption that most audio signals can be considered sparse in the STFT domain, i.e. that only a small amount of TF bins have a significant magnitude, all others being close to zero. Consequently, for a given TF bin, it is much likely that only a subset of the objects have a noticeable energy. If we assume that at most K sources are active at some TF bin (ω, t), (1) becomes invertible locally for those sources, all others being set to zero.

For each TF bin, the technique then amounts to look for the combination of active objects that minimizes the squared error between S (ω, t) and its reconstruction, among the M ! K!(M -K)! possible combinations. The side information Θ is thus the combination of active objects for each TF bin. The computational complexity of this method is O (N ω N t M ). A further refinement on [START_REF] Parvaix | Informed source separation of linear instantaneous under-determined audio mixtures by source index embedding[END_REF] which permits variable bitrates is to send the combination of active objects only for TF bins whose magnitude lies above a perceptual threshold.

MMSE of locally stationary Gaussian processes

A second technique is based on the work in [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF]. The sources are assumed to be locally stationary Gaussian processes. In that case, the TF coefficients of the mixtures can also be considered as Gaussian and the optimal MMSE estimator of the source signals is provided by the mean of the posterior (Gaussian) distribution of the sources given the mixtures [START_REF] Duong | Underdetermined reverberant audio source separation using a full-rank spatial covariance model. Audio, Speech, and Language Processing[END_REF], which writes:

µ post (ω, t) = (diagS (ω, t)) A (ω) H A (ω) diagS (ω, t) A (ω) H -1 Y (ω, t) .
(

) 2
The side information considered by this method are A and the spectrograms S m of the sources. [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF] proposes to compress S m using either image compression, e.g. JPEG, or Nonnegative Matrix Factorization (NMF). The corresponding techniques are denoted "MMSE-NMF" and "MMSE-JPG" respectively in the following. If the original S m are used, the technique is called "Oracle MMSE". The complexity of MMSE techniques is O N ω N t K 3 .

Iterative beamforming

The third technique, abbreviated as MVDR+PP, is the TF selective iterative spatial filtering algorithm presented in [START_REF] Gorlow | Informed source separation: Underdetermined source signal recovery from an instantaneous stereo mixture[END_REF]. It uses the minimum-variance distortionless response (MVDR) beamformer [START_REF] Capon | High-resolution frequency-wavenumber spectrum analysis[END_REF] in the filtering stage and during a post-processing stage, it readjusts the magnitudes of the estimated signal spectra in order to comply with the imposed equal-loudness constraint. The following modifications were made with regard to [START_REF] Gorlow | Informed source separation: Underdetermined source signal recovery from an instantaneous stereo mixture[END_REF]. First, the spatial covariance matrices were approximated directly from the transmitted spectral envelopes and the mixing coefficients according to

CXX (ω, t) = A (ω) diag Ŝ (ω, t) A (ω) H ,
and no longer from the clustered data. Second, pairwise extraction of the two most dominant signal components was abandoned in favor of a one-by-one deflation strategy. In addition, the side information was compressed with bzip21 , which uses Burrows-Wheeler block sorting text compression technique and Huffman coding, to encode the bitstream. The complexity of the MVDR+PP decoder is O N ω N t KM2 .

Iterative sources reconstruction

This method is described in [START_REF] Sturmel | Informed source separation using iterative reconstruction[END_REF] and denoted ISS using Iterative Reconstruction (ISSIR) in the following. The spectrograms S m are quantized uniformly on a logarithmic scale to yield Sm and an activity map W m is computed as:

W m (ω, t) = 1 if Sm(ω,t) m S m (ω,t) > T act 0 otherwise .
where T act is an activity threshold. The side information Sm , W m , A is compacted using the bzip2 coder. At the decoder, the method initially sets Ŝm = Sm Y0 |Y0| and then iterates N I times the following operations. First, the sources are reconstructed and the error between left and right-hand sides of (1) using Ŝm is computed. Second, this error is uniformly distributed to the sources based on W m . Finally, a Griffin and Lim iteration [START_REF] Griffin | Signal estimation from modified short-time Fourier transform[END_REF] is performed. Two variants of the method are presented, one with single resolution (2048 point STFT) denoted "ISSIR-Single" and one with dual-resolution STFT (256 and 2048 points) with transient detection, denoted "ISSIR-Dual". The complexity of the IS-SIR decoders are: O (N ω N t KM N I ) for ISSIR-Single and O ((N ω,1 N t,1 + N ω,2 N t,2 ) KM N I ) for ISSIR-Dual, where N ω,i N t,i stands for the number of TF bins for each transform.

EVALUATION

The main goal of this study is to proceed to a comparison of the objective performance of the four techniques for ISS presented in Section 3, as well as to identify some promising research directions. To this purpose, a set of 14 excerpts sampled at 44.1kHz or 48kHz from the Quaero database 2 was gathered along with all their constitutive audio objects. Each excerpt is approximately 30s long and composed of 5 to 10 objects. Two mixing scenarios were considered: linear instantaneous and convolutive mixtures using fixed short filters of order 200. In any case, the azimuth for all objects were different and separated by at least 5 °. The filters used for mixing were the Head Related Transfer Functions (HRTF) described in [START_REF] Algazi | The CIPIC HRTF Database[END_REF].

The metrics considered are the Signal to Distortion Ratio (SDR, in dB) of BSSEval [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF] and the Perceptual Similarity Measure (PSM, between 0 and 1) of PEMO-Q [START_REF] Huber | PEMO-Q -a new method for objective audio quality assessment using a model of auditory perception[END_REF]. Whereas SDR is widely used for the objective evaluation of the performance of source separation algorithms, PEMO-Q is mostly used within the audio coding community. Even if both metrics are intended to be related to the perceptual accuracy of the estimates, SDR has been acknowledged as not sufficiently taking musical noise into account [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF], which motivates the combined use of PSM. Further quantities of interest are the encoding and decoding times for all excerpts.

All techniques were then run at various levels of quality, in order to compare their respective rate-performance curves. Since objective performance appears to be highly dependent on the excerpt considered, all results are compared as in [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF] to those of the oracle MMSE estimate [START_REF] Avendano | Frequency-domain source identification and manipulation in stereo mixes for enhancement, suppression and re-panning applications[END_REF], which makes it possible to compare metrics across different excerpts. Of course, performance of (2) should not be understood as a bound whatsoever, since it is limited by its use of the phase of the mixture to compute estimates, a limitation which is overcome by some of the techniques under study. Instead, it must be seen as a handy way to make the performance of all methods as independent as possible from the excerpt considered. Performance of the MMSE Oracle estimate for both instantaneous and convolutive mixtures can be found in Fig. 2.

For a given technique, a given excerpt and a given bitrate, the estimated audio objects were first compared to the original. Second, the obtained scores were averaged so as to obtain the metric for this technique, excerpt and quality. Third, the metrics obtained by the oracle estimate on the same excerpt were subtracted so as to obtain the differential metric ∆ (method, excerpt, bitrate). Finally, for a given method, the ∆ of all excerpts were merged together and the scatter plot (bitrate, ∆) was smoothed to obtain one single rate-performance curve. With 10 quality settings for each of the 6 techniques considered on the 14 excerpts using both instantaneous and convolutive mixing, this evaluation involved computing metrics for 1680 groups of 5 to 10 stems. All results for both instantaneous and convolutive mixtures are given in Fig. 2.

From this figure, many noticeable facts may be underlined. First, the availability of a side information Θ yields a strong improvement of the performance obtained by source separation methods both in terms of SDR and PSM. Perceptually, original and estimated audio objects are hardly distinguishable, provided the bitrate is sufficient. This fact is confirmed by objective results obtained by all methods above 5 -10kbps. ISS thus indeed yields estimates of sufficient quality for active listening applications. Such bitrates are quite low compared to those achieved by traditional perceptual coders.

Concerning the difference between the methods, it can be noticed that the performance of MMSE techniques is bounded by their limitative assumptions on the signals. Indeed, they use the phase of the mixture for estimation, as well as the assumption that energies of the objects are additive, which may not be true in many cases. These issues are addressed by other methods. Through its iterative reconstruction, IS-SIR implements phase consistency of TF representations and MVDR+PP can be seen to outperform even Oracle MMSE in terms of PSM thanks to its exploitation of spatial constraints in the model. Local inversion can be seen to outperform all other methods in terms of SDR. Still, it yields some musical noise, due to the TF bins set to 0 in the reconstruction, a fact which is highlighted in the PSM results. Performance of ISSIR is seen to be a bit lower than other methods, but it is mainly because its current implementation does not benefit from the availability of several channels in the mixture. A noticeable fact about ISSIR and MMSE methods is that they are operational for mono mixtures as well as for mixtures setting the same mixing filters to different objects, which is not the case for Local Inversion nor for MVDR+PP methods. Concerning computing time, Local Inversion is by far the most computationally efficient method in both instantaneous and convolutive mixtures. While MMSE and ISSIR Single decoding methods can be seen to be possibly implemented in real time in any case, MVDR+PP currently requires more optimization for convolutive mixtures, as does ISSIR Dual for both instantaneous and convolutive mixtures.

Several interesting directions for research can be outlined. First, most techniques share a very similar side information. Indeed, a coded version of the spectrogram of the mixtures or a related quantity is often required at the decoder. Hence, we are currently exploring crossovers between methods on this point, particularly through the use of the NMF/JPG lossy compression approach. Second, constraints on spatial information as in MVDR+PP or on the phases as in ISSIR do yield a sensible improvement of the perceived quality. The corre-sponding ideas may be transferred to other methods as well.

CONCLUSION

In this paper, we provided a unified presentation of Informed Source Separation. The challenge of ISS is to permit recovery of audio objects within their mixture through the use of a small side information Θ. Several methods for this purpose were presented that share the same common principle: Θ is chosen as the set of all parameters needed to perform source separation at the decoder. An extensive evaluation of the six techniques presented was done that allowed for the first time to compare the methods using the same corpus and the same metrics.

Noticeable results of this evaluation include the fact that ISS is highly efficient for recovering audio objects at a low bitrate (5 -15kbps per object) and also that many crossovers between existing methods can be considered. It is interesting to mention that any audio separation method whose parameters can be encoded in a small amount of bits may be an interesting candidate for ISS.

Fig. 1 .

 1 Fig. 1. High level block diagram of ISS. During the encoding stage, both the objects and the mixture are known and a small side information is computed. At the decoding stage, this information is used to perform robust source separation

Fig. 2 .

 2 Fig. 2. Results for instantaneous and convolutive mixtures.

http://www.bzip2.org

www.quaero.org


Nqrzohgjh Dqg

H [ Shuwlvh

Iru Ghvljqlqj

Jxlgholqhv B Léger

O Naud

L Delière

P Cartolaro

D Gouache

&RQWULEXWLRQ RI IRUPDO PRGHOOLQJ DQG ILHOG H[SHULPHQW WR OHYHUDJH

Keywords: Formal Modelling, Decision Support Systems, Crop Protection

Pesticide use is still much too often systematic on many crops. However, there is scientific consensus that many farmers could change their crop protection practices without putting their revenue in jeopardy. There is need for crop protection management systems that yield significantly lower pesticide consumption. For many crops such management systems are not available, although knowledge about the pathosystems exists. We present here two crop protection formal guidelines (Decision Workflow Systems, DeWS,) which were elicited and formalised using the Statechart (GrapeMilDeWS) and the coloured Petri nets (POD BLé) modelling languages. Elicitation and formal modelling helped the pathologists to ameliorate their design and provided a convenient computer ready format. Moreover, the main purpose of DeWS models is to provide an exhaustive specification, which can then be transferred. This permits large experimental networks to be created. Such networks are necessary to collect data about the behaviour of the pathosystem under low input management and to assess the DeWS robustness and efficiency. Finally DeWS are learning tools for the growers, development workers and for its original designers as well.

Introduction

When it comes to sustainable agriculture, crop protection is a major issue, with concerns for environment, economics, social and health issues. In France, the problem is particularly acute as the country is the largest pesticide consumer in Europe. Yet there is a scientific consensus that lower consumption can be achieved without economic loss [START_REF] Aubertot | Pesticides, agriculture et environnement : réduire l'utilisation des pesticides et en limiter les impacts environnementaux: INRA et Cemagref[END_REF]. There is need for crop protection management systems that yield significantly lower pesticide consumption. For many crops, such management systems are not available or empirically designed by a few farmers in their own farm. Yet, the knowledge to reduce pesticide use exists.

Since 2001, The Santé Végétale Laboratory at INRA (INRA-SV) is involved in the design of a low fungicide crop protection strategy in viticulture, using state of the art epidemiological results and know-how, yet keeping in mind the economic constraints of growers, including risk aversion. The result of this research, in collaboration with Cemagref-ITAP, is the Grape-MilDeWS Decision Workflow System (DeWS).

GrapeMilDeWS aims at managing the two most important cryptogramic pathogens in the French vineyard. Erysiphe necator which causes Powdery Mildew and Plasmopara Viticola which causes Downy Mildew. Alone, the control of these two diseases leads to 70% of yearly pesticide consumption in the French vineyard. This amounts to about 15% of the overall national fungicide consumption [START_REF] Ask | Enquête sur les exploitations viticoles et leurs pratiques phyto-sanitaires[END_REF]. It is very common among growers to mix in the same spraying operation phytosanitary specialties against both diseases. Yet, the regional information bulletins from the extension services separate information about these two diseases, as their bioclimatic characteristics differ. GrapeMilDeWS is original in that it manages at the plot scale both diseases in an integrated reasoning.

Since 2009, Arvalis-Institut du végétal has joined in and undertook to design a DeWS, named "BLé", for the management of winter wheat crop protection. With wheat, the difficulty is not in the lack of data or technical information to achieve low input (or near optimal) pro-duction. Printed guidelines that integrate reasoning about several cryptogamic diseases in an a priori spraying programme revised each year do exist. Yet, techniques and indicators had never been compiled in an integrated operational tool targeted at optimising the implementation of the spraying program throughout the season.

Although wheat production leads to less annual fungicide treatments per hectare than viticulture (3.49 vs 14.31 IFT1 respectively), wheat represents one fourth of French arable land therefore reduction of fungicide quantities would have a significant impact on a national scale. The BLé DeWS manages seven diseases (O. yallundae -eyespot, E. graminis -wheat powdery mildew, S. tritici & S. nodurum septoria, H. tritici-repentis -tan spot, P. striiformisyellow rust and P. recondita -brown rust, F. graminearum and M. nivale. and M. majus -Fusarium ear blight).

DeWS can be seen as crop protection guidelines specifications. Our case studies have been elicited and formalised using the Statechart (GrapeMilDeWS) and the coloured Petri nets (POD BLé) modelling languages, providing computer ready decision processes. The main purpose of DeWS models is to provide an exhaustive specification of the decision process, in order to transfer this operational knowledge to other researchers, development workers and eventually farmers. In both these case studies, the DeWS address tactical aspects of crop protection. Tactic, seen as the sum of treatment decisions (time and product dose at the intra annual scale), has a major impact on yield when lowering the input quantities.

The design methods and tools for these two DeWSs, which implement innovative fungicide protection strategies, are presented in the following pages. A brief description of the structure of the models is given, together with sample parts of the models. In the second part, the importance of the field experiments with respect to validating the decision workflows is explained. Finally, the choices of working at the tactical decision scale, and of the modelling paradigm, are discussed, particularly in the perspective of extending this work to the farm scale

Designing novel decision procedures

The French agronomic school has developed since the 1980s the model for action theory which gives a general framework to explain the managerial behaviour of farmers; articulating decision making at the crop level with the farm level (see [START_REF] Aubry | Farmer's technical decisions representation and decision support[END_REF]. This conceptual framework has been developed in order to diagnose the farmer's practices and help the farmer improve his management. It has inspired a wide range of tools developed in France for various applications in agriculture [START_REF] Attonaty | OTELO : un simulateur a base de connaissance pour raisonner equipement et organisation du travail[END_REF][START_REF] Bergez | MODERATO : an object-oriented decision tool for designing maize irrigation schedules[END_REF]Chatelin et al., 2005;[START_REF] Cros | Simulating management strategies: the rotational grazing example[END_REF][START_REF] Debaeke | A rule-based method for the development of crop management systems applied to grain sorghum in south-western France[END_REF]. From the earliest to the latest, these systems have evolved to include biophysical simulation models taking advantage of the increase of computing power. Other international approaches to decision making for crop management should also be mentioned like [START_REF] Audsley | A foliar disease model for use in wheat disease management decision support systems[END_REF][START_REF] Henriksen | PC-Plant Protection-a Danish tool to reduce fungicide input in cereals[END_REF][START_REF] Lemaire | Evolution of Proculture, a disease risk simulation model for decision taking in Mycosphaerella graminicola control[END_REF][START_REF] Parsons | Optimising fungicide applications on winter wheat using genetic algorithms[END_REF][START_REF] Zadoks | EPIPRE, a computer-based decision support system for pest and disease control in wheat: Its development and implementation in Europe[END_REF].

Workflow modelling

The concepts of business process management, workflow modelling (W. van der Aalst & van Hee, 2002), has recently emerged in agriculture. Some authors emphasize on the need for modelling the farm business processes in order to cope with the complexity of agri-food supply chain networks [START_REF] Wolfert | Business Process Modeling of the Pesticide Life Cycle: a service-oriented approach[END_REF] and novel e-government processes [START_REF] Ntaliani | Agricultural egovernment services: An implementation framework and case study[END_REF]. If the former have focused on the information system aspects of workflow modelling, [START_REF] Guan | Hybrid Petri nets modeling for farm work flow[END_REF] have focused on the optimisation of the workflow in Japanese rice paddies.

We have used the workflow concepts to develop decision systems based on expertise and synthesised knowledge instead of using the biophysical model approach to optimise the solution through simulation. Consequently, newly designed decision workflows need to be tested in field experiments. In order to avoid errors and inconsistencies and thus shorten the development and life cycle, the decision logics can be checked thanks to Formal modelling.

The Structure of Decision Workflow Systems

The idea underlying the design of DeWS is that optimal solutions for some pest management problems cannot be computed in practice (see ch. 2 in [START_REF] Léger | Recueil et Formalisation de procédés experts pour conduire une protection intégrée du vignoble[END_REF]. Therefore, the goal of the design is to reach satisfactory performance, both in terms of crop health and reduced fungicide quantities. The specification of the decision making procedure has to synthesise: an operational knowledge, including timing tactics, with the set of decision support tools that are available to the farmers, including field observation.

The work we present here builds on the workflow concept to model prescriptive decision procedures. The result should be a formal specification model of the process ready for integration into a simulator or a decision support system [START_REF] Léger | GrapeMilDeWS: a formally designed integrated pest management decision process against grapevine powdery and downy mildews[END_REF]. We have experience that graphical workflow models can be understood by non-specialists and that elicitation and formal modelling can help the pathologists to ameliorate their design through a systematic approach of the decision logic (Léger & Naud, 2009).

In the case study conducted in viticulture, GrapeMilDeWS was elicited directly in Statechart. The ability for the expert to visualise at a glance the behaviour of the DeWS at design time has proved helpful, both for the knowledge engineer modelling it (the misunderstanding between him and the expert could be alleviated before any behaviour was displayed at run time) and for the experts who gained a better understanding of the system they had in mind, seeing it literally unfold before them.

The terminology "decision workflow" acknowledges the fact that timing and processes, that is, sequences of events and decisions all along the season, should structure system's design. Therefore, we have chosen discrete event system (DES) paradigm for modelling. This is a time abstraction for the dynamics of the patho-system (i.e. the crop, the disease and the control system) for which a detailed model would also include differential equations.

Figure 1 The Structure of a Decision Workflow System

DeWS architecture (see Figure 1) is composed of: a core decision workflow which is modelled with a DES graphical modelling language like Statecharts [START_REF] Harel | The RHAPSODY semantics of statecharts (or, on the executable core of the UML) -(Preliminary version)[END_REF], for the GrapeMilDeWS case study or Coloured Petri Nets (CPN) (e.g. [START_REF] Jensen | Coloured Petri nets: basic concepts, analysis methods, and practical use[END_REF], for the BLé DeWS. a set of decision variables which are either Boolean or Ordinal (e.g. {A-,A+,A++}). These decision variables are used to route the decision control flow from a reasoning state to another according to the latest known environment status. At design time, the semantic of these variables should be made as generic as possible. This means that they are linked for a given field implementation to concrete measurements from the environment The field and its environment

Input events

Output events but could be linked to alternative inputs for another. This provides some kind of loose coupling between the decision workflow and the observation protocol currently used. an interface layer makes the link between the current observation protocol and the corresponding decision variable. Through this interface, the raw data from the observation is mapped onto the decision variable space. For field data, this may be achieved with simple discretization tables which are configuration parameters of the system. a set of measured variables: These variables can be continuous or discrete. Values can be obtained via sensors or other information devices; measurement protocols in the field, or can be the output of a bioclimatic model. input and output events: input events can be rain forecasts, changes of a variable value… the output events are decisions for actions like spraying or performing a field survey. Designing DeWS requires first to select the target diseases. In order to build a decision system that significantly reduces the use of pesticides (under the hypothesis that its use is generalised) the system will focus on the most important sources of yield loss and fungicide consumption. Because the system addresses the tactical decision making, the diseases that are not dependent on the annual weather are excluded (e.g. wood diseases of the vine).

Design principles

The second step is to have the domain expert identify sub-targets and issues that need to be addressed along the "pest management" season. With these sub target and issues, the designers break the season in a number of decision stages (see Figure 2 ). For each of these stages, some general principles should outline a solution to reach the stage targets. The stage time bounds, whether from calendar or phenology, should be defined too.

Then the detailed reasoning workflow is written as sub stages models. The decision variables define states in the reasoning process. Each state does not necessarily yield a decision (to treat or not to treat). At one moment in the reasoning, the state may be to wait for an evolution in the system that will trigger a treatment decision or alternatively the conclusion that no treatment is needed.

The formal specification of the process is iterative. The stage's sub workflow and the inter-stage transitions may therefore evolve during the design process.

In the two following sections, we give a short overview of the two case studies on which we implemented this design method.

Stage i

Intermediate targets

Stage i+1

Stage n

Final goals

Events can be:

• landmarks linked to a « clock » or a duration • internal (action done)

• events un or ill -planned (rain or rain forecast…) The GrapeMilDeWS: D

GrapeMilDeWS as hinted the management of both Pow bud break and the final state stages. Each stage correspo fungicide treatment against treated against, the system is provide some robustness in th zero. Indeed, there are two treatments the question is lim it comes to optional treatmen treatment opportunity. This de provide hierarchy and concu provide design ergonomic. Us prohibitive number of states w UML standard (

The BLé decision wor

The BLé DeWS integrate season is divided in 6 decisio in regions where 3 sprays are the main crop. While GrapeM manages doses. Septoria wh through the dose of product part of a sub module which 2010).

The other major differenc (CPN) instead of Statecharts.

Decision workflow system with St

d previously provides a structured reasonin wdery and Downy of the vine at the plot scal e is reached at harvest. The system is orga nds to a decision about the opportunity to each of the target disease. When both d s organised for spraying both treatments at he pest management, the minimum number mandatory treatments against each diseas ited to positioning the spraying properly insid nt, the positioning problem is doubled by t esign was first formalised with Statecharts (H urrency to mealy finite state machines. T sing so-called "flat" state-machines would yie when modelling large systems. Statecharts a r, 2004) 2008) which is semantically equivalent to standard Petri Net: a CPN can be unfolded into standard PN. Petri nets are common to represent industrial logistic problems; they are also at the foundation of most workflow modelling formalisms.

Our interest in Petri Nets lies in their ability to represent explicitly and compute the state of the whole farm level decision system. In addition this formalism is more explicit with respect to resource management than what could be achieved with Statecharts. Figure 4 illustrates how we implemented the BLé DeWS with hierarchical CPN using the "CPN Tools" modelling and simulation tool [START_REF] Jensen | Coloured Petri Nets and CPN Tools for modelling and validation of concurrent systems[END_REF]

Validation methodology includes field experiments

A DeWS relies on the modelling process of an expert based specification. Accordingly, we proposed in ch. 6 Léger (2008, pp. 162-178), an assessment based on evaluation and conformity-checking. The performance and robustness of the decision system are evaluated through field experiments, at the plot level, over several years and various pedo-climatic conditions. In the same time, the conformity of the experimental implementation to the specification is checked.

GrapeMilDeWS is currently experimented in several wine producing regions of France [START_REF] Naud | Modelling and experimenting crop protection decision workflows: some lessons from GrapeMilDeWS research[END_REF]. This large scale experiment is necessary because of the lack of data about the behaviour of the powdery and downy mildews pathosystems under low input management and for various bioclimatic conditions. The data that is available concerns mostly untreated plots, or plots managed with periodic treatments according to specifications of products. These data are insufficient for the fine analysis of crop protection reasoning and tactics. GrapeMilDeWS has de facto become a learning tool for the growers and development workers involved in the experiment, and for its original designers as well [START_REF] Naud | Modelling and experimenting crop protection decision workflows: some lessons from GrapeMilDeWS research[END_REF].

With wheat, the focus is not on acquiring new biological knowledge on pathosystems but rather on integrating, into an explicit crop protection decision workflow, existing general and local knowledge, specialized models or disease-specific decision support tools. For instance, selecting the most effective DSS tool for eyespot between a very detailed agronomic decision table and the simpler cultivar resistance index to the disease.

Quantitative evaluation of the crop protection

Success criteria are defined in the first steps of a DeWS' design. In viticulture there are two major criteria. The plot must reach the grower's desired yield, for example the authorised production quota for a given wine type in a region. As for sanitary quality criterion, the percentage of infected berries should be comparable to results of a conventional practice. We consider that performance is not satisfactory when more than 5% of the berries are infected by powdery mildew. With wheat, our aim is to maximize the margin, therefore the DeWS must reach output that is equal or above the a priori optimal fungicide programs and respecting the sanitary constraints (e.g. mycotoxins).

The validity of low input systems lies in the ability to achieve satisfactory results in a number of different climatic and geographic contexts. The formal model provides an objective communication tool to extend the experiment beyond the original designer's own facilities.

Assessing the conformity of the field implementation to the specification

In order to identify potential weaknesses in the decision process specification and correct them, the DeWS design process is iterative; interleaving design phases, experimental evaluation phases, during which the decision process is closely monitored. The decision variables are recorded on a daily basis. Each decision is also recorded as well as the date of its implementation. This makes it possible to evaluate the conformity of the execution of a DeWS to its specification. As a result of this conformity evaluation, the data of a given plot for a given year may be partially or totally discarded in the performance evaluation. It can also happen, as it is the case for business workflows, that the model of the DeWS should be enhanced to account for a pertinent decision for a newly encountered situation.

The analysis method builds on the quantification of the conformity of an implementation's behaviour to its specification model behaviour [START_REF] Cook | Software process validation: quantitatively measuring the correspondence of a process to a model[END_REF]. The method is to compare the event sequence produced during the implementation of the process to the simulated event sequence produced by the specification model. There are two kinds of difficulties in assessing the conformity of processes and particularly crop protection decision processes. The first difficulty lies in the frequent impossibility to observe/record all relevant events. The second is the need to integrate quantitative time in the comparison. When seen as a simple sequence of event, conformity can be assess with a sequence edit distance measure [START_REF] Levenshtein | Binary codes capable of correcting deletions, insertions and reversals[END_REF]. However, taking into account the timing of the events makes the problem more difficult. In Léger (2008 in ch. 6), this second problem was partially addressed by taking advantage of the cycleless structure of GrapeMilDeWS' Statechart.

Many workflow modelling languages are Petri Net based and many Process mining tools are therefore adapted for Petri Nets. For instance the conformity problem is addressed by [START_REF] Rozinat | Conformance checking of processes based on monitoring real behavior[END_REF] by counting the token missing or supernumerary ones in the net when replaying an observed event sequence. The ProM framework (W. [START_REF] Van Der Aalst | ProM 4.0: Comprehensive Support for Real Process Analysis[END_REF] should be appropriate to analyse the conformity of the current BLé DeWS experiment implementation.

Discussion

Implication of choosing DES as modelling paradigm.

Our case studies were modelled as reactive systems. Simply said, this means that a decision should be applied in the field quickly after it has been taken, that response to new information should also be quick, and that anticipation under uncertain climatic events and revision are not taken into account. Although this hypothesis has proved unrealistic during the field implementation, it is a good heuristic at design time and makes the conformance checking (see above) simpler. Through field experiments in the first years, further knowledge of the time constraints and required flexibility that affect the decision system was acquired.

When implementing a DSS from the DeWS, these constraints and planning strategy could be added to the formal model. Indeed we have shown (see ch.6 in [START_REF] Léger | GrapeMilDeWS (part.2) Experimenting an IPM Decision Process against grapevine powdery and downy mildews[END_REF] that the experts when implementing the decision process tend to anticipate a number of decision as soon as they can "compute" it in order to simplify their planning. Developing a DSS would require providing the farmer with the ability to project into the possible futures, plan future operations and yet keep the possibility to "roll back" or revise the planning. Plan technologies (e.g. [START_REF] Martin-Clouaire | Modelling and simulating work practices in agriculture[END_REF] would allow such behaviour, but the tradeoffs for that would be to loose the ergonomic graph representation. An alternative would be to interleave planning and revision processes with the core reactive decision process. We have undertaken to test this latter alternative with YAWL, which is a computer language dedicated to workflows (W. M. P. [START_REF] Van Der Aalst | Design and Implementation of the YAWL System[END_REF]. Integrating reactive and planning behaviour is definitely one of the tasks ahead of this research.

Why work at the tactical scale?

Within the agronomical research community, it is common to admit that the most promising way to reduce pollution of a given cropping system, which should be run at its optimum, while preserving the revenue is to modify the cropping system itself. In other words the largest reduction in pesticide use would only be made at the strategic pluri-annual level through drastic changes. It seems that this assumption has lead to a low level of interest among crop protection researchers about the ways and methods that will allow a producer to manage a given cropping system close to a steady optimal.

Furthermore, the DeWS approach, which is process based, could be adapted to non conventional farming paradigms like integrated production or organic farming.

Yet, our experiments show that even under the traditional agronomic production paradigm, most farmers have a technically suboptimal use of fungicides (i.e. GrapeMilDeWS achieved commercial standard production with 40% to 60% less treatments, in 80% of the tests.). Our understanding of these result, is that the farmers put more emphasis on organisational risk management, and projected consequences on the yield, than on the economic loss caused by spending on unnecessary sprayings. Our efforts aim at developing decision devices which, from the earliest steps of design, take into account the farmers most stringent organisational constraints while outlining for them the safe paths to make the most productive use of their inputs. In this context, the field experiments that we undertook show that low input crop protection is economically efficient (yield and cost efficiency) and that it is technically feasible under normal production conditions.

Going upscale: the farm level

Our work has so far been limited to the plot scale, ensuring the design is safe, and efficient at that scale. The farm scale is the research step ahead for DeWS. We believe that having a formal model of the system will help us study through model checking or simulation the behaviour of the whole farm system.

One way to think about the farm scale level is a distributed decision system, either holonic or multi-agent, where spraying needs for each plot, with its own instance of the DeWS, would be managed by a planning module with optimisation algorithms.

Alternatively, if the DeWS is implemented with CPN, the farm level could also be implemented with this same formalism. This second solution would require altering the original design in order to include the resource management constraints. This CPN model could however be adapted to a commercial workflow management system easily.

Conclusion

With the Decision Workflow System methodology, we address the need for novel pest management methods at the tactical decision scale. This design methodology aims at more rational use of pesticides. This methodology blends field experiment and formal modelling to assess the quality of candidate solution, allowing incremental improvement of the designed solution. Formal methods are used to ensure the quality of experiments.

The traceability in the decision making process gives a renewed perspective in the study of decision support systems. For instance while developing a new bio-climatic model, epidemiologists and pathologist will be able to use the DeWS as a realistic and yet instrumented complete pest management strategy, which allows comparison of various candidates. In the same process, a DeWS will help identify when more information is needed to provide better decision making.

Our experience with the field experiment networks is that it is an efficient tool for scientists, development worker and farmer to learn about new crop protection principles that would at first hand go against their intuition. This is achieved by providing a safe environment: the experimental set up, but more important the structured reasoning framework.

This structured reasoning framework does not only facilitate communication within the experiment network. It also allows to gather data about performance and feasibility, including pragmatic details about use of weather forecasts or interpretation of bioclimatic models, and to relate this data to decision paths. The accumulation of cases permits to consolidate scientifically sound knowledge without implementing the usual blocs and repetitions.

In the same time, extension workers participating to the experiment have gained experience about low input strategies, and participating farmers have gained confidence in reducing fungicide intensity.

Besides continuation of experimentation of existing DeWS, further research will investigate decision at the farm scale, planning and revision of decisions, and optimisation of sampling within a plot and between the different plots of the farm.

Figure 2

 2 Figure 2 Conceptual structure for a Decision Workflow System in pest management.

Figure 3

 3 Figure 3 Excerpt from GrapeMilD Figure 3 displays several noted that the reasoning is h patho-system's epidemic risk decision making process. Th transition conditions become not last after mid flowering b transition from Stage2). The m

Figure 4

 4 Figure 4 Stage 0 top level from the BLé DeWS.

eld Downy mildew ge Treatment done Initial pseudostate Conditional node Final state DeWS' Stage 2.

	tatechart
	ng specification, for
	le. It is initialised at
		nised in 7 decision
			apply at most one
		iseases should be
			the same time. To
	r of treatment is not
	se. With mandatory
	de the stage. When
	the question of the
	Harel, 1987), which
	These mechanisms
	eld a very high if not
	are now part of the
	e
	l features of the Statechart model of GrapeM MilDeWS. It can be
	hierarchically decomposed from a high leve el evaluation of the
	ks to sub reasoning states guiding in detai il the timing of the
	e formalism is concise in that sub process		will be aborted as
	true at a higher level. For instance, the 'wa it for rain' state will
	ecause Stage2 will end at that time (see co ondition on the exit
	model is limited to the management of a singl le plot.
	rkflow: modelling with CPN
	s the pest management of 7 diseases in a u	nified process. The
	n stages. This breakdown is adapted to the	protection of wheat
	usually carried out and more generally to fa rms where wheat is
	MilDeWS manages solely the opportunity of		spraying, BLé also
	hich is the main wheat disease in most of F France is controlled
	applied. The selection of the products and	the correct dose is
	interacts with the core decision workflow		(Hernandez et al.,
	ce with GrapeMilDeWS lies in the use of C Coloured Petri Nets
	. CPN is a high level variant of Petri Nets (P PN) (Petri & Reisig,

IFT is the treatment frequency index(Champeaux,

2006) the data are from http://agriculture.gouv.fr/sections/thematiques/environnement/prevention-des-pollutions/produits-phytosanitaires6167 CIGR-Ageng 2010 2010; International Conference on Agricultural Engineering, 6-8 Septembre 2010, Clermont Ferrand

CIGR-Ageng 2010 2010; International Conference on Agricultural Engineering, 6-8 Septembre 2010, Clermont Ferrand

CIGR-Ageng 2010 2010; International Conference on Agricultural Engineering,