Single-input single-output 2. Multiple-input single-output 3. Single-input multiple-output 1. See the Wiener-Khinchin convergence theorem 2. See the central limit theorem

A B S T R A C T

This work deals with the problem of reverse audio engineering for active listening. The format under consideration corresponds to the audio CD. The musical content is viewed as the result of a concatenation of the composition, the recording, the mixing, and the mastering. The inversion of the two latter stages constitutes the core of the problem at hand. The audio signal is treated as a post-nonlinear mixture. Thus, the mixture is "decompressed" before being "decomposed" into audio tracks. The problem is tackled in an informed context: The inversion is accompanied by information which is specific to the content production. In this manner, the quality of the inversion is significantly improved. The information is reduced in size by the use of quantification and coding methods, and some facts on psychoacoustics. The proposed methods are applicable in real time and have a low complexity. The obtained results advance the state of the art and contribute new insights.

Key words-Active listening, informed inversion of dynamic range compression, informed source separation, informed spatial filtering under linear constraint, multichannel object-based audio coding, reverse audio engineering R É S U M É Ce travail s'intéresse au problème de la rétroingénierie du son pour l'écoute active. Le format considéré correspond au CD audio. Le contenu musical est vu comme le résultat d'un enchaînement de la composition, l'enregistrement, le mixage et le mastering. L'inversion des deux dernières étapes constitue le fond du problème présent. Le signal audio est traité comme un mélange post-non-linéaire. Ainsi, le mélange est « décompressé » avant d'être « décomposé » en pistes audio. Le problème est abordé dans un contexte informé : l'inversion est accompagnée d'une information qui est spécifique à la production du contenu. De cette manière, la qualité de l'inversion est significativement améliorée. L'information est réduite de taille en se servant des méthodes de quantification, codage, et des faits sur la psychoacoustique. Les méthodes proposées s'appliquent en temps réel et montrent une complexité basse. Les résultats obtenus améliorent l'état de l'art et contribuent aux nouvelles connaissances.

Mots clefs-Codage audio multipiste basé sur l'objet, écoute active, filtrage spatial informé sous contrainte linéaire, inversion informée de la compression de la dynamique sonore, rétroingénierie du son, séparation de sources informée Dans la plupart des cas, les enregistrements musicaux professionnels se soumettent à deux procédés, le mixage et le mastering, et beaucoup de formats de distribution habituels, comme par exemple la norme Compact Disc Digital Audio (CDDA), sont strictement en stéréo. Tandis que le terme « mastering » se réfère au procédé d'optimisation du mix final ainsi qu'au transfert de ce dernier sur un dispositif de stockage de données, le terme « mixage » se réfère au procédé d'assemblement de plusieurs couches d'audio préenregistrées et éditées dans un mélange composé. Chaque couche peut être imaginée comme une piste d'une table de mixage, qui alors représente une source sonore. Une source peut être à la fois une voix, un instrument, ou un groupe desdits. Le placement apparent des sources entre les haut-parleurs dans l'espace sonore stéréo est aussi connu en tant que « imaging » dans l'ingénierie du son professionnelle.

L'ingénierie du son est une discipline établie qui est employée dans plusieurs domaines qui font partie de notre vie quotidienne sans que l'on ne s'en rende compte. Pourtant, peu de gens savent comment l'audio a été produit. Si l'on considère l'enregistrement et la reproduction du son ou bien la diffusion comme exemple, on peut imaginer qu'un signal préenregistré provenant d'une source acoustique soit altéré par un ingénieur du son d'une telle manière qu'il corresponde à un certain critère pendant qu'il est joué. Le nombre de ces critères peut être grand et il dépend souvent du contexte. En général, ladite altération du signal à l'entrée est une séquence de nombreuses transformations en avant, la réversibilité desquelles porte peu ou aucun intérêt. Mais qu'en est-il de celui qui voulait faire exactement cela, c'est-à-dire d'inverser la chaîne de transformation, et ce qui est plus important, d'une manière systématique et répétitive ? Cela permettrait à l'auditeur d'interagir avec ou réinterpréter la musique stockée, par exemple, sur un CD audio et ainsi de devenir actif. Pour rendre cela possible, un enregistrement musical doit être décomposé en ses éléments constituants -au moins partiellement.

L'écoute active est un concept de la technologie musicale développé à la fin des années 1990 par Pachet et Delerue. Celui-ci a pour objectif de donner à l'utilisateur un certain degré de contrôle de la musique qu'il écoute. Au contraire du cas habituel, dans lequel un morceau de musique est simplement joué sans aucune modification conceptuelle, l'écoute active permet une expérience personnalisée de la musique à travers la réinterprétation. Son objectif est de pourvoir l'auditeur d'un plus haut niveau de confort musical et de lui donner accès à une nouvelle musique en créant des environnements sonores pour des répertoires de la musique déjà créée, dans lesquels les modifications conservent la sémantique. Un tel exemple est la réspatialisation de sources sonores ou le remixage en changeant le volume et/ou l'angle panoramique d'une source distincte. Donc, afin de mettre en oeuvre l'écoute active, il faut reconquérir l'accès aux composantes des source latentes étant donné le mix masterisé, à savoir exercer la rétroingénierie du mix.

L'objectif de la rétroingénierie du son peut être soit d'identifier les paramètres de transformation étant donné les signaux d'entrée et de sortie soit de récupérer le signal d'entrée qui appartient au signal de sortie étant donné les paramètres de transformation, ou les deux. Un modèle de signal et de système explicit est obligatoire dans chacun des cas. Le deuxième cas peut sembler banal, mais seulement si la transformation est linéaire et orthogonale et en tant que telle parfaitement inversible. Pourtant, la transformation en avant est souvent ni linéaire ni inversible. C'est par exemple le cas pour la compression de la dynamique sonore, qui habituellement est décrite par un système dynamique non-linéaire à paramètres variant dans le temps. La théorie classique de systèmes linéaires à paramètres invariants dans le temps ne s'applique pas dans ce cas, donc, une solution sur mesure doit être trouvée. De plus, pour la raison que le nombre des pistes utilisées est normalement supérieur au nombre des canaux du mélange, le mixage est sous-déterminé mathématiquement parlant. En conséquence, le démixage constitue un problème de séparation de sources qui est mal posé.

Dans son tutoriel de 2005, Knuth se sert de l'approche Bayesienne pour le design des algorithmes de séparation de sources robustes qui profitent d'un savoir a priori du problème présent pour assurer que l'on atteint une solution optimale. Il l'appelle la séparation de sources « informée » pour distinguer l'approche de la séparation de sources « aveugle » où il y a peu d'information. L'idée de base de son approche informée consiste à introduire l'information préalable du problème, qui peut être une propriété physique ou de même une loi physique, dans la solution. C'est sans doute plus spécifique que le théorème de Bayes, qui ne considère que des distributions a priori. Le modèle qui accompagne un tel problème de séparation de sources informée est donc une combinaison des distributions et de l'information a priori.

L'idée peut être appliquée au problème de la rétroingénierie par la manière suivante. Tout d'abord, le mélange est considéré comme postnon-linéaire, sous l'hypothèse que le mixage linéaire soit suivi par le mastering non-linéaire. Ensuite, le mixage et le mastering peuvent être décrits par deux systèmes déterminés en cascade. Le composant probabiliste dans le modèle est dû à la tâche mal posée. La probabilité d'entendre la source d'intérêt dans le mélange est donnée par la distribution a posteriori du signal de la source respective. La connaissance à propos de la production d'un enregistrement musical représente le savoir a priori que nous disposons. C'est typiquement les effets utilisés et le positionnement spatial des sources. L'information a priori comprend les paramètres de système, soit les coefficients des filtres et les angles panoramiques ou directions, ainsi que les paramètres de v distributions de sources, soit les paramètres de signal. Ces derniers représentent les paramètres hyper d'après Bayes.

Selon le dernier paragraphe, il est évident que la meilleure estimation possible des signaux de source est atteinte seulement quand l'information a priori sous la forme de paramètres de système et de signal est parfaitement donnée. Pour cette raison, le problème à résoudre est formulé comme suit. Étant donné les modèles de mixage, de mastering, et de sources, on cherche à récupérer les estimations des signaux de source à partir du mélange de meilleure qualité possible. Les paramètres des signaux sont estimés à partir des signaux de source originaux qui sont accessibles. Les paramètres de mixage et de mastering sont censés être connus. Le problème décrit ci-dessus a lieu d'être seulement si la récupération des composants de source est conduite sans avoir accès aux originaux. Sinon, le problème est banal. L'approche discutée doit être donc appliquée dans un scenario où la « collection » de l'information a priori est découpée de son emploi. Une fois que les sources sont mélangées, elles ne sont plus disponibles. L'approche peut être aussi caractérisée par l'accès aux signaux de source qui est limité temporellement et localement. Ainsi, on doit distinguer le processus de la création de contenu du processus de la consommation de contenu. Le créateur de contenu est responsable de fournir toutes les données nécessaires au consommateur de contenu, afin que ce dernier puisse exercer la rétroingénierie du mix de manière inaperçue et de remixer le contenu ad libitum grâce à une interface graphique. Ceci est réalisé dans un framework « encodeur/décodeur ». La tâche de l'encodeur est d'extraire un minimum de données supplémentaires provenant des signaux de source pour que le décodeur puisse récupérer leurs répliques à partir du mélange à haute qualité sonore. Ces métadonnées, si elles sont suffisamment petites, peuvent être cachées dans le signal de mélange sous la forme d'un filigrane inaudible soit doivent être rattachées au mélange lui-même. En dehors du problème évident de la rétroingénierie, il reste aussi le problème de la réduction de données concernant les paramètres du modèle, qui n'est pas négligeable. Le problème est d'autant plus difficile quand les métadonnées doivent être cachées dans le mélange ou quand le décodeur doit être exécuté en temps réel. Il mérite aussi de se questionner à propos du taux de transfert suffisant pour atteindre une bonne performance du système.

La cascade aux deux étapes qui est proposée se base sur un modèle simplifié de la chaine de production de la musique, qui consiste en une superposition de pistes monocanales ou bicanales spatialisées et éventuellement traitées avec un égaliseur dans l'étape de mixage et en la compression de la dynamique sonore dans l'étape de mastering. La borne supérieure de performance de la cascade est principalement due à la complexité sonore de la musique commerciale. Notamment, celle-ci dépend du degré d'interférence des spectres des pistes con-stituantes et de comment les sources sont distribuées dans l'espace sonore. La qualité du son après le démixage est sujette au dit « gain vectoriel ». Celui-ci s'est révélé être une fonction de a) les densités spectrale de puissance et b) le système de mixage. En règle générale, moins les spectres en question interfèrent et plus ils sont séparées dans l'espace, mieux est la qualité du son à la sortie.

La connaissance du mixage et du mastering peut être vue comme un défaut du schéma, qui le rend difficilement applicable à la musique existante, pour laquelle ce genre d'information est indisponible. Et même si des outils comme la factorisation en matrices non-négatives pour apprendre les spectrogrammes existent, sa performance est limitée. Premièrement, car cette factorisation n'est pas unique, et deuxièmement, sa fonction-coût n'est pas convexe. Dans le schéma proposé, comme dans n'importe quel schéma basé sur un modèle, une déviation des paramètres exacts provoquera une erreur additionnelle dans le résultat. Le décodeur est le plus efficace quand il est suppléé par une information très précise d'un encodeur accompagnant -ou un estimateur.

Les résultats obtenus permettent de tirer les conclusions suivantes. L'algorithme proposé est capable de fournir des estimations de signaux de source qui sont perçues comme étant plus proches aux orignaux que les algorithmes de type similaire publiés. Aussi, il mérite d'être mentionné que l'algorithme n'impose aucune restriction sur le nombre de sources ni sur leur interférence spectrale. Au contraire, il s'adapte à la constellation de signal donnée et livre les meilleures estimations possibles en temps quasi-linéaire en respectant une contrainte de puissance. Donnant une haute ressemblance aux signaux originaux au taux d'information d'accompagnement assez tolérable, étant d'environ 10 kbps par source ou canal, l'algorithme est bien adapté pour des applications d'écoute active en temps réel. Le filtre à variance minimum conservant la puissance se comporte perceptivement mieux qu'un filtre du type Wiener pour un mélange instantané aussi bien que pour un mélange convolutif. La contrainte de puissance égale garantie que les répliques récupérées retiennent le timbre et la bande passante auditive des originaux. De plus, il a été observé que les estimations bruitées sont plus appréciées par l'auditeur en comparaison avec les répliques sonnant « perforé » ou « sourd ». Avec l'approche proposée, le mix peut être décomposé en pistes séparées ou en objets du premier plan et le fond, et de la même manière la voix peut être séparée de l'instrumental pour karaoké. Au-delà de cela, il est possible d'extraire des images spatiales de sources sans changer leur placement. La séparation de sources informée (SSI) peut être vue comme un formalisme nouveau pour un principe de codage connu, puisqu'il possède les avantages suivants. SSI se laisse réaliser dans un framework modulaire qui est d'un côté rétrocompatible et d'un autre côté il a une compatibilité ascendante : le filtre utilisé peut être adapté vii à un plus grand nombre de canaux. En plus, on a pu observer qu'un rapport signal sur bruit (RSB) à fréquence pondérée peut donner des résultats aussi fiables que les métriques qui modélisent la perception humaine -mais à un coût beaucoup plus bas. Cependant, l'inconsistance entre les métriques de performance différentes rende des tests d'écoute toujours indispensable.

La performance d'un estimateur linéaire est bornée si le mélange est sous-déterminé à cause de sa résolution limitée. Cela provoque la nécessité de transmettre le résidu ou d'augmenter le nombre de canaux pour atteindre une meilleure qualité. La qualité d'un remix est largement satisfaisante si le mix est compressé sans perte. Sinon, la qualité est sensiblement détériorée. En connaissant les paramètres qui ont été utilisés pour la compression de la dynamique sonore, il est possible de récupérer le signal non-compressé du signal compressé avec une haute précision et un effort de calcul relativement bas. Si la compression est défaite avec une erreur négligeable, les signaux demixés sont presque identiques à ceux obtenus à partir d'un mix non-compressé. Le décompresseur est nécessaire pour éviter des artefacts qui ne sont possiblement pas entendus dans le mix. Comme mentionné précédemment, la qualité du son dépend du gain vectoriel qui augmente avec le nombre des canaux de mélange. Pour des applications de codage, le système de mixage peut être choisi arbitrairement et un nombre distinct de pistes peut être mixé dans un nombre de canaux inférieur de telle manière que l'intégralité des pistes décodées montre le même rapport signal sur interférence (RSI). La qualité du son peut être prédite à l'encodeur. Ainsi, le nombre de canaux peut être choisi avant le mixage réel et le niveau de qualité après le démixage peut être contrôlé a priori.

viii L A B O R AT O R Y

The thesis has been prepared in the following laboratory: Laboratoire Bordelais de Recherche en Informatique UMR CNRS 5800 351, cours de la Libération 33405 Talence Cedex France

ix A C K N O W L E D G M E N T S
First of all, I would like to thank Sylvain Marchand for letting me participate in the DReaM project while giving me the chance and the freedom to develop my own ideas. I know it has not been easy being confronted with my way of looking at things and/or my convictions. Thank you for your advocacy and your patience. I hope it paid off in the end and that I have taken a step forward in becoming what you consider a good researcher. I would also like to express my gratitude to Josh Reiss for welcoming me in his group at the Centre for Digital Music in London. Part of the work was undertaken there during my research stay. Thank you and Emanuël Habets from the International Audio Laboratories Erlangen for proofreading the papers which you have co-authored. Your comments were always constructive, on time, and on point. In the same breath, I would like to thank Corey Cheng for his editorial review of one of my papers.

Then, I would like to thank my project partners for the passionate discussions that we led over the three years: Roland Badeau, Laurent Daudet, Dominique Fourer, Laurent Girin, Antoine Liutkus, Jonathan Pinel, Gaël Richard, Nicolas Sturmel, and not least Shuhua Zhang. In particular, I want to thank Dominique for lending me an ear in both scientific and personal matters and for his loyalty. Cordial thanks to Jonathan for sharing the results of his work. Further, I would like to thank Gaël for teaching me about competition in academic research as much as Laurent Girin for sharing insight into originality of work, authorship, and the optimality of the Wiener filter.

My deepest gratitude goes to all the team assistants who made my life easier. Thank you Nicole Lun, Lebna Mizani, Brigitte Cudeville, Maïté Labrousse, Elia Meyre, Isabelle Garcia, and Philippe Biais. Not to forget are Cathy Roubineau and Magali Hinnenberger. Thank you both for your help. I could talk to you for hours. Beyond, I also want to mention Yves Métivier, Nicolas Hanusse, and Christophe Bavard (the latter two from the doctoral school), as the ones who provided me with financial support for a conference in Paris and the research stay in London. Many thanks to Elodie Duru, Christian Massus, and Jacky Chartier from "Aquitaine Science Transfert" for their support with the French paperwork. Also, I would like to thank Ben Dawson and DJ Vadim for their cooperation on the multitracks.

Last but not least, I would like to thank Joana for being the anchor in stormy weather and for not letting me lose my mind completely Most, if not all, of professionally produced music recordings have undergone two processes, mixing and mastering, and many common distribution formats, such as the Compact Disc Digital Audio (CDDA), are strictly stereo. While the term "mastering" refers to the process of optimizing the final mix and transferring it to a data storage device, the term "mixing" refers to the process of putting multiple layers of prerecorded and edited audio together to a composite mixture. Each layer can be imagined as a track in an audio mixer, which then again stands for a sound source. A source can be a vocal, an instrument, or a group of the aforesaid. The apparent placement of sources between the speakers in a stereo sound field is also known as "imaging" [1] in professional audio engineering.

Audio engineering is an established discipline employed in many areas that are part of our everyday life without us taking notice of it. But not many know how the audio was produced. If we take sound recording and reproduction or broadcasting as an example, we may imagine that a prerecorded signal from an acoustic source is altered by an audio engineer in such a manner that it corresponds to certain criteria when played back. The number of these criteria can be large and usually depends on the context. In general, the said alteration of the input signal is a sequence of numerous forward transformations, the reversibility of which is of little or no interest. But what if one wished to do exactly this, that is to reverse the transformation chain, and what is more, in a systematic and repeatable manner? It would allow the listener to interact with or reinterpret the music stored, e.g., on a Compact Disc (CD) and, hence, to become active. To render this possible, a music recording must be decomposed into its constituent elements-at least in part. . .

objectives 1.2.1 Active Listening

Active listening is a concept in music technology developed in the late '90s by Pachet and Delerue [2]. It aims at giving the user some degree of control of the music she or he listens to. As opposed to the usual case where a piece of music is simply played back without any conceptual modifications, active listening allows for a personalized music experience through reinterpretation. Its objective is to provide the listener with a higher level of musical comfort and to give access to new music by creating listening environments for existing music repertoires, in which the modifications preserve the semantics. One such example is the respatialization of sound sources or remixing by changing the volume and/or the panoramic angle of a distinct source. The idea of active listening is seized in works like [3,4,5,6], e.g. So, to enable active listening, one must reacquire access to latent source components given the mastered mix, i.e. one must reverse engineer the mix.

Reverse Audio Engineering

The objective of reverse audio engineering can be either to identify the transformation parameters given the input and output signals as in [7] or to recover the input signal that belongs to the output signal given the transformation parameters, or both. An explicit signal and system model is mandatory in either case. The second case may look trivial, but only if the transformation is linear and orthogonal and as such perfectly invertible. Yet, the forward transform is often neither linear nor invertible. This is the case for dynamic range compression (DRC), e.g., which is commonly described by a dynamic nonlinear timevariant system. The classical linear time-invariant (LTI) system theory does not apply here, so a tailored solution must be found instead. At this point, I would also like to highlight the fact that neither Volterra nor Wiener model approaches [START_REF] Ogunfunmi | Adaptive nonlinear system identification: The Volterra and Wiener model approaches[END_REF][START_REF] Avargel | Adaptive nonlinear system identification in the short-time Fourier transform domain[END_REF][START_REF]Modeling and identification of nonlinear systems in the short-time Fourier transform domain[END_REF] offer a solution and neither do describing functions [START_REF] Gelb | Multiple-input describing functions and nonlinear system design[END_REF][START_REF] Nuij | Higher-order sinusoidal input describing functions for the analysis of nonlinear systems with harmonic responses[END_REF]. These are powerful tools for identifying a time-invariant or a slowly varying nonlinear system, or for analyzing a feedback system with a static nonlinearity in regard to its limit cycle behavior. Furthermore, for the reason that the number of used tracks is usually greater than the number of mixture channels, mixing is in a mathematical sense underdetermined. Thus, demixing constitutes an ill-posed source separation problem.

organization

The remaining part of this work is organized as follows. Chapter 2 gives a brief overview of digital audio engineering including mixing an mastering. A system-theoretical classification of most commonly used studio effects and sound processors is also presented. Chapter 3 outlines the issues of inverse problems and explains why demixing is ill posed in the general case. In Chapter 4 it is shown how such an inverse problem can be addressed by means of Bayesian inference or statistical filtering. The notion of post-nonlinear mixtures is brought up as a special case of mixing and mastering. Chapter 5 is dedicated to the pursued "informed" approach. The major reverse engineering problem is defined and a feasible solution is offered. The following Chapter 7 deals with audio source separation, or demixing, which is equivalent to the inversion of mixing. There, a new spatial filter is derived. Its improved perceptual performance is confirmed in various experiments. An efficient coding strategy for the metadata is also presented. Apart from the separation of single-channel or point sources, it is shown how two-channel sources and/or their spatial images can be isolated from the mixture. It is argued that the respective source covariance matrix should be diagonal and it is shown why. A mixing system that incorporates a timing difference between the two channels for a more natural stereo effect is discussed as well. In Chapter 8, the proposed source separation algorithm is contrasted with MPEG's technology for interactive remixing. The performance of the former is assessed in a simulated active listening scenario. The inversion of mastering in the form of DRC is the central point of Chapter 9. Having defined the model of a compressor, it is demonstrated how the latter can be inverted using a novel approach. An exemplary compressor and the corresponding decompressor are given in terms of pseudocode. It is explained how the decompressor applies to a stereo mixture. The demixing and the decompression stages are combined into a cascade connection in the following Chapter 10. The cascade connection is analyzed with respect to some important performance characteristics like delay, complexity, and side-information rate and evaluated on an exemplary multitrack recording. This ends the main part of the thesis. In an extra Chapter 11, a proof of concept for a new multichannel object-based coding is presented which is a spillover from the previous chapters. With this coding scheme it is possible to control the sound quality at the decoder from the encoder in a measurable way. Chapter 12 concludes the work with a discussion and mentions possible directions for future work.

To sum up, the contribution of the thesis consists in:

1. The classification of digital audio effects 2. A statistical time-frequency model for single-channel and twochannel sound sources and their spatial images Digital audio effects and audio signal processors manipulate sound characteristics in three basic dimensions: volume, frequency and time. They are used to alter an audio signal's apparent volume, harmonic structure-or its waveform in general, so as to create sound patterns that are widely accepted and to satisfy personal taste of the composer or the engineer alike. The most common studio effects fall into one of the following categories: distortion effects which are based on gain, dynamics processors that control the apparent volume by altering the signal amplitude, filter effects which alter the frequency content, modulation effects which alter the amplitude, frequency, delay, or phase of the so-called "wet" signal over time, pitch and frequency effects that alter the pitch or create harmony, time-based effects that delay the so-called "dry" signal or add echoes, and finally spatial effects which alter the apparent direction of sound sources in a stereophonic sound field and give them the notion of spatial extent.

A list of common studio effects is given in Table 1. These are further classified based on their system-theoretical properties and grouped by the superscripts * , †, ‡, and §, which have the following meaning:

* Linear time-invariant (LTI) LTI systems are completely specified by their transfer function. † Linear time-variant (LTV)

LTV systems can be described by the Bello functions [START_REF] Bello | Characterization of randomly time-variant linear channels[END_REF]. Effects that fall into this category are barely used for music production. LTI effects can be regarded as a subclass of LTV effects. ‡ Nonlinear time-invariant (NTI) NTI systems are usually modeled with Volterra series [START_REF] Wiener | Response of a non-linear device to noise[END_REF]. Most NTI audio effects are non-invertible. § Nonlinear time-variant (NTV)

NTV systems require in general a special treatment. In the following, effects are considered as deterministic and casual. Effects with memory are lumped, i.e. their number of state variables, or taps, is finite. The focus is laid on three types of effects including panning and balance control, equalization, and compression during mastering. They are printed in boldface in Table 1. It should yet be noted that "reverb" is the most-used effect in the studio, as we are used to hearing sounds in enclosed spaces. However, on account of its complexity, dereverberation is left out of this work for a separate study. For more details on digital audio effects refer to [1,[START_REF] Zölzer | DAFX: Digital audio effects[END_REF], e.g.

mixing

The mixing stage is formulated using the following notation:

x l (n) = I i=1 A li [s li (n), θ li (n)] with l ∈ {1, 2}, (1)
where s li (n) is the ith console input routed to the lth channel, θ li (n) are the time-varying mixing parameters belonging to the composite effects operator A li applied to s li (n), and x l (n) is the output from the lth mixture channel. The mixture is thus a superposition of I distinct audio tracks. The input signals can be single-channel or two-channel, while the composite operator can be a cascade of effects. Effects can be linear or nonlinear, time-invariant or time-variant, instantaneous (memoryless) or dynamic (with memory), and may have a different number of inputs and outputs, see Table 1. Since I is usually greater than two, the mixing process is also referred to as "fold-down" or as "downmix".

mastering

On the analogy of (1), the mastering stage is formulated as:

y l (n) = A l [x l (n), θ l (n), . . . ], (2)
where θ l (n) now represents the time-varying mastering parameters belonging to the composite effects operator A l applied to x l (n). The composite operator can also have any other mixture channel or an external sound source as input parameter as indicated by ". . .". This allows to describe more sophisticated effects like the compressor. As a consequence, y l (n) denotes the mastered output.

I L L -P O S E D I N V E R S E P R O B L E M S

inverse problems

Inverse problems arise in situations where the model parameters of a system need to be inferred from measurements, and this with or without prior knowledge of its internal structure. They represent the counterpart to the ordinary forward problems, which can be solved using signal and system theory. Whereas forward problems have a unique solution, inverse problems may have many or no solution at all.

Many inverse problems are formulated as optimization problems. The sought-after model parameters are usually those that best fit the measurements or observations under an optimization criterion. As a rule, a hypothesis on the distribution of the error is made. When the latter is Gaussian, i.e. f(x) ∼ exp -x 2 , the sum of the squared errors or misfits is minimized to gain the best solution in the least-squares sense. The reason why the method of least squares is so popular lies in its simplicity. This is also why many problems in signal processing are considered to be of Gaussian nature. Beyond, the method of least squares allows for a closed-form solution.

When dealing with inverse problems, two issues are encountered. The first is to find a model of the system which is consistent with the observed data. The second is to quantify the non-uniqueness of the solution.

ill-posedness

As stated earlier, the system of equations resulting from mixing is generally underdetermined. And even though a forward operator in the form of a linear mixing system can be deduced from the design of a typical digital audio workstation (DAW) almost surely, the source separation problem is non-unique and so "ill posed" in Hadamard's terminology. There are, theoretically, infinitely many different source components, or model parameters to be more accurate, that could fit the observed mixture. Mastering poses additional problems. Ill-posed inverse problems often require to be reformulated, so that they can be treated numerically. This involves making additional assumptions in regard to the solution, such as its smoothness. This is also known as "regularization". Regularized inverse problems can further be viewed as special cases of Bayesian inference. For further reading, see [START_REF] Tarantola | Inverse Problem Theory and Methods for Model Parameter Estimation[END_REF].

P R O B A B I L I S T I C A P P R O A C H

bayesian inference

In Bayesian statistics the evidence about the true state of the world is expressed in terms of degrees of belief or Bayesian probabilities. In consequence, Bayesian inference is a method that uses Bayes' rule to update the probability estimate for a hypothesis as more evidence is acquired about the world. It so provides the posterior probability as a consequence of two antecedents, a prior probability and a function derived from a probability model for the observations which is also called the "likelihood". The former is computed as

P (H | E) = P (E | H) P (H) P (E) , (3)
which is the Bayes rule. In (3), H is the hypothesis, the probability of which is affected by the observations or evidence E. P(H) is thus the prior probability, i.e. the probability before E is observed. P(E | H) is the probability of observing E given H-the likelihood, while P(E) is termed the marginal likelihood or also model evidence. The updated prior probability P(H | E) is equal to the posterior probability, i.e. the probability of H after E is observed. This tells us the probability of a hypothesis given the observed evidence. Put into words, posterior is proportional to prior times likelihood.

The application of Bayesian inference to inverse problems may be illustrated with the following example. Let s(n) denote an unknown input signal corrupted by an interferer r(n) and x(n) the observable signal, x(n) = s(n) + r(n). Let S and R be two independent normally distributed random variables with zero mean, and s(n) and r(n) the respective realizations. Let s(n) span the one-dimensional parameter space. The prior distribution over s(n) be p(s | θ) = f S (s; 0, σ 2 s ), where θ = 0, σ 2 S is the set of hyperparameters-the parameters belonging to the prior. The probability density function of the interferer is also the likelihood to observe

x(n) given s(n), p(x | s, θ) ≡ f R (r; 0, σ 2 R ).
As the prior and the likelihood are both normal, the posterior is normal and the Bayes estimator for the input signal is

ŝ(n) = σ 2 S σ 2 R + σ 2 S x(n), (4)
which is the mean of the posterior distribution p(s

| x, θ) with s | x ∼ N σ 2 S/(σ 2 S +σ 2 R )x, σ 2 S σ 2 R/(σ 2 S +σ 2 R ) ∝ p(x | s, θ) p(s | θ).
It should further be noted that there exist other methods of Bayesian estimation to select elements of central tendency from the posterior distribution. The one presented above is the (linear) minimum mean square error (MMSE) estimator. Taking a closer look at (4), one may recognize the transfer function of the classical Wiener filter. The latter is an MMSE estimator in the frequency domain. Further details on the Bayesian approach to inverse problems can be found in [START_REF] Idier | Bayesian Approach to Inverse Problems[END_REF].

linear optimum filtering

The problem of linear optimum filtering consists in designing any type of linear discrete-time filter whose output provides an estimate of a desired response given a set of input samples. The optimization criterion is usually chosen as the mean squared error (MSE) between the desired response and the actual response, which is the Bayes risk or the cost function to be minimized. The choice of the MSE criterion leads to a second-order dependence of the cost function on the filter coefficients. The cost function has so a global minimum that defines the optimum operating point. Linear optimum filters that minimize the MSE of the estimation are commonly known as Wiener filters. In solving the minimization problem, there are no constraints imposed on the solution.

To find the optimum filter coefficients, the Wiener-Hopf equations must be solved. They model the relation between the autocorrelation of the input samples and the cross-correlation between the latter and the desired response. The Wiener-Hopf equations are much simpler to solve for transversal i.e. finite impulse response (FIR) filters. Their solution in the compact matrix form is given by [START_REF] Haykin | Adaptive Filter Theory[END_REF]

w o = R -1 p, ( 5
)
where p is the cross-correlation vector and R is, correspondingly, the autocorrelation matrix which is assumed nonsingular. The optimum filter weight are represented by the vector w o . It can be seen that the computation of the optimum filter requires knowledge of R and p.

The optimization problem described above is typically of temporal nature. Its spatial version can be found in beamforming applications. There, e.g., a linear array of uniformly spaced antenna is illuminated by an isotropic source located in the far field. At a given time instant n, a plane wave impinges on the array along a direction specified by the angle α with respect to the perpendicular to the array. Then, the beamforming problem would be, e.g., to obtain the source signal with a high signal-to-noise ratio (SNR) by taking a model of the wave field into account. The formulation of an optimum beamformer is subject to the same optimization problem. This fact will serve us later. So, at long last, let us recall two typical beamforming quantities:

1. The azimuth-dependent beam response or gain is defined as g(α) w ❍ a(α), (6) where a(α) is also called the "steering" vector. Superscript ❍ is the Hermitian or conjugate transpose operator, as the filter can be complex. The beampattern is the magnitude-squared gain.

The spatial power spectrum

P(α) w ❍ (α)Rw(α) (7)
is a measure for the mean total signal power received from the direction α.

In beamforming applications, the steering vector a depends on the underlying array geometry. Here in this work, however, it is detached from a physical sound propagation model and shall only define how the sound power is distributed across channels. The formal definition of R, p, and a in a particular scenario will be given in Chapter 7.

post-nonlinear mixtures

Post-nonlinear mixtures are special cases of nonlinear mixtures. In [START_REF] Taleb | Source separation in post-nonlinear mixtures[END_REF] a proof is provided which shows that it is not possible to handle a nonlinear mixture without nonlinear distortion. On the other hand, post-nonlinear mixtures, which in the cited work are linear mixtures followed by a nonlinear distortion function without memory, can be treated with the same indeterminacies as if these were purely linear. They model systems in which the transmission channel is linear, but the sensor array is not, and hence, it represents a source of nonlinear distortion. The two conditions to be fulfilled are that the sources are mutually independent and that the nonlinearities are both invertible and differentiable functions. The approach requires knowledge of the so-called "score" functions that represent the type of distortion. The score functions are estimated from the nonlinear mixture as they are unknown in the general case. The conclusion is that the separation is impossible without additional prior knowledge of the model, as the independence assumption is not sufficient in the nonlinear case. This result is of particular importance, because music recordings fall into that category. Further details on post-nonlinear mixtures are given in [START_REF] Jutten | Advances in nonlinear blind source separation[END_REF] and the references found therein.

As a side note, audio source separation techniques so far consider exclusively linear mixing. Recently, efforts have been made to deduce a mixing model that takes the complete music production chain into account [START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF]. There, it is argued in favor of a linear model that unifies linear effects such as reverberation with nonlinear processing such as dynamic range compression. However, the motivation for the model is to undo the mixing taking mastering into account but not to undo the mastering as such.

I N F O R M E D A P P R O A C H

basic idea

In his 2005 tutorial [START_REF] Knuth | Informed source separation: A Bayesian tutorial[END_REF], Knuth takes the Bayesian approach for the design of robust source separation algorithms that take advantage of prior knowledge of the problem at hand to assure that one reaches an optimal solution. He calls it informed source separation to distinguish the approach from blind source separation where little is known. The basic idea of his informed approach consists in introducing the prior information about the problem, which can be a physical property and physical law alike, into the solution. This is surely more specific than Bayes' priors which in the proper sense are distributions. The model that accompanies such an informed source separation problem is so a combination of prior distributions and prior information. It should be noted that there are a number of so-called score-informed source separation techniques which also use this paradigm [START_REF] Ganseman | Source separation by score synthesis[END_REF][START_REF] Hennequin | Score informed audio source separation using a parametric model of non-negative spectrogram[END_REF][START_REF] Ewert | Using score-informed constraints for NMF-based source separation[END_REF]. Most commonly, one of many non-negative matrix factorizations (NMFs) is made use of to approximate the spectrograms of the original signals, so as to apply weighted binary masks or linear Wiener filters to the mixture spectrum. The binary masks can also be interpreted as sums of delta distributions in a probabilistic sense.

concretization

Knuth's idea can be applied to the reverse engineering problem in the following manner. First of all, the mixture shall be considered as post-nonlinear, assuming that linear mixing is followed by nonlinear mastering. So, the mixing and the mastering can be described by two deterministic systems in cascade. The probabilistic component in the model shall be due to the ill-posedness of the task. The probability of hearing the source of interest in the mixture is given by the posterior distribution of the respective source signal. The knowledge of how a music recording is produced represents the prior knowledge that we have. This is typically the effects used and the spatial positioning of sources. The prior information comprises the system parameters, i.e. the filter coefficients and the panoramic angles or directions, and the parameters of the source distributions, i.e. the signal parameters. The latter are the hyperparameters according to Bayes.

Problem Formulation

From the previous section it is clear that the best possible estimate of the source signals is achievable only with prior information in the form of system and signal parameters being perfectly given. On that account, the problem to solve is stated as follows. Given the mixing, mastering, and source models, recover the source signal estimates in best possible quality from the mixture. The signals' parameters shall be estimated from the accessible original source signals. The mixing and mastering parameters shall be deemed as known.

Proposed Solution

The problem as it stated above makes sense only if the recovery of the source components is carried out without access to the originals. Otherwise the problem is trivial. The discussed approach is hence to be applied in a scenario, where the "collection" of prior information is uncoupled from its employment. Once the sources are mixed, they are no longer available, i.e. One could also characterize the approach by the temporally and locally bounded access to the source signals. So, one needs to differentiate between the process of content creation and the process of content consumption. The content creator is responsible for providing all the necessary data to the content consumer, so that the latter is enabled to unnoticeably reverse engineer the music mix and to remix the content ad libitum with the help of a graphical interface. This is realized, e.g., in an encoder/decoder framework. The task of the encoder is to extract a minimum of ancillary data from the source signals, so the decoder can recover their replica from the mixture in high-preferably perceptual-sound quality. This metadata, if small enough, can either be hidden in the mixture signal itself in the form of an inaudible watermark or must be attached to the mixture.

Associated Problem

Aside from the obvious reverse engineering problem, there is also the problem of data reduction with respect to the model parameters, which is not negligible. The problem is even more challenging when the metadata is to be hidden in the mixture or when the decoder is to perform in real time. It is also worth questioning what a sufficient data rate is for a good system performance.

P R E V I O U S W O R K S

About a decade ago, in 2003, Avendano has presented a scheme [START_REF] Avendano | Frequency-domain source identification and manipulation in stereo mixes for enhancement, suppression and re-panning applications[END_REF] similar to [START_REF] Jourjine | Blind separation of disjoint orthogonal signals: Demixing N sources from 2 mixtures[END_REF] with which one can identify, separate, and manipulate individual sound sources in a studio music recording. His scheme uses a "panning index" to identify collinear source components and clusters those into coherent time-frequency (TF) regions [START_REF] Avendano | Frequency domain techniques for stereo to multichannel upmix[END_REF][START_REF]A frequency-domain approach to multichannel upmix[END_REF]. These regions can then be manipulated by applying a "mask" that alters the magnitude of the signal components in question. In that manner one can either attenuate or accentuate the source of interest-the vocal or an instrument-and change its apparent location. These features, otherwise known as karaoke, mix-minus, and repanning, are all basic elements of active listening. Avendano's scheme, which is applicable to convolutional stereo mixtures without any restrictions with regard to the number of superposed sound sources, has one drawback: the resulting sound quality is insufficient for professional applications. A similar technique for binaural recordings has been developed by Mouba and Marchand in 2006 [30].

In order to attain a higher quality as compared to Avendano, Oh et al. presented in [START_REF] Oh | Enhancing stereo audio with remix capability[END_REF] a model-based remixing scheme that likewise allows for gain manipulations and repanning of distinct sound sources, but with the aid of additional information. The latter consists of the mixing parameters and the approximate short-time power spectral densities (STPSDs) of the sources that are to be manipulated. This information, which is transmitted alongside the stereo mixture signal, is used to best fit the remixing model in the least squares (LS) sense given new user-definable mixing parameters. The authors claim their technique to require less side information than other comparable schemes such as MPEG's spatial audio object coding (SAOC) [START_REF] Engdegård | Spatial audio object coding (SAOC) -the upcoming MPEG standard on parametric object based audio coding[END_REF][START_REF]ISO/IEC, Information technology -MPEG audio technologies -Part 2: Spatial Audio Object Coding (SAOC)[END_REF][START_REF] Herre | MPEG spatial audio object coding -the ISO/MPEG standard for efficient coding of interactive audio scenes[END_REF] to achieve the same effect, as only the STPSDs of a few selected sound sources and their mixing coefficients need to be communicated to the remixing unit. However, if the user was intended to be given the possibility to alter the entire mix, the required amount of side information would coincide with the one of SAOC.

SAOC is an object-oriented extension to spatial audio coding (SAC) [START_REF] Herre | The reference model architecture for MPEG spatial audio coding[END_REF] which combines efficient coding of audio objects and interactive rendering during playback. In MPEG's terminology, an audio object is synonymous with a sound source. SAOC so comprises two integrated parts: MPEG surround (MPS) [START_REF] Herre | MPEG surround -the ISO/MPEG standard for efficient and compatible multi-channel audio coding[END_REF] as the base technology and an object transcoder as superstructure. Using SAOC, multiple audio signals are transmitted in the form of a single-channel or a two-channel mix, transcoded to the MPS format, and rerendered; all that with the help of object metadata and rendering information. Additionally, the mix is perceptual-entropy, i.e. "lossy", coded. When operated in karaoke, or solo, mode, however, the SAOC transcoder engages in addition an object decoder. The purpose of the latter is to decompose the mixture into objects that can be manipulated individually and with a higher precision before rerendered to MPS. The additional prior information that is necessary to carry out the decomposition includes the STPSDs and the mixing parameters, including panning and gain. The correct MPEG terms hereof are "object level differences", "downmix channel level differences", and "downmix gains". These quantities are hence provided by the object encoder.

As of today, SAOC is maybe the most versatile object-based coding scheme in comparison with the aforementioned schemes that allows for interactive remixing. This is certainly one of the many reasons it has become international standard in 2010. Even so, although mono and multichannel objects can be handled in SAOC, the mixing model is very basic and thus not representative for professionally produced music recordings. Nonlinear processing such as, e.g., dynamic range compression is not taken into account and so it is not inverted.

A method to invert dynamics compression is described in [START_REF] Lachaise | Inverting dynamics compression with minimal side information[END_REF], but it requires a gain value to for each sample of the compressed signal, which is transmitted. To provide a means to control the data rate, the gain signal is subsampled and entropy coded. This approach is very generic but inefficient, as it does not rely on a compressor model.

On the other hand, transmitting the uncompressed signal together with a few typical compression parameters like threshold, ratio, attack, and release would require a much smaller capacity and yield the best possible signal quality with regard to any thinkable measure. A more realistic scenario is when the uncompressed signal is not available on the consumer side. This is the usual case for studio music recordings where the listener is offered a signal which is meant to sound "good" to everyone.

The Dolby solution for broadcast loudness issues, e.g., consists of the transmission of metadata that can be used to normalize loudness across programs and channels [START_REF]Broadcast Loudness Issues: The Comprehensive Dolby Approach[END_REF]. The metadata, that helps control the program's dynamic range, is optimized on the broadcaster side and transmitted alongside the broadcast signal. This is a convenient solution for broadcasters, not least because the metadata is compact. Dynamic range adjustment is, yet, another forward transform rather than a true inversion. Evidently, none of the existing solutions satisfy the reverse engineering objective of this work.

In recent years several methods have been proposed that address audio source separation in an "informed" scenario [START_REF] Parvaix | Informed source separation of linear instantaneous under-determined audio mixtures by source index embedding[END_REF][START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF][START_REF] Gorlow | Informed source separation: Underdetermined source signal recovery from an instantaneous stereo mixture[END_REF]. The reason for this trend is the plain fact that after decades of research "blind" or rather "semi-blind" source separation approaches to this day yield unsatisfactory quality with respect to what is considered as professional audio applications-for which quality is key; for an overview and general concepts of blind approaches see [START_REF] Comon | Handbook of Blind Source Separation: Independent Component Analysis and Applications[END_REF]. Blind speech separation techniques, on the other hand, often rely on a speech production model and/or make specific assumptions, which cannot be generally upheld for music [START_REF] Makino | Blind Speech Separation[END_REF]. But what is worse is that many sophisticated techniques are not applicable if the separation problem is ill-posed, which is when the number of observations (channels) is smaller than the number of unknowns (sources). Illposedness is yet the normal case for most music recordings, since the content is still distributed and consumed primarily in two-channel stereo format. The concept of informed source separation (ISS), hence, may be seen as a way of overcoming the limitations of blind source separation (BSS) found in today's state-of-the-art algorithms.

As an example, the idea of informing a separator with the mixing parameters and the STPSDs, as in Oh et al.'s scheme or SAOC, can also be found in [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF]. There, however, this additional information is used to calculate a generalized Wiener filter for each source component in each channel separately. This type of MSE based interference reduction takes account of the power relations between the source signals but not their spatial diversity, and neither does it invert the mixing system. . .

A U D I O S O U R C E S E PA R AT I O N

intensity stereophony

Let a stereo system be considered in which one or multiple mono signals are unevenly distributed over two independent channels, in such a way that an illusion of directionality and audible perspective is created. This is achieved by varying the amplitude of the signal sent to each channel relative to the listener. The parameters that control this relative amplitude are direction and volume, see Fig. 1. They are equivalent to the position of the panoramic potentiometer, the panpot, and the fader position on a mixing desk in a recording studio and are applied to each mono signal separately. The summation of all panpotted and volume adjusted mono signals constitutes the sound field of the mixture. This type of "artificial" stereo can be viewed as the counterpart to X-Y recording using two bi-directional microphones perpendicular to each other, and so, forming a Blumlein Pair. Albeit simple, the sonic image created is very realistic.

signal model

The source signals are modeled in the time domain as zero-mean normal stochastic processes that are mutually independent and nonstationary. Joint wide-sense stationarity is nevertheless assumed for the duration of a short time segment. The STPSD is used as a measure for how the mean signal power, or variance, distributes over time and frequency. 1 In line with this, a short-time Fourier coefficient represents a circular symmetric normal random variable:

2 s i (n, m) ∼ N 0, σ 2 s i (m) S i (k, m) ∼ CN[0, Φ s i (k, m)], (8)
where n is the time instant, m is the time segment, s i (n, m) is the ith source signal in the mth time segment, σ 2 s i is the variance, k denotes the frequency bin, and Φ s i (k, m) is thus the STPSD. 3 The symbol indicates the discrete Fourier transform (DFT). In the shorttime Fourier transform (STFT) domain, for a given time segment, the set of source signal components is considered mutually independent, too. The sources and their components are thus uncorrelated, i.e. the short-time cross-spectral densities (STCSDs) are zero. The signals are all single-channel, i.e. mono. Resorting to common studio practice, a sound source is assigned a location in the sound field via pan control. So, the image signal is4

Left Right

s I s 2 s 1

Volume (dB) Direction (°)

u i (n, m) = a 1i e 1 s i (n, m) + a 2i e 2 s i (n, m) = a i s i (n, m) ✂ u i (k, m) = a i S i (k, m), (9)
where {e 1 , e 2 } is the standard basis of R 2 , a i = [ a 1i a 2i ] ❚ ∈ R 2 is the panning vector, s i (n, m) ∈ R, and S i (k, m) ∈ C. The panning vector is defined as

a i a i a i =   sin α i cos α i   with α i ∈ [0 • , 90 • ], (10)
representing the spread of the source into the sound field. A source is placed at the panoramic angle α i between fully left and fully right as illustrated in Fig. 1. The placement can be chosen either arbitrarily or following some common mixing rules. The signal's volume can be considered inherent to the signal, i.e.

s i b i s i0 , (11)
where s i0 is the reference-level signal and b i ∈ R represents the desired volume level. Furthermore, due to the fact that

a i 2 = sin 2 α i + cos 2 α i = 1, (12)
the source power level is kept constant across the two channels. The angle range is defined in such a way that at the lower end the source appears in only the right channel, whereas when placed at the upper end the source appears in only the left channel. In the middle, the signal power is equally distributed across the two channels and the source appears in a phantom center channel. A source is deemed to be unique among all sources if the associated angle is unique.

Mixture Signal

The mixture is obtained by superposition of distinct stereo images created according to [START_REF] Avargel | Adaptive nonlinear system identification in the short-time Fourier transform domain[END_REF]. To account for professionally produced music recordings, each source signal is considered as having undergone prior processing in the form of linear and nonlinear audio effects [START_REF] Sturmel | Linear mixing models for active listening of music productions in realistic studio conditions[END_REF]. Yet, the effects are not included in the model. The mixture signal is

x(n, m) = i∈■ a i s i (n, m) ✂ x(k, m) = i∈■ a i S i (k, m) = As(k, m), (13)
where x = [ x 1 x 2 ] ❚ , s = [ S 1 S 2 ••• S I ] ❚ , x = [ X 1 X 2 ]
❚ , and I is the total number of sources. Note that the mixing system A = [ a 1 a 2 ••• a I ] is time-invariant and memoryless, i.e. instantaneous.

problem formulation

What is sought after is an operator F that transforms the mixture signal, x(n), into the source signal estimate, ŝi (n), based on a priori knowledge about the mixing given a measurable signal characteristic which is perceptually relevant. The mixing system is fully described by the sources' locations and their filter responses. The signal metric provided by the source model is the STPSD. The latter is particularly suited for the task, because it can be interpreted in different ways: in a statistical, physical, and also perceptual sense. So, postulating the preservation of the original STPSD in the estimate, the problem can be formulated as follows. Given the model parameters

θ i = {α i , Φ s i (k, m)} ∀ i, k, m, (14) find ŝi
(n) = F [x(n) | θ 1 , θ 2 , . . . , θ I ] (15a) subject to Φ ŝi (k, m) = Φ s i (k, m) ∀ i, k, m. (15b)

proposed solution

In order that the source signal components exhibit a better disjoint orthogonality [START_REF] Rickard | On the approximate W-disjoint orthogonality of speech[END_REF] in comparison to the waveform domain, the mixture is mapped onto the Fourier time-frequency representation (TFR). The transformed mixture is so expressed in terms of Fourier coefficients. A coefficient pair x(k, m) is then decomposed into its constituents parts by means of linear spatial filtering according to

Ŝi (k, m) = w 1i e 1 X 1 (k, m) + w 2i e 2 X 2 (k, m) = w ❍ i x(k, m), (16)
where

w i = [ w * 1i w * 2i ] ❍ ∈ C 2 is
the spatial filter. As the mixing system that we seek to invert can be complex in general, so is the filter. From a geometrical viewpoint, the beam in Fig. 2 is steered and amplified or attenuated, such that the signal component in the direction of α i is preserved whereas the contribution from the interfering sources is canceled out or at least minimized. In the latter case, the spatial filter shall be constrained to adjust the mean output power of the estimate to the power level of the original component. The filter response g(α i ) = w ❍ i a i from (6), where g ∈ C in the general case, can be used to that end. In the final stage, the filtered signal components are recombined into an isolated version of ŝi (n).

Well-Posed Case

The term "well posed" characterizes the case where the number of active sound sources in a TF point is at least one but not larger than the number of channels, which is two. In such a case, it exists one exact solution.

Unity-gain spatial filter

Suppose that the mixture x(k, m) is made up of a single directional source component, The beam is gain adjusted and directed such that the signal of interest is preserved and either one interferer is suppressed or, in case of multiple interferers, their mean total power is minimized.

x(k, m) = a i S i (k, m). (17)
By comparing ( 12) with ( 6) under the unity-gain constraint,

g(α i ) = 1, (18)
the source component S i (k, m) is extracted from the mixture x(k, m) by setting

w i = a i , ( 19
) so that Ŝi (k, m) = a ❚ i x(k, m). (20)

Zero-forcing spatial filter

Let two sources contribute to the mixture x(k, m) simultaneously:

x(k, m) = a i S i (k, m) + a l S l (k, m), (21)
where S i (k, m) is the signal of interest and S l (k, m) is the jammer. By enforcing identity for S i (k, m) as in [START_REF] Haykin | Adaptive Filter Theory[END_REF], and full cancellation of the jammer,

g(α l ) = 0, (22)
the sought-after weight vector becomes

w i = A -❚ g i , (23)
where

g i = [ 1 0 ] ❚ and A = [ a i a l ].
Applying the above procedure for both sources yields the separation matrix

W = A -❚ , (24)
and the source components are obtained by matrix inversion,

ŝ(k, m) = A -1 x(k, m) (25)
with ŝ = [ Ŝi Ŝl ] ❚ .

Ill-Posed Case

The term "ill posed" characterizes the case where a unique solution to the separation problem does not exist, that is when the mixture is composed of more than two source signals. An optimal solution that further complies with (15b) can be found instead by means of linearly constrained minimum-variance (LCMV) spatial filtering [START_REF] Green | A comparison of seismic array processing methods[END_REF][START_REF] Van Veen | Beamforming: A versatile approach to spatial filtering[END_REF]. What is considered as the jammer now is the sum of all interfering source signals. In consequence, the mixture x(k, m) is expressed in terms of two components:

-a unidirectional signal of interest and -a multidirectional jammer. The corresponding equation is

x(k, m) = a i S i (k, m) + r(k, m) (26a)
with

r(k, m) = l∈■,l =i a l S l (k, m). ( 26b
)
An estimate of the signal of interest Ŝi (k, m) is found by minimizing the mean jammer power, or the mean filter output power, along the direction of the signal of interest,

P(α i ) = w ❍ i R x (k, m)w i , (27)
subject to identity with respect to a given power value Φ s i (k, m). In other words, we seek after the weight vector that solves the quadratic optimization problem

w io = arg min w i P(α i ) s.t. g(α i ) = Φ s i (k, m)a ❚ i R -1 x (k, m)a i . ( 28
)
The solution to the above problem can be found by use of Lagrange multipliers, e.g., which yields

w io = R -1 x (k, m)a i Φ s i (k, m) a ❚ i R -1 x (k, m)a i . ( 29
)
When applied to the mixture signal x(k, m), the derived spatial filter will narrow the lobe of the jammer power spectrum, and the leakage from the interfering sources, thus, will be reduced. Due to the power constraint it is furthermore ensured that the mean estimate power is equal to the desired power value in every point of the TF plane. This is easily verified by plugging ( 29) into (27).

Narrowband Convolutional Case

Let the mixture be based on the narrowband assumption [START_REF] Zatman | How narrow is narrowband?[END_REF] which says that the channel's coherence bandwidth does not (significantly) fall below the bandwidth of a frequency bin. In that particular case, the channel transfer function is assumed constant over the frequency bin. This assumption holds for FIR filters if their order is strictly smaller than the STFT size, and it holds in good approximation for infinite impulse response (IIR) filters for which the impulse response decays sufficiently fast to a negligibly small value. A digital equalizer is well represented by such a system model. The respective mixture signal is

x(n, m) = i∈■ a i [h i (n) ⋆ s i (n)](m) ✂ x(k, m) ≈ i∈■ a i H i (k)S i (k, m) = AH(k)s(k, m), (30)
where h i (n) is a time-invariant filter response, ⋆ denotes convolution, H i (k) ∈ C is the filter transfer function, and H = diag (H 1 , H 2 , . . . , H I ). As a side note, binaural mixing is inherent to this model thanks to the relatively small order of a head-related impulse response (HRIR). 5 In that case, the term a i h i (n) above has simply to be replaced by h i (n), i.e. a multichannel filter response. If [START_REF] Mouba | A source localization/separation/respatialization system based on unsupervised classification of interaural cues[END_REF] is used instead of (13), the spatial filtering operation in ( 16) can be extended in such a manner that distinct source signals are separated and deconvolved as opposed to what has been plain demixing before.

Two-Channel Case

So far, a source has been considered as single-channel. To account for two-channel, i.e., stereo sources, the model from ( 8) is extended in the following manner:

  s 1i (n, m) s 2i (n, m)   ∼ N[0, Σ s i (m)]   S 1i (k, m) S 2i (k, m)   ∼ CN[0, Φ s i (k, m)], (31)
where Σ s i and Φ s i are 2 × 2 covariance matrices. The two channels of a stereo source may be interpreted as two separate mono sources, and a stereo source can be thought of as a centered image of a sound source that was recorded with two independent microphones.

A stereo source, i.e. its centered image, is positioned in the sound field via balance control according to

u i (n, m) = a 1i e 1 s 1i (n, m) + a 2i e 2 s 2i (n, m) = a i • s i (n, m) ✂ u i (k, m) = a i • s i (k, m), (32)
where s i = [ s 1i s 2i ] ❚ , s i = [ S 1i S 2i ] ❚ , and • denotes the Hadamard or entrywise product. The balance vector is defined as

a i a i a ref,i with a ref,i = a 1i if a 1i a 2i , a 2i otherwise. ( 33
)
The instantaneous mixture signal is thus given by

x(n, m) = i∈■ a i • s i (n, m) ✂ x(k, m) = i∈■ a i • s i (k, m) = A vec S(k, m), (34)
where

S = [ s 1 s 2 ••• s I ], A = [ diag a 1 diag a 2 ••• diag a I ],
and vec denotes the vectorization formed by stacking the vectors s i , i = 1, 2, . . . , I, on top of each other into a single column vector. A stereo source component is separated from the mixture by estimating the left-channel and the right-channel component simultaneously according to

ŝi (k, m) = W ❍ io x(k, m) (35)
with

W io = R -1 x (k, m) diag a i Φ s i (k, m) • diag [Φ s i (k, m)] ll [Φ s i ] ❍ * ,l diag a i R -1 x (k, m) diag a i [Φ s i ] * ,l 1 /2 l=1,2 , (36)
where [Φ s i ] * ,l is the lth column of Φ s i . Equation [START_REF] Herre | MPEG surround -the ISO/MPEG standard for efficient and compatible multi-channel audio coding[END_REF] thus represents the two-channel counterpart to [START_REF]A frequency-domain approach to multichannel upmix[END_REF]. This can be shown by replacing diag a i by a i and Φ s i by Φ s i (l = 1 in that case).

Noisy Case

In the case where the mixture is watermarked [START_REF] Geiger | Audio data hiding with high rates based on IntMDCT[END_REF][START_REF] Pinel | A high-rate data hiding technique for audio signals based on IntMDCT quantization[END_REF], the model in [START_REF] Bello | Characterization of randomly time-variant linear channels[END_REF], or [START_REF] Mouba | A source localization/separation/respatialization system based on unsupervised classification of interaural cues[END_REF], can be extended by a noise term in the following way:

x w (k, m) = x(k, m) + n(k, m), ( 37
)
where n is an additive noise component in the respective TF point. Due to the fact that high-capacity watermarking techniques exploit the on-frequency masking phenomenon, the noise term is assumed to be collinear with the noise-free mixture signal, which results in the following relation:

x w (k, m) = [1 + η(k, m)]x(k, m) (38)
with

n(k, m) = η(k, m)x(k, m), (39)
where η ∈ C represents the corruption due to the watermark. From [START_REF]Broadcast Loudness Issues: The Comprehensive Dolby Approach[END_REF] it can be seen that the estimate Ŝi (k, m) must be rectified by the term [1 + η(k, m)] -1 to compensate for the watermark. In the general case, however, η(k, m) will be unknown. We can yet give an estimate for the deviation of the magnitude with regard to a noise-free power value Φ s i (k, m), which is

|1 + η(k, m)| = Ŝi (k, m) Φ s i (k, m) . ( 40
)
This a posteriori estimate for magnitude distortion can then be used to partly compensate for errors due to watermarking.

7.6 general remarks

Precedence or Haas Effect

If a timing difference between the two channels of an image signal is wished for and the delay in samples is sufficiently small compared to the STFT length so that the two windowed signals carry almost the same content, a frequency-dependent phase shift can be added to the panning vector. The new vector is then complex (and so will be the spatial filter):

a i (n) =   sin θ i δ(n -n 1 ) cos θ i δ(n -n 2 )   a i (k) =   sin θ i ω -kn 1 cos θ i ω -kn 2   , (41)
where δ(•) is the Dirac delta function, ω is the Nth primitive root of unity, ω = e j 2π /N , and |n 2n 1 | is in the range of 1 to 5 ms [START_REF] Wallach | The precedence effect in sound localization[END_REF][START_REF] Haas | Über den Einfluss eines Einfachechos auf die Hörsamkeit von Sprache[END_REF]. In this way, the perceived width of the sound source can be increased.

Multi-Constraint Spatial Filter

If the constraints from ( 18) and ( 22) are also imposed on the filter in [START_REF] Avendano | Frequency domain techniques for stereo to multichannel upmix[END_REF], the obtained solution folds up to [START_REF] Ganseman | Source separation by score synthesis[END_REF]. This is simply because no degrees of freedom are left with regard to the number of weight coefficients to minimize the mean jammer power. As a consequence, the filter is suboptimal in the case of multiple interferers: Canceling out just one of the interfering sources, analogously to [START_REF] Ganseman | Source separation by score synthesis[END_REF], leaves a strong residual which is further amplified by the filter.

Wiener vs. PCMV Spatial Filter

It is well known that the classical Wiener filter minimizes the mean noise power at the cost of the signal of interest, which is to say that the signal of interest is also attenuated at the attempt to improve the SNR at the output. One can therefore expect the estimated signal spectra to be attenuated depending on whether the SNR in a TF point is high or low. As a direct consequence of this, the spectra of the source signals with a poor SNR exhibit missing spectral components after filtering, which may deteriorate the quality of the listening experience.

From the fact that the filter in [START_REF]A frequency-domain approach to multichannel upmix[END_REF] and the Wiener spatial filter are collinear, one can infer that their beams have the same look direction but different gains. The power-conserving minimum-variance (PCMV) spatial filter adapts the gain in order to conform with the quadratic constraint, whereas the Wiener spatial filter will likewise power down the output signal for the sake of a lower MSE, see Appendix A. The PCMV spatial filter, hence, is capable of overcoming the issue of spectral gaps, so that the replica of the original source signals are perceptually more similar to the latter in timbre. It also preserves the auditory signal bandwidth, which is essential for a natural listening experience. As a side note, this issue gave rise to various bandwidth extension techniques in the past, see [START_REF] Gorlow | Frequency-domain bandwidth extension for lowdelay audio coding applications[END_REF] and the references therein.

Mixture Covariance Matrix

The local mixture spatial covariance matrix is given by

R x (k, m) = E x(k, m)x ❍ (k, m) . ( 42
)
For an instantaneous mixture, the above expression yields

R x (k, m) = p∈P a p a ❚ p Φ s p (k, m) + q∈◗ a q a ❚ q • Φ s q (k, m) (43)
with

Φ s q (k, m) =   Φ s 1q (k, m) Φ s 1q s 2q (k, m) Φ * s 1q s 2q (k, m) Φ s 2q (k, m)   , ( 44
)
where the subset

P = {i ∈ ■ | ∀n[s 1i (n) = s 2i (n) = s i (n)]} contains the mono sources, while ◗ = {i ∈ ■ | ∃n[s 1i (n) = s 2i (n)]} = ■ \ P comprises
the stereo sources, respectively. Φ s 1q s 2q (k, m) is the qth source's, q ∈ ◗, STCSD between the left and the right channel, and Φ * s 1q s 2q (k, m) is its conjugate. R x (k, m) is nonsingular if there are (at least) two mono sources in a TF point contributing from different angles. 6 It is further Hermitian and positive-semidefinite.

To compute the PCMV stereo filter in [START_REF] Herre | MPEG surround -the ISO/MPEG standard for efficient and compatible multi-channel audio coding[END_REF], one requires the STCSDs, which can be calculated as the STFT of the block cross-covariances. To avoid the extra computational effort and to save on the data rate, one might consider skipping this step. As a result, (44) can be simplified as

Φ s q (k, m) = diag Φ s 1q (k, m), Φ s 2q (k, m) , ( 45
)
and the PCMV stereo filter then becomes

W io = R -1 x (k, m) diag [Φ s i (k, m)] ll R -1 x (k, m) ll 1 /2 l=1,2 . ( 46
)
Using ( 42) and ( 45), R x (k, m) can be reconstructed from the panning angles, the balance ratios, and the STPSDs exclusively-and so W io . From [START_REF] Rickard | On the approximate W-disjoint orthogonality of speech[END_REF] and [START_REF] Green | A comparison of seismic array processing methods[END_REF] it can be seen that when multiple stereo sources are present in the mixture, their component estimates have the same phase; only their spectral envelopes are shaped differently. Further, if the mixture is a combination of only stereo sources, W io is diagonal. As a consequence, each source component is filtered from either the left or the right channel using the PCMV mono filter

w io = Φ s i (k, m) l∈■ a 2 l Φ s l (k, m) , ( 47
)
which resembles Faller's filter in [START_REF] Faller | Parametric joint-coding of audio sources[END_REF]. Please note that the filter in [START_REF] Van Veen | Beamforming: A versatile approach to spatial filtering[END_REF] is not the square-root Wiener filter [START_REF] Vaseghi | Advanced Digital Signal Processing and Noise Reduction[END_REF], since the mixing coefficient a i (magnitude or square) is missing in the numerator. See also [START_REF] Pratt | Generalized Wiener filtering computation techniques[END_REF].

parameter quantization and coding

Here, a scalable quantization and coding strategy with a compact representation of side information is proposed. Other strategies, for spectrogram coding in particular, can be found in [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF]. Its advantage lies in the fact that it can be applied block by block, requiring hence less working memory.

Panoramic Angle

The panoramic angle α of a mono source is rounded to the nearest integer value using a mid-tread uniform quantizer defined as

Q(x) = sgn (x) • ∆ • |x| ∆ + 1 2 , ( 48
)
where ∆ is the step size and ⌊•⌋ represents the floor function.

Balance Ratio

The balance ratio a ¬ref /a ref of a stereo source is encoded using an A-law or a µ-law compressor in combination with the quantizer from [START_REF] Zatman | How narrow is narrowband?[END_REF]. For a given input x, |x| x max , the A-law compressor output is

C A (x) = sgn (x) •        A • |x| 1 + log (A) if 0 |x| x max A , x max • [1 + log (A • |x|/x max )] 1 + log (A) if x max A < |x| x max , ( 49
)
where A is the compression parameter and log is the logarithm. The output of the µ-law compressor is

C µ (x) = sgn (x) • x max • log (1 + µ • |x|/x max ) log (1 + µ) , ( 50
)
where µ is the associated compression parameter. Using A-law or µlaw compression, the signal-to-distortion ratio is kept constant over a broad range of x [START_REF] Itu-T | Pulse code modulation (PCM) of voice frequencies, November 1988[END_REF]. Common values for A and µ are 87.7 and 255, respectively.

Short-Time Power Spectrum

Irrelevancy reduction

A significant reduction of side information can be achieved in two steps: by reducing the frequency resolution of the STPSD Φ s i (k, m) in approximation of the critical bands [START_REF] Fastl | Psychoacoustics: Facts and Models[END_REF] and by quantizing the STPSD values in relation to an appropriate psychoacoustic criterion. The peripheral auditory system is typically modeled as a bank of overlapping bandwidth filters, the auditory filters, which possess an equivalent rectangular bandwidth (ERB). The ERB-rate scale puts into relation the center frequency of auditory filters with units of the ERB. Using the ERB-rate function [START_REF] Glasberg | Derivation of auditory filter shapes from notched-noise data[END_REF] we can define a relation between the frequency bin k and the critical-band index z k by

z k 21.4 log 10 (1 + 4.37f s /Nk) , ( 51
)
where N is the STFT length and f s is the sampling frequency in kHz.

The zth critical-band value is then computed as the arithmetic mean between lb(z)

= inf {k | z k = z} and ub(z) = sup {k | z k = z}, i.e. Φs i (z, m) = 1 ub(z) -lb(z) + 1 ub(z) k=lb(z) Φ s i (k, m). ( 52
)
Further, under the assumption that the the minimum just-noticeable difference (JND) level and so the maximum quantization error is 1 dB [START_REF] Fastl | Psychoacoustics: Facts and Models[END_REF], the step size ∆ is chosen as 2 dB, and the Φs i (z, m) values are quantized using (48) according to

Φ Q s i (z, m) 2 = 10 2 log 10 Φs i (z, m) + 1 2 . ( 53
)
In [START_REF] Gorlow | Informed source separation: Underdetermined source signal recovery from an instantaneous stereo mixture[END_REF] it is also shown that a perceptually motivated approximation of the STPSD is sufficiently precise to achieve high similarity with the original signals.

Redundancy reduction

To reduce the amount of side information even more, one can use the correlation of STPSD indices between adjacent TF points. This can be achieved by taking the difference between two consecutive STPSD indices in the direction of time, frequency, or between channel pairs, and by coding the difference signal based on its lower entropy. That principle is known as differential pulse-code modulation (DPCM). It is further advisable to use a nonuniform time resolution to better take time-varying aspects of music signals into account [START_REF] Princen | Audio coding with signal adaptive filterbanks[END_REF][START_REF] Sinha | Audio compression at low bit rates using a signal adaptive switched filterbank[END_REF].

Dequantization

As spatial filtering is carried out based on the availability of STPSDs, the quantized STPSD values must be converted back into their linear counterparts and extrapolated to the resolution of the STFT. This can be done by taking

Φs i (k, m) = 10 Φ Q s i (z,m) /10 ∀ k : z k = z. ( 54
)
7.8 performance evaluation

Instantaneous Mixture with Watermarking

In this section, the proposed algorithm which was given the name "Underdetermined Source Signal Recovery" (USSR) is compared with an in-house implementation of the algorithm described in [START_REF] Parvaix | Informed source separation of linear instantaneous under-determined audio mixtures by source index embedding[END_REF] and its predecessor [START_REF] Gorlow | Informed source separation: Underdetermined source signal recovery from an instantaneous stereo mixture[END_REF]. The four algorithms under test hence are:

issa The reference ISS algorithm [START_REF] Parvaix | Informed source separation of linear instantaneous under-determined audio mixtures by source index embedding[END_REF] ussr-a The predecessor of the proposed algorithm [START_REF] Gorlow | Informed source separation: Underdetermined source signal recovery from an instantaneous stereo mixture[END_REF] ussr-b The proposed algorithm using the PCMV spatial filter ussr-c The proposed algorithm using the Wiener spatial filter

Experimental setup

To exclude a performance bias due to different TFRs, all algorithms are implemented using the STFT. The latter is realized by means of a 2048-point fast Fourier transform (FFT) with a Kaiser-Bessel derived (KBD) window of the same length and a 50-% overlap. The sampling rate is set to 44.1 kHz. The effective data rate of ISSA is 86.1 kbps for the 5-track mixture and 108 kbps for the 7-track mixture. The USSR algorithm is adjusted in such a way that its data rate is more or less the same: 93.4 and 103 kbps. Also, the same watermarking algorithm [START_REF] Pinel | A high-rate data hiding technique for audio signals based on IntMDCT quantization[END_REF] is used in all four cases.

The following four metrics are used to objectively assess the sound quality: the signal-to-interference ratio (SIR), the so-termed "targetrelated" perceptual score (TPS), a frequency-weighted signal-to-noise ratio (SNRF) [START_REF] Tribolet | A study of complexity and quality of speech waveform coders[END_REF], and the "auditory" bandwidth as the counterpart of the "articulatory" bandwidth [START_REF] Tribolet | A study of complexity and quality of speech waveform coders[END_REF]. The first two metrics are computed with the PEASS toolkit [START_REF] Emiya | Subjective and objective quality assessment of audio source separation[END_REF]. The SNRF is redefined according to:

SNRF i (m) 1 Z Z z=1 10 log 10 Φs i (z, m) Φn i (z, m) , ( 55
)
where z is the ERB-scale index, Z = 39, Φs i (z, m) is the band's signal power, and Φn i (z, m) is the corresponding noise power,

Φn i (z, m) = ub(z) k=lb(z) min Ŝi (k, m) -|S i (k, m)|, 0 2 ub(z) -lb(z) + 1 . ( 56
)
The noise signal is calculated in such a way that the subjective effect of spectral gaps on sound quality is accentuated, as only lacking sound components are taken into account. Furthermore, a time resolution of 23.2 ms is used for both the SNRF and the bandwidth metric. The end result is obtained by taking the average over all time segments. As a supplement, a multi-stimulus test with hidden reference and anchor (MUSHRA) [START_REF] Itu-R | Method for the subjective assessment of intermediate quality level of coding systems[END_REF] is administered. It should serve as a check for consistency between the four chosen metrics and human perception. The test is carried out in the audiovisual production facilities within the University of Western Brittany. Sennheiser's HD 650 headphones and MOTU's UltraLite-mk3 Hybrid audio interface are used during the test. The gain of the preamplifier is adjusted to a listening level of -20 dB below the clipping level of a digital tape recording. The test signals are shortened to 20 s at the longest. The anchor is a 3.5-kHz lowpass filtered sum of the original tracks with 3 dB SIR and a 50-% spectral-gap rate. The anchor is altered in such a way that it shows similar types of impairment as the algorithms under test. A panel of nine audiovisual media students take part in the test. They are asked to score the stimuli according to the continuous quality scale, giving their degree of preference for one type of artifact versus some other type.

Two music pieces from different genres are selected: a 5-track hiphop mixture and a more complex 7-track pop-rock mixture. The hiphop piece is DJ Vadim's "The Terrorist". It is composed of a leading vocal, a synthesizer in the bassline, and a percussion section that has a kick, a snare, and a hi-hat. Phoenix's "Lisztomania" is chosen from within the pop-rock genre. It has a bass guitar together with drums forming the rhythmic section, several guitars in the harmonic section, a vocal melody, and a keyboard to create a sustained pad for the piece. The signals are 30 s long monophonic excerpts from the multitrack masters. The spatial placement of individual tracks is aligned with the commercial releases, see Table 2.

Experimental results

The results of the experiment are summarized in Figs. 34567. Figs. 34show the SIR and the TPS for each track from the two music excerpts. The corresponding SNRF and auditory bandwidth values are depicted in Figs. 56. The mean opinion scores (MOSs) with 95-% confidence intervals are plotted in Fig. 7.

As it was anticipated, USSR-C has the highest SIR. USSR-B shows a clear improvement over USSR-A. The SIR for ISSA is also quite high but always lower than for USSR-B and USSR-C, however. The TPS is fairly consistent in all three USSR variants for the hip-hop mixture, whereas a slight tendency towards USSR-B can be observed for the pop-rock mixture. ISSA has the worst TPS of all tested algorithms. In regard to the SNRF, the constrained USSR variants, A and B, perform better than the rest. Again, this is something that could be expected, since these algorithms preserve the auditory bandwidth of the signal. Further, it can be seen that the number of spectral gaps is smaller for USSR-C than for ISSA. De facto, the effect observed with USSR-C is more of a band limitation than the "arid" effect [START_REF] Gorlow | Informed source separation: Underdetermined source signal recovery from an instantaneous stereo mixture[END_REF], and as such it produces a sound that is rather "dull" than "annoying". Overall, the preferential tendencies of target-related perceptual score (TPS) are rather consistent with the MOS. Yet, the TPS seems to overrate the sound quality by some 20-40 points, which corresponds to 1-2 grades. In this regard, the SNRF provides the desired tendencies as well. This allows the conclusion that if the SNRF were properly scaled, it might just as well serve as an objective metric for the perceived sound quality, but at a much lower cost. With a mean score between "fair" and "good", the USSR algorithm is the clear winner in any of its variants. ISSA is graded as "bad" on the average, but better than the anchor. A slight preference for USSR-B, the proposed algorithm, over its predecessor USSR-A can also be noted. That preference seems to be linked with the complexity of the mixture. After all, USSR-B is assessed to perform significantly better than USSR-C, which once more highlights the fact that full bandwidth is essential for a natural listening experience.

Narrowband Convolutional Mixture

In this section, the variants B and C of the proposed algorithm are compared against each other by applying them now to a narrowband convolutional mixture. Variant C employs the MMSE alias Wiener spatial filter as opposed to variant B which uses the PCMV spatial filter, see also Section 7.8.1. In addition, different coding strategies for the spectrograms are applied. The PCMV spatial filter is used in tandem with the strategy from Section 7.7.3, whereas the MMSE spatial filter is used in conjunction with JPEG compression or NMF with low-rank modeling. Variant C is hence equivalent to the algorithm described in [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF] with the generalized Wiener filter replaced by its spatial counterpart, as in [START_REF] Liutkus | Informed audio source separation: A comparative study[END_REF].

Experimental setup

A set of fourteen musical excerpt from different genres is used. An excerpt is between 15 s and 35 s long and it comprises 5 to 10 tracks. Most of the excerpts have 5 tracks. The CIPIC head-related transfer function (HRTF) database [START_REF]The CIPIC HRTF database[END_REF] is used to simulate the channel, i.e. the transmission medium. The sources are placed at different angles separated by 5°or more from each other. Their placement is considered known at the decoder. The STPSDs are quantized and coded at different bitrates using one of the three strategies mentioned above. The HRTFs are derived from the pan angles taking the measurements for the KEMAR mannequin and are thus not included in the bitrate. The sampling rate is set to 44.1 kHz.

The objective sound quality is assessed with the aid of two metrics: the signal-to-distortion ratio (SDR) from BSS Eval [START_REF] Vincent | Performance measurement in blind audio source separation[END_REF][START_REF]BSS Eval: A toolbox for performance measurement in (blind) source separation[END_REF] and PEMO-Q's perceptual similarity metric (PSM). While the BSS Eval toolbox is used to measure the performance of "blind" source separation algorithms in the first place, the PEMO-Q [START_REF] Huber | PEMO-Q -a new method for objective audio quality assessment using a model of auditory perception[END_REF]71] software on the other hand is meant to predict the quality of low-bitrate speech and audio coders. Both metrics should hence complement each other.

Experimental results

The results of the experiment are summarized in Fig. 8. The SDR for the "oracle" MMSE spatial filter that has perfect knowledge of the spectrograms, or STPSDs, is shown in Fig. 8a and the PSM value for each of the fourteen excerpts is shown in Fig. 8b, respectively. As can be seen, both metrics vary significantly from excerpt to excerpt. From this one can infer that the estimator's performance depends on the number of sources, their spatial location, and their spectro-temporal overlap. Also, the metrics are inconsistent across excerpts. The mean oracle SDR for the dataset lies around 10 dB and the mean PSM value is approximately 0.85.

The "rate-performance" curves for the three different filtering and/ or coding strategies are given in Figs. 8c and8d for the two metrics. They are obtained by subtracting the oracle value from the respective SDR or PSM value for a track and by taking the average over all tracks and excerpts given a side-information rate. Although one could have expected that either of the two SDR-curves for the MMSE filter would overtop the SDR-curve for the PCMV filter, like the oracle, they do not. This raises the question of whether the proposed coding strategy is more efficient than the ones presented in [START_REF] Liutkus | Informed source separation through spectrogram coding and data embedding[END_REF]. The train of thought is as follows. In the previous experiment it was observed that the SIR is higher for the MMSE spatial filter if the same coding strategy is applied, see Figs. 34. Thus, one may conclude that if, e.g., NMF and DPCM coding contributed to the distortion to the same extent, the SIR for the MMSE spatial filter would also be higher in the convolutional case, and possibly the SDR as well. Looking at the PSM-curves, one can see that the combination of PCMV filtering with DPCM yields better sound quality than the oracle at any tested side-information rate. JPEG compression seems to have a higher coding efficiency than the NMF except for a very low bitrate region between 2-10 kbps per source in Fig. 8d. Also, it can be observed that the proposed algorithm provides roughly the same sound quality as the iterative predecessor algorithm [START_REF] Liutkus | Informed audio source separation: A comparative study[END_REF] at a much lower computational cost.

Separation of Spatial Images

In this section, the proposed algorithm is evaluated by applying it to a set of professionally produced music recordings from the SiSEC 2013 [72] website. The task is to decompose an artistic mixture into a subset of images that represent the foreground objects and the image of the background-where applicable. The term "background" refers to the sum of background objects. The original images are given as a reference.

Experimental setup

The following testing framework is used. With respect to the STFT, a 2048-point FFT is employed with a KBD window and a 50-% overlap between succeeding frames. The pan angle α is quantized with 7 bits while the balance ratio a ¬ref/a ref is quantized with 16 bits using the A-law compander with A = 87.6. The STPSD is quantized with 6 bits per value on a 76-band nonuniform frequency scale. The probability mass function of the difference between contiguous STPSD values is modeled with a Laplace (µ, b) distribution with µ = -0.2 and b = 2. The STCSDs are ignored by setting them to zero (no covariance).

The evaluation criteria suggested by the SiSEC 2013 committee are used. These include the performance metrics from the PEASS toolkit [START_REF] Emiya | Subjective and objective quality assessment of audio source separation[END_REF][START_REF]The PEASS toolkit -Perceptual Evaluation methods for Audio Source Separation[END_REF] and the decoder runtime in seconds times CPU clock rate in GHz. The side-information rate is also given. In addition, PEMO-Q's perceptual similarity metric PSM t and its objective difference grade (ODG) are computed as well [START_REF] Huber | PEMO-Q -a new method for objective audio quality assessment using a model of auditory perception[END_REF]71].

The SiSEC development set is used. It consists of five music pieces from different genres. The image parameters are estimated from the provided stereo tracks according to the protocol in Appendix B. The results are shown in Table 3 The results of the experiment are summarized in Table 4. As can be observed, the image-to-spatial distortion ratio (ISR) is between 6.66 and 17.4 dB for a stereo source and it is greater or equal to 18.5 dB for a mono source. Equally, the highest source-to-artifacts ratio (SAR) is obtained for a mono source, which is 27.7 dB. The TPS shows a weak correlation not only with the ISR and SAR but also with PEMO-Q's perceptual similarity metric PSM t , which then again does not take spatial hearing effects into account. The lowest TPS is at 52 %. The measured side-information rate is around 10 kbps per mono source or stereo channel. The execution time of the decoder is low and also faster than real time.

Table 5 compares the performance of the proposed algorithm with the figures reported in SiSEC 2011 for two oracle systems [START_REF] Sisec | [END_REF]. Their performance figures give an upper bound for binary masking based systems. In Table 5, positive delta values are in boldface. A significant improvement can be noticed for all items with regard to the SAR, up to 22.4 dB. The TPS is also higher in most cases, and so is the SDR. A comparison with more recent approaches is made available on the SiSEC 2013 website. A brief summary is given in Figs. 9 and10. Even though the bass and the drums estimates of the proposed algorithm are rated lower than other comparable algorithms, the corresponding mean ISR and TPS are clearly the highest. These are the most relevant metrics here. It can further be observed that six out of eight metrics show better-than-oracle performance for the proposed system. Only the two interference and artifacts related metrics are worse. This can be considered as the signal-noise uncertainty principle. The more of the signal is to be preserved in the estimate, the more noise (interference or artifacts) is to be accepted. Or, the less noise is desired in the estimate, the more of the signal is to be sacrificed. The SiSEC experiment finally proves that a binary-mask oracle is suboptimal. SAOC's own source separation scheme is referred to as "enhanced audio object separation" (EAOS), whereas the object decoder is also called an "enhanced audio object processor" [75, Fig. 6]. The latter consists of a two-to-I upmix unit [75, Fig. 5] which corresponds to I two-to-three (TTT) basic units [75, Fig. 4]. A TTT unit takes a twochannel mixture as input and outputs a mono signal from the object of interest, the foreground object (FGO), together with the background signal being the sum of all pan-potted object signals excluding the FGO signal. Every object that contributes to the background signal is further deemed a background object (BGO). A BGO is unalterable while an FGO can be altered in regard to its volume level and location. In applications where a single track is soloed or muted, like karaoke or mix-minus, an FGO is upgraded to a so-called "enhanced audio object" (EAO). An EAO signal is tantamount to an FGO signal that is error corrected and so quality improved using the residual, i.e. at the cost of a higher side-information rate.

object encoder

EAOS processes the object signals in the complex subband domain. The subband analysis is based on a hybrid filter bank that splits the time signal into 69 subband signals [START_REF] Schuijers | Low complexity parametric stereo coding[END_REF]. The STPSDs are computed as the instantaneous powers in each subband. These are quantized on a logarithmic scale and grouped over time and frequency. Finally, this metadata is DPCM coded and passed on to the decoder as side information along with the mixing coefficients and the downmix. The corresponding system model in the DFT domain is

X ❍ = I i=1 a i s ❍ i = AS ❍ , ( 57
)
where s i ∈ C K is a K-band signal vector, a i ∈ R 2 is the mixing vector, I is the number of audio objects, and X ∈ C K×2 is the mixture signal.

Superscript ❍ denotes Hermitian transpose. The I signal vectors and the I mixing vectors can be concatenated into the signal matrix S ∈ C K×I and the mixing matrix A ∈ R 2×I , such that the mixture matrix can be expressed as a product of the two matrices. The calculation of a single residual is as follows. First, all BGO signals are combined into a downmix signal [START_REF] Engdegård | Spatial audio object coding (SAOC) -the upcoming MPEG standard on parametric object based audio coding[END_REF],

X ❍ BGO = X ❍ -a FGO s ❍ FGO . ( 58
)
Second, an "auxiliary" signal for the FGO is calculated by taking the difference between the FGO and the BGO mixture signal projected onto the "look direction" of the FGO [START_REF] Engdegård | Spatial audio object coding (SAOC) -the upcoming MPEG standard on parametric object based audio coding[END_REF],

s FGO • = X BGO a FGO -s FGO . ( 59
)
Using ( 58) and ( 10), ( 59) can also be written as

s FGO • = Xa FGO -2s FGO . ( 60
)
Then, a linear combination of the (two) downmix channels is found that minimizes the reconstruction error between the modeled signal and the true auxiliary signal. For this, a system of K linear equations in two unknowns w FGO • = [ w 1,FGO• w 2,FGO• ] ❚ ∈ R 2 must be solved:

Xw FGO • = s FGO • . ( 61
)
The two coefficients that best fit the above equations in the LS sense are

ŵLS FGO • = X ❍ X Rx -1 X ❍ s FGO • pxs FGO• . (62)
The terms X ❍ X and X ❍ s FGO • are equivalent to the sample estimates of spatial correlation, Rx and pxs FGO• . For that reason, the LS estimate for w FGO • is formally identical with the MMSE estimator for the auxiliary signal. Due to [START_REF] Fastl | Psychoacoustics: Facts and Models[END_REF],

pxs FGO• = Rx a FGO -2 pxs FGO . ( 63
)
So, by rewriting [START_REF] Princen | Audio coding with signal adaptive filterbanks[END_REF] as

ŵMMSE FGO • = a FGO -2 R-1 x pxs FGO ŵMMSE FGO , (64)
the estimator can be put in direct relation to the FGO. Note that p xs FGO is the cross-correlation between the mixture and the FGO signal. The difference between the true auxiliary signal and its estimate,

r FGO • = s FGO • -X ŵMMSE FGO • , ( 65
)
yields the residual that is perceptual-entropy coded by the Advanced Audio Coding (AAC) [START_REF] Iso/Iec | Information technology -Generic coding of moving pictures and associated audio information -Part 7: Advanced Audio Coding (AAC)[END_REF] scheme at a bitrate of 20 kbps [START_REF] Falch | Spatial audio object coding with enhanced audio object separation[END_REF].

object decoder

To obtain the FGO signal, the auxiliary signal needs to be estimated first. To that end, the estimator from ( 64) is computed using the power spectra and the mixing coefficients. Both are made available for the decoder by the encoder. The correlation matrix is calculated as in [START_REF] Makino | Blind Speech Separation[END_REF]. Correspondingly, the cross-correlation vector is given by pxs FGO = a FGO Φ s FGO .

(66)

Again, the audio objects are considered mutually uncorrelated. Note that in SAOC, the cross-correlations between objects can be modeled using the inter-object cross coherences (IOCs). Yet the computation of these is optional, see Section 7.6.4. Once the estimator is computed it is plugged into [START_REF] Glasberg | Derivation of auditory filter shapes from notched-noise data[END_REF], and the enhanced auxiliary signal is obtained by adding the residual to the estimate:

sEAO • = X ŵMMSE FGO • ŝFGO• + rFGO • . ( 67
)
Solving ( 59) for s FGO using ( 58) yields the sought-after EAO signal

sEAO = 1 2 (Xa FGO -sEAO • ). (68)
Using ( 67) and ( 64), ( 68) can also be formulated as

sEAO = X ŵMMSE FGO -1 2 rFGO • . ( 69
)
The bottom line is that the TTT unit in SAOC is an MMSE estimator for the FGO signal with a particular residual coding strategy. This being the case, it fits perfectly into the ISS framework.

8.4 performance evaluation 8.4.1 EAOS vs. USSR

In the previous section it is shown that EAOS in SAOC uses a Bayes estimator in the form of an MMSE spatial filter to separate audio objects from their mixture. In this section, EAOS is compared with USSR using the same ISS testing framework.

Experimental setup

The STFT with a KBD window and 50-% overlap between segments is used for both systems under test. The PCMV spatial filter in USSR is replaced by the TTT unit when simulating EAOS. The metadata is encoded as in Section 7.7. The mixing coefficients are considered to be known. The two systems are compared with each other in terms of quality and computational complexity. The quality is assessed on the ODG scale [START_REF] Itu-R | Method for objective measurements of perceived audio quality[END_REF] while the complexity is measured by the decoder's execution time in MATLAB. The ODG score is computed with PEMO-Q. Ten multitracks taken from the QUASI database [START_REF]QUASI database -a musical audio signal database for source separation[END_REF] are converted to mono and cut down to 20-s excerpts. Each track is normalized to a reference root mean square (RMS) level of -16 dBFS. The sources are placed uniformly in space and gain adjusted, so as to have an equal SIR across the sources at the output, see Algorithm 4.

Experimental results

The results of the comparison are shown in Figs. 11-12. In the case where the residual is omitted, the bitrate is equivalent to the metadata rate for a varied number of parameter bands. In the case where the residual is used to correct the initial estimate, the metadata rate is fixed at 10 kbps and the bitrate is calculated as the sum of the latter and the residual rate increased from zero onwards. The results where the original signals are coded separately are also included. There, the bitrate is simply the coding rate.

The Freeware Advanced Audio Coder (FAAC) [START_REF][END_REF] is applied to the residual and the original tracks. The curves represent fitted averages over the data corpus. As can be seen from Fig. 11, USSR's PCMV spatial filter yields better results than EAOS's MMSE filter. This observation is consistent with the listening test results reported in Section 7.8.1. The gap between the two systems widens even further if their estimates are error corrected. At a bitrate of 30 kbps per source and beyond, the quality of FAAC-coded tracks is superior to those recovered from the mixture. As can be seen in Fig. 12, the USSR decoder is approximately half as complex as the EAOS decoder if all I estimates are error corrected. The longer runtime is explained by the fact that EAOS requires the residuals to be available in the subband domain whereas USSR does not. Hence, the loss of time is due to the extra I-fold STFT and the computation of auxiliary signals.

Interactive Remixing

In this section the ISS approach is evaluated in a concrete scenario. "Interactive remixing" is chosen as sample application. It allows the user to change the volume level of sound sources and their spatial location. Also, it is interesting to find out whether "plain" source coding is a more pragmatic solution with regard to audio quality and coding efficiency.

Experimental setup

The same testing framework as before is used. The ODG is retained as quality index. In order to simulate more realistic conditions, the stereo-to-mono converted tracks are panned to their original location, so that the downmix is prearranged as if by the sound engineer or the composer. The location is derived from the RMS channel difference in the original stereo tracks (see also Appendix B):

αi = arccot RMS 2i RMS 1i , ( 70
)
where arccot is the arccotangent. Table 6 provides a listing of the songs used in the experiment. The source signals are either recovered from the downmix using USSR or are encoded (and decoded) for comparison. For this, the FAAC and the Enhanced aacPlus [START_REF]ETSI, Digital cellular telecommunications system (Phase 2+); Universal Mobile Telecommunications System (UMTS); LTE; General audio codec audio processing functions; Enhanced aacPlus general audio codec[END_REF][START_REF]ISO/IEC, Information technology -Coding of audio-visual objects -Part 3: Audio[END_REF] coder are used. The downmix is encoded in perceptually transparent quality: at a variable bitrate in the region of 120 kbps. To simulate user interaction, ten different remixes with arbitrary new source locations and volume levels for each song and system are generated with the gains being in the range between -6 and 3 dB. The remixes of each system are then compared with the ones created from the original tracks.

Experimental results

The evaluation results are summarized in Fig. 13. It can be observed that the quality of a remix generated from an FAAC coded downmix depends on the number of sources and their spatial spread. A linearly pulse-code modulated (LPCM) mixture signal seems less sensitive to these factors. The deciding factor there is the "spectral texture" of a source signal and by how much it interweaves with other sources.

On an average, the best quality is obtained for an LPCM mixture in combination with residual coding at ca. 20 kbps per source. With a median not worse than "slightly annoying", the results gained with USSR alone are promising. Clearly worse is the grade for the scenario in which the mixture is FAAC coded. Even at a side-information rate of 30 kbps, the quality lies halfway between "annoying" and "slightly annoying". The same is true for Enhanced aacPlus at 10 kbps or FAAC at 30 kbps. The most efficient system in the experiment is Enhanced aacPlus operating at 30 kbps, as it does not necessitate availability of the mixture. Figure 13: The medians and the 25th and 75th percentiles for each system under test

I N V E R S I O N O F D Y N A M I C R A N G E C O M P R E S S I O N

dynamic range compression

Dynamic range compression or simply compression is an audio processing technique that attenuates loud sounds and/or amplifies quiet sounds, which in consequence leads to a reduction of an audio signal's dynamic range. The latter is defined as the difference between the loudest and the quietest sound measured in decibels. In the following, we will use the word "compression" having "downward" compression in mind, though the discussed approach is likewise applicable to "upward" compression. Downward compressing means attenuating sounds above a certain threshold while leaving sounds below the threshold unchanged. An audio engineer might use a compressor to reduce the dynamic range of source material for purposes of aesthetics, intelligibility, recording or broadcast limitations, etc. Fig. 14 shows the basic compressor model from [83, ch. 2] amended by a switchable RMS/peak detector in the side chain, which makes it compatible with the compressor/limiter model from [15, p. 106]. We will hereafter restrict our considerations to this basic model, as the purpose of the present work is to demonstrate a general approach rather than a solution to a specific problem. First, the input signal is split and a copy is sent to the side chain. The detector then calculates the magnitude or level of the sidechain signal using the RMS or peak as a measure for how loud a sound is [15, p. 107]. The detector's temporal behavior is controlled by the attack and release parameters. The sound level is compared with the threshold level and, for the case it exceeds the threshold, a scale factor is calculated which corresponds to the ratio of input level to output level. The knee parameter determines how quick the compression ratio is reached. At the end of the side chain, the scale factor is fed to a smoothing filter that yields the gain. The response of the gain filter is controlled by another set of attack and release parameters. Finally, the gain control applies the smoothed gain to the input signal and adds a fixed amount of makeup gain to bring the output signal to a desired level. Such a broadband compressor operates on the input signal's full bandwidth, treating all frequencies from zero through the highest frequency equally. A detailed overview of all sidechain controls of a basic gain computer is given in [83, ch. 3], e.g.

Broadband

x(n) y(n) v(n) g(n)
Figure 14: Basic broadband compressor model 9.2 system model

Feed-Forward Broadband Compression

The employed system model is based on the compressor from Fig. 14. The following simplification is additionally made: the knee parameter is ignored, i.e. the knee is "hard". The compressor is understood as a single-input single-output (SISO) system, that is both the input and the output are single-channel signals. What follows is a description of each block by means of a dedicated function.

The RMS/peak detector as well as the gain computer build upon a first-order (one-pole) lowpass filter. The sound level or envelope v(n) of the input signal x(n) is obtained by

x(n) = α v |x(n)| p + (1 -α v )x(n -1), (71a) v(n) = p x(n), ( 71b
)
where p = 1 represents a peak detector and p = 2 an RMS detector, respectively. The smoothing factor α v , 0 < α v 1, may take on two different values, α att v or α rel v , depending on whether the detector is in the attack or the release phase. The condition for the level detector to enter the attack phase and to choose

α att v over α rel v is |x(n)| > v(n -1). (72)
A formula that converts a time constant τ into a smoothing factor is given in [15, p. 109]. So,

α v = 1 -exp -2.2 T s τ v , ( 73
)
where T s is the sampling period and exp the exponential function.

The derivation of ( 73) can be found in [START_REF] Mcnally | Dynamic range control of digital audio signals[END_REF][START_REF] Zölzer | Digital Audio Signal Processing[END_REF]. The static nonlinearity in the gain computer is usually modeled in the logarithmic domain as a continuous piecewise linear function:

F(n) = -S • [V(n) -L] if V(n) > L, 0 otherwise, ( 74
)
where L is the threshold in decibel, V(n) = 20 log 10 v(n), and S is the slope. The slope is computed from the desired compression ratio R according to

S = 1 - 1 R . ( 75
)
Equivalently, (74) can be expressed in the linear domain as

f(n) = κv -S (n) if v(n) > l, 1 otherwise, ( 76
)
where l = 10 L /20 , κ = l S , and f is the scale factor before filtering. The smoothed gain g is calculated as the exponentially-weighted moving average:

g(n) = α g f(n) + (1 -α g )g(n -1)
with α g ∈ α att g , α rel g .

(77)

The decision to choose α att g instead of α rel g is subject to

f(n) < g(n -1). (78)
Finally, the broadband gain control multiplies the input signal x(n) by the smoothed gain g(n) and adds some makeup gain M to bring the compressed output signal y(n) to a desired level:

y(n) = m • [g(n)x(n)] with m = 10 M /20 . ( 79
)
Due to the fact that g and m are strictly positive, 0 < g 1, m 0, it follows that

sgn(y) = sgn(x), ( 80
)
where sgn is the signum function. In consequence, it is convenient to factorize the input signal as a product of the sign and the modulus,

x(n) = sgn(x) • |x(n)|. (81)

Stereo Linking

To avoid image shifting, it is imperative that an equal amount of gain reduction be applied to both channels of x. This can be achieved by calculating the required amount of gain reduction for x 1 (n) and x 2 (n) independently, and by applying the larger amount to both channels:

y(n) = m • [g(n)x(n)], (82)
where y = [ y 1 y 2 ] ❚ and

g(n) = min [g 1 (n), g 2 (n)]. (83)

problem formulation

The problem at hand is formulated as follows. Given the compressed signal y(n) and the compressor parameters The output of the side chain, i.e. the gain of |x(n)|, given θ, x(n -1), and g(n -1), may be written as

θ = p, L, R, α att v , α rel v , α att g , α rel g , M , (84)
m • g(n) = G [|x(n)| | θ, x(n -1), g(n -1)]. (85)
In ( 85), G denotes a nonlinear dynamic operator that maps the modulus of the input signal |x(n)| onto a sequence of instantaneous gain values m • g(n) according to the compressor model represented by θ. Using ( 85), ( 79) can be solved for

|x(n)| yielding |x(n)| = G -1 [g(n) | θ, x(n -1), g(n -1)] • |y(n)|
subject to invertibility of G. In order to solve the above equation one requires the knowledge of g(n), which is unavailable. However, since g is a function of |x|, we can express |y| as a function of |x| only, and in that manner we obtain an equation with a single unknown:

|y(n)| = H [|x(n)| | θ, x(n -1), g(n -1)], (86)
where H now represents the entire compressor. If H is invertible, i.e. bijective for all n, |x(n)| can be obtained from |y(n)| by

|x(n)| = H -1 [|y(n)| | θ, x(n -1), g(n -1)] if v(n) > l, |y(n)| otherwise. (87)
And yet, since v(n) is likewise unknown, the condition for applying decompression must be predicted from y(n), x(n -1), and g(n -1), and therefore needs the condition for toggling between the attack and release phases. Depending on the quality of the prediction, the recovered modulus |z(n)| may differ somewhat at transition points from the original modulus |x(n)|, so that in the end

x(n) ≈ sgn(y) • |z(n)| z(n). (88)
In the following it is shown how such an inverse compressor alias decompressor is derived.

Characteristic Function

For simplicity, the instantaneous envelope value v(n) is chosen over |x(n)| as the independent variable in [START_REF] Itu-R | Algorithms to measure audio programme loudness and truepeak audio level[END_REF]. The relation between the two items is given by (71). From ( 77) and ( 79), when v(n) > l,

|y(n)| = m • [α g f(n) + (1 -α g )g(n -1)] • |x(n)| (89) (76)
= m • α g κv -S (n) + (1 -α g )g(n -1) • |x(n)|. (90)
From (71),

|y(n)| = m • α g κv -S (n) + (1 -α g )g(n -1) • p v p (n) -(1 -α v )x(n -1) α v , (91)
or equivalently (note that m, α v = 0 by definition)

α v |y(n)| m p = α g κv -S (n) + (1 -α g )g(n -1) p • [v p (n) -(1 -α v )x(n -1)]. (92)
Moreover, (92) has a unique solution if G and hence H is invertible.

Moving the expression on the left-hand side over to the right-hand side, we may define

ζ p (v) α g κv -S (n) + (1 -α g )g(n -1) p • [v p (n) -(1 -α v )x(n -1)] -α v |y(n)| m p , (93)
which shall be termed the characteristic function. The zero-crossing or root v 0 of ζ p (v) bears so the sought-after envelope value v(n). Once v(n) is found (see Section 9.5), the current values of x, |x|, and g are updated according to

x(n) = v p 0 (n), |x(n)| = p x(n) -(1 -α v )x(n -1) α v , g(n) = |y(n)|/m |x(n)| , (94a) if v(n) > l, or else g(n) = α g + (1 -α g )g(n -1), |x(n)| = |y(n)|/m g(n) , x(n) = α v |x(n)| p + (1 -α v )x(n -1). (94b)
The decompressed sample is then computed as

z(n) = sgn(y) • |x(n)|. (95)
9.4.2 Attack-Release Phase Toggle

Envelope smoothing

When a peak detector is in use, α v can take on two different values. The condition for the attack phase is given by (72). It is equivalent to

|x(n)| p > x(n -1). (96)
Assuming that x(n -1) is known, what is needed to be done is to express the unknown |x| in terms of |y| such that the above equation still holds true. If α g is rather small, α g 0.1 ≪ 1, or equivalently if τ g is sufficiently large, τ g 0.5 ms at 44.1-kHz sampling, the term α g f(n) in ( 89) is negligible, so it approximates [START_REF] Huffman | A method for the construction of minimumredundancy codes[END_REF] as

|y(n)|/m ≈ g(n -1) • |x(n)|. (97)
Solving [START_REF] Schmidt | Multichannel dynamic range compression for music signals[END_REF] for |x(n)| and plugging the result into (96), we obtain

|y(n)|/m g(n -1) p > x(n -1). (98)
If ( 98) is true, the detector is assumed to be in the attack phase.

Gain smoothing

Just like the peak detector, the gain smoothing filter can be in either the attack or the release phase. The necessary condition for the attack phase in (78) may also be formulated as

v(n) > κ g(n -1) 1 /S with v(n) > l. (99)
But as the current envelope value is unknown, we need to substitute v(n) in the above inequality by something that is computable. With this in mind, ( 89) is rewritten as

|y(n)|/m = α g f(n) g(n -1) + (1 -α g ) g(n -1) • |x(n)| = 1 -α g 1 - f(n) g(n -1) g(n -1) • |x(n)|. (100)
Provided that f(n) < g(n -1) and due to the fact that 0 < α g 1, the expression in square brackets in ( 100) is smaller than one, and thus during attack

|y(n)|/m < g(n -1) • |x(n)|. (101)
Substituting |x(n)| using (94a) and solving (101

) for v(n) results in v(n) > p α v |y(n)|/m g(n -1) p + (1 -α v )x(n -1). (102)
If v(n) in ( 99) is substituted by the expression on the right-hand side of ( 102), (99) still holds true. So, the following sufficient condition is used to predict the attack phase of the gain smoothing filter:

p α v |y(n)|/m g(n -1) p + (1 -α v )x(n -1) > κ g(n -1) 1 /S . (103)
Note that the values of all variables are known, if and when (103) is evaluated.

Envelope Predictor

An instantaneous estimate of the envelope value v(n) is required not only to predict when compression is active, formally v(n) > l in [START_REF] Schuijers | Low complexity parametric stereo coding[END_REF], but also to initialize the iterative search algorithm in Section 9.5. By resorting once more to [START_REF] Huffman | A method for the construction of minimumredundancy codes[END_REF] it can be noted that in the opposite case where v(n) l, f(n) = 1, and so

|x(n)| = |y(n)|/m α g + (1 -α g )g(n -1) . ( 104
)
The sound level of the input signal at instant n is therefore

v(n) = p α v |y(n)|/m α g + (1 -α g )g(n -1) p + (1 -α v )x(n -1), (105)
which must be greater than the threshold for compression to set in, whereas α v and α g are selected based on ( 98) and (103), respectively.

Stereo Unlinking

First, one decompresses both channels of y(n) independently so as to obtain two estimates z 1 (n) and z 2 (n). Using (79), one then computes ŷ1 (n) and ŷ2 (n) from z 1 (n) and z 2 (n), and picks the channel ref for which y ref (n) ≈ ŷref (n). Finally, one updates the variables of the complementary channel ¬ref:

z ¬ref (n) = y ¬ref (n)/m g ref (n) , (106)
x¬ref (n) according to (71a), and g ¬ref (n) according to (77).

Error Analysis

Consider |x(n)| being estimated from |y(n)| according to

|x(n)| = |y(n)| g(n -1) with m = 1. (107)
The normalized error is then

e(n) = |x(n)| -|x(n)| |y(n)| (108) = |y(n)| g(n -1) - |y(n)| g(n) |y(n)| = g(n) -g(n -1) g(n) • g(n -1) . ( 109
)
As g > 0 for all n, e(n) < 0 during attack and e(n) 0 during release, respectively. The instantaneous gain g(n) can also be expressed as

g(n) = α g N m=0 (1 -α g ) m f(n -m), (110)
where N is the runtime in samples. Using (110) in ( 109), the magnitude of the error is given by

|e(n)| = N m=0 (1 -α g ) m [f(n -m) -f(n -m -1)] α g N i,j=0 (1 -α g ) i+j f(n -i)f(n -j -1) (111)
N m=0 (1 -α g ) m |f(n -m) -f(n -m -1)| α g N i,j=0 (1 -α g ) i+j f(n -i)f(n -j -1) . (112)
For γ = 1, (111) becomes

|e(n)| α g =1 = |f(n) -f(n -1)| f(n) • f(n -1) , (113)
whereas for α g → 0, (112) converges to infinity:

|e(n)| α g →0 1 α g N m=0 >0 during compression |f(n -m) -f(n -m -1)| N i,j=0 f(n -i)f(n -j -1) → ∞. (114)
Thus, the error is smaller for large α g or for short τ g . The smallest possible error is for α g = 1, which then again depends on the current and the previous value of f. The error accumulates if α g < 1 with N.

The difference between consecutive f-values is signal dependent. The signal envelope v(n) fluctuates less and is thus smoother for smaller α v or for longer τ v . f(n) is also more stable when the compression ratio R is low. For R = 1, f(n) is perfectly constant. The threshold L has a negative impact on error propagation. The lower L the more the error depends on N, as more samples are compressed with different f-values. The RMS detector stabilizes the envelope more than the peak detector, which also reduces the error. Moreover, since usually τ att < τ rel , the error due to α v is smaller during release, whereas the error due to α g is smaller during attack. At last, the error can be expected to be larger at transition points between quiet to loud signal passages and vice versa.

The above error may cause a decision in favor of a wrong smoothing factor α v in [START_REF] Giannoulis | Digital dynamic range compressor design-a tutorial and analysis[END_REF], like α att v instead of α rel v , e.g. The decision error from (98) then propagates to (103). Given that α att v > α rel v , the error due to (107) is accentuated by [START_REF] Giannoulis | Digital dynamic range compressor design-a tutorial and analysis[END_REF] with the consequence that (103) is less reliable than [START_REF] Giannoulis | Digital dynamic range compressor design-a tutorial and analysis[END_REF]. The total error in (103), thus, scales with α att vα rel v . In regard to (105), reliability of the envelope's estimate is subject to validity of both ( 98) and (103). A better estimate is obtained when the sound level detector and the gain filter are both in either the attack or the release phase. Here, too, the estimation error increases with α att vα rel v and also with α att gα rel g . The makeup gain M has no impact on the error. Stereo linking may be another source of error. It all depends on how well the reference channel is detected.

numerical approximation

An approximate solution of the characteristic function can be found, e.g., by means of linearization. The estimate from (105) may moreover serve as a starting point of an iterative search for an optimum:

v init = p α v |y(n)|/m α g + (1 -α g )g(n -1) p + (1 -α v )x(n -1). (115)
The criterion for optimality is further chosen as the deviation of the characteristic function from zero, which is initialized to

∆ init = |ζ p (v init )|. (116)
Thereon, (93) can be approximated using the equation of a straight line, ζ = a • v + c, where a is the slope and c is the ζ-intercept. The root is characterized by the equation

ζ p (v i + ∆ i ) -ζ p (v i ) ∆ i • v + ζ p (v i ) = 0, (117)
as shown in Fig. 15. The new estimate of the optimal v(n) is

v i+1 = v i - ∆ i • ζ p (v i ) ζ p (v i + ∆ i ) -ζ p (v i ) . ( 118
) v ζ p (v i ) Δ i 0 v i v i + 1 ζ p (v i + Δ i ) v i + Δ i ζ p (v)
zero-crossing or root If v i+1 is less optimal than v i , the iteration is stopped and v i is the final estimate. The iteration is also stopped if ∆ i+1 is smaller than some ǫ. In the latter case, v i+1 has the optimal value with respect to the chosen criterion. Otherwise, v i is set to v i+1 and ∆ i is set to ∆ i+1 after every step and the procedure is repeated until v i+1 has converged to a more optimal value. The proposed method is a special form of the secant method with a single initial value v init .

9.6 general remarks

Lookahead

A compressor with a lookahead function, i.e. with a delay in the main signal path as in [15, p. 106], uses past input samples as weighted output samples. Now that some future input samples are required to invert the process-which are unavailable, the inversion is rendered impossible. g(n) and x(n) must thus be in sync for the approach to be applied. For proof and reasoning see Appendix C.

Clipping and Limiting

Another point worth mentioning is that "hard" clipping and "brickwall" limiting are special cases of compression with the attack time set to zero and the compression ratio set to ∞ : 1. The nonlinearity F in that particular case is a one-to-many mapping, which by definition is noninvertible.

Logarithmic Gain Smoothing

Certain compressor models apply gain smoothing in the logarithmic domain. In that case, ( 77) is replaced by

g(n) = f α g (n) • g 1-α g (n -1), (119)
so that the characteristic function becomes

ζ p (v) = µ v -α g S (n) • g 1-α g (n -1) p • [v p (n) -(1 -α v )x(n -1)] -α v |y(n)| m p (120)
with µ = l pβS . Equations ( 98) and (103) remain valid, whereas (105) and consequently ( 115) are now to be replaced by

v(n) = p α v |y(n)|/m g 1-α g (n -1) p + (1 -α v )x(n -1). (121)
Finally, taking (119) into account, the gain value in (94b) is computed as

g(n) = g 1-α g (n -1) (122) instead.

pseudocode

The algorithm is divided into three parts, given below in the form of pseudocode. Algorithm 1 outlines the compressor that corresponds to the model from Section 9.2. Algorithm 2 illustrates the decompressor described in Section 9.4. The root-finding algorithm from Section 9.5 is summarized in Algorithm 3. T s represents the sampling period in ms.

parameter quantization and coding

The threshold level L and the makeup gain M, both given in dB, can be quantized using ( 48) on predefined ranges. The compression ratio R ∈ [1, 60] can be compressed using [START_REF] Algazi | The CIPIC HRTF database[END_REF] or [START_REF]The CIPIC HRTF database[END_REF] by setting x to R -1 and x max to 59. Instead of coding the time constants τ v and τ g , it is more appropriate to code the smoothing factors, α v , α g ∈ (0, 1], using [START_REF] Algazi | The CIPIC HRTF database[END_REF] or [START_REF]The CIPIC HRTF database[END_REF] in combination with (48). 9.9 performance evaluation 9.9.1 Experimental Setup The decompressor's performance is evaluated on a synthetic signal and also on "natural" audio material. The former is generated from

Algorithm 1 Single-input single-output compressor function Compress(x n , T s , θ) xn ← 0 g n ← 1 for n ← 1, N do if |x n | p > xn then α v ← 1 -exp (-2.2T s /τ att v ) else α v ← 1 -exp -2.2T s /τ rel v end if xn ← α v |x n | p + (1 -α v )x n v n ← p √ xn if v n > l then f n ← κv -S n else f n ← 1 end if if f n < g n then α g ← 1 -exp -2.2T s /τ att g else α g ← 1 -exp -2.2T s /τ rel g end if g n ← α g f n + (1 -α g )g n
y n ← g n x n end for return y n end function a weighted sum of Heaviside step functions, while the latter consists of twelve items including speech, sung voice, music, and jingles. All items are normalized to -16 LUFS [START_REF] Itu-R | Algorithms to measure audio programme loudness and truepeak audio level[END_REF]. A detailed overview of used compressor settings is given in Table 7. They correspond to presets of commercial compressor plug-ins. The ǫ-value in the break condition of Algorithm 3 is set to 1 • 10 -12 .

The inverse approach is evaluated using the following metrics. The root-mean-square error (RMSE),

RMSE = 20 log 10 1 N N n=1 [z(n) -x(n)] 2 , ( 123
)
given in decibels relative to full scale (dBFS), the perceptual similarity between the decompressed and the original signal, and the execution time of the decompressor relative to real time (RT). Furthermore, we present the percentage of compressed samples, the mean number of iterations until convergence per compressed sample, the error rate of the attack-release toggle for the gain smoothing filter, and finally

Algorithm 2 Single-input single-output decompressor function Decompress(y n , T s , θ, ǫ) xn ← 0 g n ← 1 for n ← 1, N do if |y n | > p √ xn • g n then α v ← 1 -exp (-2.2T s /τ att v ) else α v ← 1 -exp -2.2T s /τ rel v end if if |y n | > p (κ/g n ) p /S -(1 -α v )x n α v • g n then α g ← 1 -exp -2.2T s /τ att g else α g ← 1 -exp -2.2T s /τ rel g end if if |y n | > p l p -(1 -α v )x n α v • [α g + (1 -α g )g n ] then v n ← p α v |y n | α g + (1 -α g )g n p + (1 -α v )x n v 0 ← Charfzero(v n , ǫ) |x n | ← p v p 0 -(1 -α v )x n α v xn ← v p 0 g n ← |y n |/|x n | else g n ← α g + (1 -α g )g n |x n | ← |y n |/g n xn ← α v |x n | p + (1 -α v )x n end if x n ← sgn(y n ) • |x n | end for return x n end function
the error rate of the envelope predictor. The perceptual similarity is assessed by the PEMO-Q [START_REF] Huber | PEMO-Q -a new method for objective audio quality assessment using a model of auditory perception[END_REF] software [71] with PSM t as metric. The simulations are run in MATLAB on an Intel Core i5-520M CPU.

Experimental Results

Fig. 16 shows the inverse signal z(n) to the synthetic input signal x(n) using an RMS detector. The inverse signal z(n) is obtained from the compressed signal y(n) with an error of ca. -129 dBFS. It is visually indistinguishable from the original signal x(n). Due to the fact that the signal envelope is constant most of the time, the error is no-for a limiter (peak) or compressor (RMS) [15, pp. 109-110]. It can be observed that the inversion accuracy depends on the release time of the peak detector and not so much on its attack time for both the envelope and the gain filter, see Figs. 17, 18b. For the envelope filter, all error curves exhibit a local dip around a release time of 0.5 s. The error increases steeply below that bound but moderately with larger values. In the proximity of 5 s, the error converges to -130 dBFS. With regard to the gain filter, the error behaves in a reverse manner. The curves in Fig. 18b exhibit a local peak around 0.5 s with a value of -180 dBFS. It can further be observed in Fig. 17a that the curve for τ rel v = 1 ms has a dip where τ att v is close to 1 ms, i.e. where α att vα rel v is minimal. This is also true for Fig. 17c and Fig. 17d: the lowest error is where the attack and release times are identical. As a general rule, the error that is due to the attack-release switch is smaller for the gain filter in Fig. 18.

Looking at Fig. 19 one can see that the error decreases with threshold and increases with compression ratio. At a ratio of 10 : 1 and beyond, the RMSE scales almost exclusively with the threshold. The lower the threshold, the stronger the error propagates between decompressed samples, which leads to a larger RMSE value. The RMS detector further augments the error because it stabilizes the envelope v(n) more than the peak detector. Clearly, the threshold level has the highest impact on the decompressor's accuracy.

system overview

To account for a simplified but complete music production chain that consists of mixing and mastering, the two processing steps from Sections 7 and 9 are combined in a two-stage cascade scheme. The corresponding encoder and decoder block diagrams are illustrated in Figs. 22a and22b.

cascade encoder

Fig. 22a shows the cascade connection for a practical encoder. It contains an analysis block that computes the STPSDs, a mixdown block representing the mixing stage, and a DRC block that represents the mastering stage. The multiplexing block assembles the bitstream that comprises the compressed mixture and the coded side information. The processing steps include:

1. STFT analysis of source signals, see [START_REF] Crochiere | A weighted overlap-add method of shorttime fourier analysis/synthesis[END_REF] 2. STPSD computation from magnitude-squared spectra The algorithmic delay of the cascade connection is determined by the framing and overlap delay of the STFT and its inverse. When using a 2048-length symmetric window with 50-% overlap between frames, the algorithmic delay amounts to 2047 samples. This corresponds to 46.4 ms at a sampling rate of 44.1 kHz.

Computational Complexity

The computations performed in the cascade connection depend on the frequency content of source signals and also the signal dynamics, which vary from mixture to mixture. So, it is convenient to analyze the runtime complexity for the worst-case scenario in terms of "big O" notation. Moreover, it is desirable to establish a relationship between the running time and the following input parameters: the number of sources I, the number of frequency bands Z, and the transform length N, I < Z < N. All arithmetic operations that are counted shall require exactly one unit of time to execute. The results are shown in Table 9. The figures reveal that the decoder's complexity is comparable to the complexity of the encoder. The execution time is dominated by the I-fold STFT and its inverse.

Side-Information Rate

The side-information rate of the cascade connection is given by the number of bits communicated to the decoder per time frame. These comprise the compression parameters, the mixing parameters, and the STPSDs. The compression parameters and the mixing parameters are signaled to the decoder once at the beginning of the transmission. The STPSD values are represented as Q-bit unsigned integers. Table 10 provides an overview of the capacities that are necessary to store the STPSDs for various numbers of frequency bands. They are calculated as bitrate where L is the frame length, M is the overlap, and f s is the sampling frequency. The information rate is varied upon a subdivision the ERB by an integer factor. In general, the finer the frequency resolution the higher the observed quality of the estimates. On the other hand, the greater the number of sources the finer is to be chosen the frequency resolution to obtain a quality comparable to a sparser configuration. Whatever the case, the figures in Table 10 can be drawn upon to make an estimate for the side-information rate of the cascade connection, as the rate of the compression and mixing parameters is comparably negligible. 1Table 10 also shows the STPSD rate obtained using DPCM. The latter is implemented as follows. The probability distribution of the input symbols is derived from the number of occurrences of each possible difference value in a training set. It is observed that the difference signal both in time and frequency direction, e.g., has a Laplace(µ, b) distribution with µ ≈ -0.2 and b ≈ 2. Then, a Huffman codebook [START_REF] Huffman | A method for the construction of minimumredundancy codes[END_REF] can be generated from the input probability distribution with a mean codeword length of 3.5 bit. This corresponds to a compression ratio of 1.7 : 1, which again means that almost twice as many source signals can now be extracted from the mixture for the same amount of side information. A 2048-point FFT is employed together with a KBD window of the same size. Succeeding frames overlap by 50 %. The cascade is tested on an excerpt from Fort Minor's "Remember the Name" multitrack. The latter is decomposed into 5 mono sources and is 24 s long. The compressor setting is shown in Table 11. To exclude a performance bias due to quantization, the mastering and mixing parameters are considered known at the decoder. The STPSDs are quantized with 6 bits on a 1 /2-ERB frequency scale. By applying DPCM with Huffman coding to the quantized STPSD values, the mean side-information rate is reduced to roughly 10 kbps per source. The simulations are run in MATLAB.

= f s L -M • Q • Z • I, (124)
To evaluate the proposed cascade connection, the following metrics are used: the RMSE given in dBFS and the PSM. The PSM is computed with PEMO-Q. In [START_REF] Huber | PEMO-Q -a new method for objective audio quality assessment using a model of auditory perception[END_REF] it is said that PEMO-Q shows a slightly better performance than Perceptual Evaluation of Audio Quality (PEAQ) [START_REF] Itu-R | Method for objective measurements of perceived audio quality[END_REF].

Experimental Results

The results of the experiment are illustrated in Fig. 21. The asterisk marks the compressed mix without makeup gain, i.e. M = 0 dB. The RMSE and the SNR for each mixture signal,

SNR 10 log 10 l x l 2 l y l -x l 2 , ( 125
)
where • is the Euclidean norm, are given in Table 12.

It can be noted that the separated source signals exhibit relatively high quality when the mix is uncompressed (see dashed bar). The RMSE for the vocal reaches almost -60 dBFS while being below -40 dBFS for the rest. The decompressor's performance is practically free from error: the ∆RMSE and ∆PSM values for each track are close to zero after decompression (see lower row). The RMSE level decreases, as expected, if the makeup gain is removed from the compressed mix, but does not reach the level of the uncompressed mix. This proves that the waveform of the mix has been altered by the compressor. The RMSE difference between the two compressed mixtures is so due to scaling. On the contrary, the corresponding PSM values are equal, which shows that the PSM metric is scale-independent.

In the given example, the PSM improvement due to decompression is mostly evident for the bass track. For the other tracks, the PCMV filter provides an estimate which is perceptually very similar to the reference, even if the mix is compressed. Yet, a so-called "pumping" coming from hard compression can be heard clearly on the vocal track. The effect is more audible for faster attack and release. Hence, for the used compressor setting, the decompressor is indispensable to achieve high perceptual quality.

Other

M U LT I C H A N N E L O B J E C T -B A S E D A U D I O C O D I N G

introduction

Ever since the end of the last century, coding of audiovisual objects has been of particular interest to the MPEG, and it has gained importance in recent years. Whereas the first audio coders were all channelbased, a paradigm shift towards source-based coding was initiated by works like [START_REF] Faller | Parametric joint-coding of audio sources[END_REF]. A more recent example is MPEG's SAOC, see Chapter 8, or the work in [START_REF] Hotho | A backwardcompatible multichannel audio codec[END_REF]. The necessity for object-based coding in the sense of sound sources arises when distinct audio objects are to be stored or transmitted for the purpose of post-hoc reproduction in different environments. So far, its application fields include remixing, video gaming, home cinema or 3D audio, and there might be more in the future.

The work presented here focuses on the question how a number of given source signals or objects can be represented by a reduced number of mixture channels and recovered using the mixture and a small amount of metadata. The work by Faller [START_REF] Faller | Parametric joint-coding of audio sources[END_REF] considers only single-channel mixtures and has no means to scale the quality after demixing. The resulting quality can so be expected to be the worst possible, as a single-channel mixture exhibits the highest overlap between objects. Whereas in [START_REF] Hotho | A backwardcompatible multichannel audio codec[END_REF] Hotho et al. generalize the mixture to more than one channel and propose to use the residual to scale the audio quality up to perceptual transparency, there is no explicit control over the quality, except that the latter is said to improve with the bandwidth of the residual signal by rule of thumb. Moreover, the works in [START_REF] Faller | Parametric joint-coding of audio sources[END_REF] and [START_REF] Hotho | A backwardcompatible multichannel audio codec[END_REF] evaluate the quality empirically after rendering the decoded objects into a prescribed format such as 5.1 surround and are consequently bound to the sound reproduction system.

Though related to previous approaches, this work capitalizes on quality-driven demixing which is further independent of mixing and rendering after demixing. Moreover, the ISS approach is pursued. As it is demonstrated in Chapter 7, an underdetermined linear mixture can be decomposed into an arbitrary number of components by means of spatial filtering. When the separation is carried out in the STFT domain, the estimates show distortion in amplitude and phase. It is clear that the amount of distortion, which is due to bleed from other sources but also the filter response, decreases with the number of mixture channels because the separation problem becomes better conditioned and the array gain increases. In this chapter it is shown how these facts can be exploited to code audio objects in a controllable way.

signal and system model

Using the STFT signal representation, the source signal components S i (k, m), i = 1, 2, . . . , I, are circular symmetric complex normal random variables with zero mean and a diagonal covariance matrix R

s (k, m) = diag [Φ s 1 (k, m), Φ s 2 (k, m), . . . , Φ s I (k, m)]. {Φ s i (k, m)} k is
a i S i (k, m) = As(k, m), (126)
where A ∈ R M×I represents an instantaneous mixing system that is assumed real for practical reasons. This model is identical with that from Sections 7.2 and 7.3. Equation ( 126) is so an extension of ( 13) to M channels.

problem formulation

The problem at hand is formulated as follows. Given the mixing rule1 and the STPSDs of the source signals, find a low-rank signal space representation x(k, m) which satisfies a minimum-similarity constraint on the recovered source signals after transformation back to the original signal space. In other words, what is the minimum number of channels M min into which one can mix the source signals and yet maintain the desired quality level after demixing? The quality metric shall further relate to, but not model, human perception.

proposed solution 11.4.1 System Overview

The proposed coding scheme comprises an encoder and a decoder. Its functional principle is depicted in the form of a block diagram in Fig. 22. The analysis block performs the computation of the STPSDs of all I source signals, as indicated by Φ s . From Φ s , the number of required mixture channels M is derived that guarantees the desired quality on the decoder side. This is accomplished through a quality control mechanism that is discussed in Section 11.5. The STPSDs are quantized on an ERB-log frequency-power scale and DPCM coded, see Section 7.7.3. In addition, the Free Lossless Audio Codec (FLAC) [91] (a) The encoder can be used to reduce the file size of the mixture signal. When FLAC is the coder of choice, which is "lossless", the STPSDs can be attached to the mixture in the form of a watermark [START_REF] Geiger | Audio data hiding with high rates based on IntMDCT[END_REF][START_REF] Pinel | A high-rate data hiding technique for audio signals based on IntMDCT quantization[END_REF] before coding. Otherwise, they are embedded into a serial bitstream as a supplement to the encoded audio data. In the decoder, the demultiplexer reassembles the encoded mixture signal from the bitstream and the metadata if necessary. If lossless compression is used in the encoder, the decoding block may as well be followed by watermark extraction. The decoded STPSDs accompany the source separation which is discussed in more detail in the follow-up sections together with the mixdown.

Mixdown

Due to the fact that we can decide freely about the mixing rule, we will seek to implement the mixdown such that the decomposition of the mixture in the decoder becomes controllable and in this way the resulting signal quality predictable. It is also highly desirable for the signal quality not to depend on the mixing rule but on the number of mixture channels only. To accomplish this, one must consider how the decomposition is carried out.

Optimum filters and second-order statistics

A spatial filter that maximizes the output SIR in the MSE sense has the generic form [START_REF] Cox | Robust adaptive beamforming[END_REF]

w io = cR -1 x a i , (127)
c ∈ C. If the mixing matrix A and the input covariance matrix R s are known, R x is also known, since

R x = AR s A ❚ . (128)
So, it follows that R x is real symmetric and also positive-semidefinite. Specific problems may require the filter's response to be constrained in order to obtain a better suited solution. And thus, c is formulated differently from one filter to another. One well-known example is the minimum-variance distortionless response (MVDR) spatial filter [START_REF] Capon | High-resolution frequency-wavenumber spectrum analysis[END_REF] that has a unity-gain response with zero phase shift. The associated weight vector is

w MVDR io = 1 a ❚ i R -1 x a i R -1 x a i . (129)
The distortionless response property of the MVDR spatial filter is used in Section 11.5 to define a similarity metric.

Signal-to-interference ratio and array gain

An estimate for the ith source component in the kth frequency bin and the mth time segment is given by

Y i (k, m) = w ❍ i x(k, m). (130)
The corresponding STPSD value is

Φ y i (k, m) = E |Y i (k, m)| 2 = w ❍ i R x (k, m)w i . (131)
Using ( 128), (131) can also be written as

Φ y i (k, m) = w ❍ i a i 2 Φ s i (k, m) signal of interest + I l=1,l =i w ❍ i a l 2 Φ s l (k, m) Φ b i (k,m) residual interference or bleed . ( 132
)
The output SIR is then

SIR out i (k, m) = Φ s i (k, m) I p=1,p =i Φ s p (k, m) SIR in i (k,m) • w ❍ i a i 2 I p=1,p =i Φ s p (k, m) I q=1,q =i w ❍ i a q 2 Φ s q (k, m) G i (k,m)>1 , (133)
where SIR in is the input SIR and G is the array gain. The latter can be shown to be

G i (k, m) = a ❚ i R -1 x (k, m)a i 2 I p=1,p =i Φ s p (k, m) I q=1,q =i a ❚ i R -1 x (k, m)a q 2 Φ s q (k, m) (134)
for real A and real or complex c. As can be seen from ( 134), the array gain is a function of the mixing system and the STPSDs.

Mixing system

The mixing system is designed as an M-element vertical line array and the ith source is associated with an angle α i ,

α i = π I + 1 • i for i = 1, 2, . . . , I. (135)
α can be thought of as the angle between the propagation path and the normal to the array axis in a two-dimensional sound field. The mixing coefficients are calculated as

a l+1,i = cos (lα i ) for l = 0, 1, . . . , M -1, (136)
where cos (lα) represents an lth-order Chebyshev polynomial in cos α.

As a direct consequence of (135) A has linearly independent columns, and because of (136) A is real and has full row rank. However, a i = a l if i = l. Note that the mixing rule can be chosen arbitrarily so long as the resulting vectors are linearly independent. The above rule is simple and also allows for a geometric interpretation.

As previously stated, it is highly desirable that the quality of the estimates is independent of the mixing rule. It is hence vital to make sure that the output SIR is the same across all sources. This is accomplished with Algorithm 4 which under the assumption that the I mixture components are i.i.d. and standard normal, and with knowledge of the mixing rule, provides the input weights that yield an equal output SIR. In this way, one compensates for differences in "radiation" patterns.2 Algorithm 4 Equal-SIR out power distribution function Powdist(I, M, ǫ) (137) Moreover, as the local constellation of mixture components changes with time and frequency, we distinguish between inactive and active TF points (k, m). Active points can be determined, overdetermined or underdetermined. The number of components in a TF point, denoted as I(k, m), and also their indices can be inferred from the signaled STPSDs, {Φ s (k, m)} k . Taking all this into account, the demixing matrix W for an active TF point (k, m) is given by

for i ← 1, I do α i ← π/(I + 1) • i for l ← 0, M -1 do a l+1,i ← cos (lα i ) end for Φ b i ← 1 end for oldcost ← 0 cost ← ∞ while |cost -oldcost| > ǫ do for i ← 1, I do ρ i ← Φ b i / I l=1 Φ b l end for R x ← I i=1 ρ i a i a ❚ i oldcost ← cost cost ← 0 for i ← 1, I do Φ y i ← 1/ a ❚ i R -1 x a i Φ b i ← Φ y i -ρ i SIR out i ← ρ i /Φ b i l ← max (i -1, 1) cost ← cost + SIR out i -SIR out
W ❚ =        A + if I(k, m) < M A -1 if I(k, m) = M diag (c i ) I(k,m) i=1 A ❚ R -1 x (k, m) if I(k, m) > M (138)
∀(k, m), where A + is the Moore-Penrose pseudoinverse and

c i = Φ s i (k, m) a ❚ i R -1 x (k, m)a i (139)
is the magnitude response of the PCMV spatial filter, see Section 7.5.2.

As c ∈ R, the phase response of the PCMV filter is free of distortion. We define a similarity index (SIX) according to

SIX i (z, m) = 1 -min |Φ y i (z, m) -Φ s i (z, m)| Φ s i (z, m) , 1 • Φ y i (k, m) -Φ b i (z, m) Φ y i (z, m) , (140)
SIX ∈ [0, 1], where z is the band index on an ERB-like frequency scale, see Section 7.7.3. For the PCMV filter, (140) simplifies to

SIX PCMV i (z, m) = Φ s i (z, m) Φ MVDR y i (z, m) = Φ s i (z, m)a ❚ i R -1 x (z, m)a i . (141)
3. Phase distortion at the output is subject to bleed only

The relation between SIR out and SIX PCMV is given by

SIR out i (z, m) = Φ s i (z, m) Φ MVDR y i (z, m) -Φ s i (z, m) = Φ s i (z,m) /Φ MVDR y i (z,m) 1 -Φ s i (z,m) /Φ MVDR y i (z,m) = SIX PCMV i (z, m) 1 -SIX PCMV i (z, m) , (142)
which is invertible. For numerical reasons, however, it is advisable to convert SIX to SIR out first, and to limit the range of SIR out to ±60 dB afterwards. By weighting the SIX metric by frequency and fractional input power, we obtain another metric:

SIXFP i (m) = z Φ s i (z, m)SIX i (z, m) z Φ s i (z, m) . ( 143
)
In case of the PCMV spatial filter, SIRFP out i (m) can also be computed from SIXFP i (m) using (142). The overall value is finally calculated as the mean over the time segments in which the composite input signal power is significant:

SIXFP i = 1 |▼| m∈▼ SIXFP i (m), (144)
where ▼ = {m | z Φ s i (z, m) Φ min } and Φ min is an empirical lower bound.

Control Mechanism

As the SIXFP is a measure of similarity between the original and the estimated components, it can be used to predict the signal quality at the output before the final mixdown. For this, the local covariance matrix in (141) is computed as in (128) from the quantized STPSDs which are available after analysis and the tentative mixing coefficients. One starts with the lowest possible value for M, which is 2, and increases M until the desired SIXFP value is reached. For M = I, perfect reconstruction is expected. The stop condition can be defined globally for the entire signal or locally for a segment. One can also have a single condition for all objects or a separate condition for each one of them. One could ask that, e.g., the SIXFP value for any object is above a given threshold.

performance evaluation 11.6.1 Experimental Setup

The testing framework from Chapter 7 is used. The number of frequency bands is set to 76, which results in a mean side-information rate of 11.5 kbps per object at a sampling rate of 44.1 kHz. The proposed scheme is tested on a 10-track excerpt of 20 s length from Fort Minor's "Remember the Name" recording. All tracks are converted to mono. FLAC is used to code the mixture, which is not watermarked. The resulting audio quality is evaluated for 2-9 mixture channels.

Experimental Results

The results of the experiment are depicted in Fig. 23. It can be seen that for imperceptible quality impairment correspondent to PEMO-Q's ODG metric, one requires that SIXFP 0.99 or SIRFP 48.0 dB. This corresponds to 7 channels for the given multitrack. Further, it can be noted that ∆SIRFP ≈ 6 • ∆M, i.e., the SIRFP value increases roughly by 6 dB with each additional channel. The data-rate savings due to downmixing equal 1 -M /I. They amount to 0.3 in the above example, see the LPCM curve. Yet the lower curve says that coding the 10 mono tracks with FLAC separately is more efficient than coding the 7 channels so long as interchannel redundancy is not minimized. Even so, according to informal listening tests, perceptual transparency is already attained with 5 channels. In that case, the proposed scheme provides savings of 0.5 for the uncoded LPCM mixture or 0.2 when coded with FLAC. The ratio of side information to FLAC-coded audio data is 0.14 or less, and scales with the channel number M. The proposed two-stage cascade is based on a simplified model of the music production chain consisting of a summation of amplitudepanned and possibly equalized single-channel or two-channel tracks in the mixing stage and dynamic range compression in the mastering stage. Although one can certainly argue that commercial releases are more complex than this, as for instance that in electronic music more sophisticated compressors are employed or no compression at all as in classical or acoustic music-albeit "transparent" compression may still be applied, the cascade should be viewed as a "blueprint" rather than a "standard". As an example, compression can be avoided easily by setting the threshold to 0 dBFS or by "bypassing" the compressor and the decompressor. With regards to electronic and also pop/rock music, it should be possible to determine the characteristic function of the respective compressor and to solve it using the approach from Chapter 9.

The upper performance bound of the cascade is mainly due to the sound complexity of commercial music. Notably, it is depending on by how much the frequency spectra of constituent tracks overlap and how the sources are distributed in the sound field. The sound quality after demixing is subject to the so-called "array gain". It is shown to be a function of a) the STPSDs and b) the mixing system, see Chapter 11. As a general rule, the less the said frequency spectra overlap and the further apart they are positioned in space the better the resulting sound quality. Yet both constraints are barely met. Tracks in a music piece are either in the same key, or their keys are relative, or they are in a subdominant or dominant relationship. So, stringed instruments have a high number of interfering harmonics in the mix. Percussion instruments on the other hand cover a broad range of frequencies. In addition, traditional source positioning is such that the main sources are near the center, and thus very close to each other. When multiple sources are in one direction, they can only be distinguished by their spectral envelope but not their direction, which diminishes quality as well.

The knowledge of the mixing and the mastering can be viewed as a shortcoming of the scheme, rendering it hardly applicable to existent music releases, for which that sort of information is unavailable. And although tools such as the non-negative matrix factorization [START_REF] Lee | Learning the parts of objects by non-negative matrix factorization[END_REF][START_REF]Algorithms for non-negative matrix factorization[END_REF] to learn the spectrograms from the mix do exist, their performance is limited. First, because the said factorization is not unique and second, the cost function is non-convex. In the proposed scheme, like in any other model-based scheme, a deviation from the true parameters will cause an additional error in the result. The decoder is most effective when supplied with very accurate information by an accompanying encoder-or an estimator.

The results obtained in Chapter 7 allow the following conclusions to be drawn. The proposed algorithm is capable of providing source signal estimates that are perceptually closer to the originals than any published algorithms of similar type. It should also be noted that the algorithm does not impose any restrictions on the number of sources and neither on their spectral overlap. On the contrary, it adapts to a given signal constellation and provides the best estimates respecting a power constraint in linearithmic time. Bearing high resemblance to the original signals at a fairly tolerable side-information rate around 10 kbps per source or channel, the algorithm is well-suited for active listening applications in real time. The power-conserving minimumvariance filter performs perceptually better than a Wiener-type filter for an instantaneous and a narrowband convolutional mixture alike. The equal-power constraint ensures that the recovered replica retain the timbre and the auditory bandwidth of the originals. Also, it was observed that noisy estimates are more appreciated by the listener in comparison to "perforated" or "dull" sounding replica. With regard to spectrogram coding, JPEG compression gives better results than the NMF at a bitrate above 10 kbps. Below, the NMF seems more efficient. With the proposed approach, the mix can be decomposed into separate tracks or into foreground objects plus the background and in the same manner one can separate the vocal from the instrumental for karaoke. Further, it is possible to extract spatial images of sources without changing their location. The two channels of a stereo source can be modeled as uncorrelated to save on computation and data rate. In that case, the algorithm is most effective when the foreground objects are all single-channel and only the background has two channels (two distinct mono sources). Better results than those presented in the recent SiSEC should be possible with given covariances for the stereo sources. But, of course, at a higher side-information rate. Beyond, ISS can be viewed as a new formalism for a known coding principle for it has the following advantages. It has a modular framework that is backward compatible on the one hand and also upward extensible on the other hand. The PCMV filter, e.g., can be generalized to an arbitrary number of mixture channels, see Chapter 11, including multiple constraints. It was moreover observed that a frequency-weighted SNR can provide results just as reliable as any other metric that models human perception-but at a much lower computational cost. Nevertheless, the inconsistency between different performance metrics makes listening tests indispensable still.

The conclusions drawn from Chapter 8 are the following. EAOS is an MMSE estimator with a unnecessarily complicated residual coding strategy. This allows the conclusion that the MPEG SAOC decoder can be simplified. One may also infer that the entire SAOC system would improve, if the MMSE estimator was replaced by the PCMV filter. This is yet to be confirmed. A linear estimator's performance is bounded if the mixture is underdetermined due to its limited resolution. This gives rise to the necessity to provide the residual for a better quality. Other methods like [START_REF] Sturmel | Informed source separation using iterative reconstruction[END_REF] may also help. The sound quality of a remix is largely satisfactory if the mix is not lossy compressed. The quality noticeably degrades otherwise. It was also observed that the result is better when a fraction of the bit rate for the estimator is sacrificed in favor of a higher residual rate for the same overall data rate. Hybrid audio coding, such as aacPlus, is more efficient but also more costly and so suboptimal for real-time rendering and handheld devices.

The following Chapter 9 reveals that by knowing the parameters that were used for compression it is possible to recover the "dry" or uncompressed signal from the "wet" or compressed signal with high numerical accuracy and low computational effort. The figures prove that the SISO decompressor is real-time capable. This fact is exploited in Chapter 10. There, the decompressor is extended to two channels and combined with the PCMV estimator into a two-stage cascade that reverses mastering and mixing separately. If compression is undone with a negligible error, the demixed signals are almost identical with the ones obtained from an uncompressed mix. The decompressor is necessary to avoid artifacts such as "pumping", which might not be heard in the mix. Its accuracy could also be pivotal when effects like "reverb" are to be considered in the demixing stage.

Finally, Chapter 11 tells us that the sound quality is dependent on the array gain which increases with the number of mixture channels. For coding applications, the mixing system can be defined arbitrarily and a number of distinct tracks can be mixed into a lower number of channels in such a way that they exhibit the same SIR at the decoder. The sound quality can be foretold at the encoder. Hence, the number of channels can be chosen before actual mixing and the quality level after demixing can be controlled.

future outlook

Future work could consist in inverting other effects such as reverb in particular. The results could also be used in other disciplines with speech dereverbertion being one such example. The problem there is that the narrowband assumption does not hold anymore. One could also consider time-varying mixing and study its impact. Further, it is worth studying the effects of parameter quantization on the system's performance. How precise should the panoramic angle be? The used coding strategy for spectrograms could also be compared with other strategies for the MMSE and the PCMV filter separately. In this manner, one could better quantify the differences. Another possible direction is the estimation of compression parameters from the compressed signal given the uncompressed signal. One may also want to derive the characteristic function of more sophisticated compressor models which use a "soft" knee, parallel and multiband compression, etc. See [START_REF] Zölzer | DAFX: Digital audio effects[END_REF][START_REF] Jeffs | Dynamics Processors -Technology & Application Tips[END_REF][START_REF] Schmidt | Multichannel dynamic range compression for music signals[END_REF][START_REF] Giannoulis | Digital dynamic range compressor design-a tutorial and analysis[END_REF] and the references therein. Also, it might be interesting to see whether the decompressor is capable of restoring dynamics in over-compressed audio, see [START_REF] Goodwin | Frequency-domain algorithms for audio signal enhancement based on transient modification[END_REF][START_REF] Walsh | Adaptive dynamics enhancement[END_REF][START_REF] Zaunschirm | A sub-band approach to modification of musical transients[END_REF].

With regard to the cascade connection, one could study the effects that lossy data compression such as MP3 or AAC [START_REF] Brandenburg | MP3 and AAC explained[END_REF] brings along. It should behave similarly to high-capacity watermarking. In addition, the two-stage cascade should be tested on a larger dataset-and also with different compressor types and settings. Besides, it is thinkable to combine multiple constraints in the demixing stage if the mixture has more than two channels. This brings us to multichannel coding, i.e. the end. With the proposed scheme one would probably achieve higher data-rate savings if the redundancy between channels is also taken into account. Auditory perception is also something that could be considered in the scheme. One last direction worth mentioning is to find a direct mapping between the proposed metrics and, e.g., the corresponding mean opinion scores.

A S I G N A L AT T E N U AT I O N W I T H W I E N E R F I LT E R I N G

Based on the signal model from Section 7.2, the Wiener alias MMSE spatial filter, for an arbitrary frequency bin and for the duration of an arbitrary time segment, can be reformulated as

w io = R -1 x a i σ 2 i = σ 2 i det R x adj R x a i = σ 2 i det R x I l=1 σ 2 l adj a l a ❚ l a i = σ 2 i det R x I l=1 ρ il σ 2 l Qa l , (145)
where det R x is the determinant and adj R x the adjugate of R x . Above, The gain along the source's direction is

ρ il = det [ a i a l ] = a ❚ l Qa i with Q = 0 1 -1 0 . det
g i = w ❚ io a i (
In the ill-posed case, i.e. for I > 2, it can be noted that 0 ρ 2 il , ρ 2 uv < 1, and since σ 2 l , σ 2 v 0, the following inequalities hold true: (149) Equation ( 149) underlines that just like the classical Wiener filter, the spatial counterpart also attenuates the output signal at the attempt to minimize the MSE in TF points with a poor SIR. At worst, it may leave an audible spectral gap.

B M O D E L PA R A M E T E R E S T I M AT I O N

Consider the stereo image of a distinct source as given. From the stereo signal one can estimate the model parameters that are used as additional prior information for source separation. These parameters describe the source's location and how the signal power or variance is distributed over time and frequency.

First, one computes the zero-lag cross-covariance between the left and the right channel, and normalizes the former by the product of average powers in each channel using the RMS as measure:

corr (u 1i , u 2i ) = cov (u 1i , u 2i ) RMS 1i RMS 2i , ( 150
)
where the sample covariance is defined as

cov (u 1i , u 2i ) = 1 N N n=1 u 1i (n)u * 2i (n) (151)
with * denoting complex conjugation. The RMS is given by

RMS li = 1 N N n=1 u 2 li (n). (152)
When the source signal has zero mean, corr is identical with Pearson's correlation. The computation of (150) may be carried out in either the subband domain or the time domain. In the latter case, u li (n) is real.

The sample size N corresponds to the signal duration over which it can be considered wide-sense stationary and ergodic. The correlation coefficient corr may also be computed on a sample basis assuming non-stationarity. Then, if the sample variance

var {corr n (u 1i , u 2i )} = 1 N N n=1 [corr n (u 1i , u 2i ) -corr n (u 1i , u 2i )] 2 → 0, (153)
where

corr n (u 1i , u 2i ) = 1 N N n=1 corr n (u 1i , u 2i ), (154)
the source is considered as single-channel and its panoramic angle is estimated according to

αi = arccot RMS 2i RMS 1i , ( 155
)
where arccot is the arccotangent. In the reverse case, i.e. if the source is two-channel, its balance ratio is estimated as

â¬ref,i = RMS ¬ref,i RMS ref,i with a ref,i = 1, (156)
where ref ∈ {1, 2} is the channel with the greater RMS value and ¬ref is the channel with the smaller RMS value, respectively. The STPSD of a mono source or a stereo channel is finally estimated according to

where x(l + d) represents the sidechain signal, which is not delayed. From (161), one can see that to find |x(l)| one requires:

1. A future sample of |y|, namely |y(l + d)|.

2.

A future sample of g, namely g(l + d).

For n = 0, 1, . . .

Neither |x(d -1)| is known at instant l = 0, nor do we know anything about |x(l)| for l 0. If |x(0)| were known, we wouldn't have to look for it. Also note that (160) can no longer be expressed as an equation of a single unknown. Hence, the question of invertibility is subject to causality, which is only given for d = 0.

  power spectral density TF time-frequency (point, region, plane, etc.) TFR time-frequency representation TPS target-related perceptual score TTT two-to-three (MPEG SAOC)

3 . 4 . 5 . 6 . 9 .

 34569 The development of a constrained spatial filtering approach for sound source separation and narrowband deconvolution The derivation of a new spatial filter The inclusion of human perception to minimize the quantity of side information The elaboration of a novel and unprecedented approach for the inversion of a dynamic nonlinear time-variant operator, such as the dynamic range compressor 7. A two-stage cascade to reverse mastering and mixing 8. A novel multichannel object-based audio coding scheme The proposition of new perceptually motivated metrics for the objective assessment of sound quality Part II P R E L I M I N A R I E S D I G I TA L A U D I O E N G I N E E R I N G 2.1 audio effects and audio signal processors

Figure 1 :

 1 Figure 1: Modeling of sound sources in a sound field using the parameters direction (azimuth) and volume (radius)

Figure 2 :

 2 Figure2: Spatial pattern and power distribution of a two-channel filter. The beam is gain adjusted and directed such that the signal of interest is preserved and either one interferer is suppressed or, in case of multiple interferers, their mean total power is minimized.

Figure 3 :Figure 4 :Figure 5 :Figure 6 :Figure 7 :

 34567 Figure 3: SIR and TPS values (left column) complemented by the median, the 25th and 75th percentiles, and outliers (right column) for an excerpt from "The Terrorist" by DJ Vadim

Figure 9 :Figure 10 :

 910 Figure 9: Mean SDR, ISR, SIR, and SAR values for the SiSEC 2013 development dataset consisting of three music excerpts

Figure 12 :

 12 Figure 12: Decoder runtime as a function of the number of bands at a sideinformation rate of 30 kbps

L

  recover the modulus of the original signal |x(n)| from |y(n)| based on θ. For a more intuitive use, the smoothing factors α v and α g may be replaced by the time constants τ v and τ g . The meaning of each parameter is recapitulated below. p Detector type (peak or RMS) Threshold level in dB R Compression ratio dB in : dB out τ v,att Attack time of the envelope filter in ms τ v,rel Release time of the envelope filter in ms τ g,att Attack time of the gain filter in ms τ g,rel Release time of the gain filter in ms M Makeup gain in dB 9.4 proposed solution

Figure 15 :

 15 Figure 15: Graphical illustration of the iterative search for the root

Figure 16 :Figure 17 :Figure 18 :

 161718 Figure16: An illustrative example using an RMS amplitude detector with τ v set to 5 ms, a threshold of -20 dBFS (dashed line in the upper right corner), a compression ratio of 4 : 1, and τ g set to 1.6 ms for attack and 17 ms for release, respectively. The RMSE is ca. -129 dBFS.

Figure 19 :

 19 Figure 19: RMSE as a function of threshold relative to the signal's average loudness level (left column) and compression ratio (right column) using a peak (upper row) or an RMS amplitude detector (lower row). The time constants are: τ v = 5 ms, τ att g = 20 ms, and τ rel g = 1 s.

3 . 5 .. 8 6.Figure 20 : 4 6.

 358204 Figure 20: A two-stage cascade connection

T

  enc (I, Z, N) = O(I • N log N) • N log N) T dec (I, Z, N) = O(I • N log N)

10.5 performance evaluation 10 . 5 . 1

 1051 Experimental Setup

Figure 21 :

 21 Figure 21: RMSE and PSM values for the multitrack (upper row) and the corresponding difference values between the estimates from the de-/compressed and the uncompressed mixture signal (lower row). The asterisk ( * ) indicates M = 0 (no gain).

  the STPSD of the ith source signal. The source signals are linearly combined into an M-channel mixture signal (M < I) according to x(k, m) = I i=1

Figure 22 :

 22 Figure 22: The proposed coding scheme. The asterisk ( * ) indicates that the coding/decoding blocks may also include watermarking/signing functionality.

  (ρ 1 , ρ 2 , . . . , ρ I ) end function 11.4.3 Source Separation Equations (126), (135), and (136) constitute the mixing rule that is used on the encoder side during mixdown. Having knowledge of this rule on the decoder side means knowing the mixing matrix A, provided that the number of objects I is known. The transmission of the mixing coefficients can hence be omitted. Using (127), (128), and (130), we can formulate a joint demixing operation being y(k, m) = W ❚ x(k, m).

Figure 23 :

 23 Figure23: Mean SIXFP, SIRFP, and ODG values for the multitrack and the corresponding data-rate savings. Starting from the middle, we see that for imperceptible quality impairment, i.e. an ODG above -1, the SIRFP must be 48 or greater (shaded area). Switching over to the left, we see that at least 7 channels are necessary to reach it. The figure on the right indicates that 30 % of LPCM data can so be saved.

  Φs i (k, m) = Ŝi (k, m) m) = sin αi cos αi u i (k, m)(158)in the case of a mono source, or elseŜref,i (k, m) = U ref,i (k, m) of a stereo source. C I N V E R T I B I L I T Y O F A C O M P R E S S O R W I T H L O O K A H E A DThe output of a compressor with a delay line in the main signal path is given byy(n) = g(n)x(nd) with m = 1, (160)where d is the delay in samples. Making the substitution l = nd in (160), the above equation can be restated as|x(l)| = |y(l + d)| g(l + d) . (161) Also recall that g(l + d) = G [|x(l + d)| | θ, x(l + d -1), g(l + d -1)],

, d - 1 ,

 1 |y(l + d)| can be assumed zero due to causality, and thus known. Equation (162) yet says that to compute g(l + d), one requires a future sample of x, which is unknown. More precisely,x(l + d -1) = α v |x(l + d -1)| p + (1α v )x(l + d -2).(163) Evidently, one requires |x(l + d -1)| to compute g(l + d) as well. For the very first non-zero sample of x, i.e. for n = d or l = 0, x(d -1) = α v |x(d -1)| p + (1α v )x(d -2).

  SIRFP, and ODG values for the multitrack and the corresponding data-rate savings. Starting from the middle, we see that for imperceptible quality impairment, i.e. an ODG above -1, the SIRFP must be 48 or greater (shaded area). Switching over to the left, we see that at least 7 channels are necessary to reach it. The figure on the right indicates that 30 % of LPCM data can so be saved. 106

	xviii xxiv	contents acronyms	contents	xix
	. I thank you all very much for what you have said or done that helped me finish my thesis. Dedicated in loving memory to Waldemar Stab C O N T E N T S i introduction 1 1 introduction 3 1.1 Motivation 3 1.2 Objectives 3 1.2.1 Active Listening 3 1.2.2 Reverse Audio Engineering 4 1.3 Organization 4 ii preliminaries 7 2 digital audio engineering 9 2.1 Audio Effects and Audio Signal Processors 9 2.2 Mixing 11 2.3 Mastering 11 3 ill-posed inverse problems 13 3.1 Inverse Problems 13 3.2 Ill-Posedness 13 4 probabilistic approach 15 4.1 Bayesian Inference 15 4.2 Linear Optimum Filtering 16 4.3 Post-Nonlinear Mixtures 17 5 informed approach 19 5.1 Basic Idea 19 5.2 Concretization 19 5.2.1 Problem Formulation 20 5.2.2 Proposed Solution 20 5.2.3 Associated Problem 20 iii reverse audio engineering 21 6 previous works 23 7 audio source separation 27 7.1 Intensity Stereophony 27 7.2 Signal Model 27 7.3 System Model 28 7.3.1 Image Signal 28 7.3.2 Mixture Signal 29 7.4 Problem Formulation 29 7.5 Proposed Solution 30 7.5.1 Well-Posed Case 30 7.5.2 Ill-Posed Case 32 7.5.3 Narrowband Convolutional Case 33 7.5.4 Two-Channel Case 34 7.5.5 Noisy Case 35 7.6 General Remarks 35 7.6.1 Precedence or Haas Effect 35 7.6.2 Multi-Constraint Spatial Filter 36 7.6.3 Wiener vs. PCMV Spatial Filter 36 7.6.4 Mixture Covariance Matrix 37 7.7 Parameter Quantization and Coding 38 7.7.1 Panoramic Angle 38 7.7.2 Balance Ratio 38 7.7.3 Short-Time Power Spectrum 38 7.8 Performance Evaluation 40 7.8.1 Instantaneous Mixture with Watermarking 7.8.2 Narrowband Convolutional Mixture 42 7.8.3 Separation of Spatial Images 44 8 comparison with mpeg saoc 59 8.1 Enhanced Audio Object Separation 59 8.2 Object Encoder 59 8.3 Object Decoder 61 8.4 Performance Evaluation 61 8.4.1 EAOS vs. USSR 61 8.4.2 Interactive Remixing 63 9 inversion of dynamic range compression 67 9.1 Dynamic Range Compression 67 9.2 System Model 69 9.2.1 Feed-Forward Broadband Compression 69 9.2.2 Stereo Linking 70 9.3 Problem Formulation 70 9.4 Proposed Solution 71 9.4.1 Characteristic Function 72 9.4.2 Attack-Release Phase Toggle 73 9.4.3 Envelope Predictor 74 9.4.4 Stereo Unlinking 74 9.4.5 Error Analysis 74 9.5 Numerical Approximation 76 9.6 General Remarks 77 9.6.1 Lookahead 77 9.6.2 Clipping and Limiting 77 9.6.3 Logarithmic Gain Smoothing 78 9.7 Pseudocode 78 9.8 Parameter Quantization and Coding 78 9.9 Performance Evaluation 78 9.9.1 Experimental Setup 78 9.9.2 Experimental Results 80 10 two-stage cascade connection 89 10.1 System Overview 89 10.2 Cascade Encoder HRTF head-related transfer function 89 10.3 Cascade Decoder 89 10.4 Performance Characteristics 91 10.4.1 Algorithmic Delay 91 10.4.2 Computational Complexity 91 10.4.3 Side-Information Rate 91 10.5 Performance Evaluation 93 10.5.1 Experimental Setup 93 10.5.2 Experimental Results 94 11 multichannel object-based audio coding 97 11.1 Introduction 97 11.2 Signal and System Model 98 11.3 Problem Formulation 98 11.4 Proposed Solution 98 11.4.1 System Overview 98 11.4.2 Mixdown 99 11.4.3 Source Separation 102 11.5 Quality Control Mechanism 103 11.5.1 Quality Metrics 103 11.5.2 Control Mechanism 104 11.6 Performance Evaluation 104 11.6.1 Experimental Setup 104 11.6.2 Experimental Results 105 iv conclusion 107 12 conclusion and future outlook 109 12.1 Conclusion 109 12.2 Future Outlook 111 bibliography 113 a signal attenuation with wiener filtering 123 b model parameter estimation 125 c invertibility of a compressor with lookahead 127 L I S T O F F I G U R E S Figure Modeling of sound sources in a sound field using 40 the parameters direction (azimuth) and volume (ra-dius) 28 Figure Spatial pattern and power distribution of a two-channel filter. The beam is gain adjusted and directed such that the signal of interest is preserved and either one interferer is suppressed or, in case of multiple inter-ferers, their mean total power is minimized. 31 Figure SIR and TPS values (left column) complemented by Figure The medians and the 25th and 75th percentiles for each system under test 66 Figure Basic broadband compressor model 68 Figure Graphical illustration of the iterative search for the root 77 Figure An illustrative example using an RMS amplitude de-tector with τ v set to 5 ms, a threshold of -20 dBFS (dashed line in the upper right corner), a compres-sion ratio of 4 : 1, and τ g set to 1.6 ms for attack and 17 ms for release, respectively. The RMSE is ca. -129 dBFS. 84 Figure RMSE as a function of typical attack and release times using a peak (upper row) or an RMS amplitude de-Figure Table 12 RMSE and SNR for the three mixture signals 94 L I S T O F A L G O R I T H M S Algorithm 1 Single-input single-output compressor 79 Algorithm 2 Single-input single-output decompressor 80 Algorithm 3 Root-finding algorithm 81 IIR infinite impulse response IOC inter-object cross coherence (MPEG SAOC) IPS interference-related perceptual score ISR image-to-spatial distortion ratio ISS informed source separation JND just-noticeable difference JPEG Joint Photographic Experts Group KBD Kaiser-Bessel derived LCMV linearly constrained minimum-variance (filtering, e.g.) Mean SIXFP, L I S T O F TA B L E S Algorithm 4 Equal-SIR out power distribution LPCM linear pulse-code modulation 102 LTI linear time-invariant tector (lower row). In the left column, the attack time the median, the 25th and 75th percentiles, and out-liers (right column) for an excerpt from "The Terror-ist" by DJ Vadim 47 Figure SIR and TPS values (left column) complemented by the median, the 25th and 75th percentiles, and out-liers (right column) for an excerpt from "Lisztoma-nia" by Phoenix 48 Figure 49 Figure SNRF and auditory bandwidth values (left column) of the envelope filter is varied while the release time Table LTV linear time-variant Studio effects divided into categories and classified A C R O N Y M S AAC Advanced Audio Coding APS artifacts-related perceptual score BGO background object (MPEG SAOC) BSS blind source separation LS least squares MMSE minimum mean square error MOS mean opinion score MP3 MPEG-1 or MPEG-2 Audio Layer III MPEG Motion Picture Experts Group MPS MPEG surround MSE mean squared error as LTI ( * ), 56 CD Compact Disc complemented by the median, the 25th and 75th per-centiles, and outliers (right column) for an excerpt from "Lisztomania" by Phoenix 50 Figure Mean opinion scores and 95-% confidence intervals Table The corpus of prearranged mixes 65 Table CDDA Compact Disc Digital Audio Selected compressor settings 81 Table CIPIC Center for Image Processing and Integrated Computing Performance figures obtained for real-world audio material (12 items in total) 83 DAW digital audio workstation for the two music excerpts (left column) and the overall grades for the algorithms under test (right column) 51 Figure SDR and PSM values for the oracle MMSE spatial fil-ter (upper row) and the average rate-performance difference curves for three different filtering and/or coding strategies (lower row) 52 Figure Mean SDR, ISR, SIR, and SAR values for the SiSEC 2013 development dataset consisting of three music ex-cerpts 57 Figure Mean OPS, TPS, IPS, and APS values for the SiSEC 2013 development dataset consisting of three music ex-Table Runtime complexity of the cascade connection as a DFT discrete Fourier transform g = 1 s. 87 Figure A two-stage cascade connection 90 Figure RMSE and PSM values for the multitrack (upper row) function of the number of sources, the number of DPCM differential pulse-code modulation frequency bands, and the frame length. It is assumed that the transform length is equal to the frame length. DRC dynamic range compression 92 Table 10 Information rate of STPSD values (left) and STPSD dif-EAO enhanced audio object (MPEG SAOC) and the corresponding difference values between the estimates from the de-/compressed and the uncom-pressed mixture signal (lower row). The asterisk ( * ) indicates M = 0 (no gain). 95 Figure ference values using DPCM (right) if quantized with 6 EAOS enhanced audio object separation PEASS Perceptual Evaluation methods for Audio Source bits at 44.1-kHz sample rate and 16-kHz cutoff 93 Table 11 Compressor setting used for the complete mix ERB Separation equivalent rectangular bandwidth 93 FAAC Freeware Advanced Audio Coder PSM perceptual similarity metric The proposed coding scheme. The asterisk ( * ) indi-FFT fast Fourier transform RMS root mean square cates that the coding/decoding blocks may also in-99 FGO foreground object (MPEG SAOC) RMSE root-mean-square error clude watermarking/signing functionality. cerpts 58 Figure FIR finite impulse response SAC spatial audio coding ODG as a function of the side-information rate 62 Figure at a side-information rate of 30 kbps 63 HRIR head-related impulse response Decoder runtime as a function of the number of bands FLAC Free Lossless Audio Codec SAOC spatial audio object coding
		xx xxii	xi xvii xxi xxiii

SNRF and auditory bandwidth values (left column) complemented by the median, the 25th and 75th percentiles, and outliers (right column) for an excerpt from "The Terrorist" by DJ Vadim is held constant. The right column shows the reverse case. The time constants of the gain filter are fixed at zero. In all four cases, threshold and ratio are fixed at -32 dBFS and 4 : 1, respectively. 85 Figure RMSE as a function of typical attack and release times using a peak (upper row) or an RMS amplitude detector (lower row). In the left column, the attack time of the gain filter is varied while the release time is held constant. The right column shows the reverse case. The time constants of the envelope filter are fixed at zero. In all four cases, threshold and ratio are fixed at -32 dBFS and 4 : 1, respectively. 86 Figure RMSE as a function of threshold relative to the signal's average loudness level (left column) and compression ratio (right column) using a peak (upper row) or an RMS amplitude detector (lower row). The time constants are: τ v = 5 ms, τ att g = 20 ms, and τ rel LTV ( †), NTI ( ‡), and NTV ( §). Effects considered in this work are in boldface. 10 Table Panning used for the two music pieces 42 Table Estimated image parameters for the development set 45 Table Experimental results. The ISR, SAR, and TPS values for mono sources are framed. 54 Table A comparison between the results shown in Table 4 and the scores reported in SiSEC 2011 for two oracle systems. The upper row next to the track name represents the STFT-based system and the lower row represents the cochleagram-based system, respectively. MUSHRA MUlti-Stimulus test with Hidden Reference and Anchor MVDR minimum-variance distortionless response NMF non-negative matrix factorization NTI nonlinear time-invariant NTV nonlinear time-variant ODG objective difference grade OPS overall perceptual score PCMV power-conserving minimum-variance (filtering, e.g.) PEAQ Perceptual Evaluation of Audio Quality SAR source-to-artifacts ratio SDR signal-to-distortion ratio

Table 1 :

 1 Studio effects divided into categories and classified as LTI ( * ), LTV ( †), NTI ( ‡), and NTV ( §). Effects considered in this work are in boldface.

	digital audio engineering
	effect	l	ti	memory	i/o
		Distortion effects
	Overdrive/fuzz †				SISO 1
		Dynamics processors
	Fader/amplifier *				SISO
	Compressor/				SISO/
	limiter §				MISO 2
	Noise gate †				SISO
		Filter effects
	Equalizer *				SISO
	Roll-off *				SISO
		Modulation effects
	Chorus ‡				MISO
	Flanger ‡				MISO
	Phaser ‡				MISO
	Ring modulator ‡				MISO
	Tremolo ‡				MISO
	Vibrato ‡				MISO
		Pitch and frequency effects
	Pitch shifter *				SISO
	Harmonizer *				MISO
		Time-based effects
	Delay *				SISO
	Reverb *				SISO
		Spatial effects
	Panning *				SIMO 3
	Fattening *				SIMO

Table 2 :

 2 Panning used for the two music pieces

  .

	track	type	α	a ¬ref/a ref
	Vocal	stereo	-	0.89
	Drums	stereo	-	1.00
	Guitar	stereo	-	0.96
	"The Ones We Love" by Another Dreamer
	Vocal	stereo	-	0.99
	Bass	mono	45.0	-
	Piano	stereo	-	0.83
	Background	stereo	-	0.94
		"Roads" by Bearlin
	Vocal	stereo	-	0.90
	Drums	stereo	-	0.99
	Bass	mono	45.0	-
	Claps	stereo	-	0.99
	Background	stereo	-	0.97
		"Remember the Name" by Fort Minor
	Vocal	mono	47.9	-
	Guitar	stereo	-	0.97
		"Que Pena/Tanto Faz" by Tamy
	Vocal	stereo	-	1.00
	Drums	stereo	-	0.97
	Bass	stereo	-	0.93
	Background	stereo	-	0.93
		Ultimate NZ Tour

Table 3 :

 3 Estimated image parameters for the development set 7.8.3.2 Experimental results

Table 4 :

 4 SDR and PSM values for the oracle MMSE spatial filter (upper row) and the average rate-performance difference curves for three different filtering and/or coding strategies (lower row) Experimental results. The ISR, SAR, and TPS values for mono sources are framed.

		(d)
		1
	MMSE JPEG MMSE NMF	∆PSM (PEMO-Q)	MMSE JPEG MMSE NMF
	PCMV DPCM		PCMV DPCM
		Bitrate per source (kbps)

Table 5 :

 5 A comparison between the results shown in Table4and the scores reported in SiSEC 2011 for two oracle systems. The upper row next to the track name represents the STFT-based system and the lower row represents the cochleagram-based system, respectively.

Table 8 :

 8 Performance figures obtained for real-world audio material (12 items in total)

	a		b		c		d		e
	Peak	RMS	Peak	RMS	Peak	RMS	Peak	RMS	Peak	RMS

Table 9 :

 9 Runtime complexity of the cascade connection as a function of the number of sources, the number of frequency bands, and the frame length. It is assumed that the transform length is equal to the frame length.

Table 10 :

 10 Information rate of STPSD values (left) and STPSD difference values using DPCM (right) if quantized with 6 bits at 44.1-kHz sample rate and 16-kHz cutoff

	subdivision	number of	bitrate in kbps
	factor	frequency bands	per source
	-	39	10.1 / 5.88
	2	76	19.6 / 11.5
	3	108	28.0 / 16.3
	4	136	35.2 / 20.6
	5 . . .	163 . . .	42.2 / 24.6 . . .
	parameter	description	value
	p	Type	RMS
	L (dBFS)	Threshold	-32.0
	R (dB in : dB out )	Ratio	3.0 : 1
	τ v,att (ms) τ v,rel (ms)	Envelope attack Envelope release	5.0
	τ g,att (ms)	Gain attack	13.0
	τ g,rel (ms)	Gain release	435
	M (dB)	Makeup	9.0

Table 11 :

 11 Compressor setting used for the complete mix

Table 12 :

 12 RMSE and SNR for the three mixture signals

	mixture type	rmse (dbfs)	snr (db)
	Compressed	-31.8	3.08
	Compressed *	-36.0	7.27
	Decompressed	-62.3	33.6

Note that the STPSD is tantamount to the STFT of the auto-covariance function

The notion of spatial "images" in a source separation context can also be found in[START_REF] Duong | Underdetermined reverberant audio source separation using a fullrank spatial covariance model[END_REF], e.g.

In[START_REF] Algazi | The CIPIC HRTF database[END_REF][START_REF]The CIPIC HRTF database[END_REF], e.g., the HRIR is 200 taps long

This is by definition the case for a stereo source

In the case of an instantaneous mixture, the mixing coefficients can also be estimated from the mixture signal using the algorithm in[START_REF] Reju | An algorithm for mixing matrix estimation in instantaneous blind source separation[END_REF].

The term "mixing rule" means a set of distinct relations between input and output variables including the mixing system but also its definition

Algorithm 4 uses the MVDR spatial filter to estimate Φ y i

As can be seen from Table 8, the largest inversion error is associated with setting E and the smallest with setting B. For all five settings, the error is larger when an RMS detector is in use. This is partly due to the fact that ζ 2 (v) has a stronger curvature in comparison to ζ 1 (v). By defining the distance in (116) as ∆ p |ζ p (v)|, it is possible to attain a smaller error for an RMS detector at the cost of a slightly longer runtime. In most cases, the envelope predictor works more reliably as compared to the toggle switch between attack and release. It can also be observed that the choice of time constants seems to have little impact on decompressor's accuracy. The major parameters that affect the decompressor's performance are L and R, while the threshold is evidently the predominant one: the RMSE strongly correlates with the threshold level.

Figs. 17-18 show the inversion error as a function of various time constants. These are in the range of typical attack and release times