Paul Magron

Senior Member, IEEE Tuomas Virtanen

Online Spectrogram Inversion for Low-Latency Audio Source Separation

Keywords: Audio source separation, low-latency, online spectrogram inversion, phase recovery, sinusoidal modeling

Audio source separation is usually achieved by estimating the short-time Fourier transform (STFT) magnitude of each source, and then applying a spectrogram inversion algorithm to retrieve time-domain signals. In particular, the multiple input spectrogram inversion (MISI) algorithm has been exploited successfully in several recent works. However, this algorithm suffers from two drawbacks, which we address in this paper. First, it has originally been introduced in a heuristic fashion: we propose here a rigorous optimization framework in which MISI is derived, thus proving the convergence of this algorithm. Besides, while MISI operates offline, we propose here an online version of MISI called oMISI, which is suitable for low-latency source separation, an important requirement for e.g., hearing aids applications. oMISI also allows one to use alternative phase initialization schemes exploiting the temporal structure of audio signals. Experiments conducted on a speech separation task show that oMISI performs as well as its offline counterpart, thus demonstrating its potential for real-time source separation.

I. INTRODUCTION Audio source separation

 [START_REF] Comon | Handbook of blind source separation: independent component analysis and applications[END_REF]consists in extracting the underlying sources that add up to form an observable audio mixture. This task finds applications in many areas such as speech enhancement and recognition [START_REF] Barker | The fifth 'chime' speech separation and recognition challenge: Dataset, task and baselines[END_REF], musical signal processing [START_REF] Cano | Musical source separation: An introduction[END_REF] and hearing aid devices [START_REF] Wang | Time-frequency masking for speech separation and its potential for hearing aid design[END_REF]. In particular, hearing aids require a very low processing latency, as significant discomfort can be experienced by listeners for delays exceeding 20 ms [START_REF] Agnew | Just noticeable and objectionable group delays in digital hearing aids[END_REF]. State-of-the-art approaches for source separation consist in using a deep neural network (DNN) to estimate a nonnegative mask that is applied to a time-frequency (TF) representation of the audio mixture, such as the short-time Fourier transform (STFT) [START_REF] Wang | Supervised speech separation based on deep learning: An overview[END_REF]. Recent works such as [START_REF] Luo | TaSNet: Time-domain audio separation network for real-time, single-channel speech separation[END_REF], [START_REF]Conv-TasNet: Surpassing ideal timefrequency magnitude masking for speech separation[END_REF] operate in the time domain directly, but TF approaches remain interesting since they make it possible to better exploit the structure of sound.

Applying a nonnegative mask to the input STFT results in assigning the mixture's phase to each isolated source. Even though this yields somewhat satisfactory results in practice, it has been pointed out [START_REF] Magron | Phase recovery in NMF for audio source separation: an insightful benchmark[END_REF] that when sources overlap in the TF domain, using the mixture's phase induces residual interference and artifacts in the estimates. With the advent of deep learning, magnitudes can nowadays be estimated with a high accuracy, which highlights the need for advanced phase recovery [START_REF] Gerkmann | Phase Processing for Single-Channel Speech Enhancement: History and recent advances[END_REF], [START_REF] Mowlaee | Advances in phase-aware signal processing in speech communication[END_REF], a problem hereafter termed spectrogram inversion. Consequently, several recent works have P. Magron is with IRIT, Université de Toulouse, CNRS, France (paul.magron@irit.fr). T. Virtanen is with the Audio Research Group, Tampere University, Finland (tuomas.virtanen@tuni.fi). The work of P. Magron was conducted while he was with Tampere University and supported by the Academy of Finland, project no. 290190. focused on phase recovery in DNN-based source separation, whether phase recovery algorithms are applied as a postprocessing [START_REF] Wang | Alternative objective functions for deep clustering[END_REF], [START_REF] Magron | Reducing interference with phase recovery in DNN-based monaural singing voice separation[END_REF] or integrated within end-to-end systems for time-domain separation [START_REF] Wang | End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction[END_REF], [START_REF] Wichern | Phase reconstruction with learned timefrequency representations for single-channel speech separation[END_REF], [START_REF] Wang | Deep learning based phase reconstruction for speaker separation: A trigonometric perspective[END_REF], [START_REF] Wisdom | Differentiable consistency constraints for improved deep speech enhancement[END_REF], [START_REF] Le Roux | Phasebook and friends: Leveraging discrete representations for source separation[END_REF].

Among the variety of phase recovery techniques, the multiple input spectrogram inversion (MISI) algorithm [START_REF] Gunawan | Iterative phase estimation for the synthesis of separated sources from single-channel mixtures[END_REF] is particularly popular. This iterative procedure consists in retrieving time-domain sources from their STFT magnitudes while respecting a mixing constraint: the estimates must add up to the original mixture. This algorithm has shown promising for audio source separation when combined with DNNs [START_REF] Wang | Alternative objective functions for deep clustering[END_REF], [START_REF] Wang | End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction[END_REF]. However, the MISI algorithm has been introduced in a heuristic fashion, therefore there is currently no proof that it converges. Besides, while several recent works addressed the problem of low-latency magnitude estimation [START_REF] Naithani | Low latency sound source separation using convolutional recurrent neural networks[END_REF], [START_REF] Aihara | Teacher-student deep clustering for low-delay single channel speech separation[END_REF], [START_REF] Wang | Low-latency deep clustering for speech separation[END_REF], the MISI algorithm operates offline, as it computes the whole STFT and its inverse at each iteration. This makes it impracticable for real-time applications such as hearing aids.

In this paper, we investigate and overcome the drawbacks of the MISI algorithm. First, we propose a rigorous optimization framework for spectrogram inversion. Using the auxiliary function method we derive a procedure that is equivalent to the MISI algorithm. To the best of our knowledge, this is the first proof of convergence of MISI. Second, we propose an online adaptation of MISI that is suitable for low-latency applications. This adaptation is based on approximating the STFT and its inverse in a causal fashion by only accounting for the past context and for an arbitrarily small number of future frames. It also allows us to exploit the temporal structure of the phase to use alternative phase initialization schemes. Here, we propose to use a sinusoidal phase [START_REF] Mowlaee | Harmonic phase estimation in singlechannel speech enhancement using phase decomposition and SNR information[END_REF] instead of the mixture's phase as initial estimate. We experimentally demonstrate the potential of this technique for a speech separation task.

The rest of this paper is structured as follows. Section II presents some mathematical notations. The MISI algorithm is derived in Section III and adapted online in Section IV. Section V presents the experimental results, and Section VI concludes the paper.

II. MATHEMATICAL NOTATIONS

• A (capital, bold font): matrix.

• x (lower case, bold font, with tilde): time-domain signal, whose n-th sample is denoted x(n). • x (lower case, bold font, without tilde): TF domain vector, whose m-th entry is denoted x(m). • z (regular): scalar.

• |.|, ∠(.): magnitude and complex angle, respectively.

• x T , x H : transpose and Hermitian transpose, respectively. . : element-wise matrix or vector multiplication, and division, respectively.

III. ALGORITHM DERIVATION

In this section we derive MISI using the auxiliary function method, which proves the convergence of this algorithm.

A. Problem setting

Let us consider an instantaneous and linear mixing model:

x = J j=1 sj , (1)
where x ∈ R N is the mixture, sj ∈ R N are the J sources, and N denotes the length of the time-domain signals in samples. The STFTs of the sources are denoted by s j ∈ C M with M = F × T , where F and T are the number of frequency channels and time frames respectively, and we have

s j = As j , (2)
where the matrix A ∈ C M ×N encodes the STFT operation.

Let us assume that some STFT magnitude estimates denoted v j ∈ R M + are available (e.g., estimated beforehand using a DNN). The goal of multiple-input spectrogram inversion is to estimate time domain source signals sj given the STFT magnitude estimates v j . Since these magnitude estimates are usually not equal to the ground truth, one should allow the magnitudes of the estimated sources to deviate from those values. Thus, we consider the following objective function:

ψ(s) = j |||As j | -v j || 2 , (3)
We treat the mixing constraint (1) as a hard constraint, leading to the following optimization problem:

min s ψ(s) subject to j sj = x. (4)
Directly addressing the optimization problem (4) is however difficult since ψ is not differentiable with respect to s. Therefore, we propose to use the auxiliary function method [START_REF] Févotte | Algorithms for nonnegative matrix factorization with the beta-divergence[END_REF], which consists in finding a function ψ + (s, y) such that

ψ(s) = min y ψ + (s, y), (5)
where y is a set of auxiliary parameters. It can be shown [START_REF] Févotte | Algorithms for nonnegative matrix factorization with the beta-divergence[END_REF] that ψ is non-increasing when alternating the minimization of ψ + with respect to (w.r.t.) s and y.

B. Auxiliary function

Let us first introduce a set of auxiliary parameters y j such that |y j | = v j and rewrite (3) as:

ψ(s) = j |||As j | -|y j ||| 2 . ( 6
)
We then use the property

∀(z, z ) ∈ C 2 , ||z| -|z || ≤ |z -z |, (7)
that arise from the triangle inequality, and where equality holds if and only if ∠z = ∠z . This leads to ψ(s) ≤ ψ + (s, y) with:

ψ + (s, y) = j ||As j -y j || 2 . (8)
In order to minimize ψ + w.r.t. y under the constraint |y j | = v j , we introduce this constraint using the Lagrange multipliers method. We therefore aim at finding a saddle point for:

ψ + (s, y) + j,m λ j (m)(|y j (m)| 2 -v j (m) 2 ), (9)
where λ j ∈ R M are the Lagrange multipliers. We set the partial derivative of (9) w.r.t y at 0, which leads to:

(1 + λ j (m))y j (m) = s j (m). ( 10
)
Using the constraint |y j (m)| = v j (m), we have

|1 + λ j (m)| = |s j (m)| v j (m) . (11)
Finally, injecting ( 11) into (10) leads to the update for y j :

y j = ± s j |s j | v j , (12)
We consider the update that does not modify the phase of s j (i.e., with a '+' sign in ( 12)), as it corresponds to the equality case of Eq. ( 7). Under such an update, ψ(s) = ψ + (s, y), which shows that ψ + is an auxiliary function for ψ.

C. Including the mixing constraint

Now, let us introduce the hard mixing constraint (1) within the auxiliary function ψ + by means of the Lagrange multipliers as in [START_REF] Magron | Model-based STFT phase recovery for audio source separation[END_REF]. This results in finding a saddle point for:

Ψ(s, y, δ) = j ||As j -y j || 2 + 2  δ H ( j sj -x)   , (13)
where δ ∈ C N is the vector of Lagrange multipliers. Setting the derivative of Ψ w.r.t. sj at 0 yields:

2A H As j -2A H y j + 2 δ = 0. ( 14
)
Let us point out that the matrix A H encodes the inverse STFT [START_REF] Roux | Consistent Wiener filtering for audio source separation[END_REF]. Indeed, one can show [START_REF] Yang | A study of inverse short-time Fourier transform[END_REF] that if the synthesis window is equal to the analysis window up to a specific normalization constant [START_REF] Griffin | Signal estimation from modified short-time Fourier transform[END_REF], the STFT is an Hermitian operator.

Assuming such analysis-synthesis windows are used, A H A is the identity matrix. We define:

ỹj = A H y j = iSTFT(y j ), (15)
and therefore Eq. ( 14) rewrites:

sj -ỹj + δ = 0. (16)
Summing ( 16) over j and using the mixing constraint yields:

x -

j ỹj + J δ = 0. ( 17
)
Finally, solving for δ and injecting it in Eq. ( 16) leads to:

sj = ỹj + 1 J (x - p ỹp ). ( 18
)
Combining the updates given by Eq. ( 2), ( 12), [START_REF] Wichern | Phase reconstruction with learned timefrequency representations for single-channel speech separation[END_REF], and ( 18) yields the MISI algorithm, as introduced in the original paper [START_REF] Gunawan | Iterative phase estimation for the synthesis of separated sources from single-channel mixtures[END_REF]; this derivation therefore proves its convergence.

IV. ONLINE MISI

First, let us reshape the STFTs s j onto matrix form as they are usually processed this way: S j ∈ C F ×T . We rewrite the MISI algorithm in the TF domain, as done in [START_REF] Wang | End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction[END_REF], [START_REF] Wichern | Phase reconstruction with learned timefrequency representations for single-channel speech separation[END_REF]: ∀j,

Z j = STFT(iSTFT(S j )), (19)
Y j = Z j |Z j | V j , (20)
S j = Y j + 1 J (X - p Y p ), (21)

A. Problem setting

It is straightforward to implement ( 20) and ( 21) online, as these are performed bin-wise, but this is not the case of [START_REF] Gunawan | Iterative phase estimation for the synthesis of separated sources from single-channel mixtures[END_REF]. Indeed, the inverse STFT of S j is computed through the overlap-add (OLA) procedure as follows:

s j,t = iDFT(S j,t ) w, (22) sj (n)
= T -1 t=0 s j,t (n -tl), (23)
where S j,t is the t-th column of S j , iDFT denotes the inverse discrete Fourier transform, w is a window of length N w , and l is the hop size. For notation convenience, we consider that s j,t (n -tl) = 0 if n / ∈ {0, ..., N w -1}. Using OLA [START_REF] Mowlaee | Harmonic phase estimation in singlechannel speech enhancement using phase decomposition and SNR information[END_REF], a sample n is reconstructed by accounting for all frames whose iDFT has a temporal support that includes n, which is not suitable for online applications.

B. Online implementation

Let us assume that we are currently processing the frame indexed by t. We first decompose the OLA procedure [START_REF] Mowlaee | Harmonic phase estimation in singlechannel speech enhancement using phase decomposition and SNR information[END_REF] as:

sj (n) = t-1 k=0 s j,k (n -tl) spast j,t (n) + T -1 k=t s j,k (n -tl) sfut j,t (n) , (24)
where spast j,t (resp. sfut j,t ) contains the contributions of the previous (resp. current and future) frames. Drawing on prior work [START_REF] Beauregard | An efficient algorithm for real-time spectrogram inversion[END_REF], [START_REF] Zhu | Real-time iterative spectrum inversion with look-ahead[END_REF], [START_REF] Zhu | Real-time signal estimation from modified short-time Fourier transform magnitude spectra[END_REF], we propose hereafter to approximate sfut j,t by using only the current frame and an arbitrarily small number K ≥ 0 of future frames:

sfut j,t (n) ≈ t+K k=t s j,k (n -tl). ( 25
)
Even though this approach results in losing the contributions of some of the future time frames involved in the calculation of sj (n) (cf. IV-A), it still enforces a form of coherence over time 22) and ( 25)

7 ∀j: Z j = STFT(s past j + sfut j )
8 ∀j: Update S j,t , ..., S j,t+K using ( 20) and ( 21)

9 end 10
Compute, ∀j, s j,t from S j,t using ( 22)

11
Update the sources: ∀j, sj (tl + n) = s j,t (n) + spast

j (n) for n < l 12
Update the past segments: ∀j, spast j (n) = s j,t (n + l) for n < N w -l and 0 otherwise 13 end 14 Outputs: sj by accounting for the overlap with the previous time frames. It also preserves the coherence with the near-future frames which overlap with the current one. With such an approach, the algorithmic latency is reduced to N w + Kl samples. It is illustrated in Fig. 1. This procedure is called oMISI (for "online MISI") and summarized in Algorithm 1.

C. Phase initialization

MISI is usually initialized by assigning the mixture's phase to each source. However, its online implementation makes it possible to use an alternative initialization scheme, exploiting phase relationships over time. In particular, we propose here to use a phase model that arise from modeling audio signals as mixtures of sinusoids [START_REF] Mcauley | Speech analysis/Synthesis based on a sinusoidal representation[END_REF], [START_REF] Krawczyk | STFT phase reconstruction in voiced speech for an improved single-channel speech enhancement[END_REF]. It can be shown [START_REF] Magron | Model-based STFT phase recovery for audio source separation[END_REF] that the phase ϕ j of a source represented as a mixture of slowlyvarying sinusoids follows the relationship: where ν j,t (f ) is the normalized frequency in channel f and time frame t. We propose to use this model as a phase initialization scheme for oMISI (i.e., at line 4 in Algorithm 1). The frequencies ν are estimated from the magnitude spectra using quadratic interpolation around each magnitude peak [START_REF] Magron | Model-based STFT phase recovery for audio source separation[END_REF], [START_REF] Abe | Design criteria for simple sinusoidal parameter estimation based on quadratic interpolation of FFT magnitude peaks[END_REF].

ϕ j,t (f ) = ϕ j,t-1 (f ) + 2πlν j,t (f ), (26)

V. EXPERIMENTAL EVALUATION

A. Dataset and protocol

For evaluation, we consider a single-channel speech separation task. We use the Danish hearing in noise test dataset [START_REF] Nielsen | The Danish hearing in noise test[END_REF]. We consider three speaker pairs denoted MF, MM and FF, where M and F stand for male and female respectively, in order to cover all gender combinations. All audio files were recorded with a sampling rate of 44.1 kHz, and down-sampled at 16 kHz in our experiments. The STFT is computed using a 16 ms long Hann window, 50 and a zero-padding factor of 2. The synthesis window is defined as in [START_REF] Griffin | Signal estimation from modified short-time Fourier transform[END_REF], so that the STFT is Hermitian, as discussed in Section III-C.

Two scenarios are considered. The Oracle scenario uses the ground truth magnitude spectra of the sources. In the Estim. scenario, they are estimated using the DNN described in [START_REF] Naithani | Low latency sound source separation using convolutional recurrent neural networks[END_REF] (where the interested reader can find details on the DNN architecture and training). This DNN predicts a soft mask that is applied to the mixture to yield magnitude estimates. We compare oMISI to its offline counterpart and to the amplitude mask (AM) used as a baseline [START_REF] Naithani | Low latency sound source separation using convolutional recurrent neural networks[END_REF].

The separation quality is measured with the scale-invariant signal-to-distortion ratio improvement (SI-SDRi) [START_REF] Roux | SDR half-baked or well done?[END_REF]. We provide some audio excerpts1 for a subjective evaluation, and the code for reproducing the Oracle scenario experiments2 .

B. Results

We present in Fig. 2 the objective loss (3) and SI-SDRi over iterations for the MISI algorithm using the MF pair in the Estim. setting (similar results are obtained using the other speaker combinations). We observe a non-increasing cost function over iterations, which empirically confirms the convergence of MISI. The SI-SDRi appears to saturate at 15 iterations, thus we present hereafter the results obtained with this value. Since with oMISI, each frame is processed (K + 1) times more as in MISI, we reduce the number of iterations to 15/(K + 1) for a fair comparison. Finally, note that the SI-SDRi can further increase in the Oracle setting.

The separation results are reported in Table I. We first remark that MISI improves the performance over the baseline by approximately 0.4 dB in the Estim. scenario. In the Oracle scenario, the improvement is more significant (≈ 15 dB), which highlights the room for improvement for phase recovery. Besides, in the Estim. scenario, we observe that oMISI performs as well as MISI with K = 1. The performance of oMISI drops slightly for K = 0, which was expected as no future frame is taken into account: nonetheless, it is still improved compared to AM, and the drop in comparison to the offline method is quite small. Finally, the performance does not further improve for K = 2 in the Estim. scenario. These results demonstrate the potential of oMISI for real-time applications.

Using one future frame then appears as a good compromise between latency and performance, thus we test the initialization with the sinusoidal phase model with K = 1. However, this scheme does not improve the performance of oMISI over using the mixture's phase overall. We suggest that the speakers in the MM and MF pairs are sufficiently orthogonal (i.e., less overlapping) in the TF domain, thus the mixture's phase is a good quality initial estimate. Nonetheless, the Estim. results highlight that the FF pair is the most challenging, and the corresponding Oracle results indicate that this initialization scheme has some potential, provided accurate enough magnitude estimates.

VI. CONCLUSION

In this paper, we provided the first proof of convergence of the MISI algorithm. We adapted it to operate online without any performance loss, which is an important step towards real-time audio source separation. Future work will focus on deriving alternative spectrogram inversion algorithms based on this theoretical framework, e.g., by replacing the magnitude mismatch distance by a β-divergence, which is more adapted to audio. This algorithm will also be incorporated into an endto-end framework for time-domain source separation along with learned and more advanced [START_REF] Průša | Real-time spectrogram inversion using phase gradient heap integration[END_REF] phase models.

VII. ACKNOWLEDGEMENTS

•

  (.): real part function. • ||.||: Euclidean norm. • , .

Fig. 1 .

 1 Fig. 1. Illustration of the proposed approach in one time frame with K = 0. Algorithm 1: Online MISI 1 Inputs: Mixture X ∈ C F ×T , magnitudes V j ∈ C F ×T , number of iterations N i and future frames K. 2 Initialize the past segments: ∀j, spast j = 0 3 for t = 0 to T -1 -K do4

Fig. 2 .

 2 Fig. 2. Error (3) and SI-SDRi over iterations in the Estim. setting.

TABLE I AVERAGE

 I SI-SDRI IN DB (HIGHER IS BETTER). BOLD FONTS CORRESPOND TO THE BEST PERFORMANCE AMONG ONLINE TECHNIQUES.

			MF			MM	FF
		Latency	Estim. Oracle Estim. Oracle Estim. Oracle
	AM	16 ms	7.5	8.8	5.7	7.3	5.1	7.5
	MISI	offline	7.9	23.8	6.2	22.3	5.4	22.9
	oMISI
	mix	16 ms (K=0)	7.7	16.4	6.1	15.8	5.4	16.9
	mix	24 ms (K=1)	7.9	20.2	6.2	19.4	5.4	19.6
	mix	32 ms (K=2)	7.9	21.4	6.2	20.4	5.4	20.6
	sin	24 ms (K=1)	7.8	15.2	6.2	14.6	5.4	20.7

https://magronp.github.io/demos/spl20 omisi.html

https://github.com/magronp/omisi

We thank Roland Badeau for his insight on optimization, and Gaurav Naithani for providing the magnitude estimates.