Jort Florent Gemmeke
email: j.gemmeke@let.ru.nl

Bert Cranen
email: b.cranen@let.ru.nl

Ulpu Remes
email: ulpu.remes@tkk.fi

Sparse imputation for large vocabulary noise robust ASR

Keywords: missing data techniques, noise robustness, automatic speech

An effective way to increase noise robustness in automatic speech recognition is to label the noisy speech features as either reliable or unreliable ('missing'), and replace ('impute') the missing ones by clean speech estimates. Conventional imputation techniques employ parametric models and impute the missing features on a frame-by-frame basis. At low SNR's, frame-based imputation techniques fail because many time frames contain few, if any, reliable features. In previous work, we introduced an exemplar-based method, dubbed sparse imputation, which can impute missing features using reliable features from neighbouring frames. We achieved substantial gains in performance at low SNR's for a connected digit recognition task. In this work, we investigate whether the exemplar-based approach can be generalised to a large vocabulary task.

Experiments on artificially corrupted speech show that sparse imputation substantially outperforms a conventional imputation technique when the ideal 'oracle' reliability of features is used. With error-prone estimates of feature reliability, sparse imputation performance is comparable to our baseline imputation technique in the cleanest conditions, and substantially better at lower SNR's. With noisy speech recorded in realistic noise conditions, sparse imputation performs slightly worse than our baseline imputation technique in the cleanest conditions, but substantially better in the noisier conditions.

Automatic speech recognition (ASR) performance drops rapidly when speech is corrupted with increasing levels of unfamiliar background noise (i.e., noise not seen during training) since the observed acoustic features no longer match the acoustic models. Although in real-world environments, speech is often corrupted by several unknown and time-varying noise sources, few techniques other than multi-condition training have been proposed to enhance robustness towards non-stationary noise. Missing Data Techniques (MDT) [START_REF] Raj | Missing-feature approaches in speech recognition[END_REF] are among the most promising alternative proposals.

MDT, first proposed in [START_REF] Cooke | Handling missing data in speech recognition[END_REF], build on the assumption that one can estimateprior to decoding-which spectro-temporal elements in the acoustic representation of noisy speech are reliable (i.e., dominated by speech) and which are unreliable (i.e., dominated by background noise). In the unreliable elements, the clean speech information is considered missing, and the challenge is then to do speech recognition with partially observed data. In this work, we focus on the so-called imputation approach [START_REF] Raj | Inference of missing spectrographic features for robust automatic speech recognition[END_REF] which handles the missing elements by replacing them with clean speech estimates. Classic imputation methods include e.g. correlation and cluster-based reconstruction [START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF][START_REF] Raj | Missing-feature approaches in speech recognition[END_REF] and methods for reconstruction in the cepstral and PROSPECT domains [START_REF] Van Hamme | PROSPECT features and their application to missing data techniques for robust speech recognition[END_REF], while the state-based imputation method proposed in [START_REF] Josifovski | State based imputation of missing data for robust speech recognition and speech enhancement[END_REF] combines front-end imputation and classifier modification.

Imputation has been proven an effective technique in both small and large vocabulary tasks [START_REF] Segbroeck | Robust large vocabulary continuous speech recognition using missing data techniques[END_REF], performing better than conventional feature-enhancement techniques (cf. [START_REF] Van Hamme | Robust speech recognition using cepstral domain missing data techniques and noisy masks[END_REF][START_REF] Stouten | Robust automatic speech recognition in time-varying environments[END_REF]). However, a number of issues concerning the applicability of imputation methods in different ASR tasks remain under-investigated. First, since most work on MDT has been done on artificially constructed databases (see e.g. [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF][START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF][START_REF] Segbroeck | Robust large vocabulary continuous speech recognition using missing data techniques[END_REF]), the potentials and limitations of the missing data approach in real-world environments are not well known. Using artificially corrupted data is attractive as it allows estimating which features are reliable based on exact knowledge of the speech and noise power in each timefrequency cell. This facilitates comparison of different MDT approaches and allows for analysis of the influence of errors in reliability estimation. The results from such experiments are not, however, truly indicative for real-world conditions where the observed signal is rarely a simple addition of clean 2 A c c e p t e d M a n u s c r i p t speech and noise: in many cases, channel effects, the Lombard effect, and room reverberation also affect the observations.

Another issue is the imputation performance when the signal-to-noise ratio (SNR) is low and a substantial number of frames contains few, if any, reliable features. In previous work [START_REF] Gemmeke | Using sparse representations for missing data imputation in noise robust speech recognition[END_REF], we suggested that the observed performance loss when using conventional imputation methods at low SNR's is at least partly due to the fact that these methods work on a frame-by-frame basis. We argued that taking into account the time-context and utilising reliable features from neighbouring frames could reduce the number of imputation errors significantly. The use of time context for imputation has been explored in various other studies [START_REF] Raj | Reconstruction of incomplete spectrograms for robust speech recognition[END_REF][START_REF] Tan | Exploiting temporal correlation of speech for error-robust and bandwidth-flexible distributed speech recognition[END_REF]. Our approach to harnessing the information in neighbouring frames is by using a novel, non-parametric imputation method, sparse imputation (SI). We showed that the use of SI results in large performance gains and allows for successful missing data imputation at lower SNR's provided that the locations of the reliable time-frequency cells are estimated accurately [START_REF] Gemmeke | Compressive sensing for missing data imputation in noise robust speech recognition[END_REF].

The key concept in sparse imputation is that any speech fragment can be represented as a linear combination of a small number of example speech tokens. First, a dictionary of exemplars is constructed using fixed-length clean speech tokens. Then, a sparse linear combination of exemplars is sought using only the reliable speech features. Imputation of the unreliable features is accomplished by replacing them with the corresponding features of the linear combination of clean speech dictionary exemplars. Initially, we illustrated SI on an isolated digit recognition task where each fixed-length exemplar in the dictionary corresponded to a complete word [START_REF] Gemmeke | Using sparse representations for missing data imputation in noise robust speech recognition[END_REF]. In [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF], we successfully adapted the technique for continuous digits by using a sliding window approach and a dictionary that consists of randomly selected, fixed-length segments of clean speech. During imputation, the reliable features of each window of the speech signal are treated as a sparse linear combination of clean speech windows in the dictionary. At every instant in time, the final estimates for each spectral feature vector are then calculated as an average over the reconstructions in overlapping windows.

In [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF], the sparse imputation approach for continuous digits was evaluated on the aurora-2 digit recognition task [START_REF] Hirsch | The Aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions[END_REF] which is frequently used in noise robust ASR experiments. However, it is well known that results for small vocabulary tasks are difficult to generalise to large vocabulary continuous speech recognition. One reason for this is that speech sounds occur in a much larger number of different contexts in large vocabulary tasks, which A c c e p t e d M a n u s c r i p t might make it more difficult to model speech as a sparse combination of a small number of examples. This problem will only become more serious if the number of context frames in the SI approach is increased. In this paper, we will investigate to what extent the increased number of reliable features that comes from using multiple time-frames, in combination with the natural coherence of speech signals, will result in performance gains at low SNR's, despite the potential loss in accuracy due to increased variation.

In this work, we apply the sliding window approach for sparse imputation proposed in [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF] on large vocabulary continuous speech data from the Finnish speecon database [START_REF] Iskra | SPEECON -speech databases for consumer devices: Database specification and validation[END_REF]. The data used in the experiments are either the original speecon data recorded in real-world noisy environments or artificially constructed from mixing clean speech speecon data and noise from the noisex-92 database [START_REF] Varga | Assessment for automatic speech recognition: II. NOISEX-92: a database and an experiment to study the effect of additive noise on speech recognition systems[END_REF]. By experimenting on different window sizes and noise types, we will investigate to what extent using more time-context can improve recognition accuracy. We will compare the results obtained with sparse imputation to results obtained with a standard frame-based parametric method, cluster-based imputation [START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF][START_REF] Raj | Missing-feature approaches in speech recognition[END_REF], which has been shown to work well for the speecon database [START_REF] Remes | Missing feature reconstruction and acoustic model adaptation combined for large vocabulary continuous speech recognition[END_REF].

The rest of the paper is organised as follows. In Section 2, we discuss Missing Data Techniques for ASR and introduce the two types of reliability estimates and missing data masks used in this work. In Section 3, we briefly describe the baseline cluster-based imputation method. In Section 4, we describe the sparse imputation approach and discuss the generalisation to imputing large vocabulary speech by using a sliding time window. In Section 5, we present the experimental setup, while the results appear in Section 6 and are discussed in Section 7. Conclusions and suggestions for future research are given in Section 8.

Missing Data Techniques in ASR

Motivation

In this section, we briefly discuss the MDT framework as used for noise robust ASR [START_REF] Junqua | Robustness in Automatic Speech Recognition: Fundamentals and Applications[END_REF][START_REF] Barker | Decoding speech in the presence of other sources[END_REF]. In ASR, the basic representation of speech is a spectrotemporal distribution of acoustic power, a spectrogram. In noise-free conditions, the value of each time-frequency cell in this two-dimensional matrix is determined only by the speech signal. In noisy conditions, the value in each cell represents a combination of speech and background noise power.

A c c e p t e d M a n u s c r i p t

Assuming noise is additive, the power spectrogram of noisy speech can be approximately described as the sum of the individual power spectrograms of clean speech and noise. To mimic human hearing, often a Mel-frequency scale and logarithmic compression of the power scale are employed. Since the logarithmic compression of a sum can be approximated by the logarithm of the largest of the two terms [START_REF] Nadas | Speech recognition using noiseadaptive prototypes[END_REF], it approximately holds for noisy speech features that:

Y ≈ max(S, N ) (1)
with the (Mel-frequency) log-power spectrograms Y denoting noisy speech, S denoting clean speech and N representing the background noise. From [START_REF] Raj | Missing-feature approaches in speech recognition[END_REF] we can infer that the noisy speech features dominated by clean speech energy remain approximately uncorrupted and can be used directly as estimates of the clean speech features. The noise dominated features, on the other hand, provide only an upper bound for the clean speech, which means the clean speech features cannot be observed and are effectively missing.

Missing data masks

Elements of Y that predominantly contain speech or noise energy are distinguished by introducing a spectrographic mask M . The elements of a mask M are either 1, meaning that the corresponding element of Y is dominated by speech ('reliable') or 0, meaning that it is dominated by noise ('unreliable' c.q. 'missing'). Thus, we write:

M (k, t) = 1 def = reliable if S(k, t) -N (k, t) > θ 0 def = unreliable otherwise (2)
with M , Y , S, and N two-dimensional matrices of K × T , with frequencyband index k, 1 ≤ k ≤ K and time-frame index t, 1 ≤ t ≤ T . θ denotes a constant SNR-threshold. Smaller values of θ will result in more elements being considered as reliable in the mask but the proportion of errors implied in the assumption that S(k, t) = Y (k, t) will be larger. Larger values of θ lead to a safer model but also to fewer reliable elements for estimating the missing values.

Estimating missing data masks

In experiments with artificially added noise, an oracle mask can be computed directly with (2) using knowledge of the corrupting noise and the clean A c c e p t e d M a n u s c r i p t speech signal. The oracle mask is useful to assess the potential of missing data imputation techniques and to compare the performances of different techniques in ideal conditions.

In realistic situations, however, the masks must be estimated from the noisy speech. Many different estimation techniques have been proposed, such as SNR based estimators [START_REF] Vizinho | Missing data theory, spectral subtraction and signal-to-noise estimation for robust ASR: An integrated study[END_REF][START_REF] Remes | Missing feature reconstruction and acoustic model adaptation combined for large vocabulary continuous speech recognition[END_REF], machine learning approaches to mask estimation [START_REF] Seltzer | A Bayesian classifier for spectrographic mask estimation for missing feature speech recognition[END_REF][START_REF] Kim | Band-independent mask estimation for missingfeature reconstruction in the presence of unknown background noise[END_REF][START_REF] Weiss | Estimating single-channel source separation masks: Relevance Vector Machine classifiers vs. pitch-based masking[END_REF], methods that focus on speech characteristics, e.g. harmonicity based SNR estimation [START_REF] Van Hamme | Robust speech recognition using cepstral domain missing data techniques and noisy masks[END_REF][START_REF] Van Segbroeck | Vector-Quantization based mask estimation for missing data automatic speech recognition[END_REF], and mask estimation exploiting binaural cues [START_REF] Harding | Mask estimation for missing data speech recognition based on statistics of binaural interaction[END_REF] or correlogram structure [START_REF] Ma | Exploiting correlogram structure for robust speech recognition with multiple speech sources[END_REF] (cf. [START_REF] Cerisara | On noise masking for automatic missing data speech recognition: A survey and discussion[END_REF] and the references therein for a more complete overview of mask estimation techniques).

In this work, we use the mask estimation approach described in [START_REF] Remes | Missing feature reconstruction and acoustic model adaptation combined for large vocabulary continuous speech recognition[END_REF]. Local SNR's are obtained from comparing the noisy speech to a noise estimate which is calculated based on frames identified as non-speech by a speech/nonspeech classifier. Implementation details are given in Section 5.4. Since imputation accuracy is dependent on the quality of the missing data mask, we investigate the influence of mask estimation errors by additionally using oracle masks when the test set contains speech artificially corrupted with background noise.

Use of MDT in ASR

Techniques for speech recognition with missing data can be divided in two categories: marginalisation and imputation. In the marginalisation approach [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF]31], acoustic likelihoods are calculated by integrating over the range of possible values of the missing features and recognition is carried out primarily based on the reliable features. In the imputation approach [START_REF] Raj | Reconstruction of incomplete spectrograms for robust speech recognition[END_REF][START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF], the missing features are replaced by clean speech estimates, after which recognition can proceed without modification of the recognition system.

The marginalisation approach has been shown to be more robust against data sparsity at low SNR's than the traditional imputation methods [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF]. Imputation methods are, however, attractive for two reasons. First, after the missing features have been replaced with clean speech estimates, any recogniser developed for clean speech can be deployed without further modifications. Another benefit is that the reconstructed features can be converted to an arbitrary domain, like the cepstral domain. This is advantageous since cepstral features are known to be less correlated and better suited for processing with the state-of-the-art HMM-based ASR techniques [32]. Therefore, in this work, only imputation techniques are investigated.

A c c e p t e d M a n u s c r i p t

Cluster-based imputation

As mentioned in the introduction, we use the cluster-based imputation method proposed in [START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF] as the baseline approach for missing data speech recognition. It is a frame-based method where the unreliable feature values are estimated based on information in the observed features and a parametric clean speech model.

Modelling assumptions

The clean speech distribution model used in cluster-based imputation [START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF][START_REF] Raj | Missing-feature approaches in speech recognition[END_REF] assumes the clean speech vectors s(t) are independent and identically distributed (i.i.d.). Thus, the model will capture the statistical dependencies between spectral channels but not between time-frames. It is also assumed the clean speech data can be clustered so that the features in each cluster are approximately normally distributed, and the clean speech can be modelled using a Gaussian mixture model (GMM):

P (s(t)) = ν P (z(t) = ν)N [s(t); µ(ν), Σ(ν)] ∀ t, (3)
where ν are the cluster indices, z(t) indicates the current cluster, P (z(t) = ν) are the cluster weights i.e. prior probabilities for z(t), and µ(ν) are the cluster means and Σ(ν) the covariance matrices. Here, the cluster identities z(t) underlying s(t) are assumed unknown and modelled as a latent variable. In this work, the clusters and the distribution parameters µ(ν) and Σ(ν) were jointly estimated from a clean speech training corpus using the expectationmaximisation (EM) algorithm.

Missing data imputation

The noisy observations y(t) corresponding to individual frames of the spectrogram Y are divided into mutually exclusive reliable and unreliable regions y r (t) and y u (t) as indicated by the missing data mask (2). In clusterbased imputation [START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF][START_REF] Raj | Missing-feature approaches in speech recognition[END_REF], the clean speech estimates or reconstructions for the unreliable features are chosen so that 1) the reconstructed vectors ŝ(t) = s r (t) ∪ ŝu (t) are similar to clean speech i.e. provide the best possible fit with the clean speech distribution model while 2) the reconstructed values ŝu (t) are constrained not to exceed the observed values y u (t). Finding such reconstruction can be written as a bounded maximum a posteriori (BMAP)

A c c e p t e d M a n u s c r i p t estimation task, where the BMAP estimator for the unreliable features is given as ŝu = argmax

su∈IR U {P (s u |s r , s u ≤ y u , Λ)}, (4)
where Λ are the parameters estimated for the GMM (3) and U is the number of unreliable features in y(t). We dropped the explicit notation to indicate the dependency on t, i.e., s = s(t) and y = y(t). Note that the model contains two unknown variables: in addition to the unreliable feature values, the cluster identities z = z(t) are unknown. In (4), the latent variable z has been marginalised, but the dependency can be made explicit and (4) written as ŝu = argmax

su∈IR U { ν P (z = ν|s r , s u ≤ y u , Λ)P (s u |s r , s u ≤ y u , Λ, ν)}, (5)
where the first probability term is the posterior probability for the ν-th GMM cluster given the reliable features y r (t) and the upper bound given by the unreliable features y u (t), and the second is the cluster-conditional posterior probability for the unreliable features.

In practice, finding maximum a posteriori estimates for GMM-distributed variables is difficult. Therefore, in cluster-based imputation [START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF][START_REF] Raj | Missing-feature approaches in speech recognition[END_REF], ( 5) is approximated as

ŝu = ν P (z = ν|s r , s u ≤ y u , Λ) argmax su∈IR U {P (s u |s r , s u ≤ y u , Λ, ν)}, (6)
where the latter term is the cluster-conditional BMAP estimate for the unreliable features s u . The cluster-conditional estimate for s u is weighted with the posterior probability for cluster ν which is calculated based on the prior probability P (z(t) = ν) and cluster-conditional observation probability. In this work, we use full covariance matrices Σ(ν) and calculate the clusterconditional BMAP estimates iteratively over the frequency channels k as proposed in [START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF][START_REF] Raj | Missing-feature approaches in speech recognition[END_REF]. The covariance matrices are only assumed diagonal when evaluating the posterior probabilities for z(t) = ν.

Sparse Imputation

In sparse imputation, speech tokens are represented as a linear combination of tokens from an overcomplete dictionary of noise-free exemplars 8 A c c e p t e d M a n u s c r i p t represented as fixed-length vectors. For an unknown speech token, a sparse linear combination is sought in the dictionary using all reliable features in the entire token. Imputation of the unreliable features is then accomplished by replacing them with the corresponding values from this linear combination of the clean speech dictionary exemplars. In [START_REF] Gemmeke | Using sparse representations for missing data imputation in noise robust speech recognition[END_REF], the tokens were chosen to constitute time-normalised complete words, but for the continuous large vocabulary speech used in this work, we must apply the sliding window approach proposed in [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF].

Sparse representation of speech

The log-power spectrogram of clean speech, S, is reshaped to a single vector s of dimension D = K • T by concatenating T subsequent K-dimensional time frames. For now, we assume that T is fixed. Inspired by a similar approach in the field of face recognition [START_REF] Wright | Robust face recognition via sparse representation[END_REF], we assume that s can be represented exactly (or at least approximated with sufficient accuracy) by a linear, non-negative, combination of exemplar spectrograms a n , where n denotes a specific exemplar (1 ≤ n ≤ N ) in the dictionary of N available exemplars:

s = N n=1 x n a n = Ax subject to x ≥ 0 (7)
with x an N -dimensional activation vector. The matrix A denotes an overcomplete dictionary: A = [a 1 a 2 . . . a N ], with dimensions D × N with N ≫ D. A schematic representation of this process for a set of non-noisy spoken digits (1 through 9, "zero", and "oh") is displayed in Fig. 1 A.

Although the system of linear equations in [START_REF] Segbroeck | Robust large vocabulary continuous speech recognition using missing data techniques[END_REF] has no unique solution, research in the field of Compressive Sensing [START_REF] Donoho | Compressed sensing[END_REF][START_REF] Donoho | For most large underdetermined systems of linear equations the minimal L1-norm solution is also the sparsest solution[END_REF][START_REF] Candès | Stable signal recovery from incomplete and inaccurate measurements[END_REF] has shown that under mild conditions on A, the activation vector x can be uniquely determined if x is sufficiently sparse. This is accomplished by solving:

x = argmin x∈IR N { Ax -s 2 + λ x 1 } (8)
with a regularisation parameter λ. The requirement that the linear combination must be sparse means that it must be possible to represent speech tokens with a small number of exemplars, resulting in a small number of nonzero values in x. For spoken digits, it was shown in [START_REF] Gemmeke | Compressive sensing for missing data imputation in noise robust speech recognition[END_REF] that the representation is indeed sparse. A c c e p t e d M a n u s c r i p t

Missing data imputation

If the data contains missing values, we begin by concatenating subsequent time frames of the spectrographic mask M discussed in Section 2.2 to form a mask vector m; this is done similarly as described for s in the previous section. Using the same approach for the noisy speech spectrogram Y we construct a noisy observation vector y. The elements of y corresponding to elements of the mask vector m that are equal to 1 are the reliable coefficients y r . We use the reliable elements y r as an approximation for the corresponding elements of s, so problem (8) becomes:

x = argmin x∈IR N { A r x -y r 2 + λ x 1 } (9)
with A r pertaining to the rows of A for which m = 1. The sparse representation x obtained by solving problem ( 9) could be directly used to estimate the clean observation vector as ŝ = Ax. A schematic representation of this process for a set of pre-segmented spoken digits (1 through 9, "zero", and "oh") is displayed in Fig. 1 B.

In practice, the sparse representation is not directly used as a clean speech estimate since the reconstruction error for the reliable coefficients will generally be non-zero if we solve problem [START_REF] Stouten | Robust automatic speech recognition in time-varying environments[END_REF], so it is better to only impute the unreliable elements. Furthermore, under the assumption that noise and speech are additive in the power domain, the observed noisy speech y is an upper limit for ŝ. Incorporating these two modifications we obtain:

ŝ = ŝr = y r ŝu = min (A u x, y u ) (10)
with A u and ŝu pertaining to the rows of A and ŝ for which m = 0 and with the min-operator taking the element-wise minimum of two values. A version of ŝ that is reshaped into a K × T matrix Ŝ can be considered a denoised spectrogram representing the underlying speech signal, and as illustrated in Fig. 1 B, it can be directly used in speech recognition.

Sliding window approach

The approach described above is suitable for imputation of noisy speech tokens that can be adequately represented by a fixed number of time frames T [START_REF] Gemmeke | Using sparse representations for missing data imputation in noise robust speech recognition[END_REF]. Since arbitrary length utterances clearly do not satisfy this constraint, we adopt a sliding window approach introduced in [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF]. In this approach, A c c e p t e d M a n u s c r i p t each window is imputed separately using sparse imputation as described in Section 4.2. Subsequently, at every time frame, the different clean speech estimates resulting from any overlapping windows are combined.

Consider a noisy speech utterance Y tot represented as a spectrogram with K frequency bands and T tot time-frames. The goal of the missing data imputation process is to provide an estimate Ŝtot of the underlying clean speech S tot using a missing data mask M tot .

We slide a window of length T w through Y tot , with shifts of ∆,1 ≤ ∆ ≤ T w frames (cf. Fig. 2). Y w and M w describe the noisy speech and associated missing data mask for each window w, 1 ≤ w ≤ W . The ratio of ∆ and T w determines the degree with which subsequent windows overlap. Larger step sizes ∆ reduce computational effort, but can decrease imputation accuracy [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF]. Throughout this paper, we keep the window shift constant at ∆ = 1 frame.The total number of windows we process is W = T tot -T w + 1.

We then use, for each window, the sparse imputation approach described in Section 4.2 to provide a clean speech estimate Ŝw of the underlying clean speech S w . Since windows overlap, each frame in Y is associated with multiple clean speech estimate candidates, with the number of candidates ranging from 1 (at the beginning and end of an utterance) to T w . For each frame, the feature values of the final clean speech estimate Ŝtot are created by averaging over the multiple clean speech estimate candidates pertaining to that frame (cf. Fig. 2). The clean speech estimate is calculated using only clean speech estimates derived from windows with a nonzero number of reliable elements.

However, in very noisy conditions and particularly at the start and end of an utterance, it may happen that many adjacent windows do not contain reliable features, leaving the method unable to provide a clean speech estimate from averaging. Yet, despite the lack of information about the underlying signal, input must be provided to the ASR engine. Thus, we opted to impute silence (the average feature values per frequency band for silence states as observed in a training database) for such frames.

Experimental setup

Speech recognition system and performance evaluation

The speech recognition system used in this work is the large vocabulary continuous speech recognition system developed in the Adaptive Informatics Research Centre at the Aalto University School of Science and Technology. The acoustic models are trained with 30-hours of clean speech recorded with a A c c e p t e d M a n u s c r i p t headset in quiet conditions and selected from the Finnish speecon database (recorded with a 16 kHz sampling rate) [START_REF] Iskra | SPEECON -speech databases for consumer devices: Database specification and validation[END_REF]. The training set comprises 293 speakers (142 female and 151 male). The utterances used for training contain words, read sentences and spontaneous speech in order to have a general acoustic model valid for multiple tasks.

The decoder used in the system is a time-synchronous beam-pruned Viterbi token-pass system described in [START_REF] Pylkkönen | An efficient one-pass decoder for Finnish large vocabulary continuous speech recognition[END_REF] and the acoustic models are state-clustered, hidden Markov triphone models, constructed with a decision-tree method [START_REF] Odell | The use of context in large vocabulary speech recognition[END_REF]. There are acoustic models for 13 250 triphones and two silences. The triphones are modelled as left-right HMM with three states and silences with one state each. In total we used 1564 individual states modelled with approximately 28 Gaussians per state. Each state is also associated with gamma probability functions to model the state durations [START_REF] Pylkkönen | Duration modeling techniques for continuous speech recognition[END_REF].

The language model employs morpheme-like subword units, called statistical morphs, discovered in an unsupervised, data-driven manner [START_REF] Creutz | Unsupervised discovery of morphemes[END_REF]. These are used because word-based modelling is not feasible for highly inflected languages such as Finnish, Estonian, or Turkish. The statistical morph lexicon with 25k morpheme-like units was learned from the 160k most common words extracted from 145 million words of Finnish book and newspaper data [START_REF]The language bank of Finland[END_REF]. The variable-length, growing n-gram language model [START_REF] Siivola | Growing an n-gram language model[END_REF] used in this work was trained on the same text corpus and contains 52 million n-grams. The decoding vocabulary is in practice unlimited since all words and word forms can be represented using the statistical morphs [START_REF] Hirsimäki | Unlimited vocabulary speech recognition with morph language models applied to Finnish[END_REF].

Finally, in this work, the speech recognition performance is measured primarily in letter error rates (LER). This is because the words in Finnish are often long and consist of several morphemes so that measuring the word error rate (WER) would correspond better to measuring sentence or phrase error rates in languages such as English. Using the word error rate is also considered to over-penalise misrecognised word breaks.

Recognition task

The imputation methods are evaluated with clean speech recordings artificially corrupted with noise at different SNR's as well as with speech recorded in real-world noisy environments. In both conditions, the speech data consists of read sentences selected from the Finnish speecon database [START_REF] Iskra | SPEECON -speech databases for consumer devices: Database specification and validation[END_REF]. The artificially corrupted clean speech was constructed by mixing headset-recorded clean speech utterances with a randomly selected sample of the babble noise from the noisex-92 database [START_REF] Varga | Assessment for automatic speech recognition: II. NOISEX-92: a database and an experiment to study the effect of additive noise on speech recognition systems[END_REF] at SNR's 15 dB, 10 dB, 5 dB and 0 dB.

A c c e p t e d M a n u s c r i p t

The real-world noisy speech data is recorded in two types of environments: in a car and in public places both indoors and outdoors. These recordings have been made with three microphones: 1) with a headset, 2) with a lavalier microphone, and 3) with a microphone from 0.5 m-1 m distance (in public environments) or with a microphone mounted on the rear-view mirror (in car environments).

In the speecon documentation, the average SNR's in the public environments are estimated to be 24 dB for the headset microphone data, 14 dB for the lavalier microphone data, and 9 dB for the far-field microphone data. For the car recordings, the estimated average SNR's are 13 dB for the headset microphone data, 5 dB for the lavalier microphone data, and 8 dB for the rear-view mirror (RVM) microphone data. The RVM microphone data has a higher SNR than the lavalier microphone data because the RVM microphone (AKG Q400 Mk3T) has a limited frequency response, specifically designed for in-car use and suppressing low frequency noise.

The speech material in each of the three scenarios (public environments, car environments, and artificially added babble noise) is divided in development and evaluation sets. The composition of the sets in terms of number of utterances (#u), duration (d), number of speakers (#s), number of female speakers (#f) and number of male speakers (#m) is shown in Table 1. None of the sets share speakers with each other or with the speech data used for training the speech recognition system.

Feature extraction

Feature extraction was carried out using a 16 ms Hamming window with 8 ms overlap between subsequent windows. First-order pre-emphasis was applied to the signal using a coefficient of 0.97. After Fourier transformation, the log-power was computed in 21 triangular-shaped Mel-frequency bands. Imputation was carried out on these log-power Mel-frequency spectra to obtain clean speech estimates.

A c c e p t e d M a n u s c r i p t

After imputation, the resulting spectra were transformed to 12 Melfrequency cepstral coefficients (MFCC) and a log-energy feature, augmented with first and second-order time derivatives for a total of 39 features per time frame. Channel normalisation was applied using cepstral mean subtraction, and as a final step, a maximum likelihood linear transformation (MLLT). The MLLT, optimised during training of the acoustic models, is applied on the normalised features to improve the modelling of any remaining correlation in the normalised MFCCs as proposed in [START_REF] Gales | Semi-tied covariance matrices for hidden Markov models[END_REF].

Missing data mask estimation

In this work, the missing data masks are constructed based on local SNR estimates obtained from comparing the noisy speech to a static noise estimate calculated during speech pauses. These speech pauses were detected using an HMM-based speech/non-speech classifier described in [START_REF] Remes | Missing feature reconstruction and acoustic model adaptation combined for large vocabulary continuous speech recognition[END_REF]. Additionally, we used the MATLAB command bwareaopen to discard small, isolated regions of reliable features from the estimated mask since it was suggested in [START_REF] Cooke | A glimpsing model of speech perception in noise[END_REF] that such glimpses are not detectable to human listeners and are therefore unlikely to contain usable information. Experiments on the speecon development data also confirmed that removing glimpses comprising less than five spectro-temporal components improves speech recognition results. In experiments with artificially corrupted speech, we also computed oracle masks (Section 2.3) from which glimpses were not removed.

The SNR threshold θ for deciding whether a time-frequency component is treated as reliable or unreliable was determined by maximising recognition accuracy on the development sets described in Section 5.2. For both realworld noisy speech sets, the optimum value for estimated masks was at θ = 3 dB for both imputation methods. For artificially corrupted speech, we determined an SNR-independent threshold using the development data sets containing noise at 10 and 5 dB SNR. The optimum mask threshold value for the estimated masks was at θ = 4 dB for both imputation methods and for the oracle masks at θ = -2 dB for sparse imputation and θ = -1 dB for cluster-based imputation.

Cluster-based imputation

The clean speech model used in this work is a 5-component GMM trained using a 52-minute dataset of 500 read sentences randomly selected from the speecon training data described in Section 5.1. The clusters and distribution parameters are jointly estimated using the expectation-maximisation Page 17 of 36 A c c e p t e d M a n u s c r i p t (EM) algorithm implemented in the GMMBayes toolbox1 . The cluster-conditional bounded maximum a posteriori (BMAP) estimates are calculated in an iterative manner as described in [START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF] using a MATLAB implementation. The estimates are calculated in 10 iterations over the mel-frequency bands.

The number of 5 clusters was selected as a reasonable balance between imputation performance and computational complexity. Non-exhaustive tests on the development data showed that while more clusters do improve recognition accuracy, the increase is slight compared to the extra computational effort required.

Sparse imputation

The sparse imputation was implemented in MATLAB. The l 1 minimisation (9) was carried out using the SolveLasso solver. 2 Due to instability issues, the slower, but more robust l1 ls nonneg solver3 [START_REF] Kim | An interior-point method for large-scale l1-regularized least squares[END_REF] was used whenever the (fast) solver SolveLasso appeared to crash. When using the l1 ls nonneg solver, the regularisation parameter λ was determined using the utility function find lambdamax l1 ls nonneg. The stopping criterion of the l1 ls nonneg solver was a duality gap of 0.01. The SolveLasso solver was run for 30 iterations.

For each window size T w ∈ {1, 5, 10, 15, 20, 25, 30, 35} being considered, an initial dictionary is created by randomly extracting 4 spectrogram segments of the desired duration T w from each of the 8139 read sentences (containing 14 hours of speech) in the speecon training data described in Section 5.1. From this initial dictionary spanning 32 556 exemplars, we then randomly extract 8000 exemplars to form the final dictionary used for imputation. The dictionary size of 8000 exemplars was chosen because pilot tests showed that while using larger dictionaries improves recognition accuracy, the increase is slight enough to consider it a reasonable balance between recognition performance and computational complexity. In exemplar selection, no effort was made to balance a possible over-representation of spectra containing silence.

After reshaping the spectrograms to one-dimensional vectors as described in Section 4.1, the feature vectors form the columns of the dictionary matrix. The zero-dB level in the spectra is set such that the lowest occurring feature A c c e p t e d M a n u s c r i p t value in the dictionary is zero. Finally, the columns of the dictionary are normalised to Euclidean unit norm.

Results

Public and car environment data

The speech recognition results from our experiments with the data recorded in the public and car real-world environments are displayed in Fig. 3. The results depict the letter error rate (LER) of the sparse imputation (SI) method as a function of window size (T w ∈ {1, 5, 10, 15, 20, 25, 30, 35}). The figure also shows the performance of the cluster-based imputation (CI) method and the baseline (B) recogniser. The latter has no noise compensation other than what is implicit in the feature extraction (see Section 5.3).

One point of interest is the fact that the LERs for SI with window length T w = 1 are much higher than for any other window length. In fact, in many cases the performance at T w = 1 is even worse than the baseline performance.

Possible causes for this effect will be discussed in Section 7. In the rest of this section, we will largely ignore the data points at T w = 1, and focus on a comparison of CI and B with SI at window lengths T w ≥ 5.

Another remark that holds for virtually all testing conditions is that with SI the performance seems to have an optimum in the range T w = [START_REF] Van Hamme | PROSPECT features and their application to missing data techniques for robust speech recognition[END_REF][START_REF] Junqua | Robustness in Automatic Speech Recognition: Fundamentals and Applications[END_REF]. Generally speaking, however, the differences obtained with various window sizes are quite small. Taking window size into account when comparing the SI results with those of the other methods would unnecessarily complicate matters. In the description below we will therefore focus on gross effects that can be observed in the range T w = [START_REF] Van Hamme | PROSPECT features and their application to missing data techniques for robust speech recognition[END_REF][START_REF] Junqua | Robustness in Automatic Speech Recognition: Fundamentals and Applications[END_REF].

The first row in Fig. 3 illustrates results on the headset recorded data, the condition which resembles clean speech the most. Although some of the observed differences are statistically significant, the differences in performance of CI, SI, and B are quite small.

In the case of the lavalier microphone data, we can observe more differences between the car and public environments. In the car environment, both the baseline and CI method achieve 7 to 11 % absolute lower accuracies than on data recorded in the public environment. In the public environment, SI performs comparable with CI for window lengths in the range T w = [START_REF] Van Hamme | PROSPECT features and their application to missing data techniques for robust speech recognition[END_REF][START_REF] Junqua | Robustness in Automatic Speech Recognition: Fundamentals and Applications[END_REF], while in the car environment SI outperforms CI by some 4 % (absolute).

In the case of the far-field microphone (public environment) or rear-view mirror (RVM) microphone (car environment), the results are similar as for A c c e p t e d M a n u s c r i p t the lavalier microphone. Again, the car environment proves to be the more difficult recognition environment with a baseline LER score of 67.3 %, compared to 38.3 % in the public environment. As with the lavalier microphone data, the SI method performs substantially better than CI in the car environment, doing up to 11 % (absolute) better for window lengths in the range T w = [START_REF] Van Hamme | PROSPECT features and their application to missing data techniques for robust speech recognition[END_REF][START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF]. In the public environment, SI also performs much better than CI, although by a smaller margin.

All in all, both CI and SI have a positive effect on recognition performance in comparison to the baseline, but clearly SI performs better in the more difficult conditions (lavalier and RVM microphone data from the car environment and the far-field microphone data from the public environments) at the cost of a small performance loss in relatively clean environments.

Artificially corrupted speech: babble noise scenario

Recognition performance using the clean speech data artificially corrupted with babble noise is displayed in Fig. 4. The baseline result for the clean speech signal is LER = 3.3 %.

In the SNR = 15 dB condition, SI and CI achieve comparable accuracies with LER ≈ 7.5 %, again with the exception of T w = 1 when using an estimated missing data masks. We can also observe that using an error-free oracle masks leads to a much lower number of recognition errors: SI now comes closer to clean speech baseline recognition scores (LER = 4.1 %) while CI achieves 4.9 % LER. While in the case of the estimated mask, there is an indication of an optimum window length in the range T w = [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF][START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF], no such optimum can be seen when using the oracle mask.

In the SNR=10 dB condition, SI does significantly better than CI both for estimated and oracle missing data mask. The same can be observed in the SNR=5 dB condition, although the gap between SI and CI performance becomes much larger at lower SNR's, particularly when using the oracle mask. This indicates that SI gains more performance from the extra, error-free information contained in the oracle mask. In both SNR conditions, it is difficult to see a clear relation between the window length and SI performance, although LER seems to show a shallow minimum around T w = [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF][START_REF] Junqua | Robustness in Automatic Speech Recognition: Fundamentals and Applications[END_REF]. Remarkably, even in the T w = 1 condition, SI does better than CI when using the oracle mask.

In the SNR=0 dB condition when using an estimated mask, neither SI nor CI can reconstruct the clean speech signal to a sufficient degree to achieve a usable performance , since both methods have LER around 75 %. When A c c e p t e d M a n u s c r i p t using an oracle mask, the situation is quite different. While CI achieves a LER of 68.3 %, gaining less than 10 % from using an oracle mask, SI achieves error rates of only 11.2 % at SNR = 0 dB.

Discussion

Sparse imputation for large vocabulary continuous speech recognition

Research on noise robust MDT started out with experiments on small vocabulary tasks artificially corrupted by noise [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF][START_REF] Van Hamme | PROSPECT features and their application to missing data techniques for robust speech recognition[END_REF]. While vocabulary sizes have since increased, in the majority of cases still artificially corrupted speech has been employed [START_REF] Raj | Reconstruction of missing features for robust speech recognition[END_REF][START_REF] Srinivasan | Binary and ratio time-frequency masks for robust speech recognition[END_REF][START_REF] Van Segbroeck | Vector-Quantization based mask estimation for missing data automatic speech recognition[END_REF].

It is not until recently that research has turned to MDT on large vocabulary speech recorded in realistic conditions [START_REF] Remes | Missing feature reconstruction and acoustic model adaptation combined for large vocabulary continuous speech recognition[END_REF][START_REF] Gemmeke | Application of noise robust MDT speech recognition on the SPEECON and SpeechDat-Car databases[END_REF]. In this work, we investigated whether the improvements in recognition accuracy obtained with SI on the aurora-2 digit recognition task [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF] would generalise to a large vocabulary task.

Experiments on the speecon data indicate that SI is 1) indeed capable of significantly improving large vocabulary continuous speech recognition (LVCSR) performance on noisy speech data and 2) performs equally well on artificially corrupted data and noisy speech recorded in real-world environments where different types of microphones and different microphone-speaker distances were used.

Compared to the cluster-based imputation (CI) method, SI improved the speech recognition performance especially at low SNR's. When using an estimated missing data mask, SI performance on the cleanest conditions (headset-recorded data or artificially corrupted data at SNR=15 dB) was comparable or slightly lower than the CI performance, but SI performed better than CI in all the noisier conditions. The difference between SI and CI was most notable when using SI on the car data recorded with a rear-viewmirror (RVM) microphone, which resulted in 26 % relative error reduction in LER. In the experiments using oracle masks, SI outperformed CI at all SNR levels, and as with the estimated masks, the differences in performance grew larger at low SNR's. On the artificially corrupted data at SNR=0 dB, the relative error reduction in LER was 84 % compared to the CI performance.

Sparse representation of speech

Initially, we expressed some concern regarding the performance limits of an exemplar-based method in modelling large vocabulary speech, especially A c c e p t e d M a n u s c r i p t when using windows spanning multiple time frames. The oracle mask results reported in Section 6, however, serve as empirical evidence that such concerns are unfounded, since the use of 8000 exemplars suffices for SI to find a proper reconstruction of the underlying clean speech, even when many features are missing.

To study this issue in more detail, we did a small additional experiment in which we investigated for three window lengths the sparsity of clean (uncorrupted) speech of a random subset of 10 utterances of the speecon test database. At window lengths T w ∈ {10, 20, 30}, the utterances contain 6794, 6694, 6594 windows, respectively.

For each of the windows, the observation vector was first normalised to a Euclidean unit norm after which we recovered its sparse representation x using the l1 ls nonneg solver solver (cf. Section 5.6) and sorted the elements of this vector with respect to weight. Finally, we averaged the sorted weight vectors over all windows.

The result is an average weight vector ordered with respect to weight which indicates how many exemplars (on average) are needed to represent the windows of the selected test utterances. For each window length, the 45 largest weights are shown in Figure 5. From this figure it can be deduced that the fixed length spectrogram segments of large vocabulary speecon speech in the test set can indeed be sparsely represented. The results show that, within the accuracy of the solver, on average windowed spectra can be sparsely represented using no more than approximately 30 exemplars.

Low-noise conditions

With CI performance being consistently lower than the SI performance in all the noisier conditions, it is interesting to explore in more detail why CI performs comparable or better than SI when using an estimated mask in the cleanest conditions, i.e., the headset-recorded data or artificially corrupted data at SNR=15 dB. A factor that may contribute to the difference is that CI uses the unreliable features as an upper bound during the missing feature reconstruction. This has been shown to improve the MDT performance in various noise conditions [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF]. In SI, the upper bound is applied on the reconstructed features after selection of the exemplars as indicated in Equation [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF]. As a consequence, it is not taken into account that if some of the estimates are considered incorrect because they are larger than the observed upper bound, there is no guarantee the other features are correct since they stem from the same linear combination of exemplars. The upper bounds are likely to have the largest impact in low-noise conditions, which is where CI performs best, but the treatment of upper bounds does not explain why SI does better than CI at SNR=15 dB if the estimated masks are replaced with oracle masks. Analysing the recognition errors in detail (not shown) revealed that when CI outperforms SI, the difference is mainly due to insertion errors. This is in line with the observations in [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF]. The SI method is prone to insertion errors because even a single, isolated, reliable feature results in the entire T w -frame window being represented as a linear combination of clean speech exemplars. Depending on the decoder, such a single reliable feature could lead to the insertion of a segment/letter/word. These insertion errors are most likely when using an estimated mask that erroneously marks spurious features reliable.

It is noteworthy that in the noisier conditions, an isolated reliable feature may be all that is left of the underlying clean speech, and indeed, the same property that makes SI prone to insertion errors in relatively clean conditions is what causes the performance gain in the more difficult noise conditions where SI performs better than CI. A solution for this issue may A c c e p t e d M a n u s c r i p t be found in techniques such as weighted Viterbi decoding [START_REF] Tan | Exploiting temporal correlation of speech for error-robust and bandwidth-flexible distributed speech recognition[END_REF] or uncertainty decoding [START_REF] Liao | Issues with uncertainty decoding for noise robust automatic speech recognition[END_REF], in which the decoder takes the uncertainty of the accuracy with which clean speech was estimated into account. Preliminary experiments on artificially corrupted speech showed that SI performance indeed increases both at high and low SNR's when using a proper measure of uncertainty (cf. [START_REF] Gemmeke | Observation uncertainty measures for sparse imputation[END_REF]).

Optimal time context 7.2.1. Using multiple frames of time-context

Figures 3 and4 in Section 6 clearly indicate that increasing the time context beyond T w = 1 in the SI method improves the speech recognition performance. In most conditions, the optimal recognition performance is achieved when the time context in SI is approximately 5-20 frames. A detailed analysis of the recognition errors (not shown) revealed that increasing the time context past the optimum range systematically leads to an increased number of deletion errors and that in general, when SI outperformed CI, it was due to a reduced number of deletion errors.

Analysis of the clean estimates produced by SI at window sizes larger than 5-20 frames (not shown) revealed that the increased number of deletions is due to difficulties in finding a sparse linear combination of exemplars that would describe the high-dimensional observation sufficiently accurately. Consequently, the resulting sparse representations only capture the high energy regions of the spectrogram window and do not describe the details accurately. In addition, the number of spectrograms over which we average to calculate the final estimate increases with increasing window length, results in smoothed approximations for the clean speech spectrograms and tends to decrease the dynamic range of the feature values. Since the decoder employs mean normalisation, the smoothed, mean-normalised features result in a reduced contrast between states and may even start to resemble silence. This, in turn, can lead to recognition errors such as deletions.

Finally, as the results reported in Section 6 and the previous results in [START_REF] Gemmeke | Compressive sensing for missing data imputation in noise robust speech recognition[END_REF] indicate that missing data imputation using SI benefits from increasing the time context, the question could come to mind whether using more timecontext might also be advantageous for CI. While we do not address this question in this work, it should be noted that the computational complexity of CI is O (K 4 ), where K denotes the feature dimension, which shows that adding time context (effectively increasing the number of features per frame) would quickly become infeasible. Moreover, there is a quadratic increase A c c e p t e d M a n u s c r i p t in the amount of data required to accurately estimate the full covariances needed for CI, which can lead to training data scarcity.

SI performance for single time frames

Although the SI performance is not better than CI performance for every noise scenario and window length T w , the performance is never much worse than CI, except for T w = 1 in combination with an estimated mask. In this case, the SI performance is occasionally even worse than the uncompensated baseline system performance. However, if oracle masks are used instead, the SI performance is substantially better than CI, even with T w = 1.

There are two issues that could explain the difference. First, the way frames are treated when all features are unreliable is different in SI and CI. In SI, frames without any reliable features are imputed as silence. If several of such frames occur during a speech segment, the decoder is more or less forced to recognise these segments as silence. Since it becomes less likely that a speech segment of T w frames does not contain any reliable frames when T w is large, SI achieves better results at longer window sizes. In contrast, CI imputes frames without any reliable values by making use of the prior probability given the noisy data. In this approach, the imputed frame is less likely to be interpreted as silence all the time.

To test this hypothesis, we ran an additional experiment on the artificial babble noise data. Here, CI was modified to also impute silence for all the frames that contain no reliable features. The results (not shown) confirmed that this decreases the system performance when an estimated missing data mask is used, although the performance was still better than the SI performance at T w = 1. Interestingly, the modification improved the CI performance when an oracle mask was used, although the obtained accuracies were still substantially lower than the SI accuracies at T w = 1. In summary, this experiment shows not only that imputing silence when all features are unreliable is only a good approach in the absence of mask estimation errors, but also that the difference in treating frames without any reliable values does not fully explain the differences between SI using T w = 1 and CI.

A second factor that may contribute to the sub-optimal performance of SI when T w = 1 and an estimated mask is used, is that at T w = 1, the minimisation problem (9) gets extremely under-determined as the number of reliable features decreases. It is a property of the applied technique that the quality of the reconstruction does not deteriorate gradually, but suffers from a sudden break-down once the number of reliable features gets below A c c e p t e d M a n u s c r i p t a certain threshold. This happens because with very few reliable values, the sparsest representation can suddenly be constituted of rather arbitrary exemplar vectors. As discussed in detail in [START_REF] Gemmeke | Compressive sensing for missing data imputation in noise robust speech recognition[END_REF], it is very difficult to give an estimate on the minimum number of (reliable) features needed for imputation, but the results in the paper can be seen as an empirical indication that in low SNR conditions, a sizable proportion of single time frames does not contain enough reliable features for the SI method to work properly.

In light of these results, we may need to reconsider the SI approach to impute silence for frames that contain no reliable features, at least when a short window is used. Possibly, a better approach would be to interpolate between neighbouring frames, to use the output of the CI method for frames without any reliable values, or to use an approach related to the uncertainty decoding approach mentioned above by setting all state likelihoods to an equal value for frames which do not contain reliable features.

Influence of mask quality

It is well known that recognition accuracy improvements obtainable with MDT are highly dependent on the quality of the applied missing data mask. When comparing the CI results for estimated and oracle mask, it is obvious that the absence of mask estimation errors substantially improves recognition accuracy, with relative LER improvements ranging from 10% to 45%. The same holds for sparse imputation, although the differences are larger, particularly at low SNR's. As can be seen in Figure 4, at an SNR of 0 dB, SI achieves an accuracy of 11.2 % LER as opposed to the 73.1% with the estimated mask at T w = 20. As in our previous experiments on a digit recognition task [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF], the large performance gap between the two types of missing data masks indicates that SI can potentially perform much better than CI, provided mask estimation errors are reduced.

An equally valid conclusion is that SI is more sensitive to mask estimation errors than CI. The reason for this is that mask estimation errors which incorrectly label features reliable will mislead the search for the 'true' sparse representation associated with the underlying clean speech. Depending on the location of these features, imputed features can become very different from the underlying clean speech. Using a more conservative estimation method is no solution, since mask estimation errors which incorrectly label features as unreliable reduce the overall number of reliable features available for imputation and cause the search to miss out on features useful for distinguishing between exemplars [START_REF] Gemmeke | Compressive sensing for missing data imputation in noise robust speech recognition[END_REF].

A c c e p t e d M a n u s c r i p t

The recognition performance using error-free oracle masks suggest an even larger real-world potential for SI, provided substantially better mask estimation methods can be found. Alternatively, the SI method could be made more robust against mask estimation errors by a number of algorithmical improvements. An attractive approach would be to change the search for exemplars, i.e., the minimisation problem [START_REF] Stouten | Robust automatic speech recognition in time-varying environments[END_REF], to include the constraint that the reconstructed speech should not exceed the noisy observation. Other possibilities for improvements are the additional use of derivative features in the exemplars rather using only static features, or the use of soft missing data masks [START_REF] Barker | Soft decisions in missing data techniques for robust automatic speech recognition[END_REF]. In soft masks the binary reliability score is replaced by the probability that a spectral component is reliable, providing more robustness against mask estimation errors. For a more extensive discussion on these potential improvements, we refer the reader to [START_REF] Gemmeke | Compressive sensing for missing data imputation in noise robust speech recognition[END_REF][START_REF] Gemmeke | Sparse imputation for noise robust speech recognition using soft masks[END_REF].

Computational effort

To roughly characterise the computational effort needed, we did a small test of the running time of the CI algorithm on a machine with a Core 2 Duo E6550 2.33 GHz processor. The running time for an utterance of 756 frames (6 seconds of speech) containing 11 734 unreliable values was 80 seconds. For the SI algorithm with a window length of T w = 10 frames, the running time on this utterance was 61 seconds.

While SI is faster for this particular utterance, SI still performs at about 10 times real-time. It is therefore interesting to study its computational complexity. SI using the SolveLasso solver has a computational complexity of O (W ((RT w ) 3 + N RT w )), where W denotes the number of windows to be processed, N the number of clean speech exemplars in the dictionary, R the average number of reliable features per frame, and T w the window length in frames. In practice, the computational complexity is completely dominated by the term N RT w .

There are three ways to reduce the computational effort, each of which scale approximately linearly. The first is to increase the window shift ∆ which decreases the number of windows W . In [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF] it was shown that increasing ∆ too much reduces imputation performance, but also that small increases do not decrease accuracy. Moreover, ∆ does not have to be constant over the utterance and could for example be made dependent on the number of reliable features. The second approach is to reduce the number of features in each window. Aside from reducing the window length, which in Section 7.2 was shown to decrease performance when reduced to much, it is also possible A c c e p t e d M a n u s c r i p t to apply dimensionality reduction to the features in a window, as used in [START_REF] Wright | Robust face recognition via sparse representation[END_REF]. Finally, the dictionary size might be reduced by methods such as clustering and an algorithmic way to handle shift-invariance rather than by including time-shifted variants of the same phenomena.

Conclusions and future work

In this work, we investigated the performance of the sparse imputation missing data technique on read sentences of the Finnish speecon corpus using real-world noisy speech recordings as well as clean speech recordings artificially corrupted with babble noise. In previous research sparse imputation was shown to be an effective method for improving the noise robustness of a connected digit recognizer using the artificially noisifid aurora-2 data [START_REF] Gemmeke | Missing data imputation using compressive sensing techniques for connected digit recognition[END_REF] . The current results show that the method can be readily extended to a large vocabulary recognition task in which the speech suffers from corruptions found in real-world environments. We found that also in the speecon corpus, fixed-length spectrogram windows can be adequately represented by a sparse, linear combination of exemplars: On average less than 30 are needed. As with the aurora-2 task, sparse imputation on speecon greatly benefits from using additional time-context for imputation. With a dictionary size of 8000 randomly selected exemplars used in this study, we typically found a context of 5-20 frames (i.e., 50-200 ms) to yield the best recognition accuracies.

Experiments on artificially corrupted speech indicated that sparse imputation outperforms a conventional imputation technique by a significant margin when the ideal 'oracle' reliability of noisy speech features are used. With error-prone reliability estimates, sparse imputation performs slightly worse than our baseline imputation technique in the cleanest conditions, but significantly better at lower SNR's.

Future work in the sparse imputation framework will focus on improving the robustness toward mask estimation errors and better handling the frames which do not contain any reliable values.

9 A

 9

Figure 1 :

 1 Figure 1: Schematic representation of sparse imputation, using isolated, presegmented digits as an example. Digit labels are at the top of the log-power spectrograms, where "Z" denotes "zero" and "O" denotes "oh". Panel A shows the sparse representation of the digit "three" in the case of clean, unmasked speech. Panel B shows the same digit with background noise at -5 dB SNR. The missing data (in black) is replaced by the corresponding features of the linear combination of clean speech dictionary exemplars found. In both panels, only the five largest nonzero weights of the linear combination are shown. 10

  A c c e p t e d M a n u s c r i p t

Figure 2 :

 2 Figure 2: Schematic diagram of the sliding window approach for imputation. The dark shaded time-frame in Y tot is processed in several fixed-length imputation windows, of which we have shown Y w through Y w+3 . Within each window, the given frame takes a different position due to the window shift ∆. The corresponding time-frame in the clean speech estimate Ŝtot is the average over these individual window-based imputations.

Figure 3 :

 3 Figure3: Recognition accuracy expressed as letter error rates (LER) for the public (left pane) and the car environment (right pane). From top to bottom, the rows correspond to headset, lavalier, and far-field microphone (public environment) or rear-view mirror (RVM) microphone (car environment). In each panel, the LER is shown as a function of window size T w (in frames) for the sparse imputation (SI) method (solid line) with vertical bars around the data points indicating the 95% confidence intervals. The 95% confidence intervals for the cluster-based imputation (CI) method are indicated by dashed lines and that of the baseline recogniser (B) by dotted lines.

Figure 4 :

 4 Figure4: Recognition accuracy expressed as letter error rates (LER) for the dataset containing clean speech artificially corrupted with babble noise. The results are shown for the estimated missing data mask (left pane) and oracle missing data mask (right pane). Row one through four apply to different signal-to-noise ratio's (SNR). In each panel, the LER is shown as a function of window size T w (in frames) for the sparse imputation (SI) method (solid line) with vertical bars around the data points indicating the 95% confidence intervals. The 95% confidence intervals for the cluster-based imputation (CI) method are indicated by dashed lines and that of the baseline recogniser (B) by dotted lines.

Figure 5 :

 5 Figure 5: The sparsity of clean speech in a subset of the speecon test database. The graph shows the average weight of the 45 largest nonzero elements of x of each sparsely represented window in a random subset of 10 utterances. The results displayed here pertain to the window lengths T w ∈ {10, 20, 30}.

A

  c c e p t e d M a n u s c r i p t

Table 1 :

 1 Composition of development and test set.

			public		car			babble
	#u d (min) #s #f #m #u d (min) #s #f #m #u d (min) #s #f #m
	Development 587	60	20 7 13 288	29	10 2	8	1093	115	40 22 18
	Evaluation 878	94	30 13 17 575	57	20 12 8	1118	113	40 21 19

This toolbox is publicly available from www2.it.lut.fi/project/gmmbayes/

This solver is implemented as part of the SparseLab toolbox which is publicly available from http://www.sparselab.stanford.edu

This solver is publicly available from http://www.stanford.edu/ ~boyd/l1_ls/

Acknowledgement

The research by Jort Florent Gemmeke was carried out in the MIDAS project, granted under the Dutch-Flemish STEVIN program. The research by Ulpu Remes was supported by the Helsinki Graduate School in Computer 29