Nils Holzenberger

Mingxing Du
email: mingxing.du@polytechnique.edu

Julien Karadayi
email: julien.karadayi@gmail.com

Rachid Riad
email: riadrachid3@gmail.com

Emmanuel Dupoux
email: emmanuel.dupoux@gmail.com

Keywords: unsupervised speech processing, audio word embeddings, ABX discrimination, same-different classification, representation learning

Fixed-length embeddings of words are very useful for a variety of tasks in speech and language processing. Here we systematically explore two methods of computing fixed-length embeddings for variable-length sequences. We evaluate their susceptibility to phonetic and speaker-specific variability on English, a high resource language, and Xitsonga, a low resource language, using two evaluation metrics: ABX word discrimination and ROC-AUC on same-different phoneme n-grams. We show that a simple downsampling method supplemented with length information can be competitive with the variable-length input feature representation on both evaluations. Recurrent autoencoders trained without supervision can yield even better results at the expense of increased computational complexity.

Learning word embeddings: unsupervised methods for fixed-size representations of variable-length speech segments

Introduction

Techniques to efficiently embed words into a vector space [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Pennington | Glove: Global vectors for word representation[END_REF] have led to spectacular successes in several Natural Language Processing (NLP) tasks such as language modeling [START_REF] Ororbia | Learning simpler language models with the differential state framework[END_REF], translation [START_REF] Zou | Bilingual word embeddings for phrase-based machine translation[END_REF], topic modeling [START_REF] Cao | A novel neural topic model and its supervised extension[END_REF] and anaphora resolution [START_REF] Marasovic | A mentionranking model for abstract anaphora resolution[END_REF]. In speech, similar techniques could be very useful, and are already used for several applications such as query by example [START_REF] Settle | Query-byexample search with discriminative neural acoustic word embeddings[END_REF], spoken term discovery [START_REF] Kamper | Unsupervised word segmentation and lexicon discovery using acoustic word embeddings[END_REF] and unsupervised representation learning [START_REF] Chung | Audio word2vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder[END_REF][START_REF] Chung | Learning word embeddings from speech[END_REF]. However, before applications inspired by NLP can be deployed, a complication specific to speech has to be addressed. In text, the embedding for two instances of the same word will be the exact same vector. With audio input, due to phonetic and speaker variability, they won't be identical. Instead, we are dealing with a distribution over vectors, potentially overlapping with the distribution of another word. Therefore, a prerequisite for developing useful vector representations for spoken words is to minimize the impact of this variability.

Here, we propose to systematically explore the phonetic variability of audio word embeddings using two widely different approaches. The first one has been proposed by [START_REF] Kamper | A segmental framework for fully-unsupervised large-vocabulary speech recognition[END_REF]. It relies on the idea of downsampling the time sequence of acoustic information using a fixed number of samples. This is a simple idea, which requires no training, and is worth investigating, at least as a baseline. The second idea is to use Recurrent Neural Networks (RNN) as autoencoders, by training them to encode a sequence of frames into a vector, and to decode it back to the original sequence of frames. Contrary to downsampling, this * Work done while author was at ENS technique requires training, but no labels are needed, making it therefore applicable to a variety of speech signals.

We propose to evaluate the phonetic variability of word embeddings using two metrics, based on the computation of the within-word and between-word distance of embeddings. The first one, ABX, tests the average discriminability between pairs of words. The second one, AUC, tests for the separation, across the entire corpus, of the distribution of cosine or Euclidean distance for pairs of same phoneme n-grams versus the distribution of that distance for pairs of different phoneme n-grams.

Related work

The framework of segmental speech recognition [START_REF] Zweig | A segmental CRF approach to large vocabulary continuous speech recognition[END_REF] involves an explicit segmentation of speech utterances and a fixed-size representation for each acoustic segment. [START_REF] Glass | A probabilistic framework for segment-based speech recognition[END_REF] use mean pooling and temporal derivatives over subparts of each segment, duration and energy, while [START_REF] Ostendorf | A stochastic segment model for phoneme-based continuous speech recognition[END_REF] warp segments into a fixed length. Recent approaches represent segments with RNNs [START_REF] Tang | End-to-end neural segmental models for speech recognition[END_REF], or Deep Neural Networks (DNN) together with subsampling, pooling and duration information [START_REF] Abdel-Hamid | Deep segmental neural networks for speech recognition[END_REF][START_REF] He | Segmental conditional random fields with deep neural networks as acoustic models for first-pass word recognition[END_REF]. Fixed-size embeddings are also used for speaker verification and extracted with factor analysis [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF] or DNNs [START_REF] Li | Deep speaker: an end-to-end neural speaker embedding system[END_REF]. DNNs have been used to derive embeddings by training them with a supervised objective, although not for speech recognition per se: [START_REF] Maas | Word-level acoustic modeling with convolutional vector regression[END_REF][START_REF] Harwath | Learning word-like units from joint audio-visual analysis[END_REF] alternate convolutional and pooling layers, finishing with a fixed length vector.

In [START_REF] Levin | Fixeddimensional acoustic embeddings of variable-length segments in low-resource settings[END_REF], word embeddings are obtained unsupervisedly by first memorizing a reference set of N words, and calculating the Dynamic Time Warping [START_REF] Vintsyuk | Speech discrimination by dynamic programming[END_REF] (DTW) distance of a new target word with each of these references. This set of distances is then projected into a subspace by dimensionality reduction. This method obtains good quality embeddings with a large reference set but suffers from a high computational cost. In [START_REF] Kamper | A segmental framework for fully-unsupervised large-vocabulary speech recognition[END_REF], this approach is replaced by a simpler procedure, where a fixedsize embedding is extracted from a sequence of frames by picking 10 equally-spaced frames from the sequence. In this paper, we will explore improvements of this technique.

More recently, sequence-to-sequence autoencoders [START_REF] Chung | Audio word2vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder[END_REF][START_REF] Chung | Learning word embeddings from speech[END_REF] have been used on speech to learn fixed-length embeddings in a purely unsupervised fashion. This has the advantage of providing embeddings in low-resource settings, for datasets where a transcript is not available. This is the kind of architecture and training setup we will use in this paper.

Proposed approach

Even though both downsampling and recurrent autoencoders have been used in past work, they have never been directly compared, nor have the hyperparameters of downsampling been thoroughly explored. In addition, we compare these models with a DTW baseline, which aligns raw features to compute the distance between two sequences.

Downsampling

The downsampling techniques used in [START_REF] Kamper | A segmental framework for fully-unsupervised large-vocabulary speech recognition[END_REF] extract a fixed number of equidistant samples from a time series. Here, we explore variants of this idea.

Formally, let x1, x2, ..., x T , with xt ∈ R 40 be a sequence of 40-dimensional log mel features. Equidistant downsampling samples k vectors at interval T k-1 with proportional interpolation as needed.

Let xq i be the i-th sample, where 1 ≤ i ≤ k and 1 = q1 < q2 < ... < q k = T , qj+1 -qj = T k-1 for j = 1, 2, ..., k -1. Note • and • the floor and ceiling functions, then we have :

xq i = x q i • ( qi -qi) + x q i • (qi -qi )
When qi is an integer, we take xq i as the sample; otherwise, the sample is a weighted sum of its left and right neighbors. The closer the neighbor, the more weight it contributes. The embedding of x1, x2, ..., x T is the concatenation of xq 1 , xq 2 ..., xq k .

The acoustic information in a word is not necessarily distributed uniformly along the time axis; psycholiguistics suggest that it is mostly located at the boundaries. Thus, we introduce non-equidistant downsampling: we assume the space between two samples follows a linear progression and is symmetric with respect to the center, i.e ∆j := qj+1 -qj = k(j -1) + b and qj+1 -qj = q k-j+1 -q k-j for j = 1, 2, ..., k/2 . To illustrate the degree of non-equidistance, instead of setting k as hyperparameter, we introduce the ratio α = ∆ k/2 ∆ 1 . Using q1 = 1 and q k = T , k and b can be deduced from α. α represents the ratio of the middle space against the first space. For example if α > 1 then we take more samples at the two ends, if α < 1 we take more samples in the middle, if α = 1, then we come back to equidistant downsampling.

We further extend proportional interpolation to Gaussian weighted interpolation, the idea being to avoid loosing information. Formally, consider the sequence of log mel features as a step function i.e. f (t) = x j for j -0.5 < t <= j + 0.5, j = 1, 2, ..., T (this amounts to saying that log mel features are represented at the center of the time stride). Then for i th sample xq i , we introduce gi = N (qi, σ 2 i ), a Gaussian density function centered at qi with variance σ 2 i . We have the following formula and illustrate it in figure 1:

xq i = T +0.5 0.5 gi(t)f (t)dt Zi where Zi is the normalization term, Zi = T +0.5 0.5 gi(t)dt.
Combining the downsampling methods with Gaussian weight, we have tested 5 variants with equidistant downsampling:

• (EPI) Proportional interpolation (baseline): no hyperparameters.

• (EAG) Absolute Gaussian weight: one hyperparameter σ, as described previously.

• (ERG) Relative Gaussian weight: one hyperparameter β, setting σ proportional to the sequence length, i.e. σ = βT .

• (EALG) Absolute linear Gaussian weight: two hyperparameters k, σ1. Instead of setting σ for all samples, we ∆ 1 , qj+1 -qj = q k-j+1 -q k-j , σj = k(j -1) + σ1, σj = σ k-j+1 for j = 1, 2, ..., k/2 . • (NRLG) Relative linear Gaussian weight: two hyperparameters α, β. The only difference from the previous one is that we set the interception proportional to the sequence length, i.e. σj = k(j -1) + βT , σj = σ k-j+1 for j = 1, 2, ..., k/2 . Note that k is computed by nonequidistant downsampling hyperparameter α. Finally, preliminary results suggest that including length information can further improve the quality of the embeddings. To this end, we compute the L2 norm (approximate the energy for each frame) for each log mel feature vector, and cut off the sequence to 150 frames if larger, pad with 0 if smaller than 150 frames. After scaling the L2 norms with a trade-off hyperparameter γ, we concatenate them to the downsampled embedding. Our results suggest that with this L2 norm, we achieve an average gain on ABX error of 6.09% and 1.01% for Xitsonga and English respectively. Downsampling each sequence of log mel features to 20 samples (each the output of 40-dimensional log mel filter banks), then concatenating these samples along with the L2 norm, leads to a final embedding size of 950.

Recurrent autoencoders

RNNs, in particular Long-Short Term Memory networks (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF], are the basis of state of the art solutions for numerous speech processing tasks. LSTMs can learn fixed-size representations of variable-length sequences, in text [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Schwenk | Learning joint multilingual sentence representations with neural machine translation[END_REF] and speech processing [START_REF] Chung | Audio word2vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder[END_REF][START_REF] Chung | Learning word embeddings from speech[END_REF]. Our proposed model, shown in figure 2, follows the encoder-decoder architecture used in [START_REF] Chung | Audio word2vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder[END_REF], frequently used in machine translation [START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF], and borrows some ideas from segmental ASR.

Let xt be the t-th 40-dimensional log mel feature vector, concatenated with 3 context frames on either side, a total of 7 concatenated frames (thus xt ∈ R 280 ). The encoder, a multilayer bidirectional LSTM [START_REF] Schuster | Bidirectional recurrent neural networks[END_REF], reads a sequence x1, ..., x T and encodes it into a of hidden states h1, ... xt is the decoder's output at time step t. The network is trained on segments of no more than 150 frames.

The training data has a very large amount of samples (English: 1.7M, Xitsonga: 653K), making it impractical to evaluate the neural networks on the timescale of epochs. Instead, we set aside 10% of the training data as a validation set. During training, we randomly take 100 samples from the training data and perform gradient descent using MSE loss and the Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with standard settings on this batch. After 100 training batches (i.e. 10 4 training samples), we randomly take 1000 samples from the validation set and use them to measure validation MSE. We use the neural net with the lowest validation MSE as our final model. The larger the model, the lower the MSE, so we must use the ABX score to compare architectures.

Evaluation

Given the broad scope of applications for low-resource speech technologies, and to develop methods fairly independent of their final application, we chose to evaluate the quality of embeddings intrinsically. We evaluate the learned embeddings with ABX discrimination tasks [START_REF] Synnaeve | Phonetics embedding learning with side information[END_REF] across speakers. We sample triplets of (sequence A, sequence B, sequence X) where each sequence is a sequence of acoustic features; A and B were said by speaker 1; X was said by speaker 2, and the label associated with X (e.g. "cat", or "ow-k") is identical to that of either A or B. In this paper, the label is either a word or an n-gram of phonemes. Using the method we wish to evaluate, we compute embeddings a, b and x from sequences A, B and X. Using distance metric d, we compute pairwise distances between a, b and x. We report two different metrics for this task.

• In the first setting, denoted ABX, sequences A, B and X are words. We look at the classification task of predicting whether X shares its label yX with A or B. We predict

ŷX = yA if d(a, x) < d(b, x)
yB otherwise and report error on this classification task. Triplets are sampled such that the performance of random guessing is 50%.

• In the second setting, denoted AUC, there is no X; sequences A and B correspond to phoneme n-grams where n = 2...12. We are still looking at a classification task, with

ŷ = "same" if d(a, b) < θ "different" otherwise
We look at the ROC curve obtained by measuring precision and recall for various values of θ. The score reported is the area under the ROC curve. Higher is better, and means less sensitivity with respect to parameter θ [START_REF] Fawcett | Roc graphs: Notes and practical considerations for researchers[END_REF].

For DTW, embedding x is actually X itself, and d is the DTW cosine distance with or without length normalization. For downsampling, d is the cosine distance, shown to work best for log mel features. For neural networks, d is the Euclidean distance, as the most natural distance metric over a vector space.

Datasets

We used the English and Xitsonga datasets provided by the Zero Resource Speech Challenge 2015 [START_REF] Versteegh | The zero resource speech challenge 2015[END_REF] to design and evaluate our models. The English corpus is a subset of the Buckeye corpus [START_REF] Pitt | Buckeye corpus of conversational speech (2nd release)[END_REF] (denoted as "English" in the following), and the Xitsonga corpus is the NCHLT Xitsonga Speech Corpus [START_REF] De Vries | A smartphone-based ASR data collection tool for under-resourced languages[END_REF] (denoted as "Xitsonga" in the following). Speech is represented as a sequence of 40-dimensional log mel feature vectors, computed on a sliding window of 25 ms, offset by 10 ms. The filter banks were mean and variance normalized per file (excluding silences), to help remove background noise and reduce acrossspeaker variability. Here are, in ms, average and standard deviation for the duration of English words: 236.2 and 153.6; English 2-to-12-grams: 547.7 and 500.1; Xitsonga words: 450.8 and 290.4; Xitsonga 2-to-12-grams: 617.4 and 327.2.

Results and discussion

For each downsampling method, we have conducted several grid searches. We kept the parameters achieving the best average ABX error on both corpora, since we would like parameters which can generalize well enough across different corpora. As shown in Table 1, although the best methods for English and for Xitsonga are different, the Equidistant downsampling with Absolute Linear Gaussian weight (EALG) method, which achieves the best average ABX, performs close enough to the best method on both corpora.

We investigated varying the depth and the number of hidden units of the recurrent autoencoders. Note that the size of the embedding is equal to twice the number of hidden units. Results from table 1 show, across both languages, that the larger the size of the embedding, the higher the ABX score and AUC. Depth of the network does not correlate with better scores.

On the AUC task, on both corpora, we find that recurrent autoencoder embeddings perform best, and downsampling outperforms normalized DTW by a comfortable margin. For the ABX task, normalized DTW and our methods perform somewhat on par on the Xitsonga corpus, while normalized DTW still performs best on the English corpus.

We further explored the amount to which the neural nets specialize to the language of the training data as training progresses. Figure 4 shows how the ABX score progresses on both English and Xitsonga datasets, as the MSE loss decreases on the validation set. Note that ABX error decreases even for mismatched tests sets (across a language change). This could be due to the fact that Xitsonga and English share some of their phonetic inventory, or that recurrent autoencoders are learning generally useful speech features, not necessarily tailored to the training language. Downsampling has a similar behavior, as it

Conclusion

In this paper, we propose two parametric methods to extract fixed-size embeddings from variable-length sequences of acoustic frames. The geometry of these embeddings captures linguistic properties, sometimes better than the geometry of the original acoustic space, as measured by DTW. We have thoroughly explored different strategies of downsampling, a method with few parameters, fast to compute and robust across languages. RNNs, trained with an unsupervised objective, and without any gold word-level segmentation of the input, manage to capture acoustic regularities, and further improve over downsampling. Using larger embeddings allows these models to fit the acoustic regularities even better, with higher ABX and AUC scores for larger networks. However, these embeddings come at a higher computational cost than downsampling. Interestingly, one can find, for each method, hyperparameters which give very good results for two very different languages. The RNN autoencoders even show some ability to generalize across languages after training. This suggests a possibility to precompile language-general spoken word embedders. In future work, it would be interesting to further improve the RNN autoencoders by training them as denoising autoencoders, using data augmentation such as adding reverberation, and encouraging the autoencoder to ignore speaker information, e.g. with a well-designed loss function.

Figure 1 :

 1 Figure 1: Illustration of Gaussian weighted interpolation. Note that our log mel features are generated with a 10 ms time stride.

Figure 2 :

 2 Figure 2: RNN-based autoencoders used to learn word embeddings. For simplicity, the positional embeddings are left out.

1 ;h backward 1 ,

 11 h T , where ht = [h forward t , h backward t ] and [a, b] is the concatenation of vectors a and b. We extract 3 representations from the encoder. The last hidden state of the forward LSTM h forward T ; the last hidden state of the backward LSTM h backward an element-wise maxpooling along the time axis of the sequence of hidden states, v = maxpool t ht so that [v]i = maxt[ht]i. These are concatenated and used as a representation of the full sequence: c = [h forward T , v]. The vector c is fed to an LSTM decoder at every time step t, along with a learned embedding et ∈ R 300 of the integer t. The decoder's input at time step t is thus [c, et], and its target output is xt. We train the network with Mean Squared Error (MSE) loss:

Figure 3 :Figure 4 :

 34 Figure 3: ABX error (top) and AUC score (bottom) for neural networks (RNNAE), downsampling and DTW baselines. We report our best model for each task.

4

 4 13.0 .665 10.4 .943 ERG β = 0.03, γ = 0.4 13.0 .683 10.4 .914 EALG k = 0.2, σ1 = 0.4, γ = 0.4 12.8 .707 10.4 .961 ERLG k = 0.07, β = 0.03, γ = 0.4 13.0 .739 10.4 .960 NPI α = 0.9, γ = 0.4 13.2 .657 10.5 .967 NAG α = 1.0, σ1 = 1.5, γ = 0.4 13.0 .665 10.4 .958 NRG α = 1.1, β = 0.02, γ = 0.4 13.1 .669 10.3 .912 NALG α = 0.8, σ1 = 1.4, γ = 0.4 13.2 .709 10.4 .966 NRLG α = 1.1, β = 0.02, γ = 0.4 13.0 .696 10.4 .963 works consistently well across languages.

Table 1 :

 1 ABX error (%) and AUC score. "AE" means RNN autoencoder, with the architecture of the encoder indicated as the number of hidden units per layer. The decoder's architecture mirrors the encoder's.

	English	Xitsonga

Acknowledgements

This project is funded by the European Research Council (ERC-2011-AdG-295810 BOOTPHON), the Agence Nationale pour la Recherche (ANR-10-LABX-0087 IEC, ANR-10-IDEX-0001-02 PSL* ), Microsoft Research (joint MSR-INRIA center) and a Google Award Grant.


Neil Zeghidour
email: neilz@fb.com

Nicolas Usunier
email: usunier@fb.com

Gabriel Synnaeve

Ronan Collobert

Emmanuel Dupoux
email: emmanuel.dupoux@gmail.com

End-to-End Speech Recognition From the Raw Waveform

Keywords: speech recognition, waveform, deep, end-to-end, scattering, gammatones

. We propose two modifications to these architectures and systematically compare them to mel-filterbanks, on the Wall Street Journal dataset. The first modification is the addition of an instance normalization layer, which greatly improves on the gammatone-based trainable filterbanks and speeds up the training of the scattering-based filterbanks. The second one relates to the low-pass filter used in these approaches. These modifications consistently improve performances for both approaches, and remove the need for a careful initialization in scattering-based trainable filterbanks. In particular, we show a consistent improvement in word error rate of the trainable filterbanks relatively to comparable mel-filterbanks. It is the first time end-to-end models trained from the raw signal significantly outperform mel-filterbanks on a large vocabulary task under clean recording conditions.

Introduction

State-of-the-art speech recognition systems rapidly shift from the paradigm of composite subsystems trained or designed independently to the paradigm of end-to-end training. While most of the work in this direction has been devoted to learning the acoustic model directly from sequences of phonemes or characters without intermediate alignment step or phone-state/senome induction, the other end of the pipeline model -namely, learning directly from the waveform rather than from speech features such as mel-filterbanks or MFCC -has recently received attention [START_REF] Palaz | End-to-end phoneme sequence recognition using convolutional neural networks[END_REF][START_REF] Tüske | Acoustic modeling with deep neural networks using raw time signal for lvcsr[END_REF][START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF][START_REF] Zhu | Learning multiscale features directly from waveforms[END_REF][START_REF] Van Den | Wavenet: A generative model for raw audio[END_REF][START_REF] Ghahremani | Acoustic modelling from the signal domain using cnns[END_REF][START_REF] Zeghidour | Learning filterbanks from raw speech for phone recognition[END_REF], but the performances on the master task of speech recognition still seem to be lagging behind those of models trained on speech features [START_REF] Xiong | The microsoft 2017 conversational speech recognition system[END_REF][START_REF] Han | The capio 2017 conversational speech recognition system[END_REF].

Yet, promising results have already been obtained by learning the front-end of speech recognition systems. We focus the discussion on trainable components that can be plugged in as replacement of mel-filterbanks without modification of the acoustic model. The approach inspired by gammatone filterbanks of Hoshen et al. and Sainath et al. [START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF] achieved similar or better results than comparable mel-filterbanks on multichannel speech recognition and on far-field/noisy recording conditions. More recently, Zeghidour et al. [START_REF] Zeghidour | Learning filterbanks from raw speech for phone recognition[END_REF] proposed an alternative learnable architecture based on a convolutional architecture that computes a scattering transform and can be initialized as an approxima-tion of mel-filterbanks, and obtained promising results on endto-end phone recognition on TIMIT. However, these approaches have not been proven to improve on speech features on largescale, end-to-end speech recognition in clean recording conditions on English -admittedly one of the tasks for which melfilterbanks have been the most extensively tuned.

We present a systematic comparison of the two previous architectures of learnable filterbanks, which we will (coarsely) refer to as gammatone-based and scattering-based, and evaluate them against mel-filterbanks within an end-to-end training pipeline on letter error rate and word error rate on the Wall Street Journal dataset. Our main contributions and results are the following:

1. A mean-variance normalization layer on top of the log nonlinearity of learnable filterbanks appears to be critical for the efficient learning of the gammatone-based architecture, and makes the training of the scattering-based architecture faster;

2. The low-pass filter previously used in the scattering-based learnable filterbanks stabilizes the training of gammatone filterbanks, compared to the max-pooling that was originally proposed [START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF];

3. For scattering-based trainable filterbanks, keeping the lowpass filter fixed during training allows to efficiently learn the filters from a random initialization, whereas the results of [START_REF] Zeghidour | Learning filterbanks from raw speech for phone recognition[END_REF] with random initialization of both the filters and the lowpass filter showed poor performances compared to a suitable initialization;

4. Both trainable filterbanks improve against the melfilterbanks baseline on word error rate on the Wall Street Journal dataset, in similar conditions (same number of filters, same end-to-end training convolutional architecture). This is the first time learnable filterbanks improve against a strong mel-filterbanks baseline on a large vocabulary, speech recognition task under clean recording conditions.

The next section describes the learnable filterbanks architectures. Then, we present the end-to-end convolutional architecture used to perform the comparisons, and analyze the results of our comparative studies.

Learning filterbanks from raw speech

The two approaches that we consider for learning filterbanks from the raw waveform can be used as direct replacement for mel-filterbanks in any end-to-end learning pipeline: they are convolutional architectures that take the raw waveform as input and output 40 channels every 10ms. As such, they can directly be compared with standard mel-filterbanks, simply by changing the features stage of a neural-network-based acoustic model. The filters are then nothing more than an additional layer to the neural network and are learnt by backpropagation with the rest of the acoustic model. The first architecture we consider is inspired by [START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF], the second one is taken from [START_REF] Zeghidour | Learning filterbanks from raw speech for phone recognition[END_REF]. They are described in Table 1.

In both architectures, a convolutional layer with window length 25ms (to match the standard frame size used in melfilterbanks) is applied with a stride of 1 sample, and is followed by a nonlinearity to give 40 output channels for each sample. Then, a pooling operator of width 25ms with a stride of 10ms performs low-pass filtering and decimation. Finally, a log nonlinearity reproduces the dynamic range compression of log melfilterbanks. The parameters to be learnt are the convolution filters, and possibly the weights of the low-pass filters.

The two architectures differ by the choices of each layer of computation. Hoshen et al. and Sainath et al. use 40 realvalued filters with ReLU non-linearity, and rely on gammatones as filter values to approximate mel-filterbanks [START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF]. In their work, they use a max-pooling operator for low-pass filtering. In contrast, Zeghidour et al. [START_REF] Zeghidour | Learning filterbanks from raw speech for phone recognition[END_REF] use 40 complex-valued filters with a square modulus operator as non-linearity. Low-pass filtering is then performed by multiplying each output channel by a squared Hanning window so that, when using suitable Gabor wavelets as convolution filters, the architecture closely approximates mel-filterbanks computed on the power spectrum [START_REF] Andén | Deep scattering spectrum[END_REF].

The number of filters (40), the convolution and pooling width of 25ms, as well as the decimation of 10ms are not necessarily the optimal parameters of either trainable architecture, but these are the standard settings of mel-filterbanks (and likely the best settings for these features on standard speech recognition datasets). We keep these values fixed for the trainable architectures, so that the comparison to mel-filterbanks is carried out in the setting most favorable for the non-learnable baseline.

In the next subsections, we describe the improvements we propose for these architectures: the low-pass filter and the addition of instance normalization.

Low-pass filtering

The original papers describing the gammatone-based trainable filterbanks used max-pooling as low-pass filter, whereas the scattering-based approach uses a squared Hanning window per channel. To make sure the low-pass filter is not responsible for notable differences between the two approaches we experiment with the squared Hanning window on both architectures. For both architectures, we also propose to keep this low-pass filter fixed while learning the convolution filter weights, a setting that was not explored by Zeghidour et al. [START_REF] Zeghidour | Learning filterbanks from raw speech for phone recognition[END_REF], who learnt the lowpass filter weights when randomly initializing the convolutions.

SCATTERING GAMMATONES

Conv 11-80-400-1 1-40-400-1

Instance normalization

More importantly, we noticed that a per-channel per-sentence mean-variance normalization after log-compression is important for the baseline mel-filterbanks. Consequently, we propose to add a mean-variance normalization layer on both trainable architectures, performed for each of the 40 channels independently on each sentence. Coincidently, this corresponds to an instance normalization layer [START_REF] Ulyanov | Instance normalization: The missing ingredient for fast stylization[END_REF], which has been shown to stabilize training in other deep learning contexts.

Experimental setup

The experiments compare different versions of the trainable architectures against log mel-filterbanks on a single deep convolutional network architecture for the acoustic model. The experiments are carried out on the open vocabulary task of the Wall Street Journal dataset [START_REF] Paul | The design for the wall street journalbased csr corpus[END_REF], using the subset si284 for training, nov93-dev for validation, and nov92-eval for testing. Training is performed end-to-end on letters. We evaluate in both letter and word error rates. All our experiments use the open source code of wav2letter [START_REF] Collobert | Wav2letter: an endto-end convnet-based speech recognition system[END_REF]. In the next subsections, we describe the model, the different variants we tested and the hyperparameters.

Acoustic model

Taking either log mel-filterbanks or trainable filterbanks, the acoustic model is a convolutional network with gated linear units (GLU) [START_REF] Dauphin | Language modeling with gated convolutional networks[END_REF] trained to predict sequences of letters, following [START_REF] Liptchinsky | Letter-based speech recognition with gated convnets[END_REF]. The model is a smaller version of the convolutional network used in [START_REF] Liptchinsky | Letter-based speech recognition with gated convnets[END_REF] All convolutions have stride 1. The number of input channels of the n + 1th convolution is half the size of the output of the n-th convolution because of the GLU. There are GLU layers with a dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] of 0.25 after each convolution layer. There is an additional linear layer to predict the final letter probabilities. When predicting letters, the training and decoding are performed as in [START_REF] Liptchinsky | Letter-based speech recognition with gated convnets[END_REF]. When predicting words, we use a 4-gram language model trained on the standard LM data of WSJ [START_REF] Paul | The design for the wall street journalbased csr corpus[END_REF] and perform beam search decoding, as in [START_REF] Liptchinsky | Letter-based speech recognition with gated convnets[END_REF].

Variants

We compare the two architectures of trainable filterbanks along different axes: how to initialize the convolutions of the trainable filterbanks, the low-pass filter, and instance normalization.

Gammatone-based architecture

Initialization of the convolution weights random (rand), or with gammatone filters (gamm) that match the impulse response of a reference open source implementation of gammatones [START_REF]Gammatone-based spectrograms, using gammatone filterbanks or fourier transform weightings[END_REF];

Low-pass filter max-pooling as in [START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF], or the squared Hanning window (Han-fixed).

Scattering-based architecture

Initialization of the convolution weights random (rand), or Gabor filters (scatt) as described in Section 2.2 of [START_REF] Zeghidour | Learning filterbanks from raw speech for phone recognition[END_REF];

Low-pass filter the squared Hanning window (Han-fixed), or a low-pass filter of same width and stride initialized with the weights of the squared Hanning window but the weights are then learnt by backpropagation (Han-learnt).

Hyperparameters and training

For models trained on the raw waveform, the signal was first normalized with mean/variance normalization by sequence. The network is trained with stochastic gradient descent and weight normalization [START_REF] Salimans | Weight normalization: A simple reparameterization to accelerate training of deep neural networks[END_REF] for all convolutional layers except the front-ends. First, 80 epochs are performed with a learning rate of 1.4, then training is resumed for 80 additional epochs with a learning rate of 0.1. These hyperparameters were chosen from preliminary experiments as they seemed to work well for all architectures. Additional hyperparameters are the momentum and the learning rate for the training criterion, respectively chosen in {0, 0.9} and {0.001, 0.0001} [START_REF] Collobert | Wav2letter: an endto-end convnet-based speech recognition system[END_REF][START_REF] Liptchinsky | Letter-based speech recognition with gated convnets[END_REF]. For Letter Error Rate (LER) evaluations, the hyperparameters are selected using the LER on the validation set, validating every epoch. For Word Error Rate (WER) evaluations, the hyperparameters are chosen on the validation set using the WER, validating every 10 epochs. The model selected on LER is also included for validation. The additional hyperparameters are the (i) SOTA -speech features: for state-of-the-art and representative baselines using speech features (mel-filterbanks, spectrograms or MFCC), (ii) SOTA-waveform: state-of-the-art from the raw waveform, including our own implementation of vanilla gammatone filterbanks without instance normalization, and (iii) our baseline and the different variants of the trainable filterbanks (with instance normalization) studied in this paper.

weight of the language model and the weight of word insertion penalty (see [START_REF] Liptchinsky | Letter-based speech recognition with gated convnets[END_REF] for details). We set them between 5 and 8 by steps of 0.5, and between -2 and 0.5 by steps of 0.1, respectively. For hyperparameter selection, the beam size of the decoder is set to 2, 500; the final performances are computed with the selected hyperparameters but using a beam size of 25, 000.

Experiments

Baseline results

Table 2 contains our results together with end-to-end baselines from the literature. [START_REF] Amodei | Deep speech 2: End-to-end speech recognition in english and mandarin[END_REF] is the current state-of-the-art on the WSJ dataset; it is given as a topline but uses much more training data (∼ 12, 000h of speech) so the results are not comparable. [START_REF] Graves | Towards end-to-end speech recognition with recurrent neural networks[END_REF][START_REF] Miao | Eesen: End-to-end speech recognition using deep RNN models and WFST-based decoding[END_REF][START_REF] Chorowski | Towards better decoding and language model integration in sequence to sequence models[END_REF][START_REF] Kim | Joint ctc-attention based endto-end speech recognition using multi-task learning[END_REF] are representative results in terms of WER and LER from the literature of end-to-end models trained on speech features from 2014-2017, in chronological order. [START_REF] Tjandra | Sequence-to-sequence asr optimization via reinforcement learning[END_REF] and [START_REF]Attention-based wav2text with feature transfer learning[END_REF] are the current state-of-the-art in LER on speech features and from the waveform respectively. These comparisons validate our baseline model trained on mel-filterbanks as a strong baseline in light of recent results, as it outperforms the state-of-theart in LER by a significant margin (4.9% vs 6.1% for [START_REF] Tjandra | Sequence-to-sequence asr optimization via reinforcement learning[END_REF]), and achieves a test WER of 6.6%, better than all other end-to-end baselines ( [START_REF] Zhou | Improving end-to-end speech recognition with policy learning[END_REF] and [START_REF] Ghahremani | Acoustic modelling from the signal domain using cnns[END_REF] report WER that are below our 6.6% but are on easier closed vocabulary tasks).

Instance normalization

As described in Section 2.2, we evaluate the integration of instance normalization after the log-compression in the trainable filterbanks, which was not used in previous work [START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF][START_REF] Ghahremani | Acoustic modelling from the signal domain using cnns[END_REF][START_REF] Zeghidour | Learning filterbanks from raw speech for phone recognition[END_REF] but is used in our baseline. Figure 1 shows training LER as a function of the number of epochs for scattering-based and gammatone-based filterbanks models, with and without instance normalization. We can see that this normalization drastically improves the training stability of the gammatonebased model, while it moderately improves the scattering-based model. We observed a positive impact of instance normalization in all settings, and so only report as a reference the results of our implementation of a vanilla gammatone-based trainable filterbanks following [START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF]. Comparing gammatone (learnt)/gamm/max-pool without instance norm (under SOTA -waveform) to the results of gammatone (learnt)/gamm/maxpool in Table 2, we see a significant improvements of both LER and WER due to instance normalization, with an absolute reduction in LER and WER of 1.5% and 2.8% respectively.

Impact of the low-pass filter

For low-pass filtering, we first compare the Han-fixed setting to max-pooling for gammatone-based filterbanks (as max-pooling was previously used in [START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF]), and to Han-learnt for scattering, all with instance normalization. The tendency is that the Han-fixed setting consistently improves the results in LER and WER of both trainable filterbanks. More importantly, using either an Han-fixed or Han-learnt filter when learning scatteringbased filterbanks from a random initialization removes the gap in performance with the Gabor wavelet initialization that was observed in [START_REF] Zeghidour | Learning filterbanks from raw speech for phone recognition[END_REF] where the lowpass filter was also initialized randomly. This is an important result since carefully initializing the convolutional filters is both technically non-trivial, and also relies on the prior knowledge of mel-filterbanks. We believe the ability to use random initialization is an important first step for more extensive tuning of trainable filterbanks (e.g., trying different numbers of filters, decimation or convolution width). Compared to the literature, replacing the max-pooling by a low-pass filter and adding an instance normalization layer leads to a 23% relative improvement in LER and a 33% relative improvement in WER on nov92-eval on the gammatone-based trainable filterbanks, a significant improvement compared to the existing approach [START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF]. Our models trained on the waveform also exhibit a gain in performance in LER of 22 -31% relative compared to the state-of-the-art end-to-end model trained on the waveform with its first 6 layers being pre-trained for melfilterbanks reconstruction [START_REF]Attention-based wav2text with feature transfer learning[END_REF], and outperform various end-toend models trained on speech features, both in LER [START_REF] Kim | Joint ctc-attention based endto-end speech recognition using multi-task learning[END_REF][START_REF] Tjandra | Sequence-to-sequence asr optimization via reinforcement learning[END_REF] and WER [START_REF] Graves | Towards end-to-end speech recognition with recurrent neural networks[END_REF][START_REF] Miao | Eesen: End-to-end speech recognition using deep RNN models and WFST-based decoding[END_REF][START_REF] Chorowski | Towards better decoding and language model integration in sequence to sequence models[END_REF].

Trainable filterbanks vs mel-filterbanks

Comparing both trainable filterbanks with instance normalization to the log mel-filterbanks baseline, we observe that the performances of the Han-fixed settings and of the mel-filterbanks are comparable in terms of LER. However, we observe a consistent improvement in terms of WER of all trainable filterbanks. To the best of our knowledge, this is the first time a significant improvement in terms of WER relatively to comparable melfilterbanks has been shown on a large vocabulary task under clean recording conditions. Some improvements on the clean test of the Switchboard dataset have previously been observed by [START_REF] Ghahremani | Acoustic modelling from the signal domain using cnns[END_REF], but their comparison point is MFCC rather than melfilterbanks and the number of filters of the trainable architecture differs from their MFCC baseline.

Adding a learnable pre-emphasis layer

The first step in the computation of mel-filterbanks is typically the application of a pre-emphasis layer to the raw signal. Preemphasis is a convolution with a first-order high-pass filter of the form y[n] = x[n] -αx[n -1], with α typically equal to 0.97. This operation can be performed by a convolutional layer of kernel size 2 and stride 1, that can be plugged below timedomain filterbanks, initialized with weights [-0.97 1], then learned with the network. In Table 3, we compare the performance of identical models (all using a fixed Hanning window, and a gammatone or scattering initialization) with and without pre-emphasis. We observe a gain on both LER and WER (except on nov93-dev WER/scatt) when using pre-emphasis.

Conclusion

This paper presents a systematic study of two approaches for trainable filterbanks, which clarifies good practices and identifies better architectures to learn from raw speech. Our results show that adding an instance normalization layer on top of the trainable filterbanks is critical for learning gammatone-based architectures, and speeds up learning of scattering-based architectures. Second, the use of a fixed squared Hanning window as low-pass filter is critical to learn the scattering-based filterbanks from random initialization of the filters, and improves on max-pooling for gammatone-based filterbanks. With these two improvements, we observe a consistent reduction of WER against comparable mel-filterbanks on the open vocabulary task of the WSJ dataset, in the setting of speech recognition under clean recording condition -most likely the setting on which mel-filterbanks have been the most heavily tuned.

Acknowledgements

This research was partially funded by the European Research Council (ERC-2011-AdG-295810 BOOTPHON), the Agence Nationale pour la Recherche (ANR-10-LABX-0087 IEC, ANR-10-IDEX-0001-02 PSL*).

Figure 1 :

 1 Figure 1: Training Letter Error Rate (LER) for a gammatone-based architecture (left) and a scattering-based architecture (right), with and without instance normalization.

  since they train on the larger LibriSpeech dataset. Using the syntax C-input channels-output channels-width, the architecture we use has the structure C-40-200-13/C-100-200-3/C-100-200-4/C-100-250-5/ C-125-250-6/C-125-300-7/C-150-350-8/C-175-400-9/ C-200-450-10/C-225-500-11/C-250-500-12/C-250-500-13/ C-250-600-14/C-300-600-15/C-300-750-21/C-375-1000-1.

Table 1 :

 1 Architectures

	non-linearity	sq. L2-Pooling ReLU
	low-pass filter (wdth=400, strd=160)	sq. Hanning	max-pooling or sq. Hanning
	log-compression 2	log(1+abs(.)) log(0.01+abs(.))
	normalization	mean-var. per-channel per-sentence

(#in-#out-width-stride) of the two trainable filterbanks. Values of width and strides are given to match the standard settings of mel-filterbanks for waveform sampled at 16kHz.

Table 2 :

 2 Results on the open vocabulary task of the WSJ dataset.

		MODEL		NOV93-DEV NOV92-EVAL LER WER LER WER
		SOTA -speech features
	Deep Speech 2 [20]	-	-	-	3.1
	-(+ additional data)
	RNN-WER -tri. LM [21]	-	-	-	8.2
	RNN -WSFT decoding [22]	-	-	-	7.3
	Seq2Seq + tri. LM [23]	-	9.7	-	6.7
	Multi-task CTC/att [24]	11.3	-	7.3	-
	Att + RL [25]		-	-	6.1
			SOTA -waveform
	Att Wav2Text (+transfer) [26] -	-	6.5	-
	gamm (learnt)/gamm/max-pool	8.9 12.9 6.4	8.8
	-(without inst. norm.)
	FRONT	FILTER LOW-	NOV93-DEV NOV92-EVAL
	END	INIT	PASS	LER WER LER WER
	mel-			6.9	9.5	4.9	6.6
	fbanks
	gamm	gamm	Han-fixed 6.9 max-pool 7.2	9.1 9.3	4.9 4.9	5.9 6.0
	(learnt)	rand	Han-fixed max-pool	7 7.2	8.9 9.2	4.9 5.1	5.9 6.3
	scatt	scatt	Han-fixed 6.7 Han-learnt 6.7	8.3 8.9	4.6 4.5	6.1 6.3
	(learnt)	rand	Han-fixed 6.8 Han-learnt 6.9	8.5 8.9	4.7 4.9	5.7 5.8

Table 3 :

 3 Comparison of models trained with or without a learnable pre-emphasis layer. All models are initialized either with the scattering or gammatone initialization, and the pooling function is a fixed squared Hanning window.

	MODEL	PRE-EMP	NOV93-DEV LER WER LER WER NOV92-EVAL
	gamm	no pre-emp 6.9	9.1	4.9	5.9
	(learnt)	pre-emp	6.8	9	4.7	5.7
	scatt	no pre-emp 6.7	8.3	4.6	6.1
	(learnt)	pre-emp	6.5	8.7	4.5	5.7

The convolution for the scattering-based architecture uses 80-real valued output channels and squared L2-pooling on the feature dimension to emulate a complex-valued convolution with 40 filters followed by a squared modulus operator. Thus, after the nonlinearity, both architectures have 40

filters.[START_REF] Tüske | Acoustic modeling with deep neural networks using raw time signal for lvcsr[END_REF] [START_REF] Zeghidour | Learning filterbanks from raw speech for phone recognition[END_REF] use 1 to prevent log(0) and[START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF] use 0.01. We kept the values initially used by the authors of the respective papers and did not try alternatives. We believe it has little impact on the final performance.


Thomas Schatz
email: thomas.schatz@laposte.net

Francis Bach
email: francis.bach@ens.fr

Emmanuel Dupoux
email: emmanuel.dupoux@gmail.com

Schatz

Evaluating automatic speech

Keywords: phonetic categories, human perception, quantitative modeling, ASR, 12 machine ABX

HAL is

Introduction

The way we perceive phonetic categories (i.e. basic speech sounds such as consonants and vowels) is largely determined by the language(s) to which we were exposed as a child. For example, native speakers of Japanese have a hard time discriminating between American English (AE) /ô/ and /l/, a phonetic contrast that has no equivalent in Japanese [START_REF] Goto | Auditory perception by normal japanese adults of the sounds l and r[END_REF][START_REF] Miyawaki | An effect of linguistic experience: The discrimination of [r] and [l] by native speakers of japanese and english[END_REF]. Perceptual specialization to the phonological properties of the native language has been extensively investigated using a varieties of techniques (see [START_REF] Strange | Speech perception and linguistic experience: Issues in cross-language research[END_REF]Cutler 2012 for reviews). Many of the proposed theoretical accounts of this phenomenon concur that foreign sounds are not perceived faithfully, but rather, are 'mapped' onto one's pre-existing (native) phonetic categories, which act as a kind of 'filter' resulting in the degradation of some nonnative contrasts [START_REF] Best | A direct realist view of cross-language speech perception. Speech Perception and Linguistic Experience: Issues in Cross-Language Research[END_REF][START_REF] Flege | Second language speech learning: Theory, findings, and problems. Speech perception and linguistic experience: Issues in cross-language research[END_REF][START_REF] Kuhl | Linguistic experience and the perceptual magnet effect. Speech perception and linguistic experience: Issues in cross-language research[END_REF][START_REF] Werker | Primir: A developmental framework of infant speech processing[END_REF]. In none of these theories, however, is the mapping specified in enough detail to allow a concrete implementation. In addition, in most of the existing theories 1 , even if a fully specified mapping was available, it remains unclear how predictions on patterns of error rates could be derived from it (the filtering operation). These theories remain therefore mainly descriptive.

In this paper, we propose to leverage ASR technology to obtain fully specified mappings between foreign sounds and native categories and then use the machine ABX evaluation task [START_REF] Schatz | Evaluating speech features with the minimal-pair ABX task: Analysis of the classical MFC/PLP pipeline[END_REF][START_REF] Schatz | ABX-Discriminability Measures and Applications[END_REF] to derive quantitative predictions from these mappings regarding cross-linguistic phonetic category perception.

More specifically, our approach can be broken down into three steps. First, train a phoneme recognizer in a 'native' language using annotated continuous speech recordings. Second, use the trained system to derive perceptual representations for test stimuli in a foreign language. In this paper, these will be vectors of posterior probabilities over each of the native phonemes. Third, obtain predictions for perceptual errors by running a psychophysical test over these representations for each foreign contrast. Machine ABX discrimination tasks will be used for this.

To showcase the possibilities offered by the approach, we look at predictions obtained for three empirically-attested effects in cross-linguistic phonetic category perception. The first two effects are global effects that apply to the set of phonetic contrasts in a language as a whole. First: native contrasts tend to be easier to distinguish than non-native ones [START_REF] Gottfried | Effects of consonant context on the perception of french vowels[END_REF]. Second: patterns of perceptual confusions are function of the native language(s): two persons with the same native language tend to confuse the same foreign sounds, which can be different from sounds confused by persons with another native language [START_REF] Strange | Speech perception and linguistic experience: Issues in cross-language research[END_REF]. Thanks to the quantitative and systematic nature of the proposed approach, these effects are straightforward to study.

We show that ASR models can account for both of them. Most effects documented in the empirical literature on cross-linguistic phonetic category perception are more local however. They describe patterns of confusion observed for very specific choices of languages and contrasts. We illustrate how such effects can be studied with our method through the classical example of AE /ô/-/l/ perception by native Japanese listeners [START_REF] Goto | Auditory perception by normal japanese adults of the sounds l and r[END_REF][START_REF] Miyawaki | An effect of linguistic experience: The discrimination of [r] and [l] by native speakers of japanese and english[END_REF]. We show that ASR models correctly predict the difficulty of perceiving this distinction for Japanese listeners.

Previous attempts at specifying mappings between foreign and native categories relied on phonological descriptions of the languages involved. Analyses at the level of abstract (context-independent) phonemes, however, were found not to be sufficient to fully account for perceptual data [START_REF] Kohler | Contrastive phonology and the acquisition of phonetic skills[END_REF][START_REF] Strange | Acoustic and perceptual similarity of north german and american english vowels[END_REF]. For example, the French [u-y] contrast can be either easy or hard to perceive for native AE listeners, depending on the specific phonetic context in which it is realized [START_REF] Levy | Effects of consonantal context on perception of french rounded vowels by american english adults with and without french language experience[END_REF]. Attempting to specify mappings explicitly through finer-grain phonetic analyses certainly remains an option, but involves a formidable amount of work. An [START_REF] Best | A direct realist view of cross-language speech perception. Speech Perception and Linguistic Experience: Issues in Cross-Language Research[END_REF] Quantitative models of phonetic category perception Page 4 Schatz, JASA-EL attractive and potentially less costly alternative consists in specifying mappings implicitly, through quantitative models of native speech perception. By this, we mean models that map any input sound to a perceptual representation adapted to the model's 'native language'. This representation can take the form of a phonetic category label, a vector of posterior probabilities over possible phones or some other, possibly richer, form of representation. Predictions regarding human perception of foreign speech sounds are then derived by analyzing the 'native representations' produced by the model when exposed to these foreign sounds.

Let us now explain the rationale for turning toward ASR technology, when the goal is to model human speech perception. This approach is best understood in the context of a top-down effort, where the focus is on developing models first at the information processing level, before considering issues at the algorithmic and biological implementation levels [START_REF] Marr | Vision: A computational approach[END_REF]. Native speech perception is thought to arise primarily from a need to reliably identify the linguistic content in the language-specific speech signal to which we are exposed, despite extensive para-linguistic variations.

ASR systems, whose goal is to map input speech to corresponding sequences of words, face the same problem. ASR systems seek optimal performance, and can thus be interesting as potential normative models of human behavior from an efficient coding point of view [START_REF] Barlow | Possible principles underlying the transformations of sensory messages[END_REF], even though biological plausibility is not taken into account in their development.

We found two previous studies taking steps in the proposed direction. In the first one [START_REF] Strange | Acoustic and perceptual similarity of north german and american english vowels[END_REF]

Methods

Speech recordings

To train and evaluate ASR models, 5 corpora of recorded speech in different languages were used: a subset of the Wall Street Journal corpus (WSJ) [START_REF] Paul | The design for the wall street journal-based csr corpus[END_REF],

April 26, 2018

Quantitative models of phonetic category perception Page 5 Schatz, JASA-EL the Buckeye corpus (BUC) [START_REF] Pitt | The buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability[END_REF], a subset of the Corpus of Spontaneous Japanese (CSJ) [START_REF] Maekawa | Corpus of spontaneous japanese: Its design and evaluation[END_REF], the Global Phone Mandarin (GPM) corpus [START_REF] Schultz | Globalphone: a multilingual speech and text database developed at karlsruhe university[END_REF] and the Global Phone Vietnamese (GPV) corpus [START_REF] Vu | Vietnamese large vocabulary continuous speech recognition[END_REF]. Important characteristics of the corpora are summarized in Table 1. Two corpora in American English were included to dissociate language-mismatch effects, in which we are interested, from channel-mismatch effects due to differences across corpora in recording conditions, microphones, speech register, etc. Phonetic transcriptions were obtained by combining word-level transcriptions with a phonetic dictionary for the WSJ, BUC, GPM and GPV corpora. For the CSJ corpus, manual phonetic transcriptions were used.

For all corpora, timestamps for the phonetic transcriptions were obtained by forced alignment using an ASR system similar to those described in the next section, but trained on the whole corpus.

ASR models

State-of-the-art ASR systems are built from deep recurrent neural networks. These systems, however, typically require hundreds of hours of data to be reliably trained and we decided to focus in this study on using older, but more stable, Gaussian-Mixture based Hidden-Markov Models (GMM-HMM) to ensure reasonable performance across all corpora. Each corpus was randomly split into a training and a test set of approximately the same size, each containing an equal number of speakers. There was no overlap between training and test speakers. Models were trained with the Kaldi toolkit [START_REF] Povey | The kaldi speech recognition toolkit[END_REF] using the same recipe with the same parameters and input features to train all models 4 . The Word-Error Rate 5 (WER) on the test set for each of the resulting models is reported in Table 1.

We will not attempt to describe the inner workings of the models beyond mentioning that a generative model is trained for each phone, with explicit mechanisms for handling variability due to changes in speaker, phonetic context or word-position. We refer to the Kaldi documentation for further detail 6 . Input to the models takes the form of 39 MFCC coefficients 7 plus 9 pitch-related features 8 extracted every 10ms of signal.

These 48-dimensional input features can be seen as a universal auditory-like baseline representation that is not tuned to any particular 'native language'. The model produces 'native' representations under the form of output vectors produced every 10ms, which list the posterior probabilities, according to the model, that the corresponding stretch of speech signal belongs to each of the segment in the phonemic inventory of the model's 'native language' 9 . The test set of each corpus is decoded with each of the 5 ASR models and we also use the input features directly, without any GMM-HMM decoding, as a language-independent control, yielding a total of 6 different representations of each corpus to be evaluated.

Table 1. Word-Error-Rates obtained by the ASR systems trained on each corpus as well as the language, total duration, speech register and number of speakers for each corpus. AE stands for American English, Spont. stands for Spontaneous.

Corpus Language

Machine ABX evaluation

We evaluate our ASR models with a machine version of an ABX discrimination task [START_REF] Schatz | Evaluating speech features with the minimal-pair ABX task: Analysis of the classical MFC/PLP pipeline[END_REF][START_REF] Schatz | ABX-Discriminability Measures and Applications[END_REF] that allows us to quantify how easy it is to distinguish two phonetic categories based on representations produced by one of our models.

The basic idea is to take two acoustic realizations A and X from one of the phonetic categories and one acoustic realization B from the other category and to test whether the model representation for X is closer to the model representation for A than to the model representation for B. The probability for this to be false for A, B and X randomly chosen in a corpus is defined as the ABX error rate for the two phonetic categories according to the model. If it is equal to 0, the two categories are perfectly discriminated. If it is equal to .5, discrimination is at chance level.

For each A, B and X triplet, we use the phone-level time alignments to select corresponding model representations. Because the stimuli have variable durations, the resulting representations can have different lengths. To find a good alignment and obtain a quantitative measure of dissimilarity between A and X and B and X, we use Dynamic Time Warping based on a frame-wise symmetric Kullback-Leibler divergence for posterior probability vectors and a frame-wise cosine distance for the input features control. In the specific ABX task considered here, we select only triplets such that A, B and X occur in the same phonetic context (same preceding phone and same following phone) and are uttered by the same speaker. For each phonetic contrast an aggregated ABX error rate is obtained by averaging over stimulus order, context and speaker. Let us illustrate this through the example of the /u/-/i/ contrast. First, we average error rates obtained when A and X are chosen to be /u/ and B is chosen to be /i/ and viceversa, then we average over all possible choices of speaker and finally we average over all possible choices of preceding and following phones. We either report directly the scores obtained for individual phonetic contrasts or we average them over interesting classes of contrasts, such as consonant contrasts or vowel contrasts.

Note that, because we are studying very robust empirical effects that reflect what subjects learn outside the lab and that are expected to be observed in any welldesigned experimental task, our evaluation method focus on simplicity of application rather than detailed modeling of human performance in a specific experimental setting.

Results

See supplementary material for the raw (unanalyzed) confusion matrices obtained for each model on each test corpus. Native phonetic categories are easier to distinguish than non-native categories [START_REF] Gottfried | Effects of consonant context on the perception of french vowels[END_REF]. This is consistent with the predictions of our models shown in Figure [START_REF] Best | A direct realist view of cross-language speech perception. Speech Perception and Linguistic Experience: Issues in Cross-Language Research[END_REF] Quantitative models of phonetic category perception Page 7 Schatz, JASA-EL 1. The AE models (in red) separate AE phonetic categories better than other models (in blue). This is true even when they are tested with AE stimuli from a corpus different from the one on which they were trained, showing that the differences observed cannot be explained simply by channel-mismatch effects and reflect a true language-specificity of the representations learned by the models. Another interesting observation is that, while a moderate improvement in phone separability is observed when comparing 'native' AE models to the 'universal' input features control, the most salient effect is a large decrease in performance for 'non-native' models. A possible interpretation is that, while ASR models can provide categorical representations of 'native' speech that are much more compact than the input features, they do it at the expense of a loss of representation power for coding speech in other languages 10 .

Native-language-specific confusion patterns

WSJ BUC CSJ GPM GPV Input feat.

  , a Linear Discriminant Analysis model was trained to classify AE vowels from F1/F2/F3 formant plus duration representations. The classification of North German vowels by this model was then compared to assimilation patterns from a phoneme classification task performed by native AE speakers exposed to North German vowels. The model's predictions only partially matched observed human behavior. In the second study (Gong et al., 2010), Hidden-Markov-Models (HMM) with a structure inspired from ASR technology were trained to classify Mandarin consonants from Mel-Frequency Cepstral Coefficients 2 (MFCC). The classification of AE consonants by this model was then compared to assimilation patterns from a phoneme classification task performed by native Mandarin speakers exposed to AE consonants. There was a good consistency between model's predictions and human assimilation patterns in most cases, although the model provided more variable answers overall and differed markedly from humans in its preferred Mandarin classification of certain AE fricatives.The present work expands over these previous studies in several respects. First, we replace ad hoc speech processing models trained on restricted stimuli 3 with generalpurpose ASR systems trained on natural continuous speech. This has both conceptual and practical benefits. Conceptually, the information processing problem our models attempt to solve is closer to the one solved by humans, who have to deal with the full variability of natural speech. From a practical point of view, this allows us to capitalize on existing corpora of annotated speech recordings developed for ASR. A second difference with previous studies is that we improve on the evaluation methodology, by replacing informal analysis of assimilation patterns with quantitative evaluations based on a simple model of an ABX discrimination task, leading to clean and clearly interpretable results. Finally, we conduct more systematic evaluations, testing for two global and one local effect in cross-linguistic phonetic category perception.

Consonants

Fig. 1 .

 1 Fig. 1. (color online) ABX error-rates averaged over all consonant contrasts of AE. Left: using stimuli from the WSJ corpus test set. Right: using stimuli from the BUC corpus test set.

Acknowledgments

The authors would like to thank Naomi Feldman, Alex Cristia and anonymous reviewers for helpful comments on earlier versions of this manuscript. The research leading to these results received funding from the European Research Council under the FP/2007-2013 program / ERC Grant Agreement n. ERC-2011-AdG-295810 BOOTPHON, from the Agence Nationale pour la Recherche (ANR-2010-BLAN-1901-1 BOOTLANG, ANR-10-0001-02 PSL*, ANR-10-LABX-0087 IEC), from the Fondation de France and from the National Science Foundation (BCS-1734245). This paper was written in part while Emmanuel Dupoux was a visiting researcher at Facebook AI Research, Paris and while Thomas Schatz was a visiting researcher in the Computational Psycholinguistics Laboratory at the Massachusetts Institute of Technology.

Consonants

Consonants

Fig. 2. (color online) Two-dimensional embeddings of the different models based on the average cosine similarity between their patterns of ABX errors across the five test corpora. The distance between models in the embedding space directly reflects whether they make the same type of confusions or not. Left: for consonant contrasts. Right: for vowel contrasts. Text labels are centered horizontally and vertically on the point they represent.

The specific confusions we make between sounds of a foreign language differ according to our native language [START_REF] Strange | Speech perception and linguistic experience: Issues in cross-language research[END_REF]. Consistent with this effect, Figure 2 shows that, for both consonant and vowel contrasts, the confusion patterns obtained with the two AE models over the different corpora are more similar to each other than to the confusion patterns obtained with models trained on other languages. Confusion patterns for input features occupy a somewhat central role. In this figure, the distance between two points is proportional to the observed similarity between confusion patterns obtained from the associated models 11 . Confusion patterns on a given corpus consist of vectors listing the ABX errors for either all consonant contrasts or all vowel contrasts in this corpus. For example for a language with n consonants, n(n -1)/2 consonant contrasts can be formed and the corresponding ABX errors are listed in a vector of size n(n -1)/2. The similarity between confusion patterns of two models is defined as the average of the cosine similarity between the confusion patterns obtained with these models on each of the five corpora 12 . Importantly, the rescaling invariance of the cosine similarity ensures that our analysis of confusion patterns is independent from the average ABX error rates studied in Section 3.1.

Japanese listeners and American English /ô/-/l/

AE /ô/ and /l/ are much harder to perceive for Japanese than for AE native speakers [START_REF] Goto | Auditory perception by normal japanese adults of the sounds l and r[END_REF][START_REF] Miyawaki | An effect of linguistic experience: The discrimination of [r] and [l] by native speakers of japanese and english[END_REF]. Figure 3 shows that our models' predictions are fully consistent with this effect: when comparing the Japanese model to both AE models and to the input features, the /ô/-/l/ discriminability drops spectacularly, much Vietnamese test stimuli (from GPV) Fig. 3. (color online) Comparison of the ABX error-rates obtained with the input features, with the two AE models and with the Japanese model on the AE /ô/-/l/ contrast. ABX Error-rates for the /w/-/j/ contrast and ABX Error-rates averaged over all consonant contrasts of AE are also shown as controls. Left: using stimuli from the WSJ corpus test set. Right: using stimuli from the BUC corpus test set. more than the discriminability of two controls. This is observed both when using test stimuli from the WSJ and from the BUC corpora.The first control is the AE /w/-/j/ contrast. Like /ô/ and /l/, /w/ and /j/ are liquid consonants, but unlike those, they have a clear counterpart in Japanese. The second control is the average ABX error rate from Section 3.1. This control allows to check that there is a specific deficit of the Japanese model on AE /ô/-/l/ discrimination, that cannot be explained by an overall weakness of this model.

Discussion

Fully specified mappings between foreign sounds and native phonetic categories were obtained for several language pairs through GMM-HMM ASR systems. Coupled with a simple model of a discrimination task, they successfully accounted for several empirically attested effects in cross-linguistic phonetic category perception by monolingual listeners. This includes two types of global effects: first, that the phonetic categories of a language are overall harder to discriminate for non-native speakers than for native speakers and second, that the pattern of confusions between phonetic categories for non-native speakers is specific to their native language (e.g. native speakers of Japanese do not make the same confusions between phonetic categories of American English than native speakers of French). We also showed that the proposed model can account for a well-known local effect: American English /ô/ and /l/ are very hard to discriminate for native speakers of Japanese.

These results provide a proof-of-concept for the proposed approach to evaluating ASR systems as quantitative models of phonetic category perception. They also show promise regarding the possibility of modeling human phonetic category perception with ASR systems. Yet we do not claim, at this point, to have provided definitive evidence that the particular GMM-HMM ASR systems considered provide the best, or even a particularly 'good', such model. A host of local effects have been documented in the empirical literature on phonetic category perception beyond the one investigated here [START_REF] Strange | Speech perception and linguistic experience: Issues in cross-language research[END_REF][START_REF] Schatz | Native listening: Language experience and the recognition of spoken words[END_REF] and the empirical adequacy of the proposed models with respect to more of these effects will need to be determined before any conclusion can be reached. Effects that are hard to predict from conventional phonological analyses, such as how the phonetic or prosodic context can modulate the difficulty of perceiving certain foreign contrasts [START_REF] Levy | Effects of consonantal context on perception of french rounded vowels by american english adults with and without french language experience[END_REF][START_REF] Kohler | Contrastive phonology and the acquisition of phonetic skills[END_REF][START_REF] Strange | Acoustic and perceptual similarity of north german and american english vowels[END_REF], should be of particular interest. Finally, let us underline that we only investigated predictions obtained with one particular ASR architecture. There [START_REF] Best | A direct realist view of cross-language speech perception. Speech Perception and Linguistic Experience: Issues in Cross-Language Research[END_REF] Quantitative models of phonetic category perception Page 9 Schatz, JASA-EL are multiple ways of instantiating ASR systems, which might yield different predictions. For example, modeling variability in the signal due to the phonetic context explicitly with context-dependent phone models, as in this article, or implicitly with context-independent phone models, might affect predictions regarding the aforementioned context-dependent effects. Another example of a potentially significant decision is whether to use HMM-GMM or neural-network systems. HMM models have known structural limitations for modeling segment duration [START_REF] Pylkkönen | Duration modeling techniques for continuous speech recognition[END_REF], from which neural-network models do not suffer. Thus, neural-network ASR systems may provide better models of native perception in languages like Japanese, where duration is contrastive. The multiplicity of documented empirical effects and available computational models calls for an extensive investigation, which could in turn trigger a more systematic experimental investigation of non-native perception and result in applications in foreign language education.

Notes 1 Best 1995 being a possible exception.

2 MFCC [START_REF] Mermelstein | Distance measures for speech recognition, psychological and instrumental[END_REF] are speech features commonly used as a front-end to ASR systems. They can be thought of as moderate-dimensional descriptor (d = 13) of the whole shape of regularly-spaced spectral-slices in a mel-scale log-spectrogram. They are usually taken every 10ms and augmented with their first and second time derivatives to incorporate dynamic information, leading to 100 vector descriptors of dimension d = 39 per second of signal.

3 Previous studies used as training stimuli a limited sample of 264 AE vowels occurring either in

[hVba] context or within a unique carrier sentence [START_REF] Strange | Acoustic and perceptual similarity of north german and american english vowels[END_REF] and 3331 Chinese consonants occurring in isolated VCV context [START_REF] Gong | Towards a quantitative model of mandarin chinese perception of english consonants[END_REF].

4 See https://goo.gl/RsKMA3.

5 Error-rate obtained in a word recognition task using the trained acoustic model with a language model (in our case a word-level bigram estimated from the training set).

6 See http://kaldi-asr.org/.

7 See footnote 1.

8 Pitch features were added because two of the languages considered (Mandarin and Vietnamese) are tonal languages.

9 More specifically, we use Viterbi-smoothed phone-level posteriorgrams obtained with a phone-level bigram language model estimated on the training set of each corpus.

10 Note that [START_REF] Renshaw | A comparison of neural network methods for unsupervised representation learning on the zero resource speech challenge[END_REF] observed a different pattern when testing a neural-network-based ASR system trained on AE on the Xitsonga language: the ' AE-native' model improved Xitsonga phone separability relative to the input features control. There are, at least, two possible interpretations for this discrepancy: it could be due to general differences between GMM-HMM and neural-network architectures or it could be due to differences in the representation format chosen (they used 'bottleneck features' extracted from a middle layer of the neural network, which are not constrained to represent phonetic categories, while our posterior features are)

11 Two-dimensional embeddings are obtained with scikit-learn's non-metric multi-dimensional-scaling.

12 Observed range of cosine similarities: [0.90-0.96] for consonants and [0.85-0.94] for vowels.