Prénom Nom

Natalia Tomashenko

Mercedes, Sahar, Kévin, Salima Anthony R Carole

Etienne, Grégor, Sylvain, Walid, Mélanie Antoine Dominique

Marie Adrien Fethi

Florent, Daniel Nathalie Ozan

Amira Loïc

Emmanuelle Abdessalam

Bruno

Hakim Simon

I Also

Yuri N Matveev

Jean-François Bonastre

Denis Jouvet

BP Backpropagation BRNN Bidirectional Recurrent Neural Network CAT Cluster Adaptive Training CD Context-Dependent CDF Cumulative Distribution Function CE Cross-Entropy CI Context-Independent CMLLR Constrained Maximum Likelihood Linear Regression CMN Cepstral Mean Normalization CMVN Cepstral Mean and Variance Normalization CNN Convolutional Neural Network CTC Connectionist Temporal Classification DB

Keywords: adaptation au locuteur, apprentissage adaptatif au locuteur (SAT), réseaux de neurones profonds, modèles de mélanges Gaussiens speaker adaptation, speaker adaptive training (SAT), deep neural network (DNN), Gaussian mixture model

GMM), paramètres acoustiques dérivés de GMM (GMMD), reconnaissance automatique de la parole (RAP), modèles acoustiques, apprentissage profond

Title of thesis : Speaker adaptation of deep neural network acoustic models using Gaussian mixture model framework in automatic speech recognition systems

Résumé

Les différences entre conditions d'apprentissage et conditions de test peuvent considérablement dégrader la qualité des transcriptions produites par un système de reconnaissance automatique de la parole (RAP). L'adaptation est un moyen efficace pour réduire l'inadéquation entre les modèles du système et les données liées à un locuteur ou un canal acoustique particulier. Il existe deux types dominants de modèles acoustiques utilisés en RAP : les modèles de mélanges gaussiens (GMM) et les réseaux de neurones profonds (DNN). L'approche par modèles de Markov cachés (HMM) combinés à des GMM (GMM-HMM) a été l'une des techniques les plus utilisées dans les systèmes de RAP pendant de nombreuses décennies. Plusieurs techniques d'adaptation ont été développées pour ce type de modèles. Les modèles acoustiques combinant HMM et DNN (DNN-HMM) ont récemment permis de grandes avancées et surpassé les modèles GMM-HMM pour diverses tâches de RAP, mais l'adaptation au locuteur reste très difficile pour les modèles DNN-HMM. L'objectif principal de cette thèse est de développer une méthode de transfert efficace des algorithmes d'adaptation des modèles GMM aux modèles DNN. Une nouvelle approche pour l'adaptation au locuteur des modèles acoustiques de type DNN est proposée et étudiée : elle s'appuie sur l'utilisation de fonctions dérivées de GMM comme entrée d'un DNN. La technique proposée fournit un cadre général pour le transfert des algorithmes d'adaptation développés pour les GMM à l'adaptation des DNN. Elle est étudiée pour différents systèmes de RAP à l'état de l'art et s'avère efficace par rapport à d'autres techniques d'adaptation

Introduction

Automatic speech recognition (ASR) is the technology that enables human-machine interaction by allowing human beings to speak with a computer interface. ASR has been an active research area for decades. The progress in the development of ASR technology achieved in the last years increased the use of different ASR systems in everyday life. Interactive voice response (IVR) systems, information extraction and retrieval, automatic closed captioning, dictation, transcription of recorded speech, language learning systems, speech-to-speech translation systems (such as Skype translator ), virtual personal assistant devices (such as Apple's Siri, Google Assistant, Amazon Alexa, Microsoft Cortana, Facebook's M), and many other ASR applications are becoming an integral part of our lives. This progress has become possible largely due to the recent advances in deep learning research [START_REF] Goodfellow | Deep Learning[END_REF], which represents now the mainstream direction for speech recognition development [START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF]. Two key factors have contributed to this process: (1) substantial increase of computational power of multi-core processors, general purpose graphical processing units (GPGPUs), and GPU clusters, which allow to train more complex models with a greater number of parameters, and (2) access to more training data.

Despite all these advances, ASR systems are still domain dependent and usually can show high performance only if they are designed for a specific task or environment. Any mismatch between training and testing conditions may degrade the performance. In particular, this mismatch can be caused by different acoustic conditions (such as different speakers, recording channels, background noises, etc.). To overcome this problem, acoustic adaptation is typically applied.

The aim of acoustic model (AM) adaptation is to reduce mismatches between training and testing acoustic conditions and improve the accuracy of the ASR system for a target speaker or channel, using a limited amount of adaptation data from the target acoustic source. This thesis focuses mainly on speaker adaptation, which is aimed to reduce the mismatch caused by inter-speaker variability. Nevertheless, the developed approaches are applicable to a wider range of adaptation tasks.

Adaptation of deep neural network (DNN) AMs is a rapidly developing research area. In the recent years, DNNs have replaced conventional Gaussian mixture models (GMMs) in most state-of-the-art ASR systems, because it has been shown that DNN Hidden Markov Models (HMMs) outperform GMM-HMMs in different ASR tasks [Hinton et al., 2012a]. Many adaptation algorithms that have been developed for GMM-HMM systems, such as maximum a posteriori adaptation (MAP) [START_REF] Gauvain | Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains[END_REF], maximum likelihood linear regression (MLLR) [START_REF] Gales | Maximum likelihood linear transformations for HMM-based speech recognition[END_REF][START_REF] Leggetter | Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models[END_REF] and others [START_REF] Shinoda | Speaker adaptation techniques for automatic speech recognition[END_REF][START_REF] Woodland | Speaker adaptation for continuous density HMMs: A review[END_REF], cannot be easily applied to DNNs because of the different nature of these models.

Among the adaptation algorithms developed for DNNs, only a few take advantage of robust adaptability of GMMs (Chapter 4). The most common way of using GMMs for DNN model adaptation is through GMM-adapted features. For example, acoustic features adapted with feature-space maximum likelihood linear regression (fMLLR) technique are used as input for DNN training in [START_REF] Kanagawa | Feature-space structural maplr with regression tree-based multiple transformation matrices for DNN[END_REF][START_REF] Parthasarathi | fMLLR based feature-space speaker adaptation of DNN acoustic models[END_REF][START_REF] Rath | Improved feature processing for deep neural networks[END_REF]Seide et al., 2011a]. In [START_REF] Lei | Deep neural networks with auxiliary Gaussian mixture models for real-time speech recognition[END_REF] likelihood scores from DNN and GMM models, both adapted in the feature space using the same fMLLR transform, are combined at the state level during decoding. A temporally varying weight regression (TVWR) is explored in [START_REF] Liu | On combining DNN and GMM with unsupervised speaker adaptation for robust automatic speech recognition[END_REF], where DNN posteriors are transformed into time-varying scaling factors for Gaussian weights, using a regression model. However, none of these approaches suggests a universal method to transfer adaptation algorithms from GMM models to DNNs.

The main purpose of this thesis is to develop a framework for efficient transfer of all adaptation algorithms, developed for GMM AMs, to DNN AMs. To achieve this goal, we proposed to use so-called GMM-derived features (GMMD) as input to a DNN [START_REF] Tomashenko | Speaker adaptation of context dependent deep neural networks based on MAP-adaptation and GMM-derived feature processing[END_REF]. Then, the proposed adaptation algorithm was extended to the concept of speaker adaptive training (SAT) for DNNs [Tomashenko and Khokhlov, 2015].

The desirable property of an adaptation algorithm is flexibility. That means that adaptation makes use of all available adaptation data: it improves the speech recognition accuracy even with a small amount of adaptation data, and, when the amount of adaptation data increases, speech recognition accuracy also continues to improve, asymptotically approaching the accuracy of matched (in our case, speaker-dependent (SD)) AM [START_REF] Shinoda | Speaker adaptation techniques for automatic speech recognition[END_REF]. The effectiveness of the proposed adaptation approach was explored using the most common adaptation algorithms for GMM-HMM -MAP and fMLLR adaptation, as an example. We mostly focus on MAP adaptation in the experiments, because it can provide an additional flexibility to DNN adaptation, as it is not restricted to a single transform, as fMLLR.

Different ways for adaptation performance improvement, such as using confidence scores [START_REF] Tomashenko | On the use of Gaussian mixture model framework to improve speaker adaptation of deep neural network acoustic models[END_REF], data selection and data augmentation strategies [START_REF] Tomashenko | Exploring GMM-derived features for unsupervised adaptation of deep neural network acoustic models[END_REF], were proposed and investigated in this thesis.

Starting our research from a classical DNN architecture and simple auxiliary GMM model trained for GMM-derived feature extraction, we are interested in the following questions:

• The first question is how, in terms of topology and basic features, to more effectively train an auxiliary GMM model, which is used for GMM-derived features, in order to achieve better DNN adaptation performance [START_REF] Tomashenko | On the use of Gaussian mixture model framework to improve speaker adaptation of deep neural network acoustic models[END_REF]].

• The second question concerns the way of efficient integration of GMM-derived features into neural network architectures of state-of-the-art ASR systems [Tomashenko et al., 2016a]. Most of these systems already use normalization and speaker adaptation techniques. How the best improvement over these systems can be obtained using the proposed adaptation approach?

• Is the proposed technique complementary to other widely used algorithms for DNN adaptation, such as fMLLR or i-vectors?

• And finally, can the other more advanced neural network architectures, such as timedelay neural networks (TDNN), recurrent neural networks (RNN), and others, which nowadays have become dominant in state-of-the art ASR systems, also benefit from the proposed adaptation technique? We are particularity interested in end-to-end deep AMs (Chapter 9), because end-to-end systems are an important trend in current ASR technology.

In addition, we aim to look more deeply into the nature of the GMMD features and adaptation techniques associated with them to better understand their properties, strengths and weaknesses and the potential for improvement (Chapter 10).

• Chapter 3 gives an overview of HMM-based ASR systems, describes their principal components (including feature extraction; acoustic, pronunciation and language modeling; decoding techniques) with the main focus on acoustic modeling part. Two types of AMs are described, differing in the way they model the state probability distribution: HMM-GMM and hybrid HMM-DNN models. In addition, end-to-end ASR systems are reviewed, as they represent an important trend in current ASR technology. Performance evaluation for ASR systems is described in the final section.

• Chapter 4 reviews speaker adaptation techniques for both GMM-HMM and DNN-HMM acoustic models.

• Chapter 5 describes the speech corpora and language models (LM) used to carry out the experiments. Experimental results in each chapter follow theoretical descriptions.

• Chapter 6 introduces a GMM framework for training DNNs and provides preliminary experimental results with supervised speaker adaptation.

• Chapter 7 extends the scheme for GMM-derived feature extraction by applying a concept of SAT. It can be considered as consisting of two main parts, representing independent ideas: (1) Section 7.1 and ( 2) the rest of this chapter (Sections: 7.2, 7.3 and 7.4). Each of these parts has a corresponding section with experimental results.

In the first part, a SAT procedure for DNN training is presented. In the experiments for this chapter we explore the effectiveness of MAP and fMLLR adaptation, as well as their combination in the proposed approach for adapting an auxiliary GMM model, used for GMMD feature extraction. We look at the dependence of the adaptation behavior for different algorithms on the amount of adaptation data.

In the second part, several techniques for adaptation performance improvement are proposed and studied: using lattice scores, data augmentation and data selection. For this techniques we used a different experimental setup, with larger and more complex models.

For this chapter as well as for all the following ones experimental results are reported for unsupervised adaptation mode.

• Chapter 8 investigates various ways of integrating GMM-derived features into different state-of-the-art neural network architectures (DNN and TDNN). To build a stronger ASR system, we took as a basis conventional Kaldi [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF] recipes for AMs and aim to integrate our adaptation algorithm in these recipes. We perform this integration in different ways: feature concatenation of the proposed GMMD features with conventional features for training DNNs, fusion of posterior from different DNNs, lattice-level fusion of recognition results, and others. Also we compare the proposed approach with the most common feature-space adaptation techniques, such as fMLLR and i-vectors, for DNN and TDNN AMs. In addition, in Section 8.7 we report some results of applying the proposed GMMD features in the MGB Challenge 20161 as a part of the LIUM ASR system [START_REF] Tomashenko | LIUM ASR systems for the 2016 multi-genre broadcast Arabic challenge[END_REF]].

• Chapter 9 explores the effectiveness of the proposed adaptation technique in application to end-to-end AMs, taking as an example bidirectional long short term memory (BLSTM) acoustic models trained with connectionist temporal classification (CTC) criterion. Three different speaker adaptation algorithms have been implemented to this type of AMs and experimentally analyzed: (1) fMLLR adaptation, (2) adaptation using i-vectors, and

(3) the proposed algorithm with MAP adaptation using GMMD features. Furthermore, a comparative study of the adaptation techniques was conducted for CTC AMs and TDNN AMs trained with the traditional frame-wise cross-entropy criterion.

• Chapter 10 analyzes properties of the proposed GMM-derived features and adaptation algorithm. For this analysis, we use phoneme posterior based (PPB) features, obtained from decoding lattices. Visual t-distributed stochastic neighbor embedding (t-SNE) analysis [START_REF] Maaten | Visualizing data using t-SNE[END_REF] is performed for different phoneme groups. Davies-Bouldin (DB) index [START_REF] Davies | A cluster separation measure[END_REF] and other statistics are also used in this study.

• Chapter 11 contains a summary and discusses possible future work.

Chapter 2

Deep neural networks

This chapter introduces deep neural networks (DNNs) and provides an overview of some of the most common neural network architectures used in ASR.

Introduction

In the ASR literature, the term deep neural network (DNN) was originally referred to as feed-forward artificial neural network (or multilayer perceptron (MLP) [START_REF] Rosenblatt | Principles of neurodynamics. Perceptrons and the theory of brain mechanisms[END_REF]) with more than one hidden layer [Hinton et al., 2012a;[START_REF] Seide | Conversational speech transcription using contextdependent deep neural networks[END_REF]. Later this term was extended to the meaning of any neural network with a deep structure [START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF]. In this thesis we will use the term DNN mainly when referring to feed-forward models and explicitly specify the type of the architecture (feed-forward, convolutional, recurrent, etc.), when it is necessary.

DNNs play an important role in modern ASR systems, particularly in acoustic modeling. It was shown, that for various ASR tasks, DNN AMs outperform traditional GMM AMs [Hinton et al., 2012a].

Further in this chapter we review a feed-forward DNN architecture, training procedure including some practical considerations, and different alternative types of neural network architecture that nowadays are used in ASR systems.

Deep neural network architecture

The architecture of a conventional DNN can be described as follows [Hinton et al., 2012a]. Let denote the l-th layer of a DNN as h l , and the total number of layers in a DNN as L + 1, so that layers have indexes: 0, . . . , L. Then

h l = f l (z l ) = f l W l h l-1 + b l for 0 < l ≤ L, (2.1)
where

                                       z l = W l h l-1 + b l ∈ R N l -excitation vector, h l ∈ R N l -activation vector, W l ∈ R N l ×N l-1 -weight matrix, b l ∈ R N l -bias vector,
N l -number of neurons in layer l (or layer dimension), (2.2)

h 0 = o ∈ R N 0 -input
Input vector

Hidden layers

Output vector The activation function f l is applied element-wise to the excitation vector. Let z i be i-th element of vector z of linear activations, obtained from (2.2). For simplicity, we omit layer index l in the notation. There are several types of activation functions, among which the most commonly used are:

• Identity function:

I(z i ) = z i . (2.3)
Identity function is usually used in the output layer for regression tasks.

• Sigmoid (standard logistic) function:

σ (z i ) = 1 1 + e -z i .
(2.4)

• Hyperbolic tangent function:

tanh(z i ) = e z ie -z i e z i + e -z i .

(2.5)

Hyperbolic tangent function is connected with the sigmoid function: tanh(z) = 2σ (2z) -1 and is different from the sigmoid function in the output range of values: tanh(z) ∈ (-1, 1), while σ (z) ∈ (0, 1). So the tanh function is symmetric around zero, and is recommended, for example in [START_REF] Lecun | Efficient backprop[END_REF], for more efficient training.

• Rectified linear unit (ReLU) function:

ReLU(z i ) = max(0, z i ).

(2.6)

The ReLUs [START_REF] Jaitly | Learning a better representation of speech soundwaves using restricted Boltzmann machines[END_REF][START_REF] Nair | Rectified linear units improve restricted Boltzmann machines[END_REF] create sparse representations in a DNN structure. They are efficient in combination with dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] regularization techniques [START_REF] Dahl | Improving deep neural networks for LVCSR using rectified linear units and dropout[END_REF].

• Maxout function [START_REF] Goodfellow | Maxout networks[END_REF]:

maxout {z i } i∈R = max ({z i } i∈R ) , (2.7)
where R is the number of linear activations. A single maxout unit can be considered as a piecewise linear approximation to any convex function, and maxout network with two hidden units can approximate arbitrarily well any continuous functions.

• Softmax function [START_REF] Bridle | Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition[END_REF]:

softmax(z i ) = e z i ∑ N l j=1 e z j .
(2.8)

• L p -norm function [START_REF] Boureau | A theoretical analysis of feature pooling in visual recognition[END_REF][START_REF] Gulcehre | Learned-norm pooling for deep feedforward and recurrent neural networks[END_REF][START_REF] Zhang | Improving deep neural network acoustic models using generalized maxout networks[END_REF]:

{z i } i∈R p = ∑ i∈R |z i | p 1 p , (2.9)
where p ≥ 1 is the order of the norm, and R is the set of linear activations, for which the norm is calculated.

In this thesis we will apply sigmoid (2.4) and ReLU (2.6) activation functions for hidden layers in different DNN-HMM AM setups, and the softmax activation function (2.9) will be used for the output layer.

The type of the output layer depends on the task:

• For the regression task it is a linear layer (with the identity activation function):

h L = z L = W L h L-1 + b L .
(2.10)

• For the multi-class classification task each output neuron represents a class i ∈ 1, . . . , N L , and the value of i-th output neuron h L i represents the probability P DNN (i|o) that vector o belongs to class i. To simulate a probability distribution, the softmax function (2.9) is used:

h L i = P DNN (i|o) = softmax(z i ).
(2.11)

Training

Let y be an output vector of a DNN, corresponding to an observation vector o. As before (see Formula (2.2)), [W, b] denotes a set of DNN parameters to be estimated from training samples T = {(o m , y m ) : 0 ≤ m ≤ M}, where o m is the m-th observation vector, and y m is the corresponding target vector. The process of [W, b] parameter estimation (or DNN training) is characterized by a training criterion and a learning algorithm.

Training criteria

In order to estimate parameters of a DNN, a suitable training criterion (or a loss function) should be specified. Two aspects should be taken into account when choosing a loss function F:

1. Simplicity of evaluation.

2. Correlation with the final goal of the task.

Several popular training criteria are used for DNN training:

• Mean square error (MSE):

F MSE (W, b; T) = 1 M M ∑ m=1 F MSE (W, b; o m , y m ), (2.12)
where .13) This criterion is used for regression tasks.

F MSE (W, b; o, y) = 1 2 ∥h L -y∥ 2 = 1 2 h L -y T h L -y . ( 2
• Cross-entropy (CE):

F CE (W, b; T) = 1 M M ∑ m=1 F CE (W, b; o m , y m ), (2.14)
where

F CE (W, b; o, y) = - N L ∑ i=1 y i log h L i , (2.15)
where y i = P target (i|o) is the observed in the training set empirical probability that the observation vector o belongs to class i, and h L i = P DNN (i|o) is the same probability estimated from the DNN. This criterion is used for the classification tasks, where y is a probability distribution. Minimizing CE criterion is equivalent to minimizing the Kullback-Leibler divergence (KLD) between the empirical probability distribution (of targets) and the probability distribution estimated from the DNN. In many tasks, hard class labels are used as targets: (2.16) where c o is the class label of the observation vector o in the training set. In this case the CE criterion (2.15) becomes the negative log-likelihood criterion (NLL):

y i =    1, if c o = i 0, otherwise
F NLL (W, b; o, y) = -log h L c o .
(2.17)

Learning algorithm

Given the training criterion, the parameters of a DNN [W, b] can be discriminatively trained (DT) with the error backpropagation algorithm (BP) [START_REF] Rumelhart | Learning representations by backpropagating errors[END_REF] by propagating derivatives of a loss function. In the simplest version, the DNN parameters can be improved using the first-order gradient information as follows [START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF]:

W l t+1 ← W l t -ε∆W l t (2.18) and b l t+1 ← b l t -ε∆b l t , (2.19)
where ε is a learning rate; W l t and b l t are the weight matrix and the bias vector of the layer l after the t-th update;

∆W l t = 1 M b M b ∑ m=1 ∇ W l t F(W, b; o m , y m ) (2.20)
and

∆b l t = 1 M b M b ∑ m=1 ∇ b l t F(W, b; o m , y m ) (2.21)
are the average weight matrix gradient and the average bias vector gradient correspondingly at iteration t computed on M b samples and ∇ x F is the gradient of F with respect to x. The parameter updates in Formulas (2.18) and (2.19) are estimated on a batch of training samples. The choice of the batch size influences the final result of the training, as well as the convergence speed. In the simplest approach, which is referred to the batch training, the batch is the whole training set. An alternative approach is to use the stochastic gradient descent (SGD) algorithm [START_REF] Bishop | Pattern recognition and machine learning[END_REF][START_REF] Bottou | Online learning and stochastic approximations[END_REF], where gradients are estimated from a single sample. However, the most common technique is to compute the derivatives on small, randomly chosen mini-batches of the training samples. In this thesis we will use the last approach, which also referred to as the mini-batch SGD algorithm in the literature [Li et al., 2014c].

There are some practical considerations that have to be taken into account for efficient DNN training [START_REF] Ruder | An overview of gradient descent optimization algorithms[END_REF][START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF]. One important question is the choice of the learning rate ε. If a learning rate is too small the learning algorithm has too slow convergence, and on the contrary, if it is too large, it can prevent learning from convergence to optimal solution. Learning rate schedules [START_REF] Darken | Note on learning rate schedules for stochastic optimization[END_REF] aim to regulate the learning rate during the training according to a fixed learning rate schedule, depending on the changes in the objective function, or component-wise, depending on the geometry of the observed data and the parameter sparseness. Popular gradient descent optimization algorithms include [START_REF] Ruder | An overview of gradient descent optimization algorithms[END_REF]: momentum, Nesterov accelerated gradient (NAG) [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate o(1/k2)[END_REF], Adagrad [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF], Adadelta [START_REF] Zeiler | ADADELTA: an adaptive learning rate method[END_REF], adaptive moment estimation (Adam) [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], natural gradient descent [START_REF] Amari | Natural gradient works efficiently in learning[END_REF], and RMSprop [START_REF] Tieleman | Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF].

Batch normalization technique, proposed in [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], allows significant acceleration of the DNN training. It consists in a normalization step that fixes the means and variances of layer inputs.

Below we consider some other important issues related to DNN training procedure, such as momentum, pre-training, and regularization techniques.

Momentum

In order to speed up the training and to smooth the parameter updates, the information about the previous updates is included into the gradient update in the form of momentum [START_REF] Qian | On the momentum term in gradient descent learning algorithms[END_REF][START_REF] Rumelhart | Learning internal representations by error propagation[END_REF]. When the momentum is applied, Formulas (2.18) and (2.19) are replaced with:

W l t+1 ← W l t -ε∆W l t + α∆ t-1 (W l ) (2.22) and b l t+1 ← b l t -ε∆b l t + α∆ t-1 (b l ), (2.23)
where

∆ t (W l ) = W l t+1 -W l t (2.24) and ∆ t (b l ) = b l t+1 -b l t , (2.25)
and 0 < α < 1 is a momentum coefficient (factor).

Regularization

Overfitting can be a serious problem in DNN training due to the large number of estimated parameters. There are different ways to control and prevent over-fitting in DNN training.

One solution to reduce over-fitting is to apply some regularization terms for the DNN parameter updates. Regularization aims to incorporate some prior information into the training criteria, to prevent the model from learning undesirable configurations. The common way is to add a complexity term (penalty) to the loss function to penalizes certain configurations. The most commonly used regularization terms include:

1. L 1 -penalty: R 1 (W) = ∥vec(W)∥ 1 = L ∑ l=1 vec(W l ) 1 = L ∑ l=1 N l ∑ i=1 N l-1 ∑ j=1 |W l i j | (2.26) is based on L 1 -norm. 2. L 2 -penalty: R 2 (W) = ∥vec(W)∥ 2 2 = L ∑ l=1 vec(W l ) 2 2 = L ∑ l=1 N l ∑ i=1 N l-1 ∑ j=1 (W l i j ) 2 (2.27) is based on L 2 -norm.
In Formulas (2.26) and (2.27), W i j denotes the (i, j)-th element in matrix W, and vec(W l ) is the vector, obtained by concatenation all the columns in the matrix W l . The regularization terms are often referred to as weight decay in the literature. When the regularization is applied, the training criterion is changed to:

F reg (W, b; T) = F(W, b; T) + λ R p (W), (2.28)
where p ∈ {1, 2}, depending on the chosen penalty type; and F(W, b; T) is a CE or MSE loss function, that optimizes the empirical loss on the training set T.

Dropout

Dropout is an alternative powerful regularization technique [START_REF] Dahl | Improving deep neural networks for LVCSR using rectified linear units and dropout[END_REF][START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF][START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] which consists in removing a specified percentage of randomly selected hidden units or inputs during training or pre-training to improve generalization. When a hidden neuron is dropped out in training, its activation is set to 0. During the training with dropout, a random subset of units should be repeatedly sampled. This slows down the training process. A fast dropout training algorithm was proposed in [START_REF] Wang | Fast dropout training[END_REF] and is based on using Gaussian approximation instead of doing Monte Carlo optimization.

Pre-training

Pre-training techniques provide an effective way of weight initialization and also can propose a solution for the over-fitting problem [START_REF] Erhan | Why does unsupervised pre-training help deep learning[END_REF]. Various approaches to DNN pre-training include [START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF]]:

• Generative pre-training:

restricted Boltzmann machines (RBMs) [START_REF] Hinton | A practical guide to training restricted Boltzmann machines[END_REF][START_REF] Nair | Rectified linear units improve restricted Boltzmann machines[END_REF][START_REF] Smolensky | Information processing in dynamical systems: Foundations of harmony theory[END_REF]]. An RBM is a stochastic neural network, where binary activations depend on their neighbors and have probabilistic binary activation functions.

-Deep belief networks (DBNs) [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF].

-Denoising autoencoders [START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF].

• Discriminative pre-training (DPT). In DPT, layers can be pre-trained layer-wise using BP as follows. First, a DNN with only one hidden layer is trained discriminatively to convergence using labels. Then, another hidden layer with randomly initialized weights is inserted before the output layer. After that the new network is discriminatively trained again. Each time, when a new hidden layer is added to the network, all layers are updated using BP. This procedure is repeated until the required number of hidden layers is reached.

Alternative neural network architectures 2.4.1 Convolutional neural network (CNN)

Convolutional neural network (CNN) [START_REF] Lecun | Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks[END_REF][START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]] is a neural network architecture designed for data that has a grid-structure topology [START_REF] Goodfellow | Deep Learning[END_REF].

The development of CNNs is claimed to be originally inspired by biological neural processes. Neurophysiologists [START_REF] Hubel | Receptive fields, binocular interaction and functional architecture in the cat's visual cortex[END_REF] explored how neurons in the cat's brain responded to images on a screen in front of the cat. In this experiment they found out that in the primary visual cortex of the brain, some part of cells (so-called, simple cells) strongly respond to specific edge-like patterns, but almost do not respond to other patterns. Another part of cells (complex cells) have wider receptive fields (sub-regions of the visual field, to which the cells are sensitive) and are locally invariant to small changes in the position of the pattern. This phenomenon inspired the development of some pooling strategies in CNNs, such as, maxout units [START_REF] Goodfellow | Maxout networks[END_REF] and different models, such as, NeoCognitron [START_REF] Fukushima | Neocognitron: A hierarchical neural network capable of visual pattern recognition[END_REF], HMAX [START_REF] Serre | Realistic modeling of simple and complex cell tuning in the HMAX model, and implications for invariant object recognition in cortex[END_REF][START_REF] Serre | Robust object recognition with cortex-like mechanisms[END_REF], LeNet [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]]. The modern CNNs were proposed in [START_REF] Lecun | Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks[END_REF][START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF][START_REF] Lecun | Generalization and network design strategies[END_REF] for image processing. The following ideas were introduced into the neural network architecture [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]]: convolution, local receptive fields, shared weights and spatial or temporal subsampling (or pooling) in order to achieve a certain degree of invariance to scale, shift and distortions of the input [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF].

A typical example of a CNN architecture is depicted in Figure 2.2. It contains two convolutional layers (C1 and C2), two subsampling (or pooling) layers (S1 and S2), and several fully-connected layers. The lower-layers are composed of convolution and subsampling (or max-pooling layers). The upper-layers are fully-connected and correspond to a simple MLP with several hidden layers. Units in a layer are organized in planes, within which all the units share the same set of weights. The set of outputs from units in a plane is called a feature map.

A convolutional layer is composed of several feature maps with different sets of weights and biases. Different feature maps extract different types of local features from the input. The input to the first fully-connected layer is the set of all features maps from the lower layer.

In general, a CNN can be described by the following operations:

• Convolution. The convolution operation, denoted as

c(t) = (x * k)(t) = x(τ)k(t -τ)dτ, (2.29)
can be considered in the discrete case as multiplication of input x by a matrix kernal (also called filter) k. The output is referred to as the feature map. In practice input x is usually a multidimensional array of data and kernel k is a multidimensional array of parameters that are trained by the learning algorithm. The kernel typically has a much smaller size than the input.

• Pooling. The pooling operation maps the output at a certain region to a summary statistic of neighboring units in this region. Examples of pooling include: the maxpooling [START_REF] Riesenhuber | Hierarchical models of object recognition in cortex[END_REF][START_REF] Zhou | Computation of optical flow using a neural network[END_REF], average of a rectangular neighborhood, the L p -norm of a rectangular neighborhood [START_REF] Boureau | A theoretical analysis of feature pooling in visual recognition[END_REF], a weighted average based on the distance from the central point, and others [START_REF] Jia | Beyond spatial pyramids: Receptive field learning for pooled image features[END_REF]Swietojanski and Renals, 2016]. Pooling helps to make the representation be approximately invariant to small changes of the input. Also for many tasks, pooling is important for handling inputs of different sizes.

These operations introduce several useful properties into the neural network architecture, that can help to improve a machine learning system: sparse interactions and parameter sharing [START_REF] Goodfellow | Deep Learning[END_REF]. Also they allow a neural network to work with inputs of variable size.

Convolutional networks play an important role in the history of neural networks. In the past they were applied to many tasks including handwriting recognition [START_REF] Bottou | Comparison of classifier methods: a case study in handwritten digit recognition[END_REF], on-line handwritten word recognition [START_REF] Bengio | LeRec: A NN/HMM hybrid for on-line handwriting recognition[END_REF], face recognition [START_REF] Lawrence | Face recognition: A convolutional neural-network approach[END_REF] and others. In image processing, a deep CNN was used to win the ImageNet image classification challenge [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], and more recently, the 152-layer residual network (ResNet)1 has been introduced [START_REF] He | Deep residual learning for image recognition[END_REF] to significantly reduce the ImageNet classification error rate.

In recent years, CNNs have become widely used in various state-of-the-art applications, for example, in computer vision: [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF][START_REF] Lecun | Convolutional networks and applications in vision[END_REF][START_REF] Szegedy | Going deeper with convolutions[END_REF], in speech technology: [START_REF] Chen | Tone classification in mandarin chinese using convolutional neural networks[END_REF][START_REF] Li | Combining CNN and BLSTM to extract textual and acoustic features for recognizing stances in mandarin ideological debate competition[END_REF][START_REF] Manenti | CNN-based phone segmentation experiments in a less-represented language[END_REF], and particularly, in ASR: [Abdel-Hamid et al., 2013, 2012;Deng et al., 2013a;[START_REF] Ghahremani | Acoustic modelling from the signal domain using CNNs[END_REF]Sainath et al., 2013a,b;[START_REF] Suzuki | Domain adaptation of CNN based acoustic models under limited resource settings[END_REF]Swietojanski et al., 2014;[START_REF] Zhang | Towards end-to-end speech recognition with deep convolutional neural networks[END_REF], CNNs with a very deep VGG 2 network architecture [START_REF] Sercu | Very deep multilingual convolutional neural networks for LVCSR[END_REF][START_REF] Yu | Deep convolutional neural networks with layer-wise context expansion and attention[END_REF]. In ASR systems, for acoustic modeling, the convolution can be done either in time [START_REF] Yoshioka | Far-field speech recognition using CNN-DNN-HMM with convolution in time[END_REF] or in frequency domain [START_REF] Abdel-Hamid | Convolutional neural networks for speech recognition[END_REF], or in both [START_REF] Mitra | Time-frequency convolutional networks for robust speech recognition[END_REF]. Applying convolution in the time domain allows a neural network to be invariant to shifts in time. Using convolution in the frequency domain makes a neural network invariant to frequency.

Time-delay neural network (TDNN)

Time-delay neural networks (TDNNs) belong to a particular type of CNNs that share weights along a single temporal dimension. Initially TDNN models have been proposed for the phoneme recognition task in [START_REF] Lang | A time-delay neural network architecture for speech recognition[END_REF][START_REF] Waibel | Phoneme recognition using time-delay neural networks[END_REF], and later they were applied for spoken word recognition [START_REF] Bottou | Speaker-independent isolated digit recognition: Multilayer perceptrons vs. dynamic time warping[END_REF]] and for on-line handwriting recognition [START_REF] Guyon | Design of a neural network character recognizer for a touch terminal[END_REF] tasks. TDNNs allow the acoustic model to learn the temporal dynamics of the speech signal using short term acoustic feature vectors. Recently AMs with the TDNN topology have been shown to outperform state-of-the-art DNN-HMM ASR systems for many tasks [START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF][START_REF] Povey | Purely sequence-trained neural networks for asr based on lattice-free mmi[END_REF]. Each frame of the higher level corresponds to a longer context than the lower layers. This hierarchical temporal context expansion is different from the using of a wide contextual window of acoustic feature vectors in the input layer, as it is done in a DNN model. Different layers in the TDNN model correspond to different levels of abstraction in information, extracted from the speech signal: local patterns in speech signal can be captured by the lower layers, while more complex structures can be learned by higher levels.

Acoustic vectors

In the given example (Figure 2.3), each layer has its own context extension and subsampling characteristics. For example, the first layer operates on the window of 5 frames {o t-2 , o t-1 , o t , o t+1 , o t+2 } of the input features. We will denote it as {-2, -1, 0, 1, 2} (or simply, [-2, 2]) following the notations from [START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF][START_REF] Povey | The Kaldi speech recognition toolkit[END_REF]. Layer 2 operates on the window of 4 frames of the Layer 1: [-1, 2]. In addition, Layer 2 has a sub-sampling, so it utilizes only boundary frames {-1, 2} from context window [-1, 2]. The frame connections that are used in TDNN after sub-sampling are shown in the figure with solid lines (the light dotted lines correspond to frame connections without subsampling). Finally, we can see, that the top layer has an indirect connection to the input acoustic vectors layer by means of context window [-17, 12].

This type of neural network will be used for AMs in our experiments. The layer-wise context expansion in time dimension has been explored also in [START_REF] Amodei | Deep speech 2: End-to-end speech recognition in English and Mandarin[END_REF][START_REF] Yu | Deep convolutional neural networks with layer-wise context expansion and attention[END_REF]. Sub-sampling and context extension in the network was also used in stacked bottle-neck networks [START_REF] Grézl | Adaptation of multilingual stacked bottle-neck neural network structure for new language[END_REF], but in that setup both neural networks were trained separately.

Recurrent neural network (RNN)

Recurrent neural networks (RNNs) is a type of neural networks developed to process sequential data. RNNs provide a powerful extension of feed-forward DNN models by adding connections between different type of units, regardless from their positions within the neural network topology. Possible connections include backward connections to previous layers, self-recurrent loops of units and others. A directed cycle is a basic type of connections between different units in RNN models. The use of recurrence over the temporal dimension allows RNNs to model the dynamic temporal behavior.

Early important types of RNNs, suggested in the literature, include:

• Jordan RNN [START_REF] Jordan | Attractor dynamics and parallelism in a connectionist sequential machine[END_REF][START_REF] Jordan | Serial order: A parallel distributed processing approach[END_REF]: contains recurrent connections that allow the network's hidden units to see its own previous output, so that the subsequent behavior can be formed by the previous responses (Figure 2.4).

• Elman RNN or simple recurrent networks (SRNs) [START_REF] Elman | Finding structure in time[END_REF]: has additional context units that come from the hidden layer and are augmented with the input layer (Figure 2.5).

Input vector

Hidden layer

Output vector

Context vector The main difference of RNNs from DNNs consists in the fact that RNNs operate not only on input vectors, but also on internal states of the model. The internal states of the RNN encode the information about the temporal sequence of the past process that was already processed by this RNN. A simple RNN with a single hidden layer can be described with the observation and state equations as follows: The term W yh y t-1 in Formula (2.30) is often omitted. For training RNN models, a backpropagation-through-time (BPTT) learning algorithm is typically used [START_REF] Werbos | Backpropagation through time: what it does and how to do it[END_REF]. However, in practice, training RNNs to learn long-term temporal dependencies can be difficult due to the vanishing and exploding gradient problems [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF]].

   h t = f (W oh o t + W hh h t-1 + W yh y t-1 + b h ) y t = g(W hy h t + b y ), (2.30) where                                                          o t ∈ R D -vector of inputs, h t ∈ R N -

Bidirectional recurrent neural network (BRNN)

In many applications the output prediction of y t may depend not only on the information about the past sequence {o 1 , . . . , o t-1 }, but also on the future sequence {o t+1 , . . . , o T }. In order to capture information from the whole input sequence, the bidirectional RNN (BRNN) architecture was proposed [START_REF] Schuster | Bidirectional recurrent neural networks[END_REF]. In BRNNs, data are processed in two directions with two hidden layers, which are then inputted further to the same output layer. As shown in Figure 2.6, a recurrent forward hidden layer of BRNN -→ h computes sequence of hidden outputs for t = 1, . . . , T , and an additional recurrent layer ←h computes the backward sequence of hidden outputs for t = T, . . . , 1:

         - → h t = f (W o - → h o t + W-→ h - → h - → h t-1 + b-→ h ), ← - h t = f (W o ← - h o t + W← - h ← - h ← - h t-1 + b← - h ), y t = g(W-→ h y - → h t + W← - h y ← - h t + b y ).
(2.31)

Input vectors

Backward hidden layers

Output vectors

Forward hidden layers

-1 +1 -1 +1 -1 +1 -1 +1 Figure 2.6 Bidirectional recurrent neural network
BRNNs become very successful in many applications [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF], such as handwriting recognition [Graves and Schmidhuber, 2009] and ASR [START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF]. In ASR domain, BRNN architecture underlies many end-to-end systems [START_REF] Graves | Towards end-to-end speech recognition with recurrent neural networks[END_REF][START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF][START_REF] Hannun | Deep speech: Scaling up end-to-end speech recognition[END_REF]Miao et al., 2015a].

Long short term memory (LSTM) network

Long short term memory (LSTM) neural networks represent a special type of RNNs, which are able to learn long-term dependencies. They were introduced in [START_REF] Hochreiter | Long short-term memory[END_REF]] and explicitly designed to avoid the long-term dependency problem. The LSTM models have shown themselves to be extremely effective in many tasks, such as handwriting recognition [Graves et al., 2009;[START_REF] Pham | Dropout improves recurrent neural networks for handwriting recognition[END_REF]; machine translation [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF]; speech synthesis [START_REF] Fan | TTS synthesis with bidirectional LSTM based recurrent neural networks[END_REF]; visual recognition and description [START_REF] Donahue | Long-term recurrent convolutional networks for visual recognition and description[END_REF]; speech recognition [START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF][START_REF] Sak | Long short-term memory recurrent neural network architectures for large scale acoustic modeling[END_REF] and many others [START_REF] Greff | LSTM: A search space odyssey[END_REF].

A schematic LSTM block diagram is illustrated in Figure 2.7. Here c t is the memory cell state at time moment t. The LSTM has the ability to remove or add information to the cell state, regulated by structures called gates. Gates provide a mechanism to optionally let information through. They are composed of a sigmoid neural net layer and a pointwise multiplication operation. To protect and control the cell state, the LSTM has three gates:

• Forget gate f t decides what information is thrown away from the cell state. The computations at time t are described as follows:

                           i t = σ (W io o t + W iy y t-1 + b i ) , f t = σ W f o o t + W f y y t-1 + b f , ct = tanh (W co o t + W cy y t-1 + b c ) , c t = f t ⊙ c t-1 + i t ⊙ ct , u t = σ (W uo o t + W uy y t-1 + b u ) , y t = u t ⊙ tanh(c t ), (2.32)
where W .o terms denote the weight matrices connecting the inputs with the units; W .y terms denote the weight matrices connecting the memory cell outputs from the previous time moment t -1 with the units. The operation ⊙ denotes the element-wise multiplication of two vectors.

There are many variants of the original LSTM model, which were proposed in the literature for different tasks [START_REF] Grézl | Probabilistic and bottle-neck features for LVCSR of meetings[END_REF][START_REF] Sak | Long short-term memory recurrent neural network architectures for large scale acoustic modeling[END_REF]Yao et al., 2015]. In one popular LSTM variant, introduced in [START_REF] Gers | Recurrent nets that time and count[END_REF], peephole connections from the cell to the gates were added to the neural network architecture. This allows the neural network to learn exact timings. Another alternative structure is the gated recurrent unit (GRU), proposed in [START_REF] Cho | Learning phrase representations using rnn encoder-decoder for statistical machine translation[END_REF]. It consists in simplification of the LSTM architecture. In GRU, the forget and input gates are combined into a single update gate. In addition, the cell state and hidden state are merged, and some other modifications are made.

Chapter 3

Overview of automatic speech recognition

This chapter provides an overview of ASR systems with the focus on hidden Markov model based ASR systems and describes their principal concepts.

Introduction to automatic speech recognition

The development of ASR has a long history -since 1960 till this moment -and during this period has achieved a great progress. The success in ASR research has led to the increase of a variety of real-world applications, such as interactive spoken dialog systems, information extraction and retrieval, automatic closed captioning, dictation, transcription of recorded speech, language learning systems, etc.

The goal of ASR consists in finding a text representation for an input speech signal. In a statistical speech recognition paradigm, which is the most common one for this problem nowadays [START_REF] Benesty | Springer handbook of speech processing[END_REF][START_REF] Rabiner | A tutorial on hidden markov models and selected applications in speech recognition[END_REF]], an ASR system aims to find word sequence W = (w 1 , . . . , w N ), which is the most probable according to the trained model, for given acoustic observation feature vectors

O = [o 1 , . . . , o T ].
The principal components of an ASR system with an example of speech recognition analysis are illustrated in Figure 3 (3.1)

However, since P (W|O) is difficult to model directly, Bayes theorem is usually applied [START_REF] Jelinek | Continuous speech recognition by statistical methods[END_REF]:

P (W|O) = P(O|W)P(W) P(O) ∝ P(O|W)P(W) (3.2)
to transform into the equivalent problem :

W * = arg max W P(O|W)P(W). (3.
3)

The likelihood P(O|W) is determined by an acoustic model and the prior P(W) -by a language model. As shown in Figure 3.1 each word w n in W is represented as a sequence of basic sounds called phones. To take into account different pronunciation variations (phonetic transcriptions Q), the likelihood P(O|W) can be estimated as

P(O|W) = ∑ Q P(O|Q)P(Q|W), (3.4)
where Q is a sequence of word pronunciations Q = (Q 1 , . . . , Q N ) for the word sequence W, and each word pronunciation is a sequence of phones:

Q n = (q (n) 1 , q (n) 2 . . .), 1 ≤ n ≤ N. Then P(Q|W) = N ∏ n=1 P (Q n |w n ), (3.5)
where P (Q n |w n ) is the probability that the word w n is pronounced with phonetic transcription Q n . Hence P(Q|W) corresponds to pronunciation model and gives the probability of the phonetic sequence given the sequence of words. The pronunciation model is also called (phonetic) dictionary, or lexicon. Taking into account all three factors (acoustic, phonetic and language) and the corresponding models, Formula (3.3) can be rewritten as follows: In our example presented in Figure 3.1 the phrase W= (his, most, significant, scientific, publications) corresponds to the phonetic transcription Q=((hh ih z), (m ow s), (s ah g n ih f ih k ah n t), (s ay ah n t ih f ih k), (p ah b l ih k ey sh ah n z)).

W * = arg max

Feature extraction

A feature extraction (also referred to as acoustic front-end analysis) module is an important component of any ASR system. The purpose of the feature extraction stage is to provide a compact encoding of the speech waveform signal to be used further in the ASR system.

Speech data contains information about different aspects of the signal besides the pronounced word sequence. These aspects include language and dialect, environmental characteristics (recording channel, background noises), speaker's personal characteristics (gender, emotional state, age) and others. One of the main goals of the feature extraction module is to remove the information irrelevant for ASR from the speech signal while preserving the essential information about the spoken content. Another goal is to reduce the dimension of the obtained feature vectors and provide the optimal representation of information for the given acoustic model (AM).

Typically, the waveform file format is used for storing speech data. The continuous speech signal is transformed into a sequence of samples (or the discrete-time signal). Signal sampling can be performed at different sampling rates (usually 8-44 kHz) depending on the conditions, task and recording channel. Speech signal is divided into overlapping segments and feature vectors are computed usually every 10-15 ms using an overlapping analysis window of about 25 ms. Speech signal is supposed to be stationary in these segments (speech frames). Usually each speech frame is represented by a single parameter feature vector.

One of the simplest and most widely used speech encoding techniques is based on Melfrequency cepstral coefficients (MFCCs) extraction [START_REF] Davis | Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences[END_REF]. Also there have been various other types of features proposed for the speech recognition problem, which aim to incorporate new information into speech recognition system: perceptual linear prediction (PLP) [START_REF] Hermansky | Perceptual linear predictive (PLP) analysis of speech[END_REF], temporal patterns (TRAPS) features [START_REF] Hermansky | Temporal patterns (TRAPS) in ASR of noisy speech[END_REF], discriminatively trained features [START_REF] Povey | fMPE: Discriminatively trained features for speech recognition[END_REF] with minimum phone error objective function (fMPE features), and many others. Features generated by neural networks, such as tandem features [START_REF] Hermansky | Tandem connectionist feature extraction for conventional hmm systems[END_REF], features with split left and right temporal contexts (LC-RC) [START_REF] Schwarz | Phoneme recognition based on long temporal context[END_REF], and bottleneck (BN) features [START_REF] Grézl | Probabilistic and bottle-neck features for LVCSR of meetings[END_REF][START_REF] Yu | Improved bottleneck features using pretrained deep neural networks[END_REF] have shown to be very effective in many ASR systems.

Usually ASR systems with GMM-HMM acoustic models do not use filter-bank coefficients (fbanks) as the input representation because they are strongly correlated, and to model them well many parameters (a lot of diagonal Gaussians or full covariance Gaussians) are required [Hinton et al., 2012a]. However, for neural network acoustic models, raw filter-bank features sometimes are more preferable than MFCCs or others [START_REF] Deng | Recent advances in deep learning for speech research at Microsoft[END_REF][START_REF] Mohamed | Acoustic modeling using deep belief networks[END_REF].

Features, obtained with a GMM were used in [START_REF] Pinto | Combining evidence from a generative and a discriminative model in phoneme recognition[END_REF] for training a phoneme multilayer perceptron recognizer. This type of features is close to the one which will be used in this thesis. Augmenting the basic input features with some additional features, for example, pitch features [START_REF] Ghahremani | A pitch extraction algorithm tuned for automatic speech recognition[END_REF][START_REF] Metze | Models of tone for tonal and non-tonal languages[END_REF] is a common technique to improve ASR performance.

In the next two sections, we describe the most commonly used in ASR types of features from two categories: spectral features and neural network-based features.

Spectral features

The general scheme for feature extraction can be described as follows.

• Spectrum computation. Short time fast Fourier transform (FFT) is calculated for each speech frame.

• Auditory-like modifications. Modifications, which are motivated by psycho-physical acoustic phenomena and human perception characteristics [START_REF] Moore | An introduction to the psychology of hearing[END_REF], are performed for the obtained spectra.

For example, in case of MFCC features, to take into account different sensitivity of human hearing system to different frequencies, the powers of the obtained spectra are mapped into the Mel scale. In addition, since the human perception of loudness is logarithmic, all the Mel power-spectra values are transformed into logarithmic scale.

The obtained features are referred to as mel-filter bank (fbank) features, and sometimes are used in this thesis to train DNN AMs.

• Decorrelation. In case of MFCCs, decorrelation of the obtained features is done by discrete cosine transform (DCT).

• Derivatives. The resulting feature vectors are often appended with their first and second order temporal derivatives (delta (∆) and acceleration (∆∆) coefficients).

MFCCs are one of the most widely used input features for ASR systems. Another popular features are PLP [START_REF] Hermansky | Perceptual linear predictive (PLP) analysis of speech[END_REF]. In PLP feature extraction, the Bark scale is used (instead of the Mel scale) to compute the filter-bank filters. Then this process is followed by a linear predictive analysis, from which a cepstral representation is derived. Both, MFCC and PLP features are calculated with ∆ and ∆∆ derivatives and usually have the same dimension (39). It was shown [START_REF] Dave | Feature extraction methods LPC, PLP and MFCC in speech recognition[END_REF][START_REF] Milner | A comparison of front-end configurations for robust speech recognition[END_REF][START_REF] Psutka | Comparison of mfcc and plp parameterizations in the speaker independent continuous speech recognition task[END_REF] that PLP features can provide comparable results to MFCCs, and the superiority of one feature type over another depends on the database characteristics, environmental conditions, or system configurations.

Neural network-based features

Neural network models are often used as feature extractors in ASR systems. Features generated by these models can further be used to train other classifiers for ASR, such as GMMs or DNNs. Two important types of neural network-based features, tandem and bottle-neck features, are described below. In the tandem approach [Ellis et al., 2001;[START_REF] Hermansky | Tandem connectionist feature extraction for conventional hmm systems[END_REF][START_REF] Morgan | Deep and wide: Multiple layers in automatic speech recognition[END_REF] a neural network, trained with phones as targets, is used to generate input features fed to a con-ventional GMM-HMM model. These features can be fed directly to a GMM model after some post-processing steps (decorrelation and dimensionality reduction, for example, using principal component analysis (PCA), DCT, heteroscedastic linear discriminant analysis (HLDA) [START_REF] Kumar | Heteroscedastic discriminant analysis and reduced rank HMMs for improved speech recognition[END_REF]) as in the originally proposed approach [START_REF] Hermansky | Tandem connectionist feature extraction for conventional hmm systems[END_REF], or appended with other features, for example, with some spectral features before training a GMM model. An example of the tandem speech recognition system is illustrated in Figure 3.2. The dashed lines in the figure correspond to the stages of the feature extraction process that can be omitted.

In this thesis we propose and investigate the approach which is in some sense inverse to the tandem approach -features, derived from GMM models, are used as input representation for training DNN models.

Bottleneck features

Bottleneck (BN) features [START_REF] Grézl | Probabilistic and bottle-neck features for LVCSR of meetings[END_REF][START_REF] Yu | Improved bottleneck features using pretrained deep neural networks[END_REF] have shown to be effective in improving the accuracy of ASR systems for both DNN-HMM and GMM-HMM acoustic models. Conventionally, bottleneck features are extracted from a DNN, which is trained to predict context-independent monophone states or context-dependent triphone states. In [START_REF] Yu | Improved bottleneck features using pretrained deep neural networks[END_REF] it was shown that the use of context-dependent triphone states gives better accuracy for the final ASR system.

One of the hidden layers in a DNN, trained for BN feature extraction, has a relatively small dimension (35-80 units). This layer is referred to as a BN layer. The part of the DNN, following the BN layer, is removed from the DNN structure after the training, and the outputs from the BN layer are used as features directly or with some post processing (splicing, concatenation with other features and other modifications) to train DNN-HMM or GMM-HMM models. An example of using BN features for training a GMM-HMM acoustic model is shown in Figure 3.3.

An important property of bottleneck features -the capability of transferring the learned representation to unseen languages [START_REF] Vu | Investigating the learning effect of multilingual bottle-neck features for ASR[END_REF] -makes them (and some of their advanced versions, such as stacked BN (SBN) features [START_REF] Grézl | Adaptation of multilingual stacked bottle-neck neural network structure for new language[END_REF]) very popular in multilingual training architectures, especially for low-resource ASR systems [START_REF] Bell | Cross-lingual adaptation with multi-task adaptive networks[END_REF][START_REF] Li | Investigation of cross-lingual bottleneck features in hybrid ASR systems[END_REF][START_REF] Müller | Multilingual deep bottle neck features: a study on language selection and training techniques[END_REF].

Acoustic modeling

Statistical approach with hidden Markov models (HMMs) [START_REF] Rabiner | A tutorial on hidden markov models and selected applications in speech recognition[END_REF] is conventional in state-of-the-art ASR systems. As mentioned above (see Figure 3.1), any spoken word w is represented with a sequences of basic sounds (units). Each basic unit is modeled with a corresponding HMM.

The simplest example of a unit is a phoneme. The same phonemes may have different pronunciations depending on their context. To take into account the effects caused by coarticalation and reduction, usually context-dependent (CD) triphone units (phonemes with given left and right neighbor phonemes) are used for acoustic modeling. When phones are modeled without considering the context they are referred to as context-independent (CI) or monophone units.

Introducing triphones into the ASR system significantly increases the number of parameters to be estimated for this system. Some triphones may be missing in the training data sets (so-called unseen triphones) or have insufficient number of realizations to model them accurately. To avoid this data sparsity problem, clustering algorithms, such as data-driven clustering and tree-based clustering, are commonly used [START_REF] Hwang | Predicting unseen triphones with senones[END_REF][START_REF] Young | The HTK book[END_REF][START_REF] Young | Tree-based state tying for high accuracy acoustic modelling[END_REF] for triphone state clustering. They tie the parameters of those triphone states that are close acoustically.

Hidden Markov models

An HMM is a stochastic Markov process with hidden (unobserved) states. The hidden states can be observed indirectly through another stochastic process (or several stochastic processes) that outputs the sequences of observed symbols. A detailed introduction to HMMs for speech processing is given in [START_REF] Rabiner | An introduction to hidden markov models[END_REF].

In ASR systems, each unit (triphone or monophone) is modeled with a continuous density HMM, as illustrated in Figure 3.4 for a sequence of observation vectors O = (o 1 , o 2 , . . .). An HMM model is characterized by the following set of parameters:

• Number of states N;

• State transition probability distributions a i j , denoted by matrix A = {a i j }:

a i j = P (s t = j|s t-1 = i) , (3.7)
where

N ∑ j=1 a i j = 1, ∀i = 1, . . . , N. (3.8)
• ). Transition parameters a i j model the transition probabilities between hidden states i and j.

• Initial state probability distribution. The initial probability π i for state s i is π i = P(s 0 = i), and ∑ N i=1 π i = 1. In practice and further in this thesis, initial distribution is excluded from the analysis.

Hence an HMM is represented with model parameters Λ Λ Λ = {A, B} = {a i j }, {b j (.)} . Modeling of output probability distributions b j (.) depends on the type of acoustic model (see Section 3.3.2).

Due to computational reasons, two important assumptions for HMM in ASR systems are usually made (except for some particular examples, such as graphical models [START_REF] Bilmes | Graphical model architectures for speech recognition[END_REF]):

1. A first order Markov process assumption, which means that the stochastic process satisfies the Markov property, so that its state s t at time t depends only on the state s t-1 at the previous moment of time: The analysis of conditions where the standard model assumptions deviate from real speech data is given in [START_REF] Gillick | Don't multiply lightly: Quantifying problems with the acoustic model assumptions in speech recognition[END_REF]. These assumptions allow us to simplify the calculation of acoustic likelihood for a given sequence of observations O and states S as

P (s t |s t-
P (O|w;Λ Λ Λ) = ∑ S∈w T ∏ t=1 P(o t |s t ;Λ Λ Λ)P(s t |s t-1 ;Λ Λ Λ) = ∑ S∈w a q 0 ,q 1 T ∏ t=1 b q t (o t )a q t ,q t+1 .
(3.12)

In practice, the calculation of the likelihoods directly from Formula (3.12) is too computationally expensive and requires O(N T ) multiplications and summations, hence special algorithms were developed for this purpose. The Baum-Welch forward-backward algorithm [START_REF] Baum | An inequality with applications to statistical estimation for probabilistic functions of markov processes and to a model for ecology[END_REF] reduces the complexity to O(N 2 T ) steps. Another widely-used solution is the Viterbi algorithm [START_REF] Viterbi | Error bounds for convolutional codes and an asymptotically optimum decoding algorithm[END_REF], which finds only the most likely state sequence (using max S∈w operation instead of ∑ S∈w in Formula (3.12)).

State distribution modeling

In this section we consider some of the most common approaches for modeling state probability distributions in HMMs.

GMM-HMM

The Gaussian mixture models (GMMs) approach has been one of the most common technique to model state probability distributions (see Formula (3.9)) in ASR systems for many decades. In a GMM-HMM model, each state j is modeled as a weighted sum of M j Gaussians:

b j (o t ) = P(o t | j) = M j ∑ m=1 ω jm N o t ;µ µ µ jm ,Σ Σ Σ jm , (3.13)
where ω jm is the weight of the m'th component in the mixture for state j such that: 1. (3.14) In Formula (3.13), N (.;µ µ µ,Σ Σ Σ) is a multivariative Gaussian distribution with mean vector µ µ µ and covariance matrix Σ Σ Σ: 3.15) where d is the dimensionality of acoustic feature vector: o ∈ R d .

ω jm ≥ 0 and M j ∑ m=1 ω jm =
N (o;µ µ µ,Σ Σ Σ) = 1 (2π) d |Σ Σ Σ| exp - 1 2 (o -µ µ µ) T Σ Σ Σ -1 (o -µ µ µ) , (

Subspace Gaussian mixture models

Subspace Gaussian mixture models (SGMMs) have been proposed in [START_REF] Burget | Multilingual acoustic modeling for speech recognition based on subspace gaussian mixture models[END_REF][START_REF] Povey | Subspace Gaussian mixture models for speech recognition[END_REF]Povey et al., , 2011a] ] as an alternative approach to standard GMM-HMMs for acoustic modeling. All context-dependent HMM states in SGMMs share a common representation, based on the universal background model (UBM) 1 . The covariance matrices are shared between states. The mean vectors and mixture weights for each state are specified by a corresponding vector v j ∈ R S , where S is a subspace dimension (typically S = 50). In the simplest form SGMM can be expressed as follows [START_REF] Povey | Subspace Gaussian mixture models for speech recognition[END_REF]:

         b j (o t ) = P(o t | j) = ∑ I i=1 ω ji N o t ;µ µ µ ji ,Σ Σ Σ i , µ µ µ ji = M i v j , ω ji = exp ω T i v j ∑ I i ′ =1 exp ω T i ′ v j , (3.16)
where j ∈ {1, . . . , J} is a context-dependent speech state, modeled with a GMM with I Gaussians (usually I ∈ [200,2000]) and covariance matrices Σ Σ Σ i which are shared between states, mixture weights ω ji and means µ µ µ ji . The description of model extensions and developed training procedures can be found in [Povey et al., 2011a]. The strengths of SGMMs lie in their compactness. These models require less data for training in comparison with GMM models to achieve a comparable accuracy and have been widely used for low-resource and multi-lingual ASR systems [START_REF] Burget | Multilingual acoustic modeling for speech recognition based on subspace gaussian mixture models[END_REF][START_REF] Lu | Regularized subspace Gaussian mixture models for cross-lingual speech recognition[END_REF]. The weakness of SGMMs consists in their complexity of implementation in comparison with conventional GMMs; we will not consider these models further in the thesis.

DNN-HMM

The use of neural networks is an alternative approach for AM training, proposed in [START_REF] Bourlard | Links between markov models and multilayer perceptrons[END_REF][START_REF] Morgan | Continuous speech recognition using multilayer perceptrons with hidden markov models[END_REF], where multilayer perceptions (MLP) (feed-forward artificial neural networks (ANN)) were used to estimate emission probabilities for HMMs. In the DNN-HMM approach state probabilities in HMMs are estimated with a neural network model.

One of the commonly used approaches in state-of-the-art ASR systems is a hybrid DNN-HMM approach, where the dynamics of speech signal is modeled with HMMs, and the state observation probabilities are estimated with DNN models. In the original approach [START_REF] Bourlard | Links between markov models and multilayer perceptrons[END_REF] the MLP outputs were used directly for HMMs. But later the need for scaling the obtained probabilities with state priors was shown. The output posterior probabilities from a neural network are scaled using prior class probabilities estimated from the training data, to obtain scaled likelihoods [START_REF] Bourlard | CDNN: A context dependent neural network for continuous speech recognition[END_REF].

In a DNN-HMM system, outputs of a DNN are the state posteriors P(s i |o t ), which are transformed for decoding into pseudo (or scaled) log-likelihoods as follows:

log P(o t | s i ) = log P(s i | o t )P(o t ) P(s i ) ∝ log P(s i | o t ) -log P(s i ), (3.17)
where the state prior P(s i ) can be estimated from the state-level forced alignment on the training speech data, and probability P(o t ) of the acoustic observation vector o t is independent on the HMM state and can be omitted during the decoding process.

Typically states of context-dependent (CD) phonemes (triphones) are used as basic classes for DNN outputs. However the earlier hybrid ASR systems use context-independent (CI) phones (monophones) states. The first attempts to incorporate CD triphones into hybrid architectures [START_REF] Bourlard | CDNN: A context dependent neural network for continuous speech recognition[END_REF] were based on modelling the context-dependency using factorization: P(s i , c j |o t ) = P(s i |o t )P(c i |s j , o t ) = P(c i |o t )P(s i |c j , o t ), (3.18) where c j ∈ {c 1 , . . . , c J } is one of the clustered context classes, and s j is either a CD phone or a state in a CD phone. ANNs were used to estimate both P(s i |o t ) (or P(c i |o t )) and P(c i |s j , o t ) (or P(s i |c j , o t )) probabilities. These types of CD DNN-HMM models provided comparatively small improvement over the GMM-HMM models.

The more efficient technique for CD state modeling [START_REF] Dahl | Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition[END_REF]] uses a DNN model to directly estimate the posterior distribution of tied triphone HMM states. These states are usually obtained from a conventional GMM-HMM system through a state clustering procedure, however there are some works that propose a GMM-free training procedure [START_REF] Bacchiani | Asynchronous, online, GMM-free training of a context dependent acoustic model for speech recognition[END_REF]Senior et al., 2014;[START_REF] Zhu | Gaussian free cluster tree construction using deep neural network[END_REF]. A DNN-HMM model is illustrated in Figure 3.5.

The recent technical progress in computational hardware allowed to overcome some limitations of the first hybrid ASR systems. The shallow neural networks have been replaced with the deep neural architectures, with many hidden layers. This type of acoustic models is referred to as CD-DNN-HMM models in the literature. Modern DNN-HMM AMs have been demonstrated to significantly outperform GMM-HMM systems on various ASR tasks [START_REF] Dahl | Large vocabulary continuous speech recognition with context-dependent DBN-HMMs[END_REF][START_REF] Dahl | Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition[END_REF]Hinton et al., 2012a;[START_REF] Kingsbury | Scalable minimum bayes risk training of deep neural network acoustic models using distributed hessian-free optimization[END_REF][START_REF] Seide | Conversational speech transcription using contextdependent deep neural networks[END_REF][START_REF] Yu | Roles of pre-training and fine-tuning in contextdependent dbn-hmms for real-world speech recognition[END_REF]. A review of recent developments in DNN-HMM framework and in deep learning approach to ASR is given in [Li et al., 2015a;[START_REF] Trentin | A survey of hybrid ANN/HMM models for automatic speech recognition[END_REF][START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF]. The acoustic model parameters Λ Λ Λ = {a i j }, {b j (.)} can be estimated from a training corpus using expectation maximization (EM) algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF]. In a general case, HMMs are trained using a chosen training criterion, which corresponds to a particular One of the standard ways to train HMM models is by optimizing maximum likelihood (ML) objective function:

F ML (Λ Λ Λ) = R ∑ r=1 log p Λ Λ Λ (O r |φ r ), (3.19)
where φ r = φ (W r ) is a composite HMM model, corresponding to the (correct) transcription of the training sentence O r and W r is a sequence of words in this transcription. This objective function can be maximized by using the EM algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], known as the Baum-Welch algorithm [START_REF] Baum | An inequality with applications to statistical estimation for probabilistic functions of markov processes and to a model for ecology[END_REF][START_REF] Juang | Maximum likelihood estimation for multivariate mixture observations of markov chains (corresp.)[END_REF].

One of the main problem with the ML training criterion is that it fits to the training data, but does not take into account the ability of the model to discriminate. To eliminate this problem, alternative approaches with discriminative training criteria [START_REF] Schlüter | Comparison of discriminative training criteria and optimization methods for speech recognition[END_REF][START_REF] Vertanen | An overview of discriminative training for speech recognition[END_REF] have been proposed.

In one of the first approaches, developed in the discriminative training framework, parameters are trained to maximize the mutual information (MMI) between an acoustic observation sequence and the corresponding word sequence [START_REF] Bahl | Maximum mutual information estimation of hidden markov model parameters for speech recognition[END_REF][START_REF] Povey | Discriminative training for large vocabulary speech recognition[END_REF][START_REF] Povey | Improved discriminative training techniques for large vocabulary continuous speech recognition[END_REF][START_REF] Schluter | Comparison of discriminative training criteria[END_REF][START_REF] Schlüter | Interdependence of language models and discriminative training[END_REF] with the following objective function:

F MMI (Λ Λ Λ) = R ∑ r=1 log p Λ Λ Λ (O r |φ r )P(φ r ) ∑ φ p Λ Λ Λ (O r |φ )P(φ ) , (3.20)
where P(φ ) is the language model probability for sentence O. Here the numerator corresponds to the data given the correct word sequence W r , and the dominator corresponds to the total likelihood of the data given all possible word sequences W. In practice, W is obtained from an ASR system (in the form of N-best lists or word lattices). Hence, the MMI criterion attempts not only to make the correct hypothesis more probable, but also to decrease the probability of incorrect alternatives.

Other approaches include minimum phone error (MPE) training [START_REF] Povey | Minimum phone error and I-smoothing for improved discriminative training[END_REF], which attempts to maximize the a posteriori sentence probability, scaled by the raw phone accuracy, and a state-level minimum Bayes risk (sMBR) criterion [START_REF] Veselỳ | Sequence-discriminative training of deep neural networks[END_REF]. These two criteria can be expressed as

F MBR (Λ Λ Λ) = R ∑ r=1 ∑ φ p Λ Λ Λ (O r |φ ) k P(φ ) k A(φ , φ r ) ∑ φ ′ p Λ Λ Λ (O r |φ ) k P(φ ′ ) k , (3.21)
where k is a scaling factor, and A(φ , φ r ) is the raw accuracy of the sentence W with respect to the reference W r . Hence, A(φ , φ r ) corresponds to the number of correct phone labels (in MPE) or state labels (in sMBR).

There are also many others [START_REF] Kaiser | A novel loss function for the overall risk criterion based discriminative training of hmm models[END_REF][START_REF] Povey | Boosted MMI for model and feature-space discriminative training[END_REF] criteria, for example, minimum classification error (MCE) [START_REF] Chou | Minimum error rate training based on N-best string models[END_REF][START_REF] Juang | Discriminative learning for minimum error classification (pattern recognition)[END_REF], which focuses on sentence-level accuracy, or boosted MMI (BMMI) criterion [START_REF] Povey | Boosted MMI for model and feature-space discriminative training[END_REF].

In the BMMI criterion [START_REF] Povey | Boosted MMI for model and feature-space discriminative training[END_REF] the MMI objective function (3.20) is modified to boost the likelihoods of paths that contain more errors, as follows:

F BMMI (Λ Λ Λ) = R ∑ r=1 log p Λ Λ Λ (O r |φ r )P(φ r ) ∑ φ p Λ Λ Λ (O r |φ )P(φ )e -bA(φ ,φ r ) , (3.22)
where b is the boosting factor.

In this work for training GMM-HMM models we will mostly use the ML and BMMI criterion. For DNN-HMM models we will apply the cross-entropy (CE) criterion (see Section 2.3) and, for some models, perform additional training with the sMBR criterion.

DNN-HMM

DNNs usually use a contextual window of several (2T + 1) frames as an input vector. As shown in Figure 3.5, the input vector for the DNN is vector

o t = [o t-T , . . . , o t , . . . , o t+T ].
In a CD-DNN-HMM system, a single DNN is used to model the conditional state posterior probability P(s i |o t ) for all states, in contrast to a GMM-HMM system, where each state is modeled by a separate GMM model.

DNNs for AMs can be trained with a CE criterion (Formula (2.15)) using SGD algorithm, as described in Section 2.3. To obtain targets for each observation vector, the force-alignment of the training corpus can be produced by a GMM-HMM system, or with DNN alignments as in a GMM-free training procedure [START_REF] Bacchiani | Asynchronous, online, GMM-free training of a context dependent acoustic model for speech recognition[END_REF]Senior et al., 2014].

The CE criterion operates on the frame level, so it is possible to improve AMs using discriminative criteria [START_REF] Veselỳ | Sequence-discriminative training of deep neural networks[END_REF], which operate on the sentence level, as described in Section 3.3.3.1. Similar to training GMM-HMM systems, before the sequence-discriminative training of DNNs the numerator and denominator lattices have to be generated. In [START_REF] Povey | Purely sequence-trained neural networks for asr based on lattice-free mmi[END_REF] a sequence-discriminative training of neural network AMs using lattice-free version of the (MMI) criterion is performed without frame-level CE pre-training. In recent times, connectionist temporal classification (CTC) criterion [START_REF] Graves | Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[END_REF], which allows to train a DNN without pre-generated frame-level alignment, has attracted a lot of interest in ASR research. We will consider this approach in Section 3.7.

Pronunciation modeling

The pronunciation model (or lexicon) defines the list of words with their phonetic transcriptions in terms of a set of context-independent phonemes. Some words may have multiple pronunciations and prior probabilities associated with their occurrence frequencies.

A pronunciation system usually contains a static word pronunciation dictionary, which can be created by experts or generated automatically. However, such a static dictionary may not cover all possible words in a language, that are required for a given ASR system. Hence, a dictionary is often supplemented with a grapheme-to-phoneme (G2P) converter. Many approaches for building a lexicon (or G2P converters) can be broadly classified into two classes [START_REF] Bisani | Joint-sequence models for grapheme-to-phoneme conversion[END_REF][START_REF] Jouvet | Evaluating grapheme-to-phoneme converters in automatic speech recognition context[END_REF][START_REF] Wester | Pronunciation modeling for ASR-knowledge-based and data-derived methods[END_REF]]:

• Knowledge-based approach, where different pronunciation variants are generated by using phonological rules of the given language [START_REF] Kaplan | Regular models of phonological rule systems[END_REF]. Such rules are developed by experts and based on linguistic knowledge. They have to take into account many pronunciation exceptions that can occur in a language.

• Data-driven approach, in which phonetic pronunciations are automatically trained from a corpus of pronunciation examples. Different algorithms have been developed to train such G2P converters, including joint-sequence models [START_REF] Bisani | Joint-sequence models for grapheme-to-phoneme conversion[END_REF], conditional random field (CRF) models [START_REF] Illina | Grapheme-to-phoneme conversion using conditional random fields[END_REF][START_REF] Wang | Letter-to-sound pronunciation prediction using conditional random fields[END_REF], the statistical machine translation approach [START_REF] Laurent | Grapheme to phoneme conversion using an SMT system[END_REF], approaches based on using neural networks [START_REF] Rao | Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks[END_REF][START_REF] Yao | Sequence-to-sequence neural net models for grapheme-tophoneme conversion[END_REF]] and many others.

Often, and in this thesis also, all pronunciations of a given word are assumed to have equal prior probabilities.

Language modeling

Statistical language modeling aims to learn the joint probability function of word sequences in a given language. The language model (LM), which corresponds to P(W) in Formula (3.3), estimates a probability distribution over the sequence of words: W = (w 1 , . . . , w M ). The prior probability of W is estimated as

P(W) = M ∏ m=1 P (w m |w m-1 , . . . , w 1 ). (3.23)
For large vocabulary continuous speech recognition (LVCSR) systems, the common technique is based on using n-gram LMs [START_REF] Damerau | Markov models and linguistic theory: an experimental study of a model for English[END_REF] and approximates this probability as:

P(W) = M ∏ m=1 P (w m |w m-1 , . . . , w m-n+1 ). (3.24)
In Formula (3.24), n is the n-gram order of the given LM. It determines the length of the conditioning word history. Usually n is in the range 2-4.

LMs are trained on text corpora, by estimating n-gram probabilities of word n-gram frequencies to maximize the likelihood (ML) of the training data [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF]. Let denote C (w m , . . . , w m+k ) the number of occurrences of the k + 1 words w m , . . . , w m+k in the training corpus, then

P(w m |w m-1 , w m-2 ) ≈ C (w m-2 , w m-1 , w m ) C(w m-2 , w m-1 ) . (3.25)
The major problem with this ML estimation approach is data sparsity. There are several approaches to robust estimation of LMs:

1. Katz smoothing [START_REF] Katz | Estimation of probabilities from sparse data for the language model component of a speech recognizer[END_REF] consists in combination of discounting and backing-off :

P(w m |w m-1 , w m-2 ) =              C (w m-2 , w m-1 , w m ) C(w m-2 , w m-1 ) , if C * < C, d C (w m-2 , w m-1 , w m ) C(w m-2 , w m-1 ) , if 0 < C ≤ C * , α(w m-1 , w m-2 )P(w m |w m-1 ), C = 0, (3.26) 3.5 Language modeling
where C * is a count threshold, d is a discount coefficient and α is a normalization constant. That means, that when there is enough statistic for some n-grams, the ML estimate is used. When the amount of available statistic is not sufficient according to the chosen threshold, then the same ML estimated is used, but discounted. The "discounted probability mass" is then distributed over unseen n-grams, which are estimated with the weighted version of their (n -1)-grams. In our example (Formula (3.26

)) n = 2.
The discounting coefficient d is based on Good-Turing estimates [START_REF] Good | The population frequencies of species and the estimation of population parameters[END_REF][START_REF] Katz | Estimation of probabilities from sparse data for the language model component of a speech recognizer[END_REF]: d = (r + 1)n r+1 /(rn r ), where n r is the number of n-grams that occur r times in the training corpus.

2. Kneser-Ney smoothing [START_REF] Chen | An empirical study of smoothing techniques for language modeling[END_REF][START_REF] Kneser | Improved backing-off for m-gram language modeling[END_REF][START_REF] Ney | On structuring probabilistic dependences in stochastic language modelling[END_REF] is an improved smoothing algorithm.

3. Class-based models [START_REF] Brown | Class-based n-gram models of natural language[END_REF][START_REF] Martin | Algorithms for bigram and trigram word clustering[END_REF]] is an alternative approach to robust LM estimation, where every word w m has a corresponding class c m , to which this word belongs. Then the probability of the word sequence is estimated using class n-gram probabilities:

P(W) = M ∏ m=1 P(w m |c m )P(c m |c m-1 , . . . , c m-n+1 ). (3.27)
As for the word n-grams, class n-grams are estimated via ML, and since the number of classes (typically several hundreds) is much smaller, than the number of words, this approach helps to overcome the data sparsity problem. The words are grouped together into the classes according to the underlying nature of these classes. It can be syntactical, semantical or statistical word classes. In the statistical approach for classes, the word classes are chosen to optimize the likelihood of the training data.

4. Interpolation of LMs relies on linearly weighted combination of several LMs with the same or different n-gram order.

Various other LM approaches have been proposed for ASR systems, for example, random forest LMs [START_REF] Xu | Using random forest language models in the IBM RT-04 CTS system[END_REF], structured LMs [START_REF] Chelba | Structured language modeling[END_REF][START_REF] Emami | Using a connectionist model in a syntactical based language model[END_REF], LMs based on constraint dependency grammars (CDGs) [START_REF] Wang | The use of a linguistically motivated language model in conversational speech recognition[END_REF] and others [START_REF] Schwenk | Continuous space language models[END_REF]. In recent years neural networks have become a popular approach for language modeling [START_REF] Bengio | A neural probabilistic language model[END_REF][START_REF] Le | Structured output layer neural network language model[END_REF][START_REF] Xu | Can artificial neural networks learn language models?[END_REF]. The idea in [START_REF] Bengio | A neural probabilistic language model[END_REF] is based on mapping a discrete n-gram word distribution to a continuous representation. Since the obtained distributions are smooth functions of the word representation, this approach allows to better generalize the unseen n-grams. In [START_REF] Mikolov | Recurrent neural network based language model[END_REF] recurrent neural network based LMs have been proposed. The long short-term memory (LSTM) neural network architecture for LMs has been explored in [START_REF] Sundermeyer | LSTM neural networks for language modeling[END_REF]. Neural network based LMs are usually used for ASR hypotheses rescoring in a post-processing step.

An evaluation metric for LMs is perplexity estimated on a test dataset as:

PPL = 2 -1 M ∑ M m=1 log 2 (P(w m |w m-1 ,...,w 1 )) .
(3.28)

The lower the perplexity, the better the LM generalizes to predict unseen data.

In this thesis we will use 2-gram, 3-gram and 4-gram LMs.

Speech decoding techniques

After the acoustic, pronunciation and language models are estimated, it is possible to perform decoding, based on Formula (3.6). Existing decoding strategies can be characterized by the following aspects [START_REF] Aubert | An overview of decoding techniques for large vocabulary continuous speech recognition[END_REF]:

• Network expansion:

-Static expansion of the search space, where the entire decoding network is generated before the decoding [START_REF] Saon | Anatomy of an extremely fast LVCSR decoder[END_REF]. In this case, any search algorithm may be applied, though in practice, a time-synchronous Viterbi search is usually used [START_REF] Forney | The Viterbi algorithm[END_REF][START_REF] Viterbi | Error bounds for convolutional codes and an asymptotically optimum decoding algorithm[END_REF] with implementation based on weighted finite-state transducers (WFSTs) [START_REF] Mohri | Weighted finite-state transducers in speech recognition[END_REF].

-Dynamic (or on-the-fly) expansion of the search space [START_REF] Rybach | Lexical prefix tree and WFST: A comparison of two dynamic search concepts for LVCSR[END_REF][START_REF] Rybach | A comparative analysis of dynamic network decoding[END_REF].

In this case, the required parts of the network are generated on demand during the search. Hence, decoders with the dynamic network expansion have much lower memory consumptions, but requires additional computations during the search process, that can slow down the decoding process. There are several decoding strategies, that can be implemented in this framework, for example, the history conditioned lexical tree (HCLT) [START_REF] Ney | Improvements in beam search for 10000-word continuous speech recognition[END_REF] or WFST-based decoding with on-the-fly transducer compositions [START_REF] Mohri | Weighted automata in text and speech processing[END_REF] and on-the-fly hypothesis rescoring [START_REF] Hori | Efficient WFST-based one-pass decoding with on-the-fly hypothesis rescoring in extremely large vocabulary continuous speech recognition[END_REF].

• Search:

-Time-synchronous decoding, where all hypotheses are evaluated in parallel, frequently referred to as Viterbi decoding, which approximates the solution to Formula (3.6) by searching only for the most probable state sequence. It relies on beam pruning to restrict the number of hypotheses, developed in parallel.

-Asynchronous decoding [START_REF] Frankel | Speech recognition using linear dynamic models[END_REF][START_REF] Picheny | Speed improvement of the time-asynchronous acoustic fast match[END_REF] (for example, stack decoders [START_REF] Paul | An efficient A* stack decoder algorithm for continuous speech recognition with a stochastic language model[END_REF]), where the most promising hypothesis is followed first until the end of the speech signal is reached, thus, the search is carried out asynchronously with respect to time.

For the experiments reported in this thesis, we will apply decoding with a static network expansion and the Viterbi search with an WFST-based implementation. In the WFSTbased approach, all components of the ASR system (acoustic, phonetic and language) are represented with individual WFSTs, which are then composed into a single WFST used for decoding.

In practice, to give more strength to a LM in comparison with an AM, the LM probabilities are scaled in the log-domain2 with factor k, and Formula (3.6), is replaced with:

W * = arg max W ∑ Q P(O|Q)P(Q|W)P(W) k , (3.29)
where the optimal value for k is usually estimated empirically on the development set.

Additional improvements in ASR accuracy can be obtained by performing multiple passes over the speech data. For this purpose the decoder can generate and save multiple recognition hypotheses in the form of word lattices [START_REF] Richardson | Lattice-based search strategies for large vocabulary speech recognition[END_REF]. A word lattice consists of a set of nodes representing points in time and a set of directed arcs representing word hypotheses. Each arc can contain information about a word identity and corresponding acoustic and language model scores. Lattices can be rescored by using a higher-order (or another more advanced) language model. Also they can be transformed into another efficient representation called a confusion network [START_REF] Mangu | Finding consensus in speech recognition: word error minimization and other applications of confusion networks[END_REF].

End-to-end ASR

One important trend in current ASR technology is training deep end-to-end ASR systems [START_REF] Audhkhasi | Direct acoustics-to-word models for english conversational speech recognition[END_REF][START_REF] Bahdanau | End-to-end attention-based large vocabulary speech recognition[END_REF][START_REF] Chan | Listen, attend and spell: A neural network for large vocabulary conversational speech recognition[END_REF][START_REF] Chorowski | End-to-end continuous speech recognition using attention-based recurrent NN: First results[END_REF][START_REF] Collobert | Wav2letter: an end-to-end ConvNetbased speech recognition system[END_REF][START_REF] Fritz | End-to-end MAP training of a hybrid HMM-DNN model[END_REF][START_REF] Graves | Towards end-to-end speech recognition with recurrent neural networks[END_REF]Miao et al., 2015a;[START_REF] Ochiai | Multichannel end-to-end speech recognition[END_REF][START_REF] Yi | CTC regularized model adaptation for improving LSTM RNN based multi-accent mandarin speech recognition[END_REF]Zhang et al., 2016a[START_REF] Zhang | Towards end-to-end speech recognition with deep convolutional neural networks[END_REF][START_REF] Zhu | Learning multiscale features directly from waveforms[END_REF] that attempt to map an acoustic signal to a words sequence directly by means of neural network models. Some of these works aim to be exempted from the need of an HMM part, or presegmented training data, but still use lexicons, phoneme representations, LMs, or separate feature extractors, so not all of them are completely end-to-end.

One of the first major steps in this direction was introduced in [START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF] where, for the phoneme recognition task, a deep bidirectional LSTM RNN model was trained to map directly acoustic sequences to phonetics ones. This was done by using the connectionist temporal classification (CTC) objective function [START_REF] Graves | Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[END_REF]]. In the CTC approach, the alignment between the inputs and target labels is unknown. CTC can be implemented with a softmax output layer using an additional unit for the blank label / 0. The symbol / 0 corresponds to no output and is used to estimate the probability of outputting no label at a given time. The network is trained to optimize the total log-probability of all valid label sequences for training data. A set of valid label sequences for an input sequence is defined as the set of all possible label sequences of the input with the target labels in the correct order with repetitions and with label / 0 allowed between any labels. Targets for CTC training can be computed using finite state transducers (FSTs) [START_REF] Sak | Learning acoustic frame labeling for speech recognition with recurrent neural networks[END_REF], and the forward-backward algorithm can be used to calculate the CTC loss function.

State transition probability distribution and state priors are not required for CTC approach, in contrast to the hybrid DNN-HMM system. Several types of output units for CTC training have been explored in the literature, such as phones (or graphemes) [Miao et al., 2015a], words [START_REF] Audhkhasi | Direct acoustics-to-word models for english conversational speech recognition[END_REF] or grams [START_REF] Liu | Gram-CTC: Automatic unit selection and target decomposition for sequence labelling[END_REF]. Due to the large number of word outputs in acoustic-to-word CTC models, they require significantly more training data in comparison with traditional ASR systems [START_REF] Audhkhasi | Direct acoustics-to-word models for english conversational speech recognition[END_REF]. A maximum a-posteriori (MAP) training criterion instead of CTC was used in [START_REF] Fritz | End-to-end MAP training of a hybrid HMM-DNN model[END_REF] to train an end-to-end ASR system.

Most of the works devoted to end-to-end technology do not use any speaker adaptation techniques. In this thesis, we will implement our proposed adaptation techniques to one endto-end system, described in [Miao et al., 2015a], to explore how the end-to-end technology can benefit from speaker adaptation (Chapter 9).

Speech recognition performance evaluation

Performance evaluation is characterized by three elements: a criterion, a measure and a method [START_REF] Hirschman | Overview of evaluation in speech and natural language processing[END_REF]. In ASR the criterion is recognition accuracy, one of the most popular measures is word error rate (WER), and the method is comparing the sequence of recognized words with what was actually spoken and establishing alignment at the word level between these sequences by using a dynamic programming (DP) algorithm.

General evaluation scheme

LVCSR evaluation procedure contains two important steps:

1. Alignment of reference and recognized sequences of words. It is assumed that we have the text that was actually spoken (a reference). Alignment consists in establishing agreement at the word level between the original and recognized texts. The standard approach to alignment is the classical Levenshtein (edit distance) algorithm. If some additional information (such as time labels or phonological features of words) about reference is available, then it may be used in alignment process.

2. Computing the evaluation measure. After the alignment is processed, each word in reference and recognized sentences gets a status: correct, substitution, deletion (for a reference word) or insertion (for a recognized word). The evaluation measure is usually based on counts of substitution, deletion, insertion errors and correct words.

Alignment algorithms

Alignment of the original and recognized texts is usually performed by computing the Levenshtein distance using the DP algorithm. The Levenshtein distance is the minimal number (or weighted sum) of insertion, deletion and substitution word operations needed to transform the original text into the recognized one.

Existing alignment algorithms differ from each other by the way they compute weights for word operations. Let rec be a recognized word, ref -a word from the reference. There are two basic types of alignment [START_REF] Fiscus | The NIST speech recognition scoring toolkit (SCTK)[END_REF]].

1. Standard Levenshtein alignment (SLA). This approach is considered to be the standard method for LVCSR performance evaluation tasks. In general, SLA can be characterized by the loss function: (3.30) where N sub , N ins and N del denote the total number of word substitutions, insertions and deletions respectively; N cor -the number of correctly recognized words; w cor , w ins , w del , w sub -the constant weights assigned for operations of each type. Optimal weights (in the sense of minimizing WER) are: w cor = 0 , w ins = w del = w sub = 1. It was shown in [START_REF] Morris | From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition[END_REF]] that any choice of other weights requires special justification. This type of alignment (with some variations of weights) is the most commonly used. For example, NIST used it in Speech Recognition Scoring Toolkit for Speech-To-Text evaluation programs but sets up weights that equal (0,3,3,4) [START_REF] Fiscus | Rich transcription 2009 evaluation[END_REF]; in the HTK toolkit (for building HMMs) weights are (0,7,7,10) [START_REF] Young | The HTK book (for HTK version 3.4)[END_REF].

F = w cor N cor + w ins N ins + w del N del + w sub N sub ,
2. Time-mediated alignment (TMA) [START_REF] Fiscus | The NIST speech recognition scoring toolkit (SCTK)[END_REF]]. TMA is a variation of DP alignment where word-to-word distances are based on the time of occurrence for individual words. TMA is computed by replacing the standard word-to-word distance weights with measures based on the times of beginnings and endings of words. The formulas for time-mediated word-to-word distances are3 :

               D(correct) = |B(re f ) -B(rec)| + |E(re f ) -E(rec)| D(insertion) = E(rec) -B(rec) D(deletion) = E(re f ) -B(re f ) D(substitution) = |B(re f ) -B(rec)| + |E(re f ) -E(rec)| + 0.001, (3.31)
where B(x) and E(x) are the beginning and ending time marks of word x.

Other types of alignment reported in literature, such as word-weight-mediated, phonological [START_REF] Fisher | Better alignment procedures for speech recognition evaluation[END_REF], alignment based on multiple dimension Levenshtein edit distance [START_REF] Fiscus | Multiple dimension levenshtein edit distance calculations for evaluating automatic speech recognition systems during simultaneous speech[END_REF], modified TMA [Khokhlov and Tomashenko, 2011] and others are developed for particular applications and are not considered in this thesis.

Measures for evaluation

Most of the measures for LVCSR are based on magnitudes: N sub , N ins , N del and N cor . The common measure is WER, which is defined as the percentage ratio of the total number of word errors made by an ASR system to the total number of words in the reference:

WER = N sub + N ins + N del N re f × 100%, (3.32)
where N re f is the total number of words in the reference text. WER has been the most popular measure in continuous speech recognition assessment since the 1980s. NIST used WER as the primary measure in comparative technology evaluation programmes for ASR [START_REF] Fiscus | Rich transcription 2009 evaluation[END_REF]. Other measures were suggested in literature, such as: weighted WER [START_REF] Fiscus | The NIST speech recognition scoring toolkit (SCTK)[END_REF], sentence error rate (SER); word correct or incorrect (WCI) and sentence correct or incorrect (SCI) [START_REF] Strik | Comparing the recognition performance of CSRs: in search of an adequate metric and statistical significance test[END_REF]; number of errors per sentence (NES) [START_REF] Strik | Comparing the recognition performance of CSRs: in search of an adequate metric and statistical significance test[END_REF]; match error rate (MER) [START_REF] Morris | From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition[END_REF]; relative information lost (RIL) [START_REF] Maier | Evaluating RIL as basis of automatic speech recognition devices and the consequences of using probabilistic string edit distance as input[END_REF][START_REF] Morris | From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition[END_REF]; word information lost (WIL) and word information preserved (WIP) measures [START_REF] Morris | An information theoretic measure of sequence recognition performance[END_REF]; individual WER (IWER) [START_REF] Goldwater | Which words are hard to recognize? Prosodic, lexical, and disfluency factors that increase speech recognition error rates[END_REF] and others. In this thesis we will consider WER as a basic metric for evaluation.

Absolute and relative improvement

When several ASR systems are compared it is a common practice to report the absolute and relative change in WER values, calculated between two systems with WERs: WER 1 and WER 2 , as follows:

• Absolute WER reduction

∆ abs WER = WER 1 -WER 2 (3.33)
• Relative WER reduction

∆ rel WER = WER 1 -WER 2 WER 1 × 100%.
(3.34)

Statistical significance tests and confidence intervals

In the ASR system development, it is often important to know whether the observed difference in the performance is statistically significant or not. For this purpose several approaches have been proposed in the literature [START_REF] Bisani | Bootstrap estimates for confidence intervals in asr performance evaluation[END_REF][START_REF] Gillick | Some statistical issues in the comparison of speech recognition algorithms[END_REF][START_REF] Pallet | Tools for the analysis of benchmark speech recognition tests[END_REF]. In paper [START_REF] Gillick | Some statistical issues in the comparison of speech recognition algorithms[END_REF]] a matched-pairs test for comparing outputs of two ASR systems is described, which is applicable under the assumptions that the output of an ASR system can be divided into segments, in which the errors are independent of the errors in other segments. In paper [START_REF] Bisani | Bootstrap estimates for confidence intervals in asr performance evaluation[END_REF] a bootstrap method for significance analysis is presented. The idea is based on creating the replications of a statistic by random sampling from the test dataset with replacement (so-called Monte Carlo simulation).

In this thesis, if the results in terms of WER are reported to be significantly different, that means that the hypothesis is tested with significance level α = 5%. In addition, as it is done for example in [START_REF] Bell | Full covariance modelling for speech recognition[END_REF][START_REF] Povey | Discriminative training for large vocabulary speech recognition[END_REF][START_REF] Świętojański | Learning Representations for Speech Recognition using Artificial Neural Networks[END_REF] for baseline systems, we approximate the required WER change in terms of confidence intervals (CIs), calculated with α = 5%. For clarity, we report these CIs in some experiments for baseline systems (not for all systems to avoid redundancy).

Chapter 4 Acoustic adaptation

This chapter provides a review of adaptation techniques for both GMM and DNN acoustic models.

Speech variability and adaptation

Differences between training and testing conditions may significantly degrade recognition accuracy in ASR systems. The source of these differences may be of various nature [START_REF] Benzeghiba | Automatic speech recognition and speech variability: A review[END_REF] -speaker, channel and environment. Speech variations may occur both across different speakers (across-speaker variability) and within one speaker (within-speaker variability).

The across-speaker variability is connected with such factors as age, gender, foreign or regional accents, etc. In [START_REF] Huang | Analysis of speaker variability[END_REF], with the use of statistical analysis methods, it was demonstrated that the two principal components in variation correspond to the gender and accent. From the physiological point of view, another speaker-dependent factor is the shape of the vocal tract, that determines the potential range within which parameters of voice for a given speaker may vary.

The within-speaker variability [START_REF] Karlsson | Within-speaker variability due to speaking manners[END_REF] appears, for example, when a speaker speaks at different rates. In the past it was shown that variations in speaking rate can degrade recognition performance in ASR systems [START_REF] Mirghafori | Towards robustness to fast speech in ASR[END_REF]. This effect is typically more significant for fast speech than for slow speech, that is, the higher the speaking rate, the higher the error rate in ASR systems. Fast speech is different from normal or slow speech in several aspects, such as acoustic-phonetic and phonological characteristics. Many methods for compensating the effects of speaking rate variability were proposed in the literature [START_REF] Ban | Speaking rate dependent multiple acoustic models using continuous frame rate normalization[END_REF][START_REF] Mirghafori | Towards robustness to fast speech in ASR[END_REF][START_REF] Nanjo | Speaking rate dependent acoustic modeling for spontaneous lecture speech recognition[END_REF][START_REF] Siegler | Measuring and compensating for the effects of speech rate in large vocabulary continuous speech recognition[END_REF][START_REF] Tomashenko | Speaking rate estimation based on deep neural networks[END_REF].

Another important factor, which can be attributed to within-speaker variability, is the speech style. Reading, dictation or spontaneous speech may have different acoustic characteristics for the same speaker. In spontaneous speech, reductions of some phonemes or syllables often happen. Also it usually contains various disfluencies, such as repetitions, hesitations, filled pauses, false starts and others [START_REF] Liu | Enriching speech recognition with automatic detection of sentence boundaries and disfluencies[END_REF], which should be taken into account when building an ASR system. A speaker can speak with different volume -louder, quieter, in a whisper [START_REF] Jou | Adaptation for soft whisper recognition using a throat microphone[END_REF], or shout. Emotional state of a speaker also has an impact on the speech signal [START_REF] Scherer | Vocal communication of emotion: A review of research paradigms[END_REF].

Adaptation is an efficient way to reduce mismatches between the models and the data from a particular speaker, channel or another factor.

Three types of acoustic models (AMs) can be used in ASR systems:

• Speaker independent (SI) AM is trained on a training set of acoustic data from multiple speakers.

• Speaker dependent (SD) AM is trained on data from a single target speaker. Given enough training data, a SD AM can provide WERs several times lower than a SI AM [START_REF] Lee | A study on speaker adaptation of the parameters of continuous density hidden markov models[END_REF].

• Speaker adapted (SA) AM is initially trained on data from multiple speakers, and then is adapted using a comparatively small amount of data from a target speaker.

The aim of AM adaptation is to reduce mismatches between training and testing acoustic conditions and improve the accuracy of the ASR system for a target speaker or channel using a limited amount of adaptation data from the target acoustic source.

Adaptation types and modes

Acoustic adaptation algorithms can be divided in two types:

• Model-based adaptation transforms the model parameters in order to optimize a certain criterion, for example, to maximize posterior probability or likelihood on the adaptation data.

• Feature-space adaptation transforms the acoustic feature vectors and does not require modification of the parameters of the acoustic model, so usually it is more suitable for real-time on-line ASR systems.

Speaker adaptation techniques can be applied in various modes:

• In the supervised mode, the true transcriptions of the adaptation data are supposed to be known.

• In the unsupervised mode there are no available (correct) transcriptions of the adaptation data. In this case transcripts for adaptation can be obtained from the first decoding pass by a SI or another model. Since the adaptation with erroneous transcriptions from the first decoding pass may degrade the performance of the unsupervised adaptation compared to the supervised one [START_REF] Pitz | Improved MLLR speaker adaptation using confidence measures for conversational speech recognition[END_REF], an important question for some adaptation techniques is how to deal with these transcription errors. One popular solution is the use of confidence measures1 [START_REF] Anastasakos | The use of confidence measures in unsupervised adaptation of speech recognizers[END_REF][START_REF] Gollan | Confidence scores for acoustic model adaptation[END_REF][START_REF] Pitz | Improved MLLR speaker adaptation using confidence measures for conversational speech recognition[END_REF][START_REF] Tomashenko | On the use of Gaussian mixture model framework to improve speaker adaptation of deep neural network acoustic models[END_REF][START_REF] Uebel | Improvements in linear transform based speaker adaptation[END_REF][START_REF] Wang | A confidence-score based unsupervised MAP adaptation for speech recognition[END_REF].

. With regard to the use of the accumulated adaptation data, speaker adaptation can also be classified by the following modes [START_REF] Shinoda | Speaker adaptation techniques for automatic speech recognition[END_REF][START_REF] Woodland | Speaker adaptation for continuous density HMMs: A review[END_REF]]:

• Batch / off-line / static adaptation. This adaptation is performed after all the adaptation data are obtained.

• Incremental / on-line / dynamic adaptation [START_REF] Zhang | On-line incremental speaker adaptation with automatic speaker change detection[END_REF]] is performed each time, a new portion of adaptation data is obtained, so that the system continues to adapt over time.

A speaker diarization system may also be required to detect speaker changes. The choice of the most suitable adaptation mode depends on the application.

Normalization

Normalization attempts to minimize the effects of variations in speech. The most popular normalization techniques include:

• Cepstral mean and variance normalization (CMVN). Cepstral mean normalization (CMN) subtracts the average (cepstral) feature value. Generally, it reduces the sensitivity to channel variations. Cepstral variance normalization scales each feature coefficient to have a unit variance and empirically this has been shown to reduce sensitivity to additive noise [START_REF] Benesty | Springer handbook of speech processing[END_REF].

• Vocal-Tract-Length Normalization (VTLN) [START_REF] Lee | Speaker normalization using efficient frequency warping procedures[END_REF]. Variations in vocaltract length of different speakers cause variations in formant frequencies. VTLN is usually performed by warping the frequency-axis of the spectra of the given speaker by an corresponding warp factor before extracting cepstral features. Warp factors can be found by ML estimation of the warped utterances with respect to a given model and transcriptions.

Adaptation for GMM models

Many efficient adaptation algorithms have been developed for GMM-HMM AMs (see, for example, reviews in [START_REF] Lee | On adaptive decision rules and decision parameter adaptation for automatic speech recognition[END_REF][START_REF] Shinoda | Acoustic model adaptation for speech recognition[END_REF][START_REF] Woodland | Speaker adaptation for continuous density HMMs: A review[END_REF]). In this section we consider some of the most popular adaptation approaches, such as MAP, MLLR, as well as their modifications, and speaker space methods.

Maximum a posteriori (MAP) 4.2.1.1 Standard MAP approach

One of the most popular approaches for acoustic model adaptation is maximum a posteriori (MAP) adaptation [START_REF] Gauvain | Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains[END_REF], also referred to as Bayesian adaptation. Given some adaptation data O = (O 1 , . . . , O R ), MAP adaptation aims to maximize the following objective function:

F MAP (Λ Λ Λ) = R ∑ r=1 log p Λ Λ Λ (O r |φ r )p 0 (Λ Λ Λ), (4.1)
where φ r = φ (W r ), as before in Section 3.3.3, is a composite HMM model, corresponding to the (correct) transcription of the training sentence O r and W r is a sequence of words in this transcription. In comparison with the ML objective function (see Formula (3.19)), the likelihood in MAP-estimation is weighted by the prior distribution of the parameters p 0 (Λ Λ Λ).

In MAP adaptation a SI model is used as a prior probability distribution over model parameters. Let m denote the index of a Gaussian component in the SI acoustic model (AM), and µ µ µ m the mean of this Gaussian. Then the MAP estimation of the mean vector is

µ µ µ m = τµ µ µ m + ∑ t γ m (t)o t τ + ∑ t γ m (t) , (4.2)
where τ is the parameter that controls the balance between the maximum likelihood estimate of the mean and its prior value; γ m (t) is the posterior probability of Gaussian component m at time t. As can be seen from Formula (4.2), the smaller the occupation likelihood of a Gaussian component γ m = ∑ t γ m (t), the closer the mean MAP estimate remains to the SI component mean µ µ µ m . In MAP adaptation, every single mean component in the system is updated with a MAP estimate, based on the prior mean, the weighting, and the adaptation data.

An important advantage of MAP adaptation consists in the fact that, when the amount of adaptation data increases, the parameters asymptotically converge to speaker-dependent performance. On the other hand, one drawback of MAP adaptation is that it requires more adaptation data to be effective when compared to transformation-based techniques, such as maximum likelihood linear regression (MLLR) (see Section 4.2.2), because MAP adaptation is defined at the component level and adapts each Gaussian individually. When larger amounts of adaptation data become available, MAP begins to perform better than MLLR. The two adaptation approaches can be combined to improve the performance (see, for example, [START_REF] Digalakis | Speaker adaptation using combined transformation and Bayesian methods[END_REF]).

Other approaches developed to make MAP adaptation more robust when only a small adaptation set is available include vector field smoothing (VFS) [START_REF] Ohkura | Speaker adaptation based on transfer vector field smoothing with continuous mixture density HMMs[END_REF][START_REF] Takahashi | Vector-field-smoothed bayesian learning for fast and incremental speaker/telephone-channel adaptation[END_REF][START_REF] Tonomura | Speaker adaptation based on transfer vector field smoothing using maximum a posteriori probability estimation[END_REF], structural maximum a posteriori (SMAP) [Shinoda andLee, 1997, 2001], and the combination of MAP estimation and weighted neighbor regression (WNR) in the so-called MAP-WNR adaptation [START_REF] He | Speaker adaptation based on combination of MAP estimation and weighted neighbor regression[END_REF].

Vector field smoothing (VFS)

The vector field smoothing (VFS) technique [START_REF] Ohkura | Speaker adaptation based on transfer vector field smoothing with continuous mixture density HMMs[END_REF][START_REF] Takahashi | Vector-field-smoothed bayesian learning for fast and incremental speaker/telephone-channel adaptation[END_REF][START_REF] Tonomura | Speaker adaptation based on transfer vector field smoothing using maximum a posteriori probability estimation[END_REF] provides a solution to overcome one limitation of MAP adaptation and improves its performance when only very small amount of adaptation data is available. The VFS technique is based on the assumption that one speaker acoustic feature space can be continuously transfer to another one. Hence, it solves the problem of adapting the parameters in case of unseen adaptation data. The VFS algorithm is applied to the Gaussian mean vectors and consists of three steps [START_REF] Tonomura | Speaker adaptation based on transfer vector field smoothing using maximum a posteriori probability estimation[END_REF]:

• Estimation of transfer vectors. The transfer vectors represent the difference of the mean vectors between the initial SI model µ µ µ m and the target MAP-model µ µ µ m :

v m = µ µ µ m -µ µ µ m , (4.3)
where m ∈ G a (G a is the set of Gaussian distributions, which have associated adaptation data).

• Interpolation of transfer vectors. Interpolation is applied for those Gaussian components µ µ µ n , that do not have associated adaptation data (n ∈ G a ):

v n = ∑ k∈N(n) λ n,k v k ∑ k∈N(n) λ n,k , (4.4)
where N(n) is the set of nearest neighbors to µ µ µ n . The set of nearest neighbors N(n) is defined based on the Euclidean distance between mean vectors of Gaussian components and µ µ µ n . The weighting coefficient λ n,k depends on the distance between µ µ µ n and µ µ µ k . Then, µ µ µ n is estimated with the obtained transfer vector v n , as follows:

µ µ µ n = µ µ µ n + v n (4.5)
• Smoothing of transfer vectors. For all transfer vectors v m (m ∈ G a ), which Gaussian components are present in the adaptation set, a smoothing is performed as follows:

v s m = ∑ k∈N(m) λ m,k v k ∑ k∈N(m) λ m,k , (4.6)
µ µ µ s m = µ µ µ m + v s m . (4.7)

Structural maximum a posteriori (SMAP)

Structural maximum a posteriori (SMAP) adaptation is another approach to improve the MAP estimates obtained when the amount of adaptation data is small [Shinoda andLee, 1997, 2001]. In SMAP adaptation, parameters are shared by using a tree structure. First, a tree of Gaussian distributions is build using Kullback-Leibler divergence as a distance between mixture components. The root node corresponds to the whole acoustic space, and each leaf corresponds to a single Gaussian in the AM. Parameters of parent nodes are used as priors, and the MAP estimation is performed from the root to the leaves in a cascade way.

Maximum likelihood linear regression (MLLR)

Maximum likelihood linear regression (MLLR) is an alternative algorithm to AM adaptation, in which a set of linear transforms is estimated to map a SI model to a new adapted model in such a way that the likelihood of the adaptation data is maximized [START_REF] Gales | Maximum likelihood linear transformations for HMM-based speech recognition[END_REF][START_REF] Leggetter | Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models[END_REF], given the transformed Gaussian parameters. The MLLR can be applied either only to the means of Gaussians, or to the both means and variances. The adapted estimate of the mean µ µ µ is calculated as:

µ µ µ = Aµ µ µ + b, (4.8)
where A is d × d transformation matrix (d is the dimensionality of acoustic feature vectors) and b is a bias vector.

If we denote W = [b A] and ξ ξ ξ = [wµ 1 . . . µ d ]
T -the extended mean vector, where w represents a fixed bias offset (which is usually equals to 1) [START_REF] Young | The HTK book (for HTK version 3.4)[END_REF], then Formula (4.8) becomes more compact:

µ µ µ = Wξ ξ ξ . (4.9)
The transformation W is estimated using the expectation maximization (EM) algorithm (the details can be found in [START_REF] Gales | Maximum likelihood linear transformations for HMM-based speech recognition[END_REF][START_REF] Leggetter | Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models[END_REF][START_REF] Young | The HTK book (for HTK version 3.4)[END_REF]).

Regression classes

Typically the transformation matrix is tied over a large number of Gaussians, that allows the system to adapt all these Gaussians using a small amount of adaptation data. In general, the MLLR adaptation is very flexible. If only a small amount of adaptation data is available, then a single global transformation for all Gaussians in a GMM-HMM model can be applied. However, increasing the amount of adaptation data, it is possible to improve MLLR adaptation with the use of several more specific transformation matrices, each of which is applied to a certain group of Gaussians. Gaussians can be grouped according to the phoneme classes to which they belong. A popular approach for using such classes is based on the construction of a regression class tree.

The regression class tree is built to cluster together those Gaussians, that are close in acoustic space, so that similar Gaussians can be transformed in a similar way. The tree can be built using a SI model and a centroid splitting algorithm with the Euclidean distance measure. The tree topology makes it possible to adapt those Gaussian components, for which there is no data in the adaptation set. Also, it allows the transformations to be applied dynamically, depending on the amount of adaptation data for a particular regression class.

Constrained MLLR (CMLLR) and feature-space MLLR (fMLLR)

There are two possible ways of transforming covariance matrices: unconstrained [START_REF] Gales | Mean and variance adaptation within the MLLR framework[END_REF] and constrained [START_REF] Digalakis | Speaker adaptation using constrained estimation of Gaussian mixtures[END_REF]. In the unconstrained MLLR, transformations for means and variances of Gaussians are estimated independently. On the contrary, in the CMLLR means and variances are adapted using the same transformation:

   µ µ µ = Aµ µ µ + b, Σ Σ Σ = AΣ Σ ΣA T . (4.10)
In practice, CMLLR is used more often as an adaptation technique for features, than for acoustic models. In this case typically one regression class is used and the adaptation method is referred as feature-space MLLR (fMLLR). For fMLLR, Formula (4.10) can be rewritten for features as follows:

o t = A -1 o t -A -1 b. (4.11)
MLLR can be combined with the MAP adaptation techniques. In [START_REF] Neumeyer | A comparative study of speaker adaptation techniques[END_REF]] various ML transformation-based techniques, as well as their combination with MAP, are compared. In [START_REF] Digalakis | Speaker adaptation using combined transformation and Bayesian methods[END_REF], the mean vectors, obtained by MLLR, are used as priors for mean vectors in MAP adaptation. Transformation-based methods work well when only a small amount of adaptation data is available. Bayesian methods perform better when the amount of adaptation data increases. Hence, their combination inherits the advantages of these two approaches. Maximum a posteriori linear regression (MAPLR) was proposed in [START_REF] Chesta | Maximum a posteriori linear regression for hidden markov model adaptation[END_REF][START_REF] Chou | Maximum a posterior linear regression with elliptically symmetric matrix variate priors[END_REF], where a standard MLLR approach was reformulated under MAP framework. A combination of SMAP (Section 4.2.1.3) and MLLR was investigated in [START_REF] Siohan | Structural maximum a posteriori linear regression for fast HMM adaptation[END_REF] in the method called SMAPLR.

Speaker space methods

Cluster adaptive training (CAT)

The basic idea of speaker clustering is to find in the training dataset a cluster of those speakers who are acoustically close to a given test speaker. Then this cluster can be used directly for speech recognition, in case when we have an HMM model corresponding to this cluster. A simple example of this approach is gender-dependent AMs. Also each of these training speakers can be adapted to the test speaker, and the resulting adapted data are used to estimate the new parameters of the model [START_REF] Padmanabhan | Speaker clustering and transformation for speaker adaptation in speech recognition systems[END_REF]].

Speaker clustering is performed according to the chosen distance metric between speakers. Popular metrics include Bhatacharyya distance [START_REF] Kosaka | Tree-structured speaker clustering for fast speaker adaptation[END_REF], likelihood distance measure [START_REF] Gales | Cluster adaptive training of hidden Markov models[END_REF] and others [START_REF] Hazen | A comparison of novel techniques for instantaneous speaker adaptation[END_REF][START_REF] Yoshizawa | Unsupervised speaker adaptation based on sufficient HMM statistics of selected speakers[END_REF].

Cluster adaptive training (CAT) [START_REF] Gales | Cluster adaptive training of hidden Markov models[END_REF] can be considered as an extension of speaker clustering. Instead of a SI model, a set of cluster-specific models is trained on more homogenous datasets. In [START_REF] Gales | Cluster adaptive training of hidden Markov models[END_REF]] speaker model's mean parameters are estimated in the form of a linear combination of all the cluster means. The Gaussian component variances and priors are assumed to be the same for all clusters. At recognition time, a linear combination of models is selected where the set of interpolation weights gives a speaker-specific transform.

Eigenvoice-based adaptation

Eigenvoice approach, proposed in [START_REF] Kuhn | Rapid speaker adaptation in eigenvoice space[END_REF], also represents the adapted model in the form of a weighted sum of a small number of basis eigenvoice vectors obtained from a set of reference speakers. These eigenvoices are found using principal components analysis (PCA) for set of supervectors obtained from all means in the set of SD HMM systems. The eigenvoices with the largest eigenvalues are selected as a basis set. These vectors are orthogonal to each other and represent the most important components of variation between the reference speakers.

Speaker adaptive training (SAT)

Speaker adaptive training (SAT) aims to provide a more suitable model for speaker adaptation. A method, that explicitly compensates for the inter-speaker variations in the HMM parameter estimation process, is introduced in [START_REF] Anastasakos | A compact model for speaker-adaptive training[END_REF]. First an initial SI model is trained. Then for each speaker in the training dataset, MLLR mean (or fMLLR) transformations are estimated. Then using the obtained SD transforms, the parameters of the initial models (means, variances and mixture weights) are re-estimated on the whole training dataset. The SAT procedure tends to result in more compact AMs with higher likelihoods on the training set and smaller variances. Usually SAT AMs provide significantly better results with adaptation, in comparison with SI models.

Adaptation for DNN models

Adaptation of DNN acoustic models is a rapidly developing research area. In recent years DNNs have replaced conventional GMMs in most state-of-the-art ASR systems, because it has been shown that DNN-HMMs outperform GMM-HMMs in different ASR tasks [Hinton et al., 2012a]. Many adaptation algorithms that have been developed for GMM-HMM systems (see Section 4.2) cannot be easily applied to DNNs because of the different nature of these models. GMM is a generative model and it fits the training data so that the likelihood of the data given the model is maximized. In contrast, DNN is a discriminative model. Since DNN parameter estimation uses discriminative criteria, it is more sensitive to segmentation errors and can be less reliable for unsupervised adaptation. Various adaptation methods have been developed for DNNs. These methods, as for GMM-HMM AMs, can also be categorized in two broad classes, feature-space and model-based.

Model-based adaptation methods rely on direct modifications of DNN model parameters.

In [Swietojanski et al., 2016;Swietojanski and Renals, 2014], learning speaker-specific hidden unit contributions (LHUC) was proposed. The main idea of LHUC is to directly parametrize amplitudes of hidden units, using speaker-dependent amplitude parameters. The idea of learning amplitudes of activation functions was proposed earlier in [START_REF] Trentin | Networks with trainable amplitude of activation functions[END_REF]. Other model-based DNN adaptation techniques include linear transformations (Section 4.3.1), adaptation using regularization techniques (Section 4.3.2), subspace methods (Section 4.3.3) and others.

Feature-space adaptation methods operate in the feature space and can either transform input features for DNNs, as it is done, for example, in fMLLR adaptation [Seide et al., 2011a] or use auxiliary features (Section 4.3.6).

Linear transformation

One of the first adaptation methods developed for neural networks was linear transformation that can be applied at different levels of the DNN-HMM system:

• Input features, as in the linear input network (LIN) transformation [START_REF] Abrash | Connectionist speaker normalization and adaptation[END_REF][START_REF] Gemello | Adaptation of hybrid ANN/HMM models using linear hidden transformations and conservative training[END_REF][START_REF] Li | Comparison of discriminative input and output transformations for speaker adaptation in the hybrid NN/HMM systems[END_REF][START_REF] Neto | Speaker-adaptation for hybrid HMM-ANN continuous speech recognition system[END_REF][START_REF] Trmal | Adaptation of a feedforward artificial neural network using a linear transform[END_REF] or feature-space discriminative linear regression (fDLR) [Seide et al., 2011a;[START_REF] Yao | Adaptation of contextdependent deep neural networks for automatic speech recognition[END_REF];

• Activations of hidden layers, as in the linear hidden network (LHN) transformation [START_REF] Gemello | Adaptation of hybrid ANN/HMM models using linear hidden transformations and conservative training[END_REF];

• Softmax layer, as in the linear output network (LON) transformation [START_REF] Li | Comparison of discriminative input and output transformations for speaker adaptation in the hybrid NN/HMM systems[END_REF] or in the output-feature discriminative linear regression (oDLR) [START_REF] Yao | Adaptation of contextdependent deep neural networks for automatic speech recognition[END_REF].

Wherever the transformation is applied, weights are initialized with an identity matrix and then trained by minimizing the error at the output of the DNN system while keeping the weights of the original DNN fixed. Adaptation of hybrid tied-posterior acoustic models [START_REF] Stadermann | Two-stage speaker adaptation of hybrid tied-posterior acoustic models[END_REF] can also be considered a linear transform of posteriors. The authors of [START_REF] Dupont | Fast speaker adaptation of artificial neural networks for automatic speech recognition[END_REF]] describe a method based on linear transformation in the feature space and PCA.

Linear input network (LIN)

In the LIN [START_REF] Abrash | Connectionist speaker normalization and adaptation[END_REF][START_REF] Gemello | Adaptation of hybrid ANN/HMM models using linear hidden transformations and conservative training[END_REF][START_REF] Li | Comparison of discriminative input and output transformations for speaker adaptation in the hybrid NN/HMM systems[END_REF][START_REF] Neto | Speaker-adaptation for hybrid HMM-ANN continuous speech recognition system[END_REF][START_REF] Trmal | Adaptation of a feedforward artificial neural network using a linear transform[END_REF] and fDLR [Seide et al., 2011a;[START_REF] Yao | Adaptation of contextdependent deep neural networks for automatic speech recognition[END_REF] approaches, the linear transformation is applied to the input features, as shown in Figure 4.1. These adaptation techniques aim to linearly transform the SD features of a particular speaker to better match the SI DNN AM.

Adaptation layer LIN

Hidden layers

Output vector

Input vector In LIN adaptation the input vector h 0 ∈ R N 0 is transformed to h 0 LIN as follows:

h 0 LIN = W LIN h 0 + b LIN , (4.12)
where N 0 is the size of the input layer, W LIN ∈ R N 0 ×N 0 is the weight matrix and b LIN ∈ R N 0 is the bias vector. Since in ASR the input vector h 0 typically composed of a sequence of several speech frames:

h 0 = o t = [o t-T , . . . , o t+T ],
when the adaptation set is small, it is more preferable [START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF] to use per-frame transformation, with smaller number of parameters:

o LIN = W LIN o o + b LIN o , (4.13)
where W LIN o ∈ R D×D is the weight matrix and b LIN o ∈ R D is the bias vector, D is the dimension of acoustic feature vectors (N 0 = (2T + 1)D), and the final transformed input feature vector is constructed as:

h 0 LIN = o LIN t = [o LIN t-T , . . . , o LIN t+T ].
Based on the LIN, the adaptation parameter estimation via MAP linear regression was studied in [START_REF] Huang | Feature space maximum a posteriori linear regression for adaptation of deep neural networks[END_REF] and is referred to as feature-space MAP LIN (fMAPLIN).

Linear output network (LON)

In LON [START_REF] Li | Comparison of discriminative input and output transformations for speaker adaptation in the hybrid NN/HMM systems[END_REF] and output-feature discriminative linear regression (oDLR) [START_REF] Yao | Adaptation of contextdependent deep neural networks for automatic speech recognition[END_REF][START_REF] Yu | KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition[END_REF] adaptation algorithms, the linear transformation is applied to the softmax layer, as shown in Figure 4.2. The LON adaptation can be applied in two different ways: either before the calculation of the softmax layer weights, or after it. Both approaches lead to very similar linear transforms [START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF], but the number of parameters can differ a lot depending on the difference between the hidden and output layer dimensions.

Adaptation layer LON

Hidden layers

Output vector

Input vector In a similar manner, linear transformations can also be applied to the hidden layers, as in LHN [START_REF] Gemello | Adaptation of hybrid ANN/HMM models using linear hidden transformations and conservative training[END_REF]. Illustration of LHN adaptation is given in Figure 4.3. The

Adaptation layer LHN

Hidden layers

Output vector

Input vector hierarchical MAP approach applied to the LHN was proposed in [START_REF] Huang | Maximum a posteriori adaptation of network parameters in deep models[END_REF].

The choice of the appropriate adaptation techniques depends on the task, the number of parameters, and the size of the adaptation set. An experimental comparison of different linear transformation approaches can be found in [START_REF] Gemello | Adaptation of hybrid ANN/HMM models using linear hidden transformations and conservative training[END_REF][START_REF] Huang | Maximum a posteriori adaptation of network parameters in deep models[END_REF][START_REF] Li | Comparison of discriminative input and output transformations for speaker adaptation in the hybrid NN/HMM systems[END_REF].

Regularization techniques

In order to improve generalization during the adaptation regularization techniques, such as L 2 (or L 2 -prior) regularization [START_REF] Li | Regularized adaptation of discriminative classifiers[END_REF][START_REF] Liao | Speaker adaptation of context dependent deep neural networks[END_REF], Kullback-Leibler divergence (KLD) regularization [START_REF] Huang | Regularized sequence-level deep neural network model adaptation[END_REF][START_REF] Tóth | Adaptation of DNN Acoustic Models Using KL-divergence Regularization and Multi-task Training[END_REF][START_REF] Yu | KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition[END_REF], conservative training [START_REF] Albesano | Adaptation of artificial neural networks avoiding catastrophic forgetting[END_REF][START_REF] Gemello | Adaptation of hybrid ANN/HMM models using linear hidden transformations and conservative training[END_REF], reduction of the number of adapted parameters [START_REF] Ochiai | Speaker adaptive training using deep neural networks[END_REF][START_REF] Stadermann | Two-stage speaker adaptation of hybrid tied-posterior acoustic models[END_REF]Xue et al., 2014a] and others [START_REF] Ochiai | Speaker adaptive training using deep neural networks[END_REF][START_REF] Shen | Comparison of regularization constraints in deep neural network based speaker adaptation[END_REF][START_REF] Xue | Speaker adaptation of hybrid NN/HMM model for speech recognition based on singular value decomposition[END_REF] are used.

L 2 regularization

The key idea of the DNN adaptation with L 2 regularization [START_REF] Li | Regularized adaptation of discriminative classifiers[END_REF][START_REF] Liao | Speaker adaptation of context dependent deep neural networks[END_REF][START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF] is to update DNN parameters or only part of them by using the modified training objective function F L 2 (W, b; T), which takes into account the difference between the parameters of SI and speaker-adapted models. This difference is calculated using L 2 norm as follows:

R 2 (W SI -W) = ∥vec(W SI -W)∥ 2 2 = L ∑ l=1 vec(W l SI -W l ) 2 2 , (4.14)
where W SI and W correspond to the original SI and adapted models respectively, and vec(W l ) 2 2 is the Frobenious norm of matrix W l . The objective function is modified with this L 2 regularization term:

F L 2 (W, b; T) = F(W, b; T) + λ R 2 (W SI -W) , (4.15)
where λ is the weight regularization factor, that controls the impact of the original SI model to the adaptation criterion. The larger the regularization term is, the closer the adapted model is forced to be to the SI one.

Kullback-Leibler divergence (KLD) regularization

Using the Kullback-Leibler divergence (KLD) regularization [START_REF] Huang | Regularized sequence-level deep neural network model adaptation[END_REF][START_REF] Tóth | Adaptation of DNN Acoustic Models Using KL-divergence Regularization and Multi-task Training[END_REF][START_REF] Yu | KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition[END_REF] is another way to adapt the model. The idea of this adaptation technique consists in keeping the senone distribution, estimated from the adapted model, to be close to the distribution estimated from the SI model. To measure the distance between these distributions, KLD is used [START_REF] Yu | KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition[END_REF]. By adding this KLD as a regularization term to the adaptation criterion and removing the terms unrelated to the model parameters, the objective function with the KLD regularization can be written as follows [START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF]:

F KLD (W, b; T) = (1 -λ )F(W, b; T) + λ R KLD (W SI , b SI ; W, b; T), (4.16)
where λ is the regularization weight,

R KLD (W SI , b SI ; W, b; T) = 1 M M ∑ m=1 C ∑ i=1 P SI (i|o m ; W SI , b SI ) log P(i|o m ; W, b), (4.17)
P SI (i|o m ; W SI , b SI ) and P(i|o m ; W, b) are the probabilities that the observation vector o m belongs to class i, estimated with the SI and adapted models respectively. Unlike in L 2regularization approach to adaptation, which imposes the constraints to the DNN parameters, in the KLD-regularization approach the constraints are applied to the output probabilities.

The regularization weight λ ∈ [0, 1] can be optimized on the development set. The large values of λ provide small impact from the adaptation data to the final model.

Reducing the number of adapted parameters

Alternative regularization technique for preventing overfitting during the adaptation is to adapt not the entire DNN model, but only a subset of its parameters. In [START_REF] Stadermann | Two-stage speaker adaptation of hybrid tied-posterior acoustic models[END_REF]] only a sub-set of the hidden units with maximum variance, computed on the adaptation data, is retrained.

The number of speaker-specific parameters is reduced in [Xue et al., 2014a;[START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF] through factorization based on singular value decomposition (SVD). One approach proposed in [Xue et al., 2014a] is referred to as SVD bottleneck adaptation. It is based on applying the SVD to the fully connected weight matrix W m×n ∈ R m×n (where m ≥ n) :

W m×n = W 1 m×n Σ Σ Σ n×n (W 2 n×n ) T , (4.18)
where Σ Σ Σ n×n is a diagonal matrix with singular values of matrix W m×n on the diagonal. If

W m×n is a sparse matrix with rank r ≪ n, then Formula (4.18) can be rewritten as

W m×n ≈ W 3 m×r Σ Σ Σ r×r (W 4 n×r ) T = W 3 m×r W 5 r×n . (4.19)
Hence, the original matrix W m×n can be approximated with two smaller matrices: W 3 m×r and W 5 r×n , as shown in Figure (4.4). To apply an SVD bottleneck adaptation, an additional r-dimensional linear layer is added to the BN layer:

W m×n ≈ W 3 m×r S r×r W 5 r×n , (4.20)
where S r×r is initialized with the identity matrix for SI model. During the adaptation, this matrix S r×r is adapted to a particular speaker, while keeping matrices W 3 m×r and W 5 r×n fixed for every layer. Since in this case only r × r parameters have to be adapted instead of m × n, as r can be much smaller then m and n, this approach allows to adapt all layers of the DNN by adapting only small-sized matrices for each layer. Thus, such technique gives the possibility to decrease the required amount of adaptation data. In [START_REF] Xue | Speaker adaptation of hybrid NN/HMM model for speech recognition based on singular value decomposition[END_REF] the SVD also is applied on the weight matrices W m×n in the trained SI DNNs (Formula (4.18)), and then only the diagonal matrices with singular values Σ Σ Σ n×n are fine-tuned with the adaptation data. Since during the adaptation the weight matrices can be modified only by changing the singular values, this techniques proposes the solution for the over-fitting problem. Regularized SAT of subsets of DNN parameters is explored in [START_REF] Ochiai | Speaker adaptive training using deep neural networks[END_REF].

SI BN layer

Conservative training

When a neural network trained with a large set of parameters has to be adapted with new data that does not appropriately represent the original training data, then a problem, addressed in the literature as catastrophic forgetting [START_REF] French | Catastrophic forgetting in connectionist networks[END_REF], occurs. In particular, this problem appears when some of the targets of the original NN are not present in the adaptations set. In this situation, the previously learned information in the NN can be distorted or forgotten during the adaptation.

The conservative training [START_REF] Albesano | Adaptation of artificial neural networks avoiding catastrophic forgetting[END_REF][START_REF] Gemello | Adaptation of hybrid ANN/HMM models using linear hidden transformations and conservative training[END_REF] proposes a solution to eliminate this problem. During the adaptation, the conservative training does not assigns zero value to the targets of the missing units, using instead as target values the outputs computed by the original network.

Subspace methods

Subspace adaptation methods aim to find a speaker subspace and then construct the adapted DNN parameters as a point in the subspace.

In [START_REF] Dupont | Fast speaker adaptation of artificial neural networks for automatic speech recognition[END_REF] an approach similar to the eigenvoice technique [START_REF] Kuhn | Rapid speaker adaptation in eigenvoice space[END_REF], was proposed for the fast speaker adaptation of NN AMs. It is based on using the affine transformation in the feature space. In contrast to the standard approach, where the whole transformation is adapted, in [START_REF] Dupont | Fast speaker adaptation of artificial neural networks for automatic speech recognition[END_REF] adaptation to a new speaker is performed using PCA by optimizing several (K) principal components, which represent the most meaningful speaker specific variations. A new speaker is represented by the parameter space described by the first K eigenvoices. Adaptation is performed by estimating the corresponding K eigenvoices coefficients. This allows to reduce the required amount of adaptation data.

As noticed in [START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF], this idea can be extended to a more general case and to other adaptation techniques, for example, to any other linear transformations (LON, LHN, etc). Given the set of S speakers, for each speaker an adaptation matrix W SA can be estimated (by any adaptation technique, considered before). If a = vec(W SA ) is the vectorization of the matrix W SA , then PCA can be done for the set of S vectors in the speaker space in order to obtain the eigenvectors, that define the principal adaptation matrices. It is assumed [START_REF] Dupont | Fast speaker adaptation of artificial neural networks for automatic speech recognition[END_REF][START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF]] that the number of speakers S in the training set is big enough, so that the new speaker can be represented as a point in the space, defined by these S speakers. A new speaker is defined as a linear combination of the eigenvectors:

a = a + Ug, (4.21)
where U = (u 1 , . . . , u S ) is the eigenvectors matrix; g is the projection of the adaptation vector onto principal directions; a is the mean of the adaptation parameters of all speakers.

The eigenvectors corresponding to small variances in the PCA are discarded after the PCA procedure, and the dimensionality of the speaker space can be reduced to K < S. In the reduced speaker space the adaptation vector a can be approximated as

a ≈ a + U g, (4.22)
where U = (u 1 , . . . , u K ) is the reduced eigenvectors matrix. In Formula (4.22), a and U are estimated on the training set using data of S speakers; and g is estimated for a new speaker on the adaptation set.

In [START_REF] Wu | Multi-basis adaptive neural network for rapid adaptation in speech recognition[END_REF] a multi-basis adaptive neural network is proposed, where a traditional DNN topology is modified and a set of sub-networks, referred as bases were introduced. This DNN has a common input layer and a common output layer for all the bases. Each basis has several fully-connected hidden layers and there is no connections between neurons from different bases. The outputs of bases are combined by linear interpolation using a set of adaptive weights. The adaptation to a given speaker can be performed through optimization of interpolation weights for this speaker. The idea of this approach was motivated by the CAT, developed for GMM AMs (Section 4.2.3.1). Paper [START_REF] Tan | Cluster adaptive training for deep neural network[END_REF] also investigates the CAT framework for DNNs. A subspace learning speaker-specific hidden unit contributions (LHUC) adaptation was proposed in [START_REF] Samarakoon | Subspace LHUC for fast adaptation of deep neural network acoustic models[END_REF]].

Factorized adaptation

Factorized adaptation [Li et al., 2014a;[START_REF] Qian | Neural network based multi-factor aware joint training for robust speech recognition[END_REF][START_REF] Samarakoon | Multi-attribute factorized hidden layer adaptation for DNN acoustic models[END_REF][START_REF] Tran | Factorized linear input network for acoustic model adaptation in noisy conditions[END_REF][START_REF] Yu | Factorized deep neural networks for adaptive speech recognition[END_REF] takes into account different factors that influence the speech signal. These factors can have different nature (speaker, channel, background noise conditions and others) and can be modeled explicitly before incorporating them into the DNN structure, for example, in the form of auxiliary features [Li et al., 2014a] (see also Section 4.3.6), such as i-vectors, or can be learnt jointly with the neural network AM [START_REF] Tang | Collaborative joint training with multitask recurrent model for speech and speaker recognition[END_REF]. The first case, when factors, such as noise or speaker information, are estimated explicitly from the training and testing data, and are then fed to the DNN AM, is also known as noise-aware or speaker-aware training correspondingly [START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF], or, in general case, as context-aware training [START_REF] Tang | Collaborative joint training with multitask recurrent model for speech and speaker recognition[END_REF].

Paper [Li et al., 2014a] proposed to incorporate the auxiliary features, consisting of average noise estimates, computed for each utterance, and noise-distorted speech, into the DNN AM. These features correspond to the two factors and are included directly to the input of the softmax layer. The general scheme of this approach is shown in Figure 4.5. This approach is claimed to be related [Li et al., 2014a] to the vector Taylor series (VTS) based adaptation [START_REF] Moreno | Speech recognition in noisy environments[END_REF] technique, developed for noise-robust ASR. Also this approach can be considered from the joint factor analysis (JFA) point of view [START_REF] Kenny | Joint factor analysis versus eigenchannels in speaker recognition[END_REF].

Auxiliary features, representing different factors, can also be included into the other parts of the DNN structure. For example, they can be appended to the input feature vectors. We will consider this case in more detail in Section 4.3.6.

In paper [START_REF] Yu | Factorized deep neural networks for adaptive speech recognition[END_REF] two types of factorized DNNs were introduced: joint and disjoint models. In the first model, hidden factors (speaker and environment conditions) and triphone tied states are modeled jointly. On the contrary, in the disjoint factorized model, factors and triphone tied states are estimated separately using different DNNs. However, as it was noticed in [START_REF] Yu | Automatic speech recognition: A deep learning approach[END_REF], the total number of parameters in such networks is typically too large to use them in real-world applications.

In [START_REF] Tran | Factorized linear input network for acoustic model adaptation in noisy conditions[END_REF] an extension of the LIN adaptation (see Section 4.3.1.1), so-called factorized LIN (FLIN), has been investigated for the case when adaptation data for a given

Output vector

Input vector

Hidden layers Factor 1 Factor N Features In [START_REF] Delcroix | Context adaptive deep neural networks for fast acoustic model adaptation[END_REF] a context adaptive DNN is proposed. This DNN contains one or several factorized hidden layers. Each factorized hidden layer contains several sub-layers, which represent different acoustic conditions. The output of the factorized layer is a weighted averaging over the outputs of all sub-layers, where weights are the posterior probabilities of the factor classes. Factorized hidden layer adaptation was also studied in [Samarakoon and Sim, 2016a,b].

Multi-factor aware joint DNN training was proposed in [START_REF] Qian | Neural network based multi-factor aware joint training for robust speech recognition[END_REF], where factors are dynamically estimated using a DNN. The input to the factor extractor networks are noisy corrupted features. Each factor is modeled using several-layer DNN with a BN layer in the A different approach called collaborative joint learning is proposed in [START_REF] Tang | Collaborative joint training with multitask recurrent model for speech and speaker recognition[END_REF] where the two models -for ASR and for speaker recognition task are trained in a collaborative manner, mutually improving the performance of each other.

Multi-task learning

The concept of multi-task learning (MTL) has recently been applied to the task of speaker adaptation in several works [Huang et al., 2015a;Li et al., 2015b;[START_REF] Swietojanski | Structured output layer with auxiliary targets for context-dependent acoustic modelling[END_REF] and has been shown to improve the performance of different model-based DNN adaptation techniques, such as LHN [Huang et al., 2015a] and LHUC [START_REF] Swietojanski | Structured output layer with auxiliary targets for context-dependent acoustic modelling[END_REF].

The basic idea of MTL framework, used in [START_REF] Bell | Multitask learning of context-dependent targets in deep neural network acoustic models[END_REF]Huang et al., 2015a;[START_REF] Swietojanski | Structured output layer with auxiliary targets for context-dependent acoustic modelling[END_REF] consists in adding to the neural network structure an additional auxiliary output layer or several layers, as shown in Figure 4.7. Each output layer is associated with a specific task. Typically, a main task is used during an ASR process, but a DNN can be trained or adapted using auxiliary (secondary) tasks. This auxiliary tasks (layers) correspond to different sets of targets and usually have a lower dimension than the main task. For example, the main task can represent the senones, which correspond to a large number of context-dependent (CD) triphone tied states, and the auxiliary task can represent CI monophone states or clusters of senones from the main task. This type of the DNN architecture can be trained using a MTL approach. MTL is a machine learning technique which allowed a classifier to learn several related tasks in parallel, using a shared representation [START_REF] Caruana | Multitask learning[END_REF]. The principal goal of MTL is to improve generalization by leveraging the information, contained in auxiliary tasks. In MTL DNN training, the error vector from each output layer is back-propagated to the same last HL. Then, the error vectors, corresponding to different output layers, are combined together in a linear combination. The combined error vector is further back-propagated to the previous HLs. During the adaptation, the auxiliary tasks can be used to deal with the data sparsity problem and unseen senones.

In [START_REF] Pironkov | Speaker-aware long short-term memory multi-task learning for speech recognition[END_REF] a speaker classification task was used as an auxiliary task to train RNN-LTSM AM. In [Pironkov et al., 2016a], in addition to the speaker classification task, the i-vector extraction auxiliary task was implemented.

A slightly different idea was proposed earlier in [START_REF] Price | Speaker adaptation of deep neural networks using a hierarchy of output layers[END_REF] in the form of special hierarchy of output layers, where tied triphone state layer are followed by monophone state layer. Figure 4.8 illustrates the DNN with an auxiliary output layer, added on the top of the original main output layer, as proposed in [START_REF] Price | Speaker adaptation of deep neural networks using a hierarchy of output layers[END_REF]. During adaptation, the DNN parameters are updated to predict posterior probabilities for the over HMM states, corresponding to the auxiliary monophone output layer, which has much lower dimension then the main triphone output layer. The errors are back-propagated through the original softmax layer to the rest of the DNN. Before adaptation, the weights of the hierarchy output layer are trained using SI data, keeping fixed the layers below. During adaptation, the weights of the original network are updated, using the full hierarchical structure, but keeping the weights in the hierarchy output layer fixed. During decoding, only the main original output triphone layer is used.

Output vector

Input vector

Hidden layers A different multi-task learning approach, called collaborative joint learning, is proposed in [START_REF] Tang | Collaborative joint training with multitask recurrent model for speech and speaker recognition[END_REF]. It relies on inter-task information propagation, which can improve the performance of each task by the auxiliary information derived from other tasks.

CD triphone states

CI monophone states

Hierarchy output vector

Auxiliary features

Using auxiliary features, such as i-vectors [START_REF] Gupta | I-vector-based speaker adaptation of deep neural networks for French broadcast audio transcription[END_REF][START_REF] Karanasou | Adaptation of deep neural network acoustic models using factorised i-vectors[END_REF][START_REF] Saon | Speaker adaptation of neural network acoustic models using i-vectors[END_REF][START_REF] Senior | Improving DNN speaker independence with i-vector inputs[END_REF], is another widely used approach in which the acoustic feature vectors are augmented with additional speaker-specific or channel-specific features computed for each speaker or utterance at both training and test stages. Another example of auxiliary features is the use of speaker-dependent BN features obtained from a speaker aware DNN used in a far field speech recognition task [Liu et al., 2014]. Alternative method is adaptation with speaker codes [START_REF] Abdel-Hamid | Fast speaker adaptation of hybrid nn/hmm model for speech recognition based on discriminative learning of speaker code[END_REF][START_REF] Huang | Speaker adaptation of RNN-BLSTM for speech recognition based on speaker code[END_REF][START_REF] Xue | Fast adaptation of deep neural network based on discriminant codes for speech recognition. Audio, Speech, and Language Processing[END_REF]. In [START_REF] Murali Karthick | Speaker adaptation of convolutional neural network using speaker specific subspace vectors of SGMM[END_REF] speaker-specific subspace vectors, obtained by the SGMM model, are used as auxiliary features to adapt CNN AMs.

I-vectors

The use of speaker identity vectors (or i-vectors) has become a popular approach for speaker adaptation of DNN AMs [START_REF] Gupta | I-vector-based speaker adaptation of deep neural networks for French broadcast audio transcription[END_REF][START_REF] Karanasou | Adaptation of deep neural network acoustic models using factorised i-vectors[END_REF][START_REF] Miao | Towards speaker adaptive training of deep neural network acoustic models[END_REF][START_REF] Saon | Speaker adaptation of neural network acoustic models using i-vectors[END_REF][START_REF] Senior | Improving DNN speaker independence with i-vector inputs[END_REF]. Originally i-vectors were developed for speaker verification and speaker recognition tasks [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF], and nowadays they have become a very common technique in these domains. I-vectors can capture the relevant information about the speaker's identity in a low-dimensional fixed-length representation [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF][START_REF] Saon | Speaker adaptation of neural network acoustic models using i-vectors[END_REF].

I-vector extraction

The acoustic feature vector o t ∈ R D can be considered as a sample, generated with a universal background model (UBM), represented as a GMM with K diagonal covariance Gaussians [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF][START_REF] Saon | Speaker adaptation of neural network acoustic models using i-vectors[END_REF]:

o t ∼ K ∑ k=1 c k N (•;µ µ µ k ,Σ Σ Σ k ) , (4.23)
where c k are the mixture weights, µ µ µ k are means and Σ Σ Σ k are diagonal covariances. The acoustic feature vector o t (s), belonging to a given speaker s is described with the distribution:

o t (s) ∼ K ∑ k=1 c k N (•;µ µ µ k (s),Σ Σ Σ k ) , (4.24)
where µ µ µ k (s) are the means of the GMM, adapted to the speaker s. It is assumed that there is a linear dependence between the SD means µ µ µ k (s) and the SI means µ µ µ k , which can be expressed in the form:

µ µ µ k (s) = µ µ µ k + T k w(s), k = 1, . . . , K, (4.25)
where T k ∈ R D×M is a factor loading matrix, corresponding to component k and w(s) is the i-vector, corresponding to speaker s2 . Each T k contains M bases, that span the subspace of the important variability in the component mean vector space, corresponding to component k.

The detailed description of how to estimate the factor loading matrix, given the training data {o t }, and how to estimate i-vectors w(s), given T k and speaker data {o t (s)}, can be found, for example, in [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF][START_REF] Saon | Speaker adaptation of neural network acoustic models using i-vectors[END_REF].

Integration of i-vectors into a DNN model

Various methods of i-vector integration into a DNN AM have been proposed in the literature.

The most common approach [START_REF] Gupta | I-vector-based speaker adaptation of deep neural networks for French broadcast audio transcription[END_REF][START_REF] Saon | Speaker adaptation of neural network acoustic models using i-vectors[END_REF][START_REF] Senior | Improving DNN speaker independence with i-vector inputs[END_REF] is to estimate i-vectors for each speaker (or utterance), and then to concatenate it with acoustic feature vectors, belonging to a corresponding speaker (or utterance). The obtained concatenated vectors are introduced to a DNN for training, as shown in Figure 4.9.

In the test stage i-vectors for test speakers also have to be estimated, and input in a DNN in the same manner.

Input vector

Hidden layers Output vector stacked acoustic vectors i-vector Figure 4.9 Using i-vectors for DNN adaptation: concatenation of i-vectors with input features Unlike acoustic feature vectors, which are specific for each frame, an i-vector is the same for a chosen group of acoustic features, to which it is appended. For example, i-vector can be calculated for each utterance, as in [START_REF] Senior | Improving DNN speaker independence with i-vector inputs[END_REF], or estimated using all the data of a given speaker, as in [START_REF] Saon | Speaker adaptation of neural network acoustic models using i-vectors[END_REF]. I-vectors encode those effects in the acoustic signal, to which an ASR system is desired to be invariant: speaker, channel and background noise. Providing to the input of a DNN the information about these factors makes it possible for a DNN to normalize the acoustic signal with respect to them.

An alternative approach of i-vector integration into the DNN topology is presented in [START_REF] Miao | Towards speaker adaptive training of deep neural network acoustic models[END_REF][START_REF] Miao | Speaker adaptive training of deep neural network acoustic models using i-vectors[END_REF], where an input acoustic feature vector is normalized though a linear combination of it with a speaker-specific normalization vector obtained from an i-vector, as shown in Figure 4.10. Firstly, an original SI DNN is built. Secondly, a small adaptation network is built, keeping a SI DNN fixed. This adaptation network is learnt to take i-vectors i s as input and generate SD linear shifts y s to the original DNN feature vectors. After the adaptation network is trained, the SD output vector y s is added to every original feature vector o t from speaker s:

o t = o t ⊕ y s , (4.26)
where ⊕ denotes element-wise addition. Finally, the parameters of the original DNN model are updated in the new feature space, while keeping the adaptation network unchanged. This gives a SAT DNN model. Similar approaches have been studied in [START_REF] Goo | Speaker normalization through feature shifting of linearly transformed i-vector[END_REF][START_REF] Lee | Semi-supervised speaker adaptation for in-vehicle speech recognition with deep neural networks[END_REF]. Also i-vector dependent feature space transformations were proposed in [Li and Wu, 2015a].

In paper [START_REF] Dimitriadis | An investigation on the use of i-vectors for robust ASR[END_REF] two different i-vector representations are presented: (1) noise i-vectors, estimated only from the noise component of the noisy speech signal; and (2) noisy i-vectors, estimated from noisy speech signal. Another type of factorized i-vectors is described in [START_REF] Karanasou | Adaptation of deep neural network acoustic models using factorised i-vectors[END_REF], where speaker and environmental i-vectors are used together.

In [START_REF] Garimella | Robust i-vector based adaptation of DNN acoustic model for speech recognition[END_REF] casual i-vectors are calculated using all previous utterances of a given speaker for accumulating sufficient statistics. Exponentially decaying weights are applied on previous speech frames to give more importance to the recent speech signal, in order to make i-vectors to be able to capture more recent speaker and channel characteristics.

Attribute-based i-vectors are explored in [START_REF] Zheng | Improving large vocabulary accented mandarin speech recognition with attribute-based i-vectors[END_REF] for accent speech recognition.

D-vectors

Deep vectors or d-vectors were originally proposed for text-dependent speaker verification [START_REF] Variani | Deep neural networks for small footprint text-dependent speaker verification[END_REF]. They also described speaker information and are derived by a DNN which is trained to predict speakers at frame-level. For extracting the d-vector, firstly, a DNN is trained to classify speakers at the frame-level. Then the averaged output activations of last hidden layer are used as the speaker representation. Paper [START_REF] Li | Modeling speaker variability using long short-term memory networks for speech recognition[END_REF] explores the idea of augmenting DNN inputs with d-vectors derived by LSTM projected (LSTMP) RNNs [START_REF] Sak | Long short-term memory recurrent neural network architectures for large scale acoustic modeling[END_REF]. Both the d-vector extraction and senone prediction are LSTMP based, and the whole network is jointly optimized with multi-task learning.

Speaker codes

Adaptation with speaker codes, proposed in [START_REF] Abdel-Hamid | Fast speaker adaptation of hybrid nn/hmm model for speech recognition based on discriminative learning of speaker code[END_REF][START_REF] Xue | Fast adaptation of deep neural network based on discriminant codes for speech recognition. Audio, Speech, and Language Processing[END_REF], relies on a joint training procedure for (1) an adaptation NN from the whole training set and (2) small speaker codes, estimated for each speaker only using data from that speaker, as shown in Figure 4.11, The speaker code is fed to the adaptation NN to form a nonlinear transformation in feature space to normalize speaker variations. This transformation is controlled by the speaker code. During adaptation, a new speaker code for a new speaker is learned, optimizing the performance on the adaptation data.

In [START_REF] Xue | Direct adaptation of hybrid DNN/HMM model for fast speaker adaptation in lvcsr based on speaker code[END_REF] instead of stacking an adaptation NN below the initial speaker independent NN and normalizing speakers features with speakers codes, speaker codes are fed directly to the hidden layers and the output layer of the initial NN and the output layer of the initial NN. In [START_REF] Huang | Speaker adaptation of RNN-BLSTM for speech recognition based on speaker code[END_REF] speaker adaptation with speaker codes was applied for RNN-BLSTM AMs.

Input vector

Adaptation based on GMMs

The most common way of combining GMM and DNN models for adaptation is using GMM-adapted features, for example fMLLR, as input for DNN training [START_REF] Kanagawa | Feature-space structural maplr with regression tree-based multiple transformation matrices for DNN[END_REF][START_REF] Parthasarathi | fMLLR based feature-space speaker adaptation of DNN acoustic models[END_REF][START_REF] Rath | Improved feature processing for deep neural networks[END_REF]Seide et al., 2011a]. In [START_REF] Lei | Deep neural networks with auxiliary Gaussian mixture models for real-time speech recognition[END_REF] likelihood scores from DNN and GMM models, both adapted in the feature space using the same fMLLR transform, are combined at the state level during decoding. Another method is temporally varying weight regression (TVWR) [START_REF] Liu | On combining DNN and GMM with unsupervised speaker adaptation for robust automatic speech recognition[END_REF], where DNN posteriors are transformed, using a regression model, into the time-varying scaling factors for the Gaussian weights.

However, none of these approaches can be considered as a universal method for transfer of adaptation algorithms from GMM models to DNNs. Development and investigation of a method, that can solve this problem, led us to GMM-derived features, which we use as input to train a DNN [START_REF] Khokhlov | [END_REF]Khokhlov, 2014b, 2015]. This approach will be presented and extensively explored in Chapters 6-10.

Chapter 5

Speech corpora

This chapter describes speech corpora and language models, used in the experiments throughout this thesis.

We perform a series of experiments to explore the proposed acoustic model adaptation framework using four different corpora, which are suitable for this purpose. This corpora differ among themselves in such characteristics as:

• language;

• training dataset size;

• testing dataset sizes;

• amount of data available for adaptation;

• vocabulary size;

• number of speakers;

• type of speech;

• quality of available text transcriptions.

It is important to explore the adaptation framework in different conditions to get a more complete picture of the approach, its properties, strengths and weaknesses, and boundaries of applicability. We carry out certain types of experiments for particular corpora. Below in this chapter we describe four speech corpora used in experiments.

Wall Street Journal (WSJ)

In this thesis, the WSJ0 (CSR-1) part of the Wall Street Journal Corpus (WSJ) [START_REF] Paul | The design for the wall street journal-based CSR corpus[END_REF] was used for experiments. This corpus consists of read speech in English with texts from the Wall Street Journal news. The part of the corpus used in the experiments was recorded with a Sennheiser close-talking microphone, 16 kHz. The phoneme set consists of 39 phonemes1 . This corpus was used for experiments in Chapter 7.

Training data

For acoustic models training we used 7 138 utterances from 83 speakers (42 male and 41 female) from the standard SI-84 training set, which correspond to approximately 15 hours (12 hours of speech and 3 hours of silence) recordings.

Adaptation data

Dataset si_et_ad consists of 320 utterances from 8 speakers (5 male and 3 female). We used this dataset in experiments with semi-supervised adaptation (Section 7.1.2).

Test data and language models

Evaluation was carried out on the two standard WSJ0 evaluation tests:

TED-LIUM

The experiments were conducted on the TED-LIUM corpus [START_REF] Rousseau | TED-LIUM: an automatic speech recognition dedicated corpus[END_REF][START_REF] Rousseau | Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks[END_REF]. The corpus is comprised of TED talks3 in English. It has been released by LIUM laboratory [START_REF] Rousseau | TED-LIUM: an automatic speech recognition dedicated corpus[END_REF] within the context of the participation in the International Workshop on Spoken Language Translation (IWSLT) 2011 evaluation campaign, and later extended [START_REF] Rousseau | Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks[END_REF].

We used the last (second) release of this corpus [START_REF] Rousseau | Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks[END_REF]. This publicly available dataset contains 1 495 TED talks that amount to 207 hours (141 hours from male and 66 hours from female) speech data from 1 242 speakers, 16kHz. The phoneme set is the same, as for the WSJ corpus (Section 5.1). This corpus is used for experiments in Chapters 8, 9 and 10.

Training and test data

For experiments with SAT and adaptation we removed from the original corpus data for speakers who had less than 5 minutes of data, and from the rest of the corpus we made four datasets: training set, development set and two test sets. Characteristics of the obtained datasets are given in Table 5.1. The motivation for creating the new test and development datasets was to obtain datasets that are more representative and balanced in characteristics (gender, duration) than the original ones and more suitable for adaptation experiments.

Language models

For evaluation two different LMs are used:

1. LM-cantab is the publicly available 3-gram LM cantab-TEDLIUM-pruned.lm34 with 150K word vocabulary. The same LM is used in the Kaldi tedlium s5 recipe5 .

2. LM-lium is a 4-gram LM from TED-LIUM corpus with 152K word vocabulary. It is similar to the one currently used in the Kaldi tedlium s5_r2 recipe. 6 We conducted part of the experiments presented here using this LM in order to be compatible with the most recent Kaldi recipe and for comparison purposes with the results of the TDNN acoustic models. The only difference is that we modified a little the training set, removing from it data presented in our test and development datasets.

These two LMs are used in different series of experiments. The ASR results with different LMs are not comparable between each other. Also because of the fact that some data from the development and test sets may be part of the training corpus for LM-cantab, this LM can be biased towards the test sets, and more ASR errors are due to acoustics than due to the LM. For this reason, most of the final results are reported for LM-lium.

STC

The STC corpus is a microphone corpus (16 kHz) of read Russian speech collected at Speech Technology Center 7 .

Training data

The training set contains approximately 27 hours of speech data from 203 (111 male and 92 female) speakers. An 11k vocabulary without LM was used in evaluation. The phone set consists of 52 phonemes. This corpus is used for experiments with supervised speaker adaptation in Chapter 6.

Adaptation and test data

The adaptation experiments were conducted on data from 20 (10 male and 10 female) speakers excluded from the training set. For each speaker there are approximately 180 utterances, corresponding to 16-17 minutes of speech data. The amount of adaptation data is 5 minutes for each speaker, and the rest of the data (11-12 minutes for each speaker) is used for testing in all experiments. The test set consists of 2395 utterances. The total number of word occurrences in the test set is approximately 19K.

Arabic MGB 2016 challenge

The 2016 Multi-Genre Broadcast (MGB-2) Challenge is a controlled evaluation of speech recognition and lightly supervised alignment using Aljazeera Arabic TV channel recordings [START_REF] Ali | The MGB-2 challenge: Arabic multi-dialect broadcast media recognition[END_REF]. In Section 8.7 we report experimental results on the MGB-2 Challenge corpus, provided by the organizers to all participants.

Training and development data

The training data for AMs, provided by the organizers, consists of approximately 1 128 hours of Arabic broadcast speech, obtained from more than 2 000 broadcast shows on the Aljazeera Arabic TV channel over a period of 10 years, from 2005 until 2015. According to the MGB-2 organizers [START_REF] Ali | The MGB-2 challenge: Arabic multi-dialect broadcast media recognition[END_REF], most of the speech data (around 70%) is Modern Standard Arabic (MSA), and the remaining part contains speech in different Arabic dialects, such as Egyptian, Gulf, Levantine and North African.

The original data has a corresponding time-aligned transcription output from a lightly supervised alignment based on Aljazeera closed-captions, with varying quality of manual transcription. Table 5.2 presents some statistics for the training and development datasets.

For AM training, the LIUM has iteratively selected and aligned 648.3 hours of speech data (see the last line of Table 5.2). The detailed description of LIUM's data selection and alignment strategy is described in [START_REF] Tomashenko | LIUM ASR systems for the 2016 multi-genre broadcast Arabic challenge[END_REF]. We use this selected data for AM training.

For experiments with adaptation we use grapheme-based lexicon with 40 graphemes. The number of speakers, estimated using meta data, is 14 249. This number is considered when performing speaker adaptation or per-speaker normalization, but this may not correspond to the real number of speakers in the training corpus, as, for example, a speaker presented in different shows, will be counted several times. The estimated number of speakers in

Language models

The provided language modeling data, according to the organizers, consists of more than 110 million words from the Aljazeera8 website, collected between 2004 and 2011. The provided normalized and transliterated language modeling data contains 4 717 873 sentences, corresponding to 121 277 408 words. The automatic transcriptions of Arabic broadcast speech were also available for LM training. We use 2-gram and 4-gram LMs, built using SRILM toolkit [START_REF] Stolcke | SRILM -an extensible language modeling toolkit[END_REF]] by the LIUM. The description of these LMs are given in [START_REF] Tomashenko | LIUM ASR systems for the 2016 multi-genre broadcast Arabic challenge[END_REF]. For these LMs, modified Knesser-Ney discounting was applied. The vocabulary size is 300K.

Summary for all datasets

Summary information for all datasets, used in this thesis, is given in Table 5.3.

GMM framework for neural network AMs

In this chapter we proposed a GMM framework for adaptation of DNN AMs. It is based on paper [START_REF] Tomashenko | Speaker adaptation of context dependent deep neural networks based on MAP-adaptation and GMM-derived feature processing[END_REF]].

Introduction

The main purpose of introducing the GMM framework is to transfer GMM adaptation techniques to DNN AMs. On the one hand, GMM-HMM AMs have a long history: since 1980s they have been used in speech recognition and, as we saw in Section 4.2, speaker adaptation is a well-studied field of research for these models. Many very effective adaptation algorithms have been developed for GMM AMs. On the other hand, DNNs have achieved big advances in ASR over the past 3-6 years, and nowadays DNNs show higher performance than GMMs for different ASR tasks. Neural networks today are state-of-the-art of acoustic modeling. However, speaker adaptation is still a very challenging task for these models.

One of the motivations for the current research, described in this thesis, is the fact that many adaptation algorithms that work well for GMM systems cannot be easily applied to DNNs. Except for some feature normalization techniques, such as VTLN or CMVN (Section 4.1.2), only fMLLR adaptation technique (Section 4.2.2.2), originally developed for GMM AMs, has also become a widespread adaptation technique for DNN AMs (Section 4.3.7). However, fMLLR adaptation is limited to only one feature-space transformation. Thus, this algorithm performs well, when a small amount of adaptation data is available, but when the amount of adaptation data increases, this method is known to saturate, and does not make use of all the available adaptation data, unlike, for example, Bayesian methods. Therefore, there is no universal method for efficient transfer of all adaptation algorithms from the GMM framework to DNN models.

Another important aspect, that should be taken into account when comparing GMM and neural network AMs, is the different nature of these models. Hence, GMM and DNN models may be complementary, and ASR systems may benefit from their combination. In this thesis, we aim to take advantage of the robust adaptability of GMM AMs and existing adaptation methods developed for them, and apply these methods to DNN AMs.

Hybrid DNN-HMM systems with GMM-derived features

As we saw in Section 3.3.2.1, in a conventional GMM-HMM ASR system, the state emission log-likelihood of the observation feature vector o t for a certain tied state s i of HMMs is modeled as (see Formula (3.13)):

log P(o t | s i ) = log M ∑ m=1 ω im N im (o t | s i ), (6.1)
where M is the number of Gaussian mixtures in the GMM for state s i and ω im is the mixture weight.

In a DNN-HMM system, outputs of a DNN are the state posteriors P(s i |o t ), which are transformed for decoding into pseudo (or scaled) log-likelihoods as follows (see Formula (3.17)):

log P(o t | s i ) = log P(s i | o t )P(o t ) P(s i ) ∝ log P(s i | o t ) -log P(s i ), (6.2)
where the state prior P(s i ) can be estimated from the state-level forced alignment on the training speech data, and probability P(o t ) is independent on the HMM state and can be omitted during the decoding process.

We propose to use features, derived from a GMM model, in order to train a DNN model. Further in this thesis, we will refer to a GMM model, used for feature extraction, as an auxiliary GMM model, and to the obtained features -as GMM-derived (GMMD) features.

As we mentioned before, the construction of GMMD features for DNNs is mainly motivated by two factors. First, in the past it was shown [START_REF] Ellis | Investigations into tandem acoustic modeling for the aurora task[END_REF][START_REF] Pinto | Combining evidence from a generative and a discriminative model in phoneme recognition[END_REF][START_REF] Swietojanski | Revisiting hybrid and GMM-HMM system combination techniques[END_REF] that NN and GMM models may be complementary and their combination can provide an additional improvement in ASR performance. Secondly, this type of features makes it possible to use various GMM-HMM adaptation algorithms in the DNN framework.

The GMMD features are obtained as follows (see Figure 6.1). Firstly, acoustic feature vectors are extracted from the speech signal. It can be spectral features (Section 3.2.1), such as MFCCs, PLPs, or NN-based features (Section 3.2.2), such as BN features (Section 3.2.2.2) or other types. After that, cepstral mean normalization (CMN) can be applied to the extracted features. Then, an auxiliary HMM-GMM model is used to transform acoustic feature vectors into likelihood-based feature vectors. This auxiliary HMM-GMM can be trained in a standard way with ML objective function (Section 3.3.3) and with triphone or monophone states as basic units. For a given acoustic feature vector, a new GMMD feature vector is obtained by calculating likelihoods across all the states of the auxiliary GMM model on the given vector. Suppose o t is the acoustic feature vector at time t, then the new GMMD feature vector f t is calculated as follows:

f t = [p 1 t , . . . , p n t ], (6.3)
where n is the number of states in the auxiliary GMM-HMM model,

p i t = ϕ(P(o t | s t = i)) (6.4)
is the function of likelihood estimated using the GMM-HMM. Here s t denotes the state index at time t. The natural choice of ϕ is a logarithmic function (as in [START_REF] Pinto | Combining evidence from a generative and a discriminative model in phoneme recognition[END_REF]Tomashenko and Khokhlov, 2015;[START_REF] Tomashenko | On the use of Gaussian mixture model framework to improve speaker adaptation of deep neural network acoustic models[END_REF]) or an identity function (as in [START_REF] Tomashenko | Speaker adaptation of context dependent deep neural networks based on MAP-adaptation and GMM-derived feature processing[END_REF]).

The dimension of the obtained likelihood-based feature vector f t is determined by the number of states in the auxiliary GMM-HMM AM. At this step, we can also reduce the dimension of f t using PCA, LDA, HLDA or other analysis. This step is not necessary, as shown in Figure 6.1. The dimensionality reduction can be required to obtain a sufficient dimension of the input layer of a DNN.

After that, the features are spliced in time by taking a context of several frames (typically 2T + 1 frames (i.e. [-T..T ]1 ), where 5 ≤ T ≤ 15).

The obtained GMMD features are used to train a DNN-HMM AM. A DNN model can be trained either directly on GMMD features, as shown in Figure 6.1, and as it was done in [START_REF] Khokhlov | [END_REF]Khokhlov, 2014b, 2015], or on combination of GMMD with other conventional features, as in [Tomashenko et al., 2016a,b].

DNN adaptation using GMMD features

Speaker adaptation of a SI DNN-HMM model built on GMMD features is performed through the adaptation of the auxiliary SI GMM-HMM model, that was used for GMMD feature extraction. The adaptation of the auxiliary SI GMM-HMM model can be done by any adaptation algorithm developed for GMM-HMM AMs (Section 4.2). In this thesis we will investigate the application of the most popular speaker adaptation algorithms -MAP (Section 4.2.1) and MLLR (Section 4.2.2) for DNN AM adaptation.

Preliminary experimental results with supervised adaptation

In this section preliminary results for the proposed approach are presented. The adaptation experiments were conducted in a supervised mode (Section 4.1.1).

Baseline system

The experiments were conducted on STC corpus (Section 5.3). We used 11×13MFCC features (13-dimensional MFCC spliced across 11 frames ([-5..5]) as baseline features for training baseline DNN.

For GMMD feature extraction we first built an auxiliary speaker-independent (SI) GMM-HMM monophone model. Acoustic features in Figure 6.1 in this series of experiments are 39-dimensional MFCC+∆ + ∆∆ features, extracted from speech signal with a frame shift of 10 ms and window length of 16 ms. Features were normalized using CMN. In the auxiliary HMM-GMM, each of 52 Russian phonemes was modeled using a 3-state left-right context-independent HMM, with 30 Gaussians per state. The silence model was a 1-state HMM. The total number of states in the auxiliary HMM-GMM was equal to 157 -that is the dimension of f t (n in Formula ( 6.3)). The function φ from Formula (6.4) was an identity function in these experiments. That means, that the coordinates of feature vector f t were likelihoods:

p i t = P(o t | s t = i) (6.5)
We extracted two types of GMMD features (with and without dimensionality reduction):

• 11×157GMMD -1 727-dimensional features, obtained after splicing f t feature vectors with a context of 11 frames: [-5..5].

• 11×55GMMD -550-dimensional features. In this case, before splicing, the dimension of f t was reduced from 157 to 55 using PCA.

The three SI-DNN models corresponding to these three types of features (11×13MFCC, 11×55GMMD and 11×157GMMD) had identical topology (except for the dimension of the input layer) and were trained on the same training corpus. The auxiliary SI monophone GMM was also trained on the same data.

The SI CD-DNN-HMM systems used 1000-neuron hidden layers, and a 2008-neuron output layer. 2008 neurons in the output layer correspond to context-dependent states determined by tree-based clustering in the CD-GMM-HMM system. ReLU nonlinearities (Formula (2.6)) were used in the hidden layers. The DNN system was trained using the framelevel cross-entropy (CE) criterion and the senone alignment generated from the triphone GMM system. The output layer was a softmax layer. DNNs were trained without layer-bylayer pre-training, and dropout with hidden dropout factor (HDF = 0.2) was applied during training as a regularization (Section 2.3).

In order to explore trends in the behavior of SI and adapted models trained on the proposed GMMD features for DNNs of different depths, two DNN-HMMs were trained for each experiment, with 2 and 4 hidden layers.

An 11k vocabulary was used in evaluating both SI and adapted models. No language model was used. All described experiments were conducted on the same test set, which consists of 2 395 utterances.

The performance results in terms of WER (Formula (3.32)) for SI DNN-HMM models are presented in Table 6.1. We can see that DNN-HMMs trained on 11×13MFCC perform better than DNN-HMMs trained either on 11×55GMMD or 11×157GMMD features. In Section 6.4.3, we explore this issue in more detail and propose ways to improve the quality of SI models.

Adaptation for DNN

The adaptation experiments were conducted on data from 20 speakers (10 male and 10 female speakers) excluded from the training set (see Section 5.3 for details). The parameter τ in the MAP adaptation formula (4.2) is set to 5. Adaptation experiments are carried out in a supervised mode.

Supervised adaptation performance for different DNN-HMMs

The adaptation results for four DNN-HMMs are presented in Table 6.2. We can see that adaptation improves SI models by approximately 14-15% absolute, corresponding to 34-36% relative WER reduction (∆ rel WER) (Formula 3.34). The amount of adaptation data in these experiments was 5 minutes per each speaker. In this experiment we measure the performance of the proposed adaptation method using different amounts of adaptation data. Adaptation sets of different size, from 5 seconds to 5 minutes (for a speaker), are used to adapt a SI 4-hidden-layer DNN-HMM trained on 11×157GMMD features. The results are shown in Figure 6.3. We can see that even with 5 seconds of adaptation data, we can achieve a performance gain of about 2% absolute (5% relative) WER reduction. After using all available 300 seconds of adaptation data, the gain from the adaptation increases up to approximately 14% absolute (34% relative) WER reduction.

Fusion

As we observed in the experiments described above, the proposed adaptation technique provides a significant improvement in the ASR performance. However, from the practical point of view, it can also be useful to have a strong SI baseline for the ASR system , for example, in case we do not have adaptation data for some speakers. Also, a stronger SI model (according to WER) can potentially provide a better result than a weaker model when they are adapted. As seen from the results described above (Table 6.1), DNN-HMMs trained on 11×13MFCC perform better than DNN-HMMs trained on either 11×55GMMD or 11×157GMMD features. For the SI 2-hidden-layer DNN, the absolute difference in WER is 2.3% for 11×55GMMD and 1.1% for 11×157GMMD as compared to 11×13MFCC features. As the number of layers in DNNs increases from 2 to 4, the differences in quality between the SI DNN trained on 11×13MFCC features and SI DNNs trained on 11×55GMMD and 11×157GMMD features increase up to 3.6% and 2.8% respectively. The purpose of this set of experiments was to find out whether it is possible to improve the performance of SI systems by applying different multi-stream combination techniques, so-called late and early integration [START_REF] Pinto | Combining evidence from a generative and a discriminative model in phoneme recognition[END_REF]. More advanced analysis for fusion techniques and ways of integration of GMMD features into the state-of-the-art ASR system architecture is presented in Chapter 8.

Feature-level fusion

In this experiment, 11×13MFCC and 11×55GMMD features are concatenated, resulting in 693-dimension feature vectors, and new DNN SI models are trained. The results with feature concatenation are presented in Table 6.3. We can see that this type of features helps to reduce WER for the SI model compared to the baseline SI model for 11×55GMMD. In addition, the adaptation performance with feature concatenation is slightly better than with 11×55GMMD for the 4-hidden-layer DNN, while for the 2-hidden-layer DNN it remains unchanged. However, the performance of feature concatenation without adaptation is still worse than 11×13MFCC, especially for the deeper network. Finally, we perform state-level system fusion experiments. The outputs of the two classifiers can be combined using various multi-stream combination techniques [START_REF] Kittler | Pattern Analysis and Machine Intelligence[END_REF]] such as sum (or linear combination), product, maximum, minimum, etc. In this work results are shown for the mean of outputs of two DNNs, as this combination gave the best result in preliminary experiments. System fusion is performed at the state-level for every frame. The posterior fusion results are summarized in Table 6.3. Combining scores from DNNs trained on 11×13MFCC and 11×55GMMD features achieves 1.6% and 0.8% absolute WER improvement for SI DNNs as compared to DNNs trained on 11×13MFCC features for DNNs with 2 and 4 hidden layers respectively. Adaptation for posterior-level fusion gives about 30-32% relative WER reduction compared to SI DNNs trained on 11×13MFCC features.

Thus we see that posterior-level fusion of the two SI-DNN-HMMs gives better results than the feature concatenation approach. Moreover, posterior-level fusion helps to improve the quality of the baseline SI-system which is used for adaptation. This type of fusion gives an improvement of about 2-4% of relative WER reduction over the single best stream. This result can probably be further improved by using other fusion techniques.

Conclusions

In this chapter we proposed a novel feature-space transformation method for the adaptation of DNN-HMM models [START_REF] Tomashenko | Speaker adaptation of context dependent deep neural networks based on MAP-adaptation and GMM-derived feature processing[END_REF]. The proposed method is based on a special method of feature processing for DNNs. Input features for DNN are generated from the likelihoods of the GMM model. The described method of feature processing is effective for training DNN-HMMs when used in combination with other conventional features, such as 11×13MFCC. According to the experiments, the best performance is obtained by the posterior-level fusion technique which gives approximately 2-4% relative WER reduction compared to the single best stream (the baseline DNN trained on 11×13MFCC).

The main advantage of these GMMD features is the possibility of performing the adaptation of a DNN-HMM model through the adaptation of the auxiliary GMM-HMM. We investigate the performance of MAP adaptation for this scheme. Experiments demonstrate that MAP adaptation is very effective and, for single-stream systems, gives, on average, approximately 35-36% relative WER reduction for a 5 minute adaptation sample and 5% relative WER reduction for a 5 second adaptation sample, as compared to a SI-DNN model.

If we compare adaptation results in a multi-stream recognition system, we see that the performance improvement from the adaptation for a 5 minute adaptation set is 11% absolute (29% relative) WER as compared to the SI system.

It is worth noting that in the proposed scheme, other methods for the adaptation of the auxiliary GMM-HMM can be used instead of MAP adaptation. Thus, this approach opens new perspectives in the adaptation of DNN-HMMs, since it allows us to use approaches developed for GMM-HMM, as well as SAT (that will be the topic of the following chapter), in DNN-HMM adaptation.

Chapter 7

Speaker adaptive training

In this chapter we extend the scheme for GMM-derived (GMMD) feature extraction and apply the concept of speaker adaptive training (SAT) to DNNs, trained on GMMD features. Also several techniques for adaptation performance improvement are proposed and explored: using lattices scores in maximum a posteriori (MAP) adaptation, data augmentation and data selection techniques. This chapter is based on papers [Tomashenko and Khokhlov, 2015] and [START_REF] Tomashenko | Exploring GMM-derived features for unsupervised adaptation of deep neural network acoustic models[END_REF].

SAT DNN AM training using GMMD features 7.1.1 Proposed approach to SAT-DNN

In this section we improve the previously proposed scheme for GMMD feature extraction and apply the concept of SAT to DNNs, trained on GMMD features. In SAT, speaker adaptation is performed for the training data, independently for each training speaker, so that all the training data are projected into speaker-adaptive feature space. Then, the parameters of the SAT-DNN AM are estimated in this feature space.

The scheme of SAT-DNN training is shown in Figure 7.1. In contrast to the previous approach, explored above (Section 6.2 and Figure 6.1), for each training speaker we extract speaker-adapted GMMD features, and train SAT-DNN AM on them.

The procedure for building SAT-DNN can be summarized as follows:

1. Train an auxiliary SI GMM-HMM Λ Λ Λ over the training data. 4. Calculate GMMD features from speaker-adapted f t .

5. Train DNN AM on the obtained speaker-adapted GMMD features.

During the test, the adaptation is performed in the same manner, as described in Section 6.3. In this chapter, and further in the following chapters, we use log-likelihoods in Formula (6.4), so that The experiments were conducted on the WSJ0 corpus (Section 5.1).

p i t = log P(o t | s t = i) (7.1)
We used conventional 11×39MFCC features, composed of 39-dimensional MFCC (with CMN) spliced across 11 frames [-5..5], as baseline features and compared them to the proposed GMMD features. The two SI-DNN models corresponding to these two types of features, 11×39MFCC and 11×118GMMD, had identical topology (except for the dimension of the input layer) and were trained on the same training dataset. An auxiliary monophone GMM was also trained on the same data. For training speaker independent (SI) DNN on GMMD features, we applied the scheme shown in Figure 6.1.

The SI CD-DNN-HMM systems used four 1000-neuron hidden layers and an output layer with approximately 2500 neurons. The neurons in the output layer corresponded to the context-dependent states determined by tree-based clustering in CD-GMM-HMM. Rectified linear units (ReLUs) were used in the hidden layers. The DNN system was trained using the frame-level CE criterion and the senone alignment generated from the GMM system. The output layer was a softmax layer. DNNs were trained without layer-by-layer pre-training, and hidden dropout factor (HDF = 0.2) was applied during the training as a regularization.

Evaluation was carried out on the two standard WSJ0 evaluation tests: (1) si_et_05 and (2) si_et_20.

SAT-DNN AMs

We trained three SAT DNNs on GMMD features, as shown in Figure 7.1. For adapting an auxiliary GMM model we used two different adaptation algorithms: MAP and fMLLR. In addition, we trained a SAT DNN on GMMD features with "fMLLR+MAP" configuration, where MAP adaptation of an auxiliary GMM model was performed after fMLLR adaptation. For comparison purposes we also trained a DNN on conventional 11×39MFCC features with fMLLR. All SAT DNNs had similar topology and were trained as described in Section 7.1.2.1 for SI models. Training GMM-HMMs and fMLLR adaptation was carried out using the Kaldi speech recognition toolkit [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF].

Adaptation performance for different DNN-HMMs

Unless explicitly stated otherwise, the adaptation experiments were conducted in an unsupervised mode on the test data using transcripts obtained from the first decoding pass. The performance results in terms of word error rate (WER) for SI and SAT DNN-HMM models are presented in Table 7.1. We can see that SI DNN-HMM trained on GMMD features performs slightly better than DNN-HMM trained on 11×39MFCC features. In all experiments we consider SI DNN trained on 11×39MFCC features as the baseline model and compare the performance results of the other models with it. The SAT DNN trained on GMMD features with MAP adaptation demonstrates the best result over all cases in the dataset si_et_20. It outperforms the baseline SI DNN model trained on 11×39MFCC features and results in 20.4% relative WER reduction (∆ rel WER, see Formula 3.34). In the dataset si_et_05 the SAT DNN trained on GMMD features with fMLLR adaptation performs better than other models and gives 22.0% of ∆ rel WER. Moreover, we can see that in all experiments SAT models trained on GMMD features, with fMLLR as well as with MAP adaptation, perform better in terms of WER than the SAT model trained on 11×39MFCC features with fMLLR adaptation.

From the last row of Table 7.1, which shows the results of combining MAP and fMLLR algorithms, we can conclude that this combination does not lead to additional improvement in performance, and for the test si_et_05, it even degrades the result.

The results, presented in Tables 7.2 and 7.3, demonstrate adaptation performance for different speakers from the si_et_20 and si_et_05 datasets, correspondingly. The bold figures in the tables indicate the best performance over the three acoustic models. Relative WER reduction (∆ rel WER) is given in comparison to the baseline model, SI DNN built on 11×39MFCC. We can observe that all the three models behave differently depending on the speaker. In this experiment we measured the performance of the proposed adaptation method using different amounts of adaptation data. Adaptation sets of different sizes, from 15 to 210 seconds of speech data (per speaker), were used to adapt SAT DNN-HMM models trained on 11×39MFCC and GMMD features.

The results are shown in Figure 7.2. Relative WER reduction is given with respect to the baseline SI DNN trained on 11×39MFCC. We can see that, for adaptation sets of different size, the SAT DNN trained on GMMD features with fMLLR perform better than the SAT DNN trained on 11×39MFCC features with fMLLR. In contrast, MAP adaptation reaches the performance of fMLLR adaptation (for MFCC) only when using all the adaptation data. However, the gain from MAP adaptation grows monotonically as the sample size increases, while fMLLR adaptation reaches saturation on a small adaptation dataset. The same behavior of MAP and fMLLR adaptation algorithms is known for GMM-HMM AMs.

We conducted an additional experiment with MAP adaptation, marked in Figure 7.2 with the text "using extra data", in which we added the data from the WSJ0 si_et_ad dataset to the adaptation data. Hence, in this case we performed the adaptation in a semi-supervised mode: the transcriptions for the si_et_ad dataset were supposed to be known and the transcriptions for the si_et_05 were generated from the first decoding pass. The total duration of the adaptation data was approximately 380 seconds for each speaker. This result (21.4% of relative WER reduction) confirms the suggestion that the performance of MAP adaptation did not reach its maximum in the previous experiments.

Discussion

In this section we improved the previously proposed adaptation algorithm by applying the SAT concept to DNNs built on GMMD features and by using fMLLR-adapted features for training an auxiliary GMM model. Traditional adaptation algorithms, such as MAP and fMLLR were performed for the auxiliary GMM model used in a SAT procedure for a DNN. Experimental results on the WSJ0 corpus demonstrate that, in an unsupervised adaptation mode, the proposed adaptation technique can provide, approximately, a 17-20% relative WER reduction for MAP and 18-28% relative WER reduction for fMLLR adaptation on different adaptation sets, compared to the SI DNN-HMM system built on conventional 11×39MFCC features. We found that fMLLR adaptation for the SAT DNN trained on GMMD features outperforms fMLLR adaptation for the SAT DNN trained on conventional features by up to 14% of relative WER reduction. It has been shown, that fMLLR adaptation for GMMD features is efficient when using a small amount of adaptation data, while MAP adaptation works better when more adaptation data are used.

It is worth noting that in the proposed scheme, any other methods for the adaptation of the auxiliary GMM can be used instead of MAP or fMLLR adaptation. Thus, this approach provides a general framework for transferring adaptation algorithms developed for GMM-HMMs to DNN adaptation.

MAP adaptation using lattices scores

The use of lattice-based information and confidence scores [START_REF] Gollan | Confidence scores for acoustic model adaptation[END_REF]] is a well-known method for improving the performance of unsupervised adaptation. In this work we use the MAP adaptation algorithm for adapting the SI GMM model. Speaker adaptation of a DNN-HMM model built on GMMD features is performed through the MAP adaptation of the auxiliary GMM monophone model, which is used for calculating GMMD features. We modify the traditional MAP adaptation algorithm by using lattices instead of alignment from the first decoding pass as follows.

Let m denote an index of a Gaussian in SI AM, and µ µ µ m the mean of this Gaussian. Then the MAP estimation of the mean vector is

µ µ µ m = τµ µ µ m + ∑ t γ m (t)p s (t)o t τ + ∑ t γ m (t)p s (t) , (7.2)
where τ is the parameter that controls the balance between the maximum likelihood estimate of the mean and its prior value; γ m (t) is the posterior probability of Gaussian component m at time t; and p s (t) is the confidence score of state s at time t in the lattice obtained from the first decoding pass by calculating arc posteriors probabilities. The forward-backward algorithm is used to calculate these arc posterior probabilities from the lattice as follows [START_REF] Evermann | Posterior probability decoding, confidence estimation and system combination[END_REF][START_REF] Uebel | Improvements in linear transform based speaker adaptation[END_REF]:

P(l|O) = ∑ q∈Q l P ac (O|q) 1 λ P lm (w) P(O) , (7.3)
where λ is is the scale factor (the optimal value of λ is found empirically by minimizing WER of the consensus hypothesis [START_REF] Mangu | Finding consensus in speech recognition: word error minimization and other applications of confusion networks[END_REF]); q is a path through the lattice corresponding to the word sequence w; Q l is the set of paths passing through arc l; P ac (O|q) is the acoustic likelihood; P lm (w) is the language model probability; and p(O) is the overall likelihood of all paths through the lattice. For the given frame o t at time t we the calculate confidence score p s (t) as follows:

p s (t) = ∑ l∈S s (o t ) P(l|O), (7.4)
where S s (o t ) is the set of all arcs corresponding to state s in the lattice at time t. In a particular case, when p s (t) = 1 for all states and t, Formula (7.2) represents the traditional MAP adaptation (Formula (4.2)).

In addition to this frame-level weighting scheme, we apply confidence-based selection scheme, when we use in (7.2) only the observations with confidence scores exceeded the chosen threshold.

Data augmentation and data selection for SAT

In this work we explore two other approaches to improve the performance of SAT DNN models with MAP adaptation (Figure 7.3): data augmentation and data selection.

The first approach is based on using different values of parameter τ (in Formula (4.2)) when extracting adapted GMMD features for DNN training. In this approach we extract features for all training corpus several times for a set of τ values: {τ 1 , τ 2 , . . .}. Then, the DNN model is trained on the union of the obtained features. The intuition behind this approach is similar to that used in data augmentation [START_REF] Cui | Data augmentation for deep neural network acoustic modeling[END_REF]. The second approach, which we call data selection strategy, consists in splitting training data for each speaker in the training corpus into several parts and then performing MAP adaptation independently on each of the parts. In this chapter we use a simple implementation of this strategy -we randomly separate training data for each speaker into several subsets, so that the total amount of data in each subset is approximately equal to the average amount of data per speaker in the test set. This strategy serves as a regularization and is supposed to make adaptation more robust to the size of the adaptation set.

Hence, the original data from the training corpus are used in AM training several times with different values of τ and inside different subsets of data chosen for adaptation. The motivation for these two approaches lies in obtaining more robust SAT DNN models for MAP adaptation, especially when the training corpus is relatively small.

The GMMD feature dynamic in the training corpus for different values of τ and for different data selection strategies is shown in Figure 7.4. In both pictures "full" means that during the SAT training for a given speaker all data of that speaker from the training corpus are used for MAP adaptation, whereas "selection" means that data selection strategy is applied and training data for this speaker is randomly split into two subsets so that MAP adaptation is performed for each subset independently. Let denote T 1 and T 2 two types of features, (or more precisely, GMMD features extracted with different parameters). Every curve in Figure 7.4a and 7.4b, marked as "T 1 -T 2 ", corresponds to the average differences between T 1 and T 2 features and is calculated as follows. First, we subtract coordinate-wise features T 2 from T 1 on the training corpus. Then, we found mean (Figure 7.4a) and standard deviation (Figure 7.4b) values for each feature vector coordinate. Finally, we sort the obtained values for each feature vector dimension by descending order.

For example, in Figure 7.4 the first (blue) line in both pictures corresponds to "τ=0.1 -τ=5 (full)". This means in our notation, that we consider two types of GMMD features: T 1 : τ = 0.1 and T 2 : τ = 5, calculated on the training corpus with different τ values, and "(full)" means that in both cases features are extracted in the standard way without data selection, for each speaker using all data from this speaker available in the training corpus to perform speaker adaptation. We can see that GMMD features calculated for various τ and with (or without) data selection strategy have different amplitude and dynamic characteristics, therefore they can contain complementary information. Hence data augmentation might improve AM by making them more robust to τ and to the size of the adaptation set.

Experimental results with data augmentation, data selection and lattices scores

The experiments are conducted on the WSJ0 corpus (Section 5.1). Unlike the initial experiments with SAT and GMMD features, described in Section 7.1.2, for this series of experiments we built a novel strong baseline AM using the Kaldi speech recognition toolkit [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF], following mostly Kaldi WSJ recipe (except for GMMD-features and adaptation). The new SAT-DNN AM on GMMD features was also trained according to the new recipe.

We use conventional 11×39MFCC features (39-dimensional MFCC (with CMN) spliced across 11 frames (±5)) as baseline features and compare them to the proposed GMMD features. We train four DNN models: SI model on 11×39MFCC; SI and two SAT models on GMMD features. These four DNNs have identical topology (except for the dimension of the input layer) and are trained on the same training dataset. An auxiliary GMM is also trained on the same data.

The first SAT DNN on GMMD features is trained as described in Section 7.1.1 with parameter τ for adaptation equal to 5. The second SAT DNN on GMMD features is trained using data augmentation (with τ equal to 0.1, 1 and 5) and data selection strategy, as described in Section 7.3. For training SI-DNN on GMMD features, we apply the scheme shown in Figure 6.1.

All four CD-DNN-HMM systems had six 2048-neuron hidden layers and a 2355-neuron output layer. The neurons in the output layer correspond to context-dependent states determined by tree-based clustering in CD-GMM-HMM. The DNN is initialized with the stacked restricted Boltzmann machines by using layer by layer generative pre-training. It is trained with an initial learning rate of 0.008 using the CE objective function. After that, five iterations of training with the sMBR criterion (Formula (3.3.3.1)) are performed.

In all experiments further we consider SI DNN trained on 11×39MFCC features as the baseline model and compare the performance results of the other models with it. Evaluation is carried out on the standard WSJ0 evaluation test si_et_20. The adaptation experiments are conducted in an unsupervised mode on the test data using transcripts or lattices obtained from the first decoding pass.

For adapting an auxiliary GMM model we use MAP adaptation algorithm. We perform two adaptation experiments:

1. with traditional MAP; 2. with lattice-based MAP using confidence scores, as described in Section 7.2.

For lattice-based MAP the value of confidence threshold is 0.6. The performance results in terms of WER for SI and adapted DNN-HMM models are presented in Table 1. We can see that using confidence scores can give an additional slight improvement in MAP adaptation for DNN models over adaptation, which uses an alignment. The best result is obtained using data augmentation and data selection strategies. For comparison purposes we also train six DNN models with τ values 0.1, 1 and 5 with and without data selection strategies, but in all cases the results are worse than the one obtained combining both strategies, so we do not report other results here.

Table 7.4 Summary of WER (%) results for unsupervised adaptation on WSJ0 evaluation set si_et_20. ∆ rel WER is a relative WER reduction calculated with respect to the baseline SI AM. The WER for the baseline system is given with a confidence interval, corresponding to 5% level of significance. Experimental results demonstrate, that in an unsupervised adaptation mode, the proposed adaptation technique can provide, approximately, up to 9.9% relative WER reduction compared to the SI DNN system built on conventional 11×39MFCC features.

Conclusions

In this chapter we extended and improved the previously proposed adaptation algorithm by applying the concept SAT to DNNs built on GMM-derived features.

In the first series of experiments (Section 7.1.2), traditional adaptation algorithms, such as MAP and fMLLR were performed for the auxiliary GMM model used in a SAT procedure for a DNN [Tomashenko and Khokhlov, 2015]. Experimental results on the WSJ0 corpus demonstrate that, in an unsupervised adaptation mode, the proposed adaptation technique can provide, approximately, a 17-20% relative WER reduction for MAP and 18-28% relative WER reduction for fMLLR adaptation on different adaptation sets, compared to the SI DNN-HMM system built on conventional 11×39MFCC features. We found that fMLLR adaptation for the SAT DNN trained on GMM-derived features outperforms fMLLR adaptation for the SAT DNN trained on conventional features by up to 14% of relative WER reduction. It has been shown, that fMLLR adaptation for GMMD features is efficient when using a small amount of adaptation data, while MAP adaptation works better when more adaptation data are used.

Another contribution of this chapter consists in the way to improve the previously proposed adaptation algorithm by using confidences scores in adaptation [START_REF] Tomashenko | Exploring GMM-derived features for unsupervised adaptation of deep neural network acoustic models[END_REF]. In addition, we introduced two approaches, so called data augmentation and data selection strategies, for improving the regularization in MAP adaptation for DNN. The proposed approaches are especially suitable when the training corpus is small, or when the amount of adaptation data is not known in advance and can vary.

The second series of experiments (Section 7.4) was conducted with different DNN training recipes, criteria and different DNN topologies in comparison with the experiments described in the first part of this chapter. Particularly, all DNNs in this second series were trained with much more parameters (6 HLs×2048 neurons vs. 4 HLs×1000 neurons). Also, DNNs in the second series had sigmoid activation functions and were trained with RBM pre-training in contrast to the DNNs in the first series, which used ReLU activation functions and dropout. In addition, DNNs in the second series were trained with sMBR training criterion, while CE criterion was used in the first series of experiments. This resulted in more accurate (in sense of WER) models. We observed that, in these conditions, improvement from MAP adaptation decreased but is still significant. The relative WER reduction for adapted AM equals to approximately 10%, if we compare them with SI AM trained on the same features, and 6% if we compare them with SI AM trained on conventional features. These two values can be further improved by the proposed data augmentation and selection techniques and reach respectively 14% and 10% of relative WER reduction.

Chapter 8 Integration of GMMD features into state-of-the-art ASR systems

In this chapter we investigate various ways of integrating GMMD features into different neural network architectures (DNN and TDNN). It includes the results from papers [START_REF] Tomashenko | On the use of Gaussian mixture model framework to improve speaker adaptation of deep neural network acoustic models[END_REF], [Tomashenko et al., 2016a] and [START_REF] Tomashenko | LIUM ASR systems for the 2016 multi-genre broadcast Arabic challenge[END_REF].

System fusion

In this section we suggest several types of combination of GMMD features with conventional ones at different levels of DNN architectures. It is known that GMM and DNN models can be complementary and their combination allows to improve the performance of ASR systems [START_REF] Pinto | Combining evidence from a generative and a discriminative model in phoneme recognition[END_REF][START_REF] Swietojanski | Revisiting hybrid and GMM-HMM system combination techniques[END_REF]. Fusion is useful when the individual systems, used in combination, contain complementary information. To obtain better recognition performance we explore the following types of fusion: feature-level, posterior-level, lattice-level and others.

Feature-level fusion

In this type of fusion, also called early fusion or early integration [START_REF] Pinto | Combining evidence from a generative and a discriminative model in phoneme recognition[END_REF], input features are combined before performing classification, as shown in Figure 8.1a. In our case, features of different types -GMMD features and cepstral or BN features are simply concatenated and provided as input into the DNN model for training. This type of fusion allows us to combine different adaptation techniques in a single DNN model. For example, MAP-adapted GMMD features can be concatenated with fMLLR-adapted BN features or i-vectors, that makes adaptation more efficient for both small and large adaptation sets.

Posterior-level fusion

Posterior-level fusion is also referred to as late fusion [START_REF] Parthasarathi | fMLLR based feature-space speaker adaptation of DNN acoustic models[END_REF], late integration [START_REF] Pinto | Combining evidence from a generative and a discriminative model in phoneme recognition[END_REF], state-level score combination [START_REF] Lei | Deep neural networks with auxiliary Gaussian mixture models for real-time speech recognition[END_REF] or explicit combination [START_REF] Swietojanski | Revisiting hybrid and GMM-HMM system combination techniques[END_REF]. In this type of fusion (Figure 8.1b) the outputs of two or more DNN models are combined at the state level. The outputs of the two classifiers can be combined using various multi-stream combination techniques such as sum (or linear combination), product, maximum, minimum, etc. In this work we perform frame-synchronous fusion using a linear combination of the observation posteriors of two models (DNN 1 and DNN 2 ) as follows:

P(o t | s i ) = αP DNN 1 (o t | s i ) + (1 -α)P DNN 2 (o t | s i ), (8.1)
where α ∈ [0, 1] is a weight factor that is optimized on a development set. This approach assumes that both models have the same state tying structure.

Lattice-level fusion

The highest level of fusion operates in the space of generated word hypotheses and tries to rescore or modify recognition hypotheses provided as lattices (Figure 8.1c) or n-best lists. This type of fusion is also referred to as implicit combination [START_REF] Swietojanski | Revisiting hybrid and GMM-HMM system combination techniques[END_REF]. One of the most common techniques for ASR system combination are recognizer output voting error reduction (ROVER) [START_REF] Fiscus | A post-processing system to yield reduced word error rates: recognizer output voting error reduction (ROVER)[END_REF] and confusion network combination (CNC) [START_REF] Evermann | Posterior probability decoding, confidence estimation and system combination[END_REF]. In ROVER 1-best word sequences from the different ASR systems are combined into a single word transition network (WTN) using a dynamic programming algorithm. Based on this WTN the best scoring word is chosen among the words aligned together. The decision is based either on the voting scheme, or word confidence scores if they are available for all systems.

In CNC, instead of aligning the single best output, confusion networks built from individual lattices are aligned. In this work we experiment with the CNC approach because usually (for example, [START_REF] Evermann | Posterior probability decoding, confidence estimation and system combination[END_REF][START_REF] Xu | Minimum bayes risk decoding and system combination based on a recursion for edit distance[END_REF]) it provides better results than a simple ROVER scheme.

Other types of fusion

There are other possible ways of combining information from different ASR systems than those listed above, that also could be considered as fusion. In this chapter, in addition to those already mentioned, we also implemented the two following approaches.

The first approach is specific to the adaptation task and is related only to the acoustic model adaptation stage. It consists in using the transcripts (or lattices) obtained from the decoding pass of one ASR system in order to adapt another ASR system (cross-adaptation).

The second approach (Figure 8.1d) is related to the acoustic model training. It is possible to transfer some information from building one acoustic model to another one. In this work we used phoneme-to-speech alignment obtained by one acoustic DNN model to train another DNN model. In addition we used state tying from the first DNN model to train the second DNN. This procedure is important when we want to apply posterior fusion for two DNNs and need the same state tying for these models. Also, several types of fusion described above can be combined.

Training DNN AMs with GMMD features

In the previous chapters (6 and 7), an auxiliary GMM model was trained on MFCC features, and then this GMM model was used to extract GMMD features for further training a DNN model. In this chapter, we investigate the effectiveness of the proposed approach on another level of DNN architecture. We use bottleneck (BN) features (Section 3.2.2.2) from a DNN to train a GMM model for GMMD feature extraction. The motivation for using BN features in this approach is that for better source features we can obtain better adaptation results. BN features allow us to capture long term spectro-temporal dynamics of the signal with GMMD features and are proven to be effective both for GMM and DNN acoustic model training [START_REF] Grézl | Probabilistic and bottle-neck features for LVCSR of meetings[END_REF][START_REF] Grézl | Adaptation of multilingual stacked bottle-neck neural network structure for new language[END_REF]. To confirm this suggestion, in Section 8.6.1, we will experimentally demonstrate that BN features are more effective than MFCC for GMMD feature extraction.

The scheme for training DNN models with GMM adaptation framework is shown in Figure 8.2. First, 40-dimensional log-scale filterbank features, concatenated with 3-dimensional pitch-features1 , are spliced across 11 neighboring frames [-5..5], resulting in 473dimensional (43 × 11) feature vectors. After that a DCT transform is applied and the dimension is reduced to 258. Then a DNN model for 40-dimensional BN features is trained on these features. An auxiliary triphone or monophone GMM-HMM model is used to transform BN feature vectors into log-likelihoods vectors. At this step, speaker adaptation of the auxiliary speaker-independent (SI) GMM-HMM model is performed for each speaker in the training corpus and a new speaker-adapted (SA) GMM-HMM model is created in order to obtain SA GMMD features.

For a given BN feature vector, a new GMMD feature vector is obtained by calculating log-likelihoods across all the states of the auxiliary GMM model on the given vector. Suppose o t is the BN feature vector at time t, then the new GMMD feature vector f t is calculated as in Formulas (6.3) and (7.1).

The obtained GMMD feature vector f t is concatenated with the original vector o t . After that, the features are spliced in time taking a context size of 13 frames: [-10,-5..5,10]. These features are used as the input for training a SAT DNN.

Training TDNN AMs with GMMD features

In addition to the system described in Section 8.2, we aim to explore the effectiveness of using GMMD features to train a time delay neural network (TDNN) [START_REF] Waibel | Phoneme recognition using time-delay neural networks[END_REF]. A TDNN model architecture allows to capture the long term dependencies in speech signal. The recently proposed approaches to train TDNN acoustic models [START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF] are reported to show higher performance on different LVCSR tasks compared with the standard (best) DNN systems. We aim to incorporate GMMD features into the existing state-of-the art recipe for TDNN model [START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF]. For comparison purposes, we take a Kaldi TED-LIUM recipe with a TDNN acoustic model as a basis. An example of using GMMD features for training a TDNN is shown in Figure 8.3. Here, as before, we use BN features to train the GMM auxiliary model for GMMD feature extraction. Then GMMD features are obtained in the same way as described in Section 8.2.

There are several options to obtain the final features which are fed to the TDNN model. First GMMD features can be combined with the original MFCC features. Another variant consists in combination of GMMD features with BN features, that are used for training the auxiliary GMM model, as shown in Figure 8.3. In both cases we can also use speaker i-vectors as auxiliary features (Section 4.3.6.1). We will experimentally explore all these possibilities in Section 8.6.3.

Baseline systems and addressed questions

The experiments were conducted on the TED-LIUM corpus (Section 5.2). We used the opensource Kaldi toolkit [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF] and mostly followed the standard TED-LIUM Kaldi recipes to train two sets of baseline systems, corresponding to two different types of acoustic models (DNN and TDNN)2 and two LMs (LM-cantab and LM-lium, see Section 5.2):

1. Baseline systems with DNN AMs. AMs in this set are DNNs trained on BN features, and for the baseline with speaker adaptation we used fMLLR adaptation. LM-cantab was used for decoding.

2. Baseline systems with TDNN AMs. This set corresponds to a TDNN AMs, with i-vectors and fMLLR for speaker adaptation and LM-lium for decoding.

Hence, for each set of baseline systems we trained several acoustic models -SI and SAT AMs.

The motivation for creating multiple baseline systems is to have a possibility to compare the proposed adaptation approach not only to some SI baselines, but also to the strong SAT AMs, which use conventional speaker adaptation techniques. Also, one of the question addressed in this study, is to explore several types of basic features to build an auxiliary GMM for GMMD feature extraction. For this reason, we (in addition to the standard original baselines, proposed in Kaldi recipes) built corresponding baseline AMs using the same basic features, as we used for GMMD feature extraction in order to have a more complete picture of experimental results and possibility to compare adaptation techniques under equal conditions. As we noticed already, besides the standard feed-forward DNNs which were the main focus of all the previous experiments, in this chapter, we also explore TDNN AMs, because recently they have been shown to outperform standard DNNs for many LVCSR tasks [START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF].

Thus, two series of experiments were conducted:

1. With DNN AMs:

• Baseline AMs: Section 8.4.2;

• LM: LM-cantab;

• Proposed systems with GMMD features: Section 8.5.1;

• Results: Preliminary results to choose basic features and optimal topology of an auxiliary GMM for GMMD feature extraction, as well as the adaptation parameter τ are given Section 8.6.1. Final results are provided in Section 8.6.2.

2. With TDNN AMs:

• Baseline AMs: Section 8.4.3;

• LM: LM-lium;

• Proposed systems with GMMD features: Section 8.5.2;

• Results: Section 8.6.3.

Questions addressed in this study

Several questions addressed in this study include:

1. One of the main questions explored in this chapter is how to effectively integrate the proposed GMMD features into state-of-the art AMs (DNN and TDNN). Some examples of integration were proposed in Sections 8.2 (for DNN) and 8.3 (for TDNN). But we also explore other ways, as will be described below in this chapter.

2. Explore factors which influence GMMD feature extraction:

• Basic features for training an auxiliary GMM model which is used for GMMD feature extraction:

-For DNNs we explore 39-dimensional MFCC+∆ + ∆∆ and 40-dimensional BN features (preliminary experiments, Section 8.6.1);

-For TDNNs we explore 40-dimensional high-resolution MFCC and 40dimensional BN features (Section 8.6.3);

• Topology of the auxiliary GMM model (Section 8.6.1)

• Parameter τ in MAP adaptation (see Formula (4.2)), that controls the balance between the maximum likelihood estimate of the mean and its prior value [START_REF] Gauvain | Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains[END_REF]] (Section 8.6.3).

3. Study different types of fusion (features-, posterior-, lattice-level) of GMMD features with other features (both SI and adapted) for DNNs (Section 8.6.2) and TDNNs (Section 8.6.3);

4. Compare the proposed adaptation technique with the two most popular adaptation approaches for neural network AMs:

• fMLLR (for DNNs and TDNNs, Sections 8.6.2 and 8.6.3);

• i-vectors (for TDNNs, Section 8.6.3).

5. Explore the complementarity of the proposed adaptation technique to other adaptation approaches:

• fMLLR (for DNNs and TDNNs)

• i-vectors (for TDNNs)

6. Investigate the impact of the training criteria on the adaptation performance. We compare CE and sMBR criteria for DNNs (Section 8.6.2).

Baseline DNN AMs

We trained four baseline DNN acoustic models:

• DNN BN -CE was trained on BN features with CE criterion.

• DNN BN -sMBR was obtained from the previous one by performing four epochs of sMBR sequence-discriminative training.

• DNN BN-fMLLR -CE was trained on fMLLR-adapted BN features.

• DNN BN-fMLLR -sMBR was obtained from the previous one by four epochs of sMBR training.

For training DNN models, the initial GMM model was trained first using 39-dimensional MFCC features including delta and acceleration coefficients. Linear discriminant analysis (LDA), followed by maximum likelihood linear transform (MLLT) and fMLLR transformation, was then applied over these MFCC features to build a GMM-HMM system. Discriminative training with the BMMI objective function (Formula (3.22)) was finally performed on top of this model.

Then a DNN was trained for BN feature extraction. The DNN system was trained using the frame-level cross entropy criterion and the senone alignment generated by the GMM-HMM system. To train this DNN, 40-dimensional log-scale filterbank features concatenated with 3-dimensional pitch-features were spliced across 11 neighboring frames, resulting in 473-dimensional (43 × 11) feature vectors. After that a DCT transform was applied and the dimension was reduced to 258. A DNN model for extraction 40-dimensional BN features was trained with the following topology: one 258-dimensional input layer; four hidden layers (HL), where the third HL was a BN layer with 40 neurons and other three HLs were 1500dimensional; the output layer was 2390-dimensional. Based on the obtained BN features we trained the GMM model, which was used to produce the forced alignment, and then SAT-GMM model was trained on fMLLR-adapted BN features. Then fMLLR-adapted BN features were spliced in time with the context of 13 frames: [-10,-5..5,10] to train the final DNN model. The final DNN had a 520-dimensional input layer; six 2048-dimensional HLs with logistic sigmoid activation function, and a 4184-dimensional softmax output layer, with units corresponding to the context-dependent states.

The DNN parameters were initialized with stacked restricted Boltzmann machines (RBMs) by using layer-by-layer generative pre-training. It was trained with an initial learning rate of 0.008 using the cross-entropy objective function to obtain the SAT-DNN-CE model DNN BN-fMLLR -CE.

After that four epochs of sequence-discriminative training with per-utterance updates, optimizing state sMBR criteria, were performed to obtain the SAT-DNN-sMBR model DNN BN-fMLLR -sMBR.

Baseline SI DNN models (DNN BN -CE and DNN BN -sMBR) were trained in a similar way as the SAT DNNs described above, but without fMLLR adaptation.

Baseline TDNN AMs

We trained five baseline TDNN acoustic models, which differ only in the type of the input features:

• TDNN MFCC was trained on high-resolution (40-dimensional) MFCC features.

Baseline systems and addressed questions

123

• TDNN MFCC⊕ ⊕ ⊕i-vectors was trained on high-resolution MFCC features, appended with 100-dimensional i-vectors.

• TDNN BN was trained on BN features.

• TDNN BN⊕ ⊕ ⊕i-vectors was trained on BN features, appended with 100-dimensional ivectors.

• TDNN BN-fMLLR was trained on fMLLR-adapted BN features.

The baseline SAT-TDNN model TDNN MFCC⊕ ⊕ ⊕i-vectors is similar to those described in [START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF], except for the number of hidden layers and slightly different subsequences of splicing and sub-sampling indexes. The two types of data augmentation strategies were applied for the speech training data: speed perturbation (with factors 0.9, 1.0, 1.1) and volume perturbation. The SAT-TDNN model was trained on high-resolution MFCC features (without dimensionality reduction, keeping all 40 cepstra) concatenated with i-vectors. The 100-dimensional on-line i-vectors were calculated as in [START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF], and the statistic for i-vectors was updated every two utterances during the training.

The temporal context was [t -16,t + 12] and the splicing indexes used here were3 [-2, 2],

{-1, 2}, {-3, 3}, {-7, 2}, {0}, {0}. This model has 850-dimensional hidden layers with rectified linear units (ReLU) [START_REF] Dahl | Improving deep neural networks for LVCSR using rectified linear units and dropout[END_REF] activation functions, a 4052-dimensional output layer and approximately 10.9 million parameters.

The baseline SI-TDNN model TDNN MFCC was trained in a similar way as the SAT-TDNN described above, but without using i-vectors.

In addition to these baseline models, for comparison purpose, we trained two other baseline TDNNs (with and without i-vectors) using BN features instead of high-resolution MFCC features. The same BN features we used later for training an auxiliary monophone GMM model for GMMD feature extraction. These BN features were extracted using a DNN trained in a similar way, as described in Section 8.4.2 for the DNN AM, but on the highresolution MFCC features (instead of "filter bank ⊕ pitch" features) and on the augmented (by means of speed and volume perturbation) data base.

Finally, a TDNN with fMLLR adaptation on BN features was trained (TDNN BN-fMLLR ).

Proposed systems with GMMD features

For experiments with speaker adaptation we trained two types of acoustic models -DNNs and TDNNs.

DNN AMs on GMMD features

In this set of experiments we trained four DNNs, using the approach proposed in Section 8.2:

• DNN GMMD⊕ ⊕ ⊕BN -CE is a DNN trained without performing speaker adaptive training, which in our case means that the auxiliary GMM monophone model was not adapted.

The model was trained using CE criterion.

• DNN GMMD⊕ ⊕ ⊕BN -sMBR was obtained from the previous one by performing four epochs of sequence-discriminative training with per-utterance updates, optimizing sMBR criterion.

• DNN GMMD-MAP⊕ ⊕ ⊕BN -CE was a proposed SAT DNN model trained on speaker adapted GMMD-MAP features, with CE criterion.

• DNN GMMD-MAP⊕ ⊕ ⊕BN -sMBR was obtained from the previous one by performing four epochs of sMBR sequence training.

Models DNN GMMD-MAP⊕ ⊕ ⊕BN -CE and DNN GMMD-MAP⊕ ⊕ ⊕BN -sMBR were trained as described in Section 8.2. The GMMD features were extracted using a monophone auxiliary GMM model, trained on BN features. This GMM model was adapted for each speaker by MAP adaptation algorithm (Section 4.2.1). We took the state tying from the baseline SAT-DNN to train all other models. The purpose of using the same state tying is to allow posterior level fusion for these models.

Both DNN models were trained on the proposed features in the same manner and had the same topology except for the input features, as the final baseline SAT DNN model trained on BN features (Section 8.4.2).

The other two SI models (DNN GMMD⊕ ⊕ ⊕BN -CE and DNN GMMD⊕ ⊕ ⊕BN -sMBR) were trained in the same manner, but without speaker adaptation.

TDNN AMs on GMMD features

In this set of experiments we trained five TDNNs, using the approach proposed in Section 8.3.

• TDNN MFCC⊕ ⊕ ⊕GMMD was trained on high-resolution MFCC features appended with speaker adapted GMMD features.

• TDNN MFCC⊕ ⊕ ⊕GMMD⊕ ⊕ ⊕i-vectors is a version of the TDNN MFCC⊕ ⊕ ⊕GMMD model with 100dimensional i-vectors appended to input features.

• TDNN BN⊕ ⊕ ⊕GMMD was trained on BN features appended with speaker adapted GMMD features.

• TDNN BN⊕ ⊕ ⊕i-vectors⊕ ⊕ ⊕GMMD is a version of the TDNN BN⊕ ⊕ ⊕GMMD with 100-dimensional i-vectors appended to input features.

• TDNN BN-fMLLR⊕ ⊕ ⊕GMMD was trained on fMLLR-adapted BN features appended with speaker adapted GMMD features.

All the five TDNN models were trained in the same manner, as the baseline TDNN model (Section 8.4.3), and differ only in the type of the input features.

Adaptation and fusion results

In this section, the adaptation experiments were conducted in an unsupervised mode on the test data using transcripts from the first decoding pass obtained by the baseline SAT-DNN model, unless explicitly stated otherwise.

8.6.1 Preliminary results: impact of the auxiliary GMM and parameter τ in MAP adaptation on GMMD features

We aim to explore two factors related to the auxiliary GMM, used for GMMD feature extraction: [START_REF] Tomashenko | On the use of Gaussian mixture model framework to improve speaker adaptation of deep neural network acoustic models[END_REF] the topology of the model and ( 2) the type of input features for training this model, and choose the configuration, which is more effective for GMMD feature extraction. We experimented with the following parameters of GMM model: the total number of Gaussians and their distributions between states. Also GMM models were trained on two different types of input features: 39-dimensional MFCC and BN features, extracted as described in Section 8.4.2. In addition, we extracted features with different values of adaptation parameter τ (in Formula (4.2) τ is the parameter that controls the balance between the maximum likelihood estimate of the mean and its prior value in MAP adaptation).

In order to speed up these preliminary experiments, we performed them on a smaller (85 hours) subset of the training dataset. The performance results in terms of WER for DNN models, used for BN feature extraction, are presented in Table 8.1. Parameter Power in the table controls the distribution of number of Gaussians between states in the GMM-HMM model according to the distribution of data in the training corpus between these states. It is the exponent for the number of Gaussians according to occurrence counts. When Power = 0, all states in the GMM-HMM are modeled with the same number of Gaussians, otherwise the number of Gaussians, used to model a given state, depends on the amount of data in the training corpus belonging to this state.

We can see that for GMMD feature extraction it is better to train an auxiliary GMM model on BN features than on MFCC, and that equal distribution of number of Gaussians between states (Power = 0) performs worse than distribution which is dependent on occurrence counts. We set parameter τ = 5 for all the following experiments. As we can see from Table 8.2, using speaker-independent (SI) GMMD features in concatenation with SI BN features (#5 and #6) improves the SI baselines (#1 and #2 correspondingly) for all data sets, for both training criteria (CE and sMBR).

By comparing results for AMs trained with CE criterion in lines #5 and #7, we can conclude that the proposed adaptation technique with MAP provides for different datasets approximately 14-18% of relative WER reduction with respect to the SI AM trained on GMMD⊕BN features. For sMBR criterion (lines #6 and #8) the relative WER reduction is approximately 7-13%.

If we compare AMs adapted with fMLLR (#3, #4) with AMs adapted with MAP (#7, #8 correspondingly), we observe that for AMs trained with CE criterion (#3 and #7) the MAP-adapted AM outperforms the fMLLR-adapted AM, and relative WER reduction for different datasets is in the interval 8-11%. However, for AMs trained with sMBR criterion this gain reduces to 1-3% of relative WER reduction.

Table 8.2 shows the effectiveness of feature-level fusion of GMMD and conventional BN features. We are also interested in other types of fusion (posterior-and lattice-level). For these experiments we chose the best AM, trained on MAP-adapted GMMD features (#8) and the AM, trained on fMLLR-adapted BN features (baseline SAT AM: #4). After that, we made posterior fusion for these AMs. The result is given in Table 8.3,line #9 Finally, we applied lattice fusion (Section 8.1.3) for the same pair of models (line #10). In this type of fusion, before merging lattices, for each edge, scores were replaced by their a posteriori probabilities. Posteriors were computed for each lattice independently. The optimal normalizing factors for each model were found independently on the development set. Then the two lattices were merged into a single lattice and posteriors were weighted using parameter α. As before, value α in Formula (8.1) corresponds to the weight of the baseline SAT-DNN model. The resulting lattice was converted into the confusion network [START_REF] Mangu | Finding consensus in speech recognition: word error minimization and other applications of confusion networks[END_REF] and the final result was obtained from this confusion network.

We can see that both posterior and lattice types of fusion provide similar improvement for all three models: approximately 4%-6% of relative WER reduction in comparison with the adapted baseline model (SAT DNN on fMLLR features, #4), and 12%-18% of relative WER reduction in comparison with the SI baseline model (#2). For models #7-8 only MAP adaptation was applied. Table 8.4 Summary of unsupervised adaptation results for TDNN models on TED-LIUM corpus. The results in parentheses correspond to WER of the consensus hypothesis (they are given only for those results, that will be used further in fusion). Indices, following GMMD features, denotes the AM, used to obtain transcriptions for adaptation. This section presents the results of using the described approach for speaker adaptation as a part of the LIUM ASR system [START_REF] Tomashenko | LIUM ASR systems for the 2016 multi-genre broadcast Arabic challenge[END_REF]] in 2016 Multi-Genre Broadcast (MGB-2) Challenge in the Arabic language [START_REF] Ali | The MGB-2 challenge: Arabic multi-dialect broadcast media recognition[END_REF]. One objective of this section is to study the effectiveness of the proposed adaptation technique when it is directly applied to novel conditions: language, size of the training database, amount of adaptation data for training and test speakers, etc. The language (MSA Arabic, switching to various Arabic dialects), the quality of audio recordings and transcriptions in the training database, as well as other factors make this task more difficult for ASR, and resulted in much higher WER of the baseline system in comparison with the results, reported for WSJ0 and TED-LIUM datasets. The amount of the training data used in the Arabic MGB 2016 dataset is approximately 3.8 times more than in TED-LIUM dataset. Also, the amount of data in development and test datasets is not balanced with respect to speakers. Another aspect addressed in this experiments is to show the performance of AMs trained on GMMD features in combination (particularly, lattice-level fusion) with more different AMs than before (in the previous experiments we reported lattice-level fusion results only for two AMs).

Acoustic models

The Kaldi speech recognition toolkit [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF] was used for AM training. The corpus description is given in Section 5.4. The LIUM developed five different AMs, while the same LM set was used for every system. Below we will provide a brief description of these AMs; further details can be found in [START_REF] Tomashenko | LIUM ASR systems for the 2016 multi-genre broadcast Arabic challenge[END_REF]. All the AMs except the second one (TDNN 2 ) were built using the grapheme-based lexicon.

TDNN chain model with 1-to-1 word-to-grapheme lexicon (TDNN 1 )

The first acoustic model is a recently proposed type of model in the Kaldi toolkit, the socalled chain TDNN model [START_REF] Povey | Purely sequence-trained neural networks for asr based on lattice-free mmi[END_REF]. Training this model was done by using high-resolution PLP features (without dimensionality reduction, keeping the 40 cepstra) concatenated with 100-dimensional i-vectors for speaker adaptation. We also, as in a standard Kaldi recipe, applied data augmentation techniques before performing the actual training, namely time-warping of the raw audio by factors of 0.9, 1.0 and 1.1, as well as frame-shifting by 0, 1 and 2 frames. On top of this network, we performed a sequence-discriminative training, using the sMBR criterion (Formula (3.3.3.1)).

TDNN chain model with vowels and phonetization (TDNN 2 )

This acoustic model is a TDNN chain model built in the same way as the one described above, to the exception that no sequence-discriminative training was performed. The model was trained on a realigned vowelized training set, where words were replaced by the two best diacritization candidates provided by MADAMIRA toolkit [START_REF] Pasha | MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic[END_REF]. Thereby, the phoneme-based lexicon was used for training, while pronunciations were mapped to the grapheme words in the decoding process.

DNN on fMLLR-adapted BN features (DNN BN )

This DNN model was trained on fMLLR-adapted 40-dimensional BN features, spliced in time with the context of 13 frames: [-10,-5...5,10]. It had a 520-dimensional input layer; six 2048dimensional hidden layers with logistic sigmoid activation function, and a 8467-dimensional softmax output layer, with units corresponding to the context-dependent states. The DNN parameters were initialized with stacked RBMs by using layer-by-layer generative pretraining and trained with CE objective function. After that four epochs of sequence-discriminative training with per-utterance updates, optimizing sMBR criterion, were performed.

DNN on MAP-adapted GMMD features (DNN GMMD )

For training this DNN acoustic model we used speaker-adapted GMMD features. The MAP-adapted GMMD 130-dimensional feature vectors were concatenated with unadapted 40-dimensional BN feature vectors and spliced across 13 frames as before [-10,-5...5,10], resulting in 2210-dimensional feature vectors, for training a new DNN model. The topology of DNN GMMD is similar to DNN BN , except for the dimension of the input layer. Further DNN GMMD was trained in the same way as DNN BN model. In the experimental results, for adaptation on the Dev set, we used the best transcripts, obtained from other AMs.

TDNN on MAP-adapted GMMD features (TDNN GMMD )

For training this TDNN acoustic model we used the same features, as for the model DNN GMMD : speaker-adapted GMMD features concatenated with speaker-independent BN features. Except for the difference in the input features and the absence of data augmentation, the TDNN GMMD model was trained in the same way as TDNN 1 . We did not not perform sequence training for this model (particularly due to competition constrains), hence in the sense of training criteria this model is weaker than the TDNN 1 model, and we can not compare them directly. Also no data augmentation was applied for this AM. However, the experiments reported in this section are interesting because they can provide additional understanding of how systems built on GMMD features perform in combination with the other systems built on conventional features.

Results

As earlier in this chapter (Section 8.6.2), we applied the fusion of the recognition results from different AMs, on the word lattice level (Section 8.1.3), in the form of CNC [START_REF] Evermann | Posterior probability decoding, confidence estimation and system combination[END_REF].

Table 8.6 summaries the recognition results in terms of WER on the Dev dataset. Results are reported after rescoring word-lattices with the 4-gram LM described in Section 5.4. These word-lattices were obtained by using the 2-gram LM to decode the audio signal. Numbers in the columns 3-7 are weights of the recognition results (lattices) from corresponding AMs in CNC. Lines 1-5 represent results for single AMs, both for the 1-best result from lattices, and for the 1-best result from consensus hypotheses. Lines 6-15 show results for pairwise combination, lines 16-25 -fusion results for three models, and lines 26-30 -fusion results for four models. Finally, line 31 demonstrates the WER=19.48%, obtained by fusion of the results from all the five AMs.

If we compare lines #5 and #24, we can see, that by adding in the combination the AMs, trained with the use of GMMD features, we can achieve 7.6% of relative WER reduction in comparison with best single AM (TDNN 1 ). Even when we combine three AMs (#20, WER=19.86%), we can further slightly improve the result using GMMD features (#31, WER=19.48%) by 2% of relative WER reduction.

We do not report results on the Test dataset, because they were computed by the MGB organizers and did not cover all our experiments on the Dev dataset. But those results that were available showed that results on Test and Dev were consistent. For more details see [START_REF] Tomashenko | LIUM ASR systems for the 2016 multi-genre broadcast Arabic challenge[END_REF].

Discussion

In this section, we described the application of the proposed adaptation approach to the LIUM ASR system, that has been ranked in second position in the 2016 Arabic MGB Challenge. In comparison with the previous experiments, the AMs, described in this section, were trained on a larger speech corpus (648 hours) and were used without any a priori knowledge about test speaker statistics. Also, combining more than two models by CNC, we can observe, that GMMD features continues to provide an improvement in the overall ASR accuracy.

Conclusions

In this chapter we have investigated the GMM framework for adaptation of DNN-HMM acoustic models and combination of MAP-adapted GMMD with conventional features at different levels of DNN and TDNN architectures [Tomashenko et al., 2016a,b].

Experimental results on the TED-LIUM corpus demonstrated that, in an unsupervised adaptation mode, the proposed adaptation and fusion techniques can provide approximately, a 12-18% relative WER reduction on different adaptation sets, compared to the SI DNN system built on conventional (MFCC) features, and a 4-6% relative WER reduction compared to the strong adapted baseline -SAT-DNN trained on fMLLR adapted features. For TDNN models using the adapted GMMD features and fusion techniques leads to improvement of 15-28% of relative WER reduction in comparison with SI model trained on conventional features and 8-15% of relative WER reduction in comparison with SAT model trained with i-vectors. Hence, for both considered adaptation techniques, fMLLR and i-vectors, the proposed adaptation approach has appeared to be complementary and provide an additional improvement in recognition accuracy.

In addition, we reported the results of applying the proposed adaptation approach to the LIUM ASR system in the 2016 Arabic MGB Challenge, and demonstrated that this complicated ASR system, trained with many hundred hours of speech data and resulting from a combination of several AMs, can also benefit from the use of the adapted GMMD features [START_REF] Tomashenko | LIUM ASR systems for the 2016 multi-genre broadcast Arabic challenge[END_REF].

Chapter 9

Towards speaker adaptation for end-to-end AMs

In this chapter we explore effectiveness of the proposed speaker adaptation technique for end-to-end AMs.

Introduction

Recently, various neural end-to-end approaches to ASR have been proposed in the literature [START_REF] Audhkhasi | Direct acoustics-to-word models for english conversational speech recognition[END_REF][START_REF] Bahdanau | End-to-end attention-based large vocabulary speech recognition[END_REF][START_REF] Chan | Listen, attend and spell: A neural network for large vocabulary conversational speech recognition[END_REF][START_REF] Collobert | Wav2letter: an end-to-end ConvNetbased speech recognition system[END_REF][START_REF] Fritz | End-to-end MAP training of a hybrid HMM-DNN model[END_REF][START_REF] Hannun | Deep speech: Scaling up end-to-end speech recognition[END_REF]. End-to-end acoustic models (AMs) [START_REF] Chorowski | End-to-end continuous speech recognition using attention-based recurrent NN: First results[END_REF][START_REF] Graves | Towards end-to-end speech recognition with recurrent neural networks[END_REF]Miao et al., 2015a;[START_REF] Zhang | Towards end-to-end speech recognition with deep convolutional neural networks[END_REF] attempt to map an acoustic signal to a phoneme or grapheme sequence directly by means of neural network models (see Section 2.4). They have been developed as an alternative to the traditional hybrid HMM-DNN approach.

However, the major part of the published works, devoted to end-to-end technology, does not use any speaker adaptation techniques. This lack may be justified by the strong focus of these papers on the neural core of the technology they introduce.

A few papers have offered some preliminary and promising information about the benefits provided by some speaker adaptation techniques to end-to-end AMs. In [START_REF] Miao | An empirical exploration of CTC acoustic models[END_REF], VTLN has been applied to filterbank features, for a neural end-to-end AM training through CTC criterion, providing 3% of relative WER reduction. Speaker i-vectors, appended to the acoustic features, are used in [START_REF] Audhkhasi | Direct acoustics-to-word models for english conversational speech recognition[END_REF] for training phone and word CTC models. Also fMLLR-adapted features are used to train attention-based RNNs [START_REF] Chorowski | End-to-end continuous speech recognition using attention-based recurrent NN: First results[END_REF]. However in these works [START_REF] Audhkhasi | Direct acoustics-to-word models for english conversational speech recognition[END_REF][START_REF] Chorowski | End-to-end continuous speech recognition using attention-based recurrent NN: First results[END_REF], no comparison results with the unadapted models are given. Work [START_REF] Yi | CTC regularized model adaptation for improving LSTM RNN based multi-accent mandarin speech recognition[END_REF] proposes a CTC regularized model adaptation method for the accent adaptation task. Speaker adaptation with speaker codes of RNN-BLSTM AMs is studied in [START_REF] Huang | Speaker adaptation of RNN-BLSTM for speech recognition based on speaker code[END_REF] for the phone recognition task. In [START_REF] Huang | Speaker adaptation of RNN-BLSTM for speech recognition based on speaker code[END_REF] AMs were trained with CE criterion, and the adaptation provides about 10% of relative reduction in phone error rate.

The aim of this chapter is to explore the efficiency of speaker adaptation for end-to-end ASR systems with the focus on the proposed approach, on the example of CTC-BLSTM AMs (or shortly, CTC AMs). For this purpose we implemented three different speaker adaptation algorithms to this type of AMs and performed an experimental analysis of these methods. Furthermore, a comparative study of the adaptation techniques was conducted for CTC AMs and TDNN AMs trained with traditional frame-wise CE criterion.

Speaker adaptation for BLSTM-CTC models

In this chapter we focus on the feature space adaptation techniques for end-to-end acoustic models. In this section we describe the adaptation approach, which is based on using speaker-adapted GMMD features for training BLSTM-CTC models.

The incorporation of the adapted GMMD features into the recipe for training sequenceto-sequence AMs is straightforward. The scheme for SAT of BLST-CTC AMs models with GMM-based adaptation framework is shown in Figure 9.1.

An auxiliary monophone GMM-HMM model is used to transform acoustic feature vectors into log-likelihoods vectors. At this step, speaker adaptation of the auxiliary SI GMM-HMM model is performed for each speaker in the training corpus using correct transcriptions and a new speaker-adapted GMM-HMM model is created in order to obtain speaker-adapted GMMD features.

For a given acoustic feature vector, a new GMMD feature vector is obtained by calculating log-likelihoods across all the states of the auxiliary GMM model on the given vector. Suppose o t is the acoustic feature vector at time t, then the new GMMD feature vector f t is calculated as in Formulas (6.3) and (7.1).

The adapted GMMD feature vector f t is concatenated with the original vector o t to obtain vector x t . These features are used as the input to train a SAT BLSTM-CTC AM. For the AMs trained on features #5 and #6, the 100-dimensional on-line i-vectors were calculated as in [START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF], and the statistic for i-vectors was updated every two utterances during the training.

For AMs #7-#9 we used BN features to train the auxiliary GMM model for GMMD feature extraction. The speaker-adapted GMMD features were obtained in the same way as described in Section 9.2. Parameter τ in MAP adaptation (see Formula (7.2)) was set equal to 5 for both acoustic model training and decoding.

Adaptation results for CTC AMs

The adaptation experiments were conducted in an unsupervised mode on the test data using transcripts from the first decoding pass.

The performance results in terms of word error rate (WER) for SI and SAT AMs models are presented in Table 9.1. The first three lines of the table (#1-#3) correspond to the baseline SI AMs, which were trained as described in Section 9.3.1, where the very first line represents the Eesen baseline [START_REF] Miao | Towards speaker adaptive training of deep neural network acoustic models[END_REF]. The next six lines (#4-#9) show the results for the adapted models. The numeration in Table 9.1 coincides with the numeration in Sections 9.3.1 and 9.3.2.

The two last lines of the table (#10 and #11) are obtained with the same AMs, as the lines #8 and #9 correspondingly, but for the extraction of GMMD-adapted features in #10 and #11 we used the transcriptions from the adapted model #6). Notice, that for all other tests (#7-#9) we used transcriptions from the SI model #2.

The best result among all the systems #1-#9 is obtained by system #9, which corresponds to the use of MAP-adapted GMMD features appended with fMLLR-adapted BN features. It can be only slightly improved (#11) for two sets by using the adapted model on the first decoding pass. Among all the adaptation methods, applied separately (#4-#8), the MAP adaptation of GMMD features shows the best performance for both BN and MFCC features.

Comparison of adaptation behavior for BLSTM-CTC and TDNN

AMs

In this series of experiments we aim to compare the adaptation behavior of SAT CTC models with the different type of neural network AMs. For this purpose we chose a TDNN model topology, because such models are shown to achieve the best performance result in many state-of-the ASR systems [START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF]. These AMs were trained with the CE criterion.

For comparison we used the same set of SI and SAT AMs, as before for CTC-AMs (see Sections 9.3.1 and 9.3.2), except for #1. All SI ans SAT TDNN models were trained in a similar way and have the same model topology. They differ only in the type of the input features. These are the same AMs, as were described in Sections 8.4.3 and 8.5.2. Note that not all the TDNN AMs explored in the mentioned sections are used in the current chapter. The results for TDNN AMs are reported in Table 9.2. They include some results from Table 8.4, but we show them here again for clarity and ease of comparison with CTC AMs. Also Figure 9.2 presents the comparison of different adaptation algorithms in terms of relative WER reduction for the speakers from test and development datasets for BLSTM-CTC (Figure 9.2a) and TDNN (Figure 9.2b) AMs. The relative WER reduction is calculated with respect to the SI AMs trained on BN features.

For TDNN AMs we also added in Figure 9.2b the results obtained with the use of SAT AMs for the first decoding pass, because they provide a consistent additional improvement in performance in comparison with the use of SI AMs.

Table 9.3 shows the relative WER reduction for BLSTM-CTC and TDNN AMs in comparison with the best corresponding SI AMs (#2 for CTC and #3 for TDNN). We can see that the optimal choice of features depends on the AM architecture. For SI AMs, BNs have appeared to perform better than MFCCs for TDNN AMs, but for CTC AMs the situation is reversed. Also for SAT CTC and SAT TDNN AMs, the ranking of the systems by the WER is different.

Conclusions

This chapter explored how the end-to-end AMs can benefit from speaker adaptation and demonstrated that speaker adaptation has remained an essential mechanism for improving the performance of an ASR system in the new end-to-end speech recognition paradigm.

Experimental results on the TED-LIUM corpus showed that in an unsupervised adaptation mode, the adaptation and data augmentation techniques can provide approximately a 10-20% relative WER reduction on different adaptation sets, compared to the SI BLSTM-CTC system built on filter-bank features. In average, the best results for BLSTM-CTC AMs were obtained by using GMMD features and MAP adaptation, which can be further slightly improved by combination with fMLLR adaptation technique. We found out, that the type of the neural network AM architecture can differently influence the adaptation performance. The comparison with the TDNN-CE AMs showed that for these models, in contradiction to BLSTM-CTC AMs, MAP adaptation using GMMD features outperforms fMLLR only when it uses SAT model in the first decoding pass to obtain transcriptions for adaptation.

Also the obtained results allow us to compare TDNN-CE and BLSTM-CTC AMs in the realistic conditions, when the speaker adaptation is applied, which is important because usually end-to-end and hybrid AMs are compared on incomplete unadapted systems. The best SI TDNN-CE AM outperforms the best SI BLSTM-CTC AM by 1-7% of relative WER reduction for different test sets. For the best SAT AMs this gap in WER for TDNN-CE and BLSTM-CTC AMs increases and reaches 5-13% of relative WER reduction. Chapter 10

GMMD feature analysis

The objective of this chapter is to analyze the proposed GMMD features and the adaptation algorithm for better understanding their nature and properties at different levels.

Phoneme posterior based features

Phoneme posterior based (PPB) features are obtained from time dependent phoneme posterior scores [START_REF] Gollan | Confidence scores for acoustic model adaptation[END_REF][START_REF] Uebel | Improvements in linear transform based speaker adaptation[END_REF]] by computing arc posteriors from the output lattices of the decoder. These features contain more information about the decoding process, than the posterior probabilities from neural networks. We use this type of features to analyze the adaptation performance for TDNN acoustic models.

Let {ph 1 , . . . , ph N } be a set of phonemes and the silence model. For each time frame t we calculate p n t -the confidence score of phoneme ph n (1 ≤ n ≤ N) at time t in the decoding lattice by calculating arc posterior probabilities. The forward-backward algorithm is used to calculate these arc posterior probabilities from the lattice as follows:1 P(l|O) = ∑ q∈Q l P ac (O|q)

1 λ P lm (w) P(O) , (10.1)
where λ is the scale factor (the optimal value of λ is found empirically by minimizing WER of the consensus hypothesis [START_REF] Mangu | Finding consensus in speech recognition: word error minimization and other applications of confusion networks[END_REF]); q is a path through the lattice corresponding to the word sequence w; Q l is the set of paths passing through arc l; P ac (O|q) is the acoustic likelihood; P lm (w) is the language model probability; and P(O) is the overall likelihood of all paths through the lattice.

For the given frame o t at time t we calculate its probability P(o t ∈ ph n ) of belonging to phoneme ph n , using lattices obtained from the first decoding pass:

p n t = P(o t ∈ ph n ) = ∑ l∈S n (o t ) P(l|O), (10.2)
where S n (o t ) is the set of all arcs corresponding to the phoneme ph n in the lattice at time t; P(l|O) is the posterior probability of arc l in the lattice.

ASR Calculation of phoneme posterior probabilities

Word lattices phoneme alignments acoustic feature vector The obtained probability P(o t ∈ ph n ) of frame o t belonging to phoneme ph n is the coordinate value p n t on the new feature vector p t . Thus for a given acoustic feature vector o t at time t we obtain a new feature vector p t : p t = p 1 t , . . . , p N t , (10.3) where N is the number of phones in the phoneme set used in the ASR system.

Hence for each frame o t we have a N-dimensional vector p t , each coordinate of which represents the probability of this frame to belong to a certain phoneme. When some phonemes are not present in the lattice for a certain frame, we set probabilities equal to some very small value ε for them in the vector (where ε is a minimum value from lattices: ε ≈ 10 -9 ). In addition to this, we use state index information (position of the state in phoneme HMM: 0, 1 or 2) from the Viterbi alignment from the original transcripts.

Visual analysis using t-SNE

The PPB features were visualized using t-distributed stochastic neighbor embedding (t-SNE) analysis [START_REF] Maaten | Visualizing data using t-SNE[END_REF]. This technique allows us to visualize high-dimensional data into two or three dimensional space, in such a way that the vectors, which are close in the original space, are also close in the low dimensional t-SNE representation.

We are interested in how well the different acoustic models can cluster different phoneme states. For better visualization we used data only from inter-word phones and only from the middle state of HMM (State 1). We choose only those phonemes for which we have sufficient amount of data for analysis and perform t-SNE analysis independently on three different groups of phonemes2 :

• Vowels (UH, OW, AO, EY, ER, AA, AY, IY, EH, AE, IH, AH);

• Consonants-1: Liquids (L, R), Nasals (M, N, NG), Semivowels (W);

• Consonants-2: Stops (P, T, D, K), Affricates (CH), Fricatives (F, V, TH, S, Z, SH).

Each phoneme in these groups represents a separate cluster in our study.

Davies-Bouldin index

In order to support the visual analysis, we also use the Davies-Bouldin (DB) index [START_REF] Davies | A cluster separation measure[END_REF] to evaluate the quality of the phoneme state clusters obtained from the PPB features:

DB = 1 K K ∑ k=1 max j̸ =k σ k + σ j ρ k, j , (10.4)
where K is the number of clusters;

σ k is the scatter within the cluster k, which is our case the standard deviation of the distance of all vectors corresponding to cluster k, to the cluster center (other possible metric variants are described in [START_REF] Davies | A cluster separation measure[END_REF]); ρ k, j is a between-cluster separation measure, which in our case is the Euclidean distance between the centroids of clusters k and j.

Smaller values of DB index correspond to better clusters.

Analysis for TDNN models

In this set of experiments we compare the following three TDNN acoustic models: TDNN MFCC , TDNN BN⊕ ⊕ ⊕i-vectors and TDNN BN⊕ ⊕ ⊕i-vectors⊕ ⊕ ⊕GMMD (from Chapter 8). All the experiments described in this section (except for Figure 10.6) are performed using PPB features (Section 10.1) on the Development dataset.

First we analyzed the adaptation algorithm using t-SNE (Section 10.2). The results of the visual t-SNE analysis are given in Figure 10.2 for the group of vowels and in Figures 10.3,10.4 -for the two group of consonants. We can observe for all groups of phonemes that the adapted features (Figures 10.2b,10.3b,10.4b) form more distinct and clear phone clusters than the unadapted features (Figures 10.2a,10.3a,10.4a). Also we can note that the use of GMMD features helps to further slightly improve cluster separability (Figures 10.2c,10.3c,10.4c).

To support this visual analysis of cluster separation, we calculated DB index (Section 10.3) for all phonemes, separately for each state type, depending on its position in phoneme HMM (State 0, 1, 2). As we can see in Table 10.1, DB index decreases for all HMM states when we move from unadapted (MFCC) to adapted (BN ⊕ i-vectors) features. That confirms the fact the clusters are better for adapted features. The acoustic model with the adapted GMMD features (BN ⊕ i-vectors ⊕ GMMD) shows best result (the smallest value of DB index).

In order to more deeply investigate the adaptation behavior, we calculated additional statistics for PPB features (Table 10.2). Frame error rate (FER) is calculated on the phoneme level using only speech frames (excluding silence). Oracle FER was calculated also only on speech frames as follows: if the correct phoneme is not present in the list of all candidates in the lattices for a given frame, then it was considered as an error. We can see that FER decreases when moving from the unadapted features to the adapted ones, and then to the use of the adapted GMMD features, that correlates with the WER behavior (Table 8.4). It is interesting to note, that Oracle FER, on the contrary, increases with the adaptation. One of the possible explanation for this unusual situation can be phonetic transcriptions errors which occur in the lexicon. The adapted models, which can be more sensitive to the phonetic transcription errors, can more strictly supplant, during the decoding process, hypotheses that do not match the sound.

Decoding parameters, such as decoding beam and lattice beam, were the same for all models, but the adapted models in average have less alternative phoneme candidates for a certain frame, than the unadapted one. This can be seen in Figure 10.5a, which shows the cumulative distribution functions (CDFs) of the number of phoneme alternatives presented in the lattices for a certain frame, estimated only for speech frames. Figure 10.5b demonstrates CDFs of position of the correct phoneme (if it exists) in lattices for a certain speech frame in the list of all phoneme candidates ordered by their posterior probabilities. We can conclude from this figure, that for adapted models (especially for the AM with GMMD features), the correct candidate has less incorrect alternatives with higher probabilities than its own. Also, the average correct log-probability (it is a value from a PPB features vector, which corresponds to the correct phoneme for a given frame) has a maximum value for TDNN BN⊕i-vectors⊕GMMD model (see the last column of Table 8.4 and a histogram on Figure 10.5c).

Hence, if we compare the statistics presented in Table 10.2 and in Figure 10.5, we can conclude that the adapted models tend to be more "selective" and "discriminative" in comparison with unadapted models in the sense that: (1) they reduce the number of alternatives in the hypothesis more aggressively; (2) they give higher probability values for correct candidates; and (3) the correct phoneme candidate, if it exists in the lattice for a given frame, has in average, less incorrect competitor alternatives with higher probabilities than its own. The AM trained on the adapted GMMD features most strongly shows the same properties.

This analysis demonstrates that the acoustic models trained with the proposed adapted GMMD features perform better than the baseline adapted model not only by comparing the WER (which is the main metric), but also on the other levels.

Also, what is important, this gives us an understanding of the possible way of the adaptation performance improvement through more careful handling of the transcripts, for example, by automatically estimating their quality and reliability.

Finally, Figure 10.6 shows the statistics obtained for 42 speakers from Development, Test 1 and Test 2 data sets for TDNN BN and TDNN BN⊕ ⊕ ⊕GMMD models. We can observe that the proposed adaptation approach improves recognition accuracy for 83% of speakers.

Conclusions

We have looked from the various points of view at the proposed adaptation approach exploring the phoneme posterior based features, generated from the decoding lattices and have demonstrated, that the advantage of using MAP-adapted GMMD features manifests itself at different levels of the decoding process. This analysis also shows a possible potential and direction for improvement of the proposed adaptation approach through more careful handling of quality of the phonetics transcripts used in adaptation. This will be a focus of our future work.

Chapter 11

Conclusions and future work

Overview of contributions

We have developed and investigated a novel technique for adaptation of DNN AMs. Chapter 6 introduced a feature-space transformation method for the adaptation of DNN-HMM models [START_REF] Tomashenko | Speaker adaptation of context dependent deep neural networks based on MAP-adaptation and GMM-derived feature processing[END_REF], where input features for DNN training are generated from the likelihoods of the GMM-HMM model. The main advantage of these GMM-derived (GMMD) features is the possibility of performing adaptation of a DNN-HMM model through the adaptation of the auxiliary GMM-HMM. Experiments demonstrate that in supervised adaptation mode MAP adaptation is very effective for this scheme and gives, on average, approximately 35-36% of relative WER reduction for a 5 minute adaptation sample and 5% relative WER reduction for a 5 second adaptation sample, as compared to a SI-DNN model.

Then, Chapter 7 extended the proposed adaptation approach by applying the concept SAT to DNNs built on GMM-derived features. Traditional adaptation algorithms, such as MAP and fMLLR were performed for the auxiliary GMM model used in a SAT procedure for a DNN [Tomashenko and Khokhlov, 2015]. Experimental results on the WSJ0 corpus demonstrate that, in an unsupervised adaptation mode, the proposed adaptation technique can provide, approximately, a 17-20% relative WER reduction for MAP and 18-28% relative WER reduction for fMLLR adaptation on different adaptation sets, compared to the SI DNN-HMM system built on conventional 11×39MFCC features. It has been shown, that fMLLR adaptation for GMMD features is efficient when using a small amount of adaptation data, while MAP adaptation works better when more adaptation data are available. Another contribution of this chapter concerns a method of improving the adaptation algorithm by using confidence scores in MAP adaptation [START_REF] Tomashenko | Exploring GMM-derived features for unsupervised adaptation of deep neural network acoustic models[END_REF]. In addition, data augmentation and data selection strategies, were introduced for improving the regularization in MAP adaptation for DNN. These approaches can be especially useful when the training corpus is small, or when the amount of adaptation data is not known in advance and can vary.

Various ways of integration of adapted GMMD features into different state-of-the art neural network architectures DNN [Tomashenko et al., 2016a,b] and TDNN have been investigated in Chapter 8. Experimental results on the TED-LIUM corpus demonstrate that, in an unsupervised adaptation mode, the proposed adaptation and fusion techniques can provide approximately, a 12-18% relative WER reduction on different adaptation sets, compared to the SI DNN system built on conventional features, and a 4-6% relative WER reduction compared to the strong adapted baseline -SAT-DNN trained on fMLLR adapted features. For TDNN models, using the adapted GMMD features and fusion techniques leads to improvement of 15-28% relative WER reduction in comparison with SI model trained on conventional features and 8-15% relative WER reduction in comparison with SAT model trained with i-vectors. Hence, for both considered adaptation techniques, fMLLR and i-vectors, the proposed adaptation approach has appeared to be complementary and provide an additional improvement in recognition accuracy. In addition, we report the results of applying the proposed adaptation approach to the LIUM ASR system in the 2016 Arabic MGB Challenge, and demonstrate, that the complicated ASR system, trained with many hundred hours of speech data and comprised a combination of several AMs, can also benefit from the use of the adapted GMMD features [START_REF] Tomashenko | LIUM ASR systems for the 2016 multi-genre broadcast Arabic challenge[END_REF].

Chapter 9 has explored how the end-to-end ASR technology can benefit from speaker adaptation and has demonstrated that speaker adaptation has remained an essential mechanism for improving the performance of an ASR system in the new end-to-end speech recognition paradigm. Experimental results on the TED-LIUM corpus showed that in an unsupervised adaptation mode, the adaptation and data augmentation techniques can provide approximately a 10-20% relative WER reduction on different adaptation sets, compared to the SI BLSTM-CTC system built on filter-bank features. The best results, for BLSTM-CTC AMs, in average, were obtained using GMMD features and MAP adaptation, which can be further slightly improved by combination with fMLLR adaptation technique. We found out, that the type of the neural network AM architecture can differently influence the adaptation performance. The comparison with the TDNN-CE AMs showed that for these models, in contradiction to BLSTM-CTC AMs, MAP adaptation using GMMD features outperforms fMLLR only when it uses SAT model in the first decoding pass to obtain transcriptions for adaptation. Also the obtained results allow us to compare TDNN-CE and BLSTM-CTC AMs in the realistic conditions, when the speaker adaptation is applied, which is important because usually end-to-end and hybrid AMs are compared on incomplete unadapted systems. The best SI TDNN-CE AM outperforms the best SI BLSTM-CTC AM by 1-7% of relative WER reduction for different test sets. For the best SAT AMs this gap in WER for TDNN-CE and BLSTM-CTC AMs increases and reaches 5-13% of relative WER reduction.

Finally, in Chapter 10, we have looked from the various points of view at the proposed adaptation approach exploring the phoneme posterior based (PPB) features, generated from the decoding lattices and have demonstrated, that the advantage of using MAP-adapted GMMD features manifests itself at different levels of the decoding process. Visual tdistributed stochastic neighbor embedding (t-SNE) analysis applied to different phoneme groups, Davies-Bouldin (DB) index and analysis of other statistics from decoding lattices have shown a possible potential and direction for improvement of the proposed adaptation approach through more careful handling the quality of the phonetic transcripts, used in MAP adaptation.

The proposed adaptation approach was experimentally studied on four different speech corpora, corresponding to three languages (English (WSJ0 and TED-LIUM), Russian (STC) and Arabic (MGB-2016). The developed algorithms are integrated into the Kaldi environment. For TED-LIUM and WSJ0 corpora, we plan to prepare and release training recipes based on the proposed approach.

The most important advantage of the proposed approach is that it provides a general framework for adapting DNN acoustic models. In this thesis we used MAP and fMLLR adaptation algorithms as examples to study and demonstrate the effectiveness of the proposed framework, but any other adaptation algorithms can be used instead of them.

Perspectives

Different possible directions of future developments can be suggested for the work, presented in this thesis.

Application to other adaptation domains

The proposed adaptation framework opens new opportunities not only for the speaker adaptation task, which was the main focus of this thesis, but also to other adaptation domains, such as channel and noise adaptation, or other tasks. For example, the application of the idea, presented in works [START_REF] Khokhlov | [END_REF]Khokhlov, 2014b, 2015] for speaker adaptation, has later been successfully applied in [START_REF] Kundu | Incorporating a generative front-end layer to deep neural network for noise robust automatic speech recognition[END_REF] to noise adaptation of a DNN AM with the use of vector Taylor series (VTS) for an auxiliary GMM model.

Combination with model-based adaptation techniques

In this thesis it was demonstrated, that the proposed adaptation algorithm is complementary to the most commonly used feature-space adaptation techniques for DNNs, such as fMLLR and i-vectors. Also it is interesting to investigate the complementarity of the proposed algorithm to model-based adaptation techniques for DNNs, for example learning hidden unit contributions (LHUC), or others.

Speaker adaptive training for GMMD with transcripts from ASR

In the proposed SAT approach, at the training stage, adaptation is performed using original (correct) transcripts from the training corpus. However, at the test time, inaccurate transcripts from the ASR system are used for adaptation. Hence, speaker adaptive training and decoding are performed in different conditions. Probably, using transcriptions from the ASR system for adaptation in the SAT procedure can help to make adaptation more robust to overfitting during the training or to transcription errors during adaptation at the test time. In the proposed SAT scheme we assume that the targets and alignment for training are obtained using the correct transcripts.

Improving the quality of adaptation transcripts

The GMMD feature analysis in Chapter 10 shows a possible direction for improvement of the proposed adaptation approach through more careful handling the quality of the phonetic transcripts used in adaptation. Various approaches can be proposed for this purpose.

We assume that the fixed lexicon for the ASR system can not cover all pronunciations of words that can occur in speech. But we can not extend the lexicon enough with all additional pronunciation variants because too much pronunciations in lexicon degrades the performance of the ASR system. Hence one approach is to add possible pronunciations dynamically during decoding in a two-pass scheme.

GMMD features: high dimension and correlation problem

The dimension n of the GMMD features equals to the number of states in the auxiliary GMM. In our study we usually use a monophone GMM model with 3 states in each monophone.

Depending on the number of phonemes in the language, n can be roughly estimated as 100 < n < 200. Hence, the dimension of GMMD features is typically larger than the dimension of conventional features (fbanks, MFCC+∆+∆∆, BN, etc.) which often does not exceed 40. When we aim to train a DNN, a context extension is usually applied which may increase the dimension of the input features up to 30 times. This can lead to too high dimension of the input layer (up to 6000 units).

Using triphones in the auxiliary GMM model instead of monophones can be attractive because adaptation of a triphone GMM can provide more accurate adaptation when we have more adaptation data. But in this case the problem of high dimension becomes critical, because with triphone states the dimension of the input layer can increase in 10-30 times more in comparison with monophones.

Another aspect of this problem is correlation between GMMD feature components, which increases when the number of states in the auxiliary GMM increases.

Proposed solution

To solve this problem, standard feature dimensionality reduction and decorrelation techniques (such as PCA, LDA, HLDA, and others) can be applied.

An alternative promising solution is to use convolutional neural networks (CNNs, see Section 2.4.1) for this purpose and train CNNs directly on high-dimensional GMMD features, obtained after splicing with a wide context. Through convolutional and pooling operations, CNNs can solve the problem of a high dimension and correlation in a natural and easy way. Also, to train a CNN more effectively, it is possible to place GMMD features into 2-dimensional plane (to make a "picture" of triphone states) in such a way that acoustically close senones are locally close on this "picture".

In order to adapt GMMD features extracted using a triphone auxiliary GMM with many states, different approaches, such as vector field smoothing (MAP-VFS, Section 4.2.1.2), structural MAP (SMAP, Section 4.2.1.3), MLLR with regression classes (Section 4.2.2.1), maximum a posteriori linear regression (MAPLR) or other algorithms can be applied.

  observation feature vector,N 0 = D -feature dimension, f l (•) : R N l → R N lactivation function.

Figure 2 . 1

 21 Figure 2.1 Example of a deep neural network with an input layer, three hidden layers, and an output layer

Figure 2 . 2

 22 Figure 2.2 Example of the convolutional neural network for image processing. The figure was adapted with some modifications based on the idea from [LeCun et al., 1995].

Figure 2 . 3

 23 Figure 2.3 Example of feature expansion and subsampling in the time-delay neural network.The figure was adapted with some modifications based on the idea from[START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF].

Figure 2 .

 2 Figure 2.3 shows an example of layer-wise context expansion scheme for a TDNN model. Each frame of the higher level corresponds to a longer context than the lower layers. This hierarchical temporal context expansion is different from the using of a wide contextual

Figure 2 . 4 Figure 2 . 5

 2425 Figure 2.4 Jordan recurrent neural network

  vector of hidden state values, y t ∈ R L -vector of outputs, W hy ∈ R L×N -weight matrix connecting N hidden units to L outputs, W oh ∈ R N×D -weight matrix connecting D inputs to N hidden units, W hh ∈ R N×N -weight matrix connecting N hidden units from time t -1 to time t, W yh ∈ R N×L -weight matrix connecting the output layer at time t -1 to the hidden layer, b h -bias vector for hidden states, b y -bias vector for outputs, f (•) -hidden layer activation function, g(•) -output layer activation function.

. 1 .

 1 First, an input speech waveform is converted into a sequence of fixed-size acoustic feature vectors O = [o 1 , . . . , o T ] in a process called feature extraction. Then, the decoder tries to find the most likely sequence of words for the obtained feature vectors. The ASR task can be formulated as finding word sequence W * out of all possible word hypotheses, that maximizes the posterior probability of O: W * = arg max W P (W|O).

Figure 3 . 1

 31 Figure 3.1 Scheme of an ASR system and analysis of an example phrase "his most significant scientific publications"

  Figure 3.2 Example of the tandem speech recognition system

Figure 3 . 3

 33 Figure 3.3 Bottleneck (BN) features for GMM-HMM speech recognition system

Figure 3 .

 3 Figure 3.4 HMM-based phone model with five states and left-to-right topology. The first and the final states are non-emitting, other states have output distributions b j (.). Transition parameters a i j model the transition probabilities between hidden states i and j.

Figure 3 . 5

 35 Figure 3.5 Example of the hybrid acoustic model using a DNN to estimate posterior probabilities for HMM states

  objective function. An objective function F(Λ Λ Λ) = F(Λ Λ Λ; O) is optimized (minimized or maximized, depending on the training criterion) during the training. Here O = (O 1 , . . . , O R ) is the set of training sentences (acoustic sequences).

Figure 4 . 1

 41 Figure 4.1 Linear input network (LIN) adaptation

Figure 4 . 2

 42 Figure 4.2 Linear output network (LON) adaptation

Figure 4 . 3

 43 Figure 4.3 Linear hidden network (LHN) adaptation

Figure 4 . 4

 44 Figure 4.4 SVD-BN adaptation: f corresponds to a nonlinear activation function; h l , h l+1 are the layers of the NN between which a SI and SA BN layers are inserted.

Figure 4 . 5

 45 Figure 4.5 Factorized adaptation

Figure 4 . 6

 46 Figure 4.6 Factor extraction using BN features

Figure 4 . 7

 47 Figure 4.7 Multi-task learning (MTL) neural architecture for adaptation

Figure 4 . 8

 48 Figure 4.8 DNN architecture for adaptation with hierarchy output layer

Figure 4 .

 4 Figure 4.10 Using i-vectors for DNN adaptation: adaptation network

Figure 4 .

 4 Figure 4.11 Speaker codes for DNN adaptation

Figure 6 . 1

 61 Figure 6.1 Use of GMMD features for training DNN-HMM model

Figure 6 . 3

 63 Figure 6.3 Supervised adaptation performance depending on the amount of the adaptation data for the DNN trained on 11×157GMMD features for the STC corpus.

2 .

 2 Adapt Λ Λ Λ for the training speakers, so that for each speaker S in the training corpus a speaker-adapted GMM-HMM Λ Λ Λ(S) is created.

Figure 7 . 1

 71 Figure 7.1 SAT scheme for DNN training with GMMD features.

3 .

 3 For each speaker in the training corpus, transform all acoustic feature vectors o t of the training corpus, belonging to speaker S, into likelihood based feature vectors f t using speaker-adapted auxiliary AM Λ Λ Λ(S) and Formulas (6.3) and(6.4).

Figure 7 . 2

 72 Figure 7.2 Unsupervised adaptation performance on si_et_05 test depending on the size of the adaptation dataset. Relative WER reduction (∆ rel WER) is given with respect to the baseline SI DNN trained on 11×39MFCC.

Figure 7 . 3

 73 Figure 7.3 Data selection and data augmentation scheme for SAT. For each speaker S in the training corpus, training data of this speaker are divided into several parts (shown in gray). Then MAP adaptation is performed independently for these parts. At this stage, different values of parameter τ: {τ 1 , τ 2 , τ 3 } are applied during the GMMD feature extraction for each part. In the shown example, the size of the training corpus (in terms of the features) increases in 9 times (3(selection) × 3(augmentation)).

  Figure 7.4 Differences in GMMD-features depending on τ values. GMMD feature coordinates are sorted by descending order of the corresponding differences. Statistics are calculated on the training dataset of the WSJ0 corpus.

  Fusion for training and decoding stages.

  Fusion for training stage.

Figure 8 . 1

 81 Figure 8.1 Types of fusion

Figure 8 . 2

 82 Figure 8.2 Using speaker-adapted BN-based GMMD features for SAT of a DNN-HMM. Numbers correspond to feature or layer dimensions.

Figure 8 . 3

 83 Figure 8.3 Using speaker-adapted BN-based GMMD features for SAT of a TDNN-HMM. Numbers correspond to feature or layer dimensions.

  . Value α in Formula (8.1) (Section 8.1.2) is a weight of the baseline SAT-DNN model. Parameter α was optimized on the development set.

Figure 9 . 1 6 . 9 .

 9169 Figure 9.1 SAT for the BLSTM-CTC AM using GMMD features

Figure 9 . 2

 92 Figure 9.2 Per-speaker relative WER reduction (∆ rel WER) for speakers from test and development datasets of the TED-LIUM corpus for different adaptation algorithms with respect to the SI AMs, trained on BN features (#3). Adaptation is performed in an unsupervised mode. Results are ordered in ascending WER reduction order for each AM.

10 Figure 10 . 1

 10101 Figure 10.1 Scheme of phoneme posterior based (PPB) feature extraction

Figure 10 . 2

 102 Figure 10.2 Analysis of PPB features for vowels using t-SNE for TDNN models, trained on different basic features. Results are shown for the Development dataset of the TED-LIUM corpus.

  Figure 10.4 Analysis of PPB features for consonants-2 using t-SNE for TDNN models, trained on different basic features. Results are shown for the Development dataset of the TED-LIUM corpus.

Figure 10 . 5

 105 Figure 10.5 Adaptation analysis based on PPB feature statistics for the three TDNN models on the development data set of the TED-LIUM corpus: (a) CDF of the number of phoneme alternatives in the lattices for a certain frame; (b) CDF of the position of the correct phoneme in the list of all phoneme candidates (ordered by the posterior probability) presented in lattices for a certain frame; (c) Log-probability histogram of the correct phoneme (if it exists) in the lattice for a certain frame.

Figure 10 . 6

 106 Figure 10.6 Summary statistics for all speakers from the development and the two test datasets of the TED-LIUM corpus: (a) WERs(%) for two (SI and SAT) TDNN models: SI -TDNN BN , SAT -TDNN BN⊕ ⊕ ⊕GMMD . Relative WER reduction (%) is computed for the given WERs. Results are ordered by increasing WER values for the SI model; (b) Dependence of relative WER reduction (the same as in (a)) on average likelihood improvement, obtained by MAP adaptation of the auxiliary monophone model. The line corresponds to the linear regression model.

  8.6 Recognition results on the Dev dataset for different AMs and their fusion for MGB-2016 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 9.1 Summary of unsupervised adaptation results for CTC AMs on TED-LIUM corpus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 9.2 Summary of unsupervised adaptation results for TDNN AMs on TED-LIUM corpus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 9.3 Relative WER reduction for adapted BLSTM-CTC and TDNN AMs on TED-LIUM corpus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 10.1 Davies-Bouldin index for different types of features used in TDNN training 154 10.2 Statistics for PPB features, obtained using the corresponding TDNN models 154 Parameter that controls the balance between the maximum likelihood estimate of the mean and its prior value in maximum a posteriori adaptation [-T 1 ..T 2 ] where (T 1 , T 2 ≥ 0) -Sequence of T 1 + T 2 + 1 integer numbers in this interval: -T 1 , -T 1 + 1, . . . , T 2 -1, T 2 . Notation [-T 1 ..T 2 ] for feature vector o t means, that for this vector a new context vector o t is obtained through concatenation of neighboring vectors: o t = [o t-T 1 , ..., o t , ..., o t+T 2 ]. Usually, except for time-delay neural networks, we consider symmetrical intervals (T 1 = T 2 = T ).

	Nomenclature Notations ∆ First-order time derivatives of a feature vector ∆∆ Second-order time derivatives of a feature vector EM Expectation Maximization fbanks filter-bank (coefficients) FER Frame Error Rate GMMD GMM-Derived GMM Gaussian Mixture Model GPU Graphics Processing Unit HL Hidden Layer HMM Hidden Markov Model xviii MLLT Maximum Likelihood Linear Transform ML Maximum Likelihood MLP Multilayer Perceptron MTL Multi-Task Learning PCA Principal Component Analysis PLP Perceptual Linear Prediction PPB Phoneme Posterior Based RBM Restricted Boltzmann Machine ∆ xvii Nomenclature fMLLR feature-space Maximum Likelihood Linear Regression MMI Maximum Mutual Information Chapter 1
	KLD Kullback-Leibler Divergence ReLU Rectified Linear Unit
	LDA Linear Discriminant Analysis RNN Recurrent Neural Network
	LHN Linear Hidden Network SAT Speaker Adaptive Training
	LHUC Learning Hidden Unit Contributions SD Speaker Dependent
	LIN SGD Stochastic Gradient Descent Linear Input Network
	LM SGMM Subspace Gaussian Mixture Model Language Model
	SI	LON Linear Output Network Speaker Independent
	LSTM Long Short Term Memory SMAP Structural Maximum a Posteriori
	LVCSR Large Vocabulary Continuous Speech Recognition sMBR state-level Minimum Bayes Risk
	MAP Maximum A Posteriori t-SNE t-Distributed Stochastic Neighbor Embedding
	MBR Minimum Bayes Risk TDNN Time-Delay Neural Network
	MFCC Mel-Frequency Cepstral Coefficients UBM Universal Background Model
	MGB Multi-Genre Broadcast VFS Vector Field Smoothing
	MLLR Maximum Likelihood Linear Regression VTLN Vocal Tract Length Normalization

abs WER Absolute Word Error Rate Reduction ∆ rel WER Relative Word Error Rate Reduction. F(Λ Λ Λ) Objective function to be optimized over parameters Λ Λ Λ h l Activation vector (typically hidden) at l-th layer in a neural network Λ Λ Λ Set of model parameters N (.;µ µ µ,Σ Σ Σ) Multivariative Gaussian distribution with mean vector µ µ µ and covariance matrix Σ Σ Σ o t Speech observation vector at time t o t Context vector composed of several neighboring speech observation vectors: [o t-T 1 , . . . , o t+T 2 ]. Usually, except for time-delay neural networks, we consider symmetrical intervals (T 1 = T 2 = T ). τ

  State output (emission) probability distributions {b j (.)}, denoted by B = {b j (.)}, where b j (.) is a probability distribution of state j, and b j (o t ) denotes the likelihood of state j generating observation vector o t at time t:

	States
	Acoustic
	feature
	vectors
	b j (o t ) = P(o t |s t = j)	(3.9)

Table 5 .

 5 1 Datasets statistics for the TED-LUM corpus

	Characteristic	Dataset Training Development Test 1	Test 2
	Duration, hours	Total Male Female	171.66 120.50 51.15	3.49 1.76 1.73	3.49 1.76 1.73	4.90 3.51 1.39
	Duration	Mean	10.0	15.0	15.0	21.0
	per speaker, minutes	Minimum Maximum	5.0 18.3	14.4 15.4	14.4 15.4	18.3 24.9
	Number of speakers	Total Male Female	1 029 710 319	14 7 7	14 7 7	14 10 4
	Number of words	Total	-	36 672	35 555 51 452

Table 5 .

 5 2 Statistics for MGB-2016 training and development datasets. The first line of the table corresponds to the training, provided by the MGB organizers, and the last line corresponds to the subset of the training dataset selected for AM training. Since for the test data set the original text data was not provided by the MGB-2 organizers, and the evaluation process on the test dataset was performed by them, we do not report statistics about this dataset here.

	Dataset	Shows Duration, hours Utterances	Words
	Training (original)	2 214	1 128.0	376 011 7 815 566
	Dev	16	8.5	4 940	57 647
	Training (selected for AM training)	-	648.3	398 438 4 422 123
	the development set equals to 132.

Table 5 .

 5 3 Summary statistics for all datasets

	Arabic MGB-2016	Arabic	Multi-genre	broadcasts			648	14 249		Dev.		3.8	132	57 647	303K
	STC	Russian		Reading			27	203 (111+92)		Adapt. Test		5 11.5	20 (10+10)	-19K	-11K
	TED-LIUM	English	Microphone	Lectures			172	1029 (710+319)		Dev. Test 1 Test 2		15 15 21	14 (7+7) 14 (7+7) 14 (10+4))	36 672 35 555 51 452	150K and 152K
	WSJ	English		Reading			15 (13+2)	83 (42+41)		si_et_05 si_et_20		3.5 3.5	8 (5+3)	5 353 5 645	5K 20K
	Corpus	Language	Channel	Type	Training	Duration (speech + pause),	hours	Speakers (male + female)	Test	Adaptaion / Dev./ Test sets	Mean duration per speaker,	minutes	Speakers (male + female)	Words	Vocabulary, words

Table 6 .

 6 [START_REF] Tomashenko | On the use of Gaussian mixture model framework to improve speaker adaptation of deep neural network acoustic models[END_REF] Baseline WER results for SI-DNN models with 2 and 4 hidden layers for three types of features on the STC corpus. WERs for baseline systems are given with confidence intervals, corresponding to 5% level of significance.

		Number of
	Features	hidden layers WER, %
	11×13MFCC	2 4	39.7 ±0.70 38.0 ±0.69
	11×157GMMD	2 4	40.8 40.8
	11×55GMMD	2 4	42.0 41.6

Table 6 .

 6 2 Supervised adaptation performance for different DNN-HMMs for the STC corpus. Adaptation is performed using all the available adaptation data (5 minutes per each speaker). Absolute and relative WER reduction (∆ abs WER and ∆ rel WER) for adapted AMs is calculated with respect to the corresponding SI AMs.

	Features	Number of hidden layers	WER, % for adapted AM ∆ abs WER, % ∆ rel WER, %
	11×157GMMD	2 4	26.4 26.9	14.3 13.9	35.2 34.1
	11×55GMMD	2 4	26.8 27.1	15.2 14.5	36.2 34.8
	6.4.2.2 Adaptation performance depending on the amount of the adaptation data

Table 6 .

 6 3 Feature and posterior level fusion results on the STC corpus for 11×13MFCC and 11×55GMMD features. Adaptation is performed in a supervised mode using all the available adaptation data (5 minutes per each speaker).

	Fusion type	Number of hidden layers	WER, % for SI AM	WER, % for adapted AM
	Features	2 4	39.8 39.4	26.8 26.3
	Posteriors	2 4	38.1 37.2	27.1 26.6
	6.4.3.2 Posterior-level fusion

Table 7 .

 7 1 WER results for unsupervised adaptation on WSJ0 evaluation sets: si_et_20 and si_et_05. ∆ rel WER is a relative WER reduction calculated with respect to the baseline SI model. WER for the baseline system is given with confidence intervals, corresponding to 5% level of significance.

	Features	Adaptation (SAT)	si_et_20 WER, % ∆ rel WER, % WER, % si_et_05 ∆ rel WER, %
	11×39MFCC	SI fMLLR	9.55 ±0.77 baseline 8.03 16.0	3.23 ±0.47 baseline 2.76 14.5
		SI	9.28	2.8	3.19	1.2
	GMMD	MAP fMLLR	7.60 7.85	20.4 17.8	2.69 2.52	16.8 22.0
		fMLLR+MAP 7.83	18.0	2.78	13.9

Table 7 .

 7 2 Performance for unsupervised speaker adaptation on the WSJ0 corpus for speakers from si_et_20 evaluation set

	Speaker id WER, % for SI, 11×39MFCC	∆ rel WER, % 11×39MFCC fMLLR GMMD fMLLR	GMMD MAP
	440	8.49	24.1	22.4	6.9
	441	16.02	24.3	26.2	36.4
	442	10.59	13.2	15.8	14.5
	443	9.94	11.3	14.1	9.9
	444	10.87	11.8	18.4	26.3
	445	7.25	13.0	20.4	18.5
	446	7.13	8.5	17.0	12.8
	447	6.59	16.0	0.0	26.0
	All	9.55	16.0	17.8	20.4
	7.1.2.4 Adaptation performance depending on the size of the adaptation dataset

Table 7 .

 7 3 Performance for unsupervised speaker adaptation on the WSJ0 corpus for speakers from si_et_05 evaluation set

	Speaker id WER, % for SI, 11×39MFCC	∆ rel WER, % 11×39MFCC fMLLR GMMD fMLLR	GMMD MAP
	440	4.15	29.6	40.7	40.7
	441	4.71	43.3	56.7	16.7
	442	3.32	0	8.3	12.5
	443	1.52	-10.0	0	-20
	444	2.43	33.3	22.2	22.2
	445	3.16	0.0	5.3	15.8
	446	1.89	-15.2	-7.6	23
	447	4.87	6.3	15.6	6.3
	All	3.23	14.5	22	16.8

Table 8 .

 8 1 Impact of the parameters of the auxiliary model topology and τ (adaptation parameter in MAP, see Formula (4.2)) on GMMD feature extraction (on the development set of the TED-LIUM corpus)

	Features Number of Gaussians τ	Power WER,%
		2500	5	0.5	13.89
	MFCC	2500 3800	5 5	0.0 0.5	14.05 13.75
		3800	5	0.0	13.65
		2500	1	0.5	13.69
	BN	2500 2500	3 5	0.5 0.5	13.51 13.34
		2500	7	0.5	13.33
		2500	10	0.5	13.40
		2500	5	0.0	13.34
		3800	5	0.5	13.33
		3800	5	0.0	13.48
		10200	5	0.5	13.92
	8.6.2 Results for DNN models
	We empirically studied different types of fusion described in Section 8.1 and applied them
	to DNN models trained using GMMD-features extracted as proposed in Section 8.5.1. The
	performance results in terms of WER for SI and SAT DNN-HMM models are presented in

Table 8 .

 8 2. The first four lines of the table correspond to the baseline SI (#1, #2) and SAT (#3, #4) DNNs, which were trained as described in Section 8.4.2. Lines #4-#8 correspond to feature level fusion of conventional and GMMD features. For comparison purpose with lattice-level fusion (which we consider further in this chapter) we report WER of the consensus hypothesis 4 in parentheses for experiments #4 and #8 -these models will be used in the fusion. Parameter τ in MAP adaptation for both acoustic model training and decoding was set equal to 5.

Table 8 .

 8 2 Summary of unsupervised adaptation results for DNN models on the TED-LIUM corpus. The results in parentheses correspond to WER of the consensus hypothesis. WERs for baseline systems are given with confidence intervals, corresponding to 5% level of significance.

	# Model Features	DNN training criterion	WER,% Development Test 1	Test 2
	1 SI 2	BN	CE sMBR	13.16 ±0.35 12.14 ±0.33	11.94 ±0.34 15.43 ±0.31 10.77 ±0.32 13.75 ±0.30
	3 SAT 4	BN-fMLLR	CE sMBR	11.72 10.64 (10.57) 9.52 (9.46) 12.78 (12.67) 10.88 14.21
	5 SI 6	GMMD⊕BN	CE sMBR	12.92 11.80	11.62 10.47	15.19 13.52
	7 SAT 8	GMMD-MAP⊕BN	CE sMBR	10.46 10.26 (10.23) 9.40 (9.31) 12.52 (12.46) 9.74 13.03

Table 8 .

 8 3 Summary of the fusion results for DNN models on the TED-LIUM corpus. The results in parentheses correspond to WER of the consensus hypothesis. Relative WER reduction (∆ rel WER,%) is calculated for consensus hypothesis with respect to AM trained on BN-fMLLR (#4 in Table 8.2). The bold figures in the table indicate the best performance improvement.

	#	Fusion: #4 and #8	Development WER ∆ rel WER WER ∆ rel WER WER ∆ rel WER Test 1 Test 2
	9	Posterior fusion, α = 0.45	9.98 (9.91)	6.2	9.15 (9.06)	4.3	12.11 (12.04)	5.0
	10	Lattice fusion, α = 0.46	10.06	4.8	9.09	4.0	12.12	4.4

Table 8 .

 8 The bold figures in the table indicate the best performance. WERs for baseline systems are given with confidence intervals, corresponding to 5% level of significance. 5 Summary of the fusion results (CNC) for TDNN models on TED-LIUM corpus.Relative WER reduction (∆ rel WER,%) is calculated for consensus hypothesis with respect to the baseline TDNN MFCC⊕ ⊕ ⊕i-vectors , α is a weight of TDNN 1 in the fusion. Numeration of AMs corresponds to numeration in Table 8.4. The bold figures in the table indicate the best performance.

	#	Model Features	WER,% Development Test 1	Test 2
	1	SI	MFCC	13.69 ±0.35	11.34 ±0.33 14.38 ±0.30
	2	SAT	MFCC⊕i-vectors	11.63 (11.56) 9.62 (9.51) 13.28 (13.19)
	3	SI	BN	12.32 ±0.34	10.48 ±0.32 14.00 ±0.30
	4	SAT	BN⊕i-vectors	11.62 (11.45) 9.75 (9.69) 13.30 (13.23)
	5	SAT	BN-fMLLR	10.70 (10.60) 9.28 (9.25) 12.84 (12.83)
	6.a		GMMD #3 ⊕MFCC	11.30	9.75	13.74
	6.b	SAT	GMMD #4 ⊕MFCC	10.91	9.47	13.45
	6.c		GMMD #5 ⊕MFCC	10.39	9.31	13.27
	7.a		GMMD #3 ⊕MFCC⊕i-vectors 11.23	9.91	13.69
	7.b	SAT	GMMD #4 ⊕MFCC⊕i-vectors 10.90	9.45	13.49
	7.c		GMMD #5 ⊕MFCC⊕i-vectors 10.31 (10.29) 9.32 (9.34) 13.26 (13.20)
	8.a		GMMD #3 ⊕BN	11.07	9.75	13.55
	8.b	SAT	GMMD #4 ⊕BN	10.83	9.36	13.13
	8.c		GMMD #5 ⊕BN	10.29	9.20	13.04
	9.a		GMMD #3 ⊕BN⊕i-vectors	11.01	9.73	13.59
	9.b	SAT	GMMD #4 ⊕BN⊕i-vectors	10.82	9.32	13.20
	9.c		GMMD #5 ⊕BN⊕i-vectors	10.30 (10.22) 9.11 (9.09) 13.09 (13.02)
	10.a		GMMD #3 ⊕BN-fMLLR	10.92	9.54	13.27
	10.b	SAT	GMMD #4 ⊕BN-fMLLR	10.70	9.17	13.01
	10.c		GMMD #5 ⊕BN-fMLLR	10.15 (10.10) 9.06 (9.03) 12.84 (12.82)

Table 8 .

 8 6 Recognition results on the Dev dataset for different AMs (systems: #1-#5) and their fusion (systems: #6-#31) for MGB-2016. The second column (#) represents the system identification numbers. Numbers in the columns 3-7 are weights of the recognition results (lattices) from corresponding AMs in CN combination. The last column represents 1-best result from consensus hypotheses.

	Number of AMs in fusion	#	Fusion weights TDNN 1 TDNN 2 DNN BN DNN GMMD TDNN GMMD Lattice Cons. hyp. WER, %
		1	-	1	-	-	-	23.66	23.25
		2	-	-	1	-	-	22.67	22.56
	1	3	-	-	-	-	1	22.31	22.11
		4	-	-	-	1	-	22.05	21.89
		5	1	-	-	-	-	21.69	21.37
		6	-	-	0.525	0.475	-		21.02
		7	-	0.529	0.471	-	-		20.65
		8 0.620	-	-	0.380	-		20.52
		9	-	-	0.461	-	0.539		20.50
	2		--	0.550 -	--	0.450 0.468	-0.532		20.47 20.46
			0.598	-	0.402	-	-		20.42
			-	0.501	-	-	0.499		20.38
			0.562	0.438	-	-	-		20.36
			0.567	-	-	-	0.433		20.01
			0.432	-	0.298	0.271	-		20.11
			-	-	0.341	0.293	0.365		20.08
			-	0.423	0.264	0.313	-		19.99
			0.303	0.435	-	0.262	-		19.87
	3		0.369 -	0.332 0.357	0.298 0.340	--	-0.304		19.86 19.86
			-	0.383	-	0.249	0.368		19.81
			0.475	-	0.187	-	0.338		19.76
			0.379	-	-	0.218	0.403		19.75
			0.350	0.358	-	-	0.292		19.71
			-	0.314	0.200	0.252	0.235		19.70
			0.307	0.326	0.183	0.183	-		19.68
	4		0.252	0.270	0.207	-	0.271		19.64
			0.342	-	0.226	0.188	0.244		19.64
			0.302	0.259	-	0.193	0.246		19.49
	5		0.297	0.179	0.130	0.183	0.211		19.48

Table 9 .

 9 1 Summary of unsupervised adaptation results for CTC AMs on TED-LIUM corpus. Indices for GMMD features denote the AM, which was used to obtain transcripts for adaptation.

	# Model Features	WER,% Dev. Test 1 Test 2
	1	SI	fbanks ⊕ ∆ ⊕ ∆∆	14.57 11.71 15.29
	2 3	SI	MFCC BN	13.21 11.16 14.15 13.63 11.84 15.06
	4		MFCC ⊕ i-vectors	12.92 10.45 14.09
	5		BN ⊕ i-vectors	13.47 11.37 14.31
	6 7	SAT	BN-fMLLR MFCC ⊕ GMMD #2	12.45 10.96 13.79 11.95 10.20 14.04
	8		BN ⊕ GMMD #2	11.66 10.14 13.88
	9		BN-fMLLR ⊕ GMMD #2 11.63 9.91 13.58
	10 SAT 11	BN ⊕ GMMD #6 BN-fMLLR ⊕ GMMD #6 11.41 9.93 13.47 11.67 10.11 13.70

Table 9 .

 9 2 Summary of unsupervised adaptation results for TDNN AMs on TED-LIUM corpus

	# Model Features	WER,% Dev. Test 1 Test 2
	2 3	SI	MFCC BN	13.69 11.34 14.38 12.32 10.48 14.00
	4		MFCC ⊕ i-vectors	11.63 9.62 13.28
	5		BN ⊕ i-vectors	11.62 9.75 13.30
	6 7	SAT	BN-fMLLR MFCC ⊕ GMMD #2	10.70 9.28 12.84 11.30 9.75 13.74
	8		BN ⊕ GMMD #2	11.07 9.75 13.55
	9		BN-fMLLR ⊕ GMMD #2 10.92 9.54 13.27
	10 SAT 11	BN ⊕ GMMD #6 BN-fMLLR ⊕ GMMD #6 10.15 9.06 12.84 10.29 9.20 13.04

Table 9 .

 9 3 Relative WER reduction (∆ rel WER) for adapted BLSTM-CTC and TDNN AMs in comparison with the best SI AMs for each AM type (#2 for CTC and #3 for TDNN). ∆ rel WER values are calculated based on the results from Tables 9.1 and 9.2.

	# Features	CTC: ∆ rel WER,% Dev. Test 1 Test 2 Dev. Test 1 Test 2 TDNN: ∆ rel WER,%
	4 MFCC ⊕ i-vectors	2.20	6.36 0.42	5.60	8.21 5.14
	5 BN ⊕ i-vectors	-1.97 -1.88 -1.13 5.68	6.97 5.00
	6 BN-fMLLR	5.75	1.79 2.54 13.15 11.45 8.29
	7 MFCC ⊕ GMMD #2	9.54	8.60 0.78	8.28	6.97 1.86
	8 BN ⊕ GMMD #2	11.73 9.14 1.91 10.15 6.97 3.21
	9 BN-fMLLR ⊕ GMMD #2 11.96 11.20 4.03 11.36 8.97 5.21
	10 BN ⊕ GMMD #6	11.66 9.41 3.18 16.48 12.21 6.86
	11 BN-fMLLR ⊕ GMMD #6	13.63 11.02 4.81 17.61 13.55 8.29

Table 10 .

 10 1 Davies-Bouldin (DB) index for different types of features used in TDNN training. The DB index is calculated on PPB features produced by the corresponding model. Results are provided for the development data set of the TED-LIUM corpus.

	Features	State 0 State 1 State 2
	MFCC	1.67	1.52	1.71
	BN ⊕ i-vectors	1.53	1.36	1.41
	BN ⊕ i-vectors ⊕ GMMD	1.39	1.26	1.27

Table 10 .

 10 2 Statistics for PPB features, obtained using the corresponding TDNN models. All statistics in the table are calculated only for speech frames (excluding silence). The average log-probability of the correct phoneme is given with the standard deviation. Results are provided for the development data set of the TED-LIUM corpus.

	Features	FER Oracle FER Aver. correct log-prob.
	MFCC	5.18	0.72	-0.17 ± 0.83
	BN ⊕ i-vectors	4.11	0.75	-0.11 ± 0.64
	BN ⊕ i-vectors ⊕ GMMD 3.64	1.23	-0.08 ± 0.52

The Multi-Genre Broadcast (MGB) challenge: http://www.mgb-challenge.org/

Residual networks (ResNets) are the type of NNs, where a residual function of the input is learned using skip connections. ResNet framework was designed[He et al.,

2016] to train very deep neural networks, which are easy to optimize and which avoid poor optimization or generalization problems.[START_REF] Tomashenko | A New Perspective on Combining GMM and DNN Frameworks for Speaker Adaptation[END_REF] The VGG (Visual Geometry Group) network architecture was introduced in[START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] for ImageNet Challenge 2014 submission.

Universal background model (UBM) is a GMM model trained on all speech classes pooled together[Povey et al.,

2010]. The UBM is a popular approach to alternative hypothesis modeling in speaker recognition domain, where a GMM-UBM represents a large mixture of Gaussians that cover all speech, and in the context of speaker recognition, it is trained using the EM algorithm on data from multiple speakers, and then adapted to each speaker using maximum a posteriori (MAP) estimation[START_REF] Bimbot | A tutorial on text-independent speaker verification[END_REF][START_REF] Reynolds | Speaker verification using adapted gaussian mixture models[END_REF].

Decoders typically work in the log-domain.

http://www1.icsi.berkeley.edu/Speech/docs/sctk-1.2/sclite.htm

In ASR, confidence measures are used in different tasks to evaluate reliability of recognition results. An extensive survey on confidence measures for ASR can be found in[Jiang,

2005].

More strictly, i-vector is estimated as the mean of the distribution of random variable w(s).

ARPAbet phoneme set is used: https://en.wikipedia.org/wiki/Arpabet

These 8 speakers are the same for si_et_05, si_et_20 and si_et_ad datasets

Technology, Entertainment, Design: http://www.ted.com

http://cantabresearch.com/cantab-TEDLIUM.tar.bz2

https://github.com/kaldi-asr/kaldi/tree/master/egs/tedlium/s5; date of access: January, 2016

https://github.com/kaldi-asr/kaldi/tree/master/egs/tedlium/s5_r2; date of access: March, 2017

http://speechpro.com/

http://www.aljazeera.net

Notation [-T..T ] for feature vector o t means, that for this vector a new context vector o t is obtained through concatenation of neighboring vectors: o t = [o t-T , ..., o t , ..., o t+T ]

Pitch-features are calculated using the Kaldi toolkit[Povey et al.,

2011b] and consist of the following values[START_REF] Ghahremani | A pitch extraction algorithm tuned for automatic speech recognition[END_REF]: (1) probability of voicing (POV-feature), (2) pitch-feature and (3) delta-pitchfeature. For details see http://kaldi-asr.org/doc/process-kaldi-pitch-feats_8cc.html.

using "nnet1" and "nnet3" Kaldi setups: http://kaldi-asr.org/doc/dnn.html

For notations see Section 2.4.2 and Figure 2.3

Consensus hypothesis is obtained by finding the path through the confusion graph with the highest combined link weight[START_REF] Mangu | Finding consensus in speech recognition: word error minimization and other applications of confusion networks[END_REF]

We already used the lattice scores obtained from arc posterior probabilities to improve MAP adaptation in Section 7 (Formula (7.2)), but here we again provide the description as a reminder.

The notations are given according to the ARPAbet phoneme set: https://en.wikipedia.org/wiki/Arpabet

The Multi-Genre Broadcast (MGB) Challenge: http://www.mgb-challenge.org/

Open Keyword Search (OpenKWS) Evaluation: https://www.nist.gov/itl/iad/mig/openkws16-evaluation

Acknowledgements

This thesis was done in collaboration between two universities -University of Le Mans in France and ITMO University in Saint-Petersburg -and I thank all the people from both universities who organized this collaboration and made this double-degree program possible.

I would like to express my deep gratitude to my supervisor from the LIUM (Laboratoire d'gave me experience in the field and inspired me to complete my PhD in speech recognition. I particularly thank the Research and Development

The proposed adaptation scheme for a DNN model is shown in Figure 6.2. First, adaptation of an auxiliary SI GMM-HMM model is performed and a new speaker-adapted (SA) GMM-HMM model is created. Second, at the recognition stage, GMMD features are calculated using this SA GMM-HMM. The proposed approach can be considered as a feature space transformation technique with respect to DNN-HMMs trained on GMMD features.

Experiments #9-10 present combination of two different adaptation types: MAP and fMLLR. Is is interesting to note that in all experiments optimal value of α is close to 0.5, so all types of models are equally important for fusion. We can see that MAP adaptation on GMMD features can be complementary to fMLLR adaptation on conventional BN features.

Results for TDNN models

Summary of the adaptation results for the TDNN models is given in Table 8.4. The first five lines of the table correspond to the baseline AMs: SI (#1, #3) and SAT (#2, #4, #5). The rest of the table (lines #6-#10) shows the results for different AMs, trained using GMMD features. Since the performance of MAP adaptation depends on the quality of transcripts, used for adaptation, for these models, for comparison purposes, we present results, obtained with the use of transcripts from three different AMs, used in the first decoding pass (#3, #4, #5). These results have the corresponding letters in numeration (a, b, c).

For SI AM trained on BN features, using GMMD adapted features provides 3.2-10.1% of relative WER reduction (#3 vs #8.a). If we compare results in lines #4 and #9.b, we can see that the use of GMMD features gives an additional improvement over i-vector based adaptation (0.8-6.9% of relative WER reduction for different sets). The best result over all AMs was obtained by TDNN GMMD⊕ ⊕ ⊕BN-fMLLR (line #10.c).

To further investigate the complementary of the different adaptation techniques, we performed CNC of recognition results for different TDNN models (Table 8.5). The best results (#24 or #25) are obtained by combinations of TDNN models (#2, #7) or (#2, #10) and provide approximately 8-15% of relative WER reduction in comparison with the conventional state-of-the-art SAT baseline model TDNN MFCC⊕i-vectors , and 5-7% of relative WER reduction in comparison with the strongest baseline AM we could achieve without GMMD features (#5).

Baseline systems

We used the open-source Kaldi toolkit [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF]] and the Eesen system [Miao et al., 2015a] for the experiments presented in this chapter. Three baseline SI AMs were trained using the Eesen system in a similar manner, and differ only in the front-end processing. The following three type of features were used: The first type of features is the same as proposed in the original Eesen recipe for the TED-LIUM corpus. For the AMs with the two other types of features, also the two types of data augmentation strategies were applied for the speech training data: speed perturbation (with factors 0.9, 1.0, 1.1) and volume perturbation, as in [START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF]. The first baseline AM was trained as described in [START_REF] Miao | Towards speaker adaptive training of deep neural network acoustic models[END_REF] with the CTC criterion and the deep BLSTM architecture. The BLSTM network contains five bidirectional LSTM layers with 320 memory cells in each forward and backward sub-layer. The input features were normalized with per-speaker mean subtraction and variance normalization. The output layer is a 41-dimensional softmax layer with the units corresponding to 39 context-independent phones, 1 noise model and 1 blank symbol.

The third SI AM was trained on BN features [START_REF] Grézl | Probabilistic and bottle-neck features for LVCSR of meetings[END_REF]. A DNN model for extraction 40-dimensional BN features was trained with the following topology: one 440dimensional input layer; four hidden layers (HLs), where the third HL was a BN layer with 40 neurons and other three HLs were 1500-dimensional; the output layer was 4052-dimensional. The input features for training this BN extractor were 440-dimensional (40 × 11): 40dimensional high-resolution MFCCs spliced across 11 neighboring frames (±5).

Adapted models

The three types of AM adaptation were empirically explored in this section: fMLLR, adaptation using i-vectors, and MAP adaptation using GMMD features. For all the adapted AMs the same data augmentation strategies were applied during the training, as for the SI ones. All the SAT models were trained with the same neural network topology (except for the input

Publications International conferences and journals

This thesis mainly contains the work published in the following papers: