Imran Sheikh
email: imran.sheikh@loria.fr

Irina Illina
email: irina.illina@loria.fr

Dominique Fohr
email: dominique.fohr@loria.fr

Georges Linarès
email: georges.linares@univ-avignon.fr

Improved Neural Bag-of-Words Model to Retrieve Out-of-Vocabulary Words in Speech Recognition

Keywords: lvcsr, oov, proper names

Many Proper Names (PNs) are Out-Of-Vocabulary (OOV) words for speech recognition systems used to process diachronic audio data. To enable recovery of the PNs missed by the system, relevant OOV PNs can be retrieved by exploiting the semantic context of the spoken content. In this paper, we explore the Neural Bag-of-Words (NBOW) model, proposed previously for text classification, to retrieve relevant OOV PNs. We propose a Neural Bag-of-Weighted-Words (NBOW2) model in which the input embedding layer is augmented with a context anchor layer. This layer learns to assign importance to input words and has the ability to capture (task specific) key-words in a NBOW model. With experiments on French broadcast news videos we show that the NBOW and NBOW2 models outperform earlier methods based on raw embeddings from LDA and Skip-gram. Combining NBOW with NBOW2 gives faster convergence during training.

Introduction

The diachronic nature of news content causes frequent variations in the linguistic content and vocabulary, leading to Out-Of-Vocabulary (OOV) words problem for Large Vocabulary Continuous Speech Recognition (LVCSR). Simply appending the LVCSR vocabulary and updating the Language Model (LM) will (a) require a good amount of training data and/or (b) affect the LVCSR performance and complexity. An analysis of the OOV words reveals that majority of OOV words (56-72% [START_REF] Sheikh | OOV proper name retrieval using topic and lexical context models[END_REF]) are Proper Names (PNs). However PNs are important for obtaining accurate automatic transcriptions as well as automatic indexing of audio-video content. In this paper, we focus on the problem of retrieval of OOV PNs relevant to an audio document.

To retrieve OOV PNs relevant to an audio document we rely on their semantic context. During the training phase, diachronic text news with new (i.e., OOV) PNs are collected from the internet. This set of text documents, referred as a diachronic text corpus, is used to learn a context vector space which captures relationships between the In-Vocabulary (IV) words, PNs and the OOV PNs. During the test, the LVCSR hypothesis of the audio document is projected into the context space and then relevant OOV PNs are inferred. In our previous work [START_REF] Sheikh | OOV proper name retrieval using topic and lexical context models[END_REF] we have shown that methods based on Latent Dirichlet Allocation (LDA) topic space can perform well for retrieval of the target OOV PNs 1 . 1 For a given audio documents several OOV PNs can be relevant. The ones actually present in the audio are referred as target OOV PNs.

Alternative methods to learn word and context representations [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Pennington | Glove: Global vectors for word representation[END_REF], based on predicting the context in which words appear, have become popular. These representations have been shown to perform effectively in a range of applications and tasks [START_REF] Baroni | Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors[END_REF].

In this paper, we present an improved Neural Bag-of-Words (NBOW) model [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF]. The simple, yet impressive, NBOW model takes an average of the word vectors in the input sequence and performs classification with a fully connected layer. We propose to replace the average with a weighted sum, where the weights applied to each word (vector) are learned during the training of the model. We refer to this as the Neural Bag-of-Weighted-Words (NBOW2) model. With experiments on French broadcast news videos, we show that (a) the proposed NBOW2 model learns meaningful word importance weights, (b) the NBOW2 model, like the NBOW model, outperforms the baseline methods based on raw embeddings from LDA and Skip-gram, (c) NBOW2+ model, which combines the context vectors from the NBOW and NBOW2 models, gives faster convergence during training (d) the improved retrieval performance translates to an improvement in the recovery of the target OOV PNs.

Related Work

The task of retrieval of OOV and PNs relevant to an audio document has been presented in previous works. These include methods based on probabilistic topic models applicable to common PNs [START_REF] Bigot | Person name recognition in ASR outputs using continuous context models[END_REF][START_REF] Senay | Person name spotting by combining acoustic matching and LDA topic models[END_REF] and those addressing even the less frequent PNs [START_REF] Sheikh | OOV proper name retrieval using topic and lexical context models[END_REF][START_REF] Sheikh | Study of entity-topic models for OOV proper name retrieval[END_REF]. Word embedding based methods to retrieve relevant PNs have been tried for audio documents with multiple news events [START_REF] Fohr | Continuous word representation using neural networks for proper name retrieval from diachronic documents[END_REF]. More recently [START_REF] Sheikh | Document level semantic context for retrieving OOV proper names[END_REF] document similarity based methods have been shown to perform better, especially for retrieval of less frequent PNs. Compared to these works we explore neural networks trained to retrieve relevant OOV PNs, for audio documents with a single or coherent event.

Our methodology in this paper is related to the recent approaches of text classification with neural networks. In this context, fully connected feed forward networks [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF][START_REF] Nam | Large-scale multi-label text classification -revisiting neural networks[END_REF], Convolutional Neural Networks (CNN) [START_REF] Kim | Convolutional neural networks for sentence classification[END_REF][START_REF] Johnson | Effective use of word order for text categorization with convolutional neural networks[END_REF][START_REF] Wang | Semantic clustering and convolutional neural network for short text categorization[END_REF] and also Recurrent Neural Networks (RNN) [START_REF] Socher | Recursive deep models for semantic compositionality over a sentiment treebank[END_REF][START_REF] Hermann | The Role of Syntax in Vector Space Models of Compositional Semantics[END_REF][START_REF] Dong | Adaptive recursive neural network for target-dependent twitter sentiment classification[END_REF][START_REF] Tai | Improved semantic representations from tree-structured long short-term memory networks[END_REF][START_REF] Dai | Semi-supervised sequence learning[END_REF] have been applied. On one hand, the approaches based on CNN and RNN capture rich compositional information, and have been outperforming the state-of-the-art results in text classification; on the other hand they are computationally intensive and require careful hyper-parameter selection and/or regularisation [START_REF] Zhang | A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification[END_REF][START_REF] Dai | Semi-supervised sequence learning[END_REF]. For our task we rely on document level bag-of-words architectures mainly because they are suitable to process LVCSR transcriptions of audio documents, which are firstly prone to noise in word sequence due to word errors and secondly have no direct information about position of OOVs. Moreover, in contrast to the tasks in most state-of-the-art works in text classification, our task has a large number of output classes (OOV PNs) and the distribution of documents per OOV PN is very skewed [START_REF] Sheikh | Study of entity-topic models for OOV proper name retrieval[END_REF].

We found that the work of Ling [START_REF] Ling | Not all contexts are created equal: Better word representations with variable attention[END_REF] is related to our proposal of using different weights for words. However, they use word position based weights to improve vectors learned by the Continuous Bag-Of-Words (CBOW) [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] model. Our NBOW2 model learns a context anchor vector to assign task specific word importance weights. The NBOW2 model is a variation of our D-CBOW2 model reported earlier [23]2,3 . We will highlight the differences between these models and show that the NBOW2 model gives a better retrieval performance. However, an explicit comparison of the two is not in scope of this paper.

Neural Bag-of-Words Model

The Neural Bag-of-Words (NBOW) model [START_REF] Kalchbrenner | A convolutional neural network for modelling sentences[END_REF][START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF] is a fully connected neural network which maps input text X, a sequence of words, to one of k output labels. The input to this model is in BOW form where the index of input words are set to 1 or number of occurrences of that word. The first hidden layer has d dimensional word vectors for each word in the chosen task vocabulary. Given the word vectors vw for the words w ∈ X, the output of this layer is an average of the input word vectors: z = 1/ |X| w∈X vw. The average vector z is then fed to a fully connected layer to estimate probabilities for output labels as ŷ = sof tmax(W l z + b), where W l is k × d matrix, b is a bias vector and sof tmax(q) = exp(q)/ k j=1 exp(qj). The NBOW model is trained to minimise the categorical crossentropy loss [START_REF] Goldberg | A primer on neural network models for natural language processing[END_REF]. In our task to retrieve OOV PNs relevant to an audio document, the NBOW model is trained using a diachronic text corpus from the internet. During training IV words and PNs in a text document are given at the input and the co-occurring OOV PNs in the document are set at the output. During test, the LVCSR word hypothesis of the audio document is given at the input and the softmax probabilities at the output are used as scores to rank and retrieve the OOV PNs.

Proposed Neural Bag-of-Weighted-Words Model

While the NBOW model learns word vectors specialised for the task, we feel that it fails to explicitly use, as well as provide, the information that certain words are more important than the others for the given task. We thus propose the NBOW2 model, with the motivation to enable the NBOW model to learn and use task specific word importance weights. As compared to the NBOW model, our proposed NBOW2 model is a weighted sum composition of the input word sequence (X), calculated as:

z = 1 |X| w∈X αw vw (1)
αw are the scalar word importance weights for each word w ∈ X, obtained by introducing a global context anchor vector a as:

αw = f (vw • a) (2)
where (•) represents a dot product and f scales the importance weights to [0, 1]. We believe that the vector a, which is itself learned and updated along with the word vectors, will act as a reference for the separation and the composition of the word vectors into a context vector; hence the term anchor. For function f , common activation functions sigmoid, softmax (as in [START_REF] Sheikh | Learning to retrieve out-of-vocabulary words in speech recognition[END_REF]) and even hyperbolic tangent can be used. From our experiments we found that the sigmoid function f (x) = (1+e -x ) -1 is a better choice in terms of model convergence and accuracy. The NBOW2 model will thus include an additional anchor vector, with the training and testing both using Equations 1 and 2.

Experiments and Results

Baseline Methods

The first baseline is the LDA based method, (Method I) from our previous work [START_REF] Sheikh | OOV proper name retrieval using topic and lexical context models[END_REF], briefly described below. We also examine an extension of this method to Skip-gram word vectors4 .

• LDA Topic Space Representations LDA topic model is trained on the diachronic text corpus. A topic vocabulary, the number of topics (T ) and Dirichlet priors (α, β) are first chosen. Topic model parameters θ and φ are then estimated using Gibbs sampling algorithm [START_REF] Griffiths | Finding scientific topics[END_REF]. During test, the latent topic mixture p(t|h) of the LVCSR word hypothesis (h) is inferred. Then the likelihood of an OOV PN (ṽi) in the diachronic corpus is calculated, using p(ṽi|t) from φ, as: p(ṽi|h) = T t=1 p( ṽi|t) p(t|h). To retrieve relevant OOV PNs, p(ṽi|h) is calculated for each OOV PN ṽi and used as a score to rank the OOV PNs.

• Raw Skip-gram Vectors Skip-gram word vectors are trained for the words in the diachronic corpus. Given the word vectors and their linearity property, we obtain a representation for a test document by taking an average of vector of words in the document. This representation is referred to as AverageVec.

The K dimensional vector representation of the LVCSR hypothesis (h) is compared with the vector (ṽi) for each of the OOV PNs to calculate a score si = CosSim(h, ṽi), where CosSim(•, •) is the cosine similarity. Score si is used to rank the OOV PNs ṽi. The NBOW model can be seen as the Av-erageVec setup trained in a supervised manner but it is clear that this supervision is free of labelling costs.

Experiment Corpus

Table 1 presents the main characteristics of the French language diachronic broadcast news datasets which will be used as the train, validation and test sets in our study. Detailed description of the datasets is available in our previous works [START_REF] Sheikh | How diachronic text corpora affect context based retrieval of oov proper names for audio news[END_REF][START_REF] Sheikh | Learning to retrieve out-of-vocabulary words in speech recognition[END_REF]. The L'Express dataset will be used as a diachronic corpus to train context/topic models, in order to infer the OOV PNs relevant to Euronews videos. Non video text articles from Euronews are used as a validation/development set. The words and PNs which occur in the lexicon of our Automatic News Transcription System (ANTS) [START_REF] Illina | The Automatic News Transcription System: ANTS some Real Time experiments[END_REF] are tagged as IV and remaining PNs are tagged as OOV. The total number of OOV PNs to be retrieved in the test set, obtained by counting unique OOV PNs per video, is 4694. Out of 4694, up to 2010 (42%) OOV PNs can be retrieved with the L'Express diachronic train set. The target OOV PN coverage can be further increased by augmenting additional text datasets, as discussed in another work [START_REF] Sheikh | How diachronic text corpora affect context based retrieval of oov proper names for audio news[END_REF].

Experiment Setup

The ANTS [START_REF] Illina | The Automatic News Transcription System: ANTS some Real Time experiments[END_REF] LVCSR system is used to perform automatic segmentation and speech-to-text transcription of the test set.

The automatic transcriptions of the test audio news obtained by ANTS have an average Word Error Rate (WER) of 40% as compared to the reference transcriptions available from Euronews.

The train, dev and test texts are pre-processed as in our previous works [START_REF] Sheikh | How diachronic text corpora affect context based retrieval of oov proper names for audio news[END_REF][START_REF] Sheikh | Learning to retrieve out-of-vocabulary words in speech recognition[END_REF]. For comparison, the number of LDA topics and the dimensionality of the different neural context models are chosen to be equal and set to 400. A window size of 15 is chosen for training Skip-gram vectors. This selection of model hyper-parameters is based on performance on the validation set.

Our baseline methods (Section 3.1) are denoted as LDA and AverageVec. Proposed models (Section 2) are denoted as NBOW and NBOW2. Additionally we propose a combination of the NBOW and NBOW2 models, denoted as NBOW2+, in which the NBOW and NBOW2 averaged context vectors are concatenated together, both during training and test.

Training the NBOW group of models

For the NBOW group of models, we first train Skip-gram word vectors on the diachronic text corpus and use them to initialise the input layer vectors. This gives better performance than random initialisation. Then the NBOW, NBOW2 and NBOW2+ models are trained in 2 phases. In the first training phase, only the output parameters (W l , b), and the anchor vector (a) for the NBOW2 and NBOW2+ models, are trained, keeping the input word vectors fixed. After this first training phase, all the model parameters are trained and updated in the second training phase. We found that a model trained in two phases in such a manner gives much better retrieval performance compared to the same model in which all the parameters are trained in one go.

To control the training of all the NBOW models an early stopping criterion [START_REF] Bengio | Practical recommendations for gradient-based training of deep architectures[END_REF] based on the validation set error is used. Early stopping is applied in both the first and the second training phases. Further we applied a dropout at the input layer (word dropout) [START_REF] Dai | Semi-supervised sequence learning[END_REF][START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF]. With experiments on validation set we chose a word dropout probability of 0.9 (from among 0, 0.25, 0.5, 0.75 and 0.9) 5 . All NBOW models were trained with gradient descent algorithm with ADADELTA [START_REF] Zeiler | ADADELTA: an adaptive learning rate method[END_REF].

Figure 1 shows a graph of validation set errors, of the NBOW, NBOW2 and NBOW2+ models, as the training progresses. It must be noted that this error is like a classification error stating whether an OOV PN in the validation set document is given the highest output probability or not. It will help to analyse the learning of the models. For instance it can be observed that the NBOW2+ model gives a faster convergence without compromise in error rate. More detailed discussion on this and the retrieval performance will follow in next sections. 5 A word dropout probability p does not necessarily translate to leaving out p% of the input words.

Discussion

OOV PN Retrieval Performance

Figure 2 shows the Recall and Mean Average Precision (MAP) [START_REF] Manning | Introduction to Information Retrieval[END_REF] performance of retrieval of OOV PNs for all the methods. The graphs shown are for the reference transcriptions (left) and the LVCSR transcriptions (right) of the Euronews test set audio. The X-axis represents the number of OOV PNs selected from the diachronic corpus i.e. the 'N' in the top-N retrieved results. The Y-axis represents recall (top) and MAP (bottom) of the target OOV PNs. We can observe that our previous method based on LDA [START_REF] Sheikh | OOV proper name retrieval using topic and lexical context models[END_REF] performs better than AverageVec, which follows a similar methodology but with Skip-gram word vectors. The recall and MAP retrieval performance for NBOW, NBOW2 and NBOW2+ models is very similar and their graphs are overlapping. But as mentioned earlier, combining the NBOW and NBOW2 representations in the NBOW2+ model gives a faster convergence in training. Overall the three models clearly outperform the baseline methods in terms of recall and MAP, both for reference and LVCSR transcriptions. (Similarly they outperformed the document similarity method [START_REF] Sheikh | Document level semantic context for retrieving OOV proper names[END_REF] which improves performance for less frequent PNs, at the cost of searching the diachronic corpus. The best of these methods achieved a maximum MAP of 0.519 for reference and 0.462 for LVCSR.)

Scrutinising the training of NBOW model

We try to examine how much the choice of training hyperparameters affects the NBOW model training. Table 2 depicts the effect of applying word dropout. The comparison is in terms of maximum MAP achieved for the LVCSR hypothesis. For space constraints we show only for 100 and 400 word vector dimension NBOW models. Firstly, it is clear that word dropout improves the MAP performance significantly and secondly we cannot attribute the increase in number of training epochs to increase in word dropout. Next we increased the ADADELTA decay constant (ρ) from 0.99, as in our experiments, to 0.95. The training takes fewer epochs but the MAP performance also reduces. For instance with word dropout probability p = 0.9, the 100 and 400 dimensional models take 185 and 351 epochs respectively and achieve a maximum MAP of 0.45 and 0.5 respectively. These MAP values are significantly lower that those with ρ = 0.99. Further we also trained the (400) NBOW and NBOW2+ models with fixed number of epochs (100) in the first and (50) second training phases. The NBOW2+ model still achieved a maximum MAP (0.553) comparable to NBOW (0.556), the difference being insignificant. And these MAP values are significantly lower than that (0.568) obtained from training with early stopping. From these experiments, we can conclude that to obtain better retrieval performance with the NBOW model we need a longer training, which can be reduced by the NBOW2+ model as depicted in Figure 1. We present Figure 3 to discuss (a) the scalar word importance weights αw (b) the choice of the function f , for the NBOW2 model (see Equation 2). Considering a sample test document, the left graph of Figure 3 shows the weights assigned by the NBOW2 model with f as sigmoid activation and the right graph shows the weights assigned by f as softmax activation. Firstly, it is clear from these graphs that the NBOW2 models have learned that different words have different degree of importance. For example, for this document with the missing OOV PN kehm (https://de.wikipedia.org/wiki/Sabine Kehm), as per the left graph the top four important words are michael, formule, critique and hospitaliser and the four least important words are rester, tenir, monde and présent. Secondly, the NBOW2 model with f as softmax tends to assign higher weights to fewer words and weights close to zero to the remaining words. This could be one reason for its relatively bad retrieval performance [START_REF] Sheikh | Learning to retrieve out-of-vocabulary words in speech recognition[END_REF].

Recovery of OOV PNs with Keyword Search (KWS)

We perform an evaluation of the retrieved list of OOV PNs using an automatic Keyword Search (KWS) method [START_REF] Chen | Quantifying the value of pronunciation lexicons for keyword search in low resource languages[END_REF] which enables searching of OOV words in an LVCSR lattice. First, a list of relevant OOV PNs is retrieved with the models presented in this paper. In the second step, KWS is performed on the entire LVCSR lattice of the audio file, for each OOV PN in the list of relevant OOV PNs. Evaluation is done in terms of F1-score.

Figure 4 shows the F1-scores obtained with the top-N lists denotes the all equal weights (1/48 = 0.0208) in the simple average by the NBOW model. from NBOW2+ and LDA. We show only these two since the NBOW models have similar performance and LDA is shown to outperform AverageVec. The KWS algorithm has a matching score threshold which controls the operating characteristics, and hence recall/precision and F1-score, of the search. We show the best F1-scores corresponding to top-N OOV PN lists of different sizes (N). Beyond top-512 there is no significant difference in the F1-scores. Overall we can observe that the better the Recall and MAP of OOV PN retrieval, the better is the F-1 score6 .

Conclusion and Future Work

We examined the NBOW model for our task of retrieval of OOV PNs relevant to an audio document. We proposed a novel extension to the NBOW model, which enables it to learn the words important for the given task. With experiments on French broadcast news videos we showed that (a) the NBOW and NBOW2 models give improvements in retrieval performance as compared to the previous method (b) combining the NBOW and NBOW2 into a new model leads to a faster convergence in training. The improvements in retrieval were validated by performing recovery of the target OOV PNs with an automatic keyword search. These results motivate us to extend NBOW2 to Deep Averaging Networks (DAN) [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF], which cascade additional fully connected layers to the NBOW model. Further, instead of using a single anchor vector to obtain word importance weights, we would like to explore models with multiple anchors.

Acknowledgements

This work is funded by the ContNomina project supported by the French National Research Agency (ANR) under contract ANR-12-BS02-0009. KWS experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria, CNRS, RENATER and other Universities and organisations (https://www.grid5000.fr).

Figure 1 :

 1 Figure 1: Validation set errors during training of NBOW, NBOW2 and NBOW2+ models. ( markers indicate end of first training phase and begin of second training phase)

Figure 2 :

 2 Figure 2: OOV PN retrieval performance for Euronews audio test set. is LDA, AverageVec, NBOW, NBOW2, NBOW2+. (NBOW, NBOW2 and NBOW2+ give similar performance and their graphs are overlapping.)

Figure 3 :

 3 Figure 3: Word importance weights assigned by the NBOW2 model ( ) in a sample document with 48 words. Two variations of the NBOW2 model are shown: (left) f as sigmoid and (right) f as softmax.denotes the all equal weights (1/48 = 0.0208) in the simple average by the NBOW model.

Figure 4 :

 4 Figure 4: F1 scores for recovery of target OOV PNs.

Table 1 :

 1 Diachronic (French) news datasets used for experiments

		L'Express Euronews Euronews
		(train)	(valid)	(test)
	Type	Text	Text	Video
	Time Period		Jan -Jun 2014
	OOV PN unigrams	9.3K	3.4K	3.1K
	OOV PN Documents	26.5K	1.9K	1.9K
	Total OOV PN count	107K	6.9K	6.2K

Table 2 :

 2 The maximum MAP obtained by NBOW models of 100 and 400 dimension word vectors in given number of epochs (2 training phases). * denotes difference in MAP is statistically insignificant[START_REF] Smucker | A comparison of statistical significance tests for information retrieval evaluation[END_REF] compared to MAP with p = 0.0.

			word dropout probability (p)
		0.0	0.25	0.5	0.75	0.9
	100	MAP 0.500 0.497 * 0.514 0.536 0.540 epoch 459 403 415 458 568
	400	MAP 0.525 0.519 * 0.533 0.561 0.568 epoch 481 482 398 417 410
	4.3. Word Importance weights of NBOW2 model

This work is neither published nor under review and available as a technical report on the arXiv pre-print server.

We choose the name NBOW2 over D-CBOW2 since the name NBOW was used in a recently published work[START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF].

Word vectors from Skip-gram model give a better performance in our task than the word vectors from the CBOW model[START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF].

document similarity method mentioned at end of Section 4.1 achieved a maximum F-1 score of 0.342.


Imran Sheikh

Irina Illina

Dominique Fohr
email: dominique.fohr@loria.frgeorges.linares@univ-avignon.fr

Georges Linarès

Learning Word Importance with the Neural Bag-of-Words Model

The Neural Bag-of-Words (NBOW) model performs classification with an average of the input word vectors and achieves an impressive performance. While the NBOW model learns word vectors targeted for the classification task it does not explicitly model which words are important for given task. In this paper we propose an improved NBOW model with this ability to learn task specific word importance weights. The word importance weights are learned by introducing a new weighted sum composition of the word vectors. With experiments on standard topic and sentiment classification tasks, we show that (a) our proposed model learns meaningful word importance for a given task (b) our model gives best accuracies among the BOW approaches. We also show that the learned word importance weights are comparable to tf-idf based word weights when used as features in a BOW SVM classifier.

Introduction

A Bag-of-Words BOW represents text (a sentence, paragraph or a document) as a vector of word features. Traditional BOW methods have used word occurrence frequency and variants of TermFrequency-InverseDocumentFrequency (tf-idf) as the word feature [START_REF] Christopher | Introduction to Information Retrieval, chapter Scoring, term weighting, and the vector space model[END_REF]. Development in neural network and deep learning based language processing has led to the development of more powerful continuous vector representation of words [START_REF] Bengio | A neural probabilistic language model[END_REF][START_REF] Collobert | A unified architecture for natural language processing: Deep neural networks with multitask learning[END_REF][START_REF] Turian | Word representations: A simple and general method for semi-supervised learning[END_REF]Mikolov et al., 2013b). It was shown that these predictive word vector representations capture syntactic and/or semantic characteristics of words and their surrounding context (Mikolov et al., 2013a;[START_REF] Pennington | Glove: Global vectors for word representation[END_REF], and that they outperform the count based word (vector) representations [START_REF] Baroni | Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors[END_REF].

Many approaches in text classification are now driven by models built on word vectors. Earlier works proposed models which learned word vectors targeted for the classification task [START_REF] Maas | Learning word vectors for sentiment analysis[END_REF]. In more recent works, text classification is performed with compositional representations learned with neural networks or by training the network specifically for text classification [START_REF] Goldberg | A primer on neural network models for natural language processing[END_REF]. One such network is the Neural Bag-of-Words (NBOW) model [START_REF] Kalchbrenner | A convolutional neural network for modelling sentences[END_REF][START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF]. The NBOW model takes an average of the word vectors in the input text and performs classification with a logistic regression layer. Essentially the NBOW model is a fully connected feed forward network with BOW input. The averaging operation can be attributed to the absence of non-linearity at the hidden layer and the BOW inputs where words are set to 1 (or number of occurrences of that word) and 0. While the NBOW model learns word vectors targeted for the classification task, it does not explicitly model which words are important for given task. In this paper, we propose an improved NBOW model which learns these task specific word importance weights. We replace the average with a weighted sum, where the weights applied to each word (vector) are learned during the training of the model. With experiments on sentiment and topic classification tasks, we will show that our proposed model learns meaningful word importance weights and it can perform better than the NBOW model.

The rest of the paper is organised as follows. First in Section 2 we discuss about the related works. In Section 3 we briefly introduce the NBOW model and present our proposed model, termed as the Neural Bag-of-Weighted-Words (NBOW2) model, in Section 4. In Section 5 we give details about the experiment setup and the tasks used in our experiments. Section 6 presents a discussion on the word importance weights learned and the classification performance achieved by the proposed NBOW2 model, followed by the conclusion in Section 7.

Related Work

A variety of neural network architectures have been proposed for different language processing tasks. In context of text classification, fully connected feed forward networks (Le and [START_REF] Quoc | Distributed representations of sentences and documents[END_REF][START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF][START_REF] Nam | Largescale multi-label text classification -revisiting neural networks[END_REF], Convolutional Neural Networks (CNN) [START_REF] Kim | Convolutional neural networks for sentence classification[END_REF][START_REF] Johnson | Effective use of word order for text categorization with convolutional neural networks[END_REF][START_REF] Ling | Not all contexts are created equal: Better word representations with variable attention[END_REF] and also Recurrent/Recursive Neural Networks (RNN) [START_REF] Socher | Recursive deep models for semantic compositionality over a sentiment treebank[END_REF][START_REF] Moritz | The Role of Syntax in Vector Space Models of Compositional Semantics[END_REF][START_REF] Dong | Adaptive recursive neural network for target-dependent twitter sentiment classification[END_REF][START_REF] Sheng | Improved semantic representations from tree-structured long short-term memory networks[END_REF][START_REF] Andrew | Semisupervised sequence learning[END_REF] have been used. On one hand, the approaches based on CNN and RNN capture rich compositional information, and have been outperforming the state-of-the-art results, on the other hand they are computationally intensive and may require careful hyper-parameter selection and/or regularisation [START_REF] Zhang | A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification[END_REF][START_REF] Andrew | Semisupervised sequence learning[END_REF]. We focus our study to the NBOW model which gives an impressive performance in text classification, not far below the state-of-theart CNN and RNN systems. We propose an improved NBOW model which learns these task specific word importance weights.

Word weights based on variants of word occurrence frequency or tf-idf have been commonly used and studied in literature [START_REF] Christopher | Introduction to Information Retrieval, chapter Scoring, term weighting, and the vector space model[END_REF][START_REF] Paltoglou | A study of information retrieval weighting schemes for sentiment analysis[END_REF][START_REF] Quan | Term weighting schemes for question categorization[END_REF]. Supervised weighting schemes for adjusting tf-idf for text classification have been proposed earlier [START_REF] Kim | Credibility adjusted term frequency: A supervised term weighting scheme for sentiment analysis and text classification[END_REF]Deng et al., 2014;[START_REF] Mammadov | Proceedings, chapter A New Supervised Term Ranking Method for Text Categorization[END_REF][START_REF] Lan | Proposing a new term weighting scheme for text categorization[END_REF]. Use of a small number of important words against all the words in the text was studied by [START_REF] Islam | Proceedings, chapter Do Important Words in Bag-of-Words Model of Text Relatedness Help?[END_REF] for the task of text relatedness using Latent Semantic Analysis and Google Trigram Model. Our work is in line with recent approaches of text processing with neural networks and learns word importance weights along with the word vectors.

We found that the works by [START_REF] Ling | Not all contexts are created equal: Better word representations with variable attention[END_REF] and [START_REF] Li | Feature weight tuning for recursive neural networks[END_REF] are most related to our proposed method. [START_REF] Ling | Not all contexts are created equal: Better word representations with variable attention[END_REF] proposes position based weights to improve word vectors learned by the Continuous Bag-Of-Words (CBOW) model (Mikolov et al., 2013a). [START_REF] Li | Feature weight tuning for recursive neural networks[END_REF] proposes Weighted Neural Network (WNN) for training RNNs which learn compositional representation of text with a parse tree. The WNN weighs how much one specific node contributes to the higher-level representation. Also related are works on learning to pay attention in a sequence of input, as applied in text [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] as well as speech [START_REF] Chan | Listen, attend and spell[END_REF], image [START_REF] Xu | Show, attend and tell: Neural image caption generation with visual attention[END_REF] and protein sequence analysis [START_REF] Søren | Proceedings, chapter Convolutional LSTM Networks for Subcellular Localization of Proteins[END_REF]. The model with convolutional-pooling structures presented by [START_REF] Gao | Modeling interestingness with deep neural networks[END_REF] is also shown to capture keywords in text.

Neural Bag-of-Words (NBOW) model

The Neural Bag-of-Words (NBOW) model [START_REF] Kalchbrenner | A convolutional neural network for modelling sentences[END_REF][START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF] is a fully connected network which maps text X, a sequence of words, to one of k output labels. The NBOW model has d dimensional word vectors for each word in the chosen task vocabulary. For the words w ∈ X, corresponding word vectors v w are looked up and a hidden vector representation z is obtained as an average of the input word vectors

z = 1 |X| w∈X v w (1)
The average vector z is then fed to a fully connected layer to estimate probabilities for the output labels as:

ŷ = sof tmax(W l z + b) (2)
where W l is k × d matrix, b is a bias vector and sof tmax(q) = exp(q)/ k j=1 exp(q j ). For text classification tasks, the NBOW model is trained to minimise the categorical cross-entropy loss (Goldberg, 2015) using a stochastic gradient descent algorithm. Additional fully connected layers can be added into the NBOW model to form Deep Averaging Networks (DAN) [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF].

Proposed model: Neural

Bag-of-Weighted-Words (NBOW2)

While the NBOW model learns word vectors specialised for the classification task, it lacks to explicitly model and provide the information that certain words are more important than the others in the given classification task. While tf-idf weights capture word importance weights over a given corpus and can be used at the input of the NBOW model, we are interested in letting the model learn the word importance weights which are task specific. We thus propose the NBOW2 model, with the motivation to enable the NBOW model to provide task specific word importance weights.

It is easy to realise that the NBOW model is essentially a fully connected feed forward network with BOW input vector. The averaging operation can be attributed to the absence of non-linearity at the hidden layer and the BOW inputs where words are set to 1 and 0. To learn the word importance weights, we form a weighted sum composition of the text X as follows:

z = 1 |X| w∈X α w v w (3)
where α w are the scalar word importance weights for each word w ∈ X. Learning task specific word vectors with Equation 3 ensures that words which drive the classification task are given higher importance or α w values (c.f. Figure 1). α w are obtained by introducing a vector a in the model and are calculated as follows:

α w = f (v w • a) (4)
where v w • a represents a dot product between input word vector v w and vector a, and f scales the importance weights to range [0, 1]. Equation 4, which makes the scalar word importance weight α w a function of the distance of the word w from a in the context space, ensures that word importance are based on contextual word similarities and not biased by frequency of occurrence in the training corpus. We refer to the vector a as the anchor vector because each word vector is compared with this vector when calculating the word importance weight.

For f common activation functions including softmax, sigmoid and also hyperbolic tangent can be used. From our experiments we found that the sigmoid function f (t) = (1 + e -t ) -1 is a better choice in terms of model convergence and accuracy. However, it must be noted the softmax f could be more interesting in certain tasks because (a) the importance of a word (α w ) in an input document will be dependent not only on the distance of this word from anchor vector a but also on that of the other words in the given input document (b) being a max function the softamax f will bias the composition of input document context vector z (in Equation 3) to only a handful of input words.

To summarise the as compared to the NBOW model, the NBOW2 model will include one additional anchor vector. This anchor vector is randomly initialised before training and learned along with the word vectors and other model parameters. The model training, with stochastic gradient descent, and classification (with a forward pass) both will use Equations 3 and 4, along with the output class probability estimates ŷ = sof tmax(W l z + b) similar to the NBOW model.

Experiment Setup 1

To analyse the working and performance of our proposed NBOW2 model, we consider two common tasks: (a) binary sentiment classification on IMDB [START_REF] Maas | Learning word vectors for sentiment analysis[END_REF] and Rotten Tomatoes movie review dataset [START_REF] Pang | Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales[END_REF]) (b) topic classification of 20 Newsgroup dataset.

Sentiment Analysis

For the IMDB task we use the original dataset2 with 25000 train and 25000 test movie reviews. For Rotten Tomatoes (RT) we obtained the v1.0 dataset3 and we do 10-fold cross-validation over the balanced binary dataset of 10,662 sentences. In both IMDB and RT tasks, model training parameters4 for NBOW2 are kept similar to those chosen for NBOW by [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF] after cross validation. For NBOW and NBOW2 models '-RAND' suffix will denote random word vector initialisation and no suffix is initialisation with publicly available 300-d GloVe vectors trained over the Common Crawl [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] 5 .

20 Newsgroup Topic Classification

We use the 'bydate' train/test splits, cleaned and made available by Cardoso (2007) 6 . There are 11,293 documents in the original training set and 7,528 in the test set. For training the NBOW and NBOW2 models, we randomly extract 15% of the original train set as the validation set and use remaining 85% as the final training set. Training was performed with the ADADELTA (Zeiler, 2012) gradient descent algorithm. L2 regularisation weight of 1e-5 was applied to all parameters. Further, to add robustness, we applied 75% 7 word dropout [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF][START_REF] Andrew | Semisupervised sequence learning[END_REF]. Additionally we use early stopping when the validation error starts to increase. Similar to the sentiment analysis experiments '-RAND' suffix will denote random word vector initialisation and no suffix is initialisation with 300-d GloVe.

6 Analysis and Discussion

Word importance weights learned by the NBOW2 model

We perform an analysis of the word importance weights learned by the NBOW2 model by presenting some qualitative and quantitative results.

Visualisation of word vectors from the RT sentiment analysis task

We visually examine the word vectors learned by the NBOW and NBOW2 models. To visualise word vectors they can be projected into a two dimensional space using the t-Distributed Stochastic Neighbour Embedding (t-SNE) technique (van der [START_REF] Van Der Maaten | Visualizing high-dimensional data using t-sne[END_REF]. Figure 1 shows the two dimensional t-SNE visualisations of word vectors learned by the NBOW and NBOW2 models. 1b. For instance the words a, on, it, for, there are not so important8 for the RT sentiment classification task and are present together in region of lowest word importance. The words staid, inflated, softens can contribute to (negative) polarity of the reviews and hence have relatively higher importance weights (and are present together near the negative sentiment region in the word vector space).

To further verify our claim that, in comparison to the NBOW model, the NBOW2 model is able to distinguish words based on their importance we show Figure 1c. Figure 1c shows the word vectors learned by the NBOW model (same as in Figure 1a) but it depicts each word with (a colour based on) word importance weight learned by the NBOW2 model. It can be seen in Figure 1c that the NBOW model does not separate/group words based on word importance, even if we restrict only to the example words a, on, it, for, there.

Word importance weights v/s Tf-Idf weights as classification features

In this analysis, we compare the word importance weights learned by the NBOW2 model with tfidf weights and other word weight features proposed in the previous works. For this comparison, an SVM classifier is used for the IMDB and RT binary classification tasks. Each train/test document is represented as a sparse BOW feature vector in which each word feature is only the word weight. For NBOW2 model it is the scalar word importance weight learned by the model. We compare it with (a) classical tfidf weights (b) credibility adjusted tf-idf (cred-tfidf) weights proposed by Kim (2014) (c) binary cosine-normalised weights (bnc) and binary deltasmoothed-idf cosine-normalised (b∆'c) weights used by Maas (2011) (d) the Naive-Bayes SVM (NBSVM) method proposed by [START_REF] Wang | Baselines and bigrams: Simple, good sentiment and topic classification[END_REF]. Tfidf, bnc and b∆'c word weights are task independent word weights but cred-tf-idf and NBSVM are built based on the class/task information. It should be noted that some of these methods/features have been the earlier state-of-the-art results for IMDB and RT tasks.

The classification accuracies obtained by the SVM classifiers are reported in Table 1. The tf- Features for SVM Classifier IMDB RT bnc [START_REF] Maas | Learning word vectors for sentiment analysis[END_REF] 87.8 -b∆'c [START_REF] Maas | Learning word vectors for sentiment analysis[END_REF] 88.2 tf-idf-uni [START_REF] Kim | Credibility adjusted term frequency: A supervised term weighting scheme for sentiment analysis and text classification[END_REF] 88.6 77.1 cred-tf-idf-uni [START_REF] Kim | Credibility adjusted term frequency: A supervised term weighting scheme for sentiment analysis and text classification[END_REF] 88.8 77.5 NBSVM-uni [START_REF] Wang | Baselines and bigrams: Simple, good sentiment and topic classification[END_REF] 88 idf, cred-tf-id and NBSVM methods are denoted with a '-uni' suffix in Table 1 following the notation used by [START_REF] Kim | Convolutional neural networks for sentence classification[END_REF] . For the SVM classifier on 25k full length documents of IMDB task, the NBOW2 model weights are as good as NBSVM and b∆'c and better than bnc. But they do not perform as good as tf-idf weights. Whereas for the RT task with 1066 test sentences, the NBOW2 model word weights achieve accuracy closer to tfidf variants.

NBOW2 model Classification Performance

After the discussion on the word importance weights learnt by the NBOW2 model we compare the classification results obtained with our NBOW2 model. We compare the NBOW2 model classification accuracy to that obtained from the NBOW model [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF],

Model

IMDB RT NBOW-RAND [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF] 88.9 76.2 NBOW [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF] 89 [START_REF] Dauphin | Stochastic ratio matching of rbms for sparse high-dimensional inputs[END_REF] 79.5 SVM + BoW [START_REF] Cardoso-Cachopo | Improving Methods for Single-label Text Categorization[END_REF] 82.8 SA-LSTM [START_REF] Andrew | Semisupervised sequence learning[END_REF] 84.4 LM-LSTM [START_REF] Andrew | Semisupervised sequence learning[END_REF] 84.7 gives best accuracy among the BOW approaches.

For IMDB and newsgroup task, the accuracy of NBOW2 model is closer to that of NBOW (not statistically significant for the 20 Newsgroup). It is also evident that for RT and newsgroup classification, the performance of NBOW2 is not far from and LSTM methods. For further analysis we also trained the NBOW2 model by simply using fixed tf-idf weights in Equation 3. This gave 87.6% and 79.4% accuracy for IMDB and RT task. Thus we can state that the word importance weights of the NBOW2 model are themselves informative.

Conclusion and Future Work

We proposed a novel extension to the NBOW model, which enables the model to learn task specific word importance. With experiments and analysis on sentiment and topic classification tasks, we showed that our proposed NBOW2 model learns meaningful word importance weights. We showed that the NBOW2 model gives the best accuracy among the BOW approaches and it can outperform the NBOW model. This motivates us to explore extensions to the model, including (a) class-specific anchor vectors to obtain class-specific word importance (b) document context specific word importance.

  Figure 1ashows a plot of the word vectors learned by the NBOW model and Figure1bshows a plot of the word vectors learned by the NBOW model. Additionally in Figure1beach word is given a colour based on the word importance assigned to it by the NBOW2 model.From Figure1awe can see that NBOW model tries to separate the words in the word vector space. According to the word examples labelled in Figure1athe words appear to be grouped into regions corresponding to positive and negative sentiments of the RT movie review task. Similarly the NBOW2 model also learns to separate the words into regions of positive and negative sentiments as shown, by the same word examples, in Figure1b. If we examine the word importance assigned by the NBOW2 model, indicated by colours in Figure 1b, it is evident that the NBOW2 model also learns to separate words based on their importance 7 choice based on accuracy on validation set weights. To support this statement we show additional word examples labelled in different regions in Figure

Figure 1 :

 1 Figure 1: Visualisation of word vectors learned by the NBOW and NBOW2 models in the RT task. Word vectors are reduced to 2 dimension using t-SNE technique and shown in each plot. Plot (a) represents word vectors from the NBOW model, (b) represents words from NBOW2 model, with colours indicating the word importance weights learned by the NBOW2 model, (c) represents word vectors from the NBOW model as in (a) but depicts each word with word importance weight learned by the NBOW2 model.

Table 3 :

 3 20 Newsgroup topic classification accuracy. First group lists BOW methods; including different initialisations of NBOW[START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF] and NBOW2 (this work). The second group shows best reported results with LSTM RNN. Best method in each group is shown in bold. (0.2% corresponds to about 15 test set documents.)

We make available the source code at url https://drive.google.com/open?id=0BxkWDqd0McJ5R0ZPb l9YVW1vWW8 (Please use complete url without any spaces.)

http://ai.stanford.edu/ amaas/data/sentiment/

https://www.cs.cornell.edu/people/pabo/movie-reviewdata/

4 word vector size 300, word dropout probability 0.3, L2 regularisation weight 1e

-5 5 http://nlp.stanford.edu/projects/

glove/ 6 http://web.ist.utl.pt/acardoso/datasets/

from a BOW sentiment classification perspective; for other approaches or text analysis they might be essential


Imran Sheikh
email: imran.sheikh@loria.fr

Irina Illina
email: irina.illina@loria.fr

Dominique Fohr
email: dominique.fohr@loria.fr

How Diachronic Text Corpora Affect Context based Retrieval of OOV Proper Names for Audio News

Keywords: oov, proper names, lvcsr, diachronic corpus

Out-Of-Vocabulary (OOV) words missed by Large Vocabulary Continuous Speech Recognition (LVCSR) systems can be recovered with the help of the topic and lexical context of the OOV words captured from a diachronic text corpus. In this paper we investigate, how the choice of documents in the diachronic text corpora affects the retrieval of OOV Proper Names (PNs) relevant to an audio document. We first present our French diachronic broadcast news datasets which highlight the motivation of our study on OOV PNs. Then the effect of using diachronic text data from different sources and different time span is analysed. With OOV PN retrieval experiments on French broadcast news videos, we conclude that diachronic corpus with text from different sources leads to better retrieval performance than relying on text from single source or from a longer time span.

Introduction

Diachronic news is characterised by different topics which change with time, leading to a change in the linguistic content and vocabulary. As a result, a typical problem faced by Large Vocabulary Continuous Speech Recognition (LVCSR) systems processing such diachronic audio news is Out-Of-Vocabulary (OOV) words. In previous works it has been observed that majority of OOV words are PNs. PN percentage in OOV words being reported as :56% in [START_REF] Qin | Learning Out-of-Vocabulary Words in Automatic Speech Recognition[END_REF], 66% in [START_REF] Parada | OOV sensitive named-entity recognition in speech[END_REF], 57.6% in [START_REF] Palmer | Improving out-ofvocabulary name resolution[END_REF], 70% in [START_REF] Allauzen | Open vocabulary ASR for audiovisual document indexation[END_REF], 72% in [START_REF] Béchet | Tagging unknown proper names using decision trees[END_REF]. On the other hand, Proper Names (PNs) in audio news are of prime importance for content based indexing and browsing applications. Our work is closely related to recognition and recovery of OOV PNs i.e., PNs which appear in diachronic audio but are not present in the LVCSR vocabulary and cannot be recognised by the LVCSR system. To recognise OOV PNs in test audio documents, we rely on new PNs extracted from collections of diachronic text news from the internet (referred as diachronic corpus). Instead of using all these new PNs to recognise/recover the target OOV PNs 1 , it would be efficient to use only the relevant new PNs. In (Sheikh et al., 2015a), we proposed methods to reduce the long list of OOV PNs from the diachronic corpus to a smaller list of OOV PNs which are relevant to the test audio document. We refer to this task as retrieval of OOV PNs relevant to an audio document. To achieve this, we used the topic and lexical context of the audio document and OOV PNs. The topic and lexical context, which enable the retrieval of 1 Ideally new PNs extracted from collections of diachronic text news are OOV PNs with respect to the LVCSR vocabulary. However all new PNs are not present in the test set audio documents. Hence we use the term target OOV PNs to refer to the OOV PNs actually present in the test set audio documents. The general term 'OOV PNs' is used to refer to new PNs.

relevant OOV PNs, are derived from (and are dependent on) the diachronic corpus. In this paper we try to investigate some characteristics of the diachronic corpus which can affect the performance of retrieval of OOV PNs. In our previous works (Sheikh et al., 2015a;[START_REF] Sheikh | Study of entity-topic models for OOV proper name retrieval[END_REF] we have presented the problem of less frequent OOV PNs in the diachronic corpus, discussed topic model variations and proposed methods to handle the less frequent OOV PNs. In contrast, our study in this paper is around selection of documents for the diachronic corpus, which is used to model context. For instance we try to analyse the effect on the retrieval performance when: (a) diachronic corpus is built with text from different originating sources (b) diachronic corpus from a single source is supplemented with text resources only for the less frequent OOV PNs (c) diachronic corpus timeline extends far beyond test dataset timeline. Following our earlier works we will focus on retrieval of OOV PNs in French broadcast news videos. The rest of the paper is organised as follows. In Section 2. we briefly discuss our approach to retrieve OOV PNs relevant to an audio document using probabilistic topic models. In Section 3. we present French diachronic broadcast news datasets used in our experiments and in Section 4. we present the different configurations of diachronic corpora used in our study. Section 5. presents the experiments and results, followed by discussion in Section 6..

OOV PN Retrieval using Topic Models

As mentioned earlier, our task is to retrieve OOV PNs relevant to an audio document. To achieve this, we rely on new PNs extracted from collections of diachronic text news from the internet which we refer as diachronic corpus. Topic models are trained using a diachronic text corpus as training corpus to learn relations between words, latent topics and OOV PNs. During test, the audio news document is transcribed by the LVCSR with the standard vocabulary. In-Vocabulary IV words (including IV PNs) hypothesised by the LVCSR are then input to the topic model to infer Following our approach in (Sheikh et al., 2015a;[START_REF] Sheikh | Study of entity-topic models for OOV proper name retrieval[END_REF], Latent Dirichlet Allocation (LDA) [START_REF] Blei | Latent dirichlet allocation[END_REF] is used to model topics of the diachronic corpus. To model topics in the diachronic corpus of text documents, we choose a fixed vocabulary, the number of topics T and symmetric Dirichlet priors. Topic model parameters θ and φ are estimated using Gibbs sampling algorithm [START_REF] Griffiths | Finding scientific topics[END_REF]. Topics learned by LDA are used to retrieve OOV PNs relevant to the test document. Let us denote LVCSR hypothesis of test document by h and OOV PNs in diachronic corpus by ṽx . In order to retrieve OOV PNs, we calculate p(ṽ x |h), for each ṽx and then treat it as a score to rank OOV PNs relevant to h. With the words observed in h, the latent topic mixture [p(t|h)] T can be inferred by re-sampling from the word-topic distribution φ learned during training [START_REF] Griffiths | Finding scientific topics[END_REF]. Given p(ṽ x |t) = φ vt , the likelihood of an OOV PN (ṽ x ) can be calculated as: p(ṽ x |h) = T t=1 p(ṽ x |t) p(t|h).

Diachronic Broadcast News Datasets

Diachronic Corpus Configurations used in our Study

The topic context, which enables the retrieval of relevant OOV PNs, are dependent on the diachronic corpus used to model topic context. We try to study the effect of selection of documents for the training/diachronic corpus. In particular we study the following configurations of the diachronic corpus.

(A) Documents containing OOV PNs 3 and from the same time period as the test set, e.g. L'Express documents containing OOV PNs and corresponding to 6 months of the Euronews video test set.

(B) Documents coming from two different originating sources, e.g. L'Express and Le Figaro.

(C) Documents coming from a time period much longer than the test set (e.g. L'Express documents from 18 months for the Euronews video test set).

(D) Documents with OOV PNs collected from one source and then for the less frequent OOV PNs new documents are additionally collected from another source.

As discussed in our previous works (Sheikh et al., 2015a;[START_REF] Sheikh | Study of entity-topic models for OOV proper name retrieval[END_REF], retrieval of less frequent OOV PNs has a poor performance because there is not enough data to learn their topic distribution. This problem motivates us to try this configuration.

Experiment Setup

Corpora Setup

In our experiments, we use the datasets presented in Section 3.. The L'Express and Le Figaro datasets will be used as diachronic corpus to train the topic models.

LVCSR processing

The ANTS [START_REF] Illina | The Automatic News Transcription System: ANTS some Real Time experiments[END_REF] LVCSR system is used to perform automatic segmentation and speech-to-text transcription of the test audio news. The automatic transcriptions of the test audio news obtained by ANTS have an average Word Error Rate (WER) of 40% as compared to the reference transcriptions4 available from Euronews.

Topic Models

For training topic models, diachronic corpus words are lemmatised and filtered by removing PNs and non PN words occurring less than 3 times. Additionally a stoplist of common french words and non content words which do not carry any topic-related information is used. Moreover, a POS based filter is employed to choose words tagged as PN, noun, adjective, verb and acronym. PNs not present in the ANTS LVCSR lexicon are tagged as OOV PNs. LDA topic model is trained with this filtered vocabulary. We trained 100, 200, 300, 400 and 500 topics for each of the corpora configurations and first evaluated the performance on our Euronews (text) development set. Figure 1, shows the effect of number of topics on the OOV PN retrieval performance. The comparison in Figure 1 is in terms of Recall@100 and Map@100 [START_REF] Manning | Introduction to Information Retrieval[END_REF], which represent the Recall and MAP values obtained by considering the top 100 retrieved OOV PNs. Higher number of topics are more favourable for larger corpus and we can observe the same from Figure 1. LX-18m and LX+FIG continue to give better performance with increased number of topics but for LX and FIG the recall does not improve beyond 300 topics. The MAP however still improves for LX and Fig and we will discuss more about this difference in Section 6.1. As each diachronic corpus configuration will perform its best for a certain number of topics, choosing a fixed number of topics for analysis of corpora of different sizes may not be appropriate. However, since our study is focused on effects of corpus selection on the (OOV PN) retrieval performance, we will continue our analysis on the test set with each diachronic corpus modelled with 300 topics. While we will refer back to Figure 1 to support and improve relevant observations whenever required.

Discussion

Figure 2 helps us to analyse and compare the effects of diachronic corpus configurations A, B, C and D discussed in Section 4. Figure 2 shows the Recall and Mean Average Precision (MAP) [START_REF] Manning | Introduction to Information Retrieval[END_REF] performance of retrieval of OOV PNs. The performance on both reference (left) and LVCSR (right) transcriptions of the Euronews test set audio are shown. The X-axis represent the number of OOV PNs selected from the diachronic corpus. Y-axis represent recall (top) and MAP (bottom) of the target OOV PNs. For comparison, Figure 2 shows the retrieval results obtained with 300 topics.

What do Recall and MAP indicate

It is necessary to understand the importance and differences for the recall and MAP curves. After retrieval of the relevant OOV PNs, the top-N relevant OOV PNs are to be used for recovery/recognition of the target OOV PNs. To recover the target OOV PNs one can use phone matching [START_REF] Pan | Named entity recognition from spoken documents using global evidences and external knowledge sources with applications on mandarin chinese[END_REF], or additional speech recognition pass [START_REF] Oger | On-demand new word learning using world wide web[END_REF]; or spotting PNs in speech [START_REF] Parada | A spoken term detection framework for recovering outof-vocabulary words using the web[END_REF]. In each of these approaches, the retrieval ranks/scores may or may not be used. This is where the recall and MAP curves make a difference. The recall value at an operating point (i.e. N in the top-N choice) will be same whether the ranks of the retrieved OOV PNs are closer to 1 or to N. Whereas the MAP value is a function of the retrieval ranks. It can be seen that the MAP for LX is best but it does not have the best recall rates. This is because with smaller number of OOV PNs to choose from it makes smaller retrieval errors but at the same time gives smaller coverage of the target OOV PNs. We can also observe that the recall values for the reference and LVCSR transcriptions appear close but the MAP values capture the differences in ranks. Similarly, in Figure 1

Effects of different Diachronic Corpora

By analysing the recall and MAP curves of Figure 2 we can draw the following conclusions.

• Expanding time period of diachronic corpus (as in LX-18m) gives better target OOV PN coverage, but not the best recall rates. Additionally it gives a low MAP. Such corpora can be possibly exploited by training more number of topics or by better retrieval methods [START_REF] Sheikh | Learning to retrieve out-of-vocabulary words in speech recognition[END_REF][START_REF] Sheikh | Document level semantic context for retrieving OOV proper names[END_REF].

• Adding documents for less frequent OOV PNs is not effective (performance of LX+rFIG is similar to that of LX+FIG). We found that adding documents for less frequent OOV PNs in LX+rFIG leads to inclusion of more than 60% of FIG documents. The additional data not only increases data for learning topic representation of less frequent OOV PNs but also comes with additional less frequent OOV PNs and more instances for frequent OOV PNs. Further analysis of the ranks of the less frequent OOV PNs obtained with LX, LX+FIG and LX+rFIG showed that the ranks with LX+rFIG are better with respect to LX but similar to that with LX+FIG.

• Using documents of same time period and from multiple sources (LX+FIG) gives a good balance of recall, MAP and target OOV PN coverage.

Conclusion

Diachronic text corpora are essential for recovery of OOV words and proper names missed by LVCSR systems. We performed a study to analyse the selection of text documents in the diachronic corpus used for retrieval of OOV proper names. French broadcast news videos from a particular time period were used as the test set and text articles from different French news websites formed the diachronic text corpora. With OOV proper name retrieval based on topic context, we can conclude that (a) text from a longer time span can give increased coverage of OOV proper names (b) but corpus with text from different sources leads to better retrieval performance than relying on text from single source, even if it corresponds to longer time span (c) and less frequent OOV proper names need improvement in retrieval methods [START_REF] Sheikh | Document level semantic context for retrieving OOV proper names[END_REF] and not just additional training data.

Figure 1 :

 1 Figure 1: Recall@100 and MAP@100 for OOV PN retrieval on Euronews text (Dev set) with different number of LDA topics trained on the diachronic corpora. ( LX, FIG, LX+FIG, LX+rFIG, LX-18m)

Figure 2 :

 2 Figure 2: Recall and MAP for OOV PN retrieval on Euronews news videos with different diachronic corpora for training LDA topic model. ( LX, FIG, LX+FIG, LX+rFIG, LX-18m)

  the recall values for LX and Fig do not improve beyond 300 topics but the MAP values increase due to the improvement in ranks of the target OOV PNs within the top 100 retrieved OOV PNs.

Table 1 :

 1 Broadcast news diachronic datasets

		L'Express Le Figaro L'Express + L'Express Euronews Euronews
				Le Figaro
		(LX)	(FIG)	(LX+FIG)	(LX-18m)	(Dev)	(Test)
	Type of Documents	Text	Text	Text	Text	Text	Video
	Time Period	Jan -Jun 2014	Jan -Jun 2014	Jan -Jun 2014	Jul 2013 -Jan -Jun Dec 2014 2014	Jan -Jun 2014
	Number of Documents 1	45K	59K	104K	142K	3.1K	3K
	Vocabulary Size (unigrams) 2	150K	140K	180K	270K	42K	45K
	Corpus Size (approx. word count)	24M	18M	42M	70M	550K	700K
	Number of PN unigrams 2	57K	51K	80K	104K	12K	11K
	Total PN count	1.45M	1.3M	2.7M	4.2M	54K	42K
	Number of OOV unigrams 3	12.4K	11.9K	24.4K	37.1K	4.9K	4.3K
	Documents with OOV 3	32.3K	36.4K	73K	109K	2.25K	2.2K
	Total OOV count 3	141K	142K	320K	509K	9.1K	8K
	Number of OOV PN unigrams 3	9.3K	8.8K	18.4K	28.2K	3.4K	3.1K
	Documents with OOV PN 3	26.5K	30K	61.3K	93.5K	1.9K	1.9K
	Total OOV PN count 3	107K	103K	243K	388K	6.9K	6.2K

1 K denotes Thousand and M denotes Million, 2 unigrams occurring less than two times are ignored 3 unigrams occurring in less than three documents ignored, documents with more than 20 and less than 500 terms Note: (a) OOV and OOV PN statistics are post term-document filtering (b) 2 and 3 does not apply for Euronews the topic distribution of the test audio document. With the test document topic distribution we retrieve a list of most relevant OOV PNs for the test document.

,

  72% (3.1K out of 4.3K) of OOV words in the Euronews video dataset are PNs and about 63% (1.9K out of 3K) of the videos contain OOV PNs. An important statistic target OOV PN coverage is not shown in Table1. We use the term target OOV PN coverage to refer to the percentage of the target OOV PNs in Euronews videos which can be recovered with a diachronic corpus. The target OOV PN coverage for each of the diachronic text corpus is found to be: for LX 42%, for FIG 40%, for LX+FIG combined 52% and for LX-18m 54%.

	Table 1 presents realistic broadcast news diachronic
	datasets which will be used as the training and test sets
	in our study. These datasets also highlight the moti-
	vation for handling OOV PNs. The datasets are col-
	lected from three sources: (a) French newspaper L'Express

(http://www.lexpress.fr/) (b) French newspaper Le Figaro (http://www.lefigaro.fr/) (c) French website of the Euronews (http://fr.euronews.com/) television channel. The L'Express and Le Figaro datasets contain text news whereas the Euronews dataset contains text news as well news videos and their text transcriptions. In our study the L'Express and Le Figaro datasets will be used as diachronic corpora to train topic models, in order to infer the OOV PNs relevant to Euronews videos. TreeTagger 2

[START_REF] Schmid | Probabilistic part-of-speech tagging using decision trees[END_REF]

is used to automatically tag PNs in the text. The words and PNs which occur in the lexicon of our Automatic News Transcription System (ANTS)

[START_REF] Illina | The Automatic News Transcription System: ANTS some Real Time experiments[END_REF]

are tagged as IV and remaining PNs are tagged as OOV. ANTS lexicon is based on news articles until 2008 from French newspaper LeMonde and contains about 123K unique words. As shown in Table

1

http://www.cis.uni-muenchen.de/ schmid/tools/TreeTagger/

From our experiments we observed that adding non OOV PN documents to the diachronic corpus does not give significant improvement in the OOV PN retrieval performance.

these are approximate transcriptions; on a set of 10 manually transcribed audio files we obtained a WER of 33.8%

Acknowledgment

This work is funded by the ContNomina project supported by the French National Research Agency (ANR) under the contract ANR-12-BS02-0009.


Imran Sheikh
email: imran.sheikh@loria.fr

Irina Illina
email: irina.illina@loria.fr

Dominique Fohr
email: dominique.fohr@loria.fr

Georges Linarès
email: georges.linares@univ-avignon.fr

DOCUMENT LEVEL SEMANTIC CONTEXT FOR RETRIEVING OOV PROPER NAMES

Keywords: OOV, proper names, semantic, indexing

Recognition of Proper Names (PNs) in speech is important for content based indexing and browsing of audio-video data. However, many PNs are Out-Of-Vocabulary (OOV) words for LVCSR systems used in these applications due to the diachronic nature of data. By exploiting semantic context of the audio, relevant OOV PNs can be retrieved and then the target PNs can be recovered. To retrieve OOV PNs, we propose to represent their context with document level semantic vectors; and show that this approach is able to handle less frequent OOV PNs in the training data. We study different representations, including Random Projections, LSA, LDA, Skip-gram, CBOW and GloVe. A further evaluation of recovery of target OOV PNs using a phonetic search shows that document level semantic context is reliable for recovery of OOV PNs.

INTRODUCTION

Large Vocabulary Continuous Speech Recognition (LVCSR) based audio indexing approaches allow search, navigation, browsing and structuring of audio-video datasets [START_REF] Alberti | An audio indexing system for election video material[END_REF]. However, such datasets are diachronic and LVCSR processing can be challenging due to the variations in linguistic content and vocabulary. Thus leading to Out-Of-Vocabulary (OOV) words for LVCSR. In previous works [START_REF] Qin | Learning out-of-vocabulary words in automatic speech recognition[END_REF][START_REF] Palmer | Improving out-ofvocabulary name resolution[END_REF][START_REF] Parada | OOV sensitive named-entity recognition in speech[END_REF][START_REF] Allauzen | Open vocabulary ASR for audiovisual document indexation[END_REF][START_REF] Béchet | Tagging unknown proper names using decision trees[END_REF] it has been observed that a majority of OOV words (56-72%) are Proper Names (PNs). But PNs are important for indexing. In this paper, we focus on the problem of retrieval of relevant OOV PNs for recovery of the target OOV PNs. (We refer to all the PNs not in LVCSR vocabulary as OOV PNs and the OOV PNs present in the test audio as target OOV PNs.)

When processing diachronic audio with LVCSR systems, the number of OOV PNs can be very high [START_REF] Fohr | Continuous word representation using neural networks for proper name retrieval from diachronic documents[END_REF]. Even if good amount of training data is available, appending the LVCSR vocabulary and updating the Language Model (LM) is not always a feasible solution [START_REF] Qin | Learning out-of-vocabulary words in automatic speech recognition[END_REF]. To recognize the target OOV PNs in an audio document, we first find a list of OOV PNs which are relevant to this audio document, by using diachronic text resources. OOV PNs are inferred based on the semantic/topic context of the audio document. (We consider datasets which contain one news event per document. News documents with multiple news events are part of another study [START_REF] Fohr | Continuous word representation using neural networks for proper name retrieval from diachronic documents[END_REF].) The reduced list of relevant OOV PNs can then be used to recover target OOV PNs using phone matching [START_REF] Pan | Named entity recognition from spoken documents using global evidences and external knowledge sources with applications on mandarin chinese[END_REF], or additional speech recognition pass [START_REF] Oger | Ondemand new word learning using world wide web[END_REF]; or spotting PNs in speech [START_REF] Parada | A spoken term detection framework for recovering out-ofvocabulary words using the web[END_REF]. In this paper, we evaluate recovery of the target OOV PNs using a phonetic search. (LVCSR decoding with updated LM can give better results [START_REF] Chen | Quantifying the value of pronunciation lexicons for keyword search in lowresource languages[END_REF] but this is not our current focus.)

Retrieval and recovery of OOV PNs in audio documents, based on the topic and semantic context, has been proposed previously. Latent Dirichlet Allocation (LDA) and Latent Semantic Analysis (LSA) was used to train context models for PNs in [START_REF] Senay | Person name spotting by combining acoustic matching and LDA topic models[END_REF][START_REF] Bigot | Person name recognition in ASR outputs using continuous context models[END_REF]. However, these approaches work for PNs with significant amount of training documents. Retrieval methods adapted to handle less frequent PNs were proposed in [START_REF] Sheikh | OOV proper name retrieval using topic and lexical context models[END_REF][START_REF]Study of entity-topic models for OOV proper name retrieval[END_REF]. In [START_REF] Wintrode | Combining local and broad topic context to improve term detection[END_REF] topic context was used to update LM for LVCSR based term detection. But as mentioned, updating LVCSR LM is not always feasible [START_REF] Qin | Learning out-of-vocabulary words in automatic speech recognition[END_REF][START_REF] Fohr | Continuous word representation using neural networks for proper name retrieval from diachronic documents[END_REF]. The originality of our work is the use of document level semantic representations to model OOV PNs. We follow this approach, because it can model less frequent OOV PNs which have less training instances and secondly because this approach (unlike the rare OOV PN re-ranking technique proposed in [START_REF] Sheikh | OOV proper name retrieval using topic and lexical context models[END_REF][START_REF]Study of entity-topic models for OOV proper name retrieval[END_REF]) can be applied to different semantic representations, as discussed in this paper. As compared to [START_REF] Sheikh | OOV proper name retrieval using topic and lexical context models[END_REF] we study document level semantic context derived from different prominent representations including (a) LSA, LDA (b) Continuous Bag-Of-Words (CBOW) and Skip-gram distributed representations of words learned by neural networks [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] and (c) GloVe [START_REF] Pennington | Glove: Global vectors for word representation[END_REF].

The paper is organized as follows: Section 2 describes our approach to represent and recover OOV PNs with document context. Section 3 presents the different document representations. Sections 4 & 5 present the experiment setup and results.

METHODOLOGY

Our main objective is to identify a list of OOV PNs relevant to the given audio document. Relevant OOV PNs are retrieved based on semantic context. Then (to assess the retrieved list of OOV PNs) we try to recover the target OOV PNs using the retrieved list. For this a phonetic search is performed on the LVCSR hypothesis with each of the retrieved OOV PNs.

• Retrieval of relevant OOV PNs: To retrieve OOV PNs relevant to an audio document we rely on its semantic context. Several methods have been proposed in literature to obtain word embeddings corresponding to semantic context of words. These embeddings are derived from word cooccurrence statistics [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] and hence are dependent on the frequency of occurrence of words in training data [START_REF]Study of entity-topic models for OOV proper name retrieval[END_REF]. As a result, these methods do not learn good representations for less frequent words (and OOV PNs) or mostly discard them in training. In our task, we therefore use the semantic representation of a document as the semantic context of the OOV PN occurring in this document. During training, diachronic text news are collected from the internet and indexed with new (i.e., OOV) PNs. These set of text documents indexed with OOV PNs is referred as diachronic corpus. A semantic vector representation is learned for each of the documents in the diachronic corpus and stored as a semantic context vector for the OOV PNs in that document. OOV PNs occurring in more than one diachronic document will have multiple context vectors or in other words document specific context vectors. Multiple OOV PNs in a diachronic document will share a common context vector. During test the semantic vector representation of the LVCSR hypothesis of the audio document (H) is compared with the context vectors (C i ) for each of the OOV PNs. For retrieval, relevance score s ≈ max i {CosineSimilarity(H, C i )} is used. This approach applies to different document representations.

• Phonetic Search for Target OOV PNs: To recover the target OOV PNs a phonetic search is performed on the LVCSR hypothesis, for each of the retrieved OOV PNs. Phonetic form of the LVCSR word hypothesis is obtained and the retrieved OOV PNs are converted into their phonetic forms using a Grapheme to Phoneme (G2P) converter. We employ a search based on the classical k-differences approximate string matching algorithm [START_REF] Navarro | A guided tour to approximate string matching[END_REF]. In our case the algorithm makes a decision based on the phonetic string match score. It should be noted that it is not required to search the entire LVCSR hypothesis. The error and OOV regions in the LVCSR hypothesis can be hypothesised (as in [START_REF] Lecouteux | Combined low level and high level features for out-of-vocabulary word detection[END_REF]) and only these regions can be searched. Further, the phonetic search can be improved using techniques like searching Nbest results or phone lattice, using phone confusion matrix, etc. (which is not the main focus of this paper).

DOCUMENT LEVEL CONTEXT VECTORS

In this section we briefly present the different document level representations used to model context of OOV PNs. Random Projection is our non-semantic baseline representation, whereas LSA, LDA, CBOW, Skip-gram and GloVe are the semantic counterparts used to represent context of OOV PNs.

• Random Projections of TF-IDF vector: It is classical to represent text documents as vector of Term Frequency-Inverse Document Frequency (TF-IDF) values of the words in the vocabulary. It has been shown that Random Projections can efficiently reduce the dimensionality of TF-IDF vectors while still preserving their original similarities and distances [START_REF] Bingham | Random projection in dimensionality reduction: Applications to image and text data[END_REF]. With random projection, the N -dimensional TF-IDF vectors of D documents in a corpus (X D×N ) are projected to a K-dimensional (K << N ) subspace as:

X RP K×D = R K×D X D×N
, where R K×D is a random matrix with random unit vectors.

• LSA and LDA: LSA [START_REF] Deerwester | Indexing by latent semantic analysis[END_REF] and LDA [START_REF] Blei | Latent dirichlet allocation[END_REF] have been prominent unsupervised methods to obtain semantic/topic space representations from documents. In LSA TF-IDF matrix of documents are projected into a (K dimensional) semantic space by performing Singular Value Decomposition (SVD).

LDA is a generative probabilistic model which learns a (K) topic space in which documents are expressed as a mixture of topics and each topic is a distribution over words.

• CBOW and Skip-gram: Distributed approaches based on neural networks are being widely adopted for learning word embeddings in semantic space. The work of Mikolov et al. [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] has become popular due to its ability to handle large amounts of unstructured data with reduced computational costs and perform with high accuracy. It proposed two models: (a) the Skip-gram model, with an objective function to maximize the likelihood of prediction of contextual words given the centre word; (b) the CBOW model which predicts the centre word given the surrounding words. In our work we use these word embeddings to represent documents. During training the CBOW/Skip-gram word embeddings are learned for all the words in the diachronic corpus. Given these word embeddings and their linearity property, we obtain a representation for a document by taking an average over all the vocabulary words in the document. This document representation is referred to as AverageVec.

(Paragraph Vector, a distributed model to represent sentences, paragraphs and documents has been proposed [START_REF] Le | Distributed representations of sentences and documents[END_REF]. But it gives a poor performance in our experiments.)

• GloVe: LSA/LDA derive semantic spaces by performing factorization of document level co-occurrence statistics.

Whereas CBOW and Skip-gram scan context windows across the corpus and produce linear directions of meaning. GloVe (Global Vectors) model was proposed [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] to produce linear dimensions of meaning and capture global corpus statistics. The improved performance of GloVe reported in [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] motivates us to use GloVe. As in case of CBOW/Skip-gram we obtain a document representation from GloVe by averaging the words in the document.

DIACHRONIC BROADCAST NEWS DATASETS

We present two realistic diachronic news datasets which are our training and test sets. These datasets, described in

EXPERIMENTS AND RESULTS

In our experiments the L'Express dataset is used as diachronic corpus. Audio from the Euronews video dataset is used as the test set. The ANTS [START_REF] Illina | Automatic News Transcription System: ANTS some Real Time experiments[END_REF] LVCSR system is used to perform automatic segmentation and speech-to-text transcription of the test audio news. The automatic transcriptions of the test audio news obtained by ANTS have an average Word Error Rate (WER) of 40% as compared to the reference transcriptions.

Diachronic corpus vocabulary is lemmatized and filtered by removing PNs occurring only once, non PN words less than 4 times, and using a stop-list of common and non-content French words. Moreover, only words tagged as PN, noun, adjective, verb and acronym are retained. The filtered vocabulary has 40K PNs and 28K words. Out of the 40K PNs 17K are not present in the ANTS LVCSR lexicon and are tagged as OOV PNs. Four OOV PNs, present in test set and not present in diachronic corpus, are excluded from experiments. Context vectors discussed in Section 3 are trained with this filtered vocabulary. We tried different numbers of dimensions/topics (in range 20-1K), the best performance is obtained for 300. LDA hyper-parameters and window size for CBOW, Skip-gram and GloVe are chosen for best performance.

Performance of OOV PN Retrieval

OOV PN retrieval using context vectors, as discussed in Section 2, is evaluated on the 1415 test audio news (in Euronews dataset) which contains OOV PNs. As shown in Table 1, these 1415 documents consist of 1024 unique OOV PN unigrams occurring a total of 3128 times. However, the total number of OOV PNs to be retrieved, obtained by counting unique OOV PNs per document, is 2300. Out of the 2300, 476 (20%) occur 5 times or fewer in the diachronic corpus.

Retrieval using Word v/s Document Semantic Context

We compare the performance of retrieval of OOV PNs when using document level context vectors, as discussed in Section 2, to that when using OOV PN (word) context vectors, as proposed in earlier works [START_REF] Sheikh | OOV proper name retrieval using topic and lexical context models[END_REF][START_REF]Study of entity-topic models for OOV proper name retrieval[END_REF]. To study the difference in the performance we plot the distribution of the ranks obtained by the target OOV PNs versus the frequency of occurrence of the target OOV PNs in the diachronic corpus used for training. Figure 1 shows the rank-frequency distribution when using word (left) and document (right) context vectors from LDA. (This comparison is shown only for LDA but holds true for other representations discussed in this paper.) Word level semantic context vectors perform better for frequent OOV PNs whereas document level semantic context vectors are more uniform across different OOV PNs and hence better for retrieval of less frequent OOV PNs. The improvement in retrieval performance can be measured in terms of Recall [START_REF] Manning | Introduction to Information Retrieval[END_REF] at 5% operating point, an operating point chosen for analysis and to restrict phonetic search. Using document context vectors over word context vectors gives a 5% absolute improvement in recall with LDA based representation. This improvement is 12% with Skip-gram. This justifies the use of document level semantic context vectors, along with the fact that this approach can be used for different representations. The best recall and MAP performance on LVCSR transcriptions is obtained with the CBOW representation and the worst with Random Projections (RP) which is a non-semantic representation. It must be noted that the performance with RP is not the worst for reference transcriptions of the videos. The speech recognition errors introduced with the LVCSR degrade the document representation. But with semantic representation the degradation is lesser. The performance trend for LVCSR is: CBOW/Sg > GloVe > LDA/LSA > RP. As opposed to the results on the analogy task in [START_REF] Pennington | Glove: Global vectors for word representation[END_REF], GloVe does not outperform CBOW/Sg representations in our task. Furthermore, the Recall graph shows that document context based retrieval of OOV PNs can reduce the search list for target OOV PN drastically. The context vectors of the LVCSR hypothesis can recover up to 79-87% of the target OOV PNs within top 5% of retrieval results. Thus reducing the phonetic search to only 5% i.e. 850 OOV PNs from diachronic corpus.

Performance of Different Semantic Representations

Target OOV PN Recovery Performance

Target OOV PNs are searched in the LVCSR hypothesis as discussed in Section 2. The phonetic string corresponding to the LVCSR hypothesis is obtained using forced alignment and the OOV PNs are converted to phone strings with our G2P converter [START_REF] Illina | Multiple Pronunciation Generation using Grapheme-to-Phoneme Conversion based on Conditional Random Fields[END_REF]. As mentioned, k-differences approximate matching algorithm [START_REF] Navarro | A guided tour to approximate string matching[END_REF] is used for performing phonetic search. The search was performed only in the error regions of the LVCSR hypothesis (obtained by alignment with manual transcriptions). Figure 3 shows the F 1 -scores for recovery of the target OOV PNs. The best F 1 -score is shown for different The trends of Recall and MAP of Figure 2 reflect in the F 1 -scores of Figure 3. CBOW and Sg context vector based retrieval results give better F 1 -scores than GloVe, which is better than LDA/LSA. RP gives least F 1 -scores. However, the F 1 -score stops improving beyond top 64 retrieved OOV PNs due to the increase in false positives. The best F 1 -score 0.41 is obtained with Skip-gram based document context vector.

CONCLUSION

We proposed an new approach to represent OOV PNs and their context with document level semantic vector representations for retrieval of OOV PNs relevant to an audio document. This approach can handle less frequent OOV PNs in the training data and gives better retrieval performance compared to word level semantic vector representations. Among semantic representations, AverageVec obtained from CBOW and Skipgram neural network word embeddings perform better than the classical LDA/LSA topic/semantic spaces. Recovery of the target OOV PNs using a phonetic search gives an F 1 -score upto 0.41, thus confirming that document level semantic context is reliable for recovery of OOV PNs in diachronic audio.

Fig. 1 .

 1 Fig. 1. Rank-Frequency Distribution for Retrieval with LDA based Word (left) and Document (right) Context Vectors

Figure 2

 2 Figure2shows the OOV PN retrieval performance in terms of Recall and Mean Average Precision (MAP)[START_REF] Manning | Introduction to Information Retrieval[END_REF] on the Reference and LVCSR transcriptions of the test set. In the graphs

Fig. 2 . 2 ,

 22 Fig. 2. OOV PN retrieval with different representations ( RP, LSA, LDA, CBOW, Sg, GloVe).in Figure2, the X-axis represents the number of top-N OOV PNs selected from the diachronic corpus. The Y-axis represents recall and MAP of the target OOV PNs. The different context representations are denoted by their abbreviations.The best recall and MAP performance on LVCSR transcriptions is obtained with the CBOW representation and the worst with Random Projections (RP) which is a non-semantic representation. It must be noted that the performance with RP is not the worst for reference transcriptions of the videos. The speech recognition errors introduced with the LVCSR degrade the document representation. But with semantic representation the degradation is lesser. The performance trend for LVCSR is: CBOW/Sg > GloVe > LDA/LSA > RP. As opposed to the results on the analogy task in[START_REF] Pennington | Glove: Global vectors for word representation[END_REF], GloVe does not outperform CBOW/Sg representations in our task. Furthermore, the Recall graph shows that document context based retrieval of OOV PNs can reduce the search list for target OOV PN drastically. The context vectors of the LVCSR hypothesis can recover up to 79-87% of the target OOV PNs within top 5% of retrieval results. Thus reducing the phonetic search to only 5% i.e. 850 OOV PNs from diachronic corpus.

Fig. 3 .

 3 Fig. 3. OOV PN recovery with different representations. number of OOV PNs retrieved by the context model.The trends of Recall and MAP of Figure2reflect in the F 1 -scores of Figure3. CBOW and Sg context vector based retrieval results give better F 1 -scores than GloVe, which is better than LDA/LSA. RP gives least F 1 -scores. However, the F 1 -score stops improving beyond top 64 retrieved OOV PNs due to the increase in false positives. The best F 1 -score 0.41 is obtained with Skip-gram based document context vector.

Table 1 ,

 1 also highlight the motivation for handling OOV PNs. The L'Express dataset is collected from the website (lexpress.fr) of the French newspaper L'Express whereas the Euronews dataset is collected from the French website (fr.euronews.com) of the Euronews TV channel.

	L'Express

Table 1 .

 1 Broadcast news diachronic datasets.

		L'Express Euronews
	Type of Documents	Text	Video
	Time Period	Jan 2014 -Jun 2014
	Number of Documents*	45K	3K
	Vocabulary Size (unigrams)	150K	18K
	Corpus Size (total word count)	24M	600K
	Number of PN unigrams+	40K	2.2K
	Total PN count	1.3M	19K
	Documents with OOV	43K	2172
	Number of OOV unigrams+	55K	1588
	Total OOV count	450K	7710
	Documents with OOV PN	36K	1415
	Number of OOV PN unigrams+	17K	1024
	Total OOV PN count	200K	3128
	*K =10 3 and M =10 6 ; +unigrams occurring once excluded