D L A N G L O I S

Alignment of comparable documents: comparison of similarity measures on French-English-Arabic data

The objective, in this article, is to address the issue of the comparability of documents, which are extracted from different sources and written in different languages. These documents are not necessarily translations of each other. This material is referred as multilingual comparable corpora, These language resources are useful for multilingual natural language processing applications, especially for low-resourced language pairs. In this paper, we collect different data in Arabic, English and French. Two corpora are built by using available hyper links for Wikipedia and Euronews. Euronews is an aligned multilingual (Arabic, English and French) corpus of 34k documents collected from Euronews website. A more challenging issue is to build comparable corpus from two different and independent medias having two distinct editorial lines such as British Broadcasting Corporation (BBC) and Al Jazeera (JSC). To build such corpus, we propose to use the Cross-Lingual Latent Semantic approach. For this purpose, documents have been harvested from BBC and JSC web sites for each month of the years 2012 and 2013. The comparability is calculated for each Arabic-English couple of documents of each month. This automatic task is then validated by hand. This led to a multilingual (Arabic-English) aligned corpus of 305 pairs of documents (233k English words and 137k Arabic words). In addition A study is presented in this paper to analyze the performance of three methods of the literature allowing to measure the comparability of documents on the multilingual reference corpora. A recall at rank 1 of 50.16 per cent is achieved with the Cross-lingual LSI approach for BBC-JSC test corpus, while the dictionary-based method reaches a recall of only 35.41 per cent.

Introduction

Multilingual texts (parallel or comparable) are useful in several Natural Language Processing applications such as bilingual lexicon extraction [START_REF] Li | Improving corpus comparability for bilingual lexicon extraction from comparable corpora[END_REF], cross-lingual information retrieval [START_REF] Knoth | Using explicit semantic analysis for cross-lingual link discovery[END_REF] and machine translation [START_REF] Delpech | Evaluation of terminologies acquired from comparable corpora: an application perspective[END_REF]. A parallel corpus is a collection of aligned sentences, which are translations of each other. Parallel corpora are acquired using human translators, but this is time-consuming and requires a lot of human efforts.

In addition, these resources are not available for all the language pairs. The huge quantity of data on Internet has increased the interest for collecting multilingual corpora related to the same subject. These text resources are known as comparable corpora. Below, we give an example of two short documents, extracted from Wikipedia in English and French, which are an illustration of what is called comparable texts.

Barack Hussein Obama II born August 4, 1961 is an American politician who is the 44th and current President of the United States. He is the first African American to hold the office and the first president born outside the continental United States.

Barack Hussein Obama II, né le 4 août 1961 à Honolulu (Hawaï), est un homme d' État américain. Il est le 44e et actuel président des États-Unis, élu pour un premier mandat le 4 novembre 2008 et réélu pour un second le 6 novembre 2012.

These two comparable documents are not translation of each other, but they are related to the same topic. In addition, they embed few fragments of sentences which are translation of each other. The idea consisting in using the web to get comparable corpora is very attractive, but the issue is how to align multilingual documents sharing the same topic, or the same news? In other words, from a list of source documents, how to associate each document to its corresponding one from a list of target documents.

Producing comparable corpora has several utilities in Natural Language Processing. Among them, we can mention the improvement of machine translation systems [START_REF] Munteanu | Improving machine translation performance by exploiting non-parallel corpora[END_REF][START_REF] Abdul-Rauf | On the use of comparable corpora to improve SMT performance[END_REF]. This material could be used also to create parallel corpora for under-resourced languages (Wo lk and Marasek, 2014). Comparable corpora can be used to extract classical bilingual lexicon [START_REF] Morin | Bilingual lexicon extraction from comparable corpora enhanced with parallel corpora[END_REF] or transliteration lexicon for under-resourced languages (Abidi and Smaïli, 2011). Our purpose in this article is to lay the groundwork for an automatic press review. In other words, we would like to compare the content of two documents produced by two different news agencies. The documents may be different at the language level or at the level of the editorial lines of the news agencies. For instance, in AMIS (Access Multilingual Information opinionS ) 1 , a Chist-Era project in which we are involved in, and aims to compare the content of two videos in terms of opinions. The supported languages are Arabic, French and English.

To achieve the previous objectives, one should make the corpora comparable. Therefore, in this paper, we investigate the task of aligning comparable documents. Given a multilingual corpus which is not aligned, the objective is to automatically align source and target documents which are related to the same context. In this article, we tackle the comparability of documents in three languages: Arabic, English and French. There are two kinds of methods are tested in this paper to reach the objective of aligning documents; the first kind is based on lexical information and the second relies on latent information. We compare these methods on three different multilingual corpora; one is provided by the evaluation campaign: BUCC 2015 [START_REF] Sharoff | Bucc shared task: Cross-language document similarity[END_REF]. This corpus is extracted from Wikipedia, the experiments consist in aligning French and English documents. The second one; is more suitable for our objective, since it concerns the press. This corpus has been built in a previous work and concerns documents of Euronews [START_REF] Saad | Extracting Comparable Articles from Wikipedia and Measuring their Comparabilities[END_REF]. For this second corpus, experiments concern the alignments of the three languages Arabic, French and English. The third corpus is more challenging since, the objective is to align two different medias with two different editorial lines in different languages. The concerned medias are BBC and Al Jazeera. It is remarkable that building a comparable corpus from BBC and Al Jazeera is more difficult, since inter-lingual hyper links are missing, contrary to Wikipedia or Euronews.

The rest of this paper is structured as follows. In Section 2, we give a description of some related works about alignment of comparable articles. The three similarity measures we experiment are described in Section 3. In Section 4, we describe the data sets extracted from Wikipedia and Euronews, and in Section 5 we present the method we used to build the BBC-Al Jazeera corpus. Then, we give the results of the different approaches evaluated on the three data sets (Section 6.1). Finally, we propose a discussion and a conclusion.

Related works

In [START_REF] Fung | Multi-level bootstrapping for extracting parallel sentences from a quasi-comparable corpus[END_REF], the authors proposed three levels for non-parallel corpora. These levels are noisy-parallel, comparable and quasi-comparable corpora. Texts in noisy-parallel corpora have many parallel sentences roughly in the same order. Texts in comparable corpora have topic aligned documents, which are not necessarily translations of each other. Quasi-comparable corpora contain bilingual documents that are not necessarily related to the same topic.

Most of the researchers are interested in comparable corpora because they can be used to extract parallel texts for different purposes. This interest continues to receive increasing attention from the research community. For example, ACCURAT2 (Pinnis et al., 2012;[START_REF] Skadina | Collecting and using comparable corpora for statistical machine translation[END_REF]) is a research project dedicated to finding methods and techniques to overcome the problem of lacking linguistic data for under-resourced languages and narrow domains. For that, the project proposes approaches to align and extract lexical resources. The ultimate objective is to exploit comparable data to improve the quality of machine translation for under-resourced languages. Other researchers and projects (for example, the "TTC: Terminology Extraction, Translation Tools and Comparable Corpora" project3 ) also considered comparable corpora to improve the quality of machine translation [START_REF] Smith | Extracting parallel sentences from comparable corpora using document level alignment[END_REF][START_REF] Abdul-Rauf | Parallel sentence generation from comparable corpora for improved SMT[END_REF] and to extract bilingual lexicons [START_REF] Li | Improving corpus comparability for bilingual lexicon extraction from comparable corpora[END_REF].

The objective of some other researchers is to improve the overall comparability of the corpus [START_REF] Li | Measuring and improving comparable corpus quality[END_REF] without retrieving specific alignments between documents, but our objective is to build inter-lingual aligned document pairs; in other words, [START_REF] Li | Measuring and improving comparable corpus quality[END_REF] focused on corpus level comparability measure while we focus in this work on document level comparability measure. The methods to align comparable documents are numerous. The issue of aligning comparable multilingual documents could be considered such as an information retrieval (IR) task. Cross-Lingual Information Retrieval (CL-IR) is a special case of IR, where the language of the query is different from the language of the documents [START_REF] Ballesteros | Dictionary methods for cross-lingual information retrieval[END_REF]. In this case, the CL-IR system should unify the language of queries and documents by using for instance a Machine Translation (MT) system [START_REF] Aljlayl | On Arabic-English Cross-Language Information Retrieval: Machine Translation Approach[END_REF][START_REF] Ture | Searching to Translate and Translating to Search: When Information Retrieval Meets Machine Translation[END_REF][START_REF] Hieber | Bag-of-words forced decoding for cross-lingual information retrieval[END_REF]. The drawback of this approach is the dependence on the MT system, which affects the performance of the IR system. Moreover, the MT system needs to be developed first if it is not available.

Other different approaches are proposed in this topic. For instance in [START_REF] Ion | An expectation maximization algorithm for textual unit alignment[END_REF], the authors proposed an Expectation-Maximization algorithm to find the aligned documents. This is achieved by an iterative process which stops when the words in the aligned documents match at best the content of a translation table. This method requires a parallel corpus for training the translation table. In [START_REF] Vulić | Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings[END_REF], the authors presented a method which tries to show the relationship between word embeddings text representation and information retrieval methods. The work showed, among other points, how to model the cross-lingual semantic embedding space without the use of parallel data or bilingual dictionaries.

In [START_REF] Oshikiri | Cross-lingual word representations via spectral graph embeddings[END_REF], the authors proposed a method called Cross-Lingual Eigenword which differs from the current trend: word embedding based on neural networks such as in [START_REF] Vulić | Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings[END_REF]. The authors, instead of using the skip-gram model, extended the spectral Eigenwords [START_REF] Dhillon | Eigenwords: Spectral word embeddings[END_REF] to cross-lingual settings with sentence-alignment. Even if Cross-Lingual Latent Semantic Indexing has not been dedicated to word embedding, the authors have used it as a baseline method. This method achieves competitive results on a English-Spanish word translation task. Unfortunately it has not been tested on the alignment of comparable corpus, which is our objective in this article.

Experimented alignment methods

In this section, we present some alignment methods of the state-of-the-art, and we experiment them in Section 6. We present the hapax-based method and a dictionary-based method which have been used in BUCC 2015 shared task campaign [START_REF] Sharoff | Bucc shared task: Cross-language document similarity[END_REF]. This campaign asked the competitors to retrieve comparable documents from a Wikipedia corpus. We also experimented Cross-Lingual Latent Semantic Indexing (CL-LSI) method on the corpora yet used in [START_REF] Saad | Cross-lingual semantic similarity measure for comparable articles[END_REF][START_REF] Saad | Mining Documents and Sentiments in Cross-lingual Context[END_REF] and on the Wikipedia corpus from the BUCC2015 shared task.

Hapax

Morin et al. in [START_REF] Morin | Lina: Identifying comparable documents from wikipedia[END_REF] proposed a method based only on the lexical content of documents. The method is based on the matching of the hapax words (which occur once) in both source and target documents. For example, this can be the case for url, entity names, Hashtags, etc. Then, the more the documents have common hapax words, the more they are close to each other. In the context of this paper, this method is called HAPAX.

Dictionary-based methods

In dictionary-based methods, two cross-lingual documents d s and d t are comparable if most of words in d s are translations of words in d t . A bilingual dictionary has to be used to look-up the translations of words in both documents. The similarity can be measured as follows [START_REF] Li | Improving corpus comparability for bilingual lexicon extraction from comparable corpora[END_REF][START_REF] Otero | Measuring comparability of multilingual corpora extracted from Wikipedia[END_REF]:

sim(d s , d t ) = |w s → w t | + |w t → w s | |d s | + |d t | (1)
where |w s → w t | is the number of source words (w s ) that are in the bilingual dictionary and have translations in the target document (d t ), and |w t → w s | is the number of target words (w t ) that are in the bilingual dictionary and have translations in the source document (d s ). |d s | + |d t | is the number of words of the source and the target documents that are also in the bilingual dictionary.

Several similarity measures based on a bilingual dictionary have been proposed. Some of them consider the similarity at the corpus level [START_REF] Li | Improving corpus comparability for bilingual lexicon extraction from comparable corpora[END_REF], while others consider it at the document level, then aggregate similarities of documents in the corpus [START_REF] Otero | Measuring comparability of multilingual corpora extracted from Wikipedia[END_REF].

In our experiments, we used Li & Gaussier measure [START_REF] Li | Improving corpus comparability for bilingual lexicon extraction from comparable corpora[END_REF]; in addition, we take into account common words between the source and target documents even if they do not occur in the bilingual dictionary. For example, if the source document contains Obama, and the target document as well, this would increase the similarity. In the context of this article, we call this method DB for Dictionary-Based. By this way, the actual used formula is:

sim(d s , d t ) = |w s → w t | + |w t → w s | + 2c st |d s | + |d t | (2)
where c st is the number of identical (common) words that appear in d s and in d t . c st excludes words that are yet taken into account by Equation 1.

The drawbacks of the dictionary based approach are the dependency on a bilingual dictionary, which is not always available, and the necessity to use morphological analyzers for inflected languages. Moreover, word-to-word dictionary translations without considering the context can lead to many errors because of the polysemy, and because the text is considered only as a bag of independent words.

To avoid to use a bilingual dictionary built by hand, it is possible to automatically build it. This approach was used in [START_REF] Etchegoyhen | A portable method for parallel and comparable document alignment[END_REF], which defines a measure based on Jaccard distance between the source and the target documents. As the measure requires that both the documents are in the same language, the target document is translated into the source language by using a translation table obtained by IBM models [START_REF] Brown | The mathematics of statistical machine translation: Parameter estimation[END_REF].

Cross-Lingual Latent Semantic Indexing (CL-LSI) methods

Document similarity can be estimated at term or semantic level [START_REF] Harispe | Semantic similarity from natural language and ontology analysis[END_REF]. Generally, semantic similarity is a measure which quantifies the likeness of documents based on the latent semantic of the contents. Semantically related terms are usually referred as concepts. Semantic similarity can be measured based on a predefined ontology, which specifies the distance between concepts, or can be measured using statistical methods which find correlations between terms and contexts in a text corpus. Numerous methods in this latter category exist in literature; in [START_REF] Vulić | Probabilistic models of cross-lingual semantic similarity in context based on latent cross-lingual concepts induced from comparable data[END_REF], a list of cross-lingual word similarity is given. One of these methods, which we use in this article, is Latent Semantic Indexing (LSI).

Documents and words in LSI are projected into a reduced space using Singular Value Decomposition. Similar words and documents are mapped closer to each other in the LSI space. More precisely, the documents are represented by a term-document matrix in which each cell contains the frequency of a word into a document. This sparse matrix is rewritten into a term-to-document and a document-to-term matrices, and these matrices are reduced into a more little space by Singular Value Decomposition. Then, it is possible to project every unseen document into the LSI space and to represent the document by a numerical vector. The similarity of two documents is defined by the cosine-based distance between their LSI vectors. The method is parameterized by the size of the LSI space, which is called number of Topics. The LSI approach is extended to the Cross-Lingual domain (CL-LSI) by considering a document as the concatenation between source and target documents [START_REF] Littman | Automatic cross-language information retrieval using latent semantic indexing[END_REF]. For more details see [START_REF] Saad | Mining Documents and Sentiments in Cross-lingual Context[END_REF].

The authors in [START_REF] Blei | Latent Dirichlet Allocation[END_REF] extended LSI to a probabilistic version called Latent Dirichlet Allocation (LDA). In LDA, documents are represented as random mixtures of latent topics, these topics are probabilistic distributions over words. Each document can be perceived as a mixture of different topics, and every topic is characterized by a distribution over words. LDA is useful for modeling topics that are mentioned in a corpus while LSI is useful to map similar documents and words in a corpus into a reduced feature space (model concepts) [START_REF] Cui | Textflow: Towards better understanding of evolving topics in text[END_REF]. Among all latent semantic works, we use LSI for cross-lingual similarity measure because the aim is to map similar documents and words across languages closer to each other into a reduced feature space. In other words, we aim to model concepts rather than topics.

Data sets

To evaluate the different approaches, we decided to test on two kinds of corpora. The first one is made up of Wikipedia data and is provided by the BUCC 2015 evaluation campaign. We used it because this corpus is used by the research community. But, our focus is news data. Therefore we used also two other corpora coming from news agencies. The first one is made up of Euronews website content and has been collected in a previous work [START_REF] Saad | Extracting Comparable Articles from Wikipedia and Measuring their Comparabilities[END_REF]. The multilingual document pairs from Euronews were easy to collect because there are hyper links between documents about the same news. The second corpus uses two different news agencies: BBC for English, and Al Jazeera for Arabic. Given that BBC news articles and Al Jazeera ones are not connected by hyper links, the task of collecting them was not so simple. We describe in Section 5 the method we use to align English BBC news and Arabic Al Jazeera articles.

Wikipedia corpus

The Wikipedia data used in our experiments, consists of pairs of articles in French and English provided by BUCC 2015. Table 1 gives statistics about this corpus. In this table |D| is the number of documents in the corpus, |W | is the number of words in the corpus, and W is the average number of words per document.

Wikipedia corpus (228k documents pairs) is divided into two parts. We use more than 113k French-English couples for training the method based on CL-LSI, and 114k for test. We did not use a development corpus for Wikipedia because we did not tune parameters on this data. Each French document is compared to each English document of the test corpus. To overcome the issue of the complexity of comparison, we decided to sample the source (French) corpus by selecting 1,003 representative documents. Then, each document of the source sample has been compared to every article of the target corpus (114,802 articles).

Euronews comparable corpus

Euronews data was collected in a previous work [START_REF] Saad | Extracting Comparable Articles from Wikipedia and Measuring their Comparabilities[END_REF]. Euronews is a multilingual news TV channel available in many European languages as well as in Arabic. For the three languages and for the development and the test corpus, 3,345 documents have been used, while 27,752 documents served for the training. Details are given in Table 2. For an explanation of the notations used in this table, see Section 4.1. In a previous work, we collected comparable corpus by aligning two different news sources: BBC and Al Jazeera news [START_REF] Saad | Mining Documents and Sentiments in Cross-lingual Context[END_REF]. The challenge is less straightforward than for Wikipedia or Euronews because hyper links between comparable articles do not obviously exist for BBC and Al Jazeera. In the next sections we present the alignment method and statistics on the obtained corpus.

Alignment method

First, we crawl BBC and Al Jazeera websites to collect news articles published in 2012 and 2013 using HTTRACK tool4 . The data of BBC-Al Jazeera corpus are then split into several sub-corpora. Each sub-corpus is composed of news articles that are published in a given month. Consequently, we obtain 24 sub-corpora for each language as shown in Figure 1. The number of articles in each month is between 70 and 300. Then, we use the Cross-Lingual Latent Semantic Approach [START_REF] Littman | Automatic cross-language information retrieval using latent semantic indexing[END_REF][START_REF] Saad | Mining Documents and Sentiments in Cross-lingual Context[END_REF] to measure the similarity between Arabic and English documents. The LSI model is trained on Euronews corpus (Arabic-English part). For each month, we measure the similarity between each Arabic document and the English documents of the same month. This leads to numerous Arabic-English pairs with a similarity score for each one.

We have now to select the correctly aligned documents from these pairs. We decided that two articles are correctly aligned if and only if they share the same exact news. More precisely, two articles focusing, for example, on earthquakes are correctly aligned if this is the same earthquake in the same country, the same day. It was not possible to check manually each pair of documents. Therefore, this handwork is done on the top-15 article pairs retrieved from each of the 24 months (e.g. 360 pairs). Only 305 out of 360 were aligned correctly and have been kept for the further experiments. In the next section, we present statistics about the obtained comparable corpus.

As for Wikipedia and Euronews corpus, we give statistics on this corpus in For 2013, the maximum value is 0.89 and the minimum is 0.26. It should be noted that the similarity intervals are close to each other for all months in 2012 and 2013, except for January, February, April and May 2013. This is probably due to the nature of the items harvested for each month, where the crawler may miss some items in the collection process. Furthermore, Figure 3 shows that min-max values vary from a month to another. This is why we decided to choose the top-n similar articles rather than setting a threshold for the similarity value. The accuracy of correctly aligned documents is 85 per cent. We carried out more investigations about misaligned articles during the validation process. We found that they are all related to the same topic, but they are not related to the same news story or event. The investigation reveals that some of these articles are misaligned despite their similarity is high. The reason is that they are related to the same kind of events, but these events happened in different countries. For instance, one of misaligned news articles was related to elections, but the English article was related to elections in Bulgaria while the Arabic article was related to elections in Pakistan. We conducted a search for elections in Bulgaria in Al Jazeera collection, but we could not find any news article that is related to this event. We also found that some of these stories are local news that are covered only by one of the two broadcast news: Al Jazeera or BBC. In addition to this, it should also be noted that the crawler sometimes can not harvest all Web pages. That is why, for few months, some news can be found neither in the BBC nor in the collections of Al Jazeera. The similarity values in this figure are divided into intervals. The number of correctly aligned articles increases as the similarity value increases, up to the interval [0.6 -0.7), then it decreases for higher similarity values. The interpretation might be as follows: when the similarity is low, the articles are mostly related to the same topic but not to the same news story. As the similarity increases, the likelihood for the aligned articles to be related to the same news story increases up to a certain value, then it normally decreases again. This is because it is unlikely to find related articles written at the same time by BBC and Al Jazeera, and that have a high similarity value.

Experiments on cross-language information retrieval

In this section, we present the results for the two categories of corpus (described in Section 4) as presented in Table 4.

In the experiments, we present the results in terms of recall at rank 1 (@1 per cent) and 5 (@5 per cent) and in terms of Mean Reciprocal Rank (MRR) which is defined below:

M RR = ds∈ C 1 r(dt)

|C|

(3) Where ds is a source document, C is the source corpus, dt is the target document corresponding to ds and r(x) is the rank of document x according to the similarity measure. DB is applied on lemmatised documents because the bilingual dictionary is also lemmatised. For French and English, we used the DELA lemma lists5 . For Arabic, we used the approach we proposed in a previous work [START_REF] Saad | Mining Documents and Sentiments in Cross-lingual Context[END_REF].

For the dictionaries, we used Euradic Sci-Fran6 for French-English and a dictionary extracted from Multilingual WordNet for Arabic-English. The Euradic Sci-Fran dictionary contains 71k French entries (with 2.1 English translations per entry in average) while the Arabic-English one contains 12k Arabic entries (with 6.5 English translations per entry in average).

The CL-LSI method does not require lemmatisation. But, for verification purposes, we tested CL-LSI on lemmatised and non-lemmatised data. Performance was better with a non-lemmatised corpus. Therefore, in the following experiments, we present only results with such data. Moreover, for this method, we only kept the words occurring more than 3 times.

Experiments on multi-lingual documents extracted from single source

This section presents the performance of HAPAX, DB and CL-LSI achieved on the three corpora presented in the previous sections.

Table 5 shows the performance of the methods on Wikipedia data. Since there is no development corpus and because we decided to be in the same test conditions such as in BUCC2015, no tuning has been done. For the LSI method, 300 topics have been empirically chosen (the same value as in [START_REF] Saad | Mining Documents and Sentiments in Cross-lingual Context[END_REF]).

Table 5 shows that the DB method outperforms the two others whatever the measure. CL-LSI method performance is similar to HAPAX one, except for @5. Performance for the three methods is quite low. These results are explained by the fact that each source document is compared to 114k target documents. the diversity of target documents could mislead while retrieving the relevant document.

Results on Euronews data are given in Tables 6 and7 for respectively French-English and Arabic-English.

For HAPAX method, we present the results for the French-English pair because this method is based on the count of the same words that do exist in documents of both languages. For Arabic and English, this method is not applicable because they use different scripts.

The results, in comparison to those given for Wikipedia, are higher. Indeed, this was predictable due to the fact that, for this data set, a source document is compared only to 3,345 articles, which in fact reduces the ambiguity.

The number of topics for CL-LSI has been tuned on a development corpus; the best performance is achieved with 1,000 topics. The results obtained by CL-LSI are better than DB and HAPAX for all the metrics.

For DB method we propose an improvement to boost its results. Upgrading the method consists in selecting the best words of a document in accordance to their tfidf value. In fact, instead of comparing every word of the source document to the words of the target documents, only these representative words are taken into account. In our experiments, these best words are determined by selecting a percentage of the document's vocabulary. This percentage is optimized on a development corpus. The use of a sub-part of document's vocabulary (47 per cent) allows DB to be improved by 11.5 per cent, which outperforms CL-LSI at @1.

We propose also to linearly combine DB with HAPAX in order to improve the results. The weights of this linear combination are also tuned on the development data. Linearly combining DB with HAPAX (with a weight equal to 0.4 for HAPAX) led to an increasing of 13.7 per cent. This result was unexpected because HAPAX uses limited information in documents. For the data set Arabic-English, CL-LSI obtained the best performance with 1,250 topics. For Arabic, the use of CL-LSI shows a real jump in the performance in comparison to DB method. In fact, CL-LSI exceeds DB by 86.5 per cent. The use of a sub-part of document's vocabulary (71 per cent) allows DB to be improved by 23.9 per cent, but the performance remains lower than CL-LSI. This is probably due to the fact that the Arabic-English dictionary does not cover all words necessary to DB method. When the bilingual dictionary has a larger coverage as the one used for French-English data set, CL-LSI achieves better results than DB but the improvement is not so spectacular. We will investigate this hypothesis in concluding remarks just after experiment sections.

Experiments on multi-lingual documents extracted from multiple sources

In the previous experiments, tests have been achieved on documents extracted from the the same source. In the following experiment, we would like to test the methods on multilingual data extracted from different sources: BBC for English and Al Jazeera for Arabic.

As the BBC-Al Jazeera corpus is very small, 305 documents retrieved automatically and checked by hand (see Section 5), we used this whole corpus as test corpus. The CL-LSI models used here are trained on Euronews (such as in Section 6.1). We tested on the test corpus several values for the number of topics in CL-LSI. The results for each number of topics are given in Table 8. On the 305 documents, 750 topics led to the best performance.

Table 9 shows the performance of the methods on BBC-Al Jazeera data. We can remark that the results of DB method have collapsed. The use of sub-parts of document's vocabulary allows to partly counteract this performance, but CL-LSI remains higher than DB. Even if the performance of CL-LSI decreased also, it could be considered as more robust than DB since the decreasing is not as dramatic.

Concluding remarks about results

The previous results show that the performance depends strongly on the number of target documents a source article is compared to. For example, for DB (French-English), the performance in terms of @1 is equal to 59.12 for Wikipedia (with more than 100k target articles), and is equal to 75.19 for Euronews (with more than 3k documents). This result seems obvious because the more documents to be compared, the more risk of ambiguity there is between articles. The language, also, has a strong impact on performance. DB performance in terms of @1 is 74.14 for Euronews French-English, while it is 37.53 on Euronews Arabic-English. This is certainly due to the weak coverage of the bilingual dictionary since Arabic is strongly agglutinative. Indeed, Table 10 shows that the relative performance for DB is correlated with the dictionary coverage: for the French-English pair, the dictionary has a better coverage than for the Arabic-English pair. A method independent from a bilingual dictionary such as CL-LSI is more robust, despite the fact that its results for Arabic decrease slightly in comparison to French.

Several analyzes may be done on the results of Tables 6 and7. Concerning the pair of languages French-English of Euronews, where the comparability is measured on a corpus from the same source, we can mention that the best results are provided by DB 47 per cent at @1. This percentage means that only 47 per cent of the most representative words of the document have been kept for the comparison (as explained in Section 3.2). Several rates have been tested, but only the total number and the best one have been reported in Table 6 . The results of LSI method used with 1000 topics are low in comparison to DB, this is probably due to the fact that, texts coming from the same source, it is easier to identify the comparable multilingual documents only by using dictionaries. HAPAX leads to bad results, since this method is based only on common words between two languages from close languages.

Conclusion

In this article, we proposed a method for collecting, retrieving and aligning comparable documents. We used the CL-LSI approach in order to compare documents from two different sources: BBC and Al Jazeera. This allowed to extract Arabic-English couples of documents which have been checked by hand and this led to 305 comparable documents.

Moreover, we studied three similarity measures on a comparable documents retrieval task. We experimented these measures on three corpora strongly different in terms of source (encyclopedia, news) and in terms of size. For the dictionary-based similarity measure, we proposed to use only a selection of the document's vocabulary, stemming on the tfidf value of words. The selection of the best words showed a real improvement for DB. Actually, DB performance is improved by 11.5 per cent on Euronews French-English, and by 23.9 per cent on Euronews Arabic-English. Moreover, DB has very low performance (16.07) in terms of @1 on BBC-Al Jazeera data set, but when the selection of the most representative words is performed, the method obtains better results (35.41). HAPAX achieves, in the majority of experiments, the worst performance in terms of @1, @5 and MRR. Nevertheless, it can improve slightly the performance when it is interpolated with DB.

The results obtained by CL-LSI are better than DB and HAPAX for all the metrics and all the corpora except for Wikipedia. CL-LSI showed its strength when the linguistic resources are missing or has a weak coverage. For the data set BBC Al Jazeera, the DB method, even by selecting the vocabulary, is lower than CL-LSI in terms of performance by 29.4 per cent. Consequently, CL-LSI remains robust even on difficult task as the one concerning the identification of comparable documents from two contrasting information sources such as BBC news and Al Jazeera news. In a future work, we will use the methods described in this paper to collect comparable corpora from social networks because this media is now a source of information for journalists. Obviously, other issues will rise due to the particularity of this kind of data.

Figure 2

 2 Figure2shows the accuracy of alignment of the top-15 most similar documents of each month. The accuracy of the alignment is defined as the number of cross-lingual articles that are correctly aligned, divided by the total number of articles in the month.The ranges of similarity values of the top-15 aligned articles for the years 2012 and 2013 are shown in Figure3. The figure illustrates the minimum and the maximum of similarity values for each month. For 2012, the maximum value is 0.86 and the minimum is 0.45.

Fig. 3 .

 3 Fig. 3. Similarity ranges of the top-15 similar documents of BBC-Al Jazeera of the years 2012 and 2013

Fig. 4 .

 4 Fig. 4. Distribution of the 305 documents according to the CL-LSI similarity

Table 1 .

 1 Wikipedia comparable corpus characteristics

		English French
	|D|	228k	228k
	|W |	185M	113M
	W	810	495

Table 2 .

 2 Euronews comparable corpus characteristics

		English French Arabic
	|D|	34k	34k	34k
	|W |	6.8M	6.9M	5.5M
	W	198	200	161
	5 BBC News -Al Jazeera corpus: aligning comparable documents from
		different sources

Table 3 .

 3

	BBC		Al Jazeera
	2012-2013		2012-2013
	split by month		split by month
	Jan.		Jan.
	2012		2012
	Feb.		Feb.
	2012		2012
		CL-LSI
		trained on
		EURONEWS
	Nov.	corpus	Nov.
	2013		2013
		align
	Dec.		Dec.
	2013	Top-n	2013
		BBC-Al Jazeera
		aligned articles
	Fig. 1. Automatic alignment of BBC and Al Jazeera news stories
	Table 3. BBC-Al Jazeera comparable corpus characteristics
		English Arabic
	|D|	305	305
	|W |	246k	110k
	W	805	361
	5.2 Statistic analysis of the corpus

Table 4 .

 4 Language pairs for each corpus

	Category Corpus	Arabic	French
			English English
	non news Wikipedia		X
	news	Euronews	X	X
	news	English-BBC Arabic-Al Jazeera	X

Table 5 .

 5 Results on Wikipedia BUCC2015 data

	Method	@1 per cent @5 per cent MRR
	HAPAX	26.22	32.70	0.30
	DB	59.12	70.99	0.65
	CL-LSI 300 topics	24.00	38.60	0.31
	Table 6. Results on Euronews data (French-English)
	Method	@1 per cent @5 per cent MRR
	HAPAX		30.85	53.15	0.41
	DB		74.14	87.62	0.80
	DB 47 per cent		82.66	94.50	0.88
	DB 47 per cent + 0.4 HAPAX	84.33	95.04	0.89
	CL-LSI 1000 topics		77.46	95.58	0.85

Table 7 .

 7 Results on Euronews data (Arabic-English)

	Method	@1 per cent @5 per cent MRR
	DB	37.53	56.06	0.46
	DB 71 per cent	46.49	68.07	0.57
	CL-LSI 1250 topics	70.00	93.93	0.80

Table 8 .

 8 CL-LSI results on BBC-JSC data (test corpus)

	# topics @1	@5	MRR # topics @1	@5	MRR
	per	per		per	per
	cent	cent		cent	cent
	10 27.87	64.92	0.44	400 47.21	88.20	0.65
	20 34.10	74.75	0.52	500 48.52	88.20	0.65
	30 38.03	78.36	0.56	750 50.16 88.20	0.66
	40 39.67	81.31	0.58	1,000 47.87	86.89	0.65
	50 42.30	83.28	0.60	1,250 46.56	86.56	0.64
	100 46.89	86.56	0.64	1,500 46.23	87.21	0.64
	200 47.87	87.54	0.64	1,750 44.92	87.87	0.63
	300 49.18	87.21	0.66	2,000 46.23	88.52	0.64
	Table 9. Results on BBC-Al Jazeera data (Arabic-English)
	Method		@1 per cent @5 per cent MRR
	DB		16.07	32.46	0.25
	DB 40 per cent	35.41	60.98	0.48
	CL-LSI 750 topics	50.16	88.20	0.66

Table 10 .

 10 Coverage of bilingual dictionaries. DB performance (@1) is given between parentheses.

		Arabic-English	French-English
	Wikipedia		66.6 per cent (59.12)
	Euronews	49.0 per cent (37.53) 74.8 per cent (74.14)
	English-BBC	47.5 per cent (16.07)
	Arabic-Al Jazeera

www.accurat-project.eu

http://www.ttc-project.eu/

www.httrack.com

http://infolingu.univ-mlv.fr/DonneesLinguistiques/Dictionnaires/telecharg ement.html

http://catalog.elra.info/product_info.php?products_id=666


K Smaïli

D Fohr

C E González-Gallardo

M Grega

L Janowski

D Jouvet

A Komorowski

A Koźbia

D Langlois

M Leszczuk

O Mella

M A Menacer

A Mendez

E Linhares Pontes

E Sanjuan

D Świst

J M Torres-Moreno

B Garcia-Zapirain

A First Summarization System of a Video in a Target Language

Keywords: Video Summarization, Speech Recognition, Machine Translation, Text Boundary Segmentation, Text Summarization, Sentence Compression

In this paper, we present the first results of the project AMIS (Access Multilingual Information opinionS) funded by Chist-Era. The main goal of this project is to understand the content of a video in a foreign language. In this work, we consider the understanding process, such as the aptitude to capture the most important ideas contained in a media expressed in a foreign language. In other words, the understanding will be approached by the global meaning of the content of a support and not by the meaning of each fragment of a video. Several stumbling points remain before reaching the fixed goal. They concern the following aspects: Video summarization, Speech recognition, Machine translation and Speech segmentation. All these issues will be discussed and the methods used to develop each of these components will be presented. A first implementation is achieved and each component of this system is evaluated on a representative test data. We propose also a protocol for a global subjective evaluation of AMIS.

Introduction

Nowadays, the information is widely available in different medias: TV, social networks, newspapers, etc. The main difference in comparison to what we had one or two decades before is that people can access to the videos of social networks in foreign languages. When, the video does not necessitate any understanding, there is no main problem. In the opposite, when the information necessitates to understand the language, a human being is limited in terms of mastering foreign language, even if YouTube proposes a rough translation of some contents. In Supported by Chist-Era (AMIS project).

AMIS, a Chist-Era project 6 , the main objective is to make available a system, helping people to understand the content of a source video by presenting its main ideas in a target understandable language. The understanding process is considered here to be the comprehension of the main ideas of a video. We think that the best way to do that, is to summarize the video for having access to the essential information. Henceforth, AMIS focuses on the most relevant information by summarizing it and by translating it to the user if necessary. As a result, AMIS will permit to have another side of story of an event since we can, for instance, have the Russian version of the war in Syria.

Several skills are necessary to achieve this objective: video summarization, automatic speech recognition, machine translation, text summarization, etc. Each output of a a sub-system of AMIS can enrich in upstream or downstream the other modules. That makes AMIS working such as a workflow where the flow refers to the information necessary for a component. In this article, we will present the first result that works such as a pipeline system connecting the output of each component to the input of the next one.

Different components of AMIS

Video Summarization

We designed and developed an operational framework for summarization of newscasts and reports [START_REF] Leszczuk | Video summarization framework for newscasts and reports -work in progress[END_REF]. The framework is designed in such a way, that it allows for easy experimentation with different approaches to video summarization. The framework hosts several high-and low-level meta-data extraction algorithms (referring to our former research conducted within the scope of e.g. IMCOP project [START_REF] Baran | The imcop system for data enrichment and content discovery and delivery[END_REF]) that include detection of the anchor-person, recognition of day and night shots and extraction of low-level video quality indicators. The main summarization processes start with Shot Boundary Detection (SBD). This algorithm helps in prediction whether video is static or dynamic. Also, through SBD we can calculate and compare data per shots instead of frames which is a way more efficient while analyzing video clips over longer durations. The video quality indicators mentioned above are used for calculating coefficient of activity which is a product of two indicators -Spatial Activity and Temporal Activity. These indicators show amount of details appearing on the frame and how dynamic the frame is in comparison to the previous frame, respectively. The coefficient of activity is calculated in two steps -firstly per each frame and then as an average per each shot.

The final summarization is built from shots with higher or equal value of coefficient of activity than average value for entire video.

Speech recognition system

Automatic speech recognition (ASR) and machine translation (MT) are among the AMIS project key technologies for understanding videos. Although these items were developed as separate modules, the goal is to include them more tightly with other modalities and other processes. AMIS project deals with videos in French, English and Arabic languages. Arabic is considered as one of the foreign input language of the videos. That is why an Arabic automatic speech recognition system has been developed in Loria. This system is based on the state-of-the-art methods and is trained on large acoustic and text corpora. For that we developed ALASR: Arabic Loria Automatic Speech Recognition system [START_REF] Menacer | Development of the Arabic Loria Automatic Speech Recognition system (ALASR) and its evaluation for Algerian dialect[END_REF]. A speech recognition system needs at least two components: an acoustic model and a language model. In the following, each component is described by presenting the different steps to train such models. Training necessitates two kinds of data: acoustic and textual, which are presented in the following.

Acoustic model

The development of the acoustic model is based on Kaldi [START_REF] Povey | The kaldi speech recognition toolkit[END_REF] recipe, which is a state-of-the-art toolkit for speech recognition based on Weighted Finite State Transducers [START_REF] Mohri | Speech recognition with weighted finite-state transducers[END_REF]. The ASR system uses 13-dimensional Mel-Frequency Cesptral Coefficients (MFCC) features with their first and second order temporal derivatives, which leads to 39-dimensional acoustic features. For Arabic, 35 acoustic models (28 consonants, 6 vowels and silence) are trained. The emission probabilities of the HMM models are estimated by DNN (namely DNN-HMM). The DNN-HMM are trained by applying sMBR criterion [START_REF] Veselỳ | Sequence-discriminative training of deep neural networks[END_REF] and using 40 dimensional features vector (fMLLR) [START_REF] Gales | Maximum likelihood linear transformations for hmm-based speech recognition[END_REF] for speaker adaptation. The topology of the neural network is as follows: a 440-dimensional input layer (40 × 11 fMLLR vectors), 6 hidden layers composed by 2048 nodes and a 4264dimensional output layer, which represents the number of HMM states. And finally, the total number of weights to estimate is about 30.6 million.

Language model In Arabic, even in newspapers,several words could be simplified especially at the beginning or at the end by replacing a specific letter by another one or by omitting the hamza symbol. Unfortunately, both writing of a word may exist in a same document. This leads to share a probability over two forms that correspond to the same word. For instance, the word (uses) is the right way to write it, but people could omit the hamza and write it such as:

. Obviously, this is not the only case, several other points have to be treated. Some of them are specific to Arabic and others are used in the majority of other natural languages. Several preprocessing tasks have been done on the corpora before calculating the language model [START_REF] Menacer | Development of the Arabic Loria Automatic Speech Recognition system (ALASR) and its evaluation for Algerian dialect[END_REF]. The training corpus is composed by GigaWord and the speech transcripts. As this data set is unbalanced, a 4-gram language model, for each part of this corpus, has been developed and combined linearly. The optimal weights are determined in order to maximize the likelihood of the development corpus. Due to the memory constraints, the full 4-gram language model has been pruned by minimizing the relative entropy between the full and the pruned model [START_REF] Stolcke | Entropy-based pruning of backoff language models[END_REF].

Machine Translation

A statistical machine translation has been developed in the direction Arabic -English, since Arabic is considered such as the foreign language of the video to translate to a summarized video in English. For training, we use a parallel (Arabic -English) corpus of 9.7 million sentences extracted from United Nation corpus concerning the period of January 2000 -September 2009. A 4-gram language model has been trained on the target language of the mentioned parallel corpus. The vocabulary contains 224,000 words. The development and the test corpus are composed by 3,000 parallel sentences. This component is a statistical machine translation, but in the next version of AMIS, we will use a neural network machine translation [START_REF] Menacer | Is statistical machine translation approach dead?[END_REF].

Text Summarization

Automatic Text Summarization (ATS) is a Natural Language Processing (NLP) task [START_REF] Torres-Moreno | Automatic Text Summarization[END_REF]. The main objective of ATS is to find the most important information from a text source in order to produce an abridged and informative version. In the AMIS project, the ATS component aims to summarize a newscast or report video in French, English or Arabic languages based only on the textual information provided by the ASR or by the MT modules.

ATS is divided in three different families depending on the followed approach to generate a summary: summarization by extraction, summarization by abstraction and summarization by sentence compression [START_REF] Torres-Moreno | Automatic Text Summarization[END_REF].

We implemented mainly the extractive summarization paradigm in the project because it is more robust to external noise like speech disfluencies and ASR errors [START_REF] Bell | A system for automatic broadcast news summarisation, geolocation and translation[END_REF][START_REF] Christensen | From text summarisation to style-specific summarisation for broadcast news[END_REF]. Also, we performed some exploratory experiments using the sentence compression paradigm.

Extractive Text Summarization Extractive Text Summarization (ETS) aims to select the most pertinent segments of the transcribed video based on different criteria like information content, novelty factor and relative position. ARTEX (Autre Résumer de TEXtes), originally developed for French, English and Spanish is an ETS system described in [START_REF] Torres-Moreno | Artex is another text summarizer[END_REF] by Torres-Moreno et al.. Within the AMIS project, a Modern Standard Arabic (MSA) extension has been developed and added to ARTEX, making possible the generation of summaries in all the languages involved in AMIS.

Multi-Sentence Compression Multi-Sentence Compression (MSC) combines the information of a cluster of similar sentences to generate a new sentence, hopefully grammatically correct, which compresses the most relevant data of this cluster. Among several state-of-the-art MSC methods, Linhares Pontes et al. [START_REF] Linhares Pontes | Multi-sentence compression with word vertex-labeled graphs and integer linear programming[END_REF] used an Integer Linear Programming (ILP) formulation to guide the MSC using a list of keywords. Our system incorporates this approach to use the keywords of a transcribed video to guide the compression of similar sentences and to improve the informativeness of summaries Independently of the summarization approach ATS relies on the existence of sentences either to select, reformulate or compress the source text. One main issue to take into account is that in the AMIS project, the source from which a text summary is created is the transcript of a ASR system or its translation; this transcript does not contain any punctuation mark, hence sentences are nonexistent. In order to solve this problem, a specialized module of sentence boundary detection has been developed.

Sentence Boundary Detection Deep research has been done concerning Sentence Boundary Detection (SeBD) of the ASR transcripts covering the three languages of the project: English, French and Arabic. Written text differs from spoken language in the way the writer/speaker expresses its ideas. In spoken language, sentences are not as well defined as in written text, in this context a segment is defined by a sentence-like unit (SU). Well-formed sentences, phrases and single words can be considered SUs [START_REF] Liu | A study in machine learning from imbalanced data for sentence boundary detection in speech[END_REF].

The developed SeBD system uses mainly textual features and convolutional neural networks (CNN) to segment the transcripts and generate SUs [START_REF] González-Gallardo | Sentence boundary detection for french with subword-level information vectors and convolutional neural networks[END_REF]. Arabic SeBD module was trained with a subset of 50 million words of the Arabic Gigaword Corpus.

Besides ATS, SeBD has shown to be of vital importance for other NLP tasks in the AMIS project like NMT. For these reason a stand-alone version of the module has been deployed to be used independently by other AMIS modules.

Audio summarization

Audio summarization task is a new approach to generate speech-to-speech summaries. This is a difficult task, because in the audio signal there is not any linguistic information (words, sentences, etc.). Although low level features such as signal intensity, starting and ending time could be used for extracting and selecting informative segments, reliable methods for automatic extraction of high level features (prosodic and linguistic features) from spontaneous speech have not yet been established [START_REF] Furui | Speech-to-text and speech-to-speech summarization of spontaneous speech[END_REF][START_REF] Zhang | Active learning with semi-automatic annotation for extractive speech summarization[END_REF]. We explore several neural architectures using deep learning in order to find the best features in the hidden layer. These features will allow us to capture the most important abstract structure (linguistic level) from a low level resource (signal).

Global architecture

Several architectures are proposed to summarize a video in a target language. These architectures are presented in Figure 1. The four (SC1, SC2, SC3 and SC4) architectures are different. In SC1, a summary video is achieved directly

Objective Evaluation

In order to appreciate the relevance of the whole system developed, it is necessary to evaluate it globally, but also each component should be evaluated. In the following, we will evaluate each component since the consequences of the weakness of a system could be propagated to the other components.

Video Summarization The framework is evaluated using annotated video sequences. A pool of experts decides which frames are key frames (meaning: very important frames, core of the video) and which have to be in the summary. We use VLC media player to extract frames from single shots. Obviously, this evaluation process is time-consuming and subjective when the key frames are chosen. In order to describe the obtained results, we are calculating Precision, Recall and F1 score for each sequence and algorithm.

We considered video summarization based only on the visual evaluation. Both human and algorithm were provided with video without any additional audio description. Of course a person creating the evaluation could understand some written text appearing on the screen. Our goal was to validate if the summary created by human is similar to the summary created by algorithm, focusing on the visual part only. Such evaluation can be found in literature [START_REF] Gygli | Video summarization by learning submodular mixtures of objectives[END_REF][START_REF] Huang | Automatic performance evaluation for video summarization[END_REF][START_REF] Quemy | Unsupervised video semantic partitioning using ibm watson and topic modelling[END_REF][START_REF] Sharghi | Query-focused video summarization: Dataset, evaluation, and a memory network based approach[END_REF] just to name a few. Nevertheless the summary of news was not considered. In our research we noted that making a reasonable summary for human observer is very difficult. As a consequence comparing to an algorithm is not as precise as for other cases considered in literature.

The first problem we found is the length of summary. In Table 1 the length of summaries provided by a human observer are presented. We can see that the shortest is just 15% of the original video while the longest is 61%. Comparing such different solutions is difficult. The difference comes from very different natures of the video news, which can span from a talking head to a report from a field where there is an action. In order to help with comparing human and automatic summarization, the automatic algorithm has the information about the length of the summary provided by humans.

The automatic and human summaries were compared by precision and recall metrics. We calculated how many frames marked by human were also marked by an algorithm. So true positive means that both human and algorithm marked the same frame. True negative means that both human and algorithm did not mark specific frame. The results obtained for 50 evaluated sequences are presented in Table 2. The obtained results are not very good but even comparing summaries provided by two humans are not much better. The problem is the content, which is already a summary.

Arabic Automatic Speech Recognition : The ASR system for Arabic has been trained on an acoustic corpus of 63 hours (Nemlar [START_REF] Maegaard | Nemlar: Arabic language resources and tools[END_REF] and NetDC [START_REF] Choukri | Network of data centres (netdc): Bnsc-an arabic broadcast news speech corpus[END_REF]). The Language Model has been trained on the GigaWord. The vocabulary is composed of 95k words with an average of 5.07 pronunciations for each entry. After several tests, tuning and improvements ALASR achieves the performance presented in Table 3. This performance is achieved on a tuning and a test corpora of 31,000

Dev WER Test WER ALASR 13.07 14.02

Table 3. Performance of ALASR in terms of WER sentences for each of them. This test is done on data not extracted from our video database. The issue is that we do not have any reference transcription corpus for these videos. The evaluation is then impossible. To overcome this problem, we decided to build a pseudo-reference by aligning, for each YouTube video from Euronews channel, the automatic transcription and textual data from the corresponding Youtube and Euronews webpages. The transcription is considered as a reference if the WER is under a chosen threshold [START_REF] Jouvet | Adaptation of speech recognition vocabularies for improved transcription of youtube videos[END_REF].

Experiments have been done on a corpus of 1,300 sentences (a mixture of transcriptions from YouTube and Euronews). We have to notice, that this transcription does not correspond exactly to what has been pronounced. Consequently, the performance we provide below is under-estimated. Under these conditions, ALASR achieves a WER of 36.5.

Machine Translation The machine translation system has been evaluated on separated corpora: on a general one and on data extracted from our video database. In Table 4, we give the result of Arabic-English machine translation system evaluated on 3,000 sentences extracted from a corpus of the United Nations [START_REF] Ziemski | The united nations parallel corpus v1. 0[END_REF]. As for our ASR system, the evaluation on our database is difficult

Test (3k sentences) BLEU 39
Table 4. The evaluation of the Arabic-English MT system.

because we do not have any reference corpus for Machine Translation. To overcome this limit, we decided to create two reference corpora. system when SYSTRAN is considered such as the reference is 9.9, while the results when the translation of Google is considered as the reference is equal to 26.7. To understand the weak performance we get with SYSTRAN as reference, we used Google as a translator. The achieved BLEU for this experience is 12.8. This shows that both Google and our system fails to get good results with a translation produced by SYSTRAN.

Sentence Boundary Detection The SeBD module was approached as a classification task, where the system should decide if a target word corresponds or not to a boundary between two SUs. Table 6 shows the results of a strict evaluation for the Arabic SeBD module over 12M samples of the Arabic Gigaword corpus. The method seems to perform really well concerning the "no boundary" class (NO BOUND); both Precision and Recall achieve a value over 92%. By contrast the performance related to the "boundary" class (BOUND) drops almost 15% for Precision and 32.5% for Recall. The unbalanced nature of the data influences this drop in performance. The "no boundary" class represents the 84% of the samples, against the 16% from the "boundary" class. Further work is in develop to reduce the gap between both classes.

Subjective evaluation

In the previous section, we evaluated each component independently from the others, in this section we propose a method to evaluate the system as a whole.

The method is based on a questionnaire that involves the final users, which is considered as the best indicator of the quality of this type of systems. The evaluators of the summarized videos will be 12 participants in total, 6 in Arabic (3 men and 3 women per language). The inclusion criteria are being 18 years old, with at least high school level, while the exclusion criteria is having understanding problems, reading or writing impairment. The material to be evaluated will consist of 25 videos tackling mixed topics: Politics, Soccer, War, Homosexuality.

The number of summarized videos per user participating as tester is 3. The aim of the designed questionnaires is analyzing the quality of the video summarization with the best precision, taking into account the proposed resources. So, we will include 2 generic questions for all the videos, that can be evaluated from 0 ("Not done") to 4 ("Excellent"), and a set of 3/5 specific questions (depending of the length of the video) with 3 possible answers. In that way, we will know if the summary is understandable and if there is any part out of context, using the generic questions, and if the main ideas of the original video have been gathered by the summarization, using the video specific questions. The assessment data analysis will consist on statistical analysis of questionnaires and the application of some machine learning techniques, if possible for clustering and comparison purposes between genders, languages, etc.

Conclusion

In this article, several research aspects have been investigated through AMIS project. An understanding system of a foreign video has been developed by summarizing the source video. The objective was to capture the main idea of a video and to restore it into English. A whole system has been carried out by implementing several sub-systems, where each of them represents a real scientific challenge. The different sub-systems were assembled to give rise to the first version of AMIS. The system is operational and the results are certainly improvable, but are better than what we expected. We thought that the serialization of these systems had to produce very bad results, but this is not the case. We are currently working on developing more relevant architectures that would probably yield better results. In parallel, each sub-system is subject to regular improvements making the global system better.

Table 1 .

 1 Summary length comparison. The summaries in this table are created by human.

	Video ID Summary length Source length Percentage
	1	205	473	43%
	2	86	187	46%
	3	76	277	27%
	4	69	200	35%
	5	85	186	36%
	6	119	194	61%
	7	41	281	15%
	8	41	233	18%
	Recall Precision F1 Accuracy
	0.13	0.36 0.19	0.36

Table 2 .

 2 Performance of video summarization

Table 5 .

 5 The corpora are composed respectively by the translation of 197 videos of Euronews that correspond to 1,253 sentences achieved by Google and SYSTRAN. It means that we consider the result of the translation of Google respectively SYSTRAN as the references. The results are given in Table5. The achieved BLEU for our MT evaluation on AMIS data.

	System AMIS Google Systran
	AMIS	26.7	9.9
	Google 26.7		12.8
	Systran 10	12.9

Table 6 .

 6 Performance of the AMIS SeBD

	Class	Precision Recall F1
	NO BOUND	0.928	0.963 0.945
	BOUND	0.782	0.638 0.700

http://deustotechlife.deusto.es/amis/

Acknowledgment

We would like to acknowledge the support of Chist-Era for funding this work through the AMIS (Access Multilingual Information opinionS) project. Research work funded by the National Science Center, Poland, conferred on the basis of the decision number DEC-2015/16/Z/ST7/00559.

without using any knowledge on the audio content of the video. The content of the result of this summary is then transcribed thanks to ALASR (our speech recognition system) then translated to English. The result of the translation is integrated such as subtitles to the summarized video. SC2 is an original architecture in which an audio summary is proposed on the audio part of the original video. The result is then transcribed and translated such as in SC1.

SC3 and SC4 are similar architectures, they take benefit from the result of ALASR. The result of this step is then a text in Arabic, then the blocks (Machine Translation + Text Summarization) and (Text Summarization + Machine Translation) are respectively performed on SC3 and SC4.

In this paper, only the first architecture (SC1) is presented.