Monsieur Massih-Reza Amini Professeur

Monsieur Cyril Goutte

Rapporteur Monsieur

Gaël Dias Professeur

Monsieur Laurent Besacier Professeur

Examinateur Monsieur Societe Coffreo

Patrick Gallinari

Georgios Balikas

Massih-Reza R Amini

Twise

Hesam Amoualian

Wei Lu

Eric Gaussier

Marianne Clausel

Reza Amini

Anil Goyal

Prantapa Goswami

Prof Gael Dias

Dr Pierre-Antoine Papon

Dr. Karim Sayabou

ten-year university student period.

Abstract

Text is one of the most pervasive and persistent sources of information. Content analysis of text in its broad sense refers to methods for studying and retrieving information from documents. Nowadays, with the ever increasing amount of text becoming available online in several languages and writing styles, content analysis of text is of tremendous importance as it enables a variety of applications. To this end, unsupervised representation learning methods like topic models and word embeddings constitute prominent tools. The goal of this thesis is to study and address challenging problems in this area, focusing on both the design of novel text mining algorithms and tools, as well as on studying how these tools can be applied to text collections written in a single or several languages.

In the first part of the thesis we focus on topic models and more precisely on how to incorporate prior information of text structure to them.

Topic models are built on the premise of bag-of-words, and therefore words are exchangeable. While this assumption benefits the calculations of the conditional probabilities it results in loss of information.

To overcome this limitation we propose two mechanisms that extend topic models by integrating knowledge of text structure to them. To this end, we begin by assuming that the documents are partitioned in thematically coherent text segments. Then, the first mechanism assigns the same topic to the words of a segment. The second, capitalizes on the properties of copulas, a tool mainly used in the fields of economics and risk management that is used to model the joint probability distributions of random variables while having access only to their marginals.

Through the use of copulas we propose flexible topic models that can model different degrees of dependence between the topics of a segment.

The second part of the thesis explores bilingual topic models for comparable corpora with explicit document alignments. Typically, a document collection for such models is in the form of comparable document pairs. The documents of a pair are written in different languages and are thematically similar. Unless translations, the documents of a pair are similar to some extent only. Meanwhile, representative topic models assume that the documents have identical topic distributions, which is a strong and limiting assumption. To overcome the limitations of this assumption we propose novel bilingual topic models that incorporate the notion of cross-lingual similarity of the documents that constitute the pairs in their generative and inference processes. Calculating this cross-lingual document similarity is a task on itself, which we propose to address using cross-lingual word embeddings.

The last part of the thesis concerns the use of word embeddings and neural networks for three text mining applications. First, we discuss polylingual document classification where we argue that translations of a document can be used to enrich its representation. Using an autoencoder to obtain these robust document representations we demonstrate improvements in the task of multi-class document classification.

Second, we explore multi-task sentiment classification of tweets arguing that jointly training classification systems on correlated tasks can improve the obtained performance. To this end we show how one can achieve state-of-the-art performance on a sentiment classification task using recurrent neural networks. The third application we explore is cross-lingual information retrieval. Given a document written in one language, the task consists in retrieving the most similar documents from a pool of documents written in another language. In this line of research, we demonstrate how adapting the transportation problem for estimating document distances one can achieve important improvements.

Chapter 1

Introduction W e live in an interconnected world where new information technologies enable a fast flow of data and information. As a result of the ongoing growth of the World Wide Web, whose omnipresence in our lives persistently changes the way we make decisions and even behave, data are constantly produced at massive volumes. In this era of big data, the access to large amounts of data together with the ever-faster computing machines and data storage facilities have created huge research opportunities. It is therefore, nowadays, possible to study problems and extract valuable information at a scale and granularity that are truly unprecedented.

Among the different types of the data being made available online, text is arguably one of the most pervasive and persistent sources of information. Content analysis of text, that in its broad sense may refer to methods for studying and retrieving information from documents, has been traditionally achieved by close reading and manually coding the retrieved information. However, the voluminous amounts of data available today make it impossible for a single person or groups of people to manually examine text resources of tremendous size. On the other hand, being able to analyze and understand what is discussed online is a critical task and successfully accomplishing it has a huge potential for several real-world applications.

At the same time, the properties of text that is published online in the World Wide Web continuously change. Recent statistics reported, for instance, that the non-English Web content represents more than half of the information that is available on the Internet. 1 This entails that (i) in order to discover and exchange 1 http://www.internetlivestats.com/internet-users/#byregion 1 knowledge at world-wide scale one needs algorithms and models capable of modeling and mining text beyond English, (ii) English may not be anymore the lingua franca of the Web as the increasing multilingual content pushes users towards consuming content written in their native language(s).

But multilinguality is not the only challenge of today's Web content. The media that people use to express their opinions or publish content also evolve: other than formal documents such as news articles, micro-blogging platforms like Twitter2 are extremely popular and have become ubiquitous today. Their omnipresence, however, poses major algorithmic challenges due to the specificities of the content published there. For instance, tweets, which are short messages of up to 140 characters published on Twitter, pose several problems due to their style as symbols, abbreviations, slung and creative language are heavily used. As such media have largely democratized online content publishing and sharing, analyzing their data is also of great importance.

The above observations motivate a set of requirements when developing modern systems in order to be able to cope with the vast amounts of available data. These requirements, grouped by the particular characteristics of the data, can be summarized as follows:

Rq. 1) Considering the data type i.e., text, as well as the fact that humans produce text in an ordered way following particular morpho-syntactic rules, we need to be able to take advantage of the structure that is inherent in text data. Such structure may be, for instance, the grouping of words in thematically coherent text spans such as sentences or noun-phrases. Moreover, given the variability in the style of text, we need to be able to represent it efficiently.

Rq. 2) Considering the large number of languages online, we need to be able to represent text written in different languages using language independent representations so that different tasks like prediction can be accomplished across languages. Further, these representations should be able to take advantage of the large amounts of multilingual data, especially those that are unlabeled and cheap to obtain.

Rq. 3) Considering the large amount of unlabeled data, one needs to be able to use them in order to develop both more expressive and semantically rich representations and better performing systems.

Text Mining It follows from the above that text mining, the interdisciplinary area that the research of this dissertation belongs to, spans and borrows techniques from the domains of machine learning, natural language processing, information retrieval as well as data mining. The added requirement, imposed by the unprecedented availability of multilingual content, for efficient models that can digest and even benefit from text written in multiple languages is central for the results of the dissertation. Notably, any findings in this domain of (multilingual) text mining have a great potential for improving the performance of applications that affect our everyday lives significantly. Motivated by the above points, our driving force is the importance of text mining techniques towards understanding large amounts of text, possibly written in more that one languages, and providing efficient and effective solutions to problems that are met when dealing with such data. To this end, we propose models as well as algorithmic tools that address various challenges that arise in this fascinating research area.

Thesis statement and overview of contributions

This dissertation is organized in two parts and contributes models, tools and observations to problems that arise in the area of text mining with a focus on multilingual text mining. We build upon effective models and algorithms that address the three main requirements stated above (Rq. 1 -Rq. 3). In particular, the contributions of the dissertation aim at:

i.) Proposing probabilistic topic models that handle one or more input languages.

The models incorporate prior knowledge of text structure in the form of boundaries of thematically coherent text spans.

ii.) Exploiting the rich semantic properties of word embeddings, that are vector representations of words that capture their semantic and syntactic properties. They can be used as another source of prior knowledge for (multilingual) topic models or as compact text representations in order to improve the performance achieved for various natural language processing tasks.

Distributional Hypothesis

The shared idea that links the models of the dissertation stems from an insight which was perhaps first formulated by Harris [START_REF] Zellig | Distributional structure[END_REF] who suggested for linguistic items that "If A and B have almost identical environments . . . we say that they are synonyms." Another, perhaps more famous, statement of this principle was formulated from Firth [START_REF] Rupert | A synopsis of linguistic theory[END_REF]: "You shall know a word by the company it keeps". These statements describe the distributional hypothesis, that suggests that linguistic items with similar distributions have similar meanings. Distributional methods build on the distributional hypothesis and propose ways to compute the meaning of a linguistic item using the distribution of words around it. There is a plethora of computational models implementing distributional methods. Both the topic models used for (i.) and the word embedding models used for (ii.) belong in this family as they use word co-occurrence statistics to learn efficient word and document representations. Their difference lies on how they model co-occurrence as well as on the computational means they use to induce the representations. Topic models use a probabilistic framework and aim at modeling an underlying generative story which dictates a set of conditional independences between random variables which enables Markov Chain Monte Carlo inference algorithms for inference. Meanwhile, popular models for learning embeddings rely on a supervised prediction task. We will elaborate more on the similarities and differences of those models and we will evaluate the performance of their extensions at different tasks either when the documents of a collection are written in a single or more languages.

The dissertation contributions regarding point (i.) constitute the first part of the thesis. We utilize linguistic tools like shallow parsers and statistical tools like copulas to extend probabilistic topics models by incorporating parts of text structure. In particular, for the former point we show how to identify thematically coherent groups of words using either linguistically motivated tools or statistically motivated approaches. This results in a hierarchical document representation where a document is decomposed as a set of coherent segments; further each segment is a set of words. In both cases, by taking advantage of observations concerning which words often co-occur, we manage to better model and uncover the topics discussed in a collection. Such knowledge can be then used within a plethora of applications including text classification, document retrieval as well as collection exploration and visualization.

The dissertation contributions regarding point (ii.) constitute the second part of the thesis. We propose models and algorithms that utilize the expressiveness of word embeddings for the benefit of information retrieval and natural language processing applications. We argue that several applications can benefit from the ability of embeddings to capture the semantics of words and we demonstrate how this can be accomplished for the tasks of text classification of short and long documents, polylingual text classification where one has access to translations of a document and cross-lingual document retrieval.

A third, shorter part of the work performed during the thesis is put in Appendix A. It details algorithmic tools for fast model selection for regularized classification by exploiting unlabeled data. While the main thesis is devoted to models built on the distributional hypothesis, that concerns the word co-occurrence in the document level, this last part of work models properties in the collection level. In particular, we utilize the distribution of document categories within a collection and unlabeled data that are usually cheap to obtain to accelerate model selection and hyperparameter tuning for classification models that use regularization like Support Vector Machines. To this end, we observe that the assumption of identically and independently distributed documents (i.i.d.) between the training and the test parts of a collection, which is common in several document classification settings, may be used to accelerate model selection. Being able to estimate the distribution of categories in unseen documents, motivates learning theory bounds that, in turn, accelerate the process of hyperparameter tuning. Therefore, the work explores how the distribution of categories on unlabeled data can be approximated and evaluates the proposed bounds for model selection.

Next, we provide an overview of the contributions of the dissertation following the above points. Meanwhile, Figure 1.1 provides an overview of the thesis.

Extending Probabilistic Topic Models

Incorporating text structure to probabilistic topics models. Given a document segmentation denoting the partition of a document to coherent text spans, how can probabilistic topic models be extended to incorporate that knowledge?

The exchangeability assumption in topic models like Latent Dirichlet Allocation (LDA) often results in inferring inconsistent topics for the words of text spans like noun-phrases, which are generally expected to be topically coherent. We propose segmentLDA and copulaLDA, two novel topic models that extend LDA by integrating part of the text structure and relax the conditional independence assumption between the word-specific latent topics given the per-document topic distributions. The novel models assume that the words of text spans like noun-phrases are topically bound. The former model (segmentLDA) forces all words within a segment to be assigned to the same topic and the binding between topics within Text Structure and Topic Models [START_REF] Balikas | On a topic model for sentences[END_REF][START_REF] Balikas | Modeling topic dependencies in semantically coherent text spans with copulas[END_REF] Text Structure and Multilingual Topic Models -under review-Multilingual Topic Models and Word Embeddings -under review-

Extending Probabilistic Topic Models

Polylingual Text Classification [START_REF] Balikas | Multi-label, multi-class classification using polylingual embeddings[END_REF] CLIR with Word Movers Distance -under review-

Multitask short text classification [START_REF] Balikas | Multitask learning for fine-grained twitter sentiment analysis[END_REF] segments is maximal as all word specific topics are equal. The latter (copulaLDA) uses copulas when assigning topics to the words of sentences and is therefore more flexible as the strength of the bound between the topics is controlled by the free parameters of the copulas. To demonstrate the efficiency of the novel topic models we conduct experiments on several text datasets consisting of documents in English.

We assess the quality of the produced topics using the normalized point-wise mutual information scores, the generalization performance of the models measured by perplexity and the learned document representations as inputs to a document classification task. For this purpose we compare for each topic model a variety of segments that can be considered to be coherent. Our analysis reveals the benefits of integrating prior knowledge of text structure in topic models as well as the advantages of having flexible models and segments of various sizes to accomplish that. (Chapter 4).

Bilingual topics models for comparable corpora. Given pairs of documents that discuss the same themes to some extend, how can we extend bilingual topic models to better adapt them for such inputs? Probabilistic topic models like Latent Dirichlet Allocation (LDA) have been previously extended to the bilingual setting. A fundamental modeling assumption in several of these extensions requires the input documents to be exact translations between them. However, this assumption is strong for comparable corpora, which are, in turn, the most commonly available or easy to obtain. In this chapter we relax this assumption by proposing a binding mechanism between the distributions of the paired documents. The strength of the bound depends on each pair's semantic similarity, that we propose to estimate using bilingual word embeddings learned with shallow Neural Networks. We evaluate the proposed method by extending two topic models: a bilingual adaptation of LDA that assumes bag-ofwords inputs and a model that naturally extends those proposed in Chapter 4 in order to incorporate part of the text structure in the form of boundaries of semantically coherent segments. To demonstrate the efficiency of the novel, bilingual topic models we conduct experiments on four bilingual, comparable corpora of English documents with French, German, Italian and Portuguese documents. The obtained results demonstrate the efficiency of our approach in terms of topical coherence measured by the normalized point-wise mutual information, generalization performance measured by perplexity and accuracy in a cross-lingual document retrieval task for each of the language pairs (Chapter 5).

Word Embeddings and Text Mining Applications

Polylingual Text Classification How can translations of a document be used to improve the performance in the task of document classification?

We propose a polylingual text embedding strategy, that learns a language independent representation of texts using Neural Networks. We study the effects of bilingual representation learning for text classification and we empirically show that the learned representations achieve better classification performance compared to traditional bag-of-words and other monolingual distributed representations. Our results also demonstrate that the performance gains are more significant in the interesting case where only few labeled examples are available for training the classifiers (Chapter 6.1).

Multitask Learning for Fine-Grained Twitter Sentiment Analysis How can we improve the performance of short text sentiment classification using information from correlated tasks? How deep neural networks perform in the task?

Traditional sentiment analysis approaches tackle problems like ternary (3-category) and fine-grained (5-category) sentiment classification by learning the tasks separately. We argue that such classification tasks are correlated and we propose a multitask approach based on a recurrent neural network that benefits by jointly learning them. Our study demonstrates the potential of multitask models on this type of problems and improves the state-of-the-art results in the fine-grained sentiment classification problem (Chapter 6.2).

Cross-lingual Document Retrieval Using Word Embeddings How can we use the tools developed for the transportation problem to calculate the distance of documents written in different language and perform cross-lingual document retrieval?

We extend Word Mover's Distance, a recently proposed distance function between documents for the task of cross-lingual document retrieval (CLDR). We show that the metric can naturally incorporate various term weighting schemes and that it benefits from high quality multilingual word embeddings. Using word embeddings that incorporate information from a Knowledge Base we show that our method outperforms state-of-the-art baselines on six CLDR problems by a large margin in terms of evaluation measures like Mean Reciprocal Rank and P@1 (Chapter 6.3).

Quantification-based algorithmic tools

Quantification-Based Bounds for Supervised Model Selection Assuming access to identically and independently distributed data, how can we accelerate the model selection process of regularized classification systems using quantification?

Hyper-parameter tuning is a resource-intensive task when optimizing classification models. The commonly used k-fold cross validation can become intractable in the large scale settings as a classifier should learn billions of parameters. At the same time, in real-world, one often encounters multiclass classification scenarios where only a few labeled examples are available; model selection approaches often offer little improvement in such cases and the default values of learners are used. We propose bounds for classification on accuracy and macro measures (precision, recall, f 1 measure) that motivate efficient schemes for model selection and can benefit from the existence of unlabeled data. We demonstrate the advantages of those schemes by comparing them with k-fold cross validation and hold-out estimation in the setting of large scale classification (Appendix A).

Outline of the thesis

The rest of the dissertation is organized as follows: The next two chapters present the basic concepts and background material that the author believes are essential for the topics discussed in the thesis. In particular,

• Chapter 2 is a concise introduction to probabilistic topics models, and • Chapter 3 presents basic concepts used for representation learning with neural networks.

Then, the next two chapters are devoted to our work and contributions concerning topic models:

• Chapter 4 presents the contributions when integrating parts of text structure to monolingual topic models, while

• Chapter 5 presents an adaptation of bilingual topic models for comparable corpora.

The remaining of the contributions concern the use of word embeddings for text mining tasks:

• Chapter 6 presents work concerning the improvements one may achieve using word embeddings for polylingual classification, multitask classification and cross-lingual document retrieval.

Having presented the main thesis contributions, in Chapter 7, we offer our concluding remarks about the topics that are discussed in this thesis and we describe promising future research directions. Lastly, Appendix A which is self-contained, introduces the task of quantification as well as an algorithmic tool where quantification-based classification bounds are derived whose application accelerates model selection.

Chapter 2

Preliminaries for Probabilistic Topic Models D ocument collections are on the basis of every text mining application.

A collection C is a set of documents: C = {d 1 , . . . , d M } where each document is a set of ordered words d i = (w 1 , . . . , w N ). As the amount of data available today is unprecedented, a fundamental challenge is its analysis, especially when those data are in the form of unstructured text like documents. The challenge therefore decouples in being able to organize large amounts of text data without requiring manual labor, as the later is expensive and scales poorly for such problems. To this end, efficient algorithms that can describe the themes discussed in documents without any type of annotations are important.

Given a document collection, the first problem that naturally arises is how one can describe it in a vectorized format that would be suitable for further calculations. The vector space model [START_REF] Salton | A vector space model for automatic indexing[END_REF] (VSM), is an algebraic model for representing text as a set of vectors of identifiers. The identifiers aim at modeling the occurrence (or absence) of terms in a given document. According to the VSM, these identifiers are the indexes of the terms of the vocabulary of the collection and their values are each term's frequency.

To better describe the application of VSM to documents and collections, Figure 2.1 shows an example. The text excerpt is vectorized such that the non-zero elements express the amount of information conveyed by the frequency of the vocabulary words within the excerpt. The first element of the vector, for example, models the word "information" and is populated with 3 as "information" occurs thrice in the excerpt. Moving from the document level to the collection level (bottom part of Fig. 2.1) the vectorization process is applied to each of the N documents of the collection. Therefore, instead of a single vector, the output of the collection Information retrieval is the activity of obtaining information resources relevant to an information need...

Document Level

Vectorization 3 The Vector Space Model for a document (above) and a collection of documents (below). A document is transformed to a vector with non-zero indexes corresponding to the frequency of the words associated to these indexes. Repeating the process for each document of a collection creates the term-document matrix.

vectorization process is a matrix, namely the term-document co-occurrence matrix that comprises N vectors. Notice how the bag-of-words assumption is inherent to the vectorization process: the order of the words within documents is ignored. In the example of Figure 2.1, independently of the word order the process would result in vectors whose non-zero elements would always be the same. Also, the produced vectors and the term-document co-occurrence matrix are very sparse. The vector dimensionality equals the size of the vocabulary (V ) of the collection so that the occurrences can be modeled. The sparsity stems from the fact that only few unique words compared to V appear is each document. The vector space model received a lot of attention partially due to the performance improvements on a number of applications produced on its basis. Despite its success there are several limitations. Two of them, that motivate the contributions of this manuscript, are:

i. The inability of the model to capture the semantics of the words. As each word is represented by an index in the resulting vector, the semantic relatedness of the words is not captured. Furthermore, the properties of synonymous or polysemous words can not be modeled.

ii. The exchangeability assumption that results in the bag-of-words representation. As shown above the order of words as well as the way they are grouped in phrases, sentences etc. is lost during the vectorization step and the resulting vectors are of big dimensionality.

As shown above, the resulting matrix that models the occurrence of words in the documents of a collection can be big, noisy and sparse. Given this term-document matrix, one may feel that there should be some structure or pattern in the way that words occur in documents or co-occur with other words. The models presented in this chapter aim at uncovering these patterns. The outputs of the approaches to be presented can be used to estimate semantic similarities of text spans ranging from words to larger text passages.

In the rest, we briefly review basic concepts of topic modeling which will serve as a fundamental theoretical background for the remainder of this thesis. It illustrates how topic models can be seen as computational models that implement the distributional hypothesis. The concepts to be presented are not covered in their entirety as the purpose of the demonstration is to be used as a concise overview which is accompanied by references for further reading. Furthermore, the chapter focuses only on introductory material that the author considers necessary for a deeper understanding of the rest of the text.

The Multinomial and Dirichlet Distributions

We begin with a short overview of probability distributions that will be used when developing the topic models. Along with the definitions of these distributions we introduce the notation that will be used in the rest of the manuscript.

The Multinomial Distribution

One of the important distributions that is extensively used for topic modeling is the multinomial distribution [START_REF] Balakrishnan | A primer on statistical distributions[END_REF]. The multinomial distribution models the outcome of a k-sided dice when rolled n times. It gives the probability of any particular combination of numbers of successes for the various categories, when n independent trials are performed. The outcome of a trial is a success for exactly one of the k categories. If p i is the probability of the outcome i, it is required for each trial that

k i=1 p i = 1.
The multinomial distribution generalizes other distributions in various ways. In particular:

• If for the number of trials one has n = 1 and for the number of outcomes has k = 2, the multinomial distribution is the Bernoulli distribution.

• If n = 1 and k > 2 it is the categorical distribution, which is equivalent to the result of rolling a k-sided dice once.

• If n > 1 and k = 2 it is equivalent to the binomial distribution.

In terms of notation, assuming k outcomes and n trials, Mult(n, p), where p = (p 1 , . . . , p k ) denotes the outcome of the multinomial experiment with probabilities p. From the above, letting Cat(p) to be the result of a draw from a categorical distribution, it follows that:

Mult(1, p) = Cat(p).
The probability density function of a multinomial distribution parametrized by p that estimates the probability of observing the i-th event x i times, when we have n events in total is:

p(x 1 , x 2 , . . . , x k ) = n! x 1 • x 2 • • • x k p x 1 1 • • • p x k k , with k i=1 x i = n.

The Dirichlet Distribution

The Dirichlet distribution [START_REF] Balakrishnan | A primer on statistical distributions[END_REF], denoted Dir(α) is a family of multivariate continuous probability distributions. It is a probability distribution over the space of multinomial distributions, i.e., to generate data X from a Dirichlet distribution with parameters α = α 1 , . . . , α k you first draw a p ∼ Dir(α), and then draw the data X ∼ Mult(n, p). Therefore, the Dirichlet distribution is a distribution over distributions controlled by α. Compared to the standard multinomial draws, the Dirichlet distribution introduces an extra layer with parameters α that controls the probabilities p according to which the data X are generated. The probability density function of the Dirichlet distribution is

p(x 1 , . . . , x k |α) = 1 B(α) k i=1 x a i -1 i , where x 1 , . . . , x k > 1, k i=1 x i = 1,
where B(α) is the multivariate Beta function that is equivalent to:

B(α) = n! k i=1 α i ! , where n = k i=1 α i .

Conjugacy between the Multinomial and Dirichlet distributions

The Dirichlet distribution is the conjugate prior of the multinomial distribution.

This means that if the prior distribution of the parameters p of a multinomial follow is Dirichlet, then the posterior distribution is also a Dirichlet. This has the benefit of making the posterior distribution easy to calculate, and the Multinomial-Dirichlet conjugates are commonly used for topic modeling. To highlight this, let (p 1 , . . . , p k ) be the multinomial parameters and assume that they are sampled from:

(p 1 , . . . , p k ) ∼ Dir(α 1 , . . . , α k ),
prior to having access to data observations. After observing data X = (x 1 , . . . , x k ), for these (p 1 , . . . , p k ) where x i denotes how many times event i occurred, the parameters p for our beliefs may be updated as:

(p 1 , . . . , p k )|X ∼ Dir(α 1 + x 1 , . . . , α k + x k ).
This signifies that the data observations update our beliefs and this is modeled having the data to be pseudo counts added to the Dirichlet parameters α.

Another important comment concerns the role of the α parameters in the samples of the Dirichlet. The smaller the values of α, the sparser the obtained sample is. Therefore, in the case of the Multinomial-Dirichlet example demonstrated above, if most of the elements of α are α i 1, then the sampled values (p 1 , . . . , p k ) will be sparse, which means that only a few events will have high probability p i and the rest of them very low. This property is important for topic models, as this is how sparsity is imposed.

From the term-document co-occurrence matrix to probabilistic topic models

As we have previously discussed, the term-document co-occurrence matrix is an approach to represent the occurrence of words in the documents of a collection.

We have argued that this matrix can be big and noisy and one may expect that some patterns occur in it. In this section we introduce approaches previously proposed to reveal parts of the structure of this matrix whose goal is to model the meaning and the semantics of words as well as the thematical content of the documents that these words occur to.

One may identify two families of approaches. The models of the first try to decompose a word-context matrix by relying on matrix decomposition methods.

Popular models of this family include Latent Semantic Analysis (LSA) [START_REF] Thomas | A solution to plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge[END_REF] and the Hyperspace Analogue to Language [START_REF] Lund | Producing high-dimensional semantic spaces from lexical co-occurrence[END_REF]. The second family of methods was mainly motivated by the success of models like LSA. These models alleviate some of the limitations of LSA and also provide more interpretable outputs. To this end, they rely more on probabilistic groundings and utilize latent variables to model the latent themes that are assumed to generate the documents of a corpus. Popular models of this family are the Probabilistic Latent Semantic Analysis (pLSA) [START_REF] Hofmann | Probabilistic latent semantic indexing[END_REF] and the Latent Dirichlet Allocation (LDA) [START_REF] David M Blei | Latent dirichlet allocation[END_REF]. They take as input the documents of a collection and return a number of topics that can be used to semantically describe their content.

The tools used by the models of the first family are inspired by Linear Algebra or geometry, while the second family of probabilistic models employs Bayesian approaches. Both families of approaches allow for calculating the similarity between terms: spatial models compare terms using distance metrics in a high-dimensional space, while probabilistic models measure similarity between terms according to the degree to which they share the same topic distributions [START_REF] Cohen | Empirical distributional semantics: methods and biomedical applications[END_REF]. In the rest of the section, we briefly review LSA (Section 2.2.1) and pLSA (Section 2.2.2) and then describe the Latent Dirichlet Allocation (Section 2.2.3) in more detail.

Latent Semantic Analysis

Latent Semantic Analysis (also known as Latent Semantic Indexing) [START_REF] Thomas | A solution to plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge[END_REF] is an algebraic method used to analyze the term-document co-occurrence matrix. For LSA the context of word is the document it appears (the term-document matrix is used), while this varies for other methods: for the Hyperspace Analogue to Language the context is the surrounding words and instead of the term-document matrix a term-term matrix is used whose frequencies are calculated using a sliding window.

The goal of LSA is to find a low-rank approximation of the term-document cooccurrence matrix which results in the combination of some of the dimensions that may depend on several terms. Given a collection C = {d 1 , . . . , d M } whose vocabulary is V = {w 1 , . . . , w N }, the term-document matrix is X ∈ R M×N and x i,j denotes the number of the occurrences of word w j in d i . X can also be transformed as a result of the application of a term-weighting scheme like the term frequency-inverse document frequency (tf-idf) scheme [START_REF] Sparck | A statistical interpretation of term specificity and its application in retrieval[END_REF]. As rows of the matrix represent documents, the dot product between them calculates document similarity i.e., the more common terms two documents contain the more similar they are. Further, column product represents word similarity i.e., terms that occur in the same documents should be similar. The fundamental part for LSA is the application of the truncated SVD algorithm that approximates X with the product of three other matrices:

X ≈ U t Σ t V T t , ( 2.1)
where t denotes the number of the largest singular values that are kept. For more details on SVD we refer the interested reader a Linear Algebra textbook (e.g., [START_REF] Strang | Introduction to linear algebra[END_REF]). Due to the decomposition that results in the combination of some dimensions, in the reduced dimensional space a term can be calculated to be similar with others if they have occurred in similar contexts, regardless of whether those contexts are in the same documents. One drawback of LSI is that it lacks in terms of a solid probabilistic foundation and it may be difficult to interpret the resulting word representations [START_REF] David M Blei | Latent dirichlet allocation[END_REF].

Probabilistic Latent Semantic Allocation

Probabilistic Latent Semantic Analysis (pLSA: also known as probabilistic latent semantic indexing pLSI) [START_REF] Hofmann | Probabilistic latent semantic indexing[END_REF] is another model that can be used to analyze the term-document co-occurrence matrix. The model has evolved from LSA and instead of relying on a matrix decomposition approach like SVD it is based on a mixture decomposition. Latent topics are assumed to have generated the collection's documents and the goal is to identify them. PLSA can be considered as a generative model, although strictly speaking it is not one [START_REF] David M Blei | Latent dirichlet allocation[END_REF] due to its inability to model unseen documents.

The model assumes that topics are distributions over the words of the vocabulary of a collection and are modeled as multinomials. This means that given a particular topic k, identified by the value of the random variable z, a word w has conditional probability of occurrence such as:

p(w|z = k) = φ k,w and V w=1 φ k,w = 1.
For instance, word like "ball" and "athlete" would have higher probability given a topic "Sports" than given a topic "Science".

Further, the model assumes that each document is associated with a distribution over the topics of a collection. For the conditional probability of a topic k in a document d one has:

p(z = k|d) = θ d,k and K k=1 θ d,k = 1.
The generative process for the documents of the collection is then:

• For each document d:

-For each word position i within d:

Choose a topic z ∼ Mult(1, θ d ) Choose a word w ∼ Mult(1, φ z )
Then, the probability of each occurrence of a word in a document, that is the probability of a non-zero element of the term-document matrix, is modeled as a mixture of conditionally independent multinomial distributions:

p(w, d) = K z=1 p(z)p(d|z)p(w|z) = p(d) k p(z|d)p(w|z), ( 2.2)
where k ∈ [1, K] is the topic identifier and z the random variable that denotes the topic. These are two, equivalent formulations of the joint probability p(w, d) of pLSA. Applying the Bayes rule reveals their equivalence [START_REF] Hong | Probabilistic latent semantic analysis[END_REF]:

p(w, d) = p(d) K z=1 p(z|d)p(w|z) = p(d) K z=1 p(d|z)p(z) p(d) p(w|z) = K z=1 p(z)p(d|z)p(w|z).
During inference one needs to estimate the parameters of the model, that is θ d,k

and φ k,w , which can be achieved by applying and an expectation-maximization algorithm [START_REF] Hofmann | Probabilistic latent semantic indexing[END_REF][START_REF] Hong | Probabilistic latent semantic analysis[END_REF]. One pitfall is the lack of parameters for p(d), so we don't know how to assign probability to a new document. Another is that the number of parameters for p(z|d) grows linearly with the number of documents, which may lead to overfitting. To overcome this limitations, Latent Dirichlet Allocation that is presented in the next section extend pLSA by proposing a complete generative story.

Latent Dirichlet Allocation

Latent Dirichlet Allocation (LDA) is probably the most popular and representative topic model today. It was first proposed by Blei et al. [START_REF] David M Blei | Latent dirichlet allocation[END_REF], and its introduction has motivated a large part of research work as well as applications in several domains such as text classification systems [START_REF] David M Blei | Latent dirichlet allocation[END_REF], image processing [START_REF] Gokberk Cinbis | Approximate fisher kernels of non-iid image models for image categorization[END_REF], computational biology analyses [START_REF] Liu | Identifying functional mirna-mrna regulatory modules with correspondence latent dirichlet allocation[END_REF] and countless others.

LDA offers a robust way for identifying the topics and builds on a principled set of assumptions that enable efficient inference algorithms compared to pLSA.

Significant research attempts have been devoted to relaxing the assumption that govern LDA.

The generative story of LDA consists of the following steps:

• for each topic k ∈ [1, K]: sample per-word topic distributions φ k ∼ Dir(β)

• for the document d i , i ∈ [1, M]:
sample the per-document topic distribution θ i ∼ Dir(α)

-for the word position n of d i , n ∈ [1, N i ]
Sample the topic z of the word:

z i,n ∼ Mult(1, θ i )
Sample the term for the word position:

w i,n ∼ Mult(1, φ z i,n )
The generative story of LDA is a process that results in the terms of corpus w i,n partitioned into documents d i . The number of the topics K as well as α, β : α ∈ R K , β ∈ R V that are priors of the Dirichlet per-document and per-words distributions are required for the generative process. First, the per-word topic distributions φ k are sampled for the whole corpus. To achieve that, for each document d i a topic proportion θ i is sampled, and from this topic proportion the document terms are emitted: for each word position a topic z i,n is sampled which denotes the topic that

α θ z w φ β N D K Figure 2.2:
The graphical representation of LDA.

will generate the word and, finally, the word is drawn from that topics per-word distribution φ z i,n .

Complementary to the generative process described above, Figure 2.2 depicts the graphical model of LDA. The plate D denotes the documents of a collection.

For each document the topic mixture θ is sampled from Dir(α). Then, for each word position (plate N ) as topic z is sampled. Given the topic z and the per-word topic distributions φ the terms of the document w are sampled. In the figure, the node denoting the terms is gray because the words of the documents are observed.

Due to the fact that in reality we observe the documents and, therefore, we know their size in words, in the generative story we omitted the step where the size of the document is sampled. In some resources, one may find an additional step where the document size N i is sampled, usually from a Poisson distribution. However, this choice does not affect the inference steps and can be safely omitted. 1We further elaborate here on our comment above that LDA is a mixture model.

Mixture models use a convex combination of some base distributions in order to model the observations. A convex combination refers to a weighted sum over some base observations, whose sum of weighs equals to one. Due to the probabilistic nature of LDA and its flexibility to assign words of a document to different topics, for the topic proportions of a document one has:

K k=1 p(z = k) = K k=1 θ k = 1. (2.3)
Further, for the per-word topic distributions that model the probability of a word given a topic:

K k=1 p(w|z = k) = K k=1 φ z k = 1. (2.4)
As documents are mixture of topics (Eq. (2.3)) and topics themselves are mixtures of word probabilities (Eq. (2.4)) LDA is also referred to as an admixture model.

Admixture denotes mixtures whose basic components are mixtures themselves.

Inference While the generative story and the graphical representation of LDA describe an iterative process that is assumed to have generated a document collection, inference tries to achieve the opposite. Instead of generating the corpus, an inference strategy aims at discovering LDA's parameters when observing the words of a corpus. The parameters of the model are the per-document topic distributions (θ i ) and the per-word topic distributions (φ). Estimating those is equivalent to uncovering the latent themes (topics) of a collection. In particular, given φ one may identify the words with the highest probability for a topic while θ i is a vector representation of d i in the space of the topics. As a result, documents with similar topic distributions are expected to be semantically similar.

The two, most popular inference strategies are variational inference [START_REF] David M Blei | Latent dirichlet allocation[END_REF] and collapsed Gibbs sampling [START_REF] Thomas | Finding scientific topics[END_REF]. We review the details of the collapsed Gibbs sampling approach as it will be further used for inference of the LDA-extensions that will be proposed in the next chapters. Gibbs sampling algorithms obtain posterior samples by sweeping through each block of variables and sampling from their conditional, while the remaining blocks are fixed. In practice, for LDA the algorithm initializes randomly the topics of words. Then, during the Gibbs iterations and until convergence, it samples topics for the words occurring within documents as Multinomial draws. The probabilities of the Multinomial draw for sampling the topic of a word position i where word t is observed are given by [START_REF] Heinrich | Parameter estimation for text analysis[END_REF][START_REF] Thomas | Finding scientific topics[END_REF]:

p(z i = k|z ¬i , w) ∝ Ψ (t) k,¬i + β t V t=1 Ψ (t) k,¬i + β t (Ω (k) m,¬i + α k ) (2.5)
where "¬i" in a subscript of a count variable signifies the exclusion of the counts due to the word position i and, β t , α k are the t-th and k-th coordinate of β ∈ R V and α ∈ R K respectively. Ψ and Ω are count variables, whose content is shown in Table 2.1 that summarizes the notation used for LDA. The Gibbs sampling algorithm is then an iterative process over the words of a collection C, where Eq. (2.5) is applied and a topic for each word is sampled until convergence. Although checking convergence for Markov Chain Monte Carlo approaches is a field of research (e.g., [START_REF] Jun | Monte Carlo strategies in scientific computing[END_REF]), for topic models one may use as criteria how well semantically similar words of documents are clustered. From the count matrices Ψ and Ω one may yield the per-word and per-document topic distributions by normalizing their rows. Inference on unseen documents As LDA has a complete generative story, one may identify the topic distributions of unseen (held-out) documents by performing the Gibbs sampling inference process on those documents. In this case, for the Ψ counters the values that were observed during training are used the model is queried. Typically very few iterations (<10) are needed for the topic distributions of the unseen documents to be inferred [START_REF] Heinrich | Parameter estimation for text analysis[END_REF].

Distributional Semantics

The models presented in the previous sections are popular approaches used to represent the meaning of words or text spans. These methods, however, provide only a quantitative estimate of the semantic similarity (or meaning) between terms that is estimated by operations that quantify vector similarities. This is to differentiate them from other resources like ontologies or controlled vocabularies that have been extensively used to represent meaning or specific types of relationships, which can not be easily achieved by using the outcome of topic models.

Topic models like pLSA or LDA manage to determine the meanings (or semantics) of terms within a collection empirically from the way in which these terms are distributed across the text. From these distributional statistics, it is possible to obtain meaningful estimates of the semantic similarity between terms in an unannotated corpus of text without human intervention. This is why they can be seen as models that implement the distributional hypothesis. Being unsupervised and easily applicable to unannotated text as well as computationally efficient, is a strong advantage of these methods.

Multilingual Topic Models

As more and more multilingual content is becoming available online, there is a pressing need for developing models that can cope with text data written in different languages. Topic models like LDA manage to uncover the latent topics of a corpus and have been used to numerous applications. Their success has motivated research that resulted in bilingual and multilingual topic models [START_REF] Mimno | Polylingual topic models[END_REF][START_REF] Vulić | Probabilistic topic modeling in multilingual settings: An overview of its methodology and applications[END_REF]. These are models that extend the concept of probabilistic topic models in the case where documents are written in more than a single language. The goal is not only to learn consistent topics for each language, but also to learn topics that are aligned across the input languages. Before providing an overview of these models, we first describe their inputs and discuss the concept of parallel and comparable corpora.

Parallel and comparable corpora

The section provides some basic definitions about the properties of multilingual corpora that are important for the rest of the presentation.

Definition 1 A comparable corpus in two or more languages 1 , 2 , . . . is a set of cor- responding text collections C 1 , C 2 , . . .. Each collection consists of documents such that C 1 = {d 1 1 , d 1 2 , . . . , d 1 N 1 }, C 2 = {d 2 1 , d 2 2 , . . . , d 2 N 2
}, . . . that discuss similar topics.

It is not required for the documents of C 1 , C 2 , . . . to have one-to-one explicit alignments, that is the content of d 1 1 to be thematically comparable to the content of d 2 1 etc.. Therefore, it can also be N 1 N 2 .

A characteristic example of a comparable corpus is the set of documents comprising the English and the French Wikipedia. The number of entries for the two languages (or any other two or more languages) varies. It is not necessary for an entry in English to have a counterpart entry in French and vice-versa. However, concerning the themes (topics) underlying the collection, one can safely assume though that at least, to some extent, the English and the French documents cover similar topics like Arts, Science, Geography, . . . due to the nature of Wikipedia. A comparable corpus with explicit document alignments is a special case of a comparable corpus that requires topical alignments between documents written

α θ z 1 z 2 w 1 w 2 φ 1 φ 2 β N 2 N 1 D K K Figure 2.3: The graphical model of bilingual LDA.
in different languages. In the above-mentioned example of Wikipedia articles, one may obtain a comparable corpus with explicit document alignments by only keeping entries that have a counterpart entries in the rest of the languages. Assuming an entry about "Dog", there must be an entry about "Chien" (French for dog) and so forth. is identical to the content of d2 i , . . . . The most natural way to build or obtain a parallel corpus is by translating the documents of a language 1 to one or more languages 2 , . . . Parallel corpora are the most costly to develop as they require either human translations or high quality automatic translation systems. Notably, parallel corpora are very important for different types of applications like machine translation [START_REF] Koehn | Europarl: A parallel corpus for statistical machine translation[END_REF], multilingual topic models [START_REF] Vulić | Probabilistic topic modeling in multilingual settings: An overview of its methodology and applications[END_REF] and plenty others.

Bilingual Latent Dirichlet Allocation

Following the success of topic models like LDA, whose application on monolingual data has enabled various types of applications, topic models for multilingual content were proposed. Bilingual LDA 2 (BiLDA: Figure 2.3) is a direct extension of LDA in the bilingual setting. The input collection is assumed to be either a parallel [START_REF] Zhao | Bitam: Bilingual topic admixture models for word alignment[END_REF] or a comparable one [START_REF] Ni | Mining multilingual topics from wikipedia[END_REF][START_REF] Mimno | Polylingual topic models[END_REF][START_REF] De | Cross-language linking of news stories on the web using interlingual topic modelling[END_REF][START_REF] John C Platt | Translingual document representations from discriminative projections[END_REF]. Its generative story is as follows:

• for each topic k ∈ [1, K]: φ 1 k ∼ Dir(β), φ 2 k ∼ Dir(β)
• for each document pair d i :

sample θ i ∼ Dir(α)

-for each language ∈ { 1 , 2 }
for each of the N i words:

• sample z ∼ Mult(1, θ i ) • sample w ∼ Mult(1, φ z )
It can be seen from the generative story that BiLDA assumes that the documents of an aligned pair

d i = (d 1 i , d 2 i
) have identical topic distributions as there is a single, shared θ i topic distribution per pair. Also, as the model is a direct extension of LDA assumes the documents to be a bag-of-words.

The collapsed Gibbs sampling updates [START_REF] Vulić | Probabilistic topic modeling in multilingual settings: An overview of its methodology and applications[END_REF] for the topic of word j of document

d i is ∀ ∈ { 1 , 2 }: p z 1 ij = z k |z 1 ¬ij , z 2 , w , w 2 , α, β, ∝ Ψ 1 k,w,¬ij + β Ψ 1 k,•,¬ij + V β (Ω i,k,¬ij + α).
Notice that there are two counters, Ψ 1 , and Ψ 2 used to model the per-word topic distributions. Each hold the counts for the respective language. On the other hand, since the documents are assumed to have a shared topic distribution, there is a single Ω counter variable to model the per-document topic distribution. Further, the model handles the documents written in the different input languages symmetrically and, therefore, its extension to more than two languages is straightforward. For inference using the Gibbs sampling iterative method as well as for deriving topic distributions for unseen documents, the same process with that of LDA can be applied.

Summary

The chapter provided a short overview of basic tools and topic models that will be extended in the remaining of this manuscript. We have described the distributional hypothesis which is behind very recent models that will be used later in the thesis for several text mining applications. We also introduced and discussed the Multinomial and Dirichlet distributions that are central for several topic models as prior and posterior distributions. Following that, we provided a brief introduction to some of the most representative Bayesian topic models: we begun with LDA and pLSA before detailing LDA as well as its multilingual extension BiLDA.

Chapter 3

Preliminaries for Neural Networks O ur presentation of topic models like pLSA and LDA as well as their predecessors from the Hyperspace Analogue to Language (HAL) and LSA in Chapter 2 showed that contextual information can be used to generate vectors that successfully model word meaning. We argued that this is achieved because semantically similar words tend to have similar contextual distributions, an idea known as the "distributional hypothesis" first stated in the early 60's [START_REF] Rupert | A synopsis of linguistic theory[END_REF][START_REF] Zellig | Distributional structure[END_REF].

The distributional semantic models of the previous chapter learn word (and document) vectors. Those vectors are obtained by counting how words occur in contexts denoted by documents and applying geometric (LSA) or probabilistic techniques (pLSA, LDA) to the resulting word-document co-occurrence matrices.

Lately, a new family of distributional semantic models have gained popularity.

Instead of relying on word-document co-occurrence matrices factorization, they rely on predicting words [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. The representations learned through those methods are commonly referred to as word embeddings. Embeddings can be efficiently learned using (shallow) neural network architectures. This novel way of training the distributional semantic model and the word representations thereof is attractive because it replaces the transformation steps of the earlier approaches with a prediction step, which is a well-defined supervised learning step. In particular, given a word or a word context, one tries to predict the word context or the word that maximize the performance of the classification task. While the task of trying to predict words or word context may be of low value, the learned embeddings that are a by-product of the prediction task, have been shown to capture interesting semantic and syntactic properties of words. Recall, however, that the idea of learning parameter vectors based on an objective optimum function is also shared by Latent Dirichlet Allocation (LDA) models [START_REF] David M Blei | Latent dirichlet allocation[END_REF], where the per-word and per-document topic distribution are parameters learned to optimize the joint probability distribution of words and documents.

There are three main advantages of word embeddings: (i) the dimensions of the learned representations models can be interpreted as general "latent" semantic properties and therefore simple algebraic operations like the addition encode word properties like gender or word relationships like Country-Capital. (ii) The supervised task is defined in such a way that no annotated resources are required: the contexts can be directly extracted from free text. (iii) Some of the models like those that we will present in the rest of the chapter, scale well for huge amounts of data inputs, which is not directly achieved with probabilistic models like LDA.

LDA on the other hand, has the advantage of learning representations with some cognitive plausibility as the latent topic are shown to capture the themes of a collection, whereas such interpretations are more difficult for word embeddings.

Word embeddings have shown promising results in a plethora of tasks [START_REF] Baroni | Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors[END_REF][START_REF] Pennington | Glove: Global vectors for word representation[END_REF][START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] and constitute another family of models that implement the distributional hypothesis. In the rest of the manuscript we will evaluate the performance of models on top of word embeddings or as a way to incorporate prior knowledge to topic models. Therefore, in this chapter we review popular and high-performing models for training word embeddings also providing a concise introduction to neural networks.

Word Embeddings with shallow Neural Networks

We review here two models, the continuous bag-of-words model and the skipgram model [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF], both released as part of the word2vec1 tool, that have shown to perform well across several tasks [START_REF] Baroni | Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors[END_REF][START_REF] Schnabel | Evaluation methods for unsupervised word embeddings[END_REF][START_REF] Österlund | Factorization of latent variables in distributional semantic models[END_REF] and have, thus, gained a lot of popularity. The models rely on a shallow neural network with a single hidden layer for learning representations of words and short phrases. Naturally, every feed-forward neural network that takes words from a vocabulary as input learns word embeddings: it embeds the vocabulary identifiers into a vector space of dimension lower than the vocabulary cardinality, and those vectors are then finetuned through back-propagation to improve the performance on the task. This first layer is commonly referred to as the "Embedding Layer". Figure 3.1 depicts the embedding layer of a neural network. A word, vectorized using the one-hotencoding scheme, is associated with a dense vector of size D, where D needs to be tuned for a particular task. Since the input vector that encodes the word is of dimension V with a single non-zero element, the word's identifier i, the output of the embedding layer is a vector that corresponds to the i-th line of W . Therefore, W is a matrix that holds the embeddings of the V words.

d 1 . . . d D Output Word Embedding of dimension D . . . 1 V Input One-hot-encoded input vector W ∈ R V ×D
The main difference between such an arbitrary network that learns word embeddings as a by-product of the main task and a method such as word2vec whose explicit goal is to learn the word embeddings is its computational complexity.

Generating word embeddings with a deep architecture is simply too computationally expensive for a large vocabulary. On the contrary, the models presented next use shallow networks and are efficient. As an introductory side-note, embeddings learned with tools like word2vec are generally used as initializations of the embedding layers of networks that solve a task wit text inputs, which is similar on how pre-trained networks like VGGNet [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] are used for computer vision architectures: common weight initializations that generally provide useful features without the need for expensive training. [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. The input of the network is a word, while the output of the network is a softmax layer over the words of a corpus. It follows, that the output layer is of dimensionality V . The task used to train the word embeddings is as follows: given a word, maximize the probabilities of the words in its context. The context (also referred to as window) is up to n words before and after the word-occurrence of the input word in the text. In the example of Figure 3.2 we assume an input sequence of words "the cat sat on the mat". For illustration purposes we do not apply any type of text normalization steps (e.g., stemming). Then, given the word "sits" the network needs to maximize the probabilities of the words of the context (the, cat, on, mat). Notice, that the input layer, which in detail was shown at Figure 3.1, is linear and the only non-linear function is the softmax layer applied in the network output.

The Skipgram Model

In terms of implementation, there are some important details that have been shown to improve the performance of the network if tuned carefully. The window size n is not fixed but dynamic, and it is sampled uniformly from 1 to n, where n is a parameter of the model. The main bottleneck of the model as shown in Figure 3.2, is the softmax layer. Due to the high dimensionality of the output, which is equivalent to V , calculating it analytically is expensive. As an alternative, negative sampling estimates the probability of an output word by learning to distinguish it from draws from a noise distribution. The number of these draws (number of negative samples) is given by a parameter k, usually set to 10 ∼ 15. Negative sampling is very appealing computationally because computing the loss function scales with the number of noise words that we select (k), and not all words in the vocabulary (V ), which accelerates training. Furthermore, to limit the effect of very frequent words like stopwords (e.g., and, the, in . . .), one can discard them during training with a probability that is proportional to their frequency. Since such words are uninformative, subsampling them results at limiting their effect on the learned embeddings and accelerates training as the prediction step is not performed for every occurrence of them. The subsampling is performed creating the windows to be considered during prediction, which entails that the actual windows used may by larger than n.

The cost function J of the skipgram model as illustrated at Figure 3.2 is:

J(θ) = 1 T T t=1 -m≤j≤m,j 0 log p(w t+j |w t ) = 1 T T t=1 -m≤j≤m,j 0 log exp(w T t+j v t ) V w=1 exp(w T w v t ) , (3.1)
where w t is the embedding of the word t in hidden layer and v t the output embedding of the term. Notice that the calculation of softmax output of Eq. (3.1) the summation in the denominator over all the words of the vocabulary is a computational bottleneck.

To overcome this problem, [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] that is an approximation of the Noise Contrastive Estimation (NCE) [START_REF] Gutmann | Noise-contrastive estimation: A new estimation principle for unnormalized statistical models[END_REF][START_REF] Mnih | A fast and simple algorithm for training neural probabilistic language models[END_REF], is a sampling-based approach that tries to approximate the normalization in the denominator of the softmax with some other loss which is cheaper to compute.

Considering a word-context pair, where the context is defined using the window as discussed above, the objective of the skipgram model with negative sampling is as follows. The probability p(D = 1|w, c) stands for the probability the pair comes from the data, while the probability p(D = 0|w, c) = 1p(D = 1|w, c) is the probability that the pair does not come from the data but it is rather a negative sample. Further, one has:

p(D = 1|w, c) = σ (w • c) = 1 1 + e -w•c ,
where the model parameters w, c are to be learned. Negative sampling is a special case of NCE that approximates the probability that a word w comes from the empirical training distribution of the training data given a context c with [START_REF] Goldberg | word2vec explained: Deriving mikolov et al.'s negative-sampling word-embedding method[END_REF]:

p(y = 1|w, c) = e w i ,c 1 + e w i ,c = σ (w i , c).
The objective is to maximize p(D = 1|w, c) for the observed pairs and also maximize p(D = 0|w, c) for the negative samples:

p(D = 0|w, c) = 1 - 1 1 + e -w•c = (1 -1 + e -w•c )e w•c (1 + e -w•c )e w•c = 1 1 + e w•c = σ (-w • c),
which results in the objective function of the skipgram model with negative sampling:

J θ = - w∈V log σ (w • c) + k j=1 log σ (-w j • c) .
While the basis of the skipgram model with negative sampling is the prediction step, it has been shown that in fact this is equivalent to factorization the matrix of the pointwise mutual information (PMI) values of the word-context co-occurrence [START_REF] Levy | Neural word embedding as implicit matrix factorization[END_REF][START_REF] Li | Word embedding revisited: A new representation learning and explicit matrix factorization perspective[END_REF][START_REF] Pennington | Glove: Global vectors for word representation[END_REF] matrix. This implies that depending on how the context of the co-occurrence matrices is defined (context as document/window..) and how the matrix elements are populated (frequencies, PMI values,..) those models are rather different computational means to arrive at the same type of semantic model. The common factor between them is the underlying distributional hypothesis.

The Continuous Bag-of-Words Model

Another popular model for learning word embeddings is the continuous bag-ofwords (cbow) [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. It is very similar to the skipgram model, and it is illustrated in Figure 3.3. The cbow model learns to predict the word in the middle of a symmetric window based on the sum of the vector representations of the words in the window. The words of the window are summed and, as a result their order is not taken into account explicitly. Therefore, the model assumes a bag-of-words representation, that is highlighted in its name.

The example of Figure 3.3 is analogous to that of Figure 3.2. The input sequence of words is assumed to be "the cat sits on the mat". Given the words that surround "sits", the model's goal is then to maximize the probability of that word given as input the sum of the representations of the words "the, cat, on, the, mat".

The fact that the cbow model performs an averaging operation statistically signifies that it smoothes over a lot of the distributional information because it treats an entire context as one observation. This becomes more intense as the window size increases. In the original paper [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] the authors note that this turns out to be a useful thing for smaller datasets. On the other hand, skipgram treats each context-target pair as a new observation, and this tends to do better when we have larger datasets. N maybe an arbitrarily deep neural network, that given an input x 1 , which can be a word embeddings for instance, propagates a hidden state to its successor. The RNN can be very long [START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF].

Text Representations using deep neural networks

The skipgram and cbow models of the previous section learn word embeddings using a prediction task that relies on how words co-occur with their contexts in free text. When more information about a task is available like labeled examples, such general purpose embeddings could be used to initialize the weights of neural networks. Then, allowing the backpropagation gradients to modify the embeddings would result in fine-tuning them for the given task. A shortcoming of the traditional feed-forward neural networks is that they do not model sequences effectively. On the other hand, as text is a sequence of words, being able to capture the dependencies that this suggests may be advantageous for several tasks.

Recurrent neural networks (RNNs) are a family of networks that address this shortcoming. They are deep neural networks: once unfolded they resemble to traditional neural networks with several hidden layers. RNNs contain loops that allow information from step t to be passed to step t + 1. They can be seen as copies of the same network, each passing a message to its successor. We illustrate at Figure 3.4 an unrolled RNN whose inputs are the embeddings of T words. In the figure N is a neural network, typically consisting of a sigmoid hidden layer, although deeper architectures can be considered.

RNNs are appealing as they are supposed to model information until input t and pass it to its successor t + 1. In practice, while RNNs may be able to achieve that when the sequence of inputs is small, when the sequence becomes larger than a few elements they fail [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF][START_REF] Hochreiter | Long short-term memory[END_REF][START_REF] Hochreiter | Untersuchungen zu dynamischen neuronalen Netzen[END_REF] due to the problem of vanishing gradients, which make it difficult to train the network efficiently as the length of the input sequence increases. ) and the input of the sequence that is currently modeled x t . Rectangles denote hidden layers while the algebraic operators and the tanh function in the ellipse are applied element-wise to the vectors that are inputs to the blocks.

The long short-term memory network [START_REF] Hochreiter | Long short-term memory[END_REF] (LSTM) is a popular and state-ofthe-art network that alleviates the limitations of the vanilla RNN networks described above. To manage to encode information from long periods of time, the layers N of LSTM are more complex than those of RNNs. Instead of a single hidden neural network they have four that interact in a particular way in order to encode information from previous states efficiently. Figure 3.5 shows the internals of an LSTM unit.

We provide a brief description of the LSTM unit describing the purpose of the four hidden layers shown at Figure 3 The left-most bottom sigmoid layer is the "forget gate". Given the concatenation of h t-1 and the current input x t , it outputs a vector of values within [0, 1] that are weights quantifying how much of the C t-1 will be kept. Given the notation of the figure we have:

f t = σ (W f • [h t-1 , x t ] + b f ), (3.2)
where the square brackets "[]" applied to two or more vectors e.g., [h t-1 , x t ] denote the concatenation of their values.

While the forget gate decides which information will be removed from the cell state, the input gate decides in a two-step process which information will be added in the cell state. First, a sigmoid layer, whose output is denoted by i t , decides which values will be updated. Subsequently, a tanh layer (whose output is Ĉt ) suggests and candidate values for the state. The candidate values Ĉt are combined with element-wise multiplication with i t and are added to the cell state. Therefore, the calculation of the information to be added in the cell state and the updates of its values are given by:

i t = σ (W i • [h t-1 , x t ] + b i ), Ĉt = tanh(W c • [h t-1 , x t ] + b c ), C t = f t × C t-1 + i t × Ĉt . (3.3)
The last gate calculates the output of the current LSTM unit, which is a filtered version of the updated cell state C t . A sigmoid function is therefore applied to the concatenation of h t-1 and x t and the output is multiplied (element-wise) with the squashed values of C t that are obtained by applying tanh on its elements. Therefore:

o t = σ (W o • [h t-1 , x t ] + b o ), h t = o t × tanh(c t ).
(3.4)

The equations (3.2),(3.3), (3.4) describe the updates performed in each of the gates of the LSTM. The values of the matrices W f , W i , W c and W o as well as the corresponding biases are updated gradient descent and the back-propagation through time algorithm [START_REF] Hochreiter | Long short-term memory[END_REF].

Cross-lingual Word Embeddings

In the previous sections of the chapter we restricted the presentation of models for learning word embeddings of modeling text sequences in a single language. As we discussed in Chapters 1 and 2 however, the amount of multilingual content online steadily increases. In conjunction with the fact that resources, training data, and benchmarks are mostly for English language, which may result in a disproportionate focus or even a bias (complementary to the gender/racer bias observed by [START_REF] Bolukbasi | Man is to computer programmer as woman is to homemaker? debiasing word embeddings[END_REF]) for this language.

To overcome such issues, cross-lingual embeddings aim at learning embeddings for the words of two or more languages that share the same space. The hope is, then, that by projecting examples from a language in this space and training a model, the model will have the ability of predictions for the rest of the languages also. This of course assumes remedies to several problems, like having effective compositional models to that capture the semantics of text spans larger than words.

Recently, several models have been proposed for learning cross-lingual word embeddings. Among them, several approaches build on the successful models proposed for monolingual collections and extend models like the skipgram model with negative sampling in the bilingual or multilingual space [START_REF] Mikolov | Exploiting similarities among languages for machine translation[END_REF][START_REF] Luong | Bilingual word representations with monolingual quality in mind[END_REF][START_REF] Gouws | Bilbowa: Fast bilingual distributed representations without word alignments[END_REF][START_REF] Coulmance | Trans-gram, fast cross-lingual word-embeddings[END_REF].

The models for learning cross-lingual embeddings can be grouped with regard to the type of approach used to align the multilingual embeddings [START_REF] Ruder | A survey of cross-lingual embedding models[END_REF]:

• Monolingual mapping: this family of methods begin by learning monolingual word embeddings and try to learn a linear transformation from one space to another. It was first proposed by [START_REF] Mikolov | Exploiting similarities among languages for machine translation[END_REF] and followed by more works [START_REF] Xing | Normalized word embedding and orthogonal transform for bilingual word translation[END_REF][START_REF] Lazaridou | Hubness and pollution: Delving into cross-space mapping for zero-shot learning[END_REF] trying to relax some of the underlying assumptions.

• Pseudo-cross-lingual: The methods of this family aim at generating pseudocross-lingual datasets where dictionaries are used to replace words with their translations in order to obtain artificial contexts and then train models like skipgram [START_REF] Gouws | Simple task-specific bilingual word embeddings[END_REF][START_REF] Duong | Learning crosslingual word embeddings without bilingual corpora[END_REF] or generate multilingual documents are concatenated and their content is shuffled [START_REF] Vulić | Bilingual distributed word representations from document-aligned comparable data[END_REF].

• Cross-lingual training: the approaches of this family optimize a cross-lingual training loss using parallel, sentence aligned corpora. For instance, use autoencoders to encode sentences or documents in a source language and reconstruct it in another language [START_REF] Lauly | Learning multilingual word representations using a bag-of-words autoencoder[END_REF][START_REF] Balikas | Multi-label, multi-class classification using polylingual embeddings[END_REF].

• Joint optimization of monolingual and cross-lingual losses: the approaches of this family optimize both a cross-lingual and monolingual losses. For instance, [START_REF] Luong | Bilingual word representations with monolingual quality in mind[END_REF][START_REF] Gouws | Bilbowa: Fast bilingual distributed representations without word alignments[END_REF] extend the skipgram model to the bilingual case: the former uses the words in the source language to additionally predict their aligned words in the target language while the later introduces an L 2 sampled loss for cross-lingual regularization.

Bilbowa

As discussed above, the goal of models used to learn cross-lingual embeddings is to learn feature that generalize across languages. The goal is to learn word embeddings such that similar words in each language are close in the induced space and furthermore similar words across languages are also close. For example, the words "cat", "dog" and "chien" (French for dog) are expected to be close in the shared space. Assigning similar embeddings stands for assigning vectors that are close in terms of a distance metric like Euclidean distance or cosine similarity in the induced space. To accomplish this, they optimize a monolingual loss that encourages similar words to have similar embeddings in each languages and a cross-lingual loss that encourages similar words written in different languages to be close.

To achieve that Bilbowa [START_REF] Gouws | Bilbowa: Fast bilingual distributed representations without word alignments[END_REF] that stands for "Bilingual Bag-of-Words without

Alignments" optimizes a monolingual objective function L(•) and the cross-lingual objective is enforced as regularization by a term Ω. Therefore the overall loss function to be optimized is:

L = min ∈{s,t} w t ,h∈D L (w t , h; θ) + λΩ(θ s , θ t ). (3.5)
The first term of Eq. (3.5) captures the monolingual objective over the source s and target t languages while the second term encourages the embeddings of similar words across languages to be close. One of the advantages of this formulation is that one may use unlimited corpora for learning the monolingual embeddings and a smaller collection of parallel sentences to enforce the regularization.

For the monolingual objective Bilbowa uses the objective of the skipgram model that we presented in Section 3.1.1. For the cross-lingual objective the model utilizes sentence aligned data and approximates with a sampling method the loss:

Ω = i j a i,j r s i -r t j 2 , ( 3.6)
where a i,j encodes a translation score for the words i and j approximated using the sentence aligned parallel data and r s i , r t j are the embeddings of the words i, j in the source and target languages respectively. Intuitively, Eq. (3.6) is weighted sum that is minimized when pairs of words with high translation scores a i,j have small Euclidean distances in the embedding space. Instead of relying on an alignment tool like Giza++ [START_REF] Franz | A systematic comparison of various statistical alignment models[END_REF] which is computationally expensive for finding the a i,j from the aligned sentences, the model uses a sampling mechanism that accelerates the process.

Concept Net

Word embeddings have been shown to capture the semantics of words as well as some of their syntactic properties [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF]. The models we presented so far however, from the topic models approaches like LDA to the word emebedding models like skipgram implement distributional semantics models using strictly free text. Their cross-lingual extensions also rely on free text for each language and may also require some aligned comparable or parallel corpora. One would expect however, that combining those successful models with external information sources like knowledge graphs would yield even better representations. The work of [START_REF] Speer | Conceptnet 5.5: An open multilingual graph of general knowledge[END_REF] proposed to extend the models for learning word representations to incorporate knowledge from ConceptNet, that is linked open data resource.

ConceptNet, first released by [START_REF] Liu | Conceptnet-a practical commonsense reasoning tool-kit[END_REF], is a knowledge graph that connects words and phrases with labeled, weighted edges. The edges encode relations of "is_a" type (e.g. The word cold in English is studený in Czech). Its graph-structured knowledge is particularly useful for models like those used to train word embeddings as such relations can be utilized to learn semantic spaces that are more effective than using distributional semantics alone, as in the case of the previously presented models.

The fundamental idea behind the model used to incorporate the knowledge graph in the process of learning word embeddings is to fine-tune embeddings learned with a model like skipgram using the knowledge graph relationships. Assuming, for instance, the objective function:

L(Q) = n i=1 a i q i -qi 2 + (i,j)∈E β i,j q i -qi 2 . (3.7)
The loss of Eq. (3.7) describes the fine-tuning process used to derive the embedding of word i to be qi from an initially learned embedding q i . This process is referred to expended retrofitting [START_REF] Speer | Conceptnet 5.5: An open multilingual graph of general knowledge[END_REF][START_REF] Speer | An ensemble method to produce highquality word embeddings[END_REF]. The first term encourages the updated embeddings to be close to the original ones by minimizing their Euclidean distance

q i -qi 2 .
The second term, which is a weighted summation over the edges of the knowledge graph encourages the embeddings of words that have some relationships to be close. The weights β i,j of the sum are also taken from the knowledge graph. This makes it possible to learn embeddings for words that were out-ofvocabulary of the initially learned embeddings q i , by effectively setting α i = 0 for a word i and relying only on the knowledge graph connections. Another significant advantage of the expanded retrofitting as described above is that it can benefit from the multilingual connections in ConceptNet (e.g. The word cold in English is studený in Czech). The model learns more about each language via the translations of words in other languages, and also aligns the embeddings of similar semantically similar terms written in different languages.

Summary

The chapter presented an overview of word embeddings. Their interesting properties of capturing semantic and syntactic properties of the words were highlighted as they will be later used as external sources of information for probabilistic models or as text representations for different tasks. We also presented popular models that are based on shallow neural networks for learning embeddings for text written in one (cbow or skipgram) or more languages (BilBowa, ConceptNet). Apart from shallow neural networks, we also presented recurrent neural networks, which although computationally expensive are well suited for modeling text sequences.

For a deeper analysis of deep learning models and their properties, one may refer to the introduction of [START_REF] Michael A Nielsen | Neural networks and deep learning[END_REF] or the excellent reviews of [START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF][START_REF] Goodfellow | Deep Learning[END_REF].

39

Chapter 4

Incorporating Prior Knowledge of Text Structure to Topic Models P robabilistic topic models aim at uncovering the latent topics of a collection of documents. To identify the topics, one typically represents the documents of the collection as a bag-of-words and applies an inference approach like Gibbs Sampling [START_REF] Thomas | Finding scientific topics[END_REF]. We discussed in Chapter 2 that popular topic models (e.g., LDA [START_REF] David M Blei | Latent dirichlet allocation[END_REF]) represent documents as bag-of-words. We argue in this chapter that this can be limiting and propose to overcome it.

The exchangeability assumption, that follows from the bag-of-words, dictates that given the topic distribution of a document, the words of the document are conditionally independent. While this assumption greatly benefits the involved computations and, in particular, the calculations of the conditional probabilities, it is rather naive and unrealistic [START_REF] Heinrich | Parameter estimation for text analysis[END_REF]. A shortcoming concerns the loss of information from not accounting for the grouping of words in topically coherent spans. These can be contiguous words that form text spans like sentences that are important in the use of language.

Text structure contains useful information that could be leveraged during inference. Sentences or phrases, for instance, are by definition text spans complete in themselves that convey a concise statement. To better illustrate how text structure could help in topic identification, consider the example of Figure 4.1. It shows the topics inferred by LDA for the words (excluding stop-words) of a sentence drawn from a Wikipedia page. At the sentence level, one could argue that the sentence is generated by the topic "Cinema" since it discusses a film and its authors. LDA, however, fails and assigns several topics to the words of the sentence. Importantly, several of those topics like "Elections" and "Inventions" are unrelated. In finer textual granularity, LDA also fails to assign consistent topics in noun-phrases like

The film is a remake of the 1947 film noir classic that starred Victor Mature, Brian Donlevy and Richard Widmark. Cinema Science Elections Inventions "film noir classic" and entities like "Brian Donlevy". A binding mechanism among the topics of the words of a sentence, or a phrase, could have prevented those inconsistencies in the topic assignment process. Hence, the hypothesis we investigate in this chapter is whether taking simple text structure into account benefits topic models.

To evaluate our hypothesis, we aim at extending LDA with prior knowledge of text structure. We suggest that such knowledge can be in the form of boundaries of topically coherent text spans, like the noun-phrases of Figure 4.1. We propose two approaches to achieve that: (i.) The first assumes that the words within a coherent segment are generated by the same topic and proposes a collapsed Gibbs sampling inference process that uncovers the topic while taking into account the per-word topic distributions of the words that compose the segment [START_REF] Balikas | On a topic model for sentences[END_REF].

(ii.) The second shares the motivation that segments should be topically coherent but proposes a more flexible approach that utilizes copulas in the sampling process and, therefore, allows a few (instead of one), related topics to occur within the segment [START_REF] Balikas | Modeling topic dependencies in semantically coherent text spans with copulas[END_REF].

Both models assume some level of dependence between the topics of the words of segments. This dependence is maximal for the first model. The second, through the use of copulas has bigger modeling capacity and is more flexible. It is to be noted, that for both types of models, the documents are assumed to be segmented a priori. Different segmentation mechanisms can be used, from linguistically motivated (e.g., parsing) to statistically motivated (e.g., n-grams). We expect different approaches to text segmentation to have different advantages; we intend to evaluate the impact of this choice also.

The remainder of the chapter discusses those points in detail. It is organized as follows:

• Section 4.1 presents an overview of the related work.

• Section 4.2 defines what segments are, as the concept is fundamental for the topic models to be proposed.

• Section 4.3 describes the two novel topic models: segmentLDA and copulaLDA and details their inference processes.

• Section 4.4 presents the experiments performed to assess the quality of the proposed topic models, and

• Section 4.5 concludes with a summary of the chapter.

An overview of the relevant literature

Despite the success that vector-space models [START_REF] Salton | A vector space model for automatic indexing[END_REF] have enjoyed, they come with a number of limitations. We mention, for instance, their inability to model synonymy and polysemy and the sparse, high-dimensional induced representations.

Many research studies have pointed out these problems, and Probabilistic Latent Semantic Analysis [START_REF] Hofmann | Probabilistic latent semantic indexing[END_REF] was among the first attempts to model textual corpora using latent topics. In this chapter, we build on LDA [START_REF] David M Blei | Latent dirichlet allocation[END_REF], which is often used as a building block for topic models. In its context, the corpus is associated with a set of latent topics, and each document is associated with a random mixture of those topics. The words are assumed exchangeable, that is their joint probability is invariant to their permutation. Previous work proposed a variety of extensions to LDA in order to incorporate additional information such as class labels [START_REF] Blei | Supervised Topic Models[END_REF] and temporal dependencies between stream documents [START_REF] Wang | TM-LDA: Efficient Online Modeling of Latent Topic Transitions in Social Media[END_REF]. Here, our goal is to extend LDA by incorporating simple text structure in its generative and inference processes using copulas.

One may identify two lines of research to address the limitations due to the exchangeability assumption in LDA: extensions to account for the boundaries of text spans like sentences and extensions to account for the word order. With respect to the first line, [START_REF] Wang | Multi-document summarization using sentence-based topic models[END_REF] combines a unigram language model with topic models over sentences so that the latent topics are represented by sentences instead of terms. In [START_REF] Griffiths | Integrating Topics and Syntax[END_REF], the authors investigate a combination of a topic model with a Hidden Markov Model (HMM). They assume that the HMM generates the words that handle the long-range dependencies (semantic dependencies) and the topic model the words that handle the short range dependencies (syntactic dependencies). Also, [START_REF] Blei | Syntactic topic models[END_REF] proposed the Syntactic Topic Model whose goal is to integrate the text semantics and the syntax in a non-parametric topic model. Having the tree, the semantic consistency of each document is given by a distribution over latent topics, as in topic models, and the syntactic consistency by the fact that each element in the tree has also a distribution over the topics of its children. In another effort, [START_REF] Zhu | Taglda: Bringing document structure knowledge into topic models[END_REF] propose TagLDA, where they replace the unigram word distributions by a factored representation that is conditioned on the topic and the part-of-speech tag of a term.

The second line of research investigates how topic models can be extended to incorporate word order. In [START_REF] Shafiei | Latent Dirichlet Co-clustering[END_REF], the authors propose a four-level hierarchical structure where the latent topics of paragraphs are decided after performing a nested word-based LDA operation. A particularly interest body of work concerns collocations. They can be defined as a sequence of consecutive words that have the characteristic of a syntactic and semantic unit, such as stock market, Los Angeles Premier League [START_REF] Choueka | Looking for needles in a haystack or locating interesting collocational expressions in large textual databases[END_REF]. Previous work has mainly explored the idea of bigram collocations. For instance, [START_REF] Wang | Topical n-grams: Phrase and topic discovery, with an application to information retrieval[END_REF] studied when bigrams should be assigned as a whole to particular topics or as two disjoint unigrams in other topics. Later, [START_REF] Johnson | Pcfgs, topic models, adaptor grammars and learning topical collocations and the structure of proper names[END_REF] proposed a model that combines the LDA and adaptor grammars to incorporate word collocations in the process of topic modeling. Despite their theoretical elegance these models come with higher computational overhead. To this end, [START_REF] Jey | On collocations and topic models[END_REF] explored how various strategies of selecting bigrams to be used as artificial tokens can impact the quality of topic models. Their models although interesting and inspiring for our development, have the shortcoming of increasing the vocabulary size and therefore the sparsity of the model.

Another interesting line of research studied the task of discovering and partitioning text in topically coherent spans. In [START_REF] Du | A Segmented Topic Model Based on the Twoparameter Poisson-Dirichlet Process[END_REF][START_REF] Du | Topic segmentation with a structured topic model[END_REF] the authors rely on hierarchical Bayesian models to accomplish it. In our work here, contrary to identifying such spans, we assume documents to be topically coherent a priori, and we investigate how to leverage and incorporate this information to LDA.

The coherent text segments

In this section we discuss the idea of coherent text spans, or segments hereafter, as they are central for the subsequent development of the topic models. Given a document, there are several types of text spans that can be regarded as the document's

The film is a remake of the 1947 film noir classic that starred Victor Mature, Brian Donlevy and Richard Widmark. building blocks. These spans usually constitute semantic units pertaining to a single or few related topics and are topically coherent in this sense [START_REF] Balikas | Modeling topic dependencies in semantically coherent text spans with copulas[END_REF]. From a linguistic point-of-view, a document consists of sentences, which are meaningful text spans that convey a concise statement. In a finer level, syntactic analysis of sentences like shallow parsing reveals coherent segments like noun-phrases. Figure 4.2 illustrates the output of a shallow parsing step that identifies the noun-phrases of the example sentence of Figure 4.1, generated using the Stanford Parser. 1Both sentences and noun-phrases are text spans composed by contiguous words and can be considered semantically coherent. Further, from a computational aspect, sentence segmentation can be performed efficiently in several languages.

Also, there are several pre-trained models for shallow parsing in a variety of languages. For the above reasons, in the rest of our development we will evaluate how assuming sentences and noun-phrases to be topically coherent affects the topic modeling performance.

N -grams are consecutive words of length N . Compared to sentences or nounphrases no syntactic information is used to obtain them as they only rely on count statistics. Despite their simplicity, their positive effects on topic modeling have been shown in previous work (e.g., [START_REF] Jey | On collocations and topic models[END_REF][START_REF] Wang | Topical n-grams: Phrase and topic discovery, with an application to information retrieval[END_REF]). Therefore, n-grams are another type of segments we may consider coherent.

The segmentation of the documents in coherent segments creates a hierarchical document representation. From the higher to the lower levels this structure is described as follows: (i) documents consist of segments, which are independent between them, and (ii) segments consist of words, which are their basic units. Therefore, each document is a bag-of-segments and each segment is a bag-of-words.

These independence assumptions are important while performing inference for the topic models that will presented in the next sections.

Incorporating text structure to topic models

The previous sections provided an overview of the relevant literature with a focus on work that incorporates parts of text structure to topic models and defined coherent text segments. We continue by presenting the two contributions of this chapter. In particular, we first describe segmentLDA in Section 4.3.1 and then introduce in Section 4.3.3 copulaLDA. These are two novel topic models that relax the bag-of-words assumption that is a fundamental premise of LDA. They propose different mechanisms for incorporating text structure in their generative process.

segmentLDA: Integrating segment boundaries to LDA

Recall that a probabilistic topic model represents the words in a collection of D documents as mixtures of K "topics", which are multinomials over a vocabulary of size V . In the case of LDA, for each document d i a multinomial over topics is sampled from a Dirichlet prior with parameters α.

We extend LDA by adding an extra plate denoting the coherent text segments of a document. The graphical representation of this novel mode, called segmentLDA (segLDA) model is shown in Figure 4.3. The generative process of a document collection according to segLDA is as follows:

• For each topic k ∈ [1, K], choose a per-word distribution: φ k ∼ Dir(β), with φ k , β ∈ R V • For each document d i , i ∈ {1, . .

. , D}:

-Choose a per-document topic distribution:

θ i ∼ Dir(α), with θ i , α ∈ R K -For each segment s i,j , j ∈ {1, . . . , S i } of d i :
Sample the topic underlying the segment's words:

z i,j ∼ Mult(1, θ i )
Sample the words of the segment: (w 1 , . . . , w N i,j ) ∼ Mult(N i,j , φ z i,j )

As it is evident from the generative story, a single topic is assigned to the words of a segment as each word of the segment is sampled from Mult(N i,j , φ z i,j ). However, a topic is assigned to each and every word of the document, as in LDA. Therefore, words (and not segments) remain the basic units of the documents. This signifies that comparing topics models such as LDA and segLDA on tasks that use the topics of each word (like perplexity that we discuss in Section 4.4) is fair.

The generative process we presented above describe the generation of the collection of documents. Meanwhile, given a corpus words are observed and the goal is to infer the latent topics. For inference, we propose to use a collapsed Gibbs sampling method [START_REF] Thomas | Finding scientific topics[END_REF]. We now derive the Gibbs sampler equations by estimating the hidden topic variables. In segLDA the joint distribution of words w and topics z can be decomposed as:

p(w, z|α, β) = p(w|z, β)p(z|α), ( 4.1)
because the first term is independent of α2 and the second from β (cf. Fig. 4.3).

After standard manipulations as in the paradigm of [START_REF] Heinrich | Parameter estimation for text analysis[END_REF] one arrives at:

p(z, w|α, β) = K z=1 ∆(Ψ z + β) ∆(β) D i=1 ∆(Ω i + α) ∆(α) , ( 4.2)
where

∆( x) = Beta(x 1 , . . . , x m ) = dim x k=1 Γ (x k ) Γ ( dim x k=1 x k )
is a multidimensional extension of the beta function used for notation convenience, and Ω i , Ψ z refer to the occurrences of topics with documents and topics with terms respectively. To calculate the full conditional probability we take into account the structure of the document i and the fact that w i = {w i¬s i,j , w ¬s i,j }, z = {z i¬s i,j , z ¬s i,j }. The subscript s i,j in w s i,j , z s i,j denotes the words and the topics respectively of segment s i,j , that is the j-th segment of the i-th document. For the full conditional of topic k we have:

p(z s i,j = k|z ¬s i,j , w) = p(w, z) p(w, z ¬s i,j ) = p(w|z) p(w ¬s i,j |z ¬s i,j )p(w s i,j ) p(z) p(z ¬s i,j ) = = p(w, z) p(w ¬s i,j , z ¬s i,j ) ∝ ∆(Ψ z + β) ∆(Ψ z,¬s i,j + β) ∆(Ω i + α) ∆(Ω i,¬s i,j + α) . (4.3)
For the first term of equation Eq. ( 4.3) we have:

∆(Ψ z + β) ∆(Ψ z,¬s i,j + β) = w∈s i,j Γ (Ψ z +β) Γ ( w∈s i,j (Ψ z +β)) w∈s i,j Γ (Ψ z,¬s i,j +β) Γ ( w∈s i,j (Ψ z,¬s i,j +β)) = w∈s i,j       Γ (Ψ z + β) Γ (Ψ z,¬s i,j + β)       Γ ( w∈s i,j (Ψ z,¬s i,j + β)) Γ ( w∈s i,j (Ψ z + β)) = = A w∈s i,j (Ψ w,k,¬s i,j + β) • • • (Ψ w,k,¬s i,j + β + (N i,j,w -1)) ( w∈V (Ψ w,k,¬s i,j + β)) • • • ( w∈V Ψ w,k,¬s i,j + β + (N i,j -1)) B . (4.4)
Here, for the generation of A and B we used the recursive property of the Γ

function: Γ (x + m) = (x + m -1)(x + m -2) • • • (x + 1)xΓ (x)
; w is a term that can occur many times in a sentence and N i,j,w denotes the frequency of w in segment s i,j ; N i,j

denotes the number of words in sentence s.

The development of the second factor in the final step of Eq. ( 4.3) is similar to the LDA calculations. The difference is that the counts of topics per document are estimated given the allocation of every word of a segment to the sampled topic.

On the other hand, compared to segLDA, LDA does not incorporate any part of the local structure when sampling topics for the words. From Eq. (4.3) one yields:

p(z s i,j = k| z ¬s i,j , w) = (Ω i,k,¬s i,j + α) × w∈s i,j (Ψ w,k,¬s i,j + β) • • • (Ψ w,k,¬s i,j + β + (N i,j,w -1)) (Ψ •,k,¬s i,j + V • β) • • • (Ψ •,k,¬s i,j + V • β + (N i,j -1)) (4.5)
where Ω i,k,¬s denotes the number of words from document i assigned to topic k excluding the words of the segment currently sampled. Further, the product in the numerator of the second term results from the bag-of-words assumption for the words within the segments of d i . The possibly multiple occurrences of w in s i,j , generated by the topic k, are taken into account by the factor (Ψ w,k,¬s ij +β), which is incremented by one for every other occurrence of the word after the first. This reflects the fact that every occurrence of w comes from the same topic. For instance, if w appears twice in s i,j , then N i,j,w = 2, and the factor (

Ψ k,w,¬s ij +β)(Ψ k,w,¬s ij +β +1)
denotes the contribution of the occurrences of w to the probability that s i,j is generated by the topic k. The product in the denominator acts as a normalization term. The progressive increase of its values can be explained by the bag-of-words assumption within a segment: the product normalizes the probability of assigning Algorithm 1: A Gibbs Sampling iteration for segLDA Input: documents' words grouped in segments, α, β, K //Initialize counters Ψ , Ω for document d i , i ∈ [1, D] do for segment s i,j : j ∈ {1, . . . , S i } do Decrease counter variables Ψ , Ω according to the previous topic assignments of the words of s i,j Calculate the probabilities of the new topic of the words of s i,j (Eq. 4.5) Sample the topics of the words of s i,j using the calculated probabilities Increase counters Ψ , Ω end end the topic k to a word of the segment, given that the previous words have also been assigned to this topic. Algorithm 1 presents the steps one needs to follows during the Gibbs sampling updates of segLDA.

Note that segLDA is an extension of LDA. If the coherent text spans are reduced to words that is ∀i, j : N i,j = N i,j,w = 1 then segLDA reduces to LDA and Eq. (4.5) reduces to the standard LDA collapsed Gibbs sampling inference equations of Eq.

(2.5).

Copulas and random variables (intermezzo)

In the previous section we proposed segLDA, a novel topic model that assigns the same topics to the words of a segment. This entails that the dependency between the topics of the words of a segment is maximal. The question that arises in this case is whether one can come up with a more flexible binding mechanism. If such a mechanism exists, one could then incorporate to a topic model that would account for text structure but would also allow more flexibility within topics. In this section, we introduce copulas, a statistical tool that can be used to solve this problem.

Copulas allow one to explicitly relate joint and marginal distributions, through Sklar's theorem [START_REF] Sklar | Fonctions de Répartition à n Dimensions et Leurs Marges[END_REF]: x) axis depict the distributions of the initial (resp. transformed) samples.

Theorem 4.
Formally [START_REF] Roger | An introduction to copulas[END_REF][START_REF] Trivedi | Copula Modeling: An Introduction for Practitioners[END_REF], a p-dimensional copula C is a p-variate distribution function with C : I p = [0, 1] p → [0, 1] whose univariate marginals are uniformly distributed on I and C(u 1 , . . . , u p ) = P (U 1 ≤ u 1 , . . . , U p ≤ u p ).

As a result any multivariate distribution F can be decomposed into its marginals F i , i ∈ {1, . . . , p} and a copula, allowing to study the multivariate distribution independently of the marginals. Sklar's theorem also provides a way of sampling multivariate distributions with a large number of random variables using copulas:

F(x 1 , . . . , x p ) = F F -1 1 (u 1 ), . . . , F -1 p (u p ) = P [U 1 ≤ u 1 , . . . , U p ≤ u p ] = C(u 1 , .
. . , u p ). Hence, to sample F it suffices to sample the dependence structure modeled by copulas and then transform the obtained sample in the marginals of interest using the probabilistic integral transform. We illustrate this transformation for one variable in Figure 4.4. Sampling the copula returns, for each variate, a sample as the one indicated in the histogram of the y axis. One can then transform the sample using the quantile (F -1 ) of an arbitrary marginal.

Before proceeding further, we visit some extreme conditions of dependence illustrating the respective copulas that model them: (1) Independence, which is a frequently assumed simplification in topic models and is obtained with

p i=1 u i ,
and (2) Co-monotonicity, which is the complete, positive correlation between the random variables u p , obtained with min(u 1 , . . . , u p ).

In the rest of our development we will be using a particular family of copulas, the Archimedean copulas. Archimedean copulas are widely used copulas and are defined with respect to a generator function ψ. They take the form:

C(u 1 , • • • , u d ) = ψ -1 (ψ(u 1 ) + • • • + ψ(u d ))
. A special case of Archimedean copulas corresponds to Frank copulas, which are obtained by setting:

ψ λ (u) = -1 λ log(1-(1-e -λ )e -u
). When λ → 0, the Frank copula approaches the independency copula; when λ → ∞ it approaches the co-monotonicity copula. Hence, the Frank copula allows one to model all dependencies between complete independence to perfect dependence while varying λ from 0 to ∞. Therefore, λ can be seen as an additional hyperparameter to be tuned from the data. Figure 4.5 illustrates the positive dependence between two random variables sampled from a Frank copula. To highlight the effect of λ in the correlation imposed at the sample we visualize samples while increasing its value. Following the increase of the λ values the correlation between the values of the variates increases. To sample from the Archimedean copulas, we rely on the algorithm proposed by [START_REF] Marshall | Families of Multivariate Distributions[END_REF], which was further improved in [START_REF] Mcneil | Sampling Nested Archimedean Copulas[END_REF][START_REF] Hofert | Efficiently Sampling Nested Archimedean Copulas[END_REF] and implemented in the R language [START_REF] Hofert | Nested Archimedean Copulas Meet R: The nacopula Package[END_REF].

Lately, there is an increasing interest over the integration of copulas in machine learning applications [START_REF] Elidan | Copulas in Machine Learning[END_REF] such as classification [START_REF] Elidan | Copula Network Classifiers (CNCs)[END_REF] or structure learning [START_REF] Liu | The Nonparanormal: Semiparametric Estimation of High Dimensional UndirectedGraphs[END_REF]. Interestingly, [START_REF] Wilson | Copula Processes[END_REF] have shown how to incorporate copulas in Gaussian processes in order to model the dependency between random variables with arbitrary marginals with a practical application on predicting the standard deviation of variables in the financial sector (volatility estimation). In another generic framework, [START_REF] Tran | Copula Variational Inference[END_REF] have shown the benefits of using copulas to model complex dependencies between latent variables in the general variational inference setting.

The idea of using copulas with topic models was recently investigated in the interesting work of [START_REF] Hesam Amoualian | Streaming-lda: A copula-based approach to modeling topic dependencies in document streams[END_REF]. In the context of document streams they proposed a topic model where the dependencies between the topic distributions of two consecutive documents are captured by copulas. Here, instead of modeling the dependence between topic distributions of consecutive documents, we model the dependence between the topics assigned to the words of segments we consider coherent.

copulaLDA: Integrating segment boundaries to LDA using copulas

In the previous section (Section 4.3.2) we introduced copulas, a powerful framework for modeling the joint distribution of random variables. The capacity of copulas to model joint distributions by decoupling the underlying dependence of the variables from their marginals can be a useful property for topic modeling.

Our motivation lies in the way that we sample topics for segments: we want a mechanism to model the joint distribution using information from the marginals.

For topic modeling, the marginals describe how topics occur with each word of a segment while the joint distributions concern the topics assigned to the words of the segments. As discussed previously in the framework of segLDA, one expects a dependence mechanism to apply between the topics of segments.

In this section we develop copulaLDA (hereafter copLDA), that extends LDA by integrating simple text structure in the model using copulas. We assume that the topics that generate the terms of coherent text spans are bound. A strong binding signifies high probability for the terms to have been generated by the same topic. Therefore, as we show, the conditional independence of topics given the perdocument topic distributions does not hold. The copLDA generative model. We model the dependency between the topics underlying a segment with copulas. Notice in the graphical model how the topics of (z 1 , . . . , z N ) of a segment depend both on θ and the copulas, illustrated with its λ parameter.

Copulas provide an intuitive way to bind random variables. We are making use of them here to bind word-specific topics (the z variables in LDA) within coherent text spans, the rationale being that coherent text spans can not be generated by many different, uncorrelated topics. This leads us to the following generative model:

• For each topic k ∈ [1, K], choose a per-word distribution: φ k ∼ Dir(β), with φ k , β ∈ R |V | • For each document d i , i ∈ {1, . .

. , D}:

-Choose a per-document topic distribution:

θ i ∼ Dir(α), with θ i , α ∈ R |K| -Sample number of segments in d i : S i ∼ P oisson(ξ); -For each segment s i,j , j ∈ {1, . . . , S i }:
Sample number of words: N i,j ∼ P oisson(ξ d ); Sample topics Z i,j = (z i,j,1 , . . . , z i,j,N i,j ) from a distribution admitting Mult(1, θ i ) as margins and C as copula; Sample words W i,j = (w i,j,1 , . . . , w i,j,N i,j ):

w i,j,n ∼ Mult(1, φ z i,j,n ), 1 ≤ n ≤ N i,j .
There are two main differences between copLDA and LDA. Firstly, the former assumes a hierarchical structure in the documents: the topics that generate the words in the coherent segments exhibit topical correlation, hence the conditional independence assumption between the terms of a segment given the document per-topic distribution (θ i ) no longer holds. Secondly, this topical correlation is modeled using copulas. Figure 4.6 provides the graphical model for copLDA. For clarity, we draw each word in a coherent segment S, (w 1 , . . . , w N ) to make the dependencies explicit. Notice how the topics of those words depend on both the copula parameter λ and the per-document topic distribution θ.

There is also an important difference between copLDA and segLDA. While the latter assumes that the words (w 1 , . . . , w N ) of a segment are generated by a single topic, the former allows more flexibility. As a result, more topics may be observed within a segment. The copula hyperparameter λ as well as the family of the copulas chosen control this flexibility. Notice that in the limit of total dependence (when λ → ∞) copLDA becomes equivalent with segLDA.

The hyper-parameters α and β correspond to priors of the model. Their values can be set according to values that previous work has proposed (e.g., [START_REF] David M Blei | Latent dirichlet allocation[END_REF]) or can be tuned using the data. Similarly, the hyper-parameter λ can be chosen after exploration of a grid of possible values.

Inference with Gibbs sampling The parameters of the above model, that are φ, θ and the topics of each segment Z i,j = (z i,j,1 , • • • , z i,j,N i,j ), can be directly estimated through Gibbs sampling. Denoting Ω and Ψ the count matrices such that

Ω = (Ω i,k ) (resp. Ψ = (Ψ k,v
)) represents the count of word belonging to topic k assigned to document d i (resp. the count of word v being assigned to topic k), the Gibbs updates for θ and φ are the same as the ones for the standard LDA model [START_REF] David M Blei | Latent dirichlet allocation[END_REF]:

θ i ∼ Dir(α + Ω i ) and φ k ∼ Dir(β + Ψ k ) (4.7)
The update for the variables z is obtained as follows:

p(Z i,j |Z ¬i,j , W , Θ, Φ, α, β, λ) = p(Z i,j , Z ¬i,j , W |Θ, Φ, α, β, λ) p(Z ¬i,j , W |Θ, φ, α, β, λ) = p(Z i,j , W i,j |Θ, Φ, λ)p(Z ¬i,j , W ¬i,j |Θ, Φ, λ) p(W i,j |Θ, φ)p(Z ¬i,j , W ¬i,j |Θ, Φ, λ) = p(Z i,j , W i,j |Θ, Φ, λ) Z i,j p(Z i,j , W i,j |Θ, Φ, λ) = p(W i,j |Z i,j , Φ)p(Z i,j |Θ, λ) Z i,j p(W i,j |Z i,j , Φ)p(Z i,j |Θ, λ) ∼ p(W i,j |Z i,j , Φ)p(Z i,j |Θ, λ) = p(Z i,j |Θ, λ) N i,j n=1 φ w i,j,n ,z i,j,n (4.8)
where W , Θ and Φ stand for the whole parameter set of w, θ and φ and the probability outside the product in the last step admits a copula C λ and Mult(1, θ i ) as margins. The notation -i, j means excluding the information for i, j. Note that in case where λ → 0, the words of a segment become conditionally independent given the per-document distribution and one recovers the non collapsed Gibbs sampling updates of LDA.

From the expression of Eq. (4.8), a simple acceptance/rejection algorithm can be formulated: (1) Sample a random variable of pdf p(Z i,j |Θ, λ) using copula, and,

(2) Accept the sample with probability p(W i,j |Z i,j , Φ) = N i,j n=1 φ w i,j,n ,z i,j,n . Algorithm 2 summarizes the inference process.

Computational Considerations As the values of φ w i,j,1 ,z i,j,1 ו • •×φ w i,j,n ,z i,j,n tend to be very low, the acceptance/rejection sampling step described above is very slow in practice (see below). We propose here to speed it up by considering, for each word w i,j,n in a given segment, not the exact probability of z i,j,n , but its mean (noted M) over all the other words in the segment: j,n θ d,z i,j,n as w ij,l φ w i,j,l = 1. In the experimental part we will empirically illustrate the effect of the mean approximation as described above.

M(z i,j,n |Z -i,j , W , Θ, Φ, α, β, λ) = w ij,l ,l n z ij,l ,l n P (Z i,j |Z -i,j , W , Θ, Φ, α, β, λ) ∝ φ w i,

The Experimental Evaluation

In this section we evaluate the copLDA and segLDA models presented above. To this end, we propose both intrinsic and extrinsic evaluation tasks. The assessment

φ w i,j,1 ,z i,j,1 × • • • × φ w i,j,n ,z i,j,n end end
of the performance of topic models in an intrinsic way signifies that no application is used. Common ways to intrinsically evaluate topic models is by measuring the coherence of the produced topic or by estimating their generalization performance. On the other hand, extrinsic evaluation of topic models requires an application like text classification or document retrieval.

Models

In the experiments of this section we compare the following topic models:

(i.) LDA3 as proposed in [START_REF] David M Blei | Latent dirichlet allocation[END_REF] using the collapsed Gibbs sampling inference [START_REF] Thomas | Finding scientific topics[END_REF], (ii.) segLDA bi as described in Section 4.3.1 with the 1,000 most frequent bigrams to be considered as coherent segments, (iii.) segLDA tri as described in Section 4.3.1 with the 1,000 most frequent trigrams to be considered as coherent segments, (iv.) segLDA np as described in Section 4.3.1 with noun-phrases as coherent segments, (v.) segLDA sent as described in Section 4.3.1 with sentences as coherent segments, (vi.) copLDA bi , copLDA tri , copLDA np , copLDA sent that use copulas to extend the previous models following the development of Section 4.3.3.

In total, we considered nine models in our experiments that extend LDA using different types of segments and different binding mechanisms between the topics of the words that constitute the segments. For copLDA x models, we use the Frank copula which was reported to obtain the best performance in similar tasks [START_REF] Hesam Amoualian | Streaming-lda: A copula-based approach to modeling topic dependencies in document streams[END_REF] and was also found to achieve the best performance in our local validation settings compared to Gumbel and Clayton copulas. We have implemented the models using Python. For sampling the Frank copulas we used the R copula package [START_REF] Hofert | Nested Archimedean Copulas Meet R: The nacopula Package[END_REF] and rPY. 4 Also, λ is set to values which we found to perform well in every dataset we tried, that is to 2 for copLDA sent and to 5 for copLDA np,bi,tri . Furthermore, the hyper-parameters α and β where set to 1/K and 0.01 respectively following [START_REF] Thomas | Finding scientific topics[END_REF], where K is the number of topics. For the shallow parsing step, required for copLDA np , we used the Stanford Parser [START_REF] Klein | Accurate unlexicalized parsing[END_REF]. The text pre-processing steps performed are: lower-casing, stemming using the Snowball Stemmer and removal of numeric strings.

Datasets We use the following publicly available data collections to test the performance of the topic models:

• 20NG (20 news groups), which is a standard text dataset for such tasks as provided by [START_REF] Bird | Natural language processing with Python[END_REF],

• Reuters (Reuters-21578, the "ModApte" version), also discussed in [START_REF] Bird | Natural language processing with Python[END_REF],

• TED, that are transcriptions of TED talks released in the framework of the International Workshop on Spoken Language Translation 2013 evaluation campaign 5 (we have merged the train, development and test parts and we selected the transcriptions with at least one associated label among the 15 most common in the data 6 ),

• Wiki x , with x ∈ {15, 37, 46} and PubMed, both excerpts7 from the Wikipedia dataset of [START_REF] Partalas | LSHTC: A benchmark for large-scale text classification[END_REF] and the PubMed dataset of [START_REF] Tsatsaronis | An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition[END_REF] used in [START_REF] Balikas | On a topic model for sentences[END_REF] Basic • "Austen", where we concatenated three books 8 written by Jane Austen, available from the Gutenberg project (each paragraph is considered as a document).

Table 4.1 presents some basic statistics for these datasets. The goal when selecting them was to evaluate the models on data of different types, ranging from forum messages (20NG) to news stories (Reuters) or literature ("Austen"). In that way we exclude any possible bias due to the type or source of the text.

The effect of the mean approximation during inference Figure 4.7 compares the perplexity scores achieved in 200 documents from the "Wiki46" Wikipedia dataset of Table 4.1 by the copLDA model, when considering noun-phrases as coherent spans, with and without rejection sampling. We repeat the experiment 10 times and also plot the standard deviation. We first note that approximating Algorithm 1 by ignoring the rejection sampling step results in slightly worse performance. On the other hand, without the rejection sampling, copLDA converges faster in terms of iterations. Furthermore, the cost in terms of running time of a single iteration is significantly smaller: for instance, for 30 iterations with rejection sampling, the algorithm needs almost 6 hours, that is 100 times more than the 3.5 minutes needed without the rejection sampling. Hence, in the rest of the study, for scaling purposes, we adopt the above mean approximation. No Rejection Sampling Rejection Sampling

Intrinsic Evaluation

Intrinsic evaluation of topic models is a way of evaluating the topic models without using a real application. Such types of evaluation usually assess the coherence of the produced topics, either by manual inspection or by calculating some scores that are indicative of how often words occur in similar contexts. The most-used intrinsic measure to evaluate topic models is probably perplexity. As, however, it was shown that perplexity does not always correlate well with human judgments of the quality of the produced topics [START_REF] Chang | Reading tea leaves: How humans interpret topic models[END_REF], various measures have been proposed as alternatives [START_REF] Newman | Automatic evaluation of topic coherence[END_REF][START_REF] Mimno | Optimizing semantic coherence in topic models[END_REF][START_REF] Jey | The sensitivity of topic coherence evaluation to topic cardinality[END_REF]. In this section, we evaluate the proposed topic models with respect to several of these measures. We begin by visualizing the learned topics, we continue by reporting the perplexity scores and then, we discuss the topic coherence performance with regard to the normalized point-wise mutual information scores.

Manual inspection of the topics We begin by comparing the topics learned by

LDA and copLDA np . We choose to visualize the results for those two models as LDA is our main baseline while copLDA np integrates noun-phrase boundaries which are short and easy to visualize. For presentation purposes, we train the two topic models using the Wiki 47 dataset with 10 topics and we illustrate the top-10 words learned for each topic by the two models in Table 4.2. As one can note, since the two models have been trained on the same data with the same training parameters, the identified topics are very similar. This said, copLDA np manages to produce arguably better topics. This is for example the case for the topic "Birth"; although both models assign high probability to words like "born" and "american" due to the content of the dataset, copLDA np manages to identify several words corresponding to months which makes the topic more thematically consistent and easier to interpret compared to its LDA counterpart. In the same line, Table 4.3 visualizes the inferred topics for parts of the Wiki 47 dataset. Recall, the first sentence of the table is the example out in the beginning of the chapter (Fig. 4.1) that motivated the work presented in the chapter. Notice here that given the topic interpretations of Table 4.2, both models manage to identify intuitive topics. Note, however, how in most of the cases the text structure information used by copLDA np helps to obtain consistent topics for noun-phrases like "crime thriller film" and "raspy voice", a consistency that LDA is lacking. Of course, this is not the case for every noun-phrase of the corpus: there are cases like "Dana Lynne Goetz" where two (or more) topics are assigned to the words of the phrases. This was expected as, by definition, the strength of the bound that copLDA np applies for sampling the topics of words of a segment is not maximal but is controlled by the λ hyperparameter of the copula.

Intrinsic Evaluation: Perplexity We continue our evaluation by presenting perplexity scores of held-out documents, calculated for each of the topic model datasets of Table 4.1. Achieving lower perplexity score means that a topic model can explain unseen data more efficiently, thus it generalizes better and it is, in turn, a better model.

As a result, a good model with low perplexity should be able to infer better representations for the unseen documents. As perplexity does not use any real application to evaluate the topic models it is also an intrinsic metric. For a set of test documents C consisting of N words {w 1 , . . . , w N } the perplexity is calculated using:

perpl(C) = exp                - N i=1 log p(w i ) N                (4.9)
Hence, the best the model fits the data, the higher will be the p(w i ) and consequently the lowest the perplexity score achieved.

In order to estimate perplexity, the topic distributions of the unseen documents are required. They can be obtained by repeating the Gibbs sampling inference process for the unseen (held-out) documents. During this process, however, the per-word topic distributions learned during training are kept constant.

In our experiments here, we split the documents of a dataset randomly in two parts with 80%/20% of the documents: we use the former for learning the model and the second for calculating the perplexity scores. We use exactly the same splits for training and evaluating each of the topic models. First, notice that the lowest perplexity is consistently achieved by segLDA bi and copLDA bi . This suggests that assuming short segments to be coherent benefits perplexity scores. Such a finding is intuitive in that frequent bigrams usually refer to entities like destinations (e.g., New York, United Kingdom) and assuming them to be coherent benefits the model's generalization performance. On the other hand, models that assume larger segments, like sentences (segLDA sent and copLDA sent ) to be coherent are not competitive.

Another interesting observation concerns the effect of copulas when various segments are considered. One can identify two families of models: those whose perplexity scores greatly improve due to the use of copulas: copLDA sent and copLDA np and those that do not benefit much: copLDA bi and copLDA tri . These results suggest that the size of segment and the frequency of co-occurrence is important for deciding whether to consider copulas or not. Short and frequent segments like bigrams are by definition coherent and having a flexible binding scheme like copulas does not improve the perplexity scores. On the other hand, sentences and noun-phrases that are less thematically consistent benefit from the flexibility that copulas offer. Note that segLDA sent is the model with the fastest convergence rate with respect to the number of Gibbs iterations. On the other hand, LDA, copLDA sen and copLDA np require the same number of iterations, which depends on the dataset. copLDA bi manages to achieve the lowest perplexity scores. Notice its steep curves in the first iterations. Intrinsic Evaluation: Normalized Pointwise Mutual Information Automatically evaluating the coherence of the topics produced by topic models is a task that has received a lot of attention. The goal is to measure how coherent or interpretable the produced topics are [START_REF] Mimno | Optimizing semantic coherence in topic models[END_REF]. It has been recently found that scoring the topics using co-occurrence measures, such as the pointwise mutual information (PMI) between the top words of a topic, correlates well with human judgments [START_REF] Newman | Automatic evaluation of topic coherence[END_REF]. To achieve that, an external corpus like Wikipedia is treated like a metadocument, which is used as the basis to calculate the PMI scores of words using a sliding window and applying the equation:

P MI(w i , w j ) = log P (w i , w j )

P (w i )P (w j ) . (4.10)
Evaluating the topic coherence requires selecting the top-N words of a topic and performing the manual or automatic evaluation. Here, N is a hyper-parameter to be chosen and its value can impact the results. Very recently, Lau and Baldwin [START_REF] Jey | The sensitivity of topic coherence evaluation to topic cardinality[END_REF] showed that N actually impacts the quality of the obtained results and in particular the correlation with human judgments. In their study they conclude that aggregating the topic coherence scores over several topic cardinalities, leads to a substantially more stable and robust evaluation. Following these findings, we present in Table 4.5 the topic coherence scores as measured by the normalized pointwise mutual information (nPMI). The scores of nPMI range in [-1,1], where in the limit of -1 two words w 1 and w 2 never occur together, while in the limit of +1 they always occur together (complete cooccurrence). As in [START_REF] Jey | The sensitivity of topic coherence evaluation to topic cardinality[END_REF], for each topic, we aggregate the topic coherence scores over three different topic cardinalities: N ∈ {5, 10, 15}. The table presents the results when training the topic models for various numbers of topics K, in particular for K ∈ {25, 50, 75, 100, 150}. The numbers reported are the averages of the nPMI scores and are calculated as follows: we train the topic models for 250 Gibbs sampling iterations. After the 150-th iteration (including the 150-th) we sample the top-N words of the topics every 25 iterations. As a result, we have 5 samples in total, and for those we calculate the average nPMI scores and standard deviations and report them in Table 4.5.

One may observe from the table that, in general, increasing the number of topics decreases the coherence of the learned topics. This applies for every topic model and dataset we tried. In most of the cases, the most coherent topics in terms of nPMI are obtained with 25 topics. In terms of the types of the segments we consider, once more, short segments likes noun-phrases, bigrams and trigrams perform the best. This, however, also depends on the dataset and, less frequently on the number of topics considered. Further, compared to the perplexity results where copulas improved the performance of topic models with linguistically motivated segments, in the nPMI experiments they do not. In most of the experiments, the models with copulas perform worse than their counterparts that do not use copulas for sampling the segment topics.

Our last observation concerns the nPMI scores: depending on the dataset, one observes higher or lower scores. The highest scores are consistently obtained on the PubMed dataset, while the lowerst on the Reuters. This, of course, greatly depends on the corpus used as a meta-document to estimate the PMI scores (Wikipedia in our case). The results however, suggest, that the topics learned using PubMed documents are more coherent, and we believe that this is due to the consistent use of language in scientific articles, like PubMed documents. To highlight this in Table 4.6 we show the nPMI scores for each topic of the best performing system (copLDA np ) for 25 topics for PubMed. From the table, Topic 3 that concern cancer is the most consistent with nPMI=0.27. Interestingly, including more words of the topic in the nPMI calculation improves the scores from .21 to .29 and .31 when considering the top-5, top-10 and top-15 words respectively. On the other hand, Topic 4 that is mainly about genetics, achieves the highest top-5 nPMI scores as the words "gene", "chromosom", "mutat", "tumor", and "genet" have nPMI=. [START_REF] Coulmance | Trans-gram, fast cross-lingual word-embeddings[END_REF].

Adding more words in the nPMI calculation however, worsens the score and results in an average nPMI of .24. We conclude the nPMI evaluation by commenting on a limitation of the approach: Topic 14 with achieves the lowest nPMI score of .03, is still consistent. Its top-5 words ("patient", "group", "p", "signific", "age", "studi") that probably describe group studies, although coherent for scientists, are scored with nPMI=0.0 probably due to meta-document we used for calculating the PMI probabilities. As Wikipedia is out-of-domain, one typically expects few entries to discuss similar topics and this impacts the scores of the topic.

Extrinsic Evaluation

Compared to the previous evaluation subtasks where we presented results of intrinsic evaluation, we now present a task for extrinsic evaluation of the topic models. Extrinsic evaluation of topic models uses a real task such as clustering [START_REF] Pessiot | Improving document clustering in a learned concept space[END_REF] or classification to assess the performance of the representations learned with topic models. The topics and nPMI scores per topic identified by copLDA np on the PubMed dataset when trained with K = 25. We words of the topics are ordered from left to right by their per-word topic probabilities. We report the nPMI scores of each topic for the top-N words (nPMI N ), with N ∈ {5, 10, 15}, as well as the average nPMI as suggested by [START_REF] Jey | The sensitivity of topic coherence evaluation to topic cardinality[END_REF].

Extrinsic evaluation: text classification Document classification is a supervised learning task where a document is associated with one or more categories from a pool of M categories Y = {y 1 , . . . , y M }. The document is represented as a vector X . To perform the classification task, a learner such as a Logistic Regression [START_REF] Mccullagh | Generalized linear models[END_REF] or Support Vector Machines [START_REF] Cortes | Support-vector networks[END_REF] is trained to uncover a function f : X → Y . In the framework of topic modeling, each document can be represented as by its topic distribution. Then, given the document topic distributions a learner can be trained in order to perform the classification task and assign the categories to the documents.

To this end, we obtain the per-document topic distributions following a protocol similar to the calculations of nPMI: we train the topic models for 250 Gibbs sampling iterations. After the 150-th iteration (including the 150-th) we sample the per-document topic distributions every 25 iterations. The per-document topic distributions used as inputs to the SVMs is then the average of the 5 samples. This approach of subsampling the chain is called thinning and is common with Gibbs sampling. Gibbs samplers generate a Markov chain of samples where nearby samples are correlated and subsampling the chain of samples results in obtaining samples (here document distributions) that are less correlated and therefore more effective. Thinning allows to have independent samples. In addition, discarding the samples from the beginning of the chain is called burn-in, and we applied it to the first 149 samples, as they may not accurately represent the desired distribution.

We evaluate the presented topic models using the task of document classification. We use Support Vectors Machines (SVMs) as our learners, and in particular the implementation of Scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in python[END_REF]. We set the regularization parameter C = 10, after cross-validating it in the training parts of the datasets from the range C ∈ {10 -4 , 10 -3 , . . . , 10 4 , }. For the multi-label datasets (TED, Reuters and PubMed) we employed one-versus-rest: the SVMs return every category with a positive distance from the separating hyper-planes. Table 4.7 reports the classification scores for the micro-averaged F 1 measure for the datasets used. The F 1 measure is the harmonic mean of precision and recall:

F 1 = 2 • precision • recall precision + recall ,
where precision is the fraction between true positives and the sum of true positives and false positives, while recall is the fraction between true positives and true positives and false negatives. Micro-averaging refers to summing up the individual true positives, false positives, and false negatives of the system for different categories and aggregating them in order to get the total precision and recall. The reported scores in Table 4.7 are the averages of 10-fold cross-validation and the corresponding standard deviations are shown as subscripts.

As one can note, the highest classification scores are obtained using topic distributions obtained with copulas. Importantly, the addition of copulas consistently improves the scores of the MiF 1 evaluation measure. Notice the benefits of copulas in the table: the systems that use copulas when sampling the topics of the words of the coherent segments consistently achieve higher scores than their counterparts that do not use copulas. Importantly, those systems perform better that LDA, which builds on the bag-of-words representation. The latter highlights that incorporating parts of text structure is advantageous for applications that use document representations learned with topic models. Overall, the results suggest that assuming short segments like noun-phrases, bigrams and trigrams to be coherent is the optimal option with respect to the performance.

Our last observation concerns the low performance of the SVMs on the TED dataset. We believe that this is due to the fact that TED is a multilabel dataset with 2.42 categories/instance (Table 4.1), which is much higher that the rest of the multilabel dataset. The results suggest that an approach like one-versus-rest we used, can not model this successfully and one should employ a better suited multi-label strategy. Although interesting, multi-label classification was not the main topic of this chapter and therefore we do not explore more this direction.

Summary

In this chapter we presented segmentLDA and copulaLDA, two novel topic models that incorporate prior knowledge of text structure in the form of boundaries of coherent segments. copulaLDA uses copulas when sampling the topics of the words of segments and thus allows few, related topics to appear in a segment. On the other hand, segmentLDA applies a maximal binding between the topics of the words in a segment and, as a result, a single topic is assumed to generate its words.

To evaluate different versions of the proposed model, where we considered various types of segments to be thematically coherent, we performed a systematic comparison of the performance of the models: we compared their performance with respect to the generalization performance measured by perplexity, topical coherence measured by normalized pointwise mutual information (nPMI) and text 69 classification performance measured by the MiF 1 scores. Our results strongly suggest that prior knowledge of text structure benefits the coherence of the produced topics as well as the quality of the learned per-document topic distributions. Our analysis further suggests that, in practice, assuming short text spans like bigrams and trigrams is optimal: apart from the fact that such segments achieved the best performance in most of the evaluation tasks, to obtain them one does not need resort to linguistic tools like parsers but can rely only on counting operations. Lastly, the use of copulas is advised mostly for cases where the output of the topic models will be used for a real task, like text classification, which is though the most interesting ones.

While incorporating segments with or without the use of copulas has been shown to be beneficial, one should take into account that such methods require an overhead for segmenting the documents. While the segmentation methods we presented can benefit from parallelization in a straight-forward way, this needs to be considered. Further, sampling from copulas imposes a further overhead: not only one needs sample from complex distributions [START_REF] Hofert | Nested Archimedean Copulas Meet R: The nacopula Package[END_REF][START_REF] Hofert | Efficiently Sampling Nested Archimedean Copulas[END_REF], but also needs to transform the sample using the probabilistic integral transform as discussed in the previous sections. In conjunction with the extra free parameters (copulas family, λ, segmentation methods) that these models introduce, one may experience a significant overhead tuning the models for production purposes.

Our findings open various avenues for future research. The computational overhead discussed in the previous paragraph motivates future work on accelerating inference such as combining copulas with variational inference, which to the best of our knowledge has yet to be achieved. A second question that is raised is whether one can use these findings to improve multilingual topic models. This question in fact motivates part of the contributions concerning text structure and bilingual topic models presented in the next chapter. Another question concerns the segmentation approach used. While here we relied on frequency-based approaches like n-grams and syntactic information like noun-phrases, one may ask if similar results can be obtained using an unsupervised segmentation approaches or approaches that learn the segment boundaries and the topics jointly.

71

Chapter 5

Extending Bilingual Topic Models D ue to the ever increasing amount of multilingual content online, people are more and more confronted with documents available in more than one language. An important challenge when developing systems for such multilingual document corpora is to automatically discover and extract meaningful topics that help to better organize them and comprehend their content. Following the success of topic models in the monolingual setting, there have been recent efforts that extended them to the bilingual (or multilingual) setting.

Those extensions of topic models that account for text written in two or more languages enable a variety of interesting multilingual and cross-lingual applications.

In the previous chapter we discussed that probabilistic topic models like Latent Dirichlet Allocation (LDA) [START_REF] David M Blei | Latent dirichlet allocation[END_REF] are a family of unsupervised models that when applied to monolingual collections uncover the latent themes underlying it. Despite their success and wide adoption, we identified some of their limitations and suggested extensions to overcome them. In particular, we argued that while the bag-of-words assumption may be important during inference as it simplifies the calculation of the conditional probabilities, a more complex document model may be advantageous. Our contributions were motivated by the outcomes of linguistic or statistic pre-processing steps for text segmentation. We proposed two novel topic models that incorporate parts of the text structure in the form of boundaries of text spans and we demonstrated in a plethora of evaluation tasks the improvements that those models yield.

Our focus in this chapter is bilingual topic models. The most representative bilingual topic model is illustrated in Figure 5.2(i) and is commonly called bilingual 1 LDA (BiLDA) [START_REF] Mimno | Polylingual topic models[END_REF][START_REF] De | Knowledge transfer across multilingual corpora via latent topics[END_REF][START_REF] Vulić | Probabilistic topic modeling in multilingual settings: An overview of its methodology and applications[END_REF]. BiLDA, which as also presented in Section 2.3. extends LDA and does not require any prior, language dependent linguistic knowledge but the input collection to be in the form of pairs of thematically aligned documents. Given the pairs, a fundamental hypothesis of BiLDA is that the documents of a pair share a single per-document topic distribution θ, an observation we also highlighted in Section 2.3.2. This entails that the documents in a pair discuss exactly the same themes. Although reasonable for parallel corpora, whose pairs consist of documents that are translations, this assumption is strong for collections composed by pairs of comparable documents (e.g. [START_REF] Mcenery | Parallel and comparable corpora: What are they up to? Translating Europe[END_REF]), that is documents similar to some extent only. Figure 5.1 illustrates an example of comparable documents written in English and Portuguese. 2 As the English document is larger one would expect it to cover more topics. Hence, having a shared topic distribution between those two documents is a strong assumption.

In this chapter we propose to extend bilingual topic models. On one hand, our goal is to relax the assumption of comparable documents sharing a single topic distribution and better adapt bilingual topic models for corpora consisting of documents like those of Figure 5.1. For this purpose, instead of a shared distribution we allow the documents of a pair to have two, separate, yet bound distributions. We suggest that the strength of the bound should depend on the semantic similarity of the documents of the pair. The estimation of this similarity for documents written in different languages is a task on itself. Instead of using dictionaries, which are one-to-one or one-to-N discrete word associations and do not capture different levels of similarity, or machine translation systems, which are computationally expensive to develop, we propose to use cross-lingual word embeddings.

On the other hand, motivated by our discussion in Chapter 4, where we found or multilingual LDA.

α θ z 2 z 1 w 2 w 1 φ 2 φ 1 β N 1 N 2 D K K α θ 2 θ 1 λ z 2 z 1 w 2 w 1 φ 2 φ 1 β N 1 N 2 D K K α θ z 2 z 1 w 2 w 1 φ 2 φ 1 β N 1 S 1 S 2 N 2 D K K α θ 2 θ 1 λ z 2 z 1 w 2 w 1 φ 2 φ 1 β N 1 S 1 N 2 S 2 D K K Figure 5.2:
The topic models used in this work. From left to right: (i) BiLDA (ii) segBiLDA (iii) λ-BiLDA (iv) λ-segBiLDA. The difference of BiLDA and segBiLDA from their λ-counterparts lies on the fact that the second have separate but bound topic distributions and the strength of binding is controlled by λ.

text structure to benefit topic models, we aim at validating our findings in the multilingual setting. The arguments that support incorporating text structure in topic models hold independently of the language: words that frequently co-occur (e.g., frequent n-grams) or words that are grouped together due to the syntax (e.g., noun-phrases) should be topically coherent in every language.

The questions we attempt to answer here are twofold: (Q1) How to better adapt bilingual topic models to comparable collections? (Q2) Does this adaptation generalize well across different types of topic models? To address these questions, the chapter proposes three contributions: (i.) a novel approach that combines topic models with (shallow) neural networks for learning word embeddings allowing the former to extract latent distributions from comparable corpora, (ii.) the extension of BiLDA and of a monolingual topic model that incorporates text structure in the form of boundaries of coherent text spans, (iii.) a systematic evaluation of the novel topic models on four comparable corpora where English are paired with French, German, Italian, Spanish and Portuguese documents.

The remainder of the chapter is organized as follows: Section 5.1 presents an overview of the related work. The main contribution of this chapter is presented in Section 5.2, where we propose to incorporate information of text structure into bilingual topic models and also extend them for comparable corpora. Then, in Section 5.3 we evaluate the presented topic models and we conclude in Section 5.4.

An overview of the relevant literature

Our work lies in the intersection of the fields of multilingual topic modeling and cross-lingual word embeddings. We review the relevant literature starting with the work on multilingual topic models.

Multilingual Topic Models There are two different lines of research in the multilingual topic modeling approaches with respect to the nature of the available training inputs. The first line assumes access or attempts to create linguistic resources such as dictionaries, in order to identify the topical links and alignments between the multilingual documents of a text corpus [START_REF] Boyd-Graber | Multilingual topic models for unaligned text[END_REF][START_REF] Jagarlamudi | Extracting multilingual topics from unaligned comparable corpora[END_REF][START_REF] Zhang | Cross-lingual latent topic extraction[END_REF][START_REF] Boyd | Holistic sentiment analysis across languages: Multilingual supervised latent dirichlet allocation[END_REF]. The topic alignments between documents are not implicit in the input, and the models identify the topically relevant multilingual documents and the topic distributions while leveraging the available linguistic resources. For instance, [START_REF] Boyd-Graber | Multilingual topic models for unaligned text[END_REF] propose the multilingual topic model for unaligned text (MuTo) that discovers a parallelism in the documents of the corpus at the vocabulary level, while it assumes that similar themes are expressed in both languages. To perform the joint task of producing consistent topics in each of two languages and then aligning them, the model uses dictionaries. JointLDA is a model with similar motivations, proposed in [START_REF] Jagarlamudi | Extracting multilingual topics from unaligned comparable corpora[END_REF]. To cope with the multilingual setting, jointLDA also uses dictionaries but learns topics shared among the input languages. Those topics are distributions over the vocabulary terms of the multilingual corpus, and as a result, terms of different languages occur in a topic. Despite the advantages of such models, their requirements for several language-specific resources can be seen as a limitation. Also, [START_REF] Boyd | Holistic sentiment analysis across languages: Multilingual supervised latent dirichlet allocation[END_REF] uses multilingual topic models and incorporates a regression task like sentiment prediction to better predict sentiment.

The second, more flexible line of research, investigates topic modeling solely on the basis of the textual inputs. Those inputs, usually consist of text corpora with documents that are either parallel translations [START_REF] Zhao | Bitam: Bilingual topic admixture models for word alignment[END_REF] or comparable translations of each other [START_REF] Ni | Mining multilingual topics from wikipedia[END_REF][START_REF] Mimno | Polylingual topic models[END_REF][START_REF] De | Cross-language linking of news stories on the web using interlingual topic modelling[END_REF][START_REF] John C Platt | Translingual document representations from discriminative projections[END_REF]. Such topic models by not relying on any external resource are a better fit for unsupervised methods. The most representative model of this family is BiLDA, which extends LDA in the bilingual [START_REF] Ni | Mining multilingual topics from wikipedia[END_REF][START_REF] Mimno | Polylingual topic models[END_REF][START_REF] De | Cross-language linking of news stories on the web using interlingual topic modelling[END_REF] or the multilingual setting [START_REF] Krstovski | Online polylingual topic models for fast document translation detection[END_REF][START_REF] Mimno | Polylingual topic models[END_REF]. The difference between the bi-or multilingual setting lies on the number of the input languages, which ranges from two to several. The model we propose in this work belongs to this family of models as it assumes access to a corpus whose input documents form theme-aligned pairs. However, our model is more flexible as instead of assuming a single topic distribution per pair of documents uses two topic distributions that are linked with a binding mechanism that uses cross-lingual word embeddings to account for the level of similarity between the documents. Similarly, Latent Semantic Allocation has been extended in the bilingual setting [START_REF] Tam | Bilingual lsa-based translation lexicon adaptation for spoken language translation[END_REF] with applications to translation or [?] proposed to use a PLSA model per-language and concatenate the learned representations; we focus however on BiLDA as in the monolingual setting LDA was shown to outperform LSA [START_REF] David M Blei | Latent dirichlet allocation[END_REF] and one should expect similar findings in the bilingual setting.

Cross-lingual Word Embeddings

According to the distributional hypothesis, first stated in the early 60's [START_REF] Rupert | A synopsis of linguistic theory[END_REF][START_REF] Zellig | Distributional structure[END_REF], linguistic items such as words with similar distributions should have similar meanings. In other words, semantically similar words should have similar contextual distributions. The contextual information is usually induced assuming the context to be documents or sliding windows and is represented by populating word-context co-occurrence matrices. For words, different models that learn distributed representations have been recently proposed and those models are used as implementation models of the distributional hypothesis [START_REF] Turian | Word representations: a simple and general method for semi-supervised learning[END_REF]. To this end, the distributed representations (also known as word embeddings) associate words with dense vectors, of dimension of a few hundreds to some thousands. A distributed representation of a symbol is a vector of features that characterize the meaning of the symbol and in our case a symbol is a word. The representation is a continuous D-dimensional vector and, therefore, compact in the sense that an exponential number of symbols (words) in the number of dimensions can be efficiently represented [START_REF] Turian | Word representations: a simple and general method for semi-supervised learning[END_REF], compared, for instance, to the one-hot-encoding scheme that can only represent D symbols when using D dimensions.

Among different models, the skipgram model with negative sampling [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF],

has been shown to be efficient and effective in several applications. Such a model is a function f that projects a word w in a D-dimensional space: f (w) ∈ R D , where D is predefined. Although the model relies on a well-defined prediction task [START_REF] Baroni | Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors[END_REF], it has been shown that it is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) (Chap. 4, Eq. 4.10) of the respective word and context pairs, shifted by a global constant [START_REF] Levy | Neural word embedding as implicit matrix factorization[END_REF][START_REF] Li | Word embedding revisited: A new representation learning and explicit matrix factorization perspective[END_REF]. Despite the theoretical equivalence however, an advantage of the skipgram model compared to other models that factorize such matrices like Latent Semantic Indexing [START_REF] Hofmann | Probabilistic latent semantic indexing[END_REF] is its ability to practically scale to huge amounts of data.

The skipgram model was initially proposed for the single language setting.

However, motivated by the idea of having a single representation space shared by more languages, cross-lingual word embeddings models extended the idea in the bilingual and multilingual settings. The models can be grouped with regard to the approach used to align the cross-lingual embeddings. Models like [START_REF] Mikolov | Exploiting similarities among languages for machine translation[END_REF] followed by [START_REF] Xing | Normalized word embedding and orthogonal transform for bilingual word translation[END_REF][START_REF] Lazaridou | Hubness and pollution: Delving into cross-space mapping for zero-shot learning[END_REF] for instance, begin by learning monolingual word embeddings and try to learn a linear transformation from one space to the other. Another way to learn cross-lingual embeddings [START_REF] Gouws | Simple task-specific bilingual word embeddings[END_REF][START_REF] Duong | Learning crosslingual word embeddings without bilingual corpora[END_REF][START_REF] Vulić | Bilingual distributed word representations from document-aligned comparable data[END_REF] is by artificially generating multilingual documents by concatenating the documents of parallel or comparable corpora and then training existing monolingual models. Lastly, models like [START_REF] Luong | Bilingual word representations with monolingual quality in mind[END_REF][START_REF] Gouws | Bilbowa: Fast bilingual distributed representations without word alignments[END_REF] perform joint optimization of monolingual and cross-lingual losses. They can benefit from very big monolingual corpora for optimizing their monolingual objectives while relying on smaller corpora for optimizing their cross-lingual objective.

In this work we use Bilbowa [START_REF] Gouws | Bilbowa: Fast bilingual distributed representations without word alignments[END_REF]. It belongs to the family of models that jointly optimize monolingual and cross-lingual objectives. It extends the skipgram model for cross-lingual embeddings and trains directly on monolingual data. It uses a bilingual signal from a smaller set of raw-text sentence-aligned data to align the cross-lingual embeddings.

Combining Topic Models and Word Embeddings While topic models are trained to infer the per-word and per-document topic distributions, the skipgram model is trained by trying to predict the context of a word. Different efforts have attempted to extend the models by combining them. For instance, embeddings associate words with a single vector, which may be limiting for encoding the different meanings of polysemous words. This limitation motivated works that extend word embeddings with topic models. The purpose is for the topic models to uncover the different senses of a word, so that different embeddings can be derived for each sense [START_REF] Liu | Topical word embeddings[END_REF][START_REF] Cheng | Contextual text understanding in distributional semantic space[END_REF]. Such efforts attempt to produce better performing word embeddings.

In a relevant line of work, the purpose is to produce better topic models while taking advantage of the fact that text embeddings model semantic similarity. To The number of words in d i assigned to topic k θ i Topic distribution of

d i Ψ k,w
The number of assignments of word w to topic k φ k

Word distribution for topic k s i,j

The j th segment of document d i w i,j,k

The k th word of segment s i,j N i

The number of words of document

d i N i,j
The number of words in segment s i,j N i,j,w

The number of occurrences of word w in s i,j this end, [START_REF] Dat Quoc Nguyen | Improving topic models with latent feature word representations[END_REF][START_REF] Das | Gaussian LDA for topic models with word embeddings[END_REF][START_REF] Zhang | Improving short text classification by learning vector representations of both words and hidden topics[END_REF] extend topic models in order to encourage the models to group words that are a priori known to be semantically related into topics, where the a priori knowledge comes from training embeddings in large external corpora.

Our work belongs to this second line of research because we use word embeddings to improve the results of topic models. Differently from previous research though, word embeddings are used only to estimate the similarity of documents written in different languages. Also, our models are multilingual, while previous work investigated the intersection of topic models and word embeddings in the monolingual setting using English.

Framework

Our primary goal in this work is to adapt the bilingual topic models for comparable corpora. To accomplish that we relax the assumption of paired documents having identical topic distributions. In the rest of this section, after presenting the notation, we briefly discuss BiLDA in Section 5.2.1. To illustrate how several classes of topic models can benefit by the adaptation to comparable corpora, we introduce a novel bilingual topic model that incorporates parts of the document's structure (Section 5.2.2). We, then, extend BiLDA and the novel bilingual topic model for comparable corpora in Section 5.2.3 and Section 5.2.4.

The notation is summarized in Table 5.1. For consistency, we keep the symbols of previous work [START_REF] Wang | Distributed gibbs sampling of latent topic models: The gritty details[END_REF] to the extent of possible. We denote by 1 and 2 the different languages of a comparable corpus. As languages are handled symmetrically, for convenience we designate by \, the language different from language ∈ { 1 , 2 }. The inputs of the topic models are document pairs

d i = (d 1 i , d 2 i
), that consist of thematically aligned documents d 1 i and d 2 i , written in 1 and 2 . Depending on the model, documents are either represented as a bag-of-words, or as a bag-of-segments. Segments are text spans smaller than documents, for instance sentences, and are represented as a bag-of-words. Considering ∈ { 1 , 2 }, s i,j is the j th segment of document d i . Segmented documents have a hierarchical structure: they are composed by segments that are composed by words in turn. Depending on the model, there may exist either a single θ i topic distribution that captures the topics present in both documents of the pair d i , or two, separate yet bound topic distributions θ i , θ \ i that capture the topics of d i and d \ i respectively. The rest of the notation in Table 5.1 stands for count matrices or vectors used during inference.

The bilingual LDA

BiLDA (Figure 5.2(i)) is a direct adaptation of LDA in the bilingual setting where a parallel collection is assumed to be the model's input. Due to its effectiveness we use it as a reference in this work. BiLDA assumes that the documents of an aligned pair d i have identical topic distributions (a single and shared θ i ) and therefore discuss the same topics. Also, it expects the documents as a bag-of-words. Its generative story is as follows:

• for each topic k ∈ [1, K]: φ 1 k ∼ Dir(β), φ 2 k ∼ Dir(β)
• for each document pair d i :

sample θ i ∼ Dir(α)

-for each language ∈ { 1 , 2 }
for each of the N i words:

• sample z ∼ Mult(1, θ i ) • sample w ∼ Mult(1, φ z )
The collapsed Gibbs sampling updates [START_REF] Vulić | Probabilistic topic modeling in multilingual settings: An overview of its methodology and applications[END_REF] for the topic of word j of docu-

ment d i is ∀ ∈ { 1 , 2 }: P z ij = z k |z ¬ij , z \ , w , w \ , α, β, ∝ Ψ k,w,¬ij + β Ψ k,•,¬ij + V β Ω i,k,¬ij + α
A dot "•" occurring in the subscript of a count variable, stands for the summation over the possible values of the element it replaces, i.e.,Ψ k,

•,¬ij = V w=1 Ψ k,w,¬ij .
Also, ¬ excludes the counts for the particular variable (¬ij excludes the counts of the j-th word of d i ). Further, Dir(α) stands for a sample from a Dirichlet distribution with prior α and Mult(M, θ) stands for M samples from a Multinomial distribution parametrized by θ.

For BiLDA, as well as for the models we present next, we consider the Dirichlet hyperparameters α ∈ R K and β ∈ R V to have fixed values, implying symmetric priors. Extending the models to asymmetric priors or even learning their values could be done as in [START_REF] Asuncion | On smoothing and inference for topic models[END_REF] for example. Also, as commonly done we omit from the generative stories the steps where the sizes of segments or documents are sampled as their sizes are observed during inference. As noted, BiLDA uses a bag-of-words representation; next we present an extension that uses a more complex document representation.

Incorporating text structure into bilingual topic models

In this section, we propose segment-BiLDA (segBiLDA) that incorporates prior knowledge of text structure using a more complex document representation than bag-of-words. Although important for inference, the bag-of-words assumption is limiting. In fact, previous research in the single language domain showed the benefits of similar extensions: Wang et al. [START_REF] Wang | Topical n-grams: Phrase and topic discovery, with an application to information retrieval[END_REF] proposed a model that handles bigrams as a single token or as two unigrams depending on the topic, Lau et al. [START_REF] Jey | On collocations and topic models[END_REF] modeled frequent bigrams as separate tokens, Balikas et al. [START_REF] Balikas | On a topic model for sentences[END_REF] proposed to incorporate sentence boundaries to LDA, while Boyd et al. [START_REF] Blei | Syntactic topic models[END_REF] incorporated parse trees . These important contributions focused on the monolingual setting and used English texts for empirical evaluation. Here, we extend topic models to account for text structure in the bilingual case.

For our subsequent analysis, we define coherent text segments to be contiguous words of a document that are topically coherent. A topically coherent text segment refers to a segment whose constituent words discuss a single or very few related themes. For instance, one would expect frequent bigrams like "information retrieval" or even short sentences to be topically coherent as they generally convey a simple message. We model this property with (segBiLDA), which is illustrated in Figure 5.2(ii). segBiLDA assumes that the input text is segmented a priori and incorporates the boundaries of segments in its generative story:

• for each topic k ∈ [1, K]: φ 1 k ∼ Dir(β), φ 2 k ∼ Dir(β)
• for each document pair d i :

sample θ i ∼ Dir(α)

-for each language ∈ { 1 , 2 }
for the j segment (1 ≤ j ≤ S ):

• sample z ∼ Mult(1, θ i ) • sample segment words: (w 1 . . . w N i,j ) ∼ Mult(N i,j , φ z )
The important difference of BiLDA from segBiLDA (Figures 5.2(i) and 5.2(ii)) lies in the addition of the segment's plate. A topic is sampled per segment, and every word of a segment is associated with it. The segment boundaries limit the number of topics that appear in the segment to be equal to one. As in BiLDA though, words remain the document units and this single topic is associated with each word of the segment. Therefore, comparing the models on measures like perplexity that are calculated at the word level is fair. To infer these topics we propose a collapsed Gibbs sampling approach, that ∀ ∈ { 1 , 2 }, samples topics from: When sampling the topic z 1 i,j of the words of the segment s 1 i,j one has:

sample z 1 i,j ∼ p z 1 i,j = z k |z 1 ¬s i,j , z 2 , . . . , z M , w 1 , . . . , w M , α, β ∝ θ φ p(z 1 i,j = z k |z 1 ¬s i,j , z 2 , . . . , z M , θ, α)× p(w 1 s i,j |z 1 i,j = z k , z 1 ¬s i,j , w 1 ¬s i,j , φ, β)dφdθ ∝ θ p(z 1 i,j = z k |θ)p(θ|z 1 ¬s i,j , z 2 , . . . , z M , α)dθ× φ w∈s 1 i,j p(w|z 1 s i,j = z k , φ)p(φ|z 1 ¬s i,j , w 1 ¬s i,j , β)dφ = θ i p(z 1 i,j = z k |θ)p(θ|z 1 ¬s i,j , z 2 , . . . , z M , α)dθ i × w∈s 1 i,j φ k p(w|z 1 s i,j = z k , φ)p(φ k |z 1 ¬s i,j , w 1 ¬s i,j , β)dφ k , (5.1)
where z 1 ¬s i,j denotes the topic assignments of d 1 i excluding those referring to s 1 i,j . For the integrals of Eq. (5.1) using the Multinomial-Dirichlet conjugacy for φ and θ, one may perform the updates using their expectations, hence :

E Dir(Ω 1 k,i,¬s i,j +Ω 2 k,i +α) [θ i,k ] × w∈s 1 i,j E Dir(Ψ 1 k,w,¬s i,j +β) [φ (w) k ],
where the first term is influenced by the the topic assignments within the documents of the pair, and the second by the words of the segment. The result for the conditional probability then becomes:

p(z s i,j = z k |z ¬s ij , z \ , w , w \ , α, β) ∝ Ω i,k,¬s i,j + α × w∈s ij (Ψ k,w,¬s ij + β) • • • (Ψ k,w,¬s ij + β + (N i,j,w -1)) (Ψ k,•,¬s ij + βV ) • • • (Ψ k,•,¬s ij + βV + (N i,j -1)
) .

(5.2)

In Eq. (5.2), the product appearing in the numerator of the second term results from the bag-of-words assumption for the words of segments. The (possibly multiple) occurrences of a word w in a segment s i,j , generated by the topic k, are taken into account by the factor (Ψ k,w,¬s ij + β), which is incremented by one for every other occurrence of the word after the first. For example, if word w appears twice in s i,j , then N i,j,w = 2, and the factor (Ψ k,w,¬s ij + β)(Ψ k,w,¬s ij + β + 1) denotes the contribution of the occurrences of the word to the probability that s i,j is generated by the topic k. This way, every word of the segment contributes to the probability of sampling a particular topic. Similarly, the denominator acts as a normalization term. The progressive increase of its values can also be explained intuitively: given the bag-of-words assumption of words within a segment, the product normalizes the probability of assigning the topic k to a word of the segment, given that the previous words have also been assigned to this topic. Notice, that if the size of the segment is 1, the model as well as the sampling equations reduce to BiLDA.

The bag-of-words assumption in BiLDA results in a joint distribution of random variables (here topics) being invariant to any permutation of the variables (exchangeability). This holds for segBiLDA only locally, within segments. Globally, within a document, words are not exchangeable as the segment boundaries are utilized. While in BiLDA the topics of words are conditionally independent given the document's topic distribution, for segBiLDA they are not, as they also depend on the rest of the segment's words.

Previous work in the monolingual case suggested to incorporate various types of text structure to topic models ranging from n-grams to parse trees. segBiLDA can be considered an extension of our model [START_REF] Balikas | On a topic model for sentences[END_REF] in the bilingual setting. Rather than extending more complex models like Boyd's [START_REF] Blei | Syntactic topic models[END_REF] that may require parsing the documents, we opt for segBiLDA due to the variety of segments it can handle. For instance, one can use linguistically motivated segmentation approaches like sentence tokenization or statistically motivated segmentation approaches like frequent n-grams with the same model. Furthermore, these segmentation approaches can be accomplished efficiently and accurately across a wide range of languages without resorting to complex linguistic analysis tools like parsers.

Extracting multilingual topics from comparable corpora

BiLDA and segBiLDA assume a single topic distribution for the documents of a pair, which as illustrated in Figure 4.1 is a string assumption for comparable documents. Apart from that, the motivations for adapting the bilingual topic models to comparable corpora lie on two facts: on one hand, comparable corpora are more common and easy to obtain or to construct than parallel ones, which require additional linguistic knowledge and tools. On the other hand, recent advances on cross-lingual word embeddings resulted in methods that can be directly used to estimate the semantic similarity of documents written in different languages. The latter facilitates the application of our method to various pairs of languages without expensive resources.

For comparable corpora, we first propose the λ-BiLDA model, whose graphical model is shown in Figure 5.2(iii). In this case, instead of having a single, shared topic distribution we have a topic distribution per language shown as θ 1 and θ 2 in the figure. However, these distributions are bound between them. To model naturally the possible levels of dependence between θ 1 and θ 2 we need a binding mechanism flexible enough to model the two extreme conditions: total independence between the topic distributions of the aligned documents that should result in two distinct LDA models (one per language), and a complete dependence between them (identical topic distributions) which should result in BiLDA. Similar dependence mechanisms were previously explored under the setting of streaming documents [START_REF] Hesam Amoualian | Streaming-lda: A copula-based approach to modeling topic dependencies in document streams[END_REF], where topic distributions of earlier documents affect the distributions of later documents.

The generative process for λ-BiLDA is as follows:

• for each topic k ∈ [1, K]: φ 1 k ∼ Dir(β), φ 2 k ∼ Dir(β)
• for each document pair

d i = (d 1 i , d 2 i ):
estimate λ i with respect to the documents that form the pair

d i = (d 1 i , d 2 i ) -sample θ 1 i |θ 2 i ∼ Dir(α + λ i θ 2 i ), θ 2 i |θ 1 i ∼ Dir(α + λ i θ 1 i ) -for language ∈ { 1 , 2 }
for each of the words N :

• sample z ∼ Mult(1, θ i ) • sample w ∼ Mult(1, φ z )
The central idea is that the topic distributions of documents in one language depend on the topic distributions of documents in the other language via a binding mechanism that generates θ with a Dirichlet distribution depending on θ \ ; θ |θ \ ∼ Dir(α + λ i θ \ i ) and vice-versa. Note that from the Hammersley-Clifford theorem [START_REF] John | Markov fields on finite graphs and lattices[END_REF], fixing the two conditional distributions θ 1 |θ 2 and θ 2 |θ 1 defines in an unique way the distribution of (θ 1 , θ 2 ) which implies that our generative process is welldefined.

For inferring the topics of the observed words we propose a Gibbs sampling approach, whose derivation is given in the Appendix at the end of this chapter.

The update equations for the topics of the words are then ∀ ∈ { 1 , 2 }:

p z i,j = z k |z ¬i,j , w , α, β, λ i , θ \ ∝ Ψ k,w,¬i,j + β Ψ k,•,¬i,j + V β • Ω i,k,¬i,j + α + λ i θ \ d,k . (5.3)
Gibbs sampling algorithms obtain posterior samples by sweeping though each block of variables and sampling from their conditional, while the remaining blocks are fixed. In practice, the algorithm initializes randomly the topics of words. Then, during the Gibbs iterations and until convergence, sampling topics for words of assumes the distribution of θ \ fixed, and hence can be accessed as assumed by the generative story.

In Eq. (5.3), λ i captures the dependency between the topic distributions of the documents of d i . We use cross-lingual word embeddings for its estimation. We use the average (avg) compositional function of meaning, which was shown to be robust and effective [START_REF] Mitchell | Composition in distributional models of semantics[END_REF][START_REF] Blacoe | A comparison of vector-based representations for semantic composition[END_REF]. Having the vectors of the document pair

d i = (d 1 i , d 2 i
) in the embedded space, we then estimate λ i using the cosine similarity. Calculated in this way, λ i ∈ [-1, 1] and since θ d,k ∈ [0, 1] it follows for the second term of Eq. 5.3 that Ω i,k,¬i,j >> λ i θ d,k , which results in negligible impact for λ i θ d,k . To circumvent that we use:

λ i = λ i × |N \ i | = Ω \ d,k .
(5.4)

Notice that incorporating Eq. ( 5.4) to Eq. ( 5.3) has as an additional advantage as the model generalizes previous models. In particular, it follows that BiLDA becomes a special case of λ-MuLDA with λ = 1 (complete dependency where θ and θ \ are the same topic distributions). Also, when λ = 0 (case of independence) we have two distinct LDAs, one per language. 3

Combining the two models

To this point, we proposed segBiLDA that incorporates the boundaries of coherent segments like sentences, and λ-BiLDA, that assumes bound topic distributions for the paired documents in the two languages. The two models can be combined: λ-segBiLDA assigns consistent topics in the words of the segments of the documents and also assumes different topic distributions for each language. We illustrate λ-segBiLDA in Figure 5.2(iv). We omit the generative story, since it is a direct combination of the generative stories of segBiLDA and λ-BiLDA. The inference process is given by the following equation, whose derivation is given in the Appendix at the end of the chapter. To sample the topics of segments from the conditional distribution for ∀ ∈ { 1 , 2 }: Calculate the probabilities of the new topic of the words of s i,j (Eq. 5.5)

p z i,j = z k |z ¬i,j , w , α, β, λ i , θ \ ∝ [Ω i,k,¬s i,j + α + λ i Ω \ d,k ]× w∈s ij (Ψ k,w,¬s ij + β) • • • (Ψ k,w,¬s ij + β + (N i,j,w -1)) (Ψ k,•,¬s ij + βV ) • • • (Ψ k,•,¬s ij + βV + (N i,j -1)) . ( 5
d i = (d 1 i , d 2 i ), i ∈ [1,
Sample the topics of the words of s i,j using the calculated probabilities Increase counters Ψ , Ω end end end also presented in Algorithm 3. One may obtain similar algorithms for each of the bilingual topic models presented in this section or adapt Algorithm 3 by selecting the appropriate equation while calculating the probabilities of the topics.

Experimental Framework

The comparable corpora In order to evaluate the proposed models, we perform a series of evaluation tasks using Wikipedia documents in four language pairs as our comparable corpora. The language pairs are English-French (En-Fr), English-German (En-Ge), English-Italian (En-It) and English-Portuguese (En-Pt). Table 5.2 shows the basic statistics of these datasets. For topic modeling purposes we have sampled subsets from the full datasets (right part of the table) consisting of 10,000 documents for each pair. Since the sampling was random, it is not the same 10,000 English documents used for every language pair. Notice in the table that for each pair English is the language with the most words (N ), which was expected since often Wikipedia lemmas are first written in English and then translated to other languages. This is also why Wikipedia is a suitable comparable corpus; the English version usually includes more information on a topic compared to other languages. To extract comparable Wikipedia documents one can use the inter-87 Table 5.2: Data used for evaluating topic coherence (left) and topic modeling (right) purposes. The names signify the language pair and the language that the statistics correspond to. For instance, Wiki En En-Fr are the English documents of the En-Fr corpus. language links. For the sake of reproducibility, we have used the bilingual corpora as made available by linguatools. 4 We have cleaned the documents to remove html tags and tables using Python v2.7 and Beautiful Soup v4.5.1. 5 The statistics of Table 5.2 are after the pre-processing steps, that include lower-casing, filtering the numerical terms out, stemming using the WordNet stemmer as implemented in [START_REF] Bird | Natural language processing with Python[END_REF], stop-word removal using the stopword lists of [START_REF] Bird | Natural language processing with Python[END_REF] and finally filtering vocabulary terms with less than 4 occurrences in the corpus.

The models We evaluate the following six models for each language pair: (i)

BiLDA that has been proposed in [START_REF] Mimno | Polylingual topic models[END_REF] (ii) segBiLDA s that was presented above and uses sentences as coherent segments, (iii) segBiLDA b that is segBiLDA with the 1,0006 most frequent bigrams considered as coherent segments, as well as λ-BiLDA, λ-segBiLDA s and λ-segBiLDA b that extend the first three models for comparable corpora. We have implemented each of these models using Python, Numpy and Scipy. As commonly done, we follow previous work e.g., [START_REF] David M Blei | Latent dirichlet allocation[END_REF] Table 5.3: For each of the topic models we present three topics: City, Sports, Art for the "En-Fr" Wikipedia corpus. Notice the strong intra-semantic (words within a topic) and inter-semantic (topics across languages) coherence.

1 -2 D N 1 V 1 N 2 V 2
En-Fr 1.92M 56M Training Bilbowa. To estimate the word representations of the En-Fr, En-Ge, En-It and En-Pt pairs we used Bilbowa [START_REF] Gouws | Bilbowa: Fast bilingual distributed representations without word alignments[END_REF] to generate a dictionary of word embeddings, where words from two different languages are projected to the same space.

We have used the open implementation of Bilbowa7 with its default parameters and the training epochs set to 10. 8 The model requires parallel text, and for this purpose we used the Europarl corpus [START_REF] Koehn | Europarl: A parallel corpus for statistical machine translation[END_REF]. The statistics of the pairs of languages for the Europarl data are shown in Table 5.4.

Visualizing the topics. As an initial qualitative evaluation of the learned topics, Figure 5.3 presents for 3 topics (City, Sports, Art) the five words with the highest probability for each of the six topic models. The topics were identified after training each model for 200 Gibbs sampling iterations on the Wiki En-Fr corpus with K = 10 topics. Visual inspection of the topics reveals that the models produce topics that are intra-semantically coherent, that is the words that constitute the topics are semantically relevant. Further, the topics are inter-semantically coherent, that is the topics and aligned across languages and closely related words represent them. For instance, the "Sports" topic in English contains mostly word like "team", "play", "season" while in French one can find their (stemmed) translations: "équipe", "jouer", "saison". Although reassuring the visual inspection of the topics in not sufficient to compare the models. In the rest, we evaluate the models intrinsically, that is independently of an application as well as extrinsically in the framework of a cross-lingual document retrieval application.

Intrinsic Evaluation

Normalized PMI As we discussed in the previous chapter (Chapter 4), recent research has showed that calculating the nPMI scores for the topics learned by the models correlates well with human judgments of their quality. Following these results, we present in Table 5.5 the topic coherence scores as measured by the nPMI. Recall, the scores of nPMI range in [-1,1], where in the limit of -1 two words w 1 and w 2 never occur together, while in the limit of +1 they always occur together (complete co-occurrence). As in [START_REF] Jey | The sensitivity of topic coherence evaluation to topic cardinality[END_REF], for each topic, we aggregate the topic coherence scores over three different topic cardinalities: N ∈ {5, 10, 15}. The reference corpora for calculating the topic coherence for each language are the "Full Datasets" of Table 5.2 excluding the "Topic Modeling Subsets". For English we opt for Wiki En En-Fr , which is the biggest, whereas for the rest of the languages we use their respective Wikipedia datasets.

In Table 5.5, note that in most cases λ-segBiLDA b outperforms the rest of the models, while segBiLDA b and segBiLDA s follow. Notice, how BiLDA although competitive for low values of K does not perform as well. This is probably due to the fact that the concept of context is encapsulated in the calculation of the nPMI scores, and the segBiLDA topic models explicitly account for this. In general, increasing the number of topics from 10 to 25 or 50 seems to improve the performance measured by nPMI. For instance, in the lower part of the Although well-correlated with human judgments, for nPMI we only used a small part of the output of topic models, that is for each topic the top-N words.

Furthermore, the evaluation of nPMI suffers in that is does not account for the topical overlap between the learned topics as well as recall gaps within a topic, i.e. lack of terms which should have been ideally included. Therefore, we also report the perplexity scores of held-out documents, whose calculation requires more information from the topic models.

Perplexity Another measure presented in Chapter 4 as an intrinsic measure for comparing topic models is perplexity. In line with our analysis in the previous chapter, we also compare the topic models based on the achieved perplexity scores for each language.

Here, for the perplexity calculations we assume that the held-out documents form thematically-aligned pairs (as during training) and, depending on the topic model, shared or language-dependent per-document distributions are inferred that are used at Eq. (4.9). In the next section, where we will compare the performance of the models in an extrinsic task, we will ignore the links within the pairs to demonstrate than our models perform well under both settings.

Table 5.6 presents the perplexity scores achieved by the topic models. The reported scores are the averages of 10-fold cross-validation as follows: (i) we split the datasets in 10 disjoint sets, (ii) we repeat the training and perplexity calculation steps 10 times, each time considering the i-th set to be the held-out documents and the remaining 9 sets for training. The goal is to exclude any bias due to the split. We present the scores for K ∈ {25, 50, 100, 150}. In terms of perplexity, λ-segBiLDA b and λ-BiLDA clearly outperform the rest of the systems consistently for each language and language pair. The third best performing system is segBiLDA b and BiLDA follows. λ-segBiLDA s and segBiLDA s achieve the worst perplexity scores for every experiment. segBiLDA s , while competitive when evaluated using the nPMI scores, performs poorly in this task. Shown from a different angle, it seems that the systems who build on the bagof-words assumption (BiLDA and λ-BiLDA) consistently outperform those that incorporate the boundaries of large spans like sentences (segBiLDA s and λ-segBiLDA s ).

That was expected as it is in line with previous work [START_REF] Balikas | On a topic model for sentences[END_REF], where incorporating text structure in the form of sentence boundaries was found to lead to higher (worse) perplexity. One the other hand, incorporating the boundaries of smaller spans like bigrams, helps perplexity performance as λ-segBiLDA b seems to be the best performing model overall, especially when the number of topics increases. This is also in line with previous work: [START_REF] Jey | On collocations and topic models[END_REF] showed how bigram boundaries improve the topic model results. In fact, λ-segBiLDA b further improves segBiLDA b which is inspired by [START_REF] Jey | On collocations and topic models[END_REF] since it consistently achieves better perplexity scores.

Another interesting remark concerns the effect of λ, whose goal is to adapt the topic models for comparable corpora. Notice that λ-BiLDA, λ-segBiLDA s and λ-segBiLDA b outperform BiLDA, segBiLDA s and segBiLDA b respectively for each of the experiments and topic values. This highlights the positive effect of the proposed binding mechanism on the achieved perplexity scores. What is more, that was achieved by using a simple yet powerful mechanism (aggregation of word embeddings) for calculating the value of λ for each document pair and these results can be potentially refined when applying more complex strategies. Effectively,

this is the answer to the question (Q2) that the chapter investigates. Adapting topic models for comparable corpora improves their generalization performance and, importantly, these improvements are consistent across different topic models (here BiLDA, segBiLDA s and segBiLDA b ) and different pairs of languages.

Lastly, Figure 5.3 shows the perplexity curves for each language for 200 Gibbs iterations for every language pair and system we evaluated. There are two main observations from the Figure . First, as in Table 5.6, for each experiment λ-BiLDA achieves the lowest perplexity values among the systems that are shown. Second, segBiLDA and λ-segBiLDA are the fastest to converge. They need ∼ 50 iterations to converge, while BiLDA and λ-BiLDA need ∼ 200, that is 4x times more. In terms of computation time, the benefit is similar as the cost of Gibbs iterations are roughly the same.9

Extrinsic Evaluation

Cross-lingual Document Retrieval We conclude the evaluation of the presented topic models by reporting their performance in the framework of a cross-lingual document retrieval application. As discussed during perplexity evaluation, the model can infer the per-document topic distribution for previously unseen data.

Recall, that as The perplexity curves of the four models for the designated datasets for 200 Gibbs sampling iterations when K = 150 topics. In the left column of the figures we visualize the perplexity calculated for the English documents of the comparable pairs while in the right column the documents of the second language of the pair. The proposed model λ-BiLDA consistently outperforms the rest of the systems. may perform inference with a trained model in each language separately, without requiring the explicit links between the documents of a pair. To achieve that, the per-words topic distributions of each language are used. Then, documents with similar topic distributions written in different languages are actually similar due to the inter-semantic coherence of the topics alignments between the learned topics. This is a central observation that enables various cross-lingual applications [START_REF] Vulić | Probabilistic topic modeling in multilingual settings: An overview of its methodology and applications[END_REF] as well as cross-lingual document retrieval.

The task we propose is a cross-lingual document discovery task (CLDD), The goal is to identify counterpart Wikipedia documents due to cross-language links.

In particular, given a document d 1 i as a query, one needs to identify the corresponding document d 2 i . For instance, given an English document for "Dog" the task is to retrieve the German article for "Hund" and, vice versa given the article for "Hund" one must retrieve the article for "Dog".

Following [START_REF] Fukumasu | Symmetric correspondence topic models for multilingual text analysis[END_REF][START_REF] Vulić | Cross-language information retrieval models based on latent topic models trained with document-aligned comparable corpora[END_REF] who found bilingual topic models efficient for the task we address it using the following pipeline. For each of the four language pairs, we train topic models on 9,000 document pairs (18,000 documents). For the remaining 2,000 documents (that is 1,000 pairs of documents) we infer their topic distributions using one language at a time. We consider the cross-language links to be our golden standard. Then, given a document d 1 i whose inferred topic distribution is θ 1 i , we rank every document written at 2 according to the KL-divergence (Kullback-Leibler divergence: [START_REF] Kullback | On information and sufficiency[END_REF]) between θ 1 i and θ 2 j and using the golden links evaluate the performance. The KL-divergence measures the distance of probability distributions and is a suitable distance measure for our case as the topic distributions are probability distributions. We repeat the retrieval experiment 10 times by randomly selecting 500 documents (and their counterparts) out of the 1,000 held-out document pairs. As evaluation measure, we report the scores of Mean Reciprocal Rank (MRR) [START_REF] Voorhees | The trec-8 question answering track report[END_REF] that accounts for the rank of the true positive documents in the returned ranked list. 10 The scores of the MRR evaluation measure are given by:

MRR = 1 |D| |D| i=1 1 rank i
where |D| is the number of documents (queries) at each experiment and rank i denotes the rank of the true document to be retrieved. Further, one has MRR ∈ [0, 1] and the higher the score, the higher the rank of the true positive document in the returned list is. Table 5.7 reports the achieved scores for the document representations inferred for each topic model. The scores are the average performance of the 10 experiments accompanied by the standard deviations. In terms of notation, 1 → 2 (e.g., En→Fr) stands for the experiment where the documents of 1 (e.g., English) are used as queries and the documents of 2 (e.g., French) are to be retrieved. The results of the table clearly establish the improvements on the task due the adaptation of the bilingual topic models for comparable corpora. Notice how λ-BiLDA, λ-segBiLDA b outperform the rest of the models and especially their counterparts BiLDA and segBiLDA b in most of the experiments. The observed improvements are consistent across the language pairs and number of topics K ∈ {25, 50, 100, 150}. This suggests that quantifying the semantic similarity between the documents of the pairs during training led to discovering better topics, whose performance we evaluated in the CLDD task by trying to identify the links of held-out document pairs. It is to be noted that λ-segBiLDA s and segBiLDA s both perform poorly on the task. We believe that this is due to the fact that assuming large spans like sentences in Wikipedia documents to be thematically coherent results in per-document topic distributions unable to capture fine-grained differences between documents. In turn, such fine-grained differences are necessary for achieving high performance on the CLDD task.

Overall, our results suggest that incorporating text structure in the form of short text spans (bigrams) and adapting the bilingual topic models for comparable corpora benefits the performance on CLDD.

Summary

In this chapter we presented two extensions of bilingual topic models. First, motivated by the findings of Chapter 4 concerning prior knowledge of text structure, we proposed to incorporate such knowledge in bilingual topic models. Then, we also identified that a popular version of bilingual topic models can be better adapted to comparable corpora if a robust mechanism for calculating document similarities written in different languages is available. To achieve that we proposed to use cross-lingual embeddings that are known to capture the semantics of words. We evaluated different versions of the novel topic models with regards to Table 5.7: The scores for the CLDD task achieved by the proposed topic models for four bilingual datasets when K ∈ {25, 50, 100, 150}. The best (highest) score achieved per language and k is shown in bold. The topic distributions induced by λ-segBiLDA b achieved the highest MRR scores in most of the experiments. the topic coherence, generalization performance and cross-lingual document retrieval. Our assessment of the performance of the bilingual models suggested that both prior information of short text spans and adapting the models for comparable corpora improved the performance.

In the chapter we showed that by combining topic models with cross-lingual word embeddings one may improve the quality of the learned topics. Our future work in this setting targets to adapt the proposed models to the general setting of comparable corpora. Provided a reliable way such as high quality cross-lingual embeddings to identify links between documents, one may be able to adapt the model to the setting where no documents are not aligned in the input. On the contrary, the alignments are discovered using a document retrieval step like the one presented in the framework of CLDD.

A. Gibbs Sampling Equations for λ-BiLDA and λ-segBiLDA

We derive the Gibbs sampling equations for λ-segBiLDA:

sample z i,j ∼ P z i,j = k|z ¬s i,j , z \ , w , w \ , α, β, λ i , θ , θ \ ∝ θ i φ P z s i,j = k|z ¬s i,j , w , α, β, λ i , θ , θ \ dφdθ i ∝ θ i φ P (z s i,j = k|z ¬s i,j , θ , θ \ , λ i , α) × P (w i,j |z i,j = k, z ¬i,j , w ¬i,j , φ, β)dφdθ i ∝P (z s i,j = k|z ¬s i,j , θ \ , λ i , α) ×             θ i φ P (w s i,j |z s i,j = k, z ¬s i,j , w ¬s i,j , φ, β)dφdθ i            
For the first term, observe that sampling P (z s i,j = k|z ¬s i,j , θ \ , λ i , α) is exactly the same with sampling P (z s i,j = k|z ¬i,j ) in the case of standard LDA, replacing the Dirichlet parameter α with α + λ i θ \ . The derivation of the second term where segments have several words, follows the steps shown on [START_REF] Balikas | On a topic model for sentences[END_REF]. Hence, we deduce that:

98

P z i,j = z k |z ¬s i,j , w , α, β, λ i , θ \ ∝ Ω d,k,¬s i,j + α + λ i θ \ d Ω d,•,¬s i,j + Kα + Kλ i × × w∈s 1 ij (Ψ k,w,¬s ij + β) • • • (Ψ k,w,¬s ij + β + (N i,j,w -1)) (Ψ k,•,¬s ij + βV ) • • • (Ψ k,•,¬s ij + βV + (N i,j -1)) . ( 5.6)
In the last result, for Gibbs sampling the fraction of the first term can be simplified by omitting the denominator as in [START_REF] Heinrich | Parameter estimation for text analysis[END_REF][START_REF] Balikas | On a topic model for sentences[END_REF]:

Ω d,k,¬s i,j + α + λ i θ \ d Ω d,•,¬s i,j + Kα + Kλ i ∼ Ω d,k,¬s i,j + α + λ i θ \ d .
(

Integrating Eq. (5.7) to Eq. (5.6) leads to the desired result.

The equations for λ-BiLDA are simpler as ones does not have segments and the product in Eq. (5.5) and in the subsequent calculations is simplified to a simple term.

99

Chapter 6

Applications of word embeddings to text mining W ord embeddings, whose interesting properties of capturing the semantics of words we investigated in the previous chapter, gained a lot of attention recently. In Chapter 5 we proposed and evaluated a novel bilingual topic model that uses cross-lingual embeddings. Cross-lingual embeddings are a variant of the monolingual embeddings adapted for the case where text is written in several languages. Motivated by the performance benefits we observed, in this chapter we explore their potential in the monolingual setting in the framework of different applications.

Our main hypothesis throughout the contributions of the chapter is that incorporating expressive representations like embeddings on text mining applications should improve the performance of various text mining tasks. In other words, our goal is to develop models tailored for challenging text mining tasks that benefit from rich text representations as well as from the flexibility that neural networks offer in modeling different scenarios. While shallow architectures of neural networks can be used to learn general purpose word embeddings, such learned embeddings can be used as initializations of the first layers of deeper networks instead of using random initializations.

In the rest of the chapter we propose different models and we contribute several observations regarding three interesting tasks. In particular:

1. Learning document representations utilizing translations of a document. Our results in Chapter 5 demonstrated that access to multilingual versions of a document supported by cross-lingual embeddings improved the representations learned with topic models for a retrieval task. Motivated by this, we propose to evaluate the use of embeddings for arbitrary long text spans in the document classification task. In particular, we investigate whether one can use translations of a document to obtain better-performing representations (Section 6.1).

2. Multitask learning using neural networks. While the question we investigate in Section 6.1 concerns the effect of multiple representations of a document for a single task, we are also interested on the effect of learning jointly correlated tasks. Having different representations such as translations of a document may result in better document representations, while having different tasks may result in a form of induced bias towards selecting hypotheses that perform well across tasks. To this end we propose a neural network architecture for jointly learning the hypotheses for two different yet correlated text classification problems (Section 6.2).

Cross-lingual document retrieval as an application of the problem of optimal

transport. An important issue of text mining applications with word embeddings is to compose the representations of large text spans from the word representations provided that there exists an alternative way to calculate document distances. We argue that such a step can be omitted for several text mining applications. We propose to use a document distance metric that relies on the solution of the optimal transport problem and we demonstrate the effectiveness of this formulation on an interesting cross-lingual retrieval application (Section 6.3).

The rest of the chapter is organized as follows: Sections 6.1-6.3 contribute the models, the observations and the results for the settings described above. Then, Section 6.4 summarizes the findings of the chapter.

Polylingual text classification

Neural Networks have recently shown promising results in several machine learning and information extraction tasks [START_REF] Tang | Learning sentiment-specific word embedding for twitter sentiment classification[END_REF][START_REF] Zhang | Text understanding from scratch[END_REF][START_REF] Gao | Learning continuous phrase representations for translation modeling[END_REF]. For text classification, the use of embeddings as inputs or initializations to more complex architectures has been investigated and, for example, [START_REF] Kalchbrenner | A convolutional neural network for modelling sentences[END_REF][START_REF] Kim | Convolutional neural networks for sentence classification[END_REF] study the benefits of embeddings of sentence-length spans (sentences and/or questions). In the multilingual setting, [START_REF] Gouws | Bilbowa: Fast bilingual distributed representations without word alignments[END_REF] proposed an approach to learn bilingual embeddings exploiting parallel and non-parallel text in the languages, [START_REF] Faruqui | Improving vector space word representations using multilingual correlation[END_REF] proposed to use correlated components analysis, together with small bilingual lexicons, to learn how to project embeddings in two separate languages into a common representation space and [START_REF] Lauly | An autoencoder approach to learning bilingual word representations[END_REF] proposed an approach similar to ours that uses an auto-encoder to learn bilingual representations.

In this section we propose a mechanism for combining distributed representations of documents in different languages. In this line, each document in a given language is first translated using an existing Machine Translation (MT) tool.

The rationale behind is that translation offers the possibility to enrich and disambiguate the text, especially for short documents. Documents are then represented by aggregating the embeddings of their associated text spans in each language [START_REF] Quoc | Distributed representations of sentences and documents[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] using a non-linear auto-encoder (AE). The AE is trained on their concatenated representations and a classifier is finally trained in the polylingual space generated by the auto-encoder. The hope is that the AE can learn language independent representations of large text spans like documents by compressing its inputs in a hidden layer and thus combining information from every language in the input. Our classification results in a subset of the publicly available Wikipedia documents show that our approach yields improved classification performance compared to the case where a classical bag-of-words space is used for document representation, especially in the case where the size of the training set is small. The following subsections present our strategy for learning polylingual embeddings. Then, in the experimental part we empirically show that the learned representations constitute better classification features compared to several baselines.

Importantly, our findings suggest that polylingual representations can strongly benefit classification settings with few labeled examples.

The model for learning the polylingual embeddings

Monolingual distributed representations (DRs) project text spans into a languagedependent semantic space where spans with similar semantics are closer in that space. Here, we aim to combine two distributed representations of documents corresponding to the original document and its translation using an auto-encoder.

We will refer to those combined representations as Polylingual Embeddings (PE).

We call them polylingual as two or more languages are used to derive them and they model at the same time the text written in either of them. We suppose that the auto-encoder will disentangle the language-dependent factors and will learn robust representations on its hidden layer encoding as illustrated in Figure 6.1.

Given a document d i in English, we first translate it into French using a commercial translator, 1 . Then, we generate the distributed representations of the document and its translation using a function G that we will describe shortly as {G (d i )} 2 =1 and aggregate those DRs using an auto-encoder as follows:

• {G (d i )} 2 =1 , a trained AE • For each document d i : -Concatenate G 1 (d i ) and G 2 (d i )
-Get PE representation of d i as the hidden encoding of the AE fed with the concatenation Auto-encoders (Figure 6.1) are neural network architectures whose aim is to learn an encoding of the data by typically projecting them in a lower dimension using a single or a cascade of hidden layers ??. To this end, we try to minimize the distance between the input and the output representations, which is commonly calculated using the Euclidean distance. Instead of learning a linear projection of the data, the activation functions of the hidden layer are non-linear (such us the sigmoid function or the hyperbolic tangent function) thus adding to the modeling capacity of the architecture. The auto-encoder is learned over all concatenated distributed representations of documents using a stochastic back-propagation algorithm. In this work we consider two strategies to create the DR of each document. The first one is based on average pooling, where word representations are first obtained using the word2vec tool [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF]. This is also the approached using in Chapter 5 for obtaining the representations of documents using the cross-lingual embeddings. DR of documents, i.e. functions (G ) ∈{1,2} , are then obtained by averaging the vectors of words contained in them. In this study we consider the continuous bag of words (cbow) and the skip-gram models that generate word representations. The second strategy is based on the distributed Memory Model of paragraph vectors (DMMpv) and distributed bag-of-words of paragraph vectors (DBOWpv) models [START_REF] Quoc | Distributed representations of sentences and documents[END_REF], that extend cbow and skip-gram respectively. DMMpv and DBOWpv, instead of learning representations for words learn representations for larger spans that is whole documents for our case here. Therefore, (G ) ∈{1,2} are defined by the output of the models without further processing.

The Experimental Evaluation

The data Training neural network models to generate distributed representations benefits by large amounts of free text. To train the models that generate DRs we used such free texts in English and French:2 the left part of Table 6.1 (under "Distributed Representations") presents some basic statistics for those data. We used the same number of documents for the two languages to avoid any training bias. The raw text was pre-processed by applying lower-casing and space-padding punctuation. Similarly to previous studies [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Quoc | Distributed representations of sentences and documents[END_REF], we kept the punctuation. Publicly available implementations of the models were used with their default parameters: the word2vec tool 3 for the cbow and skip-gram and the doc2vec for the DBOWpv and DMMpv from Gensim [START_REF] Řehůřek | Software Framework for Topic Modelling with Large Corpora[END_REF].

For the classification task we used the raw version of the Wikipedia dataset of the Large Scale Hierarchical Text Classification challenge [START_REF] Partalas | LSHTC: A benchmark for large-scale text classification[END_REF]. The original dataset contains 60,252 categories; we restrict our study here in a fraction of the dataset with 12,670 documents belonging to the 100 most common categories. The right part of Table 6.1 presents basic statistics for this subset.

Baselines We use as a first baseline Support Vectors Machines (SVM) fed with the tf-idf representation of the documents, which is commonly used in text classification problems (denoted by SVM BoW ). As a second baseline, we use k-Nearest Neighbors (k-NN) and SVMs learned on the monolingual space of the DRs of English documents (denoted respectively by SVM DR and k-NN DR ). These baselines and k-NN PE ). For the former, given the labels of the k nearest training instances of a test document, the algorithm returns the labels that belong to at least p% of its nearest neighbors. For each run k ∈ {13, 14, 15} and p ∈ {0.1, 0.2, 0.3} are decided using 5-fold cross-validation on the training data. The SVMs were used in an one-vs-rest fashion; they return every label that has a positive distance from the separating hyperplane. The value of the hyper-parameter C ∈ {10 -1 , . . . , 10 4 } that controls the importance of the regularization term in the optimization problem, is selected using 5-fold cross-validation over the training data.

Our approach Using the above-presented DR model, we first generate the document embeddings in English and French. These are vectors in a d-dimensional space with d ∈ {50, 100, 200, 300}. Then, for the AE we considered as activation functions the hyperbolic tangent and the sigmoid function. The sigmoid performed consistently better and thus we use it in the reported results. The AE was trained with tied weights using a stochastic back-propagation algorithm with mini-batches of size 10 and the euclidean distance of the input/output as loss function. The number of neurons in the hidden layer was set to be 70% of the size of the input. 4 Experimental Results Table 6.2 presents the scores of the F 1 measure when 10% of the 12.670 documents were used for training purposes and the rest 90% for testing. We report the classification performance with the four different DR models (cbow, skip-gram, DBOWpv and DMMpv) and 2 learning algorithms (k-NN and SVMs) for different input sizes. The columns labeled k-NN DR and SVM DR present the (baseline) performance of SVM and k-NN trained on the monolingual DRs. 4 The code is available at http://ama.liglab.fr/~balikas/ecir2015.zip. We first notice that the average pooling strategy (cbow and skip-gram) performs better compared to when the document vectors are directly learned (DBOWpv and DMMpv). In particular, cbow seems to be the best performing representation, Comparing the two learning methods (k-NN PE and SVM PE ), we notice that k-NN PE performs best. This is motivated by the fact that distributed representations more training data are available the tf-idf copes with the complexity of the problem and leverages this wealth of information more efficiently than PE does.

Summary

We proposed a novel approach for learning embeddings of large text spans. The novel embeddings are learned using with neural networks and in particular with a denoising autoencoder. The AE embeds translations of an input document in a language independent space the hope being that by combining information from two or more languages can benefit the performance. We empirically showed the effectiveness of the novel embeddings in the bilingual setting, in the task of document classification. Our embeddings achieved better performance compared to traditional classification approaches in the interesting case where few labeled training data are available for learning.

The main limitation of this approach is the fact it relies on a translation system for obtaining the translations of the input document which are consecutively used for obtaining the PE. An interesting extension of this work would be to quantify the effect of the quality of the translations to the performance. One could, for example, start by using dictionaries to translate between languages and then proceed with more advanced and state-of-the-art methods like the translation system we used in this work. Another interesting extension concerns the compositional mechanism used to derive the representations of spans that are composed of several words. In our experiments we used the average pooling function that is efficient and robust but may result in loss of information when a text span has several words. To overcome this, one could use a special type of neural network architectures called recurrent neural networks that can be model sequences of elements like text. It is indeed this observation that motivates part of the contributions presented in the following section, where we will use recurrent neural networks for generating text span representations.

Multitask Learning with Neural Networks

In the previous section we showed that learning representations using translations of a text span in different languages improves the performance achieved.

On the other hand, this approach required a readily available translation system, which imposes a significant development and computational overhead. Our analysis illustrated, however, that combining information in the form of translations improved the results.

Semantically, translations of a document are correlated because they discuss the same ideas in different languages. Meanwhile, one may imagine scenarios comprising different tasks, strongly or weakly correlated between them instead of correlated representations. In such a case, instead of dealing with different representations of an instance, we deal with different tasks, that may be able to benefit from each other. This can be natural language processing tasks like named entity recognition and part-of-speech tagging or multimedia classification tasks such as image classification and segmentation. For each of the previous examples, while the tasks may seem at first different, information from one task may help the performance on the other task. Motivated by our previous findings, we explore such a setting by taking advantage of the modeling flexibility that neural networks offer. Typical scenarios of machine learning involve optimizing the performance on a task using an evaluation metric. To this end, a learning model or an ensemble of such models, are trained to perform the task while their free parameters are tuned to maximize the achieved performance. However, instead of only relying on the training signal of the given task, one may be able to do better by incorporating signals from related tasks. Multitask learning refers to the scenario where a learner is trained jointly on several interdependent tasks [START_REF] Caruana | Multitask learning[END_REF]. The hope is that the multiplicity of the tasks will result in more robust representations or a decision function that will, in turn, improve the performance on the given task. As a result, incorporating dependent tasks to the learning process of the main tasks helps selecting a better hypothesis.

We can motivate multitask learning approaches in different ways. First, as being inspired by human learning, where for learning new tasks we apply parts of the knowledge previously acquired from different yet related tasks. For instance, children or adults begin by understanding parts of language such as simple words and use this while improving and understanding larger spans like phrases and sentences. Second, from a pedagogical or a didactic point of view: we often learn a task using knowledge from previous, simpler tasks. For instance, in sports one first learns simple moves that are the basics and then elaborates on them. Third, multitask learning can be motivated from a machine learning perspective: multitask learning can be seen as a form of inductive bias or regularization. Inductive bias is anything that causes an inductive learner to prefer some hypotheses over others. An example of inductive bias is l 1 , which causes a learner to select sparse solutions. In multitask learning, where a learner trains for the main task uses the signals of the other asks it is easy to see why these signals can serve as inductive bias. It causes the learner to select solutions that perform well across tasks and therefore generalize better.

For neural networks, one may imagine two straightforward architectures for implementing multitask learning. The first, could be described as hard parameter sharing. The first layers of a network are shared across the two or more related tasks, and the last layers specialize on each of those tasks. Such an approach reduces the risk of overfitting for the parameters of the shared layers [START_REF] Baxter | A bayesian/information theoretic model of learning to learn via multiple task sampling[END_REF]. Apart from hard parameter sharing architectures, one can also imagine soft parameter sharing schemes. In the latter, given N tasks, there are N networks. Each tasks has a dedicated network and parameters. The parameters of these two networks are regularized however in order to be close. For example, [START_REF] Duong | Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser[END_REF] regularize such architectures with the l 2 norm, while [START_REF] Yang | Trace norm regularised deep multi-task learning[END_REF] use the trace norm. The work presented in this section belongs in the first category. We propose a model that implements a hard parameter sharing architecture for modeling and categorizing short text spans with respect to the intensity of the sentiment the convey.

In the rest of the sections we elaborate on the task, the model and the evaluation framework used.

Multitask Learning for Sentiment Classification

Automatic classification of sentiment has mainly focused on categorizing tweets in either two (binary sentiment analysis) or three (ternary sentiment analysis) categories [START_REF] Giachanou | Like it or not: A survey of twitter sentiment analysis methods[END_REF]. In this work we focus on the problem of fine-grained sentiment classification where tweets are classified according to a five-point scale ranging from VeryNegative to VeryPositive. To illustrate this, Table 6.3 presents examples of tweets associated with each of these categories. Five-point scales are widely adopted in review sites like Amazon and TripAdvisor, where a user's sentiment is ordered with respect to its intensity. From a sentiment analysis perspective, this defines a classification problem with five categories. In particular, Sebastiani et al. [START_REF] Da | Ordinal text quantification[END_REF] defined such classification problems whose categories are explicitly ordered to be ordinal classification problems. To account for the ordering of the categories, learners are penalized according to how far from the true class their predictions are.

Although considering different scales, the various settings of sentiment classification are related. First, one may use the same feature extraction and engineering approaches to represent the text spans such as word membership in lexicons, morpho-syntactic statistics like punctuation or elongated word counts [START_REF] Balikas | Twise at semeval-2016 task 4: Twitter sentiment classification[END_REF][START_REF] Kiritchenko | Sentiment analysis of short informal texts[END_REF].

Second, one would expect that knowledge from one task can be transfered to the others and this would benefit the performance. Knowing that a tweet is "Positive" in the ternary setting narrows the classification decision between the VeryPositive and Positive categories in the fine-grained setting. From a research perspective this raises the question of whether and how one may benefit when tackling such related tasks and how one can transfer knowledge from one task to another during the training phase. Our focus in this work is to exploit the relation between the sentiment classification settings and demonstrate the benefits stemming from combining them. To this end, we propose to formulate the different classification problems as a multitask learning problem and jointly learn them. Multitask learning [START_REF] Caruana | Multitask learning[END_REF] has shown great potential in various domains and its benefits have been empirically validated [START_REF] Collobert | A unified architecture for natural language processing: deep neural networks with multitask learning[END_REF][START_REF] Plank | Keystroke dynamics as signal for shallow syntactic parsing[END_REF][START_REF] Liu | Recurrent neural network for text classification with multi-task learning[END_REF][START_REF] Liu | Deep multi-task learning with shared memory for text classification[END_REF] using different types of data and learning approaches. An important benefit of multitask learning is that it provides an elegant way to access resources developed for similar tasks. By jointly learning correlated tasks, the amount of usable data increases. For instance, while for ternary classification one can label data using distant supervision with emoticons [START_REF] Go | Twitter sentiment classification using distant supervision[END_REF], there is no straightforward way to do so for the fine-grained problem. However, the latter can benefit indirectly, if the ternary and fine-grained tasks are learned jointly.

The research question that this section attempts to answer is the following:

Can twitter sentiment classification problems, and fine-grained sentiment classification in particular, benefit from multitask learning? To answer the question, the work done brings the following two main contributions: (i) we show how jointly learning the ternary and fine-grained sentiment classification problems in a multitask setting improves the state-of-the-art performance, 5 and (ii) we demonstrate that recurrent neural networks outperform models previously proposed without access to huge corpora while being flexible to incorporate different sources of data.

The Experimental Framework

In his work, Caruana [START_REF] Caruana | Multitask learning[END_REF] proposed a multitask approach in which a learner takes advantage of the multiplicity of interdependent tasks while jointly learning them.

The intuition is that if the tasks are correlated, the learner can learn a model jointly for them while taking into account the shared information which is expected to improve its generalization ability. People express their opinions online on various

Multitask outputs biLSTM LSTM LSTM LSTM LSTM LSTM LSTM X T X 1 X T -1 X 2 X 1 X T H 1 H A H M Additional Features softmax 1 softmax N . . .

VeryNegative

Beyond frustrated with my #Xbox360 right now, and that as of June, @Microsoft doesn't support it. Gotta find someone else to fix the drive.

Negative @Microsoft Heard you are a software company. Why then is most of your software so bad that it has to be replaced by 3rd party apps?

Neutral @ProfessorF @gilwuvsyou @Microsoft @LivioDeLaCruz We already knew the media march in ideological lockstep but it is nice of him to show it.

Positive PAX Prime Thursday is overloaded for me with @Microsoft and Nintendo indie events going down. Also, cider!!! :p VeryPositive I traveled to Redmond today. I'm visiting with @Microsoft @SQLServer engineers tomorrow -at their invitation. Feeling excited.

Table 6.3: The example demonstrates the different levels of sentiment a tweet may convey. Also, note the Twitter-specific use of language and symbols.

subjects (events, products..), on several languages and in several styles (tweets, paragraph-sized reviews..), and it is exactly this variety that motivates the multitask approaches. Specifically for Twitter for instance, the different settings of classification like binary, ternary and fine-grained are correlated since their difference lies in the sentiment granularity of the classes which increases while moving from binary to fine-grained problems.

There are two main decisions to be made in our approach: the learning algorithm, which learns a decision function, and the data representation. With respect to the former, neural networks are particularly suitable as one can design architectures with different properties and arbitrary complexity. Also, as training a neural network usually relies on back-propagation of errors [START_REF] David E Rumelhart | Learning representations by back-propagating errors[END_REF], one can have shared parts of the network trained by estimating errors on the joint tasks and others specialized for particular tasks. Concerning the data representation, it strongly depends on the data type available. For the task of sentiment classification of tweets with neural networks, distributed embeddings of words have shown great potential. Embeddings are defined as low-dimensional, dense representations of words that can be obtained in an unsupervised fashion by training on large quantities of text [START_REF] Pennington | Glove: Global vectors for word representation[END_REF].

Concerning the neural network architecture, we focus on Recurrent Neural Networks (RNNs) that are capable of modeling short-range and long-range dependencies like those exhibited in sequence data of arbitrary length like text. While in the traditional information retrieval paradigm such dependencies are captured using n-grams and skip-grams, RNNs learn to capture them automatically [START_REF] Dyer | Transition-based dependency parsing with stack long short-term memory[END_REF]. To circumvent the problems with capturing long-range dependencies and preventing gradients from vanishing, the long short-term memory network (LSTM) was proposed [START_REF] Hochreiter | Long short-term memory[END_REF]. In this work, we use an extended version of LSTM called bidirectional LSTM (biLSTM). While standard LSTMs access information only from the past (previous words), biLSTMs capture both past and future information effectively [START_REF] Huang | Bidirectional LSTM-CRF models for sequence tagging[END_REF][START_REF] Dyer | Transition-based dependency parsing with stack long short-term memory[END_REF]. They consist of two LSTM networks, for propagating text forward and backward with the goal being to capture the dependencies better. Indeed, previous work on multitask learning showed the effectiveness of biLSTMs in a variety of problems: [2] tackled sequence prediction, while [START_REF] Plank | Keystroke dynamics as signal for shallow syntactic parsing[END_REF] and [START_REF] Kiperwasser | Simple and accurate dependency parsing using bidirectional LSTM feature representations[END_REF] used biLSTMs for Named Entity Recognition and dependency parsing respectively. Figure 6.3 presents the architecture we use for multitask learning. In the topleft of the figure a biLSTM network (enclosed by the dashed line) is fed with embeddings {X 1 , . . . , X T } that correspond to the T words of a tokenized tweet. Notice, as discussed above, the biLSTM consists of two LSTMs that are fed with the word sequence forward and backwards. On top of the biLSTM network one (or more) hidden layers H 1 transform its output. The output of H 1 is led to the softmax layers for the prediction step. There are N softmax layers and each is used for one of the N tasks of the multitask setting. In tasks such as sentiment classification, additional features like membership of words in sentiment lexicons or counts of elongated/capitalized words can be used to enrich the representation of tweets before the classification step [START_REF] Kiritchenko | Sentiment analysis of short informal texts[END_REF]. The lower part of the network illustrates how such sources of information can be incorporated to the process. A vector "Additional Features" for each tweet is transformed from the hidden layer(s) H A and then is combined by concatenation with the transformed biLSTM output in the H M layer.

Our goal is to demonstrate how multitask learning can be successfully applied on the task of sentiment classification of tweets. The particularities of tweets are to be short and informal text spans. The common use of abbreviations, creative language etc., makes the sentiment classification problem challenging. To validate our hypothesis, that learning the tasks jointly can benefit the performance, we propose an experimental setting where there are data from two different twitter sentiment classification problems: a fine-grained and a ternary. We consider the fine-grained task to be our primary task as it is more challenging and obtaining bigger datasets, e.g. by distant supervision, is not straightforward and, hence we report the performance achieved for this task. As a result, unless otherwise stated, we optimize for the performance on the fine-grained classification tasks. For completeness, however, we also report the performance we obtain for the ternary classification task.

The data Ternary and fine-grained sentiment classification were part of the Se-mEval 2016 6 "Sentiment Analysis in Twitter" task [START_REF] Nakov | Semeval-2016 task 4: Sentiment analysis in twitter[END_REF]. We use the high-quality datasets the challenge organizers released. 7 The dataset for fine-grained classification is split in training, development, development_test and test parts. In the rest, we refer to these splits as train, development and test, where train is composed by the training and the development instances. Table 6.4 presents an overview of the data. As discussed in [START_REF] Nakov | Semeval-2016 task 4: Sentiment analysis in twitter[END_REF] and illustrated in the Feature representation We report results using various feature sets. The first one, dubbed bow is the commonly used bow of words representation of the tweets. The representation is motivated by our previous work [START_REF] Balikas | Twise at semeval-2016 task 4: Twitter sentiment classification[END_REF], where we showed that using n-grams with n ∈ {1, 2, 3} as well as character-grams of size 4 and 5 benefits sentiment classification. Here, we use all the possible n-grams and charactergrams hashed in vectors of dimension 20K and 25K respectively. The second, dubbed nbow, is a neural bag-of-words that uses text embeddings to generate lowdimensional, dense representations of the tweets. To construct the nbow representation, given the word embeddings dictionary where each word is associated with a vector, we apply the average compositional function that averages the embeddings of the words that compose a tweet. Simple compositional functions like average were shown to be robust and efficient in previous work [START_REF] Mitchell | Composition in distributional models of semantics[END_REF]. Instead of training embeddings from scratch, we use the pre-trained on tweets GloVe embeddings of [START_REF] Pennington | Glove: Global vectors for word representation[END_REF]. 8 In terms of resources required, using only nbow is efficient as it does not require any domain knowledge. However, previous research on sentiment analysis showed that using extra resources, like sentiment lexicons, can benefit significantly the performance [START_REF] Kiritchenko | Sentiment analysis of short informal texts[END_REF][START_REF] Balikas | Twise at semeval-2016 task 4: Twitter sentiment classification[END_REF]. To validate this and examine at which extent neural networks and multitask learning benefit from such features we evaluate the models using an augmented version of nbow, dubbed nbow+. The feature space of the latter, is augmented using 1,368 extra features consisting mostly of counts of punctuation symbols ('!?#@'), emoticons, elongated words and word membership features in several sentiment lexicons. The next paragraph details those extra features. Also, bow+ refers to the concatenation of these extra features with the bow representations introduced above.

Feature Engineering Similar to [START_REF] Kiritchenko | Sentiment analysis of short informal texts[END_REF] we extracted features based on the lexical content of each tweet and we also used sentiment-specific lexicons. The features extracted for each tweet include:

• number of exclamation marks, number of question marks, number of both exclamation and question marks,

• number of words written in capitals and number of elongated words, that is words with more than three occurrences of a letter such as "cooool" and "bliiiiah",

• number of negative words in a tweet,

• number of positive emoticons, number of negative emoticons and a binary feature indicating if emoticons exist in a given tweet, and

• distribution of Part-of-speech (POS) tags [START_REF] Gimpel | Part-of-speech tagging for twitter: Annotation, features, and experiments[END_REF] and distribution of POS tags in a positive and negative contexts. We consider words to occur in a negative context if a negative word proceeds then. A negative context stops when another negative words, punctuation or the end of the tweet is met.

With regard to the sentiment lexicons, we used:

• manual sentiment lexicons: the Bing liu's lexicon [START_REF] Hu | Mining and summarizing customer reviews[END_REF], the NRC emotion lexicon [START_REF] Saif | Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon[END_REF], and the MPQA lexicon [START_REF] Wilson | Recognizing contextual polarity in phrase-level sentiment analysis[END_REF],

• # of words in positive and negative context belonging to the word clusters provided by the CMU Twitter NLP tool 9• positional sentiment lexicons: sentiment 140 lexicon 10 [68] and the Hashtag Sentiment Lexicon [START_REF] Kiritchenko | Sentiment analysis of short informal texts[END_REF] We make, here, more explicit the way we used the sentiment lexicons, using the Bing Liu's lexicon as an example. We treated the rest of the lexicons similarly. For each tweet, using the Bing Liu's lexicon we obtain a 104-dimensional vector. After tokenizing the tweet, we count how many words (i) in positive/negative contexts belong to the positive/negative lexicons (4 features) and we repeat the process for the hashtags (4 features). To this point we have 8 features. We generate those 8 features for the lowercase words and the uppercase words. Finally, for each of the 24 POS tags the [START_REF] Gimpel | Part-of-speech tagging for twitter: Annotation, features, and experiments[END_REF] tagger generates, we count how many words in positive/negative contexts belong to the positive/negative lexicon. As a results, this generates 2 × 8 + 24 × 4 = 104 features in total for each tweet.

Evaluation measure To reproduce the setting of the SemEval challenges [START_REF] Nakov | Semeval-2016 task 4: Sentiment analysis in twitter[END_REF], we optimize our systems using as primary measure the macro-averaged Mean Absolute Error (MAE M ) given by:

MAE M = 1 |C| |C| j=1 1 |Te j | x i ∈Te j |h(x i ) -y i |
where |C| is the number of categories, Te j is the set of instances whose true class is c j , y i is the true label of the instance x i and h(x i ) the predicted label. The measure penalizes decisions far from the true ones and is macro-averaged to account for the fact that the data are unbalanced. Complementary to MAE M , we report the performance achieved on the micro-averaged F 1 measure, which is a commonly used measure for classification.

The models To evaluate the multitask learning approach, we compared it with several other models. Support Vector Machines (SVMs) are maximum margin classification algorithms that have been shown to achieve competitive performance in several text classification problems [START_REF] Nakov | Semeval-2016 task 4: Sentiment analysis in twitter[END_REF]. SVM ovr stands for an SVM with linear kernel and an one-vs-rest approach for the multi-class problem. Also, SVM cs is an SVM with linear kernel that employs the crammer-singer strategy [START_REF] Crammer | On the algorithmic implementation of multiclass kernel-based vector machines[END_REF] for the multi-class problem. Logistic regression (LR) is another type of linear classification method, with probabilistic motivation. Again, we use two types of Logistic Regression depending on the multi-class strategy: LR ovr that uses an one-vs-rest approach and multinomial Logistic Regression also known as the MaxEnt classifier that uses a multinomial criterion.

Both SVMs and LRs as discussed above treat the problem as a multi-class one, without considering the ordering of the classes. For these four models, we tuned the hyper-parameter C that controls the importance of the L 2 regularization part in the optimization problem with grid-search over {10 -4 , . . . , 10 4 } using 10-fold crossvalidation in the union of the training and development data and then retrained the models with the selected values. Also, to account for the unbalanced classification problem we used class weights to penalize more the errors made on the rare classes. These weights were inversely proportional to the frequency of each class.

For the four models we used the implementations of Scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in python[END_REF].

For multitask learning we use the architecture shown in Figure 6.3, which we implemented with Keras [START_REF] Chollet | [END_REF]. The embeddings are initialized with the 50dimensional GloVe embeddings while the output of the biLSTM network is set to dimension 50. The activation function of the hidden layers is the hyperbolic tangent. The weights of the layers were initialized from a uniform distribution, scaled as described in [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]. We used the Root Mean Square Propagation optimization method. We used dropout for regularizing the network. We trained the network using batches of 128 examples as follows: before selecting the batch, we perform a Bernoulli trial with probability p M to select the task to train for. With probability p M we pick a batch for the fine-grained sentiment classification problem, while with probability 1p M we pick a batch for the ternary problem. As shown in Figure 6.3, the error is backpropagated until the embeddings, that we fine-tune during the learning process. Notice also that the weights of the network until the layer H M are shared and therefore affected by both tasks.

To tune the neural network hyper-parameters we used 5-fold cross validation.

We tuned the probability p of dropout after the hidden layers H M , H 1 , H A (cf. Fig. 6.3) and for the biLSTM for p ∈ {0.2, 0.3, 0.4, 0.5}, the size of the hidden layer H M ∈ {20, 30, 40, 50} and the probability p M of the Bernoulli trials from {0.5, 0.6, 0.7, 0.8}. 11During training, we monitor the network's performance on the development set and apply early stopping if the performance on the validation set does not improve for 5 consecutive epochs.

Experimental results

Fine-grained problem Table 6.8 illustrates the performance of the models for the different data representations. The upper part of the Table summarizes the performance of the baselines. The entry "Balikas et al." stands for the winning system of the 2016 edition of the challenge [START_REF] Balikas | Twise at semeval-2016 task 4: Twitter sentiment classification[END_REF], which to the best of our knowledge holds the state-of-the-art. Due to the stochasticity of training the biLSTM models, we repeat the experiment 10 times and report the average and the standard deviation of the performance achieved.

Several observations can be made from the table. First notice that, overall, the best performance is achieved by the neural network architecture that uses multitask learning. This entails that the system makes use of the available resources efficiently and improves the state-of-the-art performance. In conjunction with the fact that we found the optimal probability p M = 0.5, this highlights the benefits of multitask learning over single task learning. Furthermore, as described above, the neural network-based models have only access to the training data as the development are hold for early stopping. On the other hand, the baseline systems were retrained on the union of the train and development sets. Hence, even with fewer resources available for training on the fine-grained problem, the neural networks outperform the baselines. We also highlight the positive effect of the additional features that previous research proposed. Adding the features both in the baselines and in the biLSTM-based architectures improves the MAE M scores by several points.

Lastly, we compare the performance of the baseline systems with the performance of the state-of-the-art system of [START_REF] Balikas | Twise at semeval-2016 task 4: Twitter sentiment classification[END_REF]. While [START_REF] Balikas | Twise at semeval-2016 task 4: Twitter sentiment classification[END_REF] uses n-grams (and charactergrams) with n > 1, the baseline systems (SVMs, LRs) used in this work use the nbow+ representation, that relies on unigrams. Although they perform on par, the competitive performance of nbow highlights the potential of distributed representations for short-text classification. Further, incorporating structure and distributed representations leads to the gains of the biLSTM network, in the multitask and single task setting.

Similar observations can be drawn from Figure 6.4 that presents the F 1 scores.

Again, the biLSTM network with multitask learning achieves the best performance.

It is also to be noted that although the two evaluation measures are correlated in the sense that the ranking of the models is the same, small differences in the MAE M have large effect on the scores of the F 1 measure.

Ternary problem Complementary to the performance on the fine-grained sentiment classification problem discussed above, we report the performance on the ternary task. Again, in order to replicate the setting of the SemEval 2016 challenges, we use the evaluation measure proposed by the challenge organizers that is the F 1 measure, calculated only for the positive and the negative categories. Table 6.6 summarizes the results for the baselines and the multitask architecture. An important detail before commenting on the performance achieved is that the values of the hyperparameters are those we found optimal for the fine-grained task, as that was our main task for the study. There are several observations from the table.

First, adding the additional features (bow and nbow compared to bow+ and nbow+) benefits every system we tested. In general, the representations that are based on the word embeddings perform better than those based on the bag-ofwords representations. Second, the neural network architectures perform better than the traditional classification systems like SVMs and Logistic Regression. Notice that the entry "Deriu et al." [START_REF] Deriu | Swisscheese at semeval-2016 task 4: Sentiment classification using an ensemble of convolutional neural networks with distant supervision[END_REF] is the winner of the task who also used neural networks, and in particular, an ensemble of convolutional neural networks.

Lastly, and perhaps most importantly, the improvements due to multitask learning are very small and therefore we can not claim any important gain. We claim however that the models perform on par and we highlight that the performance on the ternary task did not decrease while the performance in the fine-grained task increased.

Summary

In this section, we showed that by jointly learning the tasks of ternary and finegrained classification with a multitask learning model, one can greatly improve the performance on the second. This opens several avenues for future research. The first and perhaps the most straightforward would be to verify whether our findings generalize or further improve using data of different type and language. Since sentiment is expressed in different textual types like tweets and paragraph-sized reviews, in different languages (English, German, ..) and in different granularity levels (binary, ternary,..) one can imagine multitask approaches that could benefit from combining such resources.

We showed that by using a multitask learning architecture we managed to gain in terms of performance when tuning for the fine-grained problem. For the ternary problem however we did not observe similar benefits when enabling multitask learning: there are some marginal performance improvements that do not allow for claiming significant improvements. A direct extension would be to compare this outcome with the case where the fine-tuning is performed for the ternary problem. Although this is an interesting task from an experimental point of view, one can directly improve on the ternary task using distant supervision in the form of pseudo-labeling tweets based on emoticons, following for instance the work of [START_REF] Deriu | Swisscheese at semeval-2016 task 4: Sentiment classification using an ensemble of convolutional neural networks with distant supervision[END_REF].

Another research line would be to examine a hierarchical decision function in the output of the network, perhaps motivated by common hierarchical text classification systems [START_REF] Sun | Hierarchical text classification and evaluation[END_REF]. That way, one could apply a two-level prediction mechanism for the fine-grained task: first predict the ternary task (positive, negative, neutral) and then, in the second level, the intensity of the sentiment: between VeryPositive and Positive for the positive sentiment or between VeryNegative and Negative for the negative sentiment. Increasing the data size for the decisions of the first level by combining the ternary and fine-grained datasets should result in performance gains provided that the distribution over the classes does not change.

Exploring and comparing such as approach with multitask learning is part of our future work.

Lastly, while we opted for biLSTM networks here, one could use convolutional neural networks or even try to combine different types of networks and tasks to investigate the performance effect of multitask learning. Convolutions neural networks have been shown to be effective for text classification and comparing their performance with LSTMs as well as the effect of multitask learning is an interesting extension.

Cross-lingual text retrieval

As we have previously discussed and demonstrated in this dissertation, word embeddings have shown great potential in several natural language processing tasks [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Pennington | Glove: Global vectors for word representation[END_REF][START_REF] Levy | Neural word embedding as implicit matrix factorization[END_REF][START_REF] Li | Word embedding revisited: A new representation learning and explicit matrix factorization perspective[END_REF]. Their ability to capture syntactic properties as well as their property to project semantically similar words close in the induced vector space have been particularly celebrated as they alleviate limitations of the discrete representations of words based on the vector space model [START_REF] Salton | A vector space model for automatic indexing[END_REF]. The success of monolingual word embeddings, helped to develop new approaches for multilingual word embeddings. Additionally to projecting similar words (or short multiword expressions) of a single language close in the inferred vector space, multilingual embeddings project similar words across languages close in the shared vector space. Depending on the available resources and the approaches employed to learn the embeddings, there are different methods [START_REF] Klementiev | Inducing crosslingual distributed representations of words[END_REF][START_REF] Wu | Improve statistical machine translation with context-sensitive bilingual semantic embedding model[END_REF][START_REF] Gouws | Bilbowa: Fast bilingual distributed representations without word alignments[END_REF][START_REF] Moritz | Multilingual distributed representations without word alignment[END_REF][START_REF] Shi | Learning crosslingual word embeddings via matrix co-factorization[END_REF][START_REF] Mikolov | Exploiting similarities among languages for machine translation[END_REF][START_REF] Xiao | Distributed word representation learning for cross-lingual dependency parsing[END_REF][START_REF] Gardner | Translation invariant word embeddings[END_REF][START_REF] Søgaard | Inverted indexing for cross-lingual nlp[END_REF][START_REF] Vulić | Bilingual distributed word representations from document-aligned comparable data[END_REF][START_REF] Speer | Conceptnet 5.5: An open multilingual graph of general knowledge[END_REF].

In the previous sections, we have also discussed that methods of composition for obtaining representations of large text spans on top of word embeddings are particularly important. To this end, in Section 6.1 we demonstrated how averaging word embeddings performs while in Section 6.2 we relied on recurrent neural networks and LSTMs in particular. In this section we are revisiting this problem by proposing a completely different approach: we argue that in several tasks having explicit representations of large spans is not necessary as long as an efficient way for obtaining distances between spans exists.

In this section, we adapt to the cross-lingual setting the work of [START_REF] Kusner | From word embeddings to document distances[END_REF], who proposed a clever re-parametrization of the transportation problem without hyperparameters for calculating document distances from cross-lingual word embeddings.

Meanwhile, we will show that a simple modification of the optimization problem 123 allows the framework to incorporate term-weighting schemes. To validate the proposed approach, we conducted extensive experiments for cross-lingual document discovery (CLDD) in six cross-lingual settings where given a query document (e.g., English Wikipedia entry for "Dog") one needs to retrieve its corresponding document in another language (e.g., French entry for "Chien"). The novel method outperforms strong baselines previously proposed for the task by a large margin. Notably, we demonstrate the impact of the quality of the embeddings used, as well as the impact of three term weighting schemes in terms of Mean Reciprocal Rank and Precision at 1.

A Wasserstein-alike distance for Cross-lingual Document Retrieval

In this section, we demonstrate how calculating document distances can be seen as an instance of the Earth Mover's Distance problem [START_REF] Kusner | From word embeddings to document distances[END_REF].

Notation We assume access to the collections

C 1 = {d 1 1 , . . . , d 1 N } and C 2 = {d 2 1 , . . . , d 2 M } where d 1 i (resp. d 2 i
) is the i-th document written in language 1 (resp. 2 ). Let the vocabulary size of the two languages be denoted as V 1 and V 2 . For the rest of the development we assume to have access to dictionaries of embeddings E 1 , E 2 where words from 1 and 2 are projected into a shared vector space of dimension

D, hence E 1 ∈ R V 1 ×D , E 2 ∈ R V 2 ×D and E 1
k , E 2 j denote the embeddings of words k, j. As learning the bilingual embeddings is not the focus of this paper, any of the previously proposed methods can be used. A document consists of words and is represented using the Vector Space Model with frequencies. Hence, ∀i :

d 1 i ∈ R V 1 , d 2 i ∈ R V 2 and d 1 ij
is the frequency of the j-th word of document d 1 i . Importantly, the vector representations of the documents need to be l 1 -normalized. Calculating the distance of words in the embeddings space is naturally achieved using the Euclidean distance with lower values meaning that words are similar between them.

For the rest, we denote by c(k, j) = ||E 1 k -E 2 j || 2 the Euclidean distance between the words k and j in the embedding's space. Our goal is the following: given two documents d 1 n , d 2 m in two languages, estimate the distance between them by utilizing the expressiveness of their word embeddings.

The distance between two documents depends on the distances between the words they consist of. In the Earth Mover's Distance setting, the words of d 1 n can be considered as piles and the words of d 2 m as holes of earth in the D-dimensional space of the embeddings. The amount of earth in the piles and holes is described by each word's frequency. Any word of d 1 n can be transformed to any of the words of d 2 m either in total or in parts. A matrix T ∈ R V 1 ×V 2 composed by non-negative T jk elements describes how much earth from the pile of the word d 1 nj is moved to the hole of the word d 1 mk . To transform d 1 n to d 2 m for the outgoing and ingoing earth flows should be k T jk = d 1 nj and j T jk = d 2 mk , which intuitively means that every word must be transformed. Therefore, the linear optimization problem writes:

min V 1 j=1 V 2 k=1 T jk c(j, k) subject to: V 2 k=1 T jk = d 1 nj , ∀j ∈ {1, . . . , V 1 } V 1 j=1 T jk = d 2 mk , ∀k ∈ {1, . . . , V 2 } (6.1)
As transforming the words of d 1 n to d 2 m comes with the cost c(k, j), the optimization problem of Eq. (6.1) translates to the minimization of the associated cumulative cost of transforming all the words. The value of the minimal cost is the distance between the documents. Intuitively, the more similar the words between the documents are, the lower will be the costs associated to the solution of the optimization problem, which, in turn, signifies smaller document distances. For example, given "the cat sits on the mat" and its French translation "le chat est assis sur le tapis", the weights (earth piles and holes) after stopwords filtering of "cat", "sits", "mat", and "chat", "assis", "tapis" will be 1/3. Given high-quality embeddings, solving Eq. (6.1) will converge to the one-to-one transformations "cat↔chat", "sits↔assis" and "mat↔tapis", with very low cumulative cost as the paired words are similar.

The problem of Eq. (6.1) is a special case of the earth mover's distance [START_REF] Rubner | A metric for distributions with applications to image databases[END_REF]. Following [START_REF] Kusner | From word embeddings to document distances[END_REF] , we refer to it as word mover's distance (WMD) for bilingual collections. Since c(k, j) is a metric, WMD for bilingual collections is a metric [START_REF] Rubner | A metric for distributions with applications to image databases[END_REF].

The optimization problem of Eq. (6.1) requires that the vector representations of the documents d 1 n , d 2 m are l 1 -normalized. Therefore, without loss of generality one may apply any term weighting scheme (that guarantees non-negative vector elements) prior to the l 1 -normalization. In this work we investigate three schemes:

Term frequency (tf ), that represents a document using the frequency of its word occurrences.

The term frequency-inverse document frequency weighting scheme (idf ), where the term frequencies are multiplied by the words inverse document frequencies:

tf -idf (t, d) = tf (t, d) × log N + 1 df (t) + 1 .
In a collection of N documents, the document frequency df (t) is the number of documents in the collection containing the word t. The inverse document's frequency idf penalizes words that occur in many documents and is smoothed in order to prevent uninformative terms that occur in all of the documents of a collection (smoothing of the numerator); and hence to avoid zero-divisions (smoothing of the denominator).

The graph of words (gow) document representation [START_REF] Rousseau | Composition of tf normalizations: new insights on scoring functions for ad hoc ir[END_REF][START_REF] Rousseau | Graph-of-word and tw-idf: new approach to ad hoc ir[END_REF]. Following the process of Sec. 4 of [START_REF] Rousseau | Graph-of-word and tw-idf: new approach to ad hoc ir[END_REF] we represent documents by unweighted directed graphs constructed using a sliding window. Then, the word weights are:

tw-idf (t, d) = tw(t, d) × log N +1 df (t)+1
where tw(t, d) is the in-degree (number of incoming edges) of the term t in the graph of d. Gow captures long term dependences (depending on the sliding window size) and the order of the terms (the graph is directed). The terms weight tw increases with the number of contexts the term occurs with, which was shown to be a robust signal for the term's importance. Following [START_REF] Blanco | Random walk term weighting for information retrieval[END_REF], we set the window size to 6 hereafter.

The Experimental Framework

The goal of CLDD is to identify corresponding documents written in different languages. Assuming, for instance, English and French Wikipedia documents, the goal is to identify the cross-language links between the articles. The challenge is to quantify the cross-lingual document distances. Traditional retrieval approaches employing bag-of-words representations perform poorly in CLDD as the vocabularies vary across language, and words from different languages rarely co-occur.

CLDD decomposes as follows: for each article from C 1 , one needs to retrieve the corresponding article from another collection C 2 . We derive three bilingual datasets using the inter-language links of Wikipedia: (i) English-French, (ii) English-German, and, (iii) French-German, which define six CLDD problems. We consider the inter-language links to be our golden standard that will be used for calculating the evaluation measures. In the pre-processing steps we lowercase the documents, we remove stopwords, punctuation and words that occur less than three times and apply the Stanford Part-of-Speech tagger [START_REF] Toutanova | Feature-rich part-of-speech tagging with a cyclic dependency network[END_REF] to keep only the nouns, which was recommended at [START_REF] Boyd-Graber | Multilingual topic models for unaligned text[END_REF] and we also found to improve the results in our preliminary experiments. Table 6.7 (under "CLDD") summarizes these datasets.

To evaluate WMD we use both in-house and publicly available pre-trained embeddings. First, we use the open implementation of BilBOWA12 [START_REF] Gouws | Bilbowa: Fast bilingual distributed representations without word alignments[END_REF] with its default parameters. 13 BilBOWA benefits from large monolingual datasets and requires a smaller set of sentence-aligned parallel data to learn the bilingual embeddings. We construct the parallel data using the English-French and English-German parts of the Europarl v7 data [START_REF] Koehn | Europarl: A parallel corpus for statistical machine translation[END_REF]: using English as the pivot language we first construct a French-German parallel corpus and then for the English-French and English-German parallel corpora we only keep the sentences that occur in the French-German corpus. This results in ∼ 1.7M aligned sentences. Lastly, we compiled monolingual corpora with 900K articles with the same protocol using the English-French and English-German Wikipedia dumps. 14As far as pre-trained embeddings are concerned, we used the state-of-the-art embeddings 15 of [START_REF] Speer | Conceptnet 5.5: An open multilingual graph of general knowledge[END_REF], dubbed cn from ConceptNet Numberbatch hereafter. [START_REF] Speer | Conceptnet 5.5: An open multilingual graph of general knowledge[END_REF] proposed to learn word embeddings by combining distributional semantics and ConceptNet v5.5 [START_REF] Liu | Conceptnet-a practical commonsense reasoning tool-kit[END_REF] using a generalization of the retrofitting method of [START_REF] Faruqui | Retrofitting word vectors to semantic lexicons[END_REF].

We expect those embeddings to perform substantially better than those we trained with BilBOWA as much more resources were used to learn them.

The systems Previous work found the bilingual Latent Dirichlet Allocation (BiLDA) [START_REF] De | Cross-language linking of news stories on the web using interlingual topic modelling[END_REF][START_REF] Ni | Mining multilingual topics from wikipedia[END_REF][START_REF] Mimno | Polylingual topic models[END_REF][START_REF] Vulić | Probabilistic topic modeling in multilingual settings: An overview of its methodology and applications[END_REF] which is an extension of the Latent Dirichlet Allocation [START_REF] David M Blei | Latent dirichlet allocation[END_REF] in the bilingual setting to yield state-of-the-art results [START_REF] Fukumasu | Symmetric correspondence topic models for multilingual text analysis[END_REF][START_REF] Zhang | Cross lingual entity linking with bilingual topic model[END_REF][START_REF] Vulić | Cross-language information retrieval models based on latent topic models trained with document-aligned comparable corpora[END_REF][START_REF] Wang | Crosslanguage article linking with different knowledge bases using bilingual topic model and translation features[END_REF]. BiLDA is trained on either parallel or comparable corpora and learns aligned per-word topic distributions between two or more languages that best explain the latent themes of a collection. During inference it projects unseen documents in the shared topic space. In the rest, we compare:

-nBOW that represents documents by a weighted average of their words' embeddings [START_REF] Mitchell | Composition in distributional models of semantics[END_REF][START_REF] Blacoe | A comparison of vector-based representations for semantic composition[END_REF].

-BiLDA with 300 topics and collapsed Gibbs sampling for inference implemented using Numpy [START_REF] Stéfan Van Der Walt | The numpy array: a structure for efficient numerical computation[END_REF], with α = 50/K and β = 0.01. We let 200 Gibbs sampling iterations for burn-in and then sample the document distributions each 25 iterations until the 500th Gibbs iteration. For learning the topics we used 30K comparable Wikipedia documents (Table 6.7 under "BiLDA training") which is an order of magnitude bigger than what previous work used [START_REF] Vulić | Cross-language information retrieval models based on latent topic models trained with document-aligned comparable corpora[END_REF].

-Trsl that is a system that is based on off-the-shelf translation. Since in the pre-processing steps we keep only nouns, we use dictionaries between the language pairs in order to translate the documents to be retrieved in the the language of the query document. Given the translations, one may apply the different weighting schemes during vectorization. Having the vectors of the query and translated documents, we rank them according to the Euclidean distance. To generate the dictionaries we rely on Wiktionary and in particular on the methods of [START_REF] Acs | Building basic vocabulary across 40 languages[END_REF][START_REF] Ács | Pivot-based multilingual dictionary building using wiktionary[END_REF]. 16Since the translation process is based on dictionaries it is possible for a given word to have several translations: in this case we sort the words given their unigram probabilities calculated on the comparable Wikipedia datasets of Table 6.7 under "BiLDA training" and select the most frequent translation.

-WMD that is the proposed metric implemented with Scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in python[END_REF] and PyEMD [START_REF] Pele | A linear time histogram metric for improved sift matching[END_REF][START_REF] Pele | Fast and robust earth mover's distances[END_REF]. For nBOW and WMD we compare three weighting schemes (tf, idf, gow) and two types of embeddings (bw, cn). Hence, WMD cn idf applies idf term weights and uses cn embeddings.

Results

As evaluation measures we report the Mean Reciprocal Rank (MRR) [START_REF] Voorhees | The trec-8 question answering track report[END_REF] and the Precision at 1 (P@1) scores. MRR accounts for the rank of the correct answer in the returned documents. When only a single document is relevant P@1 counts how many times the correct document is returned at rank 1. Table 6.8 presents the scores achieved. First, notice that the results clearly establish the important performance improvements of WMD over the rest of the methods. In particular, WMD with the BilBOWA embeddings outperforms BiLDA in every CLDD setting except for the "En→Fr" and "Ge→Fr". Further switching from Bil-BOWA to ConceptNet Numberbatch embeddings boosts the performance of WMD significantly, achieving the best performances by very large margins for each of the six CLDD settings. This highlights the importance of high-quality embeddings. BiLDA that was found effective by previous work, outperforms nBOW, although the latter uses the ConceptNet Numberbatch embeddings that showed a great potential with WMD. While this is probably due to the weighted averaging operations which result in information loss for long documents, it further highlights the suitability of WMD for calculating document distances.

The best performing baseline system is probably Trsl that relies on an off-theshelf translation approach. Notably, Trsl performs better than WMD bw for several language pairs. Moreover, the best performing weighting scheme for this system is graph-of-words. It outperforms idf , that was found to be the most robust weighting scheme for the rest of the systems, by several points in both measures for most of the language pairs. The main limitation of Trsl concerns the out-of-vocabulary words (OOV). As with WMD we ignored OOV words. We believe however that using a more robust technique for treating OOV can result in obtaining even higher scores. To this direction, an interesting approach would be to use either subword information and select for instance the translation of the word that is most similar to the OOV or use a system that would make queries to a search tool like Google to find the translation. Another observation concerns the impact of the term weighting schemes on the performance achieved by the WMD approach. The gow scheme performs better than tf counts in most of the cases, especially when the ConceptNet Numberbatch embeddings are used. Overall, idf is the best performing weighting scheme. As an interesting direction of future work, one may improve the results of the gow by tuning parameters like the sliding window size used to construct the graph in order to generate graphs that capture the syntactic rules of the languages.

Summary

In this work we adapted the Word Movers Distance metric for CLDD. Our results demonstrate the effectiveness of the proposed approach which we attribute to the ability of the model to quantify document similarity using word level information and high quality word embeddings. Our study open avenues for future research. First, one could further improve the obtained results by tuning the parameters of the weighting schemes or adapt the proposed approach to other multilingual and cross-lingual tasks like multilingual document clustering. Second, the promising results achieved suggest that tasks like CLDD or document classification (used in [START_REF] Kusner | From word embeddings to document distances[END_REF]) are a good fit for comparing methods for learning embeddings.

A last promising research avenue would be to incorporate the distance estimation between documents that WMD provides into topic models. Our findings in Chapter 5 suggest that the quality of the learned topics when incorporating crosslingual word embeddings improves and we showcases this potential by computing a similarity metric between documents based on the cosine between document representations. The latter were generated by averaging the representations of words of a document. Our results here suggest that WMD offers a more efficient way for calculating documents distances that considers every words separately instead of the their averaged representations. We believe that integrating this to the presented topic models has the potential to further improve them.

Chapter Summary

In this chapter we investigated how three text mining tasks can benefit from the use of word embeddings. Our goal for each case was to obtain robust representations that integrate parts of external knowledge. The knowledge source was different for each task:

• In the case of polylingual classification, we assumed access to translations of a documents and used this information to enrich the learned representation using a denoising autoencoder.

• In the case of multitask learning we assumed that there exist two (or more) correlated tasks and we argued that in such a scenario jointly learning the tasks can be beneficial.

• Lastly, for the third task that was cross-lingual information retrieval we suggested that one may not need aggregate word embeddings (which was explicitly done by averaging on via the LSTM network in the first two tasks) to learn a decision function but use a formulation of the transportation problem to derive the distances of documents written in different languages.

Overall, our findings suggest that using word representations learned with models that implement the distributional hypothesis can achieve competitive performance. Furthermore, such models are flexible enough to incorporate different types of knowledge that can further improve performance.

131 a second that uses copulas which allow for more flexibility. Through extensive experimentation on various datasets and tasks we demonstrated that knowledge of text structure is indeed beneficial for topic models. As unsupervised exploration of text collections is an important task with the ever growing amount of data being generated the proposed models can help us to better understand the topics discussed in them and also extract features efficiently from them.

Bilingual Topics Models for Comparable Corpora In Chapter 5 we proposed to better adapt bilingual topic models for comparable corpora with explicit alignments. Motivated by the fact that such corpora are easier to obtain than parallel, we proposed to extend the bilingual Latent Dirichlet Allocation and allow for different topics distributions for the documents of each language. After systematic evaluation of the proposed models, we showed that extending bilingual topic models and adapting them for bilingual collections improves the topical coherence of the learned topics, the generalization performance of the models as well as the performance of the documents representations learned in the task of cross-lingual document retrieval.

Word Embeddings for Text Mining Applications

In Chapter 6 we investigated how text mining applications can benefit from word embeddings. Motivated by previous research suggesting that such word representations capture semantic and syntactic word properties we proposed models and algorithms for polylingual classification, multi-task classification and cross-lingual document retrieval. Our results confirmed that the tasks at hand can strongly benefit from rich text representations and efficient models. Our observations complemented seminal results on the fields of representation learning and deep learning for natural language processing tasks about the potential of word embeddings and neural network architectures.

Future Directions

In this section, we discuss future research directions for the topics covered in the manuscript as well as more broad topics of interest in the areas of text mining and natural language processing in general.

Incorporating Rich Text Structure to Topic Models

For the models of Chapter 4 we considered segments to be contiguous words that are observed in the text. However, one can discover far more structure in documents: from documents to paragraphs, from paragraphs to sentences and then from sentences to parse trees, one may imagine various ways to represent documents as complex structures like trees [START_REF] Blei | Syntactic topic models[END_REF] of graphs [START_REF] Rousseau | Graph-of-word and tw-idf: new approach to ad hoc ir[END_REF]. Extending topic models to account for this complex structure is an interesting problem. For instance, copulas can be extended by nesting; nested copulas in turn can model tree-like dependencies which can provide the means for further extending topic models. Another issue, which is rarely touched, is the scalability of such complex models. Addressing the questions "How complexity impacts the obtained results ?" as well as "How scalable are these algorithms for truly big data ?" would provide further useful insights.

Beyond Comparable Corpora with Explicit Alignments The setting we considered in Chapter 5 required input corpora to be in the form of pairs of documents that should be topically aligned. Our motivation was that such corpora are more common that parallel. However, in the most general case one has only access to comparable corpora without any type of alignments. Can we apply the findings of this thesis as well as those of related work to address the challenges of this exciting setting? We believe that by combining the models of Chapter 5 with the cross-lingual document retrieval process outlined in Chapter 6 is a promising research direction.

Word Embeddings for Text Mining Applications In Chapter 6 we explored three different text mining applications. For the case of polylingual classification we validated the hypothesis that translations of a document result in document representations that improve the performance in the task of text classification. The approach used here motivated experiments with more and different languages. Is the hypothesis valid for languages that are very dissimilar like English and Japanese for instance? Are there any benefits when the number of more languages used to learn the representation increases?

We also discussed multi-task learning with neural networks. We showed that by jointly learning two sentiment classification tasks for tweets improved the performance of the fine-grained text classification task. Multi-task learning is a very promising area of research as it allows for knowledge transfer between different fields. For the case of sentiment analysis, it would be interesting to see if similar results can be obtained by varying the type of text.For instance, instead of using only tweets one could also use reviews of hotels or products. The hope is that by incorporating more data in the process of training one can arrive at selecting a better hypothesis that would improve the final performance. The direction of investigating different types of tasks is also interesting: instead of focusing only on sentiment classification tasks, one may want to include other NLP tasks like Partof-Speech tagging of Named-Entity-Recognition that would add a different type of inductive bias in the process.

The last application we investigated concerned cross-lingual documents retrieval. We used the solvers for the problem of optimal transport with ground distances estimated by the cross-lingual embeddings. An important aspect of this approach is the computational cost. Although the distance estimation between documents is straightforward to parallelize, it is still computationally expensive.

Therefore, aceclerating the solution by aggressive sampling techniques so that less problems can be solved is another promising area. Also, in the community of compute vision area, earth movers distance has been shown to perform well when used as a kernel [START_REF] Gardner | Earth mover's distance yields positive definite kernels for certain ground distances[END_REF][START_REF] Reza | Kernel earth mover's distance for eeg classification[END_REF] for classification tasks. We believe that building on these findings may lead to similar findings for the Word Movers Distance presented in this chapter.

In terms of complexity, the quantification cost is reduced to the prediction for the already trained classifier, which is linear in the cardinality of the set S on which quantification is performed. The computational cost for Algorithm 4 is thus the same as 1-fold cross-validation. Additionally, as only one hypothesis is generated

A.3 Experimental Framework

To empirically evaluate the model selection method presented above we use the publicly available datasets of the LSHTC 2011 (Large Scale Hierarchical Text Classification) challenge [START_REF] Partalas | LSHTC: A benchmark for large-scale text classification[END_REF]. Specifically, we make use of the Dmoz and Wikipedia datasets containing 27,875 and 36,504 categories respectively. The datasets are provided in a pre-processed format using stop-word removal and stemming while we transformed the term-frequency vectors to the tf*idf representation. For each of the datasets we randomly draw several datasets with increasing number of classes.

Table A.1 presents the important statistics of the different datasets. As one can note, the number of categories in our datasets ranges from 250 to 2, 500, and the number of features from 26, 000 to 212, 000. An interesting property of the instances of those datasets is the fit to the power law distribution. As a result, there are several under-represented classes having a few labelled examples. Thus, model selection approaches using only a fraction of the labeled instances, such as hold out, may lead to sub-optimal decisions.

The classification problems defined from our datasets are multi-class, and we adopt a standard one-vs-rest approach to address them (the large datasets considered prevents one from using more complex multi-class approaches). The Dmoz dataset is single-labeled, i.e. each training/test instance is associated to a single target category. On the other hand, the Wikipedia dataset is multi-labeled with the average labels per instance in the training set being 1.85. We transformed the multi-label problem to single label, both in the training and the test phase, by replicating the multi-labeled instances according to the number of their labels.

In order to empirically measure the effectiveness of model selection, we compare the following three methods: (i) k-CV, using k = 5 folds, (ii) hold-out estimation with a split of 70% and 30% for the training and the validation sets, and (iii) our method using as quantification set i) an unlabeled set denoted "quantification set" in Table A |S| | is small. As mentioned above, this quantity is null for the quantification method CC, which thus agrees with our theoretical developments. The other quantification method considered, PCC, is based on the probabilities that an instance belongs to a class. When using LR, those probabilities are directly produced by the model. For SVMs, however, one needs to transform the confidence scores into probabilities, which can be done in several ways, as using a logistic function, a multivariate logistic regression function or neural networks based on logistic activation functions and without hidden layers (the latter two settings can be seen as generalizations of Platt's scaling for the multi-class problem). We obtained the best results with a simple logistic function defined as 1 1+e -σ t , varying σ from 1 to 10. Table A.2 displays the values of Max obtained for PCC for each of the dataset and for each classifier (the default hyper-parameter values of the classifiers are used), using the value of σ leading to the lowest value of Max . As one can note, although the values obtained are small in most cases (except for Dmoz 1000 and Dmoz 1500 ), there are not negligible compared to the class prior probabilities, which are in the range of 1 divided by the number of classes. Thus, the quantification method PCC does not fully agree with our theoretical development. It turns out that it also performs worse than CC in practice. We thus rely on this latter method for the rest of our experiments. Model Selection Evaluation. We evaluate model selection methods for two families of classifiers: (i) SVMs, and (ii) LR which are among the best performing models in text classification. We explore for both classifiers the value for the regularization parameter λ ∈ {10 -3 , 10 -2 , . . . , 10 4 }. We used the implementations in Python's scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] that are wrappers of the LibLinear package [START_REF] Fan | LIBLINEAR: A library for large linear classification[END_REF].

We report the scores obtained in Accuracy and Macro-F (MaF) measure when a classifier is applied on the test set. In particular, for each dataset of Table A.1 the model selection methods are used only for selecting the regularization parameter λ when optimizing for the repsective measure. After the selection of λ, the classifier is retrained on the entire training set, and we report its performance in the test set. This last step of retraining is not required for our method since the classifier is trained in the overall labeled set from the beginning. Also, as hold-out estimation may be sensitive to the initial split, we perform 10 different random splits training/validation and report the mean and the standard deviation of the scores obtained for both evaluation measures. The curve MaF corresponds to the actual MaF on the test set. Although each parameter estimation method selects the value for λ that seems to maximize the performance, the goal in this example, ultimately, is to select the value that maximizes the performance of MaF. For instance, hold-out, by selecting λ = 10 -1 , fails 166 to select the optimal λ value, while all other the methods succeed. Here, the 5-CV approach requires 1310 sec., whereas the bound approach only requires 302 sec. (the computations are performed on a standard desktop machine, using parallelized implementations on 4-cores). The bound approach is thus 4.33 times faster, a result consistent over all experiments and in agreement with the complexity of each approach (Section 3). Lastly, we notice that the curve for Bound UN with the quantification method CC follows the MaF curve more strictly than the curve with the quantification method PCC.

Table A.3 presents the evaluation of the three model selection methods using as classifiers SVM and LR respectively. As one can note, the performance of the method proposed here is equivalent to the one of cross-validation, for all datasets, and for both classifiers and performance measures (accuracy and MaF). The performance of SVM is furthermore higher than the one of LR on all datasets, and for both evaluation measures, the difference being more important for the MaF. The performance of cross-validation however comes with the cost of extra processing time, as our method achieves a k speed-up compared to cross-validation. If both methods can easily be parallelized (at least on the basis of the number of values of the hyper-parameter to be tested), k-fold cross validation requires k times more computing resources than our method.

Unlike cross-validation, hold-out estimation fails to provide a good model in many instances. This is particularly true for SVMs and the MaF measure, for which the model provided by hold-out estimation lies way behind the ones provided by Bound UN and Bound Test on several collections as Dmoz 1500 and Dmoz 2500 . The difference is less important for LR, but the final results in that case are not as good as in the SVM case.

A.4 Summary

We presented in this work a new method for model selection that is able to exploit unlabeled data (this is in contrast with current model selection methods). To do so, we have introduced quantification-based bounds for accuracy and macro performance measures. We have then shown how to apply this bound in practice, in the case where unlabeled data is available in conjunction with labeled data, and in a transductive-like setting where the instances to be classified are known in advance. The experimental results, obtained on 10 datasets with different number of classes ranging from 250 to 2,500, show that the method proposed here is equivalent, in terms of the quality of the model selected, to k-fold cross-validation, while being k times faster. It furthermore consistently outperforms hold-out estimation for SVM classification, for both accuracy and macro-F1, the difference being more important for macro-F1. Furthermore, and contrary to hold-out estimation, our method needs neither a validation/train splitting procedure nor a retraining procedure.

In our future work we plan to investigate the application of a generalized version of the proposed model selection approach in cases where more than one hyper-parameters have to be tuned. In this framework, we also plan to research the extension of the theoretical and experimental findings to multi-label classification problems i.e., multi-class classification problems where each instance can be given more than one categories at once.

Figure 2 . 1 :

 21 Figure 2.1:The Vector Space Model for a document (above) and a collection of documents (below). A document is transformed to a vector with non-zero indexes corresponding to the frequency of the words associated to these indexes. Repeating the process for each document of a collection creates the term-document matrix.

Definition 3 A

 3 parallel corpus is a comparable corpus in two or more languages 1 , 2 , . . . . The documents have explicit thematic alignments, and the content of d 1 i

Figure 3 . 1 :

 31 Figure 3.1: Given an one-hot-encoded text input, a dense layer produces a word embedding whose output propagates to the neural network. The matrix W that holds the word embeddings is fine-tuned for the particular task.

Figure 3 .

 3 Figure 3.2 illustrates the model used to learn the skipgram word embeddings [124,

Figure 3 . 3 :

 33 Figure 3.3: The continuous bag-ofwords model.

4 Figure 3 . 4 :

 434 Figure 3.4: An unrolled Recurrent Neural Network.N maybe an arbitrarily deep neural network, that given an input x 1 , which can be a word embeddings for instance, propagates a hidden state to its successor. The RNN can be very long[START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF].

  .5. A central idea for LSTMs is the cell state, illustrated in the Figure as the top continuous line [C t-1 , C t ]. The cell is an information flow whose content is updated via one element-wise multiplication and one element-wise addition operation. The rest of the hidden layers, known as gates, control how the information of the cell is updated.

Figure 4 . 1 :

 41 Figure 4.1: Applying LDA on Wikipedia documents. Notice how LDA assigns several, unrelated topics to the excerpt.

Figure 4 . 2 :

 42 Figure 4.2: Shallow parsing of a sentence with the Stanford Parser. Contiguous words in italics like "Richard Widmark" denote a noun-phrase.

Figure 4 . 3 :

 43 Figure 4.3: The graphical model of segLDA.

Figure 4 . 4 :

 44 Figure 4.4: The transformation of a random variate to multinomial (or arbitrary) marginals. The arrows illustrate the generalized inverse; the histograms in y (resp.x) axis depict the distributions of the initial (resp. transformed) samples.

  a random sample with Frank copula

Figure 4 . 5 :

 45 Figure 4.5:The positive correlation imposed to two random variates when sampling from a Frank copula with increasing values of λ. λ ranges in[START_REF] Arlot | A survey of cross-validation procedures for model selection[END_REF][START_REF] Balikas | Twise at semeval-2016 task 4: Twitter sentiment classification[END_REF][START_REF] Baroni | Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors[END_REF][START_REF] Blum | Beating the hold-out: Bounds for k-fold and progressive cross-validation[END_REF] from top-left to bottom right respectively.

  Figure 4.6:The copLDA generative model. We model the dependency between the topics underlying a segment with copulas. Notice in the graphical model how the topics of (z 1 , . . . , z N ) of a segment depend both on θ and the copulas, illustrated with its λ parameter.

Figure 4 . 7 :

 47 Figure 4.7: The effect of rejection sampling in efficiency and perplexity performance for copLDA.

Figure 4 . 8 :

 48 Figure 4.8: The perplexity curves of the investigated models for 200 Gibbs sampling iterations and different datasets.

Figure 4 .

 4 Figure 4.8 illustrates the perplexity curves of the hold-out documents for seven of the models on two of the datasets of Table 4.1 for 250 Gibbs sampling iterations.

. 5 )

 5 Notice how both assumptions are relaxed is this model: the first term of the result (discussed in the Appendix of the chapter) shown in Eq. (5.5) accounts for the topic dependence between the paired documents, while the second incorporates the segment boundaries. The Gibbs sampling updates for λ-segBiLDA are Algorithm 3: A Gibbs Sampling iteration for λ-segBiLDA Input: the words of the document pairs, cross-lingual embeddings, α, β, K for document pair

Figure 5 .

 5 3 depicts, the learned topics are aligned. Therefore, one

Figure 5 . 3 :

 53 Figure 5.3: The perplexity curves of the four models for the designated datasets for 200 Gibbs sampling iterations when K = 150 topics. In the left column of the figures we visualize the perplexity calculated for the English documents of the comparable pairs while in the right column the documents of the second language of the pair. The proposed model λ-BiLDA consistently outperforms the rest of the systems.

Figure 6 . 1 :

 61 Figure 6.1: An AE that generates the PE in its hidden layer. The dashed boxes denote the document DRs in the corresponding language.

  both as a baseline model and when used as base model to generate the PE representations. On the other hand, DBOWpv and DMMpv perform significantly worse: in the baseline setting the best cbow performance achieved is 44.25 whereas the best DMMpv configuration achieves 30.49, 14 F 1 points less. The PE representations learned on top of the four base models improve significantly over the performance of the monolingual DRs, especially for k-NN. For instance, for cbow with base-model vector dimension 200, the baseline representation achieves 40.42 F 1 and its corresponding PE representation obtains 46.33, improving almost 6 points. In general, we notice such improvements between the base DR and its respective PE, especially when the dimension of the DR representation increases. Note that the PE improvements are independent of the methods used to generate the DRs: for instance k-NN PE over the 200-dimensional PE DMMpv representations gains more than 11 F 1 points compared to k-NN DR . It is also to be noted that the baseline SVM BoW is outperformed by SVM PE especially when cbow and skip-gram DRs are used.

Figure 6 . 3 :

 63 Figure 6.3: The neural network architecture for multitask learning. The biLSTM output is transformed by the hidden layers H 1 , H M and is led to N output layers, one for each of the tasks. The lower part of the network can be used to incorporate additional information.

  for each parameter value by training to the whole set of labeled data one has just to select the hypothesis with the highest bound without the need of retraining the model in contrast to hold-out or k-fold cross-validation. More precisely, the complexity of our approach for m values of λ is O([Tr(N) + Pr(M)]×m), which is k times lower than the complexity of k-CV with re-training the learner for the selected λ value, given by O([Tr(( k-1 k )×N)+Pr( 1 k ×N)]×k×m+T(N)), where Tr(N), Pr(N) are the training and predicting costs for N examples.

Figure A. 1

 1 Figure A.1 illustrates the model selection decisions for the different methods using an SVM on the Wikipedia dataset with 1, 500 classes for the MaF measure.

Figure A. 1 :

 1 Figure A.1: Model selection process for SVM on the wiki 1500 for MaF. The squares denote the best performance for each method.

  Counter variable: per topic word assignments, Ψ ∈ R V ×K Ω Counter variable: per document word assignments, Ω ∈ R D×K

	Symbol Description
	K number of topics
	V The size of the vocabulary
	α concentration hyper-parameter of per-document topic distribution prior
	β concentration hyper-parameter of per-word topic distribution prior
	k topic variable placeholder, k ∈ [1, K]
	i document variable placeholder, i ∈ [1, D]
	θ i topic distribution of the i-th document
	φ per word topic distribution
	Ψ

Table 2 .

 2

1: Notation used for the development of Latent Dirichlet Allocation.

  Algorithm 2: A Gibbs Sampling iteration for copLDA Input: documents' words grouped in segments, α, β, K, Copula family and its parameter λ //Initialize counters Ψ , Ω for document d i , i ∈ [1, D] do for segment s i,j : j ∈ {1, . . . , S i } do Draw a random vector U = (U 1 , . . . , U N i,j ) that admits a copula C λ do / * If the mean approximation is used, the loop is done once, ignoring the acceptance condition * / for words w i,j,k , k ∈ [1, W N i,j ] in s i,j do Decrease counter variables Ψ , Ω Get z i,j,k by transforming U k to Mult. marginals with the generalized inverse Assign topic z i,j,k to w i,j,k Increase counters Ψ , Ω end while Accept the new segment topic assignments with probability

Table 4 .

 4 Statistics of the datasets used

		Docs.	|N |	|V |	Categories Categories/Instance
	TED	1,096 1.16M 30.4K	15	2.42
	PubMed 5498 1.09M 28.7K	50	1.32
	Reuters 10,788 875K 21.4K	90	1.23
	20NG	19,056 1.7M 75.4K	20	1.0
	Wiki15	1,198	162K 13.4K	15	1.0
	Wiki37	2,459	317K 19.7K	37	1.0
	Wiki46	3,657	478K 23.4K	46	1.0
	Austen	5,262	170K	6.3K	-	-

1: The basic statistics of the datasets used for evaluating the topic models. |N | denotes the total words in the corpus, |V | the vocabulary size, while "Categories" and "Categories/Instance" the size of the category set of the corpus and the average number of categories per document respectively.

Table 4 .

 4

	Profession	Science	Books	Art	Cinema	Places	Music	Birth	Elections Inventions
	profession	univers	book	art	film	state	record	born	elect	california
	world	research	new	new	televis	unit	music	american canadian	plant
	footbal	scienc	work	work	role	us	band	known	parti	use
	wrestl	professor american	paint	appear township album	best	member	invent
	play	work	publish	york	also	school	song	actress	liber	flower
	born	institut	time	american	actor	univers	also	decemb	minist	compani
	american	award	author	artist	born	serv	produc	june	hous	north
	championship	prize	also	museum	play	war	releas	april	canada	patent
	team	born	year	painter	seri	nation	new	juli	serv	inventor
	first	receiv	york	studi	star	build	singer	januari	conserv	found
	known	univers	book	art	film	township record	play	elect	work
	wrestl	research	new	new	born	state	music	footbal	canadian	first
	born	scienc american	york	televis	counti	band	born	serv	year
	world	professor author	paint	role	us	album	american	parti	photograph
	profession	work	publish american	actor	california	song	tour	member	design
	american	institut	novel	work	appear michigan	also	golf	liber	state
	name	born	time	artist	also	plant	singer	year	hous	new
	wrestler	prize	also	painter	seri	civil	releas	profession minist	use
	best	studi	writer	museum actress	popul	produc	first	state	also
	championship award	magazin	born	american flower american	season	born	build

2: The topics learned by copLDA (upper half) and LDA (lower half) in the Wiki46 dataset. Kiss of Death is a 1995 crime thriller film starring David Caruso Samuel L. Jackson and Nicolas Cage. The film is a very loosely based remake of the 1947 film noir classic of the same name that starred Victor Mature, Brian Donlevy and Richard Widmark. Kiss of Death is a 1995 crime thriller film starring David Caruso Samuel L. Jackson and Nicolas Cage. The film is a very loosely based remake of the 1947 film noir classic of the same name that starred Victor Mature, Brian Donlevy and Richard Widmark. Bertram Stern (born 3 October 1929) is an American fashion and celebrity portrait photographer. Bertram Stern (born 3 October 1929) is an American fashion and celebrity portrait photographer. Dana Hill (born Dana Lynne Goetz in Los Angeles, California; May 6, 1964 -July 15, 1996) was an American actress and voice actor with a raspy voice and childlike appearance, which allowed her to play adolescent roles well into her 20s. Dana Hill (born Dana Lynne Goetz in Los Angeles, California; May 6, 1964 -July 15, 1996) was an American actress and voice actor with a raspy voice and childlike appearance, which allowed her to play adolescent roles well into her 20s.

Table 4 .

 4 3: The discovered topics underlying the words of example documents for LDA (left) and copLDA (right). The parts of the documents in italics indicate the noun-phrases obtained by the Stanford Parser. The text colors refer to the topics described in Table4.2.

Table 4 .

 4 4 illustrates the achieved perplexity scores for the datasets of Table 4.1 and for the number of topics K ∈ {25, 50, 75, 100, 150}. The best (lowest) perplexity score per dataset and number of topics is shown in bold. There are several observation to be made from the table.

  LDA segLDA np copLDA np segLDA sent copLDA sent segLDA bi copLDA bi segLDA tri copLDA tri

	Dataset K
		1,626	1,654	1,616	1,747	1,642	1,618	1,618	1,613	1,610
		1,508	1,501	1,485	1,625	1,520	1,469	1,482	1,487	1,483
	20NG	1,464	1,460	1,435	1,600	1,464	1,426	1,435	1,437	1,432
		1,431	1,435	1,415	1,547	1,451	1,385	1,414	1,412	1,408
		1,434	1,401	1,422	1,550	1,422	1,381	1,406	1,406	1,405
		761	764	773	948	831	748	770	773	775
		748	744	748	939	815	709	754	751	751
	Austen	748	730	746	945	826	708	745	745	760
		757	729	755	953	825	696	751	746	756
		762	728	761	966	845	699	763	761	766
		1,093	1,122	1,098	1,249	1,093	1,092	1,098	1,118	1,117
		961	987	972	1,160	990	957	968	992	999
	PubMed	928	951	932	1,130	954	904	926	955	954
		930	922	914	1,110	946	884	927	930	957
		938	891	939	1,095	957	867	937	940	962
		548	568	551	594	590	555	549	620	615
		487	502	489	545	530	479	486	543	554
	Reuters	468	470	469	528	511	448	464	518	527
		464	452	459	508	499	436	455	505	518
		454	437	455	491	495	426	456	495	511
		1,645	1,645	1,640	1,681	1,636	1,652	1,640	1,651	1,661
		1,579	1,587	1,568	1,647	1,579	1,571	1,574	1,585	1,590
	Ted	1,543	1,546	1,544	1,641	1,551	1,551	1,545	1,560	1,563
		1,518	1,531	1,522	1,637	1,533	1,512	1,527	1,531	1,537
		1,501	1,503	1,505	1,634	1,505	1,492	1,508	1,518	1,519
		1,161	1,213	1,156	1,410	1,195	1,150	1,148	1,237	1,239
		1,095	1,133	1,096	1,418	1,143	1,077	1,104	1,181	1,184
	Wiki15	1,101	1,120	1,092	1,426	1,144	1,037	1,092	1,167	1,188
		1,090	1,097	1,113	1,405	1,159	1,029	1,107	1,168	1,205
		1,119	1,098	1,149	1,393	1,201	1,023	1,146	1,184	1,238
		1,357	1,422	1,343	1,636	1,386	1,358	1,356	1,446	1,451
		1,270	1,311	1,272	1,558	1,315	1,254	1,271	1,348	1,356
	Wiki37	1,260	1,292	1,271	1,584	1,305	1,207	1,269	1,325	1,355
		1,271	1,273	1,281	1,562	1,312	1,209	1,274	1,335	1,358
		1,290	1,273	1,306	1,556	1,363	1,208	1,301	1,346	1,405
		1,366	1,427	1,366	1,561	1,417	1,358	1,360	1,441	1,442
		1,259	1,278	1,261	1,455	1,302	1,211	1,254	1,312	1,336
	Wiki46	1,227	1,224	1,233	1,396	1,278	1,151	1,214	1,274	1,288
		1,216	1,181	1,226	1,355	1,276	1,153	1,228	1,266	1,303
		1,236	1,125	1,243	1,301	1,298	1,123	1,240	1,269	1,319

Table 4 .

 4 4: The perplexity scores (Eq. 4.9) achieved on each of the datasets used. The best scores for each dataset are shown in bold font.

K

  LDAsegLDA np copLDA np segLDA sent copLDA sent segLDA bi copLDA bi segLDA tri copLDA tri20NG.118 ±.07 .127 ±.07 .113 ±.06 .108 ±.07 .117 ±.07 .112 ±.06 .115 ±.07 .084 ±.05 .117 ±.06 .111 ±.07 .114 ±.06 .113 ±.06 .113 ±.07 .111 ±.07 .110 ±.06 .113 ±.06 .081 ±.06 .107 ±.06 .105 ±.06 .109 ±.07 .103 ±.06 .107 ±.06 .109 ±.07 .103 ±.05 .104 ±.06 .068 ±.06 .109 ±.06 .103 ±.06 .112 ±.06 .105 ±.06 .102 ±.06 .101 ±.06 .096 ±.05 .097 ±.06 .071 ±.06 .100 ±.06 .098 ±.06 .102 ±.06 .098 ±.06 .097 ±.06 .095 ±.06 .093 ±.06 .096 ±.05 .058 ±.05 .095 ±.05 Austen .076 ±.04 .077 ±.04 .082 ±.04 .088 ±.04 .084 ±.04 .082 ±.03 .077 ±.04 .060 ±.03 .076 ±.04 .069 ±.04 .072 ±.03 .066 ±.03 .070 ±.04 .066 ±.03 .080 ±.04 .068 ±.03 .057 ±.03 .068 ±.03 .065 ±.03 .062 ±.03 .066 ±.03 .064 ±.03 .064 ±.03 .068 ±.03 .064 ±.03 .052 ±.02 .065 ±.03 .062 ±.03 .064 ±.03 .060 ±.03 .064 ±.03 .062 ±.03 .066 ±.03 .062 ±.03 .046 ±.02 .063 ±.03 .057 ±.03 .059 ±.03 .059 ±.03 .059 ±.03 .059 ±.03 .063 ±.03 .058 ±.03 .045 ±.03 .058 ±.03 PubMed .154 ±.07 .165 ±.08 .167 ±.07 .154 ±.06 .153 ±.06 .153 ±.07 .153 ±.07 .153 ±.06 .156 ±.06 .156 ±.07 .154 ±.08 .159 ±.07 .163 ±.08 .158 ±.07 .149 ±.08 .160 ±.07 .136 ±.07 .165 ±.07 .159 ±.08 .153 ±.08 .161 ±.08 .143 ±.08 .156 ±.08 .155 ±.08 .158 ±.08 .135 ±.08 .159 ±.08 .150 ±.07 .153 ±.08 .150 ±.08 .149 ±.07 .153 ±.08 .156 ±.07 .148 ±.08 .134 ±.07 .150 ±.08 .143 ±.07 .144 ±.08 .143 ±.08 .137 ±.07 .139 ±.07 .147 ±.07 .146 ±.07 .116 ±.07 .142 ±.08 Reuters .055 ±.03 .061 ±.04 .054 ±.03 .062 ±.03 .061 ±.03 .067 ±.03 .052 ±.03 .047 ±.03 .054 ±.03 .056 ±.03 .073 ±.04 .060 ±.03 .062 ±.03 .057 ±.03 .064 ±.03 .055 ±.03 .050 ±.03 .048 ±.03 .057 ±.03 .066 ±.04 .057 ±.03 .062 ±.03 .056 ±.03 .064 ±.03 .053 ±.03 .047 ±.03 .051 ±.03 .056 ±.03 .070 ±.04 .055 ±.03 .057 ±.03 .058 ±.03 .061 ±.03 .056 ±.03 .048 ±.03 .049 ±.03 .057 ±.03 .068 ±.04 .054 ±.03 .060 ±.04 .055 ±.03 .057 ±.03 .053 ±.04 .046 ±.03 .051 ±.03 Ted .070 ±.04 .090 ±.04 .072 ±.04 .087 ±.04 .074 ±.05 .073 ±.04 .069 ±.04 .063 ±.03 .064 ±.04 .074 ±.05 .099 ±.04 .081 ±.05 .087 ±.05 .080 ±.05 .074 ±.04 .079 ±.06 .064 ±.04 .078 ±.05 .080 ±.05 .100 ±.05 .074 ±.05 .089 ±.05 .079 ±.05 .076 ±.05 .075 ±.05 .054 ±.03 .077 ±.05 .073 ±.04 .094 ±.05 .072 ±.05 .075 ±.05 .076 ±.05 .071 ±.04 .073 ±.05 .056 ±.04 .074 ±.05 .067 ±.04 .086 ±.06 .070 ±.05 .077 ±.05 .071 ±.05 .067 ±.04 .071 ±.05 .051 ±.03 .069 ±.05 Wiki15 .107 ±.06 .103 ±.06 .102 ±.06 .112 ±.06 .099 ±.06 .109 ±.06 .098 ±.06 .132 ±.05 .105 ±.06 .088 ±.06 .086 ±.05 .084 ±.05 .084 ±.06 .081 ±.06 .086 ±.06 .086 ±.06 .106 ±.06 .085 ±.06 .077 ±.06 .082 ±.06 .072 ±.06 .073 ±.06 .073 ±.05 .078 ±.05 .076 ±.06 .096 ±.06 .074 ±.06 .071 ±.05 .076 ±.05 .068 ±.05 .071 ±.05 .066 ±.05 .070 ±.05 .066 ±.06 .085 ±.05 .069 ±.05 .063 ±.05 .064 ±.05 .058 ±.05 .060 ±.05 .054 ±.05 .067 ±.05 .062 ±.05 .077 ±.05 .059 ±.05 Wiki37 .115 ±.05 .124 ±.06 .126 ±.05 .110 ±.05 .105 ±.06 .110 ±.04 .120 ±.05 .127 ±.05 .120 ±.05 .103 ±.06 .107 ±.06 .102 ±.06 .101 ±.06 .098 ±.06 .104 ±.05 .097 ±.06 .112 ±.05 .102 ±.05 .093 ±.06 .093 ±.06 .089 ±.06 .089 ±.06 .088 ±.06 .096 ±.06 .088 ±.06 .100 ±.06 .095 ±.06 .081 ±.06 .086 ±.06 .083 ±.06 .083 ±.06 .079 ±.06 .088 ±.06 .089 ±.06 .093 ±.06 .083 ±.06 .078 ±.06 .083 ±.06 .070 ±.06 .071 ±.05 .071 ±.05 .079 ±.05 .075 ±.06 .084 ±.06 .072 ±.06 Wiki46 .119 ±.05 .123 ±.06 .120 ±.05 .114 ±.06 .105 ±.05 .119 ±.05 .106 ±.06 .120 ±.05 .117 ±.05 .105 ±.06 .109 ±.06 .109 ±.06 .107 ±.06 .100 ±.06 .108 ±.06 .104 ±.06 .094 ±.06 .115 ±.06 .092 ±.06 .105 ±.06 .099 ±.06 .094 ±.06 .090 ±.06 .099 ±.06 .101 ±.06 .084 ±.06 .095 ±.06 .089 ±.06 .090 ±.06 .088 ±.06 .083 ±.06 .079 ±.06 .086 ±.06 .089 ±.06 .079 ±.06 .092 ±.06 .081 ±.06 .084 ±.06 .082 ±.06 .074 ±.06 .077 ±.06 .082 ±.06 .075 ±.06 .072 ±.05 .076 ±.06

Table 4 .

 4 5: nPMI scores (Eq. 4.10) for the topic models for different values of topics. The best scores for each dataset are shown in bold font.

Table 4 . 6 :

 46

	Topic nPMI nPMI 5 /nPMI 10 /nPMI 15	Top-15 words per topic
	1	.17	.20/.18/.13	case patient tumor diagnosi present clinic report diseas lesion find his-
				tolog examin year one rare
	2	.10	.10/.10/.09	use care health rate cost hospit system provid data payment state medic
				medicar new physician
	3	.27	.21/.29/.31	cancer mutat polyposi patient famili apc colon adenomat adenoma tumor
				fap polyp colorect intestin carcinoma
	4	.24	.40/.18/.13	gene chromosom mutat tumor genet region delet patient identifi loss
				analysi famili studi allel use
	5	.15	.14/.17/.15	pressur renal rat sodium increas furosemid blood effect hypertens signific
				plasma decreas diet p intak
	6	.17	.26/.13/.12	bind protein domain interact structur repeat activ residu region site two
				function peptid complex contain
	7	.13	.13/.14/.12	rat activ increas mice level vascular express protein effect endotheli diabet
				aorta product control mrna
	8	.05	.03/.05/.07	muscl chang dure observ studi activ mitochondri complex respons differ
				occur motor increas fiber membran
	9	.18	.22/.18/.15	protein gene sequenc human express dna acid isol region encod virus tran-
				script contain two strain
	10	.10	.14/.10/.07	patient injuri trauma sever result day fractur score care hospit injur con-
				clus admiss hour multipl
	11	.12	.13/.12/.12	patient result use procedur anastomosi urinari sunscreen pouch bladder
				protect skin ileal function complic contin
	12	.11	.14/.10/.10	effect inhibit rat activ relax ca k contract induc increas respons concentr
				phosphoryl cl aorta
	13	.16	.20/.15/.14	acid increas effect fatti concentr group level diet enzym decreas signific
				activ product lipid dietari
	14	.03	.00/.04/.05	patient group p signific age studi year differ risk associ factor result com-
				par conclus n
	15	.26	.38/.24/.16	hiv infect virus viral drug therapi patient treatment studi resist load test
				combin effect activ
	16	.24	.36/.20/.14	patient seizur tempor epilepsi lobe tle later surgeri eeg result hippocamp
				ictal method onset left
	17	.13	.14/.12/.12	neuron brain rat control loss chang increas signific epilepsi patient hip-
				pocamp activ function cell anim
	18	.08	.12/.07/.07	liver hepat seal fetal male femal infant group found anim bodi speci fetus
				per signific
	19	.10	.10/.10/.10	use test method latex studi sensit patient imag result detect specif evalu
				posit b assay
	20	.08	.11/.07/.06	use method motion measur model result system dose time determin differ
				direct mean degre studi
	21	.15	.15/.17/.15	express cell signal gene develop activ protein factor regul differenti
				growth transcript suggest role neural
	22	.20	.26/.19/.15	heart cardiac patient ventricular left p cardiomyopathi myocardi function
				normal group signific failur lv dysfunct
	23	.20	.24/.18/.17	cell express human receptor macrophag activ level studi tissu antibodi
				respons increas cultur apoptosi antigen
	24	.23	.34/.19/.15	aortic patient arteri aorta arch thorac graft oper repair left descend valv
				use aneurysm replac
	25	.19	.20/.21/.16	uterin leiomyoma women patient pregnanc treatment hysterectomi fi-
				broid result myoma embol conclus month endometri studi

K

  LDAsegLDA np copLDA np segLDA sent copLDA sent segLDA bi copLDA bi segLDA tri copLDA tri 20NG 55.34 ±1.12 54.90 ±0.[START_REF] Giachanou | Like it or not: A survey of twitter sentiment analysis methods[END_REF] 57.30 ±0.81 48.95 ±1.05 56.80 ±1.09 54.25 ±0.78 56.69 ±0.67 55.19 ±0.95 56.56 ±0.47 59.83 ±0.81 59.37 ±0.97 62.36 ±0.92 55.27 ±1.31 61.66 ±0.97 64.09 ±1.00 63.71 ±0.65 63.76 ±0.96 63.79 ±0.77 64.05 ±1.07 62.98 ±1.26 63.42 ±1.06 56.18 ±1.00 63.48 ±1.29 63.08 ±1.20 63.65 ±1.17 64.12 ±1.36 64.25 ±0.87 65.12 ±1.24 61.42 ±0.84 63.86 ±1.01 59.71 ±1.14 64.02 ±1.01 63.82 ±0.91 65.20 ±1.08 64.78 ±1.01 64.88 ±0.98 64.62 ±1.15 62.29 ±0.95 65.11 ±0.86 56.89 ±1.02 63.74 ±0.71 64.87 ±0.82 64.18 ±1.17 64.74 ±1.14 64.23 ±1.01 Wiki15 72.99 ±2.34 68.44 ±4.16 73.66 ±2.39 64.66 ±2.90 69.78 ±3.89 69.31 ±2.54 72.84 ±2.48 68.45 ±1.60 72.53 ±3.20 76.61 ±3.43 69.64 ±2.70 77.04 ±3.50 63.73 ±3.57 73.49 ±2.21 74.14 ±4.30 74.23 ±2.72 73.92 ±4.29 74.23 ±2.42 76.01 ±2.13 72.27 ±3.39 74.42 ±3.25 65.45 ±3.78 73.47 ±2.55 74.45 ±2.62 76.36 ±2.73 73.61 ±2.75 75.85 ±2.63 76.01 ±2.47 71.68 ±3.88 73.81 ±3.03 67.37 ±5.28 72.30 ±3.85 75.86 ±2.67 76.25 ±2.50 76.13 ±3.81 73.04 ±4.12 73.96 ±2.81 71.56 ±3.75 72.60 ±2.66 66.11 ±2.48 72.82 ±1.62 74.90 ±1.38 75.75 ±2.43 75.22 ±2.44 75.66 ±3.08 Wiki37 59.65 ±2.86 55.83 ±3.32 59.98 ±1.87 55.21 ±2.07 59.41 ±1.93 56.34 ±2.18 58.51 ±1.51 57.72 ±3.25 57.78 ±1.68 60.47 ±3.02 59.18 ±1.70 63.95 ±2.11 55.69 ±1.54 63.54 ±1.98 62.56 ±1.84 63.34 ±2.48 62.31 ±1.92 63.35 ±2.29 63.27 ±2.22 60.37 ±2.13 63.74 ±2.19 56.44 ±3.06 63.84 ±1.78 62.88 ±1.87 62.34 ±2.59 64.34 ±2.06 63.57 ±2.65 63.94 ±2.64 60.76 ±3.01 62.52 ±1.38 56.05 ±2.49 63.51 ±1.58 61.97 ±1.76 64.50 ±2.18 63.32 ±2.22 64.22 ±2.15 63.75 ±2.39 61.68 ±1.62 62.18 ±2.93 56.93 ±2.70 62.04 ±2.07 64.31 ±2.23 64.45 ±2.30 64.38 ±2.58 64.02 ±2.96 Wiki46 54.77 ±1.75 49.75 ±0.86 54.59 ±2.31 47.52 ±1.87 52.52 ±1.44 53.01 ±2.34 53.77 ±1.82 55.11 ±1.41 56.14 ±1.69 60.78 ±1.90 54.96 ±1.33 61.32 ±1.86 50.51 ±2.06 58.64 ±1.58 58.69 ±1.98 60.50 ±1.12 60.16 ±1.70 61.68 ±1.64 62.35 ±1.61 60.25 ±1.58 62.89 ±1.25 52.19 ±1.46 62.71 ±1.30 62.95 ±2.43 63.69 ±1.90 63.05 ±2.11 63.26 ±1.99 64.81 ±1.29 60.49 ±1.63 65.22 ±1.50 53.98 ±2.22 62.03 ±1.91 64.18 ±1.67 62.77 ±1.54 63.97 ±1.50 62.98 ±1.79 65.36 ±1.43 63.68 ±2.17 65.82 ±1.11 55.03 ±1.43 61.38 ±1.39 67.30 ±1.67 65.41 ±1.91 64.89 ±0.93 67.97 ±1.68 PubMed 43.18 ±1.15 42.80 ±1.66 44.74 ±1.17 41.00 ±2.12 49.47 ±1.84 46.13 ±1.69 49.62 ±1.41 49.55 ±0.99 51.62 ±1.24 57.59 ±1.97 53.80 ±1.57 55.26 ±1.96 53.86 ±2.36 57.37 ±1.60 54.91 ±1.95 56.95 ±1.40 57.58 ±1.80 58.15 ±1.45 64.39 ±1.35 61.26 ±2.13 63.53 ±1.62 52.64 ±2.02 63.55 ±2.00 64.42 ±1.33 61.20 ±2.00 64.53 ±1.77 62.28 ±2.18 65.50 ±0.88 62.81 ±1.99 65.75 ±2.14 59.30 ±1.96 66.02 ±1.94 65.50 ±1.34 64.15 ±2.12 66.39 ±1.55 65.22 ±1.69 68.22 ±2.36 65.37 ±1.75 68.30 ±1.48 56.58 ±2.28 67.95 ±2.16 67.81 ±1.23 68.32 ±1.63 68.94 ±1.59 67.97 ±1.92 Reuters 62.92 ±3.77 62.15 ±4.06 62.38 ±4.20 61.19 ±4.17 62.30 ±3.88 61.49 ±4.27 63.34 ±4.60 62.93 ±3.97 65.04 ±4.01 66.34 ±4.21 65.09 ±4.19 65.42 ±4.23 65.33 ±4.30 65.04 ±4.29 66.92 ±3.71 66.98 ±3.39 67.76 ±3.73 67.23 ±3.65 68.56 ±3.81 66.25 ±3.88 67.85 ±3.30 66.78 ±3.96 66.39 ±3.44 67.88 ±3.84 68.67 ±3.72 69.23 ±3.69 68.47 ±3.79 70.77 ±3.47 68.46 ±3.53 69.97 ±3.54 66.19 ±4.17 67.52 ±4.11 69.47 ±3.33 70.25 ±3.82 70.15 ±3.25 70.89 ±3.74 70.95 ±3.17 70.72 ±3.23 71.34 ±3.52 69.00 ±3.67 69.66 ±3.32 71.38 ±3.34 70.41 ±3.44 72.60 ±3.07 71.71 ±3.57 Ted 11.74 ±3.12 10.83 ±2.98 12.02 ±2.30 10.09 ±2.50 12.48 ±3.39 10.73 ±2.72 12.84 ±2.32 11.10 ±3.22 12.11 ±2.59 13.58 ±1.35 12.20 ±3.62 15.14 ±3.41 11.93 ±2.72 13.12 ±2.84 12.48 ±2.18 12.84 ±2.13 13.12 ±2.69 12.20 ±2.01 13.58 ±1.52 11.47 ±3.26 13.21 ±2.43 12.57 ±2.96 14.59 ±3.03 13.39 ±4.21 13.39 ±2.91 12.75 ±3.44 13.67 ±3.27 12.66 ±2.65 12.94 ±2.77 12.20 ±2.43 10.64 ±3.51 13.67 ±2.77 13.67 ±2.61 12.66 ±2.12 12.39 ±2.60 12.48 ±2.06 12.48 ±2.60 12.57 ±2.36 12.20 ±3.21 10.55 ±2.10 13.21 ±2.60 12.57 ±1.79 13.21 ±2.82 13.03 ±2.75 13.49 ±3.02

Table 4 .

 4 7: MiF scores for the classification task. The best scores are shown in bold font.

Table 5 .

 5 1: The notation used for the development of the topic models. Adding exponents 1 , 2 to the symbols of the lower part of the table (below the dashed line) stands for counts specific to d 1

i , d 2 i .

  D] do Calculate λ i using the cross-lingual embeddings end Segment the documents of each language //Initialize counters Ψ , Ω for language ∈ [ 1 , 2 ] do for document d i , i ∈ [1, D] do for segment s i,j : j ∈ {1, . . . , S i } do Decrease counter variables Ψ , Ω according to the previous topic assignments of the words of s i,j

Table 5 . 4 :

 54 Statistics for the Europarl corpus. We used the Europarl data to train the BilBowa word representations.

	number of topics.

Table 5 . 5

 55 Table with the averages across languages, increasing the topics increases the best performance from .135 to .151. From the table, it is evident that adapting the topic models for BiLDA λ-BiLDA segBiLDA s λ-segBiLDA s segBiLDA b λ-segBiLDA b

	En 10 .105 ±.07 .102 ±.07	.090 ±.06	.080 ±.07	.113 ±.05	.124 ±.06
	En 25 .124 ±.10 .125 ±.04	.129 ±.11	.111 ±.07	.140 ±.08	.150 ±.07
	En 50 .132 ±.05 .129 ±.10	.125 ±.06	.125 ±.10	.157 ±.08	.155 ±.05
	Fr	10 .114 ±.05 .114 ±.06	.105 ±.06	.053 ±.05	.088 ±.06	.125 ±.07
	Fr	25 .122 ±.06 .121 ±.06	.124 ±.08	.068 ±.03	.120 ±.07	.114 ±.05
	Fr	50 .124 ±.06 .120 ±.07	.136 ±.07	.080 ±.06	.123 ±.07	.133 ±.06
	Ge 10 .198 ±.09 .198 ±.11	.234 ±.10	.250 ±.08	.203 ±.10	.215 ±.10
	Ge 25 .174 ±.02 .173 ±.11	.235 ±.08	.235 ±.11	.187 ±.08	.176 ±.04
	Ge 50 .183 ±.03 .180 ±.02	.230 ±.09	.255 ±.10	.181 ±.05	.183 ±.04
	It	10 .096 ±.06 .101 ±.05	.102 ±.07	.084 ±.06	.119 ±.06	.113 ±.05
	It	25 .109 ±.04 .118 ±.05	.104 ±.09	.099 ±.02	.143 ±.07	.127 ±.06
	It	50 .119 ±.08 .125 ±.05	.122 ±.09	.131 ±.07	.137 ±.04	.142 ±.08
	Pt	10 .099 ±.05 .115 ±.07	.108 ±.13	.093 ±.09	.098 ±.04	.123 ±.06
	Pt	25 .129 ±.12 .120 ±.11	.164 ±.18	.145 ±.11	.131 ±.06	.124 ±.09
	Pt	50 .120 ±.10 .137 ±.07	.143 ±.15	.125 ±.06	.141 ±.08	.145 ±.10
	avg 10 .117 ±.07 .118 ±.08	.129 ±.10	.110 ±.08	.122 ±.06	.135 ±.07
	avg 25 .127 ±.07 .129 ±.07	.141 ±.11	.127 ±.08	.143 ±.07	.141 ±.06
	avg 50 .131 ±.06 .137 ±.07	.145 ±.09	.137 ±.08	.149 ±.06	.151 ±.06

comparable corpora improves the scores, apart from the case of segBiLDA s . For the rest of the models (BiLDA and segBiLDA b ) the λ-counterparts perform better 2 K : Topic coherence measured by the nPMI for each of the models. The averages are calculated for each model and K across languages. Overall, λ-segBiLDA b performs the best. according to the columnwise comparison of the averaged results in the lowest rows of the table.

Table 5 .

 5 6: The perplexity scores achieved by the proposed topic models for four bilingual datasets when K ∈ {25, 50, 100, 150}. The best (lowest) score achieved per language and k is shown in bold. λ-segBiLDA b achieves the best perplexity scores in most of the experiments.

Table 6 .

 6 1: Statistics after pre-processing the datasets. The distributed representations dataset refers to the data used to train G. The classification data refer to the supervised dataset used for classification purposes. aim at evaluating the value of the fusion mechanism (PE) that we propose. k-NN and SVMs were adapted to the multi-label setting (denoted respectively by SVM PE

Table 6 .

 6 DR SVM DR k-NN PE SVM PE k-NN DR SVM DR k-NN PE SVM PE 2: F 1 measures of difference algorithms. The performance of 5-fold crossvalidated SVM using the bag-of-words representation is 36.03 Also the last line of the table indicates the F 1 score of SVM with tf-idf representation (SVM BoW ). The best obtained result is shown in bold.

	106

  Comparison of the performance of the learning algorithms learned on different representations with respect to the available labeled data. The dimension of the PE representations is 300. are supposed to capture the semantics in the low dimensional space. At the same time, the neighbors algorithm compares exactly this semantic distance between data instances, whereas SVMs tries to draw separating hyperplanes among them.We now examine the performance of the PE representations taking into account the amount of labeled training data. Figure6.2 illustrates the performance of the SVM BoW and SVM PE and k-NN PE with PE representations when the fraction of the available training data varies from 10% of the intial training set to 90% and in the case where, cbow and skip-gram are used as distributed representations with an input size of 300. Note that if only a few training documents are available, the learning approach is strongly benefited by the rich PE representations, that outperforms the traditional SVM BoW setting consistently. For instance, in the experiments with 300 dimensional PE representations with cbow DRs, when only 20% of the data are labeled, the SVM BoW needs 20% more data to achieve similar performance, a pattern that is observed in most of the runs in the figure. When, however,

		0.6		cbow			0.6		skip-gram
	1 measure	0.5				1 measure	0.5
	F					F
		0.4			SVM PE k-NN PE	0.4			SVM PE k-NN PE
					SVM BoW				SVM BoW
		0.1	0.3	0.5	0.7	0.9	0.1	0.3	0.5	0.7	0.9
			Proportion of the training set			Proportion of the training set
	Figure 6.2: Finally, it is known that SVMs benefit from high-dimensional vectors such as bag-
	of-words representations. Notably, in our experiments increasing the dimension
	of the representations consistently benefits SVMs.

Table 6 .

 6 Table, the finegrained dataset is highly unbalanced and skewed towards the positive sentiment: only 13.6% of the training examples are labeled with one of the negative classes. 4: Cardinality and class distributions of the datasets.

			|D|	VeryNeg.	Neg.	Neutr.	Pos.	VeryPos.
	Ternary	Train Test	5,500 20,632	--	785 3,231 10,342 1,887	2,828 7059	--
		Train	7,292	111	884	2,019	3,726	432
	Fine-Grained	Dev.	1,778	29	204	533	887	125
		Test	20,632	138	2,201 10,081 7,830	382

Table 6 .

 6 5: The scores on MAE M for the systems. The best (lowest) score is shown in bold and is achieved in the multitask setting with the biLSTM architecture of Figure 6.3.

		bow	nbow	bow+	nbow+
	SVM ovr	0.993	0.840	0.786	0.714
	SVM cs	0.941	0.946	0.746	0.723
	LR ovr	0.965	0.836	0.731	0.712
	MaxEnt	0.946	0.842	0.701	0.715
	Balikas and Amini [10]	-	-	-	0.719
	biLSTM (single task)	0.827±0.017	0.694±0.04
	biLSTM+Multitask	0.786±0.025	0.685±0.024

Table 6 .

 6 6: The scores of F 1 measure for the systems for the ternary classification task. The hyperparameter tuning was performed for the fine-grained task.

		bow	nbow	bow+	nbow+
	SVM ovr	0.506	0.572	0.584	0.600
	SVM cs	0.499	0.546	0.576	0.590
	LR ovr	0.520	0.572	0.601	0.600
	MaxEnt	0.510	0.547	0.596	0.593
	Deriu et al. [46]	-	-	-	0.633
	biLSTM (single task)	0.580±0.02	0.613±0.04
	biLSTM+Multitask	0.582±0.03	0.617±0.05

Table 6 .

 6 7: Statistics for the data used. W denotes the size of the corpora measured in words.

Table A .

 A .1, and ii) the test set which may be available during training in a transductive alike scenario. The corresponding methods are called Bound UN and Bound Test respectively. 1: The properties of the datasets we used. The dataset name denotes the collection we sampled it from; its subscript denotes the number of categories.Evaluation of the quantification methods. We first discuss the performance of the quantification methods presented above (CC and PCC), prior to comparing the results obtained by the different model selection methods (k-fold cross-validation, hold-out estimation, Bound UN and Bound Test ). Recall that Theorem A.2.1 is based on the assumption that the quantity Max = max y∈Y |p

	Dataset	#Training inst. #Quantification inst. #Test inst. #Features # Parameters
	dmoz 250	1,542	2,401	1,023		55,610	13,902,500
	dmoz 500	2,137	3,042	1,356		77,274	38,637,000
	dmoz 1000	6,806	10,785	4,510	138,879 138,879,000
	dmoz 1500	9,039	14,002	5,958	170,828 256,242,000
	dmoz 2500	12,832	19,188	8,342	212,073 530,182,500
	wiki 250	1,917	3,095	1,003		26,699	6,674,750
	wiki 500	4,912	8,190	2,391		46,556	23,278,000
	wiki 1000	7,887	12,790	4,067		60,788	60,788,000
	wiki 1500	12,156	19,776	6,160		79,973 110,959,500
	wiki 2500	22,642	37,398	11,171	109,694 274,235,000
					C(S) y	-	M	C(S) y

Table A .

 A dmz 250 dmz 500 dmz 1000 dmz 1500 dmz 2500 wiki 250 wiki 500 wiki 1000 wiki 1500 wiki 2500 2: Evaluation of the assumption of Theorem A.2.1 concerning the quantification step. For each dataset, we present Max for the PCC quantification method.

	SVM	.0728	.0967	.1067	.1125	.0345	.0287	.0754	.0310	.0425	.0365
	LR	.0942	.0674	.0889	.1111	.0530	.0219	.0517	.0481	.0310	.0294

  .5889 .7747 .5927 .7663 ±.0158 .5746 ±.0183 .7747 .5927 wiki 500 .7445 .5257 .7449 .5254 .7440 ±.0006 .5228 ±.0031 .7445 .5254 wiki 1000 .7000 .4737 .6993 .4732 .6996 ±.0009 .4584 ±.0274 .7000 .4737 wiki 1500 .6360 .4278 .6354 .4283 .6343 ±.0049 .4230 ±.0126 .6360 .4278 wiki 2500 .5808 .3763 .5811 .3762 .5822 ±.0023 .3759 ±.0004 .5832 .3763 dmoz 250 .8260 .6242 .8270 .6243 .8260 ±.0000 .6242 ±.0000 .8260 .6242 dmoz 500 .7227 .5584 .7227 .5584 .7221 ±.0005 .5558 ±.0022 .7220 .5562 dmoz 1000 .7302 .4883 .7302 .4892 .7301 ±.0001 .4835 ±.0155 .7299 .4883 dmoz 1500 .7132 .4715 .7132 .4715 .6958 ±.0457 .4065 ±.0998 .7132 .4715 dmoz 2500 .6352 .4301 .6350 .4306 .6350 ±.0001 .3949 ±.0686 .6352 .4301 Logistic Regression wiki 250 .7527 .5423 .7527 .5423 .7464 ±.0078 .5335 ±.0134 .7527 .5423 wiki 500 .7302 .4709 .7302 .4709 .7266 ±.0056 .4633 ±.0116 .7302 .4709 wiki 1000 .6836 .4354 .6836 .4354 .6836 ±.0000 .4354 ±.0000 .6836 .4354 wiki 1500 .6166 .3801 .6166 .3801 .6166 ±.0000 .3801 ±.0000 .6166 .3801 wiki 2500 .5802 .3506 .5802 .3506 .5802 ±.0000 .3506 ±.0000 .5802 .3506 dmoz 250 .7742 .4724 .7742 .4724 .7718 ±.0047 .4692 ±.0096 .7742 .4724 dmoz 500 .6608 .4513 .6608 .4513 .6586 ±.0064 .4488 ±.0076 .6608 .4513 dmoz 1000 .6845 .3681 .6845 .3681 .6845 ±.0000 .3681 ±.0000 .6845 .3681 dmoz 1500 .6678 .3616 .6678 .3616 .6678 ±.0000 .3616 ±.0000 .6678 .3616 dmoz 2500 .5959 .3351 .5959 .3351 .5959 ±.0000 .3351 ±.0000 .5959 .3351 Table A.3: The performance of the model selection methods for SVM and Logistic Regression on the test set. For held out, we report the mean and in parenthesis the standard deviation of 10 rounds of the method.

		Bound Un	Bound Test	Hold-out		5-CV
	Dataset	Acc	MaF	Acc	MaF	Acc	MaF	Acc	MaF
	wiki 250	.7747
	SVM

http://www.twitter.com

In the rest of the dissertation we omit the steps when sizes of documents or other sub-document text spans are sampled, provided they are observed during inference.

Also commonly referred to as multilingual LDA depending on the number of the input languages

https://code.google.com/archive/p/word2vec/

http://nlp.stanford.edu/software/lex-parser.shtml

Hereafter, we consider α, β to be symmetric, that is α 1 = . . . = α K , β 1 = . . . = β V , and we denote by the scalars α, β the values of each dimension of the vector.

We dub with typewriter font the implementations of the models we use from the experiments we performed.

https://pypi.python.org/pypi/rpy2

http://workshop2013.iwslt.org/59.php

Technology, Culture, Science, Global Issues, Design, Business, Entertainment, Arts, Politics, Education, Art, Creativity, Health, Biology and Music.

https://github.com/balikasg/topicModelling/tree/master/data

We used the books: Emma, Persuasion, Sense. We considered each paragraph as a document.

This is a real example from the Wiki En-Pt collection used in our experiments.

Although by definition λ i ∈ [-1, 1] in all our experiments we found λ i >0.

urlhttp://linguatools.org/tools/corpora/wikipedia-comparable-corpora/

https://www.crummy.com/software/BeautifulSoup/bs4/doc/

We use 1K bigrams following the work of[START_REF] Jey | On collocations and topic models[END_REF] who found this number to be the optimal choice for similar corpora.

Topic 3 [City] Topic 5 [Sports] Topic 8 [Art] Topic 3 [City] Topic 5 [Sports] Topic 8 [Art] En Fr En Fr En Fr En Fr En Fr En Fr En

https://github.com/gouwsmeister/bilbowa

-size 100 -window 5 -sample 1e-4 -negative 5 -binary 0 -adagrad 1 -xling-lambda 1 -threads 12 -epochs 10

Measured on an Intel Xeon CPU E5-2643 v3 @ 3.40GHz segBiLDA iterations are ∼20% faster. Since, this may vary across datasets depending on the number of sentences/document and their length (Eq. (5.2)), we opt for only reporting the 4x speedup due to less sampling iterations without taking the faster iterations into account.

For cases where there is a single golden documents for each query, MRR is equivalent to Mean Average Precision.

http://statmt.org/

https://code.google.com/p/word2vec/

An open implementation of the system for research purposes is available at https://github. com/balikasg/sigir2017.

http://alt.qcri.org/semeval2016/task4/

The datasets are those of Subtasks A and C, available at http://alt.qcri.org/semeval2016/ task4/.

urlhttp://nlp.stanford.edu/data/glove.twitter.27B.zip

http://www.cs.cmu.edu/~ark/TweetNLP/

For a collection of sentiment lexicons the interested reader can refer to http://saifmohammad. com/WebPages/lexicons.html.

Overall, we cross-validated 512 combinations of parameters. The best parameters were: 0.2 for all dropout rates, 20 neurons for H M and p M = 0.5.

https://github.com/gouwsmeister/bilbowa

-window 5 -sample 1e-4 -negative 5 -binary 0 -adagrad 1 -xling-lambda 1

http://linguatools.org/tools/corpora/wikipedia-comparable-corpora/

https://github.com/commonsense/conceptnet-numberbatch

We use the open implementation of https://github.com/juditacs/wikt2dict.

Chronologically, this was the first contribution in the framework of the author's thesis.

They do not provide a sufficient condition since it is possible, in an adversarial setup, to achieve an upper bound of 1 on the accuracy by simply assigning instances to categories in the same proportion as in the training set.

Acknowledgements

, that I would like to deeply thank.

First, I would like to express my sincere gratitude to my advisor Prof.

Massih-

List of Publications

The following publications are included in parts or in an extended version in this thesis:

Word Embeddings and Text Mining Applications

Twitter Sentiment Quantification [START_REF] Balikas | Twise at semeval-2016 task 4: Twitter sentiment classification[END_REF] Quantification Bounds for Model Selection [START_REF] Balikas | Efficient model selection for regularized classification by exploiting unlabeled data[END_REF] Quantification-based algorithmic tools En→Fr Fr→En En→Ge Ge→En Fr→Ge Ge→Fr MRR P@1 MRR P@1 MRR P@1 MRR P@1 MRR P@1 MRR P@1 Table 6.8: The scores achieved by the systems. WMD with ConceptNet Numberbatch embeddings and the idf term weighting scheme outperforms the rest by an important gap. 1 → 2 (e.g. En→Fr ) defines a CLDD problem where the query documents are written in 1 (e.g. En) and the retrieved in 2 (e.g. Fr).

Chapter 7

Concluding Remarks T ext data are ubiquitous, posing a variety of interesting challenges based on a wealth of possible tasks. The goal of this dissertation was to propose, develop and implement models for text mining applications for text data written in a single or in multiple languages. In particular, our contributions target at answering two main questions: i.) How text structure can improve the performance of unsupervised models for uncovering the latent topics of a documents collections?

ii.) How one can take advantage of polylingual content and rich text representation for improving the performance of various tasks? Both points constitute important questions for the field of text mining and natural language processing in general. In the following sections we provide an overview of the main contributions of this thesis and discuss possible future research directions.

Summary of Contributions

The main contributions of the thesis can be summarized as follows.

Text Structure and Topic Models In Chapter 4 we proposed two novel topic models whose goal was to extend LDA by integrating prior knowledge of text structure. We defined the concept of topically coherent segments and we argued that different text spans like frequent n-grams or noun-phrases can be considered coherent. To incorporate this type of knowledge to LDA we proposed two different sampling strategies, one that assigns the same topic to the words of a segments and Appendix A Efficient Model Selection for Regularized Classification by Exploiting Unlabeled Data I n the main part of the manuscript we proposed extensions of topics models and we explored applications of word embeddings for various challenging tasks. In this appendix we present a contribution that concerns model selection. 1 Differently from the presented models, that learn word representations by modeling how words co-occur within documents, we move to a higher level and explore how categories occur in a collection of documents. By observing the distribution output by a classifier such as Logistic Regression, we propose algorithms for efficient model selection.

Model selection is an essential step in the pipeline of data analysis tasks. Having decided on the algorithm to be used, one should proceed to parameter selection that is the process of selecting a value for the model's hyper-parameter(s) expected to obtain the optimal performance on unseen examples. For instance, when using Support Vector Machines (SVM) or Logistic Regression (LR) in a classification task, one has to tune the regularization parameter λ which controls the complexity of the model.

The fundamental idea of parameter estimation methods is to validate the model's performance in fractions of the training data. In several learning scenarios however, except few labeled data, a larger set of unlabeled data may be available (for example in text classification) as the cost of assigning labels is high. This is the case for example of the transductive learning framework [START_REF] Chapelle | Semi-Supervised Learning[END_REF], where the data to be classified are available beforehand and can be leveraged during the training or inference procedures.

The situation we are investigating in this paper is when unlabeled data are available during the step of parameter selection in a classification problem. The challenge is to come up with a method that is able to leverage the information in the unlabeled data, instead of ignoring them as traditional model selection strategies such as k-fold cross validation (k-CV) do. To tackle this problem, we incorporate quantification techniques in order to infer the distribution of the examples on unlabeled data, which in turn is used to calculate upper bounds (Section 3) on the performance of a model that motivate an efficient model selection scheme (Section 4).

We place ourselves in the supervised learning paradigm where the i.i.d. assumption holds. Note that unlike semi-supervised and transductive learning that make use of the unlabeled data in the training process to improve the performance, we use the unlabeled data for hyper-parameter selection and, hence, the obtained performance in the test set depends on the amount of the available labeled data.

Our method, which is an alternative to k-CV, motivates the selection of the optimal value for the model's hyper-parameter(s) from a finite set that in turn results in the optimal performance (again from a finite set of possible performances). In this work, we propose a hyper-parameter selection method that (i) benefits from unlabeled data, (ii) performs on par with k-CV but it is k times faster and (iii) has the same complexity as hold-out estimation but performs better due to the use of unlabeled data. We demonstrate the efficiency and the effectiveness of the proposed method in Section 5 where we present multi-class text classification results on several datasets with a large number of classes.

A.1 Related Work

Several approaches have been proposed for selecting the hyper-parameters of learning algorithms. The goal is always to select the hypothesis that minimizes the generalization error, which is approximated by the estimated error [START_REF] Mohri | Foundations of Machine Learning[END_REF]. A popular method to calculate the estimated error is the hold-out procedure that splits the data in a training and a validation set; the estimated error is calculated on the latter.

The k-CV technique repeats k times the hold-out procedure: in each round the available training data are partitioned into two complementary subsets, one for training and one for validation. To reduce variability, multiple rounds of crossvalidation are performed using different partitions and the validation results are averaged over the rounds. At the end, an hypothesis is selected e.g. by retraining the classifier on all data using the best values found for the hyper-parameters, or by averaging the hypotheses [START_REF] Bengio | No unbiased estimator of the variance of k-fold cross-validation[END_REF]. A variant of this method is proposed by Blum et al. [START_REF] Blum | Beating the hold-out: Bounds for k-fold and progressive cross-validation[END_REF] with a progressive cross-validation procedure that begins by splitting the data in training and test. At each step, it tests an example which in the next round is used for training, resulting in as many hypotheses as the available test examples. To label an example, a hypothesis is randomly selected. This method has the advantage of using more examples for training than the hold-out and was

shown to select a better hypothesis. In addition, the study in [START_REF] Kohavi | A study of cross-validation and bootstrap for accuracy estimation and model selection[END_REF] reviews accuracy estimation and model selection methods based on cross-validation and bootstrap.

The former is shown to be better than the latter in different datasets, especially in terms of accuracy estimation (for which a stratified approach may be preferred).

The hold-out estimation and the k-CV when k is small are known to have large variance, a problem that can be partially compensated in k-CV by selecting high values for k (like 5 or 10) [START_REF] Arlot | A survey of cross-validation procedures for model selection[END_REF][START_REF] Arlot | Why V=5 is enough in V-fold cross-validation[END_REF]. However, k-CV and its variants are computationally expensive and may be intractable in practice if one wants to search for the appropriate values in large-scale scenarios.

We propose here a different method that can select an appropriate model on unlabeled datasets. The advantages compared to the above-mentioned methods concern its efficiency and its ability to be applied when few labeled examples are available. It dispenses with the use of validation sets which can be cumbersome to produce in unbalanced or small datasets. It is, however, intended for model selection only, whereas cross-validation and hold-out estimation can be used for performance evaluation as well.

A.2 Accuracy and Macro-F1 Quantification Bounds

In this section, we propose an upper bound on several performance measures (accuracy and macro-F1) of a given classifier C on a dataset S which doesn't need to be labeled. We then use this bound, which is based on the class distribution induced by C on S, to perform model selection.

We considered mono-label multi-class classification problems, where observations x lie in an input space X ⊆ R d . Each observation x is associated with a label y ∈ Y , where |Y | > 2. We suppose that examples consist of pairs of (x, y) identically and independently distributed (i.i.d) according to a fixed, but unknown probability distribution D over X × Y (D X will denote the marginal probability for x in X ).

In the context of text classification, x (i) ∈ X denotes the vector representation of document i and its label y (i) ∈ Y represents the category associated with x (i) . We further assume to have access to a training set S train = {(x (i) , y (i) )} N i=1 also generated i.i.d with respect to D.

Quantification. As explained below, our analysis makes use of M C(S) y

, the number of documents in the unlabeled set S assigned by classifier C to class y. Many classifiers do not directly assign a category to documents, but rather produce scores (probabilistic or not) for each category, from which a categorization decision can be made. The task of determining the number of instances of each target category in a set S is called quantification and was first proposed by Forman et al. [START_REF] Forman | Counting positives accurately despite inaccurate classification[END_REF][START_REF] Forman | Quantifying counts and costs via classification[END_REF].

Contrary to classification that identifies in which target categories an observation belongs, quantification is solely concerned with the estimation of the number of observations belonging to a target category (the positive examples). Note that a good quantifier is not necessarily a good classifier, and vice versa. For example, in a binary problem with 40 observations, a learner that outputs 20 False Positives and 20 False Negatives is a perfect quantifier but a really bad classifier.

Given a set of instances in S, quantifiers output, for each target category y of S, a number denoting the prediction of the relative frequency of category y in S. Quantification methods using general purpose learners are usually split ( [START_REF] Esuli | Optimizing text quantifiers for multivariate loss functions[END_REF]) in aggregative and non aggregative methods based on whether the quantification step requires the classification of the individual instances as a basic step or not. Quantification has been mainly used to estimate distribution drifts. We make a different |S| x∈S p(y|x), where p(y|x) is the posterior probability that an instance x of S belongs to y. We do not consider the adjusted version of those two approaches proposed in [START_REF] Bella | Quantification via probability estimators[END_REF] because they require the expensive k-fold cross-validation in the training set which is undesirable in large scale settings. Lastly, having a trained classifier, the computational complexity of quantification reduces to the prediction cost of a trained classifier.

Quantification-based Bounds. We now present our main result which consists of quantification-based upper bounds on the accuracy (denoted A C(S) ), the macroprecision (denoted MaP C(S) ), the macro-recall (denoted MaR C(S) ) and the macro-F1 (denoted MaF C(S) ) of a classifier C on a dataset S which does not need to be labeled.

Theorem A.2.1. Let S = {(x (j) )} M j=1 be a set generated i.i.d. with respect to D X , p y the true prior probability for category y ∈ Y and N y N py its empirical estimate obtained on S train . We consider here a classifier C trained on S train and we assume that the quantification method used is accurate in the sense that: Then for any δ ∈]0, 1], with probability at least (1δ): Model Selection Using Quantification Bounds. We consider here a standard regularization setting in which one aims at minimizing a combination of the empirical error and the model complexity using the following template of the objective function: ŵ = arg min R emp (w) + λReg(w)

where Reg(w) is the regularization term to avoid overfitting and R emp (.) represents the empirical error.

The parameter λ controls the trade-off between the empirical error and the regularization term. As mentioned before, λ is typically estimated through hold-out estimation or k-fold cross-validation. We propose here to estimate it on the basis of the upper bounds presented in Theorem A.2.1, as described in Algorithm 4.

As one can note, for each value of λ, a classifier is trained and quantified on the unlabeled set S. If the quantification assumption of Theorem A.2.1 is not valid, then one falls back on the Classification and Count method for quantification. The bounds, as computed by Inequalities A.1 and A.3 are used to select the "best" classifier. Tuning the hyper-parameter is, therefore, reduced to the problem of finding a classifier which yields the highest value of the bounds on a given set. In contrast with other selection methods, the set used to select the classifier can be an unlabeled set from the same distribution (unlabeled data is usually readily available, contrary to labeled data) or the test set in a transductive-like scenario.