Zheng, Hicham, Julien, Arnaud, Christopher, Rashedur, Somnath, Sadaf Anna Swen

Yuming, Tsanta Leon Paul

Hanna Leonardo

Srikanth, Shreyas Catherine Eva

Arthur

Charlotte

Pavan Pierre

Michael Patrick

Emmanuel Macron

François Hollande

Q: What currency is used in Ukraine? (Currency) A: Ukrainian hryvnia Q: Who played the role of Heisenberg in the series Breaking Bad? (Person) A: Bryan Cranston Factoid QA examples

Non-Factoid answers are answers which are descriptive in nature, such as definitions, procedures, explanation of a phenomenon etc. Often,

Synthèse en français

La recherche de réponses à des questions relève de deux disciplines : le traitement du langage naturel et la recherche d'information. L'émergence de l'apprentissage profond dans plusieurs domaines de recherche tels que la vision par ordinateur, le traitement du langage naturel, la reconnaissance vocale, etc. a conduit à l'émergence de modèles de bout en bout et les travaux actuels de l'état de l'art en question-réponse (QR) visent à mettre en oeuvre de tels modèles.

Dans le cadre du projet GoASQ1 , l'objectif est d'étudier, comparer et combiner différentes approches pour répondre à des questions formulées en langage naturel sur des données textuelles, en domaine ouvert et en domaine biomédical. Le travail de thèse se concentre principalement sur 1) la construction de modèles permettant de traiter des ensembles de données à petite et à grande échelle, et 2) l'exploitation de connaissances sémantiques pour répondre aux questions par leur intégration dans les différents modèles. Nous visons à fusionner des connaissances issues de textes libres, d'ontologies, de représentations d'entités, etc. Afin de faciliter l'utilisation des modèles neuronaux sur des données de domaine de spécialité, généralement de petite taille, nous nous plaçons dans le cadre de l'adaptation de domaine. Nous avons proposés deux modèles de tâches de QR différents, évalués sur la tâche BIOASQ de réponse à des questions biomédicales, et nous montrons par nos résultats expérimentaux que le modèle de Questions-Réponses ouvert2 convient mieux qu'une modélisation de type Compréhension machine 3 , qui est la plus courament utilisée. Nous pré-entrainons le modèle de Compréhension machine, qui sert de base à notre modèle, sur différents ensembles de données pour montrer la variabilité des performances lorsque ces modèles sont adaptés au domaine biomédical. Nous constatons que l'utilisation d'un ensemble de données particulier (ensemble de données SQUAD v2.0) pour la pré-entraînement donne les meilleurs résultats lors du test et qu'une combinaison de quatre jeux de données donne les meilleurs résultats lors de l'adaptation au domaine biomédical. Nous avons effectué des expériences à l'aide de modèles de langage à grande échelle, comme BERT4 , qui sont adaptés à la tâche de réponse aux questions. Les performances varient en fonction du type des données utilisées pour pré-entrainer BERT. Nous en avons conclu que le modèle de langue appris sur des données biomédicales, BIOBERT, constitue le meilleur choix pour le QR biomédical.

Étant donné que les modèles d'apprentissage profond visent à fonctionner de bout en bout, les informations sémantiques provenant de sources de connaissances construites par des experts n'y sont généralement pas introduites. Nous avons annoté manuellement et automatiquement un jeu de données par les variantes des réponses de BIOASQ et montré l'importance d'apprendre un modèle de QR avec ces variantes. Nous montrons l'utilité d'exploiter le Type de réponse attendu et le Types lexical de la réponse en domaine ouvert et en domaine biomédical par différentes études. Ces types sont ensuite utilisés pour mettre en évidence les entités dans les jeux de données, ce qui montre des améliorations sur l'état de l'art. Par ailleurs l'exploitation de représentations vectorielles d'entités dans les modèles se montre positif pour le domaine ouvert.

Une de nos hypothèses est que les résultats obtenus à partir de modèles d'apprentissage profond peuvent être encore améliorés en utilisant des traits sémantiques et des traits collectifs calculés à partir des différents paragraphes sélectionnés pour répondre à une question. Nous proposons d'utiliser des modèles de classification binaires pour améliorer la prédiction de la réponse parmi les K candidats à l'aide de ces caractéristiques, conduisant à un modèle hybride qui surpasse les résultats de l'état de l'art sur la plupart des ensembles de données.

Enfin, nous avons évalué des modèles de QR ouvert sur des ensembles de données construits pour les tâches de Compréhension machine et Sélection de phrases. Nous montrons la différence de performance lorsque la tâche à résoudre est une tâche de QR ouverte et soulignons le fossé important qu'il reste à franchir dans la construction de modèles de bout en bout pour la tâche complète de réponse aux questions. iv I dedicate this thesis to my parents who believed in me and my choice to pursue Computer Science throughout my life since my early school days. Thank you maa, for supporting me everyday and importantly since I moved out of India to study in France. Without your support I would have never finished what I started. Thank you paa, for buying me that amazing PC in my 3rd grade and getting me an internet connection, without that I would never appreciate the world of computers in my childhood.

I sincerely thank my supervisors, Brigitte Grau and Yue Ma who believed in my candidature when I had applied for the role and for supporting me throughout the duration of my PhD, by having regular meetings. You both are great mentors and I will remember these times forever. I would like to thank my jury members Patrice Bellot and Mohand Boughanem for patiently reading my thesis and giving valuable feedback and comments, Patrick Gallinari and Catherine Berrut for accepting to be my examiners, and Nicolas Sabouret for being the president of my jury and hosting the defense. Special thanks to Anne Vilnat for accepting to be my second supervisor because of official reasons after Brigitte's retirement and helping me finish everything easily.

Special thanks to my dear partner Swathisree who stood with me throughout my journey in France, and my friends Karthik B and Amruth AC who are my partners in crime and will always be my partners in crime. Thanks guys.

Introduction

Question Answering (QA) deals with retrieving relevant answers from documents for a given question. It is a field of research which is an intersection between two major research fields, Natural Language Processing (NLP) and Information Retrieval (IR). Unlike document retrieval performed by search engines, question answering relies on extracting suitable short answers which are more specific than lengthy documents which are topic related. Question Answering is often seen as a supervised learning problem which requires labelled data, although some exceptions exist [START_REF] Lewis | Unsupervised Question Answering by Cloze Translation[END_REF].

Question Answering can be defined in several ways based on the target task. Some of Question-Answer pair examples are presented below.

If the goal of the system is to find a relevant document (with multiple paragraphs) to a query, the task is called as Document Retrieval. If the goal is to find a paragraph or a sentence relevant to a query, the task is called as Answer Selection. If the goal is to find an answer span in the text, the task is called as Answer Extraction. All the examples for each task are shown above for a given question. The target task of the QA system defines the type of output expected from the system.

In the first approaches of QA systems, a question is analysed to determine some elements such as focus (the main entity addressed in the question), question category (factoid, non factoid, summary, yes/no, choice question etc.), and expected answer type (entity type expected from the question). The question terms (words) are used to find documents, paragraphs and sentences which might contain the answer. Answer selection (or extraction or ranking based on how the candidates are chosen) is then performed on the relevant paragraphs or sentences to choose the answer sentence (long) or answer span (short).

The Information Retrieval (IR) community focuses on document retrieval and paragraph retrieval, whose role is to fetch a set of relevant supporting texts which might contain the answer for a question. Other models are used for ranking answers candidates based on term frequencies, entities, etc. The Natural Language Processing (NLP) community works on the QA topics which use different linguistic features such as named entities, part of speech tags, expected answer types, passage and question representations (different syntactic and semantic approaches) etc. These are used as features for either document or paragraph retrieval models or answer selection/extraction models. Both the communities often use machine learning algorithms to learn different classifiers, ranking functions etc. for their individual fields. [START_REF] Jouis | Next Generation Search Engines: Advanced Models for Information Retrieval[END_REF][START_REF] Chu-Carroll | Finding needles in the haystack: Search and candidate generation[END_REF][START_REF] Grappy | Selecting answers to questions from Web documents by a robust validation process[END_REF] Domain specific question answering is a special type of question answering that deals with specialized data belonging to a domain. Open domain datasets are usually curated automatically or by using crowd sourced workers, while domain specific datasets are curated with the help of domain experts. Open domain question answering systems must be explicitly adapted to these specific domain data inorder to perform efficiently.

In biomedical domain, there are a plenty of expert curated information available in the form of ontologies, metathesaurus etc. The datasets for biomedical question answering are not large scale in size. Creating a large scale dataset in biomedical domain requires domain expertise which is expensive and a time consuming process. Therefore it is important to adapt an open domain model to biomedical domain. There are different types of question answering systems based on the type of data they use for processing. Textual data (free text, wikipedia documents, scientific articles, medical text etc.), relational databases, knowledge graphs, ontologies etc. are some of the data sources used for question answering.

A textual question answering system has series of operations either performed sequentially or in parallel. A typical QA pipeline1 seen in the perspective of Information Retrieval discipline is as shown in the Figure 1.1. The sub-modules of the QA pipeline are explained in brief below.

Question Processing

The question processing module processes a query either to analyse query terms (words) for query expansion, answer type detection etc., or to reformulate it as a SPARQL or a relational database query to find answers in structured databases or ontologies.

The answer type detection analyses the question for different expected answer types based on question words such as "Who", "What", "Where", "When", "How" etc. and determine the expected type of answer such as "Person", "Entity", "Location", "Time", "Procedure" respectively. Based on this type, the answer space can be filtered for better candidate answers.

Passage Retrieval

In a textual QA task, the data used is only textual documents (not databases or knowledge graphs) using which the Document Retrieval module applies several techniques to find the relevant documents (more than one paragraphs is termed as a document) for a question, based on the indexed set of documents. These documents are then passed on to a Paragraph Retrieval module which finds the best set of paragraphs out of the text documents to extract the answer.

Answer Processing (Long and Short)

The answer processing module processes the paragraphs or sentences in several ways such as ranking candidates based on features (word overlap, entity match, knowledge graph entity types, syntactic analysis, representation matching etc.) and extract answer spans based on named entity type match. This module uses knowledge from databases, ontology, knowledge graphs as reference to extract or find the answers.

The kind of answers such as sentences for long answers or short spans for short answers determines the processing steps involved. For long answers, a sentence selection task is used to select an answer sentence. For short answers, an extraction task namely reading comprehension or machine reading or extractive question answering task is used to extract short answer spans from sentences or paragraphs.

The three answer processing tasks used in our works are:

• Answer Sentence Selection -choosing one or more answer sentences or short paragraphs among other candidates, as correct answers for a given question.

• Reading Comprehension -extracting short answer spans from a relevant paragraph for a given question.

• Open Question Answering -extracting short answer spans from a set of paragraphs (relevant and irrelevant) for a given question.

Since Question Answering is a complex task, each module is considered to be an individual subtask and the focus is on achieving better scores on the subtask target datasets. An issue with pipeline approaches in general, is Error Propagation. If an intermediate module causes errors, the error gets propagated onto the final output.

The overall error rate would increase. In other words, the error gets propagated across different modules which might reduce the final performance. Evaluating individual module and predicting the overall behaviour in a pipeline approach is cumbersome.

1.2 Redefining the QA Pipeline with Deep

Learning

The use of deep learning models in the field of Computer Vision and Speech Processing started to rise rapidly because of the availability of 1) Large scale datasets 2) Access to use hardwares such as GPUs (Graphical Processing Unit) for machine learning. GPUs were earlier used primarily for computer graphics but recently tweaked for training deep learning models, which gave rise to more research interests in the field of machine learning towards deep learning because of the accelerated training times. This was observed in the field of question answering which changed the way some of the traditional QA models work. The intuition of using these models is to 1) avoid handcrafting features for the models, 2) avoid pipeline approaches but rely only on input data to predict the output by automatically learning these features. An ideal deep learning model would be an end-to-end model which relies only on input data and will not have issues like Error Propagation. The field of Natural Language Processing (NLP) witnessed the usage of deep learning models starting with tasks such as Named Entity Recognition [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF], Dependency Parsing [START_REF] Socher | Parsing with compositional vector grammars[END_REF], Sentiment Analysis [START_REF] Moraes | Document-level sentiment classification: An empirical comparison between SVM and ANN[END_REF] etc. The intuition or the hypothesis behind using deep learning models is that these models are good implicit feature extractors and will not need handcrafted feature engineering which the pipeline modules heavily relied on.

The deep learning models for QA would focus only on the red parts highlighted in the Figure 1.2. An ideal deep learning based QA model would perform answer extraction directly by retrieving the documents, finding the relevant ones, extracting relevant paragraphs and finally extracting the answers out of it. All the modules can be learnt based on the input data and no explicit features would be needed. But building such models has many hurdles to cross.

Hurdles on using deep learning models

Building such a deep learning model (end-to-end model) which requires zero handcrafted features and which learns solely from data has some hurdles.

• Have enough data. (Size)

• Have the right kind of labelled data. (Suitable type)

• Build a single model which does it all. (Architectural Complexity)

• Generalize the model to work on all QA tasks. (Generalization)

Size

For deep learning models to work efficiently, they have to be trained on large scale datasets. Training on small scale datasets will not result in similar performance as on large scale ones. We show this behaviour experimentally in further sections of our work. Because of this behaviour, only tasks which have sufficiently large scale labelled datasets can use deep learning models.

This behaviour causes difficulty in analysing if the model itself is performing poorly or if the dataset is not sufficient for the model to perform better. It causes a dilemma whether or not to design a new model or augment the dataset with more labeled samples and balanced label classes.

To overcome the small size of datasets for biomedical question answering, Transfer Learning was first used by (Wiese et al., 2017a). The process involves training models on large scale datasets and fine-tuning the same models on small scale datasets for the same target tasks. When this process is conducted across two or more different domain datasets, it is called as Domain Adaptation. The target task remains the same in this approach. Another type of Transfer Learning approach is to train models on a task with large scale datasets and modifying final output layer to another target task and fine tuning this model on a small scale dataset of the target task.

Suitable type

The large scale datasets required by the deep neural network models are not just any random datasets. The data should be labelled according to the task for which the model is built. To label such datasets three methods are used in common practice. 1) An automatic annotation method which results in creation of synthetic datasets; 2) Distant supervision methods with some noise; 3) Crowdsourced workers (or sometimes colleagues, students etc.) employed to annotate a dataset which is sufficiently large scale for their model. Although an important highlight is that there is no standard measure to determine a "large scale" dataset.

Architectural complexity

Building an end-to-end model which returns an answer for an input question, is more complex than building submodules, the focus was on using deep learning models on tasks such as Answer Sentence Selection and Answer Extraction using Reading Comprehension by splitting the QA process into two or more sub-tasks. More focus is seen on these topics separately than together into a single big model. The goal of building end-to-end models remained the same but the target tasks became different, that are mainly the sub tasks of the overall QA task.

Even though end-to-end models are the best models one can strive to build for some subtasks, getting certain aspects such as their neural architectures, training approaches, hyperparameters, optimizer functions, random initialization parameters etc. is a big challenge and often sometimes referred as Architecture Engineering 2 . From avoiding Feature Engineering which the traditional machine learning algorithms heavily relied on, towards neural networks which heavily relies on Architecture Engineering, one problem gives rise to another problem. Techniques like neural architecture search [START_REF] Elsken | Neural architecture search: A survey[END_REF] can be used to learn a better architecture than the ones chosen by humans. But they are computationally very expensive to do and bad for environment because the electricity and CO2 emissions they use for this purpose.

Generalization

A common problem working on QA datasets is the usability of these models on other datasets. This problem is not only specific to QA but also other applied machine learning tasks. In other words, can a model trained on one dataset perform equally well on other datasets on the same or similar task? This aspect plays a major role in determining if deep learning models trained on large scale datasets can be used to predict on datasets like the domain specific ones which are usually tedious and expensive to curate with the help of experts.

Hurdles in domain specific question answering

Domain specific question answering has several issues in common with some points discussed above. The data is usually curated by domain experts. Contrary to open domain where there is no need of expert knowledge, domain specific data demands domain expertise.

The complexity of the domain expertise makes it hard to obtain labelled datasets of large scale. One such example is the BIOASQ3 challenge where the biomedical domain experts are asked to annotate certain task datasets for question answering.

Over the period of 7 years, the data annotated by human annotators who are domain experts for factoid questions still remain less than 1000 questions.

Domain specific data has two main limitations to be addressed before using the models which work better on open domain data:

Size

Similar to applying deep learning models on small scale datasets, the same problem applies for domain specific datasets which are small scale. Construction of large scale dataset is expensive and time consuming because of the domain expertise it demands.

Domain Expertise and Vocabulary

Since the data is domain specific, the vocabulary also consists of new words from a specific domain. For example, using a word embedding space which is trained on Wikipedia might result in missing words from the vocabulary of biomedical domain data. Because the vocabulary is different, detecting named entities, noun phrases, part of speech tags and expected answer types cannot be reused from the traditional QA pipeline used in the open domain setting. There is a requirement for special use case handling for domain specific knowledge integration along with the existing QA models.

While using word embeddings in any NLP tasks, a tokenizer is used to split sentences into words and an embedding vector is found corresponding to the word from a pretrained embedding space. In domains such as biomedical domain or medical domain, the numerical values and co-efficients such as α,β,γ are important and using a tokenizer which tokenizes them differently from the tokenizer used in the pretrained word embeddings, results in missing vocabulary.

Because of the above two limitations, research on domain specific data is significantly less emphasized compared to the open domain. This explains the low results on domain specific tasks like the BIOASQ tasks4 . Techniques such as domain adaptation and transfer learning can be used to handle this situation, which are relatively easier to do with deep learning models.

Structured Knowledge for Question Answering

Structured knowledge and semantic knowledge by experts provide supporting information for textual data which are enriched, verified, accessible and stored in such a manner that computer applications can access and use it easily with querying languages. They are annotated by human experts or extracted from free text. Relational databases, knowledge graphs and ontologies are some of the examples for such knowledge.

To overcome the missing vocabulary problem in domain specific QA one way is to use domain specific textual resources for training word embedding spaces and another is to use concepts and relationships from ontologies like SNOMED CT 5 , UMLS metathesaurus6 to detect specific entities and their types in the text of QA datasets.

Research Objectives and Contributions

To address some of the issues discussed in the above sections with respect to deep learning methods, we intend to investigate and compare various Our research questions are the following:

1. How can we build models which work both on small scale and large scale datasets without dropping performance?

2. How can we leverage the structured and semantic knowledge effectively into state of the art question answering models?

In the context of this thesis work -"Question Answering with Hybrid Data and Models", Hybrid data refers to using open domain data with specific domain data in a domain adaptation process, plus integrating structured knowledge for annotating training datasets and for enriching the input data. Hybrid model refers to an addition of a post processing reranker to account for structured knowledge and collective features obtained from different paragraphs.

Deep learning models have been commonly used in the field of Natural Language Processing across various tasks. However, using them on domain specific data which are small scale and are dependent on domain specific vocabulary is not very straightforward. As addressed earlier the performance of these models rely on the Size of the datasets.

In Chapter 4, addressing our first research question, we discuss how one can use these models on small scale labelled datasets such as biomedical domain dataset for question answering and discuss different strategies involved in getting better performance. More precisely we discuss details about Domain adaptation on biomedical QA dataset (BIOASQ) by training two task models 1) Reading Comprehension and 2) OpenQA on open domain QA datasets and fine-tune them on BIOASQ data. We show how a pre-training method using OpenQA method suits better for BIOASQ task instead of the other method.

Using deep learning models has become a new trend in the field of research, which has resulted in a lot of outcomes which are incremental in nature. Although often some engineering tricks can improve a model performance slightly (sometimes significantly), research papers tend to not discuss these in detail but rather focus on the mathematical models. Often the performance reported is generalized as the result of the underlying neural network architectures and training methods, but not the subtle difference in preprocessing training data such as using a different tokenizer, different named entity recognition tools, different part of speech taggers, different embedding spaces, highlighting some entity information, using different combination of datasets etc. We present two research articles [START_REF] Seo | Bidirectional attention flow for machine comprehension[END_REF][START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] focused on introducing two different QA models based on RNNs on a same QA task. The authors report the performance on SQUAD dataset leaderboard (Rajpurkar, J. Zhang, et al., 2016). The simple method (Single attention mechanism, faster training) by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] performs better than the complex method (Multiple attention mechanisms, slower training) by [START_REF] Seo | Bidirectional attention flow for machine comprehension[END_REF], which shows that the results not always increase based on the increasing complexity.

In Chapter 5, addressing our second research question, we hypothesize that existing QA models can perform better if the input data contain more information than just textual phrases. Structured and semantic knowledge which exist already can be useful for question answering.

Our contributions are:

• Using entity information -we explore entity enriched texts for Reading Comprehension task on open domain QA dataset.

• Using answer variants -we manually annotate more answer spans which are correct besides the gold standard ones in biomedical domain and show how models can perform better when correct data is input.

• Using expected answer types to highlight entities -we highlight entities in paragraphs which match the expected answer types from the question using an existing Answer Sentence Selection QA model and compare their performance with baseline model scores and show how using the same model with slightly modified data can improve performance.

• Improving QA performance using semantic features and structured information for ranking models -The semantic information from various paragraphs which are provided with the questions in OpenQA task are not well modelled in neural network models which consider only a pair of paragraph and question inputs and ignore other paragraph information. In our work we model this collective information and semantic information as features for ranking models to improve QA performance.

The organization of the chapters of this thesis is as shown below: We introduced the context of our research work, presented the introduction to question answering, some hurdles with neural network models in QA and presented our research questions in this chapter.

In the following chapters, we detail the tasks of question answering, datasets used for benchmarking QA models and models which are widely used as state-of-the-art techniques. Chapter 3 presents the overview of QA approaches including the stateof-the-art models widely used these days on different QA tasks. Chapter 4 presents our work on building models for small scale and large scale datasets using domain adaptation. Chapter 5 presents our work on leveraging structured and semantic information into QA models to improve performance. The question answering pipeline introduced in the Introduction chapter in Figure 1.1 (is also shown in Figure 2.1 for reference) represents the whole pipeline or a complete QA system architecture. Each of these modules individually sometimes are termed as "Question Answering systems" based on different assumptions on the target subtask. Question Answering in different contexts can mean different tasks.

Publications

The organization of this chapter is as follows: we first discuss and detail about question answering based on a classification of different types of answers. Then are followed by defining question answering based on different types of tasks, specifically about Answer Sentence Selection, Reading Comprehension and OpenQA.

We also highlight and detail some modifications on Reading Comprehension task and term it as RC 2.0.

We present a list of datasets/corpora used for benchmarking the above mentioned QA tasks and explain some characteristics of open domain and closed domain datasets. Finally we present some evaluation metrics with respect to different tasks and conclude by highlighting what we address in our work.

Answer based classification

Question Answering systems can be classified based on the answers they expect by analysing the questions. Based on different types of Answers or Results a system returns to the user, we can briefly classify a QA system into one of the following below:

• Factoid Question Answering

Task based classification

In the context of factoid question answering as explained above, Question Answering field has seen a lot of different tasks commonly referred to as "Question Answering" although they differ in many ways. These systems can be classified according to their tasks.

Evolution of tasks over time

The tasks have evolved over time based on several factors such as available datasets, approaches, evaluation campaigns etc. The below list of tasks are presented in the chronological order.

1. Open domain question answering based on the pipeline approaches. -Legacy OpenQA, initiated by TREC evaluations.

2. Answer sentence selection -to focus on sentence similarity or textual entailment problems.

3. Reading comprehension and modifications -with neural network approaches.

4. Open domain question answering based on deep learning approaches. -OpenQA

Task based classification

For question answering in an open setting, a question is provided and the system is expected to return an answer. No other supporting information is provided along with the questions. The system has to retrieve a document collection and extract a short answer. This is termed as an Open QA task. In the earlier QA systems, pipeline approaches were used for this purpose using information retrieval and NLP techniques, knowledge bases and ontologies. These systems are called as Legacy OpenQA. Recent methods use deep learning methods instead of traditional pipeline methods with a goal of building end-to-end models for the same task, these systems are simply called as OpenQA.

In between the above two tasks, the focus of QA system was towards Answer sentence selection where a sentence among other sentences was to be predicted as a relevant answer or not. Focus further shifted towards answer extraction task from relevant paragraphs which were called as Reading Comprehension. There were also systems for multiple choice answers under Reading Comprehension.

In this section we define the QA tasks which we address in our work.

Answer Sentence Selection

As per the details presented in Figure 2.1, the task of Answer Sentence Selection deals with finding a sentence as the correct answer for a question. So the Answer Processing module determines which paragraph (or sentence) is the correct answer.

Given a question in natural language, the objective of this task is to find relevant sentences among candidate sentences. A sample dataset consists of a Question Q, a set of paragraphs P = {p 1 , p 2 , p 3 , ....p n } and annotated labels either being {0, 1}.

From the definition, the problem can be formulated as a ranking problem, where the goal is to give better rank to the candidate sentences that are relevant to the question. Pointwise and Pairwise approaches are the two most common approaches to learn the ranking functions.

In the pointwise approach, the ranking problem is transformed into a binary classification problem. More specifically, the training instances are triples (Q i , P ij , y ij ), where Q i is a question in the dataset, P ij is a candidate answer sentence for Q i , and y ij is a binary value indicating whether P ij contains the correct answer to Q i .

In the pairwise approach the ranking function is explicitly trained to score correct candidate sentences higher than incorrect sentences. Given a question, the approach takes a pair of candidate answer sentences and learns to predict which sentence is relevant to the question. Concretely, a score is predicted for each pair of question Q i and a candidate sentence P ij . We discuss more about several such approaches and the state of the art models in the Section 3. P 3 : in "The Iron Lady", Young traces the winding staircase of fortune that transformed the younger daughter of a provincial english grocer into the greatest woman political leader since catherine the great.

An example from TREC-QA dataset, where the sentence P 1 is the correct answer.

The example shown above is from a QA task where the answers are sentences. The models built for this task must determine if an answer sentence is correct or wrong. However, the main limitation of this task is the inability to extract short answers from the sentences. For example, the correct answer for the above example is Hugo Young, but the model can only determine if the whole sentence is an answer or contains a short answer in it.

The TrecQA dataset by (M. [START_REF] Wang | What is the Jeopardy model? A quasi-synchronous grammar for QA[END_REF] which was curated from the Trec task1 is used to benchmark this task.

Reading Comprehension or Machine Reading

To focus on the answer extraction task, the Reading Comprehension or Machine Reading task was introduced. By definition, the goal of Reading Comprehension task is to "read" a document, which technically means "analyse" or "understand" a document and answer a set of questions based on the document, similar to a Cloze test 2 where an exercise consisting of a portion of text with certain items such as entities, words, numbers etc. are removed or hidden from the text. The participant is asked to replace the missing item or items by reading the text. "Cloze tests require the ability to understand context and vocabulary in order to identify the correct language or part of speech that belongs in the deleted parts of the paragraphs." -Wikipedia 2 .

Using this intuition, the Reading Comprehension task for answer extraction was developed by [START_REF] Hermann | Teaching machines to read and comprehend[END_REF] where a question and a relevant paragraph is the input to a QA system and the answer should be a span (word, set of words, numbers, phrases etc.) from the relevant paragraph. This way an answer from the predicted paragraph can be extracted.

By definition, Reading Comprehension task needs comprehension skills to understand and answer a question. The task of QA4MRE3 (Question Answering for Machine Reading Evaluation) by [START_REF] Peñas | QA4MRE 2011-2013: Overview of question answering for machine reading evaluation[END_REF] was held in CLEF QA task, which is termed as Machine Reading and had multiple choice answers for questions. The task needed some reasoning and understanding of the paragraph to choose the correct answer among candidate answers and not the extraction of substrings from the paragraphs like in the Reading Comprehension task.

In our work we only use and refer to the Reading Comprehension task which is formally defined below. A sample dataset consists of a Question Q, a relevant paragraph P along with two string offsets -Start and End which represent the start and the end character offsets of the answer in the relevant paragraph text. The task of extracting answer span has been generally approached in an uniform manner since its inception by [START_REF] Hermann | Teaching machines to read and comprehend[END_REF]. If the answer is a single token, then a probability distribution overall the paragraph terms is used to extract the answer token. If the answer is a span (more than 1 tokens), two classifiers are used to detect the answer span.

A question Q has m terms Q = {q 1 , ....., q m } and paragraph S has n terms S = {s 1 , ....., s n }. An answer span (s, e) is a substring in S. (s, e) ∈ S.

We discuss more about several approaches and the state of the art models in the Chapter 3. The example shown above represents a sample data from a Reading Comprehension dataset SQUAD where usually more than one questions exists per paragraph. A lot of buzz around this task was created by the media,4 when a system got higher accuracy scores than the human annotator scores. Since then and before this buzz, a lot of works have been published on the SQUAD dataset and their scores are presented on its leaderboard 5 which shows the increasing interests of the community towards this task.

Following this hype, a lot of limitations of the task and the models were put forward by Yoav Goldberg6 and several others on various social media such as Medium and Twitter.

Following is a list of some of the limitations of Reading Comprehension on SQUAD dataset which gave rise to other methods and datasets later on:

• The answer is guaranteed to be in the paragraph.

• We must find the answer in a given paragraph, not elsewhere.

• Annotators see the paragraph when creating the question, resulting in high lexical similarity between question and answer.

• Adversarial manipulation of the paragraphs (appending an automatically generated noisy sentence to the paragraph) leads to significant drop of performance of many models. [START_REF] Jia | Adversarial examples for evaluating reading comprehension systems[END_REF] • No symbolic reasoning is required. Understanding questions to perform reasoning over paragraphs is not possible in traditional datasets such as SQUAD or CNN/Dailymail datasets [START_REF] Hermann | Teaching machines to read and comprehend[END_REF].

Reading Comprehension 2.0 -Modifications

There are several shortcomings from Reading Comprehension task which were listed in the previous section. To address some of them, several works have been published recently. More details about specific works are presented in the Section 3.

In this section we briefly describe three approaches which are often seen as modifications for the Reading Comprehension task. 1. Unanswerable questions -in other words Paragraphs with no answers. In the setting of open domain question answering, not always there is an answer present in the paragraph. To address this issue the SQUAD 2.0 dataset [START_REF] Rajpurkar | Know What You Don't Know: Unanswerable Questions for SQuAD[END_REF] was released which contains some questions which do not have an answer.

2. Discrete reasoning over paragraphs -To address the simplicity of the task, questions which do not need any symbolic reasoning, [START_REF] Dua | DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs[END_REF] released a dataset named DROP (Discrete Reasoning Over Paragraphs) which consists of questions which will need to resolve multiple references in a question, perhaps to multiple input positions, and perform discrete operations over them (such as addition, counting, or sorting).

3. Multihop reasoning -When a question needs several paragraphs or sentences to find references to entities which are needed to answer a question, the task is referred to as Multihop QA. For instance, a paragraph which contains an answer might not have any lexical term match with the question terms. Such situation makes the Reading Comprehension models to fail (predict wrong answers). Multihop QA models must take this into account by cross referencing entities from different paragraphs and making links to the answer. Figure 2.2 shows an example by [START_REF] Yang | Hotpotqa: A dataset for diverse, explainable multi-hop question answering[END_REF] where the answer has 5 supporting sentences from 2 different paragraphs which relates the answer to the question.

There are some simple approaches used for the above cases. For unanswerable questions, a binary classifier can be trained to determine if a paragraph contains answers to a question, if the binary predication is positive, then the Start and End linear classifiers can extract the answer from the paragraph. For Discrete reasoning over paragraphs, authors of [START_REF] Dua | DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs[END_REF] propose multi-class classification approach to determine what type of reasoning problem the question refers to and perform further steps. We discuss more works on these topics in the Section 3.

Open Domain Question Answering -OpenQA

Back to where it all started -the pioneering QA task as explained in Section 2.2 is open domain question answering. Open Domain Question Answering or OpenQA is a QA task where a given input data would contain a question and a short answer (which is a factoid answer) and no supporting paragraphs or documents. The systems should find relevant documents from some sources, and further find relevant paragraphs and perform answer processing to extract a short answer. In other words,

Task based classification

OpenQA task follows the QA pipeline, where a QA system as shown in Figure 2.1 shall perform intermediate steps in order to obtain a final answer. An example shown above shows how a relevant paragraph does not always contain an answer. P 2 is a relevant paragraph according to an Information Retriever model but the answer is not present in this paragraph. A model trained on Reading Comprehension task will extract some answer which is incorrect. To overcome this problem, RC 2.0 models can be used in combination with Answer Sentence Selection models or probabilistic models can be used to extract answers only from highest probable paragraph using a Reading Comprehension model.

Datasets/Corpora

The performance of a QA model can be tested and reported on benchmark datasets which are used by the research community. In this section we list some of the well known datasets used for reporting results on particular tasks. As explained in QA pipeline in the Figure 2.1, the section 2.2 details three main tasks and the recent variants of the Reading Comprehension task. Listed below are the datasets corresponding to these tasks. [START_REF] Lai | A review on deep learning techniques applied to answer selection[END_REF]. WikiQA questions without any correct answers are removed as done by (Y. [START_REF] Yang | Wikiqa: A challenge dataset for open-domain question answering[END_REF] Table 2.1 presents the datasets for the Answer Sentence Selection task.

Answer Sentence Selection

TrecQA is a well known dataset to benchmark a system built for Answer Sentence Selection task. It was created from the TREC Question Answering tracks (M. [START_REF] Wang | What is the Jeopardy model? A quasi-synchronous grammar for QA[END_REF]. It contains real questions created from search engine logs and sentences from news articles returned by the participating systems in TRECQA task. InsuranceQA by [START_REF] Feng | Applying deep learning to answer selection: A study and an open task[END_REF] is a large scale dataset on insurance domain. The questions and answers were collected from the website Insurance Library7 . The questions are from real world users and the answers with high quality were composed by professionals with deep domain knowledge. There could be multiple correct answers for some questions so that the number of correct answers is larger than the number of questions.

SQUAD-Sent dataset was created by us by modifying the SQUAD dataset (Rajpurkar, J. Zhang, et al., 2016) designed for reading comprehension, into an answer sentence selection dataset to provide the answers in their original context. Sentence tokenization was performed on SQUAD dataset paragraphs using spacy toolkit8 and answers were checked for an exact match of strings in the sentences. There is only one positive sentence per question and the all other sentences are negative examples.

Although Natural Questions dataset by [START_REF] Kwiatkowski | Natural questions: a benchmark for question answering research[END_REF] contains long sentence answers (like in sentence selection task) for questions, with both relevant and irrelevant paragraphs. It is not explicitly used for Answer Sentence Selection rather it is used for RC 2.0 or OpenQA. Questions come from Google search engine queries filtered by human annotators along with the answers.

Answer Extraction

The Table 2.2 presents different answer extraction task datasets for Reading Comprehension, Advancements to Reading Comprehension -RC 2.0, and OpenQA tasks.

The rise of deep learning models in the field of natural language processing gave rise to the demand for large scale labelled datasets across different tasks. For reading comprehension, the first large scale synthetic dataset was constructed by [START_REF] Hermann | Teaching machines to read and comprehend[END_REF] who used a large scale news domain corpus and converted that into a cloze style question answering dataset who goal is to find the missing entities for the query, in the paragraph. CNN and Dailymail datasets presented in the Table 2.2 shows this first large scale supervised reading comprehension dataset created synthetically which was inspired by the Cloze style QA.

Following this, a work by (Rajpurkar, J. Zhang, et al., 2016) Due to the issues pointed out in the Section 2.2.4 with respect to the SQUAD v1.0 dataset on Reading Comprehension, all the other datasets published with changes are referred as RC 2.0 in the Table 2.2 including SQUAD v2.0.

SQUAD 2.0 dataset [START_REF] Rajpurkar | Know What You Don't Know: Unanswerable Questions for SQuAD[END_REF] was released two years after the first one with additional data which contained questions without an answer. The dataset contained an additional data point about being answerable or not. The unanswerable questions were annotated by crowd sourced workers.

NewsQA is a RC 2.0 type dataset with unanswerable questions. Only the CNN articles from the dataset of [START_REF] Hermann | Teaching machines to read and comprehend[END_REF] were used and crowd sourced workers were asked to create questions in natural language. This is done to convert a synthetic cloze QA dataset to a crowd sourced human annotated QA dataset.

HotpotQA [START_REF] Yang | Hotpotqa: A dataset for diverse, explainable multi-hop question answering[END_REF]) is a special RC 2.0 dataset which pointed out the issues with Reading Comprehension or traditional RC 2.0 datasets with unanswerable questions being simple to answer without any reasoning required.

Another challenge with models trained on these datasets is the interpretability or explanation. Therefore this dataset provides supporting facts from different parts of a paragraph which are required to answer a particular question. This dataset is a Multi-hop reasoning dataset which means an answer can be answered with information taken from more than one document to arrive at the answer. This solves another shortcoming of the RC datasets such as SQUAD whose questions have high lexical similarity around the answer terms in the paragraphs.

DROP dataset [START_REF] Dua | DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs[END_REF]) is a crowd sourced dataset which is also a Multi-hop reasoning dataset like HotpotQA but has questions which require discrete operations such as addition, counting, or sorting to answer questions. Some of the other operations needed to answer these questions involve -subtraction, comparison, selection, addition, coreference resolution, different answer spans and more. The authors also propose a multiclass classifier method to classify which kind of reasoning problem the question would belong to and perform discrete operations.

QUASAR-T dataset [START_REF] Dhingra | Simple and Effective Semi-Supervised Question Answering[END_REF] consists of trivia questions which were collected manually by a reddit user and posted freely online10 . It is in the format of OpenQA task where the paragraphs are retrieved using LUCENE tool and not all the paragraphs contain the answer.

SearchQA [START_REF] Dunn | Searchqa: A new q&a dataset augmented with context from a search engine[END_REF]) is a large-scale open domain QA dataset which consists of QA pairs crawled from J!11 archive and the paragraphs are obtained by retrieving 50 webpages for each question from Google search.

TriviaQA dataset [START_REF] Joshi | Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension[END_REF] consists of QA pairs created by trivia enthusiasts and documents gathered by retrieving 50 webpages per question using Bing Web search. For our experiments involving Quasar-T and SearchQA, we use the retreived paragraphs from (S. Wang, M. Yu, Guo, et al., 2018) as done by (Y. [START_REF] Lin | Denoising distantly supervised open-domain question answering[END_REF].

CuratedTREC dataset [START_REF] Voorhees | The evaluation of question answering systems: Lessons learned from the TREC QA track[END_REF] is based on the benchmark from the TREC QA datasets from 1999, 2000, 2001 and 2002. WebQuestions [START_REF] Berant | Semantic parsing on freebase from question-answer pairs[END_REF] is designed for answering questions from the freebased knowledge base which was built by crawling Google suggest API and the paragraphs were retrieved from English wikipedia.

Natural Questions by [START_REF] Kwiatkowski | Natural questions: a benchmark for question answering research[END_REF]) is a large scale QA dataset with questions coming from Google search engine which are anonymous and aggregated. They also release a 5 way annotated test data to capture different ways of human annotations. In the paper they report some findings on 25-way annotations on 302 examples to highlight human variability for annotations. This dataset is 3 times bigger than SQUAD dataset and contains long answers (paragraphs or wikipedia HTML bounding boxes highlighted by the annotators) and short answers (answer spans highlighted by the annotators).

There have been some debate about the inductive bias in some of the datasets like SQUAD dataset where people were first shown the paragraphs and answers while annotating and were asked to frame questions based on that. This arguably gives an easier set of questions than when somebody was just asked to find answers in a paragraph which the SQUAD task is aimed for. Datasets like Natural Questions do not ask users to formulate queries but rather choose the queries based on existing query logs from their search engines to avoid such biases.

The field of Question Answering has seen a lot of datasets because of the rise in the usage of deep learning methods and the trend of releasing more models and datasets. This has changed and improved some of the issues in the former QA systems which heavily relied on feature engineering. In the section 3, we discuss some works which have had significant impact in the QA field.

BIOASQ dataset

BIOASQ 12 challenge is a large-scale biomedical semantic indexing and question answering task [START_REF] Tsatsaronis | An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition[END_REF] which has been held successfully for 7 years. The challenge proposes several tasks using biomedical data. One of the tasks focuses on Biomedical question answering (Task B Phase B) where the goal is to extract answers for a given question. Tab. 2.3: BIOASQ datasets used in all our experiments along with their splits. The numbers represent number of questions. It is a small scale expert annotated QA dataset.

Open Domain vs Closed Domain Corpus

Closed Domain corpora are those datasets that come from a specific domain source which consists of special definitions, vocabulary terms and features which corresponding to the domain such as Insurance, Medical, Biomedical, Scientific, law etc. Their size is often small scale.

Open Domain corpora are those datasets that come from a broad range of data sources and are not constrained by some special domain specific content. Usually datasets coming from News, Wikipedia, Common crawled data from the internet that do not require certain domain expertise to understand. They are termed under Open Domain data for textual corpora. Their size is often large scale.

BIOASQ Question Answering dataset is a biomedical domain dataset with questions annotated by biomedical domain experts. Domain specific (closed domain) corpus often have certain special characteristics which make them different. Some of the characteristics are listed below:

• Specialized vocabulary.

12 http://bioasq.org/

30

Chapter 2 Question Answering -Tasks and Problems

• Different answer variants (abbreviations, symbols etc.) for a same question in factoid QA.

• Small scale data.

• Created with the help of domain experts.

• Focused towards a specific community of users.

BIOASQ dataset being a closed domain dataset, was annotated by biomedical experts with the support of biomedical documents. Experts were asked to curate questions and annotate or provide answers to them.

Evaluation metrics

Different Question Answering tasks have different evaluation metrics because of different outputs. We discuss the evaluation metrics we used in our work in this section based on the corresponding QA task. We experiment mainly on the tasks of Answer Sentence Selection, Reading Comprehension, BIOASQ Question Answering and OpenQA.

Answer Sentence Selection

For a given question, there are a set of sentences out of which one or more sentences are correct. For evaluation, Mean Average Precision (MAP) (Eq. 2.1) and Mean Reciprocal Rank (MRR) (Eq. 2.2) are used.

Precision is calculated for each question if the highest ranked (or scored) sentence is retrieved correctly or not (P = 0 for each correct sentence that was not retrieved). The average is then calcuated for each question. Finally, an average over all questions is calculated.

M AP = 1 N N j=1 1 Q j Q j i=1 P (rel = i) (2.1)
with Q j being the number of relevant sentences for question j; N the number of questions, and P (rel = i) the precision at i th relevant sentence.

Evaluation metrics

In the case of MRR, each system returns some K number of answers, in rank of confidence of their correctness, for each question. The MRR is defined as the mean of the inverse rank of the first correct answer, taken over all N questions:

M RR = 1 N N i=1 RR i (2.2) RR i = 1 r i (2.3)
The score for an individual question i is the reciprocal rank r i where the first correct answer appeared (0 if no correct answer in top K (often K=5) answers). Thus, there are only six possible reciprocal ranks per question for K=5: 0, 0.2, 0.25, 0.33, 0.5, and 1. MRR is averaged over the N questions.

Instead of implementing code for computing this, once can use the scripts13 provided by Trec.

Reading Comprehension

For a simple Reading Comprehension system, an answer is present inside the paragraph. i.e the answer is a substring of a paragraph string. There are two types of evaluations commonly done by systems for this task.

• Unofficial Evaluation -Word offsets of the answers in paragraphs are evaluated.

• Official Evaluation -As defined by (Rajpurkar, J. Zhang, et al., 2016), answer strings are extracted from the word offsets, and are normalized before being evaluated.

A simple Reading Comprehension system has two classifier outputs which detect the start and end positions of the answers in the paragraph.

For the Unofficial Evaluation -Accuracy is evaluated to check how accurate the start and end positions individually are extracted by the model. For a question i, if Gold standard label is the same as Predicted label, Accuracy is 1, else the Accuracy is 0 where label is the pair of word offsets. An average over all the questions is calculated.

For the Official Evaluation -The predicted word offsets are used to retrieve the answer text from the paragraph, and are normalized to remove punctuations, stop words, extra spaces etc. and are checked for exact match between the gold standard and normalized prediction strings. If both the strings match, then Exact Match (EM) score is 1 for question i, else Exact Match (EM) is 0, an average over all the questions is calculated.

Along with Exact Match, F1 Score for each question is calculated and averaged over all the questions, using the following equation: Where tokens function tokenizes words from the string and len function calculates the number of words in the string.

F 1 = 2 recall -1 + precision -1 = 2 • precision • recall precision + recall (2.4)
These measures were initially used by (Rajpurkar, J. Zhang, et al., 2016) on the SQUAD leaderboard14 whose code can be found in their repository.

BIOASQ evaluation

BIOASQ challenge has a biomedical question answering (Task B Phase B) task where the goal is to extract answers for a given question from relevant snippets.

For a given question, there are one or more relevant snippets which contain the answer. Unlike Reading Comprehension task, not all answers are correctly annotated in the gold standard data and also not all gold standard answers are present in all the relevant snippets.

The official evaluation as defined by the task organizers is as follows. There are three measures computed by the scripts15 provided.

• Strict Evaluation -Top 1 accuracy -Where accuracy is measured using Exact Match (EM) as done by Reading Comprehension tasks for Top-1 answer.

• Lenient Evaluation -Top 5 accuracy -If the gold standard is present in Top 5 responses, where accuracy is measured using Exact Match (EM) as done by Reading Comprehension tasks.

• Mean Reciprocal Rank -As done in Answer Sentence Selection -the score for an individual question i is the reciprocal rank r i where the first correct answer appeared (0 if no correct answer in top five answers).

All the three measures are averaged over total number of questions and are reported in the challenge leader board16 , batch wise.

Contributions and Conclusion

Our work is funded by the GOASQ project17 which intends to investigate, compare, and combine two different approaches for answering questions formulated in natural language over textual, semi-structured, and structured data. One approach is the text-based question answering that directly answers natural language questions using natural language processing and information extraction techniques. The other tries to translate the natural language questions into formal, database-like queries and then answer these formal queries w.r.t. a domain-dependent ontology using database techniques. This thesis work is focused on the first approach that directly answers natural language questions using natural language processing and information extraction techniques.

As defined in our research objectives (Section 1.3), our focus is towards building QA models which work with better performance both on large scale and small scale datasets such as biomedical domain datasets. For this purpose, we show how Domain Adaptation can be done on biomedical data from open domain QA datasets using Reading Comprehension. We evaluate two models (Reading Comprehension and OpenQA) for biomedical question answering and compare their performance variation to show that the OpenQA modelling of BIOASQ performs better. We also experiment with several open domain datasets for Domain Adaptation process to show which ones perform better while adapted to biomedical domain.

We use SQUAD v1.0, v2.0, HotpotQA, NewsQA datasets for reading comprehension, QUASAR-T dataset for OpenQA, BIOASQ dataset for domain specific dataset. WikiQA, TrecQA for Answer sentence selection task.

Another research objective is to focus on leveraging structured and semantic information for existing QA models. In factoid question answering where answers are usually entities, their types (based on a custom taxonomy) can often be inferred by analysing the question keywords. These types are called as "Expected Answer Types" which are useful in traditional pipeline question answering systems. These are used to filter or eliminate candidates which does not belong to the question type. This information is seldom used explicitly in deep learning models for QA as the goal is to build end-to-end systems. We present a detailed analysis on "Expected Answer Types" for biomedical and open domain QA to verify its usefulness. We also show how entity information and expected answer type information can be used for QA tasks in open domain QA to further improve scores of deep learning models.

We annotate for more variants of answers than just use gold standard data on biomedical QA dataset and show that the Reading Comprehension models perform better when annotations are done correctly with more information. We also show how these annotations can be automatically annotated using Metamap tool which uses UMLS meta thesaurus and fetches similar results to the manual ones.

Deep learning models almost always focus towards building end-to-end systems, and not much emphasis is put towards post processing of the outputs to better rank the Top-K predictions. We use semantic features and structured information from different paragraphs provided for a question, and use it for a ranking model to improve the QA performance. We use some traditional machine learning models to rank a better answer candidate from Top-K predictions into Top-1 position and show that there is a scope for improvement on the predictions from the neural network models. This applies both on biomedical question answering and open domain question answering models on Open QA tasks.

The following chapter presents some of the state-of-the-art models on the above tasks in detail.

Contributions and Conclusion

State of the Art

In the previous chapter we have defined the Question Answering (QA) tasks, datasets, evaluation metrics commonly used. In this chapter we present the state of the art models and present a literature review of how some models and tasks evolved over time.

We present the history of question answering system development from their initial stages, and how they ended up with a structured pipeline of tasks which we see these days. The extensive use of deep learning models in machine learning also had an impact on question answering changing the way some traditional systems work. The following sections first present question answering in the past, before deep learning and with deep learning models. Recently since contextual embeddings using pre-trained language models have started to improve performance of several NLP tasks including question answering, we present some of the works in this regard which are currently in the trend.

Since a long time, Question Answering is always targetted in different ways based on the type of data used. One of the pioneering works in the field of Question Answering was BASEBALL [START_REF] Green | Deep learning with word embeddings improves biomedical named entity recognition[END_REF] which was built for answering questions about baseball games played in the American league over one season. LUNAR [START_REF] Woods | Progress in natural language understanding: an application to lunar geology[END_REF] was designed as a result of the Apollo moon mission, to help lunar geologists. Several others early systems SYNTHEX, LIFER, and PLANES are mentioned by [START_REF] Paris | Towards more graceful interaction: a survey of question-answering programs[END_REF] for the same objective of getting an answer for a question asked in natural language. There were no pipeline structure approaches as we see in Figure 1.1 earlier in those systems and people built rule based systems generally. This kind of QA systems were also built for querying databases. The QA tasks on plain text with answer sentences and spans, gained attention while TREC QA task was organized in 1999. Data such as databases, knowledge bases and graphs, triples, plain text etc. lead to different types of question answering systems. Given a question in natural language, one of the main challenges is to convert or translate the question into a query which can be used to querying a database or a knowledge base. The process involves aligning a question with the KB triples, which needs to overcome lexical gap and to adapt the question parsing to the KB schema in order to determine which phrases are entity or relation mentions. If the question answering data is only based on free text, these problems do not exist but other problems such as text understanding, part of speech tagging, entity recognition, answer type detection etc. are important to tackle [START_REF] Grau | A Corpus for Hybrid Question Answering Systems[END_REF] Our work focuses on text based question answering, mainly factoid question answering and answer sentence selection tasks.

Engineering Approaches

Different question answering systems can be generalised into a single structured pipeline manner. A recent survey by [START_REF] Allam | The question answering systems: A survey[END_REF] shows in Figure 3.1 how different sub tasks are structured into one QA structure and each contribution would come under some sub tasks generally, which is similar to the one presented in Figure 2.1. We list and explain some of the works with respect to the structured pipeline.

To retrieve an answer in a text from a question, there are three main modules namely:

• Question Processing

38

Chapter 3 State of the Art

• Document Processing

• Answer Processing

Question Processing

Question Processing module takes as input a question in natural language. This module Analyzes the question, Classifies it into a question type and Reformulates into queries for the IR engine.

Question analysis is done to analyze and determine the focus of the question. A focus is word or a sequence of words in the question which is used to determine and disambiguate what the question is looking for. For example, in the question "What is the longest river in France?", the question focus would be "longest river". If the Expected Answer Type (that defines the entity type of the answer inferred from the question) and the question focus are known, the system can determine the answer.

Pattern matching rules can be used to determine the focus based on the question type. Machine learning methods using models like Wapiti by [START_REF] Lavergne | Practical Very Large Scale CRFs[END_REF] that uses conditional random fields for sequence labelling tasks can also be used for this purpose.

Determining the question type is important to understand what kind of answer is to be extracted. This information will make the search easier by filtering the type of information to be searched. For example, if the question type is a "location" then the answer expected shall be a location and not a name of a person. Either done by rule based systems or machine learning based classifiers, the question types are classified based on taxonomies.

The effectiveness of the taxonomy chosen is directly linked to the capacity of recognizing the question type. For rule based systems, the taxonomies act as a base to construct rules such as "Where" questions would classify as "Location" class and "When" would classify as "Date" class and so on. This approach was used by [START_REF] Moldovan | Lasso: A tool for surfing the answer net[END_REF][START_REF] Hermjakob | Parsing and question classification for question answering[END_REF][START_REF] Radev | Probabilistic question answering on the web[END_REF][START_REF] Ferret | Finding an answer based on the recognition of the question focus[END_REF] as it was simple, quick and effective.

Machine learning classifiers need labeled questions to be trained on. Choosing the right set of features to represent the question and the type of classifier plays an important role on the performance. Features may vary from simple shallow word analysis to detailed syntactic and semantic features using linguistics analysis. The authors of [START_REF] Hermjakob | Parsing and question classification for question answering[END_REF] used machine learning based parsing and question classification. The authors of (D. Zhang and W. S. [START_REF] Zhang | Question classification using support vector machines[END_REF] compared various choices for machine learning classifiers using the hierarchical taxonomy proposed by [START_REF] Li | Learning question classifiers[END_REF]. Among: Support Vector Machines (SVM), Nearest Neighbors (NN), Naive Bayes (NB), Decision Trees (DT), and Sparse Network of Winnows (SNoW), they showed that with only surface text features SVM outperforms four other methods for question classification.

For the question reformulation into queries [START_REF] Azad | Query expansion techniques for information retrieval: a survey[END_REF] used entity recognition tools, stop words, part of speech taggers to extract keywords and features from question words which are appended to the expanded query along with synsets from the WordNet by [START_REF] Miller | WordNet: a lexical database for English[END_REF]. These are input to Information Retrieval engine.

Document Processing

The first step in document processing module is to extract relevant documents for an input question. A document can be defined as a set of multiple paragraphs of text. A relevant document is a document which has some semantic relationship with the question. This can be based on matching terms between question and documents, terms which are semantically nearby to each other measured using cosine similarity in embedding space, etc. An important detail to highlight here is that a relevant document (judged according to some measure) might not always contain short answer terms but might still be relevant to the question, which leads to different QA tasks.

The authors of [START_REF] Stoyanchev | Exact phrases in information retrieval for question answering[END_REF] present a document retrieval model on a question answering system, and evaluate the use of named entities, part of speech tags in a query and show that phrases extracted from questions, named entities of noun, verb, and prepositional phrases improves IR performance than just words. The authors of [START_REF] Gaizauskas | A combined IR/NLP approach to question answering against large text collections[END_REF] describe an IR model with an NLP model that performed reasonably through linguistic analysis.

In question answering datasets which require a relevant textual paragraph for a question, researchers typically use an IR method to retrieve relevant documents. The authors of (Dhingra, Mazaitis, and William W Cohen, 2017a) use ClueWeb09 service to retrieve 100 HTML documents per question and later do post processing to remove HTML syntax, non textual data, images etc. For paragraph filtering and ordering given in the Figure 3.1, different approaches do it differently. A common way of doing this is by splitting documents into paragraphs which act as answer candidates for Answer Sentence Selection task which predicts the best suitable paragraphs as answers. We detail this in section 3.2.1. Several models and approaches for this can be found on the ACL leaderboard1 .

Answer Processing

Answer Processing module functions differently for different sub-tasks of textual question answering. For long answers like answer sentences, semantic similarities are captured between questions and answer sentences. For short answers like short spans or phrases, entities, one of the several ways of processing answers is based on the expected answer types obtained in the Question Processing module as done by [START_REF] Grappy | Selecting answers to questions from Web documents by a robust validation process[END_REF].

For long answers like in TrecQA dataset2 , the authors of [START_REF] Severyn | Automatic feature engineering for answer selection and extraction[END_REF]) use the answer type and their named entities as features in their tree kernel approach on answer sentence selection and extraction task. The authors of (Gleize and Grau, 2015a) also propose a unified kernal approach for recognizing textual entailment and answer sentence selection tasks.

For short answers, the answer type can be used to infer short answers by using the named entity recognition tools. For the short answers that are entities, a question type classifier would predict a class based on the hierarchies in Figure 3.2, which can be used to extract the matching named entities that are further ranked based on some features such as embedding similarity, term overlap. The authors of [START_REF] Shih | ASQA: Academia sinica question answering system for NTCIR-5 CLQA[END_REF] propose an approach to extract answers in Chinese texts. The authors of [START_REF] Ravichandran | A Rule-based Question Answering System for Reading Comprehension Tests[END_REF] present a model for finding answers using shallow surface text patterns and manually constructed rules on the Web dataset and TREC-10 questions. The authors of [START_REF] Peng | Combining deep linguistics analysis and surface pattern learning: A hybrid approach to Chinese definitional question answering[END_REF] present an approach to capture long distance dependencies using linguistic structures to enhance patterns in chinese QA data.

Earlier works before deep learning models such as the ones by [START_REF] Punyakanok | Natural language inference via dependency tree mapping: An application to question answering[END_REF][START_REF] Cui | Question answering passage retrieval using dependency relations[END_REF] focus mainly on syntactic features such as using dependency trees and relation between terms and the distance between question and sentence syntax trees. They also include named entity features for semantic information. Instead of using strict word match between question terms and answer sentence terms, the authors of [START_REF] Cui | Question answering passage retrieval using dependency relations[END_REF] propose fuzzy relation matching based on statistical models. The authors of (Gleize and Grau, 2015a) use kernel functions to detect paraphrases, answer sentence selection and recognizing textual entailment. Some other works such as [START_REF] Heilman | Tree edit models for recognizing textual entailments, paraphrases, and answers to questions[END_REF]M. Wang and Manning, 2010;Yao et al., 2013b;[START_REF] Yih | Question answering using enhanced lexical semantic models[END_REF] focus on using tree edit distances, probabilistic tree edit models, feature extraction using dependency trees, relations, named entity types etc.

Neural Question Answering -Task based classification

Ever since the rise in the usage of deep learning models and techniques in the field of machine learning and natural language processing. Question answering domain also has witnessed the impact on a lot of question answering systems built using neural network models. For a few sub-tasks of question answering, like answer extraction from plain text, sentence selection task, these models outperform the traditional methods.

From building individual sub-tasks of question answering organized in a pipeline to building end-to-end models which completely rely on input data, deep learning is being used extensively.

Deep learning models are good at automatic feature extraction which the models learn based on the input data, contrary to traditional models which heavily rely on the hand crafted features provided by tools.

In the following section we discuss how deep learning models function on a few QA tasks following the same classification of tasks presented in Section 2. We discuss about the current state of the art models in the respective tasks, how some popular neural network models evolved over time and the advantages/disadvantages of using one model over the other.

Answer Sentence Selection

Given a question and a set of potential answer sentences, answer selection is the task of identifying which of the candidates sentences answer the question correctly. As per the question answering pipeline described in Figure 3.1, the Document Processing module outputs a list of paragraphs corresponding to a question, and the Answer Processing module shall predict which of the answer sentences (or paragraphs) are relevant to answer the question.

For this task, there are several popular labelled datasets used to benchmark results as listed in Table 2.1. The most popular dataset used by the community since more than a decade is TREC QA dataset curated and first used by (M. [START_REF] Wang | What is the Jeopardy model? A quasi-synchronous grammar for QA[END_REF] where the authors use probabilistic quasi-synchronous grammar for question answering. The dataset is designed for open domain question QA task.

Few years later, (Y. [START_REF] Yang | Wikiqa: A challenge dataset for open-domain question answering[END_REF] released a dataset named WikiQA based on Bing search engine query logs and answer sentences were annotated by humans on a crowdsourcing platform. Until this dataset was released, all the models were experimented on TrecQA dataset alone. This shows the lack of datasets for question answering and especially for the task of answer sentence selection. Table 2.1 presents the four datasets out of which two (TrecQA and WikiQA) are widely used for benchmarking models. This task also corresponds to a textual entailment task.

Often the models are also evaluated on a paraphrase corpus (Gleize and Grau, 2015a).

Models and approaches

The ACL page3 for state of the art models for question answering contains a leaderboard with scores comparing MAP (Mean Average Precision) and MRR (Mean Reciprocal Rank) of different models evaluated using official evaluation scripts4 of Trec. Authors can upload their scores directly on the leaderboard by linking their published articles.

As soon as a deep learning model was first used in this task by (L. [START_REF] Yu | Deep learning for answer sentence selection[END_REF] where the authors use a 1D CNN model, others started using different deep learning models for this task. Following the convolutional neural network approach, (Severyn and Moschitti, 2015) propose a siamese CNN model using learning to rank approach, which computes a representation of both entries, candidate passage and question, and a similarity between these two representations using a pooling layer followed by similarity matrix computation. In [START_REF] Yin | Abcnn: Attentionbased convolutional neural network for modeling sentence pairs[END_REF], the similarity of the two entries is evaluated by computing interactions between words of the two texts by an attention layer.

The authors of [START_REF] He | Multi-perspective sentence similarity modeling with convolutional neural networks[END_REF] propose a Multi-Perspective CNN for this task which is further used by [START_REF] Rao | Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks[END_REF] with a triplet ranking loss function to learn pairwise ranking from both positive and negative samples. CNNs are generally used for classification problems where the input size between question and answer pair does not vary significantly. [START_REF] Lai | A review on deep learning techniques applied to answer selection[END_REF] gives a good comprehensive review of the task and summarizes several works which use deep learning models.

According to the authors of [START_REF] Lai | A review on deep learning techniques applied to answer selection[END_REF], there are three general architectures for measuring the relevance of a candidate answer sentence to a question.

1. Siamese Architecture: In a siamese architecture [START_REF] Bromley | Signature verification using a" siamese" time delay neural network[END_REF], The same encoder (a CNN or a RNN) layer is used to build the representations for the input sentences (both the question and the answer sentence) individually.

After that, the relevance score is determined based on the encoded representations. There is no explicit interaction between the input sentences during the encoding process. [START_REF] Lai | A review on deep learning techniques applied to answer selection[END_REF] 2. Attentive Architecture: Rather than encoding representations independently, attention mechanisms can be used to allow the information from an input sentence to influence the computation of the other's representation [START_REF] Tan | Lstm-based deep learning models for non-factoid answer selection[END_REF][START_REF] Santos | Attentive pooling networks[END_REF].

3. Compare-Aggregate Architecture: In a Compare-Aggregate architecture, vector representations of small units such as words of the sentences are first compared. After that, these comparison results are aggregated to calculate the final relevance score.

The authors of [START_REF] Lai | A review on deep learning techniques applied to answer selection[END_REF] also note that boundaries of defining which model follows which architecture is often unclear because some models mix different representations inspired from different types of models making it harder to classify under a single category.

Following these definitions for neural model architectures and definitions for Pointwise and Pairwise learning approaches as defined in Section 2.2.2, the Figure 3.3 presents an overview of deep learning models used for this task. A similar overview can also be found on the ACL leaderboard 5 .

The authors of (Tayyar [START_REF] Madabushi | Integrating Question Classification and Deep Learning for improved Answer Selection[END_REF]) use a CNN model proposed by [START_REF] He | Multi-perspective sentence similarity modeling with convolutional neural networks[END_REF] using question classes (or question types as described earlier) to enhance the dataset by highlighting entities in it. Highlighting entities were done by mainly two ways called Bracketing (appending a special token before and after the entity occurrence) and Replacement (replacing the entity word with a special token) methods. They propose the above two methods for highlighting expected answer types or question classes in the answer text.

Answer Sentence Selection task was extensively studied and several works using different neural architectures were proposed. But the question answering community shifted focus towards tougher aspects of the factoid QA tasks such as answer extraction and answering tougher questions which require reasoning skills than just factoid answer type matches.

Reading Comprehension

Reading Comprehension task has been addressed in several ways. One of the popular tasks in CLEF was QA4MRE by [START_REF] Peñas | QA4MRE 2011-2013: Overview of question answering for machine reading evaluation[END_REF] which provided multiple choice questions (with one correct answer) and the goal was to understand single documents and answer a question out of the options provided.

Two of the other earliest Reading Comprehension systems are based on pattern matching techniques with bag-of-words [START_REF] Hirschman | Deep read: A reading comprehension system[END_REF], and a rule based system Quarc (Riloff and Thelen, 2000) which use rules based on the question words present in the question. The dataset consist of 115 questions in total. The authors of [START_REF] Poon | Machine reading at the university of washington[END_REF] propose an approach using information extraction methods for detecting predicate argument triples that can later be queried as a relational database, similar to converting queries in natural language to structured queries. The authors of (Gleize and Grau, 2015b) use word vectors and tree edit model on graph representations of the passages and answer choices to extract edit sequences which decide the correct answers among several choices of QA4MRE task [START_REF] Peñas | QA4MRE 2011-2013: Overview of question answering for machine reading evaluation[END_REF].

With the introduction of a new dataset by [START_REF] Hermann | Teaching machines to read and comprehend[END_REF], cloze style QA task was used to convert news domain data into QA data. Since this was synthetic, human annotated data like SQUAD dataset by (Rajpurkar, J. Zhang, et al., 2016) became more popular as they were harder than the synthetic ones. The task is to identify an answer span (a short answer) in the given paragraph which is the correct answer. The models built for this task must focus on analyzing the given paragraph to extract an answer from it. The main assumption of this task is that the answer is always present in the paragraph. This task is often referred as Machine Reading as well.

The lack of training data is one of the bottlenecks of using deep learning models. Not just question answering, but this is common to many other NLP tasks. A solution for this bottleneck is to create more data (labelled data). There are three common ways of doing this,

• Creating labelled datasets by automatically generating question and answers from a source [START_REF] Hermann | Teaching machines to read and comprehend[END_REF] -Synthetic datasets.

• Creating semi-supervised models with limited labelled data (Dhingra, Mazaitis, and William W Cohen, 2017a) -Semi-supervised datasets.

• Creating labelled datasets by human annotations (Rajpurkar, J. Zhang, et al., 2016) -Human annotated datasets.

Datasets

Synthetic datasets are the ones that can be created automatically and without expert knowledge for annotations. The authors of [START_REF] Hermann | Teaching machines to read and comprehend[END_REF] followed the approach of Cloze style reading where a summary or a paraphrase sentence related to a paragraph is used to create queries. The answer terms are one or more entities present in the query, but hidden or anonymized. The task is to identify which is the entity hidden in the query. The dataset created and released by the authors contained more than 1 million query answer pairs.

The follow up work by [START_REF] Chen | A thorough examination of the cnn/daily mail reading comprehension task[END_REF] criticizes the way these questions are framed and state that the required reasoning and inference level of this dataset is still quite simple. The authors build a simple model which outperforms the state of the art by 7-10% of the original paper results. Although this dataset is relatively simple to address the complex reading comprehension task, this work fetched more 3.2 Neural Question Answering -Task based classification attention towards the reading comprehension task especially towards using an end-to-end model.

Addressing the issues discussed above and the noise present in automatically created synthetic datasets, (Rajpurkar, J. Zhang, et al., 2016) released a new human annotated dataset for reading comprehension named as SQUAD (Stanford QUestion Answering Dataset). This dataset contains over 100,000+ questions posed by crowdworkers on wikipedia articles. The workers were asked to read a paragraph and frame questions based on the paragraph and mark the answer span in the paragraph. This was a pioneering work in reading comprehension which released a human annotated large scale dataset which led to several other works in the domain of reading comprehension for question answering.

Along with the dataset, Kaggle6 like leaderboards for question answering was released7 by the authors of (Rajpurkar, J. Zhang, et al., 2016) which made research on reading comprehension more competitive and comparable. The authors propose a baseline approach using logistic regression and textual features to extract answers and also report human accuracy score which was computed by comparing one of the three answers humans had annotated as a prediction and others as gold standard.

Although a similar leaderboard approach existed already for Answer Sentence Selection8 , the main difference introduced by the authors was a hidden test set. A participant must submit the code and model to the organizers in order to evaluate their model. That made sure that the results were not fine-tuned on the test set and the scores were reproducible and trustworthy which made a leap in transparency of reported results.

A lot of interesting approaches, tricks and models came out as a result since the availability of this dataset which showed how deep learning models would indeed perform better with more data. Most of these below mentioned works can be found on the SQUAD leaderboard9 as well.

Models and approaches

We briefly discuss some of the works contributing towards Reading Comprehension and mainly on SQUAD dataset and highlight some important points from the articles.

When variable length textual data is used such as in paragraph texts where the size might vary significantly from one to another. Using a Convolutional Neural Network (CNN) is not very straightforward as CNNs were modelled on images initially which are fixed size inputs. Several works on Reading Comprehension focus on using Recurrent Neural Networks such as Long Short Term Memories (LSTM) or Gated Recurrent Units (GRU) to model sequential and variable length data. Works by [START_REF] Weissenborn | Making neural qa as simple as possible but not simpler[END_REF][START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] use a similar simple architecture of LSTM or GRUs for both Question and Paragraph encoding and use a mechanism to learn interaction between question layer and paragraph layer. The authors of [START_REF] Weissenborn | Making neural qa as simple as possible but not simpler[END_REF] call it as Interaction Layer and [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] call is it as Aligned Question Embedding which makes an interaction of question terms with paragraphs term possible. The authors of (S. [START_REF] Wang | Machine comprehension using match-lstm and answer pointer[END_REF] used MatchLSTM which is built on Pointer networks by [START_REF] Vinyals | Pointer networks[END_REF] where the output sequence tokens must come from the input sequence. Instead of picking an output token from a fixed vocabulary, pointer network uses attention mechanism as a pointer to select a position from the input sequence as output. This was one of the first few models which used attention mechanism for this task.

In the same regard, attention mechanism which was also used extensively in sequence to sequence models for machine translation and later modified to work on Reading Comprehension by several others and almost all models using RNNs use some kind of attention mechanisms for this task to facilitate interaction between two sequences. Using attention in a bidirectional manner was later shown by [START_REF] Seo | Bidirectional attention flow for machine comprehension[END_REF] termed as BIDAF -BIDirectional Attention Flow which uses a hierarchical multi-stage architecture for modeling the representations of the context paragraph at different levels of granularity. BIDAF includes character-level, wordlevel, and contextual embeddings, and uses bi-directional attention flow to obtain a query-aware context representation. Inspite of this model being more complex than DRQA by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF], the latter performs much better on the SQUAD leaderboard (BIDAF -EM 67.974, DRQA -EM 70.733), which is indeed a surprising aspect.

Ever since RNN models became popularly used for this task, works by [START_REF] Xiong | Dynamic coattention networks for question answering[END_REF][START_REF] Hu | Reinforced mnemonic reader for machine reading comprehension[END_REF][START_REF] Shen | Reasonet: Learning to stop reading in machine comprehension[END_REF][START_REF] Huang | Fusionnet: Fusing via fully-aware attention with application to machine comprehension[END_REF] focused on using attention mechanism in different ways to handle dependency between question to paragraph, paragraph to question, left to right, right to left etc. by modelling different mechanisms with RNNs. and how many attentions are used. This comparison shows how little modifications in the neural architectures led to different results.

Although the focus is to increase metrics like exact match score or F1 score of these models for Reading Comprehension, the training time of these models is increased along with model complexity while using RNNs. That is one of the disadvantages of using RNNs (it is worse while multiple RNNs are used sequentially) where the training time is increased proportionally to increasing complexity of the RNN model.

Convolution operations on the other hand take lesser training times and hence they are better suited when time constraint is considered seriously. Following this problem (A. W. [START_REF] Yu | Qanet: Combining local convolution with global self-attention for reading comprehension[END_REF] propose QANET to use local convolutions and self attention mechanism by [START_REF] Vaswani | Attention is all you need[END_REF] which would avoid the use of RNNs but facilitates using attention mechanism. The self attention mechanism was one of the pioneering works which was famously called as Transformer model which later was used for contextual embeddings. Modifications of RNNs had almost hit a ceiling on SQUAD leaderboard followed by QANET which performed better than the previous RNN methods whose exact match score for a single model was 82.47%. This is the peak performance reported on SQUAD leaderboard for a model that does not use pre-trained language representations, which are explained in the following section.

50

Chapter 3 State of the Art

Pre-trained language representations and reading comprehension

An important milestone of using pre-trained language representations for contextual word embeddings was proposed by (M. Peters et al., 2018a) in their work Deep contextualized word representations which is famously named as ELMO -Embeddings from Language Models. The authors train a deep bidirectional language model on a large corpus to learn word vectors, these word vectors are later added as an embedding layer to downstream NLP tasks. The authors concatenate these contextual word vectors into existing state of the art models for NLP tasks such as question answering (Reading comprehension on SQUAD), textual entailment, semantic role labelling, co-reference resolution, sentiment analysis, named entity recognition and show that the performance of these models can be improved right away without modifying anything with the downstream task model. This was originally inspired by TagLM (M. E. [START_REF] Peters | Semisupervised sequence tagging with bidirectional language models[END_REF] which shows the feasibility of this on sequence labelling tasks.

The authors of ELMO (M. Peters et al., 2018a) use a baseline model of theirs which is an improved version of BIDAF model which fetches exact match score of 81.1 and adding ELMO words vectors to the same model fetches exact match score of 85.8 which is an increase 4.7 points on the SQUAD dataset test set. This work sets a new baseline for using contextual word embeddings trained on language model tasks.

ELMO uses Bi-LSTMs in their model in both the directions left and right, separately to encode the sequences. Instead of these Bi-LSTMs, [START_REF] Radford | Improving language understanding by generative pre-training[END_REF] propose to use a transformer model as proposed by [START_REF] Vaswani | Attention is all you need[END_REF]. Following ELMO and GPT, a work by [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] named as BERT -Bidirectional Encoder Representations from Transformers was released. BERT is also a popular name from the Sesame Street10 which is why a muppet is used while portraying the BERT model. Figure 3.5 compares the three models briefly. BERT is designed to pretrain deep bidirectional word representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of NLP tasks, such as question answering and language inference, without substantial task specific architecture modifications.

For question answering, as shown in Figure 3.6, the final layer of BERT is an additional layer which shall predict the Start and End of the answer span, with no new model specifically for QA. BERT-Large (A variant with large number of layers) scored 84.1% which beat the previous state of the art models at the time of release and the only peer reviewed and published paper at the time of writing. Other variants of BERT and better models have not been published yet althought the results are published on SQUAD leaderboard.

ELMO, BERT, OpenAI GPT, GPT-2 and their variants are trained on large corpus of text with very deep models. The are mainly two downsides of these models: 1) Time required to train. 2) Hardware required to train. Since algorithms can be parallelised easily, more hardware leads to lesser time.

The cost required to train these models using a sophisticated hardware set up is very expensive 11 for an individual or an academic lab to afford for such use cases. These models are made publicly available because of the reason that they are expensive to recompute and fine tuning it leads to better reuse of the models for downstream tasks. But if one wants to pre-train these models from scratch, the cost involved is expensive.

There are new variants of the above models released very frequently these days by different research teams which can be accessible and tested easily by using a repository by Huggingface12 who currently host 7 types of transformer models namely: 1)BERT, 2)GPT, 3)GPT-2, 4)Transformer XL, 5)XLNet, 6)XLM, 7)RoBERTa which can be used easily towards any NLP downstream tasks without the need of pretraining the models ourselves.

Moving on from SQUAD

The simplicity of SQUAD dataset was criticized and the assumption that the answer was always contained in the paragraph gives a pseudo positive relevance for all the paragraphs which in real time QA setting might not hold true because finding the "relevant" paragraphs which contains the answer is also a challenge.

We mainly discuss about the dataset of SQUAD in this section because the Reading Comprehension task witnessed a lot of models and progress in question answering. Several changes were proposed because of the issues of SQUAD dataset, and the authors also released a new dataset SQUAD 2.0 [START_REF] Rajpurkar | Know What You Don't Know: Unanswerable Questions for SQuAD[END_REF] with additional questions without an answer.

Because of the different other assumptions and changes proposed to the task, Section 3.2.3 and 3.2.4 discuss either a different QA setting or a modified version of Reading Comprehension task.

OpenQA: back to the original QA task

OpenQA or open domain question answering is a QA task whose goal is to retrieve answer to a given question in open domain.

The input is only a question and a text collection, and the underlying systems should perform all possible operations to return a short answer. Such operations can be from the structured pipeline as explained in Figure 3.1. Traditional Reading Comprehension models are provided with a paragraph which contains the answer. Whereas in OpenQA there is no supporting paragraph provided to extract answer directly.

A typical system would use an IR engine to retrieve relevant paragraphs for a question and perform Answer Sentence Selection to find the best set of sentences which contain the answer.

An answer extraction module such as a Reading Comprehension model can be used to extract an answer from the sentences and choose the best one among them but the assumption that the paragraph is always relevant (A relevant paragraph is a paragraph that is likely to contain the answer), does not apply for OpenQA. There can be relevant paragraphs which do not contain an answer, which makes OpenQA a more realistic and tougher problem than the task such as the one aimed by the SQUAD dataset.

Datasets

There are not many QA datasets for OpenQA which are large scale in size to use it effectively with a neural network model. For this reason works such as (K. Lee et al., 2019;[START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF]

convert a Reading Comprehension datasets into

OpenQA ones just by considering their question and answer strings and ignoring the provided gold standard paragraphs and follow the IR method either in a cascaded manner [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] or by learning both the document retrieval and answer extraction model in an end to end fashion (K. Lee et al., 2019).

Dataset statistics and details can be found in Section 2.3 in detail.

Models and Approaches

A straightforward way to approach this task is by doing it in a strongly supervised manner where the model assumes that the paragraphs retrieved from a retriever model contain the answer and the noise is ignored. The authors of [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] do it in this way where the retriever model is used to retrieve relevant paragraphs for questions, which are further used in the reader model which is a Reading Comprehension model.

But in reality, there is always noise induced in the retrieved paragraphs as some paragraphs might not contain answers when an IR approach is used. As pointed out by [START_REF] Singh | Entity based q&a retrieval[END_REF], QA is fundamentally different from IR. (K. Lee et al., 2019) highlight that IR is concerned with lexical and semantic matching, but in QA the models require more language understanding, since users are explicitly looking for unknown information which may not be lexically or semantically closed to

54

Chapter 3 State of the Art question terms. Works by [START_REF] Joshi | Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension[END_REF][START_REF] Dunn | Searchqa: A new q&a dataset augmented with context from a search engine[END_REF]Dhingra, Mazaitis, and William W Cohen, 2017a) consider this noise and follow a weakly supervised manner which removes the strong supervision and considers the noise in the gold standard data by IR systems.

Models built for OpenQA ideally should consider this noise from the retriever. Therefore works by [START_REF] Choi | Coarse-to-fine question answering for long documents[END_REF]S. Wang, M. Yu, Guo, et al., 2018) attempt to consider this noise in the model of [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] by separating the question answering into paragraph selection and answer extraction. Both these models only select the most relevant paragraphs among all retrieved paragraphs to extract answers. By doing so, only the best scored paragraph from paragraph selection is used for answer extraction. This approach neglects information present in other paragraphs and does not take negative paragraph information into consideration by doing as above.

A follow up work by (S. Wang, M. Yu, Jiang, et al., 2017) propose strength-base and coverage-based re-ranking approaches for retrieved paragraphs, which can aggregate13 the results extracted from each paragraph by an existing Reading Comprehension system like DRQA by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] to better determine the answer. However, this still suffers from the noise issue in distant supervision data because it considers all retrieved paragraphs indiscriminately. The authors of (Y. [START_REF] Lin | Denoising distantly supervised open-domain question answering[END_REF] use a paragraph selector to filter out noisy paragraphs and keep the best ones to perform answer extraction by considering the combined paragraph and answer probabilities.

The most popular pipeline QA system which raised media attention was the IBM Watson which won the jeopardy challenge against human competitors14 . IBM Watson follows a parallel component based pipeline approach [START_REF] Ferrucci | Building Watson: An overview of the DeepQA project[END_REF] whose complex workflow is as shown in the Figure 3.7.

All the above discussed approaches rely on a cascaded style model which first performs paragraph retrieval and then followed by answer extraction. Both the processes are done sequentially and not learnt together. The authors of [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] use the IR engine like a blackbox model without learning any parameters for the model. The first end-to-end model which learns to retrieve evidence from an open corpus and supervise only by question answer string pairs is proposed by (K. Lee et al., 2019). The authors highlight that the main challenge

Reading Comprehension 2.0 -Modifications and Future of QA

Reading Comprehension task was mainly experimented on SQUAD v1.0 dataset as it was the first ever human annotated large scale dataset for the task. In section 2.2.4, some problems were discussed and new methods were also introduced which gave rise to new datasets. These new methods and datasets are explained in this section and we term it as Reading Comprehension 2.0 as they are generally a variant of the original RC task.

Unanswerable questions

The goal of introducing unanswerable questions in SQUAD 2.0 by [START_REF] Rajpurkar | Know What You Don't Know: Unanswerable Questions for SQuAD[END_REF] is to learn models which can determine if an answer exists to a question in the paragraph. This is similar to the questions of TRECQA task which did not contain any answers in the documents, but on a large scale.

In the setting of open domain question answering (OpenQA), an answer is not always present in the paragraph. There are relevant paragraphs in accordance with a question, which do not contain an answer.

Neural Question Answering -Task based classification

A simple way to learn this datapoint as done by the authors of SQUAD v2.0 is by using the models of [START_REF] Levy | Zero-Shot Relation Extraction via Reading Comprehension[END_REF][START_REF] Clark | Simple and Effective Multi-Paragraph Reading Comprehension[END_REF] which learn to predict the probability that a question is unanswerable, in addition to extracting answers if the question is answerable. A binary classifier which predicts this probability can be learnt along with answer span extraction probabilities Start and End of the answer. Several works including one of the state of the art models BERT by [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] uses the above method to handle this case.

Discrete reasoning over paragraphs

As the RC task is relatively simple and does not need any reasoning required to find answers in the paragraph, the authors of [START_REF] Dua | DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs[END_REF] introduce a new dataset named DROP -Discrete Reasoning Over Paragraphs which is a more challenging reading comprehension dataset which requires discrete reasoning to answer questions which demand numerical answers, choice answers etc. and not just lexically similar term matching answers.

Figure 3.8 shows different types of questions present in the DROP dataset which require different reasoning skills to answer the question. The baseline approach proposed by the authors uses a multi-class classification approach where the traditional RC type answer offsets are extracted along with two classes to predict if it is a Count type or Arithmetic type reasoning, followed by predicting the numbers in the text. Although this is a pretty naive way of doing limited set of operations, the authors state that this is a promising approach to combine neural network methods and symbolic reasoning. The authors also created a leaderboard 15 where one can upload their systems and the website evaluates on the hidden test set automatically online.

Multihop reasoning

The DROP dataset provides a dataset and explains a baseline approach of using multi-class classification approach to solve the reasoning problem. But the answers are still relying on a single supporting paragraph. i.e. Questions in DROP dataset do not use any information from other paragraphs to answer questions.

Suppose we have a question who needs data from different paragraghs to answer a question, neither DROP or other RC datasets have such data or methods to tackle this situation. To address this issue which is called as Multihop reasoning, (Zhilin [START_REF] Yang | Hotpotqa: A dataset for diverse, explainable multi-hop question answering[END_REF] introduce this problem and release a dataset called HOTPOT QA which addresses two main issues: 1) Multihop reasoning, 2) Sentence level supporting facts to provide explainability of the answers.

The systems should not only return answers to the questions which are inferred based on different supporting paragraph information, but also provide supporting facts which resulted in that answer prediction. The dataset also contains questions where two entities are compared, two entities are used to bridge another entity. The authors also propose a baseline approach which uses strong supervision over facts in a multi-task setting. The authors created a leaderboard 16 where one can upload their systems and the website evaluates on the hidden test set automatically online.

Evaluations and Challenges

A question answering system is tested always on datasets to determine its capability.

Recently, there have been several leaderboards which compare results on a hidden dataset and report the performance of a system. These are online exclusive, some are automatic and most of them are updated frequently all over the year.

Some of such popular online leaderboards are:

• SQUAD v1.0 and v2.0 datasets by (Rajpurkar, J. Zhang, et al., 2016;[START_REF] Rajpurkar | Know What You Don't Know: Unanswerable Questions for SQuAD[END_REF] • HotpotQA dataset by (Zhilin [START_REF] Yang | Hotpotqa: A dataset for diverse, explainable multi-hop question answering[END_REF] • DROP dataset by [START_REF] Dua | DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs[END_REF] • Natural Questions dataset by [START_REF] Kwiatkowski | Natural questions: a benchmark for question answering research[END_REF] Offline Evaluation campaigns or challenges are held in order to gather research community efforts to solve a task by providing a standard dataset and evaluation metrics. Participating teams must submit either their systems (code + model) or the system predictions on a test set which the organizers will evaluate and compare with others. Usually a workshop is conducted at the end of the shared task where participants discuss their systems, approaches, tricks etc. and explain how their systems perform. To make it comparable, all the systems use the same datasets.

In this section we discuss six such workshops or conferences based on question answering and mainly focus on explaining about the workshop, data used and different tasks involved. However, this section does not provide an exhaustive list of all popular workshops involving question answering.

TREC

The TREC question answering (QA) track was the pioneer in large-scale evaluation of open-domain question answering systems. TREC is a workshop series designed to provide the infrastructure required for large-scale evaluation of text retrieval and related technologies. A "QA" track was introduced in TREC-8 in 1999 which went on for several years. The motivation was to foster research towards information retrieval systems than document retrieval systems.

Participants were given a document collection and a test set of questions, and expected to return a ranked list of five answer strings which contain an answer to the question. Human assessors read string and decided if it contained the answer or not. TREC-9 dataset contained around 500 questions which is a small scale dataset for today's standards.

In Trec QA task, QA systems return an actual answer (can be an answer sentence or a phrase), rather than a ranked list of documents, in response to a question. TREC has had a question answering track since 1999; in each track the task was defined such that the systems were supposed to retrieve small snippets of text that contained an answer for open-domain, factoid questions.

Trec QA evaluations considered the possibility that different people might have different ideas of what constitutes a correct answer. And the judgements by three individuals which were in conflict were decided by the official evaluation but the difference in the answer judgements were used to measure the judgement on system scores. More detailed version of the evaluation can be found in the document by [START_REF] Voorhees | The evaluation of question answering systems: Lessons learned from the TREC QA track[END_REF].

These datasets from the past (1999)(2000)(2001)(2002)(2003)(2004) are curated (questions without an answer were removed) by (M. [START_REF] Wang | What is the Jeopardy model? A quasi-synchronous grammar for QA[END_REF] and used popularly as TrecQA data for Answer Sentence Selection today.

CLEF

The CLEF Initiative (Conference and Labs of the Evaluation Forum, formerly known as Cross-Language Evaluation Forum) is an ongoing initiative started in 1999 in the view of supporting and promoting research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure.

CLEF initiative is split into two parts: 1) Peer-reviewed conference 2) A series of evaluation labs with some datasets released to conduct evaluations. CLEF has different tracks such as ImageCLEF, VideoCLEF, QA@CLEF, CLEF-eHealth related to different data domains and tasks.

QA track at CLEF has different language question answering tasks. Questions in this track are in natural language. However, answering some questions may need to query Linked Data and some questions may need textual inferences and querying free text data. QA@CLEF had various types of QA Reading as an answer extraction task, whereas QA4MRE was about detecting the right answer in multiple choices provided. Therefore the evaluation and the models built for both the tasks are different even though they carry the same name. More about QA@CLEF can be found on their website18 .

BIOASQ

BIOASQ19 challenge is a large-scale biomedical semantic indexing and question answering task [START_REF] Tsatsaronis | An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition[END_REF] which has been successful for 7 years. The challenge proposes several tasks using Biomedical data. One of the tasks focuses on Biomedical question answering (Task B Phase B) where the goal is to extract answers for a given question from relevant snippets.

Below is an example of a factoid question from the BIOASQ dataset for QA (task B). Each sample contains a question, answer and relevant snippets.

Question: What is the mode of inheritance of Wilson's disease? Answer: autosomal recessive Snippets: The overall sex ratio of patients was nearly 1:1, and genetic analysis of 20 families confirmed an autosomal recessive mode of inheritance.

Every year the challenge is organized in 5 batches, a training set with all the gold standard data is released earlier and the 5 test sets with all new questions and snippets are provided in each batch release. System predictions are expected to be submitted within 24 hours of release of the test set. Biomedical questions with their exact answers, relevant text snippets, concepts, articles, summaries were constructed or selected by biomedical experts from around Europe.

BIOASQ 7 is the seventh challenge and the evaluation measures for BIOASQ task B has always been the same. Strict Accuracy, Lenient Accuracy and Mean Reciprocal Rank (MRR) are the three evaluation measures used. To compute the scores, the exact match of strings between the predictions and the gold standard answers is used to decide if a system answer is correct. Strict accuracy is the rate of top 1 exact answers. Lenient accuracy is the rate of exact answers in top 5 predictions. MRR is the mean reciprocal rank computed on the top 5 system answers. These measures have been the same since the 1st challenge, although the first four challenges had triples and concepts along with snippets in the data. In the last two challenges, only relevant snippets for questions are released.

Several works in the past BIOASQ tasks have used classical question answering pipeline architecture adapted to the biomedical domain which includes modules such as question analysis, passage selection, answer selection which contribute towards the extraction of suitable answers. Some use the domain-specific information from UMLS tools such as Metamap [START_REF] Schulze | Hpi question answering system in bioasq 2016[END_REF], along with other NLP tools like Corenlp, LingPipe (Zi [START_REF] Yang | Learning to answer biomedical questions: Oaqa at bioasq 4b[END_REF].

One of the first attempts to use deep learning algorithms for the BIOASQ task was reported in BIOASQ 5 by [START_REF] Wiese | Neural Question Answering at BioASQ 5B[END_REF] where the dataset was adapted to be used as a reading comprehension dataset whose goal is to extract answers from snippets. The authors use a model trained on open domain questions, and perform domain adaptation from open domain to biomedical domain using BIOASQ data.

The models prior to the first deep learning model by [START_REF] Wiese | Neural Question Answering at BioASQ 5B[END_REF]

MediQA

The MEDIQA challenge20 aims to attract further research efforts in Natural Language Inference (NLI), Recognizing Question Entailment (RQE), and their applications in medical Question Answering (QA).

The shared task of the workshop had a QA task which is to filter and improve the ranking of automatically retrieved answers. Similar to Answer Sentence Selection, the MEDIQA task had documents (collection of paragraphs) instead of a sentence or a small paragraph as an answer. For a given question a system had to return if a question and an answer document pair is correct or wrong (0 or 1). The dataset contained 104 consumer health questions and 104 simple questions about diseases.

MediQA had an automatic evaluation system21 setup by the organizers where the participants could upload the test set predictions.

An overview of the workshop is presented by [START_REF] Abacha | Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering[END_REF] where more details about other tasks can also be found. An overview and proceedings of the workshop is presented by (Choi, Seo, et al., 2018) which also lists different works published recently.

MRQA

Conclusion

In this chapter, we have detailed about question answering models and approaches in the past, and currently used state-of-the-art systems. Indeed, the definition of question answering has evolved in many different ways, resulting in various datasets with different challenges and a large number of models with their distinct advantages and limits. According to the issues we address, we experiment on different QA tasks, including Reading Comprehension, Answer Sentence Selection, and OpenQA.

The current state-of-the-art models on question answering tasks mainly use neural network based approaches. They are mainly proposed for open domain with large datasets. In contrast, the interests in neural network models for domain specific datasets, that are small scaled datasets, are much less studied. One way of using neural network models for small sized datasets is via domain adaptation. We will explore different possibilities for doing domain adaptation and experiment different neural network models and different pre-training data for BIOASQ dataset.

The Reading Comprehension task mainly used in the context of SQUAD dataset (Rajpurkar, J. Zhang, et al., 2016) assumes that the paragraphs given along with the questions always contain an answer. This assumption does not hold good in the case of the BIOASQ dataset [START_REF] Tsatsaronis | An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition[END_REF] where there are several paragraphs that do not contain an answer. Several works on BIOASQ such as (Wiese et al., 2017b;[START_REF] Yoon | Pretrained Language Model for Biomedical Question Answering[END_REF][START_REF] Hosein | Measuring Domain Portability and ErrorPropagation in Biomedical QA[END_REF] use Reading Comprehension task modelling for BIOASQ where the paragraphs that do not contain an answer are ignored from the data. The pre-training is done on SQUAD v1.0 dataset and fine-tuning on the BIOASQ data in all the previous works. We believe that modelling this task differently considering all kinds of paragraphs from BIOASQ dataset would improve the performance of the models.

BERT model [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] is used in many QA tasks currently. Previous best scoring models, such as [START_REF] Hosein | Measuring Domain Portability and ErrorPropagation in Biomedical QA[END_REF][START_REF] Yoon | Pretrained Language Model for Biomedical Question Answering[END_REF], fine-tune the BERT language model (primary task of BERT) towards the target QA tasks like BIOASQ. [START_REF] Yoon | Pretrained Language Model for Biomedical Question Answering[END_REF] pre-train on SQUAD v1.0 dataset (Rajpurkar, J. Zhang, et al., 2016) and [START_REF] Hosein | Measuring Domain Portability and ErrorPropagation in Biomedical QA[END_REF] pre-train on Natural Questions dataset [START_REF] Kwiatkowski | Natural questions: a benchmark for question answering research[END_REF]) before fine-tuning on BIOASQ dataset. Our intuition is that the choice of the dataset to pre-train plays a major role on the performance on BIOASQ dataset. We will study the variation of performance when a QA model is pre-trained on different datasets.

The performance of neural network models is based on the dataset annotation and evaluation protocols. BIOASQ dataset lacks many variants of answer in the gold standard dataset annotated by human experts. Since it is important to be able to construct annotated corpus of a high quality at a reasonable cost, we will study how structured resources, like UMLS which contain large amount of information on biomedical domain, can be used to annotate the answer variants in order to improve the performance of the QA models.

Since end-to-end models are one of the goals of using neural networks, not much efforts have been put towards improving the outputs of a neural network model by post-processing the predictions. Our hypothesis is that the semantic features and structured information from different paragraphs can be used to post-process the predictions to further increase the performance. In the biomedical domain, there exists structured information such as UMLS which can help in detecting biomedical entities and find the matching types between question and paragraphs. In the open domain, the Expected Answer Types inferred from questions can help in finding the matching entity types between question and paragraphs. These matching types highlight the entity types and the entities which are likely to be the answers for the questions. We intend to study and use these features in different ways to improve the QA performance on different models and tasks.

In the following chapters, we detail about our two research questions of 1) Building better models for large scale and small scale data and 2) Leveraging structured and semantic information into QA models and present several hypothesis, experiments, and result comparison in detail.

4

Building Models for Small Scale and Large Scale Datasets "Deep neural network models perform better with more data!", "One needs more data to use a deep neural network model", "Less data? Not suitable to use deep learning models!, "Why is it difficulty to get annotated datasets?" are some of the common phrases we come across in recent times.

In our work, one of the goals was to build a better question answering model for domain specific data. Using deep neural network models directly on small scale datasets would not fetch the best scores because of the small scale nature of the datasets -is one of the popular beliefs among the deep learning community. Since the data (BIOASQ dataset) we came across was small scale, we wanted to experimentally determine how bad the performance of a deep neural network model (usually targeted for large scale datasets) would be on this small scale dataset.

One of the solutions to obtain optimal performance on small scale datasets is to train the models on large scale datasets, and then fine-tune (retrain the model) it towards the small scale datasets. This allows reusing the state-of-the-art models for small scale QA datasets. This process is called as Domain Adaptation and we present a method to perform domain adaptation from open domain data (which is generally large scale) towards biomedical domain data (which is generally small scale).

In this chapter we report several experiments on small scale domain specific dataset of BIOASQ task. We define the process of domain adaptation by formalizing the terms such as Pre-training (first training) and Fine-tuning (retraining), and explain three main kinds of domain adaptations.

Choosing a good model among several state-of-the-art models is a challenging task. We present certain factors which are important to consider before choosing to work on their implementations and explain why we choose one model over the others for Reading Comprehension task.

To perform domain adaptation, one must first choose a suitable QA task to perform the first training phase with a large scale dataset. We compare two QA task models (One for Reading Comprehension task and one for Open domain question answering task) for training and show one outperforms the other on BIOASQ dataset. We also hypothesize that using different datasets (sometimes more than one combined together) to pre-train a model impacts the downstream finetuning performance. To show this behaviour we pre-train a single model with different datasets and finetune it to biomedical data.

The organisation of this chapter is as follows:

• We first define and detail the state-of-the-art question answering models used in our work for 1) Reading comprehension, 2) Open domain question answering models, 3) Answer sentence selection tasks.

• We detail the concept of domain adaptation, and show how to adapt these models to biomedical domain along with general notions of pre-training and fine-tuning.

• We experiment with different word embedding spaces for a QA model to determine which performs better.

• We explain about the choice of a good QA model and compare two QA task models for pre-training.

• For performing domain adaptation, we also need a large scale dataset to pretrain the models, we compare with several datasets and report performance on domain adaptation.

Question Answering models

Question Answering is a broad domain of natural language processing which can be defined in different ways based on the types of tasks involved. As introduced in chapter 2, there are several tasks under question answering.

In this section we explain two QA models which we have used extensively in several studies throughout this research work.

• Reading comprehension, a task that deals with extracting an answer from a paragraph. The input data would ideally contain a question and a paragraph. The model is expected to predict an answer span in the paragraph.

• Open Question Answering, a task that deals with finding an answer provided a question. Sometimes there is a collection of texts provided from an IR engine and sometimes there is no paragraph provided along with the question.

Reading Comprehension

Fig. 4.1: DRQA model by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] Reading Comprehension or RC task although existed since several years, it has gained a lot of attention since the release of SQUAD dataset (Rajpurkar, J. Zhang, et al., 2016). Many works on Reading Comprehension or RC task can be found on the SQUAD leaderboard1 .

We chose a model named DRQA by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] for the following three reasons. 1) The document reader model of DRQA that performed the best on SQUAD dataset at the time of experimenting, 2) The code was released publicly2 to replicate the results reported in the paper, 3) The model was training rapidly (4 hours approximately on a single GPU) and performed better than models which took longer to train and were more complex.

DRQA has two models namely Retriever and Reader which work in a cascaded fashion when the questions are not provided with related paragraphs. Figure 4.1 shows the workflow of the overall model.

The retriever model is primarily based on a tool which uses information retrieval (non-machine learning) techniques to retrieve relevant paragraphs (or documents) for the question based on question terms. The reader model is a simple Bi-LSTM model for Reading Comprehension task which takes as input a question and a paragraph and aims at extracting an answer from the paragraph. [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] in their work so we use the same for this model.

70

Chapter 4 Building Models for Small Scale and Large Scale Datasets

E(Q) = {E(q 1 ), .., E(q m )} (4.1) E(S) = {E(s 1 ), .., E(s n )} (4.2)
A pre-attention mechanism captures the similarity between paragraph words and question words in the same layer. We call it as pre-attention mechanism because it is applied before the embeddings are encoded with LSTM layers, which is usually how attention mechanisms are used such as in [START_REF] Seo | Bidirectional attention flow for machine comprehension[END_REF]. For this purpose, a feature Falign shown in Equation 4.3 is added as an input feature to the LSTM layer.

Falign(p i ) = Σ j a i,j E(q j ) (4.3)
Where a i,j is,

a i,j = exp (α(E(s i )) • α(E(q j )) Σ j exp(α(E(s i )) • α(E(q j )) (4.4)
which computes the dot product between nonlinear mappings of word embeddings of question and paragraph.

They are followed by a 3-layer Bidirectional LSTM layers for both question and sentence encodings. Maximum length of the sequences are set as 200 tokens.

{E(q 1 ), .., E(q n )} = Bi-LSTM({ Ẽ(q 1 ), .., Ẽ(q n }) (4.5)

{E(s 1 ), .., E(s n )} = Bi-LSTM({ Ẽ(s 1 ), .., Ẽ(s n }) (4.6)
These LSTM states are connected to two independent classifiers that use a bilinear term to capture the similarity between paragraph words and question words and compute the probabilities of each token being start or end of the answer span.

P start (i) ∝ exp (p i W s q) (4.7) P end (i) ∝ exp (p i W e q) (4.8)
During prediction, the best spans are chosen from token i to token i such that i ≤ i ≤ i + 15 and P start (i) × P end (i ) is maximized.

Question Answering models

The combined probability predicted as shown above results in a set of predictions and for official evaluations like SQUAD, we only consider Top-1 prediction. For tasks like BIOASQ, Top-5 can be chosen based on the decreasing order of combined probability scores.

4.1.2 Pre-trained Large scale Language Models -BERT

Since the introduction of self attention mechanism by [START_REF] Vaswani | Attention is all you need[END_REF] which are usually termed as Transformer models, a lot of attention has been given towards using these as building blocks in several neural architectures such as in (A. W. [START_REF] Yu | Qanet: Combining local convolution with global self-attention for reading comprehension[END_REF] for question answering. Pre-trained language models which are trained on large scale architectures like BERT by [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] use transformer models as building blocks instead of LSTMs such as in ELMO by (M. Peters et al., 2018b). The authors of BERT train two large scale neural language models (BERT Base and BERT Large) using transfer models as building blocks on two tasks namely 1) Masked Language Modelling and 2) Next Sentence Prediction. Further they modify the model to be able to work on several NLP tasks such as question answering, natural language inference, named entity recognition etc. They do this by modifying the final output layer to fit into target task. In our work we only use the fine-tuning modifications on a question answering task. vectors S ∈ R H and E ∈ R H which are used only during fine-tuning for SQUAD task. The probability of word i being the start of the answer span is computed as a dot product between T i and S followed by a softmax over all of the words in the paragraph:

P i = e S•T i j e S•T j (4.9)
The score of a candidate span from position i to position j is defined as

S • T i + E • T j ,
and the maximum scoring span where j ≥ i is used as a prediction. By doing this small modification and fine-tuning the whole BERT model with this new layer at the end, the authors get state of the art results in several NLP tasks including Reading Comprehension on SQUAD dataset.

BERT has two models namely BERT Large and BERT Base which defines the number of layers and units used in the model. In our experiments, we only use BERT Base as it is faster and easier to fine-tune on a single GPU machine. BERT Large performs the best out of the two models but since our aim was not to fetch the best scores possible, we rely on BERT Base model for all of our experiments.

We use the open source code for BERT experiments from HuggingFace Inc.3 by [START_REF] Wolf | Transformers: State-of-the-art Natural Language Processing[END_REF], whose code is compatible with PyTorch and also is modifiable for several other NLP tasks models.

Open QA -Open Domain Question Answering Model

Open QA task is a straightforward question answering task whose input is just a question phrase without any additional information. One way of modeling this task which we use in our work is by using a Retriever model to retrieve relevant paragraphs for the question, followed by using a Reader model to extract answers from the retrieved paragraphs.

We present a model named PSPR -Paragraph Selector and Paragraph Reader which is an Open QA model by (Y. [START_REF] Lin | Denoising distantly supervised open-domain question answering[END_REF] whose overview is presented in the Figure 4.4 and the code is available online 4 . We do not use the DRQA model as explained in section 4.1.1 directly for this task because DRQA works only on Reading Comprehension task which assumes that all questions are provided with relevant paragraphs, which is not the case for Open QA task which contains a mixture of irrelevant and relevant paragraphs. Paragraphs for the questions are retrieved using an information retrieval model where some paragraphs are relevant and some are not.

E(Q) = {E(q 1 ), .
., E(q m )} (4.11)

E(P ) = {E(p 1 ), .., E(p n )} (4.12)
A RNN layer encodes the contextual information of the sequence.

{E(q 1 ), .., E(q m )} = RNN({ Ẽ(q 1 ), .., Ẽ(q m }) (4.13) {E(p 1 ), .., E(p n )} = RNN({ Ẽ(p 1 ), .., Ẽ(p n }) (4.14)
Using this hidden representation, a self attention operation is applied to get the question representation q:

q = j α j qj (4.15)
where α j encodes the importance of each question word against the other question words which is calculated as:

α i = exp (w b q i ) j exp (wbq j ) (4.16)
Where w is the learnt weight vector. Finally, the probability of each paragraph is calculated via a max-pooling and a softmax layer as shown below:

P r (p i |q, P ) = sof tmax max j pj i Wq (4.17)
where W is a learnt weight matrix.

Question Answering models

The softmax operation in equation 4.17 is applied over total number of paragraphs per question therefore a probability value is predicted for each paragraph. Since not all the paragraphs contain an answer in the Open QA setting, the probability scores from equation 4.17 should indicate if there exists an answer or not.

While training, the paragraphs containing the answer are highlighted as 1 and the rest as 0. While testing, the best answer candidate (â from equation 4.19) is chosen with the highest probability P r(a|q, P ) for a question q which is calculated as shown below :

P r(a|q, P ) = For each paragraph, the authors of (Y. [START_REF] Lin | Denoising distantly supervised open-domain question answering[END_REF]) extract 10 answer spans based on their decreasing order of probabilities represented in equation 4.18. Choosing a lesser value instead of 10 might affect the model performance. This is an important hyperparameter to control the number of answer predictions per paragraph.

In the implementation provided online 5 , the training process is done in three phases:

1. Pre-training the reader model to determine which paragraph has an answer and which doesn't based on answer presence.

2. Pre-training the selector model to learn a ranking function which is similar to the one in Answer Sentence Selection task.

3.

Training an overall model using the above two pre-trained models to perform Reading Comprehension by combining paragraph and answer probabilities.

Choosing a Good model

What defines as a good model? Why is it hard to choose a good model? What factors should one consider before choosing a model for training? -these are some of the questions we asked ourselves before we began performing experiments on QA tasks.

With the rise of deep learning models for many NLP tasks including the ones for question answering we have listed above, there have been exploding number of research articles and models for various tasks. It is often hard to catch up with the latest state-of-the-art research work. If someone wants to use a model to compare for the baseline performance, there has to be certain points to consider before jumping on using the latest or more visibly famous (on social media such as facebook and twitter) research models.

Here are some factors which are important to address:

1. Time required to train these models.

2. Computation power needed.

3. Model complexity and difference in performance over simple models. In an academic research lab scenario, we are often limited to smaller and basic computation resources required to run some experiments. This becomes the first most important aspect in choosing the right experimental setup and models.

Secondly, based on the complexity of the models, some models consume more time to train compared to some other simpler models who train faster. Therefore time is also an important aspect which contributes to choosing the right model needed for some experiments.

Model complexity and difference in performance over simple models is one of the aspects which seems to be less of a concern while experimenting with latest research works. When we began our work on Reading Comprehension task. We came across several works which were very similar to each other in terms of model complexity and their performance. For a QA task, research done on SQUAD dataset and the scores reported on the SQUAD leaderboard6 are a good example to show this point.

Some models were better than the rest and simpler. Particularly we stumbled upon two models, A simple model DRQA by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] which we have explained above and we use extensively in our works and another complex model named BIDAF -Bidirectional Attention Flow by [START_REF] Seo | Bidirectional attention flow for machine comprehension[END_REF].

BIDAF proposes a bidirectional attention flow and a complex RNN based architecture (atleast 2 LSTM layers) whose code7 takes ∼20 hours to compute on the SQUAD train set and fetches 67.7% EM score on the SQUAD dev set. DRQA which is a much simpler model with just 1 layer of 3 layered Bi-LSTM trains on SQUAD train set in ∼4 hours and fetches 69.5% EM score on the SQUAD dev set. The training time of DRQA model is 5 times lesser than that of the BIDAF model.

These two models were reported and published around the same with DRQA being released in March 2017 and BIDAF being released in November 2016 with a gap of around 5 months. Although DRQA was published late, the model is much simpler and fetches better scores than BIDAF on the exact same dataset. This difference in model complexity and simpler model (DRQA) performing better on a comparable experimental setup on the same dataset raises suspicions about the actual contribution of the attention layers and extra LSTM layers required for the task as presented by [START_REF] Seo | Bidirectional attention flow for machine comprehension[END_REF].

Does this phenomenon occur because of engineering tricks which was done on the DRQA code or just hyper parameter optimization ? We could not come up with a conclusion in this regard so we went ahead with choosing DRQA as a base model for all our Reading Comprehension experiments. This phenomenon shows that using a simple model sometimes is better than using the latest and famous (well discussed in social media) models which may not be always better performing. Therefore we proceed by using a simple model for our experiments.

In the following section, we introduce domain adaptation and detail about adapting QA models which are primarily modelled for open domain question answering tasks, towards biomedical domain question answering.

Domain Adaptation

Domain adaptation is a field associated with machine learning and transfer learning. This scenario arises when we aim at learning from a source data distribution a well performing model on a different (but related) target data distribution. -Wikipedia8

In our research context, we use domain adaptation to learn a model on a large scale dataset and use the same model and its parameters to learn on a small scale dataset.

In this case both the datasets are similar in nature but come from different source domains. This facilitates to use a deep neural network model effectively on small scale datasets like BIOASQ data. Training a model from scratch on a small scale dataset might not result in the best performance, therefore domain adaptation is carried out.

We first present the general process of domain adaptation before presenting different types of domain adaptations (sometimes referred as transfer learning) for biomedical QA task BIOASQ from open domain data.

The process

In this section we first briefly explain the process of domain adaptation in a generic manner. In Pre-training and fine-tuning or domain adaptation can also be done in several ways. The general approaches are listed below.

• Type 1 -The target task remains the same for pre-training and fine-tuning. Pre-training should be done on a large scale dataset from random initialization of parameters. Fine-tuning can be done on a small scale dataset by loading the model parameters from pre-trained model rather than random initialization. This approach is used when a target dataset is small scaled and using it to train a deep neural network would result in overfitting. This type of pre-training is common in computer vision field where models are pre-trained on Imagenet [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF] and fine-tuned on target image classification datasets.

• Type 2 -The tasks are different for pre-training and fine-tuning. Pre-training should be done on a large scale dataset from random initialization of parameters. Fine-tuning should be done on a different model which uses certain parameters from the pre-trained model which are frozen (non-trainable) and learns some parameters which are randomly initialized on a different task. These approaches in NLP were initially proposed for sequence labelling tasks by (M. E. [START_REF] Peters | Semisupervised sequence tagging with bidirectional language models[END_REF] which were later evolved into ELMO (Embedding Language Models) by (M. Peters et al., 2018a) which significantly improved the state of the art across a broad range of challenging NLP tasks such as question answering, textual entailment and sentiment analysis. This type of method uses special contextual text embeddings obtained from the pre-trained models that are added as features into downstream models built for another task.

• Type 3 -The tasks are different for pre-training and fine-tuning. Pre-training should be done on a large scale dataset from random initialization of parameters. Fine-tuning should be done on the pre-trained model by modifying certain layers to fit to the new task. Newly added layers can be randomly initialised and pre-trained model layers together with newly added ones are trained on the new task. This approach is similar to Type 2 approach with a difference that the reference model can be slightly modified for target task rather than building a different model. This type of approach proposed by [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] At the time of performing these experiments, DRQA model by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] was ranking on top on the SQUAD leaderboard with the source code released by the authors9 . The model is built for the Reading Comprehension task i.e. the model would take as input -a question and a paragraph, and output an answer span in the paragraph. We use this model and modify the dataset to fit the format.

Data modification

We modified the BIOASQ dataset in order to fit it into Reading Comprehension task style data by considering only those paragraphs which contained the gold standard answer and ignoring the rest. We do this the same way as (Wiese et al., 2017b) by performing distant supervision considering the gold standard answers and paragraphs.

An example from SQUAD dataset and BIOASQ dataset:

Question: The atomic number of the periodic After these modifications, we can highlight the differences between SQUAD dataset and BIOASQ datasets:

• BIOASQ can have multiple paragraphs and SQUAD has 1 paragraph.

• SQUAD paragraph always contains an answer and BIOASQ paragraphs might not always contain exact matching gold answers.

• SQUAD has 1 answer, BIOASQ has multiple answers (as shown in the above example).

DRQA considers each question and a paragraph as a pair for each sample datapoint.

It does not take into account multiple paragraphs per question or multiple answers per paragraph. Therefore in order to modify our data to match this format, we repeat the same question with multiple paragraphs creating same number of samples as the number of paragraphs and consider only one gold standard answer per paragraph. We keep only the positive paragraphs (paragraphs containing an answer) in the dataset.

The DRQA model is originally trained and tested on SQUAD v1.0 dataset. We use the same model trained on SQUAD dataset, and perform domain adaptation by finetuning the model on our modified BIOASQ dataset. Figure 4.6 shows the procedure we follow.

The Open domain model is the DRQA trained on SQUAD v1.0 with best performing parameters of the model (best scores on validation set). The Biomedical + Open For BIOASQ evaluation, if we have only 1 paragraph relevant for a question, we take 5 predictions from the same paragraph as Top-5 answers and rank it based on the decreasing score of probability. Top-1 probability answer is evaluated for the strict accuracy as per official evaluation. And Top-5 is evaluated for lenient accuracy.

If we have more than 1 paragraphs per question, we take at least 1 prediction (best candidate prediction with highest probability) per paragraph until we have 5 predictions. Further all the predictions are ranked based on the decreasing score of their answer probabilities.

Importance of Pre-Training and Fine-Tuning

To show the importance of Pre-Training and Fine-Tuning for domain adaptation to biomedical domain, and also show the necessity of doing this to be able to use deep learning methods efficiently, we experimented three approaches on a single model DRQA without altering any hyperparameters. Default parameters as those used by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] in their code10 . Lastly, Pre+Fine is the model which underwent pre-training on a large scale dataset and fine-tuning on the biomedical dataset which clearly shows an improvement over the other approaches as the model which is well learnt on a large scale dataset is tuned to fit a small scale dataset. This set of experiments show that the best approach is to do pre-training on large scale datasets and fine-tuning on small scale datasets, which is a feasible approach to use deep learning models on small scale datasets.

The difference between No-Fine and Pre+Fine shows the need for domain adaptation.

The No-Fine model is a good model with good performance because it has been trained on a large scale dataset, but it can be further improved by doing the above process.

Comparison of word embeddings for domain adaptation

While performing domain adaptation, one hurdle we came across was the set of missing vocabulary terms across different domain word embeddings. The input word embeddings for both Pre-training and Fine-tuning process should be the same for the overlapping words. Therefore a word having one word vector in word embedding space cannot be different while doing pre-training or fine-tuning. Which means a common word embedding space is required for this purpose.

An easy way is to keep the same word embeddings space for both Pre-training and Fine-tuning processes. This gives raise to several questions. Using both domain data together is better? or one is better than the other? , Which algorithm is the best for this purpose?.

In order to check with different domain word embeddings, different algorithms and different hyperparameters required to train them, we train different word embeddings with CBOW and Skipgram models of Word2Vec by (Mikolov et al., 2013) with different hyperparameters and different data from open domain and biomedical domain.

For training biomedical domain word embeddings we chose the BIOASQ 5A task data which consist of 12,834,585 PUBMED articles as an input corpus. We preprocessed this dataset to remove special characters and use the Gensim tool11 to train word embeddings with 50, 100, 200, 300, 400 dimensions with CBOW and Skipgram algorithms. We also use Global Vectors (Glove) which was trained on 840B tokens, 300 dimensions and available to download freely12 to compare performance. We also combine open domain and biomedical domain data into one big corpus and use it to train word embeddings using Word2vec.

We do an extrinsic evaluation of word embeddings by using them on the downstream task of question answering. BIOASQ Task B contains five different test batches with distinct question sets. We retrained the DRQA model on BIOASQ 2017 5B training data after removing each test set. (namely 100, 200, 300, 400) with CBOW and Skipgram models as described in (Mikolov et al., 2013), where the performance is calculated based on Strict Accuracy (Top-1) measure of BIOASQ 4B test sets. It is evident from the table that Skipgram performs better than CBOW when the dimensions are higher. But 300 dimensions is found to be optimal in terms of both strict and lenient accuracy. Increasing it to 400 dimensions did not fetch better results.

These experiments highlight the importance of choosing right word embeddings for biomedical domain QA system. Glove performs better on average because of large amount of data it is trained on, and pretraining on SQUAD which has a large set of open domain questions makes the pretrained QA model to learn better representations. Whereas the biomedical embeddings are trained on lesser data and domain specific vocabulary which has a negative impact over the pretraining of SQUAD. Therefore we further use Glove embeddings for all our experiments as it showed better performance in the above experiments.

In the above set of experiments, we consider only positive paragraphs (paragraphs containing an answer) in the datasets for answer extraction which suits the definition of Reading Comprehension task. BIOASQ datasets consists of some negative paragraphs (paragraphs that do not contain an answer) as well. Therefore in the next section we experiment by modelling the task as an OpenQA task for pre-training.

Domain adaptation with Different Models and Data

In the previous section we focussed on how domain adaptation can be done using a Reading Comprehension task model DRQA with modifying different word embedding inputs. In this section we explore two other aspects of performing domain adaptation. 1) Different QA models. 2) Different Large Scale Datasets.

Our hypothesis is that the pre-training data and different modelling can also result in performance variations. The following sections show which models and which datasets are better for domain adaptation towards biomedical domain.

Comparison of Different Pre-Training Models

In section 4.2.2 we already discussed about how a Reading Comprehension model DRQA is used for domain adaptation. We now consider another question answering task named Open QA. Open QA is a QA task where a question is given and the goal is to retrieve an answer. Since this is an open task, answer can be retrieved either from textual sources or knowledge graphs or ontologies. In our work we focus only on textual sources. An answer has to be retrieved from a set of documents or passages of textual sources as Wikipedia articles or news or scientific articles. Answers are also usually short phrases or entities.

In deep neural network approaches for Open QA, generally answers are extracted using a reading comprehension model on the subset of the retrieved documents or passages considered as relevant (Dhingra, Mazaitis, and William W. Cohen, 2017b;[START_REF] Joshi | Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension[END_REF].

One of the main differences between Reading Comprehension (RC) and Open QA tasks is that the answer must be present in the paragraphs (or documents) for Reading Comprehension (RC), but for Open QA this condition might not hold true because the retrieved documents considered to be relevant to the question might not contain However, as shown in the example one paragraph (P1) has the gold standard answer and the other (P2) does not (i.e. it does not contain the exact match of the answer string). Therefore this resembles more like an Open QA task than a Reading Comprehension task because of several reasons. 1) The existence of paragraphs without answers even though they are considered relevant. 2) Multiple paragraphs containing the same answer. Therefore we experiment by pre-training and finetuning in three different ways for BIOASQ task by taking into account the above two considerations.

In its general definition, the OpenQA task contains questions and their short answers without any given paragraphs. BIOASQ organizers already provide paragraphs in the gold standard data. Therefore, OpenQA can be formulated as a parent task which involves two child tasks, 1) Ranking the relevant paragraphs for a question and 2) Extracting a short answer from the paragraphs. In Open QA, the first task is generally referred as paragraph selection or answer sentence selection and the second task is often modelled as Reading Comprehension although there exists several correct and incorrect paragraphs. Open QA models should distinguish if the paragraph is correct and then extract the answer unlike the RC models.

We use the two models for RC and Open QA which are shown in the Figure 4.7.

Open QA model described in section 4.1.3 is used for the implementation. While adapting the model to BIOASQ, the number of answers to be extracted for BIOASQ is Top-5 and not Top-1. Because of this, instead of choosing from only the top most probable paragraph, we select top 5 answers from combined probability scores in equation 4.20, which might consider 1 or more paragraphs to extract answers from. In this set of experiments, we apply Type 1 domain adaptation.

The data for pre-training

Two datasets correspond to each of the two tasks: SQUAD V1.0 dataset for RC task and QUASAR-T dataset for Open QA task and we show below their differences.

• QUASAR-T which is based on Trivia questions is generated synthetically, and SQUAD is annotated manually by humans on a crowd-sourcing platform.

Domain adaptation with Different Models and Data

• Each question in QUASAR-T is associated to 100 sentence-level passages retrieved from ClueWeb09 dataset based on Lucene, whereas SQUAD 1.0 has 1 relevant paragraph from Wikipedia.

• Some paragraphs in QUASAR-T do not have an answer. 13Comparing the above differences with BIOASQ dataset, the QUASAR-T dataset resembles more closely to BIOASQ than that of SQUAD v1.0 due to the following reasons.

• BIOASQ data has more than 1 relevant paragraphs per question.

• Some paragraphs do not have an answer.

Experiments with Gold standard paragraphs

For studying the modelling of the BIOASQ QA task as a Reading Comprehension task, we use SQUAD v1.0 dataset for pre-training and experiment with the DRQA model as explained in the previous section. For studying its modelling as an Open QA task, we use QUASAR-T dataset for pre-training and experiment with PSPR model.

As the PSPR model is a cascaded model with paragraph selector and paragraph reader, we use the paragraph probabilities predicted by the paragraph selector and multiply them with the answer probabilities obtained using DRQA model to select the Top-5 answers which have combined higher probabilities. The PSPR model has a selector model which predicts a probability score for each paragraph which signifies the answer presence. We use these paragraph probability scores from PSPR model and the answer probability scores from DRQA model and multiply them to choose the answer candidates. The results for this experiment is highlighted in

Experiments with Longer paragraphs (modified BIOASQ data)

For the BIOASQ task we noted that the method used by (J. Lee et al., 2019) with BioBert modified the original paragraphs. For computing the BioBert model, the authors have retrained the original BERT model [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] using Pubmed and PMC articles. For applying it on the BIOASQ task, the authors use longer documents (instead of the actual snippets) from Pubmed corresponding to the data given by BIOASQ in the "documents" field to access the Pubmed documents for each question. Therefore this modification of the gold standard dataset leads to different results for BioBert compared to the gold standard data performance.

The exact pre-processing of BIOASQ dataset in order to do this is detailed in [START_REF] Yoon | Pretrained Language Model for Biomedical Question Answering[END_REF]. The authors have used different strategies and found that using full abstracts from the pubmed document is more useful than using the BIOASQ gold

Domain adaptation with Different Models and Data

standard snippet which is a small part of the abstract. The authors released the data 14 with full abstracts which we use in some of our experiments for comparison.

Similar to these modifications, the authors of [START_REF] Hosein | Measuring Domain Portability and ErrorPropagation in Biomedical QA[END_REF] perform modifications of the BIOASQ gold standard data by using pubmed abstracts as paragraphs instead of the given gold standard paragraphs (which are part of the of the pubmed abstracts). The authors also use the paragraphs obtained from a system which participated in BIOASQ paragraph retrieval task.

They found that the performance is better when the full abstract text is used instead of gold standard data in some cases, and gold standard training data worked better in some cases. The authors conclude that the paragraph retrieval also plays major role than the QA model itself and report that the domain adaptation is not always useful as the non fine-tuned model performed better in some cases. We do not compare our scores with their work because the authors pre-train their models on Google Natural Questions dataset [START_REF] Kwiatkowski | Natural questions: a benchmark for question answering research[END_REF] which contains atleast 3 times more data than SQUAD dataset and diverse set of questions obtained from Google search logs. Because of this pre-training difference with a different dataset, the results are not directly comparable with the systems which have pre-trained on SQUAD dataset. We discuss more about this in the next section.

In order to evaluate the importance of this data modification, we did two experiments and also report results on BioBert with modified paragraph data from (J. Lee et al., 2019) : 1) DRQA with longer contexts 2) BioBert with unaltered data from BIOASQ.

The results are shown in Table 4.6. The results of BioBert is as presented in (J. Lee et al., 2019) where the authors have fine-tuned the models first using SQUAD v1.0 dataset and adapted it to BIOASQ data. We use the modified dataset to experiment it with the DRQA model to determine if it would improve the performance of the pre+fine DRQA model as reported in Table 4.2. We got lower performances to that of the DRQA model trained on the original BIOASQ data.

For comparison, we try the BioBert model on the original BIOASQ data i.e. paragraphs given by BIOASQ data and not pre-processed. The results in

Conclusion

In this set of experiment we compare two QA models based on i.e. 1) Reading Comprehension task 2) Open QA task, and found that the performance is better when using an Open QA model than a Reading Comprehension model. We report the performance on different datasets and show that in some cases OpenQA modelling outperforms the state-of-the-art systems of BIOASQ which use domain adaptation using Reading Comprehension model (Wiese et al., 2017a;J. Lee et al., 2019) in average.

Based on a different pre-processing done by (J. Lee et al., 2019) on the biomedical dataset by using longer contexts from documents than shorter contexts, we found that the Reading Comprehension model performs worse on the pre-processed longer contexts (which are longer documents than paragraphs) compared to the shorter contexts (paragraphs) originally given by BIOASQ data. On the other end, a large pre-trained language model such as BERT performs much better on the pre-processed longer contexts than shorter contexts. In the previous section, the importance of Pre-training and fine-tuning a.k.a Domain adaptation can be seen on small scale datasets in order to use deep neural network models effectively. We also state that the deep learning models perform better when trained on large scale datasets. This assumption gives raise to several concerns to address before using different datasets to train deep neural network models.

Some of those concerns are:

1. How large is large enough for a dataset?

2. What is the minimum size required for deep neural network models to learn efficiently?

3. What kind of data should be used?

4. Are synthetic datasets better than human annotated ones?

How do we choose the best dataset for pre-training?

To address some of the above concerns in terms of question answering and deep learning methods, we present some experiments in this section which uses a single model namely BERT-Base [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF]. BERT15 has two models 1) BERT-Base and 2) BERT-Large. BERT-Large takes approximately 12-13 hours to fine-tune on SQUAD dataset on one GPU, whereas BERT-Base takes around 2 hours of time. Therefore we decided to use BERT-Base for our experiments. The BioBert model which we have used in other set of experiments on other datasets is pre-trained on the BERT-Base model. The goal is to experimentally understand the performance variation of QA models trained on different datasets and compare the performance variation while using it for domain adaptation.

This study was inspired by the work of [START_REF] Talmor | MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension[END_REF] who perform an empirical investigation of generalization of QA datasets across different other datasets. The authors use two models 1) Bi-LSTM based attention model -DocQA 2) BERT Base for QA and perform several experiments by training models on one dataset and testing it on another. They also do fine-tuning on target datasets (not always small scale) and show that the fine-tuning does have positive impact.

One of the conclusions from their study is that the QA models are often overfitting to source datasets and their own data distributions and not well suited when data comes from a different source or is of a different type. Therefore the authors create a single dataset called MultiQA by using several QA datasets listed in Table 4.8 where some percentage of questions were randomly picked from all datasets used in the experiments to create the MultiQA dataset which fetches the best results over all the test sets of the datasets used on the Reading Comprehension task, using the same model BERT-Base. Some important observations from this work are relevant to our work:

• A model trained on sufficiently large scale data is good at performance for generalization.

• An ideal dataset to obtain The Best performing QA model involves mixing up questions from different datasets which are created using different sources of data and different annotation schemes.

• Generalization improves when the model learns information from the target distribution of data.

Based on some observations and conclusions mentioned in the above study, we decided to perform a similar study on domain adaptation by using a single model pre-trained with different datasets to measure the performance of fine-tuning process towards biomedical domain.

Our intuition is that, a model with exact same hyperparameters perform differently on a downstream task when pre-trained with different datasets. Mainly because of the nature and the scale of the dataset. Human annotated data may perform better than synthetic ones which is why human annotated datasets are widely used for benchmarking QA models. Our goal is to experimentally determine these aspects on biomedical domain adaptation.

Experiments

We pre-train the BERT Base model on datasets from as done by the authors of [START_REF] Talmor | MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension[END_REF], we also pre-process and keep only those questions whose paragraphs contain the answers. If there are irrelevant and relevant paragraphs in the datasets, we only keep relevant paragraphs and questions . This allows to use SQUAD v1.0 models with these datasets.

For the following experiments we use the single model and the same set of hyper parameters for all the experiments as used in the code 16 for RC task. Table 4.9 shows experimental results on BIOASQ 4,5 & 6 task test sets.

The experiments are performed on the models trained on open domain large scale datasets from Table 4.8 and fine-tuned on the BIOASQ data. We have two special datasets which are combined datasets created using several other datasets into a single large dataset. 1) SQUAD v1.0, SQUAD v2.0, NewsQA, HotpotQA. 2) SQUAD v1.0, SQUAD v2.0. The above datasets are created by combining the datasets mentioned into a big collection by adding 100% of data from each set. Due to lack of time we did not experiment more combinations of these with varying amount of data from different sets. We compare our results with BioBert ( For individual BIOASQ (BIOASQ 4, 5 and 6) evaluations, on BIOASQ 4 the dataset (SQUAD v1.0, SQUAD v2.0, NewsQA, HotpotQA) fetches better scores for Strict accuracy and SQUAD v2.0 fetches better scores for MRR. On BIOASQ 5 the dataset (SQUAD v2.0, SQUAD v1.0) fetches better scores for both Strict accuracy and MRR.

On BIOASQ 6 the dataset (SQUAD v1.0, SQUAD v2.0, NewsQA, HotpotQA) fetches better scores for Strict accuracy and SQUAD v2.0 fetches better scores for MRR.

On average, the scores of SQUAD v2.0 dataset are better than the BioBert [START_REF] Yoon | Pretrained Language Model for Biomedical Question Answering[END_REF] We believe that the pre-training language modelling task of BERT on different datasets, has a greater impact on the performance of BERT on downstream tasks like QA task.

We use three BERT models trained on different domain data.

1. BERT Base Google by [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] Tab. 4.10: Experiments with BERT-BASE models trained on different text corpus for Language Modelling tasks. We pre-train these models on SQUAD V2.0 dataset and then fine-tune on BIOASQ dataset. Results averaged over 5 official test sets.

Table 4.10 shows experimental results on BIOASQ 4, 5 & 6 task datasets on different BERT-BASE models. We use the SQUAD v2.0 dataset for pre-training as shown above in table 4.9 that this dataset is the better option to pre-train a BERT model and fine-tune to BIOASQ than the rest (excluding the combination datasets).

Table 4.10 shows that the BioBert model by (J. Lee et al., 2019) is the best choice as shown earlier in their article. Even though Scibert is trained on scientific domain data, it is not the best choice for biomedical tasks as seen in the results above. Google BERT performed with least scores as it is not trained specifically on any biomedical corpus.

In the above set of experiments, we try different datasets on BERT models and experimentally verify that the SQUAD v2.0 is the better single dataset to pre-train models than SQUAD v1.0 maybe because of the presence of paragraphs without answers. And BioBert is the best suited pre-trained domain data for BERT model on biomedical tasks. While SQUAD v2.0 performs the best among single dataset pretraining, the combination dataset which combines 4 datasets (SQUAD v1.0, SQUAD v2.0, NewsQA, HotpotQA) is the best option to fetch better results.

In the work of [START_REF] Hosein | Measuring Domain Portability and ErrorPropagation in Biomedical QA[END_REF], the authors pre-train Google BERT on Natural Questions dataset by [START_REF] Kwiatkowski | Natural questions: a benchmark for question answering research[END_REF] and find that in some cases, the results are better than BioBert. We believe that this phenomenon would be different for the authors if they had trained their system with SQUAD dataset and not Natural Questions (NQ) as the NQ dataset has 3 times larger than SQUAD and contains diverse range of questions. Our results and findings in this section supports our argument that pre-training QA datasets also play a major role in downstream QA system performance. Also since the above experiments were conducted on BERT models which are trained on large scale data for language modelling, the performance gain from one dataset to

Conclusion

Using deep learning models directly on the small scale datasets (especially on different speciality domains like biomedical domain) will not fetch optimal results. In this chapter we addressed one of our research questions on building models which work both on small scale and large scale datasets to obtain better performance. Choosing right models which works better on open domain and adapting it to biomedical domain was the primary goal.

We presented and detailed three QA task models for Reading Comprehension, Open QA and Answer Sentence Selection. We also explained in brief about using BERT model for question answering. We presented some factors which we consider as important before choosing a model to work with and explained why we did some of our choices of models.

In this regard, we introduced the concept of domain adaptation from open domain to biomedical question answering and showed the impact on the performance gain. We also formally defined some terminologies used for domain adaptation and showed different ways of doing domain adaptation which is sometimes also referred to as Transfer Learning when the target task is different. The domain adaptation cannot performed be straight away because the data formats are different for the open domain dataset and the BIOASQ data, there we modified some aspects of the data and detailed about the process. This facilitates using state-of-the-art deep learning models for biomedical question answering.

We experimented these models with different word embeddings trained on open domain corpus, biomedical domain corpus and their mixture with various embeddings models like Word2vec, Glove etc. We concluded that using Glove embeddings was the best choice as it is trained on a large scale corpus.

While performing domain adaptation, choosing a good model is equally as important as choosing the right training datasets or right word embeddings. We studied 2 different QA task models suitable for biomedical domain adaptation. We concluded that the Open QA task outperforms the Reading Comprehension task for modelling 100 Chapter 4 Building Models for Small Scale and Large Scale Datasets BIOASQ task. The model of (Y. [START_REF] Lin | Denoising distantly supervised open-domain question answering[END_REF] for OpenQA task on BIOASQ outperforms BioBert model [START_REF] Yoon | Pretrained Language Model for Biomedical Question Answering[END_REF] results on certain sets which shows that simpler models using Bi-LSTMs can be better than massively large scaled models like BERT on tasks like BIOASQ.

We experimented pre-training with different Reading Comprehension datasets by keeping the same model architecture and hyperparameters to show variability in the performance on downstream domain adaptation. SQUAD 2.0 dataset by [START_REF] Rajpurkar | Know What You Don't Know: Unanswerable Questions for SQuAD[END_REF] was the best performing single dataset for pre-training the models. And a combination of 4 RC datasets for pre-training performed the best on the fine-tuning for BIOASQ dataset.

The neural network approaches usually focus on building end-to-end models and seldom focus on improving the predictions by post processing them. Generally, the objective of the neural models in QA is to score better on the Top-1 accuracy by learning the whole model on the QA dataset. Not a lot of emphasis is put on post-processing Top-K predictions. In the OpenQA task, a question is provided with several paragraphs some of which might contain the answer. Our hypothesis is that the semantic features from questions and paragraphs and structured information from different sources like UMLS can be used to further improve the performance of the OpenQA task. The predictions from the neural model are used to compute the features and input into a ML classifier which is modified as a ranker to better rank Top-1 predictions. The following chapter will focus on using different kinds of features in biomedical domain and open domain to improve the QA performance.

Our publications related to the work described in this chapter is listed below:

• In this chapter we address the topic of using existing data from different sources into the QA models to improve their performance. Ever since deep learning models have become prominent in the field of NLP, end-to-end models are on the rise which depend only on the input data to produce outputs. Our research concern in this regard is to use different ways to annotate, enrich and highlight certain aspects of the text data to improve the performance of these models without changing much of their underlying model.

When doing our experiments on BIOASQ data, we came across some issues for annotating the training data and automatically evaluating the results with gold answers. Using distant supervision with gold answers leads to omit a lot of answer variants. Thus, to overcome this problem, after studying its impact on the system results, we leverage biomedical entities and terminology present in the Metathesaurus UMLS1 for improving the corpus annotation. These biomedical terms, entities and their types can be detected in free text data using Metamap. This tool gives an easy access to use UMLS by providing an interface to input text and obtain annotations about the biomedical entities, types and concept identifiers etc.

Neural network models do not explicitly use the semantic information such as Lexical and Expected Answer Types from questions which were used to improve prediction of answers in older models. The answer types are useful in highlighting the entity types in the paragraph text, which are likely to be the answers. We study different ways for modelling this information, by enriching the input data, or better representing entities by embeddings learned on semi-structured knowledge without changing the overall architecture of the system.

In order to exploit structured and semantic information sources for question answering in open domain and biomedical domain, we annotate, enrich and highlight the existing datasets with these extra information. We use it on different QA tasks (Answer sentence selection, Reading comprehension and Open question answering tasks). All experiment results are compared with plain text results to show the performance variations.

Our third study concerns improving the system predictions in the open QA task by introducing a post-ranking process. The features used are based on the earlier mentioned expected answer types, and also on the redundancy of answers in several paragraphs. We conclude our study on the open QA task by experimenting stateof-the-art models on different and even newly built datasets, and highlight the remaining difficulty of this task when modelled by neural network approaches.

The organisation of this chapter is as follows:

• Annotation of Answer variants in BIOASQ dataset to highlight the actual performance of a Reading Comprehension model when the annotation is done extensively.

• We discuss about Expected Answer Types and Lexical Answer Types and their importance for question answering by performing a verification study with a QA model outputs.

• Annotation of entities in plain text w.r.t Expected Answer Types and using special entity embeddings for Reading Comprehension, Answer Sentence Selection and open domain question answering tasks.

• Improving the QA performance using semantic and structured information features such as lexical answer types and semantic types match from UMLS for biomedical data etc. for reranking answer candidates.

• Applying QA pipeline on different sub-task datasets to benchmark scores on Open Question Answering task.

Annotation of answer variants for BIOASQ

dataset to improve performance of existing QA models in Biomedical domain model's performance? What about true negatives while evaluating? Is the evaluation correctly done?" -These are some of the questions which we wondered about when we came across BIOASQ dataset for the first time and built a system of QA on this dataset.

In fact, our results were pretty low compared to what we had obtained when manually did the evaluation ourselves (we are not biomedical experts, but we used reference gold standard answers to decide what is correct and what is not). The dataset of BIOASQ for question answering (task B phase B) as explained in Section 2.3.2 has gold standard answers annotated by experts, based on knowledge base triples and plain text paragraphs provided as reference for annotators. The human annotators provided answer spans which are the gold standard answers do not necessarily cover all variants of the answers present in the paragraphs. i.e. these annotations are not an exhaustive list of all answer variants which occur in the paragraphs. They only provide one or two variants.

In this section we discuss and highlight the problem (with examples) and show how a model can perform when the annotation is done while highlighting all the correct variants.

Problems addressed

The evaluation measures computed by BIOASQ for task B phase B are Strict Accuracy, Lenient Accuracy and Mean Reciprocal Rank (MRR). Participating systems are required to provide an answer span (5 answers maximum) for each question. To compute the scores automatically, the exact match of strings between the predictions and the gold standard answers is used to decide if a system answer is correct. Strict accuracy is the rate of Top-1 exact answers. Lenient accuracy is the rate of exact answers in Top-5 predictions. MRR is the mean reciprocal rank computed on the Top-5 system answers.

These measures have been the same since BIOASQ 1, although the first four challenges had triples and concepts along with snippets in the data. In the last three challenges, only relevant snippets for questions are released. Similar evaluations are performed in extractive question answering and reading comprehension tasks like in SQUAD question answering task where Top-1 accuracy and F1 scores are computed by comparing exact matching strings after removing stop words and special characters. One main assumption in reading comprehension task is that the answer strings are substrings of the snippets, which implies that answers have to be extracted from the snippets. In BIOASQ, the answers are curated by human experts by analyzing the triples, concepts, and snippets (or paragraphs). Thus, the BioASQ dataset and evaluation measures are very similar to that of reading comprehension task, but the major difference apart from the dataset size are the answers instances provided as gold standard which does not contain all the occurrences, abbreviations, different forms of answers which are present in the snippets.

As in [START_REF] Wiese | Neural Question Answering at BioASQ 5B[END_REF], we transform BIOASQ Phase B as a reading comprehension task with domain adaptation (we explained this process in detail in Section 4.2.2). Gold standard answer strings and their offsets are automatically searched in the snippets for exact match and treated as answers if only they are found in the snippets, i.e., the answer string must be a substring of the snippet. We term this method as Distant Supervision method. By doing so the dataset size is reduced to 65% of Bioasq 5B train set which was suitable for adaptation. Other 35% of the questions did not have matching answers in the snippets, because of different variants of answers in the snippets, missing abbreviations, or irrelevant snippets. We also found that in BIOASQ 6B training dataset for factoid questions, 205 out of 619 questions have false negative answers (33% of the dataset). This kind of corpus annotation may result in some problems:

• Less data for learning.

• The model does not learn to extract all the variants because it does not learn from all the variants.

• Evaluation is done using such gold standard data which will lower the results even though the model is performing well.

This above explained automatic snippet annotation method results in:

• False positive: an answer mentioned in a snippet which does not answer the question. i.e. a snippet alone does not justify the answer to the question.

• False negative: a snippet answers the question but does not have the exact string compared to the gold standard string.

Below are some examples for which the answers returned from a reference system is correct (when evaluated manually) but the automatic evaluation classifies it as incorrect. In example 1, because of a missing word "channels", the predicted answer is marked incorrect. In example 2, MDR-TB stands for Multi-drug-resistant tuberculosis, which is from a relevant snippet but since the gold standard has only tuberculosis, it is marked incorrect. Contextually both are valid answers. In example 3, the paragraph has no relation with the question except that it contains the answer terms.

To overcome this problem and enrich the answer space correctly, we manually annotated 618 factoid question-answers pairs from the training dataset of 6B task, by annotating the substring of the gold standard answers in the snippets, and adding answers with abbreviations, multi-word answers, synonyms, that are likely correct answers. We explain this in detail in the following section.

5.

1 Annotation of answer variants for BIOASQ dataset to improve performance of existing QA models in Biomedical domain

Annotations of variants

Manual Annotations

To overcome the issues mentioned above, we manually annotated answer variants in the data. The details of the annotations on the BIOASQ 6B training dataset and the statistics are presented here.

Our annotations include the following type of answers:

• Exact Answer -Exact match with gold standard (GS) answers, which can also be annotated automatically, and different variants of the answers. For example, the annotation of a single GS answer "Transcription factor EB (TFEB)" resulted in 3 annotations, "Transcription factor EB", "TFEB", "Transcription factor EB (TFEB)".

• Lenient Answer -a more general form or a more specific form of an answer. An example is "Telomerase" for "Human telomerase reverse transcriptase".

• Paragraph Answer -The answer matches with gold standard but the snippet alone is not relevant to the question. -this corresponds to the false positive case.

We came across several kinds of snippets. A supporting snippet, or answering snippet, is a snippet that contains the answer and has enough elements for justifying it. It is a correct answer to the question (snippet starting at line 5 in Figure 5.1 for example).

A snippet that contains the answer without justification towards the question will not be annotated with the answer as correct and is considered as a non-supporting snippet (snippet starting at line 3 in Figure 5.1). A snippet that does not contain the answer cannot be a supporting snippet, henceforth it is an irrelevant snippet (snippet starting at line 8 in Figure 5.1).

We use Brat2 annotation tool by [START_REF] Stenetorp | BRAT: a web-based tool for NLP-assisted text annotation[END_REF] shown in Fig. 5.1 to perform the manual annotations of the snippets with the answer to the question. The annotations done include the answer string along with their character offsets in the snippet. Answers were annotated by 3 people from computer science background The annotation files are freely available3 and can be used by researchers who can obtain the BIOASQ dataset from the official website4 . Some statistics of the dataset are listed in Table 5.1 for the automatically annotated answers from gold standard data and the fully annotated data with manual annotations. The annotations are done on 618 BIOASQ 6B training dataset questions. Out of 619 factoid questions, 1 question does not have any snippets. Only 426 questions contain answers from automatic annotation. "Answers" are the count of answers present in the snippets. Avg score represents an average over the total number of questions (i.e. 618). Since in gold standard data, only 426 questions have gold answers in snippets, it is normal for the average to fall below 1. It is clear from the table that the full annotated data contain at least 3 times (1814 answers) more the number of candidate answers over the provided gold standard ones (500 answers).

We found that some answers contained the whole snippet as an answer and that 3503 snippets are repeated in the 6B train set. After filtering those repeated snippets we found 3286 different snippets containing exact matching answers extracted automatically from gold standard data and 4965 unique snippets manually annotated with correct answers.

Automatic Annotations

When we realised the problems addressed in section 5.1.2, we did not use an automatic method to annotate answer variants as it was not straightforward to do using string edit distance for abbreviations etc. Instead we proceeded with manual annotations. Once we began annotating it manually, we had some intuitions of using either rule based techniques, noun phrases and text normalizing using some pre-processing techniques.

Manual annotations are expensive in terms of time and money (if experts are paid to annotate). An automatic alternative approach although not very straightforward and accurate as manual ones, is necessary to provide as an alternate solution for future BIOASQ datasets. Hence we propose an automatic way of annotating these answer variants in this section.

Answers often refer to biomedical entities. The answer variants would also refer to the same biomedical entity with different syntactic structures. Since both represent the same biomedical entity, it is easier to detect the entity than the matching strings. We decided to use the UMLS Meta-thesaurus to determine matching entities. UMLS -Unified Medical Language System is a Metathesaurus created in 1986, which has become an important and a large resource for biomedical science. It provides over 3,100,000 biomedical concepts imported from nearly 200 vocabularies. Each concept is assigned a Concept Unique Identifier (CUI) that uniquely identifies a single meaning. To consistently categorize these huge number of concepts, 133 Semantic Types are defined in UMLS Metathesaurus. In order to further reduce the complexity of the Metathesaurus, these semantic types are divided into 14 groups, called Semantic Groups5 . Metamap6 is a tool that exploits UMLS to annotate free text containing biomedical entities. .2 shows a sample output of metamap tool for a paragraph which contains a biomedical term MDR-TB. We use PyMetamap code7 which is a python wrapper program which uses metamap server running in the background and returns metamap annotations. Metamap outputs an identifier for each entity that it recognizes in the text. This identifier is called as a "CUI". This identifier is unique for each concept found in UMLS. Metamap also gives the exact paragraph span which triggered this concept while annotating. Using this tool we first annotate the gold standard answer phrases provided by BIOASQ to obtain its "CUI". We then annotate the paragraphs with metamap in order to obtain all annotations in them and search for the gold standard answer "CUI" in them. The matching "CUIs" found this way are the exact same concept in UMLS as the gold standard answer terms, but might be syntactically different such as "Multi-drug-resistant tuberculosis" and "MDR-TB". This allows to determine the UMLS concepts for Abbreviations.

There are two cases where automatic annotations fail: 1) Missing words: Missing words from answer phrases will not be annotated if it does not belong to the UMLS concepts. 2) Gold standard answers with both Abbreviations and the expansion together, as UMLS does not highlight both in the annotations.

To determine if the automatic or the manual annotated ones are better, we experiment by training the model using both data individually and evaluate the results with gold standard data in Section 5.1.4.

Experiments

In this section we report the experiments we performed on the annotated and gold standard data. We follow the process of [START_REF] Wiese | Neural Question Answering at BioASQ 5B[END_REF] and use a machine reading model developed by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] that is pre-trained on SQUAD dataset (Rajpurkar, J. Zhang, et al., 2016) for open domain questions and fine tuned to biomedical questions. Note that we use the same model for all experiments without changing any model hyperparameters or methods.

To study the impact of training data sets on the evaluations, we train the models using separately the domain adaptation done via Distant Supervision, Automatically annotated data and Manually annotated data. We evaluate them individually.

For the detailed explanation of the QA system and domain adaptation to biomedical domain, please refer the section 4.2.2. Several embedding spaces were tested as input vectors (Kamath et al., 2017a) and the best performing ones which were the Glove embeddings trained on common crawl data with 840B tokens, were chosen as input to the system. Unknown words were initialized as zero vectors.

As BIOASQ questions have several answer paragraphs, we treat each question and a paragraph as a training sample which might often result in repeated questions with different paragraphs, i.e. for each training example, there is a question, a unique paragraph and the start and end token string offsets of the answer in the paragraph.

Since there are multiple paragraphs per question, while predicting we consider 5 answers from each paragraph based on the decreasing order of their probabilities, resulting in a set of 5*n answers where n is the number of paragraphs. Top 1 and Top 5 are chosen in this set based on their answer probabilities.

Distant supervised vs Manually annotated answers

In this section we compare the performance of systems trained on answers obtained by the method of Distant Supervision and Manually annotated answers.

We perform fine-tuning on two datasets namely • BIOASQ 5B training set, which contains the 4B training data + the answers of the 4B test data -We term it as 5B.

• BIOASQ 6B training set, which contains the 5B training data + the answers of the 5B test set -We term it as 6B.

We term the distant supervised annotated data as Gold, and manually annotated data as Anno., both for train and test splits.

The pre-trained model on open domain QA data is fine-tuned on the above listed Bioasq datasets separately. Evaluation is performed by K-fold cross validation because of the small scale of the data (Table 5.2), and on the official test sets of Bioasq 5B (Table 5.3), which were separated from the training data while fine-tuning. The explanation of scores reported in table 5.2 and 5.3 along with the corresponding experiments on the datasets listed above, is as follows. On the data of Train set mentioned in the first row, we fine-tune it with Finetune data on the second rowwhich is Gold or Anno. version of the answers.

The official evaluation measures8 using Gold or Anno. version of the test answers are highlighted in the third row. The strict and lenient accuracies along with the MRR are reported.

Gold version of 5B data contains 313 questions and Gold version of 6B data contains 428 questions. We consider the remaining questions with no matching answers as incorrectly answered, hence evaluating over all the questions of the datasets (5B -486 questions, 6B -618 questions). Annotated 5B data contains 483 questions and 6B data contains 618 questions.

Overall results of 5B test sets presented in Table 5.3 are evaluated on 150 questions from the test sets of 5B challenge whose gold standard answers are present in 6B challenge train set.

To compare our scores with the ones reported in (Wiese et al., 2017b) and also since the size of the dataset is small, we perform K-Fold ( 5) evaluations which are reported in Table 5.2. To compare with previously reported official test scores in Bioasq 5, we train on 5B training set and test on 5B test sets which are reported in Table 5.3.

The results shown in the tables 5.2, 5.3 and 5.4 highlights the improvements using manually annotated data over the distance supervised annotated data on the QA performance as well as the evaluations with Gold and Anno. versions of answers.

In Table 5.2, training on Gold and evaluating on Gold are the baseline scores. DeepQA MRR score is the K-fold evaluation score of MRR reported on 5B train set by (Wiese et al., 2017b) In Table 5.4, MRR scores are reported separately for each batch. MRR scores in general have the best scores compared to both [START_REF] Wiese | Neural Question Answering at BioASQ 5B[END_REF] and Lab Zhu, Fudan Univer by training on Anno. and evaluating on Anno. versions.

Automatically annotated vs Manually annotated answers

To determine the usefulness of automatically annotated answers done using Metamap with the manual annotated ones, we use the same methods to train as above done for manually annotated answers, but train with automatically annotated answers which are obtained as explained in section 5.1.3 of this chapter. To avoid confusion with distant supervised data and automatically annotated data by metamap, we present the results separately. In this set of experiments, we try to determine which method of annotation is better and if automatic annotation is comparable with manual annotations. If automatic annotations provide better or similar performance, this avoids the need of expensive manual annotations done by experts.

Table 5.5 shows the results of the models trained on automatically annotated data and manually annotated data. Gold Data evaluations are done on gold standard answers given by BIOASQ challenge and Annotated Data evaluations are done on respective annotated data. The results show that both automatic and manual annotations are comparable on 6B set. Lenient accuracy (Top 5) is better with manual annotated data than the automatically annotated data (3 out of 4 experiments). And for Strict accuracy (Top-1) both perform very similar except BIOASQ 5B data where manual data is better when evaluating on Annotated dataset.

Although the data used in the Table 5.5 for Manual annotated dataset and in the previous section of experiments in the Table 5.2 for Annotated dataset is exactly the same, the results on Gold Data evaluation differ largely between these two tables on Manual annotated dataset training. The scores in Table 5.5 is much higher than Table 5.2 because of the difference in the prediction module. In the method presented in

5.

1 Annotation of answer variants for BIOASQ dataset to improve performance of existing QA models in Biomedical domain

Table 5.2, we predict 5 answers per paragraph based on decreasing order of answer probabilities and choose Top 1 and Top 5 based on answer probabilities of all the resulting candidates. In the method presented in Table 5.5, if we have 1 paragraph per question then we take 5 predictions from the same paragraph as Top 5 answers and rank them based on the decreasing score of answer probabilities. If we have more than 1 paragraphs per question, we take at least 1 prediction from a paragraph which has maximum probability until we have at least 5 candidates and choose Top 1 and Top 5 based on answer probabilities of all the resulting candidates. The order of choosing the paragraphs for answer extraction is the same as provided by BIOASQ gold standard data.

These small changes in the prediction module has a large impact on the result mainly because the second method considers answers from paragraphs whose top answers might not have the highest probability compared to another paragraph top answer.

The experiments and results show that the automatically annotated dataset using Metamap performs similar to the manually annotated datasets henceforth this method is better, easier and cheaper to adapt future datasets to cover more answer variants for biomedical domain. The study reported above shows the influence of enriching the training data by manually and automatically annotating variants of gold standard answers, on the evaluation performance. We used UMLS Meta-thesaurus as a source for Metamap tool to detect and annotate biomedical vocabulary. We show the impact of the enriched data on Reading Comprehension task by experimenting on two training datasets. Our method outperforms some of the best-performing systems from BIOASQ without changing the model.

In the following section, we report our studies on using semantic features such as Expected Answer Types both in open domain and biomedical domain on improving the QA model performance in different tasks.

Expected Answer Types (EAT) and their

importance in Question Answering models

Introduction

In traditional QA systems (non deep learning approaches) on text, one of the main criteria for selecting an answer is based on recognizing the Lexical Answer Type (LAT) and the Expected Answer Type (EAT) in order to do a matching with candidate answers.

116

Chapter 5 Leveraging Structured and Semantic Information into Question Answering Models

The Expected Answer Type (EAT) is a type which determines the type of answer for a question. This type is inferred from the question. In open domain, named entity types can be used as EAT. Table 5.6 shows the taxonomy we follow to obtain EAT from the named entity types. We use Spacy tool9 to determine named entities and map them to the Expected Answer Type. Traditional QA systems on text are made of several pipeline modules: question analysis, passage selection, answer selection. Question analysis allows to extract features that are used for selecting passages and extracting the answer. Apart from the plain textual words, these features can be different from a system to another, but they all make use of the Expected Answer Type (EAT) [START_REF] Kolomiyets | A survey on question answering technology from an information retrieval perspective[END_REF].

The definition of Expected Answer Type (EAT) and some examples are presented above. Best methods for verifying if a candidate answer matches the EAT involves feature based supervised learning based on the use of different resources, as cooccurrences and presence in structured resources [START_REF] Grappy | Answer type validation in question answering systems[END_REF][START_REF] Grappy | Selecting answers to questions from Web documents by a robust validation process[END_REF][START_REF] Chu-Carroll | Finding needles in the haystack: Search and candidate generation[END_REF]. In medical domain, this verification was made using UMLS (Zi [START_REF] Yang | Learning to answer biomedical questions: Oaqa at bioasq 4b[END_REF][START_REF] Abacha | MEANS: A medical question-answering system combining NLP techniques and semantic Web technologies[END_REF].

Recent QA approaches are based on deep neural network architectures, mainly in the open domain. The authors of [START_REF] Weissenborn | Making neural qa as simple as possible but not simpler[END_REF] for their RNN based model introduce a supplementary feature that is the word embeddings of LAT. It is defined as the question word or the words around the question word (two different features) whose embeddings are averaged and appended as a feature vector to the model. However they did not report results that allows to evaluate the impact of EAT on the overall performance of the model.

For the studies presented in this chapter, some of the resources used are listed below:

UMLS is used extensively in almost all our works involving biomedical domain data. For example, a concept such as Transcription factor EB (TFEB) has a CUI C1420699 in the UMLS and belongs to the [Gene or Genome] semantic type. If a textual paragraph contains a term "TFEB" or just "Transcription factor EB" without the abbreviation, it belongs to the same concept mentioned above and UMLS tool Metamap can be used to map these text spans to the concept to determine its features and relations with other concepts.

Semantic types and semantic groups have been used in various biomedical information systems, including categorizing clinical research eligibility criteria [START_REF] Luo | Dynamic categorization of clinical research eligibility criteria by hierarchical clustering[END_REF], learning biomedical ontology [START_REF] Petrova | Formalizing biomedical concepts from textual definitions[END_REF], and representing clinical questions for medical QA [START_REF] Kobayashi | Representing clinical questions by semantic type for better classification[END_REF].

Pubmed articles are used to create word embeddings for biomedical domain words which are used for tasks such as Named Entity Recognition (NER) by (Habibi et al., 2017), Question Answering (QA) by [START_REF] Tsatsaronis | An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition[END_REF][START_REF] Wiese | Neural Question Answering at BioASQ 5B[END_REF].

In the following sections we present a study to understand the usefulness of Expected Answer Types (EAT) in Question Answering systems in biomedical domain and open domain.

Verification of the Expected Answer Types in Biomedical Domain

In the first study, we begin with biomedical domain data. To understand the importance of Expected Answer Types in biomedical QA dataset, one set of experiment involves analysing the output of our QA system for determining how many system responses correspond to the Expected Answer Type from the question. For this purpose we use the Reading Comprehension task model DRQA by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] which is explained in Section 4.1.1.

The model does not make use of any LAT or EAT information or any medical domain related resources. The model only relies on input data tagged with named entities and encoded with word embeddings. Thus the goal is to study if it can be interesting to add information regarding the Lexical Answer Type or Expected Answer Type into the model explicitly. Alternatively we can also determine if the model already captures the Answer Type information by analysing the predictions.

For the analysis, we take the gold standard answers provided by the BIOASQ and use Metamap tool12 to annotate the answers to determine their semantic types13 from UMLS. These semantic types are our gold standard answer Expected Answer Types (EAT). We determine the percentage of EAT matches provided by the gold standard answers and the QA system's response to analyze two aspects:

• If the Expected Answer Type inferred from the question by using the Lexical Answer Type (LAT), matches gold standard answer types.

• If the Expected Answer Type inferred from the question is already captured by the QA system's response even when the answer is wrong.

As we explain in the previous section, an Expected Answer Type (EAT) is the type of the answer, whereas a Lexical Answer Type (LAT) is a word or words in the question which is used to infer the EAT.

For the biomedical domain, [START_REF] Neves | BioMedLAT corpus: Annotation of the lexical answer type for biomedical questions[END_REF]) released a corpus named BiomedLat which consists of LAT and EAT annotations for BIOASQ questions14 which were manually annotated with LAT words into them and their semantic group from UMLS which are the EAT.

For our study, we consider different representations for the EAT from the questions:

• The Semantic Group of LAT. -This refers to the LAT from the question, whose semantic groups from the UMLS semantic network15 are used -SGEAT -Semantic Group Expected Answer Type.

• a word embedding for LAT (WEEAT -Word Embedding Expected Answer Type) -word embeddings for LAT words obtained using a pre-trained word embedding space. We use this to measure cosine distance between answer word embeddings and the LAT embeddings (WEEAT). For computing WEEAT when the LAT is made of several words, we compute the average of each word embedding of the LAT. When a word has no embedding, we set its vector to 0.

We use Word2Vec skipgram model with 300 dimensions from (Tomas [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] for computing word embeddings on the biomedical texts of BioAsq 5A task data which consist of 12.8 Million PUBMED articles.

To determine if the Expected Answer Type can be useful for selecting an answer, we study if the EAT given in the BiomedLat corpus by [START_REF] Neves | BioMedLAT corpus: Annotation of the lexical answer type for biomedical questions[END_REF] (the gold standard EAT for questions (GoldEAT)) matches with the BIOASQ answer inferred semantic types (the answers in the gold standard BIOASQ data (GoldAns)) and with the answer semantic types for predicted answers by our QA system (PredAns). Semantic types from the answers (gold or predicted) are inferred using the Metamap tool which annotates the answers with a semantic type.

An example to explain the above terms: Question: What disease in Loxapine prominently used for? Gold Standard Expected Answer Type (GoldEAT) : Disease Gold Standard Lexical Answer Type: disease. Gold Standard Semantic Group: DISO Disease or Syndrome.

Answer (BIOASQ given): Schizophrenia Answer Expected Answer Type (GoldAns): Disease Predicted answer (QA model output): Clozapine Predicted answer inferred Expected Answer Type (PredAns): Drug

In the above example, GoldEAT is Disease. GoldAns is Disease. Therefore the gold standard answer given by BIOASQ matches the GoldEAT which is an expected result.

PredAns is Drug which is inferred from the predicted answer Clozapine which is a wrong answer for the question, therefore the answer types do not match.

Experiments and Results

For the experiments, we consider only the factoid questions from BiomedLat corpus. We split the dataset into train and test sets (80% train and 20% test). The statistics reported in the Figure 5.3 are for the factoid question test set.

We compute cosine similarities between LAT word embeddings in questions and three different answer word embeddings which are detailed below:

• GoldStandard-maxCosine (crossed points): Answer words are Gold standard data annotated with all answer representations that have the maximal cosine similarity with WEEAT.

• DRQA-cosine-top1 (triangular points): Answer words are Top-1 answers from DRQA output. The similarities of correct (resp. false) answers are plotted above (resp. below) the X-axis.

• DRQA-maxCosine (round points): Answer words are from top 5 answers of DRQA output that have the maximal cosine similarity with WEEAT. The similarities of correct (resp. false) answers are plotted above (resp. below) the X-axis.

From Figure 5.3, we can see that gold standard answers (GoldStandard-maxCosine) show a significant correlation with LAT in terms of word embeddings, although there are 6 questions whose LAT have 0 similarity with WEEAT caused by missing word embeddings for the medical domain vocabulary. Another clear observation is that many of top-1 wrong answers the red crossed points are for gold-standard answers that have the maximal similarity with the question LAT word embedding; So all the crossed points are correct answers. The blue round points above the X-axis are correct answers returned by DRQA system with maximal cosine similarity with WEEAT; The round points under the X-axis are false answers found by DRQA system with maximal cosine similarity. The absolute value is the similarity. The green triangles stand for the top-1 results of DRQA system, where the upper parts are correct answers and the low parts are wrong answers. 0.25), which indicates that we could remove some wrong answers according to this criterion.

Moreover, Figure 5.3 shows that there are around 50% top-1 answers having zero similarity with question LAT. This could be caused by the out-of-vocabulary problem of word embeddings such as short answers with specific words that have never appeared in the training corpus.

For the round points below the X-axis, they also present an important similarity (around 0.5) correlation with WEEAT, which means that by simply selecting the answer with highest similarity as the best answer is not an effective strategy. Indeed, when we used this re-ranking strategy to select one answer from DRQA candidate answers, the strict accuracy with respect to the annotated gold standard decreased 5.2 Expected Answer Types (EAT) and their importance in Question Answering models 123 from 38% to 33%. Again, the missing word embedding for correct answers has a strong impact on this results.

The observations above show that a fine-grained study of word embeddings is important for biomedical QA systems and missing vocabulary for biomedical terms might cause performance degradations. To determine the importance of semantic types inferred from the lexical answer types from question (SGEAT) in answer words, we studied if the SGEAT types are present in the answers. We report this on three datasets, one being the Gold standard answers in BIOASQ and other two being the correct and wrong predicted answers of DRQA system (top-1).

Table 5.7 shows the count of matches of SGEAT and semantic types from answers. It is clear that many correct answers (gold standard -40/59) have a matching SGEAT.

For DRQA outputs, we compute how many correct and wrong top-1 answers has a matching SGEAT. From the reported findings, there are more correctly answered DRQA outputs (18/23) with matching SGEAT than the wrong ones (16/36) which signifies that the model already captures this information correctly in most of the cases. But 44% (16/36 questions) of wrongly answered questions have the matching Expected Answer Types, which means that this feature is useful to improve wrongly answered questions.

We studied different representations of the LAT words and EAT, based on structured taxonomy or word embeddings, and showed a correlation with the correct answers.

When comparing with the answers provided by our QA model, we observe that the wrong answers might be rejected by adding a criterion when the answer types do not match.

Verification of the Expected Answer Types in Open Domain

In our study of the open domain, we analyze the SQUAD dataset by (Rajpurkar, J. Zhang, et al., 2016) for the presence of Expected Answer Types, which is based on Wikipedia data. For analyzing this data, we need LAT and EAT information on the SQUAD dataset. The authors of (Madabushi and M. Lee, 2016) built a rule based model which is highly accurate (97.2% in their evaluations) in determining the question classes for input questions in a taxonomy defined in their work. We obtain these annotations on SQUAD dataset questions upon request which we use for the study of EAT for open domain.

We have created a simple taxonomy by using the higher level taxonomy from their work to a version which is suitable for our work as shown in the Table 5.8. In open domain data setting, the named entity types are mainly used to define the Expected Answer Types (EAT). A taxonomy must be defined for mapping named entity types to Expected Answer Types (EAT) of the questions. Table 5.8 shows the taxonomy we used. The annotations given by (Madabushi and M. [START_REF] Madabushi | High accuracy rule-based question classification using question syntax and semantics[END_REF] contain high level EAT mentioned in the table 5.8 on the left. The right side values "spacy annotated tags" are the tags which are annotated by named entity recognition tool Spacy16 which the answer words belong to. The mapping determines the which named entity type belongs to which Expected Answer Type (EAT). Since ABBR -Abbreviations, DESC-Description cannot be inferred from the named entities as they refer to textual phrases and not exclusively entities, we exclude these from the analysis. These types are present in Figures 5.4 and 5.5 because the gold standard data were not inferred using the named entity types.

EAT

The annotations on SQUAD dataset by (Madabushi and M. Lee, 2016) contained the following numbers of EAT as shown in the figures 5.4 and 5.5 across 6 different categories of EAT. We annotate answer spans using the spacy named entity recognition tool and map it with the EAT taxonomy as highlighted above to check how many of them match correctly. For the dev set: There are 8026 questions with EAT annotations (subtracting 24 ABBR and 1556 DESC tagged questions from 9606 total questions). 4029 questions out of 8026 match according to the EAT annotations, i.e. 50.19%. For the train set: There are 66,659 questions with EAT annotations (subtracting 354 ABBR and 11727 DESC tagged questions from 78,740 total questions). 31,924 questions out of 65,419 match according to the EAT annotations. i.e. 48.79%.

The above analysis shows that EAT match happens for 50.19% of questions on dev set and 48.79% of questions on train set. The non matching questions might be because of several reasons:

• Correct Expected Answer Type not being recognized by the rule based system annotations of (Madabushi and M. Lee, 2016).

• Non recognition or wrong recognition of certain entities and their types in the answers from the named entity recognition tool Spacy.

• Wrong taxonomy mapping for certain types. For ex: NORP named entity refers to HUM, LOC and ENTY EAT types.

From the scores reported above, we observe that it is difficult to use EAT as a strong feature for selecting an answer from the paragraph.

Experiment using QA model

In order to understand how many wrongly answered questions correspond to their Expected Answer Type (EAT), we experiment using QA model predictions. We only consider the questions which were wrongly answered to study if the predicted answer type corresponds to the correct expected answer type from the question.

For this set of experiments we use DRQA model on Reading Comprehension task. We train on the official SQUAD train set and predict on the official SQUAD dev set. We analyze these wrong answers to check if they match the Expected Answer Types from the questions.

The official dev set has 10,570 questions and 3,242 questions (30.67%) are wrongly answered by DRQA model. Out of the 3242 questions only 721 questions (22.23%) has an EAT from the annotated set provided by (Madabushi and M. [START_REF] Madabushi | High accuracy rule-based question classification using question syntax and semantics[END_REF] which matched the named entity inferred answer type.

This signifies that the QA model predicted only a few number of answers (22.23%) in the wrongly answered set which matched the corrected EAT but failed to capture the right entity as the answer. This shows that there is a scope for highlighting entity types as 77.77% of wrongly answered questions (2,521 questions) did not have a matching EAT. Since 22.23% did not have correct answers in spite of their matching answer types, shows that the entities were not correctly recognised.

5.2 Expected Answer Types (EAT) and their importance in Question Answering models

In the following section we address these issues by highlighting EAT in the paragraphs and using EAT in the QA models.

Using Expected Answer Types and

Embeddings for the Answer Sentence

Selection task

Expected Answer Types (EAT) is one of the vital information which is important for question answering systems to detect which type of answers do the questions require and filter out the less important answer candidates. We hypothesize that this information can be explicitly highlighted and used in the existing models to improve performance.

In the Answer Sentence Selection task, a question and a set of sentences are provided and the goal is to find which sentence is the correct answer to the question. There can be multiple correct answer sentences per question. Our intuition is that when the type of an entity (among several other entities in the sentence) present in the sentence corresponds to the Expected Answer Type (EAT) from the question, this sentence is more likely to be the correct answer sentence than a sentence that does not contain such an entity type. Our goal is to better rank these kinds of sentences. We proceed by highlighting, in the paragraphs, entities that correspond to the EAT. We explain the process in detail in the following sections.

Our contributions in this regards are as follows. We introduce two different ways of using Expected Answer Type EAT. We use a simple model of a recurrent neural network which uses a pre-attention mechanism. We experiment with several datasets along with TrecQA to determine if this would work better for a wider range of large scale datasets. To annotate other datasets with EAT information, we propose a multiclass classifier model which is trained on a dataset built by using an existing rule-based system which predicts EAT for questions.

Highlighting Single Entity and Multiple Entity Types

An answer sentence contains several named entities and some of them correspond to the Expected Answer Type (EAT). Highlighting these entities using their type might help in improving the QA system performance. The authors of (Tayyar [START_REF] Madabushi | Integrating Question Classification and Deep Learning for improved Answer Selection[END_REF] propose a method for replacing words by special token embeddings for highlighting entities that catch the EAT entity in sentences. In our work, this method is referred as "EAT (single type)" in the following experiments. The entities belong to (HUM, LOC, ABBR, DESC, NUM or ENTY). HUM refers to a description, group, individual, title. LOC refers to city, country, mountain, state. ABBR refers to abbreviation, expansion. DESC refers to a definition, description, manner, reason. NUM refers to numerical values such as code, count, date, distance, money, order etc. ENTY refers to a numerous entity types such as animal, body, color, creation, currency, disease etc. More details regarding the taxonomy can be found in (Madabushi and M. Lee, 2016).

The entities, irrespective of which class they belong to, are treated similarly by replacing them by two special tokens entity_left for entity occurrences and max_entity_left for maximum occurring entity that corresponds to an entity that is at least twice the number of occurrences when compared to the second maximum occurring entity. Entity types are recognized using a named entity recognition tool. When an entity type in a sentence matches the EAT from the question, entity_left token is used to replace the entity mentions in the sentences; same applies for the maximum occurring entity token max_entity_left as well.

Our proposition is to replace an entity according to the type it belongs to, instead of replacing all kinds of entity by just one word i.e. entity_left. We do it based on the different types of EAT it belongs to based on the taxonomy used in the original work. The intuition behind this method is that the model would learn to better map the relations between question words and specific entity type tokens when used in a model with attention mechanisms, rather than learning the relation between question words and the same generic entity type token for all entities. This way, we can learn a different behaviour with an entity about location and with an entity about a person for example.

Answer Sentence Selection Model -RNN-Similarity

The answer sentence selection task is a question answering task which is also referred sometimes as Sentence Reranking or Sentence Ranking task. It involves ranking a set of sentences S = {S 1 , ....., S m } for a given question Q, so the correct answer sentences are ranked higher. Sentence set S can contain the mixture of both negative and positive sentences relevant to the question, often more than one positive sentence.

We model this task as a pairwise similarity scoring task. For each sentence related to a question, we compute a similarity score against the question sentence and answer sentence. i.e., ( Recurrent neural networks such as LSTMs and GRUs are widely used in several NLP tasks like machine translation, sequence tagging, and question answering tasks such as reading comprehension and answer sentence selection. We propose a simple model with recurrent neural networks and an attention mechanism to capture sequential semantic information of words in both questions and sentences and predict similarity scores between them. We refer to this model further in this article as RNN-Similarity model whose code is available online17 . Figure 5.6 shows the architecture of the model.

Q i -S i,j , Q i -S i,j+1 , Q i -S i,j+2 , ....Q i -S i,j+n ).
E(Q) = {E(q 1 ), .., E(q m )} (5.1) E(S) = {E(s 1 ), .., E(s n )} (5.2)
A pre-attention mechanism captures the similarity between sentence words and questions words in the same layer. For this purpose, a feature Falign shown in Equation 5.3 is added as a feature to the LSTM layer.

Falign(p i ) = Σ j a i,j E(q j ) (5.3)
Where a i,j is,

a i,j = exp (α(E(s i )) • α(E(q j )) Σ j exp(α(E(s i )) • α(E(q j )) (5.4)
which computes the dot products between nonlinear mappings of word embeddings of question and sentence.

The above process is similar to the DRQA model by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] and PSPR model by (Y. [START_REF] Lin | Denoising distantly supervised open-domain question answering[END_REF] for both Paragraph Selector and Paragraph Reader who use LSTMs to encode Question and Paragraph words along with the pre-attention mechanism as shown in Equation 5.3 . We use 3-layer Bidirectional LSTM layers for both question and sentence encodings.

{E(q 1 ), .., E(q n )} = Bi-LSTM({ Ẽ(q 1 ), .., Ẽ(q n }) (5.5)
{E(s 1 ), .., E(s n )} = Bi-LSTM({ Ẽ(s 1 ), .., Ẽ(s n }) (5.6)
The LSTM output states are further connected to a linear layer and a sigmoid nonlinear activation function is applied on the output of the linear layer which outputs the score ranging between 0-1, which signifies the similarity between the question and the answer sentence.

We implement the RNN-Similarity model in Pytorch, and we use MSELoss (Mean Squared Error loss) to minimize the error of predictions for relevance scores. The code for the model along with default hyperparameters is publicly available on Github 18 . The QA task is presented in section 5.3.2 in detail. The model's hyperparameters remain the same for all input settings. Only the modified input text for question and paragraph as shown in section 5.3.1 is the change which shows differences in performance. We use adamax optimizer and keep the missing words as zero vectors. We create a random word embedding ranging between (-0.5 -0.5) with dimension D for each of the EAT words and encode the word with this embedding when it appears in all our experiments. Tab. 5.9: Three methods for replacing entities along with an example from TrecQA dataset

Highlighting Process

The example in Table 5.9 shows the highlighting process. The row 1 is the original text data from the dataset. The row 2 is the modification done by [START_REF] Madabushi | Integrating Question Classification and Deep Learning for improved Answer Selection[END_REF] to highlight maximum occurring entity and the EAT matching entity. The row 3 and row 4 are our modifications. The row 3 refers to an example that has an EAT as "HUM" from the taxonomy, so we replace it as entity_hum. We do the same for other expected answer types such as entity_loc for " LOC" type, entity_enty for " ENTY" type, entity_num for "NUM" type, entity_desc for "DESC" type, entity_abbr for "ABBR" type. We replace the entity mentions in the text whose types are matching the EAT from questions same as the authors of (Tayyar [START_REF] Madabushi | Integrating Question Classification and Deep Learning for improved Answer Selection[END_REF], but we use separate entity highlighting according to their type, instead of using entity_left for all entities.

We also experiment with a variant where the max_entity_left is replaced with the entity type along with other entities as done in row 4 of Table 5.9. If the maximum entity is of type "HUM", we replace it as max_entity_hum. This method is referred to as "EAT (MAX + different types)" in the following experiments.

Prediction of the EAT

In order to experiment on the datasets without EAT annotations, we had to develop our own annotation tools to annotate any dataset questions with EAT entities.

Since SQUAD-EAT (see section 5.3.5) is the result of a rule-based method with a high accuracy score (97.2% as reported in (Madabushi and M. Lee, 2016)), we use it to train a multiclass classifier based on a CNN model for text classification19 by [START_REF] Kim | Convolutional Neural Networks for Sentence Classification[END_REF], by modifying the outputs into a multi-class setting. We use the original CNN model by [START_REF] Kim | Convolutional Neural Networks for Sentence Classification[END_REF] built for binary classification of sentiments because it captures better semantic information from text than traditional ML models as pointed in their work which also uses word embeddings from Word2vec (Tomas [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] as input for their task. We further refer to our modified model as EAT Classifier. We use 300 dimensions GloVe embeddings by [START_REF] Pennington | Glove: Global vectors for word representation[END_REF]. The output classes of the classifier refer to a type based on the taxonomy such as ABBR, DESC, ENTY, HUM, LOC, NUM and a "NO_EAT" class to signify an EAT which is not in the above list of classes.

We train the multi-class classifier model using the SQUAD-EAT dataset which gets an accuracy score of 95.17% on the SQUAD-EAT dev in our experiment, according to the annotation done by [START_REF] Madabushi | Integrating Question Classification and Deep Learning for improved Answer Selection[END_REF] as reference. We release the code for the EAT Classifier and the SQUAD-EAT dataset on Github20 .

Below is an example from SQUAD-EAT with HUM: Question: Which NFL team represented the AFC at Super Bowl 50?

Expected Answer Type: HUM.

Experiments and Results

Datasets

We experimented on the TrecQA dataset which is a standard dataset used to benchmark state of the art systems for answer sentence selection task. The authors of (Tayyar [START_REF] Madabushi | Integrating Question Classification and Deep Learning for improved Answer Selection[END_REF]Madabushi and M. Lee, 2016) provide the EAT annotations for the TrecQA dataset based on their rule-based approach.

We modify the QA dataset SQUAD (Rajpurkar, J. Zhang, et al., 2016) designed for machine comprehension, into an answer sentence selection dataset to provide the answers in their original context. We name it as SQUAD-Sent. We do this by processing the dataset where each example is usually a triple of Question, Paragraph and Answer span (Text and the answer start offset in the paragraph) into a dataset where each triple is a Question, Sentence and Sentence label. The sentence label is 1 if the answer is present inside the sentence, else it is 0. We perform sentence tokenization using spacy toolkit 21 on paragraphs of SQUAD and perform a check for an exact match of answer strings in them. SQUAD-Sent is a special case dataset where there is just one positive sentence per question and the other sentences are negative examples. The motivation to do this is that we hypothesize that the Reading Comprehension task on SQUAD dataset might perform better when the input paragraph is a single sentence with an answer, instead of a paragraph with multiple sentences. Since the answer is present in only one sentence in a paragraph, reducing the paragraph size might increase the probability of finding the correct answer in the paragraph. In order to obtain the single sentence with an answer we can rely on Answer Sentence Selection task. Also since SQUAD dataset is a large scale human annotated dataset, we decided to modify this for the Answer Sentence Selection task.

For the expected answer types of SQUAD questions, we use SQUAD-EAT which is a dataset with EAT annotated questions on SQUAD v1 dataset questions which is annotated by the authors of (Tayyar [START_REF] Madabushi | Integrating Question Classification and Deep Learning for improved Answer Selection[END_REF]Madabushi and M. Lee, 2016) on our request.

WikiQA dataset by (Y. [START_REF] Yang | Wikiqa: A challenge dataset for open-domain question answering[END_REF] is another dataset used for answer sentence selection task which was built using Bing search engine query logs. We use a preprocessed version as used by [START_REF] Rao | Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks[END_REF] which has removed certain examples without any positive answers and questions with more than 40 tokens to compare the scores. The questions and answer sentences are annotated with EAT information as described in section 5.3.4.

Annotation of the entities in paragraphs

We detect the entities in the sentences using Dbpedia Spotlight tool by [START_REF] Daiber | Improving efficiency and accuracy in multilingual entity extraction[END_REF]. The detected entities by the spotlight are verified for their entity type match using the Spacy NER tool which is mapped to EAT using the mapping shown in table 5.6. Only the matching entities are highlighted and others are discarded. We also try using a named entity recognition tool to annotate entities in the text directly without using Dbpedia spotlight. Spacy22 was used to annotate plain text and the entities matching the EAT were annotated. Table 5.10 shows the statistics of the datasets with EAT annotated questions and plain word level questions (regular datasets) and the number of entities annotated in each set. EAT version of TrecQA dataset is as reported in (Tayyar [START_REF] Madabushi | Integrating Question Classification and Deep Learning for improved Answer Selection[END_REF] and available through this link23 . SQUAD-Sent was annotated by the authors of (Madabushi and M. Lee, 2016) upon our request. Wiki QA dataset although has high number of questions with a predicted EAT (the rest are NO_EAT class), the percentage of entities in the paragraphs which match the EAT are very low. This can be because of three reasons as explained earlier, 1) Undetected entities in the paragraph. 2) Wrongly predicted EAT. 3) Wrong taxonomy mapping.

Dataset

Results

Table 5.11 shows various results on different versions of datasets. Note that the questions in the following experiments of Table 5.11 contain all the questions from the datasets, which includes questions which are highlighted with EAT and questions which are not highlighted with EAT as well.

TrecQA

The current state of the art system is by [START_REF] Madabushi | Integrating Question Classification and Deep Learning for improved Answer Selection[END_REF] that uses EAT on word level model of [START_REF] Rao | Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks[END_REF]. Henceforth both results are presented.

Our model RNN-Similarity on plain word level data fetches better result than the model of [START_REF] Rao | Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks[END_REF] et al., 2018) where the MAP and MRR scores are higher for correct sentences being ranked as Top-1. In this way, we obtained the best accuracy scores by integrating the occurrence of the EAT in the answer sentences.

WikiQA

WikiQA dataset was annotated by our EAT-Classifier model which predicts EAT for the questions. NO_EAT predicted type is excluded from the experiments. A recent model by [START_REF] Tymoshenko | Cross-Pair Text Representations for Answer Sentence Selection[END_REF] which uses kernel methods outperforms all the scores of our model. We note that the performance on our EAT level models is higher than the ones on plain words.

SQUAD-Sent

SQUAD official test set is hidden to the public users. Although the difference between word level and EAT word level is little, the difference highlights the fact that the entity words replaced in the sentence would not worsen the performance of the systems; instead it improves it subtly. We would like to note that the MAP and MRR values were the same because of the existence of just 1 positive sentence amongst other negative per question. Hence we only report MRR on this dataset. Plain words - [START_REF] Rao | Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks[END_REF] performance is obtained using the implementation available online24 , which we experimented on SQUAD-Sent dataset.

One aspect to be highlighted is that the implementation 24 of word level model by [START_REF] Rao | Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks[END_REF] • WikiQA (EAT): We remove the questions with 'NO-EAT' class which were 23 questions overall. The results are better with EAT (single type) which shows that the method works well in certain cases better than different types of EAT as in the above two datasets.

The results reported in table 5.12 show that there is not a high improvement over different methods when trained only on questions with EAT information. Henceforth it is better to train models with the entire dataset and highlight EAT information only when the question contains the EAT information.

Conclusion

We report our findings on TrecQA, SQUAD-Sent and WikiQA dataset performance and show that we outperform state of the art results on TrecQA dataset25 by the two different ways of highlighting Expected Answer Types in the data compared to the plain text input.

The Expected Answer Types are a useful piece of information that used to be extensively exploited in the traditional QA systems. Using them with the current state of the art DNN systems for Answer Sentence Selection task improves the system performance. We propose a simple model using recurrent neural networks which works robustly on three different datasets without any hyperparameter tuning and annotate entities belonging to the expected answer type of the question. Our model outperforms the previous state of the art systems in the answer sentence selection task on TrecQA dataset. We also propose a model to predict the expected answer type based on the question words using a multiclass classifier trained on a rule based system's output on a large scale QA dataset.

In the above set of experiments, we observed that highlighting Expected Answer Types (EAT) in the sentences works better in the Answer Sentence Selection task.

Therefore we proceed to experiment on the Reading Comprehension task, whose goal is to extract short answers from the answer sentences, by highlighting the entities.

Using Embeddings of Entities for the Reading Comprehension task

In the previous sections we discuss about highlighting entities with respect to Expected Answer Types for Answer Sentence Selection task. In the case of Reading Comprehension task the same technique cannot be applied in a straightforward manner because the answer tokens or words are substrings of the answer sentences and if an answer word is an entity then replacing it with generic entity words such as entity_left or entity_hum etc. will result in loss of information during evaluation. Henceforth a different approach with the same intuition is carried over in this set of experiments.

We propose to take into account entities using entity embeddings for Reading Comprehension task. Usually QA systems use pre-trained word embeddings trained using Glove [START_REF] Pennington | Glove: Global vectors for word representation[END_REF]), Word2Vec (Tomas Mikolov, Sutskever, et al., 2013) or Fast-text models (Tomas [START_REF] Mikolov | Advances in Pre-Training Distributed Word Representations[END_REF] where entity words, stop words, other words are considered. Using special entity embeddings for entities have been shown to have better performance in certain state of the art methods for evaluating word embeddings [START_REF] Sherkat | Vector embedding of wikipedia concepts and entities[END_REF], (Dhingra, H. Liu, et al., 2017), (Kamath et al., 2017b) and also in the above for Answer Sentence Selection task where entities are represented with their type as a special embedding.

Often in the case of Reading Comprehension task the entity themselves play a role in being the answer tokens therefore the same technique used in the above section cannot be used here. Each entity has to be represented by its own embedding. In order to learn specialized entity embeddings for the entities we chose to learn word and entities embeddings (by highlighting entities during embeddings pre-training) in the same embedding space. The intuition behind learning a representation of words and entities is that the textual data sources such as Wikipedia dumps, news data, common crawl data etc. often contain single or multi-word entities along with regular words which can be used to represent entities in a special manner along with other non-entity words. For example, phrases "New York City" and "New Orleans" contain the word "New" which has the same word embedding in both the phrases. Our work shall use "New York City" as one single word embedding as it is a multi word entity, same for "New Orleans" which will be a single word embedding. Our hypothesis is that, doing so would improve sentence and entity matching and a better association between the LAT word of the question and the entities without any supplementary annotations required.

We describe our solution for encoding the information related to entities in a Reading Comprehension QA model and show that using entity embeddings with word embeddings outperforms certain methods which use only word embeddings on factual questions.

Motivation

Open-domain question answering systems often consists of question-answer pairs where the answers are entities. As shown in Table 5.13 & 5.14, there is a high number of questions whose answers are entities and their paragraphs contain 10 entities on average. Question answering systems input tokens in general do not distinguish between regular words and entities. Apart from using input features such as named entities, pos tags etc. the input tokens still remain at word level or character level. We propose to use a joint representation of words and entities together in the text to better represent entities. Multi-word entities (an entity described using more than 1 word) are clubbed into one single entity token.

Data

Joint Entity and Word embeddings

Concatenating a phrase of entity words into one token makes it easier to represent multi-word entities like European Union Commission and European Union Law into a richer contextual representation to distinguish between different entities. The QA system then will have to retrieve a single entity token rather than an N-gram of word tokens.

Encoding symbolic information such as knowledge bases with text documents has become a reliable way to work with such information. Word and entity embeddings are widely used nowadays in several NLP tasks. The former have shown that it is a great tool to capture both semantic and syntactic information about words (Tomas [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. Several works have been proposed for their combination [START_REF] Fang | Entity Disambiguation by Knowledge and Text Jointly Embedding[END_REF][START_REF] Yamada | Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation[END_REF] and we opt for the recent EAT algorithm proposed by [START_REF] Moreno | Combining word and entity embeddings for entity linking[END_REF] for its simplicity. This work jointly learns words and entities (Entity + Word) in a unique embedding space (EW-Emb) and restrict the entity usage to Wikipedia pages only. The underlying idea is to exploit entity mentions, or anchor texts, within Wikipedia pages to calculate the entity embeddings following (Tomas [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. When an anchor text is found in a window, EAT processes two times the same window, one for the word and another for the entity. In that way word embeddings are not downgraded and unambiguous entity embeddings are learnt. However, extra pre-processing steps are needed to ensure that anchor texts are not removed, entity entries in the vocabulary are identified, and entity entries are normalized to avoid redundancy (e.g., by applying all possible redirections in Wikipedia). These embeddings are used as input for a simple question answering system. Our implementation is based on an optimized version of Word2Vec using TensorFlow26 with parameters (embedding size = 200, learning rate = 0.025, 5 epochs, Skipgram configuration, and window size of 5). The obtained vocabulary is composed by more that 5.2M entries including 1.8M entities.

The QA system

The QA system used in this set of experiments is DRQA by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] whose model is explained in Section 4.1.1. The model remains exactly the same but we have made certain changes for the input words.

We transform the SQUAD dataset which is built using Wikipedia dumps as shown in the example below

Ent.A: Wikipage_European_Commission

At input, we use word and entities embeddings for input tokens. The multi-word entities become a single token because of entity embeddings. This results in change of offsets and answer text for the answers in the paragraph which causes issues for Exact Match evaluations as done by Reading Comprehension systems. Therefore during the preprocessing time we build a mapping between multi-word entities to their individual word offsets in the plain text paragraphs to use this during the evaluation to perform evaluation on the official dev set answers provided by SQUAD. The evaluation however is the official evaluation of SQUAD task based on plain text entities.

An example of a QA pair is shown above, first box represents a regular QA pair, and second box represents a QA pair after pre-processing with entities. Also note that in this setting, the answer is always an entity. Offset refers to character level offset to the start of the answer span. Mapping refers to the entity answer in plain text which is represented by both start and end character offsets which is used during evaluation.

We chose SQUAD data for our experiments because it is built using Wikipedia articles, which are also used to train our embeddings. To annotate entities in SQUAD data, we use DBpedia spotlight [START_REF] Daiber | Improving efficiency and accuracy in multilingual entity extraction[END_REF], an off-the-shelf tool for annotating mentions of DBpedia resources in text. We use redirections file from dbpedia27 to use the latest Wikipedia page titles for the entities after the redirections.

It was not suitable to find the wikipage for SQUAD paragraphs and use entities from the wiki dump because Wikipedia does not annotate all entity occurrences in 3) Fast text vectors Paragraphs with words (W) and paragraphs with Entities + Words (E+W) on entity annotated SQUAD dataset questions and also on the whole SQUAD dataset questions. Table 5.16 represents the results on Whole SQUAD dataset questions and 5.17 represents the results on the entity annotated SQUAD dataset questions.

One of the goal is to evaluate the performance of the system on extracting entities from the text. We evaluate with exact match and F1 scores on the retrieved results. Table 5.16 shows that using entity annotations and embeddings causes the results to fall by around 1% on the whole SQUAD dataset in comparison with EW-Emb embeddings. This might happen because the answers often are phrases and are not tagged as entities. However the EW-Emb embeddings space does not perform well when used on only words, compared to Glove embeddings space. It is then difficult to get a true conclusion about using entity embeddings on this dataset.

In Table 5.17, the questions whose answers are entities perform much better with entity annotations and embeddings by 7% increase in performance over Glove embeddings. The Entity+Word representation performs the best with EW-Emb space because of presence of entity tokens with entity embeddings.

In SQUAD dataset, the answers are textual phrases sometimes containing just one entity (like in Entity QA setting) and sometimes containing an entity along with other surrounding contextual words which might not be entities but stop words, pronouns etc. (whole dataset setting). In some cases even short answers which are entities are not annotated by named entity recognition tools. Because of this phenomenon, Entity+Word embeddings perform better only when the answers are single entities which are short answers. Longer answer phrases are better answered with only word embeddings.

Conclusion

Entities are widely present in open domain textual corpus and using a jointly learnt embedding space benefits in better performance of QA with entities. We have reported our findings of using such embeddings with an open domain QA dataset which we have annotated with entities. Results show that the entity embeddings improve retrieval of entity answers over the regular word embedding spaces when the answers are entities. The performance does not improve but gets worse when the entities are not present in the answer spans.

In the above sections, we have discussed several ways of improving QA models which are mainly based on neural networks. These models focus only on getting better performance on their Top-1 accuracy scores. In the following section, we propose to post-process the output of neural network based QA models to further improve accuracy using semantic features from different paragraphs relative to a question. To understand the importance of ranking Top-K predictions on the performance, we analyze Top-5 predictions on obtained from a QA model on a open domain dataset and a biomedical domain dataset. We first present the open domain analysis on SQUAD dataset (Rajpurkar, J. Zhang, et al., 2016) consisting of only wrongly answered questions based on the predictions by the DRQA model [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF]. We consider only the wrongly answered set to evaluate how many answers would have been correctly answered in the Top-5 predictions. Out of 3242 wrongly answered questions, 1433 questions had correct answers in Top-5 predictions. 44.2% of wrongly answered questions had a correct answer in Top-5. SQUAD dataset has 5.5 Improving the QA Performance using Semantic and Structured Resources in a Ranking model 10,570 total dev questions, 13.55% (1433/10,570) of questions which are wrong in Top-1 predictions, contained an answer in Top-5 predictions.

We report a similar analysis on BIOASQ 5B [START_REF] Tsatsaronis | An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition[END_REF] dataset on the wrongly answered questions. We use DRQA model pre-trained on SQUAD dataset and fine-tuned to BIOASQ 5B dataset. Out of 98 wrongly answered questions in the overall test set, 35 questions had correct answers in Top-5 predictions. 35.71% of wrongly answered questions had a correct answer in Top-5. BIOASQ 5B dataset has 150 overall test questions (from 5 batches), 23.33% (35/150) of questions which are wrong in Top-1 predictions, contained an answer in Top-5 predictions.

The above statistics show that there is a scope of improvement to score better on Top-1 using only the Top-5 predictions from the model. We experiment with other values of Top-K by modifying K for ranking experiments. Only for the above analysis we use Top-5 predictions.

In the case of Open Question Answering task, due to the availability of several paragraphs which might contain the answer, the answer candidates might end up overlapping among different paragraphs answers. This overlap feature between these paragraphs has proven to be highly reliable for ranking answers in feature based QA models [START_REF] Grappy | Selecting answers to questions from Web documents by a robust validation process[END_REF]. The same kind of data (Open Question Answering task data with several paragraphs) is present in the BIOASQ task where the answers are contained in more than 1 paragraph and sometimes there are zero answers.

In order to take into account, the possibility of answer overlap among different paragraphs along with the other explicit semantic features we presented in the preceding sections, we define a set of features which are detailed in the section below and build ranking models based on Top-K answers predicted by the OpenQA system.

Modelling

In this study, we intend to rank the predictions of the chosen OpenQA model, therefore we present in detail its overall process.

The Paragraph Reader model extracts answer spans as shown in the DRQA model [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF]. The model gives two probabilities (one for start and one for end token given by two classifiers) as described in equation The softmax operation in equation 5.10 is applied over total number of paragraphs per question therefore a probability value is predicted for each paragraph.

Combining the two probabilities (from eq.5.9 and eq.5.10), the overall answer (Top 1) is chosen by choosing the highest probable answer from P r(a|q, P ) for a question q which is calculated as : P r(a|q, P ) = p i ∈P P r (a|q, p i ) P r (p i |q, P ) (5.11) To perform ranking over Top-K predictions, we need K predictions for a pair of question and a set of paragraphs. We use K as a hyperparameter and test with different number of K.

The model of PSPR returns a Top-1 answer based on the maximum combined probability (equation 5.11) computed by multiplying individual probabilities of paragraph (equation 5.10) and answer (equation 5.9) per paragraph and summed over all the paragraphs per question. In our case, we use Top-K answers using the same formula, instead of Top-1 we choose Top-K and choose paragraph which resulted in the prediction.

The goal is to improve the top 1 prediction by reranking the top K predictions of the model. We model the ranking task as a classification problem where only the top answer prediction of the training data is marked as True class and other top K -1 predictions are marked as False class.

For the classification task we use 1) Random Forests, 2) Adaboost, 3) MLP classifier implementation using scikit learn28 .

Features for classifiers

We define features for the classifiers which are 1) Single features -features computed over a single paragraph. 2) Collective features -features computed over several paragraphs. 3) Semantic features for dealing with EAT or LAT verification.

The input features to the classifier models are described below.

Single Features

• Answer Probability -as computed in Equation 5.9 is the probability of answer spans obtained from the reader model (top answers have higher probabilities)

• Paragraph Probability -as computed in Equation 5.10 is the probability of paragraph obtained from the selector model (top scored paragraphs have higher probability to contain an answer)

• Paragraph and Answer length in characters

• Answer words overlap with paragraph words -Ratio of answer words overlapping with paragraph words. Length of A ∩ P

Collective Features

• Maximum value of answer probability -Maximum answer probability for a prediction, over the number of paragraphs.

• Maximum value of paragraph probability -Maximum paragraph probability for a prediction.

• Answer presence ratio -Count of number of predicted answer occurrences across different paragraphs, which is divided by the number of paragraphs.

• Summation of answer probability -If an answer prediction is repeated several times coming from different paragraphs, their probabilities are summed into one value.

• Answer rank -Rank of the predicted answer according to the model.

Semantic Features

• EAT Match -Expected Answer Type from the question matching the answer's entity type -'0' when there is no EAT match, '1' when there is an EAT match and '-1' when there is no EAT predicted in the question.

• Cosine distance between LAT and answer -Semantic feature -Lexical Answer Type words from questions and Answer prediction words are averaged and checked for cosine distance between them in Glove word embeddings.

For the feature Cosine distance between LAT and answer, the LAT words are computed using a classifier built using a CRF model named Wapiti29 by [START_REF] Lavergne | Practical Very Large Scale CRFs[END_REF] to determine which words in the question phrase are the LAT words.

The task is a sequence labelling task where the labels are "1" or "0" to predict if a word belongs to a LAT or not. For the training dataset, we use the annotations provided by (Madabushi and M. Lee, 2016) which also highlights important words which contribute as LAT words.

For the feature EAT Match -we use the EAT Classifier explained in section 5.3.4, which predicts an Expected Answer Type for an input question. The model is trained on SQUAD EAT dataset provided by (Madabushi and M. Lee, 2016).

The same set of features (except the EAT Match) apply for both open domain and biomedical datasets. We define some additional ones for Biomedical domain as listed below.

• Lexical Answer Type UMLS semantic Type match -LAT word predicted using Wapiti is annotated with Metamap for UMLS semantic type which is checked for matching with Answer's UMLS semantic type.

• Lexical Answer Type UMLS semantic Group match -LAT word predicted using Wapiti is annotated with Metamap for UMLS semantic group which is checked for matching with Answer's UMLS semantic group.

• Lexical Answer Type UMLS CUI match -LAT word predicted using Wapiti is annotated with Metamap for CUI (Concept Unique Identifier) which is checked for matching with Answer's CUI.

The above features are marked as '0' when there is no match, '1' when there is a match and '-1' when there is no EAT predicted in the question.

Experiments and Results

Classifiers

We experiment with a few classifiers which takes as input, numerical features and predicts either True or False for input data. The scikit learn package provides several algorithms for binary classification, we experimented with 10 different classifiers mentioned in the post30 which compares different algorithms. We report results on top 3 better performing classifiers -Randomforests, Adaboost and MLP Classifier. Adaboost algorithm performs better than the other two models and beats the baseline score of PSPR model by 2%. The baseline scores is computed on the predictions taken by the model implementation provided on the author's Github31 page. The K value for Top-K predictions for the above set of experiments was 3.

Open Domain Data

Results shown in Table 5.19 show that choosing K value as 3 fetches the best result on Adaboost algorithm. The more K value is increased, the lesser the accuracy because of imbalance in True and False classes for Top-1 data which leads to more negative samples.

Feature ablation

In order to understand which feature is more important compared to the rest, we do a feature ablation test by using scikit learn feature importance plot and the list below represents the 5 most important features in decreasing order of magnitude. To understand the importance of input features, we remove some features listed in the 5 most important features from above, and run the experiment to compute the decrease in performance. Results shown in Table 5.20 highlight the importance of choosing right features to increase performance. These features are important and sufficient to model the complexity of the input. Removing some features penalizes the performance. The most impactful features according to the above experiments were Maximum value of paragraph probability and Answer presence ratio, removing these two features drops the performance by 1.87%.

BIOASQ dataset

BIOASQ task expects Top-5 answers per questions to evaluate Strict Accuracy (Top-1) and Lenient Accuracy (Top 5). Therefore in this case, we use Top-K as Top-5 for this purpose. That is why Lenient Accuracy remains the same in all experiments. In the previous set of experiments on open domain data in table 5.19, we found that choosing Top 3 answers performed with better results for Top 1 ranking, and the performance decreased upon increasing K value because of increasing number of negative answers in the data. Therefore we do not increase above 5.

In this set of experiments, the baseline system is the output from PSPR model evaluated using official BIOASQ evaluation. From the results it is clear that the Randomforest algorithm performs better when comparing with the rest. We evaluate this the same way do with QUASAR-T dataset by considering a dev set to check for the best hyper parameter and later apply it on the official test sets. The most impactful features according to the above experiments were Maximum Answer Probability and Answer words overlap with paragraph words, removing these two features drops the performance by more than 1% on average on all the three datasets. The semantic feature of LAT UMLS semantic type match has an impact on the classifier performance which shows a decrease in performance when removed.

Conclusion

With the above set of experiments, we try to rank the predictions obtained from a neural network to further improve the accuracy. In the case of open domain data and two sets of biomedical domain datasets, the traditional ML classifiers work better and improve the scores up by some margins, which gives an opportunity for improving results of deep learning models. Since there are different models for different individual tasks, different datasets with different task goals exist. In this section, we try to use existing datasets which were primarily built for a different task in a way that the pipeline approach would fit and show more realistic version of the task implementation on the datasets.

For example, SQUAD dataset by (Rajpurkar, J. Zhang, et al., 2016) assumes that the relevant paragraphs always exist for a question and the systems built on this dataset should comply to this assumption. But generally getting relevant paragraphs itself is a hard task which has been researched for decades. Therefore only Open Question Answering scenario is feasible as some paragraphs may not be relevant. Therefore we try to use SQUAD in such a way that it would fit as a Open Question Answering task.

OpenQA on SQUAD dataset

One way to use a Reading Comprehension task dataset as a OpenQA dataset is by completely ignoring paragraphs and using an IR engine to fetch the paragraphs and using it as input as done by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF]. In this case, the IR engine retrieval was not based on a machine learning model but used as an off-the-shelf tool. Another way to build an end-to-end open domain question answering model is to modify the retrieval phase as a weakly supervised task and build an end-to-end with a single large scale model as done by (K. Lee et al., 2019) using the BERT model.

In our work, we consider the SQUAD dataset and the paragraphs, and split the paragraphs into sentences. In this setting we will only have one sentence with a correct answer and all other sentences will be marked negative as they do not have an answer. We hypothesize that determining a relevant sentence and extracting answers from it will be better than extracting answers from a lengthy paragraph which contains irrelevant sentences to the question. We call this dataset as SQUAD-SENT dataset.

SQUAD-SENT dataset has been experimented on Answer Sentence Selection task in section 5.3. In this section we apply the SQUAD-SENT dataset on OpenQA task for answer extraction.

Model Accuracy SQUAD-RC by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] 69.5 SQUAD-SENT on PSPR 50.10 SQUAD-OPEN by (K. Lee et al., 2019) 26.5 SQUAD-OPEN by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] 27.1 SQUAD-OPEN BM25+BERT by (K. Lee et al., 2019) 28.1 The results presented in Table 5.23 are on different variants of SQUAD datasets but evaluated for the same answer extraction task as done in official SQUAD evluation setting. SQUAD-RC by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] is the DRQA model scores on the SQUAD dataset with correct paragraphs.

SQUAD-SENT is a sentence level dataset which contains 1 correct sentence with an answer and rest are wrong sentences. The results presented above on SQUAD-SENT dataset is on the model of PSPR by (Y. [START_REF] Lin | Denoising distantly supervised open-domain question answering[END_REF].

5.6

Applying the Open QA model on different datasets SQUAD-OPEN is an open setting where no paragraphs are given and the model itself uses some techniques to first retrieve a paragraph and then perform answer extraction. The results presented above by (K. Lee et al., 2019) uses a BERT model approach and learns the paragraph retrieval and answer extraction end-to-end using a weakly supervised approach. The DRQA model by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] on the SQUAD-OPEN dataset setting use an answer retriever and answer reader in cascading fashion. Their model fetches better result than BERT (K. Lee et al., 2019) on SQUAD-OPEN data setting. The authors of [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] implement their own IR model (based on TF-IDF weighted bag-of-word vectors and include bi-gram features) without using machine learning for fetching relevant paragraphs.

An important highlight from these experiments is that the performance of the stateof-the-art model BERT [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] on SQUAD v1.0 dataset is 87%, and the same model modified to SQUAD-OPEN dataset setting which does not provide a relevant paragraph to the model fetches 26.5% (K. Lee et al., 2019) on an end-toend OpenQA BERT model. It is the similar case even for the DRQA model [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] which fetches 69.5% on SQUAD v1.0 dataset whereas the same model fetches 27.1% on SQUAD-OPEN dataset setting. This shows the wide gap in performance drop when the relevant paragraphs are not provided to the models.

Conclusion

The three different approaches with the above results show how different models trained on SQUAD dataset (some in a realistic setting without always containing a positive paragraph) cripples upon not having always some positive samples. As discussed in Chapter 4 and highlighted by [START_REF] Talmor | MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension[END_REF], current stateof-the-art deep learning models for QA are often overfitting on the task datasets and failed to generalize. In this set of experiments we show that these task datasets which are designed with specific goals do not perform similarly when some assumptions do not apply.

End-to-end models for overall Question Answering (OpenQA) seems to under perform even when pre-trained models like BERT are used. This shows the complexity of the overall QA task (fetching relevant paragraphs and extracting answers together). Task specific models like Reading Comprehension tasks are simplified QA tasks whose performance cannot be directly compared to OpenQA tasks.

TrecQA as OpenQA task

TrecQA task had been a widely popular QA task in the past that has released several small scale datasets released during 1999-2004 that are in different formats. From 2007 onwards, (M. [START_REF] Wang | What is the Jeopardy model? A quasi-synchronous grammar for QA[END_REF] created and used these datasets for Answer Sentence Selection task and a lot of works32 followed after that used this dataset for the same task.

TrecQA task organizers originally released a regex pattern answer set for the QA datasets which were used to judge the correct answers automatically. We use this pattern set to obtain the answer spans for the paragraphs in the well known TrecQA-13 (2004) dataset by (M. [START_REF] Wang | What is the Jeopardy model? A quasi-synchronous grammar for QA[END_REF]. We use the same regex patterns to determine the answer span from the answer paragraphs. We create a dataset with answer paragraphs and answer spans from the paragraph if they are relevant, if the paragraphs are not relevant then there will not be an answer in it. We call this dataset as TrecQA-RC and we release this publicly33 . The number of questions and paragraphs are the same as the one created by (M. [START_REF] Wang | What is the Jeopardy model? A quasi-synchronous grammar for QA[END_REF] but contains a binary label signifying the relevance of the paragraphs along with added information.

We apply a QA model on this dataset to have a baseline score. Since TrecQA dataset has both relevant and irrelevant snippets, it is more suitable for OpenQA task to be applied, so we apply the model of PSPR on this dataset to report the performance.

Experiments and Results

TrecQA train dataset (∼1,000 questions) is relatively very small compared to the scale of QUASAR-T dataset (∼40,000 questions) which is another dataset of the same type. We experiment by training a model just on the TrecQA dataset and predict on the test set. We also experiment using the pre-training and fine-tuning approach by first training on QUASAR-T dataset and later finetuning on the TrecQA dataset.

In table 5.24, we compare the previous state-of-the-art model results which used non neural network approaches [START_REF] Severyn | Automatic feature engineering for answer selection and extraction[END_REF]Yao et al., 2013a) which report results on answer extraction task on TrecQA dataset without releasing the dataset for this task. Both the models fetch better scores by atleast 9% higher than the neural network models which we detail below.

Model Accuracy Sequence tagging model by (Yao et al., 2013a) 67.2 Tree kernels methods by [START_REF] Severyn | Automatic feature engineering for answer selection and extraction[END_REF] 70 The assumption of always having an answer in the paragraph in datasets like SQUAD v1.0 gave rise to a huge amount of DNN based models on Reading Comprehension task. We experiment with the same assumption by having only relevant paragraphs (paragraphs with answers) to perform answer extraction. We term it as the RC mode where the paragraphs always contain an answer. We use the model of DRQA on this dataset. As expected from the previous experiments above, the Reading Comprehension -RC mode fetches the better scores than OpenQA mode on the TrecQA answer extraction as all the snippets are relevant.

The above results highlight that using neural network models does not always necessarily fetch better results than the non neural network models.

Conclusion

Specific domains, like biomedical domain, have plenty of resources handcrafted or generated or annotated by human experts. Using this along with plain text corpus for question answering is a challenging task and is not trivial. In this chapter, we addressed one of our research questions on leveraging these expert curated knowledge sources and semantic information effectively into state-of-the-art question answering models on biomedical domain and open domain systems (with other types of information apart from free text) to improve their performances.

Our goal was to utilize the availability of different sources of information and tools to enrich the plain text datasets with additional information for question answering.

We believe that extracting features from text using knowledge sources and inputting them in different ways to deep learning models will have some positive impact.

Since deep learning models are not best suited for small scale datasets, results on methods involving pre-training and fine-tuning can further improve scores if domain specific information is used in correct ways.

We highlighted the problem of lacking answer variants in biomedical domain dataset BIOASQ and manually annotated answer variants to show the significant difference the same model fetches upon annotating the answer labels correctly. We also proposed a method to automatically generate these variants using Metamap tool which fetched similar or even better results in some sets compared to manually annotated answers.

We presented the use of Expected Answer Types in question answering by verifying the presence of it in the answers. In both biomedical domain data and open domain data, there is a scope of improvement of QA model performance because there is a significant amount of wrongly answered questions whose answer types match Expected Answer Types from questions.

We used the Expected Answer Types to highlight entities in plain text by using a special method of annotation and entity embeddings to input this information.

These methods for open domain works well for Answer Sentence Selection which scores better than merely on plain text data of TrecQA dataset. This method however works better on Reading Comprehension task only when the answers consists of entities and not long text phrases, the Reading Comprehension model performs better at finding the right answer entities on SQUAD dataset.

Since deep learning models almost always focus towards building end-to-end systems, not much emphasis is put towards post processing of the outputs to better rank the

Conclusion

Top-K predictions. We used some traditional machine learning binary classification models to rank a better answer candidate into Top-1 position and showed that there was a scope for improvement on the predictions from the neural network models. This applies both on biomedical question answering and open domain question answering models on Open QA task.

We found that some datasets like SQUAD which focuses on Reading Comprehension task performs poorly when the relevant paragraphs are removed and the task is modified into an OpenQA task. We applied OpenQA models on SQUAD dataset which is modified as a sentence level OpenQA task and showed the performance drop.

We created an Open QA version of TrecQA task by adding answer spans from the paragraphs which can be extracted using the answer patterns provided by Trec. We presented some baselines scores using PSPR model and Reading Comprehension models on this dataset.

Inspite of using state-of-the-art massive language models like BERT, the performance on OpenQA tasks such as SQUAD-OPEN as shown by the authors of (K. Lee et al., 2019) is very low (26.5 on the end-to-end BERT model). This highlights the low performance on overall QA task with current state-of-the-art models.

Our publications related to the work described in this chapter are listed below:

• use using semantic and structured information features from different paragraphs and showed that it improved the performance on the BIOASQ task.

In summary, to build a best performing model for BIOASQ QA task, we conclude that one can use multiple datasets in combination to pre-train the model, and fine-tune it on all answer variants which can be annotated automatically. The predictions from the above process can be further improved for better accuracy using the ranking model which uses semantic and structured information features at the post-processing phase.

In the following, we summarize the hybrid QA system details from both hybrid model and hybrid data point of view and give some future perspectives.

Building Models for Small Scale and Large Scale Datasets

The chapter 4 presents our work on building better models for question answering which suits both small scale and large scale datasets. The definition of Hybrid Data in this context refers to building models which are trained on different datasets from different domains to perform well on QA tasks.

Using deep learning models on small scale datasets will not fetch optimal results. One of our first goals was to adapt deep learning models to work on small scale datasets effectively. Therefore we began by choosing a model which worked well on open domain QA dataset and adapted it to biomedical domain. This process is called as domain adaptation. We formally defined domain adaptation which we use extensively in many experiments. We adapted a RNN based QA model which was built on open domain datasets such as SQUAD, towards biomedical dataset BIOASQ. We show the importance of doing domain adaptation by comparing results on models with and without domain adaptation.

According to this study, we believe that the domain adaptation will play a major role for small scale and domain specific datasets. Mainly because creating large scale datasets is not an easy task and it is more difficult and expensive when the data is domain specific. Domain adaptation and transfer learning are the two most important research disciplines to facilitate current open domain models to be used on small scale datasets and different tasks.

While performing domain adaptation, we came across another way of modelling the BIOASQ task, the OpenQA task, open domain question answering. It turns out that OpenQA is more suitable for BIOASQ task than Reading Comprehension task because of the nature of BIOASQ data which contains relevant and irrelevant paragraphs in the data relative to the gold standard answers provided. We compared two RNN based models on these two tasks and adapt these models to biomedical domain. We found that OpenQA model of PSPR better performed than the model of Reading Comprehension task, which was the DRQA model. We presented some factors which we considered as important before choosing a model to experiment and explained why we did some of our choices of models.

Several ways of modelling reading comprehension task have been proposed lately from works such as HotpotQA (Zhilin [START_REF] Yang | Hotpotqa: A dataset for diverse, explainable multi-hop question answering[END_REF] and DROP [START_REF] Dua | DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs[END_REF] which point out the simplicity of the original task defined by SQUAD dataset (Rajpurkar, J. Zhang, et al., 2016). Harder QA tasks and datasets create a demand for new models which focus on tackling these problems.

While performing above experiments, we wondered about pre-training the models on different datasets. Therefore we experimented with different Reading Comprehension datasets by keeping the same model architecture and hyperparameters. We showed the variability in the performance on downstream domain adaptation when the initial models are trained with different datasets. SQUAD 2.0 dataset by [START_REF] Rajpurkar | Know What You Don't Know: Unanswerable Questions for SQuAD[END_REF] was the best performing single dataset for pre-training the models. A combination of 4 Reading Comprehension datasets for pre-training performed the best on the fine-tuning for BIOASQ dataset. For the above experiments, we used BERT model by [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] which was the state-of-the-art model while experimenting. We also compared different pre-trained BERT models and found BIOBERT (J. Lee et al., 2019) to be the best model for biomedical domain adaptation.

More data leads to better performance has been proven for biomedical domain adaptation in our experiments. In the future, we believe that more datasets should be built by explicit mention of possible biases attached with the datasets to make more people aware about the consequences.

Towards the goal of building end-to-end models which do not use explicit features but perform feature extraction themselves, less emphasis is put on classical techniques which have addressed the fundamental problems of question answering and have proposed certain useful features and methods to handle certain types of data. More emphasis should be put on creating diverse datasets, careful analysis of bias, and most importantly highlight where the model fails to fetch correct answers which provides insights on improvements required for future work. But the research articles published in the recent times do not address any negative results from the models, and also seldom do error analysis.

Leveraging Structured and Semantic Information into Question Answering Models

The Chapter 5 presents our work on leveraging semantic and structured knowledge effectively into state-of-the-art question answering models on biomedical domain and open domain systems to improve their performances.

Our goal was to utilize the existing information from different sources and tools to enrich plain text datasets. Our hypothesis was that the deep learning based QA model can be further improved using semantic and structured information from external sources pertaining to the input text. The definition of Hydrid Data in this context refers to using open domain data with domain specific data in a domain adaptation process, plus integrating structured knowledge for annotating training datasets and for enriching the input data. The definition of Hybrid model in this context refers to an addition of a post processing reranker to account for structured knowledge and collective features obtained from different paragraphs.

In this regard, we highlighted the problem of lacking answer variants in biomedical domain dataset BIOASQ and showed a large performance difference when all the answer variants are annotated and when the model learns from such annotated data. We provided manual annotations for these answer variants and proposed an automatic method which uses UMLS meta-thesaurus to generate these annotations which fetched similar results. We release the manual annotations online2 along with the code to generate automatic variants3 .

In terms of future perspectives on the BIOASQ task and the dataset, we believe that it is important to focus research on providing better annotations, proposing guidelines for people from other domains, and performing fine grained analysis on the dataset instead of just focusing on building better models which improves accuracy.

We presented the use of Expected Answer Types in question answering by verifying the presence of it in the answers. In both biomedical domain data and open domain data, there is a scope of improvement of QA model performance because there is a significant amount of wrongly answered questions whose correct answer type match as of the Expected Answer Type. On open domain, we presented some statistics of SQUAD dataset questions containing Expected Answer Types as predicted by a model of (Madabushi and M. Lee, 2016) and on the biomedical domain, we presented some statistics on the BIOASQ dataset questions containing Lexical and Expected

Answer Types annotated by [START_REF] Neves | BioMedLAT corpus: Annotation of the lexical answer type for biomedical questions[END_REF] to support the above study on verification.

We use the Expected Answer Types to highlight entities in plain text by using a special method of annotation and special embeddings to input this information.

These methods for open domain work well for Answer Sentence Selection. We release the code for the model used in this experiment4 . This method however works better on Reading Comprehension task only when the answers consists of entities and not long text phrases. The Reading Comprehension model performs better at finding the right answer entities on SQUAD dataset. In the context of Expected Answer Types from input questions, we released the model 5 which predicts an Expected Answer Type for input questions.

In terms of future perspectives we believe that neural models must better integrate entity representations of knowledge coming from free text as well as from structured resources like ontologies, knowledge bases. Works such as [START_REF] Ferré | Représentations vectorielles et apprentissage automatique pour l'alignement d'entités textuelles et de concepts d'ontologie : application à la biologie[END_REF] align two different types of vector representations that capture part of their meanings with entities and text in the form of word embeddings and ontology concepts in the form of concept embeddings. The alignment is done in a supervised manner. This can be further extended towards complex tasks like question answering where the model integrates the knowledge from two sources while learning.

Since deep learning models almost always focus towards building end-to-end systems, not much emphasis is put towards post processing of the outputs to further improve predictions. We use some traditional machine learning models like randomforests, adaboost, multilayer perceptron etc. for binary classification to rank a better answer candidate into Top-1 position and show that there is a scope for improvement on the predictions from the neural network models. This applies both on biomedical question answering and open domain question answering models on Open QA task.

Using classical ML techniques to improve predictions from deep learning models are advantageous in the following ways: 1) Cheaper compute costs to improve predictions than train a complex model. 2) Faster inference times. These classical ML models might not be applicable for the overall QA task, but does improve scores when used on the outputs of a deep learning model to further improve scores.

In the analysis of [START_REF] Talmor | MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension[END_REF], the authors point out the overfitting of models to particular task and datasets they are trained. We find that some datasets like SQUAD which is focused on Reading Comprehension task performs poorly when the relevant paragraphs are removed and the task is modified into an OpenQA task. We apply OpenQA models on SQUAD dataset which is modified as a sentence level OpenQA task and show the performance drop. We create an Open QA version of TrecQA task by adding answer spans from the paragraphs which can be extracted using the answer patterns provided by Trec. We present some baselines scores using PSPR model and Reading Comprehension models on this dataset. We release this dataset online6 . An end-to-end model for the OpenQA task was first proposed by Lee et al. (K. Lee et al., 2019), which learns the retrieval and the extraction phase together. This is one of the first works which aims at building an end-to-end model for the overall QA task. The low results obtained using the latest state-ofthe-art model shows the real complexity of the overall QA task which cannot be easily addressed using a single end-to-end deep learning model including the latest pre-trained language models like BERT.

While we performed the majority of the above set of experiments, the contextual language models such as BERT by [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] had not been released. Ever since then, the large scale pre-trained language models (LM) have changed the way most of the NLP tasks are addressed today. Earlier models used CNNs or RNNs with attention mechanisms for many tasks which are now being replaced by these large scale LM models like BERT. We believe these new models have a large scope for various NLP tasks and the traditional ML models like the classifiers we use (randomforests, adaboost etc.) still hold good in certain cases where the dataset size is small scale and might require domain expertise.

An end-to-end OpenQA model (K. Lee et al., 2019) which uses these pre-trained contextual language models failed to obtain equivalent scores compared to LSTM based models that use a pipeline approach. This shows the complexity involved in building end-to-end models for overall OpenQA task. We believe that an hybrid modelling approach is the way forward to build better systems on overall question answering task.

4.6

Experiments with data containing longer contexts (Document level) by (J. Lee et al., 2019). DRQA is a Reading Comprehension model by [START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF]. BioBert-Unaltered is the original BIOASQ dataset with questions and paragraphs which contain answers. BioBert by (J. Lee et al., 2019) is the modified BIOASQ dataset where the paragraphs are longer paragraphs (documents from respective articles), where all the models are pre-trained on SQUAD v1.0 dataset and finetuned on BIOASQ dataset. SOTA scores are reported by (J. DeepQA scores are presented by (Wiese et al., 2017b) . . . . . . . . .

5.3

Overall results calculated on official test sets from 5B task. Scores from [START_REF] Wiese | Neural Question Answering at BioASQ 5B[END_REF] and Lab Zhu, Fudan Univer are reported in Bioasq 5.113 5.4 MRR results calculated batchwise on 5B official test sets. . . . . . . . .

5.5

Experiments with manual and automatically annotated dataset. Evaluation done on both BIOASQ gold standard data and annotated datasets.

5.6

Spacy tool uses this named entity annotation scheme following OntoNotes 5 corpus, which is mapped with EAT types . . . . . . . . . . . . . . . . Résumé : La recherche de réponses à des questions relève de deux disciplines : le traitement du langage naturel et la recherche d'information. L'émergence de l'apprentissage profond dans plusieurs domaines de recherche tels que la vision par ordinateur, le traitement du langage naturel etc. a conduit à l'émergence de modèles de bout en bout. Les travaux actuels de l'état de l'art en question-réponse (QR) visent à mettre en oeuvre de tels modèles.

Dans le cadre du projet GoASQ, l'objectif est d'étudier, comparer et combiner différentes approches pour répondre à des questions formulées en langage naturel sur des données textuelles, en domaine ouvert et en domaine biomédical. Ce travail se concentre principalement sur 1) la construction de modèles permettant de traiter des ensembles de données à petite et à grande échelle, et 2) l'exploitation de connaissances sémantiques pour répondre aux questions par leur intégration dans les différents modèles. Nous visons à fusionner des connaissances issues de textes libres, d'ontologies, de représentations d'entités, etc. Afin de faciliter l'utilisation des modèles neuronaux sur des données de domaine de spécialité, nous nous plaçons dans le cadre de l'adaptation de domaine. Nous avons proposé deux modèles de tâches de QR différents, évalués sur la tâche BIOASQ de réponse à des questions biomédicales. Nous montrons par nos résultats expérimentaux que le modèle de QR ouvert convient mieux qu'une modélisation de type Compréhension machine. Nous pré-entrainons le modèle de Compréhension machine, qui sert de base à notre modèle, sur différents ensembles de données pour montrer la variabilité des performances. Nous constatons que l'utilisation d'un ensemble de données particulier pour le pré-entraînement donne les meilleurs résultats lors du test et qu'une combinaison de quatre jeux de données donne les meilleurs résultats lors de l'adaptation au domaine biomédical.

Nous avons testé des modèles de langage à grande échelle, comme BERT, qui sont adaptés à la tâche de réponse aux questions. Les performances varient en fonction du type des données utilisées pour pré-entrainer BERT. Ainsi, le modèle de langue appris sur des données biomédicales, BIOBERT, constitue le meilleur choix pour le QR biomédical.

Les modèles d'apprentissage profond visent à fonctionner de bout en bout. Les informations sémantiques provenant de sources de connaissances construites par des experts n'y sont généralement pas introduites. Nous avons annoté manuellement et automatiquement un jeu de données par les variantes des réponses de BIOASQ et montré l'importance d'apprendre un modèle de QR avec ces variantes. Ces types sont ensuite utilisés pour mettre en évidence les entités dans les jeux de données, ce qui montre des améliorations sur l'état de l'art. Par ailleurs l'exploitation de représentations vectorielles d'entités dans les modèles se montre positif pour le domaine ouvert.

Nous faisons l'hypothèse que les résultats obtenus à partir de modèles d'apprentissage profond peuvent être encore améliorés en utilisant des traits sémantiques et des traits collectifs calculés à partir des différents paragraphes sélectionnés pour répondre à une question. Nous utilisons des modèles de classification binaires pour améliorer la prédiction de la réponse parmi les K candidats à l'aide de ces caractéristiques, conduisant à un modèle hybride qui surpasse les résultats de l'état de l'art. Enfin, nous avons évalué des modèles de QR ouvert sur des ensembles de données construits pour les tâches de Compréhension machine et Sélection de phrases. Nous montrons la différence de performance lorsque la tâche à résoudre est une tâche de QR ouverte et soulignons le fossé important qu'il reste à franchir dans la construction de modèles de bout en bout pour la tâche complète de réponse aux questions.

Université Paris-Saclay

Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France Title : Question Answering with Hybrid Data and Models Keywords : Question Answering, Natural Language Processing, Machine Learning Abstract : Question Answering is a discipline which lies in between natural language processing and information retrieval domains. Emergence of deep learning approaches in several fields of research such as computer vision, natural language processing, speech recognition etc. has led to the rise of end-toend models.

In the context of GoASQ project we investigate, compare and combine different approaches for answering questions formulated in natural language over textual data on open domain and biomedical domain data. The thesis work mainly focuses on 1) Building models for small scale and large scale datasets, and 2) Leveraging structured and semantic information into question answering models. Hybrid data in our research context is fusion of knowledge from free text, ontologies, entity information etc. applied towards free text question answering.

The current state-of-the-art models for question answering use deep learning based models. In order to facilitate using them on small scale datasets on closed domain data, we propose to use domain adaptation. We model the BIOASQ biomedical question answering task dataset into two different QA task models and show how the Open Domain Question Answering task suits better than the Reading Comprehension task by comparing experimental results. We pre-train the Reading Comprehension model with different datasets to show the variability in performance when these models are adapted to biomedical domain. We find that using one particular dataset (SQUAD v2.0 dataset) for pre-training performs the best on single dataset pre-training and a combination of four Reading Comprehension datasets performed the best towards the biomedical domain adaptation.

We perform some of the above experiments using large scale pre-trained language models like BERT which are fine-tuned to the question answering task. The performance varies based on the type of data used to pre-train BERT. For BERT pre-training on the language modelling task, we find the biomedical data trained BIOBERT to be the best choice for biomedical QA. Since deep learning models tend to function in an end-to-end fashion, semantic and structured information coming from expert annotated information sources are not explicitly used. We highlight the necessity for using Lexical and Expected Answer Types in open domain and biomedical domain question answering by performing several verification experiments. These types are used to highlight entities in two QA tasks which shows improvements while using entity embeddings based on the answer type annotations. We manually annotated an answer variant dataset for BIOASQ and show the importance of learning a QA model with answer variants present in the paragraphs.

Our hypothesis is that the results obtained from deep learning models can further be improved using semantic features and collective features from different paragraphs for a question. We propose to use ranking models based on binary classification methods to better rank Top-1 prediction among Top-K predictions using these features, leading to an hybrid model that outperforms state-of-art-results on several datasets. We experiment with several overall Open Domain Question Answering models on QA subtask datasets built for Reading Comprehension and Answer Sentence Selection tasks. We show the difference in performance when these are modelled as overall QA task and highlight the wide gap in building end-to-end models for overall question answering task.

Fig. 1

 1 Fig. 1.1: A traditional IR or NLP based question answering pipeline

Fig. 1 . 2 :

 12 Fig. 1.2: Deep learning QA pipeline (only the modules highlighted in red are used)

  question answering system techniques on open domain and domain specific data. Specifically we focus on answer sentence selection task on open domain data, answer extraction by reading comprehension and open question answering tasks on both open domain data and biomedical domain datasets.

Fig. 2

 2 Fig. 2.1: A traditional IR or NLP based question answering pipeline

Fig. 2 . 2 :

 22 Fig. 2.2: An example from HotpotQA dataset by (Zhilin Yang, Qi, et al., 2018) on multi-hop reasoning for QA.

Fig. 3 . 1 :

 31 Fig. 3.1: Question Answering pipeline as defined by[START_REF] Allam | The question answering systems: A survey[END_REF]

Fig. 3 . 2 :

 32 Fig. 3.2: Hierarchical question types taxonomy by (Moldovan et al., 1999) (left) and (Li and Roth, 2002) (right)

Fig. 3 . 3 :

 33 Fig. 3.3: An overview of deep learning methods applied for Answer Sentence Selectionpresented by[START_REF] Lai | A review on deep learning techniques applied to answer selection[END_REF]

  Figure 3.4 by[START_REF] Huang | Fusionnet: Fusing via fully-aware attention with application to machine comprehension[END_REF] gives a comparison between different works by showing which model uses how many RNN, at what level 3.2 Neural Question Answering -Task based classification

Fig. 3 . 4 :

 34 Fig. 3.4: Summary of several models using an attention mechanism for Reading Comprehension. Figure by (Huang et al., 2017).

Fig. 3 . 5 :

 35 Fig. 3.5: Comparing BERT by[START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] with OpenAI GPT[START_REF] Radford | Improving language understanding by generative pre-training[END_REF] and ELMO (M.Peters et al., 2018a)

Chapter 3

 3 11 https://syncedreview.com/2019/06/27/the-staggering-cost-of-training-sota-ai-models/ 52 State of the Art

Fig. 3 . 7 :

 37 Fig. 3.7: Overall DeepQA architecture of the IBM Watson QA system

Fig

  Fig. 3.8:Table presented by (Dua et al., 2019) with different types of reasoning required for the QA in DROP dataset

Fig. 4 . 2 :

 42 Fig. 4.2: Detailed reader model of DRQA

Fig. 4 . 3 :

 43 Fig. 4.3: BERT model modified for several NLP tasks by[START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF]

Figure 4 .Chapter 4

 44 Figure 4.3 shows the final layer of BERT model modified into Reading Comprehension task SQUAD. Both the question and paragraph are packed as a single packed sequence with question and paragraph as two different embedding representations from BERT. Both Start and End which represents the answer span in the paragraph are two

  This model has two parts, namely Paragraph Selector (PS) and Paragraph Reader (PR) in a cascade fashion, that is why we named it as PSPR model. Normally this model is termed as OpenQA model. The authors of (Y. Lin et al., 2018) use DRQA model in this model for modelling the Paragraph Reader (PR).

Fig. 4 . 4 :

 44 Fig. 4.4: OpenQA model by (Y.[START_REF] Lin | Denoising distantly supervised open-domain question answering[END_REF]

  p i ∈P P r (a|q, p i ) P r (p i |q, P )

Fig. 4 . 5 :

 45 Fig. 4.5: DRQA model by[START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF] (Left) and BIDAF model by[START_REF] Seo | Bidirectional attention flow for machine comprehension[END_REF] (Right)

  the context of domain adaptation and transfer learning for deep neural networks, two terms are often used. 1) Pre-training -is a learning or training process of a model with randomly initialized model weights. 2) Fine-tuning -is a learning or training process but initialized from the model weights of the pre-trained model and not randomly initialized model weights. Both pre-training and fine-tuning together can be termed as Domain Adaptation when the domain of the data used for pre-training and fine-tuning are different. For example, open domain and biomedical domain in question answering.

Tab. 4 . 1 :

 41 Datasets used in the experiments along with their splits. The numbers represent number of questions.

Fig. 4 . 6 :

 46 Fig. 4.6: Domain Adaptation from open domain to biomedical domain using DRQA model

3)

  Pre+Fine is the DRQA model trained on SQUAD v1.0 dataset and fine-tuned on BIOASQ dataset -Domain Adaptation. Results are shown in the Table4.2 for different BIOASQ test sets. When a model is only trained on a small dataset like BIOASQ, the results are very low as shown in the column No-Pre, because deep neural network model do not perform the best when the source dataset is small scale. When a model is trained on a large dataset like SQUAD v1.0, the model can be straight away used to predict results on the biomedical dataset. No-Fine shows a clear improvement doing so, against No-Pre, because of the large scale dataset it is trained on.

4. 3

 3 Domain adaptation with Different Models and Data any answer. Multiple paragraphs can contain the answer as well. An effective Open QA model must consider all these into account. Below is an example from the dataset. Q: Which calcium channels does ethosuximide target? A: T-type calcium channels P1: ..neuropathic pain is blocked by ethosuximide, known to block T-type calcium channels,.. P2: Theta rhythms remained disrupted during a subsequent week of withdrawal but were restored with the T-type channel blocker ethosuximide.

Fig. 4 . 7 :

 47 Fig. 4.7: Left: DRQA -Paragraph Reader (RC task). Right: PSPR -Paragraph Selector and Paragraph Reader model (Open QA task)

4. 3

 3 Domain adaptation with Different Models and Data 4.3.2 Comparison of Pre-Training with Different Datasets

Fig. 4 . 8 :

 48 Fig. 4.8: BERT model training process from Language modelling task to BIOASQ QA task.

  Non justifiable paragraph (False Positive)

Fig. 5 . 1 :

 51 Fig. 5.1: Brat annotation tool

Fig. 5 . 2 :

 52 Fig. 5.2: Output of Metamap tool for a sample paragraph containing the term MDR-TB

Fig. 5 . 3 :

 53 Fig. 5.3: The distribution of answers in three different answering settings for 59 questions:the red crossed points are for gold-standard answers that have the maximal similarity with the question LAT word embedding; So all the crossed points are correct answers. The blue round points above the X-axis are correct answers returned by DRQA system with maximal cosine similarity with WEEAT; The round points under the X-axis are false answers found by DRQA system with maximal cosine similarity. The absolute value is the similarity. The green triangles stand for the top-1 results of DRQA system, where the upper parts are correct answers and the low parts are wrong answers.

  Spacy annotated tag HUM PERSON, ORG, NORP LOC LOC, GPE, NORP ENTY PRODUCT, EVENT, LANGUAGE, WORK_OF_ART, LAW, FAC, NORP NUM DATE, TIME, PERCENT, MONEY, QUANTITY, ORDINAL, CARDINAL ABBR -DESC -Tab. 5.8: Named entity annotation scheme following OntoNotes 5 corpus mapped with EAT.

Fig. 5 . 4 :

 54 Fig. 5.4: EAT statistics on SQUAD dev set -contains 8026 annotations for 10,570 questions.

Fig. 5 . 5 :

 55 Fig. 5.5: EAT statistics on SQUAD train set -contains 66,659 annotations for 87,599 questions.

Fig. 5 . 6 :

 56 Fig. 5.6: Proposed RNN-Similarity model

Chapter 5

 5 21 https://spacy.io 134 Leveraging Structured and Semantic Information into Question Answering Models

5. 5 Fig. 5 . 7 :

 557 Fig. 5.7: An example of answer predicitons from BIOASQ data

  entity annotation scheme following OntoNotes 5 corpus mapped with EAT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . replacing entities along with an example from TrecQA dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.10 Statistics of datasets with plain and EAT annotated questions. '#' refers to "Number of." #Plain Q is the number of questions in the whole dataset, #EAT Q is the number of questions which are annotated with EAT (a subset of #Plain Q), #Entities is the number of entities on average per question in the paragraphs for the #EAT Q question setonly those entities which match the EAT are annotated and not the rest. 5.11 Results reported on TrecQA, WikiQA, and SQUAD-Sent datasets. SQUAD-Sent dataset is a modified version for answer sentence selection task. RNN-S is RNN-Similarity model. . . . . . . . . . . . . . . . . . . . . . 5.12 Results reported on TrecQA, SQUAD-Sent and WikiQA datasets using RNN-Similarity model trained only on EAT annotated questions . . . . 5.13 Number of questions with at least one entity as answer, in SQUAD dataset140 5.14 Average number of entities in the SQUAD dataset paragraphs . . . . . 5.15 Number of questions in SQUAD dataset official vs pre-processed . . . . 5.16 Whole SQUAD dataset system performance with dev set of 10570 questions. EW-EMB is the entity+word embeddings . . . . . . . . . . . 5.17 Entity QA system performance on pre-processed SQUAD dev set of 2705 questions. The same EW-Emb space does not perform well when used on only word (W) representations. Glove performs the best on only words (W) representations with comparable results on Fast text models. 5.18 Experiments on QUASAR-T dataset using different algorithms with best hyper parameters with all features Single, Collective and Semantic features from above. The best K value was found to be 3 on this dataset.150 5.19 Experiments on choosing K value for Top-K predictions using Adaboost algorithm with all features mentioned above on QUASAR-T dataset. . . 5.20 Experiments on Feature ablation and importance of features. Computing decrease in performance when certain features are removed from the input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.21 Experiments using different algorithms with best hyper parameters with all features listed above for biomedical domain . . . . . . . . . . . . . 5.22 Experiments on Feature ablation and importance of features. . . . . . . 5.23 Results on SQUAD variant dev datasets in Open Domain Question Answering and Reading Comprehension settings . . . . . . . . . . . . . 5.24 Results on using TrecQA dataset for OpenQA task . . . . . . . . . . . . List of Tables Université Paris-Saclay Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, FranceECOLE DOCTORALE N°580 Sciences et technologies de l'information et de la communication (STIC) Titre : Question-réponse utilisant des données et modèles hybrides Mots clés : Question-réponse, Traitement langage naturel, Apprentissage automatique

• 2019 -Measuring semantic similarity of clinical trial outcomes using deep pre-trained language representations

-Anna Koroleva, Sanjay Kamath, Patrick Paroubek. Journal of Biomedical Informatics: X, Published in October 2019.

• 2019 -

How to Pre-Train Your Model? Comparison of Different Pre-Training Models for Biomedical Question Answering.

	tational Linguistics and Intelligent Text Processing -CICLING 2019, April
	2019.
	• 2018 -
	-Sanjay Kamath, Brigitte Grau,
	Yue Ma. Proceedings of the 7th BioASQ Workshop A challenge on large-scale
	biomedical semantic indexing and question answering. ECMLPKDD, September
	2019.

• 2019 -Predicting and Integrating Expected Answer Types into a Simple Recurrent Neural Network Model for Answer Sentence Selection. -Sanjay Kamath, Brigitte Grau, Yue Ma. 20th International Conference on Compu-

An Adaption of BIOASQ Question Answering dataset for Machine Reading systems by Manual Annotations of Answer Spans.

	-Sanjay Ka-
	math, Brigitte Grau, Yue Ma. Proceedings of the 6th BioASQ Workshop A
	challenge on large-scale biomedical semantic indexing and question answering.
	EMNLP, October 2018.
	• 2018 -Verification of the Expected Answer Type for Biomedical Question
	Answering. -Sanjay Kamath, Brigitte Grau, Yue Ma. HQA workshop, compan-
	ion proceedings of the The Web Conference 2018, April 2018.
	• 2017 -A

Study of Word Embeddings for Biomedical Question Answer- ing

. -Sanjay Kamath, Brigitte Grau, Yue Ma. 4e édition du Symposium sur l'Ingénierie de l'Information Médicale, November 2017. 1.4 Publications 2 Question Answering -Tasks and Problems

Why is ice less dense than water? Answer Passage: The molecules of water are closer together and constantly moving, whereas the molecules of ice are in a crystal lattice, meaning they're in a rigid formation. When water freezes, the molecules spread out a little more to form the crystal lattice. Since density is mass over volume, and ice has takes up more volume than water, the density of ice is lesser than that of water. Which makes ice float on water. A non-factoid QA example

  Apart from the above two types of QA, there are also Multiple choice QA, Yes/No QA, Conversational QA etc. In our work we only focus on factoid QA therefore the further sections refer to Factoid Question Answering as just Question Answering or QA.

• Non-Factoid Question Answering Factoid answers are factual answers such named entities, numerical values, lists, currencies, locations etc. For domain specific data such as medical domain the answers can be the names of medicines, diseases, proteins etc.

Question:

is the author of the book , "The Iron Lady: A Biography of Margaret Thatcher" ? P 1 : The Iron Lady ; a biography of margaret thatcher by hugo young, farrar straus giroux P 2 : this second volume of ward's lively and psychologically revealing biog- raphy begins with his honeymoon abroad in 1905 and concludes with his election as governor of new york in 1928.

Q: Who

Question: Which NFL team represented the AFC at Super Bowl 50? Answer Passage: Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the logo could prominently feature the Arabic numerals 50. Answer: Denver Broncos. Start Offset: 177 An example of RC task from SQUAD dataset (Rajpurkar, J. Zhang, et al., 2016).

  The difference between Legacy OpenQA and OpenQA is in the intermediate modules. Earlier, different information retrieval approaches, knowledge graphs and ontologies were used to process information in the intermediate modules. Currently deep learning models have replaced these intermediate modules (by modelling different intermediate tasks) and the goal is to achieve an end-to-end model which does all of it in a single model like in multi task learning.OpenQA task is a union of both Answer Sentence Selection and Reading Comprehension tasks done sequentially but with certain changes. The assumption of Reading Comprehension that the paragraph always contains an answer will not be feasible in OpenQA setting as the Answer Sentence Selection model might select a paragraph semantically nearby to the question terms but might not contain an answer.

Q: What's the capital of Ireland? P 1 : As the capital of Ireland, Dublin is... P 2 : Ireland is an island in the North Atlantic. . . P 3 : Dublin is the capital of Ireland. Besides, Ottawa is one of famous tourist cities in Ireland and ... An example from a dataset on OpenQA task.

  There are two versions of this dataset, both have the same training set (Train has manual judgements and Train-all is a noisy set with automatic judgements) but their development and test sets differ. The clean version has removed questions in the dev and test sets that did not have answers or only contained positive/negative answers, reducing the development and test sets sizes from the raw version. WikiQA dataset (Y.[START_REF] Yang | Wikiqa: A challenge dataset for open-domain question answering[END_REF] is constructed from real queries of Bing search engine and

Wikipedia data. The answers are annotated by human annotators on Amazon Mturk crowd sourcing platform. Questions with no correct answers are usually removed, resulting in the statistics presented in the table 2.1.

  released the SQUAD dataset which presents a new dataset for Reading Comprehension annotated by human crowdsourced workers. A set of human annotators were asked to read a paragraph and frame questions on it. SQUAD dataset is split into Train, Dev and Test sets. The Test dataset is hidden from public access. The only way to evaluate a

	Dataset	Annotation	Split	#Question	Task
	CNN	Synthetic	Train Dev	380,298 3,924	RC -Cloze
			Test	3,198
	Dailymail	Synthetic	Train Dev	879,450 64,835	RC -Cloze
			Test	53,182
	SQUAD	Human Annotated	Train Dev Test	87,599 10,570 9,533	RC

  tasks since 2003-2014. Some of the QA tasks in this track are, 1) BIOASQ -a biomedical challenge dataset which we use in our work. 2) QA4MRE -a QA task which has multiple choice questions related to a document. 3) QALD -a QA task on linked data. 4) Answer Validation Exercise to assess QA responses and decide whether an Answer to a Question is correct or not according to a given Text. More details can be found on the track site 17 .

QA4MRE -Question Answering for Machine Reading Evaluation was a campaign which was held during 2011-2013 aimed at evaluating Machine Reading systems through Question Answering and Reading Comprehension Tests. The definition of Reading Comprehension is slightly different these days (also in our work) as the field of question answering refer to Reading Comprehension and Machine

  got lower scores than the deep learning ones.Starting from BIOASQ 7 in 2019, BERT models are being used with different training strategies such as training with varying paragraph data and varying data the BERT model is pre-trained on for language modelling task. BIOBERT used Biomedical data to pre-train BERT model and use it for BIOASQ task (J.Lee et al., 2019). And authors of[START_REF] Hosein | Measuring Domain Portability and ErrorPropagation in Biomedical QA[END_REF] use BERT model on BIOASQ task by pre-training BERT on Natural Questions dataset. Both BIOBERT and work by[START_REF] Hosein | Measuring Domain Portability and ErrorPropagation in Biomedical QA[END_REF] obtain similar results on same sets and BIOBERT being generally better than others. Both the models first train on open domain reading comprehension task and perform domain adaptation in their best scored approaches.

  The second workshop along with being open to normal papers also had a shared task where the goal was to evaluate the generalisation of models beyond the datasets they are trained on. Intuition is that the models should do more than merely interpolate from the training set to answer test examples drawn from the same distribution but rather extrapolate on data from different distributions. A training set was created by pooling six large scale datasets and test sets were ten different test sets to test the generalisation. More details can found on the link 23 .

	works such as new models, datasets, training methods, error analysis etc. on RC
	task.
	Machine Reading for Question Answering (MRQA) workshop 22 was organized for
	the first time in 2018 to gather researchers to address and discuss important research
	topics surrounding Machine Reading (Reading Comprehension). The goal was to
	discuss aspects such as Accuracy, Interpretability, Speed, Scalability, Robustness,
	Dataset Creation, Dataset Analysis, Error Analysis. The first task mainly covered

  The Paragraph Selector model predicts a probability distribution P r (p i |q, P ) over all the retrieved paragraphs where P is the set of paragraphs for the question.The reader model gives two probabilities (one for start and one for end token given by two classifiers) as described in equation 4.7 and 4.8. The answer probability P r(a|q, P ) is computed as shown below:The Paragraph Selector uses tokenized question words Q = {q 1 , ....., q m } and tokenized paragraph words P = {p 1 , ....., p n } which are encoded using an embedding layer of dimension D.

	P r (a|q, p i ) =	P r s a j s P r e a j e	(4.10)
	j

The Paragraph Reader model extracts answer spans as shown in the DRQA model for Reading Comprehension task and predicts a probability P r (a|q, p i ) for each answer span where p i is i th paragraph in Paragraph set P .

  is being widely used in NLP tasks such as question answering, textual entailment, sentiment analysis, named entity recognition, relation extraction etc. which are easily done by modifying a final output layer of the original model and fine-tuned. Fine-tuning can be done either by learning the whole model parameters or learning only a part of the model by freezing the rest.4.2.2 Biomedical Domain Adaptation for Reading ComprehensionAs explained earlier, a neural network model like DRQA cannot be used in a straightforward way on a small scale dataset like BIOASQ dataset whose statistics are presented in table 4.1. The statistics compares BIOASQ data with two other large scale datasets and shows the difference in the number of samples. The large scale datasets SQUAD v1.0 and QUASAR-T and several others which are currently used in different Question Answering tasks are at least 100 times larger than BIOASQdataset. An assumption with deep neural network models is that it needs large scale datasets to perform better. We test this assumption by experimenting with a deep neural network model trained on small scale dataset and show the performance difference experimentally.An important aspect to highlight is that the formulation of questions, paragraphs and answers from both SQUAD v1.0 dataset and BIOASQ dataset are very similar. Only the vocabulary and paragraph lengths are different (BIOASQ dataset has shorter paragraph lengths in gold standard data). The task definition provided by the organizers which states that all the paragraphs are useful to answer the questions, enables to use a Reading Comprehension model.

	Datasets	Train	Dev	Test
	BIOASQ 4b	427	59	161
	BIOASQ 5b	544	75	150
	BIOASQ 6b	685	94	161
	SQUAD v1.0 87,599 10,570 9,533
	QUASAR-T 37,012 3,000 3,000

  Itis a member of the chalcogen group on the periodic table and is a highly reactive nonmetal and oxidizing agent that readily forms compounds (notably oxides) with most elements.

	Answer: 8
	Question: Which topoisomerase is essential in yeast?
	Paragraph 1: Yeast DNA topoisomerase II is encoded by a single-copy, essential gene.
	Paragraph 2: Topo II performs topological modifications on double-stranded DNA
	molecules that are essential for chromosome condensation, resolution, and segrega-
	tion.
	Answers: Topoisomerase II, Topo II

table for oxygen? Paragraph: Oxygen is a chemical element with symbol O and atomic number 8.

  No-Pre model is the DRQA model trained on BIOASQ dataset only -Deep learning on small scale data. 2) No-Fine model is the DRQA model trained on SQUAD v1.0 dataset only. -Deep learning on large scale data. Results reporting the importance of Pre-Training and Fine-Tuning a model. All scores are computed on the official test sets combined into one. S.Acc is Strict Accuracy, L.Acc is Lenient Accuracy (the correct answer is in the top 5) and MRR is the Mean Reciprocal Rank for the correct answer in Top-5 answers. No-Pre is for No-Pretraining on open domain, No-Fine is for No-Finetuning on biomedical domain and Pre+Fine is for Pretraining and Finetuning on open domain and biomedical domain.

	Datasets	No. of Ques Metrics No-Pre No-Fine Pre+Fine
			S.Acc	08.98	23.96	24.00
	BIOASQ 4b	161	L.Acc MRR	16.56 11.36	35.26 28.40	39.21 29.34
			S.Acc	25.91	32.17	32.43
	BIOASQ 5b	150	L.Acc MRR	34.86 29.27	45.58 36.75	47.73 38.37
			S.Acc	13.40	26.24	26.72
	BIOASQ 6b	161	L.Acc MRR	27.08 19.20	40.60 32.57	43.72 33.80
			S.Acc	16.09	27.45	27.71
	Average		L.Acc MRR	26.16 19.94	40.48 32.57	43.55 33.83
	Tab. 4.2:

1)

Table 4 .

 4 3 presents the comparison of five word embedding spaces tested on the five test sets (Test-1 to Test-5) and a set with all test sets combined (All). BIOASQ 4 with 200D performed worse on our experiments. These embeddings are given by the organizers of BIOASQ task that is available on their website.BIOASQ 5 embeddings Accuracy (top 5) on 4B test with different Embeddings: |V|= vocab,|T|= token counts were trained by us with a Skipgram model and 300D. We can see that although Glove embeddings are trained on Web Crawl data and not specifically biomedical data, it performs better than the rest trained on biomedical data because of the large training data of Glove. Wiki+BIOASQ 5 trained with Skipgram and 300D on data of BIOASQ 5 and Wikipedia articles, has second best accuracy after Glove because of the domain specific training data even though it is smaller compared to Glove's training data. Comparision of Word2vec models on 4B Test set (Testset "All" from Table 4.3)

		Nb. of	BIOASQ 4	BIOASQ 5	Wikipedia	Wiki+BIOASQ 5	Glove
		Ques. |V| = 1.7M |V| = 2.1M |V| = 2.13M	|V| = 4M	|V| = 2.2M
			|T| = 2.2B	|T| = 2.3B	|T| = 2.19B	|T| = 4.49B	|T| = 840B
	Test-1	39	33.33	46.15	46.15		51.28	58.97
	Test-2	31	35.48	48.39	45.16		48.38	51.61
	Test-3	26	38.46	65.38	65.38		65.38	61.53
	Test-4	31	35.48	45.16	51.61		41.93	41.93
	Test-5	33	45.45	57.57	60.61		60.61	66.66
	Average	-	37.64	52.53	53.78		53.52	56.14
	All	160	33.12	51.25	45.0		50.0	52.5
	Tab. 4.3: CBOW	Skipgram
			Dims	Strict	Lenient	Strict	Lenient
			100	30.62	48.75	31.25	50.0
			200	28.75	47.5	33.75	50.0
			300	31.87	48.75	31.87	51.25
			400	28.75	46.87	30.0	48.75
	Tab. 4.4:

Table 4 .

 4 4 presents a comparison of different embedding spaces trained on different dimensions

Table 4 .

 4 5 as DRQA+PS. This is different PSPR model because PSPR learns the model by combining probabilities, whereas DRQA+PS does not learn but just use output probability scores.Results are shown in Table4.5 for different BIOASQ test sets. We compare different model results with BioBert scores reported in (J.Lee et al., 2019). The scores from PSPR model shows the performance on Strict and Lenient accuracy on 4b, 5b and 6b test sets. By taking paragraph probability into account PSPR allows to better rank Top-1 correct answer than BioBert which extracts answers from all paragraphs and chooses the maximum probability scored answer only based on answer probability. Although PSPR has a reader model similar to DRQA, considering the paragraph probability improves the answer extraction in PSPR model.

	Datasets	Metrics BioBert by (J. Lee et al., 2019) DRQA DRQA+PS PSPR
		S.Acc	36.48	24.00	26.22	30.28
	BIOASQ 4b	L.Acc MRR	48.89 41.05	39.21 29.34	32.33 26.54	40.34 34.19
		S.Acc	41.56	32.43	30.62	46.59
	BIOASQ 5b	L.Acc MRR	54.00 46.32	47.73 38.37	47.86 36.96	53.76 49.55
		S.Acc	35.58	26.72	26.50	43.91
	BIOASQ 6b	L.Acc MRR	51.39 42.51	43.72 33.80	42.16 32.07	51.34 45.70
		S.Acc	37.87	27.71	27.78	40.26
	Average	L.Acc MRR	51.43 43.29	43.55 33.83	40.78 31.85	48.48 43.14

Tab. 4.5: DRQA is the Reading Comprehension model by

[START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF]

, PSPR is the Open QA model by (Y. Lin et al., 2018), DRQA+PS is answers chosen with scores by multiplying answer probabilities of DRQA with Paragraph Selector probabilities of PSPR. SOTA scores are reported by (J. Lee et al., 2019) who average the best scores from each batch (possibly from multiple different models). Results from BIOASQ 4b, 5b and 6b test sets. 7b test set cannot be evaluated yet due to lack of gold standard answers. S.Acc is Strict Accuracy, L.Acc is Lenient Accuracy and MRR is Mean Reciprocal Rank. Experiments are done with the original BIOASQ data.

Table 4

 4 Experiments with data containing longer contexts (Document level) by (J.Lee et al., 2019). DRQA is a Reading Comprehension model by[START_REF] Chen | Reading Wikipedia to Answer Open-Domain Questions[END_REF]. BioBert-Unaltered is the original BIOASQ dataset with questions and paragraphs which contain answers. BioBert by (J.Lee et al., 2019) is the modified BIOASQ dataset where the paragraphs are longer paragraphs (documents from respective articles), where all the models are pre-trained on SQUAD v1.0 dataset and finetuned on BIOASQ dataset. SOTA scores are reported by (J.Lee et al., 2019) who average the best scores from each batch (possibly from multiple different models). Results from BIOASQ 4b, 5b and 6b test sets. 7b test set cannot be evaluated yet due to lack of gold standard answers. S.Acc is Strict Accuracy, L.Acc is Lenient Accuracy and MRR is Mean Reciprocal Rank.

	.6,

  Table 4.8 individually and fine-tune the model to BIOASQ datasets from Table 4.7 individually. Small scale BIOASQ datasets used in the experiments for fine-training, with their splits. The numbers represent number of questions. Large scale datasets used in the experiments for pre-training, with their splits. The numbers represent number of questions.

	Datasets	Train Dev Test
	BIOASQ 4b 427	59 161
	BIOASQ 5b 544	75 150
	BIOASQ 6b 685	94 161
	Tab. 4.7: Datasets	Train	Dev	Test
	SQUAD v1.0 87,599 10,570 9,533
	SQUAD v2.0 130,319 11,873 8,862
	Hotpot QA	90,564	7,405 7,405
	News QA	107,673 5,988 5,971
	Tab. 4.8:

For the datasets in table 4.8, note that the SQUAD 2.0, HotpotQA, NewsQA, SearchQA and TriviaQA are not exclusively for Reading Comprehension task as some datasets have relevant and irrelevant snippets like the datasets for OpenQA task. Therefore, 4.3 Domain adaptation with Different Models and Data

  [START_REF] Yoon | Pretrained Language Model for Biomedical Question Answering[END_REF] whose work scored the best at BIOASQ 7. Since we do not have access to BIOASQ 7 gold standard data for the test sets, we cannot evaluate our models on the test set.

	Index	Datasets	Measures BIOASQ 4 BIOASQ 5 BIOASQ 6	Average
			Strict	30.33	44.58	37.21	37.37
	1	NewsQA	Lenient	46.68	57.42	52.87	52.32
			MRR	36.74	49.82	43.54	43.36
			Strict	31.61	36.72	33.58	33.97
	2	HotpotQA	Lenient	45.81	52.14	53.74	50.56
			MRR	37.07	42.55	41.97	40.53
			Strict	30.30	42.12	36.73	36.38
	3	SQUAD v1.0	Lenient	52.66	56.99	54.60	54.75
			MRR	39.51	48.16	43.83	43.83
			Strict	34.00	42.91	41.01	39.30
	4	SQUAD v2.0	Lenient	50.50	57.28	56.49	54.75
			MRR	41.30	48.02	47.17	45.49
			Strict	31.13	46.53	38.90	38.85
	5	SQUAD v2.0 & v1.0	Lenient	48.24	56.47	54.88	53.19
			MRR	37.92	50.11	44.78	44.27
			Strict	34.70	39.36	41.41	38.49
	6	SQUAD v2.0 & v1.0	Lenient	51.40	57.69	56.64	55.24
		HotpotQA, NewsQA	MRR	41.27	46.76	47.13	45.05
			Strict	28.57	44.00	42.86	38.47
	7	BioBert	Lenient	47.82	56.67	57.14	53.87
		at BIOASQ	MRR	35.17	49.38	48.41	44.32
	SQUAD v2.0 dataset pre-training seems to clearly outperform SQUAD v1.0 on average
	on Strict Accuracy and MRR, which almost all previous works such as (Wiese et al.,

2017b; J.

Lee et al., 2019)

have used for pre-training their models. 16 https://github.com/huggingface/transformers 96 Chapter 4 Building Models for Small Scale and Large Scale Datasets Tab. 4.9: Experiments with different single datasets with fine-tuning on BIOASQ data. Results averaged over 5 official test sets. The paragraphs used in these experiments has been obtained from the authors of BioBert (J. Lee et al., 2019) which is a modified long document data and not BIOASQ gold standard paragraphs. Best scores are highlighted in Bold according to the different BIOASQ tasks and Average.

  scores which are pretrained on SQUAD v1.0 dataset. In the next set of experiments we experiment several BERT models including BioBert on the SQUAD v2.0 dataset.

There are multiple BERT models popular these days such as Roberta by (Y.

[START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF]

, XLNet by (Zhilin

[START_REF] Yang | XLNet: Generalized Autoregressive Pretraining for Language Understanding[END_REF]

, DistilBERT by

[START_REF] Sanh | DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter[END_REF]

etc. Some are trained using different tweaks in the architectures and training methods, and some are trained on different datasets. We are interested in testing 4.3 Domain adaptation with Different Models and Data

  It would be interesting to see how traditional deep learning models like CNN or RNN based models like DRQA and PSPR would perform when pretrained on different datasets. Note that training DRQA model takes approximately 4 hours, whereas fine-tuning BERT on SQUAD dataset takes around 2 hours which is why we chose BERT-Base for our experiments.

4.3 Domain adaptation with Different Models and Data

another might be subtle.

2019 -How to Pre-Train Your Model? Comparison of Different Pre-Training Models for Biomedical Question Answering. -

	Sanjay Kamath, Brigitte Grau,
	Yue Ma. Proceedings of the 7th BioASQ Workshop A challenge on large-scale
	biomedical semantic indexing and question answering. ECMLPKDD, September
	2019.
	• 2018 -

An Adaption of BIOASQ Question Answering dataset for Machine Reading systems by Manual Annotations of Answer Spans

	Leveraging Structured and	5
	Semantic Information into
	Question Answering Models
		. -Sanjay Ka-
	math, Brigitte Grau, Yue Ma. Proceedings of the 6th BioASQ Workshop A
	challenge on large-scale biomedical semantic indexing and question answering.
	EMNLP, October 2018.
		4.4 Conclusion

.1: Annotation statistics

		Gold std. annotations Full annotations
	Count	Avg	Total	Avg	Total
	Answers	0.8	500	2.9	1814
	Snippets	7.7	3286	8	4965
	Questions	-	426	-	618
	Tab. 5

.5023 Tab. 5.4:

  K-fold evaluation on different train sets with Gold and Anno data. DeepQA scores are presented by(Wiese et al., 2017b)

	Train set			5B				6B
	Finetune			Gold	Anno.	Gold	Anno.
	Eval	DeepQA	Gold	Anno.	Gold	Anno.	Gold	Anno.	Gold	Anno.
	Strict	-	0.2551 0.2962 0.1666 0.3333 0.2669 0.3090 0.2265 0.3948
	Lenient	-	0.4156 0.4444 0.2991 0.5843 0.4417 0.4724 0.3511 0.6197
	MRR	0.2620	0.3138 0.3425 0.2148 0.4322 0.3334 0.3718 0.2728 0.4765
	Tab. 5.2: Train set					5B
	Finetune						Gold	Anno.
	Eval	(Wiese et al., 2017c) Lab Zhu, Fudan Univer	Gold	Anno.	Gold	Anno.
	Strict	0.3466		0.3533		0.3533	0.42	0.3133 0.4266
	Lenient	0.5066		0.4533		0.54	0.64	0.5	0.6866
	MRR	-			-		0.4256 0.5042 0.3884 0.5258
	Train set					5B
	Finetune						Gold	Anno.
	Eval	(Wiese et al., 2017c) Lab Zhu, Fudan Univer	Gold	Anno.	Gold	Anno.
	Batch 1	0.5600		0.4200		0.4733 0.5733 0.4933 0.6066
	Batch 2	0.4086		0.4839		0.4274 0.5510 0.3387 0.5215
	Batch 3	0.4308		0.3846		0.4070 0.4198 0.3185 0.3955
	Batch 4	0.3025		0.2601		0.3595 0.4474 0.4444 0.6196
	Batch 5	0.3924		0.4524		0.4271 0.4771 0.3452 0

Tab. 5.3: Overall results calculated on official test sets from 5B task. Scores from

[START_REF] Wiese | Neural Question Answering at BioASQ 5B[END_REF]

and Lab Zhu, Fudan Univer are reported in Bioasq 5. MRR results calculated batchwise on 5B official test sets.

  Training on Anno. and evaluating on Gold has low scores in almost all experiments because of the model which learns on different forms of answers, therefore predicts different forms of answers which are not present in the Gold version.In Table5.3, because of a low number of questions in the official test sets ranging from 25 to 35 questions for each batch, the scores are computed over all 5B batch test sets (5B test sets -150 questions). The scores by[START_REF] Wiese | Neural Question Answering at BioASQ 5B[END_REF] and Lab Zhu, Fudan Univer are the best official results in Bioasq 5. We calculated strict and lenient accuracy as mentioned above and our scores are better than both best official results by 6.67% for strict accuracy and 13.34% lenient accuracy on Gold version training, 7.33% for strict accuracy and 18% lenient accuracy on Anno. version training.

	version of answers fetch best results by 3.68% and 8.58% on Strict accuracy, 14%
	and 14.73% on Lenient accuracy in 5B and 6B respectively.

. Comparing the DeepQA MRR score with the Gold and Anno. 5B versions, there is an improvement of at least 17% (Anno. training and Anno. evaluation) to 8% (Gold training and Anno. evaluation). In terms of accuracy, training the model on Anno. version and evaluating on Anno. 114 Chapter 5 Leveraging Structured and Semantic Information into Question Answering Models

  Spacy tool uses this named entity annotation scheme following OntoNotes 5 corpus, which is mapped with EAT typesThe Lexical Answer Type (LAT) is the word or words in a question which determines the type of expected answers. LAT and EAT are often used in the same context but are very different from each other. LAT corresponds to the word in the question phrase which is used to infer an expected type, whereas an EAT is the inferred type and not a word in the question. Some examples of questions and answers with their lexical answer types and expected answer types are given below:

	Expected Answer Type: NO_EAT
	Semantic Group: None
	Lexical Answer Type: causes
	Spacy annotated named entity type	EAT
	PERSON, ORG, NORP	HUM
	LOC, GPE	LOC
	PRODUCT, EVENT, LANGUAGE, WORK_OF_ART, LAW, FAC	ENTY
	DATE, TIME, PERCENT, MONEY, QUANTITY, ORDINAL, CARDINAL NUM
	Tab. 5.6: Question: What was the name of the food chain owned by Gus Fring?
	Answer: Los Pollos Hermanos
	Expected Answer Type (EAT): ORG.
	Lexical Answer Type (LAT): food chain.
	Question: What was the food joint famous for?
	Answer: Fried chicken
	Expected Answer Type (EAT): NO_EAT
	Lexical Answer Type (LAT): What
	Question: Who played the role of Walter White in the series Breaking Bad?
	Answer: Bryan Cranston
	Expected Answer Type (EAT): HUM.
	Lexical Answer Type (LAT): Who.

  Statistics of datasets with plain and EAT annotated questions. '#' refers to "Number of." #Plain Q is the number of questions in the whole dataset, #EAT Q is the number of questions which are annotated with EAT (a subset of #Plain Q), #Entities is the number of entities on average per question in the paragraphs for the #EAT Q question set -only those entities which match the EAT are annotated and not the rest.

		Split #Plain Q	#EAT Q	#Entities
	Trec QA	Train Dev	1229 82	649 (52.8%) 76 (92.68%)	13.96 5.02
		Test	100	82 (82%)	7.82
	SQUAD-Sent	Train Dev	87,599 10,570	78,740 (89.99%) 9,606 (90.87%)	0.44 0.49
		Test	-	-	-
	Wiki QA	Train Dev	873 126	859 (98.39%) 124 (98.41%)	0.15 0.03
		Test	243	236 (97.11%)	0.16
	Tab. 5.10:

  Note that we test our systems on the Raw version of TrecQA test dataset. Results reported on TrecQA, WikiQA, and SQUAD-Sent datasets. SQUAD-Sent dataset is a modified version for answer sentence selection task. RNN-S is RNN-Similarity model.

	Datasets	Method	Acc.@1	MAP	MRR
		Plain words -(Rao et al., 2016)	-	78	83.4
		EAT words -(Tayyar Madabushi et al., 2018)	-	83.6	86.2
	TrecQA	Plain words -RNN-S	78.95	80.24 84.81
		EAT words (single type) -RNN-S	85.26	85.28 89.16
		EAT words (different types) -RNN-S	85.26	85.48 88.11
		EAT words (MAX+different types) -RNN-S	86.32	85.42 88.86
		Plain words -Implementation of model by (Rao et al., 2016)	-	-	58.08
		Plain words -RNN-S	83.94	-	90.5
	SQUAD-Sent	EAT words (single type) -RNN-S	84.21	-	90.65
		EAT words (different types) -RNN-S	84.26	-	90.70
		EAT words (MAX+different types) -RNN-S	84.24	-	90.69
		Plain words -(Rao et al., 2016)	-	70.9	72.3
		Plain words -(Tymoshenko and Moschitti, 2018)	-	75.59 77.00
	WikiQA	Plain words -RNN-S	56.79	69.07 70.55
		EAT words (different types -NER) -RNN-S	55.14	66.56 68.10
		EAT words (MAX+different types -NER) -RNN-S	55.14	66.25 67.92
		EAT words (single type) -RNN-S	56.38	68.63 70.59
		EAT words (different types) -RNN-S	58.4	70.04 71.56
		EAT words (MAX+different types) -RNN-S	57.20	69.17 70.89
	Tab. 5.11:

  Plain text input fetches 69.07% on our RNN-S model, whereas the entity highlighted input with different types of entities fetches 136 Chapter 5 Leveraging Structured and Semantic Information into Question Answering Models 70.04% with a slight improvement. Only a few number of entities are annotated by spotlight compared to other datasets which is shown in the table5.10. To annotate entities better we experimented using Spacy NER types (marked as NER) directly which resulted in more annotated entities but reduced the performance lower than the word level scores.

Table 5 .

 5 12 shows various results on TrecQA SQUAD-Sent and WikiQA datasets with only the questions which are annotated with EAT information in the train and test sets.

	Training datasets contain questions which are annotated with EAT information, if
	the question does not have an EAT annotated, it is discarded from the dataset below
	are the set of experiments and results:
	• TrecQA (EAT): EAT words (MAX + different types) version of the dataset
	fetches the best scores on the test set with only EAT annotated questions. The
	same experiment with same dataset performed by (Tayyar Madabushi et al.,
	2018) fetches lower results (MAP: 81.74% and MRR: 82.93%).
	• SQUAD-Sent(EAT): EAT words (MAX + different types) version of the dataset
	fetches the best scores on this dataset.

What is the main executive body of the EU? P: The European Commission is the main executive body of the Euro- pean Union. A: European Commission Ent.Q: What is the main executive body of the Wikipage_European_Union? Ent.P: The Wikipage_European_Commission is the main executive body of the Wikipage_European_Union.

Q:

  The paragraph selector model gives a probability score P r (p i |q, P ) for each paragraph p i in the paragraph set P , where W is a learnt weight matrix and the probability of each paragraph is calculated via a max-pooling and a softmax layer on question q and paragraph pi inputs as shown in the equation 5.10.

			(5.7)
	P r e (i) ∝ exp (p i W e q)	(5.8)
	P r (a|q, p i ) =	P r s a j s P r e a j e	(5.9)
	j

5.7 and 5.

8. The answer probability P r (a|q, p i ) for each answer span where p i is i th paragraph in Paragraph set P is computed as shown in the equation 5.9.

146

Chapter 5 Leveraging Structured and Semantic Information into Question Answering Models P r s (i) ∝ exp (p i W s q) P r (p i |q, P ) = sof tmax max j pj i Wq

(5.10)

  Experiments on QUASAR-T dataset using different algorithms with best hyper parameters with all features Single, Collective and Semantic features from above. The best K value was found to be 3 on this dataset. ,000 dev and test dataset questions each. Results shown in Table5.18 is conducted using all the features explained in the section above using three different algorithms. The results reported are chosen based on the best performing scores validated on the official dev set and tested on the official test set. For all the classifiers mentioned in the table 5.18, the hyperparameter Number of estimators was tested with different values from 0 -2000 and the best performing score on dev set was chosen and results on the test set are reported.For each question, we consider the answers which are predicted as True class and consider Top-1 as the highest scored candidate. If there is no True class prediction for a question, we consider the False class prediction with lowest scored candidate.

	Algorithm PSPR -Baseline Randomforest Adaboost MLP classifier We experiment with QUASAR-T dataset which is an OpenQA dataset with several Accuracy 41.1 42.86 43.23 42.23 paragraphs per question. QUASAR-T dataset contains 37,012 training dataset ques-Accuracy 1 -Baseline 41.1 2 42.40 3 43.23 4 43.03 5 42.9 6 42.8 tions and 3Top-K 10 42.63

Tab. 5.18: Tab. 5.19: Experiments on choosing K value for Top-K predictions using Adaboost algorithm with all features mentioned above on QUASAR-T dataset.

  Experiments on Feature ablation and importance of features. Computing decrease in performance when certain features are removed from the input

	4. Paragraph Probability -Sent. Prob. -09.23%
	5. Answer Probability -06.15%
	Features removed from the best model	Accuracy
	Best model score	43.23
	-Summation of answer probability	43.03
	-Answer presence ratio	42.53
	-Maximum value of paragraph probability	42.23
	-Sentence Probability	43.2
	-Answer Probability	43.06
	-(Summation of Ans Prob. + Answer presence ratio)	42.46
	-(Maximum value of paragraph probability + Answer presence ratio) 41.36
	Tab. 5.20:
	1. Maximum value of paragraph probability -Max Para. Prob. -28.46%
	2. Answer presence ratio -27.69%
	3. Summation of answer probability -15.38%

  Experiments using different algorithms with best hyper parameters with all features listed above for biomedical domain Results shown in table5.21 highlight the gain in the performance of Strict Accuracy scores. For BIOASQ 4 and BIOASQ 6 datasets, there is an increase in performance, but for BIOASQ 5 there is a small decrease in performance.Feature ablation is important to understand which feature plays important role in the classification. Here are the few important features as calculated using scikit learn feature important plot. The list below represents the 5 most important features in decreasing order of magnitude.

	Datasets	Finetune BIOASQ 4 BIOASQ 5 BIOASQ 6
		Strict	30.31	46.83	42.79
	Baseline	Lenient	45.00	52.66	53.41
		Strict	39.37	44.00	45.96
	Adaboost	Lenient	45.00	52.66	53.41
		Strict	38.75	46.00	46.58
	Randomforest	Lenient	45.00	52.66	53.41
		Strict	34.37	46.00	38.50
	MLP	Lenient	45.00	52.66	53.41
	Tab. 5.21: 1. Maximum value of answer probability -21.87%
	2. Answer words overlap with paragraph words -15.28%
	3. Paragraph Probability -11.72%
	4. Lexical Answer Type UMLS semantic Type match -10.06%
	5. Lexical Answer Type UMLS CUI match -09.82%
	Features removed	Finetune BIOASQ 4 BIOASQ 5 BIOASQ 6
		Strict	38.75	46.00	46.58
	Best model score	Lenient	45.00	52.66	53.41
		Strict	38.75	43.33	44.72
	-Max. Ans Probability	Lenient	45.00	52.66	53.41
		Strict	38.75	45.33	45.96
	-Answer overlap	Lenient	45.00	52.66	53.41
		Strict	38.75	45.33	45.96
	-LAT Semantic Type	Lenient	45.00	52.66	53.41
		Strict	38.75	44.66	45.96
	-Paragraph Probability	Lenient	45.00	52.66	53.41
		Strict	37.5	42.66	44.72
	-(Feature 1 and 2)	Lenient	45.00	52.66	53.41

152 Chapter 5 Leveraging Structured and Semantic Information into Question Answering Models Tab. 5.22: Experiments on Feature ablation and importance of features. 5.5 Improving the QA Performance using Semantic and Structured Resources in a Ranking model

Tab. 5.23:

  Results on SQUAD variant dev datasets in Open Domain Question Answering and Reading Comprehension settings

  Results on using TrecQA dataset for OpenQA task We use the OpenQA model PSPR with TrecQA dataset for all experiments with slight variations. Trec only experiment is trained and tested only on TrecQA dataset using the PSPR model. Since the TrecQA dataset is small scale, we use a "pretrain and finetune" method. Pretrained with QUASAR-T experiment is pre-trained firstly on QUASAR-T dataset and fine-tuned with TrecQA dataset.Training a DNN model for the small scale TrecQA dataset although fetches relatively similar results to compared to the pre-trained ones with QUASAR-T, it is always better to pre-train when there is a large scale dataset with similar task at disposal. Our model of RNN-SIMILARITY(Kamath et al., 2019) on Answer Sentence Selection task has results of 85.2% MAP on TrecQA dataset for selecting the right answer sentence. But using a similar RNN model like PSPR on OpenQA task which uses paragraph selection and extraction module fetches around 45.68% results which is low for the answer extraction objective.

		.8
	Trec only	45.68
	Pretrained with QUASAR-T	49.38
	RC mode	58.02
	Tab. 5.24:

2019 -Predicting and Integrating Expected Answer Types into a Simple Recurrent Neural Network Model for Answer Sentence Selection

  . -Sanjay Kamath, Brigitte Grau, Yue Ma. 20th International Conference on Computational Linguistics and Intelligent Text Processing -CICLING 2019, April 2019.

• 2018

-An Adaption of BIOASQ Question Answering dataset for Machine Reading systems by Manual Annotations of Answer Spans

  . -Sanjay Kamath, Brigitte Grau, Yue Ma. Proceedings of the 6th BioASQ Workshop A challenge on large-scale biomedical semantic indexing and question answering. EMNLP, October 2018.

• 2018 -

Verification of the Expected Answer Type for Biomedical Question Answering

  . -Sanjay Kamath, Brigitte Grau, Yue Ma. HQA workshop, companion proceedings of the The Web Conference 2018, April 2018.

	• 2017 -A

Study of Word Embeddings for Biomedical Question Answer- ing

. -Sanjay Kamath, Brigitte Grau, Yue Ma. 4e édition du Symposium sur l'Ingénierie de l'Information Médicale, November 2017.

  Lee et al., 2019) who average the best scores from each batch (possibly from multiple different models). Results from BIOASQ 4b, 5b and 6b test sets. 7b test set cannot be evaluated yet due to lack of gold standard answers. S.Acc is Strict Accuracy, L.Acc is Lenient Accuracy and MRR is Mean Reciprocal Rank. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Small scale BIOASQ datasets used in the experiments for fine-training, with their splits. The numbers represent number of questions. . . . . . 4.8 Large scale datasets used in the experiments for pre-training, with their splits. The numbers represent number of questions. . . . . . . . . . . . 4.9 Experiments with different single datasets with fine-tuning on BIOASQ data. Results averaged over 5 official test sets. The paragraphs used in these experiments has been obtained from the authors of BioBert (J. Lee et al., 2019) which is a modified long document data and not BIOASQ gold standard paragraphs. Best scores are highlighted in Bold according to the different BIOASQ tasks and Average. . . . . . . . . . . 4.10 Experiments with BERT-BASE models trained on different text corpus for Language Modelling tasks. We pre-train these models on SQUAD V2.0 dataset and then fine-tune on BIOASQ dataset. Results averaged over 5 official test sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Annotation statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 K-fold evaluation on different train sets with Gold and Anno data.

https://goasq.lri.fr/

Extraction de la réponse étant donné un ensemble de paragraphes, pertinents et non pertinents

Extraction de la réponse étant donné un paragraphe pertinent

https://github.com/google-research/bert iii

Figure from: https://web.stanford.edu/class/cs124/lec/watsonqa.pdf 1.1 Question Answering Pipeline

Chapter 1 Introduction

Chapter 1 Introduction

https://smerity.com/articles/2016/architectures_are_the_new_feature_engineering.html 1.2 Redefining the QA Pipeline with Deep Learning

http://bioasq.org/ 8 Chapter 1 Introduction

http://participants-area.bioasq.org/results/ 1.2 Redefining the QA Pipeline with Deep Learning

https://www.snomed.org/snomed-ct/five-step-briefing

https://www.nlm.nih.gov/research/umls/index.html 10 Chapter 1 Introduction

1.3 Research Objectives and Contributions

Chapter 2 Question Answering -Tasks and Problems

https://trec.nist.gov/data/qa.html

https://en.wikipedia.org/wiki/Cloze_test 2.2 Task based classification

http://nlp.uned.es/clef-qa/repository/qa4mre.php 20 Chapter 2 Question Answering -Tasks and Problems

https://www.wired.com/story/ai-beat-humans-at-reading-maybe-not/, https://www.cnet.com/news/new-results-show-ai-is-as-good-as-reading-comprehension-aswe-are/

 5 https://rajpurkar.github.io/SQuAD-

explorer/ 6 http://u.cs.biu.ac.il/~yogo/on-squad.pdf 2.2 Task based classification

Chapter 2 Question Answering -Tasks and Problems

2.3 Datasets/Corpora

https://www.insurancelibrary.com

https://spacy.io 26 Chapter 2 Question Answering -Tasks and Problems

https://rajpurkar.github.io/SQuAD-explorer/2.3 Datasets/Corpora

https://www.reddit.com/r/trivia/comments/3wzpvt/free_database_of_50000_trivia_questions/

https://www.j-archive.com2.3 Datasets/Corpora

https://github.com/usnistgov/trec_eval

https://rajpurkar.github.io/SQuAD-explorer/

https://github.com/BioASQ/Evaluation-Measures2.4 Evaluation metrics

http://participants-area.bioasq.org/

https://goasq.lri.fr/ 34 Chapter 2 Question Answering -Tasks and Problems

https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art) 3.1 Text Based Question Answering by Feature Engineering Approaches

https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art)

3.2 Neural Question Answering -Task based classification

https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art)

https://github.com/usnistgov/trec_eval 44 Chapter 3 State of the Art

https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art)

Chapter 3 State of the Art

https://www.kaggle.com/

https://rajpurkar.github.io/SQuAD-explorer/

https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art)

https://rajpurkar.github.io/SQuAD-explorer/ 48 Chapter 3 State of the Art

https://www.sesamestreet.org/ 3.2 Neural Question Answering -Task based classification

https://github.com/huggingface/pytorch-transformers 3.2 Neural Question Answering -Task based classification

Consider values calculated from different paragraphs for a question

https://www.nytimes.com/2011/02/17/science/17jeopardy-watson.html 3.2 Neural Question Answering -Task based classification

https://leaderboard.allenai.org/drop/submissions/public 58 Chapter 3 State of the Art

https://hotpotqa.github.io3.3 Evaluations and Challenges

http://nlp.uned.es/clef-qa/repository/

http://www.clef-initiative.eu/track/qaclef

http://bioasq.org/3.3 Evaluations and Challenges

https://sites.google.com/view/mediqa2019

https://www.aicrowd.com/challenges/mediqa-2019-question-answering-qa

https://mrqa2018.github.io/3.3 Evaluations and Challenges

https://mrqa.github.io/shared 64 Chapter 3 State of the Art

3.4 Conclusion

https://rajpurkar.github.io/SQuAD-explorer/

https://github.com/facebookresearch/DrQA 4.1 Question Answering models

https://github.com/huggingface/transformers

https://github.com/thunlp/OpenQA 4.1 Question Answering models

Chapter 4 Building Models for Small Scale and Large Scale Datasets

https://github.com/thunlp/OpenQA 76 Chapter 4 Building Models for Small Scale and Large Scale Datasets

4.1 Question Answering models

https://rajpurkar.github.io/SQuAD-explorer/

https://github.com/allenai/bi-att-flow 78 Chapter 4 Building Models for Small Scale and Large Scale Datasets

https://en.wikipedia.org/wiki/Domain_adaptation 4.2 Domain Adaptation

https://github.com/facebookresearch/DrQA 4.2 Domain Adaptation

https://github.com/facebookresearch/DrQA 4.2 Domain Adaptation

https://radimrehurek.com/gensim/models/word2vec.html

https://nlp.stanford.edu/projects/glove/ 4.2 Domain Adaptation

SQUAD 2.0 is a variant of SQUAD dataset which contains questions without answers. We do not use this because the reference models also do not use v2.0 to pre-train.90Chapter 4 Building Models for Small Scale and Large Scale Datasets

https://github.com/naver/BioBert-pretrained 92 Chapter 4 Building Models for Small Scale and Large Scale Datasets

https://github.com/google-research/bert

https://www.nlm.nih.gov/research/umls/index.html

5.1 Annotation of answer variants for BIOASQ dataset to improve performance of existing QA models in Biomedical domain

http://brat.nlplab.org 108 Chapter 5 Leveraging Structured and Semantic Information into Question Answering Models

https://zenodo.org/record/1346193#.W3_WUZMzZQI

http://bioasq.org/

5.1 Annotation of answer variants for BIOASQ dataset to improve performance of existing QA models in Biomedical domain

https://semanticnetwork.nlm.nih.gov/download/SemGroups.txt

https://metamap.nlm.nih.gov 110 Chapter 5 Leveraging Structured and Semantic Information into Question Answering Models

https://github.com/AnthonyMRios/pymetamap 5.1 Annotation of answer variants for BIOASQ dataset to improve performance of existing QA models in Biomedical domain

Chapter 5 Leveraging Structured and Semantic Information into Question Answering Models

https://github.com/BioASQ/Evaluation-Measures 5.1 Annotation of answer variants for BIOASQ dataset to improve performance of existing QA models in Biomedical domain

https://spacy.io/ 5.2 Expected Answer Types (EAT) and their importance in Question Answering models

https://mmtx.nlm.nih.gov/SemanticTypesAndGroups.shtml

https://metamap.nlm.nih.gov/ 118 Chapter 5 Leveraging Structured and Semantic Information into Question Answering Models

5.2 Expected Answer Types (EAT) and their importance in Question Answering models

https://metamap.nlm.nih.gov

https://mmtx.nlm.nih.gov/SemanticTypesAndGroups.shtml

https://github.com/mariananeves/BioMedLAT

https://metamap.nlm.nih.gov/SemanticTypesAndGroups.shtml 5.2 Expected Answer Types (EAT) and their importance in Question Answering models

https://spacy.io/ 5.2 Expected Answer Types (EAT) and their importance in Question Answering models

https://github.com/rsanjaykamath/RNN-Similarity

https://github.com/rsanjaykamath/RNN-Similarity

https://github.com/cmasch/cnn-text-classification

https://github.com/rsanjaykamath/EAT-classifier 5.3 Using Expected Answer Types and Embeddings for the Answer Sentence Selection task

https://spacy.io/

www.harishmadabushi.com/research/answer-selection/ 5.3 Using Expected Answer Types and Embeddings for the Answer Sentence Selection task

https://github.com/castorini/Castor 5.3 Using Expected Answer Types and Embeddings for the Answer Sentence Selection task

https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art) 138 Chapter 5 Leveraging Structured and Semantic Information into Question Answering Models

https://tinyurl.com/y8p4457e 5.4 Using Embeddings of Entities for the Reading Comprehension task

http://wiki.dbpedia.org/develop/datasets/downloads-2016-10

5.5Improving the QA Performance using Semantic and Structured Resources in a Ranking model

https://scikit-learn.org/

https://wapiti.limsi.fr/ 5.5 Improving the QA Performance using Semantic and Structured Resources in a Ranking model

https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html

https://github.com/thunlp/OpenQA 5.5 Improving the QA Performance using Semantic and Structured Resources in a Ranking model

https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art)

https://github.com/rsanjaykamath/trecqa-rc 5.6 Applying the Open QA model on different datasets

https://goasq.lri.fr/

Chapter 6 Conclusion and Future Perspectives

https://zenodo.org/record/1346193#.W3

https://github.com/rsanjaykamath

https://github.com/rsanjaykamath/RNN-Similarity

https://github.com/rsanjaykamath/EAT-classifier

https://github.com/rsanjaykamath/trecqa-rc 166 Chapter 6 Conclusion and Future Perspectives

Bibliography

Acknowledgement

Question: Where was the TV show Breaking Bad primarily filmed at? Answer: Albuquerque Expected Answer Type (EAT): LOC. Lexical Answer Type (LAT): Where.

Question: Which president of the USA did not sign the Paris climate agreement? Answer: Donald Trump Expected Answer Type (EAT): PER Lexical Answer Type (LAT): President

In biomedical QA, the answers do not belong to the open domain named entity types as shown in Table 5.6. Biomedical domain Expected Answer Types (EATs) are specific to the domain.

In the case of biomedical domain for entities, the UMLS gives references of semantic types and groups which give more granular information about the entity type which it belongs to. Therefore instead of named entity types from tools like Spacy, we use semantic types and groups from UMLS 10 . There are 133 semantic types and 14 semantic groups. Metamap tool 11 is used to detect these for biomedical entities. To perform the evaluation of Entity question answering, we preprocess the SQUAD dataset -train and dev sets to retain only those questions whose answers only contain entities. Table 5.15 shows the dataset statistics before and after the pre-processing. Questions after pre-processing contained 18% of Who, 11% of Which, 62% of What, 7% of Where types of questions. We conduct experiments with three different embedding spaces: 1) EW-Emb (entity+word embeddings) 2) Global Vectors (Glove)

Experiments and Results

Data

Conclusion and Future Perspectives

In the context of GoASQ project 1 , our goal was to investigate, compare and combine different approaches for answering questions formulated in natural language over textual data using semi-structured and structured data.

We introduced question answering by defining and explaining the general process, different types of tasks, different kinds of data used. Different QA systems are built based on different types of tasks. In this thesis, we detailed all the above aspects for the three tasks: Reading Comprehension, Answer Sentence Selection and Open Question Answering, and report experiments on several datasets on these tasks.

One of our research focuses was towards building better QA systems for domain specific datasets like biomedical domain data, although open domain is also studied on certain tasks, such as answer sentence selection. We listed some of the hurdles we encountered while building a deep learning based QA system using deep learning models and our research questions were aligned towards addressing them:

1. How can we build models which work both on small scale and large scale datasets without dropping performance?

2. How can we leverage semantic and structured knowledge effectively into state-of-the-art question answering models?

We found that a hybrid QA system that can deals with these two aspects worked the best for the QA task in biomedical domain -the BIOASQ task. In the chapter 4 we proposed to model the task as an OpenQA task which performed the best and outperformed the state-of-the-art models which model BIOASQ as a Reading Comprehension task. We showed that pre-training on multiple Reading Comprehension datasets obtained the best performance scores on a state-of-the-art model for BIOASQ.

In the chapter 5, we proposed a method to annotate the answer variants in the training and test sets automatically which improved results by a large margin. This constitutes a realistic evaluation of the system performance. We also proposed to [START_REF] Lai | A review on deep learning techniques applied to answer selection[END_REF] . . . . . . . . . . . . . . . . . .

List of Figures

3.4

Summary of several models using an attention mechanism for Reading Comprehension. Figure by [START_REF] Huang | Fusionnet: Fusing via fully-aware attention with application to machine comprehension[END_REF]. . . . . . . . . . . . . .

3.5

Comparing BERT by [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] with OpenAI GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF] and ELMO (M. Peters et al., 2018a) . . . . . . . . . . . . .

3.6

Overall process of pre-training and fine tuning BERT for different NLP tasks by [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF]