Marine Romain Arthur

Anna Christopher

Arnaud, Yuming, Rachel, Rashedur, Swen Nicolas Zheng

Sanjay

Keywords: brat formatted sentence example

Important information for public health is contained within electronic health records. Being able to extract this information automatically would allow to improve the daily medical care and to foster advances in clinical research. However, the large majority of it remains locked in documents written in natural language.

Among all the clinical aspects that are of interest in these records, the patient timeline is one of the most important. Being able to retrieve clinical timelines would allow for a better Je souhaite commencer par remercier mes encadrants de thèse, Aurélie, Olivier et Xavier pour leur soutien sans faille et leurs conseils avisés. Merci d'avoir accepté d'encadrer mon travail, pour toutes ces discussions que nous avons eu semaine après semaine et de m'avoir donné goût à la recherche.

Je remercie les membres de mon jury de thèse. Merci à Philippe Muller et Mathieu Roche d'avoir accepté d'être rapporteurs de ma thèse et à Sandra Bringay, Claire Nédellec et Guergana Savova d'avoir accepté de l'évaluer.

Je remercie Guergana, Tim, Chen et toute l'équipe du Boston's Children Hospital de m'avoir accueilli pendant deux mois dans leur laboratoire. Cette escapade américaine a été enrichissante, pleine de bons souvenirs et m'a permis d'initier des liens amicaux et scienti ques durables.

La vie d'un laboratoire c'est un va-et-vient continu de gens tous plus intéressants les uns que les autres. Un grand merci à tous les collègues de passage au LIMSI, pour toutes ces discussions passionnantes sur la science et le reste qui ont animé nos pauses cafés et nos pauses déjeuners.

understanding of some clinical phenomena such as disease progression and longitudinal effects of medications. It would also allow to improve medical question answering and clinical outcome prediction systems. Accessing the clinical timeline is needed to evaluate the quality of the healthcare pathway by comparing it to clinical guidelines, and to highlight the steps of the pathway where speci c care should be provided.

In this thesis, we focus on building such timelines by addressing two related Natural Language Processing (NLP) topics which are temporal information extraction and clinical event coreference resolution.

Beyond the obvious necessity of tackling both issues at the same time, we could argue that both topics are interdependent. Intuitively, two event mentions that are coreferent would share the same temporal information. This has a number of implications when considering that tasks are done in a speci c order (coreference resolution then temporal information extraction). Information brought by the coreference links could be used to increase the performance of a temporal information extraction approach.

Conversely, two event mentions that share the same temporal location and the same meaning have high chances to be coreferent. The additional temporal information that would be available to a coreference resolution system could be valuable and improve performance.

Based on these observations, it becomes clear that temporal information extraction and clinical event coreference resolution need to be handled jointly not only because they are needed for clinical timeline extraction but also because there are complementary.

Temporal information extraction is a complex subject that requires carefully designed annotated corpora. These resources are mostly in English and the approaches that are developed in the community are biased toward that language. This motivates our rst research question: is it possible to devise a generic approach for temporal information extraction that could be used in di erent languages?

These corpora are often packing a rich set of categorical features describing the annotated entities. Moreover, there is a large number of clinical text preprocessing tools that allow to add more information about these entities. Given this context, how could this diversity of categorical features be used in neural approaches? This brings a follow-up question: how does the use of these categorical features impact the performance of such approaches?

Coreference resolution is an active topic in the clinical domain. As we mentioned above, both temporal information extraction and coreference resolution are interlinked. This leads II to our fourth research question: what sort of temporal information could be useful for coreference resolution? Coreference resolution approaches are complex and it is very hard to improve the performance of simple vanilla models. In this context, how could this temporal information be integrated in a neural approach for coreference resolution?

To answer these questions, we present several contributions. We devise a feature-based approach for temporal relation extraction. We evaluate our system on an English corpus. Then, we perform an empirical evaluation on two corpora of documents written in English and in French and showed that a similar approach can be used for both languages.

Our next contribution consists in a hybrid approach for end-to-end temporal information extraction that incorporates categorical features. We perform an empirical study on how categorical features impact the performance of our neural-based approach. Then, we evaluate our approach on an English dataset.

Concerning coreference resolution, we devise a neural-based approach for coreference resolution in the clinical domain inspired by recent approaches in the general domain. We devise a temporal feature and evaluate its contribution in the context of an empirical study which aimed at measuring how categorical features and neural network components such as attention mechanisms and token character-level representations in uence the performance.

In addition to the above-mentioned contributions, we contribute to the NLP community by devising two resources. First, we convert the i2b2 task1c corpus to the CoNLL format. This transformation could foster NLP research on coreference resolution in the clinical domain by allowing a better reproduction of results and by facilitating corpus processing. Second, we pack our neural sequence labeling module into an open source tool called Yet Another SEquence Tagger (YASET) that can be used for any NLP sequence labeling tasks.

Résumé

Les dossiers patients électroniques contiennent des informations importantes pour la santé publique. Être en mesure d'extraire automatiquement ces informations permettrait d'améliorer les soins médicaux et de soutenir la recherche clinique. Cependant, la majeure partie de ces informations est contenue dans des documents rédigés en langue naturelle.

Parmi toutes les informations cliniques intéressantes présentes dans ces dossiers, la chronologie médicale du patient est l'une des plus importantes. Être capable d'extraire automatiquement cette chronologie permettrait d'acquérir une meilleure connaissance de certains phénomènes cliniques tels que la progression des maladies et les e ets à long-terme des médicaments. De plus, cela permettrait d'améliorer la qualité des systèmes de question-réponse et de prédiction de résultats cliniques. Par ailleurs, accéder aux chronologies médicales est nécessaire pour évaluer la qualité du parcours de soins en le comparant aux recommandations o cielles et pour mettre en lumière les étapes de ce parcours auxquelles une attention particulière doit être portée.

Dans notre thèse, nous nous concentrons sur la création de ces chronologies médicales en abordant deux questions connexes en Traitement Automatique des Langues (TAL) : l'extraction d'informations temporelles et la résolution de la coréférence dans des documents cliniques.

Au-delà de la nécessité évidente d'aborder les deux tâches en même temps, nous soutenons que les deux sujets sont interdépendants. Intuitivement, deux événements médicaux qui sont coréférents partagent les mêmes informations temporelles. Cette observation a des implications si l'on considère l'ordre dans lequel les tâches sont e ectuées (le résolution de la coréférence puis l'extraction d'informations temporelles). L'information apportée par la présence de liens de coréférence entre les événements médicaux pourrait être utilisée pour améliorer la performance d'une approche d'extraction d'informations temporelles.

Inversement, deux mentions d'événements qui partagent les mêmes informations temporelles et la même signi cation ont une grande chance d'être coréférentes. Cette information temporelle pourrait permettre d'améliorer la qualité d'un système de résolution de la coréférence.

Sur la base de ces observations, il apparaît clairement que l'extraction d'informations temporelles et la résolution de la coréférence sont des tâches qui doivent être étudiées conjointement, non seulement car elles sont nécessaires à l'extraction de chronologies médicales, mais aussi car elles sont complémentaires.

L'extraction d'informations temporelles est un sujet complexe qui nécessite des corpus soigneusement annotés. Ces ressources sont principalement rédigées en anglais et les approches élaborées dans la communauté sont biaisées en faveur de cette langue. De cette observation découle notre première question de recherche : est-il possible d'élaborer une approche générique pour l'extraction d'informations temporelles qui pourrait être utilisée pour di érentes langues ? IV Ces corpus contiennent souvent un ensemble de traits catégoriels décrivant les entités annotées. De plus, il existe un grand nombre d'outils de prétraitement des textes cliniques qui permettent d'ajouter des informations sur ces entités. Dans ce contexte, comment cette diversité de traits catégoriels pourrait-elle être utilisée dans le cadre des approches neuronales ? Par ailleurs, comment l'utilisation de ces traits catégoriels in ue-t-elle sur la performance de ces approches ?

La résolution de la coréférence est un sujet de recherche actif en TAL clinique. Comme nous l'avons mentionné précédemment, l'extraction d'informations temporelles et la résolution de la coréférence sont des sujets interdépendants. Cela nous amène à notre quatrième question de recherche : quelle sorte d'informations temporelles pourrait être utile pour la résolution de la coréférence ? Les approches développées pour la résolution de la coréférence sont souvent complexes et il est très di cile d'améliorer signi cativement la performance de modèles simples. Dans ce contexte, comment cette information temporelle pourrait-elle être intégrée dans une approche neuronale pour la résolution de la coréférence ?

Pour répondre à ces questions, nous présentons plusieurs contributions. Nous concevons une approche à base de traits pour l'extraction des relations temporelles. Nous évaluons notre approche sur un corpus de documents écrits en anglais. Ensuite, nous e ectuons une évaluation empirique sur deux corpus de documents rédigés en anglais et en français et nous montrons qu'une approche similaire peut être utilisée pour les deux langues.

Notre seconde contribution consiste en une approche hybride pour l'extraction d'informations temporelles qui incorpore des traits catégoriels. Nous e ectuons une étude empirique sur la façon dont ces traits a ectent la performance de notre approche. Ensuite, nous évaluons notre approche sur un corpus de documents écrits en anglais.

En ce qui concerne la résolution de la coréférence, nous concevons une approche neuronale inspirée par les travaux récents dans le domaine général. Nous concevons un trait temporel et évaluons sa contribution dans le cadre d'une étude empirique visant à mesurer l'impact des traits catégoriels et des di érents composants habituellement utilisés dans les approches neuronales tels que les mécanismes d'attention et les représentations au niveau des caractères.

En plus des contributions susmentionnées, nous convertissons le corpus i2b2 task1c au format CoNLL. Cette transformation pourrait favoriser la recherche sur la résolution de la coréférence dans le domaine clinique en permettant une meilleure reproduction des résultats et en facilitant l'utilisation du corpus. Deuxièmement, nous empaquetons notre module d'étiquetage de séquences dans un outil open-source appelé YASET qui peut être utilisé dans des tâche d'étiquetage de séquence en TAL. ), the number of predicted relations (pred.), the number of correct relation with and without temporal closure (corr.), precision (P), recall (R) and f1-score (F1). . . . . . . . . . . . . . . . . 3.10 MERLOT (fr) and THYME (en) corpora: DCT relation distribution. . . . . . . . 3.11 Features used by our classi ers. . . . . . . . . . . . . . . . . . . . . . . . . . . 3.12 Cross-validation results over the training corpus for all tasks. We report f1score for CONTAINER and CONTAINS tasks and accuracy for DCT task. We also report standard deviation for all models (in brackets). . . . . . . . . . . . 3.13 DR task results over the test corpus. We report precision (P), recall (R) and f1-score (F1) for all relation types. . . . . . . . . . . . . . . . . . . . . . . . . 3.14 CR task results over the test corpus. We report precision (P), recall (R) and f1-score (F1) for all relation types. . . . . . . . . . . . . . . . . . . . . . . . .

V

List of Tables

4.1

Attributes available for each token of the corpus after preprocessing. . . . . . IX 4.2 Experimentation results. We report precision (P), recall (R) and f1-score (F1) for each con guration of our model, for the best system of the Clinical Tem-pEval 2016 challenge (H.-J. [START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF]) and for the best result obtained so far on the corpus (C. [START_REF] Lin | Improving Temporal Relation Extraction with Training Instance Augmentation[END_REF]). . . . . . . . . . . . . . . . . . . . . 61 4.3 Results obtained by the intra-sentence and inter-sentence classi ers for each model of this paper. We report the number of gold standard relations (ref), the number of relations predicted by our system (pred), the number of true positives (corr), precision (P), recall (R) and f1-score (F1). . . . . . . . . . . . 63 4.4 Results obtained by our system across our four runs. We report precision (P), recall (R) and f1-score (f1 We report true positives (TP), false positives (FP), false negatives (FN), precision (P), recall (R) and f1-score (F1). . . . . . . . . 6.9 Coreference resolution scores obtained with the predicted mentions on the test corpus. Gold coreference chains are projected on the predicted mentions and the score is computed with the o cial CoNLL scorer. We report precision (P), recall (R) and f1-score (F1) for all coreference metrics. . . . . . . . . . . 6.10 NER model performance for THYME clinical events on the development corpus part (20% of the training corpus). We report precision (P), recall (R) and f1-score (F1) for each category and micro-average on all categories. . . . . . . 6.11 Detailed coreference scores on gold mentions for all con gurations. We report precision (P), recall (R) and f1-score (F1) for all four metrics averaged over 10 runs. We report the standard deviation in brackets. . . . . . . . . . . . . . 6.12 Detailed coreference scores on predicted mentions for all con gurations. We report precision (P), recall (R) and f1-score (F1) for all four metrics averaged over 10 runs. We report the standard deviation in brackets. . . . . . . . . . . .

7

Important information for public health is contained within Electronic Health Records (EHRs). The vast majority of clinical data available in these records takes the form of narratives written in natural language. Although free text is convenient to describe complex medical concepts, it is di cult to use for medical decision support, clinical research or statistical analysis. The need to access information combined to the rapid growth of EHRs led to the development of NLP approaches tailored for the clinical domain.

Information Extraction (IE) has been applied successfully to a variety of tasks in the clinical domain over the last decades. Various examples of such applications can be found in the literature. One example is the automatic extraction of codes from clinical text. It consists usually in assigning codes dealing with diagnoses, such as International Classi cation of Diseases (ICD) codes. Several datasets were released to the community to foster NLP research on this topic. For instance, [START_REF] Pestian | A Shared Task Involving Multi-label Classi cation of Clinical Free Text[END_REF] o er to work on radiology reports while [START_REF] Névéol | Clinical Information Extraction at the CLEF eHealth Evaluation lab 2016[END_REF] release a large dataset for ICD-10 coding of death certi cates.

Surveillance is also an important area of research. One valuable application is adverse event detection (Velupillai et al. 2015a). These events may be related to medical procedures [START_REF] Penz | Automated Identi cation of Adverse Events Related to Central Venous Catheters[END_REF] or drugs [START_REF] Wang | Active Computerized Pharmacovigilance Using Natural Language Processing, Statistics, and Elec-References tronic Health Records: A Feasibility Study[END_REF]. Another use-case scenario is syndromic surveillance which consists in monitoring patient records to spot disease outbreaks [START_REF] Meystre | Extracting Information from Textual Documents in the Electronic Health Record: a Review of Recent Research[END_REF] or hospital-acquired infections (Velupillai et al. 2015a).

Another active area of research is EHR enrichment for computerized decision support. [START_REF] Meystre | Extracting Information from Textual Documents in the Electronic Health Record: a Review of Recent Research[END_REF] identify several sub-areas: automatic structuring of documents which consists in converting free text by segmenting and re-arranging them according to a template, automatic summarization which allows to have a concise view of clinical narratives, and case nding which allows to search for a speci c patient according to some criteria. Also, the i2b2 initiative 1 led several corpus annotation e orts to foster NLP research. It includes the annotation of patient smoking status [START_REF] Uzuner | Identifying Patient Smoking Status from Medical Discharge Records[END_REF]) obesity and comorbidities [START_REF] Uzuner | Recognizing Obesity and Comorbidities in Sparse Data[END_REF], medication information [START_REF] Uzuner | Extracting Medication Information from Clinical Text[END_REF]) and concepts, assertions and relations [START_REF] Uzuner | 2010 i2b2/VA Challenge on Concepts, Assertions, and Relations in Clinical Text[END_REF]. All these research e orts would not have been possible without access to data. As clinical narratives contain highly sensitive personal information about the patients and their conditions, one requirement is that clinical narratives must be properly de-identi ed. Automatic de-identi cation of clinical narratives has been addressed in many research e orts and is still an active area of research [START_REF] Grouin | De-Identi cation of Clinical Notes in French: Towards a Protocol for Reference Corpus Developpement[END_REF][START_REF] Neamatullah | Automated De-identi cation of Free-text Medical Records[END_REF]Stubbs and Uzuner 2015;[START_REF] Uzuner | Evaluating the State-of-the-Art in Automatic De-identi cation[END_REF].

Devising a comprehensive list of NLP applications to the clinical domain is a di cult task. For a detailed overview of such applications, we refer the reader to literature surveys on the topic [START_REF] Meystre | Extracting Information from Textual Documents in the Electronic Health Record: a Review of Recent Research[END_REF]Velupillai et al. 2015a;[START_REF] Wang | Active Computerized Pharmacovigilance Using Natural Language Processing, Statistics, and Elec-References tronic Health Records: A Feasibility Study[END_REF]).

Among all the clinical aspects that are of interest in these records, the patient timeline (Figure 1.1) is one of the most important. Being able to retrieve clinical timelines would allow for a better understanding of some clinical phenomena such as disease progression and longitudinal e ects of medications (C. Lin et al. 2016a;W. Sun et al. 2013c). It would also allow to improve medical question answering and clinical outcome prediction systems. Accessing the clinical timeline is needed to evaluate the quality of the healthcare pathway by comparing it to clinical guidelines, and to highlight the steps of the pathway where speci c care should be provided.

In this thesis, we focus on building such timelines by addressing two related NLP topics which are temporal information extraction and clinical event coreference resolution.

Temporal Information Extraction

Intuitively, building clinical timelines requires to extract relevant temporal information from text. Temporal information extraction has already a long history in NLP but was mainly addressed in the news domain. The rst research e orts on the topic can be tracked down to the Message Understanding Conferences (MUCs) [START_REF] Sundheim | Tipster/MUC-5: Information Extraction System Evaluation[END_REF]. At that time, the scope of the e ort was limited to extracting dates and times from news reports. A major milestone was reached in 2005 with the creation of the rst full temporal annotation scheme called TimeML [START_REF] Pustejovsky | The speci cation language TimeML[END_REF]. This research e ort allows to model precisely temporal information in text and has in uenced most of the following research up to this day. The clinical domain started to develop an interest in temporal information in the early 2010's with the release of the i2b2 corpus (W. Sun et al. 2013b). The large majority of temporally annotated datasets in both the general and the clinical domains contain documents written in English. However, there are several e orts aiming at devising resources in other languages. For instance, [START_REF] Campillos | A French Clinical Corpus with Comprehensive Semantic Annotations: Development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)[END_REF] recently annotated a clinical corpus of documents written in French.

Temporal information extraction in the clinical domain can be broken down into two main steps. The rst implies the extraction of clinical event mentions and temporal expressions from clinical narratives. The second involves the extraction of temporal relations. Example 1 contains one clinical event (neck pain) and one temporal expression (July) that are linked with a temporal relation (begins-on). De nition of events, temporal expressions and temporal relations vary across corpora but are often derived from the TimeML speci cation. Following this two-step process, temporal information acquired from documents can be aggregated into a timeline.

(1) She has been experiencing [ EVENT neck pain] since [ DATE July].

→ neck pain BEGINS-ON July

Feature-based approaches for automatic temporal information extraction have progressively been replaced by neural approaches over the last ve years. However, initial hopes for a major performance improvement have rapidly faded as classical feature-based approaches still remain competitive [START_REF] Bethard | SemEval-2015 Task 6: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF].

There is an on-going e ort in the NLP community for combining both approach types. Performance has been improved by including categorical features in neural network approaches. For instance, [START_REF] Dligach | Neural Temporal Relation Extraction[END_REF] used Part-Of-Speech (POS) tags as features for temporal relation extraction in clinical narratives.

Coreference Resolution

Another linguistic phenomenon to consider when building a patient timeline is coreference. Clinical reports contain multiple mentions of the same clinical events due to the fact that the medical sta needs to follow-up on these events. Intuitively, this phenomenon brings noise when building clinical timelines and needs to be handled appropriately.

Similarly to temporal information extraction, coreference resolution has been addressed primarily in the news domain and the rst systematic evaluations can be traced back to the same conferences [START_REF] Hirschman | MUC-7 Coreference Task De nition[END_REF][START_REF] Sundheim | Overview of Results of the MUC-6 Evaluation[END_REF]. In the clinical domain, the i2b2 initiative is once again responsible for the di usion of the rst clinical dataset annotated with coreference in 2012 [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF].

Coreference resolution in the clinical domain can be divided into two main steps. In the rst one, potential coreferring text spans must be extracted. Usually, these elements fall into speci c categories such as medication or medical procedure. Once these mentions have been extracted, one must gure out which ones are referring to the same real-world clinical events. In Example 2, all bracketed mentions are referring to the same real-world medical problem.

(2) The CXR revealed [8 mm obstructing stone] ... [The renal stone] was considered to be the cause of patient's symptoms ... We recommended surgical procedure to remove [ureteropelvic stone] ...

Research e orts for coreference resolution in the clinical domain implement feature-based approaches [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF]. However, similarly to temporal information extraction, neural approaches have become increasingly popular in the general domain [START_REF] Lu | Event Coreference Resolution: A Survey of Two Decades of Research[END_REF]. Moreover, we note that hybrid approaches including categorical features in neural networks have allowed signi cant performance improvement. For instance, K. [START_REF] Lee | End-to-end Neural Coreference Resolution[END_REF] encoded speaker information, text genre and other features as dense representations in their neural-based coreference resolution approach.

Topic Interdependence

As we mentioned above, temporal information extraction and coreference resolution are two of the main topics that need to be addressed when considering the task of building clinical timelines. Event mentions need to be extracted and placed in time. Simultaneously, coreferring mentions need to be regrouped as they are referring to the same real-world event.

Beyond the obvious necessity of tackling both issues at the same time, we could argue that both topics are interdependent. Intuitively, two event mentions that are coreferent would share the same temporal information. This has a number of implications when considering that tasks are performed in a speci c order (coreference resolution then temporal information extraction). Information brought by the coreference links could be used to increase the performance of a given temporal information extraction approach.

Conversely, two event mentions that share the same temporal location and the same meaning have high chances to be coreferent. The additional temporal information that would be available to a coreference resolution system could be valuable and bring a performance improvement.

Based on these observations, it becomes clear that temporal information extraction and clinical event coreference resolution need to be handled jointly not only because they are needed for clinical timeline extraction but also because there are somehow complements.

Research Questions

Temporal information extraction is a complex subject that needs carefully designed annotated corpora. These resources are mostly in English and the approaches that are developed in the community are biased toward that language. This motivates our rst research question: is it possible to devise a generic approach for temporal information extraction that could be used in different languages?

These corpora are often packing a rich set of categorical features describing the annotated entities. Moreover, there is a large number of clinical text preprocessing tools that allow to add more information about these entities. Given this context, how could this diversity of categorical features be used in neural approaches? This brings a follow-up question: how does the use of these categorical features impact the performance of such approaches?

Coreference resolution is an active topic in the clinical domain. As we mentioned above, both temporal information extraction and coreference resolution are interlinked. This leads to our fourth research question: what sort of temporal information could be useful for coreference resolution? Coreference resolution approaches are complex and it is very hard to improve the performance of simple vanilla models. In this context, how could this temporal information be integrated into a neural approach for coreference resolution?

Contributions

In the rst part of this thesis, we address temporal information extraction from clinical narratives. We present four contributions to the topic:

• A feature-based approach for narrative container extraction from clinical narratives. We devise a competitive feature-based approach for temporal relation extraction. We test our system on an English dataset in the context of the 2016 edition of the Clinical TempEval shared task [START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF].

• An abstraction of our feature-based approach. We perform an empirical evaluation on two corpora of documents written in English and in French. We show that a similar approach can be used for both languages.

• A neural approach for end-to-end temporal information extraction that incorporates classical categorical features. We address event, temporal expression and temporal relation extraction in clinical narratives. We test our approach on an English dataset in the context of the 2017 edition of the Clinical TempEval shared task [START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF].

• An empirical study on how categorical features impact the performance of our neural-based approach. We use gold features available in the dataset but also predicted features obtained via the use of clinical text preprocessing tools.

The second part of this thesis is dedicated to clinical event coreference resolution. We present two contributions to the topic:

• A neural-based approach for coreference resolution in the clinical domain inspired by recent approaches in the general domain. We address coreference resolution on both gold mentions and predicted mentions.

• A temporal feature derived from the temporal relation that exists between events and DCTs. We test this feature in the context of an empirical study which aims at measuring how categorical features and neural network components such as attention mechanisms and token character-level representations in uence the performance.

In addition to the above-mentioned contributions, we contribute to the community by devising two resources. First, we convert the i2b2 task1c corpus to the CoNLL format. This transformation could foster NLP research on the topic by allowing a better reproduction of results and by facilitating corpus processing. Second, we pack our neural sequence labeling module into a open source tool called YASET than can be used for any NLP sequence labeling tasks.

Outline

The thesis is divided into two parts. The rst one is composed of three chapters and is related to temporal information extraction. We start by reviewing the literature on the topic and we present our contributions. The second part is composed of two chapters and is related to clinical event coreference resolution. Similarly, we begin with a survey of the literature on the topic followed by a presentation of our contributions.

Part I -Temporal Information Extraction from Clinical Narratives

• Chapter 2 -It's About Time: Temporal Information Extraction from Text. In this chapter, we review the di erent aspects of time from both linguistic and computational linguistic perspectives. We address the de nition of the three primitives of time: events, times and temporal relations. We survey the di erent approaches that have been devised for temporal information extraction from text in both the general and the clinical domains.

• Chapter 3 -Feature-Based Approach for Temporal Relation Extraction: Application to French and English Corpora. We present a feature-based approach for temporal information extraction in clinical narratives. We evaluate our approach in the context of the 2016 edition of the Clinical TempEval shared task. Then, we investigate whether the same generic approach can be applied on two di erent languages. We evaluate our approach on two datasets of clinical documents written in English and French.

• Chapter 4 -Neural Approach for Temporal Information Extraction. We present a neural-based approach for temporal information extraction. We devise a neural architecture for clinical event, temporal expression and temporal relation extraction. We investigate how categorical features impact the performance of our model. Then, we evaluate our approach in the context of the 2017 edition of the Clinical TempEval shared task.

Part II -Clinical Event Coreference Resolution

• Chapter 5 -Clinical Event Coreference Resolution. This chapter presents an overview of the current state of the literature related to coreference. We present how the notion of coreference is handled in linguistics and computational linguistics. Speci cally, we address the di erences that exist between event coreference resolution in the general and in the clinical domains. We review the supervised machine learning approaches that have been devised in the literature.

• Chapter 6 -Neural Entity-Based Approach for Coreference Resolution in the Clinical Domain. We present a neural entity-based approach for coreference resolution inspired by recent e orts in the domain. We devise a temporal feature that we integrate into our approach. We perform an empirical study on how usual neural network components such as attention mechanisms or character-level representations impact the performance of our approach.

Published Work

The material presented in Chapter 3 is based on three publications: one at the 2016 edition of the SemEval workshop [START_REF] Tourille | LIMSI-COT at SemEval-2016 Task 12: Temporal Relation Identi cation Using a Pipeline of Classi ers[END_REF]), one at the 2016 edition of the TALN conference (Tourille et al. 2016a) and one at the 2017 edition of the EACL conference [START_REF] Tourille | Temporal Information Extraction from Clinical Text[END_REF]). The material presented in Chapter 4 is based on two publications: one at the 2017 edition of the Clinical TempEval workshop (Tourille et al. 2017a) and one at the 2017 edition of the ACL conference [START_REF] Tourille | Neural Architecture for Temporal Relation Extraction: A Bi-LSTM Approach for Detecting Narrative Containers[END_REF].

The presentation of our sequence labeling tool will be published in the proceedings of the 2018 edition of the LOUHI workshop [START_REF] Tourille | Evaluation of a Sequence Tagging Tool for Biomedical Texts[END_REF].

Publication List

1. Julien [START_REF] Tourille | LIMSI-COT at SemEval-2016 Task 12: Temporal Relation Identi cation Using a Pipeline of Classi ers[END_REF]

Introduction

This chapter presents the di erent aspects of time in linguistics and computational linguistics. First, we survey how the notion of time is addressed in the linguistic literature (Section 2.2). Then, we discuss how the computational linguistics community has modeled this notion: we describe the main models (Section 2.3) and resources (Section 2.4) that are available for devising temporal information extraction approaches (Section 2.5).

Time in Natural Language: A Linguistic Perspective

Time can be described according to three primitives: events, times and temporal relations [START_REF] Derczynski | Automatically Ordering Events and Times in Text[END_REF]. In this section, we review each of them from a linguistic perspective. We focus our attention on the English language. Our choice is guided by the fact that a large number of annotated resources are available in that language and that we mostly worked on texts written in English. However, we acknowledge the fact that some language particularities may not be re ected in the overview presented in this section. For instance, tense and aspect are di erent in French and English. These disparities are re ected in annotation e orts [START_REF] Bittar | French Time-Bank: An ISO-TimeML Annotated Reference Corpus[END_REF]).

Temporal Expressions

Temporal expressions or time expressions are used in natural language to give information about when something happened, how long something lasted or how often something occurred [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF]. They may denote calendar dates, times of day, durations or sets of recurring times.

The text extent which signals the temporal expression is called a lexical trigger. [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF] identi es six types of lexical triggers: nouns (e.g. century or year), proper names (e.g. Christmas or April), adjectives (e.g. past or current), adverbs (e.g. currently or weekly), time patterns (e.g. 9:00 or '80s) and numbers (e.g. 4th as in John arrived on the 4th).

Temporal expressions convey information about position in time, duration, frequency or temporal relationship [START_REF] Biber | Longman Grammar of Spoken and firitten English[END_REF]. We discuss the rst three in this section. The last one, temporal relationship information, will be discussed in Section 2.2.3.

Position in Time.

Temporal expressions can be used to locate an event in time. In this case, they answer the when question. Such expressions can take the form of noun phrases and usually includes a determiner such as that (Example 3a). Noun phrases can also be used within prepositional noun phrases (Example 3b). Position in time can also be expressed by adverbs (e.g. now, then, today, ago or yesterday) as shown in Example 3c, using time points (Example 3d), or vaguely speci ed with time periods or intervals (Example 3e).

(3) a. Mary met him that afternoon [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF] b. The wedding was on Thursday [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF] c. John went for a walk yesterday [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF] d. We found the letter at twelve noon [START_REF] Allen | Maintaining Knowledge About Temporal Intervals[END_REF] e. We found the letter yesterday [START_REF] Allen | Maintaining Knowledge About Temporal Intervals[END_REF] There is an agreement in the literature to make the distinction between absolute and relative position in time [START_REF] Derczynski | Automatically Ordering Events and Times in Text[END_REF][START_REF] Ferro | TIDES-2005 Standard for the Annotation of Temporal Expressions[END_REF][START_REF] Mani | Robust Temporal Processing of News[END_REF]. This consideration led to the creation of three categories of temporal expressions referring to a position in time in the literature:

• Absolute. This category regroups time expressions that refer to an absolute point in time. Enough information is contained in the expression to pinpoint the exact temporal location.

• Deictic. Time expressions that are interpreted relatively to the utterance situation are categorized as deictic. These expressions usually specify a temporal distance and a direction from the utterance time (e.g. tomorrow, yesterday).

• Anaphoric. Time expressions that are interpreted relatively to another time which is not the utterance time are considered anaphoric. These expressions have usually three parts: a temporal distance, a temporal direction and an anchor for both distance and direction (e.g. that evening, a few hours later).

Temporal Duration. Temporal expressions denoting a temporal duration answer the how long question. They can take the form of noun phrases or prepositional noun phrases (Example 4a and 5). Some studies underline that bare noun phrases can be considered as prepositional noun phrases lacking the preposition for (Example 4b, [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF]). Other studies describe durative temporal expressions as intervals bound by a start and an end point where the distance between the two is known [START_REF] Derczynski | Automatically Ordering Events and Times in Text[END_REF]. Also, they cannot be placed on a calendar. These expressions are sometimes close to deictic expressions as shown in Example 5.

(4) a. They lived several years in Italy. [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF] b. His mother-in-law stayed (for) three weeks. [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF] (5) a. Duration: The Texas Seven hid out there for three weeks. [START_REF] Ahn | Towards Task-Based Temporal Extraction and Recognition[END_REF] b. Deictic: California may run out of cash in three weeks. [START_REF] Ahn | Towards Task-Based Temporal Extraction and Recognition[END_REF] Frequency in Time. Temporal expressions denoting a frequency in time answer the how often question. These expressions usually start with every/each (Example 6a). Frequency in time can also be expressed as a bare plural (Example 6b) or can be introduced by the prepositions on, at or in (Example 6c). Finally, it can also be expressed by adjectives and adverbs derived from time units (e.g. hourly, monthly, annually) as shown in Example 6d.

All examples are extracted from [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF].

(6) a. Mary writes an article or a review every month.

b. Saturdays John goes to the theatre.

c. They reviewed their stock portfolio on the first day of each month.

d. A monthly newsletter is emailed to all customers.

Events

Events in linguistics are studied under the notion of eventualities [START_REF] Bach | The Algebra of Events[END_REF] or situations [START_REF] Comrie | Aspect: An Introduction to the Study of fferbal Aspect and Related Problems[END_REF]). The literature makes the distinction between non-stative and stative eventualities [START_REF] Dowty | fiord Meaning and Montague Grammar: The Semantics of fferbs and Times in Generative Semantics and in Montague's PTQ[END_REF][START_REF] Pustejovsky | The Generative Lexicon[END_REF][START_REF] Vendler | Linguistics in Philosophy[END_REF]. Stative eventualities involve events where there is no change during the time span denoted by the event. Non-stative eventualities involve a change of state. [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF] highlights that event identi cation in text is not a simple task. Events can be realized by verbs (Example 7a), nouns (Example 7b), adjectives (Example 7c), verb phrases (Example 7d), clauses (Example 7e), sentences (Example 7f) or semantic entities (Example 7g). On some occasions, agreeing on the number of events is even di cult (Example 7f). All examples are taken from [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF].

(7) a. In scal 1989, Elco earned $7.8 million, or $1.65 a share. b. Ms. Atimadi says the war has created a nation of widows. c. They say IRA commanders are responsible for the recent bomb attacks. d. Rally's Inc. said it has adopted a shareholders rights plan. e. The Federal Bureau of Investigation says it received more than eight thousand reports of hate group crimes last year f. Telerate's two independent directors have rejected the offer. g. We know that 3,000 teens start smoking each day, although it is a fact that 90% of them once thought that smoking was something that they'd never do.

Most linguistic studies on events are centered around the verb. Authors try to classify events according to how the events they denote take place in time [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF]. The phenomenon is captured by the notion of lexical aspect which is described according to three main dimensions [START_REF] Bethard | Finding Event, Temporal and Causal Structure in Text: A Machine Learning Approach[END_REF][START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF]):

• Dynamicity: it allows to distinguish between static events (e.g. know or love), that do not involve change and dynamic events that do involve change (e.g. run or deliver).

• Durativity: it allows to distinguish between instantaneous (or punctual) events, that are conceptually one point in time (e.g. nd or blink) and durative events that last for a period of time (e.g. desire or push a cart).

• Telicity: telic events have an end point or are directed toward a goal [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF] as shown in Example 8a. Atelic events do not have a goal or endpoint (Example 8b).

The telic/atelic distinction has been coined with other pairs of terms in the literature: bounded vs. non-bounded, culminating vs. non-culminating, delimited vs. non-delimited or de nite vs. inde nite change of state [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF].

(8) a. Telic event: John fixed the roof. [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF] b. Atelic event: He drove for hours.

This three-dimension lexical aspect breakdown has been used in a large number of papers studying eventualities from a linguistic perspective. Most of them are based on the work of [START_REF] Vendler | Linguistics in Philosophy[END_REF] who makes the distinction between four event categories:

• States, that do not describe any change in the world and last for a period of time (e.g. desire or believe). They are static and durative.

• Activities, that are dynamic, durative and atelic (e.g. swim, push a cart or running).

• Accomplishments, that are dynamic, durative and telic (e.g. draw a circle or recover).

• Achievements, that are dynamic, punctual and telic (e.g. recognize or reach the top).

The four-way classi cation of [START_REF] Vendler | Linguistics in Philosophy[END_REF] formed the basis for following work on lexical aspect. For instance, [START_REF] Bach | The Algebra of Events[END_REF] further subdivides the states category by introducing the notions of dynamic states, that can be expressed by verbs occurring with progressive tenses (Example 9a), and static states that hold permanently (Example 9b). Non-states are divided into processes (corresponding to Vendler's activities) and events (Vendler's accomplishments and achievements). These events can be protracted (accomplishments) or momentaneous (achievements). He also subdivides the momentaneous category between culminations (Example 9c) and happenings (Example 9d) according to whether the event involves a transition. A overview of other linguistic theories of lexical aspect can be found in Marşic's thesis [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF].

(9) a. Mary is feeling sick.

b.

John knows the answer. c. John's father died a few years ago. d. Mary noticed John's mistake.

Temporal Relations

There are two main areas of research involving temporal relations. First, researchers tried to identify and characterize the linguistic realizations of temporal relations in natural language. Second, there is a line of work, mostly in the eld of knowledge representation, which tries to model the temporal relations using algebra in order to be able to reason about time.

Linguistic Realization

There are several mechanisms used in natural language to denote temporal relations between events and/or time expressions. All examples in this section are extracted from [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF].

Tense. According to K. [START_REF] Brown | Encyclopedia of Language and Linguistics[END_REF], tense is a grammatical category that serves to locate situations in time. It allows to locate events in time relatively to the time of utterance, also called speech time. There are three main tenses in English:

• • Past tense expresses that the event took place in the past (before speech time). Past tense can be anaphoric, thus events are interpreted relatively to other times expressed in the previous or current utterances as shown in Example 11.

(11) John went to the park and saw many squirrels.

• Future Tense expresses that the event takes place after the speech time. In English, future tense can be expressed by the modal auxiliary construction with will, be going to followed by the in nitive, simple present or be to/be about to followed by the in nitive. Time Adverbials. These adverbials convey temporal relations between the time they denote and the verbal event they depends on. They are realized by time expressions as shown in Example 13. [START_REF] Marşic | Temporal Processing of News: Annotation of Temporal Expressions, Verbal Events and Temporal Relations[END_REF] makes the distinction between time position adverbials (Example 13a), durative adverbials (noun phrases and prepositional phrases introduced by for, during, within, over, throughout, while, whilst, as long as, so long as, until, till, up Other Means. There are other means of expressing temporal relations in natural language.

Grammatical

They can be expressed at the syntactic level as in Example 14 where the event result is quali ed by a temporal expression the Sunday Election.

(14) They do not know the result of the Sunday Election.

They can also be inferred using world knowledge. This is the case for subevents as shown in Example 15 where world knowledge is necessary to interpret that the event painted is included in the event redecorated. The narrative sequence, i.e. the order of natural language utterances, can be used to infer temporal relations. In Example 17, utterance order allows us to place the event cooked after the event went.

(17) Mary went home. She cooked dinner and ate it in front of the TV.

Event coreference can also be used to infer temporal relations. If one event is mentioned several times, then temporal relations involving one mention also apply for other mentions.

Finally, inference (e.g. transitivity) allows for inferring temporal relations. If event A occurred before event B and event B occurred before event C then event A occurred before event C.

Temporal Reasoning

Once we have identi ed the explicit and implicit relations in text, we must devise a system for reasoning about them (e.g. for being able to infer other relations). This problem has been addressed in the literature under the notion of temporal algebra. A temporal algebra allows to deduce relationships between events based on their connections to other times and events using a set of rules [START_REF] Derczynski | Automatically Ordering Events and Times in Text[END_REF].

Early work in the domain can be traced back to [START_REF] Reichenbach | Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging[END_REF] who modeled events as points in time. In his model, an event duration is a series of points between a start and an end point. He introduces a set of three relations between event start and end points: before, simultaneous and after. The model has some drawbacks. Although it works well for events like love or know where one can interpret that the event is true for every point in the set, it does not work for event like draw a circle where at every point before the completion of the drawing, a circle has not yet been drawn [START_REF] Bethard | Finding Event, Temporal and Causal Structure in Text: A Machine Learning Approach[END_REF]. The model presented by [START_REF] Reichenbach | Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging[END_REF] is thus not well suited for modeling temporal relations in natural language. Following work on temporal algebra can be divided into three main categories.

Temporal Interval Logic. The most in uential work in this domain was conducted by [START_REF] Allen | Maintaining Knowledge About Temporal Intervals[END_REF] who devised a model where events and times are represented as continuous intervals with start and end points. It is a graphical model where event and time expressions are the nodes and temporal relations are the edges that connect the nodes. [START_REF] Allen | Maintaining Knowledge About Temporal Intervals[END_REF] identi es thirteen interval-interval relations (i.e. edges) and provides an algorithm for relation inference (e.g. using temporal transitivity). [START_REF] Galton | A Critical Examination of Allen's Theory of Action and Time[END_REF] identi es some aws in Allen's model. The major drawback is that the theory does not account for continuous change. The author takes the example of a ball thrown vertically in the air. There are two intervals, one for the ball going up, one for the ball going down. There is a moment in time where the ball is neither in these intervals. Allen's theory would need to represent this moment as an interval but this is contradictory with the laws of physics. [START_REF] Jixin | A General Temporal Theory[END_REF] tried to remedy to this problem by proposing to represent points as zero-length intervals.

Semi-Interval Reasoning. [START_REF] Derczynski | Automatically Ordering Events and Times in Text[END_REF] underlines that temporal logic intervals are not perfect. For instance they become intractable with medium and large scenarios and, as we discussed above, there are some problems when intervals have a duration of zero. Also, some scenarios does not t well in Allen's interval logic. Consider Example 18. Boundaries of the event control are di cult to conceptualize. The start boundary is probably the utterance time (or publication time in this case) but it is di cult to place its end boundary as the event may be nished at the time of reading or may continue in the future.

(18) Today, rebels still control the air eld and surrounding area. [START_REF] Derczynski | Automatically Ordering Events and Times in Text[END_REF] Semi-interval reasoning is devised to handle such cases by allowing to de ne only one temporal boundary of the events. One example of such model is the one of Freksa (1992) who proposes a model which allows to tackle uncertainty about events and allows to capture information from text that may not describe completely all intervals.

Point-based Reasoning. Point-based reasoning allows to model events and times as individuals points rather than intervals. It is possible to decompose intervals with begin and end points. This model type su ers from the fact that annotating text using points rather than intervals complicates the annotation task [START_REF] Derczynski | Automatically Ordering Events and Times in Text[END_REF]. Furthermore, temporal relations in text are better represented by interval or semi-interval logics rather that points. However, one major bene t of point-based reasoning is that it is very fast to process [START_REF] Verhagen | Times Between the Lines[END_REF]).

Time in Computational Linguistics

Analyzing time from a linguistic perspective is helpful to understand the mechanisms involved in the realization of time in natural language. In order to being able to model time from a computational linguistics perspective, we must settle on a linguistic model and derive an annotation schema that will be used to annotate time in text.

In this section, we present the di erent approaches that have been devised in the computational linguistics literature to model the three primitives of time: temporal expressions, events and temporal relations.

Temporal Expressions: From TIMEX to TIMEX3

There are four main types of temporal expressions in the literature [START_REF] Strötgen | Domain-Sensitive Temporal Tagging[END_REF]: dates, times, durations and sets. These expressions may be explicit, implicit, relative or underspeci ed [START_REF] Strötgen | A Baseline Temporal Tagger for All Languages[END_REF]. We note that this categorization corresponds roughly to the linguistic description from Section 2.2 in which temporal expressions may denotes calendar dates, times of day, durations or sets of recurring times. Furthermore, the linguistic distinction between absolute and relative position in time is kept.

There were several attempts to model temporal expressions in the computational linguistic literature. In this section, we describe the most in uential models: the TIMEX model series.

TIMEX. This is one of the earliest attempt to create an annotation scheme for temporal expressions. It was developed for the MUC-5 conference [START_REF] Sundheim | Tipster/MUC-5: Information Extraction System Evaluation[END_REF]. The conference included a shared task on NER that involved the extraction and categorization of dates and times. The task was proposed again in following MUC conferences until MUC-7 [START_REF] Chinchor | Overview of MUC-7/MET-2[END_REF].

The goal of the task was to identify and annotate time expressions that denote calendar dates or times with one TIMEX tag. The tag had one type attribute that could take the value date or time. There was no task related to time expression normalization. TIMEX extraction and classi cation were part of a bigger task related to slot lling in which participants were asked to assign times to events (e.g. rocket launching dates).

TIDES TIMEX2.

The second version of TIMEX was developed under the Defense Advanced Research Projects Agency (DARPA) research project Translingual Information Detection, Extraction and Summarization (TIDES) and the Automatic Content Extraction (ACE) Program. The development of this speci cation spanned over ve years between 2000 and 2005. The last and nal version of the annotation scheme is described in [START_REF] Ferro | TIDES-2005 Standard for the Annotation of Temporal Expressions[END_REF]. The TIDES TIMEX2 annotation scheme, materialized by the tag TIMEX2, aimed at annotate a wider range of English temporal expressions. The TIMEX2 tag includes six attributes:

• VAL: it contains the ISO-8601 normalized value of the temporal expression when it represents a point or interval on a calendar or a clock.

• MOD: it is lled in when the time expression includes a modi er (e.g. no more than or approximately).

• ANCHOR_VAL and ANCHOR_DIR: these two attributes are used together to indicate the orientation and anchoring of time expressions.

• SET: it is used to mark time expressions that are representing sets of time. Its only possible value is YES. The absence of the attribute implies that the time expression is not representing a set.

• COMMENT: this attribute may be used by annotators to justify decisions for ambiguous time expressions or to signal doubts during the annotation process.

TIMEX3.

The third and last version of the TIMEX annotation scheme has been developed within the TimeML [START_REF] Pustejovsky | The speci cation language TimeML[END_REF]) and ISO-TimeML [START_REF] Pustejovsky | ISO-TimeML: An International Standard for Semantic Annotation[END_REF] projects which aimed at creating a formal speci cation for representing events, temporal expressions and temporal relations in natural language. The TIMEX3 annotation scheme is heavily based on its predecessors. The main di erence is that the model does now includes two new types: duration and set. [START_REF] Strötgen | Domain-Sensitive Temporal Tagging[END_REF] reports other di erences with the TIMEX2 annotation scheme. Events are no longer be parts of temporal expressions. This is also the case for speci c pre-and post-modi ers or time expressions. Also, the TIMEX3 tag cannot be nested. Its main attributes are:

• TYPE: it allows to specify the type of the annotated temporal expression. Possible values are date, time, duration and set.

• VALUE: as for the TIMEX2 tag, it contains the ISO-8601 normalized value of the temporal expression.

• MOD: this attributes allows to specify a modi er when necessary.

• QUANT and FREQ: these two attributes allow to specify the quantity and frequency of time expressions denoting sets.

• BEGINPOINT and ENDPOINT: they are used when the related expression is a duration. They allow to specify begin and end points of the duration.

• TEMPORALFUNCTION: it is an optional attribute that allows to model anaphoric temporal expressions. It is used in conjunction with either ANCHORTIMEID whose value points to the anchor identi er or VALUEFROMFUNCTION whose value points to another temporal function.

Clinical Domain. Although the TIMEX3 annotation scheme was originally developed for the annotation of temporal expressions in various textual genres and domains, its attributes and their possible values may not re ect the particularities of a given domain. This is the case for clinical narratives for which several adaptations have been implemented in following work. Most of the adaptations involved the simpli cation of the attribute structure. W. Tapi [START_REF] Nzali | Automatic Extraction of Time Expressions Accross Domains in French Narratives[END_REF] devised an approach for automatic extraction of temporal expressions across domains in French narratives. Studying three corpora (news-based, historical and clinical), they show that time expression distribution is domain-speci c. Among other observations, the authors highlight that the clinical corpus has a high proportion of Set compared to the two other domains.

Events: Task-Dependent Modeling

Event models vary according to the task they are devised for. Besides the temporal information extraction task, which is the one we are concerned with in this thesis, we identify three other tasks in the literature: event extraction, slot lling and topic detection and tracking.

Event Extraction

In event extraction, also called event detection, the goal is to extract and classify event mentions and their arguments from text and regroup mentions according to the event they are referring to. An event is de ned as "a speci c occurrence of something that happens, often a change of state, involving participants".

The task was formalized during the ACE 2005 shared task (Linguistic Data Consortium 2005) on event extraction. The ACE program aimed at providing new datasets and annotation schemas for the development of new information extraction methods. Objects of interest included entities, times, values, relations and events.

In the context of the shared task, an event mention is a span of text, also called an extent, which is usually a sentence. Each event mention has an anchor which is the most representative word of the event mention. Within the event mention, there could be zero or more arguments. In the ACE 2005 shared task, an event is an actual real-world event, i.e. the collection of event mentions referring to it. The ACE 2005 shared task limited the scope to speci c event categories and argument positions.

System performance was evaluated with two methods. The rst one was designed as an end-to-end evaluation that takes into account the NER phase (entities, time expressions, values and events). For the second one, participants were given the gold named entities and were asked to perform argument extraction.

This type of modeling inspired later work in both general [START_REF] Mitamura | Overview of TAC-KBP 2015 Event Nugget Track[END_REF]) and biomedical domains. In the successive versions of the BioNLP shared task [START_REF] Kim | Overview of BioNLP'09 Shared Task on Event Extraction[END_REF][START_REF] Kim | Overview of BioNLP Shared Task 2011[END_REF][START_REF] Kim | Refactoring the Genia Event Extraction Shared Task Toward a General Framework for IE-Driven KB Development[END_REF][START_REF] Kim | The Genia Event Extraction Shared Task, 2013 Edition: Overview[END_REF], participants were challenged on biomedical event extraction. The task involved extracting proteins, their types and their arguments. These tasks were part of bigger tasks such as protein coreference resolution, entity relation extraction or gene renaming tasks. Moreover, several tasks were concerned with the detection of event negation and speculation.

Slot Filling and Knowledge Base Population

In slot lling based shared tasks, the goal is not anymore the annotation of event mentions and anchors in text but rather to identify knowledge about entities and events. Participants are given a xed inventory of relations and attributes and are asked to ll in these slots with values extracted from the text. Slots can be related to the type, the agent, the time and place or the e ect. This task can be considered as something hybrid between relation extraction and question answering.

The rst instance of this task can be traced back to the MUC conferences [START_REF] Grishman | Message Understanding Conference-6: A Brief History[END_REF]. Targeted topics changed regularly: naval sightings and engagements (MUC-1 and MUC-2), terrorist attacks (MUC-3 and MUC-4), joint ventures (MUC-5), succession events (MUC-6) or airplane crashes, and rocket/missile launches (MUC-7).

More recently, the NLP community started to focus on knowledge base population. The Text Analysis Conferences (TAC) introduce in 2014 a shared task [START_REF] Mitamura | Overview of TAC-KBP 2015 Event Nugget Track[END_REF] where the goal is to ll event slots with relations such as org:founded_by (who) or org:date_founded (when).

Topic Detection and Tracking

The Topic Detection and Tracking (TDT) task was a DARPA-sponsored shared task [START_REF] Allan | Topic Detection and Tracking Pilot Study: Final Report[END_REF][START_REF] Allan | On-line New Event Detection and Tracking[END_REF]. The main objective of the task was to nd and follow events in a stream of broadcast news stories. The rst iterations of the task were concerned with topics but the following years, the task switched to events, i.e. things that happen at a point in time. There were several subtasks:

• Segmentation: participants were asked to segment a stream of text into stories.

• Detection: the objective is to identify the events discussed in the stories. Two tracks were organized: retrospective and on-line detection. In both cases, the task was modeled as a clustering problem were participants must regroup stories according to the event they are discussing.

• New event detection: the goal is to mark stories that are discussing an event for the rst time.

• Tracking: participants are asked to associate stories with known events.

TDT is hard for both the organizers and the participants and was not further developed in later years until recent years. The recent development of micro-blogging platforms such as Twitter led a regain of interest for the task among the NLP community. [START_REF] Atefeh | A Survey of Techniques for Event Detection in Twitter[END_REF] presents a survey of models for event detection in Twitter. The authors identify tasks that are similar to those which were put into place for TDT in traditional media: detection of unspeci ed vs. speci ed (or planned) events and the retrospective vs. on-line detection.

Temporal Information Extraction

The last task involving event detection is temporal information extraction. This is the task we are concerned with in this thesis. In this task, the event is considered as an atomic object, generally represented by a word in a text. The goal is not to extract arguments as in previous models but rather to place an event in time relatively to other events and time expressions. The task is usually divided in three blocks:

• Event extraction: the objective is to extract event extents which are usually limited to one token.

• Time expression extraction: the objective is to extract time expressions usually modeled as TIMEX3.

• Temporal relation extraction: the objective is to extract temporal relations between events and/or temporal expressions but also between events and document creation times.

The most prominent shared tasks organized in the community around temporal relation extraction are the TempEval campaigns [START_REF] Uzzaman | SemEval-2013 Task 1: TempEval-3 -Evaluating Time Expressions, Events, and Temporal Relations[END_REF][START_REF] Verhagen | SemEval-2007 Task 15: TempEval Temporal Relation Identication[END_REF][START_REF] Verhagen | SemEval-2010 Task 13: TempEval-2[END_REF] which o er to work on news-based corpora (cf. Section 2.4.2). The annotation schema used to annotate these datasets is based on ISO-TimeML [START_REF] Pustejovsky | ISO-TimeML: An International Standard for Semantic Annotation[END_REF], cf. Section 2.4).

Temporal information extraction was originally addressed in the news domain. The specication TimeML [START_REF] Pustejovsky | The speci cation language TimeML[END_REF]) and the proof of concept corpus TimeBank [START_REF] Pustejovsky | The TIMEBANK Corpus[END_REF] annotated with this schema paved the way for the development of new methods for temporal information extraction.

Clinical Domain. In the meantime, the clinical community started expressing a strong interest in temporal information extraction from clinical narratives (Bramsen et al. 2006a;[START_REF] Hripcsak | Syndromic Surveillance Using Ambulatory Electronic Health Records[END_REF]). However, it came quickly to light that the notion of event is di erent between the general and the clinical domains.

First, medical sta is not interested in all events that appear in the text but only those which are medically relevant (Galescu and Blaylock 2012;Styler IV et al. 2014a).

Second, the forms of events, i.e. the linguistic realization of these medical events, di ers from the ones found in the general domain. The ISO-TimeML speci cation mentions that events may be expressed by verbs, nouns and adjectives. The majority of nouns in this case are nominalizations of verbal events (e.g. the detonation). Similar cases can be found in the clinical domain (e.g. intubation) but most events in the clinical sense, are noun phrases that would not be annotated according to the ISO-TimeML speci cation. Galescu and Blaylock (2012) identify this phenomenon while annotating temporal relations between medical problems, tests and treatments. In a majority of cases, these entities are not eventive in the sense of ISO-TimeML but are considered as representative of the event they are the most closely related. Styler IV et al. (2014) explains that this interpretation of disorder, medication and procedure as rightful events comes from the fact that the medical sta does not discuss them without an associated (implicit) event. In the light of this observation, Styler IV et al. (2014) infer that all entities falling into the following Uni ed Medical Language System (UMLS ® ) categories can be considered as events: Disorder, Chemical/Drug, Procedure and Sign/Symptom.

Temporal Relations

The last piece of information to be modeled in order to have a fully functional annotation scheme is the temporal relation set. One of the rst attempt to model temporal relations was done during the MUC-7 shared task [START_REF] Chinchor | Overview of MUC-7/MET-2[END_REF], where participants were asked to provide a launch date for rocket launching events. Later, [START_REF] Katz | The Annotation of Temporal Information in Natural Language Sentences[END_REF] annotated a set of fty sentences with intra-sentence temporal relations. The corpus was also annotated with syntactic information. Relations were drawn between verbs and utterance times as well as between pairs of verbs. There were only two relations in the set: precedence and inclusion. [START_REF] Setzer | On the Importance of Annotating Event-Event Temporal Relations in Text[END_REF] annotated a trial corpus of six newswire articles using the She eld Temporal Annotation Guidelines (STAG) [START_REF] Setzer | Temporal Information in Newswire Articles: an Annotation Scheme and Corpus Study[END_REF]. They devised an iterative annotation process where the rst step consists in annotating events, time expressions, signals and temporal relations that are explicit in the text or syntactically implicit. Then, they derive all inferrable annotations based on this rst phase. They repeat the two steps until every pair is annotated. One major drawback emerging from that annotation e ort is the low inter-annotator agreement. The authors identify several causes: the underspeci ed annotation guidelines, the lack of annotator training and the task di culty. [START_REF] Pustejovsky | The speci cation language TimeML[END_REF] and [START_REF] Pustejovsky | ISO-TimeML: An International Standard for Semantic Annotation[END_REF] presented the ISO-TimeML Specication which has become the basis of most following temporal annotation e orts. The annotation scheme allows to annotate events, time expressions, signals, and temporal relations between these entities. The temporal relation set is derived from Allen's interval relations: before, after, includes, is included, during, during inv, simultaneous, iafter, ibefore, identity, begins, ends, begun by and ended by. They annotated the TimeBank corpus [START_REF] Pustejovsky | The TIMEBANK Corpus[END_REF]) as proof of concept, which was used in a simpli ed version in the TempEval shared tasks [START_REF] Uzzaman | SemEval-2013 Task 1: TempEval-3 -Evaluating Time Expressions, Events, and Temporal Relations[END_REF][START_REF] Verhagen | SemEval-2007 Task 15: TempEval Temporal Relation Identication[END_REF][START_REF] Verhagen | SemEval-2010 Task 13: TempEval-2[END_REF] Clinical Domain. In the clinical domain, several temporal annotation guidelines have been devised upon the work of [START_REF] Pustejovsky | ISO-TimeML: An International Standard for Semantic Annotation[END_REF] and led to the creation of related corpora. Among these e orts, the THYME-TimeML speci cation (Styler IV et al. 2014a) proposes a full temporal annotation scheme adapted to the clinical domain. The authors annotated a corpus of 1,200 documents that have been used in the Clinical TempEval challenges [START_REF] Bethard | SemEval-2015 Task 6: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF].

One major modi cation of TimeML brought by THYME-TimeML lies in the temporal relation set. Instead of considering classic Allen's relations, the authors embrace the narrative container concept [START_REF] Pustejovsky | Increasing Informativeness in Temporal Annotation[END_REF]. According to Styler IV et al. (2014), a narrative container is a temporal bucket in which several events may fall. These containers may be realized by temporal expressions, durative events or abstract medical concepts. Another way to visualize the narrative container concept is to think of it as the imbrication of temporal intervals. Example 21, taken from Styler IV et al. (2014), illustrates the concept. In this example, the event colonoscopy happened on January 7, 2010. Thus, the temporal expression associated with the calendar date is marked as containing the event colonoscopy. All other events mentioned in the example are included in the event colonoscopy. Styler IV et al. (2014) explain that using the narrative container concept instead of classical relations allows for better annotation quality by improving the inter-annotator agreement while limiting under-and over-annotation e ects that are common in temporal annotation e orts. The authors add that this approach mimics closely the general structure of story-telling in both the general and clinical domains. Medical sta tend to cluster discussions of medical events around a given date or other event. Placing events in narrative containers and linking these containers with a few relations allows to draw quickly a useful understanding of the overall clinical timeline. Events within clusters are not ordered but the ordering can be found a posteriori with domain knowledge. The authors argue that this annotation strategy o ers a good balance between temporal informativeness and annotation quality.

schemes used in the literature to build temporally annotated corpora. Second, we describe the main annotated corpora that exist in both the general and the clinical domains.

Full Annotation Schemes

There are two main annotation schemes that have been developed in computational linguistics. ISO-TimeML [START_REF] Pustejovsky | ISO-TimeML: An International Standard for Semantic Annotation[END_REF]) is a general markup language for annotating temporal expressions, events and temporal relations in text. THYME-TimeML (Styler IV et al. 2014a) is an adaptation of ISO-TimeML to the clinical domain.

ISO-TimeML

ISO-TimeML [START_REF] Pustejovsky | ISO-TimeML: An International Standard for Semantic Annotation[END_REF]) is an ISO standard for temporal information markup in text. It is a standardization of TimeML [START_REF] Pustejovsky | The speci cation language TimeML[END_REF], a general-purpose markup language for time. This speci cation allows to annotate events, time expressions, relations between events and/or time expressions and the relative ordering of events.

Temporal expressions are captured by TIMEX3 tags (described in Section 2.3.1). Minor di erences with the original TIMEX3 annotation scheme includes the limitation of possible modi er values to a restricted set and the addition of an attribute which allows to specify the function of the temporal expression within the document (creation-time, modi cation-time, publication-time, release-time, reception-time, expiration-time or none).

Events are modeled with the combination of EVENT and MAKEINSTANCE tags. The authors distinguish between event triggers and event instances. The MAKEINSTANCE tag holds most of the attributes and creates the actual realization of the event. Such scheme allows to model complicated examples such as the one presented in Example 22 where two event instances are needed.

(22) John taught on Monday and Tuesday.

<EVENT eid="e1">taught</EVENT> <MAKEINSTANCE eiid="ei1" eventID="e1" tense="PAST" aspect="NONE" pos="VERB"/> <MAKEINSTANCE eiid="ei2" eventID="e1" tense="PAST" aspect="NONE" pos="VERB"/> According to the annotation guidelines [START_REF] Saurí | TimeML Annotation Guidelines: ffersion 1[END_REF], the authors consider events as a cover term for situations that happen or occur. Only one word should be annotated, usually the head of the minimal chunk expressing the event (e.g. a verb, a noun or an adjective e. "There is no reason why we would not be prepared", Mordechai told the Yediot Ahronot daily. f. All 75 people on board the Aero ot Airbus died.

The main event instance attributes are:

• CLASS: it allows to distinguish between classes of events. An event can be of the type occurrence, perception, reporting, aspectual, state, i-state or i-action.

• TENSE and ASPECT allows to capture common distinctions between categories of verbal phrases. The TENSE attribute can take the following values: past, present, future, none, in nitive, prespart or pastpart. The ASPECT attribute can take the following values: progressive, perfective, imperfective, prefective-progressive, imperfective-progressive, none. Non nite verbs are marked with either in nitive, prespart or pastpart values.

• POS: this attribute allows to specify the syntactic category of the text extent (usually one token) marked as an event. It can take the value adjective, noun, verb, preposition or other.

• POLARITY: this attribute allows to specify if the event is negated or not. It can take the values neg or pos.

• MODALITY: the attribute allows to mark event modality. Examples of possible values are must or should.

• CARDINALITY: this attribute is used if there is a modi er denoting a cardinality. For instance, Example 22 can also be modeled with a unique MAKEINSTANCE element whose CARDINALITY attribute will be 2.

Temporal relations can happen between two events, two times or between an event and a time. They are modeled as TLINKs with several attributes: [START_REF] Saurí | TimeML Annotation Guidelines: ffersion 1[END_REF].

(24) a. The police looked into the slayings of 14 women. In six of the cases suspects have already been arrested. b. All passengers died when the plane crashed into the mountain. c. John arrived in Boston last Thursday. d. James was CTO for two years. e. John was in the gym between 6:00 p.m. and 7:00 p.m. f. John was in the gym between 6:00 p.m. and 7:00 p.m.

• EVENTINSTANCEID or TIMEID: identi cation label of the source entity (event or time expression) involved in the relation.

• RELATEDTOEVENTINSTANCE or RELATEDTOTIME: identi cation label of the target entity (event or time expression) involved in the relation.

Besides TLINKs which are used to mark temporal relation between entities, there are two other link types that are used to annotate modality and aspect:

• SLINK tags are used to mark subordination links between two events. This relation can be modal, factive, counter-factive, evidential, negative-evidential or conditional.

• ALINK tags are used to mark the relation between an aspectual event and its argument event. Possible values are initiation, culmination, termination, continuation or reinitiation.

Finally, SIGNAL tags are used to mark text extents that make explicit the relation holding between two entities. Signals can take the form of temporal prepositions (e.g. on, in or at), temporal conjunctions (e.g. before or after), prepositions signaling modality (e.g. to) or special characters (e.g. the character / which denotes a range as in 1998/1999). Signal may be referenced in relations expressed with TLINKs, SLINKs or ALINKs.

THYME-TimeML

THYME-TimeML (Styler IV et al. 2014a) is a temporal annotation scheme developed specically for the clinical domain. As its name suggests, it is strongly inspired by the ISO-TimeML speci cation described in the previous section. Three types of elements are annotated: clinical events, temporal expressions and temporal relations between these two entity types as well as between events and document creation times. All examples in this section are taken from Styler IV et al. (2014).

As previously mentioned in Section 2.3.2, Styler IV et al. (2014) extend the de nition of an event proposed in [START_REF] Pustejovsky | ISO-TimeML: An International Standard for Semantic Annotation[END_REF] by including all pertinent events for the clinical timeline. In this context, events can be diagnostics, diseases, procedures and tumors. Also, AJCC tumor type codes (American Join Committee on Cancer Staging Codes) are annotated. These codes are events in the sense that they are assigned to a patient at a given point in his medical history. According to ISO-TimeML, they would not be annotated. However, they represent valuable information for medical sta . In this context, verbs of discussions such as talk, agree and repeat are also annotated due to the fact that they are important for legal reasons. Moreover, the annotation speci cation formalizes the notion of entities as events that has been devised in previous research e orts (e.g. in A. [START_REF] Roberts | Building a Semantically Annotated Corpus of Clinical Texts[END_REF]). Thus treatments (Example 25a) and diseases (Example 25b) can be considered as events.

(25) a. [ EVENT Levaquin] 750 mg p.o. q. day (will restart today).

b. The [ EVENT CT] showed a small rectal [ EVENT abcess].

Styler IV et al. ( 2014) made several modi cations to the ISO-TimeML speci cation to simplify the annotation process and better t the clinical domain. To improve the interannotator agreement, event modality is not modeled with SLINK anymore but with three event attributes.

The contextual modality attribute can take four values. An event is considered as actual if it did happen in the real world (Example 26a). Conditional or hypothetical events will be marked as hypothetical (Example 26b). Some events, such as diagnostics, can be expressed with caution by the medical sta in order to avoid legal repercussions. These events will be marked as hedged (Example 26c). Finally, events discussing a treatment or a disease from a generic point of view will be marked as generic (Example 26d). ( 26 The contextual aspect attribute allows to distinguish between intermittent events (e.g. vomiting), constant events (e.g. fewer) and new events (e.g. discovering a new tumor). The attributes will respectively take the values intermittent (Example 27a), n/a and novel (Example 27b). Finally, the permanence attribute allows to express the di erence between the medical concepts acute and chronic. A disease will be considered chronic if it is uncurable. In other cases, it will be marked as nite. Although this feature was originally bound to be annotated, the authors dropped it after a few annotation iterations due to the fact that the task demands a lot of medical knowledge.

Besides event modality, there are other event properties marked in the corpus. The type attribute allow to distinguish between three event types. Aspectual events encode aspectual information about other events (Example 28a). Evidential events allow to link an information source (e.g. radiography) to an observation/diagnostic based on this source (e.g. broken rib) as in Example 28b. Other events will be marked as n/a.

(28) a. The rash has not [ ASPECTUAL reappeared] and we will monitor closely. b. Her CT-scan [ EVIDENTIAL showed] a small mass in the right colon.

The polarity attribute encodes the fact that an event took place in the real world (Example 29a) or did not (Example 29b). The degree attribute allows to nuance polarity of an event. Thus, an event can be a little positive (Example 30a) or almost positive (Example 30b). Similarly, an event may be considered as little negative or almost negative.

(30) a. There is a small amount of bright T1 [ LITTLE signal].

b. Abdominal tenderness has nearly [ MOST disappeared].

Temporal expressions are modeled using TIMEX3 tags. The annotation scheme uses all the types de ned in ISO-TimeML and adds a speci c prepostexp category for temporal expressions such as preoperative, intraoperative and postoperative.

TIMEX3 tags have only one attribute class which can take several values. The attribute will take the value date (Example 31a) if the temporal expression denotes a calendar date. It can also take the value time if the expression denotes a speci c time in the day (Example 31b). The expression will be marked as a duration if it denotes a duration (Example 31c). The expression can also be marked as a quanti er (Example 31d) or a set (Example 31f) whenever it combines both a duration and a quantity. Finally, the attribute will take the value prepostexp if the temporal expression denotes the temporal phenomenon presented above (Example 31e). As we mentioned in Section 2.3.3, Styler IV et al. (2014) embrace the narrative container concept to annotate temporal relations between events and/or temporal expressions. However, other types of relations are annotated to model temporality within containers. An entity can be placed before an other entity (Example 32a). Both entities can begin at the same time (Example 32b) or ends at the same time (Example 32c). Finally, two entities can temporally overlap with each other (Example 32d). Only linguistically apparent relations are annotated.

(32) a. She vomited shortly before surgery.

b. She has had abdominal cramping since January. c. She has had no bleeding since her stitches were removed. d. The patient's rst MI occurred while she was undergoing chemotherapy.

The second type of temporal relation is the relation existing between events and document creation times. Hence, an event can happen before (Example 33a) or after (Example 33b) the document creation time. It may have started before the document creation time and still be on-going at that time (before-overlap, Example 33c). Finally, it can be considered as true at the document creation time (overlap, Example 33d). Document creation time is the same time at which the clinical examiner has seen the patient even if the document has been written after the meeting.

(33) a. This is unchanged and may be related to treatment

Corpora and Associated Shared Tasks

Several corpora for temporal information extraction have been devised in both general and clinical domains. In this section, we describe the resources based on ISO-TimeML and THYME-TimeML speci cations, which were discussed in the previous section. Table 2.1 compares these resources along four axes: languages, document types, annotation schemes and sizes.

The TimeBank and the AQUAINT TimeML Corpora

The TimeBank corpus [START_REF] Pustejovsky | The TIMEBANK Corpus[END_REF]) is a proof-of-concept resource for the ISO-TimeML speci cation. It contains 183 documents (≈ 61,000 tokens) annotated according to the speci cation. It includes biographies, description of events and broadcast news and newswire.

The AQUAINT TimeML Corpus, also known as the Opinion Corpus, is a small resource containing 73 news reports (≈ 38,000 tokens) annotated according to the ISO-TimeML specication.

The work of [START_REF] Pustejovsky | The TIMEBANK Corpus[END_REF] inspired other researchers in creating temporally annotated corpora in other languages based on the same speci cations. Among them, [START_REF] Bittar | French Time-Bank: An ISO-TimeML Annotated Reference Corpus[END_REF] led an annotation e ort on a comparable French corpus. [START_REF] Asahara | BCCWJ-TimeBank: Temporal and Event Information Annotation on Japanese Text[END_REF] and [START_REF] Caselli | Annotating Events, Temporal Expressions and Relations in Italian: the It-Timeml Experience for the Ita-TimeBank[END_REF] did the same with Japanese and Italian texts. Other e orts include the translation of the TimeBank corpus in Portuguese [START_REF] Costa | TimeBankPT: A TimeML Annotated Corpus of Portuguese[END_REF] and Romanian [START_REF] Forascu | Romanian TimeBank: An Annotated Parallel Corpus for Temporal Information[END_REF].

The TempEval Corpora

The TempEval corpora are a series of resources used in the TempEval shared tasks [START_REF] Uzzaman | SemEval-2013 Task 1: TempEval-3 -Evaluating Time Expressions, Events, and Temporal Relations[END_REF][START_REF] Verhagen | SemEval-2007 Task 15: TempEval Temporal Relation Identication[END_REF][START_REF] Verhagen | SemEval-2010 Task 13: TempEval-2[END_REF]). The corpora were annotated according to a simpli ed version of ISO-TimeML. A version of TimeBank following this new annotation scheme was provided to the participants as training corpus. The temporal relation set comprises six relations: before, after, overlap, before-or-overlap, overlap-or-after and vague.

The rst edition of the evaluation campaign [START_REF] Verhagen | SemEval-2007 Task 15: TempEval Temporal Relation Identication[END_REF]) focused on temporal relation extraction between EVENTs and TIMEX3s within the same sentence, between EVENTs and document creation times and between two main EVENTS of two consecutive sentences.

The second and third editions of TempEval [START_REF] Uzzaman | SemEval-2013 Task 1: TempEval-3 -Evaluating Time Expressions, Events, and Temporal Relations[END_REF][START_REF] Verhagen | SemEval-2010 Task 13: TempEval-2[END_REF]) o ered a task related to EVENT and TIMEX3 extraction and further subdivided the temporal relation extraction task. Participants were asked to extract temporal relations between EVENTs and TIMEX3s which are syntactically dominated by the EVENTs, between EVENTs and document creation times, between two main EVENTS of two consecutive sentences and between EVENTs which are in a syntactic dependency relation.

The Clinical E-Science Framework (CLEF) corpus

The CLEF corpus (A. [START_REF] Roberts | Semantic Annotation of Clinical Text: The CLEF Corpus[END_REF] Table 2.1: Publicly available corpora annotated with temporal information based on either the ISO-TimeML or THYME-TimeML speci cations. We compare these resources along four axes: languages, document types, annotation schemes and sizes corpus contains documents from two patient records. The document selection process aimed at selecting documents that are representative of the various document types and lengths. Also, each of the two patient records annotated with temporal information contains nine narratives, one radiology report, seven histopathology reports and the associated structured data.

These documents have been annotated with entities but also with semantic and temporal relations. The latter are modeled as CTlinks (CLEF Temporal Links) between TLCs (Temporally Located CLEF entities). The annotation scope is limited to relations that occur between an event and a time expression. Considered events include investigations, interventions and conditions. Temporal expressions are modeled as TIMEX3 elements. There are several temporal links: after, ended by, begun by, overlap, before, none, is included, unknown and includes. The other relation type annotated is the one existing between TLCs and document creation times. The relation type set is the same as the one used for the other temporal relation.

The Corpus of Galescu and Blaylock (2012)

Galescu and Blaylock (2012) annotated a subset of the corpus used during the 2010 edition of i2b2/VA challenge on relations [START_REF] Uzuner | 2010 i2b2/VA Challenge on Concepts, Assertions, and Relations in Clinical Text[END_REF]. The authors chose only the documents that were not temporally altered by the deidenti cation process. They extracted the sections related to History of Present Illness from 97 discharge summaries from Partners Healthcare, resulting in a corpus comprising 44 sections (410 sentences and 7,701 tokens).

The authors annotated clinical events, time expressions and temporal relations between these entities. The scope of the annotation e ort is limited to intra-sentence relations. Considered events include problems, tests and treatments.

Temporal expressions are annotated as TIMEX3. Similarly to other annotation e orts, the authors annotate only the type and value attributes. The type attribute can take the following values: date, time, duration or set. One TIMEX3 is added to each document to represent the document creation time.

Temporal relations are annotated as TLINKs. They can occur between events, time expressions and between events and time expressions. Relations are the same as the ones used during the TempEval shared tasks: before, after, overlap, before-overlap, overlap-or-after or vague.

The Informatics for Integrating Biology & the Bedside (i2b2) Corpus

The i2b2 corpus (W. Sun et al. 2013a) is a temporally annotated corpus of 310 discharge summaries (≈ 178,000 tokens) from Partners Healthcare and the Beth Israel Deaconess Medical Center. The resource was used during the 2012 edition of the i2b2/VA shared task on temporal information extraction. The annotation guidelines used during its creation are based on ISO-TimeML and on an earlier version of THYME-TimeML.

Clinically relevant events are annotated with EVENT tags. These events include clinical concepts (problems, tests and treatments), clinical departments (e.g. surgery or main oor), evidentials (i.e. events that indicate a source of information such as the word complained in the patient complained about), occurrences (i.e. events that happen to the patient, such as admission, transfer or follow-up). Events have several attributes: type (problems, test or treatment), polarity (positive or negated) and modality (indicates whether an event happen, is proposed, is mentioned as conditional or as possible.)

Time expression annotation follows the TIMEX3 annotation scheme. The authors added a section time to track the section creation date. For instance, the section time for the clinical history section is the date of admission whereas the section time for the hospital course section is the discharge date. TIMEX3s have three attributes: type (date, time, duration or set), value (ISO-8601) and modi er (exact or approximate)

Temporal relations are annotated with TLINKs. They occur between two TIMEX3s, two EVENTs or between a TIMEX3 and a EVENT. The initial set of relation types included before, after, simultaneous, overlap, begun-by, ended-by, during and before-overlap. However, the authors noticed that the inter-annotator agreement was low for several relation types. They decided to merge before, ended-by and before-overlap into the single class before. They performed the same for begun-by and after which were merged into the single class after. Finally simultaneous, overlap and during were merged as overlap.

The 2012 i2b2 shared task o ered three tracks: EVENT and TIMEX3 extraction, TLINK extraction (gold entities are provided) and end-to-end extraction where participants must accomplish the two rst tasks.

The Temporal Histories of Your Medical Event (THYME) Corpus

The THYME corpus (Styler IV et al. 2014a) is set of clinical documents annotated as proof-ofconcept for the THYME-TimeML speci cation. It contains two sets of 600 documents from brain cancer and colon cancer patients at the Mayo clinic. These notes have been deidenti ed. Each set is divided in three parts: train (50%), dev (25%) and test (25%).

The corpus was used during the Clinical TempEval shared tasks [START_REF] Bethard | SemEval-2015 Task 6: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF]). In the two rst editions, only the colon cancer part was used. In the last edition of the shared task, participants were asked to perform domain adaptation by training on colon cancer related documents and testing on brain cancer related documents. At this occasion, two tracks were proposed, one without any document from the target domain (unsupervised domain adaptation) and the other with 30 documents from the target domain (supervised domain adaptation).

Several subtasks were presented to the participants: time expression extraction, event extraction and temporal relation identi cation (between events and/or time expressions but also between events and document creation times). Also, two tracks were o ered for temporal relation extraction, one where gold time expressions and events were given to the participants and one where participants were asked to perform all three tasks [START_REF] Bethard | SemEval-2015 Task 6: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF].

The Medical Entity and Relation LIMSI annOtated Text (MERLoT) Corpus

The MERLoT corpus [START_REF] Campillos | A French Clinical Corpus with Comprehensive Semantic Annotations: Development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)[END_REF]) is a corpus of 500 clinical notes written in French from a hepato-gastro-enterology and nutrition ward. The documents have been de-identi ed and pseudonymized using the Medical Information Anonymization (MEDINA) tool [START_REF] Grouin | De-Identi cation of Clinical Notes in French: Towards a Protocol for Reference Corpus Developpement[END_REF]. Documents are segmented into zones to distinguish between headers and footers from the core medical content. There are several types of documents: discharge summaries, physician letters, medical procedure reports and prescriptions. The corpus is extracted from a larger collection of documents. Sampling has been done by preserving document type proportion.

The corpus has several layers of annotations. The annotation scheme contains 12 entities: anatomy, biological process or function, chemical and drugs, concept or idea, devices, disorder, genes or proteins, hospital, living beings, medical procedure, person, sign or symptom, temporal expression. A subset of these entities are considered as events: biological process or function, chemical or drug, concept or idea, disorder, medical procedure and sign or symptom. Events are characterized by several attributes that take the form of entities linked with a relation to the events. For instance, aspectual text elements (e.g. started on or interrupted) may be linked to events via relations (e.g. start or stop). We refer the reader to the original paper for an overview of all entities and relations involving events [START_REF] Campillos | A French Clinical Corpus with Comprehensive Semantic Annotations: Development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)[END_REF]).

The temporal annotation scheme follows mostly the ISO-TimeML speci cation. Time expressions are encoded as TIMEX3 entities. However, the SIGNAL tag described in [START_REF] Pustejovsky | ISO-TimeML: An International Standard for Semantic Annotation[END_REF] is not used in this annotation e ort. Instead, the text extent that is considered as SIGNAL in ISO-TimeML is embedded in the time expression. Time expressions have one attribute type which can take the following values: date, time, duration or frequency Two types of temporal relations are annotated within the corpus. The rst one concerns relations between events and/or temporal expressions. The relation set comprises six relation types: before, begins on, during, ends on, overlap and simultaneous. The second one concerns relations between events and document creation times. It is realized as an event attribute which can take the following values: before, before-overlap, overlap, after.

Other Clinical Corpora

Several other clinical corpora can be found in the literature. [START_REF] Harkema | Mining and Modelling Temporal Clinical Data[END_REF] devised an annotation scheme based on ISO-TimeML and annotated events related to X-rays and CT-scans in 446 clinical documents. [START_REF] Mowery | Temporal Annotation of Clinical Text[END_REF] annotated 24 clinical reports with temporal information. They do not follow the ISO-TimeML speci cation and annotate directly events with temporal information. [START_REF] Savova | Towards Temporal Relation Discovery from the Clinical Narrative[END_REF] annotated a corpus of 5,000K tokens following an annotation scheme based on ISO-TimeML. All clinical and non-clinical events are annotated. The authors annotate relations between events and document creation times and event-event relations. They also annotate event modality and aspect.

Approaches for Temporal Information Extraction

Several approaches have been devised in the literature for temporal information extraction ranging from rule-based methods to fully supervised machine learning approaches. In this section, we review these approaches in both the general and clinical domains. We start with time expression extraction. Then, we discuss event extraction. Finally we review methods for temporal relation extraction.

Temporal Expression Extraction

Rule-based Approaches. Rule-based approaches have been very e cient for temporal expression extraction due to the fact that the diversity of realization in text is rather small. One of the most popular rule-based system is Heideltime [START_REF] Strötgen | Multilingual and Cross-domain Temporal Tagging[END_REF], a multilingual, cross-domain, temporal tagger. It extracts time expressions as TIMEX3 entities and provides their normalization to ISO-8601 format when applicable. Heideltime targets four document types: narrative-style, news-style, English colloquial and scienti c writing. The tool is build as a Unstructured Information Management Architecture (UIMA) module and can therefore be used in NLP pipelines. It may also be improved by the community which can provide new resources for a language or a domain (e.g. [START_REF] Moriceau | French Resources for Extraction and Normalization of Temporal Expressions with HeidelTime[END_REF] Tapi [START_REF] Nzali | Automatic Extraction of Time Expressions Accross Domains in French Narratives[END_REF] investigated temporal expression extraction in French across three domains (news, historical and medical). They devised a CRF-based system that takes domain independent features as input. They also used the output of Heildeltime in the feature set. Among other results, they showed that adapting preprocessing (e.g. tokenization) to the targeted domain yields a signi cant performance improvement. Velupillai et al. (2015) built a UIMA pipeline using ClearTK [START_REF] Bethard | ClearTK 2.0: Design Patterns for Machine Learning in UIMA[END_REF]) and SVM classi ers for TIMEX3 extraction in the 2015 edition of Clinical TempEval. The authors used simple features including POS tags, orthographic, and gazeteer information based partly on Heideltime output.

H.-J. [START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF] used a HMM-SVM sequence tagger [START_REF] Joachims | Cutting-Plane Training of Structural SVMs[END_REF]) to extract TIMEX3 spans and attributes. They used a combination of several features including lexical, syntactic, discourse-level, word representation and gazeteers features. They also derived features from the output of SUTime (A. X. [START_REF] Chang | SUTime: A Library for Recognizing and Normalizing Time Expressions[END_REF]. [START_REF] Khalifa | UtahBMI at SemEval-2016 Task 12: Extracting Temporal Information from Clinical Text[END_REF] used the output of cTAKES to generate morphological, lexical and syntactic-level features. They tested two machine learning algorithms: CRF and SVM for TIMEX3 span extraction and attribute classi cation. [START_REF] Cohan | GUIR at SemEval-2016 task 12: Temporal Information Processing for Clinical Narratives[END_REF] implemented a system based on CRF to extract TIMEX3 spans. They devised morphological features and also made use of brown clustering. Attributes were extracted with a logistic regression classi er using similar features as the one used for span detection.

Event Extraction

In this section, we focus on event extraction in the context of temporal information extraction as de ned in Section 2.3.2.

Supervised Machine Learning Approaches.

Most approaches for event extraction are data-driven and use supervised machine learning algorithms. [START_REF] Bethard | Identi cation of Event Mentions and their Semantic Class[END_REF] presented STEP (System for Textual Event Parsing), a system for TimeML event recognition based on a SVM. The authors used a rich set of features including textual, morphological, syntactic, temporal and WordNet features (hypernymy relations). March and T. Baldwin (2008) presented a system for event recognition and classi cation based on a SVM. Features include tokens and POS tags in a window around the considered word. The authors also implemented feature reduction by removing stop words.

TempEval Shared Tasks. There were signi cant advances during the TempEval shared tasks. [START_REF] Jung | ATT1: Temporal Annotation Using Big Windows and Rich Syntactic and Semantic Features[END_REF] investigated the usefulness of various features for the task. They made use of simple common features such as POS tags, tokens and lemmas but also semantic role labels. [START_REF] Bethard | ClearTK-TimeML: A Minimalist Approach to TempEval 2013[END_REF] proposed the same approach that has been used for temporal expression extraction and classi cation. Based on a SVM classi er, the author used a small set of features derived from POS tags, tokens and syntactic constituency parses. The model is integrated in the tool ClearTK. [START_REF] Kolya | JU_CSE: A CRF Based Approach to Annotation of Temporal Expression, Event and Temporal Relations[END_REF] combined features extracted from Wordnet (hypernyms, hyponyms and other semantic relations) with features derived from semantic role labels for training a CRF. The authors did not obtain good performance on the task. NavyTime [START_REF] Chambers | NavyTime: Event and Time Ordering from Raw Text[END_REF]) extracted a minimalist set of features derived from tokens, POS tags and syntactic parses to train a maxent classi er. Finally, KUL (Kolomiyets and Moens 2013) is a system based on a multi-label logistic regression classi er. The features are derived from dependency and constituency parse trees.

Clinical Domain. In the clinical domain, most systems used a sequence labeling machine learning algorithm for event span extraction and a SVM for attribute classi cation. K. [START_REF] Roberts | A Flexible Framework for Recognizing Events, Temporal Expressions, and Temporal Relations in Clinical Text[END_REF] derived features from Brown clustering (P. F. [START_REF] Brown | Class-Based n-gram Models of Natural Language[END_REF]) on large biomedical and non-biomedical corpora and used them for both event span extraction and event attribute classi cation tasks. Y.-K. [START_REF] Lin | Improving Temporal Relation Extraction with Training Instance Augmentation[END_REF] extracted features from Wikipedia and Metamap [START_REF] Aronson | E ective Mapping of Biomedical Text to the UMLS Metathesaurus: the MetaMap Program[END_REF]. [START_REF] Kovačević | Combining Rules and Machine Learning for Extraction of Temporal Expressions and Events from Clinical Narratives[END_REF] derived semantic features from cTAKES [START_REF] Savova | Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): Architecture, Component Evaluation and Applications[END_REF]).

In the Clinical TempEval campaigns, Velupillai et al. (2015) used the same UIMA module created for temporal expression extraction. H.-J. [START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF] used a the same architecture as the one use for time expression extraction (HMM-SVM sequence tagger) and used similar features. For event attribute classi cation, they trained three SVM classi ers for each of the three attributes (modality, degree and polarity) with similar features extracted in a window of 5 words around the event. [START_REF] Khalifa | UtahBMI at SemEval-2016 Task 12: Extracting Temporal Information from Clinical Text[END_REF] used the output of cTAKES to generate morphological, lexical and syntactic-level features. They tested two machine learning algorithms: CRF and SVM for EVENT span extraction and attribute classi cation. Finally, [START_REF] Leeuwenberg | KULeuven-LIIR at SemEval-2017 Task 12: Cross-Domain Temporal Information Extraction from Clinical Records[END_REF] used a SVM classi er for span and attributes extraction. They rely on a small set of features based on POS and token form. They considered single tokens as event candidates.

Relation Extraction

Temporal relation extraction is mostly performed via supervised machine learning approaches although some rule-based methods gave interesting results on TempEval corpora Rule-Based Approaches. There are a few rule based systems. [START_REF] Mani | Inferring Temporal Ordering of Events in News[END_REF] devised a rule-based model to anchor event to times and obtained good result on the TimeBank corpus. [START_REF] Hagège | XRCE-T: XIP Temporal Module for Tem-pEval campaign[END_REF] obtained the best performance on event-event relations during the rst edition of TempEval with a rule-based system based on the custom parser XIP.

Data-driven Approaches. The most e ective algorithms for temporal relation extraction implement data-driven approaches. [START_REF] Boguraev | TimeML-compliant Text Analysis for Temporal Reasoning[END_REF] presented an algorithm for temporal relation identi cation and classi cation on the TimeBank corpus. The model jointly predicts both link and labels based on features derived from a nite state parser. One limitation of the work is that the authors considered only EVENT-TIMEX3 links. The authors noticed that limiting the distance between two relation arguments to four tokens gave the best results. [START_REF] Mani | Machine Learning of Temporal Relations[END_REF] proposed a supervised approach for EVENT-EVENT and EVENT-TIMEX3 relation extraction. The features are derived from the text and entity attributes. The authors also devised simple features describing how well tenses and aspects of both relation elements are compatible. [START_REF] Hepple | USFD: Preliminary Exploration of Features and Classi ers for the TempEval-2007 Task[END_REF] experienced with several classiers and several attributes in order to nd to which extent attributes contribute to correctly classify temporal relations during the rst edition of TempEval. They found that tense and aspect were not as useful in EVENT-TIMEX3 relation classi cation as they are for EVENT-EVENT relation classi cation.

Clinical Domain. In the clinical domain, K. [START_REF] Roberts | A Flexible Framework for Recognizing Events, Temporal Expressions, and Temporal Relations in Clinical Text[END_REF] used a SVM-ranker to detect links between pairs of entities and a multi-class SVM to assign a type to these links. [START_REF] Cherry | À la Recherche du Temps Perdu: Extracting Temporal Relations from Medical Text in the 2012 i2b2 NLP Challenge[END_REF] further divided the two i2b2 tasks in four subtasks: anchoring EVENT to the section time, extraction intra-sentence EVENT-TIMEX3 relations, extracting intersentence OVERLAP relations between EVENTS and determining causal relation induced TLINKS. They devised a set of features including surface, syntax, semantic and structural features and use a Maximum Entropy algorithm to learn a model. The idea of subdividing the tasks was also adopted by other teams [START_REF] Grouin | Eventual Situations for Timeline Extraction from Clinical Reports[END_REF][START_REF] Xu | An End-to-End System to Identify Temporal Relation in Discharge Summaries: 2012 i2b2 Challenge[END_REF]. Y.-C. Chang et al. (2013) used features devised from a rule based approach for TLINK extraction in a MaxEnt component. [START_REF] Nikfarjam | Towards Generating a Patient's Timeline: Extracting Temporal Relationships from Clinical Notes[END_REF] implemented the same approach but used a SVM classi er instead of a MaxEnt.

Concerning the Clinical TempEval shared tasks, Velupillai et al. (2015) used token-level features partly derived from cTAKES output. Both temporal relation extraction tasks were addressed by using a CRF. H.-J. [START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF] divided the narrative container identi cation task into six problems according to whether the link is between two EVENTs or an EVENT and a TIMEX3 and whether the link concerned entities within one sentence, in adjacent sentences or separated with more than two sentences. They trained a SVM classi er to identify if a pair of ordered entities is linked with a narrative container. In order to take into account the imbalance of the training set, they used the transitive closure of containment relations and ltered candidate pairs that are unlikely to have a TLINK. They devised a set of lters based on the modality and document creation time relation attributes of the entities and on whether the entities are in the same section. They also applied cost-sensitive learning and assigned a weight to each class during learning. The feature are derived from cTAKES and include POS tags of the entities, tense of the sentence verb, section information and sentence type. The authors also added the distance between the two entities when applicable. [START_REF] Khalifa | UtahBMI at SemEval-2016 Task 12: Extracting Temporal Information from Clinical Text[END_REF] used the output of cTAKES to generate morphological, lexical and syntactic-level features. They tested two machine learning algorithms: CRF and SVM for document creation time relation classi cation. For narrative container relation extraction, they trained four models for intra-and inter-sentence EVENT-EVENT, EVENT-TIMEX3 relations. They did not apply any lter for negative instance sub-sampling. [START_REF] Cohan | GUIR at SemEval-2016 task 12: Temporal Information Processing for Clinical Narratives[END_REF] used a logistic regression classi er to assign a document creation time relation to each EVENT entity. They used similar features as the ones used for EVENT and TIMEX3 attribute classi cation and add speci c features derived from the sentence and nearby time and date mentions. For narrative container relation identi cation, the authors used the semantic frames of the sentence. By doing so, they only consider intra-sentence relations. They derived features from a semantic role labeler and a dependency parser.

Temporal relation extraction has also been addressed using neural networks. Dligach et al. (2017) tested Convolutional and Recurrent Neural Networks for containment relation extraction. They focus on intra-sentence relations. They tested several input sequences: token, POS sequences and a combination of both. They showed that using only POS tags instead of tokens for EVENT-TIMEX3 relation extraction impact only a little the performance of the system. Furthermore, they showed that Recurrent Neural Networks (CNNs) give better performance that LSTMs on this task. They also highlighted that the neural model is not able to give better performance than a traditional feature based system and concluded that the model may not be able to generalize well on the input for this relation type.

Global Constraint Approaches. The vast majority of approaches make independent decisions on pair of entities and do not take into account the dependencies of the local decisions in the temporal graph. [START_REF] Chambers | Jointly Combining Implicit Constraints Improves Temporal Ordering[END_REF] devised a system capable of generating consistent temporal graph. They limited the scope to before and after relations. They performed training set enhancement with closure and folding over an extended set of relations. [START_REF] Yoshikawa | Jointly Identifying Temporal Relations with Markov Logic[END_REF] used Markov logic network to model global constraints for relation typing. Bramsen et al. (2006) used Integer Linear Programmingng (ILP) for global optimization of the temporal graph. They restrained the relation set to before and after relations. [START_REF] Denis | Predicting Globally-Coherent Temporal Structures from Texts via Endpoint Inference and Graph Decomposition[END_REF] used point algebra instead of interval algebra to model the relations within the TimeBank corpus. This allows for temporal relation set reduction while keeping most of the information. The authors used ILP to nd a optimal temporal graph. The material presented in this chapter is based on three publications: one at the 2016 edition of the SemEval workshop [START_REF] Tourille | LIMSI-COT at SemEval-2016 Task 12: Temporal Relation Identi cation Using a Pipeline of Classi ers[END_REF]), one at the 2016 edition of the TALN conference (Tourille et al. 2016a) and one at the 2017 edition of the EACL conference [START_REF] Tourille | Temporal Information Extraction from Clinical Text[END_REF].

Introduction

In this chapter, we focus on the extraction of temporal relations between clinical events, temporal expressions and document creation times. More speci cally, we address intra-and inter-sentence narrative container relation extraction between events and/or temporal expressions and DCT relation extraction between events and documents. We only consider the situation where gold events and temporal expressions are given.

For containment relation extraction, the objective is to identify temporal inclusion relations between pairs of entities (event and/or temporal expression) formalized as narrative container relations (cf. Chapter 2, Section 2.3.3). For DCT relation extraction, the objective is to temporally locate events according to the DCT of the documents in which they are mentioned. DCT relations include before, before-overlap, overlap and after.

Feature-based approaches have proven to be competitive with recent neural models [START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF]. In this chapter, we describe a feature-based machine learning approach for containment and DCT relation extraction. We evaluate our approach on the THYME corpus in the context of the 2016 edition of the Clinical TempEval shared task [START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF].

Most approaches presented in the literature have been devised for processing clinical documents written in English. However, temporal information extraction in clinical text is an active research area in many other languages [START_REF] Névéol | Clinical Natural Language Processing in Languages other than English: Opportunities and Challenges[END_REF]). In the second part of this chapter, we adapt our approach for clinical narratives written in French and experiment on the MERLoT corpus.

The remainder of this chapter is organized as follows. In Section 3.2, we describe the datasets that were used in our experiments. Speci cally, we address the di erences between the two annotation sets. In Section 3.3, we present our approach for containment and DCT relation extraction. Section 3.4 presents our participation to the 2016 edition of Clinical TempEval. In Section 3.5, we present our experiments using a similar model for both English and French clinical texts. Finally, we close the chapter with a conclusion (Section 3.6).

Data

In this chapter, we use the MERLoT [START_REF] Campillos | A French Clinical Corpus with Comprehensive Semantic Annotations: Development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)[END_REF]) and THYME (Styler IV et al. 2014a) corpora. As we described in Chapter 2, the de nition of a clinical event is slightly di erent in each corpus. According to the annotation guidelines of the THYME corpus, a clinical event is anything that could be of interest on the patient's clinical timeline. It could be for instance a medical procedure, a disease or a diagnosis. For the MERLoT corpus, clinical events are described according to UMLS ® Semantic Groups and Semantic Types [START_REF] Mccray | Aggregating UMLS Semantic Types for Reducing Conceptual Complexity[END_REF]. Several categories are considered as events: disorder, sign or symptom, medical procedure, chemical and drugs, concept or idea and biological process or function.

The THYME corpus comprises several attributes for clinical events. However, not all of them were used during the Clinical TempEval challenges. In the version provided to the participants, there were ve attributes given to each event: Contextual Modality (actual, hypothetical, hedged or generic), Degree (most, little or n/a), Polarity (pos or neg), Type (aspectual, evidential or n/a) and DocTimeRel (before, before-overlap, overlap or after). Events in the MERLoT corpus carry only one DocTime attribute (before, before-overlap, overlap or after).

Both corpora are annotated with temporal expressions that take the form of TIMEX3 elements. Each TIMEX3 carries one type attribute. For the THYME corpus this attribute can take the following values: date, time, duration, quanti er, prepostexp or set. For the MER-LoT corpus, the possible values are: date, time, duration or frequency. More information about THYME and MERLoT entity attributes can be found in Chapter 2, Section 2.4. We noted earlier that the THYME and MERLoT corpora do not have the same temporal relation set (cf. Chapter 2, Section 2.4). The THYME corpus is annotated following the THYME-TimeML speci cation and therefore embraces the concept of narrative container to annotate temporal relations between medical events and/or TIMEX3. The MERLoT corpus does not explicitly cover narrative container relations. However, some transformations can be undertaken to build an equivalence with the THYME corpus. In this work, we consider that during relations are equivalent to contains relations (Figure 3.1). In addition, we also consider that reveals and conducted relations imply contains relations. Furthermore, the MER-LoT corpus does not cover inter-sentence relations (relations that can spread over multiple sentences), which is the case for the THYME corpus.

The second type of relation is the one existing between events and the DCTs. Both corpora are annotated with this relation which takes the form of an attribute on event elements. Possible values are: before, before-overlap, overlap and after.

In the rest of this chapter, we will refer to clinical events as EVENTs and to containment relation as CONTAINS relations. Corpus statistics for both MERLoT and THYME corpora are presented in Table 3.1 and Table 3.3. For the THYME corpus we present separated counts for each subpart of the corpus, including the 30-document part related to brain cancer patients that was released during the 2017 edition of Clinical TempEval. For the MERLoT corpus, we present the number of contains relations created through the transformation process described above. Only a subset of all documents are annotated with CONTAINS relations in the THYME corpus. We report statistics on this corpus part in Table 3

Model Overview

Document Creation Time Relation Subtask. We treated the subtask as a supervised classi cation problem where each EVENT is classi ed into four categories (before, before-overlap, overlap or after). The features used for the classi cation are described in the appropriate sections below.

Containment Relation Subtask. Similarly to the DCT relation subtask, we cast the containment relation subtask as a supervised classi cation problem and more particularly, as a binary classi cation task applied to pairs of EVENT and/or TIMEX3 entities. As we highlighted in the previous section, the MERLoT corpus does not cover inter-sentence relations. However these relations are annotated in the THYME corpus and were included in the Clinical TempEval challenge tasks [START_REF] Bethard | SemEval-2015 Task 6: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF]. This inclusion of both types of relations brings a di culty related to training instance creation. Indeed, considering all possible pairs of entities for building the training set without any scope restriction would lead to unbalanced training examples where the negative examples largely outnumber the positive examples. Hence, some choices have been made to reduce the number of negative examples. The analysis of the training corpus shows that a large majority -around 76% -of the CONTAINS relations are intra-sentence relations, which means that the problem of their scope actually occurs for one quarter only. The remaining relations, called inter-sentence relations, spread over at least two sentences. Given this context, we have built two separate classi ers, the rst one for the intra-sentence relations, the second one for the inter-sentence relations. This distinction has two main advantages: rst, it reduces drastically the number win. a # of rel. of negative examples, which produces better results as we observed on our development set; second, the intra-sentence classi er can bene t from a larger and richer set of features coming from sentence-level linguistic analyzers.

Concerning the inter-sentence relations, considering all pairs of EVENT and/or TIMEX3 would still give us a very large amount of negative examples. We rst observed that all of them were contained within sections (no relation overlaps section boundaries). Within the scope of a section, we further noticed that inter-sentence relations within a 3-sentence window covered approximately 89% over all existing relations. A wider window would bring too much noise while giving us a very small bump on coverage. Table 3.4 shows the number of covered relations according to the size of the window, expressed in the number of sentences. The rst line corresponds to the intra-sentence level (window=1). These statistics are computed on the train and dev parts of the THYME corpus. Following these observations, we decided to limit the scope of inter-sentence relations to a 3-sentence window without crossing any section boundary.

To further reduce the number of candidates for both inter-and intra-sentence classi ers in the MERLoT and THYME corpora, we transformed the 2-category problem (contains vs. no-relation) into a 3-category classi cation problem (contains, is-contained or no-relation).

Instead of considering all permutations of events within a sentence or a sentence-window, we considered all pairs of events from left to right, changing when necessary the contains relations into is-contained relations. This strategy allowed us to divide by a factor of two the number of candidates.

Some entities are more likely to be containers. By example TIMEX3 entities are, by nature, potential containers. This is also the case for some clinical events. For instance, a surgical operation may contain other events such as bleeding or suturing. It will not be the same with the two latter in most cases. Following this observation, we have built a model to classify entities as being a potential container or not. As we will show in Section 3.4.3, this classi er obtains a high accuracy. We used its output as feature for our intra-and inter-sentence classi ers.

Finally, we developed a rule-based module to capture speci c inter-sentence CONTAINS relations within the THYME corpus. There are some strong regularities in the handling of laboratory results where the rst temporal expression contains all the results, which are expressed with EVENTs. The module we have built aims at capturing these inter-sentential regularities with the use of rules.

To summarize, our system is composed of four modules:

1. Container detection module: entities are classi ed according to whether or not they are the source of one or more CONTAINS relations;

2. Intra-sentence relation module: combinations of entities within sentences are considered (relations contains, is-contained or no-relation);

3. Inter-sentence relation module: combinations of entities within a 3-sentence window are considered. We use the same relation classes as those used for intra-sentence relations;

4. List detection module: speci c laboratory results written as lists are handled via manual rules.

Lexical Feature Representation. We implemented two strategies to represent the lexical features in both subtasks. In the rst one, we used the plain forms of the di erent lexical features. In the second strategy, we substituted the lexical forms with word embeddings. For English, these embeddings have been computed on the MIMIC III corpus (A. E. W. [START_REF] Johnson | MIMIC-III, a Freely Accessible Critical Care Database[END_REF]. Concerning the French language, we used the whole collection of raw clinical documents from which the MERLoT corpus has been built. In both cases, we computed1,2 the word embeddings using the word2vec [START_REF] Mikolov | E cient Estimation of Word Representations in Vector Space[END_REF]) implementation of gensim [START_REF] Řehůřek | Software Framework for Topic Modelling with Large Corpora[END_REF]. We used the max (Section 3.5) or the mean (Section 3.4) of the vectors for multi-word units. Lexical contexts are thus represented by 200-dimensional vectors. When several contexts are considered (e.g. right and left), several vectors are used.

Evaluation on the THYME corpus

We evaluated our approach in the context of the second phase of the 2016 edition of the Clinical TempEval shared task [START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF]. In this phase of the competition, participants were provided with gold EVENT and TIMEX3 entities and were asked to extract CONTAINS and DCT relations. Our submission used the full pipeline presented in the previous section, including the inter-sentence relation modules.

Preprocessing and Feature Extraction

We applied a four-step preprocessing on the 440 texts that were provided for the subtasks. First, we used NLTK [START_REF] Bird | NLTK: The Natural Language Toolkit[END_REF] to segment the texts into sentences with the Punkt Sentence Tokenizer pre-trained model for English provided within the framework. The second step consisted of parsing the resulting sentences. For this task, we used the BLLIP Reranking Parser (Charniak and M. Johnson 2005) and a pre-trained biomedical parsing model [START_REF] Mcclosky | Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing[END_REF]. This step gave parse trees, POS and Coarse-grained Part-Of-Speech (CPOS) tags.

In the third step, we lemmatized the corpus using BioLemmatizer [START_REF] Liu | Bi-oLemmatizer: A Lemmatization Tool for Morphological Processing of Biomedical Text[END_REF]), a tool built for biomedical literature processing. We used the POS tags from the previous step as parameters for the lemmatization.

The last step consisted in using Metamap [START_REF] Aronson | E ective Mapping of Biomedical Text to the UMLS Metathesaurus: the MetaMap Program[END_REF] to detect biomedical events and linking them, after disambiguation, to their related Semantic Types and Semantic Groups. Semantic Types and Groups are sets of subject categories (organized as a tree) that are used to categorize concepts in the UMLS ® Metathesaurus. There are currently 133 types and 133 groups [START_REF] Bodenreider | The Uni ed Medical Language System (UMLS): Integrating Biomedical Terminology[END_REF]). We chose to keep biomedical entities that had a span overlapping with at least one EVENT of the gold standard.

For both tasks, we used a combination of structural, lexical contextual features yielded from the corpora and the preprocessing steps. We selected these features based on previous research e orts on temporal information extraction [START_REF] Bethard | SemEval-2015 Task 6: Clinical TempEval[END_REF][START_REF] Uzzaman | SemEval-2013 Task 1: TempEval-3 -Evaluating Time Expressions, Events, and Temporal Relations[END_REF][START_REF] Verhagen | SemEval-2007 Task 15: TempEval Temporal Relation Identication[END_REF][START_REF] Verhagen | SemEval-2010 Task 13: TempEval-2[END_REF]). These features are presented in Table 3.5.

Algorithm Selection

A grid search strategy was applied to select the most appropriate machine learning algorithm and its hyperparameters. When using plain lexical forms of the tokens, three algorithms were considered in our search: Random Forests, Linear SVM (liblinear) and SVM with a RBF kernel (libsvm). For the second strategy, using pretrained word embeddings, we only considered the Linear Support Vector Machine for the CONTAINS relation extraction task and Random Forests for the DCT relation extraction task.

For both strategies, 5-fold cross-validation was used to choose the algorithm and its hyperparameters. Preliminary experiments on reducing the feature set revealed that selecting the most informative features may result in a higher performance. Following this observation, we implemented statistical feature selection as part of the grid search for the strategy using plain lexical forms of the tokens, reducing progressively the number of attributes, using ANOVA F-test.

The machine learning algorithms used for the nal submission are presented in Table 3.6 together with their parameters and the percentage of the feature space kept after statistical feature selection. We used the Scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine Learning in Python[END_REF]) machine learning library for implementing our classi cation models and performing statistical feature selection.

Results

We present the cross-validation accuracies of our DCT and CONTAINER models over the development corpus in a Several paths are considered when the entities spread over more than one token. b Semantic Types and Semantic Groups of the medical entities that have been detected by Metamap and that share a span overlap with the considered EVENTs. c Classi er that predicts whether an entity is the source of one or more CONTAINS relations. a Percentage of feature space kept for nal submission (using ANOVA F-test) b Tolerance for stopping criteria c Classi er that predicts whether an entity is the source of one or more CONTAINS relations.

Table 3.6: Machine learning algorithms and hyperparameters used for our nal submission to the 2016 edition of the Clinical TempEval shared task [START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF]. lexical features. However, using words embeddings gives lower performance. Concerning the CONTAINER model, we obtain high performance with both strategies. We submitted two runs with our system for the CONTAINS and DCT relation extraction tasks, one for each strategy (plain text or word embeddings). The results for both subtasks are presented in Tables 3.8 and 3.9.

Model

Plain text Word embeddings DCT 0.873 0.778 CONTAINER a 0.917 0.924 a Classi er that predicts whether an entity is the source of one or more CON-TAINS relations. ), the number of predicted relations (pred.), the number of correct relation with and without temporal closure (corr.), precision (P), recall (R) and f1-score (F1).

Concerning the DCT relation extraction subtask, we obtained above-median scores (median score: 0.724) for both runs. The second run, which relies on word embeddings to represent the lexical features of the EVENT entities, achieves better performance. These results are consistent with what was observed during the cross-validation process using the development set. The fact that the second strategy achieves the best performance is however in contradiction with the scores obtained during cross-validation, where plain text features performed best.

In the CONTAINS relation extraction subtask, we obtained above-median F1 for the rst run (plain lexical features) and median scores for the second run (word embeddings) (median score: 0.449). Using plain lexical features gives us a more balanced system than using word embeddings. With a F1 of 0.538, our system achieves performance close to the best system (0.573), thus validating our modeling choices. These results are consistent with those we obtained when testing against the development part of the corpus. The reasons for the decrease in recall when using the second strategy are however unclear and need further investigation.

Overall, our two submitted runs achieved good performance and ranked third and fth on twenty submitted runs. We reproduce the score table presented in [START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF] in Appendix A (Table A.1).

Discussion

Our feature-based approach for DCT and CONTAINS relation extraction allows for high performance in both tasks. The fact that plain text lexical features gives a lower performance for the DCT relation extraction subtask is surprising as the top 2 best systems of the shared task used similar features and obtained scores above 0.80 F1 [START_REF] Khalifa | UtahBMI at SemEval-2016 Task 12: Extracting Temporal Information from Clinical Text[END_REF]; H.-J. [START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF]). However, these approaches limit the size of the window in which features are extracted ([-5;+5]). Our approach did not implement any ltering, which may introduce noise in the classi er.

Although we obtained a high performance for CONTAINS relation extraction, the gap between our best run and the best performing system of the shared task remains large (0.035 F1). We identify several areas of improvement. First, we did not implement any ltering on entity pairs. For instance, H.-J. [START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF] discarded event pairs that have contradictory modality and doctimerel attribute values. They also apply heuristic rules to further lter pairs when considering inter-sentence relations. Implementing a similar approach could increase the overall performance of our system.

Second, our machine learning approach did not use any weighting scheme to account for class imbalance within the corpus. The THYME corpus is highly unbalanced, especially for inter-sentence relations. This can result in a biased model toward the majority class, the negative one in our case.

Finally, applying a classical grid search approach for hyperparameter optimization can lead to an under-e cient model. The optimal value for a given hyperparameter can fall between its possible values de ned in the range. Random search [START_REF] Bergstra | Random Search for Hyper-Parameter Optimization[END_REF] or a treestructured parzen estimator approach [START_REF] Bergstra | Algorithms for Hyper-Parameter Optimization[END_REF]) may be more suitable as they allow to not make any assumptions on hyperparameter value sets besides their boundaries.

Temporal Relation Extraction in the Clinical Domain: Adapting the Approach to French Clinical Text

The main motivation for this research e ort is to evaluate whereas our feature-based approach can be used for other languages than English, provided that the di erent languagesensitive resources along our preprocessing pipeline are replaced by equivalent resources in the target language. We experiment on the THYME and MERLoT corpora. Similarly to our participation to Clinical TempEval, we focused on temporal relation extraction and use the gold entities provided within the two corpora. We discarded inter-sentence containment relations as they are not annotated in the French dataset. The MERLoT corpus has been transformed into a comparable corpus according to the process described at Section 3.2. The number of DCT relations per class for both corpora is presented at Table 3

Preprocessing and Feature Extraction

The THYME corpus was preprocessed using cTAKES [START_REF] Savova | Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): Architecture, Component Evaluation and Applications[END_REF], an open source natural language processing system for the extraction of information from electronic health records. We extracted several features from the output of cTAKES: sentences boundaries, tokens, POS tags, token types and Semantic Types of the entities that have been recognized by cTAKES and that have a span overlap with at least one EVENT entity of the THYME corpus.

Concerning the MERLoT corpus, no speci c NLP pipeline exists for French clinical texts; we thus used Stanford CoreNLP system [START_REF] Manning | The Stanford CoreNLP Natural Language Processing Toolkit[END_REF] to segment and tokenize the text. We also extracted POS tags. As the corpus already provides a type for each EVENT, there is no need for detecting other clinical information.

For both DCT and CONTAINS relation extraction tasks, we used a combination of structural, lexical and contextual features yielded from the corpora and the preprocessing steps. The choice of feature is inspired by research e orts in the temporal information extraction domain [START_REF] Bethard | SemEval-2015 Task 6: Clinical TempEval[END_REF][START_REF] Uzzaman | SemEval-2013 Task 1: TempEval-3 -Evaluating Time Expressions, Events, and Temporal Relations[END_REF][START_REF] Verhagen | SemEval-2007 Task 15: TempEval Temporal Relation Identication[END_REF][START_REF] Verhagen | SemEval-2010 Task 13: TempEval-2[END_REF]. These features are presented in Table 3.11.

Experimental Setup

We divided randomly the two corpora into train and test set following the ratio 80/20. As we mentioned in the Section 3.4.4, using a traditional grid-search approach for hyperparameter optimization may result in an under-e cient model. Following this observation, we performed hyper-parameter optimization using a tree-structured parzen estimator approach [START_REF] Bergstra | Algorithms for Hyper-Parameter Optimization[END_REF], as implemented in the library Hyperopt [START_REF] Bergstra | Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures[END_REF], to select the hyper-parameter C of a Linear SVM, the lookup window around entities and the percentile of features to keep. For the latter we used the ANOVA F-value as selection criterion. We used the SVM implementation provided within Scikit-learn. In each case, we performed a 5-fold cross-validation. For the container classi er and contains relation classi er, we used the f1-score as performance evaluation measure. Concerning the DCT classi er, we used the accuracy.

Feature

Results

Cross-validation results on the training corpus are presented in Table 3.12. DCT and CON-TAINS relation extraction task results on the test set are presented respectively in Table 3.13 and Table 3.14. For both tasks, we present a baseline performance. For the DCT relation extraction task, the baseline predicts the majority class (overlap) for all EVENT entities. For the CONTAINS relation extraction task, the baseline predicts that all EVENT entities are contained by the closest TIMEX3 entity within the sentence in which they occur.

Discussion

There is a gap of 0.04 in performance between the French (0.83) and English (0.87) corpora for the DCT relation extraction task. We notice that results per category are not homogeneous in both cases. Concerning the MERLoT corpus, the score obtained for the category overlap is better (0.90) than the score obtained for before-overlap (0.69), before (0.69) and after (0.73).

Concerning the THYME corpus, the performance for the category before-overlap (0.66) is clearly detached from the others which are grouped around 0.85 (0.88 for before, 0.84 for after and 0.89 for overlap). This may be due to the distribution of categories among the corpora. val 2016 evaluation corpus3 . We increased the recall value in comparison to their results (from 0.436 to 0.47) but this measure is still the main point to improve. The scores obtained by our CONTAINS relation classi er without the use of the CON-TAINER classi er output suggest that this feature is valuable for the classi cation decision with a drop ranging from -0.103 F1 to -0.234 F1.

More globally, the best results of the Clinical TempEval shared task were 0.843 (accuracy) for the DCT relation extraction task and 0.573 (f1-score) for the CONTAINS relation extraction task, which are comparable to our results (0.87 for the DCT task and 0.53 for the CONTAINS task).

Results presented in Table 3.12 indicates that replacing lexical forms by word embeddings seems to have a negative impact on performance in every case. This performance drop need further investigation.

As for the di erence of performance according to the language, several parameters can a ect the results. First, the sizes of the corpora are not comparable. The THYME corpus is bigger and has more annotations than the MERLoT corpus. Second, the quality of annotations is more formalized and re ned for the MERLoT corpus. This di erence can in uence the performance, especially for the CONTAINS relation extraction task. Third, the lack of specialized clinical resources for French can negatively in uence the performance of all classi ers.

Concerning the quality of annotations, it has to be pointed out that Inter-Annotator Agreement (IAA) for temporal relation is low to moderate: in the MERLoT corpus, IAA measured on a subset of the corpus is 0.55 for During relations, 0.32 for conducted relations and 0.64 for reveals relations. In the THYME corpus, IAA for contains relation is 0.651. The interannotator agreement is comparable in both languages, and suggests that temporal relation extraction is a di cult task to perform, even for humans.

Overall, we managed to obtain comparable results for both English and French datasets, suggesting that our approach can be applied to at least these two languages by replacing language sensitive resources in the preprocessing pipeline.

Conclusion

In this chapter, we have presented a feature-based approach focusing on the extraction of temporal relations between clinical events, temporal expressions and document creation times from clinical notes written in English and in French. This approach, based on feature engineering, obtained competitive results with the state-of-the-art at the time of publication and led to two main observations.

Our feature engineering approach can be applied with comparable results to two di erent languages, English and French in our case, by changing language dependent resources in the pipeline.

In the next chapter, we investigate the use of neural networks for temporal information extraction from clinical narratives. Speci cally, we try to combine the bene ts of both featurebased and neural approaches by incorporating categorical features in our model. We focus on both entity and relation extraction. The material presented in this chapter is based on three publications: one at the 2017 edition of the Clinical TempEval workshop (Tourille et al. 2017a), one at the 2017 edition of the ACL conference [START_REF] Tourille | Neural Architecture for Temporal Relation Extraction: A Bi-LSTM Approach for Detecting Narrative Containers[END_REF]) and one at the 2018 edition of the LOUHI workshop [START_REF] Tourille | Evaluation of a Sequence Tagging Tool for Biomedical Texts[END_REF].

Introduction

In this chapter, we investigate the two tasks related to temporal information extraction in clinical narratives: entity extraction (event and temporal expression) and temporal relation extraction (DCT and CONTAINS relations). More speci cally, we devise a supervised approach using neural networks for entity and relation extraction, and a supervised approach based on a linear SVM for event attribute classi cation.

As we mentioned in the introduction of this thesis, annotated corpora are often packing a rich attribute set describing the entities. In this chapter, we investigate how these features can be used in neural approaches and how they will impact the performance of such approaches. Our neural-based classi er leverages categorical features extracted from the corpus itself and from the output of cTAKES. These features are used in combination with classical word and character-level embeddings. We begin by studying how categorical features can be used to further improve the performance of our containment relation extraction module. Then, we evaluate our approach in the context of the 2017 edition of the Clinical TempEval shared task [START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF].

The remainder of this chapter is organized as follows. In Section 4.2, we quickly describe the dataset that was used in our experiments. In Section 4.3, we present the models for entity, attribute and temporal relation extraction. We also describe how input embeddings are built. Section 4.4 describes our work on the in uence of categorical features on the performance of our system. Section 4.5 presents the model evaluation in the context of the 2017 edition of the Clinical TempEval shared task. We conclude in Section 4.6.

Data

In this chapter, we use exclusively the THYME corpus for our experiments. Similarly to Chapter 3, we focus on the version of the THYME corpus that was proposed to the participants during the Clinical TempEval challenges [START_REF] Bethard | SemEval-2015 Task 6: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF]. This version comprises the train, dev and test subparts of the colon cancer section of the corpus.

EVENTs are given ve attributes: Contextual Modality, Degree, Polarity, Type and Doc-TimeRel. TIMEX3s elements have only one Class attribute. The temporal relation set is limited to the containment relation, expressed as CONTAINS relations in the dataset. Furthermore, the challenge addressed both intra-and inter-sentence relation extraction. More information about the corpus can be found in Chapter 2. Corpus statistics can be found in Chapter 3, Table 3.1.

Model Overview

Event Attribute and Document Creation Time Extraction

Feature-based approaches have proven to be e cient for attribute classi cation on the THYME corpus [START_REF] Bethard | SemEval-2015 Task 6: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF]). These approaches rely on handcrafted features extracted from the sentence context. Following these previous research e orts, we devised a feature-based approach using a SVM.

We treated each EVENT attribute extraction subtask as a supervised classi cation problem. We built a common pipeline for all attributes based on a linear SVM (Figure 4.2). First, a feature vector is extracted around each entity in a window of size w. Then, we retain only the k best features, ranked via their ANOVA F-score. The last step is the classi cation. The classi er has two hyperparameters. The parameter C allows for penalizing more or less classi cation mistakes. The parameter l allows to choose the loss function (hinge or squared hinge). Hyperparameter optimization is addressed by using a tree-structured parzen estimator approach [START_REF] Bergstra | Algorithms for Hyper-Parameter Optimization[END_REF] as implemented in Hyperopt2 [START_REF] Bergstra | Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures[END_REF]. The same pipeline is used for the DCT relation extraction subtask.

Containment Relation Extraction

Similarly to the entity extraction module, our approach relies on LSTMs. The architecture of our model is presented in Figure 4.3. For a given sequence of tokens separating two entities (EVENT and/or TIMEX3), represented as vectors, we compute a representation of the context between the two concerned entities by going from left to right in the sequence (forward LSTM in Figure 4.3).

As LSTMs tend to be biased toward the most recent inputs, this implementation would be biased toward the second entity of each pair processed by the network. To counteract this effect, we compute the reverse representation with an LSTM reading the sequence backwards, from right to left (backward LSTM in gure 4.3). By doing so, we keep as much information as possible about the two entities.

The two nal states are then concatenated and linearly transformed to a n-dimensional vector representing the number of categories (concatenation and projection in Figure 4.3). Finally, a softmax function is applied.

Similarly to the approach described in Chapter 3, we build two separate classi ers for intraand inter-sentence relations and limit the scope of the inter-sentence classi er to relations that do not span over more than three sentences. For inter-sentence relations, a special token SENT is added to the input sequence to mark sentence boundaries.

An overview of the embedding computation is presented in Figure 4.4. Following [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF], the character-based representation is constructed with a Bi-LSTM. First, a random embedding is generated for every character present in the training corpus. Token characters are then processed with a forward and backward LSTM similar to the one we use in our general architecture. The nal character-based representation is the result of the concatenation of the forward and backward representations.

Inluence of Categorical Features

Our rst set of experiments aims at measuring how categorical features in uence the performance of our CONTAINS relation extraction approach.

Preprocessing

Our experimental setup mimics the conditions of the second phase of the 2016 edition of the Clinical TempEval challenge where participant were provided with gold entities (EVENTs and TIMEX3s). We focus on CONTAINS relation extraction and consider that the DCT relation is given as gold attribute. We experiment on the colon cancer part of the THYME corpus.

More speci cally, we use the train and dev part of the corpus section as training corpus. We measure the performance on the test part of the corpus with the o cial evaluation script provided during the challenges. We preprocessed the corpus using cTAKES. We extracted sentence and token boundaries, as well as token types and semantic types of the entities that have a span overlap with a least one gold standard EVENT of the THYME corpus. This information was added to the set of gold standard attributes available for EVENTs in the corpus. An overview of the attributes available for each token is presented in Table 4.1.

Experimental Setup

We implemented the two neural models described in the previous section using TensorFlow 0.12 [START_REF] Abadi | TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems[END_REF]. We trained our network with mini-batch Stochastic Gradient Descent (SGD) using Adam with a batch-size of 256. The learning rate was set to 0.001. The hidden layers of our forward and backward LSTMs have a size of 512. We kept 10% of the training corpus for a development corpus and we implemented early stopping with a patience of 10 epochs without performance improvement. Finally, we used dropout training to avoid over tting. We applied dropout on input embeddings with a rate of 0.5.

We experimented with three con gurations. In the rst one, we used only word embeddings and character embeddings. In the second one, we added the feature embeddings related to the Gold Standard (GS) attributes. Finally, in a third experiment, we added the feature embeddings related to cTAKES. For each experiment, we report precision (P), recall (R) and f1-score (F1) computed with the o cial evaluation script3 provided during the Clinical TempEval challenges.

Results

Results of the experiments are presented in

Discussion

not distinguish between intra-and inter-sentence relations, but instead considered that related entities had to occur within a window of 30 tokens. We see that the addition of attribute embeddings slightly improves the overall performance of our system (+0.008). Adding the embeddings of GS features contributes to the major part of this improvement but tends to increase the imbalance between recall and precision. On the contrary, while the attribute embeddings related to cTAKES seem to have little impact on the overall performance, they tend to restore more balanced precision and recall.

The results for respectively intra-and inter-sentence relations are presented in Table 4.3. Similarly to our global results, the intra-sentence classi er bene ts from the addition of feature embeddings with a small increase for GS features and only a very little improvement for cTAKES features.

The inter-sentence classi er exhibits the same trend: GS features do improve the performance. However, adding cTAKES features degrades it slightly (-0.013).

The closest work compared to ours is clearly [START_REF] Dligach | Neural Temporal Relation Extraction[END_REF] as it also heavily relies on neural models for extracting temporal containment relations between medical events. [START_REF] Dligach | Neural Temporal Relation Extraction[END_REF] tested both CNN and LSTM models and found CNN superior to LSTM. However, this work addressed intra-sentence relations only. Moreover, its LSTM model was not a Bi-LSTM model as ours and it did not include character-based or attribute embeddings. Finally, it distinguished EVENT-TIMEX3 and EVENT-EVENT relations while we have only one model for the two types of relations.

Perspective

From a global perspective, the work we have presented in this section shows that in accordance with a more general trend, our neural model for extracting containment relations clearly outperforms classical approaches based on feature engineering. However, it also shows that incorporating classical features in such a model is a way to improve it, even if all kinds of features do not contribute equally to such improvement. A more ne-grained study has now to be performed to determine the most meaningful features in this perspective and to measure the contribution of each feature to the overall performance, with a speci c emphasis on character-based embeddings.

The place where these categorical features are embedded in our network should also be questioned. Including features related to the concerned entities in higher-level layers of our architecture could improve the performance. This possibility will be investigated in future research e orts.

Beyond a further analysis of the characteristics of our model, we are interested in three main extensions. The rst one will investigate whether training two models, one for EVENT-TIMEX3 relations and one for EVENT-EVENT relations, as done by [START_REF] Dligach | Neural Temporal Relation Extraction[END_REF], is a better option than training one model for all types of containment relations as presented herein. The second extension consists in transposing the model we have de ned in this work for English to French, as done in Chapter 3 for a more traditional approach based on a feature engineering approach. In the last one, we plan to explore additional strategies for CONTAINS relation extraction. For instance, adding a feature predicting whether a given EVENT entity is a container or not has proved to be useful in our feature-based approach (Chapter 3), but was not implemented in our system due to time constraints.

Intra-sentence classi er

Inter We report the number of gold standard relations (ref), the number of relations predicted by our system (pred), the number of true positives (corr), precision (P), recall (R) and f1-score (F1).

A Word on Temporal Coherence

Our neural approach for CONTAINS relation extraction only considers pairs of entities when making classi cation decisions. This could result in incoherent temporal graphs. We evaluated the degree of incoherence resulting from our approach by counting the number of cycles in the temporal graphs. Our experiments on intra-sentence CONTAINS relation extraction during development suggested that temporal graph incoherence is only minor with an average number of cycles per documents close to zero in a given iteration. However, robust temporal graph consistency is important for clinical sta that rely on extracted information to make informed decisions. Hence, we will investigate global approaches for temporal graph creation [START_REF] Bramsen | Inducing Temporal Graphs[END_REF][START_REF] Denis | Predicting Globally-Coherent Temporal Structures from Texts via Endpoint Inference and Graph Decomposition[END_REF]. These approaches allows to take into account all classi cation decisions to generate the nal temporal graph.

Evaluation on the THYME Corpus: Domain Adaptation for Temporal Information Extraction

In this second set of experiments, we evaluate our approach in the context of the 2017 edition of the Clinical TempEval challenge [START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF]. Similarly to the previous edition, the participants were asked to perform entity and containment relation extraction on the THYME corpus (Styler IV et al. 2014a). However, this edition included the notion of domain adaptation. Source and target domains were di erent. Systems were trained on documents related to colon cancer and were tested on brain cancer documents. Domain linguistic variation can decrease the performance of a given approach. As annotating a corpus is expensive and time consuming, there is a strong need for developing domain adaptation approaches for clinical NLP (Miller et al. 2017a;[START_REF] Miwa | Adaptable, High Recall, Event Extraction System with Minimal Con guration[END_REF][START_REF] Zhang | Domain Adaptation for Semantic Role Labeling of Clinical Text[END_REF]. Two experimental setup were proposed by the organizers. In the rst one, no target domain annotations were provided (unsupervised domain adaptation). In the second one, a set of 30 documents related to the target domain was given to the participants (supervised domain adaptation).

Preprocessing

For unsupervised domain adaptation, participants were given the three parts of the colon cancer section of the THYME corpus for developing their systems (train, dev, test). For supervised domain adaptation, 30 documents extracted from the train part of the brain cancer section were given to the participants. Performance was measured on the test part of the brain cancer section.

We preprocessed the corpus using cTAKES 3.2.2. We extracted sentence and token boundaries, as well as token types and semantic types of the entities that have a span overlap with a least one gold standard EVENT of the THYME corpus. This information was added to the set of gold standard attributes available for EVENTs in the corpus. We also preprocessed the corpus using HeidelTime 2.2.1 and used the results to further extend our feature set.

Architecture Description

We implemented a pipeline approach using the three modules described above (Section 4.3). For entity extraction, we integrated the attributes Type for EVENT and Class for TIMEX3 in the IOB scheme.

Concerning the attribute classi cation, we trained a separate classi er for each of the three remaining attributes and the DCT relation based on lexical, contextual and structural features extracted from the documents:

-EVENT type attribute, -EVENT plain lexical form, -EVENT position within the document, -POS tags of the verbs within the right and left contexts of the considered entity, -EVENT POS tag, -Type or Class of the other entities that are present within the left and right contexts, -token unigrams and bigrams within a window around the entity.

Input vectors are built di erently depending on the subtask. For the entity extraction subtask, vectors representing tokens are built by concatenating a character-based embedding and a word embedding. Whether we are dealing with EVENT or TIMEX3 entities, we add one embedding per cTAKES attribute or one embedding representing the TIMEX3 class as detected by HeidelTime. Concerning the containment relation subtask, input vectors are built by concatenating a character-based embedding, a word embedding, one embedding per gold standard attribute and one embedding for the type of DCT relation.

Domain Adaptation Strategies

We implemented two strategies for unsupervised domain adaptation. In the rst strategy, we blocked further training of the pretrained word embeddings during network training. Since a large number of medical events mentioned in the test set are not seen during training, we believe that our system should rely on untuned word embeddings to make its prediction.

In the second strategy we randomly replaced tokens that composed EVENT entities by the unknown token4 . Given the fact that our word embeddings are pretrained on the MIMIC III corpus (A. E. W. [START_REF] Johnson | MIMIC-III, a Freely Accessible Critical Care Database[END_REF]) and on the colon cancer part of the THYME corpus, a number of tokens (and therefore EVENTs) of the test part of the corpus may not have a speci c word embedding. By replacing randomly EVENT tokens, we force our networks to look at other contextual clues within the sentence. Both strategies were applied on EVENT entity and CONTAINS relation extraction subtasks.

Supervised domain adaptation was addressed by implementing two strategies. In the rst one, we mixed the 30 texts about brain cancer to the 591 texts about colon cancer. In the second one, we randomly chose 30 texts related to colon cancer and combined them to the 30 texts about brain cancer, resulting in a balanced training corpus. Both strategies were applied on EVENT, TIMEX3 and CONTAINS extraction subtasks.

Network Training

We trained our networks with mini-batch SGD using Adam [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF] with a batch-size of 256. The learning rate was set to 0.001. Hidden layers of our forward and backward LSTMs have a size of 256. We kept 10% of the training corpus for a development corpus and we implemented early stopping with a patience of 10 epochs without performance improvements. We use dropout training to avoid over tting. We applied dropout on input embeddings with a rate of 0.5.

Results

Our model ranked rst in almost all categories. Reproductions of the challenge result tables are provided in Appendix B. Table B.1 presents the results on the TIMEX3 extraction task.

Although our approach has a lower performance on f1-score and precision, we obtained the best recall across all con gurations. The results for EVENT extraction are presented in Table B.2. Our approach obtained the best f1-score across all con gurations. Concerning the temporal relation extraction task (DCT and CONTAINS relations), our system performed best in three out of four con gurations (Table B.3).

Results for our four runs are presented in Table 4.4. The two strategies implemented for unsupervised domain adaptation yield similar results (0.01 di erence in f1-score at most), with only a very slight advantage for the strategy blocking further training of the word embeddings (STATIC strategy in the table).

For supervised domain adaptation, the two strategies also yield close results (0.04 difference in f1-score) for the EVENT and temporal relation extraction subtasks. However, the strategy consisting in taking all available annotations (ALL strategy in the table) outperforms slightly the training on a balanced corpus, especially for the extraction of CONTAINS relations. The same strategy seems to perform much better for the TIMEX3 entity extraction subtask where the gap in f1-score reaches 0.06. This superiority agrees the general observation that the size of the training corpus has often a greater impact on results than its strict matching with the target domain. Overall, in both phases (supervised and unsupervised domain adaptation) and for all strategies, results are competitive for entity and temporal relation extraction.

Discussion

The performance obtained by our system relies in part on corpus tayloring. Some sections of the test corpus related to medication and diet are not to be annotated according to the annotation guidelines. However, these sections are not formally delimited within the documents. To avoid annotating them during test time, we developed a semi-automatic approach for detecting these sections and put them aside.

Other aspects linked to the corpus limit the performance. Some sections should not be annotated as they are duplicate of other sections found in the corpus as a whole. However, we have no information on how to formally identify these sections. Furthermore, a number of temporal expressions are annotated as SECTIONTIME or DOCTIME entities. Detecting TIMEX3 entities instead decreases the precision of our model. 4.4: Results obtained by our system across our four runs. We report precision (P), recall (R) and f1-score (f1). The best f1-score performance in each phase is bolded.

Concerning the domain adaptation strategies, results of the shared task con rm that a domain shift decreases signi cantly the performance of a given approach. Although we obtained the best score in the shared task, our performance remains much lower than the best performance obtained during the previous edition of the shared task [START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF]. For instance, H.-J. [START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF] obtained 0.479 F1 for end-to-end CONTAINS relation extraction while our approach gives 0.33 F1 in the unsupervised domain adaptation track.

Interestingly, TIMEX3 extraction has a very low f1-score in comparison to the best performance obtained when no domain adaptation is implied [START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF]). H.-J. [START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF] obtained a score of 0.772 F1 while the best score obtained in the unsupervised domain adaptation track was 0.51 F1 [START_REF] Macavaney | GUIR at SemEval-2017 Task 12: A Framework for Cross-Domain Clinical Temporal Information Extraction[END_REF]. This suggests that temporal expressions are either expressed di erently or that their sentence context is not the same within the two corpus sections (colon cancer and brain cancer). EVENT extraction su ers from the same type of performance drop. Overall, our strategies may not be suitable for this type of domain shift. We will investigate other approaches for domain adaptation in future work.

Conclusion

In this chapter, we presented an hybrid approach for temporal information extraction in clinical narratives. We tackled both entity and temporal relation extraction tasks. Our approach relies on classical word embeddings and on categorical features embedded as dense vector in our network. We tested the e ect of those features while experimenting on the THYME corpus and showed that they allow for a performance increase.

Then, we evaluated the approach in the context of the 2017 edition of the Clinical Tem-pEval shared task, which proposed two tracks related to domain adaptation. Our system was ranked rst in almost all categories. The two strategies that we implemented for unsupervised domain adaptation during the shared task performed similarly. However, for the supervised domain adaptation phase, the strategy which consist in training on a balanced dataset gives lower performance. This con rms the general observation that the size of the training corpus has a great impact on neural network performance.

Compared to our feature-based approach, the model presented in this chapter allows for a signi cant performance increase, suggesting that neural approaches are more e cient for temporal information extraction. However, by including categorical features within our network, we managed to improve the performance of our system. This suggests that hybrid approaches are a promising research area and should be investigated in the future.

Part II

Clinical Event Coreference Resolution

Introduction

This chapter discusses event coreference resolution in the clinical domain. We survey how the notion of coreference is apprehended from a linguistic perspective (Section 5.2). We present the terminology used by the NLP community (Section 5.3) and go over the di erences between event coreference in the general and the clinical domains (Section 5.4). We detail the annotated corpora in the clinical domain (Section 5.5). We present the di erent approaches developed in the literature (Sections 5.6, 5.7 and 5.8). Finally, we describe the approaches that have been developed for coreference resolution in the clinical domain (Section 5.9) and present the evaluation metrics used to measure coreference system performance (Section 5.10). We close the chapter with a conclusion (Section 5.11).

Anaphora and Coreference: A Linguistic Perspective

The terms anaphora and coreference are not well de ned in the NLP community and are often used confusingly, even synonymously in some cases. In this section, we brie y go over the di erent concepts behind these terms from a linguistic perspective.

An anaphoric expression depends on the linguistic context to be correctly interpreted. Personal pronouns such as she or he (Example 34a) but also demonstrative pronouns such as that (Example 34b) are examples of such expressions. In other words, the interpretation of an anaphoric expression depends on the entities previously mentioned or inferred from what have been said [START_REF] Poesio | Linguistic and Cognitive Evidence About Anaphora[END_REF]). This phenomenon is called anaphora.

(34) a. Barbara is coming tonight. [She] will bring a cake.

b. The train [that] departed at 3pm is arrived.

Besides the linguistic context, the visual context, also called visual deixis, regroups entities that are not mentioned in the linguistic context but are shared by the participants of a given conversation (H. H. [START_REF] Clark | De nite Knowledge and Mutual Knowledge[END_REF]. For instance, the seat in Example 35, is only known by the participants to the conversation and was not mentioned previously in the linguistic context. More generally, we say that these expressions depend on the discourse situation or utterance situation in order to be interpreted [START_REF] Barwise | Situations and Attitudes[END_REF]. The set of entities introduced in the discourse situation is called Universe of Discourse in the Discourse Representation Theory (DRT) [START_REF] Kamp | From Discourse to Logic. Introduction to Modeltheoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory[END_REF]. The interpretation of references to the visual context has been the focus of many research e orts (e.g. in [START_REF] Landragin | Information Sharing: Reference and Presupposition in Language Generation and Interpretation[END_REF]). However, this question is out of scope in our thesis and we will focus on anaphora.

Poesio (2016) identi es several expression types which depend on the linguistic context. They includes noun phrases as illustrated above but also pro-verbs (Example 36a), ellipsis (Example 36b) or full verbal expressions (Example 36c). In the last example, the time at which Kim listened to her messages is determined by the discourse situation. Examples are extracted from [START_REF] Poesio | Linguistic and Cognitive Evidence About Anaphora[END_REF]. Research in computational linguistics mainly focuses on anaphoric nominal expressions although some work has been done on ellipsis resolution (e.g. in [START_REF] Dalrymple | Ellipsis and Higher-Order Uni cation[END_REF]). These nominal expressions can be divided in four categories, according to their semantic function [START_REF] Poesio | Linguistic and Cognitive Evidence About Anaphora[END_REF]):

1. Referring: noun phrases that introduce a new entity in discourse or that rely on previously introduced entities for their interpretation.

2. Quantificational: noun phrases that denote a relation part/whole between two sets of objects. In the Example 37, the expression few trains is a subset of the set of trains.

( (39) a. It rains.

b. There is a cat in the tree.

Predicative noun phrases are less dependent on the universe of discourse than other types of noun phrases [START_REF] Poesio | Linguistic and Cognitive Evidence About Anaphora[END_REF]. The interest given to predicative noun phrases in the literature lies in the fact that they do not introduce new entities in the universe of discourse and will therefore not be used as anchors for anaphoric expressions. This distinction is re ected in recent coreference annotation guidelines, for instance in the corpus OntoNotes [START_REF] Hovy | OntoNotes: The 90% Solution[END_REF] used in recent shared tasks on coreference resolution [START_REF] Pradhan | CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes[END_REF][START_REF] Pradhan | CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes[END_REF][START_REF] Pradhan | Unrestricted Coreference: Identifying Entities and Events in OntoNotes[END_REF].

Most of the work on anaphora resolution targets referring noun phrases and the selection of an anchor for anaphoric expressions. Referring noun phrases include pronouns (re exive, de nite, inde nite and demonstrative), nominals and proper names. Proper names are a special type of referring noun phrases. They are not dependent on the linguistic context. They are directly referring to an object that is encoded in their semantics. In that respect, their interpretation is not similar from the one of others noun phrases such as pronouns and nominals [START_REF] Poesio | Linguistic and Cognitive Evidence About Anaphora[END_REF].

Anaphoric expression forms vary across languages. For instance, re exives and personal pronouns can be realized as incorporated pronouns as in Example 40. In several languages such as Italian and Japanese, one argument of an anaphoric reference can sometimes be omitted. We called this phenomenon zero anaphora (Example 41). All examples are taken from [START_REF] Poesio | Linguistic and Cognitive Evidence About Anaphora[END_REF]. c. English: [John] went to visit some friends. On the way, [he] bought some wine.

Up to this point, all examples were build around the notion of identity of reference between the referring expression and its anchor, meaning that the referring expression and its anchor refer to the same object. This is a particular case of anaphora, which is called coreference. One important thing to notice is that coreference does not implies anaphora. Proper names are the most idiomatic examples. Two mentions of the same person would be coreferring to the same real-world person, but do not depend on the linguistic context to be interpreted. Similarly, anaphora does not imply coreference, several other relation types exist:

• Identity of sense: this is the case for inde nite pronouns such as one or another which refer to a di erent object of the same type (Example 42). This can also be the case for de nite pronouns, as for the paycheck pronouns (Example 43).

(42) Sally admired Sue's jacket, so she got [one] for Christmas. [START_REF] Garnham | Mental Models and the Interpretation of Anaphora[END_REF] (43) The man who gave his paycheck to his wife is wiser that the man who gave [it] to his mistress. [START_REF] Karttunen | Discourse Referents[END_REF] • Bound anaphora: this is the case when the anchor is a quanti ed expression and the referring expression is a pronoun. In that case, the pronoun behave like a variable in a loop that gets called over the set of objects referenced by the anchor as in Example 44.

(44) No Italian ever believes that the referee treated [his] team fairly. [START_REF] Poesio | Linguistic and Cognitive Evidence About Anaphora[END_REF] • Associative anaphora: the relation between the referring expression and its anchor is one of part/whole or set/subset as in Example 45. This relation type is also called bridging anaphora.

(45) We saw a at yesterday. [The kitchen] is very spacious but [the garden] is very small. [START_REF] Poesio | Linguistic and Cognitive Evidence About Anaphora[END_REF] Determining the correct relation that lies between an anaphoric expression and its anchor is not an easy task. Recasens et al. (2011) point out that even the coreference (i.e. identity of reference) can sometimes be nuanced. For instance, in Example 46, 47 and 48, the authors described the relations that link the mentions as relations of near-identity. Recasens et al. (2011) argue for a continuum ranging from identity to non-identity. (48) "[Your father] was the greatest, but he was also one of us," commented an anonymous old lady while she was shaking Alessandro's hand-[Gassman]'s best known son. "I will miss [the actor], but I will be lacking [my father] especially," he said. (Recasens et al. 2011) Intuitively, entities mentioned in the previous utterances of a given discourse are prominent for the interpretation of referring expressions. This observation has led to the discoursemodel hypothesis and to the dynamic models of discourse interpretation [START_REF] Garnham | Mental Models and the Interpretation of Anaphora[END_REF][START_REF] Karttunen | Discourse Referents[END_REF]). The discourse model hypothesis explains that context dependent expressions are processed and interpreted with respect to the current state of the universe of discourse. This led to two main considerations in the literature [START_REF] Poesio | Linguistic and Cognitive Evidence About Anaphora[END_REF]. First, the universe of discourse is constantly updated and this update potential has to be accounted for in the models. Second, the objects included in the discourse model are not limited to the ones being mentioned. A number of entities can be constructed or inferred. In Example 49, the constructed set comprising John and Mary is the anchor of the pronoun they. Propositions or abstracts concepts can also be inferred such as the fact that the court does not believe someone in Example 50. Finally, entities may have been introduced implicitly like in Example 51 where the expression the government refers to the government of Korea.

Deinitions and Terminology: The NLP imbroglio

In the previous section, we stated that an anaphoric expression depends on the entities introduced in the universe of discourse in order to be interpreted. When the relation between the anaphoric expression and its anchor is one of identity, we say that they are coreferring. However, the terms anaphora and coreference have been misused in the NLP literature. The term coreference was introduced during the rst MUC shared task [START_REF] Sundheim | Overview of Results of the MUC-6 Evaluation[END_REF] on entity coreference resolution. The term did not convey the same concept that we de ned in the previous section. [START_REF] Deemter | On Coreferring: Coreference in MUC and Related Annotation Schemes[END_REF] highlighted several conception mistakes. First, non-referring expressions were used in anaphoric relations and annotators were asked to mark bridging relations as coreference relations (Example 52).

(52) a. No solution emerged from our discussions. b. Whenever a solution emerged, we embraced it. [START_REF] Deemter | On Coreferring: Coreference in MUC and Related Annotation Schemes[END_REF] also point out that predicative anaphoric expressions were marked as coreferent with their anchor. This has some implications regarding change over time. In the annotation guidelines [START_REF] Hirschman | MUC-7 Coreference Task De nition[END_REF] for the MUC-6 and MUC-7 shared tasks, it is stated that two mentions should be marked as coreferent if the text asserts them to be coreferential at any time. Also, Henry Higgins, sales director of Sudsy Soaps and president of Dreamy Detergents should be marked as coreferent in Example 53 according to the annotation guidelines. Since coreference implies equivalence, the annotation implies that the sales director of Sudsy Soaps and the president of Dreamy Detergents is the same person, which is not correct.

(53) Henry Higgins, who was formerly sales director of Sudsy Soaps, became president of Dreamy Detergents.

The main conclusion of [START_REF] Deemter | On Coreferring: Coreference in MUC and Related Annotation Schemes[END_REF] analysis is that the relation annotated in MUC is di erent from the coreference relation discussed in linguistics. Anaphora, predication and coreference are annotated together under the same term coreference. The authors recommend that the next annotations guidelines re ect the distinction between them.

The observations formulated by [START_REF] Deemter | On Coreferring: Coreference in MUC and Related Annotation Schemes[END_REF] have been taken into account in recent coreference annotation e orts. The coreference annotations of the OntoNotes corpus [START_REF] Pradhan | Unrestricted Coreference: Identifying Entities and Events in OntoNotes[END_REF]) makes the distinction between identical and appositives coreference links. Furthermore, coreference annotations does not include generic, underspeci ed or abstract entities. However the distinction between anaphora and coreference remains loose in the sense that a coreference link will be set between two proper names as well as between a pronoun and its anchor. Both cases will be considered as examples of anaphoric coreference.

In this thesis, we borrow the terminology of the NLP community and slightly abuse the terms coreference and anaphora. Subsequently, several terms need to be de ned. Consider the following example:

(54) [John] e1 went to [[Bob] e2 's house] e3 .
[He] e1 gave [him] e2 back [[his] e2 umbrella] e4 .

Following [START_REF] Pradhan | CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes[END_REF], we de ne the coreference resolution task as identifying all mentions of entities or events in text and clustering them into equivalence classes. This task does not imply to determine to which real-world entity or event the mentions are referring to, but whether the mentions are referring to same entity or event.

The text spans that are considered as potential entity or event mentions within a coreference resolution system are called mentions, candidates or markables. In Example 54, all bracketed expressions are candidates.

As we will explain later in this chapter, coreference resolution systems usually order the mentions according to their position in the text. Typical approaches process the mentions in order and try to draw an anaphoric coreference link between the active mention, i.e. the mention currently in focus and one of its antecedents, i.e. the mentions physically located before the active mention. The active mention is also called the anaphor in the literature, even if the mention itself is not anaphoric.

Following the literature, two mentions m, n are coreferent if and only if Referent(m) = Referent(n). The coreference relation is re exive, symmetric and transitive.

A cluster of mentions referring to same entity is called a coreference chain in the literature. The term comes from the fact that if we were to sort all mentions belonging to the same cluster according to their position in a document, the resulting form would looks like a chain where each mention excepted the rst one is linked to the previous one via an anaphoric coreference link. Note that coreference chains are also called entities in the literature.

Event Coreference Resolution

Until now, we have been talking mostly about entity coreference resolution where the objective is to determine whether two entity mentions refer to the same real-world entity. Event coreference resolution targets events instead of entities. It is a growing eld of research in NLP which is considered as more complex and more challenging than entity based coreference resolution [START_REF] Lu | Event Coreference Resolution: A Survey of Two Decades of Research[END_REF]). Consider the following example: The two event mentions in Example 55 are coreferent for two reasons. First, the subtype of the events are compatible. They are both referring to a movement which implies the transport of a person. Second, their arguments are also compatible. The rst event mention has three arguments: a person argument (Georges Cipriani), an origin argument (a prison) and a time argument (fiednesday). The second event mention has also three arguments: a person argument (He), an origin argument (the prison) and an instrument argument (a police vehicle). The two overlapping arguments are compatible (i.e. they are entity-coreferent). Both constraints (subtype and argument constraints) should be met in order for two event mentions to be coreferent [START_REF] Lu | Event Coreference Resolution: A Survey of Two Decades of Research[END_REF].

In order to perform event coreference resolution, a typical event coreference system must implement several steps. First, it must extract entities and perform entity coreference resolution. Then, event mentions should be extracted and entities should be classi ed as arguments. Finally, event coreference resolution may be done.

Intuitively, event coreference resolution is harder than its counterpart on entities due to error propagation in the pipeline. Moreover, as we have seen in Chapter 2, event mentions have various linguistic realizations ranging from noun phrases to verb phrases, while entity mentions are mostly noun phrases and pronouns.

Clinical Domain. Event coreference resolution in the clinical domain is di erent for several reasons. First, as we have seen in Chaper 2, clinical events di er from general domain events where the linguistic object of interest is mostly the verb and its derivatives (e.g. nominalizations). In the clinical domain, events may be realized by noun phrases. This is the case for the treatment and the two problems mentioned in Example 56. Thus, the task of nding markables in the clinical documents may be cast as a NER problem. This di erence must be re ected in the clinical event coreference systems. Second, clinical events do not usually imply syntactically explicit arguments in the text. In Example 57a, the three problems mentioned in the sentence have a similar argument structure. They have a location argument (Youville Hospital), a date argument (2020-03-10) and an instrument argument (MRI). However, all three arguments are di cult to retrieve with a syntactic analysis of the sentence. Sometimes, the event arguments are not even in the same sentence. This is the case for the event mentions in Example 57b. The sentence is located next to the sentence presented in Example 57a. All event mentions are also related to the MRI done in Youville Hospital.

Third, event arguments may be implicit. This is the case for all event mentions in Example 57 which all have an implicit person argument, i.e. the patient. Note that sometimes the person argument may be explicit, as shown in Example 58. Fortunately, clinical documents concern only one patient, so this implicit argument does not play such an important role for coreference resolution. Finally, another speci city of clinical event coreference resolution emerges when considering the downstream applications where the task could be useful. In our case, the medical sta is interested in reconstructing the patient clinical timeline based on the patient clinical documents. In this context, the temporal location of clinical events seems to play an important role for coreference resolution. Intuitively, two event mentions sharing the same semantic meaning and the same temporal location have a high probability to be coreferent. In other words, each event mention has an implicit temporal argument. Once retrieved, this argument could be used for coreference resolution.

These domain characteristics (e.g. event mention forms or absence of arguments) seems to make the clinical event coreference resolution task more similar to entity based coreference resolution. This is re ected in the clinical NLP literature where systems and methods follow the entity based coreference resolution methods developed in the general domain.

Annotated Corpora

There are two open annotated corpora available for event coreference resolution in the clinical domain. The biomedical and the general domains are also active areas for event coreference resolution. However we will not review their associated corpora in this section. An overview of these resources can be found in K. B. [START_REF] Cohen | Coreference Annotation and Resolution in the Colorado Richly Annotated Full Text (CRAFT) Corpus of Biomedical Journal Articles[END_REF] and [START_REF] Lu | Event Coreference Resolution: A Survey of Two Decades of Research[END_REF].

The i2b2 Corpus. The i2b2 corpus [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF] is the rst publicly available dataset annotated with coreference in the clinical domain. The corpus was created to foster the development of new methods for coreference resolution in clinical text. The corpus builds upon existing annotation e orts made in the context of the i2b2 challenges. Thus, it reuses previous layers of annotations.

The corpus is composed of two sub-corpora: the i2b2/VA corpus and the Ontology Development and Information Extraction (ODIE) corpus. Both corpora provide annotated clinical documents (discharge summaries and progress notes) from various institutions.

Event types annotated in each sub-corpora are not identical. The ODIE corpus [START_REF] Savova | Anaphoric Relations in the Clinical Narrative: Corpus Creation[END_REF]) contains 164 documents and includes ten entity categories: anatomical site, disease or syndrome, indicator/reagent/diagnostic aid, laboratory or test result, none, organ or tissue function, other, people, procedure and sign or symptom. The i2b2/VA [START_REF] Uzuner | 2010 i2b2/VA Challenge on Concepts, Assertions, and Relations in Clinical Text[END_REF]) corpus contains 814 documents and annotates a smaller set of entities: problem, person, pronoun, test and treatment. Statistics about the two corpora can be found in [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF]. Interestingly, the corpus addresses both entity and event based coreference by annotating coreference chains related to people. However the clinical documents involve a limited set of persons beside the patient himself. Therefore, there are a small number of person coreference chains involving a large number of mentions, mostly pronouns.

The THYME corpus. Besides the temporal annotations (Styler IV et al. 2014a), approximately 300 colon cancer documents are annotated with within-document coreference chains (Miller et al. 2017b). The main di erence with the i2b2 corpus lies in the markable de nition. The THYME corpus considers that all nouns, noun phrases (including relative clauses), nominal modi ers, pronouns and nominalized verbs can be considered as markables. Moreover, the annotations do not cover singletons (non-coreferent mentions) and markables are not typed. Annotators were asked to annotate the longest and the most speci c span, including determiners and modifying information.

Another di erence with other general and clinical domain coreference corpora is that the THYME corpus covers other relations besides the identity and appositive relations.

A Word on Mention Extraction

There are two variants of the coreference resolution task. In the rst one, gold mentions are given and used in the coreference systems. Thus, the task does not involve mention extraction and/or classi cation but only determining the coreference links between mentions. In the second one, systems must start from scratch, without any mentions to reason with. This version of the coreference resolution task has become more and more popular over the years as the rst one is considered too simple and does not re ect the true di culty of the task.

Coreference resolution therefore implies mention extraction as a rst step. Most approaches in the general domain use rule-based systems based on the syntactic analysis of sentences (e.g. in K. Clark and Manning (2016); [START_REF] Wiseman | Learning Anaphoricity and Antecedent Ranking Features for Coreference Resolution[END_REF]). For instance, the Berkeley Coreference System [START_REF] Durrett | Easy Victories and Uphill Battles in Coreference Resolution[END_REF] builds on H. [START_REF] Lee | Stanford's Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task[END_REF] to extract mentions based on sentence parse trees and the output of a NER system. The extraction is sometimes followed by a ltering step to remove unlikely coreferent mentions such the pleonastic pronoun it. Several systems train an anaphoricity classi er to determine whether a mention is anaphoric [START_REF] Björkelund | Exploring Lexicalized Features for Coreference Resolution[END_REF][START_REF] Rahman | Supervised Models for Coreference Resolution[END_REF].

Rule-based approaches are also popular in the clinical domain. [START_REF] Grouin | Coreference Resolution in Clinical Reports. The LIMSI Participation in the i2b2/VA 2011 Challenge[END_REF] use an analysis engine based upon regular expressions of words, rules and lexicons. They create a lexicon based on the UMLS ® by selecting the relevant concept via its semantic types. Miller et al. (2017) use the output of cTAKES [START_REF] Savova | Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): Architecture, Component Evaluation and Applications[END_REF] to extract candidate mentions based on the syntactic analysis of sentences.

For both the general and the clinical domains, mention extraction systems are tuned to maximize recall. This leads to the creation of large number of candidates which have to be pruned somehow during the following processing steps.

Early Approaches for Coreference Resolution

Until the apparition of corpora in the mid 1990s and the introduction of shared tasks on the subject [START_REF] Hirschman | MUC-7 Coreference Task De nition[END_REF][START_REF] Sundheim | Overview of Results of the MUC-6 Evaluation[END_REF], most of the work on coreference resolution was theoretically inspired and rule-based. [START_REF] Hobbs | Resolving Pronoun References[END_REF] presented a syntax-based approach for pronoun resolution which is still often used as a baseline. The algorithm builds on the observation made in the literature that syntactic and morphosyntactic information plays a role for interpretation. This information is often modeled as constraints. For instance gender, number or person constraints.

Hobbs' algorithm traverses the surface parse tree breadth-rst, from left to right. It goes backwards one sentence at a time and looks for the correct antecedent which matches the pronoun constraints (gender and number). As [START_REF] Poesio | Early Approaches to Anaphora Resolution: Theoretically Inspired and Heuristic based[END_REF] underlines, [START_REF] Hobbs | Resolving Pronoun References[END_REF] is one of the rst who tries to implement a formal evaluation. The algorithm is tested with 100 examples.

Several approaches were based on common sense knowledge. [START_REF] Charniak | Toward A Model of Children's Story Comprehension[END_REF] proposed a model called DSP (Deep Semantic Processing) that takes hand-coded assertions for a group of sentences and applies deductive inferences that resolve anaphoric references. Although the system is one of the rst proposing to use inferences for anaphora resolution, it su ers from several problems including lack of evaluation (Poesio et al. 2016b).

A number of research e orts are build around the notion of salience. The idea of salience is to account for the recency of entities introduced in discourse [START_REF] Grosz | Attention, Intentions and the Structure of Discourse[END_REF]. [START_REF] Hobbs | Pronoun Resolution[END_REF] showed that 90% of pronoun anchors are in the same sentence as the pronoun and 98% are in the same or previous sentences. However, choosing the closest matching antecedent do not give good results. This observation led to the development of a framework where the notion of focus is introduced. [START_REF] Grosz | Attention, Intentions and the Structure of Discourse[END_REF] proposed two focus levels. The global focus speci es the articulation of a given discourse into segments. In other words, discourses are segmented according to topics. The second level is called the local focus. At this level, the authors model the fact that depending on the position in the text or in the conversation, the entities have a variable salience. The most prominent entities will be preferred in the case of pronominal anaphora. This notion of local focus was later divided into two sub-concepts. The discourse focus is concerned by the discourse topic whereas the actor focus is concerned by the preference for pronouns in subject positions to refer to antecedents in subject positions. This approach for accounting for the salience of entities in discourse inspired researchers in developing the centering theory [START_REF] Grosz | Centering: A Framework for Modeling the Local Coherence of Discourse[END_REF]. According to this theory, every utterance updates the local focus by introducing mentions of discourse entities. These entities are ranked at each utterance and the rst ranked is called the Preferred Center. This corresponds to the actor focus mentioned above. The theory accounts also for the discourse focus by acknowledging the existence of an object playing this role, called the backward looking center. There is a large number of models based on the theories of salience in the literature [START_REF] Brennan | A Centering Approach to Pronouns[END_REF][START_REF] Lappin | An Algorithm for Pronominal Anaphora Resolution[END_REF][START_REF] Strube | Never Look Back: An Alternative to Centering[END_REF][START_REF] Strube | Annotating Longitudinal Clinical Narratives for De-Identi cation: The 2014 i2b2/UTHealth Corpus[END_REF].

Supervised Approaches for Coreference Resolution

Supervised approaches for coreference resolution have given interesting results over the last two decades. We identify four main models in the literature. As it is (almost) always the case for any categorization, some models t with di culty in our schema, either because they are crossing several categories or because they are somehow unique. For a di erent view of the domain we refer the reader to other reviews that have been published on the topic [START_REF] Lu | Event Coreference Resolution: A Survey of Two Decades of Research[END_REF][START_REF] Martschat | Structured Representations for Coreference Resolution[END_REF][START_REF] Ng | Supervised Noun Phrase Coreference Research: The First Fifteen Years[END_REF][START_REF] Ng | Machine Learning for Entity Coreference Resolution: A Retrospective Look at Two Decades of Research[END_REF]Poesio et al. 2016a;[START_REF] Zheng | Coreference Resolution: A Review of General Methodologies and Applications in the Clinical Domain[END_REF].

Each model presented in the section is rst described by taking the most representative research e ort from the literature. Then, following [START_REF] Ng | Advanced Machine Learning Models for Coreference Resolution[END_REF], we describe the di erent model variants according to four main axes: the learning algorithm, the instance creation method, the features and the clustering algorithm.

Mention-Pair Model

The mention-pair model was rst introduced by [START_REF] Mccarthy | Using Decision Trees for Coreference Resolution[END_REF] and [START_REF] Aone | Applying Machine Learning to Anaphora Resolution[END_REF] who focused on organizations involved in business joint ventures in news articles written in English and Japanese. The model gained popularity when [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF] proposed a generalization of the model by including all noun phrases. [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF] cast the task as a classi cation problem where a classi er, a decision tree learner [START_REF] Quinlan | C4.5: Programs for Machine Learning[END_REF], is trained to decide whether a given pair of noun phrases is coreferent or not. In this model, each pair is represented as a feature vector. [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF] relied on a small set of twelve features including distance, string matching and agreement features, which were, for most of them, already used in previous work [START_REF] Cardie | Noun Phrase Coreference as Clustering[END_REF][START_REF] Fisher | Description of the UMass System as used for MUC-6[END_REF].

This rst classi cation step is followed by a clustering phase where local pairwise classication decisions are clustered together to form proper coreference chains. However, these local decisions can be contradicting. For instance, consider Example 60. A contradiction could arise if the pronoun he was wrongly classi ed as coreferent with both John and Bob mentions, but the two latter were classi ed as non-coreferent. Furthermore, even if we ignore the potential contradictions, several antecedents could have been marked as coreferent with a given mention. In this case, how do we choose the correct antecedent?

To answer this question, [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF] implemented closest-rst clustering. In this strategy, the model chooses the closest antecedent which was classi ed as coreferent with the active mention. Although the strategy allows to build a coherent set of coreference chains, it does not resolve contradictions that could arise at the classi er level.

One major issue with mention pair models lies in the creation of the training instances. The distribution of positive and negative instances is highly unbalanced since the majority of mention pairs are not coreferent. [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF] tackled this issue by implementing a heuristic rule. Positive instances are created between anaphoric noun phrases and their closest preceding antecedent. All antecedents occurring between the anaphoric noun phrases and their closest antecedents are used to build negative instances.

Learning algorithm Several machine learning algorithms have been used to learn the pairwise classi er: decision tree learners (Quinlan 1993) (e.g. in McCarthy andLehnert (1995); [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF]), memory-based learners [START_REF] Daelemans | Memory-Based Language Processing[END_REF]Bosch 2005) (e.g. in Hoste (2005); [START_REF] Recasens | A Deeper Look into Features for Coreference Resolution[END_REF]), perceptrons (e.g. in [START_REF] Bengtson | Understanding the Value of Features for Coreference Resolution[END_REF]; [START_REF] Stoyanov | Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art[END_REF]), support vector machines [START_REF] Cortes | Support-Vector Networks[END_REF]Vapnik 1995) (e.g. in Rahman andNg (2011); [START_REF] Xu | A Classi cation Approach to Coreference in Discharge Summaries: 2011 i2b2 Challenge[END_REF]), maximum entropy learners [START_REF] Bergert | A Maximum Entropy Approach to Natural Language Processing[END_REF]) (e.g. in [START_REF] Kehler | The (Non) Utility of Predicate-Argument Frequencies for Pronoun Interpretation[END_REF]) or the RIPPER rule learner (W. W. Cohen 1995) (e.g. in [START_REF] Hoste | Optimization Issues in Machine Learning of Coreference Resolution[END_REF]Ng and Cardie 2002a;[START_REF] Ng | Combining Sample Selection and Error-Driven Pruning for Machine Learning of Coreference Rules[END_REF][START_REF] Ng | Improving Machine Learning Approaches to Coreference Resolution[END_REF]).

Features

The small feature set used by [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF] was improved and extended by Ng and Cardie (2002). The authors added a set of 41 features including new mention comparisons, grammatical constraints and semantic features. Later, the availability of large corpora allowed the inclusion of lexical features [START_REF] Bengtson | Understanding the Value of Features for Coreference Resolution[END_REF][START_REF] Björkelund | Exploring Lexicalized Features for Coreference Resolution[END_REF] Several research e orts considered the use of world knowledge in the model. [START_REF] Ponzetto | Exploiting Semantic Role Labeling, WordNet and Wikipedia for Coreference Resolution[END_REF] used the category network of Wikipedia to build a taxonomy and built semantic relatedness features based on it. Rahman and Ng (2011) derived features based on YAGO [START_REF] Suchanek | Annotating Temporal Information in Clinical Narratives[END_REF].

Clustering A large number of research papers focused on improving the clustering step of the initial mention-pair model. While [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF] performed closest-rst clustering, Ng and Cardie (2002) implemented best-rst clustering where the active mention is linked to the best scoring antecedent (among those who have a coreference score above 0.5). This strategy was already proposed in [START_REF] Aone | Applying Machine Learning to Anaphora Resolution[END_REF]. Both closest-rst and best-rst clustering approaches are greedy in the sense that they do not take into account the relations between classi cation decisions. Another greedy approach, called aggressive-merge clustering, takes the transitive closure over all decisions [START_REF] Denis | Global Joint Models for Coreference Resolution and Named Entity Classi cation[END_REF][START_REF] Stoyanov | Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art[END_REF], meaning that the active mention is clustered with all its preceding coreferent antecedents.

C. Nicolae and G. Nicolae (2006) built a graph from the output of the pairwise classi er where each edge is weighted with the classi er score. The optimal partitioning of the graph is obtained by cutting repeatedly the graph until a satisfactory stopping point is reached. [START_REF] Klenner | Enforcing Consistency on Coreference Sets[END_REF] used integer linear programming to enforce correct transitivity based on the output of a pairwise classi er.

The clustering step can also be made during learning. McCallum and Wellner (2005) presented a graph partitioning model where all mentions of a given cluster must be close to each other.

There are a few comparison of clustering approaches in the literature. While comparing closest-rst and best-rst strategies, Ng and Cardie (2002) found that the latter performs best on MUC data while [START_REF] Rahman | Supervised Models for Coreference Resolution[END_REF] observed the opposite situation. [START_REF] Denis | Specialized Models and Ranking for Coreference Resolution[END_REF] compared closest-rst and aggressive-merge clustering approaches and observed that both strategies impact the metric di erently. One major weakness of all three clustering approaches, closest-rst, best-rst and aggressive-merge is that they rely only on local decisions taken independently.

Training instance creation

The method proposed by [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF] where a negative instance is build for each antecedent located between the mention and the rst true antecedent has been widely used in the literature [START_REF] Ng | Improving Machine Learning Approaches to Coreference Resolution[END_REF][START_REF] Ponzetto | Exploiting Semantic Role Labeling, WordNet and Wikipedia for Coreference Resolution[END_REF][START_REF] Rahman | Supervised Models for Coreference Resolution[END_REF]. In some cases, the heuristic is slightly modi ed. For instance, Ng and Cardie (2002) selected the closest non-pronominal antecedent for a pronominal anaphor. [START_REF] Mccarthy | Using Decision Trees for Coreference Resolution[END_REF] and [START_REF] Stoyanov | Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art[END_REF] used all mention pairs in their respective systems.

Mention-Ranking Model

The mention-pair model has two major issues. The rst one is that the features extracted from the contexts of the two mentions considered at each timestep may not be su cient for correctly making a coreference decision. The second one is that each pair of mentions is considered independently from the others and this can lead to classi cation contradictions. The mention-ranking model aims at addressing the second problem by ranking all possible antecedents and choosing the best one. Thus it removes the burden of selecting a clustering approach.

Yang et al. ( 2003) made an early attempt at designing a mention-ranking model. Their model, the twin-candidate model, takes each pair of mentions and predicts which one is better antecedent for the anaphor. The mention that win the most comparisons is selected as the correct antecedent for the anaphor. Although the model shows promising results, it does not take into account all antecedents at once.

The model presented in [START_REF] Denis | Specialized Models and Ranking for Coreference Resolution[END_REF] is one of the rst modern mentionranking model. The authors consider all candidate antecedents and computed all the pairwise scores between these candidates and the anaphor. The candidate that obtain the best score is selected as antecedent. During learning, the model learns a parameter vector such that the closest correct antecedent gets a higher score compared to the others. [START_REF] Denis | Specialized Models and Ranking for Coreference Resolution[END_REF] used a combination of features including semantic compatibility, string similarity and morphosyntactic agreement features.

In the model, every mention must have an antecedent. However, the vast majority of these mentions are non anaphoric. Hence, [START_REF] Denis | Specialized Models and Ranking for Coreference Resolution[END_REF] performed a classi cation step to determine if the active mention is anaphoric or not. One major drawback of this approach is that error made at this stage of the pipeline would propagate at the coreference resolution level.

Finally, as [START_REF] Denis | Specialized Models and Ranking for Coreference Resolution[END_REF] chose only one correct antecedent during learning. They chose the closest correct antecedent as the reference for pronouns and the closest nonpronominal antecedent for non-prononominal mentions.

Learning algorithm

In their initial model, [START_REF] Denis | Specialized Models and Ranking for Coreference Resolution[END_REF] used a maximum entropy learner [START_REF] Bergert | A Maximum Entropy Approach to Natural Language Processing[END_REF]) but other algorithms have been used in the literature. Rahman and Ng (2011) used support vector machines [START_REF] Cortes | Support-Vector Networks[END_REF], K.-W. Chang et al. (2012) used an average perceptron while [START_REF] Yang | Coreference Resolution Using Competition Learning Approach[END_REF] used a decision tree learner [START_REF] Quinlan | C4.5: Programs for Machine Learning[END_REF].

Features Most papers use features similar to those presented in Ng and Cardie (2002) and [START_REF] Bengtson | Understanding the Value of Features for Coreference Resolution[END_REF]. [START_REF] Durrett | Easy Victories and Uphill Battles in Coreference Resolution[END_REF] used mainly lexical features and heuristics for feature combination. [START_REF] Wiseman | Learning Anaphoricity and Antecedent Ranking Features for Coreference Resolution[END_REF] used a neural network model to learn feature combinations and thus alleviated the need for engineering them. They showed performance improvement on the CoNLL 2011 shared task corpus [START_REF] Pradhan | CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes[END_REF].

Clustering In [START_REF] Yang | Coreference Resolution Using Competition Learning Approach[END_REF], clustering at test time is done via tournament ranking. The antecedent that is classi ed as antecedent the largest number of times is selected as antecedent for the active mention.

Alternatives to the anaphoricity classi cation step described by Denis and Baldridge ( 2008) have been proposed in the literature. Among them, K.-W. Chang et al. (2012) built a dummy antecedent for the non-anaphoric case and give it the score 0 when considering a pair with the dummy mention. Alternatively, [START_REF] Durrett | Easy Victories and Uphill Battles in Coreference Resolution[END_REF] used a special feature that is triggered when the considered mention is the dummy mention.

Papers using dummy mentions for the non-anaphoric case use a cost-sensitive loss. For instance, [START_REF] Durrett | Easy Victories and Uphill Battles in Coreference Resolution[END_REF] and [START_REF] Wiseman | Learning Anaphoricity and Antecedent Ranking Features for Coreference Resolution[END_REF] distinguished between wrong link, false new and false anaphoric errors.

Finally, [START_REF] Denis | Specialized Models and Ranking for Coreference Resolution[END_REF] selected the closest antecedent as the correct antecedent. This approach has been embraced in a large number of research e orts. However, there are research e orts trying to learn what the best antecedent is. In K.-W. Chang et al. (2012), [START_REF] Durrett | Easy Victories and Uphill Battles in Coreference Resolution[END_REF] and [START_REF] Wiseman | Learning Anaphoricity and Antecedent Ranking Features for Coreference Resolution[END_REF], the models choose the best scoring correct antecedent for the active mention (with the current model parameters) and use it as the referent antecedent.

Entity-Based Models

Entity-based models were introduced to overcome the limitation of mention-pair and mentionranking models which rely only on local features from the mention contexts to make classication decisions.

One of the rst entity-based approach was introduced by [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF]. Similarly to the mention-pair model, [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF] processed the text in order and decide to which partially constructed cluster, or chain, the active mention has to be linked. By doing so, later coreference decisions depend on earlier ones and these decisions depend not only on the active mention and the antecedent but also on all the mentions of the entity in which the antecedent is in. In that respect, the model has been coined entity-mention model in the literature (Poesio et al. 2016a). [START_REF] Ng | Advanced Machine Learning Models for Coreference Resolution[END_REF] highlighted that the way this type of models incrementally processes a discourse makes them similar to earlier discourse models. [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF] used several features at the mention and entity levels. It is worth noting that most of the traditional features used in the literature with the mention-pair model [START_REF] Cardie | Noun Phrase Coreference as Clustering[END_REF][START_REF] Fisher | Description of the UMass System as used for MUC-6[END_REF][START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF]) can be easily transposed at the entity level via the use of logical predicates. Consider the head match feature. [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF] devised a entity-level feature that is triggered if any mention in the partially constructed entity matches the active mention.

Considering that best-rst and closest-rst clustering approaches are too greedy, [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF] modeled the search space as a Bell tree. They kept track of the k-best partial clustering and performed the search via beam search. [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF] implemented a method similar to the one described in [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF] for training instance creation to reduce class imbalance. For each anaphor that does have an antecedent, a negative instance will be created between the considered anaphor and every preceding cluster only if one mention of these clusters lives between the anaphor and the closest mention of the antecedent cluster.

Similarly to the mention-pair model, the model of [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF] su ers from the fact that each decision is taken independently from the others and therefore fails to capture the competition between antecedents.

Learning algorithm Entity-based models have been used widely in the literature [START_REF] Daumé | Practical Structured Learning Techniques for Natural Language Processing[END_REF]Daumé III and Marcu 2005a;[START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF]C. Ma et al. 2014;Rahman and Ng 2011b;[START_REF] Webster | Limited Memory Incremental Coreference Resolution[END_REF][START_REF] Yang | An Entity-Mention Model for Coreference Resolution with Inductive Logic Programming[END_REF]. [START_REF] Rahman | Supervised Models for Coreference Resolution[END_REF] proposed a cluster-ranking approach, where clusters are ranked in the same fashion as the mentionranking approaches. This approach inspired [START_REF] Wiseman | Learning Global Features for Coreference Resolution[END_REF] who used neural networks to build partial cluster representations in an mention-ranking approach. [START_REF] Webster | Limited Memory Incremental Coreference Resolution[END_REF] built a model inspired by shift-reduce parsing.

Several research e orts model the task as a clustering problem (K. Clark and Manning 2015; K. Clark and Manning 2016a;[START_REF] Culotta | First-Order Probabilistic Models for Coreference Resolution[END_REF][START_REF] Stoyanov | Easy-First Coreference Resolution[END_REF]. These models are also called agglomerative-clustering models in the literature. Entities are constructed via merge operation between partially constructed entities. As for its entity-mention counter-part, it allows to take into account information about the considered entities. In this line of approaches, the problem is modeled as a search problem where the objective is to learn the best sequence of merging operations. Similarly to [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF], Daumé III and Marcu (2005) modeled the coreference resolution task as a search over a Bell tree. They made use of the Learning as Search Optimization framework [START_REF] Daumé | Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction[END_REF]. K. [START_REF] Clark | Entity-Centric Coreference Resolution with Model Stacking[END_REF] and [START_REF] Stoyanov | Easy-First Coreference Resolution[END_REF] implemented easy-rst clustering by making easiest coreference decisions before the hardest ones. They relied on the output of a pairwise classi er for the ranking.

The task can also be modeled as a partition induction problem. Finley and Joachims (2005) trained a max-margin ranking model for learning to rank candidate coreference partitions.

Features Although the entity-mention model has not yielded any encouraging results compared to regular mention-pair or mention-ranking models, [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF] showed that it allows to use 20x less features and avoid coreference mistakes such as the one clustering a male and a female pronoun.

Most of the entity-mention models follows the initial work of [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF] by extending the usual feature set to the entity level. For instance distance features takes into account the average distance between the anaphor and the entity. However, di erent logical predicates can be used. For instance, a agreement feature can apply between all, most, any or none of the entity mentions and the active mention. K. [START_REF] Clark | Entity-Centric Coreference Resolution with Model Stacking[END_REF] added the average probability of coreference between mentions in two clusters as a feature. They learned feature combination via the use of neural networks. [START_REF] Yang | An Entity-Mention Model for Coreference Resolution with Inductive Logic Programming[END_REF] argued that the reason why the entity-mention model did not give good results is because the cluster-level features are not built adequately. Instead of devising features at the cluster level, we should build features at the mention level, by taking into account each mention in the cluster. However, the implementation of such a model is limited by the fact that machine learning algorithm usually takes a xed size input vector. Hence, the author used Inductive Linear Programming (ILP) to address this issue. They showed a performance improvement when comparing an entity-mention model learned via ILP versus a mention-pair model learned via ILP.

Clustering Best-rst and closest-rst clustering strategies that were previously used for mention-pair models can be implemented for entity-mention models in a similar fashion. Agglomerative clustering approaches learn from scratch and compare current merging decisions with good merging decisions. For [START_REF] Stoyanov | Easy-First Coreference Resolution[END_REF], good decisions are those who lead to an increase of the evaluation metric.

Training instances Most of the literature working on entity-mention models follow the method presented in [START_REF] Luo | A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree[END_REF] for the creation of training instances. Rahman and Ng (2011) did not follow this schema and created one instance between the active mention and all the partially constructed clusters. [START_REF] Culotta | First-Order Probabilistic Models for Coreference Resolution[END_REF] employed rst order predicates to construct cluster-level features. The naive way of creating one positive instance for each subset of mentions is untractable. Hence, the authors set up error driven sampling where the instances are created from the errors the model makes on the training instances. First they initialize randomly, then perform agglomerative clustering until a mistake is made. They update the weights and repeat for the next document. When the model makes an error, a negative instance is created based on the resulting clustering. A positive instance is created by merging two coreferent clusters. Update is performed via ranking so that the positive instance gets a higher score than the negative one.

Tree-Based Models

Tree-based modeling of the coreference resolution task was initially introduced by Yu and Joachims (2009) but gained popularity after [START_REF] Fernandes | Latent Structure Perceptron with Feature Induction for Unrestricted Coreference Resolution[END_REF] won the CoNLL-2012 coreference shared task [START_REF] Pradhan | CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes[END_REF]. The antecedent tree encodes all pairwise decisions at the document level. The objective in this model becomes to predict a coreference tree. One advantage of this structure is that its enforces that there is only one antecedent per mention. The non-anaphoric case is modeled by a dummy mention, located at the root of the tree. All subtrees are thus entities.

Learning algorithm

The antecedent-tree model was used in several research e orts with light modi cations (Björkelund and Kuhn 2014; K.-W. Chang et al. 2013;[START_REF] Lassalle | Joint Anaphoricity Detection and Coreference Resolution with Constrained Latent Structures[END_REF]. [START_REF] Yu | Learning Structural SVMs with Latent Variables[END_REF] did not use a dummy mention to model the non-anaphoric case and therefore the output of their model is a set of trees instead of a single tree. They used latent structural support vector machines as learning algorithm. [START_REF] Fernandes | Latent Trees for Coreference Resolution[END_REF], [START_REF] Björkelund | Learning Structured Perceptrons for Coreference Resolution with Latent Antecedents and Non-local Features[END_REF] and [START_REF] Lassalle | Joint Anaphoricity Detection and Coreference Resolution with Constrained Latent Structures[END_REF] used Latent structured perceptron [START_REF] Collins | Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms[END_REF]X. Sun et al. 2009) while K.-W. Chang et al. (2013) used stochastic gradient descent.

Features Most models make use of standard features including lexical and combination features. They do not devise any feature for the dummy antecedent with the exception of [START_REF] Lassalle | Joint Anaphoricity Detection and Coreference Resolution with Constrained Latent Structures[END_REF] who devised features for anaphoricity detection that they use for the dummy antecedent.

K.-W. Chang et al. (2013) and [START_REF] Lassalle | Joint Anaphoricity Detection and Coreference Resolution with Constrained Latent Structures[END_REF] used must-link and cannot-link constraints and applied them either during training [START_REF] Lassalle | Joint Anaphoricity Detection and Coreference Resolution with Constrained Latent Structures[END_REF] or inference (K.-W. Chang et al. 2013). The latter did observe a performance increase.

Björkelund and Kuhn (2014) allowed for non-local features such as mention types in an entity. They applied left-to-right decoding via beam-search.

Training instances Most models do not use any resampling but implement cost-sensitive loss functions. [START_REF] Yu | Learning Structural SVMs with Latent Variables[END_REF] rewarded coreferent linking by 1 and penalized linking two non-coreferent mentions by -1 3 . [START_REF] Fernandes | Latent Trees for Coreference Resolution[END_REF] and [START_REF] Björkelund | Learning Structured Perceptrons for Coreference Resolution with Latent Antecedents and Non-local Features[END_REF] used a simpler cost function: edges between two non-coreferent mentions have cost 1 while erroneous links with dummy mention is 1.5. [START_REF] Lassalle | Joint Anaphoricity Detection and Coreference Resolution with Constrained Latent Structures[END_REF] and K.-W. Chang et al. (2013) penalized all wrong links with cost 1.

Coreference Resolution in the Clinical Domain

Literature on coreference resolution in the clinical domain is rather sparse. It was initiated with the i2b2 challenge on coreference resolution [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF]. Participating teams submitted rule-based, hybrid and supervised machine learning systems. Several systems tried to use world knowledge derived from the corpus itself or from external resources such as the UMLS ® Metathesaurus, Wikipedia or Wordnet [START_REF] Fellbaum | fiordNet: An Electronic Lexical Database[END_REF]. [START_REF] Grouin | Coreference Resolution in Clinical Reports. The LIMSI Participation in the i2b2/VA 2011 Challenge[END_REF] modeled the task as a three-step process and devised an mentionpair model. First, they discarded singletons by using a SVM with some handcrafted features. Next, they classify each candidate mention pairs with a combination of string matching patterns and rules. They created a knowledge base from the training corpus which contains all pairs of words that are coreferent. Finally, mention partition is obtained via closure on the coreference graph. [START_REF] Xu | A Classi cation Approach to Coreference in Discharge Summaries: 2011 i2b2 Challenge[END_REF] split the task into three subtasks: one related to people, one related to pronouns and the other related to clinical events. They built a binary classi er to determine whether a mention related to people is a mention of the patient. They selected attributes ranging from mention position to string matching and obtained a high performance (f1-score: 0.996). Concerning the clinical event mentions, they extracted world-knowledge features from Wikipedia, WordNet and other knowledge databases. They combined these features with semantic, grammatical and lexical features. Final partitioning is done via best-rst clustering using the SVM con dence scores.

Outside the i2b2 shared task, Jindal and Roth (2013) built a semantic representation of the medical mentions by extracting their UMLS ® concept IDs via MetaMap [START_REF] Aronson | E ective Mapping of Biomedical Text to the UMLS Metathesaurus: the MetaMap Program[END_REF]. They also derived mention types via Wikipedia. The authors added a normalization step and used a deterministic rule-based algorithm to select the best antecedent. For coreference resolution of mentions referring to people, they devised a two-layer algorithm which separates the mention set in three parts: mentions corresponding to patients, mentions corresponding to any of the doctors and the rest of the mentions.

To the best of our knowledge, there was no attempt do devise an entity-based approach for coreference resolution in the clinical domain. More information about the i2b2 shared task can be found in the original paper [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF].

Evaluation Metrics

Measuring coreference resolution approach performance is hard and is an active research area. The traditional precision, recall and f1-score metrics used in the NLP literature do not t well for this task. A metric should be able to capture how well a given approach has clustered mention together compared to the gold standard. In this section, we will present the di erent metrics that are used to measure performance.

As we present the di erent metrics used in the literature for coreference resolution evaluation, we will use an example borrowed from [START_REF] Pradhan | Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation[END_REF] to illustrate the computation of these metrics:

• the key (K) is composed of two entities with mentions {a, b, c} and {d, e, f, g};

• the response (R) contains three entities with mentions {a, b}, {c, d} and {f , g, h, i};

• the mention e is missing from the response and mentions h et i are spurious.

The MUC Score

The MUC score was proposed by [START_REF] Vilain | A Model-Theoretic Coreference Scoring Scheme[END_REF] for performance evaluation in the sixth and seventh MUCs [START_REF] Hirschman | MUC-7 Coreference Task De nition[END_REF][START_REF] Sundheim | Overview of Results of the MUC-6 Evaluation[END_REF]. It is a link-based metric that looks at how the system outputs partitions in comparison to the reference. The larger the number of components in these partitions is, the worse the performance should be.

The rst step in computing the MUC score is to create partitions with respect to the key and response respectively as shown in Figure 5.1. Applied to our example, MUC precision, recall and f1-score are computed as following: where:

Recall = ∑ N k i=1 (|K i | -|p(K i )|) ∑ N k i=1 (|K i | -1) = (3 -2) + (4 -3) (3 -1) + (4 -1) = 0.40 (5.1) Precision = ∑ Nr i=1 (|R i | -|p ′ (R i )|) ∑ Nr i=1 (|R i | -1) = (2 -1) + (2 -2) + (4 -3) (2 -1) + (2 -1) + (4 -1) = 0.40 (5.2) F1-Score = 2 × precision × recall precision + recall = 2 ×
• K i is the i th key entity and p(K i ) is the set of partitions created by intersecting K i with response entities (cf. the middle sub-gure in Figure 5.1);

• R i is the i th response entity and p ′ (R i ) is the set of partitions created by intersecting R i with key entities (cf. the right-most sub-gure in Figure 5.1);

• N k and N r are the number of key and response entities, respectively.

The MUC metric su ers two major shortcomings (Bagga and B. Baldwin 1998). First, it is unable to reward successful identi cation of singleton clusters as the metric was devised for the MUC datasets which do not annotate singleton clusters. Second, the metric does not consider the impact of coreference errors. Intuitively, a mistake involving a large cluster is more damaging than a mistake involving a smaller one and the MUC metric fails to capture that aspect.

The B 3 Algorithm

The B 3 (Bagga and B. Baldwin 1998) metric was devised to address the shortcomings of the MUC score. It is a mention-based metric which allows to take into account mention cluster sizes. It was originally build to evaluate cross-document coreference resolution [START_REF] Ng | Machine Learning for Entity Coreference Resolution: A Retrospective Look at Two Decades of Research[END_REF].

The main idea behind recall computation is that each mention is assigned a credit computed as the ratio between the number of correct mentions in the predicted entity (which contains the key mention) to the size of the key entity. The nal recall score is the sum of all credits normalized by the sum of key mentions. The precision is computed by switching key and response in the computation.

The original paper from Bagga and B. Baldwin (1998) fails to specify how to score predicted mentions. The examples in the paper describe the case where predicted mentions are the same as the key mentions. Moreover, the authors did not provide any reference implementation, making it di cult to compute the score in this situation. This led to various implementations in the literature. [START_REF] Stoyanov | Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art[END_REF] introduce the notion of twinless mention which denotes mentions which are either spurious or missing from the predicted mention set. The authors describe two variations of the B 3 metric, B 3 all and B 3 0 . In the former, all predicted twinless mentions are retained whereas the latter discards them. [START_REF] Rahman | Supervised Models for Coreference Resolution[END_REF] present one other variation of the metric where predicted twinless mentions which are singletons are discarded before computing the metric. Finally, [START_REF] Cai | Evaluation Metrics For End-to-End Coreference Resolution Systems[END_REF] propose a method where key and predicted mentions are manipulated depending on whether we are computing precision or recall. [START_REF] Pradhan | Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation[END_REF] argue that the B 3 metric was intended to work without any modication in the computation and propose an implementation of the metric. This is the one we are describing in the equations below. The computation of the metric with our running example is as follows:

Recall = ∑ M k i=1 ∑ Rr j=1 |K i ∩R j | 2 |K i | ∑ N k i=1 |K i | = 1 7 × ( 2 2 3 + 1 2 3 + 1 2 4 + 2 2 4 ) = 1 7 × 35 12 ≈ 0.42 (5.4) Precision = ∑ M k i=1 ∑ Rr j=1 |K i ∩R j | 2 |R j | ∑ N k i=1 |R j | = 1 8 × ( 2 2 2 + 1 2 2 + 1 2 2 + 2 2 4 ) = 1 8 × 4 1 = 0.50 (5.5) F1-Score = 2 × precision × recall precision + recall = 2 × 0.42 × 0.50 0.42 + 0.50 = 0.46 (5.6)
The B 3 metric su er from several shortcomings. For instance, the same gold cluster is allowed to be aligned to multiple di erent system clusters and vice versa [START_REF] Luo | On Coreference Resolution Performance Metrics[END_REF]. Therefore, the metric does not correctly penalize systems that output an incorrect number of clusters. Recasens and Hovy (2011) highlight that the metric is highly sensitive to the number of singleton clusters. The metric tends to be less sensitive to how well the systems extract coreference links when in presence of a high number of singletons.

Constrained Entity-Aligned F-Measure

Constrained Entity Aligned F-Measure (CEAF) was originally proposed by [START_REF] Luo | On Coreference Resolution Performance Metrics[END_REF] to address the shortcomings of the B 3 metric. While MUC was a link-based metric, B 3 was mention-based, CEAF is devised as an entity based metric. The main idea of the metric is that an entity should only be used once in the evaluation. MUC and B 3 rely on intersections of entities and therefore entities can be used more than once during the computation.

The main idea behind CEAF metrics is to nd the optimal alignment between gold and predicted clusters. Each gold cluster is aligned to at most one predicted cluster and vice versa. [START_REF] Luo | On Coreference Resolution Performance Metrics[END_REF] describes two similarity scores in their paper: a mention-based (CEAF m ) and an entity-based version (CEAF e ).

Similarly to the B 3 metric, CEAF was considered to be underspeci ed for twinless mentions. This led to the creation of several variations of the metric in the literature that are similar to the variations implemented for the B 3 metric (cf. methods of [START_REF] Rahman | Supervised Models for Coreference Resolution[END_REF] and [START_REF] Cai | Evaluation Metrics For End-to-End Coreference Resolution Systems[END_REF] in previous section). As for the B 3 metric, [START_REF] Pradhan | Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation[END_REF] argue that CEAF was intended to work in the case where predicted mentions are di erent from key mentions and propose an implementation of the metric.

CEAF m recall is the number of aligned mentions divided by the number of key mentions while precision is the number of aligned mentions divided by the number of response mentions [START_REF] Pradhan | Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation[END_REF]). The computation with our running example is as follows:

Recall = |K 1 ∩ R 1 | + |K 2 ∩ R 3 | |K 1 | + |K 2 | = (2 + 2) (3 + 4) ≈ 0.57 (5.7) Precision = |K 1 ∩ R 1 | + |K 2 ∩ R 3 | |R 1 | + |R 2 | + |R 3 | = (2 + 2) (2 + 2 + 4) = 0.50 (5.8)
F1-Score = 2 × precision × recall precision + recall = 2 × 0.57 × 0.50 0.67 + 0.50 = 0.53 (5.9) CEAF e recall and precision are computed:

Recall = ϕ 4 (K 1 , R 1 ) + ϕ 4 (K 2 , R 3 ) N k = (2×2) (3+2) + (2×2) (4+4) 2 = 0.65 (5.10) Precision = ϕ 4 (K 1 , R 1 ) + ϕ 4 (K 2 , R 3 ) N r = (2×2) (3+2) + (2×2) (4+4) 3 ≈ 0.43 (5.11) F1-Score = 2 × precision × recall precision + recall = 2 × 0.65 × 0.43 0.65 + 0.43 = 0.52 (5.12)
where ϕ 4 (K i , R j ) denotes the similarity between a key entity K i and a response entity R j and is computed as follows [START_REF] Luo | On Coreference Resolution Performance Metrics[END_REF]):

ϕ 4 (K i , R j ) = 2 × |K i ∩ R j | |K i | + |R j | (5.13)
Although addressing the shortcomings of the previous metrics, Recasens and Hovy (2011) point out that the CEAF metric is as sensitive to the number of singletons as the B 3 score.

BLANC

The main objective of the BLANC score (Recasens and Hovy 2011) is to provide a metric that is more suitable for situations where singletons are annotated (e.g. in the ACE corpora). It also addresses the shortcoming of the MUC metric which does not take into account the impact of errors.

Two F-scores are computed, one which captures how accurately a system predicts coreference links (F coref ) and another to capture how accurately a system classi es mentions as singletons (F non-coref ). The nal BLANC score is the arithmetic mean between the two Fscores.

The original computation of the metric only addressed the situation where gold mentions were provided. [START_REF] Pradhan | Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation[END_REF] adapted the metric to the end-to-end situation where gold and predicted mentions could mismatch.

Let C k be the set of coreference links in the key and C r be the set of coreference links in the response. Let N k be the set of non-coreference links in the key and N r be the set of non-coreference links in the response. A link between a mention pair m and n is denoted as mn. The set states for our example are as follows:

C k = {ab, ac, bc, de, df, dg, ef, eg, f g} N k = {ad, ae, af, ag, bd, be, bf, bg, cd, ce, cf, cg} {ac, ad, af, ag, ah, ai, bc, bd, bf, bg, bh, bi, cf, cg, ch, ci, df, dg, dh, di} F coref is computed as follows:

C r = {ab, cd, f g, f h, f i, gh, gi, hi} N r =
R c = |C k ∩ C r | |C k | = 2 9 ≈ 0.22 (5.14) P c = |C k ∩ C r | |C r | = 2 8 = 0.25 (

The CoNLL Score

Each metric described above measures how well a system performs for coreference resolution with respect to a certain aspect. The consequence of this diversity is that it is di cult to compare system performances. The CoNLL score [START_REF] Pradhan | CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes[END_REF][START_REF] Pradhan | CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes[END_REF] has been devised as a way of combining the strength of these metrics. It is computed as the unweighted arithmetic mean of MUC, B 3 and CEAF e scores. While each metric provides a view of systems strengths and weaknesses, the CoNLL score allows to compare objectively system outputs according to three dimensions.

An implementation of all the metrics described in the section is available in the CoNLL scorer1 [START_REF] Pradhan | Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation[END_REF].

Conclusion

Coreference resolution is a complex NLP topic that brings together several research domains. Carefully annotated corpora are needed to foster new approaches for coreference resolution. As we showed in this chapter, the linguistic phenomenon called coreference has been di cult to grasp in the NLP community and led to several annotation mistakes that have since been corrected.

Beyond the annotation process, devising approaches for coreference resolution is a di cult task. The literature has evolved from local pairwise classi ers to entity-based models and succeeded in improving the state-of-the-art on the topic. However, research on coreference resolution in the clinical domain remains sparse.

In the next chapter, we present an neural entity-based approach for coreference resolution in the clinical domain. It is based on an entity-aware mention-ranking model inspired by recent research e orts on the topic [START_REF] Wiseman | Learning Global Features for Coreference Resolution[END_REF]

Introduction

This chapter addresses coreference resolution in the clinical domain. Recent neural approaches have allowed for better performance in the general domain. Speci cally, entitybased approaches seem to be a promising area of research (K. Clark and Manning 2016a;[START_REF] Wiseman | Learning Global Features for Coreference Resolution[END_REF]) although simple neural-based mention-ranking models present stateof-the-art results (K. [START_REF] Lee | End-to-end Neural Coreference Resolution[END_REF]. To the best of our knowledge, these types of models have not been transposed to the clinical domain. Hence, we present a neural entity-based approach for coreference resolution in the clinical domain. Speci cally, we experiment with various model con gurations that include character-level representations, attention mechanisms and network pretraining and measure their impact on the nal coreference score.

As we mentioned in the introduction, temporal information extraction and coreference resolution are interlinked. Temporal clues can be valuable for a coreference resolution system. In this context, we devise a temporal feature based on the relationship that exists between medical events and DCTs and embed that feature into our model.

The remainder of the chapter is organized as follows. We present the corpus that has been used in our experiments in Section 6.2. We discuss how we split the task into two subtasks in Section 6.3. We present our approach for mention extraction in Section 6.4 and for building a temporal feature in Section 6.5. We describe our model components in Section 6.6 and our experimental setup in Section 6.7. Results of our experiments are presented in Section 6.8. Finally, we discuss these results in Section 6.9 and we conclude the chapter in Section 6.10.

Data

As we saw in Chapter 5, there are only two clinical corpora annotated with coreference chains available to the community. We chose to use the i2b2 corpus task1c in our experiments [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF]. Speci cally, we selected the i2b2/VA part that does not contain documents from the University of Pittsburgh Medical Center (UPMC). This situation mimics one of the experimental setup that was proposed to the participants during the shared task. The dataset contains 194 clinical documents from the Beth Israel Deaconess Medical Center (BETH) and 230 clinical documents from Partners Healthcare (PARTNERS) (Table 6.1). The number of coreference chains, and their average and maximum lengths (i.e. the number of mentions per chain) are presented in Table 6 The corpus is annotated with ve concept (i.e. entity) types: person, pronoun, test, treatment and problem. Singletons are annotated (i.e. entities that do not take part in any coreference chain). [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF] do not mention explicitly that the medical entities (tests, treatments and problems) are event mentions. However, as we saw in Chapter 5, these entity mentions can be considered as event mentions when found in a clinical document. Pronouns participate in either people-related chains or event-related chains. The relation annotated in the corpus is the one of identity (cf. Chapter 5, Section 5.2). Other anaphoric relation types (e.g. set-subset) are not annotated.

Corpus Format Conversion

The documents are shipped as i2b2-formatted les. Each clinical text has three associated documents: a text le that contains the raw text, a concept le that holds entity o sets and types, and a chain le that tracks coreference chains. The text le holds the text content of the clinical document. It has been segmented in sentences and tokenized. There is one sentence per line and tokens are separated by spaces.

The concept le encodes entity positions following a custom bi-o set format. Each o set boundary (begin or end) is a 2-dimension coordinate where the rst dimension points to the line, and the second points to the token (Figure 6.1). This o set information is completed with the surface form of the concept and its type. There is one entity per line.

... c="thrombocytopenic" 43:3 43:3||t="problem" c="back surgery" 25:1 25:2||t="treatment" c="platelets" 42:13 42:13||t="treatment" ... The chain le contains coreference chains. A chain is composed of several entities which are encoded similarly to the format described above except that their types are omitted. The chain type is encoded via a custom chain attribute (Figure 6.2). There is one chain per line.

... c="hepatitis c cirrhosis" 12:16 12:18||c="hepatitis c cirrhosis" 20:1 20:3||c="which" 20:5 20:5||t="coref problem" c="transplant" 48:8 48:8|| c="an orthotopic liver transplant" 49:10 49:13|| c="the procedure" 50:22 50:23||c="the procedure" 53:0 53:1|| c="the procedure" 53:20 53:21||c="surgery" 59:13 59:13|| c="the liver transplant" 59:32 59:34||c="liver transplant" 88:0 88:1|| t="coref treatment" ... The corpus was released during the i2b2 evaluation campaign on coreference [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF]. Participants were asked to format their system outputs to match the i2b2 format. Evaluation was performed with a custom Python script coded for the shared task1 .

Preliminary experiments performed with the corpus and the evaluation script revealed several di culties and problems. First, the i2b2 annotation format is cumbersome to use. The vast majority of text processing tools are using character-based o sets to encode annotations within documents. This has become the usual way of devising NLP pipelines. Furthermore, by enforcing a line/token o set system, the i2b2 format can lead to annotation mismatch as the way of segmenting lines may vary (e.g. the way of splitting a double space).

Second, the evaluation script implements custom versions of the B 3 and CEAF metrics. The script has been devised before Pradhan's implementation of the coreference metrics that was used during the CoNLL challenges [START_REF] Pradhan | Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation[END_REF]. Metric computation is done by modifying key and response partitions. Preliminary comparisons between the CoNLL scorer and the i2b2 evaluation script outputs revealed discrepancies between scores. Furthermore, the script su ers from an important computation time (several minutes for the CEAF metric).

These observations motivated us to implement the transformation of the corpus to the CoNLL format. This conversion has two main advantages. It will allow the community to work on coreference in the clinical domain more easily by using a well-known and well-tested format. Furthermore, it allows to use the CoNLL scorer for evaluation which has become the reference evaluation script for researchers working on coreference resolution.

One drawback of using the CoNLL format is that the entity type information is lost. This information is important when dealing with coreference in the clinical domain as two mentions that are not of the same type are not likely to be coreferent. Thus, in order to keep this valuable information, we also convert the corpus to the brat format. Mapping between brat and CoNLL versions of the corpus is performed via o set tracking in the CoNLL version. The entire transformation process is documented and the scripts will be available online.

Conversion quality was tested by converting the resulting CoNLL les back to the i2b2 format. We asserted the quality by using the o cial evaluation script on the converted les and obtained an optimal score. Brat version statistics are presented in Table 6.3. We report the number of entities for each type (person, problem, pronoun, test and treatment)

Task Division

The annotated mentions can be divided in two groups. The rst regroups event mentions (tests, treatments and problems). The second one comprises people mentions (person). Both groups di er in several ways. First, event and people entities di er from a semantic view- point. As we saw in Chapter 5, despite the fact that clinical events are often realized by noun phrases, they hold an implicit argument structure while person-related entities do not. Second, coreference chain statistics presented in Table 6.4 show that coreference chains related to people tend to be longer (more than ten mentions per chain) and the maximum length is much larger than the one of other chain types.

Finally, the vast majority of mentions in coreference chains related to people takes the form of personal pronouns. The diversity of surface forms is therefore limited compared to the one of clinical events.

Considering these three major di erences, it seems legitimate to consider both coreference resolution tasks as di erent subtasks. It seems that the features used for supervised learning in both cases will be di erent enough to train two separate models.

Pronouns can be found in either the chains related to clinical events or those related to people. A distribution of pronouns according to the chain type is presented in Table 6.5. Pronouns are mostly singletons (60.76%) 2 . The remaining are distributed across all chain types. We note that coreference chains related to people contain less pronouns than the others (3.79%).

Following these observations, we decided to include all pronouns in both models (people and clinical events). Our hypothesis is that the models will be able to learn which pronoun has to be included in the chains.

To summarize, our approach consists in dividing the coreference task into two subtasks, one related to people mentions, the other to clinical event mentions. In both subtasks, all pronoun entities are included.

Gold vs. End-to-End Coreference Resolution. Up to this point, we considered the situation where gold mentions are given and the task is to extract coreference links. This setting is often considered as a simpli ed version of the task. To measure the e ectiveness of our approach on system-predicted mentions, we devise a NER module for mention extraction and apply our coreference resolution models on these mentions.

Mention Extraction

Before presenting how we performed mention extraction, we would like to clarify some terminology that we will use in the rest of this chapter. Coreference resolution objective is to cluster event or entity mentions that are referring to the same real-world event or entity. However, to smooth the description of our approach, we will refer to event mentions as events and people mentions as people. We will use the word real-word (event or person) to di erentiate between the two concepts. Mention extraction is performed via a hybrid approach. Events (tests, pronouns and problems) are extracted via a fully supervised machine learning approach using YASET [START_REF] Tourille | Evaluation of a Sequence Tagging Tool for Biomedical Texts[END_REF]). One model is learned for the three event types. A detailed description of the neural model is presented in Chapter 4, Section 4.3.1.

People are extracted via a combination of regular expressions and supervised machine learning. We noticed that several mentions overlap with each other, (e.g. the mentions his and his primary physician). In all overlapping situations, we removed the mentions with the smallest span length and devised a regular expression for each of them. The nal list contains six mention forms: his, her, your, the patient, the pt and our. For the main group of mentions, we learn a NER model with YASET.

Pronouns are extracted via the use of regular expressions. Pronoun list includes: it, its, that, their, them, these, they, this, those, which. Originally, the pronoun "there" was included in the list but it seems that its annotation within the corpus is not coherent. There is a large number of them which are not annotated and among those which are annotated, only one participates to a coreference chain. Therefore, we decided to remove the pronoun from the list. Extracted pronouns are added to the list of mentions in both event and people related coreference models.

Hyperparameters.

For both models, we used word embeddings computed3 with the gensim implementation of word2vec on the MIMIC III corpus. Mini-batch size is set to 8. Main Bi-LSTM hidden layer size is set to 100. We initialized character embeddings randomly and set their size to 25. The character-level Bi-LSTM hidden layer has a size of 25. We set the We report precision (P), recall (R) and f1-score (F1) for all coreference metrics.

Building a Temporal Feature

Event temporal location seems to play a major role in event coreference resolution (cf. Chapter 5, Section 5.4). Being able to take into account temporal clues in a coreference resolution system could improve its performance. However, since the i2b2 corpus does not include temporal information, it has therefore to come from another source. Usually, temporally annotated datasets contain two types of temporal relations: relations between event and/or temporal expressions and relations between event and the DCT (cf. Chapter 2). Intuitively, the DCT relation could bring a valuable information to a coreference resolution system. Two events which have similar or non-contradictory DCT relations (e.g. before and before-overlap) have a higher chance to be coreferent than two events happening respectively before and after the DCT.

Among available corpora, the THYME corpus seems to be the more suitable to learn a DCT relation extraction model. However, due to the fact that events from the i2b2 and THYME corpora have di erent de nitions, it would be dubious to try to assign a DCT relation to i2b2 events with a model learned on the THYME corpus. Event de nition in the THYME corpus is larger and regroups clinical concepts beyond the three event categories annotated in the i2b2 corpus.

The solution we adopted consists in learning a NER model for event extraction on the THYME corpus by incorporating the DCT relation into the IOB tagging scheme. The model extract events with their associated DCT relations. Then, this model is used to extract THYME events in the i2b2 corpus. This approach has several drawbacks. The model learned on the THYME corpus will extract much more events than there are in the i2b2 corpus as the de nition of an event in the THYME corpus is more loose than the one of the i2b2 corpus.

Furthermore, it is not sure that every i2b2 events will have a corresponding THYME event extracted within its span. Following this observation, we decided to build a temporal feature based on the sentence context of the events. Each i2b2 event is assigned a DCT relation based on the DCT relations of THYME events extracted in the sentence. For instance, if an i2b2 event is located in a sentence comprising two THYME-predicted events which have two DCT relations before and before-overlap, the nal DCT relation associated with the i2b2 event will be before/before-overlap. If there is no THYME event within the sentence, the DCT relation is declared as unde ned.

There is one condition for this feature to be useful: sentences must be temporally coherent, meaning that they must not contain contradictory DCT relations. From our perspective, temporally coherent sentences relate clinical events that are temporally located in a semi interval started or closed by the DCT. For instance, a sentence containing before, before/overlap and overlap DCT relations would be considered as coherent. The same goes for sentences which combine overlap and after DCT relations. However, this would not be the case for sentences having both before and after DCT relations.

We verify sentence temporal coherence in the training data (i.e. the THYME corpus). Figure 6.3 shows the number of sentences according to the number of di erent DCT relations they contain. The vast majority of sentences which contain events have only one type of DCT relation (78.86%). For those sentences, there is therefore no contradictory temporal information.

Approximately 18% of sentences contain at least two events with di erent DCT relations. Figure 6.4 presents the number of sentences per pair of DCT relations. Among those sen-

Input Embeddings

Similarly to the method presented in Chapter 4 Section 4.3.4, input embeddings are built by concatenating a character-based embedding and a pre-computed word embedding.

An overview of the character-based embedding computation is presented in Figure 4.4. Following [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF], the character-based representation is constructed with a Bi-LSTM. First, a random embedding is generated for every character present in the training corpus. Token characters are then processed with a Bi-LSTM (forward and backward LSTM in Figure 6.9). The nal character-based representation is the result of the concatenation of the forward and backward representations.

Mention Representation

An overview of mention representation computation is presented in Figure 6.10. For a given mention, embedded in its sentential context, we compute two contextual representations with two LSTMs. The forward LSTM processes the sentence from the rst token of the sentence to the last token of the mention (forward LSTM in Figure 6.10). The backward LSTM processes the sentence from the last token of the sentence to the rst token of the mention (backward LSTM in Figure 6.10). We implemented another approach during development where each contextual mention representation was composed of three parts: left and right contexts, and the mention itself. Each part was computed by processing the tokens with a bi-LSTM. However, this solution largely decreased the performance of the model.

Then, these two left and right contextual representations are concatenated. We add two feature embeddings, one to represent the temporal feature (DCT relation) and one to represent the mention type (person, pronoun, test, problem or treatment).

Finally, we add an attention mechanism. Attention on both left and right contexts is computed via a weighted sum of the LSTM hidden states. Weights are computed using a feed forward network with one hidden layer. Attention has proven to be useful for coreference resolution (K. [START_REF] Lee | End-to-end Neural Coreference Resolution[END_REF]. It allows the network to focus on the most informative parts of the input and can further increase the performance of a given approach.

Cluster-Level Representation

Similarly to [START_REF] Wiseman | Learning Global Features for Coreference Resolution[END_REF], we used a LSTM to build cluster representations. To produce a cluster representation, we process its mentions with a LSTM by order of apparition in the text. We use mention representations that have been computed following the method described in the previous section. An overview of the cluster representation computation is presented in Figure 6.11. Our hypothesis is that this cluster-level representation will be able to capture similarities within the cluster. These similarities may be found in the DCT relations, in the type of the mentions or the linguistic context of the mentions.

From an implementation viewpoint, as we process the text from left to right, we maintain n cluster representations. These representations are built with the same LSTM (i.e. the same set of parameters) and updated after each mention processing. Singleton clusters are composed of only one mention.

Pairwise Scorer

An overview of the computation process is presented in Figure 6.12. The pairwise scorer assigns a score to every antecedent-mention pair. It takes as input the active mention and the antecedent representations. The latter is composed of the mention representation as described in Section 6.6.2 and the partial cluster representation in which the antecedent belongs as described in Section 6.6.3. We add a distance feature embedding representing the number of sentences separating the active mention and its antecedent. Following K. Clark and Manning (2016), the distance is binned into the following buckets: [1, 2, 3, 4, 5-7, 8-15, 16-31, 32-63, 64+]. This input is then fed to a feed forward neural network with two hidden layers. Each hidden layer has a size equal to the half of the preceding layer size. The output of the network is the antecedent-mention pair score.

Pairwise Scorer Specialization. We implement pairwise scorer specialization according to the mention type. In this scenario, we build a di erent feed forward network for each mention type (person, pronoun, test, treatment, problem). During learning, depending on the active mention type, the corresponding feed forward network will be used to compute all pairwise scores. This approach can be seen as similar to multitask approaches which have been shown to be e cient in several domains, as for instance in sequence labeling [START_REF] Crichton | A Neural Network Multi-task Learning Approach to Biomedical Named Entity Recognition[END_REF][START_REF] Rei | Semi-supervised Multitask Learning for Sequence Labeling[END_REF]).

Training

Individual antecedent-mention scores are then combined and a softmax layer is applied. Following K. [START_REF] Lee | End-to-end Neural Coreference Resolution[END_REF], we optimize the marginal log-likelihood of all correct antecedents implied by the gold clustering:

log N ∏ i=1 ∑ ŷ∈Y(i) P (ŷ) (6.1)
where Y(i) is the set of possible assignments for each y i (Y(i) = {ϵ, 1, ..., i-1}) with the dummy antecedent ϵ and all preceding mentions.

We x the score of the dummy antecedent to zero. This strategy prevents antecedent ltering to introduce noise (i.e. the case where all gold antecedents have been pruned). In this situation, the learning objective will push the scores of non-antecedent mentions lower and not incorrectly push the score of the dummy antecedent higher (K. [START_REF] Lee | End-to-end Neural Coreference Resolution[END_REF]). We experimented with the cost-sensitive slack-rescaled learning objective presented in [START_REF] Wiseman | Learning Global Features for Coreference Resolution[END_REF] and [START_REF] Durrett | Easy Victories and Uphill Battles in Coreference Resolution[END_REF] but the learning objective presented in K. [START_REF] Lee | End-to-end Neural Coreference Resolution[END_REF] proved to be more e cient.

Wrap-Up

This section aims to summarize how the network pieces work together. Let consider the example presented in Figure 6.13. The system is processing one document, composed of one sentence. Five mentions have already been processed in the document and the current active mention is the entity "your". The mention-ranking approach consider all mentions located before the active mention as potential true antecedents. We add the case where the active mention is consider as non anaphoric.

The system has already predicted that there are three partial entities up to this point in the document composed of the following mentions: (you, you), (further chest pain) and (other symptoms, which). The rest of the process is as follows: Attention. Our attention mechanism allows for a small performance improvement, suggesting that the sentential context of the mentions brings only minor information for the classi cation. The event model is the one that bene ts the most from this mechanism.

Multitasking. The multitasking approach, where the top-end feed forward layer is specialized according to the entity type brings a large decrease in performance (-1.8 F1). One possible reason for this performance drop is the limited size of our corpus. We have only 200 documents to train on while the main corpus for coreference resolution in the general domain proposes more than 2,000 documents [START_REF] Pradhan | CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes[END_REF][START_REF] Pradhan | CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes[END_REF].

Characters Embeddings. The use of character embeddings slightly decreases the performance of our approach (-0.06 F1). The character-level representation does not bring as much information as it is the case for other tasks such as NER [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF]X. Ma and Hovy 2016). Table 6.14 presents the number of tokens that compose the corpus mentions. We report the number of unique tokens and the number of unknown tokens, i.e. those for which we do not have a pre-computed embedding. We observe that most tokens have a pre-computed embedding. Character embeddings are used to provide an alternative representation in the case where there is a high proportion of unknown tokens. They also allow to account for character casing, su xes and pre xes These features have proved to be useful in previous NER e orts [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF]). In our case, the actual form of the tokens seem to play little role for coreference resolution. Temporal Feature. The last component, the temporal feature embedding, decreases the overall performance of our approach (-1.79 F1) with a larger gap for the event model (-3.31 F1) than for the people model (-0.5 F1). Looking at these results, it seems that our temporal feature did not capture useful temporal information for coreference resolution. In that case, we need to rethink what kind of temporal information could be appropriate for coreference resolution. One other explanation could be that the quality of this feature is not su cient. The THYME and i2b2 corpora are not from the same clinical subdomain. During the 2017 edition of the Clinical TempEval shared task [START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF], we obtained a f1-score of 0.51 for DCT relation extraction. The train corpus was composed of documents related to colon cancer patients and the test corpus comprised documents related to brain cancer patients. This represents a drop of -0.246 F1 in comparison to the best result obtained during the 2016 edition [START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF] where both train and test corpora were from the same subdomain (colon cancer patients). The di erence between the THYME and i2b2 corpora seems to be more pronounced than it was during the Clinical TempEval shared task. This could lead to an even more pronounced drop in performance.

Also, the way our approach incorporates the temporal feature is maybe not the best solution. During development, we tried to use a feature embedding at token-level by concatenating it to the character-level representation and the pre-computed word embedding without success. Other possibilities should be explored such as antecedent ltering according to their DCT relation. However this needs a carefully made mapping to account for all DCT combinations.

Finally, the way of computing DCT relations may not be appropriate. Assigning only one relation instead of a concatenation of relations could improve the performance. However, this would result in a large proportion of undetermined relations.

Predicted Mentions. Concerning the decrease observed when extracting coreference links on predicted mentions, the di erence between people and event model drops can be explained by the performance of our NER model (Table 6.8). People and pronouns obtained a very high f1-score, well balanced between precision and recall. The consequence is that error propagation will be not as prominent as it is for the event model.

Although we observe a performance drop when using the character embeddings on both the people and event models, the combined score shows an improvement. This happens due to a combination of several e ects. First, we are dealing with averages. There is an outlier for the event model (Visible on Figure 6.15). If we were to remove this outlier, we would observe a performance increase for the event model. Second, event and people models are combined by pairs (one run implies with our approach implies one pair of models). It means that the way they are combined is completely arbitrary. Finally, chains sizes combined to model performances a ects the metrics when both prediction sets are combined. All these parameters explains the counter-intuitive result showed in Table 6.12. We note that the improvement is anyway very small and cannot be considered as signi cant.

Optimal Coniguration. Finally, we experiment with an optimal con guration where components that bring a performance improvement are used. For the people model, we use pretraining, the attention mechanism and the multitasking approach. For the event model, we use pretraining, the attention mechanism and the ltering of entities according to their type.

Results of this experiment are presented in Table 6.15. We report the performance obtained on gold and predicted mentions. Predicting on gold mentions, we obtain an overall f1-score of 73.82. The event model reaches a f1-score of 62.16, which is an improvement of +0.17 F1 in comparison to the best score obtained on gold mentions during development. As for the people model, the performance is lower than the best one obtained during development (i.e. attention mechanism) suggesting that the individual bene ts of each component do no add up to each other when combined or that the information captured by these three components are contradictory (-0.02 F1).

Concerning the scores obtained on predicted mentions, the overall f1-score is 65.96, representing a drop of -7.86 F1 compared the scores obtained on gold mention with the same model. The event model is the one which decreases the most with a drop of -5.95 F1 while people model performance decrease only by -0.45 F1. Table 6.15: Best con guration performance. We report precision (P), recall (R) and f1-score (F1) for all four coreference metrics (MUC, B 3 , CEAFe, CoNLL). We present the scores for the con gurations where gold mentions and predicted mentions are used.

Comparison to the Literature. We converted several system outputs submitted to the i2b2 shared task (task1c). Although we had access to other submissions besides the ones presented in this section, we were not able to convert them due to le format problems. System performances are presented in Table 6.16. Our approach is competitive with the ones presented during the workshop which reached f1-scores ranging from 68.34 to 73.41.

Most approaches devised for coreference resolution in the clinical domain use a rich feature set to build mention representations. These features are extracted from the text itself and from external resources such as clinical knowledge bases. Our model achieves similar performance without the use of such features. Incorporating this information into our model as we did in our temporal information extraction approach (cf. Chapter 6) could improve the global performance.

MUC

73.82

(± 0.47) Table 6.16: Performance comparison with other system outputs submitted during the i2b2 shared task [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF]. We converted the runs to the CoNLL format and evaluated with the o cial CoNLL scorer. We report precision (P), recall (R) and f1-score (F1) for all four coreference metrics (MUC, B 3 , CEAF e , CoNLL).

Conclusion

In this chapter, we presented a neural entity-based approach for coreference resolution in the clinical domain. Our attempt to include a cluster-level representation in our system did not bring any performance improvement. It may suggest that our approach is not suitable for the clinical domain. Other possibilities for building an entity-aware approach will be investigated. For instance we will look at recent cluster-merging approaches such as the one presented in K. Clark and Manning (2016) or joint approaches where mentions and coreference links are learned together such as the one presented in K. [START_REF] Lee | End-to-end Neural Coreference Resolution[END_REF]. The temporal feature did not bring any performance improvement. This result could due to the quality of our feature, to the way of using it into our network or to the nature of the feature itself.

Our experiments suggest that pretraining, which has proven to be valuable for coreference resolution in the general domain, allows for better performance in our situation. Then, character embeddings seem to play little role for coreference resolution as they do no bring any performance improvement. The same observation can be made for our ltering and multitasking approaches, although their poor performance could be due to the limited size of our corpus.

Overall, our model reaches a f1-score of 65.96 (when using predicted mentions) and 73.82 (when using gold mentions) and is competitive with the scores obtained during the i2b2 shared task. With the transformation of the i2b2 corpus to CoNLL format and its future release, other research teams will have the possibility to develop coreference resolution models for the clinical domain and may compare objectively their results using a reference implementation of the coreference metrics. Extracting clinical timelines from electronic health records is a complex task that is related to several NLP research topics. This thesis presented several contributions to temporal information extraction and coreference resolution in clinical narratives. This chapter makes a brief summary of the thesis (Section 7.1), discusses future research directions (Section 7.2) and addresses some of the remaining challenges for clinical timeline extraction (Section 7.3).

Summary Temporal Information Extraction

In the rst part of this thesis, dedicated to temporal information extraction, we started by reviewing the literature on the topic. We showed that time is a complex linguistic phenomenon that has been researched for a long time by the community. We presented the resources annotated with temporal information. Then, we reviewed computational linguistics approaches that have been devised for automatic temporal information extraction.

Our rst main contribution to the topic is a feature-based approach for narrative container extraction. We tested our approach on clinical documents written in English and showed competitive performance. As we mentioned in the introduction, research e orts in the NLP community tend to be biased toward the English language as most annotated corpora are written in that language. However, we showed that our approach can be adapted to other languages by replacing language sensitive resources along our preprocessing pipeline. We experimented on a corpus of clinical documents written in French and obtained comparable results to the ones obtained on the English dataset.

Our second main contribution is a neural-based approach for temporal information extraction. We devised an approach that makes use of the rich feature sets available in most corpora. We studied how the inclusion of such categorical features in uence the performance of our approach. Then, we performed an evaluation of our approach in the context of the 2017 edition of the Clinical TempEval shared task that included a track on domain adaptation and obtained competitive performance. Furthermore, noticing a lack of e cient neural sequence labeling tools in the community, we decided to pack our module for NER into an open source tool called YASET available online1 . This tool reaches state-of-the-art performance on various NER tasks.

Coreference Resolution

The second part of our thesis was related to clinical event coreference resolution. We started by reviewing the literature on the topic. We presented the di erent aspects of this phenomenon from a linguistic perspective. Speci cally, we addressed the terminology di erences between linguistics and computational linguistics. We emphasized the di erences between event coreference in the general and the clinical domains. Then, we reviewed the literature in computational linguistics. We presented the resources and the approaches that have been devised for automatic coreference resolution.

In our main contribution to the topic, we devised a neural entity-based coreference resolution approach. We performed an empirical study with several con gurations ranging from the use of an attention mechanism to pretraining the network, to measure how the use of these components in uences the overall performance. As we mentioned in the introduction, coreference resolution could bene t from temporal information. To verify this hypothesis, we devised a temporal feature based on the DCT relations that was included in our neural model. Unfortunately, this feature did not bring any performance improvement.

Future Research Directions

Starting from the work presented in this thesis, several research directions arise. First, our approaches for temporal information extraction and coreference resolution are done in a pipeline fashion where each step is independent from the other. However, learning jointly multiple tasks can be bene cial. Clues used in several subtasks add-up to each other and improve the overall performance. For instance, [START_REF] Leeuwenberg | KULeuven-LIIR at SemEval-2017 Task 12: Cross-Domain Temporal Information Extraction from Clinical Records[END_REF] devised an approach for jointly learning containment and DCT relations. For coreference resolution, K. [START_REF] Lee | End-to-end Neural Coreference Resolution[END_REF] learned a joint model for mention extraction and coreference resolution and improved the state-of-the-art in the general domain.

In addition, both NLP topics addressed in this thesis involve structured predictions where classi cation decisions depend on previously made decisions. In this context, several approaches can be used to take into account this dependence. For instance, imitation learning is an active machine learning topic which allows to model such problems adequately. It has been successfully applied for coreference resolution (K. Clark and Manning 2016a) and could be applied to temporal relation extraction as well.

Then, the experiments involving the use of categorical features in our neural network approaches showed promising results. Their individual bene t need to be assessed. Furthermore, the way of incorporating them into a neural network need also further investigation. Positioning them at the adequate depth level in our network could bring further performance improvement.

Finally, within-sentence relation extraction can bene t from the syntactic structure of the sentence. It has been tested in the general domain and showed promising results [START_REF] Miwa | End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures[END_REF]. The idea can also be implemented at the scale of the document (cross-sentence relation extraction). [START_REF] Peng | Cross-Sentence N-ary Relation Extraction with Graph LSTMs[END_REF] performed n-ary relation extraction over multiple sentences by modeling documents as graphs. They extracted drug-gene interactions from a large corpus of biomedical research papers.

Extracting Clinical Timelines: Are We There Yet?

Automatic timeline extraction from text is still an unresolved task. Among all remaining issues is the fact that most research e orts on the subject address within-document clinical timeline extraction. As we mentioned in the introductory chapter, the objective is to provide a temporal summary of a given electronic health record. This task requires corpora annotated with cross-document temporal information and coreference links. Moreover, the community needs an annotated resource that provides target timelines as examples.

This rises another issue related to data availability and corpora creation. Clinical data is sensitive by nature and must me handled appropriately. Most annotation e orts are unknown from the NLP community as they cannot be distributed freely. Moreover, the annotation process is highly costly as the knowledge needed to annotate clinical concepts can only be acquired with a thorough training.

Although unstructured data taking the form of raw text is the principal source of information in EHR, there is structured data that is not yet used in NLP research e orts. This structured data includes laboratory results, prescriptions and vital recording. Being able to leverage this structured information by combining it to information extracted from clinical documents could bene t to clinical timeline extraction.

Another issue concerns the diversity of clinical subdomains. The clinical domain is in fact a collection of subdomains where text genre, style and vocabulary may di er. As we have shown in this thesis, the performance drops signi cantly when train and test domains are not the same. As we cannot annotate all clinical subdomains, the NLP research community needs to work on clinical domain adaptation.

In this thesis, we worked with event mentions. For temporal information, we extracted relations between event mentions and/or temporal expressions, while for coreference resolution, we regrouped event mentions that are referring to the same real-world event. However, if we want to be able to query information extracted from EHRs, we need to normalize these events by associating them to a clinical concept (e.g. a clinical concept from UMLS ® ). The task is called entity normalization and is an active research topic.

Finally, another issue concerns the complexity of the involved NLP tasks. As we showed in this thesis, both temporal information extraction and coreference resolution tasks can not be considered as resolved. The linguistic phenomena involved in these tasks are di cult to capture within annotation schemes and automatic extraction approaches. Further research e orts are needed on these two complex topics.

#begin document (clinical-587); ... 14 One 520 523 14:0 -14 month 524 529 14:1 -14 prior 530 535 14:2 -14 to 536 538 14:3 -14 this 539 543 14:4 (18) 14 admission 544 553 14:5 -14 ,554 555 14:6 -14 the 556 559 14:7 (0 14 patient 560 567 14:8 0) 14 reports 568 575 14:9 -14 not 576 579 14:10 -14 feeling 580 587 14:11 -14 well 588 592 14:12 -14 with 593 597 14:13 -14 worsening 598 607 14:14 (17 14 gastric 608 615 14:15 -14 distress 616 624 14:16 17) 14 . 625 626 14:17 -... #end document Des informations importantes sont contenues dans les dossiers patients électroniques. La majeure partie de ces informations est localisée dans des textes écrits en langue naturelle. Bien que le texte libre soit pratique pour exprimer des concepts médicaux complexes, il est très di cile de s'en servir dans le cadre d'applications telles que l'aide à la décision, la recherche clinique, l'analyse statistique ou le résumé automatique. Le besoin d'accéder à ces informations combiné à l'adoption massive du dossier patient électronique a favorisé le développement d'approches en Traitement Automatique des Langues (TAL) spéci ques au domaine clinique.

Les méthodes pour l'extraction d'information ont été appliquées avec succès à une variété de tâches durant les dix dernières années. Un exemple typique est l'assignation de codes de diagnostics tels que les codes ICD. Plusieurs jeux de données relatifs au sujet ont été distribués dans la communauté TAL a n de favoriser la recherche sur ce sujet. Par exemple, [START_REF] Pestian | A Shared Task Involving Multi-label Classi cation of Clinical Free Text[END_REF] proposent de travailler sur des rapports de radiologie tandis que [START_REF] Névéol | Clinical Information Extraction at the CLEF eHealth Evaluation lab 2016[END_REF] ont distribué un ensemble de certi cats de décès annotés avec des codes ICD-10.

Un autre domaine de recherche actif concerne l'enrichissement sémantique des dossiers électroniques pour l'aide à la décision. [START_REF] Meystre | Extracting Information from Textual Documents in the Electronic Health Record: a Review of Recent Research[END_REF] identi ent plusieurs sujets : structuration automatique de documents qui consiste à segmenter le texte libre en sections selon un schéma dé ni préalablement, le résumé automatique qui permet d'avoir une vue concise des documents cliniques et la recherche de cas qui permet de chercher des patients répondant à des critères cliniques spéci ques. L'initiative i2b2 a mené plusieurs campagnes d'annotation portant sur le tabagisme [START_REF] Uzuner | Identifying Patient Smoking Status from Medical Discharge Records[END_REF], l'obésité et les comorbidités [START_REF] Uzuner | Recognizing Obesity and Comorbidities in Sparse Data[END_REF], les médicaments [START_REF] Uzuner | Extracting Medication Information from Clinical Text[END_REF]) et les concepts, assertions et relations [START_REF] Uzuner | 2010 i2b2/VA Challenge on Concepts, Assertions, and Relations in Clinical Text[END_REF].

La surveillance est aussi un champ de recherche important. Un exemple d'application est la détection d'événements indésirables (Velupillai et al. 2015a). Ces événements peuvent être relatifs à des procédures médicales [START_REF] Penz | Automated Identi cation of Adverse Events Related to Central Venous Catheters[END_REF]) ou à la prise de médicaments [START_REF] Wang | Active Computerized Pharmacovigilance Using Natural Language Processing, Statistics, and Elec-References tronic Health Records: A Feasibility Study[END_REF]. Un autre cas d'utilisation concerne la surveillance des dossiers patients dans le but de repérer des épidémies [START_REF] Meystre | Extracting Information from Textual Documents in the Electronic Health Record: a Review of Recent Research[END_REF]) ou des infections nosocomiales (Velupillai et al. 2015a).

Tous ces projets de recherche n'auraient pas été possibles sans un accès aux données. Dans la mesure où les dossiers patients contiennent des informations personnelles con dentielles, la dé-identi cation des documents qu'ils contiennent est un prérequis à toute recherche. La dé-identi cation automatique des documents cliniques est un domaine de recherche actif qui fait l'objet de nombreux travaux de recherche [START_REF] Grouin | De-Identi cation of Clinical Notes in French: Towards a Protocol for Reference Corpus Developpement[END_REF][START_REF] Neamatullah | Automated De-identi cation of Free-text Medical Records[END_REF]Stubbs et Uzuner 2015 ;[START_REF] Uzuner | Evaluating the State-of-the-Art in Automatic De-identi cation[END_REF].

Établir une liste exhaustive des méthodes en TAL utilisées dans le domaine clinique est une tâche di cile. Nous renvoyons les lecteurs aux di érentes revues de la littérature menées sur le sujet [START_REF] Meystre | Extracting Information from Textual Documents in the Electronic Health Record: a Review of Recent Research[END_REF]Velupillai et al. 2015a ;[START_REF] Wang | Active Computerized Pharmacovigilance Using Natural Language Processing, Statistics, and Elec-References tronic Health Records: A Feasibility Study[END_REF].

Parmi toutes les informations médicales qui présentent un intérêt dans les dossiers patients, la chronologie médicale (Figure D.1) est une des plus importantes. Être en mesure d'extraire ces chronologies permettrait d'acquérir une meilleure compréhension de certains phénomènes cliniques tels que le déroulement des maladies ou l'e et longitudinal des médicaments (C. Lin et al. 2016a ;W. Sun et al. 2013c). De plus, cela permettrait d'améliorer les systèmes de question-réponse et de prédiction de résultats cliniques. Par ailleurs, accéder aux chronologies médicales est nécessaire pour évaluer la qualité du parcours de soins en le comparant aux recommandations o cielles et pour mettre en lumière les étapes du parcours de soins auxquelles une attention particulière doit être fournie.

Dans notre thèse, nous nous concentrons sur la création de ces chronologies médicales en abordant deux questions connexes en TAL : l'extraction d'informations temporelles et la résolution de la coréférence dans des documents cliniques.

D.1 Extraction d'informations temporelles

Intuitivement, la création de chronologies médicales requiert d'extraire les informations temporelles pertinentes dans les documents. L'extraction d'informations temporelles a déjà une longue histoire en TAL et s'est principalement développée dans le domaine journalistique. Les premiers travaux de recherche sur le sujet remontent aux conférences MUC [START_REF] Sundheim | Tipster/MUC-5: Information Extraction System Evaluation[END_REF]. À l'époque, la portée de l'e ort se limitait à extraire les dates et les heures de reportages. Une étape importante a été franchie en 2005 avec la création de TimeML [START_REF] Pustejovsky | The speci cation language TimeML[END_REF], le premier schéma complet pour l'annotation de la temporalité. Ce schéma permet de modéliser avec précision l'information temporelle dans le texte et a in uencé la plupart des travaux de recherche suivants jusqu'à ce jour. Le domaine clinique a commencé à s'intéresser à l'extraction d'informations temporelles au début des années 2010 avec la publication du corpus i2b2 (W. Sun et al. 2013b). La grande majorité des corpus de données temporellement annotés dans les domaines général et clinique contiennent des documents rédigés en anglais. Cependant, plusieurs travaux se sont attelés à la création de ressources dans d'autres langues. Par exemple, [START_REF] Campillos | A French Clinical Corpus with Comprehensive Semantic Annotations: Development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)[END_REF] → neck pain BEGINS-ON July Au cours des cinq dernières années, les approches basées sur des traits pour l'extraction automatique d'informations temporelles ont été progressivement remplacées par des approches neuronales. Cependant, les espoirs initiaux d'une amélioration majeure des performances se sont rapidement évanouis car les approches classiques basées sur les traits demeurent encore compétitives [START_REF] Bethard | SemEval-2015 Task 6: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF][START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF]. Un courant de recherche au sein de la communauté TAL essaye actuellement de combiner les deux types d'approches. Les performances ont été améliorées grâce à l'inclusion de traits catégoriels dans les approches neuronales. Par exemple, [START_REF] Dligach | Neural Temporal Relation Extraction[END_REF] utilisent les étiquettes morphosyntaxiques pour l'extraction des relations temporelles dans les documents cliniques.

D.2 Résolution de la coréférence

Un autre phénomène linguistique à prendre en considération lors de la construction de chronologies médicales est la coréférence. Les documents cliniques contiennent de multiples mentions des mêmes événements en raison du fait que le personnel médical doit faire un suivi de ces événements. Intuitivement, ce phénomène apporte du bruit lors de la création des chronologies médicales et doit être traité de façon appropriée.

Comme pour l'extraction d'informations temporelles, la résolution de la coréférence a été abordée principalement dans le domaine journalistique et les premières évaluations systématiques remontent aux mêmes conférences [START_REF] Hirschman | MUC-7 Coreference Task De nition[END_REF][START_REF] Sundheim | Overview of Results of the MUC-6 Evaluation[END_REF]. Dans le domaine clinique, l'initiative i2b2 est à nouveau responsable de la di usion du premier corpus de données cliniques annoté avec des chaînes de coréférence [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF].

La résolution de la coréférence dans le domaine clinique peut être divisée en deux étapes principales. Premièrement, il faut extraire les portions de textes susceptibles d'êtres coréférentes. Habituellement, ces éléments entrent dans des catégories spéci ques telles que les médicaments ou les procédures médicales. Une fois que ces mentions ont été extraites, il faut regrouper celles qui font référence aux mêmes événements médicaux. Dans l'Exemple 62, toutes les mentions d'événements entre crochets font référence au même événement médical. Les e orts de recherche portant sur la résolution de la coréférence dans le domaine clinique mettent en oeuvre des approches fondées sur des traits [START_REF] Uzuner | Evaluating the State of the Art in Coreference Resolution for Electronic Medical Records[END_REF]. Cependant, les approches neuronales sont devenues de plus en plus populaires dans le domaine général [START_REF] Lu | Event Coreference Resolution: A Survey of Two Decades of Research[END_REF]. De plus, nous notons que les approches hybrides incluant des traits catégoriels dans les réseaux neuronaux ont permis une amélioration signi cative des performances. Par exemple, K. [START_REF] Lee | End-to-end Neural Coreference Resolution[END_REF] ont encodé les informations du locuteur, du genre du texte et d'autres caractéristiques sous la forme de représentations denses dans une approche neuronale pour la résolution de la coréférence.

D.3 Interdépendance des domaines

Comme nous l'avons mentionné plus haut, l'extraction d'informations temporelles et la résolution de la coréférence sont deux sujets qui doivent être abordés lorsque l'on envisage de construire des chronologies médicales. Les mentions d'événements doivent être extraites et placées dans le temps. Simultanément, les mentions doivent être regroupées si elles font référence au même événement.

Au-delà de la nécessité évidente d'aborder les deux tâches simultanément, nous avançons que les deux sujets sont interdépendants. Intuitivement, deux mentions d'événements qui sont coréférentes partagent les mêmes informations temporelles. Cela a des implications lorsque l'on considère que les tâches sont e ectuées dans un ordre spéci que (résolution de la coréférence puis extraction d'informations temporelles). Les informations apportées par la résolution de la coréférence pourraient être utilisées pour accroître la performance d'un système d'extraction d'informations temporelles.

Inversement, deux mentions d'événements qui partagent la même localisation temporelle et la même signi cation ont une forte probabilité d'être coréférentes. L'information temporelle supplémentaire qui serait disponible pour un système de résolution de la coréférence pourrait aider à améliorer la performance du système.

Sur la base de ces observations, il devient clair que l'extraction d'informations temporelles et la résolution de la coréférence doivent être traitées conjointement, non seulement parce qu'elles sont nécessaires pour la construction de chronologies médicales, mais aussi parce qu'il existe une certaine complémentarité entre les deux sujets.

D.4 Questions de recherche

L'extraction d'informations temporelles est un sujet complexe qui nécessite des corpus annotés soigneusement conçus. Ces ressources sont principalement en anglais et les approches développées dans la communauté sont biaisées en faveur de cette langue. Ceci motive notre première question de recherche : est-il possible de concevoir une approche générique pour l'extraction d'informations temporelles qui pourrait être utilisée pour différentes langues ?

Ces corpus contiennent souvent un riche ensemble de traits décrivant les entités annotées. De plus, il existe un grand nombre d'outils de prétraitement de documents cliniques qui permettent d'ajouter plus d'informations sur ces entités. Dans ce contexte, comment cette diversité de traits catégoriels pourrait-elle être utilisée dans les approches neuronales ? Cela soulève une question connexe : comment l'utilisation de ces traits catégoriels influet-elle la performance de ces approches ?

La résolution de la coréférence est un sujet actif dans le domaine clinique. Comme nous l'avons mentionné plus haut, l'extraction d'informations temporelles et la résolution de la coréférence sont liées. Cela nous amène à notre quatrième question de recherche : quel type d'information temporelle pourrait être utile pour la résolution de la coréférence ? Les approches développées pour la résolution de la coréférence sont complexes et il est très di cile de parvenir à améliorer les performances de modèles simples. Dans ce contexte, comment cette information temporelle peut être intégrée dans une approche neuronale pour la résolution de la coréférence ?

D.5 Contributions

Dans la première partie de cette thèse, nous abordons l'extraction d'informations temporelles dans les documents cliniques. Nous présentons quatre contributions sur ce sujet :

• Une approche à base de traits pour l'extraction des conteneurs narratifs dans des documents cliniques. Nous concevons une approche à base de traits pour l'extraction des relations temporelles. Nous testons notre système sur un corpus de documents écrits en anglais dans le contexte de l'édition 2016 de la campagne d'évaluation Clinical TempEval [START_REF] Bethard | SemEval-2016 Task 12: Clinical TempEval[END_REF]).

• Une abstraction de notre approche à base de traits. Nous e ectuons une évaluation empirique sur deux corpus de documents écrits en anglais et en français. Nous montrons qu'une approche similaire peut être utilisée pour les deux langues.

• Une approche neuronale pour l'extraction d'informations temporelles qui intègre des traits catégoriels. Nous abordons l'extraction d'événements, d'expressions temporelles et de relations temporelles dans les documents cliniques. Nous testons notre approche sur un ensemble de documents écrits en anglais dans le contexte de l'édition 2017 de la campagne d'évaluation Clinical TempEval [START_REF] Bethard | SemEval-2017 Task 12: Clinical TempEval[END_REF]).

• Une étude empirique de l'effet des traits sur la performance de notre approche neuronale. Nous utilisons les traits gold-standard disponibles dans le corpus, mais aussi des traits obtenus grâce à l'utilisation d'outils de prétraitement.

La deuxième partie de cette thèse est consacrée à la résolution de la coréférence dans les documents cliniques. Nous présentons deux contributions sur ce sujet :

• Une approche neuronale pour la résolution de la coréférence dans le domaine clinique inspirée des approches récentes dans le domaine général. Nous abordons la résolution de la coréférence à la fois sur les mentions gold-standard et les mentions prédites.

•

2. 1

 1 Publicly available corpora annotated with temporal information based on either the ISO-TimeML or THYME-TimeML speci cations. We compare these resources along four axes: languages, document types, annotation schemes and sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Temporal Histories of Your Medical Event (THYME) corpus descriptive statistics (all documents). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 THYME corpus descriptive statistics including only the documents annotated with narrative container relations. . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 MERLoT corpus descriptive statistics . . . . . . . . . . . . . . . . . . . . . . . 3.4 CONTAINS relations according to sentence window size in the THYME corpus. Window of size 1 corresponds to the intra-sentence level. . . . . . . . . . . . . 3.5 Features used by our classi ers. . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Machine learning algorithms and hyperparameters used for our nal submission to the 2016 edition of the Clinical TempEval shared task (Bethard et al. 2016). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 DCT and CONTAINER model accuracies on the development corpus. . . . . . 3.8 DCT relation extraction subtask: evaluation script output. We report the number of gold standard relations (ref.), the number of predicted relations (pred.), the number of correct predictions (corr.), precision (P), recall (R) and f1-score (F1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 CONTAINS relation extraction subtask: evaluation script output. We report the number of gold standard relations (ref.

  For each corpus part (BETH and PARTNERS) we report the number of coreference chains, and their average and maximum lengths. We also report aggregated numbers. . . . . . . . . . . . . . . . . . . . 98 6.5 Pronoun distribution per chain type. . . . . . . . . . . . . . . . . . . . . . . . 99 6.6 Named Entity Recognition (NER) model performance for event extraction on the development corpus part (20% of the training corpus). We report precision (P), recall (R) and f1-score (F1) for each category and micro-average on all categories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 NER model performance for people extraction on the development corpus part (20% of the training corpus). We report precision (P), recall (R) and f1-score (F1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.8 Mention extraction performance on the test part of the corpus. Metrics are computed using brateval.

  6.13 Singleton distribution across categories. . . . . . . . . . . . . . . . . . . . . . 6.14 Number of tokens, unique tokens and unknown tokens per category. . . . . . . Information Extraction . . . . . . . . . . . . . . . . . . . .

  Resolution . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Topic Interdependence . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.6 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.7 Published Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 1 . 1 :

 11 Figure 1.1: Example of timeline extracted from an electronic patient record.

  Present tense can be subdivided into timeless present (Example 10a), habitual present (Example 10b), instantaneous present (Example 10c), historic present (Example 10d) and present referring to the future (Example 10e). (10) a. Water consists of hydrogen and oxygen. b. They visit their parents every week. c. I advise you to quit. d. Just as John arrived, Mary leaves the house. e. The airplane departs at 9pm tomorrow.

  Aspect. It allows for the distinction between completed (perfective) and ongoing (imperfective or progressive) events. It can be combined with tenses (e.g. Past Perfect or Past Progressive). There is an overlap in meaning between tense and aspect which leads to potential confusion. The most idiomatic example is the selection between the Simple Past and Present Perfective. The rst one indicates that the period of time has ended while the second indicates that the period still continue at the speech time and may continue in the future as shown in Example 12.(12) a. Simple Past: John lived in London for two years.b. Present + Perfective: John has lived in London for two years.

( 15 )

 15 John redecorated his house. He rst painted the walls. Causality is one other mean to express temporal relationship. If one event causes another, thus it is before the other as shown in Example 16. (16) John fell. Mary pushed him.

(

  ) a. His anterior chest rash has not [ ACTUAL reoccurred]. b. We suspect either [ HYP. achalasia] or [ HYP. pseudoachalasia] here. c. The patient may have undergone a mild [ HEDGED stroke] d. I explained that BRAF [ GEN. mutations] have no predictive value with regard to cetuximab [ GEN. sensitivity].

  (27) a. He reports occasional bright red [ INT. bleeding] from the rectum. b. The newest [ NOVEL MRI] revealed a previously undiscovered mass.

(

  29) a. The patient has [ POSITIVE hepatosplenomegaly]. b. No evidence for new suprasellar [ NEGATIVE mass].

(

  31) a. I would probably restart her furosemide [ DATE tomorrow morning]. b. Following the patient's latest seizure, [ TIME 20 minutes ago], we are re-evaluating her medications. c. In [ DURATION the last week], his pain has signi cantly worsened. d. The patient vomited [ QUANTIFIER twice] before the surgery. e. The patient exhibits [ PREPOSTEXP post-exposure] changes. f. Mirtazapine REMERON 7.5-mg tablet 1 tablet by mouth [ SET every bedtime].

  provide a French model). The system obtained competitive results in several shared tasks such as the SemEval 2013 subtask on temporal expression extraction with a f-score of 90.30%. Other rule-based systems include SUTime (A. X. Chang and Manning 2012), a temporal tagger similar to Heideltime. The system is part of the CoreNLP text processing pipeline (Manning et al. 2014). TempEx (Mani and Wilson 2000) was one of the rst time expression tagger. It extracts and normalizes time expressions according to the TIMEX2 speci cation. It served as the base for GUTime, which is an extension of TempEx. It extracts and normalizes time expressions based on the TIMEX3 speci cation. Finally, Chronus (Negri and Marseglia 2005) was developed for the ACE TERN 2004 competition on time expression extraction and normalization. Hybrid Approaches. Several other models implement hybrid approaches. The ATT system (Jung and Stent 2013) used hand-crafted features for the TempEval time expression extraction task. The authors modeled the problem as a sequence labeling task and trained a Conditional Random Field (CRF) model to label each token in the corpus. They obtained a f1-score of 85.60%. Bethard (2013) used a similar approach on the same shared task. He exploited a simple set of features and trained a Support Vector Machine (SVM) instead of a CRF. Clinical Domain. Almost all approaches devised for temporal expression extraction in the clinical domain are hybrid. K. Roberts et al. (2013) used HeidelTime and Llorens et al. (2012) system outputs do devise features that are used in a CRF and a SVM for TIMEX3 span extraction and attribute classi cation in the 2012 i2b2/VA shared task on temporal information extraction. The same approach is implemented in MedTime (Y.-K. Lin et al. 2013) where the authors devised features from Heideltime output and UMLS ® . The system achieved a score of 88.0% in i2b2 2012 temporal relation challenge for the time expression extraction task.

  Figure 3.1: Example of the transformation of a DURING relation into a CONTAINS relation in the MERLoT corpus.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Model Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Entity Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Event Attribute and Document Creation Time Extraction . . . . . . 4.3.3 Containment Relation Extraction . . . . . . . . . . . . . . . . . . 4.3.4 Input Word Embeddings . . . . . . . . . . . . . . . . . . . . . . 4.4 Influence of Categorical Features . . . . . . . . . . . . . . . . . . . . 4.4.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 4.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.5 Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.6 A Word on Temporal Coherence . . . . . . . . . . . . . . . . . . 4.5 Evaluation on the THYME Corpus: Domain Adaptation for Temporal Information Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.2 Architecture Description . . . . . . . . . . . . . . . . . . . . . . 4.5.3 Domain Adaptation Strategies . . . . . . . . . . . . . . . . . . . 4.5.4 Network Training . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Our model is composed of three components: entity extraction (Section 4.3.1), EVENT attribute and DCT relation extraction (Section 4.3.2), and containment relation extraction (Section 4.3.3).

Figure 4 . 2 :

 42 Figure 4.2: Pipeline for entity attribute classi cation.

Figure 4 . 4 :

 44 Figure 4.4: Neural architecture for word embedding creation.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Anaphora and Coreference: A Linguistic Perspective . . . . . . . . . . 5.3 Definitions and Terminology: The NLP imbroglio . . . . . . . . . . . . 5.4 Event Coreference Resolution . . . . . . . . . . . . . . . . . . . . . . 5.5 Annotated Corpora . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 A Word on Mention Extraction . . . . . . . . . . . . . . . . . . . . . . 5.7 Early Approaches for Coreference Resolution . . . . . . . . . . . . . . 5.8 Supervised Approaches for Coreference Resolution . . . . . . . . . . . 5.8.1 Mention-Pair Model . . . . . . . . . . . . . . . . . . . . . . . . 5.8.2 Mention-Ranking Model . . . . . . . . . . . . . . . . . . . . . . 5.8.3 Entity-Based Models . . . . . . . . . . . . . . . . . . . . . . . . 5.8.4 Tree-Based Models . . . . . . . . . . . . . . . . . . . . . . . . . 5.9 Coreference Resolution in the Clinical Domain . . . . . . . . . . . . . 5.10 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10.1 The MUC Score . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10.2 The B 3 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 5.10.3 Constrained Entity-Aligned F-Measure . . . . . . . . . . . . . . . 5.10.4 BLANC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10.5 The CoNLL Score . . . . . . . . . . . . . . . . . . . . . . . . . 5.11 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

( 35 )

 35 Could you bring back [the seat] inside?

( 36 )

 36 a. Kim is making the same mistakes that I [did]. b. Kim brought the wine, and Robin [ ] the cheese. c. Kim arrived home. She [listened to the messages] on her answering machine.

(

  40) a. Italian: Giovanni i e' in ritardo così mi ha chiesto se posso incontrar[lo] al cinema. b. English: John i is late so he i asked me if I can meet [him] at the movies. (41) a. Italian: [Giovanni] andò a far visita a degli amici. Per via, Φ comprò del vino. b. Japanese: [John]-wa yujin-o houmon-sita. Tochu-de Φ wain-o ka-tta.

( 46 )

 46 For centuries here, [the people] have had almost a mystical relationship with Popo, believing the volcano is a god. Tonight, [they] fear it will turn vengeful. (Recasens et al. 2011) (47) On homecoming night [Postville] feels like Hometown, USA, but a look around [this town of 2000] shows it's become a miniature Ellis Island. [This] was an all-white, all-Christian community …For those who prefer [the old Postville], Mayor John Hyman has a simple answer. (Recasens et al. 2011)

( 49 )

 49 John and Mary came to dinner last night. [They] are a nice couple. (Poesio 2016) (50) We believe her, the court does not, and [that] resolves the matter. (Poesio 2016) (51) For the Parks and millions of other young Koreans, the long-cherished dream of home ownership has become a cruel illusion. For [the government], it has become a highly volatile political issue.[START_REF] Poesio | A Corpus-based Investigation of De nite Description Use[END_REF]

( 55 )

 55 Georges Cipriani [ EVT-1 left] a prison in Ensisheim in northern France on parole on Wednesday. He [ EVT-2 departed] the prison in a police vehicle bound for an open prison near Strasbourg.[START_REF] Lu | Event Coreference Resolution: A Survey of Two Decades of Research[END_REF]

( 56 )

 56 She received a 7 day course of [ TREATMENT amoxicillin] for [ PRONOUN which] [ PROBLEM Enterococcus] was sensitive but [ PROBLEM Klebsiella] unknown.

(

  57) a. [ TEST MRI lumbar spine] (Youville Hospital 2020-03-10): [ PROB. L3-4 osteomyelitis], [ PROB. discitis], and [ PROB. epidural abscess]. b. [ PROB. Degenerative changes] including [ PROB. multilevel central spinal stenosis].

( 58 )

 58 She also described [ PROB. sob], [ PROB. nausea], [ PROB. worsening lower ext edema].

  It includes part/whole (Example 59a) and set/subset (Example 59b) relations. (59) a. [ M1 The SFA] has severe stenotic disease. [ M2 The popliteal segment] has moderate disease without stenosis. b. [ M1 Laboratory studies]. [ M2 Mammogram]. [ M3 CT Angiogram].

(

  60) [John] e1 went to [Bob] e2 's house. [He] e1 have [him] e2 back [his] e2 umbrella.

Figure 5 . 1 :

 51 Figure 5.1: Reproduction of Figure 1 from Pradhan et al. (2014). Original caption: "Example key and response entities along with the partitions for computing the MUC score."

Figure 6 . 1 :

 61 Figure 6.1: Concept annotations extracted from the concept le associated to the document clinical-13.

Figure 6 . 2 :

 62 Figure 6.2: Chain annotations extracted from the chain le associated to the document clinical-13.

Figure 6

 6 Figure 6.9: Example of token representation computation.

Figure 6

 6 Figure 6.10: Example of mention representation computation. Mention sentential context is processed with a Bi-LSTM. The resulting dense representation is concatenated to one embedding for the DCT relation, one for the mention type and an attention representation of both left and right sentence contexts.

Figure 6

 6 Figure 6.11: Example of cluster representation computation. The cluster is composed of four mentions: the pain, a burning pain, which and her pain.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 7.2 Future Research Directions . . . . . . . . . . . . . . . . . . . . . . . 124 7.3 Extracting Clinical Timelines: Are We There Yet? . . . . . . . . . . . . 125

Figure C. 2 :

 2 Figure C.2: i2b2 task1c corpus: CoNLL formatted sentence example.

  Fig. D.1 : Exemple de chronologie médicale.

  ont récemment annoté un ensemble de documents cliniques rédigés en français. L'extraction d'informations temporelles dans le domaine clinique peut se décomposer en deux étapes principales. La première concerne l'extraction des mentions d'événements et des expressions temporelles. La seconde concerne l'extraction des relations temporelles. Dans l'Example 61, l'événement clinique (neck pain) et l'expression temporelle (July) sont liés par une relation temporelle (begins-on). Les dé nitions des événements, des expressions temporelles et des relations temporelles varient d'un corpus à l'autre, mais elles sont souvent dérivées du schéma TimeML. À la suite de ce processus en deux étapes, l'information temporelle acquise à partir des documents peut être agrégée pour former une chronologie. (61) She has been experiencing [ EVENT neck pain] since [ DATE July].

( 62 )

 62 The CXR revealed [8 mm obstructing stone] ... [The renal stone] was considered to be the cause of patient's symptoms ... We recommended surgical procedure to remove [ureteropelvic stone] ...

  Une tentative d'élaboration d'un trait temporel dérivé de la relation temporelle qui existe entre les événements et les dates de création des documents. Nous testons ce trait dans le contexte d'une étude empirique qui vise à mesurer comment les traits catégoriels et les modules neuronaux tels que les mécanismes d'attention et les représentations au niveau des caractères in uent sur la performance. En plus des contributions susmentionnées, nous concevons deux ressources. Tout d'abord, nous avons converti le corpus i2b2 task1c au format CoNLL. Cette transformation pourrait favoriser la recherche en TAL sur la résolution de la coréférence en permettant une meilleure reproduction des résultats et en facilitant le traitement des corpus. Deuxièmement, nous avons regroupé notre module neuronal d'étiquetage de séquences dans un outil open-source appelé YASET qui peut être utilisé pour toute tâche d'étiquetage de séquences en TAL.

  ). The best f1-score performance in each phase is bolded. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 6.1 i2b2/VA task1c corpus train and test le counts. . . . . . . . . . . . . . . . . . 94 6.2 i2b2/VA task1c corpus chain count, chain average length (avg. len.) and chain maximum length (max. len.). . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 6.3 i2b2 task1c corpus: brat version statistics. . . . . . . . . . . . . . . . . . . . . 97 6.4 Coreference chain statistics.

  [START_REF] Tourille | Neural Architecture for Temporal Relation Extraction: A Bi-LSTM Approach for Detecting Narrative Containers[END_REF]. "Neural Architecture for Temporal Relation Extraction: A Bi-LSTM Approach for Detecting Narrative Containers". In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics(Vancouver, Canada, July 30, 2017-Aug. 4, 2017).

	4. Associa-tion for Computational Linguistics, pp. 224-230
	5. Julien Tourille, Olivier Ferret, Xavier Tannier, and Aurélie Névéol (Aug. 2017a). "LIMSI-COT at SemEval-2017 Task 12: Neural Architecture for Temporal Information Extrac-tion from Clinical Narratives". In: Proceedings of the 11th International fiorkshop on Semantic Evaluation (Vancouver, Canada, Aug. 3, 2017-Aug. 4, 2017). Association for Computational Linguistics, pp. 597-602
	6. Julien Tourille, Matthieu Doutreligne, Olivier Ferret, Nicolas Paris, Aurélie Névéol, and Xavier Tannier (Oct. 2018). "Evaluation of a Sequence Tagging Tool for Biomedical Texts". In: Proceedings of the 9th International fiorkshop on Health Text Mining and Part I Information Analysis (Brussels, Belgium, Oct. 31, 2018). Association for Computational Linguistics Temporal Information Extraction from Clinical Narratives
	July 4, 2016-July 8, 2016). Association pour le Traitement Automatique des Langues, pp. 459-466
	3. Julien Tourille, Olivier Ferret, Xavier Tannier, and Aurélie Névéol (Apr. 2017c). "Tem-poral Information Extraction from Clinical Text". In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (Valencia, Spain, Apr. 3, 2017-Apr. 7, 2017). Association for Computational Linguistics, pp. 739-745

. "LIMSI-COT at SemEval-2016 Task 12: Temporal Relation Identi cation Using a Pipeline of Classi ers". In: Proceedings of the 10th International fiorkshop on Semantic Evaluation (San Diego, California, USA, June 16, 2016-June 17, 2016). Association for Computational Linguistics, pp. 1136-1142 2. Julien Tourille, Olivier Ferret, Aurélie Névéol, and Xavier Tannier (July 2016a). "Extraction de Relations Temporelles dans des Dossiers Électroniques Patient". In: Actes de la Conférence Traitement Automatique des Langues Naturelles 2016 (Paris, France,

21) [ EVENT Colonoscopy] ( [ TIMEX3 January 7, 2010] ): Fair/adequate [ EVENT prep.],

	Limited [ EVENT Colonoscopy] to the distal sigmoid due to an obstructive [ EVENT le-sion]. Diminutive [ EVENT polyps] of the rectosigmoid.
	a. [ TIMEX3 January 7, 2010] CONTAINS [ EVENT Colonoscopy]
	b. [ EVENT Colonoscopy] CONTAINS [ EVENT prep.]
	c. [ EVENT Colonoscopy] CONTAINS limited [ EVENT colonoscopy]
	d. [ EVENT Colonoscopy] CONTAINS [ EVENT lesion]
	e. [ EVENT Colonoscopy] CONTAINS [ EVENT polyps]
	f. [ EVENT Colonoscopy] CONTAINS [ EVENT removed]

  is a set of documents from the Royal Marsden Hospital. It contains clinical narratives, histopathology reports and imaging reports. The

	Corpora	Languages Document Type	Annotation Scheme	Size (Approx number of documents or tokens)
	TimeBank	EN	news, biographies, event descriptions	ISO-TimeML	183 doc.
	FR-TimeBank	FR	news	Adapted ISO-TimeML	109 doc.
	TimeBankPT BCCWJ-TimeBank Romanian TimeBank RO PT JP	news, biographies, event descriptions news, blogs, books, magazines news, biographies, event descriptions	ISO-TimeML Adapted ISO-TimeML ISO-TimeML	183 doc. (Translation of TimeBank) 1,000,000 tok. 183 doc. (Translation of TimeBank)
	Ita-TimeBank	IT	news	Adapted ISO-TimeML	150,000 tok.
	AQUAINT	EN	news	ISO-TimeML	73 doc.
	TempEval-1	EN	news	Modi ed ISO-TimeML	183 doc. (same as TimeBank)
	TempEval-2
	TempEval-3	EN, SP	news	Modi ed ISO-TimeML	768,075 tok. (EN); 67,819 tok. (SP)
	CLEF	EN	clinical narratives, histopathology reports, imaging reports	Custom	50 doc.
	Galescu & Blaylock	EN	discharge summaries	modi ed ISO-TimeML	7,701 tok.
	i2b2	EN	discharge summaries	THYME-TimeML beta version 310 doc.
	THYME MERLoT	EN FR	clinical reports pathology reports discharge summaries, physician letters, prescriptions procedure reports,	THYME-TimeML modi ed ISO-TimeML	1,200 doc. 500 doc.

CN, EN, IT, FR, KO, SP news Modi ed ISO-TimeML 23,000 tok. (CN); 63,000 tok. (EN); 27,000 tok. (IT); 19,000 tok. (FR); 14,000 tok. (KO); 68,000 tok. (SP)

Chapter 3 Feature-Based Approach for Temporal Relation Extraction: Application to French and English Corpora 3.1 Introduction

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.3 Model Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

42 3.4 Evaluation on the THYME corpus

  . . . . . . . . . . . . . . . . . . . .

44 3.4.1 Preprocessing and Feature Extraction . . . . . . . . . . . . . . . 45 3.4.2 Algorithm Selection . . . . . . . . . . . . . . . . . . . . . . . . 45 3.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.

5 Adapting the Approach to French Clinical Text

. . . . . . . . . . . . . 49 3.5.1 Preprocessing and Feature Extraction . . . . . . . . . . . . . . . 50 3.5.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 50 3.5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

51 3.6 Conclusion

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

  .2.

			Colon		Brain
		Train	Dev	Test 30-doc.
	# of documents	195	98	100	20
	DOCTIME EVENT SECTIONTIME TIMEX3	219 31,902 17,245 15,503 115 114 278 122 153 3,756 2,035 1,917	5 2,124 4 306
	CONTAINS	11,248	6,226	5,930	788
	Table 3.2: THYME corpus descriptive statistics including only the documents annotated with narrative container relations.
	# of documents		500
	EVENT TIMEX3		18,127 3,940
	CONTAINS		4,295
	Table 3.3: MERLoT corpus descriptive statistics
			Colon		Brain
		Train	Dev	Test 30-doc
	# of documents	293	147	151	30
	DOCTIME EVENT SECTIONTIME TIMEX3	322 38,937 20,974 18,990 168 168 284 123 154 3,833 2,078 1,952	7 2,557 8 350
	CONTAINS	11,248	6,226	5,930	788
	Table 3.1: THYME corpus descriptive statistics (all documents).

Table 3

 3

	.4: CONTAINS relations according to sentence window size in the THYME corpus. Window of size 1 corresponds to the intra-sentence level.

  Table 3.7. For the DCT model, we obtain high performance with plain

				CONTAINS
	Feature	DCT CONTAINER c Intra Inter
	Entity type Entity form Entity attributes EVENT Semantic Types and Semantic Groups b Entity Lemmas Entity POS and CPOS tags Do the entities contain other entities? Do the entities are contained by other entities? Container model output for the two considered entities Syntactic paths between the two considered entities a	✓ ✓ ✓ ✓ ✓	✓ ✓ ✓ ✓ ✓ ✓	✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓	✓ ✓ ✓ ✓ ✓ ✓
	Sentence entity forms Sentence entity types Sentence entity attributes Sentence EVENT Semantic Types and Semantic Groups Sentence entity POS and CPOS tags Sentence entity lemmas Sentence token lemmas Sentence token POS and CPOS tags	✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓	✓ ✓ ✓ ✓ ✓ ✓ ✓	✓ ✓ ✓
	Section entity forms Section entity types Section entity lemmas Section EVENT Semantic Types and Semantic Groups Section entity POS and CPOS tags Section entity attributes	✓ ✓ ✓ ✓ ✓ ✓
	Center context entity types Center context entity attributes Center context EVENT Semantic Types and Semantic Groups Center context entity container model outputs				✓ ✓ ✓ ✓
	Sentence position in section Sentence position in document	✓ ✓			✓
	Number of containers at the sentence level Number of entities between the considered entities Number of tokens between the entities Number of entities before and after at the sentence level Number of entities before and after at the section level Number of entities before and after at the document level	✓ ✓	✓	✓ ✓ ✓	✓

Table 3 .

 3 5: Features used by our classi ers.

	Strategy	Classi er	Algorithm	Parameters	% feat. a
	Plain text	CONTAINER c CONTAINS INTRA SVM (RBF) SVM (RBF) CONTAINS INTER SVM (RBF) DCT SVM (Linear)	C=10, gamma=0.01 C=10, gamma=0.01 C=1000, gamma=0.01 C=1, tol b =0.0001, normalization=l2, loss function=hinge	60 60 100 100
	Word embeddings	CONTAINER c CONTAINS INTRA SVM (Linear) LinearSVM CONTAINS INTER SVM (Linear) DCT Random Forests max features=auto, criterion=entropy, estimators=100 C=1, tol b =0.01, normalization=l2, loss function=hinge C=1, tol b =0.01, normalization=l2, loss function=squared hinge C=1000, tol b =0.01, normalization=l2, loss function=hinge	100 100 100 100

Table 3 .

 3 7: DCT and CONTAINER model accuracies on the development corpus.

	Strategy	ref.	pred.	corr.	P	R	F1
	Plain text Word embeddings 18,990 18,989 15,317 0.807 18,990 18,989 14,603 0.769	0.769 0.807	0.769 0.807

Table 3

 3

	Strategy	ref.	pred.	corr.	P	R	F1
	Plain text	5,894	3,755	2,642 2,570	0.704	0.436	0.538
	Word embeddings 5,894	2,544	1,911 1,889	0.751	0.320	0.449

.8: DCT relation extraction subtask: evaluation script output. We report the number of gold standard relations (ref.

), the number of predicted relations (pred.), the number of correct predictions (corr.), precision (P), recall (R) and f1-score (F1).

Table 3

 3

.9: CONTAINS relation extraction subtask: evaluation script output. We report the number of gold standard relations (ref.

  .10.

		THYME (en) MERLoT (fr)
	Before Before-Overlap Overlap After	29,170 4,240 37,091 8,400	1,936 2,643 12,211 1,337
	Table 3.10: MERLOT (fr) and THYME (en) corpora: DCT relation distribution.

DCT CONTAINER c CONTAINS

	Entity type Entity form Entity attributes Entity position (within the document) Container model output Document Type a Contextual entity forms Contextual entity types Contextual entity attributes Container model output for contextual entities POS tag of the sentence verbs Contextual token forms (unigrams) Contextual token POS tags (unigrams) Contextual token forms (bigrams) b Contextual token POS tags (bigrams) b	✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓	✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓	✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

a Information available only for the MERLoT corpus. b Only when using plain lexical forms. c Classi er that predicts whether an entity is the source of one or more CONTAINS relations.

Table 3 .

 3 11: Features used by our classi ers.

  Typically, the performance is lower for the categories where we have a lower number of training examples (before-overlap for the THYME corpus and categories other than overlap for the MERLoT corpus).Concerning the CONTAINS relation extraction task, results are separated by a 10 point gap (0.65 for the MERLoT corpus and 0.53 for the THYME corpus). Results obtained for the THYME corpus are coherent with those presented in Section 3.4 on the Clinical TempE-

		DCT	CONTAINER	CONTAINS	CONTAINS w/o CONTAINER
	Corpus	Plain	W2V	Plain	W2V	Plain	W2V	Plain	W2V
	MERLOT (fr)	0.830 (0.008)	0.785 (0.006)	0.837 (0.004)	0.776 (0.014)	0.827 (0.007)	0.799 (0.012)	0.724 (0.011)	0.670 (0.016)
	THYME (en)	0.868 (0.002)	0.797 (0.006)	0.760 (0.007)	0.678 (0.031)	0.751 (0.003)	0.702 (0.013)	0.589 (0.006)	0.468 (0.018)

Table 3 .

 3 12: Cross-validation results over the training corpus for all tasks. We report f1-score for CONTAINER and CONTAINS tasks and accuracy for DCT task. We also report standard deviation for all models (in brackets).

		MERLOT (fr)	THYME (en)
		P	R	F1	P	R	F1
	baseline	0.67 0.67 0.67 0.47 0.47 0.47
	before-overlap before after overlap	0.68 0.69 0.69 0.73 0.60 0.66 0.81 0.60 0.69 0.88 0.88 0.88 0.79 0.69 0.73 0.84 0.84 0.84 0.88 0.92 0.90 0.88 0.90 0.89
	micro-average	0.83 0.84 0.83 0.87 0.87 0.87
	Table 3.13: DR task results over the test corpus. We report precision (P), recall (R) and f1-score (F1) for all relation types.
		MERLOT (fr)	THYME (en)
		P	R	F1	P	R	F1
	baseline	0.43 0.15 0.22 0.55 0.06 0.11
	no-relation contains	0.99 1.00 0.99 0.96 0.98 0.97 0.75 0.57 0.65 0.61 0.47 0.53
	micro-average	0.98 0.98 0.98 0.93 0.94 0.93
	Table 3.14: CR task results over the test corpus. We report precision (P), recall (R) and f1-score (F1) for all relation types.

Table 4 .

 4 Table4.2. For comparison, we report the baseline provided as reference during the Clinical TempEval shared tasks, the results of the best system of the Clinical TempEval 2016 challenge (H.-J.[START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF]) and the best scores obtained after the challenge (C.[START_REF] Lin | Improving Temporal Relation Extraction with Training Instance Augmentation[END_REF]) on the test portion of the corpus. Both H.-J.[START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF] and C.Lin et al. (2016) rely on SVM classi ers using hand-engineered linguistic features.

	P	R	F1

2: Experimentation results. We report precision (P), recall (R) and f1-score (F1) for each con guration of our model, for the best system of the Clinical TempEval 2016 challenge (H.-J.

[START_REF] Lee | UTHealth at SemEval-2016 Task 12: an End-to-End System for Temporal Information Extraction from Clinical Notes[END_REF]

) and for the best result obtained so far on the corpus (C.

[START_REF] Lin | Improving Temporal Relation Extraction with Training Instance Augmentation[END_REF]

).

Table 4

 4

	-sentence classi er

.3: Results obtained by the intra-sentence and inter-sentence classi ers for each model of this paper.

. Chapter 6 Neural Entity-Based Approach for Coreference Resolution in the Clinical Domain 6.1 Introduction

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 6.3 Task Division . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 6.4 Mention Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 6.5 Building a Temporal Feature . . . . . . . . . . . . . . . . . . . . . . .

102 6.6 Neural Entity-Based Approach for Coreference Resolution

  . . . . . . . 106 6.6.1 Input Embeddings . . . . . . . . . . . . . . . . . . . . . . . . . 107 6.6.2 Mention Representation . . . . . . . . . . . . . . . . . . . . . . 107 6.6.3 Cluster-Level Representation . . . . . . . . . . . . . . . . . . . . 108 6.6.4 Pairwise Scorer . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.6.5 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 6.6.6 Wrap-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 6.7 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 6.7.1 Experiment Con gurations . . . . . . . . . . . . . . . . . . . . . 112 6.7.2 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . 113 6.8 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

113 6.9 Discussion

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

114 6.10 Conclusion

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

  .2.

	Institution	Train Test Total
	BETH PARTNERS		115 136	79 94	194 230
	Combined		251 173	424
	Table 6.1: i2b2/VA task1c corpus train and test le counts.
	Institution	Chain count	Chain avg. len.	Chain max. len.
	BETH PARTNERS	1,816 1,395		4.155 4.352	122 105
	Table 6.2: i2b2/VA task1c corpus chain count, chain av-erage length (avg. len.) and chain maximum length (max. len.).

  as well as the number of pairwise coreference links (coref_person, coref_pronoun, coref_test and coref_treatment). Chain statistics are discussed in Section 6.3. Overall, corpus size is much smaller than the OntoNotes corpus in the general domain which includes 2,083 documents annotated with 131,846 mentions and 97,556 coreference links. An example of a sentence transformed to both formats (brat and CoNLL) is presented in Appendix C in Figures C.1 and C.2. The description of the CoNLL le columns is presented in Table C.1.

Table 6 .

 6 4: Coreference chain statistics. For each corpus part (BETH and PARTNERS) we report the number of coreference chains, and their average and maximum lengths. We also report aggregated numbers.

		TRAIN			TEST			ALL
	Chain count	Chain avg. len.	Chain max. len.	Chain count	Chain avg. len.	Chain max. len.	Chain count	Chain avg. len.	Chain max. len.

2.4 Resources for Temporal Information ExtractionIn this section, we present resources that have been developed in computational linguistics for temporal information extraction. First, we review the two most in uential annotation

Parameters used during computation (Section 3.4): algorithm=CBOW; min-count=5; vector-size=200; window=20.

Parameters used during computation (Section

3.5): algorithm=CBOW; min-count=5; vector-size=200; window=10.

Similarly to our evaluation corpus for English, the Clinical TempEval 2016 evaluation corpus was extracted from the THYME corpus but the two corpora are di erent.

https://github.com/hyperopt/hyperopt

https://github.com/bethard/anaforatools

Replacement probability = 0.2.

https://github.com/conll/reference-coreference-scorers

https://github.com/jtourille/i2b2-coreference-evaluation

Although pronouns always refers to something in the text, this something may not fall in the annotated entity categories.

algorithm=skip-gram, vector size=100, window=8

https://github.com/jtourille/yaset

Remerciements

We report precision (P), recall (R) and f1-score (F1).

The NER model for people extraction obtains a high performance with a f1-score of 92.41. Model performance is evenly distributed between precision (93.69) and recall (91.16).

We report the nal performance on the test part of the corpus in Table 6.8. The scores are computed with brateval and after that all steps described above have been applied (supervised learning and application of regular expressions for people and pronouns). As expected, pronoun extraction obtains a very high f1-score (0.9647) with very little false negatives (16). Scores for events are similar to the ones obtained on the development corpus (from 0.8443 for problems to 0.8771 for tests). The f1-score for people is very high (0.9403). Overall, our mention extraction method reaches a balanced f1-score of 0.8803.

Coreference Resolution Evaluation. To measure the impact of mention extraction errors on the coreference score, we project the gold coreference chains on the predicted mentions and perform coreference resolution evaluation with the o cial CoNLL scorer (Table 6.9). The ceiling CoNLL f1-score that can be obtained with the predicted mentions is 91.84. Most of the performance decrease is imputable to a drop in recall for all three metrics that compose the CoNLL score. simpli ed version of the task by selecting active mentions that are known to be anaphoric. The task is then to assign the correct antecedent to the active mention.

Finally, we implement antecedent ltering. For people and events, depending on the active mention type, we consider antecedents that are of the same type or of the type pronoun. For pronouns, all antecedents are considered.

Hyperparameters

The model was built using Pytorch [START_REF] Paszke | Automatic Di erentiation in PyTorch[END_REF]). Contextual and cluster-level LSTM hidden layers sizes are xed to 100. We x the hidden layer size of the character Bi-LSTM and the one of the character embeddings to 25. Feature embedding size is set to 50.

Network is trained end-to-end using mini-batches of 1 document. We use Adam as optimization algorithm with an initial learning rate of 0.001. We apply learning rate decay of 1% at every iteration. We apply dropout on pairwise scorer hidden layers and on feature embeddings with a rate of 0.2. Dropout is also applied on input embeddings with a rate of 0.5.

We implement the learning of LSTM initial hidden states [START_REF] Gers | Learning Precise Timing with LSTM Recurrent Networks[END_REF]. Compared to initializing them randomly or keeping them at zero, this approach allowed us to increase signi cantly the performance of our model during development.

Results

Performance on Gold Mentions. Performance of all experiments on gold mentions are presented in Table 6.11. We report the score distributions over 10 runs in Figure 6.14. Surprisingly, our cluster-level representation degrades the performance of our system. For both models (people and events), the average CoNLL f1-score is lower for the con guration that includes the cluster-level representation. Also, the scores indicate that using a clusterlevel representation seems to increase precision and decrease recall. Intuitively, this additional information allows to better match mentions within clusters and therefore increase precision.

We observe that there is an overlap between the two distributions (baseline vs. cluster-rpr). It means that comparing two extreme values of these models could have resulted in an opposite conclusion. This con rms the need for distribution comparisons instead of single-shot comparisons which has already been mentioned in previous e orts (Reimers and Gurevych 2017).

Concerning the other experiments, only 3 out of 7 con gurations did perform better than our baseline approach. Network pretraining is the component that brings the largest f1-score increase. The event model bene ts the most from this pretraining step with an increase of +2.69 F1 while the people model performance is only improved by +0.43 F1.

The attention mechanism brings an increase of +0.39 F1, balanced between people and event models. Our ltering strategy does improve the performance by only +0.01 F1.

Concerning the components that degrade performance, the use of character embeddings decreases the overall f1-score of both people and event models. However, the nal decrease when models are combined is limited at -0.06 F1.

Freksa, Christian (1992). "Temporal reasoning based on semi-intervals". In: