Pedro Javier Ortiz Suárez

Yoann Dupont

Gaël Lejeune

Tian Tian

SinNer@Clef-Hipe2020 : Sinful adaptation of SotA models for Named Entity Recognition in French and German

Keywords: Named Entity Recognition, Historical Texts, German, French, ELMo, CRFs., Sentence Segmentation

In this article we present the approaches developed by the Sorbonne-INRIA for NER (SinNer) team for the CLEF-HIPE 2020 challenge on Named Entity Processing on old newspapers. The challenge proposed various tasks for three languages, among them we focused on Named Entity Recognition in French and German texts. The best system we proposed ranked third for these two languages, it uses FastText embeddings and Elmo language models (FrELMo and German ELMo). We show that combining several word representations enhances the quality of the results for all NE types and that the segmentation in sentences has an important impact on the results.

Introduction

Among the aspects for which Natural Language Processing (NLP) can be useful for Digital Humanities (DH) figures prominently Named Entity Recognition. This tasks interests researchers for numerous reasons since the application can be pretty wide. Among other usages we can cite genealogy or history for which finding mention of persons and places in texts is very useful. Researchers in digital literature have shown a great interest in Named Entities since it can help for instance to highlight the path of different characters in a book or in a book series. There can be cross-fertilization between NER and DH since some researchers showed that some particular properties of literature can help to build better NER systems [START_REF] Brooke | Bootstrapped text-level named entity recognition for literature[END_REF]. Apart from literature, NER can also be used more generally to help refine queries to assist browsing in newspaper collections [START_REF] Neudecker | Large-scale refinement of digital historic newspapers with named entity recognition[END_REF]. Like other NLP tasks, NER quality will suffer from different problems related to variations in the input data : variation in languages (multilinguality), variation in the quality of the input data (OCR errors mainly) and specificity of the application domain (literature VS epidemic surveillance for instance). These difficulties can be connected with the challenges for low level NLP tasks highlighted by Dale et al. [START_REF] Dale | Handbook of Natural Language Processing[END_REF]. In this CLEF-HIPE challenge [START_REF] Ehrmann | Extended Overview of CLEF HIPE 2020: Named Entity Processing on Historical Newspapers[END_REF], the variation in language and in text quality will be the main problems even if the specificity of the application can be of great interest.

NER in old documents represent an interesting challenge for NLP since it is usually necessary to process documents that show different kind of variations as compared to the particular laboratory conditions on which NER systems are trained. Most NER systems are usually designed to process clean data. On the other hand, there is the multilingual issue since NER systems have been designed primarily for English, with assumptions on the availability of data on one hand and on the universal nature of some linguistic properties on the other hand.

The fact that the texts processed in Digital Humanities are usually not borndigital is very important since, even after OCR post-correction, it is very likely that some noise would be found in the text. Other difficulties will arise as well in those type of documents. The variation in language is one of them since contemporary English will clearly not be the most frequent language. It is interesting for researchers to check how much diachronic variation has an influence on NER systems [START_REF] Ehrmann | Diachronic evaluation of ner systems on old newspapers[END_REF]. It makes it even more important to work on multilingual NER and to build architecture that need less training data [START_REF] Rahimi | Massively multilingual transfer for NER[END_REF]. More generally, NER in ancient texts represents a great opportunity for NLP to compare to main approaches to handle variation in texts : adapting the texts to an existing architecture via modernization or normalization [START_REF] Leaman | TaggerOne: joint named entity recognition and normalization with semi-Markov Models[END_REF] or adapting the pipeline to non standard data (OCR noise, language variants . . . ) via domain adaptation or data augmentation techniques [START_REF] Ghannay | End-to-end named entity extraction from speech[END_REF].

In Section 2 we will present a brief state-of-the-art for Named Entity Recognition with a focus on digitized documents. Section 3 and 4 will respectively devoted to the description of the dataset of CLEF-HIPE 2020 shared task [START_REF] Ehrmann | Extended Overview of CLEF HIPE 2020: Named Entity Processing on Historical Newspapers[END_REF] and the methods we developed to extract NE for French and German. The results of our systems will be described in Section 5 and in Section 6 we will give some conclusions and perspectives for this work.

State of the Art for Named Entity Recognition

Named Entity Recognition came into light as a prerequisite for designing robust Information Extraction (IE) systems in the MUC conferences [START_REF] Grishman | Design of the muc-6 evaluation[END_REF]. This task soon began to be treated independently from IE since it can serve multiple purposes, like Information retrieval or Media Monitoring for instance [START_REF] Yangarber | Unsupervised learning of generalized names[END_REF]. As such, shared task specifically dedicated to NER started to rise like CONLL [START_REF] Kim | Introduction to the conll-2003 shared task: Language-independent named entity recognition[END_REF]. Two main path were followed by the community: (I) since NER was at first used for general purposes, domain extension start to gain interest [START_REF] Evans | A framework for named entity recognition in the open domain[END_REF]; (II) since the majority of NER systems were designed for English, the extension to novel languages (including low resource languages) became of importance [START_REF] Rössler | Adapting an NER-system for German to the biomedical domain[END_REF].

Regarding approaches, one can say that NER followed the different fashions in NLP. The first approaches were based on dictionaries, gazeeters and handcrafted rules. Initially NER was considered to be solved by a patient process involving careful syntactic analysis [START_REF] Hobbs | The generic information extraction system[END_REF]. Supervised learning approaches came to fashion with the increase of available data and specifically the rise of shared tasks on the subject. Decision trees and Markov models were soon outperformed with Condition Random Fields. By taking advantage of the sequentiality of textual data, CRF helped to set new state-of-the-art results in the domain [START_REF] Finkel | Incorporating non-local information into information extraction systems by gibbs sampling[END_REF]. Since supervised learning results were bound by the size of training data, lighter approaches were tested in the beginning of the 2000's, we can cite weakly supervision [START_REF] Yangarber | Counter-training in discovery of semantic patterns[END_REF] and active learning [START_REF] Shen | Multi-criteria-based active learning for named entity recognition[END_REF].

During a time, most of promising approaches involved an addition to improve CRfs: embeddings [START_REF] Passos | Lexicon infused phrase embeddings for named entity resolution[END_REF], (bi-)LSTMs [START_REF] Lample | Neural architectures for named entity recognition[END_REF] [15] or contextual embeddings [START_REF] Peters | Deep contextualized word representations[END_REF]. More recently, the improvements in contextual word embeddings made the CRFs models disappear of the architecture reaching state-of-the-art results, see [START_REF] Stanislawek | Named entity recognition -is there a glass ceiling?[END_REF] for a review on the subject and a very interesting discussion on the limits attained by the state-of-the-art systems.

Dataset for the CLEF-HIPE shared task

The provided corpus in French and German both contain training data (train) and development (dev) data whereas, for English only development data was provided. We chose to work only on French and german The table 1 shows some statistics of this dataset. The size of the train dataset was two time higher for French than for German wheras the development sets had roughly the same size. As usual in NER, persons (Pers) and locations (Loc) are the most frequent entity types. The dataset is represented in CONLL format with one token per line. Table 2 shows an extirp of the train dataset in French. For each document, general information were provided. Newspaper and date may have been features useful for recognising entities but we dit not take advantage of it. Each document was composed of segments, starting with "# segment . . . " corresponding to lines in the original documents. Each segment was tokenized to correpsond to the CONLL format with one token per line. These two notions, segments and tokens, are very important since they do not always match the type of unit usually processed in NLP. Segments seldom correspond to sentences so that there is a need to concatenate the segments to get the raw text and then segment it into sentences. This is very interesting since it gets us close to real-world conditions rather than laboratory conditions, and we show in Section 5.2 that this segment VS sentence question has an important influence on the results. Regarding tokens, the tokenization is obviously not perfect. We can see that there are non-standard words and bad tokenization due to the OCR output (in red in Table 2). If we concataenate the tokens we get the sequence "Su. _sss allemands" instead of "Suisse allemande". These non-standard words make the Named Entity Recognition task more complicated and, again, more realistic. a "fill-in-the-gaps" gazetteers feature where tokens not found in any gazetteer are replaced by their POS, as described in [START_REF] Raymond | Reconnaissance robuste d'entités nommées sur de la parole transcrite automatiquement[END_REF]. This features used token unigrams and token bigrams in a [-2, 2] a window. tag unigrams and bigrams.

O O O _ _ NoSpaceAfter Nous O O O O O O _ _ _ serons O O O O O O _ _ _ heureux O O O O O O _ _ _ de O O O O O O _ _ _ publier O O O O O O _ _ _ de O O O O O O _ _ _ temps O O O O O O _ _ _ à O O O O O O _ _ EndOfLine
We trained a CLEF HIPE specific model by optimizing L1 and L2 penalties on the development set. The metric used to estimate convergence of the model is the error on the development set (1 -accuracy). For French, our optimal L1 and L2 penalties were 0.5 and 0.0001 respectively (default Wapiti parameters). For German, our optimal L1 and L2 penalties were 1.0 and 0.0001 respectively.

One interest of SEM is that it has a builtin sentence tokenizer for french using a rule-based approach. By default, CLEF-HIPE documents segments are only newlines. As a result, some NE mentions span across multiple segments, making it very hard to identify them correctly. It is to be expected that models trained (and labelling on) sentences would yield better performances than those trained (and labelling on) segments. SEM makes it simple to switch between different sequence segmentations, which allowed us to label sentences and output segments. SEM's sentence segmentation engine works using mainly local rules at token level rule to determine whether a token is the last of a sequence. It also uses non-local rules to remember whether a token is between parentheses or french quotes to not segment automatically within them. Since we work at token level, we had to adapt some rules to fit CLEF-HIPE tokenization. For example, SEM decides at tokenization stage whether a dot is a strong punctuation or part of a larger token, as for abreviations. This has the advantage of making sentence segmentation easier. CLEF-HIPE tokenization systematically separates dots, so we adapted some tokenization rules to apply them at sentence segmentation stage. We decided to not consider a dot as a sentence terminator if the previous token was in a lexica of titles or functions.

Another interest is that SEM has an NE mention broadcasting process. Mentions found at least once in a document are used as a gazetteer to tag unlabeled mentions within said document. When a new mention overlaps and is strictly longer than an already found mention, the new mention will replace the previous one in the document.

Contextualized word embeddings

Embeddings from Language Models (ELMo) [START_REF] Peters | Deep contextualized word representations[END_REF] is a Language Model, i.e, a model that given a sequence of N tokens, (t 1 , t 2 , ..., t N ), computes the probability of the sequence by modeling the probability of token t k given the history (t 1 , ..., t k-1 ):

p(t 1 , t 2 , . . . , t N ) = N k=1 p(t k | t 1 , t 2 , . . . , t k-1 ).
However, ELMo in particular uses a bidirectional language model (biLM) consisting of L LSTM layers, that is, it combines both a forward and a backward language model jointly maximizing the log likelihood of the forward and backward directions; ELMo also computes a context-independent token representation via token embeddings or via a CNN over characters.

When included in a downstream model, as it is the case in this paper, ELMo collapses all L layers into a single vector, generally computing a task specific weighting of all biLM layers applying layer normalization to each biLM layer before weighting. Following [START_REF] Peters | Deep contextualized word representations[END_REF], we use in this paper ELMo models where L = 2, i.e., the ELMo architecture involves a character-level CNN layer followed by a 2-layer biLSTM.

ELMo-LSTM-CRF

The LSTM-CRF is a model originally proposed by Lample et al. [START_REF] Lample | Neural architectures for named entity recognition[END_REF] is a BiLSTM pre-appended by both character level word embeddings and pre-trained word embeddings and pos-appended by CRF decoder layer. For our experiments, we follow the same approach as Ortiz Suárez et al. [START_REF] Ortiz Suárez | Establishing a new state-of-the-art for French named entity recognition[END_REF] by using the Bi-LSTM-CRF implementation of [START_REF] Straková | Neural architectures for nested NER through linearization[END_REF] which is open source and readily available 6 and is designed to easily pre-append contextualized word-embeddings to the model.

Historically English had receivend the most attention in NER, with some recent developments in German, Dutch and Spanish by [START_REF] Straková | Neural architectures for nested NER through linearization[END_REF] And additional term of comparison was identified in a recently released stateof-the-art language model for French, CamemBERT [START_REF] Martin | CamemBERT: a tasty French language model[END_REF], based on the RoBERTa architecture pre-trained on the French sub-corpus of the newly available multilingual corpus OSCAR [START_REF] Ortiz Suárez | A monolingual approach to contextualized word embeddings for mid-resource languages[END_REF].

Results and Discussion

Official shared-task results

Results of our 3 runs compared to the best run on the NERC-coarse sharedtask for french and german are given in table 3. For each of those tasks, we are the third best ranking team. We only did very minimal adaptation of existing systems. The most notable one was to use custom sentence segmentation instead of given segments for french and using some language-specific lexica for our CRF model in german. Other than that, we only optimized hyper-parameters on the dev set. This clearly illustrates the power of contextual embeddings and today's neural network architectures. This simple adaptation to the task allowed us to be among the three best ranking teams. This is encouraging in terms of usability of SotA models on real-world data.

Study of sequence segmentation

As can be seen in table 4, sentence segmentation allows to improve results by 3.5 F1 points. This is due to the fact that some entities were split across multiple segments in the original data. Using a custom sentence segmentation allows to 3. Strict results for our systems compared to the winning system have entities in a single sequence. This benefits the models both at training and at label level, where systems can access a more proper context. The cost of using another segmentation is relatively cheap, as SEM can process nearly 1GB of raw text per hour.

A per entity comparison is also available in table 4. One can see that the improvement of sentence segmentation is not very significant for locations (Loc). It is due to two facts : (I) locations are usually small in number of tokens and therefore less prone to be sperated in two segments and (II) there was less room from improvement since they were the easiest entity type to detect (86.35% F1-measure). To the contrary, products (Prod), usually longer in tokens, were very hard to predict with only 48.57% F1-measure and benefited the most from segmentation in sentences (+16 percentage points in F1-measure).

Conclusion

In this article we presented three methods developed for the Named Entity Recognition task in French and German ancient texts. The first method relied on linear-chain CRFs while the other two methods use a Bidirectional LSTM and a bidirectional Language Model (ELMo). The later outperformed the CRF model and achieved rank 3 on the NER task in both French and German. We also showed that the type of sequences used has a significant influence on the results. When we segment in sentences rather than using the segments of the dataset as it is the results are systematically much better, with an exception for locations where the gain is marginal. This proves that sentence segmentation remains a key component of efficient NLP architectures, in particular for models taking advantage of the context.

As a future work it would be interesting to assess the importance of noise in the data. For instance, by comparing the results of NER on texts obtained via different OCR tools. The influence of the qualitative jumps in the data, which is common in Digital Humanities, is an important aspect to evaluate the robustness of the system in real-world conditions rather than laboratory conditions.

Table 1 .

 1 Statistics of training and dev data in French and German

		Tokens Documents Segments	Labeled named entities
					Pers Loc Org Time Prod
	Train Fr 166217	158	19183 3067 2513 833	273	198
	Dev Fr	37592	43	4423	771 677 158	69	48
	Train De 86960	104	10353 1747 1170 358	118	112
	Dev De	36175	40	4186	664 428 172	73	53

Table 2 .

 2 Example extracted from French training dataset 4 CRFs and Contextualized Word Embeddings for NER

	4.1 CRF model
	SEM 4 [3] is a free NLP tool that relies on linear-chain CRFs [11] to perform
	tagging. SEM uses Wapiti [13] v1.5.0 5 as linear-chain CRFs implementation.
	SEM uses the following features for NER:
	-token, prefix/suffix from 1 to 5 and a Boolean isDigit features in a [-2, 2]
	window;
	-previous/next common noun in sentence;
	-10 gazetteers (including NE lists and trigger words for NEs) applied with
	some priority rules in a [-2, 2] window;

Table 4 .

 4 Detailed comparison between segments and sentences on dev dataset

	Entity	P segments sentences	R segments sentences	F1 segments sentences
	Loc	85.21 87.73 (+2.52)	87.52 87.08 (-0.44)	86.35 87.41 (+1.06)
	Org	70.62 71.33 (+0.71)	62.78 65.64 (+2.86)	66.47 68.37 (+1.90)
	Pers	80.24 84.64 (+4.40)	76.88 82.09 (+5.21)	78.52 83.35 (+4.83)
	Prod	62.96 75.86 (+12.90) 39.53 56.41 (+16.88) 48.57 64.71 (+16.14)
	Time	86.21 90.91 (+4.70)	78.12 87.72 (+9.60)	81.97 89.29 (+7.32)
	Global	81.03 84.46 (+3.43)	81.61 84.46 (+2.85)	79.52 83.01 (+3.49)

available at : https://github.com/YoannDupont/SEM

available at : https://github.com/Jekub/Wapiti

Available at: https://github.com/ufal/acl2019_nested_ner.