Rosa Stern

Benoît Sagot

Frédéric Béchet

A Joint Named Entity Recognition and Entity Linking System

We present a joint system for named entity recognition (NER) and entity linking (EL), allowing for named entities mentions extracted from textual data to be matched to uniquely identifiable entities. Our approach relies on combined NER modules which transfer the disambiguation step to the EL component, where referential knowledge about entities can be used to select a correct entity reading. Hybridation is a main feature of our system, as we have performed experiments combining two types of NER, based respectively on symbolic and statistical techniques. Furthermore, the statistical EL module relies on entity knowledge acquired over a large news corpus using a simple rule-base disambiguation tool. An implementation of our system is described, along with experiments and evaluation results on French news wires. Linking accuracy reaches up to 87%, and the NER fmeasure up to 83%.

Introduction 1.Textual and Referential Aspects of Entities

In this work we present a system designed for the extraction of entities from textual data. Named entities (NEs), which include person, location, company or organization names1 must therefore be detected using named entity recognition (NER) techniques. In addition to this detection based on their surface forms, NEs can be identified by mapping them to the actual entity they denote, in order for these extractions to constitute useful and complete information. However, because of name variation, which can be surfacic or encyclopedic, an entity can be denoted by several mentions (e.g., Bruce Springsteen, Springsteen, the Boss); conversely, due to name ambiguity, a single mention can denote several distinct entities (Orange is the name of 22 locations in the world; in French, M. Obama can denote both the US president Barack Obama (M. is an abbreviation of Monsieur 'Mr') or his spouse Michelle Obama; in this case ambiguity is caused by variation). Even in the case of unambiguous mentions, a clear link should be established between the surface mention and a uniquely identifiable entity, which is achieved by entity linking (EL) techniques.

Entity Approach and Related Work

In order to obtain referenced entities from raw textual input, we introduce a system based on the joint application of named entity recognition (NER) and entity linking (EL), where the NER output is given to the linking component as a set of possible mentions, preserving a number of ambiguous readings. The linking process must thereafter evaluate which readings are the most probable, based on the most likely entity matches inferred from a similarity measure with the context. NER has been widely addressed by symbolic, statistical as well as hybrid approaches. Its major part in information extraction (IE) and other NLP applications has been stated and encouraged by several editions of evaluation campaigns such as MUC [START_REF] Marsh | Muc-7 evaluation of ie technology: Overview of results[END_REF], the CoNLL-2003 NER shared task [START_REF] Tjong | Introduction to the conll-2003 shared task: Languageindependent named entity recognition[END_REF] or ACE [START_REF] Doddington | The automatic content extraction (ace) program-tasks, data, and evaluation[END_REF], where NER systems show near-human performances for the English language. Our system aims at benefitting from both symbolic and statistical NER techniques, which have proven efficient but not necessarily over the same type of data and with different precision/recall tradeoff. NER considers the surface form of entities; some type disambiguation and name normalization can follow the detection to improve the result precision but do not provide referential information, which can be useful in IE applications. EL achieves the association of NER results with uniquely identified entities, by relying on an entity repository, available to the extraction system and defined beforehand in order to serve as a target for mention linking. Knowledge about entities is gathered in a dedicated knowledge base (KB) to evaluate each entity's similarity to a given context. After the task of EL was initiated with Wikipedia-based works on entity disambiguation, in particular by [START_REF] Cucerzan | Large-scale named entity disambiguation based on wikipedia data[END_REF] and [START_REF] Bunescu | Using encyclopedic knowledge for named entity disambiguation[END_REF], numerous systems have been developed, encouraged by the TAC 2009 KB population task [START_REF] Mcnamee | Overview of the tac 2009 knowledge base population track[END_REF]. Most often in EL, Wikipedia serves both as an entity repository (the set of articles referring to entities) and as a KB about entities (derived from Wikipedia infoboxes and articles which contain text, metadata such as categories and hyperlinks). [START_REF] Zhang | Entity linking leveraging: automatically generated annotation[END_REF] show how Wikipedia, by providing a large annotated corpus of linked ambiguous entity mentions, pertains efficiently to the EL task. Evaluated EL systems at TAC report a top accuracy rate of 0.80 on English data [START_REF] Mcnamee | An evaluation of technologies for knowledge base population[END_REF].

Entities that are unknown to the reference database, called out-of-base entities, are also considered by EL, when a given mention refers to an entity absent from the available Wikipedia articles. This is addressed by various methods, such as setting a threshold of minimal similarity for an entity selection [START_REF] Bunescu | Using encyclopedic knowledge for named entity disambiguation[END_REF], or training a separate binary classifier to judge whether the returned top candidate is the actual denotation [START_REF] Zheng | Learning to link entities with knowledge base[END_REF]. Our approach of this issue is closely related to the method of [START_REF] Dredze | Entity disambiguation for knowledge base population[END_REF], where the out-of-base entity is considered as another entry to rank.

Our task differs from EL configurations outlined previously, in that its target is entity extraction from raw news wires from the news agency Agence France Presse (AFP), and not only linking relying on gold NER annotations: the input of the linking system is the result of an automatic NER step, which will produce errors of various kinds. In particular, spans erroneously detected as NEs will have to be discarded by our EL system. This case, which we call not-an-entity, contitute an additional type of special situations, together with out-of-base entities but specific to our setting. This issue, as well as others of our task specificities, will be discussed in this paper. In particular, we use resources partially based on Wikipedia but not limited to it, and we experiment on the building of a domain specific entity KB instead of Wikipedia.

Section 2 presents the resources used throughout our system, namely an entity repository and an entity KB acquired over a large corpus of news wires, used in the final linking step. Section 3 states the principles on which the NER components of our system relies, and introduces the two existing NER modules used in our joint architecture. The EL component and the methodology applied are presented in section 4. Section 5 illustrates this methodology with a number of experiments and evaluation results.

Entity Resources

Our system relies on two large-scale resources which are very different in nature:

• the entity database Aleda, automatically extracted from the French Wikipedia and

Geonames;

• a knowledge base extracted from a large corpus of AFP news wires, with distributional and contextual information about automatically detected entites.

Aleda

The Aleda entity repository2 is the result of an extraction process from freely available resources [START_REF] Sagot | Aleda, a free largescale entity database for French[END_REF]. We used the French Aleda databased, extracted the French Wikipedia 3 and Geonames 4 . In its current development, it provides a generic and wide coverage entity resource accessible via a database. Each entity in Aleda is associated with a range of attributes, either referential (e.g., the type of the entity among Person, Location, Organization and Company, the population for a location or the gender of a person, etc.) or formal, like the entity's URI from Wikipedia or Geonames; this enables to uniquely identify each entry as a Web resource. Moreover, a range of possible variants (mentions when used in textual content) are associated to entities entries. Aleda's variants include each entity's canonical name, Geonames location labels, Wikipedia redirection and disambiguation pages aliases, as well as dynamically computed variants for person names, based in particular on their first/middle/last name structure. The French Aleda used in this work comprises 870,000 entity references, associated with 1,885,000 variants.

The main informative attributes assigned to each entity in Aleda are listed and illustrated by examples of entries in Tab. 1. The popularity attribute is given by an approximation based on the length of the entity's article or the entity's population, from Wikipedia and Geonames entries respectively. Table 1 also details the structure of Aleda's variants entries, each of them associated with one or several entities in the base.

Unlike most EL systems, Wikipedia is not the entity base we use in the present work; rather, we rely on the autonomous Aleda database. The collect of knowledge about entities and their usage in context will also differ in that our target data are news wires, for which the adaptability of Wikipedia can be questioned.

Knowledge Acquisition over AFP news

The linking process relies on knowledge about entities, which can be acquired from their usage in context and stored in a dedicated KB. AFP news wires, like Wikipedia articles, have their own structure and formal metadata: while Wikipedia articles each have a title referring to an entity, object or notion, a set of categories, hyperlinks, etc., AFP news wires have a headline and are tagged with a subject (such as Politics or Culture) and several keywords (such as cinema, inflation or G8), as well as information about the date, time and location of production. Moreover, the distribution of entities over news wires can be expected to be significantly different from Wikipedia, in particular w.r.t. uniformity, since a small set of entities forms the majority of occurrences. Our particular context can thus justify the need for a domain specific KB. As opposed to Wikipedia where entities are identifiable by hyperlinks, AFP corpora provide no such indications. Wikipedia is in fact a corpus where entity mentions are clearly and uniquely linked, whereas this is what we aim at achieving over AFP's raw textual data. The acquisition of domain specific knowledge about entities from AFP corpora must circumvent this lack of indications. In this perspective we use an implementation of a naive linker described in [START_REF] Stern | Détection et résolution d'entités nommées dans des dépêches d'agence[END_REF]. For the main part, this system is based on heuristics favoring popular entities in cases of ambiguities. An evaluation of this system showed good accuracy of entity linking (0.90) over the subset of correctly detected entity mentions:5 on the evaluation data, the resulting NER reached a precision of 0.86 and a recall of 0.80. Therefore we rely on the good accuracy of this system to identify entities in our corpus, bearing in mind that it will however include cases of false detections, while knowledge will not be available on missed entities. It can be observed that by doing so, we aim at performing a form of co-training of a new system, based on supervised machine learning. In particular, we aim at providing a more portable and systematic method for EL than the heuristics-based naive linker which is highly dependent on a particular NER system, SXPipe/NP, described later on in section 3.2.

The knowledge acquisition was conducted over a large corpus of news wires (200,000 news items of the years 2009, 2010 and part of 2011). For each occurrence of an entity identified as such by the naive linker, the following features are collected, updated and stored in the KB at the entity level: (i) entity total occurrences and occurrences with a particular mention; (ii) entity occurrence with a news item topics and keywords, most salient words, date and location; (iii) entity co-occurrence with other entity mentions in the news item. These features are collected for both entities identified by the naive linker as Aleda's entities and mentions recognized by NER pattern based rules; the latter account for out-ofbase entities, approximated by a cluster of all mentions whose normalization returns the same string. For instance, if the mentions John Smith and J. Smith were detected in a document but not linked to an entity in Aleda, it would be assumed that they co-refer to an entity whose normalized name would be John Smith; this anonymous entity would therefore be stored and identified via this normalized name in the KB, along with its occurrence information.

NER Component

Principles

One challenging subtask of NER is the correct detection of entity mentions spans among several ambiguous readings of a segment. The other usual subtask of NER consists in the labeling or classification of each identified mention with a type; in our system, this functionality is used as an indication rather than a final attribute of the denoted entity. The type assigned to each mention will in the end be the one associated with the matching entity. The segment Paris Hilton can for instance be split in two consecutive entity mentions, Paris and Hilton, or be read as a single one. Whether one reading or the other is more likely can be inferred from knowledge about entities possibly denoted by each of these three mentions: depending on the considered document's topic, it can be more probable for this segment to be read as the mention Paris Hilton, denoting the celebrity, rather than the sequence of two mentions denoting the capital of France and the hotel company. Based on this consideration, our system relies on the ability of the NER module to preserve multiple readings in its output, in order to postpone to the linker the appropriate decisions for ambiguous cases. Two NER systems fitted with this ability are used in our architecture. NP is part of the SXPipe surface processing chain [START_REF] Sagot | SXPipe 2 : architecture pour le traitement présyntaxique de corpus bruts[END_REF]. It is based on a series of recognition rules and on a large coverage lexicon of possible entity variants, derived from the Aleda entity repository presented in section 2.1. As an SXPipe component, NP formalizes the text input in the form of directed acyclic graphs (DAGs), in which each possible entity mention is represented as a distinct transition, as illustrated in Figure 1. Possible mentions are labeled with types among Person, Location, Organization and Company, based on the information available about the entity variant in Aleda and on the type of the rule applied for the recognition. Figure 1 also shows how an alternative transition is added to each mention reading of a segment, in order to account for a possible non-entity reading (i.e., for a false match returned by the NER module). When evaluating the adequacy of each reading, the following EL module will in fact consider a special not-an-entity candidate as a possible match for each mention, and select it as the most probable if competing entity readings prove insufficiently adequate w.r.t. the considered context.

Statistical NER: LIANE

The statistical NER system LIANE [START_REF] Bechet | Unsupervised knowledge acquisition for extracting named entities from speech[END_REF] is based on (i) a generative HMMbased process used to predict part-of-speech and semantic labels among Person, Location, Organi-zation and Product for each input word 6 , and (ii) a discriminative CRF-based process to determine the entity mentions' spans and overall type. The HMM and CRF models are learnt over the ESTER corpus, consisting in several hundreds of hours of transcribed radio broadcast [START_REF] Galliano | The Ester 2 Evaluation Campaign for the Rich Transcription of French Radio Broadcasts[END_REF] put of LIANE consists in an n-best lists of possible entity mentions, along with a confidence score assigned to each result. Therefore it also provides several readings of some text segments, with alternatives of entity mention readings.

As shown in [START_REF] Bechet | Unsupervised knowledge acquisition for extracting named entities from speech[END_REF], the learning model of LIANE makes it particularly robust to difficult conditions such as non capitalization and allows for a good recall rate on various types of data. This is in opposition with manually handcrafted systems such as SXPipe/NP, which can reach high precision rates over the development data but prove less robust otherwise. These considerations, as well as the benefits of a cooperations between these two types of systems are explored in [START_REF] Béchet | Coopération de méthodes statistiques et symboliques pour l'adaptation non-supervisée d'un système d'étiquetage en entités nommées[END_REF].

By coupling LIANE and SXPipe/NP to perform the NER step of our architecture, we expect to benefit from each system's best predictions and improving the precision and recall rates. This is achieved by not enforcing disambiguation of spans and types at the NER level but by transferring this possible source of errors to the linking step, which will rely on entity knowledge rather than mere surface forms to determine the best readings, along with the association of mentions with entity references.

Linking Component

Methodology for Best Reading Selection

As previously outlined, the purpose of our joint architecture is to infer best entity readings from 6 For the purpose of type consistency across both NER modules, the NP type Company is merged with Organization, and the LIANE mentions typed as Product are ignored since they are not yet supported by the overall architecture. The linking component will therefore process ambiguous NER outputs in the following way, illustrated by Fig. 2.

1. For each mention returned by the NER module, we aim at finding the best fitting entity w.r.t. the context of the mention occurrence, i.e., at the document level. This results in a list of candidate entities associated with each mention. This candidates set always includes the not-an-entity candidate in order to account for possible false matches returned by the NER modules.

2. The list of candidates is ordered using a pointwise ranking model, based on the maximum entropy classifier megam. 7 The best scored candidate is returned as a match for the mention; it can be either an entity present in Aleda, i.e., a known entity, or an anonymous entity, seen during the KB acquisition but not resolved to a known reference and identified by a normalized name, or the special not-an-entity candidate, which discards the given mention as an entity denotation.

3. Each reading is assigned a score depending on the best candidates' scores in the reading.

The key steps of this process are the selection of candidates for each mention, which must reach a sufficient recall in order to ensure the reference resolution, and the building of the feature vector for each mention/entity pair, which will be evaluated by the candidate ranker to return the most adequate entity as a match for the mention. Throughout this process, the issues usually raised by EL must be considered, in particular the ability for the model to learn cases of out-of-base entities, which our system addresses by forming a set of candidates not only from the entity reference base (i.e., Aleda), but also from the dedicated KB where anonymous entities are also collected. Furthermore, unlike the general configuration of EL tasks, such as the TAC KB population task (sec-tion 1.2), our input data does not consist in mentions to be linked but in multiple possibilities of mention readings, which adds to our particular case the need to identify false matches among the queries made to the linker module.

Candidates Selection

For each mention detected in the NER output, the mention string or variant is sent as a query to the Aleda database. Entity entries associated with the given variant are returned as candidates. The set of retrieved entities, possibly empty, constitutes the candidate set for the mention. Because the knowledge acquisition included the extraction of unreferenced entities identified by normalized names (section 2.2), we can send the normalization of the mention as an additional query to our KB. If a corresponding anonymous entity is returned, we can create an anonymous candidate and add it to the candidate set. Anonymous candidates account for the possibility of an out-of-base entity denoted by the given mention, with respectively some and no information about the potential entity they might stand for. Finally, the set is augmented with the special not-an-entity candidate.

Features for Candidates Ranking

For each pair formed by the considered mention and each entity from the candidate set, we compute a feature vector which will be used by our model for assessing the probability that it represents a correct mention/entity linking. The vector contains attributes pertaining to the mention, the candidate and the document themselves, and to the relations existing between them. Entity attributes Entity attributes present in Aleda and the KB are used as features: Aleda provides the entity type, a popularity indication and the number of variants associated with the entity. We retrieve from the KB the entity frequency over the corpus used for knowledge acquisition. Mention attributes At the mention level, the feature set considers the absence or presence of the mention as a variant in Aleda (for any entity), its occurrence frequency in the document, and whether similar variants, possibly indicating name variation of the same entity, are present in the document (similar variants can have a string equal to the mention's string, longer or shorter than the mention's string, included in the mention's string or including it). In the case of a mention returned by LIANE, the associated confidence score is also included in the feature set. Entity/mention relation The comparison between the surface form of the entity's canonical name and the mention gives a similarity rate feature. Also considered as features are the relative occurrence frequency of the entity w.r.t. the whole candidate set, the existence of the mention as a variant for the entity in Aleda, the presence of the candidate's type (retrieved from Aleda) in the possible mention types provided by the NER. The KB indicates frequency of its occurrences with the considered mention, which adds another feature. Document/entity similarity Document metadata (in particular topics and keywords) are inherited by the mention and can thus characterize the entity/mention pair. Equivalent information was collected for entities and stored in the KB, which allows to compute a cosine similarity between the document and the candidate. Moreover, the most salient words of the document are compared to the ones most frequently associated with the entity in the KB. Several atomic and combined features are derived from these similarity measures.

Other features pertain to the NER output configuration, as well as possible false matches: NER combined information One of the two available NER modules is selected as the base provider for entity mentions. For each mention which is also returned by the second NER module, a feature is instanciated accordingly. Non-entity features In order to predict cases of not-an-entity readings of a mention, we use a generic lexicon of French forms [START_REF] Sagot | The Lefff , a freely available and large-coverage morphological and syntactic lexicon for French[END_REF] where we check for the existence of the mention's variant, both with and without capitalization. If the mention's variant is the first word of the sentence, this information is added as a feature.

These features represent attributes of the entity/mention pair which can either have a boolean value (such as variant presence or absence in Aleda) or range throughout numerical values (e.g., entity frequencies vary from 0 to 201,599). In the latter case, values are discretized. All features in our model are therefore boolean.

Best Candidate Selection

Given the feature vector instanciated for an (candidate entity, mention) pair, our model assigns it a score. All candidates in the subset are then ranked accordingly and the first candidate is returned as the match for the current mention/entity linking. Anonymous and not-an-entity candidates, as defined earlier and accounting respectively for potential out-of-base entity linking and NER false matches, are included in this ranking process.

Ranking of Readings

The last step of our task consists in the ranking of multiple readings and has yet to be achieved in order to obtain an output where entity mentions are linked to adequate entities. In the case of a reading consisting in a single transition, i.e., a single mention, the score is equal to the best candidate's score. In case of multiple transitions and mentions, the score is the minimum among the best candidates' scores, which makes a low entity match probability in a mention sequence penalizing for the whole reading. Cases of false matches returned by the NER module can therefore be discarded as such in this step, if an overall non-entity reading of the whole path receives a higher score than the other entity predictions.

Experiments and Evaluation

Training and Evaluation Data

We use a corpus of 96 AFP news items intended for both NER and EL purposes: the manual annotation includes mention boundaries as well as an entity identifier for each mention, corresponding to an Aleda entry when present or the normalized name of the entity otherwise. This allows for the model learning to take into account cases of out-of-base entities. This corpus contains 1,476 mentions, 437 distinct Aleda's entries and 173 entities absent from Aleda. All news items in this corpus are dated May and June 2009.

In order for the model to learn from cases of not-an-entity, the training examples were augmented with false matches from the NER step, associated with this special candidate and the positive class prediction, while other possible candidates were associated with the negative class. Using a 10-fold cross-validation, we used this corpus for both training and evaluation of our joint NER and EL system.

It should be observed that the learning step concerns the ranking of candidates for a given mention and context, while the final purpose of our system is the ranking of multiple readings of sentences, which takes place after the application of our ranking model for mention candidates. Thus our system is evaluated according to its ability to choose the right reading, considering both NER recall and precision and EL accuracy, and not only the latter.

Task Specificities

As outlined in section 1.2, the input for the standard EL task consists in sets of entity mentions from a number of documents, sent as queries to a linking system. Our current task differs in that we aim at both the extraction and the linking of entities in our target corpus, which consists in unannotated news wires. Therefore, the results of our system are comparable to previous work when considering a setting where the NER output is in fact the gold annotation of our evaluation data, i.e., when all mention queries should be linked to an entity. Without modifying the parameters of our system (i.e., no deactivation of false matches predictions), we obtain an accuracy of 0.76, in comparison with a TAC top accuracy of 0.80 and a median accuracy of 0.70 on English data. 8It is important to observe that our data consists only in journalistic content, as opposed to the TAC dataset which included various types of corpora. This difference can lead to unequally difficulty levels w.r.t. the EL task, since NER and EL in journalistic texts, and in particular news wires, tend to be easier than on other types of corpora. This comes among other things from the fact that a small number of popular entities constitute the majority of NE mention occurrences.

In most systems, EL is performed over noisy NER output and participates to the final decisions about NEs extractions. Therefore the ability of our system to correctly detect entity mentions in news content is estimated by computing its precision, recall and f-measure. 9 The EL accuracy, i.e., the rate of correctly linked mentions, is mea- sured over the subset of mentions whose reading was adequately selected by the final ranking. The evaluation of our system has been conducted over the corpus described previously with settings presented in the next section.

Settings and results

We used each of the two available NER modules as a provider for entity mentions, either on its own or together with the second system, used as an indicator. For each of these settings, we tried a modified setting in which the prediction of the naive linker (NL) used to build the entity KB (section 2.2) was added as a feature to each mention/candidate pair (settings SXPipe/NP- NL and LIANE-NL). These experiments' results are reported in Table 3 and are given in terms of:

• NER precision, recall and f-measure;

• EL accuracy over correctly recognized entities; therefore, the different figures in column EL Accuracy are not directly comparable to one another, as they are not obtained over the same set of mentions;

• joint NER+EL precision, recall and fmeasure; the precision/recall is computed as the product of the NER precision/recall by the EL accuracy.

As expected, SXPipe/NP performs better as far as NER precision is concerned, and LIANE performs better as far as NER recall is concerned. However, the way we implemented hybridation at the NER level does not seem to bring improvements. Using the output of the naive linker as a feature leads to similar or slightly lower NER precision and recall. Finally, it is difficult to draw clear-cut comparative conclusions at this stage concerning the joint NER +EL task.

Conclusion and Future Work

We have described and evaluated various settings for a joint NER and EL system which relies on the NER systems SXPipe/NP and LIANE for the NER step. The EL step relies on a hybrid model, i.e., a statistical model trained on a manually annotated corpus. It uses features extracted from a large corpus automatically annotated and where entity disambiguations and matches were computed using a basic heuristic tool. The results given in the previous section show that the joint model allows for good NER results over French data. The impact of the hybridation of the two NER modules over the EL task should be further evaluated. In particular, we should investigate the situations where an mention was incorrectly detected (e.g., the span is not fully correct) although the EL module linked it with the correct entity. Moreover, a detailed evaluation of out-of-base linkings vs. linking in Aleda remains to be performed.

In the future, we aim at exploring various additional features in the EL system, in particular more combinations of the current features. The adaptation of our learning model to NER combinations should also be improved. Finally, a larger set of training data should be considered. This shall become possible with the recent manual annotation of a half-million word French journalistic corpus.

Figure 1 :

 1 Figure 1: Ambiguous NER output for the segment Paris Hilton in SXPipe/NP

Figure 2 :

 2 Figure 2: Possible readings of the segment Paris Hilton and ordered candidates

Table 1 :

 1 Structure of Entities Entries and Variants in Aleda

	Entities
	ID	Type	CanonicalName	Popularity URI
	20013 Loc	Kingdom of Spain	46M	geon:2510769
	10063 Per	Michael Jordan	245	wp:Michael Jordan
	20056 Loc	Orange (California) 136K	geon:5379513
	10039 Comp	Orange	90	wp:Orange (entreprise)
	Variants
	ID	Variant	FirstName	MidName LastName
	20013 Espagne	-	-	-
	10063 Jordan	-	-	Jordan
	10029 George Walker Bush George	Walker	Bush
	10039 Orange	-	-	-
	20056 Orange	-	-	-

  , annotated with the BIO format (table2). The out-

	investiture NFS	O
	aujourd'hui ADV	B-TIME
	à	PREPADE O
	Bamako	LOC	B-LOC
	Mali	LOC	B-LOC

Table 2 :

 2 BIO annotation for LIANE training

Table 3 :

 3 Joint NER and EL results. Each EL accuracy covers a different set of correctly detected mentions

	Setting		NER		EL		Joint NER+EL
		Precision Recall f-measure Accuracy Recall Precision f-measure
	SXPipe/NP	0.849	0.768	0.806	0.871	0.669	0.740	0.702
	LIANE	0.786	0.891	0.835	0.820	0.730	0.645	0.685
	SXPipe/NP-NL	0.775	0.726	0.750	0.875	0.635	0.678	0.656
	LIANE-NL	0.782	0.886	0.831	0.818	0.725	0.640	0.680
	SXPipe/NP & 2	0.812	0.747	0.778	0.869	0.649	0.705	0.676
	LIANE & SXPipe/NP	0.803	0.776	0.789	0.859	0.667	0.689	0.678

The set of possible named entities varies from restrictive, as in our case, to wide definitions; it can also include dates, event names, historical periods, etc.

Aleda is part of the Alexina project and freely available at https://gforge.inria.fr/projects/alexina/.

www.fr.wikipedia.org

www.geonames.org

This subset is defined by a strict span and type correct detection, and among the sole entities for which a match in Aleda or outside of it was identified; the evaluation data is presented in section 5.1.

http://www.cs.utah.edu/ ∼ hal/megam/

As explained previously, these figures, as well as the ones presented later on, cannot be compared with the 0.90 score obtained by the naive linker which we used for the entity KB acquisition. This score is obtained only on mentions identified by the SXPipe/NP system with the correct span and type, whereas our system does not consider the mention type as a contraint for the linking process, and on correct identification of a match in or outside of

Aleda. 9 Only mention boundaries are considered for NER evaluation, while other settings require correct type identification for validating a fully correct detection. In our case, NER is not a final step, and entity typing is derived from the entity linking result.


Rosa Stern
email: rosa.stern@afp.com

Benoît Sagot
email: benoit.sagot@inria.fr

Population of a Knowledge Base for News Metadata from Unstructured Text and Web Data

We present a practical use case of knowledge base (KB) population at the French news agency AFP. The target KB instances are entities relevant for news production and content enrichment. In order to acquire uniquely identified entities over news wires, i.e. textual data, and integrate the resulting KB in the Linked Data framework, a series of data models need to be aligned: Web data resources are harvested for creating a wide coverage entity database, which is in turn used to link entities to their mentions in French news wires. Finally, the extracted entities are selected for instantiation in the target KB. We describe our methodology along with the resources created and used for the target KB population.

Current research conducted at the French news agency AFP focuses on the acquisition and storage of knowledge, particularly entities, relevant for the news production and usable as metadata for content enrichment. This objective sets off the need for a dedicated knowledge base (KB) relying on a light-weight ontology of entities mappable to the Linked Data framework. Identification of entities such as persons, organizations and geopolitical entities (GPES)1 in unstructured textual data, news wires in French in our case, underlie the construction and enrichment of such a KB. This specific need is met by the KB population task, now well defined within the annual TAC dedicated track (Ji et al., 2011). KB population indeed relies on the ability to link entity mentions in textual data to a KB entry (entity linking subtask, henceforth EL),2 which follows pioneer work in entity disambiguation [START_REF] Bunescu | Using encyclopedic knowledge for named entity disambiguation[END_REF][START_REF] Cucerzan | Large-scale named entity disambiguation based on Wikipedia data[END_REF]. In a similar way to systems described in Dredze et al. (2010) and Ji & Grishman (2011), we conduct EL over AFP news wires in order to obtain relevant entities meant to populate the target KB, adapting these techniques to French data. This linking process is based on Web data extraction for both coverage and the purpose of Linked Data integration, which has become a widely explored trend in news management and publishing projects, such as the ones conducted by the BBC (Kobilarov et al., 2009) or the New York Times (NYT).

Compared to other KB population settings, this knowledge acquisition process is done throughout a sequence of resources and extraction steps rather than in a cyclic way. Instead of considering one KB as both an entity resource and the target of the population task, the target KB (AFP Metadata Ontology, henceforth AMO) is viewed as initially empty and progressively augmented with entity instances. This is because the use intended for AMO does not rely on exhaustivity, but on a relevant set of entities mentioned in the daily news production. This set is not fixed a priori and must be regularly updated in order to maintain a reflection of entities' emergence in the news domain. For instance, not all cities in the world need be instanciated in AMO, but the city Fukushima should become an entry as soon as its part in major events is stated in news wires. The relevant entity set can then be matched to new entries and updates in parallel documentation resources maintained at the AFP. The KB population process is broken down into several layers of data extraction and alignment, as sketched in Figure 1:

Step 1 The models of GeoNames and Wikipedia are mapped to a unified entity model; extraction from these Web datasets based on this mapping result in an entity database named Aleda, whose schema is presented in section 2, along with its alignement with Wikipedia and GeoNames data models.

Step 2 The Aleda database obtained in Step 1 provides entities for the linking step, where entity mentions in news wires are aligned with entries from Aleda. This process, along with the use of a joint resource for entity knowledge access, is described in section 3. EL in our particular task also targets the identification of new entities (i.e. absent from Aleda) and attempts to deal with the problem of possible named entity recognition errors in queries made to the system.

Step 3 The resulting entities aligned in Aleda must be anchored in the target KB, via instantiation of the adequate ontological class. Contextual information gathered during the entity extraction and linking process can be used at this point (section 4).

Entity Extraction from Web Data

Step 1 in our architecture is based on two Web datasets: the geographical database GeoNames brings together millions of location identifiers with associated information; Wikipedia is a wide coverage resource for entities, with continuous updates and article creations for entities involved in current events. The creation of a large-scale and unified entity resource is achieved by defining a database schema dedicated to entity representation, and by aligning both Wikipedia's and GeoNames' model with it. The schema considers Person, Organization and GPE as types of entries. The building of Aleda therefore relies on the identification of Wikipedia's and GeoNames' entries to which one of these types can be assigned.

Wikipedia Exploiting Wikipedia as a large-scale entity resource has been the focus of numerous efforts, such as [START_REF] Balasuriya | Named entity recognition in wikipedia[END_REF][START_REF] Charton | Nlgbase: a free linguistic resource for natural language processing systems[END_REF]. Each Wikipedia article, referring to an entity, concept or notion, is referenced under a number of categories, often fine-grained and specific. A hierarchical or ontological organization of these categories can be inferred [START_REF] Syed | Wikipedia as an ontology for describing documents[END_REF] but a given article is not anchored in a generic conceptual class such as an entity type in a straightforward fashion. The present model alignment thus consists in a mapping from Wikipedia categories to one of the target entity types. Each article mapped to one of these types leads to adding a corresponding entity in Aleda. The selection and typing process of articles as entities makes use of heuristics based on articles' categories and infoboxes and is achieved as follows.

Around 100 of the most frequent infobox templates are retrieved from the articles' structure and manually associated with an entity type (e.g., the Politician template is typed as Person). All articles associated with one of these templates (23% of the French Wikipedia) are assigned the same type. The categories associated with these typed articles are then considered: a category appearing mostly in articles of a given type is associated with it. 20,328 categories are thus typed (e.g., all articles with an infobox and tagged by a category such as Born in * have been assigned the type Person, which is therefore associated to this category). An article is selected as referring to an entity by assigning the type with maximal category association to it. If no type can be assigned (when no categories were associated with any type), an article's infobox whose template has been typed can provide the information. If the article has no such infobox, no entity is derived from it. Aleda's schema is filled with attribute values extracted from each selected article: the columns URI, name, weight are mapped to the article's URL, the normalized name inferred from the article's title and the article's length,3 respectively. A joint table furthermore groups possible variants or labels for each entity; these are inferred from Wikipedia's redirection and disambiguation pages, which provide aliases for article titles. For person names, additional variants are automatically generated, by identifying the first, possible middle and last names. GeoNames The model alignment from GeoNames is fairly straightforward since all its entries are locations. However this database is huge and present some noisy aspects, which we aim at avoiding by limiting the extraction to entities considered relevant for news content enrichment in French. Only GPEs such as countries, cities or country divisions were selected, based on the GeoNames feature provided. Heuristics are then designed for further selection: all GPEs in France are retained, as well as all non-French GPEs with more than 200 inhabitants. Aleda's schema is filled with values provided by GeoNames for each selected GPE: the columns URI, name, weight, subtype, country-code and coordinates are mapped to the GeoNames' entity URL, name, number of inhabitants, feature (such as P.PPL for populated place), ISO country code and coordinates, respectively. The joint variants table is filled with GeoNames' labels indicated as French or without language indication.

A unified entity resource of persons, organizations and GPEs4 is obtained, with 832,452 uniquely identified entities (33%, 62% and 5% of type Person, GPE and Organization, respectively), associated with 1,673,202 variants. 5

3 Text-Entity Alignment: Entity Linking

Methodology for Entity Linking

The knowledge acquisition step, crucial for the target KB population, consists in entity extraction from a corpus of AFP news wires. This is done by aligning detected entity mentions in news wires with entries the Aleda database introduced in 2. This linking component is based on the learning of a similarity model between entity attributes, including contextual ones, and a mention in a given document. It is challenging particularly because of name variation and ambiguity, which can prevent an accurate entity identification. The named entity recognition (NER) module first detects entity mentions, along with a possible type, which become queries for the candidates selection among the database entries. The entry with the highest similarity w.r.t. the mention and the document is chosen. This similarity is computed in a way comparable to a number of linking systems (described and evaluated in Dredze et al. ( 2010) and Ji & Grishman (2011)) and using the following features: Surface similarity The candidate's name and labels, available in the entity database, are compared to the mention (exact, partial, null string match). Candidate attributes The other candidate's attributes available in Aleda, such as its weight (or popularity) and its country-code, can be indicators in cases of ambiguity. Contextual similarity Entity contextual knowledge (associated to the entity resource for the linking component in fig. 1) is made available from the corpus of Wikipedia articles. In each article, entity mentions (identified by hyperlinks) are surrounded with information such as the article's categories (i), the most salient words of the article's text (ii) and the co-occurring entities in the article (iii). As news items are indexed via keywords and topics from AFP's controlled vocabulary lists, (i) are mapped to the document keywords and topics; (ii) and (iii) are 5 A comparison with the NLGbAse resource [START_REF] Charton | Nlgbase: a free linguistic resource for natural language processing systems[END_REF], which has similar objectives, can be found in (Sagot and Stern, 2012) The candidate maximizing the similarity is selected as the mention linking. However, this selection should not occur for every query, since (i) a number of entities mentioned in news wires don't have a corresponding Aleda entry and (ii) the automatic NER can return a number of false mentions. In the case of (i), the out-of-base entity (NIL) should be output as a linking result; in (ii), the mention should be discarded and indicate a not-an-entity reading (NAE) rather than give rise to a false link; for the latter eventuality, features indicating ambiguities with the common lexicon are added to the feature set. These cases are part of the training examples and are taken into account in the prediction by including the NIL and NAE candidates in the ranking process.

Experiments and Evaluation

Experiments and evaluation are conducted over a manually annotated corpus of 96 news items, dated May-June 2009, where each entity mention's boundaries and Aleda ID, if relevant, are indicated. Mentions referring to entities absent from Aleda are identified by a normalized name. The corpus includes 1,476 entity mentions, referring to 610 distinct entities among which 28% are absent from Aleda. The corpus is also annotated with the NER system SXPipe/NP 6 to add examples of false matches to the linking model. More precisely, the NER is applied without complete disambiguation: a number of typing and segmentation alternatives are not resolved at this level, but rather passed to the linking module. The underlying idea is to leave final entity recognition decisions to a level where knowledge and semantic information about entities are available. Figure 2 illustrates this ambiguity preservation, where SXPipe/NP builds possible readings of a text segment in terms of entity mentions.

In each reading built by the NER module, each 6 http://alpage.inria.fr/ ∼ sagot/sxpipe.html mention (gold or not) is associated with a set of candidates: all Aleda entries for which the mention is a possible variant, as well as NIL and NAE instances. All mention/candidate pairs are assigned a class, positive for correct links and negative for wrong ones, and form the training set for the linking model. A pair consisting in a false mention and the NAE candidate is for instance labeled as positive.

The training examples are fed to a maximum entropy classifier7 with a 10-fold cross-validation. Based on the resulting model, each pair is then ranked according to the score assigned by the classifier, which amounts to a pointwise ranking strategy. Once mentions are locally linked to the top-ranked candidates in all readings, the latter must in turn be ranked in order to finally disambiguate the current text segment. This ranking is done by assigning to each reading the score of its top-ranked candidate or, when the reading contains a sequence of mentions, the minimum of all top-ranked candidates' scores.

On the evaluation data, the system obtains encouraging results of linking accuracy (A) over the set of correctly detected entities (NER columns), compared to the top TAC-KB population evaluation results over English data with only correct entity mentions (table 1). The overall task, i.e. of joint entity recognition and linking, is also measured and is not comparable to equivalent work to the best of our knowledge. Results show that the system's ability to detect false entity mentions should be considerably improved, but filters out some NER noise, which could not be the case in a mere sequential system where all detected mentions would be equally handled by the linking module.

Conducted over a corpus of ≈ 400,000 news items of 2009 and 2010 on 16 news topics such as Politics, Economy or Culture, the EL step results in a set of 46,616 distinct identified entities (35%, 50% and 15% of GPEs, persons and organizations), along with information retrieved from Aleda. Moreover, each entity is associated with new information gathered at each of its occurrence in the extraction corpus. Hence the most frequent association of entities with news features (see Extracted entities in fig. 1) augment the extraction list with useful knowledge for further integration.

Target KB Population and Enrichment

The entities identified during the previous step are made available for the target KB population. Given the specialized use intended for AMO in further content enrichment, a high precision rate is expected w.r.t. its instances, which requires a phase of human intervention. However this process should not come down to a time costly and off-putting task, and should rely on concrete and systematic validation criteria. The extracted entities are presented to domain specialists in order to assess their relevance w.r.t. the KB. This judgement can leverage various type of information, including the usage of entities in the extraction corpus. Each entity is submitted to validation with its Wikipedia or GeoNames URL, which allows for an unambiguous verification of its identity on the Web. Furthermore, candidates are submitted in reverse frequency order: the mention rate of an entity over a given period can by itself indicate its relevance for a KB integration. 8 The validation of less frequent entities can rely in a greater extend on contextual information derived from news features: news topics, keywords, salient words and co-occurring entities can indicate the level of contribution of a given entity to the domain, and therefore its usefulness as a KB instance for further content enrichment. Moreover, the news publication dates collected for each occurrence allow to visualize an entity salience in a given period of time, and can indicate its emergence as an important news actor at certain event peaks. Following this process, 5,759 relevant entities were selected. Their concrete integration in AMO consists in an automatic instantiation within the underlying ontological model (see AMO in fig. 1). The adequate AMO Entity subclass is determined by a straightforward mapping from Aleda entity types to AMO Entity subclasses. Finer subclasses, such as 8 Roughly 20% of the distinct entities constitute 80% of the total occurrences in corpus (more than 4 million mentions); hence examining most frequent entities quickly allows for an initial population with the most prominent instances.

Location/POI (point of interest) can be instantiated based on GeoNames features associated with the entity (e.g., locations with the GeoNames museum feature are mapped to the POI subclass). The instantiation process also considers useful information available on entities: Aleda attributes -the entity normalized name or its geographical coordinates -, are represented in the form of an owl:dataProperty. Knowledge elements extracted from the corpus, such as the news topic with which the entity is most mentioned, give rise to owl:objectProperties (whose domain and range are the considered entity and an adequate instance of the ontology, e.g. the Politics instance of the NewsTopic class).

When the process described in section 3 is repeated over new data, i.e. over the daily news flow, and along with regular updates of the Aleda to take into account new entries, the entities presented to validation can either be new w.r.t. AMO or already present as an instance. In the latter case, the additional information linked to the entity should be merged with the adequate existing node. Automatically identifying the adequate existing instance can be achieved by entity resolution techniques, such as systematic comparison of attribute values. In the case of a new entity, a new instance should be created as described above. This illustrates the challenge of dynamic enrichment of resources, comparable to the automatic detection of neologisms in the general language for lexical resources. This aspect of the KB population is intended to be at the center of further developments of our architecture.

In order to integrate AMO in the Linked Data (LD) framework, we applied entity resolution via URIs matches with existing LD datasets: all instances defining a Wikipedia URL could be linked to the equivalent DBpedia resource; 20% of AMO instances after the initial population were linked to the NYT data, thus making AMO a suitable resource for content enrichment and Web publishing in the news domain.

Figure 1 :

 1 Figure 1: Overview of entity extraction and KB population process

Figure 2 :

 2 Figure 2: SXPipe/NP output for the segment Paris Hilton compared to the document's salient words and entity mentions, respectively.The candidate maximizing the similarity is selected as the mention linking. However, this selection should not occur for every query, since (i) a number of entities mentioned in news wires don't have a corresponding Aleda entry and (ii) the automatic NER can return a number of false mentions. In the case of (i), the out-of-base entity (NIL) should be output as a linking result; in (ii), the mention should be discarded and indicate a not-an-entity reading (NAE) rather than give rise to a false link; for the latter eventuality, features indicating ambiguities with the common lexicon are added to the feature set. These cases are part of the training examples and are taken into account in the prediction by including the NIL and NAE candidates in the ranking process.3.2 Experiments and EvaluationExperiments and evaluation are conducted over a manually annotated corpus of 96 news items, dated May-June 2009, where each entity mention's boundaries and Aleda ID, if relevant, are indicated. Mentions referring to entities absent from Aleda are identified by a normalized name. The corpus includes 1,476 entity mentions, referring to 610 distinct entities among which 28% are absent from Aleda. The corpus is also annotated with the NER system SXPipe/NP 6 to add examples of false matches to the linking model. More precisely, the NER is applied without complete disambiguation: a number of typing and segmentation alternatives are not resolved at this level, but rather passed to the linking module. The underlying idea is to leave final entity recognition decisions to a level where knowledge and semantic information about entities are available. Figure2illustrates this ambiguity preservation, where SXPipe/NP builds possible readings of a text segment in terms of entity mentions.In each reading built by the NER module, each

  . Aleda is freely available at gforge.inria.fr/frs/download.php/30598/

			{Hilton} _Person_f
	{Paris} _Location	5 _Person_f {Paris Hilton}	{Hilton} _Person_m {Hilton} _Organization	6

Table 1 :

 1 NER, linking and joint results

			NER		EL	Joint NER+EL
		P	R	F	A (all/NILs) P	R	F		NAE
	AFP (French)	0.849 0.768 0.806 0.871/0.597 0.740 0.669 0.702 33%
	Top TAC-KBP (English)	-	-	-	0.821/0.891	-	-	-	-

These entity types are the usual focus of Information Extraction systems and are defined among others by the ACE entity recognition task(Doddington et

al., 2004).

The consequent slot filling subtask associates these entries to attributes and relations to other entities.

This value is seen as a weight in the extend that an article size can indicate the popularity of an entity relatively to others, particularly in cases of homonymy.

GPEs extracted from both GeoNames and Wikipedia are listed and associated by an owl:sameAs relation.

Megam: http://www.cs.utah.edu/ ∼ hal/megam/


Benoît Sagot
email: benoit.sagot@inria.fr

Rosa Stern
email: rosa.stern@afp.com

Aleda, a free large-scale entity database for French

Keywords: Named Entities, Entity Database, Named Entity Linking

Named entity recognition, which focuses on the identification of the span and type of named entity mentions in texts, has drawn the attention of the NLP community for a long time. However, many real-life applications need to know which real entity each mention refers to. For such a purpose, often refered to as entity resolution and linking, an inventory of entities is required in order to constitute a reference. In this paper, we describe how we extracted such a resource for French from freely available resources (the French Wikipedia and the GeoNames database). We describe the results of an instrinsic evaluation of the resulting entity database, named Aleda, as well as those of a task-based evaluation in the context of a named entity detection system. We also compare it with the NLGbAse database (Charton and Torres-Moreno, 2010), a resource with similar objectives.

Introduction

Identifying named entities (NE) is a widely studied issue in Natural Language Processing, because NEs are crucial targets in information extraction or retrieval tasks, but also for preparing further NLP tasks (e.g., parsing). Therefore a vast amount of work has been published that is dedicated to NErecognition, i.e., the task of identification of NE mentions (spans of text denoting a NE), and sometimes types. However, real-life applications need not only identify NE mentions, but also know which real entity they refer to; this issue is addressed in tasks such as knowledge base population with entity resolution and linking. For such a purpose, an inventory of entities is required prior to those tasks in order to constitute a reference. In this paper, we introduce such an entity database for French, Aleda, which is large-scale, freely available, and automatically extracted from free resources. After a short description of the related work in Section 2, with an emphasis on the entity database NLGbAse [START_REF] Charton | Nlgbase: a free linguistic resource for natural language processing systems[END_REF], we describe in Section 3 how we extracted Aleda from the French Wikipedia and the GeoNames database. Finally, we evaluate Aleda both instrinsically and in a task-oriented setting (Section 4).

Related work and contribution

Aleda is related to a number of works through its extensive use of Wikipedia as a base resource for entities. In particular, the tasks of entity disambiguation, linking and knowledge population have often relied on encyclopedic information about entities provided by Wikipedia. Following [START_REF] Bunescu | Using encyclopedic knowledge for named entity disambiguation[END_REF], a lot of research has been conducted in order to efficiently exploit both the content and the structure of Wikipedia [START_REF] Cucerzan | Large-scale named entity disambiguation based on wikipedia data[END_REF][START_REF] Giuliano | Instance-based ontology population exploiting named-entity substitution[END_REF][START_REF] Dredze | Entity disambiguation for knowledge base population[END_REF]Zheng et al., 2010). This is usually achieved along the lines of the following two-step process:

1. selecting the Wikipedia articles corresponding to entities, often by examinating the form of their titles, and 2. associating with the selected entities a set of attributes extracted from the Wikipedia structure: entity variants from text anchors, redirections and disambiguation pages, topic relation from article categories, lexical associations using the article texts and co-occuring entities.

The way Wikipedia is used for building Aleda (see Section 3.2) differs from this standard procedure, mostly (i) in the way entities are selected (implicitly through typing rather than by title form examination) and (ii) by the reduced amount of knowledge finally associated with entities. This is due to the fact that Aleda is not designed for a particular task but rather aims at being a stand-alone resource whose structure can be used by different applications with minimum adaptation. Aleda, which can be seen as a Wikipedia and GeoNames extraction rather than exploration, can thus be used directly in any application, avoiding potentially impeding computing costs often encountered when exploiting resources such as Wikipedia on-the-fly. Another stand-alone base extracted from the French Wikipedia exists, namely NLGbAse [START_REF] Charton | Nlgbase: a free linguistic resource for natural language processing systems[END_REF]. The extraction process used for creating NLGbAse is comparable to our own approach for building Aleda. Table 1: Quantitative information about the NLGbAse database [START_REF] Charton | Nlgbase: a free linguistic resource for natural language processing systems[END_REF].

dommage collatéral 'collateral damage' or cyclotron 'cyclotron', names of species such as oreillard 'plecotus'), but also to other types of pages (e.g., liste swadesh de l'arabe 'Swadesh list for Arabic').

Building Aleda

Since the preliminary version of Aleda, briefly described in (Stern and Sagot, 2010), the extraction process from the French Wikipedia and GeoNames has been strongly refined and the resulting database has been evaluated and used in various settings. Each entity in Aleda is associated with information such as a weight or a subtype, as well as with a set of variants (mentions), themselves associated with additional information (e.g., the first/middle/last name structure for most person names).

Extracting locations from GeoNames

For location names, the extraction process from GeoNames is fairly straightforward. We first extract from the huge GeoNames database all entities associated with a subtype considered relevant for domain-generic corpus processing in French (countries, cities and so on). However, this filtering still outputs too many entities. Therefore, we designed heuristics for selecting entities considered as relevant for processing such corpora (e.g., we retained all locations in France as well as all non-French locations for which GeoNames provide a number of inhabitant greater then 200; all mentions associated with a non-empty language tag other than French were discarded, as well as all mentions using non-latin characters). Each selected entity is extracted with its GeoNames id and URI, its standardized name and its geographical coordinates, as well as a weight computed from the number of its inhabitants.

Extracting persons and organizations from Wikipedia

For other types of entities, we designed an architecture for exploiting the French Wikipedia, following previous work [START_REF] Balasuriya | Named entity recognition in wikipedia[END_REF][START_REF] Charton | Nlgbase: a free linguistic resource for natural language processing systems[END_REF]. In order to select all relevant articles corresponding to entities, we proceed in three steps. All infoboxes 1 included in articles are first extracted. Only a small proportion of articles include an infobox, but most infobox therefore, this category is associated with the type "Person"). Entities are finally extracted as follows: for a given article, we extract the type of each of its categories that has been assigned one; the most frequent type is assigned to the article; if no type can be assigned this way, we look if the article contains an infobox; if it is the case and if this infobox was manually typed, the corresponding type is assigned to the article; if it is not the case, this article will not correspond to an entity in Aleda.

For each article which was selected as an entity, we extract the entity URI, a standardized name (the title of the article) and a weight computed from the number of lines of the wikipedia article, which can be seen as an indication of the entity's popularity. Entity variants are extracted using Wikipedia redirection links, as well as disambiguation pages: their titles are considered as variants of entities corresponding to articles they are referring to as alternatives.2 For person names, additional variants are automatically generated, by identifying the first, middle and last names (when relevant), based on the title of the article and its first line (e.g., the variants J.W. Smith and John W. Smith can be added to the entity named John William Smith).

In its last version, the Aleda entity database contains 872,087 entities associated with 1,673,202 mentions. The breakdown by major entity types is given in Table 2. A few examples of entries, together with their counterpart in NLGbAse (when it exists), is given in Table 3.

Evaluation and contextual use of Aleda

An entity base like Aleda can be evaluated with regard to both its intrinsic quality as a collection of entities and its usefullness when used for a particular task.

Intrinsic evaluation

The quality of Aleda can be evaluated by (i) mesuring the rate of variants which are correctly linked to a particular entity, i.e., which are relevant variants of the considered entity (precision) and (ii) its coverage (recall), for which a reference is needed. We evaluated the precision (i) by randomly selecting 100 variants from Aleda and checking whether the variant was relevant as such and whether the association with one or several entities was correct. Among those 100 cases, one was not a relevant entity variant and two were on the verge of irrelevance (but established redirections in Wikipedia, e.g., Martino for Martino Longhi l'Ancien); all 97 others were correct variants, with a correct link to the associated entity or entities.

For recall (ii), we use as a reference a manually annotated corpus described in (Stern and Sagot, 2010) and considered three different cases. This corpus contains 709 distinct entity mentions, corresponding to 610 distinct entities, 518 having a Wikipedia or GeoNames page.3 Among them, 437 match an entity in Aleda (over 508 distinct mentions). Among the 709 distinct mentions in this corpus, 497 are in Aleda. Among the 617 distinct pairs of entity/mention where the entity has a Wikipedia or GeoNames page, 462 exist in Aleda. The coverage in terms of entities is thus evaluated at 84.4%, 70% in terms of mentions and 74.9% in terms of entity/mention pairs.

Task-based evaluation: using Aleda in a named-entity recognition system

Aleda has been used in the context of a knowledge base construction at the French press agency (AFP): for the purposes of content enrichment with metadata and integration of the AFP production in the Linked Open Data network,4 a reference base of the most relevant entities for the AFP (e.g., person, location, organization names) is needed. In order to initiate the population of this KB, a large corpus of AFP news has been tagged with a NER system which use Aleda mentions as a lexicon for its detection rules.5 Recognized entity mentions were then linked to matching entities in Aleda; finally, those links where manually validated (or simply rejected when the NER system provided a false mention) and resolved in the cases of ambiguities (several possible entities for one mention), where the information provided by Wikipedia and Geonames URIs could help the process of disambiguation. Thanks to the large-coverage of Aleda and its compliance with the Linked Data requirements, 6 this process provided a set of 5,350 domain related entities. A second iteration using a more recent corpus and the same method enabled to enrich this initial population with a set of 2,300 new entities, which illustrate the quality of Aleda's coverage. Although not fully automated, it allowed for a fast and efficient initialization of the KB with relevant and structured data. Future population will follow the same principles but will include an automatic entity linking module. This module will also use Aleda as an entity resource and thus link entity occurrences in news wires to Aleda; new entities can then be suggested as candidates to be integrated into the AFP KB, still using Aleda information related to Linked Data.

Conclusion and perspectives

We have sketched how we have extracted Aleda, a largescale entity database for French from freely available sources, namely the French Wikipedia and the GeoNames database. We have described and illustrated Aleda and compared it briefely to NLGbAse [START_REF] Charton | Nlgbase: a free linguistic resource for natural language processing systems[END_REF], another such database.

Apart from improving the accuracy of the extraction results, in particular concerning the "definition" associated with each entry, the perspectives of this work are threefold. First, we aim at increasing the granularity of the structured information embedded in Aleda, for example by introducing sub-types for entities other than location names. Second, we intend to put together information about locations coming from Wikipedia and from GeoNames. Finally, we have already started building Aleda for languages other than French.

Table 1

 1

	NLGbAse is not limited to NEs but provides
	a conceptual structure by grouping entities in main types
	(LOC, PERS, etc.) and subtypes (PERS.HUM for humans,
	LOC.ADMI for administrative locations, etc.). It contains
	1,085,812 entries, whose distribution accross main types is
	shown in

. Note that mentions in NLGbAse are titles of Wikipedia articles; for example, Paul Ier (pape) is a mention for the entity whose English standard name is Pope Paul I, although "(pape)" will probably never been found after Paul Ier in a real text. Another issue with NL-GbAse is the fact that almost one fifth of all entries have an unknown type (see Table

1

). Such entries sometimes correspond to entities which should have been typed (e.g., musée de l'érotisme which should have been typed as an organization), often to terms that are not NEs (e.g., notions such as

1

  An infobox is a fixed-format table added to articles to consistently present a summary of some unifying aspect that share sets of articles.

		#entities #mentions
	Person	272,507	966,793
	Location	515,242	522,728
	Organization	32,963	84,949
	Company	11,740	23,356
	Product	4,675	8,060
	Work	34,960	67,316
	total	872,087	1,673,202

Table 2

 2

	: Quantitative information about Aleda
	templates used in articles corresponding to entities can be
	easily associated with a unique entity type. A type such
	as Person, Location or Organization has thus been manu-
	ally assigned to the most frequent infobox templates (136
	infoboxes). This allowed us to assign a type to 202,571 ar-
	ticles, i.e., relevant entities. In a second step, we look for
	wikipedia categories associated with these already typed
	articles. For each of these categories occuring at least four
	times, we count the number of times they appear in articles
	of each type. If a category appears mostly within articles
	of one type, this type is associated with the category. This
	happened to 20,328 categories (e.g., Naissance en 1978 ap-
	pears in 2,777 articles, 1,777 of which have an infobox,
	which in all cases have been assigned the type "Person";

This process requires to filter out other article titles the disambiguation page is pointing to.

Only those entities where considered for the evaluation of entity coverage.

4 http://linkeddata.

org/ 5 The NP system is briefly described in (Stern and Sagot,

2010).[START_REF] Balasuriya | Named entity recognition in wikipedia[END_REF] Indeed, every Aleda entity is associated with a Wikipedia or GeoNames URI, which in turn links with other Linked Data knowledge such as DBPedia and the New York Times LOD (http://data.nytimes.com).

Acknowledgements

This work was funded by the French ANR national grant EDyLex (ANR-09-CORD-008).