Karën Fort

Adeline Nazarenko
email: adeline.nazarenko@lipn.univ-paris13.frkaren.fort

Claire Ris
email: claire.ris@inist.fr

Corpus Linguistics for the Annotation Manager

Hand crafted annotated corpora are acknowledged as critical elements for the Human Language Technologies but systems have to be trained on domain specific data to achieve a high level of performance. This is the reason why numerous annotation campaigns are launched. The role of the annotation manager consists in designing the annotation protocol, sometimes selecting the source data, hiring the required number of annotators with the adequate competences, writing the annotation guidelines, controlling the annotation process and delivering the resulting annotated corpus with the expected quality.

However, for a given task, the complexity of the annotation work seems to be highly dependent on the type of corpus to annotate. Since this affects both the cost and the quality of the annotation, it is an important issue to tackle for the annotation manager. This paper illustrates the role of corpus linguistics for the management of annotations through a specific annotation campaign. We show how the corpus characteristics affect all aspects of the annotation protocol: the design of the annotation guidelines, the selection of the a sub-corpus for training, the duration of the annotator's training, the complexity of the annotation formalism, the quality of the resulting annotation.

Introduction

The term "annotation" corresponds to the process of adding (ad-) an interpretation in the form of a note on a flow of data. This definition stems from and enlarges the one given in [START_REF] Leech | Corpus annotation: Linguistic information from computer text corpora[END_REF] as the interpretative dimension of the annotation is not limited to linguistic information or to any particular type of corpus, which can be composed of speech and texts, but also of video or images.

Since the beginning of the 90s and the seminal work on the Penn Treebank project [START_REF] Marcus | Building a large annotated corpus of English : The Penn Treebank[END_REF], there has been a large endeavor to develop various types of annotated corpora, which are used for collecting linguistic data and testing linguistic theories but also and probably now more often, for training, testing and evaluating Natural Language Processing (NLP) tools.

Such annotated corpora are usually developed through annotation tasks in which one or several human annotators are asked to encode their interpretation of a given corpus in the form of annotations that are attached to that corpus. However, such annotation tasks are complex to define. What kind of interpretation is required? for what purpose? How many annotations are needed? What type of corpus and which corpora have to be annotated? What quality of annotation is expected? Those are the typical questions that should be answered when defining an annotation task in order to 1) write down annotation guidelines that explain to the annotators what kind of interpretation they are expected to deliver, 2) choose an annotation tool, 3) determine how many annotators should work in parallel, and 4)the size of the corpus to be annotated.

Even if, to date, there is no stable, well-acknowledged methodology for designing such an annotation task, it is more and more evident that the annotation manager plays a central role in this process. The growing need for annotated corpora has given rise to a new job.

The annotation manager has a critical role in the design and success of the annotation tasks, for which s/he is responsible. S/He has to interact with the various actors involved in the annotation process, mainly the client, who asks for an annotated corpus, and the annotators, who annotate the corpus. S/He has to understand the client's needs, estimate the costs, write the annotation guidelines, hire and train the annotators and evaluate the quality of the annotations before they are delivered to the client.

Corpus linguistics is important for the management of annotations, not only for the selection of the corpus to annotate -quite often the sources to annotate are selected by the client and the annotation manager has no control on the corpus choice itself -but because the corpus imposes strong constraints on the annotation task, which cannot be defined independently of the availability, size, homogeneity and internal characteristics of the corpus. The annotation manager cannot take the responsibility of an annotation task without an in-depth analysis of the corpus to be annotated. This paper illustrates the role of corpus linguistics for the management of annotations through a specific annotation campaign. The task itself is introduced in Section 2 and the underlying corpus analysis is presented in Section 3. Section 4 explains what is the role of the annotation manager in this context and how much it relies on corpus analysis.

Football Matches Annotation Task

The annotation of football (soccer) matches reports is a task that was defined in the context of the Quaero program 1 in which various Quaero partners are involved (IRISA, LIPN, INIST, among others):

-The client who expressed the need for an annotated corpus is both the primary user and the provider of the corpus. 2 The source documents -the corpus -was provided by the same client with the summarization of matches reports and various other repurposing tasks as target applications.

-The annotations were made at INIST-CNRS. The campaign was organized there by an annotation manager (Karën Fort) with few expert annotators (3 at the beginning and then 2), using an annotation interface, Glozz (Widlocher et al., 2009).

-The a priori corpus analysis was made jointly by INIST and LIPN, and the a posteriori error analysis by INIST, LIPN and IRISA (Vincent Claveau).

Figure 1 shows the role of the annotated corpus in the summarization application. The goal is to produce summaries out of the football match reports that are given as input. In the training phase (bottom part of the figure), the summarization tool is given couples of reports and associated summaries and "learns" how to derive the latter from the former. In the exploitation phase (upper part of the figure), the resulting derivation rules are exploited to derive new summaries for unknown match reports. -12 football matches in French -12 written minutes (Web) -24 transcripts of the video commentaries (1 file per half-time) -Written minutes of 4 additional matches Despite the multimedia dimension of the source corpus, the annotation is defined as text-based. No access to source videos was given to the annotators since the final summarization tool would have to rely on texts only.

The source corpus also includes various types of matches as shown on Table 1. This diversity has a direct impact on names and denominations, which differ at the national and international level. For instance, out of context, the mention les joueurs de l'équipe de France / the French team players is difficult to interpret. One does not know if it refers to the French national team as such (at the international level) or to the French team players that are playing for a specific club. For instance, Anelka is playing both in the French Team and for Chelsea. A third important characteristic to take into account is the heterogeneity of the report size in the source corpus. This is reflected in the Table 2 and illustrated on Figure 2, which presents the same action in two different report types. In whole, the corpus is unbalanced: the minutes and transcripts respectively represent 16 % and 84 % of the total 247,955 tokens. A forth point to note is the variety of the commentary sources. Videos come from 5 different channels and are commented by 4 different teams. The minutes are typed by individual commentators in reverse chronological order (as in a blog). This heterogeneity is reflected in the quality and style of the resulting texts:

-There is a major contrast between the oral and written styles. The transcripts are verbose and include linguistic features that are typical of the oral such as hesitations, disfluencies, repetitions and errors (e.g. Oh là là. le contrôle. le contrôle, le contrôle. / Oops. control. control ... or Bosingwa , Obi Mikel ... Terry ... Malouda ...).

-There is also a contrast between the commentaries that are dialogues (de toutes les manières Jean-Michel ... / anyway Jean-Michel...) and the minutes that are monologues. -Finally, there are noticeable differences among commentators' styles, each commentator having his preferred denominations (Bordeaux, les girondins, les bordelais, les marines, all referring to the same Bordeaux team players) and some being more explicit than others (especially in the description of actions).

The Role of the Annotation Manager

After analyzing the corpus to annotate, the annotation manager has to set up the annotation campaign. S/He must precisely define the task and design annotation guidelines, analyze the task complexity and estimate its costs, select and train the annotators and finally evaluate and control the quality of the produced annotations. The example of the football matches campaign shows that several of these tasks directly rely on the initial corpus analysis.

Task and Guidelines Definition

In the context of the football matches campaign, the aim is to exploit the annotations for various re-purposing applications such as the production of match summaries, the alignment of commentaries on videos, a statistical analysis of the players' activity. In this context, the relevant information is the sequence of football actions and resulting scores (who does what at what time and for which result?) and the goal is to annotate it. The set of annotations made on a match report must form a summary of the match.

Finally, the annotation manager decided to ask for the following annotations:

Actors player, team, referee, assistant referee, coach, club president This task definition represents a trade-off between the richness of the annotation and the difficulty of the annotators' work. From the task definition, the annotation manager has to design the guidelines. Their role is to explain the task to the annotators and guide their work (what pieces of texts should be annotated and how?) but also to reduce the annotation errors and to standardize the annotations made by various annotators and over time. In the context of the football match campaign, it was decided to have a single guide and annotation task, despite the corpus heterogeneity. Since the source of reports is known and the training must be done separately for each type of sources, it would have been possible to carry out two different annotation campaigns. However, that would have been a significant additional burden for the annotators and a possible source of errors, if the same annotators had to tackle with two different but somehow similar annotation tasks. The manager therefore preferred the robustness of the guidelines to the accuracy of the annotations.

These decisions had a direct impact on the annotation model. It led to simplify it, so that annotators could annotate in the same way all the documents, while listing the specific cases. The annotation manager also decided to allow for commentaries in case the annotators were uncertain what or how to annotate.

The requirement of robustness imposed that football actions be annotated in the same way in video commentaries and in written minutes. However, video commentaries con- The ball goes out., The shot action is attached to the actor (Bosingwa). Another example is presented in Figure 3. If the actors themselves are omitted, the annotators are asked to annotate the action or relation marker and add a feature "missing actor" (source or/and target actor for relations). In La balle est repoussée de la tête / The ball is pushed away with the head , the intercept action is attached to the verb (pushed).

Cost Estimates

The annotation manager has to analyze the annotation task so as to precisely estimate its cost, potentially redefine it to reduce it and provide the annotators with the appropriate assisting tools, such as an automatic pre-annotation or a convenient annotation tool.

To do so, we propose to use five complexity indicators [START_REF] Fort | Manual Annotation of Corpora: Identifying the Difficulties to Reduce them[END_REF], impacting the final cost of the task:

1) The unit discrimination is defined as the ratio of units to annotate with regard to the total number of "annotatable" units, sparse annotation being difficult to produce.

2) The frontiers adjustment is defined with regard to a standard segmentation of the text and is related to the number of units which frontiers must be changed during the annotation process.

3) The annotation language expressiveness is related to the type of annotation language that is used (the tags can be types, relations or even 2nd order annotations).

4) The tagset dimension captures the fact that the annotation work is more difficult when the degree of liberty in tagging and the number of candidate tags is higher.

5) The degree of ambiguity gives an estimate of the number of ambiguous units that can be tagged in different ways.

The complexity of the football campaign annotation we present is mainly related to unit discrimination, annotation language expressiveness and ambiguity.

Unit Discrimination

Independently of the number of annotations to produce, sparse annotations are more difficult than others. Annotation is easier if all the units have to be annotated or if the relevant ones were pre-tagged, but this is not the case in the football task which has, on the contrary, a high level of discrimination. Since the annotation is a summary, only a small proportion of transcripts has to be annotated and this discrimination complexity is higher for video transcripts, which are verbose.

To ease the work of the annotators, the annotation manager decided to pre-annotate the corpus. Some actors (proper names of players and coaches) were thus automatically pre-annotated. Unfortunately, not all the actors could be pre-annotated: the names of referees and club presidents, as well as variant denominations remained unannotated. For instance, Fernando is pretagged but not the brazilian.

Annotation Language Expressiveness

The annotation task complexity depends on the type of information that the annotators have to add. The annotation is boolean if the units are simply marked as relevant or not. Type annotation refers to the cases where a simple tag is associated to units. Part of the annotations of the football campaign belong to that category (player/coach/referee/match location) even if multi-type annotations are also expected in the case, for instance, where a name refers both to a player and to the captain. The complexity is higher for relational annotations (X makes a foul on Y) and for meta annotation (annotation on an existing annotation).

To reduce the expressiveness of the annotation language in the football campaign, the annotation manager simplified the annotation task. Actions and relations were all annotated as types on the actors. Although this might appear counterintuitive at first, it eases the annotators' work. Once the annotators understood a football action, they were able to annotate its main actor, who is usually easy to identify (if not pre-annotated). The annotators do not have to localize the exact words that express the action or the target of the relation, which would be difficult, due to the elliptic style of the reports.

The annotation manager also asked the annotators to indicate the most dubious annotations, an important information for controlling the quality of the annotations (see Section 4.3), especially in complex and non-standard annotation campaigns as the football one. However, to keep the annotation language as simple as possible, it was decided that the uncertainty would be encoded as an annotation feature rather than as a metaannotation apposed on dubious annotations.

Degree of Ambiguity

The degree of ambiguity is the complexity indicator that is the most difficult to compute. One has to estimate how many alternative tags a given text unit may have. Even if it is not possible to produce precise figures, it is easy to understand that a high degree of ambiguity characterizes the football task. Since actors, actions and relations annotations are all attached to actors, it increases their ambiguity. In addition to that, many player and team denominations are ambiguous and ellipses even increase ambiguity.

The degree of ambiguity remains the most important factor of complexity in the football task. As a matter of fact, the choices made by the annotation manager in designing the guidelines aimed at reducing the discrimination and the expressiveness complexity, but led to increasing ambiguity, which annotators had to tackle.

Evaluation

The last role of the annotation manager is to control the quality of the annotations.

Protocol

The annotation manager organized the campaign in such a way that the annotators' training and learning curve, both fundamental to the annotation quality [START_REF] Marcus | Building a large annotated corpus of English : The Penn Treebank[END_REF][START_REF] Dandapat | Complex Linguistic Annotation -No Easy Way Out! A Case from Bangla and Hindi POS Labeling Tasks[END_REF], were taken into account.

The campaign encompassed three different phases. In the training phase, the annotators annotated a small set of documents. They got used to the corpus, task, guidelines and annotation tool. The annotation manager revised the guidelines taking into account the annotators' feedback, monitored the annotation quality and speed, which were increasing. In the football campaign, the training phase also led to eliminating one of the annotators (he/she was not reliable enough).

In the pre-campaign, the annotation manager dispatched the work to the annotators, who annotated the same sample of corpus, in order to compute the inter-annotator agreement. The annotation manager performed a detailed analysis of the disagreements, uncertainty features and annotators' feedback and revised the guidelines accordingly. In parallel, the annotation manager monitored the annotation speed, which was still increasing. Once it stabilized, the annotation manager put an end to the pre-campaign phase.

In the production phase, the annotation manager dispatched the work to annotators, who worked independently of each other. The annotation manager controlled the quality of the annotation.

Quality Evaluation

Following the methodology described in (BM et al., 2005), the inter-annotator agreement was computed early in the process (in the training phase) in order to assess the reliability of the annotations. The guidelines were then revised to reduce the disagreement sources. The intra-annotator agreement was also computed during the campaign to check the annotators' ability to reproduce the annotations [START_REF] Krippendorff | Content Analysis: An Introduction to Its Methodology, second edition[END_REF].

The coefficients used to compute the agreements are usually those of the kappa family (Cohen's or Carletta's, see [START_REF] Artstein | Inter-Coder Agreement for Computational Linguistics[END_REF] for more details), which take chance into account, as opposed to simpler measures like the F-measure. However, the computation of those coefficients requires to know the number of "annotatable" units, that can only be estimated for tasks like this [START_REF] Grouin | Proposal for an Extension of Traditional Named Entities: From Guidelines to Evaluation, an Overview[END_REF]. Other measures are also being considered, like the one provided in the annotation tool [START_REF] Mathet | Une approche holiste et unifiée de l'alignement et de la mesure d'accord inter-annotateurs[END_REF].

A detailed analysis of the obtained results should then be done, in order to identify the patterns of disagreements and correct the annotations and the guidelines. The annotators' feedback, given through uncertainty features or comments, should also be analyzed and taken into account.

Conclusion

The growing need for annotated corpora gave rise to a new job, the annotation manager and it is now a challenging task to set up a stable, well-acknowledged methodology to design such annotation tasks and, more generally, for the management of annotations.

This paper reports on an experiment where a heterogeneous corpus of football match reports had to be annotated in order to identify the key actions of the football players during the matches. This showed that corpus linguistics is important for the management of annotations, not only for the selection of the corpus to annotate -quite often the sources to annotate are selected by the client and the annotation manager has no control on the corpus choice itself -but because the corpus imposes strong constraints on the annotation task, which cannot be defined independently of the availability, size, homogeneity and internal characteristics of the corpus. The annotation manager cannot take the responsibility of an annotation task without an in-depth analysis of the corpus to be annotated.

Figure 1 .

 1 Figure 1. Role of the annotations for the training of a summarization tool

  Figure 2. Comparison of the size of minutes and transcripts.

Figure 3 .

 3 Figure 3. Example of action annotation with a missing actor

Table 1 .

 1 Composition of the football corpus wrt. match types

		National Level	International Level
	Match	French national Other national	Club teams	National teams
	types	level	level
		(Ligue 1)	(Bundesliga) (Champions' League) (World Cup)
	Proportion	25%	8%	50%	17%
	Example Bordeaux-Vannes WerderBreme	Munich-Lyon	France-Serbie
			-Hambourg

Table 2 .

 2 Contrast between minutes and transcripts, the two types of football reports

	Types of report Minimal length Maximal length	Total	Proportion
	Minutes	1,116 tokens	3,627 tokens	38,919 tokens	16%
	Transcripts	6,020 tokens	11,110 tokens 209,036 tokens	84%

  Locations match location (town), field area Chronology time in match Actions center, center attempt, direct free kick, indirect free kick, corner, penalty, dribble, foul, offside, score goal, miss goal, stop goal, interception, have ball, yellow card, red card, warning, audience action Relations pass, pass attempt, combination, tackle foul, replacement, foul

Corpus AnalysisIn this task, the annotation manager had no control on the selection and composition of the source corpus that were made by the client. However, it is essential to take corpus characteristics into account in the definition of the annotation task.

http://quaero.org/

Acknowledgments

We want to thank Alain Zérouki from INIST-CNRS, for his hard work as annotator and Vincent Claveau, for hir implication and help during and after the campaign. This work was realized as part of the Quaero Programme 3 , funded by OSEO, French State agency for innovation.

Match


Julien Jourde

Alain-Pierre Manine

Philippe Veber

Karën Fort
email: karen.fort@inist.fr

Robert Bossy

Erick Alphonse

Philippe Bessières

BioNLP Shared Task 2011 -Bacteria Gene Interactions and Renaming

We present two related tasks of the BioNLP Shared Tasks 2011: Bacteria Gene Renaming (Rename) and Bacteria Gene Interactions (GI). We detail the objectives, the corpus specification, the evaluation metrics, and we summarize the participants' results. Both issued from PubMed scientific literature abstracts, the Rename task aims at extracting gene name synonyms, and the GI task aims at extracting genic interaction events, mainly about gene transcriptional regulations in bacteria.

Introduction

The extraction of biological events from scientific literature is the most popular task in Information Extraction (IE) challenges applied to molecular biology, such as in LLL [START_REF] Nédellec | Learning Language in Logic -Genic Interaction Extraction Challenge[END_REF], BioCreative Protein-Protein Interaction Task [START_REF] Krallinger | Overview of the protein-protein interaction annotation extraction task of BioCreative II[END_REF], or BioNLP [START_REF] Demner-Fushman | Themes in biomedical natural language processing: BioNLP08[END_REF]. Since the BioNLP 2009 shared task [START_REF] Kim | Overview of BioNLP'09 shared task on event extraction[END_REF], this field has evolved from the extraction of a unique binary interaction relation between proteins and/or genes towards a broader acceptation of biological events including localization and transformation [START_REF] Kim | Corpus annotation for mining biomedical events from literature[END_REF]. In the same way, the tasks Bacteria Gene Interactions and Bacteria Gene Renaming deal with the extraction of various molecular events capturing the mechanisms relevant to gene regulation in prokaryotes. The study of bacteria has numerous applications for health, food and industry, and overall, they are considered as organisms of choice for the recent integrative approaches in systems biology, because of their relative simplicity.

Compared to eukaryotes, they allow easier and more in-depth analysis of biological functions and of their related molecular mechanisms.

Processing literature on bacteria raises linguistic and semantic specificities that impact text analysis. First of all, gene renaming is a frequent phenomenon, especially for model bacteria. Hence, the abundance of gene synonyms that are not morphological variants is high compared to eukaryotes. The history of bacterial gene naming has led to drastic amounts of homonyms and synonyms which are often missing (or worse, erroneous) in gene databases. In particular, they often omit old gene names that are no longer used in new publications, but that are critical for exhaustive bibliography search. Polysemy makes the situation even worse, as old names frequently happen to be reused to denote different genes. A correct and complete gene synonym table is crucial to biology studies, for instance when integrating large scale experimental data using distinct nomenclatures. Indeed this information can save a lot of bibliographic research time. The Rename Task is a new task in text-mining for biology that aims at extracting explicit mentions of renaming relations. It is a critical step in gene name normalization that is needed for further extraction of biological events such as genic interactions.

Regarding stylistics, gene and protein interactions are not formulated in the same way for eukaryotes and prokaryotes. Descriptions of interactions and regulations in bacteria include more knowledge about their molecular actors and mechanisms, compared to the literature on eukaryotes. Typically in bacteria literature, the genic regulations are more likely expressed by direct binding of the protein, while in eukaryote literature, non-genic agents related to environmental conditions are much more frequent. The bacteria GI Task is based on [START_REF] Manine | Extraction of genic interactions with the recursive logical theory of an ontology[END_REF] which is a semantic re-annotation of the LLL challenge corpus [START_REF] Nédellec | Learning Language in Logic -Genic Interaction Extraction Challenge[END_REF], where the description of the GI events in a fine-grained representation includes the distinction between expression, transcription and other action events, as well as different transcription controls (e.g. regulon membership, promoter binding). The entities are not only protein agent and gene target but extend to families, complexes and DNA sites (binding sites, promoters) in order to better capture the complexity of the regulation at a molecular level. The task consists in relating the entities with the relevant relations.

Rename Task Description

The goal of the Rename task is illustrated by Figure 1. It consists in predicting renaming relations between text-bound gene names given as input. The only type of event is Renaming where both arguments are of type Gene. The event is directed, the former and the new names are distinguished. Genes and proteins were not distinguished because of the high frequency of metonymy in renaming events. The relation to predict between genes is a Renaming of a former gene name into a new one. In the example of Figure 1, YtaA, YvdP and YnzH are the former names of three proteins renamed CotI, CotQ and CotU, respectively.

Rename Task corpus

The Rename Task corpus is a set of 1,836 PubMed references of bacterial genetic and genomic studies, including title and abstract. A first set of 23,000 documents was retrieved, identifying the presence of the bacterium Bacillus subtilis in the text and/or in the MeSH terms. B. subtilis documents are particularly rich in renaming mentions. Many genes were re-named in the middle of the nineties, so that the new names matched those of the Escherichia coli homologues. The 1,843 documents the most susceptible to mention renaming were automatically filtered according to two non exclusive criteria:

1. Either the document mentions at least two gene synonyms as recorded in the fusion of seven B. subtilis gene nomenclatures. This led to a set of 703 documents.

2. Or the document contains a renaming expression from a list that we manually designed and tested (e.g. rename, also known as). It is an extension of a previous work by [START_REF] Weissenbacher | La relation de synonymie en Génomique[END_REF]. A total of 1,140 new documents not included in the first set match this criteria.

About 70% of the documents (1,146) were kept in the training data set. The rest was split into the development and test sets, containing 246 and 252 documents respectively. Table 1 gives the distribution of genes and renaming relations per corpus. Gene names were automatically annotated in the documents with the nomenclature of B. subtilis. Gene names involved in renaming acts were manually curated. Among the 21,878 gene mentions in the three corpus, 680 unique names are involved in renaming relations which represents 891 occurrences of genes.

Rename Task annotation and guidelines

Annotation procedure The corpus was annotated in a joint effort of MIG/INRA and INIST/CNRS. The reference annotation of the Rename Task corpus was done in two steps, a first annotation step by science information professionals of INIST with MIG initial specifications, a second checking step by people at MIG. Two annotators and a project manager were in charge of the task at INIST. The documents were annotated using the Cadixe editor 1 . We provided to them detailed annotation guidelines that were largely modified in the process. A subset of 100 documents from the first set of 703 was annotated as a training session. This step was used to refine the guidelines according to the methodology described in [START_REF] Bonneau-Maynard | Semantic annotation of the French Media Dialog Corpus[END_REF]. Several inter-annotator agreements coefficients were computed to measure the discrepancy between annotators (Fort et al., 2009). With a kappa and pi scores (for more details on those, see [START_REF] Artstein | Inter-coder agreement for Computational Linguistics[END_REF]), the results can be considered satisfactory.

The manual analysis of the 18 discrepancies led to enrich the annotation guidelines. The first hundreds of documents of the second set did not mention any renaming, leading to concentrate the annotation efforts on the first set. These documents actually contained renamings, but nearly exclusively concerning other kinds of biological entities (protein domains, molecules, cellular ultrastructures, etc.).

Guidelines

In order to simplify the task, only short names of gene/protein/groups in B. subtilis were considered. Naming conventions set short names of four letters long with an upper case letter at the end for all genes (e.g. gerE) and the same names with the upper case of the initial letter (e.g. GerE) and long names for the proteins (e.g. Spore germination protein gerE). But many irregular gene names exist (e.g. tuf), which are considered as well.

It also happens that gene or protein name lists are abbreviated by factorization to form a sequence. For instance queCDEF is the abbreviation of the list of gene names queC, queD, queE and queF. Such aggregations are acceptable gene names as well. In any case, these details were not needed by the task participants since the corpus was provided with tagged gene names.

Most renaming relations involve couples of the same type, genes, proteins or aggregations. Only 18 relations link mixed couples of genes and proteins. In case of ambiguity, annotators would consult international gene databases and an internal INRA database to help them determine whether a given couple of names were actually synonyms.

Multiple occurrences of the same renaming relation were annotated independently, and had to be predicted. The renaming pairs are directed, the former and the new forms have to be distinguished.

When the renaming order was not explicit in the document, the rule was to annotate by default the first member of the couple as the new form, and the second one as the former form. Figure 2 presents the most common forms of renaming. Revised annotations INIST annotations were systematically checked by two experts in Bioinformatics from INRA. Mainly, encoding relations (e.g. the gene encoding sigma K (sigK)) that are not renaming cases were purged. Given the number of ambiguous annotations, we designed a detailed typology in order to justify acceptance or rejection decisions in seven different sub-cases hereafter presented. Three positive relations figure in Table 2, where the underlined names are the former names and the framed names are the new ones. Explicit renaming relations occur in 261 sentences, synonymylike relations in 349 sentences, biological proofbased relations in 76 sentences.

Explicit renaming relation is the easiest positive case to identify. In the example, the aggregation of gene names ykvJKLM is clearly renamed by the authors as queCDEF. Although the four genes are con-Explicit renaming PMID 15767583: Genetic analysis of ykvJKLM mu- tants in Acinetobacter confirmed that each was essential for queuosine biosynthesis, and the genes were renamed queCDEF . Implicit renaming PMID 8002615: Analysis of a suppressor mutation ssb ( kinC ) of sur0B20 (spo0A) mutation in Bacillus subtilis reveals that kinC encodes a histidine protein kinase. Biological proof PMID 1744050: DNA sequencing established that spoIIIF and spoVB are a single monocistronic locus encoding a 518-amino-acid polypeptide with features of an integral membrane protein. catenated, there is no evidence mentioned of them acting as an operon. Furthermore, despite the context involving mutants of Acinetobacter, the aggregation belongs correctly to B. subtilis.

Implicit renaming is an asymmetric relation since one of the synonyms is intended to replace the other one in future uses. The example presents two renaming relations between former names ssb and spo0A, and new names kinC and sur0B20, respectively. The renaming relation between ssb and kinC has a different orientation due to additional information in the reference. Like in the preceding example, the renaming is a consequence of a genetic mutation experiment. Mutation names represent an important transversal issue that is discussed below.

Biological proof is a renaming relation induced by an explicit scientific conclusion while the renaming is not, as in the example where experiments reveal that two loci spoIIIF and spoVB are in fact the same one and then become synonyms. Terms such as "allelic to" or "identical to" usually qualify such conclusions. Predicting biological proof-based relations requires some biological modeling.

The next three cases are negative (Table 3). Underlined gene and protein names are involved in a relation which is not a renaming relation.

Protein encoding relation occurs between a gene and the protein it codes for. Some mentions may look like renaming relations. The example presents the gene yeaC coding for MoxR. No member of the couple is expected to replace the other one.

Homology measures the similarity between gene or protein sequences. Most of the homology mentions involve genes or proteins from different species (orthologues). The others compare known gene or protein sequences of the same species (paralogues). This may be misleading since the similarity mention may look like biological proof-based relations, as between ParA and Soj in Table 3.

Operon, regulon or family renaming involves objects that may look like genes, proteins or simple aggregations of gene or protein names but that are perceptibly different. The objects represent more than one gene or protein and the renaming does not necessarily affect all of them. More problematic, their name may be the same as one of the genes or proteins they contain, as in the example where sigA and rpoD are operons but are also known as gene names. Here, sigA (and so rpoD) represents at least two different genes. For the sake of clarity, operons, regulons and families are rejected, even if all the genes are clearly named, as in an aggregation.

The last point concerns mutation which are frequent in Microbiology for revealing gene phenotypes. They carry information about the original gene names (e.g., rvtA11 is a mutant name created by adding 11 to rvtA). But partial names cannot be partially annotated, that is to say, the original part (rvtA) should not be annotated in the mutation name (rvtA11). Most of these names are local names, and should not be annotated because of their restricted scope. It may happen so that the mutation name is registered as a synonym in several international databases. To avoid inconsistencies, all renamings involving a mutation referenced in a database were accepted, and only biological proof-based and explicit renamings involving a strict non-null unreferenced mutation (a null mutation corresponds to a total suppression of a gene) were accepted.

Rename Task evaluation procedure

The evaluation of the Rename task is given in terms of recall, precision and F-score of renaming relations. Two set of scores are given: the first set is computed by enforcing strict direction of renaming relations, the second set is computed with relaxed direction. Since the relaxed score takes into account renaming relations even if the arguments are inverted, it will necessarily be greater or equal than the strict score. The participant score is the relaxed score, the strict score is given for information. Relaxed scores are informative with respect to the ap-plication goal. The motivation of the Rename task is to keep bacteria gene synonyms tables up to date. The choice of the canonical name among synonyms for denoting a gene is done by the bacteriology community, and it may be independent of the anteriority or novelty of the name. The annotation of the reference corpus showed that the direction was not always decidable, even for a human reader. Thus, it would have been unfair to evaluate systems on the basis of unsure information.

Results of the Rename Task participants

Final submissions were received from three teams, the University of Turku (Uturku), the University of Concordia (Concordia) and the Bibliome team from MIG/INRA. Their results are summarized in Table 4. The ranking order is given by the overall F-score for relations with relaxed argument order.

Team

Prec Uturku achieved the best F-score with a very high precision and a high recall. Concordia achieved the second F-score with balanced precisions and recalls. Bibliome is five points behind with a better recall but much lower precision. Both UTurku and Concordia predictions rely on dependencies (Charniak-Johnson and Stanford respectively, using McClosky model), whereas Bibliome predictions rely on bag of words. This demonstrates the high value of dependency parsing for this task, in particular for the precision of predictions. We notice that UTurku system uses machine learning (SVM) and Concordia uses rules based on trigger words. The good results of UTurku confirms the hypothesis that gene renaming citations are highly regular in scientific literature. The most frequently missed renamings belong to the Biological Proof category (see Table 2). This is expected because the renaming is formulated as a reasoning where the conclusion is only implicit.

Discussion

The very high score of Uturku method leads us to conclude that the task can be considered as solved by a linguistic-based approach. Whereas Bibliome used an extensive nomenclature considered as exhaustive and sentence filtering using a SVM, Uturku used only two nomenclatures in synergy but with more sophisticated linguistic-based methods, in particular syntactic analyses. Bibliome methods showed that a too high dependence to nomenclatures may decrease scores if they contain compromised data. However, the use of an extensive nomenclature as done by Bibliome may complement Uturku approach and improve recall. It is also interesting that both systems do not manage renamings crossing sentence boundaries.

The good results of the renaming task will be exploited to keep synonym gene lists up to date with extensive bibliography mining. In particular this will contribute to enriching SubtiWiki, a collaborative annotation effort on B. subtilis [START_REF] Flórez | A community-curated consensual annotation that is continuously updated: The Bacillus subtilis centred wiki SubtiWiki[END_REF][START_REF] Lammers | Connecting parts with processes: SubtiWiki and Subti-Pathways integrate gene and pathway annotation for Bacillus subtilis[END_REF].

Gene Interactions Task description

The goal of the Bacteria GI Task is illustrated by Figure 3. The genes cotB and cotC are related to their two promoters, not named here, by the relation PromoterOf. The protein GerE is related to these promoters by the relation BindTo. As a consequence, GerE is related to cotB and cotC by an Interaction relation. According to (Kim et al., 2008), the need to define specialized relations replacing one unique and general interaction relation was raised in [START_REF] Manine | Learning ontological rules to extract multiple relations of genic interactions from text[END_REF] for extracting genic interactions from text. An ontology describes relations and entities [START_REF] Manine | Information extraction as an ontology population task and its application to genic interactions[END_REF] catching a model of gene transcription to which biologists implicitly refer in their publications. Therefore, the ontology is mainly oriented towards the description of a structural model of genes, with molecular mechanisms of their transcription and associated regulations.

The corpus roughly contains three kinds of genic interaction mentions, namely regulations, regulon membership and binding. The first case corresponds to interactions the mechanism of which is not explicitly given in the text. The mention only tells that the transcription of a given gene is influenced by a given protein, either positively (activation), negatively (inhibition) or in an unspecified way. The second kind of genic interaction mention (regulon membership) basically conveys the same information, using the regulon term/concept. The regulon of a gene is the set of genes that it controls. In that case, the interaction is expressed by saying that a gene is a member of some regulon. The third and last kind of mention provides with more mechanistic details on a regulation, since it describes the binding of a protein near the promoter of a target gene. This motivates the introduction of Promoter and Site entities, which correspond to DNA regions. It is thus possible to extract the architecture of a regulatory DNA region, linking a protein agent to its gene target (see Figure 3). The set of entity types is divided into two main groups, namely 10 genic entities and 3 kinds of action (Table 5). Genic entities represent biological objects like a gene, a group of genes or a gene product. In particular, a GeneComplex annotation corresponds to an operon, which is a group of genes that are contiguous in the genome and under the control of the same promoter. The annotation GeneFamily is used to denote either genes involved in the same biological function or genes with sequence homologies. More importantly, PolymeraseComplex annotations correspond to the protein complex that is responsible for the transcription of genes. This complex includes several subunits (components), combined with a sigma factor, that recognizes specific promoters on the DNA sequence.

The second group of entities are phrases expressing either molecular processes (e.g. sequestration, dephosphorylation, etc.) or the molecular state of the bacteria (e.g. presence, activity or level of a protein). They represent some kind of action that can be performed on a genic entity. Note that transcription and expression events were tagged as specific actions, because they play a specific part in certain relations (see below).

The annotation of entities and actions was provided to the participants, and the task consisted in extracting the relations listed in Table 6 The relations are binary and directed, and rely the entities defined above. The three kinds of interactions are represented with an Interaction annotation, linking an agent to its target. The other relations provide additional details on the regulation, like elementary components involved in the reaction (sites, promoters) and contextual information (mainly provided by the ActionTarget relations). A formal definition of relations and relation argument types can be found on the Bacteria GI Task Web page.

Bacteria Gene Interactions corpus

The source of the Bacteria GI Task corpus is a set of PubMed abstracts mainly dealing with the tran-scription of genes in Bacillus subtilis. The semantic annotation, derived from the ontology of [START_REF] Manine | Information extraction as an ontology population task and its application to genic interactions[END_REF], contains 10 molecular entities, 3 different actions, and 10 specialized relations. This is applied to 162 sentences from the LLL set [START_REF] Nédellec | Learning Language in Logic -Genic Interaction Extraction Challenge[END_REF], which are provided with manually checked linguistic annotations (segmentation, lemmatization, syntactic dependencies). The corpus was split into 105 sentences for training, 15 for development and 42 for test. Table 7 gives the distribution of the entities and actions per corpus and Table 8 gives the distribution of the relations per corpus.

Annotation procedures and guidelines

The semantic annotation scheme was developed by two annotators through a series of independent annotations of the corpus, followed by reconciliation steps, which could involve concerted modifications [START_REF] Manine | Extraction of genic interactions with the recursive logical theory of an ontology[END_REF] corpus was reviewed and the annotation simplified to make it more appropriate to the contest. The final annotation contains 748 relations distributed in nine categories, 146 of them belonging to the test set.

The annotation scheme was generally well suited to accurately represent the meaning of the sentences in the corpus, with one notable exception. In the corpus, there is a common phrasing telling that a protein P regulates the transcription of a gene G by a given sigma factor S. In that case, the only annotated interactions are between the couples (P, G) and (S, G). This representation is not completely satisfactory, and a ternary relation involving P, S and G would have been more adequate.

Additional specific rules were needed to cope with linguistic issues. First, when the argument of a relation had coreferences, the relation was repeated for each maximally precise coreference of the argument. Second, in case of a conjunction like "sig-maA and sigmaX holoenzymes", there should ideally be two entities (namely "sigmaA holoenzyme" and "sigmaX holoenzyme"); however, this is not easy to represent using the BioNLP format. In this situation, we grouped the two entities into a single one. These cases were rare and unlikely affected the feasibility of the task, since entities were provided in the test set.

Gene Interactions evaluation procedure

The training and development corpora with the reference annotations were made available to participants by December, 1st on the BioNLP shared Task pages together with evaluation software. The test corpus with the entity annotations has been made available by March, 1st. The participants sent the predicted annotations to the BioNLP shared Task organizers by March, 10th. The evaluation results were computed and provided to the participants and on the Web site the same day. The participants are evaluated and ranked according to two scores: Fscore for all event types together, and F-score for the Interaction event type. In order for a predicted event to count as a hit, both arguments must be the same as in the reference in the right order and the event type must be the same as in the reference.

Results of GI Task participants

There was only one participant, whose results are shown in Tables 9 and10. Some relations were not significantly represented in the test set and thus the corresponding results should be considered with caution. This is the case for RegulonMember and TranscriptionFrom, only represented two times each in the test. The lowest recall, 17%, obtained for the SiteOf relation is explained by its low representation in the corpus: most of the test errors come from a difficult sentence with coreferences.

The recall of 56% for the Interaction relation certainly illustrates the heterogeneity of this category, gathering mentions of interactions at large, as well as precise descriptions of gene regulations. For instance, Figure 4 shows a complex instance where all of the interactions were missed. Surprisingly, we also found false negatives in rather trivial examples ("ykuD

Discussion

The GI corpus was previously used in a relation extraction work [START_REF] Manine | Learning ontological rules to extract multiple relations of genic interactions from text[END_REF] based on Inductive Logic Programming [START_REF] Muggleton | Inductive Logic Programming: Theory and methods[END_REF]. However a direct comparison of the results is not appropriate here since the annotations were partially revised, and the evaluation setting was different (leave-one-out in Manine's work, test set in the challenge). Nevertheless, we note similar tendencies if we compare relative results between relations. In particular, it was also found in Manine's paper that SiteOf, TranscriptionBy and Interaction are the most difficult relations to extract. It is also worth to mention that both approaches rely on syntactic dependencies, and use the curated dependencies provided in the corpus. Interestingly, the approach by the University of Turku reports a slightly lower F-measure with dependencies calculated by the Charniak parser (about 1%, personal communication). This information is especially important in order to consider a production setting.

Conclusion

The quality of results for both challenges suggests that current methods are mature enough to be used in semi-automatic strategies for genome annotation, where they could efficiently assist biological experts involved in collaborative annotation efforts [START_REF] Lammers | Connecting parts with processes: SubtiWiki and Subti-Pathways integrate gene and pathway annotation for Bacillus subtilis[END_REF]. However, the false positive rate, notably for the Interaction relation, is still too high for the extraction results to be used as a reliable source of information without a curation step.

Figure 1 :

 1 Figure 1: Examples of relations to be extracted.

Figure 2 :

 2 Figure 2: Common types of relations to be extracted.

  Protein encodingPMID 8969499: The putative products of ORFs yeaB (Czd protein), yeaC (MoxR), yebA (CNG-channel and cGMP-channel proteins from eukaryotes), Genetic homology PMID 10619015: Dynamic movement of the ParA- like Soj protein of B. subtilis and its dual role in nucleoid organization and developmental regulation. Operon | Regulon | Family PMID 3127379: Three promoters direct transcription of the sigA (rpoD) operon in Bacillus subtilis.

Figure 3 :

 3 Figure 3: Examples of relations to be extracted.

Figure 4 :

 4 Figure 4: Examples of three missed interactions.

Table 1 :

 1 Rename Task corpus content.

		Training + Dev.	Test
	Documents	(1,146 + 246) 1,392	252 (15%)
	Gene names	18,503 3,375 (15%)
	Renamings	373	88 (24%)

Table 2 :

 2 Positive examples of the Rename Task.

Table 3 :

 3 Negative examples of the Rename Task.

Table 4 :

 4 Participant scores at the Rename Task.

		. Recall F-score
	Univ. of Turku	95.9	79.6	87.0
	Concordia Univ. 74.4	65.9	69.9
	INRA	57.0	73.9	64.4

Table 5 :

 5 . List of molecular entities and actions in GI.

	Name	Example
	Gene	cotA
	GeneComplex	sigX-ypuN
	GeneFamily	class III heat shock genes
	GeneProduct	yvyD gene product
	Protein	CotA
	PolymeraseComplex SigK RNA polymerase
	ProteinFamily	DNA-binding protein
	Site	upstream site
	Promoter	promoter regions
	Regulon	regulon
	Action	activity | level | presence
	Expression	expression
	Transcription	transcription
	Name	Example
	ActionTarget	expression of yvyD
	Interaction	ComK negatively regulates
		degR expression
	RegulonDependence	sigmaB regulon
	RegulonMember	yvyD is member of sigmaB
		regulon
	BindTo	GerE adheres to the pro-
		moter
	SiteOf	-35 sequence of the pro-
		moter
	PromoterOf	the araE promoter
	PromoterDependence GerE-controlled promoter
	TranscriptionFrom	transcription from the up-
		stream site
	TranscriptionBy	transcription of cotD by
		sigmaK RNA polymerase

Table 6 :

 6 List of relations in GI.

Table 7 :

 7 . As a third and final stage, the Distribution of entities and actions in GI.

	Entity or action	Train. + Dev. Test
	Documents	(105+15) 120	42
	Protein	219	85
	Gene	173	56
	Transcription	53	21
	Promoter	49	10
	Action	45	22
	PolymeraseComplex	43	14
	Expression	29	6
	Site	22	8
	GeneComplex	19	4
	ProteinFamily	12	3
	Regulon	11	2
	GeneProduct	10	3
	GeneFamily	6	5
	Relation	Train. + Dev. Test
	Interaction	208	64
	ActionTarget	173	47
	PromoterOf	44	8
	BindTo	39	4
	PromoterDependence	36	4
	TranscriptionBy	36	8
	SiteOf	23	6
	RegulonMember	17	2
	TranscriptionFrom	14	2
	RegulonDependence	12	1

Table 8 :

 8 Distribution of relations in GI.

Table 9 :

 9 was transcribed by SigK RNA polymerase from T4 of sporulation."). Uturku used an SVM- based approach for extraction, and it is thus delicate to account for the false negatives in a simple and concise way. University of Turku global scores.

	Event	U. Turku scores
	Global Precision		85
	Global Recall		71
	Global F-score		77
	Interaction Precision	75
	Interaction Recall		56
	Interaction F-score		64
	Event	Prec. Rec. F-score
	Global	85	71	77
	ActionTarget	94	92	93
	BindTo	75	75	75
	Interaction	75	56	64
	PromoterDependence	100 100	100
	PromoterOf	100 100	100
	RegulonDependence	100 100	100
	RegulonMember	100	50	67
	SiteOf	100	17	29
	TranscriptionBy	67	50	57
	TranscriptionFrom	100 100	100

Table 10 :

 10 University of Turku scores for each relation.

http://caderige.imag.fr/Articles/ CADIXEXML-Annotation.pdf

Acknowlegments

We thank Franc ¸oise Tisserand and Bernard Talercio (INIST) for their work on the Rename corpus, and the QUAERO Programme funded by OSEO (French agency for innovation) for its support.