Emanuela Boros
email: emanuela.boros@univ-lr.fr

Romaric Besançon
email: romaric.besancon@cea.fr

Olivier Ferret
email: olivier.ferret@cea.fr

Brigitte Grau
email: brigitte.grau@limsi.fr

The Importance of Character-Level Information in an Event Detection Model

Keywords: Information extraction, Events, Word embeddings

This paper tackles the task of event detection that aims at identifying and categorizing event mentions in texts. One of the difficulties of this task is the problem of event mentions corresponding to misspelled, custom, or out-of-vocabulary words. To analyze the impact of character-level features, we propose to integrate character embeddings, that can capture morphological and shape information about words, to a convolutional model for event detection. More precisely, we evaluate two strategies for performing such integration and show that a late fusion approach outperforms both an early fusion approach and models integrating character or subword information such as ELMo or BERT.

Introduction

In this article, we concentrate more specifically on event detection, which implies identifying instances of specified types of events in a text. The notion of event in our work is classically defined as something that happens and covers a wide spectrum, from terrorist attacks to births or nominations. The instances of these events in texts, which are called event mentions or event triggers, are annotated as words or phrases that evoke a reference type of events. The most successful approaches developed for achieving this task are currently based on neural models, which have been intensively studied to overcome fundamental limitations, specifically the complex choice of features [START_REF] Chen | Event extraction via dynamic multi-pooling convolutional neural networks[END_REF][START_REF] Nguyen | Event Detection and Domain Adaptation with Convolutional Neural Networks[END_REF][START_REF] Nguyen | Joint event extraction via recurrent neural networks[END_REF][START_REF] Nguyen | A two-stage approach for extending event detection to new types via neural networks[END_REF][START_REF] Feng | A language-independent neural network for event detection[END_REF][START_REF] Zhang | Joint entity and event extraction with generative adversarial imitation learning[END_REF][START_REF] Nguyen | Graph Convolutional Networks With Argument-Aware Pooling for Event Detection[END_REF]. All these proposed models based on Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or even Graph Neural Networks (GNNs) rely on word embeddings, a general distributed word representation that is produced by training a deep learning model on a large unlabeled dataset. Consequently, word embeddings This work was partly supported by the European Union's Horizon 2020 research and innovation program under grants 770299 (NewsEye) and 825153 (Embeddia).

replace the hard matches of words in the feature-based approaches with the soft matches of continuous word vectors. Hence, compared to previous rule-based or machine learning-based approaches, neural models are supposed to be less sensitive to the problem of unseen triggers since the distributed representations of words they exploit can account for the similarity between words.

However, this capacity may vary depending on the reasons why a trigger was not seen during the training of a model. We illustrate these different cases on the ACE 2005 dataset 4 , a standard corpus used for evaluating event detection. An unseen trigger may be a morphological variant of a trigger already seen in the training set. For instance, torturing is not present in the training data but is a variant of torture and can be considered as a trigger for the same type of events, namely Life.Injure. Moreover, torturing is likely to be present among general pre-trained word embeddings and if so, a neural event extraction model is likely to successfully detect this trigger. The situation may be different when a trigger is absent from the training data because it corresponds to a misspelled version of a reference trigger. For instance, aquitted is part of the ACE 2005 test dataset for referring to a Justice.Sentence event while only acquitted, the correct form for that word, is present in the training data. In that case, we cannot assume that the unseen word is part of general word embeddings and as a consequence, has little chance to be detected as a trigger for a Justice.Sentence event.

From a more general perspective, the problem of missing word embeddings when using pre-trained models in the context of event extraction is not marginal. The ACE 2005 dataset, for instance, covers the most common events of national and international news (from a variety of sources selected from broadcast news programs, newspapers, news reports, internet sources, or transcribed audio) and thus, it contains different types of discourse, professional or noisy discussions prone to the presence of mistakes in spelling and custom words. As a result, in this dataset, 14.8% of the words are not part of the pre-trained embeddings provided by Google, trained with word2vec on Google News [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF], 1.5% for the GloVe embeddings [START_REF] Pennington | Glove: Global Vectors for Word Representation[END_REF], and 4.5% for the fastText embeddings [START_REF] Joulin | Bag of Tricks for Efficient Text Classification[END_REF]. Different strategies were proposed and implemented for dealing with the issue of missing words in neural language models. For static word embeddings, fastText relies on a representation of words based on n-grams of characters. For contextual models, ELMo [START_REF] Peters | Deep Contextualized Word Representations[END_REF] exploits a character-based representation built with a CNN while BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] adopts a mixed strategy based on subwords, called wordpieces, where a word is split into subwords when it is not part of a predefined and restricted vocabulary [START_REF] Luong | Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models[END_REF][START_REF] Kim | Character-Aware Neural Language Models[END_REF][START_REF] Jozefowicz | Exploring the limits of language modeling[END_REF]. However, while BERT seems to be an interesting option for a large number of tasks in Natural Language Processing, its ability to handle noisy inputs is still an open question [START_REF] Sun | Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT[END_REF] or at least requires the addition of complementary methods [START_REF] Muller | Enhancing BERT for Lexical Normalization[END_REF]. This limitation may result from the dependence of BERT on a vocabulary. The alternative is to rely, as ELMo for instance, on a character model in which all words, including words with abnormal character combinations and misspellings, are processed similarly. Some researchers studied the application of CNNs to characters. For antispam filtering, the use of character-level n-grams was already experimented out of the context of deep learning models [START_REF] Kanaris | Words versus character n-grams for anti-spam filtering[END_REF]. In [START_REF] Santos | Learning character-level representations for part-ofspeech tagging[END_REF], character-level embeddings were automatically learned and joined with pre-trained word embeddings in a CNN-based model for Part-of-Speech tagging. This architecture was also used for improving the performance of a Named Entity Recognition (NER) system in [START_REF] Santos | Boosting Named Entity Recognition with Neural Character Embeddings[END_REF]. While character models have been used with success in several contexts for tackling the absence of pre-trained embeddings for all words, the use of CNNs to learn directly from characters was also investigated, without the need for any pre-trained embeddings [START_REF] Zhang | Character-level convolutional networks for text classification[END_REF]. Notably, the authors use a relatively deep network and apply it to sentiment analysis and text classification tasks. The application of character-level convolutions to language modeling was explored in [START_REF] Kim | Character-Aware Neural Language Models[END_REF] by using the output of a character-level CNN as the input to a Long Short Term Memory (LSTM) network at each time step. The same model is easily applied to various languages. However, the choice of CNN-based or LSTM-based character-level word embeddings did not affect the performance significantly [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF][START_REF] Ma | End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF[END_REF].

Our contributions in this article are more particularly focused on the integration of character-level features in event detection models for addressing the issue of unknown words. More specifically, we show that an event detection model exploiting a character-based representation is complementary to a word-oriented model and that their combination according to a late fusion approach outperforms an early fusion strategy.

Related Work

The current state-of-the-art systems for event extraction involve neural network models to improve event extraction. [START_REF] Nguyen | Event Detection and Domain Adaptation with Convolutional Neural Networks[END_REF] and [START_REF] Chen | Event extraction via dynamic multi-pooling convolutional neural networks[END_REF] deal with the event detection problem with models based on CNN. [START_REF] Nguyen | Modeling Skip-Grams for Event Detection with Convolutional Neural Networks[END_REF] improve the previous CNN models of [START_REF] Nguyen | Event Detection and Domain Adaptation with Convolutional Neural Networks[END_REF] for event detection, slightly modifying the way CNNs are applied to sentences by taking into account the possibility to have non-consecutive n-grams as basic features instead of continuous n-grams. Both models use word embeddings for representing windows of text that are trained as the other parameters of the neural network.

The authors of [START_REF] Nguyen | Joint event extraction via recurrent neural networks[END_REF] predict at the same time event triggers and their arguments in a joint framework with Bidirectional RNNs (Bi-RNNs) and a CNN and systematically investigate the usage of memory vectors/matrices to store the prediction information during the labeling of sentence features. Additionally, the authors augment their system with discrete local features inherited from [START_REF] Li | Joint Event Extraction via Structured Prediction with Global Features[END_REF].

A GNN is advocated in [START_REF] Nguyen | Graph Convolutional Networks With Argument-Aware Pooling for Event Detection[END_REF] based on dependency trees to perform event detection with a pooling method that relies on entity mentions aggregating the convolution vectors. The authors of [START_REF] Liu | Exploiting Argument Information to Improve Event Detection via Supervised Attention Mechanisms[END_REF] consider also that arguments provide significant clues to this task and adopt a supervised attention mechanism to exploit argument information explicitly for event detection, while also using events from FrameNet. Further, some researchers have proposed other hybrid neural network models with different types of pre-set word embeddings that combine different neural networks to make use of each other's abilities. A hybrid neural network (a CNN and an RNN) [START_REF] Feng | A language-independent neural network for event detection[END_REF] was developed to capture both sequence and chunk information from specific contexts and use them to train an event detector for multiple languages without any handcrafted features.

Some authors went beyond sentence-level sequential modeling, considering that these methods suffer from low efficiency in capturing very long-range dependencies. An approach that goes beyond sentence level [START_REF] Duan | Exploiting Document Level Information to Improve Event Detection via Recurrent Neural Networks[END_REF] was proposed by using a document representation using an RNN model that can automatically extract cross-sentence clues.

Recently, different approaches that include external resources and features at a subword representation level have been proposed. For example, Generative Adversarial Networks (GANs) framework has been applied in event extraction [START_REF] Zhang | Joint entity and event extraction with generative adversarial imitation learning[END_REF][START_REF] Hong | Self-regulation: Employing a generative adversarial network to improve event detection[END_REF]. Besides, reinforcement learning is used in [START_REF] Zhang | Joint entity and event extraction with generative adversarial imitation learning[END_REF] for creating an end-to-end entity and event extraction framework. An approach based on the BERT pretrained model [START_REF] Yang | Exploring Pre-trained Language Models for Event Extraction and Generation[END_REF] attempts an automatic generation of labeled data by editing prototypes and filtering out the labeled samples through argument replacement by ranking their quality.

The problem of ambiguous indicators for particular types of events, i.e., the same word can express completely different events, such as fired, that can correspond to an Attack type of event or can express the dismissal of an employee from a job, is approached in [START_REF] Liu | Event Detection via Gated Multilingual Attention Mechanism[END_REF] by using an RNN and cross-lingual attention to model the confidence of the features provided by other languages.

Motivation

Learning word representations from a corpus (word embeddings) allows us to derive a flexible similarity between words that takes into account a form of synonymy or relatedness between the words into the model. A drawback of this kind of representation is that unknown words (i.e. words unseen in the training corpus) are not well represented: they are generally associated with a random embedding even if these words are morphologically close to known words. Existing embeddings trained on very large collections of text, such as word2vec embeddings, which have proven their efficiency as initial embeddings for event extraction, do not take into account these morphological similarities: no lemmatization or stemming or even case normalization is performed.

We present in Table 1 some statistics about unknown words on the dataset we will use for training and testing our proposed approach, the ACE 2005 dataset, [START_REF] Ji | Refining Event Extraction through Cross-Document Inference[END_REF]. We report the size of the vocabulary for the whole dataset, the size of the vocabulary for the trigger words, the number of words in the test dataset not seen in the training dataset, and among those, the number of words for which a similar word (measured by a Levenshtein ratio of less than 0.3) can be found.

We see that there is an important number of words, even among the trigger words, that cannot be exploited by the models because they are not seen in the training corpus. Also, most of these words (more than 88%) have similar words in the training corpus which could be used to approximate their representation, as they are likely to be semantically close words. To illustrate this, we show in Table 2 examples of unknown words focused on triggers. The examples are pairs of words used as triggers in the test set associated with their closest trigger words (for an event of the same type) as seen in the training set (with a distance of Levenshtein ratio less than 0.3). We can see in this table that most of the pairs correspond to derivational morphology links. These semantic links are lost with the standard embedding models.

The integration of a character-based embedding model should be able to help in dealing with such cases by allowing to bridge the gap between the unknown words and representations of known words used for training the system. The same problem occurs with infrequent words, that could be better represented if they are processed at a character level.

Approach

Our approach lies in the standard supervised framework of event detection where the task is modeled as a word classification task: considering a sentence, we want to predict for each word of the sentence if it is an event trigger and associate it with its event type. The input of the system is therefore a target word in the context of a sentence and the output an event type or NONE for non-trigger words. To study the influence of character-based features, we rely on the CNN model proposed by [START_REF] Nguyen | Event Detection and Domain Adaptation with Convolutional Neural Networks[END_REF] as a core model. This core architecture is used in the two components of our overall system: the Word model and the Character model. These two components are combined using either an early fusion approach or a late fusion approach, as illustrated by Figure 1.

Word and Character CNN Models

In the Word CNN model, the context of a target word is formed by its surrounding words in the sentence, which constitutes the input of the convolution layer. To consider a limited-sized context, longer sentences are trimmed and shorter ones are zero-padded. We consider a context window for every trigger candidate where each token is associated with a word embedding and a relative position to the trigger candidate embedding. The word and position embeddings are concatenated and passed through the convolution layer. The concatenated output of convolutional filter maps forms, after a max-over-time pooling operation on each one, the representation of the input (target token and context) that finally goes through a softmax classification layer. The Character CNN model is very close to the Word CNN model, with two main differences: words are replaced by characters and there is no position embedding associated with each character.

Integration of Word and Character Models

Early Fusion The first type of integration is the early fusion model, in which the two representations of the input sequence produced by the Word and Character CNNs (i.e., the concatenation of the output vectors of their filters) are concatenated before the fully-connected softmax classification layer. Using this type of integration allows joint learning of the parameters of the two models in the training phase.

Late Fusion

The late fusion integration of the Word and Character CNNs relies on the combination of the decisions of the two models, which are trained separately and therefore learn different characteristics of the candidate trigger. Indeed, the word-level CNN combines word and position embeddings that can capture syntactic and semantic information, and of course, the relative positions of words to the candidate trigger. The character-level CNN learns more local features from character n-grams and can capture morphological information. The late fusion focuses on the individual strength of these two models by applying the following rule: we always keep the Character CNN label, except if a trigger was detected by the Word CNN but not by the Character CNN. This strategy is motivated by the fact that the Word CNN model has good coverage whereas the Character model is more focused on precision.

Experiments and Results

Dataset

The evaluation is conducted on the annotated ACE 2005 corpus. We use the same split as previous studies with this dataset [START_REF] Nguyen | Event Detection and Domain Adaptation with Convolutional Neural Networks[END_REF][START_REF] Nguyen | Joint event extraction via recurrent neural networks[END_REF]: 40 news articles (672 sentences) for the test set, 30 other documents (863 sentences) for the development set, and the remaining 529 documents (14,849 sentences) for the training set. Following the same line of work, we consider that a trigger is correct if its event type, subtype, and offsets match those of a reference trigger. We use Precision (P), Recall (R), and F-measure (F1) to evaluate the overall performance.

Hyperparameters

For the Word CNN, we consider a sliding window with a maximal size of 31 words. Hence, sentences are padded at the beginning and the end with a vector of 15 zeros (a common practice for the padding special character). The window sizes for the convolutions are in the set {1, 2, 3} and 300 feature maps are used for each window size. After each convolutional layer with orthogonal weights initialization, a ReLu non-linear layer is applied. We employ dropout with a probability of 0.5 after the embedded window of text since they contain most of the parameters and as a consequence, the possibility of being responsible for overfitting. A dropout of 0.3 is also applied after the concatenation of the convolutions. The size of the position embeddings is equal to 50, similarly to [START_REF] Nguyen | Event Detection and Domain Adaptation with Convolutional Neural Networks[END_REF]. We use the Google News word embeddings pre-trained with word2vec (size = 300).

For the Character CNN, we consider a maximum length of 1,024 for a sequence of characters: longer sequences are trimmed and shorter ones are padded with zeros. The window sizes for the convolutions are in the set {2, . . . , 10}, with 300 feature maps. The convolutional layer non-linearity and initialization are the same as for the Word CNN. The size of the character embeddings is 300. These embeddings are initialized based on a normal distribution and trained on the event detection task. A dropout of 0.5 is applied after the embedded characters. When jointly trained, in the early fusion model, the features obtained after convolutions from both models are concatenated and, similarly to the Word CNN, a dropout of 0.3 is applied afterward, before the softmax layer. We encode all the characters except space.

We train both networks (Word and Character CNNs) with Adam optimizer. During the training, we optimize the embedding tables (i.e., word, position, and character embeddings) to achieve the optimal states. Finally, for training, we use a batch size of 256 for the Word CNN and 128 for the Character CNN. When they are trained jointly in the early fusion model, we use a batch size of 128. All these hyperparameters were optimized by a grid search on the development set.

Results

We compare our model with several neural-based models proposed for the same task that do not use external resources, namely: a set of CNN-based models including a CNN model without any additional features [START_REF] Nguyen | Event Detection and Domain Adaptation with Convolutional Neural Networks[END_REF], the dynamic multipooling CNN model of [START_REF] Chen | Event extraction via dynamic multi-pooling convolutional neural networks[END_REF], the non-consecutive CNN of [START_REF] Nguyen | A two-stage approach for extending event detection to new types via neural networks[END_REF], and the Graph CNN proposed by [START_REF] Nguyen | Graph Convolutional Networks With Argument-Aware Pooling for Event Detection[END_REF]; a set of RNN-based models, represented by the bidirectional joint RNN model of [START_REF] Nguyen | Joint event extraction via recurrent neural networks[END_REF], the DLRNN model of [START_REF] Duan | Exploiting Document Level Information to Improve Event Detection via Recurrent Neural Networks[END_REF] and the DEEB-RNN model of [START_REF] Zhao | Document Embedding Enhanced Event Detection with Hierarchical and Supervised Attention[END_REF] that both rely on a document representation, and the work of [START_REF] Liu | Event Detection via Gated Multilingual Attention Mechanism[END_REF], based on a Gated Cross-Lingual Attention mechanism. Our reference models also include the hybrid model proposed by [START_REF] Feng | A language-independent neural network for event detection[END_REF], the model exploiting arguments through an attention mechanism of [START_REF] Liu | Exploiting Argument Information to Improve Event Detection via Supervised Attention Mechanisms[END_REF], and the GAIL-ELMo model of [START_REF] Zhang | Joint entity and event extraction with generative adversarial imitation learning[END_REF], based on GANs. We do not consider models that are using other external resources such as [START_REF] Bronstein | Seed-Based Event Trigger Labeling: How far can event descriptions get us?[END_REF], [START_REF] Li | Joint event extraction based on hierarchical event schemas from FrameNet[END_REF], or [START_REF] Yang | Exploring Pre-trained Language Models for Event Extraction and Generation[END_REF], since we only rely on the input text in our model. We also compare this model with four baselines based on the BERT language model, applied in a similar way to [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] for the NER task, with the recommended hyperparameters: a learning rate of 2e-5 and the split of sentences into chunks of 128 tokens.

The best performance (75.8 F1 on the test set) is achieved by combining word and position embeddings with the character-level features using a late fusion strategy. This performance relates to improvements that have been reported on other tasks when concatenating word embeddings with the output from a character-level CNN for Part-of-Speech tagging [START_REF] Santos | Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts[END_REF] and NER [START_REF] Santos | Boosting Named Entity Recognition with Neural Character Embeddings[END_REF]. From Table 3, we can also outline that adding character embeddings in a late fusion strategy outperforms all the word-based models, including complex architectures such as the graph CNN and the models based on the BERT language model. Among BERT models, it is worth noticing that the cased models perform better than the uncased ones, which confirms that the character morphology is important for the task, maybe because capitalization is connected to the recognition of named entities, which are usually considered important to detect event mentions.

However, we can see that the character embeddings are not sufficient on their own: using only the Character CNN leads to the smallest recall among all the considered approaches. However, its precision is high (71.7), which makes this model fairly reliable about the triggers it retrieves. Given this observation, we can compare the two integration strategies, early and late fusions. In the case of early fusion, where the two models are trained jointly, we notice that Table 3. Evaluation of our models and comparison with state-of-the-art systems for event detection on the blind test data. † beyond sentence level, + with gold arguments.

Approaches

Precision Recall F1 Word CNN [START_REF] Nguyen | Event Detection and Domain Adaptation with Convolutional Neural Networks[END_REF] (without entities) 71.9 63.8 67.6 Dynamic multi-pooling CNN [START_REF] Chen | Event extraction via dynamic multi-pooling convolutional neural networks[END_REF] 75.6 63.6 69.1 Joint RNN [START_REF] Nguyen | Joint event extraction via recurrent neural networks[END_REF] 66.0 73.0 69.3 RNN with document context † [START_REF] Duan | Exploiting Document Level Information to Improve Event Detection via Recurrent Neural Networks[END_REF] 77.2 64.9 70.5 Non-Consecutive CNN [START_REF] Nguyen | A two-stage approach for extending event detection to new types via neural networks[END_REF] na na 71.3 Attention-based + [START_REF] Liu | Exploiting Argument Information to Improve Event Detection via Supervised Attention Mechanisms[END_REF] 78.0 66.3 71.7 GAIL-ELMo [START_REF] Zhang | Joint entity and event extraction with generative adversarial imitation learning[END_REF] 74.8 69.4 72.0 Gated Cross-Lingual Attention [START_REF] Liu | Event Detection via Gated Multilingual Attention Mechanism[END_REF] 78.9 66.9 72.4 Graph CNN [START_REF] Nguyen | Graph Convolutional Networks With Argument-Aware Pooling for Event Detection[END_REF] 77.9 68.8 73.1 Hybrid NN [START_REF] Feng | A language-independent neural network for event detection[END_REF] 84.6 64.9 73.4 DEEB-RNN3 † [START_REF] Zhao | Document Embedding Enhanced Event Detection with Hierarchical and Supervised Attention[END_REF] 72.3 75.8 74.0 BERT-base-uncased + LSTM [START_REF] Wadden | Entity, Relation, and Event Extraction with Contextualized Span Representations[END_REF] na na 68.9 BERT-base-uncased [START_REF] Wadden | Entity, Relation, and Event Extraction with Contextualized Span Representations[END_REF] na na 69.7 BERT-base-uncased [START_REF] Du | Event Extraction by Answering (Almost) Natural Questions[END_REF] 67.2 73.2 70.0 BERT-QA [START_REF] Du | Event Extraction by Answering (Almost) Natural Questions[END_REF] 71 the precision is the highest among all the compared models. We assume that in the joint approach, the power of representation of morphological properties provided by the characters is overtaking the influence of the word and positions embedding, and the combination reproduces the imbalance between precision and recall observed for the Character CNN, the recall being the lowest among all the models except the Character CNN. In the case of the late fusion, since we have more control over the combination and we can give priority to the Character CNN to establish the labels on the trigger candidates retrieved by the Word CNN, the method takes advantage of the high precision of the Character CNN, allowing an increase of the precision from 71.7 to 87.2, while still having a high recall, also increasing the recall of the Word CNN model from 65.9 to 67.1. The late fusion integration is therefore able to take into account the complementarities of the two models. Finally, for more qualitative analysis, we examine the new triggers correctly detected by the Word + Character CNN (late fusion), in comparison with the Word CNN. We observe that among the 37 new correctly found triggers, some are indeed derivational or inflectional variants of known words in the training data, such as illustrated in Table 4. This seems to confirm that the character-based model can capture some semantic information associated with morphological characteristics of the words and manage to detect new correct event mentions that correspond to inflections of known event triggers (i.e., existing in the training data). Also, the fact that the convolution windows in the Character CNN range from 2 to 10 means that character n-grams in the same range are included in the model and contribute to the model's ability to handle different word variations.

Conclusion and Perspectives

We have proposed in this article a study of the integration of character embeddings in an event detection neural-based model using a simple CNN model as core architecture and testing early and late fusion strategies to integrate the characterbased features. The best results are achieved by combining the word-based features with the character-based features in a late fusion strategy that gives priority to the Character CNN for deciding the event type. This method outperforms more complex approaches such as Graph CNN or adversarial networks and BERTbased models. Our results demonstrate that a convolutional approach for learning character-level features can be successfully applied to event detection and that these features allow overcoming some issues concerning unseen or misspelled words in the test data. We do not integrate the character information at the embedding level as it is usually done in models considering smaller units such as ELMO with characters, FastText with character n-grams, or BERT with subwords. In a certain way, they implement another kind of early fusion than ours. However, our late fusion approach is complementary and as a perspective, we consider implementing this late fusion framework using more complex models as core models. Another way to deal with the problem of unseen words would be to exploit data augmentation strategies that would focus on increasing the variability about derivational and inflectional variants of event mentions in the training data.

Fig. 1 .

 1 Fig. 1. Word + Character CNN.

Table 1 .

 1 Statistics about unknown words in the ACE 2005 dataset.

	All words Trigger words

Table 2 .

 2 Examples of unknown words focused on triggers.

	Event Type Unknown/Closest Trigger Words
	Start-Org	creating/creation, opening/open, forging/forming, formed/form
	End-Org	crumbled/crumbling, dismantling/dismantle, dissolved/dissolving
	Transport	fleeing/flying, deployment/deployed, evacuating/evacuated
	Attack	intifada/Intifada, smash/smashed, hacked/attacked, wiped/wipe
	End-Position retirement/retire, steps/step, previously/previous, formerly/former
	using the standard training/validation/test split

Table 4 .

 4 Examples of new triggers found with the Word+Character CNN (late fusion).

	Event Type New triggers correctly found Trigger words in training data
	End-Position	steps	step
	Extradite	extradited	extradition
	Attack	wiped	wipe
	Start-Org	creating	create
	Attack	smash	smashed
	End-Position	retirement	retire

https://catalog.ldc.upenn.edu/ldc2006t06.


Dorian Kodelja
email: dorian.kodelja@cea.fr

Romaric Besançon
email: romaric.besancon@cea.fr

Olivier Ferret
email: olivier.ferret@cea.fr

Dynamic Cross-Sentential Context Representation for Event Detection

Keywords: Information extraction, event detection, global context

In this paper, which focuses on the supervised detection of event mentions in texts, we propose a method to exploit a large context through the representation of distant sentences selected based on coreference relations between entities. We show the benefits of extending a neural sentence-level model with this representation through evaluation carried out on the TAC Event 2015 reference corpus.

Introduction

This study focuses on the supervised event extraction from text, which consists in identifying in texts the words or the sequences of words, called event mentions, that mark the presence of a predefined type of events. For instance, the word pow-wow for an event of type Meet in:

Putin had invited Tony Blair to the pow-wow in Saint Petersburg's Grand Hotel Europe.

The best methods for achieving this task are generally based on neural models and operate at the sentence level, similarly to [START_REF] Nguyen | Graph Convolutional Networks with Argument-Aware Pooling for Event Detection[END_REF]. However, the sentence level is not always sufficient to get all the elements for detecting an event mention. Two main types of studies already explored the possibility to exploit information at a larger scale: on the one hand, methods that use document level information to perform event extraction at a local scale; on the other hand, methods that achieve event extraction globally at the document level through joint approaches [START_REF] Chen | Collective event detection via a hierarchical and bias tagging networks with gated multi-level attention mechanisms[END_REF][START_REF] Reichart | Multi-Event Extraction Guided by Global Constraints[END_REF][START_REF] Liu | A Probabilistic Soft Logic Based Approach to Exploiting Latent and Global Information in Event Classification[END_REF][START_REF] Yang | Joint Extraction of Events and Entities within a Document Context[END_REF]. Our work takes place among the first type of methods, which can be broken down into methods using specific information at the document scale between events [START_REF] Liao | Using Document Level Cross-Event Inference to Improve Event Extraction[END_REF][START_REF] Kodelja | Exploiting a More Global Context for Event Detection Through Bootstrapping[END_REF] or event and entities [START_REF] Hong | Using Cross-Entity Inference to Improve Event Extraction[END_REF] and methods exploiting a more global representation of documents, either through generic models such as Doc2Vec [START_REF] Duan | Exploiting Document Level Information to Improve Event Detection via Recurrent Neural Networks[END_REF] or models specifically trained for the target task as in [START_REF] Zhao | Document Embedding Enhanced Event Detection with Hierarchical and Supervised Attention[END_REF].

In this article, our contribution is a new method for taking into account the document context for event extraction. More precisely, we exploit the coreference links from the entities surrounding a candidate mention to dynamically build its context from selected event-related distant sentences. The representations of those sentences are then integrated into a sentence-level model that, similarly to recent studies [START_REF] Nguyen | Graph Convolutional Networks with Argument-Aware Pooling for Event Detection[END_REF][START_REF] Liu | Jointly Multiple Events Extraction via Attention-based Graph Information Aggregation[END_REF][START_REF] Yan | Event detection with multi-order graph convolution and aggregated attention[END_REF][START_REF] Balali | Joint event extraction along shortest dependency paths using graph convolutional networks[END_REF], is based on Graph Convolution Networks (GCN) [START_REF] Kipf | Semi-supervised classification with graph convolutional networks[END_REF].

Model

Classically, we frame event detection as a multi-class classification task for each word in a document. The label is either one of the 38 event types of the DEFT Rich ERE taxonomy [START_REF] Bies | A Comparison of Event Representations in DEFT[END_REF] or the NONE label for the absence of event mention.

Intra-sentential GCN

Our intra-sentential model is a GCN relying on syntactic dependencies, similarly to [START_REF] Nguyen | Graph Convolutional Networks with Argument-Aware Pooling for Event Detection[END_REF]. In this model, we consider as candidate each word w t in a sentence S = (w 1 , w 2 , . . . , w n ), where w i is the i-th word in the sentence, associated with an entity type e i (with e i = O if w i is not an entity head). Each of these words is represented as a real-valued vector X = x 1 , x 2 , . . . , x n built by concatenating three kinds of embeddings: a word embedding for representing the word itself, a position embedding for its relative distance to the candidate, and an entity embedding for its entity type e i . A BiLSTM is applied to the target sentence S (focused on w t through the position embedding) for producing a first contextual representation of each word. A GCN made of K convolution layers is then used for producing a contextual representation of each word taking into account the influence of distant words of S through up to K syntactic dependencies. It relies on a directed graph G where the nodes are the words of S and each edge (w i , w j ) is associated with a label L(w i , w j ) corresponding to a syntactic dependency between w i and w j . The last step consists in aggregating the sequence h K w1 , h K w2 , . . . h K wn at the last convolution layer into a final representation p t of the target word w t that can be fed to a dense layer with a softmax for the classification. [START_REF] Nguyen | Graph Convolutional Networks with Argument-Aware Pooling for Event Detection[END_REF] introduces a new pooling strategy that focuses on entities, with the assumption that entities carry a special interest for the task.

With a similar goal, we propose syntactic pooling, which also considers multiple specific words in the sentence while not requiring a prior annotation of named entities. In this case, the pooling is focused on the target word and all nouns (n), verbs (v), and adjectives (a) in the sentence:

p t = maxpool({h K wt } ∪ {h K wi,1 i n : pos(w i ) ∈ {n, v, a}}) (1)

Cross-Sentential Context Representation

Contrary to work integrating a global representation of the document [START_REF] Duan | Exploiting Document Level Information to Improve Event Detection via Recurrent Neural Networks[END_REF][START_REF] Zhao | Document Embedding Enhanced Event Detection with Hierarchical and Supervised Attention[END_REF], we chose to take into account the context of a target sentence in a more selective way, both for improving the disambiguation of candidate event mentions and limiting the parameters of the model. For the task of event extraction, the presence of common named entities is a good indicator of the contextual association of the sentences since they are typically possible arguments of similar events (for instance, different legal events concerning the same person), related events in a chronological succession (an injure event followed by a die event) or even two mentions of the same event. In the example given in the introduction, pow-wow is not a frequent word for a Meet event but the event is also mentioned with less ambiguous occurrences in the same document, in sentences sharing common entities, such as Saint Petersburg:

But the Saint Petersburg summit ended without any formal declaration on Iraq.

Context Representation Our context representation relies on the integration of a contextual representation of each entity mention e j i of the target sentence S j . For selecting the context linked to e j i , we define the function links(S j , S k , i), that gives the set of positions l in a context sentence S k of its entity mentions that are in a coreference relation with the considered entity mention e j i :

links(S j , S k , i) = {l : E(e j i ) = E(e k l )} 1 l n (2)
where E(e) denotes the entity referred by the mention e. The context of e j i is then built from the set of pairs (context sentence, mention of E(e j i )) defined as:

Links(S j , i) = (S k , l) : l ∈ links(S j , S k , i) 1 l n, k =j (3)
For each pair (S k , l) of this context, we produce an input representation, noted X k,l = x k,l 1 , x k,l 2 , . . . , x k,l n , similar to the one in Section 2.1, except for the position embeddings: in this case, the position vector of each word of S k represents the distance to the position l of the entity mention e k l . A BiLSTM is then applied to this input representation. Two extraction methods for the representation of each pair (S k , l) are considered: the Final mode (eq. 4), which concatenates the final representations of the two LSTMs, and the Mention mode (eq. 5), which extracts the representations at the position of the entity mention e k l .

Final :

h context (S k , l) = [h forward (x k,l n ); h backward (x k,l 1 )] (4)
Mention : h context (S k , l) = [h forward (x k,l l ); h backward (x k,l l )] (5)
Context Integration The context representation of the entity mention e j i is then integrated into the local context at two possible levels, as illustrated by Figure 1: either as an additional embedding in the local input representation of the entity mention, or as an additional node in the graph, associated with the node of the entity mention by a specific relation. For both integration modes, the expected representation is a vector that we obtain by aggregating the vectors of all contextual entity mentions through max-pooling:

context(e j i ) = maxpool({h context (S k , l) : (S k , l) ∈ Links(S j , i)}) (6)
For the integration as a node, we modify the dependency graph G by adding a node cn j i merging all the context representations of e j i and having h 0 cn j i = context(e j i ) as initial representation. We then define a new Context edge type between the local entity mentions and their context representation and add the corresponding edge (w j i , cn j i ) in G. For the integration at the embedding level, the context representation is concatenated to the other embeddings. For the words having no context representation, a default representation c default is used, initialized randomly, and modified during training. The context vector defined in ( 6) is then generalized to all the words of the sentence with:

c j i = context(e j i ) if |Links(S j , i)| > 0 c default otherwise (7)
and the input sequence is redefined as X j = ([x j 0 , c j 0 ], [x j 1 , c j 1 ], . . . , [x j n , c j n ]).

Experiments

Data and Preprocessing

Our training dataset is composed of 58 documents from the TAC 2015 training dataset and 288 documents from the DEFT Rich ERE (R2 V2 and V2) dataset. The validation set is composed of the remaining 100 documents from the TAC 2015 training dataset. We evaluate our proposed model on the test set of TAC 2015 to compare it with the graph model of [START_REF] Nguyen | Graph Convolutional Networks with Argument-Aware Pooling for Event Detection[END_REF].

We use the Stanford CoreNLP tool [START_REF] Manning | The Stanford CoreNLP Natural Language Processing Toolkit[END_REF] for named entity recognition (NER), coreference resolution, and syntactic analysis to produce dependency graphs from its Basic dependencies. At the document level, an entity is defined as a group of mentions in coreference. For increasing the coverage of the automatic coreference system, we merge entities mentions of which are identical. Example Generation To facilitate access to content-bearing words in the graph convolution, we filter some categories of words: punctuations, symbols, numbers, determiners, prepositions, conjunctions, or interjections. We also use a prediction mask: only nouns, verbs, and adjectives are associated with a predicted category; the other words are associated with the NONE class.

Hyperparameters The word embeddings are initialized with pretrained GloVe embeddings [START_REF] Pennington | Glove: Global Vectors for Word Representation[END_REF]. The position and the entity type embeddings are of size 50 while the dimensions of the local BiLSTM layer and the two graph convolution layers are 400 and 300 respectively. The embeddings for the words, entities, and distances are the same for the target sentences and the context sentences. The model is trained using SGD with momentum and batches of 10 examples. All average performances are computed on 10 runs with the same parameters.

Study of Model's Parameters

We first evaluate the influence of the different choices for the model's parameters:

-Intra-sentential pooling: Syntactic/Entity -Context representation extraction: Final /Mention -Context representation integration: Embedding/Node

We searched for the best values of these parameters on the validation set together with the values of less specific optimization parameters (learning rate, l2 regularization, dropout, momentum). Concerning the model's parameters, the best result is obtained using Syntactic pooling, Final extraction, and Embedding integration. These parameters are also the best, in general, in each tested configuration, but since we cannot show all results, we present in Table 1 the results for this best model, noted C-GCN, and the variations of this model when changing each of the other parameters. We observe, with a weakly significant difference (p = 0.058), that the entity pooling is slightly worse than the Syntactic pooling, which indicates that the use of a larger set of context words benefits to an enriched representation of the target word. On the contrary, the overall pooling in [START_REF] Nguyen | Graph Convolutional Networks with Argument-Aware Pooling for Event Detection[END_REF] performs worse than the entity pooling while it also considers more words than that pooling. However, this difference may come from the use of different NER tools.

Concerning the context extraction, the poor results obtained with the Mention mode could also be related to the quality of the entities or to the fact that the final representations of the context sentences are more informative than the specific representations of the entity mentions. Finally, the integration of the context representation as a node does not degrade the results in a significant way but produces a less balanced performance between precision and recall.

Comparison with State-of-the-art

We compare in this section our proposed model to the original model from [START_REF] Nguyen | Graph Convolutional Networks with Argument-Aware Pooling for Event Detection[END_REF], noted GCN nguyen , and to the best model of the TAC evaluation campaign, RPI_BLENDER, proposed by [START_REF] Hong | RPI_BLENDER TAC-KBP2015 System Description[END_REF], based on a MaxEnt classifier using a large set of lexical, syntactic and entity features. To further prove the interest of having a specific context for each example, we train a model C-GCN generic that uses all the sentences of the document as context. In this case, there is no position embedding for the context sentences, and the same representation is used as an embedding for all the words in the considered sentence. The results presented in Table 2 prove the interest of our proposition: our implementation of the GCN model, noted GCN repro , achieve results similar to the ones reported by [START_REF] Nguyen | Graph Convolutional Networks with Argument-Aware Pooling for Event Detection[END_REF] 1 and we obtain a gain of 1.8 F-score on this baseline when using the context representation (p < 0.0001). We also see that the integration of the context in C-GCN generic does not yield better results, which confirms our intuition on the interest of defining a context specific to each example.

Conclusion and Perspectives

We propose in this article a method allowing a neural model for event extraction to take into account a cross-sentential context. The approach consists in enriching the representation of entity mentions in a target sentence with a contextual embedding built using information from distant sentences where these entities also occur. The evaluation of the approach on the dataset TAC 2015 proves the interest of the method, with a significant gain over the initial model. One perspective would be to use an attention mechanism as an alternative to the max-pooling to aggregate the representations of all the mentions of an entity, which could lead to better discriminate and filter context sentences. Another one could be the use of a more elaborated GCN model able to take into account the type of the relations in the graph, such as Relational GCN [START_REF] Schlichtkrull | Modeling Relational Data with Graph Convolutional Network[END_REF].

Fig. 1 .

 1 Fig. 1. Two solutions for integrating the context representation (in red) of an entity mention into the GCN model: at the input level or by adding a node to the graph.

Table 1 .

 1 Performances on the validation set for the main model's parameters (Pavg., Ravg., Favg.: average values from 10 runs of precision, recall, and F-score; Fσ: F-score standard deviation; Fmax.: F-score maximal value).

		Pavg. Ravg. Favg.	Fσ	Fmax.
	C-GCN	75.6	50.4	60.5 0.6	60.4
	Pooling -Entity	74.8	49.2	59.3	0.9	60.2
	Extraction -Mention	75.0	48.8	59.1	1.2	58.1
	Integration -Node	76.9	48.1	59.1	1.2	59.3

Table 2 .

 2 Results on TAC 2015 test set (F max./dev : F-score for the best parameters on the dev. set; for the two reference systems, P and R are max./dev values; average values for the others).

		P	R	Favg.	Fσ	F max./dev
	RPI_BLENDER	75.2	47.7	-	-	58.4
	GCNnguyen	70.3	50.6	-	-	58.8
	GCNrepro	78.5	47.0	58.7	0.8	59.1
	C-GCNgeneric	74.5	48.4	58.6	0.6	59.0
	C-GCN	75.6	50.4	60.5 0.6	60.4

We note that we do not have exactly the same train/dev datasets because we also used the DEFT dataset as training, which can explain the slight gain in F-score.

Work partly supported by ANR under project ASRAEL (ANR-15-CE23-0018) with the FactoryIA supercomputer supported by the Ile-de-France Regional Council.


Aboubacar Tuo
email: aboubacar.tuo@cea.fr

Romaric Besançon
email: romaric.besancon@cea.fr

Olivier Ferret
email: olivier.ferret@cea.fr

Julien Tourille
email: julien.tourille@cea.fr

Better Exploiting BERT for Few-shot Event Detection

Keywords: Few-shot Event Detection, Meta-learning, BERT

Recent approaches for event detection rely on deep supervised learning, which requires large annotated corpora. Few-shot learning approaches, such as the meta-learning paradigm, can be used to address this issue. We focus in this paper on the use of prototypical networks with a BERT encoder for event detection. More specifically, we optimize the use of the information contained in the different layers of a pre-trained BERT model and show that simple strategies for combining BERT layers can outperform the current state-of-the-art for this task.

Introduction

Event extraction aims to automatically extract structured information about events from text. It can be compared to filling a form with entities, each field of this form corresponding to an argument of the event. Earlier methods for event extraction were based on handcrafted rules [START_REF] Ahn | The stages of event extraction[END_REF]. These methods were gradually replaced by machine learning algorithms with the development of statistical learning and neural networks in recent years. In this context, [START_REF] Li | Joint event extraction via structured prediction with global features[END_REF] proposes a structured prediction model based on numerous lexico-syntactic features, [START_REF] Nguyen | Event Detection and Domain Adaptation with Convolutional Neural Networks[END_REF] uses convolutional networks to exploit contextual information, [START_REF] Nguyen | Joint Event Extraction via Recurrent Neural Networks[END_REF] defines models based on recurrent networks, and [START_REF] Liu | Jointly Multiple Events Extraction via Attention-based Graph Information Aggregation[END_REF], [START_REF] Nguyen | Graph Convolutional Networks with Argument-Aware Pooling for Event Detection[END_REF], and [START_REF] Yan | Event Detection with Multi-Order Graph Convolution and Aggregated Attention[END_REF] exploit graph convolution models to capture syntactic dependencies between different parts of sentences.

While the objective of event extraction is to identify all arguments connected to a particular event, datasets since ACE 2005 [START_REF] Walker | ACE 2005 Multilingual Training Corpus[END_REF] have introduced the concept of event trigger, designating the word or group of words that indicate as clearly as possible the presence of an event in a sentence. The intention is to define a lexical anchor to help in the search for arguments. We focus in this paper on the detection and the classification of those triggers according to a restricted set of predefined types, a task generally called Event Detection or Trigger Detection.

Supervised learning methods are costly since they require large corpora that are manually annotated. Hence, a current challenge is to investigate methods that reduce the development cost of these systems. In this context, we investigate Few-Shot Learning (FSL) for trigger detection.

Few Shot Event Detection (FSED) has been recently the focus of several studies with various configurations: generalization of models to new types of events using keyword lists [START_REF] Bronstein | Seed-Based Event Trigger Labeling: How far can event descriptions get us?[END_REF][START_REF] Lai | Extending Event Detection to New Types with Learning from Keywords[END_REF], data enrichment with external resources [START_REF] Deng | OntoED: Low-resource Event Detection with Ontology Embedding[END_REF], Zero-Shot Learning with the use of class descriptions or external resources [START_REF] Zhang | Zero-shot Label-Aware Event Trigger and Argument Classification[END_REF], and FSL [START_REF] Shen | Adaptive Knowledge-Enhanced Bayesian Meta-Learning for Few-shot Event Detection[END_REF][START_REF] Chen | Honey or Poison? Solving the Trigger Curse in Few-shot Event Detection via Causal Intervention[END_REF][START_REF] Cong | Few-Shot Event Detection with Prototypical Amortized Conditional Random Field[END_REF]. Other studies have also focused on Few-Shot Event Classification, which restricts FSED to assigning an event type to a candidate trigger already identified in a sentence [START_REF] Lai | Learning Prototype Representations Across Few-Shot Tasks for Event Detection[END_REF][START_REF] Lai | Extensively Matching for Few-shot Learning Event Detection[END_REF][START_REF] Deng | Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detection[END_REF].

In this paper, our contribution focuses on a better exploitation of the BERT language model representations for FSED, more specifically by studying the importance of these different representations and by evaluating different ways to associate them.

Method

Problem Formulation

We cast FSED as a sequence labeling task [START_REF] Ramshaw | Text chunking using transformation-based learning[END_REF], using the IOB format (Inside Outside Beginning), which can be addressed as a multi-class classification task.

Recent studies in FSL use meta-learning, which is often defined as learning to learn. The main idea behind meta-learning is to train models on several tasks, each with a limited number of instances, so that the learned model can quickly perform similar tasks on new data. The methods that have emerged in recent years for solving FSL tasks with the help of meta-learning fall into three main categories: model-based methods [START_REF] Yan | Multi-task transfer methods to improve one-shot learning for multimedia event detection[END_REF][START_REF] Santoro | Meta-learning with memory-augmented neural networks[END_REF], optimization-based algorithms [START_REF] Finn | Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks[END_REF][START_REF] Nichol | On First-Order Meta-Learning Algorithms[END_REF][START_REF] Ravi | Optimization as a model for few-shot learning[END_REF], and metric-based methods [START_REF] Vinyals | Matching networks for one shot learning[END_REF][START_REF] Sung | Learning to compare: Relation network for few-shot learning[END_REF][START_REF] Snell | Prototypical networks for few-shot learning[END_REF]. Among the latter, we adopted prototypical networks [START_REF] Snell | Prototypical networks for few-shot learning[END_REF] in our study, similarly to most works about FSED.

We adopt the standard N-way, K-shot episodic formulation, as described in [START_REF] Vinyals | Matching networks for one shot learning[END_REF]. An episode E is composed of a support set and an associated query set. During each episode, the model is defined by relying on a subset S of the available labeled data, the support set, which contains N types of events and K annotated instances per type (K being generally small, e.g. 1, 5, or 10):

S = {(x 1 1 , t 1 1 , y 1 ), . . . , (x 1 k , t 1 k , y 1 ), . . . , (x N 1 , t N 1 , y N ), . . . , (x N k , t N k , y N )} where x n i = {w 1 , . . . , w L } is a sequence of tokens of length L containing a trigger of type n, t n
i , the position of the trigger, and y n , the corresponding sequence of labels. The training in this context is done by updating the weights of the model based on the prediction on the instances of the query set, which has the same structure as the support set.

Model Architecture

We use a Prototypical Network model with episodic learning to combine metalearning and FSL. An overview of our model, composed of three modules, is presented in Figure 1. Encoder module. Given a sentence x = {w 1 , . . . , w L } of length L, the objective of this module is to construct a representation e i of each word w i ∈ x in a ddimensional space. Importantly, during the episodic learning phase, only the encoder part of the model is actually updated. We use the BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] language model as our encoder, with -Max-pool: max-pooling on each dimension p for m consecutive layers. The p-th element of the embedding e i is given by: (e i ) p = max((

H i = [h 1 i , h 2 i , . . . , h 12
h 1 i ) p , . . . , (h m i ) p ) -Concat: concatenation of m consecutive BERT layers e i = [h 1 i || h 2 i || . . . ||h m i ] or e i = [h 12 i || h 11 i || . . . || h 12-m+1 i ]
-Weighted: linear combination of the 12 BERT layers. e i = 12 k=12 α k h k i , where the α k are randomly initialized and learned.

-ATT: linear combination of BERT layers for each dimension using an attention mechanism. The objective is to identify the most important layers for each dimension. The p-th element of e i is given by (e i ) p = 12 k=1 α k (h k i ) p where the α k are learnt from a linear combination of the 12 layers and a softmax normalization.

Prototypical module. The objective of this module is to build a prototype for each class and then, to classify new examples according to their similarity to these prototypes. We take the average of the examples in the support set as prototypes for each class, as proposed by [START_REF] Snell | Prototypical networks for few-shot learning[END_REF]. Since we are using the IOB format, we build a prototype for classes B and I, as well as for class O (the null class for all event types), which refers to words that are not triggers of any event. Hence, we obtain 2N + 1 prototypes for an episode composed of N types.

Classification module. This module classifies the words of a query sequence according to their similarity to the prototypes. The probability of a given word to belong to a particular class is computed according to its similarity to the class' prototype. The model is trained using the cross-entropy loss.

Experiments and Results

Experimental Setup

Evaluation dataset. We experiment on the FewEvent corpus [START_REF] Deng | Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detection[END_REF] for FSED. This corpus is composed of 70,852 event mentions divided into 100 types. We use the same split as [START_REF] Cong | Few-Shot Event Detection with Prototypical Amortized Conditional Random Field[END_REF] for comparison purposes. This split includes 80 types in the training set, 10 types in the test set, and the remaining 10 types in the validation set.

Model parameters. We use the BERT-base pre-trained model as our encoder.

To evaluate the impact of our modifications of this encoder, we rely on two models presented in [START_REF] Cong | Few-Shot Event Detection with Prototypical Amortized Conditional Random Field[END_REF]: Proto-dot, a prototypical model based on the dot product for its similarity function, which is our baseline model, and PA-CRF, an improvement of the previous model using CRFs (Conditional Random Field ) [START_REF] Lafferty | Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data[END_REF] to estimate the transition probabilities between different IOB labels as proposed by [START_REF] Hou | Few-shot slot tagging with collapsed dependency transfer and label-enhanced task-adaptive projection network[END_REF]. The PA-CRF model is the main contribution of [START_REF] Cong | Few-Shot Event Detection with Prototypical Amortized Conditional Random Field[END_REF] and is, to our knowledge, the best performing model for FSED. We take as input 128-word sentences (with padding if needed) and train the model with a learning rate of 10 -5 .

Evaluation. To test our model, we construct 3,000 episodes E i {S i , Q i } with N types of events randomly sampled for each episode. We then select K examples per class in the support set and one example per class in the query set. The examples in the support set are used to build prototypes and the examples in the query set are classified based on their similarity to these prototypes. We consider an event trigger to be correct if its type and position in the sentence are correctly predicted, as in previous work about event detection [START_REF] Cong | Few-Shot Event Detection with Prototypical Amortized Conditional Random Field[END_REF][START_REF] Cui | Edge-Enhanced Graph Convolution Networks for Event Detection with Syntactic Relation[END_REF][START_REF] Liu | Exploiting Contextual Information via Dynamic Memory Network for Event Detection[END_REF].

Results and Discussion

We compute the micro F1-score to evaluate the performance and report the means and standard deviations over 5 runs in Table 1 with different values of N and K. For the encoders Average, Concat, and Max-pool, we only consider the last 4 layers (as suggested in [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF]) for the results reported in Table 1. We compare our model to our implementation of [START_REF] Cong | Few-Shot Event Detection with Prototypical Amortized Conditional Random Field[END_REF], which gives the best current performance on the same task (BERT line in Table 1) and report the results provided in their article (BERT [Cong] line in Table 1).

Whatever the model used (Proto-dot or PA-CRF), all the encoder improvements, except the Concat configuration for the 10 ways condition of PA-CRF, significantly improve the performance compared to the classical BERT encoder. Thus, a better exploitation of the information of the BERT model leads to outperform the improvements brought by the more sophisticated model of [START_REF] Cong | Few-Shot Event Detection with Prototypical Amortized Conditional Random Field[END_REF], representing the current state of the art. Among all the tested strategies, those allowing the model to learn the weights of the linear combination of BERT's layers generally yield better results, with the Weighted strategy proving to be the best in almost all configurations.

Finally, the fact that the gains observed for the Proto-dot model are also found for the PA-CRF model, which is a more elaborate version of the Protodot model, shows that the proposed improvements are complementary to those that can be made to the other modules (prototypical and classification modules).

Influence of the episodic formulation N ways k shots. We observe logically that the task is more difficult when the number of types (N ) increases and that the results are improved with a larger number of annotated examples (K). Furthermore, we assess the robustness of the evaluation protocol by experimenting with different numbers of evaluation episodes for the encoder Average (between 500 and 5,000). The score variation between our different runs remains within a range of ± 0.5 points.

Layer analysis. We perform experiments to determine the influence of the number of layers selected for the encoders Average, Concat, and Max-pool, which do not have the capability to perform this selection on their own, contrary to Weighted and ATT. We report in Figure 2 the results obtained by these three models for the 5 ways 5 shots task, taking into account n successive layers starting from the first layer (Figure 2a) or the last one (Figure 2b). We observe that the encoder Average is more stable than the other two and that its performance tends to increase steadily with the number of layers. Its best results are also competitive with the Weighted strategy, which shows that taking into account all the layers is beneficial, even with a very simple strategy such as averaging. We can also observe that the other strategies do not exploit all the information. This is particularly noticeable for Max-pool, which probably tends to increasingly smooth out the outstanding dimensions as the number of layers increases. Concerning Concat, it seems that the strategy creates large layered representations that are probably not very informative for the model.

In this context, Figures 2a and 2b also show that the combination of the last layers seems more interesting than the combination of the first ones. This observation may reflect the intrinsic presence of more useful information for the task in these layers, or simply be explained by a more important influence of the learning linked to the task at their level because of their closeness to the model output. Finally, Figure 2c reports the limiting case of considering only one layer, with, from first to last layers, a strong increase of results until layer 4, which reaches a performance comparable to Average but with a larger variance, and then, a soft decrease.

Conclusions and Perspectives

In this article, we have studied different ways to better exploit the information contained in the BERT pre-trained model for the task of detecting events from a few examples. We have shown that the improvements brought by our proposals outperform the state-of-the-art results on this task. We plan to pursue the improvement of the encoder by studying other representation models and by enriching these representations with external knowledge or lists of examples of triggers. Furthermore, we will study how the improvements of the encoder can be combined with improvements of the prototypical and classification modules.

Fig. 1 :

 1 Fig. 1: Our few-shot event detection model based on prototypical networks.

  i ], its representations for the word w i and h j i ∈ R d . The main objective of our work is to study different options for selecting and combining the 12 layers of BERT to obtain more relevant token representations for the event detection task. More precisely: -Average: embedding e i of the word w i = average of the representations of m consecutive layers. e i = 1 m m k=1 h k i or e i = 1 m 12 k=12-m+1 h k i depending on whether the layers are aggregated from the first or the last layer.

  (a) From first layers to last layers. (b) From last layers to first layers. (c) Performance by layer.

Fig. 2 :

 2 Fig. 2: Figures 2a and 2b present the influence of the number of layers selected for the encoders Average, Concat, and Max-pool on the performance of the model. Figure 2c presents the performance of the model for only one single layer.

Table 1 :

 1 Results: mean and standard deviation of the micro F1-score over 5 trials. bold, the best performance on average; underlined, the second best. * denotes the best statistically significant model compared to the second best using the significance test of[START_REF] Dror | Deep dominance -how to properly compare deep neural models[END_REF].

	Model Encoder	5 ways 5 shots	5 ways 10 shots	10 ways 5 shots	10 ways 10 shots
		BERT [Cong]	58.82 ± 0.88	61.01 ± 0.23	55.01 ± 1.62	58.78 ± 0.88
		BERT	61.22 ± 0.90	60.84 ± 1.58	58.14 ± 1.69	59.85 ± 2.01
		average	64.34 ± 1.94	65.37 ± 0.66	61.85 ± 2.05	63.93 ± 1.08
	Proto-dot	max-pool	64.10 ± 1.78	65.80 ± 0.91	61.15 ± 1.51	63.37 ± 1.03
		concat	61.99 ± 0.46	61.94 ± 0.97	57.47 ± 0.65	59.02 ± 1.39
		weighted	65.62 ± 1.55	67.15 ± 0.88* 62.63 ± 1.18* 65.22 ± 0.98*
		ATT	65.64 ± 0.90	65.63 ± 0.46	62.22 ± 0.52	64.23 ± 0.99
		BERT [Cong]	62.25 ± 1.42	64.45 ± 0.49	58.48 ± 0.68	61.54 ± 0.89
		BERT	63.63 ± 2.01	63.66 ± 1.54	62.11 ± 1.58	62.47 ± 1.29
		average	65.09 ± 0.40	66.70 ± 0.45	62.32 ± 1.51	65.38 ± 1.71
	PA-CRF	max-pool	63.95 ± 1.99	66.94 ± 1.20	61.74 ± 1.95	64.77 ± 1.84
		concat	64.30 ± 1.99	64.31 ± 1.80	62.01 ± 1.28	61.88 ± 1.05
		weighted	66.26 ± 1.16* 66.97 ± 0.95* 63.90 ± 1.23* 67.21 ± 1.27*
		ATT	63.65 ± 1.35	66.40 ± 1.03	62.41 ± 1.73	64.32 ± 1.64

This publication was made possible by the use of the FactoryIA supercomputer, financially supported by the Île-de-France Regional Council.