Nils Holzenberger
email: nilsh@jhu.edu

Yunmo Chen

Benjamin Van Durme
email: vandurme@jhu.edu

Asking the Right Questions in Low Resource Template Extraction

Information Extraction (IE) researchers are mapping tasks to Question Answering (QA) in order to leverage existing large QA resources, and thereby improve data efficiency.

Especially in template extraction (TE), mapping an ontology to a set of questions can be more time-efficient than collecting labeled examples. We ask whether end users of TE systems can design these questions, and whether it is beneficial to involve an NLP practitioner in the process. We compare questions to other ways of phrasing natural language prompts for TE. We propose a novel model to perform TE with prompts, and find it benefits from questions over other styles of prompts, and that they do not require an NLP background to author.

Introduction

Information Extraction (IE) tasks such as event extraction (EE) and template extraction (TE) (Du et al., 2021b, inter alia) are commonly presented as slot filling for template-like structures, with fillers extracted from a textual passage. Current extraction systems powered by neural networks have achieved state-of-the-art performance but require significant amounts of annotated data for training. Given the complexity and specialized domains of many ontologies, annotation processes are often performed by domain experts, making high-quality annotations hard to obtain.

One way to mitigate data scarcity is to use natural language prompts. In the general sense, prompts are natural language (NL) strings which, fed to an IE model, serve as a flexible inductive bias across many tasks [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF]. Prompts have been widely adopted in many NLP tasks [START_REF] Levy | Zero-shot relation extraction via reading comprehension[END_REF]McCann et al., 2018, inter alia), paving the way for formulating TE tasks as Question Answering (QA) (Du and Cardie, 2020b;Chen et al., 2020b, inter alia). * Equal contributions Previous work has focused on different ways to compose prompts [START_REF] Shin | Autoprompt: Eliciting knowledge from language models with automatically generated prompts[END_REF] and approaches to tuning prompts towards better model performance [START_REF] Robert | Cutting down on prompts and parameters: Simple few-shot learning with language models[END_REF]. However, to the best of our knowledge, under the TE as QA (TE-QA) formulation, a thorough study has not been pursued to investigate the effectiveness of prompts formulated as NL questions, and humans' ability to judge the usefulness of a given prompt. In particular, we investigate whether potential end users of a TE system can be allowed to drive task definitions. For that, we let humans who have no background in TE write prompts in the form of NL questions, as an efficient way of annotating for the task. Since end users are not experts in TE modeling, and are likely more used to writing queries used in search engines, we are concerned with whether a QAbased TE system can perform well with their questions -a realistic setting for people using such IE systems. With the advancement of QA models, we would like to investigate the robustness of such systems under different types of prompts. Furthermore, we quantify whether end users are able to intuitively distinguish between higher-and lowerquality prompts. 1 The contributions of this work are as follows: 1. We design an analysis framework to empirically evaluate the performance of TE-QA systems fed with different prompts: we propose a rankingbased TE-QA model that bypasses the need to extract spans, to focus on examining the effectiveness of various prompts. 2. We collect prompts from potential end users, and thoroughly analyse prompt quality. 3. We conclude that: a) the phrasing of prompts matters; b) prompt writers without TE modeling background can write prompts leading to high model performance; c) humans are unable to gauge the quality of a prompt.

Corruplate

This Granular event is intended to capture the various aspects of a corruption event.

What event reports the corruption? who the individual being accused of corruption

Who is being accused of corruption?

corrupt-event as with the Protestplate, this enumerates the instances of a Corruption Basic event that led to the Corruplate being generated

What event reports the accusation of corruption?

What did this person/network do that was corrupt?

What event reports the corruption event?

Who has been accused of corruption?

Who was accused of corruption?

What was the corruption?

What event caused the corruption event?

Who is corrupt?

(E) (E) (Q) (Q) (Q) (Q) (Q) (Q) (Q) (Q) (S) (S) (T)
Figure 1: Top: Example template from the Granular ontology. Bottom: an example document from the Granular dataset, with spans highlighted. Expected answers for each slot are specified using superscripts. (T): template name and description, (S): slot name and description, (E): expert question, (Q): non-expert question.

2 Background TE as QA The terms QA and MRC have both been used to describe the use of a linguistic description to query a context in order to extract answers. Under its formulation, prompts2 serve as the key component that elicits information and guides the extraction process. QA has been widely explored in different NLP tasks [START_REF] Mccann | The natural language decathlon: Multitask learning as question answering[END_REF], such as semantic role labeling [START_REF] Fitzgerald | Large-scale QA-SRL parsing[END_REF], dialogue state tracking [START_REF] Gao | Dialog state tracking: A neural reading comprehension approach[END_REF], and fact verification (Weber et al., 2019).

Recent work adopts the formulation for various TE tasks, and achieves state-of-the-art performance [START_REF] Li | Entity-relation extraction as multi-turn question answering[END_REF]Wu et al., 2020;[START_REF] Li | A unified MRC framework for named entity recognition[END_REF]Du and Cardie, 2020b, inter alia). Interestingly, those works make use of different styles of prompts: hand-written NL questions [START_REF] Li | Entity-relation extraction as multi-turn question answering[END_REF], a series of NL questions instantiated from a set of templates (Du and Cardie, 2020b), descriptions from annotation guidelines rewritten as bleached definitions (Chen et al., 2020b), and mentions detected using a Conditional Random Field along with surrounding context used as ques-tions (Wu et al., 2020). These forms of prompts work and, to some extent, serve their modeling purposes well. However, there is a lack of analysis on how much the form and the phrasing of the prompt matters for model performance. Further, it is not clear whether end users can judge the quality of a prompt in terms of expected model performance.

Prompt engineering NL prompts contain information not captured by model parameters, hence serving as a flexible way to inject inductive bias, and playing the role of a hyper-parameter [START_REF] Tom | Language models are few-shot learners[END_REF]. Unlike other hyper-parameters, given the huge discrete search space, searching for an optimal NL prompt that maximizes model performance is an NP-hard problem. Prior work [START_REF] Shin | Autoprompt: Eliciting knowledge from language models with automatically generated prompts[END_REF]Schick and Schütze, 2021a,b, inter alia) investigates searching for improved NL prompts with respect to model performance using gradients as signals. These approaches usually lead to non-fluent NL prompts, e.g. a list of keywords. We are concerned with understanding how robust a model is to prompts written by humans, and whether performance benefits from a prompt being authored by an NLP expert. We rely on human annotators to explore the space of NL prompts.

Another line of work [START_REF] Qin | Learning how to ask: Querying lms with mixtures of soft prompts[END_REF]Liu et al., 2021, inter alia) abandons NL questions and optimizes soft prompts in a continuous space. It allows training the prompts along with model parameters and bypasses the difficult search problem. Although soft prompts may perform similarly or better than explicit prompts, the tuning process is no longer interpretable for humans. Moreover, soft prompts themselves are no longer interpretable for humans. In low resource scenarios, where inductive biases play an important role in compensating data scarcity, NL prompts can better capture human-provided inductive biases. Templates and events We define a template as a semantic structure that consists of slots and represents an event or a conjunction of events. The definition shares similarities with that of events used throughout IE, where slot is a counterpart to argument, allowing us to recast existing IE datasets to the framework of templates and slots.

Low resource

In prior work, event semantics has been investigated in different settings. ACE is a sentence-level event task (Walker et al., 2006) which requires the detection of event trigger and arguments residing within the same sentence. RAMS [START_REF] Ebner | Multi-sentence argument linking[END_REF] contains events with arguments spanning across multiple sentences. The Granular task proposed by the BETTER program 3 further extends the paradigm to combine multiple simple sentencelevel events, entities, factuality, and their temporal relationships to form storylines (named "templates"). Although these datasets differ in complexity, they can fit under the same template framework.

We are interested in how prompt phrasing affects a model's NL understanding ability in both high and low resource scenarios. Given the above definition of templates, we use the task of filling slots in templates (Du and Cardie, 2020a;Chen et al., 2020b;Du et al., 2021a,b, inter alia)

Prompt collection

In the ontologies associated with Granular, ACE and RAMS, each template comes with an NL description. These descriptions are meant to help humans understand the scope of the ontology, and to help when annotating documents by hand. Slots may or may not have a linguistic description. For all three datasets, we asked annotators to write questions that captured the information stated in the ontology description.

Annotation protocol

First, one expert NLP practitioner wrote a question for each slot, referred to as "expert". 4 Second, we recruited non-expert annotators to write a question for each slot in a given template. 5 These annotators have no background in TE, and we view them as potential end users of the TE system. We provided the annotators with the name of the template in the ontology6 , the description of the template, and if it exists, the description of the slot. Examples are shown in Figure 1. Annotators were also shown some templates with the expert questions already filled in, to help them get started. The templates, as well as the questions within a template, were presented in a random order, mitigating any variations in the quality of the annotations due to seeing one slot at an earlier or later point in the annotation process. For the Granular ontology, 8 annotators were hired. Each produced questions for 3 templates. Templates were assigned to annotators so that all 4 templates in the ontology were annotated equally, resulting in 6 questions for each slot. Some questions were identical across multiple annotators, resulting in less than 6 distinct questions for some slots, as can be seen in Figure 1. Similarly, 5 annotators worked on the ACE and RAMS ontologies, resulting in 3 and 1 questions per slot, respectively.

Prompt aggregation

With n questions per slot, we group questions across slots into n series (shown as SERIES in the following tables). Each series covers the entirety of the dataset's templates and contains questions that come from the smallest number of different annotators possible. This is done by assigning an index to each annotator. For each slot, questions are ordered by increasing index. The i-th question is assigned to the i-th series. Variation in performance across series is thus kept as close as possible to variations in performance across annotators. Examples are shown in Figure 1, and Tables 5 and9.

To measure the diversity of the questions collected, we compute pairwise similarities of the questions filling the same slot using SBERT [START_REF] Reimers | Sentencebert: Sentence embeddings using siamese bertnetworks[END_REF]. We report detailed statistics in Table 2 and show examples in Table 3. Pairwise similarity of questions is generally high. Including expert questions does not change the similarities much, indicating a certain amount of information being shared in the embedding space. RAMS questions show lowest absolute similarity. We hypothesize that more fine-grained ontologies such as used in RAMS make annotators write diverse questions to distinguish subtle differences. For example, the AIDA ontology used for RAMS distinguishes between one-way and twoway communication, and whether communication happens in person or not. 7 Manual analysis of questions reveals that annotators tended to write questions that allow for a narrower interpretation than the expert's, as can be seen in Table 5. We attribute this to the expert's effort to formulate questions close to the meaning of the ontology, but open enough to accommodate a wide range of answers, in an attempt to balance precision and recall. Table 4 shows the statistics of questions written by experts and non-experts. We consider ACE and Granular simpler than the AIDA ontology, as they have fewer template types and slot types (shown in Table 1). For these simpler ontologies, the expert wrote shorter, somewhat less 7 e.g.

Contact:CommandOrder:Broadcast, Contact:CommandOrder:Correspondence and Contact:CommandOrder:Meet specific questions. For the complex AIDA ontology, the expert wrote longer questions, to highlight subtle differences between templates. This was sometimes done at the expense of fluency. RAMS, similarity=0.9:

Where did the money transfer take place?

Where did the transaction occur?

Table 3: Example questions and their similarities from different datasets. Expert questions are in italics.

non-expert expert ACE 5.3 ± 1.4 (6) 5.1 ± 1.5 (6) Granular 7.1 ± 2.2 (5) 6.5 ± 2.0 (7) RAMS 5.7 ± 1.8 (4) 7.0 ± 2.4 (7) 4 Model

To answer the aforementioned questions, we would like our QA model performance to depend, as much as possible, only on prompts used, excluding the effect of other factors such as finding candidate spans for slot filling. We propose a learning-torank model that consumes a given set of candidate spans and a prompt, and picks the most probable slot fillers.

Problem formulation

Let C be a text passage, or context. We refer to a specific template and slot type as y. The set of candidate spans A in C are those to be considered when predicting U y , the subset of A that fills slot y.

We also assume access to a span in C that triggers the template. 8 The candidate spans A are given as part of the TE datasets. MOVEMENT:TRANSPORTARTIFACT:SMUGGLEEXTRACT.

The smuggling or extracting of an artifact by an agent out of one place into another. <Transporter> transported <Artifact> in <Vehicle> from <Origin> place to <Destination> place. TRANSPORTER.

Who transported the artifact that was smuggled or extracted? Who smuggled or extracted an artifact? We use a prompt to query the model to predict a set of spans to fill the slot y. We denote the prompt as Q y = (q 1 , • • • , q m ), and the text passage as

C = (c 1 , • • • , c n ).
The model predicts a set of spans U y where U y ⊂ A. Our approach is illustrated in Figure 1.

Context and span encoding

We adopt an encoder-based machine reading architecture to jointly encode a prompt Q y and passage C [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF]. The prompt Q y and the text C are concatenated with special delimiters9 and passed through the underlying encoder:

ENCODER([CLS, q 1 , • • • , q m , SEP, c 1 , • • • , c n , SEP])
where CLS is a special token whose embedding represents the whole sequence, and SEP serves as the sequence separator. We denote the output encoding of each question token q i (1 ≤ i ≤ m) as q i ∈ R d and the encoding of each text token

c i (1 ≤ i ≤ n) as c i ∈ R d . We concatenate a posi- tional embedding p i ∈ R p (1 ≤ i ≤ n) to each c i ,
marking the distance in tokens of c i to the tokens representing the template trigger. The distances are non-negative integers, so that the p i are learned embeddings. We consider the set of candidate spans

A = {a 1 , • • • , a k }, where a k = (a start k , a end k ) and 1 ≤ a start k ≤ a end k ≤ n.

Span ranking and dynamic threshold

As slots can have any number of filler spans, only picking the most probable span [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF] cannot suffice. Besides, some slots might not have any fillers. Prior work such as [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF] uses the CLS token to determine whether the question is answerable. We adopt the use of CLS, and extend it with the idea of dynamic threshold from Chen et al. (2020a), to be able to support any number of answer spans. Similar to [START_REF] Lee | End-to-end neural coreference resolution[END_REF], we compute a score s(a k ) for each candidate span a k , indicating how likely it belongs to U y :

s(a k ) = FFNN([c a start k , c a end k ])
where

[c a start k , c a end k
] is the concatenation of the two vectors representing the start and end tokens of a k , and FFNN is a feed-forward neural network. We score the CLS token the same way as other candidate spans to obtain s(CLS). It then serves as a sentinel to dynamically separate relevant spans from irrelevant spans. The underlying ranking relation to be learned is:

u CLS v, ∀u ∈ U y , ∀v ∈ V y = A \ U y
where means "ranks higher than". For model training, we employ a hinge ranking loss:

J u CLS = 1 |U y | u∈Uy [α-(s(u)-s(CLS))] + J CLS v = 1 |V y | v∈Vy [β+(s(v)-s(CLS))] + J = λJ u CLS + (1 -λ)J CLS v
where the hyper-parameter λ controls the trade-off between both losses, and hyper-parameters α and β are the margins of the loss functions. During inference, we first compute scores for all candidate spans and CLS. The predicted answer set is then decoded as U y = {a | s(a) ≥ s(CLS), ∀a ∈ A}.

Experiments

Experimental settings

We run experiments on the Granular, ACE and RAMS datasets to measure the impact of using different prompts on TE-QA. We use Unified-QA [START_REF] Khashabi | Unifiedqa: Crossing format boundaries with a single QA system[END_REF] as the ENCODER described in Section 4.2. Since our experiment setting is inherently the same as by [START_REF] Ebner | Multi-sentence argument linking[END_REF] and Chen et al. (2020c), we follow prior work and use span-level F1 as the evaluation metric. We take Chen et al. (2020c) as our baseline, shown as JOINTARG in the tables, since it obtains previous state-of-the-art results on both ACE and RAMS using gold mention spans. As far as we are aware, we are the first to publish results on the Granular dataset with this experimental setup.

For each set of prompts, we use Optuna [START_REF] Akiba | Optuna: A next-generation hyperparameter optimization framework[END_REF] to tune hyper-parameters, running 20 experiments per set. Each experiment consists of 3 training runs with 3 different random seeds. 10 During the hyper-parameter sweep, we pick the best performing model on the validation set based on micro-averaged F1 scores. With the best set of hyper-parameters, we retrain with 5 distinct random seeds and report the test scores by taking the average of 5 runs. The process intends to minimize the effect of hyper-parameters other than the choice of prompts, hence illustrating the best possible performance for each set of prompts. Therefore, we are able to make a fair comparison across different types of prompts. Further details can be found in Table 10 in the Appendix.

Results and discussion

We report results in Tables 6, 7 and 8. We compare expert questions (EXPERT), questions collected in Section 3 (SERIES), and the following different types of prompts: • SPECIALTOKENS, a single token per slot, providing a randomly initialized, trainable embedding, seen as a naive soft prompt baseline in the spirit of Qin and Eisner (2021), • the NAME of the template and slot in the ontology, separated by a space, • a fluent DESCRIPTION of the template and slot, when available in the ontology. Overall, we find that the model is able to leverage the information contained in NL prompts, as evidenced by the lower performance of SPECIAL-TOKENS. The best performance is consistently achieved by prompts formulated as questions, significantly better than the fluent DESCRIPTION.

While the expert question achieves good performance on each dataset, results suggest that knowledge of TE is not necessary to write questions that yield good performance.

Surprisingly, NAME performs consistently well, being comparable to the best-performing question on 2 out of 3 datasets. NAME consists of the template name and the slot name, separated by one space. Since slot names may be shared across templates, and since models can use the slot name as a 10 Except for RAMS, where we use single runs. 61.2 ± 0.5 56.6 ± 0.9 SERIES-3 57.5 ± 0.5 49.9 ± 1.1 SERIES-4 61.0 ± 0.4 59.5 ± 1.2 SERIES-5

62.1 ± 0.4 57.9 ± 0.9 -PRETRAINING 63.8 ± 0.3 58.1 ± 1.2 SERIES-6 61.6 ± 0.5 56.3 ± 0.9 BEST 59.6 ± 0.7 58.0 ± 1.0 WORST 61.2 ± 1.0 55.8 ± 1.4 proxy for the kind of answer expected, this could help the model spot the correct answer, simply based on surface patterns. Consistent with our findings, [START_REF] Dai | Deeper text understanding for IR with contextual neural language modeling[END_REF] show, in the context of Information Retrieval, that whether a keywordbased or a fluent query performs best can depend on both the dataset the retrieval is performed on, and the retrieval method being used. We further experiment with few-shot learning by subsampling positive and negative examples in the training data so that they appear less than a certain number of times (see Figure 2). The observations above extend to the few-shot setting: the expert question outperforms other prompts for most data regimes. In addition, we find that as above, NAME consistently achieves good performance, although with a slightly shallower learning curve.

Finally, the aforementioned Unified-QA encoder is built on T5-Base. In Tables 6 to 8, "-PRETRAIN-ING" reports the results of training the model on the line above, but using T5-Base as the encoder, thus depriving it of pretraining on QA datasets. This ablation experiment allows us to examine the effect of QA pretraining, and shows that a large part of the performance indeed comes from QApretraining. In fact, the ACE, RAMS and Granular datasets lie, in that order, along a spectrum stretching from sentence-level TE to documentlevel TE. The type of QA pretraining we have used here involves mostly sentence-level, factoid questions [START_REF] Khashabi | Unifiedqa: Crossing format boundaries with a single QA system[END_REF]. As expected, it helps more for sentence-level TE, and less for documentlevel TE. While it is always possible to map TE to a QA problem, a QA-pretrained encoder is not necessarily the best-performing encoder.

The datasets used here rely on general-domain ontologies, so that the annotators possess common sense knowledge about their templates and slots. The conclusions of this study might be different if instead, the datasets relied on specialized ontolo-gies, which the annotators would not be familiar with, such as ontologies in the legal or medical domain. Such experiments are out of scope here, and we leave it to future work.

As can be seen in Table 6, picking the right set of questions is crucial for final performance. This is more challenging in the low-resource scenario, with little validation data.

Analysis of human judgements

Can humans predict performance on TE-QA? Can humans reliably pick the best-performing question for a given slot, effectively replacing the role of the dev set? For each slot, we asked human annotators to rank questions according to how well they describe the slot, without access to data or experimental results.

Annotation protocol

For each slot of each template in the Granular ontology, we presented human annotators with all questions written for that slot. 11 We asked them to rate each question on a scale of 1 to 4:

1. This is the best out of all the questions 2. This is not the best, but among the best 3. This is not the worst, but among the worst 4. This is the worst out of all the questions Annotators were allowed to give the same rating to multiple questions, especially since some questions can be quite similar. The annotators were shown the template description as well as the slot description. There is no overlap between the annotators writing questions and those scoring questions. We obtain 3 ratings per question for the Granular dataset. We performed this study on the Granular dataset only, since performance achieved by different sets of questions was noticeably different. Examples from the annotation process are shown in Table 9.

Human Ratings (3) -Aggregate Score -Question EPIDEMIPLATE. The Epidemiplate template captures key information elements in regards to various kinds of disease outbreaks. DISEASE. Mentions of the disease that is at the heart of the disease outbreak.

1 1 1 2.0 What disease caused the outbreak? 2 1 1 1.5 What is the disease causing the outbreak? 3 2 3 1.0 What disease broke out? 4 4 4 0.8 What is the disease? PROTESTPLATE. This Granular event is intended to capture the various aspects of a protest event. ARRESTED. Description or count of those arrested.

1 1 2 2.2 Who/How many were arrested at the protests? 2 3 1 1.4 Who was arrested because of the protest? 3 2 3 0.7 Who and how many were arrested? 3 2 3 0.7 Who and/or how many people have been arrested? 2 3 2 0.7 Who was arrested during or after the protest? 2 3 2 0.7 Who was arrested in the protest? 4 4 4 0.4 Who was arrested?

Table 9: Example question ratings from the Granular dataset. The expert question is in italics.

For a given slot, the ratings of each annotator implicitly established a ranking of the questions, which we use to compute an aggregated score for each question. Using the rankings instead of the ratings normalizes the annotators' ratings. We assign each question a score of 3 i=1 1 r i where r i is the rank attributed by the i-th annotator. For questions with identical ranking, we picked the lowest possible rank -e.g. 3 questions ranked first received each a score of 1 3 . Using these scores, we can pick a BEST and WORST question for each slot, as the questions that get the highest and lowest score, respectively. Examples of scores are shown in Table 9.

The scores provided by the 3 annotators have an average Pearson correlation of 31.4%. The scores on questions implicitly provided a scoring of the annotators who wrote the questions. We assigned each question-writer the average score of their questions. The scores of question-writers ranged from 1.1 to 0.8, with the expert rated a 0.9, and ranked 5th out of 9.

Looking through the prompt set labeled BEST, we find that annotators seem to have favored questions that are grammatically correct and fluent, but with unconventional typesetting. For instance, 6 questions contain a "/", as in What did this person/network do that was corrupt? and Who and/or how many people have been injured in the protest?. Similarly, 2 questions contain a "(s)" appended to the end of a word, as in What building(s) and space(s) are the protestors occupying?. In contrast, the expert questions contain neither of those, and the best performing set contains 4 "/" and no "(s)". The encoder's tokenizer, built using a form of aggregation of characters and sub-word units, keeps certain frequent words whole, but breaks up less frequent ones. Taking an example from above: What did this person do that was corrupt? is tokenized as [What, did, this, person, do, that, was, corrupt, ?] (and similarly if replacing "person" with "network"). In contrast, What did this person/network do that was corrupt? is tokenized as [What, did, this, person, /, net, work, do, that, was, corrupt, ?]. With conventional typesetting, the tokenizer keeps occurrences of the word "network" as a single token. The encoder has thus rarely, if at all, seen the token net followed by the token work, and likely cannot compose their meaning back into that of the token network. More generally, every time the tokenizer is forced to back off to a sub-word representation, word-level information potentially useful for retrieval may be lost.

Experiments

As shown in Table 6, performance achieved by BEST and WORST questions is within that of other question prompts, neither standing out as particularly good or particularly bad. To quantify more precisely whether human scores could predict F1 score performance, we ran our models trained on questions on the test set, swapping in prompts they were not trained on. We then computed, for each given model, the Pearson correlation between F1 performance (averaged over 5 runs), and the score attributed by humans to the question. We found that the correlation was less than .1 in absolute value, so that overall, human preferences do not correlate with performance.

Conclusion

We show that NL questions written by potential end users of a TE system consistently achieve the best performance on TE-QA on all three datasets. This finding extends to few-shot learning. In practice, we find that a technical background in TE is not strictly necessary to formulate good questions, meaning that end users of the TE system can be trusted to drive the task definition by writing questions. However, humans are unable to estimate the usefulness of a particular question phrasing with respect to TE-QA model performance. Our results encourage the data-driven combination of different questions to achieve better performance on TE-QA. In addition, as TE is shifting to a low resource scenario in which ontologies are built but not annotated for, our results suggest that designers of such ontologies ought to include, for each template and slot, questions that best describe the information meant to match the template and slot. 78.5 ± 1.0 85.9 ± 0.9 82.0 ± 0.4 85.7 ± 0.6 89.2 ± 0.7 86.4 ± 0.5 SERIES-2 79.9 ± 0.5 84.9 ± 0.9 82.3 ± 0.3 86.5 ± 0.5 88.9 ± 0.6 86.7 ± 0.4 -PRETRAINING 77.7 ± 1.7 38.8 ± 2.9 51.6 ± 2.9 77.9 ± 1.9 67.0 ±

Figure 2 :

 2 Figure 2: Granular, ACE and RAMS Micro F1 as a function of slot frequency in the training data. Each data point is an average over 3 training runs with different random seeds. Bands and error bars indicate the full range of values. The bottom right bar plot corresponds to ACE, with at most 1 positive and 1 negative example per slot.

Figure 3 :

 3 Figure 3: F-score as a function of the number spans in the gold answer on the Granular dataset. Each point is an average over 5 runs. Bands indicate one standard error of the mean around the average.

Figure 4 :

 4 Figure 4: F-score as a function of the number spans in the gold answer on the ACE dataset. Each point is an average over 5 runs. Bands indicate one standard error of the mean around the average.

Figure 5 :

 5 Figure 5: F-score as a function of the number spans in the gold answer on the RAMS dataset. Each point is an average over 5 runs. Bands indicate one standard error of the mean around the average.

Table 2 :

 2 Pairwise cosine similarity (in %) of questions written for the same slot, formatted as average ± standard deviation. "With expert" includes the expert question in the pool of questions.

		with expert without expert
	ACE	89.1 ± 6.5 88.8 ± 6.6
	Granular 90.5 ± 4.4 90.2 ± 4.4
	RAMS	87.4 ± 8.8
	ACE, similarity=0.7:
	For what was the defendent being tried?
	What criminal charges is the trial for?
	Granular, similarity=0.8:
	How much was this person fined?
	What monetary punishment is the individual facing?

Table 4 :

 4

Question length in words, formatted as average ± standard deviation (median).

  INJURE.] An INJURE Event occurs whenever a PERSON entity experiences physical harm. INJURE Events can be accidental, intentional or self-inflicted. INSTRUMENT-ARG. The device used to inflict the harm.

	What weapon or vehicle was used to injure?
	What device was used to cause the injury?
	What weapon was used to harm the person?
	What device was used to inflict harm?

LIFE:

Table 5 :

 5 Example questions written for the ACE (top) and RAMS (bottom) ontologies. The expert question is in italics.

Table 6 :

 6 Test scores on Granular dataset with standard error. Best scores and scores that would be the best by changing 1 standard error are bolded. The question set with the highest dev micro F1 is underlined.

	Prompt	Micro-F1	Macro-F1
	JOINTARG *	65.6	-
	SPECIALTOKENS 65.1 ± 1.5	72.6 ± 0.6
	NAME	82.0 ± 0.3 86.3 ± 0.2
	DESCRIPTION	80.8 ± 0.2	85.9 ± 0.2
	EXPERT	82.1 ± 0.2 86.2 ± 0.2
	SERIES-1	82.0 ± 0.4 86.4 ± 0.5
	SERIES-2	82.3 ± 0.3 86.7 ± 0.4
	-PRETRAINING 51.6 ± 2.9	68.8 ± 2.2
	SERIES-3	82.1 ± 0.4 85.6 ± 0.2

Table 7 :

 7 Test scores on ACE dataset with standard error. See Table6for more details.

* is not fully comparable given the difference of encoder used.

Table 8 :

 8 Test scores on RAMS dataset with standard error. See Table6for further details.

  the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4222-4235. Association for Computational Linguistics.

	part of model	hyper-parameter name description	value range
	encoder	model	Transformer model en-	allenai/unifiedqa-t5-base
			coding the paragraph
	and prompt Christopher Walker, Stephanie Strassel, Julie Medero, training corpus (LDC2006T06). Philadelphia: Lin-encoder's parameters and Kazuaki Maeda. 2006. ACE 2005 multilingual freeze Whether to freeze the	False
	guistic Data Consortium. MLP input fea-distance feature size	size of the embeddings	{8, 16, 32, 64,128, 256, 512}
	tures		encoding the distance of
	Leon Weber, Pasquale Minervini, Jannes Münchmeyer, each token to the tem-
	Ulf Leser, and Tim Rocktäschel. 2019. NLProlog: plate anchor
	Reasoning with weak unification for question an-prior type the type of prior to use	{none, embed, logit}
	to compute the scores swering in natural language. In Proceedings of the 57th Annual Meeting of the Association for Com-for each slot
	putational Linguistics, pages 6151-6161, Florence, MLP classifier dropout feature dropout in be-	[0, 0.8]
	Italy. Association for Computational Linguistics. tween the layers of the
			MLP
	Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Ji-number of units -	{32, 64, 128, 256, 512, 1024, 2048}
	wei Li. 2020. Corefqa: Coreference resolution as number of layers -	{1, 2, 3, 4}
	query-based span prediction. In Proceedings of the trainer batch size -58th Annual Meeting of the Association for Compu-learning rate -tational Linguistics, ACL 2020, Online, July 5-10, learning rate scheduler This specific scheduler	{8, 16, 32, 64, 128, 256} [1e-6, 1e-3] reduce_on_plateau
	2020, pages 6953-6963. Association for Computa-halves the learning rate
	tional Linguistics.		when the validtion score
			stops improving
	A Appendices	optimizer number of epochs	-Maximum number of	huggingface_adamw 25
			epochs to train for
		patience	-	5
	loss function	loss trade-off	see λ in Section 4	[0, 1]
		negative margin	see β in Section 4	[1e-2, 0.5]
		positive margin	see α in Section 4	[1e-2, 0.5]

Table 10 :

 10 Hyper-parameters used for the hyper-parameter search. We use Optuna[START_REF] Akiba | Optuna: A next-generation hyperparameter optimization framework[END_REF] to find good hyper-parameters for each set of prompts, running 20 experiments per set. If there is a single value under "value range", it means this parameters was set and not searched over with Optuna. Each experiment consists of 3 training runs with 3 different random seeds, using the average micro F1 score on the validation set; except for RAMS, where we use single runs. We compute the test score of the best set of hyper-parameters by running 5 training runs with distinct random seeds and taking the average score.

			micro			macro
	prompt set	P	R	F1	P	R	F1
	SPECIALTOKENS 27.8 ± 2.9	49.9 ± 3.2	35.3 ± 3.0	38.6 ± 2.4	41.5 ± 3.1	35.5 ± 2.8
	NAME	68.3 ± 1.3 47.2 ± 0.5	55.8 ± 0.4 69.1 ± 1.4 53.4 ± 1.1 57.3 ± 1.0
	DESCRIPTION	57.8 ± 2.3	64.5 ± 2.1	60.6 ± 0.7	58.4 ± 2.5	55.0 ± 1.0	53.0 ± 1.6
	EXPERT	57.0 ± 0.4 66.8 ± 0.6 61.5 ± 0.3 59.9 ± 1.2 58.5 ± 1.2 55.6 ± 1.1
	SERIES-1	54.9 ± 2.5	62.5 ± 2.8	57.9 ± 0.7	59.9 ± 1.6	58.0 ± 1.4	54.8 ± 1.3
	SERIES-2	62.0 ± 0.4	60.6 ± 1.1 61.2 ± 0.5 64.7 ± 1.0	56.5 ± 1.2	56.6 ± 0.9
	SERIES-3	50.5 ± 0.8 67.0 ± 1.6 57.5 ± 0.5	54.0 ± 1.6	54.1 ± 0.9	49.9 ± 1.1
	SERIES-4	61.3 ± 0.6	60.7 ± 0.5	61.0 ± 0.4 67.2 ± 1.9 59.4 ± 1.0 59.5 ± 1.2
	SERIES-5	57.9 ± 1.1 67.2 ± 1.8 62.1 ± 0.4 62.5 ± 1.6 60.4 ± 0.7 57.9 ± 0.9
	-PRETRAINING	59.1 ± 0.6 69.3 ± 0.4 63.8 ± 0.3 62.6 ± 1.5 60.5 ± 1.0 58.1 ± 1.2
	SERIES-6	60.4 ± 1.3	63.1 ± 1.1 61.6 ± 0.5 62.2 ± 0.9	56.5 ± 1.3	56.3 ± 0.9
	BEST	61.7 ± 1.1	58.1 ± 2.3	59.6 ± 0.7 66.6 ± 1.3 57.2 ± 1.7 58.0 ± 1.0
	WORST	56.0 ± 2.1 68.0 ± 1.2 61.2 ± 1.0 60.4 ± 1.8 59.8 ± 0.8 55.8 ± 1.4

Table 11 :

 11 Micro and macro averaged scores on Granular dataset with standard error. Best scores and scores that would be the best by changing 1 standard error are bolded. The question set with the highest dev micro F1 is underlined. ± 0.5 82.0 ± 0.3 83.8 ± 0.2 91.0 ± 0.2 86.3 ± 0.2 DESCRIPTION 73.7 ± 0.5 89.3 ± 0.5 80.8 ± 0.2 83.3 ± 0.3 91.1 ± 0.4 85.9 ± 0.2 EXPERT 80.3 ± 0.5 84.0 ± 0.7 82.1 ± 0.2 86.1 ± 0.3 88.1 ± 0.4 86.2 ± 0.2 SERIES-1

			micro			macro
	prompt set	P	R	F1	P	R	F1
	SPECIALTOKENS 62.1 ± 1.7	68.7 ± 1.6	65.1 ± 1.5	72.0 ± 0.6	75.4 ± 0.6	72.6 ± 0.6
	NAME	75.7 ± 0.4	89.5

Table 12 :

 12 Micro and macro averaged scores on ACE dataset with standard error. See Table 11 for further details. ± 0.6 82.5 ± 0.7 82.0 ± 0.1 82.8 ± 0.6 82.4 ± 0.3 81.2 ± 0.1 EXPERT 73.5 ± 0.2 90.8 ± 0.2 81.2 ± 0.1 77.8 ± 0.2 88.2 ± 0.1 81.0 ± 0.1

	2.1	68.8 ± 2.2

Table 13 :

 13 Average micro and macro scores for the RAMS dataset, with standard error of the mean. Statistics are computed over 5 runs of the best set of hyperparameters, each with a different random seed. Best scores are bolded. Scores that would be best scores if removing or adding one standard error of the mean are bolded as well. The prompt set that would have been picked based on the dev micro F1 score is in italics.

We define higher quality prompts as prompts yielding higher model performance.

In this case, prior work has been using prompt, query, and question interchangeably.

For our purposes, experts are people who know how QA models work, and have a sense of how to inject detailed information via prompts.

Annotators were paid undergraduate students without background in the task nor in NLP modeling.

e.g. ArtifactExistence:ArtifactFailure:MechanicalFailure

A trigger span in IE is the key textual evidence that supports prediction of an event or template.

We use BERT delimiters in this description, but the actual delimiters might differ depending on the encoder design.

Annotators come from the same demographics as the annotators of Section 3 and were compensated in the same way for their work.

Acknowledgements

We thank Tongfei Chen, Zhengping Jiang, Dawn Lawrie, Doug Oard, Guanghui Qin, Kate Sanders, Nathaniel Weir, and Patrick Xia for helpful comments and feedback.