Gautier Izacard

Edouard Grave

DISTILLING KNOWLEDGE FROM READER TO RETRIEVER FOR QUESTION ANSWERING

The task of information retrieval is an important component of many natural language processing systems, such as open domain question answering. While traditional methods were based on hand-crafted features, continuous representations based on neural networks recently obtained competitive results. A challenge of using such methods is to obtain supervised data to train the retriever model, corresponding to pairs of query and support documents. In this paper, we propose a technique to learn retriever models for downstream tasks, inspired by knowledge distillation, and which does not require annotated pairs of query and documents. Our approach leverages attention scores of a reader model, used to solve the task based on retrieved documents, to obtain synthetic labels for the retriever. We evaluate our method on question answering, obtaining state-of-the-art results.

INTRODUCTION

Information retrieval is an important component for many natural language processing tasks, such as question answering [START_REF] Voorhees | The TREC-8 question answering track report[END_REF] or fact checking [START_REF] Thorne | Fever: a large-scale dataset for fact extraction and verification[END_REF]. For example, many real world question answering systems start by retrieving a set of support documents from a large source of knowledge such as Wikipedia. Then, a finer-grained model processes these documents to extract the answer. Traditionally, information retrieval systems were based on hand-crafted sparse representations of text documents, such as TF-IDF or BM25 [START_REF] Sparck | A statistical interpretation of term specificity and its application in retrieval[END_REF][START_REF] Stephen E Robertson | Okapi at TREC-3[END_REF]. Recently, methods based on dense vectors and machine learning have shown promising results [START_REF] Karpukhin | Dense passage retrieval for open-domain question answering[END_REF][START_REF] Khattab | Relevance-guided supervision for openqa with colbert[END_REF]. Deep neural networks based on pre-training, such as BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], have been used to encode documents into fixed-size representations. These representations are then queried using approximate nearest neighbors [START_REF] Johnson | Billion-scale similarity search with gpus[END_REF]. These techniques have lead to improved performance on various question answering tasks.

A challenge of applying machine learning to information retrieval is to obtain training data for the retriever. To train such models, one needs pairs of queries and the corresponding list of documents that contains the information corresponding to the queries. Unfortunately, hand-labeling data to that end is time consuming, and many datasets and applications lack such annotations. An alternative approach is to resort to heuristics, or weakly supervised learning, for example by considering that all documents containing the answer are positive examples. However, these approaches suffer from the following limitations. First, frequent answers or entities might lead to false positive examples. As an example, consider the question "where was Ada Lovelace born?". The sentence "Ada Lovelace died in 1852 in London" would be considered as a positive example, because it contains the answer "London". A second limitation is that for some tasks, such as fact checking or long form question answering, such heuristics might not be applicable directly.

In this paper, we propose a procedure to learn retriever systems without strong supervision in the form of pairs of queries and documents. Following previous work [START_REF] Chen | Reading Wikipedia to answer opendomain questions[END_REF], our approach uses two models: the first one retrieves documents from a large source of knowledge (the retriever), the second one processes the support documents to solve the task (the reader). Our method is inspired by knowledge distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], and uses the reader model to obtain synthetic labels to train the retriever model. More precisely, we use a sequence-to-sequence model as the reader, and use the attention activations over the input documents as synthetic labels to train the retriever. Said otherwise, we assume that attention activations are a good proxy for the relevance of documents. We then train the retriever to reproduce the ranking of documents corresponding to that metric.

We make the following contributions:

• First, we show that attention scores from a sequence-to-sequence reader model are a good measure of document relevance (Sec. 3.2) ; • Second, inspired by knowledge distillation, we propose to iteratively train the retriever from these activations, and compare different loss functions (Sec. 3.4) ; • Finally, we evaluate our method on three question-answering benchmarks, obtaining stateof-the-art results (Sec. 4).

Our code is available at: github.com/facebookresearch/FiD.

RELATED WORK

We briefly review information retrieval based on machine learning. We refer the reader to [START_REF] Christopher D Manning | Introduction to information retrieval[END_REF][START_REF] Bhaskar Mitra | An introduction to neural information retrieval[END_REF] for a more exhaustive introduction to the subject.

Vector space models. In traditional information retrieval systems, documents and queries are represented as sparse vectors, each dimension corresponding to a different term. Different schemes have been considered to weigh the different term, the most well known being based on inverse document frequency, or term specificity [START_REF] Sparck | A statistical interpretation of term specificity and its application in retrieval[END_REF]. This technique was later extended, leading to the BM25 weighting scheme which is still widely used today [START_REF] Stephen E Robertson | Okapi at TREC-3[END_REF]. A limitation of sparse representations is that the terms of the query need to match the terms of the returned documents. To overcome this, [START_REF] Deerwester | Indexing by latent semantic analysis[END_REF] proposed to use latent semantic analysis for indexing, leading to low-dimension dense representations of documents.

Neural information retrieval. Following the success of deep learning for other natural processing tasks, neural networks were applied to the task of information retrieval. [START_REF] Huang | Learning deep structured semantic models for web search using clickthrough data[END_REF] proposed a deep bag-of-words model, where queries and documents were embedded independently, a technique known as bi-encoder. Documents were then ranked by using the cosine similarity with the query, and the model was trained on clickthrough data from a search engine. This technique was later extended by using convolutional neural networks [START_REF] Shen | Learning semantic representations using convolutional neural networks for web search[END_REF] and recurrent neural networks [START_REF] Palangi | Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval[END_REF]. A limitation of independently embedding documents and query is that it does not capture fine-grained interactions between the query and documents. This lead [START_REF] Nogueira | Passage re-ranking with bert[END_REF] and [START_REF] Yang | End-to-end open-domain question answering with BERTserini[END_REF] to use a BERT model to jointly embed documents and query, a technique known as cross-encoder.

End-to-end retrieval. Most of the methods described in the previous paragraph were used to rerank a small number of documents, usually returned by a traditional IR systems. In the context of ad-hoc document retrieval, [START_REF] Gillick | End-to-end retrieval in continuous space[END_REF] showed that bi-encoder models could be competitive with traditional IR systems. For open domain question-answering, [START_REF] Karpukhin | Dense passage retrieval for open-domain question answering[END_REF] introduced dense passage retrieval (DPR), which uses dense embeddings and nearest neighbors search. More precisely, question and passage embeddings are obtained using a BERT-based biencoder model, which is trained on a small dataset of question and passage pairs. Then, the full knowledge source (Wikipedia) is encoded using this model, and passages are queried by computing the k-nearest neighbors of the embedding of the question. Jointly embedding the query and documents makes the application of cross-encoder models intractable to large database. To address this limitation, [START_REF] Humeau | Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring[END_REF] introduced the poly-encoder architecture, in which each documents is represented by multiple vectors instead of one. Similarly, [START_REF] Khattab | Relevance-guided supervision for openqa with colbert[END_REF] proposed a scoring function where each term of the query and documents is represented by a single vector. To make the method tractable, their system retrieves documents with an approximate score, which are then re-ranked with the exact one. Finally, [START_REF] Luan | Sparse, dense, and attentional representations for text retrieval[END_REF] conducts a theoretical and empirical study of sparse, dense and cross-attention information retrieval systems.

Unsupervised learning. Closest to our work, there is growing body of work trying to learn information retrieval systems from unsupervised data. [START_REF] Lee | Latent retrieval for weakly supervised open domain question answering[END_REF] introduced the inverse cloze task for pre-training retrievers, which can then be fine-tuned end-to-end on question-answering tasks. This pre-training scheme was later evaluated for ad-hoc document retrieval by [START_REF] Chang | Pre-training tasks for embedding-based large-scale retrieval[END_REF]. [START_REF] Guu | Realm: Retrievalaugmented language model pre-training[END_REF] proposed to augment language model pre-training with a retriever module, which is trained using the masked language modeling objective. Similarly, Lewis et al. (2020a) introduced a sequence-to-sequence model that is pre-trained by generating a target text, after retrieving a set of related texts. Lewis et al. (2020b) further train the retriever obtained in [START_REF] Karpukhin | Dense passage retrieval for open-domain question answering[END_REF] by backpropagating to the retriever the error between the generated output and the gold answer.

Simultaneously to our work, [START_REF] Yang | Is retriever merely an approximator of reader?[END_REF] proposes to train a retriever with knowledge distillation. The main difference with our method is the nature of the synthetic labels that are used to train the retriever. [START_REF] Yang | Is retriever merely an approximator of reader?[END_REF] uses the DPR reader, which includes a classifier that predicts which passage contains the answer, and can be seen as a cross-encoder reranker. This technique thus performs the distillation of a cross-encoder retriever to a bi-encoder retriever. In contrast, our method uses the internal attention scores of the reader, which does not require additional supervision besides pairs of question and answer.

METHODOLOGY

Our system is composed of two modules, the retriever and the reader, following the standard pipeline for open-domain question answering. Given an input question these modules are used in a two-step process to generate an answer. First the retriever selects support passages in a large knowledge source. Then these passages are processed by the reader, along with the question, to generate an answer. For the reader module we use the Fusion-in-Decoder model [START_REF] Izacard | Leveraging passage retrieval with generative models for open domain question answering[END_REF], which achieves state-of-the-art performance when combined with BM25 or DPR [START_REF] Karpukhin | Dense passage retrieval for open-domain question answering[END_REF].

It is based on a sequence-to-sequence architecture, and is initialized from pre-trained models such as T5 or BART [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF][START_REF] Lewis | BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension[END_REF].

The focus of this work is to train the retriever without strong supervision or weakly supervised learning based on heuristics. For this we propose to train the retriever by learning to approximate the attention score of the reader. The training scheme outlined here can be seen as a student-teacher pipeline, where the teacher, the reader module, produces targets which are used to train a student network, the reader. By doing so, we hope to leverage the signal extracted from the question-answer pairs by the reader. Since the goal of the retriever is to retrieve the most relevant passages, by training the retriever to estimate the reader attention scores, we implicitly make the assumption that these scores are a good proxy for the usefulness of a passage to answer the question.

In this section we will first describe the Fusion-in-Decoder architecture, before elaborating on the signal which is used to train the retriever, the design of the retriever, and how this module is trained.

CROSS-ATTENTION MECHANISM

First, let us briefly review the Fusion-in-Decoder model (FiD, Izacard & Grave, 2020). The underlying architecture is a sequence-to-sequence model, composed of an encoder and a decoder. The encoder independently processes n p different text inputs (s k ) 1≤k≤np . In the case of open-domain question answering based on Wikipedia, each input s k is the concatenation of the question q and a support passage, with special tokens question:, title: and context: added before the question, the title of the Wikipedia article and the text of each passage. The output representations of the encoder are then concatenated to form a global representation X of dimension

( k k ) × d,
where k is the length of the k-th segment and d is the dimension of the embeddings and hidden representations of the model. Then, the decoder processes this representation as a regular autoregressive model, alternating self-attention, cross-attention and feed-forward modules.

Only the cross-attention module explicitly takes as input the global output representation X of the encoder. If H ∈ R d denotes the output of the previous self-attention layer of the decoder, the crossattention operation consists in the following operations. First, queries Q, keys K and values V are computed by applying linear transformations:

Q = W Q H, K = W K X, V = W V X.
Then a similarity score between the query at position i, Q i , and the key at position j, K j , is obtained by computing the dot-product between these two elements, and normalized over the dimension:

α i,j = Q T i K j , αi,j = exp(α i,j ) m exp(α i,m )
.

A new representation is obtained as a sum of the values, weighted by the attention probabilities, before going through a final linear transformation W o :

O i = W O j αi,j V i,j
The operations described above are performed in parallel with different linear transformations in the case of multi-head attention. Finally a normalization layer is applied, and this pipeline is wrapped by a skip connection. See [START_REF] Vaswani | Attention is all you need[END_REF] for more details on the structure of Transformers.

CROSS-ATTENTION SCORE AS A RELEVANCE MEASURE FOR PASSAGE RETRIEVAL

In some sense, the attention scores α :,j involving the j-th key measures the importance of this key, and corresponding value, to compute the next representation. We hypothesize that it is good proxy to estimate the relevance of a passage -the more the tokens in a text segment are attended to, the more relevant the text segment is to answer the question.

Given the reader model, an input question q and a corresponding set of support passages D q = (p k ) 1≤k≤n , we obtain relevance scores (G q,p k ) 1≤k≤n for each passage by aggregating attention scores. In particular, the score G q,p k is obtained by averaging the pre-attention scores α 0,: over all the tokens in the input s k corresponding to the passage p k , all the layers and all the heads of the decoder. Note that the FiD decoder jointly processes the passages, and thus the score G q,p k depends on the other support passages. We consider other pooling operators, such as max, to aggregate attention scores over layers, heads and tokens and empirically compare them in Sec. 5.2.

Before we proceed, let us consider the following simple experiment, which is a first indication that reader attention scores are indeed a strong relevance signal. Given a question and 100 passages retrieved with DPR, our goal is to select the 10 best passages. When using the top 10 passages from DPR instead of the top 100, the performance of our reader drops from 48.2 EM to 42.9 EM. On the other hand, if we select the top 10 documents according to the attention scores, the performance only drops to 46.8 EM.

DENSE BI-ENCODER FOR PASSAGE RETRIEVAL

Ideally, we would like to rank passages according to the reader cross-attention scores. In practice however, since the passages and the question need to be processed simultaneously by the reader module it is impractical to query a large knowledge source this way. Thus, we use a retriever model composed of an embedder function E that maps any text passage to a d-dimensional vector, such that the similarity score between a question q and a passage p is defined as S θ (q, p) = E(q) T E(p). This similarity metric enables us to index all passages in the knowledge source as a preprocessing step.

Then at runtime, passages with the highest similarity score with the input question are retrieved, by using an efficient similarity search library such as FAISS [START_REF] Johnson | Billion-scale similarity search with gpus[END_REF].

For the embedder we use BERT and follow DPR by considering that the encodings E(q) and E(p) are obtained by extracting the representation of the initial [CLS] token. This leads to a representation of dimension d = 768 in the case of a base model. Differently from DPR, we use the same encoding function E for the questions and passages by sharing parameters.

DISTILLING THE CROSS-ATTENTION SCORE TO A BI-ENCODER

In this section, we describe how to train the retriever model, based on the relevance scores obtained in Sec. 3.2. For the training objective of the retriever, we propose to minimize the KL-divergence between the output S θ (q, p) and the score G q,p after normalization:

L KL (θ, Q) = q∈Q,p∈Dq
Gq,p (log Gq,p -log Sθ (q, p)), where Gq,p = exp(G q,p )

p ∈Dq exp(G q,p )

, Sθ (q, p) = exp(S θ (q, p))

p ∈Dq exp(S θ (q, p ))

.

In Sec. 5.1 we present results obtained when using alternatives to this training objective. We consider two other objectives which have been used in [START_REF] Dehghani | Neural ranking models with weak supervision[END_REF], where BM25 is used as a teacher model to train a neural ranker. A first option consists in training the retriever with a regression approach by minimizing the mean squared error:

L MSE (θ, Q) = q∈Q,p∈Dq (S θ (q, p) -G q,p ) 2 .
The second option we consider is to use a max-margin loss that explicitly penalizes inversions in the ranking estimated by the retriever:

L ranking (θ, Q) = q∈Q,p1,p2∈Dq
max (0, γsign(G q,p1 -G q,p2 )(S θ (q, p 1 ) -S θ (q, p 2 ))) .

In words, if p 1 is more relevant to answer the question q than p 2 , i.e. G q,p1 > G q,p2 , the loss pushes the retriever score of p 1 to be larger than the score of p 2 by at least a margin of γ.

ITERATIVE TRAINING

In this section, we explain how iterative training can be used with the student-teacher scheme described in the previous section, similarly to [START_REF] Khattab | Relevance-guided supervision for openqa with colbert[END_REF]. This iterative procedure can be interpreted as using the current retriever to sample negative examples, in order to train a new retriever. When learning a retriever with discriminative training, negative samples play an important role, and various strategies have been considered in previous work. [START_REF] Karpukhin | Dense passage retrieval for open-domain question answering[END_REF] compared random sampling with using the top-k passages from BM25 which do not contain the answer and with using the positive passages from other queries. Consider that for each question, we have an initial set of support documents D 0 q . We propose to use an iterative pipeline where each iteration can be described as the following 4-step process:

1. Train the reader R using the set of support documents for each question D 0 q . 2. Compute aggregated attention scores (G q,p ) q∈Q,p∈D 0 q with the reader R. 3. Train the retriever E using the scores (G q,p ) q∈Q,p∈D 0 q . 4. Retrieve top-passages with the new trained retriever E.

This multi-step procedure can be repeated multiple times. A critical point of the training procedure is the initial set of documents corresponding to each question. In Sec. 4, we compare retrievers obtained by starting from documents obtained using BM25 or cosine similarity from a BERT model. In particular, we show that while the initial performance with BERT is low, the iterative procedure allows to greatly improve the performance of the model.

EXPERIMENTS

In this section we evaluate the student-teacher training procedure from the previous section. We show that we obtain competitive performance without strong supervision for support documents.

EXPERIMENTAL SETTING

Datasets. We perform experiments on TriviaQA [START_REF] Joshi | Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension[END_REF] and NaturalQuestions [START_REF] Kwiatkowski | Natural Questions: a benchmark for question answering research[END_REF], two standard benchmarks for open-domain question answering. TriviaQA is made of questions from trivia and quiz league websites, and does not contain gold support documents. NaturalQuestions contains questions corresponding to web search queries, and gold support documents from Wikipedia. Following the setting from [START_REF] Lee | Latent retrieval for weakly supervised open domain question answering[END_REF] (2017).

We also evaluate on NarrativeQuestions [START_REF] Kočiskỳ | The NarrativeQA reading comprehension challenge[END_REF], using a publicly available preprocessed version.1 This is a reading comprehension dataset built on a corpus of books and movie scripts. For each story, questions are generated by human annotators based on a summary of the given document. We consider the full story setting, where the task is to answer questions given the entire story and not the summary used to generate question-answer pairs. Here the knowledge source is not the same for all questions: given a question the retrieval operation is performed on all passages of the associated story. These passages are obtained by dividing the story in chunks of 100 words. These stories are long documents, with an average of 60k words. While part of the documents could be processed entirely by the Fusion-in-Decoder module, it is interesting to limit the number of support passages to reduce the computational cost of the reading step.

While answers in TriviaQA and NaturalQuestions are short, NarrativeQA answers are about five words long on average, with medium length answers such as "He dismantles it and attaches it to his mother's jeep" which answers the question "What does Mark do with his radio station?". Notably a significant number of answers do not correspond to spans in the story. It is thus not straightforward to train the retriever with heuristics using question-answer pairs. In our case we use the same pipeline as for TriviaQA and NaturalQuestions, demonstrating the flexibility of our approach.

Evaluation. The model performance is assessed in two ways. First, following previous work such as DPR and ColbertQA, we report the top-k retrieval accuracy (R@k), which is the percentage of questions for which at least one passage of the top-k retrieved passages contains the gold answer. It is unclear how well this metric evaluates the retriever performance, since the answer can be contained in a passage without being related to the question. This is notably true for common words or entities.

We also report the final end-to-end performance of the question answering system composed of the retriever and reader modules. This is the metric we are fundamentally interested in. For TriviaQA and NaturalQuestions, predicted answers are evaluated with the standard exact match metric (EM), as introduced by [START_REF] Rajpurkar | SQuAD: 100,000+ questions for machine comprehension of text[END_REF]. For NarrativeQA we report the metrics proposed in the original paper: ROUGE-L, BLEU-1, BLEU-4 and METEOR.

TECHNICAL DETAILS

Initialization. Similarly to DPR, we initialize the retriever with the BERT base model, pretrained with uncased text. The Fusion-in-Decoder reader is initialized with the T5 base model. A critical component of the iterative training procedure is the initialization of the support passages D 0 q asso-Model NQ TriviaQA dev. test dev. test DPR [START_REF] Karpukhin | Dense passage retrieval for open-domain question answering[END_REF] -41.5 -57.9 RAG (Lewis et al., 2020b) -44.5 -56.1 ColBERT-QA [START_REF] Khattab | Relevance-guided supervision for openqa with colbert[END_REF] -48.2 -63.2 Fusion-in-Decoder (T5 base) [START_REF] Izacard | Leveraging passage retrieval with generative models for open domain question answering[END_REF] -48.2 -65.0 Fusion-in-Decoder (T5 large) [START_REF] Izacard | Leveraging passage retrieval with generative models for open domain question answering[END_REF] -51.4 -67.6

Ours (starting from BERT, T5 base) 39.3 40.0 62.5 62.7 Ours (starting from BM25, T5 base) 47.9 48.9 67.7 67.7 Ours (starting from DPR, T5 base) 48.0 49.6 68.6 68.8 Ours (starting from DPR, T5 large) 51.9 53.7 71.9 72.1

Table 2: Comparison to state-of-the-art models on NaturalQuestions and TriviaQA.

ciated with each question q. For this we consider different options. The first one is to use passages retrieved using BM25. We use the implementation from Apache Lucene2 with default parameters, and tokenize questions and passages with SpaCy3 . We also use passages obtained with BERT as a retriever without fine-tuning, this leads to poor initial performance. Finally in Table 2 we show that initializing D 0 q with passages obtained with DPR [START_REF] Karpukhin | Dense passage retrieval for open-domain question answering[END_REF] outperforms the two previous initializations. We train all retrievers using 100 passages. For the reader, we use 100 passages for NaturalQuestions and TriviaQA and 20 passages for NarrativeQA.

Iterative training. We apply the iterative training procedure on each dataset independently. Both the reader and the retriever are fine-tuned using the ADAM algorithm [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF], with a batch of size 64. The reader is trained for 10k gradient steps with a constant learning rate of 10 -4 , and the best model is selected based on the validation performance. The retriever is trained with a constant learning rate of 5 • 10 -5 until the performance saturates. To monitor the performance of the retriever during training, we measure the similarity between the reader and the retriever rankings. At each new training iteration the reader is reinitialized from T5 base, while we pursue the training of the retriever. We found that restarting from T5 base is important for the first iterations when starting with BERT documents. We have not tried to reinitialize the retriever between each iteration. More details on the hyperparameters and the training procedure are reported in Appendix A.2.

RESULTS

In Table 1, we report the performance of our approach for different number of self-training iterations. Generally, we observe that the accuracy of our system increases with the number of iterations, obtaining strong performance after a few iterations. Interestingly, while the initial performance with documents retrieved with BERT is very poor, our method still reach competitive scores on TriviaQA, and to a lesser extent, NaturalQuestions. However, a second observation is that the quality of the initial document sets plays an important role on the performance of the end system. Indeed, we observe that starting the procedure from BM25 documents, which are higher quality as indicated by the performance of the system at iteration 0, leads to stronger results than using BERT documents. An interesting research question would be to explore pre-training of the initial BERT model for retrieval, for example by using the inverse cloze task.

In Table 2, we report the performance of our approach, as well as existing state-of-the-art systems on TriviaQA and NaturalQuestions. In addition to initializing our method with documents retrieved with BM25 and BERT, we also train a system by starting from DPR documents. First, we observe that our method improve the performance over the state-of-the-art, even when starting from BM25 documents. This validates our assumption that it is possible to obtain strong retrievers without the need of supervision for the documents. Second, when starting from DPR passages, our method leads to a +4.5 EM improvement on TriviaQA and +2.3 EM improvement on NaturalQuestions when the final evaluation is carried out with a large reader.

Method

Iter. In Table 3, we report the performance of our method on the NarrativeQA dataset. We use the setting where the knowledge source corresponds to the whole document, and in particular, we do not use the summary. We compare our results to the best ones reported in the original paper for this setting. Similar to results obtained on NaturalQuestions and TriviaQA, we observe that training the retriever by using the attention scores of the reader leads to improvements, compared to the BM25 baseline.

Rouge-L

ABLATIONS

In this section, we investigate design choices regarding two key elements of our approach: the training objective and the aggregation of cross-attention scores. For all experiments, we consider a simplified experimental setting: a single training iteration is performed on NaturalQuestions, starting from BM25 passages.

TRAINING OBJECTIVES

In Table 4 we report the performance of our model trained with the different training objectives described in Sec. 3.3. We observe that using the KL-divergence between the aggregated scores of the reader and the scores of the retriever outperforms the other objective functions.

Method R@5 R@20 R@100 Dev EM We report all the metrics on the validation set.

HOW TO AGGREGATE CROSS-ATTENTION SCORES?

In Section 4 the cross-attention scores α are aggregated in a specific way, in order to obtain a single scalar used to train the retriever. Formally let us denote by α i,j,k,h the cross-attention scores between token i of the output and token j of the input, for the k-th layer and h-th head. Then, the scores G q,p for p ∈ D q used in Section 4 are computed as follows:

G q,p = mean j,k,h α 0,j,k,h ,
where j describes the input tokens corresponding to p. In Table 5 we explore alternatives to this choice by considering different aggregation schemes. In particular, we consider (1) taking the max over the input tokens corresponding to passage p instead of the average, (2) taking the average over the output tokens instead of taking the score of the first token, (3) taking the mean over the last six layers instead of all the layers, (4) taking the max over the layers instead of the average, (5) taking the max over the heads instead of the average. We observe that the performance of our approach is relatively stable to the choice of aggregation, and that the best result is obtained by averaging, except over the output tokens where it is best to only consider the first token.

Method R@5 R@20 R@100 Dev EM (0) mean j,k,h α 0,j,k,h 64.7 76.4 84.3 46.8 (1) mean k,h max j α 0,j,k,h 61.2 72.5 81.0 46.0 (2) mean i,j,k,h α i,j,k,h 63.5 75.3 83.1 45.8 (3) mean 7≤k≤12,j,h α 0,j,k,h 64.1 75.7 83.8 46.4 (4) mean j,h max k α 0,j,k,h 63.9 75.5 83.7 46.5 (5) mean j,k max h α 0,j,k,h 64.2 76.1 83.9 46.8

Table 5: Comparison of attention aggregation schemes on NaturalQuestions after one iteration. The index i corresponds to output tokens, j corresponds to input tokens of a given passage, h to heads and k to layers of the decoder. We report all metrics on the validation set.

CONCLUSION

In this paper, we introduce a method to train an information retrieval module for downstream tasks, without using pairs of queries and documents as annotations. Our approach is inspired by knowledge distillation, where the retriever module corresponds to the student model and the reader module corresponds to the teacher model. In particular, we use the cross-attention scores, from a sequenceto-sequence reader, to obtain synthetic targets for the retriever. We compare different ways to aggregate the scores, as well as different training objectives to learn the retriever. We show that iteratively training the reader and the retriever leads to better performance, and obtain state-of-the-art performance on competitive question answering benchmarks. In the future, we would like to explore better pre-training strategies for the retriever module, as well as better scoring functions for the retriever.

NaturalQuestions

TriviaQA Iter.

R@20 R@100 Dev EM R@20 R@100 Dev EM

Table 1 :

 1 Iterative training starting with documents retrieved with BERT and BM25. Iteration 0 corresponds to the performance of the reader trained on the set of initial support documents. We report all metrics on the validation set. et al. (2020), we use the original evaluation set as test set, and keep 10% of the training data for validation. We use the Wikipedia dump from Dec. 20, 2018 for support documents, splitting articles into non-overlapping passages of 100 tokens, and applying the same preprocessing as Chen et al.

	; Karpukhin

Table 3 :

 3 Performance on NarrativeQA.

	Bleu-1	Bleu-4	Meteor
	dev. test dev. test dev. test dev. test

Table 4 :

 4 Comparison of training objectives on NaturalQuestions after one iteration.

	Mean Squared Error	46.5	61.2	73.9	40.6
	Max-margin loss, γ = 1	60.3	73.6	82.7	45.4
	Max-margin loss, γ = 0.2 60.3	73.5	82.6	45.8
	Max-margin loss, γ = 0.1 60.2	73.5	82.6	45.1
	KL-divergence	64.7	76.4	84.3	46.8

Table 7 :

 7 Iterative training starting with documents retrieved with DPR. Iteration 0 corresponds to the performance of the reader trained on the set of initial support documents. We report all metrics on the validation set. Contrary to results reported in Table1, the reader model was not re-initialized between each iteration.

	0	77.1	84.3	46.4	78.2	84.7	65.0
	1	80.3	86.7	47.8	81.4	86.4	67.1
	2	82.4	87.9	48.2	83.5	87.4	68.1

Table 8 :

 8 Retrieval performance on the test set depending on the initial passages.

https://cs.nyu.edu/ ˜kcho/NarrativeQA

lucene.apache.org

spacy.io

For TriviaQA we use the unique human-generated answer to train the reader. In this dataset part of the answers are in uppercase. We normalize uppercase answers by converting the first letter in each word to uppercase and remaining characters to lowercase using the title Python string method.

For NarrativeQA, questions and answers in uppercase are converted to lowercase.

A.2 TRAINING

For every datasets, both the reader and the retriever are fine-tuned with a dropout rate of 10%. All models at the exception of the large reader are trained using the ADAM algorithm (Kingma & Ba, 2014) with a constant learning rate of 10 -4 for the base reader and 5 • 10 -5 for the retriever. The base reader is trained for 10k gradient steps with a batch size of 64. We train the large reader with the ADAMW algorithm [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] with a peak learning rate of 5 • 10 -5 and a linear warmup for 600 gradient steps followed by a linear decrease of the learning rate for 14.4k gradient steps.

We perform model selection on the validation performance. The retriever is trained until its performance saturates with a batch size of 64. To monitor the performance of the retriever during training, we measure the similarity between the ranking obtained with the reader score, and the ranking of the retriever. We use different metrics for this: the number of inversions between the two rankings, the proportion of passages in the retriever top-k that are also in the reader top-k and the number of passages to obtain all top-k passage of the reader.

During training and at test time, each text input of the encoder is restricted to be at most 250 token long. For NaturalQuestions and TriviaQA, we use wikipedia as a knowledge source, thus for each passage there is an associated article title. Each input is composed of the concatenation of a question, title and support passage with special tokens question:, title: and context: added before the question, the title and the text of each passage. In the case of NarrativeQA, the question and each passage are concatenated to form the different inputs.

A.3 INFERENCE

At test time, for TriviaQA and NaturalQuestions we use greedy decoding, and Beam Search with 3 beams for NarrativeQA.


Emna Amdouni
email: emna.amdouni@lirmm.fr

Clement Jonquet
email: jonquet@lirmm.fr

FAIR or FAIRer? An integrated quantitative FAIRness assessment grid for semantic resources and ontologies

Keywords: FAIR data principles, FAIRness assessment, evaluation grid, semantic Web, ontologies, semantic resources / artefacts, metadata properties

In open science, the expression "FAIRness assessment" refers to evaluating to which degree a digital object is Findable, Accessible, Interoperable, and Reusable. Standard vocabularies or ontologies are a key element to achieving a high level of FAIRness (FAIR Principle I2) but as with any other data, ontologies have themselves to be FAIR. Despite the recent interest in the open science and semantic Web communities for this question, we have not seen yet a quantitative evaluation method to assess and score the level of FAIRness of ontologies or semantic resources in general (e.g., vocabularies, terminologies, thesaurus). The main objective of this work is to provide such a method to guide semantic stakeholders in making their semantic resources FAIR. We present an integrated quantitative assessment grid for semantic resources and propose candidate metadata properties -taken from the MOD ontology metadata model-to be used to make a semantic resource FAIR. Aligned and nourished with relevant FAIRness assessment state-of-the-art initiatives, our grid distributes 478 credits to the 15 FAIR principles in a manner which integrates existing generic approaches for digital objects (i.e., FDMM, SHARC) and approaches dedicated to semantic resources (i.e., 5-stars V, MIRO, FAIRsFAIR, Poveda et al.). The credits of the grid can then be used for implementing FAIRness assessment methods and tools.

Introduction

In 2014, a group of researchers, research institutions, and publishers (called FORCE 11) defined fundamental guiding principles called FAIR (for Findable, Accessible, Interoperable, and Reusable) to make scientific data and their metadata interoperable, persistent, and understandable for both humans and machines [START_REF] Wilkinson | The FAIR Guiding Principles for scientific data management and stewardship[END_REF]. The FAIR principles emphasize the importance of semantic technologies in making data interoperable and reusable. However, ontologies -the backbone of semantic technologies-have 1 themselves to be FAIR. Until recently, not much attention has been made to quantitatively evaluating ontologies using FAIR principles; all related work or state-of-the-art methods regarding ontologies are qualitative i.e., proposing recommendations and best practices without providing a scoring mechanism. It is clear that the development of FAIRness assessment methods -i.e., ways to measure to which level a digital object implements FAIR principles-remains challenging [START_REF] Wilkinson | A design framework and exemplar metrics for FAIRness[END_REF], including for ontologies and semantic resources. In fact, the complexity of FAIRness assessment is due to the fact that the FAIR principles are expressed at a very generic level and need to be expanded and projected to specific digital objects to be more explicit. Furthermore, some criteria are very hard to evaluate by a program and sometimes require subjectivehuman expertise. For all these reasons, we believe it is essential to define a quantitative method i.e., a metric, for assessing and scoring to which degree a semantic resource is FAIR compliant -for example, to determine if a resource is "not FAIR", "FAIR" or even "FAIRer" than a certain threshold or another resource. The objective of this work is to provide a grid dispatching different values of credits to each FAIR principle, depending on its importance when assessing the FAIRness of ontologies. We talked about an integrated grid, to capture that our grid is aligned and nourished by existing generic approaches for digital objects in general (i.e., FDMM, SHARC) and approaches dedicated to semantic resources or artifacts (i.e., 5-stars V, MIRO, FAIRsFAIR, Poveda et al.). As a result, the proposed grid involves 478 credits that can be used for implementing FAIRness assessment tools. Such tools will then guide semantic stakeholders in (i) making their semantic resources FAIR and (ii) selecting relevant FAIR semantic resources for use. The rest of this paper is organized as follows: Section 2 presents related work in FAIRness assessment or alike. Section 3 describes the methodology followed to integrate the most prominent existing works into one schema and details the proposed FAIRness assessment grid. Section 4 presents candidate metadata properties -taken from the MOD 1.4 ontology metadata model-to be used to enable FAIRness assessment. Finally, Section 5 concludes and gives our perspective of developing a methodology to automatize FAIRness assessment.

Overview of related work for assessing FAIRness

We distinguish between two FAIRness assessment approaches: the first category concerns general schemes or generic tools applicable for any kind of digital object; the second category is specific for the description and assessment of ontologies or semantic resources. We review both of them chronologically.

Generic FAIRness assessment approaches

The Research Data Alliance (RDA) SHARing Rewards and Credit (SHARC) Interest Group, created in 2017, proposed a FAIRness assessment grid to enable researchers and other data stakeholders to evaluate FAIR implementations and provide the appropriate means for crediting and rewarding to facilitate data sharing [START_REF] David | FAIRness Literacy: The Achilles' Heel of Applying FAIR Principles[END_REF]. The SHARC grid defines a set of 45 generic criteria with importance levels (essential, recommended, or desirable) evaluated by answering one of four values (Never/NA, If mandatory, Sometimes, Always) to a question; questions are sometimes dependent on one another as in a decision tree.

In 2018, the RDA FAIR Data Maturity Model (FDMM) Working Group recommended normalizing FAIRness assessment approaches and enabling comparison of their results [START_REF] Bahim | The FAIR Data Maturity Model: An Approach to Harmonise FAIR Assessments[END_REF]. It describes a set of 47 generic criteria derived from each FAIR principle with priorities (essential, important, or useful). Both the SHARC grid and the FDMM recommendation assumed that some FAIR principles were more important than others. We have kept this philosophy in our methodology and kept the SHARC and FDMM outputs as influences on our FAIRness assessment score. Some FAIRness assessment tools recently appeared, such as FAIRdat tool [START_REF]SurveyMonkey Powered Online Survey[END_REF], FAIR self-assessment tool [START_REF]FAIR self-assessment tool[END_REF], OzNome 5-star tool [START_REF] Cox | OzNome 5-star Tool: A Rating System for making data FAIR and Trustable[END_REF], FAIR Metrics [START_REF] Wilkinson | Evaluating FAIR maturity through a scalable, automated, community-governed framework[END_REF], FAIR-Aware [START_REF] Mokrane | FAIR-Aware[END_REF], F-UJI . We cannot unfortunately detail them here. These tools are relevant but need to be 2 improved in order to provide a clear methodology and a tool to assess any digital object quantitatively.

Specific FAIRness assessment approaches

Before the emergence of the FAIR principles in 2011, Berners-Lee presented the foundational principles for Linked Open Data (LOD) [START_REF] Bizer | Linked Data: The Story so Far[END_REF] for making data available, shareable, and interconnected on the Web. The FAIR principles have been proposed for similar reasons with a stronger emphasis on data reusability (consideration of license agreement and provenance information). The 5-stars LOD principles were specialized in 2014 for Linked data vocabularies [START_REF] Janowicz | Five stars of Linked Data vocabulary use[END_REF] as five rules to follow for creating and publishing "good" vocabularies. Under this scheme, stars denote the quality, leading to better structure (i.e., use of W3C standards) and interoperability for reuse (i.e., metadata description, reuse of vocabularies, and alignment). The proposed 5-star rating system (later called 5-stars V) for vocabularies is simple. However, no implementation tool was developed for making the assessment automatic, and the principles are not largely referenced today. A study of the degree to which the FAIR principles align, and extend the 5-star LOD principles was proposed first in [START_REF] Hasnain | Assessing FAIR Data Principles Against the 5-Star Open Data Principles[END_REF] and later in [START_REF] Garijo | Best Practices for Implementing FAIR Vocabularies and Ontologies on the Web[END_REF]; we have incorporated this alignment in our methodology.

In 2017, the Minimum Information for Reporting an Ontology initiative published the MIRO guidelines for ontology developers when reporting an ontology in scientific reports [START_REF] Matentzoglu | MIRO: guidelines for minimum information for the reporting of an ontology[END_REF]. The MIRO guidelines aim to improve the information content descriptions' quality and consistency, including development methodology, provenance, and context of reuse information. They define 34 information items (such as "ontology name", "ontology license", "ontology URL") and specify the level of importance "must'', "should', "optional'' for each. This work was significant, but there have been no studies on how the MIRO properties align with or extend the FAIR principles. However, the MOD 1.4 metadata model (see hereafter) aligned each MIRO guideline and the corresponding metadata properties in MOD. We, therefore, used this alignment in our methodology to influence the FAIRness assessment score with the MIRO guidelines.

Dutta et al. [START_REF] Dutta | New Generation Metadata Vocabulary for Ontology Description and Publication[END_REF] reviewed and harmonized existing metadata vocabularies and proposed a unified ontology metadata model called MOD (for Metadata for Ontology Description) to facilitate manual and automatic ontology descriptions, identification, and selection. MOD is not another standard nor another metadata vocabulary, but more a set of cataloged and regrouped properties one can use to describe a semantic resource. For instance, MOD does not require the use of a specific authorship property but rather encodes that dc:creator; schema:author, foaf:maker, or pav:createdBy can be used to say so. The MOD 1.2 model later extended in MOD1.4 was used in AgroPortal -a 3 vocabulary and ontology repository for agronomy-to implement a richer, unified metadata model [START_REF] Hugo | D2.5 FAIR Semantics Recommendations Second Iteration[END_REF]. With this implementation, AgroPortal affirms to recognize 346 properties from 15 relevant metadata vocabularies (such as Dublin Core, Ontology Metadata Vocabulary, VoID, FOAF, Schema.org, PROV-O, DCAT, etc.) and map them to its unified model. Somehow, this previous work on a unified metadata model could be considered as the first step for enabling FAIRness assessment. For example, an ontology developer can focus on his/her responsibility of determining the license to use an ontology, while MOD offers means and recommendations to encode such information in a way machines can assess the level of FAIRness. Based on the MOD model, we produce in this article guidelines on how FAIR principles might be met and evaluated. Section 4 provides a clear alignment between the MOD properties and the FAIR principles. For instance, to assess F1, we rely on the existing MOD properties to encode the identifiers of an ontology (omv:uri) and (dct:identifier).

In March 2020, the FAIRsFAIR H2020 project delivered the first version of a list of 17 recommendations and 10 best practices recommendations for making semantic artefacts FAIR [START_REF] Franc | D2.2 FAIR Semantics: First recommendations[END_REF] (later revised in Dec. 2020 in a new deliverable [START_REF] Hugo | D2.5 FAIR Semantics Recommendations Second Iteration[END_REF]). For each recommendation, the authors provided a detailed description, listed its related semantic Web technologies, and outlined potential technical solutions in some cases. Similarly, best practices are introduced as recommendations that are not directly related to a FAIR principle but contribute to the overall evaluation of a semantic resource. This proposal is currently being discussed in the context of the RDA Vocabulary and Semantic Services Interest Group (VSSIG). The recommendations are also publicly available for comments on GitHub. 4Later, in September 2020, Poveda et al. considered some previous works and produced "guidelines and best practices for creating accessible, understandable and reusable ontologies on the Web" [START_REF] Garijo | Best Practices for Implementing FAIR Vocabularies and Ontologies on the Web[END_REF]. In another position paper [START_REF] Poveda-Villalón | Coming to Terms with FAIR Ontologies[END_REF], they complete a qualitative analysis of how four ontology publication initiatives cover the foundational FAIR principles. They propose some recommendations on making ontologies FAIR and list some open issues that might be addressed by the semantic Web community in the future. These two publications are very relevant for our methodology; our work is a step further. It completes this work and proposes a concrete metric necessary for further work on automatic FAIRness assessment.

Other recent related works on FAIR principles for semantic resources include a list of functional metrics and recommendations for Linked Open Data Knowledge Organization Systems (LOD KOS) products proposed in 2020 [START_REF] Zeng | FAIR + FIT: Guiding Principles and Functional Metrics for Linked Open Data (LOD) KOS Products[END_REF], a list of ten simple rules for making a vocabulary FAIR [START_REF] Cox | Ten simple rules for making a vocabulary FAIR[END_REF]. Finally, the DBPedia Archivo tool [START_REF] Frey | DBpedia Archivo: A Web-Scale Interface for Ontology Archiving Under Consumer-Oriented Aspects[END_REF], an ontology archive also released at the end of 2020 that aims to help developers and consumers in "implementing FAIR ontologies on the Web."

To design our FAIRness assessment methodology, we analyzed and merged relevant related approaches namely FDMM version v0.04, SHARC version v1.1, LOD 5-stars V, MIRO, FAIRsFAIR recommendations, and Poveda et al.'s guidelines. We consider both generic and specific approaches to provide a specialized solution for ontologies but are still influenced by more general concerns, as ontologies are a kind of digital object. The integration was not straightforward because none of the approaches used is simply and strictly aligned with the 15 sub-principles (e.g., FDMM provides 47 criteria). Two of them (i.e., MIRO and 5-stars V) were totally disconnected from the FAIR prism. Table 1 gives a summary of our selection. We classify approaches into three groups: (A) for generic approaches which set priorities for each FAIR principle or sub-principle; for specific approaches for semantic resources which: (B) includes FAIRsFAIR and Poveda et al. as these guidelines do not set priorities; (C) includes LOD 5-stars and MIRO as they are not aligned to the FAIR principles. In the next section, we explain how we proceeded to integrate all these methodologies into the proposed grid. 3 Integrated quantitative FAIRness assessment grid for ontologies

Methodology

In what follows, we explain how we quantify each approach against the FAIR principles, then for each category (A) and (B-C), and finally determine a set of final FAIR credits that could be used in evaluating any semantic resource. We chose to provide numerical credits {0;1;2;3} to respectively represent the degree of priorities/qualification of each indicator (other names for sub-principle e.g., F1, F2, F3 et F4 for F) within an approach {e.g., "none", "may", "should", "must"}. A "must" represents an essential principle, a "should" means that a principle is important except under some particular circumstance, "may" is an optional requirement, and "none" a non-revealed/specified qualification of a principle. Table 2 lists the correspondences between the five approaches, their priorities, and their attributed credits. To determine the FAIR principle credits for each group, we follow certain rules:

• Group A: we calculate the approximate average value of credits per indicator (FDMM) or principle (SHARC). For SHARC, we divide the obtained average value by the number of indicators associated with a principle. • Group B: we count the number of recommendations to determine the credits.

• Group C: we count the number "must", "should" and "optional" properties for MIRO and the number of the principles for 5-stars V. The final credits for each sub-principle are the sum of all obtained credits per methodology. An example is provided hereafter:

Example 1: We illustrate how we determine for each group (i.e., A, B and C) the credits of F1 (noted Credits F1 ): Group A:

• FDMM defines 4 "essential" indicators (F1-01M, F1-01D, F1-02M, and F1-02D). Thus, .

Results

From a semantic Web perspective, the results obtained emphasize the need for the establishment of agreement about a set of core metadata ontology descriptions, a federation model for ontologies regarding repositories and search engines, clear ontology and metadata ontology perseveration strategies within endpoints, mechanisms for references qualification, and best practices to document and communicate ontologies. Fig. 1 (c) provides final integrated FAIR credits per indicator; it shows how both generic approaches and semantic resources specific approaches address FAIRness and scores each FAIR indicator. For example, . Which is the sum of 41 (F1)+27 (F2)+ 21(F3)+24(F4). 𝐶𝑟𝑒𝑑𝑖𝑡𝑠 𝐹 = 113 Fig. 1 (c) illustrates the importance of each indicator in our integrated method. When doing the final sums, we have chosen a baseline value fixed to 10, to represent the fact that originally, as suggested by the FORCE 11 group, the FAIR principles were not ordered by importance; they were supposed to all contribute equally. The final credits are presented in our integrated FAIRness assessment grid (Table 3); the 478 credits of the grid, dispatched by each sub-principle, can be used for the assessment of any semantic resource or ontologies. Total credits 478

Fig. 1. Credits are assigned to each FAIR principle by generic approaches (a), specific approaches (b), and sums with a common baseline in our integrated grid (c).

A quick analysis of Table 3 and Fig. 1 reveals interesting points:

• The most important principles for generic and specific approaches are not the same. Generic approaches tend to emphasize principles F1 (identifier), A1 (access protocol), R1.1 (license), and R1.3 (community standards), while specific approaches emphasize principles I1 (knowledge representation), R1.2 (provenance), and I2 (use of vocabularies). This confirms our hypothesis that being FAIR is strongly dependent on the type of digital object considered and therefore FAIRness assessment methods must be customized for each type. • In the integrated grid, F1, A1, and I1 are the three sub-principles with the higher number of credits. These aspects being "generally" well addressed by ontologies, it will contribute to an overall good level of FAIRness. • Four sub-principles, important for FAIR, were completely ignored/avoided by specific approaches, except the FAIRsFAIR recommendations: F3 (link data-metadata), A1.1 (protocol openness), A1.2 (protocol security), and A2 (long term metadata). Consequently, three of these four keep the minimum number of credits in the integrated grid. • None of the specific approaches covered all of the FAIR sub-principles. This is not surprising for MIRO and 5-stars V, which preexist the FAIR movement, but it is more surprising for FAIRsFAIR and Poveda et al. whose recommendations were done specifically for ontologies or semantic resources to be FAIR. Only A1, I1, and I3 were found in the four approaches studied. This point backups our methodology, which mixes both generic and specific approaches. • Despite differences in credits assigned to the sub-principles, the sums by principles are relatively close, with a mean of 119,5. Only the R group is significantly above the mean. The group I is slightly under, mainly because it is made of only three sub-principles instead of four. • R being the most important principle may reveal the concern that ontologies and semantic resources, often developed by means of semantic Web technologies (RDFS, OWL, SKOS) are naturally equipped with good findability, accessibility, and interoperability features (e.g., URIs for identifiers, HTTP for accessibility, W3C standards for knowledge representation, claim to use vocabularies, etc.) whereas they lack reusability.

Candidate metadata properties for FAIR ontologies

In the second phase of our work, we elicited candidate metadata properties that can be used to encode information relevant for each FAIR sub-principle. Indeed, we found out most sub-principles (about 93%) might be partially or totally implemented and assessed with a series of metadata properties. In this section, we review candidate metadata properties that could be used by anyone developing (i) an ontology or semantic resource or (ii) a FAIRness assessment tool to obtain associated credits as listed in the previous section.

Candidate metadata properties to support FAIRness

Here, we reuse the MOD ontology metadata model (v1.4) [START_REF] Jonquet | Harnessing the Power of Unified Metadata in an Ontology Repository: The Case of AgroPortal[END_REF] as a reference to pick up metadata properties. MOD1.4 reviewed 346 metadata properties from 23 standard metadata vocabularies (such as Dublin Core, DCAT, VoID, ADMS, VOAF, Schema.org, etc.) to provide a list of 127 "aligned or crosswalked" properties that can be used to describe an ontology or a semantic resource. MOD allows us to unambiguously identify which property may be used; however, our grid could be implemented with any other metadata standard or combination of standards that cover all the sub-principles. The outcome of this process is a list of 58 candidate metadata properties that may be used to support FAIRness assessment and assign some credits from our grid. These metadata properties might help to assign 276 credits over a total of 478 (57%). We have separated the metadata properties for any principles from the ones for F2, which has to be treated apart. Indeed, F2 ("Data are described with rich metadata") was assigned all the properties that MOD1.4 has reviewed as relevant for ontologies that have not been assigned to another sub-principle. We refer to the first group as core metadata properties (Table 4) and to the second group as extra metadata properties (Table 5). The idea is that any ontologies using some of the 69 extra metadata properties in addition to the core 58 ones, will be "FAIRer". We identified that 46% of the FAIR principles (i.e., F2, I1, I2, R1, R1.1, R1.2, and R1.3) are totally evaluable with metadata properties, 33% are partially evaluable (i.e., F1, F4, A1, A1.2, and A2). Two principles for which we have not found any metadata property are A1.1 ("The protocol is open, free, and universally implementable.") and A1.2 ("The protocol allows for an authentication and authorization where necessary.") because they are completely related to the evaluation of the communication protocol, not the ontology persay. A sub-principle is not totally evaluable with metadata properties when it is about an aspect independent of the ontology itself but related to the library/repository hosting the ontology. For instance: F4 ("(Meta)data are registered or indexed in a searchable resource.") concerns also ontology repositories. F3 ("Metadata clearly and explicitly include the identifier of the data they describe.") is excluded from Table 4 as MOD1. [START_REF] Caracciolo | 39 Hints to Facilitate the Use of Semantics for Data on Agriculture and Nutrition[END_REF] do not yet offer a property to establish the link between an ontology and its metadata (necessary when metadata are not explicitly included in the same file than the ontology itself). Such a property is currently being discussed in the FAIR Digital Object working group of GO FAIR that shall soon release a new metadata vocabulary including fdo:hasMetadata and fdo:metadataOf properties. Even if I3 is totally evaluable with metadata, the currently proposed candidate metadata are not covering its evaluation. Here again, we need some extension to MOD to enable encoding all information required by this principle (especially alignment qualification). MOD is currently being extended as a new model compliant with DCAT2 within the RDA VSSIG and H2020 FAIRsFAIR. Qualifying the degree of FAIRness of a semantic resource or even comparing it with other semantic resources necessarily implies the use of a metric delimiting the range of values for each qualification (e.g., not FAIR, FAIR, or FAIRer). In that context, our proposed integrated quantitative grid allows for defining thresholds. For instance, the median value of the resulting total credits can be considered a minimum threshold to be FAIR. A semantic resource with a degree/score under this threshold will not be considered FAIR. Similarly, a semantic resource might be considered as "FAIRer" if it is described with extra metadata properties. In other words, answering the question: "how much is a semantic resource FAIR?" becomes possible with such a metric. In our grid, the total credits are 478, so the first threshold could be at 240 (478/2+1) and the second threshold at 451 (478-27), as illustrated in Fig. 2. Clearly, using a metric and threshold is the first required step in making the FAIRness assessment task machine-actionable and enabling the development of automatic FAIRness assessment tools. We believe it will also be beneficial for researchers to quantify the FAIRness degree of their semantic resources and compare them with other ones.

Conclusions and perspective

In this paper, we proposed an integrated quantitative grid for assessing the level of FAIRness of semantic resources and ontologies. Moreover, we provided a list of candidate metadata properties -from the MOD model v1.4-to enable FAIRness assessment and possibly implement systems based on our grid. Our grid was realized by analyzing existing related work (among others, the semantic Web community work before and since the FAIR movement) and summarizing them into one coherent scheme.

A distinct feature of our grid is to propose a metric -and thus possible thresholds-for the qualification of any semantic resource. The grid is conceived in a way that can be customized, extended, or improved by other semantic experts in further studies. This work is a starting point for developing machine-actionable FAIRness assessment tools in the semantic Web context. The motivation of this work was to go beyond the current recommendations to guide semantic stakeholders for making their semantic resources FAIR: We consider these recommendations, harmonize and integrate them to build a grid of 478 credits to assess the 15 FAIR principles.

Currently, we are using the grid to implement a FAIRness assessment tool in AgroPortal (http://agroportal.lirmm.fr/), a vocabulary and ontology repository dedicated to agri-food and based on the generic and open source OntoPortal technology . However,

  𝐶𝑟𝑒𝑑𝑖𝑡𝑠 𝐹1,𝐹𝐷𝑀𝑀 = 3 * 4 = 12 • SHARC defines 12 sub-indicators (8 essential, 4 recommended) for F. Thus the approximative . 𝐶𝑟𝑒𝑑𝑖𝑡𝑠 𝐹1, 𝑆𝐻𝐴𝑅𝐶 = (8 * 3 + 4 * 2 )÷4 = 8 Group B: • FAIRsFAIR defines 2 recommendations (P-Rec 1 and P-Rec 2) related to F1 thus, =2. 𝐶𝑟𝑒𝑑𝑖𝑡𝑠 𝐹1,𝐹𝑠𝐹 • Poveda et al. define 4 recommendations related to F1 (Rec 1, Rec 2, Rec 3 and Rec 5) thus . 𝐶𝑟𝑒𝑑𝑖𝑡𝑠 𝐹1,𝑃𝑜𝑣𝑒𝑑𝑎 𝑒𝑡 𝑎𝑙. = 3 Group C:• MIRO refers to 2 "must" properties ("A" category-basics) for F1 sub-principle: ontology version (A.4) and IRI version (A.4). Thus, =6. 𝐶𝑟𝑒𝑑𝑖𝑡𝑠 𝐹1,𝑀𝐼𝑅𝑂• LOD 5-stars V does not especially cover Findability; thus,

Fig. 2 .

 2 Fig. 2. Not FAIR, FAIR, or FAIRer: using the metric of the integrated quantitative grid.

Table 1 .

 1 Summary of related works on FAIRness assessment integrated into our approach.

	Category	Generic (A)		Specific (B, C)
	Format	grid		principles	scheme	recommendations
	Approach	SHARC FDMM	5-stars V	MIRO	FAIRsFAIR Poveda et al.
	Reference	[6]	[7]	[14]	[17]	[3]	[20]
	Year	2017	2018	2011	2017	2020	2020
	FAIR principles	after	after	before	before	after	after
	Priorities	yes	yes	n/a	n/a	no	no

Table 2

 2 Alignment between priorities in related work approaches and credits.

	Grou	Approach	none	may	should	must
	p		(0 credit)	(1 credit)	(2 credits)	(3 credits)
	A	FDMM	n/a	useful	important	essential
		SHARC	n/a	desirable	recommended	essential
	B	FAIRsFAIR	n/a	1 Rec	2 Rec	3 Rec or >
		Poveda et al.	n/a	1 Rec	2 Rec	3 Rec or >
	C	MIRO	n/a	optional	should	must

Table 3 .

 3 Integrated FAIRness assessment grid for semantic resources and ontologies.

	Principle	Basel	SHARC	FDMM	5-stars	MIRO	FAIRs	Poveda	Credits
			ine			V		FAIR	et al.
	F	F1	10	8	12	0	6	2	3	41	113
		F2	10	8	2	0	5	1	1	27
		F3	10	8	2	0	0	1	0	21
		F4	10	8	3	0	0	2	1	24
	A	A1	10	6	18	3	3	1	2	43	113
		A1.1	10	6	11	0	0	1	0	28
		A1.2	10	6	5	0	0	1	0	22
		A2	10	6	3	0	0	1	0	20
	I	I1	10	4	12	1	12	3	2	44	109
		I2	10	4	7	0	9	2	0	32
		I3	10	4	12	1	3	2	1	33
	R	R1	10	9	3	1	6	0	3	32	143
		R1.1	10	9	12	0	3	2	1	37
		R1.2	10	9	3	0	12	3	1	38
		R1.3	10	9	12	0	0	3	2	36

Table 4 .

 4 List of core metadata properties from MOD1.4 to help make an ontology FAIR.

	Principle	Credits	Metadata properties
	F	F1	29	owl:ontologyIRI, owl:versionIRI, dct:identifier
		F4	24	schema:includedInDataCatalog
	A	A1	36	owl:ontologyIRI, dct:identifier, sd:endpoint
		A2	4	omv:status, owl:deprecated
	I	I1	44	omv:hasOntologyLanguage,	omv:hasFormalityLevel,	omv:hasOntologySyntax,
				dct:hasFormat, dct:isFormatOf
		I2	22	owl:imports, voaf:hasEquivalenceWith, owl:priorVersion, voaf:similar, voaf:metadataVoc,
				dct:relation, dct:isPartOf, voaf:specializes, schema:translationOfWork, voaf:generalizes
	R	R1	8	mod:prefLabelProperty,		mod:synonymProperty,	mod:definitionProperty,
				mod:authorProperty, bpm:obsoleteProperty, mod:hierarchyProperty, mod:obsoleteParent,
				mod:maxDepth
		R1.1	37	dct:license, dct:rightsHolder, dct:accessRights, cc:morePermissions, cc:useGuidelines
		R1.2	36	dct:creator,	dct:ontributor,	pav:curatedBy,	schema:translator,	dct:source,
				prov:wasGeneratedBy, prov:wasInvalidatedBy, dct:accrualMethod, dct:accrualPeriodicity,
				dct:accrualPolicy,	omv:versionInfo,	vann:changes,	dct:hasVersion,
				omv:usedOntologyEngineeringTool,	omv:usedOntologyEngineeringMethodology,
				omv:conformsToKnowledgeRepresentationParadigm,	omv:designedForOntologyTask,
				mod:competencyQuestion, foaf:fundedBy
		R1.3	36	mod:ontologyInUse, omv:endorsedBy, mod:group, dct:accessRights
	Total		276	58 metadata properties

Table 5 .

 5 List of extra metadata properties from MOD1.4 to make an ontology FAIRer.

	Principle	Credits	Metadata properties
	F	F2	27	omv:acronym, dct:title, dct:alternative, skos:hiddenLabel, dct:description, foaf:page,
				omv:resourceLocator, omv:keywords, dct:coverage, foaf:homepage, vann:example,
				vann:preferredNamespaceUri, void:uriRegexPattern, idot:exampleIdentifier, dct:publisher,
				dct:subject,	owl:backwardCompatibleWith,	door:comesFromTheSameDomain,
				mod:sampleQueries, omv:knownUsage, dct:audience, doap:repository, doap:bugDatabase,
				doap:mailing-list,	mod:hasEvaluation,	mod:metrics,	omv:numberOfClasses,
				omv:numberOfIndividuals,	omv:numberOfProperties,	mod:numberOfDataProperties,
				mod:numberOfObjectProperties,	omv:numberOfAxioms,	mod:numberOfLabels,
				mod:byteSize, vann:preferredNamespacePrefix, dct:language, dct:abstract, mod:analytics,
				dct:bibliographicCitation,	rdfs:comment,	foaf:depiction,	foaf:logo,voaf:toDoList,
				schema:award,	schema:associatedMedia,	owl:isIncompatibleWith,	dct:hasPart,
				schema:workTranslation,	door:hasDisparateModelling,	voaf:usedBy,
				voaf:hasDisjunctionsWith,	omv:keyClasses,	void:rootResource,	mod:browsingUI,
				mod:sampleQueries,	void:propertyPartition,	void;classPartition,	void:dataDump,
				void:openSearchDescription, void:uriLookupEndpoint, schema:comments, dct:created,
				dct:modified, dct:valid, dct:dateSubmitted, pav:curatedOn, omv:IsOfType
	Total	27	69 metadata properties
	4.2

FAIR or FAIRer: How FAIR is a semantic resource?

In this paper, we will consider the terms ontologies, terminologies, thesaurus and vocabularies as a type of knowledge organization systems[START_REF] Zeng | Knowledge Organization Systems (KOS) in the Semantic Web: a multi-dimensional review[END_REF] or knowledge artefacts[START_REF] Franc | D2.2 FAIR Semantics: First recommendations[END_REF] or semantic resources[START_REF] Caracciolo | 39 Hints to Facilitate the Use of Semantics for Data on Agriculture and Nutrition[END_REF]. For simplicity, we will sometimes use "ontology" as an overarching word.

https://seprojects.marum.de/fuji/api/v1/ui/

https://github.com/FAIRsFAIR-Project/FAIRSemantics/issues/ 3 https://github.com/sifrproject/MOD-Ontology

in the future, this work will need to be further tested in other FAIRness assessment approaches and discussed within some international FAIR initiatives, for instance, RDA, GO FAIR, or projects such as FAIRsFAIR.

https://github.com/ontoportal-lirmm

Acknowledgments

This work has been supported by the Data to Knowledge in Agronomy and Biodiversity project (D2KAB -www.d2kab.org -ANR-18-CE23-0017) and the project ANR French participation in GO FAIR Food Systems Implementation Network (FooSIN -https://foosin.fr -ANR19-DATA-0019). We also thank the VSSIG (Vocabulary and Semantic Services Interest Group) of the Research Data Alliance and the H2020 FAIRsFAIR project T2.2 on "FAIR Semantics" for fruitful discussions and exchanges.