Minghan Li

Diana Nicoleta Popa

Johan Chagnon
email: johanchagnon@uowmail.edu.au

Eric Gaussier
email: eric.gaussier@univ-grenoble-alpes.fr

The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval

Keywords: CCS Concepts:, Information systems → Information retrieval, Document representation, Retrieval models and ranking BERT-based language models, long-document neural information retrieval

On a wide range of natural language processing and information retrieval tasks, transformer-based models, particularly pre-trained language models like BERT, have demonstrated tremendous effectiveness. Due to the quadratic complexity of the self-attention mechanism, however, such models have difficulties processing long documents. Recent works dealing with this issue include truncating long documents, in which case one loses potential relevant information, segmenting them into several passages, which may lead to miss some information and high computational complexity when the number of passages is large, or modifying the self-attention mechanism to make it sparser as in sparse-attention models, at the risk again of missing some information. We follow here a slightly different approach in which one first selects key blocks of a long document by local query-block pre-ranking, and then few blocks are aggregated to form a short document that can be processed by a model such as BERT. Experiments conducted on standard Information Retrieval datasets demonstrate the effectiveness of the proposed approach.

INTRODUCTION

The field of query-document information retrieval (IR) has seen increasingly rapid advance in the past decades. Learning-to-rank (LTR) models [START_REF] Li | Learning to rank for information retrieval and natural language processing[END_REF][START_REF] Liu | Learning to rank for information retrieval[END_REF] have already achieved great success in many IR applications. However, LTR models mainly rely on hand-crafted features which are time-consuming and often over-specific in definition [START_REF] Guo | A deep look into neural ranking models for information retrieval[END_REF]. With the resurgence of interest in neural networks, especially deep neural networks or deep learning, we have witnessed dramatic improvements in computer vision and natural language processing (NLP) tasks. Neural Information Retrieval (Neural IR), which refers to the use of (deep) neural networks to directly construct the ranking function for IR, has been the subject of many studies [12, 19, 25, 26, 33, 51-53, 61, 69] which have led to the development of several interesting IR models for learning representations of documents and queries.

The transformer model [START_REF] Vaswani | Attention is all you need[END_REF], which is exclusively based on multi-head attention mechanism, has shown to be higher in quality while being more parallelizable and requiring substantially less training time than models based on recurrent neural networks [START_REF] Vaswani | Attention is all you need[END_REF]. Using a multi-layer bidirectional transformer encoder, the authors of [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] have proposed Bidirectional Encoder Representations from Transformers (BERT), a method for pre-training deep bidirectional representations from unlabeled text by conditioning all layers on both left and right context. Pre-trained BERT models can be fine-tuned to produce cutting-edge models for a variety of applications. In particular, following their success in NLP, several works have focused on transformers [START_REF] Vaswani | Attention is all you need[END_REF] and derived models based on BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] for solving IR tasks [START_REF] Dai | Deeper text understanding for IR with contextual neural language modeling[END_REF][START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF][START_REF] Macavaney | CEDR: Contextualized embeddings for document ranking[END_REF][START_REF] Nogueira | Passage Re-ranking with BERT[END_REF], leading to some of the current state-of-the-art models in ad hoc IR [START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF][START_REF] Macavaney | CEDR: Contextualized embeddings for document ranking[END_REF].

One main advantages of the self-attention mechanism is that it allows to capture dependencies between tokens in a sequence regardless of their distance. However, despite its excellent results, the self-attention mechanism has difficulty to process long sequences due to its quadratic complexity in the number of tokens, which also limits the application of transformer-based models to long document information retrieval, where each document could contain thousands of tokens.

Three standard strategies based on BERT have been adopted to circumvent this problem. The first one consists in truncating long documents (e.g., [START_REF] Nogueira | Passage Re-ranking with BERT[END_REF]), the second in segmenting long documents into shorter passages (e.g., [START_REF] Dai | Deeper text understanding for IR with contextual neural language modeling[END_REF]) and the last one in replacing the complex self-attention module with sparse-attention ones (e.g., [START_REF] Jiang | Long Document Ranking with Query-Directed Sparse Transformer[END_REF]). In the first case, important information may be lost and the relevance judgement is damaged. In the second case, a hierarchical architecture can be further adopted to build a document-level representation on top of the representations of each passage [START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF]. This said, despite the state-of-the-art results this strategy may lead to, there remain issues concerning the time, memory and energy consumption associated to it. Furthermore, the consideration of passages that may not be relevant to the query may introduce noise in the final representation and limit the identification of long-distance dependencies between relevant tokens [START_REF] Ding | CogLTX: Applying BERT to Long Texts[END_REF]. In the third case, sparsity constraints may lead to miss important dependencies which can lead to under-optimal results.

The approach we propose is slightly different with these strategies, aiming at capturing, in long documents, the blocks which are the most important to decide on the relevance status of the whole document. Besides, it can be integrated to the second strategy. It is based on three main steps: (a) selecting key (i.e., likely relevant) blocks with local pre-ranking using either classical IR models or a learning module reminiscent of the judge module used in [START_REF] Ding | CogLTX: Applying BERT to Long Texts[END_REF], (b) learning a joint representation of queries and key blocks using a standard BERT model, and (c) computing a final relevance score which can be regarded as an aggregation of local relevance information [START_REF] Ho | A retrospective study of a hybrid document-context based retrieval model[END_REF].

Our contributions are two-fold. We first conduct an analysis which reveals that relevance signals can appear at different locations in documents and that such signals can be better captured by semantic relations than by exact matches. We then investigate two methods to select blocks, one based on standard IR functions and the other on a learned function operating on semantic representations, and show how to integrate these methods in state-of-the-art IR models. In this approach, as well as in previous approaches based on passages as PARADE [START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF], blocks occurring at different positions of a document are concatenated or selected in the order they occur in the document and can be seen as a digest of the elements necessary to assess the relevance of the whole document to the query. Although the blocks selected are not coherent physically, they still are coherent in that they are all relevant to the same query.

The remainder of the paper is organized as follows: Section 2 describes related work. Section 3 investigates the relation between the potential relevance and the position of a block in a document as well as the importance of fuzzy vs exact matching when selecting blocks. Section 4 presents the block selection approach based on standard IR functions whereas Section 5 describes the block selection approach based on a learned function. Sections 6 and 7 show the benefits of selecting key blocks on different collections. Finally, Section 8 summarizes our findings and concludes the paper.

Let 𝑞 denote a query and 𝑑 a document. Without loss of generality, the ranking function 𝑓 of an IR system takes the form [START_REF] Guo | A deep look into neural ranking models for information retrieval[END_REF]:

𝑓 (𝑞, 𝑑) = 𝑔(𝜓 (𝑞), 𝜙 (𝑑), 𝜂 (𝑞, 𝑑)),

(1) where 𝜓 and 𝜙 are representation functions that extract features from 𝑞 and 𝑑 respectively, 𝜂 is the interaction function that models query-document representation from (𝑞, 𝑑) pairs, and 𝑔 is the evaluation function that calculates the relevance score based on the extracted features or interaction. According to the choices made on the representation and interaction functions, neural information retrieval models can be grouped into two categories [START_REF] Guo | A deep look into neural ranking models for information retrieval[END_REF]: representation-based and interaction-based architectures. Besides these two categories, some neural information retrieval models adopt a hybrid approach.

The Deep Structured Semantic Model (DSSM) [START_REF] Huang | Learning deep structured semantic models for web search using clickthrough data[END_REF] is one of the earliest representation-based models for document ranking which uses a fully-connected network for the functions 𝜓 and 𝜙. To map the query and the documents to a shared semantic space, a non-linear projection is used. The relevance of each document given the query is then calculated with the cosine similarity between their vectors in that semantic space. Clickthrough data is then used to discriminatively train the model by maximizing the conditional likelihood of the clicked documents. Other studies in this category proposed to exploit distributed representations via DSSM variations, or relied on different representation functions [START_REF] Dilek Onal | Neural information retrieval: At the end of the early years[END_REF]. For example, ARC-I [START_REF] Hu | Convolutional neural network architectures for matching natural language sentences[END_REF] and CLSM [START_REF] Shen | A latent semantic model with convolutional-pooling structure for information retrieval[END_REF] use convolutional neural networks (CNN) for 𝜓 and 𝜙 while [START_REF] Palangi | Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval[END_REF] uses a recurrent neural networks.

One of the first neural IR models which outperformed traditional IR models is the interaction-based model referred to as Deep Relevance Matching Model (DRMM) [START_REF] Guo | A deep relevance matching model for ad-hoc retrieval[END_REF]. The interaction function 𝜂 is defined as the matching histogram mapping between each query term and the document. A feed-forward network for term-level relevance and a gating network for score aggregation in the evaluation function 𝑔 are further used. In this work, the term vectors are fixed to Word2Vec word embeddings [START_REF] Mikolov | Distributed Representations of Words and Phrases and their Compositionality[END_REF]. Similarly, Xiong et al. [START_REF] Xiong | End-to-end neural ad-hoc ranking with kernel pooling[END_REF] proposed KNRM which employs a translation matrix that utilizes word embeddings to represent word-level similarities, a unique kernel-pooling technique for extracting multi-level soft match features, and a learning-to-rank layer that combines those features into the final ranking score. The entire model is trained end-to-end, and the word embeddings are tuned to produce the desired soft matches [START_REF] Xiong | End-to-end neural ad-hoc ranking with kernel pooling[END_REF]. Inspired by the way humans assess the relevance of a document, Pang et al. [START_REF] Pang | Deeprank: A new deep architecture for relevance ranking in information retrieval[END_REF] proposed DeepRank, a model which splits documents into term-centric contexts according to each query term. A tensor containing both the word representations of query/query-centric context as well as their interactions is first built. It is then passed through a measure network, based on CNN [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] or 2D-GRU [START_REF] Wan | Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN[END_REF], to produce a representation of local relevance. Finally, the global relevance score is calculated using an aggregation network. Hui et al. [START_REF] Hui | PACRR: A Position-Aware Neural IR Model for Relevance Matching[END_REF] proposed PACRR, a model inspired by the neural models used in image recognition [START_REF] Guo | A deep look into neural ranking models for information retrieval[END_REF]. PACRR takes a similarity matrix between a query and a document as input. Then multiple CNN kernels capture the query-document interactions. Following this work, Hui et al. [START_REF] Hui | Co-PACRR: A context-aware neural IR model for ad-hoc retrieval[END_REF] provided a lightweight contextualization model called CO-PACRR which averages word vectors within a sliding window and appends the similarities to those of the PACRR [START_REF] Hui | PACRR: A Position-Aware Neural IR Model for Relevance Matching[END_REF] model [START_REF] Hofstätter | Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking[END_REF].

Since it is sometimes difficult to produce good high-level representations of long texts, the representation-based architecture is better suited to short input texts. Models in this category are good for online computing since they allow one to pre-calculate text representations. Interaction-based models, on the other hand, tend to yield better results as they can tune document representations towards a given query. Unfortunately, since the interaction function 𝜂 cannot be pre-calculated until the input pair (𝑞, 𝑑) is seen, models in this category are not as efficient for online computation as representation-focused models [START_REF] Guo | A deep look into neural ranking models for information retrieval[END_REF].

Models Based on transformers. Benefiting from pre-trained language models based on transformers [START_REF] Vaswani | Attention is all you need[END_REF], especially BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF], different research teams have developed state-of-the-art neural IR models, significantly outperforming traditional and previous neural IR models. Nogueira and Cho [START_REF] Nogueira | Passage Re-ranking with BERT[END_REF] proposed to use BERT as a re-ranker for the passage re-ranking task by fine-tuning it and achieved state-of-the-art results. The passages are truncated if too long (typically over 512 tokens). This work proved the effectiveness of fine-tuning BERT for IR problems. MacAvaney et al. [START_REF] Macavaney | CEDR: Contextualized embeddings for document ranking[END_REF], through a model called CEDR which combined BERT with other neural IR models, as PACRR [START_REF] Hui | PACRR: A Position-Aware Neural IR Model for Relevance Matching[END_REF], KNRM [START_REF] Xiong | End-to-end neural ad-hoc ranking with kernel pooling[END_REF] and DRMM [START_REF] Guo | A deep relevance matching model for ad-hoc retrieval[END_REF], and showed the benefits of this combination. Dai and Callan [START_REF] Dai | Deeper text understanding for IR with contextual neural language modeling[END_REF] proposed to first segment documents into short, overlapping passages, and then used BERT to define the relevance score of the document, using either the first passage, the best passage or the sum of all passages. Hofstätter et al. [START_REF] Hofstätter | Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking[END_REF] proposed a reranking model called Transformer Kernel, in short TK, which uses a hybrid approach based on a small number of transformer layers to contextualize query and document word embeddings separately. Then RBF-kernels [START_REF] Xiong | End-to-end neural ad-hoc ranking with kernel pooling[END_REF] are used for interaction scoring, where each kernel focuses on a specific similarity range. Experimental results show that although the effectiveness is not as good as BERT reranker, TK has strong efficiency. In a similar vein, Li et al. [START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF] explored strategies for aggregating relevance signals from a document's passages into a final ranking score, leading to a model called PARADE. A hierarchical layer, in the form of a max-pooling, attention, CNN or transformer aggregator is used to aggregate the passage representations so as to obtain a joint query-document representation for long documents. They showed that passage representation aggregation strategies can outperform techniques proposed previously. In particular, PARADE can improve results significantly on collections with broad information needs where relevance signals can be disseminated throughout the document. Grail et al. [START_REF] Grail | Globalizing BERT-based Transformer Architectures for Long Document Summarization[END_REF] also proposed a hierarchical model in which each transformer layer, used to learn a representation for each sentence of a document, is followed by an RNN which captures dependencies between the CLS tokens representing the different sentences of a document. As shown in the experiments, this model is particularly well adapted for long-document summarization.

In the above models, as transformers are limited in their input length due to their quadratic complexity, researchers have either truncated long documents or segmented them into passages. There have however been different attempts to use transformers on long documents. For example, Dai et al. [START_REF] Dai | Transformer-XL: Attentive Language Models beyond a Fixed-Length Context[END_REF] introduced a model with left-to-right recurrence between transformer windows, consisting of a segment-level recurrence mechanism and a novel positional encoding scheme. The left-to-right approach processes the document in chunks moving from left-to-right and thus not adapted to tasks which benefit from bidirectional contexts [START_REF] Beltagy | Longformer: The Long-Document Transformer[END_REF]. Child et al. [START_REF] Child | Generating Long Sequences with Sparse Transformers[END_REF] introduced several sparse factorizations of the attention matrix which reduce the quadratic complexity to 𝑂 (𝑛 √ 𝑛). Hofstätter et al. [START_REF] Hofstätter | Local self-attention over long text for efficient document retrieval[END_REF] proposed a local self-attention which considers a sliding window over the document and restricts the attention to that window in order to deal with long documents. Their model, called TKL, adapts TK [START_REF] Hofstätter | Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking[END_REF] with this mechanism. Beltagy et al. [START_REF] Beltagy | Longformer: The Long-Document Transformer[END_REF] introduced the Longformer with an attention mechanism which scales linearly with sequence length, combining windowed local-context self-attention with task-motivated global attention to encode inductive bias about the task. Longformer achieves state-of-the-art results on the character-level language modeling tasks, and when pretrained from the RoBERTa [START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF] checkpoint, it consistently outperforms RoBERTa on long document tasks. Zhao et al. [START_REF] Zhao | Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention[END_REF] proposed Transformer-XH which enables to represent structured texts. It shares similar motivation with Dai et al. [START_REF] Dai | Transformer-XL: Attentive Language Models beyond a Fixed-Length Context[END_REF] and Child et al. [START_REF] Child | Generating Long Sequences with Sparse Transformers[END_REF], and is particularly well adapted to multi-hop QA tasks [START_REF] Yang | HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering[END_REF] and fact verification tasks [START_REF] Thorne | The Fact Extraction and VERification (FEVER) Shared Task[END_REF]. Ainslie et al. [START_REF] Ainslie | ETC: Encoding Long and Structured Inputs in Transformers[END_REF] introduced the Extended Transformer Construction (ETC) model to address two key challenges of standard transformers: scaling input length and encoding structured inputs. A novel global-local attention mechanism is introduced where the local sparsity reduces the quadratic scaling of the attention mechanism. They further show that by including a pre-training Contrastive Predictive Coding (CPC) task [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF], the performance for tasks where structure matters improves even further. Zaheer et al. [START_REF] Zaheer | Big Bird: Transformers for Longer Sequences[END_REF] proposeed BigBird, which combines local and global attention with random sparse attention. Kitaev et al. [START_REF] Kitaev | Reformer: The Efficient Transformer[END_REF] only computed self-attention between similar tokens, as defined through locality-sensitive hashing.

Despite such models' described effectiveness, there remain problems. Firstly, as described in [START_REF] Zaheer | Big Bird: Transformers for Longer Sequences[END_REF], coalesced memory operations, which load blocks of contiguous bytes at once, are where hardware accelerators like GPUs and TPUs really shine. As a result, small sporadic look-ups caused by a sliding window or random element queries are not very efficient. This is addressed by "blockifying" the lookups. It is generally known [START_REF] Gray | Gpu kernels for block-sparse weights[END_REF][START_REF] Yao | Balanced sparsity for efficient dnn inference on gpu[END_REF] that GPUs cannot efficiently execute sparse multiplications, which are commonly employed by models with tailored attention mechanisms. Naively using for-loops or masking the matrix may result in even worse efficiency than the full self attention [START_REF] Jiang | Long Document Ranking with Query-Directed Sparse Transformer[END_REF]. Thus, such models with customized attention mechanisms need specifically designed tricks or customized CUDA kernels [START_REF] Beltagy | Longformer: The Long-Document Transformer[END_REF], which are inconvenient or require expertise in low-level GPU operations [START_REF] Jiang | Long Document Ranking with Query-Directed Sparse Transformer[END_REF]. Jiang et al. [START_REF] Jiang | Long Document Ranking with Query-Directed Sparse Transformer[END_REF] proposed Query-Directed Sparse Transformer (QDS-Transformer), which also induce sparsity in self-attention mechanism, containing local windowed attention and query-directed global attention. Experiments demonstrate consistent and robust advantages of QDS-Transformer over previous approaches. However, this model still needs customized TVM implementation [START_REF] Chen | TVM: an automated end-to-end optimizing compiler for deep learning[END_REF]. Secondly, as these customized attention models only rely on partial attention, their accuracy does not match, in general, the one of full attention models.

Selecting key blocks. The approach we advocate here to solve the above-mentioned problems is to select key, important blocks from a document and base the relevant score of the full document on just these blocks. Note that this differs from the approach followed in PARADE [START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF] in which the passages retained are arbitrary. It is however reminiscent of both [START_REF] Pang | Deeprank: A new deep architecture for relevance ranking in information retrieval[END_REF] and [START_REF] Ding | CogLTX: Applying BERT to Long Texts[END_REF], which are partly inspired by cognitive theory and reckon that, in order to assess the relevance of a document, humans first scan the whole document to detect relevant locations where query terms occur and then aggregate local relevance information to decide on the relevance of a document [START_REF] Ho | A retrospective study of a hybrid document-context based retrieval model[END_REF]. Compared to [START_REF] Pang | Deeprank: A new deep architecture for relevance ranking in information retrieval[END_REF], our approach is simpler conceptually and can consider blocks which do not contain query words but are nevertheless relevant to the query. We show in Sections 3 and 6 that such a fuzzy matching can help improve the block selection process and the overall IR system built upon it. The study described in [START_REF] Ding | CogLTX: Applying BERT to Long Texts[END_REF] focuses on reading comprehension, multi-hop question-answering and text classification. We show here how this approach can be simplified for IR purposes by using the same BERT model for the reasoner and judge. In addition, we investigate the use of standard models to select key blocks, which leads to an entirely different and simpler architecture.

Furthermore, we want to mention the study presented in Muntean et al. [START_REF] Ioana Muntean | Weighting Passages Enhances Accuracy[END_REF] which, in order to assign a relevance score to a document, weighs each passage differently by identifying salient terms using TF-IDF and KL divergence scores [START_REF] Büttcher | A document-centric approach to static index pruning in text retrieval systems[END_REF] are used to identify salient terms and to derive the weights. Although this model treats passages in a long document differently, the weights do not reflect the relevance to the query (salient terms are identified independently of the query). This is a major difference with our approach which aims to select blocks according to their relevance to the query. Furthermore, we focus here on neural IR models which have difficulties in dealing with long documents.

This paper is an extension of the earlier short paper published at SIGIR 2021 and entitled: "KeyBLD: Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval" [START_REF] Li | KeyBLD: Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval[END_REF]. This extension consists first in an analysis of where relevant signals appear in documents and of how blocks should be selected, and second in the proposal of a new learning-based selection method (see Section 5) and an integration within PARADE [START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF] (see Section 4.2). We have also added three new collections and several baseline models for evaluation purposes. Experimental results show the effectiveness of the proposed approaches for selecting key blocks. In particular, the method we developed for learning how to select blocks outperforms all other methods, including the ones we presented in [START_REF] Li | KeyBLD: Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval[END_REF]. Besides, selecting key passages with local pre-ranking makes the PARADE model more efficient and accurate: the results obtained are in general better or at least similar. This shows that the proposed mechanism can be deployed in different models. In addition, our approach leads to better results than sparse attention transformer models while not requiring customized CUDA kernels.

Lastly, in parallel to our work, Hofstätter et al. [START_REF] Hofstätter | Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking[END_REF] also introduce a model called IDCM which also learns to select top scoring blocks which are then used to score documents with respect to a given query. IDCM first trains a block ranking model based on BERT (and called ETM for Effective Teacher Model) on MS-MARCO 1 , prior to fine-tuning this model on each collection for document ranking. Then, a block selection model (called ESM for Efficient Student Model) is learned via knowledge distillation from ETM. Both ESM and ETM are then used to score new documents, ESM allowing the selection of the most important blocks and ETM being used to score documents on the basis of the selected blocks. This contrasts with the way we learn to select blocks: first of all, for all proposed models except the last one KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 , we do not require any pre-training on additional collections; second, we use the same model for selecting blocks and scoring documents, the rationale being that in both cases one computes a relevance score with respect to the same query. The last model that we propose can also re-use an additional BERT ranker from the KeyB(vBERT) models previously proposed to select blocks. Our approach is thus simpler (see Section 5) and finally leads to better results as shown in Section 7. Furthermore, it is interesting to note that IDCM scores each block separately and then aggregates the block scores, while our approach for improving Vanilla BERT scores a document using concatenated selected blocks. As shown in our experiments, the two strategies are effective, with a slight advantage for the latter on the TREC 2019 DL collection. We also want to mention that we recently proposed Li and Gaussier [START_REF] Li | BERT-based Dense Intra-ranking and Contextualized Late Interaction via Multi-task Learning for Long Document Retrieval[END_REF] to select key blocks for late-interaction models by training a BERT-based model with multi-task learning. This study however has a different focus that the current one, which only concerns interaction-based models.

A FINER-GRAINED LOOK AT DOCUMENTS

We take in this section a closer look at documents and the blocks they contain by addressing two questions:

(1) Are relevant signals spread over the entire document, and thus can appear in any block, or are they concentrated in particular regions, as the beginning of documents? (2) Should one rely only on exact matching of query words to select important blocks or is additional information contained in related words, as synonyms, important?

Our analyses thus aim to reveal which blocks to select with respect to their positions and how to select them.

Preliminaries

To conduct our investigation of the above points, we use four standard IR datasets, namely MQ2007, MQ2008, GOV22 (also referred to as Trec-terabyte 2004/2005/2006) and Robust04. MQ2007 and MQ2008 are standard LETOR [START_REF] Qin | Introducing LETOR 4.0 datasets[END_REF] benchmark datasets for learning to rank. GOV2 contains documents resulting from a crawl of .gov websites made in early 2004. Robust043 contains news article from the Financial Times, the Federal Register 94, the LA Times, and FBIS. In each dataset, the title of the topics have been used as queries and the content of the documents have been extracted using Anserini [START_REF] Yang | Anserini: Reproducible ranking baselines using Lucene[END_REF]. Relevance judgements can take three values: 0 (irrelevant), 1 (relevant) or 2 (very relevant). A document-query pair with an associated relevant judgement will be referred to as a labeled document-query pair. All our analyses are based on labeled document-query pairs so as to avoid assumptions on the relevance status of non-labeled pairs. Furthermore, as it has become common to first filter documents with a standard IR system prior to deploy deep IR models, we first select, for each query, the top 200 documents using BM25 and only retain the labeled document-query pairs associated with these documents. Note that the above filtering is not run on MQ2007 and MQ2008 which already rely on a small subset of documents for each query. Table 1 displays, for each collection, the number of queries and documents as well as the number of unique labeled document-query pairs (in the original dataset as well as the one filtered with BM25). The proportions of irrelevant, relevant and very relevant pairs are computed on the filtered version for GOV2 and Robust04, and on the original version for MQ2007 and MQ2008.

To divide documents into blocks, we use the recent CogLTX [START_REF] Ding | CogLTX: Applying BERT to Long Texts[END_REF] block decomposition method, which is based on a dynamic programming method, each block having a maximum of 63 tokens. This method, which was used Dataset MQ2007 MQ2008 GOV2 Robust04

Nb of queries with success on several NLP tasks, sets different costs for different punctuation marks and aims at segmenting in priority on strong punctuation marks such as ". " and "!", making it close to a sentence segmentation procedure. Lastly, to measure to which extent a block is relevant or not to a given query, we use the Retrieval Status Value (RSV) of the block which simply amounts here to the score provided by an IR model, in our case either BM25 or the cosine similarity computed on semantic representations of queries and blocks. The higher the score obtained, the likelier the corresponding block is relevant.

We now turn to answer the two questions we asked before.

Relevance signals appear at different positions in documents

We first analyze the length of documents with respect to the number of blocks they contain. To do so, we plot in Figure 1 the proportion of documents containing exactly 𝑛 blocks, 𝑛 varying from 1 to more than 125 4 . In addition, the median as well as the first and third quartiles of the distribution of the number of blocks for each dataset are provided. Even though the shape of the curve for each dataset varies, one can notice that more than 25% of the documents in MQ2007, MQ2008 and GOV2 contain more than 80 blocks, way above the size limitation of current transformer-based IR models. For Robust04, 25% of the documents have more than 30 blocks, a size that can also exceed the limitation of current transformer-based IR models 5 .

As the number of blocks contained in a document varies a lot from one document to another, for assessing where relevance signals appear in different documents, we consider 𝑝 positions (1, 2, • • • , 𝑝) and allocate each block of a document to one of the 𝑝 positions with the constraint that the first block of the document should be allocated to the first position, and the last block to the last position. This can be easily done with the following function which provides the position of the 𝑖 𝑡ℎ block of a document containing 𝑏 blocks (1 ≤ 𝑖 ≤ 𝑏):

𝑝𝑜𝑠 (𝑖) = 10 × 𝑖 𝑏 ,
where ⌈𝑥⌉ is the ceiling function which returns the integer greater than or equal to 𝑥. For example, the fifth block in a document containing 100 blocks is in the first position. Only documents that have at least 15 blocks are considered for analysis, preventing missing positions. query-document pairs (referred to as full dataset), for very relevant and relevant pairs (referred to as relevant only) and for irrelevant pairs (referred to as irrelevant only). On all collections, the average RSV score decreases when the position increases. However, the average RSV scores are still important in the middle positions and non-negligible in the last positions. Furthermore, if one assumes that all blocks of an irrelevant document are irrelevant 6 , then the difference, for a given position, between the average RSV scores of relevant and irrelevant pairs may serve as an additional indicator of whether or not relevance signals are found in that particular position. When there is no difference between the average RSV scores of relevant and irrelevant pairs at a given position, then the only conclusion one can draw is that all blocks from relevant and irrelevant documents behave in the same way with respect to the RSV score, and there is no indication that relevance signals are present at that position. On the contrary, when the average RSV score for relevant pairs is significantly higher than the one for irrelevant pairs at a given position, then there is a clearer indication that there are relevant signals at that position. This is the case for all positions of the four collections.

To complement the above analysis, we used a different RSV score, namely the cosine similarity between the semantic representations of blocks and queries obtained with the pre-trained Sentence-BERT [START_REF] Reimers | Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks[END_REF] model 7 (we simply input each query-block pair to Sentence-BERT which outputs query and block representations on which a cosine similarity is computed). Contrary to the BM25 score which is mainly 'lexical', this score aims to capture additional semantic relationships between queries and blocks. Figure 3 displays the distribution of the average cosine similarity RSV score on each position. This analysis confirms that all positions can contain relevance signals. Furthermore, the decrease in RSV scores when the position of the block increases is less marked so that the difference between blocks at the beginning, the middle and the end of a document (except the last block which gets in general significantly smaller scores) is not really important. Overall, our analysis reveals that, if terms at the beginning of a document may be more important that terms at the end, all positions in a document can contain important relevance signals and should a priori be explored for IR purposes. This conclusion is in agreement with the verbosity hypothesis [START_REF] Robertson | Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval[END_REF].

Fuzzy matching may help select better blocks on some collections

The second question we address is whether one should solely rely on an exact matching of the words present in the query to select a given block or whether fuzzy matching including words related to the query words may help retrieve better blocks. We consider here that a word related to a query word is any synonym, as provided by WordNet [START_REF] Fellbaum | WordNet: An Electronic Lexical Database[END_REF], of that query word. Other semantic relations can of course be used; we chose synonymy because it has the advantage of being a simple relation which is at least partly captured in modern word embeddings and for which useful resources such as WordNet are available. Our goal here is to assess whether using synonyms can help select useful blocks. If this is the case, then one can conclude that it may be useful to use matching strategies that go beyond an exact matching of query words. It is important to note here that if many studies have been devoted to the utility of synonyms in IR systems [START_REF] Li | Multi-view embedding-based synonyms for email search[END_REF][START_REF] Peter | Mining the web for synonyms: PMI-IR versus LSA on TOEFL[END_REF], our study differs from them in that it focuses on the use of synonyms for selecting blocks and does not aim to assess different query expansion strategies. In particular, we are not interested in assigning different weights to expanded terms even though such a strategy may lead to better query expansion results [START_REF] Fang | A re-examination of query expansion using lexical resources[END_REF][START_REF] Fang | Semantic term matching in axiomatic approaches to information retrieval[END_REF].

We first try to answer the question: Can synonyms identify blocks that would not have been identified without them? To do so, from the original query, we construct three boolean queries. The first one consists of the disjunction of all query words and will be referred to as 𝑞 𝑏𝑜𝑜𝑙 . The second one consists of the disjunction of all the synonyms from WordNet of all query words and excludes the original query words. It will be referred to as 𝑞 𝑠𝑦𝑛 𝑏𝑜𝑜𝑙 . Lastly, the third one, referred to as 𝑞 𝑟𝑎𝑛𝑑 𝑏𝑜𝑜𝑙 , which has the same length as the second one, consists of the disjunction of words randomly selected from WordNet. This last query helps assess to which extent the phenomena observed depend solely on the query length 8 . Table 2 provides an example of these three types of (("minimal" OR "earnings" OR "pay" OR "remuneration" OR "salary" OR "growth" OR "gain" OR "addition") AND NOT ("minimum" OR "wage" OR "increase"))

Random extended query boolean representation 𝑞 𝑟𝑎𝑛𝑑 𝑏𝑜𝑜𝑙 (("cadent" OR "gravely" OR "stuffiness" OR "puller" OR "complaisant" OR "sunlight" OR "profusely" or "asterism") AND NOT ("minimum" OR "wage" OR "increase")) 3 with the percentage increase with respects to the number of blocks matched by 𝑞 𝑏𝑜𝑜𝑙 . One can observe that leveraging the knowledge about synonyms enables to match more blocks: up to 22.85% increase in the number of blocks matched in the case of the Robust04 dataset (against 0.33% for the random query), 14.70% increase for GOV2 (against 0.26% for the random query, 8.72% in the case of MQ2007 (against 0.23% for the random query) and 5.26% increase for MQ2008 (against 0.16% for the random query).

To complement the above analysis, we also assessed whether synonyms can leverage relevance signals in blocks at different positions. To do so, we used a standard expansion of the original query by simply adding all synonyms of all query terms without duplicates. For comparison purposes, we did the same with the words randomly selected in WordNet. The obtained queries, an example of which is given in Table 2, will be respectively referred to as 𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙 + 𝑠𝑦𝑛𝑜𝑛𝑦𝑚𝑠 and 𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙 + 𝑟𝑎𝑛𝑑𝑜𝑚. We then computed the difference in average BM25 scores between relevant and irrelevant documents across all positions for the three types of queries: Original, Original + synonyms, Original + random. The results obtained are reported in Figure 4. As one can note, blocks in relevant documents score higher than the blocks in irrelevant documents, the gap being consistently and significantly higher when using the query with synonyms than when using the original query or the original query with additional random words. Furthermore, the two curves, Original and Original + random, are very close to each other and almost identical on GOV2. This shows that the increase in the BM25 scores when using synonyms is not due to the length of the query, and that synonyms help identify relevant signals in blocks. It thus may be useful to use matching strategies that go beyond an exact matching of query words.

In the next sections, we will present two different ways to select blocks in documents. The first one is based on standard IR functions, namely TF-IDF and BM25, to compute relevance scores between queries and blocks; it is thus based on an exact matching of words present in the query. The second one aims at learning a scoring function that exploits the semantic similarities between query words and document words. In both cases, the top scoring blocks are then used to compute the relevance score of the document.

SELECTING BLOCKS WITH STANDARD IR FUNCTIONS: TF-IDF AND BM25

TF-IDF is a popular IR model which amounts to score a document through the product of the term frequency (TF) and inverse document frequency (IDF) scores of the query words present in that document. Applied at the block level, this yields the following retrieval status value (RSV):

𝑅𝑆𝑉 (𝑞, 𝑏) 𝑇 𝐹 -𝐼 𝐷𝐹 = ∑ 𝑤 ∈𝑞∩𝑏 (ln 𝑡 𝑓 𝑏 𝑤 + 1)
𝑇 𝐹

•𝐼 𝐷𝐹 (𝑤).

ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x.

in which 𝑡 𝑓 𝑏 𝑤 corresponds to the number of occurrences of word 𝑤 in block 𝑏 and 𝐼 𝐷𝐹 (𝑤) to the inverse document frequency of word 𝑤. 𝐼 𝐷𝐹 (𝑤) is defined here according to scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] by:

𝐼 𝐷𝐹 (𝑤) = ln 𝑁 + 1 𝑑 𝑓 𝑤 + 1 ,
where 𝑁 is the number of documents in the collection and 𝑑 𝑓 𝑤 corresponds to the number of documents containing 𝑤.

For BM25 [START_REF] Robertson | The probabilistic relevance framework: BM25 and beyond[END_REF][START_REF] Robertson | The Probabilistic Relevance Framework: BM25 and Beyond[END_REF], the RSV score is defined by:

𝑅𝑆𝑉 (𝑞, 𝑏) 𝐵𝑀25 = ∑ 𝑤 ∈𝑞∩𝑏 𝐼 𝐷𝐹 (𝑤) • 𝑡 𝑓 𝑏 𝑤 𝑘 1 • (1 -𝑏 + 𝑏 • 𝑙 𝑏 𝑙 𝑎𝑣𝑔 ) + 𝑡 𝑓 𝑏 𝑤 ,
where 𝑙 𝑏 is the length of block 𝑏, 𝑙 𝑎𝑣𝑔 the average length of the blocks in 𝑑, and 𝑘 1 and 𝑏 two hyperparameters. As standard in this setting, we use the IDF formulation of Lucene version as shown in Kamphuis et al. [START_REF] Kamphuis | Which BM25 do you mean? A large-scale reproducibility study of scoring variants[END_REF] which slightly differs from the previous one:

𝐼 𝐷𝐹 (𝑤) = ln 𝑁 + 1 𝑑 𝑓 𝑤 + 0.5 .
According to [START_REF] Nottelmann | From retrieval status values to probabilities of relevance for advanced IR applications[END_REF], RSVs in different IR systems have different scales. In the boolean model, RSVs are either 0 or 1. Vector space model can generate RSV in [-1, 1] by cosine similarity or scalar product R. In this section, we rely on TF-IDF and BM25, which can be viewed as an model that rely on term matching and the scale of the RSVs is R. The blocks are ranked according to RSVs by descending order.

Block or document level IDF. As the reader may have noticed, the IDF is based on documents and not on blocks. There are two main reasons for this. First of all, considering blocks instead of documents for the IDF may artificially increase the 𝑑 𝑓 score of a word since important words of a document are likely to occur in many blocks of that document. The second reason is that one can directly re-use existing IDF scores computed at the document level. Note that all words are lowercased prior to compute TF-IDF and BM25 scores.

The overall architecture of an IR neural system relying on the above standard IR models to select blocks is presented in Figure 5. As one can note, the query-block scoring part is used to select relevant blocks across the whole document, which can be viewed as a pre-ranking strategy. The Deep Neural IR Network can represent any neural IR network which generates relevance scores for query-document pairs, scores which can in turn be used as input to a learning-to-rank (LTR) loss, be it a pairwise or listwise loss. We focus in this study on two state-of-the-art neural IR Models, namely Vanilla BERT, in which case we refer to the models obtained as KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 and KeyB(vBERT) 𝐵𝑀25 , and PARADE, in which case we refer to the models obtained as KeyB(PARADE𝑘) 𝑇 𝐹 -𝐼 𝐷𝐹 and KeyB(PARADE𝑘) 𝐵𝑀25 , 𝑘 representing in that case the number of passages retained. Figure 6 provides an illustration of Vanilla BERT and PARADE.

KeyB(vBERT)

The KeyB(vBERT) models rely on four steps:

(1) Block segmentation We adopt here the dynamic programming method proposed in [START_REF] Ding | CogLTX: Applying BERT to Long Texts[END_REF] to segment documents into blocks, each block having a maximum of 63 tokens. This method sets different costs for different punctuation marks and aims at segmenting in priority on strong punctuation marks such as ". " and "!". It is thus close to a sentence segmentation procedure. (2) Block selection Each block is assigned a relevance score provided by either TF-IDF or BM25, as described above.

(3) Query-blocks representation The most relevant blocks are then concatenated together in their order of appearance in the document and with the query to form the input of BERT (see Figure 6). As the input number of tokens for BERT is limited to 512, the last block is truncated if necessary. The number 𝑛 of selected blocks depends on the capacity of BERT and is defined by:

3 + 𝑙 𝑞 + 𝑛-1 ∑ 𝑖=1 𝑙 𝑏 𝑖 + trunc(𝑙 𝑏 𝑛 ) = 512,
where 𝑙 𝑞 denotes the length of the query and 𝑙 𝑏 𝑖 the length of a block, potentially truncated for the last selected block. The final query-representation corresponds to the [CLS] embedding of the learned BERT.

(4) Document ranking We rely here on a one-layer dense, linear network to generate the final relevance score, using a learning to rank loss computed on a mini-batch (see Section 4.3).

KeyB(PARADE𝑘)

As mentioned in Section 2, PARADE is a state-of-the-art model which computes a query-document representation on the basis of the query-passage representations. A document is first segmented into passages. Each passage is then fed, together with the query, to a BERT model. The CLS embedding thus obtained corresponds to the querypassage representation. Denoting by 𝑝 𝑖 the 𝑖 𝑡ℎ passage and 𝑝 𝑐𝑙𝑠 𝑖 the corresponding query-passage representation, one has:

𝑝 𝑐𝑙𝑠 𝑖 = 𝐵𝐸𝑅𝑇 (𝑞, 𝑝 𝑖 ). The query-passage representations are then aggregated to obtain the query-document representation, that is finally fed to a feed-forward neural network to obtain the relevance score of the document.

Four types of aggregation methods have been proposed: PARADE-Max, PARADE-Attn, PARADE-CNN and PARADE-Transformer. PARADE-Max uses a max pooling operation on the passage relevance representations. PARADE-Attn assumes that each passage contributes differently to the relevance of a document to the query and passage weights are predicted by a feed-forward network. PARADE-CNN stacks Convolutional Neural Network (CNN) layers in a hierarchical way whereas PARADE-Transformer stacks a full attention model which enables query-passage representations to interact in a hierarchical manner through a transformer. Its architecture is depicted in Fig. 6b. Because of its good behavior [START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF], we have retained this last aggregation method here. We will refer to is as just PARADE in the remainder.

Let 𝑥 (ℓ) denote the input of the ℓ transformer layer. 𝑥 (0) consists in the concatenation of the query-passage representations (𝑝 𝑐𝑙𝑠 𝑖 ). 𝑥 (ℓ+1) is then given by: 𝑥 (ℓ+1) = LayerNorm(ℎ + FFN(ℎ)), with ℎ = LayerNorm(𝑥 (ℓ) + MultiHead(𝑥 (ℓ) )).

LayerNorm refers to the layer-wise normalization described in [START_REF] Lei Ba | Layer normalization[END_REF] and MultiHead to the multi-head self-attention [START_REF] Vaswani | Attention is all you need[END_REF]. FFN is a two-layer feed-forward network with a ReLU activation in between. The [CLS] vector of the final output layer, which is denoted by 𝑑 𝑐𝑙𝑠 , constitutes the query-document representation. A linear layer is then used to generate the query-document relevance score:

𝑅𝑆𝑉 (𝑞, 𝐷) = 𝑊 𝑑 𝑐𝑙𝑠 , where 𝑊 is a learnable weight matrix.

It is important to note that the PARADE model described above can have a high complexity when the number of passages considered is large (for example, when using 10 passages, the model can not fit on a standard GPU with 11 GB memory even with mixed precision). Indeed, in that case, the input consists in a large tensor which can only be stored in a large CUDA memory [START_REF] Gao | Estimating gpu memory consumption of deep learning models[END_REF] with high computational complexity. To avoid this, the number of passages is limited to 16. When a document contains more than 16 passages, then only the first passage, the last passage and 14 sampled passages are used. Whether one restricts the number of passages or not, one problem faced by PARADE lies in the fact that non relevant passages can bring noise in the query-document representation, and hamper the computation of a precise retrieval score. To address this problem, we propose here to select a fixed, small number of passages, denoted by 𝑘. The selected passages are then concatenated and fed to a transformer as in the original PARADE model. When using standard IR functions as described above to select passages, the query-document representations denoted by 𝑑 𝑐𝑙𝑠 𝑇 𝐹 -𝐼 𝐷𝐹 and 𝑑 𝑐𝑙𝑠 𝐵𝑀25 are thus obtained. The final relevance score of the document is then obtained by: 𝑅𝑆𝑉 (𝑞, 𝐷) = 𝑊 𝑑 𝑐𝑙𝑠 𝑇 𝐹 -𝐼 𝐷𝐹 or 𝑊 𝑑 𝑐𝑙𝑠 𝐵𝑀25 .

Model training

The block/passage selection process is applied in both the training and testing phases. The BERT models used in Vanilla BERT and PARADE are fine-tuned during training. The parameters of all the other components (final layer for Vanilla BERT, final layer and transformer layers for PARADE) are directly trained. For fine-tuning and training, the following pairwise hinge loss [START_REF] Guo | A deep relevance matching model for ad-hoc retrieval[END_REF] is used:

L (𝑞, 𝑑 + , 𝑑 -; Θ) = max(0, 1 -𝑠 (𝑞, 𝑑 + ; Θ) + 𝑠 (𝑞, 𝑑 -; Θ)),
where 𝑞 represents a query, (𝑑 + 𝑞 , 𝑑 - 𝑞 ) a positive and negative training document pair for 𝑞, Θ the parameters of the model considered and 𝑠 the predicted relevance score for a query-document pair. Other choices are of course possible, including ones based on a listwise loss. We chose the pairwise hinge loss here as it is a very common choice, used in many IR methods [START_REF] Li | KeyBLD: Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval[END_REF][START_REF] Macavaney | CEDR: Contextualized embeddings for document ranking[END_REF][START_REF] Pang | Deeprank: A new deep architecture for relevance ranking in information retrieval[END_REF].

LEARNING TO SELECT BLOCKS

The analysis conducted in Section 3 suggests that a fuzzy matching procedure may be preferred over one based on an exact matching. We thus aim here to learn a scoring function that exploits the semantic similarities between words in queries and blocks using the same two neural IR model as before, Vanilla BERT and PARADE.

Improving Vanilla BERT

We focus here on the Vanilla BERT model to compute the relevance score of a block. That is, instead of just using the semantic representation of query-block pairs to compute the relevance score of a document as in the previous models, we also make use here of these representations to select the most appropriate blocks. Furthermore, we propose here to share the semantic representation for both purposes, selecting blocks and computing the overall relevance score, as both are based on the relevance information contained in each block. It is of course possible to make use of two different models, with however higher computational and training costs. The overall architecture of the model proposed is depicted in Figure 7, in which the same BERT model and linear layer are used at different time slices, first to compute a query-block representation, from which ([CLS] embedding) the relevance score of the block is derived, and then to compute the query-document representation ([CLS] embedding) based on the top ranked blocks, finally to obtain the score of the document. This query-document representation is identical to the one used in the KeyB(vBERT) model, the only difference lying in the way the blocks are selected. For the query-block represntation, both the BERT model and the linear layer are only used for scoring and are not updated via back-propagation (hence the term "eval model" used in Figure 7). The score of a block 𝑏 for a given query 𝑞 is defined by: 𝑅𝑆𝑉 (𝑞, 𝑏) 𝐵𝐸𝑅𝑇 = 𝑊 𝑑 𝑏 𝑐𝑙𝑠 , where 𝑊 𝑑 is the weight of the dense linear layer on top of BERT, and 𝑏 𝑐𝑙𝑠 is the query-block relevance representation:

𝑏 𝑐𝑙𝑠 = 𝐵𝐸𝑅𝑇 (𝑞, 𝑏).
For a given query, the BERT model is first used to generate a relevance score for each block in the document. Since the BERT model is not well fine-tuned at the beginning of the process, the block selection process acts as an almost random selector. However, with the query-document level relevance labels, after each back-propagation, the BERT model is improved and provides better relevance scores for each block in the next iterations. In return, the BERT model benefits from the block selection too. This can be viewed as a self-evolution process: the BERT model evolves to provide appropriate query-block representations to be able to select blocks, and meanwhile, appropriate query-document representations to be able to generate relevance scores for query-document pairs. Because of this self-evolution, we refer to this model as KeyB(vBERT) 𝐵𝑖𝑛𝐵 , where 𝐵𝑖𝑛𝐵 means "BERT in BERT".

Improving PARADE

We here propose to improve the PARADE model with the learning approach for selecting passages.

To do so, we first follow the same strategy as the one used for Vanilla BERT by trying to use the same modules for both selecting passages and scoring documents. A major difficulty for doing so here is that the Transformer encoder used in PARADE to score a complete document takes as input the representation of several query-block pairs. As such, it cannot be used to score a single block. For this reason, we first propose to only share the initial BERT model and the final feed-forward neural network as illustrated in Figure 8. In this new architecture, one obtains the [CLS] query-passage representation, denoted by 𝑝 𝑐𝑙𝑠 for passage 𝑝, using the base BERT model of PARADE which is shared in different time slices. The final feed-forward neural network is used to provide, from a CLS representation, a score for a query-passage pair (Fig. 8, left) and a score for a query-document pair, where a document is seen as the concatenation of passages (Fig. 8,right). The score of a passage 𝑝 for a given query 𝑞 is then defined by: 𝑅𝑆𝑉 (𝑞, 𝑝) 𝐵𝐸𝑅𝑇 = 𝑊 𝑑 𝑝 𝑐𝑙𝑠 , where 𝑊 𝑑 is the weight of the feed-forward neural network after the Transformer encoder, and 𝑝 𝑐𝑙𝑠 is the query-passage relevance representation:

𝑝 𝑐𝑙𝑠 = 𝐵𝐸𝑅𝑇 (𝑞, 𝑝). We refer to this approach as KeyB(PARADE𝑘) 𝐵𝑖𝑛𝐵 , where 𝑘 is the number of selected passages.

If the previous attempt makes it possible to reuse modules for selecting passages and scoring documents, it may however suffer from the fact that the same feed-forward neural network is required to produce a relevance score from two different CLS representations: one restricted to a single query-passage pair for selecting passages, and one resulting from an encoding, through a Transformer, of several query-passage representations for scoring the document. We propose to fix this issue by decoupling the passage scoring module from the main model, as illustrated in Figure 9 which relies on a different BERT module and feed-forward neural network to select passages. But which BERT model and feed-forward layer to use? A simple answer to this question is to re-use the pretrained BERT ranker and final feed-forward neural network of one of the KeyB(vBERT) models previously proposed as these models are compatible with the input used here. The score of a passage 𝑝 for a given query 𝑞 is in this case defined by:

𝑅𝑆𝑉 (𝑞, 𝑝) 𝐵𝐸𝑅𝑇 2 = 𝑊 2 𝑑 𝑝 𝑐𝑙𝑠 2 , where 𝑊 2
𝑑 is the weight of the feed-forward neural network following the additional BERT module, and 𝑝 𝑐𝑙𝑠 2 is the query-passage relevance representation after the additional BERT module:

𝑝 𝑐𝑙𝑠 2 = 𝐵𝐸𝑅𝑇 2(𝑞, 𝑝).
In practice, we use the BERT module and feed-forward neural network from the KeyB(vBERT) 𝐵𝑀25 model and refer to this approach as KeyB(PARADE𝑘) 𝐵𝑖𝑛𝐵2 , where 𝑘 is the number of selected passages.

EXPERIMENTS ON STANDARD IR COLLECTIONS

We conducted a first series of experiments on the same collections as the ones used in the analysis presented in Section 3, namely MQ2007, MQ2008, Robust04 and GOV2. These experiments aim at assessing the validity of the models proposed, KeyB(vBERT) and KeyB(PARADE𝑘) 10 , and at comparing them to models that have provided state-of-the-art results on these collections. These latter models are: • CEDR-KNRM: CEDR is also a reported state-of-the-art model that incorporates BERT's classification vector into existing neural models. We choose the CEDR-KNRM as the baseline for it is reported better than other variants. We have used the Georgetown IR Lab implementation 13 with the Hugging Face transformer module 14 .

For assessing the various components of the models proposed, we also used the following baseline models:

• BM25: We use the BM25 implementation of Anserini [START_REF] Yang | Anserini: Reproducible ranking baselines using Lucene[END_REF], with default hyperparameters. This model is the one presented in Section 4 and is used both as a baseline on all collections. In addition, it is used as a first stage ranker, retaining only the top 200 documents, for the neural IR models on Robust04 and GOV2. For MQ2007 and MQ2008, we converted the original documents to JSON files which can be indexed by Anserini.

We then used the BM25 model from Anserini as a baseline but not as a first stage ranker as in MQ2007 and MQ2008 each query only contains dozens of labeled documents for ranking. For all experiments, the hyper-parameters 𝑘1 and 𝑏 of BM25 are set to Anserini's default values: 0.9 and 0.4. • Vanilla BERT: This is a BERT baseline that truncates long documents to the first 512 tokens. Except for this difference in the input used for BERT, the architecture is the same as the one of KeyB(vBERT). This baseline thus allows one to evaluate the impact of key blocks. For implementation, we have used the same library as the one for CEDR-KNRM. • Random Select: This architecture is the same as the one with KeyB(vBERT) except that it does not incorporate the block selection mechanism. To be specific, a long document is also firstly segmented into blocks, but each block is given a random score. This is to say, without local block pre-ranking step, the blocks are selected randomly.

Experimental design

It is common in IR to first filter documents with a classical IR system prior to re-rank them with a more complex (and usually more time consuming) system [START_REF] Dai | Deeper text understanding for IR with contextual neural language modeling[END_REF][START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF][START_REF] Macavaney | CEDR: Contextualized embeddings for document ranking[END_REF][START_REF] Nogueira | Passage Re-ranking with BERT[END_REF]. We adopt here this approach and use BM25 as the first filtering system, retaining only, for each query, the first 200 documents on Robust04 and GOV2 as done in [START_REF] Macavaney | CEDR: Contextualized embeddings for document ranking[END_REF].

As queries in MQ2007 and MQ2008 are associated with far less than 200 documents, this filtering step is not necessary. Following [START_REF] Pang | Deeprank: A new deep architecture for relevance ranking in information retrieval[END_REF], we merged the MQ2007 and MQ2008 training sets as the training set of MQ2008 is relatively small. The validation and testing sets remain unchanged. For all neural IR models, the pairwise hinge loss [START_REF] Pang | Deeprank: A new deep architecture for relevance ranking in information retrieval[END_REF] is used for training the models. Furthermore, for all experiments, 5-fold cross-validation is used with three folds for training, one fold for validation and one fold for testing. For Robust04 and GOV2, we used the keyword (title) version of queries [START_REF] Dai | Deeper text understanding for IR with contextual neural language modeling[END_REF]. For MQ2007 and MQ2008, the default queries are used. All neural IR models based on BERT use the "BERT-Base, Uncased, L=12, H=768" 15 pre-trained language model (but not further pre-trained with additional data) for fair comparison.

For DeepRank, we followed the experimental setup of [START_REF] Pang | Deeprank: A new deep architecture for relevance ranking in information retrieval[END_REF] and used GloVe embeddings [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] of dimension 50, which are pre-trained on Wikipedia 2014 + Gigaword 5. For the preprocessing of document and query words, we applied lower-case, removed English stop words, stemmed with Krovetz stemmer [START_REF] Krovetz | Viewing morphology as an inference process[END_REF] and removed words occurring less than 5 times. The Adam optimizer is used for training the network and the learning rate is searched over values of {0.01, 0.005, 0.001}. We selected the model that leads to the best MAP score on the validation set.

For PARADE, the first and last passages are always selected and the other passages are randomly sampled 16 . The number of passages considered is a hyperparameter in PARADE that needs to be set. Following [START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF], the passages are obtained with 225 document tokens with stride size 200, the maximum passage sequence length is set as 256, and the number of passages is set to 16 for all collections. Note again that for a fair comparison, BERT is not further trained on MS-MARCO [START_REF] Nguyen | MS MARCO: A human generated machine reading comprehension dataset[END_REF]. For all BERT based IR models, we use the BERT implementation of PyTorch Huggingface library [START_REF] Wolf | HuggingFace's Transformers: State-of-the-art natural language processing[END_REF].

For the variants of PARADE we have proposed in Section 4.2, we have used TF-IDF or BM25 to select the top 5 passages. The choice of 5 passages provides a good balance for effectiveness, as ca. 1000 tokens are considered, and efficiency, as a standard RTX 2080ti GPU with 11GB memory is not able to deal with 12 passages and automatic mixed precision [START_REF] Micikevicius | Mixed Precision Training[END_REF] for example. The resulting models, based on PARADE and integrating the top 5 passages using TF-IDF or BM25 for local pre-ranking, are referred to as KeyB(PARADE5) 𝑇 𝐹 -𝐼 𝐷𝐹 and KeyB(PARADE5) 𝐵𝑀25 . The variants of PARADE proposed in Section 5.2 and with top 5 passages are are referred to as KeyB(PARADE5) 𝐵𝑖𝑛𝐵 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 . We here use the "cross-encoder/ms-marco-MiniLM-L-12-v2" 17 version model as the standalone BERT ranker for KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 . This standalone BERT ranker is firstly finetuned on each collection as the way of KeyB(PARADE5) 𝐵𝑀25 (trained with document level labels), then it can generate a RSV for each query-passage pair (this means that a query and passage are concatenated and directly input to the BERT ranker). All other settings are the same as the ones for the PARADE model described above. For comparison purposes, we also used another variant of PARADE, called PARADE5, relying on the first, last and 3 randomly chosen passages.

Each model is trained for a maximum of 10 epochs. For Robust04 and GOV2, one epoch represents 1024 batches of two pairs, each pair being of the form ((𝑞, 𝑑 + 𝑞 ), (𝑞, 𝑑 - 𝑞 )) where 𝑑 + 𝑞 is a positive document for query 𝑞 and 𝑑 - 𝑞 a negative document. Since MQ2007 and MQ2008 have more queries than Robust04 and GOV2, each epoch of these two collections is composed of 2048 batches of two pairs identical to the ones above. The negative example in a pair is generated randomly for all models from the set of documents which are either labeled not relevant or not labeled for the query. Although different negative sampling mechanisms may impact final results [START_REF] Lu | Neural passage retrieval with improved negative contrast[END_REF], the above simple negative sampling strategy achieves very good performance and has been successfully used in previous studies [START_REF] Li | KeyBLD: Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval[END_REF][START_REF] Macavaney | CEDR: Contextualized embeddings for document ranking[END_REF][START_REF] Pang | Deeprank: A new deep architecture for relevance ranking in information retrieval[END_REF]. Gradient accumulation is employed every 8 steps to fit on a single GPU with 11GB memory like a RTX 2080ti GPU, simulating a batchsize with 16 training pairs, as done in [START_REF] Macavaney | CEDR: Contextualized embeddings for document ranking[END_REF]. Automatic mixed precision [START_REF] Micikevicius | Mixed Precision Training[END_REF] is used to speed up training. We adopt a validation mechanism with validation set to report each metric on the test set. That is to say, for each evaluation metric, we obtain the best performing model in the 10 epochs on the validation set, and use this model to obtain results on the test set for this metric. Each model is trained using Adam optimizer (the transformer layers are trained with a rate of 2 * 10 -5 while the linear layer with a rate of 10 -3 ).

Results, for all models, are measured with P@1, P@5, P@10, P@20, MAP, NDCG@1, NDCG@5, NDCG@10, NDCG@20 and NDCG, NDCG and MAP being computed on all available documents for each query (the number of documents per query is 200 on Robust04 and GOV2 and varies from one query to the other on MQ2007 and MQ2008). Each metric is calculated with pytrec_eval 18 [START_REF] Van Gysel | Pytrec_eval: An extremely fast python interface to trec_eval[END_REF], which is a wrapper of trec_eval 19 . Lastly, a paired t-test is used to assess whether differences are significant or not.

Experimental results

The results obtained on all the four collections are displayed in Tables 4 to 7 20 . We first analyze the results of the KeyB(vBERT) models, prior to analyze the ones of KeyB(PARADE5) 𝐵𝑀25 and compare them.

Improving

Vanilla BERT model with selected key blocks. We propose to analyze the experimental results by answering several research questions.

RQ1 How effective are KeyB(vBERT) models compared to baseline models (BM25, DeepRank)? 17 https://www.sbert.net/docs/pretrained_cross-encoders.html 18 https://github.com/cvangysel/pytrec_eval 19 https://trec.nist.gov/trec_eval 20 As pointed out in e.g. https://github.com/Georgetown-IR-Lab/cedr/issues/22, the results obtained for CEDR-KNRM with the code provided by the authors differ from the ones reported in the original paper. We have also observed this in our experiments. The same holds for DeepRank; in that case however the original paper provides results for most of the metrics we have retained on MQ2007 and MQ2008. We have thus reported the original results in Tables 6 and7, under the name DeepRank * . These results do not change our conclusions. Table 4. Results on Robust04 dataset. Best results are in bold. For KeyB(vBERT) models, a significant difference with BM25 is marked with a 'B', with DeepRank with a 'D', with Vanilla BERT with a 'V', with CEDR-KNRM with a 'C', and with Random select with an 'R'. For KeyB(PARADE) models, a significant difference with BM25 is marked with a 'B', with DeepRank with a 'D', with PARADE with a 'P', and with PARADE5 with a '5'. A paired t-test (𝑝 -𝑣𝑎𝑙𝑢𝑒 ≤ 0.05) is used for measuring significance.

Model

P@1 P@5 P@10 P@20 MAP NDCG@1 NDCG@5 NDCG@10 NDCG@20 NDCG The first conclusion we draw from Tables 4 to 7 is that all KeyB(vBERT) models outperform both baseline models on all collections, for all metrics, by a large margin. Furthermore, on all collections, KeyB(vBERT) models significantly outperform most of metrics.

RQ2 How effective are KeyB(vBERT) models compared to standard BERT based models (Vanilla BERT, CEDR-KNRM)? As one can see, KeyB(vBERT) 𝐵𝑀25 and KeyB(vBERT) 𝐵𝑖𝑛𝐵 models outperform standard BERT models (Vanilla BERT and CEDR-KNRM) on all collections and for all metrics. KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 outperforms standard BERT models (Vanilla BERT and CEDR-KNRM) for all metrics on all collections except Robust04, for which it yields Table 6. Results on MQ2007 dataset. DeepRank* represents the results from the original paper. Best results are in bold. For KeyB(vBERT) models, a significant difference with BM25 is marked with a 'B', with DeepRank with a 'D', with Vanilla BERT with a 'V', with CEDR-KNRM with a 'C', and with Random select with an 'R'. For KeyB(PARADE) models, a significant difference with BM25 is marked with a 'B', with DeepRank with a 'D', with PARADE with a 'P', and with PARADE5 with a '5'. A paired t-test (𝑝 -𝑣𝑎𝑙𝑢𝑒 ≤ 0.05) is used for measuring significance.

Model

P@1 P@5 P@10 P@20 MAP NDCG@1 NDCG@5 NDCG@10 NDCG@20 lower results than Vanilla BERT on P@5, NDCG@5, and lower result than CEDR-KNRM on P@1, P@5 and NDCG@1. In addition, the best KeyB(vBERT) model (on each metric respectively) significantly improves the Vanilla BERT model on 6 metrics out of 10 on Robust04, on 9 metrics out of 10 on GOV2, on all metrics on MQ2007 and on 5 metrics out of 10 on MQ2008; it is furthermore significantly better than CEDR-KNRM on 2 metrics on Robust04, on 9 metrics out of 10 on GOV2, on all metrics on MQ2007, and on 6 Metrics out of 10 on MQ2008.

RQ3 Is it important to accurately select blocks?

Analysis of the position of selected blocks

We are finally interested here in analyzing the positions at which blocks are selected. To do so, we retained all documents containing at least 15 blocks and looked at which position the top eight scoring blocks occur for two models with different selection strategies, namely KeyB(vBERT) 𝐵𝑀25 and KeyB(vBERT) 𝐵𝑖𝑛𝐵 . The position is computed as in Section 3. The results obtained are displayed in Figure 11 for KeyB(vBERT) 𝐵𝑀25 and in Figure 12 for KeyB(vBERT) 𝐵𝑖𝑛𝐵 . In both Figures, a heat map is used to represent the probability the blocks selected occupy a particular position. As one can observe, both models are more likely to select blocks at the beginning of a document, this tendency being more marked in KeyB(vBERT) 𝐵𝑀25 . Interestingly, KeyB(vBERT) 𝐵𝑖𝑛𝐵 is more likely to select blocks in the second position (and even in the third position) than in the first position. This said, both models also have a non null probability to select blocks at later positions. For example, in MQ2008, the possibility of selecting blocks in the last five positions, i.e., in the second half of a document, amounts to 41.8% for KeyB(vBERT) 𝐵𝑀25 and to 42.7% for KeyB(vBERT) 𝐵𝑖𝑛𝐵 . The situation is similar on MQ2007 and GOV2, as well as on Robust04 for KeyB(vBERT) 𝐵𝑖𝑛𝐵 . These results, in line with the analysis conducted in Section 3, show the capacity of the proposed models to rely on blocks at different positions in the document; the good performance of these models further shows that the blocks selected tend to contain relevant information.

To conclude this series of experiments on standard IR collections and for illustration purposes, we display an example in Figure 13 top 8 blocks selected by the KeyB(vBERT) 𝐵𝑖𝑛𝐵 model on MQ2007. As one can see, the different blocks are distributed across different positions at the beginning, middle and end of the document. Furthermore, each block is related to the query.

EXPERIMENT ON TREC 2019 DL AND COMPARISON WITH SPARSE ATTENTION BASED MODELS AND IDCM

This section sheds additional light on the behaviour of the models proposed by comparing them to different baseline models, including sparse attention models and IDCM discussed in Section 2, on the relatively recent TREC DL dataset introduced to evaluate neural IR models. Table 10 summarizes the main characteristics of the the TREC 2019 Deep Learning Track collection. Following [START_REF] Jiang | Long Document Ranking with Query-Directed Sparse Transformer[END_REF], we aim here to rerank the official top 100 retrieved documents and use the official evaluation metric NDCG@10 on the test set, which we complement with MAP. Since this collection is larger than previously used collections, we train the models, again using the pairwise hinge loss [START_REF] Guo | A deep relevance matching model for ad-hoc retrieval[END_REF], for longer steps, i.e. for 10 epochs each epoch being composed of 15000 batches of 2 pairs (four documents). As the qrels for the training and validation sets only contain one annotated document for each query (and it is relevant), each pair is composed of a query, its relevant document and another randomly sampled document which is viewed as irrelevant. For each metric and each model, we select the hyper-parameters leading to the lowest loss on the validation set and report its performance on the test set. The other experimental settings are the same as those in Section 6.1. experiments conducted on the TREC 2019 Deep Learning Track collection [START_REF] Craswell | Overview of the trec 2019 deep learning track[END_REF] showed that QDS-Transformer improves the standard retrofitting BERT ranking baselines and outperforms more recent transformer architectures as Sparse Transformer [START_REF] Child | Generating Long Sequences with Sparse Transformers[END_REF], Longformer [START_REF] Beltagy | Longformer: The Long-Document Transformer[END_REF], and Transformer-XH [START_REF] Zhao | Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention[END_REF]. We compare here our proposed approach with this QDS-Transformer and related baselines on this same collection. Table 11 shows the results obtained. Note that for PARADE, the number of passages is set to 16 and the max query length to 30 (other settings are the same as in Section 6.1). For the other models, we report the results given in [START_REF] Jiang | Long Document Ranking with Query-Directed Sparse Transformer[END_REF]. As one can see, the best results are obtained with KeyB(vBERT) 𝐵𝑖𝑛𝐵 which outperforms all baselines and sparse attention models, including QDS-Transformer, reaching 0.707 on NDCG@10 and 0.281 on MAP. It is closely followed by KeyB(vBERT) 𝐵𝑀25 which outperforms all baseline and sparse attention models on NDCG@10, reaching 0.678 compared with QDS-Transformer's 0.667. For MAP, KeyB(vBERT) 𝐵𝑀25 is slightly below QDS-Transformer and PARADE, and on a par with TKL.

The KeyB(PARADE5) models are very close to the KeyB(vBERT) models, both in terms of NDCG@10 and MAP. They also outperform baseline and sparse attention models on NDCG@10, and outperform all baseline and sparse attention models but PARADE and QDS-Transformers on MAP (they are on a par with these to models on MAP). One can note however that on this collection the KeyB(PARADE5) models not as effective as on the previous collections NDCG@10. Li et al. [START_REF] Li | PARADE: Passage representation aggregation for document reranking[END_REF] also observed this for the PARADE model and attributed it to the fact that the from an additional pre-training on MS-MARCO and rely on a much simpler procedure to select blocks, obtain results comparable with IDCM: KeyB(vBERT) 𝐵𝑀25 is 0.001 point below IDCM on NDCG@10 and 0.004 above on MAP whereas KeyB(PARADE5) 𝐵𝑀25 is 0.007 point below IDCM on NDCG@10 and 0.007 above on MAP, and KeyB(PARADE5) 𝐵𝑖𝑛𝐵 , KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 have higher MAP results.

Overall, the results obtained on TREC DL 2019 once again prove the effectiveness of the proposed selecting key blocks approaches, which obtain results than baseline and sparse attention models without the need to customize CUDA kernels. Besides, when selecting key blocks with the trained BERT model, which corresponds to the model KeyB(vBERT) 𝐵𝑖𝑛𝐵 , one obtains the near state-of-the-art level performance [START_REF] Craswell | Overview of the trec 2019 deep learning track[END_REF] of 0.707 on this collection for NDCG@10. Lastly, the faster and less memory demanding variant KeyB(vBERT) 𝐵𝑀25 is a close competitor to KeyB(vBERT) 𝐵𝑖𝑛𝐵 and should be preferred if time or memory constraints are important.

CONCLUSION

Benefiting from pre-trained BERT models, the field of information retrieval has seen remarkable progress in neural IR models, as exemplified by the success of Vanilla BERT which has become a strong, yet simple, baseline for neural IR models. To overcome the limitations of BERT-based models regarding long documents, we have proposed to divide documents into blocks and to select only the most important key blocks. This is reminiscent of the way humans assess the relevance of a document for a given query: one first identifies blocks relevant to the query, blocks which are then aggregated to obtain the overall assessment of the document. In order to select blocks, we have investigated two approaches: the first one is straightforward and makes use of standard retrieval functions as TF-IDF or BM25; the second one learns a single BERT model used for both ranking blocks and documents. Both approaches have been shown to improve over standard baselines and previous BERT-based models. We have followed the same approach on another highly competitive neural IR model, namely PARADE, here again with improved results. All in all, selecting blocks is advantageous for the two models studied here, Vanilla BERT and PARADE. We conjecture that this selection is a way to remove passages in documents which are not relevant to the query and which are likely to bring noise when matching queries and documents.

Comparing our different proposals, if the selection strategy based on learned mechanisms performs in average better than the one based on standard IR models with similar GPU memory usage, its ranking latency is not as appealing. We thus recommend in practice to use the TF-IDF and BM25 versions of Vanilla BERT and PARADE if latency constraints are important (among these two variants, one may prefer the BM25 one which is slightly better overall). The choice between Vanilla BERT and PARADE variants, the latter being slightly better than the former on standard IR collections, and slightly worse on the TREC 2019 DL collection, depends on the collection considered.

In the future, we plan on deploying the proposed block selection approach on more complex models, which could be a way to further improve the results obtained in this study. We also plan to investigate alternative negative sampling strategies as well as ways to accelerate the selection process based on learned models.

Fig. 1 .

 1 Fig. 1. Interpolated curve of the density of the number of blocks per document for the different datasets. The vertical line in red denotes the position of the median, whereas the position of the 1st and 3rd quartile are displayed in green. For the sake of readability, only the first 125 blocks are shown here.

Figure 2

 2 Figure2displays the distribution of the average BM25 RSV score on each position with 𝑝 = 10 for all labeled query-document pairs (referred to as full dataset), for very relevant and relevant pairs (referred to as relevant only) and for irrelevant pairs (referred to as irrelevant only). On all collections, the average RSV score decreases when the position increases. However, the average RSV scores are still important in the middle positions and non-negligible in the last positions. Furthermore, if one assumes that all blocks of an irrelevant document are irrelevant6 , then the difference, for a given position, between the average RSV scores of relevant and irrelevant pairs may serve as an additional indicator of whether or not relevance signals are found in that particular position. When there is no difference between the average RSV scores of relevant and irrelevant pairs at a given position, then the only conclusion one can draw is that all blocks from relevant and irrelevant documents behave in the same way with respect to the RSV score, and there is no indication that relevance signals are present at that position. On the contrary, when the average RSV score for relevant pairs is significantly higher than the one for irrelevant pairs at a given position, then there is a clearer indication that there are relevant signals at that position. This is the case for all positions of the four collections.To complement the above analysis, we used a different RSV score, namely the cosine similarity between the semantic representations of blocks and queries obtained with the pre-trained Sentence-BERT [57] model 7 (we simply input each query-block pair to Sentence-BERT which outputs query and block representations on which a cosine similarity is computed). Contrary to the BM25 score which is mainly 'lexical', this score aims to capture

Fig. 2 .

 2 Fig. 2. Average BM25 RSV scores per position of a block in a document using the original query 𝑞.

Fig. 3 .

 3 Fig. 3. Average RSV scores (cosine similarity) per position of a block in a document using the original query 𝑞.

Fig. 4 .

 4 Fig. 4. Difference in RSV scores between relevant and irrelevant documents for the original query 𝑞, the expanded one 𝑞 𝑒𝑥𝑝 and the random expanded one 𝑞 𝑟𝑎𝑛𝑑_𝑒𝑥𝑝 across block positions.

Fig. 5 .Fig. 6 .

 56 Fig. 5. An illustration of the architecture of KeyB (e.g., TF-IDF or BM25).

Fig. 7 .

 7 Fig. 7. An illustration of the architecture of KeyB(vBERT) 𝐵𝑖𝑛𝐵 . Here, the BERT model and linear layer are used to select blocks too. While the neural model being trained with document level annotations, this model would become able to score blocks for a query.

Fig. 8 .

 8 Fig. 8. An illustration of the architecture of KeyB(PARADE𝑘) 𝐵𝑖𝑛𝐵 . Here, the BERT model and linear layer of PARADE are used to select blocks too.

Fig. 9 .

 9 Fig. 9. An illustration of the architecture of KeyB(PARADE𝑘) 𝐵𝑖𝑛𝐵2 . Here, additional BERT model and additional linear layer are used to select blocks.

Fig. 11 .Fig. 12 .

 1112 Fig. 11. The probabilities of top 8 block appearing locations in KeyB(vBERT) 𝐵𝑀25 .

7.0. 1 29 Fig. 13 .

 12913 Fig. 13. An example of top 8 blocks selected by the KeyB(vBERT) 𝐵𝑖𝑛𝐵 model on MQ2007.

Table 1 .

 1 Statistics of the datasets used.

	1,692	784	150	250

Table 2 .

 2 Example query and extensions

	Original query 𝑞	"minimum wage increase"
	Synset ("minimum")	minimal
	Synset ("wage")	earnings, pay, remuneration, salary
	Synset ("increase")	growth, gain, addition
	Expanded query 𝑞 𝑒𝑥𝑝	minimum wage increase minimal earnings pay remuneration salary growth gain addition
	Random expanded query 𝑞 𝑟𝑎𝑛𝑑_𝑒𝑥𝑝	minimum wage increase cadent gravely stuffiness puller complaisant sunlight profusely asterism
	Original query boolean representation 𝑞 𝑏𝑜𝑜𝑙	("minimum" OR "wage" OR "increase")
	𝑠𝑦𝑛 𝑏𝑜𝑜𝑙 Extended query boolean representation 𝑞

Table 3 .

 3 Statistics: number of blocks selected. Using the the eldar package 9 , we then computed, for very relevant and relevant documents, the number of blocks matching 𝑞 𝑏𝑜𝑜𝑙 , those matching either 𝑞 𝑏𝑜𝑜𝑙 or 𝑞

	Dataset	MQ2007	MQ2008	Robust04	GOV2
	# of blocks matching 𝑞 𝑏𝑜𝑜𝑙	914,901	217,035	45,566	399,549
	# of blocks matching 𝑞 𝑏𝑜𝑜𝑙 + 𝑞 𝑏𝑜𝑜𝑙 𝑠𝑦𝑛	994,657 (+8.72%) 228,451 (+5.26%) 55,976 (+22.85%) 458,277 (+14.70%)
	# of blocks matching 𝑞 𝑏𝑜𝑜𝑙 + 𝑞 𝑟𝑎𝑛𝑑 𝑏𝑜𝑜𝑙	917,026 (+0.23%) 217,378 (+0.16%) 45,717 (+0.33%) 400,570 (+0.26%)
	boolean queries.

𝑠𝑦𝑛

𝑏𝑜𝑜𝑙 , and matching either 𝑞 𝑏𝑜𝑜𝑙 or 𝑞 𝑟𝑎𝑛𝑑 𝑏𝑜𝑜𝑙 . The results are given in Table

  𝐵𝐷𝑅 0.5166 𝐵𝐷𝑉 𝑅 0.4941 𝐵𝐷𝑉 𝑅 0.4687 𝐵𝐷𝑉 𝐶𝑅 KeyB(vBERT)𝐵𝑖𝑛𝐵 0.6710 𝐵𝐷𝑅 0.5661 𝐵𝐷𝑅 0.5088 𝐵𝐷𝑉 𝐶𝑅 0.4241 𝐵𝐷𝑉 𝑅 0.2722 𝐵𝐷𝑉 𝐶𝑅 0.6289 𝐵𝐷𝑅 0.5554 𝐵𝐷𝑅 0.5249 𝐵𝐷𝑉 𝑅 0.4958 𝐵𝐷𝑉 𝑅 0.4768 𝐵𝐷𝑉 𝐶𝑅

	Baseline models
	BM25	0.5542	0.5004	0.4382	0.3631	0.2334	0.5080	0.4741	0.4485	0.4240	0.4402
	DeepRank	0.5663	0.4538	0.3907	0.3331	0.2145	0.5081	0.4386	0.4051	0.3864	0.4272
	BERT based models
	Vanilla BERT	0.6067	0.5478	0.4843	0.4088	0.2510	0.5706	0.5337	0.4945	0.4678	0.4553
	CEDR-KNRM	0.6220	0.5542	0.4840	0.4097	0.2440	0.5878	0.5253	0.5093	0.4803	0.4600
	Random Select	0.5983	0.5108	0.4730	0.4059	0.2453	0.5482	0.4856	0.4880	0.4688	0.4540
	KeyB(vBERT)𝑇 𝐹 -𝐼 𝐷𝐹	0.6146	0.5430 𝐵𝐷𝑅 0.4963 𝐵𝐷𝑅	0.4208 𝐵𝐷𝑅	0.2628 𝐵𝐷𝑉 𝐶𝑅 0.5764	0.5275 𝐵𝐷𝑅 0.5099 𝐵𝐷	0.4884 𝐵𝐷𝑉 𝑅 0.4684 𝐵𝐷𝑉 𝐶𝑅
	KeyB(vBERT)𝐵𝑀25 0.5512 PARADE based models 0.6468 𝐵𝐷 0.5622 𝐵𝐷𝑅 0.4976 𝐵𝐷𝑅 0.4241 𝐵𝐷𝑉 𝑅 0.2609 𝐵𝐷𝐶𝑅 0.6004 𝐵𝐷
	PARADE	0.6869	0.5686	0.5080	0.4309	0.2739	0.6166	0.5510	0.5146	0.5017	0.4737
	PARADE5	0.6388	0.5334	0.4868	0.4125	0.2477	0.5905	0.5215	0.5055	0.4761	0.4594
	KeyB(PARADE5)𝑇 𝐹 -𝐼 𝐷𝐹 0.6790 𝐵𝐷	0.5687 𝐵𝐷5 0.5093 𝐵𝐷5	0.4319 𝐵𝐷5	0.2714 𝐵𝐷5	0.6348 𝐵𝐷	0.5467 𝐵𝐷	0.5218 𝐵𝐷	0.4989 𝐵𝐷5	0.4710 𝐵𝐷5
	KeyB(PARADE5)𝐵𝑀25	0.6871 𝐵𝐷	0.5768 𝐵𝐷5 0.5177 𝐵𝐷5	0.4337 𝐵𝐷5	0.2757 𝐵𝐷5	0.6329 𝐵𝐷	0.5636 𝐵𝐷5 0.5304 𝐵𝐷5	0.5040 𝐵𝐷5	0.4735 𝐵𝐷5
	KeyB(PARADE5)𝐵𝑖𝑛𝐵	0.6308 𝐵	0.5479 𝐵𝐷	0.5057 𝐵𝐷	0.4200 𝐵𝐷	0.2629 𝐵𝐷𝑃 5	0.5885 𝐵𝐷	0.5329 𝐵𝐷	0.5283 𝐵𝐷5	0.4967 𝐵𝐷5	0.4687 𝐵𝐷5
	KeyB(PARADE5)𝐵𝑖𝑛𝐵2	0.6427 𝐵𝐷	0.5758 𝐵𝐷5 0.5112 𝐵𝐷5	0.4378 𝐵𝐷5	0.2779 𝐵𝐷5	0.5965 𝐵𝐷5 0.5625 𝐵𝐷5 0.5247 𝐵𝐷	0.5058 𝐵𝐷5	0.4778 𝐵𝐷5

Table 5 .

 5 Results on GOV2 dataset. Best results are in bold. Best results are in bold. For KeyB(vBERT) models, a significant difference with BM25 is marked with a 'B', with DeepRank with a 'D', with Vanilla BERT with a 'V', with CEDR-KNRM with a 'C', and with Random select with an 'R'. For KeyB(PARADE) models, a significant difference with BM25 is marked with a 'B', with DeepRank with a 'D', with PARADE with a 'P', and with PARADE5 with a '5'. A paired t-test (𝑝 -𝑣𝑎𝑙𝑢𝑒 ≤ 0.05) is used for measuring significance. 𝐵𝐷𝑉 𝐶 0.6937 𝐵𝐷𝑉 𝐶𝑅 0.6645 𝐵𝐷𝑉 𝐶𝑅 0.6125 𝐵𝐷𝑉 𝐶𝑅 0.2674 𝐵𝐷𝑉 𝐶𝑅 0.5574 𝐷 0.5414 𝐵𝐷𝑉 𝐶𝑅 0.5356 𝐵𝐷𝑉 𝐶𝑅 0.5295 𝐵𝐷𝑉 𝐶𝑅 0.4453 𝐵𝐷𝑉 𝐶𝑅

	Model	P@1	P@5	P@10	P@20	MAP	NDCG@1 NDCG@5	NDCG@10	NDCG@20	NDCG
	Baseline models
	BM25	0.6510	0.6054	0.5792	0.5362	0.2331	0.5034	0.4904	0.4867	0.4774	0.4296
	DeepRank	0.6453	0.5682	0.5143	0.4880	0.2151	0.4738	0.4363	0.4194	0.4170	0.4120
	BERT based models
	Vanilla BERT	0.6241	0.6068	0.5672	0.5475	0.2321	0.4531	0.4954	0.4837	0.4764	0.4279
	CEDR-KNRM	0.6239	0.6133	0.5886	0.5556	0.2375	0.4929	0.4891	0.4892	0.4769	0.4315
	Random Select	0.6839	0.6169	0.5984	0.5640	0.2467	0.4995	0.4811	0.4955	0.4853	0.4358
	KeyB(vBERT)𝑇 𝐹 -𝐼 𝐷𝐹	0.7122	0.6735 𝐵𝐷𝑉 𝐶𝑅 0.6446 𝐵𝐷𝑉 𝐶𝑅 0.6123 𝐵𝐷𝑉 𝐶𝑅 0.2583 𝐵𝐷𝑉 𝐶𝑅 0.5574 𝐷	0.5256 𝐷𝑅	0.5340 𝐵𝐷𝑉 𝐶𝑅 0.5269 𝐵𝐷𝐶𝑅	0.4413 𝐵𝐷𝐶
	KeyB(vBERT)𝐵𝑀25	0.6634	0.6524 𝐷	0.6303 𝐵𝐷𝐶	0.5997 𝐵𝐷𝑉 𝐶𝑅 0.2643 𝐵𝐷𝑉 𝐶𝑅 0.5171	0.5341 𝐷𝑉 𝑅	0.5272 𝐵𝐷𝑉 𝐶	0.5199 𝐵𝐷𝐶𝑅	0.4447 𝐵𝐷𝑉 𝐶𝑅
	KeyB(vBERT)𝐵𝑖𝑛𝐵 0.7651 PARADE based models
	PARADE	0.7244	0.7016	0.6631	0.6133	0.2621	0.5930	0.5518	0.5562	0.5466	0.4484
	PARADE5	0.6906	0.6429	0.6246	0.5707	0.2462	0.5463	0.5327	0.5161	0.5053	0.4386
	KeyB(PARADE5)𝑇 𝐹 -𝐼 𝐷𝐹 0.7386	0.6931 𝐵𝐷5	0.6605 𝐵𝐷5	0.6222 𝐵𝐷5	0.2728 𝐵𝐷𝑃 5	0.5569 𝐷	0.5536 𝐵𝐷	0.5498 𝐵𝐷	0.5352 𝐵𝐷5	0.4537 𝐵𝐷5
	KeyB(PARADE5)𝐵𝑀25	0.7720 𝐵𝐷	0.6931 𝐵𝐷5	0.6528 𝐵𝐷	0.6397 𝐵𝐷𝑃 5	0.2745 𝐵𝐷𝑃5	0.5806 𝐷	0.5783 𝐵𝐷5	0.5529 𝐵𝐷5	0.5624 𝐵𝐷5	0.4578 𝐵𝐷𝑃5
	KeyB(PARADE5)𝐵𝑖𝑛𝐵	0.7055	0.7196 𝐵𝐷5	0.6563 𝐵𝐷	0.6212 𝐵𝐷5	0.2680 𝐵𝐷5	0.5640 𝐷	0.5660 𝐵𝐷	0.5451 𝐵𝐷	0.5379 𝐵𝐷	0.4495 𝐵𝐷
	KeyB(PARADE5)𝐵𝑖𝑛𝐵2	0.7253	0.7034 𝐵𝐷5	0.6771 𝐵𝐷5	0.6407 𝐵𝐷𝑃 5	0.2733 𝐵𝐷𝑃5	0.5706 𝐷	0.5512 𝐵𝐷	0.5676 𝐵𝐷5	0.5591 𝐵𝐷5	0.4554 𝐵𝐷5

  𝐵𝐷𝑉 𝐶𝑅 0.4465 𝐵𝐷𝑉 𝐶𝑅 0.3702 𝐵𝐷𝑉 𝐶𝑅 0.5323 𝐵𝐷𝑉 𝐶𝑅 0.4917 𝐵𝐷 0.5043 𝐵𝐷𝑉 𝐶 0.5342 𝐵𝐷𝑉 𝐶𝑅 0.5864 𝐵𝐷𝑉 𝐶𝑅 0.6551 𝐵𝐷𝑉 𝐶 KeyB(vBERT)𝐵𝑀25 0.5526 𝐵𝐷𝑉 𝐶 0.4946 𝐵𝐷𝑉 𝐶𝑅 0.4408 𝐵𝐷𝑉 𝐶 0.3705 𝐵𝐷𝑉 𝐶𝑅 0.5305 𝐵𝐷𝑉 𝐶𝑅 0.4933 𝐵𝐷𝑉 0.5061 𝐵𝐷𝑉 𝐶 0.5339 𝐵𝐷𝑉 𝐶𝑅 0.5824 𝐵𝐷𝑉 𝐶 0.6528 𝐵𝐷𝑉 𝐶 KeyB(vBERT)𝐵𝑖𝑛𝐵 0.5597 𝐵𝐷𝑉 𝐶𝑅 0.4971 𝐵𝐷𝑉 𝐶𝑅 0.4503 𝐵𝐷𝑉 𝐶𝑅 0.3759 𝐵𝐷𝑉 𝐶𝑅 0.5457 𝐵𝐷𝑉 𝐶𝑅 0.5133 𝐵𝐷𝑉 𝐶𝑅 0.5134 𝐵𝐷𝑉 𝐶𝑅 0.5496 𝐵𝐷𝑉 𝐶𝑅 0.5969 𝐵𝐷𝑉 𝐶𝑅 0.6627 𝐵𝐷𝑉 𝐶𝑅

											NDCG
	Baseline models
	BM25	0.4186	0.3969	0.3757	0.3391	0.4527	0.3712	0.3954	0.4309	0.4962	0.5933
	DeepRank	0.4444	0.4201	0.3898	0.3473	0.4596	0.3942	0.4168	0.4468	0.5088	0.6012
	DeepRank*	0.508	0.452	0.412	-	0.497	0.441	0.457	0.482	-	-
	BERT based models
	Vanilla BERT	0.5266	0.4741	0.4257	0.3606	0.5073	0.4708	0.4808	0.5070	0.5620	0.6379
	CEDR-KNRM	0.5284	0.4768	0.4233	0.3601	0.5066	0.4814	0.4874	0.5084	0.5601	0.6380
	Random Select	0.5343	0.4768	0.4347	0.3656	0.5207	0.4808	0.4980	0.5224	0.5775	0.6499
	KeyB(vBERT)𝑇 𝐹 -𝐼 𝐷𝐹 0.4926 PARADE based models 0.5425 𝐵𝐷
	PARADE	0.5474	0.5009	0.4486	0.3747	0.5418	0.5054	0.5255	0.5499	0.5950	0.6599
	PARADE5	0.5686	0.4824	0.4370	0.3714	0.5291	0.5174	0.5142	0.5356	0.5851	0.6538
	KeyB(PARADE5)𝑇 𝐹 -𝐼 𝐷𝐹 0.5721 𝐵𝐷𝑃	0.5034 𝐵𝐷5	0.4491 𝐵𝐷5	0.3737 𝐵𝐷	0.5477 𝐵𝐷5	0.5198 𝐵𝐷	0.5221 𝐵𝐷	0.5488 𝐵𝐷5	0.5998 𝐵𝐷5	0.6645 𝐵𝐷5
	KeyB(PARADE5)𝐵𝑀25	0.5769 𝐵𝐷𝑃	0.5063 𝐵𝐷5	0.4486 𝐵𝐷5	0.3748 𝐵𝐷5	0.5494 𝐵𝐷𝑃 5	0.5151 𝐵𝐷	0.5261 𝐵𝐷5	0.5530 𝐵𝐷5	0.6021 𝐵𝐷5	0.6664 𝐵𝐷𝑃 5
	KeyB(PARADE5)𝐵𝑖𝑛𝐵	0.5580 𝐵𝐷	0.5079 𝐵𝐷5	0.4487 𝐵𝐷5	0.3740 𝐵𝐷	0.5427 𝐵𝐷5	0.5054 𝐵𝐷	0.5200 𝐵𝐷	0.5493 𝐵𝐷5	0.5978 𝐵𝐷5	0.6623 𝐵𝐷5
	KeyB(PARADE5)𝐵𝑖𝑛𝐵2	0.5709 𝐵𝐷𝑃	0.5066 𝐵𝐷5	0.4488 𝐵𝐷5	0.3766 𝐵𝐷5	0.5461 𝐵𝐷5	0.5213 𝐵𝐷	0.5266 𝐵𝐷5	0.5513 𝐵𝐷5	0.6005 𝐵𝐷5	0.6650 𝐵𝐷5

Table 7

 7 𝐵𝐷𝑉 𝐶𝑅 0.2624 𝐷𝑉 𝐶𝑅 0.1589 𝐷 0.5425 𝐵𝐷𝑉 𝐶𝑅 0.4661 𝐵𝐷𝑅 0.5382 𝐵𝐷𝑉 𝐶𝑅 0.5616 𝐵𝐷𝑉 𝐶𝑅 0.5788 𝐵𝐷𝐶𝑅 0.5891 𝐵𝐷𝑅

	Model	P@1	P@5	P@10	P@20	MAP	NDCG@1	NDCG@5	NDCG@10	NDCG@20 NDCG
	Baseline models
	BM25	0.3816	0.3316	0.2411	0.1515	0.4538	0.3297	0.4376	0.4841	0.5086	0.5243
	DeepRank	0.3992	0.2816	0.1920	0.1150	0.4356	0.3641	0.4373	0.4672	0.4878	0.4917
	DeepRank*	0.482	0.359	0.252	-	0.498	0.406	0.496	-	-	-
	BERT based models
	Vanilla BERT	0.5063	0.3650	0.2560	0.1566	0.5230	0.4508	0.5165	0.5489	0.5697	0.5810
	CEDR-KNRM	0.5050	0.3678	0.2561	0.1569	0.5220	0.4515	0.5151	0.5488	0.5674	0.5794
	Random Select	0.5000	0.3663	0.2574	0.1579	0.5196	0.4387	0.5096	0.5427	0.5611	0.5729
	KeyB(vBERT)𝑇 𝐹 -𝐼 𝐷𝐹	0.5166 𝐵𝐷	0.3862 𝐵𝐷𝑉 𝐶𝑅 0.2597 𝐷	0.1580 𝐷	0.5318 𝐵𝐷	0.4649 𝐵𝐷	0.5330 𝐵𝐷𝑉 𝐶𝑅 0.5596 𝐵𝐷𝑅	0.5755 𝐵𝐷𝑅	0.5869 𝐵𝐷𝑅
	KeyB(vBERT)𝐵𝑀25	0.5165 𝐵𝐷	0.3760 𝐵𝐷𝑉 𝑅	0.2579 𝐷	0.1582 𝐷	0.5350 𝐵𝐷𝑅	0.4629 𝐵𝐷	0.5317 𝐵𝐷𝑉 𝐶𝑅 0.5609 𝐵𝐷𝑉 𝐶𝑅 0.5788 𝐵𝐷𝐶𝑅 0.5891 𝐵𝐷𝑅
	KeyB(vBERT)𝐵𝑖𝑛𝐵 0.3819 PARADE based models 0.5254 𝐵𝐷
	PARADE	0.5089	0.3811	0.2617	0.1590	0.5375	0.4502	0.5321	0.5656	0.5799	0.5867
	PARADE5	0.4999	0.3763	0.2578	0.1574	0.5254	0.4514	0.5226	0.5523	0.5716	0.5821
	KeyB(PARADE5)𝑇 𝐹 -𝐼 𝐷𝐹 0.5369 𝐵𝐷𝑃 5 0.3829 𝐵𝐷	0.2621 𝐷5	0.1585 𝐷	0.5436 𝐵𝐷5	0.4859 𝐵𝐷𝑃 5 0.5390 𝐵𝐷5	0.5728 𝐵𝐷5	0.5851 𝐵𝐷5	0.5942 𝐵𝐷5
	KeyB(PARADE5)𝐵𝑀25	0.5267 𝐵𝐷5	0.3831 𝐵𝐷	0.2635 𝐷5	0.1592 𝐷5 0.5409 𝐵𝐷5	0.4744 𝐵𝐷𝑃	0.5373 𝐵𝐷5	0.5646 𝐵𝐷5	0.5812 𝐵𝐷	0.5907 𝐵𝐷5
	KeyB(PARADE5)𝐵𝑖𝑛𝐵	0.5229 𝐵𝐷	0.3888 𝐵𝐷𝑃 5	0.2634 𝐷5	0.1592 𝐷 0.5428 𝐵𝐷5	0.4668 𝐵𝐷	0.5354 𝐵𝐷5	0.5702 𝐵𝐷5	0.5835 𝐵𝐷5	0.5914 𝐵𝐷
	KeyB(PARADE5)𝐵𝑖𝑛𝐵2	0.5242 𝐵𝐷	0.3847 𝐵𝐷5	0.2639 𝐷5	0.1583 𝐷	0.5431 𝐵𝐷5	0.4789 𝐵𝐷𝑃5 0.5373 𝐵𝐷5	0.5689 𝐵𝐷5	0.5837 𝐵𝐷5	0.5952 𝐵𝐷5

. Results on MQ2008 dataset. DeepRank* represents the results from the original paper. Best results are in bold. For KeyB(vBERT) models, a significant difference with BM25 is marked with a 'B', with DeepRank with a 'D', with Vanilla BERT with a 'V', with CEDR-KNRM with a 'C', and with Random select with an 'R'. For KeyB(PARADE) models, a significant difference with BM25 is marked with a 'B', with DeepRank with a 'D', with PARADE with a 'P', and with PARADE5 with a '5'. A paired t-test (𝑝 -𝑣𝑎𝑙𝑢𝑒 ≤ 0.05) is used for measuring significance.

Table 9 .

 9 Ranking latencies (seconds) on MQ2007 test set for 100 queries each with 40 documents.

	Model	Latency Seconds/query Milliseconds/doc
	Vanilla BERT	6.962	0.070	1.741
	KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹	11.598	0.116	2.900
	KeyB(vBERT) 𝐵𝑀25	13.742	0.137	3.436
	KeyB(vBERT) 𝐵𝑖𝑛𝐵	211.648	2.12	52.912
	PARADE	27.909	0.279	6.977
	PARADE5	18.701	0.187	4.675
	KeyB(PARADE5) 𝑇 𝐹 -𝐼 𝐷𝐹 26.894	0.269	6.724
	KeyB(PARADE5) 𝐵𝑀25	24.320	0.243	6.080
	KeyB(PARADE5) 𝐵𝑖𝑛𝐵	242.309	2.423	60.577
	KeyB(PARADE5) 𝐵𝑖𝑛𝐵2	336.789	3.368	84.197

Table 10 .

 10 Statistics of the TREC 2019 DL document ranking task.

	Collection # Documents # Train queries # Train qrels # Dev queries # Dev qrels # Test queries # Test qrels
	TREC19 DL	3,213,835	367,013	384,597	5,193	5,478	43	16,258

ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x.The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval • 3

ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x.

MS-MARCO is a passage retrieval collection available at: https://microsoft.github.io/msmarco/ ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January

202x.

http://ir.dcs.gla.ac.uk/test_collections/gov2-summary.htm

https://trec.nist.gov/data/robust/04.guidelines.html ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x. The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval • 7

The maximum number of blocks for MQ2007, MQ2008, Robust04 and GOV2 respectively is 3225, 3225, 2959, 3311. We do not display the entire distribution for reading purposes.

The average number of tokens per block for MQ2007, MQ2008, Robust04 and GOV2 respectively is 54.39, 54.44, 53.16, 54.33. ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x.

We believe this is a reasonable assumption, at least when the blocks are not too short.

The all-MiniLM-L12-v2 version from https://www.sbert.net/docs/pretrained_models.html ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x. The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval • 9

Adding terms to a boolean query through a disjunction is likely to increase the number of blocks retrieved by the query. This said, please bear in mind that here the words added to the original query come from an external resource and may not be present in the collections queried.ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x. The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval • 11

https://github.com/kerighan/eldar ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x.

These models are developed on top of the Georgetown IR Lab implementation and are available at: https://github.com/lmh0921/keyB. ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x.

https://github.com/pl8787/DeepRank_PyTorch

https://github.com/capreolus-ir/capreolus

https://github.com/Georgetown-IR-Lab/cedr

https://github.com/huggingface/transformers ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x.

https://github.com/google-research/bert

As done https://github.com/canjiali/PARADE/blob/master/generate_data.py in line 304. ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x.

ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x. The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval • 23

https://github.com/sebastian-hofstaetter/intra-document-cascade

ACKNOWLEDGMENTS

This work has been partially supported by MIAI@Grenoble Alpes (ANR-19-P3IA-0003) and the Chinese Scholarship Council (CSC) grant No.201906960018.

We are interested here in assessing whether it is important to accurately select blocks or not. For this, we compare the results obtained by the different KeyB(vBERT) models with the ones obtained by the Random Select strategy which amounts to randomly selecting blocks. As one can also note, all KeyB(vBERT) models outperform the Random Select strategy on all collections, for all metrics. Furthermore, the best KeyB(vBERT) model is significantly better than Random Select on 10 metrics out of 10 on Robust04, on 8 metrics out of 10 on GOV2, on 10 metrics out of 10 on MQ2007, and on 8 Metrics out of 10 on MQ2008.

The above analysis shows that the KeyB(vBERT) models should be preferred over all baseline and standard BERT-based IR models. We now turn to the comparison of KeyB(vBERT) models.

RQ4 What are the differences between the different KeyB(vBERT) models? On Robust04, KeyB(vBERT) 𝐵𝑖𝑛𝐵 is the best model on 10 metrics and KeyB(vBERT) 𝐵𝑀25 on 1 metric. We further conduct significant tests between KeyB(vBERT) models. KeyB(vBERT) 𝐵𝑖𝑛𝐵 is significantly better than KeyB(vBERT) 𝐵𝑀25 on MAP and NDCG while shows no significant difference than KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 . On GOV2, KeyB(vBERT) 𝐵𝑖𝑛𝐵 is the best model on 10 metrics and KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 on 1 metric. KeyB(vBERT) 𝐵𝑖𝑛𝐵 is significantly better than KeyB(vBERT) 𝐵𝑀25 on p@1 and p@10 while shows no significant difference than KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 . On MQ2007, KeyB(vBERT) 𝐵𝑖𝑛𝐵 is the best model over all metrics and significantly outperforms KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 on MAP, P@20, NDCG, NDCG@1, NDCG@10 and NDCG@20, significantly outperforms KeyB(vBERT) 𝐵𝑀25 on MAP, P@10, P@20, NDCG, NDCG@10 and NDCG@20. On MQ2008, KeyB(vBERT) 𝐵𝑖𝑛𝐵 is the best model on all metrics but P@5, KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 on 1 metric and KeyB(vBERT) 𝐵𝑀25 on 2 metrics. The difference between all three models is however not really significant as KeyB(vBERT) 𝐵𝑖𝑛𝐵 significantly outperforms KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 on only MAP and KeyB(vBERT) 𝐵𝑀25 on P@10.

From this analysis, one can see that the model KeyB(vBERT) 𝐵𝑖𝑛𝐵 is either significantly better or on a par with KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 and KeyB(vBERT) 𝐵𝑀25 . This justifies the use of a learning mechanism to select blocks. This said, even a simple approach to select blocks as the one implemented in KeyB(vBERT) 𝐵𝑀25 can yield good results on collections such as Robust04 and MQ2008. We now turn to the PARADE models. 6.2.2 Improving PARADE with selected passages. As mentioned before, PARADE is the original PARADE model with 16 passages corresponding to the first and last passages, and 14 randomly selected passages in between, PARADE5 is another variant with only 5 passages corresponding to the first and last passages, and 3 randomly selected passages in between, and KeyB(PARADE5) models are the PARADE models with only 5 passages selected with BM25, TF-IDF or learning based approaches. We propose to analyze the experimental results by answering several research questions.

RQ5 How effective are KeyB(PARADE5) models compared to baseline models (BM25, DeepRank)? From Tables 4 to 7, one can see that KeyB(PARADE5) models outperform both baselines on all collections, the difference being significant for all metrics and all on collections but P@1 on GOV2.

RQ6 How effective are KeyB(PARADE5) models compared to PARADE and PARADE5? As one can note, on all collections, KeyB(PARADE5) 𝐵𝑀25 , KeyB(PARADE5) 𝐵𝑀25 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 obtain better average results or on a par with PARADE. For example, comparing with the original PARADE model, on Robust04, KeyB(PARADE5) 𝐵𝑀25 outperforms PARADE on 9 metrics out of 10, even though the difference is never significant. On GOV2, KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 outperforms PARADE on 8 metrics out of 10 and is significantly better on 2 metrics. On MQ2007, KeyB(PARADE5) 𝐵𝑀25 outperforms PARADE on 9 metrics out of 10 and is significantly better on 3 metrics. On MQ2008, KeyB(PARADE5) 𝑇 𝐹 -𝐼 𝐷𝐹 outperforms PARADE on 9 metrics out of 10 and is significantly better on 2 metrics.

Comparing with PARADE5, KeyB(PARADE5) 𝐵𝑀25 , KeyB(PARADE5) 𝐵𝑀25 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 obtain better average results on all collections and metrics, except KeyB(PARADE5) 𝐵𝑀25 on NDCG@1 on MQ2007 (0.5151 vs 0.5174). More precisely, on Robust04, KeyB(PARADE5) 𝐵𝑀25 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 are significantly better than ACM Trans. Inf. Syst., Vol. To appear, No. To appear, Article . Publication date: January 202x.

The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval • 25

PARADE5 on 8 metrics. On GOV2, KeyB(PARADE5) 𝐵𝑀25 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 are significantly better than PARADE5 on 7 metrics. On MQ2007, KeyB(PARADE5) 𝑇 𝐹 -𝐼 𝐷𝐹 and KeyB(PARADE5) 𝐵𝑀25 are significantly better than PARADE5 on 6 and 8 metrics respectively. On MQ2008, KeyB(PARADE5) 𝑇 𝐹 -𝐼 𝐷𝐹 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 are significantly better than PARADE5 on 8 metrics.

The model KeyB(PARADE5) 𝐵𝑖𝑛𝐵 , which reuses the BERT and feed-forward neural networks in PARADE for selecting passages, is however sometimes less effective than the other KeyB(PARADE5) models. On Robust04, it is below the other three KeyB(PARADE5) models as well as below PARADE. On GOV2, KeyB(PARADE5) 𝐵𝑖𝑛𝐵 is higher than PARADE on 5 metrics and lower on 5 metrics. On MQ2007 and MQ2007, KeyB(PARADE5) 𝐵𝑖𝑛𝐵 is mostly better than PARADE. Comparing with PARADE5, KeyB(PARADE5) 𝐵𝑖𝑛𝐵 obtains almost always better results, especially on MQ2007 and MQ2008, being significantly better on 6 and 5 metrics respectively. This shows that, despite its mitigated results on some metrics and collections, KeyB(PARADE5) 𝐵𝑖𝑛𝐵 is still a powerful approach that obtains several best results on different metrics.

RQ7 What are the differences between the different KeyB(PARADE5) models?

As one can note, the best results on each metric is somehow distributed on the different KeyB(PARADE5) models. On Robust04, KeyB(PARADE5) 𝐵𝑀25 obtains 5 best results, KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 4 and KeyB(PARADE5) 𝑇 𝐹 -𝐼 𝐷𝐹 1. On GOV2, KeyB(PARADE5) 𝐵𝑀25 obtains 5 best results, KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 3 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵 1. On MQ2007, KeyB(PARADE5) 𝐵𝑀25 obtains 5 best results, KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 3 and the other two models 1 each. On MQ2008, KeyB(PARADE5) 𝑇 𝐹 -𝐼 𝐷𝐹 obtains 6 best results, KeyB(PARADE5) 𝐵𝑀25 1 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵 2 each. Besides, as discussed above, although KeyB(PARADE5) 𝐵𝑖𝑛𝐵 does not perform well on Robust04, it is still competitive with PARADE5 on this collection and performs well on the other collections. KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 tends however to be more stable across the collections and metrics.

Overall, the PARADE variants we have introduced in general significantly outperform the PARADE5 model and are either on a par or significantly outperform the original PARADE model. This is all the more remarkable that these models use three times less passages than the original PARADE model and require less memory while being faster, as illustrated below. Lastly, the best KeyB(PARADE5) model tends to be slightly better than the best KeyB(vBERT) model, on all collections and almost all metrics, even though the difference is in general small. Their latency is however not the same (see below).

Memory usage

The memory usage of all models are similar across datasets. We thus only report here the memory usage of different models on MQ2007 as this dataset contains more queries and requires longer training than Robust04 and GOV2. The memory usage corresponds to the GPU consumption for training a given model. We remind the reader that a training batch contains two pairs consisting of four queries and four documents. The results obtained are shown in Figure 10 on two official LETOR metrics [START_REF] Qin | Introducing LETOR 4.0 datasets[END_REF].

The best models in terms of accuracy (as measured by either P@10 or NDCG@10) and memory usage are located in the top left corner: they use less memory and achieve higher results. As one can note, KeyB(vBERT) 𝐵𝑖𝑛𝐵 and KeyB(PARADE5) models are located in this area. They need less GPU memory while achieving similar or higher results than PARADE on the two metrics. Furthermore, KeyB(PARADE5) 𝑇 𝐹 -𝐼 𝐷𝐹 , KeyB(PARADE5) 𝐵𝑀25 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵 uses the same amount of memory as PARADE5 but are better on both metrics. KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 uses slightly more memory than PARADE5 but is also better on both metrics.

Ranking speed

We measure here the speed of ranking of the different models on two sets of queries: all queries from one test fold of Robust04, each with 200 documents, and a randomly selected subset of 100 queries from MQ2007, again from one test fold, each with 40 documents. Note that the documents in MQ2007 are on average longer than the (a) P@10 performance (b) NDCG@10 performance Fig. 10. GPU memory usage and effectiveness comparisons, automatic mixed precision is used for all models which would reduce memory usage. Top left models show better performance. ones in Robust04. Latency results, as well as the average time for processing a query (in seconds) and a document (in milliseconds) on a RTX 6000 GPU are reported in Table 8 for Robust04 and Table 9 for MQ2007. The passage splitting time is not counted as this step can be performed offline.

As one can see on both tables, the three fastest models are Vanilla BERT, KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 and KeyB(vBERT) 𝐵𝑀25 , the latter two being only slightly slower than the former one. Furthermore, on both collections, the KeyB(PARADE5) 𝑇 𝐹 -𝐼 𝐷𝐹 and KeyB(PARADE5) 𝐵𝑀25 are faster than PARADE. They are also faster than PARADE5 on Robust04 and only slightly slower than PARADE5 on MQ2007. These two variants, TF-IDF and BM25, because of their performance, their memory usage and their speed, represent strong alternatives to the original Vanilla BERT and PARADE models.

Regarding the models based on learning the selection block method, if their performance is higher than the one of other models, their latency is also higher: KeyB(vBERT) 𝐵𝑖𝑛𝐵 , KeyB(PARADE5) 𝐵𝑖𝑛𝐵 and KeyB(PARADE5) 𝐵𝑖𝑛𝐵2 are, at best, 10 times slower than the KeyB(vBERT) 𝑇 𝐹 -𝐼 𝐷𝐹 and KeyB(vBERT) 𝐵𝑀25 models on both collections. Their current latency may prevent their use in a commercial system. This said, there are several paths that one can follow to make them faster, including a two-stage approach at the block level, using a fast model as BM25 for filtering out less relevant blocks and using the more complex models on the remaining blocks. [START_REF] Hui | PACRR: A Position-Aware Neural IR Model for Relevance Matching[END_REF] 0.550 0.231 TK [START_REF] Hofstätter | Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking[END_REF] 0.594 0.252 TKL [START_REF] Hofstätter | Local self-attention over long text for efficient document retrieval[END_REF] 0.644 0.277 RoBERTa (FirstP) [START_REF] Dai | Deeper text understanding for IR with contextual neural language modeling[END_REF][START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF] We see here however the advantage of the KeyB(PARADE5) models which rely on fewer d-passages, more likely to be relevant to the query.

Comparison with IDCM.

As mentioned in Section 2, IDCM [START_REF] Hofstätter | Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking[END_REF] is a recently proposed model that also learns how to select blocks. The motivation behind this model is to obtain an IR model more efficient as it would only rely on a few blocks. Our motivation slightly differs as we aim to improve the overall IR system by filtering out non relevant, likely noisy blocks. Furthermore, our approach can be directly used with different IR models by selecting blocks with standard IR systems. This is the basis of the models KeyB(vBERT) 𝐵𝑀25 and KeyB(PARADE5) 𝐵𝑀25 for example. Table 11 shows a comparison of our approaches with IDCM (last four lines) where for each query the official top 100 documents are used (this setting is used for all models reported in Table 11). For IDCM, we have used the authors' notebook 21 . IDCM reaches 0.679 for NDCG@10, which is higher than all baseline and sparse attention models, and 0.273 for MAP, which is higher than all baseline and sparse attention models but TKL, PARADE and QDS-Transformer. In contrast, KeyB(vBERT) 𝐵𝑖𝑛𝐵 outperforms all models, including IDCM, on both evaluation metrics. In addition, both KeyB(vBERT) 𝐵𝑀25 and KeyB(PARADE5) variants, even though they did not benefit