Cornelia Caragea

Zhao Kang

Josiane Mothe
email: josiane.mothe@irit.fr

Analytics Methods to Understand Information Retrieval Effectiveness-A Survey

Keywords: information systems, information retrieval, system effectiveness, search engine, IR system analysis, data analytics, query processing chain, 68T20, 94A16

Information retrieval aims to retrieve the documents that answer users' queries. A typical search process consists of different phases for which a variety of components have been defined in the literature; each one having a set of hyper-parameters to tune. Different studies focused on how and how much the components and their hyper-parameters affect the system performance in terms of effectiveness, others on the query factor. The aim of these studies is to better understand information retrieval system effectiveness. This paper reviews the literature of this domain. It depicts how data analytics has been used in IR to gain a better understanding of system effectiveness. This review concludes that we lack a full understanding of system effectiveness related to the context which the system is in, though it has been possible to adapt the query processing to some contexts successfully. This review also concludes that, even if it is possible to distinguish effective from non-effective systems for a query set, neither the system component analysis nor the query features analysis were successful in explaining when and why a particular system fails on a particular query.

Introduction

Information retrieval (IR) aims to retrieve the documents that answer users' queries. It is a core functionality of many digital systems, such as web search engines and e-commerce recommender systems. The effectiveness of an IR system is not the same on all the queries it processes, this is the query factor of effectiveness variability.

In IR, documents are indexed to build document representations that will be used for the online query-document matching. The online process used to answer a user's query consists of different phases and aims to choose the documents to deliver to the user as well as their order. A typical online process consists of the following phases: query pre-processing, optional automatic query reformulation or expansion, search for documents matching the query, and retrieved document ordering (see Figure 1).

A variety of components have been defined in the literature for each phase. For example, the searching/matching, also called weighting model (it is called this because in this component each document will receive a score with regard to a given query), can be achieved by the Salton's Vector Space Model [START_REF] Salton | A vector space model for automatic indexing[END_REF], the Roberston and Spark Jones' BM25 probabilistic model [START_REF] Robertson | Relevance weighting of search terms[END_REF][START_REF] Robertson | The Probabilistic Relevance Framework: BM25 and Beyond[END_REF], the Ponte and Croft' Language Modelling [START_REF] Ponte | A Language Modeling Approach to Information Retrieval[END_REF], and others.

Defining an information search process chain implies we decide which component will be used in each phase. In addition to the wide choice of possible components, each one has a set of hyper-parameters that need to be set. For example, the number of terms to add to the query in the automatic query expansion phase is one of the parameters to be decided. A query processing chain A will not result in the same retrieved documents, and, thus, not the same effectiveness, than a query processing chain B, even when considering the same query or queries, the same collection of documents, and the same effectiveness measure. This variability in effectiveness due to the system is also named as the system factor of variability in effectiveness. Common practice in information retrieval is to tune the search process components' parameters once for all the forthcoming queries.

For example, Terrier, an open source search engine that implements state-of-the-art indexing and retrieval functionalities adapted to use on TREC-like collections [START_REF] Ounis | Terrier information retrieval platform[END_REF], provides implementations of many weighting and query expansion models, along with default values of their hyper-parameters (http://terrier.org/docs/v4.0/configure_retrieval.html accessed on 15 May 2022). The optimal values of components hyper-parameters are obtained in an empiric way and different methods have been used for that, grid search is one of the most popular [START_REF] Taylor | Optimisation methods for ranking functions with multiple parameters[END_REF]. Optimal hyper-parameter values have been shown to be collection dependant.

Some studies have shown that the optimal hyper-parameters are not only collection dependent but query dependent as well. For example, considering the query expansion component, Ayter et al. reported that for the query 442 from the TREC7 reference collection, it is best to add 2 to 5 query terms to the initial query when expanding it, while for query 432, it is best to add 20 terms [START_REF] Ayter | Statistical analysis to establish the importance of information retrieval parameters[END_REF]. Note that evaluation forums distinguish between topics and queries. In evaluation frameworks, a topic consists of a title which is generally used as the query to submit to the search engine, a description of one or two sentences that explains the title, and a narrative that depicts what a relevant document is and what information will not be relevant. Because queries are usually formed from topic title, topic and query are often used interchangeably. The relationship between search components and system performance is worth studying in order to better understand the impact of the choice one makes when designing a query processing chain.

With regard to a better understanding of IR thanks to the system features, pioneer studies were based on the results from the participants at IR evaluation forums, such as TREC 5 . The strength of such evaluation campaigns is that a series of systems, each with different query processing chains, are using the same queries, the same document sets to search in, and the same evaluation measures. Because the results are obtained on the same basis, they are comparable and analyses are made possible [START_REF] Tague-Sutcliffe | A Statistical Analysis of the TREC-3 Data[END_REF][START_REF] Banks | Blind men and elephants: Six approaches to TREC data[END_REF][START_REF] Dinçer | Statistical principal components analysis for retrieval experiments[END_REF][START_REF] Mothe | Linguistic analysis of users' queries: Towards an adaptive information retrieval system[END_REF][START_REF] Harman | The NRRC reliable information access (RIA) workshop[END_REF][START_REF] Mizzaro | Hits hits trec: Exploring ir evaluation results with network analysis[END_REF][START_REF] Harman | Overview of the reliable information access workshop[END_REF][START_REF] Bigot | Fusing different information retrieval systems according to query-topics: A study based on correlation in information retrieval systems and TREC topics[END_REF]. The material which is used in such studies is the participants' results to shared tasks, that is to say from 30 to 100 systems (called runs in shared tasks). These studies suffer, however, from the lack of structured and easily exploitable descriptions of the query processing chains that the participants used. The components used in the indexing and query processing, as well as the values of their hyper-parameters, are described in papers and working notes both in verbose ways and with different levels of detail.

Further studies considered generated query processing chains [START_REF] Ayter | Statistical analysis to establish the importance of information retrieval parameters[END_REF][START_REF] Ferro | A general linear mixed models approach to study system component effects[END_REF][START_REF] Ferro | Toward an anatomy of IR system component performances[END_REF]. Here, the data collections from shared tasks are also used, but rather than considering real participants' systems, a huge number of systems, from a few hundred to several thousands, are generated thanks to some tools [START_REF] Louedec | A massive generation of ir runs: Demonstration paper[END_REF][START_REF] Wilhelm | A tool for comparative ir evaluation on component level[END_REF]. The generated systems differ by the components used or their hyper-parameters. Such chains have the advantage of being deterministic in the sense that the components they used are fully described and known, and, thus, allowed deeper analyses. The effect of individual components or hyper-parameters can be studied.

Finally, some studies developed models that aim at predicting the performance of a given system on a query for a document collection [START_REF] Carmel | What makes a query difficult[END_REF][START_REF] Mothe | Linguistic features to predict query difficulty[END_REF][START_REF] Zamani | Neural query performance prediction using weak supervision from multiple signals[END_REF]. The predictive models can help in understanding the system effectiveness. If they are transparent, it is possible to know what the most important features are, what deep learning approaches seldom do.

The challenge for IR we are considering in this paper is:

• Can we understand better the IR system effectiveness, that is to say successes and failures of systems, using data analytics methods?

The sub-objectives targeted here are:

• Did the literature allow conclusions to be drawn from the analysis of international evaluation campaigns and the analysis of the participants' results? • Did data driven analysis, based on thorough examination of IR components and hyper-parameters, lead to different or better conclusions? • Did we learn from query performance prediction?

The more long-term challenge is:

• Can system effectiveness understanding be used in a comprehensive way in IR to solve system failures and to design more effective systems? Can we design a transparent model in terms of its performance on a query?

This paper reviews the literature of this domain. It covers analyses on the system factor, that is to say the effects of components and hyper-parameters on the system effectiveness. It also covers the query factor through studies that analyse the variability due to the queries. Cross-effects are also mentioned. This paper does not cover the question of relevance, although it is in essence related to the effectiveness calculation. It does not cover query performance prediction either.

Rather, this paper question the understanding we have of information retrieval thanks to data analytic methods and provides an overview on which methods have been used in relation to which angle of effectiveness understanding the studies focused on.

The rest of this paper is organised as follows: Section 2 presents the related work. Section 3 presents the material and methods. Section 4 reports on the results of analyses conducted on participants obtained at evaluation campaigns. Section 5 covers the system factor and analyses of results obtained with systematically generated query processing chains. Section 6 is about the analyses on the query effect and cross effects. Section 7 discusses the reported work in terms of its potential impact for IR and concludes this paper.

Related Work

To the best of our knowledge, there is no survey published on this specific challenge. Related work mainly consists of surveys that study a particular IR component. Other related studies are of relevance in IR, query difficulty and query performance prediction, and fairness and transparency in IR.

Surveys on a Specific IR Component

Probably because of its long-standing history in IR and the large number of techniques that have been developed, several surveys focused on the query expansion component. Carpineto and Romano' survey [START_REF] Carpineto | A survey of automatic query expansion in information retrieval[END_REF] includes the different applications of query expansion, as well as the different techniques. They suggested a classification of QE approaches that Azad and Deepak [START_REF] Azad | Query expansion techniques for information retrieval: A survey[END_REF] completed with a four-level taxonomy. To analyse the different methods, Carpineto and Romano did not use any data analytics, rather they used both a classification with various criteria and a comparison of method effectiveness. More precisely, the criteria they used are as follows: the data source used in the expansion (e.g., Wordnet, top ranked documents, . . . ), candidate feature extraction method, feature selection method, and the expanded query representation. With regard to effectiveness, they report mean average precision on TREC collections (sparse results). Mean average precision is the average of average precision on a query set. Average precision is one of the main evaluation measure in IR. It is the area under the precision-recall curve which, in practice, is replaced with an approximate based on precision at every position in the ranked sequence of documents, more at https://jonathan-hui.medium.com/map-meanaverage-precision-for-object-detection-45c121a31173 accessed on 15 May 2022. The authors concluded that for query expansion, linguistics techniques are considered as less effective than statistic-based methods. In particular, local analysis seems to perform better than corpus based. The authors also mentioned that the methods seem to be complementary and that this should be exploited more. Their final conclusion is that the best choice depends on many factors among which the type of collection being queried, the availability and features of the external data, and the type of queries. The authors did not detail the link between these features and the choices of a query expansion mechanism.

Moral et al. [START_REF] Moral | A survey of stemming algorithms in information retrieval[END_REF] focuses on stemming algorithms applied in the indexing and query pre-processing and their effect. They considered mainly rule-based stemmers and classified the stemmers according to their features, such as their strength, the aggressiveness with which the stemmer clears the terminations of the terms, the number of rules and suffixes considered, their use of recoding phase, partial-matching, and constraint rules. They also compared the algorithms according to their conflation rate or index compression factor. The authors did not compare the algorithms in terms of effectiveness but rather refer to other papers for this aspect.

We can also mention the study by Kamphuis et al. in which they considered 8 variants of the BM25 scoring function [START_REF] Kamphuis | Which BM25 Do You Mean? A Large-Scale Reproducibility Study of Scoring Variants[END_REF]. The authors considered 3 TREC collections and used average precision at 30 documents. Precision at 30 documents is the precision, the proportion of relevant document within the retrieved document list where this list is considered up to the 30th retrieved document. They show that there is no significant effectiveness difference between the different implantation of BM25.

These analyses focus on a single component and do not analyse the results obtained strictly speaking but rather compare them using typical report means (mainly tables of effectiveness measures averaged over queries) as presented in Section 2.3.

Effectiveness and Relevance

System effectiveness is closely related to the notion of relevance. Mizzaro [27] studied different kinds of relevance in IR, for which he defined several dimensions. He concluded that common practice to evaluate IR is to consider: (a) the surrogate, a representation of a document; (b) the query, the way the user expresses their perceive information need; and (c) the topic, that refers to the subject area the user is interested in. He also mentioned that this is the lowest level of relevance consideration in that it does not consider the real user's information need nor the perceived information need, nor the information the user creates or receives when reading a document. Ruthven [START_REF] Ruthven | Relevance behaviour in TREC[END_REF] studied how various types of TREC data can be used to better understand relevance and found that factors, such as familiarity, interest, and strictness of relevance criteria, may affect the TREC relevance assessments.

Although relevance and the way relevance assessments are collected and considered can be a factor of IR system effectiveness, in this paper, we do not discuss the relevance point and consider effectiveness in its most common meaning in the IR field, as mentioned in [START_REF] Mizzaro | How many relevances in information retrieval?[END_REF].

Typical Evaluation Report in IR Literature

With regard to hyper-parameters, we should mention that it is a common practice nowadays in IR experimental evaluation (https://www.sigir.org/sigir2012/paperguidelines.php accessed on 15 May 2022 is an example of paper guideline to write IR papers.) to analyse the hyper-parameters of the method one developed. Analysing the results is generally performed by comparing the results in terms of effectiveness in tables or graphs that show the effectiveness for different values of the hyper-parameters (see Figure 2 that represent typical reports on comparison of methods and hyper-parameters in IR papers). In these figures and tables, the parameter values change and either different effectiveness measures or different evaluation collections, or both are reported.

The purpose here is to emphasise that, even if extensive experimental evaluation is reported in IR papers, the reports are mainly under the form or tables and curves, which are low level data analysis representations that we do not discuss in the rest of this paper. Here, a deep learning-based model was used and comparisons are reported on the different training types, encoders, and batch sizes; using different effectiveness measures (nDCG@10, MRR@10, and R@1K), on different collections (here TREC DL'19, TREC-DL'20, and MSMARCO DEV). The best results are highlighted in bold font. the bottom part is a typical graph to compare different variants or hyper-parameters effect on effectiveness. Here, the lines represent different combination of hyperparameters, effectiveness is measured in terms of recall (Y-axis) for different cut-off of the retrieved document list.

Materials and Methods

The analysis of effectiveness for better comprehensive understanding of IR relies on data analysis methods and analysable data that we describe in this section.

Data Analysis Methods

System effectiveness analyses rely on different statistical analysis methods, including, but not limited to, machine learning.

Boxplot is a graphical representation of a series of numerical values that shows their locality, spread, and skewness based on their quartiles. Whiskers extend the Q1-Q3 box, indicating variability outside the upper and lower quartiles. Beyond the whiskers, outliers that differ significantly from the rest of the dataset are plotted as individual points. Effectiveness under different conditions (different queries, different values of a component parameter) is a typical series that can be represented under the form of a boxplot.

Correlation is a family of analysis that measures the relationship between two variables, its strength and direction. Correlation calculation results in a value that ranges between -1 (strong negative correlation) and 1 (strong positive correlation); 0 indicating that the two variables are not correlated. The p-value indicates the confidence or risk of error in rejecting the hypothesis that the two variables are independent. The most familiar measure of correlation is the Pearson product-moment correlation coefficient which is a normalised form of the covariance. Covariance between two random variables measures their join distance to their expected values which can be the distance to the mean for numerical data. Pearson ρ assumes linear relationship between the two variables. Spearman's correlation (r) considers the ranks rather than the values and measures how far from each other variable ranks are. r is similar to Pearson on ranks. Spearman's assumes monotonic relationship between the two variables. Kendall correlation measures the correlation on ranks, that is the similarity of the orderings of data when ranked by each of the variable values. It is affected by whether the ranks between observations are the same or not without considering how far they are as opposed to r. It is thus considered as more appropriate for discrete variables. With regard to system effectiveness, correlation is used in query performance prediction to evaluate the accuracy of the prediction: the two analysed variables are the predictor (either a single predictor or a complex one) and the observed effectiveness.

Analysis of variance (ANOVA) encompasses different statistical models and estimation procedures used to highlight differences or dependencies between several statistical groups. It is used to analyse the difference between the means of more than two groups. In ANOVA, the observed variance in a particular variable is partitioned into components that are attributable to different sources of variation. A one-way ANOVA uses one independent variable, while a two-way ANOVA uses two independent variables. The General Linear Mixed Model [START_REF] Breslow | Approximate inference in generalized linear mixed models[END_REF] extends the General Linear Model [START_REF] Mccullagh | Generalized Linear Models[END_REF] so that the linear predictor contains random effects in addition to the usual fixed effects.

Factorial analysis is used to describe variability among observed, correlated variables; it uses factors, here combinations of initial variables, to represent individuals or data in a space of a lower dimension. It uses singular value decomposition and is appropriate to visualise the link between elements (individuals) that are initially represented in a high dimensional space (variables). Two variants of factorial analysis are used in the context of IR system performance analysis. Factorial analysis is also the core model used in the Latent Semantic Indexing model [START_REF] Dumais | LSA and information retrieval: Getting back to basics[END_REF] where documents are considered in the high dimensional space of words. It is also linked to the matrix factorisation principle used in recommender systems for example [START_REF] Sarwar | Application of Dimensionality Reduction in Recommender System-A Case Study[END_REF]. Principal Component Analysis (PCA) and Correspondence Analysis [START_REF] Benzécri | Statistical analysis as a tool to make patterns emerge from data[END_REF] which differ on the pre-treatment applied to the initial analysed matrix and on the distance used to find the links between variables and individuals. Although PCA reduces the dimensionality of the data by considering the most important dimensions as determined by the eigen values of the variance/covariance matrix using Euclidian distance, CA uses the χ 2 distance on contingency matrices. Factorial analysis results on visual representations which can be manually interpreted. Among others, one interesting property of CA compared to PCA is that individuals and features can be observed all together in the same projected space. Factorial analyses are used in the context of IR effectiveness analysis.

Clustering methods is a family of methods that aims to group together similar objects or individuals. Under this group falls agglomerative clustering and k-means. In agglomerative clustering, each individual corresponds to a cluster; at each processing step, the two closest clusters are merged; the process ends when there is a single cluster. The minimum value of the error sum of squares is used as the ward criterion to choose the pair of clusters to merge [START_REF] Ward | Hierarchical grouping to optimize an objective function[END_REF]. The resulting dendogramme (tree-like structure) can be cut at any level to produce a partition of objects. Depending on its level, the cut will result in either numerous but homogeneously-composed clusters or few but heterogeneously-composed clusters. Another popular clustering method is k-means where a number of seeds, corresponding to the desired number of clusters, are chosen. Objects are associated to the closest seed. Objects can then be re-allocated to a different cluster if it is closer to the centroid of an other cluster. For system effectiveness analysis, clustering can be used to group queries, systems or even measures.

Although the previous methods are usually considered as descriptive ones, the two other groups of methods are predictive methods. That means they are used to predict either a class (e.g., for a qualitative variable) or a value (e.g., for a continuous variable).

Regression methods aim to approach the value of a dependent variable (the variable to be predicted) considering one or several independent variables (the variables or features that are used to predict). The regression is based on a function model with one or more parameters (e.g., linear function in the linear regression; polynomial, . . . ). Logistic regression is for the case the variable to explain is binary (e.g., the individual belongs to a class or not). It is used, for example, in query performance prediction.

Decision trees show a family of non-parametric supervised learning methods that are used for classification and regression. The resulting model is able to predict the value of a target variable by learning simple decision rules inferred from the data features. CART [START_REF] Li | Classification and regression trees (CART)[END_REF] and random forests [START_REF] Ho | Random decision forests[END_REF] are the most popular among these methods. They have been shown as very competitive methods. The extra advantage is that the model can combine both quantitative and qualitative variables. In addition, the obtained models are explainable. For system effectiveness analysis, the target variable is effectiveness measurement or class of query difficulty (easy, hard, medium, for example). The system hyperparameters or query features are used to infer the rules.

In this study, we do not consider deep learning methods as means to analyse and understand information retrieval effectiveness. Deep learning is more and more popular in IR but still these models lack interpretability. The artificial intelligence community is re-investigating the explainability and interpretability challenge of neural network based models [START_REF] Gunning | Explainable Artificial Intelligence; Defense Advanced Research Projects Agency (DARPA)[END_REF]. For example, a recent review focused on explainable recommendation systems [START_REF] Zhang | Explainable recommendation: A survey and new perspectives[END_REF]. Still, model explanability is mainly based on model interpretability and prominent interpretable models are more conventional machine learning ones, such as regression models and decision tree models [START_REF] Zhang | Explainable recommendation: A survey and new perspectives[END_REF].

Data and Data Structures for System Effectiveness Analysis

There are different international challenges in IR where participants use the same data collections to answer shared tasks and, thus, that can be used to deeply analyse system effectiveness, its factors and parameters. In this paper, the studied papers focused on the pioneering TREC challenge. TREC considered many different languages, but when it began and nowadays it is mainly focused on English. TREC encompasses various tasks; the most popular and running from the largest number of years is ad hoc retrieval where the task is to retrieve the relevant documents, given a query. It was also the first and unique task introduced in TREC in 1992 [START_REF] Harman | Overview of the First Text Retrieval Conference (trec-1)[END_REF]. This paper focuses on ad hoc retrieval.

System performance analyses (presented in Sections 4-6) share the same type of data structures, namely matrices.

In general, the participants' results consists in measurements across three dimensions (system, topic, measure). As a result of a challenge like TREC, we can thus build 3D matrices (see Figure 3) that report values for different systems, different topics, different effectiveness measures.

Figure 3. The 3D matrices obtained from participants' results to shared ad hoc information retrieval tasks that report effectiveness measurements for systems, topics, effectiveness measures can be transformed into 2D matrices that fit many data analysis methods.

Such a 3D matrix can be transformed into a 2D one for a given effectiveness measure where the two remaining dimensions are the systems and the topics. The resulting matrix can then be used as an input to many of the data analysis methods we presented in the previous sub-section where individuals are the systems represented according to the topics, or using the transposed matrix, individuals are topics represented according to the systems.

We can also have more information at our disposal on systems or on topics or on both. In that case, the data structures can be more complex. For example, if we consider a given effectiveness measure, systems can be represented by different features (e.g., the components that are used, their hyperparameters, . . . ). In the same way, topics can come with various features (e.g., linguistic or statistical features, . . . ) (see Figure 4). Finally, some studies consider aggregated values. For example, rather than considering each query individually, we can consider aggregated value across queries; this is commonly used to compare systems and methods at a upper level. On the other hand, it is possible not to consider each system individually but aggregate the results across systems (see Figure 5).

System Performance Analysis Based on Their Participation to Evaluation Challenges

International evaluation forums such as TREC. TREC started in 1992 and supports research in the information retrieval and provides the framework for large-scale evaluation of information retrieval have provided shared collections consisting of queries, documents, and relevance judgements, but also consisting of participants' results. For a given collection, we have different systems that ran the queries over the same document set and are evaluated the same way. Indeed, each participant to TREC tracks receives a detailed report where their runs or systems are scored according to various effectiveness measures obtained by the open evaluation tool trec_eval described in https://github.com/topics/trec-eval accessed on 15 May 2022. This opened the opportunity to mine these results.

The first to analyse the results were the organisers. In the first TREC report, the results of the different participants were compared mainly through recall precision curves [START_REF] Harman | Overview of the first TREC conference[END_REF].

Additional analyses were made a few years later. One search engine (SMART system of Cornell University) which has been one of the most effective in TREC ad hoc task, was run in its versions used in each of the first seven TREC conferences on each of the first seven ad hoc test collections [START_REF] Buckley | SMART high precision: TREC 7[END_REF]. This longitudinal analysis of a system version shows that effectiveness increases over time, whatever the system variant used.

The reliable information access workshop was dedicated to the analysis of the participants' results. The organisers reported that a system can perform very well on a query A, very bad on a query B, while an other system will do the opposite [START_REF] Harman | The NRRC reliable information access (RIA) workshop[END_REF]. At that time, it was not possible to understand the reasons of the variability in results; it was stated as being dependant on three factors: the query, the relationship between the query and the documents, and the system parameters; it was considered as a difficult problem to understand [START_REF] Harman | Overview of the reliable information access workshop[END_REF] and this difficulty remains.

Tague-Sutcliffe and Blustein [START_REF] Tague-Sutcliffe | A Statistical Analysis of the TREC-3 Data[END_REF] also reported such variability and showed that the variance due to queries was much greater than the one due to systems. This finding has encouraged the further study on the link between queries and system performance.

Banks et al. [START_REF] Banks | Blind men and elephants: Six approaches to TREC data[END_REF] considered a matrix in which rows and columns represent systems and topics/queries, and cells correspond to average precision (like the one presented in Figure 6). They also considered the retrieved document lists of each participant. Then, they applied six data analysis methods, trying to extract knowledge, relationships, clusters, . . . from these data that would help to understand better the structure or derive general conclusions from the participants' results. Among them, the authors considered the analysis of variance to look for deviations and variability in retrieval performance that could be explained by the systems, topics, or both. They also tried to extract clusters of topics and systems. On document lists, they analysed the correlations on document orders and tried to extract the sub-patterns in the sequence of retrieved documents. The authors concluded "None of the work we have done using the six approaches discussed has provided the sort of insights we were seeking, and the prospects that additional work along these lines will yield significantly better results vary but are not generally promising." [START_REF] Banks | Blind men and elephants: Six approaches to TREC data[END_REF]. Analyses were produced on web track overviews. On Web track 2009 [START_REF] Clarke | Overview of the Trec 2009 Web Track[END_REF], the organisers reported the plot representing the effectiveness of participants' system considering two evaluation measures, the mean subtopic recall and the mean precision (see Figure 7). This analysis showed that the two measures correlate, which means that a system A that is better than a system B on one of the two measures is also better when considering the second measure. When effective systems are effective, the measure that is used does not matter. In web track 2014 [START_REF] Collins-Thompson | TREC 2014 Web Track Overview[END_REF], the authors provided a different type of analysis with box plots that show the dispersion of the effectiveness measurements for each of the topics, across participants' systems (see Figure 8). This type of view informs on the impact of the system factor on the results. The smaller the box, the smaller the importance of the system factor is. Both some easy queries, for which the median effectiveness is high, and hard queries, for which the median effectiveness is low, have packed results (e.g., easy 285 topic in Figure 8 and hard 255 topic-not presented here). Both types have also dispersed results (e.g., easy 285 topic on Figure 8 and hard 269 topic-not presented here). On the same type of matrices than the ones Banks et al. used (see Figure 6), Dinçer et al. [START_REF] Dinçer | Statistical principal components analysis for retrieval experiments[END_REF], Mothe et al. [START_REF] Mothe | Linguistic analysis of users' queries: Towards an adaptive information retrieval system[END_REF], and Bigot et al. [START_REF] Bigot | Fusing different information retrieval systems according to query-topics: A study based on correlation in information retrieval systems and TREC topics[END_REF] applied factorial analyses more successfully. These studies showed that topics and systems are indeed linked. Principal Component Analysis (PCA) and Correspondence Analysis were used.

On Figure 9 we can see PCA applied on a matrix that reports average precision for different queries (variables, matrix columns) by different systems (individuals, matrix rows) at TREC 12 Robust track. We can see on the left bottom part systems that behave similarly on the same queries (they fail on the same queries, succeed on the same queries) and that behave differently from the other systems. We can see another group of systems on the top left corner of the figure. Similar results where reported in [START_REF] Mothe | Linguistic analysis of users' queries: Towards an adaptive information retrieval system[END_REF] where PCA was applied on a matrix that studied recall at TREC Novelty track. In both studies, the results showed that there is not just two groups of systems, thus emphasising that systems behave differently on different queries but that some systems have similar profiles (behave similarly on the same queries). These results are complementary from Mizzaro and Roberston' findings which show that "easy" queries, the ones that on average systems can answer pretty well, are best to distinguish good systems from bad systems [START_REF] Mizzaro | Hits hits trec: Exploring ir evaluation results with network analysis[END_REF].

Kompaore et al. [START_REF] Kompaore | Query clustering and IR system detection. Experiments on TREC data[END_REF] showed that queries can be clustered in a more sophisticated way, based on their linguistic features. The authors extracted 13 linguistic features that they used to cluster queries-based on agglomerative clustering-and obtained three query clusters.

Then, they ranked the TREC ad hoc task participants according to the result they obtained on mean average precision over the entire set of queries, and over each of the clusters. They showed that for each query cluster the best system is not the same (see Figure 10). For example, while ETHme1 was ranked first when considering the mean average precision on the entire set of queries and the query cluster 1, it is ranked the 10th on the query cluster 2. For that query cluster, the best performing system is uwgcx0, and it is LNaDesc1 for query cluster 3. This shows that there is also system profiles that can be extracted, considering topic difficulty levels.

Although these studies analysed TREC participants' systems, other studies have generated different combinations of search components and hyperparameters [START_REF] Bigot | Fusing different information retrieval systems according to query-topics: A study based on correlation in information retrieval systems and TREC topics[END_REF][START_REF] Ferro | A general linear mixed models approach to study system component effects[END_REF][START_REF] Louedec | A massive generation of ir runs: Demonstration paper[END_REF][START_REF] Kompaore | Query clustering and IR system detection. Experiments on TREC data[END_REF][START_REF] Hanbury | Automated component-level evaluation: Present and future[END_REF][START_REF] Arslan | A selective approach to index term weighting for robust information retrieval based on the frequency distributions of query terms[END_REF] and thus went a step further in understanding the factors of variability.

Figure 10. The first ranked system differs according to the query clusters. The rank of the system is on the Y-axis and the system is on X-axis. Blue diamonds correspond to the ranks of the systems when considering all the queries, pink squares when considering the query cluster 1, brown triangles are for query cluster 2, and green crosses for cluster 3. Systems on the X-axis are ordered according to decreasing effectiveness on average on the query set.

Analyses Based on Systems That Were Generated for the Study-The System Factor

The system factor is the factor that has been mentioned the first in shared tasks: systems do not perform identically. Thanks to the results the participants' system obtained in shared tasks, it has been possible to identify which techniques or systems work better in average over queries, but because the description of those systems was not enough, it has not been possible to study the important factors within the systems. This is what some studies aimed to analyse.

Two research groups deeply studied this specific point: the Information Management Systems Research Group of the University of Padua in Italy, starting from 2011 [START_REF] Di Buccio | Interactive Analysis and Exploration of Experimental Evaluation Results[END_REF] and the Information System Research Group of the University of Toulouse in France, starting from 2011 [START_REF] Compaoré | Mining information retrieval results: Significant IR parameters[END_REF]. Google Scholar was used to find what were the first pieces of work related to automatically generated IR chains in the objective to analyse the component effects. Although the two cited works did not obtain many citations, they mark the starting point of this new research track.

The analyses based on synthetic data are in line with the idea developed in 2010 in [START_REF] Hanbury | Automated component-level evaluation: Present and future[END_REF] for an evaluation framework where components would be run locally and where intermediate output would be upload so that component effects could be analysed deeper; evaluation as a service has further developed the same idea [START_REF] Hopfgartner | Evaluation-as-a-service for the computational sciences: Overview and outlook[END_REF].

One of the first implementations of the automatic generation of a large series of query processing chains was the one in Louedec et al. [START_REF] Louedec | A massive generation of ir runs: Demonstration paper[END_REF]; in line with the ideas in [START_REF] Hanbury | Automated component-level evaluation: Present and future[END_REF][START_REF] Kürsten | A large-scale system evaluation on component-level[END_REF] also implemented in [START_REF] Wilhelm | A tool for comparative ir evaluation on component level[END_REF]. It was made possible because of the valuable work that has been performed in Glasgow on Terrier [START_REF] Ounis | Terrier information retrieval platform[END_REF] to implement IR components from the literature. Other platforms can also serve this purpose, such as Lemur/Indri (https://www. lemurproject.org/ accessed on 15 May 2022); https://sourceforge.net/p/lemur/wiki/ Indri%20Retrieval%20Model/ accessed on 15 May 2022 although more centred on language models or Cherche (https://github.com/raphaelsty/cherche accessed on 15 May 2022) for neural models.

Compared to using participants' systems, generated query processing chains gives the ability to know the exact components and hyper-parameters used and thus make deeper analysis possible.

One of the pioneer studies that analysed a huge number of automatically generated systems is Ayter et al. [START_REF] Ayter | Statistical analysis to establish the importance of information retrieval parameters[END_REF]. They used about 80,000 combinations of components in query processing chains with: 4 stemmers, 7 weighting models, 6 query processing, 7 query expansion models, and various numbers of query terms and documents to consider in query expansion. They used TREC 7 and 8 ad hoc collection (100 topics in total querying the same document collection) and average precision as the effectiveness measure. Among their findings, the authors concluded that the choice of the stemmer component had little to no influence, while the weighting model had an impact on the results (see Figure 11). Other findings were that dirichletLM is the weaker search model among the 7 studied, while BB2 is among the best; this when considering also all the other parameters. Their analyses also confirmed that systems behave differently and that the choice of the components at each phase of the retrieving process, as well as the component hyper-parameters, are an important part of system successes and failures. Another important study is the one from Ferro and Silvello [START_REF] Ferro | A general linear mixed models approach to study system component effects[END_REF] followed up by [START_REF] Ferro | Toward an anatomy of IR system component performances[END_REF] from the same authors. On TREC 8, they considered the cross effect of the component choice in the query processing chain. They considered three components: the stop list (5 variants), the stemmer (5 variants), and the weighting model (16 variants), for a total of 400 possible different combinations. As an effectiveness measure, they also used average precision. They show that variants of the stop word list used during indexing does not have a huge impact, but the system should use one (see Figure 12,subfigures A,B where the blue-no stopword-curve is systematically below the others and C where the starting point of the curve-no stopword-is lower than the rest of the curves). It also showed that, given a stopword list is used, the weighing model has the strongest impact among the three studied components (subfigures B and D, where waves show that the systems have different effectiveness).

Additionally, related to these studies, CLAIRE [START_REF] Angelini | A combinatorial visual analytics system for information retrieval evaluation[END_REF] is a system to explore IR results. When analysing TREC ad hoc and web tracks, the findings are consistent with previous analyses: dirichletLM is the weaker weighting model among the studied ones, IR weighting models suffer from the absence of a stop list and of stemmer. By such exploration, the authors were able to show which is the most promising combination of components for a system on a collection (e.g., jskls model equipped with a snowball stop list and a porter stemmer).

These data analytics studies allowed to understand the influence of the components and their possible cross-effect. They are using the results obtained on different collection; they do not aim to predict results. [START_REF] Ferro | A general linear mixed models approach to study system component effects[END_REF]) where we added letters in each sub-figure for clarity. On the first row, the stop list effect is shown for different stemmers (A) and different weighting models (B). On the second row, the effect of stemmers is reported for different stop lists (C) and different weighting models (D). On the latest row, the weighting model effect is reported, for different stop lists (E) and different stemmers (F). Figure adapted with permission from [START_REF] Ferro | A general linear mixed models approach to study system component effects[END_REF], Copyright 2016, Nicola Ferro et al.

The Query Factor

Considering the Queries and Their Pre-and Post-Retrieval Features

The query is the main factor that explains variability in effectiveness [START_REF] Ayter | Statistical analysis to establish the importance of information retrieval parameters[END_REF][START_REF] Tague-Sutcliffe | A Statistical Analysis of the TREC-3 Data[END_REF][START_REF] Mizzaro | Hits hits trec: Exploring ir evaluation results with network analysis[END_REF][START_REF] Dejean | Studying the variability of system setting effectiveness by data analytics and visualization[END_REF]. The query factor, thus, attracted much research attention mainly under the query difficulty or query performance prediction research topics. Because this paper focuses on how data analytics can help IR effectiveness understanding, we do not survey query performance predictors but rather what we have learnt from the studies on the different types of query features used for effectiveness prediction.

The explanation of system effectiveness failure, which is strongly related to query difficulty in IR terminology, by the query it-self was studied considering different types of features:

• Linguistic features extracted from the query only [START_REF] Mothe | Linguistic features to predict query difficulty[END_REF][START_REF] De Loupy | Evaluation of document retrieval systems and query difficulty[END_REF][START_REF] Banerjee | Extended gloss overlaps as a measure of semantic relatedness[END_REF][START_REF] Patwardhan | Using WordNet-based context vectors to estimate the semantic relatedness of concepts[END_REF]; • Other pre-retrieval features that use information on the document collection [57-61]; • Post retrieval features that consider the retrieved documents for that query [START_REF] Zamani | Neural query performance prediction using weak supervision from multiple signals[END_REF][START_REF] Cronen-Townsend | Predicting query performance[END_REF][START_REF] Sehgal | Predicting performance for gene queries[END_REF][START_REF] Zhou | Ranking robustness: A novel framework to predict query performance[END_REF][START_REF] Vinay | On ranking the effectiveness of searches[END_REF][START_REF] Aslam | Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions[END_REF][START_REF] Zhou | Query performance prediction in web search environments[END_REF][START_REF] Shtok | Predicting query performance by query-drift estimation[END_REF][START_REF] Carmel | Estimating the query difficulty for information retrieval[END_REF][START_REF] Cummins | Improved query performance prediction using standard deviation[END_REF][START_REF] Roitman | Robust standard deviation estimation for query performance prediction[END_REF][START_REF] Chifu | Query performance prediction focused on summarized letor features[END_REF][START_REF] Zhang | Query performance prediction and classification for information search systems[END_REF][START_REF] Khodabakhsh | Semantics-enabled query performance prediction for ad hoc table retrieval[END_REF].

Mothe and Tanguy [START_REF] Mothe | Linguistic features to predict query difficulty[END_REF] considered 16 linguistic query features. They observed the correlation of these features with average recall and precision obtained by TREC participant systems. Morphological, syntactical and semantical features were considered. Syntactic links span (distance between syntactically linked words) was shown as inversely correlated to precision while polysemy value, the number of semantic classes a given word belongs to in WordNet was shown to be inversely correlated to recall.

Molina [START_REF] Molina | IRIT-QFR: IRIT query feature resource[END_REF] developed a resource where 258 features have been extracted from queries from Robust, WT10G and GOV2 collections. Among the features extracted from the query only, are aggregations (minimum, maximum, mean, quartiles, standard deviation, and total) over query terms on the number of synonyms, hyponyms, meronyms and sister terms from WordNet. The authors did not provide a deep analysis on how much each of these features correlate with the system performance.

Hauff et al. [START_REF] Hauff | A survey of pre-retrieval query performance predictors[END_REF] surveyed 22 pre-retrieval features from the literature at that time, from which some use the query only to be computed. The authors categorised these features into four groups, depending on the underlying heuristic: specificity, ambiguity, term relatedness, and ranking sensitivity. Intuitively, the higher the term specificity the easier the query. Inversely, the higher the term ambiguity, the more difficult the query is. When analysed on three TREC collections, the authors found weak correlation between the system performance and features based on the query only. Among them, the features related to the term ambiguity where the most correlated to system performance, in line with [START_REF] Mothe | Linguistic features to predict query difficulty[END_REF]. Features that consider information on the documents were found more correlated to performance than the ones based on the query only.

The query effect was also studied in link with the document collection that is searched by considering information resulting from the document indexing. These query features are grounded on the same idea as term weighting is for indexation: terms are not equivalent and their frequency in documents matters. Inverse document frequency, based on the number of documents in which the term occurs has specifically been used, but other features were also developed [START_REF] He | Inferring query performance using pre-retrieval predictors[END_REF][START_REF] Zhao | Effective pre-retrieval query performance prediction using similarity and variability evidence[END_REF][START_REF] Macdonald | Predicting query performance in intranet search[END_REF].

Finally, the query effect was also studied considering post-retrieval features [START_REF] Cronen-Townsend | Predicting query performance[END_REF][START_REF] Sehgal | Predicting performance for gene queries[END_REF][START_REF] Zhou | Ranking robustness: A novel framework to predict query performance[END_REF][START_REF] Vinay | On ranking the effectiveness of searches[END_REF][START_REF] Aslam | Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions[END_REF][START_REF] Zhou | Query performance prediction in web search environments[END_REF][START_REF] Shtok | Predicting query performance by query-drift estimation[END_REF][START_REF] Cummins | Improved query performance prediction using standard deviation[END_REF][START_REF] Roitman | Robust standard deviation estimation for query performance prediction[END_REF][START_REF] Chifu | Query performance prediction focused on summarized letor features[END_REF][START_REF] Zhang | Query performance prediction and classification for information search systems[END_REF][START_REF] Khodabakhsh | Semantics-enabled query performance prediction for ad hoc table retrieval[END_REF]. Post-retrieval predictors are categorised into clarity-based, score-based, and robustness-based approaches [START_REF] Zamani | Neural query performance prediction using weak supervision from multiple signals[END_REF][START_REF] Carmel | Estimating the query difficulty for information retrieval[END_REF]. They imply that a first document retrieval is performed using the query before the features can be calculated. Post-retrieval features mainly used the document scores.

Considered individually, post-retrieval features have been consistently shown as better predictors than pre-retrieval ones. It appeared, however, that an individual feature, either pre-or post-retrieval, is not enough to predict whether the system is going to fail or not or to predict its effectiveness. That mean that none of these individual features "explained" the system performance.

Indeed, many studies have reported weak correlation values for individual features [START_REF] Hauff | A survey of pre-retrieval query performance predictors[END_REF][START_REF] Chifu | Query performance prediction focused on summarized letor features[END_REF][START_REF] Faggioli | A new paradigm to evaluate and understand query performance prediction methods[END_REF] with the actual system effectiveness. When considering a single feature, the correlation values differ from one collection to another and from one feature to another. Moreover, they are weak. For example, Hauff et al. [START_REF] Hauff | A survey of pre-retrieval query performance predictors[END_REF] report 396 correlation values among which 13 only are over 0.5. Hashemi et al. [START_REF] Hashemi | Performance Prediction for Non-Factoid Question Answering[END_REF] reported 216 correlation values including the ones obtained with a new neural network-based predictor, with a maximum value of 0.58, a median of 0.17. Chifu et al. [START_REF] Chifu | Query performance prediction focused on summarized letor features[END_REF] reported 312 values, none of which above 0.50. In the same way, Khodabakhsh and Bagheri report 952 correlation values, none of which are above 0.46 [START_REF] Khodabakhsh | Semantics-enabled query performance prediction for ad hoc table retrieval[END_REF]. When correlation are low it is even likelier that there is either very weak or no correlation at all between the predicted value (here effectiveness) and the feature used to predict. Table 1 and Figure 13 illustrate this. For this illustration, we took IDF Max and IDF AVG which are considered as the best pre-retrieval features [START_REF] Hauff | A survey of pre-retrieval query performance predictors[END_REF][START_REF] Cummins | Improved query performance prediction using standard deviation[END_REF][START_REF] Roy | Estimating Gaussian mixture models in the local neighbourhood of embedded word vectors for query performance prediction[END_REF], as well as BM25, a post-retrieval feature. We can see that with a correlation of 0.29 for BM25 or 0.24 for IDF (see Table 1), there is no correlation between the two variables as depicted on the scatter plots (see Figure 13).

Papers on query performance prediction seldom plot the predicted and actual values which is however an appropriate mean to check whether the correlation exists or not. As a counter example of this, we should here recall that the Anscombe's quartet [START_REF] Anscombe | are collaborating with JSTOR to[END_REF] effect on the Pearson correlation illustrates that even a Pearson correlation up to 0.816 can be obtained with no correlation between the two studied variables (see Figure 14). A single query feature cannot explain a single system effectiveness, but:

1.

Combination of query features might; 2.

It may explain that systems will fail in general.

Regarding (1), a series of studies have been produced to combine query features into models [START_REF] Roy | Estimating Gaussian mixture models in the local neighbourhood of embedded word vectors for query performance prediction[END_REF][START_REF] Grivolla | Automatic Classification of Queries by Expected Retrieval Performance[END_REF][START_REF] Raiber | Query-performance prediction: Setting the expectations straight[END_REF][START_REF] Mizzaro | Query performance prediction and effectiveness evaluation without relevance judgments: Two sides of the same coin[END_REF]. Grivolla et al. [START_REF] Grivolla | Automatic Classification of Queries by Expected Retrieval Performance[END_REF] grouped queries according to the predicted performance. They trained a model that combined various features, including linguistic, pre-and post-retrieval features, and used decision tree and SVM. The experiments they reported on TREC-8 participants' results showed that the model was not robust across systems: for some of them the prediction was accurate while it was not for some others. Raiber et al. combined various features in a Markov Random Fields model and reported a maximum correlation of 0.695 and a median of 0.32 [START_REF] Raiber | Query-performance prediction: Setting the expectations straight[END_REF]. Chifu et al. [START_REF] Chifu | Query performance prediction focused on summarized letor features[END_REF] combined Letor features using a linear model and reported a maximum correlation of 0.45. The only study we found that both reported positive results and plotted predicted vs actual effectiveness is Roy et al. [START_REF] Roy | Estimating Gaussian mixture models in the local neighbourhood of embedded word vectors for query performance prediction[END_REF]. They proposed a linear combination of a word embedding based pre-retrieval feature which measures the ambiguity of each query term, with the post-retrieval NQC feature (see Figure 15). However, the model performs well for easy queries only. However, we know from Mizzaro and Robertson [START_REF] Mizzaro | Hits hits trec: Exploring ir evaluation results with network analysis[END_REF] that easy queries do not help to distinguish effective systems from non-effective ones [START_REF] Mizzaro | Hits hits trec: Exploring ir evaluation results with network analysis[END_REF]. With regard to (2), rather than considering individual system performance, Mizzaro et al. [START_REF] Mizzaro | Query performance prediction and effectiveness evaluation without relevance judgments: Two sides of the same coin[END_REF] focused on the average of average precision values over systems, this to detect the queries for which systems will fail in general. The authors showed that the correlation is more obvious between the predictor and the average system effectiveness than it was in other studies between the predictor and a single system (see Figure 16). This call also for the need to try to understand the relationship between the query factor and the system factor.

Relationship between the Query Factor and the System Factor

In Section 5, we reported studies on system variabilities and the effect of the components used at each search phase on the results when averaged over query sets, not considering deeply the query effect.

Here, we consider studies that were conducted at a finer grain. These pieces of work tried to understand the very link between the level of query difficulty (or level of system effectiveness) and the system components or hyperparameters.

It was concretely used in Ayter et al. where 80,000 combinations of components and hyper-parameters were evaluated on the 100 queries of TREC 7 and 8 ad hoc track topics. The combinations differed on the stemmer used to index the documents (4 different stemmers were used), the topic tags used to build the query, the weighting model (7 different models), the query expansion model (6 different models) and the query expansion hyper-parameters which take different values as well. The authors showed that the most important parameters, the ones that influence the results the most, depend on the difficulty of the query, that is to say whether relevant documents will be easily found by the search engine or not [START_REF] Ayter | Statistical analysis to establish the importance of information retrieval parameters[END_REF] (see Figure 17).

In Figure 17a where the easy queries only are analysed, we can see that the most influential parameter is the query expansion model used because this is the one where the tree first split, here for the value c, which corresponds to the Info query expansion model. The retrieving or matching model is the second most influential parameter. For hard queries however, the most influential parameter is the topic part used for the query. In that research the authors either used the title only, or the other topic fields, narrative and descriptive, that provide more information on the users' need related to the topic. The leaves of the tree is whether the decision for a query is "easy" (good performance), "average" or "hard" (bad performance) when following a branch from the root to the leave. The main overall conclusion is that the influential parameters are not the same for easy and hard queries; giving the intuition that obtaining the best performance cannot be by applying the same process whatever the queries are. This was further analysed in [START_REF] Dejean | Studying the variability of system setting effectiveness by data analytics and visualization[END_REF], where more TREC collections were studied with the same conclusions.

These results are in favour of considering the system component parameters, not at a global level like search grid or other optimisation methods do in IR, but rather at a finer grain.

Discussion and Conclusions

Understanding information retrieval effectiveness involves considering several dimensions. In this paper, we focused on the system and the query, while the document collection and the effectiveness measures were in the background.

From evaluation forums and shared tasks, although participants provide some detailed description of the systems they designed, the information is not enough structured or detailed to draw conclusions, except in broad strokes. The main conclusions from the analyses of shared tasks results are: • C1: it is possible to distinguish between effective and non-effective systems on average over a query set; • C2: effectiveness of systems has increased over years thanks to the effort put in the domain;

• C3: some queries are easy for all the systems while others are hard for all (see Figure 18, left-side part) but systems do not always fail or succeed on the same queries (see Figure 18, right side part). Some systems have a similar profile, they fail/success on the same queries. However, it was not possible to understand system successes and failures.

Regarding C1, we also considered the participants' results from the first 7 years of TREC ad hoc for a total of 496 systems (or runs) and considered the 130 effectiveness measures from trec_eval that evaluate (system, query) pairs, such as in [START_REF] Baccini | How many performance measures to evaluate information retrieval systems?[END_REF]. Correlation when considering pairs of measurements for a given topic and a given system are high, which means it is possible to distinguish between effective and non effective systems, it does not depend on the measure used (see Figure 19). . Some queries are easy for all the systems, some are hard for all, other depends on the system. On the TREC topic 297, all the analysed systems obtained at least 0.5 as NDCG@20, half of them obtained more than 0.65 and some obtained 0.8, which is high. For topic 255, all the systems failed but 3, only one obtained more than 0.3. The right part boxplot, as opposed to the left side ones, shows that for topic 298, the system effectiveness have a large span from 0 to almost 1.

Figure 19.

When considering a given system and a given query, the effectiveness measure used to compare the systems does not matter much: all are strongly correlated. Pearson correlation values between two effectiveness measurements on two measures for a given (system, query) pair. Correlations are represented using a divergent palette (a central colour, yellow, and 2 shades depending on whether the values go for negative-red-or positive values-blue).

Regarding C2, although it might be overgeneralisation of a single phenomenon, results from SMART system are convincing: the performances almost double when considering the results on the first participation to TREC ad hoc compared to the ones obtained 6 years later (see details Section 4).

Regarding C3, many studies agreed on this observation (see Figure 18). Moreover, systems with the same profiles belong generally to the same research group which means that they may be just small variants one from the other (e.g., different hyper-parameters but basically the same query processing chain, using the same components).

Different attempts to extract more information from official runs were not conclusive.

From automatically generated query processing chains, we have a deep knowledge on the systems, we know exactly what components are used with which hyper-parameters. From the analyses that used these data, we can conclude:

• C4: some components and hyper-parameters are more influential than others and informed choices can be made; • C5: the choice of the most appropriate components depends on the query level of difficulty.

Regarding C4: for some components or their hyperparameters, their choice will have a huge impact on the system effectiveness, for some others, there is no or little impact on effectiveness. This means that if one wants to tune or decide on not all the parameters but a few they should start by the more influential ones. Moreover, we know what is the best decision on some components: a stoplist should be used, a stemmer should be used, but the choice of the stemmer does not matter much; considering the weighting models that are implemented in Terrier, dirichletLM should be avoid, BB2 is a much better option.

Regarding C5: the choice of the most appropriate query processing chain is in relation with the query level of difficulty. In other words, different queries need different processing chains. This means that if we want to increase system effectiveness in the future, we should not just tune the system on a per collection basis: grid searching or any other more sophisticated version of parameter optimisation is not enough. What we need rather is to adapt the processing chain to the query.

From query analyses, we can conclude:

• C6: a single query feature or a combination of features have not been proven to explain system effectiveness; • C7: query features can explain somehow system effectiveness.

Despite their apparent contradiction, C6 and C7 are in fact complementary. Some query features or combination of them seems to be accurate to predict not individual systems but average system effectiveness; there is also some success on predicting easy queries. Systems are, however, more easily distinguishable based on the difficult queries, not the easy ones for which they are more homogeneous in their successes. Up to now, the accuracy of features or feature combinations has not demonstrated that they can explain system effectiveness; correlation values that are reported are seldom over 0.5 and more tricky studies do not report scatter plots.

Although we do not yet understand well the factors of system effectiveness, the studies show that not a single system, while effective in average on a query set, is able to answer all the queries well (mainly C5 in addition to C3 and C4). Advanced IR techniques can be grounded on this knowledge. Selective query expansions (SQE) for example, where a metasystem has two alternative component combinations which differ on using or omitting the automatic query reformulation phase, made use of the fact that some queries benefit from being expanded while other do not [START_REF] Amati | Query difficulty, robustness, and selective application of query expansion[END_REF][START_REF] Cronen-Townsend | A framework for selective query expansion[END_REF][START_REF] Zhao | Automatic term mismatch diagnosis for selective query expansion[END_REF]. SQE has not been proven to be very effective, certainly due to both the limited number of configurations used at that time (two different query processing chains) and the relatively poor learning techniques used at that time. Selective query processing expand SQE concept, where the system decides which one, from a possibly large number of component combinations, should be used for each individual query [START_REF] Dinçer | Statistical principal components analysis for retrieval experiments[END_REF][START_REF] Bigot | Fusing different information retrieval systems according to query-topics: A study based on correlation in information retrieval systems and TREC topics[END_REF][START_REF] Deveaud | Learning to Adaptively Rank Document Retrieval System Configurations[END_REF]. Here, the results were more conclusive. For example, Bigot et al. [START_REF] Bigot | Learning to Choose the Best System Configuration in Information Retrieval: The Case of Repeated Queries[END_REF] developed a model that learns the best query processing chain to use for a query based on subsets of documents. Although this makes the method applicable for repeated queries only, it can be an appropriate approach for real world web search engines. Deveaud et al. [START_REF] Deveaud | Learning to Rank System Configurations[END_REF] learn to rank the query processing chain for each new query. They used 20,000 different query processing chains. However, this very large number of combinations makes it difficult to use in real world systems. Arslan and Dinçer [START_REF] Arslan | A selective approach to index term weighting for robust information retrieval based on the frequency distributions of query terms[END_REF] developed a meta-model that uses eight term-weighting models that could be chosen among for any new query. The meta-model has to be train, as well as the term-weighting models; this is performed by a grid search optimisation which limit the usability. Mothe and Ullah [START_REF] Mothe | Defining an Optimal Configuration Set for Selective Search Strategy-A Risk-Sensitive Approach[END_REF] present an approach to optimise the set of query processing chains that can be chosen among a selective query processing strategy. It is based on a risk-sensitive function that optimises the possible gain in considering a specific query processing chain. The authors show that 20 query processing chains is a good trade-off between the cost of maintaining different query processing chains and the gain on effectiveness. Still they do not explain the successes and failures.

Thus, to the question "Can we design a transparent model in terms of its performance on a query", I am tempted to answer: "No, at this stage of IR knowledge; further analyses are needed". I am convinced that data analytics methods can further been investigated to analyse the amount of data that have been generated by the community, both in shared tasks and in labs while tuning systems.

The robustness of the finding across collections would also worth investigating in the future.

Figure 1 .

 1 Figure 1. An online search process consists of four main phases to retrieve an ordered list of documents that answer the user's query. The component used at each phase has various hyperparameters to tune.

Figure 2 .

 2 Figure 2. A common practice in IR literature is to analyse the effect of hyper-parameters on the overall system effectiveness and to present the results under the form of tables or graphs. The top part of this figure is a typical table that represents hyper-parameters or comparison of methods. Here, a deep learning-based model was used and comparisons are reported on the different training types, encoders, and batch sizes; using different effectiveness measures (nDCG@10, MRR@10, and R@1K), on different collections (here TREC DL'19, TREC-DL'20, and MSMARCO DEV). The best results are highlighted in bold font. the bottom part is a typical graph to compare different variants or hyper-parameters effect on effectiveness. Here, the lines represent different combination of hyperparameters, effectiveness is measured in terms of recall (Y-axis) for different cut-off of the retrieved document list. Table and Figure adapted with permission from [29], Copyright 2021, Sebastian Hofstätter et al.

Figure 4 .

 4 Figure 4. More complex data structures can be used that integrate features on topics, on systems or on both.

Figure 5 .

 5 Figure 5. Aggregated values can be considered, either on the queries or systems.

Figure 6 .

 6 Figure 6. A 2D matrix representing the effectiveness of different systems (X axis) on different topics (Y axis). This matrix is an extract of the one representing the AP (effectiveness measure) for TREC 7 ad hoc participants on the topic set of that track.

Figure 7 .

 7 Figure 7. Effective systems are effective whatever the measure used. Web track 09 participants' results considering mean subtopic recall (X-axis) and mean precision (Y-axis); each dot is a participant systems. Figure reprinted with permission from [43], Copyright 2009, Charles Clarke et al.

Figure 8 .

 8 Figure 8. In the easiest topics according to the median effectiveness on the participants' results, there are both topics with very diverse system effectiveness results (e.g., 298) and very similar ones (e.g., topic 285)-Web track 2014-topics are ordered according to decreasing the err@20 of the best system. Figure reprinted with permission from [44], Copyright 2014, Kevyn Collins-Thompson et al.

Figure 9 .

 9 Figure 9. System failure and effectiveness depend on queries-not all systems succeed or fail on the same queries. The visualisation shows the two first principal components of a Principal Component Analysis, where the data of the system effectiveness is obtained for each topic by each participants' run. MAP measure of TREC 12 Robust Track participants' runs. Figure reprinted with permission from [10], Copyright 2007, John Wiley and Sons.

  Figure10. The first ranked system differs according to the query clusters. The rank of the system is on the Y-axis and the system is on X-axis. Blue diamonds correspond to the ranks of the systems when considering all the queries, pink squares when considering the query cluster 1, brown triangles are for query cluster 2, and green crosses for cluster 3. Systems on the X-axis are ordered according to decreasing effectiveness on average on the query set. Figure reprinted with permission from [45], Copyright 2007, Kompaore et al.

Figure 11 .

 11 Figure 11. The choice of the weighting model has more impact than the stemmer used. Individual boxplots represent average precision on the TREC 7 and 8 topics when a given component is used in a query processing chain-80,000 query processing chains or component combinations were used. Figure reprinted with permission from [7], Copyright 2015, J.UCS.

Figure 12 .

 12 Figure 12. Interaction between component choices. The curves used in this representation are somehow misleading since the variables are not continuous but nevertheless can be understood, we thus kept the original Figures from[START_REF] Ferro | A general linear mixed models approach to study system component effects[END_REF]) where we added letters in each sub-figure for clarity. On the first row, the stop list effect is shown for different stemmers (A) and different weighting models (B). On the second row, the effect of stemmers is reported for different stop lists (C) and different weighting models (D). On the latest row, the weighting model effect is reported, for different stop lists (E) and different stemmers (F). Figure adapted with permission from[START_REF] Ferro | A general linear mixed models approach to study system component effects[END_REF], Copyright 2016, Nicola Ferro et al.

Figure 13 .

 13 Figure 13. No correlation using pre-or post-predictors with the actual effectiveness-IDF preretrieval predictor and BM25 post-retrieval predictor (X-axis) and ndcg (Y-axis) values on WT10G TREC collection. Although the correlation values are up to 0.35, there is no correlation.

Figure 14 .

 14 Figure 14. Pearson correlation value higher than 0.8 does not mean the two variables are correlated. The Anscombe's quartet presents four datasets that share the same mean, same number of values, same Pearson correlation value (ρ = 0.816) but for which this latter value does not always means the two variables X and Y are correlated. X and Y are not correlated on #4 despite high ρ value. #2 X and Y are perfectly correlated but not in a linear way (Pearson cannot measure other than linear correlations) #1 and #3 illustrates two cases of linear correlation. Figures generated from the data in [79].

Figure 15 .

 15 Figure 15. Predicted AP is correlated to actual AP for easy queries (the ones on the right part of the plot), although there are sparse. Figure reprinted with permission from Roy et al. [78], Copyright 2019, Elsevier.

Figure 16 .

 16 Figure[START_REF] Ferro | A general linear mixed models approach to study system component effects[END_REF]. AS feature[START_REF] Aslam | On the Effectiveness of Evaluating Retrieval Systems in the Absence of Relevance Judgments[END_REF] is correlated to the average effectiveness of a set of systems. TREC7 Adhoc collection. Pearson correlation between AAP and (a) QF[START_REF] Zhou | Query performance prediction in web search environments[END_REF], (b) AS[START_REF] Aslam | On the Effectiveness of Evaluating Retrieval Systems in the Absence of Relevance Judgments[END_REF]. Dots correspond to actual and predicted AAP for individual topics; the cones represent the confidence intervals. Figure reprinted with permission from [82], Copyright 2018, Mizzaro et al.

  (a) Easy queries (b) Difficult queries

Figure 17 .

 17 Figure 17. The parameters that affect retrieval effectiveness the most depend on the query difficulty. On (a), for easy queries, the most important parameter for search effectiveness optimisation is the choice of the query expansion component; on (b), for hard queries, the most important parameter is the topic parts used for building the query, then the weighting component and in third the query expansion model. Figure reprinted with permission from [7], Copyright 2015, J.UCS.

Figure 18

 18 Figure18. Some queries are easy for all the systems, some are hard for all, other depends on the system. On the TREC topic 297, all the analysed systems obtained at least 0.5 as NDCG@20, half of them obtained more than 0.65 and some obtained 0.8, which is high. For topic 255, all the systems failed but 3, only one obtained more than 0.3. The right part boxplot, as opposed to the left side ones, shows that for topic 298, the system effectiveness have a large span from 0 to almost 1.

  Table and Figure adapted with permission from [29], Copyright 2021, Sebastian Hofstätter et al.

Table 1 .

 1 Correlation between query features and ndcg. WT10G TREC collection. * marks the usual <0.05 p-Value significance.

	Feature

Funding: This research received no external funding.

Conflicts of Interest:

The author declares no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript: