Gilles Hubert

José G Moreno

Keywords: 5, 3 Search task identification results for the CUSTA dataset, 5, 4 Clustering performance for the WSMC12 dataset

I would like to thank the

List of Figures

Introduction

On ne voit bien qu'avec le coeur. L'essentiel est invisible pour les yeux.

Antoine de Saint-Exupéry

Users divide mental tasks in a sequence of multiple easy steps to deal with mental overload [START_REF] Kahneman | Thinking, fast and slow[END_REF]. In particular, when users interact with search systems to satisfy their information needs, they divide search tasks into sequences of queries. Those interactions are registered in query logs, and mining users' interactions enable the development of models that learn the patterns users follow to carry out their search tasks [START_REF] Hearst | Search user interfaces[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF].

Initially, when users run a search query, search systems produced a list of hyperlinks for ranked documents. Commonly known as the "ten-blue-links" approach to search, this paradigm is no longer in use [START_REF] Rosset | Leading conversational search by suggesting useful questions[END_REF]. Nowadays, search systems assist users while they carry out their search tasks using multiple supporting applications, including query autocompletion [START_REF] Li | Exploring query auto-completion and click logs for contextual-aware web search and query suggestion[END_REF]Sun and Lou, 2014), search clarification (Zamani et al., 2020), query suggestion [START_REF] Ahmad | Context attentive document ranking and query suggestion[END_REF][START_REF] Dosso | How to support search activity of users without prior domain knowledge when they are solving learning tasks[END_REF], query term prediction, e-commerce applications, product recommendations, advertising retrieval for products [START_REF] Hearst | Search user interfaces[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF] query-task mapping (Völske et al., 2019), ranking of results based on search tasks [START_REF] Ahmad | Context attentive document ranking and query suggestion[END_REF], suggestion of questions [START_REF] Rosset | Leading conversational search by suggesting useful questions[END_REF], and so on.

Furthermore, advances in speech recognition, natural language understanding, and text-to-speech synthesis enable users to leverage digital personal assistants to satisfy their information needs (Thomas et al., 2020;Zamani and Craswell, 2020). Digital assistants are now available in desktops, smartphones, tablets, smartwatches, and dedicated smart speakers. When users access digital assistants for information seeking, tracking the conversa-1 Introduction tion topic is essential so that users can carry out their search tasks [START_REF] Khatri | Contextual topic modeling for dialog systems[END_REF][START_REF] Khatri | Contextual topic modeling for dialog systems[END_REF].

To underpin those user assisting applications and retrieval methods, it is crucial to develop models that automatically learn from user interactions with search systems. As search systems evolve and new platforms emerge, centering on users to help them in their information seeking journeys, learning from user interactions is becoming increasingly important in information retrieval research [START_REF] Mehrotra | Special issue on learning from user interactions[END_REF].

Therefore, the research problem we aim to address in this dissertation is the development of models that learn search patterns from user interactions with search systems. For developing these models, along with the interactions of users with search systems, we also leverage advances in natural language processing for encoding search queries. As queries are steps users execute as part of a search task, and because of the relevance of search tasks in many user assisting applications and retrieval methods, we focus our research on search patterns related to user search tasks.

To address the research problem, the first approach is the detection of search task boundaries in user interactions with search systems. Some existing models use surrounding user queries or clicked URLs, which make them unfeasible to use in realtime interactions. Other models use heuristics with constant thresholds, but a manual statistical analysis is required to update the thresholds. We propose a neural model that learns to detect search task boundaries in query logs. The interactions of the users in the search query logs are ordered by time and grouped by user identifiers, providing a chronologically ordered set of interactions per user. Such interactions are the input to the proposed model, which is supervised [START_REF] Lugo | Segmenting search query logs by learning to detect search task boundaries[END_REF].

The second approach deals with the modeling of search tasks from user interactions with search systems. We propose Recurrent Deep Clustering (RDC), a deep clustering architecture that extends the foregoing neural model to extract user search tasks from query logs in an unsupervised way. The deep clustering architecture uses a multiobjective approach, simultaneously optimizing the query representation and the clustering of queries into search tasks. The representation for user queries is learned through a selfsupervised method, leveraging data augmentation for language inputs and dual-channel neural architectures (Lugo et al., 2021a,b).

Most existing models for search task identification are monolingual, supporting user queries normally in English only. However, users around the world access search systems in many languages. Also, a single user can issue queries in two or more languages to solve a particular information need. Another limitation of existing models for search tasks is the lack of realtime response times, which relegates them to offline modeling applications. For the third approach, we propose Multilingual Graph-based Clustering (MGBC), a multilingual unsupervised approach for search task identification, supporting user queries in several languages and providing the possibility of mapping queries to search tasks in realtime (Lugo et al., 2020a).

Overview

Finally, the fourth approach is Language-agnostic Search Task Modeling (LASTM), an unsupervised model that uses a multiobjective approach, simultaneously optimizing search task clusters and user search intents through the relationship between user queries and clicked documents. The modeling scheme for user search intent leverages user query -clicked document relationships, representing queries in a language-agnostic semantic space. This proposed model takes advantage of query-task mapping to provide results in realtime [START_REF] Lugo | Modeling user search tasks with a language-agnostic unsupervised approach[END_REF]. Moreover, the code for all contributions is publicly available in a GitHub repository1 , facilitating the replication of results and the utilization of proposed models for further research.

Overview

The remaining chapters of this dissertation are organized as follows. Chapter 2 presents a critical review of existing work, ranging from publicly available datasets to machine learning methods, to natural language models for text representation, to existing information retrieval literature centered around user search patterns. Chapter 3 proposes a neural model for learning to detect search task boundaries in query logs. In chapter 4, a recurrent deep clustering architecture for search task extraction is proposed, along with a self-supervised method to learn query representations using data augmentation. Chapter 5 develops a multilingual, unsupervised approach for identifying user search tasks, with the ability to map queries to search tasks in realtime. Chapter 6 proposes a model for user search intent, which leverages the relationship between user queries and clicked documents. At the same time, it develops a language-agnostic, unsupervised approach for modeling search tasks using semantic relationships between queries. Finally, chapter 7 explores the significance of the contributions, along with perspectives for future research directions.

Chapter 2 Background . . . sin saber que la búsqueda de las cosas perdidas está entorpecida por los hábitos rutinarios, y es por eso que cuesta tanto trabajo encontrarlas.

Gabriel García Márquez

More than half the world's population uses the internet, according to the Internet World Stats1 . Essential components in such interactions are search systems, which help users access the ever-increasing amount of information available to satisfy their information needs. Search systems allow users to perform their search tasks, relying on large web indexes to respond to users' requests. For example, Google's search index has an estimated size of around 55 billion web pages, while Microsoft's Bing search index has an estimated size of around 12 billion web pages2 .

Likewise, the most popular mobile applications, like WhatsApp Messenger, Twitter, and Facebook 3 , have search systems available for the millions of users who access these applications every day. Android, the mobile OS with the biggest user base, has a default search bar on its main screen. By the same token, Windows, the most popular desktop OS in the world 4 , has a search box right beside the Start button. The search box is also a default feature in its Windows 10 version. On top of that, the advent of smart home devices 6 2 Background and digital assistants like Amazon Alexa, Apple Siri, Google Assistant, and Microsoft Cortana, constitutes another way to interact with search systems.

All those interactions enable the development of models that learn search patterns to better assist users in their information seeking. In this chapter, we perform a critical review of existing literature, comparing and contrasting existing research related to the modeling of users' search patterns.

To structure the literature review, we consider that a wide range of needs and desires from users are converted to queries and run in search systems [START_REF] Hearst | Search user interfaces[END_REF]. Search query logs register the queries users run to fulfill their information needs. Some of those search query logs are publicly available for analysis and model extraction (Section 2.1). Mining such query logs allows the modeling of search behaviors through automatic or manual analysis of user interactions. For automatic analysis, machine learning (Section 2.2) provides methods for processing the high dimensional data present in query logs. Machine learning approaches require representing queries in a vector space. Thus, multiple language representation models (Section 2.3) have been proposed to encode the text data in user queries. Using query representations and machine learning, we can model the tasks that users perform on search systems (Section 2.4), as well as other search patterns, including the segmentation of query logs (Section 2.5), that enable the development of several user assisting applications and retrieval models to support users while they perform their information seeking.

Search query log datasets

Multiple research works (Thomas et al., 2020;Wang et al., 2013a,b;[START_REF] Zhang | Generic intent representation in web search[END_REF] use proprietary search query logs mainly for two reasons: user privacy and commercial concerns that could arise when other search systems exploit the information (Craswell et al., 2020a). However, there are some publicly available datasets that we can leverage to model search behaviors (Table 2.1).

SogouQ5 released the SogouQ User Query Log (SUQL) [START_REF] Liu | How do users describe their information need: Query recommendation based on snippet click model[END_REF]. The Lite version from June 2008 has 1,724,264 queries corresponding to around one day of search activities. Queries were registered from 519,876 user identifiers and are mostly in Chinese (Table 2.2). A subset of the SUQL collection with search tasks labels (SUQLST) contains 18,600 queries from 532 users [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]; however, SUQLST is not publicly available for download.

AOL6 released the AOL Query Log (AOLQL) in 2006 [START_REF] Pass | A picture of search[END_REF]. It contains 36,389,566 queries from 657,426 users, collected over a period of around three months [START_REF] Gayo-Avello | A survey on session detection methods in query logs and a proposal for future evaluation[END_REF]. Multiple subsets from the AOLQL collection has been released, with ground-truth labels for topical sessions [START_REF] Gayo-Avello | A survey on session detection methods in query logs and a proposal for future evaluation[END_REF][START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF] and search tasks [START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]Völske et al., 2019).

The AOLQL subset with topical sessions (AOLTS) [START_REF] Gayo-Avello | A survey on session detection methods in query logs and a proposal for future evaluation[END_REF][START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF] contains 10235 queries issued by 215 unique users. Labels for the queries in the dataset correspond to topical sessions, which are groups of successive queries related to the same information need [START_REF] Gayo-Avello | A survey on session detection methods in query logs and a proposal for future evaluation[END_REF]. Topical sessions are independent of information needs. For instance, the user with identifier 1713103 issued the query "ebay.com", which has the topical session '733'. The same user issued the query "ebay" later on, after issuing queries with other information needs like "club pogo" or "metal detecting". Nevertheless, the topical session label for the query "ebay" changed to '737' even though it reflects the same user intent as the query "ebay.com".

Dataset

Reference No. of queries Labels SUQL [START_REF] Liu | How do users describe their information need: Query recommendation based on snippet click model[END_REF] 1724264 None SUQLST (Du et al., The Webis Search Mission Corpus 2012 (WSMC12) dataset [START_REF] Hagen | From search session detection to search mission detection[END_REF] has 8840 entries with 2881 search task labels of 127 users. It is a subset of the AOLQL collection. Labels in this dataset correspond to search tasksknown as search missions -for a particular user identifier, grouping queries by user. If two users run a query for the same search task, the search task label is not necessarily the same. For instance, the user with identifier 9887420 issued the query "maps" and it was labeled with search task '2'; the user with identifier 1713103 issued the query "maps" as well, but it was labeled with search task '22'. Overall, both queries represent the same search task because user intents reflect maps websites, as the clicked URLs show: the clicked URL for the first case was maps.google.com while the clicked URL for the second case was maps.yahoo.com.

The Cross-Session Task Extraction (CSTE) dataset [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] is a subset of the AOLQL collection. The CSTE dataset has 1424 entries with 224 labels corresponding to cross-session search tasks, without grouping queries by user information or query timestamps. Because of this, ground-truth labels are independent from users and time sessions. By contrast, the Time Gap Session with Search Tasks (TGSST) dataset [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF] has the same queries than the CSTE dataset, but the 1424 entries are grouped by user identifiers and time sessions. Time sessions are clusters of chronological queries with a time span of less than 26 minutes between subsequent query pairs. Every time session has its own search task labels. If queries with the same tasks happens in different time sessions, they could have distinct search task labels. For instance, the query "precious momunts", run by user with identifier 117514, has a search task label of '3' for time session 3, while the query "precious momounts", run by the same user, has a search task label of '1' for time session 5.

Combining search queries and user clicks, the Open Resource for Click Analysis in Search (ORCAS) (Craswell et al., 2020a) contains 18.8M clicked document query pairs for 10.4M unique queries. Entries in ORCAS comprise query ID, query text, document ID, and document URL. The document IDs and URLs come from the TREC deep learning track document collection [START_REF] Craswell | Overview of the TREC 2019 deep learning track[END_REF]; by doing so, it is possible to avoid revealing the ranking mechanism of the search system. Queries are anonymized to protect user information. Queries are also processed to remove sensitive information like adult content or offensive words.

The Webis Query-Task-Mapping Corpus 2019 has three datasets with search task labels (Völske et al., 2019), where commercial search engines 78 provide suggested queries to perform data augmentation. The AOL-based Query-Task-Mapping (AOLQTM) dataset has 41780 queries and labels for 1423 search tasks. The TREC-based Query-Task-Mapping (TRECQTM) dataset has 47514 queries with labels for 276 search tasks. And the WikiHowbased Query-Task-Mapping (WHQTM) dataset has 119292 queries with labels for 7202 search tasks.

The Complex User Search Task Analysis (CUSTA) dataset [START_REF] Dosso | How to support search activity of users without prior domain knowledge when they are solving learning tasks[END_REF] comes from a study of user interactions with search systems when solving complex search tasks. The study included search tasks in computer science, psychology, and medicine, with questionaries to assess if users had experience in the field, as experienced users in a particular field already have several abstractions in that field. They also can use analogies to solve problems [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF][START_REF] Oakley | A mind for numbers: How to excel at math and science (even if you flunked algebra)[END_REF], making it easier to solve complex search tasks. Five types of learning tasks were evaluated: simple tasks, learning tasks, decision tasks, problem-solving tasks, and multicriteria tasks. Queries in the dataset are mostly in French (Table 2.2), totaling 2390 entries for 32 users.

Dataset

Query Language

Query Timestamps

Clicked URLs

Ground-truth labels AOLQTM, TRECQTM, and WHQTM are benchmark datasets for testing methods that map user queries to search tasks. Table 2.3 include reference results for the three datasets, considering Word Movers Distance (WMD), Min-Hash Locality-Sensitive Hashing (MinHash LSH), the Trie data structure 9 , which is the fastest mapping method, and ElasticSearch BM25 10 , which is the most accurate one (Völske et al., 2019).

SUQL Chinese √ √ X AOLTS English √ X √ AOLQTM English X X √ WSMC12 English √ √ √ CSTE English X X √ TGSST English X X √ TRECQTM English X X √ ORCAS English X √ X WHQTM English X X √ CUSTA French X X √
Though AOLQTM, TRECQTM, and WHQTM have ground-truth labels for search tasks, they do not have query timestamps. Thus, for models of search behaviors requiring the computation of time spans between queries, WSMC12 offers both query timestamps and ground-truth labels for search tasks (Table 2.2). Query timestamps in WSMC12 are also essential when it is required to determine whether two queries are adjacent. Finally, as search task labels in WSMC12 are not user-independent, models for search tasks can 10 2 Background leverage the CSTE dataset, which has user-independent ground-truth labels. CSTE contains, however, user queries in English only, so the CUSTA dataset can complement CSTE when testing search task models in multilingual setups.

Dataset

Machine learning

Given the large size of search query logs and the high dimensionality of their information, automatic analysis methods are crucial to model user behaviors from query logs. Automatic methods can extract models from query logs in order to make descriptions or predictions of future user needs and interactions. As machine learning provides methods to process large and high dimensional data in an efficient manner [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF], machine learning enables the modeling of user behaviors from their interactions with search systems.

Existing query logs with ground-truth labels provide an input -output relationship that the machine learning method can discover through the learning process [START_REF] Alpaydin | Introduction to machine learning[END_REF]. This is known as supervised learning, where a set of inputs is mapped to a certain output, and the parameters of the system are changed to improve the matching of the model. Nonetheless, the need for input-output samples represents a challenge for supervised learning models.

They usually need big datasets that are cleaned and labeled by humans. In most datasets, there is a long tail problem: common categories have high probabilities, providing large counts of samples. But there are a lot of categories with few or no samples. Also known as edge cases, they are unlikely but possible. These cases are challenging for machine learning models and can easily confuse them. To deal with such scenarios, a possibility is to train the models with common cases and let them learn the rest with unsupervised learning [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF].

In unsupervised learning, there is a set of inputs but no expected output, thus, the model should extract patterns from the inputs to create valid parameters. As unsupervised learning approaches do not need labels, this is a viable alternative. Examples of unsupervised learning include models that group similar items using abstract similarities or learn new categories by analogy to existing ones [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF][START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF]. Another type of machine learning is reinforcement learning, which is based on a system of punishments or rewards for the outputs generated, which tailor the system for appropriate modeling [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF].

Furthermore, machine learning models can be parametric or non-parametric. Parametric models have a fixed number of parameters in spite of the input dataset size. They are computationally faster but less flexible. Parametric models could be relatively simple like logistic regression or more complex like deep neural networks. Conversely, In non-parametric models, the number of parameters depends on the dataset. They are more flexible; however, large datasets can make them computationally intractable. Also, non parametric models could be difficult to scale because they grow as the dataset grows. An example of a non-parametric machine learning method is k-nearest neighbor (KNN). The label of one data point depends on the class with the higher number of neighbors. K defines the complexity of the model. A low k has a complex decision surface while a high k has a smoother decision surface, usually generating as output the class with the higher data density [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF].

Supervised models

In supervised learning, the classical error rate in pattern classification -the classification error rate -enables the computation of a loss function for the models. The error rate is the relationship between correct input-target outputs and the testing set size [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF]:

E = 1 |S | (x,z)∈S 0 if h(x) = z 1 otherwise (2.1)
where S is the test set, x is the input, z the expected output, and h is the learning algorithm.

Traditional supervised machine learning algorithms depend heavily on the way input data is represented. Each attribute of the proposed representation is known as a feature. It usually requires considerable time and effort to find the correct way to represent input data in order to get a good performance in the machine learning method. The process also needs high domain specific knowledge to extract the most important attributes of the input dataset. But automatically learned representations usually have a better performance than handcrafted feature extraction. When methods learn to represent input data by themselves, we are talking about representation learning. Deep learning models are a kind of representation learning. They learn not only the mapping between features and results, but the representation of input data itself [START_REF] Goodfellow | Deep Learning[END_REF].

Therefore, deep neural networks provide a way to infer models from the raw data by extracting hidden structures from the information. They also automatically extract the set of attributes that best represents the input dataset -improving the learning process by encoding representations as a nested hierarchy of simpler or less abstract representations [START_REF] Goodfellow | Deep Learning[END_REF][START_REF] Rampasek | Tensorflow: Biology's gateway to deep learning?[END_REF]. Very similar deep neural network architectures have worked for various case studies in not related domains, which proves the suitability of deep learning models to adapt to various problem domains. At the same time, it proves the capacity of deep neural models to automatically extract features from data in spite of dataset domain origin. It also shows the ability of deep neural networks to learn non-linear transformations in their hidden structures and to generate distributed representations of input datasets [START_REF] Goodfellow | Deep Learning[END_REF][START_REF] Angermueller | Deep learning for computational biology[END_REF].

Common deep learning architectures include Multilayer Perceptrons, Convolutional Neural Networks, and Recurrent Neural Networks [START_REF] Jouppi | In-datacenter performance analysis of a tensor processing unit[END_REF].

Multilayer perceptrons (MLPs) represent models of artificial neural networks that are useful for classification and regression tasks. They are also known as feedforward networks. The perceptron is the basic processing unit of the MLP. It comprises a series of weights that transform the input nodes through a mathematical operation. Then, an output node computes the result by combining such transformations into a single value. If a system uses only one perceptron, it can only approximate linear functions of input data. Thus, it only performs linear regression. But real applications usually require nonlinear regressions. MLPs implement hidden layers of perceptrons between input and output nodes. Those hidden layers enable the approximation of nonlinear functions for the input data. An MLP with a single hidden layer and enough nonlinear nodes -processing units -can approximate any continuous function in a compact domain. Therefore, they are considered universal function approximators. [START_REF] Alpaydin | Introduction to machine learning[END_REF][START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF].

Both hidden layers and output nodes in MLPs have nonlinear activation functions. Once one or more input nodes are activated, the information propagates through the whole set of interconnected nodes in the subsequent layers, up to the output layer. Such propagation is known as a forward pass. The MLP response does not depend on the past or future inputs, it only depends on the current input. Because of this, they are a good alternative for pattern classification tasks [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF].

Convolutional neural networks (CNNs) are models inspired by biological visual systems. Specifically, they are inspired by the visual cortex of the brain, where a combination of simple and complex neurons interact to build the powerful natural visual system. CNNs comprise convolution layers, nonlinear layers, and pooling layers. These models are quite successful nowadays for their ability to process spatial and multidimensional information [START_REF] Min | Deep learning in bioinformatics[END_REF]. Most of the record-breaking applications of these neural networks are part of the machine vision area, including semantic segmentation, object recognition, image classification, and image retrieval use CNNs [START_REF] Angermueller | Deep learning for computational biology[END_REF][START_REF] Dolz | 3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study[END_REF].

Current advances in parallel computing, optimization techniques, and network architectures have enabled recurrent neural network (RNN) applications in large-scale deep learning problems [START_REF] Lipton | A critical review of recurrent neural networks for sequence learning[END_REF]. They are inspired by the cyclical connection of the neurons inside our brain. Neural networks with cyclical connections also include recursive and feedback networks [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF]. Novel applications for recurrent networks include unsupervised video encoding, video captioning, biological sequence analysis, and program execution. Also, they are widely used in the natural language processing landscape [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF][START_REF] Lipton | A critical review of recurrent neural networks for sequence learning[END_REF][START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF].

Recurrent neural networks

An RNN is a type of neural network ideally suited to process sequential information. Sequences appear in multiple domains, including queries in a search log, frames in a video, or words in a sentence. In an RNN, the output for a step in the input sequence depends on the input of that step and information of the preceding steps [START_REF] Rajaraman | Mining of Massive Datasets[END_REF]. These networks are inspired by the cyclical connection of the neurons inside our brain. They store information from input sequences by using iterative function loops [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF].

RNNs process the output in the forward direction, starting from the first step in the sequence, they compute forward hidden states until the last step is reached. However, steps ahead of the current step in the input sequence can provide valuable information to the overall results. Bidirectional RNNs (BiRNNs) were proposed to process the input sequence simultaneously in a forward and backward direction. To do so, they compute forward hidden states and backward hidden states. Concatenating forward and backward hidden states enables the generation of outputs that leverage information from preceding and following steps in the sequence [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF].

To store preceding information, RNNs have a hidden state vector, which works as a memory while the network processes the sequence [START_REF] Rajaraman | Mining of Massive Datasets[END_REF]. In practical setups, the memory in the standard RNN only works effectively with information very close to the step that the network is processing. Thus, Long Short-Term Memory (LSTM) networks were proposed to enhance the performance of standard RNNs with long term information. LSTMs have the ability to save important information, forget information that is not relevant, and focus on the parts of the sequence that better serve the overall network performance. To implement the ability to focus, a two-tier configuration comprises a hidden state vector, which works as the working memory, and a cell state vector, which corks as the long term memory. A popular variant of the LSTM network is the Gated Recurrent Unit (GRU), a much simpler architecture to compute and implement. It relies on a single vector for memory purposes, decreasing the number of parameters needed [START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF][START_REF] Rajaraman | Mining of Massive Datasets[END_REF].

Equations for RNNs

Formally, given an input sequence x 1 , x 2 , . . . x T , equations for a standard RNN are defined as follows [START_REF] Martens | Learning recurrent neural networks with hessian-free optimization[END_REF]:

t i = W hx x i + W hh h i-1 + b h h i = e(t i ) s i = W yh h i + b y y i = g(s i ) (2.2)
where W hx , W hh , W yh are learnable weight matrices, b h , b y are biases, h i are the hidden states, y i are the outputs, e and g represent activation functions. Common activation functions for RNNs are the hyperbolic tangent (Equation 2.3) and the logistic sigmoid (Equation 2.4). Both functions are vector valued functions which are differentiable and non-linear. A differentiable activation function allows the use of gradient descent for neural network training. On the other hand, non-linearity makes neural networks more powerful than their linear equivalents [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF][START_REF] Martens | Learning recurrent neural networks with hessian-free optimization[END_REF].

tanh(x) = e 2x -1 e 2x + 1 (2.3) σ(x) = 1 1 + e -x
(2.4)

The following equations represent an LSTM unit [START_REF] Graves | Generating sequences with recurrent neural networks[END_REF]:

i t = σ(W xi x t + W hi h t-1 + W ci c t-1 + b i ) f t = σ(W xf x t + W hf h t-1 + W cf c t-1 + b f ) c t = f t c t-1 + i t tanh(W xc x t + W hc h t-1 + b c ) o t = σ(W xo x t + W ho h t-1 + W co c t + b o ) h t = o t tanh(c t ) (2.5)
where i is the input gate, f is the forget gate, o is the output gate, and c is the unit cell. b are biases and W are learnable weight matrices. The hidden state of the LSTM is the concatenation of h and c.

GRUs are similar to LSTM units. However, they are simpler [START_REF] Jozefowicz | An empirical exploration of recurrent network architectures[END_REF][START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF]:

r t = σ(W xr x t + W hr h t-1 + b r ) z t = σ(W xz x t + W hz h t-1 + b z ) ht = tanh(W xh x t + W hh (r t h t-1 ) + b h ) h t = z t h t-1 + (1 -z t ) ht (2.6)
where r t is the reset gate, z t is the update gate, b are biases and W are learnable weight matrices.

For the network outputs in classification systems, a softmax function (Equation 2.7) provides normalized output activations that represent the class probabilities [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF].

y j = e yj K k=1 e yj
where j = 1, 2, . . . , K (2.7)

True class probabilities are obtained by representing the true labels with a 1-of-K coding scheme [START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF], a binary vector with one-hot encoding. The cross entropy loss function (Equation 2.8) provides the target function that we minimize in order to train the network. By doing so, we minimize the classification error rate in Equation 2.1.

L = - K k=1 z k ln(y k ) (2.8)
where z represents the true class probabilities and y represents the network output probabilities.

The attention mechanism

To improve the performance of RNN architectures, the attention mechanism [START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF][START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] processes the output of the recurrent layers to focus in the most relevant parts of the sequences. The attention mechanism has been applied in search task identification, neural machine translation, language models, and several other language tasks [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]Vaswani et al., 2017). It creates a context vector c t from a weighted combination of intermediate output states (Equation 2.12) from the recurrent layer. In the global attention [START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF] with general content-based score (Equation 2.10), the weights are stored in an alignment vector a t :

a t = exp(score(h t , hs )) s exp(score(h t , hs )) (2.9) score(h t , hs ) = h T t W a hs (2.10)
where hs are the intermediate output states, h t is the last output state and W a is a learnable weight matrix. The concatenation of the context vector and the final output state into a dense layer generates the output of the attention mechanism:

ht = tanh(W c [c t , h t ]) (2.11) c t = s a t hs (2.12)
The attention mechanism takes as input the recurrent layer outputs. However, it is possible to replace the recurrent layer altogether, using a stack of self-attention layers that process input sequences directly. This stack of self-attention layers is commonly known as a transformer (Vaswani et al., 2017). As the transformer processes the input directly, without the iterations required by recurrent layers, it is faster to train using massive collections of text. Nonetheless, the longer the input sequence, the bigger the transformer becomes, requiring large computational resources for training and inference. Indeed, transformers have a complexity of O(n 2 ) in time and space [START_REF] Murphy | Probabilistic Machine Learning: An introduction[END_REF]. If resources are constrained, it is necessary to trim the input or use small batch sizes for training. Another alternative is to use long-range transformer architectures, which are specifically designed to process long sequences (Tay et al., 2020). Nowadays, transformers are widely used in natural language models and computer vision [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF][START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF][START_REF] Khan | Transformers in vision: A survey[END_REF][START_REF] Radford | Improving language understanding by generative pre-training[END_REF][START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF].

Unsupervised models

The need for large datasets that are cleaned and labelled by humans is challenging [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF][START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]Wang et al., 2013a). As mentioned before, a possibility is to train the models with common cases and let them learn the rest with unsupervised learning. As unsupervised learning approaches do not need labels, this is a viable alternative. In an unsupervised approach, there is a density estimation for the input data. No output is provided, thus, the method should discover useful patterns from the available input [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF][START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF].

Examples of unsupervised learning include dimensionality reduction and clustering models that group similar items using abstract similarities or learn new categories by analogy to existing ones [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF]. Dimensionality reduction aims to extract the most important latent information in the data to better describe the original data with a lower number of dimensions. Examples include Principal Component Analysis (PCA) and Latent Semantic Analysis (LSA), which uses global matrix factorization for document retrieval [START_REF] Dumais | Latent semantic analysis[END_REF][START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF][START_REF] Pennington | GloVe: Global Vectors for Word Representation[END_REF].

Clustering is a canonical example of unsupervised learning. Here, the cluster is not part of the input data, thus, it is a latent or hidden variable. Two goals comprise a clustering task: first, we need to find the number of clusters which better fits the existing data. Secondly, we want to know which cluster each data point pertains to [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF]. Clustering methods are essential in multiple data-driven applications. They are primarily based on partitioning, density, and hierarchies [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF]. An example is Latent Dirichlet Allocation (LDA) [START_REF] Blei | Latent dirichlet allocation[END_REF], which is a probabilistic generative model which has been successful in clustering documents into topics by using the co-occurrence of common words in them [START_REF] Li | Exploring query auto-completion and click logs for contextual-aware web search and query suggestion[END_REF].

Another type of clustering uses graphs, representing each sample in the input dataset as a node and using similarities between samples as weights in the edges of the graphs [START_REF] Chen | Graph-based clustering for computational linguistics: A survey[END_REF][START_REF] Nascimento | Spectral methods for graph clustering-a survey[END_REF][START_REF] Nie | The constrained laplacian rank algorithm for graph-based clustering[END_REF][START_REF] Wang | Scalable graph-based clustering with nonnegative relaxation for large hyperspectral image[END_REF]. This approach is commonly used for processing query log datasets [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Lucchese | Identifying task-based sessions in search engine query logs[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF].

Along with dimensionality reduction and clustering, there is another unsupervised method specifically designed to train neural network architectures. Though it is possible to pretrain neural architectures using labeled datasets, there is an unsupervised alternative that trains neural architectures without the need for ground-truth labels. Commonly known as self-training, this unsupervised alternative allows neural architectures to optimize for a certain objective that can be automatically calculated without manual labels [START_REF] Karamanolakis | Self-training with weak supervision[END_REF]. For example, the masked language modeling (MLM) [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF] objective allows the training of large language architectures using massive text datasets, without the need for any groundtruth label, which would be challenging to generate for language datasets with millions of samples.

A recent self-training configuration uses a dual-channel architecture [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] and data augmentation techniques to train models in an unsupervised way (Figure 2.1), using only 1% of existing labels. The configuration performs contrastive learning [START_REF] Yang | Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax[END_REF], maximizing the agreement [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Yang | Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax[END_REF] for self-training neural network architectures using data augmentation techniques on input samples. between the output of the encoders situated at each one of the channels. Known as the Simple framework for contrastive learning (SimCLR) of visual representations, it uses a predefined neural architecture for image classification -ResNet [START_REF] He | Deep residual learning for image recognition[END_REF], a set of stochastic data augmentations for image samples, a projection head, and a contrastive learning loss to better learn latent representations of image samples.

The whole SimCLR framework uses two identical branches, modifying the input sample to create two altered versions of the same image. The contrastive crossentropy loss is designed to minimize the distance between the pair of altered versions, comparing the altered version to the remaining altered versions of the minibatch. There is only one positive pair of augmented images in the minibatch. For helping the architecture to avoid learning trivial features and generalize better, the series of data augmentations is applied to a single channel of the dual-channel model. The projection head on top of the ResNet output is a crucial component of the architecture, helping the model to preserve more information in the latent space, which improves the overall accuracy results.

Data augmentations in SimCLR are specifically designed for image datasets. They include stochastic croppings and resizing, color transformations, rotations, and other transformations like Gaussian noise, Gaussian blur, Sobel filter, and so on. Results show it is crucial to combine several image transformations to increase the performance of the architecture. Applying single image transformations yields lower metrics than their application in tandem. Also, bigger and deeper neural architectures have a higher impact in SimCLR than when training the architecture with a supervised learning approach. The rate of improvement is higher as the dual-channel architecture grows, while the rate of improvement for the ResNet architecture alone is slower. Moreover, larger batch sizes and longer training times represent a benefit for SimCLR. This phenomenon happens because the larger batch size and longer training times expose the model to a bigger count of negative samples.

Model evaluation

There are several metrics to assess the performance of machine learning models, depending on the objective we are evaluating. Accuracy defines the relationship between the correct outputs and the total number of outputs (Equation 2.13). However, accuracy by itself could be a poor estimator of model performance in some cases. A more robust approach includes the calculation of precision and recall. Precision defines the fraction of detections reported by the model that is correct. On the other hand, recall defines the fraction of true events in the testing set that were successfully detected by the model [START_REF] Goodfellow | Deep Learning[END_REF]: A common summarization of model performance combines precision p and recall r into the F β score [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]:

Accuracy = T P + T N T P + F P + T N + F N (2.
F β = (1 + β) 2 pr β 2 p + r (2.16)
where β is the factor that controls the weight of precision and recall. When β = 1.0, we have a balanced F -score [START_REF] Goodfellow | Deep Learning[END_REF]. But in certain cases, we would like to give more weight to precision or recall in the F -score calculation, thus, when β < 1.0, the precision has more weight in the F β score, while a β > 1.0 gives more weight to recall in the F β score.

In the case of unsupervised learning models, when ground-truth labels are available, we compute the accuracy by maximizing the match between the labels predicted by the model and the ground-truth labels [START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF]:

ACC = max m n i=1 1 {y i = m(c i )} n (2.17)
where y i is the ground-truth label, c i is the label predicted by the model, and m is the matching function that ranges over all possible mappings between ground-truth and generated labels. For maximizing the matching function, the Hungarian algorithm enables the computation of the optimum mapping between predicted labels and ground-truth labels [START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF][START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF].

As far as the F β score is concerned, we also calculate it with equation 2.16, but we use a pairwise approach to compute T P , F P , T N , and F N . What matters in the pairwise approach is the relationship between the predicted labels and the ground-truth labels of a given pair of samples. For a given pair of input samples, if the predicted labels pertain to the same group and the ground-truth labels also pertain to the same group, then we have a T P . If the predicted labels do not pertain to the same group and the ground-truth labels do not pertain to the same group either, then we have a T N . If the predicted labels pertain to the same group and the ground-truth labels do not pertain to the same group, then we have a F P . Finally, if the predicted labels do not pertain to the same group and the ground-truth labels do pertain to the same group, then we have a F N [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF].

Other metrics for unsupervised learning models include the Normalized Mutual Information (NMI) and the Adjusted Rand Index (ARI) [START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF]. Considering U as the ground-truth labels and V as the predicted labels, NMI is calculated as follows (Vinh et al., 2010):

N M I(U, V ) = 2I(U, V ) H(U ) + H(V ) (2.18)
where I(U, V ) represents the mutual information and H is the entropy. While NMI is an information theoretic based metric, ARI is a paired counting based measure. ARI is calculated as follows (Vinh et al., 2010):

ARI(U, V ) = 2(N 00 N 11 -N 01 N 10 ) (N 00 + N 01 )(N 01 + N 11 ) + (N 00 + N 10 )(N 10 + N 11 ) (2.19)
where N 00 are the number of pairs that are in different clusters in both U and V, N 01 are the number of pairs that are in the same cluster in U but in different clusters in V, N 10 the number of pairs that are in different clusters in U but in the same cluster in V, and N 11 the number of pairs that are in the same cluster in both U and V.

Language representation

When processing search query logs with machine learning models, the language data in user queries needs to be represented in a vector space. But language is hard for computers to understand because it heavily depends on context, it requires a large amount of background knowledge common to the parts communicating for proper understanding, and it has an inherent ambiguity [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF]. First attempts to encode language data used onehot encodings for a large vocabulary. However, one-hot representations can not encode semantic relationships between words [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF]. Similarly, Bag-of-words [START_REF] Zhang | Understanding bag-of-words model: a statistical framework[END_REF] text representations could pose problems when encoding texts that are lexically similar but have different semantic content. They can also miss texts with different words that have the same semantic meaning [START_REF] Zhang | Generic intent representation in web search[END_REF].

Therefore, distributed representations were proposed to improve text encodings from a semantic standpoint. Distributed representations can learn to map semantically related words very close in the word embeddings space by leveraging the fact that semantically similar words appear in similar surroundings [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF][START_REF] Zhang | Generic intent representation in web search[END_REF]. According to the distributional hypothesis, the meaning of a word is related to the context in which it appears. In the field of linguistics, you can know a word by the context accompanying it. More precisely, the linguist John Firth said "You shall know a word by the company it keeps". This is formally known as distributed semantics: the semantic similarity of two linguistic expressions is related to the similarity of the contexts in which they appear [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF]. Distributed semantics also affect the quality of word vectors. The quality of word vectors depends on the quantity and quality of text datasets used to learn the word vector models [START_REF] Grave | Learning word vectors for 157 languages[END_REF][START_REF] Bojanowski | Enriching word vectors with subword information[END_REF].

Furthermore, a word can have different meanings depending on its context; thus, if we use a vector to represent it, in certain dimensions of that vector the word is close to words with a similar meaning, while in other dimensions, it should be closer to other words that share another meaning [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF]. For that reason, word vectors capture the multiclustering concept from distributed representations [START_REF] Bengio | Learning deep architectures for ai[END_REF][START_REF] Pennington | GloVe: Global Vectors for Word Representation[END_REF].

Word vectors

Based on the distributed semantics concept from linguistics and a multidimensional representation of words using vectors, word2vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] uses a one-hot encoding for the input and output vocabulary, along with an intermediate feedforward layer with 300 units to learn a word vector for each input in the dictionary. The training method for the model scans millions of texts to produce pairs of words that appear in documents. The scanning ignores words like prepositions to get more meaningful pairs. It also considers the pair of words both in the original order in which they appear in the text and the inverse order. The feedforward layer learns to predict the next word of the pair. Once it is trained, the neural layer produces a 300 dimensional word vector for each input word in the vocabulary. The word2vec multidimensional representation captures semantic relatedness, as points in space representing words with similar meanings appear nearby [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF].

Word2vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] has two types of training objectives: Skip Gram and Continuous Bag-of-Words (CBOW). Both objectives rely on surrounding context to model the word embeddings space. Skip Gram predicts a word's context from the word itself, while CBOW predicts the word from its context. To define the context, the training objectives consider a local window of words. From that context, a matrix is learned to minimize the hinge loss between related words and non-related words [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF][START_REF] Zhang | Generic intent representation in web search[END_REF]. Both Skip Gram and CBOW produce quantitatively and qualitatively similar word vectors [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Nalisnick | Improving document ranking with dual word embeddings[END_REF] NLP applications using Word2vec representations normally discard the output layer, using only the input vectors to compute semantic representations of text [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF][START_REF] Nalisnick | Improving document ranking with dual word embeddings[END_REF]. However, output vectors have important semantic relations with words that appear the texts to be transformed because the input-output dataset for training word embeddings depends on the context of the input words [START_REF] Nalisnick | Improving document ranking with dual word embeddings[END_REF]. Output vectors can be useful in certain scenarios. For instance, Dual Embedding Space Model (DESM) [START_REF] Nalisnick | Improving document ranking with dual word embeddings[END_REF] uses input word embeddings to represent the query words and output word embeddings to represent the documents in the index, using the cosine similarity to compute a relevance metric for the indexed documents. Documents are represented by the centroid vector of the averaged vectors that encode the sentences, which can be precomputed to decrease latency. The relevance depends on the cosine similarity of the word embeddings in the query and the centroids encoding the documents. The best performing embeddings are trained using a large collection of queries from Bing, using the CBOW model. The DESM model outperforms the BM25 index baseline as well as Latent Semantic Analysis (LSA) results.

The use of local context windows in word vectors like Word2Vec has one major limitation: it overlooks the global co-occurrence statistics from the training corpus. This is a major concern because of the amount of repetition that text datasets tend to contain. Because of this limitation, GloVe [START_REF] Pennington | GloVe: Global Vectors for Word Representation[END_REF] word vectors combine the local window context of Word2Vec with the global statistics present in matrix factorization methods like LSA. The combination proves effective, allowing GloVe vectors to improve on word vectors based on local context windows. Improvements are observed in word analogy tasks, similarity tasks, and named entity recognition.

As machine learning architectures usually require inputs in the form of vectors, word vectors like Word2Vec and GloVe -also known as word embeddings -are ideal for approaches that rely on machine learning models. Word vectors encode syntactic and semantic information in a continuous vector space, facilitating multiple language tasks [START_REF] Grave | Learning word vectors for 157 languages[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. Several variations of word vector representations have been proposed and they are now widely used in various successful NLP applications [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF]. Furthermore, pre-trained word vectors are now available in multiple languages [START_REF] Grave | Learning word vectors for 157 languages[END_REF]. However, an initial phase for language selection [START_REF] Kiela | Dynamic meta-embeddings for improved sentence representations[END_REF] or the use of automatic integration of models [START_REF] Conneau | Word translation without parallel data[END_REF][START_REF] Lample | Unsupervised machine translation using monolingual corpora only[END_REF] are necessary.

Moreover, when the input for the machine learning model is a sentence, we might want to compute a single vector representation per sentence. In this case, individual word vectors could be averaged, a representation approach used in several recent studies [START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF][START_REF] Saini | Automatic scientific document clustering using self-organized multi-objective differential evolution[END_REF][START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF]Zamani and Croft, 2016a,b). Another alternative is the use of convolutional neural networks to produce one vector per sentence [START_REF] Kim | Convolutional neural networks for sentence classification[END_REF].

Custom word vectors

In spite of the way word vectors map semantically and syntactically related words, they still pose problems when capturing fine grained details from sentences. In addition, they can suffer from topic shifting [START_REF] Rekabsaz | Word embedding causes topic shifting; exploit global context[END_REF][START_REF] Zhang | Generic intent representation in web search[END_REF].

To address such limitations in distributed word representations, it is possible to train the word vectors using search-related signals like relevance feedback [START_REF] Zamani | Relevance-based word embedding[END_REF][START_REF] Diaz | Query expansion with locallytrained word embeddings[END_REF][START_REF] Zhang | Generic intent representation in web search[END_REF]. Using top retrieved results from user queries, a recent work [START_REF] Zamani | Relevance-based word embedding[END_REF] proposed a query embedding that relies on the relationship between terms that appear both in the query and the resulting documents.

Another possibility is to fine tune word vectors for specific search related tasks [START_REF] Diaz | Query expansion with locallytrained word embeddings[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]. The tempo-lexical word embedding approach [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] relies on an embedding mechanism that incorporates both temporal and lexical information. This word embedding uses the task context to find query representations in a semantic space oriented to search tasks. One vector represents each query in the semantic space. The tempo-lexical word embedding is used in a global approach to identify users tasks in search logs. It takes into account the whole query log to train the word embedding, improving the subsequent results in the clustering of crosssession queries related to the same task.

Word embeddings could work for long documents but they might not reflect the context in queries, where short sentences or keywords are utilized by users to look for information. Thus, the number of words can be low and the window for context can not include the amount of information it has in long documents. In a temporal context, a 26-minute window [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF] provides the context to train the word vectors. In the tempo lexical case, additional clustering of queries according to their relatedness helps to improve the space for learning the representations [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF].

Universal language models

Often, there is not enough data to custom train end-to-end word vector models. Also, some NLP problems are not supervised. Thus, models focused on creating universal representations were proposed [START_REF] Zhang | Generic intent representation in web search[END_REF]. Such models include Bidirectional Encoder Representations for Transformers (BERT) [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF], Embeddings from Language Models (ELMo) [START_REF] Peters | Deep contextualized word representations[END_REF], and Generative Pretraining (GPT) [START_REF] Radford | Improving language understanding by generative pre-training[END_REF], now in its third version [START_REF] Brown | Language models are few-shot learners[END_REF]. Some universal language models like GPT process input language sequences in only one direction [START_REF] Radford | Improving language understanding by generative pre-training[END_REF]. Instead, ELMo uses two separate recurrent models for both directions and then join the results [START_REF] Peters | Deep contextualized word representations[END_REF]. It uses hidden states from right-to-left (RTL) and left-toright (LTR) sections of the recurrent neural network. The hidden states are then concatenated for subsequent steps, providing a bidirectional approach for processing the input. BERT also takes a bidirectional approach, taking advantage of context on both backward and forward directions, but in contrast to ELMo, it uses a transformer-based architecture. The BERT architecture is based on a bidirectional transformer and its results prove it outperforms existing alternatives in various natural language processing tasks. Results also highlight the importance of creating an architecture, pre-training it, and then performing transfer learning by fine-tuning it to perform different natural language tasks [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF]. Indeed, model pre-training for tasks based on NLP has proven instrumental in a number of applications, including natural language inference, paraphrasing, named entity recognition, and question answering. There are two trends. In the first trend, pretraining is performed in order to extract features from language. Feature extraction architectures focus on generating representations for other supervised tasks in the processing pipeline. Word vectors [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF][START_REF] Grave | Learning word vectors for 157 languages[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]) take advantage of this approach. There are word embeddings approaches with broader scopes, extracting features from sentences or paragraphs. Another scope is context. Models like ELMo [START_REF] Peters | Deep contextualized word representations[END_REF] and tempo-lexical word vectors [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] leverage context to generate word vectors from text data [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF] The second trend performs transfer learning, fine-tuning models for specific tasks. Transfer learning approaches leverage fine-tuning to perform NLP Tasks. This approach starts by training supervised models with a certain objective, usually with abundant labeled data available, and then perform transfer learning and fine-tune the model for the target goal. Although unsupervised pretraining for models leverages huge amounts of unlabeled data, supervised pretraining has also proven useful in various fields, including natural language tasks and computer vision applications [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF].

Together with pretraining, BERT uses several components as input to the model. The input can include three components [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF]:

1. Embeddings from tokens, where a CLS token marks the beginning of a sentence, and a SEP token marks the separation for cases when the pretraining requires two sentences 2. Embeddings for sentences, A for one sentence, A -B for two sentences 3. Positional embeddings for learning the position of the token with respect to the whole sentence Few additions are stacked on top of the BERT architecture for fine tuning specific tasks. Tasks include both one or two sentences for inference. For sentence labeling, the architecture takes as input the output tokens and learns to classify them. For sentence classification, the fine tuning is performed over the output of the CLS token. The same happens for pairs of sentences, where the CLS token output lets the fine tuning system learn sentiments, relatedness, etc. Another alternative is to use BERT output as features for further processing. In this case, although there is a loss in metrics with respect to the fine-tuned model, the best working configurations concatenate the last four layers outputs [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF].

Results show that BERT outperforms similar architectures in all tasks considered for evaluation, including the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019a). Unsupervised pretraining has a big impact on BERT final results. The bidirectional approach to process the input texts has a significant impact in the final results as well. Furthermore, bigger language models create incremental improvements when there are larger datasets. BERT helps problems with small datasets to also get improvements thanks to model pretraining. Similarly, as language models are less sensitive to hyperparameters when there are large training datasets, BERT parameters create a larger impact on final language model results when there is a small dataset in a fine-tuning configuration [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF].

Language models for sentences

Universal Sentence Encoder (USE) [START_REF] Cer | Universal sentence encoder for english[END_REF] learns universal representations using multitask learning on language semantic tasks [START_REF] Zhang | Generic intent representation in web search[END_REF]. To produce sentence embeddings, USE can have two configurations. The first configuration is more accurate but more complex computationally. It is based on the encoding part of the transformer architecture. Context-aware word embeddings are generated by the encoder; then, they are averaged together to produce one vector per sentence. As mentioned before (Section 2.2.1), the computational complexity is O(n 2 ) in sentence length, making it more computationally expensive as the length of the sentence grows. The second configuration averages together word embeddings and bi-gram embeddings. The resulting average vector feeds a Deep Average Network (DAN), a feedforward network which computes a fixed vector representation for the input sentence. It is less computationally expensive than the transformerbased model. Its computational complexity is linear with respect to sentence length. However, the transformer-based model is more accurate.

The training of the universal sentence encoder uses a multitask approach to improve model performance. A supervised training phase with the Stanford Natural Language Interface (SNLI) augments the unsupervised training, which relies on sources like Wikipedia, web news, discussion forums, etc. To test the model, several natural language processing tasks are utilized in a transfer learning setup, proving the ability of the architecture even when small labeled datasets are available. For transfer learning experiments, a taskspecific neural network takes as input the sentence embeddings. Furthermore, the semantic textual similarity between pairs of sentences uses the arccos to compute the angle between the embeddings. The angular distance between two sentence embeddings has been found to work better for pairwise semantic similarity tasks than the cosine similarity [START_REF] Cer | Universal sentence encoder for english[END_REF].

Most universal language models [START_REF] Cer | Universal sentence encoder for english[END_REF][START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF][START_REF] Radford | Improving language understanding by generative pre-training[END_REF] use contextual information to learn universal representations. But it is not clear how those generic representations relate to user intent [START_REF] Zhang | Generic intent representation in web search[END_REF]. Instead, Generic Intent Representation (GEN) creates a distributed space to represent queries modeling the user intent -what the user wants to get from the query. It learns from user clicks in retrieved documents, a weak supervision signal strongly related to the user intent. Training the model with query paraphrasing and question paraphrasing datasets help to improve the generalization abilities of the model [START_REF] Zhang | Generic intent representation in web search[END_REF]. Applications of the GEN [START_REF] Zhang | Generic intent representation in web search[END_REF] encoder include the generation of more relevant results, better query suggestions, and more precise answers. It also helps to deal with the long-tail effect in search engines and model query reformulation behaviors.

The GEN [START_REF] Zhang | Generic intent representation in web search[END_REF] architecture comprises three branches: word embeddings, character embeddings, and a mix encoder. The word embedding maps words to a continuous vector space. The character embedding deals with rare words. The mix encoder composes word embeddings into query encodings. It uses a BiGRU to learn cases in which the ordering of words has an impact on the query intent. Both the word and character embedding parts of the architecture include a highway neural layer to improve model capacity. The character embedding uses a CNN architecture and max-pooling before the highway layer.

The learning is performed end-to-end. It has two phases. In the first phase, it utilizes user co-clicks to learn user intent. The loss for query co-clicks is designed to minimize the distance between related query pairs while maximizing it between negative query pairs. Co-click queries are sampled from a large log. To avoid trivial negatives, the model uses noise-contrastive estimation (NCE) to get good adversarial samples. During the second training phase, the model uses a multitask approach, which combines the query co-clicks weak supervision loss along with a query paraphrasing loss and a question paraphrasing loss to improve the generalization ability of the model [START_REF] Zhang | Generic intent representation in web search[END_REF].

For evaluation purposes, an intrinsic user intent strategy is based on three datasets with increasing levels of difficulty: general, easy, and hard. The normalized discounted cumulative gain (NDCG) is the metric for query intent evaluation. The area under the curve (AUC) provides performance metrics for paraphrasing tasks. GEN consistently outperforms USE [START_REF] Cer | Universal sentence encoder for english[END_REF] sentence representation, the best performing encoding of all the baselines considered for experiments. In turn, USE gets better results than the best performing information retrieval based representation: the Relevance Language Model (RLM) [START_REF] Zamani | Relevance-based word embedding[END_REF].

Moreover, an ablation study of the GEN encoder is conducted to evaluate the multiple components of the proposed model. The most important conclusion of the ablation study is the importance of user clicks in training the model. Simple models trained with user click get results very close to state of the art text encodings, while complex methods failed to outperform Term Frequency -Inverse Document Frequency (TF-IDF) [START_REF] Robertson | Understanding inverse document frequency: on theoretical arguments for IDF[END_REF]. Thus, the supervision signal is more important than the architecture used to learn the model [START_REF] Zhang | Generic intent representation in web search[END_REF].

Multilingual language models

Multilingual Universal Sentence Encoder (MUSE) [START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF] extends previous work [START_REF] Cer | Universal sentence encoder for english[END_REF] to provide sentence embeddings suited to information retrieval tasks in a multilingual semantic space. It supports 16 languages and comprises three configurations: a CNN based model, a transformer based model, and a question answering model. All configurations use SentencePiece [START_REF] Kudo | SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF], a language independent subword tokenizer to process the input sequence. SentencePiece covers above 99% of possible tokens in all languages. The sentence encoder uses a dual encoder architecture. The first encoder generates sentence embeddings. It is shared in all the retrieval tasks, where additional layers process the sentence embeddings to produce the results.

The transformer based model relies on the encoder part of the transformer architecture. It takes into account context-aware embeddings and averages together the results from the encoder to produce one vector per sentence. The CNN based module processes the vectors for tokens using convolutional and average pooling layers, similar to the approach in previous work [START_REF] Kim | Convolutional neural networks for sentence classification[END_REF]. On the output of the convolutional network, additional dense layers process the average pooling layer results to generate the sentence embedding. The transformer based model is more computationally expensive, with a complexity of O(n 2 ) on sentence length (Section 2.2). CNN has a complexity linear with respect to sentence length, but it is less accurate than the transformer-based model [START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF].

Training is based on question answer pairs, translation pairs, and Stanford Natural Language Inference (SNLI). Google translate is used for the SNLI dataset -which only has english data -and for translating question answer pairs to balance the data in all languages considered. Downstream tasks include sentence retrieval, bitext retrieval, and retrieval question answering. Sentence retrieval returns all the sentences semantically related to the query sentence. Bitext retrieval deals with bilingual sentence pairs. And retrieval question answering process question answer tuples, considering the question as the query and retrieving the answer corresponding to that question in the tuple [START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF].

Language-agnostic BERT Sentence Embedding (LABSE) [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF] provides the sentence embeddings to represent texts in a languageagnostic latent space. Using a 12-layer transformer architecture [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF]Vaswani et al., 2017) in a dual configuration, LABSE takes the transformer's hidden state for the last token in the sentence to generate the query representation.

The training process of LABSE comprises two phases: a pre-training phase and a fine-tuning phase, with a total of 109 languages. The pretraining uses masked language modeling (MLM) [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF] and translation language modeling (TLM) [START_REF] Conneau | Cross-lingual language model pretraining[END_REF] tasks to train the 12layer transformer architecture. MLM masks words in the monolingual input phrases and trains the model to predict the masked word, using 17 billion monolingual phrases. TLM concatenates a phrase in one language with its translation, masking words and training the model to predict the masked words with 6 billion bilingual phrases. After pre-training with MLM and TLM, the LABSE architecture is fine-tuned using the translation ranking task. In translation ranking, the model learns to select the phrase in the target language, representing the best translation for the phrase in the source language [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF].

In cross-lingual text retrieval, using the TATOEBA corpus [START_REF] Artetxe | Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond[END_REF], LABSE outperforms MUSE in the 14 languages that the latter supports. It also outperforms the baseline Language-agnostic Sentence Representations (LASER) [START_REF] Artetxe | Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond[END_REF] in the 82 languages that LASER was trained with. Additionally, it can support languages not included in the training dataset, achieving an average of 83.7% in the 112 languages of the TATOEBA dataset, even though there are more than 30 languages that are not part of the training dataset for LABSE. By contrast, LASER obtains 65.5% in the 112 languages.

Overall, LABSE and MUSE provide representations for user queries in a multilingual vector space. Each query is represented with a single vector and the language coverage varies according to the language model. for representing queries in English only, GloVe leverages both local context and global statistic in language corpus. GloVe vectors can be averaged together to generate a single vector per user query, or used individually for machine learning models requiring one vector per word.

Search task identification

Leveraging language models (Section 2.3) and machine learning approaches (Section 2.2), we can identifiy information seeking patterns in search query logs (Section 2.1). The standard model for the information seeking process establishes certain steps users take when they need to look for information. Such steps include 1) recognizing the need for information, 2) formulating the query according to the information they need, the context, and the time, 3) running the query in a search system to retrieve results, and 4) analyzing the retrieved information and reformulating the query if the results are not satisfactory [START_REF] Hearst | Search user interfaces[END_REF].

To support users during the different steps of the information seeking process, it is crucial to correctly model users' queries according to their search tasks. Information needs could include complex search tasks, where users may need to multitask, shift goals, or iterate to refine queries. Moreover, search history improves personalization of search engine results and provides data to extract predictive models for supporting users while they execute their search tasks [START_REF] Hearst | Search user interfaces[END_REF]. In particular, search system logs provide fine-grained 30 2 Background

Search task identification

Bayesian methods

Graph-based clustering

Semi-supervised methods

Latent Dirichlet Allocation with Hawkes processes (LDA-Hawkes) (Li et al., 2014a) Query clustering based on weighted connected components (QC-WCC) [START_REF] Lucchese | Identifying task-based sessions in search engine query logs[END_REF] Pairwise binary classification [START_REF] Kotov | Modeling and analysis of cross-session search tasks[END_REF] Distance Dependent Chinese Restaurant Process (DD-CRP) [START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF] Query clustering based on head-tail components (QC-HTC) [START_REF] Lucchese | Identifying task-based sessions in search engine query logs[END_REF] Bestlink Support Vector Machine (Bestlink SVM) (Wang et al., 2013a) Bayesian Rose Trees (BRTs) [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF] Context Attention based Long Short-Term Memory (CA-LSTM) [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF] Tempo-Lexical Context driven Word Embedding (QRY-VEC) [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] Fig. 2.2: Recent machine learning models for search task identification, including Bayesian methods, semi-supervised methods, and graph-based clustering approaches.

details at the query level, enabling the characterization and classification of individual queries according to their tasks. A precise task identification improves user supporting applications, helping users to complete their information seeking tasks. Supporting the user while performing complex tasks is essential to avoid the cognitive load on the user. For that support, it is crucial that search systems identify the tasks related to the issued query.

For instance, query term prediction requires the search system to correctly identify the task the user is realizing to properly predict the next likely steps [START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF].

Similarly, the correct clustering of queries by search tasks enables the design of proactive search systems that can suggest next queries after the analysis of user intents. Suggestions help to support the user in the information seeking process, as the user does not have to issue additional related queries [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]. Other applications like query recommendation [START_REF] Boldi | The query-flow graph: model and applications[END_REF][START_REF] Zhao | Recommending queries by extracting thematic experiences from complex search tasks[END_REF], conversational search (Zamani and Craswell, 2020), user modeling based on tasks [START_REF] Hearst | Search user interfaces[END_REF], query -task mapping (Völske et al., 2019), personalization in e-commerce web interfaces [START_REF] Mehrotra | Topics, tasks & beyond: Learning representations for personalization[END_REF], advertisement retrieval, ranking of results according to the task, and prediction of user satisfaction based on tasks [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF], and so on, rely on accurate identification of tasks from user interactions.

Search task identification can be performed with methods using heuristicsbased models [START_REF] Hagen | From search session detection to search mission detection[END_REF]. A model based on a cascade of heuristics [START_REF] Hagen | From search session detection to search mission detection[END_REF] first segments the search query log in logical sessions; then, it performs a post-processing step to detect search tasks -known as search missions -based on the queries pertaining to the logical sessions. However, several manually set thresholds in the heuristics make it challenging to adapt heuristics-based models to other datasets without manually adjusting them. Therefore, machine learning models has been proposed for identifying search tasks (Figure 2.2), including semi-supervised clustering [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Kotov | Modeling and analysis of cross-session search tasks[END_REF]Wang et al., 2013a), Bayesian approaches (Li et al., 2014a;[START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF], and graph-based clustering [START_REF] Lucchese | Identifying task-based sessions in search engine query logs[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF].

Graph-based clustering

To identify queries representing the same search task, the Query Clustering based on Weighted Connected Components (QC-WCC) [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF] is a query clustering based on weighted connected components. QC-WCC does not need the number of clusters beforehand. It relies on the construction of a graph where the nodes correspond to the queries and the edges are weighted according to the similarities between the queries. Every query is a node in the graph. The similarities are based on two features: one contentbased computing a Jaccard index on tri-grams, and the other semantic-based exploiting Wikipedia and Wiktionary to infer the semantics. The graph is then pruned by removing weak edge distances and the query cluster are obtained from the connected components remaining in the graph.

Query Clustering based on Head-Tail Components (QC-HTC) [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF]) is a computationally simpler algorithm based on QC-WCC, although less accurate. It exploits the sequential nature of queries to decrease the computational complexity of the graph based method. The QC-HTC algorithm first builds sequences of queries according to the distance between them, creating the first set of clusters, then takes the first and last queries of a cluster to represent the set and group it with other sets depending on query distances [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF]. Using only head and tail queries in each cluster avoids the computation of the full similarity graph required for QC-WCC, making QC-HTC less computationally expensive.

Tempo-Lexical Context driven Word Embedding (QRY-VEC) [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] improves the QC-WCC [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF] algorithm for search task identification by using word embedding similarities instead of lexically 32 2 Background based similarities. Queries for the same tasks in search logs tend to have a low lexical similarity [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF], which affects methods that rely on lexical similarities or character-level information [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]. Some queries might have no lexical similarity but could be highly related from a semantic standpoint. Indeed, semantic similarity represents a better way to detect queries related to the same task. Queries for the same task clusters tend to be semantically similar rather than lexically similar, as queries in the same tasks contain more synonym words than exact words (Völske et al., 2019;[START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF].

Because of this limitation, instead of relying on lexically based similarities and retrieved documents from the Wikipedia collection [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF], QRY-VEC uses the cosine similarity on a tempo lexical word representation and documents retrieved from the ClueWeb12B [START_REF] Carterette | Evaluating retrieval over sessions: The TREC session track 2011-2014[END_REF][START_REF] Callan | The Lemur project and its ClueWeb12B dataset[END_REF] collection. The ClueWeb12B dataset comprises a more extensive set of web documents. It retrieves information even for terms that are not expository, which are more challenging to match with Wikipedia documents because of vocabulary mismatch. Furthermore, instead of segmenting queries inside time sessions, QRY-VEC identifies global search tasks for clustering. As a baseline, QRY-VEC considers cosine similarities with pretrained word embeddings [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. The tempo lexical word representations surpass the baseline, providing a more accurate method for search task identification.

Semi-supervised clustering

A pairwise approach [START_REF] Kotov | Modeling and analysis of cross-session search tasks[END_REF] uses a binary classification model to determine if query pairs pertain to the same search task. However, a postprocessing step is needed to determine the search tasks in the query log [START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF].

Bestlink SVM (Wang et al., 2013a) first trains a support vector machine to detect if a pair of adjacent queries from a user pertains to the same search tasks or not. Then, it clusters the related queries in the search log using the SVM output to establish links between queries. Bestlink SVM uses a backward context from users' queries to improve the task clustering results. Links between queries are established only with past queries in order to leverage the chronological structure of the log. For the links with past queries, a pairwise parameter allows establishing if two queries should be related or not. For instance, if the retrieved URLs are the same, if the clicked URLs are the same, or if a query is a sub-query of another query, there must be a link between them. No link between queries happens when queries are different, and the intersection of the retrieved URL sets is null. Given that very different queries can be semantically related, they can provide similar sets of related queries. Similar sets allow an automatic labeling scheme for search logs, a significant contribution because manual labeling tends to be expensive and sometimes there are only small annotated logs available. Likewise, the two most important features from the query representation in Bestlink SVM includes the cosine similarity between query embeddings and the similarity between clicked URLs.

Context Attention based Long Short-Term Memory (CA-LSTM) [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF] uses neural networks and graph-based clustering to build a pipeline for task identification. First, it learns to segment search logs in a supervised way. CA-LSTM relies on RNNs instead of SVMs, using both backward and forward queries to provide context while training the RNN to detect if a pair of adjacent queries pertain to the same task or not. Then, it uses the unsupervised QC-HTC clustering method to identify task clusters in the segmented search log.

Bayesian approaches

Latent Dirichlet Allocation with Hawkes processes (LDA-Hawkes) (Li et al., 2014a) combines LDA with Hawkes processes to identify and label search tasks from query logs. LDA performs topic modeling, identifying semantically related queries from different users, while Hawkes processes take into account time lapses between query timestamps in individual query sequences, assigning temporally close queries to the same search task.

Some search tasks are atomic, where a sequence of related queries solves the information needs. However, often information needs are complex. Users perform complex tasks by executing related subtasks, some of which are also complex information needs. Given the relationship between complex tasks and subtasks, a hierarchical model could also serve to cluster search queries [START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF].

The Distance Dependent Chinese Restaurant Process (DD-CRP) [START_REF] Blei | Distance dependent Chinese restaurant processes[END_REF] provides an approach to compute task relatedness from the hierarchies of tasks. The CRP method is a non-parametric Bayesian approach. It was proposed to model random groups of non-exchangeable data. It assumes a restaurant with an infinite number of tables. Customers enter the restaurant in tandem; they are assigned to a nonempty table based on the number of already existing customers in the table, or to an empty table depending on a scaling parameter. In this case, tables represent search subtasks, while customers are entries in the query log [START_REF] Blei | Distance dependent Chinese restaurant processes[END_REF][START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF].

The hierarchies of search tasks are extracted from the query logs using a representation that models queries as a linear combination of word vectors for its terms. Weights in the linear combinations represent the maximum likelihood for each term according to the query task relationships. [START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]. Search logs are considered as convoluted struc-tures of complex tasks and subtasks. Thus, given an on-task query collection, a DD-CRP model enhanced with word embedding distances determines subtasks in the search log [START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF]. The proposed approach does not need the number of subtasks per complex task. It also uses word embeddings to deal with lexically similar queries that have different semantics, a kind of queries where lexical based methods like bag of words or TF-IDF can not discriminate.

The DD-CRP method models the query links to other queries and not to sub-tasks. Distances between queries are based on cosine distance of word embeddings. DD-CRP uses skip-gram based embeddings to represent queries and computes a weighted average vector per query. Weights in the average come from the frequency of the terms because low-frequency words could be key for helping to identify subtotals. To compare results, [START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF] considers three baseline methods: QT-HTC [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF], LDA [START_REF] Blei | Latent dirichlet allocation[END_REF] learned over documents containing aggregation of queries, and a vanilla CRP model to process queries in the AOLQL [START_REF] Pass | A picture of search[END_REF] dataset (Section 2.1). The AOLQL is augmented with related queries from available APIs of search engines.

A qualitative approach is used to validate the results. The subject chosen is weddings. For that subject, the proposed method is the only one to detect the subtasks. A quantitative evaluation is challenging because there are no publicly available datasets with tasks -subtasks labels. Thus, the evaluation is carried out based on randomly selecting pairs that the method detected for the same subtasks, and asking users to assess if the queries pertain to the same subtasks or not. One hundred repetitions were performed, and the results of the DD-CRP method surpassed all the three baseline methods [START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF].

Bayesian Rose Trees (BRTs) [START_REF] Blundell | Bayesian hierarchical community discovery[END_REF] extend the hierarchy of search tasks to multiple levels, modeling search tasks using an agglomerative clustering approach [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]. Existing agglomerative clustering methods consider nodes in trees as binary, but complex tasks could have one or more related subtasks. Thus, it is essential to use an agglomerative method that supports multiple children from nodes. BRTs provide a clustering method with multiple children in each node. They are adapted and extended to model query hierarchical structures in an unsupervised process [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF].

BRTs represent data samples using binary features. But to better model relationships between queries, binary features can not properly represent the semantic relationships between queries. Thus, queries are represented as a set of affinities. The marginal distribution of the data is modeled as a Bernoulli distribution when using binary features to encode data. When using affinities, the Bernoulli distribution is replaced by a Gamma-Poisson distribution. Query affinities include [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]:

1. Query term affinity uses edit distance and Jaccard coefficients between query terms, as well as the proportion of common terms and the cosine similarity between term sets 2. Retrieved URL affinity uses edit distance and Jaccard coefficients between URLs, given that similar information needs tend to produce similar sets of URLs 3. Time / user session affinity is a binary feature indicating if the queries are part of the same time session and if they were issued by the same user 4. Query embedding affinity uses word embeddings, which are averaged together to form one vector per query. Cosine similarity between query vectors model the semantic relationships of queries. Word embeddings for query terms are custom-trained using the skip gram model and the AOL dataset

The unsupervised process using BRTs starts with all queries in the log as a subtree. In each step, trees are clustered using one of three operations: join, collapse, or absorb. Join creates a top node and assigns each subtree as a child to it. Collapse creates a top node whose children are the children of all the merged subtrees; the top node remains while the nodes of the merged subtrees disappear. Absorb assigns a subtree to the children of the top node; by doing so, the merged subtree node becomes a child of the top node. The clustering operation to perform and the subtrees to merge are selected to maximise the quality of the hierarchical representation. The merging process stops when possible merging operations on remaining subtrees do not improve the quality of the hierarchical representation [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF].

To avoid subtrees in nodes representing atomic tasks, a task coherence computation allows the collapse of all subtreees if such coherence surparsses a given threshold. This step is necessary because queries in atomic search subtasks are strongly related semantically -a phenomenon not observed in complex search tasks with multiple related subtasks. For computing task coherence, the Pointwise Mutual Information (PMI) metric is utilized. It is calculated using the AOL dataset query terms. Evaluation of the task identification effectiveness of the model is performed using [START_REF] Lucchese | Identifying task-based sessions in search engine query logs[END_REF] dataset, a subset of the AOL dataset with manual labels for search tasks in time sessions. Pairwise F-score is the metric used for assessing model performance. Also, tests are performed using query term prediction to test the hierarchical model effectiveness in task identification applications [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF].

Dataset

Additional search patterns based on tasks

Personalization in search systems can be implicit or explicit. In explicit personalization, users directly specify their interests using tools such as forms, surveys, or buttons for ranking. However, users tend to ignore the steps needed to recollect information as they find the process for explicit personalization difficult or undesirable. Thus, implicit personalization is better suited to create models for users. In implicit personalization, search systems automatically infer users' features utilizing their search history. The automatic processing of search history for implicit modeling of users relies on the analysis of query logs, where search systems store data they get from tracking user queries [START_REF] Hearst | Search user interfaces[END_REF]. Search history from users include query sentences, timestamps and click through data. In click through data, we can find the clicks users perform after retrieving search results, while they are browsing resulting documents or reviewing web pages from results. Clicks help search systems assess the quality of their ranking and improve relevance in personalized results [START_REF] Borisov | A neural click model for web search[END_REF][START_REF] Hearst | Search user interfaces[END_REF]. They also help to model user intent and create distributed query representations [START_REF] Zhang | Generic intent representation in web search[END_REF].

For query auto-completion (QAC) or query suggestions, several works consider click-through data or QAC logs (Bar-Yossef and Kraus, 2011; [START_REF] Boldi | Query suggestions using query-flow graphs[END_REF][START_REF] Cao | Towards contextaware search by learning a very large variable length hidden markov model from search logs[END_REF][START_REF] Li | Exploring query auto-completion and click logs for contextual-aware web search and query suggestion[END_REF]Sun and Lou, 2014). However, the two types of logs are considered separately. Given the user's interaction with the search system, these two logs are strongly correlated. A user performs a query search, generating entries in the QAC log; then, the user analyzes the retrieved documents, generating entries in the click log. If results are not satisfactory, a new query search starts, and a subsequent click log entry is added. Therefore, users' behaviors extracted from both logs have a sequential correlation. Also, users' behaviors usually are stable during periods of time [START_REF] Li | Exploring query auto-completion and click logs for contextual-aware web search and query suggestion[END_REF]. [START_REF] Li | Exploring query auto-completion and click logs for contextual-aware web search and query suggestion[END_REF] leverage both types of logs to extract and characterize user search behaviors from them. The click log provides context to the QAC log and vice-versa. Search logs are clustered according to the behavior of the users, looking for correlations at both logs. Experiments to prove the correlation between both logs, experiments include an LDA method that models logs separately, a Hidden Markov Model (HMM) that takes advantage of the correlation between both logs, and a contextual LDA, which also jointly models both logs. Using the log predictive likelihood as a metric, results prove the correlation between both types of logs: HMM and contextual LDA generate a better metric than LDA. Also, contextual LDA outperforms HMM in the log predictive likelihood, proving that it better models the behavior correlation in both types of logs.

The QAC log is not a low resolution log, which registers the query for the last keystroke and the run query. Instead, it is a high resolution log. It saves ten suggested queries for every keystroke. Thus, every keystroke generates its own session of queries in the log. Features from the high resolution log include user behavior characteristics, like the variance of typing speed, the length of the keystrokes, or the average speed. For example, if the user has a low variance in the typing speed, it can suggest that the user ignores the suggestions. Alternatively, if the user types the whole query fast, it can suggest that the user is proficient in the subject she is looking for. Additional tests were performed on the correlated model. A QAC task used contextual LDA along with a two-dimensional click model (TDCM) [START_REF] Li | A two-dimensional click model for query auto-completion[END_REF], outperforming previous methods for QAC. For the click prediction and relevance ranking, Contextual LDA was used with the Bayesian Sequential State (BSS) model [START_REF] Wang | Content-aware click modeling[END_REF], also outperforming previous works.

Personalization can also include individual features or aggregate information of search users, where other users' interactions provide clues about potential suggestions. In particular, implicit personalization can leverage ac-quired knowledge from other users' interactions to better predict user needs. Similarly, personalization could be content-based and users-based. In contentbased personalization, suggestions can be based on context related to users' query terms. In users-based personalization, information from users with similar characteristics helps to create tailored results [START_REF] Hearst | Search user interfaces[END_REF].

Search engine users change their behavior when analyzing search results already seen. For instance, in file browsers, users prefer to browse the file tree instead of using the search bar for finding files. Furthermore, users tend to repeat a percentage of the previous searches; thus, the inclusion of previous queries in suggestions helps to improve user productivity. In general, methods for assisting search engine users can help to predict the information users are likely to read or need. However, automatic suggestions can interfere with the users' intentions, hindering the usability of the search engine [START_REF] Hearst | Search user interfaces[END_REF]. Another change in user behavior can be seen when analyzing mobile and personal computer logs, suggesting that users change their behavior when searching on different platforms [START_REF] Li | Exploring query auto-completion and click logs for contextual-aware web search and query suggestion[END_REF].

Another aspect in user modelling is the analysis of user behaviors for subsequent queries. The GEN encoder [START_REF] Zhang | Generic intent representation in web search[END_REF], the query representation based on user intent (Section 2.3), can be used for modeling those past user behaviors for the same query. Distances of subsequent pairs of queries follow a bimodal distribution when using the GEN encoding. This bimodal distribution allows the identification of query reformulation behavior from four scenarios [START_REF] Zhang | Generic intent representation in web search[END_REF]):

• Topic change happens when the user focuses her attention on other tasks, generating an unrelated query

• Exploration includes a related query inside the same topic

• Specification refers to queries aimed at narrowing the results of the previous query execution

• Paraphrasing includes changes to the query that represents the exact same intent

Results using the GEN encoder bimodal distribution for query reformulation were compared to a number of query encodings, including TF-IDF [START_REF] Robertson | Understanding inverse document frequency: on theoretical arguments for IDF[END_REF], USE [START_REF] Cer | Universal sentence encoder for english[END_REF], and BERT [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF]. A 30 minutes gap was used for segmenting sessions [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]Wang et al., 2013a). Sessions without clicks or with less than three queries were ignored. The GEN encoder surpassed the performance of all the other query encodings used for comparison; however, it still trails the average performance of human labelers [START_REF] Zhang | Generic intent representation in web search[END_REF].

The long-tail effect in machine learning (Section 2.2) can also be observed in queries run by search engine users. The long-tail effect in search query logs refers to the appearance of queries with few or no previous coincidence. However, they can be strongly related to other queries in terms of user intent. This could happen, for example, when there are orthographic errors. Using the GEN encoder and Approximate Nearest Neighbors (ANN), the long tail is reduced in half, with a reasonable response time -10ms for 700M query index [START_REF] Zhang | Generic intent representation in web search[END_REF]. The ANN implementation uses hierarchical navigable small world graphs (HNSWG) [START_REF] Malkov | Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs[END_REF].

Limitations of existing search task models

Most models for search task identification are non-parametric [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]Li et al., 2014a;[START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]. Those models will grow as the size of the input query log grows, becoming computationally expensive, especially when processing large input datasets.

Also, some models requiring user identifiers can not be used in userindependent modeling scenarios [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF]Li et al., 2014a;[START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF], raising issues regarding user privacy.

When users carry out their information seeking, they can release personal information during their interactions with search systems (Craswell et al., 2020a), which brings privacy front and center, as privacy is crucial for the broader field of ethics in intelligent systems [START_REF] Manikonda | What's up with privacy? user preferences and privacy concerns in intelligent personal assis-References tants[END_REF][START_REF] Stahl | Ethics and privacy in AI and big data: Implementing responsible research and innovation[END_REF].

Moreover, some models [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]Wang et al., 2013a) require labeled datasets to train the supervised component of the model. Whether it is a neural network or a support vector machine, labels are indispensable to train them. But labeled datasets are challenging to create because of the resources required to manually label them. When considering large datasets, this limitation is even more critical (Wang et al., 2013a). Also, large collections are difficult to find publicly available, mainly because of user privacy issues and commercial concerns that could arise when other search engines exploit the information (Craswell et al., 2020a).

Furthermore, few models leverage clicked URLs, even though they are strongly related to user search intent [START_REF] Zhang | Generic intent representation in web search[END_REF]. For instance, two lexically different queries with the same clicked document are likely to reflect the same information need [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Zhang | Generic intent representation in web search[END_REF]. Also, a similarity metric based on clicked URLs is the second most important similarity in Bestlink SVM (Wang et al., 2013a), just after the most important one, which is the semantic similarity of the queries.

Another limitation of most models is the lack of realtime response times, which relegates them to offline modeling only. Updating graphs or trees when a user runs a query can cause delays in search systems, affecting the interaction with users in realtime. Most search systems need to generate results interactively. For example, systems for query autocompletion, query sugges-tion, or ranking based on tasks need to have model results as the user runs the query [START_REF] Ahmad | Context attentive document ranking and query suggestion[END_REF][START_REF] Boldi | Query suggestions using query-flow graphs[END_REF][START_REF] Borisov | A neural click model for web search[END_REF][START_REF] Li | Exploring query auto-completion and click logs for contextual-aware web search and query suggestion[END_REF]. Similarly, in conversational search, systems can have strict time thresholds to generate answers, and systems aggregating results from several indexes can discard an index result if it exceeds a certain limit of time [START_REF] Arguello | The effects of aggregated search coherence on search behavior[END_REF]Zamani and Craswell, 2020). Such time thresholds are normally in the order of milliseconds so that it is possible to interact with users (Zamani and Craswell, 2020).

At the same time, most models are monolingual, supporting mostly users queries in English only. But users worldwide can access search systems in many languages. The use of custom-trained word vectors, which use AOLQL for training, or the use of collections like ClueWeb12B or Wikipedia [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]Wang et al., 2013a), limit the number of languages the model can support, normally to English. Though it is possible to custom-train for other languages, few languages have large collections available for training text representations [START_REF] Joshi | The state and fate of linguistic diversity and inclusion in the NLP world[END_REF]. The number of languages is even smaller when considering publicly available query logs (Section 2.1). On top of that, it would be necessary to manually select the custom-trained vectors or the specific collection, or create a pipeline, inserting another method that allows selecting the word vectors depending on the query language.

Search log segmentation

As a first step in multiple search pattern models, a search log segmentation is performed to help in the subsequent modeling of user interactions [START_REF] Hearst | Search user interfaces[END_REF]. Once the segments in the search log are extracted, it is possible to identify search tasks [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]Wang et al., 2013a), analyze query reformulation behaviors [START_REF] Dosso | How to support search activity of users without prior domain knowledge when they are solving learning tasks[END_REF][START_REF] Zhang | Generic intent representation in web search[END_REF], improve query suggestions [START_REF] Ahmad | Context attentive document ranking and query suggestion[END_REF][START_REF] Dosso | How to support search activity of users without prior domain knowledge when they are solving learning tasks[END_REF], or optimize document ranking [START_REF] Ahmad | Context attentive document ranking and query suggestion[END_REF].

To perform the search log segmentation, a chronologically ordered log of search queries is partitioned into smaller sequences of queries. Those query log partitions are commonly known as sessions [START_REF] Gayo-Avello | A survey on session detection methods in query logs and a proposal for future evaluation[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF]. The boundaries for the query log partitions lie in pairs of adjacent queries. To determine if a query pair is a boundary, one may use time spans between the queries [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF]. If the time span is larger than a certain threshold, the query pair is considered a segment boundary, which means that each query belongs to a different segment. The time span for the segmentation can be 26 minutes [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF], 30 minutes [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF], or 90 minutes [START_REF] Hagen | From search session detection to search mission detection[END_REF], depending on statistical analyses of the query logs. By the same token, in some heuristic-based approaches [START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF], segmentation based on time is the first heuristic to be considered.

Search log segmentation

Time spans

Heuristics-based methods

Supervised methods

Discovering tasks from search engine query logs [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF] Geometric session detection in query logs (Gayo-Avello, 2009)

Support Vector Machine (SVM) (Wang et al., 2013a) Extracting hierarchies of search tasks [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF] From search session detection to search mission detection [START_REF] Hagen | From search session detection to search mission detection[END_REF] Context Attention based Long Short-Term Memory (CA-LSTM) [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF] Analysis of query reformulation behaviors [START_REF] Zhang | Generic intent representation in web search[END_REF] Segmenting user sessions in search engine query Logs [START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF] Fig. 2.3: Recent methods for search log segmentation, including segmentation based on time spans, heuristics-based methods, and supervised models.

Nonetheless, according to the analysis of query logs, users tend to interleave multiple search tasks in a single time session. Also, some tasks are performed during multiple time sessions. Thus, the search process for complex tasks could be iterative, lengthy, and multistage, with shifting goals [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]. More sophisticated approaches learn models [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]Wang et al., 2013a) or establish a cascade of heuristics [START_REF] Gayo-Avello | A survey on session detection methods in query logs and a proposal for future evaluation[END_REF][START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF] to determine the boundaries in the sequential query log (Table 2.3).

The cascade of heuristics [START_REF] Hagen | From search session detection to search mission detection[END_REF] uses a time span between queries as the starting rule in the procedure. The cascade method then uses additional heuristics to detect segments -logical sessions -in the query log before performing the identification of search tasks. The heuristics-based session segmentation method (HBSSM) [START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF], an improved heuristics-based method, leverages pre-trained word embeddings to provide semantic similarity measures to segment the query logs, complementing se- mantic relatedness with temporal, lexical, and clicked URLs heuristics. Thus, a cascade of heuristics is applied to each query pair, and manually set parameters provide thresholds in each heuristic to generate the output of the model. However, as mentioned before (Section 2.4), several manually set thresholds in the rules make it challenging to adapt heuristics-based methods (Gayo-Avello, 2009; [START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF] to other datasets without manually adjusting them. Instead, models can learn to segment logs using task labels, as tasks are recognized as a good atomic unit to divide search logs [START_REF] Hearst | Search user interfaces[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF]. Formally, the task segmentation is defined as follows. Let us consider a sequence of queries S = {q 1 , ..., q n }, a collection of pairs of successive queries C = {(q 1 , q 2 ), (q 2 , q 3 ), ..., (q n-1 , q n )} and its respective known labels L = {l 1 , ..., l n-1 } where l i ∈ {0, 1}, ∀i ∈ 1; n -1 . The task segmentation goal is to correctly predict the labels of unseen query pairs, where l i = 1 or l i = 0 indicate if q i+1 and q i belong to the same search task or not, respectively. Support Vector Machines (SVMs) or neural networks can be used to create the segmentation model [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]Wang et al., 2013a). In particular, CA-LSTM [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF] uses RNNs for segmenting search query logs. Three sequences encode the data, in contrast with other models [START_REF] Liu | Deep recurrent neural network for protein function prediction from sequence[END_REF], where a single input encodes data, and a bidirectional RNN processes the single input to produce the model results. Moreover, exploiting the context from query logs in CA-LSTM is necessary to generate an improvement over the existing methods and surpass its baseline approach [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]. The performance of the CA-LSTM architecture depends on at least six adjacent queries for obtaining its best metrics, requiring context around the pair of adjacent queries to properly segment the search query log.

Dataset

Limitations of existing segmentation approaches

Even though clicked URLs are related to user search intent (Section 2.4), when segmenting a log for interactive user assisting applications, it is not possible to wait until the user clicks the URL. A similar scenario arises if the model requires forward and backward queries to generate a context for the segmentation model [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF], which limits the use of the segmentation methods to offline modeling. Also, heuristics-based models pose some limitations. The thresholds for the rules are set depending on a manual statistical analysis of the search query logs [START_REF] Gayo-Avello | A survey on session detection methods in query logs and a proposal for future evaluation[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF]. No method is included to adjust the thresholds when the search query log registers new entries, which requires a manual update of the heuristics. Also, lexically different queries can pertain to the same information need (Mehrotra and Yilmaz, 2017; [START_REF] Zhang | Generic intent representation in web search[END_REF]; thus, semantic comparisons are a better choice when analyzing user search queries.

Chapter 3

Segmenting search query logs by learning to detect search task boundaries

The important thing is not to stop questioning. Curiosity has its own reason for existing. One cannot help but be in awe when he contemplates the mysteries of eternity, of life, of the marvelous structure of reality.

Albert Einstein

As search log segmentation is the first step in many search task models, analyses of search patterns, and user assisting applications (Chapter 2), in this chapter we propose a neural model to learn to detect task boundaries in search query logs. Previous models for task boundary detection rely on clicked URLs and surrounding queries to properly segment the query logs, determining if adjacent queries are part of the same search task or not. However, waiting for clicked URLs or future consecutive queries could render the use of these methods unfeasible in user assisting applications that require model results in realtime. The model proposed in this chapter uses only pairs of adjacent queries and their time span, generating results suited for realtime user assisting applications, with improved accuracy over existing approaches. We also show the advantages of fine-tuning the proposed model for adjusting the architecture to a small annotated collection.

Introduction

Search systems are an essential component of the interactions of users with the World Wide Web. They are crucial to help users access the ever-increasing amount of information available. A wide range of needs and desires from users are converted to queries and submitted to available search engines [START_REF] Hearst | Search user interfaces[END_REF]. Both automatic and manual analysis of search interactions from users enables the modeling of users' search patterns. Extracted patterns help to personalize search engine interactions. In turn, personalization allows the modification of search results depending on the user, the retrieval of advertisement according to user interests, the suggestion of queries related to user information needs, and other tasks designed to support the user while she performs her search tasks [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Hearst | Search user interfaces[END_REF].

Search system logs register the queries users run in the search engines to complete their search tasks. Mining those logs allows the identification of search tasks. Search session segmentation is the first step in multiple methods for search task identification [START_REF] Hearst | Search user interfaces[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]. In session segmentation, a sequential log of search queries is partitioned into smaller sequences of queries. The boundaries for the query log partitions lie in pairs of adjacent queries. To determine if a query pair is a boundary, one may use time spans between the queries [START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]Wang et al., 2013a). If the time span is larger than a certain threshold, the query pair is considered a session boundary, which means that each query belongs to a different session. More sophisticated approaches use heuristics-based models [START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF] or neural networks [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF] to determine the boundaries in the sequential query log. However, the use of clicked URLs [START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF] or adjacent queries [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF] in the search log -in both backward and forward directions -represents a limitation in practical setups, especially in user supporting applications that require modeling on the fly. In such applications, waiting for the clicked URL or future queries to populate the model input might be unfeasible.

Hence, we propose a bidirectional recurrent neural network (RNN) architecture that segments pairs of adjacent queries based on semantic representations of queries and time spans between queries to provide temporal information. The segmentation model determines if adjacent pairs of queries represent a boundary between search tasks, without relying on clicked URLs, character-level representations, or surrounding queries for optimum performance. Furthermore, we test the segmentation architecture in a fine-tuning setup [START_REF] Angermueller | Deep learning for computational biology[END_REF] to know how the model adapts to a small log dataset.

Related limitations

To support users during the information seeking process, it is crucial to segment query logs correctly according to their search tasks. Some methods based on time spans between queries have been proposed to extract task-based sessions from query logs [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF]. However, when analyzing search logs, multiple information needs overlap because users tend to solve various search problems during their interactions with search systems [START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Hearst | Search user interfaces[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF]. Because of this multitasking, a time-based session might contain more than one search task [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF].

Supervised models and heuristics based methods has been proposed as well (Section 2.5). But such approaches have some limitations. Heuristicsbased methods [START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF] have several manually set thresholds in their rules, making it challenging to automatically adapt to other labeled datasets if needed. Should an update is needed, a manual analysis of the new dataset is required to update the thresholds of the rules, while machine learning models could be automatically fine-tuned or retrained.

Likewise, the use of character-level information [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF] can misrepresent queries because lexically similar queries can have entirely different semantic contents. A difference in semantic content implies that lexically similar queries could pertain to non-related search tasks [START_REF] Zhang | Generic intent representation in web search[END_REF]. A related problem arises when evaluating similarities in queries based on term likelihoods [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]. Also, the need for clicked URLs [START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF] or adjacent queries to properly segment search logs [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]) could be problematic. Some user supporting applications need on the fly results, thus, they can not wait for clicked URLs or future user queries to provide forward context. Likewise, when dealing with simple search tasks like fact-finding, typically, a single query could solve the information need [START_REF] Hearst | Search user interfaces[END_REF]; thus, there is no related context available. Similarly, the average amount of queries per search task is less than four, on the basis of the data released by widely used search engines [START_REF] Gayo-Avello | A survey on session detection methods in query logs and a proposal for future evaluation[END_REF]Völske et al., 2019). For instance, there is an average of 3.2 queries per task [START_REF] Hagen | From search session detection to search mission detection[END_REF] and 3.5 queries per task [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] in publicly available search task datasets.

Because of these limitations, we propose a bidirectional RNN (BiRNN) approach that relies on a semantic representation of queries in search logs, adding time spans between queries to provide temporal information. The proposed session segmentation architecture can be fine-tuned to automatically adapt it to new datasets and does not need character level representations, clicked URLs, or adjacent queries in the search log to achieve its maximum performance.

A new task segmentation approach

Our proposed BiRNN-based architecture (Figure 3.1) processes search query vector representations and learns to identify task boundaries in pairs of suc- cessive queries. The architecture relies on a bidirectional recurrent layer, a commonly used layer for processing sequential information [START_REF] Angermueller | Deep learning for computational biology[END_REF][START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF][START_REF] Liu | Deep recurrent neural network for protein function prediction from sequence[END_REF]. Furthermore, some studies in the field of Neural Machine Translation showed that intermediate output states from recurrent layers could significantly improve the performance of the initial models [START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF]. Therefore, following these conclusions, we include an attention mechanism in the segmentation architecture to leverage the information from the intermediate output states. Additionally, completely different queries from a lexical standpoint might represent very similar information seeking needs. For example, "constantinople" and "istanbul archeology" could represent semantically related information needs because Istanbul and Constantinople refer to the same city, but there is no lexical similarity between the two queries [START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]. These discrepancies could affect the ability of methods that rely on character-level information to extract task patterns from query logs. Thus, we do not use character-level information to encode queries. Instead, we use word embeddings to capture the semantic meaning of the queries.

Input representation

We use word embeddings to represent queries in the search log (Figure 3.1).

As queries are short texts, every query is mapped to a unique vector, which allows us to keep both the syntactic and semantic information of the query content [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]. Thus, we define the vector representation of the query q i as the average of all its word embeddings w i in order to get a single vector per query, a query representation approach used in several recent studies [START_REF] Gomes | Segmenting user sessions in search engine query logs leveraging word embeddings[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]Zamani and Croft, 2016a,b). Also, we concatenate the time span in seconds between the timestamps of a query pair: d time span = timestamp(q i+1 )timestamp(q i ), where timestamp is a function that gets the timestamp in seconds of a given query. If needed, the d time span value is broadcast1 to match the input dimensions.

For the experiments to evaluate the model with adjacent queries for context, each adjacent query vector is appended to the query pair representation, respecting the sequential order in which queries appear in the search log. In this case, the input contains q -m , . . . , q 0 , q 1 , d time span , . . . , q n , where m is the number of backward queries and n is the number of forward queries.

Recurrent neural network

An RNN is a type of neural network ideally suited to process sequential information. In contrast with previous methods [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]Wang et al., 2013a), RNNs tend to scale adequately and perform well when dealing with sequential data [START_REF] Angermueller | Deep learning for computational biology[END_REF]. RNNs store information from input sequences by using iterative function loops [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF][START_REF] Rajaraman | Mining of Massive Datasets[END_REF]. To store information, RNNs have a hidden state vector, which works as a memory while the network processes the sequence in the forward direction [START_REF] Rajaraman | Mining of Massive Datasets[END_REF]. BiRNNs were proposed to process the input sequence simultaneously in a forward and backward direction. To do so, they compute forward hidden states and backward hidden states. Concatenating forward and backward hidden states enables the generation of outputs that leverage information from preceding and following steps in the sequence [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF].

In our proposed approach, we add an attention mechanism [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF] at the output of the bidirectional RNN (Figure 3.1). The forward and backward hidden states of the bidirectional recurrent layer are concatenated to the attention mechanism output before applying dropout. The proposed BiRNN-based model uses a fully connected layer that takes the dropout result as input (Figure 3.1). Then, Softmax is used to generate the results in the fully connected layer, indicating if the two queries are part of the same task or not.

Configurations of the proposed BiRNN approach

Four alternative configurations are studied for the proposed BiRNN approach as a result of two hyperparameters: time span location and recurrent unit type. The time span d time span can be placed in the initial layer (Figure 3.1) or the concatenation layer [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF], at the attention mechanism output. The latter is inspired by previous work [START_REF] Severyn | Learning to rank short text pairs with convolutional deep neural networks[END_REF], where the late concatenation of extra features improved performances in text classification. Regarding recurrent unit types, in practical setups, the memory in the standard RNN only works effectively with information very close to the step that the network is processing. Thus, LSTM networks were proposed to enhance the performance of standard RNNs with long term information. LSTMs have the ability to save important information, forget information that is not relevant, and focus on the parts of the sequence that better serves the overall network performance. A two-tier configuration provides the ability to focus. It comprises a hidden state vector, which works as the working memory, and a cell state vector, which works as the long term memory. A popular variant of the LSTM network is the Gated Recurrent Unit (GRU), a simpler architecture to compute and implement [START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF]. It relies on a single vector for memory purposes, decreasing the number of parameters needed [START_REF] Rajaraman | Mining of Massive Datasets[END_REF][START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF]. In the proposed architecture, we consider both types of recurrent units for the bidirectional layer. Two datasets were considered in this chapter: the WSMC12 and the CSTE datasets (Section 2.1). While task labels in the WSMC12 dataset correspond to search tasks inside user search sessions, with query timestamps (Table 3.1), task labels in the CSTE dataset correspond to cross-session search tasks, without grouping queries by user information or query timestamps. Therefore, to ensure the sequential order of queries in the CSTE dataset, query timestamp information comes from [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF]. Moreover, Accuracy and F 1 score enable the evaluation of the different approaches, performing 10-fold cross-validation. The Student's paired t-test provided the test for statistical significance.

Results and discussion

Technical considerations

We used the GloVe publicly available pre-trained word vectors [START_REF] Pennington | GloVe: Global Vectors for Word Representation[END_REF] for computing the search query vector representations. GloVe is a word vector space used in several recent works for short text representation and query encoding [START_REF] Edo-Osagie | Attention-based recurrent neural networks (RNNs) for short text classification: An application in public health monitoring[END_REF][START_REF] Marivate | Improving short text classification through global augmentation methods[END_REF][START_REF] Rakib | Improving short text clustering by similarity matrix sparsification[END_REF]Zamani and Croft, 2016a,b;[START_REF] Wang | Semantic expansion using word embedding clustering and convolutional neural network for improving short text classification[END_REF]. The pretrained word vectors have a dimension of 300 and are based on web corpora with 42 billion tokens from Common Crawl [START_REF] Pennington | GloVe: Global Vectors for Word Representation[END_REF]. Word vectors have a coverage of 96.2% for the WSMC12 dataset and 84.5% for the CSTE dataset.

The proposed segmentation architecture is implemented using the Tensorflow deep learning framework -an advanced dataflow system that provides one of the most efficient implementations for RNNs [START_REF] Angermueller | Deep learning for computational biology[END_REF]. To train the segmentation model, we minimized the cross-entropy loss with the Adam optimizer (Figure 3.3). The Adam optimizer used the default TensorFlow parameters β 1 = 0.9, β 2 = 0.999, and = 10 -8 . The learning rate was set to 10 -4 , batch size to 256, layer size to 32, training steps to 6 x 10 4 , and dropout to 0.3. To initialize the weights in all the layers, we used [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF] initialization scheme. The bidirectional layer size was determined by testing several sizes with recurrent units and the WSMC12 dataset (Figure 3.2). The model performance improved from an accuracy of 0.926 at 4 recurrent units, to an accuracy of 0.937 at 32 recurrent units. However, a further increase in size affects performance. Accuracy falls to 0.926 at 64 recurrent units. Because of these results, we set the bidirectional layer size to 32 recurrent units.

Evaluation results

Comparison of results

We compared the proposed approach to the state-of-the-art HBSSM approach as a baseline. We also considered other supervised machine learning approaches like logistic regression, Support Vector Machine (SVM) with linear kernel and Radial Basis Function (RBF) kernel (γ=2, C=1), K-Nearest Neighbors (n=3), Naive Bayes, Gaussian Process Classification (GPC) with (κ[γ=1]), Random Forest, AdaBoost, Decision Tree, and Quadratic Discriminant Analysis (QDA) (Table 3.2). We also evaluated the proposed model in several context scenarios (Table 3.3), using CA-LSTM [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF] as a reference. Additional to the query pair representation (Section 3.3.1), we provided query texts and clicked URLs, when available, to the baseline HBSSM approach so that it was possible to compute all the heuristics specified for the model. We also flattened the query pair representation for methods that required a unidimensional vector as input, ensuring that all methods got the query pair and its time span as input information.

Our proposed segmentation architecture obtains very satisfactory results, surpassing the remaining methods used for comparison. Moreover, GPC -the best performing method from the traditional machine learning approachesgets results close to the HBSSM performance and matches the accuracy of CA-LSTM, even exceeding it in some context scenarios. Nonetheless, results from our proposed segmentation architecture outperform the previous recurrent model in all context scenarios. The trained BiRNN GRU architecture is also faster than the HBSSM baseline. We measured the query pair processing time using a CPU-only virtual instance with a CPU frequency of 2.397 GHz, taking the average time per query pair for all the pairs in the WSMC12 dataset. The trained BiRNN GRU architecture took around 15ms per query pair, while the HBSSM approach took around 61ms. This difference in execution time represented a speedup of 4 for our proposed architecture.

Results and discussion

55 Also, we analyze the model with both GRU and LSTM recurrent units, as the decision of which to choose depends on the task and the dataset [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF]. Replacing the GRU cells with LSTMs decreases the accuracy of the segmentation architecture with the WSMC12 dataset. Accuracy drops from 0.937 to 0.931 (Table 3.2). Similar behavior is observed in the fine-tuning results for the CSTE dataset (Table 3.5), suggesting that the simpler GRU cell [START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF]) is a better choice for the search segmentation model. Additionally, given the smaller number of parameters per recurrent unit, GRU models have another advantage: a faster training time. One training step with GRU cells takes around 20.15s, while one training step with LSTM cells takes around 31.68s, representing a speedup of 1.6 for the GRU model.

Likewise, concatenating the time spans to the attention mechanism decreases the performance of the proposed architecture. Accuracy drops from 0.937 to 0.927 with the WSMC12 dataset. With the smaller CSTE dataset, the best results are obtained when performing fine-tuning with time span information at the input sample representation as well (Table 3.5). Thus, the best scenario happens when the query vectors and the time span information are part of the input sample representation. This configuration allows the computation of hidden state vectors inside the bidirectional layer that rely upon both the query pair semantic content and the time between them. Similarly, the bidirectional output vectors depend on both semantic and time information to feed the attention mechanism.

No

Impact of query context

We also analyzed how the model behaves when adding adjacent queries for session segmentation. We computed query vectors for each adjacent query and appended the query vectors to the original pair, respecting the order in which they appeared in the original search log. Half of the adjacent queries precedes the query pair, while the remaining half follows the query pair in the chronological search log. When the input includes adjacent queries for context, we do not see any improvement in the accuracy of the segmentation architecture. Although the addition of adjacent query vectors still exceeded other baseline methods, overall, none of the results improved the performance of the query pair without context (Table 3.3). Such a result is essential because the average amount of adjacent queries per search task is around three. Similarly, simple information needs like fact-finding are usually solved with only one query [START_REF] Hearst | Search user interfaces[END_REF]. Furthermore, forward context can be a critical drawback, especially when supporting user information needs. Applications like query suggestion must respond on the fly. They cannot wait for future queries to provide context to the model. The same pattern was observed when we examined only backward queries for context (Table 3.4), a strategy similar to the approach in Bestlink SVM (Wang et al., 2013a), where only previous queries in the sequential log were considered. In all tests, the segmentation model gave the best performance when the input had the query pair and the time span between them.

Context

Fine-tuning with a smaller dataset

Finally, we wanted to know the impact of pre-training the segmentation architecture before training the model with a small dataset. Scarce data poses 3.5 Summary 57 challenges to deep learning architectures, and fine-tuning is one of the alternatives to deal with small datasets [START_REF] Angermueller | Deep learning for computational biology[END_REF][START_REF] Lin | Deep learning of binary hash codes for fast image retrieval[END_REF]. Table 3.5 presents the results for the fine-tuning experiments. When performing 10-fold cross-validation on the CSTE dataset, without doing finetuning, we got an accuracy of 0.751. After pre-training with the WSMC12 dataset, we performed 10-fold cross-validation on the CSTE dataset, obtaining an accuracy of 0.803. Thus, the BiRNN architecture configuration with the time span at the input and GRU recurrent units gave the best performance when fine-tuning. Overall, it represents the best performance with the smaller CSTE dataset when we consider all the methods tested. The results mentioned above highlight the ability of the model to perform well when processing datasets with a limited number of samples, a common scenario because of the difficulty for obtaining large search query logs and the cost associated with labeling them (Wang et al., 2013a). 3.5: Fine-tuning the recurrent architecture to realize segmentation on the CSTE dataset. The model is pre-trained using the WSMC12 dataset.

Segmentation model

Summary

In this chapter, we proposed a bidirectional RNN architecture with an attention mechanism for segmenting search query logs by identifying task boundaries on them. Our proposed segmentation model processes query pairs in search logs to determine if a boundary is present or not. Experimental results showcase the improvement of the proposed architecture over the baseline method, outperforming the other approaches used for comparison. Once trained, the recurrent model is also several times faster than the heuristicsbased baseline. Furthermore, there is no need for clicked URLs, character-level representations, or additional queries surrounding the query pair to provide context to the model. This result is especially relevant given the mean number of queries per task in datasets from widely used search engines and the fact that information needs like fact-finding and other simple tasks are usually solved with only one query. Even when testing with additional context to the query pair, segmentation architecture results continue to be above all the other approaches. Also, the proposed segmentation model performs well when fine-tuning with a smaller query log dataset. Fine-tuning performance is useful given the scarcity of publicly available labeled collections from search engines and the cost of labeling large search logs.

Chapter 4

Extracting Search Tasks from Query Logs Using a Recurrent Deep Clustering Architecture

Tout ce qu'un homme peut imaginer, d'autres hommes peuvent le rendre réel.

Jules Verne

The neural model in Chapter 3 learns to detect search task boundaries from user interactions. However, it is a supervised method, requiring labeled datasets for training the model. Likewise, task boundaries are not enough to extract search tasks from user interactions. The steps users perform to carry out search tasks are not necessarily sequential. Those steps can appear interleaved in the search query log, reflecting the fact that users multitask when accessing search systems. In several search query datasets (Section 2.1), it is possible to observe that users can switch to other search tasks, and after a certain period, come back to continue the previous search task. In addition, different users can run queries to carry out the same information need.

Consequently, in this chapter, we extend the neural architecture to propose a model that can extract search tasks from user interactions. Most existing search task extraction methods use graph-based or non-parametric models, which grow as the query log size increases (Section 2.4). Deep clustering methods offer a parametric alternative, but most deep clustering architectures fail to exploit recurrent neural networks for learning text data representations. Therefore, we propose a recurrent deep clustering model for extracting search tasks from query logs. The proposed architecture leverages self-training and dual recurrent encoders for learning suitable latent representations of user queries, outperforming previous deep clustering methods. It is also a parametric approach, which offers the possibility of having a fixed-sized architecture for analyzing increasingly large search query logs.

Introduction

Users carry out their search tasks running groups of related queries in available search systems, fulfilling a wide range of information needs and desires [START_REF] Hearst | Search user interfaces[END_REF]. Query logs record the queries that users submit to search engines. Therefore, proper extraction of search tasks from query logs helps to support users while they fulfill their information needs, facilitating multiple goals like query term prediction, query recommendation, advertisement, results ranking depending on the search task, query-task mapping, and prediction of user satisfaction based on search tasks [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]Völske et al., 2019).

Along with the search queries that users submit, query logs also contain timestamps and other user-related information. Initially, query logs were segmented using the time between query timestamps to establish a session boundary and delimiting search tasks. If the time between a pair of subsequent queries was above a certain threshold in minutes, a boundary was established, signaling the end of a search task [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF]. However, according to multiple analyses of search query logs, users tend to interleave search tasks in a single time session. Also, some tasks are performed during multiple time sessions [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]. Hence, clustering models have been utilized to extract search tasks by grouping semantically related queries.

Recent models for search task extraction rely on graph-based methods or non-parametric approaches [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF], which grow as search query logs increase in size, making them more computationally expensive as the number of queries increases. By contrast, deep clustering methods [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF] offer a parametric alternative to learn latent representations of query log entries and simultaneously cluster them into interrelated groups of search queries.

Most existing deep clustering approaches do not exploit the modeling power of recurrent neural networks (RNNs), which are widely used for natural language processing (NLP) and sequential data processing [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF][START_REF] Lipton | A critical review of recurrent neural networks for sequence learning[END_REF][START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF]. Therefore, we propose a recurrent deep clustering (RDC) model to extract search tasks from query logs. RDC leverages the modeling power of recurrent neural networks in a dual encoder configuration, along with self-training, to learn a suitable latent space of user queries and simultaneously cluster them in groups of search tasks.

Related limitations

The need for large query log datasets that are cleaned and labeled by humans represents a challenge for supervised task extraction models [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF]Wang et al., 2013a). As unsupervised learning approaches do not rely on labels [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF], they could represent a better alternative for search task extraction. Clustering is an unsupervised learning approach that groups related items using abstract similarities or learns new categories by analogy to existing ones [START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF].

Several clustering methods has been proposed proposed to extract search tasks from query logs (Section 2.4). Nonetheless, graph-based and nonparametric models grow as the size of datasets increases [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF], becoming more computationally expensive. For example, the number of leaves in BRTs [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF] is directly related to the number of queries in the search query log; for graph-based methods, every entry in the search query log ends up being a node in the underlying graph. Likewise, the representation of the data is crucial for the subsequent results of clustering methods. High dimensional data tends to affect clustering methods because distances in high dimensional spaces are less effective. Dimensionality reduction methods have been widely used, including linear methods, non-linear methods, and spectral methods. Nevertheless, the latent representation obtained from dimensionality reduction can affect clustering performance; thus, deep neural networks are a viable alternative to compute latent representations [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF][START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF] for input data, without performing dimensionality reduction as a preprocessing step.

Deep neural networks can be used to simultaneously learn latent representations and cluster data, using a method commonly known as deep clustering [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF]. Also, in contrast with graph-based methods and non-parametric approaches, deep clustering models do not grow with the size of the search query log [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF]. Deep clustering appeared initially in acoustic separation and then spread to other areas of research [START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF]. Before deep clustering appeared, research focused on data representation and clustering methods independently. However, learning latent representations is at the heart of deep clustering.

Models in deep clustering rely on several neural network architectures, including autoencoders, variational autoencoders (VAEs), feedforward neural networks, convolutional neural networks, deep belief networks, and generative adversarial networks (GANs) [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF]. All architectural variations are trained to learn cluster friendly representations, combining representation learning and clustering. Deep neural networks are trained to minimize the clustering loss, optimizing the network weights for improving the predicted labels of input samples. Both GAN and VAE based architectures are generative. They do not only learn to cluster inputs; they are also able to generate samples from the clustering categories [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF].

Existing deep clustering models fail to exploit RNNs for learning latent representations of text data samples. Text data naturally fits the sequential modeling power of recurrent neural networks [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF]. Because of this, RNNs have been widely used for processing text data in NLP, generating state-of-the-art results in multiple applications [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF][START_REF] Mitchell | Artificial Intelligence: A Guide for Thinking Humans[END_REF][START_REF] Zhang | Generic intent representation in web search[END_REF]. Our proposed architecture differs from prior deep clustering methods [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF] by using RNNs in a dual encoder configuration [START_REF] Yang | Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax[END_REF] to simultaneously learn latent representations of user queries and cluster them in groups of search tasks. Also, in contrast with other approaches [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF], it provides a parametric model, which preserves its size despite the query log length. The proposed RDC model has an RNN encoder as the central component of its architecture. The architecture uses a dual encoder setup [START_REF] Yang | Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax[END_REF] (Section 2.2), a widely used configuration in representation learning, neural machine translation, and other NLP applications [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Yang | Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax[END_REF][START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF]. The recurrent encoder comprises a bidirectional recurrent layer, an attention mechanism, and a projection head [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF]. Input queries comprise a list of word embeddings q i = [w 1 , w 2 , w 3 , ...]. To form the input query's latent representation, we concatenate the output of the attention mechanism and the hidden state of the bidirectional recurrent layer, passing the concatenated tensor through a projection head (Figure 4.1). Regarding recurrent unit types for the encoder, we consider both Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF].

Search task extraction

Two phases optimize RDC in tandem: a pretraining phase and a clustering phase [START_REF] Guo | Improved deep embedded clustering with local structure preservation[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF][START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF]. Each phase has its loss; thus, we adapt the architecture of the model depending on the loss that we are optimizing (Figure 4.2).

Pretraining phase

Deep clustering methods tend to pretrain neural network layers before the clustering phase, which allows the initialization of the latent representation for input samples [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF]. During the pretraining phase, we optimize the encoder with a supervised objective. We use the dual encoder configuration [START_REF] Yang | Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax[END_REF] to pretrain the recurrent encoders according to the following objectives:

• Segmentation. In this supervised pretraining objective, the recurrent encoders are trained using the search task segmentation approach (Chapter 3). This pretraining objective determines if two adjacent queries in a chronologically ordered query log are part of the same search task or not.

The expected output of this objective is binary, and we use cross-entropy to compute the pretraining loss L P .

• Sequence. The sequence pretraining objective determines if a pair of queries appear adjacent in a chronologically ordered query log or not. Similar to the segmentation objective, the expected output of this objective is also binary, and we use crossentropy to calculate the pretraining loss L P .

• Intent. For the intent pretraining objective, queries representing the user's intent for the same search task are close in the latent representation space [START_REF] Zhang | Generic intent representation in web search[END_REF]. Therefore, we compute the cosine proximity between encoder outputs and use the Mean Squared Error (MSE) (Tan et al., 2018) between predicted cosine proximity and expected cosine proximity to calculate the pretraining loss L P . The expected cosine proximity is set to one for pairs of queries pertaining to the same search task, zero otherwise.

Clustering phase

Once the foregoing objectives have been used to pretrain the recurrent encoders, we discard the layers on top of the encoders that we used during pretraining, changing the pretraining objective to the clustering objective. The objective loss for the clustering phase L O comprises the clustering loss L C and the self-training loss L S [START_REF] Guo | Improved deep embedded clustering with local structure preservation[END_REF][START_REF] Luo | Deep clustering and conventional networks for music separation: Stronger together[END_REF].

For the clustering loss, following previous work [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF][START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF], we connect the pretrained encoders' output to a clustering layer, where the Student's t-distribution provides a kernel to compute the soft assignments for the data points representing the user queries. The Student's t-distribution is a heavy-tailed distribution that helps to preserve distances of data points when mapping from a high dimensional to a low dimensional space. When mapping data points, the distances in the high dimensional space require larger distances in the low dimensional space to remain equivalent. Hence, the natural gaps existing between clusters in a high dimensional space can easily disappear when mapping to a lower dimensional space. One way to deal with this phenomenon is to use a heavy-tailed distribution to model distances in the low dimensional space. Compared to a Gaussian distribution, the Student's t distribution has much heavier tails. For that reason, it is used in the t-Distributed Stochastic Neighbor Embedding (t-NSE) (Van der Maaten and Hinton, 2008), a widely popular technique for visualizing high dimensional data. Thus, by using the Student's t distribution in the low dimensional space for the clustering layer, it is possible to preserve the natural gaps that exist in the high dimensional space [START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF]Van der Maaten and Hinton, 2008).

Additionally, the soft assignments should match an auxiliary target distribution, which is designed to emphasize high confidence data point assignments, strengthen predictions, and normalize the loss contribution from each cluster by using the soft cluster frequencies. Therefore, we compute the clustering loss by calculating the Kullback-Leibler (KL) divergence between the Student's t-distribution and the auxiliary target distribution. Formally, given a query q i and initial cluster centroids µ j , the clustering loss L C is calculated as follows [START_REF] Guo | Improved deep embedded clustering with local structure preservation[END_REF][START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF]:

z i = encoder 0 (q i ) (4.1)
s ij = (1 + z i -µ j 2 ) -1 j (1 + z i -µ j 2 ) -1 (4.2) f j = i s ij (4.3) p ij = s 2 ij /f j j s 2 ij /f j (4.4) L C = i j p ij log p ij s ij (4.5)
where p ij is the auxiliary target distribution and f j are the soft cluster frequencies. K-means generates the initial cluster centroids µ j from the pretrained encoder representation. The Student's t-distribution in equation 4.2 has one degree of freedom [START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF].

For the self-training loss, we use the dual encoder configuration along with back translation [START_REF] Edunov | Understanding backtranslation at scale[END_REF], a self-training technique for unsupervised data augmentation [START_REF] Xie | Unsupervised data augmentation for consistency training[END_REF] that preserves the semantics of the query encodings. Adding noise to query encodings can be ineffective for creating augmented samples because the resulting samples hardly match variations from real case scenarios. Hence, to create realistic augmented samples, strong data augmentation methods focus on creating modifications that match real variations. For instance, in computer vision, it is common to use cropping, rotation, or scaling to create augmented samples for images [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Zoph | Rethinking pre-training and self-training[END_REF]. As queries are short texts, we use back translation to create realistic augmented samples for entries in the search query log (Figure 4.3).

Back translation creates paraphrases for a search query while preserving the semantics of the original query [START_REF] Edunov | Understanding backtranslation at scale[END_REF]. Formally, given a query q i , with augmented sampled b i , and because the semantics from back translation remains the same, target cosine proximity t i = 1.0, the self-training loss L S is calculated as follows [START_REF] Guo | Improved deep embedded clustering with local structure preservation[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF][START_REF] Xie | Unsupervised data augmentation for consistency training[END_REF]:

u i = encoder 0 (q i ) (4.6) v i = encoder 1 (b i ) (4.7) p i = u i v i |u i | |v i | (4.8) L S = i (p i -t i ) 2 (4.9)
During the clustering phase, both the cluster centroids and the dual encoder weights are updated by optimizing the objective loss L O [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Guo | Improved deep embedded clustering with local structure preservation[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF]:

L O = L S + γL C (4.10)
where γ is a constant. The range of γ is 0.0 < γ < 1.0 to help the model preserve the semantic space of the query encodings with the optimization of L S , while simultaneously optimizing L C to improve the clustering performance.

Experimental setup

Metrics to evaluate the performance of the models include the unsupervised accuracy (ACC), the Normalized Mutual Information (NMI), and the Adjusted Rand Index (ARI) (Section 2.2.3). The Student's paired t-test provides the test for statistical significance. To evaluate the effectiveness of RDC, we compare its performance with the following methods:

• Deep Embedded Clustering (DEC) combines feature extraction with autoencoders and clustering. It learns clustering centers by first pretraining the autoencoder on the input dataset to learn a latent representation. Then, DEC discards the decoder part of the autoencoder and uses the encoder to calculate input representations. DEC uses K-means to initialize cluster centroids, then, it minimizes the clustering loss by minimizing the KL divergence (Equation 4.5) [START_REF] Chang | Deep adaptive image clustering[END_REF][START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF].

• Improved Deep Embedded Clustering (IDEC) extends DEC by including the decoder part of the autoencoder during the clustering. Doing so aims to preserve the original structure of the input data in the latent representation space. To include the encoder, IDEC uses a loss to simultaneously optimize the clustering on the encoder output and the representational accuracy of the decoder output [START_REF] Guo | Improved deep embedded clustering with local structure preservation[END_REF].

• Point Symmetry-based Deep Clustering (SymDEC) replaces the euclidean distance that DEC uses for computing the clustering loss with the point symmetry-based distance, improving the results when clustering datasets with symmetrical input samples [START_REF] Moreno | Point symmetry-based deep clustering[END_REF].

• Deep adaptive clustering (DAC) joins feature extraction and clustering into a single neural method. To extract features, DAC relies on a deep convolutional neural network and adds a constraint, so that resulting labels converge to a one-hot encoding. The constraint assumes that a pair of input samples either pertain to the same cluster or pertain to a different cluster. Input sample similarities are unknown beforehand; thus, an adaptive approach inspired by curriculum learning [START_REF] Bengio | Curriculum learning[END_REF] is proposed. First, only pairs of input samples with similarities above or under a threshold are considered. With those pairs of images, the weights of the convolutional network are updated using back-propagation.

As the training advances and the model improves, more pairs meet the threshold criteria. When the model converges, all pairs of input samples are part of the loss computation, and the loss stabilizes. Once the loss stabilizes, it selects the label with the highest value inside the one-hot vector to determine the cluster of the input sample [START_REF] Chang | Deep adaptive image clustering[END_REF].

• Chimera network [START_REF] Luo | Deep clustering and conventional networks for music separation: Stronger together[END_REF][START_REF] Wang | Alternative objective functions for deep clustering[END_REF] uses stacked layers of bidirectional LSTMs for audio separation models. This stacked recur-rent model can handle problems like speaker-independent multi-speaker speech separation and music source separation. The Chimera architecture comprises four bidirectional layers, a dense layer to compute the vectors in the latent space, and two heads for multi-task learning: one head for unsupervised source separation and other head for supervised time-frequency mask inference. We replace the multi-task learning heads with the clustering layer in Section 4.3.2 to adapt the architecture for search task extraction from query logs.

For reference, we also include results for k-means (Tan et al., 2018), Density-based spatial clustering of applications with noise (DBSCAN) [START_REF] Ester | A density-based algorithm for discovering clusters in large spatial databases with noise[END_REF], and Hierarchical Agglomerative Clustering (HAC) [START_REF] Murtagh | Ward's hierarchical agglomerative clustering method: which algorithms implement Ward's criterion[END_REF][START_REF] Ward | Hierarchical grouping to optimize an objective function[END_REF]. Scikit-learn1 with default parameters provides the implementation for k-means, DBSCAN, and HAC. We also use publicly available implementations for DEC2 , IDEC 2 , SymDEC 2 , DAC3 , and Chimera4 with the best performing hyperparameters reported for each method.

Two datasets are considered for evaluating RDC performance: the CSTE and the TRECQTM datasets. For pretraining, two datasets are considered as well: Sequence and Segmentation pretraining objectives use the WSMC12 dataset, which has timestamps so that we can guarantee a chronologically ordered query log. The Intent pretraining objective uses the WHQTM dataset (Section 2.1).

The GloVe publicly available pre-trained word vectors5 provide the representation for the search queries [START_REF] Pennington | GloVe: Global Vectors for Word Representation[END_REF]. We use the same query representation for all the methods under testing. To train the RDC model, we use the Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. The learning rate is set to 10 -4 , batch size to 256, and dropout to 0.3. The bidirectional layer contains 32 recurrent units, and the projection head has two feedforward layers, one with 512 units and the other with 256 units. Using the Google Cloud Translation API6 , we perform the back translation augmentation for the self-training loss (Equation 4.9). Back translation is realized offline for practical purposes, using English (en) -French (fr) [START_REF] Xie | Unsupervised data augmentation for consistency training[END_REF]

Results and discussion

Results for RDC with several pretraining configurations appear in Table 4.1. RDC outperforms all the other deep clustering methods used for comparison, for both the CSTE and the TRECQTM datasets. RDC also outperforms reference methods like k-means and DBSCAN. When comparing clustering performance against HAC, we find that RDC outperforms HAC when ex-tracting short-lived search tasks, while in long search tasks, it improves over HAC in two out of three metrics (p ≤ 0.05). The CSTE dataset has mostly short-lived search tasks because the average number of user queries per task is 3.2, while the TRECQTM dataset has an average of 28.2 user queries per task, reflecting behaviors like exploration, specification, or paraphrasing that users undertake in long search tasks [START_REF] Zhang | Generic intent representation in web search[END_REF]. These results are essential because short-lived search tasks, including fact-finding, browsing, or transactions, can account for up to 85% of all the entries in a search query log [START_REF] Hearst | Search user interfaces[END_REF].

When comparing RDC with autoencoder-based models, such as DEC, IDEC, and SymDEC, the results are higher in all the metrics used for assessing clustering performance; we observe the same behavior when considering DAC, which uses convolutional neural networks. This outperformance reflects the advantage of using the modeling power of recurrent neural networks for learning representations of search queries. Chimera, a stacked recurrent architecture, also outperforms deep clustering models based on autoencoders and convolutional neural networks. However, RDC has a better clustering performance than Chimera in the three metrics used for comparison. Similarly, RDC has a more straightforward configuration than Chimera because RDC only uses two bidirectional recurrent layers for the dual encoder setup, while Chimera uses a stack of four bidirectional recurrent layers.

Self-training with back translation for queries renders pretraining effects negligible. Indeed, back translation using English (en) -French (fr) is a strong data augmentation technique. It augments data samples while preserving the semantics of the original queries. For instance, "effects of tide on columbia river"gets translated to "effets de la marée sur la rivière Columbia", and then back translated to "tidal effects on the columbia river"; "farm houses for rent in broom county"gets translated to "Maisons de ferme à louer dans le comté de broome", and then back translated to "farms for lease in broom county". Sometimes back translation corrects spelling, for instance "the cost of haveing a horse in new york"gets translated to "le coût d'avoir un cheval à new york", and then back translated to "the cost of having a horse in new york", but in general, the semantics remain the same, so the query encoding space is preserved during the clustering phase by minimizing the self-training loss.

Consequently, although the best pretraining scheme for the RDC models is the Sequence objective, surpassing the results of both Segmentation and Intent objectives, it represents no change when compared against RDC with no pretraining. For the CSTE dataset, accuracy is only 0.5% higher, and NMI is only 0.1% higher (p = 0.8); ARI has no change at all. We observe a similar behavior with the QTMT dataset. In some cases, pretraining can even end up hurting performance, as we can see with the Intent pretraining objective. Preceding results are in agreement with previous work about the effect of pretraining neural architectures [START_REF] Zoph | Rethinking pre-training and self-training[END_REF], where self-training with strong data augmentation diminishes the effect of pretraining, making it negligible. Therefore, it is possible to discard neural network pretraining,

Dataset

Cell an essential result because pretraining needs labeled datasets, which can be challenging to create (Wang et al., 2013a), while self-training with back translation is unsupervised.

Regarding recurrent units, the decision of which to choose depends on the task and the dataset [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF]; therefore, we analyze the RDC model with both GRU and LSTM cells (Table 4.2). Replacing the LSTM cells with GRUs generates a slight decrease in model performance for CSTE and TRECQTM datasets. The biggest difference happens with the TRECQTM dataset, using intent pretraining, where changing GRUs to LSTMs makes accuracy decrease 2.7%, NMI 2.5%, and ARI 1.1% (p ≤ 0.05). These changes imply that less computationally expensive GRUs are a better choice for the RDC architecture than LSTMs, although changes observed for the metrics are low, especially with the sequence or no pretraining configurations, which are the best performing setups for the RDC model.

Summary

This chapter presented RDC, a recurrent deep clustering method for extracting search tasks from query logs. The proposed method leverages selftraining and dual recurrent encoders to find latent representations for user queries, clustering them into search task groups. Experimental results show the proposed clustering method outperforms prior deep embedding clustering architectures in all the metrics used for testing. Also, RDC offers a parametric architecture for search task extraction, which preserves its size despite changes in the query log size. This size preservation represents an advantage compared to non-parametric methods and graph-based models that grow with the query log size, making them more computationally expensive as the number of queries in the search log grows.

Chapter 5

A multilingual approach for unsupervised search task identification

The best way to get a good idea is to get a lot of ideas.

Linus Pauling

Parametric models like RDC (Chapter 4) tend to have lower modeling results than non-parametric alternatives. When comparing parametric and non-parametric machine learning models, there is a trade-off between model size and accuracy [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF]. Parametric models remain constant regardless of the size of the input dataset, but they tend to be less accurate. On the other hand, non-parametric models grow as the size of the input dataset grows. However, they tend to be more accurate. Additionally, the use of GloVe word vectors for query representation limits RDC to English queries only. On top of that, we need to estimate the number of search tasks beforehand, so that we can initialize the clustering layer.

Therefore, in this chapter, we propose an unsupervised, multilingual model for search task identification. Search system logs contain queries in multiple languages, but most existing methods for search task identification are not multilingual. Some methods rely on search context for custom embeddings or external indexed collections that support a single language, making it challenging to support the multiple languages of queries run in search systems. Other methods depend on supervised components and user identifiers to model search tasks. The supervised components require labeled collections, which are challenging and costly to get in multiple languages (Section 2.4). Also, the need for user identifiers renders these methods unfeasible in user agnostic scenarios. Hence, the proposed approach in this chapter is user agnostic, enabling its use in both user-independent and personalized scenarios. Furthermore, the multilingual query representation enables us to address the existing trade-off when mapping new queries to the identified search tasks.

Introduction

To support users during the different steps of the information seeking process, it is crucial to correctly group queries in search logs according to their search tasks. Mining query logs from search engines enable the automatic modeling of search tasks. Precise modeling of search tasks is required for user assisting applications like query term prediction, query recommendations, user modeling based on tasks, personalization in e-commerce, and results ranking [START_REF] Hearst | Search user interfaces[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]Völske et al., 2019), which enhance search system support for helping users to fulfill their information needs.

Most unsupervised search task identification methods rely on custom training of word embeddings using search collections or external indexed collections to provide semantic similarities for search queries [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]. Unfortunately, these methods cannot support user queries in multiple languages. Other supervised approaches for search task identification require collections of manually labeled data to train the models (Wang et al., 2013a;[START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF], which are challenging to create because of the cost of manual labeling (Wang et al., 2013a) and the long-tail nature of search queries [START_REF] Zhang | Generic intent representation in web search[END_REF].

We propose a multilingual method for unsupervised search task identification. The proposed approach combines graph clustering methods [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] with recent general language models [START_REF] Cer | Universal sentence encoder for english[END_REF][START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF] for obtaining query representations in a multilingual semantic vector space. The proposed search task identification approach is independent of user identifiers, enabling the modeling of search tasks in user agnostic or personalized applications. We also address the existing trade-off between accuracy and query time (Völske et al., 2019) that arises when mapping new incoming queries to identified search tasks.

Related limitations

Search logs provide fine-grained details at the query level, enabling the characterization and classification of individual queries according to the information need they are related to [START_REF] Hearst | Search user interfaces[END_REF]. Hence, it is possible to cluster related queries in the search log to model the tasks that users perform on search engines to fulfill their information needs. Existing methods (Section 2.4) like Bestlink SVM (Wang et al., 2013a) and CA-LSTM [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF] require a supervised component to perform task identification. CA-LSTM also employs user identifiers to determine the adjacent queries needed to provide context to the recurrent architecture. However, user agnostic scenarios do not have user identifiers available. Also, the context required from adjacent queries could not be available, especially when dealing with sim-ple search tasks like fact-finding, which tend to have a single query [START_REF] Hearst | Search user interfaces[END_REF]. BRTs [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF] and LDA-Hawkes (Li et al., 2014a) depend on user identifiers and time sessions to compute the user/time affinity. Similarly, some graph clustering methods [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF] use time and user sessions to group the queries before the clustering. Heuristics task identification methods [START_REF] Gayo-Avello | A survey on session detection methods in query logs and a proposal for future evaluation[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF] rely on user information and timestamps to identify the search tasks, along with several manually set thresholds. Furthermore, relying on time sessions to identify tasks [START_REF] Gayo-Avello | A survey on session detection methods in query logs and a proposal for future evaluation[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]) could be misleading. According to multiple analyses of search query logs [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF], users tend to multitask during single time sessions and some complex tasks extend during multiple sessions.

Likewise, both QRY-VEC [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] and BRTs [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF] use a custom training word embedding model. The custom training performed using the tempo-lexical context [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] can avoid topic shifting [START_REF] Rekabsaz | Word embedding causes topic shifting; exploit global context[END_REF], but unfortunately, there are not enough labeled collections to train multilingual word embeddings using such context. Also, the use of an external index [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] requires time-consuming index access at the retrieval model [START_REF] Hagen | From search session detection to search mission detection[END_REF] and similar to the issue present in custom training word embeddings, there are not enough corpus for a multilingual clustering of queries. Furthermore, several methods use cosine similarity between query vectors to prune edges in the clustering graph [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] or compute the query affinities [START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]. However, the angular similarity has a better performance than the cosine similarity in semantic textual similarity (STS) between sentence pairs [START_REF] Cer | Universal sentence encoder for english[END_REF].

Task identification approach

The proposed unsupervised approach uses a multilingual query representation and a graph based clustering method to group queries related to the same search task. In contrast with previous methods [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]Wang et al., 2013a), it supports queries in multiple languages. Also, it does not utilize user identifiers [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF] and has no supervised components [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]Wang et al., 2013a). In this section, we cover the multilingual query representation and explain the graph based clustering method.

Multilingual query representation

Pretrained word embeddings have been released in several languages [START_REF] Grave | Learning word vectors for 157 languages[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. However, using pre-trained word vectors can generate topic shifting [START_REF] Rekabsaz | Word embedding causes topic shifting; exploit global context[END_REF][START_REF] Zhang | Generic intent representation in web search[END_REF] and requires an additional phase to detect the language of the query to correctly select the pre-trained model to compute the multilingual query vector. Instead, Multilingual Universal Sentence Encoder (MUSE) [START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF] provides universal representations for sentence embeddings suited to information retrieval tasks [START_REF] Zhang | Generic intent representation in web search[END_REF].

MUSE has models based on transformers (Vaswani et al., 2017) or convolutional neural networks (CNNs). We use the transformer-based model. It is more computationally expensive than the CNN based model; however, it is more accurate on several tasks, including sentence retrieval, bitext retrieval, and retrieval question answering. The transformer-based model relies on the encoder part of the transformer architecture. It takes into account contextaware embeddings and averages together the results from the encoder to produce one vector per sentence. Training is based on question-answer pairs, translation pairs, and the Stanford Natural Language Inference (SNLI) corpus. Furthermore, MUSE utilizes SentencePiece, a language-independent subword tokenizer, to process the input query text. SentencePiece covers above 99% of possible tokens in all languages. Likewise, MUSE supports queries in sixteen languages: Arabic (ar), Chinese PRC (zh), Chinese Taiwan (zh-tw), Dutch(nl), English(en), German (de), French (fr), Italian (it), Portuguese (pt), Spanish (es), Japanese (ja), Korean (ko), Russian (ru), Polish (pl), Thai (th), and Turkish (tr).

Graph Based Clustering

Existing clustering methods for search task identification rely on lexical similarities [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF] or cosine distances between word embeddings [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF]. However, the angular similarity [START_REF] Cer | Universal sentence encoder for english[END_REF] has been used in recent research [START_REF] Chidambaram | Learning cross-lingual sentence representations via a multi-task dual-encoder model[END_REF][START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF] to better discriminate text representations in natural language processing. In particular, the angular similarity has been found to perform better in STS between sentence pairs than the cosine similarity [START_REF] Cer | Universal sentence encoder for english[END_REF]; thus, we use it to compute the similarity between query pairs. Given two queries q i , q j , with multilingual vector representations v i , v j , the angular similarity S ang is defined as follows [START_REF] Cer | Universal sentence encoder for english[END_REF]:

S ang (v i , v j ) = -arccos v i v j |v i | |v j | (5.1) Algorithm 1 MGBC algorithm Input: Query log Q Output: Task labels L V ← {} , E ← {} , G(V, E) ← (V, E) for all q i ∈ Q do v i ← multilingual vector(q i ) V ← V ∪ {v i } end for for all v i , v j ∈ V do e k ← S ang (v i , v j ) E ← E ∪ {e k } end for for all e k ∈ E do if e k < η then E ← E \ {e k } end if end for for all C i ∈ G(V, E) do task i ← i for all v k ∈ C i do L [v k ] ← task i end for end for
The Multilingual Graph Based Clustering (MGBC) relies on the multilingual query representation to encode queries in the search logs (Algorithm 1). Once the queries are converted into vectors in the multilingual semantic space, a weighted undirected graph G(V, E) is created, where query vectors are nodes in the graph and S ang is the weight for the edges connecting the queries. After creating the fully connected graph, the graph is pruned by filtering out edges with S ang < η, where η is a threshold optimized during the clustering process: η = k/10, 0 < k ≤ 10, k ∈ N. The connected components C in the graph after the pruning process represent the search tasks in the query log. Every connected component receives a unique task label task i , which becomes the label for all the nodes pertaining to the connected component C i [START_REF] Chen | Graph-based clustering for computational linguistics: A survey[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Nascimento | Spectral methods for graph clustering-a survey[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF].

Results and discussion

Following recent work [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF], we use the F β score, with β = 1 for the balanced metric, and β = 0.6, which gives more weight to the precision of the search task identification (Section 2.2.3). The Student's paired t-test provides the test for statistical significance. For evaluating the effectiveness of the proposed approach, we consider a user agnostic search task identification and a personalized search task identification. We also address the existing trade-off when mapping queries to identified search tasks.

S ang (v i , v j ) = -α * arccos v i v j |v i | |v j | + (1 -α) * (S ind ) (5.2)
where α, η are parameters to be optimized during the process of clustering for search tasks. The optimization uses a grid search with α = k/10, η = k/10, where 0 < k ≤ 10, k ∈ N [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF], selecting the model with the best F 1 metric.

Results show that MGBC outperforms the baseline method in search task identification (Table 5.1, Figure 5.1). It gets better performance than both lexically based (QC-WCC) and monolingual query vectors based (QRY-VEC) methods for identifying tasks, highlighting the ability of the multilingual semantic vector space to encode queries for the modeling of search tasks.

Regarding multilingual tests, in addition to the CSTE dataset, we use the Complex User Search Task Analysis (CUSTA) dataset, which has queries mostly in French (Section 2.1). Using Google Cloud Translation API 1 , we translate the CSTE and CUSTA datasets to all the languages supported by MGBC. Running the search task identification method on automatically translated queries enables the assessment of the method in multilingual task identification.

MGBC uses the angular similarity S ang in equation 5.1 for the multilingual tests. In the results for the CSTE dataset (Table 5.2), F 1 metrics vary from 0.429 with the Turkish language to 0.484 with the French language, which are located around the F 1 metric of 0.456 for the English language, the original language of the dataset (Figure 5.2). Similarly, for the CUSTA dataset (Table 5.3), F 1 metrics vary from 0.541 with the Thai language to 0.697 with the Japanese language, which are located around the F 1 metric of 0.656 for the French language, the original language of the dataset. These results reflect the quality of the multilingual semantic space to represent the search tasks. Overall, no drop in performance is observed despite the use of automatic translation, suggesting an adequate performance in the sixteen languages for the search task identification approach. Although there exist variations in results for the different languages, they are explained by the expected differences in the automatic translation results. We use the WSMC12 dataset for the evaluation of the proposed MGBC approach in personalized search task identification. As discussed before (Section 2.1), labels in this dataset correspond to search tasks for a particular user identifier, grouping queries by user sessions. If two users run a query for the same search task, the search task label is not necessarily the same. For instance, the user with identifier 9887420 issued the query "maps" and it was labeled with search task '2'; the user with identifier 1713103 issued the query "maps" as well, but it was labeled with search task '22'. For that reason, the WSMC12 dataset allows the evaluation of the proposed method in a personalized setup.

Personalized search task identification

Along with the QRY-VEC baseline, we include methods depending on user identifiers for the input, namely, LDA-Hawkes, BRTs, and CA-LSTM [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]Li et al., 2014a;[START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF].

Search task identification results show that MGBC matches the clustering performance of CA-LSTM (Table 5.4), even though CA-LSTM is a semisupervised approach, which uses recurrent layers trained in a supervised way. Likewise, MGBC exhibits slight improvements against the other methods used for comparison, though only with p ≤ 0.2 when using the Student's t-test for statistical significance. These results indicate that the proposed method matches the identification performance of existing methods in personalized setups, without requiring any supervised components or user identifiers. As MGBC does not require user identifiers as input to the model, it can perform search task identification in user agnostic scenarios as well as personalized scenarios. Indeed, user agnostic scenarios can be crucial in current efforts to address user privacy concerns in intelligent systems. The same multilingual semantic space for query representation and the S ang similarity in Equation 5.1 enables us to address the existing trade-off when mapping new incoming queries to the identified search tasks. Previously analyzed methods for mapping queries face a trade-off between accuracy and execution time. The most accurate method uses an inverted index approach based on a BM25 retrieval model; however, its average time per query is much slower than a Trie data structure implementation, which is the fastest method (Völske et al., 2019). To address this trade-off, we utilize the Neighborhood Graph and Tree (NGT) approximate nearest neighbor method [START_REF] Iwasaki | Optimization of indexing based on k-nearest neighbor graph for proximity search in high-dimensional data[END_REF], along with S ang and multilingual query vectors. NGT2 is a high-speed approach that supports large volumes of data in high-dimensional spaces.

Three benchmark datasets have been proposed to evaluate the mapping of new incoming queries: the AOLQTM dataset, the TRECQTM dataset, and the WHQTM dataset (Section 2.1). For comparison, we use publicly available implementations for the Trie data structure3 and the BM25 retrieval model4 with default parameters (Völske et al., 2019;[START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF]. Experiments run on a virtual machine instance with 8 CPUs of 3GHz and 60GB of RAM. We compute time per query as the average time for mapping 10 4 queries, while accuracy is measured using a leave-one-out evaluation, independently selecting 100 random queries from the dataset and repeating the evaluation during 50 runs (Völske et al., 2019).

Summary 85

We test several values of k nearest neighbors for NGT (Figure 5.3), finding k = 9 as the best performing setup. NGT is several times faster than the inverted index based on BM25 (Table 5.5), keeping average times per query below half a millisecond. The speedup obtained with NGT does not imply a deterioration in the accuracy metrics for TRECQTM and WHQTM datasets. Also, AOLQTM differences are much lower than the Trie data structure drop in metrics. Similarly, NGT is more accurate than the Trie data structure in all the three datasets; nonetheless, the latter continues to be faster in terms of average time per query.

Summary

The proposed MGBC multilingual search task identification approach enables the modeling of search tasks from query logs, supporting queries in sixteen languages. Experiments show that the proposed approach outperforms baseline identification methods. Also, MGBC is user-independent, enabling its use in both user agnostic and personalized search task identification applications. Moreover, the same multilingual semantic space and query similarity of MGBC can be used with NGT nearest neighbor method to address the existing trade-off when mapping new queries to identified search tasks. NGT provides metrics at the same level of the BM25 retrieval model results; however, it is several times faster, keeping query response times below half a millisecond, a crucial aspect for running on the fly applications for supporting search engine users.

Chapter 6

Modeling User Search Tasks with a Language-agnostic Unsupervised Approach All models are wrong, but some models are useful.

George Box

Although MGBC supports queries in sixteen languages (Chapter 5), users around the world use many more languages when accessing search systems. Also, there are multiple languages with not enough datasets to train language models, arising the need for utilizing language-agnostic representations for queries, representations that can support queries even when the language is not part of the training dataset of the language model. Moreover, as discussed before (Section 2.4), clicked URLs provide a way to model user search intent from the relationship between queries and clicked documents [START_REF] Zhang | Generic intent representation in web search[END_REF]. For example, a similarity metric based on clicked URLs is the second most important in Bestlink SVM, after a similarity metric based on query semantics (Wang et al., 2013a).

Consequently, in this chapter we propose a language-agnostic, user intent aware approach to model search tasks from user interactions with search systems. The proposed approach leverages user intent modeling from clicked query-document pairs, latent representations of queries in a language-agnostic space, and graph-based clustering to model search tasks in an unsupervised approach. Experimental results demonstrate the proposed approach outperforms recent work in search task modeling, supporting user queries in multiple languages. It can also produce search task modeling results in the order of milliseconds, an essential aspect for conversational systems and user support applications requiring realtime results.

Introduction

Conversational AI systems are becoming increasingly popular because of advances in speech recognition, natural language understanding, text-to-speech synthesis, and the availability of digital personal assistants [START_REF] Khatri | Contextual topic modeling for dialog systems[END_REF]Thomas et al., 2020;[START_REF] Khatri | Contextual topic modeling for dialog systems[END_REF]. Personal assistants like Amazon Alexa, Apple Siri, Google Assistant, and Microsoft Cortana are now available in smartphones, tablets, desktops, and dedicated smart speakers [START_REF] Khatri | Contextual topic modeling for dialog systems[END_REF]Zamani and Craswell, 2020). Consequently, the increasing popularity and availability of conversational systems make conversational information seeking a major emerging area of research [START_REF] Anand | Conversational search (Dagstuhl seminar 19461)[END_REF]Zamani and Craswell, 2020).

In conversational information seeking and other search systems, modeling the search tasks that users perform to satisfy their information needs is a crucial step [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]. Search task modeling is a step in the process to make search systems more coherent, natural, engaging, and conversational [START_REF] Khatri | Contextual topic modeling for dialog systems[END_REF][START_REF] Rosset | Leading conversational search by suggesting useful questions[END_REF][START_REF] Khatri | Contextual topic modeling for dialog systems[END_REF]. Multiple user supporting applications benefit from search task modeling, including conversational question suggestion, personalization in e-commerce, product recommendations, query term prediction, query suggestions, query reformulation, and results ranking [START_REF] Hearst | Search user interfaces[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Rosset | Leading conversational search by suggesting useful questions[END_REF][START_REF] Dosso | How to support search activity of users without prior domain knowledge when they are solving learning tasks[END_REF]Völske et al., 2019). Even informative conversations with digital assistants can benefit from correctly modeling the search tasks, as the subjective perception of the quality in the conversation is strongly related to the accurate tracking of the topic [START_REF] Khatri | Contextual topic modeling for dialog systems[END_REF].

As discussed before, users around the world access search systems in multiple languages, making it essential to process users' requests with languageagnostic models. Also, search systems and user supporting applications require realtime responses when processing user information needs. For instance, multimodal search in conversational systems runs multiple processes in parallel, post-processing their outputs to generate a message answering the user request; hence, modeling can not exceed the timeout periods set on the search system (Zamani and Craswell, 2020). Similarly, user clicks are strongly related to the user intent [START_REF] Zhang | Generic intent representation in web search[END_REF]. Different queries with similar clicked URLs can pertain to the same information need [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF], and analyzing clicked URLs can help disambiguate queries (Craswell et al., 2020a).

Our contributions in this chapter are threefold. First, we propose a language-agnostic search task modeling (LASTM) approach to model search tasks from user interactions with search systems. Second, given the relationship between clicked URLs and user intent, we propose a user intent modeling technique leveraging a large scale query -clicked document collection in the query latent space. Third, to enable the utilization of LASTM in conversational search systems and user supporting applications requiring responses on the fly, we propose a realtime method for mapping incoming queries to the modeled user search tasks directly on the query latent space.

Related limitations

User interactions with search systems enable modeling the search tasks that users perform to satisfy their information needs [START_REF] Hearst | Search user interfaces[END_REF]. In particular, search query logs can be mined for search task modeling using methods such as heuristics-based models, semi-supervised clustering, bayesian approaches, and graph-based clustering (Section 2.4). However, most search task modeling methods [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF]Li et al., 2014a;[START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]Wang et al., 2013a) are monolingual. Although MGBC supports several languages through MUSE, it can only process queries in sixteen languages. Additionally, when using ClueWeb12B for calculating query similarities, MGBC can only support user queries in English.

By the same token, most search task modeling methods [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]Li et al., 2014a;[START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] fail to take into account clicked URLs when processing search query logs, even though clicked URLs have a critical correlation to the user intent [START_REF] Zhang | Generic intent representation in web search[END_REF]. Also, conversational information seeking systems and multiple applications supporting users search efforts require results on the fly. Building models from scratch when a user submits a query could create large processing times, forcing search systems to trigger timeout intervals (Zamani and Craswell, 2020). Similarly, waiting for forward queries to provide context [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF] can render models unfeasible in realtime setups. Also, some models requiring user identifiers [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF]Li et al., 2014a;[START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF] can not be used in user-independent modeling scenarios (Craswell et al., 2020a;[START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF].

User search task modeling

LASTM is an unsupervised method that leverages latent representations of queries in a language-agnostic space, user intent modeling from clicked querydocument pairs, and graph-based clustering to model user search tasks. It can also produce a realtime mapping of queries to modeled search tasks. In contrast with previous work [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF][START_REF] Hagen | From search session detection to search mission detection[END_REF]Li et al., 2014a;[START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]Wang et al., 2013a), our proposed approach supports multiple languages through a language-agnostic latent space. The proposed approach is also independent of user identifiers, enabling modeling of search tasks in both user-independent and personalized scenarios. It also differs from prior methods [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF]Li et al., 2014a;[START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Deconstructing complex search tasks: a Bayesian nonparametric approach for extracting subtasks[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] by leveraging clicked URLs to model user intent [START_REF] Zhang | Generic intent representation in web search[END_REF] in the query latent space.

Language-agnostic query representation

As users worldwide submit queries in different languages to satisfy their information needs, Language-agnostic BERT Sentence Embedding (LABSE) [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF] provides the sentence embeddings to represent user queries in a language-agnostic latent space. Using a 12-layer transformer architecture [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF]Vaswani et al., 2017) in a dual configuration, LABSE takes the transformer's hidden state for the last token in the sentence to generate the query representation.

The query representation using LABSE has the ability to perform zeroshot cross-lingual transfer, supporting queries in languages that are not part of the training dataset. When performing tests with the TAOEBA dataset [START_REF] Artetxe | Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond[END_REF], LABSE obtains an 83.7% accuracy, while the baseline Language-agnostic Sentence Representations (Artetxe and Schwenk, 2019) gets 65.5%, even though more than 30 languages in the TAOEBA dataset were not part of the LABSE training data [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF].

We use the cosine proximity [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] to compute the similarity between query representations in the language-agnostic latent space. Formally, given a pair of queries q i , q j with latent representations u i , u j , the similarity between query representations S lat is calculated as follows [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]:

S lat (u i , u j ) = u i u j |u i | |u j | (6.1)

User intent modeling

User clicks play a critical role in modeling user intent -the information need the user wants to satisfy by performing the search task [START_REF] Zhang | Generic intent representation in web search[END_REF]. Query term match between queries for the same information need can be very low; even different queries pertaining to the same search task can have similar clicked URLs [START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Zhang | Generic intent representation in web search[END_REF]. Also, analysis of clicked URLs can help disambiguate queries, revealing which documents users clicked when performing their search tasks (Craswell et al., 2020a).

To model user intent, we use the Open Resource for Click Analysis in Search (ORCAS) (Craswell et al., 2020a), a collection containing 18.8 million clicked document -query pairs for 10.4 million unique queries. Clicked docu-The other possibility is to compute the inner products faster, improving the scoring rate.

A strong alternative to improve the scoring rate is quantization. It makes scoring data points faster, improving the rate of inner product calculations. It also uses the memory bandwidth more efficiently because it can handle more data points per memory access, maximizing CPU utilization. Morevoer, quantization produces effiicency gains in terms of storage size, using more efficiently the disk or memory space where the data points are stored.

Prior research in quantization focused on optimizing the reconstruction error of the quantization method for all points in the database. However, it is better to assign a higher weight to the most relevant items while safely reducing the weight for irrelevant items. This intuition produces a weighted error optimization, commonly known as anisotropic vector quantization, where items close to the query receive a higher weight in the error computations than items orthogonal to it. By doing so, anisotropic vector quantization improves the relevance of MIPS results. ScaNN uses anisotropic vector quantization, improving over NGT (Section 5.4.3) and several other approximate nearest neighbor methods [START_REF] Guo | Accelerating large-scale inference with anisotropic vector quantization[END_REF]. Therefore, we use ScaNN to retrieve results from the ORCAS document collection.

Even though ORCAS has queries exclusively in English, doing MIPS directly on the language-agnostic latent space enables user intent modeling in any language LABSE can support. Hence, we can leverage the existing relationship between clicked URLs and user intent [START_REF] Zhang | Generic intent representation in web search[END_REF] by searching the D M database.

Formally, given a database D M = {m i } i=1,2,...,n formed from a clicked query-document dataset D Q with n data points, where each data point m i ∈ R p is the latent representation of the query q ∈ D Q in the p-dimensional language-agnostic latent space, we want to find the k most relevant documents {d j } j=1,2,...,k ∈ D M for the user query u ∈ R p . Therefore, we search for the k points with the maximum inner product with the user query u as follows [START_REF] Guo | Accelerating large-scale inference with anisotropic vector quantization[END_REF]:

M IP S(D M , u) = {d j } j=1,2,...,k = arg max mi∈D M u, m i (6.2)
Given a user query pair q i , q j with latent representations u i , u j , the similarity based on user intent S int is calculated using the Jaccard coefficient for the top thousand relevant documents in the database D M [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]:

D i = M IP S(D M , u i ) (6.3) D j = M IP S(D M , u j ) (6.4) S int (u i , u j ) = |D i ∩ D j | |D i ∪ D j | (6.5)
6.3 User search task modeling 93

Unsupervised search task modeling

We now integrate user intent modeling and language-agnostic query representations with graph-based clustering [START_REF] Chen | Graph-based clustering for computational linguistics: A survey[END_REF] to model search tasks (Algorithm 2). First, we encode queries in the latent space (Section 6.3.1); every query embedding becomes a node in the weighted graph. Then, we compute the similarities between pairs of queries to create the edges of the weighted graph. The similarity between queries S qry is a convex combination of the similarity in the latent space S lat and the similarity based on user intent S int . Given a pair of queries q i , q j with latent representations u i , u j , query similarity S qry is calculated as follows (Section 5.4):

S qry (u i , u j ) = α * S lat (u i , u j ) + (1 -α) * S int (u i , u j ) (6.6)
After finishing edge weight calculations, we prune the weighted graph, deleting edges with S qry < η. The resulting connected components C in the graph constitute the search tasks, so we assign a unique task label task i to every connected component. All the queries pertaining to a connected component receive the same task label. A grid search (Figure 6.2) optimizes parameters η and α, using η = k/10, α = k/10, 0 < k ≤ 10, k ∈ N [START_REF] Chen | Graph-based clustering for computational linguistics: A survey[END_REF][START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF].

Realtime mapping of new queries

Most search systems and user supporting applications require results in realtime. Applications like contextual topic modeling in conversational search [START_REF] Khatri | Contextual topic modeling for dialog systems[END_REF], query suggestion, or query reformulation can not afford to wait for large processing times. It is essential to return an answer in a few milliseconds. Hence, once the user performs a search request, we map the new incoming query to the labels extracted with Algorithm 2 so that we can model the search task in realtime. To do the mapping, we use the same MIPS method with anisotropic vector quantization [START_REF] Guo | Accelerating large-scale inference with anisotropic vector quantization[END_REF] that we used in Section 6.3.3.

The search task database maps the latent representation of the queries in the search log Q L to the extracted task labels L T . Formally, given a database Q T = {m i } i=1,2,...,n formed from the search query log Q L with search task labels L T returned from Algorithm 2, where each datapoint m i ∈ R p is the latent representation of the query q ∈ Q L in the p-dimensional languageagnostic space. For an incoming query q i , we compute the latent representation u i ; then, we retrieve the search task labels T of the k closest queries in the language-agnostic latent space using MIPS:

T = M IP S(Q T , u i ) (6.7)
Once we have the search task labels T of the k closest queries, we return the task label with the highest number of occurrences in T .

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Results and discussion

In this section, we analyze LASTM in user independent search task modeling and realtime mapping of incoming queries. Following previous work [START_REF] Du | CA-LSTM: Search task identification with context attention based lstm[END_REF], we calculate model performance with the F β score. We consider both β = 1.0 and β = 0.6 (Section 5.4

Search task modeling

The CSTE dataset and the CUSTA dataset are used for experiments. As a baseline, we use MGBC (Section 5.3), calculating metrics for all the languages supported by the baseline. Queries in the CSTE dataset are in English, while queries in the CUSTA dataset are mostly in French, with very few English entries (Section 2.1). Hence, we perform machine translation with the Google Cloud Translation API4 for evaluating LASTM in all the languages supported by MGBC. The proposed approach improves the search task modeling performance of the baseline method in the two datasets used for testing. Using the CSTE dataset (Table 6.1), LASTM surpasses MGBC in all the languages supported by the baseline, obtaining up to 10.9% (p ≤ 0.05) improvement in the F 1 score for the Turkish language; similarly, LASTM obtains better F 0.6 scores in fourteen out of sixteen languages, getting an improvement of up to 11.2% (p ≤ 0.05) in the Turkish language. Furthermore, the monolingual QRY-VEC method, which supports queries in English, obtains an F 1 score of 0.538 and an F 0.6 score of 0.488 [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]. Consequently, there is no loss in modeling performance when comparing LASTM to the QRY-VEC method.

Language

For the CUSTA dataset (Table 6.2), we observe improvements in fourteen out of the sixteen languages supported by MGBC; LASTM generates up to 9.2% (p ≤ 0.05) improvement in the F 1 score for the German language and up to 18.4% (p ≤ 0.05) improvement in the F 0.6 score for the Dutch language. Both the similarity between query representations S lat and the similarity based on user intent S int contribute to the search task modeling results ( effectively relies on the two similarities to compute the edges for the weighted graph.

From a language coverage perspective, the query representation for LASTM is trained with 109 languages and can perform zero-shot cross-lingual transfer to multiple more languages [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF]. In contrast, the baseline only supports sixteen languages, making LASTM language accessibility at least seven times larger when considering training languages only. When comparing LASTM to monolingual models, there is an improvement of two orders of magnitude in language accessibility (Figure 6.4). The improvements in modeling performance and language accessibility highlight the importance of considering user intent along with language-agnostic query representation for modeling search tasks.

Mapping of incoming queries

To analyze the performance of LASTM for mapping new incoming queries, we run the mapping method using the three benchmark datasets previously proposed for query-task mapping (Section 2.1): the AOLQTM dataset, the TRECQTM dataset, and the WHQTM dataset (Section 2.1). Additionally, we use a leave-one-out evaluation, independently selecting one hundred random queries from the dataset and repeating the evaluation for fifty runs. Experiments run on a virtual machine instance with 8 CPUs of 3GHz and 60GB of RAM. Metrics include accuracy, F 1 , F 0.6 , and query time. To measure query time, we take the average time for mapping a single query, using 10 4 mappings to compute the average (Völske et al., 2019). As a baseline, we use the MGBC approach for query task mapping. MGBC combines the Neighborhood Graph and Tree approximate nearest neighbor method [START_REF] Iwasaki | Optimization of indexing based on k-nearest neighbor graph for proximity search in high-dimensional data[END_REF] with the MUSE latent space for query encoding (Section 5.4.3). For reference, we also include results using the Trie5 data structure and the BM256 retrieval model (Völske et al., 2019;[START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF]. Figure 6.5 depicts the optimization experiments for the number of top k results from ScaNN to consider. After running tests for k = [1,3,5,7,9,11,13], we found that top k = 7 results from ScANN generates the optimal configuration, providing the best results for task mapping while keeping the time per query under a millisecond (Table 5.5). Low response time is an essential aspect for applications supporting users in realtime setups. Long answer times could affect the interaction of the search system with the users, especially in conversational and multimodal search systems, where a post-processing step is required to generate a response to the user request [START_REF] Khatri | Contextual topic modeling for dialog systems[END_REF]Zamani and Craswell, 2020). Similarly, long answer times could trigger inter- nal timeout intervals (Zamani and Craswell, 2020), forcing search systems to ignore search task mapping results while doing internal post-processing.

Dataset

Results from query-task mapping show that the proposed method surpases MGBC in two out of the three benchmark datasets (5.5). For the TREC-based dataset, LASTM improves the F1 score of the baseline by 2.4% (p ≤ 0.05), while keeping processing times under a millisecond. For the AOL-based dataset, LASTM surpasses the baseline method, obtaining a 7.0% improvement in the F 1 score (p ≤ 0.05); likewise, LASTM obtains similar results to BM25, but it is faster when comparing to the BM25 implementation used for experiments. For the WikiHow-based dataset, LASTM underperforms MGBC and BM25 (Table 5.5). Regarding the number of user queries per task, we find that the TREC-based dataset has an average of 28 user queries per search task, while the WikiHow-based dataset has an average of 2 user queries per task. Hence, the WikiHow-based dataset contains mostly simple tasks, which users can solve with a few queries [START_REF] Hearst | Search user interfaces[END_REF]. Task mapping results suggest that LASTM is better than the baseline and reference methods when mapping search tasks containing multiple queries, while MGBC is better when mapping simple search tasks in realtime.

Summary

In this chapter, we proposed LASTM, an unsupervised method for modeling search tasks from user interactions with search systems. The proposed model outperforms a state-of-the-art baseline both in modeling performance as well as the number of languages it can support, highlighting the importance of language-agnostic latent spaces for query representation and the importance of considering clicked URLs to model user intent. Also, it is independent of user identifiers, enabling modeling search tasks in user-independent or personalized applications. The modeling performance of LASTM, its languageagnostic capacity, and its ability to support realtime modeling can benefit search systems and user supporting applications, constituting an essential step in the effort to make search more coherent, conversational, engaging, and natural.

Chapter 7

Conclusions

It is a peculiarity of man that he can only live by looking to the future.

Viktor Frankl

In this dissertation, we proposed four models that learn search patterns from users interactions with search systems, namely, the Language-agnostic Search Task Modeling (LASTM) approach, the Multilingual Graph Based Clustering (MGBC) method for search task identification, the Recurrent Deep Clustering (RDC) model to extract users' search tasks, and a recurrent neural architecture for segmenting search query logs.

The recurrent neural architecture learns to detect search task boundaries in pairs of subsequent queries, determining if adjacent queries in chronologically ordered query logs represent a task change or not, and taking as input the query and its timestamp only. It does not rely on the context provided by surrounding queries to achieve its maximum modeling performance. Also, it is possible to fine-tune the model, allowing the processing of small query logs. This is especially important given the scarcity of publicly available labeled datasets from search systems. Once trained, it is several times faster than a heuristics-based baseline. Additionally, as the proposed architecture does not require surrounding queries for context, it can provide segmentation results for tasks with one or few queries. The mean number of queries per task in datasets from widely used search engines is low, and information needs like fact-finding and other simple tasks are usually solved with only one query.

RDC extends the foregoing recurrent architecture to propose a model for search task extraction. Taking the recurrent architecture as the encoder for the user queries, RDC implements a dual-channel configuration that allows it to simultaneously learn query latent representations and cluster queries into groups of search tasks. The learning of query representations is performed using back translation, a strong data augmentation technique that renders negligible the effects of pretraining RDC. For that reason, the supervised pretraining can be discarded, providing an unsupervised method for search task modeling. On top of the unsupervised nature of the proposed method, it is also parametric, providing a fixed-sized query encoder regardless of the number of queries in the search system log.

MGBC provides an unsupervised, non-parametric model to identify user search tasks, combining a multilingual latent space for query representation with graph-based clustering, and using the angular distance in the latent space to group related queries. MGBC supports realtime modeling, as the same multilingual semantic space and query similarity is used to address the existing trade-off when mapping new queries to identified search tasks. For that reason, MGBC keeps query response times below half a millisecond, a crucial aspect for running on the fly applications for supporting search system users. Likewise, MGBC is multilingual, facilitating the identification of search tasks in sixteen languages, including the most used ones like Chinese, English, Arabic, French, and Spanish. Moreover, as MGBC is independent of user identifiers, it can perform search task modeling in user agnostic or personalized scenarios.

Finally, LASTM models user search tasks by leveraging a languageagnostic semantic space for query representation, along with user search intent from the relationship between queries and clicked documents. As user intent modeling is also performed in the language-agnostic semantic space, LASTM can use both semantic relatedness and user search intent to group similar queries in the search system logs. The query representation can realize zero-shot transfer learning, which allows it to support queries in languages that were not part of the training collection for the semantic space. Similar to MGBC, it is possible to get realtime modeling results by mapping incoming queries to modeled search tasks; also, the proposed model is independent of user identifiers, enabling its use in user agnostic or personalized scenarios.

Indeed, a crucial aspect in the proposed models is user privacy, which is part of broader efforts in ethics for intelligent systems. None of the proposed models rely on user identifiers as direct input to the method. Though the neural model for search task segmentation requires the queries in the log to be ordered by timestamps and grouped by user, this ordering does not require direct use of the identifiers as input to the neural model. Similarly, RDC is user agnostic. It does not require user identifiers either for deep clustering or self-training with back translation. MGBC and LASTM are user agnostic as well. They take only queries as input, and the relationship between queries and clicked documents we use to model user search intent does not utilize any user identifiers.

Another crucial aspect is realtime response times, especially for interactive search. Except for RDC, all proposed models take into account the need for realtime results in modern search systems. Most search systems need to provide results in realtime. Conversational information seeking can also im-plement time thresholds when aggregating results from multiple systems, so that they can provide timely responses during the interaction with users. Exceeding such limits render the system response unusable or can introduce unnecessary delays in the process users carry out to fulfill their information needs. Therefore, MGBC, LASTM, and the proposed method for search task segmentation provide modeling results under a millisecond. The neural architecture for search task segmentation can be even faster than the heuristicsbased baseline once the neural network is trained. Moreover, both MGBC and LASTM rely on query-task mapping with fast approximate nearest neighbors methods for modeling search tasks in realtime.

Furthermore, language accessibility is at the core of the MGBC and LASTM approaches. As discussed before, most labeled and unlabeled datasets publicly available are in English, but users worldwide access search systems in many languages. Therefore, MGBC represents queries in a multilingual semantic space, supporting queries in sixteen languages. LASTM goes even further, using a language-agnostic semantic space for representing queries and modeling user search intent. When compared against existing monolingual modeling methods, LASTM, which supports more than a hundred languages, represents an improvement of two orders of magnitude in language accessibility.

Overall, our proposed models provide state-of-the-art performance, improving on existing methods for modeling search patterns, and taking into account user privacy, realtime responses, and language accessibility. Proposed models can be part of many user assisting applications and retrieval models, underpinning efforts of search systems to assist users while they carry out their information seeking. They can also be used to design search systems based on tasks, as well as to aid digital assistants to follow the topic in conversational search. Assisting users while they run the sequence of steps they devise to solve their information needs allows them to deal with the mental overload that information seeking requires.

Perspectives for further research

In our proposed non-parametric methods, we use existing ground-truth labels to perform the grid search, producing models as accurate as possible with respect to the ground-truth. Nonetheless, it would be interesting to explore ways to optimize the model using metrics that are independent of groundtruth labels. One possibility is the use of cluster validity indices, like the STR index [START_REF] Starczewski | A new validity index for crisp clusters[END_REF] or the Silhouette index [START_REF] Rousseeuw | Silhouettes: a graphical aid to the interpretation and validation of cluster analysis[END_REF]. Another possibility is the combined use of several cluster validity indices to optimize the non-parametric models [START_REF] Saini | Automatic scientific document clustering using self-organized multi-objective differential evolution[END_REF], as long as it guarantees that queries inside a search task cluster are as close as possible, while search task clusters remain as separated as possible between them.

Another interesting research direction is the use of generative methods for deep clustering [START_REF] Aljalbout | Clustering with deep learning: Taxonomy and new methods[END_REF][START_REF] Min | A survey of clustering with deep learning: From the perspective of network architecture[END_REF]. RDC outperforms existing deep clustering methods by leveraging bidirectional RNNs and selftraining to learn query representations. But comparing RDC with generative alternatives, like Variational Deep Embedding [START_REF] Jiang | Variational deep embedding: an unsupervised and generative approach to clustering[END_REF] or Deep Adversarial Subspace Clustering [START_REF] Zhou | Deep adversarial subspace clustering[END_REF], can give us hints of the impact generative architectures can have on modeling results. If the impact is positive, it is possible to devise ways to incorporate generative alternatives to perform the clustering. Similarly, it is possible to replace the clustering layer in RDC for graph-based clustering alternatives. Doing so allows the simultaneous learning of query representations and the optimization of graphs for clustering. Besides, discrete architectures (Kaiser and Bengio, 2018;Kaiser et al., 2018) for clustering could improve the modeling of search tasks because query logs can have thousands of search tasks, and quantization is already used in extreme classification methods with a large number of classes [START_REF] Guo | Accelerating large-scale inference with anisotropic vector quantization[END_REF].

Moreover, it is widely accepted that users multitask when fulfilling their information needs [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Mehrotra | Extracting hierarchies of search tasks & subtasks via a Bayesian nonparametric approach[END_REF][START_REF] Dosso | How to support search activity of users without prior domain knowledge when they are solving learning tasks[END_REF]. Most publicly available datasets with timestamps show this behaviour (Section 2.1), except for the the Complex User Search Task Analysis (CUSTA) dataset [START_REF] Dosso | How to support search activity of users without prior domain knowledge when they are solving learning tasks[END_REF]. But why does the CUSTA dataset shows little to no multitasking? There are some hints about this lack of multitasking. First, "task sets" allow users to program their memory to perform a certain task, even if it is not what users normally do. For example, when looking at a book page, a user will normally read the content, but the user can program its memory to instead count the number of f's present on it [START_REF] Kahneman | Thinking, fast and slow[END_REF]. Thus, users can perform all the steps to solve a particular search task in tandem, even though they normally multitask. Second, a continuous train of thought requires effort. So, users should exert self-control to resist the urge to stop the mental effort that a complex task requires (Kahneman, 2013), but if the user interface allows for "cues" like social network notifications or mail access, users will have a harder time keeping focused in a particular search task [START_REF] Oakley | A mind for numbers: How to excel at math and science (even if you flunked algebra)[END_REF][START_REF] Williams | Supporting workplace detachment and reattachment with conversational intelligence[END_REF]. Understanding this lack of multitasking can improve the design of search systems, allowing users to maintain focus, an essential behavior when solving complex search tasks.

Furthermore, conversational information seeking is increasingly becoming a major area of research [START_REF] Anand | Conversational search (Dagstuhl seminar 19461)[END_REF]Zamani and Craswell, 2020). Among the many research problems in conversational search, two research questions arise. First, what approaches adapt to evolving user information needs, while at the same time, leverage past interactions with users? As discussed before, mining search query logs allow search pattern modeling, enabling multiple applications for assisting users while they satisfy their information needs. Assisting applications like conversational question suggestion [START_REF] Rosset | Leading conversational search by suggesting useful questions[END_REF] and clarifying question asking [START_REF] Aliannejadi | Asking clarifying questions in open-domain information-seeking conversations[END_REF] can benefit from learned models. Though foregoing proposed models learn from user interactions, they use an offline training scheme. To update the models, it is necessary to retrain or fine-tune them with the updated set of interactions. However, offline learning is not dynamic. For conversational search, systems should be interactive, dynamically adapting in realtime to evolving user information needs. Conversely, learning from scratch while interacting with users can cause conversational search systems to generate unwanted responses. In such scenarios, users will likely abandon the search system (Zamani and Craswell, 2020). Hence, developing methods that learn when interacting with users in realtime, but take advantage of past interactions, is critical for information seeking in a conversational way.

Second, how to identify if users have experience in the subject matter of the information need they want to fulfill? Experience in a particular subject matter has an impact on the way users satisfy their information needs. Experienced users will use a different way to satisfy their information needs than users who need to learn about the subject matter, particularly when solving complex search tasks [START_REF] Hearst | Search user interfaces[END_REF][START_REF] Karanam | The role of domain knowledge in cognitive modeling of information search[END_REF]. Discovering whether the user has experience or not allows search systems to adapt their responses. That adaptation can be instrumental for proposing clarifying questions [START_REF] Rosset | Leading conversational search by suggesting useful questions[END_REF], presenting relevant passages [START_REF] Chang | Pre-training tasks for embedding-based large-scale retrieval[END_REF][START_REF] Dai | Context-aware term weighting for first stage passage retrieval[END_REF], or helping inexperienced users explore while carrying out the information seeking process [START_REF] Dosso | How to support search activity of users without prior domain knowledge when they are solving learning tasks[END_REF][START_REF] Hearst | Search user interfaces[END_REF]. Accordingly, developing approaches to dynamically adapt to evolving users' information needs, as well as identifying the experience of users in the topic of search, can be a crucial part of the evolution of search systems towards conversational information seeking.

2. 1

 1 Dual encoder configuration for self-training neural architectures using data augmentation . . . . . . . . . . . . . . . . . . . . . 2.2 Recent machine learning models for search task identification . 2.3 Recent methods for search log segmentation . . . . . . . . . . . . . . . . 3.1 Segmentation architecture with a bidirectional recurrent layer and an attention mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Performance of the segmentation architecture considering different numbers of recurrent units . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Training curves for the segmentation architecture . . . . . . . . . . . . 4.1 Recurrent neural network encoder for learning latent representations of queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Pretraining and clustering phases for search task extraction using a dual encoder configuration. . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Calculation of self-training loss using the dual encoder configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Grid search for MGBC parameters . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Threshold search for MGBC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Test NGT in the multilingual semantic space with multiple values of k nearest neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 User intent database with clicked document -query pairs . . . . . 6.2 Grid search for LASTM parameters . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Parameter for the convex combination of query similarities . . . 6.4 Improvement in language accessibility when comparing LASTM against existing work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Search task mapping results in the language-agnostic latent space for the three benchmark datasets . . . . . . . . . . . . . . . . . . . . . xiii List of Tables 2.1 Search query collections for modelling several users' search patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Features for publicly available datasets . . . . . . . . . . . . . . . . . . . . . 2.3 Reference results for mapping queries to search tasks . . . . . . . . 2.4 Confusion matrix for a binary classification model . . . . . . . . . . . 2.5 Search task identification performance for existing methods . . . 2.6 Search log segmentation results for several existing query log datasets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Entries in the WSMC12 dataset from an example user . . . . . . . 3.2 Segmentation performance for WSMC12 and CSTE datasets . . 3.3 Model performance for the WSMC12 dataset with additional adjacent queries for recurrent architectures . . . . . . . . . . . . . . . . . 3.4 Model metrics using additional adjacent queries in the search log for providing context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Fine-tuning the recurrent architecture to realize segmentation on the CSTE dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Clustering performance for RDC and baseline methods . . . . . . . 4.2 Comparison between LSTM and GRU cells in RDC . . . . . . . . . . Chapter 1

  Fig.2.1: Dual encoder configuration[START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Yang | Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax[END_REF] for self-training neural network architectures using data augmentation techniques on input samples.

  Fig. 3.1: Segmentation architecture with a bidirectional recurrent neural layer along with an attention mechanism. Input samples include the query pairs and the time span between them. Query examples in the image come from the first two entries in the CSTE dataset.

  Fig.3.2: Performance of the segmentation architecture considering different numbers of recurrent units for the bidirectional layer. Results were calculated using LSTM units and WSMC12 dataset.

  Fig. 3.3: Training curves for the bidirectional architecture. (a) Loss curves for both training and testing sets. (b) Accuracy and F 1 results for the testing set during the training of the system.

  Fig. 4.1: Recurrent neural network encoder for learning latent representations of queries. The encoder comprises a bidirectional recurrent layer, an attention mechanism, and a projection head.

Fig. 4

 4 Fig. 4.2: Pretraining and clustering phases for search task extraction using a dual encoder configuration.

  Fig. 4.3: Calculation of self-training loss using the dual encoder configuration, together with back translation to produce realistic augmented queries.

  Fig. 5.1: Grid search for α, the parameter for the convex combination of similarities, and η, the threshold for the MGBC clustering method.

Fig. 5

 5 Fig. 5.2: Threshold η search for the MGBC graph clustering method, considering English, French, and Turkish languages.

  Fig. 5.3: Test NGT in the multilingual semantic space with multiple values of k nearest neighbors. Results for AOLQTM, TRECQTM, and WHQTM datasets include (a) Accuracy (b) F 1 .

F1

  Fig.6.2: Grid search for α, the parameter for the convex combination of similarities, and η, the threshold for the LASTM clustering method.

  Fig. 6.3: α values for the convex combination of query similarities. Results include values after finishing the grid search for CSTE and CUSTA datasets.

Fig. 6

 6 Fig.6.4: Improvement in language accessibility when comparing LASTM against the baseline, RDC, and existing monolingual methods for search task modeling.

  Fig. 6.5: Search task mapping results in the language-agnostic latent space for AOLQTM, TRECQTM, and WHQTM datasets. Results include several values of top k from the ScaNN index, considering (a) Accuracy (b) F 1 .

Table 2

 2

.2: Common features in search log datasets. Only publicly available datasets are considered.

Table 2

 2

		Method	Accuracy Query time
		Trie	0.69	0.46ms
	AOLQTM	MinHash LSH WMD	0.66 0.67	2.42ms 7.16s
		ElasticSearch BM25	0.78	2.80ms
		Trie	0.66	0.51ms
	TRECQTM	MinHash LSH WMD	0.68 0.73	2.50ms 9.24s
		ElasticSearch BM25	0.80	2.95ms
		Trie	0.48	0.33ms
	WHQTM	MinHash LSH WMD	0.41 0.55	2.28ms 22.65s
		ElasticSearch BM25	0.63	4.21ms

.3: Reference results for mapping queries to search tasks.

  P are true positives, F P are false positives, T N are true negatives, and F N are false negatives. Table2.4 has a tabular representation of T P , F P , T N , and F N for a supervised binary case(Tan et al., 2018).

	20			2 Background
				13)
	P recision =	T P T P + F P	(2.14)
		Recall =	T P T P + F N	(2.15)
	where T Predicted	Predicted
		Class 0	Class 1
	Actual Class 0	T P		F P
	Actual Class 1	F N		T N
	Table 2.4: Confusion matrix for a binary classification model

Table 2

 2

		Method	Result
		QC-WCC (Lucchese et al., 2013)	0.471
	CSTE	QRY-VEC word2vec (Sen et al., 2018)	0.473
		QRY-VEC tempo-lexical (Sen et al., 2018)	0.538
		QC-WCC (Lucchese et al., 2013)	0.812
		QC-HTC (Lucchese et al., 2013)	0.800
	TGSST	DD-CRP (Mehrotra et al., 2016)	0.845
		BRT (Mehrotra and Yilmaz, 2017)	0.845
		QRY-VEC word2vec (Sen et al., 2018)	0.837
		QRY-VEC tempo-lexical (Sen et al., 2018)	0.840
		QC-HTC (Lucchese et al., 2013)	0.851
	WSMC12	LDA-Hawkes (Li et al., 2014a)	0.871
		BRT (Mehrotra and Yilmaz, 2017)	0.878
		CA-LSTM (Du et al., 2018)	0.883
		QC-HTC (Lucchese et al., 2013)	0.821
	SUQLST	LDA-Hawkes (Li et al., 2014a)	0.837
		BRT (Mehrotra and Yilmaz, 2017)	0.843
		CA-LSTM (Du et al., 2018)	0.851

.5: Search task identification performance for several query log datasets, including F 1 results for semisupervised, unsupervised, and Bayesian approaches.

Table 3

 3

	. of queries	2	4	6	8	10
	CA-LSTM	0.863	0.878	0.904	0.898	0.903
	BiRNN LSTM	0.929	0.921	0.920	0.915	0.912
	BiRNN GRU	0.935	0.920	0.927	0.919	0.917

.3: Segmentation model accuracy for the WSMC12 dataset, with additional adjacent queries for recurrent architectures. Differences between results have p ≤ 0.05 for the Student's t-test.

Table 3

 3

		No. of queries Accuracy	F 1
		2	0.935	0.882
		4	0.920	0.853
	Backward -forward	6	0.927	0.868
		8	0.919	0.852
		10	0.917	0.844
		1	0.933	0.877
		2	0.934	0.878
	Backward only	3	0.927	0.863
		4	0.928	0.865
		5	0.924	0.859
	No context	0	0.937	0.884

.4: Model metrics using additional adjacent queries in the search log for providing context. We consider several numbers of adjacent queries in the WSMC12 dataset, selecting both backward and forward contexts or backward context only.

  to create the augmented samples. Table 4.1: Clustering performance for CSTE and TRECQTM datasets, including RDC and other methods used for comparison. Differences in RDC results against all baseline methods have p ≤ 0.05 for the Student's t-test.

	Dataset	Method	Pretraining	ACC	NMI	ARI
		k-means	None	0.395	0.670	0.231
		DBSCAN None	0.199	0.343	0.027
		HAC	None	0.407	0.719	0.310
		DEC	Autoencoder	0.362	0.684	0.345
		IDEC	Autoencoder	0.347	0.681	0.348
	CSTE	SymDEC Autoencoder DAC None	0.337 0.318	0.652 0.644	0.325 0.344
		Chimera	None	0.387	0.707	0.339
		RDC	Sequence	0.420	0.735	0.355
		RDC	Segmentation	0.408	0.730	0.354
		RDC	Intent	0.331	0.641	0.334
		RDC	None	0.415	0.734	0.355
		k-means	None	0.219	0.535	0.050
		DBSCAN None	0.026	0.105	0.001
		HAC	None	0.276	0.613	0.086
		DEC	Autoencoder	0.097	0.419	0.019
		IDEC	Autoencoder	0.097	0.418	0.018
	TRECQTM	SymDEC Autoencoder DAC None	0.104 0.095	0.396 0.368	0.022 0.025
		Chimera	None	0.214	0.523	0.061
		RDC	Sequence	0.285	0.594	0.094
		RDC	Segmentation	0.246	0.566	0.080
		RDC	Intent	0.187	0.508	0.055
		RDC	None	0.284	0.590	0.095

  Table5.4: Clustering performance for the WSMC12 dataset, which has search task labels per user. The ClueWeb12B dataset provides the index similarity for comparison to the baseline method.

	Identification method	F 1	F 0.6
	LDA-Hawkes	0.871	0.864
	BRTs	0.878	0.874
	CA-LSTM	0.883	0.887
	QRY-VEC	0.851	0.880
	MGBC	0.884	0.913

3 Mapping queries to search tasks

	5.4 Results and discussion				83
	5.4.Dataset	Method Accuracy	F 1	F 0.6	Query time
		Trie	0.693	0.543	0.543	0.029ms
	AOLQTM	BM25	0.809	0.689 0.689	947ms
		NGT	0.751	0.608	0.607	0.308ms
		Trie	0.650	0.519	0.518	0.030ms
	TRECQTM	BM25	0.791	0.688	0.688	2532ms
		NGT	0.804	0.705 0.704	0.299ms
		Trie	0.471	0.310	0.311	0.032ms
	WHQTM	BM25	0.621	0.453	0.454	6.572min
		NGT	0.648	0.481 0.481	0.368ms

Table 5 .

 5 5: Model metrics for mapping queries to search tasks. Three datasets are considered to evaluate the performance of the NGT approach. Differences against Trie have p ≤ 0.05 for the Student's t-test.

  ). Moreover, we use open source implementations for ScaNN 1 , NetworkX 2 in graph-based clustering, and the publicly available pretrained model for LABSE 3 .

	Language	ISO 639-1	F 1			F 0.6
			MGBC	LASTM MGBC	LASTM
	Arabic	ar	0.447	0.521	0.395	0.490
	Chinese PRC	zh	0.480	0.539	0.473	0.513
	Chinese Taiwan zh-tw	0.482	0.540	0.476	0.515
	Dutch	nl	0.449	0.534	0.431	0.511
	English	en	0.456	0.538	0.437	0.512
	German	de	0.450	0.533	0.432	0.511
	French	fr	0.484	0.539	0.547	0.512
	Italian	it	0.452	0.540	0.434	0.517
	Portuguese	pt	0.458	0.537	0.438	0.514
	Spanish	es	0.450	0.541	0.432	0.516
	Japanese	ja	0.453	0.522	0.436	0.495
	Korean	ko	0.451	0.523	0.396	0.501
	Russian	ru	0.449	0.533	0.429	0.508
	Polish	pl	0.460	0.536	0.524	0.512
	Thai	th	0.444	0.522	0.427	0.489
	Turkish	tr	0.429	0.538	0.401	0.513
	Table 6.1: Search task modeling results for the CSTE dataset in all the lan-
	guages supported by the MGBC method. Differences between MGBC and
	LASTM results have p ≤ 0.05 for the Student's t-test.

Table 6 .

 6 3: Realtime mapping of queries to search tasks. Differences against baseline MGBC results have p ≤ 0.05 for the Student's t-test.

		Method Accuracy	F 1	F 0.6	Query time
		Trie	0.693	0.543	0.543	0.029ms
	AOLQTM	BM25 MGBC	0.809 0.751	0.689 0.689 0.608 0.607	947ms 0.308ms
		LASTM	0.802	0.678	0.677	0.490ms
		Trie	0.650	0.519	0.518	0.030ms
	TRECQTM	BM25 MGBC	0.791 0.804	0.688 0.705	0.688 0.704	2532ms 0.299ms
		LASTM	0.822	0.729 0.728	0.481ms
		Trie	0.471	0.310	0.311	0.032ms
	WHQTM	BM25 MGBC	0.621 0.648	0.453 0.481 0.481 0.454	6.572min 0.368ms
		LASTM	0.558	0.389	0.389	0.982ms

https://github.com/lelugom/search

https://www.internetworldstats.com/stats.htm

https://www.worldwidewebsize.com

https://www.cnet.com/news/10-most-downloaded-apps-of-the-decade-facebookdominated-2010-2019

4https://https://arstechnica.com/gadgets/2021/02/the-worlds-second-most-populardesktop-operating-system-isnt-macos-anymore

http://www.sogou.com

https://www.aol.com

Background

https://www.google.com

https://www.bing.com

https://docs.scipy.org/doc/numpy-1.15.0/user/basics.broadcasting.html

Extracting search tasks from search query logs

https://scikit-learn.org

https://github.com/XifengGuo/DEC-DA

https://github.com/HongtaoYang/DAC-tensorflow

https://github.com/leichtrhino/ChimeraNet

http://nlp.stanford.edu/data/glove.42B.300d.zip

https://cloud.google.com/translate

A multilingual approach for unsupervised search task identification

https://cloud.google.com/translate/docs

https://github.com/yahoojapan/NGT

https://github.com/google/pygtrie

https://github.com/nhirakawa/BM25

Modeling User Search Tasks with a Language-agnostic Approach

https://github.com/google-research/google-research/tree/master/scann

https://networkx.github.io

https://tfhub.dev/google/LaBSE/1

https://cloud.google.com/translate

https://github.com/google/pygtrie

https://github.com/nhirakawa/BM25

Conclusions

Acknowledgements

. Likewise, much gratitude to the Université Thank you very much as well to rapporteurs Pr Catherine BERRUT and Pr Guillaume CLEUZIOU.

Furthermore, much gratitude to the Agence Nationale de la Recherche (ANR) for supporting this research work through project CoST 2019 -2022, token, many thanks to all my friends and colleagues from the Institut de Recherche en Informatique de Toulouse. I sincerely appreciate your friendship, cordiality, and generosity.

Last but not least, I owe much appreciation to my family for their unconditional support, love, and encouragement. v

(3):687-700. Sun, A. and Lou, C.-H. (2014). Towards context-aware search with right click. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 847-850. Tamine, L., Melgarejo, J. L., and Pinel-Sauvagnat, K. (2020). What can task teach us about query reformulations? In European Conference on Information Retrieval, pages 636-650. Springer. Tan, P.-N., Steinbach, M., Karpatne, A., and Kumar, V. (2018). Introduction to Data Mining. Pearson Education, London,UK, second edition. Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. (2020). Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006. Thomas, P., McDuff, D., Czerwinski, M., and Craswell, N. (2020). Expressions of style in information seeking conversation with an agent. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1171-1180. Van der Maaten, L. and Hinton, G. (2008). Visualizing data using t-sne.

Journal of machine learning research, 9(11). Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems, pages 5998-6008. Venkatesh, A., Khatri, C., Ram, A., Guo, F., Gabriel, R., Nagar, A., Prasad, R., Cheng, M., Hedayatnia, B., Metallinou, A., et al. (2018). On evaluating and comparing open domain dialog systems. arXiv preprint arXiv:1801.03625. Vinh, N. X., Epps, J., and Bailey, J. ( 2010). Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. Journal of Machine Learning Research, 11(Oct):2837-2854. Völske, M., Fatehifar, E., Stein, B., and Hagen, M. (2019). Query-task mapping. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 969-972. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2019a). GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of ICLR.

User agnostic search task identification

The dataset for evaluating the clustering approach in a user agnostic scenario is the CSTE dataset. As a baseline, we use the state-of-the-art QRY-VEC [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF] method, an unsupervised task identification method that uses custom trained tempo-lexical embeddings, averaging the embeddings for each word in the query to compute a single vector per query. We also include results from QC-WCC [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF].

To compare to the QRY-VEC method, we use the same index similarity S ind than the baseline [START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF], which is based on the ClueWeb12B dataset [START_REF] Callan | The Lemur project and its ClueWeb12B dataset[END_REF]. We adjust the angular similarity S ang in equation 5.1 by the use of a convex combination of both angular and index similarities. The similarity between queries q i , q j with multilingual vectors v i , v j becomes [START_REF] Lucchese | Discovering tasks from search engine query logs[END_REF][START_REF] Sen | Tempo-lexical context driven word embedding for cross-session search task extraction[END_REF]: [START_REF] Craswell | Overview of the TREC 2019 deep learning track[END_REF]. Queries are encoded in a languageagnostic semantic space using LABSE [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF].

ments are represented using the TREC document identifier in the TREC Deep Learning document collection [START_REF] Craswell | Overview of the TREC 2019 deep learning track[END_REF]. We encode queries in ORCAS in the language-agnostic latent space [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF], creating a user intent database D M with clicked document -query pairs (Figure 6.1). To retrieve the most relevant documents for a given user query in the database, we use Scalable Nearest Neighbor (ScaNN) [START_REF] Guo | Accelerating large-scale inference with anisotropic vector quantization[END_REF], a state-of-the-art method for large-scale retrieval tasks.

In retrieval tasks, it is usual to learn latent representations of queries and documents in the same latent space. Then, we encode the query and look for documents in the latent space by performing the inner product against the query, selecting the documents that are as close as possible to the query. Such documents will have the maximum possible inner product against the query, representing the most relevant results from the documents' database. This procedure is commonly known as MIPS (Maximum Inner Product Search).

Computing the inner product of a query against all the documents in the database can become unfeasible, especially for large databases. Thus, there exist two possibilities to retrieve relevant results more quickly. One possibility is partitioning the latent space so that the number of inner product operations is reduced to the data points in the closest partition. For partitioning, there exist methods like local sensitive hashing, graph search, and tree search.