Neural text generation for query expansion in information retrieval

Keywords: Information Retrieval, Neural text generation, Neural language models, Query expansion, GPT2, Data-augmentation

Expanding users' query is a well-known way to improve the performance of document retrieval systems. Several approaches have been proposed in the literature, and some of them are considered as yielding state-of-the-art results in Information Retrieval. In this paper, we explore the use of text generation to automatically expand the queries. We rely on a well-known neural generative model, OpenAI's GPT-2, that comes with pre-trained models for English but can also be fine-tuned on specific corpora. Through different experiments and several datasets, we show that text generation is a very effective way to improve the performance of an IR system, with a large margin (+10 %MAP gains), and that it outperforms strong baselines also relying on query expansion (RM3). This conceptually simple approach can easily be implemented on any IR system thanks to the availability of GPT code and models.

INTRODUCTION

In the Information Retrieval (IR) traditional setting, a user expresses his information needs with the help of a query. Yet, it is sometimes difficult to match the query with the documents, for instance because of the query vocabulary may differ from the documents. Especially when the query is short, the performance of the system is usually poor, as it is difficult to detect the precise focus of the information need, and the relative importance of the query terms.

Query expansion aims at tackling these problems by transforming the short query into a larger text (or set of words) that makes it easier to match documents from the collection. The main difficulty of query expansion is obviously to add only relevant terms to the initial query. Several techniques have been proposed in the literature, based on linguistic resources (e.g. synonym lists) or based on the documents themselves (e.g. pseudo-relevance feedback).

In this paper, we explore the use of recent text generation models to expand queries. We experimentally demonstrate that the recent advances in neural generation can dramatically improve ad-hoc retrieval, even when dealing with specialized domains. More precisely, through different experiments, we show that:

(1) texts artificially generated from the query can be used for query expansion; (2) this approach does not only provide new terms to the query, but also a better estimate of their relative weights; (3) in addition, it also provides a better estimate of the importance (i.e. weight) of original query words; [START_REF] Charles | The TREC 2005 Terabyte Track[END_REF] this approach can also be used on specialized domains.

The paper is structured as follows. After a presentation of the related work (Sect. 2), Section 3 details the different components of our approach. Several experiments are then detailed in Section 4. Last, some concluding remarks are given in Section 5.

RELATED WORK

Query expansion is a well-established technique to try to improve the performance of an IR system. Adding new terms to the query is expected to specifically improve recall, yet, since the query is, hopefully, better formulated, it may also improve the top rank results and be beneficial to precision. One might classify the existing automatic approaches based on the resources used to expand the query.

Expansion with external resources.

One obvious way to expand a query is to add semantically related terms to it (synonyms or sharing other semantic relations like hyponyms, quasi-synonyms, meronyms...). Existing lexical resources can be used to add, for each query term, a list of semantically related terms; yet, one has to deal with different problems: existence of lexical resources for the collection language, or for the specific domain of the collection, choice of the appropriateness of certain relations, need of sense disambiguation for polysemous words... WordNet [START_REF] Miller | WordNet: An On-Line Lexical Database[END_REF] is among the best-known resources for English (general domain language) and have been used with mitigated results at first [START_REF] Voorhees | Query Expansion Using Lexical-semantic Relations[END_REF], but later shown to be effective [6, inter alia].

Expansion with pseudo-relevance feedback

Another category of studies considers only a small set of documents to help to expand or reformulate the query. To be automatic, they replace the user feedback by the hypothesis that the best ranked documents retrieved with the original query are relevant and may contain useful semantic information [START_REF] Ruthven | A survey on the use of relevance feedback for information access systems[END_REF]. It is interesting to note that in this case, not only semantically relevant terms are extracted, but also distributional/statistical information on them and on the original query terms. In this category, Rocchio, developed in the 60's for vector space model was among the first one popularized [START_REF] Christopher | Introduction to Information Retrieval[END_REF]. One of the current best known approach is RM3, which was developed in the framework of language model based IR systems [START_REF] Abdul-Jaleel | UMass at TREC 2004: Novelty and HARD[END_REF]. It is often reported to yield the best results in ad-hoc retrieval tasks, even compared with recent neural models [START_REF] Lin | The Neural Hype and Comparisons Against Weak Baselines[END_REF]. Neural approaches have also been proposed to integrate pseudo-relevance feedback information [START_REF] Li | NPRF: A Neural Pseudo Relevance Feedback Framework for Ad-hoc Information Retrieval[END_REF], yet, as it is reported by the authors, the results are still lower than traditional models with query expansion.

Expansion with collection-based resources.

Distributional thesauri have also been exploited to enrich queries. Since they can be built from the document collection (or from a large corpus with similar characteristics), they are suited to the domain, the vocabulary... Traditional techniques to build these thesauri have obtained good results for query expansion [START_REF] Claveau | Direct vs. indirect evaluation of distributional thesauri[END_REF]. Neural approaches, that is, word embedding approaches are now widely used to build such semantic resources. In the recent years, static embeddings (word2vec [START_REF] Mikolov | Distributed Representations of Words and Phrases and their Compositionality[END_REF], Glove [START_REF] Pennington | GloVe: Global Vectors for Word Representation[END_REF] or FastText [3] to name a few) were also used in IR, in particular to enrich the query. Indeed, these trainable dense representations make it easy to find new words that are semantically close to query words.

Even more recently, dynamic word representations obtained with transformer-based architectures, such as BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] or GPT [START_REF] Radford | Language Models are Unsupervised Multitask Learners[END_REF], have been proposed. They build a representation for each word according to its context, and this ability have been exploited to obtain competitive results in IR tasks [7, 12, inter alia]. BERT has been also used for query expansion in the framework of a neural IR system [START_REF] Naseri | CEQE: Contextualized Embeddings for Query Expansion[END_REF][START_REF] Zheng | BERT-QE: Contextualized Query Expansion for Document Re-ranking[END_REF], for instance based on reranking [START_REF] Padaki | Rethinking Query Expansion for BERT Reranking[END_REF]. While these studies show promising results, it is worth noting that the RM3 method for pseudo-relevance feedback, while simpler, still competes with or even outperforms most of these neural-based models (both Glove-based or equivalent and BERT-based and equivalent), as noted in [START_REF] Naseri | CEQE: Contextualized Embeddings for Query Expansion[END_REF].

Text generation

In this paper, we propose to use constrained text generation to expand queries. In this approach, the original query is used as a seed (or prompt) for a generative model which will output texts that are, hopefully, related to the query. While text generation with language model is not new, the performance of neural models based on transformers [START_REF] Vaswani | Attention is All you Need[END_REF] makes this task realistic.

In this paper, we use the Generative Pre-Trained Transformers (GPT) models. These neural models are learned by auto-regression, which means that they are unsupervisedly trained to predict the next token (word) given the previous ones. They are built from stacked transformers (precisely, decoders) that are trained on a large corpus. The second version, GPT-2 [START_REF] Radford | Language Models are Unsupervised Multitask Learners[END_REF], contains between twelve layers (smallest model) up to 60 layers (largest model) of transformers with twelve self-attention heads of 64 dimensions. Given that, GPT-2 has 1.5 billion parameters for its largest pre-trained model, released in Nov. 2019. It has been trained on a specially crafted corpus named WebText which contains more than 8M documents from Reddit (i.e. mostly English and general domain language such as discussion on press articles).

A newer version, GPT-3, has been released in July 2020; it is much more larger (175 billion parameters) and outperforms GPT-2 on any tested task. Yet, the access given to this model (through a restricted API), the size of this model (which makes fine-tuning impossible) and the problems arising on how to engineer the prompt to perform the expected generation task, made GPT-2 preferable for this piece of work.

To the best of our knowledge, using generative models to expand queries has not been explored before. Yet, using GPT for data augmentation in other NLP tasks has recently received lots of attention. For instance, generation is exploited in relation extraction tasks [START_REF] Papanikolaou | DARE: Data Augmented Relation Extraction with GPT-2[END_REF], or text classification tasks like sentiment classification [START_REF] Kumar | Data Augmentation using Pre-trained Transformer Models[END_REF], re-admission prediction and phenotypic classification [START_REF] Amin-Nejad | Exploring Transformer Text Generation for Medical Dataset Augmentation[END_REF] or fake news detection [START_REF] Claveau | Detecting fake news in tweets from text and propagation graph: IRISA's participation to the FakeNews task at MediaEval[END_REF].

GENERATED QUERY EXPANSION

In this section, a complete overview of the proposed expansion approach is first given. Additional details about the generative models and their adaptation are given in Section 3.2. The IR systems used in our experiments are presented in Section 3.3.

Overview of our approach

As it was previously explained, our approach is very simple as it relies on existing tools and techniques. From a query, multiple texts are generated by a GPT-2 model using the query as the seed. Note that the generation process is not deterministic (if some parameters such as top_k and temperature are correctly set; see next sub-section), and thus, even with the same seed, the generated texts are different. The generation of a large number of texts allows to have a large coverage of the vocabulary related to the query and a good estimation of the relative importance of words by their frequency in the generated texts. In the experiments reported below, 100 texts of 512 words (more precisely, tokens) per query are generated, unless specified otherwise (see Section 4.6). These texts are concatenated and considered as the expansion for the query. In our experiments, this new, very large, query is then fed to a simple BM25+ IR system (which are still state-of-the-art models, even compared with pure neural IR systems [START_REF] Thakur | BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models[END_REF]), but it could obviously be used in any other IR system. Figure 1 presents the whole process; details for each part are given in the following sub-sections. Note that the only task done on-line (at query time) is the generation, which corresponds to the inference step of the neural language models to generate texts. The training or fine-tuning of the model itself is done off-line.

An example of a text generated from a query (query 701 from the GOV2 collection) is presented in Fig. 2. As one can see, the generated text, while completely invented (note the barrel prices), is relevant for the query. It contains many terms, absent from the original query, that are more or less closely related to the information need. More specifically, this generated text provides:

• synonyms and orthographic variants (United States for the query term U.S.), • meronyms-metonyms (barrel for oil),

• hypernyms (energy for oil),

• more generally any paradigmatic relations (consumer, producer for industry), • and syntagmatic relations (production for oil).

It is worth noting that such texts also give a valuable information about the relative frequency of each terms (contrary to thesauri or embeddings); this frequency information is an interesting cue to value the importance of a term. U.S. oil production has been declining steadily for decades and it is not expected to reverse. In fact, some argue that it may even get worse. The long-term trend is for oil production to decline at a rate of about 1 percent per year. With production of about 8 million barrels per day now, there is no way the United States can replace its current output. The U.S. oil boom was a result of an energy revolution in the 1970s that led to increased oil production, and a significant change in the global oil market. The U.S. now produces about 2.3 million barrels of oil per day, the highest it has been in over 30 years. The United States is now the world's largest oil producer and the fourth largest oil exporter. What happened? When oil prices peaked in the 1970s, the United States was the world's largest oil producer. But over the next several decades, the United States' oil production began to decline. The decline was most pronounced in the 1980s, when the United States began to fall behind other oil producing countries. The oil price decline in the 1970s was not entirely voluntary. The United States was producing less oil and exporting more oil than it was consuming. The Federal Reserve controlled the amount of dollars in the Federal Reserve's reserves, so the United States was not exporting as much oil as it was producing. The decline in U.S. oil production was a result of the declining price of oil. The price of oil had declined from $8 per barrel in 1973 to $2.50 per barrel in 1977. In 1979, the price of oil reached a high of $15.75 per barrel. By 1983, the price of oil had fallen to $4.65 per barrel. By 1986, the price of oil had fallen to $1.86 per barrel. By the end of the 1980s, the price of oil had fallen to $1.24 per barrel. The decline in oil prices was a direct result of the energy revolution in the 1970s. The United States was the world's largest oil producer, but the United States was also the world's largest consumer of oil. When oil prices fell, so did the cost of producing oil.

Pre-trained models, fine-tuning and parameters

GPT-2 comes with several pre-trained models, having different size in terms of parameters (from 124M to 1.5B). As it was previously said, their training data was news-oriented general domain language. The largest model was used for two of the tested collections (see below). While these all-purpose models are fine for IR collections whose documents are also general domain language, it may not be appropriate for domain-specific IR collections. In the experiment reported in the next section, we use the ohsumed collection, consisting of medical documents. For this collection, we have fine-tuned the GPT-2 355M model on the documents of the collection in order to adapt the language model to the specific medical syntax and vocabulary. We use LM 𝑤 𝑞 (𝑡) 𝑐 (𝑡, 𝑞) 𝜇 > 0 a smoothing parameter Table 1: IR models (weighting functions of terms in the query and the document) for BM25+ [START_REF] Lv | Lower-bounding Term Frequency Normalization[END_REF][START_REF] Robertson | Okapi at TREC-7: Automatic Ad Hoc, Filtering, VLC and Interactive[END_REF] and Language modeling with Dirichlet smoothing LM [START_REF] Zhai | A study of smoothing methods for language models applied to ad hoc information retrieval[END_REF]; 𝑐 (𝑡, 𝑑) is the number of occurrences of term 𝑡 in document 𝑑, 𝑑 𝑓 (𝑡) is the document frequency of 𝑡, 𝑑𝑙 (𝑑) is the length of document 𝑑, 𝑎𝑣𝑑𝑙 is the average document length, 𝐶 is the collection, 𝑁 is the number of documents in the collection HuggingFace. The fine-tuning was stopped after reaching a plateau in terms of perplexity of the model (in practice it corresponds to 250,000 samples processed). Other parameters (batch size, optimizer, learning rate...) were set to their defaults. Although a larger set of medical documents could be used (from Pubmed ® for instance), this small fine-tuned model is expected to be more suited to generate useful documents to enrich the query.

Concerning the generation of texts, for reproducibity purposes, here are the main GPT-2 parameters used (please refer to Hug-gingFace Transformers 1 and GPT-2 documentations 2 ): length=512, temperature=0.5, top_p=0.95, top_k = 40.

IR Systems

In the experiments reported in the next section, we use two IR models. The first one is BM25+ [START_REF] Lv | Lower-bounding Term Frequency Normalization[END_REF], a variant of BM25 [START_REF] Robertson | Okapi at TREC-7: Automatic Ad Hoc, Filtering, VLC and Interactive[END_REF]. The parameters 𝑘 1 , 𝑘 3 , 𝑏 and 𝛿 were kept at their default value (resp. 1.2, 1000, 0.75, 1). It is implemented as a custom modification of the gensim toolkit [START_REF] Řehůřek | Software Framework for Topic Modelling with Large Corpora[END_REF]. The second IR model is Language modeling with Dirichlet smoothing [START_REF] Zhai | A study of smoothing methods for language models applied to ad hoc information retrieval[END_REF] as implemented in Indri 3 [START_REF] Metzler | Combining the Language Model and Inference Network Approaches to Retrieval[END_REF][START_REF] Strohman | Indri: A language-model based search engine for complex queries (extended version)[END_REF]. The smoothing parameter 𝜇 is set to 2 500 (a usual default value). Both models are regarded as yielding state-of-the-art performance for bag-of-words representation [START_REF] Lin | The Neural Hype and Comparisons Against Weak Baselines[END_REF]. Their RSV function can be written:

𝑅𝑆𝑉 (𝑞, 𝑑) = 𝑡 ∈𝑞 𝑤 𝑞 (𝑡) • 𝑤 𝑑 (𝑡)
with 𝑤 𝑞 (𝑡) the weight of term 𝑡 in query 𝑞 and 𝑤 𝑑 (𝑡) the weight in document 𝑑, as illustrated in Tab. 1 (from [START_REF] Lv | Lower-bounding Term Frequency Normalization[END_REF]). For RM3 expansion, we also rely on the Indri implementation; the results reported in the next section corresponds to the best performing parameters tested for each collection (number of documents considered for pseudorelevance feedback, number of terms kept, mixing parameter 𝜆).

1 https://huggingface.co/transformers 2 https://github.com/openai/gpt-2 3 https://www.lemurproject.org/indri/

EXPERIMENTS

This section is dedicated to the experimental validation of the proposed query expansion approach. After a presentation of our experimental settings, we show the results on several collections (Sect. 4.2 and 4.3). We also present additional experiments about the interest of the frequency given by the generated text (Sect. 4.5) and about the influence of the number of generated texts (Sect. 4.6).

Experimental settings

Four IR collections are used in our experiments: Tipster [START_REF] Harman | Overview of the Second Text Retrieval Conference (TREC-2)[END_REF] , Robust [START_REF] Voorhees | Overview of the TREC 2004 Robust Track[END_REF], GOV2 [START_REF] Charles | The TREC 2005 Terabyte Track[END_REF] and ohsumed [START_REF] Hersh | OHSUMED: An Interactive Retrieval Evaluation and New Large Test Collection for Research[END_REF]. Some basic statistics are given in Tab. Tipster was used in TREC-2. The documents are articles from newspaper, patents and specialized press (computer related) in English. The queries are composed of several parts, including the query itself and a narrative detailing the relevance criteria; in the experiments reported below, only the actual query part is used.

The Robust collection consists of 528,000 news articles from Tipster disks 4 and 5; there 250 topics (301-450, 601-700). As for the published work, we use the titles as queries.

GOV2 is a large collection of Web pages crawled from the .gov domain and used in several TREC tracks. In the experiments reported below, 150 queries from TREC 2004-2006 ad-hoc retrieval tasks are used; as for Tipster, only the actual query part is used (i.e. description and narrative fields are not included in the query).

Ohsumed contains bibliographical notices from Medline and queries from the TREC-9 filtering task. Its interest for our experiments is that it deals with a specialized domain, hence it contains a specific vocabulary.

Performance are assessed with standard scores: Precision at different thresholds (P@x), R-precision (R-prec), Mean Average Precision (MAP) on 1,000 first retrieved documents. When needed, a paired t-test with 𝑝 = 0.05 is performed to assess the statistical significance of the difference between systems.

General domain language

Tables 3 and 4 respectively present the results for the generallanguage collections Tipster and GOV2. For comparison purposes, we indicate the results of BM25+ with and without RM3 expansion, Indri's Language Model (LM) with and without RM3 expansion. Note that the RM3 expansion are strong baselines, as they achieve state-of-the-art performance, even compared with neural techniques (incl. static embedding or contextualized embeddingbased expansion) [START_REF] Naseri | CEQE: Contextualized Embeddings for Query Expansion[END_REF][START_REF] Yang | Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models[END_REF]. Moreover, the results reported here are for the best-performing RM3 parameters: for BM25+, this is 100 terms from the top 30 documents on Tipster and 80 terms on the top 10 documents on GOV2 ; for LM on Tipster, this is 100 terms from the top 20 documents, and 100 terms for the top 10 documents GOV2. The statistical significance is computed by comparing with the BM25+ + RM3 (noted with † ) and LM+RM3 baselines (noted with a * ).

On both collections, and on every performance measure, expanding the queries with the generated texts brings important gains compared with the system without expansion. Also, our approach outperforms RM3 expansion in almost every situation, and with a large margin on MAP, R-prec and precision on the top-ranked documents (P@5, P@10).

Specialized language

The same setting is used on the ohsumed collection. For these medical-oriented IR dataset, we report two versions of our approach: one is using the pre-trained model as before, and one relies on a model fine-tuned on the documents of the collection. The best performing setting for RM3 is 100 terms for the top 15 documents for BM25+ and 80 terms for the top 10 documents for LM. The results are reported in Tab. 5.

Here again, the GPT-based expansion significantly improves the results of the IR system and outperforms RM3 expansion. Yet, the gains are lower than for the two previous collections. This difference can be explained by the following factors:

(1) the queries are longer more complex and more specific (as can be seen in Tab. 2, few documents are relevant); (2) the generation model is not sufficiently suited to the documents.

Concerning this latter reason, we can indeed see the interest of finetuning the generation model, but better results may be obtained by using a larger set of medical documents, or adopting different fine-tuning parameters (in particular the number of epoch/samples processed, see Sect. 3.2). Unfortunately, defining a priori the best parameters for our IR task is not possible and the cost of the finetuning process makes it impossible to test a wide range of possible values.

Comparison with other expansion approaches

In this section, we position our approach with respect to other expansion approaches that have been proposed in the literature.

For this experiment, we use the Robust dataset to make our results comparable with published results; we also re-employ the same evaluation performance scores as in [START_REF] Naseri | CEQE: Contextualized Embeddings for Query Expansion[END_REF]. For this experiment, we use the Pyserini framework 4 which comes with the pre-indexed Robust collection, we thus rely on its BM25 and RM3 implementations. Our GPT2-based expansions have been generated as before with the pre-trained (non fine-tuned) large model. In Table 6 we report the performance of our approach, the results of CEQE and its variants [START_REF] Naseri | CEQE: Contextualized Embeddings for Query Expansion[END_REF], as well as the various baselines proposed in [START_REF] Naseri | CEQE: Contextualized Embeddings for Query Expansion[END_REF], including an expansion based on Glove embeddings (static-embed) [START_REF] Diaz | Query expansion with locally-trained word embeddings[END_REF] and a variant that has its vocabulary limited to terms appearing in the pseudo-relevance feedback documents (static-embed-PRF); see [START_REF] Naseri | CEQE: Contextualized Embeddings for Query Expansion[END_REF] for details. The BM25 parameters are 0.9 and 0.4 as recommended by Pyserini for Robust; in our experiments, the best performing RM3 parameters are 70 terms from the top 10 documents.

There are some slight differences between our baselines (BM25, BM25+RM3) and those of [START_REF] Naseri | CEQE: Contextualized Embeddings for Query Expansion[END_REF], maybe due to differences in the tokenizing and stemming processes of the IR frameworks. Nonetheless, the GPT-based expansion yields the best results, with significant improvement over the BM25+RM3 baseline and several points above the best CEQE configuration.

Leveraging the term importance in the generated texts

One of the interest of having complete texts that are generated is that we can collect information on the relative importance of words, to the contrary of expanding queries with a thesaurus. To observe the impact of the number of occurrences in the generated texts, we evaluate the effect of keeping the 𝑘 most frequent terms of the generated texts and either weighting them by their frequency (as done usually by BM25) or by giving a fixed weight (1/𝑘). The results for different values of 𝑘 are presented in Fig. 3. One can observe that adding terms to the query with a fixed weight slightly improves the MAP, but most of the gain is indeed brought by a proper weighting based on the frequency of the term in the generated documents. This is a big advantage of having proper texts, generated and tailored for the query, instead of related terms taken from a thesaurus or computed from an embedding. It also worth noting that the maximum MAP is reached with about 100 terms; this is interesting for a fair comparison with RM3 since this is also the typical numbers of terms yielding the best results in our experiments.

In the next experiment, we also examine how the generated texts can help to re-weight the initial query terms. The idea is that queries are often too short to get relevant information about the relative importance of each query term (often, each of them occurs only once, that is 𝑐 (𝑡, 𝑞) = 1). In this experiment, there is no

Number of generated texts

Text generation with large neural models has a non negligible computing cost: with our settings, for one query, about 40 texts (of 512 tokens) are simultaneously generated in about 5 seconds on one Tesla V100 GPU card. Thus, it is interesting to see how many generated texts are necessary and more generally what is the influence of the size of the expansion on the IR performance. Of course, the size of the generated texts (can be set as a parameter of the generation process) is also to be considered. In Figure 4, the MAP obtained for up to 100 generated texts per query is presented. For each number 𝑛 of texts, 5 runs are performed (ie. 5 sets of 𝑛 texts are generated for each query, and the 5 MAP are averaged). One can observe that a plateau is rapidly reached at around 20 texts per query; it represents about 10,000 words. It means that good performance can be yielded with a limited time and computing cost.

CONCLUSIVE REMARKS AND FORESEEN WORK

Neural approaches are increasingly used in IR, with mitigated results, especially when compared with "traditional" bag-of-word approaches [START_REF] Lin | The Neural Hype and Comparisons Against Weak Baselines[END_REF][START_REF] Thakur | BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models[END_REF][START_REF] Yang | Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models[END_REF]. Here, the neural part is successfully used outside of a "traditional" IR system (but note that it could be used with any IR systems, since it simply enriches the query). The expansion approach presented in this paper is simple and easy to implement (thanks to the availability of the GPT models and code) while offering impressive gains. The same approach could be used with other IR systems (neural or not), other approaches to enrich the query, and more sophisticated post-processing (such as re-ranking techniques).

Lot of parameters could be further optimized, especially on the GPT model side (to influence the "creativity" of the text generation), and the fine-tuning capabilities should also be explored more thoroughly (influence of bigger specialized corpus if available, precise mix between pre-trained and fine-tuning, etc.). The recent availability of GPT-35 makes it possible to even get greater gains thanks to the alleged high quality of its outputs, but necessitates to change the fine-tuning paradigm (used for OHSUMED here) to a prompt engineering paradigm. Last, let us note that the generation time of the artificial texts and the necessary GPU power may appear as a problem for some industrial contexts. Yet, these costs are not untractable (see Sect. 4.6) and can be dealt with one GPU card and a few seconds of additional processing time. Moreover, model reduction techniques, such as distillation [START_REF] Sanh | Dis-tilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter[END_REF], or TPUs could further reduce this generation time and its computational cost.

This whole approach also offers many research avenues: in this work, we have used text generation as a way to perform data augmentation on the query side, but it could also be used to augment the representation of the documents (even if in practice, the cost is still prohibitive on large collection, as seen with the doc2query and docTTTTTquery models [START_REF] Nogueira | From doc2query to docTTTTTquery[END_REF][START_REF] Nogueira | Document Expansion by Query Prediction[END_REF]). All machine-learning (neural or not) approaches based on pseudo-relevance feedback to train their model could instead use similar text generation with the advantage that they would not be limited by the number of potential relevant documents in the shortlist. And of course, similar dataaugmentation strategy could be used for other tasks than document retrieval.

More fundamentally, the recent improvements of text generation also question the relevance of the document retrieval task. Indeed, it is possible to envision systems that will be able to generate one unique document answering the user's information need, similarly to question-answering. If the generative model is trained on the document collection, the generated document will serve as a summary (which is one of the popular applications of GPT-x models) of the relevant documents. Yet, the current limitations of the models tested in this paper make them far from being suited for this ultimate task: the generated documents do deal with the subject of the query, and thus use a relevant vocabulary, but do not provide accurate, factual information (as seen in the Example in Fig. 2 about the price of oil barrels).

Figure 1 :

 1 Figure 1: Overview of the proposed query expansion approach based on artificially generated texts.

Figure 2 :

 2 Figure 2: Example of a document generated with the pre-trained GPT-2 large model from the text seed "U.S. oil industry history" (query 701 from the GOV2 collection)

Figure 3 :

 3 Figure 3: MAP (%) according to size of the expansion (number of terms), terms with fixed weight or weight depending on their frequency in the generated documents; Tipster collection

Figure 4 :

 4 Figure 4: MAP (%) vs. number of generated texts (average over 5 runs; max length of texts = 512 tokens); Tipster collection

Table 2 :

 2 2. Statistics on the IR collections used

		Tipster Robust GOV2 ohsumed
	nb of documents	170,000 528,000 25M	350,000
	nb of queries	50	250	150 En	En
	avg nb of relevant
	doc per query	849	65.5	179	21

Table 4 :

 4 Performance (%) on the GOV2 collection with query expansion; best results in bold, statistical significance over BM25+ + RM3 and LM+RM3 resp. noted † and * † 24.65 * † 34.65 * † 29.41 * 24.31 11.42

		MAP	R-Prec	P@5	P@10 P@20 P@100
	BM25+	18.27	19.94	31.88	26.04	20.50	9.48
	BM25+ + RM3	21.44	22.72	32.67	28.70	23.73	10.82
	LM	17.61	20.35	29.31	24.06	19.21	9.28
	LM + RM3	20.80	22.54	30.89	26.83	22.18	10.51
	BM25+ and expansion (no fine-tuning)	21.60	23.75	33.47 *	27.62	22.92	11.16
	BM25+ and expansion (fine-tuning)	23.07

*

Table 5 :

 5 Performance (%) on the ohsumed collection with query expansion; best results in bold, statistical significance over BM25+ + RM3 and LM+RM3 resp. noted † and *

	Model	P@20 nDCG@20 MAP Recall@100 Recall@1000
	BM25	36.57	41.93	25.74	41.65	69.33
	BM25 + RM3	39.98	45.17	30.69	46.10 ‡	75.88 ‡
	Static-Embed	36.75	42.85	26.15	42.17	71.25
	Static-Embed-PRF	37.81	44.00	27.03	43.24	72.31
	CEQE-Centroid	39.22	44.62	30.19 ‡	45.93 ‡	76.53 † ‡
	CEQE-MulPool	38.47	43.60	28.45 ‡	45.17 ‡	74.35 ‡
	CEQE-MaxPool	40.40 ‡	45.87	30.86 ‡	46.51 ‡	76.89 † ‡
	CEQE-MaxPool(fine-tuned) 39.86 ‡	45.28	30.71 ‡	46.47 ‡	76.26 ‡
	BM25	35.88	41.93	25.15	40.80	69.26
	BM25 + RM3	39.52	44.89	29.85	45.97	76.97
	BM25 + GPT	41.71 *	48.32 *	30.96 *	47.48 *	79.85 *

Table 6 :

 6 Performance (%) on Robust; first rows are results from[START_REF] Naseri | CEQE: Contextualized Embeddings for Query Expansion[END_REF] ( † and ‡ indicate statistical significance over BM25 + RM3 and Static-Embed-PRF, respectively), last rows are our results with the Pyserini framework ( * indicates statistical significance over BM25 + RM3) query expansion, since only the initial query terms are kept, but we use their frequency in the generated texts to compute the BM25+ weight 𝑤 𝑞 . The results are reported in Tab. 7 and compared with the usual weighting (i.e. BM25 weight with the frequency from the original query). It appears that there is indeed a small improvement of the MAP (+2% absolute gain), that is more noticeable at high document-cutoff values. These two experiments demonstrate the MAP R-Prec P@5 P@10 P@20 P@100 BM25+ 25.06 32.16 95.60 92.60 89.70 73.64 BM25+ with re-weighting 27.12 33.22 96.80 94.20 92.70 77.12

Table 7 :

 7 Performance (%) on Tipster with re-weighting query words

avg size of queries 6.74 2.76 3.15 7.24 language En En

https://github.com/castorini/pyserini/

https://github.com/openai/gpt-3


Antoine Chaffin
email: antoine.chaffin@imatag.com

Vincent Claveau
email: vincent.claveau@irisa.fr

Ewa Kijak
email: ewa.kijak@irisa.fr

PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided Decoding

Large pre-trained language models (LM) based on Transformers allow to generate very plausible long texts. In this paper, we explore how this generation can be further controlled to satisfy certain constraints (eg. being non-toxic, positive or negative, convey certain emotions, etc.) without fine-tuning the LM. Precisely, we formalize constrained generation as a tree exploration process guided by a discriminator that indicates how well the associated sequence respects the constraint. Using a discriminator to guide this generation, rather than fine-tuning the LM, in addition to be easier and cheaper to train, allows to apply the constraint more finely and dynamically. We propose several original methods to search this generation tree, notably the Monte Carlo Tree Search (MCTS) which provides theoretical guarantees on the search efficiency, but also simpler methods based on re-ranking a pool of diverse sequences using the discriminator scores. These methods are evaluated on two types of constraints and languages: review polarity and emotion control in French and English. We show that MCTS achieves state-of-the-art results in constrained generation, without having to tune the language model, in both tasks and languages. We also demonstrate that our other proposed methods based on re-ranking can be really effective when diversity among the generated propositions is encouraged. only be used through access to logits (e.g. GPT-3 API), this makes our approach more plug and play than PPLM.

, which motivated our approach.

Introduction

Generative language models exist for a long time, but with advent of the transformer architecture [START_REF] Vaswani | Attention is all you need[END_REF] and increasing computing capabilities, they are now able to generate well written and long texts. In particular, large models, such as the well known GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF] and GPT-3 [START_REF] Tom | Language models are few-shot learners[END_REF], have been used successfully for various applications: assisting writers, summarizing, augmentating data for subsequent NLP tasks, generating fake news [START_REF] Kumar | Data augmentation using pre-trained transformer models[END_REF][START_REF] Papanikolaou | DARE: data augmented relation extraction with GPT-2[END_REF][START_REF] Zellers | Defending against neural fake news[END_REF]]. Yet, beside the prompt used to initiate the generation process, there are few options to have control on the generation process. Being able to add some constraints on the generated texts is useful for various situations. For example, it allows to create texts that follow a certain writing style, convey a certain emotion or polarity. More critically, it can be used to prevent the inherent toxicity of language models trained on the internet, or to not reproduce gender or race stereotypes. So far, most methods necessitate to fine-tune the LM, so that it specifically learns to model this constraint, i.e. the constraint is -hopefully-incorporated in the LM. This fine-tuning approach has several drawbacks. It implies to train multiple specific LMs (one per constraint), which is costly, when even possible given the size of current state-of-the-art LM, and results in several models.

In this paper, we propose new approaches to add such additional constraints on the texts but at generation time. We exploit a discriminator that is trained to determine if a text follows a given constraint or not; its output provides information to guide the generation toward texts that satisfy this expected constraint. In order to make the most of the discriminator information, we propose an original method based on the Monte Carlo Tree Search (MCTS) algorithm [START_REF] Coulom | Efficient selectivity and backup operators in monte-carlo tree search[END_REF], namely Plug and Play Language -Monte Carlo Tree Search (PPL-MCTS). We also propose simpler methods based on re-ranking to fulfil this goal. Both approaches do not require to fine-tune the LM; adding a new constraint can thus simply be done by providing a discriminator verifying if a text comply with what is expected. More precisely, our main contributions are the following ones:

1. we propose to use MCTS to implement constrained generation and we show, on three datasets and two languages, that it yields state-of-the-art results while offering more flexibility;

2. we also explore simpler generation methods based on re-ranking and show that this kind of approach, with low computational costs, can also be competitive if the diversity within propositions to re-rank is encouraged;

3. we provide a fully functional code implementing a batched textual MCTS working with the popular HuggingFace library

Related work

The goal of constrained textual generation is to find the sequence x 1:T which maximises p(x 1:T | c), given a constraint c. Few methods address the constrained textual generation.

Class-conditional language models. Class-conditional language models (CC-LMs), as the Conditional Transformer Language (CTRL) model [START_REF] Shirish Keskar | CTRL: A conditional transformer language model for controllable generation[END_REF], train or fine-tune the weights θ of a single neural model directly for controllable generation, by appending a control code cc in the beginning of a training sequence. cc indicates the constraint to verify, and is generally related to a class c and a corresponding dataset. Trained with different control codes, the model learns

p θ (x 1:T | c) = T t=1 p θ (x t | x 1:t-1 , cc).
The constraint can then be applied during generation by appending the corresponding control code to the prompt. While this method gives some kind of control over the generation, the control codes need to be defined upfront and the LM still needs to be trained specifically for each set of cc. This is a problem since the current trend in text generation is the use of large pre-trained model which can hardly be fine-tuned (for instance, the last version of GPT, GPT-3, cannot be fine-tuned without access to very large hardware resources).

Discriminator-based methods

The general idea of discriminator-guided generation is to combine a disciminator D with a generative LM. The discriminator explicitly models the constraint by calculating the probability p D (c | x 1:T ) of the sequence x 1:T to satisfy the constraint c. This probability is directly related to p(x 1:T | c) through Bayes' rule : p(x

1:T | c) ∝ p D (c | x 1:T )p θ (x 1:T ).
Discriminator-based methods alleviate the training cost problem, as discriminators are easier to train than a LM. Moreover, any additional constraint can be defined a posteriori without tuning the LM, only by training another discriminator. The discriminators have been used in different ways to explore the search space. In the work of [START_REF] Holtzman | Learning to write with cooperative discriminators[END_REF][START_REF] Scialom | Discriminative adversarial search for abstractive summarization[END_REF], the space is first searched using beam search to generate a pool of proposals with a high likelihood, and then the discriminator is used to re-rank them. However, in addition that this search can miss sequences with high likelihood, it is biased towards the likelihood, while the best sequence might only have an average likelihood, but satisfies the constraint perfectly.

Hence, it might be more suitable to take the discriminator probability into account during decoding rather than after generating a whole sequence. In this case, the discriminator is used at each generation step to get the probability p D (c | x 1:t ) for each token of the vocabulary, and merge it to the likelihood p θ (x 1:t ) to choose which token to emit. In order to reduce the cost of using a discriminator, GeDi [START_REF] Krause | Gedi: Generative discriminator guided sequence generation[END_REF] proposes to use CC-LMs as generative discriminators. The method relies on the fact that during sequence generation, the CC-LM computes p θ (x t | x 1:t-1 , c) at each time step for all tokens of the vocabulary, so that most of the terms needed to compute p θ (c | x 1:t ) are already computed in prior steps during online generation. This approach is thus at the intersection of tuning the LM and using a discriminator: it tunes a small LM (the CC-LM) to guide a bigger one.

In Plug And Play Language Model (PPLM) [START_REF] Dathathri | Plug and play language models: A simple approach to controlled text generation[END_REF], the discriminator is used to shift the hidden states of the pre-trained transformer-based LM towards the desired class at every generation step. PPLM can be used on any LM and with any discriminator. However, PPLM needs to access the LM to modify its hidden states, while the approach we propose only requires the output logits. As some LM can

PPL-MCTS method

The approach that we propose is in line with methods using a discriminator to guide a large LM model, without the need to re-train it. Unlike previous approaches, it is able to have a long term vision on what is generated. Being able to make a short-term decision (choice of the next token x t at time step t) that is promising in the long run is based on the exploration of the search space. We propose here to use the Monte Carlo Tree Search (MCTS) for an efficient exploration of this space.

MCTS is very well suited for this problem for three reasons. First, it allows to get a local score (i.e, a score for the next token to emit) using finished sequences. Hence, this score is more meaningful than scores based only on the next step. Second, it allows to explicitly define the compromise between exploitation of promising sequences (with an high likelihood), and exploration of other potentially promising sequences (to not miss better sequences with a lower likelihood). The fact that regret, i.e the number of simulations done on a sub-optimal sequence, has a theoretical upper bound in MCTS [START_REF] Christopher | Multi-armed bandits with episode context[END_REF] is a nice guarantee that the computation time is not wasted and the search is efficient. Finally, it outputs a solution at each iteration and so can fit our computational budget by allowing to adjust the quality of the solution to calculation spent.

Text generation as tree exploration process. The search space of the text generation corresponds to a tree: its root is the prompt and the child of a node is its father's sequence with one of the |V| possible token appended. In the case of constrained generation, the goal is thus to find the path, and therefore the sequence x, with the highest p(x | c) possible without exploring the whole tree in width and depth. As mentioned previously, this probability can be computed as the product of the likelihood p θ (x) and the probability given by a discriminator p D (c | x). An illustration of such a tree can be found in Fig. 1, where the likelihood of x is forged by multiplying corresponding conditional probabilities along the path, and the classification probability is calculated at the terminal node.

Applying MCTS to text generation. MCTS is a heuristic based iterative algorithm that uses randomness to solve deterministic problems that cannot be solved using traditional approaches, often because the search space is too large to be entirely explored. Each iteration consists in four consecutive steps. In the particular context of applying MCTS to text generation, we made some adaptations:

1. Selection Recursively choose children from the root to a node that has not been expanded yet. To only explore viable sequences, the probability p θ (x i | x 1:t-1 ) of a given token x i given by the LM is used during the selection phase. To this end, the children chosen are those maximizing the Polynomial Upper Confidence Trees (PUCT) [START_REF] Christopher | Multi-armed bandits with episode context[END_REF] as defined in [START_REF] Silver | Mastering the game of go without human knowledge[END_REF]:

P U CT (i) = s i n i + c puct p θ (x i | x 1:t-1 ) √ N i 1 + n i (1)
with s i the aggregate score of the node i, n i the number of simulations played after this node, N i the number of simulations played after its parent, and c puct a constant defining the compromise between exploration and exploitation. In the task of constrained generation, we define the score of a sequence as its probability knowing the class p(x | c). 2. Expansion If the selected node is not terminal, use the LM to expand it by creating its children. [START_REF] Caccia | Language gans falling short[END_REF]. Simulation (roll-out) Sample one of these children according to p θ (x i | x 1:t-1 ), and go to a terminal node through a random walk or another pattern. 4. Backpropagation Aggregate the final score obtained at the terminal node to each parent until root. There are different strategies to aggregate scores, for example, compute the average between the actual score and the one being backpropagated, or take the maximum of the two. We take the aggregated score s i associated to the node i as the averaged probability over all simulations played after this node.

When the number of iterations has reached the allocated budget, the building of the tree stops. The token x i selected for the current decoding step can be selected as the most played node amongst the root's children nodes, or the one with the highest aggregated score. We chose to select the most played one.

These adaptations of MCTS to constrained generation define our proposed approach: Plug and Play Language -Monte Carlo Tree Search (PPL-MCTS), summarized in Fig. 2. MCTS has been used in recent work [START_REF] Leblond | Machine translation decoding beyond beam search[END_REF] for machine translation, where the authors try to optimize the metric used for evaluation in machine translation. Our work mainly differ in the target goal and the way the score is defined. Rather than directly optimizing a target metric, we use MCTS to find the sequence with the highest p(x | c) possible using a discriminator D to score how well the desired constraint is satisfied.

Model improvements. In order to allow a finer control on how the constraint is applied, we introduce a parameter α ∈ [0, 1] to control the compromise between likelihood and constraint strength, modifying Bayes' equation: 1-α . Note that PUCT (1) already considers the likelihood of the sequence, favoring the selection of nodes with high likelihoods. Setting α < 1 thus forces the algorithm to explore solutions even closer to the language model. In our experiments, we set α = 1 to strengthen the classification constraint.

p(x | c) ∝ p D (c | x) α p θ (x)
To avoid expensive roll-outs, one may also assign a value to unfinished sequences at the cost of a less precise evaluation that may be not as meaningful as when doing roll-outs. Indeed, the discriminator can be trained on sequences with variable numbers of tokens, allowing it to be used at each node without the need of simulations. In this setup, the MCTS is used as an efficient compromise between exploration and exploitation, losing part of its long view property but allowing to skew the exploration toward promising solutions.

Finally, during our first experiments, we observed that PPL-MCTS leads to repetitive patterns. This is very similar of what happens with greedy search, where a single sequence with an high likelihood is dominating the search. If such sequences also have a pretty high discriminator scores, they will be repeated often. CTRL [START_REF] Shirish Keskar | CTRL: A conditional transformer language model for controllable generation[END_REF] offers a simple yet very powerful method to avoid noisy repetitions. It applies a scalar factor I(i) to the temperature parameter τ of a given token x i that penalizes this token if it is already in the input sequence. Hence, the probability of a given token becomes:

p θ (x i | x 1:t-1 ) = exp (z i /(τ • I(i))) v exp (z v /(τ • I(v))) (2)
with the repetition penalty I(i) > 1 if x i is already in the prompt and 1 otherwise, and z 1:|V| are the neural LM predicted logits over the vocabulary V. Thus, probabilities of already emitted tokens are penalized but if the language model gives a really high score to one token (hence, it is very confident that this should be the token to emit), it may still be selected as the output token.

Datasets

Three different datasets are used in the experiments presented hereafter: amazon_polarity [START_REF] Zhang | Character-level convolutional networks for text classification[END_REF], CLS (from the FLUE [START_REF] Le | Flaubert: Unsupervised language model pre-training for french[END_REF] dataset) and emotion [START_REF] Saravia | CARER: contextualized affect representations for emotion recognition[END_REF]. The first two are Amazon reviews which have been labeled as positive or negative, so the intended task is to study the possibility of applying polarity to the generation. As CLS is in French, these two datasets will serve to ensure that the methods have the same behaviour for different languages. Emotion is a collection of tweets classified under eight basic emotions: anger, anticipation, disgust, fear, joy, sadness, surprise and trust. This dataset is supposed to be more challenging since there are more classes and texts are smaller (only composed of one sentence), hence the model needs to precisely generate the target emotion with few tokens. It is worth noting that the 3 datasets have different sizes: 4,000,000 instances in total for amazon_polarity, 20,000 for emotion and 6,000 for CLS. They are available at https://huggingface.co/datasets/.

We adapted prompts used to start the generation for each datasets depending on the data format. Amazon_polarity comes with a "title" column which corresponds to the title the user gave to the review. This field is directly used as prompt. For the two other datasets, the prompts are the very first tokens of the text field. Because texts from emotion and CLS have different lengths, the size of prompts are also different: it is arbitrarily set to 6 tokens for CLS and 4 for emotion.

Methods and baselines

Baselines. Beside PPL-MCTS, we propose several baselines and simple techniques. Most studies create proposals using beam search and then re-rank them using the product of likelihood and discriminator probability. As suggested in [START_REF] Leblond | Machine translation decoding beyond beam search[END_REF], re-ranking is competitive but needs more exploration, notably on the diversity aspect. Thus, we explore re-ranking with different variations, in the way sequences to re-rank are produced, and in the way the final sequence is chosen. Three methods are tested to generate propositions: beam search [START_REF]Speech understanding systems: summary of results of the five-year research effort at carnegie-mellon university[END_REF] (with a beam size of 3), nucleus (top-p) sampling [START_REF] Holtzman | The curious case of neural text degeneration[END_REF] (with p=0.9), as well as beam sampling (as described in [START_REF] Caccia | Language gans falling short[END_REF]). For the final choice, we also propose three different methods : argmax, where the sequence that has the highest p(x|c) is chosen; first true, where propositions are sorted by descending likelihood and the first sequence that belongs to the correct class according to the guiding discriminator is chosen; and sampling, where the distribution of p(x|c) for the propositions is normalized and the chosen sequence is sampled following this distribution. Similarly to PPL-MCTS, the likelihood part of p(x|c) is omitted (i.e, α = 1) since sequences in the pool of propositions already have a relatively high likelihood.

It should be noted that in our setting, a generated sequence corresponds to a document (e.g. a whole review). This choice makes sense for our datasets, but re-ranking at a smaller level (after each sentence, after x tokens...) would also be possible and might produce different results.

Methods from the literature We compare our results with methods from the literature. In particular, we test CC-LMs trained on the target task, similarly as CTRL. We tested this method using greedy search as well as sampling for decoding. We also propose an implementation of CC-LM trained with the classification loss initially proposed for the GeDi method [START_REF] Krause | Gedi: Generative discriminator guided sequence generation[END_REF]. These CC-LMs are further used to implement the state-of-the-art GeDi model. In the experiments reported below, we report results for GeDi models trained with and without the classification loss. Finally, we report results of PPLM. For a fair comparison, the same discriminator and LM are used for our PPL-MCTS approach, the re-ranking approaches (baselines), and PPLM.

Experimental setting

For each method, a number of tokens equals to the average length of sequences of the dataset are generated: 98 tokens for amazon_polarity, 23 for emotion and 137 for CLS. Fixing the number of generated tokens allows fair comparisons between the tested methods since the perplexity of a sequence is directly linked to its length, and its number of n-gram impacts the Self-BLEU metric. An example of generation from amazon_polarity is given in Fig. 3.

To run all of these methods, three different models are needed: one discriminator, a "vanilla" LM used as generator, and the CC-LMs used in the CTRL and GeDi approaches. For the discriminator used to guide the generation, we rely on BERT-base-cased [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF] for the English datasets and FlauBERT-largecased [START_REF] Le | Flaubert: Unsupervised language model pre-training for french[END_REF] for CLS. As vanilla LM, we use GPT-2 small models, relying on OpenAI's pre-trained model for the English datasets and on belgpt2 for the French one. The implementation and models <|startoftext|> The Revenge of making a good Halloween film.

[SEP]?????? I think this movie is a waste of time. It's not scary, it's just plain stupid. The only good thing about this film is the soundtrack.<|endoftext|> <|startoftext|> The Revenge of making a good Halloween film.

[SEP] ive seen this movie a few times and i love it. the acting is great, the story line is good, and the special effects are awesome. if you like horror movies then go see this one.<|endoftext|> We tested three values for the temperature parameter for each proposed method (1.0, 1.1 and 1.2) and we only report the results for the one yielding the best accuracy score. For PPL-MCTS, we also studied the impact of c puct by testing values 1.0, 3.0, 5.0 and 8.0 along with the different temperature values mentioned. The repetition penalty has been set to 1.2 as defined in CTRL. The number of MCTS iteration per token is set to 50, as well as the number of propositions for re-ranking, except for beam sampling where it is set to 10 because of memory limitations. Given the cost of roll-out for long sequences, we apply roll-out only on the emotion dataset to be able to run extensive experiments. Without roll-out, MCTS loses a part of its long view property but still allows to skew the exploration toward promising solutions. A study of the impact of the roll-out is detailed in a next sub-section. Parameters used for literature models are those provided by the authors. Experiments were conducted on a Quadro RTX 6000 with 80 Go of RAM.

Results

Results on the emotion, CLS and amazon_polarity datasets are reported in Table 1. The statistical significance against GeDi and PPLM is measured applying a t-test with significance level (p-value) of 1%. Results show that PPL-MCTS is competitive against task-specifically trained LMs on the constraint application aspect (high accuracy), while keeping a fair amount of diversity (low Self-BLEU) and staying close to the original distribution (low oracle perplexity). On all three datasets and metrics, it constantly yields top results; the only other method which is high-performing for all metrics and constant across the datasets is GeDi trained with the classification loss.

Another remarkable result is for the Sampling -Argmax method that selects among a pool of propositions generated using sampling, the one with the highest probability to be from the correct class. Thanks to the sampling used for generating propositions, its Self-BLEU is among the lowest of all reported values. Despite the simplicity and low computational cost of this approach, its accuracy is among the best on every dataset. These very good results should however be put into perspective of the very high perplexity of its generated texts. This indicates that the generated samples may be very different than those generated by a standard LM on this dataset. Hence, exploring accuracy/perplexity trade-offs achievable with different values of α is interesting, which is proposed in Appendix A.4.

Human evaluation

Since automatic metrics can be biased and not faithfully represent the human judgement, we conduct a human evaluation to compare with the results obtained through oracles. Because of the annotation cost, we limited tested methods to the two state-of-the-art methods (PPLM and GeDi), PPL-MCTS and the promising Sampling -Argmax. This allows to test if PPL-MCTS is indeed as efficient as GeDi and if both are better than original PPLM. Also, this should confirm that the high perplexity of the Sampling -Argmax method is due to generated texts being very different from the ones generated by other methods. Three annotators (the authors) labeled the same pool of reviews in order to measure the inter-rater agreement. Since annotators are native french speakers, the evaluation has been made on the CLS dataset.

The pool consists of 50 reviews (25 positives and 25 negatives) randomly sampled for each method, which results in 200 reviews in total. Annotators was asked to go through this (randomly shuffled) pool and give two score for each review:

1. Polarity. Rate from 1 to 5 how well the text correspond to the desired label (positive or negative). If the desired label is negative, 5 corresponds to a text which contains only criticisms, 4 to one with some positive aspect and 3 a neutral review (either as many positives as negatives, or not any polarity). A rate of 1 or 2 corresponds to a text containing mainly positive comments. This score corresponds to the accuracy automatic metric. 2. Readability. Rate from 1 to 5 how well the text is written. 5 corresponds to a text without any mistake and which is perfectly understandable. A score of 4 is given if there is some misspelling or if a passage does not make much sense. The more mistakes or problem of meaning, the lower the score. This score corresponds to the perplexity automatic metric.

The diversity within the pool of generated texts being complicated to evaluate using this protocol and the Self-BLEU being fairly accurate as a diversity metric, this property is not studied in the human evaluation.

We report scores averaged over the 3 annotators as well as the standard deviation in Fig. 2. A t-test with significance level (p-value) of 1% against PPLM (GeDi being best on both scores) is applied to test statistical significance. The results seem to be in line with conclusions from automatic metrics. GeDi, when trained with the classification loss, yields similar results as PPL-MCTS, in terms of constraint satisfaction and quality of writing. PPLM, on the other hand, generates samples of lower quality and has more difficulty for applying the constraint. Finally, given its readability score, Sampling -Argmax seems to generate samples with a low quality. Its polarity score, while being higher than PPLM, is lower than expected: given the accuracy reported by the oracle, it should be close to GeDi and PPL-MCTS. It is most likely due to the fact that evaluating the polarity of a badly written text is hard to an human, often resulting in review being scored as neutral.

Effect of the roll-out

Rolling out is costly for very long sequences, and the question of its usefulness necessarily arises.

We study how rolling out for only a fixed number of tokens (instead of until the end of the sequence) influences the performance of PPL-MCTS. For this experiment, we use the CLS dataset and set the roll-out to 0 (original result), 3, 5, 10 and 20 tokens. As one can note in Fig. 4, only 5 tokens allows PPL-MCTS to be on par with GeDi on this dataset. The roll-out size quickly improves accuracy, which then reaches a plateau. It suggests that having an horizon is really helpful but only up to a given point. Conversely, Self-BLEU and oracle perplexity values stay stable, varying respectively from 0.54 to 0.57, and from 4.98 to 5.18 as the roll-out size increases from 0 to 20 (not shown on Fig. 4). Finally, it should be noted that for a -relatively small-fixed number of tokens, the cost of the roll-out is marginal compared to the global cost of PPL-MCTS.

Conclusion

In this paper, we show that it is possible to control generation with the help of a discriminator that implements some expected constraint on the text. This flexible approach is very useful when using very large language models, such as GPT-3, whose fine-tuning computational costs are prohibitive.

In contrast, training a discriminator is easier and cheaper. The methods that we propose to mix the discriminator constraint and the generation yield performance that is equivalent to the best approaches based on LM tuning. On the other hand, such approaches are more expensive during inference, because of the additional cost of the discriminator and a more complex decoding process. PPL-MCTS offers a solution for cases where training is too costly for the downstream application or the language model is not directly accessible. GeDi tackles this extra inference cost by using CC-LM as discriminator. Seeing text generation as a tree exploration process, it lowers the cost of width exploration but the depth exploration is still an issue, since it is now very similar to a standard maximum likelihood search. Monte Carlo Tree Search provides an efficient way to determine the best local choice in the long run, lowering the cost of depth exploration. Thus, these two methods solve different facets of constrained generation, and the combination of the two is a promising perspective. Moreover, MCTS allows to precisely define the best compromise between cost and quality, while ensuring the efficiency of the search theoretically. For reproducibility purposes, our implementation is made available at https://github.com/NohTow/PPL-MCTS.

Several research avenues are opened by this work. For methods yielding high perplexity, exploring the α parameter could help to reach a better compromise between accuracy and perplexity. Similarly, the size (number of tokens considered) of the roll-out in MCTS offers some ways to control the cost/performance compromise.

Having an adaptive roll-out size, as in [START_REF] Cotarelo | Improving monte carlo tree search with artificial neural networks without heuristics[END_REF], would seem particularly suited for texts. Last, it should be noted that fine-tuning a model and controlling the generation with a discriminator can be used jointly. For instance, one can use MCTS on a tuned LM, which will most likely result in even better results because sequences considered during the search will have an overall higher quality for the considered task.

Ethics/Broader impact

The ethical risks of large LMs are well known [START_REF] Bender | On the dangers of stochastic parrots: Can language models be too big?[END_REF]. Especially when they are trained on large quantities of non curated data, it has be shown that they tend to reproduce or amplifies biases on gender, race, etc. and more generally may produce inappropriate content [START_REF] Samuel Gehman | Real-ToxicityPrompts: Evaluating neural toxic degeneration in language models[END_REF]. Constrained generation as we propose is one way to control, a posteriori of the LM training, that the generated texts respect some criteria. The ethical interests are thus important, such as adding constraint about race diversity, gender equality, non toxicity, etc. But of course the same technique could be used for malicious purposes, such as constraining generation so it produces offensive texts, targeted fake news, etc.

A Appendix

We provide in this technical appendix additional information on the experiments. Further experimental results, as well as examples, are given and discussed.

A.1 Data splits

We adapted the way we split the dataset into two parts and train/test/validation sets depending on the original datasets splits. Amazon_polarity is composed of a training set of 3 600 000 examples and a test set of 400 000. We split both in two and kept 20% of each training set for validation. Emotion already comes with train, test and validation set, hence we just split each in two. Finally, CLS is composed of a train and test sets that have a size of 6000. We split the training set in two and split the test set twice so we got two validations and test sets.

The first train and validation sets are used to train and control the training of models used for the generation: the guiding classifier, "vanilla" LM and CC-LM. The test set serves to verify their performance.

The second ones are used to train the LM oracle and the classifier used to measure the accuracy. The test set allows to verify that these models are trustable for accurate evaluation. Once all the models are trained, the controlled generation is evaluated on 900 samples generated from prompts never seen by the discriminator.

A.2 Complementary results

We tested three values for the temperature parameter for each proposed method: 1.0, 1.1 and 1.2.

As the temperature grows, the output distribution of the language model becomes more and more uniform. This means that high temperatures should result in high perplexities because the sampling deviates further from the original distribution.

For PPL-MCTS, we also studied the impact of c puct by testing values 1.0, 3.0, 5.0 and 8.0 along with the different temperature values mentioned. c puct defines the compromise between exploiting nodes that already have great scores and exploring less played but promising ones. A high c puct encourages exploration. We remind that the repetition penalty I in (2) has been set to 1.2 as defined in CTRL.

In Section 'Results', we reported only one set of parameter values for each method and dataset, the one that yields the best accuracy result. We report hereafter results with every tested set of parameters in Tables 3, 4 and 5 for respectively the emotion, CLS and amazon_polarity datasets.

Unsurprisingly, the perplexity of methods which sample on the LM logits explodes when τ grows, without a noticeable gain in accuracy. Since the diversity is already high for low τ values, it seems to be better to keep the temperature low with these approaches. Note that the couple c puct = 3, τ = 1.0 for PPL-MCTS always leads to the best result for this method. Using c puct = 8 seems to yield slightly worse results, especially with a low temperature. However, the different parameters do not greatly affect the results of PPL-MCTS.

A.3 Examples of generation

We provide an example of generation for amazon_polarity and emotion datasets using PPL-MCTS, PPLM, GeDi and Sampling -Argmax methods, respectively in Figures 5 and6. Texts generated using Sampling -Argmax are rather different as suggested by reported high perplexity results. Note that emotion texts are only one sentence while those of amazon_polarity are complete reviews. This difference motivated the choice of these datasets. Also, we preferred amazon_polarity over IMDb used in GeDi and PPLM papers because of its bigger size, suitable format and because a French equivalent is available (CLS).

A.4 Constraint strength through α

As described in the model improvments section, a parameter α can be defined to control the relative importance of the discriminator score and the language model likelihood. Thus, this parameter allows to control the constraint application strength, allowing to define a trade-off between staying close the Sampling -Argmax <|startoftext|> realistic account [SEP] ive seen many of these. they mostly in the 20's, 30's and 40's and the luscious ones i know -men like this -there's an old saying that farts, u makin' u sell it..this movie has a lot of realism to it too! and i was totally impressed on how good the kids and the predator was! will it be hard for them to make more like this? i think it will! i read that war is going to be much<|endoftext|> PPL-MCTS <|startoftext|> i feel that working with a group of people who are so passionate about the same thing is really important<|endoftext|> PPLM <|startoftext|> i feel that working hard and caring for someone i don t care for is a lot less selfish than i would be feeling for someone i<|endoftext|> GeDi <|startoftext|> i feel that working with the ladies of the family is a wonderful thing and i am very fond of the way they look and feel<|endoftext|>

Sampling -Argmax <|startoftext|> i feel that working at imgur for so many years is ill be devoted to it<|endoftext|> original LM and satisfying the constraint. Note that in all of our experiments, this parameter has been set to 1, focusing on the constraint application since the proposed methods already inherently provide legible texts.

As a proof of concept, we tested automatic metrics on a range of values for α, using the Sampling -Argmax method on the amazon_polarity dataset. We chose this method and dataset since it yields top result in term of accuracy, but also exhibits very high perplexity. In this case, it seems interesting to trade a bit of accuracy for better written texts.

Results are roughly constant when α is lower than 0.98, so it has an impact only for values between 0.98 and 1. This is due to the fact that, for a long enough sequence, p θ (x) is often relatively small compared to p D (c | x). This difference of scale annihilates the influence of α. This interval thus corresponds to values of α that rescale p D (c | x) α and p θ (x) 1-α on a same order of magnitude. As shown in Figure 7, within this regime, we can observe a linear dependency between α and the accuracy as well as the perplexity. This illustrate that a trade-off can be obtained by tuning this parameter, allowing to define the strength of the constraint application which also define how far the generation can be from the original LM distribution.

Figure 1 :

 1 Figure 1: Illustration of the constrained generation process as a tree exploration from the prompt The cat. Classification probabilities are only represented on completed sequences.

Figure 2 :

 2 Figure 2: Illustration of MCTS applied to text generation.

Figure 3 :

 3 Figure 3: Example of two constrained generations using PPL-MCTS, one on the negative class, one on the positive class, using the same prompt (in bold) from amazon_polarity.Figure 4: Accuracy (%) according to the roll-out size; CLS dataset

Figure 4 :

 4 Figure 3: Example of two constrained generations using PPL-MCTS, one on the negative class, one on the positive class, using the same prompt (in bold) from amazon_polarity.Figure 4: Accuracy (%) according to the roll-out size; CLS dataset

Figure 5 :

 5 Figure 5: Example of constrained generations using PPL-MCTS, PPLM, GeDi and Sampling -Argmax methods (from top to bottom) on the positive class, using the same prompt (in bold) from amazon_polarity.

Figure 6 :

 6 Figure 6: Example of constrained generations using PPL-MCTS, PPLM, GeDi and Sampling -Argmax methods (from top to bottom) on the 'love' class, using the same prompt (in bold) from emotion.

Figure 7 :Table 4 :Table 5 :

 745 Figure 7: Accuracy (left) and perplexity (right) of the Sampling -Argmax method according to the α parameter; amazon_polarity dataset

Table 1 :

 1 Performance (accuracy, self-BLEU and perplexity) of constrained generation methods; amazon_polarity (left), emotion (middle) and CLS (right) datasets. † (resp. * ) indicates statistically significant (p ≤ 1%) improvement against GeDi-classloss (resp. PPLM).

		amazon_polarity		emotion			CLS
	Generation	Acc.	5 -Self-Oracle Acc.	5 -Self Oracle Acc.	5 -Self Oracle
	method	↑	BLEU ↓ pplty ↓	↑	BLEU ↓ pplty ↓	↑	BLEU ↓ pplty ↓
	CC-LM -Classloss 0.82	0.79	2.56 * , † 0.89 *	0.65 †	3.72 * , † 0.89 *	0.04 * , †	50.6
	CC-LM	0.91	0.71	3.21 †	0.52	0.13 * , †	11.1	0.66	0.06 * , †	31.5
	GeDi -Classloss	0.96 *	0.6 *	5.16	0.88 *	0.68	5.57 *	0.94 *	0.4	7.99 *
	GeDi	0.96 *	0.6 *	5.16	0.54	0.52 †	4.09 * , † 0.83 *	0.31 †	11.9
	PPLM	0.89	0.66	2.84 †	0.67	0.19 †	7.31	0.79	0.23 †	8.3
	Beam
	w/ Argmax	0.88	0.85	3.14 †	0.72 *	0.49 †	3.7 * , †	0.64	0.82	3.31 * , †
	w/ Sampling	0.86	0.84	3.27 †	0.7	0.46 †	3.69 * , †	0.6	0.82	3.37 * , †
	w/ First true	0.85	0.83	3.27 †	0.65	0.38 †	3.68 * , † 0.62	0.82	3.26 * , †
	Beam sampling
	w/ Argmax	0.97 *	0.73	3.82 †	0.67	0.48 †	3.88 * , † 0.88 *	0.67	3.91 * , †
	w/ Sampling	0.92	0.76	3.68 †	0.66	0.48 †	3.88 * , † 0.76	0.63	4.07 * , †
	w/ First true	0.9	0.73	3.84 †	0.66	0.49 †	3.85 * , † 0.85 *	0.71	3.8 * , †
	Sampling
	w/ Argmax	0.99 * , †	0.17 * , †	16.5	0.87 *	0.13 * , †	11.7	0.92 *	0.12 * , †	14.3
	w/ First true	0.89	0.07 * , †	85.9	0.82 *	0.13 * , †	10.4	0.87 *	0.14 * , †	13
	w/ Sampling	0.88	0.17 * , †	16.3	0.81 *	0.13 * , †	10.4	0.81	0.06 * , †	31.8
	PPL-MCTS	0.97 *	0.63 *	5.69	0.84 *	0.37 †	4.82 * , † 0.89 *	0.54	4.98 * , †

Table 2 :

 2 Results of the human evaluation on the CLS dataset (averaged over 3 annotators). * indicates statistically significant (p ≤ 1%) improvement against PPLM.

	Generation method Polarity	Readability
	GeDi -Classloss	4, 46 ± 0, 08 * 4, 19 ± 0, 28 *
	PPL-MCTS	4, 43 ± 0, 12 * 4, 05 ± 0, 23 *
	PPLM	3, 74 ± 0, 08	3, 12 ± 0, 19
	Sampling -Argmax 4, 00 ± 0, 11	2, 83 ± 0, 33

  PPL-MCTS<|startoftext|> realistic account [SEP] ive read this book and i think it is a very good account of the life of a young man. i would recommend to anyone who likes to know about the lives of people in america, or any other country that has been around for centuries.<|endoftext|> PPLM <|startoftext|> realistic account [SEP] I enjoyed this book. It was realistic and I enjoyed the way the author described the people and places. I would recommend this book to anyone who is interested in the history of the United States.<|endoftext|> GeDi <|startoftext|> realistic account [SEP] This is a real book. It's not just about the war, but also about the people who fought it. The author has a very realistic view of what happened in Vietnam and how it affected our country. I would recommend this book to anyone interested in learning more about Vietnam or any other war that took place during time period<|endoftext|>

1st CtrlGen: Controllable Generative Modeling in Language and Vision Workshop at NeurIPS 2021.

Experiments 4.1 Performance assessmentThe goal of constrained generation is to generate samples that 1) belong to a specific class while 2) keeping the language quality of the original LM, and 3) with enough diversity across samples. We chose three different metrics to evaluate each of these aspects: 1) accuracy, which is automatically verified by an external "oracle" discriminator trained on a dataset disjoint from the one used to guide the generation; 2) perplexity, which is computed using an "oracle" LM , i.e an unconstrained LM trained on different data than the one used to train the constrained generator; 3) Self-BLEU score[START_REF] Zhu | Texygen: A benchmarking platform for text generation models[END_REF], which is the BLEU score[START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF] of a sample using the other samples as references: a high Self-BLEU score means that there is a lot of overlap between generated samples, and thus that the diversity is low. Such automatic metrics have known limitations[START_REF] Caccia | Language gans falling short[END_REF] but results of human evaluation on the CLS dataset, detailed in Section 4.6, confirm that they provide a good overview of the performance.In practice, the studied dataset (see below) is split into two parts, each part being sub-divided in train/val/test sets. The first part serves to train models used for the generation (LM and discriminator), while the second is used to train oracles which serve to compute the evaluation metrics. The test set of this second part will also be used to forge prompts for the generation. Each metric is evaluated on a pool of 900 generated samples.