Thomas Scialom
email: thomas@recital.ai

Paul-Alexis Dray

Sylvain Lamprier

Benjamin Piwowarski

Jacopo Staiano

To Beam Or Not To Beam: That is a Question of Cooperation for Language GANs

Due to the discrete nature of words, language GANs require to be optimized from rewards provided by discriminator networks, via reinforcement learning methods. This is a much harder setting than for continuous tasks, which enjoy gradient flows from discriminators to generators, usually leading to dramatic learning instabilities. However, we claim that this can be solved by making discriminator and generator networks cooperate to produce output sequences during training. These cooperative outputs, inherently built to obtain higher discrimination scores, not only provide denser rewards for training, but also form a more compact artificial set for discriminator training, hence improving its accuracy and stability. In this paper, we show that our SelfGAN framework, built on this cooperative principle, outperforms Teacher Forcing and obtains state-of-the-art results on two challenging tasks, Summarization and Question Generation.

Introduction

Natural Language Generation encompasses tasks such as Machine Translation, Summarization or Data To Text generation. The real life applications are numerous, but require highly reliable and fluent models. Despite significant advances, state-of-the-art models are still known to be de-generated, with outputs containing repetitions and even nonfactual information i.e. hallucination [START_REF] Holtzman | The curious case of neural text degeneration[END_REF].

Among the culprits is a limitation of Teacher Forcing [START_REF] Williams | A learning algorithm for continually running fully recurrent neural networks[END_REF]: the loss is computed at a token level while the aim is to produce complete sequences. Moreover, while a single ground-truth reference is considered correct, several realizations of the same content may exist. Finally, the model is subject to Exposure Bias [START_REF] Ranzato | Sequence level training with recurrent neural networks[END_REF], i.e. a mismatch between training and inference distributions -in the latter, the model has no access to ground truth for the previously generated tokens. The literature has considered this mismatch responsible for the lower quality observed when generating longer sequences [START_REF] Bengio | Scheduled sampling for sequence prediction with recurrent neural networks[END_REF][START_REF] Lamb | Professor forcing: A new algorithm for training recurrent networks[END_REF].

To overcome such Teacher Forcing limitations, a consensus has emerged: a sequence level objective should be introduced [START_REF] Ranzato | Sequence level training with recurrent neural networks[END_REF][START_REF] Yu | Sequence generative adversarial nets with policy gradient[END_REF]. A body of work has proposed to use Reinforcement Learning (RL) with standard NLG metrics like BLEU [START_REF] Wu | Google's neural machine translation system: Bridging the gap between human and machine translation[END_REF] or ROUGE [START_REF] Paulus | A deep reinforced model for abstractive summarization[END_REF]. However, NLG metrics are known to not reflect well human judgement [START_REF] Novikova | Why we need new evaluation metrics for NLG[END_REF], which explains why the resulting models tend to be qualitatively worse than their MLE baselines [START_REF] Caccia | Language gans falling short[END_REF]. To move toward less biased metrics, a natural alternative is to evaluate the output with a learned discriminator. An ideal discriminator would not be biased w.r.t. to its training set, and could therefore be considered as a perfect metric that matches human consensus. Note that discriminators are already reported to be highly accurate to distinguish human written texts from machine generated ones [START_REF] Scialom | Discriminative adversarial search for abstractive summarization[END_REF][START_REF] Zellers | Defending against neural fake news[END_REF].

In light of this observation, two concurrent approaches have been explored: i) at training time, using Generative Adversarial Networks [START_REF] Yu | Sequence generative adversarial nets with policy gradient. 489[END_REF]; and ii) at inference time, via cooperative decoding [START_REF] Gabriel | Cooperative generator-discriminator networks for abstractive summarization with narrative flow[END_REF]: a discriminator guides the search algorithm, such that the generator and the discriminator cooperate to select the generated tokens. These approaches pursue the same objective: producing texts more similar to what a human writes.

Both methodologies suffer from specific limitations. Cooperative decoding algorithms rely on a discriminator that re-ranks a limited set of candidates selected by the generator. 1 Hence, cooperative decoding algorithms are limited by the generator ability to rank relevant tokens in a good enough position. On the other hand, language GANs are learned via reinforcement learning due to the discrete nature of text. This makes them particularly unstable to train, and usually fall short compared to Teacher Forcing [START_REF] Caccia | Language gans falling short[END_REF]. In standard Language GANs, the discriminator provides a reward for the entire sequence, which can be difficult to exploit by the generator due to its sparsity [START_REF] De Masson D'autume | Training language gans from scratch[END_REF].

In this paper, we propose SelfGAN, a framework to learn language GANs in a Self -training process where the signal from the discriminator is passed to the generator in a completely new way. We consider cooperative algorithms as a way to infuse the discriminator signal. We start from a simple observation: outputs obtained via cooperative decoding are more human-like, compared to their generator-only counterparts. Inspired by recent knowledge distillation approaches, we propose to consider cooperative outputs as targets in a Teacher Forcing training process: cooperative decoding stands as a teacher we attempt to imitate through the generator network. Just like a standard GAN, both the generator and the discriminator are trained at each step. While the generator improves, it becomes adversarial to the discriminator, which benefits from the cooperative generation. The discriminator, now trained on improved sequences, also contributes to improve the cooperative generation, and so forth. Note that in SelfGANs the discriminator is only used to drive the cooperative generation and never to provide a reward signal like in standard Language GANs.

SelfGAN can be implemented with any cooperative decoding algorithm. Current cooperative approaches [START_REF] Deng | Residual energy-based models for text generation[END_REF][START_REF] Scialom | Discriminative adversarial search for abstractive summarization[END_REF] rely on "myopic" algorithms like Beam Search or Sampling that generate the tokens left-to-right. The model has to always predict the next word, and can never look back and revise past choices. In some cases, despite all the candidates being judged to likely not be human by the discriminator, the model is locked in a dead-end. This behavior is quite unnatural for humans -who often proofread their texts. We refer to this phenomenon as the left-to-right curse.

To address this left-to-right curse, we introduce Coop-MCTS, a new decoding algorithm based on Monte Carlo Tree Search (MCTS) [START_REF] Coulom | Efficient selectivity and backup operators in monte-carlo tree search[END_REF][START_REF] Kocsis | Bandit based monte-carlo planning[END_REF]. We compare Coop-MCTS to state-of-the-art cooperative decoding algorithms in two scenarios: i) inference time, as the decoding algorithm; and ii) during training, as the cooperative algorithm in SelfGAN. In both scenarios, we show that the respective resulting outputs are more likely to look like human texts and improve all the automatic metrics.

All in all, our contributions can be summarized as follows:

Related Work

Beyond Teacher Forcing To mitigate the limitations inherent to Teacher Forcing, various alternatives have been proposed. In Scheduled Sampling, Bengio et al. [START_REF] Bengio | Scheduled sampling for sequence prediction with recurrent neural networks[END_REF] proposed to condition the generation not only on the ground truth tokens, but also the generated ones. Given that only one reference is available, this introduces a new mismatch when computing the loss, this time between the generated tokens used to condition the model, and the target tokens. To take into account multiple possible references, using a sequence level metric is a potential alternative. In Mixer, Ranzato et al. [START_REF] Ranzato | Sequence level training with recurrent neural networks[END_REF] chose BLEU, the standard metric to evaluate Machine Translation. Since it is not differentiable, the task is framed in a Reinforcement Learning setup where the reward corresponds to the BLEU score given a sampled sequence of tokens. Paulus et al. [START_REF] Paulus | A deep reinforced model for abstractive summarization[END_REF] applied the same method to Summarization, using this time ROUGE. While the results improved in terms of ROUGE, the human evaluation found that the generated summaries were rated worse in term of fluency than the MLE baseline. The model learns to take advantage of metric biases, while being less correct according to human judgement.

Language GANs In theory, a perfect discriminator would be able to judge if an output corresponds to the data distribution or not. Discriminators could therefore be an interesting alternative reward compared to other metrics. In practice, we need to train the discriminator jointly with the generator, framing the task as a GAN. Language GANs are known to underperform MLE [START_REF] Caccia | Language gans falling short[END_REF], due to the unavoidable sparsity of a discriminator reward. A large body of works have proposed denser rewards: ranking or comparative discriminators [START_REF] Che | Maximum-likelihood augmented discrete generative adversarial networks[END_REF][START_REF] Li | Adversarial learning for neural dialogue generation[END_REF][START_REF] Zhou | Self-adversarial learning with comparative discrimination for text generation[END_REF], a sequential discriminator where the rewards are provided at each time step of the generation [START_REF] Semeniuta | On accurate evaluation of gans for language generation[END_REF][START_REF] De Masson D'autume | Training language gans from scratch[END_REF]. More recently, Scialom et al. [START_REF] Scialom | Coldgans: Taming language gans with cautious sampling strategies[END_REF] proposed to stabilize the GAN training by lowering the Softmax temperature to explore more structured outputs, and closer to the generator distribution.

In this work, our proposed framework allows to propagate the discriminator signal in a cooperative way, which can be seen as an alternative solution to the sparsity of the reward and the training stability.

Knowledge Distillation SelfGAN has a connection with knowledge distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], where a student is trained on outputs from the teacher. In particular, self distillation using only a generator has shown to improve generalisation on image GANs [START_REF] Zhang | Self-distillation as instance-specific label smoothing[END_REF] by acting as label smoothing. To the best of our knowledge, this work is the first to propose the idea of augmenting the teacher by coupling a discriminator to a generator. Beyond GANs, SelfGAN could serve for other applications using distillation, e.g. in semi-supervised methods that use a teacher model to create synthetic labels for unlabeled examples [START_REF] Scudder | Probability of error of some adaptive pattern-recognition machines[END_REF][START_REF] Yarowsky | Unsupervised word sense disambiguation rivaling supervised methods[END_REF].

Monte Carlo Tree Search in NLG Despite important successes in games [START_REF] Schrittwieser | Mastering atari, go, chess and shogi by planning with a learned model[END_REF][START_REF] Silver | Mastering the game of go without human knowledge[END_REF], very few works have attempted to apply MCTS to NLG. Kumagai et al. [START_REF] Kumagai | Human-like natural language generation using monte carlo tree search[END_REF] proposed to employ context-free grammar rules combined with a n-gram language model and explore the space of grammatically correct texts via a MCTS. In the context of commercial e-commerce agents, Mukherjee [START_REF] Mukherjee | An unsupervised approach to automatic response generation for conversational e-commerce agents using monte carlo tree search[END_REF] proposed to optimise with a MCTS a scoring function designed to reward grammatical correctness.

SelfGAN

Algorithm 1 SelfGAN 1: Input: a generator gen, a discriminator discr, and a cooperative decoding method decod coop 2: for n epochs do S coop ← decod coop (X, gen, discr)

5:
gen.train(srcs=X, tgts=S coop ) Standard maximum likelihood but with S coop as the target, and not S ref

6: discr.train(srcs=X, human_exs= S ref , machine_exs=S coop )
The difficulty of GAN-based approaches for NLP tasks lies in the fact that no gradient flow can be propagated from the discriminator to the generator. As discussed above, approaches from the literature circumvent this difficulty by employing RL approaches, using discriminator scores as rewards to train the generator. However, such approaches induce great instabilities in the learning process, due to the use of a non-stationary reward function in addition to the high variance associated to monte-carlo estimations of RL.

The idea in our SelfGAN approach is to transfer the sparse signal of the discriminator, classically used as rewards for a RL procedure, to the sampling mechanism of sequences that have to be favored through MLE. In that way, SelfGAN starts from a pretrained generator, that we fine-tune using sequences S coop provided by a cooperative decoding process decod coop for each condition in the training set X. This process, detailed in the next section, uses both the generator and a discriminator network to output human-like sequences S coop , for which we improve the generator likelihood via classical maximization:

max θ (x,s)∈(X,Scoop) π θ (s|x), where π θ (s|x) = |s| t=1 π θ (s t |s 0:t-1 , x)
stands for the generator probability of sequence s given the conditioning input x, with π θ implemented as a neural architecture with a softmax output function.

At each iteration of the training procedure, the discriminator network is optimized as a binary classifier on i) the human references and ii) the machine generated via the cooperative sequences:

1 |H| (x,s ref )∈H log(D(x, s ref )) + 1 |G| (x,scoop)∈G log(1 -D(x, s coop ))
where x is the source input, H is the set of pairs associating x with a human written text s ref from the data distribution, and G is a set of pairs with generated outputs s coop . D(x, s) stands for the probability, provided by the discriminator network, that sequence s is a human reference for condition x. In order to effectively guide the cooperative process at each step, the discriminator needs to be sequential: consistently with [START_REF] Scialom | Discriminative adversarial search for abstractive summarization[END_REF], we use a left-to-right mask during training, allowing discriminator predictions for unfinished sequences.

Please note that, by construction of the cooperative decoding process, we have with high probability at each iteration D(x, s coop ) >= D(x, s gen ) for any condition x ∈ X, with s coop a cooperative decoded sequence for x and s gen a sequence directly sampled from the generator according to π θ (s|x).

Based on this observation, and provided that the discriminator is sufficiently trained at each step, the generator is trained such that the probability of predicting human-like sequences is maximized. This process i) allows us to consider a sequence level metric, and ii) offers more stability compared to Reinforcement Learning, as we observe in our experiments (see section 6). Note also that, contrary to RL approaches which have to find a good balance between discriminator and generator capacities, our approach does not suffer from Vanishing Gradient [START_REF] Arjovsky | Towards principled methods for training generative adversarial networks[END_REF], since discrimination is only used for decoding, in a cooperative process for generator training. We depict the SelfGAN in Algorithm 1.

4 Decoding Mechanisms Sampling To obtain diverse outputs, it is common to sample tokens from the model distribution.

In particular, this is mandatory when there is no input at all, i.e. Unconditional NLG, for instance GPT [START_REF] Radford | Language models are unsupervised multitask learners[END_REF]. However, the longer the sequence, the more likely to sample a token from the tail of the distribution, causing degeneration [START_REF] Holtzman | The curious case of neural text degeneration[END_REF]. To mitigate this issue, common practices are to lower the Softmax Temperature and keeping only the Top K tokens [START_REF] Fan | Hierarchical neural story generation[END_REF] / the Top P probability mass [START_REF] Holtzman | The curious case of neural text degeneration[END_REF].

Beam Search is the standard algorithm to approximate the sequence maximising the output probability, by maintaining K candidates at each step. Its usage suits better conditional NLG tasks, where the diversity arises from the variety of conditioners inputs, e.g. in Summarization.

Cooperative Decoding: Combining a Discriminator and a Generator

Subject to exposure bias, neither Sampling or Beam Search are satisfying: the outputs produced are easily identified by a discriminator [START_REF] Scialom | Discriminative adversarial search for abstractive summarization[END_REF], indicating that they differ from human written text. In light of this, two concurrent works have recently proposed to use the discriminator during the decoding.

DAS local -Reranking Step By

Step In Discriminative Adversarial Search [START_REF] Scialom | Discriminative adversarial search for abstractive summarization[END_REF] a discriminator re-ranks the sub-sequence candidates at each decoding step of a Beam Search, in order to favor human-like outputs.

DAS global -Reranking Complete Sequences In a concurrent work [START_REF] Deng | Residual energy-based models for text generation[END_REF] a very similar cooperative method is proposed: this time, N complete sequences are sampled from the auto-regressive model.

The N sequences are scored by a discriminator, allowing to select the one with the highest probability to be human-like. Since the discriminator re-ranking is computed on a complete sequence, we refer to this method as DAS global , as opposed to DAS local .

Coop-MCTS: Cooperative Decoding beyond the Left-To-Right Curse

It can happen that all sequence candidates are judged by the discriminator to be machine-like rather than human-like. In such case, the cooperative decoding is stuck in a dead end; such limitation is unsatisfactory. Neither DAS local or DAS global have the ability to revise their previous decisions.

To cope with those limitations of myopic decoding strategies, we propose to consider an adaptation of MCTS for NLG. Just like in the context of games [START_REF] Silver | Mastering the game of go without human knowledge[END_REF], we consider a policy network π, the generator, that outputs a probability over all the possible actions (tokens) at each step of the sequence. The discriminator D corresponds to the value network. In MCTS, the trajectories are explored to build a tree following three steps:

1. Selection starting from the root, children nodes tokens ω are selected among the vocabulary V recursively w.r.t. the PUCT algorithm [START_REF] Rosin | Multi-armed bandits with episode context[END_REF][START_REF] Silver | Mastering the game of go without human knowledge[END_REF]:

ω = arg max ω∈V Q(s, ω) + c puct π τ (ω | s) b N (s, b) 1 + N (s, ω) (1)
where Q is the value of taking action ω in state s: in NLG, this corresponds to selecting a token among the vocabulary at step i given the source context and the sub-sequence ω 0 , ..., ω i-1 . c puct is a constant, τ the temperature that scales the Softmax, and N (s, ω) the number of times the token ω has been chosen in state s. We stop the loop when a node s o has not been expanded yet, i.e. the discriminator D has not calculated its value.

2. Extension Given the selected node, we calculate the distribution probability from the generator π(ω | s o ). We apply nucleus sampling [START_REF] Holtzman | The curious case of neural text degeneration[END_REF] to filter out the less likely tokens and reduce the number of actions. The remaining tokens constitute the children nodes, associated to their corresponding probability. At the same time, we calculate the value of the current state D(s o ) that allows to compute the backup step.

3. Backup we update Q for all the nodes that led to s o such that Q ← max (Q, D (s o )). Note that we choose to use the max instead of the average for the following reason: the value network, i.e. a discriminator, becomes more accurate as the candidate sequence grows (see Figure 2 in [START_REF] Scialom | Discriminative adversarial search for abstractive summarization[END_REF]), hence if a long sequence is judged human by the discriminator, any of its sub-sequences should be considered human-like as well. In contrast, a long sequence can be machine-like despite starting in a very human-like manner: the beginning sub-sequence should keep its human-like score.

These three steps are computed for a restricted number of simulations. Then, the next token corresponds to the root child with the most visit counts. The process continues step by step to generate the next token, until reaching either the special token End Of Sentence, or the maximum length.

5 Experimental Details

Datasets

To measure the effectiveness of SelfGAN, we experiment on two standard conditional NLG tasks: Question Generation (QG) and Summarization, consistently with previous works [START_REF] Dong | Unified language model pre-training for natural language understanding and generation[END_REF][START_REF] Scialom | Coldgans: Taming language gans with cautious sampling strategies[END_REF]:

• Question Generation: we used the SQuAD dataset [START_REF] Rajpurkar | Squad: 100,000+ questions for machine comprehension of text[END_REF], consisting of 100K triplets of Wikipedia paragraphs, factual questions, and their answers. • Summarization: we used the CNN/Daily Mail dataset (CNNDM) [START_REF] Nallapati | Abstractive text summarization using sequence-to-sequence rnns and beyond[END_REF], consisting of 300K news articles, paired with their corresponding summaries. The summaries are formed of multiple sentences, making the amount of tokens to generate much larger than for Question Generation.

Models Reported

MLE the first baseline we consider is a standard model trained via teacher forcing. As for all our experiments, we initialised the seq2seq with T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF], as detailed in Section 5.4.

ColdGAN we consider as a second baseline the current state-of-the art for language GANs, ColdGAN [START_REF] Scialom | Coldgans: Taming language gans with cautious sampling strategies[END_REF]. The authors proposed to lower the temperature when sampling the sequences during training, with the objective of stabilizing the training process.

SelfGAN can be based on any cooperative decoding algorithm. To train SelfGAN, we therefore experiment the three different cooperative algorithms described in Section 4 (DAS Local , DAS Global , and Coop-MCTS) and report the results for the corresponding SelfGAN: SelfGAN DAS-Local , SelfGAN DAS-Global , and SelfGAN Coop-MCTS .

Decoding Method at inference time For each model, any decoding method can be applied at inference time, independently from the training scheme. Therefore, for all the models described above, we report the results given each decoding method previously described (Section 4): Beam Search, DAS Local , DAS Global , and Coop-MCTS.

To the best of our knowledge, GANs and Cooperative decoding have never been directly compared before this work. A fortiori, this is the first time that a GAN model is tested with a Cooperative decoding method at inference. We investigate possible distillation effects in Section 6.

Metrics

To compare the different models, we report two type of metrics: n-gram based and discriminator.

N-gram based

We report the standard BLEU [START_REF] Papineni | a method for automatic evaluation of machine translation[END_REF] and ROUGE [START_REF] Lin | A package for automatic evaluation of summaries[END_REF]. Both measure an overlap of n-grams between the reference and the evaluated text. They differ in that BLEU is precision oriented while ROUGE is rather recall oriented.

Discriminators Both BLEU and ROUGE suffer from the aforementioned limitations. We therefore propose to consider discriminators for model evaluation. Intuitively, they measure how model outputs are similar to what a human would have written. We consider two different discriminators:

• Base is a discriminator trained on the MLE baseline outputs generated via beam search. It allows to measure the corresponding improvement from the MLE baseline. Note that it corresponds to the initial discriminator in all the GANs experiments, and the discriminator used in the cooperative search for the MLE baseline.

• Base+ Since the Base discriminator plays a role in all our experiments (except MLE+Beam Search), it is possible that a model that makes use of this Base obtains better Base results, despite bringing new biases and de-generation behaviors. For this reason, we also report Base+, a discriminator fine-tuned on all the different model outputs together. Base+ is never used by any model at training or inference time. It is thus more robust toward an undesirable adversarial generation mode, while still being comparable for the different experiments. We argue that a higher Base+ score indicates a real improvement beyond potential bias.

Implementation Details

For all experiments, we used the T5-small [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] architecture. 2 Using 4 Nvidia V100 SXM2 GPUs, SelfGAN Coop-MCTS training and evaluation takes respectively 26 hours and 1 hour on CNN/DM; 6 and 0.5 hours on SQuAD. Compared to 2 (0.5) hours to train via MLE on CNN/DM (SQuAD), we identify in the computational cost the main limitation of our work.

6 Results and discussion

Conditional Text Generation

In Table 1, we report the results for all the previously trained generators, with the different decoding algorithms presented in Section 4. By 'model', in the following, we refer to the couple composed by a trained generator and a decoding algorithm.

We report BLEU4, ROUGE-1, ROUGE-L along with scores for the discriminators Base and Base+, computed as the percentage of outputs considered as human by a given discriminator model. QG and from 8.6% to 15.3% on Summarization. This corresponds to almost twice more outputs that sound human according to the Discriminator metric.

Unconditional Text Generation

We follow the ColdGAN setup: we compared our proposed approaches on the EMNLP2017 News dataset. The evaluation takes into account both the quality and the diversity. Consistently with previous works (e.g. ColdGAN, ScratchGAN, LeakGAN), we use the following metrics:i) BLEU-5 for measuring the quality (higher better); ii) Self-BLEU-5 for measuring the diversity (lower better).

To obtain a finer comparison between models, Caccia et al. [START_REF] Caccia | Language gans falling short[END_REF] proposed to draw the curve of BLEU vs self-BLEU, by sampling with various temperatures at inference.

We denote as the standard method to generate text in this setup, as used in all the previous works we are comparing to. It is a simple left to right decoding where, at each step, a token is sampled among the Softmax probabilities scaled by the temperature.

In our Coop-MCTS, the probability of a token is given by its visit counts during the simulations. In Conditional generation, we select at each step the token with the maximum number of counts. In Unconditional generation, we sample from the tokens counts distribution.

Overall, the results are consistent with the experiments on Conditional Generation: the MLE generator decoded with our proposed MCTS (3rd row) obtains:

1. significantly better slightly lower results than ColdGAN decoded with Sample(2nd row)

2. results than when the same MLE decoded with Sample(1st row);

3. SelfGAN decoded with MCTS (4th row) obtains the best results.

Discussion

Human-like features during training In NLG, various rules are often integrated into the Beam Search to improve the quality of the outputs, for instance a length penalty [START_REF] See | Get to the point: Summarization with pointer-generator networks[END_REF] or an interdiction for 3-grams repetitions [START_REF] Paulus | A deep reinforced model for abstractive summarization[END_REF][START_REF] Dong | Unified language model pre-training for natural language understanding and generation[END_REF]. Such a need to hard code these rules indicates a discrepancy between the human output characteristics and what the model has learned. In particular, Scialom et al. [START_REF] Scialom | Discriminative adversarial search for abstractive summarization[END_REF] reported the difference between DAS and the human reference for: i) Length: the average number of tokens per output; ii) Novelty: percentage of tokens in the output that were not present in the source text; iii) N-gram repetition: percentage of N-grams that occur more than once in the output. To measure how SelfGAN learns these features by itself, we report in Figure 1 the evolution of these Human Evaluation We conduct a human evaluation to measure the models performances beyond automatic metrics. We limit the evaluation to three generators (MLE, ColdGAN, and SelfGAN Coop-MCTS) and two decoding methods (Beam Search and Coop-MCTS), for a total of 6 different models. Three professional English speakers rated 300 sampled summaries and followed the same protocol from Fabbri et al. [START_REF] Fabbri | Re-evaluating summarization evaluation[END_REF]. Four dimensions are evaluated on a Likert scale from 1 to 5 (the higher the better):

1. Consistency: the proportion of facts in the summary correct w.r.t. the source text; 2. Coherence: how well-structured and well-organized is the summary; 3. Fluency: how fluent the summary is to read; 4. Relevance: the ratio between important and excess information in the summary.

From Table 3 allows to obtain significant improvements on the former two dimensions while still maintaining comparable scores on the latter.

Analysis To further understand the benefits of selfGAN, we propose to analyze the evolution of the generator and discriminator networks through the learning process. In figure 2 (left), we first plot the average magnitude (L2 norm) of the discriminator gradients w.r.t. its parameters. We observe that ColdGAN induces important instabilities for its discriminator over time, with a highly fluctuating gradient magnitude. Conversely, thanks to its cooperative decoding process, SelfGAN produces sequences that form a more compact set for discriminator training, a variance of gradient magnitude twice lower than ColdGAN , for a comparable magnitude in average. This discriminator stability is a first explanation for the improvements of the proposed approach.

In a second plot, given on the right of Figure 2, we report the collinearity of generator gradients for the generated samples from the model with those for the corresponding human references. Higher values indicate sampling strategies that induce a useful gradient flow for the generator. For ablation purposes, we first report values for a "SelfGAN BeamSearch " approach, where we used a standard Beam Search to generate the training examples: note that it has no discriminator, hence it is not a GAN anymore. We can observe its divergence, as opposed to SelfGAN Coop-MCTS , which emphasizes the importance of the cooperative decoding for producing the example used to train the model. For SelfGAN Coop-MCTS and ColdGAN, the gradients become more co-linear with human references through time, indicating a convergence of the process towards the human distribution. We observe that SelfGAN Coop-MCTS produces more useful sequences for achieving this convergence.

Coop-MCTS as an alternative to the dead-end search When analysing the behavior for the Coop-MCTS decoding, we observed in different examples that it provides an effective mean to revise generations that eventually ended up to be unlikely. To illustrate this, we report in Table 6.3 the different MCTS steps for an ambiguous example: the conditioned answer, Super Bowl, occurs at different places of the the input. Therefore, the model has to decide which specific mention of Super Bowl to focus on: at step 17, it considers its current generation as a dead end and decides to start on new node (How). The final output is a question that arguably sounds better than the initial one.

Societal Impact Reliable NLG models can have significant societal impact with beneficial applications such as efficient information access via automatic summarization or personalized student evaluation through question generation. Conditioned Answer: Super Bowl Context:

Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24â€"10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the logo could prominently feature the Arabic numerals

Step 01: What ... Step 16: What was the name of the game that would have been known as "Super Bowl

Step 17: How ... Step 46: How is called the American football game that determines the NFL champion? Table 4: Progressive results obtained by our Coop-MCTS decoding method on Question Generation during a simulation. Until the 16th step, the generation is left-to-right. Then, the cooperation mechanism kicks in, allowing the model to safely abort this beam, by restarting a new question with How. We report the cross-attention weights on the input context for step 16 (red) and 17 (blue). tools detrimental to society, e.g. large scale creation of misleading (fake) news [START_REF] Radford | Language models are unsupervised multitask learners[END_REF]. As argued by Zellers et al. [START_REF] Zellers | Defending against neural fake news[END_REF], keeping this research open and under public scrutiny can be an effective defense.

Conclusion

In this paper we propose SelfGAN, a new framework to train Generative Adversarial Networks based on a cooperative decoding search. To overcome the left-to-right curse that limits standard search algorithms, we propose Coop-MCTS. We conducted extensive experiments on two challenging tasks: Summarization and Question Generation, obtaining state-of-the-art performance for SelfGAN both in terms of automatic metrics and within a human evaluation. As the stability of the discriminator looks to be crucial for language GANs, we plan for future works to still focus on increasing it through the definition of dynamic regularization mechanisms. Finally, we will explore how reference-less metrics, e.g. [START_REF] Scialom | Summarization asks for fact-based evaluation[END_REF], can be combined to help the exploration during the decoding.

3 :

 3 for X, S ref in training set do Start Training 4:

4. 1

 1 Standard Practices: Generator-only At decoding time, two different approaches are commonly used in NLG: Sampling and Beam Search. They respectively correspond to two different objectives.

Figure 1 :

 1 Figure 1: Average difference for Summarization between human references and model outputs for the Length (Left), the Novelty (Middle), and the 3-grams repetitions (Right) during training. The closer to 0 the less differences w.r.t. gold-references.

Figure 2 :

 2 Figure 2: Left: Moving Average of the magnitude of the discriminators gradients during training. Right: collinearity of the generators gradients between the sampled texts and their corresponding human reference for SelfGAN Coop-MCTS , ColdGAN and SelfGAN BeamSearch . Both on Summarization.

  SelfGAN and Coop-MCTS compare favorably to prior state-of-theart results on two challenging tasks, Summarization and Question Generation.

1. SelfGAN We propose a new training framework based on cooperative decoding, wherein the generated sequences are used as ground truth; 2. Coop-MCTS We improve cooperative decoding with a new decoding algorithm, Coop-MCTS, offering a solution to the left-to-right limitation of current search methods; 3. We show that combining

Table 2 :

 2 Results on Unconditional Text Generation for samples realized at three different temperatures, in terms of BLEU Vs Self-BLEU (higher better;lower better).

	Model						T=0.5	T=1		T=2
	MLE+Sample						0.42;0.29 0.31;0.11 0.18;0.07
	ColdGAN+Sample						0.47;0.21 0.33;0.08 0.22;0.06
	MLE+Coop-MCTS						0.45;0.22 0.34;0.10 0.21;0.06
	SelfGANCoop-MCTS+Coop-MCTS 0.48;0.20 0.37;0.09 0.24;0.05
	Delta Length	0.18 0.20 0.22 0.24 0.26 0.28 0.30	0	2000 ColdGAN + BeamSearch 4000 6000 Training Step SelfGANCoop MCTS + BeamSearch SelfGANCoop MCTS + Coop MCTS	8000	10000	Delta Novelty	0.14 0.16 0.18 0.20 0.22 0.24 0.26	0	2000 ColdGAN + BeamSearch 4000 6000 Training Step SelfGANCoop MCTS + Coop MCTS SelfGANCoop MCTS + BeamSearch	8000	10000	Delta Repetition	0.00 0.01 0.02 0.03 0.04 0.05 0.06	0	2000	4000 Training Step 6000 ColdGAN + BeamSearch 8000 SelfGANCoop MCTS + BeamSearch 10000 SelfGANCoop MCTS + Coop MCTS

Base was only trained on MLE+Beam Search outputs. As expected, by further training on the outputs generated by all the different models, Base+ has a higher accuracy, which consistently results in lower scores compared to Base.

  we observe significantly better results for SelfGAN Coop-MCTS w.r.t. both MLE and ColdGAN. While Coop-MCTS decoding appears overall beneficial in terms of Coherence and Relevance, but scores lower on Consistency and Fluency, its combination with SelfGAN Coop-MCTS

  Still, malicious actors can use the same technology to build

	Discr. Gradient Magnitude	0.0000 0.0005 0.0010 0.0015 0.0020 0.0025	0	2000	4000 Training Step 6000	8000 SelfGANCoop MCTS 10000 ColdGAN	Collinearity	0.4 0.1 0.2 0.3	0	2000 SelfGANCoop MCTS 4000 Training Step 6000 ColdGAN SelfGANBeamSearch	8000	10000

35th Conference on Neural Information Processing Systems (NeurIPS 2021).

Opposed to a generator that outputs probabilities for the entire vocabulary V at once, a discriminator outputs the likelihood for a specific sequence to be human-written or machine-generated. Calculating the discriminator probability for every possible sequence is therefore not realistic, as the computation grows exponentially at a pace of V l where V is the vocabulary size and l the sequence length.

As implemented in HuggingFace transformers[START_REF] Wolf | Huggingface's transformers: State-of-the-art natural language processing[END_REF].

Acknowledgments

This work was partially performed using HPC resources from GENCI-IDRIS (Grant 2021-AD011012318).

For Base and Base+ the scores correspond to the probability of being human, so higher is better for all the metrics. For SelfGAN MCTS , we experimented with 5 different seeds and the standard deviation is always inferior to 0.1 for BLEU4 and ROUGE, and inferior to 0.5% for Base and Base+.

We start by focusing on the MLE results to compare the different decoding mechanisms. We observe that all the cooperative searches outperform Beam Search. Regarding Base and Base+ metrics, DAS Local compares favorably to DAS Global . We hypothesize that invoking the discriminator to rank at each step can have more impact than using it only once on fully decoded sequences. Finally, our proposed Coop-MCTS obtains the best results by a large margin.

Regarding the different GANs, we first compare them given the default decoding mechanism, i.e.