Somayeh Jafaritazehjani
email: somayeh.jafaritazehjani@irisa.fr

Damien Lolive
email: damien.lolive@irisa.fr

John D Kelleher
email: john.d.kelleher@tudublin.ie

Style versus Content: A distinction without a (learnable) difference?

Textual style transfer involves modifying the style of a text while preserving its content. This assumes that it is possible to separate style from content. This paper investigates whether this separation is possible. We use sentiment transfer as our case study for style transfer analysis. Our experimental methodology frames style transfer as a multi-objective problem, balancing style shift with content preservation and fluency. Due to the lack of parallel data for style transfer we employ a variety of adversarial encoder-decoder networks in our experiments. Also, we use a probing methodology to analyse how these models encode style-related features in their latent spaces. The results of our experiments which are further confirmed by a human evaluation reveal an inherent trade-off between the multiple style transfer objectives and indicate that style cannot be usefully separated from content within these style-transfer systems.

Introduction

Style transfer can be defined as a Natural Language Generation (NLG) task where an input word sequence (a text, a sentence, an utterance. . . ) from an original style is rephrased in order to fit a target style while keeping its semantic information, leading to a style-transferred sequence. This task can be applied to many facets of the natural language: sentiment, complexity, authorship, formality, and so on. The primary use-case of textual style transfer is to improve language understanding between humans or between humans and machines by adapting messages to the social context or abilities of an interlocutor. However, work on style transfer can also provide insight into language more generally, for example by providing insight into the relationship between style and content. Indeed, a pre-requisite of style transfer is the presumed ability to separate style from content, and so research on style transfer can inform whether computational models can usefully learn this distinction.

Research on style-transfer often focuses on one specific aspect of textual productions, (e.g., preserving the meaning of the original sequence) and disregards the others (e.g., how present is the target style in the generated output, and the fluency of the generated sentences). However, this paper highlights the importance of evaluating style transfer systems in terms of multiple objectives where fitting the target style is counter-balanced by the needs to generate a linguistically fluent output sequence and at the same time preserving the original meaning. Furthermore, based on an analysis of a number of evaluation experiments of style-transfer systems, including a probing experiment that analyses the latent space of these systems, we identify an inherent trade-off in style-transfer between transfer strength, fluency and content preservation. We conclude that style and content cannot be usefully disentangled by these systems (at least not for the task of sentiment transfer), and so a holistic perspective on style and content that focuses on finding a suitable trade-off for the target application is likely the best approach to developing a useful style-transfer system.

The paper structure is as follows. Section 2 reviews related work on style-transfer and distinguishes between: (i) systems that conceive of style as being encoded in a set of explicitly identifiable features that can be removed and replaced, and (ii) systems that do not explicitly identify stylistic features. Section 3 introduces the style transfer evaluation objectives we use; Section 4 describes the models we evaluate in our experiments, and Section 5 describes the dataset and our experimental methodology. In Section 6 we evaluate the style-transfer models, and in Section 7 probe their representations of the input to examine what stylistic information they encode. In Section 8 we set out our conclusions and future work.

Literature review

To deal with textual style transfer, a first group of approaches consider a two-step process where markers of the source style are removed before generating the target style output sequence. Marking and removing the original style features can be achieved through frequency-based methods [START_REF] Li | Delete, retrieve, generate: a simple approach to sentiment and style transfer[END_REF] or neural networks that integrate attention modules [START_REF] Leeftink | Towards controlled transformation of sentiment in sentences[END_REF]. The generation step relies on techniques such as the retrieval of textual segments from a corpus of the target style [START_REF] Ramos | Using tf-idf to determine word relevance in document queries[END_REF][START_REF] Leeftink | Towards controlled transformation of sentiment in sentences[END_REF], or employing neural network generation techniques [START_REF] Li | Delete, retrieve, generate: a simple approach to sentiment and style transfer[END_REF].

The second group of approaches which have been mostly implemented in the literature focus on endto-end learning strategies to frame the task of style transfer. End-to-end learning approaches enable learning of a latent representations of an input [START_REF] Kelleher | Deep Learning[END_REF], encoding potentially both the input content and style. Historically, such style transfer approaches have been inspired by the Neural Machine Translation techniques, basically a sequence-to-sequence encoder-decoder architecture based on recurrent neural networks [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]. [START_REF] Ma | A semantic relevance based neural network for text summarization and text simplification[END_REF] applied this approach to text simplification and summarization (considering the style here as "verbosity"). The key idea is that the text embedding vector generated by the encoder is a style-free representation of the input text [START_REF] Rabinovich | Personalized machine translation: Preserving original author traits[END_REF]. A major limitation of this approach is the need for parallel data. In some scenarios this data challenge can be addressed by either relying on intermediate resources-e.g., using "zero-shot translation" [START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF][START_REF] Carlson | Zero-shot style transfer in text using recurrent neural networks[END_REF]-or monolingual data-e.g. using "back translation" [START_REF] Sennrich | Improving neural machine translation models with monolingual data[END_REF][START_REF] Shrimai Prabhumoye | Style transfer through back-translation[END_REF].

Alternatively, Generative Adversarial Networks [START_REF] Goodfellow | Generative adversarial nets[END_REF] have been implemented for style transfer, removing the need for parallel data. These models include a generator block which generates two sequences for each input sequence (a style transferred sequence and a reconstructed sequence), and a discriminator block (a classifier) which tries to guess whether a given sequence is style transferred or reconstructed. Generators are usually standard encoder-decoders [START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF][START_REF] Singh | Sentiment transfer using seq2seq adversarial autoencoders[END_REF][START_REF] Fu | Style transfer in text: Exploration and evaluation[END_REF][START_REF] Romanov | Adversarial decomposition of text representation[END_REF], but various refinements have been proposed. For instance, several generators (decoders) can be used, one for each style [START_REF] Fu | Style transfer in text: Exploration and evaluation[END_REF], or a variational encoder can be used for encoding [START_REF] Hu | Controllable text generation[END_REF][START_REF] John | Disentangled representation learning for text style transfer[END_REF]. In most cases, the decoders are conditioned on a latent representation (embedding) of the input sequence concatenated with an embedding of the target output style. The concept of discriminator can also be extended, for instance in order to force encoded representations of the input to either focus on its style or meaning or both [START_REF] Romanov | Adversarial decomposition of text representation[END_REF][START_REF] John | Disentangled representation learning for text style transfer[END_REF].

Evaluation aspects

This section introduces the evaluation methodologies we use in our experiments. Each of these methods is taken from the literature and each focuses on a different aspect of textual style transfer: content preservation, style transfer strength and fluency.

Style shift power Style shift power investigates how well the output of a model fits with the target style. One approach is to employ a pre-trained classifier to measure the percentage of the style-shifted sequences which are labeled with the desired style [START_REF] Fu | Style transfer in text: Exploration and evaluation[END_REF][START_REF] Li | Delete, retrieve, generate: a simple approach to sentiment and style transfer[END_REF][START_REF] Leeftink | Towards controlled transformation of sentiment in sentences[END_REF][START_REF] Singh | Sentiment transfer using seq2seq adversarial autoencoders[END_REF][START_REF] Shrimai Prabhumoye | Style transfer through back-translation[END_REF][START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF][START_REF] John | Disentangled representation learning for text style transfer[END_REF][START_REF] Hu | Controllable text generation[END_REF].

Here, we employ the same classifier type as [START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF], the TextCNN model [START_REF] Kim | Convolutional neural networks for sentence classification[END_REF] and train it on the Yelp dataset. It achieves an accuracy of %97 on predicting sentiment (positive and negative classes) on the test set of the Yelp dataset.

Content preservation Many approaches, usually inherited from historical NLG tasks, have been employed to compare how well the generated output preserves the meaning of the original sequence.

We compute a content preservation rate between an input sequence and a generated sequence as the cosine similarity between their embedded representations. The process we use to generated the embedding of a sequence follows the method proposed by [START_REF] Fu | Style transfer in text: Exploration and evaluation[END_REF]. Given a sequence of words x = [x 1 . . . w N ], each token x i is represented by an embedding e i ∈ R 100 . These embeddings are generated using a pre-trained 100-dimensional GloVe model [START_REF] Pennington | Glove: Global vectors for word representation[END_REF]. Then, we calculate the min, mean and max pooling of these word embeddings, namely m, µ, M as follows:

m = min 1≤i≤N e i,j 1≤j≤100
(1)

µ = 1≤i≤N e i,j N 1≤j≤100 (2)
M = max 1≤j≤100 e i,j 1≤i≤N
(3)

The embedding of the sequence x is then created by the concatenation of m, µ and M . In practice, sentiment markers1 are removed from the sequence x to make sure the content preservation metric indeed measures the content similarity. Finally, when processing a dataset, content preservation is averaged over all its sentences.

Fluency Following [START_REF] Zhao | Adversarially regularized autoencoders[END_REF] and [START_REF] John | Disentangled representation learning for text style transfer[END_REF], we measure the fluency of an output sequences using language model perplexity. For our experiments, we trained an RNN-based language model, consisting of single-layer RNN with GRU cells [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF], on the Yelp dataset. Again, tokens are embedded using a 100-dimensional layer which was initialized at training time with a pretrained GloVe model [START_REF] Pennington | Glove: Global vectors for word representation[END_REF].

Models

This section introduces three models employed in this work to deal with textual style transfer: a stateof-the-art adversarial network, which we refer to as the base model (Section 4.1); an encoder extension of the base model employing a novel type of encoder (in Section 4.2); a generator extension of the base model employing multi-generators (Section 4.3).

Base model

The base model is the adversarial style transfer model proposed by [START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF]. As depicted in Figure 1, this model is composed of the following components:

Encoder An encoder E which reads a sequence x of style s ∈ {1, 2}, denoted as x (s) and outputs an embedded representation denoted as z.

Generator A generator G which is initialized with z and the desired output style s. The output is a sequence of words where the content is supposed to be the same as in x and style should be s. E and G are based on a standard sequence-to-sequence architecture [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF]. In the case where s is the original style, the model forms an auto-encoder. During training, G is used twice for each input: once to generate a reconstructed sequence (i.e., in the same style as the input), and once to generate a style-transferred sequence (i.e., in the opposite style to the input).

Discriminators Two discriminators, D 1 and D 2 , one for each style s 1 , and s 2 . The style-specific discriminator, D s takes a generated sequence and predicts the probability that this sequence has the style s. In other words, depending on whether s is the same as the original style of the input sequence or different from it, it predicts whether the style of the generated sequence has been "preserved" or "transferred".

Figure 1: Given input sequences x with an original style s, E encodes x (s) in a latent representation z. G is initialized with z and the desired output style, 1 or 2, leading to x(1) or x(2) , respectively. Discriminators D 1 and D 2 aim at detecting whether the styles of their inputs (i.e., the sequences generated by G) are "preserved" or "transferred" relative to s.

This architecture is designed to be trained in an adversarial manner: the generator, G when conditioned with style s, aims at generating a sequence that should convince the discriminator, D s , that the generated sequence has the style s, whereas D s aims at detecting style-shifted sequences.

The training process involves processing two differently styled input sequences in parallel x

(s 1 ) 1 and x (s 2 )
2 , where s 2 = s 1 . This results in four generated output sequences, one output sequence per style for each input sequence, leading to two reconstructed sequences of x

(s 1 ) 1 , x (s 2 )
2 , and two style-transferred sequences x (s 1 ) 2 , and x

(s 2 ) 1 .
The adversarial training relies on 3 elementary losses. For simplification, we assume that training data has the two sequences x

(s 1 ) 1 and x (s 2 ) 2 .
• The reconstruction loss, L rec , is the negative log probability of the training data, and is computed as the cross-entropy between an original sentence, and the reconstructed sentence (equation 4). Backpropagation is carried out for the couple (E, G) using this loss.

L rec = -log Pr( x (s 1 ) 1 |x (s 1 ) 1 )-log Pr( x (s 2 ) 2 |x 2 (s 2 ) ) (4)
• The adversarial loss is defined such that minimising this loss requires minimising the precision of the discriminators in detecting that the input sentence has been transferred. Equation 5 lists this loss, note that it is computed solely on the transferred sequences. This loss is the log of the inverse probability predicted by the discriminator and so minimising this measures causes the generation block to generate style-shifted sequences with lower probability of being detected as transferred. L adv,s 2 is defined symmetrically to Equation 5.

L adv,s 1 = log(1 -D s 1 ( x (s 1 ) 2 )) (5)
• Equation 6lists the discrimination loss, L Ds , for a given style s. This loss is defined as the binary cross-entropy over "transferred" and "preserved" classes where the true labels of style-shifted and reconstructed outputs are considered as "transferred" and "preserved" respectively. For each style s, we train D s to maximize the probability of assigning these true labels to the output sequences by minimizing this loss.

L Ds = -log D s ( x (s) 1 )-log(1 -D s ( x (s) 2 )) (6)
The backpropagation for the encoder-generator couple (E, G) is carried out using the following:

L total = L rec + L adv,s 1 + L adv,s 2 (7)

Generator variant: style-specific generators

We propose a multi-generator extension of the base model which employs style-specific generators G 1 , G 2 (one for each style). The generators share the encoder component and given a z generated by this shared encoder, each generator generates a sequence in its corresponding style. Therefore, unlike the base model, in this framework, there is no need to condition the generation on the desired style vector.

The role and functioning of the discriminators in this model are the same as those of the base model. In this model, there are two reconstruction losses, one for each generator (equations 8 and 9). Adversarial loss, discriminator losses, and total loss are the same as those of the base model (equations 5, 6 and 7) and L rec in equation 7 is the summation of the L rec1 and L rec2 . The training process of this model is also the same as the base model apart from the fact that the backpropagations following the reconstruction losses and total loss are carried out for the the couples (E, G 1 ) and (E, G 2 ).

L rec1 = -log Pr G 1 ( x (s 1 ) 1 |x (s 1 ) 1 ) (8)
L rec2 = -log Pr Gs 2 ( x (s 2 ) 2 |x (s 2 ) 2 ) (9)
5 Datasets and experimental setup Datasets For this work we limit the style transfer task to sentiment and opinion polarity transfer in text and evaluate the proposed frameworks using the Yelp dataset of restaurant reviews. It is a large-scale review dataset (4.7 million reviews) where text reviews are annotated by the stars (on a scale of 5) given by users. The reviews with rating above three and below three are considered as positive and negative corpora respectively, and three-starred reviews are discarded.

The dataset is annotated at a review level. However, since the unit of analysis in our experiments is sentence, we must project the review level annotations to the sentence level. This involves a one to many mapping of labels: one review label to multiple sentences. A question that arises here is whether it is appropriate to label every sentence in a review with the overall sentiment label of the review. For example, a review may have an overall sentiment rating that is very positive but certain sentences in the review may be quite neutral. To address this problem, and following the process of [START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF], we first filtered all reviews that had more than 10 sentences, and then we removed all sentences that had more than 15 tokens. The intuition behind these filtering processes is that longer reviews are more likely to have neutral sentences, and that longer sentences are more likely to be neutral. We split the resulting dataset, including 250K negative, and 350K positive sentences, into training (70% of the sentences), development (10%) and test (20%) sets while maintaining the relative distributions of positive and negative instances within each split. Finally, we created a balanced training dataset by upsampling the negative sentences in the training split, by randomly selecting negative sentences without replacement for repetition within the training split. Upsampling was not applied in the development or test set. The vocabulary size of the final dataset is 10K after replacing words occurring less than 5 times with the <unk> token.

Model settings

The encoder and generator cells of the models in this work are single-layer RNNs with GRU and the size of hidden cells is set to 700 in base model and multi-generator model, whereas this is 3072 when using ELMo in the encoder part. Token vectors are initialized by pre-trained GloVe [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] and their size is set to 100. Discriminators are TextCNN classifiers from [START_REF] Kim | Convolutional neural networks for sentence classification[END_REF].

Multi-Dimensional System Evaluation

This section reports on four multi-dimensional experiments on sentiment transfer. Firstly, a comparison between the standard and ELMo-based encoders is presented (Section 6.1). Then, attention shifts to the generator component (Sections 6.2 and 6.3). Section 6.2 evaluates the benefits of using multiple style specific generators, and Section 6.3 evaluates the effects on content preservation of reinforcing the latent representation of the input throughout the output generation process. Finally, Section 6.4 validates our multi-dimensional experimental framework for style-transfer against human evaluations. In the following sections, we will discuss the results of each of these experiments which are listed in Table 1.

Experiment 1: ELMo-based encoder

Given the many recent examples of ELMo embeddings being used in state-of-the-art NLP systems, our first experiment evaluated whether using an ELMo-based encoder would improve the performance of the baseline style-transfer model. The ELMo-based variant of the base model was introduced in Section 4.2. The performance of the base model and its ELMo-based variant are reported in rows a and b of Table 1.

The results of these two models reveals that using ELMo-based representations of the input sequences reduces the performance of the model in all three aspects of evaluation. To better understand these results, we designed a probing experiment to see how much the latent representations of the input sequences generated by each system encoder (i.e., the z-vectors) encodes source stylistic information. We will introduce the probing experiments in detail in section 7; however, for now, the results of this probing experiment for the base model and ELMo variant are listed in rows a and b of Table 2.

Comparing rows a and b of Table 2 reveals that ELMo does appreciably worse in encoding stylistic features compared to a style transfer-specific encoder. We hypothesise that the role of encoders in style transfer models is not only encoding the information, but also, distinguishing between stylistic and content-related features which helps the style-shifting generation process. In more detail, we hypothesise that the performance of the ELMo-based model drops because relative to the style-transfer specific encoders the ELMo-base z-vector representations have encoded less information relating to the input style and this makes it more difficult for the generator to decide whether it is reconstructing the input or style-shifting the input. When a z-vector strongly encodes the input style the model has information relating to both the input style and the target style (it receives this through the style vector fed directly into the generator) and so has a comparative signal as to whether the generation task involves reconstructing or style-shifting the input. However, as stylistic information is stripped from the z-vector this signal is weakened and the task of the generator becomes less well defined and so more difficult.

The main finding of this experiment is that there is a need for style transfer-specific encoders. Given this result, in the next experiment we will test whether it is possible to improve style transfer by adjusting the generator component of the pipeline.

Experiment 2: Style-specific generators

We aim at modifying the generator component of the architecture by implementing style-specific generators and investigate their effect on the the style-shift strength. The style-specific generator variant of the baseline architecture evaluated in this experiment was described in Section 4.3. 1: Fluency, style shift strength and content preservation of transferred sequences. The higher the style-shift power and content preservation power are the better it is, but for the fluency (perplexity) the lower results show the better performance.

Model

The performance of the model with multiple style-specific generators is listed in row c of Table 1. Comparing these results with the performance of the baseline system reveals an improvement in the style-shift power and also fluency. The increase in style-shift power is due to the fact that in the multigenerator structure, each generator can learn a specific style. Also, the increase in fluency is due to the fact that each generator learns the distribution of one corpus (i.e., the distribution over the samples in the relevant style). This can provide further explanation for the increase of style-shift power in multigenerator frameworks (an increase from 78.76% to 94.41% in rows a and c of Table 1).

Furthermore, results of probing classification task from Table 2 indicates that multi-generator architectures (rows c and e) encode less source stylistic features compared to single-generator systems (rows a and d). As noted above, in a single generator model encoding input style in z-space provides a signal to the generator relating to whether it is reconstructing or style-shifting the input. However, the fact that multi-generator systems have style-specific generators obviates the need for this signal and this is reflected in the multi-generator system encoding less source stylistic features in z-vectors compared to single-generator models while avoiding the deterioration in style-shift and fluency performance observed with the ELMo encoder system. However, in spite of an increase in style-shift power and fluency, we lose content preservation power in this framework compared to base model. To deal with this, in section 6.3 we implement techniques to reinforce content throughout the generation process.

Experiment 3: Reinforcing the content

In this section, we investigate whether reinforcing the latent representation of the input sequence (zvectors) during generation improves the content preservation power of a model. We do this experiment for both the base model and the multi-generator model.

In the standard setup of the base model and also the multi-generator model z is provided once to the generator at the start of of the generation process, i.e.

x (s) 1 = G(z, x (s) 0 , s) but x (s) i>1 = G(h i-1 , x (s)
i-1 , s) with h i-1 being the hidden state of the generator, x 0 a special <start> token, and x i the token generated at step i. The strategy we test for reinforcing content involves z being provided to the generator at the start of the generation process and also being reinforced at each generation step by concatenating z to the GRU logits output vector prior to passing it to the softmax layer (this concatenation method was proposed by [START_REF] Tanti | Where to put the image in an image caption generator[END_REF]).

Table 1 row d shows that reinforcing the content throughout generation slightly improves content preservation in the base model but also results in a drop in style-shift power and fluency. For the multigenerator model (Table 1 row e), reinforcing content negatively affects all three aspects of evaluation. To provide an overall baseline for the content preservation of these systems for each system we calculated the cosine distance between the style shifted sentences generated by each system when applied to the test set and randomly selected sentences from the training set with the same style. The average of these scores across different frameworks can be considered as lower bound of content preservation rate and is 0.817. This shows that all the models perform better in content preservation than the lower bound.

Intuitively, the effect of reinforcing z throughout the generation process is dependent on what information is encoded in z. For example, if z only encodes content information then reinforcing z should not diminish style shift power; however, if z encodes both style and content information then reinforcing z may help with content preservation but this will likely be at the cost of a loss in style shift power. Also, if z encodes information from a distinct distribution of language (e.g., positive versus negative sentiment) reinforcing z may inhibit the fluency of the language model. The results of our probing experiment in Section 7 provide some insight into these dynamics.

The results of the probing task in rows d and e of Table 2 illustrates that z-reinforcement leads the models to embed less style-specific information in z. Our hypothesis for what causes this is that the generator component is trying to output a sequence in a specific style; however, if a lot of stylistic information is encoded in z and this stylistic information is repeatedly reinforced during generation then z interferes with the ability of generators to shift to a new style. In order to overcome this dynamic the systems learn to encode less stylistic information in z, thereby reducing the interference in the generator between the source style and desired style. This leads the single-generator model to lose more power to transfer the style compared to the multi-generator model (a drop of 12.8% in style-shift strength in single-generator models versus 3.7% in multi-generator frameworks in rows d and e of Table 1). The reason is the former was relying more on style-specific information in z than the latter, as we noted earlier the single-generator model may use the style information in z to determine whether to reconstruct or style shift the input. Overall, there is an inherent trade-off for single-generator models with respect to encoding stylistic information in z: reducing stylistic information in z makes the initial decision between reconstruction and style-shift more difficult, but increasing the stylistic information in z makes style-shifted generation more difficult. Multi-generator models do not need source style signals for their style-specific generators; therefore, they drop more source style in their z-space compared to singlegenerator models (7% versus 2% ). However, this negatively affects the content preservation power of these models, since, seemingly, dropping more source style in z-space leads to losing some features which are either content or the type of features which can be considered as the intersection between content and style.

Experiment 4: Validating the Methodology with Human Evaluations

To investigate whether the results of our automatic evaluation correlate with the human judgment, we did a manual test considering style-shift, content preservation and fluency. The tests were conducted in a strictly blind fashion by shuffling samples and models to avoid biases while answering the questions. To do the test, 450 samples were considered: 150 samples randomly selected from the Yelp test set and their corresponding style-transferred sequences of the base and multi-generator models. Each sample was evaluated at least 3 times and at most 8 times. The final scores are reported as the proportions for each answer, averaged over all testers and all samples. Moreover, Krippendorff's inter-rater agreements [START_REF] Krippendorff | Content analysis: An introduction to its methodology[END_REF] is reported. Below are the details for each aspect.

Style shift power In this test the testers considered one style-shifted sample at a time, and labelled it as "positive", "negative", or "neutral". There were 29 participants with an inter-rater agreement of 0.752. The results indicate that base model is successfully transferring the style of the sequences in 58.3% of the cases while this is 67.6% for the multi-generator model.

Content preservation

This test was conducted in a comparative manner, i.e. evaluators were shown a source sentence and two style-shifted sentences from the two models and were asked which sequence most closely resembles the source sentence in terms of content (disregarding the sentiment). Possible options were "equally good", "equally bad", "first sample is better", or "second sample is better". There were 22 participants in this study with an the inter-rater agreement of 0.772. In 52.2% of the cases, the base model appears to preserve the content, while this is 41.6% for the multi-generator model.

Fluency This test was evaluated in a similar way as style transfer. The evaluators judged the structural correctness of generated samples as "incorrect", "partly correct" and "correct". As a post-processing step, the "partly correct" answers were discarded since the task appeared as ambiguous for the annotators. The were 25 participants with inter-rater agreement of 0.568. The results show that the multi-generator performs slightly better than the base model ( 67.6% versus 64.2% of "correct" answers). In conclusion, these results are in line with the rows a and c of the Table 1 and so we interpret them as confirming the validity of the automatic metrics selected in our work.

Model

Probing the Relationship between Style and Content

In Section 6.3 we observed that reinforcing the latent representation of the input (z) during generation could result in improved content preservation but also consistently resulted in a deterioration in styleshift power. In this section we examine whether this trade-off is a result of stylistic information being encoded within the latent representation of the input. Inspired by the probing experiments described by [START_REF] Conneau | What you can cram into a single vector: Probing sentence embeddings for linguistic properties[END_REF], we designed probing experiment to examine what style information each of our models encoded in their latent representations.

To do the probing classification experiment, for each style-transfer framework we trained a classifier that took the z representation generated by the encoder of the framework and predicted the style of the input sequence that had been feed into the encoder. The idea here is that if we can train a classifier to accurately predict the style of an input from the z representation of an input sequence generated by the encoder, then this z vector must have information related to the style of the input sequence. Therefore, by examining the accuracy of the style classifier for each style-transfer framework we can get an insight into the amount of stylistic information the framework encodes in its z space. For each style-transfer framework, we trained a separate feed-forward network with a single hidden layer and a sigmoid output layer as the classifier using pairs of z vectors (embedded representations) of the input sequences and their original style. We employed the Yelp review dataset for training where the distribution of the "positive" and "negative" labels are balanced. Table 2 shows the performance of each classifier. For all style-transfer frameworks we were able to train accurate input style classifiers. This indicates that all the encoders include style information in the z state. There is, however, variations across the styletransfer frameworks in terms of the amount of style information they encode in their z state. In particular, reinforcing the input during generation leads to a reduction in stylistic information being encoded in z.

As discussed earlier we believe this is caused by the systems attempting to ameliorate the interference caused by reinforcing the input style when attempting to generate in another style.

Further discussion and future steps

Our results indicate that style cannot be totally separated from content. This is because encoders mistakenly strip out content when attempting to remove source stylistic features. This is most clearly seen in the interaction between the results of the probing experiment and the content preservation results for the different systems. The probing experiment revealed that reduction in encoding the stylistic features in the latent representation of the inputs as a result of reinforcing the content (table 2) can lead to a reduction in content preservation of the model. This indicates a direct relationship between the content preservation power of a model and how much of source style the model encodes in its latent space. Therefore, it can be inferred that style and content are entangled elements, which suggests that style should be considered as an integral component of a text which cannot totally be separated from the content of the text. This raises questions about the conceptual basis of the computational modelling of style which considers style as independent from content and fragmentizes style into a set of explicit linguistic elements such as specific words, markers, or syntactic structures [START_REF] Li | Delete, retrieve, generate: a simple approach to sentiment and style transfer[END_REF][START_REF] Leeftink | Towards controlled transformation of sentiment in sentences[END_REF].

To summarize, the results of this study show the necessity of employing style transfer-specific encoders as opposed to pre-trained embeddings. Furthermore, the results indicate that each of the tested frameworks has its own strength and weaknesses. Indeed, our result indicate the benefits of framing style-transfer as a multi-dimensional task: given the trade-off we observed in terms of style-shift and fluency versus content preservation it is important to consider all of these aspects when evaluating a system as understanding the trade-off systems make can inform the design of improved style-transfer systems. Finally, our results suggest that stylistic and contextual features cannot really be separated or at least not in the case context of framing sentiment as a style.

Overall, we consider the multi-generator framework as a good basis for future research, and we plan to implement alternative techniques to improve content preservation power in this architecture. Furthermore, considering the significance of doing the evaluation in a comprehensive manner, we would like to focus on introducing a single evaluation metric for this task which takes the three evaluation aspects of style-shift, content maintenance and fluency into consideration.

Figure 2 :

 2 Figure 2: Detail of the ELMo-based encoding.

Table 2 :

 2 The accuracy of the classifiers trained for each of the sentiment-transfer frameworks.

	Accuracy

The sentiment dictionary is the one from[START_REF] Fu | Style transfer in text: Exploration and evaluation[END_REF]

Acknowledgements

This study has been realized under the ANR (French National Research Agency) projects TREMoLo (ANR-16-CE23-0019) and TextToKids (AAPG 2019).

This project has also been supported by the ADAPT Centre for Digital Content Technology which is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund.

Appendix A. Samples of the style-shifted sequences From negative to positive 1. so nasty . 2. so awesome . 3. ugh delicious 4. so delicious . 5. so fun . 6. so awesome ! .


Carrière Mathieu
email: mathieu.carriere@inria.fr

Datashape

Andrew J Blumberg
email: blumberg@math.utexas.edu

Multiparameter Persistence Images for Topological Machine Learning

In the last decade, there has been increasing interest in topological data analysis, a new methodology for using geometric structures in data for inference and learning. A central theme in the area is the idea of persistence, which in its most basic form studies how measures of shape change as a scale parameter varies. There are now a number of frameworks that support statistics and machine learning in this context. However, in many applications there are several different parameters one might wish to vary: for example, scale and density. In contrast to the one-parameter setting, techniques for applying statistics and machine learning in the setting of multiparameter persistence are not well understood due to the lack of a concise representation of the results. We introduce a new descriptor for multiparameter persistence, which we call the Multiparameter Persistence Image, that is suitable for machine learning and statistical frameworks, is robust to perturbations in the data, has finer resolution than existing descriptors based on slicing, and can be efficiently computed on data sets of realistic size. Moreover, we demonstrate its efficacy by comparing its performance to other multiparameter descriptors on several classification tasks.

Introduction

Topological data analysis (TDA) is a new and rapidly evolving branch of computer science and statistics that provides tools to analyze geometric structures in data using ideas from algebraic topology. The success of clustering methods and nonlinear dimensionality reduction techniques make it clear that even crude approaches to leveraging the geometry of data can be very effective. TDA provides more refined geometric information, and indeed, there have been a variety of successful applications of TDA, including, among others: graph analysis [CCI + 20, ZW19], computational biology [ACC + 20, CR20, GPCI15, HMMB19, KDS + 18, RB19], finance [dCBSB17, GGK + 18, GK18] and computer graphics [START_REF] Mathieu Carrière | Stable topological signatures for points on 3D shapes[END_REF][START_REF] Li | Persistence-based structural recognition[END_REF][START_REF] Poulenard | Topological function optimization for continuous shape matching[END_REF].

The standard setup for the use of TDA is a data set given as a finite metric space X (i.e., points and a distance function) and a continuous function f : X → R. This function can be understood as giving a parameter that filters X and encodes how the topology changes as the parameter varies. A classical setting is when X = R n with the standard Euclidean distance and f is given as the distance to a point cloud P ⊂ R n , that is, f (x) = min { x -p : p ∈ P }. See Figure 1. Another example is when X is a graph G and f is a function defined on the nodes of G (see Supplementary Material, Section 4).

The fundamental geometric summary of TDA, the persistence diagram, characterizes the changing topology of the family of sublevel sets of f , that is, the family {F α } α∈R = {x ∈ X : f (x) ≤ α} α∈R . For instance, in the point cloud setting, the sublevel set F α is the union of balls of radius α centered on the points of P . See Figure 1. Persistence diagrams are built by increasing α from -∞ to +∞, which creates a growing sequence of sublevel sets, called a filtration, and recording the various topological changes that occur in this process. These changes are eventually summarized into a set of points in the plane R 2 , where each point represents a specific topological feature of the data set (e.g., a connected component, a cycle, a cavity...), and its coordinates are the parameter values α 1 , α 2 for which the structure appeared and disappeared in the filtration. Often one views points away from the diagonal, i.e., which represent topological features that existed at many scales, as encoding robust geometric information and near-diagonal points as noise.

Figure 1: The filtration of the distance to the point cloud function is a sequence of union of balls with growing radii. As the radius increases, topological structures appear (e.g., the loop highlighted in red in the third union) and are eventually filled in. The "birth" and "death" values are the coordinates for a point in the persistence diagram, colored to indicate their dimension (0 is red, 1 is blue).

A problem with the summaries provided by persistence diagrams is that the space of persistence diagrams is not a convenient place to do statistics. For example, it is not a vector space, and centroids are hard to compute and not necessarily unique [START_REF] Turner | Fréchet means for distributions of persistence diagrams[END_REF]. To handle this, a lot of work has gone into developing frameworks for supporting statistics, and machine learning for persistence diagrams has been developed in the past few years [AEK + 17, BGMP14, Bub15, CCO17, CGLM15, FLR + 14, KHF16, LY18, RHBK15]. However, the use of single-parameter filtrations often misses relevant information. For instance, in the point cloud setting, the scale filtration does not account for density, and so can be susceptible to outliers; for example, see Figure 4.

There are various approaches to handling the issue of variation in density in the context of the statistical frameworks for standard persistence. A more flexible general framework was introduced by Carlsson and Zomorodian: multiparameter (sometimes called multidimensional) persistence [START_REF] Carlsson | The theory of multidimensional persistence[END_REF]. This formalism encodes multiple filtration directions; the simplest example is the case of bifiltrations, that is, by filtering X with two parameters, or functions, instead of just one. For example, point cloud outliers can be detected by simultaneously filtering by scale and density.

However, working with multiparameter persistence is substantially more difficult. Analogues of persistence diagrams can be defined in restricted settings for some specific bivariate and multivariate functions [BCB18, BLO20, CKMW20, CO19] and metrics between them have been defined, as well as algorithms for their computation [START_REF] Kerber | Efficient approximation of the matching distance for 2-parameter persistence[END_REF][START_REF] Kerber | Exact computation of the matching distance on 2-parameter persistence modules[END_REF]. But in general there is no simple summary of a multiparameter persistence diagram and the integration with statistics and machine learning is still unsettled. Existing approaches for statistics in the context of bivariate functions [CFK + 19, Vip20] are defined by slicing, that is, by considering persistence diagrams associated to linear combinations of the coordinates of the bivariate function, which is known to be limited since it is equivalent to an incomplete summary of multiparameter persistence called the rank invariant [START_REF] Carlsson | The theory of multidimensional persistence[END_REF].

In this article, we provide a more refined numerical invariant of multiparameter persistence:

• We introduce the Multiparameter Persistence Image, a compact descriptor which integrates information across slices by tracking changes in adjacent slices. We also investigate empirically the stability of this invariant in the face of perturbations of the slices.

• We show how to efficiently compute this descriptor using any black box matching algorithm, and provide more details for the vineyards algorithm [CSEM06] that we use in experiments. We also provide open-source implementation for our descriptor and for the other approaches [CFK + 19, Vip20], in a public Python package [START_REF] Mathieu | Multipers package[END_REF].

• We demonstrate in several experiments classification experiments that the multiparameter persistence image has performance comparable to or superior to existing summaries.

Background

In this section, we briefly introduce the basics of single and multiparameter persistent homology as well as the vineyard algorithm [START_REF] Cohen-Steiner | Vines and vineyards by updating persistence in linear time[END_REF] we will use to compute our descriptor. We refer the interested reader to e.g., [START_REF] Edelsbrunner | Computational topology: an introduction[END_REF][START_REF] Carlsson | Topological pattern recognition for point cloud data[END_REF][START_REF] Oudot | Persistence theory: from quiver representations to data analysis[END_REF] for a more complete introduction to persistence.

Single-parameter persistent homology. Let X be a topological space and f : X → R be a continuous function. The family {F α } α∈R of sublevel sets of f , where F α := {x ∈ X : f (x) ≤ α}, induces the following growing sequence of subspaces of X, for any parameters

α 1 ≤ α 2 ≤ • • • ≤ α n , called a filtration: F α1 ⊆ F α2 ⊆ • • • ⊆ F αn .
Topological spaces connected by continuous maps induce vector spaces connected with linear transformations via the homology functors H k , for k ∈ N. We do not explicitly review the definition, but simply recall that each element in a basis of a vector space H k (F α ) should be thought of as a k-dimensional topological feature of F α (such as a connected component, a branch, a loop, a cavity, etc.), and that the linear map v i :

H * (F αi ) → H * (F αi+1 )
provides a correspondence between the features of F αi and F αi+1 . Such a sequence of vector spaces connected with linear maps is usually called the persistence module of f , denoted M (f ).

An essential property of persistence modules is that they can be canonically decomposed into a direct sum of simple modules:

M (f ) i∈I I(α bi , α di ), (1)
where I(α b , α d ), α b ≤ α d , denotes the interval module between α b and α d , which contains vector spaces of dimension 1 connected by identity maps between α b and α d , and vector spaces of dimension 0 everywhere else. An interval module intuitively represents a topological feature of X that appeared (was born) at parameter α b and disappeared (died) at parameter α d in the filtration.

A common way of representing decomposition (1) is to either use the values α bi and α di as coordinates of 2D points, which leads to a set of points in the plane called the persistence diagram dgm(f ), or, equivalently, as endpoints of intervals, which leads to a multiset of intervals called the barcode bcd(f ). Persistence diagrams and barcodes can be equipped with a distance called the bottleneck distance d B , which is computed by finding a partial matching which minimizes the difference between matched bars (and penalizes unmatched bars according to their length). Moreover, there is a standard distance that is commonly used to compare persistence modules, called the interleaving distance d I . These two metrics turn out to be the same, i.e., d I (M (f ), M (g)) = d B (bcd(f ), bcd(g)). The metric lets us express the most important property of persistent homology, namely that it is stable with respect to this distance: d I (M (f ), M (g)) ≤ f -g ∞ , for any two continuous functions f, g : X → R satisfying very mild hypotheses.

Vineyards. Persistence diagrams and barcodes are computed by constructing a filtration of a simplicial complex built from the data. The general algorithm takes O(n 3 ) time, where n is the number of simplices, which can grow rapidly in the number of data points and the range of the filtration. However, if a persistence diagram or barcode has already been computed from a function f , we can efficiently compute a new one for a perturbation f of f using the vineyard algorithm [START_REF] Cohen-Steiner | Vines and vineyards by updating persistence in linear time[END_REF], which applies a sequence of O(n) updates of the filtration.

Moreover, this algorithm, that we denote vine, also provides a matching between the bars of bcd(f ) and bcd( f

): vine(f, f ) = {bcd(f ), bcd( f ), m f, f },
where m f, f is a partial matching, that is, a 1-to-1 matching between a subset of bcd(f ) and a subset of bcd( f ). In favorable cases, this matching realizes the bottleneck distance.

Multiparameter persistent homology. Similarly to single-parameter persistent homology, a function f : X → R d leads to a multifiltration. For any a i , a j ∈ R d , we use a i ≤ a j to denote that each coordinate of a i is smaller than the corresponding coordinate of a j . In this case, the sublevel sets F a = {x ∈ X : f (x) ≤ a} still satisfy F ai ⊆ F aj as soon as a i ≤ a j , even though ≤ is only a partial order in R d . For any family of parameters a 1 , . . . , a n ∈ R d , the spaces F ai and the inclusions F ai ⊆ F aj when a i ≤ a j is called a multifiltration of f , which, after applying the homology functor again, gives raise to the multiparameter persistence module M (f ).

Unfortunately, general multiparameter persistence modules do not admit decompositions such as (1), except in very special cases [BCB18, BLO20, CO19]. However, for any two points

a i ≤ a j ∈ R d , the line : t → (1 -t)a i + ta j , t ∈ R, defines a filtration {F t } t∈R with F t = {x ∈ R d : f (x) ≤ (t)}, since (t 1 ) ≤ (t 2 ) for any t 1 ≤ t 2 ∈ R.
The associated barcode, denoted by bcd(f ) can be thought of as the barcode of the (scalar-valued) function equal to the restriction of M (f ) to , and whose bars can be plotted along in R d . The collection of such barcodes is referred to as the fibered barcode of f ; we use the term slicing to refer to the process of computing fibered barcodes. See [START_REF] Lesnick | Interactive visualization of 2D persistence modules[END_REF] for an extensive discussion of the fibered barcode.

The interleaving distance of one-parameter persistence generalizes to produce a metric d I on multiparameter persistence modules, also usually referred to as the interleaving distance [START_REF] Lesnick | The theory of the interleaving distance on multidimensional persistence modules[END_REF], which also enjoys stability properties in specific cases [START_REF] Blumberg | Stability of 2-parameter persistent homology[END_REF]. There is also a matching distance obtained as the supremum over the weighted bottleneck distances between barcodes in the fibered barcode, which serves as a lower bound for the interleaving distance.

The Multiparameter Persistence Image

Since multiparameter persistence modules cannot be readily used in statistics and machine learning applications, it is necessary to define a feature map or descriptor for it, that is, an Euclidean vector that summarizes the multiparameter persistence module. In this section we define our new descriptor, the Multiparameter Persistence Image. For simplicity, we restrict to the case of 2D persistence modules.

The basic idea of our construction is to select a collection of slices parametrized by lines { i } such that there is a natural ordering on the lines, match the barcodes of adjacent lines, and consider the region in R 2 specified by the collection of quadrilaterals with matched endpoints as vertices. In principle, one can use any black box matching algorithm, e.g., the bottleneck matching, to align the barcodes. In this work, we restrict to the vineyards algorithm for reasons of efficiency.

Let f be a bivariate function f :

X → R 2 and let R = [m 1 , M 1 ] × [m 2 , M 2 ] ⊂ R 2 be
the rectangle defined by the minimum and maximum of each function coordinate:

m 1 = min(f 1 ), M 1 = max(f 1 ), m 2 = min(f 2 ) and M 2 = max(f 2 ), where f (x) = (f 1 (x), f 2 (x)
). We call R the bounding rectangle of f . We first define the lines on which we are going to compute fibered barcodes.

Definition 3.1. For any θ ∈ [0, 2π], we let e θ denote the unit vector (cos(θ), sin(θ)). Moreover, for any x, v ∈ R 2 , we let (x, v) denote the line passing through x with direction vector v. Let L m N = { ((m 1 , m 2 ), e θi ) :

θ i = (i/N ) • (π/2), 0 ≤ i ≤ N } denote N ordered lines going through the lower left corner of R, L M N = { ((M 1 , M 2
), e θi ) : θ i = (i/N ) • (π/2), 0 ≤ i ≤ N } denote N ordered lines going through the upper right corner of R, and

L ∆ δ = { ((m 1 + iδ, m 2 ), e π/4 ) : 0 ≤ i ≤ (M 1 -m 1 )/δ} ∪ { ((m 1 , m 2 + iδ), e π/4 ) : 0 ≤ i ≤ (M 2 -m 2 )/
δ} denote an ordered set of evenly spaced lines with the same slope going through R. The Multiparameter Vineyard with lines L, denoted by V L (f ), is defined as:

L M N L m N (M1, M2) (M1, M2) (M1, M2) (m1, m2) (m1, m2) (m1, m2) L ∆ δ f1 f2 δ
V L (f ) = vine({f i } 1≤i≤N ), i.e., V L (f )
is the list of fibered barcodes and partial matchings {{bcd(f i )} 1≤i≤N , {m i } 1≤i≤N -1 }, where m i is a partial matching between the bars of consecutive fibered barcodes bcd(f i ) and bcd(f i+1 ).

Equivalently, the Vineyard Decomposition associated to V L (f ), and denoted by D L (f ), is defined as the set of all subsets

I = {b i , m (1) (b i ), m (2) (b i ), • • • }, where b i ∈ bcd(f i ) and m (p) (b i ) = m i+p • • • • • m i (b i ) and such that either i = 1 or m -1 i-1 (b i ) = ∅.
The subsets I are called the summands of the Vineyard Decomposition. The Vineyard Decomposition can be extended to multivariate functions by using multiple sets of ordered lines that always start with the same line. For example, in R 3 , one would sweep the 3D space with planes that all intersect on a given line. Sweeping each plane with lines, 2D summands can be computed in each plane and then connected through the common line to generate 3D summands.

VL(f ) DL(f )
To produce a descriptor suitable for machine learning and statistics, we now define the Multiparameter Persistence Image, which is a generalization and adaptation of the Persistence Image for barcodes d] denote a rectangle in the plane R 2 , and {P i,j } 1≤i,j≤p , p ∈ N * denote a grid of points evenly sampled on R, i.e., P i,j = (a + i(b -a)/p, c + j(d -c)/p). The Multiparameter Persistence Image of resolution p and bandwidth σ > 0 is the matrix I L,R,p,σ (f ) of size p × p such that

[AEK + 17]. Definition 3.3. Let R = [a, b] × [c,
(I L,R,p,σ (f )) i,j = I∈D L (f ) w(I) • ω( * ) exp - min ∈I P i,j , 2 σ 2 , ( 2
)
where P i,j , = min x∈ P i,j -x , w : D L (f ) → R is a weight function, and ω( * ) is a weight proportional to the minimum of the entries of the vector parametrizing the line * that achieves min ∈I P i,j , 2 . Typically, we use w(I) =

A(I) (b-a)(d-c) q
, where A(I) is the area of the convex hull of the endpoints of all bars in I and q ∈ N is a user-defined parameter. The key refinement of this construction over predecessors in the literature [CFK + 19, Vip20] (see Section 4) is that we are taking the vineyards matching between bars into account when constructing the invariant, as opposed to just using the slices.

Complexity. The complexity associated to the Vineyard Decomposition computation is the complexity of the computation of the barcode associated to the first line O(n 3 ) (where n is the number of simplices) plus the complexity of the vineyard algorithm on the set of lines O(N • n • v), where N is the number of lines, and v is the maximum number of filtration updates between consecutive lines (and goes to 1 when N → +∞). The Multiparameter Persistence Image of resolution p can be computed in O(p 2 • N • m), where m is the maximum number of bars of the fibered barcodes. [START_REF] Landi | The rank invariant stability via interleavings[END_REF] shows that the fibered barcode is stable in two senses. First, given a fixed line that has slope neither 0 nor ∞, parametrized by a vector e θ , the assignment M (f ) → bcd(f ) is stable in the sense that for 2D persistence modules M (f ) and M (g), we have

Stability. Work of Landi

d B (bcd(f ), bcd(g )) ≤ 1 ẽ d I (M (f ), M (g)) ≤ 1 ẽ f -g ∞ ,
where here ẽ is the minimum of the entries of e θ . Roughly speaking, the fibered barcode is stable in the interleaving distance with Lipschitz constant determined by the slope of the lines, where the constant approaches ∞ as the slope approaches either 0 or ∞. Second, given two lines and , Landi also shows that bcd(f ) and bcd(f ) are interleaved with a constant that depends on the slopes of and and again goes to ∞ as the lines approach either horizontal or vertical.

The stability properties of the Multiparameter Persistence Image depend on the local stability of the matched bars in adjacent lines. Proposition 3.4. Let 1 and 2 be two adjacent parallel lines parameterized by the vector e θ that are distance δ apart, e.g., consecutive lines in L ∆ δ (see Definition 3.1). Suppose that b 1 ∈ bcd(f 1 ) and b 2 ∈ bcd(f 2 ) are matched bars in the barcodes along 1 and 2 respectively. Given another 2D persistence module M (g) such that d I (M (f ), M (g)) < , then the change in the area of the quadrilateral specified by b 1 and b 2 is bounded by (2δ )/ẽ. Along a path of k parallel lines at most δ apart, the change in area of the region traced out by the bars is bounded by k • (2δ )/ẽ.

What the preceding proposition shows is that provided that the series of matchings given by the vineyards algorithm realize the bottleneck metric and are themselves stable under perturbation, any invariant based on the area of these regions which is weighted appropriately to compensate for the length of the path (i.e., the number of bars) and the slope of the lines will be stable to perturbation of the multiparameter persistence module and multifiltration in the interleaving distance or matching distance. We give precise theorem statements (and proofs) in Supplementary Material, Section 1. However, the issue of the stability of the matchings itself is a subtle one. In situations where bars merge and split apart again along the vineyard, the matching can be very sensitive to infinitesimal changes in the bars. There are two approaches to handling this. One possibility is to average the images over random perturbations of the input persistence module or the matchings. However, in practice, we can algorithmically detect whether such instability can occur for a given data set, by looking at the matchings that the bottleneck distance would produce: if multiple partial matchings achieve the bottleneck distance, then choosing arbitrarily between them might make the algorithm lose track of the topological feature. Moreover, we expect this kind of bad behavior to disappear for some well-behaved classes of multifiltrations [START_REF] Botnan | On rectangle-decomposable 2parameter persistence modules[END_REF][START_REF] Cochoy | Decomposition of exact pfd persistence bimodules[END_REF].

Experiments

In this section, we compare the Multiparameter Persistence Image to two other descriptors that have recently been proposed for multiparameter persistence, the Multiparameter Persistence Landscape [START_REF] Vipond | Multiparameter persistence landscapes[END_REF] and the Multiparameter Persistence Kernel [CFK + 19]. In the following, we only describe their computational approximations for sake of simplicity. See the associated reference for their precise definitions. Let L denote a set of lines (see, e.g., Definition 3.1).

Multiparameter Persistence Landscape [START_REF] Vipond | Multiparameter persistence landscapes[END_REF]. Let λ k : B → R d denote the usual kth-landscape vector construction (see [START_REF] Bubenik | Statistical topological data analysis using persistence landscapes[END_REF] for details), which turns a barcode into an Euclidean vector obtained as a sampling of the piecewise-linear function corresponding to the k-th landscape. The Multiparameter Persistence Landscape of a bivariate function Λ k (f ) ∈ R d×|L| is defined as the concatenation of all landscapes associated to the fibered barcodes {bcd(f )} ∈L where the lines in L have slope 1, which is the case when, e.g., L = L ∆ δ (see Definition 3.1). Multiparameter Persistence Kernel [CFK + 19]. Let K : B × B → R be any kernel between barcodes (see [START_REF] Mathieu Carrière | Sliced Wasserstein kernel for persistence diagrams[END_REF][START_REF] Kusano | Persistence weighted Gaussian kernel for topological data analysis[END_REF][START_REF] Le | Persistence Fisher kernel: a Riemannian manifold kernel for persistence diagrams[END_REF][START_REF] Reininghaus | A stable multi-scale kernel for topological machine learning[END_REF] for examples of possible kernels). The associated Multiparameter Persistence Kernel between two bivariate functions K K (f, g) is defined as

K K (f, g) = 1 |L| ∈L w( ) • K(bcd(f ), bcd(g )),
where w( ) is a weight function that depends on the slope of (in order to guarantee stability).

A potential advantage of the the Multiparameter Persistence Image is that both of these invariants depend only on the fibered barcode; persistence modules with the same fibered barcodes will end up having the same Multiparameter Persistence Landscapes and Kernels, but different Multiparameter Persistence Images. See Supplementary Material, Section 2 for an example.

Results. All experiments have been run on an AWS machine with a Xeon Platinum 8175 processor. Code for computing the descriptors and running the experiments is freely available [START_REF] Mathieu | Multipers package[END_REF]. Time series. Our first series of experiments involve time series classification. Time series analysis, e.g., periodicity detection, can be applied to the Takens embeddings of the time series [START_REF] Perea | Sliding windows and persistence: an application of topological methods to signal analysis[END_REF]. However, noise in the series might induced outliers in the resulting point cloud embeddings. Hence, it is natural to filter based on both distance to the point cloud and density estimates-see Figure 4 and [ACG + 18]. We use time series data sets from the UCR archive [DBK + 18] with moderate sizes and lengths; this ensures that the kernel matrices obtained with Multiparameter Persistence Kernel have reasonable sizes and that the point clouds obtained with the Takens embedding in R 3 have a reasonable number of points. Moreover, we use the train/test split that is suggested for each data set. See Supplementary Material, Table 1 for a description of the time series classification tasks.

We used L ∆ δ (see Definition 3.1) as our set of lines for computing vineyards and fibered barcodes (in homological dimensions 0 and 1) for each descriptor1 , with δ = ((M 1 -m 1 ) + (M 2 -m 2 ))/N , where m 1 , M 1 are the minimum and maximum distances between all points of all point cloud embeddings, and m 2 , M 2 are the minimum and maximum of the density estimates of all points of all point cloud embeddings, and N = 200. Density estimation is performed with the Distance-To-Measure [CFL + 18] with parameter m = 0.1.

We compare the accuracies of the Multiparameter Persistence Kernel computed with the Sliced Wasserstein Kernel between barcodes [CCO17] (reported as "MP-K"), the sum of the first 5 Multiparameter Persistence Landscapes (reported as "MP-L"), and the Multiparameter Persistence Image (reported as "MP-I"). Resolutions for Multiparameter Persistence Landscapes and Images are 5-fold cross-validated over the set of values {10, 50}, and the powers and bandwidths of the Multiparameter Persistence Images and Kernels are also 5-fold cross-validated with values in {0, 1} (power) and 10 {-2,-1,0,1,2} (bandwidth). We also compared to accuracies obtained with 1D persistence barcodes computed along the diagonal of the bounding rectangle and vectorized with the sum of the first five 1D-persistence landscapes (reported as "P-L") [START_REF] Bubenik | Statistical topological data analysis using persistence landscapes[END_REF], the 1D persistence image (reported as "P-I") [AEK + 17], and the persistence scale-space kernel 2 (reported as "PSS-K") [START_REF] Reininghaus | A stable multi-scale kernel for topological machine learning[END_REF]). Resolutions, bandwidths and powers for these descriptors were cross-validated as in the multiparameter case. Finally, we add the accuracies obtained with Euclidean nearest neighbor (B1), dynamic time warping with optimized (B2) and constant window width (B3), as provided and explained in [DBK + 18]. Our goal in this experiment is primarily to compare different topological summaries, although it is also interesting to see the relationship to the state of the art.

All features were trained with an XGBoost classifier, except for MP-K and PSS-K, which were trained with kernel Support Vector Machines. Results are displayed in Table 1 and computation time for homological dimension 0 are in Table 2 (homological dimension 1 can be found in Supplementary Material, Table 2). The Multiparameter Persistence Image clearly outperforms the other techniques on some data sets (such as ECG200, Plane, SwedishLeaf, MedicalImages), and remain very competitive on the others. It is also clearly faster to compute, since the Multiparameter Persistence Kernel requires computing several kernel matrices, which is costly and scales poorly, and Multiparameter Persistence Landscapes requires sorting the landscape values on every sample of every slice. One can see that multiparameter persistence summaries are clearly superior to the diagonal 1D-persistence summaries. We also confirmed that the differences in accuracies were not explained only by the resolutions of the final model selected by cross-validation in Supplementary Material, Table 3, and provided more details on cross-validation results in Supplementary Material, Table 4.

A second series of experiments focused on graph classification (see Supplementary Material, Section 4). In this case, we found that all of the multiparameter persistence summaries had essentially the same performance, and our conclusion is that the fibered barcodes alone already contain all the salient topological information. Note that the fibered barcodes remain again superior to 1D-persistence. Analysis of these images is usually done by looking at the nearest neighbor distance distributions, that is, for each cancer pixel (i.e., pixel whose brightness is above a manual threshold in the cancer image), compute the distance to its nearest immune pixel, and vice-versa. This gives two empirical distributions, whose means and variances can be used as features for classification [GMH + 18, SBSO16]. However, this approach is known to be unstable (due to the use of nearest distances), requires manual thesholding, and only represents the local neighborhoods around the cell without looking at relationships at larger spatial scales.

Since the two images are in correspondence, we can regard the brightness information as providing a pair of filter functions-this setup lends itself to analysis using multiparameter persistence. The classification task was to distinguish patients that were alive at the latest follow-up after diagnosis from those who passed away, based on the images. We used the same parameters as detailed in the previous experiments (since all 1D-persistence summaries led to the same results, we provided only one score, reported as "P"), and added a column "NN-F" for the performance of nearest neighbor distribution features fed to an XGBoost classifier. Results were averaged over 5 folds of the full 688 patients and are provided in Table 3. One can see that multiparameter persistence (and in particular the Multiparameter Persistence Image) provides a striking improvement in survival prediction over either the standard method or 1D-persistence.

NN-F P MP-K MP-L MP-I 67.2 ± 1.0 71.1 ± 1.1 77.6 ± 2.3 76.5 ± 2.7 79.1 ± 2.2 Table 3: Classification results for immunofluorescence data.

Stability. We conclude with an empirical study of the stability of the Multiparameter Persistence Image with respect to slice perturbations. We perturbed the endpoints of the slices for an arbitrary time series in ECG200 with random noise of increasing amplitude, and looked at the ratio between the 1-norm of the difference between the perturbed and clean, i.e., with no noise, Multiparameter Persistence Images, and the (normalized) noise amplitude. We also checked the accuracies on ECG200. It is clear from Figure 6 that the stability ratio looks constant and even starts to decrease when the noise amplitude is large enough, clearly showing that Multiparameter Persistence Images are empirically stable. This is also observed in the accuracy, which actually increases when noise amplitude is small before slowly decreasing as well. This also indicates that optimizing over the endpoints of the lines is another potential way of increasing the performance of models. In this article, we introduced the Multiparameter Persistence Image, a compact descriptor for multiparameter persistence which is easily computable, robust to noise, and can be used for machine learning and statistics. Experiments demonstrated that its refined discriminating capability enables it to match or outperform other methods on a variety of tasks, most notably predicting survival from images of cancerous tumors. In future work, we plan to study the relation between the multiparameter persistence module decomposition into interval modules (when it exists) and the one produced by the Vineyard Decomposition. Another next step is to extend our descriptor for multiparameter persistence to larger numbers of filter functions.

Broader Impact

Multiparameter persistence descriptors are sorely needed to enhance the reach and usability of topological data analysis, since many applications require understanding and encoding multiple filtrations at once. We believe that proposing an efficient descriptor that encodes strictly more information than just the union of all 1D-persistence diagrams associated to slices, and enabling the community to use it with an easy-to-use Python package, can have a significant impact and is an important contribution to data science and topological data analysis. In particular, we hope that our work helps to make topological methods within machine learning more accessible to practitioners.

Figure 2 :

 2 Figure 2: Examples of sets of lines used to define the Multiparameter Persistence Image.

Figure 3 :

 3 Figure 3: Example of Multiparameter Vineyard and associated Vineyard Decomposition.

Figure 4 :

 4 Figure 4: Single-parameter persistence fails at recovering the correct topology of a point cloud with outliers. From left to right: point cloud with outliers colored with density estimates; persistence diagram associated to the point cloud. Note the multiple blue points in dimension 1 (i.e., representing cycles) that are far away from the diagonal; Vineyard Decomposition computed with union of balls and density, the cycle is now detected as a set of long green bars; associated Multiparameter Persistence Image with resolution p = 200, bandwidth σ = 0.001 and power q = 5.

  Immunofluorescence images. Our second experiment deals with quantitative immunofluorescence images. The data is a set of pairs of images, where each pair consists of imaging data from a piece of human tissue from a patient suffering from breast cancer. The pixel intensities in an image denote the abundance of cells from one of two types: immune cells or cancer. That is, in one image cancer cells are bright, and in the other image immune cells are. These pixel intensities were obtained by injecting biomarkers in the tissue that would brighten the cells of the specified type. Moreover, the pixels of the two images are in correspondence since they represent the same location on the human piece of tissue. See [ACC + 20] for more details.

Figure 5 :

 5 Figure 5: From left to right: immune brightness, cancer brightness, Multiparameter Persistence Image in dimension 1, Multiparameter Persistence Landscape in dimension 1. Note how the bright spots of immune cells correspond to the dark spots of cancer cells.

Figure 6 :

 6 Figure 6: Empirical illustration of Multiparameter Persistence Image stability

Table 1 :

 1 Classification results for time series.

	Dataset	B1	B2	B3	PSS-K P-I	P-L MP-K MP-L MP-I
	DistalPhalanxOutlineAgeGroup	62.6 62.6	77.0	76.9	69.8 70.5	67.6	70.5	71.9
	DistalPhalanxOutlineCorrect	71.7 72.5	71.7	47.5	67.4 66.3	74.6	69.6	71.7
	DistalPhalanxTW	63.3 63.3	59.0	71.5	59.0 56.1	61.2	56.1	61.9
	ProximalPhalanxOutlineAgeGroup 78.5 78.5	80.5	75.9	82.0 78.0	78.0	78.5	81.0
	ProximalPhalanxOutlineCorrect	80.8 79.0	78.4	78.4	72.2 72.5	78.7	78.7	81.8
	ProximalPhalanxTW	70.7 75.6	75.6	61.4	72.2 73.7	79.5	73.2	76.1
	ECG200	88.0 88.0	77.0	67.0	74.0 74.0	77.0	74.0	83.0
	ItalyPowerDemand	95.5 95.5	95.0	-	64.7 61.1	80.7	78.6	79.8
	MedicalImages	68.4 74.7	73.7	51.1	46.2 44.3	55.4	55.7	60.0
	Plane	96.2 100.0 100.0	82.9	64.8 82.9	92.4	84.8	97.1
	SwedishLeaf	78.9 84.6	79.2	81.0	37.1 38.2	78.2	64.6	83.8
	GunPoint	91.3 91.3	90.7	90.6	84.7 80.0	88.7	94.0	90.7
	GunPointAgeSpan	89.9 96.5	91.8	-	84.5 87.0	93.0	85.1	90.5
	GunPointMaleVersusFemale	97.5 97.5	99.7	-	88.3 87.3	96.8	88.3	95.9
	GunPointOldVersusYoung	95.2 96.5	83.8	-	98.7 95.9	99.0	97.1	100.0
	PowerCons	93.3 92.2	87.8	-	83.4 76.7	85.6	84.4	86.7
	SyntheticControl	88.0 98.3	99.3	50.0	45.7 44.0	50.7	60.3	60.0
	Dataset			MP-K		MP-L	MP-I
	DistalPhalanxOutlineAgeGroup	9227.1	1038.9	217.1
	DistalPhalanxOutlineCorrect	36734.6 3492.6 833.7
	DistalPhalanxTW			9396.4		577.7	138.4
	ProximalPhalanxOutlineAgeGroup 11573.1		759.5	244.5
	ProximalPhalanxOutlineCorrect	30822.7 2169.5 497.6
	ProximalPhalanxTW		11641.7		375.4	93.4
	ECG200			1615.3	1355.6	269.0
	ItalyPowerDemand		41918.1 1939.0 417.5
	MedicalImages			147668.1 2404.7 599.5
	Plane			2036.0	1065.0	249.2
	SwedishLeaf			38045.7 3329.3	693.5
	GunPoint			1977.0	1685.7	422.1
	GunPointAgeSpan			14013.9 3945.6 1078.6
	GunPointMaleVersusFemale		14069.9 4058.8 1097.0
	GunPointOldVersusYoung		16668.1 5400.9 1388.5
	PowerCons			8808.3	3234.8	811.4
	SyntheticControl		13340.0		595.1	161.9

Table 2 :

 2 Computation time (s) for time series in dimension 0.

We also tried the other sets of lines in Definition 3.1 for the Multiparameter Persistence Kernel and Image, but did not report the results since changes were not significant.

In a few tasks where the PSS-K values are infeasible to compute, we did not report accuracy.

Acknowledgements and Funding Disclosure

Part of this work was done while the first author was working at Columbia University and funded by the United States National Institutes of Health (NIH) grant T15LM007079-28. The second author was partially funded by NIH grants 5U54CA193313 and GG010211-R01-HIV, AFOSR grant FA9550-18-1-0415, and NSF grant CNS-1514422. The authors would like to thank Mike Lesnick for helpful discussion of the algorithm and in particular the issue of Vineyard stability. The authors would also like to acknowledge Kevin Gardner's lab at Columbia University for kindly sharing their immunofluorescence images.