Séverine Affeldt

Lazhar Labiod

Mohamed Nadif

CAEclust: A consensus of autoencoders representations for clustering

Keywords: spectral clustering, ensemble learning, deep autoencoder

The CAEclust Python package implements an original deep spectral clustering in an ensemble framework. Recently, strategies combining classical clustering approaches and deep autoencoders have been proposed, but their effectiveness is impeded by deep network hyperparameters settings. We alleviate this issue with a consensus solution that hinges on the fusion of multiple deep autoencoder representations and spectral clustering. CAEclust offers an efficient merging of encodings by using the landmarks strategy and demonstrates its effectiveness on benchmark data. CAEclust enables to reproduce our experiments and explore novel datasets.

Introduction and related theory

In the context of unsupervised learning, recent studies improved classical clustering approaches by relying on deep networks for dimensionality reduction, as such networks can automatically learn important features from data [START_REF] Banijamali | Fast spectral clustering using autoencoders and landmarks[END_REF][START_REF] Leyli-Abadi | Denoising autoencoder as an effective dimensionality reduction and clustering of text data[END_REF][START_REF] Tian | Deepcluster: A general clustering framework based on deep learning[END_REF][START_REF] Yang | Modularity based community detection with deep learning[END_REF]. In particular, deep autoencoders (DAE) propose a new data representation or encoding using an unsupervised learning. Yet, the effectiveness and ease-touse of existing DAE clustering strategies are impeded by their sensitivity to hyperparameters. First, network weights initialization adds randomness to the results. Pretraining helps to mitigate this issue, but is computationally intensive. Finding the correct deep architecture is another hurdle. In almost all recent papers on deep clustering, architectures are fine-tuned for each benchmark dataset and lack of technical rationales. Specifically, the architecture is usually derived by cross-validation on a validation set for a grid of hyperparameters [START_REF] Yang | Modularity based community detection with deep learning[END_REF][START_REF]Fast spectral clustering using autoencoders and landmarks[END_REF][START_REF] Ji | Deep subspace clustering networks[END_REF]. Such determination is infeasible in practice for unsupervised clustering. Other studies reuse architectures that were historically proposed for the considered dataset, without any technical explanations. As an example, the DAE architecture used in [START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF][START_REF] Guo | Improved deep embedded clustering with local structure preservation[END_REF][START_REF] Yang | Towards k-meansfriendly spaces: Simultaneous deep learning and clustering[END_REF][START_REF] Moradi Fard | Deep k-means: Jointly clustering with k-means and learning representations[END_REF], which contains an encoder having 3 fully connected layers with 500-500-2000 units and an encoding size of 10, is taken from [START_REF] Van Der Maaten | Learning a parametric embedding by preserving local structure[END_REF], which is inspired by [START_REF] Salakhutdinov | Learning a nonlinear embedding by preserving class neighbourhood structure[END_REF] and [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF].

The CAEclust package relies on our recent work [START_REF] Affeldt | Spectral clustering via ensemble deep autoencoder learning (SC-EDAE)[END_REF] and practically addresses the above mentioned issues with a consensus optimization that reaches better predictive performance and more relevant clusterings than single data representation strategies. CAEclust embeds the landmarks strategy that strongly diminishes the computational complexity and requires no pretraining. It also fills the gap between ensemble deep autoencoders and spectral clustering in order to propose an effective approach that takes simultaneously advantage of several deep models with various hyperparameters settings. Specifically, we apply spectral clustering on an ensemble of fused encodings obtained from m different deep autoencoders.

Spectral clustering

A number of spectral clustering algorithms [START_REF] Filippone | A survey of kernel and spectral methods for clustering[END_REF][START_REF] Verma | A comparison of spectral clustering algorithms[END_REF], that propose different ways to use the eigenvectors of the normalized graph laplacian matrix [START_REF] Shi | Normalized cuts and image segmentation[END_REF] or of the walk's transition matrix [START_REF] Meila | Learning segmentation by random walks[END_REF], can be found in the literature. Their objective function typically favors low similarity between clusters and high similarity within clusters to partition n datapoints of X ∈ R n×d into k disjoint clusters. The spectral clustering algorithm, in its normalized version, use the top k eigenvectors of the normalized graph Laplacian matrix. They are the relaxations of the indicator vectors which provide assignments of each datapoint to a cluster. Hence, it amounts to maximize the following relaxed normalized association, max

B∈R n×k T r(B ⊤ SB) s.t. B ⊤ B = I (1)
with S = D -1/2 KD -1/2 ∈ R n×n is the normalized similarity matrix where K ∈ R n×n is the similarity matrix and D ∈ R n×n is the diagonal matrix whose (i, i)-element of X is the sum of X's i-th row.

The solution of ( 1) is to set the matrix B ∈ R n×k equal to the k eigenvectors corresponding to the largest k eigenvalues of S. After renormalization of each row of B, a k-means assigns each datapoint x i of X to the cluster that the row b i of B is assigned to. An advantage of the spectral clustering is that it performs well on arbitrary shaped clusters, by contrast with several other clustering algorithms (e.g. k-means). However, this approach has also difficulties to handle large-scale datasets. This is the consequence of the high complexity of the graph Laplacian construction and the eigendecomposition.

Recently, the Landmark-based Spectral Clustering (LSC ) [START_REF] Chen | Large scale spectral clustering with landmark-based representation[END_REF] (or AnchorGraph [START_REF] Liu | Large graph construction for scalable semisupervised learning[END_REF]) strategy offered a scalable spectral clustering. In particular, LSC allows to efficiently construct the graph Laplacian and compute the eigendecomposition. Specifically, each datapoint is represented by a linear combination of p representative datapoints (or landmarks), with p ≪ n. The obtained representation matrix Ẑ ∈ R p×n , for which the affinity is calculated between n datapoints and the p landmarks, is sparse. This, in turn, ensures a more efficient eigendecomposition as compared to the above mentioned eigendecomposition of S (Eq. 1).

Deep autoencoders

An autoencoder [START_REF] Hinton | Autoencoders, minimum description length and Helmholtz free energy[END_REF] is a neural network that implements an unsupervised learning algorithm in which the parameters are learned in such a way that the output values tend to copy the input training sample. The internal hidden layer of an autoencoder can be used to represent the input in a lower dimensional space by capturing the most salient features.

We can decompose an autoencoder in two parts, namely an encoder f θ , followed by a decoder g ψ . The first part provides the encoding Y of the input dataset by computing a feature vector y i = f θ (x i ) for each input training sample. Then, the encoding is transformed back to its original representation by the decoder part, following xi = g ψ (y i ).

The sets of parameters for the encoder f θ and the decoder g ψ are learned simultaneously during the reconstruction task while minimizing the loss, referred to as J given by

J AE (θ, ψ) = n i=1 L(x i , g ψ (f θ (x i ))), ( 2
)
where L is a cost function for measuring the divergence between the input training sample and the reconstructed data. The encoder and decoder parts can have several shallow layers, yielding a deep autoencoder (DAE) that enables to learn higher order features. The network architecture of these two parts usually mirrors each other.

2 CAEclust algorithm

Algorithm overview

We first obtain a set of m encodings {Y ℓ } ℓ∈ [1,m] , given an n × d data matrix X, using m DAE trained with different hyperparameters settings. Then, for each embedding Y ℓ , we construct a graph matrix S ℓ . We fuse the m graph matrices in an ensemble graph matrix S (equivalently named affinity or similarity matrix in the following) which contains information provided by the m embeddings. Finally, the spectral clustering applied to S benefits from the common subspace shared by the m deep embeddings. Hence, our proposal hinges on three challenges, namely (i) generating m deep embeddings, (ii) integrating the clustering in an ensemble learning framework and (iii) solving the clustering task in a highly efficient way. The steps of CAEclust are summarized in Algorithm 1 and illustrated by Figure 1.

Constructing graph matrices from embeddings

An autoencoder is composed of an encoder f θ and a decoder g ψ that can have multiple layers of different widths. Its cost function, given by Eq.3, measures the error between the input x ∈ R d×1 and its reconstruction at the output x ∈ R d×1 . We can learn m encodings {Y ℓ } ℓ∈ [1,m] by training DAE with different hyperparameter settings, g ψ ℓ and f θ ℓ , and Y ℓ = f θ ℓ (X) (Fig. 1, (a)).

||X -

g ψ ℓ f θ ℓ (X) || 2 . (3)
The Landmark Spectral Clustering [START_REF] Chen | Large scale spectral clustering with landmark-based representation[END_REF] and the Anchor-Graphs [START_REF] Liu | Large graph construction for scalable semisupervised learning[END_REF] strategies compute a smaller and sparser representation matrix Z ℓ ∈ R n×p that approximates a full n × n affinity matrix. This matrix is built between the landmarks {u ℓ j } j∈ [1,p] and the encoded points {y ℓ i } i∈ [1,n] (Fig. 1,(a)). CAEclust proposes to construct the graph matrix S ℓ following this idea. First, we obtain a set of p points (p ≪ n), that are the landmarks which approximate the neighborhood structure, through k-means applied on the embedding matrix Y ℓ . Then, a non-linear mapping from data to landmark is computed as follows,

z ℓ ij = Φ(y ℓ i ) = K(y ℓ i , u ℓ j ) j ′ ∈N (i) K(y ℓ i , u ℓ j ′ ) ; j ′ ∈ N (i) (4)
where N (i) indicates the r (r < p) nearest landmarks around y ℓ i . As proposed in [START_REF] Chen | Large scale spectral clustering with landmark-based representation[END_REF], we set z ℓ ij to zero when the landmark u ℓ j is not among the nearest neighbor of y ℓ i , leading to a sparse affinity matrix Z ℓ . The function K(.) is used to measure the similarity between data y ℓ i and anchor u ℓ j with L 2 distance in Gaussian kernel space K(x i , x j ) = exp(-||x i -x j || 2 /2σ 2 ), and σ is the bandwidth parameter. The normalized matrix Ẑℓ ∈ R n×p is then utilized to obtain a low-rank graph matrix,

S ℓ ∈ R n×n , S ℓ = Z ℓ Σ -1 Z ⊤ ℓ where Σ = diag(Z ⊤ ℓ 1).
As the Σ -1 normalizes the constructed matrix, S ℓ is bi-stochastic, i.e. the summation of each column and row equal to one, and the graph Laplacian becomes,

S ℓ = Ẑℓ Ẑ⊤ ℓ where Ẑℓ = Z ℓ Σ -1/2 . ( 5
)

Algorithmic optimization on the ensemble of affinity

CAEclust merges m graph similarity matrices, S ℓ , in an ensemble similarity matrix, S. This matrix contains the information provided by the encodings {Y ℓ } ℓ∈ [1,m] obtained using m DAE trained with different hyperparameters setting ℓ. Hence, the CAEclust ensemble affinity matrix is built as the summation of the m basic similarity matrices as given by Eq.6. Such aggregation follows an Ensemble Clustering idea analogous to that proposed in [START_REF] Strehl | Cluster ensembles -a knowledge reuse framework for combining multiple partitions[END_REF][START_REF] Vega-Pons | A survey of clustering ensemble algorithms[END_REF] where a co-association matrix is first built as the summation of all basic similarity matrices, and where each basic partition matrix can be represented as a block diagonal matrix,

S = 1 m m ℓ=1 S ℓ . (6)
The S matrix is approximately block stochastic for many natural problems. Hence, the first k eigenvectors of S are approximately piecewise constant over the k almost invariant rows subsets [START_REF] Meila | A random walks view of spectral segmentation[END_REF]. In the following, we show how CAEclust computes, at lower cost, B that is shared by the m graph matrices S ℓ . The B matrix is obtained by optimizing the trace maximization problem given by max

B T r(B ⊤ SB) s.t. B ⊤ B = I. (7)
We can solve Eq. 7 by setting B equal to the k eigenvectors corresponding to the largest k eigenvalues of S. However, the computation of the eigen-decomposition of S of size (n×n) is O(n 3 ). By contrast, CAEclust computes instead the k left singular vectors of the concatenated matrix,

Z = 1 √ m [ Ẑ1 | . . . | Ẑj | . . . | Ẑm ]. (8)
We naturally obtain an improvement in the computational cost of B by using the sparse matrix Z ∈ R n× m j=1 ℓ j with m j=1 ℓ j ≪ n, instead of S, which has a larger dimension (Fig. 1, (b)).

Proposition 2.1. Given a set of m similarity matrices S ℓ , such that each matrix S ℓ can be expressed as

Z ℓ Z ⊤ ℓ . Let Z ∈ R n× m j=1 ℓ j , where m j=1 ℓ j ≪ n, denoted as 1 √ m [Z 1 | . . . |Z j | . . . |Z m ],
be the concatenation of the Z ℓ 's, ℓ = 1, . . . , m. We first have, max

B ⊤ B=I T r(B ⊤ SB) ⇔ min B ⊤ B=I,M || Z -BM ⊤ || 2 F . (9)
Then, given SVD( Z), Z = UΣV ⊤ and the optimal solution B * is equal to U.

Proof. From the second term of Eq. 9, one can easily show that M * = Z⊤ B. Plugging now the expression of M * in Eq. 9, the following equivalences hold min

B ⊤ B=I,M || Z -BM ⊤ || 2 F ⇔ min B ⊤ B=I || Z -BB ⊤ Z|| 2 F ⇔ max B ⊤ B=I T r(B ⊤ ZZ ⊤ B) ⇔ max B ⊤ B=I T r(B ⊤ SB).
On the other hand, SVD( Z) leads to Z = UΣV ⊤ (with U ⊤ U = I, V ⊤ V = I) and therefore to the eigen-decomposition of S as follows:

S = ZZ ⊤ = (UΣV ⊤ )(UΣV ⊤ ) ⊤ = UΣ(V ⊤ V)ΣU ⊤ = UΣ 2 U ⊤ .
Thereby the left singular vectors of Z are the same as the eigenvectors of S.

With CAEclust, we propose a unique way to combine DAE encodings for clustering. Our method also benefits from the low complexity of the anchors strategy for both the graph affinity matrix construction and the eigen-decomposition. Specifically, the computational cost for the construction of each Z ℓ affinity matrix amounts to O(np ℓ e(t + 1)) (Alg. 1, step (b)) , where n is the number of datapoints, p ℓ is the number of landmarks for the ℓ th DAE (p ℓ ≪ n), e is the size of the DAE encoding Y ℓ (e ≪ n) and t is the number of iterations for the k-means algorithm that is used to select the landmarks. It is worth noting that CAEclust proposes a parallelized computation of the Z ℓ matrices over multiple cores. This limits the computation time of the ensemble affinity matrix Z to the most time consuming Z ℓ . Furthermore, the eigen-decomposition of Z, which leads to the B embeddings (Alg. 1, step (c)), induces a computational complexity of O(p ′3 + p ′ 2n), where p ′ is the sum of all landmarks numbers for the concatenated Z ℓ matrices, i.e. p ′ = m j=1 ℓ j ≪ n. The last k-means on B ∈ R n×k (Alg. 1 requires additional O(nctk)), where c is the number of centroïds, usually equal to k the number of eigenvectors, leading to O(ntk 2 ).

The originality and efficiency of the CAEclust ensemble method hinges on the replacement of a costly eigen-decomposition on S ∈ R n×n by an eigen-decomposition on a low-dimensional and sparse matrix Z ∈ R n× m j=1 ℓ j , with m j=1 ℓ j ≪ n (Alg. 1, step (c)). In particular, the sparsity of Z enables the use of fast iterative and partial eigenvalue decomposition.

3 CAEclust package: Implementation and Evaluation

Software architecture and functionalities

We summarize the package usage in Figure 2 and detail below CAEclust functionalities. [io] dataLoader.py loads a dataset. CAEclust proposes by default two well-known benchmark images datasets, namely USPS and MNIST. dataLoader.py is needed at line 1 of Algorithm 1.

[daeNet] daeEncoding.py sets a DAE with general parameters (e.g. optimizer function, number of epochs, batch size, encoding dimension), and any layer number and width. The method deep ae() generates the encoded data. The package proposes to generate the encodings either in serial (serial encodings.py) or in parallel (parallel encodings.py). daeEncoding.py is needed at lines 2 and 4 of Algorithm 1.

[clustering] ensClust.py computes the ensemble sparse affinity matrix (Eq. 4) through the method ensemble approximate affinity() (Algorithm 1, line 5). The deep consensus clustering is then obtained from the shared space B (Eq. 8, Eq. 7 & Proposition 2.1) using the sparse SVD() method (Algorithm 1, line 6).

[evaluation] evaluation.py provides 2D and 3D visualizations of the shared space B with commonSpace plot(). These visualizations are also proposed with the UMAP [START_REF] Mcinnes | Umap: Uniform manifold approximation and projection for dimension reduction[END_REF] transformation. When studying benchmark datasets with ground truth labels, allMetrics() function can compute three informative metrics (accuracy, NMI [START_REF] Strehl | Cluster ensembles -a knowledge reuse framework for combining multiple partitions[END_REF], and ARI [START_REF] Steinley | Properties of the Hubert-Arable Adjusted Rand Index[END_REF]). Furthermore, compare metrics() provides an html summary table to easily compare the CAEclust results with other clustering algorithms. ⇝ sparse SVD()

Relating functionalities and algorithm

Evaluations

The CAEclust ensemble package is evaluated on USPS and MNIST. The sets of encodings are made with different DAE initializations (Init.), epoch numbers (Ep.) or structures (Struct.). We trained fully connected autoencoders with an encoder f θ of three hidden layers of size 500, 750 or 1000 for MNIST and USPS, as suggested by Bengio et al. [START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF], in all possible orders. The decoder part g ψ mirrors the encoder stage f θ . The encoding dimension e corresponds to the number of expected classes, which is 10. We consider landmarks number within 100 and 1, 000 by step of 100. The accuracy reported for DAE-LSC corresponds to an average over 50 replicates (10 replicates on each of the 5 encodings per DAE structure), over all epoch and landmark numbers.

The simple use of LSC on encoded data already improves the LSC performance. Indeed, we found with LSC an average accuracy of 68.55 ± 2.25 and 77.20 ± 1.49 on the original MNIST and USPS datasets respectively, while the use of a single encoding enables to reach an average accuracy between 87.06 ± 8.27 and 91.54 ± 3.06 for MNIST ([500-750-1000] and [1000-500-750] DAE hidden layers) and between 79.72 ± 6.21 and 83.47 ± 7.40 for USPS ([750-500-1000] and [500-1000-750] DAE hidden layers). Yet, finding a priori the most appropriate DAE structure remains a challenging task. The accuracy may also vary for different landmark and epoch numbers. The ensemble strategy provided by CAEclust offers a straightforward way to alleviate these issues and avoid the fine tuning of the DAE hyperparameters. As can be seen from Table 2, CAEclust provides higher The CAEclust ensemble approach on the DAE structures (Struct., m = 6, all studied structures) enables also to reach high accuracy as compared to the single combination. For MNIST, it outperforms all the single encoding combination (93.23 ± 0.28 vs. 91.54 ± 3.06). For USPS, it reaches the third best single encoding combination with a better standard deviation (81.78 ± 3.61 vs. 81.78 ± 8.08). These good results are obtained with the added benefit of avoiding the training of a particular DAE structure, which is not possible in an unsupervised context. Figure 3 gives an overview of the common shared space B for diverse ensemble of encodings and demonstrates the good capacity of B to separate the classes.

Among the existing deep clustering methods, two approaches can now be considered as state-ofthe-art methods, namely IDEC (Improved Deep Embedded Clustering) [START_REF] Guo | Improved deep embedded clustering with local structure preservation[END_REF] and DCN (Deep Clustering Network) [START_REF] Yang | Towards k-meansfriendly spaces: Simultaneous deep learning and clustering[END_REF]. Recently, the DKM (Deep k-means) algorithm, which applies a k-means in an AE embedding space, outperformed these approaches [START_REF] Moradi Fard | Deep k-means: Jointly clustering with k-means and learning representations[END_REF]. While our CAEclust approach does not require any pretraining for the initialization of the DAE weights, it outperforms the DCN, IDEC and DKM methods in their pretrained version (Table 2). In particular, DCN treats DAE adjacent layers as restricted Boltzmann machines (RBM) and trains RBMs bottom-up to obtain good initial weights. Then, the network weights are fine-tuned. IDEC uses a greedy layer-wise pre-training with dropout before fine-tuning the entire network without dropout while DKM also relies on a pretraining step of 50 epochs before applying its main algorithm. By contrast, CAEclust only trains the DAEs without any weight initialization that could be obtained from a pretraining strategy. As can be seen from the Table 2, none of these challenging deep clustering approaches outperform CAEclust in their no pretraining version (see Table 2, Deep kmeans variants / no pret. column). Besides, even when injecting pretraining in DCN, IDEC or DKM, CAEclust still remains really competitive, as its lowest accuracy mainly outperforms the accuracy of the other approaches. Most importantly, the CAEclust ensemble based on the merging of encodings produced by various structures (Table 2, Struct. column) provides a straightforward way to combine embeddings from various DAEs without having to choose a specific architecture ( Beyond the DAEs hyperparameters, CAEclust also depends on the number of landmarks that are used for the LSC clustering. As can be seen in [START_REF]Large graph construction for scalable semi-supervised learning[END_REF][START_REF] Banijamali | Fast spectral clustering using autoencoders and landmarks[END_REF], the anchor graph strategy reduces the error rate as the number of landmarks increases. As an example, the experiments from [START_REF]Large graph construction for scalable semi-supervised learning[END_REF] demonstrate that the error rate of such an approach reaches a plateau above 800 landmarks on the USPS dataset. In [START_REF] Banijamali | Fast spectral clustering using autoencoders and landmarks[END_REF], experimental results show that the higher the number of landmarks, the better results for MNIST. Our experiments tend to confirm these behaviors, as can be seen in Figure 4, which provides the accuracy of CAEclust on MNIST and USPS, for different DAE architectures trained during 50 epochs.

As complementary experiments, we investigate an ensemble strategy of CAEclust on the number of landmarks. More specifically, for a chosen DAE architecture, we construct an ensemble sparse affinity matrix, Z, where each Ẑi is based on a different number of anchors (Alg. 1, step (b); Eq. 8).

As an example, we randomly choose m = 5 number of landmarks and evaluate the CAEclust accuracy at several numbers of epochs. As can be seen from Table 3, this landmark variant of CAEclust provides again better accuracy than the other approaches.

Illustrative Examples

CAEclust includes two tutorial notebooks. The first one, Baseline evaluations, shows the influence of the DAE structure on the clustering for a non ensemble approach. The second one, Ensemble evaluations, demonstrates the effectiveness CAEclust consensus deep clustering.

Baseline evaluations

The notebook Baseline evaluations first loads the dataset with dataLoader.py. Then, it performs k-means and LSC clusterings on the original data. For LSC, the required CAEclust functions are single approximate affinity() followed by sparse SVD(). We then set several DAEs with different structures to encode data using daeEncoding.py. Finally, we perform k-means and LSC on each encoding, evaluate the results with allMetrics() and display them using compare metrics().

Ensemble evaluations

The notebook Ensemble evaluations first loads the dataset with dataLoader.py. Then, it encodes the data with various hyperparameters using daeEncoding.py (either in serial or in parallel).

To compute the ensemble sparse affinity matrix Z, we use ensemble approximate affinity().

The clustering is obtained from the common shared space using sparse SVD(). We can compare the different ensemble strategies based on the metrics of allMetrics() and display them using compare metrics().

Figure 7 2 provides the evaluations of different CAEclust ensemble strategies on MNIST and USPS. As can be seen, all the ensemble strategies (initialization, epoch and structure) enable to reach high accuracy, NMI and ARI.

In particular, the ensemble structure on MNIST (Figure 7, a; Ens.Struct.) reaches much better results than the results obtained with the least efficient structure (Ens.Struct., 0.9082 vs. DAE[500, 750, 1000]+ The Ensemble evaluations notebook also proposes a meta ensemble strategy, for which the encodings from the initialization, epoch and structure ensemble sets are integrated in the common shared space B. For both USPS and MNIST, the CAEclust meta ensemble strategy outperforms the other ensemble strategies as well as their best encoding performance. The Figure 8 provides the UMAP transformation of the common share space B and highlights the good capacity of B to separate the classes. In practice, one could generate various encodings following a predefined grid and take advantage of all the deep representations to obtain a efficient clustering with very few hyperparameters settings constraints.

Conclusions

We developed a Python package CAEclust for deep spectral clustering in an ensemble framework. CAEclust is capable of learning a consensus solution that hinges on the fusion of multiple deep autoencoder representations and spectral clustering. It is an effective and user friendly package that proposes parallelization to overcome the computation time issue. Future extensions of this work will be dedicated to the challenging text data, which is by nature unstructured, highly dimensional and sparse. All in all, CAEclust alleviates the hyperparameters settings issue that usually impedes the efficiency of deep network based clustering techniques, and offers an efficient merging of encodings by using the landmarks strategy.

Figure 1 :

 1 Figure 1: CAEclust core algorithm overview. CAEclust algorithm first computes m encodings from DAE with different hyperparameters settings (a), then generates m sparse affinity matrix, { Ẑℓ } ℓ∈[1,m] , that are concatenated in Z (b), and finally performs a SVD on the ensemble graph affinity matrix Z (c).

Figure 2 :

 2 Figure 2: CAEclust package.

Algorithm 1 1 . 1 :

 111 provides the CAEclust pseudo-code. Each step indicates the associated Python script. The corresponding functions are detailed at Section 3.Algorithm CAEclust core algorithm 1: Input: data matrix X ⇝ dataLoader.py 2: Initialize: m DAE with different hyperparameters settings 3: Do: 4: (a) Generate m deep embedding {Y ℓ } l∈[1,m] (Eq. 3) ⇝ daeEncoding.py [serial encodings.py | parallel encodings.py] 5: (b) Construct the ensemble sparse affinity matrix Z ∈ R n× m j=1 ℓ j (Eq. 4, 8)⇝ ensemble approximate affinity() 6: (c) Compute B * ∈ R n×k by performing sparse SVD on Z (Eq. 9)

Figure 3 :

 3 Figure 3: 2D Visualization of the embeddings B from CAEclust on USPS for an ensemble of encodings based on diverse initializations (Ens. Init.), epochs (Ens. Epoch) and structures (Ens. Struct.). Colors indicate the true USPS digits labels.

Figure 4 :

 4 Figure 4: Mean clustering accuracy per number of landmarks for different DAE architectures

Figure 5 1

 5 Figure51 provides the evaluation metrics for k-means and LSC clusterings on single deep representations obtained with various autoencoders structures. As can be seen, the use of a single encoding usually improves the clustering results, both for k-means (Figure5, a & b) and LSC (Figure5, c & d). However, the results strongly depend on the DAE structure, as exemplified by Figure6which provides the UMAP representations for the most efficient deep encoding (left) and the least efficient deep encoding (right) for MNIST. In the second notebook, CAEclust proposes several ensemble solutions to alleviate such critical hyperparameters influence.

Figure 5 :Figure 6 :

 56 Figure 5: Clustering on a single deep representation with CAEclust package

  (a) MNIST -Consensus results (b) USPS -Consensus results

Figure 7 :

 7 Figure 7: CAEclust ensemble evaluations on MNIST (left) and USPS (right)

Figure 8 :

 8 Figure 8: 2D UMAP Visualization of the embeddings B from CAEclust on MNIST for an ensemble of encodings based on diverse initializations (Ens. Init.), epochs (Ens. Epoch) and structures (Ens. Struct.). Colors indicate the true digits labels.

7 :

 7 Output: Run k-means on B * to get the final clusteringCAEclust package metadata are summurized by Table1.

Table 1 :

 1 CAEclust package metadata

	Code metadata description	metadata
	Current code version	V1.0.0
	Legal Code License	GPL-3.0 License
	Software code languages, tools, and ser-	Python (= 3.8)
	vices used
	Compilation requirements, operating	Python (= 3.8); packages: scikit-learn, mat-
	environments & dependencies	plotlib, pandas, ipykernel, jinja2, umap-learn,
		tensorflow (2.6 Windows, otherwise 2.7)
	Support email for questions	severine.affeldt@u-paris.fr

Table 2 :

 2 Mean clustering accuracy comparisons (pret. denotes pretraining).

				CAEclust			Deep k-means Variants
	Data	structure	DAE-LSC	no pretraining			no pret.			pret.
				Init.	Ep.	Struct. DCN IDEC DKM DCN IDEC DKM
		500-750-1000	87.06 ±8.27	89.19 ±0.41	85.54 ±4.30
	MNIST	500-1000-750 750-500-1000 750-1000-500 1000-500-750	90.48 ±5.20 88.31 ±5.46 90.30 ±4.89 91.54 ±3.06	95.33 ±0.07 92.15 ±0.25 92.65 ±0.13 94.28 ±0.20	94.34 ±2.68 92.03 ±3.87 92.26 ±3.71 94.57 ±1.48	93.23 ± 2.84	34.8 ± 3.0	61.8 ± 3.0	82.3 ± 3.2	81.1 ± 1.9	85.7 ± 2.4	84.0 ± 2.2
		1000-750-500	90.96 ±3.98	93.87 ±0.38	95.25 ±0.59
		500-750-1000	81.78 ±8.08	80.07 ±1.95	81.36 ±5.09
	USPS	500-1000-750 750-500-1000 750-1000-500 1000-500-750	83.47 ±7.40 79.72 ±6.21 80.29 ±5.70 81.39 ±4.46	80.54 ±0.77 79.49 ±1.19 79.29 ±1.05 84.12 ±1.80	82.06 ±3.54 81.10 ±3.86 79.88 ±2.69 81.89 ±3.21	81.78 ± 3.61	36.4 ± 3.5	53.9 ± 5.1	75.5 ± 6.8	73.0 ± 0.8	75.2 ± 0.5	75.7 ± 1.3
		1000-750-500	83.08 ±5.64	85.22 ±2.14	84.96 ±3.29

clustering accuracy as compared to the use of a single encoding. In particular, the mean accuracy values obtained with the initialization ensemble strategy (Init., m = 6) can reach, 95.33 ± 0.07 for MNIST and 85.22 ± 2.14 for USPS. Results for the epoch ensemble strategy (Ep., m = 5, epochs within [50; 250]) reach 95.25 ± 0.59 for MNIST and 84.96 ± 3.29 for USPS.

Table 2 ,

 2 Init. & Ep. columns). With the Struct. ensemble, CAEclust also offers a better accuracy than other approaches.

	MNIST, 50 epochs	USPS, 50 epochs

Table 3 :

 3 Mean clustering accuracy comparisons (landmarks ensemble).

	Data	structure	50 epochs	100 epochs 150 epochs 200 epochs 250 epochs
		500-750-1000	85.44 ±1.07	95.14 ±0.35	85.43 ±3.82	95.37 ±0.24	82.81 ±0.62
		500-1000-750	94.52 ±0.18	96.04 ±0.26	95.62 ±0.19	95.33 ±0.23	95.24 ±0.14
	MNIST	750-500-1000	92.80 ±2.07	85.74 ±2.88	94.90 ±0.21	94.86 ±0.93	94.09 ±0.26
		750-1000-500	89.19 ±1.23	88.90 ±0.65	94.77 ±0.07	93.80 ±1.61	95.99 ±0.25
		1000-500-750	89.31 ±2.76	93.12 ±2.50	95.54 ±0.14	95.17 ±0.17	95.65 ±0.15
		1000-750-500	94.09 ±0.22	95.64 ±0.13	94.90 ±0.28	95.57 ±0.10	95.19 ±0.11
		500-750-1000	79.57 ±4.14	78.35 ±0.78	78.32 ±0.31	90.17 ±6.62	84.42 ±7.10
		500-1000-750	79.18 ±0.31	85.07 ±5.34	78.92 ±0.53	86.89 ±7.24	79.72 ±0.27
	USPS	750-500-1000	77.50 ±0.32	81.27 ±4.11	82.11 ±4.57	80.82 ±4.18	81.36 ±4.13
		750-1000-500	82.40 ±5.00	80.25 ±0.31	80.08 ±0.34	80.01 ±0.23	78.78 ±0.40
		1000-500-750	94.34 ±0.35	79.60 ±0.18	80.16 ±0.16	91.39 ±5.75	86.28 ±6.67
		1000-750-500	92.59 ±4.27	80.09 ±0.17	86.65 ±7.01	89.52 6.39±	87.73 ±6.42

Séverine Affeldt, Lazhar Labiod, Mohamed Nadif

Tables are images extracted from the outputs of the CAEclust Baseline evaluations tutorial notebooks.

Tables are images extracted from the outputs of the CAEclust Ensemble evaluations tutorial notebooks.

Acknowledgments

We thank Amine Ferdjaoui (Université Paris Cité, Centre Borelli UMR9010, Paris, France) for his valuable technical assistance in the design and implementation of the CAEclust IPOL demo. This work was supported by a grant overseen by the French National Research Agency (ANR) (ANR-19-CE23-0002). It also received the labelling of Cap Digital and EuroBiomed/Cancer-Bio-Santé competitiveness clusters.


Francesca Di Fazio

Francesca Di

Fazio Concezione

Heiner Müller L'invito

Heiner Müller

Marta Cuscunà

Johanny Bert

La Regista

Si Veda Didier

Keywords: intesa in nel teatro di figura, teatrali, e, di conseguenza, interrogazioni estetiche e drammaturgiche

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Concezione drammaturgica e materialità della scena. La processualità trasversale nelle evoluzioni contemporanee del teatro di figura

attraverso strutturati metodi di lavoro in dialogo con altri artisti (scenografi, dramaturg, autori e performer). Attraverso l'analisi di alcuni processi creativi, intendo mostrare come la figura del marionettista si posizioni nella creazione contemporanea quale "mente teatrale" capace di attivare connessioni plurali, necessarie alla realizzazione di spettacoli dai valori segnici stratificati, e come da tale lavoro sincronizzato spicchi una profonda attenzione drammaturgica.

Marta Cuscunà. L'autrice-regista-performer, l'aiuto regista e la scenografa La collaborazione di Marta Cuscunà con Marco Rogante, il suo aiuto-regista, è cominciata nel 2009, per lo spettacolo È bello vivere liberi!. In tale occasione, Cuscunà aveva bisogno di uno sguardo esterno e di una figura tecnica che curasse i suoni e le luci, essendo lei contemporaneamente autrice del testo, performer e regista. Il ruolo di Rogante si è poi evoluto nel tempo, diventando sempre più versatile e avvicinandosi alla pratica del dramaturg, nel senso tedesco della parola. All'inizio di ogni creazione, Marta Cuscunà condivide le ricerche tematiche con Marco Rogante, in seguito gli sottopone le prime bozze del testo, in modo tale che lui ricerchi i passaggi funzionanti e quelli che invece necessitano di modifiche. Cito dalle parole di Rogante: «Il punto centrale del mio apporto drammaturgico sta nell'aiutare Marta a scegliere i percorsi che funzionano meglio, senza portare il lavoro dove lo porterei io, ma cercando di capire dove lo porterebbe lei»4 . Un simile processo avviene durante le prove, in cui Rogante applica lo stesso principio di accompagnamento drammaturgico con chi si occupa del suono e delle luci: «non si tratta solo di sapere che cosa fanno, ma di cercare di incanalarli verso direzioni che siano funzionali all'ergonomia generale dello spettacolo». Questa pratica è stata sviluppata a livello generale all'interno del team che lavora con Marta Cuscunà, in cui ognuno ha una o più mansioni specifiche ma rimane in costante scambio comunicativo con gli altri. Di conseguenza, tutti gli elementi dello spettacolo -i pupazzi, la loro manipolazione, la scenografia, i suoni, le luci e il testo -sono legati gli uni agli altri.

La centralità di tale condivisione si è manifestata maggiormente dall'inizio della collaborazione con la scenografa Paola Villani, con cui Cuscunà e Rogante hanno realizzato finora tre spettacoli (Sorry, boys, 2015; Il canto della caduta, 2018; Earthbound, 2021). A partire da un'idea visiva di Cuscunà per i pupazzi, Paola Villani li realizza in un lungo lasso di tempo che prevede continui rimandi con Cuscunà e Rogante, il cui contributo non è solo di tipo drammaturgico ma sconfina spesso in aiuto tecnico nella costruzione dei prototipi dei pupazzi. L'azione concertata è indispensabile sia a questioni più tecniche, quali la praticità di utilizzo dei pupazzi, sia a questioni drammaturgiche: le figure devono essere funzionali all'espressione del senso veicolato dal testo.

Questa coesione è evidente nella messa a punto del dispositivo scenico per Sorry, boys (2015), spettacolo che ricostruisce il fatto di cronaca avvenuto a Gloucester (Massachusetts) nel 2008 dove, in una scuola superiore, diciassette ragazze rimasero volutamente incinte nello stesso periodo, con l'intento di creare una comunità femminile lontano dalle violenze perpetrate dagli uomini in seno alle loro famiglie. Se nella cronaca locale tutta l'attenzione si era focalizzata sulle ragazze, il testo di Cuscunà sposta il baricentro sulla comunità, sul contesto sociale, e mette in scena, sottoforma di teste appese al muro5 , da una parte gli adulti, il preside e i genitori, dall'altro i giovani padri adolescenti che sono stati "usati" dalle ragazze per restare incinte. Al centro, tra i due blocchi, un monitor di cellulare sul quale compaiono le uniche poche battute delle ragazze, sotto forma di messaggi WhatsApp. In scena sono dunque coloro che sono stati messi al muro dalla scelta delle ragazze, resi scenograficamente dalle teste attaccate al muro. Queste, inoltre, risolvevano anche un'importante necessità tecnica: il testo è composto da scambi serrati di battute tra dodici personaggi, ma gestirne contemporaneamente i corpi non sarebbe stato possibile, essendo Marta Cuscunà da sola in scena. La parete di teste appese forniva dei pupazzi autoportanti, e risolveva tale problema. Viste però le dinamiche psicologiche della storia e la necessità che il pubblico potesse entrarvi agilmente, le teste sono state dotate di movimenti (degli occhi, delle sopracciglia, della bocca, del collo) che le rendessero verosimili, che creassero nella percezione dello spettatore uno sfasamento tra la loro agilità e il loro essere fittizie. Come evidenzia Paola Villani, «le possibilità di movimenti espressivi, date le tecniche di manipolazione, non erano infinite, quindi abbiamo studiato i monologhi dei personaggi per individuare il tipo di espressività che li caratterizzasse di più durante il monologo. L'importante era che il pupazzo restituisse efficacemente la storia»6 . Per rispettare questa coerenza drammaturgica, è stato necessario rendere agile il passaggio da una testa all'altra. Anziché obbligare la performer a infilare la mano dentro ogni testa, movimento impossibile data la velocità delle battute tra i tanti personaggi, Paola Villani ha configurato un sistema di manipolazione esterno, formato da cavi e joystick che permettono a Cuscunà di manovrare le espressioni di un personaggio con una sola mano.

Vediamo dunque come il lavoro concertato della marionettista-performer-regista con l'apparato scenografico e con l'occhio esterno drammaturgico, risulta centrale per lo sviluppo di soluzioni tecniche funzionali alla drammaturgia.

Johanny Bert. Il regista e la dramaturg

Anche laddove il teatro di figura contemporaneo scelga di rifarsi a testi del repertorio teatrale, o operistico, può agire attraverso riflessioni plurali per attuare una messa in scena in cui figure e oggetti restituiscano in modo puntale i nodi ritenuti centrali del testo scelto. È il caso della realizzazione dello spettacolo Opéra du dragon ad opera del regista Johanny Bert, tratto dal libretto d'opera 7 scritto da Heiner Müller nel 1969 quale adattamento del Dragone dell'autore russo Evgueni Schwartz. Negli spettacoli di Johanny Bert, la figura è utilizzata come un linguaggio alla pari di altri quali recitazione, danza, video-arte, installazioni, musica. Ad ogni creazione, la necessità è quella di mettere in azione i mezzi più adatti a servire un tema o un testo. Nei suoi lavori la figura risulta dunque usata quale «componente di una scrittura scenica plurale. Si tratta di una protesi che può far ascoltare il testo in modo diverso» 8 . I testi da lui finora messi in scena sono sia il risultato di commissioni di scrittura (ha collaborato con diversi autori contemporanei), sia testi le cui tematiche lo hanno particolarmente colpito, come nel caso dell'allegoria politica dell'Opéra du dragon. Nel corso dei lunghi periodi di prova che dedica ad ogni lavoro, Johanny Bert collabora con diverse persone in una ricerca collettiva in funzione del testo, all'interno della quale egli mantiene «uno sguardo globale sulla convergenza dei segni utilizzati» 9 .

Tale attenzione all'incontro tra dinamiche testuali e sceniche lo ha presto portato alla ricerca di collaborazioni con un dramaturg, non appena ne ha avuto la possibilità: «mettere in relazione il linguaggio dell'autore con una scelta estetica polimorfa come quella della marionetta è una ricerca appassionante. Il dialogo con il dramaturg è importante. Non ne ero consapevole all'inizio, e soprattutto non potevo permettermi di farlo nelle prime produzioni. Tuttavia, la presenza di un dramaturg dovrebbe essere parte integrante di un gruppo di lavoro, soprattutto nei primi progetti…» 10 . Per la messa in scena dell'Opéra du dragon, il lavoro drammaturgico, eseguito in sinergia con Julie Sermon, testimonia dell'avvicinamento delle processualità del teatro di figura contemporaneo a quelle del teatro tout-court: «Come faccio per ogni spettacolo, ho iniziato facendo delle tavole drammaturgiche, dove traccio i movimenti e le articolazioni del testo, […] dove immagino i possibili futuri scenici, tenendo conto dei desideri e dei pregiudizi del regista. […] Nelle osservazioni che ho fatto, avevo naturalmente in mente che le azioni sarebbero state portate avanti da delle effigi (la figura può fare cose che un attore non può fare e viceversa), ma non era questo il punto principale». 11 Il lavoro si è così articolato su diverse linee guida: la dissociazione corpo/voce, la manipolazione a vista, l'uso di oggetti quotidiani, la musica dal vivo, l'utilizzo di video 12 .

Il testo allegorico di Müller racconta di un dragone che, avendo liberato il popolo dal colera, è da esso venerato al punto che ogni anno gli sacrifica una fanciulla. Lancillotto, innamorato di Elsa (la ragazza prescelta per quell'anno), uccide il dragone ponendo fine al suo dominio, ma presto appare chiaro che le dinamiche di tirannia continuano nonostante la morte del dragone. Il libretto si articola per dispersioni: vi si alternano dialoghi, interventi del coro, pantomime, proiezioni video, canto e musica, didascalie che condensano salti temporali o spaziali. Lo spettacolo asseconda queste forze centrifughe. I numerosi personaggi sono delle marionette a bacchetta (un tipo di marionetta proteiforme: una testa appoggiata su un bastone, ricoperto da un pezzo di tessuto che fa da corpo, attraverso il quale il marionettista può prestare i suoi arti alla marionetta), manipolate a vista da tre performer. La voce che le anima, inoltre, proviene da una fonte distante, una sola attrice che a lato della scena interpreta ogni battuta e personaggio, dando luogo a una spazializzazione dispersiva della voce 13 , come nel Bunraku giapponese. La presenza delle marionette, infine, è alternata a quella di oggetti di uso quotidiano e alle proiezioni video.

Tale frammentazione invita lo spettatore a una ricezione attiva e critica. La manipolazione a vista fa si che lo spettatore si confronti direttamente ai rapporti di dominio esistenti nella relazione manipolatore-figura, che esprime in modo immediato i temi portanti del testo (l'abuso di potere, la mercificazione dei corpi, l'abbandono della propria coscienza a un individuo ritenuto superiore).

Del resto, come sottolinea Geneviève Jolly, «le nozioni di manipolazione e sottomissione sono onnipresenti nel libretto» 14 , tanto che Lancillotto, dopo aver sconfitto il dragone, vedendo che la popolazione tende a «continuare la logica di repressione» 15 in cui ha sempre vissuto, si chiede: 11 Julie Sermon, « Rencontre Dramaturgie et marionnette », op.cit. 12 Si rimanda alla dettagliata analisi di Geneviève Jolly, L'Opéra du dragon de Heiner Müller mis en scène par Johanny Bert : du « théâtre des voix » aux voies d'un théâtre émancipateur, «Études théâtrales», vol. 60-61, no. 2-3, 2014, pp. 91-101. 13 Si veda a proposito Sandrine Le Pors, Le Théâtre des voix. À l'écoute du personnage et des écritures contemporaines, Presses Universitaires de Rennes, 2011. 14 Geneviève Jolly, op. cit., p. 97. 15 Ibid. «Come possiamo ottenere degli uomini da questi fantocci / coperti di vergogna e di scaglie di drago»16 .

Inoltre, la messa a nudo del processo di manipolazione, del suo farsi contemporaneamente al farsi dello spettacolo, costituisce un dramma "secondo" rispetto a quello in primo piano, mette in scena il processo fratturando l'azione, in accordo con il testo composito di Müller, che è continuamente spezzato da variazioni, digressioni, alternanze tra didascalie e battute. Non a caso, la messa in scena di Johanny Bert mantiene evidente tale aspetto del testo, con la scelta di far pronunciare all'attrice tutte le didascalie. Come l'autore sregola i meccanismi drammatici del suo testo, lo commenta dall'esterno, lo mostra mentre lo costruisce, così la manipolazione a vista svela l'atto del farsi, e mette attore e spettatore in un'inevitabile posizione di distanziamento: l'uno rispetto alla marionetta e alla recitazione, l'altro rispetto alla favola raccontata in primo piano, favorendo una postura critica rispetto ad essa.

La figura può quindi essere usata per assecondare alcune piste del testo, ma anche per ottenere effetti di contrasto: nella prima scena del testo viene detto che solo colui che si nutrirà di acqua bollente del lago potrà sconfiggere il colera che attanaglia la città; il popolo invoca quindi l'aiuto del dragone affinché con il suo fuoco faccia bollire l'acqua del lago. L'arrivo solenne del dragone, atteso da tutti come salvatore contro la terribile piaga, è reso, nello spettacolo, dalla comparsa di un ben più prosaico bollitore: alla presenza dell'oggetto manufatto, fuori contesto, i marionettisti si fermano, la recitante si arresta, tutti rimangono in attesa che l'acqua cominci a bollire. E quando esce il fumo, il gioco è chiaro: il venerato mostro è simboleggiato da un bollitore in plastica.

Drastico mutamento di piano simbolico, conseguente effetto comico.

Marionette manipolate a vista, dissociazione della voce, comparsa di oggetti quotidiani… La concretezza della figura, studiata drammaturgicamente nei suoi effetti, costruisce senso attraverso corrispondenze o contrasti che mettono in risalto determinati aspetti del testo, concorrendo a renderlo pregnante nell'attualità.

Gisèle Vienne. La regista, l'autore, il performer

Talvolta un autore esterno può essere co-attivatore dei processi drammaturgici atti alla trasposizione del suo stesso testo per la scena delle figure. È il caso della collaborazione tra l'autore americano Dennis Cooper e la regista franco-austriaca Gisèle Vienne. Al loro sodalizio, inoltre, si aggiunge quello con il performer Jonathan Capdevielle 17 , che partecipa attivamente al processo creativo. La loro interrelazione è stata alla base dell'adattamento di una novella di Dennis Cooper, Jerk (2008), attraverso l'utilizzo di tre diverse tecniche del teatro di figura -la manipolazione a vista di burattini, il corps-castelet e la ventriloquia. Jerk s'ispira a un terribile fatto di cronaca avvenuto in Texas negli anni '70, quando il serial killer Dean Corll con l'aiuto di due adolescenti, David Brooks e il suo amante Wayne Henley, ha ucciso più di venti ragazzi, attraverso un gioco perverso di tossicodipendenza, pratiche sessuali, violenze e torture.

Il testo, adattato da Cooper stesso per lo spettacolo di Vienne, ha un impianto narrativo sviluppato su più livelli. In un primo livello, il performer si rivolge direttamente al pubblico, presentandosi come David Brooks (uno dei due ragazzi che ha preso parte agli omicidi, l'unico sopravvissuto), e dicendo che narrerà la sua storia attraverso la manipolazione di burattini a guanto (secondo livello).

In due momenti, inoltre, il pubblico sarà invitato a leggere dei passaggi di testo in cui si descrive il rivoltante gioco criminale a cui David Brooks ha preso parte. In tal modo si aggiunge un terzo livello, di narrazione diegetica. È interessante notare 18 che, nel romanzo, il tipo di figura che l'autore fa utilizzare al protagonista è la marionetta a fili, mentre nello spettacolo si fa uso esclusivamente del burattino a guanto. La ragione è espressa da Gisèle Vienne stessa: «Con questa terribile storia di giovani che uccidono altre persone ed entrano nei loro corpi, è stato molto appropriato vedere il burattinaio mettere letteralmente le mani nei corpi dei burattini. Era un'opportunità atroce» 19 .

Il personaggio di David Brooks anima i burattini della sua storia (il suo amante Wayne, il "capo" Dean, i ragazzi uccisi, e anche i cadaveri dei ragazzi uccisi che Dean talvolta si diverte ad animare come se fossero le sue marionette), e presta voci diverse ad ognuno di essi. Oltre ai burattini, Capdevielle/Brooks interpreta anche "se stesso", attraverso la sua voce normale. Seduto su una sedia, con due burattini infilati su entrambe le braccia e uno appoggiato sulle ginocchia, il corpo del performer diventa un luogo (un corps-castelet) dove s'incrociano non solo più corpi ma anche più identità. Il meccanismo si arricchisce di complessità nel momento in cui il performer, attraverso la 17 Secondo Dennis Cooper, Jonathan Capdevielle «ha apportato un enorme contributo allo spettacolo, ha inventato molte cose. È difficile sottovalutare quanto il suo lavoro sia parte del motivo per cui lo spettacolo funziona. Non avremmo potuto farlo con nessun altro, non credo che nessun altro possa fare quello che fa lui». Intervista a Dennis Cooper, 04/02/2022. 18 Si veda Jennifer Ratet, Du littéraire au marionnettique : réécritures de textes littéraires sur la scène marionnettique contemporaine, tesi di dottorato, Art et histoire de l'art, Université Paul Valéry -Montpellier III, 2019, p. 109. 19 Gisèle Vienne, « Un itinéraire de création », in Carole Guidicelli (dir.), Surmarionnettes et mannequins : Craig, Kantor et leurs héritages contemporains / Übermarionettes and Mannequins : Craig, Kantor and their Contemporary Legacies, actes du colloque de l'Institut International de la Marionnette (Charleville-Mézières, 15-17 mars 2012), Montpellier, L'Entretemps, 2013, p. 375. marionetta di Dean, interpreta un personaggio indicato nel testo come "Dean-as-corpse" ("Deancome-cadavere"): insoddisfatto di uccidere ragazzi di cui non conosce nel profondo l'identità, Dean vuole impossessarsi anche di essa, e una volta che di loro non resta che il cadavere, egli lo trasforma nel simulacro di alcune star televisive che ama. Il performer, dunque, anima la marionetta di Dean, la quale, diventando Dean-as-corpse, anima a sua volta la marionetta del ragazzo morto, che diventa per un istante un famoso attore della tv: Dean-as-corpse (stifling a smirk, he picks up the puppet of the dead boy and waves it in the air at Wayne): Hi, Wayne. It's me, Luke Halpin 20 .

Questa moltiplicazione schizofrenica delle personalità rispecchia due caratteri fondamentali della pièce: da un lato la rottura provocatasi nella mente di David Brooks a seguito delle violenze a cui ha preso parte; dall'altro lato, il tentativo di superare il trauma rimettendo in atto l'accaduto attraverso il teatrino di burattini. Il tentativo, tuttavia, fallisce. Il momento finale della storia, in cui David Brooks racconta di come ha ucciso il suo amante Wayne, viene eseguito attraverso la tecnica della ventriloquia, scelta operata da Vienne e resa possibile dalla straordinaria abilità di Capdevielle in questa tecnica. I burattini non sono più animati, ma le loro voci, che lo spettatore ormai riconosce, risuonano nello spazio attorno al performer, immobile sulla sedia su cui siede dall'inizio, lo sguardo vitreo perso nel vuoto mentre si sfinisce nello sforzo della tecnica del lip-skill. Come lo mette in luce Isabelle Barberis, «tutto accade come se ci fosse stato un atto di divorazione, o di interiorizzazione, delle marionette» 21 . Avendole introiettate, quelle maschere ora compiono davanti allo spettatore quello che, ogni giorno, esse compiono dentro David Brooks: animare la sua mente con fantasmi ossessionanti, trascinandolo in uno stato di incantamento.

L'intuizione di Dennis Cooper di rifarsi a un teatrino di figure per raccontare una storia di malattia mentale, giochi di potere e violenza trova così un grado ulteriore di esattezza una volta trasposto sulla scena. A immagine del complesso interno di David Brooks, lo sfaccettato testo di Dennis Cooper investe, nella messa in scena di Gisèle Vienne, corpo e identità del performer, costantemente instabili, reinterrogati.

In tv!

Negli esempi che abbiamo visto, la "mente teatrale" in opera nelle interazioni del marionettista con altre professionalità artistiche sviluppa processi drammaturgici complessi, che si servono delle 20 Dennis Cooper, Jerk, testo inedito per lo spettacolo. 21 Isabelle Barberis, Jerk, de Gisèle Vienne et Jonathan Capdevielle, in «Communications» n° 92, 2013, p. 159-172. https://doi-org.ezpupv.biu-montpellier.fr/10.3917/commu.092.0159. sperimentazioni moderne del teatro di figura (corps-castelet, manipolazione a vista) o che creano nuove tecniche di manipolazione (il sistema di cavi e joystick) per un uso mirato della figura in relazione a un particolare testo. Tali esempi ci mostrano anche come il riavvicinarsi dei marionettisti al testo scritto e l'incontro con le scritture drammatiche contemporanee abbia avuto un forte impatto sul divenire del "marionettista", rendendo necessaria una pluralità di collaborazioni e lo sviluppo di una "mente collettiva".

In conclusione, abbiamo visto come delle teste appese al muro possano evocare un sentimento di impotenza, come la relazione tra figura e marionettista nella manipolazione a vista dia un'immagine immediata dei rapporti di potere, come la tecnica del bunraku di separare voce e corpo della figura crei un effetto di straniamento, come un bollitore possa stare al centro dell'intero palcoscenico esprimendo più significato di un dragone, o ancora come l'inguantare un burattino possa produrre una perversione nell'uso stesso di quel mezzo, come, infine, la ventriloquia possa figurare i fantasmi che popolano una mente sofferente. Il teatro di figura aggiunge così altre dimensioni all'accadere scenico di una visione, e quando, attraverso la costruzione drammaturgica di senso, associa la parola a immagini non didascaliche o decorative, esso arriva forse a concretizzare quei processi di visualizzazione auspicati da Müller, donando il testo, potremmo dire, come un'immagine alla televisione.

7

  Il testo di Heiner Müller è un adattamento del testo teatrale Le Dragon (non tradotto in italiano) di Evgueni Schwartz (scritto a Mosca nel 1944), a sua volta ispirato a un racconto di Hans Christian Andersen. Per la messa in scena del libretto di Müller alla Staatsoper di Berlino nel 1969, la regista Ruth Berghaus scelse il titolo di Lanzelot. Heiner Müller pubblicò il libretto d'opera nella raccolta Theater-Arbeit, con il titolo Drachenoper.

	9	Olivia	Burton,	Arpenter	l'inconnu,	[Online].

8

Johanny Bert, in Olivia Burton, L'intelligence de l'image,

[Online]

. URL: https://www.artcena.fr/actualites-de-lacreation/magazine/portraits/johanny-bert/lintelligence-des-images URL: https://www.artcena.fr/actualites-de-lacreation/magazine/portraits/johanny-bert/arpenter-linconnu. 10 Johanny Bert, « Rencontre Dramaturgie et marionnette », op.cit.

Intervista a Marco Rogante, 24/03/2022.

Ispirate alla serie fotografica di Antoine Bardot.

Paola Villani, trascrizione del workshop "Neurospasta mechanica".

Heiner Muller, L'Opéra du dragon,(Drachenoper, 1968), testo francese di Renate e Maurice Taszman, Parigi, Éditions Théatrales, 2000, p. 51.


Amine Ferdjaoui
email: amine.ferdjaoui@etu.u-paris.fr

Mohamed Nadif
email: mohamed.nadif@u-paris.fr.

WordGraph: A Python Package for Reconstructing Interactive Causal Graphical Models from Text Data

Keywords: CCS Concepts:, Computing methodologies → Cluster analysis, Probabilistic reasoning, • Mathematics of computing → Causal networks co-clustering, causal network reconstruction, text data

We present WordGraph, a Python package for exploring the topics of documents corpora. WordGraph provides causal graphical models from text data vocabulary and proposes interactive visualizations of terms networks. Our ease-to-use package is provided with a pre-built pipeline to access the main modules through jupyter widgets. It results in the encapsulation of a whole vocabulary exploration process within a single jupyter notebook cell, with straightforward parameters settings and interactive plots. WordGraph pipeline is fully customizable by adding/removing widgets or changing default parameters. To assist users with no background in Python nor jupyter notebook, but willing to explore large corpora topics, we also propose an automatic dashboard generation from the customizable jupyter notebook pipeline in a web application style. WordGraph is available through a GitHub repository 1 .

input matrix as a bipartite graph between documents and words, approximate the normalized cut of this graph using a real relaxation [START_REF] Inderjit S Dhillon | Co-clustering documents and words using bipartite spectral graph partitioning[END_REF]. The model-based methods derived from appropriate Latent Block Models (LBM) [START_REF] Govaert | Block clustering with Bernoulli mixture models: Comparison of different approaches[END_REF], rely on expectation-maximization algorithms [START_REF] Salah | Directional co-clustering[END_REF]. Co-clustering can also use matrix factorization based methods such as Non-negative Matrix Factorization (NMF) [START_REF] Febrissy | Improving NMF clustering by leveraging contextual relationships among words[END_REF] or Tri-factorization (NMTF) [START_REF] Salah | Word co-occurrence regularized non-negative matrix tri-factorization for text data co-clustering[END_REF]. By contrast, information-theoretic based methods, used with two-way contingency tables, minimize a loss in mutual information while collapsing rows and columns of the matrix according to the co-clustering [START_REF] Inderjit S Dhillon | Information-theoretic co-clustering[END_REF]; note that the optimized criterion is associated to a particular Poisson LBM [START_REF] Govaert | Mutual information, phi-squared and model-based co-clustering for contingency tables[END_REF]. Finally, the co-partitioning is also possible from an adapted version of the modularity measure usually used for networks [START_REF] Labiod | Co-clustering for binary and categorical data with maximum modularity[END_REF].

The CoClust Python package is well-known among the popular packages dedicated to co-clustering for co-occurrence matrices, in particular for document-term matrices in text mining applications [START_REF] Role | Coclust: a python package for co-clustering[END_REF]. It provides the implementations of three algorithms that proved their efficiency in handling such matrices 2 . While word embeddings derived from sophisticated models have been largely acclaimed, we argue that the association of document-term matrices, which are straightforward Bag-Of-Words (BOW) text data representations, and co-clustering, remains a successful combination to segment textual data into meaningful topics. WordGraph is the first Python package that facilitates a visual exploration of large corpora topics based on co-clustering.

Causal network inference

The reconstruction of graphical models is a well-known practice in multiple areas. These models can be learned from time series data, controlled perturbation experiments, or observational data, as found in many biological contexts.

Traditional methods that exploit unperturbed data, include Bayesian search-and-score, sparse inverse covariance estimation, maximum entropy or diffusion map. Beyond the reconstruction of graphical models, the discovery of causality among multiple variables is of great interest. For instance, it can lead to the identification of key regulatory transcription factors at the origin of complex diseases. Applied to textual data, it could provide graphical summaries of topics. Causal networks could indicate the organizational flow among words through directed networks. Constraintbased methods [START_REF] Pearl | A theory of inferred causation[END_REF] can identify structural constraints corresponding to unnecessary edges in a graph while uncovering causality from unperturbed data.

The Multivariate Information-based Inductive Causation algorithm, MIIC3 , relies on an information theoretic method that combines constraint-based learning and maximum likelihood framework [START_REF] Affeldt | Robust Reconstruction of Causal Graphical Models based on Conditional 2-point and 3-point Information[END_REF][START_REF] Affeldt | 3off2: A network reconstruction algorithm based on 2-point and 3-point information statistics[END_REF][START_REF] Verny | Learning causal networks with latent variables from multivariate information in genomic data[END_REF]. Specifically, MIIC is based on the analysis of multivariate information, which extends the mutual information to more than two variables. In practice, the MIIC integration of constraint-based methods within an information-theoretic framework greatly improves the prediction accuracy, the running time and the scaling capabilities in terms of sample size and network size as compared to traditional approaches. Hence, while MIIC has demonstrated its applicability on a variety of genomic datasets at different biological size and times scales [START_REF] Verny | Learning causal networks with latent variables from multivariate information in genomic data[END_REF], it also appears to be particularly well-suited to tackle high dimensional textual data facilitating therefore the exploration of large corpora topics. WordGraph is the first Python overlay that applies MIIC to textual data. It is also the first package that combines graph reconstruction and co-clustering to discover corpus topics and their interactions.

Pipeline overall description

Our package unfolds in three parts. First, it creates a corpus of documents either locally loaded or directly fetched from the PubMed4 online database (Fig. 1,a). The creation step includes the textual data pre-processing, and the construction of a document-term matrix, with a weighting TF-IDF (Term Frequency-Inverse Document Frequency) procedure. Then, WordGraph fits a co-clustering on the document-term matrix to explore the topics, based on Coclust (Fig. 1, b, top). In this step, a sub-matrix of the most similar terms of a given word can be extracted from the document-term matrix, based on the cosine similarity score (Fig. 1, b, bottom). As a Python overlay of the MIIC R package, WordGraph can provide a graphical model from the most representative terms -typically, the most frequent words -of a co-cluster (Fig. 1,c, top). The interactive graph visualizations use the ipycytoscape 5 Python library. The user can reorganize the nodes, obtain magnified views or click on a node to build a secondary graph, based on the top similar words extracted from the document-term matrix (Fig. 1,c,bottom).

In the following, we detail the WordGraph multilevel capabilities. Indeed, it can either be used as is, in a regular Python source, or the user can customize an interactive jupyter notebook. Most importantly, a web application can instantaneously be generated from this notebook source to allow an easy online sharing. WordGraph is the first package that combines efficient co-clustering and robust causal graphical model for text data. As it relies on straightforward BOW representations, WordGraph avoid the computational and memory costs of sophisticated language models.

A MULTILEVEL PYTHON PACKAGE

WordGraph uses the functionalities of Coclust and MIIC at 3 levels for exploring textual data. The main objective is to enable the exploration of corpora topics through causal graphical models. In particular, WordGraph proposes an interactive notebook for word graphical models reconstruction enhanced by a customizable web application.

A Python overlay for words causal network

The WordGraph package is the first Python package that proposes an overlay over the MIIC R package for textual data.

Figure 2 provides the main functionalities of WordGraph and their associated modules.

The package can locally load a corpus or fetch documents from PubMed (io module). WordGraph embeds MIIC in the build_graph method, and relies on the functions create_miic and preproc_graphML (utils module). utils also enables the corpus pre-processing. A simple usage example of WordGraph, from the PubMed fetching to the MIIC graph object computation, can be found in the Simple usage section of the online GitHub presentation of the package 6 . We provide in WordGraph a jupyter notebook that embeds this simple usage proposal (see WordGraph simple usage.ipynb).

An interactive notebook for an easy design

WordGraph offers visualization functionalities from ipycytoscape package. In particular, plot_graph displays interactive networks within a jupyter notebook, where the layout can be directly adjusted with the mouse. It gives a graph of clusters' top terms and hence, easily summarizes contents (Fig. 1,(c),top). By clicking on a node, it displays a secondary interactive graph, in which the vertices are derived by their similarity to the selected node (Fig. 1,(c), bottom).

Most importantly, WordGraph proposes an easily customizable jupyter notebook (WordGraph Custom.ipynb), where the whole pipeline can be embedded in a single cell. To this aim, it embeds several ipywidgets elements (eg. slider, button, text or numerical inputs). Figure 3 shows how the pipeline is organized in a single cell. Modifying or adding a new widget takes only a few lines of code. The Customizable notebook pipeline section of the online GitHub presentation of the package 7 provides the implementation for setting a confirm button and calling the corresponding function on click.

An interactive web application

The WordGraph Custom.ipynb was also designed to be automatically turned into a web application, with the voilá8 package. Hence, the user can exploit the WordGraph Custom.ipynb as is or customize it, and then easily obtain the corresponding web application, dockerized and ready-to-use for users non familiar with coding (Fig. 4).

QUALITATIVE EVALUATIONS

We provide qualitative evaluations of WordGraph abilities to give topics summaries for a large corpus. First, we consider a biomedical application on lung cancer, based on 10, 000 abstracts fetched from PubMed by our pipeline. Second, we consider the PubMed109 benchmark dataset. WordGraph explores the topics by fitting a co-clustering that efficiently segments the vocabulary, and then automatically learns a causal graphical model for each co-cluster.

In Figure 1, the opposition (blue edge) between NSCLC (Non-Small Cell Lung Cancer) and LUAD (LUng ADenocarcinoma) suggests that documents dealing with LUAD ignore NSCLC topic. Indeed, while LUAD belongs to NSCLC, their prognosis and gene expression are distinct and they are seen as separated entities. EGFR-positive NSCLC emerges from mutated EGFR gene (Epidermal Growth Factor Receptor). Our graph causally relates EGFR to the NSCLC acronym. ICI (Immune Checkpoint Inhibitor) treatment is also causally related with the immun node. The secondary graph, based on egfr, focuses on the treatment, which usually aims to prevent abnormal tumor growth by targeting Tyrosin Kinase Inhibitor (TKI) as with osimertinib drug.

We also provide a graphical summary of the Age-Macular Degeneration (AMD) disease from PubMed10 (Fig. 5, left), and compare it with a pairwise graph model using BioWordVec embeddings [START_REF] Zhang | BioWordVec, improving biomedical word embeddings with subword information and MeSH[END_REF] and cosine similarity scores (Fig. 5, right).

Edges are retained based on a threshold (typically 0.5), a small angle between two word vectors being a good indicator of similarity. WordGraph network thereby provides clearer interactions as it only harbors direct relationships, following on MIIC algorithm. Furthermore, WordGraph requires no arbitrary edges threshold, but automatically estimates a complexity penalty to discover unnecessary edges. Importantly, WordGraph needs no heavily pretrained model that would be specifically fitted on an appropriate large corpus. Finally, we can highlight the biomedical relevance of the WordGraph network. edema is upstream of macular, but also of intravitreal injection, the main AMD treatment. diabetes, a risk factor for AMD, is also found upstream of macular. We also notice that cnv (Choroidal Neovascularization) is located dowstream of neovascularization.

CONCLUSION

The exploration of large corpora topics is becoming harder as huge amount of textual data are being made available through multiple areas. We propose WordGraph as an ease-to-use Python package for users willing to retrieve as much information as possible from various topic areas. We have made particular efforts in order to turn WordGraph into a standalone web application for users with no background in Python nor jupyter notebook, while still allowing for customizable interface. Our qualitative evaluations on lung cancer and PubMed10 data show promising results, in particular as compared to word embeddings graphical models. The scalability of co-clustering combined with the robustness of causal network reconstruction within WordGraph provides a valuable topics exploration package.

Fig. 1 .

 1 Fig. 1. WordGraph workflow, from collecting documents to interactive causal networks between words. An example on lung cancer. (a) Load or fetch documents, clean data and build document-term matrix. (b) Extract a sub-matrix from co-clustering or cosine similarity scores. (c) Explore topics with causal graphs; red edges: direct co-occurrence, blue edges: antagonist co-occurrence, arrows: word presence causality.

Fig. 2 .

 2 Fig. 2. WordGraph package: main functionalities.

Fig. 3 .

 3 Fig. 3. Interactive pipeline in a jupyter notebook.

Fig. 4 .

 4 Fig. 4. Customizable web application.

Fig. 5 .

 5 Fig. 5. (left) WordGraph; (right) Embeddings similarity .

https://coclust.readthedocs.io/en/v0.2.1/install.html Manuscript submitted to ACM

See https://github.com/miicTeam/miic_R_package for the MIIC R implementation

https://pubmed.ncbi.nlm.nih.gov; more than 35 millions biomedical citations

https://ipycytoscape.readthedocs.io/en/latest/ Manuscript submitted to ACM

https://github.com/MLDS-software/WordGraph#1-simple-usage

https://github.com/MLDS-software/WordGraph#2-customizable-notebook-pipeline

https://voila.readthedocs.io/en/stable/using.html Manuscript submitted to ACM

15, 565 × 22, 437 documents×term,

expected classes corresponding to 10 diseases Manuscript submitted to ACM

Manuscript submitted to ACM

ACKNOWLEDGMENTS

This work was supported by a grant overseen by the French National Research Agency (ANR) (ANR-19-CE23-0002). We thank the Isambert Lab for its helpful support on MIIC.


Louis Falissard
email: louis.falissard@u-paris.fr

Séverine Affeldt
email: severine.affeldt@u-paris.fr

Mohamed Nadif
email: mohamed.nadif@u-paris.fr

Attentive perturbation: extending prefix tuning to large language models inner representations

Keywords: Large language models, Parameter efficient fine-tuning, Adapters, Prefix-tuning, Natural language processing, Natural Language Understanding

From adapters to prefix-tuning, parameter efficient fine-tuning (PEFT) has been a well investigated research field in the past few years, which has led to an entire family of alternative approaches for large language model fine-tuning. All these methods rely on the fundamental idea of introducing additional learnable parameters to the model, while freezing all pre-trained representations during training. This finetuning process is generally done through refitting all model parameters to the new, supervised objective function. This process, however, still requires a considerable amount of computing power, which might not be readily available to everyone. In addition, even with the use of transfer learning, this method requires substantial amounts of data. In this article, we propose a novel and fairly straightforward extension of the prefix-tuning approach to modify both the model's attention weight and its internal representations. Our proposal introduces a "token-tuning" method relying on soft lookup based embeddings derived using attention mechanisms. We call this efficient extension "attentive perturbation", and empirically show that it outperforms other PEFT methods on most natural language understanding tasks in the few-shot learning setting.

Introduction

The use of pre-trained large language models, such as BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] or GPT [START_REF] Radford | Language Models are Unsupervised Multitask Learners[END_REF] have been ubiquitous in a wide range of natural language processing tasks. These models are pre-trained in a self-supervised fashion on massive textual datasets and are made readily available online. The machine learning practitioner can then use these models' internal representations as a starting point -either as an initialization scheme or a feature extraction tool -to build powerful models for NLP related tasks (eg., summarization, machine translation, language generation).

The fine-tuning of pre-trained language models to perform downstream tasks requires storing and modifying a copy of all the model's parameters for each task. Yet, some powerful large language models involve a prohibitively high number of parameters (eg. 774M parameters for GPT-2 [START_REF] Radford | Language Models are Unsupervised Multitask Learners[END_REF], 175B parameters for GPT-3 [START_REF] Brown | Language models are few-shot learners[END_REF]). In such cases, fine-tuning can be readily unfeasible, in particular for areas where collected datasets are of small sample sizes. Indeed, while the smallest dataset used in the experiments presented in BERT models' seminal article contains 2.5K observations, such amount of data is generally impractical to obtain, for instance in clinical studies involving a cohort of patients.

Adapters and prefix-tuning approaches have motivated numerous investigations, as they enable lightweight fine-tuning by introducing additional learnable parameters to the model, while freezing all pre-trained representations during training [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF][START_REF] Houlsby | Parameter-efficient transfer learning for nlp[END_REF]. Hence, such approaches are interesting ways to leverage large pre-trained language models for downstream NLP tasks. Over the last few years, they have been at the origin of a wide family of alternative approaches for large language model fine-tuning.

Although each individual approach has its specificity, they can be gathered into two main categories: (i) adapter-like approaches, that modify the model's internal representations with the use of bottleneck perceptrons introduced at strategic points in every transformer stack, and (ii) prefix or prompt based approaches, that introduce learnable virtual tokens in the input sequence, which allows to modify each transformer stack's attention weights. Some attempts have been made to unify these two families of methods, but mostly consist of combining them independently in a mosaic of adapter-like methods, which both considerably raises the number of model hyperparameters and doesn't necessarily yields the expected gains in predictive performances.

In this work, we demonstrate that a straightforward extension of the prefixtuning approach, named Attentive perturbation, can efficiently outperforms the parameter efficient fine-tuning (PEFT) methods in the few-shot learning setting. In the following, we introduce in Section 2 the family of PEFT methods. Then, Section 3 provides detailed information on our novel prefix tuning extension. Section 4 describes our specific experimental settings, based on the GLUE benchmark [START_REF] Wang | GLUE: A multitask benchmark and analysis platform for natural language understanding[END_REF]. Finally, we discuss our results in Section 5, before the conclusion in Section 6.

PEFT Methods

The field of parameter efficient fine-tuning, which has been an active area of research for a few years now, proposes alternatives to this standard fine-tuning scheme that allows practitioners to leverage large language models' powerful learned representations, even with a few labeled instances. Although these approaches tend to differ in their implementation, most of them rely on the same idea, which is to freeze all the language model's parameters during training, and instead train additional, trainable ones, typically injected in the model in an additive fashion. These additional parameters are typically orders of magnitude less numerous than the actual model, leading to better performances in the few-shot learning setting, and less computational requirement for model fitting.

Since the introduction of Adapters methods [START_REF] Houlsby | Parameter-efficient transfer learning for nlp[END_REF], an entire family of PEFT methods has been introduced in the academic literature. These methods have varying degrees of performances, both in terms of parameter efficiency and predictive power. This family of methods embeds:

-Adapters and their variants [START_REF] Houlsby | Parameter-efficient transfer learning for nlp[END_REF][START_REF] Pfeiffer | AdapterFusion: Nondestructive task composition for transfer learning[END_REF], which use a 2 layers bottleneck perceptron to additively corrupt the transformer stack's inner representations at one or several points after its attention module.

-Bitfit [START_REF] Ben Zaken | BitFit: Simple parameter-efficient finetuning for transformer-based masked language-models[END_REF], which have the particularity to not introduce any additional parameter to the model, but instead limits itself to fine-tuning solely the bias terms in all of the model's layers.

-Low Rank Adaptation (LoRA) [START_REF] Hu | LoRA: Low-rank adaptation of large language models[END_REF], which additively reparameterize the queries and values projection matrices with a linear bottleneck multilayer perceptron.

-Prefix-tuning [START_REF] Liu | P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks[END_REF], which, instead of modifying the transformer's stack's signal itself, preprends to the query and value sequences additional learnable tokens, in an approach similar to continuous prompt learning.

-Prompt fine-tuning [START_REF] Lester | The power of scale for parameter-efficient prompt tuning[END_REF], a simpler variant of prefix tuning where learnable virtual tokens are prepended to the input sequence. This method, however, tends to work only for larger models, and is as such discarded from the scope of this article.

-Fusion methods such as Unified Framework for Parameter Efficient Language model Tuning (UniPELT) [START_REF] Mao | UniPELT: A unified framework for parameter-efficient language model tuning[END_REF], which leverages a combination of several adapter modules at once associated with gating mechanisms, in order to federate each approach's advantages while alleviating their respective drawbacks.

Proposed approach: Attentive perturbation

As aforementioned, apart from prefix (or prompt) tuning, most parameter efficient fine-tuning methods are based on the additive introduction to the model of a bottleneck two layer perceptron, with varying activation function, at varying points in the transformer stack. Prefix-tuning, however, does not directly modify the model's signal, but instead focuses on adapting each transformer stack's attention weights, and is based on learnable embedding. Following this latter observation, we propose a method to additively modify the model's internal representation based on embeddings, rather than bottleneck perceptrons, and choose to apply these perturbations to the attention layer's queries and values, in an approach conceptually similar to LoRA.

Applying these embeddings in an additive fashion to the attention layer's input sequence, which is continuous and of varying length by essence, is not as straightforward as the perceptron approach used in LoRA or adapter methods. We propose to do so using a soft lookup mechanism on the embeddings, derived in a similar fashion to attention mechanisms.

In other words, for each perturbation module (one for the transformer stack's queries, and another for its values), we introduce two distinct embeddings E q and E v , and combine them with the transformer layer's input in a scaled dotproduct attention module as keys, values and queries, respectively. The module's output is then additively reinjected into the layer's input sequence, prior to the transformer stack's multi-head attention module, as can be seen in Figure 1. We call these modules "attentive perturbators" and formally define them as follows for a given transformer layer:

P q (I) = I + softmax( IE ⊤ q,1 √ d I )E q,2 (1)
P v (I) = I + softmax( IE ⊤ v,1 √ d I )E v,2 (2)
where -I is the layer's input sequence of vectors -E q,1 , E q,2 are the trainable embeddings used for the queries perturbation Similar to what is done in prefix-tuning to improve optimization stability, we do not learn these embeddings directly, but instead perform a reparameterization trick using a two layer bottleneck perceptron, shared for the queries and values embeddings. To the best of our knowledge, we are the first to introduce an adapter-like module that makes use of attention based operation instead of multilayer perceptrons.

-E v,1 , E v,

Experiments settings

We choose to assess our method's performances on the GLUE benchmark [START_REF] Wang | GLUE: A multitask benchmark and analysis platform for natural language understanding[END_REF], a corpus of 8 natural language understanding tasks all formulated as single or double sentence classification problems, with varying sample sizes. However, since we limit ourselves to the few-shot learning setting, we do not use these datasets as is. Instead, we use a methodology similar to what was presented in the UniPELT paper for our experiments.

Basically, we sample observations from the GLUE benchmark in order to build a series of small datasets with 50, 100, 200 and 500 observations. Our method for building those datasets, however, differs from theirs on a few key points.

First, in [START_REF] Wang | GLUE: A multitask benchmark and analysis platform for natural language understanding[END_REF] the authors chose to build validation sets of 1, 000 observations for all sampled training datasets, which we feel is unrealistic in a few-shot learning settings (validation datasets do not typically contain more than ten times the number of training examples). Second, they use the GLUE benchmark validation datasets as test datasets. However, some of these validations datasets have considerably small sample sizes (277 for RTE, 408 for MRPC), which might lead to noisy test metric evaluation. As a consequence, for each of the glue benchmark dataset, a dataset of sample size K is built as follows:

1. The original training and validation sets were concatenated into one sampling dataset.

2. A test dataset was built from the observations, using random sampling with a sample size of half the total sample size (capped at 5K observations). For a given task, the same test dataset is used for all experiments (including all sampled training sizes).

3. From the remaining observations, K were sampled randomly, and divided with a 70/30 split into a training and validation set, respectively

Ten distinct training datasets were sampled from all glue benchmarks and for all sample sizes, so as to allow for performance metric averaging and statistical testing of performance differences.

For all these datasets, we compare our method with standard fine-tuning and 4 other parameter efficient methods, namely standard Adapters, LoRA, Prefix-tuning and UniPELT. For all methods, we follow the same training procedure and hyperparameter search that is done in the UniPELT paper. All models were fit for 50 epochs using AdamW with linear weight decay and an early stopping mechanism with a patience of 10 non increasing epochs. The batch size was fixed to 16, and all method's specific hyperparameters were set up as follows:

-Standard fine-tuning: Learning rate ∈ [2 × 10 -5 , 1 × 10 -5 ] .

-Adapters: Learning rate of 1×10 -4 and a reduction rate taken from {12, 6, 3}.

-LoRA: rank and α values of 8, and a learning rate ∈ [5 × 10 -4 , 1 × 10 -4 ].

-Prefix-Tuning: The prefix length was fixed to 50. The reparameterization trick was applied to the method with a bottleneck size of 512, and the learning rate taken from {5 × 10 -4 , 2 × 10 -4 , 1 × 10 -4 }.

-UniPELT: Prefix-tuning with 10 prefixes, adapter with a reduction factor of 16 and standard LoRA.

-Attentive perturbation (our approach): Same exact hyperparameter configuration as prefix-tuning. Embedding size fixed to 50, reparameterization trick applied with a bottleneck size of 512 and learning rate taken from {5 × 10 -4 , 2 × 10 -4 , 1 × 10 -4 }.

The proposed approach was implemented using the adapter-hub library, which already proposes implementations for all chosen baseline, as to use the same code base for all experiments to ensure fairness across all experiments.

Results

The performances of all selected PEFT methods as well as the proposed approach on the GLUE benchmark for varying sample sizes can be seen in Table1.

Overall Attentive perturbation outperforms all other baselines on average for all sample sizes. This gain in performance of more than 1 point compared to prefix-tuning, the second highest baseline, on all sample sizes except for 500 observations, where the performance gain starts to decrease, strongly supporting that the proposed approach is better suited to the few shot learning setting. The fact that the best performance gains for sample sizes of 50 and 100 is associated with the COLA dataset, which is strongly imbalanced, tends to bring further evidence to confirm this observation.

In addition, the proposed approach yields the best and second best performances on a given task for a given sample size 18 and 9 times out of 32, respectively. The second best method, Prefix-tuning, only yields the best and second best performances 5 and 16 times, respectively. In other words, our proposed approach is not in the top 2 best methods only 5 times in the 32 experiments presented in this paper.

Surprisingly, UniPELT, which fuses LoRA, Prefix-tuning and Adapter methods, does not outperform Prefix-tuning. This might be due to our experiment set up, with considerably smaller validation sets than those used in their experimental setup.

Conclusion

In this paper we introduced a new parameter efficient fine-tuning method, the attentive perturbator. This method is fairly straightforward and easy to use; it requires a comparable amount of computation to prefix-tuning, which is considerably lower than most adapter-like methods at inference time. In addition, we empirically showed that this method behaves better on average than all other selected baseline PEFT methods, as well as traditional fine-tuning, on a variant of the GLUE benchmark specifically tailored to assess model performances in the few-shot learning setting.

Fig. 1 .

 1 Fig. 1. Overview of the attentive perturbation module. The bottleneck adapter used for the reparameterization trick is hidden for simplicity

2

 2 are the trainable embeddings used for the values perturbation d I is the input sequence's dimensionality (and as such the embeddings dimensionality as well), in other words the model's hidden size

Table 1 .

 1 Experiment results. F1 scores are reported for QQP and MRPC. Spearman correlations are reported for STS-B. Matthews correlation are reported for CoLA. Accuracy measurements are reported for all other datasets. Results in bold and underlined correspond to the best and second best results for the selected dataset and sample size, respectively.

Acknowledgments. This work was supported by a grant overseen by the French National Research Agency (ANR) (ANR-19-CE23-0002). It also received the labelling of Cap Digital and EuroBiomed competitiveness clusters.