Shashanka Venkataramanan

Ewa Kijak

Laurent Amsaleg

Yannis Avrithis

AlignMixup: Improving Representations By Interpolating Aligned Features

Mixup is a powerful data augmentation method that interpolates between two or more examples in the input or feature space and between the corresponding target labels. However, how to best interpolate images is not well defined. Recent mixup methods overlay or cut-and-paste two or more objects into one image, which needs care in selecting regions. Mixup has also been connected to autoencoders, because often autoencoders generate an image that continuously deforms into another. However, such images are typically of low quality.

In this work, we revisit mixup from the deformation perspective and introduce AlignMixup, where we geometrically align two images in the feature space. The correspondences allow us to interpolate between two sets of features, while keeping the locations of one set. Interestingly, this retains mostly the geometry or pose of one image and the appearance or texture of the other. We also show that an autoencoder can still improve representation learning under mixup, without the classifier ever seeing decoded images. AlignMixup outperforms state-of-theart mixup methods on five different benchmarks. Code

Introduction

Data augmentation [START_REF] Ekin D Cubuk | AutoAugment: Learning augmentation strategies from data[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Paulin | Transformation pursuit for image classification[END_REF] is a powerful regularization method that increases the amount and diversity of data, be it labeled or unlabeled [START_REF] Dosovitskiy | Unsupervised feature learning by augmenting single images[END_REF]. It improves the generalization performance and helps learning invariance [START_REF] Simard | Transformation invariance in pattern recognition-tangent distance and tangent propagation[END_REF] at almost no cost, because the same example can be transformed in different ways over epochs. However, by operating on one image at a time and limiting to label-preserving transformations, it has limited chances of exploring beyond the image manifold. Hence, it is of little help in combating memorization of training data [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF] and sensitivity to adversarial examples [START_REF] Szegedy | Intriguing properties of neural networks[END_REF].

Mixup operates on two or more examples at a time, interpolating between them in the input space [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF] or feature space [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF], while also interpolating between target la-Image 1

Input mixup [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF] CutMix [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF] Image 2 Manifold mixup [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF] AlignMixup (Ours) bels for image classification. This flattens class representations [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF], reduces overly confident incorrect predictions, and smoothens decision boundaries far away from training data. However, input mixup images are overlays and tend to be unnatural [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF]. Interestingly, recent mixup methods focus of combining two [START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF][START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF] or more [31] objects from different images into one in the input space, making efficient use of training pixels. However, randomness in the patch selection and thereby label mixing may mislead the classifier to learn uninformative features [57], which raises the question: what is a good interpolation of images? Bengio et al. [START_REF] Bengio | Better mixing via deep representations[END_REF] show that traversing along the manifold of representations obtained from deeper layers of the network more likely results in finding realistic examples. This is because the interpolated points smoothly traverse the underlying manifold of the data, capturing salient characteristics of the two images. Furthermore, [START_REF] Berthelot | Understanding and improving interpolation in autoencoders via an adversarial regularizer[END_REF] show the ability of autoencoders to capture semantic correspondences obtained by decoding mixed latent codes. This is because the autoencoder may disentangle the underlying factors of variation. Efforts have followed on mixing latent representations of autoencoders to generate realistic images for data aug-mentation. However, these approaches are more expensive, requiring three networks (encoder, decoder, classifier) [START_REF] Berthelot | Understanding and improving interpolation in autoencoders via an adversarial regularizer[END_REF] and more complex, often also requiring an adversarial discriminator [START_REF] Beckham | On adversarial mixup resynthesis[END_REF][START_REF] Liu | Data augmentation via latent space interpolation for image classification[END_REF]. More importantly, they perform poorly compared to standard input mixup on large datasets [START_REF] Liu | Data augmentation via latent space interpolation for image classification[END_REF], due to the low quality of generated images.

In this work, we are motivated by the idea of deformation as a natural way of interpolating images, where one image may deform into another, in a continuous way. Contrary to previous efforts, we do not interpolate directly in the input space, we do not limit to vectors as latent codes and we do not decode. We rather investigate geometric alignment for mixup, based on explicit semantic correspondences in the feature space. In particular, we explicitly align the feature tensors of two images, resulting in soft correspondences. The tensors can be seen as sets of features with coordinates. Hence, each feature in one set can be interpolated with few features in the other.

By choosing to keep the coordinates of one set or the other, we define an asymmetric operation. What we obtain is one object continuously morphing, rather than two objects in one image. Interestingly, observing this asymmetric morphing reveals that we retain the geometry or pose of the image where we keep the coordinates and the appearance or texture of the other. Figure 1 illustrates that our method, AlignMixup, retains the pose of image 2 and the texture of image 1, which is different from existing mixup methods. Note that, as in manifold mixup, we do not decode, hence we are not concerned about the quality of generated images.

We make the following contributions:

1. We introduce a novel mixup operation, called Align-Mixup, advocating interpolation of local structure in the feature space (subsection 3.2). Feature tensors are ideal for alignment, giving rise to semantic correspondences and being of low resolution. Alignment is efficient by using Sinkhorn distance [START_REF] Cuturi | Sinkhorn distances: lightspeed computation of optimal transport[END_REF]. 2. We also show that a vanilla autoencoder can further improve representation learning under mixup training, without the classifier seeing decoded clean or mixed images (section 4). 3. We set a new state-of-the-art on image classification, robustness to adversarial attacks, calibration, weaklysupervised localization and out-of-distribution detection against more sophisticated mixup operations on several networks and datasets (section 4).

Related Work

Mixup [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF], concurrently with similar methods [START_REF] Inoue | Data augmentation by pairing samples for images classification[END_REF]56], introduce mixup, augmenting data by linear interpolation between two examples. While [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF] apply mixup on intermediate representations, it is [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF] who make this work, introducing manifold mixup. Without alignment, the result is an overlay of either images [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF] or features [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF]. [START_REF] Guo | Mixup as locally linear out-of-manifold regularization[END_REF] eliminate "manifold intrusion"-mixed data conflicting with true data. Unlike manifold mixup, AlignMixup interpolates feature tensors from deeper layers after aligning them. Nonlinear mixing over random image regions is an alternative, e.g. from masking square regions [START_REF] Devries | Improved regularization of convolutional neural networks with cutout[END_REF] to cutting a rectangular region from one image and pasting it onto another [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF], as well as several variants using arbitrary regions [START_REF] Harris | Fmix: Enhancing mixed sample data augmentation[END_REF][START_REF] Summers | Improved mixedexample data augmentation[END_REF][START_REF] Takahashi | Ricap: Random image cropping and patching data augmentation for deep cnns[END_REF]. Instead of choosing regions at random, saliency can be used to locate objects from different images and fit them in one [31,[START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF][START_REF] Qin | Resizemix: Mixing data with preserved object information and true labels[END_REF]57]. Exploiting the knowledge of a teacher network to mix images based on saliency has been proposed in [START_REF] Dabouei | Supermix: Supervising the mixing data augmentation[END_REF]. Instead of combining more than one objects in an image, AlignMixup attempts to deform one object into another.

Another alternative is Automix [START_REF] Zhu | Automix: Mixup networks for sample interpolation via cooperative barycenter learning[END_REF], which employs a U-Net rather than an autoencoder, mixing at several layers. It is limited to small datasets and provides little improvement over manifold mixup [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF]. StyleMix and StyleCut-Mix [START_REF] Hong | Stylemix: Separating content and style for enhanced data augmentation[END_REF] interpolate content and style between two images, using AdaIN [START_REF] Huang | Arbitrary style transfer in real-time with adaptive instance normalization[END_REF], a style transfer autoencoder network. By contrast, AlignMixup aligns feature tensors and interpolates matching features directly, without using any additional network.

Alignment Local correspondences from intra-class alignment of feature tensors have been used in image registration [9,[START_REF] Long | Do convnets learn correspondence?[END_REF], optical flow [START_REF] Weinzaepfel | Deepflow: Large displacement optical flow with deep matching[END_REF], semantic alignment [START_REF] Han | Scnet: Learning semantic correspondence[END_REF][START_REF] Rocco | End-toend weakly-supervised semantic alignment[END_REF] and image retrieval [START_REF] Siméoni | Local features and visual words emerge in activations[END_REF]. Here, we mostly use inter-class alignment. In few-shot learning, local correspondences between query and support images are important in finding attention maps, used e.g. by CrossTransformers [START_REF] Doersch | Crosstransformers: spatially-aware few-shot transfer[END_REF] and DeepEMD [START_REF] Zhang | Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers[END_REF]. The earth mover's distance (EMD) [47], or Wasserstein metric, is an instance of optimal transport [START_REF] Villani | Optimal transport: old and new[END_REF], addressed by linear programming. To accelerate, [START_REF] Cuturi | Sinkhorn distances: lightspeed computation of optimal transport[END_REF] computes optimal matching by Sinkhorn distance with entropic regularization. This distance is widely applied between distributions in generative models [START_REF] Genevay | Learning generative models with sinkhorn divergences[END_REF][START_REF] Patrini | Sinkhorn autoencoders[END_REF].

EMD has been used for mixup in the input space, for instance point mixup for 3D point clouds [6] and OptTrans-Mix for images [START_REF] Zhu | Automix: Mixup networks for sample interpolation via cooperative barycenter learning[END_REF], which is the closest to our work. However, aligning coordinates only applies to images with clean background. We rather align tensors in the feature space, which is generic. We do so using the Sinkhorn distance, which is orders of magnitude faster than EMD [START_REF] Cuturi | Sinkhorn distances: lightspeed computation of optimal transport[END_REF].

AlignMixup

Preliminaries

Problem formulation Let (x, y) be an image x ∈ X with its one-hot encoded class label y ∈ Y , where X is the input image space, Y = [0, 1] k and k is the number of classes. An encoder network F : X → R c×w×h maps x to feature tensor A = F (x), where c is the number of channels and w × h is the spatial resolution. A classifier g : R c×w×h → R k then maps A to the vector p = g(A) of probabilities over classes.

Mixup

We follow [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF] in mixing the representations from different layers of the network, focusing on the deepest layers near the classifier. We are given two labeled images (x, y), (x ′ , y ′ ) ∈ X × Y . We draw an interpolation factor λ ∈ [0, 1] from Beta(α, α) [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF] and then we interpolate labels y, y ′ linearly by the standard mixup operator

mix λ (y, y ′ ) := λy + (1 -λ)y ′ (1)
and inputs x, x ′ by the generic formula

Mix f1,f2 λ (x, x ′ ) := f 2 (Mix λ (f 1 (x), f 1 (x ′ )), (2)
where Mix λ is a mixup operator to be defined. This generic formula allows interpolation of the input or feature as

f 2 •f 1 according to input (x) : f 1 := id, f 2 := F (3) feature (A) : f 1 := F, f 2 := id, ( 4
)
where id is the identity mapping. For (3), we define Mix λ in (2) as standard mixup mix λ (1), like [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF]; while for (4), we define Mix λ as discussed in subsection 3.2. By default, we train the encoder network and the classifier by using a classification loss L c on the output of the classifier g for mixed examples along with the corresponding mixed labels:

L c (g(Mix f1,f2 λ (x, x ′ )), mix λ (y, y ′ )), (5)
where L c (p, y) := -k i=1 y i log p i is the standard crossentropy loss. More options using an autoencoder architecture are investigated in section 4.

Interpolation of aligned feature tensors

Alignment Alignment refers to finding a geometric correspondence between image elements before interpolation. The feature tensor is ideal for this purpose, because its spatial resolution is low, reducing the optimization cost, and allows for semantic correspondence, because features close to the classifier are small. Importantly, we are not attempting to combine two or more objects into one image [START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF], but put two objects in correspondence and then interpolate into one. We make no assumptions on the structure of input images in terms of objects and we use no ground truth correspondences.

Our feature tensor alignment is based on optimal transport theory [START_REF] Villani | Optimal transport: old and new[END_REF] and Sinkhorn distance (SD) [START_REF] Cuturi | Sinkhorn distances: lightspeed computation of optimal transport[END_REF] in particular. Let A := F (x), A ′ := F (x ′ ) be the c × w × h feature tensors of images x, x ′ ∈ X . We reshape them to c × r matrices A, A ′ by flattening the spatial dimensions, where r := hw. Then, every column a j , a ′ j ∈ R c of A, A ′ for j = 1, . . . , r is a feature vector representing corresponding to a spatial position in the original image x, x ′ . Let M be the r × r cost matrix with its elements being the pairwise distances of these vectors:

m ij := a i -a ′ j 2 (6)
for i, j ∈ {1, . . . , r}. We are looking for a transport plan, that is, a r × r matrix P ∈ U r , where

U r := {P ∈ R r×r + : P 1 = P ⊤ 1 = 1/r} (7)
and 1 is an all-ones vector in R r . That is, P is non-negative with row-wise and column-wise sum 1/r, representing a joint probability over spatial positions of A, A ′ with uniform marginals. It is chosen to minimize the expected pairwise distance of their features, as expressed by the linear cost function ⟨P, M ⟩, under an entropic regularizer:

P * = arg min P ∈Ur ⟨P, M ⟩ -ϵH(P ), (8)
where

H(P ) := -ij p ij log p ij is the entropy of P , ⟨•, •⟩
is Frobenius inner product and ϵ is a regularization coefficient. The optimal solution P * is unique and can be found by forming the r × r similarity matrix e -M/ϵ and then applying the Sinkhorn-Knopp algorithm [START_REF] Philip | The Sinkhorn-Knopp algorithm: convergence and applications[END_REF], i.e., iteratively normalizing rows and columns. A small ϵ leads to sparser P , which improves one-to-one matching but makes the optimization harder [START_REF] Alvarez | Gromovwasserstein alignment of word embedding spaces[END_REF], while a large ϵ leads to denser P , causing more correspondences and poor matching.

Interpolation The assignment matrix R := rP * is a doubly stochastic r × r matrix whose element r ij expresses the probability that column a i of A corresponds to column a ′ j of A ′ . Thus, we align A and A ′ as follows:

A := A ′ R ⊤ (9)
A ′ := AR.

Here, column a i of c × r matrix A is a convex combination of columns of A ′ that corresponds to the same column a i of A. We reshape A back to c × w × h tensor A by expanding spatial dimensions and we say that A represents A aligned to A ′ . We then interpolate between A and the original feature tensor A:

mix λ (A, A). (11)
As shown in Figure 2 (toy example, top right), A is geometrically close to A. The correspondence with A ′ and the geometric proximity to A makes A appropriate for interpolation with A. Symmetrically, we can also align A ′ to A and interpolate between A ′ and A ′ :

mix λ (A ′ , A ′ ). (12)
When mixing feature tensors with alignment (4), we define Mix λ in (2) as the mapping of (A, A ′ ) to either [START_REF] Cuturi | Sinkhorn distances: lightspeed computation of optimal transport[END_REF] or ( 12), chosen at random. Assignment matrix R is obtained by Sinkhorn-Knopp [START_REF] Philip | The Sinkhorn-Knopp algorithm: convergence and applications[END_REF] on similarity matrix e -M/ϵ . A is aligned to A ′ according to R, giving rise to A. We then interpolate between A, A. Symmetrically, we can align A ′ to A and interpolate between A ′ , A ′ . A, A ′ on the left (toy example of 16 points in 2D) shown semi-transparent on the right for reference.

x x ′ 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 λ mix λ (A, A ′ ) mix λ (A, A) mix λ (A ′ , A ′ ) (a) (b) mix λ (A, A ′ ) mix λ (A, A) mix λ (A ′ , A ′ ) (c) (d)

Visualization and discussion

Decoder We use a decoder to study images generated with or without feature alignment. Let f : R c×w×h → R d be a FC layer mapping tensor A to embedding e = f (A).

We use f • F as an encoder and a decoder D : R d → X mapping e back to the image space, reconstructing image x = D(e). The autoencoder is trained using only clean images (without mixup) using reconstruction loss L r between x and x, where L r (x, x ′ ) := ∥x -x ′ ∥ 2 is the squared Euclidean distance. We use generated images only for visualization purposes below, but we also use the decoder optionally during AlignMixup training in section 4.

Discussion For different λ ∈ [0, 1], we interpolate the feature tensors A, A ′ of x, x ′ without or with alignment, using [START_REF] Cuturi | Sinkhorn distances: lightspeed computation of optimal transport[END_REF] or [START_REF] Dabouei | Supermix: Supervising the mixing data augmentation[END_REF], and we generate a new image by decoding the resulting embedding through the decoder D.

In Figure 3, we visualize such generated images. Interestingly, by aligning A to A ′ and mixing using [START_REF] Cuturi | Sinkhorn distances: lightspeed computation of optimal transport[END_REF] with λ = 0, the generated image retains the pose of x and the texture of x ′ . In Figure 3(a) in particular, when x is 'penguin' and x ′ is 'dog', the generated image retains the pose of the penguin, while the texture of the dog aligns to the body of the penguin. Similarly, in Figure 3(c), the texture from the goldfish is aligned to that of the stork, while the pose of the stork is retained. Vice versa, as shown in Figure 3(b,d), by aligning A ′ to A and mixing using [START_REF] Dabouei | Supermix: Supervising the mixing data augmentation[END_REF] with λ = 0, the generated image retains the pose of x ′ and the texture of x. By contrast, the image generated from unaligned features appears to be an overlay.

Randomly sampling several values of λ ∈ [0, 1] during training generates an abundance of samples, capturing texture from one image and the pose from another. This allows the model to explore beyond the image manifold, thereby improving its generalization and enhancing its performance across multiple benchmarks, as discussed in section 4.

Experiments

Implementation details

Architecture We use a residual network as our encoder F . The output A is a c × 4 × 4 tensor. This is followed by a fully-connected layer as classifier g.

Autoencoder In Figure 3, we have used a decoder to visualize the effect of feature tensor alignment. In our experiments, we also use a decoder optionally during training of AlignMixup, to investigate its effect on representation learning under mixup. This results in a vanilla autoencoder architecture, which we denote as AlignMixup/AE. We use a residual generator [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF] as the decoder D. The encoder and decoder have the same architecture.

Training We train AlignMixup using only the classification loss L c (5) on mixed examples. For a given mini-batch during training, we mix either x or A (using either [START_REF] Cuturi | Sinkhorn distances: lightspeed computation of optimal transport[END_REF] or [START_REF] Dabouei | Supermix: Supervising the mixing data augmentation[END_REF] with alignment). We choose between the three cases uniformly at random. For AlignMixup/AE, we either use the reconstruction loss L r on clean examples, training the encoder and decoder, or the classification loss L c (5) on mixed examples, training the encoder and classifier. This gives rise to a fourth case and we choose uniformly at random. The algorithm is in the supplementary material.

Hyperparameters The hyperparameters used for different datasets are reported in the supplementary material.

Image classification and robustness

We use PreActResnet18 [START_REF] He | Deep residual learning for image recognition[END_REF] (R-18) and WRN16-8 [START_REF] Zagoruyko | Wide residual networks[END_REF] as the backbone architecture on CIFAR-10 and CIFAR-100 datasets [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF]. Using the experimental settings of Man- ifold mixup [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF] (in supplementary material), we reproduce the state-of-the-art (SOTA) mixup methods: baseline network (without mixup), Input mixup [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF], Manifold mixup [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF], CutMix [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF], PuzzleMix [START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF], Co-Mixup [31], SaliencyMix [57], StyleMix [START_REF] Hong | Stylemix: Separating content and style for enhanced data augmentation[END_REF] and StyleCutMix [START_REF] Hong | Stylemix: Separating content and style for enhanced data augmentation[END_REF] using official code provided by the authors. We do not compare AlignMixup with AutoMix [START_REF] Zhu | Automix: Mixup networks for sample interpolation via cooperative barycenter learning[END_REF] and Re-Mix [START_REF] Cao | Remix: Towards image-to-image translation with limited data[END_REF], since its experimental settings are different from ours and there is no available code.

In addition, we use R-18 as the backbone network on TinyImagenet [START_REF] Yao | Tiny imagenet classification with convolutional neural networks[END_REF] (TI) and reproduce SaliencyMix [57], StyleMix [START_REF] Hong | Stylemix: Separating content and style for enhanced data augmentation[END_REF] and StyleCutMix [START_REF] Hong | Stylemix: Separating content and style for enhanced data augmentation[END_REF] following the experimental settings of [START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF], and Resnet-50 (R-50) on Ima-geNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], following the training protocol of [START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF]. Using top-1 error (%) as evaluation metric, we show the effectiveness of AlignMixup on image classification and robustness to FGSM [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] and PGD [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF] attacks. As shown in Table 3, AlignMixup is more robust comparing to SOTA methods. While AlignMixup is on par with PuzzleMix and Co-Mixup on CIFAR-10 image classification, it outperforms Co-Mixup and PuzzleMix by 5.36% and 2.28% in terms of robustness to FGSM attacks. There is also significant gain of robustness to FGSM on Tiny-ImageNet and to the stronger PGD on CIFAR-100.

Image classification As shown in

Overconfidence

Deep neural networks tend to be overconfident about incorrect predictions far away from the training data and mixup helps combat this problem. Two standard benchmarks to evaluate this improvement are their ability to detect out-of-distribution data and their calibration, i.e., the discrepancy between accuracy and confidence.

Out-of-distribution detection According to [START_REF] Hendrycks | A baseline for detecting misclassified and out-of-distribution examples in neural networks[END_REF], indistribution (ID) refers to a test example drawn from the same distribution which the network is trained on, while a sample drawn from any other distribution is out-ofdistribution (OOD). At inference, given a mixture of ID and OOD examples, the network assigns probabilities to the known classes by softmax. An example is then classified as OOD if the maximum class probability is below a certain threshold, else ID. A well-calibrated network should be able to assign a higher probability to ID than OOD examples, making it easier to distinguish the two distributions.

We compare AlignMixup with SOTA methods trained using R-18 on CIFAR-100 as discussed in subsection 4.2. At inference, ID examples are test images from CIFAR-100, while OOD examples are test images from LSUN (crop) [START_REF] Yu | Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop[END_REF], iSUN [START_REF] Xiao | Sun database: Large-scale scene recognition from abbey to zoo[END_REF] and Tiny-ImageNet (crop); where crop denotes that the OOD examples are center-cropped to 32 × 32 to match the resolution of ID images [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF]. Following [START_REF] Hendrycks | A baseline for detecting misclassified and out-of-distribution examples in neural networks[END_REF], we measure detection accuracy (Det Acc) using a threshold of 0.5, area under ROC curve (AuROC) and area under precision-recall curve (AuPR).

As shown in Table 4, AlignMixup outperforms SOTA methods under all metrics by a large margin, indicating that it is better in reducing over-confident predictions. We further observe that Input mixup is inferior to Baseline, which is consistent with the findings of [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF]. More results are given in the supplementary material.

Calibration According to [START_REF] Morris | The comparison and evaluation of forecasters[END_REF], calibration measures the discrepancy between the accuracy and confidence level of a network's predictions. A poorly calibrated network may make incorrect predictions with high confidence. In the supplementary, we compare AlignMixup with SOTA methods using calibration plots and quantitative experiments.

Weakly-supervised object localization (WSOL)

WSOL aims to localize an object of interest using only class labels without bounding boxes at training. WSOL works by extracting visually discriminative cues to guide the classifier to focus on salient regions in the image.

We train AlignMixup using the same procedure as for image classification. At inference, following [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF], we compute a saliency map using CAM [START_REF] Zhou | Learning deep features for discriminative localization[END_REF], binarize it using a threshold of 0.15 and take the bounding box of the mask. We use VGG-GAP [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] and Resnet-50 [START_REF] He | Deep residual learning for image recognition[END_REF] as pretrained on Imagenet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] and we fine-tune them on CUB200-2011 [START_REF] Wah | The caltech-ucsd birds-200-2011 dataset[END_REF]. We follow the evaluation protocol by [START_REF] Choe | Evaluating weakly supervised object localization methods right[END_REF] and use top-1 localization accuracy with IoU threshold of 0.5 and Maximal Box Accuracy (MaxBoxAcc-v2) to compare AlignMixup with baseline CAM (without mixup), Input mixup [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF], CutOut [START_REF] Devries | Improved regularization of convolutional neural networks with cutout[END_REF] and CutMix [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF].

According to Table 5, AlignMixup outperforms Input mixup, CutOut and CutMix by 11.4%, 7.3% and 0.6% respectively using VGG-GAP and by 6.9%, 3.8% and 1.4% respectively using Resnet-50 in terms of top-1 localization accuracy. Furthermore, AlignMixup outperforms CutMix by 1.2% and 0.6% using VGG-GAP and Resnet-50 respectively in terms of MaxBoxAcc-v2. It also outperforms dedicated WSOL methods ACoL [START_REF] Zhang | Adversarial complementary learning for weakly supervised object localization[END_REF] and ADL [START_REF] Choe | Attention-based dropout layer for weakly supervised object localization[END_REF], which focus on learning spatially dispersed representations. Qualitative localization results are given in the supplementary material.

Ablation study

All ablations are performed on CIFAR-100 using R-18 as encoder F with feature tensor A being 512×4×4. We study the effect of mixing at different layers (x, A), by aligning A or not before mixing, as well as using a decoder D in an different autoencoder architectures. We report top-1 accu- Table 6. Ablations using R-18 on CIFAR-100. Top-1 classification accuracy (%): higher is better. Arch: autoencoder architecture. AE: vanilla; VAE: variational [START_REF] Diederik | Auto-encoding variational bayes[END_REF].

racy (%). All results are in Table 6. The ablation showing the effect of the number of iterations in Sinkhorn-Knopp algorithm is summarized in the supplementary material.

Layers We study the choice of layers to mix, regardless of feature alignment. According to (2), we may mix at any of two layers, represented by {x, A}. To investigate more diverse cases, we introduce an additional layer f to the encoder of AlignMixup and mix its output, which acts as the latent space of AlignMixup/AE. f could be a FC layer, which outputs a vector e ∈ R 512 , or a convolutional layer of kernel size 2 × 2 and stride 2, producing a 128 × 2 × 2 tensor E. In both cases, mixup is also represented by [START_REF] Beckham | On adversarial mixup resynthesis[END_REF], where now f 1 := f • F and f 2 := id. Mixing is now chosen from {x, A, e} or {x, A, E}. In AlignMixup (no decoder), among different choices of unaligned layer sets, mixing {x, e} results in the highest classification accuracy. Furthermore, AlignMixup/AE outperforms baseline and the best performing competitor StyleCutMix for all choices of layers, even when features are unaligned, showing a motivation to use the decoder.

Tensor alignment We investigate the effect of aligning feature tensors or not before mixing it, by using standard mixup (2) or ( 11), [START_REF] Dabouei | Supermix: Supervising the mixing data augmentation[END_REF], respectively. It is important to note that when e is a vector, we do not align it. In AlignMixup, we observe that aligning tensors A and E before mixing improves classification accuracy significantly. Furthermore, we observe that using an additional FC layer for e brings only minor improvement (81.

Conclusion

We have shown that mixup of a combination of input and latent representations is a simple and very effective pairwise data augmentation method. The gain is most prominent on large datasets and in combating overconfidence in predictions, as indicated by out-of-distribution detection. Interpolation of feature tensors boosts performance significantly, but only if they are aligned.

Our work is a compromise between a "good" handcrafted interpolation in the image space and a fully learned one in the latent space. A challenge is to make progress in the latter direction without compromising speed and simplicity, which would affect wide applicability.

Acknowledgement

This work was in part supported by the ANR-19-CE23-0028 MEERQAT project and was performed using the HPC resources from GENCI-IDRIS Grant 2021 AD011012528. This work was partially done while Yannis was at Inria. We also thank Konstantinos Tertikas for his amazing help with adapting AlignMixup to transformers.

A. Algorithm

AlignMixup and AlignMixup/AE are summarized in algorithm 1. By default (AlignMixup), for each mini-batch, we uniformly draw at random one among three choices (line 2) over mixup on input (x) or feature tensors (A, using either [START_REF] Cuturi | Sinkhorn distances: lightspeed computation of optimal transport[END_REF] or [START_REF] Dabouei | Supermix: Supervising the mixing data augmentation[END_REF] for mixing). For AlignMixup/AE, there is a fourth choice where we only use reconstruction loss on clean examples (line 7).

For mixup, we use only classification loss (5) (line 24). Following [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF], we form, for each example (x, y) in the mini-batch, a paired example (x ′ , y ′ ) from the same minibatch regardless of class labels, by randomly permuting the indices (lines 1,10). Inputs x, x ′ are mixed by ( 2),(3) (line 12). Feature tensors A and A ′ are first aligned and then mixed by ( 2),(11) (A aligns to A ′ ) or ( 2),( 12) (A ′ aligns to A) (lines 14,23).

out ← F (mix λ (x, x ′ )) ▷ (2),(3) else ▷ mode ∈ {feat, feat ′ } 13 if mode = feat ′ then ▷ choose (12) over (11) 14 SWAP (x, x ′ ), SWAP (y, y ′ ) 15 A ← F (x) , A ′ ← F (x ′ ) ▷ feature tensors 16 A ← RESHAPE c×r (A) ▷ to matrix 17 A ′ ← RESHAPE c×r (A ′ ) 18 M ← DIST(A, A ′ ) ▷ pairwise distances (6) 19 P * ← SINKHORN(exp(-M/ϵ)) ▷ tran. plan (8) 20 R ← DETACH(rP * ) ▷ assignments 21 A ← A ′ R ⊤ ▷ alignment (9) 22 A ← RESHAPE c×w×h ( A) ▷ to tensor 23 out ← f (mix λ (A, A)) ▷ (2),( 11
)
24 ℓi ← Lc(g(out), mix λ (y, y ′ )) ▷ classification loss [START_REF] Cao | Remix: Towards image-to-image translation with limited data[END_REF] In computing loss derivatives, we backpropagate through feature tensors A, A ′ but not through the transport plan P * (line 20). Hence, although the Sinkhorn-Knopp algorithm [START_REF] Philip | The Sinkhorn-Knopp algorithm: convergence and applications[END_REF] is differentiable, its iterations take place only in the forward pass. Importantly, AlignMixup is easy to implement and does not require sophisticated optimization like [31,[START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF].

B. Hyperparameter settings

CIFAR-10/CIFAR-100 We train AlignMixup using SGD for 2000 epochs with an initial learning rate of 0.1, decayed by a factor 0.1 every 500 epochs. We set the momentum as 0.9 with a weight decay of 0.0001 and use a batch size of 128. The interpolation factor is drawn from Beta(α, α) where α = 2.0. Using these settings, we reproduce the results of SOTA mixup methods for image classification, robustness to FGSM and PGD attacks, calibration and out-of-distribution detection. For alignment, we apply the Sinkhorn-Knopp algorithm [START_REF] Philip | The Sinkhorn-Knopp algorithm: convergence and applications[END_REF] for 100 iterations with entropic regularization coefficient ϵ = 0.1.

TinyImagenet We follow the training protocol of Kim et al. [START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF], training R-18 as stage-1 encoder F using SGD for 1200 epochs. We set the initial learning rate to 0.1 and decay it by 0.1 at 600 and 900 epochs. We set the momentum as 0.9 with a weight decay of 0.0001 and use a batch size of 128 on 2 GPUs. The interpolation factor is drawn from Beta(α, α) where α = 2.0. For alignment, we apply the Sinkhorn-Knopp algorithm [START_REF] Philip | The Sinkhorn-Knopp algorithm: convergence and applications[END_REF] for 100 iterations with entropic regularization coefficient ϵ = 0.1.

ImageNet We follow the training protocol of Kim et al. [START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF], where training R-50 as F using SGD for 300 epochs. The initial learning rate of the classifier and the remaining layers is set to 0.1 and 0.01, respectively. We decay the learning rate by 0.1 at 100 and 200 epochs. We set the momentum as 0.9 with a weight decay of 0.0001 and use a batch size of 100 on 4 GPUs. The interpolation factor is drawn from Beta(α, α) where α = 2.0. For alignment, we apply the Sinkhorn-Knopp algorithm [START_REF] Philip | The Sinkhorn-Knopp algorithm: convergence and applications[END_REF] for 100 iterations with entropic regularization coefficient ϵ = 0.1.

We also train R-50 on ImageNet for 100 epochs, following the training protocol described in Kim et al. [31].

CUB200-2011

For weakly-supervised object localization (WSOL), we use VGG-GAP and R-50 pretrained on Ima-geNet as F . The training strategy for WSOL is the same as image classification and the network is trained without bounding box information. In R-50, following [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF], we modify the last residual block (layer 4) to have stride 2 instead of 1, resulting in a feature map of spatial resolution 14 × 14. The modified architecture of VGG-GAP is the same as described in [START_REF] Zhou | Learning deep features for discriminative localization[END_REF]. The classifier is modified to have 200 classes instead of 1000.

For fair comparisons with [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF], during training, we resize the input image to 256 × 256 and randomly crop the resized image to 224 × 224. During testing, we directly re-NETWORK RESNET-50 Baseline 24.03 Input [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF] 22.97 Manifold [START_REF] Verma | Manifold mixup: Better representations by interpolating hidden states[END_REF] 23.30 CutMix [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF] 22.92 PuzzleMix [START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF] 22.49 Co-Mixup [31] 22.39 StyleMix [START_REF] Hong | Stylemix: Separating content and style for enhanced data augmentation[END_REF] 24.06 StyleCutMix [START_REF] Hong | Stylemix: Separating content and style for enhanced data augmentation[END_REF] 22.71 AlignMixup (ours) 22.0 Gain +0.39 size to 224×224. We train the network for 600 epochs using SGD. For R-50, the initial learning rate of the classifier and the remaining layers is set to 0.01 and 0.001, respectively. For VGG, the initial learning rate of the classifier and the remaining layers is set to 0.001 and 0.0001, respectively. We decay the learning rate by 0.1 every 150 epochs. The momentum is set to 0.9 with weight decay of 0.0001 and batch size of 16.

C. Additional experiments

Figure 1 .

 1 Figure 1. Different mixup methods. AlignMixup retains the pose of image 2 and the texture of image 1. This different from overlay (Input and Manifold mixup) or combination of two objects (Cut-Mix). Manifold mixup and AlignMixup visualized by a decoder (subsection 3.3) that is not used at training.

Figure 2 .

 2 Figure 2. Feature tensor alignment and interpolation. Cost matrix M contains pairwise distances of feature vectors in tensors A, A ′ .Assignment matrix R is obtained by Sinkhorn-Knopp[START_REF] Philip | The Sinkhorn-Knopp algorithm: convergence and applications[END_REF] on similarity matrix e -M/ϵ . A is aligned to A ′ according to R, giving rise to A. We then interpolate between A, A. Symmetrically, we can align A ′ to A and interpolate between A ′ , A ′ . A, A ′ on the left (toy example of 16 points in 2D) shown semi-transparent on the right for reference.

Figure 3 .

 3 Figure 3. Visualizing alignment. For different λ ∈ [0, 1], we interpolate feature tensors A, A ′ without alignment (top) or aligned feature tensors (bottom) of two images x, x ′ and then we generate a new image by decoding the resulting embedding through the decoder D. (a), (c) We align A to A ′ and mix with (11). (b), (d) We align A ′ to A and mix with (12). Only meant for illustration: No decoded images are seen by the classifier at training.

Algorithm 1 :reconstruction loss 8 else ▷ mixup 9 λ

 189 AlignMixup/AE (parts involved in the AE variant indicated in blue) Input: encoders F ; embedding e, decoder D; classifier g Input: mini-batch B := {(xi, yi)} b i=1 Output: loss values L := {ℓi} b i=1 1 π ∼ unif(S b ) ▷ random permutation of {1, . . . , b} 2 mode ∼ unif{clean, input, feat, feat ′ } ▷ mixup? 3 for i ∈ {1, . . . , b} do 4 (x, y) ← (xi, yi) ▷ current example 5 if mode = clean then ▷ no mixup 6 x ← D(e(F (x))) ▷ encode/decode 7 ℓi ← Lr(x, x) ▷ ∼ Beta(α, α) ▷ interpolation factor 10 (x ′ , y ′ ) ← (x π(i) , y π(i) ) ▷ paired example 11if mode = input then ▷ as in[START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF]

12

 12

Table 1 .

 1 Image classification top-1 error (%) on CIFAR-10/100 and TI (TinyImagenet). Top-1 error (%): lower is better. Blue: second best. R: PreActResnet, W: WRN. * : reported by [31].

	DATASET	CIFAR-10	CIFAR-100	TI
	NETWORK	R-18 W16-8 R-18 W16-8 R-18
	Baseline	5.19	5.11 23.24 20.63 43.40 *
	Input [69]	4.03	3.98 20.21 19.88 43.48 *
	CutMix [65]	3.27	3.54 19.37 19.71 43.11 *
	Manifold [58]	2.95	3.56 19.80 19.23 40.76 *
	PuzzleMix [32]	2.93	2.99 20.01 19.25 36.52 *
	Co-Mixup [31]	2.89	3.04 19.81 19.57 35.85 *
	SaliencyMix [57]	2.99	3.53 19.69 19.59 34.81
	StyleMix [28]	3.76	3.89 20.04 20.45 36.13
	StyleCutMix [28]	3.06	3.12 19.34 19.28 34.49
	AlignMixup (ours)	2.95	3.09 18.29 18.77 33.13
	AlignMixup/AE (ours) 2.83	3.15 17.82 18.09 32.73
	Gain	+0.06 -0.10 +1.52 +1.14 +1.76

Table 2 .

 2 Image classification top-1 error (%) and computational analysis on ImageNet using Resnet-50 for 300 epochs. Lower is better. Blue: second best. * : reported by authors; † : reported by PuzzleMix.

Table 1 ,

 1 AlignMixupand AlignMixup/AE is on par or outperforms the SOTA methods by achieving the lowest top-1 error, especially on large datasets. On CIFAR-10, AlignMixup and Align-Mixup/AE is on par with Co-Mixup and Puzzlemix with R-18 and WRN16-8. On CIFAR-100, AlignMixup outperforms StyleCutMix and Manifold mixup by 1.05% and 0.46% with R-18 and WRN16-8, respectively. On TI, AlignMixup outperforms Co-Mixup by 2.72% using R-18. From Table2, AlignMixup/AE outperforms PuzzleMix by 2.41% on ImageNet. While the overall improvement by SOTA methods on ImageNet over Baseline is around 2%, AlignMixup/AE improves SOTA by another 2.5%.Computational complexityTable 2 shows the computational analysis of AlignMixup training as compared with baseline and SOTA mixup methods on ImageNet, in terms of number of parameters and msec/batch on a NVIDIA RTX

	ATTACK			FGSM				PGD
	DATASET	CIFAR-10	CIFAR-100	TI	CIFAR-10	CIFAR-100
	NETWORK	R-18	W16-8	R-18	W16-8	R-18	R-18	W16-8	R-18	W16-8
	Baseline	89.41	88.02	87.12	72.81	91.85	99.99	99.94	99.97	99.99
	Input [69]	78.42	79.21	81.30	67.33	88.68	99.77	99.43	99.96	99.37
	CutMix [65]	77.72	78.33	86.96	60.16	88.68	99.82	98.10	98.67	97.98
	Manifold [58]	77.63	76.11	80.29	56.45	89.25	97.22	98.49	99.66	98.43
	PuzzleMix [32]	57.11	60.73	78.70	57.77	83.91	97.73	97.00	96.42	95.28
	Co-Mixup [31]	60.19	58.93	77.61	56.59	-	97.59	96.19	95.35	94.23
	SaliencyMix [57]	57.43	68.10	77.79	58.10	81.16	97.51	97.04	95.68	93.76
	StyleMix [28]	79.54	71.05	80.54	67.94	84.93	98.23	97.46	98.39	98.24
	StyleCutMix [28]	58.79	56.12	77.49	56.83	80.59	97.87	96.70	91.88	93.78
	AlignMixup (ours)	54.83	56.20	74.18	55.05	78.83	95.42	96.71	90.40	92.16
	AlignMixup/AE (ours)	52.13	54.86	76.40	55.44	78.98	97.16	95.32	91.69	92.23
	Gain	+4.98	+1.26	+3.31	+1.40	+1.76	+1.80	+0.87	+1.48	+1.60

Table 3 .

 3 Robustness to FGSM & PGD attacks. Top-1 error (%): lower is better. Blue: second best. Gain: reduction of error. TI: TinyImagenet. R: PreActResnet, W: WRN.

	Challenges From Table 1, we observe that AlignMixup
	achieves SoTA top-1 error on CIFAR-10 and CIFAR-100.
	These results are computed using 2000 epochs following
	the experimental settings of [58], which also achieves its
	best performance at 2000 epochs. While baseline mixup
	methods [28, 31, 32, 57, 65, 69] perform best at 300 epochs,
	they do not benefit from long training time. Unlike these
	methods, which perform mixup in the image space, Man-
	ifold mixup [58] and AlignMixup performs mixup in the
	feature space. We hypothesize that this takes longer train-
	ing time until the network learns some meaningful repre-
	sentations. It is even more challenging in our case, since
	we mix features at deeper layers comparing with Manifold
	mixup. Empirically, when trained for 2000 epochs instead
	of 300 epochs, the top-1 error drops from 21.64 → 19.80 for
	Manifold mixup and from 21.38 → 18.29 for AlignMixup.
	Robustness to FGSM and PGD attacks Following the
	evaluation protocol of [32], we use 8/255 l

2080 TI GPU. AlignMixup has nearly the same computational overhead as Manifold mixup while achieving 1.82% increase of accuracy. While SOTA methods like Co-Mixup and PuzzleMix are computationally more expensive than AlignMixup by 1.8× and 2.3× respectively, they are outperformed by AlignMixup by 0.6% on average. Align-Mixup/AE brings a further 1.85% gain in accuracy over AlignMixup. It is important to note that 40% increase in number of parameters of AlignMixup/AE is due to the residual decoder, which is only used in one out of five cases on clean images without mixup. Computational complexity during inference is the same for all methods. ∞ ϵ-ball for FGSM and 4/255 l ∞ ϵ-ball with step size 2/255 for PGD. We reproduce the results of competitors for FGSM and PGD on CIFAR-10 and CIFAR-100; results of baseline, Input, Manifold, Cutmix and Puzzlemix on TI for FGSM are as reported in

[START_REF] Kim | Puzzle mix: Exploiting saliency and local statistics for optimal mixup[END_REF]

and reproduced for SaliencyMix, StyleMix and StyleCutMix.

Table 4 .

 4 Out-of-distribution detection using PreActResnet18. Det Acc (detection accuracy), AuROC, AuPR (ID) and AuPR (OOD): higher is better; Blue: second best. Gain: increase in performance. TI: TinyImagenet. Additional results are in the supplementary material.

	TASK					OUT-OF-DISTRIBUTION DETECTION
	DATASET		LSUN (CROP)			ISUN		TI (CROP)
	METRIC	DET AUROC AUPR AUPR DET AUROC AUPR AUPR DET AUROC AUPR AUPR ACC (ID) (OOD) ACC (ID) (OOD) ACC (ID) (OOD)
	Baseline	54.0	47.1	54.5	45.6 66.5	72.3	74.5	69.2 61.2	64.8	67.8	60.6
	Input [69]	57.5	59.3	61.4	55.2 59.6	63.0	60.2	63.4 58.7	62.8	63.0	62.1
	Cutmix [65]	63.8	63.1	61.9	63.4 67.0	76.3	81.0	77.7 70.4	84.3	87.1	80.6
	Manifold [58]	58.9	60.3	57.8	59.5 64.7	73.1	80.7	76.0 67.4	69.9	69.3	70.5
	PuzzleMix [32]	64.3	69.1	80.6	73.7 73.9	77.2	79.3	71.1 71.8	76.2	78.2	81.9
	Co-Mixup [31]	70.4	75.6	82.3	70.3 68.6	80.1	82.5	75.4 71.5	84.8	86.1	80.5
	SaliencyMix [57]	68.5	79.7	82.2	64.4 65.6	76.9	78.3	79.8 73.3	83.7	87.0	82.0
	StyleMix [28]	62.3	64.2	70.9	63.9 61.6	68.4	67.6	60.3 67.8	73.9	71.5	78.4
	StyleCutMix [28]	70.8	78.6	83.7	74.9 70.6	82.4	83.7	76.5 75.3	82.6	82.9	78.4
	AlignMixup (ours)	74.2	79.9	84.1	75.1 72.8	83.2	84.1	80.3 77.2	85.0	87.8	85.0
	AlignMixup/AE (ours) 76.9	83.5	86.7	79.4 75.6	84.1	85.9	81.7 79.7	88.0	89.7	85.7
	Gain		+6.1	+3.8	+3.0	+4.5 +1.7	+1.7	+2.2	+1.9 +4.4	+3.2	+2.6	+3.8
	METRIC	TOP-1 LOC.		MAXBOXACC-V2
	NETWORK	VGG-GAP RESNET-50 VGG-GAP RESNET-50
	ACoL [70]	45.9	-		57.4	-
	ADL [8]	52.4	-		61.3	58.4
	Baseline CAM [71]	37.1	49.4		59.0	59.7
	Input [69]	41.7	49.3		57.1	60.6
	CutMix [65]	52.5	54.8		62.6	64.8
	AlignMixup (ours)	53.1	56.2		63.8	65.4
	Gain	+0.6	+1.4		+1.2	+0.6

Table 5 .

 5 Weakly-supervised object localization on CUB200-2011.

Top-1 loc.: Top-1 localization accuracy (%), MaxBoxAcc-v2: Maximal box accuracy

[START_REF] Choe | Evaluating weakly supervised object localization methods right[END_REF]

. Higher is better. Blue: second best. Gain: increase of accuracy.

  71 → 81.92), meaning that the major improvement comes from alignment. Overall, Align-Mixup/AE works the best when x, A, e are mixed, with A being aligned. It outperforms StyleCutMix by 1.52%.Alignment resolution Given the best settings of Align-Mixup/AE, we investigate the effect of aligning A at different spatial resolutions. The default is 4 × 4, denoted as A 4×4 . We also experiment 2 × 2 (A 2×2 ), obtained by average pooling, and 8 × 8 (A 8×8 ), by removing downsampling from the last convolutional layer. The accuracy of 8 × 8 is only slightly better than 4 × 4 by 0.02%, while being computationally more expensive. Thus, we choose 4 × 4 as the default. By contrast, aligning at 2 × 2 is worse than not aligning at all. This may be due to soft correspondences causing loss of information by averaging. ∈ R 512 instead of e, representing mean and standard deviation, respectively. We also investigate 128 × 2 × 2 tensors, denoted as M, Σ where the two variables are mixed simultaneously. As for AlignMixup and AlignMixup/AE, we investigate different combinations of layers with or without alignment. All three architectures work best when x, A, e are mixed. Alignment improves consistently on all three architectures. Both AlignMixup and AlignMixup/VAE are inferior to AlignMixup/AE. However, their best settings still outperform Baseline and StyleCutMix.

	Autoencoder architecture We compare AlignMixup
	with two autoencoder architectures: the vanilla au-
	toencoder (AlignMixup/AE), and a variational autoen-
	coder [33] (AlignMixup/VAE). The latter has two vectors
	µ, σ

Table 7 .

 7 Image classification on ImageNet for 100 epochs using ResNet-50. Top-1 error (%): lower is better. Blue: second best. Gain: reduction of error.

  ImageNet classification Following the training protocol of [31], Table7reports classification performance when training for 100 epochs on ImageNet. Using the top-1 error (%) reported for competitors by[31], AlignMixup outperforms all methods, including Co-Mixup[31]. Importantly, while the overall improvement by SOTA methods over Baseline is around 1.64%, AlignMixup improves SOTA by another 0.4%.Experiments using transformersWe apply mixup to LeViT-128S[START_REF] Graham | Levit: A vision transformer in convnet's clothing for faster inference[END_REF] on ImageNet for 100 epochs. For Align-Mixup, we align the feature tensors in the last layer of the convolution stem. The top-1 accuracy is: baseline 67.4%, input mixup 68.3%, manifold mixup 67.8%, Cut-Mix 68.7%, AlignMixup 69.9%. Thus, we outperform input mixup and CutMix by 1.6% and 1.2% respectively, which in turn outperform the baseline by 0.9% and 1.3% respectively. This means that the improvement brought by mixing is roughly doubled.Out-of-distribution detectionWe compare AlignMixup with SOTA methods, training R-18 on CIFAR-100 as discussed in subsection 4.2. At inference, ID examples are test images from CIFAR-100, while OOD examples are test images from LSUN[START_REF] Yu | Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop[END_REF] and Tiny-ImageNet, resizing OOD examples to 32 × 32 to match the resolution of ID images[START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF]. We also use test images from CIFAR-100 with

	DATASET	LSUN (RESIZE)	TI (RESIZE)
	METRIC	DET AU AUPR AUPR ACC ROC (ID) (OOD)	DET AU AUPR AUPR ACC ROC (ID) (OOD)

available at https://github.com/shashankvkt/ AlignMixup_CVPR22.git

Uniform and Gaussian noise as OOD samples. Uniform is drawn from U(0, 1) and Gaussian from N (µ, σ) with µ = σ = 0.5. All SOTA mixup methods are reproduced using the same experimental settings. Following [START_REF] Hendrycks | A baseline for detecting misclassified and out-of-distribution examples in neural networks[END_REF], we measure detection accuracy (Det Acc) using a threshold of 0.5, area under ROC curve (AuROC) and area under precision-recall curve (AuPR).

As shown in Table 8, AlignMixup outperforms SOTA methods under all metrics by a large margin, indicating that it is better in reducing over-confident predictions.

Calibration We compare AlignMixup with SOTA methods , training R-18 on CIFAR-100 as discussed in subsection 4.2. All SOTA mixup methods are reproduced using the same experimental settings. We compare qualitatively by plotting accuracy vs. confidence. As shown in Figure 4, while Baseline is clearly overconfident and Input and Manifold mixup are clearly under-confident, AlignMixup results in the best calibration among all competitors. We also compare quantitatively, measuring the expected calibration error (ECE) [START_REF] Guo | On calibration of modern neural networks[END_REF] and overconfidence error (OE) [START_REF] Sunil Thulasidasan | On mixup training: Improved calibration and predictive uncertainty for deep neural networks[END_REF]. As shown in Table 9, AlignMixup outperforms SOTA methods by achieveing lower ECE and OE, indicating that it is better calibrated.

. Calibration plots on CIFAR-100 using PreActResnet18: near diagonal is better. We plot accuracy vs. confidence, that is, probability for the predicted class. Qualitative results of WSOL Qualitative localization results shown in Figure 5 indicate that AlignMixup encodes semantically discriminative representations, resulting in better localization performance.

Object detection Following the settings of CutMix [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF], we use Resnet-50 pretrained on ImageNet using Align-Mixup as the backbone of SSD [START_REF] Liu | Ssd: Single shot multibox detector[END_REF] and Faster R-CNN [START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] detectors and fine-tune it on Pascal VOC07 [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] and MS-COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] Table 10. Ablation of the number of iterations in Sinkhorn-Knopp algorithm using R-18 on CIFAR-100. Top-1 classification accuracy(%): higher is better.

Iterations in Sinkhorn-Knopp

The default number of iterations for the Sinkhorn-Knopp algorithm in solving ( 8) is i = 100. Here, we investigate more choices, as shown in Table 10. The case of i = 0 is similar to crossattention. In this case, we only normalize either the rows or columns in [START_REF] Choe | Evaluating weakly supervised object localization methods right[END_REF] once, such that P 1 = 1/r (when A aligned to A ′ ) or P ⊤ 1 = 1/r (when A ′ aligned to A). We observe that while AlignMixup outperforms the best baseline-StyleCutMix (80.66)-in all cases, it performs best for i = 100 iterations.


Félix Vanneste

Olivier Goury

Christian Duriez

Calibration method for soft robots modeled with FEM: application to anisotropy

Keywords: Calibration and Identification, Optimization and Optimal Control, Modeling, Control, and Learning for Soft Robots, Soft Robot Materials and Design

HAL is

I. INTRODUCTION

S IM2REAL is one of the key challenge in robotics, and perhaps even more so in soft robotics. By having an accurate physical model of a robot, robust control methods can be derived, but also the robot can be simulated in its operating environment to plan and learn automatically [START_REF] Kuindersma | An efficiently solvable quadratic program for stabilizing dynamic locomotion[END_REF] [START_REF] Choi | On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward[END_REF]. In soft robotics, the challenge of having an accurate physical model is even more important because it must take into account deformations, which are derived from sometimes complex material properties. Being able to precisely calibrate these models by optimising the material parameters is certainly a step forward towards better sim2real transitions in soft robotics.

In the community, the question of using differentiable models has had recent interest since it allows gradient-based optimisation to improve accuracy, in particular to fit real data. This was especially investigated in machine learning approaches which can be easily differentiated, for example [START_REF] James M Bern | Soft robot control with a learned differentiable model[END_REF][START_REF] Kriegman | Scalable sim-to-real transfer of soft robot designs[END_REF]. In [START_REF] Hu | Chainqueen: A real-time differentiable physical simulator for soft robotics[END_REF], a differentiable model based on material point method was proposed. In a finite element modeling (FEM) context, an analytical gradient with respect to design variables is used in [START_REF] Tapia | Makesense: Automated sensor design for proprioceptive soft robots[END_REF] to optimise sensors positioning. In the particular context of soft robotics materials, [START_REF] Schumacher | Simulation-ready characterization of soft robotic materials[END_REF] proposes to fit a model with data from several mechanical tensile tests, thanks to an analytical gradient and a quasi-Newton optimisation method. It was successfully applied for hyperelastic models. Similarly in [START_REF] Connolly | Multi-objective optimization of hyperelastic material constants: A feasibility study[END_REF], FEM simulations are used to fit data obtained with simple tensile tests. See also [START_REF] Bächer | Design and Control of Soft Robots Using Differentiable Simulation[END_REF] for a more general review on differentiable simulation.

In this study, we focus on differentiable models for soft robots made of anisotropic materials. The main difference with previously mentioned work is that it does not require to bother with fitting an FEM of the tensile test that was performed, which can be cumbersome. Instead, we rely on the presence of sensors to directly fit the material parameters used in the FEM of the robot. We also consider the case of anisotropy, which implies differentiating with respect to anisotropy direction. Finally, we optimise with respect to several robot configurations and provide insights on the conditioning of the optimisation problem. The anisotropic structure of the robot is created by using a 3D printed meso-structure [START_REF] Martınez | Polyhedral Voronoi diagrams for additive manufacturing[END_REF].

In our previous work, we have shown that we can use FEM and homogenisation to reproduce the behavior of the structure [START_REF] Vanneste | Anisotropic soft robots based on 3D printed meso-structured materials: design, modeling by homogenization and simulation[END_REF]. But homogenisation is an approximation and we pointed out that the printing method (by extrusion) could add additional anisotropies along the printing direction. Once the robot structure is 3D-printed, it is therefore useful to be able to optimise the parameters of the anisotropy model from data that can be collected directly on the robot. The objective is to ensure that the mechanical model fits the data as closely as possible.

This paper proposes an efficient convex optimisation method to determine the parameters of a deformable FEM, based on the behavior of this model in several configurations of its workspace. Although this work is carried out in the context of manufacturing soft robots from anisotropic mesostructures, we would like to stress that the approach presented below is generic and could be easily extended to calibrate other deformable models based on other constitutive laws.

The contributions of the paper are the following:

• Formulation of the material parameters optimisation problem as a convex optimisation over several configurations of a soft robot. In this section, we present the FEM approach used to model anisotropic soft robots and the convex optimisation used on this model to perform open loop control.

A. Static formulation with constraints

In this paper we consider the static equilibrium of the soft robot in its current configuration. This configuration is represented by the position vector x of the nodes of the FEM. On any equilibrium configuration, we have:

f ext -f (x) + H T a λ a = 0 (1)
In this equation, f (x) is the vector of the non-linear internal forces of the deformable structure, computed by the FEM, and f ext is the vector of the external forces f ext , such as gravity forces. We use a Lagrangian formulation to model the loads exerted by the actuators H T a λ a , with H T a the Jacobian matrix of the actuation and λ a the vector of the Lagrange multipliers representing the forces exerted by the different actuators.

When the static equilibrium is perturbed (by a change in the loads exerted by the actuators), we will classically try to find a new equilibrium position. To perform this non-linear computation, a limited series expansion of the forces around the current configuration allows to bring back the problem to a (changing) linear formulation:

f (x + dx) ≈ f (x) + ∂f ∂x K(x) dx (2)
Where K(x) is the tangent stiffness matrix, that depends on the current position of the FEM nodes. In an iterative approach to find the static equilibrium, dx corresponds to the displacement between two successive positions dx = x i -x i-1 . So at each iteration, we find dx and λ a by solving this system:

K(x i-1 )dx = f ext -f (x i-1 ) + H T a λ a (3)
Depending on the case, the way of obtaining λ a varies. If we impose the actuator forces λ a , the problem is straightforward. If the actuators positions are imposed, additional equations corresponding to the actuators motion are set and λ a plays the role of Lagrange multipliers for these new equations. Note that to compute the FEM, we use the open source SOFA 1 framework.

B. Optimisation for open loop control

In the case of inverse problem (we now look for the actuation loads to match specific goals, rather than applying known value of λ a like in the direct problem), the two unknowns of Eq. (3) remain the same. But additionally, a new set of equations is defined to control the shape of the robot through specific points on the FEM mesh, called effectors. 1 https://www.sofa-framework.org/

The distance of these effectors to their goal positions (that corresponds to the input of the optimisation) are measured by δ e . Over a simulation step, this distance can be defined by a linear approximation of the nodes displacement, using FEM interpolation:

δ e = H e dx + δ 0 e (4)
with H e being a (highly sparse) rectangular matrix, and δ 0 e the distance vector, evaluated at the beginning of the step. Combining equations ( 3) and (4), we can derive the relationship between δ e and λ a by:

δ e = [H e K -1 H T a ] Wea λ a + H e K -1 (f ext -f (x i-1 )) + δ 0 e δ free e (5)
We now use this relation in a quadratic programming (QP) [START_REF] Coevoet | Software toolkit for modeling, simulation, and control of soft robots[END_REF] optimisation scheme in order to find the λ a which allows to minimise δ e , ie: to find how to actuate the robot so that selected points reach the different goal positions:

   min W ea λ a + δ free e 2
with δ min ≤ δ a = W aa λ a + δ free a ≤ δ max (and λ a ≥ 0) [START_REF] Tapia | Makesense: Automated sensor design for proprioceptive soft robots[END_REF] with δ a being the actual displacement of the actuator and δ min /δ max its limits. W aa is homogeneous to a compliance matrix, it gives the influence of the different actuators on themselves, it allows to see how they are coupled/decoupled. λ a ≥ 0 is used in case of unilateral actuation (for instance cable actuators). To have a well posed convex problem which converge to a unique solution and avoid instabilities, the QP matrix W T ea W ea has to be positive-definite. To have this property, it is necessary (but not sufficient) that the size of the actuator space is equal or less than the size of the effector space.

III. DIFFERENTIATION AND OPTIMISATION ON MULTIPLE

CONFIGURATIONS

In the model calibration problem, we aim at estimating the mechanical parameters that best fit the robot that was actually built. To verify that the model fits the data, we can continue to use an optimisation based on a set of effector points placed on the robot. However, this time, it is not about optimising the force on the actuators but the parameters of the FEM. In this section, we present the differentiation of the internal forces by these parameters, to make new Lagrange multipliers appear. Then, we show how to adapt the optimisation on several configurations, in order to calibrate the model on several points of the robot workspace.

A. Structural characterisation parameters

The mechanical parameters p of the constitutive law influence the computation of the internal forces. To use our method, these parameters need to be differentiable, which is the case for the anisotropic case described below.

f (x + dx, p + dp) ≈ f (x, p) + ∂f ∂x K(x,p) dx + ∂f ∂p dp (7)
In practice, we perform a numerical differentiation of the internal forces to obtain the value of H T p which is equal to ∂f ∂p . Written like this, the variation of the parameters dp can be assimilated to Lagrange multipliers λ p . Additionally, to have parameters working in a uniform space (for example between -1 and 1), we can apply scaling factors. For consistency of notations, we introduce Sλ p = dp, where S is a diagonal scaling matrix allowing to have all parameters in a uniform space and H T p (x, p) = ∂f ∂p S. Combining with Eq. ( 3) we now have the quasi-static equilibrium defined by:

K(x i-1 , p i-1 )dx = f ext -f (x i-1 , p i-1 ) + H T a λ a + H T p λ p (8)
Once the optimisation problem (which will be presented below) is solved, we update the parameter p i using p i = p i-1 + Sλ p and determine the new dx then continue to iterate to progressively reach an equilibrium (dx = 0 and λ p = 0). Note that in the case of a robot calibration, we will impose references to the actuators (so we know λ a or δ a ).

B. Multi-Configuration optimisation

The goal of the optimisation is to update the parameters p so that the positions of the effector points on equilibrium are as close to the data as possible. But, unlike the optimisation presented in Eq. ( 6), we seek to optimise the parameters over several different static equilibrium configurations, representative of the robot workspace.

For each of these configurations j, we will launch a different simulation by fixing the material parameters at a starting value but choosing different value of actuation λ j a (or δ j a ). Each simulation compute the FEM separately (each of them solves a different version of Eq. ( 3)). When the equilibrium is reached for each configuration, we start the optimisation of the parameters which will link them all together. Indeed the modification of the parameters has repercussions on the static equilibrium for all the configurations. To conduct the optimisation, we will look at the errors on the effectors at the level of each j configuration, and we will do the same type of condensation as the one done in Eq. ( 5):

δ j e = W j ep λ p + W j ea λ j a + δ free,j e (9)
Note that λ p is the only term that do not depend on the configuration since we look for one set of material parameters that is identical across all configurations, and again that λ j a is known (or we can define equality equations to impose δ j a by computing λ j a ). Let's suppose that we perform the optimisation on n configurations, we can aggregate these equations:

   δ 1 e . . . δ n e    ∆e =    W 1 ep . . . W n ep    Wep λ p +    W 1 ea λ 1 a + δ free,1 e . . . W n ea λ n a + δ free,n e    ∆ 0 e (10)
The vector ∆ e gathers the effectors distances to the objectives on all configurations. The goal of the minimisation is to find the set of parameters λ p which minimise the square norm of this vector:

min W ep λ p + ∆ 0 e 2 (11)
When a new value of λ p is found, the parameters p are updated and each configuration can update its quasi-static equilibrium. This ends a simulation step. It is repeated a number of steps until λ p → 0, we reach a minima and stop the optimisation.

IV. PARAMETERIZATION OF ANISOTROPY

In this section we present how we define varying mechanical properties of a mesh to create transverse behavior with as little parameters as possible in order to simplify later on the optimisation presented in III.B. Then we explain how we pass from these parametric mechanical properties of our mesh into a real 3D printed part.

A. Simulation -Mechanical structure

We optimise the mechanical parameters over a FEM. In this paper we focus on a special case of anisotropy which is transversely isotropic material. We have 5 mechanical parameters to define the elasticity tensor of any element of the mesh : 2 Young modulus (E t , E l E t , E l E t , E l ) , 2 Poisson ratio (ν t , ν tp ν t , ν tp ν t , ν tp ) and one shear modulus (µ l µ l µ l ). Using the approach of [START_REF] Li | Stable anisotropic materials[END_REF], we can reduce the parameters needed to only E t , E l E t , E l E t , E l and ν t ν t ν t while insuring a stable anisotropy. To simplify further, we focus on optimising only the Young modulus by setting ν t to a chosen value.

In addition to these mechanical parameters we also add a last parameter which is the "fiber" direction defined by a unit vector called u u u. With the printing technique used (presented in IV.B), we can only configure the fiber in the printing plan. As a consequence we use one rotation parameter to define u, we will call it θ. In the end, we can define a transversely isotropic tetrahedron with only 3 parameters: E t , E l , θ E t , E l , θ E t , E l , θ . Even with these simplifications, we cannot optimise these 3 parameters for each tetrahedron of the mesh: the number of unknowns would be too high and dependent on the mesh resolution. Moreover there is a necessary continuity of parameters between neighboring elements. This is why we place what we call control points (CP) upon the mesh, we can see on Fig. 1 a visual representation of them (red dots).

Each CP will have the triplet of parameters presented above and they will be independent from each other. Then the parameters of each CP are diffused on the mesh: for each tetrahedron the triplet of mechanical parameters are obtained by interpolation, depending on their relative proximity to the different CP. For that we use a simple spacial interpolation called Inverse Distance Weighting (IDW) [START_REF] Shepard | A two-dimensional interpolation function for irregularly-spaced data[END_REF] that allows to have a smooth transition between the different values while never going above the values it interpolate on. Using the CP and doing an IDW between them we only have to optimise 3 parameters times the number of CP used.

B. Fabrication -Slicing configuration

As presented in previous work [START_REF] Vanneste | Anisotropic soft robots based on 3D printed meso-structured materials: design, modeling by homogenization and simulation[END_REF], to build the soft structure of the robot we use classical fused filament fabrication (FFF) with a thermoplastic polyurethane filament (TPU) called NinjaFlex (Shore Hardness: 85A and a tensile modulus of 12MPa : data from the constructor) with 1.75 mm of diameter.

To be able to generate a nearly transversely isotropic material and control its different mechanical parameters we use the slicer IceSL2 . This slicer has many interesting options including the ability to "paint" on a given shape a desired flexibility. We can specify in 3D the coefficient of anisotropy as well as a direction and a density.

We can see on Fig. 1 different interpolation results between two CP for different values of the power parameter p used in the IDW (see [START_REF] Shepard | A two-dimensional interpolation function for irregularly-spaced data[END_REF]). It shows that with p = 1 the transition is too smooth and with p = 6 too sharp. As a middle ground we choose to use for all the presented examples in this paper a value of 2. This figure also demonstrate that we have a close match between the different directions put into the simulation and reality.

V. PROBLEM CONDITIONING

The approach proposed in this paper is based on a convex optimisation. This type of optimisation is sensitive to conditioning, which can be seen as a drawback. On the contrary, we show in this section that an in-depth analysis of the matrices conditioning allow to get practical and intuitive information, useful for the calibration. Given the number and the type of parameters, are there enough effector points on the structure? Are there enough configurations tested in the workspace? Were all parameters excited in the configurations so they can be identified?

As presented in IV.A, the number of parameters is directly proportional to the number of CP. If the number of CP is high, the number of unknown parameters is high and the data collected during the calibration has to be rich enough to insure convergence during the optimisation. On the other hand, having few CP is not enough to have a well posed problem: indeed if we place two CP too close to each other, especially on a zone which deforms little, then the set of parameters of these two CP will be redundant and the problem badly posed.

The input data can vary in two ways: we can place several effector points on the robot and we can choose several configurations on which we do the calibration. All these choices will have a direct impact on the convergence of the parameters, we will study this in the following.

A. Sample Problem setup

To illustrate the mathematical analysis, we will use the relatively simple example of optimising the anisotropy parameters on a cantilever soft anisotropic beam. This beam has a rectangular section and we place on its 2 extremities a CP. We suppose that the mass is known and we optimise only the two elasticity parameters and the anisotropy direction so p to be optimised is a vector of dimension 6 (dim(p) = 2 × 3 = 6).

We generate artificially goals for an input data set of p target with a direct simulation and we will try to find back with the optimization p target by starting with random values of p init .

We can see in Fig. 2 the simulation setup with the different configurations having all the same common parameters p init . The configurations 2 and 3 are rotated beam along their longitudinal axis to have information about the fiber direction influence. We also add configuration 4 where we attached the beam from the other end to equalize the importance of the mechanical parameters on either side of the beam.

B. Influence of the configurations on the convergence

As explained above the number of parameters, the variety of configurations and the placement of effectors are key. They have a direct link with the optimisation convergence.

To illustrate this, we can see on Fig. 3 convergence each with a different number of configurations of the simple beam presented previously (Fig. 2) with only one effector at its tip. When we only have one configuration (red) the QP problem get stuck on a local minima which does not allow to reduce further ∆ e . With 2 and 3 configurations we get to a ∆ e ≈ 0 but around simulation step 65. The third configuration probably does not provide enough relevant information. In comparison, the fourth configuration, inversed (the beam is now attached by its other end) brings useful information by making "visible" the effect of some parameters.

The problem converges more quickly (≈ step 50). Through this example, we can see that it is useful to have mathematical tools to understand if the optimisation problem we are creating is well posed.

Then we test the optimisation when starting with random initial p init . By minimising the beam tip position difference between the input data and the position obtained in the optimisation, we test the convergence and observe if we reach the same p target .

We have launched the optimisation 30 times with, each time, a new randomized p init and as we can see on Fig. 4 for the parameters E t and E l of the 2 CP, the QP problem converge each time to the same values which are the right p target . Note that we have chosen a certain range on the parameter space: [E min , E max ]. Also, for stability issues, we have forced the following inequality between longitudinal and transverse Young modulus : E l < E t .

C. Conditioning analysis on soft anisotropic beams

Through this optimisation example of anisotropic beams, we illustrate in this section how it is possible to study the compliance matrix conditioning, to have a better understanding about the convergence/divergence of the QP problem.

In Eq. ( 11), the matrix W ep represents the influence of the mechanical parameters variations on the effectors positions since we have, for each configuration j, δ j e = W j ep λ j p + δ j,0 e . To study conditioning, one can do a singular value decomposition (SVD) of the compliance W ep = USV T . The range of values in the matrix S (between the largest and the smallest) is related to the conditioning value of the problem. If this matrix is poorly conditioned, i.e. with a high condition number, there will be modifications of parameters that will have no effect on the movements of effectors. In practice this will be likely to make the QP optimiser fail. In this situation, we can extract the corresponding kernel vectors (the vectors of U corresponding to the small singular values, say U s ) that will provide the directions in which modifying the mechanical parameters has almost no influence on the movement of the effector points. In practice, it means the effector points do not change positions using 2 different sets of mechanical parameters p and a variation of p along the direction U s : p + αU s .

This information is very rich as it provides parameter directions that have not been sufficiently excited in the input data set of the calibration. This allows to point out which parameter or combination of parameters needs to be particularly exposed in additional configurations, allowing to improve the conditioning.

However, analysing the concrete conditioning values obtained for the matrix W ep could be difficult as they are also dependent of nominal parameters, such as the global compliance. So a large conditioning value could still sometimes be acceptable. We can reduce this effect by using the square matrix W pp (similar to W aa definition) obtained by summing up j H p K -1 H T p on each of the j configurations. If we compute W ep (W pp )

-1 we obtain a dimensional link between the parameters and the effectors for which the conditioning number is easier to interpret. We can see in Tab. I different results of conditioning for W ep (W pp )

-1 depending on combinations of number of parameters/effectors/configurations. We can see that when we optimise with less parameters and with more configurations, the conditioning values tends to decrease which improve the convergence. On the contrary, we rapidly tend to divergence cases when we have less configurations with less effectors. These conditioning values are therefore good indicators of the variety of data used for calibration (with respect to the tested parameters) but unfortunately, we cannot yet presume what is the bound from which the problem will diverge/converge.

VI. PRACTICAL USAGE: CALIBRATION OF A REAL ANISOTROPIC PARALLEL ROBOT

In [START_REF] Vanneste | Enabling the control of a new degree of freedom by using anisotropic material on a 6-DOF parallel soft robot[END_REF] as we can see in Fig. 5, we have shown that we were able to create a new DOF on a 6-DOF parallel soft robot using metamaterial and simulate this new behavior. We have also done an initial workspace assessment on the simulation.

In this previous work, mechanical parameters were fed to the model in a fairly accurate way through theoretical mechanical parameters of the soft plastic used in the 3D-printer as well as numerical homogenisation to take into account the particular foam structure of the meta-material. Nevertheless, we were controlling the robot on open loop and we witnessed large differences between the simulated workspace capability and the reality. This can come from multiple factors/errors :

• Printing technique adding defects into the structure and also adding some anisotropy along the printing direction. • homogenisation of the generated mesostructure mechanical properties. • Setup little errors (placement, motors position, mechanical fatigue, noise...). A calibration of the model based on real data, taken from the robot, should allow to significantly improve the results of the model, in particular by reducing the first two error factors.

A. QP problem setup

As effector, we have chosen the frame placed at the middle of the rod linking the 2 tripods. We have placed on it a Polhemus3 sensor that allows us to get accurate position as well as orientation of the effector during the calibration. To characterize our soft sheet mechanical properties we choose to put CP as shown on Fig. 6: two for each branch (with none at their end because they are rigidified to represent the attached part to the motors) and then 6 around the center where most of the precision is needed. The p init are given in order to be as close as possible with the previous simulated model (see A of Fig. 6), which means a uniform value of the Young modulus E t and E l along the structure and the same spiral pattern parameterized with θ.

Assuming that 3D printing is sufficiently reproducible, we consider that the robotic setup is symmetrical. The same parameters will be used in the two soft sheets. Consequently, we have 12 CP with 3 parameters each, so a problem of dim(p) = 36. For the calibration, we choose 7 configurations (which is the minimum to guarantee the positiveness of the QP matrix) giving 7 effector positions well distributed inside the workspace. Each effector position is described with 6 parameters (3 for translations, 3 for orientations) so we have dim(∆ e ) = 42.

B. Results

The optimisation converged to 36 new values giving to a new visual representation of the results with the interpolation shown on Fig. 6.B. It clearly show the shortcomings of 3D printing with respect to anisotropy and expected mechanical properties: compared to Fig. 6.A p init , E t and E l are no more homogeneous over the sheet with blue pockets being less stiff than the red. Moreover, the spiral pattern is much less pronounced.

In contrast, with this new set of mechanical parameters the general error reduction between the goal and effector was TABLE II: Errors reduction of the effector position in percent between the real setup and the simulation before and after optimisation. The first line correspond to the error reduction on configuration on which the optimisation was done and the second on newly tested configurations.

To show the validity of the calibration, we also consider new positions in the workspace (that were not used for calibration) and we compare the evolution of the error with the model using the initial (not calibrated) parameters. We can see that it was also reduced in average by 43% which shows the FEM ability to interpolate the behaviour beyond the positions on which it was calibrated.

A part of the remaining error (which is noticeably smaller than before) comes from the fact that the model is not perfect (mechanical parameters comes from homogenisation, internal contact are not taken into account, model for small displacement for linear elastic material...) and another part comes from the real setup noises that are not taken into account in the calibration (positioning error of the motors, dimension and attachment of the sheet with the motors, placement of the rigid marker...). To further decrease the error, more CPs could be added to refine the parameters evolution over the robots shape but it will require to calibrate on more configurations.

Nevertheless, the QP optimiser was able to find the best parameters given the input data. The improved model parameters can be used for more accurate control of the robot, including when looking into closed-loop control.

VII. DISCUSSION

We have presented a new method allowing to optimise mechanical parameters of a FEM as well as to formulate and solve multi-configurations QP problems. The method allows to calibrate a complex anisotropic FEM with a relatively high number of parameters (here up to 36) while using a relatively low number of different configurations (only 7), with a single 6D-measurement per configuration, and succeeds to achieve an average error reduction, before/after calibration, of more than 40%. The number of input data remains reasonable. The method is frugal compared to model-less methods.

The method is based on convex optimisation and we have shown that the combination of the number of configurations, the number of parameters, and the number of effectors have to be carefully chosen to guarantee the well-posedness of the QP problem. In section V-C, we give an indicator to evaluate the conditioning of the QP problem. However that is not a strict criteria that will predict whether the optimisation will converge or not.

But another aspect of the problem must be mentioned: convex optimisation can lead to getting stuck on local minima, in particular when the choice of parameters or the stresses in terms of deformations lead to important non-linearities where it is difficult to reach the global optimum. Moreover, we rely on numerical differentiation that assumes a good continuity of the influence of the parameters on the expression of the internal forces.

We have discussed ways to set the optimisation problem properly but it still needs some investigation in order to have a better knowledge about the problem conditioning and as much as possible, we would like to provide an automatic calibration workflow. We could test the method on various soft robots, with other actuator types and also other sensors to make the calibration.

Also we have to keep in mind that the optimisation process over real life data will try to correct the total error of our real setup (motors actuation / mounting errors...) and not only the mechanical parameters of the soft structure. It would need more effort to have a more reliant setup in order to decouple the mechanical errors specifically. Note that we are doing open loop simulation for the motors control, closing the loop and sending back more sensors informations (like real motor positions and/or torques) can help achieve this mechanical decoupling.

VIII. CONCLUSIONS AND PERSPECTIVES

The calibration method based on QP optimisation presented in this study is generic and can be applied to any geometry of soft robot as well as any FEM with other constitutive law (in principle). In this letter, we show that it is applicable in the complex case of an anisotropic mechanical model, which involves "traditional" mechanical parameters like Young modulus, but also more specific parameters like directions of anisotropy. We have demonstrated that the method can handle this complexity and is able to find satisfying optimums as long as enough informative configurations are provided.

This method could also be used for design purposes by combining mechanical parameters optimisation with actuation in order to have a robot structurally optimised for a specific task. This will be a direction for future works.

Fig. 1 :

 1 Fig.1: Interpolation between 2 CP (red dot) with, on the left each little black lines corresponding to a tetrahedron's direction and on the right we have the corresponding 3D shape generated. The right/left CP have respectively a set direction of -35°/35°. There is also a comparison with different values of p in the IDW interpolation[START_REF] Shepard | A two-dimensional interpolation function for irregularly-spaced data[END_REF].

Fig. 2 :

 2 Fig. 2: QP simulation with 4 configurations of a simple beam subject to only gravity. Purple arrows represent the new change in the configuration compared to the first (1). Configuration 2 & 3 have two different rotation around there longitudinal axis and for 4 the tip and attached part were switched.

Fig. 3 :

 3 Fig. 3: Minimisation of ∆ e in different QP problem. B1 = QP problem with one configuration, B2 with 2 ect...

Fig. 4 :

 4 Fig. 4: E t and E l convergence from the simulation of Fig. 2 for 30 different starting of p init . Blue/green lines are respectively for CP 1 /CP 2 parameters and the red dotted lines correspond to the different parameters objectives. The 2 black lines represent the upper/lower bound (E max /E min ) we have put for E.

Fig. 5 : 6 -Fig. 6 :

 566 Fig.5: 6-DOF parallel soft robot of[START_REF] Vanneste | Enabling the control of a new degree of freedom by using anisotropic material on a 6-DOF parallel soft robot[END_REF], left and middle show that we add a new independent DOF with the metamaterial and right shows that we are able to simulate this new behavior.

TABLE I :

 I Comparison of different conditioning values of W ep (W pp ) -1 with different number of parameters 4, 6 per beam and effector on them 1, 4 and finally with 3, 4 or 6 configurations. Green correspond to converging cases, orange to oscillating and red to diverging.

https://icesl.loria.fr

https://polhemus.com/