Tsiry Mayet

Simon Bernard

Clement Chatelain

Romain Herault

Domain Translation via Latent Space Mapping

Keywords: Domain translation, Semi-supervised learning, Weaklysupervised learning

In this paper, we investigate the problem of multi-domain translation: given an element a of domain A, we would like to generate a corresponding b sample in another domain B, and vice versa. Acquiring supervision in multiple domains can be a tedious task, also we propose to learn this translation from one domain to another when supervision is available as a pair (a, b) ∼ A × B and leveraging possible unpaired data when only a ∼ A or only b ∼ B is available. We introduce a new unified framework called Latent Space Mapping (LSM) that exploits the manifold assumption in order to learn, from each domain, a latent space. Unlike existing approaches, we propose to further regularize each latent space using available domains by learning each dependency between pairs of domains. We evaluate our approach in three tasks performing i) synthetic dataset with image translation, ii) real-world task of semantic segmentation for medical images, and iii) real-world task of facial landmark detection. Source code is publicly available 3 .

Introduction

In many machine learning tasks, different modalities can be modeled as different domains and different data as different views of the same reality. For example considering an autonomous vehicle context, the RGB camera, the depth map, and the segmentation map can be considered as three views of the same reality. Often one domain can be considered as an input domain, e.g. a CT (computed tomography) scan of a patient, and other domains as an output domain, e.g. organ segmentation of the scan. Here we do not consider an input nor output domain, but rather we would like to learn to translate from any domain to another one. This definition masks the classical definition of an input and output domain, as every domain can be a possible input or output. While the rising amount of available data has brought great results in domain translation tasks in a fully supervised fashion, in some fields the quantity of available supervision is limited and hard to obtain, as in the medical field where a domain expert is required to create a hand-made segmentation [START_REF] Lerouge | Ioda: An input/output deep architecture for image labeling[END_REF][START_REF] Reiß | Every annotation counts: Multi-label deep supervision for medical image segmentation[END_REF], and this low supervision setting often reduces the performance of classical deep learning model.

Acquiring complete supervision to learn this translation across the different domains can be a tedious task, since for one view all the other corresponding views in the other domains are required. We now consider the setting of incomplete supervision, where for one view, the other view might or might not be available. We investigate the task of multi-domain translation, i.e., given a set of n domains {D 1 , . . . , D n }, we would like to learn a mapping function ∀i, j f i→j : D i → D j allowing to translate a view from the domain D i into its view in D j . An illustration with n = 3 is given in Figure 1 where multiple possible configurations of the triple (d 1 , d 2 , d 3 ) are possible, and each point does not necessarily have an associated view in the other two domains available (e.g. example of available data: (d 1 , d 2 ), (d 2 , d 3 ), or only (d 1 )).

Many classical machine learning tasks fall into this domain translation framework, provided that one can reconstruct a view d i given another view d j . For example, image captioning and machine translation can be viewed as domain translation problems. In Image captioning, one domain is the image and the other is its description, and for machine translation both domains are textual. Previous approaches leverage useful assumptions and inductive bias, a common one in domain translation is the shared latent space assumption [START_REF] Chandar | Correlational neural networks[END_REF][START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF][START_REF] Grover | Alignflow: Cycle consistent learning from multiple domains via normalizing flows[END_REF][START_REF] Hoffman | Fcns in the wild: Pixel-level adversarial and constraint-based adaptation[END_REF][START_REF] Pei | Multi-adversarial domain adaptation[END_REF] -associated views of different domains can be embedded as the same code in a latent space.

In this paper, we propose LSM unified framework, which allows learning to translate one view from a domain to another domain in a semi-supervised setting when either all the views in each domain are available, or when an arbitrary number of views are available. We do so by learning a latent space for each of the domains where each view can be embedded in a manifold.

Each manifold is further constrained by the mean of distance and adversarial losses allowing LSM to learn a better representation by leveraging the intradomain dependencies (dependencies within one domain, e.g. the value of one image pixel or segmentation mask regarding its surrounding pixel values) and inter-domain dependencies (the dependencies between multiple domains, e.g. the pixel of an image with regard to the associated pixel of the segmentation mask).

The rest of this paper is organized as follows, Section 2 reviews related work to domain translation, Section 3 presents how LSM framework is structured and trained, and Section 4 describes how domain translation can be applied to three tasks (semantic segmentation, image to image translation, and facial landmark detection). The detailed settings of the training are provided in the Appendix.

Related Works

Domain translation is related to other tasks like semi-supervised learning (SSL), and domain adaptation (DA). While SSL and DA differ from domain translation, these three tasks leverage common assumptions and often use similar methods. The goal of SSL is to learn from labeled and unlabeled samples, but in SSL an input domain and output domain are clearly identified. In domain translation, a sample can be instantiated in A but not in B, in B but not in A, or in A and B. None of the domains can be considered either as 'input' nor 'output'. In DA, an input domain and an output domain are also often considered, but the input domain contains two sub-domains: a source domain and a target domain, in this setting close to SSL, the source domain contains labeled examples and the target domain is often left without supervision. The discrepancy between the two inputs domains prevents from just merging them together. In this work, we leverage two useful SSL assumptions that we will present: the Manifold Assumption that we use to learn the intra-domain dependencies, and the Shared Latent Space Assumption that we relax in order to learn inter-domain dependencies. We will then present methods that exploit the same type of assumptions, and how we take a different interpretation of these assumptions.

Manifold Assumption

According to the manifold assumption, high-dimensional data can be embedded in a lower-dimensional space [START_REF] Chapelle | Semi-supervised learning[END_REF], a latent space, with a lower degree of freedom considered. For the algorithm, working with this latent code representation of the data is easier than working with the original high-dimensional data, and learning this latent space helps to capture the underlying data structure -the intra-domain dependencies. Several architectures such as the auto-encoder and variational auto-encoder have been used to learn this latent space [START_REF] Bengio | Greedy layerwise training of deep networks[END_REF][START_REF] Hinton | Reducing the dimensionality of data with neural networks[END_REF][START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Kingma | Auto-encoding variational bayes[END_REF]. While learning intra-domain dependencies has shown effectiveness, allowing to use of additional data without associated views in the other domains [START_REF] Lerouge | Ioda: An input/output deep architecture for image labeling[END_REF], this approach alone does not leverage inter-domain dependencies when there is no supervision. The next assumption help to alleviate this issue.

Shared Latent Space Assumption

Deriving from the Manifold Assumption and mainly used in DA, the Shared Latent Space Assumption state that associated views of different domains can be mapped to a common latent code. If a view d i ∼ D i is associated with view d j ∼ D j , then di = dj . Learning to align each manifold in the same shared structure allows learning the inter-domain dependency between views of different domains. But this every-to-one mapping can be seen as an over-simplification, as multiple modalities can have very dissimilar features (e.g. consider an image and its pixel-wise semantic segmentation mask, the mask does not contain in itself information about the texture), this inter-domain discrepancy coupled with the every-to-one mapping could hinder the model when learning the translation.

Domain Translation

We will give an overview of common domain translation methods while classifying them according to the previously defined assumptions in order to better understand how we take different choices from them. These models can be divided into roughly two types illustrated in Figure 2. Methods using a single shared latent space can use adversarial learning on the feature space [START_REF] Chandar | Correlational neural networks[END_REF][START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF][START_REF] Grover | Alignflow: Cycle consistent learning from multiple domains via normalizing flows[END_REF][START_REF] Hoffman | Fcns in the wild: Pixel-level adversarial and constraint-based adaptation[END_REF][START_REF] Pei | Multi-adversarial domain adaptation[END_REF], and possibly also on the original highdimensional space [START_REF] Choi | Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[END_REF][START_REF] Grover | Alignflow: Cycle consistent learning from multiple domains via normalizing flows[END_REF][START_REF] Hoffman | Cycada: Cycle-consistent adversarial domain adaptation[END_REF][START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Li | Unsupervised domain adaptation in the wild via disentangling representation learning[END_REF][START_REF] Li | Bidirectional learning for domain adaptation of semantic segmentation[END_REF][START_REF] Liu | Unsupervised image-to-image translation networks[END_REF]38], using generative adversarial networks (GAN [10]) to generate a view from a latent code. GAN has been notoriously hard to train due to the min-max optimization that can lead to vanishing gradient [1] and is subject to mode collapse [34] (a mode collapse occurs when the generation is almost deterministic and the generator always creates the same output). Building on top of these existing works, models with multiple latent spaces often use disentanglement learning, dividing the latent code into twopart: a domain-agnostic content code, and a domain-specific style code. This more sophisticated setting allows learning a domain translation function that would combine a content code from a source domain and the style code of the targeted domain [START_REF] Huang | Multimodal unsupervised image-toimage translation[END_REF][START_REF] Lee | Diverse image-toimage translation via disentangled representations[END_REF][START_REF] Li | Unsupervised domain adaptation in the wild via disentangling representation learning[END_REF][START_REF] Liu | Gmmunit: Unsupervised multi-domain and multi-modal image-to-image translation via attribute gaussian mixture modeling[END_REF][START_REF] Sun | Dual-glow: Conditional flow-based generative model for modality transfer[END_REF]37]. Learning a disentangled representation is not straightforward as the definition in itself of content and style is ambiguous. These approaches bring great results at the cost of additional complexity and inductive bias (e.g. using adversarial learning between the content code and the style code).

Latent Space Mapping Assumption

We will now present how LSM uses the previously defined concept and how it differs from existing works by taking a different interpretation of the presented assumptions. LSM use the manifold assumption by, for each domain D i , learning a latent space where a view d i is embedded in its latent code di . This allows the approach to modelize intra-domain dependencies. Additionally, we learn the inter-domain dependencies using the shared latent space assumption, but here we decide to take a different approach from the classical full-shared latent space while keeping its simplicity. Instead of using a single shared latent space, we relax the assumption by supposing the existence of a function L i→j : Di → Dj allowing to translate a latent code from one domain to the latent code of another domain. The shared assumption is then applied between translated codes L i→j ( di ) and codes dj from the original domain. Similar ideas has been approached with success in existing works: in IODA [START_REF] Lerouge | Ioda: An input/output deep architecture for image labeling[END_REF] by the mean of pre-training an input encoder using input data, an output decoder using output data to initialize a network, and learning the final task in a supervised fashion. And in SOP [START_REF] Belharbi | Input/output deep architecture for structured output problems[END_REF] where there is no pre-training but the learning of this latent space is done in an unsupervised fashion as a regularization task. But SOP only uses the manifold assumption and does not take into account the discrepancy between a translated latent code and a latent code from the original domain.

To alleviate this issue, we will use a framework similar to that of the SOP. We further regularize each latent space by making the assumption that L i→j ( di ) should be close to dj .

LSM Framework

We now present LSM framework, illustrated in Figure 3, for domain translation between multiple modalities and multiple domains, by learning functions f Di→Dj : D i → D j . The framework learns intra-domain dependencies, for each view d i ∼ D i , the model produces its embedding di , this can be done in an unsupervised way by the mean of reconstruction task. It also relaxes the shared latent space assumption by supposing the existence of a function L i→j allowing to translate of a latent code di to another domain latent space dj , the shared assumption is then applied after the translation by the mean of a distance loss when paired information is available or by means of adversarial in an unpaired setting.

For clarity of concern and without loss of generality, we will now consider only two domains X and Y , and we will present the translation from domain X to domain Y , the same losses are applied when translating from Y to X. The under-script x → y will be used on functions that are specific to the translation of X to Y and has to be duplicated in the case of Y to X. Other functions are only applied once. This approach leverage three types of data configuration, either learning from view x ∼ p(x) when y is not available, from y ∼ p(y) when it is provided without corresponding x, or from paired views (x, y) ∼ p(x, y).

Our goal is to learn a translation function f x→y : X → Y from domain X to domain Y as ŷ = f x→y (x). Using the manifold assumption we introduce two latent spaces x (resp. ỹ) where the view x (resp. y) could be encoded and where the code would retain a summarized version of the original data. To learn this latent space, two auto-encoders are used: x = E x (x) and ỹ = E y (y) mapping a view to its latent code, and x = D x (x) and ŷ = D y (ỹ) mapping the latent code to its original view. An additional function performs the mapping between latent space X to Y as ỹ = L x→ỹ (x). Given these latent spaces and functions, f x→y can be reformulated as:

ŷ = f x→y (x) = D y (ỹ) = D y • L x→ỹ (x) = D y • L x→ỹ • E x (x). x x x ỹ y ŷ L x→ỹ E x E y D x D y 0/1 L x→ỹ (x) E y (y) DC ỹ Fig. 3: Illustration of LSM framework. The translation is presented from the point of view of X → Y 3.1 Learning X to Y Domain Translation
When a pair is available as (x, y) ∼ p(x, y), the model performs as a fully supervised method, predicting y from x.

ŷ = D y • L x→ỹ • E x (x) (1)
This allows learning the translation network by optimizing the supervised loss min

θ Ex ,θ L x→ ỹ ,θ Dy E p(x,y) L x→y (D y • L x→ỹ • E x (x), y) = L x→y (ŷ, y) (2)
Where L x→y can be a cross-entropy loss for a segmentation problem, or a MSE loss for a regression problem for example.

Intra-Domain Dependencies Regularization

When a pair of views is available as (x, y) ∼ p(x, y), or when only x ∼ p(x), or only y ∼ p(y) are available, it is possible to regularize the training of the model by adding two additional reconstruction losses. These losses based on the manifold assumption allow to learn intra-domain dependencies via the mean of reconstruction, with auto-encoder for example [START_REF] Bengio | Greedy layerwise training of deep networks[END_REF][START_REF] Hinton | Reducing the dimensionality of data with neural networks[END_REF][START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF]. The model optimizes the reconstruction of the views from their latent code as:

x = D x (x) = D x • E x (x) (3) ŷ = D y (ỹ) = D y • E y (y) (4)
The two associated reconstruction losses min

θ Ex ,θ Dx E p(x) L x→x (D x • E x (x), x) = L x→x (x, x) (5)
min

θ Ey ,θ Dy E p(y) L y→y (D y • E y (y), y) = L y→y (ŷ, y) (6)
can be used also when not all the data are available. When only x (resp. y) is available, only L x→x (resp. L y→y ) is optimized.

In machine learning, it is a common case to have plenty of unlabeled x only views, but to have only labels without the x available is more rare. For example, in a domain translation as style transfer, having y only sample is often easy to obtain as it needs collecting images of the desired style. For some other tasks like semantic segmentation, it is possible to obtain y only sample in road segmentation through the usage of simulator [START_REF] Martinez | Beyond grand theft auto V for training, testing and enhancing deep learning in self driving cars[END_REF], in medical images segmentation the use of phantom able the generation of y only sample [START_REF] Fedrigo | Development of the lymphatic system in the 4d xcat phantom[END_REF].

So far, by only taking the translation from X to Y , this framework corresponds to the SOP architecture [START_REF] Belharbi | Input/output deep architecture for structured output problems[END_REF]. LSM extends SOP in order to allow multidomain translation by taking into account the transition from Y to X and further leveraging inter-domain dependencies.

Inter-Domain Dependencies Regularization

Unlike previous approaches using the shared latent space assumption, we consider that this assumption should be relaxed and that L x→ỹ (x) should be close to ỹ. Applying inter-domain regularization on a fully shared latent space between each latent code from different views without domain-specific information might hinder the learning process by removing uncorrelated, domain-specific features which might be helpful for the domain translation task. LSM prevents this issue by creating a latent space for each domain, and regularization is done between a latent code ỹ originating from a domain and a translated latent code into this domain L x→ỹ (x). This does not penalize keeping domain-specific information in each latent code.

To enforce latent code L x→ỹ (x) to be close to ỹ, two additional losses are added, depending on the available supervision.

Supervised case

In the case where the pair of view (x, y) is available, LSM minimizes min

θ Ex ,θ L x→ ỹ ,θ Ey E p(x,ỹ) L Distỹ (x, ỹ) = d(L x→ỹ (x), ỹ) ( 7
)
where d is a distance function, as for example L 1 distance. Minimizing L Distỹ allows bringing closer the translated latent code L x→ỹ (x) and the latent code ỹ. In the supervised case, where all views are available, for example in a road semantic segmentation context considering the RGB image and its segmentation map (x, y), and their latent representation (x, ỹ). Supposing that ỹ allows reconstructing y through the decoder D y (ỹ) = y, then if L x→ỹ (x) = ỹ, obtaining ỹ using x would allow to translate x into y through the latent space mapping. In order for the translation to be accurate, one would expect L x→ỹ (x) to be close to E y (y) = ỹ. Equation 7 enforce this constraint by minimizing the distance between latent codes.

Unsupervised case When the supervision is incomplete (either only x or only y is available), we substitute the distance loss L Distỹ with a feature adversarial loss inspired by existing domain adaptation approaches as done in [35].

We would like the translated latent code L x→ỹ (x) to be invariant from the latent code ỹ. Classical machine learning models assume that all the input came from the same distribution, in this case, the model is the decoder, and the inputs are latent codes. We consider latent codes to be invariant if a domain classifier is unable to distinguish its source. For that purpose, a domain classifier DC ỹ is added on top of Y latent space in order to draw the two distributions closer. For the domain classifier, the loss is a binary cross-entropy (BCE) where the discriminator has to correctly classify the presented latent code in the right domain.

min θ DC ỹ E p(x),p(ỹ) L DCỹ (x, ỹ) = BCE(DC ỹ • L x→ỹ (x), 0) + BCE(DC ỹ (ỹ), 1) (8)
The domain classifier attempt to discern from which domain the presented latent code is sampled.

Classical adversarial approaches make the generating part maximize the error of DC ỹ , however, as stated in [36], the two distributions L x→ỹ (x) and ỹ are changing, this could lead to an optimization problem. When the generating part is optimal, the domain classifier would only have to flip its sign in order to produce the correct prediction. This would result in oscillations in the optimization. Therefore, we encourage domains to produce features L x→ỹ (x) and ỹ to be indistinguishable, using a confusion loss [35], where we compare the domain classifier decision against a uniform distribution:

min θ Ex ,θ Ey ,θ L x→ ỹ E p(x),p(ỹ) L Confỹ (x, ỹ) =BCE DC ỹ • L x→ỹ (x), 1 2 (9)
+ BCE DC ỹ (ỹ), 1 2 (10)

Final Loss

Finally, the model is trained to optimize the weighted sum of all the previous losses. For a two-way X/Y domain translation (x → y and y → x), the global loss is min

θ Ex ,θ Ey ,θ L x→ ỹ , θ Dx ,θ Dy ,θ L ỹ→x L F inal = λ 1 L x→x + λ 2 L y→y + λ 1 3 L x→y + λ 2 3 L y→x + λ 4 (L Distx + L Distỹ ) + λ 5 (L Confx + L Confỹ ) (11)
min θ DC x ,θ DC ỹ L Adversarial = L DCx + L DCỹ (12)
It allows the model to receive supervision regardless of whether the samples are drawn from a marginal distribution x ∼ p(x) and y ∼ p(y), or from their joint distribution x, y ∼ p(x, y). L Distỹ and L Confỹ applied on a latent space on domain Y further push the model to create an embedding where the translated latent code L x→ỹ (x) and the original latent code ỹ are more similar. In the general cases where all domains are similar (e.g. both domains are real images),

λ 1 3 = λ 2 3 .

Experiments

In this section, we provide an evaluation of LSM along with three baselines on three datasets. To evaluate the benefits induced by LSM, we evaluate every approach using the same architecture that we made as simple as possible. We empirically show that the use of LSM is not detrimental when a lot of supervision is available. Furthermore, LSM allows for improving the training when additional unsupervised information is available and improves the training while reducing the variance of the models. Qualitative results are available in Appendix E. In order to complexify the task, for the segmentation mask, we swapped the x position with the circle thickness and the y position with the circle radii. This changes the task from a segmentation task to a more challenging image-to-image translation task. We use the standard mean Intersection over Union (mIoU) as evaluation metric for the output prediction. Figure 4 displays two pairs of an input image and ground truth. We refer to Appendix A.1 for dataset description details.

Datasets

Facial Landmark Detection (300-W) Dataset We study the real case of facial landmark detection, on the 300 Faces In-the-Wild Challenge (300-W) [START_REF] Sagonas | faces in-the-wild challenge: The first facial landmark localization challenge[END_REF]32]. This 300 Faces In-the-Wild Challenge dataset is constituted of 300 indoor images and 300 outdoor images with associated 68 facial landmarks for each face. It provides a context with multiple modalities (image faces, and vector of positions) and with a highly structured output domain that presents strong intra-dependencies. We use the standard Normalized Root Mean Square Error (NRMSE) as metric for facial landmark detection. We refer to Appendix A.2 and [START_REF] Sagonas | faces in-the-wild challenge: The first facial landmark localization challenge[END_REF]32] for dataset description details, examples of data are given in Figure 5.

N=100% N=40% N=10%

Fig. 5: 300-W with their associated facial landmarks (red dot) and predicted landmark from LSM (blue dot) for different amounts of available (x, y) views.

More predictions in Appendix E

Sarcopenia Semantic Segmentation Dataset (SSS) We study segmentation of the third lumbar vertebra (L3) level from Computed Tomography (CT) scans. This task is needed for predicting the sarcopenia level, which is a measure of muscle atrophy, later used in cancer diagnostic [START_REF] Lanic | Sarcopenia is an independent prognostic factor in elderly patients with diffuse large B-cell lymphoma treated with immunochemotherapy[END_REF][START_REF] Martin | Cancer cachexia in the age of obesity: Skeletal muscle depletion is a powerful prognostic factor, independent of body mass index[END_REF]. We use the mean Intersection over Union (mIoU) as metric for the segmentation. We refer to Appendix A.3 for dataset description details, examples of data are given in Figure 6.

X GT N=100% N=20% N=1%

Fig. 6: SSS samples with their associated prediction from LSM for different amounts of available (x, y) views. More predictions in Appendix E

Baselines

We compare our model with 5 baselines: a basic baseline, IODA [START_REF] Lerouge | Ioda: An input/output deep architecture for image labeling[END_REF], SOP [START_REF] Belharbi | Input/output deep architecture for structured output problems[END_REF], CycleGAN [39] an unsupervised domain translation approach, and Pix2Pix We first compare each model while adapting the amount of available supervision, controlled by taking N % of the dataset as supervised pair (x, y), and splitting evenly the remaining (100 -N )% as only x samples and only y samples (unsupervised pairs). We lower N in order to see an evolution in the metrics. To ensure that no x and y in the unsupervised set could come from the same pair, we divide the dataset in two before the training, keeping one part as a bank of unsupervised pairs. This has for effect to divide the effective dataset length by two. As CycleGAN operates in an unsupervised setting, its performance remains the same for each N value. LSM is compared for different values of N for the facial landmark in Figure 7a, the sarcopenia dataset in Figure 7b and the synthetic image translation in Figure 8.

High Amount of Supervision For the highest N values (N = 100) and the 300-W and SSS dataset, there are no significant differences between models, as there is plenty of (x, y) views, the different baselines are able to learn the tasks. For SIT we found that LSM greatly improves on the metric compared to the baselines, this is explained by the fact that, unlike in the two previous tasks, there is no direct pixel-to-pixel (or spatial) correspondence between the two domains (due to the swapping of variables). We found that CycleGAN is able to perform on the 300-W and SSS tasks, where there is a pixel-to-pixel correspondence, but fails the SIT image translation, as the correlation between domains is not pixelto-pixel, this unsupervised approach is not able to understand the swapping of variables and not suited for this task.

Medium values of N (N ∈ {80, 60, 40} for facial landmark, N ∈ {80, 60, 40} for sarcopenia datasets, N ∈ {80} for synthetic dataset) only slightly affect the training. For the two first, the metric only starts to really decrease for LSM and the baselines when N < 40, this is explained by the fact that the models does not need every sample to learn the task. Those values of N are Pareto optimal for this combination of dataset and model leading to either no or a slight decrease in the metrics. For the synthetic dataset SIT, the metric start to decrease earlier as the task might be harder for the models and the amount of data is lower.

Low Amount of Supervision When N became low (N < 60 for every dataset) and the amount of supervision became limited, every model's performance start to degrade as the amount of supervision is no longer sufficient to completely learn the task and generalize over the test set.

For the 300-W and SIT datasets (Figure 7a) and the lowest amount of supervision (N = 1), LSM performances decline and reach the same NRMSE as the supervised and unsupervised baselines. While for the SSS dataset, LSM is able to better adapt to a low amount of supervision than the other supervised baselines. In the low-supervision setting, unsupervised models such as CycleGAN are able to perform well, for N ≤ 4 for 300-W, and N ≤ 2 for SSS. For the synthetic dataset, the task is impossible to solve without supervision due to the swapping of variable, and LSM stay above all the baseline for low values of N .

For the sarcopenia dataset Figure 7b, we can note that LSM performances decrease slower than the supervised baselines. We conjuncture that LSM is able to leverage the side information from only x or only y views without increasing the discrepancy between both domains. Where SOP learning schema does not Fig. 8: Impact of the number of available pairs from the synthetic image translation dataset on the mIoU metric (higher is better). Areas around LSM line represent the standard deviation (5 trained models for each baseline)

Table 1: NRMSE evaluation on the Facial Landmark Dataset according to the portion of available pairs (x, y) from the dataset

Model 100 % 60 % 10 % 5 % 4 % 3 % 2 % 1 % Baseline 1.25e-1 1.47e-1 2.25e-1 2.38e-1 2.51e-1 2.59e-1 2.87e-1 3.11e-1 IODA 1.46e-1 1.62e-1 2.20e-1 2.41e-1 2.62e-1 2.76e-1 2.72e-1 2.75e-1 SOP 1.19e-1 1.36e-1 2.07e-1 2.33e-1 2.53e-1 2.50e-1 2.64e-1 2.71e-1 LSM 9.53e-2 1.05e-1 1.81e-1 2.22e-1 2.27e-1 2.43e-1 2.48e-1 2.65e-1 Pix2Pix 1.02e-1 1.21e-1 2.00e-1 2.45e-1 2.49e-1 2.71e-1 2.64e-1 2.89e-1 CycleGAN 2.30e-1 2.30e-1 2.30e-1 2.30e-1 2.30e-1 2.30e-1 2.30e-1 2.30e-1
take into account this discrepancy and falls behind the baseline. We suppose that IODA is able to perform well in this kind of setting by providing a good parameter initialization without constraining the later learning of the model.

Table 2: mIoU evaluation on the synthetic image translation dataset according to the portion of available pairs (x, y) from the dataset

Model 100 % 60 % 10 % 5 % 4 % 3 % 2 % 1 % Baseline 5.35e-1 4.25e-1 3.25e-1 2.93e-1 3.03e-1 3.01e-1 2.86e-1 2.69e-1 IODA 3.74e-1 3.63e-1 3.20e-1 3.03e-1 2.87e-1 2.82e-1 2.83e-1 2.48e-1 SOP 4.86e-1 4.01e-1 3.19e-1 3.05e-1 3.02e-1 3.02e-1 2.89e-1 2.72e-1 LSM 7.32e-1 6.50e-1 4.32e-1 3.85e-1 3.77e-1 3.75e-1 3.63e-1 3.27e-1 Pix2Pix 5.14e-1 4.32e-1 3.22e-1 3.02e-1 2.90e-1 3.03e-1 2.93e-1 2.97e-1 CycleGAN 3.03e-1 3.03e-1 3.03e-1 3.03e-1 3.03e-1 3.03e-1 3.03e-1 3.03e-1
Table 3: mIoU evaluation on the sarcopenia dataset according to the portion of available pairs (x, y) from the dataset Model 100 % 60 % 10 % 5 % 4 % 3 % 2 % 1 % Baseline 7.99e-1 7.81e-1 6.67e-1 6.31e-1 6.23e-1 5.89e-1 4.45e-1 4.12e-1 IODA 8.08e-1 7.90e-1 7.21e-1 6.36e-1 6.33e-1 6.13e-1 5.87e-1 4.03e-1 SOP 8.13e-1 8.06e-1 6.92e-1 5.18e-1 4.98e-1 4.46e-1 4.54e-1 3.80e-1 LSM 8.38e-1 8.29e-1 7.69e-1 7.39e-1 7.23e-1 7.21e-1 6.70e-1 6.11e-1 Pix2Pix 7.74e-1 7.82e-1 5.67e-1 5.42e-1 4.78e-1 5.09e-1 5.13e-1 4.53e-1 CycleGAN 7.18e-1 7.18e-1 7.18e-1 7.18e-1 7.18e-1 7.18e-1 7.18e-1 7.18e-1

Conclusions

We presented LSM, a novel training framework allowing to learn domain translation with multiple supervision levels, either fully supervised, or with views from only some domains. LSM learns for each domain a latent space and a mapping function between each of these latent spaces. Unlike previous approaches, LSM does not make the shared latent assumption directly, but rather relaxes it by supposing the existence of a mapping function between latent spaces. And further constraints each mapped latent code to be consistent with the original latent code from one domain. This makes LSM suitable for multi-modal domain translation tasks, especially when incomplete supervision is available.

The experiments confirmed that LSM is not penalizing when a lot of supervision is available and is able to leverage additional information in order to improve the training of models in low data settings.

While LSM allows learning the translation between multiple domains in a weakly-supervised setting, it does not leverage the availability of multiple related views to enhance the translation during the inference (LSM leverages the availability of multiple related views during training only). We plan to address this issue in future works which will combine information from multiple related views as input to improve the translation. Another issue that arises from LSM is the increasing model size as the number of domains grows, this makes such an approach heavy for a high number of domains. We study the real case example of facial landmark detection, on the 300 Faces In-the-Wild Challenge (300-W) [START_REF] Sagonas | faces in-the-wild challenge: The first facial landmark localization challenge[END_REF]32]. This 300 Faces In-the-Wild Challenge dataset is constituted of 300 indoor images and 300 outdoor images with associated 68 facial landmarks for each face. It provides a context with multiple modalities (image faces, and vector of positions) and with a highly structured output domain that presents strong intra-dependencies. We rescale each image into 64 × 64 pixels and normalize the pixels values in [0, 1] and the landmarks in [0, 1]. We train on 70% of the data, keep 10% for the validation, and 20% for the testing. We use the mean-square error for input reconstruction-related tasks and prediction-related tasks. We use Normalized Root Mean Squared Error (NRMSE in equation 13) [START_REF] Cristinacce | Feature detection and tracking with constrained local models[END_REF] as the metric for the prediction task.

N RM SE(ŷ, y) = N i=1 ŷi -y i 2 N × D (13)
With N = 68 the number of landmark points ŷ ∈ R N ×2 is the landmark prediction and y ∈ R N ×2 is the landmark ground-truth, and D is the inter-ocular distance computed from y. Segmentation of the third lumbar vertebra (L3) level is useful in order to predict, from CT scan (computed tomography scan), the sarcopenia level -characterizing a level of muscle atrophy -which is later used in cancer diagnostic [START_REF] Lanic | Sarcopenia is an independent prognostic factor in elderly patients with diffuse large B-cell lymphoma treated with immunochemotherapy[END_REF][START_REF] Martin | Cancer cachexia in the age of obesity: Skeletal muscle depletion is a powerful prognostic factor, independent of body mass index[END_REF]. The segmentation task is a tedious task for the medical practitioner as the hand-made segmentation can take 4 minutes for an expert [START_REF] Lerouge | Ioda: An input/output deep architecture for image labeling[END_REF]. Two challenges arise from the data, inter-patient variability, as there are different factors that will impact the CT scan (age, sex, disease . . . ); and inter-image variability, a lot of parameters exterior the patient can impact the CT scan quality (quality of the device, the radiation dosage . . . ). This is illustrated in Figure 6 with two samples of CT scans and their ground-truth segmentation that are really different from one another. Examples of data are given in Figure 6, there are 4 classes: background (black), subcutaneous adipose tissue (purple), visceral adipose tissue (green), and skeletal muscle (red). The skeletal muscle segmentation is used to compute the sarcopenia level. The dataset contains a total of 527 pairs of CT images and their masks, we train on 70% of the data, keeping 10% for the validation and 20% for the testing. For this task, the image's background is cropped, they are resized to 128 × 128 pixels and normalized between 0 and 1. We use the crossentropy as the loss function for the segmentation-related tasks, mean-squareerror for input reconstruction-related tasks. As segmentation metric, we use the mean intersection-over-union, computed on the three classes and not on the background.

B Experimental Settings Details

We reuse code and architecture from CycleGAN and Pix2Pix implementation 6 . The encoders-decoders are made of resnet 6-blocks [39], and the discriminators are made of PatchGAN [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] for the SSS and SIT datasets.

For the 300-W dataset and the facial landmark domain (made of 68 2d coordinates), encoders are multilayer perceptron of dimension [68 × 2, 64, 256, 1024] and the opposite for the decoders. The link network L x→ỹ is made of a flattening layer, followed by a linear layer.

For Pix2Pix, for the SSS and SIT datasets, the output and the input are concatenated before going through the discriminator. For the 300-W dataset where the outputs are facial landmarks, they are repeated and concatenated on each spatial dimension of the input image before going through the discriminator.

We use Adam optimizer with β 1 = 0.9 and β 2 = 0.999 for all datasets and set the learning rate as a hyperparameter to optimize. For the SSS dataset, SIT dataset, and 300-W dataset respectively batch sizes of 16, 16, and 32 are used. We allow a computational budget of 1000 epochs with early stopping and patience of 100 epochs. Hyperparameter tuning is done using bayesian optimization, for the learning rate within the range [1e -6, 1e -1] in log space, and for each lambda in the range [1e -4, 10] in log space. The search is then run for the proportions N = 100%, 50%, and 5% on 40 runs each, we then use the best hyperparameters for each proportion of data. Table 4 list the functions used for each loss and dataset. For further detail, please refer to the GitHub implementation 7 .

C LSM Training complexity

We study the possible training overhead that LSM might induce in terms of the number of epochs. For each model, a training budget of 1000 epochs was available and for IODA, we also allow 1000 epochs for each pre-training stage. We compare how the different approaches behave within this computational budget and for each dataset. We display results in Figure 12.

We did not find that LSM uses an excessive number of epochs before early stopping (with a patience of 100 epochs), especially when compared to pretraining methods such as IODA, with an additional training stage, which can take a lot of training.

For methods using Generative Adversarial Networks such as CycleGAN and Pix2Pix, we consume all the training budget but display the best epoch per run. For unsupervised method as CycleGAN, it is not possible to perform early stopping as there is no ground truth available to compute a metric on an unpaired dataset in practice.

D Distance and Adversarial Loss Ablation

We study how the different losses contribute to the model performances while varying the amount of available supervision. We name each ablation according to the format λ 1 λ 2 λ 3 λ 4 where λ i = 1 if λ i = 0. We found that, among all configurations, the one using every loss, has the lower NRMSE and the higher mIoU. 5: Impact of losses ablation on LSM, on 300-W dataset on the NRMSE metric Model 100 % 60 % 20 % 5 % 3 % 1 % LSM 0000 1.25e-1 1.47e-1 1.98e-1 2.38e-1 2.59e-1 3.11e-1 LSM 1000 1.03e-1 1.13e-1 1.58e-1 2.54e-1 2.98e-1 3.22e-1 LSM 1100 1.05e-1 1.15e-1 1.71e-1 2.53e-1 2.67e-1 3.05e-1 LSM 1110 1.03e-1 1.13e-1 1.47e-1 2.51e-1 2.54e-1 2.93e-1 LSM 1111 9.53e-2 1.05e-1 1.46e-1 2.22e-1 2.43e-1 2.65e-1

D 1 D 2 D 3 Fig. 1 :

 231 Fig. 1: Set of three domains D 1 , D 2 and D 3 , each view can have or not have a corresponding view in the other domain, creating a situation of semi-supervised learning

Fig. 2 :

 2 Fig. 2: Two types of latent space are commonly used in domain translation (a) a single shared latent space, (b) or using multiple latent spaces

Fig. 4 :

 4 Fig. 4: SIT samples with their associated prediction from LSM for different amounts of available (x, y) views. More predictions in Appendix E

  [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] a supervised domain translation approach. Every baseline used has the same model architecture. More details on the implementation can be found in Appendix B.The basic baseline, is restricted to D y • L x→ỹ • E x as in Section 3.1, this corresponds to using only the input to output translation with the loss described in equation 2. IODA[START_REF] Lerouge | Ioda: An input/output deep architecture for image labeling[END_REF] model has a similar structure as our model and also exploits information from input data and output data, but the learning of additional information is performed offline during a pre-training phase. Namely IODA first pre-trains D x • E x and D y • E y . Then in a final phase, the modelD y • L x→ỹ • E x isinitialized using the pre-trained weights and training using equation 2. SOP [2] has a similar architecture, and uses information from input and output as a mean to regularize the training of equation 2. The difference between the SOP model and our model lies in the equation 7 and equation 8 that are not present in SOP. Namely, a distance cost brings closer representations from different origins, and an adversarial cost pushes closer different embedding to follow the same distribution. CycleGAN [39] learns domain translation in an unsupervised way. Unlike IODA, SOP, and LSM, it does not explicitly modelizes a latent space. The translation x → y is learned by a generator D y • L x→ỹ • E x and is enforced realistic using a domain discriminator, a cycle consistency further regularizes the translation. Pix2Pix [18] learns domain translation in a supervised way. Not explicitly modelizing a latent space, Pix2Pix learns a generator D y • L x→ỹ • E x . The supervision came from a discriminator that takes as input either (x, y) as the real examples or (x, D y •L x→ỹ •E x (y)) as the fakes examples, with an additional supervised loss between y and D y • L x→ỹ • E x (y).

Fig. 7 :

 7 Fig.7: Impact of the number of available pairs from (a) the 300-W dataset on the NRMSE metric (lower is better) and (b) the SSS dataset on the mIoU metric (higher is better). Areas around LSM line represent the standard deviation (5 trained models for each baseline)

  34. Thanh-Tung, H., Tran, T.: On catastrophic forgetting and mode collapse in generative adversarial networks (2020) 35. Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks (2015) 36. Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation (2017) 37. Vu, H.T.,Huang, C.C.: Domain adaptation meets disentangled representation learning and style transfer (2018) 38. Wu, Z., Han, X., Lin, Y.L., Uzunbas, M.G., Goldstein, T., Lim, S.N., Davis, L.S.: Dcan: Dual channel-wise alignment networks for unsupervised scene adaptation (2018) 39. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks (2020) A Dataset Details A.1 Synthetic Image Translation (SIT) DatasetThe synthetic image translation dataset consists of image pairs made by superposing a background texture and a ring from another texture 5 for the input domain and the associated circle segmentation for the output domain. Four parameters control the texture generation: circle x position, circle y position, circle radii, and circle thickness. In order to complexify the task, for the segmentation mask, we swapped the x position with the circle thickness and the y position with the circle radii. This changes the task from a segmentation task to a more challenging image-to-image translation task. We use the standard mean Intersection over Union (mIoU) as evaluation metric for the output prediction and generate a total of 500 images. Figure4displays two pairs of input image and ground truth. The images are 128 × 128 pixels (we randomly take a square in the texture 17 and 77 of size 128 × 128). Values are normalized in [0, 1], 500 samples are generated. We train on 70% of the data, keep 10% for the validation and 20% for the testing. We use the cross-entropy as the loss function for the segmentation-related tasks, mean-square-error for input reconstructionrelated tasks. As segmentation metric, we use the mean intersection-over-union, computed on the three classes and not on the background.

Fig. 9 :

 9 Fig. 9: Two couples of input image textures and their associated output. Here we swap generating parameter between the image and its segmentation creating an image-to-image translation task

Fig. 10 :

 10 Fig. 10: Faces with their associated facial landmarks from the 300-W dataset

Fig. 11 :

 11 Fig. 11: Two tuples of CT scans, their ground-truth segmentation mask from our dataset and the LSM translation

Fig. 12 :

 12 Fig. 12: Indices of the best epoch per model and per dataset using the fully labeled dataset (N = 100).

Fig. 13 :

 13 Fig.13: Impact of the losses on LSM on 300-W dataset on the NRMSE metric

Fig. 14 : 10 Fig. 16 :Fig. 17 :Fig. 18 :Fig. 19 :Fig. 20 :

 14101617181920 Fig. 14: Impact of the losses on LSM on SSS dataset on the mIoU metric

Table 4 :

 4 Used losses function for each dataset and branchDatasetLx→y Lx→x Ly→y LDist ỹ L Conf ỹ LDc

	Sarcopenia	CE	MSE CE	L1	BCE vs Uniform BCE
	Synthetic Segmentation BCE MSE BCE L1	BCE vs Uniform BCE
	Facial Landmark	MSE MSE MSE L1	BCE vs Uniform BCE

https://gitlab.insa-rouen.fr/tmayet/lsm-latent-space-mapping

https://www.ux.uis.no/∼tranden/brodatz.html Domain Translation via Latent Space Mapping

https://www.ux.uis.no/∼tranden/brodatz.html

https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

https://gitlab.insa-rouen.fr/tmayet/lsm-latent-space-mapping

Acknowledgments

This work was financially supported by the ANR Labcom Lisa ANR-20-LCV1-0009. We also thank our colleagues at CRIANN, who provided us with the computation resources necessary for our experiments.

Table 6: Impact of losses ablation on LSM, on SSS dataset on the mIoU metric Model 100 % 60 % 20 % 5 % 3 % 1 % LSM 0000 7.99e-1 7.81e-1 7.40e-1 6.31e-1 5.89e-1 4.12e-1 LSM 1000 8.30e-1 8.23e-1 7.85e-1 6.85e-1 4.39e-1 1.72e-1 LSM 1100 8.31e-1 8.20e-1 7.85e-1 7.04e-1 6.80e-1 3.60e-1 LSM 1110 5.52e-1 8.20e-1 7.92e-1 7.16e-1 4.54e-1 3.96e-1 LSM 1111 8.38e-1 8.29e-1 7.96e-1 7.39e-1 7.21e-1 6.11e-1 3 % 1 % LSM 0000 5.35e-1 4.25e-1 3.43e-1 2.93e-1 3.01e-1 2.69e-1 LSM 1000 4.08e-1 3.76e-1 3.26e-1 3.16e-1 3.04e-1 2.91e-1 LSM 1100 3.99e-1 3.56e-1 3.26e-1 2.96e-1 3.00e-1 2.88e-1 LSM 1110 4.03e-1 3.73e-1 3.23e-1 3.16e-1 3.00e-1 2.90e-1 LSM 1111 7.32e-1 6.50e-1 4.94e-1 3.85e-1 3.75e-1 3.27e-1

E Qualitative Results

We provide LSM prediction for each dataset and different levels of supervision to access the evolution of prediction quality according to the available supervision.

X

GT N=100% N=20% N=1%

Fig. 21: Predicted sample from LSM on SSS test set, with different proportion of (x, y) available (100%, 20%, 1%)