Cyprien Ruffino

Romain Hérault

Eric Laloy

Gilles Gasso

Pixel-wise Conditioned Generative Adversarial Networks for Image Synthesis and Completion

Keywords: deep generative models, generative adversarial networks, conditional GAN

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Generative modelling is the process of modelling a distribution in a highdimension space in a way that allows sampling in it. Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF] have been the state of the art in unsupervised image generation for the past few years, being able to produce realistic images with high resolution [START_REF] Brock | Large scale gan training for high fidelity natural image synthesis[END_REF] without explicitly modelling the samples distribution. GANs learn a mapping function of vectors drawn from a low dimensional latent distribution (usually normal or uniform) to high dimensional ground truth images issued from an unknown and complex distribution. By using a discrimination function that distinguishes real images from generated ones, GANs setups a min max game able to approximate a Jensen-Shannon divergence between the distributions of the real samples and the generated ones.

Among extensions of GANs, Conditional GAN (CGAN) [START_REF] Mirza | Conditional generative adversarial nets[END_REF] attempts to condition the generation procedure on some supplementary information y (such as the label of the image x) by providing y to the generation and discrimination functions. CGAN enables a variety of conditioned generation, such as classconditioned image generation [START_REF] Mirza | Conditional generative adversarial nets[END_REF], image-to-image translation [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Ting-Chun | High-resolution image synthesis and semantic manipulation with conditional gans[END_REF], or image inpainting [START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF]. On the other side, Ambient GAN [START_REF] Bora | Ambientgan: Generative models from lossy measurements[END_REF] aims at training an unconditional generative model using only noisy or incomplete samples y. Relevant application domain is high-resolution imaging (CT scan, fMRI) where image sensing may be costly. Ambient GAN attempts to produce unaltered images x which distribution matches the true one without accessing to the original images x. For the sake, Ambient GAN considers lossy measurements such as blurred images, images with removed patch or removed pixels at random (up to 95%). Following this setup, Pajot et al. [START_REF] Pajot | Unsupervised adversarial image reconstruction[END_REF] extend the learning strategy to enable the reconstruction instead of the generation of realistic images from similarly altered samples.

In the spirit of Ambient GAN, we consider in this paper an extreme setting of image generation when only a few pixels, less than a percent of the image size, are known and are randomly scattered across the image (see Fig. 1c). We refer to these conditioning pixels as a constraint map y. To reconstruct the missing information, we design a generative adversarial model able to generate high quality images coherent with given pixel values by leveraging on a training set of similar, but not paired images. The model we propose aims to match the distribution of the real images conditioned on a highly scarce constraint map, drawing connections with Ambient GAN while, in the same manner as CGAN, still allowing the generation of diverse samples following the underlying conditional distribution.

To make the generated images honoring the prescribed pixel values, we use a reconstruction loss measuring how close real constrained pixels are to their generated counterparts. We show that minimizing this loss is equivalent to maximizing the log-likelihood of the constraints given the generated image. Thereon we derive an objective function trading-off the adversarial loss of GAN and the reconstruction loss which acts as a regularization term. We analyze the influence of the related hyper-parameter in terms of quality of generated images and the respect of the constraints. Specifically, empirical evaluation on FashionM-NIST [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF] evidences that the regularization parameter allows for controlling the trade-off between samples quality and constraints fulfillment.

Additionally to show the effectiveness of our approach, we conduct experiments on CIFAR10 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], CelebA [START_REF] Liu | Deep learning face attributes in the wild[END_REF] or texture [START_REF] Jetchev | Texture synthesis with spatial generative adversarial networks[END_REF] datasets using various deep architectures including fully convolutional network. We also evaluate our method on a classical geological problem which consists of generating 2D geological images of which the spatial patterns are consistent with those found in a conceptual image of a binary fluvial aquifer [START_REF] Strebelle | Conditional simulation of complex geological structures using multiple-point statistics[END_REF] [START_REF] Laloy | Training-image based geostatistical inversion using a spatial generative adversarial neural network[END_REF]. Empirical findings reveal that the used architectures may lack stochasticity from the generated samples that is the GAN input is often mapped to the same output image irrespective of the variations in latent code [START_REF] Yang | Diversity-sensitive conditional generative adversarial networks[END_REF]. We address this issue by resorting to the recent PacGAN [START_REF] Lin | Pacgan: The power of two samples in generative adversarial networks[END_REF] strategy. As a conclusion, our approach performs well both in terms of visual quality and respect of the pixel constraints while keeping diversity among generated samples. Evaluations on CIFAR-10 and CelebA show that the proposed generative model always outperforms the CGAN approach on the respect of the constraints and either come close or outperforms it on the visual quality of the generated samples.

The remainder of the paper is organized as follows. In Section 2, we review the relevant related work focusing first on generative adversarial networks, their conditioned version and then on methods dealing with image generation and reconstruction from highly altered training samples. Section 3 details the overall generative model we propose. In Section 4, we present the experimental protocol and evaluation measures while Section 5 gathers quantitative and qualitative effectiveness of our approach. The last section concludes the paper.

The contributions of the paper are summarized as follows:

• We propose a method for learning to generate images with a few pixel-wise constraints.

• A theoretical justification of the modelling framework is investigated.

• A controllable trade-off between the image quality and the constraints' fulfillment is highlighted, • We showcase a lack of diversity in generating high-dimensional images which we solve by using PacGAN [START_REF] Lin | Pacgan: The power of two samples in generative adversarial networks[END_REF] technique. Several experiments allow to conclude that the proposed formulation can effectively generate diverse and high visual quality images while satisfying the pixel-wise constraints.

Image reconstruction with GAN in related works

The pursued objective of the paper is image generation using generative deep network conditioned on randomly scattered and scarce (less than a percent of the image size) pixel values. This kind of pixel constraints occurs in application domains where an image or signal need to be generated from very sparse measurements.

Before delving into the details, let introduce the notations and previous work related to the problem. We denote by X ∈ X a random variable and x its realization. Let p X be the distribution of X over X and p X (x) be its evaluation at x. Similarly p X|Y represents the distribution of X conditioned on the random variable Y ∈ Y.

Given a set of images x ∈ X = [-1, 1] n×p×c (see Figure 1a) drawn from an unknown distribution p X and a sparse matrix y ∈ Y = [-1, 1] n×p×c (Figure 1c) as the given constrained pixels, the problem consists in finding a generative model G with inputs z (a random vector sampled from a known distribution p Z over the space Z) and constrained pixel values y ∈ [-1, 1] n×p×c able to generate an image satisfying the constraints while likely following the distribution p X (see Figure 3). One of the state-of-the-art modelling framework for image generation is the Generative Adversarial Network. The seminal version of GAN [START_REF] Goodfellow | Generative adversarial nets[END_REF] learns the generative models in an unsupervised way. It relies on a game between a generation function G and a discrimination network D, in which G learns to produce realistic samples while D learns to distinguish real examples from generated ones (Figure 2a). Training GANs amounts to find a Nash equilibrium to the following min-max problem, min

G max D L(D, G) = E x∼p X log(D(x)) + E z∼p Z log(1 -D(G(z))) , (1)
where p Z is a known distribution, usually normal or uniform, from which the latent input z of G is drawn, and p X is the distribution of the real images.

Among several applications, the GANs was adapted to image inpainting task (Figure 1b). For instance Yeh et al. [START_REF] Yeh | Semantic image inpainting with deep generative models[END_REF] propose an inpainting approach which considers a pre-trained generator, and explores its latent space Z through an optimization procedure to find a latent vector z, which induces an image with missing regions filled in by conditioning on the surroundings available information. However, the method requires to solve a full optimization problem at inference stage, which is computationally expensive.

Other approaches (Figure 2) rely on Conditional variant of GAN (CGAN) [START_REF] Mirza | Conditional generative adversarial nets[END_REF] in which additional information y is provided to the generator and the discriminator (see Figure 2b). This leads to the following optimization problem adapted to CGAN min

G max D L(D, G) = E x∼p X y∼p Y |X log(D(x, y)) + E z∼p Z y∼p Y log(1-D(G(y, z), y)) . (2)
Although CGAN was initially designed for class-conditioned image generation by setting y as the class label of the image, several types of conditioning information can apply such as a full image for image-to-image translation [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] or partial image as in inpainting [START_REF] Yu | Generative image inpainting with contextual attention[END_REF]. CGAN-based inpainting methods rely on generating a patch that will fill up a structured missing part of the image and achieve impressive results. However they are not well suited to reconstruct very sparse and unstructured signal [START_REF] Demir | Patch-based image inpainting with generative adversarial networks[END_REF]. Additionally, these approaches learn to reconstruct a single sample instead of a full distribution, implying that there is no sampling process for a given constraint map or highly degraded image.

AmbientGAN [START_REF] Bora | Ambientgan: Generative models from lossy measurements[END_REF] (Figure 2c) trains a generative model capable to yield full images from only lossy measurements. One of the image degradations considered in this approach is the random removal of pixels leading to sparse pixel map y. It is simulated with a differentiable function f θ whose parameter θ indicates the pixels to be removed. The underlying optimization problem solved by AmbientGAN is therefore stated as min

G max D L(D, G) = E y∼p Y log(D(y)) + E z∼p Z θ∼p θ log(1 -D(f θ (G(z)))) . (3)
Pajot et al. [START_REF] Pajot | Unsupervised adversarial image reconstruction[END_REF] combined the AmbientGAN approach with an additional reconstruction task that consists in reconstructing the f θ (G(y)) from the twice-

altered image ỹ = f θ (G(y)) and ŷ = f θ (G(f θ (G(y)))), min G max D L(D, G) = E y∼p Y log(D(y)) + E y∼p Y log(1 -D(ŷ)) + ||ŷ -ỹ|| 2 2 . (4)
The 2 norm term ensures that the generator is able to learn to revert f θ i.e. to revert the alteration process on a given sample. This allows the reconstruction of realistic image only from a given constraint map y. However the reconstruction process is deterministic and does not provide a sampling mechanism.

Compressed Sensing with Meta-Learning [START_REF] Wu | Deep compressed sensing[END_REF] is an approach that combines the exploration of the latent space Z to recover images from lossy measurements with the enforcing of the Restricted Isometric Property [START_REF] Candes | Decoding by linear programming[END_REF], which states that for two samples

x 1 , x 2 ∼ p X , (1 -α)||x 1 -x 2 || 2 2 ≤ ||f θ (x 1 -x 2 )|| 2 2 ≤ (1 + α)||x 1 -x 2 || 2 2
where α is a small constant. It replaces the adversarial training of the generative model G (Eq. 1) by searching, for a given degraded image y, a vector ẑ such that ŷ = f θ (G(ẑ)) minimizes the 2 distance between y and ŷ while still enforcing the RIP. The overall problem induced by this approach can be formulated as:

min G L(G) = E x∼p X y∼p Y z∼p Z x1,x2∈S x1 =x2 (||f θ (x 1 -x 2 )|| 2 2 -||x 1 -x 2 || 2 2 ) 2 /3+||y-f θ (G(ẑ))|| 2 2
where ẑ = min

z ||y -f θ (G(z))|| 2 . ( 5
)
where S contains the three samples x, G(z), G(ẑ). In practice, ẑ is computed with gradient descent on z by minimizing ||y -f θ (G(z))|| 2 , and starting from a random z ∼ p Z . As a benefit, this approach may generate an image x = G(ẑ) from a noisy information y but at a high computation burden since it requires to solve an optimization problem (computing ẑ) at inference stage for generating an image.

D G x z GAN cost (a) GAN D G x z GAN cost y (b) CGAN D G y z GAN cost fθ (c) AmbientGAN D G x z GAN cost y L2 cost (d) Our approach

Proposed approach

Let introduce the formal formulation of the addressed problem. Assume y is the given set of constrained pixel values. To ease the presentation, let consider y as a n × p × c image with only a few available pixels (less than 1% of n × p × c). We will also encode the spatial location of these pixels using a corresponding binary mask M (y) ∈ {0, 1} n×p×c . We intend to learn a GAN whose generation network takes as input the constraint map y and the sampled latent code z ∈ Z and outputs a realistic image that fulfills the prescribed pixel values. Within this setup, the generative model can sample from the unknown distribution p X of the training images {x 1 , • • • , x N } while satisfying unseen pixel-wise constraints at training stage. Formally our proposed GAN can be formulated as min

G max D L(D, G)= E x∼px log(D(x)) + E z∼p Z y∼p Y log(1-D(G(y, z))) , (6)
s.t. y = M (y) G(y, z)
where stands for the Hadamard (or point-wise) product and M (y) for the mask, a sparse matrix with entries equal to one at constrained pixels location.

As the equality constraint in Problem ( 6) is difficult to enforce during training, we rather investigate a relaxed version of the problems. Following Pajot et al. [START_REF] Pajot | Unsupervised adversarial image reconstruction[END_REF] we assume that the constraint map is obtained through a noisy measurement process

y = f M (x) + ε . (7)
Here f M is the masking operator yielding to y = M (y) x. Also the constrained pixels are randomly and independently selected. ε represents an additive i.i.d noise corrupting the pixels. Therefore we can formulate the Maximum A Posteriori (MAP) estimation problem, which, given the constraint map y, consists in finding the most probable image x * following the posterior distribution p X|Y ,

x * = arg max x log p X|Y (x|y) (8)
= arg max

x log p Y |X (y|x) + log p X (x) . (9)
p Y |X (y|x) is the likelihood that the constrained pixels y are issued from image x while p X (x) represents the prior probability at x. Assuming that the generation network G may sample the most probable image G(y, z) complying with the given pixel values y, we get the following problem

G * = arg max G E y∼p Y z∼p Z log p Y |X (y|G(y, z)) + log p X (G(y, z)) . (10)
The first term in Problem [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] measures the likelihood of the constraints given a generated image. Let rewrite Equation ( 7) as vect(y) = vect(f M (x)) + vect(ε) where vect(•) is the vectorisation operator that consists in stacking the constrained pixels. Therefore, assuming vect(ε) is an i.i.d Gaussian noise with distribution N (0, σ 2 I), we achieve the expression of the conditional likelihood

logp Y |X (y|G(y, z)) ∝ -vect(y) -vect(M (y) G(y, z)) 2 2 (11)
which evaluates the quadratic distance between the conditioning pixels and their predictions by G. In other words, using a matrix notation of ( 7), the likelihood of the constraints given a generated image equivalently writes

log p Y |X (y|G(y, z) ∝ -y -M (y) G(y, z) 2 F . (12)
A 2 F represents the squared Frobenius norm of matrix A that is the sum of its squared entries.

The second term in Problem [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] is the likelihood of the generated image under the true but unknown data distribution p X . Maximizing this term can be equivalently achieved by minimizing the distance between p X and the marginal distribution of the generated samples G(y, z). This amounts to minimizing with respect to G, the GAN-like objective function Ex∼pX log(D [START_REF] Goodfellow | Generative adversarial nets[END_REF]. Putting altogether these elements, we can propose a relaxation of the hard constraint optimization problem (6) (Figure 2d) as follows min

(x)) + E z∼p Z y∼p Y log(1 - D(G(y, z)))
G max D L(D, G) = E x∼p X log(D(x)) (13) + E z∼p Z y∼p Y log(1 -D(G(y, z))) + λ y -M (y) G(y, z) 2 F
. Remarks:

• The assumption of Gaussian noise measurement leads us to explicitly turn the pixel value constraints into the minimization of the 2 norm between the real enforced pixel values and their generated counterparts (see Figure 2d).

• This additional term acts as a regularization over prescribed pixels by the mask M (y). The trade-off between the distribution matching loss and the constraint enforcement is assessed by the regularization parameter λ ≥ 0.

• It is worth noting that the noise ε can be of any other distribution, according to the prior information, one may associate to the measurement process. We only require this distribution to admit a closed-form solution for the maximum likelihood estimation for optimization purpose. Typical choices are distributions from the exponential family [START_REF] Lawrence D Brown | Fundamentals of statistical exponential families: with applications in statistical decision theory[END_REF].

To solve Problem (13), we use the stochastic gradient descent method. The overall training procedure is detailed in Algorithm 1 and ends up when a maximal number of training epochs is attained.

When implementing this training procedure we experienced, at inference stage, a lack of diversity in the generated samples (see Figure 5) with deeper architectures, most notably the encoder-decoder architectures. This issue manifests itself through the fact that the learned generation network, given a constraint map y, outputs almost deterministic image regardless the variations in the input z. The issue was also pointed out by Yang et al. [START_REF] Yang | Diversity-sensitive conditional generative adversarial networks[END_REF] as characteristic of CGANs.

To avoid the problem, we exploit the recent PacGAN [START_REF] Lin | Pacgan: The power of two samples in generative adversarial networks[END_REF] technique: it consists in passing a set of samples to the discrimination function instead of a single one. PacGAN is intended to tackle the mode collapse problem in GAN training. The underlying principle being that if a set of images are sampled from the same training set, they are very likely to be completely different, whereas if the generator experiences mode collapse, generated images are likely to be similar. In practice, we only give two samples to the discriminator, which is sufficient to overcome the loss of diversity as suggested in [START_REF] Lin | Pacgan: The power of two samples in generative adversarial networks[END_REF]. The resulting training procedure is summarized in Algorithm 2.

Algorithm 1 Proposed training algorithm

Require: D X the set of unaltered images, D Y the set of constraint maps, G the generation network, and D the discrimination function repeat sample a mini-batch

{x i } m i=1 from D X sample a mini-batch {y i } m i=1 from D Y sample a mini-batch {z i } m i=1 from distribution p Z update D by stochastic gradient ascent of m i=1 log(D(x i )) + log(1 -D(G(y i , z i ))) sample a mini-batch {y j } n j=1 from D Y sample a a mini-batch {z j } n j=1 from distribution p Z ; update G by stochastic gradient descent of n j=1 log(1 -D(G(y j , z j ))) + ||y j -M (y j ) G(y j , z j )|| 2
a i } m i=1 , {x b i } m i=1 from D X sample a mini-batch {y i } m i=1 from D Y sample two mini-batches {z a i } m i=1 , {z b i } m i=1 from distribution p Z update D by stochastic gradient ascent of m i=1 log(D(x a i , x b i )) + log(1 -D(G(y i , z a i ), G(y i , z b i ))) sample a mini-batch {y j } n j=1 from D Y sample two mini-batches {z a i } m i=1 , {z b i } m i=1 from distribution p Z update G by stochastic gradient descent of n j=1 log(1 -D(G(y j , z j ))) + ||y j -M (y j ) G(y j , z j )|| 2 F
until a stopping condition is met

Experiments

We have conducted a series of empirical evaluation to assess the performances of the proposed GAN. Used datasets, evaluation protocol and the tested deep architectures are detailed in this section while Section 5 is devoted to the results presentation.

Datasets

We tested our approach on several datasets listed hereafter. Detailed information on these datasets are provided in the Appendix A.

FashionMNIST [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF] consists of 60,000 28 × 28 small grayscale images of fashion items, split in 10 classes and is a harder version of the classical MNIST dataset [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. The very small size of the images makes them particularly appropriate for large-scale experiments, such as hyper-parameter tuning.

CIFAR10 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] consists of 60,000 32×32 colour images of 10 different and varied classes. It is deemed less easy than MNIST and FashionMnist

CelebA [START_REF] Liu | Deep learning face attributes in the wild[END_REF] is a large dataset of celebrity portraits labeled by identity and a variety of binary features such as eyeglasses, smiling... We use 100,000 images cropped to a size of 128 × 128, making this dataset appropriate for a high dimension evaluation of our approach in comparison with related work.

Texture is a custom dataset composed of 20, 000 160 × 160 patches sampled from a large brick wall texture, as recommended in [START_REF] Jetchev | Texture synthesis with spatial generative adversarial networks[END_REF]. It is worth noting that this procedure can be reproduced on any texture image of sufficient size. Texture is a testbed of our approach on fully-convolutional networks for constrained texture generation task.

Subsurface is a classical dataset in geological simulation [START_REF] Strebelle | Conditional simulation of complex geological structures using multiple-point statistics[END_REF] which consists, similarly to the Texture dataset, of 20,000 160×160 patches sampled from a model of a subsurface binary domain. These models are assumed to have the same properties as a texture, mainly the property of global ergodicity of the data.

To avoid learning explicit pairing of real images seen by the discrimination function with constraint maps provided to the generative network, we split each dataset into training, validation and test sets, to which we add a set composed of constraint maps that should remain unrelated to the three others. In order to do so, a fifth of each set is used to generate the constrained pixel map y by randomly selecting 0.5% of the pixels from a uniform distribution, composing a set of constraints for each of the train, test and validation sets. The images from which these maps are sampled are then removed from the training, testing and validation sets. For each carried experiment the best model is selected based on some performance measures (see Section 4.3) computed on the validation set, as in the standard of machine learning methodology [START_REF] Oneto | Model Selection and Error Estimation in a Nutshell[END_REF]. Finally, reported results are computed on the test set.

Network architectures

We use a variety of GAN architectures in order to adapt to the different scales and image sizes of our datasets. The detailed configuration of these architectures are exposed in Appendix B.

For the experiments on the FashionMNIST [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF], we use a lightweight network for both the discriminator and the generator similarly to DCGAN [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF] due to the small resolution of FashionMnist images.

To experiment on the Texture dataset, we consider a set of fully-convolutional generator architectures based on either dilated convolutions [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF], which behave well on texture datasets [START_REF] Ruffino | Dilated spatial generative adversarial networks for ergodic image generation[END_REF], or encoder-decoder architectures that are commonly used in domain-transfer applications such as CycleGAN [START_REF] Zhu | Unpaired image-toimage translation using cycle-consistent adversarial networks[END_REF]. We selected these architectures because they have very large receptive fields without using pooling, which allow the generator to use a large context for each pixel.

We keep the same discriminator across all the experiments with these architectures, the PatchGAN discriminator [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], which is a five-layer fully-convolutional network with a sigmoid activation.

The Up-Dil architecture consists in a set of transposed convolutions (the upscaling part), and a set of dilated convolutional layers [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF], while the Up-EncDec has an upscaling part followed by an encoder-decoder section with skipconnections, where the constraints are downscaled, concatenated to the noise, and re-upscaled to the output size.

The UNet [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] architecture is an encoder-decoder where skip-connections are added between the encoder and the decoder. The Res architecture is an encoderdecoder where residual blocks [START_REF] He | Deep residual learning for image recognition[END_REF] are added after the noise is concatenated to the features. The UNet-Res combines the UNet and the Res architectures by including both residual blocks and skip-connections.

Finally, we will evaluate our approach on the Subsurface dataset using the architecture that yields to the best performances on the Texture dataset.

Evaluation

We evaluate our approach based on both the satisfaction of the pixel constraints and the visual quality of sampled images. From the assumption of Gaussian measurement noise (as discussed in Section 3), we assess the constraint fulfillment using the following mean square error (MSE)

M SE = 1 L L i=1 y i -M (y i ) G(y i , z i ) 2 F (14)
This metric should be understood as the mean squared error of reconstructing the constrained pixel values.

Visual quality evaluation of an image is not a trivial task [START_REF] Theis | A note on the evaluation of generative models[END_REF]. However, Fréchet Inception Distance (FID) [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF] and Inception Score [START_REF] Salimans | Improved techniques for training gans[END_REF], have been used to evaluate the performance of generative models. We employ FID since the Inception Score has been shown to be less reliable [START_REF] Barratt | A note on the inception score[END_REF]. The FID consists in computing a distance between the distributions of relevant features extracted from generated and real samples. To extract these features, a pre-trained Inception v3 [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF] classifier is used to compute the embeddings of the images at a chosen layer. Assuming these embeddings shall follow a normal distribution, the quality of the generated images is assessed in term of a Wasserstein-2 distance between the distribution of real samples and generated ones. Hence the FID writes

F ID = ||µ r -µ g || 2 + T r(Σ r + Σ g -2(Σ r Σ g ) 1/2 ), ( 15
)
where T r is the trace operator, (µ r , Σ r ) and (µ g , Σ g ) are the pairs of mean vector and covariance matrice of embeddings obtained on respectively the real and the generated data. Being a distance between distributions, a small FID corresponds to a good matching of the distributions.

Since the FID requires a pre-trained classifier adapted to the dataset in study, we trained simple convolutional neural networks as classifiers for the FashionMNIST and the CIFAR-10 datasets. For the Texture dataset, since the dataset is not labeled, we resort to a CNN classifier trained on the Describable Textures Dataset (DTD) [START_REF] Cimpoi | Describing textures in the wild[END_REF], which is a related application domain.

However, since we do not have labels for the Subsurface dataset, we could not train a classifier for this dataset, thus we cannot compute the FID. To evaluate the quality of the generated samples, we use metrics based on a distance between feature descriptors extracted from real samples and generated ones. Similarly to [START_REF] Ruffino | Dilated spatial generative adversarial networks for ergodic image generation[END_REF], we rely on a χ 2 distance between the Histograms of Oriented Gradients (HOG) or Local Binary Patterns (LBP) features computed on generated and real images.

Histograms of Oriented Gradients (HOG) [START_REF] Dalal | Histograms of Oriented Gradients for Human Detection[END_REF] and Local Binary Patterns (LBP) [START_REF] Pietikäinen | Computer Vision Using Local Binary Patterns[END_REF] are computed by splitting an image into cells of a given radius and computing on each cell the histograms of the oriented gradients for HOGs and of the light level differences for each pixel to the center of the cell for LBPs. Additionally, we consider the domain-specific metric, the connectivity function [START_REF] Lemmens | Effective structural descriptors for natural and engineered radioactive waste confinement barrier[END_REF] which is presented in Appendix C.

Finally, we check by visual inspection if the trained model G is able to generate diverse samples, meaning that for a given y and for a set of latent codes (z 1 , ..., z n ) ∼ p Z , the generated samples G(y, z 1 ), . . . , G(y, z n ) are visually different.

Experimental results

Quality-fidelity trade-off

We first study the influence of the λ regularization hyper-parameter on both the quality of the generated samples and the respect of the constraints. We experiment on the FashionMNIST [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF] dataset, since such a study requires intensive simulations permitted by the low resolution of FashionMnist images and the used architectures (see Section 4.2).

To overcome classical GANs instability, the networks are trained 10 times and the median values of the best scores on the test set at the best epoch are recorded. The epoch that minimizes:

F ID -F ID min F ID max -F ID min 2 + M SE -M SE min M SE max -M SE min 2
on the validation set is considered as the best epoch, where F ID min , M SE min , F ID max and M SE max are respectively the lowest and highest FIDs and MSEs obtained on the validation set. Empirical evidences (highlighted in Figure 4) show that with a good choice of λ, the regularization term helps the generator to enforce the constraints, leading to smaller MSEs than when using the CGAN (λ = 0) without compromising on the quality of generated images. Also, we can note that using the regularization term even leads to a better image quality compared to GAN and CGAN. The bottom panel in Figure 4 illustrates that the trade-off between image quality and the satisfaction of the constraints can be controlled by appropriately setting the value of λ. Nevertheless, for small values of λ (less or equal to 10 -1 ), our GAN model fails to learn meaningful distribution of the training images and only generates uniformly black images. This leads to the plateaus on the MSE and FID plots (top panels in Figure 4).

Texture generation with fully-convolutional architectures

Fully-convolutional architectures for GANs are widely used, either for domaintransfer applications [28][4] or for texture generation [START_REF] Jetchev | Texture synthesis with spatial generative adversarial networks[END_REF]. In order to evaluate the efficiency of our method on relatively high resolution images, we experiment the fully-convolutional networks described in Section 4.2 on a texture generation task using Texture dataset. We investigate the upscaling-dilatation network, the encoder-decoder one and the resnet-like architectures.

Our training algorithm was run for 40 epochs on all reported results. We provide a comparison to CGAN [START_REF] Mirza | Conditional generative adversarial nets[END_REF] approach by using the selected best architectures. The models are evaluated in terms of best FID (visual quality of sampled Figure 5: An example of a loss of diversity when generating Texture samples with a trained UNetRes network using two different random noises z and a single constraint map y. The two samples on the top left are generated using the classical GAN discriminator whereas the samples on the top right are generated using the PacGAN approach. The loss of diversity is clearly visible on the absolute differences between the greyscaled images (bottom).

images) at each epoch and MSE (conditioning on fixed pixel values)

. We also compute the FID score of the models at the epochs where the MSE is the lowest. In the other way around, the MSE is reported at epoch when the FID is the lowest. The obtained quantitative results are detailed in Table 1.

For the encoder-decoder models, we can notice that the models using ResNet blocks perform better than just using a UNet generator. A trade-off can also be seen between the FID and MSE for the ResNet models and the UNet-ResNet, which could mean that skip-connections help the generator to fulfill the constraints but at the price of lowered visual quality.

Although the encoder-decoder models perform the best, they tend to lose diversity in the generated samples (see Figure 5), whereas the upscaling-based models have high FID and MSE but naturally preserve diversity in the generated samples.

Changing the discriminator for a PacGAN discriminator with 2 samples in the encoder-decoder based architectures allows to restore diversity, while keeping the same performances as previously or even increasing the performances for the UNetRes (see Table 1).

Table 2 compares our proposed approach to CGAN using fully convolutional networks. It shows that our approach is more able to comply with the pixel constraints while producing realistic images. Indeed, our approach outperforms CGAN (see Table 2) by a large margin on the respect of conditioning pixels (see the achieved MSE metrics by our UNetPAC or UNetResPAC) and gets close FID performance on the generated samples. This finding is in accordance of the obtained results on FashionMnist experiments. 2: Results obtained by the selected best fully-convolutional architectures on the Texture dataset for both the CGAN approach and our approach.

Model

Extended architectures

We extend the comparison of our approach to CGAN on the CIFAR10 and CelebA datasets (Table 3). We investigated the architectures described in Section 4.2. All reported results are obtained with the regularization parameter fixed to λ = 1. We train the networks for 150 epochs using the same dataset split as stated previously in order to keep independence between the images constraint maps. The evaluation procedure remains also unchanged. We use the PacGAN approach to avoid the loss of diversity issues. The experiments on both datasets show that though CGAN provides better results in terms of visual quality, our approach outperforms it according to the respect of the pixel constraints.

Application to hydro-geology

Finally, we evaluate our approach on the Subsurface dataset. We use the UNetResPAC architecture, since it performed the best on Texture data as exposed in Section 5.2. As previously, we simply set the regularization parameter at λ = 1 and, the network is trained for [START_REF] Marafioti | A context encoder for audio inpainting[END_REF] protocol. To evaluate the trade-off between the visual quality and the respect of the constraints, instead of FID we rather compute distances between visual Histograms of Oriented Gradients (see Section 4), extracted from real and generated samples. We also evaluate the visual quality of our approach with a distance between Local Binary Patterns. Indeed, Subsurface application lacks labelled data in order to learn a deep network classifier from which the FID score can be computed. The obtained results are summarized in Tables 4 and5. They are coherent with the previous experiments since the generated samples are diverse and have a low error regarding the constrained pixels. The conditioning have a limited impact on the visual quality of the generated samples and compares well to unconditional approaches [START_REF] Ruffino | Dilated spatial generative adversarial networks for ergodic image generation[END_REF]. Evaluation of the generated images using the domain-connectivity function highlights this fact on Figures 7 and7 in the supplementary materials. Also examples of generated images by our approach pictured in Figure 9 (see appendix D) show that we preserve the visual quality and honor the constraints.

Conclusion

In this paper, we address the task of learning effective generative adversarial networks when only very few pixel values are known beforehand. To solve this pixel-wise conditioned GAN, we model the conditioning information under a probabilistic framework. This leads to the maximization of the likelihood of the constraints given a generated image. Under the assumption of a Gaussian distribution over the given pixels, we formulate an objective function composed of the conditional GAN loss function regularized by a 2 -norm on pixel reconstruction errors. We describe the related optimization algorithm.

Empirical evidences illustrate that the proposed framework helps obtaining good image quality while best fulfilling the constraints compared to classical GAN approaches. We show that, if we include the PacGAN technique, this approach is compatible with fully-convolutional architectures and scales well to large images. We apply this approach to a common geological simulation task and show that it allows the generation of realistic samples which fulfill the prescribed constraints.

In future work, we plan to investigate other prior distributions for the given pixels as the Laplacian or β-distribtutions. We are also interested in applying the developed approach to other applications or signals such as audio inpainting [START_REF] Marafioti | A context encoder for audio inpainting[END_REF] Additional information:

• Instance normalization [START_REF] Ulyanov | Instance normalization: The missing ingredient for fast stylization[END_REF] is applied across all the layers instead of Batch normalization. This is involved by the use of the PacGAN technique.

• A Gaussian noise is applied to the input of the discriminator As for Cifar10, this network follows the same additional setup described in Appendix (B.2).

C. Domain-specific metrics for underground soil generation

In this section, we compute the connectivity function [START_REF] Lemmens | Effective structural descriptors for natural and engineered radioactive waste confinement barrier[END_REF] of generated soil image, a domain-specific metric, which is the probability that a continuous pixel path exists between two pixels of the same value (called Facies) in a given direction and a given distance (called Lag). This connectivity function should be similar to the one obtained on real-world samples. In this application, the connectivity function models the probability that two given pixels are from the same sand brick or clay matrix zone.

We sampled 100 real and 100 generated images using the UNetResPAC architecture (see Section 4.2) on which the connectivity function was evaluated for both the CGAN and our approach. The obtained graphs are shown respectively in Figures 6 and7.

The blue curves are the mean value for the real samples, and the blue dashed curves are the minimum and maximum values on these samples. The green curves are the connectivity functions for each of the 100 synthetic samples and the red curves are their mean connectivity functions. From these curves we observe that that our approach has similar connectivity functions as the CGAN approach while being significantly better at respecting the given constraints (see Section Table 4).

D. Additional samples from the Texture and Subsurface datasets

In this section, we show some samples generated with the UNetResPAC architecture, which performs the best in our experiments (see Section 5) compared to real images sampled from the Texture (Figure 8) and Subsurface (Figure 9) datasets. For the generated samples, the enforced pixel constraints are colored in the images, green corresponding to a squared error less than 0.1 and red otherwise.

Figure 1 :

 1 Figure 1: Difference between regular inpainting (1b) and the problem undertaken in this work (1c) on a real sample (1a).

Figure 2 :

 2 Figure 2: Different GAN Setups. G and D are the generator and discriminator networks, x and z are samples from the distributions Px and Pr, y is a label/constraint map sampled from Py and f θ is an image degradation function.

Figure 3 :

 3 Figure 3: Generation of a sample during training. We first sample an image from a training set (3a) and we sample the constraints (3b) from it. Then our GAN generates a sample (3c). The constraints with squared error smaller than = 0.1 are deemed satisfied and shown by green pixels in (3d) while the red pixels are unsatisfied.

F

  until a stopping condition is met Algorithm 2 Our training algorithm including PacGAN Require: D X the set of unaltered images, D Y the set of constraint maps, G the generation network, and D the discrimination function repeat sample two mini-batches {x

Figure 4 :

 4 Figure 4: Our approach compared to the GAN and CGAN baselines. MSE (left) and FID (right) w.r.t. the regularization parameter λ, MSE w.r.t the FID (bottom).

Figure 6 :

 6 Figure 6: Connectivity curves obtained on 100 samples generated with the CGAN approach.

Figure 7 :

 7 Figure 7: Connectivity curves obtained on 100 samples generated with our approach.

Table 1 :

 1 Results obtained by the different fully-convolutional architectures on the Texture dataset. We can remark that the encoder-decoder greatly outperforms the upscaling ones and that using the PacGAN technique helps keeping the performance of these models while restoring the diversity in the samples. The bottom part of the table refers to PacGan architectures.

		Best FID Best MSE	FID at	MSE at	Diversity
					best MSE best FID
	Up-Dil	0.0949	0.4137		1.0360	0.7057
	Up-EncDec	0.1509	0.7570		0.2498	0.9809
	UNet	0.0442	0.1789		0.0964	0.4559
	Res	0.0458	0.0474		0.0590	0.0476
	UNetRes	0.0382	0.0307		0.0499	0.0338
	ResPAC	0.0350	0.0698		0.0466	0.4896
	UNetPAC	0.0672	≤ 0.0001	0.3120	0.2171
	UNetResPAC	0.0431	0.0277		0.0447	0.0302
	Model		Best FID Best MSE	FID at	MSE at
						best MSE best FID
	CGAN-ResPAC	0.0234		0.1337	0.0340	0.2951
	CGAN-UNetPAC	0.0518		0.2010	0.0705	0.4828
	CGAN-UNetResPAC	0.0428		0.1060	0.0586	0.2250
	Ours-ResPAC		0.0350		0.0698	0.0466	0.4896
	Ours-UNetPAC	0.0672	≤ 0.0001	0.3120	0.2171
	Ours-UNetResPAC	0.0431		0.0277	0.0447	0.0302
	Table

Table 3 :

 3 epochs using the same experimental Results on the CIFAR10 and CelebA datasets. The reported performances compare CGAN to our proposed GAN conditioned on scarce constraint map.

		Model Best FID Best MSE	FID at	MSE at
					best MSE best FID
	CIFAR-10 CGAN	2,68	0.081	2.68	0.081
		Ours	3.120	0.010	3.530	0.011
	CelebA	CGAN 1.34e-4	0.0209	1.81e-4	0.0450
		Ours	2.09e-4	0.0053	5.392e-4	0.0249
		Model Best HOG Best MSE	HOG at	MSE at
					best MSE best HOG
	Subsurface CGAN	2.92e-4	0.2505	3.06e-4	1.1550
		Ours	4.31e-4	0.0325	5.69e-4	0.2853

Table 4 :

 4 Evaluation of the trade-off between the visual quality of the generated samples and the respect of the constraints for the CGAN approach and ours on the Subsurface dataset.

Table 5 :

 5 Evaluation of the visual quality between the CGAN approach and ours on the Subsurface dataset using several metrics.

	Model Best HOG Best MSE Best LBP	Best LBP
				(radius=1) (radius=2)
	Subsurface CGAN	2.92e-4	0.2505	2.157	3.494
	Ours	4.31e-4	0.0325	10.142	16.754

  .

	B.2. UNet-Res for CIFAR10
	Layer type	Units	Scaling	Activation	Output shape
	Input y	-	-	-	32x32
	Conv2D*	64 5x5	x1	ReLU	32x32
	Conv2D*	128 3x3	x1/2	ReLU	16x16
	Conv2D*	256 3x3	x1/2	ReLU	8x8
	Input z	-	-	-	8x8
	Dense	256	-	ReLU	8x8
	Residual block	3x256 3x3	x1	ReLU	8x8
	Residual block	3x256 3x3	x1	ReLU	8x8
	Residual block	3x256 3x3	x1	ReLU	8x8
	Residual block	3x256 3x3	x1	ReLU	8x8
	Conv2DTranspose*	256 3x3	x2	ReLU	16x16
	Conv2DTranspose*	128 3x3	x2	ReLU	32x32
	Conv2DTranspose*	64 3x3	x1	ReLU	32x32
	Conv2D	3 3x3	x1	tanh	32x32
	Input x	-	-	-	32x32
	Input y	-	-	-	32x32
	Conv2D	64 3x3	x1/2	LeakyReLU	16x16
	Conv2D	128 3x3	x1/2	LeakyReLU	8x8
	Conv2D	256 3x3	x1/2	LeakyReLU	4x4
	Dense	1	-	Sigmoid	1

Acknowledgements

This research was supported by the CNRS PEPS I3A REGGAN project and the ANR-16-CE23-0006 grant Deep in France. We kindly thank the CRIANN for the provided high-computation facilities.

Additional information:

• Batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] is applied across all the layers