Kailin Chia
email: kai.chia@monash.edu

Koksheik Wong
email: wong.koksheik@monash.edu

Jean-Luc Dugelay
email: jean-luc.dugelay@eurecom.fr

Data Hiding in Perceptually Masked OpenEXR Image

Keywords: HDR, OpenEXR, perceptual masking, data insertion

High dynamic range (HDR) imaging is able to capture and display significantly more colors when compared to the legacy imaging technology. Recently, HDR environment is gaining popularity and becoming common in our daily life, which resulted in the growing number of HDR images. In this work, a technique is put forward to first perceptually mask a HDR image, and then hide data into the masked HDR image. Specifically, pixels in the OpenEXR file, which are stored in half floating point precision format, are manipulated. First, a predictor is utilized to predict pixel values, where well predicted pixel locations are flagged and utilized as the venues to hide data. On the other hand, the ill predicted pixel locations are flagged as unusable. Next, each pixel is divided into segments of 5 bits, and the segments are XOR-ed and permuted to mask the perceptual semantic of the HDR image. Data hiding then takes place at locations flagged as usable, which further distorts the quality of the image. The proposed method is both reversible and separable. The basic performance of the proposed joint technique is evaluated by using 6 HDR images. In the best case scenario, 91% of pixels in the image can be utilized for data hiding purpose while the perceptual semantic of the original image is completely masked.

I. INTRODUCTION

Perceptual masking aims to mask the semantic of a content, and it has been heavily relied on for the purpose of secure communication as well as privacy preservation. Since contents are communicated, shared, and stored online these days, digital content encryption, particularly image, audio and video, have received much attention from the research community [START_REF] Kulkarni | Multimedia Encryption: A Brief Overview[END_REF]. On the other hand, data hiding is the act of inserting data into a digital content to serve specific purpose [START_REF] Cox | Digital watermarking and steganography[END_REF]. Some of the typical applications of data hiding include watermarking, steganography, tamper detection and hyperlinking. Data hiding can be applied to any digital content, including traditional contents such as audio, image, video and text document, or untraditional contents such as IP packets [START_REF] Abdullaziz | AIPISteg: An active IP identification based steganographic method[END_REF] or laser beam [START_REF] Yue | High-resolution grayscale image hidden in a laser beam[END_REF].

Among various types of digital content, image remains to be a popular one because it can be easily captured, edited, and shared. With the advancement of smart device, even an entry This work was supported by E-Science grant (01-02-10-SF0327) by MOSTI, and in part by EU Horizon 2020 -Marie Sklodowska-Curie Action through the project entitled Computer Vision Enabled Multimedia Forensics and People Identification (Project No. 690907, Acronym: IDENTITY) level model can produce high dynamic range (HDR) image, which is able to present a greater range of luminance levels. A cost effective way to obtain a HDR image is to capture several images, each of different level of exposure (viz., underexposed, normal and overexposed) of the same scene [START_REF] Debevec | Recovering high dynamic range radiance maps from photographs[END_REF]. Details captured in each image are then combined to render the scene more realistically. The higher dynamic range enables digital image to store more information about the scene, which is beneficial for applications such as medical imaging and remote sensing because the increase in stored information can provide more precise analysis. To accommodate the natural of HDR, various file formats are put forward, including JPEG2000, JPEG-XR, JPEG-XT [START_REF] Richter | JPEG XT: A New Family of JPEG Backward-Compatible Standards[END_REF], OpenEXR [START_REF] Kainz | Technical Introduction to OpenEXR[END_REF], Radiance RGBE [START_REF] Ward | Real Pixels[END_REF] and LogLUV TIFF [START_REF] Larson | LogLuv Encoding for Full-Gamut, High-Dynamic Range Images[END_REF].

With the emergence of HDR image, there is an increasing need to have proper techniques for managing HDR image and preserving privacy. In that regard, some perceptual encryption techniques are proposed to mask HDR image. For example, Lin et al. [START_REF] Lin | A tailor-made encryption scheme for high-dynamic range images[END_REF] proposed an encryption technique for RGBE image by using elementary cellular automata (ECA) function. Yan et al. [START_REF] Yan | Encryption in high dynamic range images for RGBE format[END_REF] proposed a format-compliant technique to encrypte LogLUV TIFF image format. The LogLUV pixel values are extracted from the TIFF file and XOR-ed with a stream cipher prior to recompression into a TIFF file. Ting et al. [START_REF] Ting | Format-compliant perceptual encryption method for JPEG XT[END_REF] proposed an encryption technique for JPEG XT compressed image. Specifically, DC and AC coefficients in the base and residual layers are shuffled in multiple rounds to completely mask the perceptual semantic of the image. On the other hand, researcher also invented new techniques to hide data into HDR image. Fujiyoshi et al. [START_REF] Fujiyoshi | A Blind Reversible Data Hiding Method for High Dynamic Range Images Taking Advantage of Sparse Histogram[END_REF] put forward a blind and reversible method based on properties of floating point values observed in HDR image. The method is inspired by the histogram shifting method [START_REF] Hwang | A Reversible Watermarking Based on Histogram Shifting[END_REF] designed for traditional integer-based image. Specifically, histogram of an HDR image is found to be sparse and has many empty bins. This characteristic is exploited as venues for reversible data hiding. Chang et al. [START_REF] Chang | A new distortion-free data embedding scheme for high-dynamic range images[END_REF] designed a method for Radiance RGBE [START_REF] Larson | LogLuv Encoding for Full-Gamut, High-Dynamic Range Images[END_REF] file format. A group of homogeneous representations for each pixel is generated by modifying the quadruplet {R, G, B, E}, where R, G, B, and E represent the red, green blue and exponent channels, respectively. Basically, a pixel value can be encoded in different representations, 978-1-7281-1817-8/19/$31.00 ©2019 IEEE where each representation can be decoded to the same value. Each representation is associated with m = log 2 (n) bits, where n is the number of homogeneous representations for a color. Then, the representation which is associated to the m-bit data to be hidden is selected as the new representation.

Often, both perceptual masking and data hiding are jointly utilized. For example, in the hospital setting, a nurse may need to extract patients' identification information from perceptually masked (or encrypted) CAT scan images. To protect privacy of the patients, medical images are masked, but for management purpose, patients' information is hidden so that it can be extracted without decrypting the images. Similarly, in cloud storage, user's contents are encrypted while information about the contents is embedded for administrative purpose. Therefore, the capability to hide data into perceptually masked HDR image is desired. Some conventional joint techniques include [START_REF] Zhang | Reversible data hiding in encrypted image[END_REF]- [START_REF] Huang | New framework for reversible data hiding in encrypted domain[END_REF], and to the best of our knowledge, none has been proposed for HDR image.

Nonetheless, in [START_REF] Zhang | Reversible data hiding in encrypted image[END_REF], the image pixels are encrypted by using stream cipher, then additional data can be embedded by modifying blocks of the encrypted image. Each block is further divided into two sets {S 0 , S 1 } and the block can be used to hide one bit by flipping the LSB in S 0 or S 1 , depending on the bit to be hidden. One disadvantage of this method is that the hidden data cannot be extracted before decryption. Therefore, in [START_REF]Separable reversible data hiding in encrypted image[END_REF], Zhang improves the method by making it separable, where data can be extracted without viewing the original image content. In [START_REF] Huang | New framework for reversible data hiding in encrypted domain[END_REF], to achieve separability, a specific stream encryption algorithm which can preserve correlation between neighboring pixels is proposed. Specifically, the stream encryption algorithm is followed by a permutation (applied to image blocks) to completely mask the image to ensure higher security while preserving the correlation between pixels in the image blocks. After encryption, any previously proposed DHS (difference histogram shifting) and PEHS (predictionerror histogram shifting) based RDH schemes can be deployed to hide additional information.

In this work, a technique is put forward to first perceptually mask a HDR image, then hide data into the perceptually masked image. Our work is designed for the OpenEXR format, which stores floating point values instead of integer values. The proposed method is reversible, where the original image can be perfectly restored, and it is also separable where the hidden data can be extracted without restoring (i.e., unmasking) the image. Experiments are conducted to evaluate the basic performance of the proposed joint technique for HDR image.

II. PRELIMINARIES A. OpenEXR File Format

OpenEXR is an open source HDR image file format developed by Industrial Light & Magic [START_REF] Kainz | Technical Introduction to OpenEXR[END_REF]. It supports higher dynamic range and color precision when compared to the traditional 8-bit file format. OpenEXR file is capable of storing pixels in 16-or 32-bit floating-point data type, and the pixels can be compressed in either lossless or lossy mode. OpenEXR image can have a number of user-defined channels, each with a different data type. The most common ones are the R, G and B channels [START_REF] Kainz | Technical Introduction to OpenEXR[END_REF]. Specifically, the half precision format is a simplified version of the IEEE-754 floating-point specification [START_REF] Stallings | Computer Organization and Architecture -Designing for Performance[END_REF], which is widely known for its single and double precision floating point formats. Half precision format is made up of 1 sign bit (denoted by S), 5 exponent bits (denoted by E), and 10 mantissa bits (denoted by M ). Figure 1 further details the arrangement of each group of bits. Similar to single and double precision formats, the half precision format supports denormalized numbers, positive and negative infinities, and NaN (i.e., not a number). The representable magnitude ranges from 2 -24 ≈ 5.96 × 10 -8 to (22 10 ) × 215 = 65504. Table I summarizes these representations for half precision format based on E and M .

III. PROPOSED METHOD

Since the exponent part of a pixel determines its coarse value, the exponents of a natural image are highly correlated, particularly for a group of pixels in the same region. This characteristic can be exploited for reversible data hiding, where the exponent values can be replaced by data bits but at the same time they can be completely recovered later through prediction. In addition, the altered exponent values can result in drastic change to the pixel values, causing severe quality degradation. Figure 2 shows the flow of processes in achieving perceptual masking and data hiding.

A. Preprocessing

The exponent part of all pixels are first predicted by using a pixel predictor. The purpose of this step is to identify well predictable locations, so that they can be replaced by the data to be hidden. In this work, we consider the Median Edge Detection (MED) predictor in JPEG-LS [START_REF] Weinberger | The LOCO-I lossless image compression algorithm: principles and standardization into jpegls[END_REF]. MED is a simple predictor which relies on the direction of the edges, i.e., horizontal, vertical, or no edges at all. The operation is formally described by Eq. ( 1), where x is the value to be predicted, a, b and c are the left, top and top-left pixels of x, .

MED predictor predicts all values in one pass following the raster scan order, where pixels at the boarder are utilized as the initial reference pixels.

The prediction error e = E o -E p is computed, where E p and E o are the predicted and original exponent values, respectively. To flag the usability of each exponent for data hiding purpose, the value e is analyzed. Specifically, if e = 0 (viz., the prediction is perfect), the least significant bit (LSB) of E o is modified to '0'. Otherwise, the LSB of E o is set to '1'. The original LSB of an ill predicted E o is stored in a dynamic array C. As for well predicted E o , its LSB is not stored because the entire E o can be predicted perfectly. Array C is important to achieve reversibility. Let E o denoted the modified exponent value, where |E o -E o | ≤ 1. The LSB of E o then guides the data hiding process, where '0' denotes that data can be hidden (or from the perspective of the receiverdata is hidden), while '1' means that no data is hidden.

B. Masking The Image

First, the 10-bit mantissa part M is divided and treated as two independent blocks each with 5 bits, namely, M 1 and M 2 . The subblocks M 1 and M 2 each assumes a size of 5 bits, which is the same as the length of E o to ease the masking operations. Therefore, each pixel will assume the form of [S, E o , M 1 , M 2 ], but for the purpose for this work, S is ignored. Second, M 1 and M 2 are extracted from all pixels, and shuffled by using a key κ 1 . The shuffled M 1 and M 2 form new mantissas (see Fig. 4). The potential outcomes for each pixel include:

[E o (i), M 1 (j),M 2 (k)], [E o (i), M 2 (j),M 1 (k)], [E o (i), M 1 (j),M 1 (k)], j = k, or, [E o (i), M 2 (j),M 2 (k)], j = k, (2)
where M 1 (j) denotes the M 1 block from the j-th pixel, and M 2 (k) as well as E o (i) are defined in same manners. The purpose of Stage II is to mask the details or textures (particularly recurring patterns) captured by the mantissa part, although mantissa can only cause small variation of value in comparison to the exponent part.

Next, the 4 most significant bits (4MSB) of the exponents E o for each pixel as well as its newly formed mantissa are XOR-ed with a random sequence of 0's and 1's, which is generated with the key κ 2 (Stage III). Figure 5 illustrates an example, where E o and M refer to the modified exponent and mantissa parts, respectively. Note the LSB of E o does not undergo the XOR step to maintain the flag, i.e., LSB(E o ) = LSB(E o ). This XOR step is important to break the high correlationship between the pixels, particularly the ill predicted pixels, as they will not be involve the data hiding process in Stage V (see Section III-C for detailed explanation). The newly formed pixels are then shuffled by using the key κ 3 (Stage IV) to further increase the overall entropy of the image pixels.

It is noteworthy that Stages II, III and IV are independent to each other. Therefore, they can be implemented without following any particular order. In addition, a master key κ m for content masking can be considered to generate the keys κ 1 , κ 2 and κ 3 .

C. Data Hiding

Let φ be the external data to be hidden, which is usually encrypted with a key κ 4 . For reversibility purpose, the original LSB of the exponent, denoted by C needs to be stored. Therefore, the augmented data Φ = [C|φ] is constructed, where | refers to the concatenation operation. Φ is then processed in segments of 4-bit. Data hiding is carried out by considering pixel locations where LSB(E o ) = 0 for E o being the exponent part of the pixel at Stage IV. Specifically, for each usable pixel, the most significant 4 bits of its exponent are replaced by a 4-bit segment from Φ. Data hiding will not take place if LSB(E o ) = 1. The process is repeated until all segments of Φ are inserted or when one runs out of usable pixels. Figure 6 illustrates the data insertion process, where the new exponent value is denoted as E Φ o .

D. Data Extraction and Image Recovery

The data extraction and image recovery steps are exactly the reverse of the encryption and data insertion processes. Specifically, the inserted data, i.e., Φ is first extracted, and it is processed to obtain φ and C. The intended receiver will be able to decrypt φ using κ 4 . Note that the data can be extracted without the need to decrypt the image.

To recover the original image, the encryption processes need to be reversed, i.e., inverse shuffling in Stage IV using κ 3 , XOR operation in Stage III using the same binary sequence generated by κ 2 , and inverse shuffling of the M 1 's and M 2 ' blocks in Stage II using κ 1 . Next, the LSB of the exponent part for each pixel is examined to reconstruct the image. Specifically, when LSB(E) = '1', that means 4MSB planes were not replaced, i.e., the original sequence is intact, and the first bit of C is removed and used as the LSB for E. On the other hand, if LSB(E) = '0', then 4MSB planes are recovered through prediction. Note that the predict-to-recover process must be carried out in raster scan order again to ensure the bits from C is reinserted at the correct position, as only pixels flagged as unpredictable have their original LSB stored in C (length of array C = total pixels in image).

IV. EXPERIMENT

The proposed algorithm is implemented in Matlab R2018a. 6 images from the dataset [START_REF] Fairchild | The HDR Photographic Survey[END_REF] are selected randomly for experiment purpose. Since the original image in [START_REF] Fairchild | The HDR Photographic Survey[END_REF] is large in dimension (i.e., larger than 4K), the selected images are downscaled to 6.25% of the original size. These images are shown in Fig. 7. It is confirmed that the inserted data can be extracted, and the original image can be completely reconstructed from its processed counterpart.

A. Effective Payload

First, the raw capacity achieved for each image using the proposed algorithm is computed as: number of usable pixels total pixels × 100%,

while the percentage of side information is computed as: number of unusable pixels number of usable pixels × 100%.

Here, raw capacity refers to the actual number of pixels that are flagged as suitable for hiding data, while side information percentage is the fraction of raw capacity spent on coding the array C. 70%. However, some images (e.g., CemeteryTree(2) ) yield lower raw capacity because there are less well predictable pixels. In other words, higher number of ill predictable pixels result in relatively larger side information C, and hence less available space to insert data.

B. Image Quality

This section presents the quality of the processed image from Stage V in two scenarios: the first one hides data after the prediction stage by skipping Stage II, III and IV, and; the second one hides data after the encryption stage, viz., stepping through Stage II, III and IV. By visual inspection, output image in the first scenario is already severely distorted. However, some details of the image are still visible (e.g., see outline of wheel at the lower left region in Fig. 8(c)). Nonetheless, the image produced by in the second scenario is completely distorted, i.e., there is no trace of the original perceptual semantic. This observation verifies that although replacing the exponent part with payload bits already result in a highly distorted image, Stage II, III and IV are crucial to further intensify distortion to completely mask the perceptual semantic of an image. Figure 8 shows two representative output images for both scenarios.

To quantify distortion, log-based PSNR [START_REF] Aydn | Extending quality metrics to full luminance range images[END_REF] is computed for the processed images for both scenarios, and the results are recorded in Table III. The logarithmic function is an approximation on how Human Visual System (HVS) responds to the visible luminance range. The capability to mimic how human eye response to light makes this quality metric more suitable in capturing the quality of HDR image. As expected, the PSNR values are low (i.e., close to 0) and this confirms that the processed images are highly distorted after applying the proposed image masking and data hiding technique.

C. Discussion

Last but not least, the proposed method is suitable to be applied for floating point numbers, specifically half precision format. The conventional methods [START_REF] Zhang | Reversible data hiding in encrypted image[END_REF]- [START_REF] Huang | New framework for reversible data hiding in encrypted domain[END_REF] are not suitable for floating point numbers because they are designed for integer pixels, which have different intrinsic properties. For example, 3LSB are manipulated to hide data in [START_REF] Zhang | Reversible data hiding in encrypted image[END_REF], but the technique is not readily applicable to pixels stored in floating point format. Specifically, the mantissa part of the pixel in half-precision format is rather random and it is challenging to predict them precisely. V. CONCLUSIONS In this paper, image masking and data insertion techniques are put forward for HDR image stored in the OpenEXR format. Specifically, the exponent part of each pixel is classified as predictable or not predictable. The predictable pixels are modified to hide data, while side information is stored as part of the payload for perfect image recovery purpose. To further intensify the distortion, mantissa subblocks shuffling, XOR cipher and inter-pixel shuffling are applied to the pixels. Experiments suggest that, on average, the number of usable pixels is greater than 70% of the total number of pixels. Results also suggest that the quality of an image can be completed distorted and perfectly recovered later when required.

As future work, more detailed analysis will be performed to investigate into the security aspect of the proposed method. In addition, the file structure of OpenEXR will be further explored to improve capacity and secrecy.

Figure 1 :

 1 Figure 1: 16-bit half precision format.

Figure 2 :

 2 Figure 2: Overall process of proposed method.

Figure 3 :

 3 Figure 3: Layout of MED predictor.

Figure 4 :

 4 Figure 4: An example output of mixing M 1 and M 2 blocks among pixels (Stage II).

Figure 5 :

 5 Figure 5: Masking pixel using XOR operation (Stage III).

(a) Case 1 :Figure 6 :

 16 Figure 6: Hiding data into preprocessed exponent part.

Figure 7 :

 7 Figure 7: Six HDR test images from [21].

Figure 8 :

 8 Figure 8: Output images for both scenarios.

  Table II records the capacity for each test image. Results suggest that the percentage of usable pixels is high for all HDR images considered, with an average of more than

TABLE II :

 II Hiding capacity for different test images.

	Image	raw capacity (%)	side info capacity (%)
	507	89.2549	2.9442
	CanadianFalls	74.0750	8.6622
	CemeteryTree(2)	63.5323	14.2582
	Frontier	86.3199	3.8944
	BigfootPass	90.5954	2.5238
	BalancedRock	88.5727	3.1595

TABLE III :

 III Quality assessment after processing (dB).

	Image	Log PSNR
		Scenario 1 Scenario 2
	507	3.3252	2.7679
	CanadianFalls	4.4658	3.0105
	CemeteryTree(2)	4.8320	2.7357
	Frontier	3.3314	2.5717
	BigfootPass	3.9878	3.4865
	BalancedRock	3.4036	2.7252


Alexandre Vérine

Benjamin Negrevergne

Fabrice Rossi

Yann Chevaleyre

On the expressivity of bi-Lipschitz normalizing flows

An invertible function is bi-Lipschitz if both the function and its inverse have bounded Lipschitz constants. Most state-of-the-art Normalizing Flows are bi-Lipschitz by design or by training to limit numerical errors (among other things). In this paper, we discuss the expressivity of bi-Lipschitz Normalizing Flows and identify several target distributions that are difficult to approximate using such models. Then, we characterize the expressivity of bi-Lipschitz Normalizing Flows by giving several lower bounds on the Total Variation distance between these particularly unfavorable distributions and their best possible approximation. Finally, we discuss potential remedies which include using more complex latent distributions.

Introduction

A weak smoothness condition for a function F , beyond continuity, can be enforced by requesting F to be L-Lipschitz, that is to verify

∀x 1 , x 2 ∈ X , F (x 1 ) -F (x 2 ) 2 ≤ L x 1 -x 2 2 ,
where L is the Lipschitz constant of F . A number of recent publications have demonstrated the benefits of constructing machine learning models with a small Lipschitz constant. First, models with a small Lipschitz constant have been linked with better generalization capabilities, both in terms of true risk [START_REF] Bartlett | Spectrally-normalized margin bounds for neural networks[END_REF], and adversarial risk [START_REF] Farnia | Generalizable Adversarial Training via Spectral Normalization[END_REF]. In addition, models with a small Lipschitz constants are more stable during training [START_REF] Miyato | Spectral Normalization for Generative Adversarial Networks[END_REF], and are less prone to numerical errors, a property which is particularly important in the context of invertible neural networks and normalizing flows [START_REF] Behrmann | Understanding and Mitigating Exploding Inverses in Invertible Neural Networks[END_REF].

Unfortunately, enforcing a small Lipschitz constant, either by design, or using regularization during training, can impede the ability of a model to fit the data distribution. Based on this observation, several researchers have studied the limitations of neural networks with bounded Lipschitz constants. In particular, [START_REF] Tanielian | Learning disconnected manifolds: a no GANs land[END_REF] were able to identify a family of target distributions with disconnected support that cannot be fitted with a Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative Adversarial Networks[END_REF] with a bounded Lipschitz constant. For the particular case of normalizing flows, [START_REF] Cornish | Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows[END_REF] were able to demonstrate that there exist pairs of latent and target distributions with particular topological conditions on their support, that require the model to have an unbounded Lipschitz constant.

In this paper we focus on characterizing the impact of Lipschitz constraints on the expressivity of normalizing flows. More precisely, we discuss the impact of the Lipschitz constant on the Total Variation distance (TV) between the approximated distribution and the target distribution. We give several lower bounds on the TV distance which (unlike previous works), do not assume any hypothesis on the support of the target distribution, and are thus applicable to any learning settings. Furthermore, since Normalizing Flows are often not only Lipschitz, but bi-Lipschitz, (meaning that both the inverse mapping function and the mapping itself have bounded Lipschitz constant), we also study the impact the Lipschitz constant of the mapping, on the expressivity. (Most work on the topic focus on the Lipschitz constant of the inverse mapping.) We give a new bound that depends on the Lipschitz constant of the mapping. As we will show, this new bound provide new insights and is complementary with the two other bounds we introduce. Finally, we discuss the potential remedies in the light of these new results.

Outline of the paper The rest of this paper is organized as follows. Section 2 reviews normalizing flows as well as total variation distance and the precision/recall for generative networks. The main results are presented in Section 3: a first general bound is presented in the Section 3.1, the two main theorems are presented and proved in the Section 3.2. We draw a link between the main results and the previous related work in the Section 3.3. Finally we discuss the potential remedies to the highlighted limitations in Section 4, and Section 5 concludes the paper and gives some directions of future works. In the Supplementary Material, formal proofs of the different results are presented.

Background

Normalizing Flow

A normalizing flow is an invertible density model in which both density estimation and sampling can be done efficiently. In short, training a normalizing flow consists in learning an invertible mapping between a data space X and a latent space Z. Typically, the forward direction F : X → Z (i.e. the normalizing direction) is tractable and exact and the inverse direction F -1 : Z → X (i.e. the generative direction) either has a closed form, or can be approximated using an iterative algorithm.

Suppose that P * is the true data distribution over X , and that P * admits a density function denoted p * that we wish to approximate. We first chose a d-dimensional Gaussian distribution Q over Z (a.k.a. the latent space), and its density function

q(z) = 1 ( √ 2π) d e -1 2 z 2 2 .
The choice of a normal distribution is natural since it is the most frequent latent distribution in normalizing flows. In Section 4, more complex distributions will be discussed. Then, we can define p, the approximation of p * , based on q and the mapping F : X → Z, using a simple change of variable formula:

∀x ∈ X , p(x) = | det Jac F (x)| q(F (x)).
(1)

Note that the estimated probability P (A) of any event A ⊆ X can be computed as follows:

P (A) = Q(F (A)) = F (A)
q(z)dz.

As seen in Equation 1, performing density estimation requires computing the determinant of the Jacobian matrix which can be large in practice, thus most normalizing flows have been specifically designed to make this computation efficient.

Bi-Lipschitz Normalizing Flows

In this paper, we focus on bi-Lipschitz normalizing flows, which is a mapping F whose Lipschitz constants are bounded in both directions. More specifically, we define the bi-Lipschitz property as follows.

Definition 2.1. A bijective function F : X ⊂ R d → Z ⊂ R d is said to be (L 1 , L 2 )-bi-Lipschitz if F is L 1 -Lipschitz and its inverse F -1 is L 2 -Lipschitz, i.e.: ∀x 1 , x 2 ∈ X , F (x 1 ) -F (x 2 ) 2 ≤ L 1 x 1 -x 2 2 ,
and

∀z 1 , z 2 ∈ Z, F -1 (z 1 ) -F -1 (z 2 ) 2 ≤ L 2 z 1 -z 2 2 .
Alternatively, since the mapping F is bijective, the bi-Lipschitz continuity can be expressed over F only as:

1

L 2 x 1 -x 2 2 ≤ F (x 1 ) -F (x 2 ) 2 ≤ L 1 x 1 -x 2 2 .
However, enforcing the bi-Lipschitz continuity of F results in a bounded determinant for the Jacobian matrix:

Proposition 2.1. Jac F satisfies for all x ∈ X ⊂ R d :

1 L d 2 ≤ |det Jac F (x)| ≤ L d 1 .
Proposition 2.1 comes from a characterization of the Lipschitz continuity adapted to differentiable functions [START_REF] Federer | Geometric measure theory[END_REF]:

∀x ∈ X , Jac F (x) 2 ≤ L 1 .
Therefore, through the spectral decomposition of the jacobian matrix, we can show that ∀x ∈ X ,

| det Jac F (x)| ≤ L d 1 .
Then, since F is bi-Lipschitz, the same inequality can be expressed for

F -1 : ∀z ∈ Z, | det Jac F -1 (z)| ≤ L d 2 and thus ∀x ∈ X , | det Jac F (x)| ≥ 1/L d 2 .
As we will show in the rest of this paper, this can limit the expressivity of normalizing flows. This is relevant, because many normalizing flows are bi-Lipschitz in practice, for example, the i-ResNet [START_REF] Behrmann | Invertible Residual Networks[END_REF] and the Residual Flow [START_REF] Chen | Residual Flows for Invertible Generative Modeling[END_REF] are both based on residual atomic blocks [START_REF] Kingma | Glow: Generative Flow with Invertible 1x1 Convolutions[END_REF] with atomic blocks

f i = I d + g i . Their invertibility is ensured by the Lipschitz constant Lip(g i ) ≤ L < 1. If F is composed of m residual blocks such that F = f m • • • • • f 1 , then the overall bi-Lipschitz constants satisfy Lip(F ) ≤ (1 + L) m and Lip(F -1 ) ≤ 1/(1 -L) m . Alternatively, in Glow
W i = P i L i (U i + diag(s i )), the bi-Lipschitz constants satisfy: Lip(F ) ≤ m i W i 2 and Lip(F -1 ) ≤ m i W -1 i 2 .
Notice that the bi-Lipschitzness constraints on either the function or its Jacobian determinant can frequently be weakened by increasing the depth of the network but, by doing so, the stability of the inverse can be affected [START_REF] Behrmann | Understanding and Mitigating Exploding Inverses in Invertible Neural Networks[END_REF].

Assessing the expressivity

Our goal is to understand how the bi-Lipschitz property affects the approximation ability of the network. To do so, we will compare the true data distribution P * and its density p * with the learned distribution P and its density p.

In previous works, [START_REF] Tanielian | Learning disconnected manifolds: a no GANs land[END_REF] use the maximum precision to evaluate how the true distribution P * and the generated distribution P differs. We have chosen another tool to compare both distributions: the total variation (TV) distance given as : Definition 2.2 (Total Variation Distance). For any distribution P and P * , the total variation distance is defined as the maximum difference of probabilities given to a same event A:

D TV (P * , P ) = sup A |P * (A) -P (A)|.
This choice has been made for several reasons. It is adequate to highlight the Lipschitz constraints of the mapping. It has a close connection with the precision and the recall, and yet, the TV is more general in terms of support of target distribution as explained in the followings. Finally, the TV distance can be used to compute a lower bound on the D KL divergence through the Pinsker's inequality: 2D TV (P * , P ) 2 ≤ D KL (P * P ).

Even if the main results of this paper are consisting in lower bounds on the TV distance, we compare our results to an existing one in terms of precision and recall. Therefore we give their definitions for generative models as they were given initially [START_REF] Sajjadi | Assessing Generative Models via Precision and Recall[END_REF][START_REF] Kynkäänniemi | Improved Precision and Recall Metric for Assessing Generative Models[END_REF].

Definition 2.3 (Precision α and Recall β for generative models). For α, β ∈ [0, 1], the distributions P is said to have a precision α at recall β with respect to P * if there exist the distributions ν, ν, ν * , such that P and P * can be decomposed as such:

P = αν + (1 -α)ν and P * = βν + (1 -β)ν * .
The distribution ν defined on Supp( P ) ∪ Supp(P * ) while Supp(ν) = Supp( P ) and Supp(P * ) = Supp(ν * ) It can be interpreted as such: ν represent the part of P * that P correctly models, ν is simultaneously the part of P * that P misses on their joint support and all the points that should not be represented by P . Finally, ν * cover the points of P * that the support of P could not reach and all the points on their joint support that P misestimated.

Among all the potential decompositions, i.e. the pairs (α, β), the focus is set on the maximum precision ᾱ and the maximum recall β.

Proposition 2.2 (Maximum precision ᾱ and maximum recall β). The maximum precision and the maximum recall satisfy: ᾱ = P (Supp(P * )) and β = P * (Supp( P )).

The results given by Tanielian et al. ( 2020) is an upper bound an the maximum precision for a Lipschitz F -1 and for a particular target distribution. In general having upper bounds on ᾱ or β is a stronger result than having a lower bound on the D TV since we have:

D TV (P * , P ) ≥|P * (Supp(P * )) -P (Supp(P * ))| , D TV (P * , P ) ≥|P * (Supp( P )) -P (Supp( P ))|.
This can be expressed as :

D TV (P * , P ) ≥1 -ᾱ, D TV (P * , P ) ≥ = 1 -β.
However, as soon as the support of, respectively, the target distribution P * or the estimated distribution P covers X , we have respectively ᾱ = 1 or β = 1. Thus, the maximum precision/recall become irrelevant for assessing the expressivity of the normalizing flow.

Lower bounds on the TV distance

The general idea is to look for subsets of the dataset in the data space X that may be particularly difficult to fit with a Lipschitz constrained mapping function. Intuitively, the Lipschitz constraints limit the ability of normalizing flows to contract or to expand the latent space, so we focus our analysis on very dense subsets or very sparse subsets of the data space that will likely be the most difficult to fit.

We focus first on dense subsets with arbitrary shape, and we are able to derive a positive lower bound on the TV that depends on the volume of the largest dense subset. Then, we show how to compute more specific but stronger results when considering subsets with a ball shape. First an intuitive but loose bound is derived. We then discuss two new tight bounds that are based on dense and sparse ball shaped subsets of the data space.

A first bound based on the largest dense subset of X

The first theorem is a lower bound on the TV distance between the learned distribution and the target distribution in a general setting. Intuitively, the idea is to find a subset A with an arbitrary shape that is sufficiently concentrated so that the Lipschitz constrained mapping can not concentrate enough weight from the Gaussian distribution onto this subset.

Theorem 3.1 (L 1 -Lipschitz mappings fail to capture high density subset). Let F be L 1 -Lipschitz and η A = P * (A) vol(A) be the average density over any subset A ⊂ R d . Then:

D TV (P * , P ) ≥ sup A vol(A) η A - L 1 √ 2π d . Therefore, if there is a subset A that satisfies η A > L1 √ 2π d
, then the TV is necessarily strictly positive.

Theorem 3.1 results from the definition of the estimated distribution P and the change of variable. Indeed for an arbitrary subset A of X :

P (A) = A p(x)dx = A |Jac F (x)|q(F (x))dx.
Then, since q is the density function of the normal distribution it is upper bounded by 1/ √ 2π d , and with the upper bound of the determinant of the jacobian matrix given in Proposition 2.1 :

P (A) ≤ L 1 √ 2π d A dx = L 1 √ 2π d vol(A).
In other terms, the weight assigned from the Gaussian latent distribution to the subset A is bounded by

L 1 / √ 2π d vol(A).
Consequently if there is a dense subset for which P * (A) = η A vol(A) is high enough, the TV will be strictly positive. More precisely, the total variation is greater than the difference made by the most dense subset A. The detailed proof of the Theorem 3.1 is given in Appendix A.1.

The main advantage of this formulation is to be applied to any subset of the data space, but at the expense of a loose bound on the TV.

Bounds based on dense and sparse balls

The bound in Theorem 3.1 can be further improved by making assumptions on the structure of the subset A. We choose to focus on l 2 balls instead of arbitrary subsets.

Let B R,x0 be the l 2 ball with center x 0 and radius R (i.e. B R,x0 = {x ∈ X : xx 0 2 ≤ R}). Then we can show that both high density balls and low density ones are difficult to fit properly, the former because of the Lipschitz constraint of F , the latter because of the Lipschitz constraint of F -1 .

We first consider high density balls.

Theorem 3.2 (NF with a L 1 -Lipschitz mapping F fails to capture high density balls). Let F be L 1 -Lipschitz. Then:

D TV (P * , P ) ≥ sup R,x0 P * (B R,x0 ) - RL 1 √ π .
Therefore, if we find a ball for which the true measure satisfies

P * (B R,x 0 ) R > L1 √ π
, then the TV is necessarily strictly positive.

Theorem 3.2 highlights the effect of the L 1 Lipschitz constraint of the forward mapping F . The image of a ball B R by the mapping F in constrained in a ball:

F (B R ) ⊂ B L1R .
Thus, if we consider ball with a high probability P * (B R ) in the data space , then the probability assigned to this ball

P * (B R ) = Q(F (B R )) is at most Q(B RL1
) in the latent space and is upper bounded by RL 1 / √ π [START_REF] Ball | The reverse isoperimetric problem for Gaussian measure[END_REF]. The lower bound of the TV in Theorem 3.2 is given with a linear relation with radius R and the Lipschitz constant L 1 in dimension d. It serves as an intuitive representation of the limitations induced by the Lipschitz constraint. For a given dense ball B R,x0 , the smaller R is, the greater L 1 should be to insure that the normalizing flow can properly map the Gaussian distribution onto P * . The bounds mainly serve for interpretation purpose as we can compute a tighter bound on the TV distance. The closed form of Gaussian measure of a ball B R,0 is given by function of the Gamma function Γ and the incomplete gamma function γ, therefore:

Q(B RL1,x0 ) ≤ Q(B RL1,0 ) = γ d 2 , L 2 1 R 2 2 /Γ d 2 .
The Theorem 3.3 is then less open to interpretation than the Theorem 3.2 but it proposes a tight bound.

Theorem 3.3 (NF with a L 1 -Lipschitz mapping F fails to capture high density balls). Let F be L 1 -Lipschitz. Then:

D TV (P * , P ) ≥ sup R,x0   P * (B R,x0 ) - γ d 2 , L 2 1 R 2 2 Γ d 2   .
Therefore, if we find a ball for which the true measure satisfies

P * (B R,x0 ) > γ( d 2 , L 2 1 R 2 2 )/Γ( d 2 )
, then the TV is necessarily strictly positive.

A one dimensional representation of a pathological case for Theorems 3.2 & 3.3 is shown on Figure 1. In other words no ball with a density high enough in the data space can be expanded sufficiently to have a matching probability in the latent space.

Conversely, the mapping being bi-Lipschitz, it can not contract arbitrarily:

B R/L2 ⊂ F (B R ).
If there is a low density zone mapped on the maximum of the Gaussian density, then the Normalizing Flow cannot reduce enough the probability of the corresponding zone: Notice that the assumption of a low density zone is strong but fairly reasonable. For instance, one can observe a multi-modal density with fairly well separated modes. If the modes are roughly equiprobable, we expect a mapping to assign those modes in balanced way around the mode of the Gaussian distribution in the latent space. Therefore, the low density ball is mapped on a zone wider than the ball B R/L2 and consequently the Gaussian measure associated is lower bounded by Q(B R/L2 ) as illustrated in the one dimensional example on Figure 2. Despite the lower bounds established by Pinelis ( 2020), there is no reasonably interpretable bounds, therefore we use the closed-form as in Theorem 3.3.

P (B R ) ≥ P (F -1 (B R )) = Q(B R/L2,0 ).
Theorem 3.4 (NF with L 2 -Lipschitz inverse mappings F -1 fail to capture low density balls). Let F -1 be L 2 -Lipschitz. We consider the balls centered on F -1 (0), we have the lower bound:

D TV (P * , P ) ≥ sup R   γ d 2 , R 2 2L 2 2 Γ d 2 -P * (B R,F -1 (0) )   .
Therefore, if we find a ball for which the the true measure satisfies

P * (B R,F -1 (0) ) < γ(d/2,R 2 /2L 2 2 ) Γ(d/2)
, then the TV is necessarily strictly positive.

All formal proofs are detailed in Appendix A.2 & A.4. Numerical illustrations of the behavior of the closed form bound are given in Figure 3. Note that for Theorem 3.3, the probability Q(B RL1 ) needs to be as large as possible which, given a radius RL 1 , will be harder while the dimension increases. In other terms, the Lipschitz constant L 1 will have to be consequently more increased in high dimension than in low dimension. The behavior is reversed for L 2 : from Theorem 3.4, we see that the probability Q(B R/L2 ) should be as small as possible. Then, given a radius R/L 2 , it will be harder in low dimension dimension than in high dimension. The Lipschitz constant L 2 will have a greater effect in low dimension than in high dimension.

Comparison to related work

A related set up is used in the work of [START_REF] Tanielian | Learning disconnected manifolds: a no GANs land[END_REF]. The authors consider two disconnected subsets M 1 and M 2 separated by a distance D, with equal probabilities in the latent space, i.e. P (M 1 ) = P (M 1 ) = 1/2. As a consequence, F -1 (0) is equidistant from M 1 and M 2 as illustrated in Figure 4. The original work assesses the learning abilities of their generative model, a L 2 -Lipschitz GAN [START_REF] Goodfellow | Generative Adversarial Networks[END_REF][START_REF] Arjovsky | Wasserstein GAN[END_REF], with a definition of precision and recall [START_REF] Sajjadi | Assessing Generative Models via Precision and Recall[END_REF]; [START_REF] Kynkäänniemi | Improved Precision and Recall Metric for Assessing Generative Models[END_REF]. The authors propose an upper bound of the maximum recall based on the cumulative distribution function of the 1-dimensional normal distribution Φ

(t) = t -∞ exp(-r 2 /2) 2π dr: ᾱ + 2D L 2 e -Φ -1 ( ᾱ/2) 2 ≤ 1.
Our method offers another bound on the maximum precision:

ᾱ ≤ 1 - γ d 2 , D 2 2L 2 2 Γ d 2 .
The main advantage of our bound is that it can be directly computed whereas the bound given by Tanielian et al. ( 2020) is not explicit. The advantage of their bound is that it does not depend on the dimension. The detailed proof can be found in Appendix A.5. From the link between the divergence and the maximum precision, we can derive a lower bound on the TV as a particular case of the Theorem 3.4:

Corollary 3.4.1 (NF with L 2 -Lipschitz inverse mapping). If F -1 is L 2 -Lipschitz, then we have a lower bound on the TV distance based on the distance D between M 1 and M 2 :

D TV (P * , P ) ≥ γ d 2 , D 2 2L 2 2 /Γ d 2 .
The main benefit of using maximum precision to assess the quality of the mapping is that it is well fitted to be used with the Gaussian Isoperimetric Inequality and therefore gives results that do not depend on the dimension d. The benefit of using TV distance is that we can now compute the bounds for distributions with arbitrary support. This covers more cases that the work by [START_REF] Cornish | Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows[END_REF] which only discusses the limits of the normalizing flows when the supports of the two distributions are not homeomorphic. Instead, the bounds we introduce in the present paper can be used to discuss the limitations of normalizing flows, even when dealing with two distributions with homeomorphic supports. Let us consider for instance, the case where a normalizing flow tries to map a 1-dimensional Gaussian distribution with an excessively low or high variance onto the standard normal distribution: the supports are homeomorphic, the maximum precision and recall are both 1, but our method can be used to derive a strictly positive lower bound.

Note that since the TV distance is defined as the supremum on any subspace A, the approximation errors made by the network exposed by the same Theorem can be accumulated. Suppose that, in this pathological case, the balls M 1 and M 2 are dense enough that Theorem 3.3 can be applied on both subsets. It results in two different bounds on TV : L M1 > 0 and L M2 > 0. Then, since M 1 and M 2 are disjoints, we can write :

D TV (P * , P ) ≥ |P * (M 1 ∪ M 2 ) -P (M 1 ∪ M 2 )| ≥ P * (M 1 ) -P (M 1 ) + P * (M 2 ) -P (M 2 ) ≥ L M1 + L M2
This set up is then an appropriate pathological case to study the effect of the bi-Lipschitzness of the mapping.

Discussion on the potential remedies

As mentioned earlier, increasing the Lipschitz constants of the entire network (for example, by adding extra layers) may impact invertibility and stability during training [START_REF] Behrmann | Understanding and Mitigating Exploding Inverses in Invertible Neural Networks[END_REF], and thus is not a suitable approach to improve the expressivity.

Alternatively, one can consider learning the parameters of the latent Gaussian distribution µ and Σ = σI d . However, this is equivalent to changing the Lipschitz constants of F from (L 1 , L 2 ) to ( L1 σ , L 2 σ), thus this results in trading off the expected error on very dense subsets (Theorem 3.3) with the expected error on subsets with low densities (Theorems 3.4) or vice-versa. In particular bounds becomes:

D TV (P * , P ) ≥ sup R,x0 P * (B R,x0 ) - γ( d 2 , L 2 1 R 2 2σ 2 ) Γ( d 2 )
, and

D TV (P * , P ) ≥ sup R   γ( d 2 , σ 2 R 2 2L 2 2 ) Γ( d 2 ) -P * (B R,F -1 (0) )   .
Increasing, reducing or learning the variance will indeed increase one bound and decrease the other one. In other words this can lead to a better approximation for a some particular data distributions, but it does not generally improve the expressivity of the normalizing flow.

To improve expressivity, a Gaussian Mixture latent distribution can also be considered. Indeed, Khayatkhoei et al. ( 2019) and [START_REF] Izmailov | Semi-Supervised Learning with Normalizing Flows[END_REF] have shown that such distributions can learn disconnected manifolds. When the latent distribution is a Gaussian Mixture, Theorem 3.4 does not hold anymore. Limitations similar to the ones highlighted in Theorem 3.2 still apply, but can be mitigated using learnable parameters.

We can trivially adapt the lower bound from Theorem 3.2 to the Gaussian Mixture with K equally distributed modes with learnable means µ i and covariance matrices Σ i = σ i I d :

D TV (P * , P ) ≥ sup R,x0   P * (B R,x0 ) - 1 K γ d 2 , L 2 1 R 2 2σ 2 i Γ d 2   .
As we can see here, the lower bound depends on the inverse of the number of modes K in the mixture and the variance σ i . Thus, this approach can solve the limitations highlighted in Theorem 3.3. The lower bound increases with K but the learnable σ i can compensate this augmentation. However, learning a Gaussian Mixture suffers from the same issues than another method than we would like to mention: the Variational Mixture of Normalizing Flow [START_REF] Pires | Variational Mixture of Normalizing Flows[END_REF]. They train set of K different normalizing flows and a neural network with a K-class softmax output to set the mixture. They encountered strong training difficulties to learn the mixture and, as for the Gaussian Mixture, there is yet no efficient way to learn the hyperparameter K [START_REF] Izmailov | Semi-Supervised Learning with Normalizing Flows[END_REF]. A closely related method by [START_REF] Dinh | A RAD approach to deep mixture models[END_REF] consists in a discrete partition of the data space X into K distinct subsets. K different normalizing flows would be trained on each of those partitions. This methods also suffers from training difficulties and the lack of evidence on how to set the number of partitions. To avoid a discrete set of normalizing flows, [START_REF] Cornish | Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows[END_REF] propose a continuous set on normalizing flows, a promising method to tackle the limitations highlighted by the Theorem 3.4, but it does not preserve an exact likelihood and requires a complex training process.

Overall, there exist some theoretical potential remedies to both of the limitations highlighted by Theorems 3.3 & 3.4 but further investigation is required to to deal with the technical issues.

Conclusion

We have established that the bi-Lipschitz constraints reduce the expressivity of Normalizing flows. When the dataset meets some particular conditions such as a high density zone or a low density zone between two high density zones, the reduced expressivity fails to capture the real distribution of the dataset. More specifically, we have brought to light two particular lower bounds of the total variation distance between the target distribution and the approximated one. The first bound illustrates that dense subset, and especially dense balls, of the data space can induce approximation errors. The second bound illustrates that some low density balls located between high density subsets can also result in approximations errors.

Further works could consist in completing the experimental aspects of these results. First understanding how those bound actually behave in the traditional dataset such as CIFAR10, CelebA or ImageNet. It would be interesting to develop a process to design normalizing flows based on the dataset and the Lipschitz constants of the reversible architecture.

Moreover, Theorems 3.3 & 3.4 are based on the bi-Lipschitz constraints of the network and on general properties met by the dataset. Some similar bounds could be derived for non invertible structures that satisfy local bi-Lipschitzness. One of the direction of the future work is to generalize, not only the results, but also the general framework to study the expressivity of generative lipschitz models based on the dataset.

Independently of the generative models, close attention is given on the effect the Lipschitz regularization as measure of the stability of a neural networks [START_REF] Scaman | Lipschitz regularity of deep neural networks: analysis and efficient estimation[END_REF][START_REF] Combettes | Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operators[END_REF][START_REF] Béthune | The Many Faces of 1-Lipschitz Neural Networks[END_REF], on the better generalization [START_REF] Bartlett | Spectrally-normalized margin bounds for neural networks[END_REF] or on the adversarial robustness [START_REF] Szegedy | Intriguing properties of neural networks[END_REF][START_REF] Araujo | On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory[END_REF]. The theoretical framework presented in this work could be transposed to more general applications linked to Lipschitz regularization.

A Proofs

A.1 Proof of theorem 3.1

By definition we have P (A) = A p(x)dx, then with the change of variable formula we obtain: Then, the idea is to show that the image of a ball B R by a L 1 -Lipschitz function is in a ball of radius L 1 R, and then use a reverse isoperimetric inequality the find an upper bound of the measure of a ball of a radius L 1 R.

P (A) = A |Jac F (x)|q(F (x))dx = 1 (2π) d/2 A |Jac F (x)|e -F (x) 2 /2 dx As F is L 1 -Lipschitz we have |Jac F (x)| ≤ L d 1 , then P (A) ≤ L 1 √ 2π d A e -F (x) 2 2 dx ≤ L 1 √ 2π d A dx ≤ L 1 √ 2π d vol (

Proof of

F (B R,x 0 ) ⊂ B L 1 R,F (x 0 )
First of all, for every z ∈ F (B R,x0 ), there exist x ∈ B R such that F -1 (z) = x, we have:

F (F -1 (z)) -F (x 0 ) = F (x) -F (x 0 ) ≤ L 1 x -x 0 ≤ L 1 R
Upper bound of Q(B L1R ) This bound is extracted from the work of [START_REF] Ball | The reverse isoperimetric problem for Gaussian measure[END_REF] on the Reverse Isoperimetric Inequality. First of all, it can be easily establish that Q(B L1R (F (x 0 ))) is at a maximum when F (x 0 ) = 0. From now on, we will only consider B L1R the ball centered on 0. Therefore the objective is to find an upper bound on:

Q(B L1R ) = z <L1R q(z)dz = z <L1R 1 ( √ 2π) d e -z 2 /2 dz
We can use the polar coordinates system to get another expression of the Gaussian measure with S d-1 (r) =

Q(B R,0 ) = P z 2 ≤ R 2 ,
when z follows the standard Gaussian distribution in R d . This quantity can be computed using the cumulative distribution function of the chi-square distribution, i.e.

Q(B R,0 ) = γ d 2 , R 2 2 Γ d 2 ,
where γ is the lower incomplete gamma function given by γ(x, k) = x 0 t k-1 e -t dt.

Figure 1 :

 1 Figure 1: Example of a target distribution where theorem 3.2 applies: the subset B R concentrates most of the weight in P * (B R ), but P (B R ) = Q(F (B R )) can only be as large as Q(B R L 1 ).

Figure 2 :

 2 Figure 2: Example of a target distribution for which Theorem 3.4 applies: the subset B R concentrates little weight in P * (B R ), but P (B R ) = Q(F (B R )) can only be as small as Q(B R /L 2 ).

Figure 3 :

 3 Figure 3: Representation of the Gaussian Measure of balls of radius R centered on 0. The measure is given for dimension 1, 2, 10 and then the dimensions of MNIST (Yann LeCun et al., 2010), CIFAR10 (Alex[START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] and CelebA[START_REF] Liu | Deep Learning Face Attributes in the Wild[END_REF]

Figure 4 :

 4 Figure 4: Experimental set up given by Tanielian et al. (2020)

  A), and thus T V (P * , P ) = sup A |P * (A) -P (A)| implies T V (P * , P ) ≥ sup A By definition of the TV distance, we have D TV (P * , P ) ≥ sup R,x0|P * (B R,x0 ) -Q(F (B R,x0 ))|,where B R,x0 is the ball of a radius R centered in x 0 .

  1 e -r 2 /2 dr However r d-1 e -r 2 /2 has a maximum value reached for r = √ d -1, we can have an upper bound:Lower Bound of the TV As soon as we have an upper bound on Q(B L1R ), we have:D TV (P * , P ) ≥ sup R,x0 (P * (B R,x0 ) -Q(F (B R,x0 ))) ≥ sup R,x0 (P * (B R,x0 ) -Q(B L1R,x0 ))Value of Q(B R,0 ) By construction

Lower Bound of the TV Since we have the closed form of the measure over a ball we can write :

A.4 Proof of theorem 3.4

In this section, we denote

Therefore, by analogy with the proof of Theorem 3.3:

A.5 Proof of Corollary 3.4.1

Since M 1 and M 2 are separated by a distance D the ball centered on F -1 (0) has a radius at least as big as D that we might call B D to simplify the notation. Therefore:

And