STRUCTURAL TESTING OF MODERN RECONFIGURABLE CHIPS

Keywords:

This paper presents recent developments for testing SRAM-based FPGAs using a structural approach. The specific architecture of these new chips is first presented identifying the specific FPGA test problems as well as the FPGA test properties. The FPGA architecture is then conceptually divided into different architectural elements. For each architectural element test configurations and test vectors are derived targeting the assumed fault models.

Field Programmable Gate Arrays (FPGAs) are digital devices that can implement logic circuits by programming the required function [START_REF] Brown | Field-Programmable Gate Arrays[END_REF][START_REF]Field-Programmable Gate Array Technology[END_REF]. Because of their short turnaround time and programmability in the field, they have been widely used for rapid prototyping or reconfiguration of complex digital systems. One important class is the SRAM-based FPGAs which can be easily reprogrammed any number of times. This paper focus on SRAM-based FPGAs. In such a programmable circuit, an array of logic cells and interconnection cells can be configured in the field to implement a desired designed function.

Testing for FPGAs as well as conventional digital ICs, is a difficult and important problem . A very simple approach consists in considering the FPGA as a classical digital ASIC making the problem of testing the FPGA equivalent to the problem of testing a classical ASIC. Such a classical test approach fails when applied on FPGA. The main reason is because FPGAs have a heterogeneous architecture mixing interconnect elements, logic modules and RAM cells. Indeed, usual digital VLSI circuits include a few number of large blocks, each block being of a specific nature: a random logic block, a RAM block, buses… In a classical digital test approach, a specific test technique is consequently used for each block according to its nature. On the contrary, typical FPGAs can be viewed as composed of a large number of small blocks, all the blocks are similar but include logic modules, RAM cells and interconnect elements. These heterogeneous elements are strongly interconnected making the usual test techniques difficult to apply. As the matter of fact, a typical heterogeneous FPGA block can be viewed as a good recollection of all the problems encountered in testing: Heter .

Heter .

Heter .

Heter .

Heter .

Heter .

Heter .

Heter .

Homog.Array of RAM cells

Homog. Array of Interconnect cells

Homog.

Array of Logic cells

Fig. 1. Homogeneous Arrays

Then, a specific and adequate test approach can be applied to each homogeneous cell according to its nature. But, it is obvious that each specific test approach must take into account the connection of the considered cell with the other ones. For example, the test approach proposed for the RAM cells must account for the connections between the RAM cells and the logic cell and interconnect cell.

These remarks try to give a general idea of the complexity of FPGA testing. On the other side, typical SRAM-based FPGAs have an important test property that must absolutely be considered and used in a test approach. As previously mentioned, almost all the blocks into the FPGAs are similar meaning that the small heterogeneous block is repeated forming a regular two-dimensional array of mxm heterogeneous blocks. From the test point of view, the regular structures present some properties that simplified the test approach. So, applying the above conceptual division to the whole FPGA means that we have to propose a specific test approach for each two-dimensional array: a) a two-dimensional array of logic cells, b) a two-dimensional array of interconnect cells, c) a two-dimensional array of RAM cells.

According to this practical test strategy, the published works usually target specific array . As an example, Inoue and al. address the problem of testing the array of look-up table in [START_REF] Inoue | Universal Test Complexity of Field-Programmable Gate Arrays // 4th Asian Test Symposium[END_REF], Huang and al. address the problem of testing the array of logic cells in [START_REF] Huang | An Approach for Testing Programmable / Configurable Field Programmable Gate Arrays[END_REF]. Other authors focus on different test strategies as Abramovici and al. dealing with BIST for FPGA in [START_REF] Abramovici | No-Overhead BIST for FPGAs[END_REF][START_REF] Stroud | Evaluation of FPGA Ressources for Built-In Self Test of Programmable Logic Blocks[END_REF][START_REF] Abramovici | ILA BIST for FPGAs: A Free Lunch with Gourmet Food // 2nd IEEE International On-line Testing Workshop[END_REF], Lombardi and al. dealing with diagnosis in [START_REF] Lombardi | Diagnosing Programmable Interconnect Systems for FPGAs[END_REF].

Following this practical divide and conquer test strategy, the author has proposed first a test procedure targeting the array of interconnect cells [START_REF] Renovell | Testing the Interconnect Structure of Unconfigurated FPGA[END_REF][START_REF] Renovell | Test of RAM-Based FPGA: Methodology and Application to the Interconnect[END_REF][START_REF] Renovell | Testing the Interconnect of RAM-Based FPGAs[END_REF], second another test procedure targeting the array of logic cells in [START_REF] Renovell | Test Pattern and Test Generation Methodology for the Logic of RAM-Based FPGA[END_REF][START_REF] Renovell | Testing the Configurable Logic of RAM-based FPGA[END_REF][START_REF] Renovell | SRAM-based FPGAs: A Fault Model for the Configurable Logic Modules // Field programmable Logic Conference[END_REF], third a test procedure for the array of LUT/RAM modules in [START_REF] Renovell | SRAM-based FPGAs: Testing the RAM mode of the LUT[END_REF][START_REF] Renovell | SRAM-Based FPGA: Testing the LUT/RAM modules[END_REF], and finally a test procedure for the array of interconnect/logic interface cells in [START_REF] Renovell | SRAM-Based FPGA: Testing the Interconnect/Logic Interface[END_REF]. In this paper the test of the twodimensional array of logic cells is presented in section 2, the test of the two-dimensional array of interconnect cells in section 3 and the test of the two-dimensional array of RAM cells in section 4.

The logic cells

This section is devoted to the test of the logic cells. This problem has been discussed in detail by the authors in [START_REF] Renovell | Testing the Interconnect Structure of Unconfigurated FPGA[END_REF][START_REF] Renovell | Test of RAM-Based FPGA: Methodology and Application to the Interconnect[END_REF][START_REF] Renovell | Testing the Interconnect of RAM-Based FPGAs[END_REF]. This section summarizes the main results. The logic cells usually consists of three types of logic modules: D flip-flops, multiplexers and look up table units (LUT). The multiplexers and the look-up tables are typical configurable devices while the D flip-flop are not really configurables. In fact, the control signals of the flip-flop (reset, clock...) are configurables by means of multiplexers. Because we concentrate here on the definition of test configuration for configurable devices, only multiplexers and look-up tables are considered in this section.

In figure 2, we have an example of classical 4-to-1 multiplexer with of 2 bit address A0,A1. In a typical FPGA representation, the data inputs E0,E1,E2,E3 are represented because they are Operation Inputs while the 2 bit address are not represented because they are Configuration Inputs not available during normal operation. In practice the internal logic structure can vary or is not really known, and so the fault model associated to this device is the stuck-at of the 6 different inputs E0,E1,E2,E3,A0,A1 and the single output S. It can be demonstrated that all the stuck-at-0/1 of the inputs and output of the FPGA multiplexer can be detected by using 4 test configurations. As the matter of fact, a test configuration is associated to each multiplexer address. For each test configuration, a sequence of 2 test vectors is applied. These 4 test configurations are illustrated in figure 3 where the configurations are symbolically represented by a connection between an input and output. In a more general way, we can say that a multiplexer with 2 n addresses (n address bits) require 2 n test configurations.

Assuming now that the LUT is a particular type of multiplexer as illustrated in figure 4, the test conditions are identical for the multiplexer and the LUT. Hence, we can use the vectors previously defined for the multiplexer. In such conditions, we found that the exclusive-OR and complemented exclusive-OR vectors must be applied on the LUT configuration inputs and an exhaustive sequence of 2 n vectors must be applied on the LUT operation Inputs. This is equivalent in defining 2 test configurations called TC XOR and TC XNOR and defining 2 corresponding test sequences called TS XOR and TS XNOR . The 2 test configurations are symbolically represented by a XOR (Е) or XNOR symbol inside the LUT. The test configurations and test vectors defined for the isolated multiplexer and LUT are now used to define test configurations and test vectors for the logic cell. The logic cell is an interconnection of modules such as multiplexers, LUTs and flip-flops and so, a test configuration of a logic cell is an aggregate of the test configurations of the modules as illustrated in figure 5 with the Xilinx 4000 logic cell [START_REF]Xilinx. The Programmable Logic Data Book[END_REF]. In order to 'cover' all the test configurations of all the modules in the cell, several test configurations must obviously be defined for the logic cell. We demonstrated that using our technique, only 5 test configurations are required for the Xilinx 4000. Figure 5 gives an example of test configuration for the logic cell of the Xilinx 4000. Concerning, the test sequences associated to each test configuration, they are obtained from the test sequences of each module. In fact, the test sequences of the modules are simply justified through the other modules.

The minimization of the number of Test Configuration using the module Test Configurations leads to only 5 Test Configurations for completely testing the complex XILINX 4000 CLB. The problem now is to define Test Configurations and Test Sequences for the mxm array of logic cells. In case of an array of logic cells, the problem consists in controlling and observing the whole logic cells. Individual access to each logic cell is not possible in practice. Indeed, a FPGA does not have enough I/O pads to control and observe each logic cell in parallel from outside.

For these reasons, the logic cells are interconnected in a special way forming one-dimensional arrays of cascaded logic cells. The length of the one-dimensional array is not important. The number and length of the arrays only depends on the number of available I/O pads. In practice, the most convenient solution is illustrated in Figure 6 where a mxm array of logic cells are distributed in m one-dimensional arrays of m logic cells. Using this scheme, the m one-dimensional arrays are tested in parallel.

In fact, a one-dimensional array of m cascaded logic cells can be viewed as an iterative circuit. Each logic cell received a local input from the previous logic cell and produces a local output connected to the next logic cell. The left most local inputs are controllable primary inputs and the right most local outputs are observable primary outputs. In addition, each logic cell receives a number of controllable primary inputs (black arrows) that are common to every logic cell in the FPGA.

Fig. 6. One-dimensional arrays of 3 CLBs

Using the concept of one-dimensional array, it is clear in Figure 6 that an 'embedded' logic cell is controlled through logic cells located on its left side and observed through logic cells located on its right side. Consequently, the proposed test configurations and sequences for the whole array of logic cells must guarantee propagation through a given logic cell of signal controlling other logic cells on its right side, and guarantee propagation through a logic cell of observing signals from other logic cells on its left side. In the example of the Xilinx 4000, we simply define 5 test configurations for the whole array corresponding to the 5 test configurations of the single logic cell. In our approach, all the m 2 logic cells in the FPGA are configured exactly in the same way. Knowing that the 5 basic Test Configurations guarantee the complete test of any isolated CLB, we have demonstrated that the proposed interconnecting principle guarantees the controllability and observability of any embedded CLB. The demonstration is based on the fact that the test configurations used for the CLB implement in fact XOR or XNOR functions. The one-dimensional array corresponds consequently to a cascade of XOR (XNOR) functions that have well-known controllability and observabillity properties.

At this point, it must be noted that the complete Test Procedure has been simulated using an iterative array of 4 CLBs giving 100% coverage of the assumed fault models. These simulations validate the proposed test configurations and test sequences.

The interconnect

This section is devoted to the test of the interconnect cells. This problem has been discussed in detail by the authors in [START_REF] Renovell | Testing the Interconnect Structure of Unconfigurated FPGA[END_REF][START_REF] Renovell | Test of RAM-Based FPGA: Methodology and Application to the Interconnect[END_REF][START_REF] Renovell | Testing the Interconnect of RAM-Based FPGAs[END_REF]. The interconnect structure of this type of RAM based FPGA is composed of a mxm array of « Switch Matrix » interconnected by k metal lines. This regular array of interconnect cells is illustrated in the simplified example of figure 7 where m=5 and k=4. ..E k ), South pins (S 1 ...S k ) and West pins (W 1 ...W k ). Inside the Switch Matrix, some pairs of pins can not be connected and are called non-connectable pins. Some pairs of pins can be connected and are called connectable pins. Figure 7 gives an example of switch matrix with 4 pins on each side. In figure 7 the connectable pins are linked by a dotted line. Figure 7 gives also an example of configuration where some connectable pins are connected (full lines). In the remainder of the paper we consider as an example that any set of pins with the same (different) number i are connectable (non-connectable). The considered set of connectable pins illustrated in figure 7 corresponds to the Xilinx 4000 family and so, the results presented here can be directly applied.

It is now necessary to define adequate fault models for this particular interconnect cells and lines. Due to the nature of the elements, we consider fault models classically used for interconnections i.e. Opens and Shorts: 1) For any line: Open called Line-Open.

2) For any pair of line : Short called LinePair-Short.

3) For any pair of non-connectable pins: Short called Permanent-Connection. 4) For any pair of connectable pins: Short called Permanent-Connection.

5) For any pair of connectable pins: Open called Permanent-Disconnection.

Obviously, the faults concerning the non-connectable pins are independent of the switch matrix configuration while faults concerning the connectable pins depend on the switch matrix configuration. A test configuration that connects for example pins N4 and S4 makes the Permanent-Connection fault between connectable Pins N4 and S4 redundant and so, un-testable. While a test configuration that does not connect the pins make the fault non-redundant. It is easy to demonstrate that a minimum of 3 test configurations are required to make all the faults under consideration non-redundant. Several set of 3 test configurations can be defined and figure 8 gives an example of 3 test configurations called the Orthogonal, the Diagonal-1 and the Diagonal-2.

The problem now is to use the 3 previous test configurations not to test a single isolated interconnect cell but to test the complete mxm array of interconnect cells. The approach used here is similar to the one used in the previous section for the logic cells i.e. all the interconnect cells in the array have the same test configurations. This method gives obviously 3 test configurations for the complete array that are illustrated in figure 8. Using these 3 test configurations, the complete array can be conceptually considered as a global bus. The concept of bus with shorts and opens allow to use the previously published works about the bus testing problem [START_REF]Xilinx. The Programmable Logic Data Book[END_REF][START_REF] Kautz | Testing for Faults in Wiring Networks[END_REF][START_REF] Goel | Electronic Chip-in Place Test[END_REF][START_REF] Jarwala | A New Framework for Analyzing Test Generation and Diagnosis Algorithms for Wiring Networks[END_REF]. It has been demonstrated that a n bits bus can be tested for any short and open with log 2 (n) vectors. For the considered array, the resulting number of test vectors is: log 2 (2km).

The RAM

This section is devoted to the test of the array of RAM cells that are embedded in the LUT/RAM modules. The LUT/ RAM module is assumed to be part of a logic cell and it is assumed to be configured in RAM mode. This problem has been discussed in detail by the authors in [START_REF] Renovell | SRAM-based FPGAs: Testing the RAM mode of the LUT[END_REF][START_REF] Renovell | SRAM-Based FPGA: Testing the LUT/RAM modules[END_REF] Considering first a single isolated module in RAM mode, the module operates as a classical RAM and any type of existing RAM test can be used. This is very interesting because the problem of RAM testing has been investigated for a long time and very mature algorithms exist [START_REF] Abadir | Functional Testing of Semiconductor Random Access Memories[END_REF][START_REF] Van De Goor | Testing Semiconductor Memories: Theory and Practice[END_REF]. A well-known class of test algorithms for RAM circuits are the MARCH algorithms.

The RAM fault models usually considered in the march tests are:

-SAF: The stuck-at fault can be defined as follows: The logic value of a stuck-at cell or line is always 0 or 1 and cannot be changed to the opposite value.

-AF: The address decoder faults concern faults in the address decoder. Different types of faults are usually considered. The first fault assumes that no cell will be accessed with a certain address. The second fault assumes that a certain cell is never accessed. The third fault assumes that multiple cells are accessed simultaneously with a certain address. And finally, the last fault assumes that a certain cell can be accessed with multiple addresses.

-TF: The transition fault is a special case of the SAF. It is defined as follows. A cell or line which fails to undergo a 0 1 transition when it is written is said to contain an up transition fault; similarly, a down transition fault is the impossibility of making a 1 0 transition.

-CF: The coupling fault involves 2 cells and can be defined as follows. A write operation which generates a up or down transition in one cell changes the contents of a second cell. Different types of coupling faults are usually considered. The inversion coupling fault (CFin) assumes that an up or down transition in one cell inverts the contents of a second cell. And the idempotent coupling fault (CFid) assumes that an up or down transition in one cell forces the contents of a second cell to a certain value, 0 or 1. In this unique test configuration, the common primary inputs connected to every module include the control signals (Read/ Write, Enable..) and address inputs of the LUT/RAM module, and the clock (CK) of the Dffs. The Read/Write control signal and the Dff clock can be adequately used to shift a value from the primary data input to the primary output, through the Dffs and through the modules. In this configuration, the MATS++ algorithm can be adapted taking into account the shift trough the different cells: This adaptation is called the shifted MARCH++ algorithm.

Conclusion

This paper gives a general overview of a structural test approach proposed for testing RAM-based FPGA taking into account their configurability. The SRAM-based FPGA architecture is first discussed identifying the specific FPGA test problems as well as the FPGA test properties. The FPGA architecture is then conceptually divided into different architectural elements such as the logic cells, the interconnect and the RAM cells. For each architectural element appropriated fault models are proposed, and test configurations and test vectors are derived targeting the fault models under consideration.

Algorithm

  a) Sequential elements (flip-flops), b) Mixed architecture (logic/interconnect/RAM), c) Multiple representation level (module/gate/transistor) d) Fault model definition (stuck/short/open/memory). To face the difficulty of handling an heterogeneous block, a practical solution consists in conceptually dividing the heterogeneous block into different homogeneous sub-blocks as represented in figure 1. An interesting solution can be to divide the heterogeneous block into the following homogeneous blocks: a) The logic cell, b) The interconnect cell, c) The RAM cells.Heter .

Fig. 3 .

 3 Fig. 2. Classical Mux and FPGA Mux

Fig. 4 .Fig. 5 .

 45 Fig. 4. Test of the Look-up Table: a -TC XOR ; b -TC XNOR

Fig. 7 .

 7 Fig. 7. FPGA Interconnect StructureA switch matrix is a programmable connecting element receiving k lines on each side. The lines are connected to the switch matrix pins called North pins (N 1 ...N k ), East pins (E 1 ...E k ), South pins (S 1 ...S k ) and West pins (W 1 ...W k ). Inside the Switch Matrix, some pairs of pins can not be connected and are called non-connectable pins. Some pairs of pins can be connected and are called connectable pins. Figure7gives an example of switch matrix with 4 pins on each side. In figure7the connectable pins are linked by a dotted line. Figure7gives also an example of configuration where some connectable pins are connected (full lines). In the remainder of the paper we consider as an example that any set of pins with the same (different) number i are connectable (non-connectable). The considered set of connectable pins illustrated in figure7corresponds to the Xilinx 4000 family and so, the results presented here can be directly applied.

Fig. 8 .

 8 Fig. 8. The 3 Test Configurations: a -orthogonal; b -Diagonal-1; c -Diagonal-2

Fig. 9 .

 9 Fig. 9. The pseudo shift register

  Note that DRAM circuits are more sensitive to CF than SRAM circuits. Dealing in this paper with SRAM based FPGA, we will use the SAF, AF and TF fault models. It can be observed in Table1that MATS, MATS+, Marching 1/0 and MATS++ are able to detect some of the fault models under consideration. It seems interesting to use the MATS++ test because it covers all the considered fault models and the number of test vectors is very low.Note that only one test configuration is required to test a single module in RAM mode. Considering now the complete array of LUT/RAM module, the approach used here is similar to the one used in the previous sections for the logic cells and the interconnect cells i.e. all the LUT/RAM modules in the array have the same test configurations. This method gives obviously 1 test configuration for the complete array. In order to guarantee full controllability and observability of each module, we propose to connect the output of the LUT/RAM module in RAM mode to the input of the DFF included in the logic cell. The output of the Dff is connected to the data in of the following LUT/RAM module. This particular test configuration called pseudo shift register is illustrated in figure9with m=3.


Thomas Lucas

Jakob Verbeek

Auxiliary Guided Autoregressive Variational Autoencoders

Generative modeling of high-dimensional data is a key problem in machine learning. Successful approaches include latent variable models and autoregressive models. The complementary strengths of these approaches, to model global and local image statistics respectively, suggest hybrid models that encode global image structure into latent variables while autoregressively modeling low level detail. Previous approaches to such hybrid models restrict the capacity of the autoregressive decoder to prevent degenerate models that ignore the latent variables and only rely on autoregressive modeling. Our contribution is a training procedure relying on an auxiliary loss function that controls which information is captured by the latent variables and what is left to the autoregressive decoder. Our approach can leverage arbitrarily powerful autoregressive decoders, achieves state-of-the art quantitative performance among models with latent variables, and generates qualitatively convincing samples. VAE encoder Latent variables z VAE decoder f(z) PixelCNN decoder Reconstruction original image x KL divergence regularization Reconstruction auxiliary image y

Introduction

Unsupervised modeling of complex distributions with unknown structure is a landmark challenge in machine learning. The problem is often studied in the context of learning generative models of the complex high-dimensional distributions of natural image collections. Latent variable approaches can learn disentangled and concise representations of the data [START_REF] Bengio | Representation learning: A review and new perspectives[END_REF], which are useful for compression [START_REF] Gregor | Towards conceptual compression[END_REF] and semi-supervised learning [START_REF] Kingma | Semi-supervised learning with deep generative models[END_REF][START_REF] Rasmus | Semi-supervised learning with ladder networks[END_REF]. When conditioned on prior information, generative models can be used for a variety of tasks, such as attribute or class-conditional image generation, text and pose-based image generation, image colorization, etc. [START_REF] Yan | Attribute2image: Conditional image generation from visual attributes[END_REF][START_REF] Van Den Oord | Conditional image generation with PixelCNN decoders[END_REF][START_REF] Reed | Parallel multiscale autoregressive density estimation[END_REF][START_REF] Deshpande | Learning diverse image colorization[END_REF]. Recently significant advances in generative (image) modeling have been made along several lines, including adversarial networks [START_REF] Goodfellow | Generative adversarial nets[END_REF][START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF], variational autoencoders [START_REF] Kingma | Auto-encoding variational Bayes[END_REF][START_REF] Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF], autoregressive models [START_REF] Oord | Pixel recurrent neural networks[END_REF][START_REF] Reed | Parallel multiscale autoregressive density estimation[END_REF], and non-volume preserving variable transformations [START_REF] Dinh | Density estimation using real NVP[END_REF].

In our work we seek to combine the merits of two of these lines of work. Variational autoencoders (VAEs) [START_REF] Kingma | Auto-encoding variational Bayes[END_REF][START_REF] Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF] can learn latent variable representations that abstract away from low-level details, but model pixels as conditionally independent given the latent variables. This renders the generative model computationally efficient, but the lack of low-level structure modeling leads to overly smooth and blurry samples. Autoregressive models, such as pixelCNNs [21], on the other hand, estimate complex translation invariant conditional distributions among pixels. They are effective to model low-level image statistics, and yield state-of-the-art likelihoods on test data [START_REF] Salimans | Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications[END_REF]. This is in line with the observations of [START_REF] Kolesnikov | PixelCNN models with auxiliary variables for natural image modeling[END_REF] that low-level image details account for a large part of the likelihood. These autoregressive models, however, do not learn a latent variable representations to support, e.g., semi-supervised learning.

The complementary strengths of VAEs and pixelCNNs, modeling global and local image statistics respectively, suggest hybrid approaches combining the strengths of both. Prior work on such hybrid models needed to limit the capacity of the autoregressive decoder to prevent degenerate models that completely ignore the latent variables and rely on autoregressive modeling only [START_REF] Gulrajani | PixelVAE: A latent variable model for natural images[END_REF][START_REF] Chen | Variational lossy autoencoder[END_REF]. In this paper we describe Auxiliary Guided Autoregressive Variational autoEncoders (AGAVE), an approach to train such hybrid models using an auxiliary loss function that controls which information is captured by the latent variables and what is left to the AR decoder. That removes the need to limit the capacity of the latter. See Figure 1 for a schematic illustration of our approach.

Using high-capacity VAE and autoregressive components allows our models to obtain quantitative results on held-out data that are on par with the state of the art in general, and set a new state of the art among models with latent variables. Our models generate samples with both global coherence and low-level details. See Figure 2 for representative samples of VAE and pixelCNN models.

Related work

Generative image modeling has recently taken significant strides forward, leveraging deep neural networks to learn complex density models using a variety of approaches. These include the variational autoencoders and autoregressive models that form the basis of our work, but also generative adversarial networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF][START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF] and variable transformation with invertible functions [START_REF] Dinh | Density estimation using real NVP[END_REF]. While GANs produce visually appealing samples, they suffer from mode dropping and their likelihood-free nature prevents measuring how well they model held-out test data. In particular, GANs can only generate samples on a non-linear manifold in the data space with dimension equal to the number of latent variables. In contrast, probabilistic models such as VAEs and autoregressive models generalize to the entire data space, and likelihoods of held-out data can be used for compression, and to quantitatively compare different models. The non-volume preserving (NVP) transformation approach of [START_REF] Dinh | Density estimation using real NVP[END_REF] chains together invertible transformations to map a basic (e.g. unit Gaussian) prior on the latent space to a complex distribution on the data space. This method offers tractable likelihood evaluation and exact inference, but obtains likelihoods on held-out data below the values reported using state-of-the-art VAE and autoregressive models. Moreover, it is restricted to use latent representations with the same dimensionality as the input data, and is thus difficult to scale to model high-resolution images. Autoregressive density estimation models, such as pixelCNNs [START_REF] Oord | Pixel recurrent neural networks[END_REF], admit tractable likelihood evaluation, while for variational autoencoders [START_REF] Kingma | Auto-encoding variational Bayes[END_REF][START_REF] Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF] accurate approximations can be obtained using importance sampling [START_REF] Burda | Importance weighted autoencoders[END_REF]. Naively combining powerful pixelCNN decoders in a VAE framework results in a degenerate model which ignores the VAE latent variable structure, as explained through the lens of bits-back coding by [START_REF] Chen | Variational lossy autoencoder[END_REF]. To address this issue, the capacity of the the autoregressive component can be restricted. This can, for example, be achieved by reducing its depth and/or field of view, or by giving the pixelCNN only access to grayscale values, i.e. modeling p(x i |x <i , z) = p(x i |gray(x <i ), z) [START_REF] Chen | Variational lossy autoencoder[END_REF][START_REF] Gulrajani | PixelVAE: A latent variable model for natural images[END_REF]. This forces the model to leverage the latent variables z to model part of the dependencies among the pixels. This approach, however, has two drawbacks. (i) Curbing the capacity of the model is undesirable in unsupervised settings where training data is abundant and overfitting unlikely, and is only a partial solution to the problem. (ii) Balancing what is modeled by the VAE and the pix-elCNN by means of architectural design choices requires careful hand-design and tuning of the architectures. This is a tedious process, and a more reliable principle is desirable. To overcome these drawbacks, we propose to instead control what is modeled by the VAE and pixelCNN with an auxiliary loss on the VAE decoder output before it is used to condition the autoregressive decoder. This allows us to "plug in" powerful high-capacity VAE and pixelCNN architectures, and balance what is modeled by each component by means of the auxiliary loss.

In a similar vein, [START_REF] Kolesnikov | PixelCNN models with auxiliary variables for natural image modeling[END_REF] force pixelCNN models to capture more high-level image aspects using an auxiliary representation y of the original image x, e.g. a low-resolution version of the original. They learn a pixelCNN for y, and a conditional pixelCNN to predict x from y, possibly using several intermediate representations. This approach forces modeling of more high-level aspects in the intermediate representations, and yields visually more compelling samples. [START_REF] Reed | Parallel multiscale autoregressive density estimation[END_REF] similarly learn a series of conditional autoregressive models to upsample coarser intermediate latent images. By introducing partial conditional independencies in the model they scale the model to efficiently sample high-resolution images of up to 512×512 pixels. [START_REF] Gregor | Towards conceptual compression[END_REF] use a recurrent VAE model to produces a sequence of RGB images with increasing detail derived from latent variables associated with each iteration. Like our work, all these models work with intermediate representations in RGB space to learn accurate generative image models.

Auxiliary guided autoregressive variational autoencoders

We give a brief overview of variational autoencoders and their limitations in Section 3.1, before we present our approach to learning variational autoencoders with autoregressive decoders in Section 3.2.

Variational autoencoders

Variational autoencoders [START_REF] Kingma | Auto-encoding variational Bayes[END_REF][START_REF] Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF] learn deep generative latent variable models using two neural networks. The "decoder" network implements a conditional distribution p θ (x|z) over observations x given a latent variable z, with parameters θ. Together with a basic prior on the latent variable z, e.g. a unit Gaussian, the generative model on x is obtained by marginalizing out the latent variable:

p θ (x) = p(z)p θ (x|z) dz. (1)
The marginal likelihood can, however, not be optimized directly since the nonlinear dependencies in p θ (x|z) render the integral intractable. To overcome this problem, an "encoder" network is used to compute an approximate posterior distribution q φ (z|x), with parameters φ. The approximate posterior is used to define a variational bound on the data log-likelihood, by subtracting the Kullback-Leibler divergence between the true and approximate posterior:

ln p θ (x) ≥ L(θ, φ; x) = ln(p θ (x)) -D KL (q φ (z|x)||p θ (z|x)) (2) = IE q φ [ln(p θ (x|z)] Reconstruction -D KL (q φ (z|x)||p(z)) Regularization . (3)
The decomposition in (3) interprets the bound as the sum of a reconstruction term and a regularization term. The first aims to maximize the expected data log-likelihood p θ (x|z) given the posterior estimate q φ (z|x). The second term prevents q φ (z|x) from collapsing to a single point, which would be optimal for the first term.

Variational autoencoders typically model the dimensions of x as conditionally independent,

p θ (x|z) = D i=1 p θ (x i |z), (4)
for instance using a factored Gaussian or Bernoulli model, see e.g. [START_REF] Kingma | Auto-encoding variational Bayes[END_REF][START_REF] Kingma | Improved variational inference with inverse autoregressive flow[END_REF][START_REF] Yan | Attribute2image: Conditional image generation from visual attributes[END_REF]. The conditional independence assumption makes sampling from the VAE efficient: since the decoder network is evaluated only once for a sample z ∼ p(z) to compute all the conditional distributions p θ (x i |z), the x i can then be sampled in parallel.

A result of relying on the latent variables to account for all pixel dependencies, however, is that all low-level variability must also be modeled by the latent variables. Consider, for instance, a picture of a dog, and variants of that image shifted by one or a few pixels, or in a slightly different pose, with a slightly lighter background, or with less saturated colors, etc. If these factors of variability are modeled using latent variables, then these low-level aspects are confounded with latent variables relating to the high-level image content. If the corresponding image variability is not modeled using latent variables, it will be modeled as independent pixel noise. In the latter case, using the mean of p θ (x|z) as the synthetic image for a given z results in blurry samples, since the mean is averaged over the low-level variants of the image. Sampling from p θ (x|z) to obtain synthetic images, on the other hand, results in images with unrealistic independent pixel noise.

Autoregressive decoders in variational autoencoders

Autoregressive density models, see e.g. [START_REF] Larochelle | The neural autoregressive distribution estimator[END_REF][START_REF] Germain | MADE: Masked autoencoder for distribution estimation[END_REF], rely on the basic factorization of multi-variate distributions,

p θ (x) = D i=1 p θ (x i |x <i ) (5)
with x <i = x 1 , . . . , x i-1 , and model the conditional distributions using a (deep) neural network. For image data, PixelCNNs [START_REF] Oord | Pixel recurrent neural networks[END_REF][START_REF] Van Den Oord | Conditional image generation with PixelCNN decoders[END_REF] use a scanline pixel ordering, and model the conditional distributions using a convolution neural network. The convolutional filters are masked so as to ensure that the receptive fields only extend to pixels x <i when computing the conditional distribution of x i . PixelCNNs can be used as a decoder in a VAE by conditioning on the latent variable z in addition to the preceding pixels, leading to a variational bound with a modified reconstruction term:

L(θ, φ; x) = IE q φ D i=1 ln p θ (x i |x <i , z) -D KL (q φ (z|x)||p(z)). ( 6
)
The regularization term can be interpreted as a "cost" of using the latent variables. To effectively use the latent variables, the approximate posterior q φ (z|x) must differ from the prior p(z), which increases the KL divergence. [START_REF] Chen | Variational lossy autoencoder[END_REF] showed that for loss (6) and a decoder with enough capacity, it is optimal to encode no information about x in z by setting q(z|x) = p(z). To ensure meaningful latent representation learning [START_REF] Chen | Variational lossy autoencoder[END_REF] and [START_REF] Gulrajani | PixelVAE: A latent variable model for natural images[END_REF] restrict the capacity of the pixelCNN decoder. In our approach, in contrast, it is always optimal for the autoregressive decoder, regardless of its capacity, to exploit the information on x carried by z. We rely on two decoders in parallel: the first one reconstructs an auxiliary image y from an intermediate representation f θ (z) in a non-autoregressive manner. The auxiliary image can be either simply taken to be the original image (y = x), or a compressed version of it, e.g. with lower resolution or with a coarser color quantization. The second decoder is a conditional autoregressive model that predicts x conditioned onf θ (z). Modeling y in a non-autoregressive manner ensures a meaningful representation z and renders x and z dependent, inducing a certain non-zero KL "cost" in [START_REF] Deshpande | Learning diverse image colorization[END_REF]. The uncertainty on x is thus reduced when conditioning on z, and there is no longer an advantage in ignoring the latent variable for the autoregressive decoder. We provide a more detailed explanation of why our auxiliary loss ensures a meaningful use of latent variables in powerful decoders in 3.3. To train the model we combine both decoders in a single objective function with a shared encoder network:

L(θ, φ; x, y) = IE q φ D i=1 ln p θ (x i |x <i , z) Primary Reconstruction + IE q φ   E j=1 ln p θ (y j |z)   Auxiliary Reconstruction - λ D KL (q φ (z|x)||p(z)) Regularization . (7)
Treating x and y as two variables that are conditionally independent given a shared underlying latent vairable z leads to λ = 1. Summing the lower bounds in Eq. (3) and Eq. ( 6) of the marginal log-likelihoods of y and x, and sharing the encoder network, leads to λ = 2. Larger values of λ result in valid but less tight lower bounds of the log-likelihoods. Encouraging the variational posterior to be closer to the prior, this leads to less informative latent variable representations.

Sharing the encoder across the two decoders is the key of our approach. The factored auxiliary VAE decoder can only model pixel dependencies by means of the latent variables, which ensures that a meaningful representation is learned. Now, given that the VAE encoder output is informative on the image content, there is no incentive for the autoregressive decoder to ignore the intermediate representation f (z) on which it is conditioned. The choice of the regularization parameter λ and auxiliary image y provide two levers to control how much and what type of information should be encoded in the latent variables.

It is optimal for the autoregressive decoder to use z

Combining a VAE with a flexible decoder (for instance an autoregressive one) leads to the latent code being ignored. This problem could be attributed to optimization challenges: at the start of training q(z|x) carries little information about x, the KL term pushes the model to set it to the prior to avoid any penalty, and training never recovers from falling into that local minimum. [START_REF] Chen | Variational lossy autoencoder[END_REF] have proposed extensive explanations showing that the problem goes deeper: if a sufficiently expressive decoder is used, ignoring the latents actually is the optimal behavior. The gist of the argument is based on bits-back coding as follows: given an encoder q(z|x), a decoder p(x|z) and a prior p(z), z ∼ q(z|x) can be encoded in a lossless manner using p(z), and x can be encoded, also losslessly, using p(x|z). Once the receiver has decoded x, q(z|x) becomes available and a secondary message can be decoded from it. This yields and average code length of:

C BitsBack = E x∼D,z∼q(.|x) [log(q(z|x)) -log(p(z)) -log(p(x|z))].
C BitsBack corresponds to the standard VAE objective. A lower-bound on the expected code length for the data being encoded is given by the Shannon entropy:

H(D) = E x∼D [-log p D (x)]
, which yields:

C BitsBack = E x∼D [-log(p(x)) + D KL (q(z|x)||p(z|x))] ≥ H(D) + E x∼D [D KL (q(z|x)||p(z|x))].
If p(.|x j<i ) is expressive enough, or if q(.|x) is poor enough, the following inequality can be verified:

H(D) ≤ E x∼D [-log p(x|x j<i )] < H(D) + E x∼D [D KL (q(z|x)||p(z|x))]
This is always true in the limit of infinitely expressive autoregressive decoders. In that case, any use of the latents that p might decrease performance. The optimal behavior is to set q(z|x) = p(z) to avoid the extra KL cost. Then z becomes independent from x and no information about x is encoded in z. Therefore, given an encoder, the latent variables will only be used if the capacity of the autoregressive decoder is sufficiently restricted. This is the approach taken by [START_REF] Chen | Variational lossy autoencoder[END_REF] and [START_REF] Gulrajani | PixelVAE: A latent variable model for natural images[END_REF]. This approach works: it has obtained competitive quantitative and qualitative performance. However, it is not satisfactory in the sense that autoregressive models cannot be used to the full extent of their potential, while learning a meaningful latent variable representation.

In our setting, both (Y, X) have to be sent to and decoded by the receiver. Let us denote C V AE the expected code length required to send the auxiliary message, y. Once y has been sent, sending x costs: E z∼q(z|x) [i log(p(x i |z, x j<i ))], and we have:

C V AE = E x∼D,z∼q(.|x) [log(q(z|x)) -log(p(z)) -log(p(y|z))] (8)
C AGAV E = C V AE + E z∼q(z|x) [- i log(p(x i |z, x j<i ))]. (9)
Using the fact that the Shannon entropy is the optimal expected code length for transmitting X|Z, we obtain

C AGAV E ≥ C V AE + H(X|Z)
The entropy of a random variable decreases when it is conditioned on another, i.e. H(X|Z) ≤ H(X). Therefore, the theoretical lower-bound on the expected code length in our setup is always better when the autoregressive component takes Z into account, no matter its expressivity. In the limit case of an infinitely expressive autoregressive decoder, denoted by * , the lower bound is attained and

C * AGAV E = C V AE + H(X|Z) ≤ C V AE + H(X).
In non-degenerate cases, the VAE is optimized to encode information about X into a meaningful Z, with potentially near perfect reconstructions, and there exists > 0 such that H(X|Z) < H(X) -, making the lower bound stricly better by a possibly big margin.

This analysis shows that in our setup it is theoretically always better for the autoregressive model to make use of the latent and auxiliary representation it is conditioned on. That is true no matter how expressive the model is. It also shows that in theory our model should learn meaningful latent structure.

Experimental evaluation

In this section we describe our experimental setup, and present results on CI-FAR10.

Dataset and implementation

The CIFAR10 dataset [START_REF] Krizhevsky | Learning Multiple Layers of Features from Tiny Images[END_REF] contains 6,000 images of 32×32 pixels for each of the 10 object categories airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. The images are split into 50,000 training images and 10,000 test images. We train all our models in a completely unsupervised manner, ignoring the class information.

We implemented our model based on existing architectures. In particular we use the VAE architecture of [START_REF] Kingma | Improved variational inference with inverse autoregressive flow[END_REF]

p(y i |µ i , s i ) = σ (c + c y i /c |µ i , s i ) -σ (c y i /c |µ i , s i ) . ( 10
)
For the pixelCNN we use the architecture of [START_REF] Salimans | Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications[END_REF], and modify it to be conditioned on the VAE decoder output f (z), or possibly an upsampled version if y has a lower resolution than x. In particular, we apply standard non-masked convolutional layers to the VAE output, as many as there are pixelCNN layers. We allow each layer of the pixel-CNN to take additional input using non-masked convolutions from the feature stream based on the VAE output. This ensures that the conditional pixelCNN remains autoregressive.

To speed up training, we independently pretrain the VAE and pixelCNN in parallel, and then continue training the full model with both decoders. We use the Adamax optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with a learning rate of 0.002 without learning rate decay. We will release our TensorFlow-based code to replicate our experiments upon publication.

Quantitative performance evaluation.

Following previous work, we evaluate models on the test images using the bitsper-dimension (BPD) metric: the negative log-likelihood divided by the number of pixels values (3×32×32). It can be interpreted as the average number of bits per RGB value in a lossless compression scheme derived from the model.

The comparison in Table 1 shows that our model performs on par with the state-of-the-art results of the pixelCNN++ model [START_REF] Salimans | Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications[END_REF]. Here we used the importance sampling-based bound of [START_REF] Burda | Importance weighted autoencoders[END_REF] with 150 samples to compute the BPD metric for our model. 1 We refer to Figure 2 for qualitative comparison of samples from our model and pixelCNN++, the latter generated using the publicly available code.

Effect of KL regularization strength.

In Figure 3 we show reconstructions of test images and samples generated by the VAE decoder, together with their corresponding conditional pixelCNN samples for different values of λ. As expected, the VAE reconstructions become less accurate for larger values of λ, mainly by lacking details while preserving the global shape of the input. At the same time, the samples become more appealing for larger λ, suppressing the unrealistic high-frequency detail in the VAE samples obtained at lower values of λ. Note that the VAE samples and reconstructions become more similar as λ increases, which makes the input to the pixelCNN during training and sampling more consistent.

For both reconstructions and samples, the pixelCNN clearly takes into account the output of the VAE decoder, demonstrating the effectiveness of our aux- iliary loss to condition high-capacity pixelCNN decoders on latent variable representations. Samples from the pixelCNN faithfully reproduce the global structure of the VAE output, leading to more realistic samples, in particular for higher values of λ.

For λ = 2 the VAE reconstructions are near perfect during training, and the pixelCNN decoder does not significantly modify the appearance of the VAE output. For larger values of λ, the pixelCNN clearly adds significant detail to the VAE outputs.

Figure 4 traces the BPD metrics of both the VAE and pixelCNN decoder as a function of λ. We also show the decomposition in regularization and reconstruction terms. By increasing λ, the KL divergence can be pushed closer to zero. As the KL divergence term drops, the reconstruction term for the VAE rapidly increases and the VAE model obtains worse BPD values, stemming from the inability of the VAE to model pixel dependencies other than via the latent variables. The reconstruction term of the pixelCNN decoder also increases with λ, as the amount of information it receives drops. However, in terms of BPD which sums KL divergence and pixelCNN reconstruction, a substantial gain of 0.2 is observed increasing λ from 1 to 2, after which smaller but consistent gains are observed.

Role of the auxilliary representation

The auxilliary variables are taken into account: 3.3 shows that in theory it is always optimal for the autoregressive decoder to take the latent variables into account. Figure 5 demonstrates this empirically by displaying auxiliary representations f (z) with z sampled from the prior f (z) as well as nine different samples from the autoregressive decoder conditioned on f (z). This qualitatively shows that the low level detail added by the pixelCNN, which is crucial for loglikelihood performance, always respects the global structure of the image being conditioned on. The VAE decoder is trained with λ = 8 and weights very little in terms of KL divergence. Yet it controls the global structure of the samples, which shows that our setup can be used to get the best of both worlds. Figure 6 demonstrates that the latent variables z of the encoder have learned meaningfull structure with latent variable interpolations. Samples are obtained by encoding ground truth images, then interpolating the latent variables obtained, decoding them with the decoder of the V AE and adding low level detail with the pixelCNN.

The auxilliary loss is necessary: The fact that the autoregressive decoder ignores the latent variables could be attributed to optimization challenges, as explained in 3.3. In that case, the auxilliary loss could be used as an initialization scheme only, to guide the model towards a good use of the latent variables. To evaluate this we perform a control experiment where during training we first optimize our objective function in Eq. ( 7), i.e. including the auxiliary reconstruction term, and then switch to optimize the standard objective function of Eq. ( 6) without the auxiliary term. We proceed by training the full model to convergence then removing the auxiliary loss and fine-tuning from there. Figure 7 displays groundtruth images, with corresponding auxiliary reconstructions and conditional samples, as well as pure samples. The reconstructions have become meaningless and independent from the ground truth images. The samples display the same behavior: for each auxiliary representation four samples from the autoregressive component are displayed and they are independent from one another. Quantitatively, the KL cost immediately drops to zero when removing the auxiliary loss, in approximately two thousand steps of gradient descent. The approximate posterior immediately collapses to the prior and the pixel CNN samples become independent of the latent variables. This is the behavior predicted by the analysis of [START_REF] Chen | Variational lossy autoencoder[END_REF]: the autoregressive decoder is sufficiently expressive that it suffers from using the latent variables.

Effect of different auxiliary images.

We assess the effect of using coarser RGB quantizations, lower spatial resolutions, and grayscale in the auxiliary image. All three make the VAE reconstruction task easier, and transfer the task of modeling color nuances and/or spatial detail to the pixelCNN.

The VAE reconstructions in Figure 8 (a) obtained using coarser color quantization carry less detail than reconstructions based on the original images using 256 color values, as expected. To understand the relatively small impact of the quantization level on the reconstruction, recall that the VAE decoder outputs the continuous means of the logistic distributions regardless of the quantization level. Only the reconstruction loss is impacted by the quantization level via the computation of the probabilities over the discrete color levels in Eq. [START_REF] Goodfellow | Generative adversarial nets[END_REF]. In Figure 8 (b) we observe small but consistent gains in the BPD metric as the number of color bins is reduced, showing that it is more effective to model color nuances using the pixelCNN, rather than the latent variables. We trained models with auxiliary images down-sampled to 16×16 and 8×8 pixels, which yield 2.94 and 2.93 BPD, respectively. This is comparable to the 2.92 BPD obtained using our best model at scale 32×32. We also trained models with 4-bit per pixel grayscale auxiliary images, as in [START_REF] Kolesnikov | PixelCNN models with auxiliary variables for natural image modeling[END_REF]. While the grayscale auxilliary images are subjectively the ones that have the best global structure, the results are still qualitatively inferior to those obtained by [START_REF] Kolesnikov | PixelCNN models with auxiliary variables for natural image modeling[END_REF] with a pixelCNN modelling grayscale images. Our model does, however, achieve better quantitative performance at 2.93 BPD. In Figure 9 (a) we show samples obtained using models trained with 4-bit per pixel grayscale auxiliary images, in Figure 9 (b) with 32 color levels in the auxiliary image, and in Figure 9 (c) and (d) with auxiliary images of size 16×16 and 8×8. The samples are qualitatively comparable, showing that in all cases the pixelCNN is able to compensate the less detailed outputs of the VAE decoder and that our framework can be used with a variety of intermediate reconstruction losses.

Conclusion

We presented a new approach to training generative image models that combine a latent variable structure with an autoregressive model component. Unlike prior approaches, it does not require careful architecture design to trade-off how much is modeled by latent variables and the autoregressive decoder. Instead, this tradeoff can be controlled using a regularization parameter and choice of auxiliary target images. We obtain quantitative performance on par with the state of the art on CIFAR10, and samples from our model exhibit globally coherent structure as well as fine details.

Fig. 1 .

 1 Fig. 1. Schematic illustration of our auxiliary guided autoregressive variational autoencoder (AGAVE). The objective function has three components: KL divergence regularization, per-pixel reconstruction with the VAE decoder, and autoregressive reconstruction with the pixelCNN decoder.

Fig. 2 .

 2 Fig. 2. Randomly selected samples from unsupervised models trained on 32×32 CI-FAR10 images: (a) IAF-VAE [15], (b) pixelCNN++ [25], and (c) our hybrid AGAVE model. For our model, we show the intermediate high-level representation based on latent variables (left), that conditions the final sample based on the pixelCNN decoder (right).

Fig. 3 .

 3 Fig. 3. Effect of the regularization parameter λ. Reconstructions (a) and samples (b) of the VAE decoder (VR and VS, respectively) and corresponding conditional samples from the pixelCNN (PS).

Fig. 4 .

 4 Fig. 4. Bits per dimension of the VAE decoder and pixelCNN decoder, as well as decomposition in KL regularization and reconstruction terms.

Fig. 5 .

 5 Fig. 5. The column labeled f (z) displays auxiliary representations, with z sampled from the unit Gaussian prior p(z), accompanied by ten samples of the conditional pixelCNN.

Fig. 6 .Fig. 7 .

 67 Fig.6. The first and last columns contain auxilliary reconstructions, images in between are obtained from interpolation of the corresponding latent variables. Odd rows contain auxilliary reconstructions, and even rows contain outputs of the full model.

Table 1 :

 1 , and use logistic distributions over the RGB color Bits per dimension (lower is better) of models on the CIFAR10 test data.

	Model	BPD .|z .|xj<i
	NICE	[7] 4.48
	Conv. DRAW	[11] ≤ 3.58
	Real NVP	[8] 3.49
	MatNet	[2] ≤ 3.24
	PixelCNN	[21] 3.14
	VAE-IAF	[15] ≤ 3.11
	Gated pixelCNN	[20] 3.03
	Pixel-RNN	[21] 3.00
	Aux. pixelCNN	[17] 2.98
	Lossy VAE	[5] ≤ 2.95
	AGAVE, λ = 12 (this paper) ≤ 2.92
	pixCNN++	[25] 2.92

values. We let the intermediate representation f (z) output by the VAE decoder be the per-pixel and per-channel mean values of the logistics, and learn perchannel scale parameters that are used across all pixels. The cumulative density function (CDF), given by the sigmoid function, is used to compute probabilities across the 256 discrete color levels, or fewer if a lower quantization level is chosen in y. Using RGB values y i ∈ [0, 255], we let b denote the number of discrete color levels and define c = 256/b. The probabilities over the b discrete color levels are computed from the logistic mean and variance µ i and s i as

The graphs in Figure4and Figure8are based on the bound in Eq. (7) to reduce the computational effort.

Acknowledgments This work has been partially supported by the grant ANR-16-CE23-0006 "Deep in France" and LabEx PERSYVAL-Lab (ANR-11-LABX-0025-01).