Guillaume Le Guludec

Ehsan Miandji

Fellow, IEEE Christine Guillemot

Deep Light Field Acquisition Using Learned Coded Mask Distributions for Color Filter Array Sensors

Keywords: Light Field imaging, compressed sensing, deep learning, inverse problems

Compressive light field photography enables light field acquisition using a single sensor by utilizing a color coded mask. This approach is very cost effective since consumer-level digital cameras can be turned into light field cameras by simply placing a coded mask between the sensor and the aperture plane. This paper describes a deep learning architecture for compressive light field acquisition using a color coded mask and a sensor with Color Filter Array (CFA). Unlike previous methods where a fixed mask pattern is used, our deep network learns the optimal distribution of the color coded mask pixels. The proposed solution enables end-to-end learning of the color-coded mask distribution and the reconstruction network, taking into account the sensor CFA. Consequently, the resulting network can efficiently perform joint demosaicing and light field reconstruction of images acquired with color-coded mask and a CFA sensor. Compared to previous methods based on deep learning with monochrome sensors, as well as traditional compressive sensing approaches using CFA sensors, we obtain superior color reconstruction of the light fields.

I. INTRODUCTION

Light field imaging has recently gained interest in the research community, due to its potential for a variety of applications, going from computational photography, e.g. by enabling genuine post-capture refocusing, to medical imaging and virtual and augmented reality. While in conventional 2D imaging, each sensor element sums all the light rays emitted by one 3D scene point over the lens aperture, i.e. records a 2D projection of the 3D points on the image plane, light fields instead record the radiance along each ray emitted by the 3D points according to different orientations. The light field can hence be seen as capturing an array of viewpoints (or sub-aperture images) of the scene, leading to a 4D raybased scene representation. However, capturing a light field in a way that is memory and computationally efficient, as well as accurate, is challenging. Several camera architectures have been designed to capture light fields. Early attempts at capturing high resolution light fields include large camera arrays [START_REF] Wilburn | High Performance Imaging using Large Camera Arrays[END_REF] or a single camera placed on a moving gantry [START_REF] Levoy | Light Field Rendering[END_REF]. While these devices can produce high quality images both in the spatial and angular domain, they are in general quite bulky

The paper has been submitted for review on Dec. 1st. This work was supported in part by the EU H2020 Research and Innovation Programme under grant agreement No 694122 (ERC advanced grant CLIM), and in part by the french ANR research agency in the context of the artificial intelligence project DeepCIM. The authors are with Inria Rennes -Bretagne-Atlantique 263 Avenue Général Leclerc, 35042 Rennes Cedex, France (e-mail: firstname.lastname@inria.fr). and costly in terms of storage. More practical solutions include light-weight devices like lenslet-based cameras [START_REF] Ng | Light Field Photography with a Handheld Plenoptic Camera[END_REF] or angle sensitive pixel cameras [START_REF] Hirsch | A switch-able light field camera architecture with angle sensitive pixels and dictionary-based sparse coding[END_REF]. These devices, however, sacrifice spatial resolution for angular resolution, hence capture views with a significantly lower spatial resolution compared to traditional 2D cameras.

More recent light field camera designs consider coded masks instead of micro-lens arrays to modulate 4D light fields into 2D projections that can be captured by a digital camera sensor. Reconstruction algorithms are used, based on the compressed sensing paradigm [START_REF] Candès | Stable signal recovery from incomplete and inaccurate measurements[END_REF], to restore the original light field from its projections [START_REF] Babacan | Compressive sensing of light fields[END_REF][START_REF] Marwah | Compressive Light Field Photography using Overcomplete Dictionaries and Optimized Projections[END_REF][START_REF] Miandji | Multi-shot single sensor light field camera using a color coded mask[END_REF]. Compressed sensing provides a theoretical framework that enables the reconstruction of high-dimensional data from lower-dimensional projections when assuming additional signal constraints such as sparsity in a particular transform domain. Coded mask acquisition techniques are cost effective since a consumer-level digital camera can be used for light field acquisition by placing a coded mask in front of the sensor.

While compressed sensing originally provides theoretical guarantees for the signal reconstruction under signal sparsity assumptions [START_REF] Miandji | On probability of support recovery for orthogonal matching pursuit using mutual coherence[END_REF][START_REF] Chang | An improved rip-based performance guarantee for sparse signal recovery via orthogonal matching pursuit[END_REF][START_REF] Tropp | Greed is good: algorithmic results for sparse approximation[END_REF], it has been empirically shown that deep priors on the signal, like the deep generative hypothesis [START_REF] Bora | Compressed sensing using generative models[END_REF], outperform the sparsity priors for a great number of image reconstruction tasks. The authors of [START_REF] Bora | Compressed sensing using generative models[END_REF] also provide theoretical evidence that deep priors can replace the traditional sparsity hypothesis. Moreover, reconstruction techniques based on deep networks usually allow for a reconstruction of the signal in one forward pass through a deep model, making them orders of magnitude faster than traditional iterative methods like the orthogonal matching pursuit [START_REF] Pati | Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition[END_REF].

Deep learning architectures have been proposed in [START_REF] Vadathya | Learning light field reconstruction from a single coded image[END_REF][START_REF] Gupta | Compressive light field reconstructions using deep learning[END_REF][START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] for light field reconstruction from a sparse set of measurements recorded on a monochrome sensor. For compressive light field acquisition, Nabati et al. [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] were able to get state-of-the-art results with a monochrome sensor using a fully convolutional network and a low-entropy random red-greenblue-white (RGBW) color-coded mask. In [START_REF] Guo | Deep spatial-angular regularization for compressive light field reconstruction over coded apertures[END_REF], the authors propose a convolutional network architecture to compute the coded sub-aperture images. The light field reconstruction problem is then solved using an iterative optimization approach with a deep spatio-angular regularization prior. The end-toend pipeline is applied to each color channel independently. None of the previous work takes into account the fact that camera sensors are usually equipped with a color filter array (CFA) [START_REF] Bayer | Color Imaging Array[END_REF] performing a form of color compression. Much research has been done in trying to find optimal CFAs and demosaicing algorithms for 2D imaging. The authors of [START_REF] Henz | Deep joint design of color filter arrays and demosaicing[END_REF] obtained state-of-the-art results on traditional 2D images by jointly learning a deep demosaicing network and optimizing the CFA itself. Note that the learning of optimal masks has also been considered in [START_REF] Chakrabarti | Learning sensor multiplexing design through back-propagation[END_REF], [START_REF] Sitzmann | End-toend optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging[END_REF], [START_REF] Wu | Phasecam3d -learning phase masks for passive single view depth estimation[END_REF], but for completely different problems. The authors in [START_REF] Chakrabarti | Learning sensor multiplexing design through back-propagation[END_REF] address the problem of jointly optimizing a color filter array for 2D image sensors and of the de-mosaicing process. The authors in [START_REF] Sitzmann | End-toend optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging[END_REF] aim at extending the depth of field from 2D captures by proposing a wave-based image formation model and a joint optimization of a diffractive optical element together with a de-convolution method. The problem addressed in [START_REF] Wu | Phasecam3d -learning phase masks for passive single view depth estimation[END_REF] is monocular scene depth estimation by using learned phase masks. None of the three methods addresses the problem of light field acquisition with optimized color coded masks that we consider here.

In this paper, we propose a deep learning architecture for an end-to-end optimization of both the color-coded mask distribution and the reconstruction algorithm. The problem of optimizing the coded mask has previously been addressed in [START_REF] Marwah | Compressive Light Field Photography using Overcomplete Dictionaries and Optimized Projections[END_REF] by minimizing the mutual coherence of the equivalent sensing matrix, and in [START_REF] Guo | Deep spatial-angular regularization for compressive light field reconstruction over coded apertures[END_REF] by using deep learning techniques. In contrast to this prior work, we propose a new approach based on learning the distribution of the mask pixels. Indeed, the learned distribution admits a higher representation power compared to a fixed mask pattern learned over a given training set. In other words, the mask distribution generalizes well to unseen test data, as shown by our results in Section V. Moreover, it has been shown that adaptive sampling, e.g. via learning a mask pattern, cannot outperform naive random sampling in terms of Mean Square Error (MSE) of the estimation [START_REF] Arias-Castro | On the fundamental limits of adaptive sensing[END_REF]. In addition, unlike previous methods where the mask is learned independently, our proposed framework jointly optimizes the mask distribution and the reconstruction network in an end-to-end optimization approach.

We also consider the case where the sensor is equipped with a color filter array which is taken into account in the optimization of the color coded mask distribution. We show that the proposed framework can efficiently perform joint demosaicing and light field reconstruction of images acquired with a colorcoded mask and a CFA-equipped sensor. Experimental results show that using a CFA, compared to monochrome sensors as in [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF], enables a superior reconstruction quality of a wide variety of light field images.

In summary, our contributions are as follows:

• We propose a novel reconstruction architecture for light field acquisition with learned color coded masks. We show that the results obtained with the proposed architecture are superior, with an average PSNR gain of 1.36 dB, when compared to state-of-the-art methods. • We then propose an end-to-end learned generator of the color coded mask distribution, which is shown to further improve the reconstruction quality. • We also introduce a Bayer CFA in the acquisition and reconstruction pipeline and show that, when combined with the learned color coded mask, it further improves the reconstruction quality. The complete pipeline outperforms by almost 2 dB the state-of-the-art method. • We further include the learning of the color filter array in the pipeline and show that it improves over using a

Bayer CFA, especially in presence of noise. • Finally, we show that the approach is robust to noise when the noise level is low. However, when the noise level increases, the learned CCM converges towards a uniform transparent mask, impacting the overall reconstruction performance. To cope with this problem, we further introduce an entropy-based regularization of the codedmask and show that this regularization constraint leads to an average of 2 dB improvement in presence of a high level of noise.

II. RELATED WORK

Mask-based cameras with compressed sensing

Programmable aperture approaches have first been considered to sequentially capture subsets of light rays. The idea consists in time-multiplexing 2D slices of the 4D light field on the sensor, using a programmable non-refractive mask placed at the aperture as in [START_REF] Liang | Programmable Aperture Photography: Multiplexed Light Field Acquisition[END_REF]. The latter design exploits the fast multiple-exposure feature of digital sensors. The authors in [START_REF] Veeraraghavan | Dappled photography: Mask enhanced cameras forheterodyned light fields and coded aperture refocusing[END_REF] use instead optical heterodyning to frequency multiplex the 4D Fourier transform of the light field into spatio-angular bands on the 2D sensor. A good overview of the above camera designs can be found in [START_REF] Wetzstein | State of the Art in Computational Plenoptic Imaging[END_REF].

Since the light field data is typically high dimensional and compressible, its acquisition can be placed in a compressive sensing framework, in which the sensing matrix is materialized by a coded physical mask. Thanks to the use of a coded mask, instead of recording a spatial multiplex of 2D slices of the light field, as in micro-lens array based camera architectures, the photosensor records a set of linear measurements from which a higher resolution light field can be reconstructed. This compressive sensing principle is applied in [START_REF] Xu | A high-resolution lightfield camera with dual-mask design[END_REF][START_REF] Xu | High-resolution Lightfield Photography Using Two Masks[END_REF] where the 2D sensor captures optically coded projections using two attenuation masks separately placed at the aperture plane and in front of the sensor. Given the measurements recorded on the sensor, the light field is then reconstructed using a least square minimization with a total variation regularization constraint. Similarly, the authors in [START_REF] Babacan | Compressive Light Field Sensing[END_REF] place a randomly coded mask on the aperture plane to obtain incoherent measurements of the light field. Multiple shots are captured as random linear combinations of angular images by separately opening one region of the aperture and blocking the light in the others.

The authors in [START_REF] Marwah | Compressive Light Field Photography using Overcomplete Dictionaries and Optimized Projections[END_REF] propose a camera architecture that records optically coded projections on a single image sensor using a monochrome mask, while the authors in [START_REF] Miandji | Compressive Image Reconstruction in Reduced Union of Subspaces[END_REF] and [START_REF] Miandji | Multi-shot single sensor light field camera using a color coded mask[END_REF] use respectively a random stationary or a moving colorcoded mask to extract incoherent measurements. In both cases, the light field is then reconstructed using a compressive sensing framework, assuming that the light field is sparse in a domain defined by an overcomplete dictionary [START_REF] Marwah | Compressive Light Field Photography using Overcomplete Dictionaries and Optimized Projections[END_REF], [START_REF] Miandji | Multi-shot single sensor light field camera using a color coded mask[END_REF] or an ensemble of 2D separable dictionaries [START_REF] Miandji | Compressive Image Reconstruction in Reduced Union of Subspaces[END_REF]. The authors in [START_REF] Nguyen | Multi-mask camera model for compressed acquisition of light fields[END_REF] introduce an Equivalent Multi-Mask Camera (EMMC) model which unifies most existing single lens mask-based light field cameras, allowing for a flexible configuration of a variety of sensing schemes.

Mask-based cameras with deep reconstruction

While the first solutions were considering classical sparse reconstruction methods, the problem of light field reconstruction can also be efficiently solved using deep learning techniques [START_REF] Vadathya | Learning light field reconstruction from a single coded image[END_REF][START_REF] Gupta | Compressive light field reconstructions using deep learning[END_REF][START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF][START_REF] Inagaki | Learning to capture light fields through a coded aperture camera[END_REF]. The authors in [START_REF] Vadathya | Learning light field reconstruction from a single coded image[END_REF][START_REF] Gupta | Compressive light field reconstructions using deep learning[END_REF][START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] assume a pre-defined and fixed mask pattern and propose convolutional neural network architectures to reconstruct the light field from the set of measurements using the coded mask. Given coded measurements, the method in [START_REF] Gupta | Compressive light field reconstructions using deep learning[END_REF] generates two coarse light fields which are then fused to generate the final estimate of the original light field.

The authors in [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] introduce a sensing matrix which modulates both color and angular information of a light field into 2D sensor measurements. The sensing matrix together with the coded measurements are fed into a CNN-based network to reconstruct the light field. In contrast, the authors in [START_REF] Inagaki | Learning to capture light fields through a coded aperture camera[END_REF] pose the coded aperture acquisition and light field reconstruction as an auto-encoder and optimize the mask pattern together with the parameters of the reconstruction algorithm in an endto-end auto-encoder learning. A convolutional kernel of size 1 × 1 is used to simulate the coded aperture process. Then, two sequential sub-networks, the second one being based on the VDSR network [START_REF] Kim | Accurate image super-resolution using very deep convolutional networks[END_REF], are used to reconstruct the light field from the coded measurements. Our acquisition model in this paper is distinct from [START_REF] Inagaki | Learning to capture light fields through a coded aperture camera[END_REF] since we place a mask between the aperture plane and the sensor. A learned convolutional network architecture is used in [START_REF] Guo | Deep spatial-angular regularization for compressive light field reconstruction over coded apertures[END_REF] to compute the coded subaperture images, from which the light field is reconstructed using an iterative optimization approach with a deep spatioangular regularization prior. The end-to-end pipeline is applied on each color channel separately.

Unlike the above solutions that assume the sensor to be monochrome, use a fixed mask pattern, or independently optimize a coded aperture pattern, here we propose an end-toend learning framework to jointly optimize the color-coded mask distribution and the reconstruction network. We also consider the case where the sensor is equipped with a CFA, which is commonly used for digital consumer-level cameras.

Phase-mask coded aperture camera designs

The authors in [START_REF] Elmalem | Learned phase coded aperture for the benefit of depth of field extension[END_REF] introduce a solution for enhanced depthof-field based on a binary phase-mask composed of a ring pattern, whereby each ring introduces a different phase-shift to the wavefront emerging from the scene. The phase-mask is designed such that an end-to-end trained CNN is able to restore from the aperture coded image an all-in-focus image, hence with a large depth of field. A layer of the CNN models the phase-mask parameters (ring radii and phase). A similar phasecoded aperture camera is proposed in [START_REF] Haim | Depth estimation from a single image using deep learned phase coded mask[END_REF] for monocular depth estimation. An optical phase mask provides depth-related color characteristics that are then used for estimating scene depth with a fully convolutional neural network. Given that depthdependent defocus 'bokeh', or the point spread function, depends on the amplitude and phase of the aperture, a solution is proposed in [START_REF] Wu | Phasecam3d -learning phase masks for passive single view depth estimation[END_REF] to end-to-end optimize a phase mask and an algorithm that allows accurate scene depth estimation from a single viewpoint. Light propagation from the scene to the

Π CCM Σ li g h t b ea m optical axis u • x • ξ x,u • 1 γ
Fig. 1: Two-plane parameterization with color-coded mask.

A light beam is characterized by the coordinates of its intersection with the reference plane Π and the sensor plane Σ, denoted respectively u and x. The beam intersects the mask plane at coordinates

ξ x,u = (1 -γ)x + γu.
sensor and the modulation by the mask are modeled as frontend layers of a deep neural network. The problem addressed in [START_REF] Chakrabarti | Learning sensor multiplexing design through back-propagation[END_REF] is instead the design of color filter arrays for 2D image sensors, by end-to-end learning of the color pattern and of the image reconstruction which is therefore a de-mosaicing problem. The authors in [START_REF] Sitzmann | End-toend optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging[END_REF] propose a wave-based image formation model for 2D cameras and an approach for joint optimization of a diffractive or refractive element together with a de-convolution method for achromatic extended depth of field and snapshot super-resolution 2D imaging.

In contrast, the proposed approach differs from all the above methods, since it concerns compressive light field acquisition, and does not concern coded aperture cameras using phase masks nor problems of depth estimation or depth of field extension with 2D cameras.

III. MATHEMATICAL FRAMEWORK

A. Light fields

We adopt the two-plane parameterization of light fields, in which any light field can be represented by a collection of light rays passing through two points on a pair of parallel planes. In addition to a sensor plane Σ, we consider a reference plane Π that usually coincides with the aperture plane. A light ray is then defined by a set of coordinates on the sensor plane (x, y), called the spatial coordinates, together with a set of coordinates on the reference plane (u, v) called the angular coordinates, see Figure 1. The first set of coordinates serves to describe the point of incidence of the ray on the sensor plane Σ, whereas the second set (u, v) defines the direction of the light ray. To simplify the notations, in the remainder of the paper we use x = (x, y) to designate the pair of spatial coordinates, and u = (u, v) to designate the pair of angular coordinates.

In this framework, a light field is a real function of a fivedimensional real vector space defined as

L : Σ × Π × Λ R 5 → R (x, u, λ) → L(x, u, λ),
where λ corresponds to a given wavelength and Λ is the space of all possible wavelengths. The projection of the light field on the sensor plane is then given by

I(x) = Π×Λ L(x, u, λ)φ(λ)dudλ, (1)
where φ weights the sensitivity of the sensor to various wavelengths. For simplicity, we also consider the vignetting effect to be included in L.

B. Compressed light field acquisition

Compressed light field acquisition aims at reconstructing a light field using lower-dimensional projections captured on the sensor plane. The compressed sensing theory indeed tells us that, under conditions of incoherence between the measurement space and the signal space, a signal can be recovered from a sparse set of measurements, provided the signal is sparse in a particular transform domain. The incoherence property is satisfied by taking random measurements. Concerning light field acquisition, this implies applying an additional linear modulation of the light field before the projection. This can be expressed as

I(x) = Π×Λ L(x, u, λ)M (x, u, λ)dudλ (2)
where M represents a modulation of the light field that may depend on the spatial coordinates, the angular coordinates, and the wavelength. 1) Color-coded mask: We are interested in the case where the modulation is performed by a color-coded mask (CCM) placed in front of the sensor plane. The mask performs some filtering or light attenuation and can be defined by a function m CCM : (u, λ) → m CCM (u, λ). Simple geometric considerations show that in this case

I(x) = Π×Λ L(x, u, λ) • m CCM (ξ x,u , λ)dudλ (3)
where we assume the distance between the reference and sensor planes to be unit, and ξ x,u = (1 -γ)x + γu with γ being the distance between the sensor and the mask. Note that this is equivalent to the previous equation with the additional constraint that the modulating function satisfies M (x, u + ∆u, λ) = M (x + γ∆u, u, λ). Figure 1 gives a visual explanation for equation 3.

2) Color filter array: The above equations implicitly assume monochromatic sensors. Nonetheless, it is possible to rewrite these formulae such that the acquisition is performed using a sensor equipped with a color filter array. A color filter array is a grid placed directly on the sensor plane that defines how much of every wavelength a given pixel will be sensitive to. It can thus be defined, much like the CCM, by a function m CFA : (x, λ) → m CFA (x, λ). CFAs are generally periodic, and the most widespread CFA pattern is probably the Bayer pattern.

3) Effective modulating function: In the presence of both a color-coded mask m CCM and a CFA m CFA , the sensing equation becomes

I(x) = Π×Λ L(x, u, λ)m CCM (ξ x,u , λ)m CFA (x, λ)dudλ (4) = Π×Λ L(x, u, λ)M effective (x, u, λ)dudλ, ( 5
)
where M is the effective mask resulting from the product of both the CFA that performs color compression and the CCM that performs the angular compression, as well as color compression. Because the actual measurement of the image is performed on a finite grid, the above equation can be re-written in a discretized form as

I(x) = u,λ L(x, u, λ) • M effective (x, u, λ), (6)
with the sum being over a finite set of size ω • ν • κ, where ω is the overall spatial resolution, ν is the angular resolution, and κ is the spectral resolution, that is, the number of color channels. In what follows, we assume κ = 3.

IV. DEEP ACQUISITION AND RECONSTRUCTION

ARCHITECTURE

We describe an architecture that allows us to define and train a deep convolutional network to reconstruct a light field from compressed measurements recorded by a CFA-equipped sensor, along with the optimization of the CCM distribution. The end-to-end architecture for mask generation, as well as the compressed acquisition and reconstruction, is depicted in Figure 2.

We consider a (discretized) light field as a 5D tensor of dimensions (ω x , ω y , ν u , ν v , κ). Similarly, the discretized version of the effective modulating function M can be considered as a 5D tensor of same dimensions. Note that the values of the effective modulating function correspond to the values of a uniform light field modulated by this function. The sensing equation in ( 6) can be functionally reformulated as

I = Sensing(L, M ), (7)
where Sensing(.) is a bilinear operator performing the element-wise multiplication of the tensors followed by summation over the angular and spectral domains. Note that in this formalism, the measurement matrix is implicitly defined, since Sensing(.) is a bilinear operator. The authors of [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] use this formalism to devise a simple reconstruction scheme. The idea is to simply feed the sensed image along with the effective modulating function values in a way that is readily processable by a feed-forward fully convolutional network. The values of the modulating function are presented as a (ω x , ω y , ν • κ) tensor, which is interpreted as a collection of feature maps, each feature map corresponding to a color channel of a subaperture view. The modulating values are then concatenated with the sensed image to yield a ν • κ + 1-channels image. Our idea is to learn the values of the modulating function together with the light field reconstruction network.

A. Modulating tensor generation

We consider color-coded masks in which the transmittance of all pixels are drawn from the same distribution and are independent from one another. The modulation is processed by element-wise multiplication with a stack of sub-aperture modulating images which are translated versions of each others, with a translation proportional to γ. Therefore, only pixels sufficiently far apart will be correlated, which ensures that the values of the modulation function are locally independent and identically distributed. Given that the CNN will not be able to capture correlation between distant pixels, we can assume that the elements of the modulating functions corresponding to a CCM are i.i.d. without sacrificing the accuracy.

We therefore seek to find a suitable probability distribution for the transmittance of the modulating tensor pixels. To this end, we train a feed-forward generator network

seed → f gen (seed, θ) (8)
to output a κ-dimensional transmittance for each pixel, that is, an element of [0, 1] κ , given a random seed drawn from a distribution in a d-dimensional space (in our experiments, we used the standard normal distribution N (0, 1) on R d ). This function is then applied element-wise on a random (ω x , ω y , ν u , ν v ) tensor to yield the modulating tensor (ω x , ω y , ν u , ν v , κ). The effective modulating tensor is then computed as the entrywise product of the previously generated color-coded mask tensor with the modulating tensor of the CFA, which consists of identical copies of the color array for each sub-aperture view, see Figure 2. The process of generating the effective modulating tensor is differentiable, and thus, when linked to a subsequent differentiable reconstruction network, allows for an end-to-end implementation and learning of the mask generation and light field reconstruction pipeline. In all our experiments, we used a dense generator network composed of 3 dense hidden layers with ReLu activations, and a final dense layer with output dimension of 3 and sigmoid activation, to ensure that the output corresponds to an actual color.

B. Reconstruction neural network

We use a convolutional neural network with skip connections (ResNet [START_REF] He | Deep residual learning for image recognition[END_REF]) as our reconstruction model. It consists of a stack of identical skip connection blocks, as depicted in I.

C. End-to-end learning of the mask generation and light field reconstruction pipeline

The end-to-end learning of the complete mask generation, acquisition and reconstruction architecture depicted in figure 2 is performed by minimizing the L 1 distance between the ground truth input light fields and the light fields reconstructed by the network

L = x,u,λ |L g.t. (x, u, λ) -L reconstructed (x, u, λ)|
The end-to-end architecture is fully differentiable, which makes it possible to train everything by gradient backpropagation. No additional regularization term was used.

V. EXPERIMENTAL RESULTS

We assessed the proposed framework assuming both a monochrome sensor and a sensor with a built-in CFA. We also considered both a fixed distribution for the color-coded mask and a learned one with the proposed color coded mask generator.

We compare our results with those obtained with the dictionary-based reconstruction method of [START_REF] Nguyen | Multi-mask camera model for compressed acquisition of light fields[END_REF] and with the solution proposed in [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] and using monochromatic color filter arrays. Note that the authors in [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] considered three different distributions for the color coded mask:

• uniform distribution in [0, 1] 3 ; • either red, green or blue with equal probability (RGB); • either red, green, blue or white with equal probability (RGBW). since they found that the RGBW distribution performed best, we are comparing with their results using the RGBW mask. We found that the proposed reconstruction network indeed gave similar performances for both the RGBW and the uniform distributions.

A. Data sets

We evaluate the proposed framework using natural light fields from the Stanford Lytro Light Field Archive [START_REF] Raj | Stanford Lytro Light Field Archive[END_REF] and the Lytro light field data set provided by Kalantari et al. [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF]. These natural light fields have been captured using the Lytro Illum camera and have an angular resolution of 14×14 and 8×8, respectively. Since these light fields suffer from strong vignetting effects on peripheral views due to mechanical and optical imperfections, we only take into account the 5 × 5 central angular images as reference (a.k.a ground truth) in our experiments.

All models were trained on 122 light fields. Among these 122 light fields, 100 were originally provided by [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] as a training dataset (72 are original light fields from [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF], 28 original from the Stanford dataset [START_REF]Stanford lytro light field archive[END_REF]). The remaining 22 light fields were originally provided for validation by [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] and we removed two light fields that were too similar to the Orchids light field, since it is conventionally used for comparison with other methods. We instead used 14 light fields from Linköping University (these light fields will be made available online soon) as our validation set.

B. Training details

While we could train all networks from scratch in a straightforward manner, we used instead the training flow illustrated in Figure 5. Fine-tuning a reconstruction network pre-trained for another task not only gives a reduced training time but also gives better performances.

All models were trained using the Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] with the hyper-parameters β 1 = 0.9 and β 2 = 0.999, and with a numerical stability term = 10 -5 for all four tasks using batches of size 16. For the initial task (i.e. monochromatic sensor + uniform CCM distribution) we used an initial learning rate α = 5 • 10 -3 and reduced it by a factor 10 1/4 every 30 epochs. We first trained the model for 210 epochs. We also found it beneficial to train for 120 additional epochs after the network had converged, restarting from a fresh optimizer (i.e. one that needs to gather the statistics again) with the same initial learning rate and the learning rate decay schedule.

The remaining three tasks were all trained using an initial learning rate of α = 5 • 10 -4 for 60 epochs, reducing the learning rate by a factor 10 1/4 every 10 epochs. Following a standard procedure, we trained on light field patches of size 100 × 100 in the spatial domain. Due to the full translational II: PSNR (dB) comparison with a dictionary-based reconstruction method [START_REF] Nguyen | Multi-mask camera model for compressed acquisition of light fields[END_REF] and with reference deep learning based methods [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] and [START_REF] Guo | Deep spatial-angular regularization for compressive light field reconstruction over coded apertures[END_REF], using a one shot acquisition scheme.

(i) (ii) (ii) † (iii) (iv) (iv) † (v) †

C. Comparison of effective modulating tensors

Figure 4 shows a collection of color-coded masks and effective modulating tensors (calculated by element-wise multiplication of the color coded mask and of the color filter array) that we obtained and/or used in our experiments. Figure [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] and [START_REF] Guo | Deep spatial-angular regularization for compressive light field reconstruction over coded apertures[END_REF], using a one shot acquisition scheme. 6 represents the corresponding learned distributions over the color space. Figures 4b and4c shows the aspect of the learned CCM in the case of a monochromatic sensor for 1D and 8D seeds, respectively. One can note that the 1D mask shows less variability in the colors than the 8D mask. This can be explained by the dimension of the support of the distribution (smaller in the 1D case), which can be visualized in Figures 6a and6b. In both cases, however, the generated colors can span a wide range of hues.

Method (i) (ii) (ii) † (iii) (iv) (iv) † (v) †
Figures 4e and4d show the appearance of the learned CCM in the case of a Bayer CFA for 1D and 8D seeds respectively. These masks are almost indistinguishable, even though figures 6c and 6b show that the support in the 1D case is 1-dimensional, while in the 8D case it is 3-dimensional. In fact, even though the model for the 8D color generator has enough capacity to span the entire cube, the learned distribution practically fits on a 1-dimensional manifold, showing a very distinct single degree of freedom. This is in contrast with the case with monochromatic sensor. Our hypothesis is that in the case of no CFA, the task of color compression is performed only by the CCM (in addition to the angular compression), while in the Bayer case, the CFA will help to perform the color-compression too; thus, reducing the need for the CCM to span a large portion of the color cube and present a diversity of hue. Section IV-A discusses these phenomena in more details.

D. Comparison to the state of the art

Tables II and III present a detailed comparison of the various reconstructions obtained using our method with two reference methods, one method using a classical dictionary-based reconstruction followed by a total variation regularization (called Dict-TV) in [START_REF] Nguyen | Multi-mask camera model for compressed acquisition of light fields[END_REF] and the deep learning method of [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF]. Note that although the model in [START_REF] Nguyen | Multi-mask camera model for compressed acquisition of light fields[END_REF] allows for any number of acquisitions (multiple shots), for the sake of the comparison here we consider only one shot acquisition. Moreover, note that since the authors in [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] show that the RGBW mask gives superior results with their architecture, this mask is used for the results of [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] reported here. To be able to compare to [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF], all the light fields were cropped on the right and bottom after reconstruction to be of spatial size 350 × 500. Since our color-coded mask is non-deterministic, we take the mean of the PSNR over 32 independent realizations. The estimated standard deviation is always < 0.06 dB. We also include a comparison with the method of [START_REF] Guo | Deep spatial-angular regularization for compressive light field reconstruction over coded apertures[END_REF], using the implementation provided by the authors. Since the trained model provided by the authors solves the task of reconstructing a 7x7 light field from 2 shots, we had to retrain their model for solving the task 1 → 5x5. We used the default hyper-parameters provided in their implementation, but used the same training dataset we used for training our model. [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF]. Note how the use of Bayer CFA allows for a more accurate recovery of colors, especially in high-saturation areas (see zoom-in; colors better visualized on a monitor). PSNR is indicated for the whole light field, not only the central view.

A comparison of the quantitative metrics in Table II shows that the proposed pipeline with the novel reconstruction network, the learned mask distribution, and the color filter array leads to superior performance with respect to both [START_REF] Nguyen | Multi-mask camera model for compressed acquisition of light fields[END_REF] and [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF]. Qualitative results for a variety of configurations for our method using two selected light fields, namely Orchids and Tulips, are shown in figures 7 and 8, respectively. We observe a better reconstruction quality with regards to the color information when compared with the results of [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF]. Note that the superiority of using a Bayer CFA over a monochrome one is especially visible on colorful light fields like Orchids and Tulips. Figure 9 also shows that the structures in the epipolar plane images are well preserved, hence the parallax is well reconstructed. Videos of reconstructed light fields, affirming the faithful parallax reconstruction, can be accessed at http://clim.inria.fr/research/DeepLFCam/index.html.

To further assess the robustness of our model, we also applied the proposed acquisition and reconstruction pipeline to synthetic light fields of the HCI dataset [START_REF] Wanner | Datasets and benchmarks for densely sampled 4d light fields[END_REF]; the results are summarized in figures 11 and 10. We would like to emphasize the fact that the light fields in the training set, which are captured using a Lytro camera, are significantly distinct to those of the HCI dataset. When comparing reconstructed central views and EPIs of synthetic light fields, as shown in Figures 10 and11, one can see that the learned model performs well for light fields that are very different from the training ones. We invite the reader to view the animated reconstructed light fields at http://clim.inria.fr/research/DeepLFCam/index.html.

E. Ablation study

1) Proposed reconstruction network: While inspired by the reconstruction network of [START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF], our reconstruction networks differ in the use of skip connections in the same manner as ResNet [START_REF] He | Deep residual learning for image recognition[END_REF]. When comparing column (i) with the last column of Table II, one can see that our architecture, even without a CFA and with a uniform coded-mask, is superior on each of the test light fields, with an average PSNR gain of 1.36dB.

2) Use of the color filter array: Table II shows the impact of the CFA and the CCM on the reconstruction quality. It can be seen that using a Bayer CFA instead of a monochromatic one contributes to improving the PSNR, in both cases of a uniform CCM or of a learned one. In addition to these quantitative results, we found that using a Bayer CFA instead of a monochromatic sensor was beneficial in terms of color reconstruction quality, while maintaining the parallax reconstruction. Table II, and figures 7 and 8, give a more detailed comparison of the different configurations of our pipeline, making apparent the benefits of using a Bayer CFA for color reconstruction.

3) Joint learning of the CFA: In addition to the learning of the coded mask, we also conducted experiments to jointly learn the color filter array. In our experiments, we learn a 4 × 4 pattern initialized with random values following a uniform distribution U(0, 1). At each training step the values are clipped to remain between 0 and 1, ensuring the physical feasibility of the CFA. Column (v) † of table II shows that jointly learning the CFA yields better results than using either a fixed monochrome sensor or a Bayer CFA, although

Noise

Low improving over the Bayer CFA by less than 0.1 dB. However the effectiveness of learning the CFA becomes much more visible in presence of noise as shown in Section V-F, yielding improvement of more than 1 dB compared to using a fixed Bayer CFA. Figure 12 shows color filter arrays corresponding to various levels of noise.

4) d-dimensional seed versus 1-dimensional seeds: Since the color distribution of the pixels of the CCM is learned by training a continuous mapping from R d to [0, 1] 3 , the support of the distribution is a manifold whose dimension is at most d. If for instance we choose d = 1, the support of the distribution will be a continuous and piece-wise differentiable curve within the color cube. No such mapping can therefore perfectly fit distributions whose support is 2D or 3D. It is, nonetheless, possible to approximate any distribution, but at the expense of an arbitrarily complex winding curve. For this reason, we expect generators with high-dimensional seeds to perform better in general. A visual representation of this fact is presented by Figure 6. We empirically found that using an 8D seed, instead of a 1D one, leads to an increase in PSNR of about 0.15dB.

5) Impact of CCM resolution:

The initial design assumes that the coded mask M (ξ, λ) has a feature size M , i.e. the size of each element or pixel of the mask, and that it is placed at a distance d from the sensor plane (0 ≤ d ≤ D) in a way that d M is sufficiently small with respect to the visible spectrum (corresponding to the wavelength in the range between 0.4 and 0.8 µm). Under these conditions, the diffraction effect induced by the mask M can be ignored [START_REF] Asif | FlatCam: Thin, Lensless Cameras Using Coded Aperture and Computation[END_REF]. However, we have further assessed the impact of the CCM resolution, i.e. of the size of the CCM elements, on the reconstructed light fields. In that case, the reconstruction problem becomes a joint problem of reconstruction and de-convolution. Table V shows the impact of the CCM resolution on the reconstruction. One remarks that diminishing the resolution of the CCM can, up to a certain point, improve the performance of the reconstruction network. One explanation to this seemingly counter-intuitive fact is that, as we reduce the resolution, we also reduce the variety of patterns to which the reconstruction network can better adapt. We also hypothesize that the presence of the Bayer CFA makes the system more robust to a lower resolution of the CCM, as some color multiplexing is still performed at the pixel level. Decreasing too much the CCM resolution hurts the performance, as it makes the sensing matrix less incoherent.

F. Noise Analysis 1) Noise model: In order to assess the robustness of our method under practical conditions, we have corrupted the coded projections with sensor random noise. Our corruption model takes into account shot noise due the quantized nature of light, readout noise caused by spontaneous emission of electrons in the sensor, and rounding happening during analogdigital conversion. Denoting I the normalized ground truth intensity (i.e. proportional to the irradiance) of a pixel, the normalized intensity measured by the sensor is given by: However, with this model, I sensor is not differentiable with respect to I, because of the rounding operator and the sampling of the Poisson distribution. This non-differentiability prevents the back-propagation of gradients to elements upstream of the sensing operator. In order to make it differentiable, we replaced the rounding operation by a uniform additive noise n rounding ∼ U(-1/2, 1/2) and p(gcI) by a Gaussian variable gcI + √ gcI • n shot where n shot ∼ N (0, 1). Such substitutions preserve the order of magnitude of the corruption applied to the signal, while allowing to back-propagate gradients up to the CFA and CCM.

In our experiments, we set σ readout = 40, N = 2 14 -1 and c = 20000, which are typical parameters corresponding to a medium-low quality photosensor. We consider two values of gain: g = 1 and g = 0.33, which correspond on our dataset to noise levels of about 1.5% and 3% respectively. We experimented with both a fixed Bayer CFA and a trainable CFA. Table IV shows that our model is robust to noise, when the noise level is not too high. It also demonstrates that learning the CFA further increases the reconstruction quality in presence of noise. When the noise level is high, to avoid the harmful impact of the noise on the learned CCM and preserve a good robustness of the approach, we introduce, as explained below, an entropy-based regularization of the CCM.

2) Entropy regularization of the coded mask: We noticed that, as we increase the level of noise, the color distribution learned by the coded mask tends to become less diverse. With moderate noise levels, the coded mask rapidly converges to a binary mask, both with a fixed Bayer CFA and a learned CFA. When increasing the noise level, the coded mask converges to a uniform transparent mask. One possible explanation is that because of the noise, the coded mask learns very early to increase its light efficiency and falls into a local minimum where the CCM distribution is very simple, because the task of reconstructing from simply multiplexed measurements is easier than the task of reconstructing from incoherent measurements, since the reconstruction network has to learn how to make use of the sensing matrix. In order to overcome this problem, we encouraged variety in the CCM distribution by adding a novel regularization term on the output of the CCM generator network. This regularization term is given by: R CCM = max(0, H threshold -ĤKL (c 1 , . . . , c n ))

where c 1 , . . . , c n are n color values sampled from the CCM generator, ĤKL (.) is the Kozachenko-Leonenko entropy estimate [START_REF] Kozachenko | Sample estimate of the entropy of a random vector[END_REF], and H threshold is a threshold value. This regularizer effectively enforces the coded mask distribution to maintain an entropy above the threshold H threshold . The Kozachenko-Leonenko estimator is a continuous, piece-wise differentiable estimator [START_REF] Lombardi | Nonparametric k-nearestneighbor entropy estimator[END_REF], and is easily implementable in modern deep learning frameworks, making it suitable for gradient-based learning with automatic differentiation. Table IV shows the effectiveness of the regularization, especially in the case of strong noise where it yields an average improvement of about 2 dB.

VI. CONCLUSION

We have presented an end-to-end learning framework for compressive light field acquisition with coded masks. While the state-of-the-art method only considers monochrome sensors, we also consider sensors with in-built CFA and we have show its impact on the reconstruction quality. Compared to state-of-the-art methods, our framework also learns the distribution of the colour coded mask, which is shown to outperform masks based on fixed uniform distributions. The reconstruction network using skip connections is also shown to outperform the reference architecture in the same acquisition conditions.

In addition to the learning of the coded mask distribution, we have considered the joint learning of the CFA, whose importance has been demonstrated in this paper by the superior quality of color reconstruction of the pipeline. A possible avenue for research could also be the extension of our work to multi-shot light field acquisition, which has been shown to greatly improve the reconstruction quality [START_REF] Nguyen | Multi-mask camera model for compressed acquisition of light fields[END_REF]. Since we learn a distribution for the mask, as opposed to a fixed learned mask, multi-shot acquisition can be achieved by merely modifying the reconstruction network that takes multiple realizations of the mask distribution as input.

Fig. 2 :

 2 Fig.2: End-to-end mask generation, acquisition and reconstruction architecture. The color-coded mask generator is first fed a (ω x , ω y , ν, d) tensor, with d being the dimension of an individual seed. It then outputs a (ω x , ω y , ν, κ) tensor corresponding to the CCM modulating function, where κ = 3 is the number of color channels. The tensor corresponding to the modulating function of the CFA is simply the repetition of the 2D CFA pattern for every view (sub-aperture images). The tensor corresponding to the effective modulating function is then computed by taking the element-wise product of the two tensors. The sensed image is subsequently simulated by the Sensing operator using the effective tensor and the input light field. Finally the effective sensing tensor is concatenated to the sensed image for producing a (ω x , ω y , νκ + 1) tensor to be fed into the fully convolutional reconstruction network.

Fig. 3 :

 3 Fig. 3: Skip connection block architecture. It is composed of three (convolution layer + exponential linear unit activation + batch normalization) blocks. Unlike the standard residual connection that adds the input filters to some intermediate filters, the last block takes the concatenation of the input and output filters of the second block as input.

( a )Fig. 4 :

 a4 Fig. 4: Color coded masks (CCM) and effective tensors obtained by element-wise multiplication of the CCM and of the CFA.

Method

Fig. 5 :

 5 Fig.5: Training flow between the different tasks. An arrow between two tasks indicates that the training of the end-to-end architecture for the target task was done using a reconstruction network pre-trained on the source task.

  equivariance of the end-to-end architecture, it is then straightforward to use the trained network on larger light fields. We also applied on-the-fly data augmentation on each patch by randomly changing the hue in the interval [-0.1, 0.1], as well as changing the saturation by a factor randomly chosen in [0.75, 1.5].

  (i) Monochromatic sensor with random uniform color-coded mask distribution. (ii) Monochromatic sensor with learned color-coded mask distribution. (iii) Bayer CFA with random uniform color-coded mask distribution. (iv) Bayer CFA with learned color-coded mask distribution. (v) Learned CFA with learned color-coded mask distribution. The symbol † indicates the use of 8-dimensional seeds, while no symbol indicates a 1-dimensional seed.

Fig. 6 :

 6 Fig. 6: Representation of the learned distributions over [0, 1] 3 for different configurations. The distribution is visualized by sampling points from the output of the per-pixel generator. Points are displayed at coordinates (r, g, b) with the corresponding color. The distributions of colors for 1dimensional generator seeds clearly lie on a 1-dimensional manifold, while the distributions for 8-dimensional generator seeds can span the whole color cube.

( a )

 a Monochromatic sensor with uniform CCM distribution (PSNR: 32.21 dB) (b) Monochromatic sensor with learned CCM distribution (PSNR: 32.46 dB) (c) Bayer CFA with uniform CCM distribution (PSNR: 32.48 dB) (d) Bayer CFA with learned CCM distribution (PSNR: 32.87 dB) (e) Deep learning method of [29] (PSNR: 30.99 dB) (f) Ground truth

Fig. 7 :

 7 Fig.7: Reconstructed central view of the Orchids light field using the proposed pipeline in comparison with the method of[START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF]. Note how the use of Bayer CFA allows for a more accurate recovery of colors, especially in high-saturation areas (see zoom-in; colors better visualized on a monitor). PSNR is indicated for the whole light field, not only the central view.

( a )

 a Monochromatic sensor with uniform CCM distribution (PSNR: 40.91 dB) (b) Monochromatic sensor with learned CCM distribution (PSNR: 41.68 dB) (c) Bayer CFA with uniform CCM distribution (PSNR: 42.14 dB) (d) Bayer CFA with learned CCM distribution (PSNR: 43.02 dB) (e) Deep learning method of [29] (PSNR: 38.26 dB) (f) Ground truth

Fig. 8 :

 8 Fig. 8: Error maps of the reconstructed central view of the Tulips light field using the proposed pipeline in comparison with the method of [29]. The error maps are scaled by a factor 10 for visibility purpose. PSNR is indicated for the whole light field, not only the central view.

( a )Fig. 9 :

 a9 Fig. 9: Visualization of the error of epipolar plane images (EPI) of the Orchid light field. These are obtained by spanning the horizontal middle line of the light field. That is, displaying the mapping (ω x , ν u , λ) → L(ω x , ω middley

  c] (p(gcI) + n readout ) where [.] denotes rounding, clip [0,c] (.) = max(min(., c), 0), N = 2 b -1, with b the number of bits used to code the digital converted measure, c is the full well capacity of the pixel (in number of electrons), g is a gain factor proportional to the ISO gain of the pixel and the exposure time, p(gcI) is a random variable following a Poisson distribution P(gcI) and n readout ∼ N (0, σ readout ).

Fig. 10 :

 10 Fig. 10: Visualization of epipolar plane images (EPI) of the Mona's Room and Dino synthetic light fields from the HCI dataset [39]. Reconstructed using the Bayer CFA and learned CCM distribution architecture. The error maps as scaled by a factor 10.

  (a) "Mona's room", ours (PSNR: 34.72 dB) (b) "Mona's room", ground truth (c) "Dino, ours" (PSNR: 33.78 dB) (d) "Dino", ground truth

Fig. 11 :

 11 Fig. 11: Reconstructed central view of the Mona's Room and Dino light fields from the HCI dataset [39] using the proposed pipeline with Bayer CFA and learned CCM architecture.

Fig. 12 :

 12 Fig. 12: Learned color filter array patterns for different levels of noise. Note how the light efficiency increases with the level of noise.

TABLE I :

 I Architecture of the reconstruction network.

  Buttercup 31.53 31.71 31.76 32.06 32.37 32.40 32.41

			29.64	29.98	30.09
	Cars	30.85 30.82 31.04 30.86 31.15 31.18 31.22	27.12	29.88	27.40
	Orchids	32.23 32.45 32.46 32.53 32.92 32.99 32.89	28.41	30.99	28.67
	Rock	31.06 31.08 31.17 30.83 31.15 31.20 31.22	27.15	30.11	30.62
	Seahorse	33.45 33.45 33.52 33.21 33.33 33.33 33.61	29.92	32.36	29.64
	Tulips	41.01 41.19 41.75 42.16 42.13 42.89 42.89	40.83	38.26	41.61
	White rose 32.81 32.82 32.88 32.54 32.95 32.92 32.96	28.23	31.84	26.18
	Average	33.28 33.36 33.51 33.46 33.71 33.85 33.89	30.19	31.92	31.78
	TABLE

Dict-TV

[START_REF] Nguyen | Multi-mask camera model for compressed acquisition of light fields[END_REF]

Deep

[START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF]

DeepSAS

[START_REF] Guo | Deep spatial-angular regularization for compressive light field reconstruction over coded apertures[END_REF]

TABLE III :

 III Deep[START_REF] Nabati | Fast and accurate reconstruction of compressed color light field[END_REF] DeepSAS[START_REF] Guo | Deep spatial-angular regularization for compressive light field reconstruction over coded apertures[END_REF] Buttercup 0.9371 0.9362 0.9358 0.9378 0.9413 0.9413 0.9425 SSIM comparison with reference deep learning based methods

			0.9216	0.9073
	Cars	0.9604 0.9593 0.9596 0.9586 0.9607 0.9612 0.9617	0.9539	0.9230
	Orchids	0.9653 0.9644 0.9646 0.9643 0.9659 0.9668 0.9672	0.9571	0.9388
	Rock	0.9294 0.9283 0.9285 0.9227 0.9270 0.9275 0.9279	0.9238	0.9250
	Seahorse	0.9716 0.9691 0.9707 0.9700 0.9697 0.9711 0.9724	0.9661	0.9496
	Tulips	0.9788 0.9775 0.9782 0.9814 0.9821 0.9832 0.9830	0.9719	0.9802
	White rose 0.9583 0.9579 0.9572 0.9538 0.9572 0.9568 0.9577	0.9516	0.9069
	Average	0.9573 0.9561 0.9564 0.9555 0.9577 0.9583 0.9589	0.9494	0.9395

  .57 32.08 32.04 28.54 29.26 29.54 30.67 Cars 30.69 30.16 30.88 31.00 25.73 28.50 26.06 29.95 Orchids 32.18 31.75 32.63 32.73 28.25 30.09 29.70 31.48 Rock 30.51 29.86 31.07 31.04 28.69 28.57 28.72 30.02 Seahorse 32.28 32.23 32.78 33.06 27.44 30.09 28.93 31.63 Tulips 40.66 40.39 40.50 41.77 37.33 37.57 38.77 39.07 White rose 32.00 31.64 32.47 32.66 24.95 29.55 26.74 31.61 Average 32.85 32.37 33.20 33.47 28.70 30.52 29.78 32.06

				level			High level
	Learned CFA	No		Yes		No		Yes
	Entropy regularizer	No	Yes	No	Yes	No	Yes	No	Yes
	Buttercup	31.64 30

TABLE IV :

 IV PSNR (dB) comparison of different models under noisy measurements. The first four columns correspond to g = 1 (low noise level), while the last four correspond to g = 0.33 (high noise level). Configurations not learning the CFA use a fixed Bayer CFA instead. Boldface figures indicate the best performing model for the corresponding level of noise.

TABLE V :

 V Performance comparison between different resolutions of the coded mask. The size of the CCM's pixels is indicated relative to the size of a the sensor's pixels. For all configurations the sensor is equipped with a Bayer CFA, and the CCM's pixels follow a uniform distribution.

	CCM pixel size	1	2	4	8
	PSNR	33.47	33.63	33.62	33.38
	SSIM	0.9555 0.9568 0.9568 0.9555