Wei Zhang

Emmanuel Baudouin

Marie Cordier

Gilles Frison

Bastien Nay

One-Pot Synthesis of Metastable 2,5-Dihydrooxepines Through Retro-Claisen Rearrangements: Methods and Applications

Keywords: 2,5-dihydrooxepines, sigmatropic rearrangement, density functional calculations, total synthesis, herbicide

come   L'archive ouverte pluridisciplinaire

Introduction

Pioneering reports by Vogel, [START_REF] Vogel | [END_REF] Dreiding, 2 and Rhoads 3 on the valence tautomerism between unsubstituted 2,5-dihydrooxepines 1 and syn-2-vinylcyclopropane-1-carboxaldehydes 2 (Scheme 1a, R = H) highlighted an equilibrium in favor of 2. In the 1990s, Reißig, 4 Boeckman, 5 and then Yamaguchi,6 demonstrated that the retro-Claisen (also 1-oxa-Cope) seven-membered product can be strongly favored by the presence of donor-acceptor stabilizing groups on the vinylcyclopropane (VCP) moiety. 7 More recently, the use of stabilizing substituents allowed the development of catalytic enantioselective synthesis of 2,5dihydrooxepines, 8 or their use as synthetic intermediates towards polysubstituted 5-membered rings. 9 However, these works only concern VCP-carboxaldehyde substrates (2, R = H), since for yet untargeted reasons, linear syn-VCP ketones (R ≠ H) are unreactive towards [3,3]-sigmatropic rearrangements. There are two exceptions with ketone substrates, when they are part of an oxalyl system, 10 or of a cyclic 1,3-diketone. 11 Furthermore, even though this retro-Claisen [3,3]-rearrangement often occurs at room temperature or below through a concerted mechanism, it rapidly enters in detrimental competition with the Cloke-Wilson rearrangement towards dihydrofurans (3, Scheme 1a), 12 which has been theoretically investigated by Fabian and Reißig. 13 Such a competition can have dramatic consequences when heating is needed to install the VCP system, leading to mixtures of products (case of 7a-b) or full recovery of the Cloke-Wilson product (case of 9) (Scheme 1b). 11,14 Overall, the reversibility of the retro-Claisen rearrangement and the competing Cloke-Wilson reaction result in metastable 2,5-dihydrooxepines, which have precluded further synthetic developments. As far as we know, this rearrangement has only been once and fortuitously reported in a total synthesis context, that of salvileucalin B by Reisman. 15 Scheme 1. Synthetic strategies developed towards 2,5-dihydrooxepines.

The oxepane ring is present in a wide variety of natural products and we thought this retro-Claisen rearrangement could be further extended to the rapid construction of some of them. [START_REF] Boyd | General reviews on oxepane chemistry[END_REF] In particular, radulanins (4a-c) are phenolic compounds containing a 2,5dihydrooxepine, isolated from liverworts (Fig. 1). 17 Herein we describe the experimental and theoretical studies we performed on VCP-substituted ketones to tame this sensible retro-Claisen rearrangement prior synthetic application (Scheme 1d). Our rearrangement is part of a simple tandem sequence, starting with the annulation of a 1,3-cyclohexadione (10) by conjunctive 1,4-dibromo-2-butene reagents (11) 21 to form VCP intermediate 12, which spontaneously cyclized into fused 2,5dihydrooxepine 13. Remarkably, a similar sequence was reported by Nickl in 1958, performed in methanol at reflux, but only resulted in isomeric 2-vinyl-2,3-dihydrofuran 9. 22 Quantum chemical calculations will be described to understand the outcome of this rearrangement, comparatively with stable linear VCP-1,3diketones, and competing with the thermodynamically favoured [1,3]-sigmatropic rearrangement (Scheme 1a). 12 This theoretical study closely follows the trend of our experiments, and rationalizes the required experimental strict control of the temperature to obtain the seven-membered ring in reasonable yields. As an application, radulanin A (4a) was successfully synthesized by this methodology, after a challenging aromatization of the cyclohexenone ring on 13. Based on an allelopathic hypothesis, we supposed that radulanins would affect plant physiology due to their structural analogy with the bibenzyl natural product lunularic acid (15, Figure 1), 17d,23 and finally demonstrated the weedkiller property of natural product 4a.

Results and Discussion

We first used 1,4-dibromo-2-butene 11a as a suitable conjunctive reagent to generate VCP 12a 21 from 1,3-cyclohexadione 10a at 20 °C (Scheme 2). To our delight, by contrast with Nickl's result, 22 this reaction resulted in 2,5-dihydrooxepine 13a in 52% isolated yield, through the retro-Claisen rearrangement of VCP intermediate 12a (not isolated), when it was conducted in the presence of Cs2CO3 in DMSO (for structure confirmation, see the X-ray crystallography of diol 22, Scheme 4). [START_REF]Crystallographic data of compounds 22 and 24 are available free of charge at the Cambridge Crystallographic Data Centre under number[END_REF] Cloke-Wilson product 16a (7%) and O-alkylated product 17a (4%) were also observed as minor side products, while 1 H NMR spectra of 13a occasionally showed the presence of non-isolatable traces of VCP intermediate 12a. Importantly, the reaction temperature had to be strictly maintained at 20°C at maximum, while purifications were performed over chilled chromatography columns (<10°C), in order to avoid the formation of substantial amounts of undesired dihydrofuran 16a. Extensive optimization of the reaction was performed (bases, solvents, temperature), with yields being limited by the dihydrooxepine stability (see Charts S1-S3 in the SI). Interestingly, the use of tertiary amines as bases strongly favored the formation of dihydrofuran 16a, while ether and alcohol solvents majored the O-alkylation. Among carbonates, Cs2CO3 was the most efficient, minimizing the formation of 16a. Lowering the temperature at 0°C in DMF did not improve yields. In all cases, the temperature window of this transformation was narrow, impacting the reproducibility of the reaction under uncontrolled "room temperature" (>20 °C). Incidentally, heating 13a at 80°C in DCM for 12 hours led to 16a in 63% yield.

Scheme 2. Tandem vinylcyclopropanation/retro-Claisen rearrangements toward 2,5-dihydrooxepines (13) from various 1,3-cyclohexadiones (10) (isolated yields shown). -a The NMR yield of this reaction, before isolation, was estimated to 74%, showing some loss of material during purification over chilled (<10°C) silica gel column; b Ratio of products 13 (2,5-dihydrooxepine) and 16 (2,3-dihydrofuran) based on NMR of the crude product; c Regioisomeric 3-Me/4-Me ratio. The reaction was applied to various cyclic 1,3-dicarbonyl substrates (10a-j) and 1,4-dibromo-2-butenes (11a-f), resulting in substituted 2,5-dihydrooxepine products (Scheme 2). Unsymmetrical dibromobutenes 11c-f led to regioisomeric products in favor of the 3-substituted dihydrooxepine for obvious steric reasons (3:1 to 5:1 ratios), while 11b afforded poor yields of the seven-membered ring product, clearly favouring Cloke-Wilson product 16e. More importantly, an ester (13j), a phenylboronate (13k) or an acetal (13l) were well tolerated during the reaction, while products 13h and 13i prefigured the radulanin skeleton. We finally tested other types of 1,3-dicarbonyl substrates to expand the scope of this transformation. 1,3-Cycloheptanedione 10g and 1,3-indanedione 10h resulted in VCP adducts 12m and 12n in 51 and 71% yields, respectively, which were unable to furnish the corresponding dihydrooxepines, even at higher temperatures. However, 1,3-cyclopentadione 10i afforded dihydrooxepine 13o in 20% yield. Since this product was found stable at temperatures up to 80°C, the low yield was attributed to the many tars formed and to the formation of O-alkylation product (17o). Finally, βketolactam 10j behaved fairly well in this tandem process if we consider that vinylcyclopropanation in that case furnishes an equimolar mixture of syn-and anti-VCP isomers (relatively to the ketone). Dihydrooxepine 13p was obtained in 50% yield according to NMR, as an inseparable mixture with the trans-VCP isomer.

Comparatively, the reactivity of diketone 10a upon cyclopropanation contrasts with that of acetylacetone 18, which afforded VCP 19 with no retro-Claisen rearrangement at 20°C, while it quantitatively yielded dihydrofuran 20 at 100 °C (Scheme 3). Compound 20 could also be obtained in 80% yield by heating 19 in DMSO at 100°C, although needing longer reaction times (5 days). These observations motivated the following theoretical study not only to explain the different reactivity of ketones 10a and 18, but also to understand the metastability of dihydrooxepin 13a. Scheme 3. Reaction outcome of acetylacetone under various conditions. No retro-Claisen product was observed, even at intermediate temperatures.

Compared to the retro-Claisen rearrangement of VCPcarboxaldehydes, 13 that of VCP-ketones has never been theoretically investigated. At our best computational level, CCSD(T)/6-311++G(2d,2p)//M06/6-311G (d,p) including solvent effect (Fig. 2a; see the SI for details), 2,5-dihydrooxepine 13a was kinetically favoured, yet nearly isoenergetic with VCP 12a (∆Gr = -0.7 kcal•mol -1 , corresponding to a 13a:12a ratio of ca. 3:1 according to Boltzmann distribution law if thermal equilibrium is assumed). This reaction operates through lower-energy boat-like transition state TS1 (∆G0 ‡ = +21.7 kcal•mol -1 ), confirming it can proceed easily and reversibly. Alternatively, 2,3-dihydrofuran isomer 16a showed better stability (∆Gr = -8.7 kcal•mol -1 ), with a higher activation energy of transition state TS2 (∆G0 ‡ = +30.9 kcal•mol -1 ), suggesting a more energetic and irreversible reaction. These results indicate a possible competition between the [1,3] (thermodynamic process) and the [3,3]-rearrangements (kinetic process) depending on the reaction temperature, as experimentally observed. A transition structure for the direct isomerization of 2,5-dihydrooxepine 13a to 2,3-dihydrofuran 16a was sought, yet without success. Combined with the energetic profile, this fact strongly supports a thermal isomerization through VCP 12a reversibly formed from 13a. Furthermore, by contrast with Fabian's results on the [1,3]-rearrangement of VCPcarboxaldehydes, 13 we did not observe a biradicaloid character for TS2. It could be explained by the strong donor-acceptor character of 12a. Additional comparisons of activation energies of both reactions in vacuo and in DMSO also support the polarized or zwitterionic character of TS2 that may be stabilized by DMSO (see the SI for more discussion).

To explain the absence of formation of dihydrooxepine 21 from 1,1-diacetyl-2-vinylcyclopropane 19, calculations at the same level were performed (Fig. 2b), showing higher thermodynamic stability of substrate 19 compared to 21 (∆Gr = +4.9 kcal•mol -1 ), thus disfavouring a retro-Claisen rearrangement through TS3. The higher relative Gibbs free energy of 21 compared to 13a may indicate higher conformational constraints in the acyclic substrate, resulting in lower conjugation of the oxy enone system of 21 (see SI for more details). Finally, as previously, Cloke-Wilson product 20 is thermodynamically favoured (∆Gr = -7.0 kcal•mol -1 ), with a higher activation energy of TS4 (∆G0 ‡ = +33.7 kcal•mol -1 ). These last results, which, as for the study of the reactivity of 12a, do not take into account the possible influence of some experimental factors such as the presence of Cs2CO3, nevertheless allow to reproduce and explain the trend observed experimentally (Scheme 3).

The synthetic potentiality of bicyclic 2,5-dihydrooxepines was then challenged by targeting the 2,5-dihydrobenzoxepine system. Although common desaturation methods for cyclohexenone aromatization (e.g. using halogenation reagents, IBX, CAN, MnO2, DDQ or Saegusa-Ito reaction) proved unsuccessful due to the thermal sensibility of 13a, a useful transformation into 2,5dihydrobenzoxepine 24 was serendipitously discovered after an Upjohn dihydroxylation of 13a into diol 22 (Scheme 4) and mesylation. Indeed dimesylate 23, in spite of poor reactivity toward elimination under basic conditions (e.g. Cs2CO3, t-BuOK, DBU), underwent rapid and clean conversion into aromatic compound 24 in 85% yield, in the presence of TBAF in DMSO. [START_REF] Bérubé | [END_REF] The structure of this compound was confirmed by X-ray crystallography [START_REF]Crystallographic data of compounds 22 and 24 are available free of charge at the Cambridge Crystallographic Data Centre under number[END_REF] and we were delighted to find a new entry to the synthesis of natural 2,5-dihydrobenzoxepines. This transformation could result from a TBAF-promoted elimination of the mesylate group in position 4, resulting in a conjugated system (25) and favoring the elimination of the second mesylate, positioning an olefin at position 3,4 in 26. Final tautomerization and aromatization into 27 would release 2,5-dihydrobenzoxepine 24. With these new transformations in hand, the tandem cyclopropanation/retro-Claisen rearrangement strategy was applied to a short synthesis of radulanin A. Starting from 5-(2phenylethyl)-1,3-cyclohexanedione 10f (available from cinnamaldehyde), 26 and 1,4-dibromo-2-methyl-2-butene 11c (available from isoprene), 22,27 the reaction proceeded as expected, providing 2,5-dihydrooxepine 13i (Scheme 2) in 52% isolated yield, as a 3:1 regioisomeric mixture. In an optimization procedure, dihydroxylation was directly performed on the crude reaction mixture of 13i, allowing the formation of diol 28 in 40% yield (2 steps) upon purification from its 4-methyl isomer (Scheme 5). Aromatization of the 6-membered ring was then undertaken from 28, after mesylation of 28 in the presence of Ms2O in pyridine and through TBAF-promoted elimination, [START_REF] Bérubé | [END_REF] finally furnishing radulanin A (4a) in 57% yield over 2 steps. This challenging transformation was particularly efficient if we consider the trisubstituted nature of the double bond to be regioselectively installed. The NMR data of 4a were fully consistent with those of the literature. 18 Scheme 5. Total synthesis of radulanin A (4a).

Having identified structural similarities with lunularic acid (15), an allelopathic natural product interacting with the abscisic acid pathways and inhibiting seed germination and plant growth, 23 radulanin A (4a) was evaluated on Arabidopsis thaliana seedlings. It showed remarkable phytotoxic effects at 50 µg/mL after 24h, with a persistent effect after 120 h (Figure 3), to be compared to the similar effect of glyphosate at 25 µg/mL. Interestingly, the dihydrobenzoxepine core alone (24) was nearly inactive at the concentration of 100 µg/mL. Although the mechanism of action and toxicity profile of 4a have to be evaluated, such a relatively simple compound may constitute a new phytopharmaceutical lead. Indeed, there is an urgent need for alternatives to current herbicides, some of which, being a subject of controversy, are threatened to be banned. 28 24) on Arabidopsis thaliana seedlings, comparatively to glyphosate (line 2) (DMSO 1% taken as a reference). The picture was taken after 96h (see Figure S3 for activities at 24 and 120 h).

Conclusions

We demonstrated that the tandem vinylcyclopropanation/retro-Claisen rearrangement of 1,3-cyclohexadione substrates can be synthetically useful for the preparation of 2,5-dihydrooxepines when properly controlling the reaction temperature. The reaction was theoretically studied, matching experimental results, and explaining the reactivity difference observed between substrates derived from cyclic or linear 1,3-diketones. This study thus complements the chapter on this powerful rearrangement discovered in the 1960s. [START_REF] Vogel | [END_REF][2][3][4][5][6] Importantly, this one-pot retro-Claisen rearrangement was applied to a short total synthesis of a natural product, radulanin A, especially using an ingenious final aromatization procedure. With this compound in hand, we demonstrated for the first time its phytotoxic property, validating our allelopathic hypothesis and paving the way to the development of natural herbicides, which may have better environmental profiles than those currently used synthetic derivatives.

Experimental Section

General procedure for tandem cyclopropanation/retro-Claisen sequence towards 2,5-dihydrooxepines: Cs2CO3 (1.1 equiv.) was added to a solution of cyclohexadione 10 (1 equiv.) in DMSO (0.1 M) and the solution was stirred at 20°C (thermostated) for 20 min. Then a solution of 1,4-dibromo-2-butene 11 (1.1 equiv.) in DMSO was added to the mixture at 20˚C (precisely). The reaction was stirred at 20˚C for another 12 h, before being diluted with Et2O and quenched with chilled water (0˚C). The organic layer was washed 3 times with brine. The aqueous phases were further extracted 3 times with diethyl ether. The combined organic extracts were dried over Na2SO4 and concentrated under reduced pressure, avoiding heating above 20°C. The residue was then purified by flash chromatography on silica gel, using a chilled column (<10°C) to furnish dihydrooxepin 13.

  They were previously prepared by Snieckus18 and Yoshida19 by ring-closing olefin metatheses, while Yamaguchi used a Mitsunobu reaction (Scheme 1c).20 Interestingly, Kawase reported in 1986 the direct formation of a 2,5-dihydrobenzoxepin ring (7a-b) by the alkylation of phenolic substrates (5a-b) with bis-electrophile 1,4dibromoisoprene 6, but this work suffered from very limited yields (Scheme 1b).14

Figure 1 .

 1 Figure 1. Structure of radulanins and their structural analogy with lunularic acid.

Figure 2 .

 2 Figure 2. Energy profile of the rearrangement reactions of substrates 12a (a) and 19 (b). Gibbs free energies relative to starting materials are given in kcal•mol -1 .

Scheme 4 .

 4 Scheme 4. Transformation of 2,5-dihydrooxepine 13a into 2,5dihydrobenzoxepine 24, and possible mechanism for this transformation. ORTEP structures are shown at a probability level of 60%.

Figure 3 .

 3 Figure 3. Herbicide activities of radulanin A (4a) and 2,5-benzoxepine (24) on Arabidopsis thaliana seedlings, comparatively to glyphosate (line 2) (DMSO 1% taken as a reference). The picture was taken after 96h (see FigureS3for activities at 24 and 120 h).

Acknowledgements

We thank Sorbonne Université and the Graduate School of Molecular Chemistry for providing a PhD grant to WZ. We thank Mr. Davy Lin for experimental support in the synthesis of compound 10f. The CNRS, Sorbonne Université and Ecole Polytechnique are acknowledged for financial supports.

Graphical abstract:

radulanin A challenging aromatization


Pierre Allain

Laurent Guillo

Christine Guillemot

4D Anisotropic Diffusion Framework with PDEs for Light Field Regularization and Inverse Problems

come   L'archive ouverte pluridisciplinaire

4D Anisotropic Diffusion Framework with PDEs for Light Field Regularization and Inverse Problems

Pierre Allain, Laurent Guillo, Christine Guillemot, Fellow, IEEE Inria Rennes Bretagne-Atlantique, 263 Avenue Général Leclerc, 35042 Rennes Abstract-In this paper, we consider the problem of vectorvalued regularization of light fields based on PDEs. We propose a regularization method operating in the 4D ray space that does not require prior estimation of disparity maps. The method performs a PDE-based anisotropic diffusion along directions defined by local structures in the 4D ray space. We analyze light field regularization in the 4D ray space using the proposed 4D anisotropic diffusion framework by first considering a light field toy example, i.e., a tesseract. This simple light field example allows an in-depth analysis of how each eigenvector influences the diffusion process. We then illustrate the diffusion effect for several light field processing applications: denoising, angular and spatial interpolation, regularization for enhancing disparity estimation as well as inpainting.

Index Terms-Light Field, regularization, anisotropic diffusion, inverse problems

I. INTRODUCTION

I NVERSE problems appear in a plethora of image pro- cessing and computer vision applications. Many image processing problems can indeed be formalized as the recovery of an image given noisy or incomplete observations. This is for example the case of image denoising, inpainting or super-resolution. A common strategy for dealing with ill-posed inverse problems consists in introducing some prior knowledge on the kind of typical images we try to restore, which helps restricting the class of admissible solutions. The smoothness prior has been one of the most widely used assumptions in computer vision together with regularization schemes using a variational regularization or partial differential equations (PDEs). To preserve edges in images, the regularization is often performed with some anisotropy, i.e. along privileged directions defined by structure tensors giving information on local image structures. These regularization schemes turned out to be very powerful tools for solving a number of inverse problems in 2D imaging.

As the capture of 4D light fields from real scenes grows in popularity, the need for solving similar inverse problems with these 4D data is expected to rise. However, the very large volume of data with angular consistency that needs to be preserved along the processing chain makes the above problems quite challenging. The authors in [START_REF] Goldluecke | The variational structure of disparity and regularization of 4D light fields[END_REF] propose a regularization scheme, leveraging the variational structure in light fields, for denoising and super-resolution. They use 2D structure tensors computed in epipolar plane images (EPI) to estimate disparity between light field views. With the help This work was supported by the EU H2020 Research and Innovation Programme under grant agreement No 694122 (ERC advanced grant CLIM) of the estimated disparity, they project all the views on a target viewpoint and solve the regularization using a global optimization framework with total variation (TV) priors. The final regularizer is written as the contributions of regularizers on vector fields in EPI 2D spaces, with user-defined constants which adjust the amount of smoothing on the different views or EPIs. Errors in disparity maps may however translate into annoying artefacts in particular along edges.

In this paper, we propose instead a regularization method operating directly in the 4D ray space that does not require prior estimation of disparity maps. The method performs a PDE-based diffusion with anisotropy steered by a tensor field based on local structures in the 4D ray space that we extract using a 4D tensor structure. To enhance coherent structures, the smoothing along directions, surfaces, or volumes in the 4D ray space is performed along the eigenvectors directions. Although anisotropic diffusion is well understood for 2D imaging, its interpretation and understanding in the 4D space is far from being straightforward. We first analyse the behaviour of the diffusion process on a light field toy example, i.e. a tesseract (a 4D cube). We particularly illustrate the effect of the diffusion along each of the 4 eigenvectors forming the diffusion tensors.

The proposed ray space regularizer is a tool enabling us to tackle a variety of inverse problems in the ray space. We first illustrate the potential of the proposed 4D anisotropic diffusion for light field denoising. This led us to introduce a novel denoising method that we called Anisotropic Diffusion Denoising 4D (ADD4D). Denoising results show that the proposed method compares well to the best state of the art method [START_REF] Alain | Light field denoising by sparse 5D transform domain collaborative filtering[END_REF], even outperforming it at high noise levels, while being considerably faster. Preliminary light field denoising results with ADD4D have been presented in [START_REF] Allain | Light field denoising using 4D anisotropic diffusion[END_REF]. However, in this paper, in addition to the in-depth analysis on the tesseract toy example, we further revise the tuning of the model parameters aiming at a more intuitive use. We then demonstrate the ability of the presented approach to perform both angular and spatial light field interpolation. The use of the proposed anisotropic diffusion framework as a 4D regularization of the light field, by removing inter-view inconsistencies and noise, is also shown to improve disparity estimation in epipolar plane images. We finally illustrate the effect of the proposed framework in the context of inpainting.

II. RELATED WORK

In this section, we give a brief review of methods related to our work, starting with the main regularization and diffusion methods widely used in 2D image processing. We then review the few papers of the litterature dealing with regularization for inverse problems in light field processing.

2D regularization. The goal here is not to give a complete review of the numerous methods for 2D regularization, but instead to briefly remind the principles of those related to this work. This section therefore focuses on PDE based regularization. The framework of diffusion based on PDEs has been widely used for image regularization, going from classical isotropic diffusion, to non linear diffusion [START_REF] Perona | Scale space and edge detection using anisotropic diffusion[END_REF] and anisotropic smoothing [START_REF] Weickert | Theoretical foundations of anisotropic diffusion in image processing[END_REF]. Although sometimes referred to as anisotropic diffusion, non linear diffusion locally adapts the response of the smoothing filter so that the diffusion process is lessened near edges and is stronger in homogenous areas. One step further has been to introduce locally adapted filters for performing truly anisotropic diffusion close to image structures such as edges [START_REF] Weickert | Theoretical foundations of anisotropic diffusion in image processing[END_REF]. Anisotropic regularization (or diffusion) refers to a smoothing in privileged spatial directions with different weights. The author in [START_REF] Weickert | Theoretical foundations of anisotropic diffusion in image processing[END_REF] uses a field of diffusion tensors to find privileged directions of diffusion. The diffusion tensor T (symmetric and positive-definite matrix) is derived in each point p = (x, y) of the image from the spectral elements of the structure tensor as:

T = λ + θ + θ T + + λ -θ -θ T -, (1)
where θ -and θ + are the eigenvectors, and λ -and λ + the corresponding eigenvalues, of the 2 × 2 symmetric and semipositive-definite matrix called structure tensor (also called the Di Zenzo tensor [START_REF] Zenzo | A note on the gradient of a multi-image[END_REF]) defined as T = ∇I∇I T . Note also that the tensor can be regularized with Gaussian kernels, in order to have increased coherency in the retrieved geometry.

A number of anisotropic diffusion methods have then been proposed, whose main differences lie first on the definition of the diffusion tensor defining the smoothing geometry, and on the PDE used to perform the smoothing. To give an example, coherence-enhancing anisotropic diffusion is proposed in [START_REF] Weickert | Coherence-enhancing diffusion filtering[END_REF], which increases the diffusivity along the direction given by θ -when the coherence measured by (λ + -λ -) 2 increases. The PDE performing the smoothing can also take several forms, using a divergence-based formulation [START_REF] Weickert | Theoretical foundations of anisotropic diffusion in image processing[END_REF], a trace-based formulation [START_REF] Tschumperlé | Vector-valued image regularization with PDEs: A common framework for different applications[END_REF]. In [START_REF] Sapiro | Anisotropic diffusion of multivalued images with applications to color filtering[END_REF], [START_REF] Tschumperlé | Constrained and unconstrained PDE's for vector image restoration[END_REF], the smoothing is modelled as the juxtaposition of two oriented 1D Gaussian smoothing processes along orthonormal directions with different weights. The link between these three formulations is established in [START_REF] Tschumperlé | Vector-valued image regularization with PDEs: A common framework for different applications[END_REF]. The author in [START_REF] Tschumperlé | Fast Anisotropic Smoothing of Multi-Valued Images using Curvature-Preserving PDE's[END_REF] further introduces a curvature-preserving PDE that aims at respecting specific curvature constraints, by adding to the trace-based PDE a term taking curvatures of integral curves of the diffusion tensor field into account. A graph Laplacian regularizer is proposed in [START_REF] Pang | Graph laplacian regularization for image denoising: Analysis in the continuous domain[END_REF] for 2D image denoising, assuming that the original image (patch) is smooth with respect to a defined graph in which vertices correspond to pixels on a 2D grid connected via edges. However, with high dimensional data as light fields, the dimension of the graph can rapidly become an issue.

Light field inverse problems and regularization. Although, at an early stage, inverse problems in light field imaging have naturally arised with the emergence of real light field cameras. Research effort has in particular been dedicated to denoising and super-resolution, in particular for overcoming technological limitations of plenoptic cameras. The authors in [START_REF] Mitra | Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior[END_REF] use a Gaussian mixture model (GMM) prior for light field patches, and then exploit this prior within a Bayesian inference framework for light field denoising. In [START_REF] Sepas-Moghaddam | Light field denoising: exploiting the redundancy of an epipolar sequence representation[END_REF], the light field is denoised by stacking EPIs in a 3D volume using the VBM4D video denoising method [START_REF] Boracchi | Video denoising, deblocking, and enhancement through separable 4-D non local spatio-temporal transforms[END_REF], while the authors in [START_REF] Dansereau | Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter[END_REF] apply a hyper-fan shaped filter in a 4D discrete Fourier transform domain. The BM3D denoising algorithm [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF] is extended in [START_REF] Alain | Light field denoising by sparse 5D transform domain collaborative filtering[END_REF] to light fields. The only approach of regularization proposed so far for light field denoising is the one of [START_REF] Goldluecke | The variational structure of disparity and regularization of 4D light fields[END_REF] using a global variational framework integrating contributions of regularizers on vector fields computed in EPI 2D spaces. Regularization based on anisotropic total variation is also used in [START_REF] Dansereau | Richardson-lucy deblurring for moving light field cameras[END_REF] for light field deblurring. A regularization term is used to enforce the equiparallax constraint, hence to enforce valid light field geometry.

Light field spatial and angular super-resolution has also received a lot of attention [START_REF] Levin | Understanding camera tradeoffs through a bayesian analysis of light field projections[END_REF], [START_REF] Bishop | The light field camera: Extended depth of field, aliasing, and superresolution[END_REF], [START_REF] Mitra | Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior[END_REF], [START_REF] Farrugia | Super resolution of light field images using linear subspace projection of patch-volumes[END_REF] using either Gaussian priors in Bayesian inference frameworks or learning methods as in [START_REF] Farrugia | Super resolution of light field images using linear subspace projection of patch-volumes[END_REF] and [START_REF] Yoon | Learning a deep convolutional network for light-field image super-resolution[END_REF]. In a similar manner as in [START_REF] Goldluecke | The variational structure of disparity and regularization of 4D light fields[END_REF], the authors in [START_REF] Wanner | Variational light field analysis for disparity estimation and super-resolution[END_REF] first estimate disparity maps for each view and project all the views into the target one within a global variational optimization framework. Errors in disparity maps may however translate into annoying artefacts in the superresolved light field.

Another important light field processing problem that can benefit from the proposed regularization framework is light field view interpolation. Existing approaches for view interpolation can be broadly divided into two main categories. A first category of methods relying on depth image based rendering techniques makes explicit use of geometric or depth information to warp source views onto target viewpoints. The various existing methods differ in the way scene geometry is estimated, e.g. using optical flows between the input views as in [START_REF] Georgiev | Spatio-angular resolution tradeoff in integral photography[END_REF] or using structure tensors computed in epipolar plane images as in [START_REF] Wanner | Variational light field analysis for disparity estimation and super-resolution[END_REF] where the authors cast the view synthesis problem in a variational framework. However, the quality of the synthesized views obtained by the above approaches in general very much depend on the quality of the depth or disparity information which is difficult to estimate in textureless and occluded regions. The quality of depth maps can also be enhanced thanks to anisotropic regularization as in [START_REF] Heber | Variational shape from light field[END_REF] where the authors proposed a method based on active wave-font sampling and the variational principle to regularize the depth map of the center view.

A second category of methods based on deep learning techniques have also recently appeared either explicitly or implicitly exploiting depth information. An architecture is proposed in [START_REF] Yoon | Learning a deep convolutional network for light-field image super-resolution[END_REF] based on first deep convolutional neural network (CNN) to perform spatial super-resolution followed by a second CNN to synthesize intermediate views. The authors in [START_REF] Flynn | Deep stereo: Learning to predict new views from the world's imagery[END_REF] use a convolutional neural network to estimate novel views from a set of input views with wide baseline. An architecture based on a cascade of two convolutional neural networks is proposed in [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] to estimate both the disparity maps and color of synthesized views from the four corner views of a dense light field with small disparity.

In this paper, we propose instead a regularization scheme directly operating in the 4D ray space that can be applied to solve inverse problems in light field processing. The method does not require prior depth estimation. We first demonstrate the interest of the proposed approach for three applications: denoising, angular super-resolution or view interpolation, and spatial interpolation. We further illustrate the potential for light field inpainting as well as for regularizing light fields in order to improve the quality of estimated depth maps.

III. 4D ANISOTROPIC DIFFUSION

Let L(x, y, u, v) denote the 4D representation of a light field, describing the radiance of a light ray parameterized by its intersection with two parallel planes [START_REF] Gortler | The lumigraph[END_REF], and where (u, v) denote the angular (view) coordinates and (x, y) the spatial (pixel) coordinates.

A. 4D structure tensor

To perform light field regularization along pre-defined directions which would preserve edges, one has to first characterize local structures in the image, or here the light field. While light field regularization methods are usually applied per Epipolar Plane Image (EPI) of the light field with directions given by 2D structure tensors often smoothed with a 2D Gaussian kernel, we consider instead a direct computation of 4D structure tensors in the 4D ray space, which is defined as the symmetric and semipositive definite 4 × 4 matrix [START_REF] Weickert | Coherence-enhancing diffusion of colour images[END_REF]:

T = 1 C C c=1 ∇L c ⊗ ∇L c , ( 2
)
where C is the number of color channels. The notation ∇L c denotes the gradient or partial derivatives in the 4 dimensions of the color channel L c . A smoothed version T ω = T * G ω (x, y, u, v) of the structure tensor is in practice computed to retrieve a more coherent geometry, where G ω (x, y, u, v) is a Gaussian kernel smoothing along the four light field dimensions, with ω = [ω x , ω y , ω u , ω v ]. Its spectral elements, i.e. the eigenvectors ν i , i = 1 . . . 4 and the eigenvalues λ i , i = 1 . . . 4 of the 4D structure tensor T respectively give the orientations of the local structures in the 4D ray space and the vectorvalued variations along these structures. We therefore have:

T ω = 4 i=1 λ i ν i ν T i ( 3
)
The eigenvector ν 1 with the largest eigenvalue λ 1 gives the dominant orientation of the local structures in the 4D ray space. The ratios between the eigenvalues gives information on the type of structure present in the local neighborhood.

B. Trace-based PDE Diffusion

To be adapted to the local structures in the 4D ray space of the light field, the diffusion process must be driven by a specific tensor, called diffusion tensor chosen so that it has the same set of eigenvectors ν i , i = 1 . . . 4 as the structure tensor T. A diffusion tensor is therefore constructed as:

D = 4 i=1 β i ν i ν T i , (4)
where the coefficients β i set the smoothing strength along the direction given by ν i , and are chosen according to the goal of the diffusion.

The diffusion is thus formulated by the trace-based PDE:

∂L c ∂t = trace(DH c ) , (5)
where H c is the 4D Hessian matrix of the c channel of L, defined as:

H c =       ∂ 2 Lc ∂x 2 ∂ 2 Lc ∂x∂y ∂ 2 Lc ∂x∂u ∂ 2 Lc ∂x∂v ∂ 2 Lc ∂y∂x ∂ 2 Lc ∂y 2 ∂ 2 Lc ∂y∂u ∂ 2 Lc ∂y∂v ∂ 2 Lc ∂u∂x ∂ 2 Lc ∂u∂y ∂ 2 Lc ∂u 2 ∂ 2 Lc ∂u∂v ∂ 2 Lc ∂v∂x ∂ 2 Lc ∂v∂y ∂ 2 Lc ∂v∂u ∂ 2 Lc ∂v 2       , ( 6
)
and D is a diffusion tensor to be defined according to the regularization goals. This formulation extends the 2D anisotropic diffusion [START_REF] Perona | Scale space and edge detection using anisotropic diffusion[END_REF] to 4D light fields. Each eigenvector ν i in ( 4) is responsible for setting a direction for the diffusion process. Since these vectors are eigenvectors of the 4D light field structure tensor, these orientations depend on the local structure properties. In order to best preserve the structures in the 4D ray space, when running the diffusion, it is very important to understand how these orientations capture anisotropy in the 4D ray space, which is much less straightforward than in 2D. To understand how each eigenvector influences the diffusion process, we first consider a 4D cube, i.e., a tesseract, as illustrated in Fig. 1, where we show its 2D projection. Because of its symmetry and of the fact that such shape contains faces, edges, and corners, it provides a good example to illustrate how the eigenvector oriented diffusion behaves on 4D structures. Fig. 2 shows the effect of the diffusion process when the diffusion tensor is based on a single eigenvector. In Fig. 2a, the diffusion tensor is based on the fourth eigenvector ν 4 , hence follows a direction corresponding to the highest degree of singularity in the 4D space. For the tesseract, these degrees of singularity are located at the hyper-corners, i.e., at the corners of corners in the 2D projection. The diffusion process therefore smoothes these hyper-corners, i.e., rounds the squares at angular corners of the tesseract. In Fig. 2d, the diffusion tensor is based on the third eigenvector ν 3 that defines a direction corresponding either to the corners of the edges, or to the edges of the corners. In Fig. 2g, the second eigenvector ν 2 has a direction corresponding to any 2D corner in the 4D space or to edges of edges. We can see that the diffusion along ν 2 leads to a more global smoothing of the original structure as compared to the diffusion along ν 3 . The same applies for ν 1 regarding ν 2 . One can note that diffusing along ν 2 , ν 3 and ν 4 using

C. Influence of eigenvectors on the diffusion tensor

D = ν 2 ν T 2 + ν 3 ν T 3 + ν 4 ν T 4
tends to create a 4D hypersphere, i.e., a glome.

D. Setting the diffusion tensor parameters

The diffusivity coefficients β i introduced in (4) control the strength of the diffusion in the eigenvectors orientations, hence the amount of anisotropy. Therefore, their definition is mostly responsible of the diffusion steering, regarding both its intensity and its direction. Indeed, the inequality of the β i coefficients produces the anisotropy that is needed to strengthen the structures in most of the inverse problems applications we consider here. In some cases, as in the following section, it is relevant to express these coefficients as functions f i depending on the quantity of structure ψ. This means the diffusion steering, in addition of being expressed on a basis defined by structures (the eigenvectors), is also driven by the structures intensity. We define this intensity such as: ψ =

IV. DENOISING

In order to preserve the coherence of the light field structures, while removing incoherent information such as noise, the function f i must verify certain properties. First, the diffusion should be isotropic in regions where there is no strong apparent direction, i.e.: lim

D ψ→0 = aI ⇒ f 1 (0) = f 2 (0) = f 3 (0) = f 4 (0). (7)
Conversely, in regions where structures are clearly appearing, no diffusion should be performed in order to preserve geometry, i.e.:

lim

D ψ→inf = 0 ⇒ f 1 (ψ) = f 2 (ψ) = f 3 (ψ) = f 4 (ψ) = 0. (8)
Also, the diffusion must be performed along directions describing the local structures. The singularity of structures is described by the eigenvectors having the lowest eigenvalues.

The more structure there is, the more the lowest eigenvalued eigenvectors must drive the diffusion. When the structure is clearly shaped, the diffusion should only correct structure singularities with the highest degree. Applying such reasoning to the eigenvectors, we obtain the condition:

(a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l)
f 1 (ψ) ≤ f 2 (ψ) ≤ f 3 (ψ) ≤ f 4 (ψ). (9)
We therefore propose the function f i meeting the conditions of equations ( 7), [START_REF] Tschumperlé | Vector-valued image regularization with PDEs: A common framework for different applications[END_REF] and, (9):

f i (ψ) = e -ψ ρ i γ , with ρ 1 ≤ ρ 2 ≤ ρ 3 ≤ ρ 4 ( 10
)
where ρ i is a parameter related to a quantity of local variation ψ at which diffusion along ν i will importantly change the intensity, and γ is a parameter defining the exponential behaviour of the function. This parametrization allows tuning the diffusion process on the basis of structure properties. The parameters γ and ρ i need however to be adjusted according to the application. Below, we illustrate the effect of these parameters in the context of light field denoising.

A. Model tuning

The best set of diffusion model parameters ρ i may depend on the light field and on the desired smoothing intensity. Finding the best values for this set of parameters ρ i may therefore be a tedious task. Below, we propose a simplified parametrization to facilitate the model tuning.

Instead of parameterizing independently each parameter ρ i , we propose the following expression:

ρ i = ρ r α i-4 , (11)
which allows a simplification of the model, being left with only two parameters. The first one, ρ r , is related to a local variation intensity. The second one, α, is related to the amount of anisotropy for the diffusion. Fig. 3 plots different functions f i (ψ) parameterized by ρ i , taking ρ r = 0.4 and α = 2. Below, we study the influence of these parameters on the denoising process. For each experiment, we set γ = 2, and the same time step of 0.1 to better show the influence of the parameter ρ r on the denoising process. 1) Model range and noise intensity: Fig. 4 shows the evolution of the PSNR along the iterations of the model integration, when varying the parameter ρ r , with different levels of noise. Note that the PSNR is the Peak Signal to Noise Ratio computed as 10log 10 1 M SE , where the MSE is the mean square error between all the original (with no noise) and all the denoised views over the three RGB color components. The light field used in this experiment is Bike of the EPFL dataset [START_REF] Rerabek | New light field image dataset[END_REF]. The integration time t is related to the number of iterations n by the time step dt = 0.1 such as: t = n dt. The integration carried out to solve (5) is performed using the Runge-Kutta 4 scheme. Fig. 4a shows that when no noise is applied to the light field, higher values of ρ r yield higher deterioration (smoothing) of the light field. Fig. 4b and4c, show that, for low noise (σ = 10 255 , σ = 40 255 ), a low value of ρ r (respectively 0.08 and 0.24) prevents the model to damage the light field structures. On the other hand, a low value of ρ r is not suitable for high noise levels as shown in Fig. 4d, where the model is unable to diffuse along the structures, hence many iterations are needed to remove the noise. For high values of ρ r , we can observe a more efficient behaviour regarding noise reduction. But in this case, the model can damage the structures (more than with low values of ρ r ), as we can see when the number of iterations increases. Fig. 5 shows, in a symmetric manner compared with Fig. 4, the evolution of the PSNR along the iterations, when varying the levels of noise, with different values of the parameter ρ r . Fig. 5b-d show that, for different values of the parameter ρ r , when increasing the number of iterations, the model converges to the same PSNR curve, at both low and high noise levels. This shows that, even for very different initial noise intensities, a same denoising model (with same parameters) will tend to reshape the light field in almost the same fashion, provided ρ r or the number of iterations is high enough. In other words, the noise is cleaned more or less quickly while the model is on its way to its inherent convergence state. In the following denoising experiments, we chose to set the parameter ρ r to 0.4, which appeared to be a good trade-off between preserving data structures and efficiency in terms of noise removal.

2) Anisotropy: We now study how the coefficient α controlling the amount of anisotropy influences the denoising process. Fig. 6d-f shows denoising results for α set to 1 (isotropic), 2, and 5, after 20 iterations, using the StonePillarsOutside light field. Even if this number of iterations may not yield the best reachable PSNR for each of the α values, it allows analyzing the model behaviour when varying the parameter α. While the isotropic denoising yields too blurry images (see Fig. 6d), a high value of α tends to sharpen too much the details, even to structure the noise instead of removing it (see Fig. 6f).

As the anisotropy increases (Fig. 6i), the best PSNR is obtained for a higher number of iterations, as we can see in Fig. 6c, highlighted by the red line. Increasing the time step can lower the number of iterations, thus the computation time, but can affect the stability of the model integration. Indeed, the non-linearity of the model is likely to increase with its anisotropy. In the end, a trade-off has to be found depending on how fine one wants to recover texture details, and how fast the denoising reaches the best result. We have observed experimentally that setting ρ r to 0.4 and α to 2 was yielding a good compromise between efficiency, sharpness and number of iterations.

B. Denoising results with the tesseract toy example

In order to illustrate the behaviour of the denoising model, we propose to apply it on the tesseract toy example presented above. The goal is to observe how the proposed denoising model behaves on a light field containing both very high and very low variations in the 4D space. Fig. 7 shows denoising results, after 20 iterations, with both the 2D regularization method described in [START_REF] Tschumperlé | Vector-valued image regularization with PDEs: A common framework for different applications[END_REF] applied independently on each view, and the proposed 4D anisotropic diffusion. One can observe that the proposed diffusion performs well in terms of denoising compared with the 2D approach of [START_REF] Tschumperlé | Vector-valued image regularization with PDEs: A common framework for different applications[END_REF]. The 4D angularities of the tesseract are being smoothed in every dimension.

C. Denoising results with real light fields

In order to assess the denoising performance of the 4D anisotropic diffusion with the proposed model on real light fields, in comparison with state of the art methods, e.g., with [START_REF] Alain | Light field denoising by sparse 5D transform domain collaborative filtering[END_REF], we considered the same 12 light fields of the EPFL dataset [START_REF] Rerabek | New light field image dataset[END_REF]. The light fields have been extracted from the raw captures using Dansereau's Matlab light field toolbox [START_REF] Dansereau | Decoding, calibration and rectification for lenselet-based plenoptic cameras[END_REF] in the same conditions as in [START_REF] Alain | Light field denoising by sparse 5D transform domain collaborative filtering[END_REF].

Because of the strong irregularities in the angular dimension resulting from the vignetting effect, we chose to denoise only the subapertures that are not affected by the later and therefore limit the size of light fields to 11 × 11 central views. Indeed, because of its regularization properties, we will show later that our method also corrects artefacts at angular corners (i.e., the vignetting effect), hence the measured PSNR, when [START_REF] Alain | Light field denoising by sparse 5D transform domain collaborative filtering[END_REF] for the PSNR values in dB of the reference methods). A visual comparison can be found in Fig. 10.

considering all the views, would not only reflect the denoising performance.

In the experiments, we chose to use the same model parameterized by ρ r = 0.4, α = γ = 2, and the tensor smoothing is set to ω = 1.1 in the spatial and angular directions. This parameterization gives a denoising model of relatively low anisotropy, but allows the removal of noise in a reasonable number of iterations. One iteration takes 7 minutes to compute on a i7-6600U without any parallelization, and best PSNR values are usually obtained from 4 iterations to 20 depending on the light field and the model. Note that the computation time for each iteration is not scene dependent and is the same for light fields of same dimension. For comparison, LFBM5D performs approximately in 7 hours on an octo core processor (but on 15 × 15 light fields). On a 10 iterations basis, our method gives an estimated 26 times faster result. Detailed results can be found online1 to show angular consistency. In Fig. 8, we observe that our model is able to remove low noise while keeping fine original structures. Our model is also able to deal with high levels of noise as shown in Fig. 9 and10. In Fig. 10, we can see that the results qualitatively match the best existing denoising method. In Table I we can see that the denoising model we used outperforms other methods for a high level of noise, while comparing well for low levels of noise.

Fig. 11 shows an example of denoising result with a light field captured under low illumination conditions. In this experiment, the denoising model is parameterized as a small variant of the expression proposed in [START_REF] Tschumperlé | Fast Anisotropic Smoothing of Multi-Valued Images using Curvature-Preserving PDE's[END_REF], i.e. by setting the model parameters as ρ i = ρ i+1 α i-4 . Our method is able to remove noise due to poor capture conditions. We also provide in Tab. II additional restoration results to show the efficiency of the proposed approach in other conditions than additive Gaussian noise. The restoration of light field quantized using the Floyd-Steinberg dithering method yields particularly good results, as presented in Fig. 12. Indeed, the dithering preserves image structures and color gradients, thus allowing more efficient reconstruction using the tensor diffusion approach.

D. Influence of the number of views

It is interesting to study to which extent the light field structure is retrieved when the number of light field views increases. In the following experiments, we use the original parameterization model proposed in [START_REF] Tschumperlé | Fast Anisotropic Smoothing of Multi-Valued Images using Curvature-Preserving PDE's[END_REF], with ρ r = 0.4 and α = 2. Fig. 13 shows how the error evolves along the iterations [START_REF] Rerabek | New light field image dataset[END_REF]. We also give the rounded number of iterations at which the maximum PSNR is reached. The denoising model is parameterized by ρr = 0.4 and α = 2.

for different numbers of views. In Fig. 13a, we can see that the maximum overall PSNR increases with the number of views (up to 9x9 views). Starting from 11x11 views, the error increases because of the vignetting effect of the light-field capturing device. Indeed, as mentioned in section IV-C, light fields captured with Lytro Illum cameras are impacted by lens deformations producing angular inconsistencies. For a high number of views (higher than 11 × 11 in the case of Lytro Illum captures), these irregularities are being corrected by the denoising process, and tend to increase the difference with the original light field.

For 13 × 13 views, the vignetting effect starts to clearly appear (also see Fig. 14e,f and Fig. 15). The strong intensity variations between vignetted (almost entirely black) and inner views produce angular structures which are difficult to properly smooth. Increasing the anisotropic coefficient α could enhance these contours, but at the expense of sharpening the noise in other regions. Such problem can also be viewed in Fig. 7f (middle and bottom), where the denoising process yields smoothing of structures at angular edges and corners. This side effect is further discussed in this section.

If we now consider only the central views of the light field, we can see in Fig. 13b that the maximum PSNR value is almost the same from 5x5 to 15x15 views. However, as the number of views increases, this maximum is reached with fewer iterations because of the larger distance to angular edges, and consequently the lower angular boundary effect. On the other hand, when the number of views increases, more angular inconsistencies are corrected, and eventually propagated to the center view, causing the PSNR to decrease.

To better see that effect, Fig. 14 shows the PSNR obtained after denoising for each view of the light field. We can see that, at the corner views, the PSNR decreases when the number of views increases. One can also observe in Fig. 14d that the values of PSNR are not symmetric around the angular axes, values at the angular coordinates [0,0] being lower than others. This can be explained by the vignetting effect and inconsistencies between micro-lenses.

As mentioned above, for 13x13 views (see Fig. 14e), the vignetting effect is clearly visible. The error on the corner views of the light field becomes more important than in inner views. Indeed, as we can observe in Fig. 15, the denoising process yields structure smoothness likely to print scene structures on vignetted views, thus increasing the error at this angular location. This effect can be lowered by increasing the anisotropic coefficient α, so the diffusion of the denoising model can better fit the contrasted structures, such as the ones the vignetting produces in the angular space. Eventually, for 15x15 views (see Fig. 14e), we observe a high PSNR rebound for the angular corner views which are almost fully black with no neighboring contrasted structures.

E. Sparsity

The proposed method can be challenged when the angular consistency between views decreases. Indeed, sparse light fields, with large baselines, do not always provide continuity of structures in the angular dimension. Fig. 16a shows a denoising example on a light field from the Stanford dataset [START_REF]The Stanford light field archive[END_REF]. The foreground of the scene (highlighted by the bottom white line) presents sharp structures in the angular space because of the high disparity at this location. Indeed, in the corresponding epipolar plane image, we can see that the structures are strongly aliased, with little inter-view continuity.

During the denoising process (see Fig. 16c), the sharp structures are difficult to preserve. Any smoothing of structures in the angular space, also leads to significant blurriness of the views. This effect is related to the toy experiment presented in Fig. 7, and more specifically to the f-bottom view. Indeed, spatial structures with high inter-view disparity corresponds to hyper corners of the tesseract. These 4D singularities are difficult to denoise while keeping intact their complex structure.

F. Non-Lambertian surfaces

Light reflections during light field captures can produce complex 4D geometries. We propose to apply our denoising method on a light field containing non-Lambertian surfaces. In Fig. 18, we can see that the reflected structures on the water are correctly denoised.

V. MISCELLANEOUS APPLICATIONS

The proposed 4D tensor-based anisotropic diffusion can be a useful tool for various applications, beyond denoising. Here, we consider view (or angular) interpolation, as well as spatial interpolation. The goal is to highlight the fact that our approach deals with light fields as 4D volumes, and can perform well in different application contexts.

A. View interpolation

We use instead the proposed anisotropic diffusion approach to estimate intermediate views by strengthening structures across views. While for the denoising application, isotropic diffusion is required to remove noise in homogeneous areas, the diffusion process for view interpolation is required to preserve existing structures of original views, while creating new intermediate views. By setting the values of the parameters β i of (4) as β 1 = 0, β 2 = 0.25, β 3 = 0.5, and β 4 = 1, we obtain a regularization model suitable for the angular interpolation problem. The increasing values of beta are explained by the goal of first propagating the strongest structures (hence with a stronger diffusivity coefficient) and then the homogeneous regions. The orientations of the strongest structures are given by the 4 th eigenvector and correspond in the view interpolation problem to strong luminance variations in both angular dimensions due to the missing views to be interpolated. Before computing the tensor, and only for the purpose of the tensor computation, we apply a convolution with a Gaussian kernel on the light field as L w = L * G w , with w = [w x , w y , w u , w v ]. In the following experiments, we set w x = w y = 0 and w u = w v = 0.8. For the view interpolation problem, the structure tensor is used to find coherent orientations in the III: PSNR using integer magnification setup, and for factor 2 and 3. High increment from nearest neighbor to bicubic for magnification factor 2 is due to the misalignment of the nearest neighbor method for even factors. angular u and v dimensions, hence is smoothed only in these dimensions.

Fig. 19 shows the results obtained with two light fields that have been sub-sampled with a factor of 2. The figures show the reconstructed views along the iterations, thus illustrating the diffusion effect. Most contrasted structures appear first, and homogeneous areas are progressively filled. A lower PSNR is obtained for the Desktop light field due to angular inconsistencies. The noise of the subaperture images is also corrected, yielding a lower PSNR, when computed with respect to the noisy input images. The computation can be speeded up by increasing β 2 to tend towards orthogonal isotropy, such that β 1 = 0 and β 2 = β 3 = β 4 = 1. In this case, homogeneous regions are reconstructed faster, but at the cost of smoothed structures.

Let us point out the fact that we chose to initialize the views to synthesize with black pixels. This choice is interesting to observe the diffusion model behaviour. However, it penalizes the computational time. Indeed, it takes a higher number of iterations to diffuse the color information, compared to the case where this initialization would be based on a combination of neighboring views. The combination of neighboring views would give initial views with blurry areas. Even if the model is able to structure them, this initialization may introduce slightly biased structures yielding local minima when running the diffusion. In our experiments, the best results have been I. obtained using a black initialization.

B. Spatial interpolation

Tensor diffusion can also find applications in the field of spatial interpolation. 2D anisotropic diffusion has already been considered for image magnification. It is nevertheless interesting to show how 4D anisotropic diffusion can extend structures in the 4D ray space for light field spatial interpolation. For this purpose, we spatially downsize all light field views and then try to restore the resolution back to the original one, as explained in Fig. 20. The spatial interpolation process is initialized using nearest neighbor interpolation. Indeed, linear and cubic interpolation can introduce a bias with respect to real structures, due to their averaging effect. However, for even magnification factors, the nearest neighbor method can introduce a one pixel shift in the initialization. In the following experiments, the model parameters are set as β 1 = 0, β 2 = β 3 = β 4 = 1 and w u = w v = 0. For a magnification factor 2, we set w x = w y = 0, and for a factor 3, because it is necessary to help detecting structures orientation, we set w x = w y = 0.8.

Tables III gives the PSNR obtained with the diffusion based light field spatial interpolation method with two magnification factors (2 and 3), in comparison with a nearest or a bicubic interpolation. The table gives the PSNR values both for the luminance only and for the three RGB color components. The light fields are from the EPFL, INRIA [START_REF]Inria Lytro capture dataset[END_REF], and HCI [START_REF]HCI light field dataset[END_REF] datasets. It is to be noted that our method yields lower PSNR than bicubic interpolation for a magnification factor of 2 for Cone Head, Monas Room and Watch light fields. The reason comes from the repetition of small structures in the scene which are sensitive to initial shifting of nearest neighbor initialization. On average, the anisotropic diffusion yields a 0.84 dB increment as compared to bicubic for factor 2 magnification, and a 1.87 dB increment for factor 3. These numbers can be related to those obtained by state of the art methods applied to spatial super-resolution, despite the different experimental setups. That is, original low resolution pixels are preserved in our case, whereas spatial super-resolution methods produce blurring of low resolution images due to the downsampling method (floating point magnification factors). In [START_REF] Farrugia | A simple framework to leverage stateof-the-art single-image super-resolution methods to restore light fields[END_REF], their method and the one proposed by [START_REF] Rossi | Geometry-consistent light field superresolution via graph-based regularization[END_REF] produce a 1.43 dB increase as compared to bicubic reconstruction for a magnification factor 3. We can therefore see that spatial interpolation using anisotropic diffusion yields a coherent increase as compared to spatial super-resolution methods. Fig. 21 shows some visual results for the light field Papillon, with a magnification factor of 3. The second row shows that the proposed method enables a better recovery of fine details (see e.g. legs of the butterfly). More visual comparisons can be found on the project web site2 .

C. Light field regularization and disparity estimate

Disparity estimation methods based on local structure orientations in epipolar plane images [START_REF] Goldluecke | The variational structure of disparity and regularization of 4D light fields[END_REF] benefit from the ability to easily compute disparity everywhere on the light field. However, they suffer from inter-view inconsistencies and noise (even low) in homogeneous area. These drawbacks can yield noisy disparity estimate and poor confidence information φ, that we define empirically as a function of the quantity of spatio-angular structure in the 2D epipolar plane images: φ(λ + , λ -) = (λ + + λ -) 0.2 . One can refer to (1) for the two dimensional context. We propose to apply our method as a preprocessing of such disparity estimation methods. The goal is to enforce a maximum regularization of structures with minimum deformations. Using the denoising model introduced in Section IV, the diffusion has to operate only on the fourth eigenvector direction ν 4 . This parametrization is therefore the same as setting ρ r = ∞ and α = ∞. In Fig. 22a-b, we observe that the regularization of the light field strengthens the continuity of the confidence estimate. The application of a same lower bound threshold on confidence highlights how homogeneous areas without disparity information are more easily detected. In Fig. 22e-f, we can see that the preprocessing regularization yields a more robust disparity estimate. Of course, the regularization of structures could also be enforced using third or second order eigenvectors for the diffusion, but at the cost of a greater structure smoothing. Such trade-off has been discussed in Section IV.

D. Inpainting

Tensor diffusion has been used for reconstructing removed parts of 2D images, such as in [START_REF] Tschumperlé | Vector-valued image regularization with PDEs: A common framework for different applications[END_REF]. The idea is to diffuse color information inside the masked area in the direction of structures present at edges of the mask. This approach let the information to fill in the inpainted area so that the continuity of structures is preserved. In Fig. 17, we show that our method is able fill the area of the three lines of text with coherent information regarding the neighboring structures.

VI. CONCLUSION

Regularization of light fields in the 4D ray space provides an efficient method for a wide range of applications. Using structural properties in the four dimension space, we showed that structural oriented diffusion can drastically reduce noise while keeping most of the essential spatio-angular structures of the light field. The denoising model we proposed offers an easy parametrization and can be adapted according to the purpose (smoothness, structures sharpness, process speed). We believe that other models can be designed, to benefit even more of the angular consistency of noised views, and to tackle more efficiently the sparsity and its inherent 4D sharpness. We have also shown promising results for spatial and angular interpolation, as well as for inpainting. Indeed, by simply tuning the weights of the tensors obtained by each eigen component, it is possible to apply the method to a variety of inverse problems associated to light fields.

Fig. 1 :

 1 Fig. 1: Tesseract as a 4D light field toy example: (a) Set of 25 × 25 light field views; (b) Subaperture image of 25x25 pixels image containing a square at the center of 21x21 pixels for the central views (fully black image at the periphery of the light field).

Fig. 2 :

 2 Fig. 2: Tesseract diffusion. (a-c) Tensor is constructed using ν 4 . (d-f) Tensor is constructed using ν 3 . (g-i) Tensor is constructed using ν 2 . (j-l) Tensor is constructed using ν 2 . (a,d,g,j) Initialization and diffusion orientation in red. (b,e,h,k) Diffusion at early iterations. (c,f,i,l) Close-up on (from top to bottom) center view [u=12,v=12], edge view [u=22,v=12] and corner view [u=22,v=22].

Fig. 3 :

 3 Fig. 3: Functions f i (ψ) controlling the smoothing strength of the diffusion tensor along orientations defined by the four eigenvectors ν i , with ρr = 0.4 and α = 2.

Fig. 4 :

 4 Fig. 4: Denoising behaviour for different noise levels applied to the Bikes light field. Each curve represents the PSNR evolution along the diffusion iterations, for different values of the parameter ρr, in the cases of (a) No noise; (b) additive Gaussian noise with σ = 10 255 ; (c) σ = 40 255 ; and (d) σ = 100 255 . The parameter α controlling the anisotropy is set to 2.

Fig. 5 :

 5 Fig. 5: Denoising behaviour for different models applied to the Bike light field of the EPFL dataset [33]. Each curve represents the PSNR evolution along the diffusion iterations for different noise standard deviations σ G , using a model parameterized by (a) ρr = 0.16; (b) ρr = 0.32; (c) ρr = 0.48; (d) and ρr = 0.80. The parameter α controlling the anisotropy is set to 2.

  (a) Original (center view) (b) Noised, σ = 50 255 , overall PSNR 14.15 dB.

Fig. 6 :

 6 Fig. 6: Influence of the anisotropic coefficient α on light field denoising using StonePillarsOutside of the EPFL dataset [33]. (a) Original center view; (b) center view with additive white Gaussian noise (σ = 50 255 ); (c) Influence of the anisotropic coefficient α on denoising efficiency. (d-f) Denoising results with parameter values on the blue line in Fig. (c): (d) Isotropic diffusion after 20 iterations. Structures are smoothed, no matter what their directions are. (e) Important structures are preserved, while textures are flattened. (f) Structures are preserved, thus yielding finer textures. However, the noise is getting structured in homogeneous areas (ground, sky, shades) instead of being isotropically smoothed, thus increasing error. (g-i) Best PSNR (red line in (c)).

Fig. 7 :

 7 Fig. 7: Denoising of the Tesseract toy example. (a,b) Gaussian noise added to tesseract example (standard deviation σ = 40 255 ).(c,d) After 20 iterations, using the 2D denoising method proposed in[START_REF] Tschumperlé | Vector-valued image regularization with PDEs: A common framework for different applications[END_REF]. (e,f) After 30 iterations using the proposed 4D denoising model. PSNR computed over all images.

  (a) Original. (b) Overall PSNR 28.13 dB. (c) Overall PSNR 35.10 dB.

Fig. 8 :Fig. 9 :

 89 Fig. 8: Denoising of Iso Chart 12 light field. Center views. (a) Original. (b) Gaussian noise of σ = 10 255 . (c) Using our method, 4 iterations.

  (a) Overall PSNR 14.15 dB. (b) Overall PSNR 29.91 dB. (c) Overall PSNR 30.10 dB, 19 iterations. (d) Overall PSNR 14.15 dB. (e) Overall PSNR 28.47 dB. (f) Overall PSNR 30.54 dB, 17 iterations. (g) Overall PSNR 14.15 dB. (h) Overall PSNR 30.22 dB. (i) Overall PSNR 30.82 dB, 23 iterations.

Fig. 10 :

 10 Fig. 10: Denoising of the Bikes, Flowers and Color Chart 1 light fields [33]. Center views. (a,d,g) Gaussian noise of σ = 50 255 . (b,e,h) Result obtained using LFBM5D with parameterization provided by the authors [2]. (c,f,i) Using our method. An extended quantitative comparison with other methods can be found in TableI.

  (a) Original center view. (b) After 15 iterations.

Fig. 11 :

 11 Fig. 11: Denoising of light field under low illumination conditions (Low-Light Flowers capture from the INRIA dataset [35]). The noise present in the input image results only from the poor capture conditions (no synthetic noise added). The denoising model is parameterized by ρ 4 = 0.4, ρ 3 = 1 2 ρ 4 , ρ 2 = 1 4 ρ 3 , and ρ 1 = 1 8 ρ 2 .

Fig. 12 :

 12 Fig. 12: Restoration of light field damaged using the Floyd-Steinberg dithering method with only 2 levels per color channel (Fountain and Vincent 2 light field of the EPFL dataset [33]). The denoising model is parameterized by ρ 4 = 0.4, ρ 3 = 1 2 ρ 4 , ρ 2 = 1 4 ρ 3 , and ρ 1 = 1 8 ρ 2 .

Fig. 13 :

 13 Fig. 13: PSNR evolution when denoising the light fields with Gaussian noise (σ = 50 255 ). The PSNR values are averaged over the 12 test light fields of the EPFL dataset [33].

Fig. 14 :

 14 Fig. 14: Denoising PSNR values for each light field view, with different angular croppings, and after 20 iterations. Each PSNR value corresponds to the average over the 12 Lytro Illum light fields of the EPFL dataset [33]. The added noise is Gaussian and of standard deviation σ = 50 255 .

Fig. 15 :

 15 Fig. 15: Light field views of the Friends 1 light field of the EPFL dataset [33], cropped such that u, v ∈ [0, 4]. During the denoising process, the diffusion tends to print views at the vignetted locations because of the aliasing.

  Fig. 16: Denoising results with the Treasure Chest light field (spatially cropped) of the Stanford dataset [36], with Gaussian noise (σ = 50 255 ). The model has difficulty coping with the high disparities at the foreground of the scene (bottom horizontal white line). Epipolar plane images are vertically stretched by a factor 4 for sake of clarity. Best viewed zoomed.

  (a) Original. Center view. (b) Inpainted. Center view. (c) Original. EPI at middle text. (d) Inpainted. EPI at middle text. (e) Original. EPI at bottom text. (f) Inpainted. EPI at bottom text.

Fig. 17 :

 17 Fig. 17: Inpainting of the light field capture Leaves from the INRIA dataset. Experiment parameterized by: β 1 = 0, β 2 = β 3 = β 4 = 1, ωx = ωy = ωu = ωv = 1.1 and wx = wy = 2.9, wu = wv = 0.

  Added Gaussian noise, σ = 50 255 . Overall PSNR 14.15 dB. (c) Overall PSNR 30.28 dB, 30 iterations.

Fig. 18 :Fig. 19 :

 1819 Fig. 18: Denoising results with the Fountain Pool light field [33], containing non-Lambertian surfaces. The denoising model is parameterized by ρ 4 = 0.4, ρ 3 = 1 2 ρ 4 , ρ 2 = 1 4 ρ 3 , and ρ 1 = 1 8 ρ 2 . Epipolar plane images are stretched in the angular direction by a factor 4 for sake of clarity. Best viewed zoomed.

Fig. 20 :

 20 Fig. 20: Subsampling method for spatial interpolation. Only integer magnification factors are allowed. For performance evaluation, if image size does not match the partitionning, a cropping based on the corresponding modulo is applied.

Fig. 21 :

 21 Fig. 21: Spatial interpolation example. Original light field (a,e) is spatially down sampled by a factor 3 and reconstructed to original dimensions (b-d and f-h). For this experiment, we set : ωx = ωy = 2.3 and ωu = ωv = 1.1, and wx = wy = wu = wv = 0.

  (a) Original (center view) and epipolar spatio-angular confidence. (b) Regularized (center view) and epipolar spatio-angular confidence. (c) Epipolar plane image of (a), left. (d) Epipolar plane image of (b), left. (e) Disparity estimate, opacity indexed to confidence. (f) Disparity estimate, opacity indexed to confidence.

Fig. 22 :

 22 Fig. 22: Light field regularization for disparity estimation enhancement. Diffusion is performed only in the fourth eigenvector ν 4 direction, and yields a more robust epipolar disparity estimate. Left: using original light field. Right: using regularized light field. Epipolar spatio-angular confidence φ in (a) and (b) is thresholded identically to filter out low confidence value, and used as alpha channel for disparity display in (e) and (f). The light field is BouquetFlower2 from the INRIA dataset.

TABLE I :

 I Comparison of light field denoising methods (courtesy of

TABLE II :

 II Light field restoration with different types of noise. The PSNR values (in dB) are averaged over the 12 EPFL light fields

  Diffusion Nearest Bicubic Diffusion Nearest Bicubic Diffusion Nearest Bicubic Diffusion

			Integer magnification, factor 2			Integer magnification, factor 3
	Light field		RGB			Luminance			RGB			Luminance
	Nearest Bicubic Bikes 24.54 26.98	28.43	25.32 28.25	29.58	23.08 24.03	26.05	23.81 24.79	26.80
	Bench in Paris	20.71 22.97	23.70	21.10 23.55	24.16	19.45 20.08	21.22	19.83 20.48	21.56
	Friends 1	25.98 27.52	29.17	26.79 28.26	30.03	24.44 25.14	27.75	25.16 25.83	28.63
	Sphynx	24.83 26.65	28.54	25.40 27.29	29.30	23.53 23.92	26.31	24.05 24.43	26.95
	Bee 2	25.62 28.20	29.17	28.02 31.65	32.52	24.50 25.51	27.14	26.80 28.06	29.83
	Duck	21.67 24.66	24.91	22.78 25.81	26.19	20.49 21.09	22.43	21.56 22.21	23.55
	Fruits	24.55 27.61	28.61	25.40 28.83	29.79	23.49 24.17	26.12	24.31 25.03	27.05
	Rose	27.96 30.73	32.20	29.05 31.87	33.45	26.56 27.45	29.80	27.54 28.29	30.84
	Mini	24.30 26.27	27.64	24.98 26.97	28.34	23.08 23.46	25.52	23.73 24.10	26.17
	Buddha	29.90 33.19	33.55	30.07 33.37	33.72	28.51 29.47	30.88	28.66 29.63	31.03
	Cone Head	33.11 36.64	35.54	35.57 39.12	38.51	32.03 32.59	33.63	34.51 35.16	36.29
	Monas	29.28 32.94	32.92	30.05 33.49	34.03	28.15 28.90	30.54	29.91 29.58	31.44
	Watch	30.16 37.35	36.18	30.16 37.35	36.18	28.96 30.77	32.11	28.96 30.77	32.11
	Papillon	32.00 35.62	37.50	32.01 35.75	37.82	30.86 32.20	34.65	30.87 32.22	34.84
	Still Life	22.61 24.79	25.41	22.72 24.87	25.50	21.01 21.20	22.75	21.10 21.27	22.82
	Average	26.48 29.47	30.23	27.29 30.43	31.27	25.21 26.00	27.79	26.05 26.79	28.66
	Average increment -2.99 from bicubic	0	+0.76	-3.14	0	+0.84	-0.79	0	+1.79	-0.74	0	+1.87

TABLE

  1 8 ρ 2 .

		33										33
		32										32.5							5x5 views, max:31.95dB 7x7 views, max:32.05dB
		29 31 30										32 31.5							9x9 views, max:32.03dB 11x11 views, max:32.02dB 13x13 views, max:32.01dB 15x15 views, max:32dB
	PSNR	27 28									PSNR	31 30.5
		26										30
		25							5x5 views, max:31.07dB 7x7 views, max:31.35dB		29.5
		24							9x9 views, max:31.35dB 11x11 views, max:31.19dB		29
		23							13x13 views, max:29.55dB 15x15 views, max:28.79dB		28.5
		22										28
		0.5	1	1.5	2	2.5	3	3.5	4	4.5	5	0.5	1	1.5	2	2.5	3	3.5	4	4.5	5
						Integration time									Integration time
					(a) Overall.					(b) Center view (rescaled plot).

i=1 λ i , which is considered to be a good measure of local variation[START_REF] Tschumperlé | Anisotropic Diffusion Partial Differential Equations in Multi-Channel Image Processing : Framework and Applications[END_REF].In the end, depending on the application, the diffusion can be either driven by constants: β i = A i as for spatial and angular interpolation as we will see later, or by the local structures: β i = f i (ψ), as we will detail next.

http://clim.inria.fr/research/AD4D

http://clim.inria.fr/research/AD4D