Morgane Vincent Clément

Bachir, Minmin, Florent, Thibauld Nicolas Ali

Lucas

Merci Bien Sûr À Tristan

Thibaud B Jérémy

Axel De

Micheline Véronique

Alina Sandra

Christophe Sébastien

In this CIFRE thesis, a collaboration between the Centre de Mathématiques et leurs Applications, École Normale Supérieure Paris-Saclay, and the company DxO Labs, we tackle the problem of the additive decomposition of an image into base and detail. Such a decomposition is a fundamental tool in image processing. For applications to professional photo editing in DxO Photolab, a core requirement is the absence of artifacts. For instance, in the context of contrast enhancement, in which the base is reduced and the detail increased, minor artifacts becomes highly visible. The distortions thus introduced are unacceptable from the point of view of a photographer.

The objective of this thesis is to single out and study the most suitable filters to perform this task, to improve the best ones and to define new ones. This requires a rigorous measure of the quality of the base plus detail decomposition. We examine two classic artifacts (halo and staircasing) and discover three more sorts that are equally crucial: contrast halo, compartmentalization, and the dark halo. This leads us to construct five adapted patterns to measure these artifacts. We end up ranking the optimal filters based on these measurements, and arrive at a clear decision about the best filters.

In the first part of the dissertation we study the widely used guided and bilateral filters. Indepth analysis of the guided filter and confrontation with the bilateral filter are performed. An asymptotic analysis of the filter when its support tends towards zero permits to link it with the Perona-Malik anisotropic diffusion. It is shown that the guided filter does not have the edge amplification term which has been proven to cause the staircase effect; this is experimentally verified with an implementation that simulates this equation. This new filter has no halo, nor contrast halo. We then review the bilateral filter along with its main fast approximations, and the solutions to the staircase effect provided in the literature.

The second part of the dissertation deals with multi-scale filters. We begin by studying a method called exposure fusion that fuses bracketed exposure sequences of images. We extend it to contrast enhancement by simulating the sequence from a single image. The study of this particular case leads us to identify the core principle of the contrast manipulation in exposure fusion. This yields further improvement in the proposed algorithm. Then, we study the local Laplacian filter, for which we propose a compact formula when interpreted in a scale-space. This interpretation reestablishes the translation invariance. Furthermore, the scale-space allows to replace the guide with the result of an arbitrary edge-aware filter, thus reducing the luminance halo. Lastly, we study the weighted least squares filter that also performs a multi-scale decomposition of the image. Its main artifact is unveiled and partially corrected.

This systematic analysis of the main decomposition filters in the literature and the identification of their respective artifacts leads us to propose a quantitative method for comparing them. For each one of the five proposed artifacts types, we create a pattern-measure pair. After setting the filter's parameters so that they extract the same amount of detail, the filters are applied on this collection of test-images, and the presence of each artifact is measured. We then rank the different method according to the quality of the decomposition and conclude. Two filters stand out, including one we propose.

Je tiens avant tout à remercier mes rapporteurs Marcelo Bertalmío, Pierre Kornprobst et Sylvain Paris pour avoir participé à l'évaluation de mon travail, pour leurs conseils éclairants et leurs remarques enrichissantes. Je remercie également Christine Graffigne qui m'a fait l'honneur de présider mon jury. J'exprime toute ma gratitude à DxO qui m'a accueilli pour cette thèse CIFRE.

Je remercie à Benoît Chauville, encadrant de ma thèse à DxO, qui m'a orienté dans mon travail et donné le temps de traiter chaque sujet en profondeur.

Je tiens à remercier Wolf Hauser, avec qui j'ai eu le plaisir de travailler à DxO. J'ai beaucoup appris de ses présentations et de nos discussions. Ses remarques et questions ont éclairé mes recherches tout au long de la préparation de cette thèse.

Je souhaite remercier très chaleureusement Gabriele Facciolo. Ce travail a aussi bénéficié de sa collaboration remarquable. J'ai beaucoup apprécié nos très éclairantes discussions, et ses conseils et encouragements m'ont été précieux.

Ce travail n'aurait pas été possible sans les spécialistes de l'image de DxO, qui ont participé à l'évaluation des algorithmes. Je remercie notamment Benoît, Wolf, Gabriele, Bao, William, mais aussi tous les traiteurs d'image de l'équipe : votre participation a été essentielle pour mener à bien ce travail.

Merci enfin à toutes

Résumé

Dans cette thèse CIFRE en collaboration entre le Centre de Mathématiques et de leurs Applications, École Normale Supérieure Paris-Saclay et l'entreprise DxO, nous abordons le problème de la décomposition additive d'une image en base et détail. Une telle décomposition est un outil fondamental du traitement d'image. Pour une application à la photographie professionnelle dans le logiciel DxO Photolab, il est nécessaire que la décomposition soit exempte d'artefact. Par exemple, dans le contexte de l'amélioration de contraste, où la base est réduite et le détail augmenté, le moindre artefact devient fortement visible. Les distorsions de l'image ainsi introduites sont inacceptables du point de vue d'un photographe.

L'objectif de cette thèse est de trouver et d'étudier les filtres les plus adaptés pour effectuer cette tâche, d'améliorer les meilleurs et d'en définir de nouveaux. Cela demande une mesure rigoureuse de la qualité de la décomposition en base plus détail. Nous examinons deux artefacts classiques (halo et staircasing) et en découvrons trois autres types tout autant cruciaux : les halos de contraste, le cloisonnement et les halos sombres. Cela nous conduit à construire cinq mires adaptées pour mesurer ces artefacts. Nous finissons par classer les filtres optimaux selon ces mesures, et arrivons à une décision claire sur les meilleurs filtres.

Dans la première partie de la thèse nous étudions les filtres bilatéraux et le filtre guidé. Une analyse approfondie du filtre guidé et une confrontation avec le filtre bilatéral sont réalisées. Une analyse asymptotique du filtre quand son support tend vers zéro permet de faire le lien avec la diffusion anisotropique de Perona-Malik. Il est démontré que le filtre guidé ne possède pas le terme d'amplification des contours dont il a été prouvé qu'il provoquait l'effet d'escalier (staircase effect) ; cela est expérimentalement vérifié par une implémentation qui simule l'équation. Ce nouveau filtre ne possède pas de halo, ni de halo de contraste. Nous examinons ensuite les filtres bilatéraux et leurs approximations rapides, ainsi que les solutions au staircase effect proposées dans la littérature.

La suite de la thèse traite de filtres multi-échelle. Nous commençons par étudier une méthode baptisée exposure fusion qui fusionne des séquences d'images avec des variations d'exposition. Nous l'étendons au rehaussement de contraste par la simulation d'une séquence d'images. L'étude de ce cas particulier nous mène à identifier le principe à l'oeuvre dans la manipulation de contraste d'exposure fusion. Cela nous permet d'améliorer encore l'algorithme proposé. Nous poursuivons avec le filtre local Laplacian filter, pour lequel nous proposons une formule compacte lorsque interprété dans un scale-space. Cette interprétation permet de rétablir l'invariance par translation. De plus, l'utilisation d'un scale-space permet de remplacer le guide par le résultat d'un filtre de lissage avec préservation des contours arbitraire, et ainsi de réduite le halo de luminance. Pour finir, nous étudions le filtre weighted least squares qui propose également une décomposition multiéchelle d'une image. Son artefact principal est révélé et partiellement corrigé.

Cette analyse systématique des principaux filtres de décomposition en base et détail de la littérature et de ceux que nous proposons, ainsi que l'identification de leurs artefacts respectifs, nous conduit à proposer une méthode quantitative pour les comparer. Pour chacun des cinq types d'artefacts proposés, nous créons une paire mire-mesure. Après avoir réglé les paramètres des filtres de sorte qu'ils produisent des décompositions comparables, les filtres sont appliqués sur les mires et la présence de chaque artefact est mesurée. Nous classons alors les différents algorithmes selon la qualité de la décomposition et concluons. Deux filtres sortent du rang, dont un proposé dans cette thèse.

Introduction

This CIFRE thesis has been undertaken in a collaboration of Centre de Mathématiques et leurs Applications, École Normale Supérieure de Saclay, with the company DxO, where I worked in the image processing team on DxO Photolab (formerly Optics Pro), a photo editing software  . The team is working at producing the best quality images from RAW pictures, but also from JPEG files produced by any camera. In this context it has been observed that it is often necessary to decompose an image in what we intuitively call a base layer and a detail layers.

The object of the thesis is the automatic additive decomposition of digital images in base and details layers, with the particular purpose of local contrast manipulation. It aims at adding more clarity to the image by enhancing its detail. This problem is directly related to the so-called retinex theory, [LM] originally proposed in the seventies as a theory of the human perception of color. This theory has later been used to enhance digital images. In this context retinex enhancement algorithms try to transform the digital images so that the result is close to what a human observer would have seen by looking at the original scene [JRWa,PSM,Get]. This goal has often been simplified as "seeing in the shadows" (of the digital image). The tone-mapping operators also belong to that category. The tone-mapping problem has the contradictory objectives of reducing the dynamic of an image while preserving the local contrast. This is needed in high-dynamic range imaging, where the dynamic range of an image must be reduced prior to display or printing (because of the small dynamic range of standard screens and printers). Retinex and tone-mapping operators can be divided in two categories: those which perform a base and detail decomposition; those which do not and deliver directly a enhanced image.

The simplest available tool for that is the combination of a low-pass filter and of high-pass filter, which decompose an image in its low frequencies (base) and high frequencies (detail) content. This is used for example in the unsharp mask technique, [MLLY, PRM] which can be computed with the Fourier transform. Wavelet transforms localize the frequency analysis in the image and can thus be used as well [Mal]. Morphological filters like the grain filter and the area filters [Vin, MG] are another class of filter that can be used for enhancement. Closing, opening (used in the top hat filter for example) or the median filter are another option [Ser]. Anisotropic diffusion PDE filters [PM] are another classical option to compute a base. They have the double objective of smoothing and simultaneously enhance the image (Coherence-Enhancing Diffusion Filtering [Wei] for example). Denoising filters can also be viewed as methods for decomposing an image into base and detail, the noise standard deviation playing then the role of a scale parameter. The base is the recovered image while the detail corresponds to the removed noise. This is the case of the bilateral filter, which usage for base and detail decomposition is widespread, but was originally designed as an image denoiser. The total variation regularization [ROF] was also originally intended as a denoising algorithm but also adapts excellently to a base-detail separation, often called cartoon+texture decomposition [Mey, Gue].

 See http://www.dxo.com/us/photography/photo-software/dxo-photolab



In brief, there is a wide panoply of image filters that may be used with the purpose of decomposing an image into base and detail. In this thesis, our purpose is to review the most relevant such decomposition methods, to find and improve the best ones and possibly define new ones. This requires rigorous measurements evaluating the quality of the results. As we shall see, we will be led to measure the various artifacts produced by each sort of filter.

The difficulty of the problem lies in our notion of "base" and "detail". Indeed, while linear filtering would smooth them out, our notion of base may retain sharp edges in the base and exclude them from the detail. Thus, such a decomposition is both additive and in essence non-linear. Our research methodology is to understand, improve and evaluate edge-preserving smoothing filters, i.e. filters that compute a base. During the study we shall define the artifacts, specific to a filter or, more often, typical of a class of filters. We shall base our definition of the artifacts on the subjective feedback of DxO image experts, that we aim at transforming into rigorous quantitative measurements. Those ratings are highly non-linear. We first systematically try in this thesis to correct the unveiled artifacts for each filter. Notably, no filter is actually exempt of artifacts, as we shall define them. However, the artifacts are not equivalently annoying from the point of view of a photographer, and the pregnancy of each defect may vary, so that many an artifact may fall below a subjective "objectionable" threshold.

We eventually select the algorithms that offer the best compromise among those artifacts, thanks to a quantitative measure carried out on the artifacts we isolated. In our final ranking, we take into account the complexity of each filter. Indeed, this parameter, though often in contradiction with the quality of the decomposition, may be decisive when it comes to select a filter in an already long and complex image processing pipeline.

In short, this dissertation develops a methodology for the quantitative evaluation of the quality of the base and detail decompositions of any image filter. After a careful examination of many filters and of their artifacts, we end up creating a set of test-patterns, one for each of the five identified artifacts, and five metrics that go along the proposed test-patterns. The method takes in input any filter with its parameters fixed, except for one that controls the quantity of detail extracted by the algorithm. This last parameter is set so that the L 2 norm of the produced detail matches a predetermined number. The value of this L 2 norm is in fact an average of the values of detail L 2 norms obtained with a representative test set of natural images. The equalization of the L 2 norms of the detail proposed by each filter ensures that the filters can be fairly compared. This leads to evaluate quantitatively the five artifact measures for all filters on all test-patterns and eventually to propose a ranking method and a final ranking for all examined filters. As we shall see, two classic -but improved by us -emerge from this study.



. Retinex methods

The Retinex theory was first formulated by Edwin H. Land in  [Lan]. It was a ground breaking attempt to model how the human visual system (HSV) perceives colors in a scene. This theory was further formalized by Land and McCann [LM]. They established that the visual system does not perceive an absolute lightness but rather a relative lightness, namely the variations of lightness in local image regions. This was proven by the experiments using Mondrian patterns [Lan, Lan], were they showed that color sensation is not directly linked to the spectral characteristics of the perceived signal: patches with different reflectance are perceived with different colors even when they have the same spectral light distribution because of a change in the spatial illumination. This is what A. Rizzi et al. called color constancy [RM]. In early results, Land assumed that three independent sets of receptors exist and that the comparison of these three receptor outputs gives the sense of color. He named this system Retinex, a neologism made of retina and cortex. Although the original work did not involve digital images, Retinex can be used to enhance digital images, as suggested by Land himself.

Implementations and derivations of Retinex have been an active research field which now counts a wealth of publications. As explained in a recent overview of Retinex methods by Petro et al. [PSM], the many implementations can be divided in two groups. The first group explores the image relative lightness using a variety of image paths or comparing the current pixel color to a set of random pixels [Lan], [Lan], [FM], [MR], [PFR + ]. The second group uses a convolution mask or variational techniques to compute a locally enhanced image [Lan], [JRWb], [JRWa], [KES + ], [BF], [MPS], [MMOC], [BCP].

Nowadays, the most prominent retinex implementation is an alternative to the initial random walk algorithm published by Land [Lan]. This implementation computes the lightness as the ratio between the value of a pixel and the average value of the surrounding samples. Taking for example a Gaussian filter G σ , the operation amounts to set L(x) := I(x) (I * Gσ)(x) , which implies log L(x) := log I(x) -log(I * G σ )(x).

(.)

This equation (.) is the so-called single-scale retinex (SSR) method, explored by Jobson et al.

in [JRWb] and later extended by the same authors to multiple scales [JRWa]. The last is called multiscale retinex (MSR) and its formula is

MSR{u}(x, i) = N n=1
w n SSR{u, n, i}(x)

= N n=1 w n log u(x, i) -log (G σn * u(i))(x) , (.)
where N is the number of scales, w n is the weight of each scale and G σn (x) = C n exp(x 2 /2σ 2 n ), a Gaussian kernel with normalization factor C n . An excellent overview of the retinex theory and algorithms can be found in Bertalmío's book of [Ber], along with connection to percetuallybased variationnal techniques [PAPBC, FBPC] and ACE.

The Automatic Color Enhancement (ACE) proposed by Gatta et al. [GRM] is strongly related to Retinex. It was further developed in [RGM, RGM, BCPR]. It has been proven by Bertalmío et al. in their excellent paper [BCP], that "can be seen as a particular antisymmetrization of the KBR [Kernel-Based Retinex] model". This last method, compared to Retinex, has the advantage of improving the contrast in both the dark and bright parts of an image, whereas Retinex has a tendency to move the histogram to the right, and thus to shrink contrast in the bright where u : Ω → [0, 1] is the input image and s α : [-1, 1] → R is the slope function



s α (t) = min max{αt, -1}, 1 , (.)
where α is a user-set parameter (displayed in Figure .). The final result is a stretching of ACE{u} to [0, 1], as many of its values are negative. We shall analyze in Chapter  the link between ACE and the bilateral filter.

The retinex filters create objectionable halo artifacts. For this reason, they are not acceptable for contrast enhancement in professional photography. Figure . illustrates this fact and shows the superiority of filters performing a base + detail decomposition, like the filter MGF which will be developed in this thesis. In the next sections, we detail our contributions chapter by chapter. Chapters  to  proceed to detailed analyses of filters and to the detection, explanation, and when possible correction of their artefacts. The long Chapter  gives the final evaluation methodology. Multi-scale retinex introduces a halo around the lighthouse, but MGF does not. The multi-scale guided filter is a base and detail decomposition algorithm; the decomposition obtained for the luminance part of the input image is displayed on the first line. Both algorithms work on the luminance only. Note that MSR does not manage to preserve the contrast of the lighthouse facade, while MGF does; besides, the base and detail decomposition gives much flexibility to the algorithms, which could be used for example to further increase the local contrast.





. Chapter : Guided filter

In Chapter , we start our analysis of the guided filter [HSTb]. Its artifacts, a contrast halo and a luminance halo, are explained. A comparison of the filter's performance is made with the related bilateral filter. We show that attempts to find a correspondence between the parameters of both filters are vain; the guided filter does not have the edge-preserving capability of the bilateral filter. We however present in Chapter  a new filter based on GF that reduces its artifacts while keeping the very desirable property of being a locally affine transformation of the guide image, which avoids the staircase effect.

The guided filter The guided filter (GF) has two steps: the first one computes a linear transform of a guidance image in small patches. In each patch ω, GF solves:

E a(y), b(y) = x∈ω(y) a(y)v(x) + b(y) -u(x) 2 + a(y) 2 , (.)
where u is the input image, v the guide, a smoothing parameter and ω the patch. This model ensures that the gradients in the filtered patches are proportional to the gradients of the guide image, and avoids the staircase effect of the bilateral filter. On another hand, it introduces a contrast halo and a luminance halo. The second step aggregates the filtered values of all overlapping patches. This is equivalent to averaging the coefficients (a, b) of the overlapping window so the final output is

GF{u}(x) = ā(x)v(x) + b(x), (.)
where (ā, b) are the aggregated linear coefficients. Equation (.) has an analytic solution, making the filter extremely fast to compute, since it requires only local averages, that can be computed in linear time thanks to integral images.

The contrast halo artifact in GF This main artifact of the guided filter comes from the fact that the edges are preserved, but the area around them is preserved too. We show an example of the resulting phenomenon in Figure .. It is especially present when the filter is used with a large radius. Indeed, the guided filter can't smooth out half of a window and keep the other half as it is; the choice is often an intermediate decision: half smoothed, half kept. Thus, it also creates a luminance halo artifact.

The luminance halo artifact in GF The luminance halo artifact arises when edges are not well preserved by the filter. This is the case with the guided filter, as shown in Figure .. Compared to its competitor the bilateral filter, the guided filter smooths less the textures that should be removed and smooths more the edges that should be preserved.

. Chapter : Iterated guided filter

Chapter  introduces the guided filter and its artifacts, namely, the contrast halo and the luminance halo. A comparison to the bilateral filter shows that its edge-preserving and smoothing property does not put the bilateral filter in the shade. On another hand, the guided filter has the neat advantage not to exhibit the staircase effect. This makes this filter particularly desirable for contrast enhancement. Figure 1.4: Comparison of the bilateral and guided filter for a test pattern containing a step edge and a sawtooth structure. In the left row, the parameter equivalence is = ( σr 2 ) 2 , in the right row it is = σ 2 r . The spatial parameter used here is r = σs = 3 and the range parameter is σr = 50. Obtaining the same reduction of the oscillating structure as the bilateral forces the guided filter to lose more contrast on edges.



A Partial Differential Equations Analysis of the Guided Filter

In [BCM] the authors proved the presence of a staircase effect in the bilateral filter by showing that it is asymptotically equivalent to a Perona-Malik equation containing a reverse heat equation term creating shocks along zero-crossings of the Haralick edge detector [Har]. Following the same methodology, we prove in Chapter  that the guided filter is equivalent to one iteration of an anisotropic diffusion partial differential equation, that can be interpreted as the first, diffusive, term of a Perona-Malik equation. This explains why the guided filter does not show staircase artifacts.

Theorem .. Consider a D image u(x, y) ∈ C 3 (Ω). Let f 1 (x, y) be a nonnegative compactly supported radial kernel. We assume that the filter is normalized, namely f 1 (x, y)dxdy = 1; and symmetric xf 1 (x, y)dx = yf 1 (x, y)dy = 0. Set ˜ = /M 20 where is the edge preserving parameter of the guided filter, and M 20 = f 1 (x, y)x 2 dxdy = f 1 (x, y)y 2 dxdy. Finally, let f σ be the scaled kernel: f σ (x, y) = σ -2 f 1 (x/σ, y/σ). Then, for (x, y) ∈ Ω, Following the interpretation of this theorem, we implement an iterated guided filter with a small radius that simulates this equation and prove that it is halo free. This filter can be simply written IGF σ {u}(t, x) = āσ (t, x)IGF (1) σ {u}(t -1, x) + bσ (t, x). (.)

In Figure ., a confrontation of the results of this filter to the ones obtained by the classic bilateral filter shows that it is no longer affected by any staircase effect. As a consequence of the absence of the edge reinforcement term, the smoothing effect is stronger. Furthermore, we propose two other versions of the filter. One involves a guide image and another accelerates the filter by computing the linear coefficient a only once. The iterated guided filter causes no staircase artifact. Parameters used here: = σ 2 r = 0.01 2 with the input dynamic range in [0; 1]; r = σs = 1 with the input image of size 250×250; number of iterations T = 50. The bottom graph displays the restrictions of the three above images to the vertical straight lines drawn on the images. The staircase effect of the bilateral filter (orange line) doesn't appear on the guided filter version (red line).



. Chapter : Bilateral filter

Chapters  and  are dedicated to the fast and recent guided filter, link it to the anisotropic diffusion and compare it to the bilateral filter. Those two last filters are the most widespread filters for the computation of an image base.

In Chapter , we present the bilateral filter. We recall its long history, and describe its main descendants: the joint (or cross) bilateral filter [ED, PSA + ], the bilateral filter with regression [BCM], the unnormalized bilateral filters [APH + , APH + , MT]. Furthermore, we make the link between the bilateral filter and ACE (the Automatic Color Enhancement), that belongs to the retinex family. We also explain the staircase effect first described, and solved, by Buades et al. [BCM].

Two others chapters dedicated to the bilateral filters follow this one. A review of the numerous schemes proposed to correct the staircase effect (Chapter ), and a review of the fast approximations, particularly usefull when the filter is used with large spatial neighborhood as in the base and detail decomposition problem (Chapter ). However, since the unnormalized bileral filter is defined in this chapter, we get ahead and present its fast approximations here. Likewise, we propose in this Chapter  a fast approximation of the bilateral filter with regression and a multi-scale filter based on it. This last filter gives us the opportunity to define and explain the dark halo artifact. This chapter, along with the two following ones on the staircasing corrections and the fast approximations of the bilateral filter, is directly inspired by the book by S. Paris, P. Kornprobst, J. Tumblin and F. Durand [PKTD]. Whereas this book aims at giving an extensive presentation of the bilateral filter and its applications, we concentrate on its usage for base and detail decomposition. Nonetheless, we approach several points already reviewed in the  book, e.g., the  different proposed extensions and its fast approximations. We highlight below the main differences between our Chapters , ,  and Paris, Kornprobst, Tumblin and Durand book. Concerning this Chapter  on the bilateral filters, we present supplementary filters and links:

• we make the link with ACE (Automatic Color Enhancement) [GRM];

• we review the unnormalized bilateral filters [APH + , MT], along with their fast approximations;

• we propose a fast approximation for the bilateral filter with regression;

• we propose a multi-scale bilateral filter with regression.

We pursue the review of the bilateral filter with the staircase effect corrections in Chapter . There are two kinds of corrections: the first modify the bilateral filter so that the slopes are taken into account, e.g., the bilateral with regression filter, the trilateral filter, the symmetric bilateral filter; these have been reviewed in Paris et al. book, so the differences comes down to:

• a more detailed presentation of the trilateral filter, with pseudo-codes;

• the introduction of a symmetric bilateral filter similar to Elad's one [Ela].

The second kind of approximations however is not described in [PKTD]. It consist in postprocessing the filtered image to correct the staircase artifact. The described corrections are:

• the blending described by Durand and Dorsey [DD];

• the minimal isotropic smoothing effect in the separable kernel approximation [PVV];

• the Poisson correction proposed by Bae et al. [BPD];

• the selective diffusion of Kass and Solomon [KS].

Concerning the fast approximations, most of them are reviewed in the book. Nonetheless, we add to the list filters posterior to  and sometimes give more detailed descriptions:

• in the local histograms, Weiss [Wei] approximation is described in the book, yet we give of it a more in-depth description: we present the earlier Huang's algorithm and give for both pseudo-codes. Furthermore, we review Porikli's  version that uses integral histograms, and discuss the usage of box spatial kernels;

• the fast approximations of the unnormalized bilateral filter and to the bilateral filter with regression are given in Chapter ;

• we present a supplementary class of fast approximations based on the usage of polynomials range kernels;

• the domain transform is also reviewed, this filter can be thought as a bilateral filter when used with a small spatial kernel.



The bilateral filter The principle of bilateral filtering appeared with Yaroslavsky () [Yar] and Lee () [Lee]. The variant we study was proposed by Smith and Brady who called it "SU-SAN" () [SB]. It was re-proposed by Tomasi and Manduchi under the name "bilateral filter" in  [TM]. All of these similar filters can be termed neighborhood filters; the only differences lies in the shape of the range and space kernels. The performance of these algorithms is justified by the same arguments: inside a homogeneous region, the gray level values slightly fluctuate because of noise or texture. In that case, the bilateral filter computes a mean. At a contrasted edge separating two regions, if the gray level difference between both regions is significantly larger than σ r , then the algorithm computes averages of pixels belonging to the same region as the reference pixel. Thus, the algorithm does not blur the edges, which is its main scope.

The version popularized by Smith and Brady and Tomasi and Manduchi uses a Gauss weighting function depending on a filtering parameter σ r (range kernel), as well as a Gauss spatial kernel: dy is the normalization factor and σ s is now a spatial filtering parameter.

The cross bilateral filter [ED], or joint bilateral filter [PSA + ], computes its range kernel according to a second guide image v: where the normalization factor C is computed accordingly. This is used for example for flash/no flash image denoising, where the edge information of the flash image are used to filter the no flash image, with better colors but more noise.

Staircasing

The staircase artifact is illustrated in figure . . In this figure we simplified the range and spatial kernels by using simple boxes. This allows a simple visualization, in the -dimensional case, of what pixels are taken into account in the averaging process. The blue arrows are the intensity differences u(x) -u(y). The dotted box shows the boundaries of the range and spatial kernels: outside of this box, all the bilateral weights are zero. Then, it is easy to see that for the current pixel (namely the intersection of the two blue dotted lines at the center of the box) the averaged value has a higher intensity than the initial one. By applying the bilateral averaging on each pixel of the blue line, one obtains the red line. The "propagation of the plateau" that one can observe is what we call the "staircase artifact". It amounts to an undesirable edge enhancement. Averaging the differences (u(y) -u(x)) using the bilateral weights actually directly computes the detail layer. Because this layer intensities oscillate around zero, the normalization factor can be removed without distorting the filtered image. This accelerates the filter and reduces staircasing; however it also reduces its smoothing properties and introduce a small, guided filter like contrast halo. This can be understood with the alternative definition of the unnormalized bilateral filter based on BF: UBF{u}(x) = C(x)BF{u}(x) + 1 -C(x) u(x), (.)

The unnormalized bilateral filter

where C is the bilateral filter normalization factor. .6: Explanation of the staircase effect for a bilateral filter with simplified range and spatial kernels. The current pixel is at the intersection of the vertical and horizontal blue dotted lines. The dotted black rectangle indicates which pixels will be considered in the average. Light blue vertical arrows stand for the intensity difference between the current pixels and the pixels in the rectangle. Since the current pixel has more neighbors (in the bilateral definition) on the right side of the edge, its bilaterally averaged value gets up and closer to the plateau's value.

The bilateral filter with regression This filter, introduced by Buades et al. in [BCM], estimates a regression plane at each pixel. Unlike the standard bilateral filter that estimates a constant, the filter, used with small σ s , no longer causes staircase artifacts. We call k = k(x, y) the weights of the bilateral filter at point (0, 0) for the image u = u(x, y). The bilateral filter with regression does find arg min a,b,c x,y k(ax + by + c -u) 2 .

(.)

The final result is simply BFR{u}(x) = c(x). We complete this chapter by analyzing a last candidate to attenuate the staircase effect, the unnormalized bilateral filter.

. Chapter : Staircase effect corrections

In Chapter  we showed that the bilateral filter not only preserves the edges, but is prone to sharpening them. This effect has been described and mathematically justified by Buades et al. in  [BCM], who call it the staircase effect. Indeed, bilateral-based filters tend to create piece-wise constant signals separated by numerically created edges, thus adopting the aspect of a staircase. From the contrast enhancement and tone-mapping point of view the same effect is sometimes called the gradient reversal artifact, because the complementary detail layer, at places where edges have been reinforced in the base layer, contains reverted gradients. The problem is that when using the bilateral filter for contrast enhancement and tone-mapping, the detail layer gets stretched and the base layer compressed. The recombination of their results causes the gradient reversal artifact. Since this artifact is particularly annoying in contrast manipulation methods, many authors have tried to correct it. The solutions can be divided in two categories. The first category of correction does not modify the filter, but corrects the artifact in a post-processing step. The second one directly modifies the filter to make it handle smoothly the slopes. We review in this chapter both categories of corrections. This chapter is inspired by the Paris et al. book on the bilateral filter [PKTD]. The differences with our review are highlighted in the previous Section ..



Example: Durand-Dorsey correction In the manner of the Durand-Dorsey correction, most of the post-processing step that aim at removing the staircase effect of the bilateral filter apply Gauss filters to the bilateral-smoothed image. The difficulty is then to find the right standard-deviation of the Gauss filter and where to apply it.

Durand and Dorsey in [DD] brought a simple answer; they apply only one Gauss filter with small standard-deviation and then blend between this blurred image and the non-blurred one in function of the normalization factor of BF. Let α be the linear interpolation coefficient between the bilateral filter result FBF{u} and its blurred version FBF{u}. This coefficient varies with log C. The function α = f log(C) is defined as α(x) = log C(x) /log C max , where C max is the maximal possible value for C, i.e. C max = y G σs (xy). The corrected image is then FBF{u} corr (x) = α(x)FBF{u} + 1 -α(x) FBF{u}(x). (.)

Another correction iteratively smooths the bilateral results, with Gaussian filters of increasing width. This is the selective diffusion of Kass and Solomon [KS]. At each iteration, they choose between the image before and after blurring by measuring locally the distance to the original image, and keeping the closest one. The idea is that if blurring the bilateral result brings it closer to the original image, then the blurred version should be preferred.

Example result with the selective diffusion Figure . displays the result of the selective diffusion applied to the bilateral filter, in the context of contrast enhancement. It succeeds in removing a large part of the gradient reversal artifact (a consequence of the staircase effect) visible as a dark and white bands along the top of the trees. Although this method works globally well, it seems unable to remove the staircases everywhere, especially in the corners (see Figure .(c)). Furthermore, it is not computationally efficient. Indeed, numerous iterations are needed to correct the staircase effect, and this computation time adds to the computation time of the filter itself.

. Chapter : Fast bilateral filters

The bilateral filter has rapidly become ubiquitous in image processing and is now used in a tremendous number of applications. The original filter needs to compute a different kernel at each pixel which makes it slow, nay not affordable for large images and (consequently) large spatial support. Hence the need for a fast implementation.



In Chapter  we review the numerous fast bilateral filters proposed in the literature. The history of the fast bilateral filter starts with the fast Durand and Dorsey approximation () [DD], who presented the original idea, that would be extensively explored later, of sampling the intensity range so as to linearize the convolution. The Gaussian convolution can then be computed using one of the numerous fast schemes available. No fast and exact implementation of the bilateral filter has been proposed yet. Thus the competition between the numerous proposed schemes not only lies in the speed but also on the precision and the unavoidable artifacts. Furthermore, for several schemes the speed depends on the parameters used and on the dimension in which the filter operates. Thus we eventually present a palette of effective filters rather than a definitive winner.

This chapter is also inspired by the Paris et al. book on the bilateral filter [PKTD]. The differences with our review are highlighted in Section .. This formulation of the bilateral is exact, and shows that it can be computed by series of linear convolutions: one per possible value of u(x). The acceleration strategy is then to compute the exact result for a limited number of intensities only. This amounts to sampling the intensity range and to linearly interpolate between these layers v(x, i) and C(x, i) for values that lie between the samples.

The piecewise-linear approximation

The bilateral grid The piece-wise linear method has later been improved by Paris et al. [PD, CPD] in the bilateral grid. This method also linearizes the convolution and downsamples the signal to reduce computational complexity, but also gives a more formal definition of this fast approximation thanks to a high dimensional interpretation of images, and a gain in precision due to a better subsampling in the range domain. This approximation is probably one of the most effective, and one of the most representative of the literature on the fast bilateral filters. We quote the excellent review by Paris, Kornprobst, Tumblin and Durand [PKTD] to give a brief overview of the bilateral grid:



The authors consider the S × R domain [S is the spatial domain and R the range domain] and represent a gray-scale image u as defined on a D grid as a D function Γ by Γ(x, y, z) = u(x, y), 1 if z = u(x, y), (0, 0) otherwise.

(.)

With this representation, they demonstrate that bilateral filtering amounts to convolving Γ with a D Gaussian whose parameters are (σ s , σ s , σ r ) : Γ = Γ * G σs,σs,σr . They show that the bilateral filter output is BF{u}(x, y) = Γ x, y, u(x, y) . This process is illustrated in Figure ..

The acceleration strategy is then to subsample the grid before the application of the Gaussian filter; this step can use a fast box subsampling that does not respect the Shannon condition because of the ensuing low-pass filter. Hence the convolution with the separable kernel is computed using consecutive 3 × 1 kernels on a much small volume, which largely compensates for the cost of subsampling and tri-linear upsampling.



Local histograms Other fast schemes [Por] are based on the interpretation of the bilateral filter as an average of local histograms. Indeed, for uniform spatial kernels, BF can be rewritten FBF loc.hist. (x) = 1 C(x) j h Ω (j)G σr j -u(x) j , (.)

C(x) = j h Ω (j)G σr j -u(x) ,
where j belongs to the discrete intensity range of the input image and h Ω (j) is the local histogram value at pixel x and for intensity j.

Polynomial approximations A last class of fast filters use polynomials range kernels [Por, CSU]. Let's explain this with a trigonometric polynomial. Assume the range kernel has the form

k M σr (t) = M n=-M
α n exp(i2πt) n , (.)

with i 2 = -1. Here, σ r stands for the range parameter of the bilateral filter. Set Ω the neighborhood of the pixel x and G σs the spatial Gaussian kernel with standard-deviation σ s . With such a range kernel, the bilateral filter can be written BF poly. {u}(x)

= 1 K(x) y∈Ω G σs (y) M n=-M α n exp i2nπ u(x -y) -u(x) u(x -y) = 1 K(x) M n=-M
α n exp -i2πnu(x) y∈Ω G σs (y) exp i2πnu(xy) u(xy).

(.)

The decomposition is the same for the normalization factor, K(x) = y∈Ω G σs (y)k M σr u(xy) -u(x) .

(.)

The last equation involves a convolution of the image exp i2πnu(x) u(x) with the spatial Gaussian kernel G σs . In other terms, the bilateral filter is obtained by a series of Gaussian convolutions.

. Chapter : Exposure fusion

In Chapter  we explore an alternative option for contrast enhancement, in which no base and detail decomposition is involved. Exposure Fusion is a high dynamic range imaging technique to fuse a bracketed exposure sequence into a high quality image. We show that one can extend this method to the more general context of improving the overall contrast of any image, turning Exposure Fusion into a new and simple contrast and color enhancement operator. To do so, bracketed images are simulated from a single output and fused by exposure fusion. We demonstrate that the resulting algorithm competes with state of the art retinex methods.

Furthermore, we unveil a serious drawback of the original fusion technique. Indeed, it tends to create, albeit unexpectedly, an output image which dynamic range is higher than any of the input images. This flaw of the method forces either to clip the fused image, thus to loose precious information from the (potentially simulated) bracketed sequence, or to compress the dynamic range, which provokes a loss of contrast with respect to the input images. We show and explain this effect. After careful diagnosis, we arrive at the important and counter-intuitive conclusion that exposure fusion does not have the faculty to reduce the edges' amplitude. The effectively operated tone-mapping is the consequence of two effects: the haloing due to the Laplacian pyramid, and the saturation (i.e., clipping) of the input LDR images of the sequence. This saturation flaw, also present in the introduced simulated exposure fusion, is solved in Chapter .

Furthermore, Chapter  introduces Burt and Adelson's Laplacian pyramid [BA] in the context of tone-mapping; we see in Chapter  that this has been successfully reused in more recent multi-scale base and detail decomposition filters. We now summarize the exposure fusion process.

The exposure fusion This method fuses the best parts of the different images in an input sequence into a high quality image. Three metrics are used to determine what pixels are the best and should be kept in the final result: contrast, saturation and well-exposedness. Each measure is pixel-wise. The first measures the amount of local contrast using a small Laplacian kernel. The second is the standard-deviation of the color channels at each pixel. The last one measures the distance to the mean value 0.5. Those three values are multiplied, then normalized so that the weight maps of all images in the input sequence sum to one. Rather than directly fusing the images using these weiht maps, the authors propose a multiscale fusion, using the method introduced by Ogden et al. [OABB]. This technique builds the Laplacian pyramid [BA] of the output image by blending the Laplacian pyramids of the input images according to the Gaussian pyramid of the weight maps. We will denote Lpyr{u} the Laplacian pyramid of the input image u, Gpyr{w} the Gaussian pyramid of the weights, and the scale. The blending operation is then

Lpyr{v, }(x) = N k=1 Gpyr{ w, } k (x)Lpyr{u, } k (x).
(.)

The fused image is obtained by collapsing the constructed pyramid Lpyr{v}.

Starting from this fusion method, our single image contrast enhancement algorithm consists in the simulation of the bracketed exposure sequence, which is then merged using Equation (.), as shown in Figure .. We call this the simulated exposure fusion.

Saturation in the exposure fusion

As we prove, saturation occurs in the original methods by Mertens et al.. Even though weights are normalized and none of the input images exceed the final dynamic range, the fused image can inherit a larger dynamic range than any of the input images. We illustrate this phenomenon using the authors' demonstration image in All input images are in the correct dynamic range. The fused result however has a greater dynamic. The experiment is carried out with gray levels images for the sake of clarity; we thus do not use the saturation metric: ωs = 0. The other parameters are ωc = 1, ωe = 1. We clipped out-of-range values in (d).

The original exposure fusion method [MKR] simply clips the values that exceed the dynamic range, but this results in saturated areas in the final image. One can obviously shift and scale the intensity to make them fit the output dynamic range, but then incurs into the risk of loosing part of the contrast gains on the input images. We are then stuck in the unpleasant situation where either we decide to compress the dynamic, but lose contrast, or we apply again a tone-mapping operator, which is what our method was initially designed for! We present in Chapter  a more natural way to avoid this problem.

. Chapter : Edge reduction in the simulated exposure fusion

In Chapter , we improve on the method presented in Chapter  in two ways: first we correct the saturation artifact that we proved to be inherent to the classic exposure fusion method. Second, we propose a smarter way to simulate the bracketed exposure sequence by automatically choosing the number of brightened and darkened images, so that images with unequally distributed histograms between their left and right sides are better enhanced. We uncover a novel artifact of our method, namely the creation of spurious edges in areas with smooth contrast changes (smooth edges). The issue is solved by replacing the sharp threshold (i.e. clipping) in the remapping function by a smoother function. Furthermore, the general algorithm thus designed can be used to improve on itself in the HDR context. In this generalized version of the fusion, additional simulated bracketed images are built from the input, thus yielding a richer choice of contrasts than those provided by the physical brackets.

The proposed method eventually resembles the local Laplacian filter, being also a multi-scale edge-aware smoothing filters. The similarities and dissimilarities of both filters are discussed in Chapter . Simulated exposure fusion method For the Laplacian pyramid blending is not capable of dynamic range compression, the solution to the out-of-range artifact of the simulated exposure fusion method is to reduce the dynamic range of the input images. We thus design remapping functions that improve the contrast where needed while keeping the overall dynamic of the simulated exposure in a smaller range than the output one. We show in [PHK]. Local Laplacian filters could roughly be explained either as a single image exposure fusion algorithm similar to the method described in Chapter , or as a multi-scale unnormalized bilateral filter. The latter interpretation was given by Aubry et al. in their analysis of the filter, where they made the link with the bilateral filter and the multi-scale version of the anisotropic diffusion [Ela, BC].

The local Laplacian filter (LLF) is versatile and can be used for a wide variety of contrast  manipulations tasks, ranging from edge-aware smoothing to local contrast enhancement with dynamic reduction. Unlike most filters, LLF constructs the Laplacian pyramid of the output image; a final operation collapses the pyramid and builds the filtered image. Each Laplacian coefficient is computed independently using a dedicated remapping function, which shape is chosen in function of the application. The fast version (FLL) uses the Durand-Dorsey [DD] slicing strategy. It greatly speeds up the execution by computing only a reduced number of remapped images.

In Chapter , we first expound the local Laplacian filters and their fast approximation. Then, we show their strong connection with the exposure fusion method [MKR, MKVR]. We see that a fast local Laplacian filter can be computed using the exposure fusion framework with very little difference in the final result. Finally, we describe the artifacts of these filters. Indeed, although they have proven to be one of the best suited filters for base plus detail decomposition for contrast manipulation, the local Laplacian filters have some drawbacks, the major ones being a loss of translation-invariance and luminance halos. We now proceed to give a formal definition of these filters.

The fast local Laplacian filter output pyramid

We have seen in Equation (.) that the output Laplacian pyramid Lpyr{v ef } of the exposure fusion (EF) method is a weighted combination of the Laplacian pyramids of the K images u k of the bracketed exposure input sequence. In FLL, interpolation weights are computed at each scale and for each remapped images. They can be pre-computed too. Denoting A i the interpolation weight pyramid associated with the remapped image u i , we have

Lpyr{v fll , }(x) = S i=1 A i (l, x)Lpyr{u i , }(x). (.)
This shows structure similarities of FLL and EF: both blend a sequence of images according to contrast weights.

Similarity with the exposure fusion

The fast local Laplacian filter and Exposure fusion can be written in an extremely similar way. But are they equivalent? Although FLL does not use as input a sequence of images, it actually generates several images from the input, and merges different pieces of the latter using Laplacian pyramid decompositions. More precisely, FLL needs no quality measurement, because it knows which intensity band has been corrected and therefore must be retained for the final image. As in EF, FLL constructs the Laplacian pyramid of the final image. A significant difference, however, is that local Laplacian filters recompute the weight maps at each scale, while EF calculates them only at the finest scale and then subsamples them.

In Figure . we examine the difference between filtering results of EF and FLL's weighting methods. Put another way, we try to reproduce the output of FLL with EF. In order to do so, we generated K images with the remapping functions of FLL and fused them with weights constructed as in FLL. We shall denote this modified EF version by ẼF. Finally the only difference between ẼF and F LL are the weights in the multi-scale blending. The resulting processed images are visually very similar, but not identical. There are more low frequency halos in the FLL result. We measured for this experiment a psnr (peak signal-to-noise ratio) of 40dB between both results, meaning that they are very similar indeed.

Translation invariance in the local Laplacian filter Due to the Laplacian pyramid, FLL is not translation invariant. We realized the following experiment: a test-pattern was constructed using a single line repeated several times to make it two-dimensional, this test-pattern is denoted #0. This test-pattern was then shifted by one pixel to the right, we denote it #1. Figure . displays these test-pattern in blue in the plots (b) and (c). We filtered these test-pattern with LLF and  superimposed in red the result on the input image. Noticeably, the red lines in (b) and (c) differ, which evidences that the filter is not translation invariant. We show in Chapter  that the meanshift an low-frequency oscillations visible in this experiment are additional symptoms of the loss of translation invariance caused by the downsampling.

. Chapter : Compact formula and scale-space local Laplacian filter

In Chapter , we presented the local Laplacian filter (LLF) and scanned its structural analogy with exposure fusion [MKR, MKVR]. We showed that despite some excellent results, LLF suffers from three artifacts, namely, its lack of translation-invariance, its luminance halos and a slight staircase effect. The lack of translation-invariance is particularly annoying because it creates irregularities, small bounces and a mean-shift. Fortunately, all of these issues are solved in Chapter   by our scale-space local Laplacian filter. We start by dissecting the local Laplacian filter and proposing a compact formula by reformulating the local Laplacian filter in a scale-space setting. This amounts to removing the downsampling and upsampling steps of the original filter. Besides giving a clean mathematical description of the filter, a welcome outcome of this re-interpretation of the filter is the reinstatement of the translation invariance property which LLF lacked. Furthermore, this interpretation puts in evidence the implicit guide used in LLF ; this guide, that we call oracle, can then be replaced by the result of an arbitrary previous filter. We therefore explore the influence of the oracle in this new framework. We show that edge-aware smoothing filters used as oracle reduce the luminance halo but increase the staircase effect, while a simple Gaussian filtered oracle (as used in the original filter) has no staircase effect but sometimes visible luminance halos. We finally compare the results of this extended scale-space local Laplacian filter with the standard local Laplacian filter in the context of base plus detail image decomposition.

A compact formula

The scale-space local Laplacian filter (SLF) has a compact formulation that fully describes the filter. Denoting D = G σ -G σ +1 the difference-of-Gaussian operator and g(x, ) = G σ * u (x) the reference intensity in r the remapping function we get

SLF{u}(x) = max-1 =0 D * r u -g(x, ) (x) + G σ max * u (x).
(.)

Second compact formula using slicing We can express the exact scale-space local Laplacian filter using the "sliced" formulation of the bilateral filter introduced by Durand and Dorsey [DD]. This actually completes our previous expression of SLF by providing another compact and insightful formulation.

   SLF{u}(x) = max-1 =0 ṽ x, , (G σ * u)(x) + (G σ max * u)(x) ṽ(x, , g) = (G σ -G σ +1 ) * r(u -g) (x) (.)
In this equation, ṽ is what we could call a "Laplacian layer": Laplacian coefficients at scale of the remapped input image according to the reference intensity g. The output image SLF{u} is constructed from these layers, by selecting at each pixel the Laplacian coefficients in a particular layer, depending on the value (G σ * u)(x). This value acts as a guide indicating for each pixel and each scale how the input image should be remapped to get the enhanced final result. Put another way, the guide (G σ * u) is used to pick the value of the Laplacian coefficient in the "right layer".

There are as many layers as the number of possible intensities for the guide, and constructing SLF amounts to collect the "correct" values in the pre-computed layers. The fast approximation consists in pre-computing only a reduced set of Laplacian layers, and, for values of the guide that have no pre-computed layer, to linearly interpolate between the two closest pre-computed values.

As we shall see in Section ., this guide is implicit in the original local Laplacian filters, whereas our scale-space interpretation reveals its presence and allows its replacement.

Implicit oracle-based single-scale filter

In [APH + ], Aubry et al. introduced the unnormalized bilateral filter (UBF), given in Equation (.). They show that this is the filter used in LLF when the pyramid has only one scale, i.e., the finest ones and the residual. Thanks to the scale-space interpretation of the filter, we can define the filter that is used in SLF for an arbitrary pyramid depth. We call this filter the unnormalized oracle-based bilateral filter (UOBF), because it uses an oracle, in a similar way to the joint image in CBF. The unnormalized bilateral filter is a spacial case of UOBF where the oracle is the input itself. It is defined as:

UOBF{u, v}(x) = v(x) + y G σs (x -y)G σr u(y) -v(x) u(y) -v(x) . (.)
We call v the oracle because it is the value that controls, for each pixel x, whether a pixel y in its neighborhood will participate a lot in the computation of the result or not. We shall explore in Section . the different filters and the improvements we can derive from the replacement of this oracle by more sophisticated ones. This leads to a third and last compact formulation of SLF:

SLF{u}(x) = u(x) - max-1 =0 UOBF σ {u, g(x, )}(x) -UOBF σ +1 {u, g(x, )}(x) , (.)
in which it becomes clear that the local Laplacian filter belongs to the bilateral pyramid family. Indeed in Equation (.), the right-hand term collapses a pyramid made of bilateral filters. We display in Figure . the application of the new oracle-based filter UOBF on a noisy edge.

Edge-aware oracles in SLF The oracle used in SLF is g(x, ) = G σ * u. We show in Figure . that it can be replaced by other filters, and in particular edge-preserving ones. This helps reducing the luminance halo but increases staircasing.

. Chapter : Weighted least squares filter

We have presented in Chapter  and Chapter  two multi-scale approaches based on the Laplacian pyramid of Burt and Adelson [BA]. We present in Chapter  another effective multi-scale filter, the weighted least squares filter (WLS). It was proposed by Farbman three years before in  [FFLS] and does not involve pyramids. Unlike other schemes based on the bilateral filter [FAR,CPD], this edge-preserving smoothing approach is grounded on the weighted least squares optimization framework. It is defined as the minimization of an energy composed of a data term that minimizes the distance between the  input image and the result, and of a regularization term that penalizes the gradients of the output, except across significant gradients of the input image. Hence, the resulting image follows the input image on its edges and is smoothed elsewhere. The authors proposed two different strategies to build a multi-scale edge-preserving decomposition of an image on this concept. We show that WLS has objectionable artifacts. The most serious is the compartmentalization effect, that breaks the homogeneity of flat regions when they are split in smaller regions with different areas (e.g. branches of a tree with uniform sky as background). The second one is an asymmetric halo. We present two ways to correct these artifacts.

The first proposed solution remedies to compartmentalization by adding in the functional "remote gradients" terms, so that disconnected regions with similar values are linked and move together. The second solution directly prevents important intensity shifts in flat regions. This works well because these are the places where compartmentalization is mostly visible.

Despite our findings and the improvements, we conclude that this filter is not well adapted to contrast enhancement. Indeed, it remains heavy in terms of memory usage and not computationally efficient. Furthermore, our corrections add to its complexity. Nevertheless, for applications on small images or for which computational time is not an issue, one can find in this chapter new good options for an additive base and detail decomposition.

Filter definition As presented by Farbman et al. [FFLS], given the input image u, edge-aware filtering consist in seeking a new image v, which, on the one hand, is as close as possible to u, and, at the same time, is as smooth as possible everywhere, except across significant gradients in u. This translates into seeking the minimum of

arg min v x v(x) -u(x) 2 + λ a x (u, x) ∂v ∂x 2 (x) + a y (u, x) ∂v ∂y 2 (x) , (.)
where x denotes the spatial location of a pixel. The data term minimizes the distance between v and u, while the regularization term strives to achieve smoothness by minimizing the partial derivatives of v. The smoothness requirement is enforced in a spatially varying manner via the  smoothness weights a x and a y , which depend on u, and are defined as

a x (u, x) = ∂ ∂x (x) α + -1 and a y (u, x) = ∂ ∂y (x) α + -1 , (.)
where the image is the log-luminance channel of u.

The compartmentalization artifact

The WLS filter has two noticeable artifacts. The first one is compartmentalization, which happens when a large region with a constant intensity (e.g. the sky) is compartmented by a thin network in the foreground, typically the branches of a tree. This creates small regions with the same constant intensity as the underlying large region, yet disconnected. Because WLS takes into account the direct neighbors only, these small regions are then free to evolve independently. The smaller their area, the lower the steadying influence of the data term compared to the gain obtained by reducing the gradients at the edge of that element. Thus, the lower its area/perimeter ratio, the stronger a small region will affected. Obviously this effect increases with λ. In Figure ., this compartmentalization occurs with the sky fragments between tree's branches, that become brighter than the rest of the sky. The second artifact is a luminance halo on the dark side of the edges only. This is also visible in Figure . at the horizon.

Super-connected WLS filter Among other modifications of the energy, we propose to solve the compartmentalization artifact by adding "remote gradients". They are intensity differences computed with pixels which are not direct neighbors of x. This ensure that close yet separated elements with the same intensity will move together. We can write the new regularization term

λ nmax n=1 a x (u, x, n) (v(x + nv x ) -v(x)) 2 + nmax n=1 a y (u, x, n) (v(x + nv y ) -v(x)) 2 ,
where we consider n max neighbors (in the original WLS filter, n max = 1), v x = (0, 1) and v y = (1, 0) are unit vectors, and the smoothness coefficients in directions x and y become

a x (u, x, n) = |u(x + nv x ) -u(x)| 2 + -1 , (.) a y (u, x, n) = |u(x + nv y ) -u(x)| 2 + -1 . (.)  (

. Chapter : Multi-scale guided filter

In the previous Chapter , we looked at the weighted least squares filter, which proposes a multiscale decomposition of an image by successive applications of the filter without downsampling, similarly to previous multi-scale decomposition based on the bilateral filter -apart from the local Laplacian filter. This last filter and the one we introduce here are based on the local Laplacian pyramid.

In this chapter, we propose a simple multi-scale implementation of the guided filter based upon the Laplacian pyramid of an image. As we shall see, a straightforward implementation leads to the creation of the dark halo artifact, typical of the multi-scale filters based on the Laplacian pyramid. We encountered the same artifact in the multi-scale bilateral filter with regression, described in Chapter . We show that a simple modification in the pyramid reconstruction solves the problem. This correction takes advantage of the guided filter linear model. It leads to a fast filter giving a very clean base and detail decomposition. The comparison we carried out on thirteen filters in Chapter  showed that this filter is effectively one of the best options available.

The mutli-scale guided filter is a direct transposition of the mutli-scale bilateral filter with regression (see Section .) where BFR have been replaced by the guided filter. Indeed, we observe that the guided filter gives a direct measure of the edge reduction with its coefficient ā. Since the dark halo artifact is created when an edge is reduced but the corresponding Laplacian coeffcients in the next finer scale are not, a simple correction is to apply the same coefficient ā to them. Including this modification, the multi-scale guided filter (MGF) is defined as

z max = GF{Lpyr{u, max }} z = GF{Upsample(z +1 ) + Upsample(ā +1 )Lpyr{u, }}, (.)
where ā +1 is the guided filter's coefficient at scale + 1. The final result is given by z 0 . We display in Figure . the detail layer produced with this filter.

. Chapter : Final evaluation and comparison of the filters

At this stage of the dissertation, we have presented and examined the virtues and defects of the most prominent existing filters, and proposed several new ones. From the bilateral filters, in Chapter ,  and , to the local Laplacian filter in Chapter  and , passing by the guided filter, Chapter ,  and , the weighted least squares filter in Chapter , the exposure fusion in Chapter  and , we explored a large part of the literature on the edge-aware smoothing filter,



Luminance halo

Staircase effect Compartmentalization

Contrast halo Dark halo Figure 1.18: Five final test-patterns used to measure the presence of the different artifacts: the luminance halo, the staircase effect, the compartmentalization, the contrast halo and the dark halo.

concentrating on the ones causing the least artifact and, is possible, low computation needs. Furthermore, for each studied filter, we named and defined its most cruel defects and proposed at least one alternative version diminishing these artifacts.

In Chapter , we compare the filters that we presented in the previous chapters. We perform a quantitative evaluation of the five main artifacts of the contrast enhancement we met during this thorough review, namely, the simple (luminance) halo, the contrast halo, the staircase effect (edge sharpening), the compartmentalization (closing effect) and the dark halo (described in Chapter , seen in Chapters  and ). For each of these artifacts we propose a test-pattern specifically designed to reveal it, along with a way of quantifying it. This evaluation gives a clear overview of the capacity of the tested filter to perform a clean base and detail decomposition. Based on the proposed measures, we eventually propose a ranking of twelve representative filters in the literature along with those proposed in this thesis. However, not all contrast enhancement filters are based on base-detail decomposition. For the sake of completeness, additional comparisons are provided with well established tone-mapping filters that do not perform this decomposition e.g., multiscale retinex (MSR), automatic color correction (ACE) and simpler methods based on histogram equalization.

Methodology for the artifact evaluation

We first design a set of five test-patterns, one for each identified artifact. Each one is paired with a measurement method giving a quantitative evaluation of the presence of the artifact for each filter. Then, we establish of short list of filters, that we believe to be representative of the variety of filters proposed in the literature. In order to fairly compare the filters, we propose a rule to set their parameters. To that aim we develop a general procedure, based on the average L 2 norm for the detail of a small set of representative natural images. Once these preparing steps are accomplished, we evaluate the presence of artifacts for the twelve selected filters. This study yields an objective filter ranking, and leads to declare winners three filters achieving a clean base and detail decomposition. We apply those filters on natural images and confirm the ranking. We now describe summarily these filters.



. Synopsis of analyzed filters, contributions

We just presented a synthesis of the dissertation, but not of its conclusions (which are revealed in Chapter ). Since our above synthesis is nothing but short, and the dissertation considerably longer, we feel compelled to present a synopsis of the filters and of our contributions on their understanding and improvement.

List of filters in order of apparition, of their abbreviations, and our contribution All filters mentioned below are formally defined in the thesis, their artifacts are identified and they are compared in the final contest; quantitatively if a they give a base + detail, and qualitatively otherwise. For most of them we propose improved variants.

• MSR (mutli-scale retinex): defined and compared in the final contest;

• ACE (automatic contrast enhancement): defined, compared in the final contest, a formal relation to the bilateral filter established;

• GF (guided filter): defined, compared in the final contest, leads to uncover the contrast halo, compared to the bilateral filter, improved with a multi-scale scheme;

• IGF (iterated guided filter): proposed a new filter, compared in the final contest, analyzed and linked to the Perona-Malik anisotropic diffusion;

• BF (bilateral filter), and variants: defined, compared in the final contest, leads to define the staircase effect, reviewed its variants and the staircase effect corrections;

• FBF (fast bilateral filters): reviewed the fast approximations of the bilateral filters, proposed a fast bilateral filter with regression, proposed a multi-scale implementation of the same filter, leads to define the dark halo artifact;

• EF (exposure fusion): defined, leads to uncover its out-of-range effect, identified the core principle in the contrast manipulation;

• SEF (simulated exposure fusion): proposed an extension of EF to single-image contrast enhancement, compared in the final contest;

• LLF (local Laplacian filter): defined, compared in the final contest, linked to the exposure fusion, explored the different and undesired effects of the pyramidal structure;

• SLF (scale-space local Laplacian filter): proposed a new filter, introduced a compact formulation, introduced the oracle-based unnormalized bilateral filter UOBF implicitly used and extended Aubry's LLF analysis and link to the bilateral filter;

• WLS (weighted least squares filter): defined, compared in the final contest, leads to define the compartmentalization artifact, improved in two new filters, one that penalizes gradients at a greater distance, one that detects and preserves the flat areas;

• BGRF (bilateral grain filter): proposed a new filter based on the morphological grain filter, compared in the final contest;

• DT (domain transform): defined, compared to the bilateral filter and compared in the final contest;

• L 0 -IS (L 0 image smoothing): defined and compared in the final contest.

 1 Introduction en français

Nous donnons dans ce chapitre une traduction des premières sections de l'introduction en anglais du Chapitre . Cette thèse CIFRE a été réalisée en collaboration entre le Centre de Mathématiques et de leurs Applications de l'École Normale Supérieure Paris-Saclay et la société DxO, où j'ai travaillé avec l'équipe de traitement d'images sur le logiciel de développement de photos DxO Photolab  (anciennement DxO Optics Pro). L'équipe travaille à produire des images de grande qualité à partir d'images RAW, mais aussi à partir de fichiers JPEG produits par n'importe quelle caméra. Dans ce contexte, il a été observé qu'il est souvent nécessaire de décomposer une image dans ce que nous appelons intuitivement la base et le détail.

L'objet de la thèse est la décomposition additive automatique des images numériques en couches de base et de détail, avec comme but la manipulation du contraste local. Cette opération vise à ajouter plus de clarté à l'image en améliorant ses détails. Ce problème est directement lié à la théorie dite retinex. Initialement proposée dans les années soixante-dix comme théorie de la perception humaine de la couleur, cette théorie a ensuite été utilisée pour améliorer les images numériques. Dans ce contexte, les algorithmes d'amélioration retinex tentent de transformer les images numériques de sorte que le résultat soit proche de ce qu'un observateur humain aurait vu en regardant la scène originale. Cet objectif a souvent été simplifié comme "voir dans les ombres".

Les opérateurs de tone mapping (mise en correspondance des tonalités ou mappage des tons en français) appartiennent également à cette catégorie. Le problème du tone mapping a les objectifs contradictoires d'en même temps réduire la dynamique d'une image et de préserver le contraste local. Une telle opération est nécessaire dans l'imagerie à grande gamme dynamique (HDR), où la plage dynamique d'une image doit être réduite avant l'affichage ou l'impression (en raison de la faible plage dynamique des écrans et des imprimantes standard). Les opérateurs retinex et de tone  Voir http://www.dxo.com/us/photography/photo-software/dxo-photolab  mapping peuvent être divisés en deux catégories : ceux qui effectuent une décomposition en base et détail ; ceux qui ne le font pas et produisent directement une image améliorée.

L'outil le plus simple disponible est la combinaison d'un filtre passe-bas et passe-haut, qui décomposent l'image en son contenu de basse fréquences (base) et de haute fréquences (détail). Ceci est utilisé par exemple dans la technique de rehaussement de contraste local unsharp mask, qui peut être calculé avec la transformée de Fourier. Les transformées en ondelettes localisent l'analyse fréquentielle dans l'image et peuvent ainsi être utilisées aussi. Les filtres morphologiques comme le filtre de grains et les filtres de surface sont une autre classe de filtre qui peut être utilisée pour cette décomposition. La fermeture, l'ouverture (utilisée dans le top hat filter par exemple) ou le filtre médian sont une autre option. Les filtres EDP comme la diffusion anisotropique sont une autre option classique pour calculer une base. Ils ont l'objectif de simultanément lisser et améliorer l'image (coherence-enhancing diffusion filtering par exemple). Les filtres de débruitage peuvent également être considérés comme des méthodes de décomposition d'une image en base et en détail, l'écart type du bruit jouant alors le rôle d'un paramètre d'échelle. La base est l'image filtrée tandis que le détail correspond au bruit supprimé. C'est le cas du filtre bilatéral, dont l'utilisation pour la décomposition de base et de détail est très répandue, mais qui à l'origine a été conçu pour le débruitage d'image. La régularisation par la variation totale est également conçue à l'origine comme un algorithme de débruitage mais s'adapte parfaitement à une séparation basedétail, alors souvent appelée cartoon + texture.

En bref, il existe une vaste panoplie de filtres d'image qui peuvent être utilisés dans le but de décomposer une image en base et en détail. Dans cette thèse, notre objectif est de passer en revue les méthodes de décomposition les plus pertinentes, de trouver et d'améliorer les meilleures et éventuellement d'en définir de nouvelles. Cela nécessite des mesures rigoureuses pour évaluer la qualité des résultats. Comme nous le verrons, nous serons amenés à mesurer les différents artefacts produits par chaque sorte de filtre.

La difficulté du problème réside dans notre notion de "base" et de "détail". En effet, alors que le filtrage linéaire les lisserait, notre notion de base conserve les contours principaux dans la base et les exclut du détail. Ainsi, une telle décomposition est à la fois additive et non linéaire. Notre méthodologie de recherche consiste à comprendre, améliorer et évaluer les filtres de lissage qui préservent les contours, c'est-à-dire les filtres qui calculent une base. Au cours de l'étude, nous allons définir les artefacts, spécifiques à un filtre ou, plus souvent, typiques d'une classe de filtres.

Nous baserons notre définition des artefacts sur l'évaluation qualitative des experts en image de DxO, que nous visons à transformer en mesures quantitatives rigoureuses. Ces évaluations sont hautement non-linéaires. Nous essayons d'abord systématiquement dans cette thèse de corriger les artefacts dévoilés pour chaque filtre. Notamment, aucun filtre n'est réellement exempt des artefacts, tels que nous allons les définir. Cependant, les artefacts ne sont pas tous également gênants du point de vue d'un photographe, et la présence de chaque défaut peut varier, de sorte que beaucoup d'artefacts peuvent tomber au-dessous d'un seuil subjectif "inacceptable" (trop visible pour être acceptable).

Nous sélectionnons finalement les algorithmes qui offrent le meilleur compromis parmi ces artefacts, grâce à une mesure quantitative réalisée sur les artefacts que nous avons isolés. Dans notre classement final, nous prenons en compte la complexité de chaque filtre. En effet, ce paramètre, bien que souvent en contradiction avec la qualité de la décomposition, peut être décisif lorsqu'il s'agit de sélectionner un filtre dans une chaîne de traitement d'images déjà longue et complexe.

En bref, cette thèse développe une méthodologie pour l'évaluation quantitative de la qualité des décompositions en base et détail de tout filtre d'image. Après un examen attentif de nombreux filtres et de leurs artefacts, nous finissons par créer un ensemble de mires, une pour chacun des cinq artefacts identifiés, et cinq métriques associées aux mires proposées. La méthode prend en entrée n'importe quel filtre avec ses paramètres fixes, à l'exception de celui qui contrôle la quan- tité de détails extraite par l'algorithme. Ce dernier paramètre est fixé de façon que la norme L 2 du détail produit corresponde à un nombre prédéterminé. La valeur de cette norme L 2 est en fait une moyenne des valeurs des normes L 2 du détail obtenues avec un ensemble représentatif d'images naturelles. L'égalisation des normes L 2 du détail extrait par chaque filtre assure que les algorithmes peuvent être comparés équitablement. Cela conduit à évaluer quantitativement les cinq mesures d'artefacts pour tous les filtres sur toutes les mires et à proposer finalement une méthode de classement ainsi qu'un classement final de tous les filtres examinés. Comme nous le verrons, deux filtres classiques -mais améliorés par nous -émergent de cette étude.



. Les méthodes retinex

La théorie Retinex a été formulée pour la première fois par Edwin H. Land en  dans [Lan]. C'était une tentative révolutionnaire de modéliser comment le système visuel humain (SVH) perçoit les couleurs dans une scène. Cette théorie a été formalisée par Land et McCann [LM]. Ils ont établi que le système visuel ne perçoit pas une luminosité absolue mais plutôt une luminosité relative, à savoir, les variations de luminosité dans des régions locales de l'image. Cela a été prouvé par les expériences utilisant des formes "de Mondrian" [Lan, Lan], où ils montrèrent que la sensation de couleur n'est pas directement liée aux caractéristiques spectrales du signal perçu : les patchs de réflectances différentes sont perçus avec des couleurs différentes même lorsqu'ils émettent la même distribution spectrale de lumière à cause d'un changement local dans l'illumination. C'est ce que A. Rizzi et al. appellent color constancy (la constance de la couleur) [RM].

Dans ses premiers résultats, Land a supposé que trois ensembles indépendants de récepteurs existent et que la comparaison de ces trois sorties de récepteurs donne le sens de la couleur. Il a appelé ce système Retinex, un néologisme fait de rétine et de cortex. Bien que le travail original n'implique pas d'images numériques, Retinex peut être utilisé pour améliorer les images numériques, comme suggéré par Land lui-même.

Les implémentations et dérivations de Retinex ont été un domaine de recherche actif qui compte maintenant une multitude de publications. Comme expliqué dans un récent aperçu des méthodes Retinex par Petro et al. [PSM], les nombreuses implémentations peuvent être divisées en deux groupes. Le premier groupe explore la luminosité relative de l'image en utilisant un grand nombre de chemins dans l'image ou en comparant la couleur du pixel courant à un ensemble de pixels aléatoires [Lan]. Le second groupe utilise un masque de convolution ou des techniques variationnelles pour calculer une image améliorée localement [Lan], [JRWb], [JRWa], [KES + ], [BF], [MPS], [MMOC], [BCP].

De nos jours, l'implementation la plus répandue de Retinex est une alternative à l'algorithme initial par marche aléatoire publié par Land [Lan]. Cette implémentation calcule la luminosité comme le rapport entre la valeur d'un pixel et la valeur moyenne des échantillons environnants. Prenant pour exemple un filtre gaussien G σ , l'opération revient à définir L(x) := I(x)

(I * Gσ)(x) , ce qui implique log L(x) := log I(x) -log(I * G σ )(x). (.)
Cette équation (.) est la méthode dite du Retinex à une seule échelle (single scale retinex ou SSR en anglais), explorée par Jobson et al. dans [JRWb] et plus tard étendue par les mêmes auteurs à plusieurs échelles [JRWa]. Cette dernière est appelée Retinex multi-échelles (multiscale Retinex ou MSR en anglais) et sa formule est :

MSR{u}(x, i) = N n=1 w n SSR{u, n, i}(x) = N n=1 w n log u(x, i) -log (G σn * u(i))(x) , (.)
où N est le nombre d'échelles, w n le poids de chaque échelle et G σn (x) = C n exp(x 2 /2σ 2 n ), un noyau gaussien avec son facteur de normalisation C n . Un excellent aperçu de la théorie et des algorithmes de Retinex peut être trouvé dans le livre de Bertalmío [Ber], ainsi que la connexion aux techniques variationnelles basée sur la perception [PAPBC, FBPC] et ACE, que nous abordons maintenant.

L'Automatic Color Enhancement (ACE, Rehaussement Automatique de Couleur en français) proposé par Gatta et al. [GRM] est fortement lié à Retinex. Il a été développé plus avant



ACE{u}(x) = y∈Ω\x s α u(x) -u(y) x -y , x ∈ Ω, (.) où u : Ω → [0, 1] est l'image d'entrée et s α : [-1, 1] → R est



. Chapitre : Le filtre guidé

Dans le Chapitre , nous commençons notre analyse du filtre guidé [HSTb]. Ses artefacts, un halo de contraste et un halo de luminance, sont expliqués. Une comparaison des performances du filtre est effectuée avec un filtre apparenté, le filtre bilatéral. Nous montrons que les tentatives de mise en correspondance des paramètres des deux filtres sont vaines ; le filtre guidé n'a pas la même capacité de préservation des contours que le filtre bilatéral. Nous présentons cependant dans le Chapitre  un nouveau filtre basé sur GF qui réduit ses artefacts tout en gardant sa propriété très désirable d'être localement une transformation affine de l'image guide, ce qui évite l'effet d'escalier (staircase effect en anglais).

Le filtre guidé Le filtre guidé (guided filter, GF) a deux étapes : la première calcule dans des patchs les coefficients qui minimisent la distance entre une transformation linéaire de l'image guide et l'image d'entrée. Dans chaque fenêtre (patch) ω, GF résout :

E a(y), b(y) = x∈ω(y) a(y)v(x) + b(y) -u(x) 2 + a(y) 2 , (.)
où u est l'image d'entrée, v le guide, un paramètre de lissage et ω une fenêtre. Ce modèle garantit que dans les patchs filtrés, les gradients sont proportionnels aux gradients de l'image guide, ce qui évite l'effet escalier présent dans le filtre bilatéral, par exemple. D'un autre côté, il introduit un halo de contraste et un halo de luminance. La deuxième étape agrège les valeurs filtrées de tous les patches qui se chevauchent. C'est équivalent à la moyenne des coefficients (a, b) des patchs qui se chevauchent, ainsi l'image filtrée s'écrit

GF{u}(x) = ā(x)v(x) + b(x), (.)



Une analyse du filtre guidé avec les équations aux dérivées partielles

Dans [BCM] les auteurs ont prouvé la présence d'un effet d'escalier dans le filtre bilatéral en montrant qu'il est asymptotiquement équivalent à une équation de Perona-Malik contenant un terme d'équation de la chaleur inverse créant des chocs le long des passages par zéro du détecteur de bord de Haralick [Har]. En suivant la même méthodologie, nous prouvons dans le Chapitre  que le filtre guidé est équivalent à une itération d'une équation différentielle partielle de diffusion anisotropique, qui peut être interprétée comme le premier terme, diffusif, d'une équation de Perona-Malik. Cela explique pourquoi le filtre guidé ne montre pas l'artefact d'escalier.

Theorem .. Considérons une image D u(x, y) ∈ C 3 (Ω). Soit f 1 (x, y) un noyau radial nonnégatif de support compact. Nous supposons que le filtre est normalisé, c'est-à-dire que f 1 (x, y)dxdy = 1 ; et qu'il est symétrique, c'est-à-dire xf 1 (x, y)dx = yf 1 (x, y)dy = 0. Nous posons ˜ = /M 20 , où est le paramètre de préservation des contours du filtre guidé, et

M 20 = f 1 (x, y)x 2 dxdy = f 1 (x, y)y 2 dxdy. Pour finir, soit f σ le noyau mis à l'échelle : f σ (x, y) = σ -2 f 1 (x/σ, y/σ). Alors, pour (x, y) ∈ Ω, GF σ {u}(x, y) -u(x, y) = σ 2 M 20 ˜ |∇u(x, y)| 2 + ˜ ∆u(x, y) + O(σ 3 ) . (.) Remark ..
˜ = |∇u(p)| 2 . . if ˜ |∇u(x, y)| 2 , GF σ {u}(x, y) -u(x, y) = σ 2 M 20 ˜ |∇u(x, y)| 2 ∆u(x, y) + O(σ 3 ) ; . if ˜ = |∇u(x, y)| 2 , GF σ {u}(x, y) -u(x, y) = σ 2 M 20 2 ∆u(x, y) + O(σ 3 ) ; . if ˜ |∇u(x, y)| 2 , GF σ {u}(x, y) -u(x, y) = σ 2 M 20 ∆u(x, y) + O(σ 3 ) .
Suite à l'interprétation de ce théorème, nous implémentons un filtre guidé itéré avec un petit rayon qui simule cette équation et prouve qu'il est sans halo. Ce filtre peut être simplement écrit IGF σ {u}(t, x) = āσ (t, x)IGF (1) σ {u}(t - Ce chapitre, avec les deux suivants sur la correction de l'effet d'escalier et les approximations rapides du filtre bilatéral, est inspiré par le livre de S. Paris, P. Kornprobst, J. Tumblin et F. Durand [PKTD]. Alors que ce livre vise à donner une présentation détaillée du filtre bilatéral et de ses applications, nous nous concentrons sur son utilisation pour la décomposition en base et détail. Néanmoins, nous abordons plusieurs points déjà passés en revue dans le livre, e.g., les différentes extensions proposées et ses approximations rapides. Nous soulignons ci-dessous les principales différences entre nos Chapitres , ,  et le livre de Paris, Kornprobst, Tumblin et Durand. Concernant ce chapitre sur les filtres bilatéraux, nous présentons des filtres supplémentaires :

• le filtre bilatéral non normalisé [APH + , MT], avec ses approximations rapides ;

• nous proposons une approximation rapide pour le filtre bilatéral avec regression ;

• nous faisons le lien entre le filtre ACE (Automatic Color Enhancement, rehaussement automatique de couleur), et le filtre bilatéral.

Nous poursuivons la revue du filtre bilatéral avec les corrections de l'effet d'escalier dans le Chapitre .

Il existe deux types de corrections : la première modifie le filtre bilatéral de sorte que les pentes soient prises en compte, par exemple le filtre bilatéral avec régression, le filtre trilatéral, le filtre bilatéral symétrique ; ceux-ci ont été revus dans le livre de Paris et al.. Les différences entre cette partie et le livre se résument donc à :

• une description plus détaillée du filtre trilatéral, avec des pseudo-codes ;

• l'introduction d'un filtre bilatéral symétrique similaire à celui proposé par Elad [Ela].

Le deuxième type d'approximations n'est toutefois pas décrit dans le livre [PKTD]. Il consiste à traiter l'image déjà filtrée pour corriger l'artefact d'escalier. Les corrections décrites sont :

• le mélange (blend) décrit par Durand et Dorsey [DD] ;

• l'effect de lissage isotropique minimal dans filtre bilatéral avec l'approximation du noyau séparable [PVV] ;

• la correction de Poisson proposée par Bae et al. dans [BPD] ;



• la diffusion sélective (selective diffusion) de Kass et Solomon [KS].

En ce qui concerne les approximations rapides, la plupart d'entre elles sont examinées dans le livre. Néanmoins, nous ajoutons des filtres postérieurs à  à la liste, et donnons parfois des descriptions plus détaillées. Notamment :

• dans les histogrammes locaux, l'approximation de Weiss [Wei] est décrite dans le livre, mais nous en donnons une description plus détaillée : nous présentons l'algorithme de Huang antérieur et donnons pour les deux les pseudo-codes. En outre, nous passons en revue la version de Porikli qui utilise des histogrammes intégraux, et discutons de l'utilisation des noyaux spatiaux carrés ;

• l'approximation rapide du filtre bilatéral non normalisé est donnée dans le Chapitre  ;

• nous présentons une classe supplémentaire d'approximations rapides basées sur l'utilisation de noyaux d'intensité polynomiaux ;

• la domain transform [GO] est également revue, ce filtre peut être vu comme un filtre bilatéral quand il est utilisé avec un petit noyau spatial.

Le filtre bilatéral Le principe du filtrage bilatéral est apparu avec Yaroslavsky () [Yar] et Lee () [Lee]. La variante que nous étudions a été proposée par Smith et Brady qui l'ont appelée "SUSAN" () [SB]. Il a été re-proposé par Tomasi et Manduchi sous le nom de "filtre bilatéral" en  [TM]. Puisque cet artefact est particulièrement gênant dans les méthodes de manipulation de contraste, de nombreux auteurs ont essayé de le corriger. Les solutions peuvent être divisées en deux catégories. La première catégorie de correction ne modifie pas le filtre, mais corrige l'artefact dans une étape de post-traitement. La seconde modifie directement le filtre pour lui permettre de gérer drectement les pentes. Nous passons en revue dans ce chapitre les deux catégories de corrections.

Ce chapitre a été inspiré par l'excellent ouvrage de Paris et al. sur le filtre bilatéral [PKTD]. Les différences avec notre revue sont mises en évidence dans la précédente Section ..

Exemple: la correction de Durand et Dorsey À la manière de la correction de Durand-Dorsey, la plupart des étapes de post-traitement visant à supprimer l'effet escalier du filtre bilatéral appliquent des filtres gaussiens à l'image lissée bilatéralement. La difficulté est alors de trouver le bon écart-type du filtre et de savoir où l'appliquer.

Durand et Dorsey dans [DD] ont apporté une réponse simple ; ils appliquent seulement un filtre gaussien avec un petit écart-type, puis font un mélange entre cette image floue et la nonfloue en fonction du facteur de normalisation de BF. Soit α le coefficient d'interpolation linéaire entre le résultat du filtre bilatéral FBF{u} et sa version floutée FBF{u}. Ce coefficient varie avec log C. La fonction α = f log(C) est définie comme α(x) = log C(x) /log C max , où C max est la valeur maximale possible pour C, i.e. C max = y G σs (xy). L'image corrigée est alors FBF{u} corr (x) = α(x)FBF{u} + 1 -α(x) FBF{u}(x).

(.)

Une autre correction itérative lisse les résultats bilatéraux, avec des filtres gaussiens de largeur croissante. C'est la diffusion sélective de Kass et Salomon [KS]. À chaque itération, ils choisissent entre l'image avant et après le flou en mesurant localement la distance à l'image originale, et en gardant la plus proche. L'idée est que si flouter le résultat bilatéral le rend plus proche de l'image originale, alors cette version floue devrait être préférée. [DD]. Ils ont présenté l'idée originale, qui sera largement explorée plus tard, d'échantillonnage de la gamme d'intensités afin de linéariser la convolution. La convolution gaussienne peut alors être calculée en utilisant l'un des nombreux algorithmes rapides disponibles. Aucune implementation rapide et exacte du filtre bilatéral n'a encore été proposée. Ainsi, la concurrence entre les nombreux schémas proposés réside non seulement dans la vitesse mais aussi dans la précision et les inévitables artefacts qu'ils introduisent. De plus, pour plusieurs schémas, la vitesse dépend des paramètres utilisés et de la dimension dans laquelle le filtre fonctionne. Ainsi, nous présentons finalement une palette de filtres efficaces plutôt qu'un gagnant définitif.



Ce chapitre a été inspiré par l'excellent ouvrage de Paris et al. sur le filtre bilatéral [PKTD]. Les différences avec notre revue sont mises en évidence dans la précédente Section ..

L'approximation linéaire par morceaux (piecewise-linear BF) Le schéma d'approximation rapide de Durand-Dorsey est basé sur la discrétisation des valeurs possibles de u(x) dans le noyau bilatéral. Considérons l'équation (.) du filtre bilatéral pour un pixel fixe x :

BF{u}(x) = 1 C(x) y∈Ω G σs (x -y)G σr u(y) -u(x) u(
v(x, i) = 1 C(x, i) y∈Ω G σs (x -y)G σr u(y) -γ(i) u(y) (.) = 1 C(x, i) y∈Ω G σs (x -y)H(y, i) (.) et C(x, i) = y∈Ω G σs (x -y)G σr u(y) -γ(i) (.) = y∈Ω G σs (x -y)I(y, i). (.)
Cette formulation du bilatéral est exacte et montre qu'elle peut être calculée par une série de convolutions linéaires : une par valeur possible de u(x). La stratégie d'accélération consiste alors à calculer le résultat exact pour un nombre limité d'intensités seulement. Cela revient à échantillonner la plage d'intensité et à interpoler linéairement entre ces couches v(x, i) et C(x, i) pour les valeurs comprises entre les échantillons.

The bilateral grid La méthode linéaire par morceaux a ensuite été améliorée par Paris et al. [PD,CPD] dans la grille bilatérale. Cette méthode linéarise également la convolution et souséchantillonne le signal pour réduire la complexité de calcul, mais donne également une définition plus formelle de cette approximation rapide grâce à une interprétation dans une dimension plus élevée des images et un gain de précision grâce à un meilleur sous-échantillonnage. Cette approximation est probablement l'une des plus efficaces et l'une des plus représentatives de la littérature sur les filtres bilatéraux rapides. Nous citons l'excellente revue de Paris, Kornprobst, Tumblin et Durand [PKTD] pour donner un bref aperçu de la grille bilatérale :

The authors consider the S × R domain [S is the spatial domain and R the range domain] and represent a gray-scale image u as defined on a D grid as a D function Γ by

Γ(x, y, z) = u(x, y), 1 if z = u(x, y), (0, 0) otherwise. (.)
With this representation, they demonstrate that bilateral filtering amounts to convolving Γ with a D Gaussian whose parameters are (σ s , σ s , σ r ) : Γ = Γ * G σs,σs,σr . They show that the bilateral filter output is BF{u}(x, y) = Γ x, y, u(x, y) . This process is illustrated in Figure ..

La stratégie d'accélération consiste alors à sous-échantillonner la grille avant l'application du filtre gaussien ; cette étape peut utiliser un sous-échantillonnage rapide qui ne respecte pas la condition de Shannon parce qu'il est suivi d'un filtre passe-bas. Par conséquent, la convolution avec le noyau séparable est calculée en utilisant des noyaux 3 × 1 consécutifs sur un très petit volume, ce qui compense largement le coût du sous-échantillonnage et du suréchantillonnage tri-linéaire. Local histograms D'autres schémas rapides [Por] sont basés sur l'interprétation du filtre bilatéral comme moyenne des histogrammes locaux. En effet, pour les noyaux spatiaux uniformes, BF peut être réécrit



FBF loc.hist. (x) = 1 C(x) j h Ω (j)G σr j -u(x) j , (.) C(x) = j h Ω (j)G σr j -u(x) ,
où j appartient à la gamme d'intensités discrètes de l'image d'entrée et h Ω (j) est la valeur de de l'histogramme local au pixel x et pour l'intensité j.

Polynomial approximations Une dernière classe de filtres rapides utilise des noyaux d'intensité polynômiaux [Por, CSU]. Nous l'expliquons ici avec un polynôme trigonométrique. Supposons que le noyau de la gamme a la forme

k M σr (t) = M n=-M α n exp(i2πt) n , (.)  avec i 2 = -1. Ici, σ
= 1 K(x) y∈Ω G σs (y) M n=-M α n exp i2nπ u(x -y) -u(x) u(x -y) = 1 K(x) M n=-M α n exp -i2πnu(x) y∈Ω G σs (y) exp i2πnu(x -y) u(x -y).
(.) La décomposition est la même pour le facteur de normalisation,

K(x) = y∈Ω G σs (y)k M σr u(x -y) -u(x) . (.)
La dernière équation implique une convolution de l'image exp i2πnu(x) u(x) avec le noyau Gaussien spatial G σs . Autrement dit, le filtre bilatéral est obtenu par une série de convolutions gaussiennes.

Nous renvoyons au Chapitre  précédent (introduction en anglais) pour la partie de l'introduction qui concerne :

• la fusion d'exposition et son application à une seule image (Section . et Section .) ;

• l'analyse du local Laplacian filter et son extension (Section . et Section .) ;

• le filtre weighted least squares (Section .) ;

• la comparaison finale des méthodes entres elle (introduite en Section .).

Nous résumons toutefois ci-dessous la liste de nos contributions.



. Synopsis des filtres analysés, contributions

Nous venons de présenter une synthèse de la thèse, mais pas de ses conclusions (qui sont révélées au Chapitre ). Puisque notre synthèse ci-dessus est tout sauf courte, et la thèse considérablement plus longue, nous nous sentons obligés de présenter une synopsis des filtres et de nos contributions à leur compréhension et amélioration.

Liste des filtres dans l'ordre d'apparition, de leurs abréviations, et de notre contribution Tous les filtres mentionnés ci-dessous sont formellement définis dans la thèse, leurs artefacts sont identifiés et comparés dans le concours final ; quantitativement s'ils donnent une décomposition en base + détail, et qualitativement sinon. Pour la plupart d'entre eux, nous proposons des variantes qui les améliorent.

• MSR (mutli-scale retinex) : défini et comparé dans le concours final ;

• ACE (automatic contrast enhancement) : défini, comparé dans le concours final, une relation formelle avec le filtre bilatéral est établie ;

• Guided filter, GF : défini, comparé dans le concours final, conduit à découvrir le halo de contraste, comparé en détail avec le filtre bilatéral, amélioré avec le schema multi-échelle ;

• Iterated Guided filter, IGF : proposition d'un nouveau filtre, comparé dans le concours final, analysé et relié à la diffusion anisotropique de Perona-Malik ;

• bilateral filter, BF, and variants : défini, comparé dans le concours final, conduit à définir l'effet d'escalier, revu en détail ainsi que ses variantes et les corrections de l'effet d'escalier ;

• fast bilateral filters, FBF : revue des approximations rapides du filtre bilatéral, proposition d'un filtre bilatéral avec régression rapide, proposition d'une implémentation multi-échelle de ce même filtre, qui conduit à la découverte et la définition de l'artefact du halo sombre ;

• exposure fusion, EF : défini, conduit à découvrir son artefact de dépassement de la dynamique, identification du principe fondamental pour la manipulation du contraste ;

• simulated exposure fusion, SEF : proposition d'une extension de EF au rehaussement de contraste pour une seule image (SEF), comparé dans le concours final ;

• local Laplacian filter, LLF : défini, comparé dans le concours final, relié à exposure fusion ; exploration des différents effets indésirables de sa structure pyramidale ;

• scale-space local Laplacian filter, SLF : proposition d'un nouveau filtre, introduction d'une formulation compacte, introduction du filtre bilatéral non normalisé basé sur un oracle (UOBF) complétion de l'analys de LLF faite par Aubry et al. ;

• weighted least squares filter, WLS : défini, comparé dans le concours final, conduit à définir l'artefact de cloisonnement, amélioré dans deux nouveaux filtres, l'un pénalisant les gradients à une grande distance et l'autre détectant et préservant les zones plates ;

• bilateral grain filter, BGRF : proposition d'un nouveau filtre basé sur le filtre morphologique de grain, comparé dans le concours final ;

• domain transform, DT : défini, comparé au filtre bilatéral et comparé dans le concours final ;

• L 0 Image Smoothing, L 0 -IS : défini et comparé dans le concours final.

 

In this chapter, we present a thorough description of the guided filter. Its artifacts, a contrast and a luminance halos, are explained. A comparison of the filter's performance is made with the related bilateral filter. We show that attempts to find equivalence between the parameters are vain; the guided filter does not have equivalent edge-preserving capability to the bilateral filter.

We shall however present in the next chapter a new filter based on GF that reduces its artifacts while keeping the very desirable property of being a linear transformation of the guide image in each patch, which avoids the staircase effect.

. Introduction

The Guided Filter (GF) was proposed by K. He, J. Sun and X. Tang in  in "Guided Image Filtering" [HST]. A preliminary conference version of this paper had been published in  [HSTb]. It is closely related to image matting and in particular to the matting Laplacian matrix [LLW], [HSTa]. GF has since been widely used in image processing. The main reason for such a success is that this filter is able to achieve high quality results, remains close to the bilateral filter, while drastically reducing the computational time. The filter was further accelerated in  [HS] by its inventors. It also avoids the appearance of staircase artifacts, also called by the authors "gradient reversal". One can actually view the Guided Filter as a simplified version of the bilateral filter, where the pixel-wise intensity difference weighting is replaced by a global measurement of the pixels intensity variation computed as a local variance. This change speeds up the filter but also causes some "contrast halo artifacts" as we shall see.

The authors of the guided filter described their invention in the following terms.

In this paper, we propose a novel explicit image filter called guided filter. The filtering output is locally a linear transform of the guidance image. On one hand, the guided filter has good edge-preserving smoothing properties like the bilateral filter, but it does not suffer from the gradient reversal artifacts. On the other hand, the guided filter can be used beyond smoothing: With the help of the guidance image, it can make the filtering output more structured and less smoothed than the input.

The Guided Filter has been used in many areas such as: stereo vision, for cost-volume refinement in [TM], stereo-matching in [HBR + ]; high-quality real-time O(N ) stereo matching algorithm [HRB + ], [DMMVC]. It has been used for image matting in [HSTa] and image dehazing algorithms in [HST]. As a base/detail decomposition algorithm it has been used for improvement of the Exposure Fusion [MKVR] algorithm in [SKBa] but also for flash/non-flash image fusion (see e.g. [SM]). Several other applications can benefit from GF: demosaicing in [MTO], optical flow estimation [HRB + ], interactive image segmentation [HRB + ], saliency detection [DXY], and illumination rendering [BEM]. Some generalizations of the guided filters have also been proposed, for example a weighted version of the guided filter [LZZ + ] with adaptive parameter and a "gradient domain" version [KCWL], where the gradients are filtered by the guided filter. A generalization of the guided filter with a "shape-adaptive local support" has also been considered [LSM + ]. In [ZSXJ] (the "Rolling Guidance Filter") the authors introduce an iterative scheme based on the joint bilateral filter where the guide is recursively filtered. They present a version using the guided filter. This last work is related to our proposition of an iterated guided filter. The above mentioned paper [SM] also uses an iterative scheme.



. Guided Filter

Perhaps the most important aspect of the guided filter is the local linear relation that is established between the guidance image v and the output image GF raw {u} in a window ω(y). We use the notation GF raw to denote the first step of the guided filter. At this step, and in each window indepently, the guided filter output is a linear transformation of the guide. For each window ω(y) of radius r (size is (2r + 1) 2 ), we have

GF raw {u}(x) = a(y)v(x) + b(y), ∀x ∈ ω(y) , (.)
where a(y), b(y) are some linear coefficients assumed to be constant in ω(y). This local linear model ensures that GF raw {u} has an edge only if v has an edge, because

∇GF raw {u}(x) = a(y)∇v(x), ∀x ∈ ω(y) . (.)
In each window ω(y), the raw guided filter is the result of fitting a linear model (.) to the input image u by minimizing the cost function

E a(y), b(y) = x∈ω(y) a(y)v(x) + b(y) -u(x) 2 + a(y) 2 . (.)
Here, is a regularization parameter penalizing large values of a(y). The underlying model is a decomposition u(x) = GF raw {u}(x) + n(x) where n is a component such as noise or texture that we want to separate from the base GF raw {u}(x). The minimization of the energy (.) amounts to minimizing the difference between u and the base GF raw {u}, i.e. n, in while maintaining the linear model in (.). Moreover, the parameter penalizes large values of coefficient a and  thus helps removing the small variations in u. Equation (.) is the linear ridge regression model [DS], [FHT] and its solution is given by

a(y) = 1 |ω| x∈ω(y) v(x)u(x) -µ(y)ū(y) σ 2 (y) + , (.) b(y) = ū(y) -a(y)µ(y). (.)
Here, µ(y) and σ 2 (y) are respectively the mean and variance of v in ω(y), |ω| is the number of pixels in ω(y), and ū(y) = 1 |ω| x∈ω(y) u(x) is the mean of u in ω(y). Once the linear coefficients a(y), b(y) have been obtained, the filtering output GF raw {u}(x) can be computed by (.).

Interestingly, the numerator in equation (.) is the empirical covariance between the input image u and the guide v and µ is the empirical variance of v. Thus a(y) and b(y) can be expressed as a(y) = Cov{v, u}(y)

Var{v}(y) + , (.) b(y) = Mean{u}(y) -a(y)Mean{v} , (.)
where Mean denotes the mean in the window ω. However, a pixel x is involved in all the overlapping windows ω(y) containing it. Thus the value of GF raw {u}(x) in (.) varies when computed in different windows. A simple strategy is to average all the possible values of GF raw {u}(x). Thus, after computing (a(y), b(y) for all windows ω(y) in the image, the filter's output is given by

GF {u}(x) = 1 |ω| y|x∈ω(y) a(y)v(x) + b(y) . (.)
Due to the symmetry of the box window, the linear coefficients can be averaged instead, so that

GF {u}(x) = ā(x)v(x) + b(x) (.) with ā(x) = 1 |ω| y∈ω(x) a(y) (.) b(x) = 1 |ω| y∈ω(x) b(y) (.)
where (.) and (.) are the average coefficients of all windows overlapping x.

Considering the modification introduced by (.), GF {u}(x) is no longer a scaling of v(x) in ω(x), because the linear coefficients ā(x), b(x) vary spatially. But as ā(x), b(x) are the output of a mean filter, their gradients can be expected to be much smaller than the gradient of v near strong edges. Thus, we still expect that ∇GF {u} ā∇v, meaning that abrupt intensity changes in v are mostly preserved in GF {u}.

. Variants and their pseudo-code

.. Guided filter

We give in Algorithm  the pseudo-code of the original guided filter. All the operations in the pseudo-code are pixel-wise. The Mean ω operator is the sample mean in a window ω, defined as

Mean ω {v}(y) = 1 |ω| x∈ω(y) v(x) . (.)



The four first lines of the algorithm compute the mean, variance and covariance of each window in images u and v. We obtain the image C of the local covariance between v and u along with the image of the local variance of v.

v ← Mean ω {v} // Empirical mean of v in windows ω  C ← Mean ω {vu} -vū // Empirical covariance of v and u in ω  V ← Mean ω {v 2 } -v2 // Empirical variance of v in windows ω  a ← C/(V + ) // equation (.)  b ← ū -av // equation (.)  ā ← Mean ω {a} // Average overlapping estimators a: equation (.)  b ← Mean ω {b} // Average overlapping estimators b: equation (.)  return GF {u} ← āv + b // equation (.)
The authors of the guided filter [HST] suggest the use of a box filter for the mean computation. It can be implemented with integral images, making the filter O(N) with N the number of pixels in the image.

.. Fast guided filter

A Fast Guided Filter [HS] has more recently been proposed in  by the same authors. It speeds up the filter by making computations on a down-sampled version of the image for the computation of the variance and for the coefficients a, b, ā and b. This reduces the filter complexity to O(N/s 2 ), where s denotes the sub-sampling factor. Indeed, when applied to large images, the guided filter is often used with a large radius r. One can then subsample the images submitted to the mean filter and therefore substantially reduce the amount of memory involved and the required computations. The images used in the algorithm are indeed low-frequency when r is large. This version is described in the pseudo-code Algorithm . This fast guided filter is an approximation using nearest-neighbor or bilinear interpolation for sub-sampling. Yet the results are indistinguishable for large r, for instance when r = 16 and s = 4, the execution time is × smaller according to the authors.

.. Guided filter for color images

The guided filter can be extended to color images. This is useful when an edge to preserve has strong color contrast but light gray level contrast. One can then filter each channel of the input color image u according to the color guide v. The color edges are then transfered to each channel and color edges are well preserved. The color cost function is:

E col a(k), b(k) = x∈ω(y) a(y) T v(x) + b(y) -u(x) 2 + a(
u ↓ ← subsample{u, s}  v ↓ ← subsample{v, s}  r ↓ ← r/s // window ω has size (2r ↓ + 1) 2  ū↓ ← Mean ω {u ↓ } // Empirical mean of u ↓  v↓ ← Mean ω {v ↓ } // Empirical mean of v ↓  cov ↓ ← Mean ω {v ↓ u ↓ } -v↓ ū↓ // Empirical covariance of v ↓ and u ↓  var ↓ ← Mean ω {v ↓2 } -v↓2 // Empirical variance of v ↓  a ↓ ← cov ↓ /(var ↓ + ) // equation (.)  b ↓ ← ū↓ -a ↓ v↓ // equation (.)  ā↓ ← Mean ω {a ↓ } // equation (.)  b↓ ← Mean ω {b ↓ } // equation (.)  ā ← upsample{ā ↓ , s}  b ← upsample{ b↓ , s}  return GF fast {u} ← āv + b // equation (.)
where the bold face is used to denote vectors. Its minimum is obtained for

a(y) = (Σ(y) + I) -1   1 |ω| x∈ω(y) v(x)u(x) -µ(y)ū(y)   (.)
with I is the identity matrix and Σ(y) the variance-covariance matrix defined as:

Σ(y) = 1 |ω| x∈ω(y) v(x)v(x) T -µ(y)µ(y) T , (.) with µ = v. The coefficient b is given by b(y) = ū(y) -a(y) T µ(y) . (.)
The linear coefficient a(y), b(y) of overlapping windows are then averaged as in equation (.) and (.) to give on couple per pixel ā(x), b(x) . The output (gray) image is finally obtained with:

GF col {u}(x) = ā(x) T v(x) + b(x) . (.)
A pseudo-code fo the color version of the guided filter is given in Algorithm .

. Understanding the guided filter and its artifacts

The guided filter can be used in two different ways:



Algorithm : Guided Filter algorithm with a color guide input : input image u input : color guide image v input : smoothing parameter input : window radius r (box window will have size (2r + 1) 2 ) output:

filtered image GF col {u}  ū ← Mean ω {u} // mean of u in ω  v ← Mean ω {v} // mean of v in ω (3 × 1 vector)  c ← Mean ω {vu} -v ū // covariance of v and u in ω (3 × 1 vector)  Σ ← Mean ω {vv T } -vv T // equation (.)  a ← Σ/(s + ) // equation (.)  b ← ū -a T v // equation (.)  ā ← Mean ω {a} // equation (.)  b ← Mean ω {b} // equation (.)  return R ← āT G + b // equation (.)
. v = u: the guide image is different from the input image. This allows to transfer the edges of the guide on the input image (eventually with a little smoothing depending on the parameter ) and is used for example in dehazing applications [HST], where one can refine the haze transmission map using the color input image as a guide.

. v = u: the guide is the input image itself. This case correspond to edge-aware image smoothing, and the parameter is set according to amount of detail to be removed.

We will focus on this second use: indeed, the edge-aware smoothing effect of the guided filter is particularly interesting for our main application, contrast manipulation.

Thanks to its local linear model, the guided filter with v = u avoids the sharpening effect of the bilateral filter. Indeed, rewriting equation (.) for v = u gives a(y) = Var{u}(y)/(Var{u}(y) + ).

(.) Hence a ≤ 1, which means that edge magnitudes can only be reduced by the filter. Moreover, the averaging process in equation (.) can only reduce the gradient conservation of v at edges, because edges are generally surrounded by flat areas (thus the surrounding coefficient a are smaller than the coefficients localized on the edge).

The main artifact of the Guided Filter is what we will call the "contrast halo artifact", which comes from the fact that the edges are preserved, but the area around them is preserved too. We show an example of the resulting phenomenon in Figure .. A second artifact is the appearance of luminance halos. See figure .. This happens when the edge is not well preserved. The contrast halo appears close to the edges, when the variance is high. It is especially present when the filter is used with a large radius. Indeed, the guided filter can't smooth out half of a window and keep the other half as it is ; the choice is often an in-between decision : half smoothed, half kept.

Structure transfer with the guided filter

The joint bilateral filter (also called the cross bilateral filter) can also be used for this kind of structure transfer with a very similar result (see Evolution of the linear coefficients a(x), b(x) as a function of r, and the image content Note that when v = u, since we want to transfer a structure, we need a coefficient a that is different from zero everywhere, otherwise the value maintained in the output would be b = ū -av, i.e. the output would be smoothed out. Thus, we will prefer to keep very small. With close to zero, the guided filter scales the guide image v to the input image u. Indeed, we have

GF {u} Cov{u, v} Var{v} (v -v) + ū ,
if we consider the approximation (ā, b) (a, b). One can see in that formulation that the guided filter first removes the high frequencies in u, then adds the high frequencies of the guide image v -v, with a coefficient Cov{u, v}/Var{v}, which adapts the amplitude of this high frequency component to the scale of u. For example, it is then possible to use two images with different dynamic range, or a negative image, e.g. -u. On the other hand when v = u, we want to have a = 0 most of the time, and a = 1 at edges. Thus, the parameter should be set to a larger value. Furthermore, the local linear model is valid in square windows of size (2r + 1) 2 and one must keep in mind that the coefficients a(y), b(y) before aggregation are constant in that window. Hence, in a window containing both an edge to preserve and some texture to smooth out, the filter cannot do both well, and must take a balanced decision. This explains the apparition of the "contrast halo artifact". This also shows the importance of the parameter r in the filter: with a large r more contrast halo will appear, but with a small r the smoothing effect in flat windows is very light (only very high frequency texture can be removed with a tight window). The Figure . shows this contrast halo artifact in function of r.

Concerning the averaging of the estimators in equation (.) and (.). Due to this averaging, the output GF {u} do not respect the linear model in (.) any longer. But this gives more robustness to the filter. Once again, we can distinguish two cases: In each graph we superimpose the original line of a real image (blue line) with the results of the guided filter (red line) and the bilateral filter (green line). One can easily see the two artifacts of the guided filter: first, the amplitude of the edge in the guided filter results is always smaller than the original edge. This is the luminance halo artifact. Second, the structures are preserved around the edges, much more with the guided filter than with the bilateral filter, as seen in graph (d). This is the contrast halo artifact. The parameters used are r = σs = 6, 2 = σr = 30.

• When the variance in the input image is homogeneous, the linear coefficients a(y), b(y) have only small variations. In that case, the second mean filter in equation (.) and (.) gives indeed a more robust estimation against the noise, because more values are aggregated for the computation of b (equivalent to a larger window).

• When the variance is not homogeneous, for example at the interface between two almost constant areas but with different intensities (step edge): Then the aggregation process will smooth the variations of the linear coefficients a(y), b(y) so that the edge-preserving property is diminished: indeed, the edge-preserving coefficient a can only be reduced.

This second averaging process thus helps in the smoothing part but diminishes the capacity of the filter to maintain edges.

Effect of the window content

We have

ā(x), b(x) → (0, 0) for Var{v}(p) ā(x), b(x) → (0, ū(x)) for Var{v}(p)
Let us express the edge-preserving capability of the guided filter in function of the height h of a step edge (see Fig..). Consider the window ω(y) of size (2r + 1) 2 , centered on the step edge. This window can't be perfectly centered because the center of the edge lies between two pixels. The variance of the window is Var{v}(y) = ( h 1 ) 2 (1 -

1 (2r+1) 2 )
This value tends rapidly towards ( h 2 ) 2 when r gets larger so we will keep that value in the following.

The linear coefficients a(y), b(y) can therefore be rewritten using these relations as We draw here the evolution of the coefficient a for v = u and a window centered on a step edge. We present four different curves, for four different values . The values of are chosen so that the coefficient a is exactly 0.5 for a specific edge height, using α = (hα/2) 2 . It follows that we know, for a specific α, that edges with a height inferior to hα will be smoothed, and edges with height superior to hα will be preserved.

a(y) = h 2 h 2 + 4



(y) = (1 -a(y))ū(x)
and we can now have a closer look at the behavior of those coefficients for a specific height h.

Furthermore, one can show that for a specific α set so that a = 0.5 for h α , then a = 0.9 for h = 3h 1

a = 0.1 for h = 1 3 h 1 .
This means that there is an interval of width 8 3 h 1 in which the windows are neither really preserved nor smoothed. In other terms, there is a ratio equal to  between the height of an edge that will be smoothed and the height of an edge that will be preserved. This is interesting because we can now precisely set the parameter : We know that for a specific α , the step edge height  that will be half smoothed, half kept is

h mid α = 2 √ α .

√

α . To give an example, a typical value for the bilateral filter parameter σ r is . In that case, following the authors recommandation, we use = 10 2 . This leads to preserve edges of height  and to smooth edges of height . On the other hand, the bilateral filter will smooth edges of height  and preserve edges of height 3σ r = 30. Hence, the guided filter has more luminance halo than the guided filter. This can be observed in The same conclusion can be turned in another way: using the guided filter for base + detail decomposition without luminance haloes need a small , and thus the detail it produces will be of very low variance.

Effect of the coefficients aggregation

In Figure ., one can see the effect of the aggregation step on the linear coefficient a. While the smooth increase of its value at strong edges is desirable, the fact that its amplitude is also reduced is not, as it causes a luminance halo.

Guided filter kernel and comparison with bilateral filter's one

We recall the bilateral filter's kernel W BF (x, y, u):

W BF (x, y, u) = 1 |W BF (x, y, u)| exp - x -y 2 2σ 2 s exp - |u(x) -u(y)| 2 2σ 2 r . (.)
The authors of the guided filter [HST] show that their filter has an explicit kernel, that can be expressed by

W GF (x, y, u) = 1 |ω| 2 z:(x∈ωz,y∈ωz) 1 + u(x) -µ(z) u(y) -µ(z) σ 2 (z) + (.)
which shows some analogy to the bilateral filter kernel.

Proof. (as given by the authors in [HST]). Due to the linear dependance between u and GF {u}, the filter kernel is given by W (x, y) = ∂GF {u}(x)/∂u(y). Putting (.) into (.) and eliminating b, we obtain

GF {u}(x) = 1 |ω| z∈ωx a(z) v(x) -µ(z) + ū(z) . (.)
The derivative gives

∂GF {u}(x) ∂u(y) = 1 |ω| z∈ω(x) ∂a(z) ∂u(y) v(x) -µ(z) + ∂ ū(z) ∂u(y) . (.)
In this equation, we have where δ y ∈ ω(z) is one when y is in the window ω(z) and zero otherwise. On the other hand, the partial derivative ∂a(z)/∂u(y) in (.) can be computed from (.):

∂ ū(z) ∂u(y) = 1 |ω| δ y ∈ ω(z) = 1 |ω| δ z ∈ ω(y) (.)



∂a(z) ∂u(y) = 1 σ 2 (z) +   1 |ω| x∈ω(z) ∂u(x) u(y) v(x) - ∂ ū(z) ∂u(y) µ(z)   = 1 σ 2 (z) + 1 |ω| v(y) - 1 |ω| δ(z ∈ ω(y)) . (.)
Putting (.) and (.) into (.), we obtain

∂GF {u}(x) ∂u(y) = 1 |ω| 2 z∈ω(x),z∈ω(y) 1 + u(x) -µ(z) u(y) -µ(z) σ 2 (z) + . (.)
This is the expression of the filter kernel W (x, y).

Comparison with the bilateral filter

Concerning the parameter equivalence, the authors in [HST] suggest to use

r ↔ σ s ↔ σ 2 r .
The spatial equivalence is clear (Figure .), but we will see here that the edge preserving property is different in the two filters.

Concerning the second equivalence ↔ σ 2 r , the problem is more complicated. As we already saw, the guided filter is less "selective" than the bilateral filter. First, the choice is made differently. The bilateral filter compares pixel intensities in a one-to-one way. On the other hand, the guided filter measures the variance of the whole window and takes its decision accordingly. This is why a contrast halo appears. Second, the guided filter in less selective, i.e. for the same smoothing effect on an edge with a certain gradient, another edge with a stronger gradient will be better preserved by the bilateral filter than with the guided filter. gives the same amount of smoothing in the texture part, yet exhibits a strong luminance halo for the edge, which is not present for the bilateral. The second setting = ( σr 2 ) 2 better protects the edge but also fails to filter enough the textural part. Once again, the bilateral filter proves to be more selective that the guided filter. Step edge preservation with the bilateral filter (blue line) and the guided filter (green line). The abscissa shows the edge height, and the ordinate its "amount of preservation". The closer its value to zero, the more preserved the edge is. For the bilateral filter, this preservation comes from the fact that pixels from the opposite side of the edge will not be used in the averaging. For the guided filter, edge preservation relies on the multiplicative coefficient a. Note that this result is shown before aggregation, after which a often gets smaller. On the left, the figure shows the results for the standard equivalence = σ 2 r . On the right, with the parameter equivalence = ( Figure 2.12: Comparison of the bilateral and guided filter for a test pattern containing a step edge and a sawtooth structure. In the left row, the parameter equivalence is = ( σr 2 ) 2 , in the right row it is = σ 2 r . The spatial parameter used here is r = σs = 3 and the range parameter is σr = 50. Obtaining the same reduction of the oscillatory structure as the bilateral forces the guided filter to more contrast loss in edges.



In the previous chapter have been introduced the guided filter and its artifacts, namely, the contrast halo and the luminance halo. A comparison to the bilateral filter showed that its edge-preserving and smoothing property does not put the bilateral filter in the shade. On another hand, the guided filter has the neat advantage not to exhibit the staircase effect. This makes the filter particularly desirable for contrast enhancement.

By performing an asymptotic analysis for the guided filter when its support tends to zero, we obtain its tangent partial differential equation and prove that it is similar to the Perona-Malik diffusion equation, but deprived of its edge enhancement term that was shown to cause staircase artifacts. This explains why the guided filter actually has no such staircase artifacts. This analysis also yields a simple solution to reduce the guided filter's halos. We define an iterated guided filter that simulates the found nonlinear parabolic equation, and show that its solutions are halo free. A practical application to local detail enhancement confirms the effectiveness of the new filter.

. Guided filter relation to anisotropic diffusion A Partial Differential Equations Analysis of the Guided Filter

We now analyze the guided filter with partial differential equations. We refer to [BCM] for a similar methodology applied to the bilateral filter, in which the authors explain the apparition of the staircase effect by the fact that the bilateral filter is asymptotically equivalent to a Perona-Malik equation containing a reverse heat equation term creating shocks along zero-crossings of the Haralick edge detector [Har]. The same paper proposes a modification of the bilateral filter avoiding this shock-creating term. We prove here that the guided filter is equivalent to one iteration of an anisotropic diffusion partial differential equation, that can be interpreted as the first, diffusive, term of a Perona-Malik equation. This explains why the guided filter does not show staircase artifacts.

.. The Perona-Malik anisotropic diffusion

The early Perona-Malik [PM] "anisotropic diffusion" reads

u t = div(g(|Du| 2 )Du) (.)



where u = u(t, x) is the time-dependent image and Du(t, x) its derivative at x = (x, y), and g : [0; +∞) → [0; +∞) is a smooth decreasing function satisfying g(0) = 1, lim s→+∞ g(s) = 0. For example the function

g(z) = S z + S (.)
satisfies these conditions. The role of g is to stop the diffusion process at edges, where the image gradient is is high. Inserting (.) in (.),

u t = div DuS |Du| 2 + S (.)
one observes the following asymptotic behaviors:

• If |Du| 2 S, then u t ∆u • If |Du| 2 S, then u t div(S ∇u |∇u| 2 )
The first case leads back to the classic heat equation; the second case however contains a term for edge accentuation, as shown in equation (.). Developping equation (.), we have

u t = ∂ ∂x u x S |Du| 2 + S + ∂ ∂y u y S |Du| 2 + S = S |Du| 2 + S (u xx + u yy ) - S (|Du| 2 + S) 2 (u x ∂ ∂x (u 2 x + u 2 y ) + u y ∂ ∂y (u 2 x + u 2 y )) = S |Du| 2 + S ∆u - 2S (|Du| 2 + S) 2 u x (u xx u x + u yx u y ) + u y (u xy u x + u yy u y ) = S |Du| 2 + S ∆u - 2S (|Du| 2 + S) 2 Du T D 2 uDu (.)
where the Du T D 2 uDu term is a diffusion in the gradient direction, but is inverted by the minus sign. To understand this term, it is enough to consider the Taylor expansion of u in the gradient direction at a point x,

u(x + θDu) -u(x) = Du.θDu + 1 2 D 2 u(θDu, θDu) + O(θ 2 ) ,
and to notice that Du T D 2 uDu = 1 θ 2 D 2 u(θDu, θDu). Thus, the second term of equation (.) is (up to a factor) the opposite second derivative of u in the gradient direction and therefore a reverse one-dimensional heat equation. Its order of magnitude is the same as the first term (because of the squaring in Du T D 2 uDu) so its influence can't be neglected. We show in figure . the shock effects caused by the presence of this term. Note that the directional second derivative term Du T D 2 uDu is nothing but the Haralick [Har] edge detector. Indeed its zero-crossings characterize the inflexion points of the gradient in the direction of the gradient.

To summarize, the Perona-Malik anisotropic diffusion smooths the image in direction orthogonal to the gradient and enhances it in the gradient direction. We will demonstrate that the guided filter loses this edge-enhancement property. On the negative size, it therefore smooths more the image across its edges. On the positive side, it has no staircase effect.

.. Asymptotic behavior of the guided filter when it is localized

A pseudo-code of the guided filter is presented in Algorithm . For the asymptotic study of the filter, we focus on the (main) case of usage v = u and will work with a continuous definition of the guided filter. It will be denoted by GF σ {u}(x, y), were (x, y) are the horizontal and vertical coordinates of a pixel p and σ is the width of its kernel. We shall define for any function v(x, y) its local mean, weighted by a filter f σ , by

v(x, y) = f σ (h, l)v(x -h, y -l)dhdl, (.)
where f σ (x, y) stands for the local window ω of the guided filter. The authors in [HST] suggested to use a square window in order to take advantage of the integral images, but mention that any kernel form can be used. We consider here a general case. Hence, our continuous definition of the guided with v = u will be

GF σ {u}(x, y) = ā(x, y)u(x, y) + b(x, y) (.) where a(x, y) = Var{u}(x, y) Var{u}(x, y) + (.) and b(x, y) = 1 -a(x, y) u(x, y) , (.)
where the local variance Var{u}(x, y) is defined as

Var{u}(x, y) = f σ (h, l)u 2 (x -h, y -l)dhdl -ū2 (x, y) . (.) Theorem .. Consider a D image u(x, y) ∈ C 3 (Ω).
Let f 1 (x, y) be a nonnegative compactly supported radial kernel. We assume that the filter is normalized, namely f 1 (x, y)dxdy = 1; and symmetric xf 1 (x, y)dx = yf 1 (x, y)dy = 0. Set ˜ = /M 20 where is the edge preserving parameter of the guided filter, and

M 20 = f 1 (x, y)x 2 dxdy = f 1 (x, y)y 2 dxdy. Finally, let f σ be the scaled kernel: f σ (x, y) = σ -2 f 1 (x/σ, y/σ). Then, for (x, y) ∈ Ω, GF σ {u}(x, y) -u(x, y) = σ 2 M 20 ˜ |∇u(x, y)| 2 + ˜ ∆u(x, y) + O(σ 3 ) . (.) Remark .. Theorem . means that the image edges are preserved when ˜ |∇u(x, y)| 2 , because σ 2 M 20 ˜ /|∇u(x, y)| 2 0.
On the other hand, the filter is a diffusion by the isotropic heat equation when ˜ |∇u(p)| 2 . The transition between both behaviors is smooth, and a half-half compromise is observed when

˜ = |∇u(p)| 2 .  . if ˜ |∇u(x, y)| 2 , GF σ {u}(x, y) -u(x, y) = σ 2 M 20 ˜ |∇u(x, y)| 2 ∆u(x, y) + O(σ 3 ) ; . if ˜ = |∇u(x, y)| 2 , GF σ {u}(x, y) -u(x, y) = σ 2 M 20 2 ∆u(x, y) + O(σ 3 ) ; . if ˜ |∇u(x, y)| 2 , GF σ {u}(x, y) -u(x, y) = σ 2 M 20 ∆u(x, y) + O(σ 3 ) .
Proof. We first analyze the raw guided filter before the aggregation of coefficients performed in equations (.) and (.). We can then write the filter output GF raw σ {u}(x, y)

GF raw σ {u}(x, y) = a(x, y)u(x, y) + b(x, y) . (.)
We now study the behavior of the filter when σ → 0.

Let us denote by u x (x, y) the first derivative of u(x, y) in x and u xx (x, y) its second derivative in x. Without loss of generality by changing the axes and the origin and adding a constant to u, we can assume that (x, y) = (0, 0), that u(0, 0) = 0, and that the gradient of u at (x, y) is null in the direction of y, so that u y (0, 0) = 0. Let us now consider the Taylor expansion to the second order of u at (0, 0),

u(x, y) = αx + βx 2 + γxy + ζy 2 + O(σ 2 ) , (.) with α = u x (0, 0), β = 2u xx (0, 0), γ = u xy (0, 0)
, and ζ = 2u yy (0, 0). By developing the expression ū(0, 0) we get

ū(0, 0) = 1 σ 2 f x σ , y σ u(x, y)dxdy = 1 σ 2 f x σ , y σ (αx + βx 2 + γxy + ζy 2 + O(σ 2 ))dxdy = 1 σ 2 f x σ , y σ (βx 2 + ζy 2 ) dxdy + O(σ 3 ) = σ 2 M 20 (β + ζ) + O(σ 3 ) , (.)
The terms with odd exponent cancel out because of the kernel's symmetry. Equation (.) is obtained by substituting the variable x/σ by x and y/σ by y so that

1 σ 2 f x σ , y σ x 2 dxdy = 1 σ 2 f (x , y )(σx ) 2 σdx σdy = σ 2 f (x , y )x 2 dx dy = σ 2 M 20 ,
and the same substitution is used for 1 σ 2 f x σ , y σ y 2 dxdy. Similarly,

ū2 (0, 0) = 1 σ 2 f x σ , y σ (αx + βx 2 + γxy + ζy 2 + O(σ 2 )) 2 dxdy = 1 σ 2 f x σ , y σ (α 2 x 2 + 2βζx 2 y 2 + γ 2 x 2 y 2 ) dxdy + O(σ 3 ) = α 2 σ 2 M 20 + (2βζ + γ 2 )σ 4 M 22 + O(σ 3 ) ,



where

M 22 = f (x, y)x 2 y 2 dxdy. Finally, Var{u}(0, 0) = ū2 (0, 0) -ū2 (0, 0) = α 2 σ 2 M 20 + (2βζ + γ 2 )σ 4 M 22 -(β + ζ) 2 σ 4 M 2 20 + O(σ 3 ) = α 2 σ 2 M 20 + O(σ 3 ) . (.)
From equations (.) and (.) follows that

a(0, 0) = α 2 α 2 + ˜ + O(σ 3 ) , (.) using ˜ = /(σ 2 M 20 ), and from equation (.) we obtain b(0, 0) = σ 2 M 20 ˜ α 2 + ˜ (β + ζ) + O(σ 3 ) . (.)
Let us recall equation (.):

GF raw σ {u}(x, y) = a(x, y)u(x, y) + b(x, y) . Hence, GF raw σ {u}(0, 0) -u(0, 0) = b(0, 0) because u(0, 0) = 0. Furthermore, we have α 2 = |∇u| 2 and (β + ζ) = ∆u, therefore from equation (.) we obtain GF raw σ {u}(0, 0) -u(0, 0) = σ 2 M 20 ˜ |∇u(0, 0)| 2 + ˜ ∆u(0, 0) + O(σ 3 ) . (.)
This equation is therefore valid for any (x, y).

Let us now extend the above asymptotic result to GF σ . The guided filter aggregates the linear coefficients of overlapping windows. It therefore performs an additional averaging step by the same window f σ , namely computes ā(0, 0) -1 u(0, 0) + b(0, 0) = b(0, 0) .

(.)

Thus, we just have to convolve the coefficient b(x, y) by the window f σ and obtain the result of the convolution at (0, 0):

GF σ {u}(0, x 0 ) -u(0, 0) = f σ * σ 2 M 20 ˜ |∇u(x, y)| 2 + ˜ ∆u(x, y) + O(σ 3 ) (0, 0) = σ 2 f σ * M 20 ˜ |∇u(x, y)| 2 + ˜ ∆u(x, y) (0, 0) + O(σ 3 )
where * denotes the convolution. Since u is C 3 in Ω, by expanding at (0, 0) the function

M 20 ˜ |∇u(x,y)| 2 +˜ ∆u(x, y), which is therefore C 1 , we finally obtain GF σ {u}(0, 0) -u(0, 0) = σ 2 M 20 ˜ |∇u(0, 0)| 2 + ˜ ∆u(0, 0) + O(σ 3 ) . (.)
This ends the proof as this relation is valid for every (x, y).

Remark .. Equation (.) can be interpreted as one step of the evolution of the filtering process. We can express this evolution as a time evolution by setting dt = σ 2 (and t = ndt if we wish to consider n iterations). So we get the evolution

GF dt (x, y) -u(x, y) dt = M 20 ˜ |∇u(x, y)| 2 + ˜ ∆u(x, y) + O(dt 1 2 )
which can be considered as the first step of a Perona-Malik like equation,

du(t, x, y) dt = M 20 ˜ |∇u(t, x, y)| 2 + ˜ ∆u(t,
x, y), with u(0, x, y) = u(x, y).



. Iterated Guided Filter

We introduce here the Iterated Guided Filter as a straightforward derivation implementation of the mathematical analysis led in the preceding section, which showed that the guided filter is asymptotically equivalent to a well posed Perona Malik equation when the filtering neighborhood size σ tends to zero. This opens the way to a much more local iterated filter and raises the hope to get rid of all artifacts (in particular the halos) caused by the use of a fixed neighborhood. Meanwhile, the iteration can ensure that the filter keeps a similar filtering effect compared to the original guided filter. In other words, the iterated guided filter solves the guided filter's artifacts at the price of more iterations and therefore more computational time. The filtered results of the iterated guided filter are noticeably different from the Guided Filter results as we shall see. This is not attributable only to the artifact correction, but also to the different edge detection. As it uses a smaller radius, the edge detection is done at a finer scale. As a result, the preserved parts of the filtered image can be significantly different, but generally for the better, as finer results will be detected as part of the base.

Algorithm description

Basically, the iterated guided filter iterates the guided filter. However, we shall also examine strategies to reduce the computational time, leading to three different versions that will be distinguished by a different superscripts for the three versions IGF

(1)

σ , IGF (2)
σ and IGF

Definition .. We use the superscript t to denote the iterations and set

IGF (1) σ {u}(t, x) = āσ (t, x)IGF (1) σ {u}(t -1, x) + bσ (t, x) , (.)
where . is the local mean defined in Equation (.),

a(t, x) = Var IGF (1) σ {u} (t -1, x) Var IGF (1) σ {u} (t -1, x) + , (.) and b(t, x) = 1 -a(t, x) Mean IGF (1) σ {u}(t -1) (x) , (.) with IGF (1)
σ {u}(t = 0) = u. This is just the guided filter definition with v = u and where the input u used at each iteration is the filtered output at the previous iteration IGF

(1) σ {u}(t -1). Formally, we therefore have

IGF v {u}(t, x) = GF IGF (1) σ {u}(t -1) (x) , (.)
where the guided filter's guide is the input itself.

Algorithm  gives the pseudo-code of the IGF

(1)

σ . We compare the results of this filter to the classic bilateral filter in Figure . to verify that it is not affected by any staircase effect. As a consequence of the absence of the edge reinforcement term, the smoothing is stronger.

The iterated guided filter in this first version can't be guided by another image than itself. We therefore introduce a second version IGF

σ {u}  IGF (1) σ {u}(t = 0) ← u // Initialization  for t = 1, . . . , T do // Apply GF with given parameters , r and v = u  IGF (1) σ {u}(t) ← GF IGF (1) σ {u}(t -1)

 Definition .. IGF

(2) σ : Iterated guided filter with a constant guide. Here, the guide v stays unmodified over the iterations. IGF

(2) σ {u}(t) is both the filtered output at iteration t and the input at t + 1, with u is the input image and IGF

(2)

σ {u}(t = 0) = u. IGF (2) σ {u}(t, x) = ā(t, x)v(x) + b(t, x) (.)
where

a(t, x) = Cov v, IGF (2)
σ {u}(t -1) (x) Var{v}(x) + (.) and b(t, x) = Mean IGF (2) σ {u}(t -1) (x) -a(t, x)v(x) , (.)
with IGF

(2)

σ {u}(t = 0) = u. This can also be written

IGF (2) σ {u}(t) = GF IGF (2) σ {u}(t -1) , (.)
where the guided filter is used with the guide v.

Algorithm : Iterated guided filter v (IGF

σ ) input : image u input : guide v input : smoothing parameter input : radius r input : number of iterations T output:

IGF (2) σ {u}  IGF (2) σ {u}(t = 0) ← u // Initialization  for t = 1, . . . , T do // Apply GF with given parameters , r and guide v  IGF (2) σ {u}(t) ← GF IGF (2) σ {u}(t -1) Algorithm  gives the pseudo-code of IGF (2)
σ . An illustration is given for this filter in Figure .(c). It smooths less than the previous version, because the texture from the guide are preserved and keep being transfered over the iterations. However, because the linear coefficient a measures the covariance between the two images, its value will decrease in flat regions. (By flat region we mean a region with low variance with respect to .) Indeed, the image IGF

(2) σ {u}(t) gets smoother and smoother, and therefore differs more and more from the guide. At edges, since they are preserved, the linear coefficients do not change. Thus, the smoothing effect in flat areas increases (although it is a slight increase in comparison with IGF

(1) σ ) at each iteration. The coefficient ā in equation (.) does not participate to the smoothing nor evolve much over the iterations. Thus, the third version of the iterated guided filter gives up computing it at each iteration. This saves several convolutions and is therefore more efficient. Indeed, the iterated guided filter with v = u (IGF σ , this filter computes only once the coefficient ā. We remind that image v is the guide and is kept unmodified over the iterations; u is original input image and IGF

(3) σ {u}(t) is both the filtered output at iteration t and the input at iteration t + 1.

IGF (3) σ {u}(t) = ā(t, x)v(x) + b(t, x) (.)
where

a(t, x) = Cov{v, u}(x) Var{v}(x) + (.) and b(t, x) = Mean IGF (3) σ {u}(t -1) (x) -a(x)v(x) , (.)
with IGF

(3)

σ {u}(t = 0) = u.
This last filter has a lower smoothing power compared to versions v1 and v2. Indeed, the edge (and structure) preserving coefficient a is computed only once, at the first iteration of the filter. One therefore needs to modify the parameter to achieve the same "amount of smoothing".

A pseudo-code for the third version of the iterated guided filter is given in Algorithm . It is straightforward to deduce from the guided filter pseudo-code in Algorithm  the first and second versions, using equations (.) (v) and (.) (v). All three algorithms have a O(T N ) complexity, with T the number of iterations and N the number of pixels. Mean filters can be implemented with integral images which makes the filter complexity independent from the window's radius. However, this argument is not as crucial as in the original guided filter case, because the iterated guided filter is designed to use small radii (typically between  and ). For the same reason, the down-sampling strategy doesn't apply here.

Algorithm : Iterated guided filter algorithm input : input image u input : guide image v input : smoothing parameter input : window radius r (box window will have size (2r + 1) 2 ) input : number of iteration T output: filtered image IGF

(3) (3) σ need a higher to achieve a similar smoothing effect. This is due to the fact that coefficient a is not updated across the iterations.

σ {u}  ū ← Mean ω {u} // mean of u in windows ω  v ← Mean ω {v} // mean of v in windows ω  C ← Mean ω {vu} -vū // covariance of v and u in windows ω  V ← Mean ω {v 2 } -v2 // variance of v in windows ω  a ← C/(V + ) // equation (.)  ā ← Mean ω {a} // equation (.)  IGF (3) σ {u}(t = 0) ← u // Initialization  for t = 1, . . . , T do  IGF (3) σ {u}(t) ← Mean ω {IGF (3) σ {u}(t -1)} // mean of IGF (3) σ {u}(t) in ω  b(t) ← IGF (3) σ {u}(t) -av // equation (.)  b(t) ← Mean ω {b(t)} // equation (.)  IGF (3) σ {u}(t) ← āv + b(t)  return IGF (3) σ {u}  (a) Input (b) IGF (1) σ (t = 50) (c) IGF (2) σ (t = 50) (d) IGF (3) σ (t = 50) (e) IGF



We show in Figure . the filtering results with the three different versions. Noticeably, the amount of smoothing differs from one to the other. Indeed, in the first version (v), the guide is more and more smooth across the iterations, whereas the guide stays the same in the second version (v). In the third (v) version three, the gradient-preserving coefficient ā is kept unchanged across the iterations, whereas in v it takes the smoothness of the input image into account (covariance term).

Tuning the parameters to get equivalent smoothing effects

To get similar the spatial smoothing, we set either T and r according to the equivalent Gaussian kernel we want. Indeed, the Gaussian convolution can be approximated by K passes of box filtering. Wells [Wel] suggested to select r according to σ 2 = 1 12 K (2r + 1) 2 -1 . Since each iteration of the guided filter corresponds to two box filters, we can use the relation

r GF = 1 6 T (2r IGF + 1) 2 -1 1/2 , (.)
where r GF is the radius of the guided filter and r IGF the radius of the iterated guided filter.

Concerning the edge-preserving parameter , there is no clear equivalence, yet using /T in the first two versions of the iterative guided filter seems to work in practice. The third iterative version has no dependence on the number of iterations T , yet has a stronger smoothing effect than the original guided filter. For this reason we use /2 in our experiments. The second iterated version keeps a small contrast halo related to window's width: We used here r = 4 and this remaining contrast halo would be smaller with a smaller r. One reason to choose r > 1 is that the larger r, the less iterations we need for a fixed final spatial smoothing. We therefore use the largest r for which the contrast halo is not objectionable. This value might nevertheless depend on the viewer and on the image resolution. We found that r = 3 or r = 4 are acceptable. The third (fast) iterated version stays close to the second one but presents a small luminance halo.

. Results

Figure . presents another application to a gray scale image. The input and filtered images are displayed on the top row, and detail (input -filtered) is showed on the bottom row, with a contrast factor of  for visualization. The iterated guided filter solves the issues of the guided filter. Another example is given in figure ., where each channel of the input color image is filtered according to the luminance channel (= 0.2989×Red+0.5870×Green+0.1140×Blue). We compute the color coefficients before the filter using u color /u luminance and add them back after filtering. Thus, only luminance contrast is enhanced in this experiment. The texture of the table is better enhanced with the iterative version around.

We show in Figure . the results for color filtering. Note that the guided filter with color is slower than the guided filter by a factor of almost . Indeed, the computational cost for using a color guide is slightly less than  times the cost of a gray guide, and one needs to filter each of the three channels of the color image. This is also valid for the iterated versions v and v, but and IGF

(2) σ and /2 for IGF

(3) σ . Radius of GF is rGF = 26 and rIGF = 4 for the iterative versions, with the number of iterations T = 50. rGF is computed from equation (.), so that the spatial smoothing of all filters is the same. The detail in the "detail layer" row in multiplied by 6 for visualization purposes. The "enhanced" images are computed as: enhance{u} = 0.125 + 0.750×u + 6×(u -GF {u}). The last row displays a zoomed in part of the enhanced images. different for the fast version (v), which does not require recomputing the linear coefficient a at each iteration. So these coefficients are only computed once, and in comparison with the number of iterations (often more than ) this cost can be neglected, making this third version only three times slower (due to the filtering of three channels). Hence, compared to the original color guided filter, the first two iterated color guided filters have a complexity factor T , whereas the third one has a factor T /3 (as for the gray versions).



Artifacts

The Iterated Guided Filter still produces a small luminance halo. Moreover, according to the radius used, it can still show the contrast halo we observed with the Guided Filter. Obviously, a contrast halo made with a radius of  is still way less visible than with a radius of , which is a standard value for the original Guided Filter.

. Conclusion

In this chapter, we presented the guided filter. Its main advantages are a fast and exact implementation, a structure transfer capability and the absence of over-sharpening (staircase) artifacts.

We demonstrated the last property by showing the link between the guided filter and the Perona-Malik anisotropic diffusion. Furthermore, our proposition of an iterated guided filter solves the two main inconveniences of the filter, namely the contrast halo and luminance halo. We also went farther and proposed two variants of the iterated guided filter: the first variant, version two (v), accepts a guide different from the input image and can then be used for structure transfer; the third version (v) is a fast approximation of the second one, that unfortunately reintroduces some luminance halo. We then showed the efficiency of the new filter in the case of extreme local contrast enhancement.

Although the authors in [HST] defend themselves of proposing a fast approximation to the  bilateral filter, one can argue that the goals and tools of both classes of filters are closely related. Indeed, the idea of the bilateral filter is to prevent averaging pixels with distant intensities, even if they are spatially close. The guided filter, by measuring the local variance, applies the same principle: high variance areas, i.e.where pixels intensity variations are strong, are not averaged. Furthermore, it was proven in [Bar] that the underlying PDE of the bilateral filter is a variant of the Perona-Malik equation. Hence, both filter belongs to the same family of edge-stopping diffusion filters.

 GF color IGF (
The next chapter will concentrate on the bilateral filter, and review its numerous fast approximations proposed since .



The previous Chapters  and  are dedicated to the fast and recent guided filter, link it to the anisotropic diffusion and compare it to the bilateral filter. Those two last filters are the most widespread filters for the computation of an image base. In this chapter, we present the bilateral filter. We recal its long history, and describe its main descendants: the joint (or cross) bilateral filter [ED,PSA + ], the bilateral filter with regression [BCM], the unnormalized bilateral filters [APH + ,APH + ,MT]. Furthermore, we make the link between the bilateral filter and ACE (Automatic color enhancement) that belongs to the retinex family. We also explain the staircase effect first described, and solved, by Buades et al. [BCM]. Two others chapters dedicated to the bilateral filters will follow. A review of the numerous schemes proposed to correct the staircase effect (Chapter ), and a review of the fast approximations, particularly usefull when the filter is used with large spatial neighborhood as in the base and detail decomposition problem (Chapter ). However, since the unnormalized bileral filter will be defined in this chapter, we get ahead and present its fast approximations here. Likewise, we propose in this Chapter  a fast approximation of the bilateral filter with regression and a multi-scale filter based on it. This last filter gives us the opportunity to define and explain the dark halo artifact.

Paris, Kornprobst, Tumblin and Durand  book "Bilateral filter: theory and applications"

This chapter, along with the two following ones on the staircasing corrections and the fast approximations of the bilateral filter, is directly inspired by the S. Paris, P. Kornprobst, J. Tumblin and F. Durand book [PKTD]. Whereas this book aims at giving an extensive presentation of the bilateral filter and its applications, we concentrate on its usage for base and detail decomposition. Nonetheless, we approach several points already reviewed in the  book, e.g., the different proposed extensions and its fast approximations. We highlight below the main differences between our Chapters , ,  and Paris, Kornprobst, Tumblin and Durand book. Concerning this chapter on the bilateral filters, we present supplementary filters:

• the unnormalized bilateral filters [APH + , MT], along with their fast approximations;

• we propose a fast approximation for the bilateral filter with regression;

• we establish a formal link with the filter ACE (Automatic Color Enhancement) [GRM].

We pursue the review of the bilateral filter with the staircase effect corrections in Chapter . There are two kinds of corrections: the first modify the bilateral filter so that the slopes are taken into  account, e.g., the bilateral with regression filter, the trilateral filter, the symmetric bilateral filter; these have been reviewed in Paris et al. book, so the differences between our review and theirs comes down to:

• a more detailed presentation of the trilateral filter, with pseudo-codes;

• the introduction of a symmetric bilateral filter similar to Elad's one [Ela].

The second kind of approximations however is not described in [PKTD]. It consist in postprocessing the filtered image to correct the staircase artifact. The described corrections are:

• the blending described by Durand and Dorsey [DD];

• the minimal isotropic smoothing effect in the separable kernel approximation [PVV];

• the Poisson correction proposed by Bae et al. [BPD];

• the selective diffusion of Kass and Solomon [KS].

Concerning the fast approximations, most of them are reviewed in the book. Nonetheless, we add to the list filters posterior to  and sometimes give more detailed descriptions:

• in the local histograms, Weiss [Wei] approximation is described in the book, yet we give of it a more in-depth description: we present the earlier Huang's algorithm and give for both pseudo-codes. Furthermore, we review Porikli's  version that uses integral histograms, and discuss the usage of box spatial kernels;

• the fast approximations of the unnormalized bilateral filter is given in Chapter ;

• we present a supplementary class of fast approximations based on the usage of polynomials range kernels;

• the domain transform is also reviewed, this filter can be thought as a bilateral filter when used with a small spatial kernel.

. Introduction

The principle of bilateral filtering appeared with Yaroslavsky () [Yar] and Lee () [Lee].

The variant we study was proposed by Smith and Brady who called it "SUSAN" () [SB]. It was re-proposed by Tomasi and Manduchi under the name "bilateral filter" in  [TM]. All of these similar filters can be termed neighborhood filters. We call neighborhood filter any filter which computes a pixel by taking an average of the values of neighboring pixels with a similar grey level value. In Yaroslavsky () [Yar] and Lee () [Lee] it is proposed to average pixels belonging to the neighborhood G(x, σ r ) ∩ B σs (x). This filter can be rewritten in a more continuous form as

YF σr,σs u(x) = 1 C(x) Bσ s (x) u(y)e - |u(y)-u(x)| 2 2σ 2 r dy (.) where x ∈ Ω and C(x) = Bσ s (x) e - |u(y)-u(x)| 2 2σ 2 r
dy is the normalization factor. Only pixels inside B σs (x) are averaged. In later versions the gray level threshold was replaced by a Gauss weighting function depending on a filtering parameter σ r [SB], [TM]. These algorithms, instead of considering a fixed spatial neighborhood B σs (x), weigh the distance to the reference pixel x,



BF σr,σs u(x) = 1 C(x) Ω u(y)e - |y-x| 2 2σ 2 s e - |u(y)-u(x)| 2 2σ 2 r dy , (.)
where

C(x) = Ω e - |y-x| 2 2σs 2 e - |u(y)-u(x)| 2 2σ 2 r
dy is the normalization factor and σ s is now a spatial filtering parameter. We show in figure .(b) a representation of the bilateral kernel at an edge. There is no significant difference between YF h,σs and BF h,σs . The performance of both algorithms is justified by the same arguments. Inside a homogeneous region, the gray level values slightly fluctuate because of the noise or texture. In that case, the first strategy computes an arithmetic mean of the neighborhood and the second strategy a Gaussian mean. At a contrasted edge separating two regions, if the gray level difference between both regions is significantly larger than σ r , both algorithms compute averages of pixels belonging to the same region as the reference pixel. Thus, the algorithm does not blur the edges, which is its main scope.

We refer to [PKTD] for a extensive review of the applications of the bilateral filter. Quoting it: "[The bilateral filter] has been used in various contexts such as denoising [ASG], [BM], [LFSK], texture editing and relighting [OCDD], tone management [BPD], [BM], [DD], [ED], [Ela], [PSA + ], demosaicking [RS], stylization [WOG], and optical-flow estimation [ST], [XCS + ].". This overview gives an idea of the wide adoption of this filter in the community, and more generally the usefulness of base and detail decomposing filters image processing.

. The bilateral filter and its implementation

The bilateral filter, as defined in equation (.), has a simple implementation with O(N 2 ) complexity. It is usual to reduce it to O(r 2 N ) by restricting the convolution to a (2r + 1) × (2r + 1) window (usually r = 2σ S ). But the complexity remains high when the filter is used with large spatial support.

Algorithm  presents an implementation of the bilateral filter. Numerous fast approximations have been proposed to accelerate this filter. However, none of them is able to reproduce the exact bilateral filter. Nevertheless, some of them have turned out to be really close. They will be detailed in the next section  on the main fast approximations.



do  k(x, y) = G σs (x -y)G σr u(x) -u(y) // Compute current pixel's weights  ω(x) = y k(x, y) // Normalization factor  BF{u}(x) = ω(x) -1 y k(x, y)u(y) // Compute output value  return BF{u}

Limitations and artifacts of the bilateral filter

The first limitation of the original bilateral filter is its execution time. Since it needs to recompute the kernel at each pixel, the execution is very slow for large images or a large spatial standard deviation σ s . The second limitation is the so-called staircase artiofact [BCM] -namely a tendency of the filter to create jumps (staircases) along the inflexion lines of smooth regions. The staircase artifact is illustrated in figure . and figure .. In this figure we simplified the range and spatial kernels by using simple boxes. This allows a simple visualization, in the dimensional case, of what pixels are taken into account in the averaging process. The blue arrows are the intensity differences u(x) -u(y). The dotted box shows the boundaries of the range and spatial kernels: outside of this box, all the bilateral weights are zero. Then, it is easy to see that for the current pixel (namley the intersection of the two blue dotted lines at the center of the box) the averaged value has a higher intensity than the initial one. By applying the bilateral averaging on each pixel of the blue line, one obtains the red line. The "propagation of the plateau" that one can observe is what we call the "staircase artifact".

This spurious edge reinforcement causes a staircase, or "contrast reversal" artifact when the filter is used for contrast enhancement. This effect is visible in figure ..

. On the link between ACE and the bilateral filter

In this section we demonstrate that ACE has the same formula as the residual of the bilateral filter. The difference is that the spatial kernel has slow decay, in 1/||x|| and that the range kernel does not discard values with distant intensity but rather limits their influence. To the best of our knowledge, such a link has not been suggested yet.

Theorem .. Let u : Ω → [0, 1] be the input image, F : R 2 → [0, 1] the filter kernel, H : R → [0, 1] an influence function and C : Ω → [0, 1] the normalization factor. Denote x = (x 1 , x 2 ) and y = (y 1 , y 2 ) the D-coordinates of pixels in Ω. ACE and the bilateral residual u -BF{u} are both written in the same form:

v(x) = 1 C(x) y∈Ω F (x -y) H (u(y) -u(x)) 



Proof. Let recall the bilateral filter definition:

BF{u}(x) = 1 C BF (x) y∈Ω G σs (x -y)G σr u(x) -u(y) u(y), (.)
where G σs is the Gaussian spatial kernel, G σr the range Gaussian kernel, and C BF (x) = y∈Ω G σs (xy)G σr u(x) -u(y) the normalization factor. As presented in the unnormalized bilateral filter [APH + , APH + ] Section ., equation (.) can be written

BF{u}(x) = u(x) - 1 C BF (x) y∈Ω G σs (x -y)G σr u(x) -u(y) u(x) -u(y) . (.)
We thus have

u -BF{u}(x) = 1 C BF (x) y∈Ω F BF (x -y)H BF (u(y) -u(x)), (.)
where

F BF (x) = G σs (x) and H BF (t) = G σr (t)t.
As for ACE, it is defined as

ACE{u}(x) = y∈Ω\x 1 x -y s α u(x) -u(y) , (.) thus ACE{u}(x) = 1 C ACE (x) y∈Ω F ACE (x -y)H ACE (u(y) -u(x)) (.) with F ACE (x) 1/ x x ∈ Ω\(0, 0) 0 x = (0, 0), (.)
H ACE (t) = s α (t), and C ACE (x) = 1 everywhere in Ω.

In summary, the only essential difference between ACE and BF is that the former computes the detail layer (and directly enhances it, as we shall see) whereas the latter computes the base layer. Another difference lies in the absence of normalization in ACE, allowed by the point we just mentioned. Indeed, similarly to the unnormalized bilateral filter (see Section .), averaging intensity differences (u(y) -u(x)) that oscillate around zero permits to remove the normalization term. The other difference between both filters is the form of the functions F (.) and H(.). As such, ACE can be expressed as the detail layer given by an unnormalized bilateral filter with modified spatial and range kernels.

The difference between the kernels is relevant: in ACE the spatial kernel is scale-invariant and the range kernel, rather than excluding pixels with distant intensity from the averaging, limits their influence by a threshold. Various range functions for the bilateral filter have been investigated in the context of robust statistical estimation by Durand et al. [DD] in . The range function used in ACE is known as the Huber minimax [Hub]. It was previously studied in the context of anisotropic diffusion by Black et al. in  [BSMH]. We reproduce in Figure . an illustration from [BSMH] showing Huber's minmax norm ρ(.), its derivative ψ(.), and the edge-stopping function g(.). The derivative ψ(.) is proportional to the influence function [HRRS]. In Theorem . we denoted it by H(.). This function characterizes the bias that a particular measurement has on the solution [BSMH]. The Huber minmax edge-stopping function g() is defined as  Remarkably, ACE range kernel prevents the filter from creating staircase patterns in the filtered image, that is, contrast reversal artifacts in the result. This is because outliers are not rejected but simply clipped. On the other hand, it makes ACE prone to halos artifacts, particularly visible when the used spatial kernel is Gaussian. However, with the 1/ x kernel, the halo is somehow "dissolved" because of its width, thus not visible. We display some results of ACE in Figure ..

(x) = ρ (x)/x. We have ρ(x, σ) = x 2 /2σ + σ/2 |x| ≤ σ, |x| |x| > σ, (.) ψ(x, σ) = x/σ |x| ≤ σ, sign(x) |x| > σ, (.) g(x, σ) = 1/σ |x| ≤ σ, sign(x)/x |x| > σ. (.) Remark that ψ(x, 1/α) = s α (x). We report in

. Unnormalized bilateral filter

The unnormalized bilateral filter (UBF) was proposed by Mathieu Aubry in his articles on the local Laplacian filter [APH + ], [APH + ]. It is extensively described in Chapter . His observation is that the bilateral filter can be rewritten in a way that keeps the average intensity of the image even if the normalization factor is removed. On the other hand, removing the normalization factor allows to reduce the filtering effect in the vicinity of the edges and then to reduce the staircasing artifact.

And because one no longer needs to compute this normalization factor, the filter is faster than the original one. From this point of view, the unnormalized bilateral filter is the only filter that with a unique modification both accelerates the bilateral filter and diminishes its sharpening property. Furthermore, it can benefit from several acceleration schemes dedicated to the bilateral filter. We shall see however that removing the normalization is not without drawbacks. In particular, the UBF smoothing strength is lowered, especially at edges and for small or thin objects.

The unnormalized bilateral filter has the simple expression

UBF{u}(x) = u(x) + y∈Ω G σs (x -y)G σr u(y) -u(x) u(y) -u(x) . (.)
Compared to the bilateral filter (cf. equation (.)), the unnormalized version averages the intensity differences u(y) -u(x) rather than the intensity u(y) themselves, and the input image is added to keep the overall intensity of the image. With the normalization factor, this would just be a rewriting of the bilateral filter. Yet in equation (.) is it safe to remove C because when the sum tends towards zero the output value tends to the input value, so there is no intensity shift. Contrarily to the bilateral filter, the spatial kernel G σ S has to be normalized:

G σs (x) = exp - x 2 2σ 2 s   y∈Ω exp - y 2 2σ 2 s   -1 , (.)
because the removed normalization factor doesn't compensate it any more. The definition (.) can be rewritten as a blend between the original image u and the bilateral filter result BF{u} involving the bilateral normalization factor C that we recall is defined as

C(x) = y∈Ω G σs (x -y)G σr u(y) -u(x) . (.)
This leads to a second definition of the unnormalized bilateral filter:

UBF{u}(x) = C(x)BF{u}(x) + 1 -C(x) u(x). (.)



The definition of BF is given in equation (.). Equation (.) makes it easy to understand the behavior of the filter: the normalization factor C is large in flat areas (where the range kernel gives values close to one to most pixels), and gets smaller when it comes across image edges (where the range kernel gives values close to zero to many pixels). This results in keeping the original image at the edges and the bilateral filter result on the flat areas. This behavior resembles the one of the guided filter (see Chapter ). We shall elaborate later on "contrast halo artifact" that UBF is prone to create. We provide the pseudo-code of the filter in Algorithm . This is the "brute force" version, and we shall present later its fast approximation. However, note that because the normalization factor is removed, this version is nearly twice faster than the original bilateral filter. The algorithm, for each pixel of the input image (line ), starts by computing the bilateral weights (line ) in the window Ω. Finally, it computes the weighted sum of the differences u(xy) -u(x) in Ω and adds it to the input intensity u(x) to obtain the output value UBF{u}(x) (line ).

k(x, y) = G σs (x -y)G σr u(y) -u(x) // Compute output value using equation (.)  UBF{u}(x) = u(x) + y∈Ω k(x, y) u(y) -u(x)
This algorithm can be accelerated using separable kernels, polynomials range kernels or the layered approximations. Since this last approximation is used by M. Aubry et al. [APH + ] for the fast local laplacian filter, we concentrate on this fast approximation of the unnormalized bilateral filter.

Algorithm  describes its pseudo-code. This algorithm requires to set the number of intensity samples S. This number is usually chosen in function of the range parameter σ r , as this layering can be interpreted as a sampling of the range kernel. Thus, a small kernel requires a small "range period" and therefore a large number of layers. On the contrary, a big parameter σ r won't need a large number of layers to acheive a good approximation. The authors in [APH + ] recommend to sample the intensity range every σ r . The algorithm starts by computing the "range period" (line ) according to the dynamic range and the chosen parameter S. Then, for each intensity sample, it computes the layer (line ), then convolves it by the Gaussian spatial kernel (line ). The output image UBF fast {u} is then updated (line ) using the interpolation weights (computed at line ): the output pixels which value do not correspond to an intensity sample are linearly interpolated from the two closest layers.

We show in Figure . the difference between the bilateral filter (red line) and the unnormalized bilateral filter (orange line) in two different configurations: the first (image a) is the filtering of a smooth edge. This figure shows the reduction of the staircase artifact: the over-sharpening is less present with UBF (the orange line stays closer to the blue one at the edge). The second configuration (image b) is the filtering of the same test-pattern where we added noise. It shows that UBF smoothes less than BF. This last property is often a drawback because one needs to increase  Algorithm : Unnormalized Bilateral Filter, Fast approximation using layers input : image u input : spatial parameter σ s input : range parameter σ r input : radius r: window Ω has size (2r + 1) 2 input : number of samples of the intensity range S output: filtered image UBF fast {u}  ν = min(u) -max(u) /(S -1) // gap between two intensity samples

 UBF fast {u} = u // initialization  foreach intensity sample j ∈ {0, 1, ..., S -1} do  γ j = min(I) + j × ν
// value of intensity sample // Following operations are pixel-wise: the value of σ r to obtain a similar smoothing effect, which reduces the edge-preserving property of the filter. Furthermore, as we see in equation (.), UBF keeps the original image values where the normalization factor is small. This happens at edges and thin lines. These parts of the image are thus not filtered, which makes a contrast halo to appear in contrast enhancement applications.

 α j = max(0, 1 -|u -γ j |/ν) // interpolation weights  H j = G σr (u -γ j )(u -γ j ) //

"A new class of image filters without normalization"

Peyman Milanfar and Hossein Talebi recently published a paper on filters without normalization [MT]. While they claim to present a new class of filters, replacing the normalized by the unnormalized ones, the idea behind is basically the same as presented by M. Aubry et al.

[APH + ].
The authors' proposition is to use a constant normalization factor α for the entire image. In the unnormalized bilateral filter, this constant factor is implicitly set to  (with a normalized spatial Gaussian kernel). The authors in [MT] set α so that it is the closest to the original normaliza-  tion factors everywhere. They provide (quoting [MT]) "an analytically sound and numerically tractable choice for the scalar α > 0 that gives the best approximation to [the filter] in the leastsquares sense". They eventually give this value:

α = 1 1 N x∈u C(x) , (.)
with N the number of pixels in the image u. Thus, the best constant α to approximate the normalized filter is the mean of all the normalization factors in the image. We adopt the notations from [MT] for the few formulas reported below. As defined in the original paper (quoting):

Consider the vectorized image y of size n as the input, and the vectorized image z as the output of the filtering process. The general construction of a filter begins by specifying a symmetric positive semi-definite (PSD) kernel k ij ≥ 0 that measures the similarity, or affinity, between individual or groups of pixels.

Indices i and j are pixels. The kernel k ij is for example the bilateral one. The normalized weights are defined as

w ij = k ij n j=1 k ij . (.)
Still following the paper notation, the output with matrix notation is:

z = Wy, (.)
where the i-th row of the filter matrix W is the vector [w i1 , ..., w in ] and produces the i-th output pixel. As said in [MT], "the filter matrix W is a normalized version of the symmetric positive definite affinity matrix K constructed from the unnormalized affinities k ij , 1 ≤ i, j ≤ n". They then write W as a product of two matrices

W = D -1 K (.)
where D is a diagonal matrix with diagonal elements

[D] ii = n j=1 k ij = d i .
Whereas the normalized filter (.) can be written

W = I + D -1 (K -D), (.)
they replace the normalization matrix D -1 by the constant α, what defined the approximation W:

W = I + α(K -D). (.)
By minimizing the following cost function using the matrix Frobenius norm:

min α W -W(α) 2 , (.)
and with some approximations they get

α = 1 n i=1 d i . (.)
Noticeably, prior to un-normalizing the filter they need to compute the normalization factors at each pixels.

The key properties of this approximate filter, as they give, are (quoting):

• Regardless of the value of α, the rows of W always sum to one.



• While the filter W is not necessarily symmetric, the approximate W is always symmetric. The advantages of having a symmetric filter matrix are many, as documented in the recent work [Mil].

• The normalized filter weights in W are typically non-negative valued. The elements in W however, can be negative valued, meaning that the behavior of the approximate filter may differ from its reference value.

We observe that the unnormalized bilateral filter can be written in the exact same way, with α = 1. Let recall UBF with notations from [MT]:

z ubf i = y i + n j=1 k ij (y j -y i ) (.)
That is, in matrix notations:

z ubf = y + (K -D)y (.) = (I + (K -D)) y.
(.)

The properties given above then apply to UBF. Concerning the symmetric filter matrix W, according to P. Milanfar in [Mil] (quoting):

Symmetrizing the smoothing operator is not just a mathematical nicety; it can have interesting practical advantages as well. In particular, three such advantages are that () given a smoother, its symmetrized version generally results in improved performance; () symmetrizing guarantees the stability of iterative filters based on the smoother; and () symmetrization enables us to peer into the complex behavior of smoothing filters in the transform domain using principal components.

The same authores later published [TM] a method for image enhancement based on the Laplacian operator using this un-normalization strategy.

. Bilateral Filter with regression

The bilateral filter with regression [BCM] incorporate a way to estimate a plan rather than a constant for each pixel, thus handeling better the slopes where, as proven by the authors, the original bilateral filter has a staircase effect. They alos proved that the bilateral filter with regression has not this artefact when the size of the spatial neighborhood tends towards zero.

The bilateral filter with regression (BFR) was introduced by Buades et al. [BCM] as an extension of the standard bilateral filter reducing its staircase effect (see section "Artifact" in .). It consists in the estimation, for each pixel, of the best fitting plane according to the bilateral weights.

The bilateral filter with regression is defined as follows in [BCM].

We call BFR σr,σs {u} the value obtained at x = (x 1 , x 2 ) by finding the plane locally approximating u in the following sense

min α,β,γ Ω k(x, y) u(y) -αy 1 -βy 2 -γ 2 dy (.)
where

k(x, y) = e - y-x 2 2σ 2 s e - |u(y)-u(x)| 2 2σ 2 r . (.)



Then, the restored value at x is given by αx 1 + βx 2 + γ. The weights used to define the minimization problem are the same as the ones used by the neighborhood filter. Thus, the points with a grey level value close to u(x) will have a stronger influence in the minimization process. The only difference with BF is the replacement of an average by a linear regression. The minimization process is made explicit, since we can easily derive the normal equations. Thus, the computation of the above linear regression reduces to the solution of a 3×3 linear system.

One should not confuse this regression strategy with the strategy used in the DAD denoising algorithm [PRMF], where this regression plane is subtracted from the patch before a second filtering step is applied with new bilateral weights. A similar two-step method (estimation of a plan, then filtering after subtraction of this plan) is used in the trilateral filter. This filter is described in Section ..

We call k = k(x, y) the weights of the bilateral filter at point (0, 0) for the image u = u(x, y). The bilateral filter with regression does finds

arg min a,b,c x,y k(ax + by + c -u) 2 . (.)
Differentiating this energy with respect to a, b, c and equating the result to zero gives the following system of equations,

  ρ x 2 ρ xy ρ x ρ xy ρ y 2 ρ y ρ x ρ y ρ     a b c   =   xku yku ku   , (.)
where ρ x = x,y xk(x, y), ρ xy = x,y xyk(x, y), etc. and in all equations u, k stand for u(x, y), k(x, y).

. Fast bilateral filter with regression

The bilateral with regression can be accelerated easily using the piecewise-linear approximation or the bilateral grid. Algorithm  present the pseudo-code of the regression bilateral filter implemented approximated with the piecewise-linear strategy.



. Multi-scale bilateral filter with regression

For the time of this short section, we move ahead to the multi-scale filters, presented in details starting at Chapter . In particular, we refer to the Section . for a precise prensentation of the Laplacian pyramid used in the following.

The multi-scale bilateral filter with regression is a straightforward multi-scale implementation of BFR using the Laplacian Pyramid. It is described in Algorithm . The exact bilateral filter with regression is used at each scale (no need to use the fast one, because the spatial standard deviation is only 1 pixels). This filter is described in Algorithm .

Algorithm : Multi-scale guided filter with regression (MBR) input : image u input : parameters σ s , σ r and r input :

parameter l max output: filtered image v  Lpyr{u} ← LaplacianPyramid(u) // compute Laplacian pyramid until scale lmax  v lmax ← BFR {Lpyr{u, l max }} // initialization: filter residual  for scale l from l max -1 to 0 do // from coarsest to finest scale  v l ← Upsample(v l-1 ) + Lpyr{u, l} // upsample and add Laplacian coefficients  v l ← BFR{v l }
// filter the new image using σs, σr and r We display in Figure . the filtering result of this algorithm and compare it to the original bilateral filter with regression. The parameters we used are the same except for the spatial standard deviation σ s : for the single-scale version we used σ s = 32 and for the multi-scale σ s = 1 and l max = 5. But since the input image is downsampled in the (dyadic) pyramid the spatial support is 2 5 = 32 in BFR too. The single-scale version presents staircasing (see Figure . (e) at the edges of the obelisk and top of the trees), because σ s = 32 is a large spatial support. In this case indeed, estimating a regression plane rather than a constant does not help much because the bilateral weights constraint the plane to have very low first order coefficients (the weights used  in BFR are the same as in BF). However, the multi-scale version uses very small windows (σ s is typically between 1 and 3) so the plane estimation is effective and the filter therefore successfully removes the staircase effect. This is clearly visible in Figure . (c). In return the luminance halo slightly increases, but stays contained (see results with the test-pattern in Section ., Table 

.).
But the manipulation of the coefficients in the Laplacian Pyramid is not without dangers. Indeed, the pyramid is constructed so that the exact image can be recovered by collapsing the pyramid, that is, the Laplacian coefficients at each scale perfectly match the upsampled image from the previous scale until the finest one. The procedures for Gaussian and Laplacian pyramid construction are described in Chapter . Yet in our algorithm the different levels of the pyramid are smoothed independently. Thus when we upsample the smoothed image and add the Laplacian coefficients of the subsequent level, they may not properly compensate their respective oscillations. This effect has been described in the excellent paper by Facciolo et al. [FPM]. In our filter, it creates what we called the "dark halo" artifact. This is in fact an inverted luminance halo, dark around dark objects (in the detail layer) and bright around bright objects. We display a case where it is particularly visible in Figure .. It arises at thin object, for example the streetlight and the top to the signboard. We show a zoom in those two parts in (f). The enhanced result presented in (e) shows that it creates a strongly visible incoherence.

Conclusion

The multi-scale bilateral filter with regression has two advantages, namely, the correction of the staircase effect and the speed, but one unacceptable drawback: the dark halo artifact. In [FPM] the authors eliminate those spurious oscillatory patterns by removing, at each scale, the high frequencies, which are eventually filtered at a finer scale -because they progressively become the medium and low frequencies as the image gets recursively upsampled. In our case however we cannot apply this strategy, because this would mean increasing σ s , otherwise this filter's work would be discarded by the additional low-pass filter. Therefore the computational time and the staircase effect would both increase.

In Chapter  we consider the replacement of the bilateral filter with regression by the guided filter. As we shall see, this filter is more appropriate for this multi-scale scheme.



 

In the previous chapter on the bilateral filter, we have seen that BF not only preserves the edges, but also is prone to sharpening them. This effect has been described and mathematically justified by Buades et al. in  [BCM], who call it the staircase effect. Indeed, bilateral-based filters tend to create piece-wise constant signals separated by numerically created edges, thus adopting aspect of a staircase. From the contrast enhancement and tone-mapping point of view the same effect is sometimes called the gradient reversal artifact, because the complementary detail layer, at places where edges have been reinforced in the base layer, contains reverted gradients. The problem is that when using the bilateral filter for contrast enhancement and tone-mapping, the detail layer gets stretched and the base layer compressed. The recombination of their results causes the gradient reversal artifact.

Since this artifact is particularly annoying in contrast manipulation methods, many authors have tried to correct it. The solutions can be divided in two categories. The first category of correction does not modify the filter, but corrects the artifact in a post-processing step.

The second one directly modifies the filter to make it handle smoothly the slopes. We review in this chapter both categories of corrections. Nevertheless we shall skip two of the corrections, namely the bilateral filter with regression [BCM] and the unnormalized bilateral filter [APH + ]. Both have already been presented in Chapter .

. Introduction

Several authors have presented a post-filtering correction step to remove the staircase artifact. F. Durand and J. Dorsey [DD] proposed a blend between a low-pass version of the input image and its bilaterally filtered one weighted by the normalization term. They justify this choice by explaining that the bilateral filer is not robust at edges because it misses information. The authors of the separable kernel bilateral filter [PVV] also proposed to prevent the staircase effect by enforcing a minimal isotropic smoothing effect everywhere. We shall review this correction method in section .. In , Bae et al. [BPD] use Poisson reconstruction on the filtered image; this solution is presented in Section .. One another important proposition was made by Kass and Solomon, the authors of the smoothed local histogram filters [KS], where they iteratively smooth the bilaterally filtered image according to the distance to the input image. Roughly, its idea is that if after a Gaussian filter has been applied to the bilateral output, the image get closer to the input image than to the bilaterally filtered one, then one should keep the Gaussian filtered one. The decision is local, and the process is done in an iterative manner with increasing standard deviations. We shall review this process in section ..

Other approaches modify the bilateral filter so that it handles piece-wise linear signals rather than piecewise constant ones, as implicitly assumed in the standard definition. The  trilat- eral filter [CT] also aims at "smoothing signals towards a sharply-bounded, piecewise-linear approximation". It is a two-step filter, where the local slopes are estimated first, then used to "tilt" the bilateral kernel. This algorithm is analyzed in section .. The  paper by M. Elad [Ela] also proposed to handle the piecewise-line case by symmetrizing the bilateral kernel. We review this method in section .. The  paper "The staircasing effect in the neighborhood filters and its solution" [BCM] proves that the staircase effect can be removed by computing at each pixel the regression plane that best fits the signal using the bilateral weights, rather that a simple scalar.

We developped a methodology for measuring the staircasing amplitude. This is presented in Chapter  on the local Laplacian filter, in Section ...

This chapter is again inspired by Paris et al. for its in depth presentation of the bilateral filter [PKTD]. The differences with our review are highlighted in Chapter  on the bilateral filter.

. A minimal isotropic smoothing effect in the separable bilateral filter

In the  fast approximation of the bilateral filter by separable kernel [PVV] (reviewed in section .), the authors describe a trick to avoid the bilateral staircase effect. The idea is to compel a minimal isotropic smoothing effect everywhere in the image, independently from the image content. This is realized by constraining the bilateral kernel.

In the separable kernel method, the filtering is realized through a horizontal D filter followed by a vertical D filter. In both D kernels, independently from the spatial parameter σ s , the authors consider a centered sub-window with one pixel radius. Thus, they consider the three pixels at the center of the kernel. Their values are constrained in order to ensure a minimal smoothing effect: the two side pixels values are set to be greater than or equal α-times that of the center pixel. They chose α = 0.25 in their implementation, so that the minimum smoothing kernel (in the centered sub-window with  pixels width) is 1 6 [1; 4; 1]. This way, the authors force a minimal smoothing effect everywhere, even at very sharp edges. This trick, however, cannot help for more low-frequency edge sharpening. We do not integrate this kernel modification in Algorithm  (in section .) in order to keep it simple. The modification would be simply to add after line  (before the normalization):

w d (x, x -d) ← max {αw d (x, x), w d (x, x -d)} w d (x, x + d) ← max {αw d (x, x), w d (x, x + d)} ,
with d = (1, 0) when processing in the horizontal direction, and d = (0, 1) when processing the vertical one.

. Blending at edges in the piece-wise linear bilateral filter

In [DD], bilateral filter is interpreted as a robust estimator. The authors state that at edges, the estimator, namely the bilateral filter, has not enough information available for a precise estimation of the base layer: the statistical estimator computed at these pixels has access to little data, leading to a high uncertainty. Hence, their correction is to blend the filtered signal with the original image where the number of neighbors used for the average computation is small. This number is directly given by the normalization factor. More precisely, the authors' idea is to linearly interpolate between the filtered image FBF{u} and FBF{u}, according to the logarithm of the normalization factor log C, where FBF{u} = G σcorr * FBF{u} is a smoothed version of the filtered image. We call the interpolated image FBF{u} corr . One has FBF{u} corr → FBF{u} when C is high (which means a large number of neighbors, in the bilateral definition) and FBF{u} corr → G σcorr * FBF{u} when C is small (for edges, corners, isolated pixels). The authors take σ corr = 2 in practice. They use the logarithm of C "because it better extracts uncertain pixels". We recall the definition of the normalization factor (given in equation .):

 (a) input u (b) 3(u-FBF{u}) (c) √ u+3(u-FBF{u}) (d) coefficient α (e) 3(u -FBF corr {u}) (f ) √ u+3(u-FBF corr {u})
C(x) = y k(x, y) = y G σs (x -y)G σr u(x) -u(y) . (.)
Let α be the linear interpolation coefficient between FBF{u} and FBF{u}. This coefficient varies with log C. The function α = f log(C) missing in the paper, we define

α(x) = log C(x) log C max , (.)
where C max is the maximal possible value for C, i.e.C max = y G σs (xy). The corrected image is then

FBF{u} corr (x) = α(x)FBF{u} + 1 -α(x) FBF{u}(x) (.)
This correction resembles the unnormalized bilateral filter, extensively described in Section ., in that it blends the filter's result and its second smoothed version according to the normalization term. In UBF, the second image is the input image itself and the blend term the normalization term itself. Note that UBF reduces indeed the staircase artifact yet does not completely avoid it.

We show in Figure . and Figure . that this correction is not well adapted to correct the staircase effect. Although it indeed alleviates the staircase effect at thin and sharp edges, it fails to remove it in the other cases. A smoothing with σ corr = 2 seems to be often too small to compensate for the over-sharpening created by the bilateral filter, and, more importantly, the normalization factor seems not to be a good detector for the staircase effect, because it principally detects the center of the edges, whereas the staircase correction should concentrate on the borders of the edges.



(a) input u (b) 3(u -FBF{u}) (c) √ u + 3(u -FBF{u}) (d) coefficient α (e) 3(u -FBF corr {u}) (f ) √ u + 3(u -FBF corr {u})

. Gradient reversal removal with the Poisson equation

The algorithm for tone management published by Bae et al. in  [BPD] uses the bilateral filter to decompose the image in two layers (base and detail), which histogram are modified so as to match the style of a target image. Because of the bilateral filter, and because their technique can strongly increase the detail, their result presents gradient reversals. The authors address this problem by constraining the gradients of the detail layer to be of the same sign and inferior than or equal to the gradients of the input image. Let u be the input image and d the detail: they build the gradient field v = (x v , y v ) :

x v =      0 if sign(∂d/∂x) = sign(∂u/∂x) ∂u/∂x if |∂d/∂x| > |∂u/∂x| ∂d/∂x

. Selective diffusion

In their excellent paper [KS], M. Kass and J. Solomon generalized the fast strategies brought by the literature for the bilateral filter using local histograms [Por, Por, Wei, PH] and proposed a wide variety of efficient filters that can be expressed in terms of local histogram operations (median filter, erosion, dilatation, bilateral filter, mean-shift and a novel closest-mode filter, dominant-mode filter, histogram equalization etc.) with arbitrary spatial kernel, and in particular they show how all of these filter can be computed in constant time (O(N ), where N is the number of pixels) using a Gaussian spatial kernel. Moreover, they present a particularly clever way of removing the over-sharpening (that we call here the staircase artifact for the bilateral filter) arising in most of those filters. We quote below the authors (we updated the notation for the sake of this dissertation's consistency):



Local image histograms alone say nothing at all about the spatial layout of their data samples [Koenderink and Doorn ]. They contain no indication of a gradual spatial shift from one mode to another. Thus, in order to track a blurred edge accurately, more information must be extracted from the original images. We propose extracting this information by supplementing edge-preserving histogram-based filters with a diffusion step. Our basic observation is that wherever blurring our edge-preserving filter causes it to get closer to the original, the blurred version is preferable as a base layer.

Let F {u} be the output of an edge-preserving smoothing filter. Our goal is to construct a modified output image F {u} which is diffused from F {u} anywhere that diffusion causes it to agree more closely with the original input image u. We will do this iteratively, considering a variety of different Gaussian blurring kernels G σ i in turn. In our experience, sampling the blurs by ratios of √ 2 works well. Let F {u} 0 = F {u} be the original output of the filter. Then we will construct F {u} i from F {u} i-1 by selectively blending between F {u} i-1 and a blurred version b i = F {u} i-1 * G σ i . An important observation is that we only want to update a pixel with a blurred version if an entire region around that pixel of size σ i is improved by the blurring. Accordingly we construct error metrics to measure the local L 2 deviation of the unblurred and blurred versions from the original image:

err u = F {u} i -u 2 * G ησ i (.) err b = b i -u 2 * G ησ i (.)
where η controls the region size. We have found η = .2 works well. Let r = err b /err u be the ratio of the error of the blurred version to the unblurred version.

Where r is larger then one, we prefer the unblurred version. Where r is smaller, we blend towards the blurred one. The exact blending is probably unimportant. The particular formula we use is Although this method works globally well, it seems unable to remove the staircases everywhere, especially in the corners (see Figure .(c)). Furthermore, it is not computationally efficient. Indeed, numerous iterations are needed to correct the staircase effect, and this computation time adds to the computation time of the filter itself. Algorithm  describes the pseudo-code of this method.

F {u} i =    b i r < .5 2(r -.5)( F {u} i-1 -b i ) + b i r ∈ [.5, 1) F {u} i-1 r ≥ 1 (.)  (

. Symmetric bilateral filter

In , M. Elad [Ela] proposed an improvement of the bilateral filter in order to treat piecewiselinear signals. As described by Paris et al.in their excellent book [PKTD], the modification consists in comparing the intensity of the filtered pixel with the average of another pixel and its symmetric point

BF sym {u}(x) = 1 C sym (x) y G σs (x -y)G σr (v(y) -u(x))v(y), (.)
where v(y) = u(y) + u(2xy) /2 is the average between the two symmetric pixels (with respect to x).

In a very similar way to Elad's symmetric bilateral filter, we shall introduce here a method to prevent the bilateral filter from creating staircases. The modification is rather simple, but unfortunately not well adapted to fast implementations.

The symmetric bilateral filter (SBF) computes the actual bilateral filter kernel at each pixel, then takes the minimal kernel value for each pair of symmetric pixels of the kernel. In other terms, the bilateral filter kernel is made symmetric by taking only the minimal values, which ensures that the edge-preserving property is kept. Indeed, this process can only reduce the weights associated to neighborings pixel, which means that their influence can only be reduced in the averaging (or kept as it is, if the symmetric weight is identical). But pixels with large intensity difference do not see their weight increased (before normalization), thus SBF still preserves the edges.

On the other hand, as the bilateral filter's kernel cannot be asymmetric after this modification, the staircase effect is removed. Indeed, it is precisely the asymmetry of the bilateral kernel that produces staircase effects at strong edges (see Figures . and ., in Section  on the staircase effect).



Algorithm : Selective diffusion [KS] input : filtered image F {u} input : input u input : Ratio for sampling the blur α (recommended α = √ 2) input : Size of the region for the error metric η (recommended=η = .2) output: corrected image without over-sharpening

F {u}  σ ← σ min // initialization  F {u} 0 ← u // initilalization  i ← 1  while σ ≤ σ max do  b i ← F {u} i-1 * G σ // smooth previously corrected image  err u ← ( F {u} i-1 -u) 2 * G ησ // equation 5.5  err b ← (b i -u) 2 * G ησ // equation 5.6  r ← err u /err b // ratio of errors  foreach pixel x ∈ u do // blending using equation 5.7  if r(x) < .5 then  F {u} i (x) ← b i (x)  else if r(x) ∈ [.1, 1) then  F {u} i (x) ← 2 r(x) -.5 F {u} i-1 (x) -b i (x) + b i (x)  else  F {u} i ← F {u} i-1  i ← i + 1  σ ← ασ  F {u} ← F {u} i-1  return F {u}



The only difference with Elad's symmetric bilateral filter [Ela] is that we take the minimal value taken by the range kernel on the two symmetric values, rather than computing both values according to the distance between their mean value and the central pixel.

We recall the form of the bilateral kernel (before normalization):

k(x, y) = G σs (x -y)G σr u(x) -u(y) . (.)
Starting from k, the symmetric bilateral kernel k sym is defined as

k sym (x, y) = min k(x, y), k(x, x -(y -x)) . (.)
As usual, the kernel is normalized by setting

w sym (x, y) = z k sym (x, z) -1 k sym (x, y). (.) We show in Figure .
 some examples of kernels of the original versus symmetric bilateral filters.

In Figure . we present the results of filtering for two images and experimentally verify that the staircase effect is removed. However, since the number of pixels averaged at each position of the output image is generally smaller (it cannot be greater, and often symmetrizing the kernel leads to the "exclusion" -weights put to zero -of many pixels), the filter's capacity to remove noise is diminished. More specifically, near a strong edge one can expect that the filtering of the noise or texture will be inexistent. Indeed, near the edge, for each strong coefficient there will be a very small coefficient on the other side of the edge. Thus the smoothing effect is altered and one should observe a texture halo artifact. In Figure ., we display the filtering result of SBF versus BF. One can verify that the denoising capacity of the symmetric bilateral filter is seriously diminished at borders. In particular at the corners of the light gray square, the number of neighbors used in the averaging is reduced to zero. Hence, these pixels are simply not denoised. The structure halo is, however, less visible than the one observed with the guided filter (discussed in Chapter ). Indeed, unlike the guided filter which completely stop filtering when contrasted regions enter its neighborhood, the symmetric bilateral filter continue averaging in the direction parallel to the edge. In other terms, the guided filter stops filtering at edges and the symmetric bilateral filter simply reduce its robustness to noise (because the mean is estimated with fewer pixels), to such an extent that some pixels are not denoised at all. Furthermore, one can expect this situation to be rather common in more complex images, where the content is rarely symmetrical. For this reason and the unsuitability to fast implementation, we do not consider this method as a valid option for base+detail decomposition.

. Trilateral filter

The trilateral filter consists in two bilateral filters: the first one is a standard bilateral filter on the gradients of the input image ∇u, the second one is a slightly modified bilateral filter where the range weights are computed using the intensity difference between the current pixel and a plane P rather than the central pixel of the current window. Let BF{∇u}(x) be the output of the first step, i.e.the filtering of the gradient of the input image:

BF{∇u}(x) = 1 C(x) y G σs (x -y)G σr ∇u(y) -∇u(x) ∇u(y), (.)
where C is the normalization factor computed using the gradients as well, Thanks to its symmetry, the staircase effect is avoided. On the other hand, the number of pixels used in the averaging process is systematically less than or equal to the number of pixels used in the original bilateral filter, thus reducing its denoising property.

C(x) = y G σs (x -y)G σr ∇u(y) -∇u(x) . (.)  (
The filtered gradient field is then used to define a plane P at each pixel:

P (x, y) = u(x) + yBF{∇u}(x). (.)
This plane is used in the second step. It is removed from the data in the modified bilateral filter that we denote by TF (for trilateral filter, although it is rather defined by the association of two filters).

By removing the plane P that locally approximates the signal, the authors "tilt" the bilateral kernel, as shown in Figure .(c). The second bilateral filter would normally be defined as

TF{u}(x) = u(x) + 1 C(x) y
G σs (xy)G σs u(y) -P (x, y) u(y) -P (x, y) , (.) with C updated accordingly as

C(x) = y G σs (x -y)G σs u(y) -P (x, y) . (.)
But the authors add in this modified bilateral kernel a third term, the function f θ . This function aims at avoiding the averaging of pixels with dissimilar gradients. Quoting them: "Tilting greatly improves smoothing abilities of the trilateral filter in high gradient regions, but also ensures that the filter window can extend beyond local boundaries into regions of dissimilar gradients. Unless we exclude these regions from the filter window, the trilateral filter will blunt or blur sharp ridges and corner-like features where the bilaterally smoothed gradient BF{u} changes abruptly (e.g.arrow  in Figure .(b))" [CT]. The function f θ then excludes pixels which gradient is too different from the current pixel's gradient. It is defined as follows:

f θ (x, y) = 1 if BF{∇u}(y) -BF{u}(x) < R 0 otherwise. (.)
Yet this definition does not ensure that the neighborhood is a connected region, as they require.

Rather than computing this function (which is also time consuming), the authors use a "stack of min-max gradient images", a pyramid-based structure (where each level keeps the original image size) where in each level is stored, for each pixel, the values min and max in neighborhood with increasing size. We refer to the author's paper [CT] for a more detailed description  . To find the   largest (connected) region where f θ = 1, one simply needs to find, for each pixel, the highest level of the pyramid in which the min and max values are within BF{∇u}(x) ± R. The final trilateral filter is thus defined as

TF{u}(x) = u(x) + 1 C(x) y G σs (x -y)G σs u θ (x, y) f θ (x, y)u θ (x, y), (.)
where u θ (x, y) = u(y) -P (x, y). The normalization factor is updated accordingly as

C(x) = y G σs (x -y)G σs u θ (x, y) f θ (x, y). (.)
Due to this modification, the trilateral filter averages only connected pixels. The output of this filter, rather than piecewise constant, is piecewise linear. The shocks are moved to the nd order derivative.

Although this algorithm has seven internal parameters, only one (σ θc ) is left to the user; the authors proposed strategies to automatically set the others. Algorithm  and Algorithm  carefully detail each algorithmic step of the method  . f  We found an error in the paper: according to their description of the min-max stack construction, i.e."each pixel (m, n) in any nonzero level K holds min and max values for the 3×3 surrounding pixels found in level (K -1) at (m + [0, ±2 K-1 ], n + [0, ±2 K-1 ])", the size of the equivalent window at level  is (2 K+1 -1) × (2 K+1 -1) and not (2 K + 1) × (2 K + 1) as given is the paper [CT]. Indeed, the radius r of the min or max filter at a level K is r(K) = 2 K-1 , and then the equivalent radius at level  is r(K) = K n=1 r(n) = 2 K -1 (geometric series with ratio ). Hence, the width of the square equivalent neighborhood is 2r + 1 = 2 K+1 -1.

 The authors' implementation ("example code" given at http://www.cs.northwestern.edu/~jet/ publications.html) is inconsistent with the paper description of the minStack algorithm. Indeed, the neighborhood they consider in the min and max filters at pixel

(m, n) is (m + [0, ±1], n + [0, ±1]) instead of (m + [0, ±2 K-1 ], n + [0, ±2 K-1 ]
) as explained in the paper [CT]. Its makes the equivalent neighborhood size at level  fall to (2K + 1)×(2K + 1). We take this implementation as a reference in the pseudo-code we give here. The implementation differs for the computation of σ rθ too: the averaging in a circular neighborhood mentioned in the paper is absent from the implementation.



Algorithm : Trilateral filter [CT] input : input u input : spatial parameter σ sθ output: trilateral filtered image TF{u} // Compute the image gradients ∇u using forward differences

 foreach pixel x do  ∇ x u(x) = u(x + 1, y) -u(x, y)  ∇ y u(x) = u(x, y + 1) -u(x,
num(x) ← y G σs (x -y)G σr ∇u(y) -∇u(x) ∇u(y)  denom(x) ← y G σs (x -y)G σr ∇u(y) -∇u(x)  BF{∇u}(x) ← num(x)/denom(x) // equation (.)  Compute minStack{ ∇u } // Algorihtm 15  Compute maxStack{ ∇u } // Algorihtm 15 // Apply trilateral filter to u  foreach pixel x do  K ← largest K that satifies both conditions: minStack{ ∇u }(x, K) ≥ BF{∇u}(x) -R and maxStack{ ∇u }(x, K) ≤ BF{∇u}(x) + R  f θ (x) ← unit square centered in x of width 2K + 1  u θ (x, y) ← u(y) -u(x) -yBF{∇u}(x) // "un-slanted" image  num(x) ← y G σs (x -y)G σs u θ (x, y) f θ (x, y)u θ (x, y)  denom(x) ← y G σs (x -y)G σs u θ (x, y) f θ (x, y)  TF{u}(x) ← u(x) + num/denom //
 stack{u, 0} ← u // initialization  foreach level k ∈ {1, 2, . . . , N } do  foreach pixel x do  stack{u, k}(x) ← stack{u, k -1}(x)  foreach pixel y in a 3×3 window do  stack{u, k}(x) ← filter{stack{u, k}(x), u(y)}  return stack{u}  
As we saw in the previous chapters, the bilateral filter has rapidly become ubiquitous in image processing and is now used in a tremendous number of applications. The original filter, invented by Yaroslavsky () [Yar] and Lee () [Lee], studied by Smith and Brady () [SB], and reproposed by Tomasi and Manduchi () [TM] needs to compute a different kernel at each pixel which makes it slow, nay not affordable for large images and (consequently) large spatial support. Hence the need for a fast implementation of the filter. In this chapter, we review the numerous fast bilateral filter of the literature. The history of the fast bilateral filter starts with the fast Durand and Dorsey approximation () [DD], who presented the original idea, that would be extensively explored later, of sampling the intensity range so as to linearize the convolution. The Gaussian convolution can then be computed using one of the numerous fast schemes available. As we shall see, no fast and exact implementation of the bilateral filter has been proposed yet. Thus the competition between the numerous proposed schemes not only lies in the speed but also on the precision and the unavoidable artifacts. Furthermore, for several schemes the speed depends on the parameters used and on the dimension in which the filter operates. Thus we eventually present a palette of effective filters rather than a definitive winner.

. Introduction

The first fast bilateral filter was proposed by F. Durand and J. Dorsey in  [DD]. They introduced the fundamental idea of linearizing the convolution by applying the formula only on a reduced set of intensity samples. This method is called the piece-wise linear approximation, or layered approximation [PKTD]. This is a layered approximation where each intensity sample defines a layer on which a linear convolution can be applied. One can then use the fast Fourier transform or appropriate sub-sampling to speed-up the linear Gaussian filtering step. The filtered layers are combined to produce the approximated bilateral filter. This work is fundamental and paved the way to later accelerations. T.Q. Pham and L.J. Van Vliet [PVV] proposed in  a different way to accelerate the filter by presenting a separable bilateral filter. The following year B. Weiss [Wei] introduced an acceleration of both median and bilateral filters using distributive histograms. S. Paris et al. proposed improvements of the layered approximation in two others publications, with the article of S. Paris and F. Durand in  [PD] (extended in a  journal paper [PD]) and the bilateral grid of J. Chen, S. Paris and F. Durand in  [CPD]. Concurrently, G. Guarnieri [GMR] improved the  Durand method both in quality and execution time, by inverting the order of the division and the linear convolution (as in [PD]) and suggested to use recursive filtering for the Gaussian filtering implementation. In  came out the first O(N ) bilateral filter by F. Porikli [Por], using integral histograms, thus improving upon  Weiss's  filter. It is followed one year later by another O(N ) filter by Q. Yang, K.H. Tan and N. Ahuja [YTA] that extends Durand and Dorsey's  paper.

Another kind of fast approximation uses the Gauss-polynomial decomposition. This started in  by the publication of another O(N ) bilateral filter by Chaudhury, Sage and Unser [CSU]. It gave rise to other papers using the same idea [Cha], [Cha], [Cha], [SK] [GCb], [GCa].

Using the "range-space" domain introduced by Durand and Paris [DD], [PD], some extended it to higher dimensions, allowing fast color-weighted bilateral filtering. In , A. Adams, N. Gelfand, J. Dolson and M. Levoy [AGDL] published the Gaussian kd-tree. In  the same authors published a similar approach for fast bilateral filtering in high-dimensional spaces [ABD], still using slicing.

One main interest of those high-dimensional filters is to accelerate the non-local means algorithm. Indeed, this denoising filter is a bilateral filter where the weights are computed according to the distance between patches rather than between pixels intensity values. The performance of these methods is not competitive for gray-scale bilateral filters because they spend much extra time preparing the data structures. These filters are specially useful for denoising, where it makes sense to use color information. In the base + detail decomposition, we generally do not work with color, as it contains little useful additional information compared to the luminance. So we shall consider that those high dimensional filters are out of our current scope.

Given the limitations of the bilateral filter, many new designs of fast edge-preserving filters have been investigated. The O(N ) time Edge-Avoiding Wavelets (EAW) [Fat] are wavelet transforms with explicit image-adaptive weights. But the kernels of the wavelets are sparsely distributed in the image plane, with constrained kernel sizes (to powers of two), which may limit the applications. In , Gastal and Oliveira [GO] proposed another O(N ) time filter known as the Domain Transform filter. The key idea is to iteratively and separably apply D edge-aware filters. The O(N ) time complexity is achieved by integral images or recursive filtering. This filter is particularly useful for color images. Although this filter is not an approximation of the bilateral filter, it is worth considering, in this review, as its smoothing effect is relatively similar and its execution time very small. This filter, as well as the guided filter, can be considered as an alternative to fast bilateral filters. We summarize in Table . the list of fast approximations and give their complexity.

This chapter has been inspired by the book by Paris et al. on the bilateral filter [PKTD]. The differences with our review are highlighted in Chapter  on the bilateral filter.

. Separable kernel

In , T. Q. Pham and L. J. Van Vliet [PVV] proposed the separable bilateral filter. This very simple acceleration applies two consecutive one-dimensional bilateral filters to the input image, one for each dimension. Although the bilateral filter is not separable, the results aren't very far from the true bilateral. But this fast filter remains a poor approximation and acceleration. Its main inconvenience is its inability to properly filter the textures.

Algorithm  works as follows: first, a horizontal bilateral filter is applied to the input image (lines -). The output image FBF sep. is obtained by applying a second one-dimensional bilateral filter in the vertical direction to the previous result (lines - again). The intermediary image is denoted by v in the pseudo-code (line ). This separable version has a O(N σ s d) complexity instead of O(N σ d s ) for the true bilateral filter (where σ s is the radius of the window and d is the dimensionality).

Used with a small radius or a small intensity parameter σ r , this approximation is faster than the layered approximation [DD]. Indeed, the complexity of this last approximation decreases with these parameters: when σ r is large, the number of required layers become smaller. As for the  Section Name and authors

Complexity

.

Brute force [SB], [TM] O

N σ d s . Separable kernel [PVV] O N σ s d . Local histograms [Wei] O N log(σ s ) . Integral histograms [Por] O N b .
Layered approximations and the bilateral grid [DD], [GMR], [PD], [PD], [YTA], [CPD] O

N + N σ 2 s R σr
. Polynomial range kernels [CSU], [SK], [Cha], [Cha], [Cha], [GCb], [GCa], [NPC] O spatial parameter σ s , when it is set to a large value, one can subsample the layers more aggressively and then reduce the computational complexity. This algorithm is described in Section .. On the other hand, the complexity of the separable kernel approximation increases slowly with the dimensionality, unlike other implementations. This separable kernel idea is also used in the more recent work by Gastal et al. [GO], where they describe an effective way to avoid the apparition of vertical or horizontal lines by iterating the filter while reducing its spatial parameter σ s . We refer to Section . for more details on this trick.

We present in Figure . the results of the application of the separable bilateral filter and compare it to the exact bilateral filter. One can see some vertical lines appearing. This is the drawback of this implementation. The very sharp transitions are nevertheless well preserved.



Staircase effect correction

As discussed in section , the bilateral suffers from an "over-sharpening" at strong edges. Interestingly, the authors brought their own solution: they compel a minimum smoothing effect everywhere by constraining the shape of the bilateral kernel. We shall describe it in section ..

. Local histograms

In this section, we present two methods [Wei], [Por] , to accelerate bilateral filters using constant spatial filters (box filters), and arbitrary range kernels. Their key observation is that when using constant spatial weights, a bilateral filtering amounts to a weighted average of the local histogram (namely the histogram of the current patch). Thus efficient non redundant schemes to compute local histograms yield fast bilateral filters.

In the first publication using that strategy [Wei], B. Weiss was more concerned by the median filter than by the bilateral filter. Nonetheless, as the method -by using a hierarchy of partial histograms -computes efficiently local histograms, the bilateral filter is presented as an extension of his work. Two years later, F. Porikli [Por] published three efficient ways to compute bilateral filters. Among them, one relates to the bilateral filter with a spatial box kernel. It uses integral histograms, another fast way to compute local histograms.

In the special case of a spatial box filter, the bilateral filter weights do not depend on the distance to the center of the patch: they only depend on the intensity of the pixels in the patch. Then a histogram of the current patch is sufficient to compute the bilaterally filtered value. Let us assume that at each image pixel x the intensity histogram h Ω is computed from an x-centered box window Ω with radius r and width 2r + 1. The spatial box kernel k S allows rewriting the standard bilateral filter's equation

FBF loc.hist. (x) = 1 C(x) y∈Ω k S y G σr u(x -y) -u(x) u(x -y) , C(x) = y∈Ω k S y G σr u(x -y) -u(x) , using the local histogram: FBF loc.hist. (x) = 1 C(x) j h Ω (j)G σr j -u(x) j , (.) C(x) = j h Ω (j)G σr j -u(x) ,
where j belongs to the discrete intensity range of the input image and h Ω (j) is the local histogram value at pixel x and for intensity j.

The sum over the intensity j doesn't depend on the window size any more, thus making the equation . run in constant time per pixel. What determines the overall complexity of the algorithm is the dynamic range and the way the local histogram is computed.

The two methods we are going to present are based on the storage of intermediate local histograms. Let us start with B. Weiss's algorithm. This method succeeds in reducing the number of needed operations to update a row of local histograms. It improves on the idea underlying Huang's algorithm [Hua]. Huang's algorithm strategy is to compute the output column per column, with a sliding window along the rows. The local histogram of the current column is updated at each new row by adding and subtracting the few pixels that entered or left the current  window. Weiss's observation is that this scheme still has a lot of redundant operations: for two consecutive columns, the major part of the added and removed pixels to the local histogram are the same. Thus, while keeping the sliding window idea, he proposes to compute all the columns at the same time, using a wisely designed set of histograms. Indeed, updating a local histogram for each column does not reduce the complexity; whereas updating the set of histograms (by adding and removing a row of pixels) requires less updates, because of the structure of this set. This set of histograms is composed of one large histogram that is a rough approximation of the local histograms, and of several smaller histograms that refine that histogram to the exact one. On the other hand, Porikli's algorithm uses integral histograms: as the integral image enables the computation of the mean of any rectangle in an image with very few operations, the integral histograms give a quick way to obtain the histogram of any rectangle in the image, and from this to derive a fast bilateral filter.

Weiss's algorithm

Ben Weiss [Wei] introduced in  a fast algorithm for median filtering, also useful for bilateral filtering with a spatial box kernel. This fast median filter is O(N log r) instead of O(N r) for the fastest previous method (Huang,  [Hua]), where r denotes the radius. The proposed fast bilateral filter is an exact implementation of a bilateral filter with spatial box kernel and any range kernel. This implementation is only valid for gray scale images. It could be extended to color images by using three-dimensional histograms. Note that this would considerably increase the computational time, as the convolution of the histogram would be in D (a local histogram is a 256 × 256 × 256 volume for a  bits color image) instead of D. Its complexity is O(N log r). This paper was published after the Durand-Dorsey [DD] approximation in  but also after the  Paris-Durand [PD] improvement of the above-named paper. B. Weiss explains that the Paris-Durand-Dorsey approximation suffers from drawbacks that his methods have not: first, the layered approximation "is not translation-invariant: the exact output is dependent on the phase of the subsampling grid". Second: "the discretization may lead to a further precision loss, particularly on high-dynamic-range images with narrow intensity-weighting functions". It is indeed remarkable that although the box-shaped spatial kernel isn't standard, B. Weiss's algorithm is a fast yet exact implementation of a bilateral filter.

Let us start with Huang's fast median filter [Hua], which has inspired the B. Weiss version. The pseudo-code is given in algorithm . Note that indices in this pseudo-code may be negative or higher than the size of the input image I. We deliberately choose to not handle the image borders, in order to clarify the pseudo-code. This algorithm can actually be used to compute a bilateral filter (with constant spatial kernel) by replacing line  with  lines:

. Compute numerator: num ← j h Ω (j)G σr j -u(l, c) j . Compute denominator: denom ← j h Ω (j)G σr j -u(l, c) . FBF loc.hist. (l, c) ← num/denom,
where j is an intensity and h Ω (j) the local histogram value of the square window Ω of size (2r+1) 2 for the intensity j.

Huang's algorithm proceeds column per column (line ), with a sliding window (line ) which direction changes (line ) from one column to the next one, so as to minimize the number of updates needed to get the correct histogram h Ω . With such a "snake-shaped" path, the number of pixels to be added or subtracted at each window displacement (that is to say: for each output pixel) is 2 × (2r + 1). Once the local histogram h Ω is computed (lines  and ), it is easy to  Algorithm : Huang's Fast Median filter [Hua] B. Weiss's observation concerning Huang's algorithm is that there are still a great amount of redundant calculations: although we save time by wisely updating the histogram when going through the rows, each pixel is still added and subtracted from 2r + 1 windows because the process is repeated for each column. B. Weiss then proposes an efficient scheme that avoids this redundancy.

First, the only way to save calculations is to compute all the columns at the same time. This means that instead of updating one histogram like in Huang's version, Weiss's version updates all histograms of the current line (that is to say: updates the histogram of each row). But at this point, as B. Weiss says, this is just a rearrangement of operations; the runtime is unchanged. Note that this rearrangement needs to store one histogram for each row of the output image. On the other hand, the median filter is local, so its exact computation can be obtained from the original image cropped in several smaller images, leading to less memory consumption.

Weiss's algorithm takes advantage of the distributive property of the histogram:

H A∪B (u) = H A (u) + H B (u)
to reduce the number of operations required to update the histograms of a row. The main idea is to store one large partial histogram and several small other partial histograms, creating a "set of histograms" H * . The histogram H c of the column c is obtained by the distributivity property, which amounts to add several histograms. The histogram values can be negative.

In function of the number T of histograms used to decompose each row histogram, the complexity of this algorithm is O(N √ r) with T = 2; O(N 3 √ r) with T = 3; O(N log r) with T = O(log r). The general pseudo-code, valid for any of the above-mentioned algorithms, is described in algorithm . As in algorithm , the fast median filter can easily be transformed in a fast bilateral filter by replacing at lines  and  the "find median value" by "compute bilateral output from local histogram using equation .".

At lines  and  we add a line to the set of histograms H * . This is done simply by adding or 

Porikli's algorithm

Following his  paper that provides a strategy to efficiently compute local histograms [Por], Fatih Porikli published in  a paper describing a fast bilateral filter with a box spatial kernel, that takes advantage of the local histograms [Por]. This paper actually gives three efficient ways to approximate the bilateral filter. We just introduced the first one, which we are going to present in this section. The second one uses a polynomial range kernel. This is described in section ..

In the third proposition, F. Porikli shows that Gaussian range and arbitrary spatial kernels can be expressed by Taylor series as linear filter decompositions.

The single modification F. Porikli brought in B. Weiss's algorithm is the integral histogram. But this nevertheless allows to decrease the complexity from O(N log r) to O(N ), as extracting local histograms from the integral histogram has a complexity independent of the window radius r.

The integral histogram is the storage, at each pixel, of the histogram of the (rectangle) region between the origin and the current pixel. The last pixel therefore is the histogram of the whole image. From this integral histogram, it is easy to extract local histograms of any radius, by using the four histograms disposed at the corners of that region. Let's write H(x, y, b) the integral histogram at position (x, y), and Ω the rectangular region, which goes from the top-left pixel (x top , y top ) to the bottom-right pixel (x bot , y bot ). The local histogram value at any bin b is then denoted h Ω (b) and obtained as follows:

h Ω (b) = H(x top -1, y top -1, b) -H(x bot , y top -1, b) -H(x top -1, y bot , b) + H(x bot , y bot , b) . (.)
Hence, one needs only three arithmetic operations per bin to compute the local histogram of a rectangle window of any size. Now, all this would be useless if the integral histogram construction needed to literally compute each region histogram from the origin to the current pixel.

Fortunately, it can be computed recursively, in a way that avoids any redundant calculation. Call Q(u(x, y)) the bin of the current pixel, then

H(x, y, b) = H(x -1, y, b) + H(x, y -1, b) -H(x -1, y -1, b) + Q(u(x,

Using a smaller number of bins in the histogram

One can use less bins in the histogram than the actual number of intensity values in the input image. This results in less memory consumption and a faster computation. The complexity of this histogram-based fast approximations is indeed dependant of the number of bins b used in the histograms, so that we write it O(N b). For example, B. Weiss suggests either to dither highprecision data into -bits before processing, and notes that it "introduces surprisingly little error", or to downsample the intensity into the histogram which "yields better accuracy".



v(x) ← b h Ω (b)G σr b -u(x) b // bilateral-weighted average  C(x) ← b h Ω (b)G σr b -u(x) // normalization  FBF int.hist. {u}(x) ← v(x)/C(x)

Usage of a box spatial kernel

The two algorithms we just presented use box spatial kernels. And one would probably ask: is there any visible difference between a Gaussian kernel and a box kernel? B. Weiss's answer is that it may indeed create "visual artifacts [that] may resemble faint Mach bands", but adds "these artifacts tend to be drowned out by the signal of the preserved image". Actually, an imperfect frequency response is particularly visible when there is a lot of contrast in the signal (imagine a white pixel alone in a dark region and the box-filtered result). Yet the bilateral precisely avoids to blend contrasted regions, thus makes this artifact less visible. In fact, the smaller the range parameter, the lesser this spurious appearance visibility. As proposed by B. Weiss, an iterative scheme would make the box-filter converge to a Gaussian filter. To avoid the cartoonish look, he suggests to iterate the filtering while keeping at each iteration the original intensity to guide the intensity weights.

Related methods

In the same vein, JJ. Francis and G De Jager published in  a paper on a bilateral median filter [FDJ]. They propose to replace the weighted mean of pixels by a weighted median of pixels. We won't present this algorithm further; we just mention here its existence and note that the local histogram implementation of the bilateral filter seems very suitable to this modification.

In , M. Kass and J. Solomon [KS] generalized the use of those local histograms, so that the spatial kernel can have any form. They apply their method to several filters, namely the median, min and max filters, closest mode filter, a "dominant mode" filter and the bilateral filter. They introduce their paper in those words:

Here, we present an efficient and practical method for computing accurate derivatives and integrals of locally-weighted histograms over large neighborhoods. The method allows us to compute the location, height, width and integral of all local histogram modes at interactive rates. Among other things, it enables the first constant-time isotropic median filter, robust isotropic image morphology operators, an efficient "dominant mode" filter and a non-iterative alternative to the mean shift.

In addition they present a method to "combat the over-sharpening that is typical of histogrambased edge-preserving smoothing" (and bilateral filtering). This last part is presented in Section . They call it the selective diffusion.



Perreault et al. [PH] presented in  another paper for fast median smoothing using local histograms, that can also be used for accelerating the bilateral filter.

. Piecewise-linear and bilateral grid approximations

S. Paris and F. Durand have been major contributors to the accelerations of the bilateral filter. F. Durand and J. Dorsey were the first to propose a fast approximation [DD]. Their original idea would later inspire other fast schemes [PD], [PD] and [CPD] and [YTA] to approximate BF.

F. Durand and J. Dorsey [DD] started by the "piecewise-linear bilateral filter". They remarked that fixing the reference pixel intensity in the formula defining the bilateral gives back a regular convolution. One can then compute the exact bilateral filter with R convolutions, where R is the number of intensity values in the image. But this would not accelerate the filter. Hence, the authors proposed to compute the exact bilateral result for a small subset of image intensity values, and to derive the other values by interpolation. To further accelerate the filter, the convolutions are computed on sub-sampled images.

In the bilateral grid [PD, PD, CPD], Paris and Durand present the bilateral filter as a Gaussian filter on the image's graph, hence adding a dimension. Filtering a gray (-dimensional) image with the bilateral filter is therefore equivalent to filtering the graph of the image, viewed as a D sparse image (or in continuous as a Hausdorff measure), with a standard Gaussian kernel. For a color image this leads to filtering a D sparse image. The voxel coordinates in this high-dimensional space are the initial spatial coordinates followed by the color channels treated as coordinates. But these channels are also taken as the image values at the same voxel.

The piecewise-linear approximation

The bilateral filter is not a priori fit to fast computation because its kernel is different at each pixel. This is due to the edge-stopping function G σr u(y) -u(x) . However, consider the bilateral filter equation (.) for a fixed pixel x

BF{u}(x) = 1 C(x) y∈Ω G σs (x -y)G σr u(y) -u(x) u(y), (.) with C(x) = y∈Ω G σs (x -y)G σr u(y) -u(x)
. This is equivalent to the (x dependent) convolution of the function H u(x) : y → G σr u(y) -u(x) u(y) by the kernel G σs . Similarly, the normalization factor w is the convolution of I u(x) : y → G σr u(y) -u(x) by G σs . The only dependency on pixel x that makes it differ from a convolution is the presence of the x-dependent value u(x) in G σr . Starting from this observation, the authors' acceleration strategy [DD] is to discretize the set of possible signal intensities into N layers values {γ(i)}, and to compute a linear Gaussian convolution for each such value:

v(x, i) = 1 w(x, i) y∈Ω G σs (x -y)G σr u(y) -γ(i) u(y) (.) = 1 w(x, i) y∈Ω G σs (x -y)H(y, i) (.)  and w(x, i) = y∈Ω G σs (x -y)G σr u(y) -γ(i) (.) = y∈Ω G σs (x -y)I(y, i) (.)
The final output FBF piecewise of the filter for a pixel x is then a linear interpolation between the output v(x, i) of the two closest values γ(i) of u(x). This corresponds to a piecewise-linear approximation of the original bilateral filter (note however that it is a linearization of the whole functional, not of the influence function). The pseudocode is given in Algorithm . All operations in that pseudo-code are pixel-wise. At line  the intensity sample is computed, and used at lines  and  to compute what we call a layer of the image at the intensy γ(i) (layer I is the normalization). The layers are convolved with the Gaussian spatial kernel (lines  and ) then divided to get the bilateral result for the current layer (line ). The output image is updated at each layering (line ) using linear interpolation weights (line ).

I  I(i) ← G σr u -γ(i)  H(i) ← H(i) * G σs // convolve layers  Ī(i) ← I(i) * G σs  v(i) = H(i)/ Ī(i) // bilaterally filtered layer i  α(u, i) = max(0, 1 -|γ(i) -u|/a) // interpolation weights  FBF piecewise ← FBF piecewise + α(u, i)v(i) // update output  return FBF piecewise
The recommended number of layers is D/σ r , where D = max u-min u is the image dynamic range of the input image. Thus the minimal allowed sampling rate of the range Gaussian kernel is 1/σ r . Indeed, one needs enough layers in order to correctly interpolate the filtered values that fall between the layers. The authors [DD] use linear interpolation.

These same authors propose two different strategies for the Gaussian convolution. The first, exact, uses the fast Fourier transform, with O(N log N ) complexity. This makes the filter complexity fall from O(N σ 2 s ) (for the original bilateral filter with truncated spatial kernel) to O(N layers N log N ). But the filter can be further accelerated by computing the convolutions on subsampled versions of the images. The strategy is then to strongly subsample the image without respecting the Nyquist-Shannon sampling theorem, which allows a very fast subsampling. The authors [DD] use the nearestneighbor algorithm, "because it does not modify the histogram". Remark that this is a dangerous  procedure particularly in presence of noise. It was replaced in the  paper [PD] by a box downsampling described in the next section (see Equation (.)). Then, a Gaussian filter with very small kernel is applied on the subsampled images. The bilaterally filtered layer is then upsampled before the output image is updated. The authors do not specify the method used for upsampling. However, the closely-related bilateral grid uses bilinear interpolation with good results, so this method can be safely used here as well. Algorithm  presents the pseudo-code of this filter. The main differences between Algorithm  and Algorithm  appear in red.

Algorithm : Piecewise-linear FBF with subsampling (FBF pw.sub. ) input : image u input : range standard deviation σ r input : spatial standard deviation σ s input : Number of layers N layers input : subsampling factor s output: FBF pw.sub. {u}  FBF pw.sub. ← 0 // initialization  a = (max u -min u)/N layers // gap between two intensity samples

 b = min u  u ↓ ← subsample u, s  foreach sampled intensity γ(i) with i ∈ {0, 1, . . . , N layers } do  γ(i) ← ai + b // current intensity sample  H ↓ (i) ← G σr u ↓ -γ(i) u // compute layers H and I  I ↓ (i) ← G σr u ↓ -γ(i)  H↓ (i) ← H ↓ (i) * G σs/s // convolve layers  Ī↓ (i) ← I ↓ (i) * G σs/s  v ↓ (i) = H↓ (i)/ Ī↓ (i) // bilaterally filtered layer i  v(i) ← upsample v ↓ (i), s  α(u, i) = max(0, 1 -|γ(i) -u|/a)
// interpolation weights  FBF pw.sub. ← FBF pw.sub. + α(u, i)v(i) // update output  return FBF pw.sub.

The bilateral grid

Although the piecewise-linear approximation [DD] does not use the "range-space" domain as presented in [PD], it uses the same ideas. Indeed, they both linearize the convolution and downsample the signal to reduce computational complexity. Moreover, the piecewise-linear approximation can be seen as a layering of this range-space domain. From this point of view, the improvements brought by Paris et al. [PD] are: a more formal definition of this fast approximation thanks to a high dimensional interpretation of images, and a gain in precision due to a better subsampling in the range domain. However, they do not gain more speed-up for the bilateral filter. This is done in  by Chen et al. [CPD] through GPU parallelization.

For the presentation of the bilateral grid, we quote Paris, Kornprobst, Tumblin and Durand [PKTD, PKTD]. Notations have been updated.

Inspired by the layered approximation of Durand and Dorsey [DD], Paris and Durand [PD] have reformulated the bilateral filter in a higher dimensional homogeneous space. They described a new image representation where a gray-level image  is represented in a volumetric data structure that they named the bilateral grid. In this representation, a D image u is represented by a D grid Γ where the first two dimensions of the grid correspond to the pixel position x = (x, y) and the third dimension corresponds to the pixel intensity u(x). In addition, this D grid stores homogeneous values, that is, the intensity value u is associated with a non-negative weight w and stored as a homogeneous vector (wu, w). Using this concept, Paris and Durand [PD] showed that the bilateral filter corresponds to a Gaussian convolution applied to the grid, followed by sampling and normalization of the homogeneous values.

More precisely, the authors consider the S × R domain [S is the spatial domain and R the range domain] and represent a gray-scale image u as defined on a D grid as a D function Γ by

Γ(x, y, z) = u(x, y), 1 if z = u(x, y), (0, 0) otherwise. (.)
With this representation, they demonstrate that filtering bilateral exactly corresponds to convolving Γ with a D Gaussian whose parameters are (σ s , σ s , σ r ) : Γ = Γ * G σs,σs,σr . They show that the bilateral filter output is BF{u}(x, y) = Γ x, y, u(x, y) . This process is illustrated in Figure ..

Using the same arguments as in Durand-Dorsey [DD], the authors subsample the grid (using nearest neighbors) before filtering it with a high-dimensional Gaussian kernel. They recommend using the parameters σ s and σ r to subsample the grid. This yields a complexity

O |S| + |S| σ 2 s |R| σ 2 r ,
where |S| is the size of the spatial domain (number of pixels) and |R| is the size of the range domain.

Algorithm  describes the pseudo-code of this method. The grid Γ is constructed at line  and convolved at line , after box subsampling (line ). Lines  and  can actually be replaced by one single operation:

Γ([x/σ s ], [y/σ s ], [z/σ r ]) ← Γ([x/σ s ], [y/σ s ], [z/σ r ]) + u(x, y), 1 , (.)
with Γ initialized with zeros and where [.] is the closest-integer operator. Upsampling is realized at line . Authors use linear upsampling; Algorithm  describes the pseudo-code of the upsample function. Note that it is not necessary to upsample Γ everywhere but only at voxels x, y, u(x, y) .

In fact, the upsampling can be done on the fly at line  and line . By using equation (.) at lines  and  and upsampling on the fly at lines  and , one avoids the storage of the full resolution grid, and thus saves a large amount of memory. Lines  to  perform the slicing step and the output values are obtained at line  after normalization. In Algorithm , . and . are the closest superior integer and closest inferior integer operators, respectively.

In their book [PKTD], Paris et al. explain the difference between this approximation and the piecewise-linear one [DD]:

The major difference is in the way the downsampling is performed. The layered approximation encounters difficulties at discontinuities: it averages adjacent pixels with different values, e.g., a white and a black pixel ends up being represented by one gray value that poorly represents the original signal. In comparison, the bilateral grid subsampling strategy preserves adjacent pixels with different intensities, because they are far apart along the intensity axis. In the white and black pixels case, the bilateral grid retains the two different values involved and thus is able to produce better results. Figure . illustrates this behavior. The bilateral grid should be preferred over the layered approximation, because both approaches perform equivalently fast. Algorithm : Fast bilateral filter with the bilateral grid [PD] (FBF grid ) input : image u input : smoothing parameters σ s (space), σ r (range) output: FBF grid {u}  Γ ← build the bilateral grid using equation (.)



 Γ ← subsample(Γ, σ s , σ s , σ r ) // equation (.)  Γ ← Γ * G 1,1,1 // 3D Gaussian convolution. Each // component is filtered independently  Γ ← upsample( Γ , σ s , σ s , σ r ) //



Algorithm : upsample function: tri-linear upsampling. input : downsampled grid Γ input : position (x, y, z) of the full resolution voxel input : down and up-sampling parameters σ s and σ r output: linearly interpolated value Γ(x, y, z)

 x ← x σs -x σs  y ← y σs -x σs  z ← z σr -z σr  Γ(x, y, z) ← i∈{0,1} i∈{0,1} k∈{0,1} Γ ( x σs + i, y σs + j, z σr + k)|x -i||y -j||z -k|  return Γ(x, y, z)



The bilateral grid can be extended to cross bilateral filter [ED], [PSA + ] (also called joint bilateral filter) and filtering of color images. Color filtering however is not well suited for the GPU implementation of Chen et al. [CPD]. For the cross bilateral filter, one simply need to construct the grid Γ with the guide image v giving the position of the pixels and the input u giving its value:

Γ(x, y, z) = u(x, y), 1 if z = v(x, y) (0, 0) otherwise . (.)
As S. Paris et al. observed in [PD], Felsberg et al. [FFS] present a method called channel smoothing that is closely related to the bilateral grid. Quoting: Channel Smoothing Felsberg et al. [FFS] described an efficient smoothing method based on a careful design of the intensity weighting function. They showed that Bsplines enable the discretization of the intensity range into a small set of channels. Filtering these channels yields smooth images with preserved edges akin to the output of the bilateral filter. B-splines allowed for a precise theoretical characterization of their filter using robust statistics. The downside of B-splines is the higher computational effort required to handle them.

We refer to their excellent paper for further details.

. Polynomials approximations

The first paper using a polynomial range kernel for the bilateral filter is [Por], in . Following this, the main contributions on this way of approximating the bilateral filter are provided by K.N. Chaudhury. He first published [CSU] with D. Sage and M. Unser. This article generalizes Porikli's work and gives the key element of the next papers: [Cha], [Cha], [Cha], [GCb], [GCa]. One can also find contributions in [SK]. The same method was used in [Get] in automatic color enhancement (ACE), in , to accelerate the filter.

Porikli started with a polynomial function and explained that "a bilateral filter can be interpreted as a weighted sum of the spatial filtered responses of the powers of the original image". Although the following papers use different polynomial functions to approximate the Gaussian range kernel of the bilateral filter, this is the key idea of those approximations. As we will explain soon, choosing the right polynomial function allows to perform a bilateral filter with a series of simple Gaussian convolutions. Those "right polynomial function" have what K. N. Chaudhury called the "shiftability property".

Let's explain this with a trigonometric polynomial. Assume the range kernel has the form

k M σr (t) = M n=-M α n exp(i2πt) n , (.) with i 2 = -1.
Here, σ r stands for the range parameter of the bilateral filter. Set Ω the neighborhood of the pixel x and G σs the spatial Gaussian kernel with standard-deviation σ s . With such a range kernel, the bilateral filter

BF poly. {u}(x) = 1 K(x) y∈Ω G σs (y)k M σr u(x -y) -u(x) u(x -y) (.)  can be written BF poly. {u}(x) = 1 K(x) y∈Ω G σs (y) M n=-M α n exp i2nπ u(x -y) -u(x) u(x -y) = 1 K(x) M n=-M α n exp -i2πnu(x) y∈Ω G σs (y) exp i2πnu(x -y) u(x -y). (.)
The decomposition is the same for the normalization factor,

K(x) = y∈Ω G σs (y)k M σr u(x -y) -u(x) . (.)
The last equation involves a convolution of the image exp i2πnu(x) u(x) with the spatial Gaussian kernel G σs . In other terms, the bilateral filter is obtained by a series of Gaussian convolutions.

Because the range kernel is even, one only needs M + 1 convolutions, where M is the order of the polynomial. Numerous fast algorithms can be used for a fast approximation of the Gaussian convolution (in general with a complexity independent of the spatial parameter σ s , which explains why those algorithms are often referred as O(N ) algorithms). The challenge is then to obtain a good approximation of the range kernel with the smallest possible order M . Indeed, the final complexity of the algorithm is O(M N ) where N is the number of pixels in the image. K.N. Chaudhury describes in [Cha] what he calls the "shiftability property" that allows to use this sort of approximations. Trigonometric polynomials have the desired shiftability property, like any function of the form φ

(x) = c 1 exp(α 1 x) + ... + c M exp(α M x), along with the polynomials φ(x) = c 0 + c 1 x + ... + c M x M .
Here is how he defines a shiftable function:

We say that a function φ(x) is shiftable in R d if there exists a fixed (finite) collection of function φ 1 (x), ..., φ M (x) such that, for every translation τ in R d , we can write

φ(x -τ ) = c 1 (τ )φ 1 (x) + ... + c M (τ )φ M (x).
We call the fixed function φ 1 (x), ..., φ M (x) the basis functions, c 1 (τ ), ..., c M (τ ) the interpolating coefficients, and M the order of shiftability. Note that the coefficients depend on τ , and are responsible for capturing the [action of the translation].

We report in Algorithm  the pseudo-code of the method presented in [GCb, GCa]. In this paper, the authors use the complex exponential to approximate a Gaussian range kernel, with order M :

k M σr (t) = M n=-M c n exp(inωt) . (.)
The bilateral filter numerator can then be written as (.)

In the mentioned paper [GCa, GCb], the authors compute the coefficients c n by minimizing the error between this complex trigonometric polynomial k M σr and a target Gaussian for a fixed order M . One can also compute the Fourier series of the Gaussian range kernel and only use the M first terms. The larger M , the more precise the approximation, but also the more computationally expensive the filter. In Algorithm , the symbol " * " stands for "complex conjugate".

Algorithm : Fast bilateral filter using a polynomial range kernel as presented in [GCb, GCa]: "Shiftable Bilateral Filtering" (FBF poly. ) input : image u input : order M and coefficients c n of the range kernel (-M ≤ n ≤ M ) input : standard deviation σ s of the spatial kernel G σs output:

Approximation FBF poly. {u}  Set P (x) = 0 and Q(x) = 0 for all x  foreach n = -M, ..., M do  G(x) = exp(inωu(x))  F (x) = G(x)u(x)  H(x) = c n G(x) *  Compute F = F * G σs and Ḡ = G * G σs  P (x) = P (x) + H(x) F (x)  Q(x) = Q(x) + H(x) Ḡ(x)
 Set FBF poly. {u} = P (x)/Q(x).

Fast Gaussian convolution algorithms

One needs a fast Gaussian convolution to achieve a fast running time for the algorithm. P. Getreuer [Get] made an excellent survey of the fast Gaussian convolution algorithms that details the filters. Here is his conclusion:

There is no single Gaussian convolution algorithm that is clearly best; the right choice is a consideration of aspects like accuracy, speed, memory, and ease of implementation. The results from this survey suggest the following recommendations (where T is a threshold roughly equal to ):

• For high accuracy, use FIR (finite impulse response filter) for σ < T and Deriche or Vliet-Young-Verbeek for σ ≥ T . • For the best accuracy, use FIR for σ < T and DCT for σ ≥ T .

• For the best speed, use SII (stacked integral images) or box filtering.

• For ease of implementation, use extended box filtering or Alvarez-Mazorra.

We refer to the original article for a description of these different methods. Most paper use FIR and Vliet-Young-Verbeek approximations for the polynomials approximations. This can be explained by the fact that the Fourier approximation is relatively poor (often assuming negative values) on the tails compared to that around the origin. Since the operating region for large pixel differences is precisely the tail, this can result in artifacts around edges.



. Domain transform

Around the same time of the publication of the guided filter [HSTb, HST] by Kaiming He et al. ( and ), Eduardo S. L. Gastal and Manuel M. Oliveira published the domain transform [GO]. Their paper presents a new edge-aware smoothing filter with very short running time. As for the guided filter, it is not an approximation of the bilateral filter. However, as the visual result is close and the running time small, the algorithm is worth considering.

The domain transform's key idea is the definition of a D transform that preserves the geodesic distance between points of the graph x, v(x) (with v a one-dimensional signal) and the real line. That is, the one-dimensional signal v : Ω → R, respectively v : Ω → R 3 (color image), is expressed in the higher dimensional space R 2 , respectively R 4 , then adaptively wrapped to R so that the geodesic distance between samples is preserved.

This method is related to the high-dimensional interpretation of the edge-preserving filters proposed (among others) in [Bar] and in the bilateral grid [PD, CPD]. Furthermore, it has been shown in [SKB] that for a small window, one obtains a bilateral effect by a direct Gaussian diffusion on the image's manifold. Although the domain transform uses the l1-norm metric on the manifold rather than the Euclidean one, a similar interpretation is possible for onedimensional signals. However, the domain transform is not defined for two-dimensional signals, thus we cannot generalize this interpretation further.

Once this isometric transform applied to the signal, a convolution with an isotropic Gaussian kernel can be applied. This convolution is done on a one-dimensional signal, leading to short execution time.

One way to see this transformation is that the intensity differences between adjacent pixels are  transformed into spatial distances by using the geodesic distance on the image's graph. As a result, pixels with distant intensities fall apart. Hence, an isotropic Gaussian filter averages them but little. A difficulty of this process is that the transformed signal is no longer uniformly sampled, which requires specific algorithms for the Gaussian convolution. The authors suggest three different methods, all implementable with an O(N ) complexity (with N the number of pixels).

Filtering of two-dimensional signals is performed in a separable fashion, through successive applications of vertical and horizontal isometric transform and Gaussian diffusion. This strategy resembles the separable kernel approximation [PVV]. The two-passes filtering process (horizontal then vertical or vice-versa) is iterated so that the information is well propagated. The authors recommend three iterations. To avoid the separable kernel approach typical artefact, that is, stripes along the last filtered axis, they come up with a new stratagem: they observe that "the length of the stripes is proportional to the size of the filter support used in the last pass" and thus propose to reduce the filter's standard deviation at each iteration, which successfully remove stripes. The transformation, however, is computed only once in each direction.

Domain transform

The domain transform relies on the vision of the bilateral filter as operating in a D space [SKB, Bar]. For a D RGB color image, this defines a manifold in R 5 . Let x = x, u(x) be a point on this manifold: it is described by its spatial coordinates x and its intensity values u(x). Let F (x, ŷ) be an edge-preserving filter kernel in D and DT{u} the filtered image. It can be generically expressed as

DT{u}(x) = Ω u(y)F x, ŷ)dy, (.)
where Ω F (x, ŷ)dy = 1. The authors [GO] propose to compute the coefficients F (x, ŷ) of the filter in a transformed domain with reduced number of dimensions, so that the evaluation is faster. Let H be the equivalent filter kernel in the transformed domain: they want where evaluating t and H is faster than evaluating F . As the authors explain, such a transformation for a D signal does not exist in general [GO], but exists in the D case. The domain transform then defines an isometry between curves on the D manifold in R 2 (gray image) or R 4 (for RGB color image) and the real line. This transform preserves the geodesic distances between points on theses curves. Denoting ct(x) = t(x), the authors define an isometry that preserves the distance

|ct(x + h, u(x + h) -ct(x, u(x))| = | x + h, u(x + h) -x, u(x) | (.) = h + |u(x + h) -u(x)|. (.)
Dividing both sides of equation (.) by h and taking h → 0 yields

ct x (x) = 1 + |u x (x)|, (.)
where the absolute value was removed because the authors constrain ct to be monotonically increasing. The function ct x (x) is the derivative of ct(x) with respect to x. Integrating on both sides and taking ct(0) = 0 gives

ct(z) = z 0 1 + |u x (x)| dx. (.)
For a signal with c channels the transformation becomes where u k denotes the image u taken at its k-th channel and u x,k is the derivative of u k in x.



Quoting the authors:

By reducing the dimensionality of the filter from c +  to , it may seem that we lost the ability to control its support over the signal's space and range (i.e., to control the values of σ s and σ r , in bilateral filter notation). But, as we show, one can encode the values of σ s and σ r in the transformation itself.

The key idea here is that scaling the filter amounts to scaling the signal. They therefore scale the signal before computing the domain transform, which allows to scale differently the different dimensions, and then gives a total control over the smoothing parameters. When scaling the filter with a coefficient 1/a, its standard-deviation is multiplied by the same factor 1/a. Denoting by σ H the standard deviation of the D smoothing filter, one obtains σ d = σ H /a, hence:

a d = σ H σ d , (.)
where d stands for s or r k (spatial or range parameter, respectively). The authors fix σ H = σ s , so that a s = 1. For simplicity, they also use a single value σ r for every channel.

 ct x ← 1 + σ s /σ r c k=1 |u x,k | // derivative of ct in hor. direction  ct y ← 1 + σ s /σ r c k=1 |u y,k | // derivative
 v ← u // initialization  for i ∈ 1, ..., N do  σ H i ← σ H √ 3 2 N -i √ 4 N -1
// filter std: equation 14 in [GO11] // Apply a smoothing filter for non-uniformly sampled signal: normalized convolution, interpolated convolution or recursive filtering

 v ← D-Gaussian-filter-along-x v, ct hor. , σ H i  v ← D-Gaussian-filter-along-y v, ct ver. , σ H i  return v

Smoothing irregularly sampled points in the transformed domain

Once the signal is transformed through the domain transform, i.e. u w (ct(x)) = u(x), it is no longer regularly sampled. The authors [GO] propose three different methods for filtering u w .

Normalized convolution

The signal is considered uniformly sampled with missing samples. By taking advantage of the fact that ct(x) is monotonically increasing, the authors implement this efficiently using a "moving-average" approach with a box filter (with O(N ) complexity, N is the number of pixels). This can be further accelerated using GPU [GO].

v(x) = 1 C(x) y∈D(Ω) u(y)H(ct(x), ct(y)), (.)
where C(x) = y∈D(Ω) H(ct(x), ct(y)) is the normalization factor, and the kernel is defined as

H(ct(x), ct(y)) = δ{|ct(x) -ct(y)| ≤ r}, (.)
where r is the radius of the box filter, and the Boolean function δ returns  where the condition is true and  elsewhere. The pseudo-code of this algorithm is presented in Algorithm . The version with the moving average strategy is in Algorithm . the signal. [...] This is the same interpretation as the D Beltrami flow PDE [SKB]." With a box kernel, the output image v is computed as

Interpolated convolution

v(x) = 1 2r ct(x)+r ct(x)-r L ω (y)dy, (.)
where L ω is the linearly-interpolated signal in the transformed domain. As the authors state, it "does not need to be uniformly resampled, since the area under its graph can be explicitly computed using the trapezoidal rule". The pseudo-code is given in Algorithm . Pixel values outside the bounds of the image are assumed to equal the nearest pixel border value.

Recursive filtering

The recursive filter is defined in the transformed domain as

v[n] = (1 - a d )u[n] + a d v[n -1], with d = ct(x n ) -ct(x n-1
). This causal filter is applied twice, first left to right, second right to left to obtain a symmetric response. As the authors prove the feedback coefficient is computed in function of σ H as a = exp(-√ 2/σ H ). Its implementation is also O(N ). It is presented in Algorithm .

We give the pseudo-code in Algorithm . It begins with the computation of the derivative along the x and y axis for each color channel (lines  and ), used to compute the derivatives of the domain transform (lines  and ) and then the final domain transform at lines  and . At lines  and  the image v is smoothed with a one-dimensional Gaussian kernel in horizontal and vertical directions successively, according to the distance between points in the transformed signal. The output image is obtained after the N iterations of these two D Gaussian filters. Borders are handled by setting the domain transform values at inf, so that the averaging is stopped. Figure . displays the result of the application of the domain transform to a gray image and compares it to the exact bilateral filter.

Artifacts

There are some restrictions. The distance considered between points is geodesic, instead of Euclidean for the bilateral filter. This means that pixels from two opposite sides of a thin but contrasted edge will not be averaged together whereas the bilateral filter would use them all for the computation of the output value. This may be seen as an advantage of as an inconvenience according to the context; concerning tone-mapping, this is not a desired property.

area(x) ← u(x + 1) + u(x) ct(x + 1) -ct(x) /2  s(x) ← x y=1 areas(y) = s(x -1) + area(x)  x low ← smallest x s.t. ct(x) -ct(x low ) ≤ r  x up ← greatest x s.t. ct(x up ) -ct(x) ≤ r
// for center part only (see Figure 6.9)

 cp ← s(x up ) -s(x low -1)
// left part (see Figure 6.9)

 α ← (ct(x) -r) -ct(x low -1) ct(x low ) -ct(x low -1)  u ct(x) -r ← u(x low -1) + α u(x low ) -u(x low -1)  lp ← u ct(x) -r + u(x low ) (1 -α) ct(x low ) -ct(x low -1) /2
// right part (see Figure 6.9) of a part of the sky that is disconnected by the branches of a tree. One does not want to have the disconnected parts treated in a different way than the rest of the sky.

 α ← (ct(x) + r) -ct(x up ) ct(x up + 1) -ct(x up )  u ct(x) + r ← u(x up ) + α u(x up + 1) -u(x up )  rp ← u ct(x) + r + u(x up ) α ct(x up + 1) -ct(x up ) /2 //
v(x) ← u(x) + a ct (x) v(x -1) -u(x) ;  u ← v ;
As explained by K. He et al. in the guided filter paper [HST], the domain transform has a staircase effect, which causes the gradient reversal artifact when used for contrast enhancement, as shown in the Figure . reproduced from [HST].

Moreover, as mentioned by the domain transform authors: "One feature of our filters is that their responses stop at strong edges. This is in contrast with the bilateral filter, whose kernel can cross edges". Indeed, the geodesic distance used to weigh the pixels averaging can be severely different from the Euclidean distance in regions with strong gradients. This can be seen as an artifact or not, depending on the context. Concerning the local contrast amplification, it may cause a "compartmentalization" artifact: some adjacent but disconnected components with the original same color can be treated differently. Furthermore, this property makes the filter not suitable for denoising. For noisy images, the domain transform will actually smooth very little the image, because the small gradients induced by the noise artificially separate pixels that should be averaged together, because the geodesic distance is very sensitive to noise. Furthermore, as demonstrated in Figure ., the domain transform is helpless for contrasted patterns.

. Conclusion and recommendations on the fast bilateral filters

To decide for a "winner" among the considered approximations of the bilateral filter is no easy task. Indeed, the choice depends on the application in view. Hence, we are going here to compare them by their degree of approximation and by their complexity. Depending on both factors, this will give some clues on the choice to make, according to the considered application. Table . gives the complexities of the reviewed filters of this section; Table . reports the execution times evaluated by the authors of the different methods. This last table can give an idea of the order of magnitude one can expect from these approximations; however, it does not ranks the filters by their speed. This is an impossible goal, for this highly depends on the complex interaction between software optimization and hardware configuration. Furthermore, fast filters sacrifice accuracy, and each method has its drawbacks: this should be taken into account while comparing the filters.

Strictly speaking of bilateral filter's approximations (i.e. excluding the guided filter and the domain transform), the smallest execution times are obtained with the local histograms approx-  imations of Porikli et al. [Por] and the layered approximation of Yang et al. [YTA], followed by the bilateral grid of Paris, Chen et al. [CPD]. The association of the moving histogram of Perreault et al. [PH] with Porikli's method was proposed by He et al. in their paper on the guided filter [HSTb, HST]. It uses the modern processor's SIMD instructions ("Single instruction, multiple data"). When a GPU is available, however, the bilateral grid seems to be the fastest method. Contrarily to the other methods, the execution time of this last method is dependent on the parameters σ s and σ r : its execution is inversely proportional to them thanks to the downsampling step, so it might be a good option even with CPUs when dealing with large radii. The local histogram approximations suffer from the fact that their spatial kernel is a square (the authors propose an approximation for "arbitrary spatial kernels" but this is at the cost of more computation time), and also from the memory consumption. Indeed, one needs to store several histograms at the same time, and for the integral histogram, this means one per pixel. However, they are faster than the layered approximations for small spatial kernels. (One should not use the layered approximations for small parameters σ s and σ r because the subsampling strategy will not apply).



Viewed as a fast Ersatz of the bilateral filter, the guided filter can be a good choice, as it is faster than any other fast bilateral for gray scale image processing. It has an exact and fast implementation, and Bauszat [BEM] showed that it can be further accelerated with graphic hardware, before the publication of the fast guided filter [HS]. If color matters, we recommend the domain transform or the color guided filter that might be faster for large radii, because it allows downsampling (and therefore a fast guided filter algorithm). However, these two very fast filters do not actually perform a bilateral filter. In particular, they lose the ability to gather pixels that have the save intensities but are separated by another group of pixels with different intensities (e.g. the panes of a window). Furthermore, both filters introduce their own drawbacks: the contrast halo artifact for GF and the geodesic distance for DT. Concerning the guided filter, it has been extensively explored in Chapter . The high-dimensional approximations by Adams et al. [AGDL] and [ABD] are not faster than those two non-bilateral filters. The bilateral grid can be used for color but is not well suited for GPU acceleration; the local-histogram filters cannot be guided by a color image.

If precision matters, the polynomial approximations are a good option. Indeed, the difference with the original bilateral filter can be controlled and the filter is still fast with a high precision. The bilateral grid approximation is also able to control the precision by choosing the downsampling factors, but for the bilateral grid particularly, it considerably increases the computational time and memory needs. Their results is also rather close to the original bilateral filter. The speed depends on the filter parameters, as well as on the polynomial approximations.

 posure fusion

In the previous chapters, we studied the two most important edge-preserving soothing filters in the literature, namely, the bilateral filter and the guided filter. In this chapter, we explore an alternative option for contrast enhancement, in which no base and detail decomposition is involved.

Exposure Fusion is a high dynamic range imaging technique to fuse a bracketed exposure sequence into a high quality image. We show that one can extend this method to the more general context of improving the overall contrast of any image, turning Exposure Fusion into a new and simple contrast and color enhancement operator. To do so, bracketed images are simulated from a single output and fused by exposure fusion. We demonstrate that the resulting algorithm competes with state of the art retinex methods. Furthermore, we shall unveil a serious drawback of this fusion technique. Indeed, it tends to create, unlike expected, an output image which dynamic range is higher than any of the input images. This artifact forces either to clip the fused image, thus to loose precious information from the (potentially simulated) bracketed sequence, or to compress the dynamic range, which provokes a loss of contrast with respect to the input images. We shall show and explain this effect in the last section of the chapter. After careful diagnosis, we arrive at the important and counter-intuitive conclusion that exposure fusion does not have the faculty to reduce the edges' amplitude. The effectively operated tone-mapping is the consequence of two effects: the haloing due to the Laplacian pyramid, and the saturation of the input LDR images of the sequence.

The saturation artifact, also present in the introduced simulated exposure fusion, will be solved in the next chapter. This chapter introduce Burt and Adelson's Laplacian pyramid [BA] in the context of tonemapping; we shall see in Chapter  that this has been successfully reused in more recent multi-scale base and detail decomposition filters.

. Introduction

The dynamic range of real scenes is generally higher than the one of our camera sensors. To capture the entire dynamic range, photographers are led to acquire a sequence of images with different exposure times: long times capture information in dark parts of the scene and saturate the brights ones, while short exposures time capture relevant information in the brights parts. The result of this acquisition is called a bracketed exposure sequence. This sequence must then be merged into a high dynamic range (HDR) image, which gets a far higher number of bits than those that can be displayed on normal screens. Thus the HdR image needs to be remapped to the  low dynamic range (LDR) of most displays through a tone-mapping operator, which alters the colors to make them fit all in the  bits Procrustean bed.

Exposure Fusion [MKR,MKVR] was introduced by T. Mertens, J. Kautz and F. Van Reeth in  as an alternative way of constructing an LDR image of a bracketed exposure sequence. This method does not build an intermediate HDR picture. In a nutshell, it directly selects for each pixel the values, among the provided pictures, which should be kept in the final image. As a result, the fused image combines the best areas of the several input images. Although similar techniques already existed [BK], this technique has brought interesting and successful answers to two crucial questions: how to detect the best pixel from the provided set of images, and how to seamlessly merge those pixels in the final image.

In this chapter, we introduce the new technique of simulating a bracketed exposure sequence acquisition from a single LDR image, extending Exposure Fusion to color and contrast enhancement methods. We will first review the wide literature on contrast enhancement, often called retinex method. We then examine the basic ideas of exposure fusion. Modeling the effect on the underlying physical image of bracketing, leads us to propose simulated bracketing as a way to artificially enrich image information. Using Exposure fusion on simulated bracketed images delivers a new retinex like algorithm. The last part of the chapter shows results and compares them to the state-of-the-art Multiscale Retinex. We also demonstrate that this algorithm improves on itself when served with bracketed images.

. Exposure Fusion methods

For a review of the work that Exposure Fusion [MKR, MKVR] has inspired, we cite the excellent state-of-the-art review of the Exposure Fusion literature in Hafner and Weickert  [HW] (Section ".. Exposure fusion"):

Classical high dynamic range (HDR) methods combine several low dynamic range (LDR) images to one HDR image with the help of the exposure times and the camera response function; see, e.g. [MPMP], [DM], [MN], [TKTS]. However, displaying those HDR results on standard monitors or printing them requires to compress the HDR again. This process is called tone mapping; see [RHD + ] for a survey and [ ČWNA] for a discussion and evaluation of various tone mapping operators. Since tone mapping is not the focus of this work, we restrict our discussion to the most related operators. In their gradient domain tone mapper, Fattal et al. [FLW] account for the local contrast adaption of the visual system by attenuating large gradients, and maintaining or even enhancing the smaller ones. Similarly, Durand and Dorsey [DD] decompose the HDR image into a base and a detail layer. Then, they compress the base while keeping the details. Reinhard et al. [RSSF] apply first a global transform, and locally increase the contrast afterwards. Also, Mantiuk et al. [MMS] show and discuss the importance of the contrast adaption of the human visual system w.r.t. tone mapping. Most related to our work is the two-stage tone mapper of Ferradans et al. [FBPC] that applies a variational contrast enhancement in the second stage.

However, if the goal is a displayable and well-exposed LDR image, there is a popular alternative to the described two-step procedure of HDR imaging and tone mapping, namely exposure fusion [MKVR]. Here, the task is to skip the HDR image generation by a direct fusion of the differently exposed LDR images to an overall wellexposed composite. Such an exposure fusion approach has several advantages: First, there is no need to know the exposure times or the camera response function. It is  even possible to include images that do not follow the HDR imaging model, e.g. flash and no flash photographs or images from different cameras. Second, this one-step approach allows a direct tuning of the final results without the detour via an intermediate HDR image. Obviously, exposure fusion is related to tone mapping. However, the different types of input data ask for different algorithmic requirements and different model assumptions.

In the meantime, exposure fusion has even developed to an own research area with various publications that we review next. Most existing exposure fusion methods pursue the following processing pipeline: In the first step, based on exposure fusionspecific quality measures, weighting maps are determined for each of the input images. Such quality measures are, for instance, the magnitude of the Laplacian [Bog], [MKVR], the entropy [Gos], [HP], or the colour saturation [MKVR], [SCSB], [SKBb]. Another idea, e.g. applied by Raman and Chaudhuri [RC] or by Singh et al. [SCSB], is to decompose the input images into base and detail layers. Then, the amount of detail is considered as measure to determine the input image weights. In the second step, these weighting maps are com-bined with the input images to form the final composite. Here, the fusion strategies vary from region-based blending [Gos] and pixel-wise weighted averaging [RC], [HP], [SCSB], [SCB], [SKBb] to gradient domain fusion [hCH], [STC + ] and pyramid-based techniques [BK], [Bog], [MKVR]. Different to those two-step approaches, Raman and Chaudhuri [RC] propose a variational method to directly compute the fused composite. However, this requires a smoothness constraint of the final image that may lead to over-smoothed blurry results. A more suitable idea by Kotwal and Chaudhuri [KC] is to formulate the output image as a weighted average of the input. Then, they design an energy on this composite.

To summarize, the classic approach is to construct a high dynamic range image from a series of low dynamic range ones taken with different exposition times (but all other parameters of the camera must be kept fixed). Several papers propose methods for the fusion, of which the most used is probably Debevec and Malik's method [DM]. Then, the HDR image must be compressed to LDR through a tone-mapping operator. Several techniques again are available in the literature, for example Fattal et al. in  [FLW] and Durand and Dorsey,  also [DD]. On the other hand the exposure fusion approach is different, in the sense that the HDR image is not constructed: the output image is directly constructed out of the input bracketed sequence. The critical points are then to wisely select what part of which image will be used in the fused result (often several images are used simultaneously for a pixel, so blending weights must be computed), but also to seamlessly fuse those different parts of the input images. Several propositions are made in the exposure-fusion literature, recapitulated in the above review by Hafner and Weickert. One very popular is the "exposure fusion" method by Mertens et al., published in , and to which we bring an extension here. We first review this method, then propose our extension.

. Exposure Fusion

Exposure fusion first measures the perceptual quality of each pixel in each image of the input sequence. Three pixel-wise metrics are used: the contrast C, saturation S and well-exposedness E. We will denote in the following by ij the position of the pixel in a image, by c the color channel, and by k the position of the image in the input sequence. The contrast metric uses the absolute value of a discrete Laplacian filter applied to the grayscale version of the image. Denoting  by K Laplacian a Laplacian kernel, we set

C ij,k = 1 3 3 c=1 I ij,c,k * K Laplacian . (.)
The authors use for K Laplacian the sum of differences over the four nearest neighbors. The saturation metric is the standard-deviation of the pixel's color,

S ij,k = 1 3 3 c =1 (I ij,c ,k - 1 3 3 c=1 I ij,c,k ) 2 . (.)
Finally, the well-exposedness metric measures how close the pixel's value is to the median value . using a Gauss curve:

E ij,k = 3 c=1 exp - (I ij,c,k -0.5) 2 2σ 2 , (.)
with σ = 0.2. To account for multiple color channels, this measure is made on each channel separately and the results are multiplied.

The quality measure of each pixel is finally obtained as a product of these three metrics. By using the product, the authors force their method to only keep pixels which are acceptable for the three qualities simultaneously. To allow the user to choose the importance given to each quality measure, they added a power function to each one, with parameters ω c , ω s and ω e (by default equal to ):

W ij,k = (C ij,k ) ωc .(S ij,k ) ωs .(E ij,k ) ωe . (.)
For the blending process, the resulting weights need to be normalized as

W ij,k = N k =1 W ij,k -1 .W ij,k . (.)
At this point, each input image has its normalized weight map. As the authors explain, one could directly use them to fuse the images. But such an operation would lead to strong seams due to the sharp variations in the weights. They instead propose a multiscale fusion, using the method introduced by Ogden et al. [OABB]. This technique builds the Laplacian pyramid [BA] of the output image by blending the Laplacian pyramids of the input images according to the Gaussian pyramid of the weight maps. The fused image is obtained by collapsing the constructed pyramid. We will denote L{I} the Laplacian pyramid of the input image I, G{W } the Gaussian pyramid of the weights, and l the scale. The blending operation is then:

L{R} l ij = N k=1 G{ W } l ij,k .L{I} l ij,k . (.)
The algorithm  describes the whole process, from the quality measurements to the multiscale fusion.

While the sum of the weights is guaranteed for every pixel to be equal to , this does not imply that the reconstructed image belongs to the initial interval. In fact it may well happens that saturations occur in the dark or bright part. Avoiding them is possible by applying an affine rescaling of the image's dynamic to fit it to the standard interval [0, 255]. In our experiments, the resulting image generally presented no artifacts. The authors however present a case where the output image suffers from a very low frequency halo, giving an unnatural sensation (see fig.  of their paper [MKVR]). We describe and explain this effect in Section ..



L{R} l ij ← L{R} l ij + G{ W } l ij,k .L{I} l ij,k  R ← collapse Laplacian pyramid L{R} Input LDR image Output LDR image Sequence of LDR images

Exposure fusion

Generation of the input sequence

. Simulated exposure fusion: fusion from a single image

The difficulty in local tone-mapping operators is to adapt the contrast modification to different areas and avoid unnatural behaviors at edges such as halo or edge sharpening. Since Exposure Fusion achieves very successfully the similar task of selecting and seamlessly merging areas from images with significant exposure changes, we propose to adapt the algorithm to make it work for a single image. The idea is to generate an input sequence simulating for this sole image its underexposed or overexposed versions, tuning Exposure Fusion into an image enhancement operator. This process is displayed in Figure .. The first question we encountered is: how to generate the sequence? We found that the choice of the over-or under-exposure processes is not that critical. Indeed Exposure Fusion metrics are designed to always select the best pixels among the available input images. In other words, Exposure Fusion will measure what correction, among the proposed ones, is the best for each input pixel. It is therefore only necessary to present a sequence which enhances the contrast at all levels of the dynamic. In the RAW case, the captured image is

u ij = min [E ij .∆t 0 -s] + , S , (.)
where E is the scene irradiance, ∆t 0 is the exposure time, and s and S are respectively the black level and the white saturation value.

[.] + denotes the positive part. From this model we can estimate the irradiance of the scene:

E ij = u ij + s ∆t 0 . (.)
The parts of E saturated in u are lost. We call v k an image generated from u with the exposure time ∆t k . Using (.),

v ij,k = min ∆t k ∆t 0 u ij + ∆t k -∆t 0 ∆t 0 s + , S .
The value s is small and can be neglected. In addition, taking the positive part is unnecessary as all terms are positive. We therefore obtain a simple expression for the generating a bracketed image,

v ij,k = min ∆t k ∆t 0 u ij , S . (.)
Most cameras use powers of two for the exposure time. To keep generality we will use ∆t k = λ k ∆t 0 with k ∈ Z and λ a parameter superior to one, for example λ = 2. Hence:

v ij,k = min λ k u ij , S . (.)
When we do not have access to the raw picture, the problem is slightly different because nonlinearities, typically a gamma-correction and a color balance, have been previously applied to the picture u. Adapting the model gives

u ij = f (min [E ij ∆t 0 -s] + , S ) , (.)
where f () is the composition of all the non-linearities of the aquisition process. In that case, the generated images should be obtained using

v ij,k = f (min λ k f -1 (u ij ), S ) . (.)
However, although it is possible to recover f from the sequence of images [DM], this is impossible from a single one. The only option is then to make a guess about the form of f and to simulate enough bracketed images compatible with it. Most JPEG images have undergone a multiplicative color balance and a gamma-correction, which is a power function. Thus, approximating f by a power function seems appropriate. Denoting by p the exponent, we deduce from (.) that the input sequence can be generated by setting

v ij,k = min λ pk u ij , S p .
The image will again be saturated at S and we don't need to saturate below this value so the final expression is

v ij,k = min λ pk u ij , S . (.)
This leads to the favorable conclusion that the generation process is simply the same for RAW and JPEG images: we just use the identity for the function f by setting p = 1 in the RAW case. To artificially increase the exposure time (there is no reason to decrease it as we can't recover saturated parts) we therefore must use k > 0.

For a more intuitive use of the algorithm, we propose a way to compute λ from another parameter: the maximal contrast amplification factor authorized in the algorithm, α. This value is reached when generating the last images of the sequence, i.e. when k = N , whith N the number of paris to generate (see below). λ is then determined from α by λ = α 1 pN . Each used multiplier generates a pair of images. Indeed, applying a multiplier λ pk > 1 creates saturation. In order to prevent this information loss, we propose two functions: f dark saturates the image in the dark parts, while f bright saturates it in the bright parts. The important parameters thus left to the user are the maximal multiplicative factor α applied to the input image, and the total number N of images to generate. Denoting by t an intensity, the remapping function are:

f dark (t, k) = max{0, α k/N (t -1) + 1} f bright (t, k) = min{1, α k/N t}
Because factors are equal or superior to  the fused image is guaranteed not to loose contrast. We drew these functions for the various values of k (denoting the position in the generated input sequence) in figure .. The pseudo-code  describes the very simple steps of our algorithm: first, the generation of the input sequence, and then the application of Exposure Fusion.

. Results

Our experiments indicate that this method challenges the well known and very effective Multiscale Retinex [LM,JRWa,PSM]. It seems indeed able to increase both the lighting and contrast in  Algorithm : Exposure fusion from a single image input : u input image input : N number of image pairs to generate input : f dark and f bright the remapping functions input : ω s , ω c , ω e : exposure fusion parameters output: R: fused image  for k ∈ {-N, . . . , 0, . . . , N } do dark areas, thus revealing information in the shadows. Furthermore, even the bright parts of the input image are improved. This is particularly relevant as Multiscale Retinex tends to compress details in the bright areas, and generally gives grayish skies. These observations are confirmed by figure .. Concerning the colors, exposure fusion from a single image shows more saturation. However, our result presents the "out-of-range" artifact. White-saturated values are observable in the Figure . (b) on the girl's hair and in the sky. Black-saturated values are more difficult to spot, but are most probably present in the trees shadows in the reflect on the pickup's window. We discuss this artifact in the next section, and give its solution in next chapter.

 if k ≤ 0 then  I ij,k ← f bright (I ij , -k)  else  I ij,k ← f dark (I ij , k)  R ←
An IPOL workshop is available at http://ipolcore.ipol.im/demo/clientApp/demo.html?id= 77777000007, letting the user try the two presented methods on his own images and explore the effect of each parameter. Even though weights are normalized and none of the input images exceed the final dynamic range, the fused image can inherit a larger dynamic range than any of the input images. The original exposure fusion method [MKR, MKVR] simply clips the values that exceed the dynamic range, but this results in saturated areas in the final image. The authors added in their  paper [MKVR] the following remark: "Another issue concerns out-of-range artifacts. The pyramid reconstruction does not guarantee that the resulting intensities lie within [0, 1], even if the original intensities were restricted to this domain. (. . . ) One can simply fix this issue by shifting and scaling the intensities, at the risk of reducing contrast." We are then stuck in the unpleasant situation where either we decide to compress the dynamic, but lose contrast (see for example Figure .(b)), or we apply again a tone-mapping operator, which is specifically what our method was initially designed for. We shall however present an alternative way to avoid this saturation. But we shall first explain the apparition of this artefact. As will soon become clear, it is due to the multiscale blending.

. Saturation in the exposure fusion method

Constructing an image that combines the most contrasted, saturated and well-exposed parts of each image of a given sequence supposes that the method is able to keep the small variations, the local contrast (i.e. structures and textures -the detail). Exposure fusion succeeds in selecting these "best" parts and to fuse them seamlessly. However, constraining the fused result to respect the initial dynamic range is more complex: it requires the method to be able to reduce the edges' amplitude. But exposure fusion is fully based on the computation of averages of Laplacian coefficients. Thus, the exposure fusion mechanism that might reduce the edges' amplitude is the blending of high amplitude Laplacian coefficients (from high amplitude edges) with lower ampli- tude Laplacian coefficients (from lower amplitude edges). This seldom happens because weights are designed to select the most contrasted regions. Thus, in the same way as exposure fusion preserves the local contrast of each input images, it preserves their edges. In Figure . we experimentally show this effect. We designed an input sequence composed of two test-patterns. The first one has values equal to zero everywhere except in a small band in its center; this band is not saturated and has some local contrast (noise) so that exposure fusion will assign large weights to it. The second test pattern is well-exposed and contrasted for its most part, except in the same centered band where it is saturated to white. Thus, exposure fusion will fuse the center band of the input  with the side parts of input . These inputs are displayed in In fact, we just demonstrated that saturation is a vital element of exposure fusion. A bracketed exposure sequence obviously contains saturation; such a method would not be used otherwise. But it is also clipping (i.e. saturation) that allows exposure fusion to produce an image with reduced dynamic range compared to the potential HDR image one could compute using the same bracketed exposure sequence. Understanding this is of prime importance when thinking about our simulated exposure fusion (SEF) method: indeed, we have the choice to clip the input images or not. It is now apparent that without clipping, our simulated exposure fusion would not be able to enhance any image. This clarification also leads us to a simple solution to the "out-of-range artifact": to reduce the dynamic range of the input sequence. The Laplacian coefficients are displayed by scale, from the finest one (top row) and in descending order of fineness towards the bottom. The fused Laplacian coefficients (dark green) are a weighted combination of the input Laplacian coefficients (the weights are not showed here). The fused Laplacian coefficients often follow the coefficients of the input image that has the greatest amplitude. As a result, the fused image combines the greatest variations (and edges!) of each different input image, which explains its increased dynamic range. This simple experiment shows that exposure fusion cannot reduce edge amplitude at will. In fact, edge reduction is a consequence of the blending of large Laplacian coefficients (from input 1) with smaller Laplacian coefficients (from input 2). In this experiment, this is not enough to prevent a saturation of the fused result (c). Figure 7.11 displays a more complex case where three input images are fused. In this chapter, we improve on the method presented in chapter  in two ways: first we correct the saturation artifact that we proved to be inherent to the classic exposure fusion method. Second, we propose a smarter way to simulate the bracketed exposure sequence by automatically choosing the number of brightened ant darkened images, so that images with unequally distributed histograms between their left and right sides are better enhanced. We shall also uncover a novel artifact of our method, namely the creation of spurious edges in areas with smooth contrast changes (smooth edges). We solve the issue by replacing the sharp threshold (i.e. clipping) in the remapping function by a smoother function.



Furthermore the general algorithm thus designed can be used to improve on itself in the HDR context. In that case of application more simulated bracketed images are built from the input, thus permitting to obtain more contrasted regions than those provided by the physical brackets.

The proposed method eventually resembles to the local Laplacian filter, member of the multiscale edge-aware smoothing filters. The similarities and dissimilarities shall be discussed in the next chapter.

. Clipping the remapping functions

As demonstrated in Chapter , Section ., the exposure fusion method [MKR, MKVR] has a dynamic extension artifact. This extension is problematic because the fused result often does not fit the typical -bits dynamic range, thus requiring either a simple clipping of the out-ofrange values or a problematic additional tone-mapping step. For our single-image exposure fusion method however, a simple fix is to reduce the dynamic range of the input images in the generated sequence. We show here that this can be done in a way that preserves relevant information of each input image, and that it allows to reduce the edges' amplitude. This method can be extended to real bracketed exposure sequences, thus correcting the dynamic extension artifact.

Let us recall the remapping function we defined for our single-image exposure fusion method. There are two parameters: α is the maximum contrast factor used in the sequence generation; N is the number of bracketed pairs of images to simulate from the input one. We use positive and negative indexes for a pair of images with the same contrast factor but clipping in the bright or dark side; so the pair of images with the largest contrast factor is (-N, N ). We keep the input Here, the number of generated images is 4 (fifth image is the input one); the maximal contrast factor is 4 and the "allowed" dynamic range of the input image is β = 2/3 of the final dynamic range.

image, which index is 0.

f remap (t, k) = f bright (t, k) = min{1, α -k/N t} if k ≤ 0 f dark (t, k) = max{0, α k/N (t -1) + 1} if k > 0 (.)

Notation and terminology

We shall use the convention that in the simulated bracketed exposure sequence, images are numbered from -N to N . The reason is that images which number has the same absolute value have the same contrast enhancement factor (see Figure .). The negative numbers correspond to images that enhance the left hand part of the histogram, and thus saturate the bright pixels (right side of the histogram). Theses images are brighter, so we called the generative function f bright .

The image with positive index in the generated sequence enhance the right hand side of the histogram and saturate the dark pixels of the input images. They are generated using the function f dark . We keep the (unmodified) input image in the sequence; its index is 0. The total number of images in the sequence is then 2N + 1.

In order to reduce the dynamic of the generated images while keeping enough relevant information, we need to adapt the clipping process to the generated images. This leads to clip the bright values of the image with index -N that enhance the dark values the most, to clip the dark values of the image with index N that enhance contrast of the bright pixels, and to equally distribute the non-saturated intervals for in-between images. Formally, the clipping function is defined by

clip(t, k) = max offset(k), min{offset(k) + β, t} , (.) with offset(k) = k + N 2N (1 -β), (.)
and k ∈ {-N, -N + 1, . . . , N }. The new parameter β controls the dynamic range of the simulated bracketed images. We then use Figure . presents the result of this method for a real image. This improved generation method prevents saturation indeed. However, the colors seems to be affected by this specific generation method. Indeed, the clipping process actually alters colors, because they can have very different values and thus it often happens that one channel is saturated but nevertheless considered as a good pixel (because clipping reduces the number of good pixels in the input sequence). The simple workaround we use is to work on luminance only: we first convert the input to a gray-level image using

f rem,clip (t, k) = clip f remap (t, k), k (

. Asymmetric bracketed exposure

We shall further improve the bracketed image sequence generation by authorizing a different number for darkened or brightened images. Indeed, placing the input image at the center of the sequence implicitly assumes that the image needs contrast enhancement in the bright areas as much as in the dark ones. But this is rarely the case. Hence, in order to decide the proper number of images to generate in the left and right side of the input images in the sequence, we use the median value of the input (luminance) image, because it gives a good estimation of the proportion of dark and bright pixels. Formally, we define N b and N d for the number of bright and dark images to generate, respectively. The total number of images (including the input one) is then

N b + N d + 1 = 2N + 1.
The values of N b and N d are computed from the user-set parameter N and the median value Median{u}:

N b = [2N (1 -Median{u})] (.) N d = 2N -N b , (.)
where [.] denotes the closest-integer operator and the input image dynamic in [0, 1]. We redefine the remapping function using N max = max{N b , N d }:

fremap (t, k) = fbright (t, k) = min{1, α -k/Nmax t} if k ≤ 0 fdark (t, k) = max{0, α k/Nmax (t -1) + 1} if k > 0.
offset(k) = k + N b N b + N d (1 -β). (.)
We show in Figure . the remapping functions with this asymmetric distribution of contrastenhanced images in the generated input sequence and the corresponding fused results. This modification avoids increasing contrast when it is not necessary, for example on the white columns of the house.

. Introducing smooth clipping functions

An issue we encountered is the creation of shocks in areas with smooth gradients, as shown in Figure .. This artifact is caused by the artificial edges introduced when clipping in the generation process.

In order to avoid this distortion of the original image, we modified the saturation process so that it does not create edges: the brutal clipping is replaced by a smooth transition. When the values exceed the allowed dynamic (β), we progressively reduce the remapping function's derivative (the decay behaves like 1/x) until it reaches zero. The exact formulation is unimportant but we give it for the sake of completeness. The clipping function only is modified.

smooth-clip(t, k) = max 0, min{1, g(t -offset(k) - β 2 ) + offset(k) + β 2 } (.)  f remap,clip 0
g(t) = t if |t| ≤ β 2 sign(t) β 2(λ-1) + log |t| -β(λ-2) exp(λ-1) 2(λ-1) 2(λ-1) β if |t| > β 2 , (.)
where λ a parameter controlling the speed of the derivative decay. We set λ = 5, because it worked well in our experiments.

. Results

We now compare the results we obtained with those of the Retinex methods shown in Figures ., . and .. We shall also compare the "new" simulated exposure fusion presented in this chapter, to the version proposed earlier in Chapter , so that the cumulated improvements due to the clipping in the generated sequence, the asymmetric generation and luminance-only effects can be observed simultaneously. Furthermore, we shall compare our output with the local Laplacian filters [APH + , APH + ], that is described in Chapter .

It has been often observed that Retinex tends to shrink contrast in bright areas. This is particularly visible in Figure . (c) in the sky, but also on the lighthouse in Figure . (e). Furthermore, on this last image the retinex output contains visible luminance halos around the lighthouse. On the contrary, our simulated exposure fusion method improve the contrast even in the bright parts of the image and does not create halo artifacts. Concerning ACE, we shall produce a better enhancement of the faint variations. Indeed, our method better reveals the details, as can be observed in In images (g) and (h) we enhanced the local contrast using α = 8 and two different parameters σ. This parameter controls the height of the edges, like β in our method. The number of images N is directly computed from σ using N = 1/σ. The output image with LLF largely exceeds the input dynamic range, thus we added a final stretching step. We used the Simplest Color Balance algorithm (SCB) [LLM + ] that allows saturation of a small percentage of black and white pixels. We set this percentage to 1%. The last image (i) is the result of the Durand-Dorsey tone-mapping algorithm [DD] where the bilateral filter is replaced by fast LLF. A log function was applied to the base layer obtained with α = -1 and the detail layer was added back; a final stretching maintained the output image in [0, 1]. We used SCB here too. More specifically, the displayed image (i) is: SimplestColorBalance log(255 × LLF{u} + 1)/ log( 256



clearly appears that a simple increase of the local contrast along with a reduction of the underlying base layer (because of the final normalization step) is not enough to enhance the darkest areas of the original image. On the other hand, this method is good at enhancing local contrast. The result in (i) improves the visibility in originally dark areas of the image, but lacks contrast. In comparison, our result in (e) has a better visibility and contrast everywhere.

The simulated exposure method we presented ends being comparable to the fast local Laplacian filters [APH + , APH + ]. In a few words, this last method fuses Laplacian coefficients of several modified versions of the input image; the modifications consist in increasing contrast in a particular intensity range and compressing it elsewhere. This filter is presented in Chapter .

Although it would seem at first sight that we just re-created LLF, at least three notable differences tell to the contrary. The most important one is that we use different contrast factors between the different images. This allows to reduce the number of images to generate, because it specifically depends on these factors, because images with high factors quickly exceed the authorized dynamic range and that it is indispensable to produce at least one image that improves the contrast in every different part of the input intensity range. Hence, our method generally needs fewer simulated images, because it reserves the large contrast factors to areas that need it (often the far left part of the histogram) and keep smaller contrast factor in areas that do not need strong enhancement (the lighthouse in Figure . for example). And because the number of images to fuse is smaller, our method is faster and less memory-demanding. A second difference resides in the physically-based simulation process. This gives the result a more natural aspect, as can be seen in the figures of this section. The third notable difference lies in the brightening/darkening property of our method. Unlike LLF that only increases the local contrast, the simulated exposure fusion also improves the global exposition of the original image, thanks to exposure fusion metrics and an appropriate simulated bracketed exposure sequence. Furthermore, our method builds a bridge between exposure fusion and the local Laplacian filters. To the best of our knowledge, this has not been remarked yet.

To conclude this comparison section, the proposed method seems to outperform state-of-theart retinex algorithms as MSR and ACE, because it is able to both greatly improve visibility in dark areas and preserve (and enhance!) contrast in bright areas. Furthermore, no artifact were observed in the results. The local Laplacian filters method can be considered as a retinex-like method: indeed, we showed in Section . the link between ACE and the unnormalized bilateral filter, and, as will become clear in Chapter , the local Laplacian filter is based on the unnormalized bilateral filter. Compared to LLF results, our algorithm produces more natural images but also generally more contrasted and with a better exposition correction.

We believe that our method could be further improved on two points: first, the number of images to generate should be automatically computed as the smallest integer that avoids nonenhanced zones  . Second, the fusion weights could be computed more efficiently: the quality metrics may not be really useful because most of the time there are only one or two images that are not clipped for each portion of the input dynamic range; so the fusion weights could be directly given in the generation process to the image with the higher contrast factor for this range portion. We did not try this option yet. However, we suspect that it would need for some further tuning to properly handle transition between images for example.

 Actually, I did try to solve this problem but because of the specific way the simulation process is designed, this results in a rather complex equation: α -1/N log N ≥ 1-β 2β , where N is the number of images, i.e. the unknown, α > 1 the contrast parameter and β ∈ (0, 1] the allowed dynamic range in the input sequence. The problem is that the enhancement coefficients are different between two consecutive images and that they themselves depend on the parameter N that we want to find.

. Application to natural bracketed exposure sequences

In this section, we apply the generation process to each image of a real bracketed exposure sequence. First of all, we correct the dynamic expansion artifact of the original method. Then we show that further improved results can be obtained with a direct application of our method. We call this algorithm simulated exposure fusion (SEF).

In the case of an already existing bracketed exposure sequence, strong contrast enhancement factors are no longer required. Indeed, each region of the input dynamic range is supposed to be well-exposed in at least one image of the sequence. Hence, unlike the presented method simulated exposure fusion that simulates longer exposition times, we simply want here to improve the contrast of the already well-exposed parts of the input images to be fused.

As in our simulation-based method, the solution to the dynamic expansion artifact is brought by the dynamic reduction of the input images. In other words, the application of the remapping functions allows to both increase local contrast of the result and reduce its dynamic. We shall see that further local contrast enhancement can be drawn from this specific dynamic reduction strategy by forcing the fused result to fit a reduced range dynamic and applying a final stretching step.

We extended the input sequence by simulating more contrasted images for each input of the bracketed exposure sequence. We used the remapping functions with the smooth clipping defined  8.10). A final normalization step has been performed for both images. We used the "Simplest Color Balance" algorithm [LLM + 11] which allows a small percentage of clipping for both white and black values. We fixed this percentage to .5%, so that a maximum of 1% of the pixels values (a color pixel counts 3 values) is clipped in the displayed results. Our result (a) has more contrast and saturation than for the Mertens et al. output (b).

in Section . with a small contrast parameter, for example α = √ 2 in Figure .. To prevent the dynamic expansion artifact presented in Section ., we set the parameter β < 1. This amounted to reduce the dynamic of the input images.

In The fused image is compared to the original exposure fusion method in Figure .. We used the Simplest Color Balance (SCB) algorithm [LLM + ] which authorizes a small percentage of clipping for both white and black values. We fixed this percentage to .5%, so that a maximum of 1% of the pixels values (a color pixel counts  values) is clipped in the displayed results. This has the effect of reducing the contrast in the standard exposure fusion result because of its dynamic expansion and can enhance contrast in our result depending on the parameter β. The fused image with SEF has more local contrast than with EF: for the clouds in the small top window, but also the content of the shelf on the left.

We did not work with luminance here, because it is unclear how to handle the color coefficients of the different input images. The parameters for exposure fusion were fixed to ω c = ω s = ω e = 1: we equally weight the contrast, saturation and well-exposedness measures.

We present in Figure . (a) and (b) two further examples of application of the method. An input sequence is displayed on the top row; the EF result is displayed in (c). It was normalized with SCB as for the previous example, as well as the SEF fused results. The first one (a) was obtained with the same remapping function as in Figure .. The second result (b) did not enhance contrast in the input images but simply (and greatly) reduced their dynamic. The fused result got then a small dynamic range that was finally extended by the normalization step. Both methods better enhance the local contrast than EF. The first result (a) looks slightly better exposed because the brightest picture of the input sequence contains dark parts that are enhanced by the contrast factor. On the other hand, the second result (b) got more local contrast because the allowed dynamic range of the input sequence was smaller -hence the final stretching was larger.

The Figure . shows that despite still present, the improvement brought by our method is less visible for sequences with more images. Indeed, among the generated images, a lot of them remain unused, because they do not contain relevant information. Moreover, the application of SEF on such sequences rapidly yields to very long sequences ( images in this example), which is  Images copyright owner: Jacques Joffre. http://www.hdrsoft.com/examples.html an inconvenient for large images. We believe that this method could be further improved by a wiser generation of the extended sequence, by taking into account the relevant information of each image. This may allow to reduce the length of the simulated sequence and thus the efficiency of the algorithm. [PHK]. Local Laplacian filters could roughly be explained either as a single image exposure fusion algorithm similar to the method we described in Chapter , or as a multi-scale unnormalized bilateral filter. The latter interpretation was given by Aubry et al. in their analysis of the filter, where they made the link with the bilateral filter and the multi-scale version of the anisotropic diffusion [Ela, BC].



The local Laplacian filter (LLF) is versatile and can be used for a wide variety of contrast manipulations tasks, ranging from edge-aware smoothing to local contrast enhancement with dynamic reduction. Unlike most filters, LLF constructs the Laplacian pyramid of the output image; a final operation collapses the pyramid and builds the filtered image. Each Laplacian coefficient is computed independently using a dedicated remapping function, which shape is chosen in function of the application. The fast version (FLL) uses the Durand-Dorsey [DD] slicing strategy. It greatly speeds up the execution by computing only a reduced number of remapped images.

In this chapter, we first expound the local Laplacian filters and their fast approximation. Then, we show their strong connection with the exposure fusion method [MKR, MKVR]. We shall see that a fast local Laplacian filter can be computed using the exposure fusion framework with very little difference in the final result. Finally, we describe the artifacts of these filters. Indeed, although they have proven to be one of the best suited filters for base plus detail decomposition for contrast manipulation, the local Laplacian filters have some drawbacks, the major ones being a loss of translation-invariance and luminance halos.

. The local Laplacian filter

We describe in this section the Local Laplacian filters (LLF). In its original version, the local Laplacian filter modifies (almost) independently each pixel of the input image by constructing the "appropriate" Laplacian pyramid.

Let us denote by x = (x 1 , x 2 ) the position of a pixel in the image. The Gaussian and Laplacian pyramids of an image u at pixel x and scale l will be respectively written Gpyr{u, l}(x) and Lpyr{u, l}(x). The Burt et al. [BA], Gaussian pyramid of u is constructed by recursively downsampling the image by factors of two until its size is only one pixel. The last scale, the coarser one, will be denoted by l max . The Laplacian pyramid at scale l corresponds to the difference between two scales l and l + 1 of the Gaussian pyramid, the second one being upsampled by a factor two. The last scale of the Laplacian pyramid is called the residual. It simply is the coarsest scale of the Gaussian pyramid. Formally,

Gpyr{u, l}(x) = u(x) if l = 0 Downsample Gpyr{u, l -1} (x) if l > 0 (.) Lpyr{u, l}(x) = Gpyr{u, l}(x) -Upsample Gpyr{u, l + 1} (x) if l < l max Gpyr{u, l}(x) if l = l max (.)
where the Downsample and Upsample operators are defined in Algorithm  and Algorithm , respectively. The filter K used for downsampling and upsampling is the one defined by Burt et al.in  [BA]: .25, .4, .25, .05] (in D)

k = [.05,
K = k T k (in D). (.)
The input image can be recovered by "collapsing" the Laplacian pyramid, that is, recursively upsampling and adding the Laplacian coefficients, starting from the residual. Indeed, Gpyr{u, l} = Lpyr{u, l} + Upsample Gpyr{u, l + 1} and Gpyr{u, 0} = u. The Collapse operator is presented in Algorithm . In order to handle images with arbitrary height and width, Upsample adds a line and/or a column when needed so that the height and width of the upsampled image are the same than before downsampling (parameters odd h and odd w at line ). When performing the convolution in the downsampling procedure, the borders are replicated. In the upsampling procedure, border handling is made explicit at lines  and . The Laplacian coefficients contain a space and scale-localized information. This means that a "good" Laplacian coefficient is obtained when the input image has the desired properties at the corresponding space location and scale. The LLF method makes the most of this observation by computing each Laplacian coefficient of the final pyramid from an improved version of the input image -improved so that it has the desired properties at the particular space and scale localization of the Laplacian coefficient. In other words, a modified version of the input image is computed for each output Laplacian coefficient, and this modification depends both on the spatial position and the scale of the concerned coefficient. Then the "good" coefficients are copied from the Laplacian pyramid of the corresponding modified input into the output Laplacian pyramid. This process in described in Algorithm .

Algorithm : Downsample input : image u output: v the downsampled image  H ← height of u  W ← width of u  K ← Burt and Adelson's kernel defined in (.)  ū ← u * K // convolve the image  foreach pixel (x 1 , x 2 ), x 1 ∈ {1, . . . , H/2 -1}, x 2 ∈ {1, . . . , W/2 -1} do  v(x 1 , x 2 ) ← ū(2x 1 , 2x 2 ) //
 u lmax ← Lpyr{u, l max } // residual  for scale l from l max -1 to 0 do  odd h ← height(Lpyr{u, l}) -2 × height(u l+1 )  odd w ← width(Lpyr{u, l}) -2 × width(u l+1 )  u l ← Lpyr{u, l} + Upsample u l+1 , odd h , odd w  return u 0 
Figure . gives a visual explanation in a simplified case. An input D signal is given on the top left; along with two scales of its Laplacian pyramid (L 0 and L 1 , bottom left). Two modified versions of this input signal (with two scales of their Laplacian pyramids) are given in the center columns. In this illustration, we want to reduce the edge amplitude, thus each pixel on the left of the edge has the same modified version of the input signal : unmodified on the left part but clipped on the right one (in green); and each pixel on the right has the same remapped signal: unmodified on the right, clipped on the left (in purple). Then, the "good" coefficients from the green and purple pyramids are copied in the output Laplacian pyramid (bottom right) and the final signal (top right) is obtained by collapsing this pyramid. Figure . gives a visual explanation in the D case.

The modified versions are obtained through the application of a "remapping function", on which the user has full control. This remapping function gives, in the spatial and scale support of the considered Laplacian coefficient, the properties that one wants to obtain in the final image, i.e.edge reduction or enhancement, detail reduction or enhancement. We call this remapping function r(). S. Paris et al. [PHK, PHK] proposed

r(t) = g + sign(t -g)σ r (|t -g|/σ r ) α if |t -g| ≤ σ r g + sign(t -g)(β(|t -g| -σ r ) + σ r ) if |t -g| > σ r (.)
where σ r distinguishes between edges from detail, α is a parameter for smoothing (α > 1) or amplifying (α < 1) the details, and β a parameter to decrease (β < 1) or increase (β > 1) the height of edges. Finally, g is the fixed point of r and is used along with σ r to separate the fine variations from the large ones, which are treated differently. Pixels which intensity is further than σ r from the reference intensity g are considered as part of the large variations, while the others belong to the fine variations. The authors call this parameter g to represent the Gaussian pyramid value corresponding to the same position as the current Laplacian coefficient. The remapping functions one can obtain with different parameters are displayed in Figure ..

In this method, many pixels are remapped and Laplacian coefficients computed, but not used. Aware of this drawback, the authors [PHK,PHK] presented a way to accelerate the algorithm by avoiding the computation of remapped pixels and Laplacian coefficients that would not be used. It simply consists in limiting the considered neighborhood to the pixels that have an influence on the current Laplacian coefficient. The pseudo-code of this method is given in Algorithm ; the operations performed on sub-regions only of the input images are marked in red. The complexity of this method is O(N log N ) where N is the number of pixels. Paris et al.also suggest to further reduce the computational cost of their method by remapping a downsampled version of the input image for coarse scales. We do not describe this version here. We shall however describe in the following the fast approximation of LLF published by M. Aubry et al.in  and  [APH + , APH + ].



. Fast approximation of the local Laplacian filters using the slicing method

Right after the publication of the Local Laplacian Filters in , M. Aubry, F. Durand and the authors of LLF published the "Fast Local Laplacian Filers", an approximation of FLL that allows acceleration "on the order of 50". Furthermore, they show the relation with anisotropic diffusion and the bilateral filter, and introduce the unnormalized bilatearl filter that we described in Section ..

The fast version speeds up the execution by computing only S remapped images (where S is about ) instead of computing a remapped image for each different output Laplacian coefficient. The authors recommend to take a number of slices equals to ( + dynamic/σ r ). This number of slices is an important parameter for the approximation. Indeed, with a too reduced number of slices, artifacts like luminance halos can appear and notably alter the result. The pseudo-code of this fast method is given in Algorithm .

f (t) = t + α(t -g) exp -(t-g) 2 /2σ 2 r . (.)
where α and σ r are two parameters: the first one allows to choose between local contrast enhancement (α > 0) and edge-aware smoothing (α = -1), the second one makes the distinction between small variations (which amplitude is below σ r ) and large ones (amplitude above σ r ).

More generally it can be written f (t) = t + α(t -g)f (t -g) where f is a continuous function.

As the authors say, it includes the functions r() of Paris et al.with f (t -g) = (t -r(t))/(t -g).



. Similarities and differences between local Laplacian filter and exposure fusion

The fast LLF (FLL) is actually very similar to the exposure fusion (EF, see Section .). Although it does not use as input a sequence of images, it actually generates several images from the input, and merges different pieces of the latter using Laplacian pyramid decompositions. More precisely, FLL needs no quality measurement, because it knows which intensity band (i.e."slice", or "layer") has been corrected (with the appropriate contrast modification function) and therefore must be retained for the final image. As in EF, LLF and FLL construct the Laplacian pyramid of the final image. A significant difference, however, is that local Laplacian filters recompute the weight maps at each scale, while EF calculates them only at the finest scale and then subsamples them.

We review exposure fusion (EF) in Chapter . We have seen that the output Laplacian pyramid Lpyr{ũ EF } is a weighted combination of the Laplacian pyramids of the K images u k of the bracketed exposure input sequence. The normalized weight map associated to each input image is denoted W k . With these notations, EF can be written

Lpyr{ũ EF , l}(x) = K k=1 Gpyr{ W k , l}(x)Lpyr{u k , l}(x). (.)
The fused image EF{u} is finally obtained by collapsing the pyramid Lpyr{ũ EF }. Observe that the fast local Laplacian algorithm can be written in pretty much the same way: Line  in Algorithm  reads

Lpyr{ũ FLL , l}(x) ← (1 -a)Lpyr{u i , l}(x) + aLpyr{u i+1 , l}(x), (.)
where FLL{u} = Collapse(Lpyr{ũ FLL }). The interpolation weight map a in Equation (.) depends on Gpyr{u, l}(x), as for the position i of the blended images in the pre-computed sequence.

Although it is not equivalent, it plays the same role as Gpyr{ W k } in Equation (.). These interpolation weights can be pre-computed too. Denoting A i the interpolation weight pyramid associated with the remapped image u i (according to the intensity sample γ i ), we have

Lpyr{ũ FLL , l}(x) = S i=1 A i (l, x)Lpyr{u i , l}(x). (.)
Hence, the structures of FLL and EF are similar. Both blend a sequence of images according to some weights. But, unlike EF, the local Laplacian filters build their own sequence of images from a single one, like in the extension of EF proposed in Chapter . Furthermore, the computation of the weights is different: in EF the weights are computed from quality metrics, and at the finest scale. Then they are downsampled in a Gaussian pyramid. In FLL however, weights are computed at each scale, hence A i (l, x) = Gpyr{A i (0), l}(x).

In Figure . we examine the difference between filtering results of EF and FLL's weighting methods. Put another way, we try to reproduce the output of FLL with EF. In order to do so, we generated K images with the remapping functions of FLL and fused them with weights constructed as in FLL. We shall denote this modified EF version by ẼF. In EF the last scale is processed, so we processed it in LLF too. We denote this version L LF. Finally the only difference between ẼF and L LF are the weights in the multi-scale blending. The resulting processed images are visually very similar, but not identical. There are more low frequency halos in the FLL result. We measured for this experiment a psnr (peak signal-to-noise ratio) of 40dB between both results, meaning that they are very similar indeed.



. Artifacts in the local Laplacian filters

The local Laplacian filters present several problems that we have attempted to correct. On the one hand, we have seen that the Gaussian pyramid introduces artefacts, in the form of a rebound near the contours or a slight change in the average intensity of the image. These are in fact two symptoms of the same problem: the sub-sampling introduces an approximation, and the sampled values are used to guide the contrast corrections applied to the image. The approximations are then apparent in the final image in the form of asymmetries and "rebounds". The scale-space version of LLF that we introduce in the sequel solves this problem. On the other hand, depending on the contrast correction function used, LLF produces either a slight halo of luminance or a little reinforcement of the contours. Our proposal to work in a Gaussian scale-space allows us to use different "oracles", allowing to limit the luminance halo. Moreover, this new interpretation of the filter makes it possible to propose a compact formula. We shall introduce this scale-space interpretation and the compact formula in the Chapter .

.. A translation-variant filter

We realized the following experiment: a test-pattern was constructed using a single line repeated several times to make it two-dimensional. There is no variation in the vertical direction. We name this test-pattern #0. This test-pattern was then shifted by one pixel to the right -we extend the plateau on the left part and remove a column on the right, so that both have the same size. We name this test-pattern #1. Figure . displays these test-pattern in blue in the plots (b) and (c).

We filtered these test-pattern with LLF and superimposed in red the result on the input image.

The remapping function used in LLF is displayed in Figure . (a); it preserves the local contrast but reduces the edges' amplitude (dynamic compression). Noticeably, the red lines in (b) and (c) differ. Hence, the filter is not translation invariant. But that's not all: two more observations can be made from this experiment. First, although the input test-pattern #0 is perfectly symmetric, its filtered result is not: the red line in plot (a) has a negative high-frequency bounce on the right hand side of the edge, whereas on the left hand side the edge is slightly smoothed. The same asymmetry, yet inverted, appears in plot (b). Second, there is a shift in the mean value of the results. Indeed, both filtered signals have their average intensity higher than the input test-patterns. In fact, both of these artifacts are additional symptoms of the loss of translation invariance caused by the downsampling.



In the local Laplacian filters, the output Laplacian coefficients are computed from remapped images, and this remapping depends on the value Gpyr{u, l}(x) (for the Laplacian coefficient Lpyr{ũ, l}(x)). Because the pyramid is not translation-invariant, the values used in the remapping can change severely, even for a very small modification of the input image, e.g. experiment in Figure .. Moreover, when an edge or other structure is not aligned with the sampling grid, which is the more common situation, this results in its asymmetric deformation. In our experiment, this creates at fine scales the high-frequency bounce close to the edge, from one side or the other depending on the position of the edge relatively to the sampling grid. At coarser scales, the same artifact creates the shift in the mean value of the result compared to the input. In short, the only difference between both observed artifacts is a difference of scale.

.. Staircase effect and halo

The Local Laplacian Filter suffers from the staircase effect. We presented this artifact in Chapter . This effect is particularly visible in the iterated bilateral filter, where it creates a staircase in the intensities. The same effect creates an "plateau expansion", that is also known as an oversharpening artifact. For not perfectly sharp edges, both sides are expanded by the filter, resulting in a sharper transition than in the original image. In a way, this is the contrary of the halo artifact. Indeed, the halo comes from an averaging of pixels from both sides of an edge ; whereas the oversharpening arises when the filter chooses to replace the edge's transition values by an average of values that all belong to one side or the other, therefore expanding the plateaus.

Compared to the bilateral filter, this artifact in LLF is seriously reduced, to such an extent that it is barely noticeable. However, it still exists. More importantly, we frequently observed a luminance halo. This halo is visible for example in the clouds, and we called it the "black clouds effect". The authors themselves discuss this artefact in their article [PHK], [PHK]. As we shall explain in the upcoming chapter, this phenomenon is due to the remapping function's reference g, because it is set to be a blurred version of the input.

Staircasing measurements

We used a unique test-pattern for each edge width (i.e. the smoothness of the edge). We finally combined all the test-patterns to get a result (an image) that allows  Test-pattern parameters: standard-deviations for blurring go from 0 to 30 pixels.

to see the evolution of the staircasing as a function of the edge smoothness. To reword, each line observed in the final image was extracted from a test-pattern with a D edge (made with an D horizontal signal repeated along the vertical axis). Thus, we filtered as many test-patterns as the number of lines in the displayed images.

For a better comparison between ESLF and the bilateral filters, we used the following remapping functions:

r1 (t) = t -t exp{-t 2 /(2σ 2 R )} r2 (t) = r1 (t) (.)
which have a closer form to the Bilateral range kernel and allow to use the same parameter σ r . We give in Figure . a preview of the results obtained with the bilateral filter (BF), the bilateral filter with regression (BFR) as proposed by Buades et al. [BCM], the scale-space local Laplacian filters (SLF) and the extended scale-space local Laplacian filters (ESLF). The last two filters are introduced in the upcoming chapter.

Interpretation

The test-pattern used in these experiments is black on the left side and white on the right side. The transition is sharp in the test-pattern used for the top lines of the displayed images, and is progressively smoothed with a Gaussian kernel to get the following lines. These images display the difference between the input and the filtered image, i.e. the detail layer. A dark area on the left and a light area on the right are the evidence of a halo. On the contrary, a light zone on the left and a dark one on the right are the evidence of a staircase effect. From the top to the bottom, we observe the proportion of each artifact for smooth edges of increasing width. What appears first is the strong staircase effect of the bilateral filter, especially for thin edges. On the other hand, this is the filter with the faintest halo. Concerning the bilateral filter with regression (BFR) the staircase effect is greatly reduced but does not disappear. The halo is slightly increased relatively to the standard BF. While ESLF better preserves a step-edge (see the uppermost line of the test-pattern) and slightly diminish the halo, it increases the staircase effect compared to SLF (compare Figure . (c) and (d)). We shall explain this in the next chapter. The bottom row is our result for detail-reduction. In our experiment, there is no detail, the filter is then supposed to let the input image unmodified. ESLF is plotted in orange, while SLF appears in red. Our modification succeeds in removing artifacts we observed before (see Sectionı 9.4).



.. Oscillations

Figure . shows some filtering results by SLF and ESLF. The line of ESLF{u} is plotted in orange, while SLF{u} appears in red; the input test-pattern is plotted in blue. Our interest here is in the spurious oscillations that are visible near the edge in plots (d), (h), (i) and (j). For the last three ones indeed, the remapping function used is designed for local contrast reduction. So why is the filtered result different from the input? In a few words, the edge in this test-pattern is considered as a detail in some pixels during the filtering process. Indeed, the distinction between base and detail variations is made according to the reference intensity g in the remapping function: we have seen that in Section . of this chapter. Since this "guide" g is nothing but a blurred version of the input image (Gaussian pyramid), at edges, g is not close to the input image anymore. Therefore, when getting closer to the edge the image is considered as detail before, by getting even closer to the edge the interpretation changes to base, creating those spurious oscillations.

 10 Compact formula for the local Laplacian filter and its scale-space extension

In the previous chapter, we presented the local Laplacian filter (LLF) and scanned its structural analogy with exposure fusion [MKR, MKVR]. We showed that despite some excellent results, LLF suffers from three artifacts, namely, its lack of translation-invariance, its luminance halos and a slight staircase effect. The lack of translation-invariance is particularly annoying because it creates irregularities, small bounces and a mean-shift. Fortunately, all of these issues will be solved in this chapter by our scale-space local Laplacian filter.

We start by dissecting the local Laplacian filter and proposing a compact formula by reformulating the local Laplacian filter in a scale-space setting. This amounts to removing the downsampling and upsampling steps of the original filter. Besides giving a clean mathematical description of the filter, a welcomed outcome of this re-interpretation of the filter is the reinstatement of the translation invariance property which LLF lacked. Furthermore, this interpretation puts in evidence the implicit guide used in LLF; this guide, that we shall call oracle, can then be replaced by the result of an arbitrary previous filter. We therefore explore the influence of the oracle in this new framework. As we shall see, edge-aware smoothing filters used as oracle reduce the luminance halo but increase the staircase effect, while a simple Gaussian filtered oracle (as used in the original filter) has no staircase effect but sometimes visible luminance halos. We finally compare the results of this extended scale-space local Laplacian filter with the standard local Laplacian filter in the context of base plus detail image decomposition.

. The scale-space point of view

Our goal is to give a clean interpretation of the local Laplacian filters by reformulating them in a Gaussian a scale-space rather than in a Gaussian pyramid. This amounts to removing the downsampling and upsampling steps in the local Laplacian filters. One should remark that the Gaussian pyramid is not identical to a downsampled Gaussian scale-space. Indeed, the upsampling step (interpolation) is not a convolution because it gives a different kernel at even and odd positions. In fact in an image this gives four different filters, one for each configuration of even and odd lines and columns. We can get a representation of the Gaussian pyramid filters at scale 1 for the different positions by successively downsampling and upsampling a Kronecker delta. Let δ(x, y) be a Gaussian scale-space kernel ( = 1) Gpyr kernel for (odd,odd) position Gpyr kernel for (odd,even) position Gpyr kernel for (even,odd) position Gpyr kernel for (even,even) position

Figure 10.1: Kernels at scale 1 in the Gaussian scale-space (blue line) and the Gaussian pyramid. There are four different kernels, one for each different configuration of even and odd line and column. The shift between the yellow and green kernels with respect to the other ones is for visualization purposes only.

Kronecker delta at position (x, y). The Gaussian pyramid kernel is

k Gpyr (x, y) = Upsample(Downsample(δ(x, y))). (.)
The downsampling and upsampling procedures are described in Algorithm  and Algorithm ; they use Burt and Adelson's [BA] filter f = [.05, .25, .4, .25, .05]. On the other hand, the kernel in the Gaussian scale-space is the convolution of the same Kronecker delta in the Fourier domain with σ = 1. We display in Figure . the kernel k Gpyr in the four possible configurations and compare it to the unique filter in the Gaussian scale-space. The Gaussian scale-space Gss{u} of the input image u is defined as

Gss{u}(x, l) = u(x) if l = 0 u * G σ l (x) if l > 0 , (.)
where * denotes the convolution, l is the scale (or level) and G σ l is a normalized Gaussian kernel with standard deviation σ l . The total number of levels in the discrete Gaussian and Laplacian scale-space, taking into account the finest one, is then l max + 1. We use standard deviations powers of two: σ l = 2 l-1 σ ref and σ ref = 1. The Laplacian scale-space is the difference between two consecutive scales of the Gaussian scale-space. Its last scale, the residual, is the same as the Gaussian scale-space:

Lss{u}(x, l) = Gss{u}(x, l) -Gss{u}(x, l + 1) if l < l max Gss{u}(x, l) if l = l max . (.)
For the sake of simplicity in the upcoming developments, let G 0 be the Dirac mass δ 0 . This way, the notation Gss{u}(x, l) = G σ l * u (x) is true in general.

In the same way as for the Laplacian pyramid, the input image can be recomposed from its Laplacian scale-space by collapsing it. In the scale-space case, it simply amounts to summing all levels:

u(x) = lmax l=0
Lss{u}(x, l).

(.)

Pseudo-codes

We now present the pseudo-code of the exact and fast versions of the scale-space local Laplacian filters. Indeed, the filter benefits from the same fast approximation as the standard LLF (slicing). However, the complexity and the memory consumption are higher with this filter, because in the scale space each scale has the same size as the highest resolution image (whereas the sub-sampling  process in the Gaussian Pyramid allows to save time and memory).

In Algorithm  we present the exact version of the scale-space local Laplacian filter. The only difference with the exact version (see Algorithm ) of LLF is the absence of down and upsampling because of the Gaussian scale-space. Algorithm  describes how this scale-space is computed. The operation at line  is optional but allows to reduce the complexity from O(N 2 ) to O(N log(N )) with N the number of pixels. This technique is proposed by Paris et al. [PHK, PHK]. At line  the remapping is pixel-wise.

The fast approximation of SLF is described in Algorithm . Likewise, this is an adaptation of the Aubry et al. method (which pseudo-code is given in Algorithm ) where we replace the Gaussian and Laplacian pyramids by scale-spaces. We recall that this fast approximation relies on the computation of the exact result of the filter for only a reduced set of samples of the intensity range. The obtained images are called layers (a layer is associated to each intensity sample). Finally, each pixel which intensity does not correspond to any sample is interpolated between the two closest layers (i.e. the two layers which corresponding intensities are the closest to the current pixels' intensity).

Operations at lines , , ,  and  are performed pixel-wise. This fast algorithm has an additional parameter S. As advised by Aubry et al. [APH + ,APH + ], one can take σ r as interval between two intensity samples, that is, S = D/σ r + 1 where . is the closest superior integer operator, D the dynamic range and σ r the range parameter, for remapping function r(t) = t + αtG σr (t).

The number of scales used is the maximum possible in the Gaussian pyramid sense. In the Gaussian (dyadic) pyramid, the last level l max is attained when whether the height or the width of the image is 1 pixel. That is, l max is such that d min (l max ) = 1 with

d min (l) = d min (l -1)/2 , (.)
with d min (0) = min{height(u), width(u)}. We kept this definition in our scale-space method. Following the implementation provided by the authors of the original LLF and of the fast version FLL, the residual is not modified.

Algorithm : Scale-space Local Laplacian Filter input : image u input : remapping function r output: filtered image SLF{u}  Compute the Gaussian scale-space Gss{u} of u // Algorithm 40  foreach coefficient at position x and scale l do // Get Gaussian scale-space value for the remapping function

 g ← Gss{u}(x, l)  Determine sub-image v of u needed to evaluate Lss{v remap }(x, l)  v remap ← r(v -g) // apply remapping function  Compute Laplacian scale-space Lss{v remap } of v remap  Lss{ũ}(x, l) ← Lss{v remap }(x, l) //
0 , γ 1 , ..., γ S-1 } regularly spaced  λ ← γ 1 -γ 0 // intensity step between two samples  for each intensity sample γ i with i ∈ {0, 1, . . . , S -1} do  u remap ← r(u -γ i ) // apply remapping function  Compute the Gaussian scale-space Gss{u remap } of u remap  for each scale l from 1 to l max -1 do  α ← max(0, 1 -|γ i -Gss{u}(l)|/λ) // interpolation weights // Compute the Laplacian scale-space at scale l  Lss{u remap }(l) ← Gss{u remap }(l) -Gss{u remap }(l + 1) // Update output Laplacian scale-space (initialized with zeros)  Lss{ũ}(l) ← Lss{ũ}(l) + αLss{u remap }(l)  Lss{ũ}(l max ) ← Gss{u}(l max ) // residual is not modified  SLF{u} ← lmax l=0 Lss{ũ}(l) // collapse output scale-space
Algorithm : Computation of the Gaussian scale-space of an image input : image u input : number of scales l max (finest scale is 0 and coarsest one is l max ) output: Gss{u} the Gaussian scale-space of u

 σ ref = 1 // fixed  Gss{u}(0) ← u // finest scale (l=0) of Gss{u} is the input image itself  Make u periodic by symmetrization; get u per with double size  ûmirror = FFT{u per } // Fourier transform of uper  for each scale l from 1 to l max do  σ ← 2 l-1 σ ref // Gaussian standard-deviation for the current scale  Ĝσ ← exp{-(2π 2 σ 2 ) ζ 2 } // Gaussian kernel in the Fourier domain  ū ← FFT -1 {û per × Ĝσ } // convolution in Fourier Domain  Gss{u}(l) ← crop(ū)
// we only need the first quarter of the result  we need to compute the FFT of the input image and l max inverse FFT (total number of scales is l max + 1, taking the finest into account). Our algorithm (see Algorithm ) requires the computation of a Gaussian Scale-Space for the input image and for each remapped image. The overall complexity of our method is then O N log(N )Sl max , with S the number of samples.

. Compact formula for the local Laplacian filters

Let us now build the local Laplacian filter in the scale-space. The scale-space local Laplacian filter (SLF) algorithm, described in Algorithm , is simply the exact LLF (which pseudo-code is given in Algorithm ) where we replaced the Gaussian and Laplacian pyramids by Gaussian and Laplacian scale-spaces. This pseudo-code will help us to construct the formal description of SLF. Starting at line , we progressively unfold the expression of the scale-space local Laplacian filter for an arbitrary pixel x. The output is given by collapsing the scale-space progressively constructed at line . That is,

SLF{u}(x) = lmax l=0 Lss{ũ}, (.)
which can be written

SLF{u}(x) = lmax-1 l=0 (G σ l -G σ l+1 ) * u (x) + G σ lmax * u (x), (.)
where u is the remapped input image. The rightmost part of Equation (.) is the residual of the Laplacian scale-space. The Laplacian scale-space Lss{ũ} in Equation (.) represents the Laplacian scale-space of the final image under construction. For a specific couple (x, l), the Laplacian coefficient Lss{ũ}(x, l) is computed line  from u . The image u is the result of the application of r to the input u according to the reference intensity G σ l * u (x):

u = r (u -(G σ l * u) (x)) . (.)
This is obtained by merging together lines  and . There is one different image u for each pixel of each scale, because it is remapped according to g, the Gaussian coefficient at scale l and pixel x. We use here a remapping function r with the same form as used by the authors of the fast local Laplacian filters [APH + , APH + ]. This function is nevertheless equivalent to the Paris et al. function: r(t, g) = r(t -g) + g. It will be useful in the coming developments. There is no need to add the constant g after remapping the intensity differences with r. Indeed, in the local Laplacian filter the remapped images are used for the computation of Laplacian coefficients, which are insensitive to this constant. Inserting Equation . in Equation . gives the final equation:

SLF{u}(x) = lmax-1 l=0 (G σ l -G σ l+1 ) * r u -G σ l * u (x) (x) + G σ lmax * u (x). (.) By denoting D l = G σ l -G σ l+1 the difference-of-Gaussian operator and g(x, l) = G σ l * u (x)
the reference intensity in r we get

SLF{u}(x) = lmax-1 l=0 D l * r u -g(x, l) (x) + G σ lmax * u (x). (.)
This formula is our more compact formula for the Local Laplacian Filter.



Second compact formula using the fast LLF point of view

The fast version of local Laplacian filters by Aubry et al. is based on the piecewise linear interpretation of the bilateral filter (described in Section .), transposed to FLL. Although this scheme is meant to accelerate the filter by downsampling in the space and range domains, this formulation is exact. We can express the scale-space local Laplacian filters from the same point of view. This actually completes our previous expression of SLF by providing another compact and insightful formulation:

   SLF{u}(x) = lmax-1 l=0 ṽ x, l, (G σ l * u)(x) + (G σ lmax * u)(x) ṽ(x, l, g) = (G σ l -G σ l+1 ) * r(u -g) (x).
(.)

In this equation, ṽ is what we could call a "Laplacian layer": Laplacian coefficients at scale l of the remapped input image according to the reference intensity g. The output image SLF{u} is constructed from these layers, by selecting at each pixel the Laplacian coefficients in a particular layer, depending on the value (G σ l * u)(x). This value acts as a guide indicating for each pixel and each scale how the input image should be remapped to get the enhanced final result. Put another way, the guide (G σ l * u) is used to pick the value of the Laplacian coefficient in the "right layer".

There are as many layers as the number of possible intensities for the guide, and constructing SLF amounts to collect the "correct" values in the pre-computed layers. The fast approximation consists in pre-computing only a reduced set of Laplacian layers, and, for values of the guide that have no pre-computed layer, to linearly interpolate between the two closest pre-computed values. As we shall see in Section ., this guide is implicit in the original local Laplacian filters, whereas our scale-space interpretation reveals its presence and allows its replacement.

A quick review of Aubry et al. analysis of the local Laplacian filter (and why ours effectively go further)

In their paper [APH + , APH + ], Aubry et al. make the link between the local Laplacian filters, the bilateral filter, and the anisotropic diffusion. They also present a new filter, the unnormalized bilateral filter, that we review in Section .. In the following, we put ourselves back in the context of (Gaussian and Laplacian) pyramids and reproduce and review the steps of Aubry et al. analysis of the local Laplacian filters. Their work suggests the form of the single-scale filter used in LLF. Yet our scale-space interpretation, besides the exact and compact formulation of the filter, allows a deeper understanding of the edge-aware manipulation of the Laplacian coefficients that occurs at scales superior to zero.

The authors in [APH + , APH + ] first consider a remapping function of the form

r(t) = t -(t -g)f (t -g) (.)
where f is a continuous function. In order to make the link with the bilateral filter, we take f (t) = G σr (t) = exp -t 2 /(2σ 2 r ) with σ r the range parameter, i.e. the standard deviation of the range Gaussian kernel, as in the bilateral filter. They then consider a pyramid with only two scales, that is, Lpyr{ũ, 0} and Lpyr{ũ, 1}. The residual is not processed, so Lpyr{ũ, 1} = Lpyr{u, 1}. At the finest scale, the Laplacian pyramid of the output is

Lpyr{ũ, 0}(x) = r u(x) -G σ 1 * r(u) (x), (.)



where * is the convolution and G σ 1 a normalized approximately Gaussian kernel. Indeed, in LLF this Gaussian convolution is in practice performed by successively downsampling and upsampling r(u). By expanding the remapping function r, we have

Lpyr{ũ, 0}(x) = u(x) -(u(x) -g)G σr u(x) -g -G σ 1 * u -(u -g)G σr u -g (x), (.)
which, using Lpyr{u, 0}(x) = u(x) -G σ 1 * u (x) and replacing g(x) by u(x) since we are at the finest scale, can be simplified as

Lpyr{ũ, 0}(x) = Lpyr{u, 0}(x) + G σ 1 * u -u(x) G σr u -u(x) (x). (.)
This reduction is possible only for the finest scale, because we used the fact that g(x) = u(x), which is not true for the other scales. By upsampling the residual and adding it to the equation (collapsing the pyramid), we get

LLF{u}(x) = u(x) + G σs * u -u(x) G σr u -u(x) (x), (.)
where we replaced σ 1 by σ s to stress the resemblance with the bilateral filter. With the same objective in mind, we re-write Equation (.) as

LLF{u}(x) = u(x) + y G σs (x -y)G σr u(y) -u(x) u(y) -u(x) . (.)
This is the definition of the unnormalized bilateral filter. The authors observe that "one may achieve cross filtering" with LLF, but let this case for further studies. We shall come back to the "guidance" process in LLF soon (see Section .).

Concerning deeper pyramids (more than two levels, the finest and the residual), the authors give, in a similar spirit, the filter that computes the output Laplacian coefficients.

Lpyr{ũ, l}(x) = D l * u -g(x) G σr u -g(x) (x), (.) where g(x) = (G σ l * u)(x) and D l = G σ l -G σ l+1 .
Written in a closer form to the bilateral filter,

Lpyr{ũ, l}(x) = y D l (x -y)G σr u(y) -g(x) u(y) -g(x) . (.)
Because g(x) = u(x) the expression cannot be collapsed as above. As the authors conclude their analysis, "this shows that each level of the output pyramid is a local average of differences over a neighborhood of x". We shall reveal in the next section the implicit filter that is used for scales superiors to zero in LLF.

Note however that Equation (.) does not exactly describe the local Laplacian filter. As we said before, although the difference-of-Gaussian operator D l is described as the difference between two Gaussian kernels, in practice the blurred versions of the input are obtained through successive downsampling and upsampling, using the methods described in Algorithm  and Algorithm , respectively. Using the definition of Gpyr and Lpyr given in Equation (.) and Equation (.), respectively, the exact Laplacian coefficients of the final image are written Lpyr{ũ, l}(x) = Lpyr{r u -Gpyr{u, l}(x) , l}(x).

(.)  Thus, using the Upsample operator (Algorithm ), the final result is obtained from the previous Equation (.) and the following recurrence relation (collapsing):

ũl = Upsample(ũ l+1 ) + Lpyr{ũ, l} ũlmax = Gpyr{u, l max } , (.)
where the finest scale is the output image: LLF{u} = ũ0 . Although this is already a compact formulation of the local Laplacian filter, it does not have the clarity and completeness of the scalespace formulations given in Equation (.).

To conclude on the LLF analysis carried out by the authors in [APH + ,APH + ], they demonstrate that LLF is actually the unnormalized bilateral filter when the pyramid is only  scales deep. They consequently also make the link with the anisotropic diffusion. In the multi-scale case however, they only give the form under which the Laplacian coefficients are computed. In our analysis we go further and present the underlying single-scale filter, that is, as will soon become clear, a sort of "guided" unnormalized bilateral filter: indeed, it is situated between the unnormalized bilateral filter and the unnormalized cross bilateral filter. Furthermore, while the last Equation . is already a compact formulation, it only expresses an intermediary result -the Laplacian coefficients. Our formulation with the scale-space in Equation . is more complete as it expresses the final filter directly.

. Oracle-based unnormalized bilateral filter

First of all, we recall the definition of the unnormalized bilateral filter proposed by Aubry et al. in  [APH + , APH + ]. We shall indeed refer to this filter many times in this section.

UBF{u}(x) = u(x) - y G σs (x -y)G σr u(y) -u(x) u(y) -u(x) .
(.)

Alternatively, this filter can be written

UBF{u}(x) = 1 -C(x) u(x) + C(x)BF{u}(x), (.)
where C(x) is the bilateral filter normalization factor in x and BF the bilateral filter.

The cross or joint bilateral filter [ED], [PSA + ], uses a second image (we call it v), related to the image to be filtered, for the computation of the range weights. For example with a flash/no-flash pair of images in a low-light context: the no-flash image, noisy but with better colors, is filtered (denoised) according to the flash image with higher signal-to-noise ratio. In the unnormalized case, this filter can be written

UCBF{u, v}(x) = u(x) - y G σs (x -y)G σr v(y) -v(x) u(y) -u(x) , (.)
where in our example u is the input no-flash image and v the flash image from with the range weights are computed. Like the unnormalized bilateral filter, UCBF has an alternative form similar to Equation (.); the only difference being the replacement of the bilateral filter BF{u} by the cross bilateral filter CBF{u, v}.



Furthermore, let us introduce the remapping function

r(t) = t + αtf (t), (.)
where f is a continuous odd function and α a parameter that allows to choose between detail amplification (α > 0) and detail reduction (α < 0). In practice we restrict ourselves to the detail smoothing case, i.e. α = -1, which places SLF in the bilateral filtering context. For the same reason we use

f (t) = G σr (t) = exp -t 2 /(2σ 2 r ) . That is, r(t) = t -tG σr (t). (.)
We shall prefer this writing rather than r defined in Equation (.) because it makes the reference intensity explicit and thus clarify our developments. Both expressions are equivalent: r(t) = r(t -g) + g. Remark that the addition of the constant g will often be omitted because it is discarded when convolving with the difference-of-Gaussian operator D l = G σ l -G σ l=1 (both Gaussian kernels are normalized).

On the importance of the oracle in the local Laplacian filters

Let's now have a closer look at the impact of the "reference intensity" g used in the remapping function of the local Laplacian filter. Indeed, this guide G σ l * u depends on the scale l. As a consequence, it is not possible to collapse the pyramid in the definition of SLF in Equation ., although it would be possible if the guide were identical at each scale. For example, using the input image u in place of G σ l * u and collapsing the pyramid leads to

SLF{u}(x) = lmax-1 l=0 (G σ l -G σ l+1 ) * r u -u(x) (x) + G σ lmax * u (x), (.)
which is in fact the same as

SLF{u}(x) = (G σ 0 -G σ lmax ) * r u -u(x) (x) + G σ lmax * u (x) (.)
because the Gaussian convolutions cancel each other between the different scales. Since G σ 0 is a Kronecker delta, it can be simply removed because r(u -u(x)) in x is zero. By expanding the remapping function using r(t) = t -tG σr (t), we get

SLF{u} = G σ lmax * u (x) -G σ lmax * u -u(x) -G σr u -u(x) u -u(x) (x). (.)
The G σ lmax * u terms cancel each other and the constant u(x) can be taken out the convolution. Finally, by expanding the convolution:

SLF{u} = u(x) + y G σ lmax (x -y)G σr u(y) -u(x) u(y) -u(x) . (.)
This is nothing but the unnormalized bilateral filter.



Interpretation The local Laplacian filters, when the guide G σ l * u is replaced by the input image, simply is the unnormalized bilateral filter with spatial parameter σ lmax . This gives an idea of the importance of this guide in LLF. Indeed, UBF as well as BF loose their ability to filter when used with large spatial standard deviation, because the number of neighbors with the same intensity increase. This makes the weights given to pixels with slightly different intensity to decrease, and, in turn, the filter tends to average only pixels that have the same intensity. Hence, the effectiveness of LLF is due to the guide introduced in the (unnormalized) bilateral filter. From now on, we shall call this guide an oracle.

In the next section, we study the single-scale filter implicitly used in SLF. As we shall see, this oracle defines a new (unnormalized) filter different both from the bilateral filter and the cross bilateral filter.

.. Implicit edge-aware filter in the scale-space local Laplacian filters

Our compact formulation of the scale-space local Laplacian filter is given in Equation .. However, we remind it here for the sake of readability:

SLF{u}(x) = lmax-1 l=0 (G σ l -G σ l+1 ) * r u -G σ l * u (x) (x) + G σ lmax * u (x). (.)
At a specific scale l and pixel x, the Laplacian scale-space SLF{u} can be written

Lss{ũ}(x, l) = y G σ l -G σ l+1 (x -y)r u(y) -g(x, l) (.)
where g(x, l) = G σ l * u (x). Using r(t) = t -tG σr (t), Equation (.) can be rewritten

Lss{ũ}(x, l) = y G σ l -G σ l+1 (x -y)u(y) - y G σ l -G σ l+1 (x -y)G σr u(y) -g(x, l) u(y) -g(x, l) , (.)
because the Gaussian kernels are normalized so the constant g(x, l) in the left part of the equation is discarded. This constant cannot be removed in the right part because of the range kernel.

Interpretation: This equation shows that Lss{ũ}(x, l) is actually the difference between two Laplacian coefficients. The first term in the equation is the standard Laplacian scale-space, made of the difference between two successive scales of the Gaussian scale-space. The second term however is the difference between two successive scales of a bilateral-like filter. This particular filter has the form of the unnormalized bilateral filter proposed by Aubry et al. in [APH + ,APH + ], but is different: it uses an oracle g. This is not a cross or joint unnormalized bilateral filter [ED, PSA + ] either, because the oracle would be used in the range kernel only (see Equation (.)). This is a new filter. We shall call it in the following the unnormalized oracle-based bilateral filter  (UOBF).

It is defined as:

UOBF{u, v}(x) = v(x) + y G σs (x -y)G σr u(y) -v(x) u(y) -v(x) . (.)
We call v the oracle because it is the value that controls, for each pixel x, whether a pixel y in its neighborhood will participate a lot in the computation of the result or not. It can be considered as  a general framework including the unnormalized bilateral filter as a particular case: indeed, using the input itself as oracle brings UOBF back to the unnormalized bilateral filter (see its definition in Equation (.)). The oracle v used in SLF is g(x, l) = G σ l * u. We shall explore in Section . the different filters and the improvements we can derive from the replacement of this oracle by more sophisticated ones. Inserting Equation (.) in Equation (.), the Laplacian coefficients can be rewritten

Lss{ũ}(x, l) = (G σ l -G σ l+1 ) * u (x) -UOBF σ l {u, g(x, l)}(x) -UOBF σ l+1 {u, g(x, l)}(x) , (.)
where the indices indicates the spatial standard-deviation of the filters. Hence, by collapsing the scale-space we get

SLF{u}(x) = u(x) - lmax-1 l=0 UOBF σ l {u, g(x, l)}(x) -UOBF σ l+1 {u, g(x, l)}(x) . (.)
This is yet another compact and complete formula for the scale-space local Laplacian filter. In Equation (.) the two terms of the additive base and detail decomposition of the input u can be easily identified: SLF is used for detail smoothing, thus SLF{u} is the base layer; the rightmost part of the equation is then the detail layer. Intuitively, the detail layer is obtained by collapsing an edge-aware scale-space constructed from this new bilateral-like oracle-based filter. We show in Section . the multi-scale decomposition obtained with it. But we first concentrate on its properties in a single-scale context.

.. The single-scale unnormalized oracle-based bilateral filter

In Equation (.) the filter is unnormalized; in the same way as for the unnormalized bilateral filter (see Equation (.) and Equation (.)), it can be rewritten in function of a normalized filter, UOBF{u, v}(x) = 1 -C(x) v(x) + C(x)OBF{u, v}(x), (.)

where C(x) = y G σs (x-y)G σr u(y)-v(x) is the normalization term and 0 ≤ C ≤ 1 because the spatial kernel G σs is normalized. Finally, OBF is the (normalized) oracle-based bilateral filter:

OBF{u, v}(x) = 1 C(x) y G σs (x -y)G σr u(y) -v(x) u(y). (.)
Once again, using the input image itself as oracle brings OBF back to the bilateral filter.

In the same way as for the unnormalized bilateral filter, this filter does not filter where the normalization factor C is small (generally at edges). Instead, it takes the oracle value (Equation (.)). Figure . compares the filtered results of UOBF, UBF and BF for a test-pattern. This allows to appreciate the importance of the oracle since this is the sole difference between UBF and UOBF.

The fact that the edges of u are replaced by those of the oracle by UOBF is not problematic in the case of SLF. Indeed, for SLF we are only interested in the difference between two applications of UOBF. Since only the spatial parameter changes between the two filters (i.e. the oracle is the  This name might not be the greatest name for this filter. However, among the flourishing bilateral filter descendants, namely the joint [ED], cross [PSA + ], dual [BMM] and even the guided [CTC] bilateral filters, not to mention other trilateral [CT], multilateral [BR] or joint multilateral [LTL] filters, we decided to go for this (unnormalized) oracle-based bilateral filter (UOBF, OBF) so as not to pick one already existing name. The same parameters are used for all filters: σ l = 2 pixels and σr = .2. The oracle used in UOBF is Gσ l * u. It is drawn in dark blue. Compared to BF, UBF has a lighter smoothing effect. On the contrary, UOBF has a stronger smoothing than both BF and UBF. In fact, UOBF closely follows its oracle, except at the edge where most of the differences averaged have the same sign (positive at the top of the edge, negative at the bottom).

same), the difference is null at places where UOBF returns v. This filter is thus particularly well suited for the computation of edge-aware Laplacian coefficients. We examine in the next section the edge-aware multi-scale decomposition allowed by the oracle-based unnormalized bilateral filter.

Conclusion on the scale-space local Laplacian filters

The scale-space local Laplacian filters are closely related to the bilateral filter; in fact, as demonstrated by Aubry et al., it is a multi-scale unnormalized bilateral filter. The normalization can be removed in a "safe" way as one manipulates Laplacian coefficients, which in average are null. We showed however that the authors inserted a guide, that we call an oracle in order to make the distinction with the cross (or joint) bilateral filter. This oracle allows an effective multi-scale decomposition of the detail layer, which is not possible with the bilateral filter (nor the unnormalized one), as explained in  by Farbman et al. [FFLS]. We examine in the next section the multi-scale base+detail decomposition realized in SLF and compare it to the bilateral pyramid.

. Bilateral pyramids

The history of multi-scale bilateral image decomposition is relatively recent and also fairly short.

In , two papers are published, one by Fattal et al. [FAR] and the other by Chen et al. [CPD]. They use a bilateral pyramid as a tool for, respectively, image fusion and transfer of photographic look. However, the proposed schemes are different, as we shall see very soon. The following contributions on the multi-scale edge-aware base+detail decomposition topic showed the imperfections of the bilateral filter and proposed alternative schemes. This is the case with the weighted least squares (WLS) filter proposed by Farbman et al. in  [FFLS] and a local extrema-based filter proposed by Subr et al. in  [SSD]. Fattal's edge-avoiding wavelets [Fat] in  also enter this category, with the older paper by Li et al. in  [LSA]. Still on the wavelet topic, Hanika et al. propose in  an "edge-optimized à-trous wavelets" [HDL] that allows a mutli-scale base+detail decomposition and avoids the artifacts due to the decimation in Fattal's method. This work is in continuation of the  paper by Dammertz et al. [DSHL]. Our interest here is in the usage of the bilateral filter for multi-scale decomposition, so we concentrate  In contrast, the basic algorithm also considers all the pixels marked with green lines. At j = 3 the basic algorithm averages in the gray pixel in column 6, but the fast algorithm never sees a contribution from that pixel. Illustration and caption reproduced from [FAR07].

on the two  papers.

Fattal et al. bilateral pyramid

In this method the input image is recursively filtered with increasing spatial parameter σ s,j , adjusted so that the combined effect of the successive filters has a spatial standard-deviation of 2 j σ s , with j the level of the pyramid. To preserve edges during this process, the range parameter is reduced at each iteration. They set σ r,j = σ r /2 j . That is,

v j+1 = BF σ s,j ,σ r,j {v j }, (.)
where v 0 is the input image. As explained by the authors [FAR] (quoting) (. . . ) we do not subsample the v j because such downsampling would blur the edges in v j . In addition downsampling would prevent the decomposition from being translation invariant and could introduce grid artifacts when the coarser scales are manipulated.

We recognize here the artifacts we described in Chapter . The detail layers at different scales are then simply computed by the difference between two consecutive scales of the pyramid. The author proposes an efficient scheme for the computation of this pyramid, based on the algorithme à trous method [Mal, HKMMT]. The key idea is to constantly use kernels with a very few non-zero entries. This is possible thanks to the recursive aspect of the method, as demonstrated for Gaussian kernels in [Bur]. In fact, the authors use for each level a kernel with 5 × 5 nonzero coefficients: those coefficients are separated by more and more zeros, as shown in Figure .. This scheme saves many operations and the author reports shorter running times than the  bilateral grid fast approximation [PD].

The Chen et al. bilateral pyramid

The bilateral filtering in this method is applied to the input image with increasing space and range parameters. This is particularly adapted to the fast approximation proposed by the same authors, the bilateral grid. It is described in Chapter . Indeed, the complexity of this method decreases when the smoothing parameters increase.

Both schemes are not well suited for multi-scale base and detail image decomposition. As explained by Farbman et al. [FFLS], Chen's method does not well preserve the edges in the high scales, and Fattal's scheme oversharpens the edges, creating difference layers with reversed gradients with respect to the initial image. In fact, this is a consequence of the staircase effect, reinforced by the recursive application of the bilateral filter. Compared to these methods, the local Laplacian filter has two advantages: first, it does not present a strong staircase effect as in [FAR] and its luminance halo is smaller than in [CPD], because its range parameter σ r is not modified across the scales. As we have seen, the scale-dependent oracle is used instead to give the range weights in the oracle-based unnormalized bilateral filter. Second, LLF allows downsampling, therefore saving a large amount of memory and computations. We display in Figure . the pyramids obtained with a Gaussian filter, BF, UBF, and eventually with UOBF. The filtered images at different scales are superimposed for each filter and displayed in the column on the left. The "Laplacian" coefficients, i.e.the difference between two consecutive levels of the pyramids  are in the right column. In order to compare the results with SLF, the range parameter σ r used in the bilateral filter is kept unchanged over the scales. Each method filters the input image to produce the different scales (no recursion). The first row shows the classic Gaussian and Laplacian pyramids. The two middle rows show the multi-scale decomposition generated by the bilateral filter and its unnormalized version. As the scale increases, these filters do not produce smoother images; in practice the "Laplacian" coefficients between two coarse scales can have high frequencies; Yet it is quite unsettling for a multi-scale decomposition to present roughly the same frequency content at each scale of the decomposition. We have seen that Fattal et al. and Chen et al. suggested ways to get around this, but their solutions present unacceptable artifacts. The last row presents the results obtained with UOBF. The right side plot is obtained in a different way than above. Indeed, this pyramid is computed using a scale-dependent oracle G σ l * u, like the Gaussian pyramid displayed in the top left plot. But the "Laplacian" coefficients are obtained from the difference between two scales of a pyramid that uses the same oracle (see Equation (.)). As seen in Figure . the UOBF filter does not respect the input edges but those of the oracle, which explains that the bottom left plot resembles the top left one. As for the SLF detail coefficients in the bottom right plot, they accurately capture the details at multiple scales.

Put another way, the insertion of an oracle in the bilateral filter allows a proper multi-scale base plus detail decomposition. This proves the importance of such an oracle in the local Laplacian filters. However the previous contributions on LLF [PHK, PHK, APH + , APH + ] do not discuss it: the default oracle is the Gaussian pyramid. We explore in the next section some decomposition produced using different oracles.

. A new framework using different oracles in the scale-space local Laplacian filters

The interpretation of the local Laplacian filters in a Gaussian scale-space allowed us to propose a complete, clean and compact formula for the filter. It also helped to reveal the implicit oraclebased bilateral filter used in SLF. We showed that this modified bilateral filter succeeds in creating  Note that unlike other sections we use the term "pyramid" or scale-space indistinctly here, by reference to [CPD, FAR]. For the sake of clarity, we shall soon return to "scale-space" for methods that does not involve re-sampling . an edge-aware pyramidal decomposition, unlike previous schemes based on the bilateral filter. Furthermore, our scale-space interpretation of the local Laplacian filters has the desirable property of translation invariance, in which the original LLF is lacking, causing several artifacts (see Section .). We shall verify its disappearance in this section. But that is not all: after the discovery of an oracle in LLF's skeleton we decided to make some experiments with it. We shall discover that the luminance halo artifact of the original method can be alleviated by edge-preserved smoothed oracles. This, unfortunately, is paid by the reappearance of a staircase effect and an increased complexity. Once again, we face the dilemma where we have on the one hand a good preservation of edges that comes with the staircase effect and, on the other hand, no staircase but a luminance halo. The improvement brought by our general framework lies in that this decision is left to the user.



Translation invariance Before extending the scale-space local Laplacian filter to different oracles, let examine is behavior with respect to translation. In Chapter  we saw that LLF was not invariant by translation. This creates two artifacts, namely, small bounces and an intensity shift (see Figure . in Chapter ). They are in fact the same artifact at two different scales, and are originated in the sampling of the pyramid. Indeed, the oracle used to remap the input image and compute the output pyramid in LLF is downsampled. Hence, the remapping itself strongly depends on the downsampling grid, which, in turn, causes strong discrepancy between the filtered results of an image and its translated version -even for small translation, e.g., 1 pixel. Figure . shows the result of the application of SLF to our simple test pattern. This test-pattern has already been used in the Chapter  when we described the artifacts that the non-translation-invariance gives rise to in LLF. It is constituted of the same step-edge repeated along the vertical axis. We used the standard oracle, i.e. (G σ l * u). Unlike the previous result obtained with LLF, there is no difference between the two plots (a) and (b) in Figure .. In other terms, the scale-space interpretation in translation-invariant. Furthermore, the absence of the bounce and "mean-shift" we observed in LLF prove that they indeed come from the sampling of the pyramid.

One could think that the oracle does not bring much freedom in the original filter because of the down-sampling. But, although using the Gaussian pyramid is particularly convenient because it is already computed, nothing forces the oracle to be down-sampled. Indeed, it is used to remap the intensity of the full resolution input image at any scale. We want to solve the luminance halo  artifact of LLF. We thus propose to replace the standard oracle G σ l * u by the result of an edgeaware filter, for example, the bilateral filter.

The extended scale-space local Laplacian filters

It has been shown in a previous section that the scale-space local Laplacian filter has a compact formulation (Equation .). We recall its definition:

SLF{u}(x) = lmax-1 l=0 D l * r u -g(x, l) (x) + G σ lmax * u (x), (.)
where g(l) = G σ l * u is the oracle and D l = G σ l -G σ l+1 , with G σ l a normalized Gaussian kernel with standard deviation σ l . We recal that σ l = 2 l-1 σ ref and σ ref = 1. In the particular case of l = 0 we denote G σ l=0 the Kronecker delta. Finally, r() is an odd remapping function. We call extended scale-space local Laplacian filters (ESLF) the filter where the oracle is free, that is, the oracle is an input of the filter. It is given by the user, in the same way as the guide image in the cross bilateral filter, for example. The compact formula of ESLF is

ESLF{u, v}(x) = lmax-1 l=0 D l * r u -v(x, l) (x) + G σ lmax * u (x), (.)
where v is the scale-dependent oracle. This filter is in fact a new general framework in which SLF is a special case. In the same way as in Equation (.), it can be written

ESLF{u, v}(x) = u(x) - lmax-1 l=0 UOBF σ l {u, v(l)}(x) -UOBF σ l+1 {u, v(l)}(x) . (.)
We present results of the filter in the following. We tested several oracles. First, the local Laplacian filter itself: we apply SLF to the input image and then use the result as oracle in ESLF. We also tried the bilateral filter and the guided filter. In these figures, the oracle is obtained with a remapping function that removes the fine variations but keeps strong edges. Indeed, we want an oracle that already describes the base layer. That is, the oracle needs to be smooth and to closely follow the input image. At edges, an edge-aware smoothed oracle still follows the input image, while a Gaussian filtered oracle does not give a good reference intensity for the remapping function.

Results on our test-pattern

.. Luminance halo reduction and staircase effect

One notable benefit of ESLF is the luminance halo reduction. But this comes unfortunately at the expense of an increase in the staircase effect produced by the filter when used for detail smoothing.

In our experiment (Figure .), we use a smooth edge (a step-edge convolved with a gaussian kernel of standard-deviation 3 pixels). We observe the evolution of the ESLF result in function of the parameter σ r of the remapping function r1 used to compute the oracle. by using the different guides shown in (a). We can observe the difference between the input image and all the results in (c).

In Figure . (a), the input image and the guide for σ r = 0 are indistinguishable. This is because using σ 0 amounts to use the identity function as a remapping function. However, it does not mean that the final image in Figure . (b) is the same that the input one: as we have seen before, it amounts to apply the unnormalized bilateral filter. The functions r1 and r2 are independent.

This experiment shows that the smaller σ r in the oracle, the lesser the amplitude of the luminance halo. This is particularly visible in Figure . (c), where the staircase effect appears in the middle of the edge and the luminance halo on the borders; for small σ r the borders stay close to zero but in the center's inverted oscillation the amplitude is higher. It is the contrary for large σ r . Note that the standard oracle is σ r = 1. This allows us to draw two conclusions: first, we demonstrated again the presence of a staircase effect in SLF, second, the oracle chosen in LLF is the one with the least visible staircase effect, but also with the worst luminance halo. That is, between staircase effect and luminance halo the LLF's authors [PHK, APH + ] chose the latter.

On the contrary, we believe that this decision belongs to the user. Nonetheless, the Gaussian filter is the easiest oracle to compute and the luminance halo in LLF is rather large, what makes its acceptable in many case. This is consistent with Trentacoste et al. findings [TMHD].

.. Results

We present in this section some additional filtering results with natural images. We have seen that although edge-aware oracles can reduce the halo spontaneously created by the scale-space local Laplacian filters, they reinforce spurious staircasing effects. Thus, although the staircasing is not visible (yet present) in the standard version, it becomes visible -and that is not acceptable. In the end, the original artifact (luminance halo) is preferable to the one we amplified by correcting it.

We show below some proof to support that statement. Two different images are shown: the first will demonstrate the staircasing amplification; the second will present a case were the luminance halo is the most visible, and demonstrate that it still acceptable. The parameter used in the experiment carried in

• σ r = 0.1 (image dynamic in [0, 1]); • r(t) = t -t exp -t 2 /(2σ 2 r ) ;
• (l max + 1) = 8 (then σ lmax = 2 8-2 = 64;

• S = 50 (We use a sliced version of SLF; yet 50 is largely enough to avoid artifacts).

A summary of the strength of the staircase effect and luminance halos is given in Note that contrarily to the intuitive idea that an increase in the number of scales of the pyramid (thus possibly larger halos) reduces the halo visibility is not always true. For example, in Figure ., an increase in the number of scales used in the filter does not allow to reduce the luminance halo around the streetlight.

. Conclusion

In this chapter, we explored the link between the local Laplacian filters interpreted in a scale-space and the bilateral filters. The scale-space enabled the formal exploration of the paramount feature of LLF: its multi-scale construction. In particular, we demonstrated the prime importance of the oracle, which distinguishes SLF from a multi-scale bilateral filter more than the removal of the normalization. Our conclusions apply to the original (without scale-space) local Laplacian filters, as well as our extended filter proposition. We have made the following contributions:

• two compact formulations for the local Laplacian filters interpreted in a scale-space. These formulae enable a formal discussion and comparison with other filters ;

• a thorough analysis of the scale-space local Laplacian filters. While the authors of the fast LLF [APH + ] already made the link with the bilateral filter and anisotropic diffusion, we go further and reveal the implicit bilateral-like filter and oracle that are used to construct an edge-aware pyramid. In the fashion of the previous chapter where we linked LLF with the exposure fusion method [MKR, MKVR], we bring to light its similarity with previous bilateral pyramids [FAR, CPD], and show that LLF falls into this category;

• the introduction and analysis of the single-scale filter used to construct this edge-aware pyramid, called the oracle-based bilateral filter;



• the solution to the artifacts caused by the lack of translation-invariance of LLF by using a Gaussian scale-space rather than a pyramid;

• the introduction of a new framework, ESLF, which allows to use different oracles to guide the filtering. By using an edge-aware smoothed version of the input image, we also bring the solution to the luminance halo remaining in SLF.

 

We have seen in Chapter  and Chapter  the local Laplacian filter [PHK, APH + ], a multi-scale approach based on the Laplacian pyramid of Burt and Adelson [BA]. Another effective multi-scale filter, the weighted least squares filter (WLS), was proposed by Farbman three years before in  [FFLS].

Unlike other schemes based on the bilateral filter [FAR,CPD], this edge-preserving smoothing approach is grounded on the weighted least squares optimization framework. It is defined as the minimization of an energy composed of a data term that minimizes the distance between the input image and the result, and of a regularization term that penalizes the gradients of the output, except across significant gradients of the input image. Hence, the resulting image follows the input image on its edges and is smoothed elsewhere. The authors proposed two different strategies to build a multi-scale edge-preserving decomposition of an image on this concept.

As we shall see, this filter has objectionable artifacts. The most serious is the compartmentalization effect, that breaks the homogeneity of flat regions when they are split in smaller regions with different areas (e.g. branches of a tree with uniform sky as background). The second one is an asymmetric halo. We shall present in this chapter attempts to correct these artifacts.

After a presentation of the filter in Section ., we describe the halo artifact (Section .) and tackle it in Section .. We then propose two ways to reduce the compartmentalization in Section . and Section .. The first proposed solution remedies to compartmentalization by adding in the functional "remote gradients" terms, so that disconnected regions with similar values are linked and move together. The second solution directly prevents important intensity shifts in flat regions. This works well because these are the places where compartmentalization is mostly visible. Section . describes complementary attempts to improve the filter.

Despite our findings and the improvements, we shall conclude that this filter is not well adapted to contrast enhancement. Indeed, it remains heavy in terms of memory usage and not computationally efficient. Furthermore, our corrections add to its complexity. Nevertheless, for applications on small images or for which computational time is not an issue, one can find in this chapter new good options for an additive base and detail decomposition.

. Algorithm description

The weighted least squares filter (WLS) et al. [FFLS] could not be better presented than by the authors themselves: "Given an input image u, we seek a new image v, which, on the one hand, is as close as possible to u, and, at the same time, is as smooth as possible everywhere, except across  significant gradients in u". This translates into seeking the minimum of

arg min v x v(x) -u(x) 2 + λ a x (u, x) ∂v ∂x 2 (x) + a y (u, x) ∂v ∂y 2 (x) (.)
where (quoting the authors again -notations are updated so as to make them consistent with the rest of the manuscript):

x denotes the spatial location of a pixel. The goal of the data term (v(x) -u(x)) 2 is to minimize the distance between v and u, while the second (regularization) term strives to achieve smoothness by minimizing the partial derivatives of v. The smoothness requirement is enforced in a spatially varying manner via the smoothness weights a x and a y , which depend on u. Finally, λ is responsible for the balance between the two terms; increasing the value of λ results in progressively smoother images v.

The smoothness weights a x (u, x) and a y (u, x) are defined as

a x (u, x) = ∂ ∂x (x) α + -1 and a y (u, x) = ∂ ∂y (x) α + -1 , (.)
where the image is the log-luminance channel of u, e.g., = log(0.2989u R + 0.5870u G + 0.1140u B +s), where s is a small constant. The exponent α is another parameter controlling edge preservation. Quoting the authors, "the exponent α (typically between 1.2 and 2.0) determines the sensitivity to the gradients of u". The small constant "typically 0.0001", prevents divisions by zero.

At pixels where the input u has strong gradients the smoothness coefficients are small and let the data term get the upper hand, therefore transferring the edges from u to v. On the contrary, in areas where the gradients of the input image are small, the regularization (balanced by the parameter λ) enforces minimal gradients in v, which smooths between the edges.

As observed by the authors [FFLS], WLS is nothing but a (linear) anisotropic diffusion [PM, BSMH, Wei, AK]. Indeed, the edge-stopping coefficients do not depend on the output's gradients and do not change during the diffusion.

Noticeably, the image acts as a guide akin to the guide image in the cross (or joint) bilateral filter [ED, PSA + ]. This amounts to saying that the input image u is being filtered according to another related image. For example with a flash/no-flash image pair, the no-flash image, with better colors, is denoised according to the flash image which has bad colors but a smaller noise level.

The minimization problem has an analytic solution, given by the following equation:

(I + λL u )v = u (.)
where

L u = D T x A x D x + D T y A y D y
, where D x and D y are forward difference operators, A is a diagonal matrix containing the coefficients a, and I is an N × N identity matrix. (N is the number of pixels). L u therefore is a five-point spatially inhomogeneous Laplacian matrix, which form is as follows

Lu =                             d(
d(N )                             , (.)



where i is the index for the (column-wise) vectorized images of smoothness coefficients a x and a y given in Equation (.). The diagonals 1 and -1 handle the a y smoothness coefficients and diagonals h and -h the a x ones, where h denotes the image's height. The diagonal with elements d is minus the sum of the elements in the line, so that for each line of L u the sum of the elements is null. This gives a rather large (N × N ) matrix to invert. Fortunately, it is sparse, with only  non-zero diagonals, so solving Equation (.) is not a problem thanks to dedicated solvers. The authors uses the multiresolution preconditioned conjugate gradient solver described by Lischinski et al. [LFUS].

We give the pseudo-code of the filter in Algorithm . It consists in () computing the coefficients a x (u) and a y (u), () creating the sparse matrix (I + λL u ) and finally () solving Equation (.).

Multi-scale decomposition

In [FFLS], Farbman et al. present two methods for computing the progressive coarsening sequence v 1 , . . . , v lmax . The first one is to solve the linear system in Equation (.) l max times, each time increasing the value of the parameter λ. In other words,

v l+1 = WLS c l λ {u} (.)
for some initial value of λ and some factor c. This decomposition is akin to Chen et al.'s one proposed in [CPD], presented in Chapter . A coarsening sequence generated in this manner and the corresponding detail layers is shown in Figure .. The second method is to apply the operator iteratively by v l+1 = WLS c l λ {v l }.

(.)

In this method the image is repeatedly smoothed, similarly to mean-shift filtering [CM] and to the multi-scale bilateral transform of Fattal et al. [FAR]. The latter is described in Chapter .

The authors still increase λ by a factor of c at each iteration, as this results in a more significant  smoothing at each iteration. The authors found this iterative scheme better suited for applications that discard or attenuate some of the details, such as image abstraction (with α = 1.8 or 2.0). We show in Figure . and Figure . the multi-scale decomposition of an image with the first and second proposed methods, respectively. In these examples we take l max = 3. We call base the filtered images at each level, so that there is a total of three bases v 1 , v 2 and v 3 with different levels of filtering. For convenience, we use the notation v 0 for the input image u. The three detail layers are the difference between two consecutive levels of the pyramid

d l = v l -v l+1 .
(.)

The input image can be exactly recovered by collapsing the pyramid:

u = lmax-1 l=0 d l + v lmax . (.)
Similarly to Burt et al. Laplacian pyramid, we call the last base layer v lmax the residual. As explained by the authors, the iterative method tends more strongly towards piecewise constant regions separated by strong edges, which encourages them to recommend it for image abstraction, while they recommend the first method for HDR compression and multi-scale detail enhancement.

Results and artifacts

We now present two contrast enhancement experiments using both proposed multi-scale decompositions. Figure . (b) shows the result of method  described in Equation (.). Figure . (c) uses method , described in Equation (.). Farbman et al. recommend method  for contrast enhancement and tone-mapping rather than method , which gives a nearly piece-wise constant result that they judge better suited for image abstraction. In this example we used three levels for the decomposition (l max = 3). To make the most of the multi-scale decomposition, the finer scales are more enhanced and the base layer (i.e. the last scale of the pyramidal decomposition) is shrunk in a Durand-Dorsey-like [DD] tone-mapping by enhance(u) = log(255v lmax + 1)/ log(256) + d 2 + 2d 1 + 3d 0 .

(.)

The obtained images are stretched to fit the output dynamic range, but we allow for robustness a saturation on the left and right hand sides of the histogram for 0.1% of the pixels. This saturation is called "simplest color balance" in [LLM + ].



The compartmentalization artifact

The WLS filter has two noticeable artifacts. The first one is compartmentalization, which happens when a large region with a constant intensity (e.g. the sky) is compartmented by a thin network in the foreground, typically the branches of a tree. This creates small regions with the same constant intensity as the underlying large region, yet disconnected. Because WLS takes into account the direct neighbors only, these small regions are then free to evolve independently. The smaller their area, the lower the steadying influence of the data term compared to the gain obtained by reducing the gradients at the edge of that element. Thus, the lower its area/perimeter ratio, the stronger a small region will affected. Obviously this effect increases with λ. In are never equal to zero. The gradients of the strong edges are then still costly. The functional minimization therefore leads to a contrast reductions of those edges, which in turn creates a luminance halo. Surprisingly, this halo is much more pronounced in its negative part than in its the positive one. As we shall discover soon, this is due to the usage of the logarithm for the computation of the smoothness coefficients.

Another artifact of WLS is observed in [HST]: the intensity shift. When increasing the parameter λ (in order to give more smoothness to the base layer), the detail layer sometimes contains entire (relatively constant) regions. This is visible in the Figure . (g) for example where the street floor and the sky become particularly bright. This means that the detail layer may contain low amplitude variations, which is often not desirable. This is again a compartmentalization symptom, which is enhanced when the detail layer is enhanced. The issue here is not the fact that disconnected regions (yet close and with the same color) get diverging colors, but rather that a single large region undergoes a spurious intensity shift.

We worked on these issues and proposed two different versions of WLS, each solving one of the above described nuisances. As already announced, we shall first handle the halo (Sections . and .), then the compartmentalization artifact (Sections . and .). To reduce halos, we modify the coefficients a x,p and a y,p to relax the smoothness constraint on the edges. As for the compartmentalization, two solutions are discussed: the first one uses distant gradients to "keep in contact" separated regions. Another successful option is to detect, then protect, flat areas. Indeed, the compartmentalization artifact is particularly annoying in constant regions like a clear sky. The values of α are those used by the authors for method 1 (α = 1.2) and for method 2 (α = 1.8). The parameters λ were chosen so that the PSNR of both images is the same. This is a way to compare the "quantity of detail" extracted by the filter. The detail layers clearly display a thin black halo. The section shown in (a) (red line) is displayed, for the base layers, in (e), and for the detail layers in (f ). The detail lines on the right show the thin but high amplitude dark halo (peaks toward the bottom) and large yet smaller amplitude bright halo (peaks towards the top).



While working with this filter, we observed the creation of shocks at edges, namely the same phenomenon that we called staircase effect for the bilateral filter (see Chapter ). This effect is not visible in the original filter, at least we could not see it in our experiments. For this reason, we do not consider it as an artifact of WLS. However, it could appear, because nothing really prevents it. We shall discuss this issue in Section ..

Execution time, memory consumption

The computational time of this algorithm depends on the solver. We tried with the very efficient Matlab solver  and the execution is fast. However, the memory consumption becomes really high for large images, despite the usage of sparse matrices. This is an issue because the problem can't be solved on smaller pieces of the image. Min et al. in their  paper [MCL + ] present an efficient scheme to solve a similar weighted least square problem for edge-aware smoothing.

. Asymmetric luminance halo

The first artifact we tackle in the WLS filter is the asymmetric halo. As we shall understand in this section, it comes from the fact that the edges contrast is reduced even when their gradient are large.

This artifact is shown in This is due to the usage of the logarithm in the guide image -the image on which the gradients are measured. Figure . shows the results when removing the logarithm in the formula : the asymmetry disappears. The negative part of the luminance halo now is as large and with the same amplitude as the positive part. So, what happened? Applying a logarithm to the luminance (guide image) before the computation of the gradients increased the gradients in the dark parts, which in turn decreased the smoothness coefficients of these areas, thus reducing the smoothness constraints. In the bright parts on the other hand, the gradients were reduced by the logarithm and therefore the smoothness constraint increased. At the end of the day, a stronger regularization is imposed on bright parts than on dark parts. This causes an asymmetry in the luminance halo: in the dark (negative) part the gradients are free to take high values, which produces a high amplitude halo with small width, whereas in the bright regions (the positive part of the halo) the gradient keeps small values, thus propagating the halo farther.

Link with the total variation

We recall that the regularization term in WLS writes

λ a x (u, x) ∂v ∂x 2 (x) + a y (u, x) ∂v ∂y 2 (x) (.) where a x (u, x) = ∂ ∂x (x) α + -1
with the log-luminance channel of the input image u. In the following experiment, we replace by u itself and set α = 1. Omitting the parameter λ, Equation (.) becomes

∂u ∂x (x) + -1 ∂v ∂x 2 (x) + ∂u ∂y (x) + -1 ∂v ∂y 2 (x). (.)
Since is negligible and because we initialize v with u, the partial derivatives cancel, and we get the total variation. We show the result of this setting in Figure .. Remark that even if at the initialization the energy is like the total variation, the smoothness coefficients depend on a guide image that is different from the output one, so it progressively diverges and this is still the square of the gradients that is penalized.

Conclusion on the black halo artefact

The asymmetry in the luminance halo comes from the logarithm. We have seen that by replacing the log-luminance by the luminance removes this artifact. We have also seen that the halo is due to the fact that the smoothness coefficients are not small enough to let the data term take over. Consequently, edge contrast is reduced. In practice, three effects of the WLS model are combined. First, the logarithm. The smoothness coefficients a x (u, x) and a y (u, x) are smaller in dark areas relatively to the bright areas, whatever the parameters value: this is because the logarithm stretches gradients in dark zones and compresses it in the bright ones. The second effect to take into account to understand the behavior of WLS is that the smoothness constraint cannot be interpreted pixel-wise. Through the gradient term, a pixel can affect the entire image. The stronger the smoothness constraint, the wider the propagation. When the smoothness constraint is soft, the pixels are more independent from their neighbors -yet, as we have seen, this constraint is never null. The third effect concerns the smoothness coefficients values at edges. One would like this coefficient to be zero for sufficiently high edge like the street light edges in Figure .. Yet the contrast at these edges is squeezed by WLS, which means that in the model, it is cheaper to compress these edges rather than to keep them unchanged. The reason is that the smoothness coefficients are not null at edges. At first sight this is a major issue in the filter. We then try to remedy it. Yet, as wee shall see soon, setting the coefficients to zero at edges yields to oversharpening. Indeed, by removing the constraint on the gradients, the edges are free to increase; and often this is what happens because it allows more smoothness in both sides of the edges.

Advantage of an asymmetric halo

Notably the black halo is not an artifact as visible as the compartmentalization. In fact, the negative part of the halo is often way less visible than its positive part.

We show in Figure . what would give contrast enhancement with an asymmetric halo, but in the reverted direction, i.e., the positive part (bright one) thinner but with a higher amplitude than the negative part. To that aim we designed a test-pattern made of a step-edge centered in the dynamic range (in practice the test-pattern goes from 0.2 to 0.8). We then added a small variance Gaussian noise that served as detail. This test-pattern is displayed in Figure . (a). We then used it for contrast enhancement, through a very simple formula: enhance(u) = u + 2(u -WLS{u}). The parameters used were α = 1 and λ = 10. In the original method, the smoothness coefficients a x (u, x) and a y (u, x) are computed using the log-luminance, that is, in our case, = log(u) since our test-pattern has only one channel. As we have already seen, this creates an asymmetry in the smoothness constraint: the dark areas get smaller coefficients than the bright areas, which creates the asymmetry in the luminance halo: in the positive part the amplitude is lower and the width larger than in the negative part. But this particular shape makes the halo less noticeable than other  where lum is the input (gray-scale) image, and in (c) with = log(1 -lum). Thus the (asymmetric) halo is reverted between (b) and (c). This allows to see that a thin-but-high-amplitude white halo is more visible than a black one.

The bottom plot displays superimposed sections of the two different detail layers.

shapes, and more specifically than the opposite one, i.e. a halo with a higher amplitude yet smaller width on its positive side than on its negative one. In order to get such a halo, we simply reversed the contrast of the guide image: the smoothness coefficients are computed using = log(1 -u) (the input image's dynamic is [0, 1]). A section of the two detail layers obtained is displayed in the plot of Figure .. The contrast-enhanced test-pattern obtained with and are shown in (b) and (c). We find that the reverted halo is much more visible than the initial one, which is barely noticeable. In fact, this is a well-known photography rule: the human eye is attracted by white parts first. Thus the white halo "pops out" instantly when looking at a picture, while the black halo is more discreet, even if both have the same "strength", namely the same amplitude and width.

. Halo reduction by improving the regularization term

We shall now try to reduce the luminance halo in WLS. The first thing we did is to switch from the log-luminance to the luminance in the smoothness coefficients computation. This brings symmetry back in the luminance halo, but does not reduce it. On the contrary, it makes it more visible, since the asymmetry allowed to hide it. What we propose now is to modify the smoothness coefficients so that the edge contrast is not reduced in the solution v. We simply need to reduce their value for the edges we want to protect. As can be seen in the plots, this completely removes the luminance halo, because the two sides of the edges become completely independent. However, another consequence is the oversharpening of the edges. The gradients at edges are indeed now free to increase; since it allows more smoothness for regularized regions, it often happens. We already fell in the same trap in Chapter  on the scale-space local Laplacian filter. The oversharpening is known for the bilateral filter as the staircase effect; it has extensively been described in Chapter . The consequence in the final enhanced image is a gradient reversal artifact. The Figure . shows the detail layer obtained with this modification (in (b)) and compare it with the standard version (a).

.. Threshold and erosion on the smoothness coefficients

Coefficients erosion

We now try to solve this issue by widening the areas where the smoothness constraint is almost null. The idea is to relax the constraint on the border of the edges too, so that more gradients are allowed to increase, and, in turn, the increase of each gradient is smaller and less visible in the final result.

We made the following experiment: we added an erosion step in the algorithm, after the hardthresholding. That is, the smoothness coefficients were normally computed, but then we set to 0.0001 those smaller than 50, as we did in Figure . (top row). In addition to this, we eroded the thresholded smoothness coefficients. This way, the smoothness constraint was removed for the pixels surrounding the edges too. We expected the oversharpening to be removed, because by allowing more pixels to "move" (in the range domain) their modification would be smaller. We display the result (detail layer only) of this experiment in Figure . (c). On the bottom row of Figure . we display on the left sections of the original image, the standard WLS result and this experiment's result, and on the right the corresponding detail layers. The structuring element used was a 3×3 pixels wide square.

The coefficients' erosion considerably reduces the oversharpening artifact (compare top and bottom row in Figure .), although it does not make it disappear completely. In the detail layer of Figure . (c) some gradient reversals are still visible (see the vertical one on the street light and the horizontal one in the clouds). Furthermore, this affects the entire image, since only the smallest coefficients are retained. Thus the smoothness constraint becomes smaller overall. We were not satisfied by this result and tried to get something better by using a function with a smooth transition rather than a hard threshold. Figure 11.10: Section of an image filtered with WLS (left column) and detail layers (right column). The original image is shown in blue, the WLS-filtered with original coefficients appears in orange, while the WLS-filtered image with modified coefficients is drawn in green. On the top row use use a threshold at 50 and in the second row a threshold at 50 followed by an erosion with a 3 × 3 structuring element. The oversharpening effect is seriously reduced by the erosion.

.. Modification of the smoothness coefficients' function

In the two previous experiments, we tried to remove the luminance halo by reducing the smoothness constraint on the edges by thresholding the coefficients. This made oversharpening appear and showed that, once again, one has to find a balance between those two artifacts. We diminished the oversharpening by applying an additional erosion to the modified coefficients, but the success of this trick was moderate. Here we shall try to tackle the problem by directly modifying the function proposed by the authors [FFLS] (see Equation (.)). We designed a new function f (.) and applied the composite function to the guide image . Unless otherwise notified, we use = u. We call a our modified coefficients. They are defined as

a z (u, x) = f ∂ ∂z (x) α + -1 , (.)
where z is either the x or y direction, and

f (t) = t 1 π tan -1 t -s ω + 1 2 β . (.)
This function, unfortunately, has three parameters s, ω and β. These many parameters prevent exhaustive exploration of the different settings. However, experiments led us to a particular set of parameters, which gave good results. It is notable that with this set of parameters, erosion is no longer necessary. We used s = 1, ω = 1000 et β = 8. A comparison between the initial and modified coefficients is presented in Figure ..

Filter calibration Our next step is to "cross-calibrate" the filter with other available filters. The method we use for this is to set the PSNR of the detail image (averaged over five reference images).

The fixed PSNR value that we want to obtain is the one obtained on average for these  images with the original guided filter. Our filter has a total of five parameters, in which four are fixed: the three ones of the function f (s, ω and β, set respectively to 1, 1000 and 8) the parameter α that we fixed to 1. Hence, the only parameter left is λ which controls the overall amount of smoothing.

Following the procedure we just explained, we got λ = 50. We show in Figure . the results obtained with this configuration. We denote WLS this modified filter. We increase the number of neighbors taken into account in WLS. We represent here only the right side of the pixel; the same addition is made in practice for the bottom side. In the initial scheme only the direct neighbor is considered; we add some additional neighbors.

Results

The result on this image and with these settings is very good: no luminance halo, staircasing, or contrast halo are visible. However, the contrast is well enhanced. This experiment demonstrates the importance of smooth transitions between the strong and the weak coefficients, because it avoids sharp transitions in the optimized result v. Into the bargain, the compartmentalization phenomenon persists but it is quite weak. It is in fact less present than in the original WLS.

. Super-connected weighted least squares filter

Regarding the compartmentalization artifact, our idea was to insert in the smoothing term nonlocal gradients making pixels interact at larger distance. Indeed, in WLS only the gradients with the four nearest neighbors are taken into account; our modification consists in increasing this neighborhood to pixels located for example at a distance of three and five pixels (what we call paws).

The same coefficients a x (u, x) and a y (u, y) are used. We show in Figure . an illustration of this process. By forcing the filter to minimize low gradients between near but disconnected pixels (those connected through the paws), we hope to counteract the effect of independent variation of nearby areas of the same intensity but separated by a few pixels of different intensity (that is: to counteract the compartmentalization). Indeed, the compartmentalization artifact arises when two (disconnected) areas with the same initial intensity become in the result filtered images of different intensities. By penalizing the gradients between those two areas, this should not happen any more. For example in the case of the scale on the arrow in Figure ., the pieces of sky inside the metal scale will be connected (thanks to the long paws) to the sky from outside the scale. In practice, this idea works (see Figure .) but is not completely satisfactory. Indeed, we need many paws to effectively reduce the compartmentalization, but the addition of paws greatly complicates the problem. As we said earlier, the matrix to invert in WLS is huge, but the solution can be calculated relatively quickly thanks to its sparsity. However, for each paw we are adding a diagonal in the matrix L, and this considerably increases the algorithm's complexity. For this reason, we have not pursued research in this direction. We can write the new regularization term

λ nmax n=1 a x (u, x, n) (v(x + nv x ) -v(x)) 2 + nmax n=1 a y (u, x, n) (v(x + nv y ) -v(x)) 2 , (.)
where we consider n max neighbors (in the original WLS filter, n max = 1), v x = (0, 1) and v y = (1, 0) are unit vectors, and the smoothness coefficients in directions x and y become With the matrix notation, this amounts to adding to the matrix L u new difference operators. Let denote D x,n the forward difference operator with the n th neighbor in direction x. So D x = D x,1 . We use the same notation for direction y and for the smoothness coefficients matrix A x,n and A y,n . The minimization problem stays the same, i.e.

a x (u, x, n) = |u(x + nv x ) -u(x)| 2 + -1 , (.) a y (u, x, n) = |u(x + nv y ) -u(x)| 2 + -1 . (.)  (
(I + λL u )v = u, (.) but the matrix L u is now defined as

L u = nmax n=1 D T x,n A x,n D x,n + nmax n=1 D T y,n A y,n D y,n . (.)
We have added two additional diagonals to L u (see its matrix writing in Equation (.)) for each neighbor taken into account and for each of the two dimensions x and y. The total number of non-zero diagonals is then 1+4×n max . Clearly, this has the inconvenience of drastically increasing the computational time, because the matrix becomes denser.

In practice, we would need to measure far away "gradients"; the closest ones are not of paramount interest. Indeed, our aim is to jump over discontinuities. This is why we decided to use only a reduced number of neighbors. More specifically, we considered only the 3 and 5-pixels away gradients. In short, we consider n ∈ {1, 3, 5}. Figure . displays the result obtained with this algorithm and compare it to the original one. As can be observed, the compartmentalization artifact is reduced: areas that move (they come out as brighter part of the sky inside the metal scale in the detail layer, for example) in the initial result (see images (b) and (d)) are better preserved in the super-connected WLS result (images (c) and (e)).

It has been demonstrated in [Ela, BC] that the bilateral filter can be seen as a multi-scale version of the anisotropic diffusion. By introducing those supplementary intensity difference  terms in the filter, we make the smoothness constraint multi-scale, thus making WLS closer to the bilateral filter.

. Flatness detection

The places where the compartmentalization artifact is particularly annoying are flat areas "cut" by some hollow object. Thus, we tried to detect these areas and to prevent any shift in their intensity during the optimization.

Algorithm : Flatness detector input : image u input : threshold t and radius r output: image flat containing the flatness score  for each pixel x do

 for each pixel y ∈ [-r, r] 2 do  T (y) = c∈{R,G,B} (u c (x) -u c (x -y)) 2  flat(x) = y∈[-r,r] 2 s t (T (y)) // equation (.)
We shall modify the functional so as to increase the influence of the data term in the flat regions (according to an arbitrary detector). In order to do this, the minimization problem is rewritten as

arg min v x γ(x) v(x) -u(x) 2 + λ a x (u, x) ∂v ∂x 2 (x) + a y (u, x) ∂v ∂y 2 (x) , (.)
where we added the coefficient γ in front of the data term. We shall set γ 1 in flat regions, and γ = 1 elsewhere.

For the purpose of detecting flat areas, we use the basic algorithm described in Algorithm . It works as follows: for each pixel of the input image, and for a considered square neighborhood of width 2r + 1, we count the number of pixels whose absolute intensity difference with the central pixel is inferior to a user-set threshold t. In the pseudo-code, this threshold appears in the function s t (.) defined by

s t (x) = 1 if x < t 0 otherwise. (.)
The flatness detector in Algorithm  outputs a score that we call flat. We then give a value to the coefficient γ as a function of this score, using

γ(x) = 1 + ρ flat(x) (2r + 1) 2 4 , (.)
where r is the radius of the window for the flatness detector and (2r + 1) 2 is the number of pixels in that window. The parameter ρ allows the user to control the penalization strength. Finally, the exponent 4 was found manually. Overall, this flatness criterion is arbitrary and could be replaced by any other flatness detector. By denoting Γ the diagonal matrix containing the γ values, the analytic solution is given by Unlike the super-connected WLS, this variant does not increase the complexity, except for the flatness computation. The pseudo-code of this modified filter is easily obtained from Algorithm , where we add a line at the beginning for the computation of the flatness coefficients using Algorithm  and Equation (.). We also modify the matrix construction at line  to include those flatness weights in accordance with Equation (.). We show in Figure . the result of this algorithm. The parameters used are the same as in Figure . (in which the original image is displayed) so as to have comparable results. The WLS filter with flatness detection successfully removes most of the compartmentalization artifact. Noticeably, it also removes the bright part of the luminance halo for this specific image, because the whole sky is detected as flat. The flatness coefficients are observable in (c). This solution works well but is not totally satisfactory because some small areas are still not detected by the flatness detector.

(I + λΓ -1 L u )v = u. (.)  (

. Isotropic filter and Laplacian penalization

In this section we shall present two further attempts to improve WLS. The first one consists in making the smoothing isotropic, by using the same smoothness coefficients for the x and y directions. This brings WLS even closer to the Perona-Malik anisotropic diffusion. This modification has however only a very small impact on the result, and no particular advantages. The second attempt also concerns the smoothness term. We replaced the penalization of the first derivative by the second order derivative ; this allows more variation in the smooth parts of the image.

.. Isotropic smoothness coefficients

We replace the directional smoothness coefficients a x (u) and a y (u) by a single isotropic smoothness coefficient a(u) defined by

a(u, x) =   ∂ ∂x 2 (x) + ∂ ∂y 2 (x) α/2 +   -1 . (.)
In practice we use their modified version a (u) = f (a(u)), where f is the function defined in Section .., Equation (.). Its parameters are fixed to s = 1, ω = 1000 and β = 8. We recall that this modification reduces the luminance halo in WLS by reducing the smoothness constraint   displays the result obtained with this filter and compare it to the original one. The results are very similar: the only observable difference seems to be the amplitude of the extracted detail in (c). But this is due to the fact that the norm of the gradient in this image is globally larger than the absolute value of the gradients in x and y. This explains the difference of smoothing because stronger gradients imply smaller smoothness constraint on the output WLS{u}.

This modification of WLS does not improve the filtering. It does not degrade the result either so we kept it in the following experiment, which is to replace the gradient by the Laplacian in the smoothness term.

.. Smoothness by Laplacian penalization

This time, the output image v is given by solving

arg min v x v(x) -u(x) 2 + λã(u, x) ∂ 2 v ∂x 2 2 (x) + ∂ 2 v ∂y 2 2 (x) . (.)
That is, we replaced the first order derivative by the second order one in the regularization. This way, we hope to better handle slopes in the signals. Indeed, with this functional we allow more  variation in the smooth parts, thus perhaps less halos. The smoothness coefficients are updated too. We use

ã(u, x) = ∂ 2 ∂x 2 + ∂ 2 ∂y 2 + -1 . (.)
Here again, we reduce the strength of the coefficients at edges with the function f (see Equation (.)). But this modification produces spurious ringing at edges. Figure . shows this effect. While the solution of WLS is to cancel the second order derivative of v, this new filter requires to cancel the fourth order derivative. It plainly does not help to get a better result. The oscillations are introduced at edges are not a desirable property.

. Conclusion

The WLS filter is powerful for defining a multi-scale base and detail decomposition but not quite adapted to large consumer images because it requires too much memory and is not computationally efficient. This is nevertheless a good filter that we recommend to use when time or memory are not an issue, or when dealing with smaller images. We showed in this chapter several ways to remove both the luminance halo and compartmentalization artifacts, namely, the introduction of the function f (.) that decreases the smoothness constraint on edges (Section .), therefore removing the halo (beware however the oversharpening artifact that may arise when completely removing this constraint). Both the super-connected WLS filter (Section .) and the flatness detector (Section .) significantly reduce the compartmentalization artifact.

 

In the previous Chapter , we looked at the weighted least squares filter, which proposes a multi-scale decomposition of an image by successive applications of the filter without downsampling, similarly to previous multi-scale decomposition based on the bilateral filter -apart from the local Laplacian filter, based on the Laplacian pyramid.

In this chapter, we propose a simple multi-scale implementation of the guided filter that rely on the Laplacian pyramid of an image. As we shall see, a straightforward implementation leads to the creation of the dark halo artifact, typical of the multi-scale filters based on the Laplacian pyramid. We encountered the same artifact in the multi-scale bilateral filter with regression, described in Chapter . We show that a simple modification in the pyramid reconstruction solves the problem. This correction takes advantage of the guided filter linear model. It leads to a fast filter giving a very clean base and detail decomposition. The comparison we carry out on thirteen filters in the next Chapter  shows that this filter is effectively one of the best options available.

. Multi-scale guided filter

The multi-scale guided filter (MGF) improves on the guided filter in quality and on the iterated guided filter in execution time. It relies on a multi-scale representation of the image: it uses the Laplacian pyramid as described by Burt and Adelson [BA]. At each scale, starting from the coarsest one, the guided filter is applied. We also present a variant that uses the iterated guided filter introduced in Chapter . We shall call it the multi-scale iterated guided filter (MIGF).

A direct implementation of the multi-scale guided filter as the recursive form

z lmax = GF{Lpyr{u, l max }} z l = GF{Upsample(z l+1 ) + Lpyr{u, l}}, (.)
where Lpyr{.} is the Laplacian pyramid, Lpyr{., l max } its residual, u the input image and z l the filtered image at scale l. The downsamping and upsampling procedures, as well as the construction of the Laplacian pyramid are described in Chapter , in Algorithm , , and Equation (.) respectively. When using the standard guided filter for image smoothing, the guide is the input itself. In the recursive scheme of Equation (.) however, the filtered image progressively diverges from the input one, so we have an alternative: either to guide the filter with the filtered result at the previous iteration, or with the unmodified input image. The former produces a stronger smoothing effect, but it also generates a stronger dark halo. We thus choose the latter. We refer to the Section . on the multi-scale bilateral filter with regression where this artifact was first  discovered and explained. The reason behind the dark halo reduction in the multi-scale guided filter is that rather than recursively distorting the edges, at each scale the detected edges are in a way restored by the filter. Indeed, we recall that the guided filter output is

z(x) = ā(x)v(x) + b(x), (.)
where v is the guide image. The coefficient ā tends to zeros where there is no edge. That is, the edges in z are transferred from v. In fact, before the aggregation step (see Chapter  on the guided filter), the coefficients a and b are constant in each patch and in this case we directly have ∇z = a∇v. So, in Equation (.) the guided filter at scale l is guided by Gpyr{u, l}, i.e., the Gaussian pyramid at scale l of the input image. This prevents the progressive distortion of the edges and thus reduces the dark halo artifact. The Gaussian pyramid construction is detailed in Equation ..

The pseudo-code of this filter is given in Algorithm . It calls the iterative guided filter (v2), because it allows the user to choose between the original guided filter (with T = 1) of its iterated version (T > 1). Indeed, one may iterate the filter at each scale to increase the global smoothing. In practice we found it unnecessary, because the mutli-scale scheme already allows to obtain large spatial standard-deviation for the filter. Hence the default parameter for the number of iterations T is 1. We recall in Algorithm  the pseudo-code of IGFv (proposed in Chapted ). We also recall the guided filter pseudo-code in Algorithm . The multi-scale guided filter complexity is O(rT N ) where r is the radius, T the number of iterations and N the number of pixels.

At Line  in Algorithm  we modify the parameter . We used smaller values for coarser levels:

l = /(l + 1), (.)
where is the parameter set by the user, the one used at the finest scale when l = 0. A similar strategy was used by Fattal et al. [FAR] in their multi-scale bilateral filter. The experiments we carried out with this filter showed that this setting produces a more appreciable decomposition, with more high-frequency details than low-frequency ones. We found it more appropriate for contrast enhancement. Note that the multi-scale retinex does enhance more the high frequencies than the low ones. See Section . and Equation (.) for a description of this filter. Some results obtained with this filter can be seen in detail separation, it is safer to let details be less pronounced along edges than risking artifacts". In their paper they propose a multi-scale edge-stopping diffusion that remove the staircase artifact but introduce a contrast halo. They argue that "since the edge itself is perceptually dominant (. . . ) a possible loss of texture detail is significantly less disturbing compared to an incorrect (. . . ) edge." Our experiments seems to confirm this statement; in Figure . for example, image (f) has a stronger contrast halo than in (e) (it is visible in (b) and (c)). Yet the difference is barely noticeable.

Conclusion

We demonstrated in this section that it was possible to produce a good base and detail decomposition using the Laplacian pyramid. The local Laplacian filter uses this pyramidal scheme too, but there is a notable difference between our filter and LLF. Indeed, LLF needs the full resolution image to compute each different scales of the output Laplacian pyramid. On the contrary, MGF filters directly the downsampled images. Another filter using a similar pyramidal scheme is the recursive multi-scale bilateral filter proposed by Fattal et al. [FAR]. But rather than downsampling the images they use an algorithme à trou, which drastically reduces the kernel size for coarse levels but is still applied to every pixel of the input image. On the contrary, in our scheme we save a lot of computation because we filter downsampled images -with small kernels too.

 

13 Quantitative artifact evaluation and a final filter ranking

We have so far presented and examined the virtues and defects of the most prominent existing filters, and proposed several new ones. From the bilateral filters, in Chapter ,  and , to the local Laplacian filter in Chapter  and , passing by the guided filter, Chapter ,  and , the weighted least squares filter in Chapter , the exposure fusion in Chapter  and , we explored a large part of the literature on the edge-aware smoothing filters, concentrating on the ones causing the least artifact and when possible with low computation needs. Furthermore, for each studied filter, we named and defined its most cruel defects and proposed at least one alternative version diminishing these artifacts.

In this last chapter, we compare the filters that we presented in the previous chapters. We shall perform a quantitative evaluation of the five main artifacts of the contrast enhancement we met during this thorough review, namely, the simple (luminance) halo, the contrast halo, the staircase effect (edge sharpening), the compartmentalization (closing effect) and the dark halo (described in Chapter , seen in Chapters  and ). For each of these artifacts we propose a test-pattern specifically designed to reveal it, along with a way of quantifying it. This evaluation gives a clear overview of the capacity of the tested filter to perform a clean base and detail decomposition. Based on the proposed measures, we eventually propose a ranking of thirteen representative filters in the literature along with those proposed in this thesis. However, not all contrast enhancement filters are based on base-detail decomposition. For the sake of completeness, additional comparisons are provided with well established tone-mapping filters that do not perform this decomposition e.g., multi-scale retinex (MSR), automatic color correction (ACE) and simpler methods based on histogram equalization.

. Introduction

We shall first design a set of five test-patterns, one for each identified artifact. Each one is paired with a measurement method giving a quantitative evaluation of the presence of the artifact for each filter. The test patterns are constructed in Section .. Then, we establish of short list of filters, that we believe to be representative of the variety of filters proposed in the literature. But we face an additional difficulty: to fairly compare filters, we must have a rule to set their parameters. To that aim we develop a general procedure. We first compute an average L 2 norm for the detail of a small set of representative natural images. This parameter fixing procedure is detailed in Section .. Once these preparing steps are accomplished, we evaluate the presence of artifacts for the thirteen selected filters in Section .. This study yields an objective filter ranking, and leads to declare winners three filters achieving a clean base and detail decomposition. In the following Section . we apply those filters on natural images and confirm the ranking.



. Test-patterns for qualitative evaluation of the artifacts

In this section we present the five measures designed to quantitatively evaluate the five canonical artifacts of edge-aware filters. Each measure is associated with a test-pattern specifically designed to detect and measure one of the artifacts. We limit our evaluation to the five main artifacts we met along our review in this thesis. They are: the luminance halo, the staircasing, the compartmentalization the contrast halo and the dark halo.

.. The luminance halo

The luminance halo is the most common artifact of edge-aware filters. It simply corresponds to the fact that an edge has been smoothed, even slightly, while it should have been preserved. This is an artifact that can naturally arise in every method, since it is linked to the amount of smoothing, always accessible with a parameter. What we measure here could be better described as the capacity of the filter to preserve the edges given a certain amount of detail to extract. Indeed, we shall use this metric on filters with fixed parameters -fixed so that every filter extracts the same L 2 norm in average for a representative image dataset. The setting of the parameters is described in Section .. This means that we effectively measure the presence of this artifact for comparable enhanced results.

In order to measure the luminance halo, we constructed a test-pattern that contains several straight edges with varying amplitude. The expert's intuition is that such edges should be conserved in the basis as soon as their contrast is large enough. The measure then simply quantifies the distance between the filtered image and the input one.

Test-pattern

The test pattern is in reality constituted of six different test-patterns. Each one has an edge with different intensity and they are far apart. This separation prevents any interaction between edges with different heights. The six edge heights are {0.3, 0.4, 0.5, 0.6, 0.7, 0.8}. Each test pattern has the same size of the composite test-pattern displayed in Figure . (a). Only the middle band of each one is displayed here. The measure is realized on this composite image.

The halo measure Based upon this test-pattern, the halo measure is defined as

H = Ω 1 u 0 <0 (x)(|u 0 (x) -u 1 (x)| + ) 2 dx + Ω 1 u 0 >0 (x)(|u 1 (x) -u 0 (x)| + ) 2 dx, (.)
where x is a pixel, u 0 is the input image, u 1 is the filtered image, u 0 is the second derivative of u 0 and 1 is the indicator function. We denote by Ω the image's domain. |.| + denotes the positive part. Because a few strong differences are more visible than numerous small ones, we square the  positive part in Equation (.). In practice we use two masks for the indicator functions, the first (1 u 0 <0 ) is a rectangle of width 40 pixels placed on the left side of the edge, the second (1 u 0 >0 ) has the same size and is placed on the right side.

Which filter performs worst

The worst filter for the luminance halo artifact is the guided filter. We show in Figure . what it produces for our test-pattern, using the fixed parameters given in Table ..

.. The staircase effect

The staircase effect has been described in Chapter  (Section ). It is typically present in the bilateral filters; it is sometimes referred to as an edge sharpening effect. Numerous correction schemes have been proposed in Chapter . However, those corrections often fail at effectively correcting it; even the bilateral filter with regression that has been proposed specifically to solve this artifact can present it in some conditions such as a large spatial support.

The staircase test-pattern Like in the luminance halo test-pattern, the image we use for measuring the staircase effect is a composite of six different test-patterns, each of which contains an increasingly smooth edge. A step edge is convolved with a Gaussian kernel in the Fourier domain with parameters σ = {0.7, 1.4, 2.8, 5.6, 11.2, 22.4}. Figure . shows a preview of these testpatterns in the second image from the left. The middle band of each image only is displayed. The measure is computed using this composite image.

The staircase measure Using the test-pattern we just described, we measure the edge reinforcement in the six bands simultaneously using

S = Ω 1 u 0 <0 (x) |u 1 (x) -u 0 (x)| + 2 dx + Ω 1 u 0 >0 (x) |u 0 (x) -u 1 (x)| + 2 dx. (.)
In practice we use masks for the indicator functions: 1 u 0 <0 is a rectangular binary mask with height the image's height and 40 pixels width (standard-deviation of the tested filters). It is placed on the left side of the edge. As for 1 u 0 >0 , the mask is a rectangle with the same size placed on the right side of the edge.

Worst filter

The worst filter is definitively the bilateral filter in this case. We display in artifact appears. The worst case is always the finest edge, but the attenuation when the edge width increases depends on the filter, see Table ..

.. The compartmentalization effect

The compartmentalization effect arises when a constant color region (typically a wall, or the sky) is divided in small pieces by, for example, by the superposition of a grid or of tree branches. We have seen this artifact in Chapter  on the weighted least squares filter (WLS), but it also appears with the grain filter described in Annex, in Section .. This artifact consists in an intensity shift in constant regions; its magnitude depends on the area of the region. The smaller the region, the stronger the artifact. The total variation [ROF, CS] presents a similar artifact that would rather be called closing, like in the morphological operation: it consists in the removal of local extrema with small area. This effect will be measured with our test-pattern too.

The compartmentalization test-pattern

The compartmentalization test-pattern is made of bright squares and rectangles of different areas disposed on the dark background. We display it in Fig- ure . (third image from the left). This image is slightly smoothed to avoid aliasing.

The compartmentalization measure Using the test-pattern described above, the measure is defined by P = Var ((u 0 -u 1 )1 white ) = σ 2 white , (.)

where σ 2 white is the variance of the detail layer in the bright squares deprived of a 2-pixels wide band along the borders that exclude pixels influenced by the edge. This mask appears in blue in the illustration of Figure .. The removed pixels in the squares are shown in red.

Filters with worst compartmentalization

The most spectacular filter concerning this artifact is WLS. Figure . displays the result obtained with this filter using the parameters set in Section .. With the false colors used, the bright regions appear in yellow and the dark ones in gray; in the detail layer in (c), one can clearly see that WLS "lights" the small shapes. The small spaces between the rectangles are sometimes affected too, but that is not taken into account by our measure. The higher score for the compartmentalization however is not obtained by WLS but by the domain transform filter (see Table .). Yet the compartmentalization present in DT is related to its luminance halo artifact; in fact edges are not well preserved by this filter, which causes the small elements to be smoothed out even if their contrast is high. Thus, all shapes in the test-pattern are affected. For filters like WLS however, there is a distinct separation between shapes that are affected (those that are lighten up in the detail layer) and those that are not.



.. Contrast halo

The contrast halo appears when regions containing details and close to edges are not filtered. This artifact is typical of the guided filters and has been described in Chapter . Measure Using the test-pattern described above, the contrast halo is measured by comparing the variance of the detail layer in the interior of the bright rectangles with the variance on the border of these rectangles, as shown in Figure .. Because of the luminance halo, this ratio can sometimes be inferior to one, i.e., the variance in the exterior side of the bright rectangles becomes higher than in the interior. We thus simply measure the maximum between one and the ratio. Formally, this gives

The test-pattern for contrast halo

C = max 1 , σ 2 ext σ 2 int -1, (.)
where the subtraction of 1 only aims at giving the same minimum to C than the other measures, which will be useful in the final comparison. The two measures of variances are obtained thanks to masks, displayed in Figure .. The value of σ 2 ext is measure in the blue regions and σ 2 int in the yellow ones. ext is computed and the yellow on where σ 2 int is computed. The red lines are excluded from the computation (those are the antialising pixels), as well as the white part of this image. On the left we display the expected detail, i.e., the noise we added in the white rectangles.

Filters with worst contrast halo

.. The dark halo

The dark halo appear when manipulating the Laplacian pyramid coefficients. It has been explained in detail in Section . when we proposed the multi-scale bilateral filter with regression.

Test-pattern

The test-pattern for measuring the dark halo is constituted of dark bands on a bright background, inspired by the image of Figure .. We simulated the streetlight-like pattern at different scales and applied a Gaussian blur with standard-deviation σ = 0.7 to remove aliasing.

Measure

The dark halo is a reverted halo. Contrarily to the luminance halo, it amplifies edges. In our test-pattern, where the edges are simulated by thin dark bands, it provokes an increase of the intensity next to the edges. We then compute the gradient along the x direction of the filtered image u 1 , and then measure how much negative the gradient is on the left part of the white band in between two dark lines and how much positive it is in the right part. Formally, this writes also produces this artifact. We superimposed in the same plot the base given by LLF. The multiscale guided filter does not have a dark halo artifact, despite the use of the Laplacian pyramid. A straightforward implementation does, but the problem can be easily corrected, as described in Chapter .

D = (| ∂u 1 ∂x | -) 2 1 left + (| ∂u 1 ∂x | + ) 2 1 right , (

. List of selected filters and parameters setting

In this section we elaborate the list of filters that will be part of the final competition (Section ..) and discuss how to set fair parameters for each (Section ..). Indeed, when we shall compare in Section . the strengths of their respective artifacts, it is of prime importance to first ensure that they are compared in a condition where the final result (contrast enhancement) is similar. Our strategy is as follows. We shall select the thirteen most representative (and competitive) filters; to give them equal chances we fix their parameters so that the L 2 norm of the delivered detail layer is equal for all of them.

.. List of tested filters

In the course of this manuscript we have presented and explained numerous different filters and variants. We list them all in Table .. Some filters do not perform a base and detail decomposition. We shall consider them too; we list them in the separate Table .. In order to make the comparison clear and understandable, we chose to restrict ourselves to thirteen most representative filters.

The selected filters are listed in Table ., along with their parameters. For the bilateral filters, we keep the most representative of the fast approximations, namely, the bilateral grid. We also keep the fast bilateral filter with regression proposed in Chapter  and the multi-scale bilateral filter with regression described in the same chapter, in Section .. We also consider the domain transform  filter presented in Chapter . This filter has a small complexity, which makes it attractive for many applications. The guided filter is inevitable. Concerning our propositions of iterated guided filters we kept the version where the guide remains unmodified over the iterations (IGFv). We also add to the list the multi-scale guided filter introduced in Chapter , as well as the multi-scale iterated guided filter (based on IGFv, introduced in Chapter  too). The original weighted least squares is put in competition in this chapter, not its modified version of Chapter  because they increase its complexity. The fast local Laplacian filter is considered, but not its scale-space version proposed in Chapter , mainly because its time and memory overload due to the scale-space and oracle computation exclude it from the competition. Finally, we added a modified grain filter, presented in Annex (in Section .) adapted from the original morphological grain filter to our needs of a base and detail decomposition. This filter gives an overview of what could be expected from the morphological filters in the context of additive base and detail decomposition. We then consider a filter representative of the total variation field, the TV-L 1 using the Chambolle-Pock method, and another filter based on the minimization of the L 0 norm of the gradients that we briefly present now.

In [XLXJ], Xu et al. propose to smooth an image while preserving its main edges by minimizing the number of non-zero gradients. Formally, they seek an output image v that minimizes

min v x (v(x) -u(x)) 2 + λC(v) , (.)
where C() counts the number of pixels x whose gradient is not zero, and λ a parameter that controls the amount of smoothing. Such a problem is particularly difficult to solve. The authors thus introduce two auxiliary variables and solve the modified problem, that approximates the original one, by alternatively minimizing two problems, each of them with a closed-form solution. The approximation is controlled by a parameter β: a large value approaches the initial Equation (.).

The minimization is then iterated, starting with a small β that is progressively increased. According to the authors, 20 to 30 iterations are generally needed and most of the computation is spent on FFT. The low curvature image simplifier filter (LCIS) proposed in  by Tumblin and Turk [TT] can decompose an image in a base and detail layers well adapted to contrast manipulation. Their filter is related to the anisotropic diffusion. The solution of their partial differential equation tends to regions with uniform gradients (low curvature), instead of constant regions in AD. Hence, their filter produce a piece-wise affine approximation of the input image rather than a piecewise constant one. Unfortunately, the solution of this equation is a slow iterative process that makes it unpractical for large images. Moreover, the tone-mapping operator they propose involves several applications of LCIS. To give an order of magnitude, Fattal et al. [FLW] indicates . minutes for a 751 × 1130 pixels image. Moreover, the coefficients in LCIS must be adapted to each image [DD], which does not correspond to our need of an automatic decomposition. For these many reasons we omit LCIS in our comparison. It can anyway be represented by the iterated guided filter, since they belong to the same family of anisotropic diffusion related iterated filters.

In , Subr et al. [SSD] proposed an edge-preserving filter in which they define the detail as the oscillations between local extrema. They first find the extrema locations, then construct two envelopes using an edge-aware diffusion technique proposed by Levin et al. [LLW]. The base layer is then obtained as the mean between the maximal and minimal envelopes. This filter has three drawbacks from our point of view: first, the produced detail has very high amplitude oscillations, which are not appropriate to tone-mapping or contrast enhancement. Second, it causes a strong compartmentalization effect. This can be verified in In , Fattal et al. [FLW] published a "gradient domain high dynamic range compression" technique. This method directly computes the output image by defining its gradients in function of the input image's one. More precisely, it computes the gradients and applies a spatially-varying compressing function that reduces the amplitude of large edges and preserves the amplitude of small ones. That is, G(x, y) = ∇I(x, y)Φ(x, y), where G are the gradients of the output image, ∇I(x, y) those of the input image and Φ the compressing function. Because the D gradient field G is not necessarily integrable, they approximate the solution by seeking the minimum of ∇I(x) -G 2 dx, where x = (x, y). This least squares problem leads to solving a large system of equations. In order to perform base and detail decomposition one can modify Φ so that the large edges are preserved and the small ones smoothed out. This algorithm actually already exists and is called WLS; it is part of the final contest.

Ward Larson et al. [LRP] in  proposed a global histogram adjustment method, efficient when the input histogram has empty portions, but limited when the input exhibits a uniform histogram. In , Pattanaik et al. [PFFG] presented a tone-mapping operator based upon psychophysically-derived filter banks. This technique has the drawback of presenting luminance halos. Other global tone-mapping operators can be found in e.g. Reinhard et al. [RSSF] and Drago et al. [DMAC]. A third option is to perform both global and local manipulation, as proposed in the two-stage algorithm by Ferradans et al. [FBPC], later extended by Cyriac et al. in [CBKVC]. These methods are based upon neural and psyphophysical models of visual perception. In the same spirit, Benzi et al. [BEK] recently used the virtual retina simulator [WK], developped by Wohrer et al. in  in neuroscience to model the retina (it transforms a video into spike trains) to build a tone-mapping operator for videos. We refer to the Reinhard et al. book [RHD + ] that gives a good review of a number of tone-mapping operators until . Other resources on tone-mapping operators can be found in, e.g., Eilertsen et al. work [EMU] for HDR video, or [LCTS] for psychophysical experiments on tone mapping operators compared with linearly mapped HDR scenes displayed on a HDR screen. [Gue] -λ = .205 L 0 image smoothing (L 0 -IS)



[XLXJ] κ = 2 λ = .002

Table 13.3: The different filters, their fixed and their variable (only one) parameters. For IGFv2, the couple of parameters (r, σeq) = (2, 40) gives T = 400 iterations; for the filters with range subsampling (FBF, FBR and FLL) the number of samples is S = 64; for WLS, the guide image is the default one, i.e., the log-luminance of the input image.

Figure 13.11: Images used to set the parameters of all methods. Gray conversion is computed here by averaging the three color channels.

.. Parameters

Once the list of filters fixed, we need to set fair parameters. Using the "default" parameters given by the authors wouldn't be right; we need to set them so as to obtain similar results for all methods in the enhanced images. Indeed, a "prudent" filter might cause less artifacts (at least, they would be less visible), but the detail enhancement might be insufficient by then. We therefore propose to equalize the L 2 -norm of the detail layer. In practice, since the detail depends on the content of the images and since it can have very different aspects, we select five different images and equalize the average where N is the number of pixels in the image. Like before, u 0 is the input image and u 1 the filtered one. The PSNR is measured on each image filtered with the current parameter, then averaged.

The canonical value we try to get is PSNR = 16.23 dB. We obtained this value with the same five pictures and the original guided filter. The detail layers obtained with each filter and with the parameters given in

. Artifact measurements

In this section, we measure the strength of each of the four artifacts presented above. A table will display the results for all the filters listed above. Our method is simple: for each filter, using the parameters given in Table ., we measure the tested artifact using equations given in Section .. We display the detail layers given by each filter in .. Indeed, the detail is the most informative part and this is the easiest way to distinguish between the filters. Each table comes with a plot that presents the value of the respective measure, sorted in descending order: each time, the smallest the value, the better the filter; this way the filters are directly ranked. In a second part, we make a summary of the five tables.

Luminance halo In Table . we present the results for the luminance halo and the detail layers of each filter. For a definition of the abbreviations we refer to Table .. The worst filter in this case are the L 0 image smoothing and the guided filter. The first because when the edge gradient is below a threshold it gets reduced, and this reduction is propagated relatively far. As for the guided filter we saw in Chapter  it is unable to both well preserve the edges while smoothing sufficiently the low variance areas of an image.

The staircase effect In Table . we present the detail layers obtained with each filter for the staircase effect test-pattern. Once again L 0 -IS is the worst filter: indeed, minimizing the L 0 norm of the gradients tends to create constant parts in the image, which corresponds exactly to the staircase effect. Unsurprisingly, the (fast) bilateral filter comes next after L 0 -IS. The apparition of the staircase effect in the bilateral filters has been described in Chapter . The bilateral with regression obviously reduces this artifact (it was designed for that purpose). It is nevertheless not completely removed: Indeed, using large spatial standard-deviation σ s as we do here makes it unable to correct the staircase effect for high-frequency edges (in the first bands of the test-pattern).

The weighted least squares filter has, to a certain extent, this artifact too, with the particularity that it is way more marked in the dark side of the halo. As we have seen in Chapter , this is due to the use of a logarithm in the gradient penalization: dark part of the images are allowed to move more than the bright ones.

The compartmentalization artifact In Table . we present the results obtained with the compartmentalization test-pattern. The expected detail is a constant image: indeed, the test-pattern does not contain texture, but only very contrasted edges. On the contrary, filters with compartmentalization tend to "light up" some of the squares or rectangles of the test-pattern in function of their area. The worst result for this measure is the domain transform (DT), that tends to smooth out the small objects whatever their contrast. Note that contrarily to WLS that lights up only some shapes in function of their areas, DT affects every shape. So for this filter the compartmentalization is linked to the luminance halo. As expected, WLS has a very high score too. This effect was described in Chapter . In a few words, because even the edges see their gradients slightly penalized, is becomes sometimes worthy, in terms of energy minimization, to reduce those edges if the area inside is small enough, because the data term will not compensate the gain (having too few pixels). The second worst filter is TV-L 1 . Indeed, this filter is prone to removing the edges of objects and "closing" regions with small area. Then comes the bilateral grain filter (see the Annex, Section .). This filter is based on the removal of shapes (here they correspond to the yellow squares and rectangles) in function of their area; the compartmentalization effect is then obviously present. We observe however that the very small shapes are not present, which shows that our grain rejection based on the contrast works. Next, L 0 -IS has also a bad score, for the same reason as presented in the luminance halo paragraph. In its case the compartmentalization is not  really annoying because it seems to affect the shapes whatever their area. Other non-zero results are mainly due luminance halo, to which our test-pattern cannot be completely insensitive. Note also that the "contour highlighting" visible in FBF, BFR and L 0 -IS is due to the staircase effect. This however does not influence the value of our measurement.

The contrast halo results

The measures realized with the contrast halo test-pattern are presented in Table .. As explained in Section ., with this test-pattern we aim at measuring if detail is affected in the vicinity of an edge, put another way, we measure if the smoothing is the same in the vicinity of edges as at a certain distance from it. The expected detail layer is shown in the table (top row, middle column). The filter that obtains the highest and therefore worst score is the guided filter. This artifact has been extensively described in Chapter , and solved in Chapter  with the iterated guided filters IGFv, IGFv and IGFv. We nevertheless see here that this artifact is reduced but still present. In fact, every filter based on the guided filter keeps a bit of contrast halo. Among them, the least impacted by this artifact is the multi-scale guided filter (MGF) proposed in Chapter . Interestingly, the bilateral filters (FBF and FBR) have a score superior to one, even slightly. Note that, unfortunately, our test-pattern is not independent from the luminance halo. Indeed, it tends to make the ratio in Equation (.) inferior to one, because the halo makes the variance to increase next to the edges. This explains why FLL has a score of 1 despite a small contrast halo visible on the detail layer. The best results here are obtained by L 0 -IS and MGF, also WLS fares well in spite its luminance halo. The result of BGRF is due to the compartmentalization artifact, because panes in the bottom right have smaller areas than in the top left corner.

Dark halo measurements

The dark halo measurements are presented in Table .. This artifact appears when manipulating the multi-scale coefficients in the Laplacian pyramid (see the multi-scale bilateral filter with regression in Section ., or the multi-scale guided filter in Chapter ). In brief, manipulating coefficients in the pyramid is hazardous because their oscillations at different scales, that normally compensate when collapsing the pyramid, are very dependent, so filtering each scale independently generally creates inconsistency in the collapsed image. The expected detail layer is zero everywhere, or with luminance halos (our dark halo measure is insensitive to the luminance halo artifact). Detail layers with dark halo are detected by their blue halo around the originally dark lines. Obviously, only multi-scale filters are affected. In practice, MBR is the worst and FLL only slightly suffers from it (see Figure .). The multi-scale MGF and MIGF however does not have this artifact, because the interaction between scales is taken into account (see Chapter ).

Summary and conclusion on the comparative experiments

We display in Figure . a summary of the measurements obtained for each artifact. The procedure to synthesize the measurements is as follows: first, we square the measures to make them more discriminant, then we normalize each different measure by dividing its scores by the standard deviation of the thirteen filters' values. After the normalization the scores are weighted according to the annoyance they introduce in the enhanced results. Indeed, the artifacts are not equally annoying, so a simple normalization would not provide a faithful ranking from the expert's viewpoint. We discuss this point in the next paragraph. We finally sum the normalized measures.

According to the final ranking, the best methods are the multi-scale guided filter (MGF), the fast local Laplacian filter (FLL) and in third place the fast bilateral filter with regression (FBR). We shall see in Table . that this last filter still creates too much staircase to be acceptable. As we stated above, the artifacts are not comparable. One can thus give an ordering of the artifacts themselves. In collaboration with the DxO image experts we worked with, we give the following ranking, in ascending order of acceptability. The most annoying artifact when performing a contrast enhancement by enhancing the detail is the staircase effect, because is creates edges that do not exist in the original image. Then, the compartmentalization because it breaks the initial homogeneity of some areas. Next, the dark halo. Indeed, this artifact gives sometimes quite unnatural results (see Figure . with MBR, the streetlight for example. See also Figure .). The contrast halo comes in fourth position; regions with initial homogeneous texture get distorted after enhancement, being less enhanced next to the edges. This artifact however needs to be strong to become objectionable, which justifies its position. Note that Eilertsen et al. in [EMU] make the same observation and suggest that because the edge itself is perceptually dominant a loss of texture or detail is less disturbing compared to an incorrect edge. In last position we have the luminance halo, which, among the artifacts, is the most acceptable. Indeed, it is present in most methods, and arguably in our own perception as modeled by the retinex theory. This ordering of the artifacts is reported in Table . and in the Figure . using weights.

Another difficulty with the measure we propose is the highly nonlinear response of the subjects to the artifacts. That's why we advocate here for a second and complementary way to select the best filters, based on a rejection criterion. As we shall see, this second classification confirms our previous results. Intuitively, a filter is rejected when an artifact is too conspicuous in the enhanced result. Such a rejection is necessarily subjective, hence we mobilized five image experts at DxO and provide the result of their judgment performed on the set of images presented in Figure .. An image is rejected if at least four of the five judges marked it as objectionable. The experiment was carried-out on contrast enhanced images with an high factor on the detail. The comparison is performed by flipping between the input images provided for reference and the enhanced ones. Note that this strategy is similar to the work of Trentacoste et al. [TMHD] on the luminance halo. Hence, for each artifact we obtained a binary decision, "acceptable" or "unacceptable", summarized in green and red in Table .. Each test-pattern disqualifies at least one method:

• the staircase effect invalidated L 0 -IS, FBF and BFR;

• the compartmentalization invalidated DT, TV-L 1 , WLS and BGRF;

• the dark halo invalidated MBR only (This artifact is small enough not to be objectionable in LLF);

• the contrast halo invalidated GF, MIGF and IGFv; (MGF has a slight contrast halo but so small that it is indistinguishable in real images -see • the luminance halo invalidated DT, GF, L 0 -IS; (WLS and FLL could also be considered as rejected but, as we said, a small luminance halo remain acceptable in most of the cases -see Table . and Table .).

This table shows that only a few filters succeeded passing the five artifact tests. This classification confirms the podium obtained in Table ., where the first places are attributed to FLL and MGF. The difference between both rankings only concerns MBR which is rejected here because of its too strong staircase artifact.

Filters complexity

Another important parameter that we must finally take into account in is the computational time of each method. Indeed, the decomposition of an image in base and detail is only a part of complex image pipelines, and generally positioned ahead of other treatments. Hence, the rapidity requirement. Furthermore, as we have seen the filters always have at least one parameter controlling the amount of detail extracted by the method, or, put another way, the amplitude of what is considered as detail. This parameter greatly changes the final result of enhancement chains, so one may leave it to the user for fine-tuning. This, however, requires displaying a preview of the final result in real-time. Giving a ranking of the filters in function of their execution time is difficult, as it is highly dependent both on the implementation and the machine used for the tests. We can nonetheless base our ranking on the theoretical complexities. These are summarized in Table ..

The guided filters have a linear complexity with respect to the image size N . The iterated versions introduce T , the number of iterations. It is of several hundreds for IGFv, yet way smaller for the multi-scale version. Indeed, the equivalent standard-deviation needed at each scale is much smaller. The multi-scale filters MGF, MIGF, FLL and MBR have basically the same complexity as their single-scale equivalent, except that their process involves 4 3 N pixels instead of N . This number is the total number of pixels in a Gaussian/Laplacian pyramid when N → ∞. The fast approximations of the bilateral filter using the Paris-Durand approach (FBF, FBR and FLL) have a dependence on R, the dynamic range if the image. But they downsample the volume N R in function of σ s and σ r , so the higher these parameters, the faster the filter. The domain transform has been reviewed in Chapter , it is one of the fastest filter available for edge-aware smoothing; it has a O(N ) complexity with a smaller increase than the GF and FBF in function of the dimensionality.

Among the three top-ranked filters found in this section, the multi-scale guided filter is the Table 13.5: Summary of the considered filters' complexity (and abbreviations).

fastest. Indeed, it requires to compute only two pyramids: the input one, progressively modified, and the guide's pyramid. The guided filter computed at each scale is very fast. On another hand, though fast too, FLL requires the construction of R/σ r pyramids and additional interpolation weights to compute the (supplementary) output pyramid. With the parameters given in table . this represents 11 pyramids. The fast bilateral filter with regression lags far behind, as it needs 2 FFTs and 9 IFFTs per slice (R/σ r slices), but also requires solving a linear system of equations (to compute the coefficients of the linear regression) for each pixel. This aspect of the filter, along with its staircase artifact, are two arguments that rule FBR out of the competition.

. Application to natural images

We present the decomposition results of the thirteen considered filters for five natural images in a series of six tables. The first two display the base layers, see where EAF stands for any edge-aware filter. That is, we apply a square-root to the base layer and slightly shrink it, while the detail layer is amplified. We suppose the input dynamic range in [0, 1].

The blurry image with the obelisk (fourth column) is particularly relevant for the staircase effect. One can easily distinguish the inverted contrast band along the obelisk for the bilateral-based filters (apart from the multi-scale bilateral filter with regression), L 0 -IS and the iterated guided filters. The trellis image (third column) is particularly relevant for the contrast halo artifact, as  In Le Guen et al. paper [Gue] at page , one sees that each iteration has roughly  operations per pixel (op/pix) and that the recommended iteration number is about  (page ). This amounts to a complexity of  op/pix. As this decomposition is rather local, the number of iterations is arguably independent from the image size, for a given scale parameter. Strange horizontal oscillations with amplitude 1 appear in BGRF and TV-L 1 results. This is due to the conversion from double to unsigned 8bits integer needed in their implementation. This is not due to the algorithm itself, and have extremely little influence on the measures. On the contrary, yellow on the left and blue on the right reveal luminance halo but this effect can be better appreciated in Table 13.6.

Strange horizontal oscillations with amplitude 1 appear in BGRF and TV-L 1 results. This is due to the conversion from double to unsigned 8bits integer needed in their implementation. This is not due to the algorithm itself, and have extremely little influence on the measures.



. Conclusion

In this chapter, we wrapped up the previous developments on the filers, those studied and those introduced. We set up a methodology based on the artifacts defined during the dissertation. We defined five measurement pattern pairs enabling a quantitative evaluation of the base and detail decomposition's quality. Our procedure evaluates the filters according to criteria that faithfully reflect the photographers' aspirations. Notably, contrarily to most existing methods we did not attempt to measure the quality of the tone-mapped images, but rather of the algorithms themselves. We thus placed ourselves in a more difficult case where we cannot expect that artifacts introduced in the decomposition are compensated in consecutive steps of the algorithm. The good point is that, once a filter is validated by our set of measures, it has the guarantee not to introduce objectionable artifacts in any image, making our methodology more general.



We addressed in this dissertation the fundamental problem of the base and detail decomposition of an image for contrast enhancement. In the industrial context in which this thesis was undertaken, our work was driven by two key criteria: the corrected photograph must still look natural, that is, without visible distortions; the algorithm must be fast so as to integrate seamlessly into a complex pipeline and to display to the user a preview in real-time. This thesis's contribution is twofold: first, it is an in-depth review and analysis of filters computing a base, leading to propose improved ones -notably one that surpasses them all; second, it analyzes the causes of the artifacts and proposes a methodology to quantify them, hence obtaining an objective way to rank all base+detail decomposition filters.

Our analysis of existing base + detail filters has established formal links sometimes ignored, bridging for example the gap between ACE and the bilateral filter, or connecting the recent guided filter to an anisotropic diffusion PDE. The detailed analysis of each filter has revealed their artifacts and their cause, often leading us to propose improved forms. For instance, we solved the guided filter's contrast and luminance halo with an iterative scheme ; we extended exposure fusion to single image contrast enhancement; we proposed a more flexible and precise scale-space adaptation of the local Laplacian filter, expressed with a new compact formula. One of the introduced filters, the multi-scale guided filter, actually ranks above all studied filters, while having a low computational cost.

This thorough review, our proposition of new filters and the identification of the five main artifacts of the edge-aware smoothing filters led us to the second part of our work, namely, the comparison of the different methods based on their artifacts. To this aim we designed a pattern for each specific artifact, along with a measure quantifying its magnitude. We then proceeded to the final evaluation of thirteen edge-aware smoothing filters and their ranking. This task was not as simple as it looks in retrospective. Indeed, each filter has its own definition of a detail -the aspect of the detail layers can be very different. We thus proposed a way to set the parameters of the filters so that their detail layers have a prefixed oscillation (simply measured by an L 2 norm).

The aggregation of the filters' rankings according to each of the five artifacts was not trivial either. Indeed, the perceived annoyance for each artifact is not uniform. It was necessary to attribute an adequate weight to each artifact measurement. This weighting was steered by DxO image specialists' subjective evaluation of each artifact.

To summarize, we have intended to provide the community with a clear identification of the different types of artifacts in the base and detail decomposition filters together with a methodology to measure each of them.

It is noteworthy that our method does not try to evaluate the quality of final images. We placed ourselves in a more difficult case where we cannot expect that artifacts introduced in the decomposition are compensated in consecutive steps of the algorithm. In fact, this methodology is more precise since in case of a bad score, it directly delivers the faulty artifact. On the other  hand, our proposition cannot handle filters that do not perform a base and detail decomposition. It would be extremely interesting to be able to rank the final results for all contrast manipulation filters, not just those performing a base + detail decomposition. We surmise that our use of artifact measurement patterns could be extended in that direction. The difficulty remaining is to define an objective enhancement measure applicable to all filters, so as to compare them all for a prefixed amount of enhancement.

It turned out that the three filters that stand out are multi-scale filters. We believe that this is attributable to the reduction of their range and spatial parameters for MGF and MBR -the smaller the parameters, the smaller the artifacts -and, in the case of LLF, to the multi-scale oracle (Gaussian pyramid).

From another point of view, it would be really interesting to explore the link between our findings concerning the artifacts (and their nuisance) and the psychological studies on the human visual system. For example, why the luminance halo is the least objectionable artifact has certainly to do with the fact that such a halo is in fact present in our vision (in the Mach bands illusion). But could we explain the ordering of the remaining artifacts?

  

In this annex, we first describe the grain filter and then present a new "bilateral grain filter" that we propose. It computes the base by integrating, along the branches of the tree of shapes, the intensity of the connected components. We introduce weights in the equation so that the base contains not only the shapes with large areas, but also these with an high contrast. Furthermore, the transition between fully accepted shapes and rejected ones (that goes in the detail layer) is smooth, which gives the decomposition an aspect more suitable for contrast enhancement.

. Bilateral grain filter

We introduce in this section a grain-based filter that constructs the detail layer by attributing weights to the grains in function of their size and contrast, in a similar way as in the bilateral filter. The detail is then defined as composed of the grains with small area and whose contrast with respect to the background is low. We shall define what we mean by grain contrast in this section. The introduction of these spatial and range terms in the grain filter have the interest of producing a detail layer more suitable for contrast manipulation.

The tree of shapes [MG, BCM] is constructed by fusing the trees of connected components of upper and lower level sets [CMM]. A shape is defined as the interior of a level set. That is, a shape corresponds to the connected component and its "holes". The tree of shapes is a non redundant representation of an image using an inclusion tree of the shapes. An illustration of this tree is provided in Figure ..

Filter definition

The grain filter, on which we shall base the filter proposed in this section, consists in pruning the tree of shapes: given an area threshold, which is the sole parameter of the algorithm, all shapes with area smaller than the threshold are removed. The reconstructed image from this pruned tree of shapes gets simplified, but keeps its main level lines. This is what makes this filter attractive for contrast enhancement. Indeed, because the level lines are not modified (some are removed, the other stay unchanged) this filter cannot create luminance halo, nor staircasing.

Our filter works as follows: given a pixel x in the image, we find the smallest shape in the tree that contains it. This corresponds to the end of a branch of the tree. Then we progressively go back up in the tree until the root is reached. We integrate along this path the intensity difference between the consecutive shapes. This would amount to reconstruct the original image, but in order to construct a base and a detail layer we weigh the intensity differences following two criteria: the area of the shape and its contrast. This resembles the spatial and range parameters of the bilateral filter so we called this filter the bilateral grain filter (BGRF).



The filter can then be expressed as follows BGRF{u}(x) = T t=0 I(u(x), t + 1) -I(u(x), t) + 1 -g (A(u(x), t)) h (C(u(x), t)) , (.)

where u is the input image and t is an index going from 0 for the end of a branch to T , the index of the root. Note that T can be different for different pixels; it depends on the length of the branch on which the pixel lies x. We denote I(u(x), t) the intensity of the shape including x at index t in the tree. Here g(.) and h(.) are two weighting functions for the area A(u(x), t) and the contrast C(u(x), t), respectively. They are described in Equation (.) and Equation (.). We shall come back to Equation (.) in the following. We now define the area and contrast measure of a shape. The area A of shape(u(x), t) is its number of pixels. We then define the growth speed S of the shape along the path to the root by S(u(x), t) = A(u(x), t + 1) -A(u(x), t).

(.)

When S(u(x), t) is small, i.e., when the area grows slowly, the shape's contrast is high: the level lines are very close. On the contrary, when S(u(x), t) is high, this means that the shape's contrast with respect to its background is low, because two consecutive shapes have very different areas.

We thus define our contrast measure as C(u(x), t) = N S(u(x), t) + 1 , (.)

where N is the area of the entire image (number of pixels). The 1 at the denominator avoids division by zero. By multiplying the inverse of S by N , we set the contrast C to the same range of the area A, i.e., [1, N ]. This intuitive definition of the contrast gives 1 when the contrast is null and N when it is maximal.

Weighting functions

It is interesting to examine the histograms of the shapes' area A and shapes' growth speed S in an image. We observe in These two histograms however do not tell whether it is mostly the small shapes that have a high contrast, the large ones or if this behavior is uniformly distributed. This information is given in Figure .. In this figure, the columns give the growth speed (low on the left, high on the right) and the rows give the area. For display we use a

√

A and √ S scale. From this image one can see first that there is no pair with high growing speed (namely low contrast) and large area (see the bottom right part of the figure). The pairs are concentrated on the top left, which means that the vast majority of shapes have both a small area and a small area speed growth, namely have a high contrast. The part in which we are interested is the top of the image (small areas), albeit even the large areas can be considered as detail depending on the image's scale. But within the top we only want the right part, where the contrast is small. Indeed, we do not want to place the strongly contrasted shapes in the detail layer. Indeed doing so would risk a compartmentalization. We therefore propose two sigmoid functions, one for the area, one for the contrast, that we call f (.) and h(.), respectively: Multiplying the two weighting functions g(.) and h(.) ensure that the shape migrates to the detail (i.e., g(.)h(.) = 1) only when both measures go above the smooth thresholds λ and ρ. In the implementation, the minimal intensity difference is 1. Furthermore, for consecutive shapes with more than 1 as intensity difference, we suppose that several shapes with the same size are stacked.

In that case the weight given to those stacked shapes is 1 because C = N , therefore h(C) = 0. This is why in Equation (.) we omit the weights for the term I(u(x), t + 1) -I(u(x), t) The pseudo-code of the filter is presented in Algorithm .

Note:

In practice the parameters let to the user are ρ = sign(ρ) abs(ρ), λ = sign(λ) abs(λ) and β = √ β rather than directly λ and ρ, because this is more intuitive to set. The parameters indicated in the following figures are λ and ρ.



Algorithm : Bilateral grain filter (BGRF) input : image u input : parameter λ limiting the migration of the shapes in the detail layer in function of their area (a higher λ means smaller area shapes in the detail layer) input : parameter ρ limiting the migration of the shapes in the detail layer in function of their area growing speed (a higher ρ means higher contrast in the base layer) output: BGRF{u}: the filtered image  ToS u ← FLST{u} // construct the tree of shapes of u We present in Figure . the results obtained with this method. In this figure the parameters λ and β are fixed, and we change the parameter ρ in the six presented base/detail couples. By keeping in the base layer the small shapes with a high contrast value C, the detail layer looses contrast. The compartmentalization however is not really reduced. Furthermore, it seems that most of the small shapes migrate to the base layer when increasing ρ. The decomposition that we get with this filter seems to produce a detail layer containing only medium-scale shapes (i.e., the large shapes but only the ones smaller than the parameter λ) with low contrast (high growing speed for the area). This is not exactly what we wanted.

Conclusion

Despite the improvement of the decomposition for contrast enhancement over the initial grain filter, the results are still not satisfactory. Indeed, the compartmentalization effect could not be removed, creating "distortions" in the images after enhancement. See for example Figure .: in the detail layer the areas between the white columns of the building on the right, some contrasted black structures appear, as well as white contrasted elements on the top of the building. This is not desired in the decomposition. In fact, this just shows that the compartmentalization effect is still present and annoying. Another example is given in Figure . between the branches of the tree. Furthermore, the detail layer globally contains only intermediary size shapes: the small shapes generally stay in the base layer, because of the contrast term. This gives flat results in every color example shown in this section.

Furthermore, we believe that one should also take into account not only the area, but also the shape of the connected component, in order to admit thin elongated shapes (possibly with a large area) as detail rather than base.

The results section however suggests that this filter could be used for other tasks such as im-

Figure 1 . 1 :

 11 Figure 1.1: Plot of the function sα used in ACE

  Figure1.2: MSR and the mutli-scale guided filter (MGF). Multi-scale retinex introduces a halo around the lighthouse, but MGF does not. The multi-scale guided filter is a base and detail decomposition algorithm; the decomposition obtained for the luminance part of the input image is displayed on the first line. Both algorithms work on the luminance only. Note that MSR does not manage to preserve the contrast of the lighthouse facade, while MGF does; besides, the base and detail decomposition gives much flexibility to the algorithms, which could be used for example to further increase the local contrast.

  Figure1.3: Contrast halo artifact in the guided filter: the smoothing is reduced near strong edges. On the left: step-edge 1D-signal with a small noise (blue line) and its smoothed version with the guided filter (red line). On the right, we show the detail layer: difference between the two signals on the left: input -filtered (green line). The detail layer is almost flat in its center, where the input signal has its step-edge. For comparison, to input noise of the test-pattern (expected detail) is displayed below (blue line). The difference between these two signals is also presented (red bottom line), showing that the obtained detail almost perfectly equals the noise everywhere except at the middle where the difference contains the input noise. Parameter used are r = 16 and = 0.03 2 .

  GF σ {u}(x, y) -u(x, y) = σ 2 M 20 ˜ |∇u(x, y)| 2 + ˜ ∆u(x, y) + O(σ 3 ) . (.) Remark .. Theorem . means that the image edges are preserved when ˜ |∇u(x, y)| 2 , because σ 2 M 20 ˜ /|∇u(x, y)| 2 0.On the other hand, the filter is a diffusion by the isotropic heat equation when ˜ |∇u(p)| 2 . The transition between both behaviors is smooth, and a half-half compromise is observed when˜ = |∇u(p)| 2 . . if ˜ |∇u(x, y)| 2 , GF σ {u}(x, y) -u(x, y) = σ 2 M 20 ˜ |∇u(x, y)| 2 ∆u(x, y) + O(σ 3 ) ; . if ˜ = |∇u(x, y)| 2 , GF σ {u}(x, y) -u(x, y) = σ 2 M 20 2 ∆u(x, y) + O(σ 3 ) ; . if ˜ |∇u(x, y)| 2 ,GF σ {u}(x, y) -u(x, y) = σ 2 M 20 ∆u(x, y) + O(σ 3 ) .

  Figure 1.5: The iterated guided filter causes no staircase artifact. Parameters used here: = σ 2 r = 0.01 2 with the input dynamic range in [0; 1]; r = σs = 1 with the input image of size 250×250; number of iterations T = 50. The bottom graph displays the restrictions of the three above images to the vertical straight lines drawn on the images. The staircase effect of the bilateral filter (orange line) doesn't appear on the guided filter version (red line).

  Recently introduced by Aubry et al. in [APH + ], the unnormalized bilateral filter reads:UBF{u}(x) = u(x) + y∈Ω G σs (xy)G σr u(y) -u(x) u(y) -u(x) .(.)

  Figure1.6: Explanation of the staircase effect for a bilateral filter with simplified range and spatial kernels. The current pixel is at the intersection of the vertical and horizontal blue dotted lines. The dotted black rectangle indicates which pixels will be considered in the average. Light blue vertical arrows stand for the intensity difference between the current pixels and the pixels in the rectangle. Since the current pixel has more neighbors (in the bilateral definition) on the right side of the edge, its bilaterally averaged value gets up and closer to the plateau's value.

  Figure 1.7: Effect of the selective diffusion. Images are enhanced with DxO's contrast enhancement tool using the standard bilateral filter (b) or the bilateral filter with the selective diffusion (c). Most of the gradient reversal artifact has been removed thanks to the selective diffusion.

G

  The Durand-Dorsey fast approximation scheme is based on the discretization of the possible values of u(x) in the bilateral kernel. Consider the bilateral filter equation (.) for a fixed pixel xBF{u}(x) = 1 C(x) y∈Ω G σs (xy)G σr u(y) -u(x) u(y), (.)where C is the normalization factor. This is equivalent to the (x dependent) convolution of the function H u(x) : y → G σr u(y) -u(x) u(y) by the kernel G σs . Similarly, the normalization factor C is the convolution of I u(x) : y → G σr u(y) -u(x) by G σs . The only dependency on pixel x that makes it differ from a convolution. Starting from this observation, the authors' acceleration strategy is to discretize the set of possible signal intensities into N layers values {γ(i)}, and to compute a linear Gaussian convolution for each such value:v(x, i) = 1 C(x, i) y∈Ω G σs (xy)G σr u(y) -γ(i) u(y) σs (xy)G σr u(y) -γ(i)

Figure 1 . 8 :

 18 Figure 1.8: Illustration reproduced from [PD06]. Bilateral filter with the bilateral grid for a 1D signal. A first step is to fill the S × R domain with the signal values: the second line displays the resulting values Γ on the grid. The third line displays it after the convolution by the Gaussian kernel with standard deviation σs, σr. Then, the fourth line shows the result of the division of the two above grid values (the bilateral filter's normalization). The orange dots depict the pixel's positions. The last line is the reconstructed filtered signal, after the "slicing" operation.

Figure 1 . 9 :

 19 Figure1.9: Simulated exposure fusion method (SEF).

Figure

  Figure 1.10: We show here a section taken in the input sequence (represented on the images on the right column). All input images are in the correct dynamic range. The fused result however has a greater dynamic. The experiment is carried out with gray levels images for the sake of clarity; we thus do not use the saturation metric: ωs = 0. The other parameters are ωc = 1, ωe = 1. We clipped out-of-range values in (d).

  Figure 1.11: Comparison of SEF (b) with ACE (c) and MSR (d) (with chromaticity preservation, [PSM14]). The remapping functions and the corresponding parameters are specified in Figure 8.4 (c).

  Figure ., in the top right corner, the clipped remapping functions used for the output shown below, and compare it to two retinex methods: multi-scale retinex (MSR) and automatic color enhancement (ACE). Some other improvements are applied to the remapping functions: in function of the input image histogram, we generate more or less dark or bright images, so as to improve the contrast where needed only. Furthermore, using smooth remapping functions helps obtaining cleaner smooth edges.. Chapter : Local Laplacian filter and connection to other operatorsIn Chapter  and Chapter  we describe the exposure fusion method and the framework proposed to extend it to the single image case through the generation of a simulated bracketed exposure sequence. This fusion algorithm is based on the manipulation of Laplacian pyramids, and has demonstrated the usefulness of such a multi-scale image representation. We focus in Chapter  on the local Laplacian filters. They use the same Laplacian pyramid but in the context of multi-scale local contrast manipulation.The local Laplacian filters have originally been proposed in  by Paris, Hasinoff and Kautz[PHK]. A fast version was proposed the same year by Aubry, Paris, Hasinoff, Kautz and Durand [APH + ]. The initial conference papers were extended to journal papers in  for the Aubry et al. fast local Laplacian filters [APH + ] and in  for the Paris et al. original local Laplacian filters

  Figure 1.12: First row: original image (a), base layer (b) with the modified exposure fusion ( ẼF) and corresponding detail layer (c). Second row: base (d) and detail (e) layers obtained with the modified fast local Laplacian filters ( F LL). The range parameter used is σr = 25/255. More low-frequency halos are visible in the FLL output. Overall, the difference between both results is minor.

Figure 1

 1 Figure1.14: Test-pattern (light blue) and its filtered version by the bilateral filter (orange); the unnormalized bilateral filter (green); and the unnormalized oracle-based bilateral filter (red). The same parameters are used for all filters: σ l = 2 pixels and σr = .2. The oracle used in UOBF is Gσ l * u. It is drawn in dark blue. Compared to BF, UBF has a lighter smoothing effect. On the contrary, UOBF has a stronger smoothing than both BF and UBF. In fact, UOBF closely follows its oracle, except at the edge where most of the differences averaged have the same sign (positive at the top of the edge, negative at the bottom).

  Figure 1.15: First row: differences (×5) -Second row: Tone-Mapping: u T M = .125 + .750 * ESLF{u} + 5 × (u -ESLF{u})

  Figure 1.16: Decomposition of the luminance of (a) in base (b) plus detail (c) with the WLS filter. The parameters used are α = 1.2 and λ = 6.4. The compartmentalization artifact is clearly visible in the branches of the tree on the right hand side of the detail image.

  Figure 1.17: Detail layer (b) computed with MGF with parameters: r = 2, maximal number of scales possible, no iterations, and = 0.4 2 . In (c), the enhancement algorithm we use is simply enhance(u) = 0.125+0.750MGF{u}+ 3(u -MGF{u}). The input image dynamic range is in [0, 1], and we treat the luminance only.

Figure 1 . 1 :

 11 Figure 1.1: Tracé de la fonction sα utilisée dans ACE

  Figure 1.2: MSR et le filtre guidé multi échelles (MGF) introduit dans cette thèse. Retinex multi-échelles introduit un halo autour du phare, mais pas MGF. Le filtre guidé multi-échelle est un algorithme de décomposition en base et détail ; la décomposition obtenue pour la partie luminance de l'image d'entrée est affichée sur la première ligne, à gauche. Les deux algorithmes ne prennent en compte que la luminance pour l'amélioration de contraste. Noter que MSR ne parvient pas à préserver le contraste de la façade du phare, contrairement à MGF ; de plus, la décomposition de base et de détail donne beaucoup de flexibilité aux algorithmes, ce qui pourrait être utilisé par exemple pour augmenter encore le contraste local.

  Figure 1.4: Comparaison des filtres bilatéraux et guidés pour une mire qui contient un bord en marche d'escalier et une structure en dents de scie. Dans la ligne de gauche, l'équivalence utilisée pour les paramètres est = ( σr 2 ) 2 . Dans la ligne de droite, elle est = σ 2 r . Le paramètre spatial utilisé est r = σs = 3 et le paramètre d'intensité est σr = 50. Pour obtenir le même lissage de la structure oscillante avec les deux filtres (colone de droite), le filtre guidé préserve moins bien le contour.



Figure 1

 1 Figure 1.5: Le filtre guidé itéré ne provoque aucun artefact d'escalier. Paramètres utilisés ici : = σ 2 r = 0.01 2 avec la plage dynamique d'entrée dans [0; 1] ; r = σs = 1 avec l'image d'entrée de taille 250 × 250 ; nombre d'itérations T = 50. Le graphique du bas affiche une superposition des trois images au-dessus, pour les lignes verticales tracées sur les images. L'effet escalier du filtre bilatéral (ligne orange) n'apparaît pas sur la version du filtre guidé (ligne rouge).

  Figure 1.7: Effet de la diffusion sélective. Les images sont améliorées avec l'outil d'amélioration de contraste de DxO utilisant le filtre bilatéral standard (b) ou le filtre bilatéral avec la diffusion sélective (c). La majeure partie de l'artefact d'inversion de gradient a été supprimée grâce à la diffusion sélective.

Figure 1 . 8 :

 18 Figure 1.8: Illustration reproduite à partir de [PD06]. Filtre bilatéral avec la grille bilatérale pour un signal 1D. Une première étape consiste à remplir le domaine S × R avec les valeurs du signal : la deuxième ligne affiche les valeurs résultantes Γ sur la grille. La troisième ligne l'affiche après la convolution par le noyau gaussien avec l'écarttype σs, σr. Ensuite, la quatrième ligne montre le résultat de la division des deux valeurs des grille ci-dessus (la normalisation du filtre bilatéral). Les points orange représentent les positions des pixels. La dernière ligne est le signal filtré reconstruit, après l'opération de découpage (slicing).

Figure 2 . 1 :

 21 Figure 2.1: Guided filter principle and comparison with the bilateral filter. Figure reproduced from [HST13].

Figure

  Figure 2.2: Illustration of the first guided filter artifact: detail smoothing is reduced near strong edges. On the left: step-edge 1D-signal with a small noise (blue line) and its smoothed version with the guided filter (red line). On the right, we show the detail layer: difference between the two signals on the left: input -filtered (green line). The detail layer is almost flat in its center, where the input signal has its step-edge. For comparison, to input noise of the test-pattern (expected detail) is displayed below (blue line). The difference between these two signals is also presented (red bottom line), showing that the obtained detail almost perfectly equals the noise everywhere except at the middle where the difference contains the input noise. Parameter used are r = 16 and = 0.03 .

  Figure2.5: In each graph we superimpose the original line of a real image (blue line) with the results of the guided filter (red line) and the bilateral filter (green line). One can easily see the two artifacts of the guided filter: first, the amplitude of the edge in the guided filter results is always smaller than the original edge. This is the luminance halo artifact. Second, the structures are preserved around the edges, much more with the guided filter than with the bilateral filter, as seen in graph (d). This is the contrast halo artifact. The parameters used are r = σs = 6, 2 = σr = 30.

Figure 2

 2 Figure 2.6: A step edge with height h. The red frame delimits the window ω(y). Its radius is r = 6. The orange frame shows what other pixels are taken into account during the aggregation process.

  Figure2.7: We draw here the evolution of the coefficient a for v = u and a window centered on a step edge. We present four different curves, for four different values . The values of are chosen so that the coefficient a is exactly 0.5 for a specific edge height, using α = (hα/2) 2 . It follows that we know, for a specific α, that edges with a height inferior to hα will be smoothed, and edges with height superior to hα will be preserved. We used h1 = 10, h2 = 20, h3 = 40 and h4 = 80. Those values are shown on the figure with the vertical dashed lines.

  Figure2.7: We draw here the evolution of the coefficient a for v = u and a window centered on a step edge. We present four different curves, for four different values . The values of are chosen so that the coefficient a is exactly 0.5 for a specific edge height, using α = (hα/2) 2 . It follows that we know, for a specific α, that edges with a height inferior to hα will be smoothed, and edges with height superior to hα will be preserved. We used h1 = 10, h2 = 20, h3 = 40 and h4 = 80. Those values are shown on the figure with the vertical dashed lines.

  b

  Figure . and Figure ..

Figure 2 . 8 :

 28 Figure 2.8: Effect of the averaging of the linear coefficients.

Figure 2 . 9 :

 29 Figure2.9: Spatial kernels of the bilateral filter (red line) and the guided filter (blue line) in a smooth area. Parameters: σs = 6 (bilateral) and r = 6 (guided). The equivalence σs = r is good.

  Figure . shows that a cross-equivalence of parameters between the guided and the bilateral is hard to establish. In figure .(d), the smoothing is too strong. In figure .(c), it is not strong enough. Figure . shows the step edge case behavior of the filter.

  Figure ., along with figures . and ., clearly show the impossibility to set in the guided filter to ensure similar edge preserving properties for both filters. The filters are different, and this goal is unattainable. The setting proposed by the authors [HST] (right row in the Figure .)



  Figure 2.10: Comparison between the bilateral and the guided filter results for two different parameter settings: standard equivalence = σ 2 r and another equivalence = ( σ R2 ) 2 . The spatial parameter is set so that r = σs. We used here σr = 27 and σs = 3.

  Figure 2.11:Step edge preservation with the bilateral filter (blue line) and the guided filter (green line). The abscissa shows the edge height, and the ordinate its "amount of preservation". The closer its value to zero, the more preserved the edge is. For the bilateral filter, this preservation comes from the fact that pixels from the opposite side of the edge will not be used in the averaging. For the guided filter, edge preservation relies on the multiplicative coefficient a. Note that this result is shown before aggregation, after which a often gets smaller. On the left, the figure shows the results for the standard equivalence = σ 2 r . On the right, with the parameter equivalence = ( σr 2 ) 2 .

Figure 3 . 1 :

 31 Figure 3.1: Effect of the second derivative term when it is inverted. It creates shocks.

σ

  where the guide v can be different from the input u. Thus, even in the v = u configuration the input/output image will evolve with time but not the guide, thus avoiding the very sharp edge-stopping aspect of the first version (see Figure .).Algorithm : Iterated guided filter v (IGF

Figure 3 . 2 :

 32 Figure 3.2: The iterated guided filter causes no staircase artifact. Parameters used here: = σ 2 r = 0.01 2 with the input dynamic range in [0; 1]; r = σs = 1 with the input image of size 250×250; number of iterations T = 50. The bottom graph displays the restrictions of the three above images to the vertical straight lines drawn on the images. The staircase effect of the bilateral filter (orange line) doesn't appear on the guided filter version (red line).

σ

  ) needs six mean filters per iterations, whereas this third version (IGF (3) σ ) only needs two. So the filter is approximatively three times faster, as the mean filters are the most computationally demanding operations of the guided filter (see Algorithm ).  Definition .. IGF (3) σ : Fast approximation of the iterated guided filter with a constant guide. Compared to IGF (2)

Figure 3

 3 Figure 3.3: Different versions of the filter. Parameters are: = 0.01 2 (unless notified otherwise); r = 1 and T = 50. The last version IGF

Figure

  Figure . displays results obtained with all presented versions of the guided filter. The very left column (except from the top image which is the input) shows filtering and contrast enhancement results with the original guided filter. The contrast halo artifact is clearly visible in the zoomed-in part displayed in the bottom image. The next columns present the same results obtained with the iterated versions. The contrast halo artifact is solved. The detail layers produced by the three versions are rather different. As seen in Figure . the first version smooths more than the other two; this is particularly visible on the dark bars.The second iterated version keeps a small contrast halo related to window's width: We used here r = 4 and this remaining contrast halo would be smaller with a smaller r. One reason to choose r > 1 is that the larger r, the less iterations we need for a fixed final spatial smoothing. We therefore use the largest r for which the contrast halo is not objectionable. This value might nevertheless depend on the viewer and on the image resolution. We found that r = 3 or r = 4 are acceptable. The third (fast) iterated version stays close to the second one but presents a small luminance halo.Figure . presents another application to a gray scale image. The input and filtered images are displayed on the top row, and detail (input -filtered) is showed on the bottom row, with a contrast factor of  for visualization. The iterated guided filter solves the issues of the guided filter. Another example is given in figure ., where each channel of the input color image is filtered according to the luminance channel (= 0.2989×Red+0.5870×Green+0.1140×Blue). We compute the color coefficients before the filter using u color /u luminance and add them back after filtering. Thus, only luminance contrast is enhanced in this experiment. The texture of the table is better enhanced with the iterative version around.We show in Figure . the results for color filtering. Note that the guided filter with color is slower than the guided filter by a factor of almost . Indeed, the computational cost for using a color guide is slightly less than  times the cost of a gray guide, and one needs to filter each of the three channels of the color image. This is also valid for the iterated versions v and v, but



Figure 3

 3 Figure 3.4: Parameter used are = 0.06 2 for GF ; /T for IGF (1) σ

  Figure 3.5: The top line displays the input image (on the left) followed by the filtered versions. The bottom line displays the detail layers obtained (with a factor 6 for visualization). The iterated version of the guided filter gets rid of both luminance and contrast halos of the guided filter.

  Figure 3.7: Each filter can handle a color guide. IGF(1) σ is a special case here because it requires a filtered color image at each iteration, thus it filters each color channel at each pass, unlike the other ones that filter each color channel independentely (but still using the same color guide). Parameters are: GF = 0.06 2 and IGF (1) σ

Figure 4 . 1 :

 41 Figure 4.1: Bilateral filter principle. Figure reproduced from [TM98]

Algorithm  :

 : Standard O(r 2 N ) bilateral filter (BF) input : input u input : range standard deviation σ r input : spatial standard-deviation σ s input : window width r (usually 2σ s ) output: BF{u} // First loop on the whole image  foreach pixel x do // Second loop on the current pixel's window  foreach pixel y in the x-centered window of size (2r + 1) 2

  Figure4.2: Explanation of the staircase effect for a bilateral filter with simplified range and spatial kernels. The current pixel is at the intersection of the vertical and horizontal blue dotted lines. The dotted black rectangle indicates which pixels will be considered in the average. Light blue vertical arrows stand for the intensity difference between the current pixels and the pixels in the rectangle. Since the current pixel has more neighbors (in the bilateral definition) on the right side of the edge, its bilaterally averaged value will be closer to the plateau's value.

Figure 4 . 3 :

 43 Figure 4.3: The number of neighbors is unbalanced for concave signals. This causes the staircase artifact. Figure reproduced from [BCM06].

Figure 4 . 5 :

 45 Figure 4.5: Huber minimax norm ρ(.), its derivative ψ(.) and the corresponding edge-stopping function g(.). This norm is a modification of the L1 norm with a quadratic part around zero. See Equation (.), Equation (.) and Equation (.), respectively. Figure reproduced from [BSMH98].

Figure 4 . 6 :

 46 Figure 4.6: Figure reproduced from [DD02]. The ACE method uses the Huber minimax influence function, while the classic bilateral filter uses a Gaussian. In ACE the σ parameter is 1/α.

  g

  Figure . (reproduced from [DD]) the plot of different influence functions considered in Durand et al. paper. The Gauss function from the bilateral filter (green line) and the Huber one (red line) show the different treatment of outliers made by BF and ACE.



  Figure 4.7: Contrast enhancement with ACE for two different spatial kernels: Gaussian (b) and 1/ x (c). The contrast factor used here is α = 5. These results are taken from the IPOL archive  of Getreuer's ACE implementation [Get12].

  Algorithm : Unnormalized Bilateral Filter (UBF) input : image u input : spatial parameter σ s input : range parameter σ r input : radius of the window r output: filtered image UBF{u}  foreach pixel x do  foreach pixel y in the x-centered window Ω of size (2r + 1) 2 do // Compute bilateral weights for current pixel 

  Figure 4.8: Filtering with the bilateral filter and the unnormalized bilateral filter. We show here the profile of a testpattern (abscissa for pixel position; ordinate for pixel intensity). Parameters: σs = 16 and σr = 0.20. Compared to BF, UBF reduces the staircasing artefact (a) but smoothes less (b).

  Figure 4.9: First row, displays in (a) the input, then the filtered images. The middle row displays the detail layer, with a amplification factor of 6 for visibility purposes. The bottom row shows an example of contrast enhancement: enhanced = input + 5 × detail. Parameter σs = 16 and image size is 400×400. For this image, while UBF succeeds in removing the edge-sharpening effect of BF (visible along the bars), it looses the ability to filter inside thin elements, because the number of similar pixels is too small.

Algorithm  :

 : Bilateral filter with regression (BFR). (Exact) input : image u input : spatial standard-deviation σ s input : range standard-deviation σ r output: BFR{u} the filtered image  foreach pixel x = (x, y) do  Compute bilateral filter weights k = G σs (xy)G σr (u(x) -u(y))  Compute ρ x 2 , ρ y 2 , ρ xy , ρ x , ρ y and ρ in x  Compute xku, yku and ku in x  Find coefficients (a, b, c) at x by solving the linear system of equations in (.)  Give to the output the value of c: BFR{u}(x) = c A pseudo-code of the standard O(r 2 N ) implementation of the bilateral filter with regression is given in algorithm .

Algorithm

  : Fast bilateral filter with regression (BF R fast ) input : image u input : spatial standard-deviation σ s input : range standard-deviation σ r input : number of layers N output: BF R fast {u}  foreach layer with itensity s do  Compute a layer of the image at s: layer(u, s) = uG σr (u -s)  Compute ρ x 2 , ρ y 2 , ρ xy , ρ x , ρ y and ρ: convolve the layer with  different kernels  Compute xwu, ywu and wu ( more convolutions of the layer)  And update the nine images computed at the previous layer using linear interpolation.  foreach pixel x do  Find coefficients (a, b, c) at x by solving the linear system of equations in (.)  Give to the output the value of c: BFR fast {u}(x) = c

  Figure 4.10: Results obtained with the multi-scale bilateral filter with regression (MBR). Parameters: σs = 1, σr = 0.05, r = 2, lmax = 5. Concerning the exact bilateral filter with regression, all the parameters are equal except for σs = 2 5 = 32, which is equivalent to the spatial support of MBR. It is clear from the comparison of the detail layers that MBR can both filter with a large spatial support and remove the staircase effect. The multi-scale filter is also faster to compute because the kernel used at each scale is drastically smaller (only 5 × 5 pixels in this example).

  Figure 4.11: "Dark-halo" artifact in the multi-scale bilateral filter with regression. This distortion is due to the suppression of the (necessary) ringing in the Laplacian pyramid. The enhancement algorithm we use is simply enhance(u) = 0.125 + 0.750 MBR{u} + 3(u -MBR{u}). The input image dynamic range is in [0, 1].

Figure 5

 5 Figure 5.1: Parameters are σs = 20, σr = 0.02 (image dynamic in [0;1]) and σcorr = 2. The dynamic in figure (d) is [0;1] also. The colormap goes from dark blue (zero) to dark red (one) through green and yellow.

Figure 5

 5 Figure 5.2: Parameters are σs = 20, σr = 0.04 (image dynamic in [0;1]) and σcorr = 2. The dynamic in figure (d) is [0;1] also. The colormap goes from dark blue (zero) to dark red (one) through green and yellow.

  otherwise, (.) and similarly for the component y v . The corrected detail layer is obtained by solving the corresponding Poisson equation. We reproduce in Figure . the illustration given by the authors, showing the correction of the gradient reversal artifacts in the output images.

Figure 5

 5 Figure 5.3: Figure reproduced from [BPD06]. From left to right: input image, image enhanced without correction, image enhanced with the Poisson correction.

  Figure 5.4: Effect of the selective diffusion. Images are enhanced with DxO's contrast enhancement tool using the standard bilateral filter (b) or the bilateral filter with the selective diffusion (c). Most of the gradient reversal artifact has been removed thanks to the selective diffusion.

Figure

  Figure . displays the result of the selective diffusion applied to the bilateral filter, in the context of contrast enhancement. It succeeds in removing a large part of the gradient reversal artifact (a consequence of the staircase effect) visible as a dark and white bands along the top of the trees.Although this method works globally well, it seems unable to remove the staircases everywhere, especially in the corners (see Figure .(c)). Furthermore, it is not computationally efficient. Indeed, numerous iterations are needed to correct the staircase effect, and this computation time adds to the computation time of the filter itself. Algorithm  describes the pseudo-code of this method.

  Figure 5.5: Comparison of the symmetric bilateral filter kernel (b) with the original one (c) (both normalized), for the position indicated by a red square in the input image (a).The symmetric bilateral kernel still adapts well to the image content. Thanks to its symmetry, the staircase effect is avoided. On the other hand, the number of pixels used in the averaging process is systematically less than or equal to the number of pixels used in the original bilateral filter, thus reducing its denoising property.

  Figure 5.6: Comparison of the symmetric bilateral filter with the original one (with an exact implementation), for two images. Column (a) displays the input images, column (b) the results (base and detail layers) obtained with the symmetric bilateral filter and column (c) displays the results (base and detail layers) of the original bilateral filter. The detail layers are multiplied by a factor 6 for visualization purposes. The parameters used for filtering are: σr = 0.1 (dynamic range in [0; 1]), σs = 12 (images size is 330×330 for top one and 250×250 for the bottom one).Comparing the detail layers of the top image (2nd row), it appears that the symmetric bilateral filter removes the staircase effect. This effect can be observed in the original bilateral filter detail layer as alternating dark and bright lines along the vertical black column. The bottom image confirms that the filtering in a more general case is not altered by the symmetrization. The edge-preserving property is preserved.

Figure 5

 5 Figure 5.8: (reproduced from [CT05]) Filter extent for one scan-line of an image.

Figure 5

 5 Figure 5.9: (reproduced from [CT05]) Difficult image features: (1) Ridge-like and valley-like edges, (2) high-gradient regions, (3) similar intensities in disjoint regions.

  Figure 6.1: Filtering with the bilateral filter and the separable bilateral filter (SBF) (algo. 16). The parameters are σr = .08 (dynamic in [0; 1]), σs = 10 and r = 25. The SBF result is not really clean: one can see in (d) some vertical lines created by the second pass of SBF. On the other hand, the strong edges are still well preserved (see the dark object in (d) and (c)).

Figure 6

 6 Figure 6.2: (Reproduced from [PVV05]) Scheme for the separable kernel bilateral filter.

  () (FMF) input : image u input : window radiuts r (width = 2r + 1) output: FMF fast {u}  c ← 0, r ← 0 // Initialize columns and rows  d ← +1 // Initialize direction: first row is processed from top to bottom  h Ω ← compute histogram of u for columns c -r to c + r and lines l -r to l + r  foreach column c do  foreach lines l do // Update histogram (shift of one row)  Add values of u for line l + d(r + 1) and columns c -r to c + r to h Ω  Subtract values of u for line l -d(r + 1) and columns c -r to c + r from h Ω  R fmf (l, c) ← find median value of h Ω // Retrieve median value  l ← l + d // Update row r in the right direction // Update histogram (shift of one column to the right)  Add values of u for column c + r + 1 and lines l -r to l + r to h Ω  Subtract values of u for column c -r -1 and lines l -r to l + r from h Ω  d ← -d // Update direction retrieve the median value (line ): it lies in the first index for which the sum of values to that index reaches 2r 2 + 2r + 1.

Figure 6 . 3 :

 63 Figure 6.3: Figure reproduced from Weiss' paper [Wei06]. This figure shows a layout for processing sixty-three columns at once. It is the three-tiered analogue of Figure 4, this time "viewed" from the side. There is a single shared histogram P31 [yellow] corresponding to the central window; eight partial histograms [orange] at sevenpixel intervals; and for each of these, six small partial histograms [red] at unit intervals; sixty-three histograms altogether. Each input pixel is added/subtracted to each histogram intersecting its column.

  y)) (.) with the initial condition H(0, 0, b) = 0. Hence the integral histogram at a pixel (x, y) is obtained first by copying the histogram value of the previous pixel, then by the propagation operation shown in equation ., i.e. with three arithmetic operations per bin and per pixel.The pseudo-code given in Algorithm  describes this fast bilateral filter. First comes the computation of the integral histogram (line ), then for each pixel, the extraction of the local histogram (line ) with the desired radius. Finally, it computes at each pixel the bilateral filtered value using Equation (.) (line ).

Algorithm

  : Piecewise-linear fast bilateral filter (FBF piecewise ) for a D gray image input : image u input : range standard deviation σ r input : spatial standard deviation σ s input : Number of layers N layers output: FBF piecewise {u}  FBF piecewise ← 0 // initialization  a = (max u -min u)/N layers // gap between two intensity samples  b = min u  foreach sampled intensity γ(i) with i ∈ {0, 1, . . . , N layers } do  γ(i) ← ai + b // current intensity sample  H(i) ← G σr u -γ(i) u // compute layers H and

Figure 6 . 4 :

 64 Figure 6.4: Illustration reproduced from [PD06]. Bilateral filter with the bilateral grid for a 1D signal. A first step is to fill the S × R domain with the signal values: the second line displays the resulting values Γ on the grid. The third line displays it after the convolution by the Gaussian kernel with standard deviation σs, σr. Then, the fourth line shows the result of the division of the two above grid values (the bilateral filter's normalization). The orange dots depict the pixel's positions. The last line is the reconstructed filtered signal, after the "slicing" operation.

Figure 6 . 5 :

 65 Figure 6.5: This figure is reproduced from the Paris-Durand paper [PD06]. In the piecewise-linear approximation (a), the downsampling is realized before the layering step. In this configuration, discontinuities are represented by only one intensity value which poorly approximates them. On the other hand, in the bilateral grid scheme (b), the discontinuities are represented by two distinct values in the downsampled S×R domain, even after downsampling. The original function (in red) is the same as in Figure 6.4. The corresponding downsampled representation of the intensity is shown under (a) or behind (b).

GG

  σs (y)u(xy) exp inωu(xy) (.)  and with a normalization factor (denominator) σs (y) exp inωu(xy) .

Figure 6 . 6 :

 66 Figure 6.6: Figure from [NPC17]. The visual artifacts in (c) are cluttered around sharp edges in the original image.This can be explained by the fact that the Fourier approximation is relatively poor (often assuming negative values) on the tails compared to that around the origin. Since the operating region for large pixel differences is precisely the tail, this can result in artifacts around edges.

DT{u}

Figure 6 . 7 :

 67 Figure 6.7: Figure reproduced from the original paper [GO11]. Curve C defined by the graph x, u(x) , x ∈ Ω (left). In l1 norm, x + h, u(x + h) -x, u(x) = h + d = h + |u(x + h) -u(x)| (center). Arc length of C, from u to w (right).

Figure 6 . 8 :

 68 Figure 6.8: Figure reproduced from original paper [GO11]. 1D edge-preserving using ct(z) (noted ct(u) int the plot). (a) Input signal u. (b) ct(z). (c) Signal u plotted in the transformed domain (Ωw). Signal u filtered in Ωw with a 1D Gaussian (d) and plotted in Ω (e).

  Figure . illustrates the use of a domain transform for filtering the D signal I, shown in (a) in its original domain Ω. (b) shows the associated domain transform ct(z) computed using Equation (.). (c) shows signal u in the transformed domain Ω w or, more compactly, u w ct(z) = u(z). The result of filtering u with a Gaussian filter H in Ω w is shown in (d). (e) shows the desired filtered signal obtained by reversing

Figure 6 . 9 :

 69 Figure6.9: Indices for the interpolated convolution described in Algorithm 28

/

  Figure 6.10: As well as the bilateral filter, the domain transform has a staircase effect. This creates gradient reversal when used for contrast enhancement. Figure reproduced from [HST13].

  Figure 6.11: Base+detail decomposition with the domain transform (center column) and the exact bilateral filter (right column). Detail is amplified by a factor 3 for visualization. Enhanced images are obtained with 0.125 + 0.750 × base + 3 × detail. The separation is different with the two filters. For this image with a lot of gradients, the domain transform produces a detail of lower amplitude. On the other hand, the staircase effect (sharpening of strong edges) in domain transform is smaller than in the bilateral filter result. Parameters are: σs = 8 (image size is 280×420 pixels) and σr = 0, 125 (image dynamic in [0; 1]). For the domain transform, the number of iterations for the two-passes 1D filter is N = 3, and the recursive strategy is used.

Algorithm

  : Exposure Fusion input : input sequence of images I input : ω s , ω c , ω e : weights for saturation, contrast and well-exposedness measures, respectively output: fused image R  foreach image at position k ∈ {1, 2, ..., N } in the input sequence do  Compute contrast metric C using eq. (.)  Compute saturation metric S using eq. (.)  Compute well-exposedness metric E using eq. (.)  Compute weight map W k of the current image using eq. (.)  Normalize weights using eq. (.)  foreach image at position k ∈ {1, 2, ..., N } in the input sequence do  Compute Gaussian pyramid of weights G{ W } k  Compute Laplacian pyramid of input images L{I} k  foreach coefficient at position ij and scale l do  Update Laplacian pyramid of the output image:

Figure 7 . 1 :

 71 Figure 7.1: Simulated exposure fusion method (SEF).

Figure 7 . 2 :

 72 Figure 7.2: Approximation of the image capture process

Figure 7 . 3 :

 73 Figure 7.3: Remapping functions used to generate the input "bracketed" sequence, here with parameter α = 6 and N = 4.

  Figure 7.4: Tone-Mapping with the "Simulated Exposure Fusion": original (top left) and tone-mapped (top right) with the proposed method. The remapping functions displayed in Figure 7.3 were used: 5 images in the sequence (including the input one) and maximal contrast factor α = 6. Comparison with Multiscale Retinex on the intensity channel [PSM14] (bottom left), with 0.1% of saturation in both black and white values for the final "Simplest Color Balance". Comparison with Automatic Color Enhancement (ACE) [Get12] (bottom left) with parameter α = 8 (maximal authorized contrast enhancement). The generated images of our method and their corresponding weights are displayed in Figure 7.5.

  Saturation occurs in the original methods by Mertens et al., as shown in Figure . and Figure ..

Figure 7

 7 Figure 7.6: Input sequence (top row) and the corresponding normalized weights maps (bottom row). The default parameters were used for this experiment: ωc = 1, ωs = 1, ωe = 1. (Images courtesy of Min H. Kim.)

  Figure .(a), (b). We display the center line in the plot of the same figure. The fused image's (yellow line) edges height is the average of the two input heights. If the same or another image of the sequence has large edges in the reverse direction, then the fused image can overstep the input dynamic range: see Figure ..

Figure 7 . 9 :

 79 Figure7.9: Laplacian coefficients for a line of the input sequence and the fused result (same line as in Figure7.8). The Laplacian coefficients are displayed by scale, from the finest one (top row) and in descending order of fineness towards the bottom. The fused Laplacian coefficients (dark green) are a weighted combination of the input Laplacian coefficients (the weights are not showed here). The fused Laplacian coefficients often follow the coefficients of the input image that has the greatest amplitude. As a result, the fused image combines the greatest variations (and edges!) of each different input image, which explains its increased dynamic range.

  Figure7.10: Fusion of input 1 and input 2 with exposure fusion. Parameters: ωc = 1, ωs = 0 (gray level images), ωe = 1. This simple experiment shows that exposure fusion cannot reduce edge amplitude at will. In fact, edge reduction is a consequence of the blending of large Laplacian coefficients (from input 1) with smaller Laplacian coefficients (from input 2). In this experiment, this is not enough to prevent a saturation of the fused result (c). Figure7.11 displays a more complex case where three input images are fused.

Figure 7

 7 Figure7.11: Edge preservation in exposure fusion and dynamic extension. In this experiment, the input sequence has three images and two contrasted bands: input 1 holds the "well-exposed" first band (saturated in the other images); input 3 holds the well-exposed second band (saturated in the other images); input 2 holds the wellexposed parts between the bands. By blending the well-exposed parts together, exposure fusion creates an image too contrasted to fit in the input dynamic range.

  Figure8.1: Modified remapping functions aimed at reducing the dynamic of the input images. So as to preserve important information, the saturation must be done wisely. In particular, we keep values that are the most brightened for the far left part of the histogram, values that are the most darkened for the far right part of the histogram, and proceed gradually for values in-between. Here, the number of generated images is 4 (fifth image is the input one); the maximal contrast factor is 4 and the "allowed" dynamic range of the input image is β = 2/3 of the final dynamic range.

  .) An illustration of these clipped remapping functions is given in Figure ..  In Figure . we show the result of single-image exposure fusion when the clipped remapping functions are used and compare it to the result without clipping. We used a similar test-pattern than the one used in Figure ., of which we display the central line. This original line appears in Figure . (a) and (b) as the orange line in the bottom plot. The fused result (dark blue) is superimposed. The same lines taken in the generated images are displayed in the top plots of (a) and (b). The couple of plots in (a) uses the remapping functions of Figure . (a), which produce a fused result with extended dynamic range. On the other hand, the couple of plots in (b) uses the remapping functions of Figure . (b) and does not create out-of-range values.

u

  lum = 0.2989 × u red + 0.5870 × u green + 0.1140 × u blue , (.) then enhance the luminance only, and finally reintroduce the color coefficients c chan = u chan /(u lum + ), modification, we can no longer use the color saturation parameter of the Mertens et al. method. Although this trick gives slightly less vivid results, it successfully resolves the color alteration artifact previously introduced. The result using luminance only is displayed in Figure . (c).

  Figure 8.2: The used test-patterns are similar to the one used in Figure 7.10. Input sequences are generated using remapping fonctions displayed in Figure 8.1 (a) (top) and Figure 8.1 (b) (bottom), that is, the bottom couple of plots use the remapping function with a reduced dynamic. Comparing top and bottom results: out-of-range values are almost completely removed.

  Figure 8.4: The original image in (a) is enhanced using a symmetric sequence (b) and an asymmetric one (c). The asymmetric sequence better enhances contrast in originally dark regions while having a limited effect on the columns. It produces a globally better exposed output (with value around .5). Parameters used for sequence generation: α = 8, N = 6 and β = 1/2. Exposure fusion parameters: ωc = 1, ωs = 0, ωe = 1. The luminance only was processed.

  Figure ., for the bushes and the part of the front house behind the columns particularly. The third row in Figure . displays a result obtained with the fast local Laplacian filter (fast LLF) [APH + , APH + ].

  Figure 8.7: Comparison of SEF (b) with ACE (c) and MSR (d) (with chromaticity preservation, [PSM14]).The remapping functions and the corresponding parameters are specified in Figure 8.4 (c).

  Figure 8.9: Comparison of SEF (a), (b), (c) with ACE (e) and MSR (f ) (with chromaticity preservation, [PSM14]). The remapping functions used in each SEF result are displayed on the corresponding images. All SEF results use α = 8. Image (a) uses β = 1/3 and Ntot = 2N + 1 = 7 with the smooth clipping; image (b) uses β = 1/2, Ntot = 2N + 1 = 4 with smooth clipping too. Image (c) displays results obtained in Chapter 7, except that we applied the method to the luminance only for a fairer comparison with the other results -so (c) uses hard clipping in [0, 1], and sequence generation is symmetric. The respective remapping functions are displayed above the SEF results. The third row displays results obtained with fast LLF. Images (g) and (h) are a direct outputs of fast LLF whereas image (i) uses LLF ability to decompose the input in base+detail and Durand and Dorsey's tone-mapping [DD02].

  Figure 8.11: Comparison of the standard result (b) obtained with exposure fusion [MKR07, MKVR09] with the result (a) obtained with an "extended" sequence (shown in Figure 8.10).A final normalization step has been performed for both images. We used the "Simplest Color Balance" algorithm [LLM + 11] which allows a small percentage of clipping for both white and black values. We fixed this percentage to .5%, so that a maximum of 1% of the pixels values (a color pixel counts 3 values) is clipped in the displayed results. Our result (a) has more contrast and saturation than for theMertens et al. output (b).

  Figure . we show the extended sequence generated from a four-images sequence (displayed in Figure .): we generated three images for each input of the sequence with a reduced dynamic range. The remapping functions are displayed at the bottom left corner of the figure.

  Figure 8.12: Second example  of application of the generative functions to a real bracketed exposure sequence. The three input images are displayed on the top row. The bottom row shows the result of simulated exposure fusion (SEF) for two sets of parameters, in (a) and (b) (the remapping function are displayed above the corresponding result). The result obtained with the original method of Mertens et al. is displayed in (c). Each output image has been normalized using the method described in Figure 8.11. The parameters used in (a) are α = √ 2, N = 2 and β = 1/2; in (b) α = 1,N = 2 and β = 1/3. The exposure fusion parameters are the same for the three images: ωc = ωs = ωe = 1. The SEF results have more contrast than with EF.

Figure 9 . 1 :

 91 Figure 9.1: Simplistic illustration of the principle used in LLF: on the top line, a one-dimensional signal with an edge, and below two scales of its (rescaled) Laplacian pyramid. On the top left is the original image and in the top middle are the two remapped versions. Below them, two scales of their (rescaled) Laplacian pyramid (middle and bottom rows). The right hand side of the illustration is obtained by merging the "non-clipped" parts of the Laplacian coefficients of the remapped signals (middle columns), and the final output (top right) after collapsing the Laplacian pyramid. Figure reproduced from [PHK15].



  Algorithm : Local Laplacian filters (LLF). O(N 2 ) algorithm. input : u the input image input : r the remapping function output: LLF{u}  Compute the Gaussian pyramid Gpyr{u} of u  Create an empty Laplacian pyramid Lpyr{ũ} // initialization  foreach pixel x and scale l do  g = Gpyr{u, l}(x)  u = r(u, g) // remap the input image in function of g  Compute the Laplacian pyramid Lpyr{u } of u  Lpyr{ũ, l}(x) ← Lpyr{u , l}(x) // update output Laplacian pyramid  LLF{u} ← Collapse Lpyr{ũ}

Figure 9 . 2 :

 92 Figure 9.2: Figure and legend reproduced from [PHK15]. Family of point-wise functions for edge-aware manipulation (. . . ). The parameters α and β control how detail and tone are processed respectively. To compute a given Laplacian coefficient in the output, the original image is filtered point-wise using a nonlinear function r(t) of the form shown. This remapping function is parameterized by the Gaussian pyramid coefficient g, describing the local image content, and a threshold σr used to distinguish fine details (red) from larger edges (blue).

Figure 9 . 3 :

 93 Figure 9.3: Figure and legend reproduced from [PHK15]. "Overview of the basic idea of our approach. For each pixel in the Gaussian pyramid of the input (red dot), we look up its value g. Based on g, we remap the input image using a point-wise function, build a Laplacian pyramid from this intermediate result, then copy the appropriate pixel into the output Laplacian pyramid. This process is repeated for each pixel over all scales until the output pyramid is filled, which is then collapsed to give the final result. For more efficient computation, only parts of the intermediate pyramid need to be generated."

Algorithm

  : Fast Local Laplacian filters (FLL) input : image u input : remapping function r input : number of slices S output: image FLL{u}  Compute the Gaussian pyramid Gpyr{u} of the input u  Regularly sample the intensity range with S values {γ 1 , γ 2 , γ 3 , ..., γ S }  foreach intensity sample γ i do  u i = r(u, γ i ) // remap u in function of γi  Compute Laplacian pyramid Lpyr{u i }  foreach pyramid coefficient at position x and scale l do  g ← Gpyr{u, l}(x) // same coefficient but in the Gaussian pyr.  Find i such that γ i and γ i+1 are the closest intensity samples from g  Compute a such that g = (1 -a)γ i + aγ i+1 // Linearly interpolate the output Laplacian coefficient from the precomputed pyramids  Lpyr{ũ, l}(x) ← (1 -a)Lpyr{u i , l}(x) + aLpyr{u i+1 , l}(x)  FLL{u} ← Collapse Lpyr{ũ} In Aubry et al. [APH + , APH + ], the proposed remapping function is different from Paris et al. [PHK, PHK]. They use a Gaussian-based one, thus closer to the bilateral filter:

  Figure 9.4: First row: original image (a), base layer (b) with the modified exposure fusion ( ẼF) and corresponding detail layer (c). Second row: base (d) and detail (e) layers obtained with the modified fast local Laplacian filters ( F LL). The range parameter used is σr = 25/255. More low-frequency halos are visible in the FLL output. Overall, the difference between both results is minor.

  Figure 9.5: Loss of translation invariance with LLF. The (a) remapping function preserves the local contrast but reduces the edges amplitude. Test-pattern #1: The spurious bounce didn't disappear. In illustration (b), I the edge was shifted by one pixel on the right, and the position of the bounce changed.

  Figure 9.6: Each figure above is the difference between the reference and the filtered version of the test-pattern, amplified with a factor 4 for visualization. Diff = .5 + 4 * (I ref -I filtered ). We recall the results of the Bilateral Filter (BF) and the Bilateral Filter with regression in the first row. In the second row, we show the results for our two Local Laplacian Filters. Parameters: σr = 0.20; nb scales = 6 (max σs = 16); nb samples = 64 (and last scale is not processed).Test-pattern parameters: standard-deviations for blurring go from 0 to 30 pixels.

  Figure 9.7: Some filtering results by SLF and ESLF. The first line shows our result for edge-reduction. The remapping function r1 is used to get the oracle for the second filtering step with ESLF, which uses the remapping function r2.The bottom row is our result for detail-reduction. In our experiment, there is no detail, the filter is then supposed to let the input image unmodified. ESLF is plotted in orange, while SLF appears in red. Our modification succeeds in removing artifacts we observed before (see Sectionı 9.4).

  update output Laplacian scale-space  Obtain SLF{u} by collapsing Laplacian scale-space Lss{ũ} // Equation 10.4 Complexity The complexity of the fast Fourier transform (FFT) is O N log(N ) , with N the number of pixels of the image. Thus, the complexity of the algorithm used to compute the Gaussian Scale-Space representation of the image (see Algorithm ) is O N log(N )l max , because  Algorithm : Fast scale-space local Laplacian filters (fast SLF) input : image u input : remapping function r input : number S of intensity samples output: filtered image SLF{u}  Compute Gaussian scale-space Gss{u} of input image u  Sample the intensity range between min(u) and max(u) with S values {γ

  Figure10.2: Test-pattern (light blue) and its filtered version by the bilateral filter (orange); the unnormalized bilateral filter (green); and the unnormalized oracle-base bilateral filter (red). The same parameters are used for all filters: σ l = 2 pixels and σr = .2. The oracle used in UOBF is Gσ l * u. It is drawn in dark blue. Compared to BF, UBF has a lighter smoothing effect. On the contrary, UOBF has a stronger smoothing than both BF and UBF. In fact, UOBF closely follows its oracle, except at the edge where most of the differences averaged have the same sign (positive at the top of the edge, negative at the bottom).

Figure 10 . 3 :

 103 Figure 10.3: Four scales of the multiscale bilateral decomposition on a 1D row of pixel intensities. Black lines indicate pixels at scale j -1 that are used by the fast algorithm to compute the value of the pixel in column 3 at scale j.In contrast, the basic algorithm also considers all the pixels marked with green lines. At j = 3 the basic algorithm averages in the gray pixel in column 6, but the fast algorithm never sees a contribution from that pixel. Illustration and caption reproduced from [FAR07].

  Figure 10.4: Progressively coarsening a signal using different edge-preserving schemes. The coarsened versions are shown superimposed on the signal (using different shades of blue: lighter is coarser). The corresponding detail signals are plotted in shades of red below. Figure and caption reproduced from [FFLS08].

Figure 10 . 5 :

 105 Figure10.5: The first column shows the pyramid obtained by filtering at different scales the input image. The second column shows the difference between two consecutive levels of the pyramid (for the pyramid on the same row), except for the last row concerning UOBF. For this filter indeed, an oracle is used in the filtering, and this oracle depends on the scale. The Laplacian coefficients in the bottom right plot are obtained by the difference between two consecutive levels of a pyramid that uses the same oracle for those levels, whereas in the bottom left a different oracle is used at each scale. The oracle used is g(l) = Gσ l * u, i.e. the Gaussian scale-space displayed in the top left plot. This is coherent with Equation (.). Thus, the bottom right plot displays the detail layer's Laplacian coefficients of UOBF. The parameter σr remains unchanged over the scales. Unlike with BF or UBF, the Laplacian coefficients obtained with UOBF effectively contains only low-scale variations in the low levels; moreover, they do not contain large oscillations due to edges like with the Gaussian filter.

  We show in Figure . and Figure . results with our test-pattern, with SLF used as oracle. It allows to improve the behavior of the filter both for the dynamic reduction case (top line in Figure . and Figure .) and detail smoothing case (bottom line in Figure .).

  Figure10.7: The first line shows our result for edge-reduction. The remapping function r1 is used to get the oracle for the second filtering step with ESLF, which uses the remapping function r2. The bottom row is our result for detail-reduction. In our experiment, there is no detail, the filter is then supposed to let the input image unmodified. ESLF is plotted in orange, while SLF appears in red. Our modification succeeds in removing artifacts we observed before. (see Section 9.4)

  Figure 10.9: Evolution of the Guide (SSLLF) and of the result in function of the parameter σr of the remapping function r1.

  Figure . and Figure . are:



  Figure10.10: Pictures "Obelisk" and "Pantheon" and the line (in orange) of the image that is used in the 1D graphs.

Algorithm  :

 : Weighted least squares filter (WLS) input : image u input : luminance (guide) image input : parameters λ (amount of smoothing) and α (edge sensitivity) output: image v  ← 0.001 // small number to prevent divisions by zero // Compute affinities between adjacent pixels based on gradients of  for each pixel x except last row do  d y (x) ← (x, y) -(x, y + 1) // forward difference in y  a y (x) ← (|d y (x)| α + ) -1 // smoothness coefficient in y  Fill last row of a y with zeros  for each pixel x except last column do  d x (x) ← (x, y) -(x + 1, y) // forward difference in x  a x (x) ← (|d x (x)| α + ) -1 // smoothness coefficient in x  Fill last column of a x with zeros // Construct a five-point spatially inhomogeneous Laplacian matrix  Construct the sparse matrix A ← (I + λL u ) // see Equation (.) for Lu  Solve the system Au = v // u, v are vectorized version of images u, v

  Figure11.1: Multi-scale decomposition using WLS and the first multi-scale decomposition method (see Equation 11.5). The parameter used are those given by the authors for the same decomposition with the easter island statues picture. That is, λ = 0.1, λ = 0.8, λ = 6.4 for the iterations 1, 2 and 3, respectively. The gradient sensitivity parameter is kept constant over the iterations with the value α = 1.2.

  Figure 11.4: Compartmentalization artifact in WLS. Decomposition of the luminance of (a) in base (b) plus detail (c) with the WLS filter. The parameters used are α = 1.2 and λ = 6.4. The compartmentalization artifact is clearly visible in the branches of the tree on the right hand side of the detail image.

  Figure ., this compartmentalization occurs with the sky fragments between tree's branches, that become brighter than the rest of the sky. This phenomenon was already at work in the Figure . and Figure ., on the right, between the tree trunk and the building, but also, at a smaller scale, at places where the sky is cut by the street lights. The halo artifact The second artifact is an asymmetric luminance halo around the edges. It is shown in Figure . and Figure .. It occurs in this filter because the coefficients a x (x) and a y (x)

  Figure 11.5: Asymmetric lumiance halo in WLS (the thin black halo). Original color image (a) and luminance input image (b) that is filtered with WLS with two sets of parameters. The detail layers are shown in (c) and (d).The values of α are those used by the authors for method 1 (α = 1.2) and for method 2 (α = 1.8). The parameters λ were chosen so that the PSNR of both images is the same. This is a way to compare the "quantity of detail" extracted by the filter. The detail layers clearly display a thin black halo. The section shown in (a) (red line) is displayed, for the base layers, in (e), and for the detail layers in (f ). The detail lines on the right show the thin but high amplitude dark halo (peaks toward the bottom) and large yet smaller amplitude bright halo (peaks towards the top).

Figure

  Figure 11.6: The only difference with Figure 11.5 is the replacement of log( ) by (i.e. the removal of the logarithm).This makes the asymmetry in the halo to disappear.

  Figure 11.7: On the left: original and filtered (base) image; on the right: detail layer. Parameters are α = 1 and λ = .5.

  Figure11.8: Experiment with a test-pattern (a) made of a step-edge (centered in the [0, 1] dynamic range) with some Gaussian noise. We display contrast-enhanced results in (b) and (c), obtained using u = u + 2(u -WLS{u}) whith the parameters α = 1 and λ = 10. The smoothness coefficients are obtained in (b) with = log(lum) where lum is the input (gray-scale) image, and in (c) with = log(1 -lum). Thus the (asymmetric) halo is reverted between (b) and (c). This allows to see that a thin-but-high-amplitude white halo is more visible than a black one. The bottom plot displays superimposed sections of the two different detail layers.

Threshold

  Figure11.9: Detail layers obtained with the original WLS filter (a) and parameters α = 1.2 and λ = 0.5, with = u. In (b) the smoothness coefficients below 50 are set to 0.0001. In (c) they undergo the same treatment, with a supplementary erosion with a 3 × 3 pixels structuring element. The luminance halo disappears, but (b) has an overshoot that is not completely removed in (c). In (c) some sharp and square-shaped transition appear, due to the erosion.

  with threshold and erosion (base layer) (d) with threshold and erosion (detail layer)

  Figure 11.11: Original coefficients for α = 1 (blue line) and the same coefficients modified with the function f (.) (red line). Parameters are s = 1, ω = 1000 and β = 8. Same with the coefficients obtained with α = 2 (yellow and purple lines).

  Figure 11.12: Results with the modified filter WLS. Only the luminance channel of the input (a) is filtered; the color in the base layer (b) and the contrast-enhanced image (c) is reintroduced afterward, by multiplying by the color coefficients. The parameters used are α = 1, λ = 50, s = 1, ω = 1000, β = 8, and guide image is the simple luminance = 0.2989R + 0.5870G + 0.1140B where R, G and B are the three color channels. The detail layer (and thus the output image) has no objectionable luminance halo, nor gradient reversal artifact.

  Figure 11.14: Application of the super-connected WLS filter. The input image (a) is filtered with the standard WLS and the detail layer (multiplied by 5 for display purpose) is shown in (b). Then, the super-connected WLS filter is applied and the detail layer (same multiplicative factor) shown in (c). The paws length are 3 and 5-pixels long. The zoom-in shown in (d) and (e) focus on a part of the input image where the initial WLS has compartmentalization, and the super-connected one has less. The artifact is not completely removed.

  Figure 11.15: Comparison of the detail layers given by the original WLS filter (a) and the proposed one (b) with the flatness detector. The flatness coefficients γ used in (b) are displayed in (c). In this last illustration the scale is [1, 11] (from dark blue to white in (c)). The parameters used are λ = 6.4, α = 1.2 and we use the log-luminance. Concerning the flatness detector, we use ρ = 10, r = 1 and t = 0.015. Most of the compartmentalization effect is removed. Note that the white halo in the sky is removed too.

  Figure 11.16: Comparison of the detail layers obtained using the same smoothness coefficients in both directions (c) and with the different coefficients in the directions x and y (b). So image (b) uses the absolute value of the gradients in x and y, shown respectively in (d) and (e). On the other hand image (c) uses the norm of the gradient shown in (f ). For visibility purposes, a log 10 has been applied to images (d), (e) and (f ); the displayed range in [-3, 0]. In both filters we reduced the coefficients at edges with the function f . The parameters used are: α = 1 and λ = 49, with = u. There is almost no difference between detail layers (b) and (c). The version with "isotropic" coefficients (c) has a slightly smaller smoothing power, but this is due to the gradients that are globally stronger in (f ) than in (d) and (e). Thus the smoothness constraint is smaller for (c).

  Figure11.17: Base+detail decomposition (b) of image (a) using WLS with the penalization of the 2 nd order derivative. A tone-mapped result using this decomposition is shown in (d). The decomposition given by this filter is wrong: ringing appears and detail extraction is very light. (Only luminance is filtered here).

FigureFigure 12 . 1 :

 121 Figure12.1: Section of the test pattern displayed in Figure12.3 (a), with the filtered versions superimposed. The red line stands for MGFDH and the yellow one for MGF. One can clearly observe the dark halo produced by the former, and that it is removed in the latter.

Figure 12 . 2 :

 122 Figure 12.2: Section of the test pattern displayed in Figure 12.3 (a) where we added a simulated detail (noise), shown in Figure 12.3 d).The input is displayed in blue, the filtered version are superimposed in red and yellow for MGFDH and MGF, respectively (top plot). In the bottom plot, we display the difference between the actual noise and the detail layers produced by MGFDH and MGF. This difference is expected to be zero. MGF better extract the detail but its detail layer also contains the thin black bands on the right.

  Figure 13.1: Test patterns for artifacts measurements.

  Figure 13.2: Luminance halo artifact with GF. The input test pattern (a) and its decomposition are represented with false colors. Dynamic range is [0, 1] for (a) and (b) and [-0.04, +0.04] in (c).

Figure

  Figure 13.3: Staircase effect with FBF (bilateral grid). The input test pattern (a) and its decomposition are represented with false colors. Dynamic range is [0, 1] for (a) and (b) and [-0.1, +0.1] in (c).

  Figure 13.4: Compartmentalization effect with WLS. The input test pattern (a) and its decomposition are represented with false colors. Dynamic range is [0, 1] for (a) and (b) and [-0.1, +0.1] in (c).

Figure 13

 13 Figure13.5: Illustration of the mask used in the compartmentalization measure: in order to quantify the movement of the bright areas (yellow areas when displayed with false colors, in Figure13.4 for example), we measure the standard deviation of the detail layer in the interior of the squares only (blue regions). A two pixels wide area (red regions) is excluded because of the slight smoothing we applied to avoid aliasing.

  This test-pattern consists in a texture (noise) surrounded by contrasted edges with different widths. We display it in Figure . and with false colors in Figure . (a).

  For the contrast halo artifact the worst filter if definitively GF. The Figure . shows the filtering result on this dedicated test-pattern: the texture is hardly removed next the dark barriers.



Figure 13

 13 Figure 13.6: Contrast halo artifact with GF. The input test pattern (a) and its decomposition are represented with false colors. Dynamic range is [0, 1] for (a) and (b) and [-0.1, +0.1] in (c).

  Figure 13.8: Worst filters for the dark halo artifact: BFR and FLL. The plot on the right shows a section of the input test-pattern (blue) and, superimposed, the filtered results of FBR (red) and FLL (yellow).

Figure 13

 13 Figure 13.9: Masks used for the dark halo measure given in Equation (.). The white bands between the dark thin lines are divided in two: the left part (in blue) and the right one (in red). The test-pattern is shown in Figure 13.1 and in false color in Figure 13.8.

  Figure 13.10: Base and detail decomposition with the local extrema filter. The extremely pronounced compartmentalization effect makes the filter useless for contrast enhancement. The parameter used in this experiment, the width of the neighborhood considered for the extrema localization is 10. The detail layer in (c) is multiplied by 3 for visualization purposes.

  Iterated guided filter v2 (IGFv)  r = 2, σ eq = 40 = .0038 2 Fast bilateral filter (FBF) (bilateral grid)  σ s = 40 σ r = .1178 Fast bilateral with regression (fast BFR)  σ s = 40 σ r = .0749 Domain transform, recursive filter (DT) . σ s = 40 σ r = 0.1166 Weighted Least Squares (WLS)  α = 1.2 λ = 0.5 Fast local Laplacian filter (FLL)  l max = 6 σ r = 0.103 Multi-scale BFR (MBR) . σ s = 2, l max = 6 σ r = .0375 Multi-scale guided filter (MGF)  r = 2, l max = 6 = .0385 2 Iterated MGF (MIGF)  r = 2, l max = 6, T = 64 = .0047 2 Bilateral grain filter (BGRF) . λ = 250, β = 50 ρ = 43 Total variation using L 1 norm (TV-L 1 )

  L 2 -norm of each filter. In practice, we only set one parameter per filter. The other ones are fixed according to the values advised by the authors. We list in Table . those parameters and give in the last column the parameter obtained with our procedure. The five images we used are displayed in Figure ..In practice we rather set the PSNR (Peak Signal to Noise Ratio) of the images because it is independent of their size and that the values are more natural to use. PSNR(u 0 -u 1 ) = -10 × log 10 1

Figure 13

 13 Figure 13.12: Final comparison of the measured values. Ranking according to the (normalized) values measured with each test-pattern. Lower is better. Weights for the different measures: luminance halo: 1/5; contrast halo: 2/5; dark halo: 3/5; compartmentalization: 4/5; staircase effect: 5/5.

  Abbr.

  7: Staircase effect measurement. The dynamic range of all results is [-0.03, +0.03]. The staircasing appears as blue lines on the left part of the test-pattern and a yellow line on the right part.

FBRFLL

  Table13.8: Compartmentalization effect measure. The displayed dynamic is[-0.1, +0.1]. The more yellow squares and dark blue lines, the more compartmentalization. Table13.9: Contrast halo measurement. The displayed dynamic range is [-0.1, +.01]. The less texture is extracted along the lines and columns, the stronger the contrast halo. 10: Dark halo measurement. The displayed dynamic range is [-0.1, +.01]. A blue halo around the original dark blue lines gives evidence of the artifact. This is the case with MBR mainly and FLL slightly.well as the first image with the hat. The hat is also good at showing the luminance halo, at the transition between the hat and the ceiling. The image in the second row can help see the luminance halo around the streetlight and in the clouds. The compartmentalization artifact is visible between the branches in the picture of the fifth column.We show in Figure . and Figure . some contrast enhancement results for filter that do not perform a base and detail decomposition.

  13: Natural images, table 3/6: detail layer (1/2) (centered around 127.5 and multiplied by a factor 3 for visualization).

  14: Natural images, table 4/6: detail layer (2/2) (centered around 127.5 and multiplied by a factor 3 for visualization).

  15: Natural images, table 5/6: enhanced images (1/2) using enhance(u) = .125 + .750 × EAF{u} + 3 × (u -EAF{u}). 16: Natural images, table 6/6: enhanced images (2/2) using enhance(u) = .125 + .750 × EAF{u} + 3 × (u -EAF{u}).



  Figure 13.13: Comparison of SEF (b) with ACE (c) and MSR (d) (with chromaticity preservation, [PSM14]). The remapping functions and the corresponding parameters are specified in Figure 8.4 (c).

Figure 15

 15 Figure 15.1: Tree of shape (a) and grain filter (b). The grain filter consists in removing the shapes which area is inferior to a threshold. For the original image of size 256 × 256 given in the top left corner of (b), the grain filter gives the top right, bottom left and bottom right images for this threshold set to 10, 40 and 800, respectively. Both illustrations are reproduced from [MG00].

  Figure 15.2: Histogram of the area (a) and contrast (b) values of all the shapes in the tree obtained with the FLST for the image of the trees (774 × 518 pixels). Both histograms are clipped on the horizontal axis at value 400 but continues farther. The vertical axis is in log scale.

Figure 15 Figure 15 . 4 :

 15154 Figure 15.3: Distribution of the pairs(A, S). The area A is given by the vertical axis, on top the area is small and increases toward the bottom. The contrast measure S is given in the horizontal axis: small values on the left and high ones on the right. This means for S that the high-contrast values are on the left while the low contrast ones are on the right. A natural logarithm was applied to the concentration values. This figure shows the number of pairs (A(u(x), t), S(u(x), t)) that fall in each "2D bin".

  in Equation (.) and Equation (.) are drawn in Figure .. The weighting we use is then 1-g (A(u(x), t)) h (C(u(x), t)) .(.)



  foreach shape of ToS u do  A shape ← Area(shape)  S shape ← Area(GetParent(shape)) -A shape // speed of growth: Equation (.) C shape ← N/ S shape + 1 // contrast: Equation (.) // Associate weight to current shape  w shape ← 1 -g λ,β (A shape ) × h ρ,β (C shape ) //Equation (.) and Equation (.)  root ← root of ToS u // root is the largest shape in u  BGRF{u}(x) ← Value(root) // initialize all pixels with the root's value  foreach pixel x of u do  shape ← GetSmallestShape(x) // smallest shape containing x in ToSu  while shape = root do  BGRF{u}(x) ← BGRF{u}(x) + w shape // integration  shape ← GetParent(shape) // trace the tree of shapes

Figure 15

 15 Figure15.5: Effect of decreasing the contrast parameter ρ. In all these experiments, the other parameters are fixed: λ = 150 and β = 10 -6 . A small β turns the functions g(.) and h(.) into simple thresholds. The smaller ρ, the less contrasted the detail layer. The contrasted shapes migrate back in the base layer. The effect on the compartmentalization is nonetheless mitigated; one can consider that it is removed for ρ = 150 or under in this experiment, yet the detail layers for these values do not contain much information. We recall that ρ = sign(ρ)ρ 2 , λ = sign( λ) λ2 and β = β2 .

  Figure 15.7: Example of decomposition and contrast enhancement using BGRF. λ = 150 2 , ρ = 300 2 , β = .



  Figure 15.10: λ = 300 2 , ρ = 150 2 , β = 100 2 . Image size is 1312 × 2000 pixels. The enhancement is done using the simplest color balance with 0.1% of saturation on the left and 0.1% on the right of the histogram of the enhanced luminance image. The color coefficients are added afterward. We use here v = log(255 × BGRF{u} + 1)/ log(256) + (u -BGRF{u}).

  Figure 15.11: λ = 300 2 , ρ = 250 2 , β = 100 2 . Image size is 1312×2000 pixels. The enhancement is done using the simplest color balance with 0.1% of saturation on the left and 0.1% on the right of the histogram of the enhanced luminance image. The color coefficients are added afterward. The enhanced luminance image v is obtained from u as follow: v = log(255 × BGRF{u} + 1)/ log(256) + (u -BGRF{u}).

  Nous présentons ainsi à nouveau les méthodes retinex (Section .), l'analyse du filtre guidé et la proposition d'un filtre guidé itéré (Sections . et .), et les trois chapitres sur le filtre bilatéral (Sections ., . et .), où nous présentons le filtre et

ses variantes, l'effet de staircasing et les différentes manières de le corriger et pour finir les approximations rapide du filtre. Nous renvoyons au Chapitre  précédent pour la partie de l'introduction qui concerne la fusion d'exposition et pour notre proposition de simulated exposure fusion (Section . et Section .), l'analyse du local Laplacian filter et son extension (Section . et Section .), du filtre weighted least squares (Section .), la proposition du multi-scale guided filter (Section .), mais aussi pour la comparaison finale des méthodes entre elles (introduite en Section .). Nous redonnons toutefois le résumé et la liste des contributions, en Section ..

  Figure 1.3: Artefact de halo de contraste dans le filtre guidé : le lissage est réduit près des contours de fort contraste. A gauche : signal 1D en marche d'escalier avec un bruit faible (ligne bleue) et sa version lissée avec le filtre guidé (ligne rouge). Sur la droite, nous montrons la couche de détail. En haut, nous avons la différence entre les deux

	1
						0.05
	0.9
	0.8					0
	0.7					-0.05
	0.6
						-0.1
	0.5
						-0.15
	0.4
	0.3					-0.2
	0.2					-0.25	detail (= input -filtered)
	0.1			input signal			original noise (expected detail) (-.1 offset for display)
				filtered signal	-0.3	difference detail -noise (-.2 offset for display)
	0
	50	100	150	200	250		50	100	150	200	250
						

où (ā, b) sont les coefficients linéaires agrégés. L'équation (.) a une solution analytique, rendant le filtre extrêmement rapide à calculer, puisqu'il ne nécessite que des moyennes locales. De plus, ces dernières peuvent être calculées en temps linéaire grâce à des images intégrales.

L'artefact du halo de contraste dans GF L'artefact principal du filtre guidé vient du fait que les contours sont conservés, et la zone alentour également. Nous montrons un exemple du phénomène qui en résulte dans la Figure .. L'artefact est particulièrement présent lorsque le filtre est utilisé avec un grand rayon. En effet, le filtre guidé ne peut pas lisser la moitié d'une fenêtre et garder l'autre moitié telle quelle ; le choix qui est fait est souvent une décision intermédiaire : à moitié lissée, à moitié conservée. Ainsi, il crée également l'artefact du halo (que nous appelons halo de luminance pour le distinguer clairement du halo de contraste).

L'artefact du halo de luminance dans GF

L'artefact du halo de luminance apparaît lorsque les bords ne sont pas bien conservés par le filtre. C'est le cas du filtre guidé, comme le montre la Figure .. Comparé à son concurrent le filtre bilatéral, le filtre guidé lisse moins les textures qui doivent être enlevées et lisse davantage les bords qui devraient être conservés. . Chapitre : Le filtre guidé itéré Le Chapitre  introduit le filtre guidé et ses artefacts, à savoir le halo de contraste et le halo de luminance. Une comparaison avec le filtre bilatéral montre que ses propriétés de préservation des contours et de lissage ne font pas d'ombre au filtre bilatéral. D'autre part, le filtre guidé présente l'avantage de ne pas avoir l'effet d'escalier. Cela rend ce filtre particulièrement souhaitable pour l'amélioration du contraste.

signaux de gauche : entrée -filtrée (ligne verte). La couche de détail est presque plate en son centre, où le signal d'entrée a son bord d'étape. Pour comparaison, le bruit d'entrée du motif de test (c'est le détail attendu) est affiché dessous (au milieu, ligne bleue). La différence entre ces deux signaux est également présentée (en bas, ligne rouge), qui montre que le détail obtenu est presque partout égal au bruit sauf au milieu, où la différence contient le bruit d'entrée. Le paramètre utilisé est r = 16 et = 0.03 2 .

  Les Chapitres  et  sont dédiés au filtre guidé, un filtre de lissage récent et rapide préservant les contours. Ils le relient à la diffusion anisotrope et le comparent au filtre bilatéral. Ces deux derniers filtres sont les filtres les plus répandus pour le calcul d'une base, dans le cadre de la décomposition d'une image en base et détail.Dans le Chapitre , nous présentons le filtre bilatéral, ses avantages et ses inconvénients. Nous rappelons sa longue histoire, et décrivons ses principaux descendants : le filtre bilatéral conjoint (ou croisé, joint and cross bilateral filters) [ED, PSA + ], le filtre bilatéral avec régression[BCM], et les filtres bilatéraux non normalisés [APH + , APH + , MT]. De plus, nous faisons le lien entre le filtre bilatéral et ACE (Automatic Color Enhancement, Amélioration Automatique de la Couleur en français), qui appartient à la famille retinex.Nous décrivons et montrons l'effet d'escalier, suivant sa description et sa solution par Buades et al.[BCM]. Les nombreux schémas proposés pour corriger cet artefact seront passés en revue dans le Chapitre .Son principal inconvénient en pratique étant sa lourdeur en calculs, nous poursuivons dans le Chapitre  par une revue des approximations rapide du filtre bilatéral. La version rapide du filtre bilatéral avec régression et du filtre bilatéral non normalisé sont décrites au Chapitre .

Le filtre bilatéral avec regréssion Ce filtre, introduit par Buades et al. dans

	0.85 La moyenne des différences (u(y) -u(x)), en utilisant les poids bilatéraux, calcule directement 0.8 1 1 la couche de détail. Parce que les intensités du détail oscillent autour de zéro, le facteur de nor-
	0.8 malisation peut être supprimé sans déformer l'image filtrée. Cela accélère le filtre et réduit l'effet 0.6
	0.4 d'escaliers ; cependant, il réduit également la quantité de lissage et introduit un léger halo de con-
	0.75 traste comme dans filtre guidé. On peut le comprendre avec cette définition alternative du filtre 0.2 ← σ R →
	0.7 bilatéral non normalisé basée sur BF :		← σ S →		-0.15 0		(b) Range kernel -0.1 -0.05 0 0.05 0.1	0.15
	0.65 où C est le facteur de normalisation du filtre bilatéral. UBF{u}(x) = C(x)BF{u}(x) + 1 -C(x) u(x), 8 9 10 ×10 -4 ← σ R → 6 7							1 2σS +1	(.)
									5
									4
	0.6								3
							test-pattern bilateral filter	2 0 1							← σ S →
	0.55	10	15	20	25	30	35	40	45	-20	-15	-10	-5	0	5	10	15	20
				(a)							(c) Spatial kernel
	Figure 1.6: Explication de l'effet escalier pour un filtre bilatéral avec des noyaux spatial et d'intensité simplifiés.
	Le pixel courant est à l'intersection des lignes pointillées bleues verticales et horizontales. Le rectangle noir en
	pointillé indique quels pixels seront pris en compte dans la moyenne. Les flèches verticales bleues représentent
	la différence d'intensité entre le pixel courant et les autres pixels du rectangle. Puisque le pixel courant a plus de
	voisins (dans le sens bilatéral) du côté droit du bord, sa valeur bilatéralement moyennée se rapproche de la valeur
	du plateau.
	Effet d'escalier (staircasing effect) L'artefact d'escalier est illustré dans la Figure .. Dans cette Tous ces filtres similaires peuvent être appelés filtres de voisinage ; les figure, nous avons simplifié les noyaux spatial et d'intensité en utilisant de simple fenêtres. Cela seules différences résident dans la forme du noyau d'intensité et du noyau spatial. La performance permet une visualisation simple, dans le cas à une seule dimension, des pixels pris en compte dans de ces algorithmes est justifiée par les mêmes arguments : à l'intérieur d'une région homogène, les le processus de moyennage. Les flèches bleues sont les différences d'intensité u(x) -u(y). La valeurs du niveau de gris fluctuent légèrement en raison du bruit ou de la texture. Dans ce cas, le zone délimitée par la ligne en pointillés montre les limites des noyaux spatiaux et d'intensité : en filtre bilatéral calcule une moyenne. À un bord contrasté séparant deux régions, si la différence de dehors de cette zone, tous les poids bilatéraux sont nuls. Il est alors facile de voir que pour le pixel niveau de gris entre les deux régions est significativement plus grande que σ r , alors l'algorithme courant (à savoir, l'intersection des deux lignes pointillées bleues au centre de la boîte), la valeur calcule des moyennes de pixels appartenant à la même région que le pixel de référence. Ainsi, moyenne a une intensité plus élevée que la valeur initiale. En appliquant la moyenne bilatérale l'algorithme ne rend pas les bords flous, ce qui est son objectif principal. sur chaque pixel de la ligne bleue, on obtient la ligne rouge. La "propagation du plateau" que l'on La version popularisée par Smith et Brady et Tomasi et Manduchi utilise une fonction de peut observer est ce que nous appelons l'artefact d'escalier (staircasing effect). Cela revient à un pondération gaussienne dépendant d'un paramètre de filtrage σ r (noyau d'intensité), ainsi qu'un noyau spatial gaussien : renforcement indésirable des contours principaux.
	BF{u} σr,σs (x) = un plan de régression à chaque pixel. Contrairement au filtre bilatéral standard qui estime une 1 C(x) Ω u(y)e -|y-x| 2 2σ 2 s e -|u(y)-u(x)| 2 2σ 2 r dy, [BCM], estime (.)
	où C(x) = Ω e constante, le filtre, utilisé avec de petits σ s , ne provoque plus l'artefact d'escalier. Nous appelons -|y-x| 2 2σ 2 s e -|u(y)-u(x)| 2 2σ 2 r k = k(x, y) les poids du filtre bilatéral au point (0, 0) pour l'image u = u(x, y). Le filtre bilatéral dy est le facteur de normalisation et σ s est un paramètre de avec régression trouve filtrage spatial. Le cross bilateral filter (filtre bilateral "croisé" ou "transversal") [ED], ou le joint bilateral filter (filtre bilatéral "conjoint" ou "partagé") [PSA + ], calcule son noyau d'intensité en fonction arg min a,b,c x,y k(ax + by + c -u) 2 . (.)
	d'une seconde image, une image guide v : Le résultat final est simplement BFR{u}(x) = c(x). Nous complétons ce chapitre en analysant un
	CBF{u} σr,σs (x) = dernier candidat pour atténuer l'effet d'escalier, le filtre bilatéral non normalisé. -|y-x| 2 -|v(y)-v(x)| 2 1 u(y)e 2σ 2 s e 2σ 2 r dy, C(x) Ω Le filtre bilatéral non normalisé Introduit récemment par Aubry et al. in [APH + ], le filtre (.)
	où le facteur de normalisation C est calculé en conséquence. Cela est utilisé par exemple pour le bilatéral non normalisé est définit comme :
	débruitage d'un couple d'image avec flash / sans flash, où l'information des contours de l'image
	prise avec flash est utilisée pour filtrer l'image sans flash, qui a de meilleures couleurs mais aussi UBF{u}(x) = u(x) + G σs (x -y)G σr u(y) -u(x) u(y) -u(x) . (.)
	plus de bruit.			y∈Ω
							

.

 Chapitre : Corrections de l'effet d'escalier

	Dans le Chapitre  nous avons montré que le filtre bilatéral ne préserve pas seulement les bords,
	mais qu'il est aussi enclin à les renforcer. Cet effet a été décrit et justifié mathématiquement par
	Buades et al. en  [BCM], qui lui ont donné le nom d'effet d'escalier (staircasing). En
	effet, les filtres basés sur le bilatéral ont tendance à créer des signaux constants par morceaux
	séparés par des arêtes créées numériquement, prenant ainsi l'aspect d'un escalier. Du point de
	vue du rehaussement de contraste et du mappage de tons, le même effet est parfois appelé artefact
	d'inversion de gradient (gradient reversal artifact), car la couche de détail complémentaire, aux
	endroits où les bords ont été renforcés dans la couche de base, contient des gradients inversés. Le
	problème est que lorsque le filtre bilatéral est utilisé pour l'amélioration du contraste et le mappage
	de tons, la couche de détail est étirée et la couche de base compressée. La recombinaison de leurs
	résultats provoque l'artefact d'inversion de gradient.

Exemple de résultat avec la diffusion selective La

  Figure . affiche le résultat de la diffusion sélective appliquée au filtre bilatéral, dans le cadre de l'amélioration du contraste. Cette méthode réussit à enlever une grande partie de l'artefact d'inversion de gradient (qui, comme nous l'avons vu, est une conséquence de l'effet d'escalier) visible sous la forme de bandes foncées et blanches à la lisière des arbres. Bien que cette méthode fonctionne globalement bien, il semble impossible de retirer l'effet d'escalier partout, en particulier dans les coins (voir Figure .(c)). En outre, il n'est pas efficace sur le plan du temps de calcul. En effet, de nombreuses itérations sont nécessaires pour corriger l'effet escalier, et ce temps de calcul s'ajoute au temps de calcul du filtre lui-même.

.

 Chapitre : Filtres bilatéraux rapides

	Le filtre bilatéral est rapidement devenu omniprésent dans le traitement d'image et est maintenant
	utilisé dans un très grand nombre d'applications. Le filtre original doit calculer un noyau différent
	à chaque pixel, ce qui le rend lent, voire non abordable pour de grandes images et (par conséquent)
	un large support spatial. D'où la nécessité d'une implémentation rapide.
	Dans le Chapitre  nous passons en revue les nombreux filtres bilatéraux rapides proposés
	dans la littérature. L'histoire du filtre bilatéral rapide commence avec l'approximation rapide de
	Durand et Dorsey ()

  par le noyau G σs . De la même manière, le facteur de normalisation C est la convolution de I u(x) : y → G σr u(y) -u(x) par G σs . La dépendance à x dans G σr est la seule chose qui diffère d'avec une convolution. A partir de cette observation, la stratégie d'accélération des auteurs est de discrétiser l'ensemble des intensités de signal possibles dans les N layers valeurs {γ(i)}, et de calculer une convolution gaussienne linéaire pour chacune de ces valeurs :

	où C est le facteur de normalisation. C'est équivalent à la convolution de la fonction H u(x) :
	y → G σr u(y) -u(x) u(y)
	y),	(.)
	

  r représente le paramètre d'intensité du filtre bilatéral. Soit Ω le voisinage du pixel x et G σs le noyau gaussien d'écart-type σ s . Avec un tel noyau, le filtre bilatéral peut être écrit BF poly. {u}(x)

  The coefficients a(k), b(k) of the local linear model are computed at lines  and , in which the coefficient of each window ω(k) is stored at pixel k. At lines  and , the coefficients of the overlapping windows are aggregated, and the final image GF {u} is computed at line .

	Algorithm : Guided Filter algorithm (All operations are pixel-wise)
	input : input image u
	input : guide image GF {u}
	input : smoothing parameter
	input : window radius r (box window will have size (2r + 1) 2 )
	output: filtered image R
	 ū ← Mean ω {u}	// Empirical mean of u in windows ω



  The preserved edge heights are h

		preserved α	= 6	√	α and the
	well smoothed heights are h smoothed α	= 2 3

  layer at γj

		Hj = G σs * H j				// convolve the layer with the truncated
							//	normalized spatial Gaussian kernel
											// update output image
		0.9					0.9
		0.8					0.8
		0.7					0.7
		0.6					0.6
		0.5					0.5
		0.4					0.4
		0.3					0.3
		0.2			test-pattern Bilateral filter	0.2				test-pattern Bilateral filter
					Unnoramlized Bilateral				Unnoramlized Bilateral
		0.1					0.1
		10	20	30	40	50	60	10	20	30	40	50	60



UBF fast {u} = UBF fast {u} + α j Hj

  equation (.)

 return TF{u}

Algorithm : minStack and maxStack algorithms input : input u input : number of levels N input : filter (min or max) output: stack of filtered image with height N

  Algorithm : Local Histograms Bilateral Filter (FBF int.hist. ) for a D gray image input : image u input : range standard deviation σ r input : window radius r output: FBF int.hist. {u}  H ← compute the integral histogram

			// equation (.)
		// Retrieve local histogram with radius r
		h Ω ← compute local histogram in Ω(x)	// equation (.)
		// Compute bilateral output value

 foreach pixel x and x-centered window Ω(x) do 

  Algorithm 23

	 foreach pixel (x,y) do	// slicing and division
		num(x, y) ← Γwu x, y, u(x, y)	// Γwu is the 1st component of	Γ



denom(x, y) ← Γw x, y, u(x, y)

// Γw is the 2nd component of Γ  FBF grid {u}(x, y) ← num(x,

y)

denom(x,y)

  of ct in ver. direction

	 Compute ct hor. where ∀x, ct(x) ← x 0 ct x  Compute ct ver. where ∀y, ct(y) ← y 0 ct y	// hor. domain transform // ver. domain transform



  Images owner is unknown. Sequence can be found at http://www.hdrsoft.com/examples.htmlIn Chapter  and Chapter  we described the exposure fusion method and proposed a framework to extend it to the single image case through the generation of a simulated bracketed exposure sequence. This fusion algorithm is based upon the manipulation of Laplacian pyramids, and has demonstrated the usefulness of such a multi-scale image representation. We focus in this chapter on the local Laplacian filters. They use the same Laplacian pyramid but in the context of multi-scale local contrast manipulation. The local Laplacian filters have originally been proposed in  by Paris, Hasinoff and Kautz[PHK]. A fast version was proposed the same year by Aubry,Paris, Hasinoff, Kautz and Durand [APH

	9 Local Laplacian filters and connection
	to other operators
		

+ ]. The initial conference papers were extended to journal papers in  for the Aubry et al. fast local Laplacian filters [APH + ] and in  for the Paris et al. original local Laplacian filters

  [PHK, PHK]. The next section will present the faster O(sN ) version (s is the number of slices) proposed by Aubry et al. [APH + , APH + ]. This one is called the fast local Laplacian filter (FLL). It uses the Durand, Paris et al. [DD, PD, PD, CPD] slicing method.

Two pseudo-codes are given: the O(N 2 ) version, and its accelerated version with O(N log N ) complexity, both proposed by Paris  et al.

  parameters odd h , odd w output: ū↑ the upsampled image of size (2H + odd h , 2W + odd w ). Burt and Adelson's kernel defined in (.)  u pad ← increase size of u by replicating its first and last lines and columns  u ↑ pad ← initialize with zeros an image of size (H , W ) = (2H + 4, 2W + 4)  foreach pixel (x 1 , x 2 ) with x 1 ∈ {0, . . . , H + 1} and x 2 ∈ {1, . . . , W + 1} do

re-sample

 return v  Algorithm : Upsample input : image u of size (H, W ) input :  H ← height of u  W ← width of u  K ←  u ↑ pad (2x 1 , 2x 2 ) ← 4 × u pad (x 1 , x 2 ) //

factor 4 for normalization  ū↑ pad ← u ↑ pad * K // interpolate with the same filter K  ū↑ ← remove  first and (2odd h ) last lines and  first and (2odd w ) last columns from ū↑ pad // remove padding Algorithm : Collapse input : Laplacian pyramid Lpyr{u} output: image u

Table

  .. The plots in Figure . display a summary of the experiments by superimposing the results of the four filters for one selected line of the images.

  On the other hand, this makes MGF not translation-invariant, like the local Laplacian filter (see Chapter ). Our filter also tends to consider small elements as detail even if they are contrasted (like the bands in Figure .), which is often not desirable. Apart from this, MGF seems to perform a clean base and detail decomposition. This in fact will be confirmed in next Chapter . where the different methods are compared. Since this filter is faster than LLF, it appears as one of the best option for base and detail decomposition for contrast enhancement.

	1
	0.9
	0.8
	0.7
	0.6
	0.5
	0.4
	0.3
	0.2	input test-pattern
	0.1	MGFdh (no shrinkage) MGF (with shrinkage)
	0
		50	100	150	200	250	300	350	400	450	500	550	600
	0.4
	0.35										zero MGFdh (no shrinkage)
	0.3										MGF (with shrinkage)
	0.25
	0.2
	0.15
	0.1
	0.05
	0
	-0.05
		50	100	150	200	250	300	350	400	450	500	550	600
							

  The corresponding test-patterns are displayed in Figure ..

  Table13.1: List of base and detail decomposition filters presented in the thesis.

	Algorithm	Chapter	Fixed	Set
	Method	Section Algorithm	Reference
	Exact bilateral filter (BF)				[Yar, Lee, SB, TM]
	Exact bilateral filter with regression (BFR)		.		[BCM]
	Fast bilateral filter with regression (FBR)		.		Chapter 
	Multi-scale bilateral with regression (MBR)		.		Chapter .
	Exact and fast unnormalized BF (UBF)		.	, 	[APH + , APH + ]
	Fast BF: separable		.		[PVV]
	Fast BF: local (integral) histograms		.		[Wei, Por, YTA]
	Fast BF: piece-wise linear, bilateral grid (FBF)	.	, 	[DD, PD, CPD]
	Fast BF: polynomials		.		[Por, CSU]
	Domain transform		.		[GO]
	Perona-Malik anisotropic diffusion		.	-	[PM]
	Guided filter (exact and fast versions, GF)			, , 	[HSTb, HST, HS]
	Iterated guided filter (v,v,v, e.g. IGFv)			, , 	Chapter 
	Local Laplacian filter (LLF)				[PHK, PHK]
	Fast local Laplacian filter (FLL)				[APH + , APH + ]
	Scale-Space LLF (exact and fast, SLF)			, 	Chapter 
	Bilateral pyramid (Fattal et al.)			-	[FAR]
	Bilateral pyramid (Chen et al.)			-	[CPD]
	Weighted least squares filter (WLS)		.		[FFLS]
	WLS with halo reduction		.		Chapter 
	Super-connected WLS		.		Chapter 
	WLS with flatness protection		.		Chapter 
	Multi-scale guided filter (MGF)				Chapter 
	Grain filter		.	-	[MG]
	Bilateral grain filter (BGRF)		.		Chapter .
	Method	Section Algorithm	Reference
	Multi-scale retinex	.	-	[JRWa, PSM]
	Automatic Color Enhancement	.	-	[GRM, Get]
	Simulated exposure fusion			Chapter 
	Table 13.2: Contrast manipulation filter without base+detail decomposition
		

Table

  . are presented in Table . and Table ..

	

  Table ., Table ., Table ., Table . and Table

Table

  . and Table .);

		DT	L 0 -IS	BGRF	FBF	FBR	FLL	GF
	1	staircase effect
	2 compartmentalization
	3	dark halo
	4	contrast halo
	5	luminance halo
		IGF	MGF	MIGF	MBR	TV-L 1	WLS
	1	staircase effect
	2 compartmentalization
	3	dark halo
	4	contrast halo
	5	luminance halo
			

Table 13

 13 .4: Exclusion of some filters due to unacceptable artifacts (red). Only LLF and MGF have no objectionable artifacts.

Table

  . and Table.. The following pair displays the detail layers. See Table. and Table.. These tables may be the more appropriate to compare the results because the details better highlight differences between filters. The third pair of tables present enhanced images. See Table. and Table.. The enhancement algorithm is very simple and does not involve a final stretching (we use clipping instead) so as to provide comparable results. The displayed images are computed using enhance(u) = .125 + .750 × EAF{u} + 3 × (u -EAF{u}), (.)

  Table 13.6: Luminance halo measurement. The dynamic range of all results is [-0.05, +0.05].

Table 13 .

 13

	

11: Natural images, table 1/6: base layer (1/2).

WLS

Table

13

.12: Natural images, table 2/6: base layer (2/2).

Remerciements

Method ms/Mp

Gray, CPU Integral histograms, Weiss  [Wei]  Bilateral grid () [PD, CPD]  †

Constant time BF, Porikli  [Por] B = 16

 † Porikli  [Por] with moving histogram [PH] and SSE  Edge-avoiding wavelets, Fattal et al.  [Fat]  † Weighted least squares, Farbman et al.  [FFLS]  Trigonometric, Chaudhury et al.  [CSU] (σ r = 30)

 † Domain transform, Gastal et al.  [GO] ( iterations) NC 55 ‡ RF 30 ‡ Guided filter, He et al. [HSTb, HST]  † Fast guided filter, He et al. [HS] (s = 4) 4 ‡ ‡

Gray, GPU

Bilateral grid, Chen et al.  [CPD]  † Constant time BF, Yang  [YTA]  † Weighted Least Squares, Farban et al.  [FFLS]  * * Domain transform [GO] (NC)  ‡ Guided filter, Bauzsat et al.  [BEM]  †

Color, CPU

Gaussian KD-tree, Adams et al.  [AGDL] ≥ 10000 † Permutohedral Lattice, Adams et al.  [ABD] ≥ 1000 † Domain transform, Gastal et al.  [GO] ( iterations)

NC

 † RF  † Guided filter, He et al. [HSTb, HST]  † Fast guided filter, He et al. [HS]  ‡ ‡ Adaptive manifolds, Gastal et al.  [GO]  †

Color, GPU

 * * * Adaptive manifolds, Gastal et al.  [GO] - † † : given by the authors. : measured by He et al. [HST]. : reported by Chen et al. in [CPD].

: reported by Fattal et al. in [Fat]. ‡ : no execution time is given for the gray case in [GO], but one can expect to have little more than one third of the color execution time. ‡ ‡ : He et al. in [HS] report a ">10× speedup" for their O(N/s 2 ) fast guided filter. * * : reported by Gastal et al. in [GO]. * * * : reported by Gastal et al. in [GO].

. Multi-scale guided filter with Laplacian coefficients shrinkage

The multi-scale guided filter works well, but despite the usage of the original image as a guide the dark halo artifact is still present. We shall now explain why and how to resolve this inconvenience.

The reason for the presence of a remaining dark halo is that there is still an inconsistency between the upsampled edges and the Laplacian coefficients that are added to it. Indeed, even the strong edges can be compressed in the guided filter -all the more so as GF is one of the filters with the strongest luminance halo, as will be shown in Chapter .. Therefore, for most edges the Laplacian coefficients are computed on greater edges than the upsampled ones (the image is upsampled after it has been filtered with GF), which creates this artificial bounce. Fortunately, the coefficient ā of the guided filter directly gives us the measure of the shrinkage; therefore all we need is to apply the same coefficients to the Laplacian coefficients. Including this modification, the multi-scale guided filter becomes

where āl+1 is the guided filter's coefficient at scale l + 1.



Algorithm : Guided filter (GF) input : image u input : guide image v input : parameter input : parameter r (patch radius) output:

// average of coefficients of the overlapping patches

// average of coefficients of the overlapping patches

Algorithm : Iterated guided filter (IGFv) input : image u input : guide image v input : guided filter's parameters and r input : parameter T : number of iterations output:



This filter's pseudo-code is described in Algorithm . It uses the (iterated) guided filter in Algorithm  that outputs its coefficients ā. At lines - in IGFs we recompute the coefficients ā using the parameter µ rather than . This allows to use a very small value (we typically take µ = 10 -12 ) and then to avoid shrinking the Laplacian coefficients more than needed.

Algorithm : Iterated guided filter with shrinkage (IGFs) input : image u input : guide image v input : Parameters , r and T output:

// output coefficients used at the following scale Algorithm : Multi-scale (iterated) guided filter with shrinkage (MGF) input : image u input : parameters r, T , , l max output:

We experimentally demonstrate that the filter described in Equation (.) solves the dark halo artifact in Figure .. We used the test-pattern displayed in Figure . (a). It is composed of a bright background on which several dark bands with different width are superimposed. In the same Figure . we show the detail layers given by MGFdh in the central column, and by MGF in the right column. The top line corresponds to the filtering of the input test pattern in (a), and the bottom row to the filtering of the same test-pattern after we added noise (simulate the detail). This noise is shown in (d). This last experiment shows that MGF is as capable as MGFdh to extract detail. In Figure . we present sections of those results. From the top plot, without noise, it is clear that the dark halo has been removed. The second and third plots show that MGF is in fact better than MGFdh: in the third plot we display the difference between the expected detail layer (i.e., the noise we added) and the detail layer given by both filters. We want the difference to be zero. In practice, the oscillations of the yellow line are often smaller, especially on the right part of the test-pattern where the bands are large. On the other hand, we observe that MGF smooths out the black bands on the left part of the pattern. This is because they tend to disappear when the image is downsampled, so the coefficients ā inherited from the previous scale are close to zero, which suppresses the oscillation. The version with coefficient shrinkage also seems to have more contrast halo. However, we found this artifact less perceptible than the dark halo. A similar observation is made by Eilertsen et al. in [EMU]: "since the aim of the spatial filtering is base - Keywords: base and detail decomposition, edge-aware smoothing, quality assessement, tone-mapping, image enhancement, computational photography.



Abstract:

In this CIFRE thesis, a collaboration between the Center of Mathematics and their Applications, École Normale Supérieure de Cachan and the company DxO, we tackle the problem of the additive decomposition of an image into base and detail. Such a decomposition is a fundamental tool in image processing. For applications to professional photo editing in DxO Photolab, a core requirement is the absence of artifacts. For instance, in the context of contrast enhancement, in which the base is reduced and the detail increased, minor artifacts becomes highly visible. The distortions thus introduced are unacceptable from the point of view of a photographer.

The objective of this thesis is to single out and study the most suitable filters to perform this task, to improve the best ones and to define new ones. This requires a rigorous measure of the quality of the base plus detail decomposition. We examine two classic artifacts (halo and staircasing) and discover three more sorts that are equally crucial: contrast halo, compartmentalization, and the dark halo. This leads us to construct five adapted patterns to measure these artifacts. We end up ranking the optimal filters based on these measurements, and arrive at a clear decision about the best filters. Two filters stand out, including one we propose.