One of the main goals of digital imagery is to improve the capture and the reproduction of real or synthetic scenes on display devices with restricted capabilities. Standard imagery techniques are limited with respect to the dynamic range that they can capture and reproduce. High Dynamic Range (HDR) imagery aims at overcoming these limitations by capturing, representing and displaying the physical value of light measured in a scene. However, current commercial displays will not vanish instantly hence backward compatibility between HDR content and those displays is required. This compatibility is ensured through an operation called tone mapping that retargets the dynamic range of HDR content to the restricted dynamic range of a display device. Although many tone mapping operators exist, they focus mostly on still images. The challenges of tone mapping HDR videos are more complex than those of still images since the temporal dimensions is added. In this work, the focus was on the preservation of temporal coherency when performing video tone mapping. Two main research avenues are investigated: the subjective quality of tone mapped video content and their compression eciency.

Indeed, tone mapping independently each frame of a video sequence leads to temporal artifacts. Those artifacts impair the visual quality of the tone mapped video sequence and need to be reduced. Through experimentations with HDR videos and Tone Mapping Operators (TMOs), we categorized temporal artifacts into six categories. We tested video tone mapping operators (techniques that take into account more than a single frame) for the dierent types of temporal artifact and we observed that they could handle only three out of the six types. Consequently, we designed a post-processing technique that adapts to any tone mapping operator and reduces the three types of artifact not dealt with. A subjective evaluation reported that our technique always preserves or increases the subjective quality of tone mapped content for the sequences and TMOs tested.

The second topic investigated was the compression of tone mapped video content.

So far, work on tone mapping and video compression focused on optimizing a tone map curve to achieve high compression ratio. These techniques changed the rendering of the video to reduce its entropy hence removing any artistic intent or constraint on the nal results. That is why, we proposed a technique that reduces the entropy of a tone mapped video without altering its rendering. Our method adapts the quantization to increase the correlation between successive frames. Results showed an average bit-rate reduction under the same PSNR ranging from 5.4% to 12.8%.

Chapter 1 Introduction

Since the dawn of the eld of photography, photographers and scientists have striven to solve two main issues: how to capture a scene with the highest delity? how to best reproduce the perception of a human observer when the capture of a scene is projected on a targeted display?

To achieve both of these goals, the eld of photography has been in constant evolution since the rst picture taken by Nicéphore Niépce in the mid-1820s. Nevertheless, current capture and display technologies are still limited in regard to the dynamic range that they can achieve as illustrated in Figure 1.1.

The rst observation that we can make from this gure is that a camera adapts the captured dynamic range to a scene. Indeed, in cameras, a parameter eV (exposure Value) allows us to tune the exposure of a captured scene. Second, displays lack such an adaptation and can only achieve luminances lying in a xed range. Finally, most cameras capture a higher dynamic range than can be directly reproduced on a display.

From these three observations, we understand that a scene captured by a camera needs to be adapted to the targeted display characteristics.

In digital photography, pixel values correspond to a relative value of a standard representation. The conversion of physical values, measured by sensors, to this standard representation is performed in the camera and is dened by a camera response function.

The resulting image can then be displayed on any commercial display, its rendering depends on the display. If the dynamic range of a camera is wider when compared to what the chosen standard pixel format can represent, then information gets lost. To summarize, the traditional digital imaging pipeline suers from several shortcomings: during the capture, information gets lost as a camera cannot record all the dynamic range present in a scene, during the storing, information captured by the sensor's camera is adapted to the standard representation capabilities, values that correspond to a standard of representation. The backward compatibility between HDR content and Low Dynamic Range (LDR) displays is ensured by a tone mapping operation.

Tone mapping an HDR image amounts to retargeting physical values, with a virtually unlimited bit-depth, to a constrained space (2 2n color hue over 2 n tonal level, n being the targeted bit-depth). This limited bit-depth means that many similar HDR values will be tone mapped to the same LDR one. Consequently, contrast between neighboring pixel as well as spatially distant areas will be reduced. Furthermore, LDR displays have a low peak luminance value when compared to the luminance of a real scene. Consequently, captured color information will have to be reproduced at dierent luminance level. To summarize, tone mapping an HDR image amounts to nding a balance between the preservation of details, the spatial coherency of the scene and the delity of reproduction. This balance is usually achieved by taking advantage of the many weaknesses of the human visual system. Finally, the reproduction of a scene is sometimes constrained by an artistic or application dependent intent. That is why, a lot of Tone Mapping Operators (TMOs) have been designed with dierent intents: from simulating the human vision to achieving the best subjective quality.

Due to the lack of HDR video content and the assumption that TMOs developed for images would behave correctly for videos, the tone mapping eld has for a long time only focused on still images. However, with the rising interest of both the digital cinema industry (ACES workow) and the television broadcasters (demos at NAB, IBC, etc.) in HDR video content, tone mapping HDR video sequence has received a great deal of attention lately. That is why, the goal of this thesis is to assess the maturity of the video tone mapping eld especially in regard to the mass-distribution of HDR content to the end-user. More precisely, we want to know if the current tone mapping techniques are robust enough to tone map HDR videos without user-interaction. Furthermore, those tone mapped content will need to be distributed to the end-consumer before being displayed. Indeed, uncompressed video content (HDR or LDR) are represented by a too large amount of data to t the storage or broadcast requirements of current video processing pipelines. Work has been performed on tone mapping and video compression, but focused on optimizing the tone map curve to achieve high compression ratio. These techniques modify the mapping performed by a TMO to reduce the entropy of a sequence. However, this modication alters the visual perception of this content and can impair any artistic intent or desired rendering. We would like to increase the compression eciency of tone mapped video content without altering their rendering.

Structure of the Thesis

The rst chapter of this thesis provides the necessary knowledge to understand how HDR imaging techniques capture, represent and display more luminance levels than traditional LDR imagery. A special attention is given to the tone mapping operation.

In Chapter 3, we rst verify the assumption regarding the application of a TMO Introduction designed for still images to an HDR video. We generated several HDR synthetic video sequences to test the behavior of the dierent types of TMO present in the literature.

Those experiments led us to identify six types of temporal artifact that occur when applying separately a TMO to each frame of an HDR video sequence: Global and Local Flickering Artifacts (FA), Temporal Noise (TN), Temporal Brightness Incoherency (TBI), Temporal Object Incoherency (TOI) and Temporal Hue Incoherency (THI).

We then provide a description of the few existing video tone mapping techniques, namely techniques that rely on other frames than the current one to perform the tone mapping. Those methods solve only three out of the six types of described artifact:

Global and Local FA and TN. Finally, we identify two additional types of temporal artifact caused by video tone mapping techniques.

Through the study performed in Chapter 3, we observe that three types of temporal artifact are not yet accounted for. Consequently, we present in Chapter 4 a technique to deal with these three types of artifact. However, a subjective evaluation reported that our method was only ecient when those temporal artifacts are of global nature.

We then modied, in Chapter 5, our technique to make it local. Finally, we conducted a subjective evaluation to evaluate whether reducing those artifacts increases the subjective quality of tone mapped video content.

In Chapter 6, we propose to analyze the relationship between video tone mapping and video compression. More precisely, we study the relation between the compression eciency of tone mapped video content and the preservation of temporal coherency in video tone mapping. We show that the choice of a TMO greatly inuences the compression ratio that a codec can achieve and hence the quality of decoded content for targeted bit-rates.

However, changing the TMO to achieve higher compression eciency amounts to changing the rendering of the tone mapped video. That is why, we propose a technique to increase the compression eciency of any tone mapped video content without altering its rendering. This technique can be adapted to any TMO and is presented in Chapter 7.

Finally, a summary as well as a discussion on future work related to this thesis are presented in Chapter 8.

Contributions

The work presented in this thesis brings the following contributions to the video tone mapping eld:

A state of the art on HDR imagery.

A description of the dierent types of artifact occurring when applying a TMO, designed for still images, to an HDR video.

A survey of video tone mapping.

A corpus of synthetic video sequences specically designed to test TMOs.

Two post-processing techniques to reduce temporal artifacts in video tone mapping.

A survey of tone mapping and video compression.

A technique to increase the compression eciency of tone mapped video content without altering its rendering.

An analysis of most of the current HDR video sequences publicly available.

Publications

Most of the work presented in this thesis is published in the following papers: International Journals R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, "Zonal Brightness Coherency for Video Tone Mapping," Signal Processing: Image Communication, vol. 29, no. 2, pp. 229-246, 2014. High Dynamic Range (HDR) imaging aims at capturing, representing and reproducing the physical value of light [START_REF] Reinhard | High Dynamic Range Imaging, 2nd Edition: Acquisition, Display, and Image-Based Lighting[END_REF], Myszkowski et al., 2008, Banterle et al., 2011, McCann and Rizzi, 2011]. Apart from being able to represent more data with a virtually unlimited bit-depth, HDR imaging will most likely reduce the dierence of rendering between displays. For all those reasons, both academic (COST HDRi 1005 action) and industrial (Technicolor, Dolby, Philips, etc.) scientists strive to bring this technology to the end-consumer. Although HDR concepts allow us to capture and reproduce all the color gamut and luminance perceived by a human observer, the limitations of both capture and display technologies prevent a one to one correspondence between a real scene and its display on a monitor. Furthermore, capture and reproduction are not the only processing required to enjoy a multimedia experience. Figure 2.1 illustrates the many types of signal processing that HDR contents undergo before reaching the consumer. Content creation corresponds to either the capture of a real-word scene or the generation of a synthetic image or video.

International Conferences

Post-processing encompasses all the creation processes to add artistic intent on content, for example color grading or addition of virtual eects. Encoding allows to compress the amount of data required to represent any content, the encoding can be lossless or lossy. The distribution to the consumer can be performed by either storing content on a storage disc (DVD, Blu-Ray, etc.) or by broadcasting it over a network (Digital Television Broadcast, Internet Streaming, etc.). Decoding is the reverse operation of the encoding and consists in reconstruction the video from the bit-stream stored on a storage disc or broadcasted over a network. Finally, the display is the interpretation by a display of the pixel values. This chapter describes briey basic concepts of HDR imaging and is structured as follows:

Section 2.1. Fundamentals: introduces some basic concepts required to understand HDR imaging.

Human Vision System

Photoreceptors

The retina of the human eye includes 130 million cells sensitive to light and called photoreceptors. Photoreceptors, via a process called phototransduction, convert luminous energy (photons) into an electric signal. This signal is transmitted thought the optic nerve and allows the brain to interpret the viewed scene. The photoreceptors respond to the number of incoming photons as well as their wavelengths.

[ [START_REF] Naka | An attempt to analyse colour reception by electrophysiology[END_REF] tted the photoreceptors response as:

R R max = C n C n + C n 50 + b, (2.1)
where R and R max are the neuronal response and maximal response of the photoreceptors to the grating stimuli C, C 50 represents the semi-saturation constant, meaning the value for which half of the upper bound is reached, n is the photoreceptor's sensitivity and b the background discharge. This response corresponds to an S-shaped curve on a log-linear plot, with saturation equal to 0 (because it is impossible to have negative occurrence of a phenomenon) and a maximal intensity.

However, unless by drastically decreasing the sensitivity (i.e. steepness of the slope), this response cannot cover all the wide range of luminance that we can perceive. This is due to the fact that although we can perceive luminance ranging from 10 -6 to 10 6 cd/m 2 , we cannot perceive this range simultaneously. Instead, we need to adapt to our environment.

In order to adapt, two types of photoreceptor cells exit: cones (5 million) and rods (125 million) [START_REF] Reinhard | High Dynamic Range Imaging, 2nd Edition: Acquisition, Display, and Image-Based Lighting[END_REF]. Cones are principally gathered in the center of the retina, also called the fovea. They are responsible for the perception of color, ner details and rapid changes. Three kinds of cone with dierent peak wavelength sensitivities exist:

Long-564 nm (red), Medium-533 nm (green) and Short-437 nm (blue) [Fairchild, 2005].

The colors that we perceive are obtained thanks to the combination of these three types of cones. The dierence in peak wavelength sensitivity means that two packets of light with dierent spectral power distributions can produce the same response, which is called metamerism [Hunt, 2005]. Cones can only operate in daylight (photopic vision ranging from 0.01 to 10 6 cd/m 2 ) and stop functioning when there is not enough light entering the eye.

Rods are highly sensitive to light (i.e. number of photons), they are scattered over Background in High Dynamic Range Imaging the outside fovea and are used in peripheral vision. They are responsible for low light levels (scotopic vision ranging from 10 -6 to 10 cd/m 2 ). As there is only one type of rod sensitive around the blue green wavelength, we experience a lack of color at night. This property, namely the changing of color appearance when illumination changes, is known as the Purkinje eect (a.k.a the Purkinje shift) [Barlow, 1957].

Cones and rods can operate simultaneously in low but not quite dark lighting situations (i.e. when the luminance level ranges between 0.01 and 10 cd/m 2 ). This is called mesopic vision regime.

Threshold versus Intensity (TVI)

Back in 1834, Weber hinted that when adapted to a uniform background, the perception of a variation of intensity is usually linearly proportional to the background intensity. Consider a person carrying a 1 kg weight, if we add 0.1 kg on top of it, the person will perceive the dierence of weight. However, if the initial weight is 10 kg, the adding of 0.1 kg will be hardly noticeable. This relationship is known as Weber's law.

This relation has been applied to the eld of light perception and several experiments were conducted to verify it [START_REF] Blackwell | An analytical model for describing the inuence of lighting parameters upon visual performance[END_REF], Barten, 1992]. These experiments are usually made by increasing the intensity of a gray patch on a uniform gray background.

When the patch becomes visible, the Just Noticeable Dierence (JND) is reached. These experiments shaped the Threshold Versus Intensity (TVI) functions that proved that Weber's law is only true in the range where the photoreceptors are fully responsive (see Figure 2.3). Two equations t the experimental results:

log 10 T s (L w ) =      -2.86 log 10 (L w ) ≤ -3.94 log 10 (L w ) -0.395 log 10 (L w ) ≥ -1.44 (0.405log 10 (L w ) + 1.6) 2.18 -2.86 otherwise (2.2) log 10 T p (L w ) =      -0.72 log 10 (L w ) ≤ -2.6 log 10 (L w ) -1.255 log 10 (L w ) ≥ 1.9 (0.249log 10 (L w ) + 0.65) 2.7 -0.72 otherwise (2.3)
where T s (respectively T p ) represents the TVI response function in the scotopic (respectively photopic) vision regime and L w the luminance [START_REF] Ferwerda | A model of visual adaptation for realistic image synthesis[END_REF].

Colorimetry

Colorimetry is the eld of assigning code values to perceived colors. There are several ways to represent perceived colors using color spaces. A color space is an abstract mathematical representation designed to describe the way color can be represented as a combination of code values (i.e. color components or color channels). Furthermore, the term gamut is used to designate a complete range or scope of a color space.

Many color spaces exist and have dierent purposes. We distinguish two types of color space: full gamut and display-dependent. In 1931, the Commission Internationale de l'éclairage (CIE) dened the standard CIE 1931 XYZ color space which includes cd/m 2 2 8 RGB tri-stimulus values, are highly correlated, Ward proposed to share the exponent between the three color channels. Each pixel is stored in 32 bits, with 8 bits for each of the three color mantissas: R m , G m and B m and another 8 bits for the common exponent E:

R m = 256R 2 E-128 , G m = 256G 2 E-128 , B m = 255B 2 E-128 , with (2.4) E = log 2 (max(R, G, B) + 128 .
(2.5)

This format allows to cover 76 orders of magnitude (i.e. the number of powers of 10 that can be represented). However, this representation does not cover the full gamut of color.

Converting an image to the XYZ color space before computing the shared exponent allows to solve this issue. This format is then referred to as the XYZE format. The RGBE or XYZE format covers a huge range of luminance, more than what the human eye can perceive.

OpenEXR File Format

By redistributing the quantization steps in a more human restricted dynamic range, it is possible to achieve a ner quantization over a more restrictive range. That solution is provided by the half oating point format which is part of the specication of the OpenEXR le format [START_REF] Magic | Openexr[END_REF]. The OpenEXR le format relies on the IEEE 754 16-bit oat standard [Hough, 1981] and is dened as:

H =                0 if (M = 0 E = 0) (-1) S 2 E-15 + M 1024 if E = 0 (-1) S 2 E-15 1 + M 1024 if 1 ≤ E ≤ 30 (-1) S ∞ if (E = 31 M = 0) N aN if (E = 31 M > 0) (2.6)
where S is the sign, M the mantissa and E the exponent. This representation allows to cover around 10.7 orders of magnitude while using 48 bpp (3 • 16 bits: S = 1 bit, E = 5 bits and M = 10 bits). The half-oat pixel representation is used in the new standard Academy Color Encoding System (ACES) which is currently undergoing industry-wide production trials by the major Hollywood studios and standardization by the Society of Motion Picture and Television Engineers (SMPTE).

Other le formats can be used to store HDR values such as the LogLuv encoding [Ward Larson, 1998] that allows to encode HDR values within a TIFF image.

Real-World HDR capture

Recall that HDR images can be generated using a renderer to generate CG content.

However, how to capture an HDR image of a real-world scene? Current commercially available sensors are limited with respect to the amount of light that they can record.

It is usually not enough to recover all the light information present in an outdoor scene.

Background in High Dynamic Range Imaging

In addition, the resulting images use a display-dependent color space that does not represent the absolute physical values of light.

To recover physical values from a camera, one can calibrate a camera. Calibration consists in measuring the Camera Response Function (CRF), that is to say the code value assigned by a camera to a recorded physical value [START_REF] Mann | Being 'undigital' with digital cameras: Extending dynamic range by combining dierently exposed pictures[END_REF]. Once

the CRF is known, it is possible to invert it to obtain the physical values (luminance) of the image.

The second limitation is the inability of current sensors to record all the light information present in a scene. Indeed, sensors cannot capture information smaller or higher than a certain threshold. The exposure Value (eV) setting denes these thresholds and pixels with a luminance lying outside them are clipped. In, photography, exposure is the amount of light per unit area reaching a photographic lm sensor, as determined by shutter speed, lens aperture and scene luminance. Exposure is measured in lux seconds, and can be computed from the eV and scene luminance in a specied region. In photography jargon, exposure generally refers to a single capture for a given shutter speed, lens aperture and scene luminance.

To overcome this limitation, it is possible to capture the same scene several times with dierent exposures. Using the CRF, one can combine those exposures to obtain an HDR image with all the information present in the scene. This processing is called bracketing (Figure 2.5). Several types of bracketing techniques exist:

Temporal bracketing consists in taking images with dierent exposures one after another [Debevec andMalik, 1997, Mitsunaga andNayar, 1999].

Spatial bracketing relies on a neutral lter superimposed onto a sensor to have spatially varying exposure [START_REF] Schoberl | High dynamic range video by spatially non-regular optical ltering[END_REF].

Multi-sensor bracketing uses beam-splitter to divide the light, which results in having dierent exposures with multiple sensors embedded in a single camera [START_REF] Tocci | A versatile HDR video production system[END_REF].

Multi-camera bracketing combines several cameras (usually two) with a rig and dierent neutral gray lters to generate the dierent exposures.

All these techniques have their pros and cons and no solution currently stands out. The common point is that these techniques capture the scene with dierent exposures in order to create an HDR content. However, with the improvement of sensor capabilities, it is highly probable that in the near future, sensors will be able to capture the wide range of visible light and color gamut without having to resort to bracketing techniques.

Display of HDR/LDR Content

CG renderers or bracketing techniques allow to generate HDR content that cover the wide range of physical luminance values of a real world scene. However, to address a display, content needs to be represented in a display-dependent color space, potentially with a much more restricted dynamic range than an HDR scene. Creating HDR content raises two questions:

How to reproduce HDR content on current commercial displays of more restricted capabilities?

Assuming that displays capabilities increase, how to display LDR content on future HDR displays?

This section provides the necessary information to comprehend how to retarget HDR/LDR content to LDR/HDR ones. First, we describe current commercial display capabilities. Then, we detail the HDR to LDR conversion using Tone Mapping Operators (TMOs). Section 2.3.3 describes the future generation of display: HDR displays.

Finally, the LDR to HDR conversion is presented in the last section.

Display Capabilities

As stated before, current displays technology cannot represent the full color gamut and luminance range existing in the real world. Most of those displays use only 8 bits for each of the three color components and can therefore represent 65,025 colors over 255 levels of gray (16,777,216 code values). The way these 3 bytes interact to shape color is dened through a display-dependent color space (Section 2.1.3).

cd/m 2 cd/m 2 γ V L v L v = kV γ , R = R 1 γ , G = G 1 γ , B = B 1 γ .
consequently, the gamma correction is not required anymore if the gamma encoding is designed such as it compensates the non-linearity of a display [Poynton, 1996].

HDR to LDR conversion

During the transition period between HDR and LDR imaging, captured HDR content will still need to be displayable on LDR monitors. Indeed, traditional display will not vanish instantly and hence backward compatibility is mandatory. The conversion from HDR content to LDR ones is performed by Tone Mapping Operators (TMOs). In HDR imaging, the pixels represent the physical scene luminance (expressed in cd/m 2 ) stored as oating point values. In the case of LDR imaging, the pixels are assigned code values corresponding to a display-dependent color space. Furthermore, the term luminance is replaced by luma, which corresponds to the weighted sum of gamma-encoded R G B components of a color video.

In the introduction, we outlined the dierent challenges of tone mapping that are:

the mapping of a wide range of luminances with unlimited bit-depth to a limited amount of code values (255 for 8 bits), the mapping of all colors perceivable by a human observer to a limited bit-depth (65,024 colors for each luma level with 8 bits), the reproduction of this color at a dierent luminance level, the preservation of spatial details (contrast) on a limited bit-depth, the preservation of any artistic intent imposed on the HDR content, the usage of this content for a specic application other than visualization.

Figure 2.7 illustrates the three steps that compose a TMO:

The mapping operation: compresses HDR luminance values to t in the range

[0-1].
The gamma encoding: redistributes the tonal level closer to how our eyes perceive them (see Gamma Encoding in Section 2.3.1).

The quantization: converts oating point values to integer code values corresponding to the used bit-depth (i.e. [0; 2 n -1] for n bits). This operation consists in scaling the gamma encoded values to the maximum value desired (i.e. 2 n -1)

and then rounding them to the nearest integer.

The last two steps (gamma encoding and quantization) can be considered xed although the γ value is tunable. That is why when describing a TMO, these two processes are usually left out. Table 2.1 summarizes the main notations that we use for the dierent steps of tone mapping.

The results of TMOs can be quite dierent since they depend on the targeted applications. Over the last two decades, a multitude of TMOs have been developed

L w cd/m 2 0; +∞ L m 0; 1 L g 0; 1 L d 0; 2 n -1 I w,m,g,d also describe it.
Two edge-aware operators are also detailed: [START_REF] Fattal | Gradient domain high dynamic range compression[END_REF], [START_REF] Durand | Fast bilateral ltering for the display of high-dynamic-range images[END_REF] along with a generic implementation of edge-aware lters for tone mapping. [START_REF] Ledda | A local model of eye adaptation for high dynamic range images[END_REF] operator is detailed as part of the HVS operator.

Then we describe the iCAM06 [Kuang et al., 2007a] as part of CAM operators. Finally we present several subjective evaluations that assess the quality of TMOs. Note that all the presented TMOs have been designed for HDR images, their application to video sequences as well as video TMOs are described in Chapter 3.

Display Adaptive Tone Mapping:

The main limitation of LDR display is their inability to reproduce high luminance values because their peak luminance ranges from 100 to 800 cd/m 2 . In addition, displays can use dierent color gamuts and dynamic ranges. For all those reasons, [START_REF] Mantiuk | Display adaptive tone mapping[END_REF] proposed a TMO that provides the least perceptually distorted LDR image on a targeted display. This TMO adaptively adjusts the rendering of an HDR image based on the display's characteristics. To inverse the display model, several characteristics of the targeted display are needed: the gamma value γ, the peak luminance display l, the black level b, the screen reectivity k and the ambient illumination a. Figure 2.9 illustrates the rendering of 3 predened display types: LCD Oce, LCD Bright and CRT. The characteristics of the predened displays can be found on the documentation page of pfsTMO [Grzegorz Krawczyk, 2007]. This TMO is among the best rated TMO in several subjective evaluations. As it uses a monotonously increasing tone map curve, it is classied as a global TMO.

Photographic Tone Reproduction

Another global TMO well rated by subjective evaluations is the Photographic Tone Reproduction algorithm [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF].

This TMO is based on photographic techniques and allows to choose the exposure of a tone mapped image. It uses a system designed by Adams [Adams, 1981] to rescale HDR images at a dened exposure:

L s = α k L w , (2.9) k = exp 1 n p np x=1 log(d + L w (x)) , (2.10)
where α is the chosen exposure, L w the HDR luminance image and L s the scaled luminance image. The geometric mean k (a.k.a. the key value) is an indication of an lter decomposes an image into several low-frequency subbands with dierent cut-o frequency [Burt, 1981]. Local TMOs usually choose one layer or combine all the layers of a Gaussian pyramid. This operator however, chooses, for each pixel, the layer of the pyramid that best approximates its neighborhood. To achieve that, the dierence between successive layers is normalized and a threshold allows to select the right layer for each pixel. Equation 2.11 is modied to include L a :

d 10 -6 cd/m 2 n p L m = L s 1 + L s 1 + L s ω 2 , ω L m α ω α = 18% ω L s L a L a
L m = L s 1 + L a 1 + L s ω 2 .
(2.12)

Figure 2.10 illustrates both global and local tone mapping results of the same HDR image.

Gradient Domain Compression

In 2002, a new trend in tone mapping appeared: edge-aware tone mapping. Starting from the fact that we are more sensitive to contrast than to absolute values, this type of TMO compresses edges and background dierently.

The Gradient Domain Compression algorithm [START_REF] Fattal | Gradient domain high dynamic range compression[END_REF] performs the tone mapping in the gradient domain. A gradient eld ∆H is computed at each level of a Gaussian pyramid. A scaling factor is then determined for each pixel of each layer based on the magnitude of the gradient:

ϕ k = α ∆H k ∆H k α β , (2.13)
where ∆H k is the gradient eld of the layer k while α determines which gradient magnitude remains unchanged. Gradients of larger magnitude than α are attenuated (assuming that β < 1), while gradients of smaller magnitude are magnied. In [START_REF] Fattal | Gradient domain high dynamic range compression[END_REF], these parameters are set to 0.1 times the average gradient magnitude for α, and to a value between 0.8 and 0.9 for β. The scaling factors are propagated and accumulated from level to level in a top-down fashion. Although these scaling factors are computed using a Gaussian pyramid, they are used only to manipulate the gradients of the ner resolution to prevent halo artifacts. As the modied gradient eld may not be integrable, the LDR image is computed by nding the output image whose gradient is the closest to the modied gradient eld (in a least mean square fashion).

I m I w L I m = α • I w T MO(B l ) + L l=0 β l D l L w , L w B l D l l th D l = B l-1 -B l B 0 = L w I w I m β l α Figure 2
.12: Comparison of Edge-Aware TMOs using only one detail layer. From left to right: bilateral lter [START_REF] Durand | Fast bilateral ltering for the display of high-dynamic-range images[END_REF], weighted least square lter [START_REF] Farbman | Edge-preserving decompositions for multi-scale tone and detail manipulation[END_REF] and recursive lter [START_REF] Gastal | Domain transform for edge-aware image and video processing[END_REF].

gain often adapts to the image to tone map (i.e. normalization factor). Note that using only one detail layer corresponds to using the tone mapping technique described for the bilateral lter (Figure 2.11). Figure 2.12 provides results given by dierent edge-aware TMOs using only one detail layer.

A Local Model of Eye Adaptation Human Visual System (HVS) operators aim at reproducing, on a display with limited capabilities, the perception of a human observer when looking at a real world scene. The goal here is not to provide the best subjective quality but rather to simulate perceptual concepts. Take for example viewing the same scene under a bright or a dim illumination. The color perception of the scene changes when the illumination changes (under dim illumination, perceived colors shifts toward the blue end of the color spectrum due to the rod's peak sensitivity) [Barlow, 1957].

This is called the Purkinje shift and is a perceptual phenomenon, the hue captured by the sensor is not changed . Consequently, when a TMO maps the HDR values to LDR ones, this shift in color is lost.

Another example is how our perception adapts to the wide range of luminance present in the real world. Consider someone walking outdoors on a bright day and entering a dim environment (e.g. a theater). At rst, as he is adapted to the outdoor ambient illumination, he perceives nothing inside the theater. After a while, as he begins to adapt to the dim ambient illumination, his perception of the theater will increase until he is fully adapted. This is called the dark adaptation and the same eect also applies for light adaptation. four main operators deal with these aspects:

Visual Adaptation Model [START_REF] Ferwerda | A model of visual adaptation for realistic image synthesis[END_REF], Time-Dependent Visual Adaptation [START_REF] Pattanaik | Time-dependent visual adaptation for fast realistic image display[END_REF], Local Model of Eye Adaptation [START_REF] Ledda | A local model of eye adaptation for high dynamic range images[END_REF] and Perceptually Based Tone Mapping [START_REF] Irawan | Perceptually Based Tone Mapping of High Dynamic Range Image Streams[END_REF]. Figure 2.13 describes the workow of [START_REF] Ledda | A local model of eye adaptation for high dynamic range images[END_REF] operator. First the HDR image is separated into CIE photopic (Y [Vos, 1978]) and scotopic (Y [Crawford, 1949]) luminance given by: Y = 0.256R + 0.67G + 0.065B, image [START_REF] Ledda | Evaluation of tone mapping operators using a high dynamic range display[END_REF], Kuang et al., 2010]. This evaluation is easier to set up and allows to test TMOs that preserve an artist intent (color grading, contrast enhancement, etc.).

Finally the non-reference method compares several tone mapped images that are ranked by an observer [Drago et al., 2003a], [START_REF] Kuang | Evaluating HDR rendering algorithms[END_REF], [adík et al., 2008], [START_REF] Petit | Assessment of video tone-mapping : Are cameras ' S-shaped tone-curves good enough?[END_REF]. This method is the easiest to set up and amounts to choosing the preferred result without knowing the HDR reference.

No method is free of problems and most of the work performed so far focused only on the subjective quality of the tone mapped content. Furthermore, the result of a TMO is highly correlated to the targeted display as outlined in [START_REF] Mantiuk | Display adaptive tone mapping[END_REF]. That is why dierent studies provide varying results although a general trend can be outlined.

The lack of reliable objective quality metrics for tone mapped content increases the need for subjective evaluation. Unfortunately, the development of objective metrics is usually validated by user's study experiment, hence subjective evaluation of TMO and objective metrics are still an open eld.

HDR Displays

TMOs allow to convert an HDR image into an LDR one displayable on legacy monitor.

This process results in a loss of quality, usually in term of contrast and color reproduction. To avoid using techniques that reduce the quality of the displayed HDR images, Seetzen et al. proposed in [Seetzen et al., 2003, Seetzen et al., 2004] a design to create HDR displays by combining two light modulation devices: a modulated backlight with an LCD panel. The eective dynamic range of the image created is going to be a product of the dynamic range of both modulators. Seetzen et al. proposed two types of backlight that can be used for creating an HDR display. The rst approach employed a digital light processing (DLP) projector producing a modulated backlight that later falls onto the back of an LCD panel [START_REF] Seetzen | A High Dynamic Range Display Using Low and High Resolution Modulators[END_REF]].

In the second design, an hexagonal matrix of individually controlled white Light Emitting Diodes (LEDs) is used to produce the backlight for the LCD panel. A university spin-o company, SunnyBrook Technologies (later known as BrightSide), further developed the technology. The company built a small amount of displays, mostly for the purpose of research and for advancing the technology. Their two most well known displays were the LED based DR37-P and a projector-based SBT1.3. The DR37-P model used 1,395 controlled LEDs to provide a backlight for a 37" LCD display with a resolution of 1920x1080, with an eective contrast of 200,000 : 1 [START_REF] Seetzen | High dynamic range display systems[END_REF].

This technique has been used by Sim2 to build the rst commercially available HDR display, the SIM2-HDR47E that can achieve 4,000 nits (nit is a unit of luminance equivalent to one cd/m 2 ). An evaluation of current HDR displays has been proposed

in [START_REF] Wanat | Physical and Perceptual Limitations of a Projector-based High Dynamic Range Display[END_REF].

LDR to HDR Conversion

The development of HDR displays allows to reproduce a wider variety of scenes with greater delity. However, even HDR displays have limitations (peak luminance, quantization of luminance step, etc.). By matching the physical luminance recorded in a scene to the luminance displayed on a device, disparities between the reproduction of dierent displays should be reduced.

However to display an LDR image or video sequence on an HDR display, its dynamic range needs to be expanded. We distinguish two types of such techniques: Expand Operator (EO) and inverse Tone Mapping Operators (iTMO). EO represents the expansion of LDR content when no information of prior tone mapping has been performed (i.e.

without knowing if the content was originally HDR). On the contrary, an iTMO reconstructs an HDR image or video sequence by performing the inverse operation performed by a TMO. For example, consider an HDR video that has been tone mapped using a TMO. By using information related to the application of the TMO, the iTMO will be able to reconstruct the HDR video sequence with greater delity than the EO. Note that there is no consensus on those two acronyms, it is not unusual to nd articles where both terms have the same meaning. We provide these two denitions because several aspects explained afterwards requires such a dierentiation.

Expand Operators (EO)

An example of an EO is proposed by [START_REF] Akyüz | Do HDR displays support LDR content?[END_REF] where the expansion is computed by:

L w = L p L d -L d,min L d,max -L d,min γ , (2.17)
where L p is the peak luminance of the HDR display, γ is a non-linear scaling factor and L w and L d are the HDR luminance and LDR luma respectively. L d,min and L d,max are the image minimum luma and maximum luma respectively and tting experiments provide γ values of 1, 2.2 or 0.45.

Another EO was designed by conducting two psychophysical studies to analyze the behavior of an EO across a wide range of exposure levels [START_REF] Masia | Evaluation of reverse tone mapping through varying exposure conditions[END_REF]. The authors then used the results of these experiments to develop a gamma expansion technique applied on each LDR color channel C d to obtain the HDR color channel C w :

C w = C d γ , (2.18)
where γ is computed by: that fell on the sensor. The resulting image can then directly be displayed on an HDR monitor.

γ = ak + b = a log(L d,H ) -log(L d,
As will be seen in Section 2.4.2, iTMOs are useful to design backward compatible HDR compression scheme. Note that the loss of information, when performing a TMO-iTMO process, is only due to the quantization step. Without this step and with enough metadata, any TMO is fully invertible without any loss.

Video Compression

Uncompressed video content (HDR or LDR) require too much data to t the storage or broadcast requirements of current video processing pipelines. Table 2.2 summarizes the bit-rates required to store uncompressed LDR or HDR videos as well as the storage capacity of legacy storage discs and targeted digital broadcast bit-rates. The ratio between the broadcaster's bandwidth and uncompressed data bit-rate is of the order of 200. That is why video codecs (coder-decoder) compresses the size of video data to t in the broadcast bit-rates or storage disc capacity.

We distinguish two main types of codec: lossless and lossy. Lossless encoders remove redundant information that can be derived at the decoder side in such a way that no information gets lost. However their compression ratio is quite low (usually below 10). Lossy encoders remove information based on a rate-distortion criterion, say a trade-o between the distortion and the bit-rate. Lossy encoders usually achieve high compression ratio (above 100).

This section aims at giving an overview of dierent codecs and techniques used to compress LDR or HDR video content. First we provide a description of the ITU-T H.265 / MPEG-H Part 2 'High Eciency Video Codec' (HEVC) [START_REF] Sullivan | Overview of the High Eciency Video Coding (HEVC) Standard[END_REF].

Then we present HDR backward compatible compression techniques.

HEVC

HEVC is the successor of the ITU-T H.264 / MPEG-4 Part 10 'Advanced Video Coding' (AVC) codec [START_REF] Wiegand | Overview of the H.264/AVC video coding standard[END_REF]. Developed by the Joint Collaborative Team on Video Coding (JCT-VC), it was released in January 2013 and is reported to double AVC compression ratio. The HEVC test Model (HM) is currently in its version 14.0. HEVC is a block-based codec that exploits both spatial and temporal correlations between the code values of the pixels to achieve a high compression ratio. Figure 2.16

gives an overview of the dierent processes and their relationship inside a block-based encoder. To exploit these correlations, blocks are predicted using two types of prediction: Intra or Inter. Intra prediction relies on spatial correlation to predict the current block using blocks already decoded in the current frame. Inter prediction exploits the temporal correlation by predicting the current block using blocks from a set of previous/subsequent decoded frames. Using a rate-distortion function, the best predictor is selected among the Inter and Intra prediction. The predicted block is then subtracted from the original block to obtain the residual blocks.

To encode the residuals, the current block is rst converted to the frequency domain using a frequency transform (T, i.e. discrete cosine transform or discrete sine transform).

The resulting harmonic coecients are then quantized (Q) before being fed to the entropy coder. The bitstream is composed of the encoded residuals and the prediction side-data to reproduce the same prediction on the decoder side. Note that in oder to use the same pixels value on the decoder and encoder side, the blocks are reconstructed on the encoder side to be used for the prediction process of future blocks.

As this thesis deals with temporal coherency, a detailed explanation of the Interprediction is given hereafter. To predict the current block, a block-based motion estimation is performed to nd the motion vector pointing to the best temporal predictor. The best predictor is the block that minimizes the distortion with the current block to be encoded. An example of distortion metrics is the Sum of Absolute Dierences (SAD)

or the Mean Square Error (MSE). This motion estimation is performed for dierent block-sizes (from 64 to 4 organized in a quad-tree in the HM). The selected motion vector is the one that minimizes a rate-distortion function. The Inter-prediction results from performing a motion compensation on a reference frame using the selected motion vectors. In video compression, it is generally considered that the closest the prediction, the more ecient the compression. the artistic intent of the tone mapped content. [Mantiuk et al., 2006a] proposes a technique that takes as input both an HDR version and a color-graded LDR version of the video. The TMO is computed so as to minimize the distortion with the LDR reference provided. This model allows to regain some sort of control on tone mapped content, but it is less optimal in term of compression eciency. That is why, the new trend in HDR video compression is based on a perceptual curve, also called inverse Electro-Optic Transfer Function(EOT F -1 ), that transforms HDR sequence (oating-point data) into a high bit-depth representation (integer data with quantization step not visible by the human eye) [START_REF] Mantiuk | Perception-motivated high dynamic range video encoding[END_REF], Mantiuk et al., 2006b, Motra and Thoma, 2010, Miller et al., 2013, Kunkel et al., 2013]. The generated video is directly compressed using a single high-bit depth codec. At the decoder side, the decoded sequence is computed by the inverse of the perceptual curve to reconstruct the HDR sequence.

Backward compatibility is achieved by simulcast, say providing two streams without joint compression.

Summary

In this chapter, we rst introduced the fundamentals to understand how the human visual system reacts to light (Section 2.1). We then described in Section 2.2 the motivations behind HDR imaging and how both bracketing techniques and CG renderers can capture/generate more information than that can be displayed on current display devices. In Section 2.3, we presented the current technology to build HDR display along with techniques that transform HDR/LDR content to address LDR/HDR displays. Finally in Section 2.4, we presented general video compression schemes, both for HDR and LDR content, required to distribute these video to the end-users. To sum up, we Background in High Dynamic Range Imaging described all the mandatory steps of an HDR pipeline from the content generation to the end-consumer's display (Figure 2.1).

Chapter 3 Video Tone Mapping

In the previous chapter, we introduced tone mapping and dierent TMOs designed to tone map HDR images (Section 2.3.2). Due to the lack of high quality HDR video content, the temporal aspect of tone mapping has been dismissed for a long time. However, with recent developments in the HDR video acquisition eld [START_REF] Tocci | A versatile HDR video production system[END_REF], Kronander et al., 2013], more and more HDR video content are now available. That is why we propose in this chapter to evaluate the temporal behavior of the dierent types of TMO designed for still images. We will show that naively applying a TMO to each frame of an HDR video sequence leads to dierent types of temporal artifact that we classify in six categories. Then we will describe video TMOs, that is to say TMOs that rely on information outside the current frame to tone map it. We will show that only three out the six types of artifact encountered can be solved by these techniques. Finally, we describe two new types of temporal artifact that are introduced by video TMOs. This chapter is structured as follows: Section 3.1. Temporal Artifacts: describes the dierent types of temporal artifact along with their main causes. Section 3.2. Video TMOs: presents techniques to reduce temporal artifacts by extending TMOs to more than one frame. Section 3.3. Temporal Artifacts caused by Video TMOs: details how these techniques, intended to reduce artifacts, entail other types of temporal artifact.

Temporal Artifacts

In this section, we describe temporal artifacts encountered when applying naively a TMO to each frame of an HDR video sequence. We propose to classify these artifacts into several categories: overall brightness has changed because the relative area of the sky in the second frame is smaller, hence reducing the chosen normalization factor (99 th percentile).

Local Flickering Artifacts correspond to the same phenomenon as its global counterpart but on a reduced area. They appear mostly with TMOs that map a pixel based on its neighborhood. Small changes of this neighborhood in consecutive frames may result in a dierent mapping. Edge-aware TMOs are particularly prone to such artifacts as they decompose an HDR image into a base layer and one or more detail layers.

As each layer is tone mapped independently, a dierence in the ltering in successive frames results in local ickering artifacts. The top row of Figure 3.3 represents a zoom on a portion of the computed base layer of 3 successive frames. Note how the edges are less ltered out in the middle frame compared to the other two. Applying the bilateral lter [Tomasi andManduchi, 1998, Durand and[START_REF] Durand | [END_REF] operator results in a local ickering artifact in the tone mapped result (bottom row).

Figure 3.4: Example of temporal noise amplication due to the application of an edge-aware TMO (left, [START_REF] Gastal | Domain transform for edge-aware image and video processing[END_REF]) compared to a global TMO (right, [START_REF] Drago | Adaptive Logarithmic Mapping For Displaying High Contrast Scenes[END_REF]).

Temporal Noise

The second type of temporal artifact presented here, namely Temporal Noise (TN), is a common artifact occurring in digital video sequences. Noise in digital imaging is mostly due to the camera and is particularly noticeable in low light conditions. On images, camera noise has a small impact on the subjective quality, however, for video sequences, its variation over time makes it more noticeable. It is the reason why denoising algorithms are commonly applied to video sequences to increase their subjective quality.

As most TMOs aim at reproducing minute details, they struggle to distinguish information from noise. Consequently most of current TMOs increase the noise rather than reducing it. Edge-aware TMOs are particularly prone to such artifacts as they enhance details while compressing more harshly a ltered version of the original image.

An example of temporal noise enhanced by the application of an edge-aware TMOs is illustrated in Figure 3.4.

Temporal Brightness Incoherency

Temporal Brightness Incoherency (TBI) artifacts occur when the relative brightness between two frames of an HDR sequence is not preserved during the tone mapping.

As a TMO uses for each frame all its available range, the temporal brightness relationship between frames is not preserved throughout the tone mapping operation. Consequently, frames perceived as the brightest in the HDR sequence are not necessarily the brightest in the LDR one.

For example, TBI artifacts appear when a change of illumination condition in the HDR sequence is not preserved during the tone mapping. Consequently, temporal information (i.e. the change of condition) is lost, which changes the perception of the scene (along with its artistic intent). Figure 3.5 illustrates a TBI artifact by plotting the geometric mean of both HDR and LDR sequences. Note that, although the geometric mean greatly varies in the HDR sequence, it remains stable in the LDR one. This is due to the fact that a TMO searches for the best exposure for each frame. As it False color luminance (top row) and tone mapped result using [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] operator (bottom row). Both frames appear at the same level of brightness although the false color representations indicate dierent levels.

Video TMOs

Applying a TMO naively to each frame of a video sequence leads to temporal artifacts.

The aim of Video TMOs is to prevent or reduce those artifacts. Video TMOs rely on information outside the current frame to perform their mapping. So far, Video TMOs are techniques that extend or post-process TMOs designed for still images to tone map HDR video sequences. These techniques can be divided in 3 categories: For each category, we provide a description of the general technique along with dierent state of the art references.

Global Temporal Filtering

Global temporal ltering aims at reducing global ickering artifacts (Section 3.1.1).

Two main approaches have been formulated so far: ltering the tone map curve or the parameters of a TMO.

Filtering the Tone Map Curve Recall that global operators compute a monotonously increasing tone map curve (Section 2.3.2). Abrupt changes of this curve in successive frames results in ickering artifacts. By applying a temporal lter to these tone map curves, global ickering artifacts can be reduced. This ltering is usually performed in two passes: one to compute a tone map curve per frame and a second to lter those curves temporally.

The Display Adaptive [START_REF] Mantiuk | Display adaptive tone mapping[END_REF] operator is able to perform such a temporal ltering on the nodes of a computed piece-wise tone map curve. The eciency of this ltering is illustrated in Filtering TMO's Parameters Most global TMOs rely on parameters to compute their tone map curve. These parameters usually correspond to image statistics that tend to be unstable over time (e.g. the 99 th percentile, the geometric mean, etc.).

For example, the Photographic Tone Reproduction [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] operator relies on the geometric mean (also called key value) to scale an HDR image to the best exposure. One temporal extension of this operator lters this key value along a set (a) [START_REF] Mantiuk | Display adaptive tone mapping[END_REF] operator without the temporal ltering, (pfstmo option = -d pd=lcd_oce [Grzegorz Krawczyk, 2007]).

(b) Histogram and piece-wise tone map curve without the temporal ltering.

(c) Histogram and piece-wise tone map curve with the temporal ltering.

(d) [START_REF] Mantiuk | Display adaptive tone mapping[END_REF] operator with the temporal ltering. sequence. An oset is added to avoid an overlap between the curves, the smoothing eect of both techniques ( [START_REF] Kang | High dynamic range video[END_REF] and [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF]) are compared to [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] operator.

of previous frames [START_REF] Kang | High dynamic range video[END_REF]. As a consequence, this method smooths abrupt variations of the frame key value throughout the video sequence. This technique is capable of reducing ickering for sequences with slow illumination variations. However, for high variations, it fails because it considers a xed number of previous frames. That is why, [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] proposed a method that adapts dynamically this number.

The adaptation process depends on the variation of the current frame key value and that of the previous frame. Moreover, the adaptation discards outliers using a min/max threshold. This solution performs better than [START_REF] Kang | High dynamic range video[END_REF] and for a wider range of video sequences. The computed key values for both of these techniques and the original algorithm are plotted in Figure 3.11. The green curve ( [START_REF] Kang | High dynamic range video[END_REF])

smooths every peak but also propagates them to successive computed key values. The red curve however ( [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF]) reduces the abrupt changes of the key value without propagating it.

Another temporal extension of the Photographic Tone Reproduction operator has been proposed in [START_REF] Kiser | Realtime Automated Tone Mapping System for HDR Video[END_REF]. It rst modies the scaling of the HDR frame (Equation 2.9) to perform the temporal ltering:

L s = • 2 2(B-A)/(A+B) k L w = a k L w (3.1)
where A = L max -k and B = k-L min with L max and L min the maximum and minimum value of L w . k corresponds to the key value (a.k.a. the geometric mean). The temporal ltering consists of a leaky integrator applied to the three variables (a, A and B) used to tune the tone mapping:

v t = (1 -α v )v (t-1) + α v v t (3.2)
where v t represents any of the three variables at time t and α v is a time constant giving the strength of the temporal ltering. The tone mapping equation remains the same (Equation 2.11 in Section 2.3.2).

Many other TMOs lter their parameters temporally including [START_REF] Pattanaik | Time-dependent visual adaptation for fast realistic image display[END_REF], [START_REF] Durand | Interactive tone mapping[END_REF], [START_REF] Ledda | A local model of eye adaptation for high dynamic range images[END_REF], [START_REF] Irawan | Perceptually Based Tone Mapping of High Dynamic Range Image Streams[END_REF] and [Van Hateren, 2006]. Most of them either aim at simulating the temporal adaptation of the HVS or at reducing global ickering artifacts.

Local Temporal Filtering

Global temporal ltering cannot apply to local TMOs as such operators rely on a spatially varying mapping function. As outlined in Section 3.1.1, local changes in a spatial neighborhood cause local ickering artifacts. To prevent these local variations of the mapping along successive frames, video tone mapping operators can rely on a pixel-wise temporal ltering.

For example, the Gradient Domain Compression operator [START_REF] Fattal | Gradient domain high dynamic range compression[END_REF] has been extended by [START_REF] Lee | Gradient domain tone mapping of high dynamic range videos[END_REF] to cope with videos. Recall that this operator computes the LDR result by nding the output image whose gradient eld is the closest to a modied gradient eld. [START_REF] Lee | Gradient domain tone mapping of high dynamic range videos[END_REF] propose to add a regularization term which includes a temporal coherency relying on a motion estimation:

x,y

∆L d (x, y, t) -G(x, y) 2 + λ x,y L d (x, y, t) -L d (x + δx, y + δy, t -1) 2 (3.3)
where L d is the output LDR luma at the precedent or current frame (t -1 or t) and G is the modied gradient eld. The pairs (x, y) and (δx, δy) represent respectively the pixel location of a considered pixel and its associated motion vectors. The parameter λ balances the distortion to the modied gradient eld and to the previous tone mapped frame.

Another operator [START_REF] Ledda | A local model of eye adaptation for high dynamic range images[END_REF]] performs a pixel-wise temporal ltering. However the goal with this operator is to simulate the temporal adaptation of the human eye on a per-pixel basis. Besides increasing the temporal coherency, pixel-wise temporal ltering also has denoising properties. Indeed, many denoising operators rely on temporal ltering to reduce noise [START_REF] Brailean | Noise reduction lters for dynamic image sequences: a review[END_REF]. Performing such a ltering during the tone mapping allows to keep the noise level relatively low.

Detection and Reduction of Artifacts by Post-Processing

The techniques presented so far in this section focus on preventing temporal artifacts (mostly ickering) when tone mapping video sequences. These a priori approaches consist in either preprocessing parameters or modifying the TMO to include a temporal ltering step. Another trend analyzes a posteriori the output of a TMO to detect and reduce temporal artifacts.

One of these techniques ( [START_REF] Guthier | Flicker reduction in tone mapped high dynamic range video[END_REF]) rst detects ickering artifacts by assuming that they are of global nature. An artifact is detected if the overall brightness Figure 3.12: Results of [START_REF] Farbman | Edge-preserving decompositions for multi-scale tone and detail manipulation[END_REF] operator without (top row) and with (bottom row) [START_REF] Guthier | Flicker reduction in tone mapped high dynamic range video[END_REF] post-processing.

dierence (a.k.a. the geometric mean) between successive frames of a video sequence is greater than a brightness threshold (dened using either Weber's law [Ferwerda, 2001] or Steven's power law [START_REF] Stevens | Brightness Function : Eects of Adaptation[END_REF]). As soon as an artifact is located, it is reduced using an iterative brightness adjustment until reaching the brightness threshold.

Note that this technique performs an iterative brightness adjustment using oating point values to avoid loss of signal due to clipping and quantization. Consequently, the TMO's implementation needs to embed the ickering reduction operation before the quantization step. This technique relies only on the output of the TMO and hence can be applied to any TMO. Figure 3.12 illustrates the reduction of a global ickering artifact when applying the post-processing proposed in [START_REF] Guthier | Flicker reduction in tone mapped high dynamic range video[END_REF].

Temporal Artifacts caused by Video TMOs

In the previous section, we presented solutions to reduce temporal artifacts when performing video tone mapping. These techniques target mostly the ickering artifact as it is considered one of the most disturbing one. However, these techniques can generate new temporal artifacts: Temporal Contrast Adaptation (TCA) and Ghosting Artifacts (GA).

Temporal Contrast Adaptation

To reduce global ickering artifacts, many TMOs rely on global temporal ltering.

Depending on the used TMO, the lter is either applied to the computed tone map curve [START_REF] Mantiuk | Display adaptive tone mapping[END_REF] or to the parameter that adapts the mapping to the image [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF], Kiser et al., 2012]. However, when an actual change of illumination occurs, as presented in Section 3.1.3, it also undergoes temporal ltering. Conse- quently, the resulting mapping does not correspond to any of the conditions but rather to a transition state. We refer to this artifact as Temporal Contrast Adaptation (TCA). These artifacts also occur when performing a post-processing to detect and reduce artifacts as presented in Section 3.2.3. Indeed, [START_REF] Guthier | Flicker reduction in tone mapped high dynamic range video[END_REF] technique only relies on the LDR results to detect and reduce artifacts. If one has no information related to the HDR video then a change of illumination suppressed by a TMO cannot be anticipated nor predicted.

Ghosting Artifacts

Similarly to global temporal ltering, local temporal ltering generates undesired temporal artifacts. Indeed, pixel-wise temporal ltering relies on a motion eld estimation which is not robust to change of illumination conditions and object occlusions. When the motion model fails, the temporal ltering is computed along invalid motion trajectories which results in Ghosting Artifacts (GA). Figure 3.14 illustrates a ghosting artifact in two successive frames resulting from the application of [START_REF] Lee | Gradient domain tone mapping of high dynamic range videos[END_REF] operator. This artifact proves that pixel-wise temporal ltering is only ecient for accurate motion vectors. When a motion vector associates pixels without temporal relationship, ghosting artifacts will occur. Those "incoherent" motion vectors should be accounted for to prevent ghosting artifacts as these have been shown to be the most disturbing type of artifact [START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR-Video[END_REF].

Summary

In this chapter, we proposed a classication in six types of temporal artifact that occur when applying separately a TMO to each frame of an HDR video sequence. Only two type of artifact had been outlined and addressed in state of the art work (global and local ickering). Then, we described Video TMOs that extend TMOs design for still images to the temporal domain. We have shown that these techniques could generate two new types of temporal artifact. Furthermore, recent subjective evaluations have been performed on video tone mapping [START_REF] Petit | Assessment of video tone-mapping : Are cameras ' S-shaped tone-curves good enough?[END_REF], Melo et al., 2013, Eilertsen et al., 2013]. They all draw the same conclusion that no solution yet exists and that further research in video tone mapping is required. Chapter 4

Global Brightness Coherency

In the previous chapter, we described why applying naively a TMO to an HDR video sequence leads to temporal artifacts. As summarized in Table 3. describes when the BC post-processing fails to preserve temporal brightness coherency.

Temporal Contrast and Number of Tonal Levels

We dene the temporal contrast as the ratio between the highest and the lowest luminance value in a video sequence. However this metric is not very meaningful if the number of dierent tonal values between the highest and lowest luminance values is not accounted for.

For HDR imaging, the contrast and the associated number of tonal level is limited by the used representation format (RGBE, half-oat, etc.). The spatial contrast of each frame is the ratio between the maximum and minimum luminance values of this frame, while the temporal contrast of a video is the ratio between the maximum and minimum luminance values of all the frames of the video. Obviously the temporal contrast is always greater or equal to the spatial contrast. Figure 4.1 illustrates the dierence between temporal and spatial contrasts in a video sequence. In this gure, the maximum spatial contrast is 6.4 orders of magnitude while the temporal one is 8.4 orders of magnitude. Note that we computed those contrasts using the 1 % and 99 % percentile (to remove outliers values).

For LDR imaging, the spatial and temporal contrasts for a given display is xed and depends on the white and black points of the display (Section 2.3.1). The number of tonal levels depends on the bit-depth n, which corresponds to the size of a quantization bin q (the width of each tonal value):

q = L d,max -L d,min 2 n -1 , (4.1)
where L d,max and L d,min correspond to the white and black points of the display respectively. When capturing an image with a wide range of luminance, the contrast limitation results in over-exposed (too much light hitting the sensor) and/or under-exposed (not enough light) pixels. The amount of over/under exposed pixels depends on the exposure (eV setting) of a camera. For still images, this parameter is set to maximize the number of well-exposed pixels, that is to say to have the nest granularity for the biggest amount of pixels. Additionally, the exposure is also often used to produce artistic eects such as all white or all black backgrounds. In other words, setting the exposure can be seen as a form of tone mapping of a real scene to get an LDR image.

When capturing a video, setting the exposure is more dicult as it is hard to fore- prone to TBI artifacts with the only dierence that the mapping is performed between the real scene and the LDR picture. To better understand this artifact, we simulated a camera trying to capture the Desert sequence (Appendix A.3) using the CameraTMO [START_REF] Petit | Assessment of video tone-mapping : Are cameras ' S-shaped tone-curves good enough?[END_REF]. We plotted the used dynamic range in Figure 4.2 along with several frames of the tone mapped result. We observe that the variations of the computed key values are high and that the perception of the scene is not at all consistent over the video (going from over exposed in frame 10 to really dark in frame 150). Temporal brightness incoherency usually appears only with user-generated content that use per default the automatic mode.

For professional content, the camera's exposure is xed for a specic shot, which ensures temporal brightness coherency. If the temporal contrast is too high, insertion of "cuts" provides a transition between two exposure settings. Furthermore, over/under exposed pixels undergo color grading at the post-production stage. This process can be compared to manually tone mapping each frame of an HDR sequence to achieve the best chosen representation of the video.

Although temporal exposure adaptation or even xing the exposure settings per scene can lead to satisfying results, they are not well suited to extreme changes in illumination as the capture is still a limitation. However, capturing an HDR video with a high temporal contrast is not an issue thanks to bracketing techniques but its tone mapping is still an issue since the targeted number of tonal levels and contrast are too limited to well depict it. Existing TMOs mimic the exposure adaptation of LDR capture device, hence resulting in temporal brightness incoherency. As a TMO uses all the available range for each frame, the HDR brightness coherency is not preserved throughout the tone mapping operation. Consequently, frames perceived as the brightest in the HDR sequence are not necessarily the brightest in the LDR one. To solve this issue, we propose a technique, called Brightness Coherency (BC), that aims at preserving the temporal brightness coherency while still beneting from the increased detail reproduc-

k t f t k t f (L t ) = exp 1 n p np x=1 log(d + L t (x)) , L t t th d n p χ χ
We denote k v (L w ) the geometric mean of the HDR anchor frame and k v (L m ) the geometric mean of the corresponding tone mapped frame. Note that the video analysis is performed prior to the tone mapping and we need to compute the key value of the anchor frame in the LDR domain k v (L m ) during the video analysis (which is a preprocessing step).

Now that both k v (the anchor's geometric means) are known, we perform the BC post-processing on each frame of the tone mapped sequence. As we want to preserve the relative dierence of brightness between each frame and the anchor, the post-processed LDR luminance should satisfy the following equation:

k t f (L t w ) k v (L w ) = k t f (L t bc ) k v (L bc ) (4.3)
To make this ratio hold, we post-process the tone mapped luminance L t m to obtain the post-processed tone mapped luminance L t bc :

L t bc = ζ + (1 -ζ) k t f (L t w ) • k v (L m ) k v (L w ) • k t f (L t m ) L t m = s t • L t m (4.4)
where s t represents the scale ratio of the t th frame and ζ is a user-dened parameter to avoid low scale ratio (default value 0.1). The inuence of ζ is detailed in Section 4.3.1.

The remaining of the tone mapping is performed as before (gamma encoding, scaling and quantization to integers). Note that although the L t m values are supposed to be in the range [0;1], values lying outside this range are not clipped until the gamma encoding.

The consequence is that, for scale ratio below 1, pixels that would have been clipped

during the tone mapping still contain useful information. Similarly, well-exposed pixels may be clipped when the scale ratio is above 1.

Results

The BC technique aims at preserving the overall temporal brightness coherency of the HDR sequence in the LDR one. This attribute allows our technique to handle artistic eects such as fading that other TMOs cannot cope with. We experimented with dierent HDR sequences (Desert, UnderBridgeHigh, MtStMichelWhite and MtStMichelBlack detailed in Appendix A) to assess the eciency of our method in terms of: reduction of Temporal Brightness Incoherency (TBI) artifacts, reduction of Temporal Ob ject Incoherency (TOI) artifacts, preservation of fading eects.

We will discuss each of these in detail in the following sections.

Temporal Brightness Coherency

Preserving the relative levels of brightness throughout a video is the main goal of our technique. We have applied the [START_REF] Drago | Adaptive Logarithmic Mapping For Displaying High Contrast Scenes[END_REF] operator to tone map the Desert sequence with and without the BC post-processing. Figure 4.4 plots the geometric mean of the HDR and tone mapped video sequences. Improved temporal brightness coherency is achieved especially between the beginning (which should be very dim) and the end (which should be bright) of the sequence. Note that a global ickering artifact occurs between frame 248 and 249, which is partly reduced thanks to the BC post-processing.

Figure 4.5 illustrates the inuence of χ by providing the tone mapped results of the four frames presented in Figure 4.1 and 4.2. Recall that χ is a user-dened parameter to select the anchor frame depending on its geometric mean. The HDR geometric mean k t f of those frames, from left to right, is: 0.9205, 630, 440 and 1070 cd/m 2 . When the BC post-processing is not used, all frames appear of similar overall brightness.

Furthermore, the dimmest frame (right frame) is actually the brightest one in the HDR sequence. When the BC is used with χ='Max', the tone mapped brightest HDR frame is not changed while all the other frames are dimmer. Coherency is improved, however only few details are visible in the leftmost frame. When using the median geometric mean as anchor (χ='Median'), more details appear in the leftmost frame but some pixels are over-exposed in the 2 nd and last frame.

The other parameter of the BC post-processing is ζ, which prevents low scale ratio.

Its eect are illustrated in Figure 4.6, where its value ranges from 0.01 to 1. Too low a ζ will result in low details reproduction for dim HDR frames while a too high one will lessen the eect of the BC post-processing. Eectively, this parameter controls the trade-o between reproduction of spatial details and temporal brightness coherency. Tone mapped results using [START_REF] Drago | Adaptive Logarithmic Mapping For Displaying High Contrast Scenes[END_REF] operator without (top-row) and with the BC post-processing (ζ = 0.05, 2 nd row χ='Max' and bottom-row χ='Median'). By using the BC post-processing, the overall brightness coherency is preserved. Left: [START_REF] Pattanaik | Time-dependent visual adaptation for fast realistic image display[END_REF] and [START_REF] Mantiuk | Display adaptive tone mapping[END_REF] with and without temporal ltering. Right: [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF], [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] without and with the BC post-processing.

Temporal Object Coherency

Temporal Ob ject Incoherency (TOI) occurs when an object's brightness, which was stable in the HDR sequence, varies in the LDR one. If we consider a small area visible in successive frames with constant HDR luminance value, the luma of the tone mapped area should have very low variations for the perception of the object to be coherent.

To assess the performance of the BC post-processing, we tone mapped the Under-BridgeHigh sequence using dierent TMOs. We then considered a pixel with the same HDR luminance value along 28 frames. Figure 4.7 shows the evolution of this pixel for all the TMOs tested. The pixel's luma variations are high with all the used TMOs while our technique keeps these variations very low.

Video Fade

A video fade occurs when a shot gradually fades to (or from) a single color, usually black or white. In LDR video processing, a fade to black is computed using an alpha-blending:

R t = α t O t + (1 -α t )F, (4.5)
where O t is the t th original frame of the sequence, F a black image and R t the t th resulting fade to black frame. α t ranges from 1 to 0 and controls the speed of the fading. We consider the speed of the fading as the number of intermediate frames n t needed to satisfy R t = F.

Although video fade eects are well known in LDR video processing, HDR video fade has not been addressed yet. This is due to the fact that HDR luminance represents [START_REF] Pattanaik | Time-dependent visual adaptation for fast realistic image display[END_REF] [ [START_REF] Mantiuk | Display adaptive tone mapping[END_REF] [ [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] [ [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] [ [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] with BC The upper image corresponds to frame 25 while the lower one to frame 35. a photometric quantity expressed in cd/m 2 , while LDR luma represents a code value relative to a display-dependent color space. However, a black or white point is not dened in HDR. For this reason, we dene the white point as the maximum HDR luminance value of the image and the black point its minimum. To make the video fade eect perceptually linear, we apply Equation 4.5 in the log domain.

To preserve video fade eects, the parameter ζ (Equation 4.4) must be set to 0. To deal with fade to black or fade to white, the anchor must be dened as the brightest (χ="Max") or the dimmest (χ="Min") frame respectively. A fading can also be detected during the video analysis using a fade detection technique [Alattar, 1997]. Video TMOs cannot handle video fade eects as they do not analyze the video before performing the tone mapping.

In this section, we show some results obtained when applying the BC algorithm on video fade sequences. Figure 4.9 illustrates the MtStMichelBlack sequence in false color as well as after applying [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] operator with and without the BC post-processing. Without post-processing, the TMO fails to reproduce the fade to black eect. Figure 4.10 shows the results of tone mapping a fade to white sequence. Similarly to the fade to black, the video fade eect is preserved only if the BC post-processing is applied.

Limitations of the BC Post-Processing

The BC post-processing was evaluated in [START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR-Video[END_REF], Melo et al., 2013] and was proven to have less temporal artifacts than other TMOs. However, these evaluations also reported that the provided results were often too dim which lessened their subjective quality. Indeed, the range associated with each frame corresponds to a portion of the Figure 4.9: MtStMichelBlack video sequence represented in false color representation (top-row) and after applying [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] operator without (middle-row) and with the BC post-processing (bottom-row). Black fade eect illustrated through frames 1, 33, 66 and 99 (from left to right). (top-row) and after applying [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] operator without (middle-row) and with the BC post-processing (bottom-row). White fade eect illustrated through frames 1, 33, 66 and 99 (from left to right). available dynamic range (this portion being dened by the scaling). Figure 4.11 plots the used dynamic range of a tone mapped sequence with and without using the BC post-processing (left). Without the BC, the maximum LDR luma remains close to its maximum value (i.e. 255). On the other hand, the BC technique reduces the used dynamic range up to 2/5 of its maximum value. The dierence of results between the TMO alone (top frames on the right) and the TMO associated with the BC postprocessing (bottom frames on the right) illustrates this reduced contrast. The rightmost frames correspond to the anchor frame used, which is not aected by the BC postprocessing. On the contrary, as the BC only preserves the overall brightness coherency, the sky's brightness in the bottom left frame is scaled down although it is the brightest element in the video sequence. From this example, we see that preserving only the global brightness coherency not only reduces the spatial contrast but also impairs the reduction of Temporal Ob ject Incoherency (TOI) artifacts.

Summary

In this chapter, we presented a post-processing method to preserve the overall HDR brightness coherency in a tone mapped sequence. Our method relies on an anchor based on the geometric mean to scale each frame of a tone mapped video sequence.

Two parameters tune the post-processing, χ to choose the anchor and ζ to prevent a too low scale ratio. We provided results regarding the reduction of TBI and TOI artifacts as well as the preservation of artistic eects such as fading. This technique has been published in: R. Boitard, K. Bouatouch, R. Cozot, D. Thoreau and A. Gruson, "Temporal Coherency for Video Tone Mapping," in Proc. SPIE, Applications of Digital Image Processing XXXV, 2012.

The BC post-processing was evaluated in [START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR-Video[END_REF], Melo et al., 2013] and was proven to have less temporal artifacts than other TMOs. However, these evaluations also reported that the provided results were often too dim which lessened their subjective quality. Indeed the BC post-processing trades o spatial contrast to increase the temporal brightness coherency. As this technique deals only with the overall brightness, each pixel of each frame is scaled down without discrimination. In other terms, no pixel, apart from those belonging to the anchor frame, is mapped to the maximum of the dynamic range. Consequently, the range associated with each frame corresponds to a portion of the available dynamic range (this portion being dened by the scaling).

Chapter 5 Zonal Brightness Coherency

In the previous chapter, we presented the Brightness Coherency (BC) post-processing that aims at preserving the temporal brightness coherency. Similar to the method proposed in [START_REF] Farbman | Tonal stabilization of video[END_REF], the BC technique performs well when the brightness uctuations in a scene change in a global way. However, for local uctuations, this technique scales similarly each pixel of a frame, resulting in a lack of spatial contrast due to the reduced dynamic range as pointed out in [START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR-Video[END_REF]. To overcome this problem, we propose to make the BC local by relying on a zonal system. The resulting Zonal Brightness Coherency (ZBC) post-processing is presented in this chapter that is organized as follows:

Section 5.1. Global and Local Post-Processing: describes the cause of the reduced spatial contrast when using the BC and proposes a solution to overcome it.

Section 5.2. Zonal Brightness Coherency (ZBC) Post-Processing: describes the modication required to make the BC post-processing local.

Section 5.3. Results: presents results regarding the preservation of temporal brightness object and hue coherency, it also reports a subjective evaluation to assess the dierence in subjective quality between no post-processing or the use of the BC/ZBC.

Global and Local Post-Processing

In the BC post-processing, the global brightness coherency is ensured by matching the LDR key value ratio with the HDR one (Equation 4.4). However using only one key value per frame is not always enough as shown in Figure 5 in mind, we propose in the next section, to preserve the local brightness coherency using key values per zone rather than per frame.

Zonal Brightness Coherency Post-Processing

In the previous section, we outlined the issues related to the BC algorithm. They are due to the use of only one key value k f per frame that represents only an indication of a frame's overall brightness. However, preserving the overall brightness coherency does not preserve local brightness coherency.

In order to improve this aspect, we propose to apply the BC algorithm to zones rather than to a whole frame. To this end, a histogram-based segmentation algorithm divides each frame into segments in the luminance domain. As the segment's boundaries change from frame to frame, ickering artifacts may appear as detailed in Section 5.2.1. To prevent ickering, we compute video zones based on the segments' key values. We dene each video zone by two luminance values kept constant throughout the video sequence.

The frame segmentation as well as the video zones determination are performed during the video analysis step. To apply the BC per zone, we compute a scale ratio per zone instead of per frame. As in the BC algorithm, the ZBC scale ratio is used to post-process any TMO's output.

The proposed decomposition of each frame in segments is similar to [START_REF] Krawczyk | Lightness Perception in Tone Reproduction for High Dynamic Range Images[END_REF] which divided an image in frameworks based on the its histogram. However, [START_REF] Krawczyk | Lightness Perception in Tone Reproduction for High Dynamic Range Images[END_REF] was looking for the brightest area in each framework to have a reference white and did not deal with videos. The steps of our solution are described in detail in the following sections.

Frame Segmentation

Segmenting a frame is usually done either in the spatial or the luminance domain. We chose to segment the HDR frames in the luminance domain for the following reasons.

First, we want to preserve the spatial brightness coherency. By using the luminance, we make sure that two spatially distant areas of same luminance have the same scale ratio. Second, as spatial segmentation is never perfect, a scale ratio, not related to the luminance of a given pixel, could be used because of its spatial neighborhood. Third, spatial areas cannot be kept coherent throughout the video, especially when objects appear or disappear. That is why the segmentation is performed in the luminance domain and consists of several successive operations:

1. Compute the luminance histogram of each frame (bin width w b in the log domain) 2. Find local maxima in the histogram (higher than threshold th) 3. Remove local maxima that are too close to each other (distance ρ) 4. Find local minima between successive local maxima 5. Dene local minima as segment's boundaries 6. Compute the key value for each segment (Eq. 2.10) Three parameters, namely w b , th and ρ, allow to tune the segmentation. w b corresponds to the width of each bin of the histogram (expressed in the log domain). Each local maximum of the histogram must be higher than the threshold th. Finally, ρ corresponds to the minimum distance between two local maxima (in log scale). The eects of these parameters on the segmentation are detailed in Section 5.2.4. The whole frame segmentation process is summarized in Figure 5.2. In this example, the algorithm efciently divides the HDR frame into three separate luminance segments. Figure 5.3 presents, in false color, the corresponding luminance segments computed in Figure 5.2.

We now have several segments per frame whose boundaries change from frame to frame. Figure 5.4 plots the video analysis of the UnderBridgeHigh sequence, showing the results obtained for one key value k f per frame (left) and each j th segment's key value k j s per frame (right). However, the change of a segment's boundaries through- out the video is a source of ickering artifacts. Large variations of segment boundaries between successive frames result in dierent key values and hence dierent scaling factors. Figure 5.5 illustrates a change of boundaries between frames 127 and 128 of the UnderBridgeHigh sequence. To avoid this problem, we compute video zones that have constant boundaries (in term of luminance values) throughout the video. with the video zones' key values k z for each frame. We see that the key value within each zone varies continuously, which ensures temporal coherency. Now that we have video zones, we apply the BC algorithm to each video zone of each frame. We consider two cases: a pixel luminance lies within a video zone or on the boundary.

Video Segmentation

Applying BC to video zones

After computing a key value for each video zone of each frame, we apply the BC algorithm to each zone. For each pixel located within a video zone (say, its luminance lies in-between the boundaries of a video zone), we apply the scale ratio dened in Equation 4.4. In this case, the key value k t f (L w ) of the t th HDR frame is replaced by the key value of the j th zone of the t th HDR frame k t,j z (L w ). Similarly, the highest video zone's key value of the HDR sequence is denoted k vz (L w ) while k vz (L m ) denotes its corresponding LDR key value.

At each video zone's boundary, we dene a blending zone (b z ) containing all the pixels whose luminance is close to this boundary up to a distance δ. To each pixel luminance within a b z , we assign a scale ratio s which is a combination of those of the video zones sharing this boundary. This weighted scale ratio prevents abrupt scale ratio changes that could cause high spatial contrast not present in the HDR frame. δ is a user-dened parameter in the log domain. The weight w l of the lower scale ratio s l assigned to a pixel x lying in a blending zone is dened as follows:

w l (x) = exp -(log(Lw(x))-log(b l )) 2 2σ 2 s h + s l , (5.1)
where the luminance b l represents the lower bound of b z . s h and s l are the scale ratios, used in Equation 4.4, of the two video zones sharing the boundary and are used to normalize each weight. To compute the weight w h of the higher scale ratio s h , b l is replaced by the higher bound b h in Equation 5.1. As for σ, it corresponds to the width of the weighting function support:

σ = δ 2 2 log(3) (5.2)
Finally the scale ratio s l,h of a pixel x is given by: s l,h (x) = s l w l (x) + s h w h (x),

(5.3)

where δ is the boundary width expressed in the log domain (set to 1 for all our experiments). Figure 5.7 plots the weight distribution function in a blending zone b z . Outside the zone delimited by b l and b h , the pixels are scaled by only one scale ratio. Inside this zone, the scale ratio is computed using Equation 5.3.

Parameters

Our solution relies on six parameters (χ, ζ, δ, b w , th and ρ). χ and ζ are the same parameters as the ones used in the BC algorithm. χ selects the anchor zone depending on its geometric mean. ζ prevents low scale ratio to trade o brightness coherency for a better preservation of details in dim frames. We found that setting it to 0.1 for every tested sequence provides good results.

Regarding δ, it is used to prevent abrupt changes of scale ratio on zone's boundaries.

Too low a value results in such abrupt changes while a too high one increases the number of pixels that lie on a boundary. As mentioned in Section 5.2.3, we found that setting it to 1 provides good results for all the tested sequences.

The three last parameters (namely b w , th, ρ) allow to determine the number of video zones. As all three parameters are highly correlated, it is hard to set each of them separately. That is why, we propose a way which facilitates the tuning of these parameters. We express the bin's width b w so as to achieve the same quantization as for 8-bit LDR imagery (256 gray levels corresponding to 8 stops in dynamic range):

b w = θ 8 256 (5.4)
where θ is a parameter used to increase or decrease width of the quantization step (binning). A high θ provides a coarse quantization thereby reducing the precision of the histogram. From b w , we can derive the number of bins n b used for each segmentation as:

n b = r b w , (5.5)
where r is the dynamic range of the HDR video sequence.

Recall that a local maximum of the histogram must be higher than the threshold th. Consequently, th must be coherent with the number of bins:

th = τ n p n b , (5.6)
where n p is either the number of pixels in the image for the frame segmentation or the number of key values in the histogram for the video segmentation. If τ is lower than 1, then the number of selected peaks increases otherwise it decreases. Experiments showed that a good setting for all the tested sequences is θ = 1 and τ = 2. Finally, we express the distance between two peaks ρ directly in the log domain. More peaks are discarded with a high ρ, reducing then the number of segments/zones used. The number of video zones directly inuences the locality of the ZBC technique.

We found that a value ranging from 0.5 to 1.5 oers a good compromise to tune the locality of our algorithm. Note that using only one video zone amounts to use the BC algorithm.

To summarize, we recommend to set δ = 1, θ = 1, τ = 2 and tune only ρ for the locality (default value = 0.65) and ζ for the details/brightness coherency trade o (default value = 0.1).

In conclusion, like the BC method, the ZBC resorts to a video analysis step prior to the tone mapping operation. This analysis relies on a histogram-based segmentation as shown in Figure 5.8. Performing the segmentation on a per frame basis results in several segments per frame. The key value of the j th segment of each frame is noted k j s (L w ) and k j s (L m ) respectively for the HDR and LDR frames. Using the segments'

F rame Segmentation V ideo Segmentation Anchor k vz (L w ) k vz (L m ) k t,j z (L w ) j t k f k s k z k v (L w ) k vz (L w ) k v (L m ) k vz (L m ) j t k t,j z (L m ) s j s j,j+1 b z Zones z j b j,j+1 z s j s j,j+1 Zone Scaling χ iCAM 06 k j z (L w ) k j z (L m )
the beginning of the sequence and the brightest ones at the end of the sequence have approximately the same value, which is not coherent. When using the BC technique (Figure 5.13c), this incoherency is reduced thanks to the scaling, but the brightness ratio is still not preserved. With its zonal preservation system, the ZBC algorithm (Figure 5.13d) solves both issues. Note that albeit the coherency is preserved between zones, it is not necessarily true inside the dimmest zones. This is due to the ζ parameter that prevents low scale ratio. For example, in the UnderBridgeHigh sequence, a low bound ζ is assigned to the scale ratio to prevent all dark frames under the bridge. This bounding explains the dierence in brightness of the downside of the bridge between the beginning and the end of the sequence. In other words, the ζ parameter trades o temporal brightness coherency to preserves detail in dimmer parts of a video.

Preserving the temporal local brightness coherency amounts to preserving temporal object coherency as described in Section 3.1.4. Figure 5.14 shows frames 80, 100, 120 and 140 of the UnderBridgeHigh sequence tone mapped with and without using the BC or the ZBC. Without any post-processing (top-row), the lower side of the bridge in frame 80 (that is a dim zone) is as bright as the sky in frame 140. With the BC technique (3 rd row), the bridge brightness appears more stable, however the sky's brightness is signicantly reduced. Only the ZBC (bottom row) preserves both aspects, providing frames with higher contrast and brightness coherency. However, dierences are still noticeable in the brightness of the downside of the bridge, due to the use of ζ that prevents a too low scale ratio. Eectively the BC technique allows to preserve the downside of the bridge temporal coherency but fails on the sky's coherency. On the other hand, the ZBC preserves both local areas thanks to its zonal system providing a good compromise between preservation of temporal brightness and object coherency while avoiding loss of spatial contrast.

Results regarding the T unnel sequence are presented in Figure 5.15. Note how the TMO alone (2 nd row) fails to reect the variations of the overall brightness of the HDR sequence. In addition, the appearance of the outside of the tunnel is almost burnt out.

By applying the BC technique (3 rd row), more overall brightness coherency is achieved, however the outside of the tunnel still lacks details. The ZBC technique (bottom row)

preserves both the coherency and the appearance of the outside of the tunnel.

Hue Coherency

In Section 3.1.5, we described the Temporal Hue Incoherency (THI) artifacts that occurs when the balance between tristimulus values in successive frames is not temporally preserved. The main reason of this imbalance is color clipping that desaturates color channels at dierent temporal positions. As the ZBC post-processing preserves the temporal object coherency, it will also preserve the temporal hue coherency either by always or never clipping the color channels. By studying Figure 3.9 and Figure 5.16, we notice that temporal hue coherency of the studied pixel and patch is preserved when the ZBC is used.

We demonstrated that our technique reduces TBI, TOI and THI artifacts. However, reducing artifacts does not always result in greater subjective quality, especially ) for the HDR (5.13a), [Ramsey et al., 2004] alone (5.13b), with the BC (5.13c) and with the ZBC (5.13d). The HDR brightness ratio between the two higher curves (red and black) is only preserved when using the ZBC (5.13d). Concerning the two lower curves (blue and green), both the ZBC and the BC solve the temporal brightness coherency problem. The lack of coherency of each curve along the sequence is due to the ζ parameter that prevents low scale ratio. and after applying [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] operator alone (2 nd row), with the BC (3 rd row) and with the ZBC (bottom row). Frames number, from left to right: 1, 100, 200 and 300.

(a) Left: [START_REF] Tumblin | Tone reproduction for realistic images[END_REF] operator with the ZBC (frame 29). Right: the temporal evolution of the central pixel of the square.

(b) Zoom on the area outlined by the rectangle in frame 4, 13, 29 and 45 (from left to right). if the overall brightness is altered. Consequently, to validate our ZBC technique, we performed a subjective evaluation presented in the next section.

Subjective Evaluation

To evaluate whether preserving spatio-temporal brightness coherency improves the subjective quality, we conducted a subjective experiment using the non-reference method ( Evaluating TMOs in Section 2.3.2).

For the comparison, we chose a forcedchoice pairwise comparison as it provides the most accurate results as detailed in [START_REF] Mantiuk | Comparison of Four Subjective Methods for Image Quality Assessment[END_REF]. The experiments were run in a dark room containing a 4K display device (TVlogic 56 Lum560W). The monitor was divided into 4 parts, the two HD videos to be compared were displayed on the upper-left and upper-right parts of the screen.

We experimented with three TMOs ( [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] which rely on global temporal ltering, Exponential mapping [Banterle, 2011] which is a global TMO and [START_REF] Li | Compressing and companding high dynamic range images with subband architectures[END_REF] which is a local TMO) and three HDR sequences UnderBridgeHigh, TunnelHD and GeekFlat). We performed the test for three conditions: the TMO alone (TMO), the TMO with the Brightness Coherency (BC) and the TMO with the Zonal Brightness Coherency (ZBC). The observers were asked to choose the video that they preferred. Each combination of two conditions was tested twice (inverting the left-right order of the showed videos). A choice is considered as coherent if the same video is Figure 5.17: Results of the subjective evaluation per video. The quality score and condence interval were computed using [START_REF] Lee | Subjective Quality Evaluation via Paired Comparison: Application to Scalable Video Coding[END_REF]. When the condence interval (yellow segment) includes the quality score 0.5, no conclusion can be drawn.

chosen for both left-right orders. The experiment consisted of 54 comparisons (3 TMOs

• 3 Sequence • 3 Conditions • 2 Combinations). 18 people (aged from 23 to 59) participated in the experiment, among them 14 are computer scientists without knowledge regarding HDR and tone mapping while the last four had prior experience with TMOs.

The results were tted in a Bradley-Terry model using Branderson's equation [START_REF] Lee | Subjective Quality Evaluation via Paired Comparison: Application to Scalable Video Coding[END_REF]. The condence intervals were computed using a non-coherent choice as a tie, while coherent choices were reported as win/lose results. Figure 5.17 plots the quality scores of each technique as well as their condence intervals for each tested video. These results show that for a given TMO, applying the BC post-processing does not improve the subjective quality. On the contrary, when comparing the ZBC with either the TMO+BC or the TMO alone, the ZBC post-processed video is preferred for the UnderBridgeHigh and Tunnel sequence while no conclusion can be drawn for the GeekFlat sequence. This is mostly due to the fact that only two video zones were determined using the default parameters in the case of the GeekF lat sequence.

This subjective evaluation conrmed the improvement brought by our ZBC method.

Summary

In this chapter, we proposed an improvement of the Brightness Coherency (BC) algorithm so as to overcome its limitations. Our modied post-processing technique aims at preserving the local brightness coherency as well as the contrast in each frame. To this end, we divide each frame into segments using a histogram-based segmentation of HDR images. We then dene video zones according to the resulting frame's segments. Finally, we apply the BC algorithm to each video zone independently. This Zonal Brightness Coherency (ZBC) algorithm better preserves both the temporal local brightness coherency and the spatial contrast in each frame. Furthermore, our technique handles, better than the BC, temporal artifacts that no other solution addressed before: Temporal Brightness Incoherency (TBI), Temporal Object Coherency (TOI) and

Temporal Hue Incoherency (THI). Finally, we conducted a subjective evaluation to evaluate whether our technique increase the subjective quality of the tone mapped video content. This technique has been published in:

R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, "Zonal Brightness Coherency for Video Tone Mapping," Signal Processing: Image Communication, vol. 29, no. 2, pp. 229-246, 2014.

In this chapter, we saw that preserving temporal coherency leads to higher subjective quality content and we proposed post-processing techniques to address temporal artifacts not considered before. In addition to the subjective quality of a video, another aspect can benet from temporal coherency: the compression eciency. Consequently, we propose in the next part to study temporal coherency with respect to the compression eciency.

Chapter 6

Compression of Tone Mapped

Video Contents

In the previous chapters, we presented the dierent types of temporal artifact that impair the subjective quality of tone mapped video content. We proposed two postprocessing techniques to reduce the Temporal: Brightness, Object and Hue Incoherency artifacts (TBI, TOI and THI). A subjective evaluation reported that post-processing video content with the ZBC increases their subjective quality. However, no such conclusion could be drawn when using the BC meaning that the eciency of this technique highly depends on the content. However compression also aects the subjective quality of tone mapped video contents. Indeed, we mentioned in Section 2.4 that codecs are required to compress the size of video data to t in the broadcast bit-rates or storage disc capacity. This chapter proposes a study on the relationship between video tone mapping and video compression. It is organized as follows:

Section 6.1. Tone Mapping and Video Compression: describes the need of tone mapping to compress HDR and LDR video content.

Section 6.2. Temporal Coherency and Inter-Prediction: analyses the impact of preserving temporal coherency on the quality of the Inter-prediction.

Section 6.3. Compression of Tone Mapped Video Sequences: presents some results of compression eciency.

Video Tone Mapping and Video Compression

In LDR digital imagery, all the signal processing operations are performed on perceptually encoded integer values (that is to say linear in regard to the perception). However, HDR data represent linear physical quantities in oating point values. Consequently, legacy video processing such as video compression are not optimized for HDR content.

That is why tone mapping is required to transform HDR content into integer values before any video is being fed to a codec. In this section, we address the compression of

LDR LDR

Dec P SN R = 10 • log 10 (2 n ) 2 M SE , n M SE = 1 M • N M -1 x=0 N -1 y=0 [F o (x, y) -F d (x, y)] 2 , F o F d (M, N ) D z D x D z D x ∆E 00 Y D z D x Y C b C r
Broadcasting LDR LDR HDR condition quality bit-rate distortion distortion Only LDR Only HDR LDR and HDR Table 6.1: Summary of broadcasting conditions for compression scheme illustrated in Figure 6.1 along with their associated criterion. represents the distance (in display order) between the reference and the predicted frame.

TMO D = 1 D = 2 D = 4 D =
of temporal coherency in video tone mapping.

Temporal Coherency and Inter-Prediction

Thanks to the high level of correlation existing between successive frames of a video sequence, codecs achieve high compression ratio via the Inter-prediction. However, independently tone mapping frames of an HDR video sequence disrupts this temporal coherency. In this section, we propose to assess the impact that the preservation of temporal coherency in video tone mapping has on motion estimation.

In LDR video compression, the quality of the motion estimation, and hence the interprediction, is evaluated by computing the PSNR between the predicted frame and the reference one. For our study, we tone mapped the UnderBridgeHigh sequence using 4 dierent congurations of a TMO: [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] operator alone, with ickering reduction ( [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF]] operator), with the BC post-processing and with both the ickering reduction and the BC. We evaluated the performance of each TMO by computing the PSNR between the predicted frame resulting from the motion estimation and the frame to predict. The motion estimation was performed at temporal distances corresponding to the hierarchical Group Of Pictures (GOP) structure specied in the main conguration prole of HEVC [START_REF] Bross | High eciency video coding (hevc) text specication draft 10[END_REF]. Table 6.2 reports the average PSNR of the predicted frames at temporal distances 1, 2, 4 and 8 for a block-size of 8.

Results show that using the BC post-processing increases the quality of the predicted frame. Note that reducing ickering artifacts does not improve the Inter-prediction for the used video sequence.

It is usually reckoned in video compression that improving the prediction results in a higher compression eciency. We verify the validity of this assumption in the next Figure 6.3: Rate-distortion curves of the UnderBridgeHigh and GeekFlat sequences for targeted bit-rates 500 kilo bits per second (kbps), 1, 2 and 4 Mega bps (Mbps). The top curves correspond to the use of the BC post-processing. section.

Compression of Tone Mapped Video Sequence

Quality of the LDR Decoded Video The main objective of TMOs is to achieve the best subjective quality. However, when tone mapped contents are compressed using a codec, this quality is impaired. That is why, this section addresses the evaluation of the quality of decoded tone mapped LDR sequences. For our evaluation, we compared the same congurations for the TMO used in the previous section for four HDR sequences (UnderBridgeHigh, GeekFlat, Sun and Tunnel). Figures 6.3 and 6.4 plots the ratedistortion curves obtained using the main conguration prole with the HEVC test Model (HM) 10.0 [START_REF] Bross | High eciency video coding (hevc) text specication draft 10[END_REF]. As expected, the increased quality of the interprediction obtained with the BC post-processing results in a PSNR gain ranging from 1 to 4 dB.

In addition, we propose to illustrate the types of artifact that are generated by compressing video contents. Figure 6.5 illustrates a zoom on a frame of the Tunnel sequence for each of the targeted bit-rates. Notice the appearance of compression artifacts around the car for bit-rates 250 and 125 kbps.

Results show that reducing ickering artifacts does not inuence the compression eciency while using the BC post-processing does. This is explained by the fact that the BC post-processing increases temporal coherency and hence the quality of the Interprediction Furthermore, the distortion generated by the codec can cause artifacts that degrade the subjective quality of tone mapped video content. To sum up, achieving the best subjective quality, when performing video tone mapping, does not guarantee high quality video when it is distributed. We have demonstrated that preserving the temporal coherency using the BC post-processing allows to decrease the distortion due to the codec (higher PSNR) when broadcasting tone mapped video sequences. However, decoded LDR content can be used to address an HDR display as illustrated in Figure 6.1. That is why, we assess in this section the impact of the BC post-processing on the quality of the HDR reconstructed sequence on the decoder side.

To reconstruct an HDR video sequence from the decoded LDR video sequence, we use an inverse Tone Mapping Operator (iTMO). As all the used congurations are based on [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] operator (see Section 2.3.2), we apply the same iTMO to reconstruct an HDR video sequence. The dierent steps of the iTMO are explained as follows:

convert n bits integer to oating point values show that using the BC post-processing reduces the quality of the reconstructed HDR sequence even without applying the codec. This can be explained as follows. TMOs assign several HDR values to a same LDR value. This results in a loss of information regarding the HDR sequence. Contrary to other TMOs that use the full available range, the BC post-processing assigns a variable and reduced range to each frame to preserve the brightness coherency. As a result, a BC-based reconstructed HDR video sequence is more distorted due to these reduced ranges.

L t g = f loat(L t d ) 2 n -1 , invert the γ encoding L t bc = L t g γ , (optional) invert the BC scaling (Equation 4.4): L t m = L t bc s t , -1 k f s t L s = L t m 1-L t m L t w = L t s k f α α k f γ s t L t
To illustrates the meaning of the MSE in the log domain, we show for dierent bit-rates a zoom on the reconstructed HDR frame 139 of the UnderBridgeHigh video sequence. Those HDR images are represented in false color luminance in Figure 6.8.

When only the TMO is applied, only small dierences on the chimney are noticeable (top row). When adding the compression at high bit-rates, banding artifacts begin to appear in the sky (2 nd row). Finally when lowering the bit-rates, banding artifacts increase in strength and details on the building's windows disappear (bottom row).

Results presented in this section show that reducing ickering artifacts has no eect either on the quality of the LDR video or on the reconstructed HDR video sequence.

On the contrary, the BC post processing preserves temporal brightness coherency while achieving a higher compression eciency. However, it distorts the reconstructed HDR video sequence more than other methods. Finally, for all congurations, the codec generates more distortion than the tone mapping.

Summary

In this chapter, we studied the relation between the compression eciency of tone mapped video content and the preservation of temporal coherency in video tone mapping. Two criteria were evaluated in regard to video compression: the distortion of the LDR decoded video and that of the HDR reconstructed video. Experiments were performed to assess the inuence of reducing ickering artifacts and/or preserving temporal brightness coherency. Results showed that using the BC post-processing increases the compression eciency of tone mapped video content. Furthermore, the subjective evaluation performed in Section 5.3.4 reported that applying the BC post-processing does not improve nor decrease the subjective quality. Consequently, using the BC post-processing allows to reduce the future distortion caused by a video codec while maintaining the same level of subjective quality.

Regarding the HDR reconstruction, using the BC post-processing or the ickering reduction technique alter marginally the distortion. However, when combined with a codec, the BC post-processing generates higher distortion even at high bit-rates. Finally, results showed that the main cause of distortion is compression whether we use the BC

or not. This study has been published in: To go further, we believe that these experiments should be performed for a wider range of HDR video sequences and TMOs. In addition, the ZBC post-processing should be included as it was reported to increase the temporal coherency and to improve the subjective quality as well. The only limitations of the ZBC with respect to compression eciency is that we would need to transmit the scale ratio corresponding to each pixel in order to invert the ZBC scaling. Included so many information would more than likely impair the compression eciency when the reconstruction of the HDR content is required.

However, one limitations of using the BC or ZBC post-processing is that this latter alters the rendering of a TMO. If this rendering was considered as an artistic intent performed previously in the pipeline, then the gain in compression is not enough to change this intent. Consequently, using the BC or ZBC is benecial only when the LDR video sequence is not constrained to a certain rendering. That is why, we propose in the next chapter a technique to increase the compression eciency of tone mapped video content without altering its rendering of a TMO.

Chapter 7

Motion-Guided Quantization for

Video Tone Mapping

In the previous chapter, we showed that preserving temporal coherency allows to improve the compression eciency of tone mapped video content. Other techniques aim at optimizing the tone map curve of a TMO to increase the quality of decoded content [Mai et al., 2011, Koz and[START_REF] Koz | [END_REF]. However, these techniques optimize the rendering of the video to achieve high compression ratio and hence do not preserve artistic intent nor achieve the best subjective quality.

However, the tone map curve is not the only aspect that can inuence the compression eciency. Indeed, the last operation performed by a TMO consists in quantizing oating point values to integer ones. We believe that by making this quantization temporally coherent a higher compression eciency can be achieved. To this end, we propose to adapt the quantization to increase the temporal correlations between successive frames. This chapter presents this technique and is organized as follows:

Section 7.1. Quantization in Video Tone Mapping: details the traditional quantization performed in video tone mapping.

Section 7.2. Motion-Guided Quantization: describes our technique to increase the compression eciency of tone mapped video content without altering their rendering.

Section 7.3. Results: assesses the eciency of our technique for compressing and denoising tone mapped video content.

Quantization in Video Tone Mapping

In this chapter, the term quantization refers to converting a oating point value to an integer one. When quantizing a value, one has only three choices: oor ( • ), ceil ( • ) or round ( • + 0.5 ). This quantization is dierent from the adaptive quantization, usually considered in imagery, that consists in optimizing the distribution of the quantization bins in regard to an image's cumulative distributive function [START_REF] Yang | Adaptive quantization using piecewise companding and scaling for Gaussian mixture[END_REF].

F t t th cd/m 2 γ = 1/2.2 [0; 2 n -1] F t g 2 n -1 F t d = Q(F t g ) = (2 n -1)F t g + 0.5 == F t s + 0.5 Q(•) n • (2 n -1)F t g F t d F t s (2 n -1)F t g F t g C b C r N R F t-1 d F t s (x, y) (δx, δy) F p F p (x, y) = F t-1 d (x + δx, y + δy) C b C r F p F t s F t,MGQ d = GQ(F t s ) = F t s F t s -F p ≥ 0 F t s F t s -F p < 0 GQ(•) • • F t s F p 2 n -1 M GQ F t-1 d F t s F t,MGQ d F t s F p F t,MGQ d F t s F t,MGQ d δ F p F t s F t,MGQ d =    F t s 0 ≤ F t s -F p < δ F t s -δ < F t s -F p < 0 F t s + 0.5 δ = 0 δ = 1 δ = ∞ δ = 0 δ = ∞ δ = 1 Ω F p -F t s < δ F t,MGQ d (Ω) = F p (Ω) δ F t,MGQ d (Ω) F p (Ω) F t s F p δ δ F t,

Results

Our technique aims at increasing the compression eciency while adapting to any TMO without altering its intent. In this section, we show that the distortion obtained with our technique and the rounding quantization are very close. Finally, we report the compression eciency of tone mapped content with and without using our technique.

Quantization Loss

Integer quantization assigns several oating point values to the same integer. This process obviously results in a loss of information in the quantized signal. We assess the loss due to the quantization by computing the PSNR between the unquantized current frame F t s and the quantized one F t,MGQ d . Table 7.3 reports the PSNR using three dierent quantizations: Rounding, MGQ with δ = 1 (noted M GQ 1 ) and MGQ with δ = ∞ (named M GQ Inf ). As mentioned earlier, our quantization technique provides a slightly more distorted sequence than the rounding quantization. This distortion is no greater than one code value for all the quantized pixels (contrary to the rounding quantization that entails a maximum distortion of half a code value). For comparison, the distortion due to a lossy codec is always greater than or equal to one code value. rate) for the tested video sequences. The sequence TunnelHD has been also tone mapped using [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] TMO. The Students sequence however has been tone mapped using [START_REF] Farbman | Tonal stabilization of video[END_REF] operator. The results show that for the same quality, the M GQ Inf provides an average bit-rate reduction of 8.5% for all the test sequences. Note that the Sun and Tunnel sequences perform better than the other two. This is due to the fact that these sequences are relatively noisy and our quantization technique reduces some of the temporal noise.

M GQ

1 M GQ Inf M GQ M GQ Inf M GQ 1 δ M GQ δ M GQ Inf Sequence Y U V Sun -

Improving Performances

Our method has some limitations, the main one being its computational complexity due to the motion estimation. Furthermore, the proposed implementation is sub-optimum with respect to the prediction process performed in HEVC. Because, rst our technique does not follow the GOP hierarchical pattern that is used in a codec. Indeed, it only uses motion compensation between successive frames. Second, the Intra-prediction process should also benet from our quantization. Third, a block-based codec uses a ratedistortion cost function to select the best predictor for each block while our solution only relies on a distortion metrics.

To summarize, the MGQ technique, if implemented in the coding loop instead of being a pre-processing, should provide an even higher compression eciency. It would allow to tune the quantization separately for each of the available prediction modes.

The selected mode and its associated quantization would depend on the codec's ratedistortion function rather than solely on the distortion. Regarding the trade-o parameter δ, it could be linked to the rate-distortion function to achieve a higher compression eciency while reducing the quantization distortion. Furthermore, the computational complexity would no longer be an issue as the motion estimation is already performed for the Inter-frame prediction. However, to implement our approach inside the coding loop of a block-based encoder (say HEVC), the computations within the codec should be performed with oating point values rather than integer. Finally, our method should be tested with the compression of native HDR sequence using perceptual encoding (Section 6.1).

We argue that our method can adapt to any application where tone mapping is

Denoising Application

As mentioned above, our method performs better for the Sun and Tunnel sequences because it reduces the temporal noise. When compression is not the targeted applications, our method can reduce the noise present in a tone mapped video sequences.

Indeed, in the previous section, the MGQ was guided by the value of the dierence between F t s -F p . Instead of adapting to the Inter-predicted frame F p , we adapt our quantization to a denoised frame F n . The way F n is computed is not relevant to this paper and any existing denoising technique can be used [START_REF] Brailean | Noise reduction lters for dynamic image sequences: a review[END_REF]. For our experiments, we will consider a simple temporal-ltering with motion compensation:

F t n (x, y) = M l=-N
F t-l (x -δx t,t-l , y -δy t,t-l ) w(l)

(7.5)
where (δx k,k-l , δy k,k-l ) is a motion vector obtained through a motion estimation between frames F t-l and F t . N (respectively M) represents the number of non-causal (respectively causal) extents of the averaging window and w(l) are the weights or the lter coecients. Note that causal frames are expressed with integer values while noncausal ones with oating point values (including the current one which is in our case

F t s ).
For our experiments we used only two frames in the lter bank: the previous one F t-l d and the current one F t s . We tested our method on the same set of sequences and TMOs as in Section 7.3.2. To assess the performance of our method when compared to the rounding quantization, we compute the PSNR between the quantized frame (either

F t,Round d or F t,MGQ d
) and the desired denoised frame F n . We report those PSNR in Table 7.5. For all test sequences, we achieve at least 1 dB of gain using the MGQ technique when compared to the rounding technique. The main advantage of using our technique rather than performing a denoising after the tone mapping lies in the fact that our technique does not introduce additional artifacts to the sequence. Indeed, denoising usually results in a smoothing which is source of problems when performed on edges. However, the amount of denoising that our technique can achieve is limited to the quantization which is not sucient for really noisy sequences.

Summary

In this chapter, we pointed out that, when performing tone mapping, rounding quantization is not ecient. We chose a quantization that aimed at improving the compression eciency of tone mapped video content. Our technique relies on the motion compensation between two successive frames of a sequence to adapt the quantization during the tone mapping. Results showed an average bit-rate reduction under the same PSNR ranging from 5.4% to 12.8%. The proposed method allows a trade-o between compression eciency and quantization distortion of the original video. We also applied our technique to denoising to show that our method can be generalized to deal with any applications where tone mapping is needed. The work presented in this chapter has been published in:

R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, "Motion-Guided Quantization for Video Tone Mapping," in IEEE International Conference on Multimedia and Expo (ICME), 2014.

As mentioned in Section 7.3.3, the main limitation of our technique is its computational complexity. However, if implemented in the loop of a codec, not only the complexity issue would no longer exist, but the eciency would be increased.

Chapter 8

Conclusion

In this thesis, we rst began by outlining the limitations of the standard imagery pipeline. To overcome these limitations, we described the emergent High Dynamic

Range imaging techniques from capture to rendering in Chapter 1. In the HDR imaging pipeline, we focused on the role of tone mapping that ensures backward compatibility between future HDR content and current LDR displays. Although tone mapping has been an active research eld for more than one decade, few researchers have investigated the temporal aspect of tone mapping.

From this fact, we rst conducted several experiments to test state of the art TMOs with several HDR video sequences. From these experiments, we described the source of six types of temporal artifacts occurring when applying naively a TMO to an HDR video sequence. We then listed and experimented with dierent techniques that aimed at coping with temporal artifacts. After showing that several temporal artifacts were not accounted for in video tone mapping, we drafted the requirements for an ecient and future proof technique. These requirements were the reduction of the aforementioned artifacts as well as its adaptability to any TMO. Indeed, since the tone mapping eld is in constant evolution, adapting to any TMO allows technique to be future-proof. From these two requirements, we chose to rely on an analysis of the HDR video sequence in order to post-process any TMO's output. Our post-processing consists in preserving the temporal coherency between the brightest frame of the sequence and all the other ones. The Brightness Coherency (BC) method has been presented in Chapter 4 and in an international conference: Reducing artifacts allows to increase the subjective quality of a video. However, as videos are bound to be distributed to the end user, they will undergo compression.

Compressing a video sequence for mass distribution inevitably results in a loss of information which leads to a decrease of the subjective quality. As storage disc capacity and broadcaster's bandwidth are limited, compressed content bit-rates need to at least be below this limitation. Consequently, the level of entropy that any content can achieve is as important as the compression eciency of a codec. That is why, we proposed, in Chapter 6, a survey on the benets of preserving temporal coherency in video tone mapping before the compression stage. We demonstrated that even if the tone mapping achieves the best subjective quality envisioned, it is more than likely that the compression stage will impair the quality of this content. We also showed that optimizing the TMO to increase the compression eciency can remove any artistic eect in the tone map result. This survey has been published in an international conference: Through this survey, we understood that it is possible to modify the mapping performed by a TMO to alter the entropy of a tone mapped video sequence. However, this modication changes the visual perception of this video and can impair any artistic intent or desired rendering. We wanted a technique that reduce the entropy of a video sequence while preserving its rendering. The last operation performed by a TMO is the quantization that consists of a simple rounding to the nearest integer. As this quantization is by nature not visible (the number of quantization bins is higher than the number of Just Noticeable Dierence in an 8 bits video), modifying this quantization will not alter the rendering. Consequently, we designed a Motion-Guided Quantization (MGQ) technique that adapts the quantization operation of any TMO to minimize the distortion between successive frames through a motion compensation. Thanks to the MGQ, the Inter-prediction performed in any codec will be improved, hence less residuals will need to be encoded. Results showed an average bit-rate reduction under the same PSNR ranging from 5.4% to 12.8%. The proposed method allows a trade-o between compression eciency and the distortion between the unquantized and quantized tone mapped video. We also applied our technique to denoising to show that it can be generalized to deal with any application where tone mapping is required. The work presented in Chapter 7 was published in an international conference: R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, "Motion-Guided Quantization for Video Tone Mapping," Accepted in IEEE International Conference on Multimedia and Expo (ICME), 2014.

Future Work

We suggest three research of avenues as future work, two related to the work presented in this thesis and one more exploratory.

Reduction of Temporal Artifacts

In Chapter 3, we described two types of temporal artifacts generated by video TMOs: Ghosting Artifacts and Temporal Contrast Adaptation. Those issues have not been dealt with in this thesis and we believe that designing generic solutions to reduce those two types of artifact is mandatory to develop the eld of video tone mapping.

Ghosting Artifacts appear because temporal ltering put in correspondence pixels without relation in successive frames. To minimize these mismatch associations, motion estimation and compensation are performed to align pixels. However, foolproof motion estimation does not exist and hence mismatch motion vectors will remain.

When performing temporal ltering along the motion vectors, these mismatch vectors create artifacts highly visible as they introduce edges in smooth areas. To reduce these eects, we need an ecient way to detect mismatch vectors. As long as these mismatch vectors cannot be detected accurately, video tone mapping operators that rely on spatio-temporal ltering will be prone to ghosting artifacts.

Regarding Temporal Contrast Adaptation artifacts, they are caused by TMOs that rely on global temporal ltering. Global temporal ltering is an ecient technique to reduce global ickering artifacts but creates artifacts in case of change of illumination.

Adapting the global temporal ltering by removing outliers as in [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF] solves this issue but only if the change of illumination is of global nature. For local changes of illumination, a pixel-wise comparison of change of illumination in the HDR and LDR sequence can be used to detect artifacts. If a change of illumination occurs in the HDR sequence and not in the LDR one, then an artifact is present. Once an artifact is detected, a post-processing operation, similar to [START_REF] Guthier | Flicker reduction in tone mapped high dynamic range video[END_REF], but on a per pixel basis, could be used to alleviate it.

Adaptive Quantization

The second research avenue corresponds to the implementation of the Motion-Guided Quantization in the loop of an encoder. As stated in Chapter 7, the current pre-processing method does not take into account the group of picture hierarchy used in a codec, nor does it adapt the quantization to the Intra-prediction.

Furthermore, the only criterion to adapt the quantization is the distortion while in the codec it could be adapted to the Rate Distortion Optimization (RDO).

Another lead could be to use the adaptive quantization when encoding HDR video content perceptually. Indeed, current standardization bodies are investigating perceptual transform to convert HDR values (oating point values that are physically linear) to perceptually uniform values (represented by integer on a limited bit-depth). We believe that using the adaptive quantization in this context could also improve the compression eciency of HDR content.

HDR Color Encoding

The last research avenue is related to the perceptual encoding of HDR values to perceptually linear ones. Two main transforms are of interest for video compression: the Electro-Optic Transfer Function (EOTF) and the color dierence encoding. EOTF is a constant transformation that encodes luminance values to luma values [START_REF] Mantiuk | Perception-motivated high dynamic range video encoding[END_REF], Miller et al., 2013, Kunkel et al., 2013, Touzé et al., 2014].

Color dierence encoding separates luma from chroma channels. This type of representation allows to subsample chroma as we are less sensitive to color dierence than luma dierence. Evaluating the minimum bit-depth required to encode HDR content with those dierent EOTFs and color dierence encoding would allow to discriminate perceptually relevant information from quantization noise.

Afterword All those research avenues along with the work presented in this manuscript demonstrate that video tone mapping is not a solved problem. Furthermore, with the recent public announcements of HDR commercial displays and cameras, it is likely that more issues related to video tone mapping will arise in the near future.

Résumé en Français

Depuis les origines du domaine de la photographie, les professionnels de l'imagerie ten-

A.3 IRISA Sequences

In order to test the dierent artifacts presented in Chapter 3, we created several HDR sequences. Those synthetic sequences were rendered using Mitsuba [Jakob, 2010] and the modeling was performed with Blender.

ColorCheckBall

The ColorCheckBall sequence is composed of 100 qHD frames generated at the frequency of 25 fps. It represents a ball moving in front of a ColorChecker (checkerboard array of 24 scientically prepared colored squares in a wide range of colors). Part of the ColorChecker is under a bright lamp while the rest of it lies in shadows.

The ball moves from the dim part to the bright one.

GeekFlat

The GeekFlat sequence is composed of 400 Full HD frames generated at the frequency of 25 fps and represents a camera moving in a 3-room at. Two well illuminated rooms are separated by a dim corridor. This sequence is a good test for brightness coherency as some TMO render the dim corridor brighter than the two rooms.

Its analysis is plotted in

Soccer

The Soccer sequence is composed of 250 Full HD frames generated at 25 fps.

This video takes place in a football stadium where two gures pass a ball before scoring.

All of the stadium is in shadows while the other half is under direct sunlight. Its analysis is plotted in A.5 Linköping Sequences

The sequences were captured using an system developed in collaboration between a camera manufacturer (SpheronVR) and the Visual Computing Laboratory at Linköping University in Sweden. The camera is a multi-sensor imaging system that captures HDR images with a resolution of 2336x1752 pixels at up to 30 fps with a dynamic range of over 24 f-stops. The high bandwidth HDR data stream can be viewed in real time through GPU processing, and is written without compression to a le on an external storage unit. A detailed overview of the imaging hardware setups and the image reconstruc- tion algorithms can be found in [START_REF] Kronander | Unied HDR reconstruction from raw CFA data[END_REF], [START_REF] Kronander | A unied framework for multi-sensor HDR video reconstruction[END_REF]. The public version of these footages have been down-sampled from the Full HD resolution 7 to HD Ready.

Students

The Students sequence is composed of 250 HD Ready frames captured at the frequency of 25 fps Several students walking in a dim corridor with a view on the outside through a window on the background. The analysis is plotted in Figure A.8.

Hallway2 and Hallway3

The Hallway2 sequence is composed of 330 HD Ready frames captured at the frequency of 25 fps. The Hallway3 sequence is composed of 350 HD Ready frames captured at the frequency of 25 fps. They both illustrate a walkthrough in a corridor of the Linköping university. Their analysis are plotted in Figure A.9.

R

  . Boitard, K. Bouatouch, R. Cozot, D. Thoreau and A. Gruson, "Temporal Coherency for Video Tone Mapping," in Proc. SPIE, Applications of Digital Image Processing XXXV, 2012. D. Touzé, Y. Olivier, S. Lasserre, F. Leléannec, R. Boitard and E. Francois, "HDR Video Coding based on Local LDR Quantization," in HDRi2014 -Second International Conference and SME Workshop on HDR imaging, 2014.

Figure 2

 2 Figure 2.2: Spectrum of the sun light and the full visible spectrum

Figure 2

 2 Figure 2.5: 1 st to 5 th images: dierent exposures of the same scene, two successive images are separated by 4 f-stops. Rightmost image: false color representation of the generated HDR image.

Peak

  Luminance and ContrastIn addition to the limited code values available to encode dierent colors or gray levels, displays have two other limitations: their peak luminance and their contrast. The peak luminance indicates the maximum amount of brightness that can be perceived by an observer while staring at the display. It is achieved when displaying the maximum code value represented by the tuple (R = 255, G = 255 and B = 255), also known as the white point of a display. Similarly, the black point of a display is represented by the tuple (R = 0, G = 0 and B = 0). On an LCD-based display, the black point is never truly 0 cd/m 2 because black is simulated by the crystals of the panel being completely shut, along with a polarized layer behind the crystals, to prevent light from the backlight to go through. Yet precisely because the

Figure 2 .

 2 Figure2.8 depicts the workow of this TMO. First, the HDR image is tone mapped using the default parameters of the TMO. In the same time, the TMO computes the response R orig of the Human Visual System (HVS) to the HDR image. Note that this response may be computed on an enhanced version (denoised, sharpened etc.) of the HDR image. Then the TMO applies an inverse display model on the resulting LDR image and computes the response R disp of the HVS. This TMO computes a piece-wise tone map curve where the location of the curve's nodes are rened to minimize an error metric between R orig and R disp responses.

Figure 2 .

 2 Figure 2.10: Comparison of the global and local version of [Reinhard et al., 2002] operator. From left to right: global, local and dierence in luma.

Y

  = 0.702R + 1.039G + 0.433B, (2.16) where R, G and B are the three color components of the HDR image. A bilateral lter is then applied separately on Y and Y to determine the ambient illumination. The σ p σ s Background in High Dynamic Range Imaging and an observer chooses the TMO that provides the closest reproduction of the HDR

Figure 2

 2 Figure 2.17: Example of the workow of a backward compatible scalable HDR video codec.

  Figure 3.2: Global ickering artifacts due to the use of the 99% percentile on two successive frames of the Desert sequence ([Farbman et al., 2008] operator).

Figure 3 . 3 :

 33 Figure 3.3: Example of local ickering artifacts when applying an edge-aware TMO to 3 consecutive frames. Top row: base layer computed using the bilateral lter. Bottom row: corresponding tone mapped result.

Figure 3 . 6 :

 36 Figure 3.6: Example of a temporal contrast incoherency when a change of illumination occurs.False color luminance (top row) and tone mapped result using

Figure 3 . 7 :

 37 Figure 3.7: Example of TBI and TOI artifacts. False color luminance (top row) and tone mapped result using[START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] operator (bottom row). The TBI artifact is represented by the overall brightness of each frame that is not coherent between the HDR and LDR frames. The TOI artifact is represented by the pixels' values of the downside of the bridge similar in the HDR sequence while greatly varying in the LDR one.

Figure 3 . 9 :

 39 Figure 3.9: Example of temporal hue incoherency due to color clipping. Each color channel desaturates at dierent temporal positions.

  Figure 3.10. The top row in Figure 3.10 provides the independently tone mapped version of three successive frames of an HDR video sequence. The second row displays the corresponding piece-wise tone map curve on top of their histogram. Note how the tone map curve of the middle frame is dierent from the other two, resulting in a change of overall brightness (global FA) in the tone mapped result. The third row shows the temporally ltered version of the piece-wise tone map curves. Finally, the bottom row provides the tone mapped frames after reducing the global ickering artifact.

Figure 3 .

 3 Figure 3.10: Reduction of global ickering artifacts by temporally ltering the tone mapping curve.

Figure 3 .

 3 Figure 3.11: Evolution of the frame key value computed for every frame of a video

Figure 3 .

 3 Figure 3.13: Example of temporal ltering of the tone map curve when a change of illumination occurs. Top row: tone mapped result (frame number 130, 140, 149 and 150) using[START_REF] Mantiuk | Display adaptive tone mapping[END_REF] operator with the temporal ltering active (pfsTMO implementation[Grzegorz Krawczyk, 2007]). Bottom row: histograms of frames 130 (left) and 150 (right) along with the corresponding tone map curve for frames 130, 140 and 150.

Figure 3 .

 3 Figure 3.13 shows how the temporal ltering of the tone map curve causes this artifact during a change of illumination. Note how the tone map curve, plotted on top of the histograms, shifts from the rst illumination condition (frame 130) toward the second state of illumination (frame 150, see Figure 3.6 for the false color luminance). As the tone map curve has anticipated this change of illumination, frames in the illumination transition are tone mapped incoherently.

Figure 3 .

 3 Figure 3.14: Ghosting Artifacts appearing on two successive frames. It is most noticeable around the two forefront columns (red squares).

  1, technical solutions to reduce temporal artifacts in video tone mapping focus mostly on ickering artifacts. That is why, we propose, in this chapter, a technique to address the reduction of Temporal Brightness Incoherency (TBI) artifacts. Recall that TBI artifacts occurs when the temporal brightness relationship between frames of an HDR video sequence is not preserved throughout the tone mapping operation. This chapter is organized as follows: Section 4.1. Temporal Contrast and Number of Tonal Levels: outlines the dierences between the HDR and LDR spatial/temporal contrast and sampling along with a description of LDR techniques to optimize the spatio-temporal sampling. Section 4.2. Brightness Coherency Post-Processing: describes our technique to reduce TBI artifacts when performing video tone mapping. Section 4.3. Results: presents some results regarding the preservation of brightness and object coherencies. Section 4.4. Limitations of the Brightness Coherency Post-Processing:

Figure 4

 4 Figure 4.1: Left: spatial and temporal contrasts in the Desert sequence. MinPerc and MaxPerc stand for minimum and maximum percentile, set at 1 % and 99 % respectively. k f is the key value computed for each frame. Right: frames 10, 50, 150 and 250 of the corresponding sequence in false color representation.

  Figure4.2: Left: example of camera's exposure adaptation, the results were obtained using the camera TMO[START_REF] Petit | Assessment of video tone-mapping : Are cameras ' S-shaped tone-curves good enough?[END_REF]. The LDR Min and Max value have been mapped back to HDR value to illustrate the dynamic range that the camera TMO covers. k f correspond to the frame's key value computed using Equation4.2. Right: tone mapped frame 10, 50, 150 and 250 of the corresponding sequence.

Figure 4

 4 Figure 4.4: Left: geometric means of the HDR and tone mapped video sequence. Right, upper row: global ickering artifact occurring between frame 248 and 249 due to the appearance of the sun. Right, lower row: global ickering artifact reduced using the BC post-processing.

Figure 4 . 5 :

 45 Figure 4.5: Example of BC post-processing when tone mapping the Desert sequence.

Figure 4 . 6 :

 46 Figure 4.6: Inuence of ζ from left to right: 0.01, 0.2, 0.5 and 1.

Figure 4 . 7 :

 47 Figure 4.7: Evolution of a pixel throughout the LDR video sequence for dierent TMOs.

  Figure 4.8 displays the area surrounding the pixel studied in Figure 4.7. The visual perception of the underside of the bridge is completely dierent between frame 25 (upper row) and 35 (lower row). Only our solution preserves the overall appearance of the bridge over time.

Figure 4 . 8 :

 48 Figure 4.8: Area surrounding the pixel studied in Figure 4.7 with the considered TMOs.

Figure 4 .

 4 Figure 4.10: MtStMichelWhite video sequence represented in false color representation

Figure 4 .

 4 Figure 4.11: Reduced dynamic range and loss of contrast due to the BC algorithm. Left: maximum LDR luma value of the UnderBridgeHigh sequence with and without using the BC post-processing. Right: reduced contrast and TOI artifact (sky's brightness) due to the global scaling. Frames 120 and 160 tone mapped with the TMO alone (top row,[START_REF] Ramsey | Adaptive temporal tone mapping[END_REF]] operator) and with (bottom row) the BC post-processing.

  Figure 5.1: Spatial areas and luminance segments for frame 50 of the UnderBridgeHigh sequence. The false color luminance (left) suggests 3 spatial areas (top green bridge, blue separation and bottom green bridge) while the luminance histogram (right) only two luminance segments. The vertical green line indicates the location of the frame's key value (a.k.a. geometric mean).

Frame

  Figure 5.2: Example of histogram-based segmentation with frame 108 of the Under-BridgeHigh sequence. (a) illustrates this frame in false color luminance while (b) to (e) summarize all the segmentation steps.

Figure 5

 5 Figure 5.6: Video segmentation of the UnderBridgeHigh sequence. Left: segmented key value histogram where z j->j+1 b

Figure 5 . 7 :

 57 Figure 5.7: Weight distribution function of the blending zone. w l corresponds to the weight of Zone l scale ratio and w h the weight of Zone h scale ratio.

  Figure5.13: Video zone key value (k j z ) for the HDR (5.13a),[Ramsey et al., 2004]

Figure 5 .

 5 Figure 5.14: UnderbridgeHigh video sequence represented in false color representation (top row) and after applying [Li et al., 2005] operator alone (2 nd row), with the BC (3 rd row) and with the ZBC (bottom row). Frames number, from left to right: 80, 100, 120 and 140.

Figure 5 .

 5 Figure 5.15: Tunnel video sequence represented in false color representation (top row)

Figure 5 .

 5 Figure 5.16: Example of preserved temporal hue coherency thanks to the ZBC. The result without using the ZBC is provided in Figure 3.9.

Figure 6 . 4 :

 64 Figure 6.4: Rate-distortion curves of the Sun and Tunnel sequences for targeted bitrates 125, 250, 500 and 1000 kbps.

Figure 6. 5 :

 5 Figure 6.5: Zoom on the decoded frame 2 of the Tunnel sequence (TMO = [Ramsey et al., 2004]) at bit-rates 1000, 500, 250 and 125 kbps(from left to right). Artifacts around the red car appear for bit-rates 250 and 125 kbps. Corresponding PSNR with the original frame 1000 kbps = 40.82 dB, 500 kbps = 40.20 db, 250 kbps = 38.85 dB and 125kbps = 37.71 dB.

Figure 6 . 8 :

 68 Figure 6.8: Zoom on the reconstructed HDR frame 139 of the UnderBridgeHigh video sequence represented in false color luminance. Top row represents the original HDR frame (left) and the reconstructed HDR frame without compression (right, MSE = 0.0620). 2 nd row represents the decoded/reconstructed HDR frame for bit-rates 4000 kbps (left, MSE = 0.1093) and 2000 kbps (right, MSE = 0.1130). Bottom row represents the decoded/reconstructed HDR frame for bit-rates 1000 kbps (left, MSE = 0.1269) and 500 kbps (right, MSE = 0.1416).

R

  . Boitard, D. Thoreau, R. Cozot and K. Bouatouch, "Impact of Temporal Coherence-Based Tone Mapping on Video Compression," in Proceedings of the 21st European Signal Processing Conference (EUSIPCO), 2013.

R

  . Boitard, D. Thoreau, R. Cozot and K. Bouatouch, "Impact of Temporal Coherence-Based Tone Mapping on Video Compression," in Proceedings of the 21st European Signal Processing Conference (EUSIPCO), 2013.

R

  . Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, "Survey of temporal brightness artifacts in video tone mapping," in HDRi2014 -Second International Conference and SME Workshop on HDR imaging, 2014. L'état de l'art du video tone mapping nous a permis d'identier trois types d'artefacts temporels non-résolus par les méthodes existantes. Deux choix étaient possibles pour réduire ces artefacts: créer un nouveau vidéo TMO ou bien proposer une méthode générique qui permet de réduire ces artefacts lorsque l'on utilise n'importe quel TMO. Comme le domaine du tone mapping est en perpétuelle évolution, nous avons choisi la seconde solution pour que notre méthode puisse s'adapter aux futurs TMOs développés. Nous avons donc proposé une technique de correction de vidéo tone mappée en se basant sur une analyse de la vidéo HDR. Un traitement a posteriori préserve la cohérence temporelle de la vidéo tone mappée en s'assurant que le ratio de brillance entre l'image la plus brillante de la séquence HDR et les autres images soit préservé dans la vidéo LDR. Notre technique baptisée Brightness Coherency (BC) est présentée dans le Chapitre 4 et a donné lieu à un article de conférence internationale : R. Boitard, K. Bouatouch, R. Cozot, D. Thoreau and A. Gruson, "Temporal Coherency for Video Tone Mapping," in Proc. SPIE, Applications of Digital Image Processing XXXV, 2012. Cependant, lorsque notre méthode a été utilisée lors d'une évaluation subjective, les participants ont constaté qu'elle produisait un résultat sous-exposé lorsque le contraste spatio-temporel était élevé. En eet, comme notre technique ne se base que sur la brillance globale d'une image, elle n'arrive pas à gérer un contraste de nature locale. Nous avons donc modié notre traitement a posteriori pour le rendre local. Cette nouvelle méthode baptisée Zonal Brightness Coherency (ZBC) détermine des zones vidéos dénies par deux valeurs de luminance constantes durant toute la séquence. Nous utilisons ces zones vidéos pour préserver la cohérence temporelle de brillance entre la zone la plus brillante et toutes les autres zones. Cette technique a également été testée lors d'une évaluation subjective et les résultats indiquent qu'elle préserve toujours ou améliore la qualité subjective pour les diérentes séquences et TMOs testés. Le traitement a postériori ZBC a été présenté dans le Chapitre 5 et a été publié dans un journal international : R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, "Zonal Brightness Coherency for Video Tone Mapping," Signal Processing: Image Communication, vol. 29, no. 2, pp. 229-246, 2014. Réduire les artefacts temporels permet d'améliorer la qualité visuelle d'un contenu. Cependant, ces contenus vidéo sont voués à être distribués au grand public. Or les moyens de distribution traditionnels sont limités en termes de capacité de stockage ou de bande passante. Ces contenus vont donc devoir être compressés avant de pouvoir être distribués. La compression vidéo entraine inéluctablement une perte d'information qui diminue la qualité de la vidéo décodée. Dans ce contexte de compression de contenu tone mappée, deux caractéristiques principales permettent d'améliorer la qualité du contenu décodé. La première est l'ecacité du codec (coder-decoder ) utilisé tandis que la seconde correspond à l'entropie du signal. Comme le tone mapping consiste en une transformation d'une vidéo HDR, le choix du TMO inuence l'entropie de la vidéo LDR à encoder. Nous avons donc étudié dans le Chapitre 6 les bénéces de préserver la cohérence temporelle lors du tone mapping, sur la qualité de la vidéo décodée. Nous avons démontré que l'utilisation du traitement a posteriori BC permet d'améliorer la qualité d'une vidéo reconstruite. De plus, les résultats indiquent que même si la meilleure qualité visuelle possible est obtenue après le tone mapping, l'étape de compression va très probablement dégrader cette qualité. Cette étude nous a montré qu'un équilibre entre la qualité visuelle du tone mapping et l'ecacité de la compression est nécessaire pour éviter qu'un traitement ne dégrade la qualité de l'autre. Cette étude a donné lieu à un article dans une conférence internationale : R. Boitard, D. Thoreau, R. Cozot and K. Bouatouch, "Impact of Temporal Coherence-Based Tone Mapping on Video Compression," in Proceedings of the 21st European Signal Processing Conference (EUSIPCO), 2013. Cette étude sur la relation entre le video tone mapping et la compression nous permet de constater qu'il est possible de modier la transformation eectuée par un TMO pour améliorer l'entropie de la vidéo à encodée. Cependant, cette modication altère le rendu de cette vidéo, changeant par conséquence une quelconque intention artistique qui pouvait être imposée sur le rendu nal. A partir de cette observation, notre objectif a été de développer une méthode qui améliore l'entropie d'une vidéo tone mappée sans altérer son rendu. La dernière opération d'un TMO est la quantication des valeurs ottantes en entier sur une profondeur de bits dénie. Cette quantication consiste en un arrondi au plus proche entier et n'altère pas le rendu d'un contenu. En eet, le pas de quantication utilisé en imagerie LDR est plus petit que notre seuil de détection d'une diérence (Just Noticeable Dierence en anglais). A partir de cette observation, nous avons implémenté une méthode qui adapte la quantication de n'importe quel TMO pour minimiser la distorsion entre images successives en utilisant une compensation de mouvement. Cette méthode a été baptisée Motion-Guided Quantization (MGQ) et permet d'améliorer la qualité de la prédiction Inter images qui est réalisée au sein d'un codec. Par conséquence, la quantité de résidus à encoder est moindre et l'ecacité de compression améliorée. Les résultats montrent une réduction moyenne du débit pour le même Peak Signal to Noise Ratio (PSNR) allant de 5.4% to 12.8%. De plus, la méthode proposée permet un compromis entre le gain en ecacité de compression et la distorsion avec la vidéo originale non-quantiée grâce à un paramètre de régularisation. Notre technique a également été utilisée pour débruiter des vidéos tone mappées et nous avons démontré qu'elle peut être généralisée à n'importe quelle application qui nécessite une opération de tone mapping. Ces travaux ont été présentés dans le Chapitre 7 et ont été publiés dans une conférence internationale : R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, "Motion-Guided Quantization for Video Tone Mapping," Accepted in IEEE International Conference on Multimedia and Expo (ICME), 2014. 1: Spatial resolution of the sequences presented in this appendix. "p" in 1080p and 720p stand for progressive scanning.

Figure A. 1 :

 1 Figure A.1: Analysis of the Sun (left) and Tunnel (right) sequences.

  Figure A.2 (right) plots its analysis.

Figure A. 2 :

 2 Figure A.2: Analysis of the UnderBridgeHigh (left) and ColorCheckBall (right) sequences.

  Figure A.4 (right).

Figure A. 3 :

 3 Figure A.3: Analysis of the HueDisc (left) and ModernFlatAnalysis (right) sequences.

Figure A. 4 :

 4 Figure A.4: Analysis of the Temple and GeekFlat sequences.

Figure A. 5 :

 5 Figure A.5: Analysis of the LightHouse (left) and Desert (right) sequences.

Figure

  sequence is composed of 250 Full HD frames captured at the frequency of 25 fps. A car is entering a tunnel during a sunny day. The change of illumination between the two conditions is slow but at one point we can still see the outside of the tunnel while being completely accommodated to the inside. Its analysis is plotted in Figure A.7 (left).

Figure A. 6 :

 6 Figure A.6: Analysis of the Disco and GeekFlat sequences.

Figure A. 7 :

 7 Figure A.7: Analysis of the TunnelHD and FireEater2 sequences.

FireEater2The

  Figure A.7 (right).

Figure A. 8 :

 8 Figure A.8: Analysis of the Students sequence.

Figure A. 9 :

 9 Figure A.9: Analysis of the Hallway2 (left) and Hallway3 (right) sequences.

Table 3 .

 3 1 gives an overview of the temporal artifacts presented in this chapter. All the solutions provided to reduce temporal artifacts focus on ickering. Furthermore,

	techniques that rely on temporal ltering can generate other types of artifact. The
	analysis of problems occurring in video tone mapping, along with the solutions presented
	in this chapter, has been published in:
	R. Boitard, D. Thoreau, K. Bouatouch, and R. Cozot, "Temporal Coherency in
	Video Tone Mapping , a Survey," in HDRi2013 -First International Conference
	and SME Workshop on HDR imaging, 2013.
	R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, "Survey of temporal
	brightness artifacts in video tone mapping," in HDRi2014 -Second International
	Conference and SME Workshop on HDR imaging, 2014.

Table 3 .

 3 1: Summary of temporal artifacts and their causes.

	Temporal Artifact	Cause	Solution
	FA	Temporal unstability of	Global temporal
	(Global Flicker)	Parameters	ltering
	FA	Dierent spatial ltering	Pixel-wise
	(Local Flicker)	in successive frames	temporal ltering
	TN	Camera noise	Spatial and/or temporal
	(Noise)		ltering (pixel-wise)
	TBI	Change of illumination
	(Brightness)	Adaptation of the TMO
	TOI	Adaptation of the TMO
	(Object)
	THI	Saturation of color channel
	(Hue)	(due to clipping)
	TCA	Due to global
	(Contrast)	temporal ltering
	GA	Due to pixel-wise
	(Ghosting)	temporal ltering

Table 6

 6

	8

.2: PSNR of the Inter-predicted frame resulting from motion estimation. D

Table 6 .

 6 3: Average distortion for the UnderBridgeHigh sequence using the MSE metrics in the log domain (for clarity purposes, all values are multiplied by 100).

	TMO	500 kbps	2 Mbps	4 Mbps	Original
	Reinhard et al.	15.34	13.55	11.04	3.28
	Reinhard et al. + BC	27.65	21.79	18.82	4.00
	Ramsey et al.	15.39	13.63	11.12	3.38
	Ramsey et al. + BC	25.51	21.80	18.81	4.18
	TMO	125 kbps	500 kbps	1 Mbps	Original
	Reinhard et al.	9.55	7.25	6.66	2.33
	Reinhard et al. + BC	11.30	8.50	7.94	2.71
	Ramsey et al.	9.54	7.26	6.68	2.29
	Ramsey et al. + BC	11.25	8.48	7.94	2.70

Table 6

 6

.4: Average distortion for the Tunnel sequence using the MSE metrics in the log domain (for clarity purposes, all values are multiplied by 100).

Table 7 .

 7 4: Average percentile bit-rate reduction under the same PSNR when comparing the Rounding and the M GQ Inf quantization techniques. The BD-rate is computed using piece-wise cubic interpolation.

		12.8%	-40.1%	-40.6%
	Tunnel	-10.4%	-31.7%	-32.7%
	Students	-5.6%	-18.8%	-17.5%
	TunnelHD	-5.4%	-21.7%	-24.5%
	Average	-8.5%	-28.1%	-28.8%

  Results showed that only three out of the six types of artifact are dealt with. Furthermore, these techniques can generate two new types of temporal artifact. The descriptions of temporal artifact and video tone mapping was detailed in Chapter 3 and presented at two workshops:

	R. Boitard, D. Thoreau, K. Bouatouch, and R. Cozot, "Temporal Coherency in
	Video Tone Mapping , a Survey," in HDRi2013 -First International Conference
	and SME Workshop on HDR imaging, 2013.
	R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, "Survey of temporal
	brightness artifacts in video tone mapping," in HDRi2014 -Second International
	Conference and SME Workshop on HDR imaging, 2014.

  R. Boitard, K. Bouatouch, R. Cozot, D. Thoreau and A. Gruson, "Temporal Coherency for Video Tone Mapping," in Proc. SPIE, Applications of Digital Image Processing XXXV, 2012.However, when evaluated in a subjective study, our technique was reported to have too low spatial contrast when the temporal contrast was high. Indeed, as this technique preserves only the overall frame brightness coherency, it fails when the temporal contrast is of local nature. Consequently, we modied the BC post-processing to make it local.The Zonal Brightness Coherency (ZBC) technique determines constant video zones over the whole sequence, each zone being bounded by two luminance values. It preserves then the temporal coherency between the brightest zone of the sequence and all the other zones. The ZBC post-processing was evaluated in a subjective evaluation which reported that it always preserves or increases the subjective quality of the tone map video. The ZBC technique has been presented in Chapter 5 and resulted in an international journal:

R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, "Zonal Brightness Coherency for Video Tone Mapping," Signal Processing: Image

Communication, vol. 29, no. 2, pp. 229-246, 2014.

  Travaux de la thèseDurant cette thèse, nous avons en premier lieu déni les limitations de l'imagerie traditionnelle et évalué les aspects de l'imagerie HDR qui permettent de dépasser ces limitations. Une fois les diérences entre HDR et LDR bien maitrisées, nous nous sommes intéressés au tone mapping qui permet la rétrocompatibilité entre les contenus HDR et les écrans LDR. L'état de l'art du tone mapping nous a permis de comprendre qu'une multitude de Tone Mapping Operators (TMOs) existait avec des buts diérents : de la simulation du système visuel humain au plus beau rendu possible. Ces méthodes nécessitent souvent d'aner le réglage de paramètres en fonction du contenu et aucune technique ne permet d'avoir un rendu satisfaisant pour un large panel d'images. De plus, toutes ces méthodes n'ont été testées que pour des images et non pour des vidéos De plus, ces vidéo TMOs pouvaient également créer deux nouveaux types d'artefacts temporels. Le détails des diérents types d'artefacts temporels ainsi que la description des vidéo TMOs sont présentés dans le Chapitre 3 ainsi que dans deux articles de workshop : R. Boitard, D. Thoreau, K. Bouatouch, and R. Cozot, "Temporal Coherency in Video Tone Mapping , a Survey," in HDRi2013 -First International Conference and SME Workshop on HDR imaging, 2013.

	En imagerie numérique, cette adaptation se fait au niveau du capteur de la caméra
	qui transforme la luminance mesurée en valeur de pixel standardisée. Les images ainsi
	obtenues peuvent donc être directement achées sur un écran traditionnel. Cependant,
	tous les écrans qui utilisent un même standard de représentation ont des caractéristiques
	techniques diérentes, le rendu du contenu varie donc d'un écran à un autre. De plus,
	comme la caméra peut capturer une dynamique de valeurs supérieure à celle qu'un
	écran peut représenter, une partie de l'information sera forcément perdue durant cette
	transformation.	cd/m 2
	Pour résumer, l'imagerie telle que nous la représentons traditionnellement pose
	plusieurs problèmes :

tent de résoudre deux principaux problèmes : comment capturer une scène réelle avec la plus grande délité? comment reproduire l'apparence de cette scène sur un écran aux caractéristiques techniques limitées? Pour essayer de résoudre ces deux questions, la photographie analogique puis numérique n'a cessé d'évoluer. Cependant, les technologies de captation et d'achage sont toujours très limitées en termes de capacité d'achage comme illustré sur la Figure 8.1. Grâce à cette gure, nous constatons que pour capturer une partie de la gamme de luminance qui existe dans une scène réelle, une caméra a besoin de s'adapter. En eet, les caméras possèdent un facteur d'exposition (eV) qui permet de modier l'exposition d'une scène. Cependant, les dispositifs d'achage (TV, écrans, etc..) ne peuvent reproduire que des luminances appartenant à un intervalle xe donné (celui-ci dépendant du dispositif ). Enn, une caméra peut en général capturer plus que ce que l'on peut reproduire sur un écran. A partir de toutes ces observations, on comprend qu'une scène capturée par une caméra a besoin d'être adaptée aux capacités de l'écran ciblé. pendant la captation, de l'information est perdue puisque la caméra ne peut pas capturer toute la gamme de luminance d'une scène. pendant la conversion en valeur de pixel standardisée, l'information capturée par la caméra est adaptée aux capacités de représentation du standard avant d'être quantiée. HDR. La raison étant que très peu de contenu vidéo HDR existait jusque-là et que l'anage de paramètres est fastidieux pour une vidéo. Nous avons ainsi étudié le comportement des TMOs lorsqu'ils sont utilisés pour transformer une vidéo HDR. Grâce à cette étude, nous avons isolé six types d'artefacts temporels qui apparaissent lorsqu'un TMO est appliqué indépendamment à chaque image d'une vidéo. Quelques techniques ont ainsi été développées pour tone mapper des images successives d'une vidéo HDR de façon cohérente. Ces techniques, appelées vidéo TMOs, ont été testées sur le même corpus de séquence et nous avons constaté qu'elles ne permettaient de résoudre que trois des six types d'artefacts.

Acknowledgments

First of all, I would like to thank my three supervisors: Kadi Bouatouch, Rémi Cozot and Dominique Thoreau. Each of you helped me with dierent aspects of this thesis and were complimentary of each other. Thank you for the support, fruitful discussions and most importantly all the time you spent advising me.

I would also like to thanks my colleagues from Technicolor and the FRSense team to provide a work place that is more than just a place to work. I will denitely miss the coee breaks, lunch breaks and any type of breaks we had. I am really glad to have met you all.

Many thanks as well to Adrien Gruson, Mickaël Ribardière and Ricardo Marquès

Contributions

Les travaux réalisés dans le cadre de cette thèse ont permis d'apporter les contributions suivantes au domaine du video tone mapping:

Un état de l'art sur l'imagerie HDR.

Une description des diérents types d'artefacts temporels qui apparaissent lorsque l'on applique un TMO créé pour des images xes à une vidéo HDR.

Un état de l'art du video tone mapping.

Un corpus de séquences synthétiques conçu pour mettre à mal les TMOs.

Deux techniques de traitement a postériori pour réduire les artefacts temporels lors d'une opération de video tone mapping.

Appendix A HDR Sequences

In this appendix, we provide an analysis of the HDR sequences used throughout our experiments. The analysis of these sequences is presented by plotting their minimum and maximum luminance value per frame. We added the 1 st and 99 th percentile luminance value per frame to better assess the dynamic range of the sequence. The key (geometric mean) which represents an indication of the overall brightness is also plotted for each frame. These sequences were captured using dierent techniques and we propose to classify them by the research laboratory or project that captured or generated them.

The dierent spatial resolutions of these sequence are listed in Table A.1.

A.1 Max Planck Institute fur Informatiks (MPII) Sequences

Two sequences from the Max Planck Institute fur Informatiks (MPII) have been made publicly available.

Sun

The Sun sequence is composed of 860 VGA frames and illustrates a drive on a highway in the vicinity of Saarbruecken. Temporal contrast is achieved though the sun that is either in full view or obscured by trees.

A.2 OpenFootage Sequence

OpenFootage is a website that provides HDR panoramas of high resolution (10,000 by 5,000 pixels). Video sequences can be extracted from the panorama through panning in the high resolution image.