Cédric Walbrecq
email: cedric.walbrecq@mines-ales.fr

Baptiste Magnier
email: baptiste.magnier@mines-ales.fr

Isabelle Marc
email: isabelle.marc@mines-ales.fr

A Temporal Tone Mapping Adaptation: An Application using Logarithmic Camera

Keywords: high dynamic range video, tone mapping, logarithmic sensor, temporal artifacts

Noise, flicker, temporal inconsistencies disturb the tone mapping creation for High Dynamic Range image (HDR). A method to alleviate these artifacts in tone mapped HDR video sequences is presented in this paper. A temporal filtering is developed in order to minimize undesirable artifacts in the low dynamic range image reconstruction process. In the experiments, tone mapping is applied on video sequences obtained by a logarithmic camera. This temporal process provides visual comfort without any kind of blink and prevents Low Dynamic Range (LDR) videos to be disturbed by some random brightness values of isolated pixels.

I. INTRODUCTION

Capture of real world scenes, where luminance may span over six decades from bright sunny areas to dark shadows, may be done using High Dynamic Range (HDR) techniques [1]. However, as the current displays are still based on standard 8 bits technologies, it is still necessary to realize the mapping of real world luminance to 8 bits luma (brightness in a digital image). A lot of research has been done to develop Tone Mapping Operators (TMOs) for many years, mostly for still images. With the growing interest in HDR videos, specific tools have to be designed, as the straightforward approach, which consists in applying these TMOs on each successive frame of video sequences, leads to unacceptable visual artifacts such as visual noise, flickering, ghosting and brightness and color incoherencies.

Our purpose is to propose a complete system (hard + soft) to capture videos of real world scenes and to display them on a conventional standard dynamic range screen. In this paper, we propose a method to decrease temporal artifacts which can be applied after global tone mapping such as the one developed by Drago et al. [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF]. This method may be applied whenever the processing of HDR video may be impeded by non-uniform illumination, for instance, in object tracking or detection (security, industrial inspection, autonomous automotive, medicine, etc.).

II. RELATED WORKS A. HDR Capturing techniques

HDR images can be obtained by different methods. The most popular is based on conventional linear camera and multi exposition technique (also known as temporal bracketing): a standard camera with limited dynamic range may be used with alternating short and long exposures. Debevec and Malik proposed a method to merge images with different exposures into a single wide range radiance map [START_REF] Debevec | Recovering high dynamic range radiance maps from photographs[END_REF]. This technique of multi exposures requires a complete static scene when capturing the images, as even very small movement in the scene will produce a noticeable artifact, known as ghosting effect [START_REF] Srikantha | Ghost detection and removal for high dynamic range images: Recent advance[END_REF]. A lot of work has been dedicated to ghost detection and ghost removal; most of them deal with images [START_REF] Srikantha | Ghost detection and removal for high dynamic range images: Recent advance[END_REF] and more recently with videos [START_REF] Zhang | Patch-based correlation for deghosting in exposure fusion[END_REF]. But this issue may be better overcome with the use of a dedicated HDR imaging device. Hardware methods to capture HDR images include the use of multiple imaging devices, or devices with special sensors such as logarithmic sensors. These sensors are well adapted to HDR imaging, as they are able to capture a range of about six orders of magnitude with an unique exposure. They differ from conventional sensors (CCDs or CMOS) in the way the light absorbed in the sensor is converted into a voltage. When the former operate in integration mode (for each pixel, the photocurrent, i.e electric current through a photosensitive device, is integrated in a capacitor during exposure time and a charge to voltage converter is used), in the former, the photocurrent is directly converted to output voltage by using the exponential I(V) curve of MOS transistor in sub-threshold polarization mode [START_REF] Darmont | Society of Photo-optical Instrumentation Engineers[END_REF] [START_REF] Cheng | A logarithmic CMOS image sensor with adjustable photo-response[END_REF]. If logarithmic sensors are well suitable for HDR imaging, they present a noticeable disadvantage: due to the large dispersion in transistor characteristics, they are prone to noise, especially for low luminance levels.

B. Tone Mapping Operators

Numerous tone mapping operators have been developed; most of them are working on the luminance value of pixels, with no consideration of their chromatic components. They may be classified into two main categories: the global ones (which use the same function to convert HDR luminance values into Low Dynamic Range (LDR) values for each pixel) and the local ones (which adapt the conversion function by taking in consideration the luminance values of the pixel and of its neighborhood). Each category exhibits advantages and disadvantages: when local operators often lead to a better preservation of details, the computational cost of the global ones is generally lower, and in consequence, they may be of greater interest for video treatment. When applied to videos, global operators are also more robust to temporal artifacts, which are likely to appear when mapping functions are applied on successive frames. A description of these artifacts can be found in [8]. Several studies devoted to qualitative TMO assessment emphasized the fact that the most disturbing ones seem to be flickering and Temporal Brightness Incoherency (TBI) [9], [10]:

• Flickering is mainly caused because TMOS adapt their mapping using image statistics, when variation between the statistical brightness information (such as the maximum/minimum/ average brightness, the brightness histogram, etc.) of consecutive frames appears, similar HDR luminance are mapped to different LDR luma. • Temporal Brightness Incoherency (TBI) artifacts occur when the relative brightness between two frames of an HDR sequence is not preserved during the tone mapping.

For each frame, as a TMO uses all its tied available range, the temporal brightness relationship between frames is not preserved throughout the tone mapping operation. Different methods were proposed to overcome these issues. Ramsey et al. [11] propose to detect flickers in video sequences: flicker occurs when luminance variation between two consecutive frames is beyond a given threshold; in the other cases; the change in luminance is assumed to correspond to true lighting condition changes. When flicker is revealed, luminance is averaged over a number of frames that depends on its value, and when flicker is dismissed, no average is performed. Photographic TMO as proposed by Reinhard and al. [12] is then applied. Another approach can be found in [START_REF] Mantiuk | Display adaptive tone mapping[END_REF]: temporal filtering is directly done on the tone map curve instead of on average values. Guthier et al. [10] have developed a post processing method which can be applied whatever the TMO is used. They also recommend to detect flickers before performing an iterative brightness adjustment, and in their work, flicker detection threshold is based on Just Noticeable Differences (JND) according to Stevens' law. All these methods focus on flicker reduction, and are not able to cope with TBI artifact, i.e. their responses to light condition variation are not satisfactory. Boitard and al. [8] intend to address TBI problem, by preserving the same ratio between luminance and log average luminance before and after tone mapping. However, their method decreases spatial contrast, as the available dynamic range is scaled down when lighting transitions occur.

III. A METHOD FOR TEMPORAL ADAPTATION OF GLOBAL TONE MAPPING

Our method is inspired by the tone mapping operator proposed by Drago et al. [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF]. It is a global operator which is designed to be efficient whatever the lighting conditions are, with a relatively small computational cost, and easily set to fit the display characteristics.

A. Adaptive Logarithmic Tone Mapping Scheme

The TMO developed in [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF] is based on psychophysical considerations: according to Weber-Fechner law, the response of the human visual system to light stimuli may be modeled by a logarithmic curve for a wide range of luminance levels. The first step is to gather luminance value for each image pixel, thanks to a conversion from RGB to XYZ spaces [START_REF] Smith | The CIE colorimetric standards and their use[END_REF], as the Y component represents the physical luminance.

Then, the visual adaptation process is taken into account by scaling the real luminance by the adaptation luminance level. This adaptation level is expressed as the log average value across the whole image:

L m = exp   1 m • n • m i=1 n j=1 log(Y (i,j) )   , (1)
where m and n represent the number of lines and columns of the image respectively. The real world luminance is scaled by this adaptation level L m :

L w = Y L m . (2)
The mapping curve for each image pixel luminance is a logarithmic function of this scaled luminance and of its maximum value in the current frame. Drago et al. bring out that while the logarithm with base 10 strongly compresses high luminance levels, logarithm with base 2 provides good contrast and better preserves details in dark areas. Consequently, they propose to use different bases for the logarithm function in the mapping curve, according to the pixel luminance value. The LDR luma value is given by the following expression:

L d = L dmax • 0.01 log 10 (L wmax + 1)
• log(L w + 1)

log 2 + (( Lw Lwmax ) log(b) log(0.5) ) • 8 , (3)
where L w represents the scaled world luminance, L max the scaled maximum value within each frame, L dmax corresponds to the maximum luminance capability of the displaying medium. The use of the L dmax value as a parameter of the mapping function enables to adapt the luminance to a given device. The b factor influences the path from a logarithmic base to another, consequently, the way low to medium luminance levels are mapped. The authors advocate for a value of b between 0.7 and 0.9. Once the novel value of L d is obtained, this value is tied to the new component Y in the XYZ space. Then, the X and Z values are modified in order to keep the same ratio between color channels as before compression. The last step is the conversion back to 8 bits RGB color space (LDR). In a nutshell, this method rescales the HDR image in a logarithmic manner, in order to maximize the contrast between the different brightness levels. The maximum luminance of the image is used as the upper limit and the bias tunes for the contrast adaptation.

This algorithm can be applied to each frame of video sequences, as it can be performed in real time, but temporal defects are likely to appear in LDR sequences, see Fig. 3 to 5.

B. A Temporal Adaptation Tone Mapping Method

The main drawback of the previous detailed approach is that it does not take into account discontinuities of luminosity in consecutive frames. The proposed method considers the displacement of the observed scene dynamic along the time, i.e., along the different frames for a video. In consecutive frames, the maximum of HDR image constantly changes. The aim here is to limit high variations with a logical coherence between consecutive frames by introducing temporal tone mapping operators.

This temporal process is based on the maxima average on the T last consecutive frames. As the algorithm presented in eq. 3, the temporal tone mapping is applied to the luminance component Y using the L m value computed as in eq. 1.

Furthermore, the maximum value L max of the Y image is computed. Then, the average of the T last frames L max is computed:

L mT = 1 T • t i=0 L max (T -i), ( 4
)
where T represents the temporal values. This temporal average and the Y channel are scaled by L m , the logarithmic luminance average:

L maxT = L mT L m . (5)
Thus, the new pondered maxima are introduced into the tone mapping equation:

L dT = L dmax • 0.01 Log 10 (L maxT + 1)
• log(L w + 1)

log 2 + (( Lw L maxT ) log(b) log(0.5) ) • 8 . ( 6
)
The different steps of the temporal algorithm are summarized in Fig. 1. Following experimental results illustrate the significance of the time inclusion in the tone mapping process.

IV. EXPERIMENTAL VALIDATION

A. Experimental method

A set of videos was captured, which is intended to contain some typical variations in lighting conditions:

• no variation (the objects, the light source and the camera are fixed, cf. Fig. 2), • small blinking LED source, as shown in Fig. 3, • abrupt variations due to turn on or turn off of a halogen lamp and a static camera, as illustrated in Fig. 4 and5.

Videos were captured by a logarithmic sensor camera (NIT MC1003-1VC; 1280×1024 pixels) in indoor environments. These videos are processed according to eq. ( 3), and eq. ( 6). The parameters remain the same in both cases: L dmax = 100cd/m 2 and b = 0.85. For the time adapted one, filtering is performed on 17 frames; it corresponds to a temporization of about 0.5s when our operator is running on a I5-4200U processor. b) Flicker reduction: A small blinking source is produced in the second experiment. Originally, only the few pixels which correspond to LED image are modified, while the overall brightness does not change at all, as it can be seen from Fig. 3(a), the HDR maximum luminance follows the source, but the average luminance is constant. However, Fig. 3 (c),(e) and (g) show that the video processed by original TMO suffers from flickering, while our time adapted operator prevents from this flicker generation (see Fig. 3 (d),(g) and (h)). Consequently, as the Drago algorithm readjusts the image brightness as a function of the image maximum, the LDR luma average behaves like the range level should be and inversely, the range should not be stable (because of the flicker, as in [START_REF] Vonikakis | A comparison framework for the evaluation of illumination compensation algorithms[END_REF]). On the contrary, the temporal algorithm leads to a stable luma average and a LDR range similar to the HDR evolution.

c) Temporal Coherency: In the third video sequence, the overall luminance undergoes a sharp transition when halogen lamp is turned on and as expected, HDR Dynamic range increases as average value does too. With original TMO processing, video exhibits strong inconsistencies at the time the lamp is turned on, Fig. 4(c) shows that the background suddenly becomes very dark. On the contrary, with time adapted TMO, the light transition does not lead to immediate background modification. A smooth transition is observed towards the final state where all the details can be seen in the dark and clear areas as well. From a perceptual point of view, transformation of lighting discontinuity into slow change makes it easier to keep visual adaptation and prevents from glare. A similar behavior is observed when the lamp is suddenly turned off, as illustrated in Fig. 5.

V. CONCLUSION

In this article, a method to alleviate temporal artifacts (noise, flicker, temporal inconsistencies) in tone mapped HDR video sequences is presented. Temporal filtering is applied to the maximum luminance value of each frame before its use in the tone mapping function. Combining this method with the acquisition by logarithmic sensor may lead to real time HDR video processing. This temporal process provides visual comfort without any kind of blink and to give access to the real world luminance variation. Future works concern applications, including video assistance as user interfaces, autonomous vehicles and robotic tasks.

Fig. 1 .

 1 Fig. 1. Summary of the temporal tone mapping adapatation.

Fig. 2 .

 2 Fig. 2. Average of the luma level and range level evolution concerning a static scene. The HDR scaled luminance is obtained by linear compression.

Fig. 3 .Fig. 4 .

 34 Fig. 3. Tone mapping responses to local blinking source.

Fig. 5 .

 5 Fig. 5. Tone mapping responses to sharp decrease in overall luminance.