Aakanksha Rana

Directeur De

Frederic Dufaux

Giuseppe Valenzise

Professeur Jury Mme Sabine Susstrunk

Senior Lecturer, M Rafal Mantiuk

Professeur Mme Céline Loscos

Professeur M Alan Chalmers

M Frédéric Dufaux

M Giuseppe

Spécialité « Signal et Images » présentée et soutenue publiquement par

Keywords: HDR, Local Features, Computer Vision, Deep Learning, GAN HDR, Caractéristiques locales, Vision par ordinateur, Apprentissage approfondi, GAN

High Dynamic Range (HDR) imaging enables to capture a wider dynamic range and color gamut, thus enabling us to draw on subtle, yet discriminating details present both in the extremely dark and bright areas of a scene. Such property is of potential interest for computer vision algorithms where performance degrades substantially when the scenes are captured using traditional low dynamic range (LDR) imagery. While such algorithms have been exhaustively designed using traditional LDR images, little work has been done so far in context of HDR content. In this thesis, we present the quantitative and qualitative analysis of HDR imagery for such task-specific algorithms.

This thesis begins by identifying the most natural and important questions of using HDR content for low-level feature extraction task, which is of fundamental importance for many high-level applications such as stereo vision, localization, matching and retrieval.

By conducting a performance evaluation study, we demonstrate how different HDR-based modalities enhance algorithms performance with respect to LDR on a proposed dataset.

However, we observe that none of them can optimally to do so across all the scenes.

To examine this sub-optimality, we investigate the importance of task-specific objectives for designing optimal modalities through an experimental study. Based on the insights, we attempt to surpass this sub-optimality by designing task-specific HDR tone-mapping operators (TMOs).

In this thesis, we propose three learning based methodologies aimed at optimal mapping of HDR content to enhance the efficiency of local features extraction at each stage namely, detection, description and final matching. By spatial adaptation of a given filter using a regression based approach, we showcase our three models learn to adaptively map the HDR content by bringing invariance to luminance transformations at all the aforementioned stages. We evaluate the performance of all the learning-based models on a proposed HDR dataset of 8 (indoor/outdoor) real scenes where it outperforms existing mapping functions across different feature extraction algorithms.

Finally, this thesis presents three end-to-end deep learning based generic tone mapping (DeepTMOs) designs which cater to desired task-specific characteristics over a wide spectrum of linear input HDR images. With the goal of avoiding any specific filtering dependency with a differentiable design as required in previously proposed models, our DeepTMOs serves as a baseline which can be fine-tuned for any computer vision specific task at hand.

Table of Contents

List of Figures

1.1 (a) shows an example case from [119], where matching of salient points (common in both) is shown using blue lines between two images of same scene taken at different hours of the day. (b) shows the efficiency measure Repeatability rate RR over a large dataset of LDR day/night images using the state-of-the-art techniques. The crest and the troughs in the curves illustrates that image captured during the daylight matches well with only other day-time images and not with the ones captured in the dark. . . . . . . . . . . . . . .

2.1

Taken from [START_REF] Debevec | Recovering high dynamic range radiance maps from photographs[END_REF]. Multiple exposures of a Church scene along with final Radience Map. .

3.3

Relative gains by best LTM, GTM (abbreviated using Table 3.1), Linear, Log and Pu HDR encodings with respect to LDR for different test datasets (scenes indicated by progressive numbers on x-axis). The dotted line shows the absolute R-scores of LDR. . . . . . . . . 3.4 Scatter plots for HDR based formats (TMOs abbreviated using Table 3.1) with respect to LDR on proposed dataset using detector: Harris and SURF . . . . . . . . . . . . . . 3.5 Scatter plots for best performing GTM and LTM (abbreviated using Table 3.1) with respect to HDR-Pu encoding on proposed dataset using detector: Harris and SURF . . .

5.7

Day/Night matching using SURF. Row I: 2 HDR images from Invalides scene are displayed after log scaling [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF]. Correct and incorrect matches are shown with yellow and red lines respectively. Green lines represent the special case of mismatch due to repetitive structure.

Row II: the feature matching using our proposed DesTMO (11 correct and 3 incorrect matches). Row III: using Reinhard TMO (3 correct and 11 incorrect matches). Row IV: using MantiukTMO (4 incorrect and 3 correct matches). . . . . . . . . . . . . . . . . 5.8 Optimal Tone Mapping Design. The tone mapping function is modulated by the SVR-based guidance model, which predicts optimal parameter maps using the characteristic features. . . . . . . . . . . . . . . . . . . . . . . . . 5.9 Generation of Training Set. Ground-truth parameter maps are generated by minimizing the total energy determined from a set of images of the same scene, undergoing lighting variations, using the procedure in Section 5.4.2. . . . . . . . . . . . . . . 5.10 Repeatability Rates (RR) computed for OpTMO using a corner (Harris) and a blob (SURF) keypoint detector. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.11 Keypoint Detection I: Average Repeatability Rates (AvgRR) computed on different TMOs using various keypoint detection schemes. The average is calculated over all test scenes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.12 Keypoint Detection II: Average Repeatability Rates (RR) computed using BTMO [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF], DetTMO [START_REF]Learning-based Adaptive Tone Mapping for Keypoint Detection[END_REF] and the proposed OpTMO for each test scene using Harris keypoint detector. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OpTMO. The cluttered keypoints in DetTMO are highlighted using the red squares. 5.15 Descriptor Matching I computed on different TMOs using SURF, SIFT, FREAK, BRISK descriptor extraction schemes. The average is calculated over all test scenes. 5.16 Descriptor Matching II. Matching Score comparison between BTMO [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF],

OpTMO, DetTMO [START_REF]Learning-based Adaptive Tone Mapping for Keypoint Detection[END_REF] and DesTMO [START_REF]Learning-Based Tone Mapping Operator for Image Matching[END_REF] over all the scenes in the HDR dataset using SURF feature extraction scheme. . . . . . . . . . . . . . . . . . . . . . .

Descriptor Matching III. Matching Score comparison between DesTMO vs

OpTMO over all the scenes in HDR dataset. The points represented using o corresponds to FREAK feature detection scheme and 2 corresponds to SURF scheme. 5.18 Image Matching I. mAP % scores for the 9 different LDR modalities using 4 feature extraction schemes. Scores are averaged over 8 lighting change datasets. . . 5.19 Image Matching II. Day/Night matching using SURF. Row I: 2 HDR images from Invalides scene are displayed after log scaling [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF]. Correct and incorrect matches are shown with yellow and red lines, respectively. Green lines represent the special case of mismatch due to repetitive structure. Row II: the feature matching using our proposed OpTMO (21 correct and 3 incorrect matches). Row III: using DetTMO (13 correct and 6 incorrect matches). Row IV: using DesTMOusing (11 correct and 3 incorrect matches). Row V using Reinhard TMO (3 correct and 11 incorrect matches). Row VI: using MantiukTMO (3 correct and 4 incorrect matches). 5.20 generator are trained alternatively, first a gradient step of discriminator then of generator. While the discriminator is trained to discriminate between real and fake image pairs, the generator learns to fool the discriminator by producing synthetic tone mapped images. By doing this, the generator effectively models the underlying distribution of real ground-truth tone mapped images, thus yielding high quality results once completely trained.

6.2 Here we show detailed architecture of both the discriminator and generator of DeepTMO-R and DeepTMO-S. The only difference for DeepTMO-S is the addition of skipped connections in the case of generator. The generator is framed as an encoder-decoder architecture, where the input HDR image is first passed to an encoder, which subsequently down-samples it to a compact representation. This representation is then forwarded through the decoder which up-samples it to the size of the input HDR. While the encoder consists of Convolution front end component G (F ) and first five residual blocks G (R) , the decoder is composed of next four residual blocks G (R) and a deconvolution component G (B) . Residual Blocks consist of two sequential convolution layers applied to the input, producing a residual correction that is in turn added to the input to yield the final output. Discriminator consists of a patchGAN [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF][START_REF] Li | Precomputed real-time texture synthesis with markovian generative adversarial networks[END_REF] architecture which is applied patch wise on the concatenated input HDR and tone mapped LDR pairs. The final prediction is an average of all the patches over the image. . . . . . . . . . . . . . . . . 6.4 Compared between the three proposed architecture DeepTMO-R, DeepTMO-S, DeepTMO-HD (I). From the insets, DeepTMO-R suffers from blurriness issues in the wall and lowermost window panels. DeepTMO-HD and DeepTMO-S both are able to preserve the finer details, though the window panels are much more clearly visible in case of DeepTMO-HD . . . . . . .

Comparisons between three architectures (II). As seen in the inset, while

DeepTMO-R simply results in blurred outputs in the bark of tree, DeepTMO-S tries to refine them but is faced by checkerboard artifacts [START_REF] Gao | Pixel deconvolutional networks[END_REF][START_REF] Odena | Deconvolution and checkerboard artifacts[END_REF]. The DeepTMO-HD provides best results amongst the three methods while preserving the fine details, contrast and sharpness in the image. . . . . . . . . 6.6 Comparison of our method with the respective BestTMOs based on the TMQI scores with the highlighted zones for scene: The Canadian Falls (row I), The Grotto (row II) and the Bar Harbor Sunrise (row III). Zoom-ins for each scene highlights mapping outputs for DeepTMO-HD and the respective BestTMO for the corresponding HDR-linear input. We notice that our model has no saturation effect in waterfall (row I), preserves finer details in sky (row II) and effectively balances luminance for the house (row III). The TMQI scores for each scene are provided alongside each TMO. . . . . . . .

List of Tables

Introduction

High Dynamic Range (HDR) technology has gained immense popularity for its ability to represent a wide range of colors and luminous intensities present in real-world environments [START_REF] Dufaux | High Dynamic Range Video: From Acquisition, to Display and Applications[END_REF][START_REF] Mantiuk | High Dynamic Range Imaging[END_REF]. In a sense, these images enable us to draw on subtle, yet discriminating details present both in the extremely dark and bright areas of a scene, which would otherwise get lost in traditional low dynamic range (LDR) imagery. With recent advancements in artificial intelligence, be it in the form of self-driving cars or automated surveillance devices, such high-contrast preserving HDR property is quintessential for the proficiency of underlying computer vision algorithms. In other words, these algorithms should be able to analyze effectively, each and every region in a scene without much uncertainty.

Though such algorithms are exhaustively customized for LDR images captured under different conditions, they fail miserably in high-contrast scenes having high or low luminescence [START_REF] Gupta | SMD: A Locally Stable Monotonic Change Invariant Feature Descriptor[END_REF][START_REF] Verdie | TILDE: A temporally invariant learned detector[END_REF][START_REF] Zagoruyko | Learning to compare image patches via convolutional neural networks[END_REF][START_REF] Zhou | Evaluating local features for day-night matching[END_REF]. Since high-contrast scenes are extremely common in the real world, it becomes quite critical in cases like automated vehicles where human lives are involved. Thus, it necessitates the application of HDR technology for viability of computer vision algorithms. While several algorithms have been exhaustively designed for interpreting over or under exposed scenes using LDR images, little work has been done so far in context of HDR content.

This thesis is focused on the analysis of 'enriched' HDR images for the benefit of low-level visual features correspondence problem, which is the bedrock for many other high-level computer vision algorithms including, registration and stereo vision, motion estimation and localization, matching, retrieval and recognition of objects and actions.

More specifically, the thesis investigates the fundamental challenges involved in using HDR imaging and derives the optimal ways of using HDR content for enhancing the robustness of such tasks. shows an example case from [119], where matching of salient points (common in both) is shown using blue lines between two images of same scene taken at different hours of the day. (b) shows the efficiency measure Repeatability rate RR over a large dataset of LDR day/night images using the state-of-the-art techniques. The crest and the troughs in the curves illustrates that image captured during the daylight matches well with only other day-time images and not with the ones captured in the dark.

Introduction

Context and Objectives

The robustness of computer vision applications can be construed from a three-level feature hierarchy namely, low-level, mid-level and high-level. Since the latter two levels build heavily from the former, the efficacy in low-level analysis has been considered quintessential [START_REF] Yi | LIFT: Learned Invariant Feature Transform[END_REF].

Generally, the low-level analysis is defined and evaluated in terms of 'visual features correspondence' problem [START_REF] Schmid | Evaluation of interest point detectors[END_REF]. The problem is formulated by drawing correspondence between images using visual features extraction algorithms. Visual features are the discriminative signatures that contain local information from the salient locations in the images. The correspondence between these features defines the 'match-ability' between the two contents.

An example from [START_REF] Zhou | Evaluating local features for day-night matching[END_REF] is shown in Figure 1.1 (a), depicting the correspondence between a day and night scene using LDR images.

Several attempts including local [START_REF] Gupta | SMD: A Locally Stable Monotonic Change Invariant Feature Descriptor[END_REF][START_REF] Tang | A novel feature descriptor invariant to complex brightness changes[END_REF][START_REF] Wang | Local intensity order pattern for feature description[END_REF], global [START_REF] Shan | Illumination normalization for robust face recognition against varying lighting conditions[END_REF] normalization models and learning-based method [START_REF] Verdie | TILDE: A temporally invariant learned detector[END_REF][START_REF] Yi | LIFT: Learned Invariant Feature Transform[END_REF], have been made to ensure better luminance invariant designs in LDR imagery. However, these techniques are practically inefficient to completely compensate the loss of information or comprehend the change in spatial configurations of objects present in a scene. Consequently, these algorithms fail to find true correspondences between similar objects and result in sharp decline in performance. An example [START_REF] Zhou | Evaluating local features for day-night matching[END_REF] of day/night matching is shown in Figure 1.1, where in the performance scores of state-of-the- One alternative would be to optimize each feature extraction pipeline for HDR images.

But it would be quite impractical and cumbersome specifically for existing learning-based pipelines which would require a large amount of geometrically-calibrated HDR dataset.

Not to mention, it might complicate the direct plug-in possibilities with existing mid-level and high-level computer vision pipelines.

In this thesis, we hence opt for other solution. We concentrate on the HDR images on the input side and explore which is the best way to employ such images in LDR-optimized feature extraction algorithms.

Some HDR based studies [2,[START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF] have recently investigated the impact of using HDR images on features detection performance. Since the algorithms are LDR-optimized, they first convert the HDR content into an LDR image using some Tone-Mapping Operators( TMOs) and then apply feature detection techniques. These studies however, do not explore other modes of using HDR (such as linear) and also lacks the impact of using different varieties of existing TMOs.

Research in HDR imagery has always been addressed from a perceptual perspective point of view. Therefore, all modes of using HDR, referred as 'modalities' in this thesis, have been accustomed to human-vision attributes [START_REF] Cadik | Evaluation of HDR tone mapping methods using essential perceptual attributes[END_REF] e.g. preserving image aesthetics, contrast etc. One common way of assessing the HDR content is via tone mapping. By definition, TMOs are the models aiming to map HDR content in a suitable 8-bit LDR representation for displaying content on standard display screens. For instance, a popular technique involves the compression of estimated luminance e.g., using edge preserving filters such as bilateral [START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF] from HDR scenes in order to produce a visually pleasing tone-mapped output.

Conceptually, perceptual objectives are quite unrelated to the task-specific performance criteria such as precision score for feature correspondence. Unlike visual perception, the feature extraction pipelines follow strict designs to develop invariance in sparsely located what are the best possible ways of using HDR content ?

Attempting to answer the aforementioned questions, in this thesis we proceed with the following step-wise methodology:

1. We first investigate the HDR and its several corresponding modalities using stateof-the-art feature extraction approaches subject to their covariance and invariance under drastic lighting variations. By proposing a geometrically calibrated dataset, the analysis of various forms of HDR inputs is based on an elaborated study of their performance on two key feature stability stages: (i) keypoint detection, (ii) descriptor extraction.

2. Based on state-of-the-art techniques of local feature extraction, we design the optimal methodologies to use HDR content aiming at facilitating stable and efficient correspondences between grayscale images in real-world luminance conditions, which are quite challenging for traditional LDR images. These luminance conditions comprise of substantial changes in terms of day-night lighting variations in outdoor scenes and changes in position of strong reflectors resulting in highly saturated regions in indoor scenes.

3. We evaluate the performance of the proposed optimal methodologies of using HDR content and compare them to the state-of-the-art approaches in image matching scenarios together with a standard repeatability and feature distinctiveness measures.

Contributions

The following contributions, mainly to the field of HDR image analysis, are presented in this thesis. The published articles are reported in List of Publications in Section 7.2.

1. We investigate how much gains can HDR bring over LDR for the keypoint detection task, and which are the best modalities of using HDR to obtain it. To this end, we additionally capture a dataset with two scenes having a wide range of illumination conditions. This contribution is has been presented in the following article:

Structure of the thesis

This thesis is structured into 7 chapters • Finally, in Chapter 7, we present concluding remarks and briefly describes future work perspectives.

•

Introduction

Chapter 2

Background and State of the Art

High Dynamic Range (HDR) imaging has been a subject of interest in graphics community over the past decades, inspiring to capture and reproduce a wide range of colors and luminous intensities of real world on a digital canvas. Primarily, the research in this domain started with problems in generation, acquisition and display. An excellent brief overview can be found in [START_REF] Chalmers | Hdr video past, present and future: A perspective[END_REF][START_REF] Mantiuk | High Dynamic Range Imaging[END_REF]. A simple method of HDR capture involves multiple exposure images of the same scene taken at different time-exposure settings. To display such scenes on standard display screen, a variety of Tone Mapping Operators (TMOs) have been designed, promising the most honest representation of real world luminosity and color gamut. Several processing methodologies such as de-ghosting [START_REF] Debevec | Recovering high dynamic range radiance maps from photographs[END_REF] have also been addressed to provide a refined and artifact-free HDR representation. However, HDR imagery has not been fully explored for computer vision problems such as feature extraction.

In this chapter, we first briefly describe the classical way of handling HDR content, where and how it has been used in the computer vision applications. Then, we provide the details of feature extraction techniques which have been addressed throughout the thesis. camera response curve and normalizing by the exposure change, we obtain a single HDR image by weighted averaging of pixel values across these different exposures. While various weighting strategies have been proposed in the literature, we adopted the setting proposed in [START_REF] Debevec | Recovering high dynamic range radiance maps from photographs[END_REF] for capturing our dataset.

HDR Imaging

HDR imaging for Display

Conventional display technology assume that the input image is LDR. In order to compress the dynamic range of an HDR image to LDR, a great variety of TMOs addressing different perceptual objectives have been proposed in the past years.

Tone mapping operators have been classified into several categories principally based upon how they handle the contrast, color and luminosity in a given HDR scene [START_REF] Banterle | Advanced High Dynamic Range Imaging: Theory and Practice[END_REF]. Overall, these algorithms have been classified into global and local approaches. The global methods such as [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF][START_REF] Larson | A visibility matching tone reproduction operator for high dynamic range scenes[END_REF][START_REF] Schlick | An adaptive sampling technique for multidimensional integration by ray-tracing[END_REF] apply the same compression function to all the pixels of an image. For the local techniques such as [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Pattanaik | Adaptive gain control for high dynamic range image display[END_REF][START_REF] Tumblin | Two methods for display of high contrast images[END_REF], a tone-mapped pixel depends on the values of neighboring pixels. Even though global approaches are faster to compute, their resulting LDR outputs do not maintain adequate contrast in the images and thus the scene appears somewhat washed out. The local tone mapping functions, conversely do not face these issues and are generally capable of handling contrast ratios, meanwhile preserving the details. However, these operators result in some prominent 'halo' effects around the high frequency edges, thereby giving unnatural artifacts in the scenes. Another kind of function is a perceptual mapping operator [START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF][START_REF] Fattal | Gradient domain high dynamic range compression[END_REF][START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF] which, inspired from the human visual system, models attributes such as adaptation with time, discrimination at high contrast stimuli and gradient sensitivities. Although these methods yield detailed outputs at high computational cost, the aesthetic appeal of generated images is questionable. All these tone mapping methods have aimed to produce images which are quite close to what an individual would perceive in reality. The performance evaluation of these TMOs have been widely studied only from a perceptual point of view [START_REF] Cadik | Evaluation of HDR tone mapping methods using essential perceptual attributes[END_REF][START_REF] Ledda | Evaluation of tone mapping operators using a high dynamic range display[END_REF] and TMOs, generally, has been used only for display applications.

Fine tuning of parameters to enhance the perceived visual quality of tone mapped image has been previously explored in the TMO literature [START_REF] Banterle | Advanced High Dynamic Range Imaging: Theory and Practice[END_REF][START_REF] Reinhard | High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting[END_REF]. Mostly, such parameters

HDR Imaging

were tuned either by a trial-test or grid-search based approach to yield favorable outputs for a wide variety of scenes [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF][START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF][START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF]. Although some works even propose to automate the parameter selection [START_REF] Reinhard | Parameter estimation for photographic tone reproduction[END_REF], the tuned values are applied globally over the scene.

HDR Imaging for Computer Vision Applications

Literature of HDR imaging applied to computer vision problems is not very vast. It is only recently that HDR imaging has been considered in the computer vision applications such as local feature analysis [START_REF] Bronislav | Evaluation of feature point detection in high dynamic range imagery[END_REF][START_REF] Kontogianni | HDR imaging for feature detection on detailed architectural scenes[END_REF], video surveillance [START_REF] Boschetti | An optimal video-surveillance approach for HDR videos tone mapping[END_REF][START_REF] Jinno | Tone-mapping for an hdr surveillance system using SIFT features[END_REF] and photogrammetric applications [START_REF] Suma | Evaluation of the effectiveness of HDR tone-mapping operators for photogrammetric applications[END_REF]. In the following, we briefly describe various applicative scenarios where HDR imagery has been proposed to enhance the task specific performances.

1. Local features: Considering both detection and description stages, in [START_REF] Chermak | Enhanced feature detection and matching under extreme illumination conditions with a hdr imaging sensor[END_REF][START_REF] Chermak | HDR imaging for feature tracking in challenging visibility scenes[END_REF],

the added value of using HDR video has been studied in the context of matching in outdoor locations, as well as pedestrian and vehicle tracking. In [START_REF] Chermak | HDR imaging for feature tracking in challenging visibility scenes[END_REF], authors compared the feature matching performance of SURF and SIFT descriptors using a dataset of both indoor and outdoor HDR (16-bit) images. Another interesting work has been carried out by Pribyl et al. [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF], where authors presented an evaluation of the repeatability of state-of-the-art keypoint detectors on images under different transformations (lighting, viewpoint, distance) for different LDR/HDR modalities, including simple global and local TMO's but not the original HDR values. Only based on description assessment, [START_REF] Cui | Robust point matching in hdri through estimation of illumination distribution[END_REF] presented a normalization approach, where TMO is used to remove lighting-dependent information from an HDR picture, and leaving only the object's texture. In [START_REF] Cui | Robust point matching in hdri through estimation of illumination distribution[END_REF], results are shown in terms of SIFT descriptors matching performance, on a limited dataset of two images, in comparison to two popular TMO's.

2.

Tracking and Video Surveillance: HDR imaging has attracted interest in the field of video surveillance. Early work on analyzing TMO for surveillance applications was carried out by [START_REF] Boschetti | An optimal video-surveillance approach for HDR videos tone mapping[END_REF], who propose to combine the properties of local and global TMO's for object detection and tracking. However, that work lacks the comparison in terms of detection accuracy with other TMO's. In [2], an interesting scenario of enhanced people detection and tracking in indoor HDR scenes is presented using only one sequence.

3. Photogrammetric Applications: Suma et al. [START_REF] Suma | Evaluation of the effectiveness of HDR tone-mapping operators for photogrammetric applications[END_REF] presented the added value of using HDR imagery and evaluated the performance of different TMOs in the context of photogrammetric applications by estimating the enhanced count of features in different TMOs over LDR. [START_REF] Kontogianni | HDR imaging for feature detection on detailed architectural scenes[END_REF] made similar investigation with the enhanced number of local invariant features on detailed architectural scenes in HDR over LDR images.

Note that the number of detected feature points is not itself a sufficient indicator of detection performance.

Background and State of the Art

4. Face detection: In [START_REF] Korshunov | Impact of tone-mapping algorithms on subjective and objective face recognition in HDR images[END_REF], authors evaluated TMOs for face recognition applications based on the subjective and the objective evaluation metrics. [START_REF] Pereira | Automatic face recognition in hdr imaging[END_REF] investigated the power of high contrast tone mapped images for automatic face recognition using sparse representation techniques.

5. Privacy Protection: Rerabek et al. [START_REF] Rerabek | Evaluation of privacy in high dynamic range video sequences[END_REF] considered the implications of having HDR content on privacy protection; a subject of great practical interest in surveillance scenarios.

One commonality amongst all these studies is the use of existing perception-based TMOs. These techniques have been directly used to convert HDR images to LDR. It is difficult to contemplate whether the adopted procedure is optimal or not. In Chapter 3, we discuss this question with experimental evaluations for feature extraction algorithms.

Additionally, most of the studies were tested on small sets of 1 or 2 scenes.

Local Visual Features

Feature extraction algorithms play a critical role in several computer vision pipelines.

As discussed in Chapter 1, most of these algorithms are optimized for LDR imagery.

Literature confining these algorithms is immense but essentially revolves around its two stages, namely keypoint detection and descriptor extraction. Keypoint detection methods look for covariant salient locations in a scene that can be repeatedly detected when the scene is undergoing drastic geometrical and photometric transformations [START_REF] Harris | A combined corner and edge detector[END_REF][START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF][START_REF] Schmid | Evaluation of interest point detectors[END_REF]. Later, descriptor extraction algorithms are applied to extract discriminative invariant signatures from these selected keypoint locations [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF][START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF][START_REF] Yi | LIFT: Learned Invariant Feature Transform[END_REF].

Over the past couple of decades, these algorithms have shifted the course from handcrafted to learning based mechanisms, but the competitiveness of some classical methods such as SIFT, SURF can be observed over the vast range of transformations [START_REF] Zhou | Evaluating local features for day-night matching[END_REF]. In the following, we discuss various detection and descriptor extraction algorithms which we have adopted to design the evaluation framework throughout the thesis.

Keypoint Detection

Algorithms for keypoint detection, in general, have been categorized in corner [START_REF] Harris | A combined corner and edge detector[END_REF][START_REF] Verdie | TILDE: A temporally invariant learned detector[END_REF] and blob detectors [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF]. We discus the principles in designing the corner and blob detectors, in the following.

The concept of corner-like keypoint detection methods has gained popularity for lowlatency vision tasks due to high speed, less computational complexity and competitive accuracy [START_REF] Tuytelaars | Local Invariant Feature Detectors: A Survey[END_REF]. By definition, corners exhibit low correlation with neighboring pixels in all directions. The most basic and widely adopted corner detectors [START_REF] Förstner | Detecting interpretable and accurate scale-invariant keypoints[END_REF][START_REF] Harris | A combined corner and edge detector[END_REF][START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF]

M = I 2 x Ixy Iyx I 2 y , (2.1)
where each component represent the directional derivative. Thereafter, different methods are proposed in the literature to localize the extrema 'keypoint' [START_REF] Schmid | Evaluation of interest point detectors[END_REF]. [START_REF] Harris | A combined corner and edge detector[END_REF] describes the response R point score for each pixel x by computing the associated eigenvector and the directional intensity variation, given as:

R(x) = det{M (x)} -k • tr{M (x)} 2 , (2.2)
where k is tuned empirically. This method is not only efficient in practice for real-time corner detection, but also optimal for locating center of junctions and circular symmetric structures [START_REF] Harris | A combined corner and edge detector[END_REF]. For [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF], the algorithm relies on the same aforementioned second moment matrix M , but explicitly computes its eigenvalues different from the previous Harris detector using the following function

R = min(λ 1 , λ 2 ). (2.3)
Although this enhances the computational requirements, the feature points detected are better localized.

Although the corner detectors are computationally fast and robust to variations such as translation and rotation, their designs are weak to handle scale variations. Additionally, since most of them are located on the object boundaries, corner points are prone to failure with scene content changing its spatial configuration such in lighting variations. To address this, blob detection methods have been designed to detect regions. Blob detectors have been discussed in details in [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF]. In this thesis, we have mainly focused on widely adopted SIFT and SURF methods to evaluate our models.

SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] is one of the first and most commonly used blob detectors which is based on the principles of Laplacian of the Gaussian (LoG) (L). The algorithm is considered invariant to scale, rotation, illumination and viewpoint. For a given image I, the convolution take place at different scales t using the Gaussian kernel given as

G(x, t) = 1 2πt exp -x 2 2t
(2.4) and then, the multi-scale image Gaussian pyramid is obtained. Since LoG is computationally expensive, one simple method to approximate it, is by computing the difference of two close levels at each given scale level of multi-scale pyramid. The proofs are described in [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF].

Generally, the approximation is given as Although, DoG has been introduced for reducing the computational complexity, the practicality of SIFT has always been questioned for real-time tasks such as tracking.

∇ 2 L(x, t) ≈ t δt (L(x, t -δt) -L(x, t)) (2.
Therefore, the SURF detector has been introduced in [START_REF] Bay | SURF: Speeded up robust features[END_REF] to compensate the computational requirement. It uses an integer approximation of the determinant of Hessian which is computed on different layers of a multi-scale representation. Different from SIFT, the SURF detectors rely on bi-directional Gaussian filtering to reduce the time-complexity.

Another category of detector include FAST [START_REF] Rosten | Machine learning for high-speed corner detection[END_REF] and BRISK [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF]. The basic strategy for both methods is inspired from corner like detectors, however, the underneath principle is entirely different. In FAST, a pixel is considered a keypoint if its N (value is 16) contiguous surrounding pixels (on a circle) are either brighter or darker then its intensity value. This detection method is considered to be the first learning based method, even though learning techniques have been used simply to speedup the localization process. BRISK on the other hand is a an extension detection scheme satisfying the covariance to scale and rotation by using a multi-scale image representation.

These methods are computationally fast and are widely used for real time applications such as object localization and tracking.

Descriptor Extraction

The descriptor extraction algorithms have gone hand-in-hand with the keypoint detection literature and have been thoroughly studied (see, e.g., [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF]). The main goal is to represent the visual information present in a local image patch in the form of a unique invariant signature which can help to find its true correspondence in other images. In this thesis, we consider the following four widely accepted feature extraction schemes as namely:

BRISK [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF] and FREAK [START_REF] Ortiz | Freak: Fast retina keypoint[END_REF] (corner based), SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] and SURF [START_REF] Bay | SURF: Speeded up robust features[END_REF] (blob based) to evaluate the performances of our models.

BRISK [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF]

Performance Evaluation Metrics

Keypoint detection and descriptor extraction performance on the LDR images are measured using the standard criteria of Repeatability Rate (RR) and Matching Score (MS) respectively, as detailed in [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF][START_REF] Mikolajczyk | A Comparison of Affine Region Detectors[END_REF]. Whereas for the evaluation of the full image matching, the mean average precision (mAP) scores [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF] are usually computed. In this thesis, we compare our proposed models with the tone-mapping models by computing their performances using these feature extraction metrics. In the following, we detail RR, MS and mAP metric.

• Keypoint accuracy measure: RR is the most common and widely used measure of detector efficiency. In mathematical notations, it is defined as To define a correct match, feature location is taken into account. Two descriptors yield a true positive match if they correspond to two keypoints/regions which are repeated [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF] in the reference and query images. Similarly, a match is labeled as a false positive if the corresponding keypoints are not repeated.

r ref ( ) min(n ref , n test ) , (2.6
In summary, only nearest neighbor (NN) and nearest neighbor distance ratio (NNDR) can reduce the possibility of one-to-many matching scenarios of the descriptors. In most of our evaluation frameworks, we employed the NNDR matching strategy to compare the performance of our TMO with other techniques.

• Feature Matching Efficiency: MS gives only the estimate of correct matches, while in practice, many incorrect matches may occur. Therefore, for completeness, the performance of feature extraction algorithms is computed using mAP score. To this end, first a Precision-Recall (P-R) curve is generated by varying the matching strategy parameter th from 0 to 1. Recall is defined as the fraction of true positives over total correspondences and precision is given as the ratio of true positives to the total number of matches. Once the P-R curves are generated for each scene, we then compute the mAP scores by determining the area under the curves.

Chapter 3

Local Feature extraction in HDR Imagery under Drastic Lighting Variations

Overview

Adverse lightening conditions can significantly deteriorate the performance of keypoint detectors and descriptors in conventional LDR imagery. Several local and global normalization models [START_REF] Gupta | SMD: A Locally Stable Monotonic Change Invariant Feature Descriptor[END_REF][START_REF] Tang | A novel feature descriptor invariant to complex brightness changes[END_REF] have been designed to obtain better luminance invariant features. But these techniques are somewhat inefficient in practice. Poor performance of these algorithms

Local Feature extraction in HDR Imagery under Drastic Lighting Variations

mainly accounts to the loss or change in the spatial configurations of the details present in a scene [START_REF] Verdie | TILDE: A temporally invariant learned detector[END_REF][START_REF] Zhang | Adaptive bilateral filter for sharpness enhancement and noise removal[END_REF].

HDR imagery brings potential to surpass these limitations and consequently enhance the feature extractor's output as accounted in [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF][START_REF] Chermak | Enhanced feature detection and matching under extreme illumination conditions with a hdr imaging sensor[END_REF]; thanks to its wider dynamic range which enables to capture details in both dark and bright regions. However, it is not clear which are the best methods of employing HDR and weather these gains are significant on a real dataset.

This chapter investigates the potential of HDR for feature extraction stages i.e.keypoint detection and description, and in particular, addresses the following research questions:

1. is HDR capable to achieve substantial quantitative gains in terms of feature stability to luminance changes compared to LDR? are these gains consistent? the other illumination settings, in order to estimate the average feature stability. This is accomplished using two popular corner point (Harris [START_REF] Harris | A combined corner and edge detector[END_REF]) and blob detectors (SURF [START_REF] Bay | SURF: Speeded up robust features[END_REF]).

Some previous frameworks for evaluating the detectors and the descriptors have been proposed in the literature as discussed in Section 2. [START_REF] Kontogianni | HDR imaging for feature detection on detailed architectural scenes[END_REF] and [START_REF] Chermak | HDR imaging for feature tracking in challenging visibility scenes[END_REF] framework report an increase in the number of detected feature points using HDR based modalities over LDR.

However, the number of detected feature points is not itself a sufficient indicator of detection performance. Additionally, based on their results, it is difficult to draw precise conclusions on what makes certain HDR modalities perform better than others. Pribyl et al. [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF] presents an evaluation of the repeatability on simple images under different transformations (lighting, viewpoint, distance) for a few LDR/HDR modalities but not the original HDR values. Aforementioned studies, mainly focused on evaluating detector performance only, with one or more different HDR representations [START_REF] Bronislav | Evaluation of feature point detection in high dynamic range imagery[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF]. [START_REF] Chermak | Enhanced feature detection and matching under extreme illumination conditions with a hdr imaging sensor[END_REF] evaluated the full feature extraction pipeline but with only a single HDR representation.

In this chapter, we focus on standard measures of feature stability under illumination changes along with analyzing the performance of many popular tone-mapping approaches which have been evaluated thoroughly from a perceptual point of view, but whose effectiveness in feature extraction has not been investigated so far. Additionally, we explicitly

HDR Luminance Change Dataset

39

compare direct feature extraction stages on HDR images with a tonemap-then-extract approach [START_REF] Kontogianni | HDR imaging for feature detection on detailed architectural scenes[END_REF].

In a nutshell, three major contributions in this chapter are:

HDR Luminance Change Dataset

Accuracy measurement of feature detection and matching is based on RR criterion and mAP, which relies on the precise localization of key-points in both reference and test images, so that correspondences between detected features points can be unequivocally assessed [START_REF] Schmid | Evaluation of interest point detectors[END_REF]. Unfortunately, the great majority of existing HDR image and video datasets are not adapted to this end, as images are not geometrically calibrated. The only such existing HDR dataset adequate for a confined low-level evaluation has been proposed in [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF] (we refer to it as 2D and 3D Lighting Dataset in the rest of this chapter), where a scene with controlled lighting conditions has been captured. However, the number of lighting conditions is quite limited.

In this chapter, we propose two different lighting setups: Project Room and Light Room (Figure 3.1), focusing mainly on lighting changes and variation in dynamic range of the scenes, which are recognized to be some of the most critical points in LDR feature detection, and are those for which HDR technology could bring most benefits.

Project Room (PR). is defined as log 10 (L max /L min ), where L min and L max are the minimum and maximum HDR brightness values, respectively. Both properties give an indication of the variety of illumination conditions contained in the dataset.

Evaluation of Keypoint Detectors in HDR imagery

Keypoint Detectors

Feature extraction has been studied in vast details in computer vision literature where several techniques have been proposed and evaluated as detailed in Section 2.2, taking into account different challenging transformations. In this section, we focus on the two most widely used interest point detection schemes, i.e., corner and blob detectors, which are often used in several real time applications. In spite of several existing schemes for these approaches, we select two common detectors that have been used in similar evaluations for LDR content [START_REF] Bronislav | Evaluation of feature point detection in high dynamic range imagery[END_REF][START_REF] Chermak | HDR imaging for feature tracking in challenging visibility scenes[END_REF].

For corner interest point detector, we employ the popular Harris corner point (Harris) detector [START_REF] Harris | A combined corner and edge detector[END_REF], which is based on the autocorrelation score computed from local intensity change in an image. For blob detection, the experiments are carried out with the highly robust SURF [START_REF] Bay | SURF: Speeded up robust features[END_REF] detector.

Considered LDR/HDR modalities

For each illumination change dataset, we consider the following LDR and HDR modalities:

• LDR best exposed image: we take the subjectively best LDR exposure shot for each illumination setup, i.e., the one that a human surveillance operator would select based on large details with smallest area of over-or under-exposed pixels; • Tone-mapped image: we consider two global (GTM) and seven local (LTM) TMO's (see Table 3.1) to convert HDR pictures to 8-bit LDR, which are representative of the most popular tone mapping techniques for rendering HDR on LDR displays proposed in the literature [START_REF] Cadik | Evaluation of HDR tone mapping methods using essential perceptual attributes[END_REF];

• HDR linear values (HDR-Lin), i.e., photometric luminance values stored in the HDR file;

• HDR perceptually encoded values: we consider a simple logarithmic (HDR-Log)

encoding, according to Weber-Fechner's law; or the perceptually uniform encoding (HDR-PU) proposed in [5], which accounts for the drop of sensitivity at lower

Experimental Results and Discussion

Experiments are carried out on proposed datasets (LR,PR) and lighting dataset (2D,3D)

of [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF]. The only measure of accuracy considered in this chapter is the repeatability rate (R-score) as discussed in Section 2.2. For this evaluation, we used = 35px, which is less than 1% the image size, similar to [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF]. Also the evaluation scheme is confined to the strongest 200 key-points in marked RoI's (Region of Interests). This not only limits the feature point detection in pertinent areas, but also helps to ensure a fair comparison of the blob or corner key-point detection on diverse datasets, as different detectors result in varying number of keypoints which can bias the R-score.

The experimental study is conducted in two phases. In the first phase, for each dataset, one scene is selected as a reference image, and the repeatability is computed with the other scenes (test images). Relative gains are recorded for all best HDR based modalities (GTM,

Local Feature extraction in HDR Imagery under Drastic Lighting Variations

Light Room HDR versus LDR. In all conditions and for both key-point detectors, average values show significant gains of HDR or tone-mapped images over single LDR exposure. This is consistent with what has been found in [START_REF] Chermak | Enhanced feature detection and matching under extreme illumination conditions with a hdr imaging sensor[END_REF]. However, based on the results from scatter plots in Figure 3.4, we observe that there are some scenarios where best LDR records higher performance than the rest of the HDR based formats. We believe that this is mainly due to significant illumination differences in pertinent regions of image pairs. HDR encodings. The best average repeatability scores are in general obtained with PU-HDR encoded values. This is not surprising, as HDR formats store most of the pertinent information in the scene, and it is therefore promising to research towards application of feature extraction on these modalities. From results, it is also clear that these encodings give significantly better results than photometric HDR-Lin. This is a non-obvious conclusion of this work, i.e., that HDR-Lin is not appropriate to be used for feature extraction

Evaluation of Keypoint Detectors in HDR imagery

45

Light Room algorithms, especially by observing huge variation in its behavior. This seems to suggest that feature extraction algorithms designed and optimized for LDR content somehow require the optimally scaled pixel values. Also for such algorithms, even the parameter tuning is not an option as the range of photometric luminance value can largely vary with the content.

0 0.2 0.4 0.6 0.8 1 M a n t i u k A s h i k h m i n W a r d S c h l i c k R e i n h a r d D r a g o P a t t n a i k C h i u L i n L o g F a t t a l P u Repeatability 0 0.2 0.4 0.6 0.8 1 S c h l i c k P a t t n a i k L i n R e i n h a r d A s h i k h m i n M a n t i u k W a r d C h i u F a t t a l L o g D r a g o P u Repeatability Project Room 0 0.2 0.4 0.6 0.8 1 W a r d A s h i k h m i n M a n t i u k S c h l i c k P a t t n a i k L i n F a t t a l R e i n h a r d D r a g o L o g C h i u P u Repeatability 0 0.2 0.4 0.6 0.8 1 P a t t n a i k M a n t i u k A s h i k h m i n F a t t a l W a r d S c h l i c k R e i n h a r d L i n L o g D r a g o C h i u P u Repeatability
HDR versus TMO's. Average R-scores gains over LDR by tone mapping techniques are either comparable or lower than those of HDR encoding formats. However better performances recorded by some TMO's draws significant attention, such as Drago [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF],

Chiu [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF] and Fattal [START_REF] Fattal | Gradient domain high dynamic range compression[END_REF]. In addition to the performance evaluation for LTMs', it is interesting that the gradient-based local techniques, i.e., Fattal and Chiu TMO's, have shown comparable gains than other LTM techniques, in specific scenarios. This is inverse to observations in perceptual applications [START_REF] Cadik | Evaluation of HDR tone mapping methods using essential perceptual attributes[END_REF], where these two LTMs' are deemed as worst performers. This further establishes that there is less congruency between visually pleasing tone mappings and vision-task-based optimal mapping technique. Another important point to note here is that these tone mappings perform better with blob detectors than corner point, and it is consistent with the observations in the literature comparing detectors [START_REF] Schmid | Evaluation of interest point detectors[END_REF].

In addition to all the observations, it is also worth mentioning that there is no unanimous winner amongst these tone mapping techniques using either detection criterion. 3.1. The choice has been entirely based on trade-off between their performance in keypoint detection task in Chapter 3 and perception based tasks in [START_REF] Cadik | Evaluation of HDR tone mapping methods using essential perceptual attributes[END_REF]. Additionally, note that we do not include PU encoded HDR. This is mainly due to the absence of luminance-based calibrated scenes from 2D and 3D lighting dataset.

Feature extraction

In this section, following 4 popular feature extraction schemes are assessed. Both gradientbased histograms and computationally fast binary descriptors are employed for the evaluation.

• SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]. This classic scheme is constituted of a blob keypoint detector (based on difference of Gaussians) and a gradient-based descriptor. The SIFT descriptor is a 128-dimensional histogram formed by concatenation of the image gradients computed on 4x4 grid spatial neighborhood around the detected keypoint.

• SURF [START_REF] Bay | SURF: Speeded up robust features[END_REF]. SURF scheme is composed of a computationally efficient blob type detector mainly based on the Hessian matrix approximation along with a descriptor computed as the sum of the Haar wavelet response around the point of interest.

• BRISK [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF]. With major focus on computational efficiency, the BRISK feature extraction is made up of a fast multi-scale detector and a binary descriptor. The detection module is an extension of corner-based detectors like AGAST and FAST.

The descriptor is a binary string computed by brightness comparisons on circular sampling patterns around the detected regions.

Evaluation of full Features Extraction Pipelines in HDR imagery 47

Project Room Light Room 2D Lighting 3D Lighting • FREAK [START_REF] Ortiz | Freak: Fast retina keypoint[END_REF]. Similar to BRISK scheme, it has the same BRISK detector along with a binary descriptor called FREAK. Similar to BRISK descriptor, FREAK also uses a concentric rings arrangement, but the sampling grid is non uniform as inner circular rings have exponentially more points.

Experimental Results and Discussion

Our test setup comprises a total of 29 images (8 Project Room + 7 Light Room + 7 2D-Lighting + 7 3D-Lighting) for each image representation. In the first part of experimental validation, we look at the overall feature extraction performance, by computing the mAP over all datasets. To this end, we evaluate matching using a test bench of 182 image pairs (56 Project Room + 42 Light Room + 42 2D-Lighting + 42 3D-Lighting). Following the detection protocol from [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF], we first select 400 keypoints with the strongest detector response and measure the keypoint stability using the RR. Then, for the descriptor part, we compute standard precision-recall (P-R) curves [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF] for measuring the accuracy of matching.

For each image pair, we compute the PR curve by varying th from 0.0 to 1.0 and record the average precision value. After this, for each format and either feature extraction scheme, a mAP score is obtained by averaging the average precision calculated on such 182 image pairs (see Table 3.3). Both metrics RR and mAP are defined in Section 2.2.3.

Furthermore, to understand how detector and descriptor contribute to the overall performance, we expand our analysis to individual datasets and compute mAP and RR.

In Figure 3.8, we report side-by-side the mAP and RR for each extraction scheme for all datasets, respectively. It is evident that in most of the cases higher RR entails higher mAP scores, i.e., having more stable keypoints strongly influences the overall matching performance. Nevertheless, there are few exceptions, e.g. RN and FA in 3D-Lighting dataset, discussed later in this Section. In the following, we examine in detail the main conclusions obtained from our results.

HDRLin versus all. The results in Table 3.3 show that HDRLin representation is consistently the worst performing using all extraction schemes. This is coherent with the

L D R R N G D R R N M A F A C H D U H D R L og H D R L N mAP mRR Project Room(FREAK) 0 0.2 0.4 0.6 0.8 1 Percentage L D R R N G D R R N M A F A C H D U H D R L og H D R L N mAP mRR Light Room (FREAK) 0 0.2 0.4 0.6 0.8 1 Percentage L D R R N G D R R N M A F A C H D U H D R L og H D R L N mAP mRR 2D Lighting (FREAK) 0 0.2 0.4 0.6 0.8 1 Percentage L D R R N G D R R N M A F A C H D U H D R L og H D R L N mAP mRR
3D Lighting (FREAK)

Local Feature extraction in HDR Imagery under Drastic Lighting Variations

previous findings in [START_REF] Bronislav | Evaluation of feature point detection in high dynamic range imagery[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF], and is mainly due to the low keypoint repeatability, which increases the probability of false positives. This leads to the first conclusion of this work, i.e., HDRLin is not appropriate to be used for feature extraction algorithms, for both detector and descriptor.

HDRLog/TMO versus LDR. On average, all HDR formats show significant gains of (at least) 8% mAP over single LDR exposure (see Avg/Formats in Table 3.3). This partially accounts to having more false matches in LDR due to loss in local textural information in lighting transformations. Another reason which is evident from Figure 3.8, is the low repeatability rate which reduces the number of true positives.

HDRLog versus TMO's. mAP scores obtained from HDRLog and different TMOs are relatively comparable. This implies that there are not significant advantages in using a floating-point HDR representation over 8-bit TMs. Alternative HDR encodings could improve further mAP scores, such as the PU encoding [5], as reported for keypoint repeatability in [START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF]. However, those representations require photometrically calibrated HDR pictures, which might not be available in practice.

Comparison with previous studies. Previous studies [START_REF] Bronislav | Evaluation of feature point detection in high dynamic range imagery[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF] have reported that local TMO approaches such as Fattal or Chiu consistently provide more stable keypoints (in terms of repeatability) under illumination changes, compared to TMOs which are generally considered good from a perceptual perspective, such as Reinhard. The results of this work show that those trends are less evident when the overall feature extraction pipeline is considered. For instance, from Figure 3.8 we observe that some TMOs achieve better repeatability rates but lower overall mAP scores compared to others formats, e.g., this is the case for RN and FA tone mappings in Project Room and Light Room dataset using BRISK and FREAK, or for RN and FA in 3D Lighting dataset using SIFT. We deduce that in those cases, although the fraction of repeated keypoints is lower, the corresponding descriptors are more discriminative, i.e., they yield a lower rate of false positives, or equivalently, a higher portion of matches are true. Another important point to note is that these tone mappings perform well with all feature extraction scheme for different lighting transformations, with marginal gains for SIFT. In addition to all the observations, it is also worth mentioning that there is no unanimous winner amongst these tone mapping techniques for all extraction criterion. This study further strengthens our argument that there might be quite a large room for improvement in feature extraction performance at detection and description stages by designing optimal tone mapping schemes for HDR, which can ensure high average precision as well as repeatability rates, and that can be easily fused with current recognition algorithms.

Consequently, in the following chapter, we will primarily investigate the key criteria for designing the optimal TMO, starting with the keypoint detection. Then, we will further approach to designing a corresponding optimal TMO which aims at enhancing the efficiency in keypoint detection.

The work presented in this chapter has resulted in the following publications: Chapter 4

Tone Mapping Operator for Efficient Keypoint Detection

Overview

TMOs have traditionally been designed to display HDR pictures in a perceptually favorable way and mainly preserve the human-vision attributes such as image aesthetics and perceptual contrast. However, when such tone-mapped images are to be used for computer vision tasks such as keypoint detection, these design approaches are suboptimal [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF][START_REF] Rana | An evaluation of HDR image matching under extreme illumination changes[END_REF] and needs to be re-calibrated. No related work exists in the literature, which aims at designing a detection-optimized tone mapping technique or comprehending the related criteria involved.

In this chapter, we address the problem of optimal TMO design for keypoint detection task. Specifically, we investigate the following questions a) what are the factors to be considered in the TMO design when targeting keypoint detection tasks ?, and b) how can we optimize a TMO for such tasks under drastic illumination variations.

To answer the aforementioned questions, this chapter initially discusses the suboptimality of existing TMOs and derives guidelines to design a keypoint-optimized TMO.

To that end, a comparison is drawn between the optimization of existing TMO parameters with respect to: a) Repeatability Rate RR and b) correlation coefficient CC between tone-mapped images of the same scene with lighting variations. CC measures the statistical similarity between a pair of tone-mapped images. The goal here is to find whether optimizing a TMO with respect to RR leads to higher keypoint stability over the per-pixel similarity (using CC) between the tone-mapped images.

Building upon the observations from optimality study, in this chapter, we introduce a novel learning based adaptive TMO for robust keypoint detection. Our proposed framework aims at enhancing the repeated detection of sparse keypoint locations (e.g. corners) in high-contrast areas of scenes undergoing complex real-world illumination transitions such as day/night change. To this end, we initially introduce an adaptive TMO which can be

Optimizing a TMO for Keypoint detection

In this section, we present a study on the parametric optimization of TMOs using two factors: CC and RR, aiming at enhancing the keypoint detection performance under drastic lighting change scenarios. Our main aim is to find out what factor could be the most interesting to define an optimized TMO.

RR is a conventional performance measure of keypoint detection algorithms and is computed on repeated occurrences of detected keypoints in test and reference images.

Hence, optimization of TMOs with respect to RR could estimate the optimal detection performance gains.

On the other hand, the CC computes the statistical similarity between distribution of images. Theoretically, a CC-optimized TMO should improve keypoint detection performance. This is mainly because as a high statistical similarity of tone mapped images should increase the probability of detection of similar keypoints. Such scenarios in return could be highly interesting for optimizing a potential class of TMOs which are based on illumination normalization such as [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF]. In simple words, what if a substantial improvement in keypoint detection performance can be achieved just by estimating the ideal reflectance maps from the HDR images using TMOs such as [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF].

In the following, we first detail the considered TMO. Next, we briefly discuss the feature detection methods, followed by metrics and dataset selection. Finally, we describe the optimization strategies of considered models.

Considered TMO

In this study, we employ two well-known Retinex inspired approaches using: a) Gaussian model [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF] and b) Bilateral model [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF] for tone mapping. We choose these TMOs as they are promising to bring illumination invariant (as much as possible) reflectance maps; an ideal case for efficient keypoint detection.

According to Retinex theory of physical image modeling, we assume that I, the HDR image to be tone mapped, is the product of the luminance L of the scene (which varies with different illumination conditions) and of the reflectance R characterizing objects of the scene, i.e., I = R • L. The luminance L is generally assumed to be spatially smooth [START_REF] Gonzalez | Digital Image Processing[END_REF],

while reflectance contains fine-grained details, texture and edges which are relevant for detection [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF][START_REF] Schmid | Evaluation of interest point detectors[END_REF]. Once L is estimated, the final reflectance image is given by R = I/L.

In the following, we briefly describe the aforementioned luminance estimation TMO: a). Gaussian tone mapping (GTM) model gives the reflectance image R g as R g = I/L g where

L g = I * G σ , (4.1)
where G σ is a Gaussian filter where the parameter σ depends on image size [m × n], i.e.

L b (x) = 1 N y∈S G σs ( x -y ) • G σr ( I x -I y )I y , (4.2)
where x and y are pixel locations, S is the set of neighborhood locations, G σr and G σs

Tone Mapping Operator for Efficient Keypoint Detection

are Gaussian filters with variances σ r and σ s referred to as range and spatial parameter respectively. N is a normalization factor term :

N = y∈S G σs ( x -y )G σr ( I x -I y ). (4.3)
It is important to note that when the range parameter increases, the model gradually approaches Gaussian convolution. This is mainly because the Gaussian G σr widens and flattens, and essentially, it becomes nearly constant over the intensity interval of the image.

Conversely, when the spatial parameter increases, larger details like edges get smoothened in the image.

An example of luminance estimation with their corresponding reflectance image is shown in Figure 4.1.

Keypoint point detection

Keypoint detection has been widely studied in computer vision literature where several techniques have been proposed and evaluated [START_REF] Schmid | Evaluation of interest point detectors[END_REF] taking into account different challenging transformations. In this study, we focus on the two most widely used keypoint detection schemes, i.e., corner and blob detectors. We select two common detectors that have been used in previous HDR imagery based evaluations [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF] and are often used in several real-time applications. For corner interest point detector, we employ the popular Harris corner point detector [START_REF] Harris | A combined corner and edge detector[END_REF], which is based on the autocorrelation score computed from local intensity change in an image. For blob detection, we carried out experiments with the highly robust SURF [START_REF] Bay | SURF: Speeded up robust features[END_REF] detector.

Metrics

In this study, we build our framework using following metrics.

• RR (see Section 2.2) is a standardized method detailed in [START_REF] Schmid | Evaluation of interest point detectors[END_REF] to measure the detector accuracy. It is given as the fraction of keypoints detected in the reference image which are repeated in the test image to the minimum of a total number of detected points in test or reference image. A keypoint is considered to be repeated in the test image if: a) it is detected as a keypoint in the test image, and b) it lies in a circle of radius centered on the projection of the reference keypoint onto the test image. determines the keypoint detection error rate. RR is given as R i ( ) min(nr,n i ) , where R i ( ) is the number of keypoints detected in the reference image which are repeated in the test image, n r , n i is the number of detected keypoints in reference and test image respectively.

• Correlation Coefficient (CC) is well-known to quantify the strength of a linear relationship between two variables. In this study, we have used this metric to measure the correlation between two image maps. Values close to 1 indicate that there is a positive linear relationship whereas 0 points no linear relationship between image maps.

Datasets

We considered the HDR dataset with substantial illumination changes proposed in the previous chapter [START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF]. It is composed of 2 parts: Project-Room with 8 lighting conditions and Light-Room with 7 lighting conditions.

Optimization of TMOs

Let I 1 (x) and I 2 (x) be two images of the same scene, illuminated by two different illumination maps L 1 (x) and L 2 (x). According to Retinex theory

I 1 (x) = L 1 (x) • R 1 (x)
and

I 2 (x) = L 2 (x) • R 2 (x
). An ideal Retinex algorithm estimates L 1 (x) and L 2 (x) such that two ideal reflectance maps are equal R 1 (x) = R 2 (x). In such an ideal scenario, keypoint detection performance should be enhanced considerably as the keypoints in reflectance maps would be identical.

Tone Mapping Operator for Efficient Keypoint Detection

However, Retinex is a mathematically ill-posed problem [START_REF] Jobson | Properties and performance of a center/surround retinex[END_REF]. In practice, it often implies that R 1 = R 2 . Besides this, all perceptually optimized Retinex based models aim at finding the best luminance estimation for a given scene such as I 1 = L 1 • R 1 , rather than optimizing on aforementioned image pairs like (R 1 , R 2 ) which is the classical way of measuring keypoint detection performance.

Therefore, we firstly investigate the maximization of the correlation between reflectance images pair. The main motivation is that highly correlated reflectance maps should result in detected keypoints that are alike, thereby, enhancing the keypoint repeatability.

Alternatively, we also investigate the optimization of considered models with respect to repeatability rate which will help to analyze the maximum gains achievable with conventional TMO.

In summary, we optimize the considered Retinex based GTM and BTM models in two ways: 1) with respect to correlation of reflectance maps of image pairs and, 2) with respect to detector repeatability. For the first method, we iteratively optimize the GTM and BTM models parameters with respect to CC on both datasets using each detector.

More specifically, we iteratively tune the parameter σ for GTM and σ s , σ r for BTM with the aim of maximizing the overall CC using each detector.

Correlation based parameter tuning for Project Room dataset in illustrated in Figure 4.2 (a). It depicts that for GTM, a higher σ, i.e. high variance Gaussian blur, minimizes the absolute differences between the reflectance image pairs. The same observation also holds for range and spatial parameters of BTM model as shown in Figure 4.2 (b). Thereafter, using these correlation based optimized models, we generate the tone mapped images as correlation-coefficient-Gaussian-tone-mapping (CCGTM) and correlation-coefficient-Bilateral-tone-mapping (CCBTM) for both datasets.

For the second method of optimization with respect to detector repeatability, we tune the GTM and BTM parameters with the aim of maximizing the overall repeatability rate (RR) of keypoints using both Harris and SURF detector. Corresponding results are shown in Similarly, we generate the tone mapped images, repeatability-rate-Gaussian-tone-mapping (RRGTM) and repeatability-rate-Bilateral-tone-mapping (RRBTM), for both datasets and using each detector.

Experimental Results and Discussion

We evaluate the CCGTM, CCBTM, RRGTM, RRBTM optimized tone mappings models for keypoint detection using Harris and SURF detectors in For each TMO, we measure the overall keypoint detection accuracy using the RR performance metrics. Initially, we compute the individual RR using a particular detector to [START_REF] Bronislav | Evaluation of feature point detection in high dynamic range imagery[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF], our keypoint detection scheme is confined to the strongest 1000 keypoints. This is mainly to ensure a fair comparison of the blob or corner keypoint detection, as different detectors result in a highly different number of keypoints. We use a fixed detection error rate, i.e. = 3px, which is .03% of image size (see section 5.4.5).

CC vs RR based optimization. We observe from Figure 4.4(a),(b) that CCGTM and CCBTM records substantial gap in performance with respect to RRGTM and RRBTM using both Harris and SURF detectors. We can conclude that high correlation between reflectance image pairs does not directly guarantee the stability of keypoint detection.

However, this can be explained as follows. Keypoints are localized using non-max suppression technique as explained in [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]. The magnitude per pixel depicts the probability of that pixel to be detected as a keypoint which generally rely on second order derivatives. An example of response map using Harris detector [START_REF] Harris | A combined corner and edge detector[END_REF] is shown in Figure 4.6, where high pixel values indicate higher likelihood to be considered as keypoint. As a consequence, local keypoints are in general sparingly distributed in detailed areas of an image. This process is much more complex than a simple statistical correlation computed at the pixel level. This is further illustrated in Figure 4.5. It can be observed that correlation of response maps are linearly proportional to RR whereas in contrast the correlation of the reflectance image maps shows scattered behavior. Therefore, this leads to the major conclusion of this study.

In order to design a keypoint-detection-optimal tone mapping, the traditional Retinex based approaches need to take into account the detector response maps while estimating the reflectance images for a given scene.

Comparison with Traditional TMOs. From Finally, we also observe that the performance gains are significantly larger for Project Room than Light Room. This is mainly due to the fact that lighting transformations are much tamer for Light Room dataset, which entails smaller performance variations in RR when comparing different TMOs.

Learning a TMO for Efficient Keypoint Detection

The experimental study in the previous section concludes that optimizing TMO parameters with respect to RR leads to higher keypoint stability over the per-pixel similarity between the tone-mapped images. Though this study points to the parametric sensitivity in TMOs, In this section, we therefore develop a novel learning-based adaptive tone mapping operator, referred as DetTMO, aims at enhancing keypoint stability under drastic illumination variations. To this end, we design a pixel-wise adaptive TMO which is modulated based on a model derived by Support Vector Regression (SVR) using local higher order characteristics.

Our idea is mainly motivated by the conclusions of our previous study in Section 4.2 where optimizing tone mapping parameters for keypoint detection is shown to yield significant gains in RR. However, in that study optimal TMO parameters are computed globally on the whole image using grid search and, more importantly, for a given scene. Here, instead, we propose to learn TMO parameters based on the local features of the scene. Specifically, since keypoints are sparsely detected and depend on their neighborhood properties, we argue that local parametric modulations in TMOs can enhance the keypoint detection probability by adaptively mapping pixels based on their local higher-order characteristics.

To predict such optimal modulations in this context, we are inspired by the success of regression-based "task-optimization" models. In the literature, regression-based models have been explored for several image processing problems [START_REF] Kim | Single-image super-resolution using sparse regression and natural image prior[END_REF][START_REF] Ni | Image superresolution using support vector regression[END_REF][START_REF] Takeda | Kernel regression for image processing and reconstruction[END_REF]. Here, we employ SVR, which has been successfully used, e.g. in image super-resolution [START_REF] Ni | Image superresolution using support vector regression[END_REF], and which enables to cope with large variability in the input training samples compared with low-dimensionality approaches using explicit functions such as polynomial regression.

Additionally, we propose a simple detection-similarity-maximization model to generate appropriate training samples for the SVR. We initially consider several HDR image pairs which are taken with the same viewpoint with different lighting variations. Then, we define our objective function to find optimal modulation maps for such scenes so that the similarity of the detector response maps is maximized. For the defined objectives, the optimization is carried out using stochastic gradient descent (SGD) [START_REF] Bottou | Stochastic Gradient Descent Tricks[END_REF] by deriving the required partial derivative architecture.

In the following, we initially provide the details of our learning-based adaptive TMO approach, the similarity maximization model for generation of training set, the SVR training and proposed dataset. Later, we present the experimental results and analysis.

General Framework

Let ϕ be a tone mapping function which maps the linear-valued HDR content of an image I to an output LDR I . In general, for a pixel x, TMO operates as:

I (x) = ϕ(I(x), θ),
where I(x) ∈ , I (x) ∈ [0, 255] and θ represents a vector of parameters.

For several existing TMOs [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF][START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF][START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF], parameter θ serves diverse objective such as filter shape and size, brightness control, but all motivated for visual perception. Such parameter is often set as globally for an image and further chosen by trial and test procedures. For example, θ serves as variance in ChiuTMO [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF], sharpening constant in ReinhardTMO [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] and range and spatial variance in bilateral filtering based TMO [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF].

Based on these observations, we assume function ϕ as an extension of existing tone mapping functions which can be modulated spatially by adapting their vector of parameters.

The idea here is to facilitate the local adaption of function ϕ at sparse keypoint locations to further ease their identification and detection. In this chapter, we call the corresponding vector of parameters as modulation maps so as to distinguish their purpose of modulating the TMO locally from global parametric tuning. The modulation maps are given as θ(x) = {θ 1 (x), θ 2 (x).., } and our proposed TMO operates as: I (x) = ϕ(I(x), θ(x)).

To predict the modulations maps, we propose to learn a model by employing SVR [START_REF] Smola | A tutorial on support vector regression[END_REF] while complying with the following two constraints: (a) To distinguish the keypoint and its neighborhood locations, (b) To bring invariance (as much as possible) to the non-affine lighting variations in the physical world scenes.

By using the radial basis kernel mapping, our SVR minimizes the non-linear problem of predicting modulation maps θ by linearly separating the input samples in high-dimensional space. We refer the reader to [START_REF] Smola | A tutorial on support vector regression[END_REF] for more details about kernel-based SVR optimization model. Fig. 4.7 outlines the general framework of our proposed keypoint optimal TMO.

Adaptive Tone Mapping Operator

Many tone mapping approaches aim at separating scene illumination, which can display large dynamic range variations, from the reflectance of objects, which instead has lower dynamic range characteristics [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF]. Following this idea, our tone mapping function ϕ is expressed as: ϕ = I • L -1 , where the illumination component L is estimated by an adaptive version of bilateral filtering [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF] and is given as:

L(x, θ) = 1 W • y∈Ω G θ 1 (x) ( x -y ) • G θ 2 (x) ( I(x) -I(y) )I(y), (4.4)
where G is a Gaussian kernel. Here, modulation vector θ has two components: θ 1 and θ 2 , also known as spatial and range variance. For each pixel location x, y is a pixel in neighborhood set Ω and the normalization factor

W = y∈Ω G θ 1 (x) ( x -y ) • G θ 2 (x) ( I(x) -I(y) ).

Learning a TMO for Efficient Keypoint Detection

63

It is important to note that we have built our model using the bilateral filtering, mainly because its proposed adaptive formulation facilitates the integration of local modulation in the proposed TMO. Moreover, it has been previously studied in the context of keypoint detection in HDR imaging in varying lighting conditions as discussed in Section 4.2. However, any other tone mapping techniques with parametric formulations such as [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] could be used as well with our proposed framework.

Generation of Training Set: Detection Similarity Maximization Model

Suppose we are given a set of HDR scenes where each scene has images captured from the same viewpoint but with different lighting conditions. To train the SVR for our proposed model, we need to compute the "ideal" modulation maps (θ 1 , θ 2 in our case) for a scene which ensures high keypoint stability. In other words, for a scene undergoing lighting variations, we need to estimate the modulations ensuring maximum keypoint repeatability.

To this end, one solution is to design an optimization model which maximizes the RR of multiple images of a given sequence.

RR is a measure of detector efficiency, as defined in the previous section. Since RR is a non-smooth and non-differentiable function, it cannot be directly used to define the similarity objective of our optimization model. Therefore, we instead propose an alternative solution to use differentiable detector response maps R and design a model that maximizes the similarity between these response maps of image pairs drawn from a given sequence. R is a score map which determines a pixel's strength to be a keypoint and it mainly depends on the choice of keypoint detection algorithm.

Our response map R is generated by a Harris corner detector [START_REF] Harris | A combined corner and edge detector[END_REF]. It is based on the autocorrelation scores computed per pixel using the second-order moment matrix, and is given as:

R ϕ(x, θ) = det{M ϕ(x, θ) } -k • tr{M ϕ(x, θ) } 2 (4.5)
where M is the second order moment matrix as detailed in [START_REF] Harris | A combined corner and edge detector[END_REF]. k is the sensitivity factor (k = 0.04). Here, we have focused on the corner-based detectors as they are computationally inexpensive and highly used for real time applications, e.g. tracking, wide-view panorama creations, etc. However, the model could be extended to region or blob-based detectors as well.

Objective: Let S be a scene consisting of N HDR images with lighting variations as shown in Fig. 4.8 (a). Let P = (1, 2), (2, 3).., be the set of K = N 2 pair combinations of N images. Our aim is to maximize the response similarity by minimizing the following objective function:

F(θ) = 1 K {i,j}∈P Φ(R i (θ), R j (θ)), (4.6)
and obtain the resulting modulation maps θ = {θ 1 , θ 2 } as shown in Fig. 4.8 (b) and (c). Inspired by max-margin formulations for image retrieval tasks [START_REF] Rana | Feature learning for the image retrieval task[END_REF], we define function Φ using the logistic penalty

Tone Mapping Operator for Efficient Keypoint Detection

Φ(R i , R j ) = log(1 + exp( -R i • R j ). (4.7)
where is a penalty control factor, R i and R j are the response maps corresponding to the images i, j ∈ S, and • denotes the scalar product.

Optimization using SGD. We optimize the objective function in Eq. (4.6) using Stochastic Gradient Descent (SGD) [START_REF] Bottou | Stochastic Gradient Descent Tricks[END_REF]. To do so, we build the partial derivative architecture required for the SGD implementation as follows.

To estimate θ maps at each iteration t, SGD update rule is given as:

θ t+1 = θ t -γ t • ∇Φ {i,j}t (θ t ), (4.8)
where γ t is a learning rate that can be made to decay with t as γ t = γ 0 /(t + 1) and the gradient for the objective function in Eq. (4.6) is replaced (as detailed in [START_REF] Bottou | Stochastic Gradient Descent Tricks[END_REF]) with the gradient of a randomly chosen sample pair {i, j} at time t, i.e.

∇Φ {i,j} (θ t ) ∂Φ(R i , R j ) ∂θ θt . (4.9)
We computed the gradient required in Eq. (4.8) using the chain rule as follows, To train an SVR model invariant to illumination variations, we first select random samples from keypoint and neighborhood locations across all the given images with varying lighting conditions. For each sample, we then consider a local patch of size s × s centered at that pixel. Next, we compute a feature vector which includes: a) the second-order detector response scores F 1 , b) the gradient magnitudes F 2 and c) the local intensity patterns F 3 .

∇Φ {i,j} (θ) = ∂Φ ∂R i • ∂R ∂ϕ i • ∂ϕ i ∂θ , ∂Φ ∂R j • ∂R ∂ϕ j • ∂ϕ j ∂θ ( 4
The second-order response scores are based on the choice of the keypoint detector.

Therefore, our response score feature for each pixel x in patch s × s is given as:

F 1 (x) = det{M (x)} -k • tr{M (x)} 2 .
The gradient magnitudes for each pixel in the local patch is computed as:

F 2 (x) = G 2 x (x) + G 2 y (x)
, where G x and G y are the gradients in horizontal and vertical directions. The local intensity patterns for each patch is recorded by subtracting the value of centered pixel from other pixels and given as:

Luminance change HDR dataset

We propose an HDR dataset with 8 different HDR scenes as shown in Fig. 4.10. The Light Room, Project Room and Poster are the publicly available datasets and have been used for evaluating HDR for keypoint detection problems [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF]. However, these 3 scenes have been captured in indoor locations and hence, they are less challenging in terms of physical-world illumination transformations such as day/light change. Therefore, we captured 5 additional scenes including 1 indoor Camroom and 4 famous outdoor locations in Paris: Notre-Dame, Louvre, Invalides and Grande Arche. The Camroom scene is shot with a Canon Mark III camera in the presence of powerful 2K Watt reflectors. All the other outdoor HDR scenes are captured with Canon 700D camera at different times of the day. To create the HDR images, LDR images have been fused using the algorithm in [START_REF] Debevec | Recovering high dynamic range radiance maps from photographs[END_REF]. Note that all scenes are geometrically calibrated.

Experimental Setup

We test our proposed model for keypoint detection task on 8 HDR scenes. We initially compare our DetTMO with the non-adaptive bilateral filtering based tone mappings BTMO and its globally optimized version BTMO(opt) as discussed in Section 4.2. Similar to our tone mapping function, both these TMOs are based on illumination normalization where the luminance L is estimated using bilateral filter. However, both these TMOs use global range and spatial variances. Moreover, BTMO(opt) is a variant of BTMO with an additional step of global parameter optimization, and approximates the maximum possible RR that can be achieved with BTMO model.

Learning a TMO for Efficient Keypoint Detection

67

Then, we compare our model with state-of-the-art perception based TMOs: Chi-uTMO [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF], DragoTMO [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF], ReinhardTMO [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] and MantiukTMO [START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF]. We considered these TMOs as they have been previously applied for HDR evaluation studies [START_REF] Suma | Evaluation of the effectiveness of HDR tone-mapping operators for photogrammetric applications[END_REF] for similar keypoint detection task.

We evaluate all these TMOs using popular and widely used corner detection schemes:

Harris [START_REF] Harris | A combined corner and edge detector[END_REF], Shi-Tom [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF], FAST [START_REF] Rosten | Machine learning for high-speed corner detection[END_REF] BRISK [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF]. In addition, even if our formulation is optimized for corner detection, we also test our TMO with respect to blob detectors such as SURF [START_REF] Bay | SURF: Speeded up robust features[END_REF] and SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]. Since our model is designed for one image scale, we employed single-scale implementation for all keypoint detection schemes to ensure a fair comparison.

The detection performance is measured in terms of RR (as discussed in Section 4. 3.3) with an error rate of 5 pixels. Namely, a keypoint is considered to be repeated in the test image if it lies in a circle of radius 5 centered on the projection of the reference keypoint onto the test image. Implementation. We use the SVR implementation of LibSVM [START_REF] Chang | Libsvm: A library for support vector machines[END_REF] using the Radial Basis Function (RBF) kernel. To obtain the optimal values of SVR parameters, the regularization cost and epsilon-SVR are tuned by 5-fold cross validation from the range of [2 -5 , 2 15 ] and = [2 -10 , 2 5 ], respectively.

Training and Implementation details

We use the HDR Toolbox [START_REF] Banterle | Advanced High Dynamic Range Imaging: Theory and Practice[END_REF] for the implementation of the considered TMOs. Moreover, we use the Matlab's Computer Vision toolbox for Harris, Shi-Tom, FAST, BRISK and SURF, and Vlfeat for SIFT. Similar to previous keypoint evaluation studies [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF], we selected the strongest 500 keypoints from each test image.

Evaluation Results

Quantitative Results: We perform a thorough evaluation of our proposed DetTMO in quantitative terms as shown in Fig. 4.11 and Fig. 4.12. We basically evaluate the performance of our method over all test scenes using the Harris corner detector. In tone mapping function using the proposed learned model significantly improves the keypoint stability across both the indoor and outdoor scenes.

Comparison with popular TMOs. We evaluate the performance of our method across different keypoint detection schemes including both corner and blobs. In Fig. 4.12, we initially compute the RR for all scenes for each considered TMO and then average them to compute the Average Repeatability Rate (AvgRR). We observe that for either detector (corner or blob) our proposed model outperforms all the other TMOs (perception based or keypoint-based). Further, the lower standard deviations observed with our proposed TMO shows higher stability of keypoints than other perception-based TMOs. Although our algorithm has been optimized for corners, it gives comparable or better performance with respect to other methods on blob detectors. This is partially due to the single scale implementation of the blob detectors used in this evaluation. However, the performance may differ when the multi-scale blob detection is taken into account.

We compare our DetTMO with popular and visually pleasing Reinhard TMO [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] and MantiukTMO [START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF]. In Fig. 4.13, we show that our method produces the highest number of repeated keypoints, even though both Reinhard TMO [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] and MantiukTMO [START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF] produce more visually appealing images. Chapter 5

Learning a Tone Mapping Operator for Efficient Image Matching

Overview

Conventional TMOs have found to be sub-optimal for the feature extraction task, which includes a detection and a description stage. So far, we leveraged on learning the keypoint characteristics to design an optimal TMO for a stable detection only. In this chapter, we address the full feature extraction pipeline, including the description stage, to design an optimal TMO for efficient image matching.

More specifically, the goal of this chapter to find an optimal TMO which can enhance the extraction of stable features for scenes under complex real-world illumination transitions, such as day/night change. To this end, the chapter first proposes a descriptor-optimal TMO design, referred to as DesTMO, which solely aims at the extraction of invariant (as much as possible) descriptors from high-contrast areas of the scenes. Later, we introduce an optimal TMO, OpTMO, for full feature extraction chain (including both detectors and descriptors) which simultaneously enhances the detection rates and matching of features extracted from HDR scenes. Both the proposed task optimal TMOs namely, DesTMO and OpTMO, follow a learning based paradigm similar to the DetTMO in Chapter 4, but with entirely different design objectives.

Altogether, this chapter proposes

• a descriptor-optimal DesTMO which facilitates the extraction of luminance invariant descriptors.

• a locally adaptive, image-matching-optimal OpTMO which collectively address the detection and description stages of the feature extraction pipelines.

Learning a Tone Mapping Operator for Efficient Image Matching

• an efficient method for generating appropriate training samples to circumvent the difficulty to train SVR in the context of DesTMO and OpTMO respectively. Additionally, we propose their differentiable surrogate objective functions.

• an evaluation of DesTMO and OpTMO against state-of-the-art methodologies. Furthermore, we show an applicative scenario of object localization.

Descriptor Optimal Tone Mapping Operator (DesTMO)

Our design idea is motivated by the detector optimal TMO of the previous chapter where significant gains in Repeatability Rate [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF] were observed when optimal TMO parameters (controlling TMO's shape and size) were learned pixel-wise. However, we mainly focused on designing a tone mapping model for corner-like keypoint detection task, while here we consider a different problem, i.e., an optimal TMO for the extraction of discriminative descriptors.

To design DesTMO, initially a tone mapping function is introduced, which can be locally modulated by spatially varying its parameters. Its parameter maps are predicted by means of a learned illumination-invariant guidance model. Our guidance model is driven by the SVR and relies on the gradient orientation-based features that are extracted from densely sampled patches from the HDR content.

Unlike corner detection, descriptor extraction depends on the large set of neighborhood pixel-set (or patch) which are processed altogether to formulate the discriminative unique signature. Hence, we propose to learn the TMO parameters locally but based on patch-level information from the scenes. Specifically, since each descriptor is restricted to a patch size such as 16 × 16 in SIFT and SURF, we learn the TMO parameters on patches of the same size.

Since there is no standard dataset to train or test any model for DesTMO, we propose a simple 'descriptor similarity-maximization' approach to generate appropriate training samples. The objective function aims to maximize the similarities of descriptors if they are extracted from images from the same location but with lighting variations.

Proposed Model

I (x) = ϕ(I(x), θ), (5.1)
where I ∈ m×n , I is of size m×n with pixel values in the [0, 255] range, and θ represents a vector of modulation maps, θ = {θ 1 , θ 2 }, where θ k is of size m × n. Secondly, the framework consists of a guidance model where an SVR predicts the optimal values of these modulation maps θ by using the densely extracted local features from the HDR content. To this end, initially, the HDR image is densely sampled into patches of size s × s and from each such patch a SIFT feature f is extracted. Then, these features are fed to the regressor which in turn predicts parameter values for modulation map θ 1 , θ 2 . Note that the regressor output for each feature is applied over the size s × s in these modulation maps, corresponding to exact location of the sampled patch from which the feature is extracted. Such patch level tuned vector parameters θ 1 , θ 2 are later used by ϕ to obtain the tone mapped image I .

Tone Mapping Function

Inspired by illumination normalization TMOs [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF][START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF] and similar to Chapter 4, our tone mapping function ϕ in Eq. (5.1) is expressed as:

ϕ = I • L -1
, where the illumination component L is estimated by a variant of bilateral filtering [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF] and is given as:

L(x, θ) = 1 W • y∈Ω G θ 1 (x) ( x -y ) • G θ 2 (x) ( I(x) -I(y) )I(y), (5.2)
where G is a Gaussian kernel and for each pixel location x, the pixel y is in the neighborhood set Ω. The normalization factor W is equal

to y∈Ω G θ 1 (x) ( x -y ) • G θ 2 (x) ( I(x) -I(y) ).
Here, the modulation vector θ has two components: θ 1 and θ 2 . They are often globally referred to as spatial and range variance respectively and control the behavior of function ϕ.

For example, if θ 2 is predicted higher at a patch location, its corresponding Gaussian kernel widens and flattens behaving like a Gaussian blur [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF], and finally, a blurred luminance L is estimated. In such condition, the final tone mapped pixels, which are obtained by normalizing the estimated L for the corresponding patch, will preserve the structures such as gradients.

Notice that we opted for bilateral filtering because its proposed formulation facilitates the integration of the core concept of local modulation. However, any other tone mapping function with parametric formulations such as [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] could be used.

Learning a Tone Mapping Operator for Efficient Image Matching

f i ) = (ω T f i + b),
where ω, b are estimated by minimizing the mean square error. However, such function is often incapable of separating the non-linearly sampled data, like our case where f i is the SIFT feature with size 128, and o(i) = θ k(i) , where k = 1, 2. Therefore, with such given inputs, we use the non-linear SVR [START_REF] Smola | A tutorial on support vector regression[END_REF] which maps the input vector f i into high dimensional space using the kernel ψ where data becomes linearly separable and is given as

r(f i ) = (ω T ψ(f i ) + b).
To fit the desired non-linear SVR prediction function, the following optimization problem is solved:

min ω,b,ξ,ξ * 1 2 ω 2 + C n i=1 (ξ i + ξ * i )
subject to:

θ k(i) -(ω T ψ(f i ) + b) ≤ χ + ξ i , (ω T ψ(f i ) + b) -θ k(i) ≤ χ + ξ * i , ξ i , ξ * i ≥ 0, i = 1..n
where ξ, ξ * are the slack variables, C represents the cost which is imposed for samples that exceed the error χ. For further understanding of the non-linear SVR optimization problem, we refer the reader to [START_REF] Smola | A tutorial on support vector regression[END_REF].

Generation of Samples

To train the SVR, we need to find appropriate training features and their corresponding supervised observations θ 1 , θ 2 as shown in Fig. 8.9. To this end, we propose a two step solution. First, we identify key locations in a scene, where we can extract meaningful descriptor features. Second, we build a model to find the optimal θ 1 and θ 2 that maximize

Descriptor Optimal Tone Mapping Operator (DesTMO)

75 the similarity between those descriptors which are captured from the same key locations of the scene.

To identify key locations, we first detect keypoints independently in each log-scaled HDR image of the scene using the DoG [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] detector. We then iteratively check, for each detected keypoint, whether it is found at about the same location in other images of the same scene, taken under different illumination conditions. If it is detected in the majority of these images, we call it a key location. As we just want to collect 'meaningful' key locations with majority occurrence under lighting variations, any other format could also be used instead of log-HDR.

From each key location, we use SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] as training feature, extracted from linear HDR content. More specifically, it is given as concatenation of 16 unnormalized cells i.e.[x 1 , ...

x 16 ]
where each cell can be compactly defined as [START_REF] Dong | Multi-view Feature Engineering and Learning[END_REF][START_REF] Vedaldi | VLFeat: An Open and Portable Library of Computer Vision Algorithms[END_REF]: Similarity model: We assume a scene S consisting of n HDR images with lighting variations as shown in Fig. 8.9. We consider P = (1, 2), (2,3).., to be the set of K = N 2 pair combinations of N descriptors extracted from a key location. Our aim is to minimize the following objective function:

h(Θ|p)[x] = G δ (Θ -∠∇p(y))G σ(y -x) ∇p(y) d(y) (5.3
F(θ) = 1 K {i,j}∈P Φ(h i (θ), h j (θ)). (5.4)
We define function Φ using the logistic penalty (similar to max-margin formulations in [START_REF] Rana | Feature learning for the image retrieval task[END_REF]),

Φ(h i , h j ) = log(1 + exp( -h T i h j ).
(5.5)

We optimize the objective function in Eq. (5.4) using a robust optimization technique, Stochastic Gradient Descent (SGD) [START_REF] Bottou | Stochastic Gradient Descent Tricks[END_REF]. SGD update rule to estimate θ maps at each iteration t is given as:

θ t+1 = θ t -γ t • ∇Φ {i,j}t (θ t )
, where γ t is a learning rate which decays with t as γ t = γ 0 /(t + 1) and the gradient for the objective in Eq. (5.6) is replaced (as detailed in [START_REF] Bottou | Stochastic Gradient Descent Tricks[END_REF]) with the gradient of a randomly chosen sample pair {i, j} at time t, i.e.∇Φ {i,j} (θ t )

∂Φ(h i ,h j ) ∂θ θt .

Learning a Tone Mapping Operator for Efficient Image Matching

Results and Discussion

Experimental Setup

We build the test setup for image matching using the HDR luminance dataset shown in Fig. 5.3 which consists of 4 indoor and 4 outdoor scenes as detailed in Chapter 4. We compare our proposed DesTMO with the classical perception based TMOs: BTMO [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF],

ChiuTMO [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF], DragoTMO [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF], ReinhardTMO [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] and MantiukTMO [START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF].

The BTMO in [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF] and ChiuTMO [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF] are also based on normalizing the estimated luminance L but use global parametric settings. DragoTMO [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF] maps the HDR content based on adaptive logarithmic scaling. ReinhardTMO [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] and MantiukTMO [START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF] are well known tone mapping techniques for high visual quality outputs with appealing brightness and contrast. We considered these TMOs as they have been previously applied for HDR evaluation studies [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF][START_REF] Suma | Evaluation of the effectiveness of HDR tone-mapping operators for photogrammetric applications[END_REF] for the related task of feature detection.

To effectively evaluate the impact of descriptor extraction scheme, we selected the strongest 500 keypoints using the DoG detector [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] for each tone mapped image. Then, we use four popular and widely used SURF [START_REF] Bay | SURF: Speeded up robust features[END_REF] and SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] descriptor schemes as well as the FREAK [START_REF] Ortiz | Freak: Fast retina keypoint[END_REF] and BRISK [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF] binary descriptors.

Metrics:

We evaluate the descriptor performance using the standard measures of Matching Score and mAP rates as detailed in Section 2.2.3. To define a match, we use the standard nearest neighbor distance ratio (NNDR) matching strategy. Implementation. We use the SVR implementation of LibSVM [START_REF] Chang | Libsvm: A library for support vector machines[END_REF] using the Radial Basis Function (RBF) kernel. To obtain the optimal values of SVR parameters, the regularization cost and epsilon-SVR are tuned by 10-fold cross validation from the range of [2 -5 , 2 15 ] and [2 -10 , 2 5 ], respectively. We use the HDR Toolbox [START_REF] Banterle | Advanced High Dynamic Range Imaging: Theory and Practice[END_REF] for the implementation of the considered TMOs, Matlab's Computer Vision toolbox for SURF, FREAK, BRISK and Vlfeat [START_REF] Vedaldi | VLFeat: An Open and Portable Library of Computer Vision Algorithms[END_REF] for SIFT.

Training and Implementation details

Evaluation Results

We perform a thorough evaluation of our proposed DesTMO quantitatively using the matching score and mAP. We initially show in Fig. 5.4 the performance of our method over all test scenes using the SURF descriptor, where we compare our algorithm with BTMO [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF] and the best exposure LDR. Our results clearly show that predicted local modulation of the bilateral filtering helps in preserving the invariance of the local gradient and hence, boosts the average number of correct matches in both the indoor and outdoor scenes. However, we observe small gains in outdoor scenes such as Invalides. This can be explained by strong lighting transitions and is partially due to increased false matches due to repetitive structures in the images as shown in Fig. 5.7. Note that, we use threshold th = 0.2 to avoid ambiguous matches and to improve the readability of descriptor matching in Fig. 5.7.

Comparison with popular TMOs. We evaluate the performance of our method across different descriptor extraction schemes including both gradient based and binary descriptors. In terms of average matching score, we observe that for every extraction scheme, our DesTMO yields a higher number of correct matches, as shown in Fig. 8.6. Furthermore, in Fig. 5.6, we compute the mAP rates by averaging the area-under-the-curve of PR curves of the complete dataset. We observe that for every descriptor extraction scheme, our

proposed model outperforms all the other TMOs. Additionally, we compare our proposed TMO with popular and visually pleasing Reinhard TMO [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] and MantiukTMO [START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF] in Fig. 5.7, where we show that our method produces a higher number of correct matches in difficult day/night matching.

Optimal Tone Mapping Operator for Image Matching

In the previous Chapter, we designed a detector-optimal DetTMO, controlled by a guidance model which is learned to understand the keypoint's locally extremal and covariant characteristics. Similarly, we introduced a descriptor-optimal DesTMO where the guidance model is mainly trained to facilitate the invariant densely-sampled descriptor extraction.

However, both these TMOs only handle one aspect at a time, namely, keypoint detection or descriptor extraction. This is inefficient in practice for the image matching task, e.g., a poor detector degrades descriptor matching [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF].

Notice that optimizing a TMO considering keypoint detection and description concurrently is not trivial, as the corresponding design objectives are generally different and somehow contrasting. For instance, an optimal TMO for detection aims to produce covariant feature points, while a TMO optimal for description should guarantee some form of invariance to transformations over a local neighborhood. In addition, optimal detection requires an accurate localization of keypoint position, while optimal description is a patch-level process. In Chapter 3 [START_REF] Rana | An evaluation of HDR image matching under extreme illumination changes[END_REF], we have showed that TMOs that are optimal for detection are not necessarily so when the full matching chain is considered.

In this section, we introduce an optimal tone mapping operator (OpTMO) to enhance the detection and matching of features extracted from HDR scenes captured under complex real-world illumination transitions. For OpTMO, we initially introduce a tone mapping function similar to DetTMO, which can be locally modulated by varying spatially (pixelwise) its parameters as a function of the HDR content characteristics. Afterwards, we propose a guidance model to map HDR-based local characteristics features (detection and description-based) to a low-dimensional TMO parameter space, by means of a support vector regressor (SVR) [START_REF] Smola | A tutorial on support vector regression[END_REF].

For OpTMO, we obtain per pixel ground-truth TMO parameters by solving an optimization problem which simultaneously ensures: 1) stable keypoint detection; and 2) keypoint description robust to illumination changes. Since these two objectives are, in general, non differentiable, we also propose the proxy cost functions which enables to compute the required derivatives and obtain an optimal solution. In the following, we detail each step.

Optimal Tone Mapping for Image Matching

Generation of Training Set

In this section, we address the problem of generating an adequate ground truth set for training the SVR-based model. We aim to find such ground truth parameter maps θ, which result in efficient image matching (i.e. mAP score) for a scene which undergoes drastic lighting variations, as shown in Figure 5.9. In this section, we, therefore, formulate an objective function f , which we minimize over the θ to yield the optimal parameter maps.

The proposed total energy f represents the difference in the image matching pairs. We quantify this difference in terms of both keypoint detection and descriptor extraction stages, depicted as 'Detection Response' and 'Description', respectively, in Figure 5.9. Finally, we propose to optimize the objective using the SGD based optimization method to obtain the optimal θ.

In the following, we first discuss the formulation of the objective function f . Then, we detail the considerations with respect to image matching components in view of designing the objective function f . Finally, we detail the SGD-based method to optimize the objective to obtain the optimal θ.

Objective Function

We aim to optimize θ to tone map an image for the full feature extraction pipeline.

Therefore, the objective function should consolidate each stage of the feature extraction pipeline i.e. to locate and extract the feature. Henceforth, we introduce two energy terms dedicated to keypoint localization (E det ) and descriptor extraction (E des ), respectively and define a combined objective function as:

minimize θ f (θ) = E det (θ) + E des (θ). (5.6)
where each energy term is computed over a scene consisting of N HDR images with lighting variations as shown in Fig. 5.9 (a). Let P = (1, 2), (2, 3), . . . is the set of K = N 2 pair combinations of N images. The E det term aims to ensure the covariance of the corner response maps. Conversely, the E des term helps in retaining the invariance of the discriminative patterns around the key locations in the image pairs when undergoing drastic transformations. Both terms are detailed as follows:

The Energy term E det To ensure efficient matching, we observe that it is important to enforce the similarity in detection response maps [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF]. This is mainly because high similarity response maps increases the probability of detection of keypoints at similar locations and thereby enhances the probability of correct matches.

We define the detection similarity term E det , by summing the penalty computed from each pair in the set K, as:

E det = λ det K {i,j}∈P C 1 (R i (θ), R j (θ)). (5.7)
For each sample pair {i, j} ∈ P , we penalize the response maps dissimilarity by a logistic cost function given as:

C 1 (i, j) = log(1 + exp( c -R i • R j ), (5.8)
where c is the penalty control factor, R i and R j are the response maps corresponding to the images i, j which belongs to a scene, and • denotes the scalar product. The selection of R is detailed later in this section.

Inspired by the max-margin formulations applied to retrieval [START_REF] Rana | Feature learning for the image retrieval task[END_REF] or classification tasks [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], we use the logistic function as the penalty in our detection term. It is a smooth differential operator and ideally penalizes less if there is high similarity and vice-versa.

Note that the term E det is somewhat similar to the one we proposed in detector optimal TMO in Chapter 4. But, in this work, we include an additional factor λ det which weights

Learning a Tone Mapping Operator for Efficient Image Matching

the penalization corresponding to detection.

Selection of Response : From handcrafted [START_REF] Schmid | Evaluation of interest point detectors[END_REF] to deep-learning [START_REF] Yi | LIFT: Learned Invariant Feature Transform[END_REF] era, the concept of corner-like keypoint detection methods has gained popularity for low-latency vision tasks due to high speed, less computational complexity and competitive accuracy. By definition, corners exhibit low correlation with neighboring pixels in all directions. The most basic and widely adopted corner detectors [START_REF] Förstner | Detecting interpretable and accurate scale-invariant keypoints[END_REF][START_REF] Harris | A combined corner and edge detector[END_REF][START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF] localize the extrema primarily by computing the per pixel gradient autocorrelation matrix, given as:

M = I 2 x I xy I yx I 2 y , (5.9)
where each component represent the directional derivative. Thereafter, different methods are proposed in the literature to localize the extrema "keypoints" [START_REF] Schmid | Evaluation of interest point detectors[END_REF]. In this work, we use [START_REF] Harris | A combined corner and edge detector[END_REF] which describes the response for each pixel x without directly computing the eigenvectors of M as:

R(x) = det{M (x)} -k • tr{M (x)} 2 , (5.10)
where k is tuned empirically.

Similar to the baseline as discussed in Section 4.3.3, we employ the detector response in Eq. (5.10) mainly because it is based on the popular structural matrix M , which is simpler to differentiate than alternative approaches, thus aiding in backpropagation. Note

that alternate detection methods could also be used, but our choice has been made entirely based on the computation complexity and ease of use in backpropagation.

The Energy term E des

The energy term E des aims to penalize the dissimilarity of the descriptors extracted from the tone-mapped images. Previously in DesTMO, we proposed a densely sampled patch-based method where a model is learned to predict global parametric values for an individual patch. Hence, not only the method optimized θ for a patch globally, but it also lacked the consideration of keypoint localization. In contrast, the image matching pipeline additionally relies on the localization of the descriptors. Hence, here we argue that it is important to compute the gradient orientation impact per pixel and to focus on its locations prior to designing a descriptor-based penalty function. It not only helps in preserving the salient locations but also avoids any "look-alike" redundant matches [START_REF]Learning-Based Tone Mapping Operator for Image Matching[END_REF]. Therefore, we propose to constraint the penalization to the dissimilarity of those descriptors that belong to some potential keypoint region. We define E des as:

E des = λ des K {i,j}∈P C 2 (D i (θ) -D j (θ)), (5.11)

Optimal Tone Mapping Operator for Image Matching

83

where C 2 is the Euclidean distance and λ des is a weighting factor. To apply the constraint in practice, we compute the descriptor D after the keypoint localization which is obtained by applying the softargmax operation S [START_REF] Chapelle | Gradient descent optimization of smoothed information retrieval metrics[END_REF] on the resulting response map. In general terms, S is given as

S = i exp(βz i ) j exp(βz j ) • i (5.12)
where z i is the pixel location and β is a hyper-parameter for defining the shape parameter.

The softargmax operation is a differentiable function to obtain local optima and helps in avoiding the cluttering in response maps. Cluttering refers to a phenomenon when several keypoints are located close to each other [START_REF]Learning-Based Tone Mapping Operator for Image Matching[END_REF].

To compute an accurate keypoint localization, we define the final gradient orientation around each pixel location computation as follows:

D =    h(ν|p), if S(R) ≥ Λ 0, otherwise (5.13)
where h(ν|p) is the gradient orientation feature map explained later in Eq. (5.14) and Λ is the maximum softargmax value in a 16 × 16 neighborhood window of the considered pixel.

It simply means that if the softargmax response score for the considered pixel location is maximum in its neighborhood window, only then the gradient orientation map is taken into account to contribute in the final descriptor-based penalty term in Eq. (5.11).

Selection of h : A common image matching approach relies on the similarity of features extracted from patches corresponding to detected keypoint locations. One widely used descriptor extraction algorithm is the Scale Invariant Feature Transform (SIFT) [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] which is a concatenation of 16 unnormalized cells i.e. [c 1 , ...c 16 ], where each cell can be compactly defined as [START_REF] Dong | Multi-view Feature Engineering and Learning[END_REF][START_REF] Vedaldi | VLFeat: An Open and Portable Library of Computer Vision Algorithms[END_REF]:

h(ν|p)[c] = G δ (ν -∠∇p(y))G σ(y -c) ∇p(y) d(y), (5.14)
where c is the center location of the cell in the restricted square patch p of size 16 × 16.

The independent variable ν represents the gradient orientation ranging in [0, 2π]. Moreover, G represents the Gaussian kernel with standard deviation σ and an angular dispersion parameter δ. Once histograms are computed, they are normalized and concatenated into a single 128-dimensional descriptor.

SGD Implementation Details

We optimize the objective function in Eq. (5.6) using stochastic gradient descent (SGD) [START_REF] Bottou | Stochastic Gradient Descent Tricks[END_REF].

It is a fast and robust optimization technique to estimate the incremental gradient descent by its stochastic approximation using a randomly chosen sample from the initial set. To implement the SGD based optimization, we follow the backpropagation procedure. We initially build the required partial derivative framework with the objective given in Eq. (5.6).

It is more formally expressed as

∇C {i,j} (θ) = ∂C 1 ∂R l • ∂R l ∂ϕ l • ∂ϕ l ∂θ + ∂C 2 ∂R l • ∂R l ∂ϕ l • ∂ϕ l ∂θ l=i,j (5.15)
Then, following the SGD rule, we iteratively estimate θ by randomly selecting sample (i, j) from the set P . Finally, we compute the gradient of the objective in Eq. (5.6), that is:

∂f ∂θ θt = 1 K {i,j}∈P ∂C ∂θ θt (5.16)
where

C = λ det • C 1 (R i , R j ) + λ des • C 2 (D i , D j ), (5.17)
with the single (i, j) selected image pair. Thereafter, at each iteration t, SGD update rule is given as:

θ t+1 = θ t -γ t • ∇C {i,j}t (θ t ), (5.18)
where γ t is a learning rate is made to decay with t as γ t = γ 0 /(t + 1), and the gradient for the objective function in Eq. (5.6) is replaced by the gradient of a randomly chosen sample pair {i, j} at time t, i.e.

∇C {it,jt} (θ t ) ∂C(R it , R jt , D it , D jt ) ∂θ θt . (5.19)
For SGD based optimization, we start from a randomly initialized set of θ which are updated iteratively using the update rule in Eq. (5.18). In total, the model comprises 3

hyper-parameters: γ 0 , λ det , λ des . To estimate these hyperparameters, we follow the standard approach used in [START_REF] Verdie | TILDE: A temporally invariant learned detector[END_REF] and take a small set of pairs from P and perform a simple crossvalidation using the grid search method in the log scale. For the SGD related optimization and convergence proofs along with the aysmptotic analysis, we refer the reader to [START_REF] Bottou | Stochastic Gradient Descent Tricks[END_REF].

This proposed mechanism for finding the optimal parameters θ for a function ϕ using SGD is generic, i.e. one can easily tune the parameter maps of any TMOs that can be expressed as Eq.(5.1).

Support Vector Regressor Training for OpTMO

Consider the sample set of characteristic features F = {f 1 , . . . , f n } and the corresponding output denoted by Y = {θ k(1) , . . . , θ k(n) } where k = 1, 2 in our case. To build our predictor model, we want feature samples which capture distinctive information for both descriptor and detector. To that end, we build our feature sample f i by concatenating two parts: a)

the gradient-based SIFT pattern [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] (64 dimensional feature); and b) the 5 × 5 grid-based detector response feature [START_REF]Learning-based Adaptive Tone Mapping for Keypoint Detection[END_REF] (25 dimensional feature). This forms a total dimension of 89.

The features f k are computed from the original HDR linear values, without any processing. This is not contradictory with the need to perform a TMO as, locally, HDR images generally display limited dynamic range [START_REF] Bronislav | Evaluation of feature point detection in high dynamic range imagery[END_REF]. Finally, for each training sample, we get the following input-output corresponding pairs {(f 1 , θ k(1) ), ..., (f n , θ k(n) )} and formulate our prediction problem using SVR. To fit the desired nonlinear SVR prediction function, the corresponding optimization problem is solved using the dual maximization approach.

Experimental Results and Discussion

Dataset : We consider the HDR dataset presented in our previous designs of DetTMO and DesTMO, which is composed of 8 different HDR scenes as shown in Fig. 5.3.

Evaluation Metrics

We evaluate the keypoint detection and descriptor extraction performance on the tone mapped images using the standard measures of Repeatability Rate (RR) and Matching Score (MS) respectively, as detailed in Section 2.2.3. For the evaluation of the full image matching, we compute the mean average precision (mAP) scores [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF].

Learning a Tone Mapping Operator for Efficient Image Matching

Evaluation Setup

We test our proposed OpTMO for image matching task on 8 HDR scenes at detection and description levels and compare with state-of-the-art TMOs. The HDR dataset is composed of a total of 52 images. For detection and description stage, we formulated a total of 280 test image pairs respectively from the 8 scenes.

We compare the proposed OpTMO with classical perception-based TMOs, including:

BTMO [START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF], ChiuTMO [START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF], DragoTMO [START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF], ReinhardTMO [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] and MantiukTMO [START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF].

In addition, we also consider our previously proposed DetTMO [START_REF]Learning-based Adaptive Tone Mapping for Keypoint Detection[END_REF] and DesTMO [START_REF]Learning-Based Tone Mapping Operator for Image Matching[END_REF],

which are optimized methods for detection and description only, respectively.

SVR Training and Implementation

We use the SVR implementation of LibSVM [START_REF] Chang | Libsvm: A library for support vector machines[END_REF] using the Radial Basis Function (RBF) kernel. The optimal values of SVR parameters, the regularization cost and epsilon, are obtained by 10-fold cross validation from the range of

Keypoint Detection

We evaluate all the considered TMOs using Harris [START_REF] Harris | A combined corner and edge detector[END_REF], FAST [START_REF] Rosten | Machine learning for high-speed corner detection[END_REF], BRISK [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF], SURF [START_REF] Bay | SURF: Speeded up robust features[END_REF] and SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] (as detailed in Section 2.2). We selected these detection methods based on the state-of-the-art studies in evaluating the performance of TMOs [START_REF] Bronislav | Feature point detection under extreme lighting conditions[END_REF][START_REF] Chermak | Enhanced feature detection and matching under extreme illumination conditions with a hdr imaging sensor[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF][START_REF]Learning-Based Tone Mapping Operator for Image Matching[END_REF] and also due to their popularity in real time applications [START_REF] Tuytelaars | Local Invariant Feature Detectors: A Survey[END_REF].

RR [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF] is sensitive to the number of detected keypoints and the error rate . For instance, large variations in the number of keypoints across different scenes might lead to biased average scores. Therefore, we fix the keypoint detection to the strongest N keypoints as suggested in prior TMO evaluation studies [START_REF] Bronislav | Evaluation of feature point detection in high dynamic range imagery[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF][START_REF] Rana | An evaluation of HDR image matching under extreme illumination changes[END_REF]. The impact of N and over average RR score is shown in Fig. 5.10. Overall increase in number of keypoints leads to an increase in average RR but the growth slows down after a certain number, partially due to the detection of cluttered keypoints. On the other hand, increase in the average RR with the increasing is in coherence with the findings of [START_REF] Schmid | Evaluation of interest point detectors[END_REF]. Here, we choose the values N = 500 and = 10.

Implementation We use the HDR Toolbox [START_REF] Banterle | Advanced High Dynamic Range Imaging: Theory and Practice[END_REF] for the implementation of the considered TMOs. Moreover, we use the Matlab's Computer Vision toolbox for Harris, FAST, BRISK and SURF, and Vlfeat for SIFT.

Comparison We perform a thorough evaluation of our proposed OpTMO quantitatively using the RR measure. In Fig. 8.12, we initially show the performance of our OpTMO and other state-of-the-art TMOs in terms of RR averaged over all test scenes. For the sake of completeness, we also report the average RR obtained using HDR linear photometric values (HDRLin), without any tone mapping. Our results clearly show that the proposed OpTMO outperforms all the perception-based TMOs. In addition, the significant drop in performance with HDRlin demonstrates that HDR linear values are highly sub-optimal for keypoint detection task, similar to what is found in previous studies [START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF][START_REF] Rana | An evaluation of HDR image matching under extreme illumination changes[END_REF].

In Fig. 5.12, we expand our experimental test bench for each scene and compare the performance of our OpTMO with the globally optimized BTMO [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF] and our previously proposed detector-optimal DetTMO [START_REF]Learning-based Adaptive Tone Mapping for Keypoint Detection[END_REF]. The per scene gains of OpTMO over BTMO prove that local modulation of parameters significantly improves the keypoint stability. In addition, we observe that the gain in performance between local and global optimization depends significantly on content characteristics. Especially for indoor scenes, which have been acquired by varying locally the illumination and introducing stark shadows, local

Learning a Tone Mapping Operator for Efficient Image Matching

parameter tuning enables to obtain important RR gains. We also notice that OpTMO achieves similar (within 2-4% per scene) RR as DetTMO, which is optimized for keypoint detection only and thus provides an upper bound in the achievable repeatability.

In order to further confirm these observations, we report a head-to-head comparison of OpTMO versus BTMO and DetTMO, respectively, in Fig. 5.13, for two different detectors:

Harris (corner) and SURF (blob). OpTMO has higher RR whenever a point (representing a specific scene and illumination condition) is above the 45 • line. As expected, we observe that this is often the case for BTMO, while for DetTMO the two methods have very similar performance. As mentioned above, the loss in keypoint repeatability compared to DetTMO is expected, and is mainly due to two reasons. On one hand, the additional descriptor-level cost term in Eq. ( 5.11) changes the objective function with respect to detector repeatability only (as in DetTMO). On the other hand, the use of the softargmax localization measure in Eq. ( 5.12) reduces cluttering of keypoints in our OpTMO. This is illustrated on a detail of the "Project-Room" scene in Fig. 5.14. For instance, cluttered keypoints are detected near the beaver's eyes in DetTMO, whereas OpTMO handles such detections efficiently.

Interestingly, the composite objective function in Eq. (5.6) enables to achieve RR almost as good as DetTMO, but with a significantly improved descriptor matching and thus overall image matching performance, as shown in the next section. .12 -Keypoint Detection II: Average Repeatability Rates (RR) computed using BTMO [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF], DetTMO [START_REF]Learning-based Adaptive Tone Mapping for Keypoint Detection[END_REF] and the proposed OpTMO for each test scene using Harris keypoint detector.

Descriptor Matching

We perform a thorough evaluation of our proposed OpTMO for descriptor matching using BRISK [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF], FREAK [START_REF] Ortiz | Freak: Fast retina keypoint[END_REF], SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] and SURF [START_REF] Bay | SURF: Speeded up robust features[END_REF] descriptors. We use the matching score (MS) as performance measure considering the NNDR matching criteria with a threshold value th = 0.5.

Implementation We use the Matlab's Computer Vision toolbox for FREAK, BRISK

and SURF, and Vlfeat for SIFT, with their default parameter settings.

Comparison In Fig. 5.15, we compare the average OpTMO MS with respect to stateof-the-art TMOs. Overall, we attain significant gains in terms of MS using all feature extraction methods. With th = 0.5 (default value [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF][START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF]), the gains are considerable for gradient-based features schemes such as SIFT and SURF, which is expected by design given the definition of the descriptor signature in Eq. (5.13).

To further analyze these results quantitatively, in Fig. 5.16 we report per scene comparison between the competing TMOs that are observed from Fig. 5.15. We observe that for each scene (indoor or outdoor) our OpTMO outperforms all the other TMOs. As in Section 5.4.7, we observe considerable gains with respect to traditional BTMO, confirming the potential of local parameter optimization. In comparison to DetTMO, we observe that gains are not as high as what are obtained with BTMO. This can be explained by the higher RR of the DetTMO (Fig. 5.12) which improves the probability of the correct matches. Interestingly, we also observe that in many scenes DetTMO and DesTMO perform equally well, e.g., Invalides and Project-Room scenes. This is mainly because DesTMO is not optimal for detection, which entails a higher number of false matches.

Finally, we show the per image-pair analysis in Fig. 5.17 to further analyze the behavior of individual test pairs. We observe that our OpTMO improves the MS over DesTMO across the whole dataset (i.e., the gains are not concentrated on specific image pairs). In fact, there is not a single case where there is a significant drop in OpTMO's performance against the descriptor-optimal DesTMO, which again confirms the advantages of simultaneously optimizing the TMO for keypoint detection and description. In addition, the OpTMO produces consistent gains even if a binary descriptor such as FREAK is employed, in spite of the use of a gradient-based cost function in Eq. (5.13).

Note that MS is sensitive to the choice of th. Therefore, in the following section, we perform a global image matching evaluation using mAP to overcome the impact of the threshold.

Image Matching

We evaluate the full image matching chain by computing mean average precision (mAP) scores over the complete dataset. We obtain the mAP rates by averaging the area-underthe-curve of PR curves [START_REF] Mikolajczyk | A performance evaluation of local descriptors[END_REF]. The results per TMO are reported in Fig. 5.18. We observe that for every descriptor extraction scheme our proposed model outperforms all the other TMOs. High mAP scores imply that our model obtains more correct matches and reduce the probability of false matches. An illustration of matching results is given in Fig. 8.14,

showing that the proposed full-chain optimal tone mapping improves the matching efficiency in drastic lighting variations. Notice that ReinhardTMO and MantiukTMO provide poor image matching results compared with the proposed approach, although they provide Matching Score comparison between BTMO [START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF], OpTMO, DetTMO [START_REF]Learning-based Adaptive Tone Mapping for Keypoint Detection[END_REF] and DesTMO [START_REF]Learning-Based Tone Mapping Operator for Image Matching[END_REF] over all the scenes in the HDR dataset using SURF feature extraction scheme. OpTMO over all the scenes in HDR dataset. The points represented using o corresponds to FREAK feature detection scheme and 2 corresponds to SURF scheme.

Applications

Localization of objects is a high-precision and pivotal task in many computer vision applications, e.g.to find region of interest for fine-grained recognition challenges. For localization, first a homography matrix is computed by finding the best matching correspondences 92

Learning a Tone Mapping Operator for Efficient Image Matching

between the target and the test image. Then, the desired object is localized based on the estimated geometric relationship. In Fig. 5.20 and Fig. 5.21, we show a similar applicative scenario of localization of selected objects such as structures in images undergoing both lighting and rotational transformations. We compare the performance of our proposed image-matching optimal TMO and the widely used ReinhardTMO over three scenes, namely Louvre, ProjectRoom and Notre Dame. In Fig. 5.20, we first find the corresponding matches between the two scenes using the SURF scheme for each TMO. Then, based on those resulting matches, we estimate the homography as proposed in [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF]. We observe that our model gives more correct corresponding matches in all three scenes as compared to ReinhardTMO. In challenging outdoor scene such as Louvre where there is a direct impact of sunshine, we observe that ReinhardTMO results in all incorrect matches, mainly concentrated in the brightest regions. In Fig. 5.21, we overlay the results on the test tone-mapped images to show where exactly our desired object should be located based on the obtained correspondences. In Louvre and ProjectRoom scenes, we observe that tone-mapped images using our proposed model result in correct localization of the desired object in the test image, as compared to ReinhardTMO. In the Notre Dame scene, the impact of illumination on the target region is smaller, and we are able to find correct overlaying results using both tone mappings.

Conclusions

We propose a novel task optimal TMOs to improve the descriptor discriminability and image matching efficiency under drastic changes of lighting conditions. To this end, we first Finally, we demonstrated the performance of our method over other TMOs in a simple localization based application scenario. Our proposed task-optimal TMO can be applied to So far, we build our optimal TMO model by proposing the variants of existing tone mapping functions and learning their parameter maps. Although we obtained gains on image matching task over the state-of-the-art, the proposed models are bounded by the limited functionalities of these exiting tone mapping functions. Hence, in the following, instead of learning the tone mapping parameters, we will focus on directly learning a tone mapping function.

The work presented in this chapter has resulted in the following publications: [START_REF] Eilertsen | Hdr image reconstruction from a single exposure using deep cnns[END_REF][START_REF] Endo | Deep reverse tone mapping[END_REF][START_REF] Gharbi | Deep bilateral learning for real-time image enhancement[END_REF], our architecture avoids the requirement of explicitly defining a task specific loss function. This happens mainly because our networks are trained to model by themselves, loss functions adapted from the underlying training data.

To train these proposed models, we accumulate data from available HDR image sources.

However, a major challenge while training the models arise from the absence of any publicly available training dataset. Selecting ground truth through a subjective evaluation over a large dataset is a highly tedious tasks. Thus, it necessitates the requirement of an objective quality assessment metric which can quantify the tone mapping performance of each TMO for any possible scene. For our task, we select a well known metrics namely Tone Mapped Image Quality Index (TMQI), which is used to rank 13 widely used tone mapping operators.

Using this, for each HDR input, we select the one which ranks topmost on this objective metric score as our ground truth tone mapped output.

Finally, our DeepTMO implicitly learns the best characteristic of all available global, local and perceptually based TMOs over a wide variety of scenes. In a sense, both through its architectural design and the underlying dataset, it is conditioned to preserve global features (such as overall structures, contrasts and luminance) as well as local finer details (such as local texture patterns) thus yielding high quality visually pleasing tone mapped outputs.

In a nutshell, 1. We propose the first deep learning based tone mapping operator, which can generate visually pleasing realistic tone mapped outputs for a wide variety of HDR inputs.

2. We fully explore and compare 3 different cGAN architectures designed specifically for generating high resolution LDR tone-mapping outputs preserving overall structural information as well as local fine-grained details.

3. We overcome the challenge of unavailability of ground truth tone mapped images for our HDR dataset by utilizing an objective metric to quantify and rank various TMOs.

4. We provide an extensive comparison of our proposed methodology with thirteen different tone mapped operators over large dataset of 105 images.

5. Our proposed model can be explored for future deep learning based task-optimal TMO designs.

Deep Learning for HDR Image Analysis

Recently, deep Convolutional neural networks (CNNs) have been utilized extensively for multiple high dynamic range imaging tasks such as reconstructing HDR using a single

Deep Learning for HDR Image Analysis

99

exposure LDR [START_REF] Eilertsen | Hdr image reconstruction from a single exposure using deep cnns[END_REF], predicting and merging various high and low exposure images for HDR reconstruction [START_REF] Endo | Deep reverse tone mapping[END_REF] or yielding HDR outputs from dynamic LDR inputs [START_REF] Kalantari | Deep high dynamic range imaging of dynamic scenes[END_REF]. CNNs have also been modeled to learn an input-output mapping as done for de-mosaicking and de-noising by [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] or learning an efficient bilateral grid for image enhancement [START_REF] Chen | Bilateral guided upsampling[END_REF]. [START_REF] Gharbi | Deep bilateral learning for real-time image enhancement[END_REF] have recently proposed a deep bilateral tone mapper, but it works only for 16-bit linear images and not the conventional 32-bit HDR images. A major drawback of all these past techniques is that, although the learning process is completely automated, one still needs to explicitly specify an effective loss function that the CNN learns to minimize. Thus the quality of resulting output is dependent a lot on the choice of our loss function. Formulating a loss function that constrains the CNN to yield sharp tone-mapped LDR from their corresponding linear-valued HDR is complex and an ill posed problem. Although authors in [START_REF] Eilertsen | Hdr image reconstruction from a single exposure using deep cnns[END_REF] formulated the inverse tone mapping task by formulating the problem in log domain, but in our case, we rather work in linear domain to avoid any form of information loss.

Generative Adversarial Networks

Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF] have attended lot of attention owing to their capability to model the underlying target distribution by forcing the predicted outputs to be as indistinguishable from the target images as possible. Through this, they implicitly learns an appropriate loss function, eliminating the requirement of hand crafting one by an expert. This property has enabled them to be utilized for a wide variety of image processing tasks such as super-resolution [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF], photo-realistic style-transfer [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF] as well as semantic image in-painting [START_REF] Yeh | Semantic image inpainting with perceptual and contextual losses[END_REF].

For our task, we employ GANs under a conditional setting, or better called as conditional GANs (cGANs) [START_REF] Mirza | Conditional generative adversarial nets[END_REF], where the generated output is conditioned on the input image. One distinctive feature of cGAN framework is that they learn a structured loss where each output pixel is conditionally dependent on one or more neighboring pixels in the output image. Thus, this effectively constrains the network by penalizing any possible structure that differs between output and target. This property is quite useful for the task of tonemapping where we only want to compress the dynamic range of an HDR image, keeping the structure of output similar to our desired target. For this specific reason, cGANs have been quite popular for the task of image-to-image translation, where one representation of a scene is automatically converted into another, given enough training pairs [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] or in an unsupervised setting [START_REF] Liu | Unsupervised image-to-image translation networks[END_REF][START_REF] Tung | Adversarial inversion: Inverse graphics with adversarial priors[END_REF][START_REF] Zhu | Unpaired image-to-image translation using cycleconsistent adversarial networks[END_REF]. However, a major limitation of using cGANs is that it is quite hard to generate high resolution images due to training instability and other optimization issues. The generated images are either blurry or contain noisy artifacts.

In [START_REF] Chen | Photographic image synthesis with cascaded refinement networks[END_REF], motivated from perceptual loss [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], a direct regression loss is derived to generate high resolution 2048 × 1024 images, but this method fails on preserving fine-details and textures. In [START_REF] Wang | High-resolution image synthesis and semantic manipulation with conditional gans[END_REF] significant improvement have recently been shown on the quality of high-resolution generated outputs through a multi-scale generator as well as discriminator. Training dataset consists of input HDRs and their corresponding best-TMQI ranked tone mapped outputs. Both the discriminator and generator are trained alternatively, first a gradient step of discriminator then of generator. While the discriminator is trained to discriminate between real and fake image pairs, the generator learns to fool the discriminator by producing synthetic tone mapped images. By doing this, the generator effectively models the underlying distribution of real ground-truth tone mapped images, thus yielding high quality results once completely trained. All our cGAN based tone mapping architectures have two basic modules, namely a generator and discriminator, both of which are conditioned on the linear HDR input.

Proposed Methodology

Both the generator and the discriminator compete with each other, the generator trying to fool the discriminator by producing high quality real looking tone mapped images for the given input HDR, while the discriminator trying to discriminate between real and synthetically generated HDR-LDR image pairs. Our basic discriminator architecture is similar to a PatchGAN [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF][START_REF] Li | Precomputed real-time texture synthesis with markovian generative adversarial networks[END_REF] which classifies patches over the entire image and averages over all of them. Similarly, the basic generator architecture comprises of an encoder-decoder network where the input HDR is given first to an encoder to yield a compressed representation which is then passed to the decoder finally resulting in a tone mapped image. Our three subsequent architectures are variants of this basic framework where DeepTMO-R is a baseline. DeepTMO-S adds skip connections between the encoder and decoder, thus effectively shuttling low level information at the time of prediction. And DeepTMO-HD constructs a multi-scale generator discriminator network, which helps in predicting tone mapped images which are both structurally consistent with the input HDR B) . Residual Blocks consist of two sequential convolution layers applied to the input, producing a residual correction that is in turn added to the input to yield the final output. Discriminator consists of a patchGAN [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF][START_REF] Li | Precomputed real-time texture synthesis with markovian generative adversarial networks[END_REF] architecture which is applied patch wise on the concatenated input HDR and tone mapped LDR pairs. The final prediction is an average of all the patches over the image. which are similar to that of generator in DeepTMO-R. The final prediction from the back end G B 1 is then concatenated with the front end output of the finer-scale generator G 2 . This is then passed through the back end component

Deep Tone Mapping Opeartor for HDR Imaging

G (B) 2
to yield a tone mapped output. Thus effectively our model utilizes both the coarser and finer scale information to make a prediction which results in better retaining of overall structure and minute low level details. Our discriminator architecture has identical architecture only that we give to it two different scales of input, the original and its 2× down sampled version. This forces the coarse-to-fine generator to take care of both global and local details. and at the same time preserves fine grained information recovered at different scales over the entire image.

To build our GT dataset, we provide the details in Section 6.6. All our model architectures are based upon cGAN [START_REF] Mirza | Conditional generative adversarial nets[END_REF] which implicitly learns a mapping from observed HDR image x and random noise vector z, to tone mapped LDR image y, given as: G : x, z -→ y.

In the following, we discuss all the 3 different architectural models in details.

DeepTMO-R

Inspired from [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], our first architecture is composed of two fundamental building blocks namely a discriminator (D) and a generator (G). As shown in Figure 8.17a, input to G consists of an H × W × C size luminance channel of an HDR image normalized between [0,1] where C = 1. While during training we set H = W = 512, inference can be performed with any larger size input. The output from G is a tone mapped image of similar size as the input. D on the other hand, takes as input pairs of luminance channels of HDR and tone mapped LDR images, and predicts whether they are real tone mapped images or fake. Thus in a way, both G and D compete with each other, G trying hard to produce outputs which cannot be distinguished from real image pairs, while the adversary D trying to detect 'fake' image pairs produced by G. Next, we discuss the architectures for both G and D which are our adaptation from [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF][START_REF] Zhu | Unpaired image-to-image translation using cycleconsistent adversarial networks[END_REF] which show impressive results for style transfer and super-resolution tasks.

Generator Architecture Our Generator is formulated as an encoder-decoder architecture as shown in Figure . 8.17a. Overall, it consists of a sequence of 3 components: convolution front end G (F ) , a set of residual blocks G (R) and deconvolution back end G (B) . G (F ) consists of 3 different convolution layers which perform subsequent down-sampling operation on their respective inputs. G (R) is composed of 9 different residual blocks each having 2 convolution layers, while G (B) consists of 3 convolution layers each of which up-samples its input by a factor of 2. For subsequent details, please see 6.4.1.

Discriminator Architecture As shown in Figure 8.17b, our discriminator architecture resembles a 70 × 70 PatchGAN [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF][START_REF] Li | Precomputed real-time texture synthesis with markovian generative adversarial networks[END_REF] model, which aims to predict whether 70 × 70 overlapping image patches are real or fake. Such a patch-level GAN discriminator, models the high-frequency information by simply restricting its focus upon the structure in local image regions. Moreover, it contains smaller number of parameters compared to a full-image size discriminator, and hence can be easily used for any-size images in a fully convolutional manner. This discriminator is run across the entire image, and all the responses over various patches are averaged out to yield the final prediction. Further details can be found in 6.4.2

Although the generator architecture in this case yields high quality results however it still lacks the ability to reconstruct precisely local low level information as is highlighted 104 6. Deep Tone Mapping Opeartor for HDR Imaging in Figure 6.4. Thus based upon the assumption that the underlying structure between an HDR and Tone mapped LDR image remains intact, we try to improve upon the current architecture by shuttling directly the low level information from the encoder layers to the decoder layer outputs.

DeepTMO-S

Various past HDR reconstruction methods, have used skip connections [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] for generating HDR scenes from single exposure [START_REF] Eilertsen | Hdr image reconstruction from a single exposure using deep cnns[END_REF] or multi-exposure [START_REF] Endo | Deep reverse tone mapping[END_REF] LDR images. The basic idea had been that since, both LDR and HDR scenes are different renderings of the same underlying structure, at a particular scale, their structures are also more or less aligned.

Hence, it is possible to effectively transmit low level details from input to output scenes, circumventing the bottleneck of the encoder-decoder architecture. As shown in Figure 8.17a, we modify the generator, by adding skip connections between each layer i and layer n-i, n being the total number of encoder-decoder layers, which as a result concatenates all the channels at layer i with layer n-i.

As seen in Figure 6.4, Skip connections are quite helpful in retaining fine structural details and preserve contrast better, thus yielding high-quality results. However in certain cases, the quality of generated images are still unsatisfactory with certain checkerboard artifacts (see Figure 6.5). This necessitates a generator and discriminator architecture that caters to both the finer details as well as the high level semantics to generate the final tone-mapped image.

DeepTMO-HD

While generating high resolution tone-mapped images, it is quite evident that we need to pay attention to both low level finer details as well as high level semantic information. To this end, motivated from [START_REF] Wang | High-resolution image synthesis and semantic manipulation with conditional gans[END_REF], we propose a new architecture that outlines a coarse-to-fine generator and a multi-scale discriminator in the algorithmic pipeline. We next propose further details regarding both of these architectures.

Coarse-to-Fine Generator As shown in Figure 8.18a, our Coarse-to-Fine Generator consists of two sub-architectures, namely, a Global Generator network G 1 and a local enhancer network G 2 .

The architecture for G 1 is similar to G with the components, convolutional front end, set of residual blocks and convolutional back end being represented respectively as:

G (F ) 1 , G (R) 1 , G (B)
1 . Only difference is in the number of convolution layers and their output channels for each of the components which are clearly represented in the Figure 8.18a. G 2 is also composed of similar three components given by: G

(F ) 2 , G (R) 2 and G (B) 2
(refer to ?? for additional details). encompasses local low-level information and global structured details together in the same tone-mapped output. Hence we get a much more structurally preserved and finely refined output as can be visualized in Figures 6.4 and 6.5.

Multi-scale Discriminator

Classifying the high resolution tone-mapped output as being real or fake is a big hurdle for the discriminator too. This can be easily solved by using larger receptive field (with a deeper network) or larger convolution kernels. However it would in turn require higher memory demand, which is already a constrain while training high resolution HDR images. We basically retain the same network architecture for the discriminator as used previously, but apply it on two different scales of input calling it a Multi-scale Discriminator. We hereafter refer these two discriminators as D 1 and D 2 .

Inputs to the discriminators D 1 and D 2 are the original outputs generated from G This feature aids the Generator to generate more globally consistent images. D 1 on the other hand, operating at a finer scale, aids in generating more precise finer details.

Tone-Mapping Objective Function

The ultimate goal of our generator G is to convert high resolution HDR inputs to tone mapped LDR images, while the discriminator D aims to distinguish real tone-mapped images from the ones synthesized by the generator. We train the entire generator-discriminator architecture in a fully supervised setting. For training, we give a set of pairs of corresponding images {(x i , y i )}, where x i is the luminance channel of HDR input image while y i is the luminance output of the corresponding tone-mapped LDR image. Note that to obtain the final color output we use the used the simple ratios as used in all the previous TMO [START_REF] Fattal | Gradient domain high dynamic range compression[END_REF][START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF].

Underneath we define the objective functions to train DeepTMO-R and DeepTMO-S as well as a modified variant for DeepTMO-HD.

Deep Tone Mapping Opeartor for HDR Imaging

DeepTMO-R and DeepTMO-S

The basic principle behind Conditional-GANs [START_REF] Mirza | Conditional generative adversarial nets[END_REF] is to model the conditional distribution of real tone-mapped images given an input HDR via the following objective:

L cGAN (G, D) = E x,y [log D(x, y)] + E x,z [log(1 -D(x, G(x, z)))] (6.1)
where G and D compete with each other; G trying to minimize this objective against its adversary D, which tries to maximize it, i.e.

G * = arg min G max D L cGAN (G, D).
In order for G to make predicted tone-mapped output being closer to real tone-mapped images, we additionally add a regularization term in the form of L1 distance between them which is given as:

L L1 (G) = E x,y,z [||y -G(x, z)|| 1 ] (6.2)
Thus the final objective function for both DeepTMO-R and DeepTMO-S can be defined as:

G * = arg min G max D L cGAN (G, D) + λL L1 (G) (6.3)
where λ is a regularization coefficient set to 100.

DeepTMO-HD

Since our DeepTMO-HD consists of two discriminator networks D 1 and D 2 , our objective for the DeepTMO-HD architecture is:

G * = arg min G max D 1 ,D2 k=1,2 L cGAN (G, D k ) (6.4)
In order to stabilize the Generator training and constrain it to yield natural image statistics at multiple scales, we append to the existing cGAN loss, an additional feature matching loss L F M (G, D k ) (similar to perceptual loss [START_REF] Dosovitskiy | Generating images with perceptual similarity metrics based on deep networks[END_REF][START_REF] Gatys | Image style transfer using convolutional neural networks[END_REF][START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF]),given by:

L F M (G, D k ) = E (x,y) T i=1 1 N i [||D (i) k (x, y) -D i k (x, G(x))|| 1 ], (6.5)
where D i k is the i th layer feature extractor of discriminator D k (from input to the i th layer of D k ), T is the total number of layers and N i denotes the number of elements in each layer. In short, we extract features from different layers of each of the discriminator and match these intermediate representations from real and generated images. Apart from this we also append a perceptual loss L P as is used in [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF] to our objective which has shown to 6.4. DeepTMO-R Architecture 107 further improve the results:

L L P (G) = N i=1 1 M i [||F (i) (y) -F (i) (G(x))|| 1 ] (6.6)
where F (i) denotes the i th layer with M i elements of the VGG network [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF].

Henceforth, our final objective function for a DeepTMO-HD can be written as:

G * = arg min G max D 1 ,D2 k=1,2 L cGAN (G, D) + β k=1,2 L F M (G, D k ) + B (6.7)
where B = γL L P (G). β and γ controls the importance of L F M and L L P with respect to L cGAN and both are set to 10.

DeepTMO-R Architecture

In this section we specify, detailed architectural details of DeepTMO-R including the generator and discriminator.

Generator Architecture

G (F ) has first a convolution layer consisting of 64 filters kernels (or output channels) each of size 7 × 7 applied with a stride of (1,1) and padding (0,0). Next two convolution layers with 128 and 256 filter kernels respectively and each with a size 3 × 3 and stride (2,2)

and padding (1,1). Each of these three layers are followed by batch norm with batch size = 1 (also called instance normalization [START_REF] Ulyanov | Instance normalization: The missing ingredient for fast stylization[END_REF]) and Relu [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF]. Following this, we have ) which is a set of 9 residual blocks (as shown in figure 6.2c, each of which contains two 3 × 3 convolutional layers, both with 256 filter kernels. Next, for G (B) we have two de-convolutional or transposed convolution layers with 128 and 64 filter kernels, each having a filter size of 3 × 3 and fraction strides of 1 2 . Both the layers have instance normalization and Relu added after the convolution. Finally we have another convolution layer of size 7 × 7 and stride 1 followed by a tanh activation function at the end. We additionally add some noise to the generator by putting dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] layers in each of the residual blocks.

G (R

Discriminator Architecture

Discriminator architecture consists of 4 convolution layers of sizes 4 × 4 and stride (2,2).

From first to the last, the number of filter kernels is 64, 128, 256 and 512 respectively. Each of the convolutional layer is appended with an instance normalization (except the first layer) and then leaky ReLU [START_REF] Maas | Rectifier nonlinearities improve neural network acoustic models[END_REF] activation function (with slope 0.2). Finally we apply a convolutional layer at the end to yield a 1 dimensional output which is followed by a sigmoid function.

Deep Tone Mapping Opeartor for HDR Imaging

Training and Implementation Details

The training paradigm for DeepTMO is inspired by the conventional GAN approach [START_REF] Goodfellow | Generative adversarial nets[END_REF],

where alternate stochastic gradient descent (SGD) steps are taken for discriminator (D) followed by the Generator (G). For both D and G, all the weights corresponding to convolution layers are initialized using zero mean Gaussian noise with a standard deviation of 0.02 while the biases are set to 0. Drawing from the efficacy of instance normalization over image generation tasks [START_REF] Ulyanov | Instance normalization: The missing ingredient for fast stylization[END_REF], we apply batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] using a batch size equal to 1. All the instance normalization layers are initialized using gaussian noise with mean 1 and 0.02 standard deviation.

All our training experiments are performed using Pytorch [START_REF] Paszke | Automatic differentiation in pytorch[END_REF] deep learning library with mini-batch stochastic gradient descent (SGD) where batch size is set to 4. We utilize an ADAM solver [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] whose initial learning rate is fixed at 2 × 10 -4 for first 100 epochs and then, allowed to decay linearly to 0 until the final epoch. Momentum term β 1 is fixed at 0.5 for all the epochs. To add noise z to different layer in the generator, we apply a dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] of 50 % for each convolution layer.

We also employ random jitters by first resizing the original image to 700 × 1100 and then randomly cropping to size 512 × 512. An additional mirroring is also performed before passing it through the network. All our networks are trained from scratch. Training is done on a 12 Gb NVIDIA Titan-X GPU for 1000 epochs and takes roughly 1-3 days depending upon the architecture. Inference is performed on test images of size 1024 × 2048 and takes less than a second. For all the other handcrafted TMOs, we used the MATLAB-based HDR Toolbox [START_REF] Banterle | Advanced High Dynamic Range Imaging: Theory and Practice[END_REF] and Luminance HDR software1 . For each tone mapping operator, we enable the default parametric setting as suggested by the respective authors.

Building HDR Dataset

In order to design a deep CNN based TMO, it is essential to obtain a large amount of dataset with a wide diversity of real-world scenes and cameras. We overcome this challenge by gathering several publicly available HDR datasets. For training the network, a total of 698 images are collected from various different sources [1,6,[START_REF] Database | Mpi hdr image database[END_REF][START_REF] Debevec | Recovering high dynamic range radiance maps from photographs[END_REF][START_REF] Ethyma | Ethyma database for high dynamic range images[END_REF][START_REF] Froehlich | Creating cinematic wide gamut hdr-video for the evaluation of tone mapping operators and hdr-displays[END_REF][START_REF] Krawczyk | Mpi hdr video database[END_REF][START_REF] Pfstools | Pfstools image database[END_REF][START_REF] Rad | Multidimensional image enhancement from a set of unregistered and differently exposed images[END_REF][START_REF] Rana | Evaluation of feature detection in HDR based imaging under changes in illumination conditions[END_REF][START_REF] Xiao | High dynamic range imaging of natural scenes[END_REF].

From the HDR video datasets from aforementioned sources, we select the frames manually so that no two chosen HDR images are similar. All these HDR images have been captured from diverse sources which is beneficial for our objective i.e., learning a general TMO. To further strengthen the training, we apply several data augmentation techniques such as random cropping and flipping. We consider 105 images from [START_REF] Fairchild | The hdr photographic survey[END_REF] for testing purposes.

Target Tone Mapped Images Different TMOs results in different output tone mapped images, and consequently, this raises an essential question about which is the best tone mapped image for the input HDR scene. Several subjective studies [START_REF] Banterle | Advanced High Dynamic Range Imaging: Theory and Practice[END_REF] built on different hypothesis attempt to answer this question on small database of maximum 15-20 scenes.

However, these solutions are impractical for large scale learning tasks. For training our Deep TMO models, a best tone mapped image is required as a target for each HDR scene. Due to unavailability of such GT images, we build the target set by using an objective Metric TMQI [START_REF] Yeganeh | Objective quality assessment of tone-mapped images[END_REF]. TMQI is a state-of-the-art objective metric which assess the quality of images on 1) structural fidelity which is a multi scale analysis of the signals, and 2) naturalness, which is derived using the natural image statistics. To find the GT for each training image, 13 classical TMO are considered [START_REF] Banterle | Advanced High Dynamic Range Imaging: Theory and Practice[END_REF] which includes [4,[START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF][START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF][START_REF] Fattal | Gradient domain high dynamic range compression[END_REF][START_REF] Larson | A visibility matching tone reproduction operator for high dynamic range scenes[END_REF][START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF][START_REF] Pattanaik | Adaptive gain control for high dynamic range image display[END_REF][START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF][START_REF] Schlick | An adaptive sampling technique for multidimensional integration by ray-tracing[END_REF][START_REF] Tumblin | Two methods for display of high contrast images[END_REF] and gamma and log mappings [START_REF] Banterle | Advanced High Dynamic Range Imaging: Theory and Practice[END_REF]. Then, based on the TMQI scores, the best tone mapped output for an individual scene is selected. The selection of these tone mappings is inspired from the following subjective evaluation studies [START_REF] Cadik | Evaluation of HDR tone mapping methods using essential perceptual attributes[END_REF][START_REF] Ledda | Evaluation of tone mapping operators using a high dynamic range display[END_REF] which highlight the distinctive characteristics of mapping functions, which we aim to inculcate in the learning of our Deep TMOs.

Results and Evaluation

In this section, we present an overview of our tone mapping models in terms of visual quality and quantitative performance. We first discuss the specific characteristics of three proposed models, including their adaptation to display, scene content, and sharpness in rendering high-resolution tone mapped outputs. Later we compare the performance of our best quality results with several handcrafted tone mapping methods [4,[START_REF] Chiu | Spatially nonuniform scaling functions for high contrast images[END_REF][START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF][START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF][START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF][START_REF] Pattanaik | Adaptive gain control for high dynamic range image display[END_REF][START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF][START_REF] Schlick | An adaptive sampling technique for multidimensional integration by ray-tracing[END_REF] on 105 images of test dataset [START_REF] Fairchild | The hdr photographic survey[END_REF]. Note that these scenes are different from training set and have not been seen by any of the architectures while training.

Comparison of the Three Architectures

We compare our three architectures in Figures 6.4 and 6.5, where in each column we showcase the full tone-mapped image for each of the architectures together with a cropped inset. In Figure 6.4, all the architectures give over all good quality LDR results with some subtle differences. From the cropped insets in second row, both DeepTMO-S and DeepTMO-HD retain the texture on background wall, while DeepTMO-R slightly blurs them out. Same goes for the bottommost window panels. This is quite evident, due to the lack of skip connections in its architecture (Figure 8.17), that help in transferring information from encoder to the decoder layers for better reproduction of output. While comparing between DeepTMO-HD and DeepTMO-S, we see DeepTMO-HD is able to better preserve very minute details such as frames in lower window in the inset. This is largely due to the presence of its two scale generator which caters to both finer details and 110 6. Deep Tone Mapping Opeartor for HDR Imaging coarser structures from the high-contrast linear HDR maps while generating tone-mapped image.

Next, in Figure 6.5 we show another interesting high-contrast natural scene where the three architecture provide similar contrast images but with some prominent visible effects.

From the cropped insets, we see that DeepTMO-R results in blurry effect on the textured bark of tree, similar to previous example. DeepTMO-S on the other hand, doesn't produce any blurriness, but instead, we notice pronounced repetitive checkerboard artifacts. Such artifacts have been recently discussed in deep-learning based image rendering problems [START_REF] Gao | Pixel deconvolutional networks[END_REF][START_REF] Odena | Deconvolution and checkerboard artifacts[END_REF] and are mainly caused due to no direct relationship among intermediate feature maps generated in de-convolutional layers. Nevertheless, it is still an open problem. DeepTMO-HD, on the other hand, gives us sharper and checkerboard free images while preserving the fine-details too. One possible reasoning behind this can be that the discriminator part working at the finer scale, effectively distinguishes a fake image through these artifacts, and thus forces the generator to subsequently generate checkerboard free images.

Overall, we observe that the result produced by our models are aesthetically pleasing images with a good balance of color contrast and details. Even though we selected GT images based on TMQI which is a color blind metric, our generator could develop an implicit understanding of contrast and saturation through the underlying data distribution and reproduce such impressive colors in the output images. Unlike past CNN based HDR techniques, which target to minimize Euclidean distance between the GT and output, our model adapts the cost based upon the target dataset and is effectively able to hallucinate high quality tone mapped results. Amongst the three models, our DeepTMO-HD successfully maps the linear-input HDR content in the most refined manner, preserving the minute details and the overall structural content. We henceforth use DeepTMO-HD for all the subsequent comparisons.

Comparison with TMOs

We begin the comparison of DeepTMO model against the Best Ranked Tone Mapper (BRTM) test scenes to assess the overall generalization capability. To obtain the best ranking tone mapped test scene, we follow a similar paradigm as provided in Section 6.6.

Later, one-to-one comparison are drawn between the existing TMOs and the proposed tone mapping model. To further demonstrate the generalization capability of DeepTMO models on all the 105 real world scenes, in Figure 8.19, we show a distribution plot of the number of scenes against the TMQI Scores. For completeness, we also add scores achieved by BRTMs. The curves clearly show that the generated tone mapped images for all models compete closely with the best available tone mapped images on the TMQI metrics with DeepTMO-H fairing the best among all. Hence, consolidating our main motivation of using this TMO.

Comparison with Best Ranked Tone Mapped Scenes

Comparison with TMOs

We first provide quantitative analysis in Table 8.2, to compare the performance of our proposed model with the existing approaches. For each method, the TMQI scores are averaged over 105 scenes of the test dataset. Final averaged results

show that all our three proposed tone mapping model outperform all the existing methods, hence proving their generalization capability over the rest. Moreover, while all three proposed models compete closely, our DeepTMO-HD model performs slightly worse than the DeepTMO-S. This is perhaps because the skip-connections provide marginal gains in some of the scenes. However, we believe that multi-scale DeepTMO-HD architecture is slightly more stable and produces visually pleasing results, and hence, we use it in the rest of the paper for comparisons.

Next, in Figure 8.20, we demonstrate qualitative comparisons of our model with four existing methods namely, Mantiuk [START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF], Reinhard [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF], Fattal [START_REF] Fattal | Gradient domain high dynamic range compression[END_REF] and Durand [START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF] over 5 real world scenes from the test dataset. For clarity of results, we choose only top four methods based on the scores in Table 8.2. From all the five scenes, we observe that our proposed tone mapping adapts to the content and preserves the contrast and fine details in the most convincing way in comparison to all the other local and perceptual TMOs.

While perceptual methods such as [START_REF] Mantiuk | A perceptual framework for contrast processing of high dynamic range images[END_REF] gives well balanced fine detailed information about the scenes, the resulting outputs are somewhat washed out in terms of overall perceptual brightness and visual appeal. Same goes for [START_REF] Durand | Fast bilateral filtering for the display of high-dynamic-range images[END_REF] for scene 1 and scene 2. The approach in [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] loose on preserving details in the overly saturated regions e.g., clouds in scene 1.

Although in these five scenes, approach in [START_REF] Fattal | Gradient domain high dynamic range compression[END_REF] provides better quality results as compared to other hand-crafted methods, in scene 1 and 5 it slightly lags behind our DeepTMO-HD

Limitations

Though our model successfully demonstrates generality in addressing wide variety of scenes, its expressive power is limited by the amount of available training data and quality of its corresponding ground truths. Due to unavailability of subjectively annotated 'best tone mapped images' for the HDR training dataset, we resort to an objective TMQI metrics to build the corresponding ground truth LDR. However, the metric itself is not as perfect as the human visual system. We illustrate this point in figure 6.9. The images ranked lower by TMQI metric in column 3 and 4 are somehow more interesting than their best-ranked counterpart in column 2. For e.g., in row 1, while the best ranked TMO outputs bursty effects near the lamp and doesn't effectively preserves the textual details in the book, TMOs in column 3 and 4 preserves better structural details and naturalness. Similarly in row2 for the best TMO, we can visualize some overall hazy effects as well as some noisy artifacts on the front board, which isn't the case in other two TMO's.

Such samples can eventually restricts the generation power of our model for darker scenes, specifically in high illumination regions such as lamps or bulbs. The discriminator no longer forces the generator to eradicate such bursty effects as the underlying ground truth samples, with which it is trained, are noisy. To overcome these challenges we provide

Conclusion

In this chapter, we have presented three different deep tone mapping architectures for a perceptual objective. Tailored in a generative-discriminative framework, the proposed models are trained to output realistic looking tone-mapped images, that duly encompass all the various distinctive properties of the available tone mapping operators. For completeness, we provide an extensive comparison among the three proposed frameworks highlighting the role that each design variation plays in their respective reproduced outputs.

Our deep tone mapping models also overcome the frequently addressed blurry or tiling effects in recent HDR related works [START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR-Video[END_REF][START_REF] Endo | Deep reverse tone mapping[END_REF], a problem of significant interest for several high-quality learning-based graphic rendering applications as highlighted in [START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR-Video[END_REF]. By simply learning an HDR-to-LDR specific cost function, the proposed models successfully preserve desired output characteristics such as underlying contrast, lighting and minute details present in the input HDR at the finest scale. Lastly we validate the versatility of our methodology through a detailed quantitative and qualitative comparisons with existing tone mapping techniques.

The work presented in this chapter has been submitted to SIGGRAPH'18.

Chapter 7

Conclusions

Summary

This thesis presents the quantitative and qualitative analysis of HDR imagery for specific task measures. More specifically, this thesis explores advantages of HDR content for specific tasks and timely addresses the corresponding limitations and challenges. In the following three major aspects, we highlight the contributions made in this thesis. Thirdly, to handle a large variety of HDR real-world images, we present three end-to-end deep learning based generic tone mapping designs which caters to desired task-specific characteristics. Our previously proposed TMOs essentially requires a specific filtering design which need to be differentiable. Since not all the TMOs can satisfy this criteria, DeepTMOs overcome these drawbacks and learn from a variety of filtering characteristics while following an easy differentiability. Furthermore, DeepTMOs can serve as a baseline

Conclusions

for future task-optimal HDR designs as they can be fine tuned for any specific task by simple cascading.

In the following, we expand the above mentioned aspects in correspondence to chapters of the thesis. We evaluate the performance of our proposed framework in terms of keypoint repeatability for state-of-the-art keypoint detectors. Our experimental results showcase the efficiency of our proposed learning-based adaptive TMO which yields higher keypoint stability when compared to existing perceptually-driven state-of-the-art

Investigation of HDR imagery for Dynamic Scenes

As evident from the results from Figures 4.13 The presented learned models theoretically should adapt in accordance to the invariance property of the feature extraction algorithms, as shown in Figure 5.20 for in planar rotation.

However, for out-of-the plane rotation problems that are specific for a mobile capturing platform, our models need to be practically re-calibrated. Note that no state-of-the-art local feature extraction algorithms is best under all transformations [START_REF] Yi | LIFT: Learned Invariant Feature Transform[END_REF][START_REF] Zhou | Evaluating local features for day-night matching[END_REF]. Therefore, instead of simply learning the regressor models with basic corner and descriptor based features, multiple characteristics from feature extraction algorithms, e.g. the ones evaluated in [START_REF] Zhou | Evaluating local features for day-night matching[END_REF], needs to be infused. Additionally, it requires a proper geometrically calibrated dataset which needs to be created.

Perception Vs Vision

In

HDR Analysis in Temporal domain

Deep Learning in HDR Imagery with small datasets

In comparison with millions of LDR annotated images, publicly available HDR dataset are very small, which in turn limits the exportability of HDR technology. Though an optimal solution would be to create a large HDR dataset with subjectively evaluated ground truths, this would be quite a cumbersome and tedious task. A work around can be reverse engineering the training dataset by reconstructing their corresponding HDR using recent deep learning based inverse tone mapping models [START_REF] Eilertsen | Hdr image reconstruction from a single exposure using deep cnns[END_REF][START_REF] Endo | Deep reverse tone mapping[END_REF].

Another alternative future work can be to rely on limited amount of samples, augmented with some noisy samples and then, utilizing a weakly supervised learning paradigm [START_REF] Lu | Learning from weak and noisy labels for semantic segmentation[END_REF].

For tasks such as HDR-to-LDR mapping, completely unsupervised learning is also possible without giving any input-output pairs [START_REF] Zhu | Unpaired image-to-image translation using cycleconsistent adversarial networks[END_REF]. The intuition is to allow the network to decide by itself which is the best possible tone-mapped output simply by independently modeling the underlying distribution of input HDR and output tone mapped images.

Chapter 8

Résumé de thèse Nos résultats expérimentaux démontrent l'efficacité de notre proposition de TMO • Apprentissage approfondi de l'imagerie HDR avec de petits ensembles de données.

En comparaison avec des millions d'images annotées LDR, les ensembles de données HDR accessibles au public sont très petits. Cela limite à son tour l'exportabilité de la technologie HDR. Bien qu'une solution optimale serait de créer un grand ensemble de données HDR avec une évaluation subjective, Si l'on se fie aux vérités du terrain, ce serait une tâche fastidieuse. Une solution alternative peut être l'ingénierie inverse de l'ensemble de données de formation en reconstruisant leur HDR correspondant à l'aide du HDR des modèles récents de TMO inverses basés sur l'apprentissage profond [START_REF] Eilertsen | Hdr image reconstruction from a single exposure using deep cnns[END_REF][START_REF] Endo | Deep reverse tone mapping[END_REF]. Une autre alternative pour les travaux futurs peut être de s'appuyer sur la quantité limitée d'échantillons, augmentés de avec des échantillons bruités et ensuite un paradigme d'apprentissage faiblement supervisé [START_REF] Lu | Learning from weak and noisy labels for semantic segmentation[END_REF]. Pour des tâches telles que le mappage HDR vers LDR, un apprentissage totalement non supervisé est également possible, sans donner de paires d'entrées-sorties [START_REF] Zhu | Unpaired image-to-image translation using cycleconsistent adversarial networks[END_REF]. L'intuition est de laisser au réseau le soin de décider, par lui-même qui est la meilleure sortie tonemapping possible simplement en modélisant de manière indépendante la distribution sous-jacente des images HDR d'entrée et des tonalités de sortie.

1 Introduction 1 . 1

 111 Context and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Structure of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Background and State of the Art 2.1 HDR Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3. 1

 1 Harris corner detection in a lighting setup from Project Room (Row-1) and Light-Room (Row-2) datasets with local, global TMOs and best exposures LDR. . . . . . . . . . . 3.2 Dynamic Range Vs Image-Key plots for (a) Light-Room(LR). (b) Project-Room. (c) Lighting 2D. (d) Lighting 3D [11]. . . . . . . . . . . . . . . . . . . . . . . . . . .

3 . 6 4 . 2 4 . 3 4 . 7

 36424347 Average gain recorded by different formats over the LDR. . . . . . . . . . . . . . . . 3.7 Example images from the datasets. . . . . . . . . . . . . . . . . . . . . . . . 3.8 Mean average precision (mAP) and mean repeatability rate (mRR) over the four considered datasets and feature schemes. mAP and mRR are computed on 56 image pairs, for the Project Room dataset, and over 42 image pairs for Light Room, 2D and 3D Lighting datasets. . . . . . . . . . . . . . . . . 3.9 An example of image matching for two TMOs. The true positive and false positive matches are shown with green and red lines respectively. The TM in (a) achieves a higher repeatability (24 %) than that in (b); however, most of the matches in (a) are false positives, thus the AP for (b) is higher than in (a) (95 % vs. 87 %, respectively). . . . . . . . . . . . . . . . . . . . . . . 4.1 Reflectance images R g and R b from original image I using the Gaussian and Bilateral luminance maps L g and L b respectively. . . . . . . . . . . . . . . . Parameters vs Correlation Coefficient (CC) for Project Room dataset. (a) σ vs CC for Gaussian tone mapping (GTM) model. (b) σ r and σ s contours for Bilateral tone mapping (BTM) model with color magnitudes showing average CC scores. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Parameters vs Repeatability Rate (RR) for Project Room dataset. (a) σ vs RR for both SURF and Harris detector for repeatability rate Gaussian TMO (RRGTM). (b) σ r and σ s contours for Harris detector and (c) for SURF detector for repeatability rate Bilateral TMO (RRBTM) with color magnitudes showing RR scores. . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Row 1. (a) and (b) Average repeatability score and standard deviation for the both correlation and response based optimized approaches using Harris and SURF detector respectively. Row 2. (c) and (d) Average repeatability score and standard deviation for the reflectance models (GTM and BTM) and other commonly used TMs on Project Room and Light Room dataset 4.5 Scatter plots. (a) correlation coefficients of reflectance maps CC(R i , R j ) vs corresponding repeatability rate RR(R i , R j ), (b) correlation coefficients of response maps CC(Resp m , Resp n ) vs corresponding repeatability rate RR(R m , R n ) for HDR log-encoded Project room dataset. . . . . . . . . . . 4.6 An example showing (a) image and its corresponding (b) Harris response map. Learning based DetTMO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8 Generation of training set. The samples images undergoing different lighting variations shown in (a) are used to generate the θ 1 and θ 2 modulation maps in (b) and (c) respectively, using the detection similarity maximization model. 4.9 Training an SVR. The sample pixel (red) with s × s neighborhood (blue) is chosen to extract the features maps (F 1 , F 2 , F 3 ) using response scores, gradients and intensity patterns respectively. . . . . . . . . . . . . . . . . . 4.10 Sample images from HDR dataset. The HDR Dataset is composed of 8 scene from different indoor/outdoor locations. . . . . . . . . . . . . . . . . . . . . 4.11 Quantitative Results I : Repeatability Rates (RR) computed using DetTMO, BTMO(opt) and BTMO for each test scene using Harris keypoint detector. Note that while testing DetTMO for a particular scene we assured that the training for DetTMO is done on all other scenes. . . . . . . . . . . . . . . . 4.12 Quantitative Results II : Average Repeatability Rates (AvgRR) computed on different TMOs using various keypoint detection schemes. The average is calculated over all test scenes. . . . . . . . . . . . . . . . . . . . . . . . . . . 4.13 Repeated Keypoints. Row I: 2 HDR images from Invalides scene taken at different day-time. HDR images are displayed after log scaling[27]. Row II: the repeated keypoints using our proposed DetTMO (66 repeated keypoints out of strongest 200 keypoints). Row III: the repeated keypoints using Reinhard TMO (7 repeated keypoints out of strongest 200 keypoints). Row IV: the repeated keypoints using MantiukTMO (5 repeated keypoints out of strongest 200 keypoints). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 DesTMO. The architecture of our proposed TMO. . . . . . . . . . . . . . . 5.2 Training Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Scenes from HDR luminance dataset. The dataset is composed of 8 scene from different indoor/outdoor locations. . . . . . . . . . . . . . . . . . . . . 5.4 Matching Score computed using DesTMO, BTMO and LDR for each test scene using SURF descriptor. . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Average Matching Scores computed on different TMOs using SURF, SIFT, FREAK, BRISK descriptor extraction schemes. The average is calculated over all test scenes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Mean Average Precision (mAP) rates computed on different TMOs using SURF, SIFT, FREAK, BRISK descriptor extraction schemes. The average is calculated over all test scenes. . . . . . . . . . . . . . . . . . . . . . . . .

5. 13

 13 Keypoint Detection III. The head to head comparison between (a) OpTMO vs BTMO and, (b) OpTMO vs DetTMO. Each point represent an image pair with different lighting conditions from the HDR dataset. The points represented using o depict the Harris corner detector and 2 represents the SURF blob detector. . . . . 5.14 Keypoint Detection IV. Harris corner keypoints on the DetTMO and proposed

  (Match & Locate) Row I: Pair of HDR images from Louvre, ProjectRoom and Notredame scenes, with one reference and other being a selected region undergone lighting change and rotation. Row II: the feature matching using our proposed OpTMO. Row III: using Reinhard TMO. . . . . . . . . . . . . . . . . . . . . . 5.21 (Match & Locate) Final patch localization results shown by overlaying the matched area for each scene using OpTMO and Reinhard TMO . . . . . . . . . . 5.22 Computation time in sec (log scale). The time is computed by running all TMOs for an image size (512 × 512) on a Intel Xeon CPU 4 cores processor, 16 Gb RAM windows 7 machine. . . . . . . . . . . . . . . . . . . . . . . . . 6.1 We illustrate here the training pipeline of our Deep Tone Mapping Operator (Deep TMO). Training dataset consists of input HDRs and their corresponding best-TMQI ranked tone mapped outputs. Both the discriminator and

6. 7 8 . 4

 784 Quantitative performance comparison of DeepTMO-R, DeepTMO-S and DeepTMO-HD with the BestTMOs. . . . . . . . . . . . . . . . . . . . . . . 6.8 Qualitative Results. Five sample scenes from Fairchild HDR dataset, taken with different natural lighting variations. . . . . . . . . . . . . . . . . . . . . 6.9 Sample cases where top scoring TMQI's TMO shows not-so-visually desirable outputs. In column I, we have tone mapped outputs from DeepTMO-H, in column II for BRTM , while in column III and column IV we provide results for two other top ranking TMO's. . . . . . . . . . . . . . . . . . . . . . . . . 8.1 En (a), nous montrons un exemple de [119], où l'appariement des points saillants (communs aux deux images) est représenté par des lignes bleues entre deux images de la même scène prises à des heures différentes de la journée. En (b), nous montrons le taux de répétabilité de la mesure d'efficacité sur un grand ensemble de données d'images LDR jour/nuit en utilisant les techniques de pointe. La crête et les creux dans les courbes illustrent que l'image capturée pendant le jour correspond bien avec seulement d'autres images de jour et non avec celles capturées dans l'obscurité. . . . . . . . . . 8.2 RR moyen enregistré par différents formats sur le LDR. . . . . . . . . . . . 8.3 Average RR et standard deviation pour les approches optimisées basées sur la corrélation et la réponse en utilisant respectivement un détecteur Harris et un détecteur SURF. Rangée 2. (c) et (d) Score moyen de répétabilité et écart-type pour les modèles de réflectance (GTM et BTM) et autres TMs couramment utilisés pour Project Room et Light Room dataset . . . . . . Learning based DetTMO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Average Repeatability Rates (AvgRR) calculée sur différents TMOs en utilisant divers schémas de détection de points clés. La moyenne est calculée sur toutes les scènes de test. . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Exemples d'images de HDR dataset. HDR Dataset est composé de 8 scènes de différents endroits intérieurs et extérieurs. . . . . . . . . . . . . . . . . . 8.7 Repeated Keypoints. Row I: 2 images HDR de la scène Invalides prises à différentes heures du jour. Les images HDR sont affichées après la mise à l'échelle du journal [27]. Row II: les points clés répétés en utilisant notre DetTMO proposé (66 points clés répétés sur les 200 points clés les plus forts). Row III: les points clés répétés à l'aide de Reinhard TMO (7 points clés répétés sur les 200 points clés les plus forts). Row IV: les points clés répétés à l'aide de MantiukTMO (5 points clés répétés sur les 200 points clés les plus forts). . . .

2

  images HDR de la scène Invalides sont affichées après la mise à l'échelle du journal. Les correspondances correctes et incorrectes sont indiquées par des lignes jaunes et rouges, respectivement. Les lignes vertes représentent le cas particulier d'inadéquation due à une structure répétitive. Row II : l'appariement des caractéristiques à l'aide de notre proposition OpTMO (21 correspondances correctes et 3 incorrectes). Rangée III : en utilisant DetTMO (13 correspondances correctes et 6 incorrectes). Row IV : utiliser DesTMO (11 correspondances correctes et 3 incorrectes). Ligne V en utilisant Reinhard TMO (3 correspondances correctes et 11 incorrectes). Row VI : utiliser MantiukTMO (3 correspondances correctes et 4 incorrectes). . . . . 143 8.15 Pipeline de formation de Deep Tone Mapping Operator (Deep TMO) basé sur les GANs. L'ensemble de données apprentisage se compose des HDR d'entrée et de leurs sorties correspondantes les mieux classées en fonction de l'TMQI et de l'indice de qualité du ton. Le discriminateur et le générateur sont formés alternativement, d'abord une étape de regression du discriminateur puis du générateur. Tandis que le discriminateur est formé pour distinguer les paires d'images réelles des fausses, le générateur apprend à tromper le discriminateur en produisant des images en tons synthétiques. Ce faisant, le générateur modélise efficacement la distribution sous-jacente des images réelles de la tonalité de vérité au sol, ce qui donne des résultats de haute qualité une fois l'entraînement terminé. . . . . . . . . . . . . . . . . . . . . 144 8.16 Nous proposons un TMO basé sur l'apprentissage profond (appelé DeepTMO) qui donne des résultats de haute qualité subjective sur un large éventail d'images HDR à valeur linéaire. Notre variante proposée de cGANs est une architecture multi-échelle qui donne des résultats d'apparence naturelle et sans artefacts en haute résolution. Alors que les TMOs classiques sont sensibles à l'accord des paramètres pour une sortie souhaitée, notre modèle apprend à traiter efficacement une plus large gamme de contenus HDR en modélisant la distribution sous-jacente de toutes les sorties de cartographie des tons cibles disponibles. En concurrence avec les meilleurs résultats des cartes tonales subjectives de qualité supérieure sur 3 types de scènes différentes : claires, nuageuses et sombres, nous montrons surtout la polyvalence de notre méthode qui préserve efficacement les textures, les détails des structures et le contraste. Les résultats détaillés sont présentés aux sections 6 et 7 de chapitre 6. Enfin, notre modèle DeepTMO est assez rapide et prend en moyenne 0,02 seconde pour le tone mapping d'une image HDR de taille 1024 × 2048. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.17 Nous présentons l'architecture détaillée du discriminateur et du générateur de DeepTMO-R et DeepTMO-S. La seule différence pour DeepTMO-S est l'ajout de connexions sautées dans le cas d'un générateur. Le générateur est encadré comme une architecture codeur-décodeur, où l'image HDR d'entrée est d'abord transmise à un codeur, qui la sous-échantillonne ensuite en une représentation compacte. Cette représentation est ensuite transmise par le décodeur qui l'échantillonne à la taille du HDR d'entrée. Alors que le codeur se compose du composant frontal Convolution G (F ) et des cinq premiers blocs résiduels G (R) , le décodeur se compose des quatre blocs résiduels suivants G (R) et du composant déconvolution G (B) . La discriminateur se compose d'une architecture patchGAN qui est appliquée à chaque patch sur les paires HDR d'entrée concaténées et les paires LDR de tonalités mappées. La prédiction finale est une moyenne de tous les patchs sur l'image. . . . . . 8.19 Nous comparons les performances quantitatives de DeepTMO-R, DeepTMO-S et DeepTMO-HD avec celles des BestTMO. . . . . . . . . . . . . . . . . . 8.20 Resultat Qualitatif de DeepTMO-HD. . . . . . . . . . . . . . . . . . . . . . .

3. 1

 1 Local(L) and Global(G) TMOs. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Different image modalities for feature extraction. . . . . . . . . . . . . . . . . . . . 3.3 Mean Average Precision (mAP %) scores for the 10 considered representations using 4 feature extraction schemes. Scores are averaged over 4 lighting change datasets. Highest mAP score for each scheme is shown in bold. Best Avg/Formats and Avg/Schemes scores are double underlined. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 1

Figure 1 . 1 -

 11 Figure 1.1 -(a) shows an example case from[119], where matching of salient points (common in both) is

3 .

 3 We discuss the sub-optimality of existing TMOs and what is needed to design a keypoint-optimized TMO. To that end, we draw comparison between the optimization of existing TMO parameters with respect to: a) task-specific measure i.e. Repeatability Rate RR and b) statistical correlation coefficient CC between pairs of tone-mapped images of the same scene with lighting variations. CC measures the statistical similarity between a pair of tone-mapped images. This is presented in the following paper: A. Rana and G. Valenzise and F. Dufaux, "Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes", 2016 IEEE Workshop on Multimedia Signal Processing (MMSP 2016), Montréal, Canada, 2016. 4. We design a learning-based adaptive tone mapping framework which aims at enhancing keypoint stability by design a pixel-wise adaptive TMO. The regression based model is driven by Support Vector Regression (SVR) using keypoint characteristics. Additionally, we propose a simple detection-similarity-maximization model to generate appropriate training samples. We present this contribution in the following paper: A. Rana and G. Valenzise and F. Dufaux, "Learning-based Adaptive Tone Mapping for Keypoint Detection", The International Conference on Visual Communications and Image Processing(ICME), Hong Kong, China, 2017.

5 . 6 .

 56 We propose to optimally tone-map a high dynamic range (HDR) content for invariant descriptor extraction under drastic illumination variations. We employ a learned model to predict optimal modulation maps that help to locally alter the intrinsic characteristics (such as shape, size) of the tone mapping function. The detail of this work is available in the following paper: A. Rana and G. Valenzise and F. Dufaux, "Learning-based Tone Mapping Operator for Image Matching", IEEE International Conference on Image Processing (ICIP'2017), We address the sub-optimality of TMOs by collectively addressing both stages of keypoint detection and descriptor extraction in the feature matching framework. We develop a two-step framework, consisting of: a) a luminance-invariant guidance model based upon a Support Vector Regressor (SVR) to optimally adapt the tone mapping function for image matching; and b) an energy maximization model to generate appropriate training samples considering each independent proxy function. The article describing this work is currently under revision: A. Rana and G. Valenzise and F. Dufaux, "Learning-based tone mapping operator for efficient image matching", IEEE Transaction of Multimedia(TMM), 2017 accepted 7. We propose a deep learning based Tone mapping operator which predicts high quality tone mapped outputs over a wide spectrum of linear HDR images. The proposed model is designed for perceptual objectives. However, this is the first end-to-end learnable TMO which can be fine-tuned for any computer vision specific task such as image matching. The following article describing this work is submitted: A. Rana*, P. Singh*, G. Valenzise, F. Dufaux and N. Komodakis. "Deep Tone Mapping Operator for High Dynamic Range Imagery", ACM SIGGRAPH, 2018 submitted.

••

  Chapter 2 discusses the background of HDR imagery using conventional acquisitiongeneration-display approaches and brief history of HDR imagery for computer vision applications. The chapter provides details about benchmark studies on fundamental visual feature extraction algorithms. The quantitative evaluation metrics for accessing the feature extraction and HDR imaging, used in following chapters, is revisited in detail. Technical contributions of the thesis begin in Chapter 3, by addressing the performance evaluation of the different HDR formats for two stage of local feature extraction, which is the keypoint detection and description. We propose a geometrically calibrated HDR luminance change dataset. In this chapter, we investigate how different HDR formats can impact the keypoint detection and descriptor performance and if there is any format which yields stable keypoints across these scenes. Specifically, we compare 11 image formats and test the keypoint detection and full feature extraction efficiency on them using two different detectors. • Based on preceding observations, in Chapter 8 we propose a learning based adaptive tone mapping framework for HDR images which results in stable and efficient keypoint 1.3. Structure of the thesis detection. In this chapter, we initially present an experimental study showing whatit takes to optimize a tone mapping function for a metric task such as keypoint detection. Later, we present a regression based guidance model to predict the desired pixel-wise modulation maps by using the linear HDR content from scenes captured with varying lighting conditions. After addressing the keypoint detection, in Chapter 5, we move to full feature extraction. To this end, the chapter first proposes a descriptor-optimal TMO design which solely aims at the extraction of invariant (as much as possible) descriptors from high-contrast areas of the scenes. Later, an optimal TMO OpTMO for full feature extraction chain (including both detectors and descriptors) is introduced which simultaneously enhances the detection rates and matching of features by inculcating proxy cost functions; to fuse the relevant information from independent design objectives.• In Chapter 6, we design the first end-to-end deep learning based TMO. Being trained with a perceptual objective in its primal stage, the GAN based network defines a universally applicable tone mapping function which yields most natural images. The proposed model can be simple fined tuned with any desired objective such as image matching and eradicates the need of designing any proxy cost functions. Instead, it provides a baseline architecture to explore HDR imagery for several other domain specific analysis tasks such as medical image analysis or high resolution remote sensing tasks.

Figure 2 . 1 -

 21 Figure 2.1 -Taken from [24]. Multiple exposures of a Church scene along with final Radience Map.

Figure 3 . 1 -

 31 Figure 3.1 -Harris corner detection in a lighting setup from Project Room (Row-1) and Light-Room (Row-2) datasets with local, global TMOs and best exposures LDR.

Figure 3 . 2 -

 32 Figure 3.2 -Dynamic Range Vs Image-Key plots for (a) Light-Room(LR). (b) Project-Room. (c) Lighting 2D. (d) Lighting 3D [11].

  luminance levels. Notice that PU encoding needs photometrically calibrated HDR pixels as input. Both Log and PU values are rescaled in [0, 1]. In total, 13 different image formats are thus considered for each lighting condition. We stress the difference between HDR encoded values and GTM pixel values: the former are the result of a simple transfer function and encoded using floating point values; the latter, instead, are the result of a content-dependent operation, and are encoded on 8-bit, integer precision.

Figure 3 . 3 -Figure 3 . 4 -Figure 3 . 5 -Figure 3 . 5 .

 33343535 Figure 3.3 -Relative gains by best LTM, GTM (abbreviated using Table 3.1), Linear, Log and Pu HDR encodings with respect to LDR for different test datasets (scenes indicated by progressive numbers on x-axis). The dotted line shows the absolute R-scores of LDR.

Figure 3 . 6 -

 36 Figure 3.6 -Average gain recorded by different formats over the LDR.

3. 4

 4 Evaluation of full Features Extraction Pipelines in HDR imagery 3.4.1 Considered LDR/HDR modalities We consider a total of 10 different image modalities (listed in Table 3.2) including the standard 8-bit LDR, 2 floating point HDR representations (HDRlog and HDRlin) and 7 different 8-bit TMO HDR representations. These consist of 2 global and 5 local TMOs. In this chapter, we have considered a subset of the TMOs introduced in Table

Figure 3 . 7 -

 37 Figure 3.7 -Example images from the datasets.

3 .

 3 Local Feature extraction in HDR Imagery under Drastic Lighting Variations

Figure 3 . 9 -

 39 Figure 3.9 -An example of image matching for two TMOs. The true positive and false positive matches are shown with green and red lines respectively. The TM in (a) achieves a higher repeatability (24 %) than that in (b); however, most of the matches in (a) are false positives, thus the AP for (b) is higher than in (a) (95 % vs. 87 %, respectively).

50

 50

Figure 3 .

 3 [START_REF] Boschetti | An optimal video-surveillance approach for HDR videos tone mapping[END_REF] shows an example of image matching for the Project room dataset, using RN and FA tone mappings and BRISK features. It is clear that, although the number of matches is lower in RN, they are "better quality", in the sense that most of them are true positives. Conversely, in FA, although the basis of possible matches is larger, most matches are indeed false, which reduces the average precision as reported by the mAP scores in Figure3.8.

3. 5 . Conclusion 51 3. 5 Conclusion

 5515 In this chapter, we presented a comprehensive evaluation of different HDR and LDR based modalities for visual feature detection and matching, under changes in illumination conditions. The analysis based on the RR and mAP scores on different scenes confirms the potential of HDR techniques over single LDR exposures. For both detection and matching, our results confirms that the linear HDR values are inappropriate to be used for visual recognition tasks. Furthermore, we observe that local TMO's producing very appealing results in terms of rendering quality are not necessarily the best option for image analysis.More interestingly, we have also observed that local TMOs with very high repeatability rate for feature detection are not necessarily the best option when the full feature extraction pipeline is considered. Although we measured a consistent gain in the average repeatability scores when using direct HDR pixel values over tone-mapping, it does not hold true for the test pairs individually. Hence, it is difficult to comment what is better before extracting the features, a) to encode the HDR pixel approximatively linear to perception, or b) directly tone map.

  Figure 4.1 -Reflectance images R g and R b from original image I using the Gaussian and Bilateral luminance maps L g and L b respectively.

σ

  = α • max(m, n). When targeting visual perception, the parameter σ is tuned so as to reduce visual artifacts like halos observed around detected edges. This model with a ingle parameter is simple and computationally very fast. b). Bilateral tone mapping (BTM) model is a a precise, non-linear and edge preserving filter where R b is computed as R b = I/L b . The luminance estimation L b (x) is given as:

4. 2 .

 2 Optimizing a TMO for Keypoint detection 57

Figure 4 . 2 -

 42 Figure 4.2 -Parameters vs Correlation Coefficient (CC) for Project Room dataset. (a) σ vs CC for Gaussian tone mapping (GTM) model. (b) σ r and σ s contours for Bilateral tone mapping (BTM) model with color magnitudes showing average CC scores.

Figure 4 . 3 -

 43 Figure 4.3 -Parameters vs Repeatability Rate (RR) for Project Room dataset. (a) σ vs RR for both SURF and Harris detector for repeatability rate Gaussian TMO (RRGTM). (b) σ r and σ s contours for Harris detector and (c) for SURF detector for repeatability rate Bilateral TMO (RRBTM) with color magnitudes showing RR scores.

58

 58

Figure 4 .

 4 Figure 4.3 (a) for the Gaussian model, and in Figure 4.3 (b) and (c) for the Bilateral model.

  Figure 4.4(a),(b). Additionally, we compare 5 different local and global high performing TMOs [21, 27, 29, 67, 90] with our RRGTM and RRBTM tone mapping models as shown in Figure 4.4(c),(d).

Figure 4 . 4 -

 44 Figure 4.4 -Row 1. (a) and (b) Average repeatability score and standard deviation for the both correlation and response based optimized approaches using Harris and SURF detector respectively. Row 2. (c) and (d) Average repeatability score and standard deviation for the reflectance models (GTM and BTM) and other commonly used TMs on Project Room and Light Room dataset

Figure 4 . 5 -

 45 Figure 4.5 -Scatter plots. (a) correlation coefficients of reflectance maps CC(R i , R j ) vs corresponding repeatability rate RR(R i , R j ), (b) correlation coefficients of response maps CC(Resp m , Resp n ) vs corresponding repeatability rate RR(R m , R n ) for HDR logencoded Project room dataset.

60 4 .

 4 Tone Mapping Operator for Efficient Keypoint Detection over all the possible images pairs for each scene (Light Room and Project Room), i.e., each lighting condition of a scene is, in turn, the reference and the other conditions of that scene are the test images. Thereafter, we compute the average of RR over all such possible image pairs (52 pairs for Project Room and 42 pairs for Light Room) for a particular scene. Similar

  Figure 4.4(c),(d), we show that optimizing the traditional models in Eqs (4.1) and (4.2), RRGTM and RRBTM respectively, lead to large performance gains in terms of RR when compared to existing local and global TMOs. It shows the necessity to optimize the tone mapping operators with respect to detection tasks.

4. 3 .

 3 Learning a TMO for Efficient Keypoint Detection 61 it does not provide any keypoint-detection-optimized TMO model. Therefore, the problem of designing an optimal TMO for the keypoint detection task remains open.

4 .

 4 Tone Mapping Operator for Efficient Keypoint Detection

Figure 4 . 7 -

 47 Figure 4.7 -Learning based DetTMO.

Figure 4 . 8 -

 48 Figure 4.8 -Generation of training set. The samples images undergoing different lighting variations shown in (a) are used to generate the θ 1 and θ 2 modulation maps in (b) and (c) respectively, using the detection similarity maximization model.

.10) 4 . 3 . 4

 434 Support Vector Regressor Training for DetTMO An illustration for SVR training is shown in Fig 4.9. Let's assume that a scene with multiple images captured under different lighting variations is given for training. Further,

Figure 4 . 9 -Figure 4 . 10 -

 49410 Figure 4.9 -Training an SVR. The sample pixel (red) with s × s neighborhood (blue) is chosen to extract the features maps (F 1 , F 2 , F 3 ) using response scores, gradients and intensity patterns respectively.

66 4 .

 4 Tone Mapping Operator for Efficient Keypoint Detection assume that the optimal modulation maps for the same scene are also given as described in Section 4.3.3.

  F 3 (x) = I(x) -I(c), where c is the pixel at center location. These individual features are normalized and concatenated to form the final feature vector {F 1 , F 2 , F 3 } of dimension 3s 2 representing a training sample.

For

  each test scene, we build the training set with 10,000 samples and use it to train and validate the SVR model. This training set is drawn from other scenes excluding the corresponding test scene. For instance, to test the Project Room scene, we build the training set by randomly selecting the samples from all other 7 scenes. For each training sample, we compute feature on a small patch size of 5 × 5 while following the feature extraction procedure from Section 4.3.4. Higher patch-size is not advisable as pixel correlation diminishes with increasing distance. Conversely smaller patch-size may extract insufficient information.

Fig. 4 .Figure 4 . 11 -

 4411 Figure 4.11 -Quantitative Results I : Repeatability Rates (RR) computed using DetTMO, BTMO(opt) and BTMO for each test scene using Harris keypoint detector. Note that while testing DetTMO for a particular scene we assured that the training for DetTMO is done on all other scenes.

4. 3 .Figure 4 . 12 -

 3412 Figure 4.12 -Quantitative Results II : Average Repeatability Rates (AvgRR) computed on different TMOs using various keypoint detection schemes. The average is calculated over all test scenes.

Figure 4 . 13 -

 413 Figure 4.13 -Repeated Keypoints. Row I: 2 HDR images from Invalides scene taken at different daytime. HDR images are displayed after log scaling[27]. Row II: the repeated keypoints using our proposed DetTMO (66 repeated keypoints out of strongest 200 keypoints). Row III: the repeated keypoints using Reinhard TMO (7 repeated keypoints out of strongest 200 keypoints). Row IV: the repeated keypoints using MantiukTMO (5 repeated keypoints out of strongest 200 keypoints).

4 .

 4 Tone Mapping Operator for Efficient Keypoint Detection4.4 ConclusionsIn this chapter, we first investigate the impact of two factors for the optimization of tone mappings models. Build on its observations, later, we propose a new learning based adaptive tone mapping framework aiming at enhancing the keypoint detection performance under drastic lighting change scenarios. To that end, we train a Support Vector Regressor using local characteristic features to learn a model which spatially modulates the proposed pixelwise adaptive TMO. Further, we introduce a simple and effective method for generating the training set to learn the SVR for the given problem. We evaluate our model on our proposed HDR benchmark dataset of indoor/outdoor scenes. Our model significantly outperforms state-of-the-art TMOs on the HDR dataset and also achieve state-of-the-art results across different keypoint detection algorithms.Our current tone mapping model is designed for keypoint detection only. In the following, we plan to design a descriptor based tone mapping framework optimal for invariant descriptor extraction.The work presented in this chapter has resulted in the following publications: 1. A. Rana and G. Valenzise and F. Dufaux, "Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes", 2016 IEEE Workshop on Multimedia Signal Processing (MMSP 2016), Montréal, Canada, 2016. 2. A. Rana and G. Valenzise and F. Dufaux, "Learning-based Adaptive Tone Mapping for Keypoint Detection", The International Conference on Visual Communications and Image Processing (ICME), Hong Kong, China, 2017.

Fig. 5 .

 5 Fig. 5.1 outlines the framework of our proposed algorithm. It primarily consists of a tone mapping function ϕ which maps the linear-valued HDR content of an image I to an output LDR I . Similar to Chapter 4, it is expressed as

Figure 5 . 1 -

 51 Figure 5.1 -DesTMO. The architecture of our proposed TMO.

Figure 5 . 2 -

 52 Figure 5.2 -Training Pipeline

  ) where x is the center location of the cell in the restricted square patch p of size 16 × 16. The independent variable Θ represents the gradient orientation ranging from [0, 2π]. Moreover, G represents the Gaussian kernel with standard deviation σ and an angular dispersion parameter δ.

Figure 5 . 3 -

 53 Figure 5.3 -Scenes from HDR luminance dataset. The dataset is composed of 8 scene from different indoor/outdoor locations.

For

  each test scene, we build the training set with 5000 training samples and use it to train and validate the SVR model. Given a test scene from our dataset Fig. 5.3, the training set is drawn from the other 7 scenes. For each training sample, we compute the SIFT feature on a patch size of 16 × 16.

Figure 5 . 4 -Figure 5 . 5 -

 5455 Figure 5.4 -Matching Score computed using DesTMO, BTMO and LDR for each test scene using SURF descriptor.

5 .Figure 5 . 6 -

 556 Figure 5.6 -Mean Average Precision (mAP) rates computed on different TMOs using SURF, SIFT, FREAK, BRISK descriptor extraction schemes. The average is calculated over all test scenes.

Figure 5 . 7 -

 57 Figure 5.7 -Day/Night matching using SURF. Row I: 2 HDR images from Invalides scene are displayed after log scaling[START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF]. Correct and incorrect matches are shown with yellow and red lines respectively. Green lines represent the special case of mismatch due to repetitive structure. Row II: the feature matching using our proposed DesTMO (11 correct and 3 incorrect matches). Row III: using Reinhard TMO (3 correct and 11 incorrect matches). Row IV: using MantiukTMO (4 incorrect and 3 correct matches).

Figure 5 . 8 -

 58 Figure 5.8 -Optimal Tone Mapping Design. The tone mapping function is modulated by the SVRbased guidance model, which predicts optimal parameter maps using the characteristic features.

Fig. 8 .

 8 Fig. 8.4 outlines the general framework of our proposed optimal TMO for image matching. It primarily consists of a tone mapping function ϕ which maps the linear-valued HDR content of an input image I to an output LDR I . Our choice of the tone mapping function is same as described in 5.2.2. Secondly, the framework consists of a guidance model where a learned SVR predicts an optimal parameter map θ based on local HDR content characteristics. In the following subsections, we discuss the design of our proposed framework and how to generate training data for the SVR.

Figure 5 . 9 -

 59 Figure 5.9 -Generation of Training Set. Ground-truth parameter maps are generated by minimizing the total energy determined from a set of images of the same scene, undergoing lighting variations, using the procedure in Section 5.4.2.

Figure 5 . 10 -Figure 5 . 11 -

 510511 Figure 5.10 -Repeatability Rates (RR) computed for OpTMO using a corner (Harris) and a blob (SURF) keypoint detector.

[ 2 - 5 ,

 25 2 15 ] and [2 -10 , 2 5 ], respectively. To train and validate the SVR model, we build the training set with 5000 sample feature set for each test scene. This training set is drawn from other scenes excluding the corresponding test scene. For instance, to test the Project Room scene, we build the training set by randomly selecting samples from all other 7 scenes. For each training sample, we randomly select a pixel location and compute characteristic features around the selected location, while following the feature extraction procedure described in Section 5.4.3. A window of 16 × 16 is selected around the pixel location for computing the gradient based SIFT pattern part of the feature f i . Whereas a 5 × 5 grid based detector response is used for the second part of the f i .

Finally, we observe from Fig. 5 .Figure 5

 55 Figure 5.12 -Keypoint Detection II: Average Repeatability Rates (RR) computed using BTMO[START_REF] Rana | Optimizing Tone Mapping Operators for Keypoint Detection under Illumination Changes[END_REF], DetTMO[START_REF]Learning-based Adaptive Tone Mapping for Keypoint Detection[END_REF] and the proposed OpTMO for each test scene using Harris keypoint detector.

Figure 5 . 13 -Figure 5 . 14 -

 513514 Figure 5.13 -Keypoint Detection III. The head to head comparison between (a) OpTMO vs BTMO and, (b) OpTMO vs DetTMO. Each point represent an image pair with different lighting conditions from the HDR dataset. The points represented using o depict the Harris corner detector and 2 represents the SURF blob detector.

5 .Figure 5 . 15 -

 5515 Figure 5.15 -Descriptor Matching I computed on different TMOs using SURF, SIFT, FREAK, BRISK descriptor extraction schemes. The average is calculated over all test scenes.

5. 4 .Figure 5 . 16 -

 4516 Figure 5.16 -Descriptor Matching II. Matching Score comparison between BTMO [84], OpTMO,DetTMO[START_REF]Learning-based Adaptive Tone Mapping for Keypoint Detection[END_REF] and DesTMO[START_REF]Learning-Based Tone Mapping Operator for Image Matching[END_REF] over all the scenes in the HDR dataset using SURF feature extraction scheme.

Figure 5 . 17 -

 517 Figure 5.17 -Descriptor Matching III. Matching Score comparison between DesTMO vsOpTMO over all the scenes in HDR dataset. The points represented using o corresponds to FREAK feature detection scheme and 2 corresponds to SURF scheme.

Figure 5 . 18 -

 518 Figure 5.18 -Image Matching I. mAP % scores for the 9 different LDR modalities using 4 feature extraction schemes. Scores are averaged over 8 lighting change datasets.

Figure 5 . 19 -

 519 Figure 5.19 -Image Matching II. Day/Night matching using SURF. Row I: 2 HDR images fromInvalides scene are displayed after log scaling[START_REF] Drago | Adaptive logarithmic mapping for displaying high contrast scenes[END_REF]. Correct and incorrect matches are shown with yellow and red lines, respectively. Green lines represent the special case of mismatch due to repetitive structure. Row II: the feature matching using our proposed OpTMO (21 correct and 3 incorrect matches). Row III: using DetTMO (13 correct and 6 incorrect matches). Row IV: using DesTMOusing (11 correct and 3 incorrect matches). Row V using Reinhard TMO (3 correct and 11 incorrect matches). Row VI: using MantiukTMO (3 correct and 4 incorrect matches).

Figure 5 . 20 -Figure 5 . 21 -

 520521 Figure 5.20 -(Match & Locate) Row I: Pair of HDR images from Louvre, ProjectRoom and Notredame scenes, with one reference and other being a selected region undergone lighting change and rotation. Row II: the feature matching using our proposed OpTMO. Row III: using Reinhard TMO. (a) OpTMO (b) ReinhTMO (c) OpTMO (d) ReinhTMO (e) OpTMO (f) ReinhTMO

Figure 5 . 22 -

 522 Figure 5.22 -Computation time in sec (log scale). The time is computed by running all TMOs for an image size (512 × 512) on a Intel Xeon CPU 4 cores processor, 16 Gb RAM windows 7 machine.

  generated training samples by proposing a objective function capturing both the detection and description stages of the feature extraction pipeline. Later, we trained a Support Vector Regressor using local characteristics to learn a model which predicts spatially varying TMO parameters. We evaluate the proposed OpTMO on a HDR dataset of indoor/outdoor scenes where it outperforms state-of-the-art TMOs across different image matching algorithms.

5. 5 .

 5 Conclusions 95 different detection/description approaches and can be directly fused with any local feature based applications such as structure from stereo, scene reconstruction, object tracking, recognition and photogrammetric applications.

Figure 6 . 1 -

 61 Figure 6.1 -We illustrate here the training pipeline of our Deep Tone Mapping Operator (Deep TMO). Training dataset consists of input HDRs and their corresponding best-TMQI ranked tone mapped outputs. Both the discriminator and generator are trained alternatively, first a gradient step of discriminator then of generator. While the discriminator is trained to discriminate between real and fake image pairs, the generator learns to fool the discriminator by producing synthetic tone mapped images. By doing this, the generator effectively models the underlying distribution of real ground-truth tone mapped images, thus yielding high quality results once completely trained.

Figure 8 .

 8 Figure 8.15 presents an overview of our training algorithm. We train 3 deep learning based models in order to reconstruct tone-mapped LDR images from HDR images. This is effectively done by generating LDR images (fake pair in Figure 8.15) which are identical or even better than the ground-truth (GT) tone mapped images (real pair in Figure 8.15).

6. 3 .Figure 6 . 2 -

 362 Figure 6.2 -Here we show detailed architecture of both the discriminator and generator of DeepTMO-R and DeepTMO-S. The only difference for DeepTMO-S is the addition of skipped connections in the case of generator. The generator is framed as an encoder-decoder architecture, where the input HDR image is first passed to an encoder, which subsequently down-samples it to a compact representation. This representation is then forwarded through the decoder which up-samples it to the size of the input HDR. While the encoder consists of Convolution front end component G (F ) and first five residual blocks G (R) , the decoder is composed of next four residual blocks G (R) and a deconvolution component G (B) . Residual Blocks consist of two sequential convolution layers applied to the input, producing a residual correction that is in turn added to the input to yield the final output. Discriminator consists of a patchGAN[START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF][START_REF] Li | Precomputed real-time texture synthesis with markovian generative adversarial networks[END_REF] architecture which is applied patch wise on the concatenated input HDR and tone mapped LDR pairs. The final prediction is an average of all the patches over the image.

  (a) Generator Architecture for DeepTMO-HD is modified version of the Generator architecture of DeepTMO-R as shown in Figure 8.17a. DeepTMO-HD generator is basically a form of coarse-to-fine generator. While the finer Generator G 2 has original image as its input, input to G 1 is a 2× down sampled version. This down sampled image is then effectively passed through subsequent components G

6. 3 .

 3 Proposed Methodology 105 As illustrated in Figure 8.18a, at the time of inference, while the input to a local enhancer network is a high resolution HDR image (2048 × 1024), the global generator receives a 2× down sampled version of the same input. The local enhancer network, effectively makes tone-mapped predictions, paying attention to local fine-grained details (due to its limited receptive field on a high resolution HDR input). At the same time, it also imbibes from the global generator, a more coarser prediction (as its receptive field has much broader view). Thus the final generated output from G (B) 2

  sampled version respectively. The discriminators are then trained together to discriminate between real and synthetically generated images. D 2 , through working on a coarser scale of image, has a larger receptive field thus having a global view of the image.

Figure 6 .Figure 6 . 4 -Figure 6 . 5 -

 66465 Figure 6.4 -Compared between the three proposed architecture DeepTMO-R, DeepTMO-S, DeepTMO-HD (I). From the insets, DeepTMO-R suffers from blurriness issues in the wall and lowermost window panels. DeepTMO-HD and DeepTMO-S both are able to preserve the finer details, though the window panels are much more clearly visible in case of DeepTMO-HD

6 .Figure 6 . 6 -

 666 Figure 6.6 -Comparison of our method with the respective BestTMOs based on the TMQI scores with the highlighted zones for scene: The Canadian Falls (row I), The Grotto (row II) and the Bar Harbor Sunrise (row III). Zoom-ins for each scene highlights mapping outputs for DeepTMO-HD and the respective BestTMO for the corresponding HDRlinear input. We notice that our model has no saturation effect in waterfall (row I), preserves finer details in sky (row II) and effectively balances luminance for the house (row III). The TMQI scores for each scene are provided alongside each TMO.

6. 7 .

 7 Results and Evaluation113this case fails to recover the overly saturated regions, our model successfully preserves the fine details in the sky along with the waterfall and mountains in the background. Similarly in row 2, while SchlickTMO fails to preserve saturated regions in sky, our model is able to preserve both textures of the clouds as well as the bird in sky. We additionally are able to retain textures of the leaves which again for Schlick get partly saturated. Same goes for a dark scene in third row, where our trained DeepTMO-HD generator compensates the lighting and preserves the overall contrast of the generated scene which isn't the case for MantiukTMO. Hence we see that, learned with different mapping characteristics, our multi-scale generator network accordingly adapts for each scene to provide an image of convincing quality, which was missing in all the existing hand-crafted tone mapping methods.

6 .

 6 Deep Tone Mapping Opeartor for HDR Imaging and provides darker results. Altogether, we observe that our DeepTMO-HD addresses the generality concept and adapts to a variety of HDR content. Based on its high level understanding of the content in the scene, it efficiently recovers convincing colors, structure and the finer details of the scene.

Figure 6 . 7 -

 67 Figure 6.7 -Quantitative performance comparison of DeepTMO-R, DeepTMO-S and DeepTMO-HD with the BestTMOs.

Firstly, based on

  the limited state-of-the-art studies, we primarily identify the most natural and important questions in this direction. We begin with a performance evaluation study on what specific advantages can HDR images bring over LDR for a given feature extraction task. To this end, we propose a geometrically calibrated dataset with wide range of illumination condition. We then observe how different HDR modalities can impact feature extraction performance. Since no modality performs best across all scenes, we address the need of an optimal design to use HDR information and further, investigate the factors influencing the design essential for optimal modalities through a small experimental study.Secondly, considering our findings from evaluation study, we propose three in-line methodologies to optimally use the HDR information to enhance the efficiency of local features extraction. By adapting a variant of Bilateral filtering, we showcase models' learning capability by brining invariance to luminance change at three levels, namely, keypoint detection, description and final matching. Finally, we evaluate the performance of all the learning-based models on a proposed HDR dataset of 8 indoor/outdoor scenes where it outperforms state-of-the-art TMOs across different feature extraction algorithms.

••

  The keypoint detection and description designs are independent in nature. After optimizing the TMO for keypoint detection, we address tone-mapping optimality for full feature extraction algorithms. To this end, we first propose a descriptor-optimal TMO design which solely aims at the extraction of invariant (as much as possible) descriptors from high-contrast areas of the scenes. Then, we collectively address both stages of keypoint detection and descriptor extraction in the feature matching framework. To this end, we first propose an energy maximization model to generate appropriate training samples by subsequently addressing the detection and description costs. Then, by locally altering the intrinsic characteristics of tone mapping function, guidance model is learned to predict optimal parameter-maps.We evaluate both proposed TMOs on a HDR dataset of indoor/outdoor scenes where they outperforms state-of-the-art TMOs across different image matching algorithms.Our proposed task-optimal TMOs showcase their versatility when applied to different detection/description approaches and hence, can be directly plugged into various local feature based applications. (Refer to Chapter 5) We propose deep learning based TMOs which predict high quality tone mapped outputs over a wide spectrum of linear HDR images. Currently, the end-to-end deep learning based TMOs are trained with a perceptual objective to yield most natural images, thanks to the large availability of HDR images. Due to ease in backpropagation, the proposed model can be simply fined tuned with any desired objective such as image matching. Therefore, it eradicates the need of designing any proxy cost functions. Additionally, these models can serve as a baseline architecture to explore HDR imagery for several other domain specific analysis tasks such as medical image analysis or high resolution remote sensing tasks.Based on conditional generative adversarial network (cGAN), our proposed architectures learn to adapt to a wide variety of content and tackle HDR-specific challenges such as contrast, brightness and luminance, while preserving fine details. Aiming for high quality LDR, we address some prominent issues like blurring, tiling patterns and saturation artifacts encountered in past HDR related deep learning methods.To this end, we propose an additional high resolution prediction mechanism which caters to such finer details. To further leverage on the large availability of unlabeled high dynamic range data to train our network, we rely on an objective HDR quality metrics called Tone Mapping Image quality Index or TMQI. Finally, we demonstrate that our proposed deep tone mapping models generate high quality realistic output images and outperform all other classical tone mappings to generalize well over a larger spectrum of real-world scenes.(Refer to Chapter 6) 7. Conclusions7.2 Future Research DirectionsWide scale availability of HDR images and videos databases have opened up new analytical perspectives for future research. In this concluding section, we discuss several possible extensions of this thesis.

  , 8.14, information preserved by HDR images facilitates the extraction of highly stable and luminance invariant local features. The tested scenarios consider HDR images that are taken from a static scenes. However, real-time scenarios can be dynamic and consequently more challenging. This is mainly due to different combinations of physical transformations such as geometric (rotation, viewpoint change), deformational variations and due to sensor noise. One practical scenario includes lighting + viewpoint changes with mobile platforms such as drones.

Chapter 6 ,

 6 we proposed the deep learning based tone mapping methods for perceptual tasks. As already mentioned in Section 6.1, one possible extension is to design a task-optimal deep-learning based TMO by fine tuning the DeepTMO model. This could help us to compare the results obtained from deep networks learned from two different objectives (perceptual and computer vision) over an HDR dataset. Since HDR technology gives representation of real-world scenes closer to human eyes, the comparison between the two models will further open up the possibilities of research in deeper understanding of these networks, finding their technical explanations about inspirations drawn from human brain.

Figure 8 . 1 -

 81 Figure 8.1 -En (a), nous montrons un exemple de [119], où l'appariement des points saillants (communs aux deux images) est représenté par des lignes bleues entre deux images de la même scène prises à des heures différentes de la journée. En (b), nous montrons le taux de répétabilité de la mesure d'efficacité sur un grand ensemble de données d'images LDR jour/nuit en utilisant les techniques de pointe. La crête et les creux dans les courbes illustrent que l'image capturée pendant le jour correspond bien avec seulement d'autres images de jour et non avec celles capturées dans l'obscurité.

Figure 8 . 3 -Figure 8 . 4 .

 8384 Figure 8.3 -Average RR et standard deviation pour les approches optimisées basées sur la corrélation et la réponse en utilisant respectivement un détecteur Harris et un détecteur SURF. Rangée 2. (c) et (d) Score moyen de répétabilité et écart-type pour les modèles de réflectance (GTM et BTM) et autres TMs couramment utilisés pour Project Room et Light Room dataset

Figure 8 . 4 -

 84 Figure 8.4 -Learning based DetTMO.

Figure 8 . 6 -Figure 8 . 5 -

 8685 Figure 8.6 -Average Repeatability Rates (AvgRR) calculée sur différents TMOs en utilisant divers schémas de détection de points clés. La moyenne est calculée sur toutes les scènes de test.

Figure 8 . 7 -

 87 Figure 8.7 -Repeated Keypoints. Row I: 2 images HDR de la scène Invalides prises à différentes heures du jour. Les images HDR sont affichées après la mise à l'échelle du journal [27]. Row II: les points clés répétés en utilisant notre DetTMO proposé (66 points clés répétés sur les 200 points clés les plus forts). Row III: les points clés répétés à l'aide de Reinhard TMO (7 points clés répétés sur les 200 points clés les plus forts). Row IV: les points clés répétés à l'aide de MantiukTMO (5 points clés répétés sur les 200 points clés les plus forts).

8. 1 . 4 8 .

 148 Figure 8.8 -L'architecture de DesTMO.

Figure 8 . 9 -Figure 8 .Figure 8 . 11 -Figure 8 . 12 -

 898811812 Figure 8.9 -Apprentissage DesTMO

Figure 8 . 13 - 8 .

 8138 Figure 8.13 -Average Matching Score (MS) calculés sur différents TMOs en utilisant les schémas d'extraction des descripteurs SURF, SIFT, FREAK, BRISK, BRISK. La moyenne est calculée sur toutes les scènes de test.

8. 1 . 5 Figure 8 . 14 -

 15814 Figure 8.14 -Image Matching II. Correspondance jour/nuit à l'aide de SURF. Row I : 2 images HDR de la scène Invalides sont affichées après la mise à l'échelle du journal. Les correspondances correctes et incorrectes sont indiquées par des lignes jaunes et rouges, respectivement. Les lignes vertes représentent le cas particulier d'inadéquation due à une structure répétitive. Row II : l'appariement des caractéristiques à l'aide de notre proposition OpTMO (21 correspondances correctes et 3 incorrectes). Rangée III : en utilisant DetTMO (13 correspondances correctes et 6 incorrectes). Row IV : utiliser DesTMO (11 correspondances correctes et 3 incorrectes). Ligne V en utilisant Reinhard TMO (3 correspondances correctes et 11 incorrectes). Row VI : utiliser MantiukTMO (3 correspondances correctes et 4 incorrectes).

Figure 8 . 15 -

 815 Figure 8.15 -Pipeline de formation de Deep Tone Mapping Operator (Deep TMO) basé sur les GANs. L'ensemble de données apprentisage se compose des HDR d'entrée et de leurs sorties correspondantes les mieux classées en fonction de l'TMQI et de l'indice de qualité du ton. Le discriminateur et le générateur sont formés alternativement, d'abord une étape de regression du discriminateur puis du générateur. Tandis que le discriminateur est formé pour distinguer les paires d'images réelles des fausses, le générateur apprend à tromper le discriminateur en produisant des images en tons synthétiques. Ce faisant, le générateur modélise efficacement la distribution sousjacente des images réelles de la tonalité de vérité au sol, ce qui donne des résultats de haute qualité une fois l'entraînement terminé.

Figure 8 . 16 -

 816 Figure 8.16 -Nous proposons un TMO basé sur l'apprentissage profond (appelé DeepTMO) qui donne des résultats de haute qualité subjective sur un large éventail d'images HDR à valeur linéaire. Notre variante proposée de cGANs est une architecture multi-échelle qui donne des résultats d'apparence naturelle et sans artefacts en haute résolution. Alors que les TMOs classiques sont sensibles à l'accord des paramètres pour une sortie souhaitée, notre modèle apprend à traiter efficacement une plus large gamme de contenus HDR en modélisant la distribution sous-jacente de toutes les sorties de cartographie des tons cibles disponibles. En concurrence avec les meilleurs résultats des cartes tonales subjectives de qualité supérieure sur 3 types de scènes différentes : claires, nuageuses et sombres, nous montrons surtout la polyvalence de notre méthode qui préserve efficacement les textures, les détails des structures et le contraste. Les résultats détaillés sont présentés aux sections 6 et 7 de chapitre 6. Enfin, notre modèle DeepTMO est assez rapide et prend en moyenne 0,02 seconde pour le tone mapping d'une image HDR de taille 1024 × 2048.

8 .

 8 Résumé de thèse (a) Architecture de générateur avec et sans 'skip-connections'. (b) Architecture de discriminateur.

Figure 8 . 17 -

 817 Figure 8.17 -Nous présentons l'architecture détaillée du discriminateur et du générateur de DeepTMO-R et DeepTMO-S. La seule différence pour DeepTMO-S est l'ajout de connexions sautées dans le cas d'un générateur. Le générateur est encadré comme une architecture codeur-décodeur, où l'image HDR d'entrée est d'abord transmise à un codeur, qui la sous-échantillonne ensuite en une représentation compacte. Cette représentation est ensuite transmise par le décodeur qui l'échantillonne à la taille du HDR d'entrée. Alors que le codeur se compose du composant frontal Convolution G (F ) et des cinq premiers blocs résiduels G (R) , le décodeur se compose des quatre blocs résiduels suivants G (R) et du composant déconvolution G (B) . La discriminateur se compose d'une architecture patchGAN qui est appliquée à chaque patch sur les paires HDR d'entrée concaténées et les paires LDR de tonalités mappées. La prédiction finale est une moyenne de tous les patchs sur l'image.

  L'architecture du générateur pour DeepTMO-HD est une version modifiée de l'architecture du générateur de DeepTMO-R comme le montre la Figure 8.17a. Le générateur DeepTMO-HD est essentiellement une forme de générateur grossier à fin. Tandis que le générateur plus fin G 2 a l'image originale en entrée, l'entrée G 1 est une version échantillonnée 2×. Cette image échantillonnée est ensuite effectivement passée à travers les composants suivants G à ceux du générateur dans DeepTMO-R. La prédiction finale de l'extrémité arrière G B 1 est ensuite concaténée avec la sortie de l'extrémité avant du générateur à plus petite échelle G 2 . Ceci est ensuite passé à travers le composant G (B) 2 pour produire une sortie avec correspondance des tonalités. Ainsi, notre modèle utilise efficacement l'information à l'échelle plus grossière et à l'échelle plus fine pour faire une prédiction qui permet de mieux retenir la structure globale et les moindres détails des bas niveaux. L'architecture du discriminateur a une architecture identique mais nous lui donnons deux échelles d'entrée différentes, l'original et sa version échantillonnée 2×. Cela oblige le générateur à s'occuper à la fois des détails globaux et locaux.

Figure 8 . 19 - 3 . 4 .

 81934 Figure 8.19 -Nous comparons les performances quantitatives de DeepTMO-R, DeepTMO-S et DeepTMO-HD avec celles des BestTMO.

Figure 8 . 20 -

 820 Figure 8.20 -Resultat Qualitatif de DeepTMO-HD.

152 8 .•

 8 Résumé de thèse 8.1.6 Orientations futures de la recherche La disponibilité à grande échelle des bases de données d'images et de vidéos du HDR a ouvert la voie à de nouvelles analyses, les perspectives d'avenir de la recherche. Dans cette section, nous discutons de plusieurs extensions possibles de cette thèse. Investigation de l'imagerie HDR pour les scènes dynamiques. Comme le montrent les résultats des Figures 4.13 et 5.19, l'information préservée par les images HDR facilite l'extraction de caractéristiques locales très stables et invariantes de luminance. Les scénarios prennent en compte les images HDR qui sont prises à partir de scènes statiques. Cependant, Les scénarios peuvent être dynamiques et, par conséquent, plus difficiles. Cela s'explique principalement par le fait qu'il n'y a pas la même chose. Les combinaisons de transformations physiques telles que les transformations géométriques (rotation, changement de point de vue), des variations déformatrices et dues au bruit des capteurs. Un scénario pratique comprend l'éclairage + l'éclairage des changements de points de vue avec des plates-formes mobiles telles que les drones. Les modèles présentés doivent théoriquement s'adapter en fonction de l'invariance des algorithmes d'extraction d'entités, tel qu'illustré à la figure 5.20 pour la rotation planaire. Cependant, pour les problèmes de rotation hors plan qui sont spécifiques à une capture mobile. nos modèles ont besoin d'être recalibrés de manière. Il est à noter qu'il n'y a pas d'équipement de létat les algorithmes d'extraction de caractéristiques locales sont meilleurs sous toutes les transformations [116, 119]. Par conséquent, au lieu d'apprendre simplement les modèles régresseurs avec un coin de référence et un descripteur basé sur le coin et le descripteur caractéristiques multiples, caractéristiques multiples à partir d'algorithmes d'extraction de caractéristiques, par exemple celles qui ont été évaluées. dans[119], a besoin d'être infusé. De plus, il nécessite un calibrage géométrique approprié qui doit être créé. • Perception Vs Vision. Au chapitre 6, nous avons proposé les TMOs basés sur l'apprentissage profond pour les TMOs perceptuels. Comme nous l'avons déjà mentionné à la section 6.1, l'une des extensions possibles est la conception d'une tâche optimale. TMO basé sur l'apprentissage profond en affinant le modèle DeepTMO. Cela pourrait nous aider à comparer les résultats obtenus à partir de réseaux profonds tirés de deux objectifs différents (vision perceptuelle et vision par ordinateur) sur un ensemble de données HDR. Puisque la technologie HDR donne une représentation de scènes du monde réel plus proche de l'oeil humain, la comparaison entre les deux modèles ouvrira davantage les possibilités de recherche dans la compréhension plus profonde de ces réseaux. Il motivera en outre la recherche d'explications techniques inspirées par le cerveau humain. • Analyse HDR dans le domaine temporel. Une grande partie de cette thèse explore 8.1. Résumé 153 la polyvalence de l'imagerie à gamme dynamique élevée pour améliorer la stabilité des caractéristiques locales dans les images RVB. Cependant, l'une des extensions naturelles est la mise à niveau de l'analyse pour les vidéos HDR. L'information dans le domaine temporel a beaucoup plus de potentiel pour les applications de vision par ordinateur en temps réel telles que les tâches de surveillance. Une grande quantité d'informations perdues dans des scènes à faible contraste peut être reconstruite à l'aide d'une modélisation prédictive en utilisant une dimension temporelle supplémentaire des vidéos HDR. Cela pourrait améliorer les performances dans plusieurs tâches vidéo telles que les applications de surveillance, le suivi en temps réel, l'analyse des gestes et des actions.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 8.8 L'architecture de DesTMO. . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

8.9 Apprentissage DesTMO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 8.10 L'architecture de OpTMO. . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 8.11 Apprentissage OpTMO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 8.12 Average RR calculée sur différents TMOs en utilisant divers schémas de détection de points clés. La moyenne est calculée sur toutes les scènes de test.141 8.13 Average Matching Score (MS) calculés sur différents TMOs en utilisant les schémas d'extraction des descripteurs SURF, SIFT, FREAK, BRISK, BRISK. La moyenne est calculée sur toutes les scènes de test. . . . . . . . 141 8.14 Image Matching II. Correspondance jour/nuit à l'aide de SURF. Row I :

  ) where r ref is the number of keypoints detected in the reference image which are repeated in the test image, and n ref and n test are the number of detected keypoints

	36	2. Background and State of the Art
		in [70] namely threshold based matching, nearest neighbor (NN) matching and nearest
		neighbor distance ratio (NNDR). In threshold-based matching two descriptors are
		said to have a match if the distance between them is below a certain threshold. As a
		descriptor can have several matches, this strategy is error-prone and is seldom used.
		In nearest neighbor (NN) matching, a descriptor A finds its match B only if A is
		the nearest neighbor to B and if the distance between them is below a threshold.
		Nearest neighbor distance ratio (NNDR) extends NN by introducing a threshold to
		the ratio of the distance descriptors. More precisely, a descriptor finds a good match
		if the ratio between its distance from the first closest match and its distance from
		the second closest match is less than a given threshold th. These distances depend
		on the descriptor type, i.e.Hamming distance metric is used for binary descriptors
		and Euclidean distance is used for non-binary descriptors.
		in the reference and test image, respectively. A keypoint is considered to be repeated
		in the test image if: a) it is detected as a keypoint in the test image, and b) it lies in
		a circle of radius centered on the projection of the reference keypoint onto the test
		image. Generally, the number of detected keypoints in images vary at large which
		might lead to a bias in the final RR. Therefore, in our evaluation framework, we
		select the strongest N number of keypoints to avoid any form of numerical bias.
		• Descriptor Matching: The match-ability of a decriptor is measured using the MS. It
		is defined as the fraction of correct matches to the total number of correspondences
		in the image pair. A match has been defined using three different matching strategies

  3. Local Feature extraction in HDR Imagery under Drastic LightingVariations

	Abbreviations	Description
	LDR	Best exposure LDR image of the scene
	RNG(G)	A global scaled mapping operator [90]
	DR(G)	An Adaptive logarithmic mapping [27]
	RN(L)	A local dodging-and-burning operator [90]
	MA(L)	Perceptual method for contrast processing [67]
	FA(L)	Gradient domain HDR compression [35]
	CH(L)	Spatially non-uniform scaling algorithm [21]
	DU(L)	A fast bilateral filtering technique [29]
	HDRLog	Logarithmic encoding in accordance to Weber-Fechner's law
	HDRLin	Linear photometric luminance values stored in the HDR file

Table 3 . 2 -

 32 Different image modalities for feature extraction.

Table 3 .

 3 .8 -Mean average precision (mAP) and mean repeatability rate (mRR) over the four considered datasets and feature schemes. mAP and mRR are computed on 56 image pairs, for the Project Room dataset, and over 42 image pairs for Light Room, 2D and 3D Lighting datasets.3.4. Evaluation of full Features Extraction Pipelines in HDR imagery 49

	Repr.	Feature Extraction Schemes SIFT SURF BRISK FREAK	Avg/Repr.
	LDR	55	62	60	61	59.5
	RNG	69	70	71	65	67.5
	DR	72	72	71	73	72
	RN	72	70	73	72	72
	MA	74	75	62	62	68.3
	FA	68	67	62	66	65.8
	CH	68	71	64	66	67.3
	DU	64	72	68	71	68.8
	HDRLog	75	66	67	68	69
	HDRLin	44	30	50	41	41.5
	Avg/Schemes 66.8	65.6	65.5	65

3 -Mean Average Precision (mAP %) scores for the 10 considered representations using 4 fea- ture extraction schemes. Scores are averaged over 4 lighting change datasets. Highest mAP score for each scheme is shown in bold. Best Avg/Formats and Avg/Schemes scores are double underlined.

•

  We investigate how much gains can HDR bring over LDR for the feature extraction stages and which are the best modalities of using HDR to obtain it. To this end, we prepared the framework with 11 HDR based modalities, 2 keypoint detection and 4 full feature extraction schemes. Additionally, we propose a geometrically calibrated HDR luminance change dataset with a variety of lighting variations. The analysis based on quantitative performance measures of keypoint detection and feature matching scores on different scenes confirms the potential of HDR techniques over single LDR exposures. For both detection and matching, we observe the linear HDR values are inappropriate to be used for LDR optimized visual recognition tasks.In case of TMOs, we observe that their performance varies with the type of scene, exhibiting their nature of content-dependence. Furthermore, we observe that all the local TMOs producing very appealing results are not necessarily the best option for image analysis tasks. More interestingly, we have also observed that local TMOs

	with very high repeatability rate for feature detection are not necessarily the best
	option when the full feature extraction pipeline is considered. For individual test
	pairs cases, we find no modality which is absolutely outperforming. Hence, it remains
	unclear whether HDR pixels should be encoded approximatively linear to perception
	or directly tone map using existing functions. Therefore, a more optimal means of
	modality needs to be designed. (Refer to Chapter 3)
	• We develop a learning based adaptive tone mapping framework for HDR images which
	results in stable and efficient keypoint detection. To this end, we initially conduct an
	experimental study investigating what it takes to optimize a tone mapping function
	for a specific task such as keypoint detection. Build on the observations, we propose
	a new learning-based adaptive tone mapping framework which aims at enhancing
	keypoint stability under drastic illumination variations. To this end, we design a
	pixel-wise adaptive TMO which is modulated based on a model derived by Support
	Vector Regression (SVR) using local higher order characteristics. To circumvent the
	difficulty to train SVR in this context, we further propose a simple detection-similarity-
	maximization model to generate appropriate training samples using multiple images
	undergoing illumination transformations.

  Dans le premier aspect, nous identifions les questions les plus naturelles et les plus importantes sur la base des études limitées de l'état de l'art dans cette direction. Nous commençons par une étude d'évaluation des performances sur les avantages spécifiques que les images HDR peuvent apporter par rapport aux images LDR pour une tâche donnée d'extraction de features. A cette fin, nous proposons un jeu de données géométriquement calibré avec un large éventail de conditions d'éclairage. Nous observons ensuite comment les différentes modalités du HDR peuvent avoir un impact sur les performances d'extraction

	8. Résumé de thèse
	Cette thèse présente l'analyse quantitative et qualitative de l'imagerie HDR pour des
	tâches spécifiques. Cette thèse explore tout d'abord les avantages du HDR pour des tâches
	spécifiques, ensuite, aborde en temps opportun les limites et les défis propres à chaque
	tâche.
	Dans cette thèse, nous soulignons les contributions apportées dans les trois aspects
	majeurs qui suivent.
	En adaptant une variante du filtrage bilatéral, nous mettons en évidence la capacité
	d'apprentissage des modèles. Nous proposons d'apporter l'invariance au changement de
	luminance à trois niveaux : (1) la détection des points clés, (2) la description et (3) la
	mise en correspondance des images. Enfin, nous évaluons la performance de tous les
	Bien que ces algorithmes soient entièrement personnalisés pour les images LDR capturées modèles proposés en fonction de l'apprentissage, sur la base d'un ensemble de données
	dans des conditions différentes, ils échouent lamentablement dans les scènes à contraste HDR proposé de 8 scènes intérieures/extérieures. Nous montrons en outre comment notre
	élevé ayant une luminance élevée ou faible [44, 110, 117, 119]. Puisque les scènes à contraste modèle surpasse les Opertateur de Tone mapping (TMOs) les plus perfectionnés à travers
	élevé sont extrêmement courantes dans le monde réel, cela devient très critique dans des différents algorithmes d'extraction de caractéristiques.
	cas comme les véhicules automatisés où des vies humaines sont en danger impliqués. Ainsi, Troisièmement, pour traiter une grande variété d'images HDR du monde réel, nous
	l'utilisation de la technologie HDR est néceéssaire pour la viabilité des algorithmes de présentons trois modèles génériques de cartographie tonale basés sur l'apprentissage profond
	vision par ordinateur. Bien qu'un grand nombre d'algorithmes aient été concus pour pour 'end-to-end' qui répondent aux caractéristiques spécifiques des tâches souhaitées.
	interpréter des scènes surexposées ou sous-exposées à l'aide d'images LDR, peu de travail a Les TMOs que nous avons proposés précédemment exigent essentiellement une concep-
	été fait jusqu'à présent dans le contexte du contenu HDR. tion de filtrage spécifique qui doit être différentiable. Comme tous les TMO ne peuvent
	Cette thèse est centrée sur l'analyse d'images HDR "enrichies" au profit d'un problème pas satisfaire ces critères, DeepTMOs surmonte ces inconvénients et apprend d'une variété
	de correspondance des caractéristiques visuelles de bas niveau, qui est la base de nombreux

8.1 Résumé

La technologie HDR (High Dynamic Range) a acquis une immense popularité pour sa capacité à représenter une large gamme de couleurs et d'intensités lumineuses présentes dans des environnements réels

[START_REF] Dufaux | High Dynamic Range Video: From Acquisition, to Display and Applications[END_REF][START_REF] Mantiuk | High Dynamic Range Imaging[END_REF]

. Dans un sens, ces images nous permettent de dessiner des détails subtils, mais discriminants, présents à la fois dans les zones extrêmement sombres et claires d'une scène, qui autrement se perdraient dans l'imagerie traditionnelle à gamme dynamique basse (LDR). Avec les récents progrès de l'intelligence artificielle, que ce soit sous la forme de voiture autonaume ou de dispositifs de surveillance automatisés, un tel contraste préservant les propriétés HDR est essentiel pour la compétence des algorithmes de vision par ordinateur sous-jacents. En d'autres termes, ces algorithmes devraient être capables d'analyser efficacement chaque région d'une scène sans trop d'incertitude. autres algorithmes de vision par ordinateur de haut niveau, y compris l'enregistrement et la vision stéréoscopique, le mouvement l'estimation et la localisation, l'appariement, la récupération et la reconnaissance d'objets et d'actions. Plus spécifiquement, la thèse examine les défis fondamentaux liés à l'utilisation du HDR et en déduit les moyens optimaux d'utiliser le contenu HDR pour améliorer la robustesse de telles tâches. des features. Puisqu'aucune modalité n'est la plus performante sur toutes les scènes, nous devons répondre au besoin d'une conception optimale qui utilise l'information du HDR. Nous étudions plus en détail les facteurs qui influencent la conception essentielle pour des modalités optimales par le biais d'une étude expérimentale. Deuxièmement, nous proposons trois méthodologies spécifiques tout en tenant compte des résultats de l'étude d'évaluation. Nous proposons d'utiliser de manière optimale les informations HDR pour améliorer l'efficacité de l'extraction des caractéristiques locales. de caractéristiques de filtrage et qui est facilement différentiable. De plus, les DeepTMOs peuvent servir de référence pour les futures conceptions HDR optimales des tâches, car elles peuvent être affinées pour n'importe quelle tâche spécifique en simple cascade.

Table 8 .

 8 2 -Résultats quantitatifs. résultats moyens à l'TMQI sur un ensemble de 105 images. DeepTMO-HD TMO 0.87 ±0.06 DeepTMO-S TMO 0.88 ±0.07 DeepTMO-R TMO 0.86 ±0.08

	TMOs	TMQI
	Ward TMO	0.71 ±0.07
	Pattnaik TMO	0.78 ±0.04
	Log TMO	0.72 ±0.09
	Gamma TMO	0.76 ±0.07
	Ashikh TMO	0.70 ±0.06
	Durand TMO	0.81 ±0.10
	Tumblin TMO	0.69 ±0.06
	Drago TMO	0.81 ±0.06
	Schlick TMO	0.79 ±0.09
	Reinh TMO	0.84 ±0.07
	Fattal TMO	0.81 ±0.07
	Chiu TMO	0.70 ±0.05
	Mantiuk TMO	0.84 ±0.06

Local Feature extraction in HDR Imagery under Drastic Lighting Variations

http://qtpfsgui.sourceforge.net/

définir explicitement une fonction de perte spécifique à une tâche. Cela se produit principalement parce que nos réseaux sont formés pour modéliser par eux-mêmes les fonctions de coût adaptées à partir des données de formation sous-jacentes. Nous fournissons les détails architecturaux des trois réseaux dans la Figure8.17 and 8.18a. Pour former ces modèles proposés, nous accumulons des données à partir des sources d'images HDR disponibles. Cependant, un défi majeur lors de la formation des modèles découle de l'absence de tout modèle public l'ensemble des données disponibles sur la formation. Sélectionner la vérité terrain par une évaluation subjective plutôt qu'un grand ensemble de données est une tâche très fastidieuse. Elle nécessite donc l'exigence d'un objectif métrique d'évaluation de la qualité qui permet de quantifier les performances de cartographie tonale de chaque TMO pour n'importe quelle scène possible. Pour notre tâche, nous sélectionnons une métrique bien connue, à savoir le Tone Mapped Image Quality Index (TMQI), qui est utilisé pour classer 13 TMOs largement utilisés. En utilisant ceci, pour chaque entrée HDR, nous sélectionnons celui qui se classe le plus haut sur ce score métrique objectif TMQI. Les résultats visuels produits par nos modèles DeepTMO dans différentes scènes sont illustrés dans le Figure 8.15. Notre DeepTMO apprend implicitement la meilleure caractéristique de tous les TMOs globales, locaux et perceptifs disponibles sur une grande variété de scènes. En un sens, tant par sa conception architecturale que par l'ensemble des données sous-jacentes, il est conditionné pour préserver les données globales (telles que les structures globales, les contrastes et la luminance) ainsi que les détails locaux plus fins (tels que les motifs de texture locale), ce qui permet d'obtenir des résultats de haute qualité visuellement agréables.Nos modèles de TMO profonds permettent également de surmonter les effets de flou ou de carrelage fréquemment étudiés dans les travaux récents liés au HDR[START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR-Video[END_REF][START_REF] Endo | Deep reverse tone mapping[END_REF], un problème d'intérêt significatif pour plusieurs applications de rendu graphique basé sur l'apprentissage de haute qualité, comme souligné dans[START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR-Video[END_REF]. En apprenant simplement une fonction de coût spécifique HDR-to-LDR, les modèles proposés préservent avec succès les caractéristiques de sortie souhaitées telles que le contraste sous-jacent, l'éclairage et les détails minuscules présents dans le HDR d'entrée à l'échelle la plus fine. Enfin, nous validons la polyvalence de notre méthodologie par une comparaison quantitative et qualitative détaillée avec les TMOs existantes. Nous comparons les performances quantitatives de DeepTMO-R, DeepTMO-S et DeepTMO-HD avec celles des BestTMO dans le Figure 8.19 et la Table 8.2. Nous démontrons que les TMOs profonds que nous proposons génèrent des images de sortie réalistes de haute qualité et surpassent tous les autres TMOs classiques pour bien généraliser sur un plus large spectre de scènes du monde réel (dans le Figure 8.20). En un mot, 1. Nous proposons le premier opérateur de tone mapping basée sur l'apprentissage en

Résumé de thèse

Conclusions

Publications

 Chapter 6Deep Tone Mapping Opeartor for HDR Imaging

Overview

With a given task-based objective, so far we proposed models by relying on specific characteristics of a given tone mapping function. In fact, a variant of Bilateral filtering has been adopted to showcase model's learning ability. However, not all the TMOs are differentiable and consequently, difficult to be learned using the proposed methods. Moreover, an individual TMO addresses only some specific characteristics which might be desired based on the content. This raises a natural question whether a more general tone mapping function can be formulated which can be easily trained for any given task and adapt itself for all the real world scenes.

In this chapter, we address this question by designing a generic end-to-end TMO which adapts itself for all the real world scenes considering the desired task-specific characteristics.