Vedad Hulusic
email: vedad.hulusic@telecom-paristech.fr

Kurt Debattista

Giuseppe Valenzise

Frédéric Dufaux

A

Keywords: High Dynamic Range, Perceived Dynamic Range, Subjective

  L'archive ouverte pluridisciplinaire

Evaluation, Predictor Model

Introduction

High dynamic range (HDR) technology [START_REF] Banterle | Advanced high dynamic range imaging: theory and practice[END_REF][START_REF] Dufaux | High Dynamic Range Video. From Acquisition, to Display and Applications[END_REF] enables the capture, storage, transmission and display of the full range of real-world lighting and colors, with a significant increase in precision when compared to traditional low dynamic range (LDR) imaging. One of HDR's main features is its ability to reproduce very bright and very dark portions of a scene concurrently. The span between these extrema in the brightness scale is commonly referred to as the dynamic range of a picture.

The dynamic range of image or video content is frequently reported by many HDR applications. It is typically computed as the ratio between the maximum and minimum pixel luminance of an image, which will be referred to as pixelbased dynamic range (DR) in this paper. Such a computation can be biased due to image noise or singularities, such as isolated pixels with extreme luminance values. Furthermore, such measures do not capture the complex behavior of the human visual system's (HVS) response and perception of lightness, including its intrinsic content-dependency [START_REF] Gilchrist | An anchoring theory of lightness perception[END_REF]. The perceived dynamic range (PDR) of HDR content and its assessment in HDR conditions still remain unexplored. The accurate prediction of PDR would be important for a number of applications.

It could be used to optimize and evaluate inverse tone mapping operators (IT-MOs) [START_REF] Banterle | Inverse tone mapping[END_REF][START_REF] Rempel | LDR2HDR: On-the-fly reverse tone mapping of legacy video and photographs[END_REF][START_REF] Simone | Dynamic range expansion of video sequences: A subjective quality assessment study[END_REF], HDR compression methods and HDR reproduction systems [START_REF] Mantel | Subjective quality of videos displayed with local backlight dimming at different peak white and ambient light levels[END_REF]; it could be used for developing objective image quality measures and quantifying aesthetic attributes [START_REF] Aydin | Automated aesthetic analysis of photographic images[END_REF]; it provides an objective means to select content for HDR presentations and subjective studies [START_REF] Narwaria | An objective method for High Dynamic Range source content selection[END_REF]; and in general it would help to better understand lightness and color perception, by extending studies on the anchoring problem [START_REF] Gilchrist | An anchoring theory of lightness perception[END_REF] to complex stimuli and HDR conditions.

To the authors' knowledge, this work is the first attempt to assess and predict the perceived dynamic range of HDR images under HDR conditions. Currently neither a standardized methodology nor an HDR data set with annotated 2 measurements of this perceptual attribute exist. For this purpose, a subjective study with 23 participants was designed and conducted, using a set of 36 HDR images (chosen from a larger pool) with different characteristics and content semantics, including indoor/outdoor scenes, natural/man-made scenes and other variations. This work is an extended version of the pilot study that investigated only the achromatic stimuli [START_REF] Hulusic | Perceived dynamic range of HDR images[END_REF]; this extension has increased the number of participants, added chromatic stimuli to the study and used the data to propose and evaluate a predictive model. While dynamic range is generally measured based only on the brightness of a picture, well-known color appearance phenomena such as Hunt or Helmholtz-Kohlrausch effects [START_REF] Fairchild | Color appearance models[END_REF] tend to question this assumption and rather lean towards the hypothesis that dynamic range perception changes from achromatic to chromatic stimuli. Therefore, in this work, both the achromatic and chromatic images were used and the correlation of the subjective scores is investigated. This paper makes the following contributions:

• a subjectively annotated data set with PDR values, using complex, chromatic and achromatic stimuli and HDR viewing conditions (using an HDR display) was created 1 ;

• a novel test methodology for measuring perceived dynamic range, partially inspired by the subjective assessment methodology for video quality (SAMVIQ) [START_REF] Blin | SAMVIQ -Subjective assessment methodology for video quality[END_REF] is proposed;

• based on the results of the study, the Pearson's correlations between mean opinion scores (MOS) and five image features, i.e., pixel-based dynamic range (correlation coefficient r = 0.87 achromatic, r = 0.84 chromatic), image key (r = -0.60, r = -0.61), area (modified to account for non-linearity; r = 0.87, r = 0.87), contrast (r = -0.19, r = -0.22) and colorfulness (r = -0.47 chromatic only) were analyzed; 1 The data set is available on the project website: http://pdr.lefca.net

• the effect of chromatic information on perceived dynamic range was investigated and the relation between chromatic and achromatic quantified (r = 0.99);

• a model for predicting the perceived dynamic range for both achromatic (adjusted R 2 = .89) and chromatic (adjusted R 2 = .87) images has been 60 proposed.

In the rest of the paper, the abbreviations and terms presented in Table 1 will be used.

Related Work

The study of perceived dynamic range shares some common features with perceived lightness, image contrast and image quality. Most of these psychoperceptual theories lack sufficient validation with complex stimuli, and have never been tested in HDR conditions.

Perceived Lightness

Lightness is a measure of relative brightness of elements of a scene, and has been extensively studied within the field of perceptual psychology. One of the most popular models in lightness perception is the intrinsic image model [START_REF] Bergstrom | Common and relative components of reflected light as information about the illumination, colour, and three-dimensional form of objects[END_REF][START_REF] Barrow | Recovering intrinsic scene characteristics, Comput[END_REF][START_REF] Gilchrist | The perception of surface blacks and whites[END_REF][START_REF] Adelson | The perception of shading and reflectance[END_REF]. This is typically based on a multi-component approach, which consists in factorizing the perceived scene as the interaction of different elements, e.g., surface reflectance, illumination and three-dimensional form or depth values. However, Gilchrist argued that these models are incomplete and proposed a new anchoring model [START_REF] Gilchrist | An anchoring theory of lightness perception[END_REF], based on a combination of global and local anchoring of lightness values. On the whole, the anchoring models promote the idea that the perception of lightness is determined by the brightest patch of the scene. The human visual system then scales the rest to this maximum, generating an internal, scene-dependent scale of light and dark. Furthermore, Li and Gilchrist [START_REF] Li | Relative area and relative luminance combine to anchor surface lightness values[END_REF] observed that anchoring is affected by the relative area of the brightest patch.

The Retinex theory [START_REF] Land | Lightness and retinex theory[END_REF] attempts to generate an output that is most similar to what a human observer would perceive by looking at the real scene where an image was taken. Compared with the anchoring theories, the Retinex theory indirectly arrives to the same conclusion through a probabilistic formulation. This is achieved by averaging luminance values along paths of pixels originating from each point of the picture [START_REF] Provenzi | Mathematical definition and analysis of the retinex algorithm[END_REF], while taking into account the relative distance between patches of different brightness.

Image Contrast

While image contrast and dynamic range have similar perceptual connotation, they describe different aesthetic attributes. Global contrast measures [START_REF] Peli | Contrast in complex images[END_REF][START_REF] Rizzi | A proposal for contrast measure in digital images[END_REF][START_REF] Matkovic | Global contrast factor-a new approach to image contrast[END_REF] model image sharpness, which is a fine-scale image feature, as the perceptual experiments by Haun and Peli [START_REF] Haun | 1: Measuring the perceived contrast of natural images[END_REF] clearly display. On the contrary, the perceived dynamic range is a measure of the magnitude of the global difference in the perceived image brightness. This is closely related to the "tone" aesthetic attribute as defined in work by Aydin et al. [START_REF] Aydin | Automated aesthetic analysis of photographic images[END_REF]. However, in their work this attribute is computed as a variation of the pixel-based dynamic range and does not take into account other perceptual factors. In a series of studies by Calabria and Fairchild it has been shown that perceived image contrast for LDR images is a function of image lightness, chroma, and sharpness [START_REF] Calabria | Perceived image contrast and observer preference i. the effects of lightness, chroma, and sharpness manipulations on contrast perception[END_REF] and the model of perceived image contrast and observed preference data were proposed [START_REF] Calabria | Perceived image contrast and observer preference ii. empirical modeling of perceived image contrast and observer preference data[END_REF].

In their recent work, Vangorp et al. defined a model of local adaptation and use it to measure dynamic range as the ratio between the brightest and darkest image region in which people can still see some details [START_REF] Vangorp | A model of local adaptation[END_REF]. The model predicts the maximum visible dynamic range for any given scene, based on both glare and local adaptation. The results revealed that the greatest decrease in perceived dynamic range, compared to the physical DR, occurs in darker scene regions due to glare. On the other hand, local adaptation causes a significant loss of dynamic range visibility in brighter parts of the scenes.

Image Quality

Image quality metrics are usually based on either predicting the visibility of distortions [START_REF] Winkler | Chapter 5: Perceptual video quality metrics-a review[END_REF][START_REF] Kopf | Quality prediction for image completion[END_REF] or magnitude of aesthetic attributes [START_REF] Aydin | Automated aesthetic analysis of photographic images[END_REF]. While such metrics usually do not consider perceived dynamic range explicitly, there is evidence that the latter is highly related with the overall image quality. In the study by Akyüz et al., when two versions of the same image were compared, observers generally preferred the higher dynamic range one [START_REF] Akyüz | Do HDR displays support ldr content?: a psychophysical evaluation[END_REF].

Therefore, assessing the perceived dynamic range of HDR pictures and videos can make a great contribution in design and evaluation of HDR methods, for example, inverse tone mapping operators (iTMOs) [START_REF] Banterle | Inverse tone mapping[END_REF][START_REF] Banterle | A psychophysical evaluation of inverse tone mapping techniques[END_REF], where the dynamic range of LDR content is expanded for displaying on an HDR display. Most of the current approaches to enhance the perception of dynamic range are based on heuristics. Meylan et al. [START_REF] Meylan | The reproduction of specular highlights on high dynamic range displays[END_REF] showed that, when expanding the dynamic range of LDR content, specular highlights have to be allocated a significant range.

Similarly, other studies tend to boost the brightest pixels of the LDR scene in order to "approximate the visceral response associated with the higher contrast and overall brightness in the original scene" [START_REF] Rempel | LDR2HDR: On-the-fly reverse tone mapping of legacy video and photographs[END_REF][START_REF] Banterle | Inverse tone mapping[END_REF]. The results of the subjective experiment conducted in this paper provide a groundtruth for designing more perceptually meaningful dynamic range metrics.

Furthermore, as pointed out by Narwaria et al. [START_REF] Narwaria | An objective method for High Dynamic Range source content selection[END_REF], having a perceptually annotated HDR image dataset with MOS values is necessary for both HDR source content selection in subjective studies and for testing HDR processing algorithms. However, in that work the proposed objective measure is tailored to a dynamic range reduction task, such as in the case of tone mapping evaluation.

Moreover, no formal subjective evaluation is proposed to verify the perceptual relevance of that method. Predictive models, if successful, can serve the same role as annotated databases without requiring time-consuming data collection.

Motivation and Methodology

Computational metrics that predict PDR could allow for automatic organization, ordering and presentation of HDR content. This work's goal is to analyze how humans perceive dynamic range of HDR content on complex scenes, and be able to predict the PDR of a scene automatically. The research method employed involves data collection and analysis of HDR content from real-world HDR images. This data is compared with objective measurements; these are subsequently used to build a PDR model. This section describes the choice of objective metrics and describes the experimental methodology.

Objective Evaluation

The objective measurements proposed here are based on the insights of a previous pilot study [START_REF] Hulusic | Perceived dynamic range of HDR images[END_REF], on achromatic HDR images, which inspired the work presented here; two measures (pixel-based DR and image key) are retained from the study and another three (area, contrast and colorfulness) are included based on analysis of that study's results and previous findings on these topics [START_REF] Li | Relative area and relative luminance combine to anchor surface lightness values[END_REF][START_REF] Calabria | Perceived image contrast and observer preference i. the effects of lightness, chroma, and sharpness manipulations on contrast perception[END_REF][START_REF] Fairchild | Color appearance models[END_REF]. Three objective measures were used in the pilot study: dynamic range (DR), image key (IK) and spatial perceptual information (SI). After obtaining MOS scores, a significant correlation was found between the DR and MOS (Spearman's rank-order correlation coefficient r s = .788; p < .001) and IK and MOS (r s = -.671; p < .001), while the correlation between the SI and MOS

was not found to be significant (r s = .037; p = .830).

In addition, analysis of the results revealed that there might be a significant contribution to the perceived dynamic range based on the area of the brightest parts of the image. The images with relatively small light sources (e.g. Zentrum, WaffleHouse, LasVegasStore) had the highest difference between the perceived and pixel-based DR. This is in accordance with the discussions in the literature on image contrast and anchoring theories where the area and distance between the contrasting patches have been found to be the significant image features [START_REF] Fairchild | The HDR photographic survey[END_REF][START_REF] Gilchrist | An anchoring theory of lightness perception[END_REF]. In these studies, usually two types of white are discriminated: the diffuse, below a certain threshold; and specular or self-luminous, that represent values above the diffuse white threshold. A couple of studies on preferred diffuse range reported similar findings. The ITU document 6C/146-E, revealed that 90% of participants were satisfied with an upper limit of diffuse white being set to 2,400 cd/m 2 [START_REF] Itu-R | [END_REF]. In a second study, the same threshold (90% preference level) was been reported for three groups of participants with the following values:

4,677 cd/m 2 for technical, 3,090 cd/m 2 for arts and 1,995 cd/m 2 for naive participants [START_REF] Daly | 1: Distinguished paper: Viewer preferences for shadow, diffuse, specular, and emissive luminance limits of high dynamic range displays[END_REF]. Both experiments were conducted on a dual modulation HDR display system [START_REF] Seetzen | 2: A high dynamic range display using low and high resolution modulators[END_REF], with the ability to reproduce luminance levels in the range from 0.004 to 20,000 cd/m 2 . Seven images were evaluated, six of which were real-world structured stimuli, using 34 participants.

In addition, dynamic range is often confused with contrast in some contexts.

Nevertheless, it was necessary to investigate perceptual relation between the two attributes, by inspecting whether a predictor of local contrast can further explain DR perception. Finally, as a number of psychophysical phenomena, such as the Helmhotz-Kohlrausch effect, link color appearance to dynamic range and contrast, colorfulness has been selected as an objective measure for chromatic images. The main goal was to investigate whether an attribute such as DR, generally considered monochromatic, could be perceived differently when the chrominance changes even if the luminance is kept constant.

Objective Metrics

Following the results of the pilot study, DR and IK were maintained as metrics and were augmented by another three measures based on the above observations -the Area of specular regions, the Contrast and the Colorfulness.

The Area was motivated by the diffuse white thresholds; the Contrast as the magnitude of the global lightness difference is considered as a crucial factor in perception; and the Colorfulness since it is directly related to the perceived brightness of regions with constant luminance.

The pixel luminance values were first scaled to the display range with the following equation:

L = L -min(L) max(L) -min(L) • (Disp max -Disp min ) + Disp min , (1)
where Disp min = 0.03 cd/m 2 and Disp max = 4250 cd/m 2 in our setup. The DR is then calculated after excluding 1% of the darkest and brightest pixels in the image, using

DR = log 10 max(L ) min(L ) , ( 2
)
where L is the image with scaled values.

The image key IK ∈ [0, 1], a measure proposed by Aküz and Reinhard [START_REF] Akyüz | Color appearance in high-dynamic-range imaging[END_REF],

was defined as:

IK = ln(avg(L)) -ln(min(L)) ln(max(L)) -ln(min(L)) , (3)
where the avg(L) was computed as ln(avg(L)) = Σ ij ln(L(i, j) + δ)/N , with δ = 10 -5 to avoid singularities and N was the number of pixels. Once again, min(L) and max(L) were calculated robustly, after excluding 1% of the darkest and the brightest pixels.

200

The Area was calculated as:

Area = Σ ij (L(i, j)), L(i, j) > 2400 cd/m 2 , (4)
and represents the number of pixels greater than the diffuse white threshold value. The value of 2,400 cd/m 2 was selected as recommended in ITU document [START_REF] Itu-R | [END_REF], as discussed in Section 3.1. It is in accordance with the results of the study by Daly et al. [START_REF] Daly | 1: Distinguished paper: Viewer preferences for shadow, diffuse, specular, and emissive luminance limits of high dynamic range displays[END_REF], where different thresholds were found for different levels of user expertise. As participants in our study ranged from naive to tech-205 nical, but not necessarily in this particular domain, it was reasonable to select a value towards to middle of the interval.

The Contrast measure is an adapted version of the work by Peli [START_REF] Peli | Contrast in complex images[END_REF] and Rizzi et al. [START_REF] Rizzi | A proposal for contrast measure in digital images[END_REF]. Let L j be the image at level j of a Gaussian pyramid, of size N/2 j × M/2 j , obtained by decimating the original image with a Gaussian low-pass filter, and let N 8 (L j ) be the 8-neighborhood for a given pixel in L j .

In this study, three levels were used, j ∈ [START_REF] Dufaux | High Dynamic Range Video. From Acquisition, to Display and Applications[END_REF][START_REF] Banterle | Inverse tone mapping[END_REF], corresponding to 1.24, 2.47 and 4.95 cycle per degree (cpd), as the peaks of the contrast sensitivity function (CSF) for different light adaptation luminance lie approximately within that spatial frequency interval [START_REF] Kunkel | Perceptual design for high dynamic range systems[END_REF]. The cycles per degree were calculated based on the image dimensions, screen size and viewing distance. The contrast was computed as:

C = j∈[2,4] LC j 3 , (5)
where

LC j = |L j x,y -N8(L j x,y )| 8 L j W × L j H , (6)
uses perceptually uniform luminance values [START_REF] Aydın | Extending quality metrics to full dynamic range images[END_REF]. L j W and L j H are the width and the height of the image L at the j-th scale.

Experimental Method

The main aim of this study was to investigate the correlation between PDR and the five objective measures, for both achromatic and chromatic images, and, subsequently, to propose a model for calculating dynamic range based on these measures. Below, the design, participants, apparatus, stimuli and the experimental procedure are described.

Design

In the study, a subjective evaluation of PDR of both achromatic and chromatic images was conducted. The participants were asked to evaluate the overall impression of the difference between the brightest and the darkest part(s) in the images. The independent variable was the image content, while the dependent variable was the reported PDR of the image. The study was conducted in two separate sessions, one for achromatic and another for chromatic images; at least one day was allowed between sessions. The sessions were ordered randomly and equally.

Three possible evaluation methods are typically considered during the design of experiments of this type: paired comparison, ranking and rating. Paired comparisons were ruled out due to their impracticality with large data sets.

The efficient pair comparison techniques [START_REF] Silverstein | Efficient method for paired comparison[END_REF] can be used under certain assumptions. However, due to multidimensionality and non-deterministic DR appearance, these assumptions in our case were violated. While the ranking methods are straightforward, and quick to conduct, as with pairwise comparisons, they provide no information on the magnitude of the differences. Therefore, this method has been designed in order to use the advantages of the three methods: it permits ranking of the stimuli, a direct comparison between the image pairs, and it uses the continuous scale for subjective scores.

The evaluation method was inspired by the Subjective Assessment Methodology for Video Quality (SAMVIQ) [START_REF] Blin | SAMVIQ -Subjective assessment methodology for video quality[END_REF], adapted to static images. The data set consisted of 36 images, selected from the pool of 137 images, and divided into three subsets of 12 pseudo-randomly selected images in a randomized order (see section 3.3.4). The images thumbnails (422 × 238 px) were presented across a 3 × 4 grid, with the corresponding subjective scores, initially set to zero, below each image. The red color of the score indicated that the image had not been yet evaluated. All thumbnails were tone-mapped [START_REF] Mantiuk | Display adaptive tone mapping[END_REF] for two main reasons: the images were rather small, thus making it inappropriate for subjective evaluation; and to discourage ratings based on the thumbnail appearance only. The evaluation session was not time constrained. Each subset was evaluated independently, allowing participants to re-evaluate any image within, but not across subsets as many times as they wanted. The rating was performed on a 0-100 vertical continuous scale, divided into five equal intervals with corresponding labels: very low, low, medium, high and very high, included for general guidance.

Upon completion, all participants were interviewed, using a short structured questionnaire as described in Section 3.3.5.

Participants

24 participants volunteered for the study. All of them reported normal or corrected-to-normal visual acuity and were screened for color blindness using the Ishihara test prior to the session with chromatic images. One male participant (age 49) was found to be color blind before proceeding to the second session with chromatic images, and therefore his scores for the achromatic stimuli were discarded and not used in the analysis. From the remaining 23 participants 17 were male and 6 female, with the age ranging from 23 to 40 (with an average age of 29). 12 participants were assigned to the chromatic condition for their first session, while the other 11 first evaluated the achromatic images during their first session.

Apparatus

All the experiments took place in a dark and quiet room. The stimuli were displayed at full HD (1920 × 1080 pixels) on an HDR SIM2 HDR47ES4MB 47" screen that allows for displaying >90% of Rec 709 color gamut [41]. It was utilized in the DVI Plus (DVI+) mode, that allows for directly and independently controlling backlight LEDs and LCD pixel values, based on the dual-modulation algorithm [START_REF] Zerman | Effects of display rendering on HDR image quality assessment[END_REF]. The ambient illumination in the room was measured between the screen and participants at 2.154 lux. The luminance of the screen when turned off was 0.03 cd/m 2 . The distance from the screen was fixed to three heights of the display, with the eyes in the middle of the display, both horizontally and vertically.

Landscape Portrait

Stimuli

Initially, all the images from the HDR Photographic Survey [START_REF] Fairchild | The HDR photographic survey[END_REF] were considered. The landscape images were downsized and cropped to 1920 × 1080 size, while all the portrait images were first downsized to 1920 pixel width and then three images were cropped out: the top-aligned, the middle-aligned and the bottom-aligned one, see Figure 1. This resulted in a total of 131 landscape HD HDR images. After computing the DR, IK and SI [START_REF] Hulusic | Perceived dynamic range of HDR images[END_REF], a set of 33 images was selected for the study; selected to maintain objective measures evenly distributed across the set. The features used in this study show the same trend 2.

Since most of the images from the Fairchild's data set are photographs of nature, a single frame from the Market HDR video sequence proposed in MPEG by Technicolor [START_REF] Lasserre | Description of HDR sequences proposed by technicolor, ISO/IEC[END_REF] and a frame from both the Carousel and Bistro video sequences from the Stuttgart HDR Video Database [START_REF] Froehlich | Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays[END_REF] were added to the test data set, see Figure 3. All 36 images were converted to the corresponding achromatic images, using BT.709 primaries to compute relative luminance [START_REF] Itu-T | Parameter values for the HDTV standards for the studio and for international programme exchange[END_REF].

Procedure

Upon entering the experimentation room, the participants were first given the instructions to read and asked if they had any questions about the nature of the experiment and their task. This was followed by a training session, where the task was first explained using feature abstractions, inspired by the study by Aydin et al. [START_REF] Aydin | Automated aesthetic analysis of photographic images[END_REF], see Figure 4. They were told that in this example the magnitude of the feature increases from minimum in the first gray block to maximum in the last, black and white block. After this, the simulation of the experimental framework was displayed with three images with high, low and average pixelbased DR range on each screen. The first three images were evaluated by the experimenter explaining that they correspond to the top, bottom and the middle of the rating scale respectively, and the rest of the images by the participant in order so stabilize their opinion. None of the six training images were part of the test data set. In both the written and verbal instructions, they were asked to evaluate the overall impression of the difference between the brightest and the darkest part(s) of the images. If the participants demonstrated a full understanding of the experimental task, they were asked to fill in the basic information form and the corresponding session commenced.

The images in each subset were evaluated independently; once the participant selected the next subset the results of the previous one could not be changed. However, they could evaluate all the images in the current subset in any order and as many times as they needed. This allowed for multiple comparisons between the images and fine adjustments of the scores. In order to evaluate an image, the participant had to click on the thumbnail. The selected HDR image was displayed full screen for evaluation. Once the participant was ready to give a rating they clicked on the presented image and a rating scale was produced. The scale appeared in the far right side of the screen. After the score was given, the initial thumbnail preview of the current subset was redisplayed with the updated score for the evaluated image, see Figure 5. After the completion of the test, the experimenter had a short structured discussion about the test with all the participants. The questions used in the experiment were:

1. On a scale 1-10, how tiring was the experiment in terms of visual comfort

Thumbnail preview (updated score)

Fullscreen HDR view + Rating scale and fatigue?

Fullscreen HDR view Thumbnail preview

Rate and click on Submit

Click on the image

Click on a thumbnail

• Have you been bothered by some particular images? 2. On a scale 1-10, how difficult did you find it to evaluate the images?

• What did you find difficult? Why? Which (type of) images?

3. With how many ratings are you confident (you think your score is correct)? 4. Would you change anything in the experiment?

• If so, what? How? Why? 5. Is there anything else you would like to comment on? 6. Was it easier to evaluate grayscale or color images?2

Results

Results were analyzed with a series of statistical tests to form a better understanding of the captured data. This section presents results across the participants testing for group, image and chromatic conditions and correlations comparing MOSs with the objective measurements.

Overall Results Across Participants

The effect of the independent variables on the PDR scores provided by the participant was analyzed in a 2 (session) × 2 (color ) × 36 (scenes) factorial design. session is a between-participants variable reflecting the order of chromatic-achromatic presentation whereby 12 participants were first presented with the chromatic condition and the other 11 with the achromatic condition on their first session. color is a within-participants variable that corresponds to the chromatic and achromatic results and scenes is also a within-participants variable corresponding to the 36 images presented to the participants.

Results were analyzed using a mixed-model factorial ANOVA. The main effect of session was not significant F(1, 21) = 2.06 , p = 0.17. This indicates that the session ordering did not have a significant effect on the results.

The main effect of color was also insignificant, F(1, 21) = 0.40 , p = 0.40.

Again this indicates that there was no significant difference between the scores for the chromatic and achromatic set. Further analysis of the chromatic and achromatic results is given below.

The main effect of scenes was significant F(35, 735) = 98.73 , p < 0.01 as expected. Analysis of the differences and correlations with objective measurements will be given below.

In order to analyze the agreement across participant scores, Kendall's coefficient of concordance W was employed. W reports a value of 1 for absolute agreement across all judges and 0 for complete disagreement. For the achromatic condition, Kendall's coefficient of concordance was significant (W = 0.81, p < 0.01) and similarly for the chromatic condition (W = 0.76, p < 0.01). These results indicate a very strong agreement across participants' allocated scores.

Correlations of MOS

In order to analyze the correlations among the PDR values provided by the participants per scene, results were collapsed into a mean opinion score (MOS) per scene. The MOSs are correlated with the four (five for chromatic) objective measures presented in Section 3.2. The scatter plots for each of the five objective measures are provided in Figure 6 and Figure 7.

Since the calculated measures were on different scales, they were scaled using the equation:

x i = x i -1 n n i=1 x i max(X) -min(X) (7)
so that they are all represented with the same order of magnitude. The results of the correlation between the MOS values and objective measures can be seen in Table 2.

Initially, when comparing the Pearson's (r) and Spearman's (r s ) correlation coefficients, the only substantial difference was found for the Area measure

), an evident trend of the root function was noticeable for this measure. A number of root functions were tested in order to linearize the data and the best was found to be:

Area = 4 √ Area. ( 8
)
With the fitted data (Area ) the Pearson's correlation coefficient was almost identical to the Spearman's (see Table 2), which eventually resulted in a more robust predictor model, see Section 5.

The correlations between the chromatic MOS and achromatic MOS were r = 0.99 and r s = 0.98, both significant at p < 0.01, indicating the chromatic and achromatic scores were very highly correlated. 8, the extended box plot depicts the distribution of the perceived DR scores, with the corresponding mean and median values, confidence intervals and outliers. In addition, the pixel-based DR is also presented to visually display the correlation between the subjective and objective scores.

Post-experimental Inquiry

The comments obtained upon completion of each session of the study were fairly consistent across all the questions. The average score of the answer to the first question was 3.93, which means that there were instances which were slightly annoying to observe. Around 60% of participants reported that the brightest images with the visible sun or light sources were uncomfortable to look at.

The average score to the second question was 4.17. Several participants reported that the images with "very small but very bright portions of the image"

were more difficult to evaluate and that the area of the brightest parts (i.e. light sources, sun) could have an affect on the perception of DR. A number of participants mentioned that they think that the PDR varies depending on the proximity of the brightest and the darkest parts in the image. For example, if there is the sun in one corner of the image and a part of the scene in shadow in the other corner (e.g. OtterPoint) versus if there is the sun behind the tree (e.g. DevilsTower ).

It was reported that the user confidence for the given scores (question 3; i.e.

images that were relatively easy to evaluate and to which they think they gave an objective score) was 70.74%. The images constituting the remaining 29.26% correspond to the images with a medium DR, which was in accordance with the increased confidence interval for these images, see Figure 8.

Generally, all the participants liked the experimental design and there were no major complaints regarding the framework (question 4). One participant suggested that it might be better to have more than 12 images in a subset.

Initially, there were two subsets, each consisting of 20 images. However, the pilot study revealed that it might be too difficult to perform comparisons among that many images. Furthermore, ten images are used in the SAMVIQ methodology, upon which this framework had been developed.

In the general comments section (question 5), the participants reported that the possibility of re-evaluating images was very helpful and liked the fact that they could evaluate images in any order without time constraints. They also reported that the overall image brightness might be affecting the PDR.

The responses to the last question revealed that it was slightly easier to evaluate achromatic images (60.87% of participants) as opposed to chromatic (26.09%). 13.04% of participants reported that it was the same in terms of difficulty. Most of the participants from the first group (i.e. the 60.87% who found achromatic easier) reported that this could be due to the extra chromatic information which compounded the complexity of the choice.

These comments were a valuable input to the study and were considered when choosing the predictors in the regression model (see Section 5).

PDR Predictor Model

This section looks into producing a model that can predict PDR based on the results of the experiment. To achieve this, multivariate linear regression is employed and analyzed.

Creating the Model

The results from the pilot study [START_REF] Hulusic | Perceived dynamic range of HDR images[END_REF] and the previous section indicate that the pixel-based DR is a generally good predictor of the PDR, but there are cases where it fails (see Section 6), and that it could be potentially improved if other measures are considered. The results were also used as an indication of the measures that could be used as variables for predicting the outcome in the multiple regression analysis. In particular, a model is fit to the data and used to predict values of the outcome from several predictors.

Miles and Shelvin [46] suggest that when expecting a large effect size, a sample size of around 40 is sufficient for two to four predictors. This is close to our initial sample of 36 images. Prior to employing the regression, the data for the Area measure was linearized using the Equation 8, and all the measures were feature-scaled using Equation 7. The hierarchical method was utilized in order to see the behavior of the model with new predictors. Since pixel-based DR was known to be a good predictor of the perceived DR, it was selected for the first block in the hierarchy. The forced entry method was selected for DR. All other predictors (IK, Area', C and Col) were added to the second block. For this block, the forward method was used. This method calculates the contribution of each predictor by looking at the semi-partial correlation with the outcome. The model summary with the corresponding R, R 2 , adjusted R 2 and the significance of the F change values are presented in Table 3.

The R values show the multiple correlation coefficients between the predictors and the outcome. In the first case where there is only one predictor (DR), In the stepwise regression, at each iteration, the BetaIn value is estimated for each predictor that has not been included into the model, as if it were entered into equation at this stage. The standardized b-value, BetaIn, indicates the number of standard deviations that the outcome will change as a result of one standard deviation change in the predictor. Based on this value, the t-statistics for these values are computed and based on its significance the next predictor is entered, until there are no predictors with significant value less than 0.05, see Table 4.

The Model Parameters

After finding the independent variables that significantly improve the prediction of the outcome variable, the model coefficients, for the model using DR and Area' were calculated, see Table 5.

Therefore, the regression model for the achromatic images is defined as:

M DR ≈ 0.573DR + 0.448 4 √ Area (9)
while for the chromatic images it is: In Figure 9 the observed scores (MOS), pixel-based DR and MDR values are displayed for each scene. A significantly higher correlation between the MOS and MDR than between the MOS and pixel-based DR is evident. The cases with the highest discrepancy between the latter will be further discussed in Section 6.

M DR ≈ 0.506DR + 0.471 4 √ Area (10)
In order to verify whether the model fits the data or if it is skewed by a few extreme cases (outliers), the standardized residuals (SR), obtained by dividing the residual by an estimate of their standard deviation, were examined. In addition, Cook's distance (CD) was calculated in order to check for the influential cases [START_REF] Cook | Detection of influential observation in linear regression[END_REF]. This test analyzes whether the regression model is stable across the sample, by calculating the overall influence of a particular case on the model.

Based on the findings by Cook and Weisberg [START_REF] Cook | Residuals and influence in regression[END_REF], values greater than 1 may be cause for concern. Computing the residual statistics on a case-wise basis, the results revealed that there were no cases with the absolute value of the SR greater than 3.29. For three achromatic images: HancockKitchenInside, OCanadaNo-Lights bottom and SequoiaRemains top this value was -2.25, -2.441 and 2.238 respectively. In the case of chromatic images there was only HancockKitchenInside image for which the SR value was -2.748. Cook's distance (CD) for the three achromatic images was .170, .167 and .087 respectively, while for the Han-cockKitchenInside chromatic image this value was .253 respectively. Therefore, based on these results, the sample represents an accurate model.

Multicollinearity between predictors makes it difficult to assess the individual contribution of a predictor. If the two predictors are highly correlated, there is a high possibility that each accounts for similar variance in the outcome. Multicollinearity was tested by using the variance inflation factor (VIF), see Table 5. As argued by Myers, a value greater than 10 indicates a potential problem [START_REF] Myers | Classical and modern regression with applications[END_REF]. In addition, values less than 0.2 indicate a potential multicollinearity and should be further investigated [START_REF] Menard | Applied logistic regression analysissage university[END_REF]. As VIF values for both achromatic and chromatic models were 1.859, there is no indication of multicollinearity between the predictors in the models.

Calculating Model Generalizability

The R 2 values show how much of the variability in the MOS is accounted for by the predictors. For the achromatic images using only DR as a predictor R 2 = .75, while for the same model with chromatic images R 2 = .704, which means that DR accounts for 75% and 70.4% of total variation of the MOS respectively. Adding Area' as the second predictor, this percentage rises to 89.3% for achromatic, and 86.8% for chromatic. 3 indicate a very good cross-validity of the model.

Calculating the change statistics, the significance of the change in R 2 by adding new predictors can be calculated. This is usually done by calculating the F-ratio by using the following equation:

F = (N -k -1)R 2 k(1 -R 2 ) ( 11
)
where N is the number of cases, and k is the number of predictors in the model.

In Table 3, a significance of the F change is provided. In both the achromatic and chromatic image models, there was a significant change in R 2 value for addition of the Area' (p < .001).

Table 6: Eight achromatic (top) and eight chromatic (bottom) scenes with the highest difference between the MOS and pixel-based DR, along with the differences between the MOS scores and predicted perceived dynamic range by the proposed model (MDR). The scores are calculated as the absolute difference between the feature-scaled values obtained using Equation 7), and multiplied by 100 for better readability. These results demonstrate that in 14 out of the 16 cases the model predicts the PDR better than the pixel-based DR measure. Table 7 shows the correlations between the MOS scores and both pixel-based DR and MDR, performed on these subsets. For both achromatic and chromatic scenes, the correlation between the MOS and DR is not significant (p > 0.05). Nevertheless, the correlation between the MOS and predicted perceived dynamic range (MDR) is significant in all cases. These chosen cases demonstrate why the predictive method can be considered superior to the DR method when predicting PDR. While traditional dynamic range measures can be an acceptable predictor of PDR, they can also be inaccurate in some cases. In order to develop a generic model that can truthfully predict PDR, sensory and cognitive processes involved in extraction of this attribute from complex stimuli need to be taken into account. Previous research has shown that the area of the brightest patches and the image topology affect the perception of lightness and contrast [START_REF] Li | Relative area and relative luminance combine to anchor surface lightness values[END_REF][START_REF] Provenzi | Mathematical definition and analysis of the retinex algorithm[END_REF][START_REF] Vangorp | A model of local adaptation[END_REF], and therefore should be considered in constructing such a PDR model.

Scene

Furthermore, image contrast and colorfulness are important factors in visual perception and, based on previous findings [START_REF] Aydin | Automated aesthetic analysis of photographic images[END_REF][START_REF] Fairchild | Color appearance models[END_REF][START_REF] Calabria | Perceived image contrast and observer preference i. the effects of lightness, chroma, and sharpness manipulations on contrast perception[END_REF][START_REF] Calabria | Perceived image contrast and observer preference ii. empirical modeling of perceived image contrast and observer preference data[END_REF], could be involved in the process related to the extraction of observed image attributes.

In this study a new experiment for subjective evaluation of perceived dynamic range in HDR images has been designed and conducted. The results were used to generate a subjectively annotated data set of 36 HDR images, with both MOS scores for achromatic and chromatic data. This data set can be used in future HDR content studies on algorithm or metric validation and aesthetic attribute analysis and modeling. This is, to the best of the authors' knowledge, the first study on perceived DR using complex stimuli and HDR conditions.

The results of the Kendall's coefficient of concordance show that there was a high agreement between participants' scores in both sessions. Using the mixed model factorial ANOVA the effect of scenes was found as significant, while neither the session order nor chromaticity of the data were found to produce significant differences in the results. In addition, the correlation between the chromatic and achromatic MOSs was computed, and the results demonstrated very high correlations. Furthermore, the correlations between the PDR and five objective measures were computed. All correlations, except for the contrast (C), were found as significant for both achromatic and chromatic images, with very small discrepancies between the two conditions.

Models that can predict PDR of both achromatic and chromatic images were generated using multivariate linear regression. The regression model revealed that only pixel-based DR and linearized Area measures had significant contribution to the predictor model. Cross-validation of the model showed that the model could accurately predict the outcome in a different sample, i.e. the model can be generalized.

Finally, eight scenes with the highest discrepancies between the MOS and pixel-based DR values were selected from both achromatic and chromatic sets.

The statistics showed that the PDR prediction was significantly improved when the Area predictor contributed to the model.

Although the results show that, in the overall, the PDR prediction with the proposed model is closer to the MOS it is likely that there will be images where this is not the case due to the excessive complexity of the HVS and the related processes in the perception of such visual attributes. Therefore, in the future, we would like to further investigate this topic by looking at these and other perceptual factors that could be involved in this process at a lower level.

Furthermore, the existing objective metrics have to be redesigned and, possibly, novel ones developed targeted directly at HDR content. While this is beyond the scope of this paper, it was evident that a gap exists in this area. Another direction at which we would like to expand this work is the analysis of aesthetic attributes. Finally, we are interested in extending this work to video content and investigating the temporal aspects. In the case of video, the perceptual phenomena behind the perception of dynamic range can be more complex. While, for a static image, the luminance range reproducible by an HDR display matches the steady-state dynamic range of the HVS, temporal variations of this range, e.g., due to a change from a bright to a dark scene, can span a much broader interval of luminance than the HVS could process at a given adaptation level, causing maladaptation phenomena and visual discomfort [START_REF] Kunkel | Perceptual design for high dynamic range systems[END_REF]. It is known that light/dark adaptation is not instantaneous, which results in higher masking for larger temporal variations of the luminance range. This entails a loss of contrast sensitivity in the maladaptation phase, but could enhance the overall perception of bright-dark differences on short time segments. Therefore, initial studies will be conducted with a similar methodology, using short clips, and the scores will be correlated with dynamic range models similar to those discussed in this work.

Figure 1 :

 1 Figure 1: Preprocessing of the two images from the Fairchild's data set: the landscape image (left) was first downsized from 4288×2848 to 1920×1275, and then cropped to 1920×1080 pixels; the portrait image (right) was first downsized from 2704×4015 to 1920×2851 pixels and the three HD images were cropped -top, middle and bottom.

Figure 2 :Figure 3 :

 23 Figure 2: Image features: pixel-based DR (top-left), IK (top-right), Area (middle-left), Contrast (middle-right) and Colorfulness (bottom), all sorted by pixel-based DR.

  In order to reproduce an image on the display, the luminance values should not exceed the range of the display. Values in excess of the maximum display brightness, for both the chromatic and achromatic images, were first clipped to the diplay's peak luminance value of 4250 cd/m 2 , and the images were then processed using the algorithm developed by Zerman et al.[START_REF] Zerman | Effects of display rendering on HDR image quality assessment[END_REF] and displayed as RGB images in DVI+ mode.From the set of 36 test images, six images with the lowest and six with the highest pixel-based DR were selected in two new subsets: imagesLow and imagesHigh. When generating each session for the experiments, each subset of 12 images was composed of two randomly selected images from the imagesLow subset, two from imagesHigh and the rest from the remaining 24 images. This preserved the consistency of measures among the subsets.Your task is to evaluate the overall impression of the difference between the darkest and the brightest part(s) of the presented images.

Figure 4 :

 4 Figure 4: The abstraction of the attribute to be evaluated.

Figure 5 :

 5 Figure 5: The illustration of the test procedure.

(r = 0 .

 0 64 and r s = 0.87 for achromatic, and r = 0.64 and r s = 0.88 for chromatic stimuli). By observing the correlation scatter plots (Figures 6 and 20

Figure 6 : 21 DynamicFigure 7 :

 6217 Figure 6: Achromatic MOS across the four objective measures (top-left to bottom-right): pixel-based DR, IK, Area and Contrast.

Finally, for assessing

  how well the sample represents the entire population, that is how accurately the model can predict the outcome in a different sample, a cross-validation of the model was performed. If the prediction on another sample is similarly correct, then the model can be generalized. Cross-validation was calculated by adjusting the R 2 values by estimating how the R 2 values were derived from the population from which the sample was taken. The adjusted R 2 values give an indication of how well the model generalizes. The closer the values to the R 2 ones are, the better the prediction from the sample is. For example, the difference between the R 2 and the adjusted R 2 values for the achromatic model with two predictors is 0.893 -0.887 = 0.006, which means that if the model was derived from the population, instead of from the sample, it would account for 0.6% less variance in the MOS. The values presented in Table

Table 1 :

 1 A list of abbreviations and terms used in the paper.

	Term	Meaning
	PDR	Perceived dynamic range
	MDR	Dynamic range as predicted by the proposed regression model
	HVS	Human visual system
	iTMO	Inverse tone mapping operator
	MOS	Mean opinion score
	CSF	Contrast sensitivity function
	IK	Image key
	SI	Spatial information
	C	Contrast
	Col	Colorfulness
	F(x,y)	F-ratio (test statistic used in ANOVA)
	p	The probability value
	r	Pearson's correlation coefficient
	rs	Spearman's rank correlation coefficient
	W	Kendall's coefficient of concordance
	R	The multiple correlation coefficient
	R 2	The coefficient of determination
	adj. R 2	The adjusted R 2 coefficient
	B	The regression coefficient (unstandardized)
	Beta	Standardized regression coefficient
	SR	Standardized residuals
	CD	Cook's distance
	VIF	Variance inflation factor

Table 2 :

 2

	MOS		Achromatic			Chromatic
	Measure DR	IK	Area' C	DR	IK	Area' C	Col
	r	0.87* -0.60* 0.87* -0.19 0.84* -0.61* 0.87* -0.22 -0.47*
	r s	0.87* -0.55* 0.89* -0.24 0.84* -0.57* 0.90* -0.27 -0.43*
	In Figure

Pearson's r and Spearman's rs correlation coefficients between MOS values for both achromatic and chromatic images and five objective measures: DR, IK, Area, C and Col. * denotes significance at p < 0.01.

Table 3 :

 3 The summary of the model with R, R 2 , adjusted R 2 and F change significance values for both achromatic and chromatic images, presented across the hierarchies. simple correlation between MOS and this measure. In case of achromatic images, the R value increases from .866 with DR only, to .945 with DR and Area' measures. Similar R values can be observed for the chromatic images (R = .839 for DR and R = .932 for DR and Area' measures).

			Achromatic			Chromatic
	Model	R	R 2	adj. R 2	sig. F change	R	R 2	adj. R 2	sig. F change
	DR	.866 .750 .743	.000 .839 .704 .695	.000
	DR, Area' .945 .893 .887	.000 .932 .868 .860	.000
	this is a

Table 4 :

 4 Estimated BetaIn value, t-statistics and its significance for each predictor. For both achromatic and chromatic models the Area' predictor has the highest (significant) t value, and was therefore included in the model. In the following iteration none of the remaining variables (IK, C and Col for the chromatic model) had significant t value, and thus were not included into the model.

		Achromatic		Chromatic
	Model BetaIn	t	Sig. BetaIn	t	Sig.
	IK	-.137 -1.303 .202	-.181 -1.610 .117
	Area'	.516	6.657 .000	.553	6.418 .000
	C	-.113 -1.325 .194	-.150 -1.640 .110
	Col				-.219 -2.362 .024
	IK	.069	.889 .381	.031	.361 .720
	C	-.049 -0.838 .408	-.082 -1.291 .206
	Col				-.116 -1.725 .094

Table 5 :

 5 Model parameters for the achromatic and chromatic images.

Table 7 :

 7 The Pearson's r and Spearman's rs correlation coefficients between MOS values and both pixel-based DR and MDR for achromatic and chromatic scenes. *Correlation significant at p < 0.01; **Correlation significant at p < 0.05.

	MOS	Achromatic	Chromatic
	Measure DR	MDR	DR	MDR
	r	0.584 0.912*	0.354 0.881*
	r s	0.429 0.833** 0.357 0.762**
	7. Conclusions and Future Work

This question was asked only upon completion of the second session.

Acknowledgments

The authors would like to thank Emin Zerman for the help with the DVI+ content generation and with development of the test framework. We would like to thank all the participants that volunteered in the subjective study. This work was supported by Region Ile de France, in the framework of the FUI 4EVER2 project. Debattista is partially supported by a Royal Society Industrial Fellowship.

ElCapitan_b

OCanadaNoLights_b

BloomingGorse ( 7. There is a high correlation between the MOS and MDR, and significant deviations in some cases of traditional DR computation. The scores are sorted by the MOS value.

Chromatic images

Analysis of High Discrepancy Scenes

As was shown in Section 4, a straightforward metric such as pixel-based DR can be a good predictor of dynamic range for many of the images. However, there are some particular cases where it fails to truthfully predict the PDR. In this section the scenes with highest discrepancies between pixel-based DR and 540 MOS values are investigated, for both the achromatic and chromatic images.

Two subsets of eight scenes with the highest discrepancies were selected, see Figure 10. The values obtained by taking the absolute value of the subtracted feature-scaled scores are presented in Table 6.