E Bosc

M Köppel

R Pépion

M Pressigout

L Morin

P Ndjiki-Nya

P Le Callet

CAN 3D SYNTHESIZED VIEWS BE RELIABLY ASSESSED THROUGH USUAL SUBJECTIVE AND OBJECTIVE EVALUATION PROTOCOLS?

Keywords: Virtual view synthesis, multi-view video, 3DTV, quality assessment, quality metrics

This paper addresses the problem of evaluating virtual view synthesized images in the multi-view video context. As a matter of fact, view synthesis brings new types of distortion. The question refers to the ability of the traditional used objective metrics to assess synthesized views quality, considering the new types of artifacts. The experiments conducted to determine their reliability consist in assessing seven different view synthesis algorithms. Subjective and objective measurements have been performed. Results show that the most commonly used objective metrics can be far from human judgment depending on the artifact to deal with.

INTRODUCTION

Recently, multi-view video processing has gained a growing interest. 3D video refers to two main applications: 3D television (3DTV), that provides a depth feeling, and Free Viewpoint Video (FVV), that allows navigation inside the scene [START_REF] Smolic | 3D video and free viewpoint VideoTechnologies, applications and MPEG standards[END_REF]. These emerging applications make the problem of evaluating the 3D visual experience a huge subject of investigation. As pointed out in [START_REF] Seuntiens | Visual Experience of 3D TV[END_REF], the assessed factors are more numerous than in traditional 2D video: image quality, visual comfort and depth are to be taken into consideration. This paper focuses on image quality.

Multi-view video plus depth (MVD) data [START_REF] Merkle | Multi-view video plus depth representation and coding[END_REF] can be used to offer 3DTV or FTV. MVD data consist of two types of videos: a first set of conventional video sequences acquired from the same scene at slightly different viewpoints, referred as "texture data"; a second set of associated depth video sequences, referred as "depth data". Depth data provide information on scene geometry and help in virtual intermediate view generation. When targeting either 3DTV or a FTV application, virtual view generation is very likely to be required. Indeed, the appreciation of a 3D content relies on the stereopsis phenomenon: an observer needs to be presented a pair of stereoscopic images with a strong binocular disparity. Human brain is then able to fuse the pair of images and to interpret the 3D scene. Thus, 3DTV displays should provide the appropriate stereoscopic pairs to ensure the immersion feeling. On the other hand, for FTV applications, a user may wish to navigate around the scene, which makes virtual view synthesis generation essential. Finally, depending on the available bandwidth or on the decoder, all the acquired video sequences may not be available. In this case, virtual view generation is also needed. Considering the users demand for acceptable image quality as a minimum, the quality of reconstruction of virtual views cannot be ignored. [START_REF] Meesters | A survey of perceptual evaluations and requirements of three dimensional TV[END_REF]. Among them, the keystone effect that makes the image look like a trapezoid; the ghosting effect that is a shadow-like artifact; the cardboard effect when depth is perceived as unnatural, as discrete incoherent planes. Synthesis errors can be added to this list as projection errors can occur. These new types of artifacts have to be taken into consideration when evaluating synthesized views.

Many new distortions have been listed in

However, up to now, there is no dedicated assessment framework, nor objective metric for 3D video quality evaluation. [START_REF] Tikanmaki | Quality assessment of 3D video in rate allocation experiments[END_REF] addressed the problem of measuring the quality of a synthesized view from encoded color and depth video. When trying to determine the optimal ratio between color and depth data in the context of 3D video compression, the authors observed that PSNR (Peak Signal to Noise ratio) and VSSIM (Video Structural SIMilarity index) led to different conclusions, regarding the compression choices. PSNR seemed unstable depending on the direction of the targeted virtual viewpoint. Though, the subjective scores correlated the VSSIM scores. Recently, [START_REF] El-Yamany | Evaluation of depth compression and view synthesis distortions in multiview-video-plus-depth coding systems[END_REF] reconsidered the synthesis quality evaluation framework. The authors showed the importance of the chosen reference for synthesis quality evaluation: they pointed out the fact that depending on the chosen reference (original view or control synthesis, i.e. image synthesized from uncompressed data), PSNR scores do not measure the same distortion. They showed that distortions from compression may be masked by distortion from synthesis process. Consequently, they recommend to use the control synthesis as a reference when assessing a codec performances. Peak Signal to Perceptual Noise Ratio (PSPNR) [START_REF] Zhao | Perceptual measurement for evaluating quality of view synthesis[END_REF] derived from PSNR, is the metric used by the 3D Video (3DV) group of MPEG. In [START_REF] Ekmekcioglu | Depth based perceptual quality assessment for synthesized camera viewpoints[END_REF], the authors proposed a new full reference metric that takes into consideration depth data and consequently the regions that are more likely to be distorted in the synthesis. The new framework is validated by its high correlation score with the perceptual-like metric Video Quality Metric (VQM) [START_REF] Pinson | A new standardized method for objectively measuring video quality[END_REF] results, and subjective assessments from 15 nonexpert observers.

This paper investigates the reliability of different objective metrics on still synthesized images as still images can be a plausible case for FTV. The test objective metrics are pixelbased as well as perceptual-like metrics. The images are synthesized with seven different Depth Image Based Rendering (DIBR) algorithms. Subjective assessments allow to evaluate the correlation between human perception and objective measurements.

ALGORITHMS

In this section different depth-image-based rendering (DIBR) methods are presented. DIBR defines the process of synthesizing "virtual" views at a slightly different viewing perspective using an image or video and the associated per pixel depth information. A critical problem in DIBR is that regions occluded in the original view may become visible in the "virtual" view, an event also referred to as disocclusion. In the absence of original image data two extrapolation paradigms address this inherent problem: 1) One can preprocess the depth information in a manner that no disocclusion occur in the "virtual" view, or 2) replace the missing image areas (holes) with known suitable image information. In the following, a short overview will be given on relevant work in disocclusion handling in 3D video.

Fehn preprocesses the per pixel depth information with a 2D Gaussian low-pass filter [START_REF] Fehn | Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV[END_REF]. This way large discontinuities are smoothed out in the depth map and dissoclusions do not appear in the "virtual" image. However, this leads to geometric distortions in the virtual view. Larger baselines yielding to more disturbing artifacts. In a rectified camera setup this method fails to close holes on the left or right border image. Therefore, these areas are treated in two different ways. Either the border is cropped and the image is resized to the original size or the holes on the border are inpainted with [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF]. These methods are referred to as A1 and A2 respectively in the rest of the paper. The cropping method is suitable for a stereo video where one view only is transmitted and the other one is rendered at the decoder side. In multi-view scenarios, this method is not applicable because all views, the original as well as the virtual views, have to be cropped to preserve the stereo impression. This would lead to image information losses in all views.

Tanimoto et al. [START_REF] Mori | View generation with 3D warping using depth information for FTV[END_REF] proposed a 3D view generation system. They are using an inpainting method [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF] to fill missing parts in the "virtual" image. This algorithm is adopted as the reference software for MPEG standardization experiments in the 3D Video group. This method is referred to as A3 in the rest of the paper.

Müller et al. [START_REF] Müller | View synthesis for advanced 3D video systems[END_REF] proposed an hole filling method embedded in a 3D video system. Holes are filled linewise with neighboring background information. The corresponding depth values at the hole boundary are examined row-wise to find background color samples to copy into the hole. This color extrapolation of the suitable background pixel leads to better results than a simple linear interpolation. Generally, due to depth estimation, some boundary background pixels in fact belong to foreground objects. Thus their color information would lead to foreground color propagation into the hole. This method is referred to as A4 in the rest of the paper.

In texture synthesis methods the unknown regions are synthesized by copying content from the known parts of the image to the missing regions. Ndjiki-Nya et al. [START_REF] Ndjiki-Nya | Depth image based rendering with advanced texture synthesis[END_REF] proposed a hole filling approach for DIBR systems based on patch-based texture synthesis. Holes with small spatial extend are closed by solving Laplacian equations. Larger holes are initialized by median filtering and then optimized via texture synthesis. This method is referred to as A5 in the rest of the paper.

Köppel et al. [START_REF] Köppel | Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering[END_REF] extended the A5 approach by a background sprite. The sprite stores valuable background image information and is updated frame-wise. Using the original and synthesized image information from previous frames temporally consistency is achieved in a sequence. This method is referred to as A6 in the rest of the paper. In the conducted subjective tests only images are analyzed. Thus, the capabilities of the approach to achieve temporal consistency in a sequence is not investigated. Algorithms A2-A6 support multi-view scenarios.

Non-filled sequences (i.e. with holes) are referred to as A7 in the rest of the paper.

EXPERIMENTAL PROTOCOL

The experiments have two main objectives: first to determine the tested algorithms performances and second, to assess the reliability of objective metrics for 3D images. Three test sequences have been used to generate four different viewpoints, that is to say twelve synthesized sequences for each test algorithm (84 synthesized sequences in total): Book Arrival (1024×768, 16 cameras with 6.5cm spacing), Lovebird1 (1024×768, 12 cameras with 3.5cm spacing) and Newspaper (1024×768, 9 cameras with 5cm spacing). Altogether 43 naive observers participated in the subjective assessment session. The session was conducted in an ITU conforming test environment. Absolute categorical rating (ACR) [START_REF]ITU-T p.910 subjective video quality assessment methods for multimedia applications[END_REF] was used to collect perceived quality scores: stimuli are presented in a random order and are evaluated through a coarse resolution rating scale. Observers notes are then averaged, which is called MOS (Mean Opinion Score). The stimuli were displayed on a TVLogic LVM401W, and according to ITU-T BT.500 [START_REF] Bt | Methodology for the subjective assessment of the quality of television pictures[END_REF]. Considering the large size of the tested database, only key frames of the rendered sequences were presented to the observers, as still images can also be a plausible scenario for FTV. Key frames were also evaluated through different objective metrics through MeTriX MuX Visual Quality Assessment Package [START_REF]MetriX MuX page[END_REF]. For both objective metrics, the reference was the original acquired image.

RESULTS

Subjective ratings are illustrated on Figure 1. Algorithms are ordered by MOS ratings. For a given algorithm its rank varies depending on the data set. This suggests that the algorithms performances depend on the inner sequences properties (i.e. the depth range, the camera acquisition parameters).

On Figure 2, subjective scores are plotted over objective scores in order to find a correlation. Let d be the camera baseline of a sequence. Top graph shows the performances when synthesizing with large baseline between reference and target view (2 × d), and bottom graph corresponds to a shorter baseline (d). As expected, it is observed that the shorter the baseline, the higher the scores (for objective as subjective measurements). For a view synthesized with short baseline, it can be observed that two PSNR scores varying from 20dB to 28dB (A1 and A4), MOS score remain nearly constant (from 2.6 to 2.5). As well, a variation of 2dB (A1 and A7) corresponds to about 1 MOS point. This is significant considering the coarse scale of MOS scores (from 1 to 5).

Statistical analyses have been conducted over the objective and subjective measurements. In order to determine whether classes of algorithms could emerge, a Student's ttest has been performed over the MOS scores for each test algorithm: on Table 1, statistically dependent pairs can be distinguished. It clearly indicates the statistical divergence of three algorithms: A7, A3 and A1 distributions differ from the other algorithms'. A7 and A3 count no statistically dependent pair, and A1 counts only one. In addition, Table 1 also indicates the required minimum number of observers that allows the statistical distinction (values in bold are higher than 24). It shows that the final ranking is obtained when 32 observers participate (VQEG recommends 24 observers).

The test with metrics other than PSNR led to nearly the same observations regarding the algorithms performances and their correlation with MOS scores. Besides, Table 2 confirms that all metrics are very correlated, even pixel-based ones compared to perceptual-based ones. Note that perceptuallike SSIM is very correlated to pixel-based PSNR (83.9%). Table 3 expresses the correlation coefficients between objective metrics and MOS scores, for the whole fitted measured points. It can be observed that the metrics closest to human judgment are WSNR (Weighted Signal-to-Noise Ratio), PSNR and NQM (Noise Quality Measure), (42.3%, 38.6% and 38.6% respectively). WSNR is a CSF-based weighting function and PSNR and NQM are pixel-based metrics. These metrics are also highly correlated according to Table 2. However, Figure 2 reveals the inconsistency between MOS and Fig. 1. MOS scores for the different sequences.

A1 A2 A3 A4 A5 A6 A7 A1 ↑(32) ↑(<24) ↑(32) o (>43) ↑(30) ↑(<24) A2 ↓(32) ↑(<24) o (>43) o (>43)o (>43) ↑(<24) A3↓(<24)↓(<24) ↓(<24) ↓(<24)↓(<24) ↑(<24) A4 ↓(32) o(>43) ↑(<24) o(>43) o(>43) ↑(<24) A5 o(>43) o(>43) ↑(<24) o(>43) ↑(28) ↑(<24) A6 ↓(30) o(>43) ↑(<24) o (>43) ↓(28) ↑(<24) A7↓(<24)↓(<24)↓(<24)↓ (<24)↓(<24)↓(<24)
Table 1. Results of Student's t-test. Legend:↑: superior, ↓: inferior, o: statistically equivalent. Reading: Line"1" is statistically superior to column "2". Distinction is stable when "32" observers participate.

PSNR. And indeed, the algorithms rankings according to each metric, listed on Table 4, show a very important difference between human scores and metrics for A1 algorithm. It is ranked as the best of this set by humans but worst by the metrics. A6 generates the best objective results but subjective evaluations assign its quality as not as good. A5 generates coherent objective and subjective results. For A5, A2, A3 and A4 the results of objective metrics correspond with human scores. This suggests that the reliability of the objective metrics differ depending on the rendering algorithm used, i.e. on the induced artifact. Algorithms can induce nonperceptible or non-annoying artifacts. Then, this implies that commonly used metrics are not suited for assessing virtual synthesized views as they inflict serious costs to relatively acceptable degradations. These results point out the need for a new 3D-adapted metric. Table 4. Rankings according to measurements.

CONCLUSION

This paper addresses the issue of evaluating virtual synthesized views with the traditional objective metrics. The assessments of the seven test algorithms by objective measurements and subjective ratings show that among all tested objective metrics, WSNR and pixel-based PSNR and NQM are the most correlated with perceptual evaluation provided by MOS scores. However, the results also show PSNR's inability to predict human experience. New methods are then required for assessing virtual synthesized views as pixel-based and perceptual-based metrics fail. Depth should be taken into account in such a metric as recently proposed in [START_REF] Ekmekcioglu | Depth based perceptual quality assessment for synthesized camera viewpoints[END_REF], because view synthesis produces geometric distorsions. Registration process according to the original view coupled with weighted critical areas could be investigated in future work to build a new metric. In addition, paired comparisons experiments should be hold on still images and video sequences in the future to refine the presented results.

Fig. 2 .

 2 Fig. 2. Correlation between MOS and PSNR according to the baseline distance between reference and target view.

Table 2 .

 2 Correlation coefficients between objective metrics in percentage.

Table 3 .

 3 Correlation coefficients between subjective and objective scores in percentage.

		A1	A2	A3	A4	A5	A6	A7
	MOS	2.388	2.234	1.994	2.250	2.345	2.169	1.126
	Rank order	1	4	6	3	2	5	7
	PSNR	18.752 24.998 23.180 26.117 26.171 26.177 20.307
	Rank order	7	4	5	3	2	1	6
	SSIM	0.638	0.843	0.786	0.859	0.859	0.858	0.821
	Rank order	7	4	6	1	1	3	5
	MSSIM	0.648	0.932	0.826	0.950	0.949	0.949	0.883
	Rank order	7	4	6	1	2	2	5
	VSNR	13.135 20.530 18.901 22.004 22.247 22.195 21.055
	Rank order	7	5	6	3	1	2	4
	VIF	0.124	0.394	0.314	0.425	0.425	0.426	0.397
	Rank order	7	5	6	2	2	1	4
	VIFP	0.147	0.416	0.344	0.448	0.448	0.448	0.420
	Rank order	7	5	6	1	1	1	4
	UQI	0.237	0.556	0.474	0.577	0.576	0.577	0.558
	Rank order	7	5	6	1	3	1	4
	IFC	0.757	2.420	1.959	2.587	2.586	2.591	2.423
	Rank order	7	5	6	2	3	1	4
	NQM	8.713 16.334 13.645 17.074 17.198 17.201 10.291
	Rank order	7	4	5	3	2	1	6
	WSNR	13.817 20.593 18.517 21.597 21.697 21.716 15.588
	Rank order	7	4	5	3	2	1	6
	SNR	12.848 19.094 17.276 20.213 20.267 20.274 14.403
	Rank order	7	4	5	3	2	1	6
	PSNR hsvm 13.772 19.959 18.362 21.428 21.458 21.491 15.714
	Rank order	7	4	5	3	2	1	6
	PSNR hsv	13.530 19.512 17.953 20.938 20.958 20.987 15.407
	Rank order	7	4	5	3	2	1	6

ACKNOWLEDGMENTS

This work is partly supported by the French ANR-PERSEE project n o ANR-09-BLAN-0170, and ANR-CAIMAN project n o ANR-08-VERS-002.


Student Member, IEEE Emilie Bosc

Romuald Pépion

Member, IEEE Patrick Le Callet

Martin Köppel

Patrick Ndjiki-Nya

Muriel Pressigout

Luce Morin

Towards a New Quality Metric for 3-D Synthesized View Assessment

Keywords: image-based rendering (DIBR), multiview, quality assessment, quality metrics, video plus depth, view synthesis

3DTV technology has brought out new challenges such as the question of synthesized view evaluation. Synthesized views are generated through a depth image-based rendering (DIBR) process. This process induces new types of artifacts whose impact on visual quality has to be identified considering various contexts of use. While visual quality assessment has been the subject of many studies in the last twenty years, there are still some unanswered questions regarding new technological improvement. DIBR brings new challenges mainly because it deals with geometric distortions. This paper considers the DIBR-based synthesized view evaluation problem. Different experiments have been carried out. They question the protocols of subjective assessment and the reliability of the objective quality metrics in the context of 3DTV, in these specific conditions (DIBR-based synthesized views). They consist in assessing seven different view synthesis algorithms through subjective and objective measurements. Results show that usual metrics are not sufficient for assessing 3-D synthesized views, since they do not correctly render human judgment. Synthesized views contain specific artifacts located around the disoccluded areas, but usual metrics seem to be unable to express the degree of annoyance perceived in the whole image. The study provides hints for a new objective measure. Two approaches are proposed: the first one is based on the analysis of the shifts of the contours of the synthesized view; the second one is based on the computation of a mean SSIM score of the disoccluded areas.

T HE emerging 3-D video applications have encouraged numerous investigations in various fields from video acquisition to display technologies. These applications are known as 3-D television (3DTV) and free viewpoint video (FVV) [START_REF] Smolic | 3-D video and free viewpoint Video-Technologies, appli-PSNRSSIM MSSIMVSNRVIF VIFPUQI IFC[END_REF]. 3DTV provides a depth feeling thanks to an appropriate 3-D display. FVV interactively allows the user to control the viewpoint in the scene. Considering the demand for high-quality visual content, the success of 3-D video applications is closely related to its ability to provide viewers with a high quality level of visual experience.

A reliable assessment protocol for synthesized views is urgently needed. The brand-new 3-D applications bring out new elements but, so far, their complexity may have been under-estimated. Thanks to recent studies that brought this complexity into focus, it is now more understandable. Seuntiens [START_REF] Seuntiens | Visual experience of 3D TV[END_REF] reported that not only image quality should be taken into account for an assessment framework, but visual comfort and depth feeling as well. These last aspects are still under investigation ( [START_REF] Banks | Consequences of incorrect focus cues in stereo displays[END_REF], [START_REF] Tam | Human stereoscopic vision: Research applications for 3D-TV[END_REF]). Concerning image quality, new distortion types have appeared, as discussed in [START_REF] Meesters | A survey of perceptual evaluations and requirements of three dimensional TV[END_REF], [START_REF] Boev | Stereoscopic artifacts on portable auto-stereoscopic displays: what matters[END_REF], [START_REF] Cancellaro | Stereo video artifacts introduced by a distributed coding approach[END_REF]. Some of them are related to the synthesis process, others are related to compression methods or display technologies. A few of them are:

• keystone effect : the image looks like a trapezoid;

• ghosting effect: this is a shadow-like artifact;

• cardboard effect: depth is perceived as unnatural, as discrete incoherent planes; • puppet-theater effect: characters in the scene appear miniaturized; • staircase effect: discontinuities appear between adjacent blocks.

El-Yamany et al. [START_REF] Noha | Evaluation of depth compression and view synthesis distortions in multiview-video-plus-depth coding systems[END_REF] showed that distortions from compression may be masked by distortion from the synthesis process. Consequently, they recommend using the control synthesis (i.e., image synthesized from uncompressed data) as a reference when assessing codec performances. Concerning the subjective evaluation sessions, another aspect to be considered is the video content: as mentioned in [START_REF] Huynh-Thu | Video quality assessment: From 2-D to 3-D-Challenges and future trends[END_REF], compared to 2-D, video content may be determinant in the evaluation by observers. Consequently, considering the factors brought by 3-D video, both subjective and objective methods of evaluation have been studied for an adaptation to 3-D. Subjective quality assessment is delicate while addressing a new type of conditions because one has to define the optimal way to get reliable data. In [START_REF] Pan | Quality metric for approximating subjective evaluation of 3-D objects[END_REF], the authors propose a strategy to develop a quantitative model that estimates 3-D perceptual quality, in the context of mesh representation of 3-D objects. Based on a prior study on quality factors such as geometry resolution, texture resolution, and shading complexity, the authors derived a model that approximates the perceptual quality of mesh-based 3-D objects. In our study, i.e. in the case of use of Multiview video plpus depth (MVD) data and DIBR algorithms, the right protocol to assess the visual quality with observers is still an open question, since new conditions are introduced. The adequate protocol might vary according to the purpose ( impact of compression, of DIBR techniques comparison). In the absence of any better subjective quality assessment methodologies for the evaluation of synthesized views, the measurement is mostly obtained through 2-D validated protocols. Chen et al. [START_REF] Chen | New requirements of subjective video quality assessment methodologies for 3DTV[END_REF] proposed new requirements for subjective video quality assessment. In particular, the authors point out the fact that the minimum of 15 observers, as recommended in ITU-BT.500, may not be sufficient because of the fluctuation of viewers' assessment in 3-D. Therefore, they suggest reconsidering the statistical analysis of viewers' opinion for appropriate rejection of incoherent responses. For 3DTV applications, the authors essentially explained how test material (3-D displays and content disparity) affects depth rendering. This calls for a careful analysis of future experiment results. Concerning objective methods, various approaches have been proposed but it is still a delicate topic. In the context of polygonal meshes based representation, in [START_REF] Cheng | A 3d perceptual metric using justnoticeable-difference[END_REF], the authors introduce the concept of just-noticeable-difference for meshes: they investigate a perceptual threshold where humans can just distinguish the difference between two levels-of-details (this refers to the complexity of the 3-D mesh).

The added value, compared to 2-D conventional video, comes from the exploitation of multiple video sequences acquired at different viewpoints in the scene. These video sequences can be processed into different 3-D representations [START_REF] Smolic | Coding algorithms for 3DTV-a survey[END_REF]: image-based representations (conventional stereoscopic video, multi-view video, etc.), surface-based representations (polygonal meshes, etc.), point-based representations or depth image-based representations (2-D+Z, multi-view video plus depth, layered depth video, etc.) among others. This study is in line with the depth image-based representation context, especially using multi-view video plus depth data, referred to as MVD in the rest of the paper.

MVD designates the association of multiple conventional color videos, referred to as "texture data," and their correspondent depth video sequences, referred to as "depth data." Depth image-based rendering (DIBR) algorithms synthesize a novel view of the scene, different from those captured by the cameras. They require information on scene geometry and texture information of the scene at slightly different viewpoints, provided by depth data and texture respectively.

The problem of assessing 3-D video quality first refers to the object under test. There is no standardized quality assessment framework for 3-D video, so the object under test is chosen depending on the desired application, on the used 3-D representation (mentioned above) and on the chosen display (i.e., stereoscopic, auto-stereoscopic or multi-autoscopic). For instance, in the case of stereoscopic videos, quality of right and left views may be separately assessed, or tools allowing assessing the fused views quality may be used as in [START_REF] Boev | Towards compound stereo-video quality metric: a specific encoder-based frame-work[END_REF].

In our application context, the very first criterion to consider is the quality of the synthesized view. It is justified by the following reasons. Firstly, 3DTV technology relies on the stereopsis phenomenon. This designates the fact that the human brain fuses two slightly different images, presented on each eye, and interprets the 3-D content [START_REF] Patterson | Human factors of 3-D displays[END_REF]. As a result, 3-D displays should provide the appropriate stereoscopic images to ensure depth feeling. Yet, the captured views may not be stereo-compliant depending on the display characteristics. In that case, view synthesis is needed to create the correct stereoscopic pairs. Secondly, in the case of FVV, smooth navigation into the scene requires the generation of nonacquired views. Thirdly, for broadcast situations, constraints on bandwidth limit the amount of data to be transmitted. Generally, virtual views are synthesized from compressed texture and depth data. So compression performances can be evaluated by the rendered views quality. Fourthly, new compression methods using view synthesis prediction have recently been proposed: Na et al. [START_REF] Na | Joint coding of multi-view video and corresponding depth map[END_REF] proposed a coding method for MVD using virtual view synthesis prediction. Morvan et al. [START_REF] Morvan | Joint depth/texture bitallocation for multi-view video compression[END_REF] proposed a joint depth/texture bit allocation algorithm for MVD compression that minimizes the distortion of the intermediate rendered view where original view is taken as a reference for the distortion measure. Consequently, our study focuses on the quality of the virtual synthesized view. More precisely, our study considers virtual synthesized views in specific conditions: as a preliminary experiment, and because we would like to free ourselves from the displays whose technology and artifacts are not under our control, we propose to primarily estimate the quality of DIBR-based synthesized views in 2-D. This is actually a plausible context in a FVV application too.

Moving Pictures Expert Group (MPEG, ISO/IECJTC 1/SC 29/WG 11) created a working group in order to develop a basic framework that, from a limited number of views, enables FVV. Four exploration experiments (EE) have been established ( [START_REF]Description of exploration experiments in 3-D video coding[END_REF]): EE1 for improved depth generation, EE2 for extrapolation capabilities of view synthesis, EE4 for coding experiments and EE5 for 4-view configurations. They provide specific requirements concerning the test conditions, including coding conditions and objective quality evaluation conditions. Especially, the performances of the proposed tools will be based on Peak Signal-to-Noise ratio (PSNR) in some cases. In the meantime, the Video Quality Expert Group (VQEG) created the 3DTV Work Group ( [START_REF]VQEG 3DTV Group[END_REF]) in order to conduct tests and report the results to ITU or other standards organizations. VQEG is concerned by questions related to the experimental design of subjective assessment methods.

Thus, the quality of different 3-D video formats and 3-D video coding strategies are to be evaluated by researchers, and the outcome will determine the chosen solutions in the industry. Therefore it seems to be of the utmost importance and urgency to confirm that usual quality assessment tools are easily applicable to 3-D synthesized views. Then, in this context, we consider the reliability of the commonly used 2-D video metrics when assessing DIBR rendered virtual views. The contribution of this paper comes from the two main questions we studied. First, how well-adapted are the used subjective assessment protocols in the case of DIBR rendered virtual views? Second, is there a correlation between commonly used 2-D video metrics scores and subjective scores when evaluating the quality of DIBR rendered virtual views? Because DIBR rendered virtual views are 2-D images, usual 2-D metrics are naturally thought of as appropriate for assessing their quality. The problem we consider has not been treated in the conditions we propose, in the literature, yet. In our context, the results of our study confirm the common idea that 3-D video is not just the extension of 2-D video and that commonly used 2-D video metrics are not sufficient to assess the synthesized view quality. The synthesized images quality is measured through pixel-based or perceptual-like metrics. The images are synthesized with seven different DIBR algorithms. Correlation with human perception is estimated through subjective assessments.

This document is organized as follows: Section I is a review of the latest works in the field of quality evaluation in 3-D Video. Section II addresses the view synthesis process. Section III considers the first question regarding the subjective assessment protocols while Section IV studies the reliability of the objective tools. Section V proposes directions for future work and Section VI concludes the paper.

I. QUALITY EVALUATION IN THE CONTEXT OF MVD

Previous studies related to investigations on quality metrics for 3-D images are mentioned in this section.

In our context (MVD representation of 3-D scenes), conventional 2-D objective quality models have been spontaneously thought of for assessing 3-D videos, because their efficiency has been proved for the evaluation of 2-D image quality. You et al. [START_REF] You | Perceptual quality assessment for stereoscopic images based on 2-D image quality metrics and disparity analysis[END_REF] addressed their capabilities in the context of stereoscopic images. The authors applied different 2-D quality metrics on a single distorted view. Various types of distortions were studied. They shown that, if used directly, the tested metrics were not robust to changes of distortion types, but if the disparity quality was appropriately combined, their capabilities could be improved.

Previous works also questioned the reliability of commonly used 2-D metrics. Yasakethu et al. [START_REF] Yasakethu | Quality analysis for 3-D video using 2-D video quality models[END_REF] studied the relationship between subjective and usual objective measures (PSNR, SSIM, and VQM), in the context of stereoscopic videos and also 2-D+Z videos. Sequences are compressed at different bit-rates and then evaluated by objective measures and through a stereoscopic display and an auto-stereoscopic display with 15 expert observers. Objective metrics are applied to decoded frames, but not to synthesized views as in our study. Yet, an auto-stereoscopic display is used for the subjective measurement, so a synthesis process is used. However, this process used by the display is blind. Statistical tools are used to show that 2-D metrics might be used, separately on each view/type of sequence to assess 3-D video quality. Tikanmaki et al. [START_REF] Tikanmaki | Quality assessment of 3D video in rate allocation experiments[END_REF] studied the assessment of 3-D encoded video in the context of 2-D+Z, and the authors considered the synthesized view quality. When trying to determine the optimal ratio between color and depth data, the authors observed that PSNR and VSSIM led to different conclusions, according to the compression choices. PSNR seemed unstable depending on the direction of the targeted virtual viewpoint. Yet, the subjective scores correlated the VSSIM scores.

Concepts of psychophysics, used in 2-D conditions (contrast sensitivity [START_REF] Mullen | The contrast sensitivity of human colour vision to redgreen and blue-yellow chromatic gratings[END_REF], contrast masking and facilitation effects [START_REF] Klein | Seven models of masking[END_REF], local contrast [START_REF] Schreiber | Fundamentals of electronic imaging systems: some aspects of image processing[END_REF] etc.) can certainly be exploited in these new conditions, but previous studies suggested that new concepts have also been introduced. Our study evaluates the reliability of commonly used 2-D objective and subjective measurements when assessing DIBR-based rendered virtual views. This is of great interest because aforementioned working groups are currently planning to rely on the results of objective and subjective measurements inspired from 2-D protocols, to assess new methods enabling 3-DTV.

II. VIRTUAL VIEW SYNTHESIS ALGORITHMS

In this section, different DIBR methods are presented. DIBR is the process of synthesizing "virtual" views at a slightly different viewing perspective, using an image or video and the associated per-pixel depth information. This novel view generation can be divided into two steps. In a first step, the texture information is re-projected into the 3-D world using the per-pixel depth information. At the next step, these 3-D space points are projected into the 2-D image plane of the "virtual" camera. The re-projection from 2-D to 3-D and the following projection from 3-D to 2-D are called image warping.

A critical problem in DIBR is that regions occluded in the original view may become visible in the "virtual" view, an phenomenon also referred to as disocclusion. This is particularly problematic in case of extrapolation beyond the baseline of the original views as shown in Fig. 1. This figure shows an extrapolation scenario. One "virtual" camera view is extrapolated from one or two original ones. Scene areas that are visible in the original cameras are marked with blue lines. Those visible in the "virtual" view are highlighted with red lines. Disoccluded areas are visible in the "virtual" camera but invisible in the original ones and must therefore be reconstructed synthetically. The baseline is the distance between the projection centers of two cameras as shown in Fig. 1. A larger baseline between original and "virtual" camera leads to larger disocclusions.

In the absence of original image data, two extrapolation paradigms address this inherent disocclusion problem: 1) one can preprocess the depth information in a manner that no disocclusion occurs in the "virtual" view [START_REF] Fehn | Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV[END_REF], or 2) replace the missing image areas (holes) with known suitable image information [START_REF] Mori | View generation with 3-D warping using depth information for FTV[END_REF], [START_REF] Müller | View synthesis for advanced 3-D video systems[END_REF], [START_REF] Ndjiki-Nya | Depth image based rendering with advanced texture synthesis[END_REF], [START_REF] Köppel | Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering[END_REF]. In the following, a short overview will be given on relevant work in disocclusion handling in 3-D video. The presented methods were evaluated via subjective and objective tests. The results are presented in the following sections.

Fehn [START_REF] Fehn | Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV[END_REF] pre-processes the per-pixel depth information with a 2-D Gaussian low-pass filter. The corresponding block diagram is depicted in Fig. 3. In this way large discontinuities are smoothed out in the depth map and disocclusions do not appear in the "virtual" image. Fig. 2(b) shows a rendered result for a part of frame 141 of the sequence "Lovebird1". The filtered depth map (cf. Fig. 2(g)) is used to warp the original view (cf. Fig. 2(a)) to the "virtual" position. The lowpass filtering of the depth map leads to geometric distortions in the "virtual" view (cf. Fig. 2(b)). The distortions are especially remarkable at background-foreground transitions.

The distortions are visible on the stairway in the backround which appears curved in the "virtual" view (cf. Fig. 2(b)). Also, the person in the foreground is widening to the left. In a rectified camera setup, the method proposed by Fehn [START_REF] Fehn | Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV[END_REF] fails to extrapolate holes on the left or right image border. Therefore, these areas are dealt with in the following in two different ways. Either the border is cropped and the image is interpolated to its original size, or the holes on the border are inpainted using the method proposed by Telea [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF]. These two methods are referred to as A1 and A2 respectively in the rest of the paper. The result shown in Fig. 2(b) is rendered by using method A2. The cropping method (A1) is only suitable for a stereo video, where only one view is transmitted and the other is rendered at the decoder side. In multi-view scenarios, A1 is not applicable because all views (the original as well as the "virtual" views) have to be cropped to preserve the stereo impression, which would lead to image information losses in all the views. Furthermore, by cropping

.n) Warping

Fig. 3. Block diagram of the view synthesis approach proposed by Fehn [START_REF] Fehn | Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV[END_REF], noted A1. The depth map is pre-processed with a 2-D Gaussian low-pass filter. In this way large discontinuities are smoothed out in the depth map. The filtered depth map is used to warp the original view to the "virtual" view. Holes in the left or right image border cannot be filled with this method in a extrapolation scenario. Therefore the holes at the borders are either inpainted [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF] (algorithm A2), or the borders are cropped and the image is finally interpolated to its original size. and subsequently interpolating the image, the distortions in the frame are enhanced.

Four methods that will be described hereafter, use the pixelwise depth information (cf. Fig. 2(f)) to project the original frame (cf. Fig. 2(a)) to the "virtual" position. The projected image and the associated depth map with the disoccluded areas are shown in Fig. 2(c) and 2(h) respectively. The different methods fill in these holes in distinctive ways. Rendered results are shown for frame 141 of the "Lovebird1" sequence. Here, the original camera 8, is used to render camera 6.

Tanimoto et al. [START_REF] Mori | View generation with 3-D warping using depth information for FTV[END_REF] proposed a 3-D view generation system that will be referred to as A3 in the following sections. They use an inpainting method [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF] to fill in missing parts Block diagram of the view synthesis approaches proposed by Tanimoto et al. [START_REF] Mori | View generation with 3-D warping using depth information for FTV[END_REF] (noted A3) and Müller et al. [START_REF] Müller | View synthesis for advanced 3-D video systems[END_REF] (noted A4) for a view extrapolation scenario. The pixel-wise depth information is used to warp the original image to the "virtual" view. Subsequently, Tanimoto et al. [START_REF] Mori | View generation with 3-D warping using depth information for FTV[END_REF] cover the holes in the "virtual" image by using the inpainting method proposed in [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF]. Müller et al. [START_REF] Müller | View synthesis for advanced 3-D video systems[END_REF] fill the holes line-wise with neighboring background information.

in the "virtual" image. Their framework is primarily used to interpolate "virtual" views between two original cameras, where just small holes are to be closed. The block diagram for their extrapolation scenario is shown in Fig. 4. Fig. 2(d) shows a rendered result. The inpainting method [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF] used to fill the holes (cf. Fig. 2(c)) introduces blur into the disoccluded area. In the extrapolation scenario the blurry regions are more obvious in the "virtual" view than in the interpolation case. Due to a missing foreground-background separation, the foreground image information is inserted into the disoccluded region. The inpainting method [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF] fails to reconstruct complex textures. This algorithm was recently adopted as the reference software for MPEG standardization experiments in the 3-D Video group. First experiments have been conducted by MPEG to explore the extrapolation capabilities of the reference software [START_REF]Description of exploration experiments in 3-D video coding[END_REF].

Müller et al. [START_REF] Müller | View synthesis for advanced 3-D video systems[END_REF] proposed a hole filling method embedded in a 3-D video system that will be referred to as A4 in the following sections. The block diagram for the extrapolation scenario is shown in Fig. 4. Holes are filled line-wise with neighboring background information. The corresponding depth values at the hole boundary are examined row-wise to find background color samples to be copied into the hole. A rendered result is shown in Fig. 2(i). This method fails to reconstruct vertical or oblique structures and complex textures, as can be seen in the background of the rendered cut-out on the stairway. It is thus suitable for the reconstruction of simple and smooth non-textured backgrounds. Color extrapolation of the suitable background pixels leads to better results than a simple linear interpolation. Generally, due to depth estimation errors, some boundary background pixels may in fact belong to foreground objects. Thus, the color hints Müller et al. [START_REF] Müller | View synthesis for advanced 3-D video systems[END_REF] exploit would lead to foreground color propagation into the hole.

In texture synthesis methods, the unknown regions are synthesized by copying content from the known parts of the image to the missing regions. Ndjiki-Nya et al. [START_REF] Ndjiki-Nya | Depth image based rendering with advanced texture synthesis[END_REF] proposed a hole filling approach for DIBR systems based on patch-based texture synthesis. This method is referred to as A5 in the rest of the paper. The corresponding block diagram is shown in Fig. 5. Fig. 2(e) shows a rendered result. Holes with small spatial extent are closed by solving Laplacian equations. Larger holes are initialized by median filtering and then optimized via texture synthesis. Thanks to the usage of texture synthesis, it is possible to reconstruct complex textures, as can be seen in the background area in Fig. 2 Fig. 5. Block diagram of the view synthesis approach proposed by Ndjiki-Nya et al. [START_REF] Ndjiki-Nya | Depth image based rendering with advanced texture synthesis[END_REF] (noted A5). The original image and the associated pixel-wise depth information is warped to the original view. Then the disocclusions in the depth map are filled. Finally, the holes in the image are initialized and subsequently refined with texture synthesis.

background area is finite and may not be a good representative of the texture to be predicted, even the best match may be a bad compromise and lead to artifacts in the synthesized texture. Due to the utilized texture synthesis approach [START_REF] Ndjiki-Nya | Depth image based rendering with advanced texture synthesis[END_REF] only straight edges can be accurately reconstructed.

The presented view synthesis methods A1-A5 render a sequence frame-by-frame. Information from previous or subsequent frames is not considered. This can lead to temporal inconsistencies such as flickering.

Köppel et al. [START_REF] Köppel | Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering[END_REF] extended the A5 approach by a background sprite. The sprite stores valuable background image information and is updated frame-wise. Using the original and synthesized image information from previous frames, temporal consistency is achieved in a sequence. A rendered result is shown in Fig. 2(j) and the block diagram is depicted in Fig. 6. Background texture is copied from the sprite to the disoccluded area so that background details are preserved. The image information from previous frames is selected via the pixel-wise depth information. However, the latter not always reliable. Therefore, incorrectly copied background image information can degrade the quality of the "virtual" view. Also, the blocking artifacts described for method A5 can occur as well. In subjective tests conducted in section III only images are evaluated. Thus, the capabilities of the approach [START_REF] Köppel | Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering[END_REF] to achieve temporal consistency in a sequence are not investigated in this work. This method is referred to as A6 in the rest of the paper.

Unfilled sequences (i.e. those with holes) are referred to as A7 in the rest of the paper (cf. Fig. 2(c)).

III. SUBJECTIVE MEASUREMENTS

The objective of the experiments is to determine whether two efficient protocols of subjective assessment of quality (in a 2-D context) are still adapted to assess the quality of DIBR-based synthesized images. Three test sequences have been used to generate four different viewpoints, that is to say twelve synthesized sequences for each tested algorithm (84 synthesized sequences in total): Book Arrival(1024×768, 16 cameras with 6.5-cm spacing), Lovebird1 (1024×768, 12 cameras with 3.5-cm spacing) and Newspaper (1024×768, 9 cameras with 5-cm spacing). Altogether, 43 naive (i.e., nonexpert) observers participated in the subjective assessment session. They were mostly computer science students with no prior knowledge of the sequences. The session was conducted in an ITU conforming test environment. Absolute categorical rating (ACR) [START_REF]ITU-T p.910 subjective video quality assessment methods for multimedia applications[END_REF] and Paired comparisons [START_REF] Keelan | Handbook of image quality: characterization and prediction[END_REF] were used to collect perceived quality scores. The stimuli were displayed Fig. 6. Block diagram of the view synthesis approach proposed by Köppel et al. [START_REF] Köppel | Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering[END_REF] (noted A6). The original view and the associated depth map are projected to the "virtual" position. Next, the background sprite is updated with original background data and the holes in the "virtual" image are updated with background information from the sprite. The remaining holes are first initialized and subsequently refined with texture synthesis. on a TVLogic LVM401W, and according to ITU-T BT.500 [START_REF] Bt | Methodology for the subjective assessment of the quality of television pictures[END_REF]. Considering the large size of the tested database, only key frames were presented to the observers, since still images can also be a plausible scenario for FVV.

A. Subjective Methodologies

As explained in the introduction, subjective quality assessment is delicate when addressing new type of conditions because one has to define the optimal way to get reliable data. New conditions are introduced, and the right protocol to assess the visual quality with observers is still an open question. Recently, authors considered these new conditions and concepts, such as the presence, or the perception of depth, in [START_REF] Ijsselsteijn | Effects of stereoscopic presentation, image motion, and screen size on subjective and objective corroborative measures of presence[END_REF] and [START_REF] Seuntiens | Visual experience of 3D TV[END_REF]. In the absence of any better subjective quality assessment methodologies for the evaluation of synthesized views, the measurement is mostly obtained through 2-D validated protocols.

Absolute Categorical Rating Method : The absolute categorical rating (ACR) [START_REF]Subjective video quality assessment methods for multimedia applications[END_REF] method consists in presenting several test conditions (also called trials) to observers. A single trial is presented only once and the all the trials are presented in a random order. Observers score the test item according to a discrete category rating scale. The five-level (ACR-5) used scale is shown in Table I. Each adjective corresponds to an integer ranging from 1 to 5. Observers' opinion scores are then averaged by computing a mean opinion score (MOS). ACR requires many observers to minimize the contextual effects (previously presented stimuli influence the observers' opinion, i.e. presentation order influences opinion ratings).

Paired Comparisons (PC) Method: With this method, a pair of images is presented to the observer who selects the one that best satisfies the specified judgment criterion, i.e. the best image quality. Responses from many observers yield to an interval-scale ordering of images: they can be converted to scale values using Thurstone-Mosteller's or Bradley-Terry's [START_REF] Handley | Comparative analysis of Bradley-Terry and Thurstone-Mosteller paired comparison models for image quality assessment[END_REF]. This leads to a hypothetical perceptual continuum. The presented experiments follow Thurstone-Mosteller's model where naive observers were asked to choose the preferred item from one pair. Paired comparisons are time consuming: for N items, N (N -1) 2 comparisons are executed.

B. Suitability of Usual Subjective Assessment Protocols

Statistical analyses have been conducted over subjective measurements. A Student's t-test has been performed over the MOS scores, and over the PC scores for each algorithm. This is meant to provide knowledge on the statistical equivalence of the algorithms. Tables II andIII show the results of the statistical tests over MOS and PC values respectively. In both tables, the number in parentheses indicates the required minimum number of observers that allow the statistical distinction (VQEG recommends 24 participants as a minimum, values in bold are higher than 24 in the tables). Table III indicates that only A2-A4 and A2-A6 are statistically equivalent. Besides, this is confirmed by Table II, because the same groups are identified among others (A5-A2, A6-A4 for example). In the case of PC test, most of the algorithms can be statistically distinguished with less than 24 participants. However, in the case of ACR results, the final distinction seems stable when 32 observers participate. This suggests that fewer observers are needed for a PC-based subjective assessment. However, this method is very time-consuming: first because of the large number of comparisons; and second, more observers are required in order to avoid the contextual effect. For these experiments, in average, one observer needed 45 minutes to achieve the PC test but only 15 minutes for the ACR test.

IV. OBJECTIVE MEASUREMENTS

The objective of the experiments is to assess the reliability of objective metrics in the case of images synthesized from DIBR algorithms, by determining the tested algorithms performances. The same test material as in subjective measurements (Section III) was used in these objective measurements. Key frames were evaluated through different objective metrics through MeTriX MuX Visual Quality Assessment Package [START_REF]MetriX MuXpage[END_REF]. For both objective metrics, the reference was the original acquired image.

A. Objective Tools

Different objective metrics are used in the presented experiments:

• Peak Signal-to-Noise Ratio (PSNR) measures the signal fidelity of a distorted image compared to a reference. It is based on the measure of the mean squared error (MSE).

• As an alternative to pixel-based methods, universal quality index UQI [START_REF] Wang | A universal image quality index[END_REF] is a perceptual-like metric. It models the image distortion by a combination of three factors: loss of correlation, luminance distortion, and contrast distortion.

• PSNR-HVS [START_REF] Egiazarian | New full-reference quality metrics based on HVS[END_REF], based on PSNR and UQI, is meant to take into account the Human Visual System (HVS) properties. PSNR-HVSM [START_REF] Ponomarenko | On between-coefficient contrast masking of DCT basis functions[END_REF] is based on PSNR but takes into account contrast sensitivity function (CSF) and between-coefficient contrast masking of DCT basis functions. • Single-scale Structural SIMilarity (SSIM) [START_REF] Wang | Image quality assessment: From error visibility to structural similarity[END_REF] combines image structural information: mean, variance, covariance of pixels, for a single local patch. Multi-scale SSIM (MSSIM) is the average SSIM scores of all the patches of the image. • Visual Signal-to-Noise Ratio (VSNR) [START_REF] Chandler | VSNR: a wavelet-based visual signal-to-noise ratio for natural images[END_REF] is also a perceptual-like metric: it is based on a visual detection of distortion criterion, helped by CSF. • Weighted Signal-to-Noise Ratio (WSNR) that uses a weighting function adapted to HVS, denotes a weighted signal-to-noise ratio. • Visual Information Fidelity (VIF) [START_REF] Sheikh | Image information and visual quality[END_REF] uses a statistical model and an image distortion model to assess an image.

VIFP is a pixel-based version of VIF. • Information Fidelity Criterion (IFC) [START_REF] Sheikh | An information fidelity criterion for image quality assessment using natural scene statistics[END_REF] uses a distortion model to evaluate the information shared between the reference image and the degraded image. • Noise quality measure (NQM) [START_REF] Damera-Venkata | Image quality assessment based on a degradation model[END_REF] quantifies the injected noise in the tested image.

B. Correlation Between Objective and Subjective Measures

From the whole set of tested data, measures yield the performances of the algorithms. Objective and subjective measurements lead to different rankings, depending on the metric, as showed on Table IV. Subjective-based rankings point A1 as the best out of the seven algorithms. But objectivebased rankings judge it as the worst. Similarlyl, A6 obtains the best results for numerous objective metrics, but subjective assessments judge the synthesized views as "poor". A2, A3, A4 and A5 ranks seem to be relatively coherent with subjective evaluations. This table suggests that objective metrics detect non-annoying artifacts. Another remark concerns the fact that the objective metrics rankings are very close to each other. This suggests that the tested objective metrics are correlated. This is confirmed in Table V. This table also shows the high correlation with pixel-based and more perceptual-like metrics such as PSNR and SSIM (83.9%).

Results also show that the algorithm performances are linked to the sequences properties: Fig. 7 shows the MOS scores from ACR and Fig. 8 shows the scaled values from PC, obtained for each algorithm, depending on the sequence. Algorithms are ranked by obtained scores. The ranks vary depending on the tested sequence. A4 seems to perform better on Lovebird1 than the other algorithms, and it is ranked as the best algorithm but only for this sequence. However, note that all the scores for this sequence are lower than those of the other sequences, regardless the algorithm. This suggests that the algorithms performances are tied to the sequence content or corroborates [START_REF] Huynh-Thu | Video quality assessment: From 2-D to 3-D-Challenges and future trends[END_REF] concerning the influence of the content on human opinion. The same trend is observed with PC results, and with the objective measures.

The consistency between objective and subjective measures is evaluated by computing the correlation coefficients for the all the fitted measured points. Table VI show the results. Note that none of the tested metric is 50% close to human judgment when assessing synthesized views. According to MOS and PC scores, the most correlated metrics are PSNR HSV M (38.1% and 42.2% resp.), PSNR HSV (37.3% and 41.9% resp.), PSNR (38.6% and 40.0% resp.) and WSNR (42.3% and 36.9% resp.). These are pixel-based or perceptual-like metrics. In order to further consider the consistency, Fig. 9 shows the ACR values over the PSNR HSV M scores. The figure shows that a difference of nearly 9 dB (A1 and A5) leads to nearly the same absolute rate. V. FUTURE DIRECTIONS These results point out the lack of crucial information that would help the assessment of synthesized views. In this section, we propose an analysis of the location of the artifacts and the first results for new objective assessment tools.

A. Analysis of Artifacts

Comparing the algorithms performances depending on the baseline distance, it can be observed that the shorter the baseline, the better the scores are, as expected (for both subjective and objective scores). Also, the algorithm rankings remain stable from short to larger baseline distance (large distance is defined as twice the acquisition camera baseline distance). This suggests that each algorithm induces a specific type of distortion that is located in the disoccluded areas while the rest of the image remains close or relatively perceptually acceptable to humans.

The analysis of the location of the artifacts, illustrated in Fig. 10 confirms the assumption concerning the close relationship between algorithms and distortion types. The images are obtained by computing the difference between original and synthesized view, and then applying a threshold T h to detect the most important errors (identified as critical areas). It is defined as follows, to detect only the most important errors in the synthesized image:

T h = max(I-I ) 10
where I is the original image, and I is the synthesized view.

In our case, unlike the commonly encountered artifacts in video coding, artifacts are mainly located around the disoccluded areas, while the rest of the image remains close or the same as the original. Most of the video coding standards rely on a block-based DCT, and the resulting artifacts are often scattered in the whole image, as studied in [START_REF] Yuen | A survey of hybrid MC/DPCM/DCT video coding distortions[END_REF]. Artifacts resulting from DIBR algorithms are different and located in specific areas.

As explained in Section II, the critical problem in DIBR is the disocclusions that are more likely to occur around sharp discontinuities. Thus the artifacts are located in these specific areas. By extrapolating the missing areas, the tested algorithms induce errors in these areas. A1, A2 and A3 induce blur in these regions, making the background-foreground transitions unnatural. The distortions seem to be even more visible and annoying if the color contrast between foreground and background is important, which make these cases critical. The best methods, in the sense of objective metrics are A4, A5 and A6. All three show fewer errors around the disoccluded areas (Fig. 10). However their most significant errors are located in the text areas. All the objective metrics penalize A1 and A3 because the count numerous errors around the disoccluded areas.

So, the type and the location of artifacts are specific in the case of DIBR-based synthesized images, and are different from the current cases of 2-D images. Here, a region may be slightly transferred or resized, depending on the chosen extrapolation method (if the method chooses to assign the background values to the missing areas, object may be resized). If the region is still visually coherent, subjective scores should be correct, but pixel-based metrics should penalize this type of image (case of A1). For this reason, we believe that a novel assessment metric should consider the degradations of the image coupled with a penalizing factor associated to the transformation of the critical areas. This, in order to measure the amount of geometric distortions.

B. Analysis of the Contours Consistency

Our assumption concerning the specific location of artifacts around the disoccluded areas, and more precisely around strong depth discontinuities, led us to consider the contours of the synthesized view. Then, we propose to penalize important shifts of contours. The block diagram in Fig. 11 illustrates the principles of this approach.

As illustrated in Fig. 11, we first apply a Canny Edge Detector on the reference image and on the synthesized view in order to detect their contours. Then a motion vectors estimation is computed over the contours of the synthesized view, in comparison with the contours of the reference view. Considering a neighborhood of pixel consisting of a 3 × 3 window, neighbor vectors whose angle with the motion vector of the central pixel is up to 45 • are considered as deviant and central pixel is penalized by a vote from each deviant vector. Then, the score of the central pixel is the sum of the votes of each neighbor vectors. The final quality score of the image is obtained by averaging all the votes computed previously for each pixel.

As pointed out in Section IV, the thin consistency between objective and subjective measures was illustrated by two distinct items obtaining the same subjective score but two very different objective scores; or two distinct items obtaining the same objective score but two very different subjective scores. We considered these four items. As a preliminary experiment, we then assessed the quality of four critical items with an objective metric (SSIM) and with the proposed measure. Results are plotted in Fig. 12. Concerned about the clarity of the presented results, we scaled the points obtained with the proposed approach, according to the same affine transform: we first scaled two of the obtained points so that they lay on the corresponding scores of the SSIM; then we applied the same obtained transformation to the two other points. The arrows on the figure, illustrate the fact that the proposed approach moves the points closer to a linear regression line. The consistency with the subjective scores is improved. The correlation coefficients are 18% and 84% for SSIM and proposed approach respectively. Fig. 12. MOS scores over scores of objective metrics (SSIM and proposed approach in Section V-B).

Considering the consistency of the contour shifts, these very first results are encouraging and are to be investigated in future work.

C. Analysis of the distorted areas

In a second approach, based on the observations mentioned in Section V-A, we considered the distorted areas. Fig. 13 shows the block diagram of the proposed approach. It consists in applying an objective metric only on the critical areas. The critical areas are defined by the main differences between the reference view and the synthesized view, as mentioned in Section V-A. Then we apply SSIM measure on these areas and the final score is the mean SSIM scores normalized by the amount of distorted pixels. As previously, we assessed the quality of four critical items with an objective metric (SSIM) and with the proposed measure. Results are plotted in Fig. 14. As explained above, we scaled the points obtained with the proposed approach. The figure shows that the consistency with subjective scores is slightly improved with the proposed approach. The correlation coefficients are 18% and 78% for SSIM and the proposed approach respectively.

The observation of the disoccluded areas into an objective measure seems encouraging but further investigation should be run.

VI. CONCLUSION

Subjective measurements (ACR and Paired comparisons) were coupled with objective measurements in the presented experiments, assessing views synthesized from seven different algorithms. The experiments considered the commonly used methods for assessing conventional images, subjectively as well as objectively. Concerning the subjective evaluations, Paired comparisons and ACR results highly correlate. But statistical analyses show that fewer observers were required for Paired comparisons tests to establish the algorithms distinctions. However, this is a time-consuming method, often avoided by researchers. Moreover, when the number of items to be compared is high, the test is hardly feasible. Concerning the objective metrics, the results show that usual objective assessments hardly correlate with subjective assessments. Rankings of algorithms from objective metrics are not reliable, when regarding the subjective results. The presented experiments revealed that using only the objective metrics is not sufficient for assessing virtual synthesized views, though they give information on the presence of errors. The analysis of their capabilities coupled with the analysis of the distortions induced by view synthesis might help in designing a new metric. We presented the first results of two measurement approaches. The first proposal was based on the analysis of the orientation of the edge shifts. The second proposal was based on the analysis of the disoccluded areas. Both approaches gave encouraging results when considering items deviant with usual 2D metrics. The correlation with the subjective measurements is encouraging. Further experiments should be performed on video sequences in the future, to refine the presented results.

Fig. 2 .

 2 Fig. 2. DIBR results for frame 141 of the "Lovebird1" sequence. A baseline of 7cm is used. Camera 8 is used to render camera 6. (a) Original view. (f) Corresponding depth map. (b) "Virtual" camera view obtained with the method proposed by Fehn[START_REF] Fehn | Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV[END_REF] using the Gaussian filtered depth map shown in (g). (c) Projected "virtual" view with disocclusions (marked black) using the original depth map (f). (h) Corresponding warped original depth map with disocclusions (marked white). (d) Result of the hole filling method proposed by Tanimoto et al.[START_REF] Mori | View generation with 3-D warping using depth information for FTV[END_REF]. (e) Result of the hole filling method proposed by Ndjiki-Nya et al.[START_REF] Ndjiki-Nya | Depth image based rendering with advanced texture synthesis[END_REF]. (i) Result of the hole filling method proposed by Müller et al.[START_REF] Müller | View synthesis for advanced 3-D video systems[END_REF]. (j) Result of the hole filling method proposed by Köppel et al.[START_REF] Köppel | Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering[END_REF].

  Fig. 4.Block diagram of the view synthesis approaches proposed by Tanimoto et al.[START_REF] Mori | View generation with 3-D warping using depth information for FTV[END_REF] (noted A3) and Müller et al.[START_REF] Müller | View synthesis for advanced 3-D video systems[END_REF] (noted A4) for a view extrapolation scenario. The pixel-wise depth information is used to warp the original image to the "virtual" view. Subsequently, Tanimoto et al.[START_REF] Mori | View generation with 3-D warping using depth information for FTV[END_REF] cover the holes in the "virtual" image by using the inpainting method proposed in[START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF]. Müller et al.[START_REF] Müller | View synthesis for advanced 3-D video systems[END_REF] fill the holes line-wise with neighboring background information.

Fig. 7 .Fig. 8 .

 78 Fig. 7. Mean ACR Scores for the different sequences.

Fig. 9 .

 9 Fig. 9. Correlation between ACR and PSNR HSM .

Fig. 10 .Fig. 11 .

 1011 Fig. 10. Location of most important errors depending on used synthesis algorithm (in red).Original image DIBR-based synthesized imageEdge detectionEdge detection

Fig. 13 .

 13 Fig. 13. Block diagram of the proposed approach in Section V-C.

Fig. 14 .

 14 Fig. 14. MOS scores over scores of objective metrics (SSIM and proposed approach in Section V-C).

  ) ↑(<24) ↑(32) o (>43) ↑(30) ↑(<24) A2 ↓(32)↑(<24)o (>43)o (>43)o (>43) ↑(<24) A3↓(<24)↓(<24) ↓(<24) ↓(<24) ↓(<24) ↑(<24) A4 ↓(32) o(>43)↑(<24) o(>43) o(>43) ↑(<24) A5o(>43) o(>43)↑(<24) o(>43) ↑(28) ↑(<24) A6 ↓(30) o(>43)↑(<24)o (>43) ↓(28) ↑(<24) A7↓(<24)↓(<24)↓(<24)↓ (<24) ↓(<24) ↓(<24)TABLE II RESULTS OF STUDENT'S T-TEST WITH ACR RESULTS. LEGEND:↑:SUPERIOR, ↓: INFERIOR, O: STATISTICALLY EQUIVALENT. READING: LINE"1" IS STATISTICALLY SUPERIOR TO COLUMN "2". DISTINCTION IS STABLE WHEN "32" )↑(<24)↑(<24)↑(<24)↑(<24)↑(<24) A2↓(<24) ↑(28) o(<24)↓(<24)o(>43)↑(<24) A3↓(<24) ↓(28) ↓(<24)↓(<24)↓(<24)↑(<24) A4↓(<24)o(>43)↑(<24) ↓(<24) ↑(43) ↑(<24) A5↓(<24)↑(<24)↑(<24)↑(<24) ↑(<24)↑(<24) A6↓(<24)o(>43)↑(<24)↓(<43)↓(<24) ↑(<24) A7↓(<24)↓(<24)↓(<24)↓(<24)↓(<24)↓(<24)

  (e). The used patches are rectangular, which may lead to block artifacts. As the known

	Depth Map (1...n)	Warping	Depth Map Filling
	Textured Image (1...n)	Warping	Hole Initialisation	Texture Synthesis	Filled Frame (1...n)

TABLE III RESULTS

 III OF STUDENT'S T-TEST WITH PAIRED COMPARISONS RESULTS. LEGEND:↑: SUPERIOR, ↓: INFERIOR, O: STATISTICALLY EQUIVALENT. READING: LINE"1" IS STATISTICALLY SUPERIOR TO COLUMN "2". DISTINCTION IS STABLE WHEN "LESS THAN 24" OBSERVERS PARTICIPATE. PSNR HSV 13.53019.512 17.953 20.93820.958 20.987 15.407

		A1	A2	A3	A4	A5	A6	A7
	MOS	2.388 2.234 1.994 2.250 2.345 2.169 1.126
	Rank order	1	4	6	3	2	5	7
	PC	1.40380.50810.2073 0.53110.93630.4540 -2.0547
	Rank order	1	4	6	3	2	5	7
	PSNR	18.75224.998 23.180 26.11726.171 26.177 20.307
	Rank order	7	4	5	3	2	1	6
	SSIM	0.638 0.843 0.786 0.859 0.859 0.858 0.821
	Rank order	7	4	6	1	1	3	5
	MSSIM	0.648 0.932 0.826 0.950 0.949 0.949 0.883
	Rank order	7	4	6	1	2	2	5
	VSNR	13.13520.530 18.901 22.00422.247 22.195 21.055
	Rank order	7	5	6	3	1	2	4
	VIF	0.124 0.394 0.314 0.425 0.425 0.426 0.397
	Rank order	7	5	6	2	2	1	4
	VIFP	0.147 0.416 0.344 0.448 0.448 0.448 0.420
	Rank order	7	5	6	1	1	1	4
	UQI	0.237 0.556 0.474 0.577 0.576 0.577 0.558
	Rank order	7	5	6	1	3	1	4
	IFC	0.757 2.420 1.959 2.587 2.586 2.591 2.423
	Rank order	7	5	6	2	3	1	4
	NQM	8.713 16.334 13.645 17.07417.198 17.201 10.291
	Rank order	7	4	5	3	2	1	6
	WSNR 13.81720.593 18.517 21.59721.697 21.716 15.588
	Rank order	7	4	5	3	2	1	6
	PSNR HSVM13.77219.959 18.362 21.42821.458 21.491 15.714
	Rank order	7	4	5	3	2	1	6
	Rank order	7	4	5	3	2	1	6

TABLE IV RANKINGS

 IV ACCORDING TO MEASUREMENTS

ACKNOWLEDGMENT

This work is partly supported by French National Research Agency as part of ANR-PERSEE project (ANR-09-BLAN-0170), and ANR-CAIMAN (ANR-08-VERS-002) project.