1 Architecture for 3D representation and coding

The general architecture of the 3D video coding scheme is described in gure 1. The goal of this architecture is to evaluate the methods and algorithms developed within the Persee project, and compare them to state of the art coding schemes. Subjective evaluations will be performed on the tested methods, to evaluate their pertinence with regard to rendered visual quality.

In a rst phase, partners will exchange input/output les in order to perform evaluation of methods without the need for software integration.

Several types of experiments are planed:

• build a reference 3Dvideo coding chain, in order to compare the proposed methods to state of the art coding schemes. The state of the art codecs for MVD data are H264 and 3DHTM, which is currently under development by the Mpeg consortium.

• perform objective/subjective evaluation of depth compression algorithms which have been developed within the Persée consortium and which are described in the following sections of the deliverable.

• exploit perceptual data extracted from the video in the coding and rendering units, such as visual attention or saliency. The scheme presented in gure 2 exploits such information for introducing blur in videos, both for rate reduction and visual quality enhancement.

The following sections provide some tools developed by PERSEE contributors, in order to build and evaluate the 3D video coding scheme.

Z-cam

Disparity map coding by Don't Care Regions

In the image+depth multiview video format, each view n = 1, . . . , N is represented with a texture and a depth (or, equivalently, disparity) map. Given v n , v n+1 and d n , d n+1 the texture and disparity maps at views n and n + 1, respectively, it is possible to synthesize any intermediate view v k , k ∈ [n, n + 1] using a depth imagebased rendering (DIBR) algorithm. Essentially, any DIBR algorithm synthesizes pixel values in v k by properly translating pixels from views n and n + 1, according to the disparity information contained in the disparity maps, and by possibly resorting to some inpainting technique to ll holes due to occlusions in one of the neighboring views. In this context, the goal of multiview video coding is maximizing the quality of the view synthesized at the decoder, provided that the total number of bits required to encode and transmit v n , d n , v n+1 and d n+1 has to be within a predetermined bit budget. This is in general a joint coding problem, since the rate-distortion (RD) performance of texture/disparity at view n will depend on the RD characteristic of the other three signals assuming that the DIBR algorithm is given.

Instead, in the following discussion we focus on the simpler task of coding disparity maps independently. That is, given uncompressed textures v n and v n+1 , we want to encode independently d n and d n+1 only. This simplied perspective is interesting as it allows as to focus on a basic dierence between disparity and conventional video coding. In disparity video coding, the rate to be minimized is the one required to encode d n and d n+1 . However, coding quality is measured on the synthesized view v k . This leads to two conclusions: 1) the quality metric used in disparity coding should capture the distortion in disparity compensation produced by coarse quantization of disparity maps; and 2) it is possible to give up delity and reduce bit rate in the reconstructed disparity maps if this does not aect too much the synthesized view.

Test of a geometric distortion metric on synthesized views

As mentioned above, at the basis of DIBR there is disparity compensation, i.e., pixels in the view v k to be synthesized are obtained by properly displacing pixels in views n and n + 1. Therefore, coding errors in the disparity maps correspond to displacement errors in the synthesized view. Given the dierent nature of these two source of distortion, we aim at describing displacement errors through a geometric distortion measure.

We consider the geometric distortion measure (GM) proposed in [START_REF] D'angelo | A full-reference quality metric for geometrically distorted images[END_REF]. This metric has been originally proposed in the watermarking eld to quantify the perceptual impact of geometric manipulations aimed at hindering the extraction of a watermark embedded into an image. In a nutshell, this approach consists in projecting the displacement vectors (of a dense displacement eld) along the directions parallel to the gradients of the local image structures, with the rationale that only this displacement component is going to be visible to users. The intensity and the direction of local image structures is obtained through Gabor lters. By averaging, for several directions, a combination of local image structure and intensity (in a given direction) of the displacement vectors, it is possible to predict the mean opinion score (MOS) a human Figure 3: MOS vs. bit rate for coding disparity in the Teddy image. The synthesized view v4 is obtained by DIBR on compressed disparities d2 and d6 . The rate term here refers to the total bits needed to encoded d 2 and d 6 .

observer would give. When we consider the disparity map coding detailed above, a natural question arises: can we use the GM metric to predict the visual impairment introduced by errors in the coding of disparity?

In order to answer this question, we tested the sensitivity of GM to synthesis errors induced by quantization errors in the disparity maps. To this end, we compressed disparity maps d 2 and d 6 of views 2 and 6, respectively, for the Teddy image in the Middlebury Dataset [START_REF]Middlebury stereo vision dataset[END_REF], using JPEG with dierent quality factors. We denote with d2 and d6 the reconstructed noisy disparities at the decoder. Then, we synthesized v4 using d2 and d6 , by means of a simple DIBR algorithm which lls holes in the synthesized view by copying the closest pixel value from either view 2 or 6. In order to single out the contribution to distortion due to compression only, we compared v4 with a groundtruth image v 4 generated using uncompressed d 2 and d 6 . We then predict the distortion between v 4 and v4 using GM, which yields a prediction of the MOS. A rate/MOS curve is shown in Figure 3, where it is clearly visible that the GM metric is absolutely not able to provide meaningful prediction of the MOS for the synthesized view. Similar results have been obtained also for the Doll image [START_REF]Middlebury stereo vision dataset[END_REF].

From these results we can infer that GM is not a suitable metric for synthesis error produced by noisy disparity maps. In fact, even at the maximum JPEG quality, the rounding and truncation entailed in the coding process make the reconstructed disparities being slightly dierent from the groundtruth. This produces isolated errors in the synthesized view, which however have a very strong impact on the computation of GM. Consider also that GM has been designed, tuned and validated with subjective tests performed on images corrupted with smooth displacement errors, so there are no guarantees that it can work in the context of view synthesis.

Increasing coding eciency of disparity maps through Don't Care Regions

Dierently from conventional video coding, where the signal to be compressed is the one that has to be displayed to users, the coding of disparity map video has to be optimized for the display of synthesized views. This introduces a certain degree of freedom in the coding of disparity which is not present in texture coding. For example, in very smooth regions of the frame, having even a relatively large disparity error would not lead necessarily to a large error in the synthesized view. This concept can be formalized and embedded into a state-of-the-art video encoder in order to increase the coding eciency of disparity maps.

Denition of Don't Care Regions (DCR)

A pixel v n (i, j) in texture map n, with associated disparity value d n (i, j), can be mapped into a corresponding pixel of the view n + 1 through a view synthesis function s(i, j; d n (i, j)). In the simplest case where the views are captured by horizontally aligned cameras, s(i, j; d n (i, j)) corresponds to a displacement in the x direction by d n (i, j), that is:

s(i, j; d n (i, j)) = v n (i, j + d n (i, j)). (1)
We dene the view synthesis error, ε(i, j), as the absolute error between the synthesized pixel s(i, j; d n (i, j)) and the actual pixel value v n+1 (i, j) in v n+1 . That is:

ε(i, j; d n (i, j)) = |s(i, j; d n (i, j)) -v n+1 (i, j)| . (2)
If d n is compressed, the reconstructed value dn (i, j) employed for view synthesis may dier from d n (i, j) by an amount e(i, j) = dn (i, j) -d n (i, j), resulting in a (generally larger) view synthesis error ε(d n (i, j) + e(i, j)). We dene the Don't Care Region DCR(i, j) = [DCR low (i, j), DCR up (i, j)] as the smallest interval of disparity values containing the groundtruth disparity d n (i, j), such that the view synthesis error for any point of the interval is smaller than ε(i, j; d n (i, j))+τ , for a given threshold τ . The denition of DCR is illustrated in Figure 4. Note that DCR intervals are dened per pixel, thus giving a very precise information about how much error can be tolerated in the disparity maps.

Use of DCR for increasing coding eciency of disparity maps

Having a per pixel DCR enables a higher degree of freedom in the encoding of disparity maps, which can be compressed with a larger distortion (i.e., lower rate), without actually compromising too much the quality of the displayed synthesized view. Specically, DCR can aect three aspects of disparity coding:

• Motion estimation;

• Mode decision;

• Coding of residuals.

Motion estimation During motion estimation, the encoder search for each macroblock or sub-block B a corresponding region in a reference frame which minimizes the Lagrangian cost function

D mv + λ mv R mv , (3)
where R mv is the bit rate associated to the cost of representing the motion vector and the prediction residual, and λ mv is a Lagrange multiplier. The term D mv is the sum of absolute dierences (SAD) between the pixels in the current block and their predictor.

We can add more degrees of freedom to motion estimation considering that the term D in (3) can be discounted by the per pixel DCR's. To this end, we manipulate the values of prediction residuals r(i, j) involved in the computation of SAD in (3) according to the following soft-thresholding function:

r =      r(i, j) -(DCR up -d n (i, j)) if r(i, j) > DCR up -d n (i, j), r(i, j) -(DCR low -d n (i, j)) if r(i, j) < DCR low -d n (i, j), 0 otherwise. (4)
The discounted SAD is then computed as

D mv = (i,j)∈B r (i, j)) (5)
and substituted to D in (3).

Mode decision

In the mode decision phase, the rate-distortion cost of each coding mode (Inter with all possible sub-block partitions, Intra with 16 × 16, 4 × 4 or 8 × 8, Skip) is evaluated in order to select the optimal modes for a given macroblock and for its partitions. In order to nd the optimal mode conguration, the encoder minimizes a function similar to (3):

D md + λ md R md , (6)
where D md is the SAD or SSE (sum of squared dierences) distortion entailed by each coding mode, R md is the associated bit rate and λ md is a Lagrangian multiplier. As before, for each coding mode we can change the term D md to D md by applying (4) to the residuals obtained with that coding mode.

Coding of prediction residuals Once motion vectors and coding modes have been computed, the prediction residuals are transformed and quantized before being entropy coded and written in the bitstream. According to DCR denition, it is not necessary to reconstruct the true groundtruth disparity at each pixel. Instead, any value that is inside the DCR would be acceptable in terms of view synthesis distortion. However, in this case altering the values of prediction residuals r(i, j) to r (i, j) using equation ( 4) is not optimal from the coding point of view, since this soft-thresholding heuristic on prediction residuals in the pixel domain does not give any guarantees of reducing the bit rate for disparity coding. A more correct approach (originally proposed in [START_REF] Cheung | Sparse representation of depth maps for ecient transform coding[END_REF] for the case of still images) would consist in performing transform-domain sparsication of prediction residuals, subject to the DCR constraints in the pixel domain. This is left to future work.

We implemented the modied motion estimation and mode decision in the JM reference software implementation (v. 18) of H.264/AVC. Figure 5 shows the rate-distortion curve obtained by coding 5 frames of the Kendo video sequence. With τ = 5 we have a gain in PSNR at low rates of 0.4 dB and a rate saving of almost 60%, while with τ = 7, ∆PSNR = 0.36 and ∆Rate = 67% using Bjontegaard metric. These results are particularly encouraging since most of the gain is obtained mainly through a higher eciency in mode decision and motion estimation, with e.g. the number of skip macroblocks almost doubling when using our method. We expect even a larger gain when also the transform domain sparsication of prediction residuals will be taken into consideration jointly with mode decision and motion estimation. The PSNR is computed between the synthesized view v4 and the groundtruth center view v 4 .

Ecient Depth Map Compression based on

Lossless Edge Coding and Diusion In this context, we also observed that these smooth surfaces can be eciently approximated by interpolating the luminance values located at their boundaries, instead of using models based on piecewise-linear functions where the coecients need to be encoded [START_REF] Merkle | The eects of multiview depth video compression on multiview rendering[END_REF]. To this end, we can observe that depth maps share similarity to cartoonimages. Mainberger et al. [START_REF] Mainberger | Edge-based image compression with homogeneous diusion[END_REF] proposed a dedicated cartoon-image encoder, that -in low bitrate conditions-beats the JPEG-2000 standard. After a Canny edge detection, the edge locations are encoded with a lossless bi-level encoder, and the adjacent edge pixel values are lossy quantized and subsampled. At the decoding stage, an homogeneous diusion is used to interpolate the inside unknown areas from lossy decoded edges. Indeed, the demonstrated performances -while beating state of the art codecsreach the limit of 30dB. We revisited this edge-based compression method by proposing improvements to t the high quality, low bitrate, and specic requirements of depth maps. Finally, we greatly increase the diusion-based depth map encoding performance, which might be generalized to all kinds of images. In the next section, the encoding process is described. In section III the new decoding and diusion methods are explained. Results, performances and comparison with state-of-the-art methods are given in Section IV. Conclusions are then drawn in section V.

Encoding

The encoding is a 3 step process: rst is detection of edges, then encoding of the edge location and nally encoding of the edge, border and seed pixel values.

Edge detection

Dierent operators exist to extract the contour of an image. An optimal edge detector should provide:

• a good detection: the algorithm should nd as much real edges as possible.

• a good localization: the edges should be marked as edges as close as possible to the real edges.

• a good robustness: as much as possible, the detector should be insensitive to noise.

In our context of depth map edge coding, several requirements are added. The quality of reconstruction by diusion should be maximized, while minimizing the number of edges that will be needed for later diusion. In order to maximize the quality of edges for diusion, the localization of contours should be quasi-perfect (see section III for explanation). The detection of contours should be good but avoiding an over-detection.

Up to a certain limit, weak contours (i.e. with a low gradient) might be useless to the reconstruction and might unnecessarily increase the edge coding cost. Also, noisily detected pixels should be avoided for the same reason.

The Marr-Hildreth edge detector combined with Canny like hysteresis thresholding is used in [START_REF] Mainberger | Edge-based image compression with homogeneous diusion[END_REF], but suers from error of localization at curved edges. The widely used Canny edge detector has also been benchmarked. It relies on a 5x5 gradient preltering to cope with noise before local maxima edge detection. This preltering step also makes this detector vulnerable to contour localization errors, as illustrated in Fig. 6(c), where inexact selection of adjacent edge pixels lead to improper diusion.

Oppositely Sobel has the advantage of an accurate contour localization as shown in Fig. 6(d) at the cost of noisy detection. Pixels with a bi-dimensional gradient amplitude larger than a threshold λ are extracted. To cope with the edge noise, contours c shorter than a certain value (c < 14) are excluded. Used with sharp depth maps, this gives well-localised contours.

Encoding the contour location

As in [START_REF] Mainberger | Edge-based image compression with homogeneous diusion[END_REF], a bi-level edge image containing the exact location of previously detected edges is rst encoded using the JBIG (Joint Bi-level Image Experts Group) standard. This is a context-based arithmetic encoder enabling lossless compression of bi-level images. We use the JBIG-Kit [START_REF]Joint bi-level image experts group: Information technology -progressive lossylossless coding of bi-level images[END_REF], a free C implementation of the JBIG encoder and decoder. The progressive mode is disabled to reduce the required bitrate.

Encoding the contour values

Once the edge pixel locations have been encoded, the pixel luminance values have also to be losslessly encoded following our initial requirements. The authors in [START_REF] Mainberger | Edge-based image compression with homogeneous diusion[END_REF] proposed to store the pixel values on both sides of the edge, instead of the pixel values lying on the edge itself. Indeed, for blurry contours, this might be valuable to interpolate the inner part of the edge and code the luminance values on both sides. However, with sharp depth maps, the pixel values lying directly on an edge, as illustrated in Fig. 6(b), alternate between one side or another from this edge and couldn't be interpolated correctly.

With the Sobel edge detection not thinned to a single edge pixel, we ensure to retain at least one pixel value from each side of the frontier edge (Fig. 6(d)).

We keep the idea of storing the pixel values by their order of occurrence along the edge to minimize signal entropy. A path with x directional priorities (E, S, W, N, NE, NE SE, SW and NW) is used. As the intrinsic properties of pixels along an edge or isophote are their small luminance variation, then we propose to compute the dierential values of edge pixels in a Dierential Pulse Code Modulation (DPCM) way. From this optimized path encoding method, the stream of DPCM values is then encoded with an arithmetic coder.

Additionally to these edges we also encode two kinds of information. The pixel values from the image border are stored to initiate the diusion-based lling from borders.

Inspired by the work of [START_REF] Belhachmi | How to choose interpolation data in images[END_REF] on dithering for nding optimal data for interpolation, we propose to sparsely deploy, at regular intervals, some seeds of original depth pixels as shown in Fig. 6(e). While having low overhead, we discovered that this helps accurate reconstruction by initializing the diusion in large missing areas. As we will see, it additionally greatly speeds up the diusion process.

Thus, these extra border and seed pixels are coded in DPCM and added to the dierential edge values. The resulting le is thus composed of the payload of the JBIG data and of the arithmetic encoded bitstream of the DPCM edge, border, and seed pixel values.

Decoding and diusion

Decoding contour location and pixel values

Once the edge location from JBIG payload is decoded, the edge pixel values are decoded and positioned following the same order in which they were encoded: the path along contour location respecting directional priorities.The border and seed values are also re-positioned following a predened location.

Reconstructing the missing values by diusion

We now have a sparse depth map containing only the edge, border and seed pixel

values. An homogeneous diusion-based inpainting approach is used to interpolate the missing data. This method is the simplest of the partial dierential equations (PDEs) diusion method, and has the advantage of low computational complexity. It directly stems from the heat equation:

I t=0 = I δI δt = ∆I
where I is the decoded edge image before diusion that will constitute the Dirichlet boundaries of the equation. The diused data then satises the Laplace equation ∆I = 0. The diusion process is run in a hierarchical manner, each diusion step being in addition helped with seeds and appropriate initialization. These three improvements have been introduced in the classical diusion approach to limit the number of iterations required to converge, hence to speed up the entire process:

Hierarchical diusion A Gaussian pyramid is built from I. The diusion process is rst performed on a lower level of the pyramid and the diused values are then propagated to a higher level (3 levels are used and shown good performance). The propagation of the blurred version of the diused pixel values from an lower level to an upper one helps to initialize the diusion in unknown areas.

Middle range initialization On the highest level, instead of starting from unknown value of I set at 0, we propose to initialize unknown values to the half of the possible range: 128 for an 8 bit depth map. This facilitates and speeds up the process of diusion by limiting the number of required iterations to converge.

Seeding As explained in section 3.2.3, some seeds are chosen from a regular pattern both to accelerate the diusion process and to provide accurately initialized values in large unknown areas. Indeed, this denitely achieves a fast and accurate diusion -with a gain of 10 dB-for a quasi-exact reconstruction of the depth map.

Experiments

Conditions

The performances of the proposed compression method are evaluated on an original resolution depth map from a Multiview Video-plus-Depth (MVD) sequence Breakdancers from Microsoft [START_REF] Zitnick | High-quality video view interpolation using a layered representation[END_REF]. The depth maps were estimated through a color segmentation algorithm. The choice of this sequence is motivated by the presence of sharp edges of objects on depth maps. Most other raw MVD sequences might be suitable once they would be sharpened i.e. post-processed with a bilateral lter, as it is often required in practice [START_REF] Smirnov | 3d video processing algorithmspart i[END_REF].

Depth map quality evaluation

The reconstruction quality of our PDE-based method is investigated and compared with the JPEG and JPEG2000 compressed versions. First, to illustrate the dierence (a) of quality reconstruction on edges, the three methods are compared at equal Peak-Signal-to-Noise-Ratio (PSNR),(46 dB, JPEG with a Quality factor Q=75, JPEG2000-Q=25). A zoom on the head of a character presenting initially sharp edges, highlights the dierence of edge quality depending on the compression type (Fig. 7). While at high PSNR, the JPEG (a) and JPEG2000 (b) versions of the depth map tend to blur the edges. This is commonly referred to as ringing artifacts. It appears on JPEG because of the lossy quantization of discrete cosine transform coecients within 8x8 pixels blocks. The same but less pronounced ringing eect appears on JPEG2000 at equal PSNR, due to lossy quantization following wavelet transformation. In addition, JPEG and JPEG2000 cannot eciently reconstitute the smooth gradient on uniform areas. At the opposite, our proposed approach stores the exact edges and interpolated regions between these edges, resulting in a smooth gradient restitution on slanted surfaces and non distorted edges.

(b) (c) (d) (e) (f) (g) (h)
Thus we evaluate the global depth-map rate-distortion performances of the three encoding methods. Fig. 8 shows that our approach outperforms JPEG2000 except in very low or high bitrate conditions. An adjustment of the number and location of seeds in these conditions might however improve the performance.

View synthesis quality evaluation

The impact of depth compression methods on rendering is measured by calculating the PSNR of a synthesized view (from a pair of uncompressed textures and compressed depth maps), with respect to an original synthesized view (from a pair of uncompressed textures and depth maps). The corresponding synthesized view from two original depth maps is then the reference. VSRS 3.0 [START_REF]Reference softwares for depth estimation and view synthesis[END_REF] is used for view interpolation from this 2-view dataset. The R-D synthesis performance, illustrated in Fig. 9, justies the edge-coding approach: undistorted edges permits an accurate and ecient view coding and rendering. Again, the PSNR measure shows its limitation of objective evaluation on perceived quality. Even at equal PSNR, our synthesized view (Fig. 7h) not only outperform in term of bitrate the existing methods (Fig. 7e,f ), but also improved the view rendering perceived quality.

Conclusion

We proposed a new method for lossless-edge depth map coding based on optimized path and fast homogeneous diusion. Our method, combined with a Sobel edge detection, provides a simple but ecient compression of edges that enables perfect restoration of the contours in depth maps. Also, it outperforms the state of the art JPEG and JPEG2000 compression methods. Thanks to careful edge selection and seeding, we also manage to increase the quality reconstruction of previous works based on edge image coding. Also, this lossless edge coding method could be locally applied to color image compression, especially on uniform areas. In this case the edge detection method should probably be optimized depending on edge smoothness. Finally, a depth map video encoder is in our scope for future research. 4 Layered Depth Image Representations for improved virtual view synthesis in a rate-constrained context.

Introduction

A multi-view video is a collection of video sequences captured for the same scene, synchronously by many cameras at dierent locations. Associated with a view synthesis method, a multi-view video allows the generation of virtual views of the scene from any viewpoint [START_REF] Buehler | Unstructured lumigraph rendering[END_REF][START_REF] Zitnick | High-quality video view interpolation using a layered representation[END_REF]. This property can be used in a large diversity of applications [START_REF] Smolic | Coding algorithms for 3dtv -a survey[END_REF], including Three-Dimensional TV (3DTV), Free Viewpoint Video (FTV), security monitoring, tracking and 3D reconstruction. The huge amount of data contained in a multi-view sequence needs an ecient compression [START_REF] Merkle | Ecient prediction structures for multiview video coding[END_REF].

However, the compression algorithm is strongly linked to the data representation as well as to the view synthesis methods. View synthesis approaches can be classied in two classes. Geometry-Based Rendering (GBR) approaches use a detailed 3D model of the scene. These methods are useful with synthetic video data but they become inadequate with real multi-view videos, where 3D models are dicult to estimate.

Image-Based Rendering (IBR) approaches are an attractive alternative to GBR. Using the acquisition videos accompanied by some low-detailed geometric information, they allow the generation of photo-realistic virtual views.

The Layer Depth Image (LDI) representation [START_REF] Shade | Layered depth images[END_REF][START_REF] Yoon | A framework for representation and processing of multi-view video using the concept of layered depth image[END_REF] is one of these IBR approaches.

In this representation, pixels are no more composed by a single color and a single depth value, but can contain several colors and associated depth values. It extends the 2D+Z representation, but instead of representing the scene with an array of depth pixels (pixel color with associated depth values), each position in the array may store several depth pixels, organised in layers. This representation is shown in Figure 10.

I-LDI Layers turn out to be more compact, with a less spread pixel distribution, and thus easier to compress than LDI, while enabling visual rendering of similar quality as with classical LDI. It eciently reduces the multi-view video bitrate while enabling photo-realistic rendering, even with complex scene geometry.

This representation reduces eciently the multi-view video size, and oers a fast photo-realistic rendering, even with complex scene geometry. Various approaches to LDI compression have been proposed [START_REF] Duan | Compression of the layered depth image, Image Processing[END_REF][START_REF] Yoon | A framework for representation and processing of multi-view video using the concept of layered depth image[END_REF][START_REF] Yoon | Coding of layered depth images representing multiple viewpoint video[END_REF], based on classical LDI's layers constructions [START_REF] Cheng | Generation of layered depth images from multi-view video[END_REF][START_REF] Yoon | A framework for representation and processing of multi-view video using the concept of layered depth image[END_REF]. The problem is that layers generated are still correlated, and some pixels are redundant between layers.

This paper addresses the problem of LDI construction aiming at improving the trade-o between compactness or compression eciency and original or virtual view rendering quality. The rst construction method is incremental, hence the name incremental LDI (I-LDI). The iterative construction procedure allows reducing the interlayer correlation. The number of layers is signicantly reduced for an equivalent nal rendering quality. Techniques are then described to overcome visual artifacts, like sampling holes and ghosting artifacts [START_REF] Cheng | Generation of layered depth images from multi-view video[END_REF][START_REF] Müller | Reliability-based generation and view synthesis in layered depth video[END_REF][START_REF] Zitnick | High-quality video view interpolation using a layered representation[END_REF]. The I-LDi construction method allows reducing the correlation between layers. However, artifacts remain which result from depth discontinuities, in particular after depth map compression.

Color Depth Synthesized from Ballet [START_REF] Zitnick | High-quality video view interpolation using a layered representation[END_REF], views 435 at t = 0, with incremental method [START_REF] Jantet | Incremental-ldi for multi-view coding[END_REF].

In order to further improve the trade-o between the compactness of the representation and the quality of the view synthesis, a novel object-based LDI representation is then proposed. This representation organises LDI pixels into two separate layers (foreground and background) to enhance depth continuity. The number of layers inside a LDI is not the same for each pixel position. Some positions may contain only one layer, whereas some other positions may contain many layers (or depth pixels). If several depth pixels are located at the same position, the closest belongs to the foreground, visible from the reference viewpoint, whereas the farthest is assumed to belong to the background. If there is only one pixel at a position, it is a visible background pixel, or a foreground pixel in front of an unknown background. The construction method makes use of a background-foreground segmentation method based on a region growing algorithm. Once the foreground/background classication is done, the background layer is most of the time not complete. Some areas of the background may not be visible from any input view. To reconstruct the corresponding missing background texture, the construction method then uses inpainting algorithms on both texture and depth map images. The advantage is that the costly inpainting algorithm is processed once, during the LDI classication, and not during the synthesis of each view. The resulting object-based LDI representation leads to good compression eciency due to the fact that depth pixels from a real 3D object belong to the same layer, increasing the compression eciency thanks to higher spatial correlation, hence eective spatial prediction of texture and depth map. Moreover, these continuous layers can be rendered eciently (in terms of both speed and reduced artifacts) by using mesh-based rendering techniques.

LDI representations: Background

The concept of LDI has rst been introduced in [START_REF] Shade | Layered depth images[END_REF], for complex geometries. An LDI contains potentially multiple depth pixels at each discrete location in the image. More precisely, given a set of viewpoints and one depth map per view, the classical algorithm for constructing a LDI [START_REF] Cheng | Generation of layered depth images from multi-view video[END_REF][START_REF] Yoon | A framework for representation and processing of multi-view video using the concept of layered depth image[END_REF] proceeds in three steps, as summarized in Figure 12. First, an arbitrary viewpoint is chosen as the reference viewpoint. This reference viewpoint is usually chosen among input viewpoints, but this is not an obligation. Then, each input view is warped onto this reference viewpoint using a DIBR method. Finally, all these warped views are merged into a single LDI model, where each pixel position may contain many layered depth pixels. There are many merging policies depending on the application. Keeping all depth pixels results in unnecessarily highly redundant layers. It is preferable to keep at each pixel location, only pixels whose depth value signicantly diers from that of the others. We use a threshold ∆ d on the depth value to eliminate pixels with very similar depth values.

The size of the representation grows only linearly with the observed depth complexity in the scene. The number of layers is not limited by the denition, and depends on the complexity of the scene. In practice, an LDI is often limited to a small number of layers. The rst three layers of such a LDI are shown in Figure 13. Layered pixels are ordered according to their depth value. The rst layer is composed of pixels with smallest depth, the second layer contains pixels with second smallest depth, and so on. We observe that, except for the rst one, layers are partially empty, but nonempty pixels are sparsely distributed all over the layer. Furthermore, many pixels are redundant between the layers. These characteristics make it dicult to eciently compress the LDI.

In [START_REF] Müller | Reliability-based generation and view synthesis in layered depth video[END_REF], the authors propose to project the reference view onto each additional viewpoint, in order to recover disoccluded textures, so called residual data. These disoccluded depth-pixels can either be preserved onto their original viewpoint, or projected onto the reference viewpoint to be included into the LDI. The temporal extension of LDI is called LDV, for Layered Depth Video.

An alternative construction algorithm, that we call I-LDI (for Incremental LDI), is described in Section 4.3.1. This incremental algorithm reduces the number of pixels inside an LDI while enhancing the nal rendering quality. A layer reorganization algorithm is proposed in Section 4.5, improving synthesized virtual views quality, in a rate-constrained context. Pixels from each layer are reorganized to enhance depth continuity. A compression method is also proposed which exploits temporal correlations ant inter-layer correlations.

Incremental Layer Depth Image (I-LDI) Representation

This section introduces the incremental construction scheme, based on extra information extraction [START_REF] Müller | Reliability-based generation and view synthesis in layered depth video[END_REF]. The algorithm is incremental, which means that input views are treated sequentially, in a xed order. Figure 14 illustrates the incremental construction scheme.

I-LDI construction algorithm

The algorithm starts with an empty LDI, for which the reference viewpoint is chosen freely. The reference view is used to create an I-LDI with only one layer (the view itself ). The reference viewpoint is often one of the input viewpoints, but is sometimes an intermediate viewpoint between two input views. Then, this I-LDI is warped iteratively on every other viewpoint (in a xed order). The synthesized view is then compared with the original acquired view, and the discovered information is isolated, using a logical exclusion dierence between the real view and the warped I-LDI to compute the residual information. This discovered information is warped back into the reference viewpoint and inserted in the I-LDI layers. By this method, only required residual information from side views is inserted, and no pixels from already dened areas are added to the L-LDI. On the other hand, all the information present in the MVD data is not inserted in the I-LDI.

The rst three layers of such an I-LDI are presented in Figure 15. Compared to LDI layers, I-LDI layers contain fewer pixels, and these pixels are grouped in connected clusters. Indeed, with this method, only required extra (or residual) information from side views is inserted, and no pixels from already dened areas are added to the I-LDI.

On the other hand, all the information present in the MVD data is not inserted in the I-LDI, reducing the correlation between layers.

The ordering of input views and the rendering method used to synthesize virtual views from the I-LDI are factors which have an important impact on the compactness of the representation and on the quality of the synthesized views. The rst input view to be used in the I-LDI construction is the one which will contribute the most to the nal I-LDI. This view should thus be the one for which the camera viewpoint is the nearest to the reference viewpoint of the I-LDI. The reference viewpoint is very often chosen as one of the input cameras, in this case, the rst layer of the I-LDI is almost identical to the reference view. The only dierence is located around depth discontinuities, where the JPF method uses a condence scoring to remove ghosting The rendering method used at each iteration to synthesize virtual views from I-LDI, should verify some properties. A classical point based projection generates three kinds of artifacts: disocclusions, cracks and ghosting artifacts. We use the JPF method to handle cracks without the need of an additional ltering step. The JPF method also lls in disocclusions.

Warping

In this section, we explicit the equations used in the warping process. Let (X, Y, Z, 1) be a 3D point in homogeneous coordinates, which is projected onto pixel p 1 = (x 1 , y 1 , 1) in view V 1 and pixel p 2 = (x 2 , y 2 , 1) in view V 2 . Pixel coordinates p i in view V i are derived from the projection equations:

ω i •   x i y i 1   =   C i 1,1 C i 1,2 C i 1,3 C i 1,4 C i 2,1 C i 2,2 C i 2,3 C i 2,4 C i 3,1 C i 3,2 C i 3,3 C i 3,4   ×     X Y Z 1     (7)
where C i is the 3 × 4 projection matrix depending on the viewpoint i, and ω i is an arbitrary scale factor. Knowing both camera parameters and the depth map Z p1 associated to view V 1 , the warping equations provide p 2 coordinates as a function of p 1 and Z p1 . Warping algorithm works in two steps. The rst step uses p 1 and Z p1 to estimate the 3D point (X, Y, Z p1 ). The second step uses this estimated 3D point to evaluate the pixel position p 2 in the new viewpoint image.

To solve the rst step, we need to inverse the projection equation [START_REF] Buehler | Unstructured lumigraph rendering[END_REF]. Let (L 1 ), (L 2 ) and (L 3 ) be the three linear equations corresponding to the matrix notation [START_REF] Buehler | Unstructured lumigraph rendering[END_REF] which are combined as follows:

(C 1 2,2 • 1 -C 1 3,2 • y 1 ) • (L 1 ) + (C 1 3,2 • x 1 -C 1 1,2 • 1) • (L 2 ) + (C 1 1,2 • y 1 -C 1 2,2 • x 1 ) • (L 3 ) (8)
Unknown parameters Y and ω 1 can then be eliminated by simplifying equation [START_REF] Cheng | Generation of layered depth images from multi-view video[END_REF] giving:

X • det C 1 .,1 C 1 .,2 p 1 + Z p1 • det C 1 .,3 C 1 .,2 p 1 + det C 1 .,4 C 1 .,2 p 1 = 0 (9)
where C 1 .,i is the i th column of the C 1 matrix. A direct form for the point's abscissa X is given by equation ( 9), and the same kind of equation could be written to estimate Y .

Compared to a classical matrix inversion, some coecients of the equation ( 9) only depend on p 1 and do not change during warping of all layered depth pixels at a same pixel location. By implementing this optimization, we reduce by 49% the number of multiplications needed to warp a full LDI, and almost as much for it's time consumption.

Each pixel is warped independently of the others. To avoid the use of a depth buer, we implemented the McMillan's priority order list algorithm [START_REF] Mcmillan | A list-priority rendering algorithm for redisplaying projected surfaces[END_REF]. Warping results are shown in gure 17(a).

Holes lling by inpainting

Directly applying warping equations may cause some visual artifacts, due mostly to disocclusion and sampling [START_REF] Cheng | Generation of layered depth images from multi-view video[END_REF][START_REF] Müller | Reliability-based generation and view synthesis in layered depth video[END_REF][START_REF] Zitnick | High-quality video view interpolation using a layered representation[END_REF]. This section describes our simple inpainting method to ll sampling holes, visible in gure 17(a).

Let V p be the pixel color at the p position, and W p a neighborhood around p. Undened pixels p can be interpolated as:

V p = 1 k q∈Wp V q (10)
where k is the number of dened pixels within W p . Blended pixels in a view can be identied by an edge detector performed on the associated depth map. Let p be a pixel position, we estimate the depth mean dp and depth variance v p within a neighborhood W p around p. Pixels near a boundary have a high variance, but among these pixels, only those from the background side of a boundary may cause a visible artifact. We then remove pixels p such v p > ∆ v and d p > dp where ∆ v is a threshold.

The result of our ghosting removal method is visible in gure 18. The silhouette behind the person is erased.

Onject-based Layer Depth Image (O-LDI) Representation

In order to overcome artifacts which result from depth discontinuities, in particular after depth map compression, a novel object-based LDI representation is proposed. This representation organises LDI pixels into two separate layers (foreground and background) to enhance depth continuity. If depth pixels from a real 3D object belong to the same layer, then compression is more ecient thanks to higher spatial correlation which improves eective spatial prediction of texture and depth map. Moreover, these continuous layers can be rendered eciently (in terms of both speed and reduced artifacts) by using mesh-based rendering techniques.

The number of layers inside a LDI is not the same for each pixel position. Some positions may contain only one layer, whereas some other positions may contain many layers (or depth pixels). If several depth pixels are located at the same position,

Background-Foreground segmentation

This section presents a background-foreground segmentation method based on a region growing algorithm, which allows organising LDI's pixels into two object-based layers.

First, all positions p containing several layers are selected from the input LDI. They dene a region R, shown in Figure 19, where foreground and background pixels are easily identied. Z F G p denotes foreground depth, and Z BG p denotes background depth at position p. For each position q outside the region R, the pixel P q has to be classied as a foreground or background pixel.

The classied region grows pixel by pixel, until the whole image is classied, as shown in Figure 20. For each couple of adjacent positions (p, q) around the border of region R such that p is inside R and q is outside R, the region R is expanded to q by classifying the pixel P q according to its depth Z q . For classication, Z q is compared to background and foreground depths at position p. An extra depth value is then given to position q, so that q is associated with both a foreground and a background depth value.

P q ∈          foreground if (Z BG p -Z q ) > (Z q -Z F G p ) so Z F G q = Z q and Z BG q = Z BG p background if (Z BG p -Z q ) < (Z q -Z F G p ) so Z F G q = Z F G p and Z BG q = Z q

Background lling by inpainting

Once the foreground/background classication is done, the background layer is most of the time not complete (see Figure 20(b)). Some areas of the background may not be visible from any input view. To reconstruct the corresponding missing background texture, one has to use inpainting algorithms on both texture and depth map images. The costly inpainting algorithm is processed once, during the LDI classication, and not during each view synthesis. Figure 21 shows the inpainted background with Criminisi's method [START_REF] Criminisi | Object removal by exemplar-based inpainting[END_REF].

Compression

Both classical LDI, I-LDI and object-based LDI are compressed using the Multi-view Video Codec (MVC) [START_REF] Tanimoto | Reference softwares for depth estimation and view synthesis[END_REF], both for texture layers, and for depth layers. The MVC codec, an amendment to H.264/MPEG-4 AVC video compression standard, is DCTbased and exploits temporal, spatial and inter-layer correlations. However, MVC does not deal with undened regions on LDI layers. To produce complete layers, each layer is lled in with pixels from the other layer, at the same position, as shown in

View Synthesis

There exists a number of algorithms to perform view rendering from a LDI. This section briey presents the two methods which have been implemented, focusing respectively on eciency and quality.

The fastest method transforms each continuous layer into a mesh, which is rendered with a 3D engine, as shown in Figure 23. The foreground mesh is transparent on background region in order to avoid stretching around objects boundaries. Our rst experiments, with this method, have shown the feasibility of real time rendering for an eight-views auto-stereoscopic display.

The second method improves the visual quality of synthesized views by using a point-

Experimental results

Experiments have been conducted on Breakdancers and Ballet data sets from MSR [START_REF] Zitnick | High-quality video view interpolation using a layered representation[END_REF].

Parameters of the 8 acquisition cameras and all associated depth maps are already estimated and provided within the data. The viewpoint number 4 is considered as the reference viewpoint. Only frames for time t = 0 are considered.

For the LDI construction, all 8 acquired views are warped into the reference viewpoint. A small merging threshold value ∆ d = 0.1 is used in following comparisons. For the I-LDI construction, views are inserted in a B-hierarchical order (4; 0; 7; 2; 6; 1; 5; 3). Thresholds are set by experiments: ∆ v = 20 for boundary detection and ∆ k = 60% • N for inpainting, where N is the number of pixels within the W p window. Both inpainting and boundary detection are done within W p a 11 × 11 window.

All 8 input views are used, but all pixels from each view are not inserted in the LDI.

Because of the depth threshold in the LDI construction, and of the exclusion dierence in I-LDI construction, some pixels from side views are ignored. Figure 25 presents the ratio of pixels from each view which is really inserted in the LDI. We can observe that few pixels are inserted from view 5 to 8, means these views become almost useless with the I-LDI construction scheme. Using only a subset of acquired views (the reference and the extreme views) provides almost the same I-LDI layers.

Figure 26 shows the ratio of dened pixels in each layers for both LDI and I-LDI construction schemes. For both constructions, the rst layer contains 100% of it's (a) Classical LDI.

(b) Object-based LDI. In the rst place, a LDI restricted to two layers, is constructed from three input views: the reference view 4 and side views 3 and 5 alternatively. To deal with unrectied camera sets and reduce correlation between layers, we use the Incremental LDI construction algorithm described in [START_REF] Jantet | Incremental-ldi for multi-view coding[END_REF]. The corresponding object-based LDI is obtained by applying our region growing classication method on the classical LDI.

Classical LDI and object-based LDI are compressed using the MVC algorithm, as explained in section 4.7. Several quantization parameters were used, from QP=18 to QP=54, producing compressed output data ows with bit-rates going from 1 Mbit/s to 25 Mbit/s. These compressed data ows are used to synthesize virtual views onto viewpoint 6, using the pixel-based projection method.

In the second place, the state-of-the-art method for multi-view video coding is used with the same input data. Views 1, 3, 5 and 7 are coded with the MVC algorithm with various quantization parameters, then the compressed views 5 and 7 are used to synthesize virtual views onto viewpoint 6, using the MPEG/VSRS software [START_REF] Tanimoto | Reference softwares for depth estimation and view synthesis[END_REF]. Finally, all synthesized views are compared to the original view 6, using the SSIM comparison metrics. Figure 27 presents all the results as three rate distortion curves.

For each quantization parameter, object-based LDI can be better compressed than classical LDI, resulting in a smaller bitrate. The rendering quality is also better, resulting in a higher SSIM for the same quantization parameter. Combining these two advantages, the rate distortion curve for the object-based LDI is higher than the one for classical LDI, for every bitrate.

Conclusions and future work

This paper has presented two methods for constructing layered depth image representations from natural multi-view images. The rst method is an incremental procedure which uses iterative warping. The minimum information to ll disocclusion areas is inserted into LDI layers which makes layers easier to compress. They contain 80% less pixels, and with a more compact distribution. To overcome visible artifacts, some simple solutions have been proposed. The sampling holes lling by pixel interpolation provides good results. The ghosting removal by depth discontinuity detection may cause some luminosity discontinuity between textures from dierent views.

In future work, we will investigate an improved disocclusion detection to insert into the I-LDI all occluded textures. An alpha merging approach will be used to reduce luminosity discontinuity artifacts. Finally, the compression stage will be investigated with a full video sequence. [START_REF] Bosc | Bit-rate allocation for multi-view video plus depth[END_REF] A content based method for perceptually driven joint color/depth compression 5.1 Introduction 3D Video applications [START_REF] Smolic | 3D video and free viewpoint Video Technologies, applications and MPEG standards[END_REF], such as 3D Television (3DTV) or Free Viewpoint Television (FTV), require the use of several conventional video sequences to ensure depth sensation, or to oer novel views of a scene. For these purposes, the use of color and geometry information of the scene is the key. MVD data refer to a specic representation of an observed scene and meet this need. They consist in a set of conventional color video sequences and an associated set of depth video sequences, all acquired at slightly dierent viewpoints.

A rst issue refers to the need for an ecient MVD compression method, considering the huge amount of data to be processed. Up to now, there is no standardized compression method for MVD sequences. Most of the proposed compression methods rely on the extension of state-of-the-art 2D codecs. The most popular is H264/AVC [START_REF] Sullivan | The h. 264/AVC advanced video coding standard: Overview and introduction to the delity range extensions[END_REF] whose 3D extension (standardized for Multi-View-Video representation, MVV), namely H.264/MVC for Multi-view Video Coding [START_REF] Merkle | Ecient compression of multiview video exploiting inter-view dependencies based on h. 264/MPEG4-AVC[END_REF], has been the subject of many adaptations for MVD compression [START_REF] Muller | Coding and intermediate view synthesis of multiview video plus depth[END_REF]. However, the exploitations of the spatial inter-view redundancies in both types of data turn out to be insucient in particular cases. For instance, Merkle et al. [START_REF] Merkle | Ecient prediction structures for multiview video coding[END_REF] observed that in case of large disparity between the dierent views of multi-view sequences, the predictions structures did not result in an improved coding eciency.

A second issue refers to the synthesis of novel views from decoded data. New intermediate viewpoints can be generated from depth and color data through Depth-Image-Based-Rendering [START_REF] Fehn | Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV[END_REF] (DIBR) methods. Previous studies already pointed out the impact of depth encoding on the synthesized frames. Compression-related artifacts that may be imperceptible in depth maps cause important distortion during the synthesis process [START_REF] Merkle | The eects of multiview depth video compression on multiview rendering[END_REF].

Many methods have been proposed recently in order to address the aforementioned issues. Various encoding strategies are possible to achieve depth map compression.

Several studies have proposed bit-rate-control methods [START_REF] Daribo | Motion vector sharing and bitrate allocation for 3D Video-plus-Depth coding[END_REF][START_REF] Morvan | Joint depth/texture bit-allocation for multi-view video compression[END_REF] relying on the objective quality of the resulting synthesized views, or on a distortion model [START_REF] Liu | Joint video/depth rate allocation for 3D video coding based on view synthesis distortion model[END_REF]. A popular and ecient strategy is the post-processing of depth maps after decoding [START_REF] Silva | A depth map Post-Processing framework for 3D-TV systems based on compression artifact analysis[END_REF]. Depthadapted encoding methods [START_REF] Graziosi | Compressing depth maps using multiscale recurrent pattern image coding[END_REF][START_REF] Morvan | Platelet-based coding of depth maps for the transmission of multiview images[END_REF][START_REF] Sarkis | Depth map compression via compressed sensing[END_REF] have also been proposed. Section 5.2 gives a review of these methods. Our work is in line with the depth-adapted encoding strategy since the method proposed in this paper relies on a content-based representation of the depth map.

The main purpose of this novel framework is to preserve the consistency between color and depth data. Our strategy is motivated by previous studies [START_REF] Merkle | The eects of multiview depth video compression on multiview rendering[END_REF] of artifacts occurring in synthesized views: most annoying distortions are located around strong depth discontinuities [START_REF] Bosc | Towards a new quality metric for 3D synthesized view assessment[END_REF] and these distortions are due to misalignment of depth and color edges in decoded images. Thus the method is meant to preserve edges and to ensure consistent localization of color edges and depth edges. It is based on a 2D still image codec, namely LAR [START_REF] Pasteau | Locally adaptive resolution (LAR) codec[END_REF] (Locally adapted Resolution). The LAR codec is based on a quad-tree representation of the images. In this quad-tree, the smaller the blocks, the higher the probability of the presence of a depth discontinuity. Analogously, big blocks correspond to smooth areas. The quad-tree representation contributes in the preservation of depth transitions when target bit-rate decreases. Another original contribution of the proposed method relies on the use of the decoded color data as an anchor for the enhancement of the associated decoded depth, together with information provided by the quad-tree structure. This is meant to ensure consistency in both types of data after decoding.

This paper is organized as follows: Section 5.2 introduces the compression issues for MVD data. Section 5.3 presents the proposed method. Section 5.4 denes the experimental protocol and gives the results. Finally Section 5.5 concludes the paper.

Compression of mvd sequences and quality of synthesized views

This section presents the main issues related to MVD compression and a review of the proposed methods addressing these problems in the literature.

Most of the proposed compression methods for MVD data rely on the extension of state-of-the-art 2D codecs. Sets of color and depth sequences can be separately encoded through existing 2D methods. This is an evident encoding strategy because depth maps, being monochromatic signals, are considered as conventional sequences.

However, depth maps are not natural images. They provide structural information of the scene: large and smooth regions often belong to the same depth plane. The closer the depth plane is from the acquiring camera, the lighter the region. This leads to smooth areas with sharp edges. The edges correspond to depth transitions.

Previous studies [START_REF] Bosc | Bit-rate allocation for multi-view video plus depth[END_REF][START_REF] Merkle | Multi-view video plus depth representation and coding[END_REF][START_REF] Vetro | Towards a 3D video format for auto-stereoscopic displays[END_REF] have shown that coding artifacts on depth data can dramatically inuence the quality of the synthesized view. Particularly, the sharp edges of the depth maps are prone to synthesis errors even when depth maps are uncompressed.

As pointed out in a recent study [START_REF] Bosc | Towards a new quality metric for 3D synthesized view assessment[END_REF], the synthesis process, with DIBR methods, induces specic artifacts located around the edges of objects. These errors are notably due to depth map inaccuracy, numerical rounding, hole lling method in DIBR, or both.

Consequently, errors occurring in these specic critical areas of the depth maps are enhanced by coarse compression. The impacts of depth compression on visual quality of synthesized views can be explained by the fact that 2D codecs are optimized for human visual perception of color images. Thus, artifacts, that may be imperceptible when visualizing the depth map, produce distortions because during the synthesis, the warping process relies on wrong depth values. The impacts of depth compression was observed in dierent studies [START_REF] Bosc | Bit-rate allocation for multi-view video plus depth[END_REF][START_REF] Merkle | The eects of multiview depth video compression on multiview rendering[END_REF].

Consequently, eorts have been directed in order to propose depth compression methods more adapted to the special features of depth maps. Morvan et al. [START_REF] Morvan | Platelet-based coding of depth maps for the transmission of multiview images[END_REF] proposed to represent the depth map thanks to platelets (piecewise linear functions).

The depth map is rst divided through quad-tree decomposition and each block is approximated by a platelet. The platelet-based compression outperformed JPEG2000 in the study. An additional interesting comparison would be that against H.264/AVC.

Moreover, in this study, the gain is evaluated with respect to the depth distortion (in PSNR). This protocol of validation is questionable because since the artifacts in the two compared methods are dierent, their impact on the synthesis may also be dierent. Yet, the quality of the synthesized views generated from the decoded depth maps is not presented. Graziosi et al. [START_REF] Graziosi | Compressing depth maps using multiscale recurrent pattern image coding[END_REF] also proposed a block partioning method associated to a least-square prediction for depth map compression. In this method, the validation is also achieved by comparing the depth map distortion from dierent compression scheme (JPEG2000 and H.264 intra). The method includes the use of a dictionary, containing concatenations of scaled versions of previously encoded image blocks. Sarkis et al. [START_REF] Sarkis | Depth map compression via compressed sensing[END_REF] proposed a depth compression method based on a subsampling in the frequency domain followed by a reconstruction using a non-linear conjugate gradient minimization scheme. The method also meant to preserve the particular features of the depth map. The method outperformed JPEG and JPEG2000.

As Morvan et al. and Graziosi et al., our method relies on a block partioning. Contrary to the aforementioned methods, we choose to evaluate the performances of our method against H.264, whose artifacts in the depth map induce less annoying distortions than JPEG2000 (Gibbs eect in the depth map induce disastrous distortions in the synthesized view). The quantization used in our method is also dierent since we choose to modify the block partioning of the depth map according to the target bit-rate. The next section will present our proposed method that is meant to preserve object edges to maximize the synthesized view quality.

Proposed method

Since color and depth sequences can be encoded through two dierent schemes, the proposed method enables the use of any compression method for the color sequences.

In our case, we use H.264 for color sequence encoding since it has proved its eciency for conventional color media compression. Moreover, encoding color with a standard codec enables backward compatibility with classical 2D video. This section presents the depth map encoding strategy.

Depth map encoding method

To address the rst constraint regarding the preservation of the depth map edges, a content-based representation and encoding is required. We choose to base our method on the LAR method because the quad-tree representation of this method matches the characteristics of the depth maps. The LAR method is based on the assumption that an image can be considered as the combination of two components: the global information and the details, which are respectively the at image and the local texture.

The at image and the local texture both rely on the same quad-tree representation.

Each pixel in the at image is assigned the mean value of the pixels of the block it belongs to. Each pixel in the local texture is then assigned the compensated error. Since depth maps do not contain high frequency areas, the local texture (that is to say the details) is not essential and represents an avoidable additional cost of compres- Quad-tree decomposition The quad-tree decomposition is dependent on the local gradient of the depth image. Given a threshold Y for the local gradient, the image is split into blocks: the higher the local activity, the more splits. This leads to small blocks around object edges and bigger ones in continuous areas.

We denote P [Nmax...Nmin] the quad-tree partition with N max and N min the maximal and minimal allowed block sizes, expressed as power of 2. Let I be an image and I (x, y) a pixel of I with coordinates(x, y). The block b N (i, j) in I is noted I(b N (i, j)),

expressed as: b N (i, j) =        (x, y) ∈ N x × N y | N × i ≤ x < N × (i + 1) , N × j ≤ y < N × (j + 1) , N ∈ [N min . . . N max ] (11)
As explained before, the quad-tree partition P [Nmax...Nmin] relies on the analysis of the local gradient. Then, the decomposition can be expressed as:

Algorithm 1 Prediction of lowest level of the pyramidal decomposition Require: Ll is the estimated representation of the image at the decoder side, for level l, P [Nmax...Nmin] is the quad-tree partition. G.init(x init ) for l = l max . . . the legs of the chair). The synthesized frames obtained in Figure 30 are generated from original color data and decoded depth maps in order to visually assess only the impact of depth quantization (i.e. not the combined eect of both color and depth compression).

Thus, we propose a quantization achieved through the evolution of the quad-tree representation of the image. Small blocks are costly and a way to reduce the bit-rate is to reduce the number of small blocks. This implies that the quad-tree representation can change according to the target bit-rate. The number of small blocks is directly related to the value of the threshold Y . Thus, an increasing threshold Y decreases the bit-rate, so that the representation of the image contains larger blocks. This corresponds to a spatial quantization that aects the values of the depth. It results in assigning the same depth to object that were not formerly in the same depth plane. The dynamic range of the depth is reduced but the global structure is preserved. Figure 30 shows that the proposed method (second column) renders sharp depth transitions. The synthesized frame in Figure 30, fourth column, shows improvements compared to the previous strategy, third column. Figure 31 gives the quad-tree representations and the resulting depth maps using two dierent thresholds for the quad-tree decomposition.

It shows that the semantic information of the image is preserved.

In depth structure of the sequence.

Depth reconstruction at decoder side

To address the second constraint regarding the consistency between color and depth edges, reconstruction step is included at the depth map decoder side, right after the rst estimation of the smallest blocks, as explained in Section 5.3.1. The additional step described in this section can be considered as a second pass of the depth reconstruction.

It consists of a multi-lateral ltering aided by the quad-tree representation whose principles are partially based on the description of Lai et al. method [START_REF] Lai | Depth map processing with iterative joint multilteral ltering[END_REF]. In our proposed method, the decoded associated color image is used to enhance only the blocks smaller or equal to 4 × 4 in the depth map. Small blocks are likely to be located around depth discontinuities. Thus, it is believed that improving the accuracy in these regions, according to the decoded associated color, will ensure consistency between color and depth edges. Let C be the decoded associated color image, and L0 the lowest level image of the depth pyramidal decomposition. Let Ω be the set of pixels such as:

Ω = L0 (x, y) L0 (x, y) ∈ L0 b N (i, j) , N ∈ [N min . . . 4] (15)
The reconstruction, noted L0r (x, y), of any pixel belonging to Ω is expressed as:

Γis the pixel window used for the calculation; Luma is the luminance component of the decoded color image; Luma(p) and Luma(q) are pixels of the luminance component of the decoded color image; σ d , σ s , σ c are standard deviations related to the spatial domain, the depth range domain (similarity of depth values), and the color range domain, respectively. Figure 32 gives the overview of the proposed method. In this gure, at the encoding step, black blocks correspond to non transmitted blocks.

Experiments

Protocol

The proposed method is compared to state-of-the-art codec H.264 in intra mode. As preliminary studies, the experiments concern only still images. First frames of views 6 and 10 from Book Arrival were encoded through both encoding methods. Afterwards, decoded color and depth maps were used to compute the intermediate view 8, through

Results

For the performance comparisons, a pixel-based metric (PSNR) and a more perceptionoriented metric VIF (Visual Information Fidelity [START_REF] Sheikh | Image information and visual quality, Image Processing[END_REF]) are considered. Figure 34 depicts the rate-distortion curve obtained by computing the PSNR scores, and the VIF scores of the synthesized views, with respect to the original acquired view. At high bitrates (higher than 2bpp), the proposed method obtains better PSNR scores. However, under 2bpp, H.264 performs better.

The curve based on VIF scores shows that H.264 and the proposed method give similar results at high bit-rates (higher than 2bpp). However, contrary to the curve based on PSNR, the curve based on the perception-oriented VIF shows that the proposed method performs better at low bit-rates.

A visual appreciation is also useful to evaluate the methods. Figure 35 gives snapshots of the obtained synthesized views for 0.1bpp and 0.9bpp. Ghosting eect is perceptible with both methods behind the head of the man. However, the quad-tree based method preserves better the vertical edges: the vertical dark lines of the posters are better rendered with the data encoded with the proposed method. At low bit-rate (0.1bpp), Figure 35 gives snapshots of the synthesized views. Although, PSNR score shows lower performances for the proposed method at low bit-rate, the observation of Figure 35 shows improvements around the edges of the synthesized objects. The ghosting eect around the head of the man is less strong with the proposed method.

The crumbling artifacts occurring around the leg of the chair at 0.1bpp with H.264 are no longer perceptible with the proposed method.

Conclusion

We proposed a novel framework whose main purpose is to preserve consistency between color and depth edges. Depth encoding is based on a 2D still image codec, namely LAR (Locally Adapted Resolution). It consists in a quad-tree representation of the images.

The quad-tree representation contributes in the preservation of edges in depth data.

The originality of the proposed method relies on the proposed quantization method and the use of the decoded color data as an anchor for the associated depth enhancement at the decoder side. The proposed method showed visual performances similar to H.264 at high bit-rates and some improvements at lower bit-rates because it preserves better the object edges. Future work should focus on the use of a more perception-oriented criterion for the quad-tree decomposition. A method to choose automatically the Y model, for any sequence, should also be investigated. Finally, the method should be extended to exploit temporal redundancies in the whole sequence.

Figure 2 :

 2 Figure 2: Blur-based example of framework.

Figure 4 :

 4 Figure 4: Denition of DCR for a given threshold τ .

Figure 5 :

 5 Figure 5: RD performance on 5 frames of the Kendo sequence, with a modied H.264/AVC encoder to embed DCR, for several values of τ . The bit rate refers to the total rate required to encode disparity maps of view 3 and 5.

Figure 6 :

 6 Figure 6: (a) A Breakdancer depth map, (b) the encoded and decoded Sobel edge and seed pixels (red selection on (a)), (c) the Canny edges (blue selection), (d) the selection of pixel values adjacent to Canny edges (c) as in [20], with an intruder edge pixel in orange that will lead to bad diusion, (e) the proposed Sobel selection of edge pixel values, exactly located from both side of the frontier edge

Figure 7 :

 7 Figure 7: Upper row: zoom on the head of a dancer on original View#3 (V3) depth map (a) highlights by comparison the ringing artifact on JPEG (b) and JPEG2000 (c). Our method (d) based on exact edges and homogeneous diusion prevent this eect. (Depth maps have been sharpened for distortion visibility). Lower row: zoom on synthesized view V4 without (e) or with JPEG (f ), JPEG2000 (g) and our method (h) compressions at equal PSNR (43dB) referenced to (e).

Figure 8 :

 8 Figure 8: Rate-Distortion performance of the V3 depth map for dierent quality factors of JPEG and JPEG2000, and dierent Sobel detection thresholds of our method.

Figure 9 :

 9 Figure 9: Rate-Distortion performance of synthesized V4 with the bitrate of V3, for dierent quality factors of JPEG and JPEG2000, and dierent Sobel detection thresholds of our method.

Figure 10 :

 10 Figure 10: Two rst layers (color + depth map) of a classical LDI.

Figure 11 :

 11 Figure 11: Structure of a Layer Depth Image (LDI).

Figure 12 :Figure 13 :

 1213 Figure 12: Classical LDI construction scheme.

Figure 14 :Figure 15 :

 1415 Figure 14: Step of I-LDI construction for view i, with residual information extraction.

Figure 16 :

 16 Figure 16: Residual information obtained with 16(a) a point based view synthesis and 16(b) the JPF method. The rst column shows the rendered view of an intermediate I-LDI and the second column shows the corresponding residual information which should be inserted into the I-LDI.

  This inpainting solution is used both during the rendering stage, and during the I-LDI construction. During the rendering stage, it improves the visual quality by interpolating all missing pixels. During the I-LDI construction, it is used carefully to ll only the sampling holes, and to leave disocclusion areas unchanged. Results are shown in gure 17(b). If a pixel is undened due to a sampling eect, it should be surrounded by many dened pixels, which mean a high k value. If the pixel is undened due to a large disocclusion area, it should be surrounded by many undened pixels, which mean a low k value. The classication is done by comparing k with a threshold ∆ k .(a) Basic Warping (b) Sampling holes lled

Figure 17 :

 17 Figure 17: View warping and sampling holes lling

  (a) Without boundaries detection. (b) With boundaries detection.

Figure 18 :

 18 Figure 18: Ghosting artifacts removal results from rendering view. (W p is a 11 × 11 window)

Figure 19 :

 19 Figure 19: Initialising state of the region growing algorithm.

Figure 20 :

 20 Figure 20: Final layer organisation with the region growing classication method.

Figure 20

 20 Figure20shows the result of classication by region growing.

Figure 22 .

 22 Figure22. This duplicated information is detected by the MVC algorithm, so that

Figure 21 :

 21 Figure 21: Background layer obtained after texture and depth map inpainting with the Criminisi's method [10].

Figure 22 :

 22 Figure 22: Final layers of an object-based LDI.

Figure 23 :

 23 Figure 23: Fast 3D rendering of a high detailed foreground mesh, onto a low detailed background mesh.

Figure 24 :

 24 Figure 24: Rendering comparison between classical and object-based LDI.

Figure 25 :Figure 26 :

 2526 Figure 25: Utilization rate of acquired views during layers construction

  /VSRS) MVC on V.1-3-5-7 -VSRS V.6

Figure 27 :

 27 Figure 27: Rate distortion curves rstly for LDI (object-based or not) compressed by MVC and rendered by our point-based projection, and secondly for multiview video compressed by MVC and rendered with VSRS algorithm.

Figure 28 depicts

 28 Figure 28 depicts this principle.

Figure 28 :

 28 Figure 28: Assumption of LAR method.

Figure 29 :

 29 Figure 29: Quad-tree decomposition (Book Arrival).

  l 1 do Estimate Ll as in the LAR method end for for each block of P such as N = N max . . . N min do Given P [ N max . . . N min ] , then L0 (b N (i, j)) = L1 end for for each block of P such as N = N min do L0 (b ( N min )(i, j)) = Mean value of the closest block b N of P such as N > N min end for return L0

Y = ae bR ( 14 )R

 14 this study, we empirically determined a model allowing the choice of Y depending on the target bit-rate. Based on the analysis of the synthesized view quality scores obtained for various values of Y , and according to the corresponding bit-rate R of the encoded frames, we opted to an exponential model such as: is the target bit-rate, a and b are two constants. Previous experiments showed that a = 30 and b = -12 gave good results for the tested sequence. Note that these values will dier from one MVD sequence to another, since the representation of the quad-tree depends on the structure of the sequence, thus Y model depends on the ]

Figure 30 :Figure 31 :

 3031 Figure 30: Comparison of two decoded depth maps at 0.06bpp, using the LAR method or the proposed method of rate control.

Figure 32 :

 32 Figure 32: Overview of the proposed method.

Figure 33 :

 33 Figure 33: Overview of the experimental protocol

Figure 34 :

 34 Figure 34: Performance comparisons, in terms of PSNR and VIF, between the original view and the synthesized view.

The value of the threshold Y strongly inuences the nal representation of the image. Figure 29 gives an example of quad-tree decomposition for the rst frame of Book Arrival sequence.

Compression scheme The compression scheme in the LAR method is based on a pyramidal decomposition [START_REF] Babel | Interleaved s+ p pyramidal decomposition with rened prediction model[END_REF]. The pyramid, built from I, consists of a set of images, noted as {L l } l=lmax l=0

, as a multi-resolution representation of the image, where l max is the top of the pyramid and l = 0 is the lowest level, i.e. the full resolution image. At each level, the image is expressed by:

The LAR method allows the prediction of each level of the pyramid, from top to bottom. For each level, the associated image of errors, also relying on the quad-tree decomposition, can be transmitted to compensate the prediction errors. At the decoder side, from the top to the bottom, the image is reconstructed.

Compression cost is mainly due to the encoding of small blocks. This is why in our proposed method small blocks are not transmitted (those are blocks whose size is such as N = N min ). This is achievable thanks to the pyramidal decomposition. The encoding of small blocks is related to the image of errors corresponding to the lowest level, i.e. L 0 . The lowest level is not encoded in the method we propose, and the image will be rened at the decoder side thanks to the analysis of the values of the nearest neighbor blocks whose size is such as N > N min : they will be predicted, depending on the values of their closest larger blocks. This allows bit-rate savings. The pseudo code of this prediction is given in Algorithm 1.

Rate control in depth map

Pasteau et al. [START_REF] Pasteau | Locally adaptive resolution (LAR) codec[END_REF] suggested applying a quantization step depending on the block sizes, in the case of conventional images. Our experiments revealed that in the case of depth map compression, this was not an adequate strategy because the smaller the blocks, the coarser was the quantization (this allowed bit rate savings because small block are costly). Yet, small blocks correspond to strong depth discontinuities and errors occurring in these areas may have disastrous eect at the synthesis step. Figure 30 shows the impact of the quantization as suggested in Pasteau et al. [START_REF] Pasteau | Locally adaptive resolution (LAR) codec[END_REF]