Mira Rizkallah

Member, IEEE Thomas Maugey

Member, IEEE Christine Guillemot

Geometry-Aware Graph Transforms for Light Field Compact Representation

Keywords: Light Fields, Energy Compaction, Transform coding, Super-rays, Graph Fourier Transform I

 .

In this paper, we address the problem of graph transforms optimization for light fields energy compaction and compact representation. Indeed, light fields record illumination of light rays emitted by a scene in different orientations. The captured data for a static light field is represented by a 4D function LF (m, n, x, y), and contains redundant information in both the spatial and angular dimensions. Those correlations could in principle be represented by a huge non separable graph connecting pixels within and across views of the entire light field. The basis functions of a graph Fourier transform [START_REF] Shuman | The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains[END_REF] could then be used to decorrelate the color signal residing on the graph vertices. However, such a graph would have a very high number of vertices, each vertex corresponding to a light ray. This makes the diagonalization of the laplacian matrix unfeasible, hence, the computation of the graph Fourier transform not practical.

To lower the dimensionality of the problem, we propose to partition the big graph structure into smaller ones that are coherent and correlated inside and across the views. This can be viewed as cutting unreliable edges from the global graph. To perform this partitioning, we group similar pixels within and across views based on the concept of super-rays defining the supports of the set of local graph transforms. The concept of super-ray has been introduced in [START_REF] Hog | Super-rays for efficient light field processing[END_REF] as an extension to light fields of the concept of super-pixels.

The authors in [START_REF] Su | Rate-distortion optimized super-ray merging for light field compression[END_REF] used super-rays as the supports of separable shape-adaptive Discrete Cosine Transform (DCT). Super-pixels are used in [START_REF] Rizkallah | Graph-based transforms for predictive light field compression based on super-pixels[END_REF] as the supports of local graph transforms, and tested in a predictive scheme based on view synthesis. The angular transform is however applied on superpixels that are co-located on all views, hence not exploiting scene geometry, due to the difficulty to design separable graph transforms that at the same time follow the scene geometrical information and preserve angular correlations. We come back on this point in the sequel.

In this paper, we address the problem of designing local super-ray based non separable and separable graph transforms following the scene geometry. Towards this goal, we first propose a specific super-ray construction method to limit shape variations of the super-pixels forming a given superray. Despite the local support of limited size defined by the super-rays, the Laplacian matrix remains of high dimension and its diagonalization to compute the transform eigen vectors is computationally expensive. An intuitive way to solve this problem is to perform the transform in a separable manner: a first spatial transform applied per super-pixel inside each view, then an angular transform between corresponding super-pixels across the views to capture angular dependencies. We have however observed that if the shape of the super-ray undergoes a slight change between views, the basis functions computed from the graph laplacian have very different forms from one super-pixel to the corresponding ones in the other views, resulting in a decreased correlation between spatial transform coefficients.

The difficulty is therefore how to optimize the spatial transforms applied on each super-pixel of the super-ray in such a way that the angular correlation is well preserved. Preserving angular correlation is important in order to best compact the light field energy. The angular correlation is preserved, only if the eigen vectors of the spatial transforms computed independently on different shapes (the super-pixels forming the super-ray) are reasonably consistent, i.e. only when the shapes of the transform supports are approximately isometric. We propose in this paper a novel method to optimize the spatial transforms in such a way that the basis functions approximately diagonalize their respective Laplacians while being coherent across the views, given the scene geometry.

Experimental results show that the proposed super-ray construction method yields, for the light fields considered in the tests, up to 60 percent coherent supports out of all super-rays, which facilitates the application of a separable graph transform. The results also show that the optimized separable graph transform yields higher energy compaction, and significant rate-distortion performance gains, compared to the non optimized separable transform, when some superrays are shape-varying across the views. The proposed simple coding scheme based on these local separable transforms is shown to outperform light field coding schemes based on HEVC-lozenge and JPEG Pleno [START_REF] Jpeg | Jpeg pleno light field coding vm 1.1[END_REF] at high bitrate following the common test conditions.

In this paper, the contributions are as follows:

• We propose a novel light field representation approach based on geometry-aware local graph transforms with a support defined by super-rays. • We define the notion of separable graph transforms that we apply locally in each super-ray. This allows us to capture both spatial and angular dependencies inside the light field. • We develop a graph optimization method with a geometric association of nodes. We design consistent transforms for more than 2 graph supports to deal with the problem of inconsistent basis functions when the corresponding super-pixels inside different views are not isometric. The optimization is performed per group of frequency bands to reduce complexity and by fixing a reference to limit error propagation. Using the consistent transforms allows us to preserve angular correlations, and thus a good energy compaction of the separable graph transforms.

II. RELATED WORK

We first briefly review prior work on graph transforms design for signal (and in particular image) energy compaction, problem related to the core of the paper. For sake of completeness, the proposed transforms being validated in a complete coding scheme, we also give a brief overview of recent work on light field compression.

A. Graph Transforms

Recently, graph signal processing has been applied to different image and video coding applications, especially for piecewise smooth images. In [START_REF] Shen | Edge-adaptive transforms for efficient depth map coding[END_REF], [START_REF] Kim | Graph based transforms for depth video coding[END_REF], the authors propose a graph-based coding method where the graph weights are defined considering pairwise similarities between pixel intensities. Another efficient graph construction method has been proposed in [START_REF] Hu | Multiresolution graph fourier transform for compression of piecewise smooth images[END_REF] for piecewise smooth images. For each signal in a block, they select the Graph Fourier Transform minimizing the rate distortion cost. A signed graph Fourier transform has also been proposed in [START_REF] Su | Graph fourier transform with negative edges for depth image coding[END_REF] for depth map coding, accounting for negative weights between pixels.

For natural images, most of the work has focused on designing sparse graphs or using graph templates that capture principal gradient-based structures in images [20][21]. This is mostly useful in textured images. While most of the aforementioned transform coding strategies did not account for the graph coding cost, in a later work [START_REF] Fracastoro | Graph transform learning for image compression[END_REF], a rate-distortion optimized graph learning approach has been proposed to code natural images while taking into account both the sparsity of the transformed coefficients and the graph coding cost. Several graph based approaches have also been proposed to code intra and inter predicted residual blocks in video compression, using generalized graph Fourier transform [START_REF] Hu | Intra-prediction and generalized graph fourier transform for image coding[END_REF], simplified graph templates transforms [START_REF] Egilmez | Graph-based transforms for inter predicted video coding[END_REF], or separate line graph based transforms [START_REF] Lu | Symmetric line graph transforms for inter predictive video coding[END_REF].

In this paper, we build graphs that follow the scene geometry and we then propose separable graph based transforms that best exploit light fields spatial and angular correlation.

B. Light Fields Compression

Existing light fields compression solutions can be broadly classified into two categories: approaches directly compressing the lenslet images or approaches coding the views extracted from the raw data. Methods proposed for compressing the lenslet images mostly extend HEVC intra coding modes by adding new prediction modes to exploit similarity between lenslet images (e.g. [START_REF] Conti | New hevc prediction modes for 3d holoscopic video coding[END_REF], [START_REF]Hevc-based light field image coding with bi-predicted selfsimilarity compensation[END_REF], [START_REF] Conti | Hevc-based 3d holoscopic video coding using self-similarity compensated prediction[END_REF], [START_REF] Li | Compression of unfocused plenoptic images using a displacement intra prediction[END_REF]). The authors in [START_REF] Tabus | Lossy compression of lenslet images from plenoptic cameras combining sparse predictive coding and jpeg 2000[END_REF] propose a lenslet-based compression scheme that uses depth, disparity and sparse prediction followed by JPEG-2000 residue coding.

The second category of methods consists in encoding the set of views which can be extracted from the lenslet images after de-vignetting, demosaicing and alignment of the microlens array on the sensor, following e.g. the raw data decoding pipeline in [START_REF] David | White lenslet image guided demosaicing for plenoptic cameras[END_REF]. Several methods code the views as pseudo video sequences using HEVC [START_REF] Liu | Pseudosequence-based light field image compression[END_REF], [START_REF] Rizkallah | Impact of light field compression on focus stack and extended focus images[END_REF], or the latest JEM coder [START_REF] Jia | Optimized inter-view prediction based light field image compression with adaptive reconstruction[END_REF], or extend HEVC to multi-view coding [START_REF] Ahmad | Interpreting plenoptic images as multiview sequences for improved compression[END_REF]. Low rank models as well as local Gaussian mixture models in the 4D rays space are proposed in [START_REF] Jiang | Light field compression with homography-based low-rank approximation[END_REF] and [START_REF] Verhack | Steered mixture-of-experts for light field coding, depth estimation, and processing[END_REF] respectively. View synthesis based predictive coding has also been investigated in [START_REF] Zhao | Light field image coding via linear approximation prior[END_REF] where the authors use a linear approximation computed with Matching Pursuit for disparity based view prediction. The authors in [START_REF] Jiang | Light fields compression using depth image based view synthesis[END_REF] and [START_REF] Su | Graph-based light fields representation and coding using geometry information[END_REF] use instead a the convolutional neural network (CNN) architecture proposed in [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] for view synthesis and prediction. The prediction residue is then coded using HEVC [START_REF] Jiang | Light fields compression using depth image based view synthesis[END_REF], or using local residue transforms (SA-DCT) and coding [START_REF] Su | Graph-based light fields representation and coding using geometry information[END_REF]. The proposed transforms could also be used for residue coding. However, to best assess their de-correlation advantage, in the experiments reported below, they are directly applied on the color values of the entire 4D light field data.

III. SUPER-RAYS AND GRAPH CONSTRUCTION

The compression efficiency of any coder based on block partitioning and transform coding does undeniably depend on the way the partitioning is done, and on how the resulting segmentation adheres to object boundaries. While traditional transforms such as 2D DCT applied on a square or rectangular support may fail due to high frequencies captured on the object boundaries, here we rely on a segmentation of the entire 4D light field into super-rays.

A. Light field Segmentation in Super-Rays

Segmentation is an important step of many editing algorithms. While this problem has been widely addressed for 2D images and videos, a few methods exist for light fields [START_REF] Berent | Unsupervised extraction of coherent regions for image based rendering[END_REF], [START_REF] Mihara | 4d light field segmentation with spatial and angular consistencies[END_REF], [START_REF] Wanner | Globally consistent multilabel assignment on the ray space of 4d light fields[END_REF]. The regions or segments extracted by these methods, often corresponding to objects in the scene, are too large for defining local graph transform supports, as targeted here for reducing the complexity of the basis function computation. To overcome this problem, it is natural to consider instead light field over-segmentation. In [START_REF] Zhu | 4D light field superpixel and segmentation[END_REF], a depth estimation is used to propagate an initial over-segmentation into superpixels of a reference view to all the views of the light field. The authors use the SLIC algorithm described in [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF] for the reference view over-segmentation. This initial segmentation is then iteratively refined by optimizing an energy function based on segmentation smoothness inside and between the views along with a color, position and disparity uniformity prior. In this paper, we consider instead the concept of super-ray introduced in [START_REF] Hog | Superrays for efficient light field processing[END_REF] as an extension of super-pixels [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF] to group light rays coming from the same 3D object, i.e. to group pixels having similar color values and being close spatially in the 3D space. The method performs a k-means clustering of all light rays based on color and distance in the 3D space. To deal with dis-occlusions, a slightly modified formulation is proposed in [START_REF] Su | Rate-distortion optimized super-ray merging for light field compression[END_REF] where the dense depth information is also used in the clustering. When the depth information is not fully reliable, this method results in inconsistent super-rays across views. In addition, the signalling cost of such a global light field segmentation is high. In order to make the super-rays more consistent across the views, we suggest a modified version where we compute super-pixels in the top-left view as shown in Figure 1. Then, using the disparity map, we project the segmentation labels to all the other views. Namely, having a segmentation map in the top left view and the corresponding disparity map, we compute the median disparity per superpixel, and use it to project the segmentation mask to the other views. More precisely, the algorithm proceeds row by row. In the first row of views, we perform horizontal projections from the top-left I 1,1 to the N -1 views next to it. For each other row of views, a vertical projection is first carried out from the top view I 1,1 to recover the segmentation on view I m,1 , then N -1 horizontal projections from I m,1 to the N -1 other views are performed, as shown in Figure 2.

An example of segmentation is shown in Figure 2, where we show a cropped area consisting of both background and foreground objects. The red and green super-pixels are computed in the initial segmentation of the top left view with SLIC. The blue are super-pixels obtained after disparity-based projection. At the end of each projection, some shapes are projected in all the views without interfering with others. Those typically represent flat regions inside objects (for example, the superpixel in red in I 1,1 ). While others, mainly consisting of occluded and occluding segments end up superposed in some views. In this case, the occluded pixels are assigned the label of the neighboring super-ray corresponding to the foreground objects (i.e. having the higher disparity). As for appearing pixels, they will be clustered with the background super-rays (i.e. having the lower disparity). An example of super-ray that ends up with different shapes in the views is marked in green in the segmentation of I 1,1 . The difference is not very remarkable since we deal with dense light fields where the disparity is not very high, it is a matter of one or two pixels at most. This method performs well on light fields with small baselines. There are however some limitations that appear in the case of wider baselines. We may end up in this case with more occlusions due to higher disparities. With large disparities, objects (or pixels) that are completely occluded in the topleft view but visible in other views cannot be constructed in super-rays.

B. Graph Construction

In order to jointly capture spatial and angular correlations between pixels in the light field, we first consider a local non separable graph per super-ray. More precisely, if we consider the luminance values in the whole light field and a segmentation map S, the k th super-ray SR k can be represented by a signal f k ∈ R N k defined on an undirected connected graph G = {V, E} which consists of a finite set V of vertices corresponding to the pixels at positions m, n, x, ys.t.S(m, n, x, y) = k. A set E of edges are built as follows.

We first connect each pixel (m, n, x, y) in the set V and its 4-nearest neighbors in the spatial domain (i.e. the top, bottom, left and right neighbors with coordinates (m, n, x -1, y), (m, n, x + 1, y), (m, n, x, y -1), (m, n, x, y + 1)). A pixel can have a maximum of four spatial connections if the four neighbors belong to the set V, and can have less if it is on the border of the super-pixel.

We then find the median disparity value d of the pixels inside the super-ray k in the top-left view. Using this disparity value, we project each pixel in super-ray k with coordinates (m, n, x, y) in the 4 nearest neighboring views (i.e. the top, bottom, left and right neighboring view). We end up with four projected pixels with coordinates (m -1, n, x -d, y), (m + 1, n, x + d, y), (m, n -1, x, y -d), (m, n + 1, x, y + d). If a projected pixel belongs to the set of vertices V, then we connect it to the original pixel (m, n, x, y). In this way, a maximum of four angular connections can be found for each pixel if the pixel is not occluded in the neighboring views. The weights of all connections are set to 1. An illustrative example of a graph built inside a super-ray is shown in Figure 3 for four views.

IV. GRAPH TRANSFORMS

In this section, we focus on the design of suitable transforms for the signals (color or residues) residing on the local graphs defined above.

A. Non Separable Graph Transform

Let us consider the k th super-ray SR k and its corresponding local graph G. We start by defining its adjacency matrix A with entries A mn = 1, if there is an edge e = (m, n) between two vertices m and n, and A mn = 0 otherwise. The adjacency matrix is used to compute the Laplacian matrix L = D -A, where D is a diagonal degree matrix whose i th diagonal element D ii is equal to the sum of the weights of all edges incident to node i. The resulting Laplacian matrix L is symmetric positive semi-definitive and therefore can be diagonalized as:

L = U ΛU ( 1
)
where U is the matrix whose rows are the eigenvectors of the graph Laplacian and Λ is the diagonal matrix whose diagonal elements are the corresponding eigenvalues. The laplacian eigenbases U are analogous to the Fourier bases in the Euclidean domain and allow representing the signals residing on the graph as a linear combination of eigenfunctions akin to Fourier Analysis. This is known as the Graph Fourier transform. For the signal f k defined on the vertices of the local graph, the transformed coefficients vector fk is defined in [START_REF] Shuman | The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains[END_REF] as:

fk = Uf k (2)
The inverse graph Fourier transform is then given by

f k = U fk (3)
Although this would be the ideal decorrelating transform for the signal, the Laplacian of such graph, despite the locality, remains of high dimension (almost 6000 nodes per superray) leading to a high transform computational cost. To limit the computational cost, we then consider separable local transforms.

B. Coherent Separable Graph Transform

The separable graph transform is defined by a first spatial transform followed by a second angular transform as detailed below.

1) First spatial graph transform: If we consider the luminance values in only one sub-aperture image v of the light field and a segmentation map S, the k th super-ray SR k,v can be represented by a signal f k,v ∈ R N k,v defined on an local spatial graph with only connections in the spatial domain (i.e. between the neighboring pixels in a super-pixel, but not across the views in a super-ray). N k,v denotes the number of pixels in sub-aperture v that belong to the super-ray k. The matrix U k,v , being the eigen-vectors of the spatial laplacian L k,v , is used to compute the first spatial graph transform : For the signal f k,v defined on the vertices of the graph, the transformed coefficients vector fk,v is defined in [START_REF] Shuman | The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains[END_REF] as:

fk,v = U k,v f k,v (4)
The inverse spatial graph Fourier transform is then given by

f k,v = U k,v fk,v (5)
2) Second angular graph transform: In order to capture inter-view dependencies and compact the energy into fewer coefficients, we perform a second graph based transform, in the angular dimension. Note that, for a given super-ray, we do not necessarily have the same number of pixels, hence coefficients resulting from the spatial transforms, in all the views. For a given band b (coefficients corresponding to the b th eigenvectors of the spatial transforms), we construct a graph of N b vertices corresponding to the views where the band b exists. Edges are drawn between each node and its direct four neighbors. Isolated nodes are connected to their nearest neighbor.

The Adjacency is used to compute the inter-view angular unweighted Laplacian as

L b k = D b k -A b k with D b k the degree matrix. L b
k can be diagonalized as:

L b k = U b k ΓU b k (6)
For a specific band number b and super-pixel k, the band signal is defined as

f b k = { fk,v (b), v ⊆ [1, • • • , M × N ]} ∈ R N b .
The angular Graph Transform consists of projecting the signal onto the eigenvectors of L b k as:

f b k = U b k f b k (7)
The inverse angular Graph Transform is then given by

f b k = U b k f b k (8)
3) Coherence of spatial graph transforms in corresponding super-pixels: The spatial graphs in the different super-pixels forming one super-ray may not have the same shape. Furthermore, we have observed that for a specific super-ray, when the spatial graph topology in the corresponding super-pixels undergoes a slight change, the basis functions of each spatial graph transform are different and thus incompatible with each others (refer to Figure 4 before optimization), resulting in decreased correlation of the spatial transform coefficients across views. This is shown in the sequel to severely decrease the efficiency of the angular transform.

Basically, during the diagonalization procedure, the eigenfunctions are only defined up to sign flips for Laplacians having a simple spectrum (if the eigenvalues have a multiplicity of 1, for example connected graphs). Therefore, even having the same shape in two different views, we may end up with two opposite eigen-vectors for a specific eigenvalue during the diagonalization.

Moreover, eigenvectors computed independently on two different shapes (i.e. corresponding to two different Laplacians) can be expected to be reasonably consistent only when the shapes are approximately isometric. Whenever this assumption is violated, it is impossible to expect that the k th eigenvector of a Laplacian L si in view i will correspond to the k th eigenvector of another Laplacian L sj in view j. If the basis functions do not behave consistently on the corresponding points of the two shapes, the two signals defined on those two Laplacians will be projected onto incompatible basis functions (see Figure 4), and therefore we cannot guarantee any correlation to be preserved after performing the first spatial graph transform.

4) Coherent spatial graph transform: In order to overcome those limitations, we consider an approach which aims at finding coupled basis functions. We propose a graph optimization which differs from [START_REF] Kovnatsky | Coupled quasi-harmonic bases[END_REF] in several manners:

• First, the association and correspondence points between the supports (i.e. super-pixels in different views) are defined based on the scene geometry and not manually as in [START_REF] Kovnatsky | Coupled quasi-harmonic bases[END_REF]). • Also, we design consistent transforms for more than 2 supports (N = 2 in [START_REF] Kovnatsky | Coupled quasi-harmonic bases[END_REF]) by fixing one reference and optimizing the other basis functions in other views with respect to the reference in order to reduce the complexity of the overall problem. • Third, thanks to fixing one reference, we limit the error propagation between different basis functions of different supports that may appear if we perform an iterative optimization. More precisely, suppose that, in the super-ray k in a reference view o and a target view i, we have two Laplacians L k,o and L k,i with size (n o × n o ) and (n i × n i ) respectively. They can be diagonalized as:

Geometric associations

L k,o = U k,o Λ o U k,o L k,i = U k,i Λ i U k,i (9)
If the two Laplacians are equal, we make sure that their eigenvectors are compatible with sign flips accordingly. We check the first value of the each eigenvector and flip its sign if the value is negative. In the case where the super-pixel shapes in the sub-aperture images are not isometric, we propose to diagonalize one specific spatial graph Laplacian L k,o and find U k,o . Then, we search for basis vectors Ûk,i that approximately diagonalize any other spatial graph Laplacian L k,i and at the same time preserve correlations after the transform. We start by posing the problem as follows:

Û * k,i = min Ûk,i of f ( Û k,i L k,i Ûk,i ) + α (F U k,o -G Ûk,i ) 2 F , s.t. Û k,i Ûk,i = I. ( 10
)
where we seek to minimize the weighted sum of two terms subject to the orthonormality constraint of the computed basis functions Ûk,i . The first term is a diagonalization term that aims at minimizing the energy residing on off-diagonal entries (of f (M) = i =j m ij ). The second term aims at enforcing coherence between the two spatial graph transforms and is defined as follows.

Based on the geometry information we have in hand, we can actually define, a priori, a set of correspondences between L k,o and L k,i . More precisely, we suppose that we have a set of p corresponding functions represented by matrices F and G of sizes (n 0 × p) and (n i × p) respectively. An example of F and G is shown in figure 5.

The basis functions of both Laplacians are supposed to be consistent if the Fourier coefficients of the functions F and G on L k,o and L k,i are approximately equal i.e. if F U k,o G Ûk,i . To avoid over-determining the problem, we use the farthest point sampling technique restricting the correspondence points to a maximum of 15 points.

If we parametrize the new basis functions of L k,i as being a linear combination of the old basis functions, we can write Ûk,i = U k,i B where B is a matrix of combination coefficients, that plays a role of reflecting and rotating the original basis vectors in U k,i so that they will align the best way with U s0 while almost diagonalizing the laplacian L k,i . Using the diagonalizing property of U k,i , we can re-write Equation [START_REF] Verhack | Steered mixture-of-experts for light field coding, depth estimation, and processing[END_REF] as

B * = min B of f (B Λ i B) + α (F U k,o -G U k,i B) 2 F , s.t. B B = I, (11)
It is important to note that the first term of the above problem does not guarantee a preserved increasing order of the eigenfunctions. It is therefore more convenient to use an alternative penalty equal to B Λ i B -Λ i 2 F that relates not only to the diagonalization property, but also to the distribution of the energies across the basis functions after the optimization.

B * = min B B Λ i B -Λ i 2 F + α (F U k,o -G U k,i B) 2 F , s.t. B B = I, (12)
The problem in Equation ( 12) is a non linear optimization problem with an orthogonality constraint, which can be solved by iterative minimization algorithms. In our case, we used Matlab optimization toolbox (interior point method of the fmincon function) to solve it. The gradients of the cost function terms are given in appendix A.

Since we are dealing with large datasets and a large number of super-rays, it is convenient to use parallel computing to independently compute eigen-basis for the different super-rays. Also, in contrast with the way the optimization is performed in [START_REF] Kovnatsky | Coupled quasi-harmonic bases[END_REF], in order to reduce the complexity of the problem, we propose to split it into smaller problems that are independent: we pick a small number z of eigenvectors to be optimized at a time. Then, for each disjoint group l of z eigenvectors in U k,i , we formulate a sub-problem by expressing z new eigenvectors as a linear combination of z old eigenvectors. Noticing that

U k,i = [ U 1 k,i , U 2 k,i , ..., U l k,i ]
and

Λ i =     Λ 1 i 0 0 0 0 Λ 2 i 0 0 0 0 .. 0 0 0 0 Λ l i     (13)
For each group of z eigenvectors, we find B l of size (z × z) that will minimize the objective function on the subset of eigenvectors.

B * l = min B l B l Λ l i B l -Λ l i 2 F + α (F U l k,0 -G U l k,i B l ) 2 F , s.t. B l B l = I, (14)
At the end of the optimization stage, most of the eigenvectors are thereby compatible across views and the transform will necessarily preserve any correlation already observed between views. An example of the second eigenvector of a super-ray before and after optimization is shown in Figure 4. While eigenvectors corresponding to higher frequencies are harder to adjust, the low frequency eigenvectors can be easily optimized. In our application, this is not a big problem since we have a high energy compaction in lower frequency bands, and those are the bands that matter the most for reconstruction. After performing the segmentation and two transforms, most of the energy of the color signal is indeed expected to be concentrated in a very small number of coefficients. In the following section, we aim at exploiting this energy compaction property to efficiently code the redundant information present in the light field using the tools introduced above.

V. LIGHT FIELD CODING SCHEME

The overall steps of the compression algorithm are shown in Figure 6. The top left view of the Light Field is separated into uniform regions using the SLIC algorithm to segment the image into super-pixels [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF], and its disparity map is estimated. Using both the segmentation map and the geometry information, we construct consistent super-rays in all views as explained in section III. The non separable and separable transforms described above are then locally applied on each super-ray. The transformed coefficients are then quantized and encoded to be stored or transmitted. The segmentation map of the reference view and a disparity value per super-ray also need to be transmitted as side information to the decoder.

1) Segmentation map and disparity values coding: The segmentation map of the reference view is encoded using the arithmetic edge coder proposed in [START_REF] Daribo | Arithmetic edge coding for arbitrarily shaped sub-block motion prediction in depth video compression[END_REF]. The contours are first represented by differential chaincode [START_REF] Freeman | On the encoding of arbitrary geometric configurations[END_REF] and divided into segments. Then, to efficiently encode a sequence of symbols in a segment, AEC uses a linear regression model to estimate probabilities, which are subsequently used by the arithmetic coder. Disparity values are encoded using an arithmetic coder.

2) Grouping and transform coefficients coding: The energy compaction is not the same in all super-rays. This can be explained by the fact, that the segmentation may not well adhere to object boundaries, resulting in high angular frequencies after optimization of the first spatial transform. Fig. 7: Consistent Super-rays performance:In the first three columns, we have the original top left corner view, its corresponding disparity map and super pixel segmentation using the SLIC algorithm [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF] respectively. In the fourth column, we show horizontal and vertical epipolar segments taken both from the 4D light field color and our final labeling in specific regions of the image(the red blocks). We use the prism color map in Matlab for the segmentation, just for illustration purposes.

To optimize the coding performance, we divide the set of super-rays into four classes, where each class is defined according to an energy compaction criterion.

First, we learn a scanning order. More precisely, at the end of the two graph transform stages, coefficients are grouped into a three-dimensional array R where R(i SR , i bd , v) is the v th transformed coefficient of the band i bd for the superray i SR . Using the observations on all the super-rays in some training datasets (Flower1,Friends), we can find the best ordering for scanning and quantization. We sort the variances of coefficients with enough observations in decreasing order and we follow this decreasing order during the scanning process.

Then, each super-ray with N coefficients belongs to class i if the mean energy per high frequency coefficient is less than 1, where the high frequency coefficients are the last round(N × i/4) coefficients following the scanning order of the super-rays coefficients defined previously. We start by finding the superrays in the first class than remove them from the search space before finding the other classes, and idem for the following steps.

We code a flag with an arithmetic coder to gives the information of the class of super-rays to the decoder side. In class i, the last round(N × i/4) coefficients of each superray are discarded. The rest of the coefficients are grouped into 32 uniform groups. The quantization step sizes in groups are defined with a rate-distortion optimization taking into account a big number of observed coefficients. At the end of this stage, for each class, each group is coded using the Context Adaptive Binary Arithmetic Coder (CABAC) from the HEVC H.265 reference coder.

VI. EXPERIMENTAL ANALYSIS

For performance evaluation, we consider real light fields captured by plenoptic cameras from the datasets used in [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] and [START_REF] Viola | A graph learning approach for light field image compression[END_REF]. We consider the 8 × 8 central sub-aperture images cropped to 364 × 524 in [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF], and 9 × 9 cropped to 432 × 624 from [START_REF] Viola | A graph learning approach for light field image compression[END_REF] in order to avoid the strong vignetting and distortion problems on the views at the periphery of the light field. The disparity map of the top left view of each light field has been estimated using the method in [START_REF] Jiang | Depth estimation with occlusion handling from a sparse set of light field views[END_REF]. The estimated disparity map is used to construct super-rays as described in Section III.

A. Assessment of the proposed super-ray construction method

In this section, we assess how the proposed super-ray construction method deals with occluded and dis-occluded parts, and to which extent the super-rays are consistent despite uncertainty on the disparity information. Figure 7 shows examples of super-rays obtained with different real light fields captured by a Lytro Ilum camera (Flower 2, Rock used in [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF], and FountainVincent, StonePillarInside used in [START_REF] Viola | A graph learning approach for light field image compression[END_REF]). In the first three columns, we have the original top left corner view, its corresponding disparity map and super pixel segmentation using the SLIC algorithm [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF] respectively. In the fourth column, we show horizontal and vertical epipolar segments taken both from the 4D light field color information and our final segmentation in specific regions of the image (the red blocks). We can see that we are following well the object borders, especially when the disparity map is reliable. Also, we have always attained a high percentage of coherent super-rays across views (higher than 40% as measured with Cons(%) in the fifth column). More precisely, Cons(%) gives the percentage of coherent super-rays: A super-ray is coherent when it is made of super pixels having the same shape in all the views, with or without a displacement.

At the end of this segmentation stage, we end up with a segmentation map with consistent super rays in flat objects and shape-varying super-rays mainly on the borders.

B. Analysis of proposed graph based optimized transforms

In this section, we analyze the performance of our optimization process described in section IV-B and its effect on the transform coding efficiency. In all the experiments, for each super-ray we find the super-pixel L so that is on the topleft most of the light field, and fix it as reference for the coupling process. We therefore optimize the maximum number of eigenvectors defined as f loor( n0 10 ) × 10 with n 0 being the number of pixels in the reference super-pixel. An example of input and output of the coupling process for a shape-varying super-ray is illustrated in Figure 8. We see that the consistency of eigenvectors in the different graphs is much better after our optimization. If we project the light field signal residing in the super-ray on the optimized coupled eigenvectors, the inter-view correlation is better preserved compared to the non optimized eigenvectors.

1) Energy Compaction of the spatial transform: Figure 9 shows the energy compaction observed in the spatial transform domain, then in the spatio-angular transform domain, i.e. after performing the first spatial transform and after performing both spatial and angular transforms on the color signal of the light fields. The energy compaction is computed for both optimized and non optimized cases. It denotes the percentage of energy if we keep some of the coefficients and discard others. For the spatial transform, we gather the transform coefficients of all super-pixels, and then we scan them following the intuitive order increasing order of the Laplacian eigenvalues to compute the compaction. For the spatio-angular compaction, we follow the learned sub-optimal scanning order using different observations from the different datasets as explained in section V-2.

If we compare the energy compaction of the spatial transforms only (red and blue curves) for different datasets, we observe that we may loose in terms of energy compaction for some datasets after optimization. In order to explain such loss, we analyze how the graphs are varying under the new basis functions after optimization. An example is shown in Figure 10 where edges between highlighted nodes are added implicitly in the graph after coupling. The new underlying Laplacian is computed as Lk,i = Ûk,i Λ k,i ÛT k,i . The underlying assumption behind the optimization procedure is that the signal can be modeled by a modified Gaussian distribution (Gaussian Markov Random Field) with a modified precision matrix which is equivalent to the new Laplacian matrix with some added small weights. Since this procedure is modifying the original graph structure, it may, in some cases, bring some high frequencies. 2) Correlation and Energy Compaction after angular transform: The gain in compaction after the spatio-angular transform is clear in Figure 9 when we perform the optimization. This is due to the fact that we are able to preserve angular correlations after the spatial transform, which will be subsequently exploited by the angular transform.

In order to assess the performance of our coupling process in preserving the correlation, we draw in Figure 11, the correlation matrices and the covariance matrices for some bands after the first transform with shape-varying super-rays. If we restrict our attention to the first column, We see that after the first transform that is not optimized, we have uncorrelated transform coefficients due to the perturbation of eigenvectors computed on super-pixels having slightly different shapes. This problem is almost resolved with our coupling procedure in the second column, where we can observe more correlation between the coefficients of the same band in neighboring from the dataset used in [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] and two others (Friends and StonePillarsInside) taken from the datasets in [START_REF] Viola | A graph learning approach for light field image compression[END_REF].

Original graphs before coupling New graphs after coupling views. Furthermore, the logarithm of the variances (values lying on the diagonal in the covariance matrices) being higher in the first low frequency bands and decreasing when moving further from the DC, shows the energy compaction of the first transform. As for the values of the off-diagonal elements of the covariance matrices, they show how correlated are the transformed coefficients after the first transform inside the views. If we observe the off-diagonal values and compare them with or without optimization, we find out that the optimization performs better for low frequencies than for high frequencies and is therefore more able to retrieve coherent basis functions.

After the second angular transform per band, for both cases with or without optimization, we compute the logarithm of coefficients' variances after the second transform and illustrate it in the third row where the x-axis and y-axis correspond to the band number and the view number respectively. A compaction of the energy in fewer coefficients is observed in the optimized case compared to the non-optimized case, especially when we focus on the top-left region. Some inter-view high frequencies are sometimes still there and might be due to the presence of some super-rays are made of super-pixels that adhere well to borders in some views while not adhering in some others due to disparity rounding effects.

3) Impact of disparity errors: When the disparity information is not reliable, dis-occluded pixels may be clustered with a wrong super-ray, resulting in high frequencies, hence poor energy compaction, after the spatial transforms in those specific regions. However, experiments with synthetic data sets (for which the ground truth depth is known) have shown that, with the considered disparity estimation method [START_REF] Jiang | Depth estimation with occlusion handling from a sparse set of light field views[END_REF], the depth map errors lead to a mislabelling for less than 2% of the pixels, which has a negligible effect on the energy compaction and on the RD performance.

4) Impact of super-rays size: The size of super-rays may have an impact on the rate distortion performance especially when the disparity information is reliable and there is a lot of homogeneous objects. If we have large objects, we might want to merge some small super-rays which makes a non separable graph transform practically unfeasible. Here comes the advantage of an optimized separable graph transform where one can define the number of eigenvectors to be optimized depending on the homogeneity of the shape-varying super-rays inside the views. In this case, the segmentation and disparity costs will more likely drop also since we also have less contours and values to code.

In our experiments, however, we use a uniform segmentation into super-pixels. We fix the number of super-rays to 2800 for the light fields in [START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF], and 4000 for the light fields in [START_REF] Viola | A graph learning approach for light field image compression[END_REF]. We have observed that when we have a small number of super-rays, the disparity errors may have an impact on the compensation and therefore result in a decreased PSNR-Rate performance. On the other hand, having a very large number of super-rays increases the rate needed for segmentation and limits the dimension of each super-ray, resulting in a smaller benefit in terms of de-correlation of the proposed spatioangular transform.

C. Rate-distortion performance comparative evaluation

We assess the compression performance obtained with our graph based transform coding schemes against four schemes: encoding the views as a video sequence following a lozenge order (HEVC lozenge) [START_REF] Rizkallah | Impact of light field compression on focus stack and extended focus images[END_REF], or using different prediction orders in the same vein as multi-view coding (HEVC pseudo), [START_REF] Liu | Pseudosequence-based light field image compression[END_REF], JPEG Pleno VM 1.1 software [START_REF] Jpeg | Jpeg pleno light field coding vm 1.1[END_REF] and HLRA [START_REF] Jiang | Light field compression with homography-based low-rank approximation[END_REF] In the simulations, the basic configuration files of JPEG Pleno VM have been used with small changes in order to be applied on 9 × 9 views. For HEVC-lozenge, the base QPs are set to 20, 26, 32, 38 and a GOP of 4 is used. The HEVC version used in the tests is HM-16.10.

In Figure 12, our coding scheme based on both non separable and separable graph transforms is investigated against HEVC-lozenge [1], JPEG pleno 1.1 [START_REF] Jpeg | Jpeg pleno light field coding vm 1.1[END_REF], HEVC-pseudo ( [START_REF] Liu | Pseudosequence-based light field image compression[END_REF]) and HLRA ( [START_REF] Jiang | Light field compression with homography-based low-rank approximation[END_REF]) for three light fields with 9×9 views, from the ICIP 2017 Grand Challenge [START_REF] Viola | A graph learning approach for light field image compression[END_REF]. Further experiments are also depicted in Figure 13 for 8 × 8 light fields 1 . For the separable case, we compare the optimized and the non optimized graph transform. In Table I, we restrict our attention to the optimized separable graph based transform case that we denote by optseparable GBT scheme that can be applied no matter how big the super-rays are. It shows the rate allocation of our method, at low and high bitrates, for the different light fields.

We can observe that, for most of the light fields used in our tests, the non separable graph transform yields a better ratedistortion performance compared to the separable case for a fixed number of super-rays. While the non optimized graph transform fails to compact the energy of the light field, the optimized graph transform is performing better and sometimes almost catches the non separable case. One major advantage of the separable optimized case is that it can be applied on super-rays of large dimensions without facing the basis functions computational complexity issue of the non separable case. More precisely, the use of the separable transform (including its optimization) leads to a time saving of around 60% compared with the non separable transform. Furthermore, the number of eigenvectors to be optimized can be defined by the encoder and does not have to be necessarily large. Fig. 12: Rate distortion performance of our graph based coding schemes (Non separable, not optimized and optimized separable graph transforms) compared to HEVC lozenge [START_REF] Rizkallah | Impact of light field compression on focus stack and extended focus images[END_REF], HEVC pseudo [START_REF] Liu | Pseudosequence-based light field image compression[END_REF], HLRA [START_REF] Jiang | Light field compression with homography-based low-rank approximation[END_REF] and JPEG Pleno VM 1.1 [START_REF] Jpeg | Jpeg pleno light field coding vm 1.1[END_REF] for the 9 × 9 light fields used in the ICIP 2017 Grand Challenge [START_REF] Viola | A graph learning approach for light field image compression[END_REF] following the common test conditions. Moreover, we can observe that the proposed method outperforms JPEG Pleno VM 1.1 [START_REF] Jpeg | Jpeg pleno light field coding vm 1.1[END_REF] and HEVC lozenge [START_REF] Rizkallah | Impact of light field compression on focus stack and extended focus images[END_REF] at high bitrate. Figure 12 also shows that it yields lower RD performances when compared with two other reference methods (HEVC-pseudo [START_REF] Liu | Pseudosequence-based light field image compression[END_REF] and HLRA [START_REF] Jiang | Light field compression with homography-based low-rank approximation[END_REF]). This is mainly due to the fact that the proposed scheme does not incorporate any spatial mechanism to exploit correlation between the local transform supports while the two reference methods benefit from the efficient HEVC intra prediction mechanisms.

Also, the bitrate allocated to the segmentation and disparity is very large, especially at low bitrate (almost reaching 30 percent for most datasets) and could be further reduced.

Note that the decoder needs to compute the optimized basis functions for the non consistent super-rays, inducing some computational complexity. However, the optimization can be performed independently on each super-ray, in a parallel manner.

VII. CONCLUSION

In this paper, we have addressed the problem of local geometry-aware graph transform design for light field energy compaction and compact representation. The transform support is based on super-rays constructed in a way that their shape remains coherent across the different views. We have first considered both non separable graph transforms.

Despite the limited size of the transform support, the Laplacian matrix of such graph remains of high dimension and its diagonalization to compute the transform eigenvectors is computationally expensive.

To solve this problem, we then considered a separable spatio-angular transform. We have shown that, when the shape of corresponding super-pixels in the different views undergoes small changes, the basis functions of the spatial transforms are not coherent, resulting in a decreased correlation between spatial transform coefficients. We hence proposed a novel transform optimization method that aims at preserving angular correlation even when the shapes of corresponding superpixels (i.e. forming one super-ray) are not isometric. This procedure has been shown to increase energy compaction of the separable spatio-angular graph transforms and bring substantial rate-distortion performance gains compared to a non optimized case. The proposed optimized spatio-angular graph transforms can be applied on both color or residual signals and can be easily parallelized to reduce the complexity on the decoder side.

ACKNOWLEDGMENT

The authors would like to thank Elian Dib, Navid Mahmoudian Bidgoli and Pierre Allain from Inria for their help in extracting and running the JPEG Pleno VM 1.1 [START_REF] Jpeg | Jpeg pleno light field coding vm 1.1[END_REF] and HEVC CABAC encoder. Also, we would like to thank Effrosyni Simou and Eda Bayram from EPFL, who were of a great help with various discussions about the subject.

APPENDIX A GRADIENTS OF THE OBJECTIVE FUNCTION TERMS

The gradients of the two terms in the optimization of equation 12 are provided below:

∇ B B Λ i B -Λ i 2 F = ∇ B tr (B Λ i B -Λ i ) (B Λ i B -Λ i ) = ∇ B tr (B Λ i B -Λ i )(B Λ i B -Λ i ) = ∇ B tr(B Λ i BB Λ i B -B Λ i BΛ i -Λ i B Λ i B + Λ i Λ i ) = 4(Λ i BB Λ i B -Λ i BΛ i ) (15)
As for the coupling term, with a similar derivation as the first gradient and using the trace derivation properties in [START_REF] Petersen | The matrix cookbook[END_REF], we get:

∇ B ( (F U s0 -G U si B) 2 F ) = 2U si G(G U si B -FU s0 ) (16)

Fig. 1 :

 1 Fig.1: The result of our proposed light field segmentation for a dense light field Fountain Vincent (from EPFL light fields dataset) with an estimated disparity. From left to right, the original view I 1,1 , the disparity map of the original view, the super-pixel segmentation of I 1,1 and an example of vertical and horizontal epipolar segments taken from both original 4D RGB light field and the 4D segmentation labels (The red lines inside the images show from where the epipolar line is extracted).

Fig. 2 :

 2 Fig. 2: Image showing the super-ray construction. The algorithm proceeds row by row. In the first row, only horizontal projections are performed. In every other row, first a vertical projection then N -1 horizontal projections are performed. The red super-pixel in I 1,1 is consistent across views, whereas the super-pixel in blue is shape-varying.

  Angul r grap c nne tio s Spatial graph connections

Fig. 3 :

 3 Fig. 3: Example of edges drawn inside a super-ray. We can see the connections within super-pixels in each view (i.e. spatial connections in red), as well as connections between pixels belonging to different views (i.e. angular connections in blue, only a subset of the connections is shown for illustration purpose). The color assigned to the vertices are the color values of the light field.

Fig. 4 :

 4 Fig. 4: Second eigenvector of different super-pixels belonging to the same super-ray before and after optimization. Only two sets of geometric associations are shown for illustration purposes.

Fig. 5 :

 5 Fig.5: Example of correspondence functions F and G computed for a small shape-varying super-pixel. The graph nodes are labeled in both graphs following a vertical scan line. In the second view, we have one disappearing node and another appearing one with respect to the first view.

Fig. 6 :

 6 Fig. 6: Overview of proposed coding scheme.

Fig. 8 :

 8 Fig. 8: Illustration of the output of the optimization process for a super-ray in 4 views. The first row corresponds to a super-ray accross four views of the light field. The signal on the vertices correspond to the color values lying on superpixels corresponding to the same super-ray and the blue lines denote the correspondences. The second to fourth rows are illustrations of basis functions before and after optimization. The signals on the vertices are the eigenvectors values.

Fig. 9 :

 9 Fig.9: Energy compaction with or without optimization of the first spatial transform for four datasets (Seahorse, Rock,Flower2 and Cars) from the dataset used in[START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF] and two others (Friends and StonePillarsInside) taken from the datasets in[START_REF] Viola | A graph learning approach for light field image compression[END_REF].

Fig. 10 :

 10 Fig. 10: Image showing the old graphs before coupling and the new graphs after optimization. New edges with absolute weight values larger than 0.04 are shown as blue lines connecting highlighted nodes.

Fig. 11 :

 11 Fig. 11: Advantage of our optimization in terms of energy compaction. The three rows correspond to (1) correlation matrices of the spatial transformed coefficients of the first ten bands, (2) the log of the absolute value of the covariance matrices of the 64 first bands of the spatial transformed coefficients, and (3) the logarithm of the variance of the coefficients after the angular transform, respectively. The two columns show the two cases: without or with our optimization.

  Rate distortion performance of our graph based coding schemes (Non separable, not optimized and optimized separable graph transforms) compared to HEVC lozenge[START_REF] Rizkallah | Impact of light field compression on focus stack and extended focus images[END_REF] for the 8 × 8 light fields of[START_REF] Kalantari | Learning-based view synthesis for light field cameras[END_REF].

		45				Flower 2				45				Cars					45			Rock				50				Seahorse
		40								40									40							45
																										40
	PSNR (dB)	30 35							PSNR (dB)	30 35								PSNR (dB)	30 35						PSNR (dB)	35
																										30
		25					HEVC lozenge Non Separable Separable not optimized Separable optimized		25						HEVC lozenge Non Separable Separable not optimized Separable optimized		25				HEVC lozenge Non Separable Separable not optimized Separable optimized		25					Non Separable Separable optimized Separable not optimized HEVC lozenge
		20	0	0.1	0.2	0.3	0.4	0.5	0.6	20	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	20	0	0.1	0.2	0.3	0.4	0.5	20	0	0.05	0.1	0.15	0.2	0.25	0.3	0.35	0.4
						Bitrate (bpp)								Bitrate (bpp)							Bitrate (bpp)							Bitrate (bpp)
				Fig. 13: Light Field								Rate allocation(in %) for the opt-separable GBT scheme Overall bitrate Segmentation Disparity Coefficients
						Cars (364 × 524)					0.2563 bpp (PSNR = 42.24dB) 0.0212 bpp (PSNR = 25.23dB)	2.69% 32.55%	0.55% 6.60%			96.76% 60.85%
						Flower2 (364 × 524)			0.2710 bpp (PSNR = 40.77dB) 0.0362 bpp (PSNR = 29.18dB)	2.69% 20.17%	0.55% 4.14%			96.76% 75.69%
						Rock (364 × 524)					0.1951 bpp (PSNR = 41.68dB) 0.0306 bpp (PSNR = 31.10dB)	4.00% 25.49%	0.82% 5.23%			95.18% 69.28%
						Seahorse (364 × 524)			0.2302 bpp (PSNR = 42.99dB) 0.0612 bpp (PSNR = 33.88dB)	2.65% 9.97%	0.74% 2.78%			96.61% 87.25%
						Friends (432 × 624)			0.1464 bpp (PSNR = 41.73dB) 0.0294 bpp (PSNR = 33.38dB)	3.89% 19.39%	0.10% 5.10%			96.01% 75.51%
						StonePillarInside (432 × 624)	0.2204 bpp (PSNR = 39.07dB) 0.0212 bpp (PSNR = 32.85dB)	2.59% 26.89%	0.54% 5.66%			96.87% 67.45%
						FountainVincent (432 × 624)	0.2448 bpp (PSNR = 40.37dB) 0.0330 bpp (PSNR = 30.38dB)	2.12% 15.76%	0.57% 4.24%			97.31% 80.00%

TABLE I :

 I Rate allocation performed by the proposed coding scheme with the optimized separable graph transform. The rate is divided into three parts used for coding the segmentation, disparity and transform coefficients.

Visual results can be found on http://clim.inria.fr/research/GBT/GBT.html

This work has been supported in part by the EU H2020 Research and Innovation Programme under grant agreement No 694122 (ERC advanced grant CLIM).