Christian Galea

Christine Guillemot

DENOISING OF 3D POINT CLOUDS CONSTRUCTED FROM LIGHT FIELDS

Keywords: Light fields, point clouds, depth estimation, denoising

Light fields are 4D signals capturing rich information from a scene. The availability of multiple views enables scene depth estimation, that can be used to generate 3D point clouds. The constructed 3D point clouds, however, generally contain distortions and artefacts primarily caused by inaccuracies in the depth maps. This paper describes a method for noise removal in 3D point clouds constructed from light fields. While existing methods discard outliers, the proposed approach instead attempts to correct the positions of points, and thus reduce noise without removing any points, by exploiting the consistency among views in a light-field. The proposed 3D point cloud construction and denoising method exploits uncertainty measures on depth values. We also investigate the possible use of the corrected point cloud to improve the quality of the depth maps estimated from the light field.

INTRODUCTION

Light field imaging has recently captivated the interest of both research and industry due to numerous applications such as view synthesis, augmented reality, and post-production. A light field can be seen as capturing an array of viewpoints (called sub-aperture images in particular in the case of microlens based plenoptic cameras) of the imaged scene with varying angular coordinates, leading to a 4D representation of the imaged scene, spanning spatial and angular dimensions [START_REF] Levoy | Light field rendering[END_REF][START_REF] Steven J Gortler | The lumigraph[END_REF]. The availability of multiple views enables the computation of scene depth that can then be used to construct a 3D model, e.g. in the form of a 3D point cloud, thanks to the light field camera parameters. However, 3D point cloud construction methods are prone to outliers caused by inaccuracies in the depth maps arising from deficiencies in the depth estimation method itself, from matching ambiguities, or from factors such as lens distortion and sensor noise [START_REF] Wolff | Point cloud noise and outlier removal for image-based 3d reconstruction[END_REF].

Methods proposed in the literature attempt to solve the problem by either refining the depth maps, or the point clouds themselves. Notable methods include the anisotropic denoising method in [START_REF] Sun | Denoising point sets via L0 minimization[END_REF] which uses L 0 minimisation to recover sparse sharp features, assuming that the surface of common objects is piece-wise smooth almost everywhere except for a small number of sharp features. Outliers are removed in [START_REF] Radu | Towards 3d point cloud based object maps for household environments[END_REF] by employing a statistical analysis of the points' geometrical properties, while the approach in [START_REF] Wolff | Point cloud noise and outlier removal for image-based 3d reconstruction[END_REF] employs geometric and photometric consistency measures. The method in [START_REF] Arvanitis | Real-time removing of outliers and noise in 3d point clouds applied in robotic applications[END_REF] removes outliers using Robust Principal Component Analysis (RPCA) and denoises the resulting point cloud with a bilateral filter [START_REF] Zheng | Bilateral normal filtering for mesh denoising[END_REF][START_REF] Fleishman | Bilateral mesh denoising[END_REF]. However, these methods are usually designed in the context of improving a particular task in a 3D reconstruction pipeline, such as surface reconstruction, or make explicit assumptions on the input scene.

This paper considers the problem of denoising 3D point clouds derived from depth maps estimated from light fields. The method does not remove any points or perform explicit smoothing as done in existing methods. Instead, the proposed method attempts to correct the point positions by leveraging the consistency among views and by considering both geometric and photometric information. To the best of the authors' knowledge, the proposed approach: (i) is the first method that considers point clouds of whole scenes (rather than single objects only) generated from light field cameras, and (ii) attempts to improve the position of outlier points, rather than simply detect and remove them as done by existing methods. It is also shown that the proposed 3D point cloud denoising method can be used in turn to improve the quality of depth maps by re-projecting the ameliorated points into the set of 2D planes.

The rest of this paper presents a description of the proposed methods in Sections 2 and 3, followed by their evaluation in Section 4. Concluding remarks and proposals for future work are finally given in Section 5.

3D POINT CLOUD CONSTRUCTION USING DEPTH UNCERTAINTY

The algorithm starts by first estimating disparity between views using the ProbFlow optical flow estimation algorithm [START_REF] Wannenwetsch | Probflow: Joint optical flow and uncertainty estimation[END_REF], which also generates uncertainty measures on the estimated disparity values. The uncertainty estimation is exploited to combine the disparity maps obtained for each view such that a value is extracted from the disparity map having the highest estimated reliability. Specifically, the disparity map for a given view can be es-

Z c = b • f • f d • max(H, W ) s c • f d • ss + b • f • max(H, W ) (1)
where W and H are the width and height of the captured image, respectively, b is the baseline (in millimetres, mm), f is the focal length (mm), f d is the focus distance (mm), and ss is the sensor size [START_REF] Honauer | A dataset and evaluation methodology for depth estimation on 4d light fields[END_REF]. The X c -and Y c -coordinates may then be obtained as follows:

X c = ( x W -1 -1 2 ) • ss • Z c f , Y c = ( y H-1 -1 2 ) • ss • Z c f (2)

3D POINT-CLOUD DENOISING

The fused disparity maps still contain several discrepancies with respect to ground truth maps as a consequence of inaccuracies in both the original disparity maps and the uncertainty estimation. For instance, edges around objects tend to be blurred, corresponding to a smooth transition from foreground to background. This manifests in the point cloud as sparse points which do not belong to any object in the 3D space. To denoise the point clouds, the proposed method exploits the redundancy due to the fact that several areas of a scene are captured by multiple cameras. These regions which are assigned similar disparity values in the majority of views yield areas of high point densities in the 3D space that can be termed as inliers. On the other hand, any incorrect values are likely to be contained within a few disparity maps only, and are represented by points which lie in sparsely-populated areas that appear to float in free space, called outliers (see the orange points in Figure 1). While outliers could be removed, as done by most works in literature, the re-projection of the point cloud into 2D image planes would yield missing pixels. Instead, the proposed method attempts to improve the position of points deemed to be outliers, by comparing their geometric and photometric properties with those of inlier points.

Outlier detection

We first need to classify each point as being either an outlier or an inlier. This is done by exploiting information on local densities of points, utilising the Statistical Outlier Removal (SOR) filter proposed in [START_REF] Radu | Towards 3d point cloud based object maps for household environments[END_REF] to exploit the differences in point densities exhibited by inliers and outliers. While this method was designed to remove outliers using statistical properties of the distances among points, in the proposed approach it is simply used to detect the outliers which need to be corrected. Formally, a point i is classed as an outlier if the distance d i to its K nearest neighbours satisfies the following condition:

d i ≥ µ( d K ) + m • σ( d K ) (3)
where d K is a vector containing the average distance of each point to its K nearest neighbours, µ( x) and σ( x) are functions computing the average and standard deviation, respectively, of the vector x, and m is the standard deviation multiplier.

Volume segmentation

Each 3D point must lie on specific rays emanating from the camera viewpoints (i.e. from the pixels in the image planes of the cameras). The position on this line is given by the depth value of the 3D point. Hence, the true position of the outlier is constrained to lie on this line, thereby significantly reducing the number of possibilities in the 3D space. The parametric equation L(t) of a line in 3D may be given by:

L(t) = R + V t (4)
where R = p 1 is a point on the line, V = p 2 -p 1 is a vector parallel to the line containing the points p 1 and p 2 which correspond to the end-points, and t is a scalar parameter representing the position on the line [START_REF]Lines and planes in R 3[END_REF]. Another observation is that a point is likely to have a similar colour to neighbouring points. Lastly, points should also be relatively close in terms of their geometric distance to other visually similar points. Given the above observations, the proposed method first determines the ray on which an outlier point must lie, and then determines nearby inlier points by constructing a frustum around this line as shown in Figure 2. The projection in the 3D space of the four neighbouring pixels (located at the corners of the n × n window centred on the considered pixel) in the disparity map yield four rays corresponding to the boundary edges of the frustum.

The histogram of the number of points along the depth axis is then computed, where a peak represents a cluster of points. Due to the similarity in terms of colour among neighbouring points, the RGB values of an outlier point are compared to those of the inlier points (within each group) using Euclidean distance, retaining only those points having a distance in this colour space that is less than an upper limit l c .

Point position improvement

Given the clusters of inliers, three approaches are then used to correct an outlier's position. The first approach determines if the ray intersects an object that can be well represented with a plane, such as a wall or floor. This is done by attempting to fit planes to the points in a scene using the Random Sample Consensus (RANSAC) method as implemented in the Point Cloud Library [START_REF]Plane model segmentation[END_REF]. Given a plane P defined as follows:

ax + by + cz = q (5)
where q is a scalar, and a, b, c are the coefficients of the normal to the plane, the intersection of the plane P with the ray L(t) is then found by first computing t:

t = q -(ax 1 + by 1 + cz 1 ) ax 2 + by 2 + cz 2 (6)
Substitution of t in Equation ( 4) then yields the coordinates of the point where the ray L(t) and the plane P intersect, corresponding to the position of an object on the surface. Otherwise, if the distance D i between the intersection of the plane and the inliers is greater than a threshold l D1 , the point is determined to be too far away from similar inliers and the outlier point is kept unchanged. Given points p 1 and p 2 defined in Section 3.2 having coordinates (x 1 , y 1 , z 1 ) and (x 2 , y 2 , z 2 ), respectively, the squared (perpendicular) distance between an inlier p 0 = (x 0 , y 0 , z 0 ) and the ray L(t) is given by [START_REF] Weisstein | Point-Plane Distance[END_REF]:

D 2 i = [(x 1 -x 0 ) + (x 2 -y 1 )t] 2 + [(y 1 -y 0 ) + (y 2 -y 1 )t] 2 + [(z 1 -z 0 ) + (z 2 -z 1 )t] 2 (
7) While the previous approach attempted to determine whether an outlier belongs to a plane fitted to a large set of points, the second approach uses a similar methodology but uses a plane that is locally fitted to the inliers within each group using Singular Value Decomposition (SVD) [START_REF] Söderkvist | Using svd for some fitting problems[END_REF]. The same threshold l D1 is used to determine whether the new point coordinates are too far away from the inliers.

Finally, if the above two approaches fail, the inliers are not well-represented with a plane and may thus form part of a curved surface. RPCA is employed to find the new point coordinates in this scenario, which decomposes a matrix into low-rank and sparse matrices as follows:

M = L + S ( 8
)
where M is the matrix representing the point cloud, L is the low-rank matrix that can be considered to correspond to the inliers, and S is a sparse matrix corresponding to the outliers [START_REF] Arvanitis | Real-time removing of outliers and noise in 3d point clouds applied in robotic applications[END_REF]. A threshold l D2 is used to determine whether the new point coordinates are close enough to the inliers. If plausible coordinates are found by any above method, the coordinates which are the closest to the corresponding cluster is selected and the outlier is set as an inlier. On the other hand, if no method produces satisfactory coordinates, the point is checked again in subsequent iterations (where an iteration spans all the non-corrected outliers). Thus, each iteration increases the number of inliers that can be used to correct the remaining outliers.

EVALUATION

Five scenes from the INRIA sparse synthetic light field database1 are considered, each containing a 9 × 9 grid of synthetic images at a resolution of 512 × 512 pixels. The ground truth and reconstructed 3D point clouds are compared using the accuracy and completeness measures as described in [START_REF] Schöps | A multi-view stereo benchmark with high-resolution images and multi-camera videos[END_REF][START_REF] Steven | A comparison and evaluation of multi-view stereo reconstruction algorithms[END_REF]. the results are shown in Table 1, where method A1 represents the point clouds derived from the projection of the disparity maps computed by ProbFlow [START_REF] Wannenwetsch | Probflow: Joint optical flow and uncertainty estimation[END_REF], method A2 represents the point clouds derived from the projection of the disparity maps fused using uncertainty estimation, and method A3 represents the point cloud derived after the proposed denoising method has been applied.

Each type of disparity map is compared to the corresponding ground-truth disparity map, using the commonly used Mean Square Error (MSE), Q25, and BadPix(e) measures, where the latter computes the percentage of pixels having an error larger than e. The results are shown in Table 2, where methods A1 and A2 represent the disparity maps obtained from ProbFlow and the disparity maps fused using uncertainty estimation, respectively, while method A3 denotes the re-projected disparity maps from the 3D point clouds corrected with the proposed method.

All measures are averaged over all views in a scene. Values of the parameters are set to n = 17, l c = 20, l D1 = 40, l D2 = 20, pr = pc = 15, K = 50, and m = 1.

As shown in Tables 1 and2, the fusion of disparity maps using uncertainty estimation does not always lead to improvement in performance. This is mostly due to the measure of uncertainty which is not always sufficiently precise. However, in most cases, it is evident that uncertainty estimation can indeed improve the quality of both 3D point clouds and disparity maps, and is thus an effective manner of combining multiple candidate disparity maps.

Comparing the denoised point clouds and corresponding disparity maps with the original versions (i.e. methods A3 and A2, respectively), it is also evident that the proposed denoising method is effective in improving the quality of the point clouds and disparity maps. Indeed, the BadPix measures (which use low error tolerances) indicate that improvements can be made even at the sub-pixel level, despite the light fields containing a substantial number of relatively large disparities (in the range [-26, 12] pixels).

It is interesting to note that the proposed denoising method is quite robust given that the objects within the scenes considered are typically quite similar in terms of colour, thus making it difficult to determine the object to which an outlier should be assigned. This is also true for the 'Chair' light field, which yields the highest MSEs for all methods. One reason is that this scene exhibits the largest disparities compared to the other scenes. Moreover, since even the disparities computed by ProbFlow contain substantial errors, the inliers are not highly robust and some outliers are undetected by the SOR filter, which impede the effectiveness of the proposed approach. Nevertheless, improvements in terms of other evaluation metrics are still evident.

CONCLUSIONS AND FUTURE WORK

A method for noise removal in 3D point clouds acquired from light fields has been described. The proposed approach first exploits uncertainty measures to have better estimates of depth values and then corrects the positions of outlier points by exploiting geometric and photometric properties together with known camera parameters. This is in contrast to existing methods which simply remove outliers. The proposed pipeline was shown to achieve promising performance in both 2D and 3D, thus leading the way for future avenues of research. Future work includes the use of a more advanced method to detect outliers, and the consideration of non-Lambertian surfaces.

  This work was supported by the EU H2020 Research and Innovation Programme under grant agreement No 694122 (ERC advanced grant CLIM).

Fig. 1 :

 1 Fig.1: Framework: for each image, disparity maps are extracted from four views in the horizontal and vertical directions along with uncertainty measures. The disparity maps are fused into a single map using uncertainty. The fused maps from all views are then projected into 3D to yield the 3D point cloud, which is denoised. Detected outliers are represented in orange, which are shown in green after the proposed method is applied.

Fig. 2 :

 2 Fig. 2: Outliers reside on the red ray, and are located in a frustum bounded by the yellow rays. Two clusters of inliers are found within this frustum, depicted in blue.

Table 1 :

 1 Quality evaluation of estimated point clouds with results averaged over all views. The best results are in boldface.

	Light fields	A1	Accuracy (1.0) A2	A3	A1	Accuracy (5.0) A2	A3	Completeness (1.0) A1 A2 A3	Completeness (5.0) A1 A2 A3
	Aretheuse	0.107	0.111	0.153	0.423	0.439	0.566	0.056	0.059	0.059	0.270	0.285	0.285
	Chair	0.120	0.125	0.137	0.418	0.428	0.465	0.066	0.068	0.068	0.274	0.277	0.278
	Lion	0.065	0.059	0.079	0.283	0.261	0.337	0.030	0.036	0.036	0.146	0.180	0.178
	Philosophers	0.320	0.262	0.335	0.759	0.729	0.862	0.192	0.155	0.158	0.665	0.649	0.669
	Toys cloud	0.202	0.210	0.253	0.613	0.618	0.725	0.133	0.140	0.141	0.549	0.569	0.585
	Average	0.163	0.153	0.191	0.499	0.495	0.591	0.095	0.092	0.093	0.381	0.392	0.399

Table 2 :

 2 Quality evaluation of estimated disparity maps with results averaged over all views. The best results are in boldface.

	Light fields	A1	MSE A2	A3	A1	BadPix (0.01) A2	A3	A1	BadPix (0.03) A2	A3	A1	BadPix (0.07) A2	A3	A1	Q25 A2	A3
	Aretheuse	0.57	0.63	0.59	69.39	68.16	67.59	33.00	32.12	30.88	14.53	13.70	12.55	0.80	0.76	0.75
	Chair	5.69	5.58	7.68	66.06	63.93	63.38	32.26	29.61	29.02	14.99	13.33	13.26	0.70	0.66	0.65
	Lion	0.51	0.52	0.40	76.12	71.68	71.24	42.67	33.05	32.03	19.96	13.47	12.63	1.06	0.88	0.87
	Philosophers	1.63	0.70	0.45	71.63	78.52	77.49	42.57	47.74	45.56	23.97	23.18	20.54	0.87	1.18	1.12
	Toys cloud	0.68	0.75	0.60	78.90	78.12	77.70	48.48	46.80	45.60	24.84	23.00	20.96	1.21	1.16	1.13
	Average	1.82	1.64	1.94	72.42	72.08	71.48	39.80	37.86	36.62	19.66	17.34	15.99	0.93	0.93	0.90

Database will soon be made available at http://clim.inria.fr/ DataSoftware.html