Tao Ku

Remco C Veltkamp

Bas Boom

David Duque-Arias

Santiago Velasco-Forero

Jean-Emmanuel Deschaud

Francois Goulette

Beatriz Marcotegui

Sebastián Ortega

Agustín Trujillo

José Pablo Suárez

José Miguel Santana

Cristian Ramírez

Kiran Akadas

Shankar Gangisetty

SHREC 2020 Track: 3D Point Cloud Semantic Segmentation for Street Scenes

Keywords: SHREC 2020, 3D Point Cloud, Semantic Segmentation, Benchmark * Track organizers

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

With the application of LiDAR sensors in many areas, such as autonomous driving and augmented reality, efficient understanding of large-scale 3D point clouds is a challenging task. Many LiDAR datasets [START_REF] Hackel | Semantic3d.net: A new large-scale point cloud classification benchmark[END_REF][START_REF] Geiger | Are we ready for autonomous driving? the KITTI vision benchmark suite[END_REF][START_REF] Roynard | Parislille-3d: A point cloud dataset for urban scene segmentation and classification[END_REF][START_REF] Zolanvari | Dublincity: Annotated lidar point cloud and its applications[END_REF] have been released as benchmarks for various 3D applications. Thee benchmarks usually provide cleaned 3D point clouds with well-annoated labels ignoring the problems of noises, sampling pattern difference and weak reflective strength. However, it is still difficult to develop an effective and efficient solution. There are two main reasons:

One is that the raw data from LiDAR scanners consist of a tremendous amount points returned from all possible reflective objects, which brings in heavy computation cost. Another reason is that the points are usually nonuniformly distributed, unstructured and unordered, which means that it is not easy to apply mature 2D deep convolutional networks.

Since the input point clouds are orderless and usually sparse, feature learning methods for point clouds can be divided into three categories in terms of the format of input point clouds: projection based, voxel based and point-set based. Projection-based methods [START_REF] Boulch | Unstructured point cloud semantic labeling using deep segmentation networks[END_REF][START_REF] Chen | Multiview 3D object detection network for autonomous driving[END_REF] try to solve 3D problems via 2D approaches. Thus, 3D point clouds are projected to 2D images through different viewpoints. Then, common 2D CNNs are capable to deal with 3D tasks. Voxel based methods [START_REF] Maturana | Voxnet: A 3D convolutional neural network for real-time object recognition[END_REF][START_REF] Tchapmi | Segcloud: Semantic segmentation of 3D point clouds[END_REF] usually map raw points to voxel representations first, then 3D CNNs are adopted to extract features. However, such methods do not naturally capture the inherent structure information of 3D point clouds. Therefore, point-set based methods are proposed to learn representative features directly from point coordinates. PointNet [START_REF] Qi | Pointnet: Deep learning on point sets for 3D classification and segmentation[END_REF] is the pioneer learning point-wise features for 3D classification and segmentation tasks. Further work [START_REF] Landrieu | Large-scale point cloud semantic segmentation with superpoint graphs[END_REF][START_REF] Thomas | Kpconv: Flexible and deformable convolution for point clouds[END_REF][START_REF] Hu | Randla-net: Efficient semantic segmentation of large-scale point clouds[END_REF] have shown their power in processing and learning from 3D point clouds. However, due to the limited main memory and possible information loss from downsampling large point clouds, it's still hard for these learning-based methods to train on large point clouds without downsampling.

In this Shape Retrieval Challenge (SHREC) track on 3D Point Cloud Semantic Segmentation for Street Scenes, we compare different frameworks to segment semantic objects from large-scale 3D point clouds. Both handcrafted and learning-based methods are included for evaluation. The contributions are summaried as:

• We provide a large-scale 3D street-scene point cloud dataset for 3D semantic segmentation.

• We evaluate different algorithms on the dataset and help finding solutions for large-scale 3D point cloud processing.

• The results show that point-set based end-to-end learning methods can outperform hand-crafted methods.

Street3D Benchmark

Related Work

In recent years, several annotated datsets for outdoor 3D point cloud semantic segmentation have been pub-lished. Semantic3D [START_REF] Hackel | Semantic3d.net: A new large-scale point cloud classification benchmark[END_REF], recorded with a terrestrial laser scanner (TLS), is a large outdoor 3D LiDAR dataset with over a billion points and 8 class labels. However, it has only 30 samples for training and testing, and the huge amount of points per sample (because of high precision of TLS) leads to heavy memory usage when training without downsampling. KITTI [START_REF] Geiger | Are we ready for autonomous driving? the KITTI vision benchmark suite[END_REF] offers nearly 15 thousands of scans of outdoor objects, but has only three classes (car, pedestrian and cyclist) and is labelled with bounding boxes instead of being point-wise annotated. It concerns more on 3D object detection instead of 3D semantic segmentation in large-scale scenarios. Paris-Lille-3D [START_REF] Roynard | Parislille-3d: A point cloud dataset for urban scene segmentation and classification[END_REF] is another worth noting 3D LiDAR dataset with 50 classes from which 10 classes are used for testing. It has over 143 million points with 7 different scans and is recorded by a mobile laser scanner (MLS). DublinCity [START_REF] Zolanvari | Dublincity: Annotated lidar point cloud and its applications[END_REF] provide a manual annotation of over 260 million laser scanning points with 13 hierachical classes in one scene. It has 3 coarse level classes (building, ground and vegetation) and 10 refined classes (e.g. door, tree and sidewalk). It is one of the densest urban LiDAR dataset recorded with an aerial laser scanner (ALS). These datasets have either the problem of huge size of a single scan (Semantic3D and DublinCity) or few scans for training and testing (Paris-Lille-3D and DublinCity).

In contrast to these datasets, our Street3D dataset provides less dense but more scenes for outdoor 3D semantic segmentation tasks. It consists of a large amount of well annoated points with 5 classes and 80 different street scenes. The size of each scene makes a trade-off between precision and productivity. A comparison is summaried in Tab. 2.

Dataset

The point cloud data is provided by Cyclomedia Technology, where we selected 80 3D point cloud for street scene and manually labelled them. The point clouds are recorded using both a panoramic image capturing device that capture every 5 meter together with a Velodyne HDL-32 Lidar sensor that performs continues capturing on the car. The recorded pointcloud is meshed, where for each pixel in the panoramic images, we compute the intersection (by ray tracing) with the dense mesh from the original pointcloud. The point of intersection provides us both color from the panoramic images capturing device together with position information.

Each point cloud represents a street scene and contains a group of objects. We used the open source software Cloud Compare [START_REF] Cloudcompare | 3D point cloud and mesh processing software open source project[END_REF] to manually label point clouds. Objects are labelled into 5 meaningful classes and an extra 'undefined' class. They are: 'Building', 'Car', 'Ground', 'Pole' and 'Vegetation'. Only the five meaningful classes will be evaluated. The 80 3D point clouds are randomly divided into train and test sets with 60 for training and 20 for testing. For each point cloud, there are over two million points. Summary information of the dataset is in Tab. 1. Examples of the dataset are visualized in Fig. 1. The dataset is made publicly available at https: //kutao207.github.io.

Evaluation

We adopt the evaluation criteria that have been widely applied in 3D semantic segmentation tasks, that is the Overall Accuracy (OA) and mean Intersection over Union (mIoU).

Generally, OA reports the percent of points in the data set which are correctly classified, and mIoU is the average of per-class IoU. The IoU of class i is defined as:

IoU i = TP i GT i + Pred i -TP i (1)
where TP i , GT i , Pred i denote the correctly classified number of points, the ground truth point number, and the predicted point number for class i, respectively.

Methods

In this part, we introduce four novel methods as well as a baseline method -PointNet++ for our 3D point cloud semantic segmentation task. The architectures of these four novel methods are varied. P4UCC is a nonlearning framework of a four-staged pipline which aims to classify the 5 classes progressively. Spherical DZNet projects 3D point clouds to 2D images and train a CNN model for semantic segmentation. Then the results are back-projected to 3D world and interpolated for 3D point clouds. ResGANet adopts graphs to encode the geometric information of 3D point clouds and use a residual graph attentional network to train an end-to-end model to predict point cloud semantic labels. GRandD-Net combines multi-scale features and Unet architecture to build a powerful model for the 3D point cloud semantic segmentation task. More details are described as follows. This method is contributed by authors Sebastián Ortega, Agustín Trujillo, José Pablo Suárez, José Miguel Santana, and Cristian Ramírez. Our proposal is a fourstaged pipeline that aims to progressively classify ground, car, pole, building, and vegetation points in vehicle-borne point clouds. Its general architecture can be seen in Fig. 2. Each stage aims to identify only a certain class. The first one is dedicated to ground point detection and also generates pixel-wise features which are useful for subsequent stages. Stage 2 looks for the detection of car points, Stage 3 filters pole points and Stage 4 splits the remaining points between building and vegetation classes. The code is available at https://github.com/ChanoOT/P4UCC.

P4UCC: Progressive 4-staged Urban Cloud Classifier

Ground Detector

Ground detection starts by dividing the cloud horizontally in a grid with squared cells of 1m 2 . Pixel features are computed using the points in each cell: minimum height, H [START_REF] Ortega | An image-based method to classify power line scenes in lidar point clouds[END_REF]; point accumulation, A; and geometric features such as omnivariance, O; and eigensum, E [START_REF] Blomley | Shape distribution features for point cloud analysis-a geometric histogram approach on multiple scales[END_REF]. Minimum height is used to select the ground candidates: all points in the cloud with height lower than H i j + 0.25. Those points are then grouped using a hierarchical approach, euclidean distance and cutoff of 0.5 m. Cluster centroids and their inclination angles with respect to the LiDAR sensor are computed. In case the sensor position is not known, the mean of point positions of the maximum A i j can be used as approximation. This can be seen in Fig. 3. The point clusters deemed ground are the ones with an inclination angle lower than the one of the vehicle in which the sensor is mounted. For the SHREC benchmark, it was approximated to 1.5 degrees. The cluster with the largest point size is also added.

Car Detector

Points from cells with E i j < 1.25 are considered for car detection. These points are grouped using the same hierarchical clusters of Stage 1. A cell descriptor is calculated as ∆X * ∆Y. A planarity descriptor [START_REF] Zhao | Classifying airborne lidar point clouds via deep features learned by a multi-scale convolutional neural network[END_REF] is also calculated per cluster. A cluster is considered to represent a car when its area is shorter than 40m 2 and the planarity is greater than 0.2. Examples of this can be seen in Fig. 4, right.

Pole Detector

Points from cells with O i j < 2 are considered for pole detection and clustered. Curvature change [START_REF] Blomley | Shape distribution features for point cloud analysis-a geometric histogram approach on multiple scales[END_REF], dominant component, minimum relative height and deviation of the XY component values are computed per cluster. Point clusters are classified as poles when the dominant component is Z, the curvature change is lower than 0.01, the relative minimum height is lower than 1 and the deviation of the values is lower than 1 in both X and Y components. This can also be seen in Fig. 4 (left).

Building Detector

The remaining points are grouped as well. A recursive algorithm for extraction of vertical planes based in MLESAC [START_REF] Torr | MLESAC: A new robust estimator with application to estimating image geometry[END_REF] is applied to each cluster. Clusters with a majority of points in a vertical plane and with ∆Z > 2.5m are considered to represent buildings. Otherwise, they are classified as vegetation. This leads to the final result that can be seen in Fig. 5. This method is contributed by authors David Duque-Arias, Santiago Velasco-Forero, Jean-Emmanuel Deschaud, Francois Goulette and Beatriz Marcotegui. We propose an automatic algorithm for point cloud segmentation using only geometrical data [x, y, z]. The proposed methodology is based on 2D projections. The main process relies on a spherical projection from the sensor point of view while bird eye view (BEV) projection is used for ground detection. The FAISS library [START_REF] Johnson | Billionscale similarity search with gpus[END_REF] is used in order to reduce computational cost at several steps in the process. The proposed algorithm is presented in Fig. 6 and it is mainly divided into four steps: 1) Coarse ground detection from Bird-Eye View (BEV) by means of quasi-flat zones; 2) Spherical projections and feature extraction; 3) Train Deep Learning model with 2D images; 4) Postprocessing and backprojection 2D predictions to 3D using a kNN classfier (with k = 3). The code is available at https://github.com/daduquea/sphericalDZ.

Spherical DZNet

Coarse Ground Detection

High class imbalance is a typical issue in point cloud segmentation. In order to reduce it, ground points are identified first. A simple quasi-flat zones algorithm from the BEV projection [START_REF] Hernández | Point cloud segmentation towards urban ground modeling[END_REF] is used for this purpose, assuming that ground elevation varies smoothly.

Segmentation in 2D projections

In the spherical projection, each point cloud is represented by an image of 1024 × 64 pixels corresponding to the azimuthal and elevation angles, respectively. A set of five features is selected in this study: z coordinate, depth (distance to sensor) and the normal vector in 3D coordinates denoted by (N x , N y , N z ). A very simple normal estimation based on depth image gradient [START_REF] Nakagawa | Estimating surface normals with depth image gradients for fast and accurate registration[END_REF] is used. Only points that have not been classified as ground are projected. If several 3D points fall in the same projected pixel, the classical approach to project the closest 3D point to the scanner at each pixel of the 2D image is used. Spherical projection has been chosen because the dataset is acquired with a terrestrial scanner. The scanner position is not given. We estimate it from the BEV as the barycentre of the empty space surrounded by the highest point density. Fig. 7 shows ground truth and computed features of '5D4KVPBP' from train set after ground extraction.

We trained a Deep Learning model using a variation of Unet architecture [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] as presented in Fig. 8. This choice was motivated by its capability to include input features at different steps of the decoding stage of the network. Specifically, the original input is directly concatenated to the features computed by the network as a shortcut connection, as is shown in the Fig. 8. Including shortcuts in the model has demonstrated several advantages in image segmentation tasks [START_REF] Drozdzal | The importance of skip connections in biomedical image segmentation[END_REF] such as increasing convergence speed and stabilizing weight updates. Two strategies have been tested: 1) Predict labels using trained model (The code is available at https://github.com/daduquea/ sphericalDZ); 2) Remove last layer of the model and predict labels using kNN classifier. Results were slightly better and faster in the second case.

As the model is trained in 2D, it is required to backproject predicted labels to point clouds. In a similar way as proposed by [START_REF] Milioto | Rangenet ++: Fast and accurate lidar semantic segmentation[END_REF], we perform kNN to label every 3D point that is not projected in 2D image. We experimentally found that best results are obtained with k equal to three.

Postprocessing

In order to improve the classification of some points, we propose an automatic postprocessing step to correct some of the predicted labels using mathematical morphology. We performed the postprocessing in three steps:

Poles: Poles are thin structures in front of other larger structures. Thus, they appear as regional minima in the depth projection. Those minima are added to the detected poles. Then, only structures with a geodesic elongation [START_REF] Lantuéjoul | Geodesic methods in quantitative image analysis[END_REF] larger than 20 are preserved.

Building: In the BEV, structures higher than 2.5 meters and with an elongation larger than 20 are considered as buildings.

Small regions: Regions classified as vegetation or building, with less than 100 pixels in the spherical view are given the class label of the large surrounding neighbors.

ResGANet: Residual Graph Attentional Networks

Preprocessing

In this part, we downsample the large 3D point clouds to smaller sizes. We apply a voxel subsampling method [START_REF] Hu | Randla-net: Efficient semantic segmentation of large-scale point clouds[END_REF] with voxel size set to 0.1m so that dense areas can be forced to be sparse while still contain enough information. Then, we adopt a farthest point sampling [START_REF] Qi | Pointnet: Deep learning on point sets for 3D classification and segmentation[END_REF] algorithm to sample the reduced point clouds to a fixed point number 8096.

ResGANet Architecture

Our propose Residual Graph Attentional Networks to learn representative features directly from point coordinates. It consists of four main parts, namely, Graph Construction, Residual Graph Convolution Module, Graph Pooling and Attentional Module. The overall architecture is shown in Fig. 10.

Graph Construction

The graph construction module aims at creating graph representations over point clouds. Given a 3D point cloud with N points, matrix X ∈ R N×3 denotes the input coordinates. Though there are many methods to construct a graph, in this paper, we define a graph using k-nearest neighbor (kNN) algorithms. Namely, for each point in a point cloud, we query the k nearest neighbors and connect the point with these neighbors. Considering that our graph is undirected, but kNN searching cannot ensure the graph is symmetrical, we define the adjacency as follows,

A(i, j) = δ j (i) + δ i ( j) 2 exp -β||x i -x j || 2 (2)
where δ i ( j) is an indicator function indicating whether node j is in the k-nearest neighbors of node i. If node j is in the k-nearest neighbors of node i, δ i ( j) = 1, otherwise, δ i ( j) = 0. x i denotes the feature vector of node i. In our approach, we set the kNN query number as 16 throughout our experiments.

Residual Graph Convolution Module

Similar to the residual block in ResNet [START_REF] He | Deep residual learning for image recognition[END_REF], we define our residual graph convolution module by two graph convolution units and a skip connection from the input to the output addition. We adopt the graph convolution concept in [START_REF] Sandryhaila | Discrete signal processing on graphs[END_REF] as

Y = AXW (3)
where X ∈ R N×C is the input features, Y ∈ R N×D is the filtered output, W ∈ R C×D is the weight matrix. Notice that the A also encodes a (K + 1)-scalar weight variables. Further, the residual block is defined as

Y res = G(G(X, A, d), A, d) + X (4)
where X denotes the feature input, A is the corresponding adjacency matrix and d is the degree which denotes the highest power of the dajacency matrix. G(•) is the graph convolution function.

Graph Pooling

We use kNN max pooling to define our graph pooling. Given the point cloud coordinates X 0 ∈ R N 0 ×3 , in order to preserve the structure of point clouds, farthest point sampling [START_REF] Qi | Pointnet: Deep learning on point sets for 3D classification and segmentation[END_REF] is used to iteratively choose the desired N 1 points. Based on that, we obtain the reduced point coordinates X 1 ∈ R N 1 ×3 and the corresponding index in X 0 . Suppose the input feature is F 0 ∈ R N 0 ×D . Then, for each point in the reduced points, kNN is used to find the K nearest features in F 0 such that we obtain a feature matrix of size N 1 × K × D . Finally, max pooling operation is applied to get the new feature of each selected point. An illustration of graph pooling is shown in Fig. 11. Note that graph pooling changes the point cloud structure. Thus, the adjacency matrix in afterwards graph convolutions needs to be updated by pooled points' coordinates.

Attentional Module

As attention has been successfully applied in many areas, the proposed graph attention module focuses on gath-ering information from important points. That is, the attention module tries to model point relations by finding out the importance of one point over all other points, and strengthening the feature of strong points while supressing weak point representations.

The attention module has three branches. Two branches are to calculate pairwise similarity, then the similarity output is fed into a sigmoid function to generate the attention matrix. Each element of the matrix indicates the relative influence one point has on another point compared to all other points. Instead of directly applying pairwise vector production, we first use two graph convolution layers to dynamically learning different representations of the identical input feature. Then, in case of rapid gradient descending, we use a residual-like operation by adding the attention output with the original input feature to get the final output. In this way, we re-balance the features of different points by considering each point's influence on others.

Given a feature matrix of X with N points, our attention output can be modeled as:

Y att = sigmoid( f 1 (X) f T 2 (X) √ D ) f 0 (X) + X ( 5
)
where f 0 (•), f 1 (•) and f 2 (•) are feature extractors, and D is the number of feature channels.

Interpolation

In this part, we need to backward project the downsampled predicted labels to the raw point clouds. We use kNN to do the interpolation. Give the downsampled point coordinates with predicted labels and raw point coordinates, we determine the raw per-point label with k = 3 for the kNN search.

GRanD-Net: Grid subsampling, RandLA-Net and Dilated Convolutions

This method is contributed by authors Kiran Akadas and Shankar Gangisetty. We propose a deep-learning based 3D semantic segmentation of point clouds inspired from the works of RandLA-Net [START_REF] Hu | Randla-net: Efficient semantic segmentation of large-scale point clouds[END_REF] known as GRanD-Net shown in Fig. 12. We improve the RandLA-Net [START_REF] Hu | Randla-net: Efficient semantic segmentation of large-scale point clouds[END_REF] by considering point features in terms of (x, y, z) coordinates, subsample using Grid subsampling and adopting dilated convolutions [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF] to gain better efficiency for semantic segmentation of large-scale 3D point clouds. The code is available at https://github.com/ KiranAkadas/GRanDNet.

Data Preparation

To efficiently process data of such large-scale 3D scenes, we down-sample the point clouds using grid subsampling of KPConv [START_REF] Thomas | Kpconv: Flexible and deformable convolution for point clouds[END_REF] with a fixed grid size 0.06m. In order to get back the original number of semantic labels from the predictions, we index projections for upsampling the point clouds. The training dataset augmented by scaling and rotation.

Data Loading

To load the data in batches, we generate the data flow for each batch. For a given batch size n and the steps in each epoch s, (n × s) point clouds are reserved for each epoch. To avoid ordered learning by the GRanD-Net model, we feed the data randomly. The k-Nearest Neighbours (kNN) algorithm is used with a pre-defined set of k neighbours being selected of all the sub-sampled points. If the sampled points are less than the given predefined k points, we pick the points with replacement. To prepare a batch of point clouds, we generate the neighbour indices for every point in a point cloud. These are used to get the relative point features. We then randomly sample 25 % of points to be reduced in the next phase while down-sampling and simultaneously track the indices for up-sampling. The pooling indices are obtained using kNN search for every sampled point.

Training

We train the sampled point clouds over several batches of data. The loaded point clouds of dimensions (N, d in ), where N is the number of points in the point cloud and d in is the number of features associated with each point in the point cloud, are processed using the dilated residual blocks (DRBs) shown in Fig. 12. Each of the DRB includes multiple units of local spatial encoding and attentive pooling stacks. The DRBs are connected through skip-connections as proposed in RandLA-Net [START_REF] Hu | Randla-net: Efficient semantic segmentation of large-scale point clouds[END_REF]. Unlike normal convolutions used in RandLA-Net, we use dilated/atrous convolutions [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF] to implement the DRB in order to increase the receptivity of the filters without affecting the resolution and gain better efficiency shown in Fig. 13. The dilated convolutions incorporate multi-scale features, essential for semantic segmentation. These dilated convolutions make our proposed GRanD-Net faster and more efficient since we are increasing the area of filter coverage without increasing the parameters and affecting the feature learning, thus reducing the number of convolutions. The local spatial encoding in DRB uses the centre points and their kNN neighbours to encode the point cloud using relative positional information. At each step, we apply random sampling with DRB using the points we loaded earlier to reduce the size of input point cloud to 25%. The attentive pooling is used as a replacement to max-pooling in order to compute the attention score for every feature which is further aggregated to avoid loss of information and learn important local information. We use GeLU [START_REF] Hendrycks | Gaussian error linear units (gelus)[END_REF] as the activation layer in our GRanD-Net model for better learning of non-linear features. GeLU prevents strong negative activations which may affect the model. The curvature and non-monotonicity of GeLU is used to learn complex functions much better compared to ReLU and leaky ReLU. The output of the stacked DRBs is up-sampled and passed through multi-layer perceptrons (MLP) followed by fully connected layers shown in Fig. 12. The use of skip-connections and MLP while upsampling ensures that the labelling is accurate. Our approach follows an all-inclusive up-sampling approach that refines labels gradually unlike simple interpolation, which would result in a single label for a group of points ignor- ing the demarcation of classes. The use of interpolation would further require the use of a post-processing technique to refine the output. Finally, the predicted semantic labels for every point are obtained as the output of the model with dimensions (N, d out ), where d out is the number of classes in the dataset.

Experimental Settings

We train our proposed GRanD-Net model for 50 epochs with a train-validation split of 3:1 for the 5 classes leaving the undefined class. A four layered network is used with feature sizes of 16, 64, 128, and 256 while training. We train GRanD-Net using the Adam optimizer with a learning rate of 0.01 and a decay rate of 0.05. A grid size of 0.06 is fixed for grid-subsampling while training and we select k = 16 nearest neighbours to be queried. To train our model, we sample a fixed number (N) of 65,536 points from each point cloud as the input and use a batch size of

PointNet++

We choose PointNet++ [START_REF] Qi | Pointnet++: Deep hierarchical feature learning on point sets in a metric space[END_REF] as our baseline method for the evaluation. PointNet++ takes point coordinates as input and is able to learn deep features efficeiently and robustly. It's a very popular base algorithm for many 3D point cloud related tasks.

The implementation for semantic segmentation task is available via https://github.com/intel-isl/Open3D-PointNet2-Semantic3D.

Results and Discussion

Experimental Results

The evaluation results are shown in Tab. 3. Overall accuracy (OA) and mean Intersection over Union (mIoU) are evaluated for all classes, and for each class, the IoU metric is also calculated. We also plot the confusion matrix of each algorithm in Fig. 16 to help investigate the weakness of each model.

There are 5 methods under evaluation: Baseline (PointNet++), P4UCC, Spherical DZNet, ResGANet and GRanD-Net. P4UCC is based on a combination of 4 hand-crafted detectors, the other four methods are learning-based. In these four learning-based methods, Spherical DZNet is a two-stage approach with a ground detector and a projection based deep learning architecture for training and predicting, while the other three take point coordinates as input and learning representations directly from points in an end-to-end way.

Computation Complexity and Timing

In Tab. 4, we show the number of trainable parameters and inference time as criterias for computation complexity and timing, since these 5 methods are run on different platforms and each has very different architecture and input format. We list the time as a reference.

Discussion

As shown in Tab. 3 and Fig. 16, we have the following observations:

• From obtained results with Spherical DZNet, we have identified some drawbacks related to spherical projections: 1) Objects size strongly depends on distance to sensor; 2) High variation of objects shape according to sensor perspective. Both of them may affect the capability of the model to learn. In addition, we identified a particularly low score in pole class. We think it is mainly due to the presence of traffic sign of point-cloud 5D4L1RW5. Few traffic signs are present in the dataset. Moreover, this one is close to the scanner and has over 150 thousand 3D points.

• It is worth noting that P4UCC is the only nonlearning-based approach but still outperforms 3 learning-based methods (Baseline, Spherical DZNet and ResGANet) by using four detectors to identify point categories progressively. It shows that handcrafted method is still reliable and may outperform learning-based methods.

• GRanD-Net achieves superior performance on four of the classes, except pole. We observe that the resulting IoU of ground, vegetation, building, and car classes are segmented accurately as the dataset distribution in these classes is high and learnt better. We 16 [START_REF] Boulch | Unstructured point cloud semantic labeling using deep segmentation networks[END_REF]. Based on the confusion matrix and visual inspection, we observe that few instances of the pole class are mislabelled as building shown in Fig. 15, due to their proximity to the building points and also the insufficiency of the pole training points.

• It is obvious that 'Building' and 'Ground' are easy to be correctly classified while 'Car' and 'Pole' are hard to be segmented. Considering the proportion of 'Car' and 'Pole' in the train/test set is 2.82%/2.19% and 0.47%/0.78%, all four learning-based methods cannot segment minority classes well. Non-learningbased P4UCC performs better on minority classes.

• We also observe that all three point-set learningbased methods (Baseline, ResGANet and GRanD-Net) have to downsample the large point clouds before model training due to the limit of hardware computation power. It may more or less result in information loss and influence the segmentation performance.

It is obvious that the end-to-end learning approaches are the trend to tackle with the 3D point cloud semantic segmentation task. They usually have powerful ability to segment certain classes, however they may also suffer from imbalanced data. The traditional hand-crafted method may not outperforms learning-based methods on certain classes but when the data is imbalanced, they can show their robustness and reliability.

Another issue worth noting is the limited classes of our Street3D dataset. For the 3D semantic segmentation task, it's sufficient to have 5 classes. However, it may not be enough for specific tasks. For example, 'road' is an important category for autonomous driving application. However, Street3D has 'ground' instead of 'ground'. It's not easy to recover road from only 3D 'ground' points. But our Street3D can still contribute to training and validating deep learning networks for autonomous driving application. The 4 learning-based methods in this paper can be easily transferred to or finetuned on a dataset with a 'road' label as what have usually been done in 2D CNN applications.

Conclusions

In conclusion, this track has drawn attention on 3D point cloud semantic segmentation for street scenes using multiple approaches. We provide a street scene dataset composed of 80 scans which are well annotated with five class labels. We introduce four novel and different methodologies that outperform a state-of-theart deep learning based method (PointNet++) for the new Street3D benchmark. As there is a non-learning method (P4UCC) which also outperforms learning-based approaches, it shows that well-designed feature descriptors for the classification could have more importance in the segmentation than the learned features, specifically with unbalanced data. The four methods have focused not only on different architectures but also on different features (covariance-based features in P4UCC, multi-scale features in GRanD-Net, graph representations in Res-GANet, spherical descriptors in Spherical DZNet), which opens the possibility of combining some of the descriptors from each discussed proposal in a new architecture. The task is still challenging as small data size classes ('Car' and 'Pole') are still hard to be classified. More work need to be done for solving the performance imbalance over classes.

Acknowledgement

The work "Spherical DZNet" was partially funded by REPLICA FUI 24 project and ARMINES.

Figure 1 :

 1 Figure 1: Examples of the Street3D datasets. The bottom right strings are the filename of each scene.

Figure 2 :

 2 Figure 2: Multi-stage architecture of the vehicle-borne point cloud segmentation process.

Figure 3 :

 3 Figure 3: Minimum height clusters and sensor position (left figure, sensor in red) vs final ground cells (right).

Figure 4 :

 4 Figure 4: Examples of pole (left, green) and car (right, green) clusters in two clouds.

Figure 5 :

 5 Figure 5: Final result in 5D4KVPG4 cloud. Building, pole, vegetation, car and ground are represented with red, blue, green, magenta and yellow colors, respectively.

Figure 6 :

 6 Figure 6: Workflow of automatic point cloud segmentation.

Figure 7 :

 7 Figure 7: Crops of spherical projections of point cloud.

  This method is contributed by authors Tao Ku, Remco C. Veltkamp and Bas Boom. We introduce the Residual Graph Attentional Networks (ResGANet) for the 3D point cloud semantic segmentation task. The ResGANet takes the semantic segmentation task as a supervised learning task. We use the 60 point clouds with semantic labels in training dataset to train and validate our model. As shown in Fig. 9, the whole pipeline is composed of three parts: preprocessing, model training & evaluation, and backprojection. As our model training is based on the proposed Residual Graph Attentional Networks (ResGANet) and the input is in a fixed point number, we need to have a voxel subsampling first and then sampling the point cloud to a fixed size. After training and evaluation, we also need to back-project the valuation results on downsampled point clouds to the original ones. The code is available at https://github.com/keruast/ResGANet.

Figure 8 :

 8 Figure 8: Proposed Network architecture. Yellow boxes represent Conv2D layers of 5x5 with strides (1,2), red boxes Conv2D of 1x1 followed by a dropout of 0.1. Blue color means Upsampling2D layer. Violet arrows illustrate concatenation operator. Note: Original input is directly concatenated to the features computed by the network.

Figure 9 :

 9 Figure 9: Pipeline of the proposed ResGANet method.

Figure 10 :

 10 Figure 10: Overall architecture of the proposed Residual Graph Attention Networks (ResGANet). ResGANet consists four main components: graph construction, residual graph module, attention module and graph pooling. The number after ResGCM and ATM denotes the filter number while the number after graph pooling means the pooling stride.

Figure 11 :

 11 Figure 11: The illustration of graph pooling. The goal is pooling input N 0 × 3 points with N 1 × D feature to N 1 × 3 points with N 1 × D feature. Each point in the figure represents a feature vector. The red ones denotes points chosen by farthest point sampling algorithm while blue ones denotes rest unchosen points. Each green circle after step 2 indicates the kNN search results. Final dark red ones are max pooling output for each chosen point feature together with their K-nearest point features.

Figure 12 :

 12 Figure 12: The proposed architecture of GRanD-Net. The abbreviations followed in the architecture are DRB: Dilated residual block, RS: Random sampling, US: Up-sampling, FC: Fully connected layer, DP: Dropout.

Figure 13 :

 13 Figure 13: Convolutions: (left) 3 × 3 regular convolution mostly used in CNNs. (right) 3 × 3 dilated convolution with d=2 covering an area of 5 × 5 is used in GRanD-Net resulting in lesser strides and parameters.

Figure 14 :

 14 Figure 14: Qualitative results of GRanD-Net on the test set (5D4KVQ9U and 5D4KX3TQ point clouds).

Figure 15 :

 15 Figure 15: Qualitative results of GRanD-Net on the test dataset. The black circle shows the pole mislabelled as building.

Figure 16 :

 16 Figure16: Confusion matrix for all the five comparison algorithms. The x-axis represents the predicted labels while the y-axis denotes the groundtruth labels. In this figure, each subfigure shows clearly the classification ability of each algorithm. We normalized each row of the confusion matrix so that it is intuitive to show the probability of correct predictions and incorrect predictions.

Table 1 :

 1 Summary of Train and Test datasets of our Street3D benchmark.

	Split	Num. of Samples	Num. of Per-sample Points	Total Points	Undefined Building	Class Ratio Car Ground	Pole	Vegetation
	Train	60	2.40M -4.67M	217M	8.37%	17.05% 2.82% 54.65% 0.47%	16.64%
	Test	20	2.30M -4.50M	72.8M	6.08%	22.07% 2.19% 53.96% 0.78%	14.92%

Table 2 :

 2 Comparison of some outdoor 3D LiDAR datasets. For LiDAR type, TLS means Terrestrial Laser Scanning, MLS stands for Mobile Laser Scanning and ALS, Aerial Laser Scanning.

	Annotation

Name

LiDAR Type Num. of Points Num. of Scans Num. of Classes

Table 3 :

 3 Evaluation results of our Street3D benchmark.

	Method	OA (%) mIoU (%) Building	Car	Ground Pole Vegetation
	Baseline	91.30	66.39	82.52	40.13	89.10	39.46	80.72
	P4UCC	94.13	72.25	84.35	60.51	96.46	40.18	79.75
	Spherical DZNet	93.89	67.30	83.16	49.93	96.46	27.52	79.41
	ResGANet	93.55	71.39	82.96	57.66	94.40	32.94	88.98
	GRanD-Net	97.83	86.40	93.66	83.92	98.10	61.79	94.55
	4 with 500 steps per epoch. The best performing GRanD-
	Net model is frozen with a mean intersection over union
	(mIoU) of 84.11%. The frozen model is used to predict
	segments for the 20 test point clouds that contain a total
	of 72,753,747 points. The resulting mIoU is 86.4% with
	an overall accuracy (OA) of 97.83% for 5 classes shown
	in Tab. 3.

Table 4 :

 4 Computation cost for participated methods.

			Number of	Seconds per
	Method	Platform	trainable	million points for	Other time
			parameters	inference
	P4UCC	CPU Intel i5 dual-core, 8GB RAM, Matlab2018b	-	-	17.91 seconds per million points
	Spherical DZNet	CPU Intel Xeon E-2186G, 32GB RAM, GPU Nvidia Titan X 12 GB	88.8K	-	Spherical projection: 0.15s, 2D training one epoch: 3.8s, 2D prediction: 1.15s, 2D to 3D: 0.81s, kNN search: 50s
		CPU Intel i7-8700, 16GB
	Baseline	RAM, GPU Nvidia	0.97M	768.13s	-
		GTX-1080ti 11 GB
		CPU Intel i7-8700, 16GB
	ResGANet	RAM, GPU Nvidia	1.76M	850.31s	-
		GTX-1080ti 11 GB
		Trained on Google Cloud
		Platform, CPU -4 vCPU,
	GRanD-Net	15 GB Memory (Powered	0.99M	219.37s	-
		by Intel Skylake), GPU-
		NVIDIA Tesla T4 16GB


Maude Guillier

Francis Repoila

RNA and gene control

The study of RNA has led to fundamental discoveries in molecular biology and genetics. RNA is central to the decoding of genetic information, playing a role both as the messenger (mRNA) and as the major component of the apparatus that carries out protein synthesis (tRNA and rRNA). RNA also performs crucial functions in gene expression, as illustrated by transcribed RNA sequences that are spliced out (introns), or by RNAs with catalytic activity (ribozymes). With the more recent discovery of regulatory non-coding RNAs, and the identification of control elements within mRNAs that do not require "classical" regulatory factors (riboswitches), RNA has become a major and unavoidable player in the control of gene expression in all domains of life. In bacteria, it is involved in adaptation processes and in the responses to environmental cues. The breadth of functions and roles of RNA in bacterial gene regulation is the focus of this special issue.

The first chapters illustrate the different pathways of the biogenesis of regulatory RNAs, and the mechanisms for controlling their activity via ribonucleases and RNA sponges. The diversity of molecular processes mediating RNA regulation is then highlighted in chapters devoted to riboswitches, antisense RNAs, and base-pairing RNAs, as well as to the protein cofactors involved in RNA regulation, such as ProQ, Hfq or Rho.

In parallel to the diversity of modes of action, multiple roles have been described for regulatory RNAs in bacterial physiology. This is exemplified in this issue by the role of RNAs in the carbon-nitrogen balance and in heterocyst formation in cyanobacteria, the link between bacterial growth and RNA metabolism and the possible function of RNAs in host-pathogen interactions via extracellular vesicles.

Importantly, the discovery and the understanding of RNA functions have greatly benefited from technological developments and, conversely, also provided grounds for the establishment of new technologies. A chapter discussing the strengths and weaknesses of the recent advances in high-throughput methods applied to RNA also features in this issue.

The current fundamental knowledge presented here is the result of a body of work carried out in various laboratories around the world. They are intellectually elegant and exciting. Nevertheless, the reader should keep in mind that the study of RNA continues to have applications in medicine and industry; RNAi technology or the use of CRISPR in genome editing are just a couple of obvious examples. Furthermore, as illustrated by the authors of this collection of articles, we still have a lot to discover and understand within the RNA world. Maude Guillier did her PhD studies on the translational feedback control of ribosomal proteins synthesis in Escherichia coli at the IBPC in Paris with Drs. Claude Chiaruttini and Mathias Springer. She then started to work on small RNA-mediated regulation as a postdoctoral researcher in the lab of Susan Gottesman at the NIH, USA, from 2004 to 2008, where she studied the role of sRNAs in modulating membrane composition. She continued to develop work centered on bacterial RNA regulation after she moved back to France. She is now a PI at the IBPC in Paris and her major research interests focus on the interplay between transcriptional and post-transcriptional regulatory circuits as well as the mechanisms of control of bacterial translation.

Francis Repoila started to work on regulatory RNAs more than 20 years ago as a post-doc in the laboratory of Susan Gottesman (NIH, USA), where he contributed to the first systematic discovery of sRNAs in Escherichia coli, and their involvement in bacterial adaptation in response to environmental cues by working on the thermocontrol of the RpoS sigma factor. Since his return to France, as a Research Associate, he has contributed to the discovery of sRNAs in Gram-positive bacteria: in Listeria, in Pascale Cossart's lab (at the Pasteur Institute), and in Enterococcus at INRAE (at the Micalis Institute). Since a few years now, he investigates switches enabling enteric bacteria to become pathogens within the intestine, by global dynamic approaches where RNA is a key compound to decipher regulatory pathways.