Luis Roldão
email: luis.roldao@inria.fr

Raoul De Charette
email: raoul.de-charette@inria.fr

Anne Verroust-Blondet

3D Semantic Scene Completion: a Survey

Keywords: Semantic scene completion, 3D Vision, Semantic Segmentation, Scene understanding, Scene reconstruction, Point cloud Sensor Pose Estimation, SS: Semantic Segmentation, OC: Object Classification, SNE: Surface Normal Estimation

14 2988 - ‡ Simulated sensor. † Synthetically augmented. a Mesh annotations from [52]. b Derivates from NYUv2 [133] by rendering depth images from mesh annotation. c Derivates from subset of SUNCG [137] where missing RGB images were rendered. d Derivates from ScanNet [24] by fitting CAD models to dense mesh.

INDOOR OUTDOOR

NYUv2 [133] / NYUCAD [START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF] (Real-world / Synthetic)

SUNCG [137] (Synthetic)
SemanticKITTI [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] (Real-world) Abstract Semantic Scene Completion (SSC) aims to jointly estimate the complete geometry and semantics of a scene, assuming partial sparse input. In the last years following the multiplication of large-scale 3D datasets, SSC has gained significant momentum in the research community because it holds unresolved challenges. Specifically, SSC lies in the ambiguous completion of large unobserved areas and the weak supervision signal of the ground truth. This led to a substantially increasing number of papers on the matter. This survey aims to identify, compare and analyze the techniques providing a critical analysis of the SSC literature on both methods and datasets. Throughout the paper, we provide an in-depth analysis of the existing works covering all choices made by the authors while highlighting the remaining avenues of research. SSC performance of the SoA on the most popular datasets is also evaluated and analyzed.

Introduction

Understanding the 3D surroundings is a natural ability for humans, who are capable to leverage prior knowledge to estimate geometry and semantics, even in large occluded areas. This proves more difficult for computers, which has drawn wide interest from computer vision researchers in recent years [START_REF] Guo | Deep learning for 3D point clouds: A survey[END_REF]. Indeed, 3D scene understanding is a crucial feature for many applications, such as robotic navigation or augmented reality, where geometrical and semantics understanding is key to leverage interaction with the real world [START_REF] Garg | Semantics for robotic mapping, perception and interaction: A survey[END_REF]. Nonetheless, vision sensors only provide partial observations of the world given their limited field of view, sparse sensing, and measurement noise, capturing a partial and incomplete representation of the scene.

To address this, Scene Completion (SC) has been used for a long time to infer the complete geometry of a scene given one or more 2D/3D observations, historically using a wide variety of more or less sophisticated interpolation methods. With recent advances of 3D deep learning, Semantic Scene Completion (SSC) has been introduced as an extension of SC, where semantics and geometry are jointly inferred for the whole scene, departing from the idea that they are entangled [137]. Consequently, along with the addition of semantics, the SSC task has significantly departed from original SC in terms of nature and sparsity of the input data. Fig. 1 shows samples of input and ground truth for the four most popular SSC datasets. The complexity of the SSC task lies in the sparsity of the input data (see holes in depth or LiDAR input), and the incomplete ground truth (resulting of frame aggregation) providing a rather weak guidance. Different from object-level completion [START_REF] Avetisyan | Scan2CAD: Learning CAD model alignment in RGB-D scans[END_REF]172] or from scene reconstruction where multiple views are aggregated [109,182], SSC requires in-depth understanding of the entire scene heavily relying on learned priors to resolve ambiguities. The increasing number of large scale datasets [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF]133,137,140] have encouraged new SSC works in the last years. On connex topics, 3D (deep) vision has been thoroughly reviewed [START_REF] Guo | Deep learning for 3D point clouds: A survey[END_REF][START_REF] Li | Deep learning for LiDAR point clouds in autonomous driving: A review[END_REF][START_REF] Liu | Deep learning on point clouds and its application: A survey[END_REF][START_REF] Lu | Deep learning for 3d point cloud understanding: A survey[END_REF] including surveys on 3D representations [START_REF] Ahmed | A survey on deep learning advances on different 3d data representations[END_REF] and taskoriented reviews like 3D semantic segmentation [168,175], 3D reconstruction [109, 182], 3D object detection [START_REF] Jiao | A survey of deep learning-based object detection[END_REF], etc. Still, no survey seems to exist on this hot SSC topic and navigating the literature is not trivial.

In this article, we propose the first comprehensive and critical review of the Semantic Scene Completion (SSC) literature, focusing on methods and datasets. With this systematic survey, we expose a critical analysis of the SSC knowledge and highlight the missing areas for future research. We aim to provide new insights to informed readers and help new ones navigate in this emerging field which gained significant momentum in the past few years. To the best of our knowledge this is the first SSC survey, which topic has only been briefly covered in recent 3D surveys [START_REF] Guo | Deep learning for 3D point clouds: A survey[END_REF][START_REF] Lu | Deep learning for 3d point cloud understanding: A survey[END_REF].

To study SSC, this paper is organized as follows. We first introduce and formalize the problem of Semantic Scene Completion in Sec. 2, briefly brushing closely related topics. Ad-hoc datasets employed for the task and introduction to common 3D scene representations are covered in Sec. 3. We study the existing works in Sec. [START_REF] Avetisyan | Scan2CAD: Learning CAD model alignment in RGB-D scans[END_REF], highlighting the different input encodings, deep architectures, design choices, and training strategies employed. The section ends with an analysis of the current performances, followed by a discussion in Sec. 5.

Problem statement

Let x be an incomplete 3D representation of a scene, Semantic Scene Completion (SSC) is the function f (.) inferring a dense semantically labeled scene ŷ such that f (x) = ŷ, best approximates the real 3D scene y. Most often, x is significantly sparser than y and the complexity lies in the inherent ambiguity, especially where large chunks of data are missing, due to sparse sensing or occlusions (see Fig. 2). Subsequently, the problem cannot be addressed by interpolating data in x and is most often solved by learning priors from (x, y) pairs of sparse input and dense 3D scenes with semantic labels.

The nature of the sparse 3D input x greatly affects the task complexity. While 3D data can be obtained from a wide variety of sensors, RGB-D/stereo cameras or LiDARs are commonly employed. The former, for example, provides a dense description of the visible surfaces where missing regions correspond to occluded areas, as shown in Fig. 2a. This reduces the SSC task to estimating semantic completion only in the occluded regions [137]. Conversely, LiDAR data provides considerably sparser sensing, with density decreasing afar and point-wise returns from laser beams cover an infinitesimal portion of the space leading to a high proportion of unknown volume, as shown in Fig. 2b.

The rest of this section provides the reader with a brief overview of related tasks, covering early works and foundations.

Related tasks

Semantic scene completion inspires from closely related tasks like shape completion, semantic segmentation, and more recently semantic instance completion. Thereof, SSC benefits from individual insights of these tasks which we briefly review their literature pointing to the respective surveys.

Completion. Completion algorithms initially used interpolation [START_REF] Davis | Filling holes in complex surfaces using volumetric diffusion[END_REF] or energy minimization [START_REF] Kazhdan | Poisson surface reconstruction[END_REF]100,139] techniques to complete small missing regions. Completion works were first devoted to object shape completion, which infers occlusionfree object representation. For instance, some trends exploit symmetry [START_REF] De Charette | 3D reconstruction of deformable revolving object under heavy hand interaction[END_REF]108,134,143,148] and are reviewed in [START_REF] Mitra | Symmetry in 3D geometry: Extraction and applications[END_REF]. Another common approach is to rely on prior 3D models to best fit sparse input [START_REF] Li | Shape completion from a single RGBD image[END_REF][START_REF] Li | Database-assisted object retrieval for real-time 3D reconstruction[END_REF]107,123,131]. In recent years, model-based techniques and new large-scale datasets enlarged the scope of action by enabling inference of complete occluded parts in both scanned objects [START_REF] Dai | Shape completion using 3Dencoder-predictor CNNs and shape synthesis[END_REF][START_REF] Han | Highresolution shape completion using deep neural networks for global structure and local geometry inference[END_REF]106,118,130,135,141,150,170,172] and entire scenes [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF][START_REF] Firman | Structured prediction of unobserved voxels from a single depth image[END_REF]181]. Moreover, contemporary research shows promising results on challenging multi-object reconstruction from as single RGB image [START_REF] Engelmann | From points to multi-object 3D reconstruction[END_REF]. For further analysis, we refer readers to related surveys [START_REF] Han | Image-based 3D object reconstruction: State-of-the-art and trends in the deep learning era[END_REF][START_REF] Mitra | Symmetry in 3D geometry: Extraction and applications[END_REF]170]. [START_REF] Boulch | SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks[END_REF][START_REF] Boulch | Unstructured point cloud semantic labeling using deep segmentation networks[END_REF]142] and were quickly replaced by the use of 3D CNNs which operate on voxels [START_REF] Dai | 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation[END_REF][START_REF] Maturana | VoxNet: A 3D convolutional neural network for real-time object recognition[END_REF][START_REF] Meng | VV-Net: Voxel VAE net with group convolutions for point cloud segmentation[END_REF]120,145,153], tough suffering from memory and computation shortcomings. Point-based networks [START_REF] Landrieu | Large-scale point cloud semantic segmentation with superpoint graphs[END_REF][START_REF] Li | PointCNN: convolution on X-transformed points[END_REF]115,116,147,158] remedied this problem by operating on points and quickly became popular for segmentation, though generative tasks are still a challenge. Therefore, if the point space is predefined, SSC can be seen as a point segmentation task [180]. We refer readers to dedicated surveys on semantic segmentation [START_REF] Gao | Are we hungry for 3D LiDAR data for semantic segmentation? a survey of datasets and methods[END_REF]167].

Semantic Instance Completion. Different from SSC, semantic instance completion is applied only at the objects instance level and cannot complete scene background like walls, floor, etc. Most existing methods require instance detection and complete the latter with CAD models [START_REF] Avetisyan | Scan2CAD: Learning CAD model alignment in RGB-D scans[END_REF][START_REF] Nan | A search-classify approach for cluttered indoor scene understanding[END_REF]129] or a completion head [START_REF] Hou | RevealNet: Seeing behind objects RGB-D scans[END_REF][START_REF] Hou | RevealNet: Seeing behind objects RGB-D scans[END_REF][START_REF] Müller | Seeing behind objects for 3D multi-object tracking in RGB-D sequences[END_REF]104,172]. Notably, [START_REF] Avetisyan | SceneCAD: Predicting object alignments and layouts in RGB-D scans[END_REF] detects both objects and the scene layout to output a complete lightweight CAD representation. These approaches are commonly based on 3D object detection [114,132,136] or instance segmentation [START_REF] Hou | 3D-SIS: 3D semantic instance segmentation of RGB-D scans[END_REF]. Furthermore, some recent methods tackle multiple object completion from single images either through individual object detection and reconstruction [START_REF] Gkioxari | Mesh R-CNN[END_REF][START_REF] Izadinia | IM2CAD[END_REF][START_REF] Kundu | 3D-RCNN: Instance-level 3D object reconstruction via render-and-compare[END_REF]172] or joint completion by understanding the scene structure [103,113]. Song et al. [137] were the first to address semantic segmentation and scene completion jointly, showing that both tasks can mutually benefit. Since then, many SSC works gather ideas from the above described lines of work and are extensively reviewed in Sec. 4.

Datasets and representations

This section presents existing SSC datasets (Sec. 3.1) and commonly used 3D representations for SSC (Sec. 3.2).

Datasets

A comprehensive list of all SSC ready datasets is shown in Tab. 1. We denote as SSC ready any dataset containing pairs of sparse/dense data with semantics label. Note that while 14 datasets meet these criteria, only half has been used for SSC among which the four most popular are bold in the table and previewed in Fig. 1. Overall, there is a prevalence of indoor stationary datasets [START_REF] Armeni | Joint 2D-3D-semantic data for indoor scene understanding[END_REF][START_REF] Chang | Matterport3D: Learning from RGB-D data in indoor environments[END_REF][START_REF] Dai | ScanNet: Richly-annotated 3D reconstructions of indoor scenes[END_REF][START_REF] Handa | SceneNet: Understanding real world indoor scenes with synthetic data[END_REF][START_REF] Hua | SceneNN: A scene meshes dataset with annotations[END_REF]133,137,164] as opposed to outdoors [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF][START_REF] Griffiths | Synthcity: A large scale synthetic point cloud[END_REF]105].

Datasets creation. Synthetic datasets can easily be obtained by sampling 3D object meshes [START_REF] Fu | 3D-FRONT: 3D furnished rooms with layouts and semantics[END_REF][START_REF] Handa | SceneNet: Understanding real world indoor scenes with synthetic data[END_REF]137] or simulating sensors in virtual environments [START_REF] Dosovitskiy | CARLA: An open urban driving simulator[END_REF][START_REF] Gaidon | VirtualWorlds as proxy for multi-object tracking analysis[END_REF][START_REF] Griffiths | Synthcity: A large scale synthetic point cloud[END_REF]126]. Their evident advantage is the virtually free labeling of data, though transferability of synthetically learned features is arguable. Real datasets are quite costly to record and annotate, and require a significant processing effort. Indoor datasets [START_REF] Armeni | Joint 2D-3D-semantic data for indoor scene understanding[END_REF][START_REF] Chang | Matterport3D: Learning from RGB-D data in indoor environments[END_REF][START_REF] Dai | ScanNet: Richly-annotated 3D reconstructions of indoor scenes[END_REF][START_REF] Hua | SceneNN: A scene meshes dataset with annotations[END_REF]133,164,165] are commonly captured with RGB-D or stereo sensors. Conversely, outdoor datasets [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF][START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF]105] are often recorded with LiDAR and optional camera. They are dominated by autonomous driving applications, recorded in (peri)-urban environment, and must subsequently account for dynamic agents (e.g. moving objects, ego-motion, etc.). Fig. 3: Semantics distribution. Class-wise frequencies of the most popular real datasets prove to be highly imbalanced.

Ground truth generation. An evident challenge is the dense annotation of such datasets. Specifically, while indoor stationary scenes can be entirely captured from multiple views or rotating apparatus, 3D outdoor dynamic scenes are virtually impossible to capture entirely as it would require ubiquitous scene sensing. Subsequently, ground truth y is obtained from the aggregation and labeling of sparse sequential data {y 0 , y 1 , ..., y T } over a small time window T . Multi-frame registration is usually leveraged to that matters, assuming that consecutive frames have an overlapping field of view. For RGB-D datasets, mostly stationary and indoors, it is commonly achieved from Structure from Motion (SfM) [START_REF] Nair | High accuracy TOF and stereo sensor fusion at interactive rates[END_REF]128] or visual SLAM [128] (vSLAM), which cause holes, missing data, and noisy annotations [START_REF] Fuentes-Pacheco | Visual simultaneous localization and mapping: a survey[END_REF]. Such imperfections are commonly reduced by inferring dense complete geometry of objects with local matching of CAD models [START_REF] Firman | Structured prediction of unobserved voxels from a single depth image[END_REF][START_REF] Guo | Support surface prediction in indoor scenes[END_REF]164], or post-processing hole filling techniques [140]. In outdoor settings, point cloud registration techniques [111,112] enable the co-location of multiple LiDAR measurements into a single reference coordinate system [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF]. Interestingly, while frequently referred to as dense, the ground truth scenes are often noisy and non-continuous in real datasets, being in fact an approximation of the real scene. Firstly, regardless of the number of frames used, some portions of the scene remain occluded, see Fig. 4d, especially in dynamic environments. Secondly, sensors accuracy and density tend to steadily decrease with the distance, as in Fig. 4b. Thirdly, rigid registration can only cope with viewpoint changes, which leads to dynamic objects (e.g. moving cars) producing traces, which impact on the learning priors is still being discussed [121,124,169], see Fig. 4c. Finally, another limitation lies in the sensors, which only sense the geometrical surfaces and not the volumes, turning all solid objects into shells. To produce semantics labels, the common practice is to observe the aggregated 3D data from multiple virtual viewpoints to minimize the labeling ambiguity. This process is tedious and (c) Object motion [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] (d) Occlusions [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] Fig. 4: Inaccuracies of SSC ground truth. Providing semantics and geometrics annotation in real-world datasets proves to be complex, and the resulting process is imperfect -which lead to noisy supervision signal. Ground truth mislabeling is a well known bias (a). Sparsity is common in LiDAR-based datasets (b). Object motion causes temporal traces (c), and occlusions are inevitable in dynamic scenes (d).

prone to errors 1 , visible in Fig. 4a. Ultimately, semantics distribution is highly imbalanced as shown in the two most used indoor/outdoor datasets Fig. 3.

Indoor datasets. From Tab. 1, NYUv2 [133] (aka NYU-Kinect) is the most popular indoor real-world dataset, composed of mainly office and house room scenery. Despite complete 3D ground truth scene volumes not being originally provided, they have been generated in [START_REF] Guo | Support surface prediction in indoor scenes[END_REF] by using 3D models and 3D boxes or planes for producing complete synthetically augmented meshes of the scene, generating 1449 pairs of RGB-D images and 3D semantically annotated volumes. Extension of the mesh volumes to 3D grids has been done in [137]. However, mesh annotations are not always perfectly aligned with the original depth images. To solve this, many methods also report results by using depth maps rendered from the annotations directly as in [START_REF] Firman | Structured prediction of unobserved voxels from a single depth image[END_REF]. This variant is commonly named as NYUCAD and provides simplified data pairs at the expense of geometric detail loss. Additional indoor real-world datasets as Matterport3D [START_REF] Chang | Matterport3D: Learning from RGB-D data in indoor environments[END_REF],

SceneNN [START_REF] Hua | SceneNN: A scene meshes dataset with annotations[END_REF] and ScanNet [START_REF] Dai | ScanNet: Richly-annotated 3D reconstructions of indoor scenes[END_REF] can be used for completing entire rooms from incomplete 3D meshes [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF]. The latter has also been synthetically augmented from 3D object models and referred to as CompleteScanNet [164], providing cleaner annotations. Additionally, smaller SUN3D [165] provides RGB-D images along with registered semantic point clouds. Note that datasets providing 3D meshes or point clouds easily be voxelized as detailed in [137]. Additionally, Stanford 2D-3D-S [START_REF] Armeni | Joint 2D-3D-semantic data for indoor scene understanding[END_REF] provides 360 • RGB-D images, of interest for completing entire rooms [START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF]. Due to real datasets small sizes, low scene variability, and annotation ambiguities, synthetic SUNCG [137] (aka SUNCG-D) was proposed, being a large scale dataset with pairs of depth images and complete synthetic scene meshes. An extension containing RGB modality was presented in [START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF] and known as SUNCG-RGBD. Despite its popularity, the dataset is no longer available due to copyright infringement2 , evidencing a lack of synthetic indoor datasets for SSC, that could be addressed using SceneNet [START_REF] Handa | SceneNet: Understanding real world indoor scenes with synthetic data[END_REF].

Outdoor datasets. SemanticKITTI [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] is the most popular large-scale dataset and currently the sole providing single sparse and dense semantically annotated point cloud pairs from real-world scenes. It derivates from the popular odometry benchmark of the KITTI dataset [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF], which provides careful registration and untwisted point clouds considering vehicle's ego-motion. Furthermore, voxelized dense scenes were later released as part of an evaluation benchmark with a hidden test set 3 . The dataset presents big challenges given the high scene variability and the high class imbalance naturally present in outdoor scenes (Fig. 3b). SemanticPOSS [105] also provides single sparse semantic point clouds and sensor poses in same format as the latter to ease its implementation. Furthermore, synthetic SynthCity additionally provides dense semantic point clouds and sensor poses. It has the advantage of excluding dynamic objects, which solves the effect of object motion (cf. Fig. 4c), but not occlusions (cf. Fig. 4d).

Scene representations

We now detail the common 3D representation for SSC output, shown in Fig. Occupancy grid encodes scene geometry as a 3D grid, in which cells describe semantic occupancy of the space. Opposed to point clouds, grids conveniently define neighborhood with adjacent cells, and thus enable easy application of 3D CNNs, which facilitates to extend deep learning architectures designed for 2D data into 3D [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF][START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF][START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF]124,137,174,177]. However, the representation suffers from constraining limitations and efficiency drawbacks since it represents both occupied and free regions of the scene, leading to high memory and computation needs. Voxels are also commonly used as a support for implicit surface definition which we now describe.

Implicit surface encodes geometry as a gradient field expressing the signed distance to the closest surface, known as the Signed Distance Function (SDF). While fields are continuous by nature, for ease of implementation they are commonly encoded in a discrete manner. The value of the gradient field is estimated at specific locations, typically at the voxel centers [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF], or at the point locations for point clouds [121]. Implicit surface may also be used as input [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF][START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF]137,160,174,177] to reduce the sparsity of the input data, at the expense of greedy computation. For numerical reason, most works encode in fact a flipped version (cf. f-TSDF, sec. 4.1). Meshes, explained in detail below, can be obtained from the implicit surface, by using meshification algorithms such as marching cubes [90].

Mesh enables an explicit surface representation of the scene by a set of polygons. Implementing deep-learning algorithms directly on 3D meshes is challenging and most works obtain the mesh from intermediate implicit voxel-based TSDF representations [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF] by minimizing distance values within voxels and applying meshification algorithms [90]. Other alternatives contemplate applying view inpainting as in [START_REF] Han | Deep reinforcement learning of volumeguided progressive view inpainting for 3D point scene completion from a single depth image[END_REF] or using parametric surface elements [START_REF] Groueix | AtlasNet: A papier-mâché approach to learning 3D surface generation[END_REF], which are more oriented to object/shape completion. Furthermore, recent learning-based algorithms such as Deep Marching Cubes [START_REF] Liao | Deep marching cubes: Learning explicit surface representations[END_REF] enable to obtain continuous meshes end-to-end from well sampled point clouds, but similarly have not been applied to fill large areas of missing information or scenes.

Semantic Scene Completion

The seminal work of Song et al.

[137] first addressed Semantic Scene Completion (SSC) with the observation that semantics and geometry are 'tightly intertwined'. While there have been great progress lately, the best methods still perform little below 30% mIoU on the most challenging SemanticKITTI benchmark [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF], advocating that there is a significant margin for improvement.

Inferring a dense 3D scene from 2D or sparse 3D inputs is in fact an ill-posed problem since the input data are not sufficient to resolve all ambiguities. As such, apart from [START_REF] Firman | Structured prediction of unobserved voxels from a single depth image[END_REF][START_REF] Geiger | Joint 3D object and layout inference from a single RGB-D image[END_REF][START_REF] Lin | Holistic scene understanding for 3D object detection with RGBD cameras[END_REF]179], all existing works rely on deep learning to learn semantics and geometric priors from large scale datasets reviewed in Sec. 3.1.

In the following, we provide a comprehensive survey of the SSC literature. Unlike some historical computer vision tasks, for SSC we found little consensus and a wide variety of choices exists. As such we also focus on the remaining avenues of research to foster future works.

The survey is organized in sections that follow a standard SSC pipeline, with each section analyzing the different line of choices. We start in Sec. 4.1 by reviewing the various input encoding strategies, broadly categorized into 2D/3D, and discuss their influence on the global problem framing. Following this, we study SSC deep architectures in Sec. 4.2. While initially, the task was addressed with vanilla 3D CNNs, other architectures have been leveraged such as 2D/3D CNNs, point-based networks, or various hybrid proposals. Sec. 4.3 presents important design choices, such as contextual aggregation which greatly influences any geometrical task like SSC, or lightweight designs to leverage the burden of 3D networks spanning from compact 3D representations to custom convolutions. We discuss the training strategies in Sec. 4.4, along with the losses and their benefits. Finally, a grouped evaluation of the metrics, methods performance and network efficiency is in Sec. 4.5.

We provide the reader with a digest overview of the field, chronologically listing methods in Tab. 2 -where columns follow the paper structure. Because SSC definition may overlap with some reconstruction methods that also address semantics, we draw the inclusion line in that SSC must also complete semantics and geometry of unseen regions. We indistinctively report as SSC any method meeting these criteria. Looking at Tab. 2, it illustrates both the growing interest in this task and the current lack of consensus on input encoding, architectures, etc.

Input Encoding

Given the 3D nature of the task, there is an evident benefit of using 3D inputs as it already withholds geometrical insights.

As such, it is easier to leverage sparse 3D surface as input in the form of occupancy grid, distance fields, or point clouds.

Another line of work uses RGB data in conjunction with depth data since they are spatially aligned and easily handled by 2D CNNs.

3D grid-based. In most works, 3D occupancy grid (aka Voxel Occupancy) is used [START_REF] Garbade | Two stream 3D semantic scene completion[END_REF]124,164,169], encoding each cell as either free or occupied. Such representation is conveniently encoded as binary grids, easily compressed (cf. Sec. 4.3.4), but provides little input signal for the network.

An alternative richer encoding is the use of TSDF (Fig. 6c), where the signed distance d to the closest surface is computed at given 3D locations (usually, voxel centers), as in [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF][START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF]. Instead of only providing input signal at the measurement locations like occupancy grids or point cloud, TSDF provides a richer supervision signal for the network. The sign for example provides guidance on which part of the scene is occluded in the input. The greedy 3D computation can be the field yet lacks thorough study on the benefit of TSDF encoding. While f-TSDF is still commonly used in recent SSC [START_REF] Chen | Am2fnet: Attention-based multiscale & multi-modality fused network[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF]174,176,177] or SC [START_REF] Denninger | 3D scene reconstruction from a single viewport[END_REF], other approaches stick with original TSDF [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF]160,176]. Furthermore, the benefit of f-TSDF over TSDF is questioned in [START_REF] Garbade | Two stream 3D semantic scene completion[END_REF]176] [START_REF] Gupta | Learning rich features from RGB-D images for object detection and segmentation[END_REF], which keeps more effective information when compared to the single channel depth encoding [START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF]. Additional non-geometrical modalities such as RGB or LiDAR refraction intensity provide auxiliary signals specifically useful to infer semantic labels [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF][START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF]180]. In practice, some works have shown that good enough semantic labels can be inferred directly from depth/range images [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF][START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF]169] to guide SSC. Interestingly, the vast majority of the literature only accounts for surface information while ignoring any free space data provided by the sensors (see Fig. 2). While free space labels might be noisy, we believe it provides an additional signal for the network which was found beneficial in [121,164]. Conversely, Roldao et al. [124] relate that encoding unknown information brought little improvement.

Architecture choices

Directly linked to the choice of input encoding, we broadly categorize architectural choices into 4 groups. In detail: Volume networks leveraging 3D CNNs to convolve volumetric grid representations, Point-based networks which compute features on points locations, View-Volume networks that learn the 2D-3D mapping of data with 2D and 3D CNNs, and Hybrid networks that use various networks to combine modalities of different dimension. All architectures output N × C data (N the spatial dimensions and C the number of semantic classes) where the last dimension is the probability of either semantic class at the given location. In most works, the final prediction is the softmax output with the class probabilities. We refer to Fig. 7 for a visual illustration of either architecture type.

Volume networks. As they are convenient for processing grid data 3D CNNs (Fig. 7b) are the most popular for SSC [START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF][START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF][START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF][START_REF] Guedes | Semantic scene completion combining colour and depth: preliminary experiments[END_REF]137,152,157,160,164,169,174,176,177]. Since completion heavily requires contextual information it is common practice to use a U-Net architecture [125] (see Fig. 7), i.e. encoder-decoder with skip connections. The benefit of the latter is not only to provide contextual information [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF], pointaugmented: [180], mix-2D-3D: [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF][START_REF] Cai | Semantic scene completion via integrating instances and scene in-the-loop[END_REF][START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF] Fig. 7: Architectures for SSC. For compactness, we do not display all connections but rather focus on the global architectures and information exchanges between the different networks. F stands for any type of fusion. but also to enable meaningful coarser scene representation, used in SSC for outputting multi-scale predictions [124,174] or for enabling coarse-to-fine refinement [START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF].

There are two important limitations of 3D CNNs: their cubically growing memory need, and the dilation of the sparse input manifold due to convolutions. To circumvent both, one can use sparse 3D networks like SparseConvNet [START_REF] Graham | 3D semantic segmentation with submanifold sparse convolutional networks[END_REF]174] or Minkowski Network [START_REF] Choy | 4D spatio-temporal ConvNets: Minkowski convolutional neural networks[END_REF] which conveniently operates only on input geometry, thus allowing high grid resolution where each cell contains typically a single point. While they were found highly beneficial for most 3D tasks, they show limited interest for SSC since the output is expected to be denser than the network input. Considering the output to be sparse, Dai et al. [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF] used a sparse encoder and a custom sparse generative decoder to restrict the manifold dilation, applied for SC rather than SSC. This is beneficial but cannot cope with large chunks of missing data. An alternative is [169] that first performs pointwise semantic labeling using a sparse network. To enable a dense SSC output in [174], authors merge the output of multiple shared SparseConvNets applied on sub-sampled non-overlapping sparse grids. Both [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF]174] are a clever use of sparse convolutions but somehow limit the memory and computational benefit of the latter. In [START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF], variational optimization is used to regularize the model and avoid the need for greedy high-capacity 3D CNN.

View-volume networks. To take advantage of 2D CNNs efficiency, a common strategy is to use them in conjunction with 3D CNNs as in [START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF][START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF]124], see Fig. 7a. Two different schemes have been identified from the literature. The most common, as in [START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF][START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF], employs a 2D CNN encoder to extract 2-dimensional features from 2D texture/geometry inputs (RGB, depth, etc.), which are then lifted to 3D and processed by 3D CNN (Fig. 7a, 3D-backbone). Optional modalities may be added with additional branches and mid-fusion scheme [START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF][START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF]. In brief, the use of sequential 2D-3D CNNs conveniently benefits of different neighboring definitions, since 2D neighboring pixels might be far away in 3D and vice versa, thus providing rich feature representation, at the cost of increased processing. To address this limitation, the second scheme (Fig. 7a, 2D-backbone) projects 3D input data into 2D, then processed with normal 2D CNN [124] significantly lighter than its 3D counterpart. The resulting 2D features are then lifted back to 3D and decoded with 3D convolutions, retrieving the third dimension before the final prediction. This latter scheme is irrefutably lighter in computation and memory footprint but better suited for outdoor scenes (cf. Sec. 4.5), as the data exhibits main variance along two axes (i.e. longitudinal and lateral). Hybrid networks. Several other works combine architectures already mentioned, which we refer to as hybrid networks [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF][START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF]121,122,180], see Fig. 7d. A common 2D-3D scheme combines 2D and 3D features (e.g. RGB and f-TSDF) through expert modality networks in a common latent space decoded via a 3D CNN [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF] (Fig. 7d, mix-2D-3D). This enables additional benefit with the combined use of texture and geometrical features. Similarly, IPF-SPCNet [180] performs semantic segmentation from RGB image on an initial 2D CNN and lifts image labels to augment ad-hoc 3D points (Fig. 7d, point-augmented). In [121, 122], Point-Net [116] encodes geometrical features from sub-set of points later convolved in a bird eye view (BEV) projection with 2D CNN in a hybrid architecture manner (Fig. 7d, point-2D). Another strategy employs parallel 2D-3D branches to process the same data with different neighborhoods definition contained in the 2D projected image and 3D grid as in [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF]. Recently, S3CNet [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF] combines 3D voxelized f-TSDF and normal features with a 2D BEV [START_REF] Chen | Multi-view 3D object detection network for autonomous driving[END_REF], passing both modalities through sparse encoder-decoder networks for late fusion (Fig. 7d, parallel-2D-3D), achieving impressive results in outdoor scenes. A similar architecture is proposed by [START_REF] Abbasi | Deep 3D semantic scene extrapolation[END_REF] to perform what they refer to as scene extrapolation [138], by performing extrapolation of a half-known scene into a complete one.

Design choices

The significant sparsity difference between the input data and the expected dense output imposes special design choices to be made, specifically to ensure efficient flow of information. In the following, we elaborate on the most important ones such as contextual awareness (Sec. 4.3.1), position awareness (Sec. 4.3.2), and fusion strategies (Sec. 4.3.3). Finally, we detail lightweight designs (Sec. 4.3.4) to efficiently process 3D large extent of sparse data, and the common refinement processes (Sec. 4.3.5).

Contextual awareness

To correctly complete the missing information in the scene, it is necessary to gather contextual information from multiple scales, which enables to disambiguate between similar objects present in the scene. This makes it possible to capture both local geometric details and high-level contextual information [137]. A common strategy to accomplish this is to add skip connections between different convolutional layers to aggregate features from different receptive fields [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF][START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF]124,137,174,177]. Additionally, serial context aggregation with multi-scale feature fusion can be used as proposed in [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF], Fig. 8a. In [137], the use of dilated convolutions (aka 'atrous') [171] are proposed to increase receptive fields and gather context information at Such convolutions are only suitable for dense networks (as opposed to sparse networks), and even then should only be applied in deeper layers of the network after dilation of the input manifold. In [START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF], a feature aggregation module is introduced by using Atrous Spatial Pyramid Pooling blocks (ASPP) [START_REF] Chen | DeepLab: Semantic image segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs[END_REF], which exploits multi-scale features by employing multiple parallel filters with different dilation rates, Fig. 8c. A lightweight ASPP is presented in [START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF]. Dilated convolutions in the ASPP module can be replaced by Residual dilated blocks [START_REF] He | Deep residual learning for image recognition[END_REF] to increase spatial context and improve gradient flow. A Guided Residual Block (GRB) to amplify fused features and a Global Aggregation module to aggregate global context through 3D global pooling are proposed in [START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF]. An additional feature aggregation strategy is presented in [177], where multi-context aggregation is achieved by a cascade pyramid architecture, Fig. 8b. In [START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF] multi-scale features are aggregated together following a Primal-Dual optimization algorithm [110], which ensures semantically stable predictions and further acts as a regularizer for the learning.

Position awareness

Geometric information contained in voxels at different positions has high variability, i.e. Local Geometric Anisotropy.

In particular, voxels inside an object are homogeneous and likely to belong to the same semantic category as their neighbors. Conversely, voxels at the surface, edges, and vertices of the scene provide richer geometric information due to the higher variance of their surroundings. To deal with this, PALNet [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF] proposes a Position Aware loss (cf. Sec. 4.4.1), which consists of a cross entropy loss with individual voxel weights assigned according to their geometric anisotropy, providing slightly faster convergence and improving results.

Likewise, AM 2 FNet [START_REF] Chen | Am2fnet: Attention-based multiscale & multi-modality fused network[END_REF] supervises contour information by an additional cross entropy loss as a supplementary cue for segmentation.

In the same line of work, EdgeNet [START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF] calculates Canny edges [START_REF] Canny | A computational approach to edge detection[END_REF] then fused with an f-TSDF obtained from the depth image. Hence, it increases the gradient along the geometrical edges of the scene. Additionally, detection of RGB edges enables the identification of objects lacking geometrical saliency. The same network is used in [START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF] to predict complete scenes from panoramic RGB-D images.

Similarly, [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF] introduces an explicit and compact geometric embedding from depth information by predicting a 3D sketch containing scene edges from an input TSDF. They show that this feature embedding is resolution-insensitive, which brings high benefit, even from partial low-resolution observations.

Fusion strategies

SSC requires outputting both geometry and semantics. Though highly coupled -geometry helping semantics and vice-versa -, there is a natural benefit to use inputs of different natures for example to provide additional texture or geometry insights. We found that about two-thirds of the literature use multi-modal inputs though it appears less trendy in most recent works (see Tab. 2 col 'Input'). For the vast majority of multi-input works, RGB is used alongside various geometrical input [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF][START_REF] Chen | Am2fnet: Attention-based multiscale & multi-modality fused network[END_REF][START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Guedes | Semantic scene completion combining colour and depth: preliminary experiments[END_REF][START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF][START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF], 180] as it is a natural candidate for semantics. Even without color, the fusing of 2D and 3D modalities is often employed because it enables richer feature estimation. This is because 2D and 3D neighborhoods differ, since 2D data results of planar projection along the optical axis of the sensor. Subsequently, a common strategy consists of fusing geometrical features processed with different 2D / 3D encoding to obtain richer local scene descriptors. In [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] depth and occupancy are fused while [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF] uses depth along with TSDF-like data. As mentioned earlier (cf. Sec. 4.1), TSDF provides a gradient field easing the network convergence. Finally, applicationoriented fusion is also found such as fusing bird eye view along with geometrical inputs as in [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF] -which is better suited for outdoor SSC.

We group the type of fusion in three categories, shown in Fig. 9. Fusion applied at the input level (Early fusion), at the mid-level features (Middle fusion), or at the late/output level (Late fusion). They are respectively referred to as E, M, and L in column 'Fusion strategies' Tab. 2.

Early fusion. The most trivial approach is to concatenate input modalities [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF][START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Guedes | Semantic scene completion combining colour and depth: preliminary experiments[END_REF][START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF]180] before any further processing, see Fig. 9a. There are two strategies here: when spatially aligned (e.g. RGB/Depth) inputs can be concatenated channel-wise; alternatively, inputs can be projected Fig. 9: Fusion Strategies. To accommodate for multiple input modalities (Mod. 1, Mod. 2), several fusion strategies are found in the literature. Here, F stands for fusion and might be any type of fusion like concat C , sum + , multiply × , convolutions, etc. to a shared 3D common space (aka features lifting). For spatially aligned modalities, it is common to use pairs of normals/depth [START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF] or RGB/semantics [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] and to process them with 2D CNNs. The second strategy lifts any 2D inputs to 3D -assuming depth information and accurate inter-sensors calibration -and processes it with 3D networks. This has been done with RGB/depth [START_REF] Guedes | Semantic scene completion combining colour and depth: preliminary experiments[END_REF], depth/semantics [START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF], points/semantics [180]. Except when using points, this second strategy leads to a sparse tensor since not all 3D cells have features. Noteworthy, [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF][START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF]180] use semantics, which is first estimated either from RGB or depth-like data. A 2D or 3D network processes the concatenated tensor, and while it logically outperforms single-modality input [START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF] there seems to be little benefit to apply early fusion.

Middle fusion.

To exploit all modalities, middle fusion uses expert networks that learn modality-centric features. A straightforward fusion strategy is employed in [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF] where the features are simply concatenated and processed with a U-Net style architecture (cf. Fig. 9b, single-stage), which improves over early fusion but still limits the exchange of information between modalities. The information flow is improved in [START_REF] Chen | Am2fnet: Attention-based multiscale & multi-modality fused network[END_REF][START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF][START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF] by fusing modality-centric features in a multi-stage manner (cf. Fig. 9b, multi-stage); meaning that low-level features from different modalities are fused together and aggregated with fused mid/high level features gathered similarly. While ultimately the number of fusion stages shall be a function of the input/output size, 3 stages are often used [START_REF] Chen | Am2fnet: Attention-based multiscale & multi-modality fused network[END_REF][START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF][START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF], though [START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF] claims 4 stages boost performances with similar input/output. The fused mechanism can be a simple summation [START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF] or concatenation [START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF], but [START_REF] Chen | Am2fnet: Attention-based multiscale & multi-modality fused network[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF] benefit from smarter selective fusion schemes using respectively RefineNet [START_REF] Lin | RefineNet: Multi-path refinement networks for high-resolution semantic segmentation[END_REF] and Gated Recurrent Fusion. Overall, the literature consensus is that middle fusion is highly efficient for SSC. The ablation studies of Liu et al. [START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF] reports that any selective fusion schemes bring at least a 20% performance boost over simple sum/concat/max schemes. [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF], see Fig. 9c. The straightforward strategy in [START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF] is to apply fusion -namely, element-wise multiplication -of two SSC branches (a 3D guidance branch, and a semantic completion branch), followed by a softmax. The benefit still appears little (5 to 10%) given the extra computational effort. Similarly, color and geometry branches are concatenated and shallowly convolved before softmax in [START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF], also providing a small benefit (less than 3%). A unique strategy was proposed in the recent S3CNet [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF] where the output of parallel 2D top-view and 3D SSC are fused together in a semantic-wise manner. While it was only evaluated on outdoor scenes -which setup naturally minimizes vertically overlapping semantic labelsablation reports an overall 20% boost.

Late fusion. Few works use late fusion for SSC

Summarizing the different strategies, Middle fusion appears to be the best general SSC practice, though Late fusion was found beneficial in some specific settings. On fused modalities, RGB/geometry fusion boosts performance but at the cost of an additional sensor need, but even using fusion of geometrical input with different encodings is highly beneficial. An interesting insight from [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF] advocates that RGB or geometry can be fused with edge features as they provide additional boundaries guidance for the SSC network.

Lightweight designs

A few techniques for lightweight designs are often applied for SSC with the aim of addressing two separate problems: how to improve the memory or computation efficiency, and how to design meaningful convolutions to improve the information flow. We detail either problem and its solutions below.

Memory and computation efficiency. Voxel grids are often used as input/output encoding of the 3D data since current datasets provide ground truth in such a format. However, only a tiny portion of the voxels are occupied which makes the naive dense grid inefficient in memory and computation. Memory wise, a few works use compact hierarchical 3D representation inspired from pre-deep learning, like Kd-Tree [START_REF] Bentley | Multidimensional binary search trees used for associative searching[END_REF] and Octree [START_REF] Meagher | Geometric modeling using octree encoding[END_REF]. Octree-based deep networks are often used for learning object reconstruction [119,120,151,153] though little applied on real semantic scene completion problem [START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF]152,157]. Meanwhile, deep Kd-Networks [START_REF] Klokov | Escape from cells: Deep Kd-networks for the recognition of 3D point cloud models[END_REF] proposal seems less appropriate and has not yet been applied to SSC. Computation-wise, [START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF] proposed a custom network architecture with adjustable multi-scale branches in which inference and backpropagation can be run in parallel, subsequently enabling faster training and good performance with low-capacity dense 3D CNNs. Alternatively, few SSC or SC works [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF]174] use sparse networks like SparseConvNet [START_REF] Graham | 3D semantic segmentation with submanifold sparse convolutional networks[END_REF] or Minkowski [START_REF] Choy | 4D spatio-temporal ConvNets: Minkowski convolutional neural networks[END_REF] which operate only in active locations through a hash table. While sparse convolutions are very memory/computation efficient, they are less suitable for completion, since they deliberately avoid filling empty voxels to prevent dilation of the input domain. To remedy this for the SSC task, dense convolutions are still applied in the decoder, which subsequently reduces sparse networks efficiency. Overall, while Kd-/Octree-networks are highly memory efficient, the complexity of their implementation has restricted a wider application. Contrastingly, sparse networks [START_REF] Choy | 4D spatio-temporal ConvNets: Minkowski convolutional neural networks[END_REF][START_REF] Graham | 3D semantic segmentation with submanifold sparse convolutional networks[END_REF] are more used [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF][START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF]169,174].

Efficient convolutions.

A key observation is the spatial redundancy of data since neighboring voxels contain similar information. To exploit such redundancy, [174] proposes Spatial Group Convolutions (SGC) to divide input volume into different sparse tensors along the spatial dimensions which are then convolved with shared sparse networks. A similar strategy is followed by [START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF], dividing the volumetric space into a set of eight interleaved voxel groups and performing an auto-regressive prediction [117]. Dilated convolutions are also widely used for semantic completion methods [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF][START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF]124,137,160,177], since they increase receptive fields at small cost, providing large context, which is crucial for scene understanding as discussed in Sec. 4.3.1. Dilated convolutions with separated kernels are proposed in [177] by separating the input tensor into subvolumes. This enables to reduce the number of parameters and consider depth profiles in which depth values are continuous only in neighboring regions. DDRNet [START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF] also introduces Dimensional Decomposition Residual (DDR) block, decomposing 3D convolutions into three consecutive layers along each dimension, subsequently reducing the network parameters. In [START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF], this concept is extended with the use of anisotropic convolutions, where the kernel size of each 1D convolution is adaptively learned during training to model the dimensional anisotropy.

Refinement

Refinement is commonly used in many vision tasks, but little applied in SSC. VD-CRF [176] extends SSCNet [137] by applying Conditional Random Field (CRF) to refine output consistency, achieving little over 4% gain. Additionally, S3CNet [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF] presents a 3D spatial propagation network [START_REF] Liu | Learning affinity via spatial propagation networks[END_REF] to refine segmentation results after fusion of 2D semantically completed bird eye view image and 3D grid. Additional partial refinement is applied in [START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF]164] to fuse SSC predictions from different viewpoints, by either softmax applied to overlapping partitions [START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF]164] or an occupancy-based fusion policy [164]. Though few works address the refinement problem, some notable performance boosts are found in the literature, thus being an encouraging topic to explore.

Training

We now detail the SSC training process, starting with the SSC losses (Sec. 4.4.1), and subsequently the implemented training strategies (Sec. 4.4.2).

Losses

We classify the SSC losses found in the literature in 3 broad categories: geometric losses which optimize geometrical accuracy, semantics losses which optimize semantics prediction, and consistency losses which guide the overall completion consistency. Note that other non-SSC losses are often added and that the type of SSC losses are commonly mixed -specifically, geometric+semantics [START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF][START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF]152] or all three types [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF]121,122,160]. We refer to Tab. 2 for a quick overview of the losses used by each method. In this section, we also refer to ŷ as SSC prediction and y as ground truth, though for clarity we add a subscript notation to distinguish between SSC encoding. For example, y mesh corresponds to the ground truth mesh.

Geometric losses. These losses penalize the geometrical distance of the output ŷ to ground truth y, in a self-unsupervised manner.

On occupancy grids outputs (ŷ occ ), Binary Cross-Entropy loss (BCE) is most often used [START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF][START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF]121,122,152] to discriminate free voxels from occupied. Assuming a binary class mapping where all non-free semantic classes map to 'occupy'. It writes:

L BCE = - 1 N N i=0 ŷocc i log(y occ i )-(1-ŷ occ i )log(1-y occ i ) , (1)
with N the number of voxels. The drawback of such loss is that it provides little guidance to the network due to its sparsity. Smoother guidance can be provided by outputting an implicit surface (ŷ SDF ) through minimization of the predicted signed distance values in ŷSDF and corresponding SDFencoded mesh (y SDF ) -using 1 or 2 norms.

On points outputs (ŷ pts ), if SSC is approached as a generative task, the above losses could also be used to penalize distance to a ground truth mesh, though it might be more suitable to apply points-to-points distances, thus assuming a ground truth point cloud (y pts ). To that end, permutation invariant metrics as the Chamfer Distance (CD) [START_REF] Fan | A point set generation network for 3D object reconstruction from a single image[END_REF] or the Earth Mover's Distance (EMD) [START_REF] Fan | A point set generation network for 3D object reconstruction from a single image[END_REF][START_REF] Kurenkov | DeformNet: Free-form deformation network for 3D shape reconstruction from a single image[END_REF] have been employed for object completion tasks [START_REF] Fan | A point set generation network for 3D object reconstruction from a single image[END_REF]172] but have not been explored yet for SSC because of their computational greediness [START_REF] Fan | A point set generation network for 3D object reconstruction from a single image[END_REF]. We highlight that such losses could provide an additional geometric supervision signal when used in conjunction with semantic losses described below.

Semantic losses. Such losses are suitable for occupancy grids or points and can accommodate for either C classes (considering only semantics classes of occupied voxels or points) or C + 1 classes (considering all voxels/points and 'free space' being the additional class). Note that only the second case (C + 1 classes) enforce reconstruction, so the first one (C classes) would require additional geometric losses. Cross-Entropy loss (CE) is the preferred loss for SSC [START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF][START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF][START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF]124,137,174,177], it models classes as independent thus considering the latter to be equidistant in the semantic space. Formally, supposing (y, ŷ) it writes:

L CE = - 1 C N i=0 N c=0 w c ŷi,c log e yi,c C c e y i,c , (2)
assuming here that y is the one-hot-encoding of the classes (i.e. y i,c = 1 if y i label is c and otherwise y i,c = 0). In practice, (y, ŷ) can be either occupancy grids (y occ , ŷocc ) or points (y pts , ŷpts ). A rare practice from [160] is to address classification with BCE (Eq. 1) through the sum of C binary classification problems between each semantic class and the free class. However, such a practice is unusual and arguably beneficial.

Recently, PALNet [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF] proposed the Position Aware loss (PA), a weighted cross-entropy accounting for the local semantics entropy to encourage sharper semantics/geometric gradients in the completion (cf. Sec 4.3.2). The loss writes:

L PA = - 1 N N i=0 C c=0 (λ + αW LGAi )ŷ occ i,c log e y occ i,c C c e y occ i,c , (3)
with λ and α being simple base and weight terms, and W LGAi being the Local Geometric Anisotropy of i that scales accordingly to the semantic entropy in its direct vicinity (i.e. W LGA lowers in locally smooth semantics areas). We refer to [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF] for an in-depth explanation. From the latter, L PA leads to a small performance gain of 1-3%. Noteworthy, this loss could easily accommodate point clouds as well.

Note that geometric or semantics losses can only be computed on known ground truth location, due to the ground truth sparsity. Additionally, because SSC is a highly imbalanced problem (cf. Fig. 3), class-balancing strategy is often used.

Consistency losses. Different from most semantics losses, these losses [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF]121,122] provide a self-supervised semantic signal. In [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF] the completion consistency (CCY) of predictions from multiple partitioned sparse inputs is enforced via a Kullback-Leibler divergence. Differently, [121,122] enforces spatial semantics consistency (SCY) by minimizing the Jenssen-Shannon divergence of semantic inference between a given spatial point and some given support points. This self-supervision signal is available at any position within the scene. However, the strategy for support points is highly application dependent and while suitable for outdoor scenes which have repetitive semantic patterns, we conjecture it might not scale as efficiently to cluttered indoor scenes.

Overall, few self-supervised or even unsupervised strategies exist and we believe that such type of new losses [173] should be encouraged.

Training strategies

The vast majority of SSC works are trained end-to-end for single-scale reconstruction, although others prefer multiscale supervision to output SSC at different resolutions. Few works also employ multi-modal supervision commonly relying on auxiliary 2D semantic segmentation. It is also possible to train n-stage networks with coarse-to-fine strategies, or even train with adversarial learning to enforce realism. Strategies are illustrated in Fig. 10 (with link color indicating the stage) and reviewed below.

End-to-end. Most architectures (Fig. 10a, top left) are trained end-to-end and output a single scale SSC [19, 22, 32, 33, 43, 54, 76-78, 89, 121, 137, 152, 177] -often similar to the input size. Training that way is straightforward and often offers minimal memory footprint. Noteworthy, [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF] -which does geometric completion only -gradually increases sparsity during training to ease the completion of large missing chunks. To guide the training, multi-scale SSC outputs can also be supervised, typically from the early layers of a U-Net decoder. A simple, yet efficient multi-scale strategy [124,174] is to minimize the sum of SSC losses at different resolutions (Fig. 10a, top right), thus also enforcing coarse SSC representations in the network. In [174], two different scales are predicted, versus four in [124] providing down to 1:8 (1 over 8) downscaled SSC. In the latter, authors also report that the decoder can be ablated to provide very fast inference at coarsest resolution (370FPS at 1:8 scale). When available, some works leverage multi-modal supervision relying on intermediate auxiliary tasks, typically 2D or 3D semantics, later used along original input data to infer the final SSC (Fig. 10a, bottom), as in [START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF]180]. The latter could also be trained in a two-stage manner. In general, end-to-end training is conducted from scratch, though some [START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF]137,177] report pretraining on the synthetic SUNCG dataset.

Coarse-to-fine. ScanComplete [START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF] also follows a multiscale strategy somehow close to [124,174], though training in a coarse-to-fine manner (Fig. 10b). In detail, three sequential training are achieved at increasingly higher resolutions, with each stage network taking as input the ad-hoc sparse input and the previous stage SSC prediction (for stage>1). Interestingly, no one explored a continuous curriculum learning setting, which could yield stabler training and performance improvement. Still, [START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF] (intentionally omitted Fig. 10b) applies a unique coarse-to-fine proposal in a fully end-to-end manner, via parallel backpropagations in all scales. Of simi-lar spirit, [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF] proposes an iteration-based progressive refinement during training for scene completion, but insights of such strategy are not deeply discussed.

Adversarial. Even SSC ground truth has large missing chunks of data, leading to ambiguous supervision. To address this, [START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF]159,160,164] use adversarial training (Fig. 10c), since the discriminator provides an additional supervision signal. This is straightforwardly implemented in [START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF]164], where the discriminator classifies ground truth from generated SSC (aka real/fake). In [159, 160] of same authors, 2 discriminators are used in a somehow similar fashion to discriminate both the SSC output and the latent depth or semantics features to enforce deep shared representation. Additionally, [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF] employs a Conditional Variational Autoencoder (CVAE) to generate completed border sketches to be fed to the main SSC branch. Despite few works on the matter, adversarial appears a logical choice to improve SSC consistency and provide additional self-supervision. Both [159] and [164] report a 10%-15% boost on several datasets.

Finally, on implementation -where mentioned -only [START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF]124,159,160, 180] train with Adam optimizer, [START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF] with a mix of Adam/SGD, and all others use only SGD with momentum 0.9 and 10 -4 weight decay, except for [START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF][START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF]151,152, 177] using 5 × 10 -4 . The training most often uses standard learning rate scheduler [19, 43, 76-78, 89, 151, 152, 177, 180] though sophisticated scheduling [START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF] or fixed learning rate [START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF] are also used. Because of 3D greediness, the common practice is to train with small batch size of 1 [START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF], 2 [START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF], 3 [START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF], 4 [START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF][START_REF] Cherabier | Learning priors for semantic 3D reconstruction[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF]124,177,180], 8 [151, 152, 159, 160] or 16 [START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF] to fit in standard 12GB GPUs.

Evaluation

We now provide an in-depth evaluation of the field, reviewing first the common metrics (Sec. 4.5.1), the qualitative and quantitative performance of the literature (Sec. 4.5.2), and the networks' efficiency (Sec. 4.5.3).

Metrics

Joint Semantics-Geometry. The preferred metric for SSC is the mean Jaccard Index or mean Intersection over Union (mIoU) [START_REF] Everingham | The Pascal visual object classes challenge: A retrospective[END_REF], which considers IoU of all semantic classes for prediction, without considering free space. It writes

mIoU = 1 C C c=1 TP c TP c + FP c + FN c , (4)
where TP c , FP c and FN c are the true positives, false positives and false negatives predictions for class c, respectively. Since ground truth is commonly semi-dense for real-world datasets, evaluation is performed in known space only.

Geometry only. Because mIoU considers semantic classes, the pure geometrical reconstruction quality is not encompassed. Therefore Intersection over Union (IoU), along with Precision and Recall are commonly used on the binary free/occupy scene representation, obtained by mapping all semantic classes to occupy. Alternatively, any distance metrics from Sec. 4.4.1 (i.e.

1 , 2 , EMD or CD) may be used as in [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF] though less used in real datasets, due to their lower precision when sparsity increases.

On common practice, we highlight that evaluation on real indoor or outdoor datasets is usually performed differently. This results of the common sensors setup, respectively RGB-D (indoor) and LiDAR (outdoor), providing significantly different density information. Referring to Fig. 2, in real indoor [137] the geometrical IoU is evaluated on input occluded regions while the mIoU is evaluated on input occluded (blue) and observed (red) surfaces. In real outdoor [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] the IoU and mIoU are commonly evaluated on the entire known space, regardless of whether regions were observed or occluded in the input. Obviously, synthetic datasets can cope with either practice. In the following, we describe the common practices and report semantics metrics (mIoU) along with geometrical ones (Precision, Recall, IoU).

Performance

We report the available mIoU and IoU performance on the most popular SSC datasets in Tab. 3, which are all obtained from voxelized ground truth. For non-voxel methods the output is voxelized beforehand. Additionally, detailed classwise performance of top five methods for SemanticKITTI, NYUv2 and SUNCG are presented in Tabs. 4, 5 and 6, respectively. From the performance Tab. 3, the mIoU of the best methods plateaus around 75 -85% on synthetic indoor dataset, 52% on real indoor, and 30% on real outdoor. Importantly, note that most indoor datasets performance are evaluated at 1:4 of the original ground truth resolution -that is 60×36×60 -for historical reasons 4 . Additionally, some methods refine parameters on NYUv2 and NYUCAD after SUNCG pre-training and are shown separately in Tab. 7. This makes indoor / outdoor performance comparison tricky. It is interesting to note that IoU -geometrical completion (i.e. ignoring semantics)is way higher than best mIoU. In detail, best IoU are 78% on real indoor, and 57% on real outdoor. Qualitative results of a dozen of methods are shown in Fig. 11 for indoor datasets, and Fig. 12 for outdoor datasets.

Overall, one may note the synthetic to real best performance gap of indoor datasets, which is approx. 10 -35% mIoU and 10 -18% IoU. While a difference is expected, once again it highlights that geometry has a smaller synthetic/real domain gap compared to semantics. On a general note also, most methods perform significantly better on IoU than on mIoU, demonstrating the complexity of the semantics scene completion. In fact, the ranking of methods differs depending on the metric. For example, on NYUv2 (indoor) the recently presented SISNet has best mIoU (52.4%) and IoU (78.2%) by a large margin thanks to their semantic instance completion block and iterative SSC architecture. By reliyng solely on SSC, CCPNet [177] gets best indoor mIoU (41.3%), although achieved through SUNCG pre-training. Conversely, 3DSketch [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF] achieves similar mIoU (41.1%) by training solely on NYUv2 and Chen et al. [START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF] is second overall on IoU (73.4%) with the same setup. On Se-manticKITTI (outdoor) S3CNet [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF] has best mIoU (29.5%) and Local-DIFs [121] best IoU (57.7%). Note also the large difference between best indoor/outdoor metrics. While only a handful of methods [START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF]124,137,174] are evaluated in both setups, they indeed perform significantly worse on outdoor data -though indoor/outdoor performance should be carefully compared given the different resolution. This is partially explained by the higher sparsity in outdoor datasets, visible in 'input' of Fig. 12. Another explanation is the higher number of classes in SemanticKITTI versus NYU and the extreme class-imbalance setup given that minor classes are very rarely observed, see Fig. 3. On general qualitative results, either indoor (Fig. 11) or outdoor (Fig. 12) results show that predictions are accurate in large homogeneous areas (walls/ground, floor, buildings) and most errors occur at object boundaries. This is evident in Tab. [START_REF] Avetisyan | Scan2CAD: Learning CAD model alignment in RGB-D scans[END_REF], where most methods achieve high performance in the largest classes of SemanticKITTI, but struggle with predictions in less represented ones (e.g. bicycle, motorcycle, person). Worth mentioning, S3CNet [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF] achieves considerably larger scores in rare classes (+25%, +37%, +38% respectively), more than twice when compared to next best classed scores. The reason for such behavior is regrettably not deeply explored in their work.

Inputs. To ease interpretation, col 'Input' in Tab. 3 shows the nature of input used, where 'G' is Geometry of any type (depth, TSDF, points, etc.) -possibly several -and 'T' is Texture (RGB). From Tab. 3 using both geometry and texture (G+T) performs among the best indoor, such as SISNet [START_REF] Cai | Semantic scene completion via integrating instances and scene in-the-loop[END_REF] using both RGB and TSDF, and 3DSketch [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF] which relies on textural edges and depth. Generally speaking, G+T enables the prediction of non-salient geometric objects (i.e. Table 3: SSC performance on the most popular datasets without pretraining. The relatively low best mIoU scores on the challenging real outdoor SemanticKitti [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] (29.5%) and real indoor NYUv2 [133] (41.1%) shows the complexity of the task. In the 'method' column, we indicate variants with an offset. To better interpret the performance, column 'Input' shows the type of input modality used where 'G' is Geometry (depth, range, points, etc.) and 'T' is Texture (RGB). Note that all indoor datasets commonly report performance for 60 × 36 × 60 grids for historical reasons though 4x bigger input is commonly treated, cf. Sec. 4.5.2. Top 5 methods are highlighted in each column from red to white. paints, windows, doors) as shown in Fig. 11a by the door predicted 3DSketch [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF] and missed by SSCNet [137]. Noteworthy, among the best mIoU methods [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF]177] all use TSDF-encoding as geometrical input. On outdoor datasets, only TS3D [START_REF] Garbade | Two stream 3D semantic scene completion[END_REF] uses texture without significant improvement. More works are required to evaluate the benefit of RGB modality on outdoor data.

Architecture and design choices. One may notice the good performance of hybrid networks [START_REF] Cai | Semantic scene completion via integrating instances and scene in-the-loop[END_REF][START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF]180] (Fig. 7d), which we believe results from richer input signal due to the fusion of multiple modalities. We also argue that multiple neighboring definitions (2D and 3D) provide beneficial complementary signals. For instance, S3CNet combines 2D BEV and 3D f-TSDF for late fusion through postprocessing refinement, achieving the best semantic completion performance on SemanticKITTI [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] by a considerable margin (+5.7% mIoU). Qualitative results of the approach are shown in Fig. 12c,12d. Similarly, JS3CNet [169] ranks second in the same dataset (23.8% mIoU and 56.6% IoU) with point-wise semantic labeling through SparseConvNet architecture [START_REF] Graham | 3D semantic segmentation with submanifold sparse convolutional networks[END_REF] and dense semantic completion using a point-voxel interaction module, enabling to better infer small vehicles as shown in circled areas of Fig. 12e,12f. Analogously, PALNet [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF] middle fuses depth image and f-TSDF features, achieving good performance on NYUv2 (34.1% mIoU and 61.3% mIoU) and NYUCAD (46.6% mIoU and 80.8% mIoU) datasets, such performance can also be attributed to its position-aware loss, to be discussed next.

Contextual awareness (Sec. 4.3.1) seems also to play an important role for the task. This is noticeable with CCP-Net [177] encouraging results given the use of a single geometric input (see Fig. 11d,11e). Note however that in addition to its lightweight design, the output of CCPNet is higher in resolution (240 × 144 × 240) which was proved to boost performance [177]. The performance of SISNet [START_REF] Cai | Semantic scene completion via integrating instances and scene in-the-loop[END_REF] is also remarkable thanks to instance-wise completion at high resolution and iterative SSC. On position awareness (Sec. 4.3.2) it seems to boost intra-class consistency together with inter-class distinction. For example 3DSKetch [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF] and PALNet [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF], both use position awareness and achieve high performances in indoor An interesting observation is the high density of the completion even regarding the ground truth, visible in Figs. 12b, 12g, 12h. This relationship is studied in [START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF], where sparsity is exploited by removing input data to impulse unknown space completion.

Synthetic data pre-training. Pre-training on large SUNCG is a common workaround to improve performance on the smaller NYUv2 and NYUCAD datasets. Comparison between Tabs. 3 and 7 show that the technique always brings performance gains although the gap becomes less important for most recent methods (i.e. +5.8% mIoU for SSCNet vs. +2.8% mIoU for CCPNet). Since SUNCG is no longer legally available, this practice is less common in recent works.

Network efficiency

In Tab. 8, network parameters and floating-point operations (FLOPs) are listed -where possible -with separation of in- [START_REF] Firman | Structured prediction of unobserved voxels from a single depth image[END_REF]133,137] or 256 × 32 × 256 for outdoor dataset [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF].

door and outdoor networks because they have different output resolutions. Notice the extreme variations between networks, which scale from 1:144 in number of parameters and 1:1260 in FLOPs. Chen et al. [START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF] and LMSCNet [124] are by far the lightest networks with the fewest parameters and lower FLOPs, in indoor and outdoor settings respectively. They also account for the lower number of operations, which canthough not necessarily [START_REF] Ma | Shufflenet v2: Practical guidelines for efficient cnn architecture design[END_REF] -contribute to faster inference times. Furthermore, the use of sparse convolutions [START_REF] Graham | 3D semantic segmentation with submanifold sparse convolutional networks[END_REF] is commonly applied as a strategy to reduce memory overhead in [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF][START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF]169,174].

Discussion

Despite growing interest, there are still major challenges to solve SSC as the best methods still perform poorly on real datasets (see Tabs. 5, 4). In this section, we wish to highlight important remaining issues and provide future research directions.

Best practices for SSC. Among the various viable choices for SSC, some were proven highly beneficial. For instance, contextual aggregation (Sec. 4.3.1) improves the information flow. In that sense, fusion of low and high level features with different receptive fields provides a contextual signal for the network, benefiting both the identification of finegrained details and the overall scene understanding. Like for semantic segmentation, features aggregation significantly boosts performance, typically with UNet [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF]124,151,169,174], Cascade Pyramid [177] or Feature Aggregation Modules [START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF]. Further geometrical cues often boost SSC, whether if it is multiple geometrical representations (e.g. depth + voxel, Sec. 4.1) or boundaries (e.g. edges, Sec. 4.3.2). Multi-task and auxiliary tasks can also bring a performance boost. This is commonly done in the form of separated semantic segmentation [169] or instance completion [START_REF] Cai | Semantic scene completion via integrating instances and scene in-the-loop[END_REF] to improve input to SSC. Sketch supervision [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF] -which comes at virtually no cost -was shown particularly helpful to boost indoor performance but might be less beneficial for outdoor scenarios given larger semantic geometrical variance (i.e. vegetation). On networks, sparse convolutions [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF][START_REF] Dai | SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans[END_REF]169] can reduce memory needs and therefore enable higher resolution, although they must be combined with dense convolutions. Similar for dilated convolutions that have been used in the wide majority of SSC works [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF]124,137,160,164,177]. On training, [121] shows that free space supervision close to the geometry can provide sharper inference, and we believe adversarial training (Sec. 4.4.2) is key to cope with the ground truth ambiguities. Another way to have an additional signal is the use of position aware losses [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF] which provides additional spatial supervision and was shown to bring performance improvements in both indoor [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF] and outdoor scenarios [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF]. On evaluation, we encourage authors to evaluate on both indoor and outdoor datasets which exhibit different challenges. Finally, for real-time applications, more works like [START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF]124] should account for lightweight and fast inference architectures (Sec. 4.5.3). Supervision bias. An important challenge for completion results from the big imbalance ratio between free and occupied space (9:1 in both NYUv2 [133,137] and SemanticKITTI [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF]) which biases the networks towards free space predictions. To deal with this problem, random undersampling of the major free class is often applied [137] to reach an acceptable 2:1 ratio. The strategy reportedly improves completion performance (i.e. +4% IoU [137]) and is widely employed [START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF]174,177]. Similarly, the loss can be balanced to favor occupy predictions [START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF]. Again, few works like [121] efficiently benefit from free space information.

Semantic class balancing. Imbalance is also present in the semantic labels, especially in outdoor datasets, where there is a prevalence of road or vegetation (see Fig. 1 and3). Classbalancing can be applied to mitigate imbalanced distribution, usually weighting each class according to the inverse of its frequency [124], though prediction of under-represented classes still suffers (e.g. pedestrian or motorcycle in [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF]). This may have a catastrophic impact on robotics applications. An approach worth mentioning is S3CNet [START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF], where Fig. 11: Performance of indoor Semantic Scene Completion on NYUCAD [START_REF] Firman | Structured prediction of unobserved voxels from a single depth image[END_REF] and SUNCG [137]. Methods with RGB modalities (i.e. 3DSketch) enable detection of color salient objects as the highlighted door in row (a). Position awareness also contributes to better reconstruction consistency and inter-class distinction as seen in rows (b), (c) by PALNet [START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF] and 3DSketch [START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF]. Furthermore, SISNet [START_REF] Cai | Semantic scene completion via integrating instances and scene in-the-loop[END_REF] overcomes all other methods through their scene-instance-scene loop architecture, seen in row (b). Multi-scale aggregation also improves reconstruction performance as seen on rows (d), (e), where CCPNet [177] achieves the best performance on SUNCG [137]. combined weighted cross entropy and position aware loss (cf. Sec. 4.4.1) achieve impressive improvements in underrepresented classes of SemanticKITTI. We believe SSC could benefit from smarter balancing strategies.

Object motion. As mentioned in Sec. 3.1, real-world ground truth is obtained by the rigid registration of contiguous frames. While this corrects for ego-motion, it doesn't account for scene motion and moving objects produce temporal tubes in the ground truth, as visible in SemanticKITTI [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] (Fig. 4c). As such, to maximize performance, the SSC network must additionally predict the motion of any moving objects. To evaluate the influence of such imperfections for SSC, some works reconstruct target scenes by accounting only for a few future scans [121,169]. Results show marginal comple-tion improvement from the application of such a strategy. An alternative proposal [START_REF] Kim | Remove, then revert: Static point cloud map construction using multiresolution range images[END_REF], is to remove dynamic objects from the detection of spatial singularities after frames registration. On the challenging SemanticKITTI [START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF], because there are few insights to classify dynamic objects, all methods tend to predict vehicles as stationary (cf. Fig. 12) -producing appealing results but being punished by dataset metrics. This obviously results from the dataset bias, given the abundance of parked vehicles.

The introduction of larger synthetic datasets [START_REF] Dosovitskiy | CARLA: An open urban driving simulator[END_REF]126] could be an interesting solution to fight ground truth inaccuracies.

Datasets extendable for SSC. Because semantic labeling is the most complex and costly, we denote that a large amount of Sparse input scene subsampling 4 3D-FRONT [START_REF] Fu | 3D-FRONT: 3D furnished rooms with layouts and semantics[END_REF] 2020 Synthetic Indoor → Mesh --Sparse input from virtual RGB-D - † Synthetically augmented.

Table 9: SSC-extendable datasets. To promote research on SSC we highlight that existing 3D semantic datasets could be extended for SSC, at the cost of processing work (cf. col. 'Extension'). While some extensions could be obtained with little processing (e.g. Replica [140], 3D-Front [START_REF] Fu | 3D-FRONT: 3D furnished rooms with layouts and semantics[END_REF]), others are significantly more complex (e.g. nuScenes [START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF]).

existing 3D semantics datasets [START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF][START_REF] Fu | 3D-FRONT: 3D furnished rooms with layouts and semantics[END_REF][START_REF] Hackel | Semantic3D.net: A new large-scale point cloud classification benchmark[END_REF]127,140,144,149] could also be extended to SSC at the cost of some processing effort. A selective list of these SSC-extendable datasets is in Tab. 9 and we believe that their use should be encouraged to serve the interest of research on SSC. Interestingly, most need little processing for SSC (e.g. sparse input generation from 3D meshes or point clouds, virtual sensor configurations) [START_REF] Hackel | Semantic3D.net: A new large-scale point cloud classification benchmark[END_REF]127,140,144,149], though some require more complex processing (e.g. aggregation of sparse inputs [START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF]). We also encourage the use of autonomous driving simulators such as CARLA [START_REF] Dosovitskiy | CARLA: An open urban driving simulator[END_REF], SYNTHIA [126] for synthetic dataset generation, devoid of dynamic objects and subsequent registration problems. More extensive surveys on RGB-D and Lidar datasets are provided in [START_REF] Firman | RGBD datasets: Past, present and future[END_REF][START_REF] Gao | Are we hungry for 3D LiDAR data for semantic segmentation? a survey of datasets and methods[END_REF].

Conclusion

This paper provided a comprehensive survey on contemporary state-of-the-art methods for 3D Semantic Scene Completion. We reviewed, and critically analyzed major aspects of proposed approaches, including important design choices to be considered, and compared their performance in popular SSC datasets. We believe that this survey will support further development in the field, aiming to provide new insights and help inexpert readers to navigate the field.

Fig. 1 :

 1 Fig. 1: Popular datasets for Semantic Scene Completion (SSC). From an incomplete input view, the SSC task consists in the joint estimation of both geometry and semantics of the scene. The figure shows the 4 most popular datasets for SSC, each showing input data and ground truth. The complexity of SSC lies in the completion of unobserved / occluded regions and in the sparse supervision signal (notice that real ground truth is incomplete). (Images source [124, 137]).

Fig. 2 :

 2 Fig. 2: Scene acquisition. A camera (RGB, RGB-D, Depth) senses dense volumes but produces noisy depth measurements (a), while LiDAR -more accurate -is significantly sparser (b). (Inspired from: [137]).

Fig. 5 :

 5 Fig. 5: SSC representations. Several 3D representations coexist in the literature. Its choice has a major impact on the method to use, as well as the memory or the computation needs. (Source: [28, 137, 154, 160])

Fig. 6 :

 6 Fig. 6: TSDF variants. Projective TSDF (b) is fast to obtain but view-dependent. TSDF (c)[START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Chen | Real-time semantic scene completion via feature aggregation and conditioned prediction[END_REF][START_REF] Chen | 3D semantic scene completion from a single depth image using adversarial training[END_REF][START_REF] Dai | ScanComplete: Large-scale scene completion and semantic segmentation for 3D scans[END_REF] 152, 160] solves the view dependency but gradient is stronger at farther areas from the surface, being inadequate for learning-based methods. In contrast, f-TSDF (d)[START_REF] Chen | Am2fnet: Attention-based multiscale & multi-modality fused network[END_REF][START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF]137,174,177] has strongest gradient near the surface. (Source: [137]).

  avoided using projective TSDF (p-TSDF, cf. Fig.6b) which only computes the distance along the sensing path [101], but with the major drawback of being view-dependent. Highlighted bySong et al. [137], another limitation of TSDF or p-TSDF lies in the strong gradients being in the free space area rather than close to the surface where the networks need guidance. This is noticeable in Fig.6b, 6c since the red/blue gradients are afar from the surface. To move strong gradients closer to the surface, they introduced flipped TSDF (f-TSDF, Fig.6d) such that f-TSDF = sign(TSDF)(d max -d) with d max the occlusion boundary, showing improvement in both network guidance and performance[137]. However,

  Point-based networks. To ultimately prevent discretization of the input data, a few recent works [121, 122, 180] employ point-based networks, see Fig.7c. In 2018, [172] first proposed to apply permutation-invariant architecture [116] to object completion with promising results. However, its use for generative SSC is hindered by the limited points generation capacity, the need for fixed size output, and the use of global features extraction. To date, only SPC-Net [180] relies solely on a point-based network -X Conv[START_REF] Li | PointCNN: convolution on X-transformed points[END_REF] -to predict the semantics of observed and unobserved points. The fixed size limitation is circumvented by assuming a regular distribution of unobserved points, addressing the problem as a point segmentation task. Overall, we believe point-based SSC has yet attracted too few works and is a viable avenue of research.

  Fig.8: Multi-scale contextual aggregation. While context is indubitably important for SSC, different strategies are used to aggregate features from various spatial/scale contexts. Color blocks stand for convolutions with different dilation rates. F stands for any type of fusion.

  Middle Fusion, single-stage:[START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF], multistage.:[START_REF] Chen | Am2fnet: Attention-based multiscale & multi-modality fused network[END_REF][START_REF] Li | Anisotropic convolutional networks for 3D semantic scene completion[END_REF][START_REF] Li | RGBD based dimensional decomposition residual network for 3D semantic scene completion[END_REF][START_REF] Liu | 3D gated recurrent fusion for semantic scene completion[END_REF]

  Late Fusion[START_REF] Cheng | S3CNet: A sparse semantic scene completion network for LiDAR point clouds[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF]

FFig. 10 :

 10 Fig. 10: Training strategies. Most SSC architectures are trained end-to-end (a) outputting single or multi-scale SSC. Additionally, multi-modal supervision training commonly lift semantic features calculated on sparse input to a second stage network. Coarse-to-fine (b), similarly to multi-scale relies on multiple size predictions, but trains in a multi-stage coarse to fine manner. Finally, Adversarial training (c) discriminates between ground truth and predicted scenes. F stands for fusion of any type.

Fig. 12 :

 12 Fig. 12: Performance of outdoor Semantic Scene Completion on SemanticKITTI [6]. LMSCNet [124] proposes a lightweight architecture with small performance decrease, rows (a), (b). S3CNet [21] achieves SoA performance by their sparse bird's eye view and 3D f-TSDF feature encoders, rows (c), (d). Two-stage JS3CNet [169] performs point-wise semantic segmentation and semantic scene completion sequentially, enabling better completion as seen in rows (e), (f). Finally, Local-DIFs [121] enables continuous surface prediction, thanks to deep implicit functions, which enable predictions of considerably larger spatial extent, rows (g), (h).

Table 1 :

 1 SSC datasets. We list here datasets readily usable for the SSC task in chronological order. Popular datasets are bold and previewed in Fig.1. Classes show the total number of semantic classes and when it differs, SSC classes in parenthesis.Semantics. Traditional segmentation techniques reviewed in [102] were based on hand-crafted features, statistical rules, and bottom-up procedures, combined with traditional classifiers. The advances in deep learning have reshuffled the cards [167]. Initial 3D deep techniques relied on multiviews processed by 2D CNNs

  Lightweight Design -GrpConv, Group Convolution. DDR, Dimensional Decomposition Residual Block. RAB, Residual Attention Block. MSO, MultiScale Optimization. Losses -Geometric: BCE, Binary Cross Entropy. 1 , L1 norm. Semantic: CE, Cross Entropy. PA, Position Awareness. Consistency: CCY, Completion Consistency. SCY, Spatial Semantics Consistency.

		Input Encoding		Architecture	Design choices				Training		Evaluation	Open source
		(Sec. 4.1)		(Sec. 4.2)		(Sec. 4.3)				(Sec. 4.4)		(Sec. 4.5)
		2D			3D
	RGB	Depth/Range/HHA	Other (seg., normals, etc.)	Occ. Grid	TSDF	Point Cloud	Network type	Contextual Awareness	Position Awareness	Fusion Strategies	Lightweight Design	Refinement	End-to-end	Coarse-to-fine	Multi-scale	Adversarial	Losses	NYU b [38, 133]	SUNCG c [86, 137]	SemanticKITTI [6]	Other [3, 13, 24, 86, 164]	Framework	Weights
	2017 SSCNet [137] a						volume	DC									CE					Caffe
	2018 Guedes et al. [51]						volume	DC		E							CE					-
	ScanComplete [28]						volume				GrpConv						1 CE					TF
	VVNet [54]						view-volume	DC		E							CE					TF
	Cherabier et al. [22]						volume	PDA		E	MSO						CE					-
	VD-CRF [176]						volume	DC									CE					-
	ESSCNet [174]						volume				GrpConv Sparse						CE					PyTorch
	ASSCNet [159]						view-volume Mscale. CAT									CE					TF
	SATNet [86]						view-volume	ASPP		M/L							CE					PyTorch
	2019 DDRNet [77]						view-volume LW-ASPP		M	DDR						CE					PyTorch
	TS3D [43] a						hybrid	DC		E							CE					-
	EdgeNet [32]						volume	DC		M							CE					-
	SSC-GAN [20]						volume	DC								BCE CE					-
	TS3D+RNet [6]						hybrid	DC		E							CE					-
	TS3D+RNet+SATNet [6]						hybrid	DC		E							CE					-
	ForkNet [160]						volume	DC								BCE CE					TF
	CCPNet [177]						volume	CCP DC			GrpConv						CE					-
	AM 2 FNet [16]						hybrid	DC		M						BCE CE					-
	2020 GRFNet [89]						view-volume LW-ASPP DC		M	DDR						CE					-
	Dourado et al. [33]						volume	DC		E							CE					-
	AMFNet [79]						view-volume LW-ASPP		L	RAB						CE					-
	PALNet [78]						hybrid	FAM DC		M							PA					PyTorch
	3DSketch [17]						hybrid	DC		M	DDR					BCE CE CCY					PyTorch
	AIC-Net [76]						view-volume FAM AIC		M	Anisotropic						CE					PyTorch
	Wang et al. [152]						volume				Octree-based					BCE CE					-
	L3DSR-Oct [157]						volume				Octree-based					BCE CE					-
	IPF-SPCNet [180]						hybrid			E							CE					-
	Chen et al. [19]						volume	GA Module								BCE CE					-
	LMSCNet [124]						view-volume	MSFA			2D						CE					PyTorch
	SCFusion [164]						volume	DC									CE					-
	S3CNet [21]						hybrid			L	Sparse					BCE CE PA					-
	JS3C-Net [169]						volume				Sparse						CE					PyTorch
	Local-DIFs [121]						point-based									BCE CE SCY					-
	2021 SISNet [11]						hybrid			M	DDR						CE					PyTorch

a These original works were significantly extended in

[START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF]

,

[START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF]

, or [124]. b Includes both NYUv2 [133], NYUCAD [38]. c Includes both SUNCG [137], SUNCG-RGBD [86]. d Includes both ScanNet [24], CompleteScanNet [164]. Contextual Awareness -DC, Dilated Convolutions. (LW)-ASPP, (Lightweight) Atrous Spatial Pyramid Pooling. CCP, Cascaded Context Pyramid. FAM, Feature Aggregation Module. AIC, Anisotropic Convolutional Module. GA, Global Aggregation. MSFA, Multi-scale Feature Aggregation. PDA, Primal-Dual Algorithm. Fusion Strategies -E, Early. M, Middle. L, Late.

Table 2 :

 2 Semantic Scene Completion (SSC) methods.

  Despite the few SSC methods using points input, it is commonly used for object completion[START_REF] Huang | Deep neural network for 3D point cloud completion with multistage loss function[END_REF] 155, 166, 172].

	2D representations. Alternatively, depth maps or range im-
	ages (i.e. 2D polar-encoded LiDAR data) are common 2D
	representations storing geometrical information, therefore
	suitable candidates for the SSC task. Indeed many works [6,
	21, 54, 76-79, 84, 86, 159] used either representation alone
	or in conjunction with other modalities. Opposite to point
	cloud but similarly to grid representation, such encoding
	enables meaningful connexity of the data and cheap pro-
	cessing with 2D CNNs. Some works propose to transform
	depth into an HHA image
	with
	experiments showing it may slightly harm the performance or
	bring negligible improvement. Except for the lighter p-TSDF,
	all TSDF-variants require high computation times and hinder
	real time implementations.

3D point cloud. Despite the benefit of such representation, only three recent SSC works [121, 122, 180] use point cloud as input encoding. In [180], RGB is used in conjunction to augment point data with RGB features, whereas [121, 122] reproject point features in the top-view space.

  8: Multi-scale contextual aggregation. While context is indubitably important for SSC, different strategies are used to aggregate features from various spatial/scale contexts. Color blocks stand for convolutions with different dilation rates. F stands for any type of fusion. low computational cost. The strategy became popular among most works[START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF][START_REF] Dourado | EdgeNet: Semantic scene completion from RGB-D images[END_REF][START_REF] Dourado | Semantic scene completion from a single 360-Degree image and depth map[END_REF][START_REF] Garbade | Two stream 3D semantic scene completion[END_REF][START_REF] Guo | View-volume network for semantic scene completion from a single depth image[END_REF][START_REF] Li | Depth based semantic scene completion with position importance aware loss[END_REF][START_REF] Li | Attention-based multimodal fusion network for semantic scene completion[END_REF][START_REF] Liu | See and think: Disentangling semantic scene completion[END_REF]124,137,160,164,177].

Table 4 :

 4 Detailed SSC class performance on SemanticKITTI[START_REF] Behley | SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences[END_REF] dataset. Best 5 methods are presented and ordered in decreasing mIoU performance from top to bottom. 93.8 53.2 41.9 43.6 66.2 61.4 38.1 29.8 53.9 40.3 52.4 CCPNet † ∧ [177] G 25.5 98.5 38.8 27.1 27.3 64.8 58.4 21.5 30.1 38.4 23.8 41.3 3DSketch [17] G+T 43.1 93.6 40.5 24.3 30.0 57.1 49.3 29.2 14.3 42.5 28.6 41.1 ForkNet † ∧ [160] G 36.2 93.8 29.2 18.9 17.7 61.6 52.9 23.3 19.5 45.4 20.0 37.1 IPF-SPCNet [180] G+T 32.7 66.0 41.2 17.2 34.7 55.3 47.0 21.7 12.5 38.4 19.2 35.1

		Method				Input	road (15.30%)	sidewalk (11.13%)	parking (1.12%)	other-gr. (0.56%)	building (14.1%)		car (3.92%)	truck (0.16%)	bicycle (0.03%)	motorcycle (0.03%)	other-veh. (0.20%)	vegetation (39.30%)	trunk (0.51%)	terrain (9.17%)	person (0.07%)	bicyclist (0.07%)	motorcyclist (0.05%)	fence (3.90%)	pole (0.29%)	tr. sign (0.08%)	mIoU
		S3CNet [169]				G 42.0 22.5 17.0 7.9 50.2 31.2 6.7 41.5 45.0 16.1 39.5 34.0 21.2 45.9 35.8 16.0 31.3 31.0 24.3 29.5
		JS3CNet [169]			G 64.7 39.9 34.9 14.1 39.4 33.3 7.2 14.4 8.8 12.7 43.1 19.6 40.5 8.0 5.1 0.4 30.4 18.9 15.9 23.8
		Local-DIFs [121]			G 67.9 42.9 40.1 11.4 40.4 34.8 4.4 3.6 2.4 4.8 42.2 26.5 39.1 2.5 1.1	0 29.0 21.3 17.5 22.7
		TS3D+DNet+SATNet [6] G 62.2 31.6 23.3 6.5 34.1 30.7 4.9 0	0	0.1 40.1 21.9 33.1 0	0	0 24.1 16.9 6.9 17.7
		LMSCNet-SS [124]		G 64.8 34.7 29.0 4.6 38.1 30.9 1.5 0	0	0.8 41.3 19.9 32.1 0	0	0 21.3 15.0 0.8 17.6
	Method	Input	ceil. (0.74%)	floor (12.44%)	wall (9.67%)	win. (2.12%)	chair (2.03%)	bed (9.17%)	sofa (6.78%)	table (4.14%)	tvs (0.53%)	furn. (36.64%)	objs. (15.74%)	mIoU
	SISNet [11]	G+T 54.7

∧

Results reported on resolution different to (60 × 36 × 60). CCPNet: (240 × 144 × 240). ForkNet: (80 × 48 × 80).

† Pretraining on SUNCG.

Table 5 :

 5 Detailed SSC class performance on NYUv2 [133] dataset. Best 5 methods from Tabs. 3 and 7 are presented and ordered in decreasing mIoU performance from top to bottom. Results reported on resolution different to (60 × 36 × 60). CCPNet: (240 × 144 × 240). * Texture input not used due to absence in SUNCG.

	Method	Input	ceil. (2.68%)	floor (12.27%)	wall (33.55%)	win. (5.79%)	chair (1.80%)	bed (5.95%)	sofa (4.94%)	table (2.90%)	tvs (0.36%)	furn. (15.04%)	objs. (14.73%)	mIoU
	3DSketch ‡ [17] G* 97.8 91.9 84.1 72.6 60.8 86.8 81.7 68.7 52.6 75.7 68.2 76.5
	Wang et al. [152] G 98.2 92.8 76.3 61.9 62.4 87.5 80.5 66.3 55.2 74.6 67.8 74.8
	CCPNet ∧ [177] G 99.2 89.3 76.2 63.3 58.2 86.1 82.6 65.6 53.2 76.8 65.2 74.2
	ESSCNet [174] G 96.6 83.7 74.9 59.0 55.1 83.3 78.0 61.5 47.4 73.5 62.9 70.5
	EdgeNet [32]	G* 97.2 94.4 78.4 56.1 50.4 80.5 73.8 54.5 49.8 69.5 59.2 69.5

∧ ‡ Results provided by authors.

Table 6 :

 6 Detailed SSC class performance on SUNCG [137] dataset. Best 5 methods are presented and ordered in decreasing mIoU performance from top to bottom.

  ∧ 94.3 ∧ 67.1 ∧ 41.3 ∧ 93.4 ∧ 91.2 ∧ 85.1 ∧ 55.0 ∧ Input: Geometry (depth, range, points, etc.), Texture (RGB).

					Indoor
				Real-world		Synthetic
				NYUv2 [133]		NYUCAD [38]
				60 × 36 × 60		60 × 36 × 60
	Method	Input Prec. Recall IoU mIoU Prec. Recall IoU mIoU
	2017 SSCNet [137]	G 59.3 92.9 56.6 30.5 75.0 96.0 73.0	-
	2018 VVNet [54]	G 69.8 83.1 61.1 32.9 86.4 92.0 80.3	-
	VD-CRF [176] G	-	-	60.0 31.8	-	-	78.4 43.0
	SATNet [86] G+T 67.3 85.8 60.6 34.4	-	-	-	-
	2019 EdgeNet [32] G+T 79.1 66.6 56.7 33.7	-	-	-	-
	ForkNet [160] G	-	-63.4 ∧ 37.1 ∧	-	-	-	-
	CCPNet [177] G 78.8

∧ Results reported at a different resolution. CCPNet: (240 × 144 × 240). ForkNet: (80 × 48 × 80).

Table 7 :

 7 SSC performance on indoor datasets with SUNCG pre-training. Synthetic data pre-training results on slight performance gains for all methods.

	scenes with 3DSketch ranking second in mIoU among non-
	pre-trained methods and third overall (Tabs. 3 and 7), visible
	in Figs. 11a, 11b, 11c. Similarly, S3CNet dominates per-
	formance in SemanticKITTI as already mentioned, which
	performance is noticeable in Figs. 12c, 12d.

Table 8 :

 8 Network statistics. Number of parameters and FLOPs are reported per method, grouped by resolution output: 60 × 36 × 60 for typical indoor datasets

https://futurism.com/tech-suing-facebook-princeton-data

https://competitions.codalab.org/competitions/22037 (a) Point Cloud (b) Voxel Grid (c) Implicit Surface (d) Mesh

In their seminal work, for memory reasonSong et al. [137] evaluated SSC only at the 1:4 scale. Subsequently, to provide fair comparisons between indoor datasets and methods, most other indoor SSC have been using the same resolution despite the fact that higher resolution ground truth is available. Recent experiments in[START_REF] Chen | 3D sketch-aware semantic scene completion via semi-supervised structure prior[END_REF] advocate that using higher input/output resolution boosts the SSC performance significantly.

Indoor Outdoor

Real-world Synthetic Real-world NYUv2 [133] NYUCAD [START_REF] Firman | Structured prediction of unobserved voxels from a single depth image[END_REF] SUNCG