Firmin Didot'

Également Hugo

Xi

Timothée, Robin, Mathis, Thomas, Othman, Yannis, François, Praveer, Abdou, Sergey, Victor, Philippe Yang

Théo Raphael Benjamin

Romain Shell

Clément, Rahima, David, Xuchong, Julien, Nicolas Georgia Tom

Sophie

Simon Nguyen

Pierre-Alain Langlois

Matthew Fisher

Oliver Wang

Yang Xiao

Xuchong Qiu

Mathieu Aubry

Vladimir Kim

Bryan Russell

Keywords: BIM, machine learning, reconstruction, semantic

A T E X, using the beautiful tufte-latex class 1 and a hand-made overlay inspired by Aaron Turon's thesis Understanding and expressing scalable concurrency and Marie-Morgane Paumard's thesis, Solving jigsaw puzzles with deep learning.

Résumé

L'émergence du Building Information Model (BIM) dans les domaines de l'industrie du BTP et de la gestion de la ville révolutionnent notre manière de concevoir, construire, gérer et entretenir nos bâtiments. Les modèles BIM rassemblent à la fois des informations géométriques sur le bâtiment, mais également des informations sémantiques permettant d'identifier chaque composant logique (mur, dalle, fenêtre, etc.). Cette information est généralement créée manuellement à la conception d'un nouveau bâtiment; cependant, seul 1% du parc immobilier à reconstituer est renouvelé chaque année. La génération automatique de maquettes BIM à partir de capteurs tels que des appareils photos ou des LIDARs (générant directement des nuages de points) constitue dans ce contexte un besoin de plus en plus important. L'objectif de cette thèse est le développement de nouvelles méthodes pour la reconstruction de maquettes BIM, qui incluent à la fois les informations géométriques et sémantiques. Un important effort de recherche est déployé pour rendre les méthodes existantes plus robustes aux contraintes présentes dans les différents cas d'usage pratiques. La reconstruction 3D débute généralement par l'acquisition de nuages de points par LIDARs ou par photogrammétrie, c'est-à-dire la triangulation de points clés mis en correspondance entre différentes photographies d'un même lieu, avant la reconstruction de la surface sous-jacente. Dans le contexte du bâtiment, la triangulation est rendue difficile par la présence de zones sans texture dans lesquelles les algorithmes peinent à détecter des points-clés. De plus, certaines parties du bâtiment peuvent être absentes des données d'entrée en raison des occultations ou d'erreurs imputables à l'opérateur. En ce qui concerne la sémantique au sein des nuages de points, il existe d'importantes ambiguïtés entre les différentes classes: il peut par exemple être ardu de repérer sur un nuage de points la séparation entre une porte et un mur. Dans cette thèse, nous proposons trois contributions techniques afin de traiter ces limitations.

Tout d'abord, nous proposons la première méthode permettant la reconstruction planaire par morceaux de bâtiments à partir d'images utilisant des segments de droites comme primitives. En effet, bien que les vastes zones sans texture dans les bâtiments (murs blancs par exemple) rendent difficile la détection de points clés, les segments de droites restent visibles et détectables (par exemple grâce au changement de luminance à l'intersection entre deux murs). L'utilisation de ces segments de droites rend notre méthode robuste aux surfaces sans texture, et nous sommes en mesure de reconstruire des scènes pour lesquelles les méthodes basées sur les points-clé échouent.

La seconde contribution traite le problème de la reconstruction de maillage à partir d'un ensemble d'images de basse résolution prises à des positions et orientations connues de l'espace. Dans ce contexte, les méthodes traditionnelles échouent, et apprendre des a priori directement sur de grandes bases de données de formes 3D nous permet tout de même d'effectuer une reconstruction. Plus précisément, notre méthode apprend des a priori afin de fournir une première approximation de la forme à reconstruire que nous affinons ensuite en ajoutant des contraintes géométriques. Notre méthode fournit directement un maillage et se compare positivement avec les méthodes de l'état de l'art dont la sortie est limitée à un nuage de point bruité.

Notre troisième contribution est VASAD, un jeu de données pour la reconstruction volumique et sémantique, que nous avons créé à partir de modèles BIM bruts disponibles en ligne. C'est le premier jeu de données de cette taille (62.000m²) à fournir simultanément des annotations géométriques et sémantiques. En plus de ce jeu de données, nous proposons 2 méthodes pour reconstruire de manière conjointe la géométrie et la sémantique à partir d'un nuage de point et nous démontrons que notre jeu de données propose des défis susceptibles de fortement influencer les recherches futures.

Mots-clé BIM, apprentissage, reconstruction, sémantique.

Contents

II SURFA C E R E C O N S T R U C T I O N F R O M 3 D L I N E S E G M E N T S

I V VA S A D : A VO L U M E A N D S E M A N T I C D ATA S E T F O R B U I L DI N G R E C O N -STRUCTIO N F R O M P O I N T C L O U D S

4.19

Reconstruction with our method where the gathering of inliers for a plane model relies on the distance to the plane intersection for lines already supporting a plane (left, our approach), and to the plane only (right, standard approach). . . . . . . . . . . . . . . . . . . . . . . . .

4.20

Reconstruction with our method where all structural lines are treated specifically (left), and considered as two textural lines, i.e., one for each supported plane (right). . . . . . . . . . .

4.21

Reconstruction with a regularization on corners only (left), vs on edges only (right). . . . . .

4.22

Reconstruction with a regularization on corners and edges (left), vs on corners only (right). Part I

4.23

PROLOGUE

Introduction

Chapter 2 ]

1.1 motivation

1.1.1 BIM
Since the spread of computers in almost every professional field, construction and civil engineering have benefited a lot from this technology. Early in this interaction, the idea grew to save time and money by making digital models for buildings. Though it might seem obvious to use computers for building design, it was soon understood that bringing more information into the digital model could lead to gains in the whole building life-cycle steps [START_REF] Ullah | An Overview of BIM Adoption in the Construction Industry: Benefits and Barriers[END_REF], such as operations, maintenance, repair or asset management. The Building Information Models were invented to fill this purpose: not only do they contain the geometry of the building, but many operational details such as building parts, infrastructure, network, metadata and documentation (see Figure 1.1). Modern software has been built to make it convenient to generate such complex files, e.g., ReVit, BIM 360, ARCHICAD, or TEKLA Structures. While efforts are being made to standardize BIM files, the use of this digital twins gets more and more generalized. A visual representation of a building and its corresponding BIM file is provided on Figure 1.2.

Automatic model generation

While this software allows the generation of rich data during the building design, it is expensive to manually create BIM files for existing building. As a matter of fact, this involves getting precise measurements, guessing what is behind hidden spots, gathering all the data in the software and of course, avoiding mistakes along this tedious work. Again, computer science can have an impactful role to play in automating this task [START_REF] Huang | Bim, machine learning and computer vision techniques in underground construction: Current status and future perspectives[END_REF]. While parts of the infrastructure require specific sensors to recover (e.g., electric network in walls), recovering the logical functionality of each building component (walls, windows, stairs, etc.) as a semantic label is doable for a human by visualizing the data. Indeed, knowing these labels could allow automatic material assignment when conducting simulation of the thermal performance of the building.

In this context we can ask: in the spirit of BIM models, is it possible with simple sensors such as regular cameras or LIDARs (see 1.2.1) to recover both the geometry and the semantic class of each component in a given building? This thesis is dedicated to answering that question. Our problem is twofold and we discuss here the definition of both aspects. Actual building [START_REF]Villa le sextant[END_REF] Representation of the BIM model

Structure and walls

Hydraulic network highlighted

Electric network highlighted

approach

We provide here a general description of the tasks involved in the thesis and general bibliographic elements. A detailed related work is provided for each of our contributions in sections 2.2, 5.2 and 8.2.

Sensors

Every signal-processing routine starts with a sensor acquisition. For buildings, it is possible to use several kinds of sensors:

] RGB cameras: Regular cameras produce an image as a grid of pixels, each of which contains an RGB byte triplet which encodes a red/green/blue light color information. While this is a very natural way of capturing and representing our environment as we see it; this does not allow a direct access to a geometric information. With this type of data, recovering the 3D information requires additional processing, which we develop below.

] RGB-D cameras: Also called 2.5D sensors, these devices (e.g., Microsoft Kinect, Intel Realsense and Apple's Face ID) make use of structured light to produce a depth information on top of the color information for every pixel. A point cloud can then directly be obtained thanks to the camera calibration parameters. Though they are still slightly more expensive than regular cameras, these sensors become more and more common as they are being integrated on modern smartphones.

] LIDAR sensors: Laser-based sensors produce a point cloud where each point represents the intersection between a laser beam and a physical surface. These devices are the most expensive but their price is decreasing fast. They are by far more precise than RGB-D camera and they cover a larger field of view.

Because they measure physical quantities, all of these sensors are subject to noise from various sources. These sources include the manufacturing precision of the optical components, the general condition of the optical components, the homogeneity of the atmosphere between the sensor and the captured object, and specific properties of the surface which may be incompatible with a proper capture (e.g., a mirror for RGB-D cameras and LIDAR sensors).

Dealing with noise is central when working on real data and specific methods perform denoising [START_REF] Mukesh C Motwani | Survey of image denoising techniques[END_REF][START_REF] Tian | Deep learning for image denoising: a survey[END_REF]. For this reason, algorithms can be designed first on synthetic data for which the level of noise is controlled, before being applied to real data.

3D Reconstruction

Reconstructing the geometry from photographs or LIDARs is usually referred to as 3D Reconstruction in the computer vision literature. This process aims at retrieving the shape and appearance of real objects. The notion of virtual shape is very ill defined as there are many ways to represent a shape in 3D: point clouds, meshes, primitive set, implicit functions. The choice of the correct representation depends on the input, and the use we want to make of the 3D reconstruction.

In the particular case of BIM models, current software mostly represents the shapes as meshes, more specifically closed meshes. This allows us to define an interior and an exterior for any part in the scene, which is crucial to perform thermal and acoustic simulations. This distinction is paramount since most approaches in the computer vision literature are focused on surface reconstruction (i.e., generating surfaces which do not necessarily partition the space between interior/exterior) rather than volume reconstruction. In this dissertation, when dealing with building reconstruction, we strive to propose methods which reconstruct proper volumes as they are more useful for downstream applications. Additionally, the parts in BIM models are usually represented in an idealized version, i.e., the way the architect drew them. In order to get closer to this representation, it is usual to add constraints such as piecewise planarity and/or encourage right angles in the final reconstruction. These constraints aim to account for planar walls typically intersecting at right angles.

3D Reconstruction has the particularity to be an ill-defined problem: the representation we have of the object we try to reconstruct is often partial and there are therefore many ways to perform reconstruction while respecting the input data.

Images as input. When dealing with images, two major classes of methods are being used:

] Photogrammetry: given a set of input images, the idea is to first find corresponding point pairs across different images to calibrate the cameras (i.e., guessing the relative camera position to one another as well as the internal parameters such as the focal length and the optical center). It is then possible to obtain a point cloud by triangulation techniques. In practice, the bundle adjustment methods [START_REF] Lutz | Structure-from-Motion revisited[END_REF] do both tasks (calibration and triangulation) simultaneously, but it is necessary to densify the matches.

] Prior-based methods: Assuming that we can decompose an image in a depth image, a reflectance image and an illumination model [START_REF] Berthold | Determining lightness from an image[END_REF], the Shape from shading algorithms propose to recover the shape of an object assuming just the illumination and reflectance are known [START_REF] Zhang | Shape-from-shading: a survey[END_REF]. These methods make strong assumptions on the scene and work well only under specific conditions. More recently, the advent of deep learning techniques (see 1.2.4) allows to directly learn strong priors from large-scale datasets of objects.

Point cloud as input. When dealing with point clouds, we already have access to a geometric information which may be sufficient for some applications. However, point clouds have major flaws: to well represent a shape, they need to be particularly dense (typically hundreds of millions for a building), and the sampling needs to be as uniform as possible (to avoid missing details). This is a very heavy and inefficient representation, and due to the capture from fixed point of views, the raw point clouds are usually not uniformly sampled (see Figure 1.3). As previously mentioned, raw point clouds are also subject to noise, which makes the estimation of the underlying surface less precise. Finally, for many applications, the normal information associated to each point of the surface is often required. Though this can be estimated by various methods [START_REF] Boulch | Fast and robust normal estimation for point clouds with sharp features[END_REF], the accuracy of the method highly depends on the point cloud aforementioned qualities. For these reasons, an important research effort has been conducted to perform surface prediction from a point cloud. Classical methods make use of priors such as the smoothness of the underlying surface [START_REF] Michael Kazhdan | Poisson surface reconstruction[END_REF] or negligible noise [BMR + 99]. In the case of BIM, surface reconstruction is not enough: we need a volume reconstruction, i.e., a partition of the space between empty and full parts. This is a harder task in general and fewer methods provide such a partition as an output [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF][START_REF] Nan | Polyfit: Polygonal surface reconstruction from point clouds[END_REF].

Semantic segmentation

Our objective also includes semantic segmentation, i.e., the process of labeling each building part. In general, this task is still an important research challenge. Simple shapes like planes [START_REF] Schnabel | Efficient RANSAC for point-cloud shape detection[END_REF] can efficiently be retrieved. When the segments are well separated by a high curvature, a region growing based algorithm [ZPK + 02] can be used. However, for many practical tasks, and in the context of BIM reconstruction in particular, the parts we want to segment are not described by their geometry, but by their functional use. The IFC format often used for BIM projects defines parts such as slabs, roofs, windows, doors to name a few. These parts are usually represented by a closed volume and a label. Most of the time, the parts do not overlap and the separation between two objects is visually clear. However, when looking at the point cloud only (in the case of a LIDAR sensor), it can be very difficult to tell different components apart (see Figure 1.3). Moreover, parts in buildings can be ambiguous: in modern buildings flat roofs can for instance be mistaken for slabs. For these reasons, it makes sense to learn the shape priors directly on the data instead of hand crafting it. Again, recent deep-learning-based techniques show promising results in this direction (see 1.2.4).

Deep Learning

In recent years, the deep-learning-based techniques have spread through all the tasks studied in computer vision and allowed major breakthrough. Even though these techniques have been first developed during the early years of computer vision [START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF], two main factors account for the recent success of these algorithms:

] Data:

contributions 1.3.1 Main contributions

As we noticed in the previous section, many topics are concerned by the subject of BIM reconstruction. In this dissertation, we are interested in four particular aspects:

] 3D reconstruction of buildings.

] Reconstruction from photographs taken thanks to RGB cameras.

Original mesh

Point cloud (darker points are closer to the sensor) ] Reconstruction from point clouds obtained from LIDAR scans.

] The use of machine learning, especially deep learning to leverage large scale datasets to perform both reconstruction and semantic segmentation.

This dissertation aims at paving the way towards efficient methods to automatically reconstruct and label each logical part of a building. As such, we bring three contributions which deal with the four aforementioned aspects (see Figure 1.4):

] Surface Reconstruction from 3D Line Segments

In this contribution, we are interested in the pure problem of surface reconstruction in the context of buildings. In particular, we address one of the most important failure cases of the traditional methods, i.e., the reconstruction of textureless areas, which are omnipresent in modern buildings. To that end, we develop an original reconstruction method based on line segments instead of points.

] 3D Reconstruction by Parameterized Surface Mapping

In this contribution, we are interested in the possibility to use machine learning to learn priors for reconstruction. Moreover, we are interested in leveraging multiple views of an object to further refine the obtained mesh.

] VASAD: a Volume and Semantic dataset for Building Reconstruction from Point Clouds

With this contribution, we bring into play the semantics along with geometry to make the synthesis of both aspects. We propose VASAD, a dataset built from freely available BIM products, and show that we can directly learn on models for the 3D reconstruction with semantic task, and therefore come closer to a full automation of the BIM generation process.

We open source the code and data for our contributions on Github [START_REF] Langlois | Github repositories of Pierre-Alain Langlois[END_REF].

Other contributions

Our other contributions in the course of the PhD will not be discussed in this dissertation.

They include:

] Pose from shape [XQL + 19]: a 3D pose estimation for objects from a single picture.

] NeeDrop: a work including an unsupervised loss to learn to reconstruct implicit surfaces from a dataset of point clouds.

organization of the dissertation

Our surface reconstruction from 3D line segments algorithm is presented in part II while our 3D reconstruction by parameterized surface mapping algorithm is presented in part III.

Our dataset VASAD and its baselines are presented in part IV and part V concludes this dissertation.

Part II SURFACE RECONSTRUCTION FROM 3D LINE SEGMENTS

Introduction

[ Chapter 1 Chapter 3 ]
Synopsis In man-made environments such as indoor scenes, when point-based 3D reconstruction fails due to the lack of texture (see Figure 2.1), lines can still be detected and used to support surfaces. We present a novel method for watertight piecewise-planar surface reconstruction from 3D line segments with visibility information. First, planes are extracted by a novel RANSAC approach for line segments that allows multiple shape support. Then, each 3D cell of a plane arrangement is labeled full or empty based on line attachment to planes, visibility and regularization. Experiments show the robustness to sparse input data, noise and outliers.

reconstruc ting textureless surfaces

Numerous applications make use of 3D models of existing objects. In particular, models of existing buildings (e.g., BIMs) allow virtual visits and work planning, as well as simulations and optimizations, e.g., for thermal performance, acoustics or lighting. The building geometry is often reconstructed from 3D point clouds captured with LIDARs or using cameras and photogrammetry. But with cameras, registration and surface reconstruction often fail on indoor environments because of the lack of texture and strong view points changes: salient points are scarce, point matching is difficult and less reliable, and when calibration nonetheless succeeds, generated points are extremely sparse and reconstructed surfaces suffer from holes and inaccuracies. Yet, recent results hint it is possible to rely on line segments rather than points. Lines are indeed prevalent in man-made environments, even if textureless. From robust detection [START_REF] Grompone Von Gioi | LSD: a line segment detector[END_REF][START_REF] Salaün | Multiscale line segment detector for robust and accurate SfM[END_REF] and matching [ZYL11, WWH09, HS12] to camera registration [EE11, SMM16b, SMM17, MDR18] and 3D segment reconstruction [START_REF] Hofer | Incremental line-based 3D reconstruction using geometric constraints[END_REF][START_REF] Hofer | Efficient 3D scene abstraction using line segments[END_REF], lines can be used when photometric approaches fail for lack of texture. But as opposed to point processing, line-based surface reconstruction has little been studied [START_REF] Witt | Maximally informative surface reconstruction from lines[END_REF][START_REF] Mentges | Surface reconstruction from image space adjacency of lines using breadth-first plane search[END_REF]. This paper presents a novel approach to do so.

A change of paradigm is needed to consider 3D line segments rather than points. Transposing point-based methods to lines is difficult as many point-related assumptions do not hold for line segments. Indeed, points should be numerous enough (often, in thousands), with a uniform enough sampling, with an accurate enough detection and matching, and most of all, they must belong at most to one primitive. On the contrary, only a few tens of lines (rarely hundreds) are typically detected, and their density and sampling uniformity is so low that they cannot directly support a good surface reconstruction. Also, due to noise in local gradients and varying occlusions depending on viewpoints, segment detection is less accurate and often leads to over-segmentation and unstable end-points which are ignored by most 2D line matchers. Only after image registration and 3D segment reconstruction can 2D detections be related to actual fragments of a 3D line segment, moreover possibly differing according to the different viewpoints. Besides, curvy shapes as cylinders may yield unstable occlusion edges (silhouettes), yielding noise or outliers. Finally, some 3D lines identify straight edges that are creases between two planar surfaces, and thus support two shapes, contrary to points, that only support one shape.

Belonging to two primitives rather than one requires reconsidering shape detection. In particular, in greedy iterative methods, removing all data supporting a detected shape could prevent detecting other shapes because all or a significant fraction of features would then be missing. For instance, it would not be possible to detect all the faces of a cube given only its edges (see Figure 3.3). And even if enough 3D data remain for detection, shape sampling would be affected and some shapes would be less likely or unlikely to be detected.

Overview. We propose the first complete reconstruction pipeline that inputs 3D line segments with visibility information and outputs a watertight piecewise-planar surface without self-intersection (cf. Figure 2.2). We first extract primitive planes from the line cloud, distinguishing two kinds of line segments: textural lines, supporting a single plane, and structural lines, at the edge between two planes. Then, we label each 3D cell of the plane arrangement as full or empty by minimizing an energy based on line type, line segment support, visibility constrains and regularization.

Our main contributions are as follows:

-We define a robust and scalable plane detection method from 3D line segments, without scene assumptions. This novel non-straightforward RANSAC formulation takes into account a key specificity of lines vs points, namely that they can support up to two primitives (at edges), which breaks the greedy iterative detection traditionally used with points.

Pictures

-We totally recast the surface reconstruction approach of [CLP10, BdLGM14] into a line-based setting. We meaningfully and efficiently generalize data fidelity and visibility from points to line segments, taking care of lines supporting two planes. We also feature a simpler and lighter treatment of noise.

-We validate our method on existing datasets, and provide new ones to assess line-based reconstruction quality.

related work

Surface reconstruction has been extensively studied from 3D points [BTS + 14] and/or images [START_REF] Furukawa | Multi-view stereo: A tutorial[END_REF]. The conventional multi-view stereo reconstruction algorithms essentially focus on finding pointwise correspondences across images to first find the relative position of images to each other, and then to extract a dense point cloud by incorporating epipolar constraints and designing photometric errors as detailed in [START_REF] Furukawa | Multi-view stereo: A tutorial[END_REF]. Schoenberger et al. have developed COLMAP [START_REF] Lutz | Structure-from-Motion revisited[END_REF], a method which leverages both geometric and photometric constraints to build a point cloud from images. This algorithm outperforms the state-of-the-art on several public benchmarks. A surface can then be obtained from the dense point cloud thanks to meshing methods which leverage priors on the surface smoothness [START_REF] Michael Kazhdan | Poisson surface reconstruction[END_REF] or the point cloud density [START_REF] Labatut | Robust and efficient surface reconstruction from range data[END_REF].

As opposed to the point-based methods, we consider here the input to be 3D line segments (with viewpoints), that can be sparse, missing, noisy and corrupted with outliers. We aim at an idealized piecewise-planar and watertight surface to be consistent with the current representation of buildings in BIM models.

To deal with sparse data, some methods detect planes based on 3D features and dominant orientations [START_REF] Sudipta | Piecewise planar stereo for image-based rendering[END_REF], possibly with a Manhattan-world assumption [START_REF] Furukawa | Manhattan-world stereo[END_REF], and create piecewise-planar depth maps taking into account visibility and local consistency. Other approaches consider 2D image patches back-projected on 3D planes [START_REF] Mičušík | Multi-view superpixel stereo in urban environments[END_REF][START_REF] Bourki | Patchwork stereo: Scalable, structure-aware 3D reconstruction in man-made environments[END_REF]. In contrast, our method produces a watertight mesh, does not impose a few specific orientations, and can work with 3D features only, not requiring images and not working at pixel level.

Another approach to little input data is to extend boundaries and primitives until they intersect [START_REF] Castellani | Improving environment modelling by edge occlusion surface completion[END_REF]. It however does not ensure a watertight reconstruction either. This is only achieved by methods that create a volumetric partition of the 3D space and extract the surface from full and empty cells. The partition can be a voxel grid [START_REF] Schnabel | Completion and reconstruction with primitive shapes[END_REF], a 3D Delaunay triangulation [START_REF] Labatut | Efficient multi-view reconstruction of large-scale scenes using interest points, Delaunay triangulation and graph cuts[END_REF][START_REF] Wan | Variational surface reconstruction based on Delaunay triangulation and graph cut[END_REF] or a plane arrangement [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF][START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF].

Wireframe reconstruction is what most lined-based methods focus on: rather than surfaces, they study how to generate meaningful 3D line segments [JKTS10, HWB13, HMB16, IS17], after line matching is performed [START_REF] Schmid | Automatic line matching across views[END_REF]. And more general curves than lines are not used beyond structure from motion [START_REF] Nurutdinova | Towards pointless Structure from Motion: 3D reconstruction and camera parameters from general 3D curves[END_REF].

Surface reconstruction with lines in addition to points has received a modest attention. Baillard and Zisserman [START_REF] Baillard | Automatic reconstruction of piecewise planar models from multiple views[END_REF] reconstruct planes from a 3D line and a neighboring detected point. It requires lines surrounded with texture and is outlier-sensitive. It also does not prevent self-intersections nor guarantees watertightness. The authors of [START_REF] Bay | 3D from line segments in two poorly-textured, uncalibrated images[END_REF] segment images into likely planar polygons based on 3D corner junctions and use best supporting lines to reconstruct polygons in 3D. For 2.5D reconstruction, extracted 3D lines [START_REF] Schmid | Automatic line matching across views[END_REF] are used with a dense height map to build a line arrangement on the ground plane and create geometric primitives and building masks [START_REF] Zebedin | Fusion of feature-and area-based information for urban buildings modeling from aerial imagery[END_REF]. In [START_REF] Sudipta | Piecewise planar stereo for image-based rendering[END_REF], pairs of 3D lines generated from vanishing directions provide plane hypotheses, validated by 3D points. The surface is a set of planar patches created from plane assignment to pixels. [START_REF] Sugiura | 3D surface reconstruction from point-and-line cloud[END_REF] adds points uniformly sampled on the 3D lines to the Delaunay triangulation, introducing extra parameters, and although visibility is treated without sampling, the method is unlikely to work on scenes with only sparse lines. [HMB14] also shows a meshing improvement using 3D line segments.

Surface reconstruction from line segments only, when points fail due to the lack of texture, has little been studied.

The method described in [START_REF] Zaheer | Shape from angle regularity[END_REF] presents a single-view surface reconstruction based on 2D line segments. Lines are paired from segment extensions along their direction, and planes orientations are sought by RANSAC, hypothesizing mutually orthogonal corresponding 3D lines. Articulating lines are found at plane intersections to construct a multi-plane structure. Our structural lines are called 'articulation' or 'articulating' lines in [START_REF] Zaheer | Shape from angle regularity[END_REF]. They are discovered late, to set plane offsets, whereas we differentiate them early at plane detection. For robotic mapping, [START_REF] Witt | Maximally informative surface reconstruction from lines[END_REF] considers all combinations of two non-collinear coplanar line segments as plane hypotheses. Line segments are then assigned to possibly multiple planes in a face complex built from plane intersections. The reconstructed surface is made of faces depending on an occlusion score. Compared to our approach, this method does not scale well to many lines, is sensitive to outliers, relies on a number of conservative heuristics that can be detrimental to surface recall, involves no regularization, and does not reconstruct a watertight mesh. As for [START_REF] Mentges | Surface reconstruction from image space adjacency of lines using breadth-first plane search[END_REF], it first reprojects 3D lines into images that see them, studies the intersection of segments in 2D rather than planes in 3D, and infers plane hypotheses. The surface is made from image faces back-projected onto a possible 3D plane. Although less sensitive to outliers, this method involves heuristics and no proper regularization, and it reconstructs a non-watertight mesh with floating polygons and possible self-intersections.

Extracting 3D planes from line segments has little been treated; the literature focuses on point clouds, chiefly ignoring line clouds. The most popular scheme for points, which is robust to sparsity contrary to region growing as in [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF][START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF], is RANSAC [START_REF] Choi | Performance Evaluation of RANSAC Family[END_REF][START_REF] Schnabel | Efficient RANSAC for point-cloud shape detection[END_REF]. But as explained below, it cannot straightforwardly be applied to line segments because it relies on different distribution hypotheses and because of the possible association of a single segment to several primitives, also invalidating line discretization into points. Still, [START_REF] Witt | Maximally informative surface reconstruction from lines[END_REF] takes line segments as input, but plane detection is somewhat exhaustive, hence with scalability issues, and sensitive to outliers. Using laser data, [START_REF] Cabo | Mobile laser scanner data for automatic surface detection based on line arrangement[END_REF] exploits 3D lines to detect planes, but it uses strong properties of LIDAR acquisition, namely line parallelism and large and dense data.

An open question is if multi-model methods [ZKM05, TF08, IB12, MF14], which assume non-overlapping segmented data, can be adapted not only to large inputs but also to multiple shape support [START_REF] Kriegel | Clustering high-dimensional data: A survey on subspace clustering, pattern-based clustering, and correlation clustering[END_REF][START_REF] Baadel | Overlapping clustering: A review[END_REF], as absolutely required for line segments.

Surface reconstruction from a plane arrangement is a common topic, with variants enforcing plane regularity [LWC + 11, MMBM15] or level of detail [START_REF] Verdie | LOD Generation for Urban Scenes[END_REF], or offering reconstruction simplicity [START_REF] Nan | Polyfit: Polygonal surface reconstruction from point clouds[END_REF]. It is largely orthogonal to our work. Here we build on [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF], with line-specific data and visibility terms.

line extraction

In this work, we assume we have as input 3D line segments as well as their viewpoints.

Concretely, for our experiments, we extract 3D line segments thanks to the Line3d++ algorithm [START_REF] Hofer | Line3D++[END_REF][START_REF] Hofer | Efficient 3D scene abstraction using line segments[END_REF]. First, when the calibration is not provided, we compute it thanks to the classical feature matching and filtering method with bundle adjustment available in COLMAP [START_REF] Lutz | Structure-from-Motion revisited[END_REF]. Then, lines are detected thanks to the a contrario method LSD [START_REF] Grompone Von Gioi | LSD: a line segment detector[END_REF]. Line3d++ essentially consists of 4 steps, plus 1 optional step. It first generates potential line matches across the images based on epipolar constraints. The matches are then filtered based on the heuristic that correct matches usually support each other, i.e., a good potential 3D segment should be close to other 3D segments. After this process, each remaining 2D detection which has at least one match generates a 3D hypothesis thanks to the match that has the highest confidence, as computed during the previous step. Finally, the 2D detections are clustered based on the spatial proximity of their 3D hypothesis, and a 3D line segment is determined for each output cluster. As an optional refinement step, a bundle adjustment can be applied to ensure that the final 3D line segments fit their 2D residuals. The output 3D line segments along with their 2D residuals and the camera poses constitute the input to our method.

plane detection from 3d line segments

The first step of our approach is to detect planes that are supported by line segments in the input line cloud ℒ. We use the RANSAC framework [START_REF] Fischler | Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography[END_REF] as it scales well to large scenes and deals well with a high proportion of outliers, which are unavoidable in photogrammetric data. The general idea is to randomly select two candidate lines in the input line set and to find the lines which are close to the plane generated by the candidate lines. Repeating this process and keeping the plane with the largest number of inlier lines statistically allows to recover the most salient plane of the line set (see Figure 3.2). Our whole algorithm is summarized in Figure 3.5 and we provide here a detailed explanation. As argued above and shown experimentally (cf. Sect. 4), a key requirement is to allow a line to belong to two planes. Lines supporting one plane are considered textural; lines supporting two planes are deemed structural. Yet some actual texture lines may support additional "virtual" planes, as when a line is drawn around an object, e.g., at the borders of a frieze around the walls of a room, which belongs both to the vertical walls and to an non-physical horizontal plane.

Candidate plane construction

We generate candidates by sampling the minimum number of observations required to create a model, i.e., two non-collinear line segments to define a plane. Two 3D segments 𝑙 𝑎 , 𝑙 𝑏 can be coplanar in two ways: they can be parallel, or their supporting infinite lines can intersect. With noisy real data, the latter can be relaxed using a maximum small distance 𝜖 between the lines. We discard parallelism because, when reconstructing man-made environments such as buildings, it may generate many bad planes. Indeed, two random vertical segments (e.g., detected on windows) are parallel but statistically unlikely to support an actual, physical plane (e.g., segments on different facades). We thus threshold the angle ∠(𝑙 𝑎 , 𝑙 𝑏 ), which also excludes the degenerate case of collinear segments.

Greedy detection and multi-support issues

We sample planes as line pairs and perform an iterative extraction of the most significant planes, i.e., with the largest number of supporting segments after a given number of sampling trials. However, contrary to usual RANSAC, we cannot remove supporting segments at once as they may actually belong to two planes; it would lead to detecting the main planes only, missing planes with a smaller support. Conversely, we cannot consider all segments as available at each iteration: it would statistically lead to multiple detections of the same large planes and again miss planes with small support.

A natural way to allow a datum to be part of several detection in greedy RANSAC is to remove inliers for model sampling but not for data assignment to models [START_REF] Zaheer | Shape from angle regularity[END_REF]. But for sparse data (which is the case with line segments), it fails to detect models with little data support, e.g., preventing detecting all the faces of a cube from its sole edges (see Figure 3.3).

Another way to allow the same datum to seed several models is to bound their number, i.e., 2 for lines supporting planes. But it does not work either as it often associates a line twice to more or less the same plane.

Our solution, as described below, is to bound the number of supported planes per line segment, but with an additional condition to prevent shared segments to belong to similar planes.

Candidate plane generation

We note Λ(𝑃 ) the set of line segments supporting a plane 𝑃, Π(𝑙) the set of planes supported by a line segment 𝑙 ∈ ℒ, with | Π(𝑙) | ≤ 2, and ℒ 𝑖 the set of segments supporting 𝑖 plane(s) for 𝑖 in 0, 1, 2.

We construct these sets iteratively by generating candidates planes 𝑃 and assigning them segments 𝑙 ∈ ℒ, some of which may have already been assigned to another plane Π(𝑙). Only line segments in ℒ 2 are discarded from the pool of available segments to support a plane, as they already support two planes. Initially, ℒ 0 = ℒ, and

ℒ 1 = ℒ 2 = ∅.
As line segments are not put aside as soon as they are assigned to a plane, they can be drawn again to generate new candidate models. However, generating several times the same plane (with the same supporting line segments) would not only reduce efficiency, but also make some models little likely to be drawn, as models with a large support would be sampled much more often. To prevent it, after drawing a first line segment 𝑙 𝑎 ∈ ℒ 0 ∪ ℒ 1 , there are two cases. If 𝑙 𝑎 ∈ ℒ 0 , i.e., if 𝑙 𝑎 has not been assigned to any plane yet, then the second segment 𝑙 𝑏 can be drawn unconditionally in ℒ 0 ∪ ℒ 1 as it will always yield a new model. If 𝑙 𝑎 ∈ ℒ 1 , i.e., if 𝑙 𝑎 has already been assigned to some plane 𝑃 ′ , with Π(𝑙 𝑎 ) = {𝑃 ′ }, then lines in Λ(𝑃 ′ ), i.e., supporting 𝑃 ′ , are excluded when drawing the second segment 𝑙 𝑏 . This ensures 𝑙 𝑎 , 𝑙 𝑏 cannot participate to the same already existing model. As the number of extracted planes is typically less than a few hundred, this drawing can be optimized by incrementally keeping track of the sets Λ (𝑃 ) = ℒ (ℒ 2 ∪ Λ(𝑃 )), that have not already been assigned to a detected plane 𝑃.

Note that we do not prevent a line pair to be redrawn when it previously failed to generate an accepted model (for lack of planarity, parallelism or poor support). It is not an issue as it does not lead to unbalanced chances to detect a plane. Yet, when the number of input line segments is not too large, we can perform a systematic drawing of all line pairs, possibly exploiting the above filtering. In this case, all possible models are considered and at most once.

Inlier selection

After picking a candidate plane 𝑃, we populate the support Λ(𝑃 ). For this, we go through each segment 𝑙 ∈ ℒ 0 ∪ ℒ 1 and assign it to Λ(𝑃 ) if close enough to 𝑃, i.e., if 𝑑(𝑙, 𝑃 ) ≤ 𝜖.

Several distances can be used, such as the average or the maximum distance to the plane.

If 𝑙 already supports some other plane 𝑃 ′ , i.e., if Π(𝑙) = {𝑃 ′ }, then also assigning 𝑙 to 𝑃 would make it a structural segment. As such, we impose that it lies close to the line at the intersection of both planes, i.e., 𝑑(𝑙, 𝑃 ∩ 𝑃 ′ ) ≤ 𝜖 (see Figure 3.4). This condition is stronger than imposing both 𝑑(𝑙, 𝑃 ) ≤ 𝜖 and 𝑑(𝑙, 𝑃 ′ ) ≤ 𝜖 as the angle between 𝑃 and 𝑃 ′ could be small and 𝑙 could then be close to both 𝑃 and 𝑃 ′ although far from their intersection. This condition is actually crucial. Without it, we would tend to associate 𝑙 to two planes 𝑃 and 𝑃 ′ which are very similar, and fail to detect crease lines.

3.2.5

Plane selection

Last, we sample 𝑁 iter models and keep the plane with the largest number of inliers. (See Figure 3.5 for the abstract version of the algorithm.) This plane detection differs from [START_REF] Zaheer | Shape from angle regularity[END_REF], that samples and populates planes from 2D line pairs instead of 3D lines, making inlier search quadratic, not linear, and requiring heuristically to only consider pairs defined by intersecting segment extensions, which is highly unstable due to noise in endpoints and which induces plane splitting at occlusions. We have none of these downsides. Besides, structural lines in [START_REF] Zaheer | Shape from angle regularity[END_REF] are found with heuristics after RANSAC, considering plane pairs and candidate lines, which only makes sense as they have few (<10) planes. We get them directly, without heuristics, in greater number, and for many more planes.

3.2.6

Plane refitting

After each plane 𝑃 best is selected, it is actually refitted to its inliers Λ best before being stored into Π, based on the (signed) distance of the segment endpoints, weighted by the segment length. As it changes the plane equation, we check whether the slice centered on the refitted plane 𝑃 ′ with thickness 𝜖 now contains extra segments. If so, they are added as inliers and refitting is repeated.

Plane fusion

Modeling a building may require different levels of details, including small plane differences such as wall offsets for door jambs, baseboards or switches. But setting a small 𝜖 to do so may easily break a wall or a ceiling into several fragments because it is not perfectly planar due to construction inaccuracies or load deflections. Each country actually has standards (official or not) defining construction tolerances, e.g., 1 cm error every 2 m for walls.

To prevent this arbitrary fragmentation while preserving details, we add a post-processing plane fusion step with a tolerance higher than 𝜖, i.e., with a maximal distance threshold 𝜖 fus > 𝜖 to the plane refitted on the union of inliers. This allows merging at 𝜖 fus accuracy several plane fragments detected at 𝜖. However, to make sure it applies only to cases described above, we impose a maximum angle 𝜃 fus when merging two planes and minimum proportion 𝑝 fus of common inliers. Specifically, we consider all pairs of planes in Π whose angle is less than 𝜃 fus , sort them, pick the pair with the smallest angle, and try merging it. If it succeeds, the two planes are removed, the new refitted plane is added, and the priority queue based on angles is updated before iterating. If it fails, the pair of planes is discarded and the next pair is considered. This is similar to a heuristics used in Polyfit [START_REF] Nan | Polyfit: Polygonal surface reconstruction from point clouds[END_REF].

Plane limitation

To make sure not too many planes are given to the surface reconstruction step, because of possible limitations (cf. Sect. 4.7), the algorithm may be stopped after at most 𝑁 max (best) greedy detections.

INPUT: set ℒ of 3D line segments ℒ 0 ← ℒ, ℒ 1 ← ∅, ℒ 2 ← ∅, Π ← ∅ while | ℒ 0 ∪ ℒ 1 | ≥ 2 and | Π | < 𝑁 max do Λ best ← ∅ // Current
∀ 𝑙 ∈ Λ best , Π(𝑙) ← Π(𝑙) ∪ {𝑃 best } // Planes supported by 𝑙 ℒ 2 ← ℒ 2 ∪ (Λ best ∩ ℒ 1 ) ℒ 1 ← (ℒ 1 (Λ best ∩ ℒ 1 )) ∪ (Λ best ∩ ℒ 0 ) ℒ 0 ← ℒ 0 (Λ best ∩ ℒ 0 ) end while OUTPUT: set of planes Π, with related support Λ(𝑃 ) 𝑃 ∈Π

surface reconstruction

The second step of our approach is surface reconstruction based on detected planes and observations of 3D line segments. Rather than selecting plane-based faces with hard constraints for the surface to be manifold and watertight [START_REF] Nan | Polyfit: Polygonal surface reconstruction from point clouds[END_REF], we follow [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF][START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF] and consider a scene bounding box, partition it into 3D cells constructed from the planes, and assign each cell with a status 'full' or 'empty' depending on segment visibility, with a regularization prior coping with sparse and missing data. The reconstructed surface is then the interface between full and empty cells (see Figure 3.6). By construction, it is watertight and free from self-intersections.

2D Plane arrangement

Extracted surface with labels (white: 0, i.e., empty; grey: 1, i.e., full) Figure 3.6: Extracting the reconstructed surface from the plane arrangement Our contribution is a total reformulation of [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF][START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF] in terms of lines, making the difference between textural and structural lines, and with a lighter treatment of noise in data.

The volume partition is given by a cell complex 𝒞 made from an arrangement of planes detected in the line cloud. For each cell 𝑐 ∈ 𝒞, we represent occupancy by a discrete variable 𝑥 𝑐 ∈ {0, 1}: 0 for empty and 1 for full. A surface is uniquely defined by a cell assignment x ∶ 𝒞 ↦ {0, 1}, where x(𝑐) = 𝑥 𝑐 . The optimal cell assignment x is defined as the minimum of an energy 𝐸(x) which is the sum of three terms: a primitive term 𝐸 prim (x) penalizing line segments not lying on the reconstructed surface, a visibility term 𝐸 vis (x) penalizing surface reconstructions on the path between observations and their viewpoints, and a regularization term 𝐸 regul (x) penalizing complex surfaces.

𝐸(x) = 𝐸 prim (x) + 𝐸 vis (x) + 𝐸 regul (x)
(3.1)

Dealing with noise

To deal with noise in input data, [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF] introduces slack in the choice of cells penalized for not being at the reconstructed surface and lets regularization make the right choices. The resulting formulation and resolution is heavy. Instead, we assume that plane extraction (Sect. 3.2) did a good-enough job: any segment supporting a plane (resp. two planes) is considered as a noisy inlier and is projected on the plane (resp. the intersection of the two planes). A segment not supporting any plane is treated as an outlier for data fidelity (no penalty for not being on the reconstructed surface) but not for visibility (penalty for not being seen from viewpoints if hidden by reconstructed surfaces).

Primitive term

𝐸 prim (x) penalizes line segments that support planes but do not lie on the reconstructed surface. But it does not penalize the presence of matter in front of segments w.r.t. viewpoints, letting the visibility term do it. Segments that support no plane are ignored as if outliers.

For a segment 𝑙 supporting one plane 𝑃, and for each viewpoint 𝑣 seeing at least a part of 𝑙, we consider the set 𝐶 of all cells 𝑐 immediately behind 𝑙 w.r.t. 𝑣, possibly only along a fraction 𝑙 𝑐 of 𝑙 due to occlusions (cf. Fig. 3.7(a)). Each 𝑐 ∈ 𝐶 is penalized if not full, with a cost 1-𝑥 𝑐 .

For a segment 𝑙 supporting two planes 𝑃 1 , 𝑃 2 , a cell behind 𝑙 w.r.t. viewpoint 𝑣 needs not be full. Any configuration is valid as long as the space around 𝑙 is not empty (see Fig. 3.7(b)): salient edges, reentering edges or planes (if the seemingly structural line happens to only be textural). To penalize only when all three cells 𝑐 around a visible fraction of 𝑙 are empty (ignoring the cell in front), we consider a cost of max(0, 1 -∑ 𝑐 𝑥 𝑐 ), which is equal to 1 in this case, and 0 in other configurations.

Both textural and structural cases can be covered with a single formula, where we weigh the cost by the length of the visible fraction of 𝑙 and normalize it by a scale of interest 𝜎:

𝐸 prim (x) = ∑ 𝑙∈ℒ 1 ∪ℒ 2 ∑ 𝑣∈𝒱(𝑙) ∑ 𝐶∈𝒞(𝑙,𝑣) | 𝑙 𝐶 | 𝜎 max(0, 1-∑ 𝑐∈𝐶 𝑥 𝑐 ) (3.2)
where ℒ 1 ∪ℒ 2 is the set of segments 𝑙 supporting at least one plane, 𝒱(𝑙) is the set of viewpoints 𝑣 seeing 𝑙, 𝒞(𝑙, 𝑣) is the set of cells 𝑐 adjacent to 𝑙 but not in the triangles of sight from 𝑣 to non occluded fragments of 𝑙 (locally 1 or 3 cells as to whether 𝑙 belongs to 1 plane or 2 planes), 𝑙 𝐶 is the set of fragments of 𝑙 in each cell 𝑐 ∈ 𝐶, and | 𝑙 𝐶 | is the sum of the lengths of segment fragments in 𝑙 𝐶 .

Visibility term

𝐸 vis (x) penalizes reconstructed surface boundaries between viewpoints and segments, as [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF][START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF]. It measures the number of times a 3D segment 𝑙 is to be considered an outlier as it should not be visible from a given viewpoint 𝑣, weighted by the length of the visible parts 𝑙 𝑣,𝑓 of 𝑙 on the offending faces 𝑓 (possibly fragmented due to occlusions). Contrary to 𝐸 prim (x), all segments are considered in 𝐸 vis (x), not just segments supporting a plane:

𝐸 vis (x) = 𝜆 vis ∑ 𝑙∈ℒ ∑ 𝑣∈𝒱(𝑙) ∑ 𝑓∈ℱ(𝑙,𝑣) | 𝑙 𝑣,𝑓 | 𝜎 | 𝑥 𝑐 +𝑣 𝑓 -𝑥 𝑐 -𝑣 𝑓 | (3.3)
where ℱ(𝑙, 𝑣) is the set of faces 𝑓 of the complex intersected by the visibility triangle (𝑙, 𝑣), at some unoccluded segment fractions 𝑙

3.3.4

Regularization term 𝐸 regul (x) penalizes surface complexity as the sum of the length of reconstructed edges (see Figure 3.8) and the number of corners (see Figure 3.9), with relative weights 𝜆 edge , 𝜆 corner , as defined in [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF]. Area penalization makes little sense here due to the low density of observations in some regions. We look for the most simple surface, which we assume is the surface that corresponds to a balanced minimization of these 2 measures (short edge length, few corners) while being consistent with the observation (line segment attachment to plane and visibility). 𝐸 regul (x) is higher-order in the existence of a corner depends on 8 cell values. (We assume any 4 planes have no intersection.) As in [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF], we use a mixed-integer programming formulation of the regularization terms. (3.4) can be rewritten as a standard linear program by introducing for each max term a new slack variable 𝑧 𝑠 ∈ R:

minimize x,z ∑ 𝑙∈ℒ\ℒ 0 ∑ 𝑣∈𝒱(𝑙) ∑ 𝑠∈ l𝑣 | 𝑠 | 𝜎 𝑧 𝑠 subject to 0 ≤ 𝑥 𝑐 ≤ 1, 𝑐 ∈ 𝒞. 𝑧 𝑠 ≥ 1 -∑ 𝑐∈𝒞 𝑐 ∩ (𝑣 𝑠) = 𝑠 𝑥 𝑐 𝑧 𝑠 ≥ 0 (3.5)
where z is the vector of all variables 𝑧 𝑠 , 𝑣 𝑠 is the visibility triangle formed by the line segment 𝑠 and the point of view 𝑣 and l𝑣 is the set of subsegments of 𝑙 as seen from viewpoint 𝑣.

The regularization term also is not linear. Minimizing the edges length and the number of corner indeed involves higher-order constraints: the presence of an edge (resp. corner) depends on the value of 4 (resp. 8) adjacent cells (full or empty). We reformulate these constraints, that are hard to solve, using linear terms only as proposed by [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF], i.e., using the absolute value of linear combinations of cell values, which can also be turned into an equivalent linear program with extra slack variables.

The resulting energy minimization problem is formulated as mixed-integer programming, with integral values (0 or 1) for the occupancy of cells 𝑥 𝑐 and continuous values for slack variables. As solving it is NP-hard, we also do as in [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF]: we relax the problem for optimization, i.e., the problem is solved for 𝑥 𝑐 ∈ [0, 1] for all 𝑐 ∈ 𝒞. This corresponds to a linear program that can be solved efficiently using off-the-shelf solvers. The obtained fractional values for variables 𝑥 𝑐 are rounded independently of each other.

3.3.6

Properties of reconstructed surface By construction, the surface we produce is watertight, even if the input data is very sparse, and not self-intersecting. Our process treats outliers (with RANSAC at plane detection stage, and regularization during reconstruction) and noise (with a model tolerance at plane detection stage and via projections when reconstructing). It has also several positive properties:

• Insensitivity to line over-segmentation: if a 3D line segment 𝑙 is split, 𝐸(x) does not change and thus the same surface is reconstructed. This provides robustness to oversegmentation, which is a common weakness of line segment detectors. (It may however change inlier-ness.)

• Little sensitivity at endpoints: given a line segment 𝑙, slightly changing its endpoint only makes a marginal change to 𝐸(x). (Yet it may change inlier-ness too.)

• Insensitivity to dummy planes: given a 3D cell assignment x, if an extra plane is randomly inserted in the arrangement, the value of 𝐸 vis (x) does not change as it only depends on surface transitions encountered on visibility path.

Results

[ Chapter 3

Chapter 5 ]

datasets

We experimented both with real and synthetic data. The real datasets consist of images of a 'MeetingRoom' from [START_REF] Salaün | Robust SfM with little image overlap[END_REF], of a 'Barn' from Tanks and Temples [KPZK17], of a 'DeliveryArea', a 'Bridge' and of a corridor named 'Terrains' from ETH3D [SSG + 17].

All scenes are poorly textured (walls of uniform colors). The synthetic datasets include a 'TimberFrame' house [START_REF] Jain | Exploiting global connectivity constraints for reconstruction of 3D line segments from images[END_REF] as well as two new synthetic datasets, that have been publicly released: 'HouseInterior' is a living room, with both large planar areas (walls, floor and ceiling) and smaller details (chair and table legs); 'Andalusian' is the outside of a modern house; it is piecewise-planar and uniformly white. MeetingRoom was calibrated with LineSfM [Sal17, SMM17] and we recalibrated the other real datasets using COLMAP [START_REF] Lutz | Structure-from-Motion revisited[END_REF], with distortion correction as it impacts line detection. The synthetic datasets came with their exact calibration.

Dataset #img | ℒ | | Π | | Π fus | | ℒ 0 | | ℒ 1 | | ℒ 2 | #
We then ran Line3D++ [START_REF] Hofer | Line3D++[END_REF], as defined in [START_REF] Hofer | Efficient 3D scene abstraction using line segments[END_REF], to detect and reconstruct 3D line segments.

Finally, we ran our plane detection and surface reconstruction, using a complete plane arrangement as baseline (see Sect. 4.7). Tab. 4.2 lists default parameters for all datasets. We often had to tweak 𝜎 𝑝 of Line3D++ to get decent input lines, and sometimes our 𝜆 edge = 𝜆 corner . Tab. 4.1 reports detection statistics.

observations on the input data

3D line segments detectors

There has been some recent approaches to adapt structure-from-motion (SfM) methods from points to line segments [START_REF] Zhang | Structure and motion from line correspondences: Representation, projection, initialization and sparse bundle adjustment[END_REF][START_REF] Salaün | Robust SfM with little image overlap[END_REF]. However, few methods focus on the actual production of 3D line segments [JKTS10, HMB16], possibly reconstructing more lines than just what the SfM algorithms would produces, by leveraging on the calibration to match or significantly augment matched lines. To our knowledge, only Line3D++ [START_REF] Hofer | Line3D++[END_REF][START_REF] Hofer | Efficient 3D scene abstraction using line segments[END_REF] provides code for this denser 3D line segment reconstruction.

In our experiments, we thus use Line3D++ to detect 3D lines segments from a set of images, and to provide as well sub-segments associated the each viewpoints.

4.2.2

Modest quality of input data from Line3D++

Although quite efficient, Line3D++ produces a somewhat noisy output on which we have little control:

] It can miss some important lines (see Artwork 1 (a)), which may lead to relevant planes not being detected.

] It also generate many outliers (see Artwork 1 (b)), which pressurizes the visibility and regularization terms.

] A number of actual, physical 3D lines (including shadows) are given multiple reconstructions (see Artwork 1 (c)).

These flaws are also visible on the line segments extracted on our datasets as seen on Figures 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8.

We expect any improvement on these aspects, for a 3D line segment detector to be used as a preliminary stage to our method (which is out of the scope of this work), to have a particularly positive impact on our results.

qualitative evaluation of reconstructions

Comparing to point-based reconstruction

To show the relevance of lines for scenes with little or no texture, in contrast to point-based methods (which are doubtlessly superior on textured scenes), we compare our method to a point-based piecewise-planar reconstruction [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF] on HouseInterior (cf. Fig. 4.8). Even when densely sampling point on the ground-truth surface as seen from the viewpoints, [CLP10] yields a reconstruction with missing details (e.g., the lounge table) due to missing primitives in hidden area (e.g., under the table). Moreover, [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF] uses a regularization that minimizes the reconstructed area, which is relevant for points uniformly sampled on the surface but strongly penalizes unsampled regions (e.g., invisible planes of lounge table). In contrast, our method leads to a better plane discovery and a reconstruction robust to non-uniform sampling. (We also tried reconstructing from points sampled on the 3D lines, but the result is poor; many planes are missed as points belong at most to one plane. As lines mostly lie on edges, the area cost also dominates the data term and

Comparing to other line-based reconstruction methods

As said above, there are very few reconstruction methods based on lines. [START_REF] Mentges | Surface reconstruction from image space adjacency of lines using breadth-first plane search[END_REF] mostly reconstructs a soup of planes, sometimes with adjacencies, but without any topological guarantee.

[WM14] provides a slightly more behaved mesh, but reconstructions still look messy and overly simple, although usable enough for robotic planning. No code nor data are available for comparing with either of these methods.

Our reconstruction with sparse data are illustrated on the datasets Andalusian (Figure 4.1), Delivery Area (Figure 4.2) and MeetingRoom (Figure 4.6). We also experimented on more textured datasets such as TimberFrame (Figure 4.4) and Barn (Figure 4.3) or less textured datasets such as HouseInterior (see Figure 4.8), and datasets with thin structural objects such as beams with Bridge (Figure 4.5).

Compared to usual point clouds, our 3D line clouds are extremely sparse. Despite the noise on inliers and the number of outliers due to Line3D++, our method is able to reconstruct a good approximation of the scenes, which illustrates the robustness of our approach. Still Barn shows that it is hard to reconstruct a sieve-like shape (balcony) due to the visibility lines traversing it.

quantitative evaluation of reconstructions

We performed quantitative evaluations of the quality of our reconstructions using our datasets with ground truth. In this section, we describe the metrics we used in our experiments, as well as our setting and principles when varying a parameter or input data to study the sensitivity of our method (in following sections).

Metrics to assess the quality of surface reconstruction

We used 4 metrics derived from the Metro distance.

To compare a reconstructed mesh 𝑀 recons with a ground-truth mesh 𝑀 gt , we first sample 2 millions of points on the surface of each mesh. We then compute the distances of each point of 𝑀 recons to their nearest neighbour in 𝑀 gt ; we note this set 𝐷 recons-→gt . Conversely, we also compute the set of distances 𝐷 gt-→recons .

The 4 metrics we use are the following:

] The max Metro distance measures the worst reconstruction error:

max(𝐷 recons→gt ⋃ 𝐷 gt→recons )
] The mean Metro distance measures the average reconstruction error:

mean(𝐷 recons→gt ⋃ 𝐷 gt→recons )
] The "95%-completeness" is the 95% percentile of 𝐷 gt→recons . It measures the distance under which most of the ground truth surface has been reconstructed.

] The "95%-precision" is the 95% percentile of 𝐷 recons→gt . It measures the distance under which most of what has been reconstructed is close enough to the ground truth.

We use these metrics to assess the quality of reconstructed surfaces as well as to define the value of our parameters (see following sections). Histograms of distances for HouseInterior are plotted on Fig. 4.9. Regarding precision, most of the points sampled on the reconstruction (91.4%) lie at less than 5 cm to the ground truth, showing that our RANSAC planes fit well the underlying surface and that our energy properly balances data fidelity and regularization. The error profile for completeness is similar, and 95% of the points on the ground truth are less than 8 cm to the reconstruction. It shows our regularization term do not over-smooth too much the surface by erasing details that would penalize completeness.

4.4.2

Varying parameters or input data

In the following sections, starting from the default parameter setting in Table 4.2, we vary the value of a chosen parameter (e.g., 𝜆 vis ) or the quantity of input data (e.g., the number of detected 3D line segments), and we plot a graph representing the impact on a quantitative assessment of the reconstruction.

For these experiments, we evaluate on the synthetic dataset HouseInterior, for which we have a ground truth. Although the results are specific to this dataset, we observed that the conclusions are relatively general. In particular, these variation studies on HouseInterior lead our choice of the best default parameter values (see Section 4.4.3), but we did not observe a strong need to alter this parameter setting when running on other datasets, although small changes could sometimes provide slightly better results.

Please note that although we may vary a parameter continuously, the labeling of the cells in the plane arrangement as full or empty is discrete and thus can lead to strong changes on the metrics when the altered cells are large or when the measure is based on a maximum distance. As a consequence, curves showing the impact of parameter variations can display significant discontinuities.

Also, due to the RANSAC stage, our algorithm is not deterministic. For each set of parameters, we actually ran our method 5 times, and we report in the graphs both the average value of the quantity we monitor over these 5 runs as well as its standard deviation.

Impact of Line3D++ parameters. The 3D line segments were obtained using the default parameter settings of Line3D++ [START_REF] Hofer | Line3D++[END_REF]. We tried playing around with the parameters but did not get significantly better segments. In particular, we studied the influence of 𝜎 𝑝 , a major regularization parameter of Line3D++, on the HouseInterior dataset. It is depicted on Figure 4.10. The default value of Line3D++ for 𝜎 𝑝 is 2.5, which is in a low plateau area of the graph.

Impact of the number of input images. We made a variant of the dataset MeetingRoom with 100 images and studied qualitatively the impact of providing a variable number of images as input (keeping a calibration based on all 100 images). Results are depicted by Figure 4.11 and compared to a point-based approach, namely Colmap [SF16] + Poisson [START_REF] Michael Kazhdan | Poisson surface reconstruction[END_REF] reconstruction. When going down from 100 images to 50 images only, the quality of our reconstruction is progressively reduced, but the general shape of the room as well as a number of details are preserved. In contrast, the point-based reconstruction with 100 images is filled with holes and degrades rapidly when the number of images decreases.

Impact of the number of 3D line segments. We randomly sampled 3D line segments produced by Line3D++. The influence of the number of lines segments as input to our method is illustrated on Figure 4.12. As can be expected, the more lines, the better (in general), as it leads to a larger diversity of possible reconstructions, with more details. The mean Metro distance plateaus to a small value around 700 lines. Quite naturally, the max Metro distance remains sensitive until a few thousands lines are provided. 95%-precision also plateau a bit after a thousand lines, while 95%-completeness also retains a limited sensitivity. The computation time of the visibility term however increases, although mostly linearly in the number of lines.

Parameter setting and sensitivity study

We strove to reduce as much as possible the number of parameters of our method. Still, it has a few parameters, that have to be set. In this section, we study how to assign a value to these parameters, either using a formal argument (for the number of RANSAC iterations 𝑁 iter ) or using an empirical justification (for the other parameters).

The parameter default setting is recalled in These parameters can be slightly adapted depending on the model. For instance, increasing 𝜆 vis will dig more into the volumes, but will also make the model more sensitive to outliers. Likewise, increasing 𝜆 corner and 𝜆 edge will lead to more regularization, which is useful when data are missing, but can also lead to a loss of details. In the following, we study the sensitivity of our method under different parameter settings, to discover ranges of parameter values leading to a good and relatively stable behavior. Table 4.2 resulted from this sensitivity study. Number of RANSAC iterations 𝑵 iter . Our choice for setting the value of the number of RANSAC iterations 𝑁 iter can be justified as follows.

The number of 3D line segments detected in the scenes of our datasets varies between 1,000 and 10,000, and the number of detected planes is in practice limited to about 150 (see Table 4.1). After the largest planes (with the most inliers) have been detected, the inlier rate of the current best plane can be quite low. Yet we must make sure that planes with a small line support are eventually detected, and first of all, sampled.

To define 𝑁 iter , we want to make sure at 𝛽 = 99% chance that we sample the best plane assuming it is supported by at least 𝛼 = 1% of the line segments in the current value of ℒ 0 ∪ ℒ 1 . If we call 𝑋 the event "not finding the best fitting plane after 𝑁 iter iterations among data which contain an inlier rate of 𝛼", its probability is:

𝑃 (𝑋) = (1 -𝛼 𝑘 ) 𝑁 iter
where 𝑘 = 2 is the number of line segment samples needed to generate a plane hypothesis. The criterion 𝑃 (𝑋) ≤ 1 -𝛽 in turn yields:

𝑁 iter ≤ log(1 -𝛽) log(1 -𝛼 𝑘 )
The right term evaluates to 46,049 with the given values for 𝛼 and 𝛽. Given that the number of candidate line segments actually decreases at each iteration, as structural lines are detected, this is a worst case analysis. In practice, in our experiments, we set 𝑁 iter = 50000. On a 12-core CPU, the whole RANSAC process (finding all planes) takes about 10 minutes for an upper bound of 10,000 lines with 50,000 iterations, which represents a minor fraction of the total reconstruction time.

Maximum number of detected planes 𝑵 max . We introduced a possible limit on the number of planes discovered by our RANSAC variant. The influence of that maximum number of detected planes 𝑁 max is shown on Figure 4.16.

As can naturally be expected, the more planes, the better. The mean Metro distance plateaus to a small value after 90 planes. So does the maximum Metro distance although with a small value but slightly larger variance. Both the 95%-precision and the 95%completeness also plateau to a small value after 90 planes.

The only drawback of adding more planes, in the case of a surface reconstruction based on a full-extent plane arrangement, is the increasing computation time, as adding a plane has a cubic time complexity.

Weight of the visibility term vis . The impact of varying the value of parameter 𝜆 vis is represented on Figure 4.13. We observe a plateau starting a bit before 10 -1 (our default parameter), which starts deteriorating after 10 1 and even more after 10 2 . As for the regularization parameters below, this observation qualitatively also applies to the other datasets.

Weight of the edge term edge . The impact of varying the value of parameter 𝜆 edge is represented on Figure 4.14. The error remains small when 𝜆 edge is under 10 -1 , in particular around value 10 -2 (our default parameter). As for 𝜆 edge , the error remains small when 𝜆 corner is under 10 -1 , in particular on the plateau around value 10 -2 (our default parameter).

Computation times

Although computing the visibility term is linear in the number of sub-segments, it is the most time-consuming part as it depends mostly on the number of cells in the plane arrangement, which is up to cubic in the number of planes. Time required for performing a whole reconstruction varies from 30 minutes (MeetingRoom) to 3 hours 30 minutes (TimberFrame). Creating the linear program from scene data takes more time than solving it.

Robustness of plane detection

To explore the robustness of our RANSAC formulation, we experimented with a toy example made of the 12 edges of a cube. We seek to extract the 6 planes associated to the 6 faces of the cube. We consider two types of perturbations: noise and outliers. The cube has an edge length of 2. We add noise to each segment endpoint, drawn from a uniform distribution with standard deviation ranging from 0 to 0.35. Outliers, from 0 to 50, are segments generated by uniformly picking pairs of points in a 2-radius ball. Finally for each couple (noise, #outliers), we report the number of planes that include the 4 edges of an actual face of the cube, using parameters 𝜖 = 0.06 and 𝑁 iter = 100, and averaging over 20 iterations.

Results are presented on Fig. 4.17. As expected, with a low level of perturbation, all planes are perfectly extracted. As the level of perturbation increases, for both noise and outliers, the rate of missed detections increases. Yet, even with a high level of noise, corresponding to a highly distorted cube (very non planar faces), we get a mean of 3.95 planes, which is a major failure. Last, we compared with a regularization using only corners or edges, which yields lower quality reconstructions.

ablation study

To show that all line specificities in our method are useful if not crucial, we performed an extensive ablation study. If one line supports at most one plane, as in an ordinary RANSAC framework, and if all lines supporting a detected plane are thus put aside before performing the following plane detection, as is the case in a greedy detection scheme, then fewer lines are available to detect succeeding planes, and less planes are detected. Experimentally, on HouseInterior, this detects only 45 planes instead of 120 with our method. The impact on reconstruction is illustrated on Figure 4.18: only the main planes are more or less reconstructed appropriately. If inliers are decided only from the distance to the model, rather than from the distance to the intersection of the plane model with the plane that already supports the inlier in case the line is structural, i.e., if we make no difference between lines in ℒ 0 and lines in ℒ 1 when gathering inliers for a given model (see Section 3.2), then the algorithm tends to associate a line twice with more or less the same plane. Experimentally, on HouseInterior, this detects only 92 planes, vs 120 with our algorithm. See Figure 4.19 for an illustration of the impact on the reconstruction.

If structural lines are treated as textural lines, i.e., if a structural line in ℒ 2 is simply treated as two textural lines in ℒ 1 (one for each supported plane), then reconstruction totally fails (see Figure 4.20). This illustrates the importance of distinguishing structural lines form textural lines, not only for plane detection but also for surface reconstruction.

If regularization penalizes only the length of edges, rather than only the number of corners, the reconstruction is not as good, as can be seen qualitatively on Figure 4.21. This fact has already been observed by [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF]: providing a good balance between 𝜆 edge and 𝜆 corner , the regularization on both corners and edges is slightly better.

If regularization penalizes only the number of corners, rather than both corners and edges, the reconstruction also is not as good (ground floor), although it can be locally better (armchair), as can be seen qualitatively on Figure 4.22. This fact has also already been observed by [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF].

If regularization also penalizes the surface area, as a measure of surface simplicity like in [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF][START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF], the metrics do not improve but rather tend to deteriorate, as can be seen on Figure 4.23. The fact is this term makes little sense on the kind of building scenes we are considering because we want to be able to reconstruct large surfaces with little data, that mostly lies on their boundaries only.

limitations and perspectives

The quality of 2D and 3D line segments at input (from Line3D++) is the main bottleneck of our method. Improving them would be very helpful. Mainly, it would be specially relevant to merge points and lines treatments into a single framework to offer a smooth transition from textured regions to textureless areas. Also, in our experiments, we used the full-extent plane arrangement, i.e., with planes extending all the way to the scene bounding box. This is not intrinsic to our method; it merely provides a baseline. Because of a cubic complexity in the number of planes, the acceptable number of planes is limited to a few hundreds, which is in practice often enough for a single room or the exterior of a building, but not enough for a complete BIM model. (It is also easy to keep the best few hundred planes after RANSAC detection to make sure the pipeline succeeds.) Yet, preliminary experiments with a coarse-to-fine approach show promising results for scaling to large scenes. In the cell complex, limiting the plane extent with a heuristic on a coarse voxel-based partition [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF] or the 3D adaptation [START_REF] Bauchet | Kinetic shape reconstruction[END_REF] of the 2D kinetic polygonal plane partitioning [START_REF] Bauchet | KIPPI: kinetic polygonal partitioning of images[END_REF] would also reduce the complexity.

Moreover, defining a notion of extent for line-detected planes, similar to 𝛼-shapes in the case of points [START_REF] Edelsbrunner | On the shape of a set of points in the plane[END_REF] but adapted to lines [START_REF] Van Kreveld | On the shape of a set of points and lines in the plane[END_REF][START_REF] Van Kreveld | Watertight scenes from urban LiDAR and planar surfaces[END_REF], could also be used to introduce so-called 'ghost planes', corresponding to unobserved, hidden planes at occluding edges of observed surfaces [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF][START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF].

Last, global regularization weights favor highly sampled surfaces. Adapting them to be more sensitive to weakly supported surfaces as in [START_REF] Jancosek | Multi-view reconstruction preserving weakly-supported surfaces[END_REF] could improve the results. They also need to be adjusted for each scene to perform the best reconstruction; thus, automatizing the parameters search would greatly improve the convenience of use of the pipeline.

conclusion

In this work, we proposed a surface reconstruction pipeline from images using 3D line segments. The whole approach takes into account the line set properties: small size, sparsity, density variations. We first presented a RANSAC approach designed for plane extraction in such line sets. It is robust to over segmentation of lines. Then, we adapted a surface reconstruction method using a plane arrangement to fit line specificities. In particular, we designed our data term for multiple viewpoints and insensitivity to dummy plane insertion. Finally, experiments on both synthetic and real datasets show the practical interest and robustness of our method. Future work also includes finding means to better normalize the regularization parameters, although we believe there will still be some dependency on the type of scene, in particular depending on whether the scene is inside or outside. More general perspectives include developing a general framework accepting both 3D points and line segments and being able to smoothly adapt from point-based paradigm on textured scenes [START_REF] Chauve | Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data[END_REF] with proper regularization [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF], to line-based reconstruction on textureless areas.

One of the main perspective is to create a unified approach to deal with heterogeneous line/point clouds. It would involve developing a new criterion for plane extraction and reformulating the energy for surface extraction to be well balanced between lines and points.

Transition

In this part, we have used a direct method to perform 3D reconstruction of buildings. We tackled specific issues met in building environments and paved the way towards an efficient algorithm. However, as discussed, limitations remain. In particular, some parameters must be adapted for each scene, and there is no semantic information in the output. Both problems require strong and complex priors to be solved. With the growing amount of data publicly available online and the generalization of machine learning techniques throughout all computer vision tasks, a promising way to build strong priors is to directly learn them from the data. In the next part, we conduct experiments on a dataset of objects, in order to study how one can learn reconstruction priors and incorporate geometric constraints to enhance the overall resulting mesh.

Part III 3D RECONSTRUCTION BY PARAMETERIZED SURFACE MAPPING

Introduction

[ Chapter 4

Chapter 6 ]

Synopsis We introduce an approach for computing a 3D mesh from one or more views of an object by establishing dense correspondences between pixels in one or different views and 3D locations on a learnable parameterized surface. Our insight is to optimize a novel geometric cycle-consistency loss between the learnable parameterized surface and input views over an embedded representation of the target shape, allowing the reconstruction of a 3D shape that closely aligns with its depiction in one or more views. In addition, we introduce a multi-view shape encoder that can jointly be trained with the AtlasNet surface parameterization. We demonstrate the efficacy of our approach on the ShapeNet-COCO dataset and show that our approach outperforms XNOCS [SRV + 19] when averaging over three shape categories. Reconstructing a shape from one or more views is a long-standing problem in computer vision [SCD + 06]. When we, as humans, are relatively far away from an object where binocular parallax is negligible, we are nonetheless able to infer a representation of its geometry, including its hidden faces, and as we move closer, we are able to incorporate information from the new viewpoints and from both eyes. An ideal system should be able to reconstruct a shape, including inferring hidden faces, from a single view and update the reconstruction as more image evidence is gathered. To achieve this goal, a system must integrate information from a learned prior for the shape (single-view reconstruction) with geometric constraints when more than one view is available (multiview reconstruction). This goal is important for scanning, augmented reality, and robotic navigation and manipulation applications.

learning 3d reconstruction

For shape reconstruction, we argue that the choice of shape representation is important. In this work, we seek to output a surface-parameterized mesh reconstruction of a shape from one or more input views. As we will show, the parameterized surface representation provides a natural medium for establishing dense correspondences across views and allows leveraging recent work that learns surface-parameterized shape priors [GFK + 18]. Moreover, the representation lends itself to rasterization, which makes it possible to reason about contours. In contrast, other shape representations, such as points, voxels, and implicit surfaces, do not easily offer such benefits.

Classical multi-view 3D reconstructions approaches establish dense correspondences between pixels or image regions depicting the same surface locations, followed by triangulation to yield a 3D shape [SCD + 06]. However, these approaches fail to find correspondences in flat, non-textured regions, and they cannot extrapolate to reconstruct nonvisible/occluded surfaces. In addition, they typically yield point clouds, whereas meshes are more useful in most real-world downstream applications.

Incorporating data-driven shape priors into the 3D reconstruction process has been used to address the above limitations [GFK + 18, PFS + 19]. However, an ideal shape prior should be able to adapt in order to finely align with its depiction in the input views [LWR + 19]. Furthermore, the prior should not require manual annotation of the surface parameterization, i.e., landmarks of shape parts, and should be learnable from training data.

To address the above issues, we propose a method that learns to establish dense correspondences between image pixels and a learned parameterized surface, which is trained on a corpus of shapes from the same class, without known surface parameterizations. The approach has two steps. First, as opposed to the method of Kulkarni et al. [START_REF] Kulkarni | Canonical surface mapping via geometric cycle consistency[END_REF] that uses a fixed template, we generate an initial shape template with a multi-view shape encoder, which extends AtlasNet [GFK + 18] to multiple viewpoints. Second, this template, specific to the shape we want to reconstruct, is refined using a cycle-consistency loss which allows the shape to adapt and finely align with its depiction in the input views.

We evaluate our method compared to a recent state-of-the-art approach for 3D reconstruction from multiple views [SRV + 19]. We show that our method outperforms the baselines when averaging over three shape categories. Moreover, as opposed to XNOCS [SRV + 19] and traditional multi-view stereo methods [SCD + 06], our shape parameterization naturally yields a mesh that can be used out-of-the-box for most graphics applications.

related work

Single-view reconstruction. Single-view scene reconstruction is a classic problem in computer vision. Most successful method learn 3D orientation priors over images or image patches that are trained for example by supervised learning [START_REF] Saxena | Make3D: Learning 3D scene structure from a single still image[END_REF][START_REF] Eigen | Depth map prediction from a single image using a multi-scale deep network[END_REF].

Recently, methods have tried to relax the explicit 3D supervision requirement by learning class-specific reconstruction priors over, for example, a deformable mesh [START_REF] Kanazawa | Learning categoryspecific mesh reconstruction from image collections[END_REF]. This approach generates impressive textured reconstructions from a single image, but requires extra supervision in the form of keypoints in correspondence during training, whereas we learn these as well.

More recently, Canonical Surface Mapping (CSM) [START_REF] Kulkarni | Canonical surface mapping via geometric cycle consistency[END_REF] predicts UV mapping from a single image onto a canonical model, trained entirely using self-supervision (via the images themselves) by introducing a geometric cycle-consistency term. Our work is inspired by this, but differs in two key ways. First, we do not assume a fixed canonical model, rather we optimize over a learned surface parameterization to better match the observed data. Second, CSM works only for single views, whereas we develop a multi-view cycle-consistency term to better integrate multi-view information when available. This is critical when reconstructing both the mapping and the underlying model jointly, due to the ill-posed nature of the problem. Data-driven multi-view reconstruction. When given multiple views, it is possible to infer scene geometry by finding correspondences subject to geometric, e.g., epipolar constraints. Classical approaches such as multi-view stereo (MVS) [SCD + 06] are powerful and well established tools with many uses. However, MVS fails in cases where correspondences are ambiguous, e.g., occluded surfaces, texture-less regions, and view-dependent effects. To overcome these limitations, these methods often employ weak smoothness priors. Recently, data-driven methods have been introduced that attempt to extend MVS to leverage much stronger priors to deal with ambiguities [KHM17, HMK + 18]. Our work differs from these works in that we reconstruct a mesh rather than a point cloud, which typically requires stronger data priors, as seen in the single-view setting.

Other approaches for learning mesh reconstruction rely on manual alignment between the cameras and shape generator and optimize a photometric loss, which is not robust to illumination changes [LWR + 19]. Alternately, Pixel2Mesh++ [START_REF] Wen | Pixel2Mesh++: Multi-view 3D mesh generation via deformation[END_REF] presents a feedforward neural network that maps from input views and a base sphere mesh to an output adapted mesh. However, this approach differs fundamentally in that it does not establish a dense surface mapping from pixels to the reconstructed mesh, which is required for fine scale alignment.

More recently, a number of methods for multi-view novel view synthesis have proposed eliminating the explicit mesh reconstruction altogether and instead learn features, for example, in a voxel grid [STH + 19, LSS + 19] or a set of orthogonal planes spaced in 3D [ZTF + 18]. These approaches demonstrate strong view synthesis results, but cannot be used in cases where mesh reconstructions are still required.

Method [ Chapter 5

Chapter 7 ]

learning a multi-view parameterized surface mapping

We consider a shape 𝑆 and a collection 𝒱 𝑆 of views of 𝑆, where a view 𝑣 = (𝐼, 𝜋) ∈ 𝒱 𝑆 consists of an image 𝐼 with associated camera intrinsics and extrinsics 𝜋. We seek to output a surface mesh ℳ of the shape in object coordinates. We assume that a template 𝒯 parameterizes the output surface mesh and that the mapping 𝒱 𝑆 → 𝒯 → ℳ produces the output. Our goal is to learn this parameterized surface mapping from image pixel locations to the output surface. For this, we present a two-step approach, depicted in Figure 6.1. The first step (initialization) aims at estimating an initial template, represented by a latent vector z S and the weights of an AtlasNet decoder 𝜙 [GFK + 18]. The second step (optimization) improves the shape representation using a cycle-consistency loss.

Initialization

Multi-view shape encoder. To compute the initial latent shape feature z S , we propose a multi-view shape encoder ℰ jointly trained with the shape parameterization 𝜙, cf., Fig. 6.2.

For each view 𝑣 = (𝐼, 𝜋) ∈ 𝒱 𝑆 , the image 𝐼 is encoded using a convolutional neural network 𝒞. Now the resulting features are expressed in a coordinate system linked to the view. Therefore, we apply a neural network ℛ that transforms the encoded representation with respect to the view camera parameters 𝜋 in a representation expressed in the canonical pose. The multi-view encoder ℰ aggregates by average pooling the features extracted from each view. The resulting representation ensures equivariance with respect to the camera parameters and can handle a variable number of input views. Denoting |𝒱 𝑆 | the size of the view collection 𝒱 𝑆 , the output of our multi-view shape encoder ℰ is the average of encoded transformed views:

z S = ℰ(𝒱 𝑆 ) = 1 |𝒱 𝑆 | ∑ (𝐼,𝜋)∈𝒱 𝑆 ℛ(𝒞(𝐼), 𝜋). (6.1)
We assume here a spherical parameterization of a shape collection 𝒮: the template 𝒯 we consider is a sphere, which covers the range of genus-zero shapes. We learn our parameterization using an AtlasNet decoder 𝜙. To do so, we need to establish correspondences between each shape 𝑆 ∈ 𝒮 in the collection. We achieve this goal by establishing per-shape correspondences 𝒬 𝑆 via the template 𝒯. While there are a variety of spherical parameterizations of a shape [START_REF] Praun | Spherical parametrization and remeshing[END_REF], we use a gnomonic projection due to its simplicity. To assign correspondences between the points at the surface of a 3D object and a sphere centered on the object centroid, the gnomonic mapping consists in associating each point on the object to its central projection on the sphere (see Figure 6.3).

We jointly optimize the parameters of ℰ and 𝜙 using the shape prior loss ℒ shape as the sum of 𝐿 1 losses over shapes 𝑆 ∈ 𝒮 and established 3D point correspondences (q, q ′ ) ∈ 𝒬 𝑆 between the template 𝒯 and the shape 𝑆:

ℒ shape (𝜙, 𝒮) = ∑ 𝑆∈𝒮 ∑ (q,q ′ )∈𝒬 𝑆 ∥𝜙 ℰ(𝒱 𝑆 ) (q) -q ′ ∥ 1 (6.2)
Training procedure.

Given a dataset of 3D shapes, we render multiple views and jointly train the parameters for the multi-view shape encoder ℰ and the parameterization 𝜙. We follow the network architectures and two-stage training procedure described in Groueix et al. [GFK + 18] to train a single-view model comprising the parameterization 𝜙 and single-view encoder 𝒞 using loss ℒ shape . We then jointly train the multi-view shape encoder and parameterization by initializing the parameters for 𝜙 and 𝒞 from the single-view training step.

In our implementation, 𝒞 is a ResNet-18 [START_REF] He | Deep residual learning for image recognition[END_REF], ℛ is a multi-layer perceptron with two layers and the AtlasNet decoder 𝜙 is a multi-layer perceptron with four layers.

Optimization

Given the estimated shape produced at the initialization, it is possible to further improve the reconstruction. Similar to CSM [START_REF] Kulkarni | Canonical surface mapping via geometric cycle consistency[END_REF], we employ cycle-consistency in our reconstruction loss to learn the parametric surface mapping: after mapping a pixel location p in an image to the surface mesh via the template, it should project back to the original location p. We further encourage that the projected point lies within a segmentation mask 𝒳 𝑣 of the depicted shape, in each view 𝑣 (see Figure 6.5). The segmentation mask can be obtained with an off-the-shelf instance segmentation algorithm [START_REF] He | Mask R-CNN[END_REF].

The cycle-consistency loss is graphically described on Figure 6.4. Let 𝑣 = (𝐼, 𝜋) be a view in collection 𝒱 𝑆 and ℱ 𝑣 ∶ R 2 → 𝒯 be a learnable mapping from a 2D pixel location p in image 𝐼 to a point on template 𝒯. We note 𝒫 𝜋 ∶ R 3 → R 2 the function projecting a 3D point on the image plane and define the cycle projection function

𝒦 ∶ R 2 → R 2 as 𝒦(p) = 𝒫 𝜋 ∘ 𝜙 z S ∘ ℱ 𝑣 (p).
Our reconstruction loss is the sum of squared-reprojection errors and squared-Chamfer distances to the segmentation mask over all pixels, weighted by 𝜆 = 0.25:

ℒ rec (ℱ 𝑣 , 𝜙 z S ) = ∑ p∈𝒳 𝑣 (‖𝒦(p)-p‖ 2 2 + 𝜆 min p ′ ∈𝒳 𝑣 ‖𝒦(p)-p ′ ‖ 2 2 ) (6.
3) Shape refinement procedure. We initialize the latent vector z S and 𝜙 with the weights obtained at the initialization step. In our implementation, the mapping ℱ 𝑣 is a 4-level UNet [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF] with a ResNet backbone. ℱ 𝑣 has 3 output dimensions which are normalized in order to represent a UVW mapping on the template sphere 𝒯. To properly initialize the cycles, the UNet weights are set by training to reconstruct the UVW rendering of the initial shape. Once the pipeline is initialized, we optimize both the UNet and the decoder using ℒ rec .

Implementation details

We optimize our full loss ℒ rec in stages, starting with the shape prior loss ℒ shape . Given a dataset of 3D shapes, we render multiple views and jointly train the parameters for the multi-view shape encoder ℰ and the parameterization 𝜙. We follow the network architectures and two-stage training procedure described in Groueix et al. [GFK + 18] to train a single-view model comprising the parameterization 𝜙 and single-view encoder 𝒞 using loss ℒ shape . We then jointly train the multi-view shape encoder and parameterization by initializing the parameters for 𝜙 and 𝒞 from the single-view training step. We assume that the relative poses (extrinsic parameters) and camera calibration (intrinsic parameters) for the input views are known. These can be either given a priori (e.g., assuming precalibrated cameras and IMU-based SLAM from a smartphone), or computed from the data by traditional structure-from-motion methods [START_REF] Lutz | Structure-from-Motion revisited[END_REF]. We optimize the shape prior loss ℒ shape between the ground truth and predicted shapes using the Adam optimization with a learning rate of 1e-3. Given the trained parameterization 𝜙 and multi-view shape encoder ℰ, we optimize the reconstruction loss ℒ rec over all views 𝒱. First, we set the latent vector to be the output of the multi-view shape encoder applied to the input image collection z S ← ℰ(𝒱). Second, we initialize the ℱ by making a UVW rendering of the initial mesh (each pixel contains the UVW coordinate of the corresponding point on the mesh), and by optimizing the L2 loss between this rendering and the ℱ output over the ℱ parameterization for 30 iterations. This allows the initial pixel-to-UVW mapping to be consistent with the initial mesh. Next, we optimize the reconstruction loss ℒ rec over the mapping ℱ and parameterization 𝜙 for 220 more iterations (the latent vector z S stays fixed during this optimization).

Pixel sampling method. At inference time, we randomly sample 2500 pixels per mask per iteration to prevent memory overflow.

Network architectures. In the multi-view shape, the convolutional neural network encoder 𝒞 is a ResNet-18 [START_REF] He | Deep residual learning for image recognition[END_REF] (yellow blocks in Figure 6.2). Latent spatial transformation network (LSTN, orange blocks in Figure 6.2) is a multi-layer perceptron with two layers. Finally, the AtlasNet decoder is a multi-layer perception with four layers.

The mapping ℱ, which maps the image space to the template space, is implemented with a 4-level UNet [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF].

Optimization parameters. We optimize the loss via Adam over mini-batches comprised of sampled pixels from each view image. We initialize the UNet with random weights and we first optimize solely over the UNet's weights for 300 iterations with learning rate 1e-4 in order to properly initialize it. Then, the UNet weights as well as the decoder's weights are jointly optimized for 2200 more epochs with a learning rate of 1e-5.

design choices

Template choice

The AtlasNet [GFK + 18] framework allows the use of different templates to deform. The default choice is an atlas: a set of (typically 25) planar primitives. Unfortunately, this is incompatible with our UNet-based architecture for the pixel-to-UVW mapping. Indeed, mapping an input pixel to a 3D point on the atlas involves choosing on which of the 25 primitives we want to map the pixel on. This is a hard (discrete) choice, and thus not naturally back-propagable. One could imagine making a soft choice (i.e., predicting a distribution of probability of the point presence on each primitive), but this involve important changes in the architecture, the loss and introduces uncertainty about the convergence of the optimization. We first made experiments with a single plane as a primitive template. While this allowed to efficiently develop the pipeline, limitations soon appeared. As reported in the original AtlasNet paper, the raw performance of the single plane primitive is very limited. Moreover, since our procedure continuously optimizes the pixel to template position, the pixel closely mapped to the plane boundaries have a clipped gradient in order not to move past this border. This introduces local minima in the optimization and caused failure cases.

It was consequently natural to use a borderless primitive such as a sphere. Mapping a pixel to this template involves no choice and no gradient clipping.

6.2.2

Using the UVW parameterization When using the sphere as a template, we can use the following classical UV-mapping:

𝑢 =0.5 + 𝑎𝑟𝑐𝑡𝑎𝑛2(𝑑 𝑥 , 𝑑 𝑧 ) 2Π (6.4) 𝑣 =0.5 - 𝑎𝑟𝑐𝑠𝑖𝑛(𝑑 𝑦 ) 𝑃 𝑖 (6.5)
where (𝑑 𝑥 , 𝑑 𝑦 , 𝑑 𝑧 ) represents any point on the unit sphere. This parameterization is similar to the latitude/longitude system on earth and as such, it presents a discontinuity line (akin to the international date line) on the U coordinate as represented on Figure 6.6.

As our algorithm maps pixels to a position on the sphere, this is an issue, because the points on the sphere cannot cross the discontinuity line during the optimization. We therefore use 3 output channels instead of 2 on our UNet and use a simple normalization to obtain a point on the sphere. While this parameterization does not involve any discontinuity, the risk is for the norm of the output channel to either vanish or diverge. In practice, we did not notice such a phenomenon.

Optimized parameters

The authors of [LWR + 19] only used the latent vector as an optimization parameter. This assumes that the latent space build at initialization time (see subsection 6.1.1) is expressive enough. In order to check this assumption, we took a trained decoder, generated a latent model thanks to the first input image of a model, and optimized the decoder output against the ground truth with the Chamfer distance. We tried optimizing over the latent U coordinate V coordinate Figure 6.6: Rendering of the UV coordinates (Black: 0, White: 1) for a unit sphere vector on the first hand, and over the decoder's weights on the other hand (see Figure 6.7). It turned out that recovering small details and achieving a very high F-score was not possible when optimizing only over the latent vector, hence our choice to optimize over the decoder's weights.

Positional encoding

The AtlasNet decoder is a multi-layer perceptron modeling a deformation of a surface. Recently, the positional encoding method has been introduced to help this kind of network to model finer details [MST + 20]. This method consists in providing as input to the network a frequency decomposition of the coordinates. In our case, while this improves the loss during training, it also decorelates the points on the template surface. As shown by Figure 6.8, even if the point cloud looks correct, the wireframe of the mesh shows a lot of foldings and self-intersections. In our case we therefore concluded that it was better to preserve the ability of neighbour points to keep being correlated during the update of the decoder weights.

Regularization

Foldings and self-intersections are bad in general: first, they usually cause additional troubles in downstream applications; second, our optimization process is based on the assumption that we can continuously deform the template to the target shape. When reaching a deformation state that involves self-intersection, this generates discontinuities in the pixel to UV mapping which make the optimization even harder. In order to avoid these issues, we have tried introducing regularization losses for the Chamfer loss:

] Edge regularization: L2 penalization of the distance between neighbour vertices.

] Normal consistency: cosine distance between the normals of each pair of points which share an edge.

] Laplacian regularization: we add the Laplacian of the transformation to encourage neighbour points on the sphere to also be neighbours on the target surface.

Input point cloud Wireframe of the mesh This approach has several drawbacks: first, it is hard to find the weights for the regularization terms; second, these regularization terms tend to prevent the model from learning details in the priors. In the end, we obtained better qualitative and quantitative results by introducing the gnomonic distance as a learning loss for the optimization step (see 6.1.1).

Results

[ Chapter 6

Chapter 8 ]

We report qualitative and quantitative performance compared to the baselines. We consider the task of reconstructing a depicted shape given a set of views with known camera and object segmentation mask for each view.

dataset

For our controlled setup, we evaluated on the publicly available ShapeNetCOCO dataset [SRV + 19], which has rendered views for three shape categories ("airplane", "car", "chair") from ShapeNet [CFG + 15]. We used the published training and validation splits containing 11,469 and 2,868 shape instances, respectively. We evaluated on 100 random shape instances belonging to the validation split for each shape category (300 instances in total). The dataset includes for most shape instances 20 rendered views with widely varying camera distances from the target shape, with the rendered views composited on single still images from the COCO dataset [LMB + 14]. Note that the rendered views in the ShapeNetCOCO dataset are more challenging than rendered views from Choy et al.

[CXG + 16] due to the composited natural image background and larger distances from the camera's viewpoint to the shape. We also found that the camera calibration was not accurate for the Choy et al. [CXG + 16] rendered views, which limits the accuracy of the reconstructions. Note that the ShapeNetCOCO dataset has world origin at the shape instances' centroid. Finally, we use the ground truth segmentation masks provided with the dataset.

evaluation criteria

We evaluate the quality of the output reconstructions by reporting the F1-score (F1) and symmetric Chamfer distance (CD). Following Wang et al. [START_REF] Wen | Pixel2Mesh++: Multi-view 3D mesh generation via deformation[END_REF], we apply their method for uniformly sampling points on the surface of the ground-truth shapes, yielding 10k points per shape. Following Wen et al. [SRV + 19], we sample 10k points from the output shapes. Given a threshold 𝜏 which is 1% of the ground-truth mesh bounding box diagonal, we compute the fraction of output points that are within 𝜏 of a ground-truth point (precision) and vice versa (recall). We follow Knapitsch et al. [START_REF] Knapitsch | Tanks and temples: Benchmarking large-scale scene reconstruction[END_REF] and report the F1-score, which is the harmonic mean of precision and recall. We follow

experimental results

Optimization process

We represent on Figure 7.2 what happens during the optimization process of ℒ rec on an example shape from the airplane category. The step count starts after the UNet has been initialized (see 6.1.3 for the detailed procedure). As expected, we notice that the mesh gets closer to the ground truth as optimization evolves. Moreover, the wings on the initial shape continuously move and deform towards their final position, which tends to show that the priors obtained at initialization are correctly exploited.

Baselines

We compare against XNOCS [SRV + 19] with multi-view aggregation of 5 images and with separate networks trained for each shape category. To be comparable, we train our approach by randomly sampling (only) five images for the input views.

We also compare against a fixed single-template baseline. For this baseline, we randomly choose a template shape for each shape class and compare the chosen template against the validation set. (The shapes are oriented and scaled consistently across the dataset.) We report the average across ten randomly selected templates for each category. For a particular selected template, this baseline is an upper bound for CSM [START_REF] Kulkarni | Canonical surface mapping via geometric cycle consistency[END_REF].

7.3.3

Results We train our method (Our full model) for each shape category separately. We report results in Table 7.1. Averaging over the three shape categories, our approach outperforms the baselines for both the F1-score and the Chamfer distance criteria although we solve a harder task, i.e., meshing rather than point generation. Note that the template baseline performs surprisingly well. In fact, as noted in Tatarchenko et al. [TRR + 19], there is large overlap of the shapes in the train/val splits. We show qualitative results in Figure 7.1, 7.3, 7.4, 7.5 and 7.6. On Figure 7.1, we also show the whole sequence of input images from which the model was created.

The point clouds generated by XNOCS are very noisy. We observe that creating a mesh from the output point clouds may be difficult, while our approach produces meshes directly.

Our method is able to recover specific shape details of the reconstructed shape (see the reactors on Figure 7.1 and 7.3). In the airplane (Figure 7.3) and car (Figure 7.4) categories, we notice that many models share similar shapes with very slight differences (e.g., car trunk shape, position of the airplane reactors). Our optimization step allows to more precisely recover these details. We notice that the chair category is harder to properly reconstruct. This is due to the fact that the transformation between a sphere and a chair requires additional distortion compared to the other categories, e.g., to create thin legs. Moreover, the models in this category include non genus 0 objects, which cannot perfectly be reconstructed by a sphere. Therefore, optimization is harder and can more easily get trapped into local minima.

ablation study

The results of an ablation study are presented in Table 7.2. We consider the method with or without the optimization step. We also consider using a Chamfer loss rather than the gnomonic loss.

Impact of the optimization stage

We see that the multi-view encoder performs reasonably well, in particular on planes and cars, but it is not sufficient to reach the state-of-the-art. It is however a good prior in the full setting: the model with optimization performs better than the multi-view encoder alone on every categories and metrics (from 16 to 24 points), given as input the estimated shape produced at the initialization step.

Chamfer loss vs gnomonic loss

We compare our ℒ shape loss defined with a gnomonic projection of the sphere to the Chamfer distance loss, that uses closest points to establish correspondences [GFK + 18], which often leads to non-smooth and non-injective mappings with local self-intersections and foldovers. We first optimize the multi-view encoder and the AltlasNet decoder (MV encoder) and observe that the two models obtain similar scores. It seems that both losses are suitable for the task. However, taking the next step (Full model), we can observe that using the gnomonic projection leads to a higher F1-score. Our interpretation is that this is due the non-smooth surface reconstructed with the Chamfer loss. We show in Figure 7.7 two meshes reconstructed after MV-encoder training, with Chamfer distance (left) and gnomonic projection (right). The first model seems to better fit the plane shape, but has a lot of self-intersections (each color discontinuity) while the second model is very smooth. This smoothness is, in practice, a better starting point for the reconstruction optimization.

Chamfer loss Gnomonic mapping loss

discussion and limitations

While our approach outperforms the baselines, we found several limitations. First, while the gnomonic projection outperforms the Chamfer loss, there are still remaining reconstruction artifacts when mapping from a sphere to a target shape to due the lack of bijectivity; most shapes are not star-shaped volumes, even simple cars. Second, the spherical template shape restricts the outputs to the set of genus-zero shapes, which does not allow us to reconstruct well shapes with arbitrary topologies, which is the case of many chairs (cf. Figure 7.1). In fact, both issues are present in chairs which are harder to reconstruct for XNOCS too.

conclusion

To wind up, we have presented a method to learn a dense UVW mapping of a parameterizable shape template. We introduce a multi-cycle loss to resolve ambiguities inherent in single image shape and UVW prediction, and show that our method outperforms a recent state-of-the-art approach. In addition, our method is able to reconstruct a mesh, rather than a point cloud, which makes it more useful for downstream 3D applications.

An interesting research direction is to use UVWs predicted from multiple images along with traditional projection-based techniques to optimize for the final mesh textures.

Transition

In this part, we studied how machine learning can enhance the reconstruction task by incorporating meaningful priors into the reconstruction process. Thanks to our method, we are able to generate meshes out of low resolution calibrated images representing small objects with little to no texture. In the next part, instead of pictures, we are interested in point clouds obtained by LIDAR sensors, and we aim at inferring semantics in addition to a volume reconstruction. While the template deformation approach is promising, there are inherent difficulties to adapt it in the context of buildings: the genus of a building can be high; the deformation from a simple template is greater than in the case of objects and this 3D representation does not naturally handle multiple labels. In part II, we used a partition of the 3D space which, as we will see, more easily handles multiple labels. In the next part, we strive to use both machine learning and a partition of space to achieve both 3D volume reconstruction and semantics inference by focusing on the main obstacle: the lack of relevant data to learn from for this task. Additionally, we propose several methods to jointly reconstruct both geometry and semantics and evaluate on the test set of the dataset. We show that the proposed dataset is challenging enough to stimulate research.

Introduction

3d reconstruction for buildings

Monitoring the life cycle of existing buildings is of great interest for the building industry.

In this context, being able to reconstruct a Building Information Model (i.e., a digital representation) of the current state of a building is required [START_REF] Thomson | Automatic geometry generation from point clouds for BIM[END_REF]. It implies reconstructing both the volume geometry and the semantics of the building.

In industrial contexts, the first step to perform model reconstruction is often to capture a 3D scan using LIDAR technology. This type of sensors allows precise depth measurements and therefore provides high-quality geometric features. Moreover, normals are efficiently estimated on such raw data [START_REF] Boulch | Fast and robust normal estimation for point clouds with sharp features[END_REF]. However, LIDAR sensors suffer from inherent flaws: there often are missing measurements due to mirrors or surfaces which are transparent to the LIDAR light frequency. Moreover, measurements are usually taken from a fixed point of view which generates a non uniform sampling with respect to the underlying surface. Finally, and as for every sensors, some surfaces may be completely missing from the input data because they were not seen from any point of view. Reconstructing building parts from partial data involves the use of strong priors. This motivates the use of machine learning based approaches to learn these priors directly from existing data.

In this context, many approaches have been developed for 3D semantic segmentation for point clouds [BPM20, QSMG17 One of the main reasons is the lack of suitable datasets. The existing datasets usually tackle one task or the other, and datasets containing both geometric and semantic infor-mation are focused on objects (i.e., furniture) rather than building components. Therefore, these components are usually presented as open surfaces instead of closed instance volumes which is required for digital building reconstruction.

This work intends to help filling the gap toward full digital mockup reconstruction both in terms of dataset and methodology. We introduce a new dataset called VASAD, for Volume And Semantic Architectural Dataset. It is composed of 6 full building models, with volume description and semantic labels of each building component. It represents approximately 62,000 m 2 of building floors, making it large enough for the development and evaluation of learning-based approaches. The objective of this synthetic dataset is to leverage machine learning techniques for architectural reconstruction, regarding both geometry and semantics.

Second, we present a deep neural network for joint semantic and geometric reconstruction. It is built over of a semantic feature extractor followed by a dense voxel-reconstruction network. Quantitative evaluations on VASAD show that it outperforms the state-of-theart reconstruction method [PNM + 20] that we also modified for semantic reconstruction purposes.

related work

Surface reconstruction Reconstructing surfaces has been part of the earliest tasks that the computer vision community has tried to address. Assuming that the point clouds are directly sampled on the underlying surface, efficient methods [BMR + 99] allow mesh reconstruction. However, the point clouds are usually noisy and a smoothness prior can help handling outliers [START_REF] Michael Kazhdan | Poisson surface reconstruction[END_REF]. In the context of LIDAR scans in man-made environments, piecewise-planar reconstruction provides an idealized model of the scene [START_REF] Boulch | Piecewise-planar 3D reconstruction with edge and corner regularization[END_REF][START_REF] Nan | Polyfit: Polygonal surface reconstruction from point clouds[END_REF] or an hybrid idealized/free-form model [START_REF] Lafarge | Surface reconstruction through point set structuring[END_REF]. [START_REF] Josep | Unsupervised robust planar segmentation of terrestrial laser scanner point clouds based on fuzzy clustering methods[END_REF] or a graph-cut [START_REF] Golovinskiy | Min-cut based segmentation of point clouds[END_REF] optimization. These different formulations allow the user to hand-craft priors they have on their data: [START_REF] Schnabel | Efficient RANSAC for point-cloud shape detection[END_REF] is efficient to recover piecewise-planar regions of a point cloud, and it is robust to noise; [START_REF] Golovinskiy | Min-cut based segmentation of point clouds[END_REF] allows to use local properties as descriptors and to use pairwise relationships as a propagation prior. While these methods can be very effective in particular cases, they fail at handling classes which involve complex priors. Again, recent methods leverage deep learning to directly learn complex priors on massively available data. Pioneering work in this direction includes PointNet [START_REF] Ruizhongtai | PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation[END_REF], a simple multi-layer perceptron which leverages the pooling operator to approximate any continuous order-invariant function on a set of point coordinates with features. Followup work [START_REF] Ruizhongtai | PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space[END_REF] focused on building a local analysis to improve performance, and progressively, proposals have been made to generalize the convolution operator to point clouds [LBS + 18, TQD + 19, WQF19, BPM20], pushing further the number of handled classes, the complexity of the detected classes, and the size of the handled point clouds. While these methods rely almost purely on point geometry, RGB-D sensors allow to jointly leverage photometric and geometric information to enhance the segmentation performance [START_REF] Dai | 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation[END_REF].

Joint reconstruction and semantic segmentation The intuition that both tasks of reconstruction and semantic segmentation can leverage each other becomes more and more popular. Pioneering work in this direction [SYZ + 17] simultaneously complete and label voxels obtained from a single depth image with full supervision. Recent work try to directly leverage the synergy between both tasks in a single-view setup by affinity learning [ZCX + 19], by supervising the reconstruction task thanks to a pre-trained segmentation network [GHL + 20], by introducing a content consistency constraint [START_REF] Chen | Towards scene understanding: Unsupervised monocular depth estimation with semantic-aware representation[END_REF] or by independently estimating the geometry on the segmented classes and adding a merging step [AAB19, WZW + 20].

Implicit representation is a major tool to achieve both tasks, because it allows to formulate the problem as an optimization task over the whole 3D space whose data term can be hand-crafted [START_REF] Häne | An overview of recent progress in volumetric semantic 3d reconstruction[END_REF] or learned [CSO + 18]. In this context, some research projects began to get interested in buildings [DRB + 18], but essentially stayed at the scale of a room and its furniture, and the lack of data has prevented researchers to address the reconstruction of the full structure of a building with semantics.

Need for a joint semantic/volume dataset Existing datasets mainly focus on a single task which is either surface reconstruction or semantic segmentation.

Semantic segmentation is usually handled at point level on various point cloud types such as car-embedded LIDARs, e.g., SemanticKITTI [GLU12 Surface reconstruction aims at reconstructing a 2-manifold based on partial view of the scene (either point cloud or images). Multiple datasets exist for this task, including ShapeNet [CFG + 15], DFaust [START_REF] Bogo | Dynamic FAUST: Registering human bodies in motion[END_REF] or MatterPort3D [CDF + 17]. In the most general case, this task does not impose to reconstruct closed surfaces. However, several approaches, including implicit function estimation, tackle the task by learning a dense function over the whole 3D space which labels any 3D point as inside or outside; this function is learned using closed shapes, (volumes) as a supervision.

Volume datasets are often obtained after a heavy pre-processing step for closing shapes. In [MON + 19, PNM + 20], the shapes of ShapeNet are closed using the marching cubes algorithm [START_REF] Lorensen | Marching cubes: A high resolution 3D surface construction algorithm[END_REF].

Semantic and volume approaches are very rare due to the lack of datasets containing both information.

In [PNM + 20], the authors propose a synthetic dataset, SyntheticRooms, constructed with ShapeNet objects and planes with small thickness for walls and floor. Even if not used in [PNM + 20], semantic information can be retrieved and used for learning. However, the dataset is object-oriented and is not realistic for the reconstruction of the full volume of a building: the building components are reduced to floor and walls with a fixed thickness.

S3DIS [START_REF] Armeni | Joint 2D-3D-Semantic Data for Indoor Scene Understanding[END_REF] contains volume information encoded as a set of 3D boxes with semantics. The classes are more numerous than in SyntheticRooms, including windows and doors, but the volume representation is mostly furniture focused i.e., it is not accurate at all for the building structure. For example, volumes are built for each room independently, thus, floor and walls are given arbitrary thicknesses; moreover, some holes (missing labels) make the date incomplete (see Figure 8.1).

To address the lack of data for volume and semantic reconstruction, we present a new dataset.

Empty column

Lack of thickness in a wall

Holes in the walls

Dataset

presentation of the dataset

As opposed to previous datasets, our primary objective is to build a realistic database for building component reconstruction that is focused on the building's entire structure instead of furniture. VASAD is derived from six freely available BIMs covering more than 62,000 m 2 . The scenes vary from a large scale hospital to a small residential building. The six buildings are presented in Figure 9.1 (a) and (b), with interior views on Artwork 2, Artwork 3, Artwork 4, Artwork 5, Artwork 6 and Artwork 7. The semantic classes and their color representations are listed on Table 9.1. These classes have been obtained by filtering the components names in the original BIM models. We strove to build classes which are relevant to properly model arbitrary building structures, and to avoid the overlap between the classes, though some of them can be challenging to tell apart (e.g., partition vs bearing wall, slab vs ceiling, partition vs pillar). As opposed to S3DIS [ASZS17] which covers 6,000 m 2 , VASAD contains the full structure information, e.g., the volumes represent the full structural components of the building and are not limited to a few millimeters from the surface.

Beam

Bearing

3d representation

Given an oracle which tells the class of an arbitrary 3D point on the scene, it is possible to generate a wide variety of 3D representations, e.g., meshes or voxels. In order to build such an oracle, an important condition is that the input shapes are closed so that the interior/exterior notions are correctly defined. It is practice almost the case in VASAD, though some construction errors still remain (e.g., due to their complex structure, some stairs railing are not always closed meshes without self-intersections).

Our oracle is built on the idea that any ray starting from a point inside a given shape intersects its surface an odd number of times. In order to introduce some robustness with respect to the errors in the input meshes, we use 3 axis-aligned rays to determine a point's label. A majority vote helps deciding on the correct label. The detailed algorithm is described on Figure 9

point cloud simulation

In many methods [PNM + 20], the input point clouds are created synthetically by uniformly sampling the ground-truth surface. In this work, we go one step closer to a realistic LIDAR scan by sampling points by ray shooting from a set of viewpoints.

Viewpoint generation

Modern acquisition methods of indoor and outdoor environments make use of UAVs [START_REF] Chiang | Development of lidar-based uav system for environment reconstruction[END_REF] to allow point cloud capture and registration from almost arbitrary viewpoints. To sim-While this method leads to a more realistic simulation, it could still be improved: LIDARs typically embed a rotor which makes the distribution of shooting directions around the viewpoint non isotropic. Moreover, the sampled point's position on the ray is usually noisy. The level of noise varies with the incidence angle with respect to the normal of the scanned surface, the distance between the sensor and the scanned surface, and the texture and albedo of the surface. Improvements on the baseline virtual scanning method we provide could lead to more realistic features and close the gap towards real data. (3) WestRiverSideHospital

pvsrnet

Voxels are a very natural way to apply learning methods to 3D reconstruction because they allow a direct generalization of the methods used for images. In particular, the UNet [RFB15] architecture has successfully been applied for image segmentation. Its direct 3D generalization has already been used in order to perform a dense volume segmentation from sparse annotations [ÇAL + 16]. One of the main drawbacks of the voxel representation is its memory consumption. In practice, attempts have been made to exploit the sparsity of the reconstructed space [START_REF] Tatarchenko | Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs[END_REF]. This approach involves leaving the decision whether to subdivide the space to reconstruct to the network. In our case, we keep a vanilla UNet and we leverage point convolution networks [START_REF] Boulch | FKAConv: Feature-Kernel Alignment for Point Cloud Convolution[END_REF] which are able to process a larger part of space at once and to therefore leverage an additional context. This is particularly important in the context of semantic segmentation of buildings: locally, a ceiling and a slab can look very similar, and the relative position of these elements in the building can help disentangling this ambiguity. The method is summarized on Figure 10.1. The point cloud with normals is first processed at large scale by a FKAConv-based [BPM20] UNet. A feature is computed per point at this stage. Since the 3D space area is too big to be fully processed as voxels, only a smaller part (in practice a cubic chunk of 2m40 side) is accumulated in a set of 48x48x48 voxels: when several points fall in the same voxel, the voxel feature is computed as the average of the points' features. At this stage, only a subset of the voxels close to the surface is filled with information. Then, the voxels are fed to a 3D UNet whose role is to propagate the information across the whole volume.

Point cloud with normals

FKA-Conv UNet

Point cloud with features

Voxel accumulation

Voxels with features

Voxels with labels

In theory, this model is end-to-end trainable, but this was not practically tractable with our computation means. Therefore, we first trained the FKA-Conv UNet thanks to the point's label that are recorded during the virtual scan (See 9.4.2). The loss we use is a cross-entropy loss and the features we keep are the raw logits (before the softmax). The obtained point clouds with features are then used to build chunks of voxels. Each voxel chunk is a cube whose side has 48 voxels of 5cm each. The second step consists in using these chunks to train the 3D Unet thanks to, again, a cross entropy loss applied on the voxels.

For simplicity, we refer to this approach as PVSRNet for Point-Voxel Semantic Reconstruction Network.

semantic convolutional occupancy network

ConvONet [PNM + 20] reconstructs volumes as an implicit representation from a 3D point cloud. It is based on the simpler Occupancy Networks [MON + 19] which consist in a multi layer perceptron mapping the 3D space to an occupancy (typically with value 1 inside the shape and 0 outside), conditioned on a latent representation of the whole input. A major limitation of Occupancy Networks [MON + 19] is that they make use of a unique latent vector to represent the whole input point cloud, regardless of its scale and complexity which prevents it to be used for scene reconstruction in practice.

In ConvONet [PNM + 20], the authors first subdivide the input point cloud into large voxels. For each of these large voxels, a PointNet [START_REF] Ruizhongtai | PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation[END_REF] encoder takes as input the points located in the considered voxel and provides a local latent vector. This set of latent vectors is then further refined thanks to a 3D UNet [ÇAL + 16]. Finally, for each arbitrary point in the 3D space, occupation is obtained by trilinear interpolation of the latent vector, which is fed to a decoding multi-layer perceptron similar to the one used in [MON + 19].

To obtain semantic information on top of pure geometry, we modify the predicted occupancy function: instead of only predicting a binary occupancy, for a given point, we predict a semantic class, including void. The network is then trained with a standard cross-entropy loss. At inference time, the authors of [PNM + 20] directly apply the marching cube [START_REF] Lorensen | Marching cubes: A high resolution 3D surface construction algorithm[END_REF] algorithm on the output logit, which allows a smooth interpolation between the marching cube voxels. In our case, we first apply a hard argmax on the logits, and the marching cubes algorithm is applied for each non-void semantic class against all other classes. Therefore, the global reconstructed mesh includes one mesh per class. Using a hard softmax does not allow a smooth interpolation (hence the voxelized aspect of the output), but it allows to perfectly stitch two volumes of different class, without in-between void or overlap, which is a typical requirement for BIM applications. If a smoother mesh is needed for a specific application, it is possible to use a finer marching cubes resolution, at the cost of additional computation time. ] PVSRNet: As described in section 10.2, PVSRNet is composed of two sub-networks. One dealing with points (FKAConv) and the other processing voxels (3D-UNet). In FKAConv, a 6m-diameter ball is used to select subsets of points at training time. The obtained point features are then aggregated in voxels of 5cm. Each chunk of voxels has 48 voxels per side, which yields a 2.40m square. In order to label the voxels, we follow a strategy that has a bias towards the full space, as to balance the fact that 70% of the models is composed of void space. For each voxel, we sample 27 points on a regular grid and get their labels thanks to the algorithm described in Figure 9.2. If and only if all points are labeled with void, the voxel gets a void label. Otherwise, the voxel gets the value of the most represented full class among the points. This allows us to recover thin objects such as windows which often fail to be represented by the majority of the 27 points. For each sampled point on the surface, the line segment between the point and the point of view from which it was sampled constitutes a visibility information. This information is stored at voxel level thanks to a binary variable which stores whether a given voxel is intersected by a visibility segment or not. The routine to efficiently compute this information is described on Figure 10.2.

] Semantic Convolutional Occupancy Network: We strive to mimic the data format used in the original implementation. We randomly sample cubes of 4m side in the 3D space and apply a random Y-axis rotation. We select the input points which lie in this cube. As a supervision, and to be fair with the bias towards full spaces we introduced when preparing the data for PVSRNet, we first sample points on a 5cm voxel grid, and 27 points per voxel on an inner regular grid. This generates more points than needed in the original algorithm, we therefore apply the following selection: for each 5cm-side voxel, we make sure to have at least one point per class that is represented in the voxel. This typically yields more than half of the 1M points we wish to consider per chunk. The remaining points are randomly selected among the rest.

Results

[ Chapter 10

Chapter 12 ]

metrics

The task of volume reconstruction with semantic is manifold, therefore it needs different metrics to quantitatively assess each aspect of the resulting 3D model. In particular, both geometric and semantic aspects need to be evaluated. Moreover, we not only propose metrics that evaluate the volume aspect of the reconstruction, but we also propose surface metrics that evaluate the quality of the reconstructed surface.

Volume metrics

Evaluation is carried out by uniformly sampling points 𝒫 𝑣 in the union of the bounding box of the ground-truth and the predicted 3D models. In practice, we sample 10 million points and we noticed that increasing the number of points left the metrics stable. We then find the ground-truth label 𝑙 𝑝,𝑔𝑡 and predicted label 𝑙 𝑝,𝑝𝑟𝑒𝑑 for each point 𝑝 ∈ 𝒫 𝑣 thanks to the algorithm presented in Figure 9.2. All the volume metrics that we build are based on these labels. We denote by 𝑣𝑜𝑖𝑑 the void label.

] Semantic IoU (intersection over union): Among each point in 𝒫 𝑣 that are labeled as full in either the ground truth or the prediction, we look at the ratio of these points which have the same label in both the ground truth and the prediction. ] Geometric IoU: Some models can predict a correct partition of the 3D space between the binary labels (full/void), but misclassify the full space (e.g., wall vs window confusion).

In this case, even if the semantic is incorrect, the geometry is still recovered. We evaluate this thanks to the geometric IoU: For the sake of visualization and understanding, each row 𝐶 𝑖 is normalized by the total number of points whose ground truth class is 𝑖: ∑ 𝑗 𝐶 𝑖,𝑗 .

IoU 𝑔 = | 𝑝

Surface metrics

Our surface metrics evaluate the quality of the geometry of the reconstructed surface.

To do so, we sample points 𝒫 𝑠,𝑔𝑡 on the ground-truth mesh and points 𝒫 𝑠,𝑝𝑟𝑒𝑑 on the predicted mesh. In practice, | 𝒫 𝑠,𝑔𝑡 | = | 𝒫 𝑠,𝑝𝑟𝑒𝑑 | = 10 millions.

We then evaluate the distance between each point in 𝒫 𝑠,𝑝𝑟𝑒𝑑 and its nearest neighbour in 𝒫 𝑠,𝑔𝑡 : 𝐷 𝑝𝑟𝑒𝑑→𝑔𝑡 = {||𝑝 -𝑁 𝑁 𝒫 𝑠,𝑔𝑡 (𝑝)|| for each 𝑝 ∈ 𝒫 𝑠,𝑝𝑟𝑒𝑑 } where 𝑁 𝑁 𝑌 (𝑥) is the nearest neighbour of 𝑥 in the set 𝑌. Symmetrically, we evaluate the distance between each point in 𝒫 𝑠,𝑔𝑡 and its nearest neighbour in 𝒫 𝑠,𝑝𝑟𝑒𝑑 : 𝐷 𝑔𝑡→𝑝𝑟𝑒𝑑 = {||𝑝 -𝑁 𝑁 𝒫 𝑠,𝑝𝑟𝑒𝑑 (𝑝)|| for each 𝑝 ∈ 𝒫 𝑠,𝑔𝑡 } All the surface metrics we build are based on these distances.

] Mean surface distance: the average surface error is given by averaging all the nearest neighbour distances we calculated. This distance is usually called the Chamfer distance in the literature. Av. dist. = mean(𝐷 𝑝𝑟𝑒𝑑→𝑔𝑡 ∪ 𝐷 𝑔𝑡→𝑝𝑟𝑒𝑑 )

] Max surface distance: the worst surface error that we make, also called the Metro distance. Max. dist. = max(𝐷 𝑝𝑟𝑒𝑑→𝑔𝑡 ∪ 𝐷 𝑔𝑡→𝑝𝑟𝑒𝑑 )

] Precision: this metric tells how close the reconstructed surface is to the ground truth.

To avoid considering the worst errors, a threshold of 95% is taken to discard the 5% worst distances: Prec. = 𝑝𝑒𝑟𝑐𝑒𝑛𝑡𝑖𝑙𝑒 95% (𝐷 𝑝𝑟𝑒𝑑→𝑔𝑡 )

] Accuracy: this metric evaluates to what extent the ground truth surface is reconstructed. As well as for the precision, a threshold of 95% is used to discard the 5% worst distances: Acc. = 𝑝𝑒𝑟𝑐𝑒𝑛𝑡𝑖𝑙𝑒 95% (𝐷 𝑔𝑡→𝑝𝑟𝑒𝑑 )

Metric Oracle

Since all our metrics involve a discretization through point sampling, we also provide an oracle which computes each metric for the validation model against itself on Figure 11.1.

surface reconstruction

We present quantitative (Table 11. In both methods, the use of voxels on one hand, and the marching cubes algorithm on the other hand generate highly complex meshes. Because the amount of triangles we can display at once is limited, we had to decimate the meshes thanks to the "Quadric Edge Collapse Decimation" provided by the software Meshlab [START_REF] Cignoni | MeshLab: an Open-Source 3D Mesh Processing System[END_REF]: for each 2.40m-side chunk, we set a maximum budget of 2500 faces. This simplification was used only for that the oriented normal information already indicates locally in which direction the void is located. Finally, we report the scores of PVSRNet in the last row of Table 11.1. It leverages both the normals and the rich semantic features from the FKAConv sub-network at voxel level. With respect to the surface metrics, it is slightly worse but comparable to the other variants, except for the maximal distance, due to very rare, but very penalizing missing parts such as the parking lot of Figure 11.2, last row. The main reason of this local failure is the extremely low point density on this part of the building.

semantic segmentation

The semantic classes in buildings are highly non uniformly distributed in terms of volume.

Structural classes such as bearing walls, partition and slabs are more represented than smaller features such as beams, doors, pillars, railings, stairs or windows. Consequently, we see that in spite of the bias towards full spaces in our supervision (see Figure 10.4) which increases the supervision signal on the smaller features, they are harder to infer while the majority classes are still well recovered by both methods. However, the Semantic Convolutional Network is not as good as PVSRNet at resolving ambiguities (see the ambiguity partition/bearing wall on Figure 11.3).

Other classes sometimes have an inherent ambiguity which is well observed in the results (see the confusion matrix in Figure 11.1):

] Partition / bearing wall: partitions tend to be thinner than bearing walls, and bearing walls tend to be mostly on the outermost part of buildings. However, exceptions remain which can make these classes hard to distinguish.

] Bearing wall / window: windows tend to be located on bearing walls, and the transition between the two classes can be hard to see on the point cloud. Variations in the normals and thickness changes are typical markers of a transition between a bearing wall and a window.

] Partition / door: similarly, and for the same reasons as for bearing walls and partitions, a close door can be a hard feature to distinguish from its neighboring partition at point cloud level.

] Slab / roof: In modern architecture, flat roofs are becoming more and more common. It is even possible for people to access these parts of the building. As a consequence, there is an intrinsic ambiguity between the two classes.

In spite of these ambiguities, we notice that the Semantic Convolutional Occupancy Network is not able to properly recover the small classes. In particular, we can notice the windows on Figure 11.2, the door and the beams on 11.1 as well as the confusion matrix on Figure 11.1 also show that our method benefits a lot from the FKAConv [BPM20] point features (+10% in IoU).

One major limitation of PVSRNet is that the transition between the semantic classes is not entirely satisfactory yet. This is especially true when the transition does not include a significant change in the surface normal: doors/partitions, windows/bearing walls for instance. Moreover, PVSRNet is limited by the voxel resolution which prevents it to properly represent classes with small details such as railings (see Figure 11.4).

Input point cloud

Ground truth voxel based approach. However, by using ConvONet as a baseline, we bring forward some limitations of the implicit representations: in spite of their inherent ability to represent any arbitrary closed surface, we show that irregularly-sampled surfaces are more efficiently reconstructed by a simpler voxel-based method. The issue is not limited to surface reconstruction, we also notice that the smaller classes are better recovered by our method and that the ambiguities are better resolved. One key difference between the voxel-based methods and the implicit representations lies in the loss: while the crossentropy loss on voxels is dense (it provides supervision on each point of the 3D space), supervising an implicit representation involves a discretization strategy. It is future work to investigate methods with hybrid voxel/implicit supervision in order to take advantage of both representations.

Towards BIM models

As an output, we have a set of closed volumes within which the semantic label is constant. This is probably sufficient to perform thermal or acoustic simulations. However, in the case of building modification, it may be necessary to have access to an idealized shape (e.g., a true parallelepiped instead of a set of voxels). This can be achieved thanks to specific mesh simplifications algorithms [START_REF] Cignoni | A comparison of mesh simplification algorithms[END_REF]. Moreover, it maybe necessary to further separate each instances (e.g., two perpendicular walls may be composed of 2 separate pieces). In any case, the BIM format does not define a unique way to represent a building and as such, this post processing step heavily depends on the downstream application.

conclusion

We introduced VASAD, a dataset for geometric and semantic reconstruction of buildings. This dataset aims at addressing the lack of data focused on building structures which is a key components of modern BIM models in architecture and civil engineering. This dataset includes 6 clutterless (empty) buildings. A strategy to sample point of views in these buildings has been developed which further allows to simulate LIDAR scans. Moreover, a routine to label any arbitrary 3D point allows to build raw data for a large variety of 3D representations. We provide the full implementation to produce the data from the raw IFC file, and we therefore allow this dataset to be enriched from more IFC files. We have shown that this dataset is challenging enough for modern machine-learning-based reconstruction methods. We were able to highlight the limitations of learned implicit representations and we were able to propose a voxel-based method to perform semantic reconstruction of building which is able to recover small classes and to address some of the inherent class ambiguities. However, the non-uniform sampling of the input point cloud makes it challenging to reconstruct a smooth surface, and the transition between classes is not always clean. As such, we believe that this dataset paves the way for contributions that will eventually enable fully automatic so-called Scan-to-BIM methods.

Part V EPILOGUE

Conclusion

[ Chapter 11

12.1 looking back

In terms of computer vision, automatizing the generation of a BIM model from sensors such as cameras or LIDARs involves the task of 3D reconstruction with semantic inference. 3D reconstruction is a complex task. As we have recalled, various input sensors are available to produce meaningful input data (images and point clouds for instance), and the choice of the 3D output representation (mesh, voxel, implicit representation, among others) plays a central role. Moreover, as for most of the tasks dealing with real-world data, the algorithms performance depend on the environment: in the case of buildings we highlighted two specificities in particular: the overwhelming presence of textureless surface, and the highly-structured geometry (large piecewise-planar areas, big volumes of the same semantic class such as walls, ceilings and roofs). With these observations in mind, we studied how to design a 3D reconstruction algorithm with semantics adapted to buildings.

] In part II, we focused on pure geometric reconstruction from a set of images with reasonable quality. We strove to understand and tackle the specificities of 3D reconstruction in the context of building (presence of textureless surfaces, presence of visible line segments due to the geometric regularity of the environment) and we were able to design a new algorithm which is able to outperform previous approaches on both synthetic and real data [START_REF] Langlois | Surface Reconstruction from 3D Line Segments[END_REF].

] In part III, we took a step away from the building environments to better understand how to learn meaningful priors from a large dataset. We designed an algorithm which outputs a mesh from a set of low-quality calibrated images of an object using template deformation [LFW + 21]. This approach competes with state of the art methods and is able to directly output a mesh.

] In part IV, we directly address the task of volumetric geometric reconstruction of buildings with semantics from LIDAR scans by introducing VASAD, a dataset specifically designed to learn this task. We introduce effective baselines to solve the task and show that structured 3D representations are the most promising to solve this task.

These contributions allow to push the boundaries of reconstruction algorithms and pave the way for more contributions in the generation of BIM models. We found that the use of machine learning is particularly effective to achieve this goal and that some of the specific corner cases met when dealing with buildings can be addressed.

looking ahead

With this work, we get closer to a scan-to-BIM algorithm. However, several aspects of the problem still need to be explored. In part IV, we studied the 3D reconstruction of buildings with semantics from artificial LIDAR scans. As discussed, a next step is to check that our assumptions hold well on real data. Moreover, LIDAR sensors remain expensive. It would therefore be relevant to generalize our algorithm to images as input, but again, there is a lack of relevant data to learn this task at building scale. When working on part IV, we noticed that the BIM files are usually not strongly constrained: for instance, there are several possibilities to split a continuous bearing wall (see Figure 12.1), depending on the construction method and the downstream task for which the model will be used. Clarifying these ambiguities or introducing conventions could pave the way for the introduction of instance segmentation in which not only would the parts be annotated by class, but each instance would be separated as in current BIM files. As mentioned previously, the BIM digital representations usually include more than the geometry and the semantics of the visible parts. In particular, they can include information on the different networks (electrical, hydraulic), even if they are mostly hidden in the walls and slabs (see Figure 12.2). Recovering such invisible network would require additional sensors (e.g., Hall-effect based sensors among others) and a specific data processing.

Full BIM Model

Hydraulic network only

4.3. 1

 1 Comparing to point-based reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Comparing to other line-based reconstruction methods . . . . . . . . . . . . . . . . . 4.4 Quantitative evaluation of reconstructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.1 Metrics to assess the quality of surface reconstruction . . . . . . . . . . . . . . . . . . 4.4.2 Varying parameters or input data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.3 Parameter setting and sensitivity study . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.4 Computation times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.5 Robustness of plane detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Ablation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Limitations and perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .III 3D REC O N S T R U C T I O N B Y PA R A M E T E R I Z E D S U R FA C E M A P P I N G 5 INTRODUCTION 5.1 Learning 3D reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 METHOD 6.1 Learning a Multi-View Parameterized Surface Mapping . . . . . . . . . . . . . . . . . . . . . 6.1.1 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1.2 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1.3 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Design choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Template choice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Using the UVW parameterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.3 Optimized parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.4 Positional encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.5 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 RESULTS 7.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Evaluation criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Optimization process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.3 Results compared to baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Ablation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.1 Impact of the optimization stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Chamfer loss vs gnomonic loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Discussion and limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure

  Figure 1.1: A BIM model and its different components.

Figure

  Figure 1.2: A building (Villa Sextant) and its digital twin.

Figure

  Figure 1.3: Even on synthetic data, it is difficult to distinguish a door on a point cloud.

Figure

  Figure 1.4: Summary of our contributions (framed) and their relationship to the topics cover by the thesis subject (black).

3

  image samples out of the 32 images used for reconstruction Colmap [SF16] + Poisson [KBH06] failed reconstruction

Figure

  Figure 2.1: Failed pointbased reconstruction in a textureless environment

Figure

  Figure 3.1: Input lines and reconstructed surfaces on 4 datasets (from left to right): TimberFrame, HouseInterior, Barn, Terrains.

Figure

  Figure 3.2: One iteration of our RANSAC process: two random line segments are selected (left) and the lines fitting the corresponding candidate plane are selected (right).

Figure

  Figure 3.3: Because line segments are sparse, removing the inliers immediately after a plane detection leads to missing planes in the simple case of a cube whose detected lines are the edges (in red: the detected planes at each step).

Figure

  Figure 3.5: RANSAC-based plane detection from 3D lines segments, differentiating structural lines from textural lines.

Figure

  Figure 3.7: Primitive and visibility terms.

Figure

  Figure 3.8: Non-penalized and penalized configurations of the 4 cells adjacent to each edge. Courtesy of [BdLGM14]

Figure

  Figure 3.9: Non-penalized and penalized configurations of the 8 cells adjacent to each corner. Courtesy of [BdLGM14]

  (a) Missing detections of 3D lines in the input: visible edge between floor and wall in a view of the MeetingRoom dataset (left), and extracted lines with Line3D++ where that important edge is missing to recover the floor (right). (b) Spurious detections of 3D lines in the input: view from the MeetingRoom dataset (left), and reconstructed line cloud using Line3D++where vertical outlier lines float above the table (right).(c) Spurious duplication of 3D lines in the input: view from the Andalusian dataset (left), and the extracted lines with Line3D++ where some actual 3D lines in the scene are detected many times at slightly different locations (right).

Figure

  Figure 4.1: Results on Andalusian

Figure

  Figure 4.2: Results on Delivery Area

Figure

  Figure 4.3: Results on Barn

Figure

  Figure 4.4: Results on Timberframe

Figure 4

 4 Figure 4.5: Results on Bridge

Figure

  Figure 4.9: HouseInterior: histograms of distance errors w.r.t. ground truth (m).

Figure

  Figure 4.10: Impact of 𝜎 𝑝 on the max / mean Metro distance (top), and the 95% precision / completeness (bottom).

Figure 4 .

 4 Figure 4.13: Impact of 𝜆 vis on the max / mean Metro distance (top), and the 95% precision / completeness (bottom).

Figure 4 .

 4 Figure 4.14: Impact of 𝜆 edge on the max / mean Metro distance (top), and the 95% precision / completeness (bottom).

Figure 4 .

 4 Figure 4.15: Impact of 𝜆 corner on the max / mean Metro distance (top), and the 95% precision / completeness (bottom).

Figure

  Figure 4.16: Impact of the number of planes 𝑵 max on the max/mean Metro distance (top) and on the 95% precision/completeness (bottom).

Figure

  Figure 4.17: Robustness of RANSAC on lines for a cube defined by its edges. Value in grid is the average number of cube planes found, depending on the perturbation.

Figure

  Figure 4.18: Reconstruction with our method where a line may support up to two planes (left, our RANSAC), or at most one plane (right, standard RANSAC).

Figure

  Figure 4.19: Reconstruction with our method where the gathering of inliers for a plane model relies on the distance to the plane intersection for lines already supporting a plane (left, our approach), and to the plane only (right, standard approach).

Figure

  Figure 4.20: Reconstruction with our method where all structural lines are treated specifically (left), and considered as two textural lines, i.e., one for each supported plane (right).

Figure

  Figure 4.21: Reconstruction with a regularization on corners only (left), vs on edges only (right).

Figure

  Figure 4.22: Reconstruction with a regularization on corners and edges (left), vs on corners only (right).

Figure 4 .

 4 Figure 4.23: Impact of 𝜆 area on the max / mean Metro distance (top), and the 95% precision / completeness (bottom).

Figure

  Figure 5.1: Input: a set of 5 calibrated low-resolution images with arbitrary backgrounds. Output: a 3D mesh

  Figure 6.1: Networks for inferring a depicted shape. Our pipeline consists of two steps. First, a trained encoder generates an initial latent shape representation from the calibrated images.Next, the decoder is iteratively improved through geometric and mask constraints to further improve reconstruction.

Figure

  Figure 6.2: Multi-view shape encoder. Our multi-view shape encoder ℰ aggregates transformed features given any number of input views and corresponding camera parameters. A decoder 𝜙 then maps the encoded representation to the output mesh.

Figure

  Figure 6.3: The gnomonic mapping: each point sampled on the ground truth shape is projected on the sphere using a central projection.

  Figure 6.4: The cycle loss through an AtlasNet Decoder.

Figure

  Figure 6.5: The mask loss. We use a 2D Chamfer distance to make sure that the reconstructed mesh masks fits the detected mask. Only the projections which fall outside of the mask and inside the frame are penalized.

Figure

  Figure 6.8: Failed initialization of a plane shape with positional encoding.

Figure

  Figure 7.1: Qualitative results. For each sub-figure: Input (top), Ground truth (bottom left), our reconstruction mesh (bottom middle) and XNOCS output points (bottom right).

  of optimization)

Figure

  Figure 7.2: Representation of the optimization process. Grey: the mesh being optimized. Red: ground truth as a point cloud

Figure

  Figure 7.3: Qualitative results on the airplane category.

Figure

  Figure 7.4: Qualitative results on the car category.

Figure

  Figure 7.5: Qualitative results on the chair category.

Figure

  Figure 7.6: Qualitative results on the chair category.

Figure

  Figure 7.7: Visualization of learned surface parameterization depending on the loss used. Discontinuities in colors indicate selfintersections.

Figure

  Figure 8.1: A few shortcomings of the volume annotations in S3DIS [ASZS17].

Figure

  Figure 9.4: Representation of the sampled point cloud on the ground truth mesh (test set). The point distribution is not uniform: it is denser closer to viewpoints.

Figure

  Figure 10.1: PVSRNet is composed of two networks, one operating on a raw point cloud for rich semantic feature extraction (FKAConv sub-network) and the second one on voxels for semantic volume occupancy (3D UNet).

  and outputs are different for PVSRNet and SemConvONet (the semantized version of ConvONet). This section describes how the VASAD dataset is prepared for training with each method.

Figure

  Figure 10.2: Fast visibility computation. The voxels considered on a line of sight (highlighted) are computed by querying the voxels in which lie the mid-points (green) of the line segment end-points and the intersections (red) of the line segment and the voxels facets, all of which are easily computed.

  IoU 𝑠 = | 𝑝 ∈ 𝒫 𝑣 such that 𝑙 𝑝,𝑔𝑡 ≠ 𝑣𝑜𝑖𝑑 and 𝑙 𝑝,𝑔𝑡 = 𝑙 𝑝,𝑝𝑟𝑒𝑑 | | 𝑝 ∈ 𝒫 𝑣 such that 𝑙 𝑝,𝑔𝑡 ≠ 𝑣𝑜𝑖𝑑 or 𝑙 𝑝,𝑝𝑟𝑒𝑑 ≠ 𝑣𝑜𝑖𝑑 |

  1) and qualitative results (see Figure 11.2, Figure 11.3, Figure 11.4, Figure 11.5, Figure 11.6, Figure 11.7 and Figure 11.8) on the VASAD dataset.

  Figure 11.3, the railings on Figure 11.4 and the windows on Figure 11.6 and Figure 11.7. The IoU scores on Table

Figure

  Figure 11.6: Qualitative results on the test set (NBU_Medi-calClinic).

  Figure 11.8: Qualitative results on the test set (NBU_Med-icalClinic).A failure case: due to the point of view being far away, the point sampling is too sparse, leading to a missing chunk.

Figure

  Figure 12.1: A few different partition possibilities for a bearing wall (sectional view from above).

Figure

  Figure 12.2: Most of the hydraulic network of VASAD's Medical Clinic is not directly visible.

Finally, and

  closer to this work, promising 3D representations could allow a future breakthrough regarding the purely geometric and semantic aspects. The authors of [KAB + 21] experiment 3D reconstruction as a set of 3D boxes which could be an interesting way to represent the very regular shapes of buildings, and which is a volume representation, and the neural render fields [MST + 20] allow a promising representation of a scene learned from images.

  Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 BIM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Automatic model generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Organization of the dissertation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	I PROLO G U E
	1 INTRODUCTION
	1.1

1.2.2 3D Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 Semantic segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.4 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 1.3.1 Main contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2 Other contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4

  Inlier selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.5 Plane selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.6 Plane refitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.7 Plane fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.8 Plane limitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Surface reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Dealing with noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Primitive term . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 Visibility term . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.4 Regularization term . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Observations on the input data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 3D line segments detectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Modest quality of input data from Line3D++ . . . . . . . . . . . . . . . . . . . . . . . 4.3 Qualitative evaluation of reconstructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 INTRODUCTION 2.1 Reconstructing textureless surfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 METHOD 3.1 Line extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Plane detection from 3D line segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Candidate plane construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Greedy detection and multi-support issues . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Candidate plane generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 3.3.5 Solving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.6 Properties of reconstructed surface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 RESULTS 4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii 4.2

  3D Reconstruction for buildings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Building information models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Presentation of the dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 3D representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Point cloud simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Viewpoint generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.2 Ray shooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 Train/test split . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Surface reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Semantic segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Looking back . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Looking ahead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Line3d++ failing cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Exterior overview and interior details of NBU_OfficeBuilding. . . . . . . . . . . . . . . . . . 3 Real [Com17], exterior overview and interior details of Sextant. . . . . . . . . . . . . . . . . . 4 Exterior overview and interior details of WestRiverSideHospital. . . . . . . . . . . . . . . . . 5 Real, exterior overview and interior details of Trapelo. . . . . . . . . . . . . . . . . . . . . . . 6 Exterior overview and interior details of OTC-ConferenceCenter. . . . . . . . . . . . . . . . . 7 Exterior overview and interior details of NBU_MedicalClinic. . . . . . . . . . . . . . . . . . . A BIM model and its different components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 A building (Villa Sextant) and its digital twin. . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Even on synthetic data, it is difficult to distinguish a door on a point cloud. . . . . . . . . . 1.4 Summary of our contributions (framed) and their relationship to the topics cover by the thesis subject (black). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Failed point-based reconstruction in a textureless environment . . . . . . . . . . . . . . . . . 2.2 Line-based 3D reconstruction pipeline. This paper covers from Input to Output. . . . . . . . . One iteration of our RANSAC process: two random line segments are selected (left) and the lines fitting the corresponding candidate plane are selected (right). . . . . . . . . . . . . . . . 3.3 Because line segments are sparse, removing the inliers immediately after a plane detection leads to missing planes in the simple case of a cube whose detected lines are the edges (in Extracting the reconstructed surface from the plane arrangement . . . . . . . . . . . . . . . . 3.7 Primitive and visibility terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Non-penalized and penalized configurations of the 4 cells adjacent to each edge. Courtesy HouseInterior: histograms of distance errors w.r.t. ground truth (m). . . . . . . . . . . . . .4.10Impact of 𝜎 𝑝 on the max / mean Metro distance (top), and the 95% precision / completeness (bottom). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.11 Variable number of input images: comparison of our method vs Colmap [SF16] + Poisson [KBH06] reconstruction on a variant of MeetingRoom with 100 images. . . . . . . . . . . . . 4.12 Impact of the number of input lines on the max/mean Metro distance (top), and the 95% precision/completeness (bottom). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.13 Impact of 𝜆 vis on the max / mean Metro distance (top), and the 95% precision / completeness (bottom). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.14 Impact of 𝜆 edge on the max / mean Metro distance (top), and the 95% precision / completeness (bottom). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.15 Impact of 𝜆 corner on the max / mean Metro distance (top), and the 95% precision / completeness (bottom). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.16 Impact of the number of planes 𝑵 max on the max/mean Metro distance (top) and on the 95%

	8 INTRODUCTION List of Artworks List of Figures
	1 8.1 9 DATASET 9.1 V EPILO G U E 12 CONCLUSION 1.1 2.1 4.6 MeetingRoom: (1) image sample, (2) segments from Line3D++ [Hof16, HMB16], (3) our 12.1 REFERENCES reconstruction, point-based reconstructions with Colmap [SF16] (4) then Chauve et al. [CLP10],
	(5) Polyfit [NW17], (6) Poisson [KBH06], (7) Delaunay [LPK09]. . . . . . . . . . . . . . . . .
	VI APPE N D I X 4.7 Terrains: (1) image sample, (2) segments from Line3D++ [Hof16, HMB16], (3) our recon-
	struction, point-based reconstructions with Colmap [SF16] (4) then Chauve et al. [CLP10], (5)
	Polyfit [NW17], (6) Poisson [KBH06], (7) Delaunay [LPK09].

10 METHOD 10.1 Reconstruction approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 PVSRNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Semantic Convolutional Occupancy Network . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 Data preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 RESULTS 11.1 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.1 Volume metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.2 Surface metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.3 Metric Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 11.4.1 Inherent ambiguities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.2 PVSRNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.3 Towards BIM models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 3.1 Input lines and reconstructed surfaces on 4 datasets (from left to right): TimberFrame, HouseInterior, Barn, Terrains. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 red: the detected planes at each step). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Inlier selection. We assume the black plane has already been detected and we are selecting inliers for the pink plane. Both the red and green line segments are close to the planes but only the green segment is selected because it is close enough to the plane intersection (i.e., within the yellow cylinder). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 RANSAC-based plane detection from 3D lines segments, differentiating structural lines from textural lines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 of [BdLGM14] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Non-penalized and penalized configurations of the 8 cells adjacent to each corner. Courtesy of [BdLGM14] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Results on Andalusian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Results on Delivery Area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Results on Barn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Results on Timberframe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Results on Bridge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8 HouseInterior: (1) an image of the dataset, (2) points densely sampled on surface, (3) reconstruction with [CLP10], (4) failed reconstruction with [CLP10] from points sampled on lines, (5) 3D lines detected with Line3D++ [Hof16], with noise and outliers, (6) our reconstruction, which is nonetheless superior. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 4.9 precision/completeness (bottom). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.17 Robustness of RANSAC on lines for a cube defined by its edges. Value in grid is the average number of cube planes found, depending on the perturbation. . . . . . . . . . . . . . . . . . 4.18 Reconstruction with our method where a line may support up to two planes (left, our RANSAC), or at most one plane (right, standard RANSAC). . . . . . . . . . . . . . . . . . . . . . . . . .

  For each sub-figure: Input (top), Ground truth (bottom left), our reconstruction mesh (bottom middle) and XNOCS output points (bottom right). . . . . . . 7.2 Representation of the optimization process. Grey: the mesh being optimized. Red: ground truth as a point cloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Qualitative results on the airplane category. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Qualitative results on the car category. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Qualitative results on the chair category. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Qualitative results on the chair category. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Visualization of learned surface parameterization depending on the loss used. Discontinuities in colors indicate self-intersections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 A few shortcomings of the volume annotations in S3DIS [ASZS17]. . . . . . . . . . . . . . . 9.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 3D point labeling from a labeled mesh algorithm. . . . . . . . . . . . . . . . . . . . . . . . . Qualitative results on the test set (NBU_MedicalClinic). . . . . . . . . . . . . . . . . . . . . . 11.3 Qualitative results on the test set (NBU_MedicalClinic). . . . . . . . . . . . . . . . . . . . . . 11.4 Qualitative results on the test set (NBU_MedicalClinic). . . . . . . . . . . . . . . . . . . . . . 11.5 Qualitative results on the test set (NBU_MedicalClinic). The sampling is highly non uniform in this area. Our method is more robust to the sampling inconsistencies . . . . . . . . . . . . 11.6 Qualitative results on the test set (NBU_MedicalClinic). . . . . . . . . . . . . . . . . . . . . . 11.7 Qualitative results on the test set (NBU_MedicalClinic). . . . . . . . . . . . . . . . . . . . . . 11.8 Qualitative results on the test set (NBU_MedicalClinic).A failure case: due to the point of view being far away, the point sampling is too sparse, leading to a missing chunk. . . . . . . . . . Additional qualitative results for our reconstruction method by parameterized surface mapping. 1st column: ground truth; 2nd column: ours: 3rd column: XNOCS [SRV + 19] . . . . . . . . Dataset statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Parameters (all datasets have metric dimensions). . . . . . . . . . . . . . . . . . . . . . . . . Ablation study. Comparison of multi-view encoder and full model for Chamfer and gnomonic loss. † : AtlasNet (single view) makes inference in canonical coordinates as opposed to camera coordinates for our method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9.1 Semantic classes in VASAD along with their color scheme. . . . . . . . . . . . . . . . . . . .

	List of Tables 4.1 7.1 Multi-view shape reconstruction on the ShapeNetCOCO [SRV + 19] validation set. We report F1-score (F1) and squared-symmetric 100xChamfer distance (CD), averaged over each class. g l o ssary BIM Building Information Modeling. vi, viii, x, 3-7, 9, 10, 15, 16, 85, 86, 90, 102, 110, 111, 114, 115 7.2 List of Terms and Acronyms GPU Graphics Processing Unit. 9
	IFC Industry Foundation Classes: a file format for BIM models. 7, 111
	LIDAR Laser imaging, detection, and ranging. iii, iv, 3, 6, 7, 9, 15, 18, 82, 85-87, 90, 92, 94, 108, 111, 114,
	115

Impact of 𝜆 area on the max / mean Metro distance (top), and the 95% precision / completeness (bottom). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Input: a set of 5 calibrated low-resolution images with arbitrary backgrounds. Output: a 3D mesh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.1 Networks for inferring a depicted shape. Our pipeline consists of two steps. First, a trained encoder generates an initial latent shape representation from the calibrated images. Next, the decoder is iteratively improved through geometric and mask constraints to further improve reconstruction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Multi-view shape encoder. Our multi-view shape encoder ℰ aggregates transformed features given any number of input views and corresponding camera parameters. A decoder 𝜑 then maps the encoded representation to the output mesh. . . . . . . . . . . . . . . . . . . . . . 6.3 The gnomonic mapping: each point sampled on the ground truth shape is projected on the sphere using a central projection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 The cycle loss through an AtlasNet Decoder. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 The mask loss. We use a 2D Chamfer distance to make sure that the reconstructed mesh masks fits the detected mask. Only the projections which fall outside of the mask and inside the frame are penalized. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Rendering of the UV coordinates (Black: 0, White: 1) for a unit sphere . . . . . . . . . . . . 6.7 Impact of the choice of the parameters used for optimization . . . . . . . . . . . . . . . . . . 6.8 Failed initialization of a plane shape with positional encoding. . . . . . . . . . . . . . . . . xii 7.1 Qualitative results. 9.3 Viewpoint generation algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Representation of the sampled point cloud on the ground truth mesh (test set). The point distribution is not uniform: it is denser closer to viewpoints. . . . . . . . . . . . . . . . . . . 10.1 PVSRNet is composed of two networks, one operating on a raw point cloud for rich semantic feature extraction (FKAConv sub-network) and the second one on voxels for semantic volume occupancy (3D UNet). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Fast visibility computation. The voxels considered on a line of sight (highlighted) are computed by querying the voxels in which lie the mid-points (green) of the line segment end-points and the intersections (red) of the line segment and the voxels facets, all of which are easily computed.103 11.1 Left: the confusion matrix for SemConvONet (with normals). Right: the confusion matrix for PVSRNet with normals and surface semantics as input. . . . . . . . . . . . . . . . . . . . . . 11.2 12.1 A few different partition possibilities for a bearing wall (sectional view from above). . . . . . 12.2 Most of the hydraulic network of VASAD's Medical Clinic is not directly visible. . . . . . . . 13.1 Additional qualitative results for our reconstruction method by parameterized surface mapping. 1st column: ground truth; 2nd column: ours: 3rd column: XNOCS [SRV + 19] . . . . . . . . 13.2 11.1 Quantitative results: we evaluate both the reconstructed volume and surface. Metrics: accuracy, precision, average distance, maximum distance, semantic intersection over union, and geometric intersection over union (considering only full/void). . . . . . . . . . . . . . . . . . . . . . . . MVS Multi-view stereo. 62 RANSAC Random Sample Consensus. xi, 14, 16, 17, 20-22, 29, 42, 43, 47, 52, 56 RGB Red Green Blue. 5, 9 RGB-D Red Green Blue Depth. 6, 87

  The present-day steady release of large scale datasets [DDS + 09, CFG + 15] has allowed training deep learning models at large scale.

	] Computing power: One of the first major breakthrough was accomplished thanks to a
	smart implementation of deep neural networks on a modern GPU for image classification
	[KSH12].
	The influence of deep learning on our task is twofold:
	] Deep reconstruction methods: Classical methods can only make use of the data from
	the specific scene they are trying to reconstruct, as well as hand-crafted priors. While
	this allows impressive results [SF16], the algorithm often fail where a human would
	perfectly understand the geometry of a scene thanks to his/her own experience of his/her
	surrounding world. Deep learning methods allow to go past this limitation by enabling
	scene completion [SYZ + 17] or single-view reconstruction [GFK + 18, PFS + 19, MON + 19].
	Moreover, the encoder/decoder architecture of these methods allow handling various
	inputs, and as such, [GFK + 18, PFS + 19, MON + 19] also allow reconstruction from a point
	cloud thanks to the powerful PointNet [QSMG17] encoder.
	] Deep segmentation methods: As mentioned in 1.2.3, classical methods struggle to
	model complex classes. The deep-learning-based methods have first brought tremendous
	improvements on image segmentation [RFB15, LSD15]. These networks have been used
	to label point clouds by back-projecting the 2D segmentation information [BGLSA18],
	and specialized networks for directly segmenting 3D data have appeared [QSMG17,
	QYSG17, ÇAL + 16, TQD + 19].

  best set of coplanar lines repeat 𝑁 iter times //// Sample a candidate plane by sampling 2 lines Pick 𝑙 𝑎 ∈ ℒ 0 ∪ ℒ 1 // 1 st sample line //// For 2 nd sample, exclude lines of the plane of 𝑙 𝑎 if any Pick 𝑙 𝑏 ∈ ℒ 0 ∪ ℒ 1 Λ(Π(𝑙 𝑎 )) // 2 nd sample line //// Make candidate plane and check consistency 𝑃 ← plane(𝑙 𝑎 , 𝑙 𝑏 ) next if 𝑃 degenerate or 𝑑(𝑙 𝑎 , 𝑃 ) > 𝜖 or 𝑑(𝑙 𝑏 , 𝑃 ) > 𝜖 //// Gather line support for plane 𝑃 Λ ← {𝑙∈ℒ 0 | 𝑑(𝑙, 𝑃 ) ≤ 𝜖} ∪ {𝑙∈ℒ 1 | 𝑑(𝑙, 𝑃 ∩ Π(𝑙)) ≤ 𝜖} //// Remember best candidate if | Λ | > | Λ best | then Λ best ←Λ, 𝑃 best ←𝑃 end if

	end repeat
	//// Update data structures
	Π ← Π ∪ {𝑃 best }	// Set of detected planes
	Λ(𝑃 best ) ← Λ

best // Support of best plane

  Number of images #img, number of 3D line segments | ℒ |, number of 3D planes before fusion | Π |, number of 3D planes after fusion | Π fus |, number of segments supporting no plane | ℒ 0 |, one plane | ℒ 1 | or two planes | ℒ 2 |, and total number of sub-segments #res.

			res Figure
	Andalusian	249 1234 160 148 242 597 395 14503 4.1
	DeliveryArea 948 1586 160 160 30 771 785 29222 4.2
	Barn	410 7936 160 141 41 2157 5738 83989 4.3
	TimberFrame 241 7268 140 131 264 4507 2497 79024 4.4
	Bridge	110 7437 150 102 338 4168 2931 48315 4.5
	MeetingRoom 32 831 135 130 25 383 423 9028	4.6
	Terrains	42 3223 120 105	9 356 2858 18189 4.7
	HouseInterior 159 1995 120 106	1 286 1708 18304 4.8

Table 4

 4

	.1:
	Dataset statis-
	tics

  Table 4.2. Besides, the normalizing factor 𝜎 is set to 1 m. 𝜖 𝜖 fus 𝜃 fus 𝑝 fus 𝑁 iter 𝑁 max 𝜆 vis 𝜆 edge 𝜆 corner

		Table	4.2:
	2 cm 3 𝜖 10°20% 50k 160 0.1 0.01 0.01	Parameters (all datasets
		have	metric
		dimensions).

  Sridhar et al. [SRV + 19] and report squared-symmetric Chamfer distance (multiplied by 100). Note that these metrics are point-based, not meshed-based (which our network outputs), in order to be comparable with XNOCS [SRV + 19].

	GT		ours		XNOCS
	GT		ours		XNOCS
	GT		ours		XNOCS
	GT	ours	XNOCS	GT	ours	XNOCS

  compared to baselines

		Airplane	Car	Chair	Avg.	Table	7.1:
	Method	F1↑	CD↓	F1↑ CD↓ F1↑	CD↓	F1↑ CD↓	Multi-view shape recon-
	Single template	36.7 0.42 23.3 0.53 13.3 1.04 24.4 0.66	struction	on
	XNOCS [SRV + 19] 62.3 0.08 33.2 0.19 27.2 0.20 40.9 0.16 Our full model 56.3 0.04 41.7 0.07 25.5 0.28 41.2 0.13	the ShapeNet-COCO [SRV + 19] validation set.
								We	report
								F1-score (F1)
								and	squared-
								symmetric
								100xChamfer
								distance (CD),
								averaged over
								each class.

  Synopsis. 3D Scene Reconstruction has important applications in industrial contexts, and the advent of Building Information Models (BIM) has made it possible to drastically improve the operational costs of existing buildings. While the community has mostly focused on surface reconstruction or semantic segmentation as separate problems, the joint reconstruction of both volumes and semantics has little been discussed, mostly due to the lack of large scale volume datasets with semantic annotations.In this work, we introduce a new dataset called VASAD, for Volume And Semantic Architectural Dataset. It is composed of 6 models, with full volume description and semantic labels. It represents approximately 62,000 m2 of building floors, making it large enough for the development and evaluation of learning-based approaches.

	[ Chapter 7
	Chapter 9 ]

  , TQD + 19]. On the other hand, advances have been made for surface reconstruction [CXG + 16, GFK + 18, MON + 19, PFS + 19] and more recently for large scene reconstruction [PNM + 20]. However few methods tackle the semantic volume reconstruction [SYZ + 17] task and when it comes to full building reconstruction, related work is even scarcer [MvAB + 20].

  To avoid using hand-crafted priors, recent methods tend to leverage big datasets of 3D objects [CFG + 15] and scenes [GLU12, BGM + 19, HSL + 17, ASZS17, RDG18] to directly learn meaningful priors. Early works use a voxel-based representation [CXG + 16] to apply similar neural network structures as in 2D. More recent works have enabled learned 3D reconstruction on meshes [GFK + 18, WZL + 18] and using implicit functions [PFS + 19, MON + 19, PNM + 20]. Another research direction[START_REF] Chen | BSP-net: Generating compact meshes via binary space partitioning[END_REF], leverages binary space partitioning, which generates simple piecewiseplanar closed meshes that are similar to the output of the computer-aided design editors typically used to build BIM models. These methods usually involve an encoder/decoder neural network architecture, and make an important use of the PointNet [QSMG17] point cloud encoder.

	Semantic segmentation of point clouds Early methods [NL13] formulate the point cloud
	segmentation problem as a region-growing [BJ88] problem, a model-fitting problem [SWK07],
	a clustering problem

  , BGM + 19] or nuScenes [CBL + 19], outdoor LIDAR, e.g., Semantic3D [HSL + 17] or NPM3D [RDG18], or indoor scenes, e.g., S3DIS [ASZS17], MatterPort3D [CDF + 17] or ScanNet [DCS + 17].

  In recent years, the construction industry has developed a new type of digital model called Building information model (BIM) for better building conception, maintenance and modification. While 3D geometric models have essentially been used for building conception, BIMs include a richer information: the semantic class of each building component (e.g., wall, ceiling, floor...), as well as other technical information. While this semantic data is usually created during the design of a new building, it is especially useful for existing buildings. However, manually creating such semantized models is costly. The main objective of our dataset is to show that it is possible to use BIM models available in the wild to learn semantic and volume priors to perform reconstruction from point clouds which are typically obtained from LIDAR scans.

			[ Chapter 8
	9.1	building information models	Chapter 10 ]

Table 9

 9

	.1: Se-
	mantic classes
	in VASAD along
	with their color
	scheme.

break end if end for OUTPUT: label for point 𝑝

  .2. INPUT: a 3D point 𝑝 and a mesh ℳ of the BIM model, structured as a set of object meshes ℳ 𝑠 with a semantic label label = label(void) // Initialize the point label to void (i.e., empty) queries ← {Ray(𝑝, e 𝑥 ), Ray(𝑝, e 𝑦 ), Ray(𝑝, e 𝑧 )}// Consider 3 orthogonal directions for ℳ 𝑠 𝑛 odd-inter ← 𝑛 odd-inter + 1 If at least two rays vote for being inside ℳ 𝑠 label ← label(ℳ 𝑠 ) > // Assign the label of ℳ 𝑠

	end if end for if 𝑛 odd-inter ≥ 2 do	point labeling from a labeled // Figure 9.2: 3D mesh algorithm.

𝑠 ∈ ℳ do // For every semantic instance ℳ 𝑠 in the global mesh 𝑛 odd-inter ← 0 // Initialize the counter of rays having an odd number // of intersections with the object mesh ℳ 𝑠 for query ∈ queries do if | query ∩ ℳ 𝑠 | mod 2 = 1 do // Count one for each query ray // having an odd number of intersections with ℳ

  As previous approaches essentially focused on surface reconstruction only, the task of reconstructing both the volume and the semantics that we propose is novel. We introduce two methods to perform volume reconstruction with semantics. The first one is a composite point-voxel model which extracts rich semantic features with a raw-point network and then handles semantic reconstruction with a voxel network. The second one is a direct extension of the state-of-art ConvONet method for surface reconstruction [PNM + 20] to handle semantics.

		10
		Method
			[ Chapter 9
	10.1	reconstruc tion approaches	Chapter 11 ]
			Artwork 4: Ex-
			terior overview
			and	interior
			details	of
			WestRiverSide-
			Hospital.

  ∈ 𝒫 𝑣 such that 𝑙 𝑝,𝑔𝑡 ≠ 𝑣𝑜𝑖𝑑 and 𝑙 𝑝,𝑝𝑟𝑒𝑑 ≠ 𝑣𝑜𝑖𝑑 | | 𝑝 ∈ 𝒫 𝑣 such that 𝑙 𝑝,𝑔𝑡 ≠ 𝑣𝑜𝑖𝑑 or 𝑙 𝑝,𝑝𝑟𝑒𝑑 ≠ 𝑣𝑜𝑖𝑑 | ] Confusion matrix: when working with classification tasks, the error is usually not evenly spread across the different classes. The best way to quantitatively visualize this is by building a confusion matrix 𝐶 whose term 𝐶 𝑖,𝑗 counts the number of points in 𝑝 ∈ 𝒫 𝑣 whose ground truth class is 𝑖 and whose predicted class is 𝑗. A perfect classification would therefore generate a diagonal confusion matrix. 𝐶 𝑖,𝑗 = | 𝑝 ∈ 𝒫 𝑣 such that 𝑙 𝑝,𝑔𝑡 = 𝑖 and 𝑙 𝑝,𝑝𝑟𝑒𝑑 = 𝑗 |

(1) NBU_OfficeBuilding Artwork 2: Exterior overview and interior details of NBU_Of-ficeBuilding.

Remerciements

(1)

(2)

(3) (4) (5) (6) (7) (1)

(2)

(3) (4) (5) (6) (7) (1)

(2)

(3) (4)

(5) (6) ulate this kind of acquisition given a mesh, some constraints have to be satisfied. First, viewpoints should not be inside a full volume. Moreover, in a realistic setup, viewpoints are spread across the model so that most surface points are visible from at least one viewpoint. Enforcing the latter condition is computationally expensive. We propose a heuristic in order to tackle this task with a cheaper computational cost. When a surface is not seen from any viewpoint, the area in the void close to this surface is also unseen from the viewpoints already stored, therefore, we sample viewpoint candidates in the void and check that they are not seen from the viewpoints already stored. Once we are unable to find such points, the viewpoint generation algorithm stops. A pseudo code for the algorithm is available on Figure 9

Ray shooting

From each viewpoint, directions are uniformly sampled to define a set of rays. The first intersection between each ray and the global mesh gives us the scanned point. We also save the normal of the surface at the scanned point, which can also be efficiently estimated in practice [START_REF] Boulch | Fast and robust normal estimation for point clouds with sharp features[END_REF], as well as the label of the intersected instance in the global mesh. This provides a more realistic point cloud than a uniform sampling on the global mesh. Not only does it prevents point sampling on invisible mesh surfaces, e.g. where two objects are in contact, but it also provides a typical non uniform distribution of points across the surface (see Figure 9.4).

(2) Sextant display purposes, and the quantitative evaluation was carried on the raw output data. In the figures representing the input point clouds, each point is colored darker as it gets closer to the viewer, in order to better perceive the underlying geometry. Again, this only has a visualization purpose and it does not carry any additional information.

The scanned point cloud is given to all the tested networks as input, but additional input data are also considered: the normal vector associated to each point, the visibility information (see Figure 10.2), and the per-point classification vector obtained thanks to FKAConv [START_REF] Boulch | FKAConv: Feature-Kernel Alignment for Point Cloud Convolution[END_REF]. Table 11.1 summarizes all the tested combinations of input.

We first report the scores for two baselines: the oracle and ConvONet [PNM + 20]. The latter is trained on our data by aggregating all the non-void semantic labels into a full label. The task being to segment the space between inside and outside the objects volume.

We train 2 versions of SemConvONet: first with raw point clouds as input, and second with additional normal information as an input to the PointNet encoders. Compared to the original, purely geometric algorithm, we notice that only the accuracy slightly worsens. The rest of the metrics tends to improve.

As an ablation study for our approach, we trained three models, not relying on point cloud semantics. This is tantamount to using the voxel sub-network of PVSRNet alone. Similarly to the SemConvONet experiment, we consider variants with and without the input normals at the voxel level. In spite of the voxel discretization inherently restricting the precision of the reconstructed surface (see the difference in reconstruction quality on the floor on Figure 11.2), our method does better than any version of ConvONet, whether semantics is used or not. Experiments also show that Convolutional Occupancy Network is very sensitive to the input sampling (cf. Figure 11.2, 4th row). As a matter of fact, experiments originally conducted by the authors of [PNM + 20] include a dense uniform sampling on the input meshes, as opposed to our dataset in which the input points are more realistically sampled from virtual point of views.

For information, we reported two experiments in which we included the visibility information, though PVSRNet did not seem to benefit from it. A possible explanation is The dataset is built from 6 building information models designed by various architects all of which can model a same building with slight differences. As previously mentioned in Section 11.3, the classes can have an inherent ambiguity, especially when only considering the point cloud as an input. A perspective for dataset improvement, besides augmenting it with new buildings, is to enrich the models with texture and/or material simulation. The LIDAR simulation could then be more sophisticated and produce intensity based on the