List of Tables

 [START_REF] Mehta | Ccd or cmos image sensor for photography[END_REF]. . . . . . . . . . . . . 2.3 Mobile laser scanning utilizes GNSS/IMU positioning for direct georeferencing of point clouds [START_REF] Kukko | Mobile Laser Scanning-System development, performance and applications[END_REF] . . . . . . . . . . . . 2.4 The pulse sensor topology forms a 6-neighborhood: the points considered is colored in red, and connection is denoted by a red arrow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Echo sensor topology: each echo is connected to all echoes of all neighboring pulse, the points considered is colored in red, and connection is denoted by a red arrow . . . . . . . . . . . . . . . . . . . 2.6 Schematical illustration of ALS data acquisition [START_REF] Roncat | The geometry of airborne laser scanning in a kinematical framework[END_REF] . . . TLS Terrestrial Laser Scanning.

Acronyms

UAV Unmanned Aerial Vehicle.

Abstract

Indoor/Outdoor modeling of buildings is an important issue in the field of building life cycle management. It is seen as a joint process where the two aspects collaborate to take advantage of their semantic and geometric complementary. This global approach will allow a more complete, correct, precise and coherent reconstruction of the buildings. This thesis is part of the Building Indoor/Outdoor Modeling (BIOM)

ANR project that aims at automatic, simultaneous indoor and outdoor modelling of buildings from image and dense point clouds. The first ambition of the BIOM ANR project is to integrate heterogeneous data sources for buildings modeling. The heterogeneity is both in: data type (image/ LiDAR data), acquisition platform (Terrestrial/ Aerial), acquisition mode (dynamic/static) and point of view (indoor/outdoor).

The first issue of such modeling is thus to precisely register this data. The work carried out has confirmed that the environment and the type of data drive the choice of the registration algorithm. Our contribution consists in exploiting the physical and geometric properties of the data and the acquisition platforms in order to propose potential solutions for all the registration problems encountered by the project. As in a building environment, most objects are composed of geometric primitives (planar polygons, straight lines, openings), we chose to introduce registration algorithms based on these primitives. The basic idea of these algorithms consists in the definition of a global energy between the extracted primitives from the data-sets to register and the proposal of a robust method for optimizing this energy based on the RANSAC paradigm. Our contribution ranging from the proposal of robust methods to extract the selected primitives to the integration of these primitives in an efficient registration framework. Our solutions have exceeded the limitations of existing algorithms and have proven their effectiveness in solving the challenging problems encountered by the project such as the indoor (static mode)/outdoor (dynamic mode) registration, image/LiDAR data registration, and aerial/terrestrial registration.

Key words

Registration, Geometric primitives, Planar polygons, 3D Segments, Openings, Global robust energy, Optimization, RANSAC.

Résumé

La modélisation intérieure/extérieure des bâtiments est un enjeu important dans le domaine de la gestion du cycle de vie des bâtiment. Elle est vue comme un processus conjoint où les deux aspect collaborent pour tirer partie de leurs complémentarités sémantiques et géométriques. Cette approche globale permettra une reconstruction complète, correcte, précise et cohérente des bâtiments. Cette thèse s'inscrit dans le cadre du projet ANR BIOM (Modelisation Intérieur/Extérieur de Bâtiments) qui vise à la modélisation automatique et simultanée de l'intérieur et de l'extérieur de bâtiments à partir d'images et de nuages de points denses. La première ambition du projet ANR BIOM est d'intégrer des sources de données hétérogènes pour la modélisation des bâtiments. L'hétérogénéité est à la fois dans :

• Le type de données (image/données LiDAR).

• La plate-forme d'acquisition (Terrestre/Aérienne).

• Le point de vue (intérieur/extérieur).

• Le mode d'acquisition (dynamique/statique)

Les approches de l'état de l'art traitent en général soit l'extérieur, soit l'intérieur La maximisation de cette énergie conduira à une maximisation de chevauchement et • Recalage du LiDAR aérien et LiDAR terrestre en utilisant soit l'algorithme basé sur les polygones planaires soit l'algorithme basée sur des segments 3D.

• Recalage du LiDAR aérien et l'image terrestre en utilisant l'algorithme basé sur les segments 3D.

• Recalage de l'image aérienne et l'image terrestre en utilisant l'algorithme basé sur les segments 3D.

Mots clés

Recalage, Primitives géométriques, Polygones planaires, Segments 3D, Ouvertures, Energie globale robuste, Optimisation, RANSAC.

I would like to dedicate my thesis to my beloved parents.

Acknowledgements

This thesis has been a rich and unique experience for me that cannot be completed Chapter 1

Context and research problem 1.1 Introduction

Our thesis is part of the Building Indoor/Outdoor Modeling (BIOM) project that aims at automatic, simultaneous indoor and outdoor modelling of buildings from heterogeneous data. The heterogeneity is both in data type (image and Light Detection and Ranging (LiDAR)) and acquisition platform: terrestrial indoor/outdoor or aerial acquisition. The objective is to achieve a complete, geometrically accurate, semantically annotated but nonetheless lean 3D CAD representation of buildings and objects they contain in the form of a Building Information Modeling (BIM) that will help manage buildings in all their life cycle (renovation, simulation, deconstruction). The first issue of such modeling is thus to precisely register the data. The work carried out has confirmed that the environment and the type of data drive the choice of the registration algorithm. So, the objective of this thesis is to explore fundamental properties of the data and the acquisition platforms in order to propose potential solutions for all the registration problems encountered by the BIOM project.

BIOM project

The Building Indoor/ Outdoor Modelling (BIOM) project aims at automatic, simultaneous indoor and outdoor modelling of buildings from images and dense point clouds. The goal of the project is to achieve a complete, geometrically accurate, semantically annotated but nonetheless lean 3D CAD representation of buildings and objects they contain in the form of a Building Information Model (BIM) that will help manage buildings in all their life cycle (renovation, simulation, deconstruction). We view indoor and outdoor building modelling as a joint process where both worlds fruitfully cooperate and benefit one another both in terms of semantics and geometry. The hope is that this holistic scene understanding and reconstruction approach will lead to more complete, correct, and geometrically accurate building models.

The first challenge of the BIOM project is to accommodate for heterogeneous data as full building modeling calls for data acquisition inside and outside the building

Objectives

The aim of this PhD is to address all registration issues faced by the BIOM project, as a major objective of the project is exploiting jointly data from various sensors (images/LiDAR) and viewpoints (terrestrial indoor/outdoor, aerial). Four key objectives for the BIOM project have been identified:

• Registration of outdoor heteregeneous data.

• Registration of indoor data.

• Indoor/outdoor landmark extraction.

• Global indoor/outdoor registration.

• Image/LiDAR registration.

Building Information Modeling

A Building Information Modeling (BIM) represents a comprehensive digital representation of a built facility with great information depth. It is based on parametric CAD technology and represents a new approach of "Virtual Building Construction" [START_REF] Woo | Bim (building information modeling) and pedagogical challenges[END_REF]. The basic principle of BIM is to share knowledge resource for information about a facility forming a reliable basis for decisions during its life-cycle; defined as existing from earliest conception to demolition [START_REF] Borrmann | Building information modeling: Why? what? how?: Technology foundations and industry practice chapter 1 building information modeling-why? what? how?[END_REF]. The collaboration by different stakeholders at different phases of the life cycle of a facility can help to insert, extract, update or modify information in the BIM to support and reflect the roles of that stakeholder.

BIM has a great potential for operation and maintenance (O&M) and facility management (FM) of buildings. However, most of the existing buildings have no BIM.

Research in creating BIM for existing buildings has received growing attention in recent years. But, it remains a challenging task [START_REF] Hossain | Bim for existing buildings: potential opportunities and barriers[END_REF]

Registration problem

The principle of registration according to [START_REF] Monnier | Mise en cohérence de données laser mobile sur un modèle cartographique par recalage non-rigide[END_REF] consists in making at least two sets of data geometrically consistent. Generally, one of the data sets serves as a reference on which the second is registered. The goal is to determine the transformation necessary to best reconcile the data with each other. For this, we can summarize a registration problem in two important steps. The first consists in extracting common features in the datasets. The second consists in using these features in order to determine the optimal transformation to apply.

Images registration

Registration is a fundamental task in image processing used for aligning two or more images of the same scene with reference to a particular image. These images can be taken at different times, from different sensors, or from different viewpoints. Image registration became a large research axis in medical sciences, remote sensing and in computer vision.

The main image registration steps

According to [START_REF] Nag | Image registration techniques: a survey[END_REF], the following steps are necessary to solve an image registration problem 1. Feature detection:

It is the key task of the Image Registration process which can be realised automatically or manually. These features must be easily detectable and physically interpretable and identifiable.

Feature matching:

This step essentially establishes the correspondence between the features detected in the images that we want to register them. Different feature descriptors and similarity measures besides spatial relationships among the features are adopted to set up an accurate accordance.

Transformation estimation:

For alignment of the sensed image with the reference image the parameters of the geometric transformation must be estimated. These parameters are computed with the established feature correspondence obtained from the previous step.

Criteria of image registration techniques

A registration model can be characterized by the following elements:

1. Dimensionality:

This specifies the dimensions of different possible registrations. It may be 2D-2D, 2D-3D or 3D-3D.

Domain of transformation:

It can be global when the whole image must be registered or it can be local when only a part of the image is taken into consideration for the registration.

Geometric transformation:

It may be rigid (translation, rotation, reflection), affine (translation, rotation, scaling, reflection, shearing), projective or non-linear.

Attributes:

These are extracted features from images that can guide the registration process.

Inter-image distance:

It is a function that can be used to measure a distance between the attributes of the two images to be registered. This distance must be minimum when the similarity between the two images is maximum.

Optimization method:

The last fundamental aspect of a registration algorithm is the choice of the optimization method whose goal is to determine the best transformation that minimizes or maximizes a cost function. In optimization, the convexity of a criterion is a desirable property.

Features detection approaches

The choice of the registration features is an essential step which is made according to three approaches [START_REF] Vincent | Recalage d'images médicales[END_REF].

Iconic approach

Iconic methods can be considered low-level approaches. Iconic methods use the dense information carried by the gray levels of the image.

Geometric approach

This approach consists in extracting common geometrical characteristics in the images, which are called primitives (e.g: points, curves). These units carry high level information. The choice of the primitives is based on certain criteria:

• Easy feature detection.

• Distribution of primitives over the entire image.

• Robustness to noise, artifacts, changes related to acquisition.

Hybrid approach

This method consists of combining several different types of attributes. We can distinguish three types of combination: the combination of geometric primitives of different natures, the combination of different information from gray levels and the combination of geometric and iconic approaches.

Point clouds registration

The need to digitally represent the world around us has led to the development of technologies to scan an environment in three dimensions. Scanning methods produce a set of 3D points called "point cloud". Point clouds has become the primary data format to represent the 3D world with the fast development of high precision sensors such as LiDAR. Because the sensors can only capture scans within their limited view range, we need several points of view to cover the scene of interest. The point clouds from these different points of view are each defined in a reference relative to the sensor. So a registration is required to generate a large 3D scene. The aim of registration is to grouping all point clouds in a common coordinate system. There are two main families of registration methods depending on the number N of input point clouds: pairwise (N = 2) and multi-view (N > 2) registration.

Pairwise registration

We can consider pairwise registration as a special case of multiview registration where only one pair of overlapped views is considered. According to [START_REF] Gojcic | Learning multiview 3d point cloud registration[END_REF], the traditional pairwise registration pipeline consists of two stages: the coarse alignment stage, which provides the initial estimate of the relative transformation parameters and the refinement stage that iteratively refines the transformation parameters by minimizing the 3D registration error under the assumption of rigid transformation.

Multiview registration

Multiview registration can be formulated as an optimization problem based on incorporating cues from multiple views. The goal of optimization is to find a transformation for each view such that the overlapping regions of the transformed views are aligned with each other as closely as possible. This kind of methods aim at resolving hard or ambiguous cases that arise in pairwise methods. If the transformations are restricted to be rigid, the problem is characterized as multiview rigid registration.

Multiview registration is more complicated than pairwise registration. The simple integration of local pairwise alignments will lead to the loop closure problem. Thus, a global method should be adopted. The process of solving the loop closure problem is also called loop-closing. Certain global methods take advantage of local pairwise alignments and perform loop-closing by diffusing the transformation errors in the initially aligned view pairs [START_REF] Tang | Hierarchical multiview rigid registration[END_REF].

Contributions

The work carried out has confirmed that the environment and the type of data drive the choice of the registration algorithm.

Our contribution consists in exploring the fundamental properties of the data and acquisition platforms in order to propose primitives based registration algorithms able to solve all registration problems encountered by the BIOM project.

The basic idea of these algorithms consists in the definition of a global robust energy between the extracted primitives from the data-sets to register and the proposal of a robust method for optimizing this energy based on the RANSAC paradigm. In a building environment, most objects are composed of planar surfaces delimited by straight lines. In addition, we can represent a building as a set of fac ¸ades, where the openings are the most obvious common elements of these entities. So, our selected primitives for this work are: planar polygons, 3D segments and openings (vertical rectangles).

To extract these primitives:

• We have proposed an efficient extraction algorithm (openings detection).

• We have adapted existing methods (planar polygons detection).

• We have improved an existing method (3D segments detection and reconstruction from images).

• We have directly used an existing method (3D segments detection from LiDAR data).

Our contribution has two aspects: methodological aspect and application aspect as shown in Figure 1.2. • Planar polygons based registration algorithm.

• Openings based registration algorithm.

• Hybrid registration algorithm.

• 3D segments based registration algorithm.

Our application aspect consists in using the proposed algorithms to solve the registration problems encountered by the BIOM project. All the developed algorithms as part of our thesis follow one of the three possible indicated schemes in the Figure 1.3 to optimize the global robust energy. The choice of the schema to follow depends on the type of the geometric primitives (heterogeneous, homogeneous) extracted in each data-set as well as the quantity.

• If the two data-sets contain few homogeneous primitives, we can directly apply our RANSAC.

• If the two data-sets contain many homogeneous primitives, we can cluster them and associate the obtained clusters to simplify our RANSAC.

• If the two data-sets contain heterogeneous primitives, we must associate them in each data-set before applying our RANSAC. The main registration problems encountered by the BIOM project are as follows.

Indoor/outdoor registration problem

The registration of indoor and outdoor scans with a precision reaching the level of geometric noise represents a major challenge for Indoor/Outdoor building modeling.

The lack of overlap between indoor and outdoor data is the most encountered obstacle, more so when both data sets are acquired separately and using different types of sensors. Among our proposed algorithms, we can consider the following algorithms as potential solutions of this challenging problem:

A planar polygons based algorithm

It is an efficient algorithm based on polygon detection and matching. Its strong points lie in the fact that it exploits with the very small overlap between indoor and outdoor scans of the same building by extracting points lying in the buildings' interior from the outdoor scans as points where the laser ray crosses detected fac ¸ades. It follows the RANSAC paradigm and uses a global robust energy in order to select the best solution.

An openings based algorithm

This proposed algorithm is based on openings detection and matching. An opening is considered among the common entities that can be seen from inside and outside.

It is a rectangular shape composed of four segments. The algorithm registers the detected openings from both inside and outside by minimizing a global robust energy between the corresponding segment sets. This minimization is based on the RANSAC paradigm.

Hybrid registration algorithm

The two previous algorithms are efficient to deal with indoor/outdoor registration problem, but they have some limitations. In order to exceed these limitations, we have combined them. This combination has produced a very efficient hybrid algorithm. This new algorithm has proven its performance to solve the indoor/outdoor registration problem.

Image/LiDAR data registration problem

Image/LiDAR data registration is a very difficult problem given the very strong difference in modality. This difference makes it difficult to extract comparable features between the two modalities. Conversely, with our primitives and our RANSAC based approach, we assume that the best transformation is the one which optimizes the global robust energy between all the extracted primitives and therefore which is the most consistent with all the information of the datasets to be registered. Our 3D segments based registration algorithm represents a potential solution for this regis-tration problem. This algorithm based on the definition of a global robust energy between two segments sets and the minimization of this energy according to the RANSAC paradigm.

Aerial/Terrestrial registration problem

The aim of our thesis is not only registration of heterogeneous data (image and LiDAR data) but also the registration of data acquired by heterogeneous acquisition platforms (aerial/terrestrial). By exploring our previous solutions, we can deal with this problem in the following forms:

• For the aerial/terrestrial registration problem, we chose to focus on the particular problem of aerial image / terrestrial image registration for the following reasons:

• we have already studied the LiDAR data /LiDAR data registration in the first part of our thesis (chapter 4).

• we have already studied the image/LiDAR data registration in the second part of our thesis (chapter 5).

• As the 3D segments based algorithm has proven its effectiveness in solving the problem of image/LiDAR data registration and LiDAR data /LiDAR data registration (openings based registration), we hoped to test it in solving the image/image registration problem.

We have adapted our 3d segments based algorithm to solve this problem.

Thesis outline

This thesis is organized as follows:

• Chapter 2: Data description This chapter is dedicated to the description of the data used for this study (image/LiDAR data) and the acquisition platforms.

• Chapter 3: Primitives detection

This chapter is divided into two main parts: A first part is dedicated to the classification of primitive extraction methods, followed by an explanation of the criteria for evaluating the performance of algorithms. A second part deals with the methods of extraction of the geometric primitives which interest us, namely the planar polygons, the 3D line segments and the openings.

• Chapter 4: Indoor/Outdoor Registration This chapter deals with the indoor/outdoor registration problem. We start with a state of the art followed by the proposal of two initial solutions (a planar polygon based solution and an openings based solution. Then, we combine these two solutions to overcome their limitations.

• Chapter 5: Image/LiDAR data Registration

This chapter deals with the problem of heterogeneous data (Image/LiDAR data) registration. We start by classifying some existing methods according to their types and attributes, citing the advantages and limitations of each category.

Then, we explain our 3D segments based algorithm.

• Chapter 6: Aerial/Terrestrial registration This chapter has been approached the problem with the registration of data acquired by heterogeneous acquisition platforms (aerial/terrestrial). We start by illustrating some existing works followed by the adaptation of the 3D segments based algorithm to solve this kind of problem.

• Chapter 7: General Conclusion

This chapter reflects on the contributions of the thesis and suggests on directions of future works.

Publication List

Data description 2.1 Introduction

Generally, to solve any problem, you have to choose the appropriate algorithm to apply. This choice should not be random. It must be guided by the type of data and the environment. The aim of our thesis is the development of potential algorithms for heterogeneous data registration in building environment. The heterogeneity is both in data type (image and LiDAR) and acquisition platform: terrestrial indoor/outdoor or aerial acquisitions. To achieve this objective, we must understand and study the fundamental properties of this data, the acquisition platforms and the environment.

Image data

Image definition

There is no universal definition of an image. According to [START_REF] Mishra | Image acquisition and techniques to perform image acquisition[END_REF], an image is a 2D function f (x, y) where (x, y) is the coordinate in two dimensional space called a pixel, which represents the smallest unit of the image , and f is the color at that coordinate. A digital image is a rectangular array of pixels also called a Bitmap.

Pixel is an abbreviation of picture element. Each pixel stores a color at this point in the image. In general, digital images have two types of color representation:

• Binary images assign only one of two values to a single pixel: 1 or 0 (black or white) and are used to represent geometric sets or image masks.

• Black and white images, also called grayscale or panchomatic images are made of different shades of gray corresponding to the luminosity received by the sensor on the full visible spectrum (400 to 800nm). These different shades are usually sampled on 8 bits between 0 (black) and 255 (white) and intermediate values correspond to different shades of gray.

• Color images are made up of colored pixels. The spectrum -the band of colors produced when sunlight passes through a prism -includes all primary colors, of which the human eye can perceive seven to ten million. The electronic capture and display of color is commonly based on the RGB (Red, Green, and Blue) color system which represents each channel on 8 bits (from 0 to 255) leading to a 24-bit representation that defines 16.8 million colors.

Image acquisition model

The images are generated by combination of a light source and the reflection or absorption of the energy by the elements of the scene of interest. In a general way, to capture the image, we use a sensor according to the nature of the light. The light energy is transformed into digital image using the sensor. By combining the input electrical energy and sensor material, that is responsive to the particular energy that is being detected, incoming illumination energy is transformed into voltage [START_REF] Mishra | Image acquisition and techniques to perform image acquisition[END_REF]. The output waveform is the response of the sensor and this response is digitized to obtain a digital image as shown in Figure 2.1.

Techniques to perform image acquisition

Image Acquisition process totally depends on the hardware system. An image sensor converts an image into a digital signal. Nowadays, there are two main technologies which is used in image sensor: Complementary Metal Oxide Semiconductor (CMOS) and Charge Coupled Device (CCD).

CMOS

According to [START_REF] Mehta | Ccd or cmos image sensor for photography[END_REF], the working principle of CMOS sensor is the following: photons from objects are falls on the photodiode and it converts it into the charges. Then this charge will apply to the capacitor that will convert the charges into the voltages. Finally Analog to digital converter will convert the voltages into the digital signal as shown in

CCD

According to [START_REF] Mishra | Image acquisition and techniques to perform image acquisition[END_REF], CCD image sensors convert light into electrons .

Once the light is converted into electrons, it reads the value (accumulated charge) of each cell in the image. A CCD transports the charge across the chip and reads it at one corner of the array. An analog-to-digital converter (ADC) then turns each pixel's value into a digital value by measuring the amount of charge at each photosite and converting that measurement to binary form. The Table 2.1 studies the advantages and limitations of the two types of image sensors. This study is taken from [START_REF] Mishra | Image acquisition and techniques to perform image acquisition[END_REF] Overall, CMOS sensors are much less expensive to manufacture than CCD sensors, are rapidly improving in performance. They traditionally consume little power. CCD sensors consume as much as 100 times more power than an equivalent CMOS sensor.

CCD sensors have been mass produced for a longer period of time, so they are more mature. They tend to have higher quality pixels, and more of them [START_REF] Mishra | Image acquisition and techniques to perform image acquisition[END_REF].

LiDAR data

LiDAR concept

LiDAR is an acronym for light detection and ranging. It refers to a remote sensing technology that emits intense, focused beams of light and measures the time it takes for the reflections to be detected by the sensor. This information is used to compute ranges, or distances, to objects. In this manner, LiDAR is analogous to radar (radio detecting and ranging), except that it is based on discrete pulses of laser light [START_REF] Carter | An introduction to LiDAR technology, data, and applications[END_REF]. As LiDAR uses its own laser light energy source to illuminate an area of the atmosphere or the targets to be studied, it is considered an active optical analysis method [START_REF] Flamant | Comprendre la science du LiDAR[END_REF].

LIDAR principle

According to [START_REF] Mehendale | Review on lidar technology[END_REF], LiDAR employs the method of sending laser light on to the target and measuring the reflected light back to recognize the variation in wavelength and arrival time of the reflected light. The measurement is done by calculating the laser return times and their wavelengths. It generates precise, high quality, and sometimes even a three-dimensional map of the environment that it scans. The generated map of the area in focus, helps in its characterization and examination. A typical LiDAR system consists of a scanner, laser, and sometimes specialized GPS receiver as well. Other elements that are essential for data collection and analysis are optics and photodetectors.

The design principle behind LiDAR is the reflection of light. This principle is to shine a light beam on to a surface and calculate the time it takes to return to its source.

The LiDAR system sends laser light on to the target and measures the reflected light to see the variation in wavelength and arrival time of the reflected light. From these measurements, it can calculate the distance to draw the digital representation of the target. Since light travels at a very high speed, the calculation of the exact distance through LiDAR is very fast. The formula that the analysts use to calculate the distance is given in equation (2.1)

D = c ∆T 2 (2.1)
where, D = The distance of the object. c = Speed of light.

∆ = Time required by the light to travel.

The LiDAR system fires many laser lights on to the surface. The sensor on the system measures the time taken for the reflected light to reach the sensor. This goes on repeating until a complex map of the surface is constructed.

LiDAR acquisition platforms

LiDAR technology can perform in two forms: static (Terrestrial laser scanning) and dynamic (Mobile laser scanning, Mobile mapping system, Airborne Laser Scanning and Unmanned Aerial System).

Terrestrial laser scanning

Terrestrial laser scanning (TLS), also referred to as terrestrial LiDAR (light detection and ranging) or topographic LiDAR, is a technique which uses laser light in order to measure with high speed in a dense regular pattern directly 3D coordinates of points on surfaces and surface brightness from terrestrial position [START_REF] Pfeifer | Overview of TLS systems, overall processing and applications, theory and application of laser scanning[END_REF]. This measurement is realized by emitting laser pulses toward the target and measuring the distance between it and the device [START_REF] Dumbrell | Advances in Ecological Research[END_REF]. The number and variety of applications of TLS instruments continue to increase.

By emitting laser pulses toward numerous points on land, TLS can acquire their XYZ coordinates. This acquisition is based on measuring the distance from the device to the target. The table 2.2 presents the main characteristics and performances of terrestrial laser scanners.

Terrestrial laser scanning (TLS)

Absolute accuracy 1 cm at distance 100 m Range

From 100 m to 6 km

Acquisition time

Variable according to the monitored surface (15 mn for a standard application) Acquisition frequency ≤ 300 kHz Wavelength 0.78 µm to 1 mm

Field of vision 360°horizontally and up to 80°vertically

Equipment weight

Between 20 and 45 kg Table 2.2: Main characteristics and performance of terrestrial scanner solutions (Coccia, 2021)

Mobile laser scanning

Mobile mapping system (MMS): According to [START_REF] Kukko | Mobile Laser Scanning-System development, performance and applications[END_REF], we can consider a mobile mapping system as a kinematic surveying system utilizing motion With state-of-the-art sensors, the achieved point clouds capture object details with good accuracy and precision [START_REF] Kukko | Multiplatform mobile laser scanning: Usability and performance[END_REF]. In general, the spatial coverage of data in MLS is achieved by the movement of the vehicle and motion-tracking positioning devices, as shown in Sensor Topology: The sensors used to capture point clouds often have an inherent topology. According to [START_REF] Guinard | Sensor-topology based simplicial complex reconstruction from mobile laser scanning[END_REF], Mobile Laser Scanners sample a regular grid in (θ,t) where(θ) is the rotation angle of the laser beam and (t) the instant of acquisition. As the vehicle moves at a variable speed and can turn, the sampling is not completely uniform in space. For a 2π rotation in θ the pulses number N p is not an integer, so it considers that a pulse of P i has six neighbors as shown in Figure 2.4. However, this topology concerns emitted pulses, not recorded echoes.

One pulse might have 0 echo (no target hit) or up to 8 as most modern scanners can record multiple echoes for one pulse if the laser beam intersected several targets.

They approached the problem by connecting an echo to each of the echoes of its pulses' neighbors as shown in Figure 2.5. This idea is a potential solution to keep all possible edge hypotheses before filtering them. produces high-resolution topographic data and allows very precise three-dimensional mapping of the surface of the earth [START_REF] Mlekuž | Airborne laser scanning and landscape archaeology[END_REF]. With airborne laser scanning, it is possible to determine the plane-ground distance and the inclination angles of the scanning platform. These tasks can be performed by frequent sending of short laser pulses from a flying platform (plane) towards the ground and measuring the return times of the reflected signal [START_REF] Kurczyński | Airborne laser scanning in poland-between science and practice[END_REF]. The trajectory of the plane can be measured using both GPS and inertial measurement.

The Figure 2.6 shows a detailed illustration of ALS data acquisition.

Unmanned Aerial System

New ways to perform laser scanning surveys more cost-effectively are offered by Unmanned Aerial Systems (UASs), which opens doors to many new change-detection applications. According to [START_REF] Kukko | Airborne laser scanning using uass[END_REF], UASs can generally be categorised into two types:

• Fixed-wing systems provide users with a longer operation time and support larger payloads due to better fuel economy. They allow for more speed, which makes this type of UAS more suitable for large areas or long-distance missions.

But They are less favourable for small-area surveys with complex terrain or objects rich in features that need to be captured.

• Rotorcrafts allow for slow or even stationary flight speeds and offer excellent manoeuvrability. They do not require an airstrip for take-off and landing.

Therefore, they enable the use of relatively low-cost sensors.

Conclusion

Data are the basic and input elements of any processing chain. They guide the choices of the algorithms to be applied in order to solve the studied problem. In this chapter,

we have focused our study on the fundamental properties of the data used in the context of our thesis (image and LiDAR). This study is essential to defend the choice of the proposed solutions in the next chapters. By grouping a set of points, we can create a more complex element which represents a geometric primitive. In a building environment, most objects are composed of planar surfaces delimited by straight lines. In addition, we can represent a building as a set of facades, where the openings are the most obvious common elements of these entities. So, we must effectively and precisely extract these components to be able to use them as attributes for more complex data processing such as registration and modeling. This chapter is divided into two main parts: A first part is dedicated to the classification of primitive extraction methods, followed by an explanation of the criteria for evaluating the performance of algorithms. A second part deals with the methods of extraction of the geometric units which interest us, namely the planar polygons, the lines, and the openings.

Classification of primitives extraction methods

Several methods have been proposed in the literature to deal with the problem of detecting primitives in 3D point cloud. We can classify these methods in four categories.

Clustering based methods

The clustering is a way for extracting geometric primitives from a 3D point cloud.

This kind of algorithms is in the category of unsupervised learning and requires in most cases a post-processing step to refine the results. The most used algorithms for this kind of methods are:

• Region growing (RG):

It consists in segmenting the 3D data using neighborhood information. It groups iteratively nearby points until some criterion is reached. The proximity may be based not only on spatial distance but also on similarity between local features [START_REF] Hojjatoleslami | Region growing: a new approach[END_REF].

• K-Means:

It consists in fixing a number k of clusters and randomly selecting k points as seeds. Points of the cloud are grouped according to the seed they are closest to based on some distance metric. These seeds move to the centroid of their group that represent the centroid of the detected clusters. The process iterates the steps assignment/motion until nothing changes [START_REF] Yadav | A review of k-mean algorithm[END_REF]).

• J-Linkage:

This method deals with the problem of fitting multiple instances of a model in the presence of noise and outliers. The characteristic function of the set of random models that fit the point is associated to each point. Then, points having a similar characteristic function are clustered using a tailored agglomerative clustering (J-linkage). The method does not require adjustment of the parameters except the consensus threshold [START_REF] Toldo | Robust multiple structures estimation with Jlinkage[END_REF].

Vote accumulation based methods

This family of method works in parameter space and needs a voting step in most cases. The Hough Transform (HT) has been proposed in [START_REF] Duda | Use of the Hough transformation to detect lines and curves in pictures[END_REF] and it represents the standard of the accumulation algorithms. It constructs an accumulation space on a parameter sampling in which similar geometric elements coincide.

The object candidates are obtained as local maxima in the accumulation space.

Hypothesis and validation based methods

This family of methods consists in the generation of primitive guesses and the selection of the best one using a validation criterion (cost function, energy. . . ). The random sample consensus (RANSAC) is the most representative of this family of methods. It is a simple and powerful tool for 3D data segmentation. It is known to be particularly robust to outliers, provided sufficient samples are drawn. RANSAC was proposed in [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF]. The algorithm is iterative and based on a random selection of a minimal or small subset of samples, which is used for estimating the parameters of a model fitting them. Then, using its proper loss function (originally the number of outliers of the model), it obtains a score for the tested model, i.e., how well the model fits to the data. Several variants of the RANSAC paradigm have been proposed in order to improve its performance. [START_REF] Torr | MLESAC: A new robust estimator with application to estimating image geometry[END_REF] proposed two modifications of RANSAC called: MSAC (M-estimator SAmple and Consensus) and MLESAC (Maximum Likelihood Estimation SAmple and Consensus). The difference between RANSAC and these generalizations lies in the cost function.

Selection and decision based methods

Most detectors encounter multiple challenges such as: parameter tuning, control of false detection and execution time. The a contrario approach proposed in [START_REF] Moisan | From gestalt theory to image analysis[END_REF] A comparison between these four classes of methods has been carried out in Table 3.1, which underlines the advantages and limitations of each class of methods.

Performance evaluation

The most used metrics to evaluate the performance of a primitive detection algorithm are: the accuracy, timing and robustness.

Accuracy

The accuracy of the detection is obtained by comparing ground truth with the obtained results. If the error between the ground truth and the obtained result increases, the accuracy decreases and vice versa.

Robustness

A good algorithm must be robust to noise, incomplete data and outliers.

• Robustness to noise:

The robustness to noise can be measured by the stability of the detection accuracy if the dataset contains noise.

• Robustness of incomplete data:

For some acquisition tools, the output data may have missing parts. So the robustness of the algorithm is represented by its capacity to complete the shape if some parts of it are not captured, such as polygons with holes.

• Robustness to outliers:

The robustness to outliers can be measured as the ability of the algorithm to distinguish between points coherent with the model (inliers) from the others (outliers).

Timing

This criterion is related to the speed of the algorithm execution and it depends on the data size and computing power of the devices. The data size is the representative size of the datasets targeted by the algorithm. We will only consider algorithms that process the input data in less than one minute.

Planar polygons extraction

3D data can be represented as a 3d point cloud. This point cloud can be unorganized (e.g., LiDAR data) or organized (e.g., range images). The high computation and memory costs of dense point cloud processing requires us to go through a concise and meaningful abstraction of the data. In man-made environments where most objects are represented by planar surfaces, the planar polygons are an adequate abstraction of the data.

State of the art

Modeling a 3D point cloud as a set of planes is known as the plane fitting problem.

Plane fitting is a very active field of research. Hough transform, random sample consensus (RANSAC) and region growing are the most popular approaches to detect planes in point clouds. The RANSAC algorithm [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF]) represents a potential solution to iteratively find an accurate model for observed data that may contain a large number of outliers and noise. An adaptation of RANSAC to the plane fitting problem has been proposed in [START_REF] Schnabel | Efficient ransac for point-cloud shape detection[END_REF]. It ensures a precise and fast extraction of the plane provided that the parameters are properly adjusted. Hough transform(HT) [START_REF] Borrmann | The 3d Hough transform for plane detection in point clouds: A review and a new accumulator design[END_REF] is also a robust method to detect plane. The high computation cost is its main limitation. Several improvements have been proposed to overcome this limitation. A new approach has been proposed in [START_REF] Limberger | Real-time detection of planar regions in unorganized point clouds[END_REF]. It works by casting votes for a set of clusters on a spherical accumulator using a trivariate Gaussian kernel. The clusters are generated by clustering sets of approximately co-planar points. An efficient triple-region growing method has been proposed in [START_REF] Hu | An automatic 3d registration method for rock mass point clouds based on plane detection and polygon matching[END_REF] for plane detection in rock mass point clouds. According to the bounding box and specified resolution, this method starts by dividing point clouds into small voxels. Afterwards, these voxels are regarded as growing units (seeds) and clustered into completed planes by region growing. Finally, the corresponding polygons are calculated by concave hull method.

A new hypothesizing and selection strategy based framework has been proposed in [START_REF] Wonka | Polyfit: Polygonal surface reconstruction from point clouds[END_REF]. The intersection of the extracted planar primitives has been performed in order to generate a reasonably large set of face candidates. Finally, an optimal subset of the candidate faces is selected through optimization. Polylidar3D

is a concave polygon extraction method from 3D data which has been proposed in [START_REF] Castagno | Polylidar3d-fast polygon extraction from 3d data[END_REF]. This method perfoms by modelling the 3D point cloud as a polygonal (triangular) mesh. The obtained concave polygons represent flat surfaces in an environment with interior cutouts representing obstacles or holes.

The interest of polygons compared to planes are the following:

• The memory space required is considerably reduced (in terms of number of points).

• They enable the calculation of certain relevant features, for example, centroid, area, perimeter.

• The polygons have a spatial extent limited to areas where they have support points in input data. They therefore form a simple and compact summary of our scans.

Our proposed solutions

Due to its robustness to noise and outliers, Random Sample Consensus (RANSAC) has become the most popular method for LiDAR point cloud segmentation. Despite this success, it can generate false segments consisting of points from several nearly coplanar surfaces. False planes made up of points from different planes or roof surfaces represent a real obstacle for RANSAC [START_REF] Xu | Investigation on the weighted ransac approaches for building roof plane segmentation from LiDAR point clouds[END_REF]. In order to overcome the limitations of RANSAC, we have exploited two methods depending on the nature of the data. tor. We use for that a recent method [START_REF] Guinard | Planar polygons detection in Lidar scans based on sensor topology enhanced RANSAC[END_REF] that exploits the sensor topology to extract compact planar patches instead of planes:

RANSAC Based on Sensor Topology

• Sample selection: as we are looking for compact planar patches, once a first sample point is drawn randomly from the point cloud, the other two are drawn • Region growing: instead of computing the distances of all the points of the cloud to the hypothetical plane, a region is grown starting at the first sample in order to recover a compact planar patch.

At each iteration, the planar patch with most inliers is selected and approximated by a single polygon using the α-shape algorithm [START_REF] Edelsbrunner | On the shape of a set of points in the plane[END_REF]. Figure 3.2

shows the pipeline steps. Figure 3.3 shows the inliers of the detected planes from an outdoor scan.

Figure 3.4: Robust estimation of a plane using MSAC.

Polygon MSAC

For scans acquired in a static mode (e.g., indoor scans), we do not have access to the sensor topology. So, we could not use the aforementioned method for the extraction of planar polygons. This is why, we have proposed a straightforward adaptation of M-estimator Sample Consensus (MSAC) which is a RANSAC extension that provides a potential solution to the spurious plane problem [START_REF] Torr | MLESAC: A new robust estimator with application to estimating image geometry[END_REF].

Whereas RANSAC gives the same unit score to all inliers:

C RANSAC (e i ) =        0 if ϵ 2 i < t 2 1 otherwise (3.1)
MSAC gives each point a penalty score measuring how well the point corresponds to the model:

C MSAC (ϵ i ) = min(ϵ 2 i , t 2 ) (3.2)
where ϵ i is the distance of LiDAR point P i to the current hypothetical plane and t is a distance threshold, parameter of the algorithm.

As RANSAC, MSAC produces hypothetical planes by randomly selecting three input points. The score of the sample plane is simply the sum of (3.2) over all the points P i of the input scan. When the scores have been computed for all planes, the one with the highest score is extracted, its inliers (points P i such that ϵ i < t) are removed from the point cloud, and the process is iterated until the score of the best plane is below a given threshold, as detailed in Figure 3.4. For each detected plane, we project its inliers to the plane and extract planar polygons by computing the α-shape [START_REF] Edelsbrunner | On the shape of a set of points in the plane[END_REF]) of these projected inliers. Figure 3.5 shows the inliers of the detected planes from an indoor scan and the polygons computed by the α-shape for the plane with the most inliers (corresponding to the ceiling).

Alpha shape

This technique has been proposed in [START_REF] Edelsbrunner | On the shape of a set of points in the plane[END_REF]. It can be derived Linear structures are considered among the best abstractions of LiDAR data especially in urban environment [START_REF] Hofer | Efficient 3d scene abstraction using line segments[END_REF].

State of the art

Several studies have focused on 3D line segment detection in 3D datasets. A fast and precise method to detect sharp edge features has been proposed in [START_REF] Bazazian | Fast and robust edge extraction in unorganized point clouds[END_REF]. This method works by analyzing the eigenvalues of the covariance matrix computed from each point's k-nearest neighbors. A new framework to extract line segments from large-scale point clouds has been proposed in [START_REF] Lin | Facet segmentation-based line segment extraction for large-scale point clouds[END_REF]. First, a binary classifier is trained which, for each 3D point, predicts the likelihood of lying on a contour. Then, regularly spaced points with high contour scores are found and linked into a graph of candidate contours. Finally, an optimal subset of those candidates is selected as final wireframe edges, by approximate inference in a higher-order random field defined over the graph edges and their adjacency rela-

tions. An efficient algorithm has been proposed in [START_REF] Lin | Line segment extraction for large scale unorganized point clouds[END_REF], which is able to accurately extract line segments from large-scale unorganized raw scan point clouds.

The 3D line-support regions are also extracted at the same time and fitted by the Line-Segment-Half-Planes (LSHP) structure, which provides a geometric constraint for line segments, making their detection more reliable and accurate.

Classification of 3D line detection methods

Following [START_REF] Lu | Fast 3d line segment detection from unorganized point cloud[END_REF], the problem of 3D line segment detection can be classified into three categories:

• Point based methods: use least square fitting of 3D line segments after detecting the boundary points. The main problem of this kind of method is the nonrobustness to the noise.

• Plane based methods: a 3D line segment can be generated by the intersection of two 3D planes. The drawback is that the endpoints of the intersection line are generally difficult to determine.

• Image based methods start by converting the 3D point cloud into images. They then extract 2D line segments for each image. Finally, the 2D line segments are reprojected to the point cloud to get the final 3D line segments. The difficulty of this kind of method is that the adequate resolution of image rendering is hard to determine.

Selected method

In this work, we have chosen the algorithm proposed in [START_REF] Lu | Fast 3d line segment detection from unorganized point cloud[END_REF]

Line detection from an image

According to [START_REF] Akinlar | EDLines: A real-time line segment detector with a false detection control[END_REF], traditional line segment detection algorithms can be summarized in the following steps:

• Computing an edge map, generally using Canny edge detector [START_REF] Canny | A computational approach to edge detection[END_REF] • Applying the Hough transform in order to extract all lines that contain a certain number of edge points.

• Breaking lines into segments using gap and length thresholds.

This traditional scheme is slow and leads to the generation of many false detections.

To correct and overcome this problem, new families of algorithms based on a contrario theory has been proposed. These algorithms become currently state of art methods for line detection. Using a contrario theory, the algorithm can automatically define if a line is meaningful or not with a score called Number of False Alarms. The first proposed algorithms in this category are EDLines [START_REF] Akinlar | EDLines: A real-time line segment detector with a false detection control[END_REF] and Line Segment Detector (LSD) (Von [START_REF] Gioi | LSD: a line segment detector[END_REF]. After, a multi-scale extension of LSD (multiscale line segment detector (MLSD)) was proposed in [START_REF] Salaün | Multiscale line segment detector for robust and accurate sfm[END_REF].

EDLines: It is a real-time line segment detector. It is based on the edge segment chains produced by a novel edge detector, Edge Drawing (ED). This detector includes a line validation step applying the Helmholtz principle, which lets it control the number of false detections.

LSD: It is a linear-time Line Segment Detector which works on any digital image without parameter tuning. The idea is to detect segments as connected components of pixels with similar gradient direction. However, because of noise or lack of contrast, some components may be separated by a few pixels and the detector then breaks the line into several shorter segments. This causes an over-segmentation phenomenon, noticeable in high resolution images.

MLSD:

It is a multi-scale extension to the popular LSD. Its multi-scale nature makes it much less prone to over-segmentation, more robust to low contrast and less sensitive to noise. Being based also on the a contrario theory, it retains the parameter-less advantage of LSD, at a moderate additional computation cost.

3D line segments reconstruction

3D reconstruction from images is a very active area of research. The first studies have been done through exploiting image feature-points and their invariant descriptors (e.g., SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]). As in man-made environment, most of objects are structured and can be outlined by a bunch of line segments, a new generation of 3D line reconstruction methods is born. However, different from point-based reconstruction, reconstructed 3D lines may have large displacement from the ground truth pose in spite of a very small sum of reprojection errors. We distinguish two families of methods of 3D line segments reconstruction.

Pairwise reconstruction

For this family of methods, we only need two overlapping images to perform the reconstruction. According to [START_REF] Baillard | Automatic line matching and 3d reconstruction of buildings from multiple views[END_REF] and [START_REF] Heuel | Matching, reconstructing and grouping 3d lines from multiple views using uncertain projective geometry[END_REF], the two end points of a segment generate two epipolar lines in the other images.

These two lines define a region, called the epipolar beam, which necessarily intersects or contains the corresponding segment. This reduces the complexity of the search for corresponding segments.

Multi-viewer reconstruction

If more than two images are available, it is easy to extend the above pairwise reconstruction method to deal with multiple views reconstruction. A novel approach for the reconstruction of straight 3D line segments has been proposed in [START_REF] Jain | Exploiting global connectivity constraints for reconstruction of 3d line segments from images[END_REF]. The main idea is to impose global topological constraints given by connections between neighbouring lines to generate an improved reconstruction. Additionally, this approach does not employ explicit line matching between views, thus making it more robust against image noise and partial occlusion. It removes outliers by merging independent reconstructions, that are generated from different base images. A new method for reconstructing Line segments (LS) in structured scenes has been proposed in [START_REF] Li | 3d line segment reconstruction in structured scenes via coplanar line segment clustering[END_REF]. This method represents a potential solution to

Selected method

We chose to work with the method developed in [START_REF] Hofer | Efficient 3d scene abstraction using line segments[END_REF]. This method uses an oriented image sequence as input, whose camera poses can be obtained by any conventional Structure From Motion (SfM) pipeline. It establishes a large set of potential line correspondences between images using weak epipolar constraints and separates correct from incorrect matches for each segment individually using a scoring formulation based on mutual support. It generates a final line-based 3D model by clustering 2D segments from different views, using an efficient graph-clustering formulation. We used the corresponding code available on https://github.com/ manhofer/Line3Dpp to recover the 3D lines. This code uses LSD detector to detect 2D lines in images. To improve the performance of this algorithm we have replaced LSD by our more robust variant of MLSD. Figure 3.9 shows the impact of the applied modifications in the results.

Openings detection

Introduction

Recently, the topic of building reconstruction has drawn increasing interest. It has various applications, among them urban planning and virtual tourism. 3D city model represents a very important platform in a smart city. This platform is essential to produce environmental simulation and master planning applications. Detailed building facade structures can be recovered in high density point clouds acquired by Terrestrial Laser Scanning (TLS). The automated segmentation of building facades and the detection of their elements can reduce the effort of 3D reconstruction of existing buildings and even entire cities. Openings are considered the most important feature of facades and play a crucial role in generating photo-realistic building models. The similarity and repetitive patterns of these elements can help us to discover facade structures. These components are considered as premier cues for indoor outdoor registration. Therefore, they must be detected correctly and with great precision.

State of the art

Many works have dealt with the windows detection problem. A hole based windows extraction method was proposed in [START_REF] Pu | Extracting windows from terrestrial laser scanning[END_REF], where a segmentation process was performed in order to group laser points in planar segments. A TIN (Triangular Irregular Network) of the wall segments is generated after recognizing them. The windows points, extrusion points and door points are identified by classifying the hole points. An automatic approach to facade and window detection from mobile LiDAR Data by combining two strategies was proposed in [START_REF] Wang | Window detection from mobile LiDAR data[END_REF]. which have a special interest due to their regularities in appearance. Another method for window and door detection has been proposed in [START_REF] Nguatem | Localization of windows and doors in 3d point clouds of facades[END_REF], which applies a model selection based on Bayes factor and uses a Monte Carlo simulation to generate and score competing models. This method has proven its ability to distinguish between various window and door shapes. A robust algorithm to locate opening corners for terrestrial LiDAR points has been proposed in [START_REF] Li | A sliding window method for detecting corners of openings from terrestrial LiDAR data[END_REF].

This algorithm searches and analyzes the facade with a globally wide sliding window that reduces the influence of noise in horizontal and vertical directions. The changes in facade elements were inspected by computing gradients in each sliding window, applying statistical analysis and deriving the symmetry information. The authors of [START_REF] Recky | Window detection in complex facades[END_REF] have proposed a modified gradient projection method designed for automatic processing of complex building facades. This algorithm is able to detect many different windows types without a learning step. An automatic method to detect windows and estimate their parameters in 3D LiDAR point clouds has been proposed in [START_REF] Aijazi | Automatic detection and feature estimation of windows for refining building facades in 3d urban point clouds[END_REF]. This method fuses information from both symmetrical and temporal correspondences to estimate window features. A new algorithm for openings detection in indoor point clouds has been proposed in [START_REF] Assi | Energy function algorithm for detection of openings in indoor point clouds[END_REF]. Holes corresponding to openings were detected after identifying walls in 2D binary image. Each opening was submitted to an energy function with two terms: data and coherence. Those functions help the authors to determine if an opening is due to a window/door or an object obstructing the acquisition.

Proposed method

According to [START_REF] Pu | Extracting windows from terrestrial laser scanning[END_REF], we can define the following characteristics for an opening:

• Most openings have a rectangular vertical shape of limited extent (a few meters) positioned within a vertical plane (the wall).

• Openings which are not covered with curtains cause holes on the wall segments where the Laser beams traverse the wall plane.

• Openings which are covered with curtains reflect the Laser beam which produces LiDAR points which are not on the wall plane.

According to these properties we propose a robust method to detect openings in LiDAR data compatible with any acquisition mode (mobile outdoor or static indoor)

which is summarized in figure 3.10.

LiDAR data segmentation and facade selection

RANSAC is the most used method for LiDAR data segmentation. But, it has some limitations such as under-segmentation (grouping points from nearly co-planar but disjoint surfaces in a single planar primitive). In order to overcome the limitation of RANSAC, we chose to use the two solutions proposed in Section 3.4.2 depending on the nature of the data and the acquisition platform.

Segmentation

Evidence of openings detection

As the LiDAR beams usually cross the facade through openings, we propose to start by detecting the evidences of openings as the intersection points of these beams and the detected facades using Ray Tracing:

1. For each point P i , we trace a ray R i from the LiDAR optical center O to P i .

2. We find the intersection points P j i of R i with the supporting planes P j of each fac ¸ade polygon F j .

3. If one P j i lies inside the polygon and the distance between P i and P j > d min ,

we add P j i to the list E j of evidences of openings on wall j. In Section 3.7.3, we use d min = 0.1 m in order to exclude noisy points that may still represent points of the facade.

Outline openings extraction

For each wall plane P j the evidences of openings E j are grouped in vertical rectangles:

• Extract the connected components of E j with a distance threshold.

• Estimate the minimum bounding rectangle of each component.

• For each rectangle, transform each 2D corner to a 3D point and create the four 3D line segments corresponding to its edges to get a 3D representation of our shape as shown in Figure 3.11.

Conclusion

In this chapter we have tackled the problem of detecting geometric primitives which represent the core of data processing. The building environment has the following properties • Most objects are composed of planar surfaces delimited by straight lines.

• We can present a building as a set of facades, where each one is composed of a set of openings.

Given properties above, our study was focused on the classification of primitives extraction methods followed by the following parts:

• Planar polygon extraction from LiDAR point clouds.

• 3D Line segments detection from LiDAR point clouds.

• Line segments detection and reconstruction from image sequences.

• Openings detection from LiDAR point clouds.

For each part, we started with a bibliographic study of existing works followed by the description of our proposed or selected methods. The obtained results show the efficiency of our methods (proposed or selected) to extract the different geometric primitives. These geometric primitives will be used as basic elements for more complex processes. These processes will be studied in detail in the next chapters.

Chapter 4

Indoor/Outdoor Registration

Introduction

The registration of indoor and outdoor scans with a precision reaching the level of geometric noise represents a major challenge for Indoor/Outdoor building model- As an opening in the fac ¸ade is the unique common entity that can be seen from inside and outside, it can help the registration of indoor and outdoor point clouds. So, we have integrated openings in a registration framework in order to introduce a second solution for the indoor/outdoor registration problem.

The two proposed solutions (planar polygons based algorithm and openings based algorithm) are very effective for realizing the indoor/outdoor registration. But they have some limitations. in order to overcome these limitations, we have proposed a hybrid solution, which represents a combination of both. In this work, we assume the outdoor (mobile mapping) scan to be the reference as its georeferencing system has direct access to GNSS data, so our problem is to register the indoor scans together and with the outdoor scan.

State of the art

3D Point clouds registration

Integrating multi-platform, multi-angle, and multi-temporal data is a core step in several fields of research such as computer vision, remote sensing and 3D modeling. To achieve this integration, a registration step is necessary. For Terrestrial Laser Scanning (TLS), there are two main families of registration methods depending on the number N of input point clouds: pairwise (N = 2) and multi-view (N > 2) registration. In this work we are interested in the pairwise category. The most used approach to solve this problem is feature-based registration which can be decomposed in three main steps: extracting geometric features (points, curves, planes, and/or surfaces), selecting the correspondences, and using selected correspondences to estimate the optimal transformation [START_REF] Dong | Hierarchical registration of unordered tls point clouds based ondadatata binary shape context descriptor[END_REF]. The Iterative Closest Point (ICP) algorithm is considered as the most used approach in the registration of point clouds [START_REF] Besl | A method for registration two 3-d shape bacterial availability[END_REF]. ICP starts with two overlapping point clouds and initial guess. The transformation parameters can be iteratively estimated by generating pairs of corresponding points and minimizing the error metric. The major disadvantage of this method is the convergence towards a local solution if the initial data are not spatially close or if the initial transformation is poorly estimated.

Several variants of the ICP algorithm have been proposed to improve its robustness such as using a point-to-plane error metric. At each iteration of the algorithm, the relative pose that gives the minimal point-to-plane error is usually estimated using a standard nonlinear least-squares method. [START_REF] Low | Linear least-squares optimization for point-to-plane ICP surface registration[END_REF] proposes an approximation of the nonlinear optimization problem with a linear least-squares one that can be solved more efficiently. An extension of the ICP framework to nonrigid registration that uses the same convergence properties of the original algorithm was proposed in [START_REF] Amberg | Optimal step nonrigid ICP algorithms for surface registration[END_REF]. Another approach [START_REF] Sharp | ICP registration using invariant features[END_REF] proposes to use Euclidean invariant features in a generalization of ICP registration of range images.

To find the correspondence of 3D range camera, the authors proposed to use either spherical harmonics or the second order momentum. Another method for detecting uncertainty in pose has been introduced in [START_REF] Gelfand | Geometrically stable sampling for the ICP algorithm[END_REF], where the transformations that can cause unstable sliding in the ICP algorithm have been estimated using a sampling strategy and the points that best contain this sliding have been picked. In order to minimize the search space for correspondence between two point clouds and to increase the accuracy of the registration, the authors of [START_REF] Rabbani | An integrated approach for modelling and global registration of point clouds[END_REF] have used standard primitives, namely planar and cylindrical surfaces, to introduce two solutions suitable in industrial environments: an indirect solution which separates object fitting and registration steps, a direct solution which simultaneously determines the shape and pose parameters of the objects as well as the registration parameters.

A Normal Distribution Transform (NDT) algorithm and its variants are proposed to deal with the problem of 3D points cloud registration. A certain cost is obtained that evaluates the alignment quality between the two point clouds, according to the desired transformation between the two point clouds and using a set of Gaussian distributions with different probability density functions (PDFs) to represent the referenced point clouds [START_REF] Dong | Registration of large-scale terrestrial laser scanner point clouds: A review and benchmark[END_REF] [START_REF] Magnusson | Scan registration for autonomous mining vehicles using 3d-ndt[END_REF]. A registration workflow which simulates the well-known RANSAC was proposed in [START_REF] Al-Durgham | Ransac approach for automated registration of terrestrial laser scans using linear features[END_REF] to register overlapping terrestrial laser scans by using 3D linear features to establish an automatic matching strategy.

The authors of [START_REF] Theiler | Automatic registration of terrestrial laser scanner point clouds using natural planar surfaces[END_REF] have used virtual tie points generated by the intersection of three non-parallel planes in two different scans. So, the objective of the registration algorithm is to search for the assignment which preserves the geometric configuration of the largest possible subset of all tie points.

The authors of [START_REF] Forstner | Efficient and accurate registration of point clouds with plane to plane correspondences[END_REF] have proposed methods to efficiently register point clouds by introducing new optimal and sub optimal direct solutions based on plane-to-plane correspondences for determining the relative motion.

A fast 3D point cloud registration in cluttered urban environments has been proposed in [START_REF] Xiao | 3d point cloud registration based on planar surfaces[END_REF], where the registration is defined as a correlation problem.

Other works based on deep learning techniques have been proposed in the literature, such as the method proposed in [START_REF] Elbaz | 3d point cloud registration for localization using a deep neural network auto-encoder[END_REF], where the authors have introduced an innovative point cloud registration algorithm based on two original approaches. The first consists in using super-points (selected by a random sphere cover set) as the basic units for matching, instead of key-points. Whereas the second consists in encoding local 3D geometric structures using a deep neural network auto-encoder.

Indoor/Outdoor registration

The registration of indoor and outdoor scans is a challenging problem for Indoor/Outdoor building modeling. Two registration and georeferencing methods for exterior and interior point clouds, in the case of heritage building recording, are tested in [START_REF] Murtiyoso | Comparison and assessment of 3d registration and georeferencing approaches of point clouds in the case of exterior and interior heritage building recording[END_REF]. The first corresponds to the independent georeferencing method, in which the indoor and outdoor scans were georeferenced separately. The second is a keypoint based registration method. These tie points concern mainly openings such as windows, which are evenly distributed throughout the building. Windows are manually segmented in the indoor and outdoor point clouds.

Then keypoints are detected for both sides of the windows, resulting in two sets of keypoints. The work presented in [START_REF] Assi | Assessment of a keypoints detector for the registration of indoor and outdoor heritage point clouds[END_REF] has proven that the use of key points rather than the entirety of a point cloud for an ICP algorithm can improve its performance to register indoor and outdoor scans. An approach for automatically aligning the non-overlapping indoor and outdoor parts of a 3D building model computed from image based 3D reconstructions has been proposed in [START_REF] Koch | Automatic alignment of indoor and outdoor building models using 3d line segments[END_REF]. This alignment has been performed by identifying corresponding 3D structures that are part of the interior and exterior model (e.g.

Planar polygons based registration

The work carried out has confirmed that the environment and the type of data drive the choice of the registration algorithm. For example, in man-made environments, where most objects are bounded by planar surfaces, the ideal is to choose a method of registration based on plane correspondences [START_REF] Theiler | Automatic registration of terrestrial laser scanner point clouds using natural planar surfaces[END_REF]

Detecting points inside buildings in the outside scan

As the LiDAR beam usually passes through windows as shown in Figure 4.6, we propose to start by detecting fac ¸ades from the outdoor scan, then detecting indoor points as points behind a fac ¸ade by ray tracing. For each given threshold, RANSAC estimates F ≤ R planes, so at most K = R -F planes are undetected as shown in Figure 4.7. Inspired by [START_REF] Bughin | Planar patch detection for disparity maps[END_REF], we can define the error as being the probability that a significant existing planar region is not detected. Another possibility is the probability that the number of detected planes is less than the number of undetected planes:

N F A(σ) = N iter × p(K ≥ F | σ) (4.2)
The probability p(K ≥ F | σ), can then be upper bounded by the tail of the binomial law of parameter P defined by:

P (σ) = 2σ × diag 2 v (4.3)
where diag is the diagonal diameter of the bounding box of the 3D point cloud and V its volume. Then:

p(K ≥ F | σ) ≤ K j=F K j P (σ) j (1 -P (σ)) K-j (4.4)

Ray Tracing

Once the fac ¸ades are detected, we detect indoor points by ray tracing:

1. For each point P i , trace a ray R i from the LiDAR optical center to P i .

2. Find the intersection points P j i of R i with the supporting planes P j of each fac ¸ade polygon F j .

3. Test if P j i is inside the polygon (using the CGAL library (https://www.cgal. org)).

4. In order to remove ouliers close to the fac ¸ade plane, we additionaly require the point to be at least 1 m away from the intersected polygon: (dist(P i , P j i ) > 1m).

5. If one P j i satisfies both criteria (inside the polygon and sufficiently far from the fac ¸ade), tag it as an indoor point.

A result of this indoor point detection method is presented in

Matching planar polygons

Selection of correspondences is a crucial step for the registration. If we have at least three correct correspondences of polygons with independent normals, it is possible to find the relative rotation/translation between the indoor and outdoor scans. We will start by presenting a simple threshold based matching in order to introduce the three main criteria used to match polygons, then propose a more robust matching based on clustering algorithm.

Threshold based matching

Let us call:

• P 1 and P 2 detected planar 3D polygons from the two LiDAR scans: indoor scan (Scan 1 ) and outdoor scan(Scan 2 ).

• n 1 and n 2 the corresponding plane normals.

• g 1 and g 2 the corresponding centroids.

• P 1,2 their bisector plane.

• P ′ 1 and P ′ 2 the projections of P 1 and P 2 on the plane P 1,2 .

We propose first a simple filter based on three measures:

1. Angle: Angle(P 1 , P 2 ) = ⟨n 1 , n 2 ⟩ 2. Distance: as there is not a standard definition for this distance, we have chosen to define it as the sum of distances from the centroid of each polygon to the bisector plane:

Distance(P 1 , P 2 ) = dist(g 1 , P 1,2 ) + dist(g 2 , P 1,2 ) (4.5)
3. Overlap:

Overlap(P 1 , P 2 ) = |P ′ 1 ∩ P ′ 2 | min (|P ′ 1 |, |P ′ 2 |) (4.6)
where |.| denotes the area. Note that we did not use the union on the denominator because a planar part of the inside scene is seen through an opening from the outside so only a very limited portion of it can be detected which would result in very low overlap with a more standard definition.

In practice, finding appropriate thresholds for these three criteria is tedious and leads often to multiple or no matchings. This is why we propose a more robust approach.

Cluster based matching

We propose a more robust matching for polygons based on three main principles:

clustering the polygons (by direction then by offset), enumerating match hypotheses between the clusters and evaluating which hypothesis is the best.

Algorithm 1 Greedy direction clustering

Input: P i a set of planes with normals n i and number of inliers n i . ϵ a tolerance angle (typically π/4 rad).

Clusters initialization:

• C 1 = {P 1 } where P 1 is the plane with most inliers • C 2 = {P 2 } where P 2 is the plane with most inliers among planes for which n i .n 1 < cos(ϵ) • C 3 = {P 3 } where P 3 is the plane with most inliers among planes for which n i .n 1 < cos(ϵ) and n i .n 2 < cos(ϵ) Mark P 1 , P 2 and P 3 as processed and all other P i as unprocessed Each cluster C k has a normal ⃗ c k computed as the weighted centroid of all the normals of the planes in C k Let P cur be the unprocessed plane with the most inliers. Mark P cur as processed. compute

k min = arg min k 1 -|⃗ n i .⃗ c k | compute d i = 1 -|⃗ n i .⃗ c k min | If d i < ϵ, add P i to the cluster C k min .
Call C h and tag as horizontal the cluster for which 1 -|⃗ z.⃗ c k | is minimum. Call C v 1 and C v 2 and tag as vertical the two remaining clusters.

Direction clustering:

For each input scan, we greedily cluster planes P i according to their normals ⃗ n i by decreasing number of inliers n i as detailed in Algorithm 1 that produces 3 clusters C h , C v 1 and C v 2 for each scan.

Direction cluster matching

We associate C h for the two scans. We associate each remaining cluster (C v 1 and C v 1 ) to the vertical cluster from the other scan with the smallest angle.

Alignment: we compute the rotation that aligns the three clusters: the vertical C v 1 and C v 2 and the horizontal clusters using the method described in Section 4.4.3

Hypotheses enumeration: for each associated direction cluster, we enumerate possible plane matches:

• For each of the three associated main cluster pair

A m = (C 1 k 1 , C 2 k 2 ) we will call I m the set of pairs of planes (P 1 i 1 ∈ C 1 k 1 , P 2 i 2 ∈ C 2 k 2 )
• for each (J 1 , J 2 , J 3 ) in I 1 × I 2 × I 3 , compute the translation that aligns the planes in (J 1 , J 2 , J 3 ) using the method described in Section 4.4.4.

• note that I 3 is extracted from the horizontal clusters which cover the displacement along z, I 2 is extracted from the vertical clusters which have the smallest angle with the y axis and I 1 is extracted from the vertical clusters which have the smallest angle with the X axis.

• keep the translation calculated by the best hypothesis.

Selection of the best hypothesis:

To asses which hypothesis is the best, we define an energy that ( 1) is robust to outliers as many planes detected in one scan have no counterpart in the other (2) favors important polygon overlaps and (3) favors small distances over these overlapping parts. For robustness, we need a distance threshold d thr above which a polygon is just considered as an outlier. Then for two polygons P 1 , P 2 with centroids O 1 , O 2 in the same cluster, we define a robust error:

E d thr (P 1 , Scan 2 ) = 1 d 2 thr P ∈P 1 min(d 2 thr , dist(P, Scan 2 ) 2 )dP (4.7)
We see that the energy is 0 when P 1 is completely overlapped by polygons of Scan 2 with a distance of 0 (the perfect case) and increases as distance augments and overlap decreases up to |P 1 | when the overlap is empty and/or the distance is over d thr . In practice we make the approximation:

E d thr (P 1 , Scan 2 ) ≈ P 2 ∈Scan 2 |P 2 1 ∩ P 1 2 | max(0, d 2 thr -Distance(P 1 , P 2 ) 2 ) d 2 thr (4.8)
Note that we do not use our modified relative Overlap function anymore as a large overlap surface should be favored to a small one as it correponds to more input points. Finally, we can write our global robust energy as:

E R (Scan 1 , Scan2) = P 1 ∈Scan 1 E d thr (P 1 , Scan 2 ) (4.9)
We consider that the best hypothesis is the one that maximizes this energy.

Rotation estimation

Assuming that we have an association between three directional clusters of two scans:

• C h 1 and C h 2 the horizontal clusters of scans 1 and 2, ⃗ c h 1 and ⃗ c h 2 their normals.

• C v 1 1 the first vertical cluster of scan 1 and C v 1 2 the associated vertical cluster of scan 2, ⃗ c v 1 1 and ⃗ c v 1 2 their normals.

• C v 2 1 the second vertical cluster of scan 1 and C v 2 2 the associated vertical cluster of scan 2, ⃗ c v 2 1 and ⃗ c v 2 2 their normals.

We want to find the rotation that best aligns these three pairs of directions. Let ⃗ u 1 = ⃗ c h 1 and ⃗ u 2 = ⃗ c h 2 . We are looking for the two vectors:

{⃗ v 1 , ⃗ v 2 } = arg min ⃗ v 1 ∈{ ⃗ c v 1 1 , ⃗ c v 2 1 },⃗ v 2 ∈{ ⃗ c v 1 2 , ⃗ c v 2 2 } ⟨v 1 , v 2 ⟩ (4.10)
We want to create a first orthogonal basis M 1 from u 1 and v 1 , and a second orthogonal basis M 2 from u 2 and v 2 , so we have:

• ⃗ q 1 =the projection of ⃗ v 1 on the orthogonal plane to ⃗ u 1

• ⃗ q 2 =the projection of ⃗ v 2 on the orthogonal plane to ⃗ u 2

P roj orth P lane ⃗ u 1 (⃗ v 1 ) = ⃗ v 1 - (⃗ v 1 .⃗ u 1 ) |⃗ u 1 | 2 ⃗ u 1 (4.11) P roj orth P lane ⃗ u 2 (⃗ v 2 ) = ⃗ v 2 - (⃗ v 2 .⃗ u 2 ) |⃗ u 2 | 2 ⃗ u 2 (4.12) ⃗ l 1 = ⃗ u 1 × ⃗ q 1 (4.13) ⃗ l 2 = ⃗ u 2 × ⃗ q 2 (4.14)
we have:

M 1 =       u 1 (x) q 1 (x) l 1 (x) u 1 (y) q 1 (y) l 1 (y) u 1 (z) q 1 (z) l 1 (z)       (4.15)
and

M 2 =       u 2 (x) q 2 (x) l 2 (x) u 2 (y) q 2 (y) l 2 (y) u 2 (z) q 2 (z) l 2 (z)       (4.16)
We can calculate the rotation as the basis transfer matrix between M 1 and M 2 :

M 2 = RM 1 (4.17) R = M 2 M -1 1 (4.18)
We consider the matrix R as the optimal rotation that aligns the three directional Clusters of two scans.

Translation estimation

We have defined a hypothesis as three pairs of planes. Each pair is extracted from

two matched clusters (C i ∈ scanA, C j ∈ scanB).
For each hypothesis we have:

pair l = {P lane i,l (N i,l , d i,l ), P lane j,l (N j,l , d j,l )}.
where l represents the pair index in the hypothesis and i,j represent the indices of the two matched clusters. For each pair we have the following properties:

N l = RN i,l (4.19
)

N l T = d i,l -d j,l (4.20)
therefore according to the two equations above and the definition of a hypothesis, we can deduce the following linear system:

      N 1 (x) N 1 (y) N 1 (z) N 2 (x) N 2 (y) N 2 (z) N 3 (x) N 3 (y) N 3 (z)             T x T y T z       =       d i,1 -d j,1 d i,2 -d j,2 d i,3 -d j,3       (4.21)
by solving this linear system, we can find the translation that aligns the three pairs of planes. The first evaluation of our algorithm was carried out on two interior scans from which a perfect registration was done manually. Each input scan was subsampled with factor of 10, yielding around 260,000 points. Starting from this perfect position, we have altered one of the two scans with an initial translation error ranging from a few centimeters to a few meters and an initial rotation error ranging from a few degrees to a few tens of degrees as shown in Table 4.1. The chosen setting allowed us to estimate 22 planes from the first scan, and 23 planes from the second scan. The obtained results show that our algorithm is able to register the two scans within a reasonable calculation time regardless of the initial error. Afterwards, we performed the registration of indoor and outdoor scans. As we do not know the ground truth we only considered the visual results. We consider that the obtained results demonstrate our algorithm's ability to efficiently register indoor and outdoor data as shown in Figure 4.9.

Evaluation and discussion

Achieving precise results requires fine-tuning of the algorithm's parameters. The iteration number of MSAC must be calculated according to the number of points of the scan to ensure the robustness of the algorithm. The number of estimated planes depends on the inlier threshold and the minimum size of a planar region, whereas the number of extracted polygons depends on the value of α. If we estimate more planes, we can generate more hypotheses and use more polygons for the evaluation of each hypothesis, and therefore we can find a more precise result.

Openings based registration

The topic of building reconstruction draws increasing interest and has various applications such as urban planning, virtual tourism. BIM is also becoming a de facto standard to handle the whole life cycle of buildings and 3D city models are central to manage smart cities and in particular to realize environmental simulations. Combining indoor and outdoor modeling of buildings remains a challenging task. The first issue of such modeling is to precisely register the indoor and outdoor data sources.

The lack of overlap between indoor and outdoor data is the most encountered obstacle, more so when both data sets are acquired separately and using different types of sensors. Openings are the most obvious common entity to link the inside and outside data. As such, it can help the registration of indoor and outdoor point clouds. So it must be automatically, accurately and efficiently extracted. Therefore, in order to improve the indoor/outdoor registration, we integrate the openings in a registration framework after having extracted them using the method proposed in Section 3.7.

Registration is usually performed by matching features. In the present work, the features are the two sets of vertical openings rectangles detected from indoor and outdoor scans. Selection of correspondences is a crucial step for a successful registration because a bad choice can lead to a bad estimate of the optimal transformation. In our case, the openings are not characteristic enough to match them robustly independently. As an opening is defined by a rectangular shape composed of four segments, two horizontal and two vertical, we can write our registration problem as a minimization of a global robust energy between two segment sets. To achieve this minimization with high accuracy and robustness, we must select the right matches and choose the good optimization algorithm.

Global robust energy between segment sets

We propose to define a robust energy between two sets of 3D segments in a way that minimizing this energy will favor important overlaps between segments and small distances over these overlaps while being robust to outliers, which is to be expected as it is possible that openings extracted form one data set will have no counterpart in the other. This will implicitly also minimize angles between the segments as an important angle with an important overlap implies a large distance. Given two sets of 3D segments

[A 1 i B 1 i ] and [A 2 j B 2 j ] with center points G k i = (A 1 i + B 1 i )/2 and direction vectors d k i = ---→ A k i B k i /|| ---→ A k i B k i ||,
we start by defining a relative overlap between two segments over the projection p on the bisector of the two segments:

overlap([A 1 i B 1 i ], [A 2 j B 2 j ]) = |p([A 1 i B 1 i ]) ∩ p([A 2 j B 2 j ])| min(|p([A 1 i B 1 i ])|, |p([A 2 j B 2 j ])|)
We define this bisector as the line passing through the closest point P i,j to (A

1 i B 1 i )
and (A 2 j B 2 j ) and oriented by the bisector v i,j of d 1 i and d 2 j . Calling:

[v] × =       0 -v z v y v z 0 -v x -v y v x 0       such that [v] × u = v × u (4.22)
we have:

P i,j = ([d 1 i ] t × [d 1 i ] × + [d 2 j ] t × [d 2 j ] × ) -1 ([d 1 i ] t × [d 1 i ] × A 1 i + [d 2 j ] t × [d 2 j ] × A 2 j ) (4.23)
The projection on the bisector is:

p(P ) = P i,j + ( ---→ P i,j P .v i,j )v i,j (4.24)
and the projected points have a linear coordinate:

c(P ) = ---→ P i,j P .v i,j (4.25)
The length of the projected segment is then defined as

|p([A 1 i B 1 i ])| = max(c(A 1 i ), c(B 1 i )) -min(c(A 1 i ), c(B 1 i )) (4.26)
and the overlap length of the projected segments as:

|p([A 1 i B 1 i ]) ∩ p([A 2 j B 2 j ])| = min(max(c(A 1 i ), c(B 1 i )), max(c(A 2 j ), c(B 2 j ))) -max(min(c(A 1 i ), c(B 1 i )), min(c(A 2 j ), c(B 2 j ))) (4.27)
We then define an energy between a segment L 1 ∈ L 1 and a segment set L 2 :

E d thr (L 1 , L 2 ) =|L 2 1 |d 2 thr - (4.28) L 2 ∈L 2 |L 2 1 ∩ L 1 2 |max(0, d 2 thr -dist(L 1 , L 2 ) 2 ) (4.29)
where d thr is a distance threshold above which a line is just considered as an outlier and dist is a metric distance between segments, for instance:

dist([A 1 i B 1 i ], [A 2 j B 2 j ]) = 1 4 dist(A 1 i , [A 2 j B 2 j ]) + dist(B 1 i , [A 2 j B 2 j ]) + dist(A 2 j , [A 1 i B 1 i ]) + dist(B 2 j , [A 1 i B 1 i ]) (4.30)
For each line pair, this denergy is 0 if L 1 is completely overlapped by segments of L 2 with 0 distance over these overlaps. Finally we can write our final symmetrized robust energy between 3D segments sets as:

E S (L 1 , L 2 ) = L 1 ∈L 1 E d thr (L 1 , L 2 ) + L 2 ∈L 2 E d thr (L 2 , L 1 ) (4.31)

RANSAC optimization

RANSAC has proven its robustness and efficiency as an optimization algorithm in several applications. In this part, we describe a new RANSAC based on two matching strategies. The first consists in matching the vertical planes representing the walls of the two scans whereas the second consists in matching the selected segments in the matched walls. The advantage of the integration of these two matching strategies is to simplify the selection of the two pairs of segments and optimize the calculation time of the algorithm by reducing the number of iterations. At each RANSAC iteration:

• We randomly select a wall plane from each scan.

• We randomly select one vertical segment and one horizontal segment in each selected wall plane.

• We associate the vertical segments and the horizontal segments between the two selected sets.

• We compute the transform (rotation and translation) that best aligns the matched 3D segments using the method of Section 4.5.3.

• For this transform, we estimate the global robust energy between all segments of the two sets using (4.31).

The final registration is given by the transformation that minimizes the global robust energy.

Transform estimation

Rotation estimation

Once two pairs of segments {v i , v j } (vertical) and {h i , h j } (horizontal) are associated, we estimate the rotation that best aligns the corresponding two 3D lines. Let us call

d i v , d j v , d i
h and d j h the unit norm vectors of v i , v j , h i and h j . We start by creating orthonormal bases O i = (x i , y i , z i ), where:

x i = d i v y i = d i h -(d i h • x i )x i ||d i h -(d i h • x i )x i || z i = x i × y i . (4.32)
We then compute the rotation R that aligns the associated segments as the base change matrix between O i and O j :

R = O j O i -1 . (4.33)

Translation estimation

To estimate the translation, we start by defining the point to line distance:

dist(p, L = a + dt) = ||d ∧ (a -p)|| ||d|| = [d] × (a -p) (4.34)
assuming that d is normalized, and again calling [d] × the matrix of the cross product with d. We look for the translation t that minimizes:

M SACepsilon = i ||[d i ] × (a i -(p i + t))|| 2 (4.35)
The minimum is reached where the gradient vanishes:

∇ t ϵ(t) = -2 i [d i ] t × [d i ] × (a i -(p i + t)) = 0. (4.36)
Calling:

w = i [d i ] t × [d i ] × (p i -a i ) M = - i [d i ] t × [d i ] × (4.37)
We get as translation vector:

t = M -1 w. (4.38) (a) (b)

Evaluation and discussion

In this section, our contribution consists in proposing an openings based registration algorithm to address the challenging problem of indoor/outdoor registration with a unified formalism. Our proposed algorithm works without any constraints on the initial position of the two scans, unlike iterative methods that require a good ap-proximation of the initial transformation to be able to converge towards the correct solution. The key step of our algorithm is the detection of openings which was carried out using an automatic method applicable in indoor and outdoor data. The obtained results have proven the performance of our algorithm to register indoor/outdoor data whatever the initial position and orientation as shown in Figure 4.10. We defined a global robust energy between two segments sets and we proposed a robust approach to minimize this energy using the RANSAC pardigm.

Hybrid solution

In Sections 4. In order to remove these uncertainties, we have tried to combine the two solutions.

This combination allows us to generate a hybrid solution able to perform the indoor/outdoor registration with great precision and exceeding the limits of the two previous solutions.

Valid hybrid associations

In this approach, each scan (indoor scan or outdoor scan) is simplified by two types of geometric primitives: openings and planar polygons. In order to accelerate and simplify the matching of these components between the two scans, we propose to start by efficiently associating the supporting planes of polygons and the openings of the same scan. This is done using the Algorithm 2.

Global hybrid energy

We propose to define a hybrid energy between two hybrid association sets. Each association sets corresponds to a scan. Each association is formed by a supporting plane of a polygon and a supporting plane of an opening extracted from the same scan. As an opening is defined by a rectangular shape composed of four segments, two horizontal and two vertical ones, the minimization of the hybrid energy must favor important overlaps both between polygons and between segments and small distances over these overlaps. For robustness, we need a distance threshold d thr above which a polygon or a 3D segment is considered as an outlier. This hybrid energy is inspired by the equations (4.9) and (4.31).

E d thr (A i (P i , L i ) ∈ Scan 1 , Scan 2 ) ≈ d 2 thr -A j ∈Scan 2 P j ∈A j |P ′ i ∩P ′ j | min(|P ′ i |,|P ′ j |) max(0, d 2 thr -Distance(P i , P j ) 2 )+ |L i |d 2 thr -A j ∈Scan 2 L j ∈A j |L ′ i ∩ L ′ j |max(0, d 2 thr -Distance(L i , L j ) 2 ) (4.39)
where:

• |.| represents the area for the polygons and the length for the 3D segments.

We see that the energy is null when P j completely overlaps by P i (relative overlap is 1) with a distance Distance(P i , P j ) = 0 and L j completely overlaps L i with a distance Distance(L i , L j ) = 0.

Finally we can write our final symmetric hybrid energy as:

E h (Scan 1 , Scan 2 ) = A i (P i ,L i )∈Scan 1 E d thr (A i (P i , L i ), Scan 2 ) + A j (P j ,L j )∈Scan 2 E d thr (A j (P j , L j ), Scan 1 ) (4.40)

RANSAC based optimization of the hybrid energy

To carry out the registration of the two association sets, we now have to find the rotation and the translation that minimize the hybrid energy (4.40). In this section, we describe a new adaptation of RANSAC based on two matching steps. The first consists in matching the valid associations between the two scans whereas the second consists in matching the selected segments and polygons in the matched associations.

The advantage of the integration of these two matching steps is to simplify the selection of the two pairs of segments and the pair of polygons, necessary to calculate the transformation at each iteration, and optimize the calculation time of the algorithm by reducing the number of iterations.

At each RANSAC iteration:

• Randomly select an association from each scan ( A i (P o i , P p i ) ∈ Scan 1 and A j (P o j , P p j ) ∈ Scan 2 ).

• Make the two planes (P o i , P o j ) coincident by applying a rotation R around their line of intersection, this rotation can be computed using the method of Section 4.6.4..

• Randomly select a corner (the intersection point of a horizontal segment and a vertical segment) from the associated openings of P o i and P o j .

• Compute the translation in the plane between the two selected corners (we have only two degrees of freedom).

• Find the missing degree of freedom of the translation by computing the distance between the associated planar polygons of P p i and P p j .

Finally, keep the sampled transformation (rotation + translation) that has the minimum hybrid energy (4.40).

Rotation around intersection line

We compute the rotation around the intersection lines of two planes P 1 ,P 2 as the following:

• N 1 is the normal vector of P 1 .

• N 2 is the normal vector of P 2 .

• U is the unit vector of the intersection line computed according the the equation (4.41).

U = N 1 ∧ N 2 (4.41)
As N 1 and N 2 are normalized, the rotation angle will be:

α = arccos(N 1 • N 2 ).

Conclusion

In this chapter, we have addressed the indoor /outdoor registration problem which represents a major challenge for Indoor/Outdoor building modeling. We have ex- methods use Mutual Information (MI) as a statistical metric to register image and LiDAR data [START_REF] Lu | Fast 3d line segment detection from unorganized point cloud[END_REF]. A mutual information (MI) approach was proposed in [START_REF] Mastin | Automatic registration of LiDAR and optical images of urban scenes[END_REF]. This method performs the registration by seeking the camera matrix that maximizes the MI between the distribution of image features and projected LiDAR features. The authors of [START_REF] Wang | Automatic registration method for mobile LiDAR data[END_REF] search for the optimal camera pose through maximizing the MI between camera image and LiDAR attribues (LiDAR intensity image, LiDAR elevation image) [START_REF] Wang | Automatic registration method for mobile LiDAR data[END_REF].

The major limitation of these methods lies in the decreasing registration accuracy when only LiDAR elevation image is used. An hybrid online calibration method for a laser scanner mounted on a mobile platform also equipped with a imaging system has been proposed in [START_REF] Miled | Hybrid online mobile laser scanner calibration through image alignment by mutual information[END_REF]. The inaccurate transformation between camera and laser scanner has been optimized by maximizing the mutual information (MI) between these two kinds of data resulting in an accurate online calibration.

Frequency based registration

The most popular frequency based method is phase correlation. Due to differences in data characteristics, these frequency based methods cannot be applied directly to register images and LiDAR data. [START_REF] Zhu | Fast and robust registration of aerial images and LiDAR data based on structrual features and 3d phase correlation[END_REF] has proposed a new registration method to deal with the problem of aerial image and LiDAR registration. This method is based on structural features and 3D phase correlation, in order to address significant geometric distortions and nonlinear intensity differences between the aerial and LiDAR intensity images. The 3D phase correlation is used to detect control points (CPs) between aerial images and LiDAR data in the frequency domain, that will be used to correct the exterior orientation elements. The main limitation of this method lies in the use of the intensity images of LiDAR only, which contains less information about the topography as compared to LiDAR elevation images (Kumar [START_REF] Kumar Mishra | A review of optical imagery and airborne LiDAR data registration methods[END_REF].

According to this summary comparison, we advocate that the best methods to apply for image/LiDAR registration in urban environments are mutual information and straight line based methods. We have chosen to develop a straight line based algorithm for the following reasons:

• In building environment, most objects are composed of planar surfaces delimited by straight lines.

• On our scenes of interest, we have both overlapping images and LiDAR scans.

• There are efficient algorithms in the literature to automatically and precisely extract these features from both images and LiDAR data.

Overview and contributions

In a man made environment, we advocate that the best feature to use for image/L-iDAR data registration is the straight line segment. Straight line segments can be reliably, accurately and automatically extracted from both LiDAR and image data under sufficient image overlap. They aggregate more information than points therefore less sensitive to more dense noise and more frequent than more complex primitives (rectangles).

In this chapter, we propose a new method for image/LiDAR registration, that consists in:

• Extracting 3D segments from the two datasets.

• Defining a global robust energy between the two segments sets that can be minimized by RANSAC or simulated annealing.

RANSAC works by matching pairs of segments. The Figure 5.1 shows details of our pipeline .

3D Segment Extraction

3D Line Segment Extraction from LiDAR Data

3D line segment detection from LiDAR data is a crucial step in our registration procedure. We have proceeded to the detection of these features according to the method detailed in Section 3.5.

3D segments based registration

Registration is usually performed either based on a global energy between the two data sets to register, or by matching features. In our case, the line segments are not characteristic enough to match them robustly independently. This is why we preferred to define a global robust energy between two segment sets and propose a robust approach to minimize this energy. We propose to work with the energy defined in Section 4.5.1. It is a robust energy between two 3D segment sets in a way that minimizing it will favor important overlaps between segments and small distances over these overlaps while being robust to outliers, as many lines extracted form one data set will have no counterpart in the other. This will implicitly also minimize angles between the segments as an important angle with an important overlap implies a large distance.

3D segments directional clustering

In order to simplify the minimization of this energy between segments sets, we start by clustering the segments (by direction). Then, we define the rotation between the segments sets by matching clusters and not individual segments in order to be more robust and precise. This is done using our proposed greedy algorithm described in Algorithm 3.

Algorithm 3 Greedy direction clustering

1: Input: Set of segments L, each segment L i = [A i B i ] has a director vector v i = --→ A i B i , a length len i = ||v i || and a unit direction d i = v i /len i . 2: Initialize an empty set of 3D segment clusters C. We will call direction of a cluster C the weighted mean of the directions of the 3D segments:

d(C) = L i ∈C sign(v i • v 1 )v i || L i ∈C v i || 3:
For each segment L i in descending order of length (from longest to shortest):

• If C = ∅ or max C∈C d i • d(C) < cos ϵ, create a new cluster C = {L i } and add it to C.
Note that this is equivalent to finding if L i has an angle smaller than ϵ with an existing cluster direction but this formulation with dot products is faster to compute. • Else add L i to the cluster arg max d i • d(C).

Valid cluster associations

To define a rotation between the two segment sets, it is sufficient to associate two pairs of segments from the two sets. However this is not very precise, so we propose to associate the segment clusters defined above. We propose two possible sets of tween two segments sets. we have chosen to work with the energy (4.31) which has been defined in the openings based algorithm. We want this minimization to be insensitive to initialization, so we propose two randomized optimizers: Simulated Annealing (SA) and RANSAC.

Simulated annealing optimization

We use simulated annealing to explore the translation and scale parameters. In simulated annealing, a new solution is iteratively computed in the vicinity of the current solution and this new solution is accepted with a certain probability depending on its energy (the global robust energy in our case). However for rotations we want to speed up the process by only exploring valid cluster associations to limit the complexity of the search, see Algorithm 5. Compute

∆ E = E(M N ) -E 0 if (∆ E < 0) or (random < e -∆ E Tmax ) then M 0 = M N E 0 = E(M N ) end if if (E 0 < E(G)) then G = M 0 E(G) = E 0 end if end for T max = α * T max end while
The solution witch has the minimum energy is considered as the optimal solution

RANSAC optimization

The adaptation of RANSAC to valid associations is quite straightforward. At each RANSAC iteration, we randomly select a valid cluster association, then randomly select one 3D segment in each of the associated clusters. If the distance between the supporting lines of two segments is smaller than a threshold d min chosen to be the expected noise level in the corresponding scan, we reject it because matching coplanar segments will lead to degenerate scale estimation. We then compute the rotation based on cluster association using the method of Section 5.5.3 and estimate the scale/translation that best aligns the associated 3D segments.

Finally, we keep the sampled transformation that has the minimum robust energy.

Optionally, we can refine this solution by matching all pairs of segments that are close enough in the two scans (with a distance smaller than d thr ) and recompute the scale/translation based on all these associations to have a more precise estimation.

Scale/translation estimation

The goal of this step is to estimate the scale/translation that minimizes the distance between matched segments pairs. To do this, we define the point to line distance as

dist(p, L = a + dt) = ||(a -p) ∧ d|| ||d|| . (5.5) Assuming that d is normalized, this writes dist(p, L = a + dt) = ∥[d] × (a -(sp + t))∥ . (5.6)
Our goal is to find the optimal translation t and scale s that minimize

ϵ(s, t) = i ||[d] i ×(a i -(sp i + t))|| 2 .
(5.7)

The minimum is reached where the gradient vanishes:

∇ t ϵ(s, t) = 2 i [d i ] 2 × (a i -(sp i + t)) = 0, (5.8) ∇ s ϵ(s, t) = 2 i p t i [d i ] 2 × (a i -(sp i + t)) = 0.
(5.9) Calling:

w 1 = i [d i ] 2 × a i M 1 = i [d i ] 2 × M 2 = i [d i ] 2 × p i w 2 = i p t i [d i ] 2 × a i = - i (d i × p i ) • (d i × a i ) M 3 = i p t i [d i ] 2 × p i = - i ∥d i × p i ∥ 2 we get    t s    =    M 1 M 2 M t 2 M 3    -1    w 1 w 2    .
(5.10)

Iterative Closest Line (ICL)

We have integrated the clustering and the global robust energy in the Iterative Closest Line (ICL) paradigm in order to compare it with our algorithm in terms of performance and robustness.

Matching step

We applied a filter based on three criteria to select the correspondences between two 3D segment sets: Angle, distance and overlap.

• Angle: in order to simplify the validation of this criterion, we preferred to select segments belonging to the associated clusters (to avoid choosing a threshold for this criterion);

• Distance: we have defined this distance according to (4.30).

• Overlap: we have defined the overlap between two 3D segments according By using SVD (singular value decomposition) of A, we can find R as: R = UV T (5.14)

Limitations

In practice, finding appropriate thresholds for the distance and overlap is tedious and often leads to multiple or no matchings. In addition, ICL requires a good approximation of the initial transformation to be able to converge towards the right solution.

Our RANSAC based approach is the best solution for the registration of two 3D segment sets. The selection of the best transformation as the one which optimizes the global robust energy between all the extracted 3D segments makes our solution the most consistent with all the information of the datasets to be registered.

Evaluation and discussion

The evaluation of our approach was carried out in two steps using two data types:

Evaluation on realistic data

The first evaluation of our approach was carried out on realistic data. We tried to simulate a realistic setting with access to a perfect ground truth by generating of two copies of a real LiDAR line cloud. The first copy has been generated by removing 25% of the lines and adding noise. The second copy has been generated by removing 33% of the lines, adding noise and applying an arbitrary known transformation (rotation, scale factor, translation). The objective of this first evaluation is:

• Compare the robustness and the speed of the two optimization algorithms (simulated annealing and RANSAC) in order to choose the best to apply.

• Evaluate the performance of our algorithm using the best optimization algorithm.

• Make a comparison between the efficiency of our algorithm and the Iterative Closest line algorithm (ICL). • we have already studied the LiDAR/LiDAR registration in the first part of our thesis (interior/exterior registration).

Conclusion

• we have already studied the image/LiDAR registration in the second part of our thesis.

• As the 3D segments based algorithm has proven its effectiveness in solving the problem of image/LiDAR registration and LiDAR/LiDAR registration(openings based registration), we hope to test it in solving the image/image registration problem.

State of the art

As the features of aerial and terrestrial data are complementary, several research works are focused on the exploration of joint data processing methods to overcome the shortcomings of methods based on aerial or terrestrial data alone. Aerial and terrestrial data can be acquired as images or 3D point clouds. Integrating data from these two acquisition platforms requires a registration step. Several works in the literature have addressed the problem of aerial/terrestrial registration.

An efficient method for the registration of airborne and terrestrial mobile LiDAR has been proposed in [START_REF] Teo | Surface-based registration of airborne and terrestrial mobile LiDAR point clouds[END_REF]. It used the least squares 3-D surface matching algorithm to minimize the Euclidean surface distance between two

datasets. An algorithm based on line features has been proposed in [START_REF] Hansen | Line-based registration of terrestrial and airborne LiDAR data[END_REF] which represents a potential solution for terrestrial and airborne LIDAR data registration. The basic idea of this algorithm consists in using orientation histograms to find the rotation and a generate and test scheme to find the translation 123 parameters by matching all possible combinations of two line segments. The authors of [START_REF] Cheng | Automatic registration of terrestrial and airborne point clouds using building outline features[END_REF] have been proposed a new solution that can achieve automatic horizontal registration with Airborne Laser Scanning (ALS) and TLS data based on building contour features extracted from both data. A coarse registration has been carried out using the four-point congruent method. Then a refinement of the horizontal registration of the building outlines is conducted. A hierarchical method for registration of ALS and terrestrial image-based point clouds has been proposed in [START_REF] Baghani | Automatic hierarchical registration of aerial and terrestrial image-based point clouds[END_REF]. In order to overcome the problems of the line-based methods, a novel polar parameterized mathematical model was extended for horizontal matching. An innovative target composed of three perpendicular planes that combine the properties of plane and plane and volume target has been implemented in [START_REF] Urbančič | New target for accurate terrestrial laser scanning and unmanned aerial vehicle point cloud registration[END_REF]. This new target enables the precise determination of reference target points in aerial and terrestrial point clouds.

This property make it suitable for accurately registering point clouds produced from Unmanned Aerial Vehicle (UAV) images and terrestrial laser scans.

3D

3D Line segments detection and reconstruction for an orthoimage

For aerial data, we have a single orthoimage. In order to get 3D line segments from this image, we started by detecting 2D lines using our improved version of MLSD.

Afterwards, we got a 3D model from each 2D line by considering that the z coordinate equal to 0 as shown in Figure 6.2

3D segments based algorithm adaptation

The 3D segments based algorithm has proven its efficiency to solve the image/LiDAR registration problem, it can be a potential solution for aerial image/terrestrial image registration. This algorithm takes two 3D segments sets as input. It starts by clustering the 3D segments of each data-set according to their directions. Afterwards, it associates the obtained clusters to find possible rotations. Then 3D segments from associated clusters are matched in order to find the translation and scale factor minimizing a global robust energy between the two segment sets. We have applied the following adaptations on this algorithm to make it able to solve the current registra-tion problem.

Valid cluster associations

To define a rotation between the two segment sets, it is sufficient to associate two pairs of segments from the two sets. However this is not very precise, so we propose to associate the segment clusters. As for aerial data most detected lines lie on the (horizontal) ground, we do not have a vertical cluster, so we have removed the vertical cluster from terrestrial data. Now, we can associate any pair of aerial clusters to any pair of terrestrial clusters if they have a compatible angle.

RANSAC optimization

For the original algorithm, at each RANSAC iteration, we randomly select a valid cluster association, then we randomly select one 3D segment in each of the associated clusters. For the aerial data, we have chosen to work with an orthoimage, where all the segments are coplanar. In this situation, if we only use two segments pairs, we obtained a degenerate scale estimation. By adding a third segment pair, we can get additional information about the distance between segments which can help to correctly estimate the scale factor.

Evaluation and discussion

As it is difficult to construct a ground truth for aerial/terrestrial data registration, we evaluated the registration quality on the 3D visual results. By applying our algorithm, we could register an aerial segment set composed of 271 lines and a terrestrial segment set composed of 846 lines. By fixing the iteration number to 2000 iterations, we got the final registration result after 762.459s. The obtained results demonstrate our algorithm's ability to efficiently register aerial and terrestrial data as shown in

Conclusion

In this chapter, we have addressed the problem of aerial/terrestrial registration that can take several forms depending on the modality (image or LiDAR). We have chosen to address the problem of registration of aerial image / terrestrial image. The very strong difference in point of view makes it difficult to extract comparable features between the two data. Conversely, with our 3D segments and our RANSAC-based approach, we assume that the best transformation is the one which minimizes the global robust energy between all the extracted 3D segments and therefore which is the most consistent with all the information of the data-sets to be registered. The adaptation of our 3D segments based algorithm has made it very effective in dealing with this difficult problem.

Chapter 7

Conclusion

Contributions

The aim of this PhD was to address all registration issues faced by the BIOM project, as a major objective of the project is exploiting jointly data from various sensors (images/LiDAR) and viewpoints (terrestrial indoor or outdoor, aerial). The work carried out has confirmed that the environment and the type of data drive the choice of the registration algorithm. So, our objective was to explore fundamental properties of the data and the acquisition platforms in order to propose potential solutions for all the registration problems encountered by the project. As in a building environment, most objects are composed of geometric primitives (planar polygons, straight lines, openings), we chose to introduce registration algorithms based on these primitives.

The basic idea of these algorithms consists in the definition of a global robust energy between the extracted primitives from the data sets to register and the proposal of a robust method for optimizing this energy based on the RANSAC paradigm. Our solutions have exceeded the limitations of existing algorithms and have proven their effectiveness in solving the challenging problems encountered by the project. The registration of indoor and outdoor scans represents a major challenge for indoor and outdoor building modeling. The lack of overlap between indoor and outdoor data is the most challenging obstacle, more so when both data sets are acquired separately and using different types of sensors. In order to solve this problem, we started by proposing two solutions separately:

• A first solution based on polygon detection and matching. The strong points of this algorithm lies in the fact that it exploits the very small overlap between indoor and outdoor scans of the same building due to the property of laser rays crossing detected fac ¸ades.

• A second solution based on openings, which represent the unique common entities that can be seen from inside and outside. Therefore, it can help the registration of indoor and outdoor point clouds.

Given the obtained results, we can consider that the two proposed algorithms are potential solutions for the indoor/outdoor registration problem, though they have some limitations. In order to overcome these limitations, we have combined the two proposed algorithms. This combination produced an efficient hybrid algorithm able to perform indoor/outdoor registration with great precision.

Image/LiDAR registration and aerial/terrestrial registration are very hard problem.

The strong difference in point of view and modality makes it difficult to extract comparable features between the two modalities. Conversely, with our primitives (3D segments) and our RANSAC-based approach, we assume that the best transformation is the one which minimizes the global robust energy between all the extracted primitives and therefore which is the most consistent with all the information of the data sets to register. The obtained results have proven the ability of our 3D segments based algorithm to deal with this kind of registration problems. Table 7.1 represents a synthesis of our contributions.

have more characteristic features to match.

• Aerial/Terrestrial registration:

Our main perspective for this difficult task is to exploit the different proposed algorithms to deal with the different modalities that can take the problem. For instance, we can perform the aerial LiDAR/terrestrial LiDAR registration using planar polygons as features. We can also solve the aerial LiDAR/terrestrial image registration or the aerial image/terrestrial LiDAR registration problem using 3D segments as features.

ity of an observed geometric event under a noise model (null hypothesis) H 0 (also called, in some papers: the a contrario noise model, the background model or the naive model) and then declare the event meaningful when this probability is small enough. Generally, the independent uniform distribution over the considered elements is represented by the noise model.

The general principle of this approach can be summarized as follows:

• Define a null hypothesis H 0 that is a probability distribution on elements, and take a small number ϵ.

• Observe a geometric event (that is a configuration of elements) in an image, and denote it by E.

• Compute the probability of E under H 0 .

• If this probability is smaller than a threshold computed to ensure that on the average there are less than ϵ detections in an image where the elements would be distributed according to H 0 , then declare E as ϵ-meaningful.

B.3 Line Segment Detector (LSD)

LSD is considered one of the best and most popular methods among proposed line detectors. It accurately detects segments and does not use any threshold tuning, relying instead on the a contrario methodology.

Following [START_REF] Gioi | LSD: a line segment detector[END_REF] The validation step is based on the a contrario approach and it takes into consideration the number of aligned points to inform the decision. Given an image i and a rectangle r, we will note k(r, i) the number of aligned points and n(r) the total number of pixels in r. Then, the expected number of events which are as good as the observed one is

N T est • P H 0 [k(r, I) ≥ k(r, i)], (B.1)
where the number of tests N T est is the total number of possible rectangles being considered, P H 0 is the probability on the a contrario model H 0 and I is a random image following H 0 . Under hypothesis H 0 , the probability that a pixel on the a contrario tially tried (The precision p is initially set to the value T π but other multiples are also tested to cover the relevant range of value).

B.4 LSD parameters improvement B.4.1 Reduce Region Radius

In some cases, the T angle-tolerance method produces a wrong interpretation. This At each iteration, the algorithm removes the farthest pixels of the region to reduce the region's radius to 75 % of its value.

Improvement

Reducing the radius to 75 % from its previous value at each iteration will penalize some pixels that are good candidates. We propose to reduce the radius to 95 % of its value instead 75 %. This solution will:

• Solve the problem that can arise at the region growing process.

• Allow a large number of pixels to be examined.

Convolution with a Gaussian kernel

The scaling is performed by a Gaussian sub-sampling: the image is filtered with a Gaussian kernel to avoid aliasing and then sub-sampled. The standard deviation of

B.6 MLSD improvements B.6.1 Angle tolerance

For empirical reasons, MLSD has set the value of angle tolerance to π 8 , as for LSD. LSD uses an 8-connected neighborhood for region growing. The tolerance T is set to π 8 radian. It is a reasonable choice which makes the probability that a pixel is aligned point at 1 8 .

For MLSD, at the coarsest scale, we only detect segments classically with LSD. At subsequent scales, we first use information from the previous scale to find new segments. So, the components can belong to the same line, or to parallel and close lines that were merged at a coarser scale. In this case a tolerance angle of π 8 is too fine and will eliminate some good segments that can be merged, this is why we have observed that taking a tolerance angle of π 4 remains a reasonable choice and allows to test a very large set of merged segments.

Convolution with a Gaussian kernel

For the pyramid image construction, MLSD smooths each scaled image with a Gaussian kernel. It fixes the σ value to 0.8 and kernel size to 3 × 3. At each scale i, MLSD multiplies the initial σ value by 2 i . So, for some scales we have a small sigma value equal to 0.8, which can not eliminate more noise. For other scales, σ value is very big, which can eliminate more noise but remove more details from the image. To solve this problem, we proposed to fix the σ value for all the scale to 1.6. This value is not small enough (to be able to remove noise) and is not large enough (to not remove details in the image). This choice of the σ value leads us to fix the kernel size at 11 × 11, because the kernel radius must be equal to 3σ + 1.

Bound to quantization error

According to [START_REF] Kim | Detector with focus: Normalizing gradient in image pyramid[END_REF], interpolation while constructing the image pyramid usually causes a difference between the gradients of the original image and the scaled image. Pixels with small gradient magnitude correspond to flat zones. Also, they naturally present a higher error in the gradient direction computation due to the quantization of their values. In LSD and MLSD the pixels with gradient magnitude smaller than a threshold D are therefore rejected. LSD and MLSD compute the threshold D as D = q sin(T ) , (B.6)

where T is the tolerance angle and q is a bound on the possible error in the gradient value due to quantization effects. For empirical reasons, LSD and MLSD fix the q value to 2. From our point of view, using a fixed value for all the scales is not a good solution due to gradient variation. Our proposed solution consists in using different values for this parameter at each scale. For each scaled image, the value of this parameter can be defined according to the noise level.

B.7 Evaluation and discussion

For the moment the evaluation of our improved versions of MLSD and LSD focuses on the visual aspect, i.e. is a qualitative evaluation. This evaluation was carried out on images of different sizes.

B.7.1 Big image

The first evaluation was performed on a big image of 18 Mpixels (size 5184 × 3456)

as

B.8 Conclusion and future works

In this appendix, we have proposed parametric improvements for two existing algorithms LSD and MLSD. the obtained results showed that the improved versions of the two algorithms exceeded the limits of the original versions. For the medium and big images, the improved MLSD represents a potential solution to detect 2D segments.

For the small images, The improved LSD has also proven its effectiveness but improved MLSD remains at the top of the tested detectors. We propose as perspectives to introduce an adaptive threshold for the gradient magnitude calculated according to noise level in each image and add a quantitative evaluation using a ground truth.

3. 1

 1 Schematics illustrating the sensor topology approach . . . . . . . . . 3.2 Pipeline details of sensor topology based RANSAC . . . . . . . . . . . 3.3 Inliers of the estimated planes from an outdoor scan computed with Sensor topology based RANSAC . . . . . . . . . . . . . . . . . . . . . 3.4 Robust estimation of a plane using MSAC. . . . . . . . . . . . . . . . 3.5 Top: Inliers of the estimated planes from an indoor scan, Bottom: Polygons extracted from the prominent plane. . . . . . . . . . . . . . 3.6 Construction of the Delaunay triangulation. (A) Voronoi diagram for a set of points. (B) Delaunay triangulation (Zhou and Yan, 2014). . . 3.7 The basic idea of alpha shape (www.cgal.org) . . . . . . . . . . . . . 3.8 3D line segments detection from an indoor scan and colored according to the scale (where the scale of the point distribution in the neighbourhood of a given point p represents the distance between p and its third closest neighbouring point) : (a) Indoor point cloud, (b) Detected planes, (c) Extracted 3D lines. . . . . . . . . . . . . . . . . 3.9 Reconstructed 3D line cloud: (a) An image from the 2D image sequence, (b) The reconstructed lines using LSD detected lines, (c) The reconstructed lines using MLSD detected lines. . . . . . . . . . . . . 3.10 Openings detection pipeline . . . . . . . . . . . . . . . . . . . . . . . 3.11 Illustration of opening detection steps on an outdoor scan: (a) initial outdoor scan. (b) principle of opening evidence detection. (c) detected openings evidence (green points). (d) connected components of opening evidence. (e) openings outlines (in orange). . . . . . . . . 3.12 Openings detection in an indoor scan. . . . . . . . . . . . . . . . . . 3.13 Openings detection results in several outdoor scans. . . . . . . . . . . 4.1 Outdoor MLS scan acquired with an MMS . . . . . . . . . . . . . . . 4.2 Indoor scan acquired in static mode inside the Zoological Museum of Strasbourg. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Indoor scans of the ground floor of the Zoological Museum of Strasbourg. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Indoor and outdoor scans acquired in static mode at the Zoological Museum of Strasbourg. . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Pipeline Details of planar polygons based registration algorithm. . . . 4.6 LiDAR rays that pass through windows. . . . . . . . . . . . . . . . . . 4.7 Design of the contrario validation method. . . . . . . . . . . . . . . . 4.8 Indoor points detected from an outdoor scan, pink: the detected points, blue : inliers of the vertical plane that represents the fac ¸ade . 4.9 Result of the test of our algorithm on two pairs of indoor/outdoor scans, Top: the initial position of each pair, Bottom: The position of each pair after registration, pink points represent the indoor points detected from the outdoor scans. . . . . . . . . . . . . . . . . . . . . 4.10 Indoor/Outdoor registration results: (a) Position of the two scans before registration, (b) the position after registration (Green: indoor scan, Blue: outdoor scan, Pink: outline of outdoor openings). . . . . 4.11 Uncertainties of the two proposed solutions, (a) planar polygons based solution: an uncertainty in the horizontal direction parallel to the facade, (b) openings based solution, an uncertainty in the direction orthogonal to the facade. . . . . . . . . . . . . . . . . . . . . 4.12 Indoor/Outdoor registration results by combining polygons and openings:(a) the result of the first step by considering only the openings, (b) the final result by adding the polygons translation , indoor scan (green points), outdoor scan(blue points), indoor points of the outdoor scan (black points), openings outlines of the outdoor scan (red lines). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Pipeline details of our proposed solution for image/LiDAR data registration problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Comparison of the convergence speed and the robustness of RANSAC and simulated annealing: X axis represents time in milliseconds, Y axis represents the energy, green points represent the minimums estimated by RANSAC and the red points represent the minimums estimated by simulated annealing. . . . . . . . . . . . . . . . . . . . . . 5.3 Comparison between the performance of ICL and our algorithm. (a) Initial position, (b) ICL result, (c) Our result . . . . . . . . . . . . . . . 5.4 Terrestrial image/Terrestrial LiDAR registration results: (a) position of the two line clouds before registration (red:image lines, black LiDAR lines), (b) position of the two line clouds after registration, (c,d) registration of image lines and the LiDAR scan. . . . . . . . . . 5.5 Aerial image/Aerial LiDAR registration results: (a) aerial LiDAR scan, (b) extracted lines from the aerial LiDAR scan, (c) an aerial image, (d) Reconstructed lines from an aerial image sequence , (e) Registration result. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Reconstructed 3D line cloud from a Terrestrial image sequence: (a) An image from the terrestrial image sequence,(b) MLSD detected lines, (c) The reconstructed lines. . . . . . . . . . . . . . . . . . . . 6.2 Reconstructed 3D line cloud from an orthoimage: (a) The orthoimage,(b) MLSD detected lines, (c) The reconstructed lines. . . . . . . . 6.3 Aerial/Terrestrial registration: 3D lines reconstructed aerial images(red), 3D lines reconstructed from terrestrial images (blue) (a): the initial position, (b): positions after registration, (c): position of pedestrian path lines after registration . . . . . . . . . . . . . . . . . . . . . . . B.1 Line Support Regions illustration (Von Gioi et al., 2012) . . . . . . . B.2 Aligned points illustration (Von Gioi et al., 2012) . . . . . . . . . . . B.3 Estimation of the number of tests. (Von Gioi et al., 2012) . . . . . . . B.4 An example of region growing problems . . . . . . . . . . . . . . . . B.5 Comparison between the performance of LSD and MLSD before and after improvement on a big size image . . . . . . . . . . . . . . . . . B.6 Comparison between the performance of LSD and MLSD before and after improvement on a small size image . . . . . . . . . . . . . . . . B.7 Comparison between the performance of LSD and MLSD before and after improvement on a medium size image . . . . . . . . . . . . . . B.8 Another comparison between the performance of LSD and MLSD before and after improvement on a medium size image . . . . . . . . .

  et se limitent alors souvent à une seule pièce, au mieux un seul étage, utilisant des a priori forts de parallélisme et d'orthogonalité qui ne sont pas nécessairement vérifiés. De notre point de vue, aucune méthode n'a proposé une modélisation conjointe intérieur/extérieur dans le cadre d'un formalisme unifié. La numérisation d'un objet à partir d'une seule station d'acquisition est souvent difficile. Pour couvrir toutes les faces de l'objet, plusieurs points de vue différents sont nécessaires. Cela est dû à des raisons diverses, soit le champ de vision du scanner est limité, soit l'objet est de dimension ou d'architecture complexe, qui pourrait être aussi un obstacle empêchant une visée directe et génèrant des masquages sur la surface à scanner. Les nuages de points provenant de ces stations multiples sont définis chacun dans un repère relatif au scanner laser. Un regroupement dans un même système de coordonnées de l'ensemble des nuages de points acquis depuis les différentes stations est nécessaire. Ce processus s'appelle recalage ou encore consolidation. Les travaux menés ont confirmé que l'environnement et le type de données conditionnent le choix de l'algorithme de recalage. Notre contribution consiste à exploiter les propriétés géométriques et physiques des données et des plateformes d'acquisition afin de proposer des solutions potentielles à tous les problèmes de recalage rencontrés par le projet. Comme dans un environnement de bâtiments la plupart des objets sont composés de primitives géométriques (polygones planaires, lignes droites, ouvertures), nous avons choisi d'introduire des algorithmes de recalage basés sur ces primitives. L'idée de base de ces algorithmes consiste en la définition d'une énergie globale entre les primitives extraites à partir des jeux de données à recaler et la proposition d'une méthode robuste pour optimiser cette énergie basée sur le paradigme RANSAC. Nos contributions vont de la proposition de méthodes robustes pour l'extraction des primitives sélectionnées à l'intégration de ces primitives dans un cadre de recalage efficace. Le recalage intérieur/extérieur avec une précision proche de celle d'un scan (environ 1 cm) représente un défi majeur pour la modélisation intérieure/extérieure des bâtiments. L'absence de chevauchement entre les données intérieures et extérieures est l'obstacle le plus souvent rencontré, d'autant plus lorsque les deux ensembles de données sont acquis séparément et à l'aide de différents types de capteurs. Pour faire face à ce problème, nous avons proposé deux solutions initiales puis nous les avons combinées. La première solution repose sur la détection et la mise en correspondance des polygones planaires. Le point fort de cette solution réside dans le fait qu'elle exploite le très faible chevauchement entre les scans intérieurs et extérieurs du même bâtiment en détectant les points situés à l'intérieur des bâtiments lors des scans extérieurs comme des points où le rayon laser traverse les fac ¸ades détectées à travers les ouvertures. Cette solution définit une énergie globale robuste entre deux ensembles de polygones et propose une méthode robuste pour maximiser cette énergie basée sur le paradigm RANSAC.

  but also from an aerial point of view to model roof. The BIOM project also aims at exploiting the complementarity of image and LiDAR data. Another challenge is coping with incomplete data due to occlusions by furniture inside and urban and mobile objects outside. Last but not least, BIOM aims at modeling a large variety of architectural styles, different interior scene layouts, and a high amount of different objects that may be contained within the scene. State-of-the-art approaches treat outdoor and indoor worlds separately: most indoor reconstruction approaches focus on detailed modelling of single rooms whereas only very few have dealt with 3D modelling of complete floors (under Manhattan world assumptions). To the best of our knowledge, no works have been proposed, yet, that model buildings outdoor and indoor simultaneously within one single comprehensive framework. The figure1.1 shows the main acquisition modes of the BIOM project. This project is funded by ANR, the French national research funding agency.

Figure 1

 1 Figure 1.1: The main acquisition modes of the BIOM project

Figure 1 . 2 :

 12 Figure 1.2: Overview of our contribution

Figure 1 . 3 :

 13 Figure 1.3: Possible schemes for our algorithms.

  Aerial LiDAR/ Terrestrial LiDAR registration: using planar polygons based algorithm • Aerial image/ Terrestrial LiDAR registration: using 3D segments based algorithm • Aerial LiDAR/ Terrestrial image registration: using 3D segments based algorithm • Aerial image/ Terrestrial image registration: using 3D segments based algorithm

Figure 2

 2 Figure 2.1: Image Acquisition Model[START_REF] Mishra | Image acquisition and techniques to perform image acquisition[END_REF]

  Figure 2.2. There are two CMOS detector types: Passive Pixel and Active Pixel.

Figure 2

 2 Figure 2.2: Image Acquisition Model[START_REF] Mehta | Ccd or cmos image sensor for photography[END_REF]

  tracking and time synchronized data acquisition, regardless of the platform in use. A GNSS receiver(s) with appropriate antenna(s) for positioning and an IMU for platform and sensor attitude determination and high-frequency positioning represent the main components of the navigation system. Mobile laser scanning (MLS): Mobile laser scanning (MLS), is a rapid and flexible method for acquiring three-dimensional topographic data. Mobile Laser Scanning, being a sub-category of MMS, based on Light Detection and Ranging (lidar) sensors is a rapid and flexible technology for acquiring high-resolution three-dimensional data.

  Figure 2.3. The survey is conducted as the ground vehicle moves around, while the navigation system tracks the vehicle's trajectory and sensor attitude, used to produce a 3D point cloud from the range data collected by the onboard scanners. When the ground vehicle moves around, the survey is conducted. The vehicle's trajectory and sensor attitude, used to produce a 3D point cloud from the range data collected by the onboard scanners, are tracked by the navigation system. The survey is conducted as the ground vehicle moves around (Kukko et al., 2013).

Figure 2 . 3 :

 23 Figure 2.3: Mobile laser scanning utilizes GNSS/IMU positioning for direct georeferencing of point clouds[START_REF] Kukko | Mobile Laser Scanning-System development, performance and applications[END_REF]

Figure 2

 2 Figure 2.4: The pulse sensor topology forms a 6-neighborhood: the points considered is colored in red, and connection is denoted by a red arrow

Figure 2

 2 Figure 2.6: Schematical illustration of ALS data acquisition[START_REF] Roncat | The geometry of airborne laser scanning in a kinematical framework[END_REF]

  3D data can be represented in several forms. The most used representations are: depth map and point cloud. Each representation is adapted to some families of processing methods. The core of 3D data processing is the extraction and the reconstruction of geometric information. The point is considered a basic element of geometry.

Figure 3 . 1 :

 31 Figure 3.1: Schematics illustrating the sensor topology approach

Figure 3 . 2 :

 32 Figure 3.2: Pipeline details of sensor topology based RANSAC

Figure 3

 3 Figure 3.5: Top: Inliers of the estimated planes from an indoor scan, Bottom: Polygons extracted from the prominent plane.

from

  Delaunay triangulation, which offers a concrete definition of a shape to represent the structure of a set of points. It represents a Delaunay triangulation (2D or 3D) for which each of the triangles or tetrahedron ("simplex") is compared to the value of an alpha (α) parameter determined by the Euclidean distance between the points 3.7.

Figure 3

 3 Figure 3.7: The basic idea of alpha shape (www.cgal.org)

  The key idea is to segment the input data into facets by performing local k-means clustering on carefully selected seeds. These facets provide sufficient information for determining linear features in the local planar region. The false positive detected line segments are filtered by introducing the concept of Number of False Alarms (NFA) into 3D point cloud context. The authors of[START_REF] Hackel | Contour detection in unstructured 3d point clouds[END_REF] presented a contour detector in unstructured 3D point clouds based on three main steps:

  to extract 3D line segments from the LiDAR data. It is a simple and efficient algorithm that starts by segmenting the point cloud into planar 3D regions via region growing and merging. All the points belonging to the same planar region are projected into the supporting plane of this region to form a 2D image. Then, 2D contour extraction and least square fitting are performed to detect 2D line segments. Finally, these 2D line segments are transformed back into the 3D frame to get the 3D segments as shown in figure 3.8. This algorithm belongs to the image based category.

Figure 3

 3 Figure 3.8: 3D line segments detection from an indoor scan and colored according to the scale (where the scale of the point distribution in the neighbourhood of a given point p represents the distance between p and its third closest neighbouring point) : (a) Indoor point cloud, (b) Detected planes, (c) Extracted 3D lines.

  solve the uncertainties in 3D line segments reconstruction through LS match grouping, space plane estimation and image LS back-projection. By introducing a new minimal four-vector line representation based on Plücker coordinates, the authors of (Zhou et al., 2020) propose a method to analyze the uncertainty of line reconstruction. Each component of the compact representation has a certain physical meaning about the 3D line's orientation or position. The reliability of the reconstructed lines can be measured through the confidence interval of each component in the proposed representation. The article provides a quantitative evaluation of accuracy.

  Figure 3.9: Reconstructed 3D line cloud: (a) An image from the 2D image sequence, (b) The reconstructed lines using LSD detected lines, (c) The reconstructed lines using MLSD detected lines.

  The first strategy used Principal Component Analysis (PCA) to cluster point clouds into potential facade regions, while the second extracts the facade by applying RANSAC (Random Sample Consensus) plane fitting to potential facade regions. Finally, the windows detection is achieved in two main steps: potential window point detection and window localization. The authors of[START_REF] Tuttas | Window detection in sparse point clouds using indoor points[END_REF] have proposed a new method for detecting windows in airborne multi-aspect laser scanning data which represent a sparse point cloud. They started by detecting facade planes using point normals and Region Growing (RG) algorithm. Then, the windows are detected using the indoor points (points lying behind the detected facade planes)

Figure 3

 3 Figure 3.10: Openings detection pipeline

Figure 3 .

 3 Figure 3.11: Illustration of opening detection steps on an outdoor scan: (a) initial outdoor scan. (b) principle of opening evidence detection. (c) detected openings evidence (green points). (d) connected components of opening evidence. (e) openings outlines (in orange).

Figure 3 .

 3 Figure 3.12: Openings detection in an indoor scan.

Figure 3 .

 3 Figure 3.13: Openings detection results in several outdoor scans.

  ing. The lack of overlap between indoor and outdoor data is the most encountered obstacle, more so when both data sets are acquired separately and using different types of sensors. State of the art approaches generally deal with either the indoor or the outdoor, and often use strong priors of parallelism and orthogonality that are not necessarily verified. To achieve the desired accuracy, clues must be extracted from internal and external data and then matched. It is very restrictive because these clues should be visible from the inside and the outside. In this work, we explored fundamental properties of our data to propose potential solutions for this challenging problem. Given the specificity of the Lidar data that crosses the windows, we have performed an extraction of the planar polygons that offer us a compact and localized representation through the detection of the facades (vertical planar polygons with sufficient extent). Then, we used them to propose an efficient registration algorithm based on planar polygons as the first solution.

  openings like windows). reasons as shown inFigures 4.2 and 4.3. The Figure 4.4 shows some indoor and outdoor scans acquired in static mode at the zoological museum of Strasbourg. The indoor acquisition was performed using the FARO Laser scanner Focus3D X 330.FARO Laser scanner Focus3D X 330: According to[START_REF] Kallisto | Scanner 3d -laser -faro focus 3d x330[END_REF], FARO Laser scanner Focus3D X 330 is ideal for both indoor and outdoor surveying applications.It offers everything you would expect from a professional laser scanner. The FARO Laser scanner Focus3D X 330 has an extra long range of 330 m. It is equipped with a GPS and adapted for outdoor scanning, even in strong sunlight. Its accuracy is of the order of ± 2 mm and its acquisition speed reaches up to 976,000 points/second.

Figure 4 . 2 :

 42 Figure 4.2: Indoor scan acquired in static mode inside the Zoological Museum of Strasbourg.

Figure 4 . 3 :

 43 Figure 4.3: Indoor scans of the ground floor of the Zoological Museum of Strasbourg.

Figure 4 . 4 :

 44 Figure 4.4: Indoor and outdoor scans acquired in static mode at the Zoological Museum of Strasbourg.

  or primitive correspondences. The basic idea of our contribution is to perform an extraction of the planar polygons, from both the indoor and outdoor data, then to match them by clustering polygons according to their normal direction, then by their offset in the normal direction. We use this clustering to find possible polygon correspondences (hypotheses) and estimate the optimal transformation for each hypothesis. Finally, a global robust energy is computed for each hypothesis in order to select the best one as shown in Figure4.5. The polygon extraction step has been carried out using the methods proposed in Section 3.4.2. The interest of polygons relative to planes is that they have a spatial extent limited to the areas where they have supporting points in the input data, so they form a simple and compact summary of our LiDAR scans. Given the specificity of LiDAR data that pass through windows, we propose to start with the extraction of the buildings' interior points captured from external scans; these are the points where the laser ray crosses the fac ¸ades through apertures, mostly windows. Afterwards, we perform the registration.

Figure 4

 4 Figure 4.5: Pipeline Details of planar polygons based registration algorithm.

Figure

  Figure 4.6: LiDAR rays that pass through windows.

  Figure 4.8. Finally, the polygon detection algorithm of Section 3.4.2 (polygon MSAC) is run only on the points detected as indoor from the outdoor scan, yielding a limited number of polygons.

Figure 4

 4 Figure 4.8: Indoor points detected from an outdoor scan, pink: the detected points, blue : inliers of the vertical plane that represents the fac ¸ade

Figure 4 . 9 :

 49 Figure 4.9: Result of the test of our algorithm on two pairs of indoor/outdoor scans, Top: the initial position of each pair, Bottom: The position of each pair after registration, pink points represent the indoor points detected from the outdoor scans.

Figure 4 .

 4 Figure 4.10: Indoor/Outdoor registration results: (a) Position of the two scans before registration, (b) the position after registration (Green: indoor scan, Blue: outdoor scan, Pink: outline of outdoor openings).

  Figure 4.11: Uncertainties of the two proposed solutions, (a) planar polygons based solution: an uncertainty in the horizontal direction parallel to the facade, (b) openings based solution, an uncertainty in the direction orthogonal to the facade.

  Figure 4.12: Indoor/Outdoor registration results by combining polygons and openings:(a) the result of the first step by considering only the openings, (b) the final result by adding the polygons translation , indoor scan (green points), outdoor scan(blue points), indoor points of the outdoor scan (black points), openings outlines of the outdoor scan (red lines).

  ploited the geometric and physical properties of our data to propose potential solutions for this challenging problem. As we worked in a building environment, where most objects are represented by a planar surface that can be simplified by planar polygons, we started by proposing a planar polygons based registration algorithm. This first algorithm matches planar polygons by clustering polygons according to their normal direction, then by their offset in the normal direction. It uses this clustering to find possible polygon correspondences (hypotheses) and estimate the optimal transformation for each hypothesis. The best transformation is selected according to a global robust energy. An opening in the fac ¸ade is the unique common entity that can be seen from inside and outside. It can help the registration of indoor and outdoor point clouds. For this reason, we have integrated openings in a registration framework in order to propose a second solution. As an opening is defined by a rectangular shape composed of four segments, two horizontal and two vertical, we preferred to write our registration problem as a minimization of a global robust energy between two segment sets. The two proposed solutions are very efficient for realizing the indoor/outdoor registration. But they have some limitations. In order to overcome these limitations and benefit from the advantages of the two solutions, we have tried to combine them. This combination allows us to generate a hybrid solution able of performing indoor/outdoor registration with good qualitative precision.

Figure 5 . 1 :

 51 Figure 5.1: Pipeline details of our proposed solution for image/LiDAR data registration problem.

Algorithm 5

 5 Simulated Annealing Input: M 0 : initial transformation (randomly chosen valid cluster association and randomly chosen scale/translation) E 0 : initial transformation energy G: Current transformation T max : initial temperature T min : final temperature α: cooling rate M axIter: maximum number of iterations G = M 0 , E(G) = E 0 while T max > T min do for Iter = 1 . . . MaxIter do With a small probability p jump , randomly select a new valid cluster association and corresponding rotation. Randomly sample a new scale/translation close to the current one, and call M N the resulting transformation.

Figure 5 . 2 :Figure 5

 525 Figure 5.2: Comparison of the convergence speed and the robustness of RANSAC and simulated annealing: X axis represents time in milliseconds, Y axis represents the energy, green points represent the minimums estimated by RANSAC and the red points represent the minimums estimated by simulated annealing.

  Figure 5.5: Aerial image/Aerial LiDAR registration results: (a) aerial LiDAR scan, (b) extracted lines from the aerial LiDAR scan, (c) an aerial image, (d) Reconstructed lines from an aerial image sequence , (e) Registration result.

  Figure 6.1: Reconstructed 3D line cloud from a Terrestrial image sequence: (a) An image from the terrestrial image sequence,(b) MLSD detected lines, (c) The reconstructed lines.

  Figure 6.2: Reconstructed 3D line cloud from an orthoimage: (a) The orthoimage,(b) MLSD detected lines, (c) The reconstructed lines.

Figure 6 . 3 .

 63 Figure 6.3. By inspecting the pedestrian path, we can clearly see that the registration error does not exceed a few centimeters

  Figure B.1: Line Support Regions illustration (Von[START_REF] Gioi | LSD: a line segment detector[END_REF]

Figure B. 2 :

 2 Figure B.2: Aligned points illustration[START_REF] Gioi | LSD: a line segment detector[END_REF]

  Figure B.4: An example of region growing problems

  shown in Figure B.5. LSD vs improved LSD With the two versions of LSD, we could not detect all the possible lines. The detected lines with LSD are very short as shown in Figure B.5(a). Some detected lines have become longer with the improved version as shown in Figure B.5(b). MLSD vs improved MLSD With the two versions of MLSD, we have detected more lines than LSD. The detected lines with MLSD are longer than the detected lines by lsd by they are interrupted as shown in Figure B.5(c). With the improved MLSD, we have detected more lines than MLSD and most of them are complete as shown in Figure B.5(d).

(

  Figure B.6: Comparison between the performance of LSD and MLSD before and after improvement on a small size image

(

  Figure B.7: Comparison between the performance of LSD and MLSD before and after improvement on a medium size image

  1.1 Publication List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Comparison between CCD and CMOS . . . . . . . . . . . . . . . . . 2.2 Main characteristics and performance of terrestrial scanner solutions (Coccia, 2021) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Comparison of LiDAR systems mounted on different platforms (Cheng, Success Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chen, Liu, Xu, Wu, Li and Chen, 2018) . . . . . . . . . . . . . . . . . 3.1 Comparison between primitives extraction methods . . . . . . . . . . 4.1 Result of indoor/indoor registration tests . . . . . . . . . . . . . . . . 5.1 Performance tests of our algorithm on synthetic data using different initial errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 6.1 Possible solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 7.1 Synthesis of our contributions . . . . . . . . . . . . . . . . . . . . . . 131

List of Figures

1 1.1 The main acquisition modes of the BIOM project . . . . . . . . . . . 1.2 Overview of our contribution . . . . . . . . . . . . . . . . . . . . . . 1.3 Possible schemes for our algorithms. . . . . . . . . . . . . . . . . . . 2.1 Image Acquisition Model(Mishra et al., 2017) . . . . . . . . . . . . . 2.2 Image Acquisition Model

  minimisation de la distance sur ces chevauchements. Afin de simplifier cette maximisation, nous avons procédé à une étape de regroupement des polygones planaires de chaque ensemble selon la direction de la normale. Ce regroupement a généré trois clusters directionnels (un horizontal et deux verticaux). L'association de ces clusters nous a permis de trouver la rotation optimale. La génération et l'évaluation des hypothèses (des correspondances possibles de polygones) nous ont permis de trouver la translation optimale. La deuxième solution initiale repose sur la détection et la mise en correspondance des ouvertures. Comme une ouverture est considérée parmi les entités communes qui peuvent être vues de l'intérieur et de l'extérieur, elle peut aider au recalage des nuages de points intérieurs et extérieurs. Le recalage DAR fournissent des informations de surface très précises et robustes. D'autre part, l'image fournit des détails à haute résolution, mais la géométrie 3D estimée à partir de l'appariement dense est moins robuste et précise. Par conséquent, l'intégration des données de ces deux sources peut conduire à une segmentation sémantique robuste et une reconstruction complète de scènes 3D. Donc, un autre objectif de cette thèse consiste à résoudre le problème du recalage image/données LiDAR. Ce problème est très difficile compte tenu de la très forte différence de modalité. Contrairement aux plateformes terrestres qui donnent des informations complètes et denses sur les faces latérales et ne donnent pas d'informations significatives sur les faces supérieures. Donc, afin d'obtenir une couverture complète des zones urbaines, la fusion des vues aériennes et terrestres est nécessaire. Les données aériennes peuvent être acquises sous forme d'images ou des données LiDAR et c'est le même cas pour les données terrestres. Donc le problème du recalage aérien/terrestre peut prendre plusieurs formes :

	Cette différence rend difficile l'extraction de caractéristiques comparables entre les
	deux modalités. À l'inverse, avec nos primitives et notre approche basée sur RANSAC,
	nous supposons que la meilleure transformation est celle qui optimise une energie

est généralement effectué en faisant correspondre des caractéristiques ou bien en optimisant une energie globale entre les données à recaler. Dans notre cas, les caractéristiques sont deux ensembles d'ouvertures verticales détectées à partir de scans intérieurs et extérieurs. Cependant, ces ouvertures ne sont pas assez caractéristiques pour permettre les mises en correspondance de manière robuste. Comme une ouverture est définie par une forme rectangulaire composée de quatre segments, deux horizontaux et deux verticaux, nous avons modélisé notre problème de recalage comme une minimisation d'une energie globale robuste entre deux ensembles de segments et nous avons proposé une méthode robuste pour minimiser cette energie suivant le paradigme RANSAC. Sachant que la minimisation de cette énergie va entraîner une maximisation de chevauchement et minimisation de la distance sur ces chevauchements. Les deux solutions initiales sont très efficaces pour réaliser le recalage intérieur/extérieur, mais elles ont quelques limitations. La solution basée sur des polygones planaires a une incertitude dans la direction horizontale parallèle à la fac ¸ade, tandis que la solution basée ouvertures a une incertitude dans la direction orthogonale à la fac ¸ade. Donc, afin de supprimer ces incertitudes, nous avons proposé une solution basée sur des associations de polygones planaires et d'ouvertures. Cette nouvelle solution a prouvé sa performance pour recaler les scans intérieurs et extérieurs. L'analyse et la reconstruction de scènes 3D à partir d'images et données LiDAR est un domaine de recherche actif en vision par ordinateur. D'une part, les données Li-globale robuste entre toutes les primitives extraites et donc qui est la plus cohérente avec toutes les informations des jeux de données à recaler. Notre algorithme de recalage basé sur des segments 3D représente une solution potentielle à ce genre de problème. Cet algorithme est basé sur la minimisation d'une énergie globale robuste, qui a été définie dans l'algorithme de recalage basé ouvertures, entre deux ensembles de segments 3D. Cette minimisation est basée sur le paradigme RANSAC. Afin de simplifier cette minimisation, nous avons commencé par regrouper les segments 3D de chaque jeu de données selon leurs direction. Les clusters obtenus sont associés pour trouver les rotations possibles, puis les segments 3D des clusters associés sont mis en correspondance afin de trouver le facteur d'échelle et la translation minimisant l'énergie définie. Nous avons choisi de travailler avec des segments 3D pour résoudre ce problème parce que dans un environnement bâti, il semble que la meilleure caractéristique à utiliser pour le recalage image/données Li-DAR soit le segment 3D. Les segments 3D peuvent être extraits de manière fiable, précise et automatique à la fois à partir des données LiDAR et à partir des données image surtout si nous avons un chevauchement suffisant d'images. Elles agrègent plus d'informations que les points, sont donc moins sensibles au bruit et sont plus fréquentes que les primitives plus complexes (rectangles). L'objectif de cette thèse est non seulement le recalage des données hétérogènes (image et données LiDAR) mais aussi le recalage des données acquises par des plateformes d'acquisition hétérogènes (aériennes/terrestres). En utilisant des plateformes aéroportées au milieu urbain, on peut voir les faces supérieures des bâtiments, mais les faces latérales manquent toujours.

du cluster vertical pour les données terrestres :

	Comme pour les données aériennes, la plupart des lignes détectées se trouvent
	sur le sol (plan horizontal), nous avons donc supprimé le cluster vertical des
	données terrestres. Puis nous avons associé n'importe quelles paires de clusters
	terrestres avec n'importe quelles paires de clusters aériens s'ils sont un angle
	compatible.

• Recalage de l'image aérienne et le LiDAR terrestre en utilisant l'algorithme basé sur les segments 3D. Dans le cadre de cette thèse, nous avons choisi de traiter le problème du recalage aérien/terrestre sous forme de problème du recalage de l'image aérienne et l'image terrestre. Ce choix n'est pas arbitraire, il est basé sur les raisons suivantes : • Nous avons déjà étudié le recalage (données LiDAR/données LiDAR) où notre algorithme basé sur les polygones planaires a prouvé son efficacité pour traiter le problème du recalage intérieur/extérieur. • Nous avons déjà étudié le recalage (image/ données LiDAR) comme un deuxième objectif de notre thèse où notre algorithme basé sur les segments 3D a prouvé sa performance. • Comme l'algorithme basé sur les segments 3D a prouvé son efficacité pour résoudre le problème du recalage (image/données LiDAR) et recalage (données LiDAR/données LiDAR), recalage basé sur les ouvertures, nous sommes motivés pour tester sa performance pour résoudre des problèmes de recalage image/im-age. Nous avons réalisé le recalage d'une ortho-image aérienne et une séquence d'images terrestres. Ce recalage a été realisé en utilisant l'algorithme basé sur les segments 3D après avoir appliquer les adaptations suivantes : • Suppression • Sélection de trois paires de segments pour RANSAC : Pour l'algorithme original, à chaque itération de RANSAC, nous sélectionnons aléatoirement une association valide de cluster, puis nous sélectionnons aléatoirement un segment 3D dans chacun des clusters associés. Pour les données aériennes, nous avons choisi de travailler avec une ortho-image, où tous les segments sont coplanaires. Dans cette situation, si nous n'utilisons que deux paires de segments, nous obtenons une estimation d'échelle dégénérée. En ajoutant une troisième paire de segments, nous pouvons obtenir des informations supplémentaires sur la distance entre les segments, ce qui peut aider à estimer correctement le facteur d'échelle.

Les différentes solutions proposées dans le cadre de notre thèse ont dépassé les limites des algorithmes existants et ont prouvé leur efficacité pour résoudre les problèmes de recalage rencontrés par le projet ANR BIOM.

Table 2 .

 2 3 shows a simple comparison between the main laser scanning systems: MLS,

	TLS and ALS.
	Platforms System	Scanning	Scanning	Point Cloud	Application Ar-
		Abbre-	Perspective	Range	Density	eas example
		viation
	Airborne ALS	Top view	Surface	Relatively	Terrain map-
				shape	sparse	ping, 3D urban
						areas
	Vehicle	MLS	Side view	Stripe	Dense	Road mapping,
				shape		3D urban areas
	Tripod	TLS	Side view	Point	Dense	Deformation
				shape		monitoring,
						reverse engi-
						neering

Table 2

 2

.3: Comparison of LiDAR systems mounted on different platforms

[START_REF] Cheng | Registration of laser scanning point clouds: A review[END_REF]

Table 3

 3

	Method	Advantages	Limitations
		+ Meaningful segmentation -Sensitive to initials condi-
	Clustering	+ Few parameters	tions (seeds) -Sensitive to noise and out-
			liers
		+ Special case: J-Linkage
		allows to detect multiple in-
		stances of a model at the
		same time
	Vote accumulation	+ Potential solution for missing data	-Dependent on parameters space quantization
		+ Pretty natural for the de-
		tection of multiple objects
	Hypothesis and validation	+ Simple + Robust to outliers	-Need many parameters -Requires a significant num-
			ber of iterations depending
			on the data size
	Selection and decision	+Potential solution for con-	-Dependent of fixed thresh-
		trolling false detection	old over the probability of
			false alarm

represents a potential solution for this kind of problem. A primitive detection algorithm has two main parts: model candidates selection and validation. Performing a validation based on the approach allows a control of the expected number of false detection, which does not require parameter settings. LSD (Line Segment .1: Comparison between primitives extraction methods Detector) is the most popular algorithm of this family of methods. It has been proposed in (Von Gioi et al., 2012) to detect locally straight contours in an image.

  Algorithm 2 Associations Creation Input: a set of supporting planes of polygons S 1 = {P p } and a set of supporting planes of openings S 2 = {P o } extracted form the same scan. Output: a list of associations of hybrid supporting planes L = {A(P o , P p )}. for each supporting plane P o do for For each supporting plane P p do if Angle(P o ⊥ , P p ⊥ ) < ϵ then Create an association A(P o , P p ) and add it to L.

	end if
	end for
	end for

  Our algorithm takes 3D segments as feature and extracts them from both LiDAR and image data with specific state of the art algorithms. We used the global robust energy between two segments sets, which has been defined in Section 4.5.1, and we proposed a robust approach to minimize this energy. In order to simplify this minimization, we started by clustering the 3D segments of each data set. The clusters are associated to find possible rotations, then 3D segments from associated clusters are matched in order to find a the translation and scale factor minimizing the defined energy. The obtained results demonstrate the efficiency and robustness of our algorithm to register heterogeneous data. Our main perspective on this work is to use planar polygons as primitives or to use combinations of more segments to have more characteristic features to match.

		Aerial LiDAR	Aerial Image
	Terrestrial LiDAR Planar polygons based algorithm 3D segments based algorithm
	Terrestrial Image	3D segments based algorithm	3D segments based algorithm
		Table 6.1: Possible solutions
	As we have four possible ways to solve the same problem (aerial/terrestrial regis-
	tration), we have chosen to exploit one of the four (aerial image/ terrestrial image
	registration) for the following reasons:

In this chapter, we have dealt with the heterogeneous data (Image/LiDAR data) registration problem by proposing a new primitives based registration algorithm.

Data

The data used for this study was acquired by two different means (mobile LiDAR scan for the outside and static scans for the inside) on the Zoological Museum of Strasbourg.

Outdoor data

The outdoor data used to experiment our method is a Mobile Laser Scanning (MLS) (cf Fig. 4.1 acquired with the Stéréopolis II Mobile Mapping System (MMS) [START_REF] Paparoditis | Stereopolis ii: A multi-purpose and multi-sensor 3d mobile mapping system for street visualisation and 3d metrology[END_REF]. The acquisition system gives access to the sensor topology inherent to such MLS acquisitions, that is usually lost during export to formats such as .las or .ply. The data was collected from three streets, north, south and east of the museum, the west fac ¸ade facing a park inaccessible to the MMS. Each outdoor scan contains approximately 3 million points.

Indoor data

The indoor data used in our study is composed of 30 static LiDAR scans of the inside of the Musée Zoologique, one or two per room. Each scan consists of roughly 500 million points, which were downsampled to around 2 million points for practical

Chapter 5

Image/LiDAR data Registration

Introduction

The volume of datasets acquired by optical and LiDAR systems is rapidly increasing due to demand from a variety of applications including remote sensing, 3D mapping and autonomous driving. 3D scene analysis and reconstruction from image and Li-DAR is an active research area in computer vision. On the one hand, the LiDAR data provides highly accurate and robust surface information. On the other hand, the image provides high resolution details and rich spectral information [START_REF] Kim | Integration of photogrammetric and Li-DAR data for realistic 3d model generation[END_REF] but the 3D geometry estimated from dense matching is less robust and accurate, in particular in homogeneous, specular or repetitive regions and near depth discontinuities. Hence, integrating data from these two sources can lead to a more robust and complete semantic segmentation and reconstruction of 3D scenes. Following a classical methodology (Kumar [START_REF] Kumar Mishra | A review of optical imagery and airborne LiDAR data registration methods[END_REF], the problem of image/Li-DAR data registration is decomposed in three main steps: (i) Feature extraction, (ii) Feature matching, (iii) Transformation model estimation.

State of the art

There is a considerable amount of prior work on image/LiDAR registration. We start by classifying some existing methods according to their types and attributes, citing the advantages and limitations of each category.

Keypoint based methods

Corners are the most used key points for image/image registration, which can be extended to image/LiDAR registration [START_REF] Ding | Automatic registration of aerial imagery with untextured 3d LiDAR models[END_REF]. The uniqueness and high precision in the localization are the strongest properties of corners [START_REF] Kumar Mishra | A review of optical imagery and airborne LiDAR data registration methods[END_REF]. However, corners are not always easily matched between LiDAR and image.

Linear feature based methods

Straight lines are the most used feature for image/LiDAR registration because they can be automatically, accurately and efficiently extracted from the LiDAR and image data [START_REF] Habib | Integration of photogrammetric and LiDAR data in a multi-primitive triangulation environment[END_REF], especially if we have a set of overlapping images [START_REF] Deng | Automatic registration between LiDAR and digital images[END_REF].

Structural feature based methods

High level structural features such as rectangles can help increasing the robustness of both detection and matching steps for image/LiDAR registration. These structural features can be extracted from both data sets as connected segments [START_REF] Liu | A systematic approach for 2d-image to 3d-range registration in urban environments[END_REF]. This kind of features can be efficiently used to estimate camera translation.

Camera rotation can be estimated using at least two vanishing points. The major limitation of these methods consists in the dependence on either the strong presence of parallel lines to infer vanishing points or availability of feature pair correspondences [START_REF] Wang | Automatic registration method for mobile LiDAR data[END_REF]. Moreover, these methods are not effective when applied to aerial data.

Mutual information based methods

Statistical and information theoretic methods have demonstrated excellent performance for a wide variety of 2D-2D and 2D-3D registration applications. Mutual information, a central concept of information theory, consists in measuring the statistical correlation of two random variables, which is a measure of the amount of information one random variable contains about the other [START_REF] Cover | Elements of information theory[END_REF]. Recent valid associations depending on whether we have a vertical cluster for each scan.

LiDAR vertical cluster selection

We assume that all LiDAR scans are vertically oriented (this is the case for most modern scanners) and select the cluster with the smallest angle with the Z-axis (0, 0, 1) as being the vertical cluster.

Image vertical cluster selection

We assume that the images are upright (the real world vertical projects as a nearly vertical 2D vector in all images) and select the vertical cluster of the line cloud reconstructed from the images according to Algorithm 4.

Algorithm 4 Image vertical cluster

for each cluster i do for each 3D line j in cluster i do for each 2D image im associated with line j do Compute S im the verticality score weighted by the length of the 2D segment l i = [a i b i ] in the image im that has been used for the reconstruction of line j using the method proposed in Section 5.5.2 end for Compute av j : the average verticality score for all images associated with line j end for Compute AV i : the average verticality score for all 3D lines in cluster i end for Select the cluster which has the smallest score as the vertical cluster

Verticality score calculation

We have an image im which contains 2D segment

We compute S im the verticality score weighted by the length of the 2D segment as the angle between d i and y-axis (0, 1) according to the equation (5.1):

If the two segment sets S 1 and S 2 (image and/or LiDAR) can be vertically oriented (the assumptions above are true), we obviously associate the vertical cluster of S 1 to the vertical cluster of S 2 , and then associate any non vertical cluster of a S 1 with any non vertical cluster of S 2 . In the other case (at least one scan cannot be vertically oriented), we associate any pair of clusters of S 1 to any pair of clusters of S 2 . As for each cluster we have two possible direction vectors:{d(C), -d(C)}, we can:

In the two cases, for each cluster association

2 )}, we have sixteen possible forms. If one of these sixteen forms satisfies equation (5.2), we consider that this association is valid, we use the retained form to calculate the rotation. We reject the cluster association, if the equation (5.2) is not validated by any form.

Rotation estimation

Once the best form of the valid cluster association As = {(s

is selected, we want to find the rotation that best aligns the corresponding two pairs of directions. After selecting the directing vectors of two clusters. We start by creating orthonormal bases O i = (x i , y i , z i ), where:

We then compute the rotation R that aligns the associated clusters as the base change matrix between O 1 and O 2 :

(5.4)

Optimization of a global robust energy between two segments set

To carry out the registration of the two segment sets, we now have to find the rotation, the scale factor and the translation that minimize a global robust energy be-to (4.27).

So at each iteration, we select the set of segments pairs which validates the three criteria as the matching set.

Optimization step

At each iteration:

• We have estimated the rotation matrix that aligns the direction vectors of the matched 3D segments.

• We have used the equations of Section 5.5.4 in order to find the scale factor and the translation between the matched segments after having aligned them using the estimated rotation.

• We computed the energy (global robust energy) of the estimated transformation (rotation, scale factor, translation).

Finally, we keep the sampled transformation that has the minimum energy.

Rotation estimation

We have a set of matched lines. Each line L i has a direction vector d i . Following [START_REF] Alshawa | lCL: Iterative closest line a novel point cloud registration algorithm based on linear features[END_REF], the rotation matrix can be computed from line directions only by minimizing the following function:

(5.11)

The solution of this function is of closed form. For doing so, the two following means are defined:

(5.12)

The cross covariance matrix of the two data sets is:

The various experiments carried out have proven the robustness and speed of RANSAC

Evaluation on real data

Afterwards, we performed the registration of heterogeneous data: (terrestrial images/terrestrial LiDAR data) as shown in Figure 5.4 and (aerial images/aerial LiDAR data) as shown in Figure 5.5. As it is difficult to construct a ground truth for heterogeneous data registration, we evaluated the registration quality on the 3D visual results. The obtained results demonstrate our algorithm's ability to efficiently register image and LiDAR data. In addition, our algorithm has proven its robustness to small disturbance of verticality.

Achieving precise results requires fine-tuning of the algorithm's parameters. The distance threshold must be chosen reasonably so as not to consider a large number of lines as outliers. The number of iterations of RANSAC must be large enough to ensure the robustness of the algorithm.

Chapter 6

Aerial/Terrestrial registration

Introduction

The aim of this thesis is not only to register heterogeneous data (image and LiDAR) but also to register data acquired by heterogeneous acquisition platforms (aerial/terrestrial). By using airborne platforms in an urban environment, we can see the top faces of buildings, but the side faces are still missing. Unlike Terrestrial platforms which give complete and dense information from side faces and do not give significant information from top faces. So in order to achieve a complete coverage of urban areas, the fusion of both aerial and terrestrial views is necessary. Aerial data can be acquired in the form of images or LiDAR and it is the same case for terrestrial data.

So the problem of aerial/terrestrial registration can take several forms:

• Aerial image/Terrestrial image registration

Future work

The studied problems in this thesis, the proposed solutions and the obtained results open several perspectives:

• Openings detection:

We can allow more flexible shapes for openings, for instance by extracting corner points for each connected component and connecting them using parametric B-spline curves. We can also perform a semantic segmentation of the input point clouds to improve the accuracy and robustness of openings detection.

• Indoor/outdoor registration:

Our main perspective concerning this part is to extract outdoor points seen from the indoor scans in order to get additional information, if some are present.

• Image/LiDAR registration:

Our main perspective on this part is to use combinations of more segments to

Appendices Appendix A Implementation

All the developed algorithms in this work have been implemented in C++ using the following libraries:

• CGAL library https://www.cgal.org/: to carry out the geometric calculations.

• Boost library https://www.boost.org/: to calculate polygons intersection.

• Eigen library https://eigen.tuxfamily.org/dox/ for matrix calculation.

• PCL library https://pointclouds.org/ for point cloud processing.

• OpenCV library https://opencv.org/ for image processing

Appendix B

MLSD Improvement

B.1 Introduction

The traditional line segment detection algorithms (Canny edge, Hough transform) are slow and generate many false detections. To correct and overcome these problems, a new family of algorithms based on a contrario theory has been proposed. These algorithms have become the current state of art methods for line detection.

Using a contrario theory, the algorithms can automatically define whether a line is meaningful with a score called Number of False Alarms. The first proposed algorithms in this category are EDLine [START_REF] Akinlar | EDLines: A real-time line segment detector with a false detection control[END_REF] and LSD [START_REF] Gioi | LSD: a line segment detector[END_REF]. Later, a multi-scale extension of LSD (MLSD) was proposed in [START_REF] Salaün | Multiscale line segment detector for robust and accurate sfm[END_REF]. This appendix is devoted to the explanation of the general principle of LSD and MLSD as well as the illustration of our improvements applied to the two detectors in order to boost their performance. Before that, we introduce and explain the general principle of the a contrario theory.

B.2 A contrario method

According to [START_REF] Desolneux | When the a contrario approach becomes generative[END_REF], the a contrario method is a statistical approach based on hypothesis testing. This approach is used in order to detect geometric meaningful events in images. The basic idea consists in computing the probabil-model is an aligned point is:

As k(r, I) follows a binomial distribution, we can write the probability term as the following:

where B(n, k, p) is the tail of the binomial distribution:

The exhaustive choice is to take all the rectangles starting and ending at image pixels.

In the Gaussian kernel is fixed to 0.8 and the size to 7 × 7. As we know, the size of the kernel used to blur an image can have a significant effect on the result of the blurring and any downstream analysis carried out on the blurred image. In addition, larger σ values may remove more noise, but they will also remove details from an image.

A σ equal to 0.8 and a kernel size equal to 7 × 7 is not a good choice. So for better effect, we set σ to 1.6 to be able to remove more noise and preserve image details.

This choice of the σ value leads us to fix the kernel size at 11 × 11, because the kernel radius must be equal to 3σ + 1.

B.5 Multi-scale line segment detector (MLSD) [START_REF] Salaün | Multiscale line segment detector for robust and accurate sfm[END_REF] propose a multi-scale extension of LSD. This multi-scale nature makes it much less prone to over-segmentation, more robust to low contrast and less sensitive to noise, while keeping the parameter-less advantage of LSD and still being fast.

MLSD uses the same background model as generally used in a contrario methods.

The authors of MLSD have seen that the size of the picture matters a lot for segment detection. A smaller version of the same picture often yields fewer but proportionally longer segment detections, while the original picture may yield a lot of fragmented segments. They thus use a multi-scale approach to find long segments at coarse scales, refining their location at a finer scale. At each scale, They consider new segment candidates at the same locations than segments at the previous scale, possibly