Chapter 1 Introduction

Background

The first idea of intelligent vehicle was born in 1960s [START_REF] Broggi | Intelligent vehicles[END_REF]. However, the level of technique at that time didn't allow people to make that dream come true. But, during the past two decades, intelligent vehicle has achieved great growth with the improvement of sensor techniques. In 1990s, Bundeswehr University Munich tested a vehicle with a 1758 km trip from Munich to Copenhagen in Denmark and back. In 95% of the trip, the vehicle was in a autonomously running state. In the early of this century, the research in intelligent vehicle is gradually developed into the test of performance in realistic scenarios. The ideal of advanced driving assistance systems (ADAS) is proposed by many researchers. It means that intelligent vehicle is still controlled by the driver, but there exists a monitoring system that detect possible dangerous situations to provide warning to the driver or take in charge the vehicle in emergency case. In 2003, the Defense Advanced Research Projects Agency (DARPA) launched a race named by Grand Challenge for autonomous vehicles. All the participants are demanded to autonomously run more than 200 km in unstructured environments. This challenge attracted many top-level research institutes (See Fig. 1-1). Recently, several important achievements have taken in the intelligent vehicle community. In August of 2012, google company announced that its self-driving cars had completed over 300,000 miles with no accidents. And, in the same year, the states of Nevada, Florida and

Problem Statements and Objectives

Sensing the environment around a vehicle is a crucial ability for intelligent vehicle application. The most important work is to understand road scene. The commonly used sensors to scan the environment are 2D/3D Laser Ranger Finders (LRFs), RADARs, cameras (monocular, stereo vision, fisheye and omnidirectional) and GPS.

Both LRFs and RADARs are active sensors. Compared to optical sensors, they have the advantages of long distance detection ability, wide sensing range, and robust performance in evening, and in foggy or rainy day. However, they usually lack of high spatial resolution. Cameras are passive sensors. Cameras can provide high spatial resolution information and visual feature. In certain road scene such as road signs and traffic lights, cameras can work merely. GPS can be used for the drawing map and giving location information. To obtain robust and good performance, as described in [2], sensor fusion is commonly used by researcher for intelligent vehicle. In this thesis, we focus on the usage of LRF, camera (fisheye) and GPS.

Our research is part of the project CPER "Intelligence du Véhicule Terrestre", conducted within IRTES-SET Lab of UTBM. It aims to develop a multisensors system to robustly and precisely analyze and represent the road scene. The followings are the main objectives and contributions in our works:

1.The first objective is to fuse sensors data. Our perception system is composed of a LRF and a fisheye camera. Determining the position relation between between LRF and camera is an important work for further research. Extrinsic parameters calibration between LRF and fisheye camera refers to calculate the rigid transformation between their coordinate systems. In this work, a new calibration method is proposed. Meanwhile, three known fisheye model are tested and evaluated.

2.The second objective is to detect road in outdoor scenarios. Perceiving the presence of road is a crucial ability for intelligent vehicle. This task is mainly conducted by camera sensor because the equipped LRF is 2D. In this thesis, a road detection method based on illuminance invariant space and HSI space is proposed. This method mainly addresses over-saturated or under saturated issues that occur in cloudy days. 3.The third objective is to detect obstacles in front of running vehicle. Security issues for intelligent vehicle always can attract the attention from most of people. Detecting robustly obstacles around the vehicle can effectively prevent us from collisions.

We propose a new method based on LRF, GPS and camera for obstacle detection in the thesis. Our contribution consist in using features obtained from map to overcome the difficulty related to visual features which is invalid in motion blur case. 4.The final objective is to track objects. Tracking objects can give us prior knowledge regarding objects movements. This knowledge is a favor to prejudge the motion of objects in following time. We combine image space and LRF space to conduct object tracking. Our contribution consists in using weak visual features to perform tracking without losing objects in a long-term test.

Thesis Organization

The rest of the thesis is divided into five chapters: Chapter2: In this chapter, we present a method for extrinsic calibration between fisheye camera and LRF. It is mainly based on the LRF scan plane and several LRF measurements. The method is tested and evaluated by simulated data and tested by real data.

Chapter3: In this chapter, a camera and LRF based road detection approach is presented. It conducts a preliminary road detection in illuminance invariant image and then refines the results in HSI space.

Chapter4: In this chapter, we proposed a method based on GPS, LRF and camera for obstacle detection. It mainly copes with the problems caused by motion blur which stems from object and camera movements. The method is evaluated by the ground truth data.

Chapter5: In this chapter, we propose an approach for object tracking. It is mainly composed of two steps: the extraction of small region in image and the small region growth in LRF space.

Chapter6: In this chapter, conclusions and some research perspectives for this thesis are presented.

Chapter 2

Extrinsic calibration between fisheye camera and laser range finder

Introduction

This chapter deals with the issue of extrinsic calibration between fisheye camera and laser range finder. The aim of extrinsic calibration is to determine the rigid transformation between LRF and fisheye camera. In the field of intelligent vehicle, the road environment is often perceived by various sensors, such as video cameras, laser rang finder. However, each sensor has some weaknesses. In a complex traffic environment, using only a single sensor to perceive and analyze the environment could limit the accuracy and robustness. Recently, LRF and camera mounted together on a car have become very common for perceiving the environment. On the one hand, LRF measure distances between itself and the detected objects with high accuracy within wide-area view but with low resolution. On the other hand, camera provides visual information around itself. These visual information provide many important clues for applications such as object recognition, obstacle detection. However, a conventional camera suffers from narrow field of view. Multiple conventional cameras can be employed to expand the perception area, but it will make extra expense and cause some real-time processing problems when the vehicle is traveling with high speed. Compared with conventional camera, camera with fisheye lens is then an attractive choice in many applications, as it provides a large field of view (FOV) with a single sensor. Due to these reasons, fisheye camera and 2D LRF are combined in our research. The used experimental platform is shown in Fig. 2-1. Before performing road scene understanding, the rigid transformation between LRF and fisheye camera has to be known.

The rest of this chapter is organized as follows. Section 2.2 introduces the state of the art. Section 2.3 describes three different kinds of fisheye camera models. Section 2.4 presents LRF plane based extrinsic calibration between LRF and fisheye camera.

State of the Art

Most of state of the art works about calibration between LRF and camera fall into two categories depending on the type of LRF: 2D LRF or 3D LRF. Paper [3] suggests a new method for calibration between 3D LRF and a stereo camera. The considered 3D LRF is built by moving a 2D LRF along one of its axes. By rotating the 2D scanner around its radial axes , it is possible to obtain the spherical coordinates of the points measured. The method firstly computes the three transformation matrices from LRF reference frame to stereo camera reference frame when the LRF is placed in three different specified orientations around its rotation axis. Then, based on the obtained three transformation matrices, the rigid transformation matrix between the LRF and the center of rotation of LRF is determined. Finally, the transformation matrix between LRF in any arbitrary angle and the stereo-camera is computed. This method needs to put pattern into several different poses. In paper [4], the authors propose a self calibration method between a 3D LRF and an omnidirectional camera.

Based on the 3D LRF measurements, a depth image named bearing angle (BA) image is constructed. Several corresponding points between BA image and intensity image are selected manually. With these points, the extrinsic parameters between the two sensors frames are estimated. However, the accuracy of this method is depending on the resolution of the 3D LRF. High resolution of 3D LRF often means a high cost.

In paper [5], a visible LRF is used to estimate transformation between both LRF and camera. However, in many applications, laser spots emitted by LRF are often invisible for camera.

Paper [START_REF] Zhang | Extrinsic calibration of a camera and laser range finder (improves camera calibration)[END_REF] introduces a method based on distance constraints from camera to 2D LRF system. A chessboard is placed in the common field of view (FOV) of the two sensors. A set of geometrical constraints on rigid transformation between LRF and camera can be defined by changing the chessboard location in the common FOV.

Based on this set of geometrical constraints, the extrinsic parameters can be determined. This method needs to change the chessboard location and is affected by the orientation between the chessboard and LRF. In paper [START_REF] Mei | Calibration between a central catadioptric camera and a laser range finder for robotic applications[END_REF], the authors provide an approach based on 2D LRF with both visible and invisible trace. This method is an extended version of the approach in paper [START_REF] Zhang | Extrinsic calibration of a camera and laser range finder (improves camera calibration)[END_REF]. It also needs to put a chessboard placed in the FOV of the two sensors. The constraint conditions are not based on the laser points directly but the straight lines consisting of them. In paper [START_REF] Brim | Modeling and calibration of coupled fish-eye CCD camera and laser range scanner for outdoor environment reconstruction[END_REF], the authors present an approach for the calibration between a fisheye camera and laser range finder. The main idea is to justly use regular lens to get extrinsic parameters and then to replace it with fisheye lens. In paper [START_REF] Naroditsky | Automatic alignment of a camera with a line scan LIDAR system[END_REF], the authors propose a minimal approach to determine the extrinsic parameters between a 2D laser scanner and a camera, using only six measurements of a planar calibration board.

The work in this chapter also focuses on extrinsic calibration between 2D LRF and fisheye camera. There are two contributions in this research. The first one is the evaluation and comparison between three different models that have been proposed to fisheye camera modeling. The second one is the proposition of a novel approach of extrinsic calibration between the two sensors. The proposed method requires LRF and camera to observe a chessboard moved in their common field of view. By analyzing successive LRF measurements, a set of points located in the laser beams plane can be detected. These detected points are then used to estimate the equation of the plane of the laser beams in the camera coordinate system. Finally, two geometrical constraints based on the equation of this plane and this set of points are constructed to estimate the extrinsic parameters between the fisheye camera and the LRF. The performance of the approach is evaluated through experiments on both simulated and real data.

Fisheye Camera Modeling

Camera with fisheye lens provides a wide angle vision, but cause great distortions in the image. It makes then the conventional camera model invalid. Up to now, there is no unified projection model for fisheye camera. In this chapter, three models proposed by Scaramuzza [10], Kannala [START_REF] Kannala | A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses[END_REF] and Mei [START_REF] Mei | Single view point omnidirectional camera calibration from planar grids[END_REF], are studied.

Scaramuzza's representation for fisheye camera

The fisheye model proposed in paper [10] is illustrated in Fig. 2-2. It consists of fisheye lens surface, image plane and sensor plane. The sensor plane is a hypothetical plane orthogonal to the mirror axis, with the origin lying on the plane-axis intersection. In practice, the sensor plane corresponds to the camera CCD plane, where the pixels are stated in physical size. The image plane corresponds to the image, where the pixels are expressed in pixel coordinates. The image point 𝑀 corresponding to the scene point 𝑃 0 is produced by three steps. Firstly, 𝑃 0 is mapped onto fisheye lens surface the vector corresponding to the image point M , so we have:

P s = f (m) =   m g(||m||)   =      u v g(||m||)      (2.1)
where g(||m||) represents the corresponding point of M on fisheye lens surface, ||m|| is the module of point m on sensor plane. In paper [10], the authors consider that the function g represent the curve line of fisheye lens surface and obeys the following polynomial form:

g(||m||) = a 0 ||m|| 1 + a 1 ||m|| 2 + • • • + a N ||m|| N (2.2)
where a i , i = 0, 1, 2

m =   u v   = A   u v   + T = AM + T (2.3)
where A ∈ R 2×2 and T ∈ R 2 . A is a stretch matrix and T is a translation vector.

Based on g(||m||), A and T, the complete model of fish-eye camera is represented as follows:

λ • Ps = λ • f (m) = λ • f (AM + T) = λ •   AM + T g(||AM + T ||)   , λ > 0 (2.4)
Fisheye camera calibration corresponds to the estimation of A, T and

a i , i = 0, 1, 2, • • • N .
Although these intrinsic parameters are not determined directly, they can be estimated with the help of a calibration pattern. There are three steps for estimating these intrinsic parameters. Firstly, assuming A = I (unit matrix) and T = 0, the extrinsic parameters between the calibration pattern and the camera are estimated. Secondly, the coefficients of the function g are calculated by incorporating several observations of the calibration pattern. Finally, non-linear refinement is used for all parameters to obtain more accurate results. For more details, the reader can refer to paper [10].

Kannala's representation for fisheye camera

In paper [START_REF] Kannala | A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses[END_REF], the authors propose to divide the full fisheye camera projection model into two parts: the radially symmetric part and the asymmetric part.

Radially symmetric part

The radially symmetric part of fisheye camera projection model is illustrated in Fig. 234. 𝑃 is a scene point, 𝑝 is the image of 𝑃 in fisheye image, 𝑝 ′ is the image of 𝑃 in pinhole Figure 2-4: Fisheye camera projection model in paper [START_REF] Kannala | A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses[END_REF]. The image of the point P is p whereas it would be p' by a pinhole camera image, 𝜃 is the angle between principal axis and the incoming ray, 𝑟 is the distance between the fisheye image point 𝑝 and the principal point and f is the focal length. As illustrated in Fig. 234, the radially symmetric part of fisheye camera projection model is different from the pinhole camera model. In Generally, it is designed to obey one of the following projections:

𝑟 =2𝑓 𝑡𝑎𝑛(𝜃/2) (𝑠𝑡𝑒𝑟𝑒𝑜𝑔𝑟𝑎𝑝ℎ𝑖𝑐 𝑝𝑟𝑜𝑗𝑒𝑐𝑡𝑖𝑜𝑛) 𝑟 =𝑓 𝜃 (𝑒𝑞𝑢𝑖𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑝𝑟𝑜𝑗𝑒𝑐𝑡𝑖𝑜𝑛) 𝑟 =2𝑓 𝑠𝑖𝑛(𝜃/2) (𝑒𝑞𝑢𝑖𝑠𝑜𝑙𝑖𝑑 𝑎𝑛𝑔𝑙𝑒 𝑝𝑟𝑜𝑗𝑒𝑐𝑡𝑖𝑜𝑛) 𝑟 =𝑓 𝑠𝑖𝑛(𝜃) (𝑜𝑟𝑡ℎ𝑜𝑔𝑜𝑛 𝑎𝑙𝑎𝑛𝑔𝑙𝑒 𝑝𝑟𝑜𝑗𝑒𝑐𝑡𝑖𝑜𝑛) (2.5)
To facilitate the calibration work, the authors of paper [START_REF] Kannala | A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses[END_REF] propose to use a general model suitable for different types of lens. The proposed general form is:

𝑟(𝜃) = 𝑘 1 𝜃 + 𝑘 2 𝜃 3 + 𝑘 3 𝜃 5 + 𝑘 4 𝜃 7 + 𝑘 5 𝜃 9 + • • • (2.6)
where 𝑘 1 , 𝑘 2 , 𝑘 3 , 𝑘 4 , 𝑘

Asymmetric part

In practice, real lens may deviate from precise radial symmetry and therefore an asymmetric part is added to obtain a full projection model. For wide application, the authors propose a flexible mathematical distortion model to represent the asymmetric part. This flexible distortion model consists of two terms. One term acts in the radial direction and has the following form:

∆ 𝑟 (𝜃, 𝜙) = (𝑙 1 𝜃 + 𝑙 2 𝜃 3 + 𝑙 3 𝜃 5 )(𝑖 1 𝑐𝑜𝑠𝜙 + 𝑖 2 𝑠𝑖𝑛𝜙 + 𝑖 3 𝑐𝑜𝑠2𝜙 + 𝑖 4 𝑠𝑖𝑛2𝜙) (2.7)
and the other term is concerned with the tangential direction:

∆ 𝑡 (𝜃, 𝜙) = (𝑚 1 𝜃 + 𝑚 2 𝜃 3 + 𝑚 3 𝜃 5 )(𝑗 1 𝑐𝑜𝑠𝜙 + 𝑗 2 𝑠𝑖𝑛𝜙 + 𝑗 3 𝑐𝑜𝑠2𝜙 + 𝑗 4 𝑠𝑖𝑛2𝜙) (2.8)

Full model

The full fisheye camera projection model is made up of radially symmetric part and asymmetric part. By combing 2.8, 2.7 and 2.6, an image point X d = (𝑥 𝑑 , 𝑦 𝑑 ) in millimeter coordinate system of an image point can be represented as follows:

X d = 𝑟(𝜃)e r (𝜙) + ∆ 𝑟 (𝜃, 𝜙)e r (𝜙) + ∆ 𝑡 (𝜃, 𝜙)e 𝜙 (𝜙) (2.9)
where e r and e 𝜙 are the unit vectors in radial and tangential directions. To get the full projection model, millimeter coordinate system is needed for transformation into image pixel coordinate system. Let (𝑢, 𝑣) be the pixel coordinate, (𝑢 0 , 𝑣 0 ) is the principal point, and 𝑛 𝑢 and 𝑛 𝑣 represent the number of pixels unit distance in horizontal and vertical directions respectively. Finally, the full model is written as follows:

⎛ ⎝ 𝑢 𝑣 ⎞ ⎠ = ⎡ ⎣ 𝑛 𝑢 0 0 𝑛 𝑣 ⎤ ⎦ ⎛ ⎝ 𝑥 𝑑 𝑦 𝑑 ⎞ ⎠ + ⎛ ⎝ 𝑢 0 𝑣 0 ⎞ ⎠ (2.10)
This full model is defined by 23 parameters. To facilitate computation, the asymmetric part is often ignored. In this case, the full model has only 9 parameters and is denoted by 𝑝 9 in the following. To solve these parameters, several observation points on a calibration pattern are used. For more details, reader can refer to paper [START_REF] Kannala | A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses[END_REF].

Mei's representation for fisheye camera

In paper [START_REF] Mei | Single view point omnidirectional camera calibration from planar grids[END_REF], the authors use an unified projection model representing omnidirectional camera to approximate fisheye camera. It is based on the model proposed by Geyer and Barreto in papers [START_REF] Geyer | A unifying theory for central panoramic systems and practical implications[END_REF] and [START_REF] Barreto | Issues on the geometry of central catadioptric image formation[END_REF], and is illustrated in Fig. 2-5. This model contains three parts: a unit sphere, a sensor plane and an image plane. Let 𝐹 𝑚 represents the coordinate system centered at 𝐶 𝑚 (centre of the sphere) and 𝐹 𝑝 is the coordinate system centered at 𝐶 𝑝 .

A world point 𝜒 𝐹𝑚 = (𝑋, 𝑌, 𝑍) is projected onto the image plane p using the Figure 2-5: Unified projection model in paper [START_REF] Mei | Single view point omnidirectional camera calibration from planar grids[END_REF] following steps:

1. The world point χ is projected onto the unit sphere to obtain the point χ s ,

(χ) Fm -→ (χ s ) Fm = χ ||χ|| = (X s , Y s , Z s ) 2. χ s Fm is then converted to F p system C p = (0, 0, ξ), (χ s ) Fm -→ (χ s ) Fp = (X s , Y s , Z s + ξ)
. ξ is a parameter depending on the mirror type located at C m 3. Then, the point (χ s ) is projected onto the sensor plane

π mu point χ d , (χ s ) Fp -→ (χ d ) Fp = ( Xs Zs+ξ , Ys Zs+ξ , 1)
4. The final projection involves the camera intrinsic matrix K

𝜒 d -→ p = K𝜒 d = ⎡ ⎢ ⎢ ⎢ ⎣ 𝑎 𝑥 𝑎 𝑥 𝛼 𝑢 0 0 𝑎 𝑦 𝑣 0 0 0 1 ⎤ ⎥ ⎥ ⎥ ⎦ 𝜒 d (2.11)
where (𝑢 0 , 𝑣 0 ) the principal point, 𝛼 the skew, 𝑎 𝑥 and 𝑎 𝑦 the focal length of the camera in terms of pixel dimensions in the x and y direction respectively.

Approximation of fisheye lens

The great distinction between fisheye image and classic image is that there is great distortion along radial direction in fisheye image. In paper [START_REF] Fitzgibbon | Simultaneous linear estimation of multiple view geometry and lens distortion[END_REF], the authors show that the great radial distortion in image can be approximated as a division model. In paper [START_REF] Ying | Can we consider central catadioptric cameras and fisheye cameras within a unified imaging model[END_REF], the fisheye image can be considered as a conventional image but with great radial

𝑝 𝑢 = 𝑘 1 𝑝 𝑑 1 -𝑘 2 𝑝 2 𝑑 ;
(2.12)

where 𝑘 1 and 𝑘 2 are two scalar parameters depending on the fisheye lens type. Given the focal length 𝑓 = 1, the projected point 𝜒 of a word point on sensor plane under pinhole model can be written as:

𝜒 = (𝑥 𝜒 , 𝑦 𝜒 , 1) = (𝑋/𝑍, 𝑌 /𝑍, 1) (2.13)
where (𝑥, 𝑦, 1) are the homogeneous coordinates of the word point on the sensor plane.

In C.Mei's representation, with 𝜉 = 1, the projected point 𝜒 𝑑 of this word point is:

𝜒 d = (𝑥 𝜒 𝑑 , 𝑦 𝜒 𝑑 , 1) = ( 𝑋 𝑠 𝑍 𝑠 + 1 , 𝑌 𝑠 𝑍 𝑠 + 1 , 1) = ( 𝑋 𝑍 + ||𝜒|| , 𝑌 𝑍 + ||𝜒|| , 1) (2.14)
By algebraic manipulation, we obtain the following relation from equations 2.13 and 2.14:

𝑝 𝜒 = 2𝑝 𝜒 𝑑 1 -𝑝 2 𝜒 𝑑 (2.15)
where 2.15 has the same form as equation 2.12. (The influence of the mentioned three models on the calibration will be discussed in experiment section of this chapter.)

𝑝 𝜒 = √ 𝑥 2 𝜒 + 𝑦 2 𝜒 and 𝑝 𝜒 𝑑 = √ 𝑥 2 𝜒 𝑑 + 𝑦 2 𝜒 𝑑 . Equation

Proposed Method for Extrinsic Calibration Between Fisheye Camera and LRF

The objective of extrinsic calibration between fisheye camera and LRF is to determine the geometric transformation between the two sensor systems.

As illustrated in Fig. 23456, given a point 𝑃 , this transformation is expressed by:

𝑃 𝑐 = 𝑅 * 𝑃 𝐿 + 𝑇 (2.16)
where 𝑃 𝑐 represents the coordinates of 𝑃 in the camera system, 𝑃 𝐿 represents the coordinates of 𝑃 in the LRF system, 𝑅 and 𝑇 are the rotation matrix and translation vector between the camera system and LRF system respectively. The elements of R and T have to be estimated by calibration.

The framework of the proposed approach is shown in Fig. 234567. Firstly, several known points (defined in the following section) are determined (section 2.4.1).

Then, the normal vector of laser plane is estimated (section 2.4.2). Two geometrical

Known points estimation

A known point in our method is a point for which its coordinates in LRF coordinate systems and camera coordinate systems are determined. In paper [3], a laser pointer is used to determine the known points for a system of a stereo camera and LRF.

Spurring by this idea, a method based on the corner point of a chessboard is proposed to find out known points for our system composed of a fisheye camera and LRF. The procedure is illustrated in Fig. 2345678. A chessboard is moved down slowly from a position above the plane of the laser beams in the field of view of the LRF. During the way down, the LRF measurements are checked continuously. When the chessboard is above the laser beams plane, the LRF measurements remain almost constant can be considered as the background measurements, as we consider that there is no other moving object in the scene. When the chessboard intersects with the laser beams So when the first difference between the LRF measurements and the background measurements is detected during the way down, the corner point (red point in Fig. 2345678) of the chessboard is believed to be intersecting with the laser beams plane, that is, being on the laser beams plane. The corner point at this position can then be considered as a known point. Its coordinates in LRF system can obtained directly from LRF measurements and the coordinate in camera system can be calculated using the calibration toolbox introduced in [10]. This procedure is repeated until several known points are obtained.

Normal vector of laser plane estimation

It is known that three points which don't stand on a line can define a plane uniquely in space. To find out the coordinates of normal vector of laser plane in camera coordinate system, it is then necessary to know the coordinates of three points at least on laser plane in camera coordinate system. Fortunately, based on the method described in the previous step, the obtained known points are the proper ones for estimating the coordinate of normal vector of laser plane in camera coordinate system. However, there may exist several outliers in obtained known points. These outliers may cause error estimations when using linear regression method to calculate the normal vector of laser plane. So, the first job to us is to remove the outliers before solving laser plane.

Random sample and consensus

RANSAC (Random Sample and Consensus) algorithm was first introduced by [START_REF] Fischler | Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography[END_REF].

It's a method to estimate parameter for a specific model with several outliers in input data. In paper [START_REF] Zuliani | The multiransac algorithm and its application to detect planar homographies[END_REF], a datum is considered to be an outlier if it will not fit the "true" model instantiated by the "true" set of parameters within some error threshold that defines the maximum deviation attributable to the effect of noise. Generally, despite many improvement versions, the RANSAC algorithm is essentially composed of two steps that are repeated in an iterative fashion:

1) Hypothesize: First minimal sample sets (MSSs) are randomly selected from the input dataset and the model parameters are computed using only the elements of the MSSs.

2) Test: In the second step, RANSAC checks which elements of the entire dataset are consistent with the model instantiated with the parameters estimated in the first step. The set of such elements is called consensus set (CS). RANSAC terminates when the probability of finding a better ranked CS drops below a certain threshold.

In our method, RANSAC is used to eliminate the outliers from known points set.

Let denote x 𝑖 = (𝑥 𝑖 1 , 𝑥 𝑖 2 , 𝑥 𝑖 3 ) 𝑇 , (𝑖 = 1, 2, ...𝑁 ) the i-th point set on plane. The used mathematical model is as follows:

⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ 𝜏 1 𝑥 1 1 + 𝜏 2 𝑥 1 2 + 𝜏 3 𝑥 1 3 + 𝜏 4 = 0 . . . . . . 𝜏 1 𝑥 𝑁 1 + 𝜏 2 𝑥 𝑁 2 + 𝜏 3 𝑥 𝑁 3 + 𝜏 4 = 0 (2.17)
where 𝜏 1 , 𝜏 2 , 𝜏 3 , 𝜏 4 are the model parameters. To group these equations in matrix form, we can get:

⎡ ⎢ ⎢ ⎢ ⎣ (x 1 ) 𝑇 1 . . . . . . (x N ) 𝑇 1 ⎤ ⎥ ⎥ ⎥ ⎦ 𝜏 = X𝜏 = 0 (2.18)
So the estimation of the parameters vector which instantiates the plane can be transformed as:

𝜏 * = 𝑎𝑟𝑔𝑚𝑖𝑛 ||𝜏 =1|| ||X𝜏 || 2 (2.19)
For each data point x, with above equation, the fitting algebraic error is defined as follows:

𝑒 x = ([x 𝑇 1]𝜏 * ) 2 ||𝜏 * 1:3 || (2.20)
After obtaining of least three "true" known points, a least-square regression based approach is used to estimate the normal vector of laser plane in camera coordinate systems.

Multiple linear regression

Regression analysis is one of useful statistical tools for analyzing multifactor data. Its broad appeal and usefulness results from the conceptually logical process of using an equation to reveal the inherent relationship between variables. Similarly, it also can be regard as an interesting theory duo to elegant underlying mathematics and a well developed statistical theory. Successful use of regression requires an appreciation of both the theory and the practical problems that typically arise when the technique is employed with real-world data [START_REF] Montgomery | Introduction to linear regression analysis[END_REF].

The simplest regression model is unary linear model, that is, a model with a single regressor x that has a relationship with a response y that is a straight line. Multiple linear regression which involves more than one regressor variable is the extension of the unary linear model. Plane normal vector estimation belongs to this case.

However, the linear regression is not robust to noise. This is the reason why we have to use RANSAC algorithm to remove outliers firstly. In camera coordinate system, the plane equation can be expressed as follows:

𝑁 𝑥 (𝑋 𝑖 -𝑋 0 ) + 𝑁 𝑦 (𝑌 𝑖 -𝑌 0 ) + 𝑁 𝑧 (𝑍 𝑖 -𝑍 0 ) = 0 (2.
⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎣ 𝑁 𝑥 𝑁 𝑦 𝑁 𝑧 ⎤ ⎥ ⎥ ⎥ ⎦ = 𝜆 𝑣 ⎡ ⎢ ⎢ ⎢ ⎣ 𝑁 𝑥 𝑁 𝑦 𝑁 𝑧 ⎤ ⎥ ⎥ ⎥ ⎦ (2.27)
Given the assumption

||𝑁 𝑣 || = 1, that is the inner product (𝑁 𝑣 , 𝑁 𝑣 ) = 1. Let 𝐸 = ⎡ ⎢ ⎢ ⎢ ⎣ ∑ ∆𝑋 𝑖 ∆𝑋 𝑖 ∑ ∆𝑋 𝑖 ∆𝑌 𝑖 ∑ ∆𝑋 𝑖 ∆𝑍 𝑖 ∑ ∆𝑋 𝑖 ∆𝑌 𝑖 ∑ ∆𝑌 𝑖 ∆𝑌 𝑖 ∑ ∆𝑌 𝑖 ∆𝑍 𝑖 ∑ ∆𝑋 𝑖 ∆𝑍 𝑖 ∑ ∆𝑌 𝑖 ∆𝑍 𝑖 ∑ ∆𝑍 𝑖 ∆𝑍 𝑖 ⎤ ⎥ ⎥ ⎥ ⎦ (2.28)
We have:

EN v = λ v N v ⇒(E -λ v )N v = 0 (2.29) (2.30)
N v is the eigenvector corresponding to the minimum eigenvalue of the following equation:

|(E -λ v )I| = 0 (2.31)
where I is identity matrix. The geometrical constraints reveal the LRF measurements relationship equations in 3D space. Based on previously obtained the known points set and normal vector, two geometrical constraints can be constructed. For one single-planar LRF, it is known that all laser beams lay on one plane. This characteristic is used to build the first constraint condition. Let 𝑆 𝑙 denote the set of LRF measurements, the first constraint is:

Geometrical constraints construction

𝑁 𝑣 (𝑅 * 𝑃 𝑗 𝑙 + 𝑇 -𝑃 𝑖 𝑘𝑐 ) = 0 (𝑃 𝑗 𝑙 ∈ 𝑆 𝑙 ) (2.32)
where 𝑃 𝑗 𝑙 is the coordinates of the j-th laser point of 𝑆 𝑙 in LRF coordinate system, 𝑃 𝑖 𝑘𝑐 is the coordinates of the i-th known point in fisheye camera system. The geometrical interpretation is illustrated in Fig. 23456789. In camera coordinate system, the vectors belonging to the laser beams plane are perpendicular to 𝑁 𝑣 .

The second constraint is based on the known points. From the previous statements, the coordinates of the "known points" are known in the two sensor coordinate systems. Let 𝑃 𝑖 𝑘𝑙 denote the coordinates of the i-th known point in LRF coordinate system. By taking 𝑃 𝑖 𝑘𝑙 and 𝑃 𝑖 𝑘𝑐 into equation 2.16, the second constraint is given as follows:

𝑅 * 𝑃 𝑖 𝑘𝑙 + 𝑇 -𝑃 𝑖 𝑘𝑐 = 0 (2.33)
To summarize the above description, the two geometrical constraints are:

⎧ ⎨ ⎩ 𝑁 𝑣 (𝑅 * 𝑃 𝑗 𝑙 + 𝑇 -𝑃 𝑖 𝑘𝑐 ) = 0 𝑅 * 𝑃 𝑖 𝑘𝑙 + 𝑇 -𝑃 𝑖 𝑘𝑐 = 0 (2.34)
To solve this equation, the following equations are proposed:

𝑀 𝑖𝑛 ⎧ ⎨ ⎩ 𝑊 1 ∑ 𝑁 𝑖=1 ∑ 𝑀𝑝 𝑗=1 𝑁 𝑣 (𝑅 * 𝑃 𝑗 𝑙 + 𝑇 -𝑃 𝑖 𝑘𝑐 ) = 0 𝑊 2 ∑ 𝑁𝑝 𝑖=1 𝑅 * 𝑃 𝑖 𝑘𝑙 + 𝑇 -𝑃 𝑖 𝑘𝑐 = 0 (2.35)
where 𝑀 𝑝 and 𝑁 𝑝 are the number of laser points and known points respectively.

Levenberg-Marquardt [START_REF] Ranganathan | The levenberg-marquardt algorithm[END_REF] algorithm is applied to find the optimal solution to equation 2.35. However, in practice, the solution of 𝑅 and 𝑇 may not make the two equations to reach their minimum values simultaneously. For example, if 𝑀 𝑝 is far greater than 𝑁 𝑝 , the minimization process will mainly focus on the first equation. To make a balance between the two constraints, two parameters 𝑊 1 and 𝑊 2 are used to adjust the influence of 𝑀 𝑝 and 𝑁 𝑝 on the minimization process.

Experiments

This section describes results obtained with simulated and real data. The program is implemented in Matlab. With real data experiment, it is hard to give performance estimation due to lack of ground-truth data. Therefore, simulated data are used firstly to estimate the performance of our method.

Simulation tests

The relative position and orientation of the LRF and fisheye camera are randomly set to [10; 300; 200](𝑚𝑚) and [75 𝑜 , -5 𝑜 , 5 𝑜 ] respectively. The coordinates of the known points are calculated based on relative pose of the camera with respect to the LRF.

Influence of the known points number in ideal case (without noise) Influence of known points number in noisy case

Comparison of calibration results

This experiment evaluates how the noise on laser data affects the performance and compares these results with the result obtained using the approach proposed in paper [START_REF] Mei | Calibration between a central catadioptric camera and a laser range finder for robotic applications[END_REF]. Gaussian noise with zero mean and standard deviation (from 1 cm to 10 cm) is added to all laser points. For each noise level, 150 independent random trials are

The reason

The average calibration error is shown in Fig. 2-12. The proposed approach is better than the approach in paper [START_REF] Mei | Calibration between a central catadioptric camera and a laser range finder for robotic applications[END_REF] with respect to the estimation of the rotation matrix. For the translation matrix, the two approaches get almost the same results

for noise level from 1cm to 5cm. With noise level over 5 cm, the proposed method performs better.

Real data experiments

In real experiments, the used fisheye lens is a Fujinon FE185C057HA1 which provides up to 185 degrees wide angle. The used camera is a pixellink PL-B742 with 1.3 megapixels (1280x1024). The used mono-layer LRF is a LMS221 with 1 degree resolution, 180 degrees field of view and up to 80m measurement range. The used computer is a normal laptop with intel core i5. All devices are mounted on the front of the vehicle (as already illustrated in fig. 2-1). For the last trial, the three approaches are compared with respect to the average reprojection errors. The results are shown in Table 2

Analysis of fisheye model choice

Extrinsic calibration between LRF and fisheye camera

In this experiment, real data are used to evaluate the performance of the proposed approach. As explained in section 2.4, the method requires to determine the coordinates of at least three points (known points) on the laser beams plane in camera coordinate system. In order to locate these points conveniently, two rectangle boards are used. Firstly, one is fixed in the view of the LRF and camera. And, another one is drew close to the fixed board until there is only one laser beam to pass through the gap between the two boards. Finally, the chessboard is slowly moved down along with the gap.

Comparison of calibration results

In this experiment, we compare our approach with the method proposed in paper [START_REF] Mei | Calibration between a central catadioptric camera and a laser range finder for robotic applications[END_REF].

Application in ICP algorithm

In order to illustrate the interest of extrinsic calibration between LRF and fisheye camera, we use the calibration result to integrate both LRF measurements and color information in ICP algorithm (Iterative Closet Points). ICP algorithm was introduced in the early 1900s [START_REF] Chen | Object modelling by registration of multiple range images[END_REF] and was further developed by various researchers. The most cited version is the one proposed in [START_REF] Besl | Method for registration of 3-d shapes[END_REF]. Since it was born, ICP algorithm is widely used in many research areas, especially in geometric alignment of 3D models.

Basic ICP algorithm

Given two roughly aligned shaped represented by point clouds, the ICP algorithm will implement the following tasks:

1) Generate temporary correspondences from the two clouds of points.

2) Estimate the relative rigid body transformation between the two clouds.

The first step is the key factor to the final estimation for rigid body transformation. In order to get a relatively good correspondence for a point, the ICP algorithm iteratively perform the following steps:

◇ Matching: the nearest neighbor of each data point in the points clouds is found.

◇ Minimization: the error metric for whole data set is minimized.

◇ Transformation: data points are transformed using minimization result.

The algorithm is terminated based on the number of iterations or the relative change in the error metric.

Closet point with color information

Generally, ICP algorithm converges very quickly, however several problems may occur:

1) Local minima: Instead of the global minimum, the algorithm may converge towards one of multiple local minima in the error metric

2) Noise and outliers: Outliers and noise may play a great side effect on the minimization process of the error metric, which leads to faulty results.

3) Partial overlap: the point clouds may lose partial information regarding the object due to the partial overlap.

To solve these problems, many variants have been introduced based on the basic ICP concept. In a review paper [START_REF] Rusinkiewicz | Efficient variants of the icp algorithm[END_REF], the authors classified the proposed variants of the algorithm as affecting one of the six subtasks:

◇ Selection (Choosing subsets of input point sets).

◇ Matching.

◇ Weighting (correspondences).

◇ Outlier removal.

◇ Error metric.

◇ Minimization.

Our work belongs to matching part. In the original ICP algorithm, the euclidean distance was used as the criterion to determine the closet point. Let 𝑆 𝑣 and 𝑆 ′ 𝑣 denote the two points sets observed by a sensor from two different viewpoints. The problem to find the closet point of the point 𝑑 𝑖 in 𝑆 𝑣 can be converted to the following question:

𝑎𝑟𝑔 min

𝑑 𝑖 ∈𝑆𝑣,𝑑 ′ 𝑖 ∈𝑆 ′ 𝑣 ‖𝑑 𝑖 -𝑑 ′ 𝑖 ‖ (2.36)
where 𝑑 ′ 𝑖 is a point in 𝑆 ′ 𝑣 . Euclidean distance is useful in many cases. However, it may become invalid if the partial overlap case happens. To handle this situation, many additional features are proposed. The color information is one of them. To obtain the corresponding color value, the extrinsic parameters between vision sensor and range sensor are needed. As done in paper [START_REF] Joung | 3d environment reconstruction using modified color ICP algorithm by fusion of a camera and a 3d laser range finder[END_REF], the modified version of euclidean distance is:

𝑎𝑟𝑔 min

𝑑 𝑖 ∈𝑆𝑣,𝑑 ′ 𝑖 ∈𝑆 ′ 𝑣 ‖𝑑 𝑖 -𝑑 ′ 𝑖 ‖ + √ 𝛼 1 (𝑎 1 -𝑎 2 ) 2 + 𝛼 2 (𝑏 1 -𝑏 2 ) 2 + 𝛼 3 (𝑐 1 -𝑐 2 ) 2 (2.37)
where 𝛼 1 , 𝛼 2 , 𝛼 3 are weight coefficients and 𝑎 𝑥 , 𝑏 𝑥 , 𝑐 𝑥 (x=1,2,3) are respectively R,G and B values in RGB model.

Real data experiment

Conclusion and Future Works

In this chapter, we firstly give a brief review about the state of the art of extrinsic calibration between LRF and camera. Extrinsic calibration is used to determine the rigid transformation between LRF and fisheye camera. It can help researcher to overcome a single vision sensor drawbacks for perceiving the environment. Three sorts of fisheye model are introduced. Unlike classic vision camera, fisheye camera provides a wide angle vision but delivers images with great distortions. It makes the traditional pinhole model invalid. However, a unified fisheye model doesn't exist. To find out a relatively good fisheye model is important to the extrinsic calibration work. After that, we introduce our extrinsic calibration method in detail. Generally speaking, it is based on the laser scanning plane equation and some known points on it. In the experiment section, we used simulated and real data to show the effectiveness of our method. At last, an interesting application of extrinsic calibration is presented.

The drawback of our method is that it takes some time to determine the known points. As future works, to improve the accuracy and to save user time, the following works could be attempted:

1) Infrared detection card (IR card) could be used to save the time required to locate the laser beams. IR card emits clearly visible light when illuminated by laser diode. This allows to locate easily.

2) Some special plane information could be considered to improve the accuracy. In the proposed method, only laser beam plane is employed to construct the geometrical constraints. However, in fact, some other special planes can be explored to develop the geometrical constrains (e.g., the plane formed by the view point and the LRF scanning line on the pattern).

Chapter 3

Road Detection Based on Fisheye

Camera and Laser Range Finder

Introduction

For autonomous vehicles and Advanced Driver Assistance Systems (ADAS), an important task is to keep the vehicle traveling in a safe region and prevent collisions. To meet that requirement, the vehicle has to perceive the structure of the environment around itself. The free road surface ahead of the vehicle has then to be detected.

In addition, a robust effective road detection system also plays an important role in higher other tasks such as vehicle and pedestrian detection (see chapter 4). The derived free road space can indeed provide a significant contextual information to reduce the region-of-interest for searching targets (cars, pedestrians,...), which contributes to reach a reasonable computational cost and to remove false detections.

In our work, we aim at performing road detection using a monocular camera with fisheye lens and a 2D LRF. An example of expected result is illustrated in Fig. 3-1.

Compared to classic lens, fisheye lens has greater FOV providing more information about the scene. But the disadvantage is the great distortion appearing in the images. Therefore, we propose to use the color space as feature space. In paper [START_REF] Alvarez | Road detection based on illuminant invariance[END_REF],

the authors prove that the log-chromaticity based illumination invariant grayscale image is more suitable than HSI (as done in paper [START_REF] Rotaru | Color image segmentation in HSI space for automotive applications[END_REF]) for road detection. Howev-er, in our research, we notice that using only illumination invariant image can cause over saturation problem or under saturation problem in some cases such as cloudy situation. So, in this chapter, a novel approach combining log-chromaticity space (as in paper [START_REF] Alvarez | Road detection based on illuminant invariance[END_REF]), HSI space [START_REF] Chen | Contrast-based color image segmentation[END_REF] Section 3.6 shows real data experimental results and compares results of the proposed approach against illuminant-invariance based algorithm [START_REF] Alvarez | Road detection based on illuminant invariance[END_REF]. Conclusions are given in section 3.7.

State of the Art

Road detection has been widely studied for past several years and many approaches have been proposed. According to the used equipments, methods can be categorized into three types: approach based on LRF only, approach based on camera only, approach based on both LRF and camera.

In papers [START_REF] Vandapel | Natural terrain classification using 3-d ladar data[END_REF] and [START_REF] Douillard | On the segmentation of 3D LIDAR point clouds[END_REF], the authors proposed approaches based on 3D LRF data.

The road information is segmented from points cloud. The advantage of LRF is that it can provide reliable range measurements that are not likely affected by the illumination. In certain cases, LRF-based methods can perform very well. However, a limitation of these methods is that LRF can't offer visual information, for example, traffic signals and object appearance. Yet, in many applications, such as object recognition and tracking, visual information is crucial for autonomous vehicle. Besides that, the cost of 3D LRF sensor is still very high.

Compared to LRF, camera can offer substantive visual information in favor to the recognition of on-road objects and traffic signals. Moreover, such passive sensor is not affected by the interference problem between the same type of devices. Generally, road detection based on vision is a challenging work for autonomous vehicle in outdoor scenario due to the background changement with vehicle traveling and the presence of many moving objects on the road whose movement is hard to predict. Furthermore, the structure of road is not fixed and the materials, illumination and weather conditions have effects on the road appearances. Therefore, a variety of vision-based approaches have been developed by researchers.

Generally, camera based approach is divided into monocular based and stereo based. In paper [START_REF] Gallup | Piecewise planar and non-planar stereo for urban scene reconstruction[END_REF], stereo camera is used for urban scene reconstruction. Firstly, the depthmap of stereo camera is utilized to estimate a set of piecewise planes in image. These piecewise planes belonging to the same one are then linked as one complete planar. These different completed planes are labelled as different classes.

Although depthmap is a useful tool for estimating the vertical plane, it can't work well for horizontal plane. In general, the road plane is a horizontal plane. In paper [START_REF] Soquet | Road segmentation supervised by an extended v-disparity algorithm for autonomous navigation[END_REF], authors propose to use V-disparity map to detect the road area in the image.

The V-disparity map is the image which counts the number of consistent points with same disparity value along vertical direction in disparity image. In V-disparity map, the road surface, in ideal case, is an oblique line from upper left to right down, and the obstacles, in general, are vertical lines. These lines in V-disparity map are then detected by hough transformation. However, for hough transformation, it is hard to decide the proper number of the lines.

In paper [START_REF] Kohli | Robust higher order potentials for enforcing label consistency[END_REF], Conditional Random Fields (CRF) based monocular classification is used to segment multiple scene objects in the field of view (FOV) of a single camera. Using conventional CRF based methods for segmentation generally makes

the assumption that all pixels in any small segmentation belong to the same object.

However, the pixels on the boundary of an object can be shared by multiple object classes. The authors propose a high order potential function as soft constraint to deal with the problem. Nevertheless, the accuracy of detection is still not very high and the problem formulated as an energy minimization task is a NP-hard problem. In paper [START_REF] Dahlkamp | Selfsupervised Monocular Road Detection in Desert Terrain[END_REF], a mixture of Gaussians in RGB color space and a Gaussian distribution are used to model the road. The pixels can be classified by the property of corresponding gaussian model. But the drawback of this sort of method is hard to decide the proper number of Gaussians. In paper [START_REF] Kong | Vanishing point detection for road detection[END_REF], the texture orientation of pixels and a soft voting

Framework Overview

The general framework diagram of the proposed approach is shown in Fig. 3-2. The input is a fisheye camera image. In a fisheye image, the middle part is the context of log-chromaticity space where the illumination invariant angle is obtained. Then, a classification step based on the histogram of this image (as in paper [START_REF] Jones | Statistical color models with application to skin detection[END_REF]) is implemented. If a pixel is classified as road, its value is set to 1. Otherwise, its value is set to 0. The scattered road pieces are connected to form road binary image by connected-component and fill-in hole algorithm. However, there may exist error classification due to over saturation or under saturation problem. A validation step based on LRF data is used to check and correct these errors using a refined procedure based on HSI color space information.

Coarse Road Detection Based on Illumination Invariant Image

Coarse road detection step is based on log-chromaticity space in paper [START_REF] Kim | Illumination invariant road detection based on learning method[END_REF]. It is introduced by Finlayson in paper [START_REF] Finlayson | On the removal of shadows from images[END_REF] and used to find the illumination invariant feature.

1 dimensional illumination invariant images

The 1 dimensional illumination invariant image is based on the Lambertian model introduced in paper [START_REF] Horn | Robot Vision[END_REF]. In this model, RGB color of a pixel is represented by a spectral power distribution and surface reflectance function. Let E(ζ, x, y) denote the spectral power distribution, 𝑆(𝜁, 𝑥, 𝑦) the surface reflectance function, the RGB color formed at a pixel (𝑥, 𝑦) can be described as follows:

𝜌(𝑥, 𝑦) 𝑘 = ∫ 𝐸(𝜁 𝑘 , 𝑥, 𝑦)𝑆(𝜁, 𝑥, 𝑦)𝑄 𝑘 (𝜁), 𝑘 = 𝑅, 𝐺, 𝐵 (3.1)
where 𝑄 𝑘 (𝜁) denotes the spectral sensitivity of the 𝑘 channel camera sensor, 𝜁 the wavelength of light. The integral usually occupies the range of the visible light. Based on the assumption that the camera sensor obeys to Dirac delta functions (𝑄(𝜁) = 𝑞 𝑘 𝛿(𝜁 -𝜁 𝑘 )), Equation 3.1 can be simplified as:

𝜌 𝑘 = 𝐸(𝜁 𝑘 , 𝑥, 𝑦)𝑆(𝜁 𝑘 , 𝑥, 𝑦)𝑞 𝑘 (3.2)
If the illumination follows the Planckian's laws described in paper [START_REF] Wyszecki | Color science: concepts and methods, quantitative data and formulae[END_REF], a light with a spectral power distribution can be parameterised by its colour temperature 𝑇 :

𝐸(𝜁) = 𝐹 𝑐 1 𝜁 -5 𝑒 -𝑇 𝜁 𝑐 2 (3.3)
where 𝑐 1 and 𝑐 2 are constants, and 𝐹 is a variable adjusting the overall intensity of the light. Taking equation 3.3 into 3.2, it becomes:

𝜌 𝑘 = 𝐹 𝑐 1 𝜁 -5 𝑒 -𝑇 𝜁 𝑐 2 𝑆(𝜁 𝑘 )𝑞 𝑘 , 𝑘 = 𝑅, 𝐺, 𝐵 (3.4)
By dividing 𝜌 𝑅 and 𝜌 𝐵 by 𝜌 𝐺 , then we have the log-chromaticity coordinate as follows:

𝜔 𝑗 = 𝑙𝑜𝑔 𝜌 𝑗 𝜌 𝐺 = 𝑙𝑜𝑔 𝑠 𝑗 𝑠 𝐺 + 1 𝑇 (𝑒 𝑗 -𝑒 𝐺 ), 𝑗 = 𝑅, 𝐵 (3.5)
where 𝑠 𝑘 = 𝑐 1 𝜁 -5 𝑘 𝑆(𝜁 𝑘 )𝑞 𝑘 depends on the surface and camera. 𝑒 𝑘 = -𝑐 2 /𝜁 𝑘 depends on camera. With temperature T change, for a given surface, 𝜔 will move along a straight line in log-chromaticity space. The direction of this line is determined by (𝑒 𝑘 -𝑒 𝐺 ) which depends on camera, but is independent of the surface and illumination. It means that the surface color under different illumination lies on a straight line in the log-chromaticity space and all those lines are parallel each other for different surface colors with slope (e k -e G ) (See Fig. 34). The 1-d illumination invariant image can be determined by projecting log-chromaticity of pixels into the direction orthogonal to the vector (e k -e G ).

Illumination invariant direction estimation

According to the above description, it is known that the direction (e k -e G ) is the key point for deriving the 1 dimensional illumination invariant images. In log-chromaticity space, the direction of the line perpendicular to parallel lines formed by different color surfaces under different illuminations is defined as illumination invariant direction D i . This direction can be expressed angle (R θ ) of its slope (See Fig. 34) and can be determined by its entropy energy. Let I gi denote the grayscale image derived from projecting the log-chromaticity coordinate of a given color image onto a line l R θ with slope R θ . The entropy energy E R of I gi based on its histogram H R is defined as follows:

E R = - N b i=1 H R (i)log(H R (i)) (3.6)
where N b is the number of bins of the histogram H R . The entropy value depends on the value of R θ . If R θ is consistent with D i , the log-chromaticity values of pixels will scatter in same bin of H R . Thus, it leads to a low value of the entropy energy E R .

Conversely, if R θ deviates from D i , a high value of E R will be expected. Hence, the illumination invariant direction can be determined as long as the minima of E R is solved. In paper [START_REF] Finlayson | Intrinsic images by entropy minimization[END_REF], the authors propose to determine R θ based on a single image content. This approach is not robust for many different images. Effectively, the quality of one single image content can be affected by many factors and it is not sufficient for estimating 𝐷 𝑖 . In paper [START_REF] Alvarez | Road detection based on illuminant invariance[END_REF], a method based on a set of images content is proposed and is proved to be more robust than the approach proposed in paper [START_REF] Finlayson | Intrinsic images by entropy minimization[END_REF]. In our research, we adopt this method. Firstly, a set of color images under different illuminations is chosen, and each of them is mapped into log-chromaticity space. And then, the points of the image in this space are projected onto the line with 𝑅 𝜃 initialized to 0 to form a new grayscale image. For the new grayscale image, Chebyshev's theorem proposed in paper [START_REF] Amidan | Data outlier detection using the chebyshev theorem[END_REF] is then applied to reject outlier points.

Among the remaining points, the middle 90% points are picked out to computer the histogram of the image. The bin width of the histogram is fixed by Scott's rule [START_REF] Scott | On optimal and data-based histograms[END_REF].

The Scott's rule is defined as :

𝐵𝑖𝑛 𝑤𝑖𝑑𝑡ℎ = 3.5𝑁 -1/3 𝐼 𝑠𝑡𝑑(𝐼 𝑖 ) (3.7)
where 𝑠𝑡𝑑 stands for stand deviation, and 𝑁 𝐼 is the number of pixels of image 𝐼 𝑖 . After obtaining the histogram, the entropy energy 𝐸 𝑅 of the image can be calculated using equation 3.6. However, the derived energy is only for 𝑅 𝜃 = 0. To get full results, 𝑅 𝜃 is varied from 0 to 360 by a fixed step, and the entropy is also estimated again for each value of 𝑅 𝜃 . Finally, the average entropy energy of the set of images for each angle is obtained (See Fig. 345). The angle corresponding to the minima average entropy is the expected illumination invariant direction. Fig. 3456shows the transformation from RGB image to illumination invariant grayscale image.

Classification based on illumination invariant image histogram

After the illumination invariant grayscale image computation, a classification based on normalized histogram of road model is applied. The aim is to classify the illumination invariant grayscale image into two classes "road" and "non-road". As in paper [START_REF] Tan | Color model-based real-time learning for road following[END_REF],

the road model is a fixed small field in front of vehicle in each image (that is reasonable

   G rmin < G i < G rmax (i = 1, 2, ..., N ) λ i > λ f (λ f > 0) (3.8)
where λ f is a threshold which is set to 0.25 (empirical value) and N is the number of pixels in the image. The result of classification is illustrated in Fig. 34567. The white pixels are labeled as "road" and the black pixels are "non-road".

However, as shown in fig. 34567, many scattered pieces of road pixels are presented in the derived result. To form a more complete road image, further processing is needed. Firstly, a connected-component algorithm is applied to the binary image obtained by the previous classification algorithm to form a connected groups. Then, flood-fill operation based on morphology is used to fill holes in each connected group. In fact, flood-fill operation brings the intensity values of dark areas that are surrounded by lighter areas up to the same intensity level as surrounding pixels. These connected groups, which are not connected to the predefined road region S r , are removed. Finally, a relatively more complete road binary image is formed as illustrated in Fig. 345678. Given the extrinsic parameters between the LRF and the fisheye camera known, the laser scanning plane corresponds to a line in image. In this paper, this line is called as dividing line. So if the pixels above dividing line in the image 𝐼 𝑑𝑟 are labeled as "road", the validation step should consider that an error occurs. Obviously, merely this condition is not sufficient if we consider the pixels below dividing line are falsely classified. To address this issue, the information of the amount of road pixels are used.

Suppose 𝐼 𝑑𝑟𝑛 is the current road image, and 𝐼 𝑑𝑟 𝑛-1 is the previous one. By observation, we find that the amount of road pixels is a useful tracking information for checking if there exists errors in the image 𝐼 𝑑𝑟 . Because it remains relatively stable between two consecutive frames if there is no error occurs due to over or under saturation. So, if there exists dramatic change in the amount of road pixels between two consecutive frames, it have a high probability that the errors have happened. In summary about the above two cases, the conflict checking condition is:

⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ 𝑃 𝑢𝑝𝑛 ∈ 𝑃 𝑟𝑛 𝑀 𝐹𝑛 > (1 + 𝛽)𝑀 𝐹 𝑛-1 𝑀 𝐹𝑛 < (1 -𝛽)𝑀 𝐹 𝑛-1 (3.9)
where 𝑃 𝑢𝑝𝑛 denotes a pixel above the dividing line in the image 𝐼 𝑑𝑟𝑛 , 𝑃 𝑟𝑛 represents the set of road pixels in the image 𝐼 𝑑𝑟𝑛 , 𝑀 𝐹𝑛 and 𝑀 𝐹 𝑛-1 are the amounts of road pixels in the image 𝐼 𝑑𝑟 𝑛-1 and 𝐼 𝑑𝑟𝑛 respectively. 𝛽 is a threshold set manually. In experiments, it is set to 0.1 (experience value).

Refined road detection procedure

The framework of the refining procedure is shown in Fig. 3456789. It firstly adopts two consecutive frames (𝐼 𝑑𝑟𝑛 , 𝐼 𝑑𝑟 𝑛-1 ) as inputs to find out two fields: discrepancy field

Distance image computation

The distance image is based on Euclidean metric adopted in HSI space. Compared with RGB model, HSI model is compatible with the vision psychology of human eyes [START_REF] Zhang | A new method of color image segmentation based on intensity and hue clustering[END_REF]. It is defined as : this channel. This characteristic in H channel also can explain why color processing method can't perform more better than grayscale processing approach. In paper [START_REF] Rotaru | Color image segmentation in HSI space for automotive applications[END_REF],

I = R + G + B 3 S = 1 - 3 R + G + B × min(R, G, B) H = arccos( 0.5 × [(R -G) + (R -B)] (R -G) 2 + (R -B)(G -B) ) (3.10)
the authors analyze the role of the histogram of the two components S and I in image segmentation procedure. They consider that intensity values alone (without position

d f = (I SI -I IO ) 2 + (I SS -I SO ) 2 (3.12)

False road pixels removing in distance image

In the distance image, a deciding threshold is calculated to rule out the false road pixels. Generally, this threshold is associated with the prior knowledge of road. In value is less than a predetermined upper limit threshold 𝑇 1 , it is also reserved as road.

our
The thresholds 𝑇 1 and 𝑇 2 are based on the average value and standard deviation of 𝑆 𝑟 area in the distance image, and they are defined as follows:

⎧ ⎨ ⎩ 𝑇 1 = 𝑀 𝑠𝑟 + √ 𝑓 𝑑 * 𝑠𝑡𝑑 𝑠𝑟 𝑇 2 = 𝑀 𝑠𝑟 - √ 𝑓 𝑑 * 𝑠𝑡𝑑 𝑠𝑟 (3.13)
where 𝑀 𝑠𝑟 is the average value of 𝑆 𝑟 area in the distance image and 𝑠𝑡𝑑 𝑠𝑟 the standard deviation. Fig. 3-13 shows the result of removing false road pixels in the distance image. For the convenience of statement, the outcome of this procedure is called as

𝐼 𝑓 .

Experimental results

The experimental data are composed of 336 images from four different sequences.

To reduce the computational time, all images are down sampled to 640 × 512 pixels resolution. The ground truth is labelled manually.

In first experiment, the algorithm is tested in HSI and RGB space. For each frame, we record the accuracy of detection result. The accuracy is defined as follows: In second experiment, the proposed algorithm is evaluated in detail. For quantitative evaluation, three indicators (as in paper [START_REF] Zymnis | Compressed sensing with quantized measurements[END_REF]) are calculated: 1) Accuracy; 2) Type I error rate; 3) Type II error rate. The accuracy is defined in the same way in first experiment. The type I error evaluates the cases: when road pixel is falsely classified as road pixels. Let 𝐵 𝑠 denote the amount of background pixel, 𝐵 𝑒 the amount of error background pixel. The type I error is defined as:

𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = 𝐵 𝑡 + 𝑅 𝑡 𝑀 𝐼 . ( 3
𝑇 𝑦𝑝𝑒 𝐼 𝑒𝑟𝑟𝑜𝑟 = 𝐵 𝑒 𝐵 𝑠 . (3.15)
The type II error evaluates the cases: when background pixel is falsely classified as road pixel. Let 𝑅 𝑠 denote the amount of road pixel, 𝑅 𝑒 the amount of error road pixel. The type II error is defined as:

𝑇 𝑦𝑝𝑒 𝐼𝐼 𝑒𝑟𝑟𝑜𝑟 = 𝑅 𝑒 𝑅 𝑠 . (3.16)
The average of these three indicators on the 336 images are resumed in Table 3 In last experiment, the proposed algorithm is compared with the approach proposed in [START_REF] Alvarez | Road detection based on illuminant invariance[END_REF] based only on illumination invariant. To compare the results, the same indicators are adopted. The results are shown in Table 3.3. We notice that the most significant improvement is obtained for the sequence 2, for which the percentage of refined images is the greater. Finally, it is to notice that the proposed approach outperforms the only illumination-invariant based algorithm for each indicator.

Fig. 3-16 shows some experimental results obtained by the proposed approach. As shown, we can see that the proposed algorithm can detect road pixels well. Nevertheless, the first and third images illustrate that some patterns (as white line in road centre) on the road can affect the performance of the proposed algorithm.

Fig. 3 and the method proposed in paper [START_REF] Alvarez | Road detection based on illuminant invariance[END_REF]. The lost road part in the second image of middle row (method in paper [START_REF] Alvarez | Road detection based on illuminant invariance[END_REF]) is filled as it can be seen in the second image of the last row (our method). The redundant road part in the first image of middle row is removed with our method. All above results prove that the combination of various information of image can permit to improve road detection.

Conclusion and Future Works

In this chapter, we presented an efficient algorithm for road detection in outdoor scenarios. The proposed method combines HSI color information, illumination invariant image and LRF measurements to extract road area from the fisheye image. It firstly conducts preliminary road detection in illumination invariant image. A coherence checking based on LRF measurements and the amount of road pixels is then applied to derive the road image. Unqualified images are finally refined in HSI color space.

Compared to the road detection method only based on illumination invariant image, the experiment results have shown that the proposed approach can permit to achieve some improvements for road detection. For future work, we can attempt the combination of different sorts of color space. We constated that illumination invariant may also lose the color or intensity information of objects when it reduces the effect of shadow. Searching for a proper color descriptor should bring some improvement.

Chapter 4

Multisensor Based Obstacles Detection in Challenging Scenes

Introduction

Obstacles detection is a broad research field and plays an important role in industry.

For autonomous vehicles, it aims at making the passenger to stay in a safe situation.

To achieve that goal, the vehicle has to perceive the obstacles around itself. In outdoor scenarios, obstacles detection is a tough work because the background is varying with the traveling of the vehicle and the appearance of obstacles is not predictable. For the past decade, many researchers put great efforts into settling these problems in the intelligent vehicle research community. Some researchers pay their attentions to the obstacles detection in daylight time, and others focus on nightly obstacles detection.

In this chapter, multi sensors ( fisheye camera, 2D LRF and GPS receiver) based obstacle detection method is proposed to handle a challenging case like in Fig. 4-1. In this case, the obstacles to detect have serious motion blur problem.

The rest of this chapter is organized as follows: Section 4.2 introduces the state of the art. Section 4.3 presents the framework overview of the proposed approach.

Section 4.4 describes in details how to extract potential obstacle areas. Section 4.5

shows how to locate the real obstacle position from these potential obstacles areas.

State of the Art

According to the used equipments, most of the proposed approaches for obstacle detection can be divided into three types: passive sensors based, active sensors based and both of them based.

Generally, camera based approaches are in common use for obstacles detection.

Among these approaches, some of them employ a priori knowledge of obstacles such as color [START_REF] Guo | Color modeling by spherical influence field in sensing driving environment[END_REF], vertical and horizontal edges [START_REF] Bucher | Image processing and behavior planning for intelligent vehicles[END_REF], texture [START_REF] Kalinke | A texture-based object detection and an adaptive model-based classification[END_REF], and symmetry of objects [START_REF] Zielke | Intensity and edge-based symmetry detection with an application to car-following[END_REF] to separate the obstacles from the background. These methods often have the benefit to be simple and efficient, but they are not robust to illuminance and weather changes. Some methods are based on the estimation of ego-motion [START_REF] Kruger | Real-time estimation and tracking of optical flow vectors for obstacle detection[END_REF][52] [START_REF] Molineros | Monocular rearview obstacle detection using residual flow[END_REF]. In paper [START_REF] Jazayeri | Vehicle detection and tracking in car video based on motion model[END_REF], the authors propose a method based on the motion trace of feature points

Possible Region of Obstacle Presence Extraction

As illustrated in Fig. 4-2, there are several procedures before the possible obstacle region extraction step. We introduce some important steps in the following sections. in image plane, it can be mapped to the ground plane point P g by the following transformation:

𝑃 𝑔 = ℎ ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ -𝑐𝑜𝑠𝛽 𝑓𝑢 𝑠𝑖𝑛𝛼𝑠𝑖𝑛𝛽 𝑓𝑣 𝑝𝑢𝑐𝑜𝑠𝛽 𝑓𝑢 -𝑝𝑣𝑠𝑖𝑛𝛼𝑠𝑖𝑛𝛽 𝑓𝑣 -𝑐𝑜𝑠𝛼𝑠𝑖𝑛𝛽 0 𝑠𝑖𝑛𝛽 𝑓𝑢 𝑠𝑖𝑛𝛼𝑐𝑜𝑠𝛼 𝑓𝑣 -𝑝𝑢𝑠𝑖𝑛𝛽 𝑓𝑢 -𝑝𝑣𝑠𝑖𝑛𝛼𝑐𝑜𝑠𝛽 𝑓𝑣 -𝑐𝑜𝑠𝛼𝑐𝑜𝑠𝛽 0 0 𝑐𝑜𝑠𝛼 𝑓𝑣 -𝑝𝑣𝑐𝑜𝑠𝛼 𝑓𝑣 + 𝑠𝑖𝑛𝛼 0 0 -𝑐𝑜𝑠𝛼 ℎ𝑓𝑣 𝑝𝑣𝑐𝑜𝑠𝛼 ℎ𝑓𝑣 -𝑠𝑖𝑛𝛼 ℎ 0 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 𝑢 𝑥 𝑣 𝑦 1 1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ (4.1)
where 𝑓 𝑢 , 𝑓 𝑣 are the horizontal and vertical focal lengths, 𝑝 𝑢 , 𝑝 𝑣 are the optical center coordinates. Similarly, to convert any point 𝑃 𝑔 (𝑔 𝑥 , 𝑔 𝑦 , -ℎ, 1) in IPM image to normal view image, the following transformation can be applied:

𝑃 𝑖 = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 𝑓 𝑢 𝑐𝑜𝑠𝛽 + 𝑝 𝑢 𝑐𝑜𝑠𝛼𝑠𝑖𝑛𝛽 𝑝 𝑢 𝑐𝑜𝑠𝛼𝑐𝑜𝑠𝛽 -𝑠𝑖𝑛𝛽𝑓 𝑢 -𝑝 𝑢 𝑠𝑖𝑛𝛼 0 𝑠𝑖𝑛𝛽(𝑝 𝑣 𝑐𝑜𝑠𝛼 -𝑓 𝑣 𝑠𝑖𝑛𝛼) 𝑐𝑜𝑠𝛽(𝑝 𝑣 𝑐𝑜𝑠𝛼 -𝑓 𝑣 𝑠𝑖𝑛𝛼) -𝑓 𝑣 𝑐𝑜𝑠𝛼 -𝑝 𝑣 𝑠𝑖𝑛𝛼 0 𝑐𝑜𝑠𝛼𝑠𝑖𝑛𝛽 𝑐𝑜𝑠𝛼𝑐𝑜𝑠𝛽 -𝑠𝑖𝑛𝛼 0 𝑐𝑜𝑠𝛼𝑠𝑖𝑛𝛽 𝑐𝑜𝑠𝛼𝑐𝑜𝑠𝛽 -𝑠𝑖𝑛𝛼 0 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 𝑔 𝑥 𝑔 𝑦 -ℎ 1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ (4.2)

Central line marker detection

The lane central marker is used as a reference line in later road model mapping section. In paper [START_REF] Alvarez | Geographic information for vision-based road detection[END_REF], the authors propose to map the road shape model to driver's view. However, it involves a transformation in four different kinds of coordinate systems. Such transformation often brings many errors, and meanwhile, it is hard to determine all of transformational parameters. To avoid this problem, we propose to map road shape model to the image I o (IPM image after distortion removal).

However, this mapping requires some marks on the road. Road boundary and lane central marker are the two choices. In our method, the lane central marker is the favorable one because it is not often covered by other objects in the traveling journey.

Given two-lane straight street, the lane marker detection flow chart is shown in Fig. 45.

Figure 4-5: Lane marker detection flow chart

Filter

The image I o is firstly sent to a filter that aims at getting a high response to the lane markers and removing irrelevant pixels. Two separable Gaussian kernels as in paper [START_REF] Aly | Real time detection of lane markers in urban streets[END_REF] are used to compose the filter. One for vertical direction, and another for in horizontal direction. The horizontal direction is a smoothing Gaussian kernel with expected width of lane segment. The filter for vertical direction is a second derivative of Gaussian kernel with the height of the lanes. One example of the performance of the proposed filter is shown in Fig. 456. As seen, the lane marker have higher response The second condition is a line tracking model. In practice, some parts of a road may haven't obvious markers or have other similar figures (zebra stripes). It brings the difficulties to detect central line marker. To make the approach get more robust, a tracking model is explored. In fact, the tracking model is a cache which conserves the central line marker positions of the latest five frames. Given 𝑙 𝑎𝑣𝑒 the average position of central line marker positions of the latest five frames, a candidate is accepted if it fits the the following condition:

Candidate lines refinement

0.9 * l ave < l c < 1.1 * l ave (4.3)
where l c is the position of the central line marker candidate. Otherwise, l ave is considered as the central line marker. However, if five successive central line marker candidates are outside this range, the cache will be cleaned and updated.

Road shape model computation

Road shape model reflects the geometry of the road such as left turn, right run, or cross-like junction. This contextual information is a very useful clue in many applications. To acquire ahead road shape model, driving direction information is necessary. To obtain this information effectively, it needs to transform GPS coordinate from 3D world geodetic system to 2D local surface system. The details for these coordinate conversion algorithms are given in appendix A.

Openstreetmap

GIS is the most important factor for building the road shape model. It is extracted from OpenstreetMap database. OpenstreetMap is a project to build a free geographic database of the world. It gathers all sorts of information including building, road, waterways and so on. There are three primitive type data combined with free form tagging scheme in OpenstreetMap to describe geographic feature: nodes, ways and relations. The first two types play an important role in the proposed approach and are introduced in details.

Nodes are fundamental elements. They are points in space and the only primitive to have position information [START_REF] Bennett | OpenStreetMap[END_REF]. The other two primitives are based on the nodes.

Each node has its latitude and longitude stored in decimal format up to 7 decimal places. This guarantees the latitude and longitude resolution can reach round 1𝑐𝑚 at the equator or 0.6𝑐𝑚 at Greenwich. This accuracy is enough for most applications. Ways are lists of nodes arranged in order. They can be used to describe something with linear features, such as roads and paths. In the map, a way must have at least two nodes and a maximum of 2000 nodes. The ordering of these nodes in a way is preserved, so the way often has direction. The direction is from the first node to the last one. If the first and last nodes of a way are the same, the way forms a closed area. For a more complex shape than a simple polygon, this can be done using several ways and a relation to link them. Additionally, for a road, the way is usually placed down the center line of the physical feature. The road skeleton model is a link which concatenates the road nodes within a map database one by one (See Fig. [4][5][START_REF] Zhang | Extrinsic calibration of a camera and laser range finder (improves camera calibration)[END_REF][START_REF] Mei | Calibration between a central catadioptric camera and a laser range finder for robotic applications[END_REF][START_REF] Brim | Modeling and calibration of coupled fish-eye CCD camera and laser range scanner for outdoor environment reconstruction[END_REF][START_REF] Naroditsky | Automatic alignment of a camera with a line scan LIDAR system[END_REF][10][START_REF] Kannala | A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses[END_REF][START_REF] Mei | Single view point omnidirectional camera calibration from planar grids[END_REF]. However, great gap between two adjacent road nodes often appears in map database. Before concatenating the road nodes, it needs to operate a linear interpolation. Firstly, the GPS data are used to pick out which road the target vehicle is traveling within the map and to identify traveling direction on this road. Then, the corresponding road nodes are extracted from the map database (OpenStreetMap [START_REF] Haklay | OpenStreetMap: user-generated street maps[END_REF]). Let R c denote the road where the vehicle is traveling, R i (i = 1, 2..., N ) the i-th road node of R c in the map database. The linear interpolation operation is implemented between two adjacent road nodes if the following condition is satisfied.

d R i R i+1 < m λ (i = 1, 2, ..., N -1) (4.4)
where d R i R i+1 is the distance between R i and R i+1 nodes, m λ is a threshold set manually. In our approach, it is set to 2 (unit meter). After that, the closest node to the vehicle location is chosen from the road nodes and interpolation nodes and is used as the starting point for the road skeleton construction. From this point, the paper [START_REF] Yang | A new approach for in-vehicle camera obstacle detection by ground movement compensation[END_REF], is used to filter these groups by the following two conditions:

   G p > K gp G g > K gg (4.5)
where G p is the peak number of a group in polar histogram, G g is the pixels number of the group, K gp and K gg are rough thresholds to eliminate the small groups. In our application, these two parameters are set to 25 and 75 (empirical value). A group is conserved if the above two conditions are satisfied. As a result, the remaining group fields represent the possible regions of obstacles presence.

⎧ ⎨ ⎩ |𝜃 𝐿 𝑗+1 -𝜃 𝐿 𝑗 | < 𝜃 𝐿 |𝑀 𝐿 𝑗+1 -𝑀 𝐿 𝑗 | < 𝑀 𝐿 (4.6)
where 𝑀 𝐿 𝑗+1 and 𝑀 𝐿 𝑗 are the LRF distance of the two adjacent correspondences respectively, 𝜃 𝐿𝑗+1 and 𝜃 𝐿𝑗 are the corresponding LRF angle of them respectively, 𝜃 𝐿 and 𝑀 𝐿 are empirical thresholds. In experiment, 𝜃 𝐿 and 𝑀 𝐿 are set to 4 and 0.5 meter respectively. As a result, these different correspondences gather into several different classes. For each class, a rectangle filter window (See Fig4-16) is assigned to each of them. The center of this window is located at the centroid of this class.

The width and height of the window are based on the minimum bounding box of the class. They are determined by following equations:

⎧ ⎨ ⎩ 𝑊 𝑤𝑖 = 1.2 * 𝑊 𝑐𝑖 ; 𝐻 𝑤𝑖 = 1.2 * 𝑊 𝑐𝑖 + 𝐻 𝑐𝑖 ; (4.7)
where 𝑊 𝑤𝑖 and 𝐻 𝑤𝑖 are the width and height of the window of the i-th class respectively, 𝑊 𝑐𝑖 and 𝐻 𝑐𝑖 are the width and height of the minimum bounding box of i-th class. A pixel in the normal view image 𝐼 𝑛𝑣 belongs to an obstacle if it is within the corresponding filter window. Otherwise, the pixel is discarded. The minimum bounding box which contains remaining pixels and the corresponding class are the final output.

Experiments

The experimental data consist of a road database obtained from openstreetmap and two sequences (sequence1: 231 frames, sequence2: 171 frames) with the corresponding LRF measurements and GPS data captured by our experimental vehicle. The ground truth is labelled manually. For the sequence 1, the results obtained using our method are compared with the results obtained with two other methods proposed in [66] and [START_REF] Felzenszwalb | Object detection with discriminatively trained part-based models[END_REF]. Public codes for the two methods are available on the internet. Some results of these three approaches are shown in Fig. 4-17. We can notice that the appearance of the vehicle in Fig. 4-17(a) is blur, especially for the front part of the vehicle. Meanwhile, we can see that all the outcomes of the method in paper [66] are incorrect and the method proposed in paper [START_REF] Felzenszwalb | Object detection with discriminatively trained part-based models[END_REF] detect nothing. The reason that make the two methods invalid is the loss of visual feature. Fig. 4-17(d) shows that, to an extent, our method can overcome this issue and give more reasonable results. For the sequence 2, the results obtained using our method are compared with the results obtained with the method proposed in [66]. Some results of the two approaches are shown in Fig. 4-18. As illustrated in Fig. 4-18(a), the appearance of the pedestrian is not very clear, especially for the leg. Meanwhile, in Fig. 4-18(b), similar with the experiment results in sequence 1, the method proposed in paper [START_REF] Felzenszwalb | Object detection with discriminatively trained part-based models[END_REF] still can't detect the object. From the Fig. 4-18(c), we can see that our method can output more reasonable results. It is noticeable that, in Fig. 4-18(c), the right detection result lose partial leg information of the pedestrian. That is because, in road detection procedure, the lost leg part are treated as road. We also evaluate the influence of LRF measurements based confirmation on the performance of the proposed approach.

The results are shown in Table 4.1. For sequence 1, one can notice that the LRF measurements based confirmation improve the correct rate greatly. However, the hit rate declines sightly due to the shift of LRF measurements of obstacles. For sequence 2, we can see that the two criteria are higher with LRF measurements based confirmation case than without LRF measurements based confirmation case. The reason which makes hit rate better is that the super correspondence can provide additional object information when the motion blur happens at leg of pedestrians and super correspondence lay on the leg. In other words, without LRF measurements based confirmation, the leg part in the image is often treated as road making the hit rate worse.

Condition

Conclusion and Future Works

In this paper, we presented a novel method to detect obstacles with motion blur case.

The obstacle visual features are usually useless. This is a big challenge for many object detection methods. To handle this issue, GIS, road information and LRF measurements are combined with image information. The GIS and road information are applied to extract possible regions of obstacle presence and the LRF measurements are used to filter these regions.

In the possible regions of obstacle presence extraction procedure, the IPM technique is firstly used to remove the perspective effect to the undistorted road image and undistorted fisheye image. Then, a road model ahead of the vehicle is constructed by using GPS data and Openstreet map. After that, this model is mapped into IPM image by aligning the road skeleton with the central line marker in that image. The road boundary can be determined during the mapping procedure. Potential obstacles area can be found out using the road information and relative road boundary information.

In obstacle confirmation step, LRF measurements are used. LRF measurements can provide the width information of target obstacles which can help us to eliminate the disturbance of "tail" pixels such as shown in Fig. 4-16. The LRF measurements are firstly screened by the road boundary. And , the remaining data are then grouped into several classes. In each class, a filter window based on the bounding boxes of this class is built. This window is then applied to filter the "non-obstacle" pixels.

For future works, the following aspects could be expected to get improvements:

1.Currently, all the road nodes in the vehicle running road are gathered manually. For future works, we would like to develop a method to extract these nodes automatically.

2.For the LRF measurements classification, the geometrical shape which is made up of LRF measurements on the objects can be exploited to help grouping these measurements. For most of the cars, their geometrical shapes consisting of LRF measurements on them are similar. This property is useful to us to classify the LRF measurements.

3.

For roads without lane marker, it is hard to map the road model shape into the image. To deal with this case, it is a considerable choice to detect the road boundary. 91 a complete region representing a tracked object. Real date experiments are shown in Section 5.9. Finally, a conclusion is presented to end this chapter.

State of the Art

Although there are many methods regarding objects tracking, most of them can be categorized by their appearance descriptors. In this chapter, we group them into four classes: raw pixel descriptor based, local feature descriptor based, statistical descriptor based and online self-learning descriptor based.

Raw pixel descriptor reflects the basic statical property of object appearance such as color distribution, pixel displacements, correlation information of object pixels and so on. In paper [START_REF] Silveira | Real-time visual tracking under arbitrary illumination changes[END_REF], target objects are encoded by their color or intensity values in image. In paper [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF], local binary pattern (LBP) technique is used to represent object appearance. However, this sort of approaches is not robust enough due to the accumulation errors during the tracking process. To address this problem, other information are explored. In paper [START_REF] Wu | Contextual flow[END_REF][71], optical flow is used to represent targets.

Although optical flow information can help to locate moving object in the image, it doesn't work very well in the presence of motion blur and illumination fluctuation.

Other researchers [72][73] propose covariance representation based on affine-invariant metric or log-euclidian metric to capture the targets in image. This sort of approaches is computational efficient and robust to illumination changes and occlusion due to seizing the intrinsic correlation properties of tracking object. Nevertheless, because of using pixel-wise statistics, it is also easily affected by image noise. In papers [START_REF] Van De Weijer | Boosting color saliency in image feature detection[END_REF][75],

histogram of target object is adopted by researcher. Color histogram based on HSV [START_REF] Van De Weijer | Boosting color saliency in image feature detection[END_REF] is applied in object representation. In paper [START_REF] Gevers | Color feature detection[END_REF], a rg-histogram based on normalized RGB color mode is proposed to represent aim object. However, the methods will gradually drift from targets during the tracking procedure. To improve the robustness of color or intensity histogram based methods, many researchers suggest to embody other information into it. In paper [START_REF] Birchfield | Spatiograms versus histograms for regionbased tracking[END_REF], a color-spatial joint model is used to describe the color distribution in spatial space. In paper [START_REF] Nejhum | Online visual tracking with histograms and articulating blocks[END_REF], tracked region is divided into several patches. By the spatial layout information of these patches and their intensity histograms, the final tracking location can be determined. In paper [START_REF] Ning | Robust object tracking using joint color-texture histogram[END_REF], texture information is integrated into color histogram. In paper [START_REF] Wang | Integrating color and shape-texture features for adaptive real-time object tracking[END_REF], gradient orientation histogram [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] and color histogram are utilized to determine the tracking position.

Local feature descriptor means to represent targets based on several local interesting points or their combination. In paper [START_REF] Abdel-Hakim | Csift: A sift descriptor with color invariant characteristics[END_REF], a SIFT together with color invariants based descriptor is applied in object tracking. In paper [START_REF] Scovanner | A 3-dimensional sift descriptor and its application to action recognition[END_REF], 3D SIFT based on bag of words descriptor is proposed. In paper [START_REF] Willems | An efficient dense and scaleinvariant spatio-temporal interest point detector[END_REF], an extended SURF visual features are proposed for object tracking. In paper [START_REF] Kim | Real time object tracking based on dynamic feature grouping with background subtraction[END_REF], corner features are considered for the construction of appearance model. Recently, learning based methods are often proposed in several literatures. Different sort of visual features are packed to train a set of weaker learners. These weaker learning classifiers are then sent to construct a composed classifier which is applied in object tracking. In paper [START_REF] Babenko | Visual tracking with online multiple instance learning[END_REF], Harr-like features [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF] are used to construct weak classifiers. Generally, local feature descriptor is robust to shape deformation and rotation. However, feature point detection is often disturbed by image noise and background changes. For feature set based weak learners, the computational cost is expensive.

Statistical descriptor use different kinds of statistical models to fit tracked objects.

These models could be Gaussian model, kernel model and template in subspace. In paper [START_REF] Yu | Differential tracking based on spatial-appearance model (sam)[END_REF], gaussian model is introduced into object tracking field. The authors use a set of gaussian models to define a density function to approximate target appearance. Although the gaussian model based method can get reliable results, it is hard to determine the number of gaussian function. In paper [START_REF] Comaniciu | Kernel-based object tracking[END_REF], a kernel-based model is introduced to represent target object. The authors use spatially isotropic kernel to regularize target color histogram and mean shift based on Bhattacharyya metric to locate target position. In paper [START_REF] Leichter | Tracking by affine kernel transformations using color and boundary cues[END_REF], edge information is integrated into kernel function. Other researchers focus on subspace for object tracking. In paper [START_REF] Wen | Online spatio-temporal structural context learning for visual tracking[END_REF], target object is considered as linear combination of several basis templates in subspace. In paper [START_REF] Mei | Robust visual tracking using &# x2113; 1 minimization[END_REF], the authors propose a tracking method based on sparse representation.

Initialization

The input for the initialization step is the previous image with annotated tracked target by a tracking box and the tracked object's LRF measurements. The target in the tracking box in the previous image is regularized into 24x24 size (Fig. 5-2). The tracked object's LRF measurements are used to obtain object information. Explicitly, object information contains: the positions of the projected points of the tracked object's LRF measurements in the image, the distances of the tracked object's LRF , the angles of the tracked object's LRF . The distances and angles of the tracked object's LRF are used to calculate the average distance, average angle, the maximum distance residual and the maximum degree residual. For statement convenience, we

𝑅 𝑙𝑝𝑝 = 𝑃 𝑏𝑟𝑦 - ∑ 𝑁 𝑖=1 𝑃 𝑖 𝑟𝑦 𝑁 𝑃 𝑏𝑟𝑦 -𝑃 𝑏𝑙𝑦 (5.1)
where 𝑁 is the amount of the tracked object's LRF measurements. The obtained 𝐴 𝑑𝑖 , 𝐴 𝑎𝑛 , 𝑀 𝑑𝑖 , 𝑀 𝑎𝑛 and 𝑅 𝑙𝑝𝑝 are packed as key information to send to frame information buffer.

Frame Information Buffer

The frame information buffer stores key information of three previous frames. These key information in the buffer are used to calculate three types of decision factors:

distance, angle and low part proportion. The distance decision factor is a range decided by the average distance and average maximum distance residual of the three previous frames. Let 𝐷 𝑑𝑖 denote the distance decision range factor, 𝐹 𝑎𝑑𝑖 and 𝐹 𝑚𝑑𝑖 the average distance and average maximum distance residual of the three pervious frames, 𝑘 the current frame, then we have:

𝐷 𝑑𝑖 =[𝐹 𝑎𝑑𝑖 -𝐹 𝑚𝑑𝑖 , 𝐹 𝑎𝑑𝑖 + 𝐹 𝑚𝑑𝑖 ] (5.2) ⎧ ⎨ ⎩ 𝐹 𝑎𝑑𝑖 = 𝐴 𝑘-3 𝑑𝑖 +𝐴 𝑘-2 𝑑𝑖 +𝐴 𝑘-1 𝑑𝑖 3 𝐹 𝑚𝑑𝑖 = 𝑀 𝑘-3 𝑑𝑖 +𝑀 𝑘-2 𝑑𝑖 +𝑀 𝑘-1 𝑑𝑖 3
where 𝐴 𝑘 𝑑𝑖 and 𝑀 𝑘 𝑑𝑖 are the average distance and maximum distance residual in the frame k. The angle decision factor (𝐷 𝑎𝑛 ) is also a range and can be calculated by the same way. The LPP decision factor, which is also range , is determined by the following expression:

𝐷 𝑙𝑝𝑝 =[0.95 * 𝐹 𝑙𝑝𝑝 , 1.05 * 𝐹 𝑙𝑝𝑝 ] (5.3) 𝑤𝑖𝑡ℎ 𝐹 𝑙𝑝𝑝 = 𝑅 𝑘-3 𝑙𝑝𝑝 +𝑅 𝑘-2 𝑙𝑝𝑝 +𝑅 𝑘-1 𝑙𝑝𝑝 3
where 𝑅 𝑘 𝑙𝑝𝑝 is the LPP in the frame k. The above decision factors are used in region growth in the following step.

Object Representation

Taking into account motion blur case, we tend to adopt raw pixel information based descriptor to encode tracked objects. The frequently used method is to encode object using color or intensity value directly. As edge visual features of objects are not very clear in motion blur case, we mainly focus to encode their central field. The proposed object descriptor is divided into two directions: vertical and horizontal directions.

As the encoding way in both directions is the same, in the following we will only introduce how to encode the object in the vertical direction.

Estimation of Small Region

Interesting small region (ISR) is a field that partially covers a target object. To determine ISR, we firstly define a searching area as a rectangular field, which has the same center of the tracking box in the previous frame, its width and height are three times larger than the tracking box's width and height respectively. An example of searching area is shown in Fig. 5-4. In the searching area, a group of scanning grids

Growth Strategy

ISR is the field merely containing a small part of the tracked object. To get a relative complete field of the tracked object, this region is expanded. Generally, there exist three region growth strategies: single direction, bidirectional and mixture way. For single direction and mixture way strategies, the key point is to know where is the brink of the object. However, in practice, it is hard to find out this information. For easy to find out these two information than to determine the brink. In the proposed approach, the bidirectional growth strategy is adopted. This strategy is performed in the horizontal and vertical directions. We will firstly introduce the horizontal direction based growth, and then the vertical direction based one.

As described above, to perform bidirectional strategy we need to know the partial location information of the tracked object in the image and where it should stop. The first information is derived from the previous section (estimation of small region).

The second information can be obtained in LRF space, which is defined as a 2D reprojection space of LRF measurements and it completely overlaps with the image.

Each element in the space is characterized by three attributes: distance, position and angle. Fig. 56shows an example of LRF space. The growth of ISR in the horizontal direction is implemented in LRF space. Let's define the region corresponding to ISR in LRF space as LISR. It means that the growth of LISR is equivalent to the growth of ISR. Taking the middle point in LISR as the center, the horizontal brink of LISR is firstly expanded until the point (in LRF space), which is not within the range 𝐷 𝑑𝑖 . Then, a shrink operation is applied to the horizontal brink if the angle of the points in the expanded LISR is beyond the range 𝐷 𝑎𝑛 . Through the expansion and shrink procedures, the target object LRF measurements and its horizontal brink can be determined.

Because of the lack of the upper and lower limits of the tracked object in LRF space, it is hard to use the same way to expand the LISR in vertical direction. To address this problem, an alternative way is proposed. We firstly expand the LISR in vertical direction to a predefined height, which is calculated by multiplying the width of LISR and the ratio of the tracking box's height to its width in the first frame.

However, this may lead to drifting gradually from the target during the tracking procedure (Fig. 567). To tackle this issue, the position of spanned region has to be According to this principle, if 𝑅 𝑙𝑝𝑝 of that region is higher than the upper limit of the range 𝐷 𝑙𝑝𝑝 and the average distance in that region is less than 𝐹 𝑎𝑑𝑖 , that region is translated upwards until its LPP fall into 𝐷 𝑙𝑝𝑝 . In other words, when the vehicle is approaching to tracked objects, their 𝑅 𝑙𝑝𝑝 should get small. Conversely, if 𝑅 𝑙𝑝𝑝 of that region is smaller than the lower limit of the range 𝐷 𝑙𝑝𝑝 and the average distance in that region is more than 𝐹 𝑎𝑑𝑖 , that region is translated downwards until its LPP fall into 𝐷 𝑙𝑝𝑝 . The above conditions can be summarized as follows:

⎧ ⎨ ⎩ 𝐷 𝑎𝑠 < 𝐹 𝑎𝑑𝑖 :↓ (𝑅 𝑙𝑝𝑝 > 1.05𝐹 𝑙𝑝𝑝 ) 𝐷 𝑎𝑠 < 𝐹 𝑎𝑑𝑖 :↑ (𝑅 𝑙𝑝𝑝 > 0.95𝐹 𝑙𝑝𝑝 ) (5.5)
where 𝐷 𝑎𝑠 denotes average distance of spanned region.

Experiment

Test results in real scenarios

In the first experiment, the proposed algorithm is tested in two long term videos captured by our experimental car moving in outdoor scene. There are totally 1000 frames in each video and several different cases (normal, presence of motion blur, background change, scale and appearance changes, over exposure and shadow). A man riding a bike is our target. The proposed method is firstly tested on the first video.

Comparison results

In the second experiment, the proposed method is compared with the method presented in paper [START_REF] Kalal | Tracking-learning-detection[END_REF]. To get ground truth, we manually annotate the tracked objects with a minimun bounding box. The criteria indicator is the overlapping rate (OR).

Let 𝑆 𝑏 denote the area of minimum bounding box in the image, 𝑆 𝑡 the area of the tracking box in the image. OR is defined as follows:

𝑂𝑅 = 𝑆 𝑏 ∩ 𝑆 𝑡 𝑀 𝑎𝑥(𝑆 𝑏 , 𝑆 𝑡 ) . ( 5.6)
The comparison results are shown in Fig. 5-16 and Fig. 5-17. The red line represents the results obtained from the proposed approach. The blue one is the results derived from the approach presented in paper [START_REF] Kalal | Tracking-learning-detection[END_REF]. In Fig. 5-16, the method proposed in paper [START_REF] Kalal | Tracking-learning-detection[END_REF] missed the target from frame 100 to frame 600. After frame 600, it gets back the target. The reason is that the method in [START_REF] Kalal | Tracking-learning-detection[END_REF] has a big buffer which stores the descriptors of the tracked object in the past frames. This buffer can help the method to get back the target when it is lost. However, in certain cases, it also does harm to track the object without other judgment conditions because it may reserve the inaccurate object descriptor. In Fig. 5-17, from frame 1 to frame 600, the proposed method performs a little better than the approach in [START_REF] Kalal | Tracking-learning-detection[END_REF] overall. From frame 630 to frame 670, the outcomes of the proposed approach get worse than the method proposed in paper [START_REF] Kalal | Tracking-learning-detection[END_REF]. The reason is that the performance is affected by LRF measurements. In deed, if the LRF measurements get worse, the outcome of our approach will go bad. After frame 700, our method outperforms the approach proposed in paper [START_REF] Kalal | Tracking-learning-detection[END_REF]. The reason that the performance of the approach proposed in paper [START_REF] Kalal | Tracking-learning-detection[END_REF] decrease rapidly is that the variation of the appearance of target object and motion blur occur simultaneously from frame 705 to frame 715.

Figure 5-16: Comparison results between the proposed method and the approach presented in paper [START_REF] Kalal | Tracking-learning-detection[END_REF] for first video is the examples from the method in paper [START_REF] Kalal | Tracking-learning-detection[END_REF]. Below is the results of the proposed method. From Fig. 5-18, after vibration of vehicle, the method proposed in paper [START_REF] Kalal | Tracking-learning-detection[END_REF] gradually lost the target. The same things also happens in Fig. 5-19. The presence of motion blur does very harm to the approach proposed in paper [START_REF] Kalal | Tracking-learning-detection[END_REF]. To an extent, the proposed approach can handle this issue.

Conclusion and Future Works

This chapter presented a method for tracking objects. It establishes a list of samples in a searching area to determine the interesting small region. This region is mapped to LRF space. In this space, a bidirectional growth strategy is applied to the small However, the occlusion problem is not tackled by the proposed method. Considering the low quality of the images, it is a challenge to find a proper descriptor to deal with the problem. This work can be treated as a direction of future research. Besides that, the case of objects leaving and reappearings is not tackled in our method.

111

Chapter 6

Conclusion and Future Works

Conclusion

The issues addressed in this thesis refer to several basic aspects of road scene understanding in perception systems of intelligent vehicle. A complete and precise description of road scene state will pave a smooth way for intelligent vehicle applications.

The following paragraphs provide a general conclusion for the methods proposed in the thesis.

In the first, a method regarding extrinsic calibration between fisheye camera and Laser Ranger Finder is presented in chapter2. The extrinsic calibration is based on the intrinsic parameters estimation of fisheye camera. Three indicators are proposed to compare three type models of fisheye camera introduced in literature. Based on the selected fisheye model, a chessboard pattern is applied to locate several points on the LRF scanning plane. The scanning plane is then estimated by these points. Based on the obtained scanning plane and several points on this plane, two geometrical constraints are built. Simulation and real experiments show the effectiveness of the proposed method. Finally, we applied it for motion trajectory estimation.

In the second, fisheye camera and LRF based road detection method is introduced in chapter3. It firstly uses histogram based classification method to implements a preliminary road detection in illuminance invariant image. The derived result is then checked by a coherence principal based on the LRF measurements and the amount of road pixels. The unqualified road detection result is refined by a distance image based classification method in HSI space. The real experiment results show the effectiveness of the proposed approach in reducing the "over saturated" or "under saturated" errors caused by the method merely based on illuminance invariant image in cloudy day.

In the third, a LRF, GPS and fisheye camera based obstacle detection approach is proposed in chapter4. Firstly, a distortion removal operation is applied to both the fisheye image and the road image (chapter3). Then, the perspective effect is removed from the undistorted fisheye image and the undistorted road image by inverse perspective mapping algorithm. After that, the center lane marker in undistorted fisheye image is detected. Meanwhile, a road model is built by the GIS information obtained by GPS data and Openstreetmap. This model is mapped into the undistorted fisheye image by aligning the road skeleton to the center lane marker, in order to determine the complete road boundary information. The potential obstacle regions can be found out by combining the road boundary information and road image without perspective effect. Finally, LRF measurements are used to pick out the real obstacle regions from these candidate regions. The real experiments show the effectiveness of the proposed method in coping with obstacle detection in presence of motion blur.

In the fourth, we present a method for object tracking in chapter 5. It encodes target objects by weighted colors. Based on the derived code, a list of samples in searching area is sorted to determine the interesting small region. This small region is expanded in LRF space according to three decision factors obtained from the position of tracking box and the LRF measurements of target objects in previous images. The long term real experiment results show the effectiveness of the proposed tracking algorithm.

Future Works

In the author's point of view, some recommendations are given to improve the work.

For extrinsic calibration between camera and LRF, different sensors could be attempted such as 3D LRF and stereo vision. Meanwhile, extrinsic calibration between the combination of several kinds of cameras (fisheye, stereo, omnidirectional) and LRF is also an interesting work. The property of straight line in fisheye image is another important research direction.

For road detection, a robust color descriptor for road deserves researching. Besides that, GIS features in map could be an available information to improve the results.

Combination of several cameras is also a considerable option. Segmentation technique based on condition random field (CRF) also deserves attempting.

For obstacle detection, optical flow with fisheye camera should be interesting to improve detection results. To build more kinds of road model, it can make the proposed method more generalized.

For object tracking, the mixture of gaussian models can be used to represent target objects to deal with the occlusion problem. Multi-instances based self-learning techniques could be attempted for shortly object disappearance. .

List of Tables

125

R ésum é :

La perception de sc ènes routi ères est un domaine de recherche tr ès actif. Cette th èse se focalise sur la d étection et le suivi d'objets par fusion de donn ées d'un syst ème multi-capteurs compos é d'un t él ém ètre laser, une cam éra fisheye et un syst ème de positionnement global (GPS). Plusieurs étapes de la chaîne de perception sont étudi ées : le calibrage extrins èque du couple cam éra fisheye/t él ém ètre laser, la d étection de la route et enfin la d étection et le suivi d'obstacles sur la route. Afin de traiter les informations g éom étriques du t él ém ètre laser et de la cam éra fisheye dans un rep ère commun, une nouvelle approche de calibrage extrins èque entre les deux capteurs est propos ée. La cam éra fisheye est d'abord calibr ée intrins èquement. Pour cela, trois mod èles de la litt érature sont étudi és et compar és. Ensuite, pour le calibrage extrins èque entre les capteurs, la normale au plan du t él ém ètre laser est estim ée par une approche de RANSAC coupl ée à une r égression lin éaire à partir de points connus dans le rep ère des deux capteurs. Enfin une m éthode des moindres carr és bas ée sur des contraintes g éom étriques entre les points connus, la normale au plan et les donn ées du t él ém ètre laser permet de calculer les param ètres extrins èques. La m éthode propos ée est test ée et évalu ée en simulation et sur des donn ées r éelles. On s'int éresse ensuite à la d étection de la route à partir des donn ées issues de la cam éra fisheye et du t él ém ètre laser. La d étection de la route est initialis ée à partir du calcul de l'image invariante aux conditions d'illumination bas ée sur l'espace log-chromatique. Un seuillage sur l'histogramme normalis é est appliqu é pour classifier les pixels de la route. Ensuite, la coh érence de la d étection de la route est v érifi ée en utilisant les mesures du t él ém ètre laser. La segmentation de la route est enfin affin ée en exploitant deux d étections de la route successives. Pour cela, une carte de distance est calcul ée dans l'espace couleur HSI (Hue,Saturation, Intensity). La m éthode est exp ériment ée sur des donn ées r éelles. Une m éthode de d étection d'obstacles bas ée sur les donn ées de la cam éra fisheye, du t él ém ètre laser, d'un GPS et d'une cartographie routi ère est ensuite propos ée. On s'int éresse notamment aux objets mobiles apparaissant flous dans l'image fisheye. Les r égions d'int ér êts de l'image sont extraites à partir de la m éthode de d étection de la route propos ée pr éc édemment. Puis, la d étection dans l'image du marquage de la ligne centrale de la route est mise en correspondance avec un mod èle de route reconstruit à partir des donn ées GPS et cartographiques. Pour cela, la transformation IPM (Inverse Perspective Mapping) est appliqu ée à l'image. Les r égions contenant potentiellement des obstacles sont alors extraites puis confirm ées à l'aide du t él ém ètre laser. L'approche est test ée sur des donn ées r éelles et compar ée à deux m éthodes de la litt érature. Enfin, la derni ère probl ématique étudi ée est le suivi temporel des obstacles d étect és à l'aide de l'utilisation conjointe des donn ées de la cam éra fisheye et du t él ém ètre laser. Pour cela, les r ésultats de d étection d'obstacles pr éc édemment obtenus sont exploit és ainsi qu'une approche de croissance de r égion. La m éthode propos ée est également test ée sur des donn ées r éelles.

  (a) Carnegie Mellon University 1 (b) The Stanford Racing Team 2

Figure 1 - 1 :

 11 Figure 1-1: Example of participants in 2003 Grand Challenge

Figure 1 - 2 :

 12 Figure 1-2: Google new prototype driverless car 3

Section 2 .

 2 5 deals with results and evaluation of the simulation tests and real data based experiments. Finally, a conclusion ends the chapter.

Figure 2 - 1 :

 21 Figure 2-1: The experimental platform. The fisheye camera is on the top bracket and the LRF is fixed in front of the vehicle

Figure 2 - 2 :

 22 Figure 2-2: Fisheye model

Figure 2 - 3 :

 23 Figure 2-3: Affine transformation between the sensor plane and the image plane

Figure 2 - 6 :

 26 Figure 2-6: Extrinsic calibration between the two sensors

Figure 2 - 7 :

 27 Figure 2-7: Framework of the proposed extrinsic calibration approach

Figure 2 - 8 :

 28 Figure 2-8: Corresponding points determination methodology

Figure 2 - 9 :

 29 Figure 2-9: First constraint. The laser beams plane is colored by blue. In camera coordinate system, any vector on this plane is perpendicular to the vector N v

Figure 2 - 10 :

 210 Figure 2-10: Performance w.r.t the number of known points in ideal case

Figure 2 - 11 :

 211 Figure 2-11: Performance w.r.t the number of known points in noisy case

RTFigure 2 - 12 :

 212 Figure 2-12: Comparison of calibration results under different noise levels

Figure 2 - 13 : 2 :

 2132 Figure 2-13: Two patterns used in convergence rate test

( a )

 a Approach in paper[START_REF] Mei | Single view point omnidirectional camera calibration from planar grids[END_REF] (b) Approach in paper[10]

Figure 2 - 14 :

 214 Figure 2-14: Point reprojection using the calibration results of the approaches proposed in[START_REF] Mei | Single view point omnidirectional camera calibration from planar grids[END_REF] and[10]

( a )Figure 2 - 15 :

 a215 Figure 2-15: Calibration results for real data considering 3 known points(a) or 8 known points (b)

Figure 2 - 16 :

 216 Figure 2-16: Projection of LRF measurements in the fisheye image

  (a) With LRF measurements only (b) With LRF measurements and color information obtained from fisheye image

Figure 2 -

 2 Figure 2-17: Estimation of motion trajectory under two different case

  and LRF information is proposed. It firstly derives the coarse road binary image by histogram based classification of the illumination invariant grayscale image. Then, a validation step is applied to check the coarse road binary image. Finally, a refined process based on HSI space is carried out. The rest of the chapter is organized as follows: Section 3.2 introduces the state of the art. Section 3.3 presents the general framework overview. Section 3.4 describes the coarse road detection based on log-chromaticity space. Section 3.5 introduces how HSI color feature and LRF are used to refine the coarse road detection results.

  scheme are employed to seek the vanishing point of road. Firstly, several vanishing points are picked out as candidates. For each candidate, it is estimated by a soft scheme voting strategy based on a local region defined by the authors. The voting strategy is based on texture orientation of the pixels in the local region and ratio coefficient decided by the diagonal length of the image and the distance between the pixels in the region and vanishing point. Each candidate has a confidence as estimation results. The candidate with the highest confidence is considered as the real vanishing point. And, finally, this vanishing point will work with the texture orientation to determine the road region in the image. However, this approach is not suitable for urban case because the boundary of road is often covered by parked or moving car. In paper[START_REF] Darms | Map based road boundary estimation[END_REF], Structure-From-Motion (SFM) is used to estimate a map-based road boundary model.

Figure 3 - 1 :

 31 Figure 3-1: The left image is original fisheye image, the right image is the road image which we aim to obtain.

Figure 3 - 3 :

 33 Figure 3-3: The context of the captured traffic scene is extracted from the original fisheye image.

Figure 3 - 4 :

 34 Figure 3-4: Left image is a color checker, right grahp represents the color in color checker under different illumination maps to log-chromaticity space. A set of different color surfaces under different illuminations form several parallel lines. l R θ is a line perpendicular to these parallel lines. The projection of log-chromaticity of pixels into l R θ form a 1-d illumination invariant image.

Figure 3 - 5 :

 35 Figure 3-5: Average entropy minimization for different values of the angle R θ

Figure 3 - 6 :Figure 3 - 7 :

 3637 Figure 3-6: The context of the captured traffic scene in RGB space is converted to illumination invariant grayscale image

Figure 3 - 8 : 3 . 5 . 1

 38351 Figure 3-8: Scattered pieces of road pixels are connected to form the so called road binary image.

Figure 3 - 9 :Fig. 3 -

 393 Figure 3-9: Framework of refined procedure

Figure 3 -

 3 Figure 3-10: ROI extraction

Fig. 3 -

 3 Fig.3-11 shows an example of the content of a RGB image in the H,S and I channels. As illustrated in this figure, it is known that: I (intensity) component has the most useful information, S (saturation) component has a few available information, H (Hue) component hardly provides useful information. Hue component represents color properties and only the content with great color change can be discriminated in

Figure 3 - 11 :

 311 Figure 3-11: The content of the image in H,S,I channels.

Figure 3 - 12 :

 312  The histograms for S,I and SI plane shown in[START_REF] Rotaru | Color image segmentation in HSI space for automotive applications[END_REF]

Figure 3 - 13 :

 313 Figure 3-13: Process of removing the false road parts in distance image

Figure 3 - 14 :

 314 Figure 3-14: Image is divided into equal intervals

Figure 3 - 15 :

 315 Figure 3-15: The configuration of the used experimental platform

Figure 3 - 16 :

 316 Figure 3-16: Experimental results. (Top row) Original image; (Middle row) Detection result; (Last Row) Ground truth.

Figure 3 - 17 :

 317 Figure 3-17: Experimental results. The images of the first row are original images, the second row are results obtained using the approach proposed in paper [25], the third are the results obtained by the proposed method in this paper

Section 4 .

 4 [START_REF] Zhang | Extrinsic calibration of a camera and laser range finder (improves camera calibration)[END_REF] gives the real data experimental results. Conclusions are presented in Section 4.7.

  to detect and track target vehicles ahead with the same running direction as the observer vehicle. Low level features such as corner, intensity and horizontal line are extracted firstly. Then, these features in each frame are projected vertically to form a 1-D profile. All consecutive 1D profiles along the time axis are linked to generate features motion trace image. In this image, a motion model based on hidden Markov model(HMM) is used to separate the background and target vehicle. This method is restricted to detect vehicles ahead with the same running direction as the observer vehicle. Besides, a proper motion probability model of target vehicle in HMM is hard to decide and the threshold for the separation of the line of background and target vehicle in 1-D profile is based on a experiential value. In paper[START_REF] Molineros | Monocular rearview obstacle detection using residual flow[END_REF], a method based on optical flow residual is used to detect the obstacle rear to the vehicle. It firstly find feature correspondence in consecutive frames. Then, these features are transformed to bird eye view (BEV). In the BEV image, they are classified into ground/nonground plane, and the features belonging to ground plane are used to estimate the ground plane ego-motion. Finally, based on the estimation of ground ego-motion, a residual motion map with respect to the ground is calculated. Using this map, the

Figure 4 - 1 :

 41 Figure 4-1: The challenging case: motion blur effect for the red vehicle in the image

Figure 4 - 2 :

 42 Figure 4-2: Processing flow chart for obstacle detection

Figure 4 - 3 :

 43 Figure 4-3: IPM coordinate system

Fig. 4 -

 4 Fig.4-4 shows an IPM example. The left image is the undistorted image of an original fisheye image, and the right image is the corresponding IPM image. As illustrated in Fig.4-4(b), the lane width is fixed and appears horizontal.

Figure 4 - 4 :

 44 Figure 4-4: IPM example: In IPM image, the borders of road appears parallel.

Figure 4 - 6 :

 46 Figure 4-6: Filter performance

Figure 4 - 7 :

 47 Figure 4-7: The red area in the image is the predefined fix area. The yellow line expresses the center of this area

Figure 4 - 8 :Figure 4 - 10 : 1 . 2 .

 4841012 Figure 4-8: WGS84 coordinate system

Fig. 4 -

 4 Fig.4-11 shows several nodes in OpenstreetMaps.

Figure 4 -Figure 4 - 12 :

 4412 Figure 4-11: OpenstreetMap. (a) The map under editor view. The red circle points are nodes. (b) The map in normal view

  interpolation nodes and road nodes are concatenated one by one along the traveling orientation to constitute the skeleton of the road ahead the vehicle (See Fig.4-12). The road delimiter model consists of a set of lines piecewise parallel to the road skeleton. The width of these piecewise parallel lines depends on the road width information estimated according to the road attributes in the database and additional countries national road legislation. The road model is shown in Fig.4-13.

Figure 4 -

 4 Figure 4-13: Road model

Figure 4 - 14 :

 414 Figure 4-14: Road model mapping

Figure 4 -

 4 Figure 4-15: (a) Red lines in the image I r shows the boundary of the road. Non-road pixels within the road boundary are conserved (b) Corresponding polar histogram

Figure 4 - 16 :

 416 Figure 4-16: The searching windows in normal view image

Figure 4 -

 4 Figure 4-17: (a) Original image (b) Results obtained with the approach of paper [66] (c) Results obtained with the approach of paper [67]. (d) Results obtained with our approach.

Fig. 4 -

 4 Fig.4-19 shows some examples of such failure case. As illustrated in the first row of Fig.4-19, the pedestrians around the road boundary, the proposed method can't

Figure 4 -

 4 Figure 4-18: (a) Original image (b)Results obtained with the approach of paper [66] (c)Results obtained with our approach.

Figure 4 -

 4 Figure 4-19: (a) error case (b)correct case

Figure 5 - 1 :

 51 Figure 5-1: Processing flow chart for object tracking

Figure 5 - 2 :

 52 Figure 5-2: Regularize procedure

3 Figure 5 - 3 :

 353 Figure 5-3: Encoding procedure in one color channel

Figure 5 - 4 :

 54 Figure 5-4: Example of searching area: the red rectangle is the tracking box, the green rectangle is the searching area

Figure 5 - 5 :

 55 Figure 5-5: Group of scanning grids

Figure 5 - 6 :

 56 Figure 5-6: LRF space. The intensity denotes the degree of corresponding measurement.

Figure 5 - 7 :

 57 Figure 5-7: Tracking box drifting from the target

7 Figure 5 - 8 :

 758 Figure 5-8: Test results in normal case

Figure 5 - 9 :

 59 Figure 5-9: Test results in motion blur case

Figure 5 - 10 :

 510 Figure 5-10: Test results in background change case

Figure 5 - 11 :

 511 Figure 5-11: Test results in the appearance and scale changes case.

Figure 5 - 12 :Figure 5 - 13 :

 512513 Figure 5-12: Test results in normal case

Figure 5 - 14 :

 514 Figure 5-14: Test results in motion blur, appearance change and scale change case

Figure 5 - 15 :

 515 Figure 5-15: Test results in over exposure case

Fig. 5 -

 5 Fig.5-18 and Fig.5-19 show some examples of visual comparison results. Above

Figure 5 - 17 :

 517  Comparison results between the proposed method and the approach presented in paper[START_REF] Kalal | Tracking-learning-detection[END_REF] in second video

Figure 5 - 18 :

 518 Figure 5-18: Examples of comparison experiment in first video

Figure 5 - 19 :

 519 Figure 5-19: Examples of visual comparison results (second video)

2. 1 (

 1 The configuration of the 8 poses used for applying the approach proposed in paper[START_REF] Mei | Calibration between a central catadioptric camera and a laser range finder for robotic applications[END_REF] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 The convergence time of three methods in solution optimization procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Average reprojection errors . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Calibration results obtained by the two approaches . . . . . . . . . . 3.1 Comparison of the proposed method performance in RGB nd HSI color space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Performance of road detection considering the proposed approach . . 3.3 The performance of road detection considering the method proposed in paper [25] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 The performance of the proposed approach with and without LRF measurements. 𝐻𝑖𝑡 𝑅𝑎𝑡𝑒 = 𝑇 ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑜𝑏𝑠𝑡𝑎𝑐𝑙𝑒𝑠 𝑇 ℎ𝑒 𝑡𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑔𝑟𝑜𝑢𝑛𝑑 𝑡𝑟𝑢𝑡ℎ 𝑜𝑏𝑠𝑡𝑎𝑐𝑙𝑒𝑠 𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑅𝑎𝑡𝑒 = 𝑇 ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑜𝑏𝑠𝑡𝑎𝑐𝑙𝑒𝑠 𝑇 ℎ𝑒 𝑡𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑜𝑏𝑠𝑡𝑎𝑐𝑙𝑒𝑠 . . . . . . . . . . . . . . . . . . . . 𝑥 𝑙 , 𝑦 𝑙 ) will be derived by the following equations: ⎧ ⎨ ⎩ 𝑥 𝑙 = 𝑥 𝑓 + 𝑛𝑒 -𝑠𝐿 𝑠𝑖𝑛(𝑠(𝛽 𝑛 -𝜎 0 )) 𝑦 𝑙 = 𝑦 𝑓 -𝑛𝑒 -𝑠𝐿 𝑐𝑜𝑠(𝑠(𝛽 𝑛 -𝜎 0 )) (A.7)

  5

  [START_REF] Chen | Object modelling by registration of multiple range images[END_REF] where 𝑁 𝑣 = [𝑁 𝑥 , 𝑁 𝑦 , 𝑁 𝑧 ] is the plane normal vector, 𝑃 𝑖 = [𝑋 𝑖 , 𝑌 𝑖 , 𝑍 𝑖 ] 𝑇 is any point on the plane and 𝑃 0 = [𝑋 0 , 𝑌 0 , 𝑍 0 ] 𝑇 a fixed point on the plane. In our case, the fixed point is regarded as the centroid point 𝑃 𝐶 = [𝑋, 𝑌 , 𝑍] 𝑇 of the known points set. For Given ∆𝑋 𝑖 = 𝑋 𝑖 -𝑋, ∆𝑌 𝑖 = 𝑌 𝑖 -𝑌 , ∆𝑍 𝑖 = 𝑍 𝑖 -𝑍, we can get the partial derivative𝑥 ∆𝑋 𝑖 + 𝑁 𝑦 ∆𝑌 𝑖 + 𝑁 𝑧 ∆𝑍 𝑖 )∆𝑋 𝑖 -2𝜆 𝑣 𝑁 𝑥 = 0 𝑥 ∆𝑋 𝑖 + 𝑁 𝑦 ∆𝑌 𝑖 + 𝑁 𝑧 ∆𝑍 𝑖 )∆𝑌 𝑖 -2𝜆 𝑣 𝑁 𝑦 = 0 𝑥 ∆𝑋 𝑖 + 𝑁 𝑦 ∆𝑌 𝑖 + 𝑁 𝑧 ∆𝑍 𝑖 )∆𝑍 𝑖 -2𝜆 𝑣 𝑁 𝑧 = 0

	Under normalization constraints of 𝑁 𝑣 (𝑁 2 𝑥 +𝑁 2 𝑦 +𝑁 2 𝑧 = 1), to use lagrange multipliers,
	the above equation is transformed to:
						𝑁
				𝑑 =	∑	𝑑 𝑖 -𝜆 𝑣 (𝑁 2 𝑥 + 𝑁 2 𝑦 + 𝑁 2 𝑧 -1)	(2.24)
						𝑖=1
								(2.25)
	of 𝑑:
		𝑁 ∑ 𝑖=1 𝑁 (𝑁 𝜕𝑑 𝜕𝑑 𝜕𝑁 𝑥 = 2 𝜕𝑁 𝑦 = 2 ∑ 𝑖=1 𝑁 (𝑁 𝜕𝑑 𝑖=1 𝜕𝑁 𝑧 = 2 ∑ (𝑁 (2.26)
	By grouping these equations into matrix form, we have:
	⎡	∑	∆𝑋 𝑖 ∆𝑋 𝑖	∑	∆𝑋 𝑖 ∆𝑌 𝑖	∑	∆𝑋 𝑖 ∆𝑍 𝑖
	⎢ ⎢ ⎢ ⎣	∑ ∑	∆𝑋 𝑖 ∆𝑌 𝑖 ∆𝑋 𝑖 ∆𝑍 𝑖	∑ ∑	∆𝑌 𝑖 ∆𝑌 𝑖 ∆𝑌 𝑖 ∆𝑍 𝑖	∑ ∑ ∆𝑌 𝑖 ∆𝑍 𝑖 ∆𝑍 𝑖 ∆𝑍 𝑖
	convenience, the equation 2.21 is rewritten as:
								(𝑃 𝑖 -𝑃 𝐶 )𝑁 𝑣 = 0	(2.22)
	The normal vector estimation problem can be converted the following minimization
	problem:
							𝑁
			𝑎𝑟𝑔𝑚𝑖𝑛	∑	𝑑 𝑖 (𝑑 𝑖 = ||𝑁 𝑣 (𝑃 𝑖 -𝑃 𝐶 )|| 2 )	(2.23)
							𝑖=1

Table 2 .

 2 .3. As illustrated in Table2.3, the performance of the Scaramuzza's approach is better than the performance of the other two method.

	Approach	Average reprojection errors (unit:pixel)
	Scaramuzza's approach	0.55
	C.Mei's approach	0.66
	Kannala's approach	1.2

3: Average reprojection errors

Based on these experiments, we find that the model proposed by Mei haven't a good ability to deal very well with corner extraction in great distortion case. The computation cost regarding intrinsic and extrinsic parameters optimization for the method proposed by Kannala is too expensive. The method proposed by Scaramuzza is a good option according to the test results with respect to convergence time, corner extraction and reprojection errors. Therefore, the model proposed by Scaramuzza is chosen in our approach.

Table 2 .

 2 From the simulation results, within this range, there are no big differences between the two outcomes.

	Approach		R		T(m)
	Approach proposed	0.034	-0.135	-0.990	1.413
	in this paper	0.999	0.019	0.032	0.309
		0.014	-0.991	0.135 -0.929
	Approach proposed	0.031	-0.140	-0.990	1.402
	in paper [7]	0.999	0.025	0.028	0.314
		0.020	-0.990	0.140 -0.921

8 known points and 8 poses are used respectively. The calibration results are shown in Table 2.4. From these results, we can see that the two methods give almost the same results. According to LMS211 technique manual, the average measurement error of LRF is about 2𝑐𝑚. 4: Calibration results obtained by the two approaches

  case, we have LRF measurements and image information. Combining the two types of information can make the threshold more robust. Let 𝐿 𝑑 denote dividing line (already define in conflict checking) in fisheye image, 𝐿 𝑏 the estimation of road boundary. In our case, 𝐿 𝑏 has two options. One depends on the road boundary of previous image. Generally, there is no drastic change of road boundary between two consecutive frames. The road boundary of previous image can be used to approximate the road boundary of the current frame. Another option is to estimate road boundary by LRF measurements. In practice, we predefine a line parallel to 𝐿 𝑑 and below it, 𝑑 𝜆

the distance between the two lines in vertical direction. This predefined line can be treated as an estimation of road boundary. The way to choose 𝐿 𝑏 obey the following principle: 𝐿 𝑏 is the road boundary of previous image if the position of road boundary of previous image is blow 𝐿 𝑑 in vertical direction; Otherwise, 𝐿 𝑏 is the predefined line. In our experiments, 𝑑 𝜆 is set to 30. In the distance image, one road pixel is considered as false road pixel if it is above 𝐿 𝑑 in vertical direction. If one pixel is located in the field between 𝐿 𝑑 and 𝐿 𝑏 and its value is less than a predetermined upper limit threshold 𝑇 2 , it is reserved as road. Similarly, if one pixel is below 𝐿 𝑏 and its

Table 3 .

 3

	Proposed approach

.14)

where 𝐵 𝑡 and 𝑅 𝑡 are the amount of correct background pixels and correct road pixels respectively, 𝑀 𝐼 is total pixel number. The average accuracy for each sequence is resumed in Table3.1. We can see that the proposed approach performs better in HSI than in RGB. 1: Comparison of the proposed method performance in RGB nd HSI color space

Table 3 . 2

 32 .2.𝑁 𝑖𝑚𝑎𝑔𝑒 denotes the number of images refined in the sequence. As shown, we that the amount of refined image in sequence 2 is more than the others. That is because there exist many buildings around the road. It makes the work to distinguish the sidewalk from road get harder.

			Propose approach
	Sequence	Acc	Type I Type II 𝑁 𝑖𝑚𝑎𝑔𝑒
	(number of images)
	1 (56)	0.9908 0.0057 0.0228	28
	2 (128)	0.9469 0.0122 0.1583	87
	3 (111)	0.9688 0.0252 0.0500	53
	4 (41)	0.9766 0.0180 0.0466	23

: Performance of road detection considering the proposed approach

Table 3 .

 3 3: The performance of road detection considering the method proposed in paper[START_REF] Alvarez | Road detection based on illuminant invariance[END_REF]

		Approach proposed in [25]
	Sequence	Acc	Type I Type II
	(number of images)
	1 (56)	0.9582 0.0080	0.1383
	2 (128)	0.8930 0.0227	0.3347
	3 (111)	0.9283 0.0188	0.2148
	4 (41)	0.9437 0.0180	0.0466

-17 compares some experimental results obtained by the proposed approach

Table 4 .

 4 1: The performance of the proposed approach with and without LRF measurements. 𝐻𝑖𝑡 𝑅𝑎𝑡𝑒 = 𝑇 ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑜𝑏𝑠𝑡𝑎𝑐𝑙𝑒𝑠 𝑇 ℎ𝑒 𝑡𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑔𝑟𝑜𝑢𝑛𝑑 𝑡𝑟𝑢𝑡ℎ 𝑜𝑏𝑠𝑡𝑎𝑐𝑙𝑒𝑠 𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑅𝑎𝑡𝑒 =

	Hit Rate Correct Rate

http://archive.darpa.mil/grandchallenge/Teams/stanfordracing.html

http://www.dotwnews.com/focus/connected-cars-the-latest-motoring-innovations# close_subscription

Mots-cl és : cam éra fish-eye, t él ém ètre laser, GPS, calibrage, d étection de route, d étection d'obstacle, suivi d'objet.

the captured traffic scene, and the remaining part is useless black area (see Fig. 3-3).

The useless area, having side effect on solving illumination invariant grayscale image, is firstly removed. grayscale image is computed from the mapping of the image from RGB space to moving obstacles around vehicle can be identified. Generally, road plane has very few texture features and is difficult to extract many saliency features. If feature points are not sufficient, it is hard to estimate precisely the entire road ego-motion. Other researchers utilize stereo vision [START_REF] Bertozzi | GOLD: a parallel real-time stereo vision system for generic obstacle and lane detection[END_REF] [55] [START_REF] Oniga | Processing dense stereo data using elevation maps: Road surface, traffic isle, and obstacle detection[END_REF] to estimate the position of obstacles. In paper [START_REF] Labayrade | Real time obstacle detection in stereovision on non flat road geometry through "v-disparity" representation[END_REF], the authors construct a V-disparity image based on stereo image pair to detect obstacles on the road. The drawback of this method is that it assumes road is dominant along the image rows and it can be sensitive to roll angle changes. In paper [START_REF] Oniga | Processing dense stereo data using elevation maps: Road surface, traffic isle, and obstacle detection[END_REF], the authors suggest to detect obstacles based on the density of digital elevation map. But this method can't discriminate the sidewalk around free road area. In practice, it is a dangerous case.

In paper [START_REF] Rebai | Moving obstacles detection and tracking with laser range finder[END_REF], a LRF is utilized for obstacles detection task. LRF is independent to illumination change. It is an active sensor which can provide reliable and high accuracy range measurements. The geometrical figures consisting of LRF measurements are employed to represent vehicles, and predict their location using an extended Kalman filter (EKF). However, in fact, it's hard to describe all kind of objects around the vehicle merely using a few simple geometrical shapes. In paper [START_REF] Weiss | Robust driving path detection in urban and highway scenarios using a laser scanner and online occupancy grids[END_REF], obstacles are detected using an occupancy grid map which is transformed into regularly spaced grid of cells. Nevertheless, the resolution of obstacles depends on the grid map. The higher resolution grid map is, the more memory it takes.

In paper [START_REF] Monteiro | Tracking and classification of dynamic obstacles using laser range finder and vision[END_REF], both LRF and camera are used to detect and recognize objects in front of a vehicle. The LRF measurements provide the regions of interest (ROI), and a classifier based on support vector machine (SVM) is applied to recognize the content in ROI. Our research is belonging to this line.

Nevertheless, all of the methods described above don't consider the motion blur case. If motion blur emerges in image, many salient features (corner, SURF, FAST) will become invalid, and put the relevant methods in failure. Most of motion blur stems from objects movement in a scene during a long time exposure. In weak lighting scene, the long time exposure of camera often occurs. It means motion blur has a high probability to appear in this case.

Chapter 5

Objects tracking based on small-region growth

Introduction

In this chapter we will present a novel object tracking algorithm. Because of the constant fluctuation of the outside world, it often brings many difficulties for outdoor object tracking. One of problems is hard to get high quality images. For low quality images, many visual based methods may become invalid due to the missing of features information of tracked objects. To address this sort of issue, a method based on LRF measurements and raw pixel information is proposed. It makes use of raw pixel information to locate interesting small regions firstly. And then these small regions are mapped to LRF space. In this space, each small region will growth to a complete region which covers most part of the corresponding target through some growth decision factors. The proposed method is tested in two long-term video which contains 1000 frames. The rest of the chapter is organized as follows: Section 5.2 introduces the state of the art. Section 5.3 presents the framework of the proposed method. Section 5.4 describes in details the initialization step for the method in details. Section 5.5 introduces the frame buffer information function. Section 5.6 presents the way to encode the tracked objects. Section 5.7 describes how to locate interesting small regions. Section 5.8 introduces how to expand an interesting small region to become Online self-learning descriptor is based on online self learning classifier. It firstly trains the classifier by several previous frame samples, after which it estimates the target position in image. Using the new estimated sample in current frame, the classifier is updataed. The method in paper [START_REF] Grabner | Semi-supervised on-line boosting for robust tracking[END_REF] is based on this technique. However, this method often suffers from drift problem due to accumulation errors in the tracking procedure. In paper [START_REF] Kalal | Tracking-learning-detection[END_REF], the authors use multi instances to address this problem.

The conventional self-learning usually only uses positive samples. Multi-instances based self-learning proposed by the authors also adopt negative samples around the positive ones. This method can help classifier discriminating non-targets around a target.

All above mentioned methods are based on a relatively high quality image. However, in our case, motion blur of tracked objects may emerge in image. Only visual based method may not lead to a robust result. To cope with this problem, a LRF space and image based method is proposed in our work.

Overview of the Proposed Algorithm

Framework overview

The framework of the proposed tracking algorithm is shown in Fig. 5-1. It takes previous tracked target's LRF measurements and previous image with annotated targets by tracking boxes as starting point. For the first frame, target is annotated with a tracking box manually, and its LRF measurements are picked out by the same way. In the initialization step, the image content in the tracking box is regularized into standard size and the tracked target's LRF measurements are used to calculate key information which are sent to frame information buffer. After that, in object representation step, the standard size image content is encoded by a method based on weighted RGB values. By this sort of code, in the current undistorted fisheye image, a list of patches is launched to determine an interesting small region that partially contains the target object. And then, a growth strategy based on current Appendix A Transformation from WGS84 to Extended Lambert II Given (𝛼 𝑔0 , 𝛽 𝑔0 ) the latitude and longitude from GPS receiver, the coordinate (𝑥 𝑙 , 𝑦 𝑙 ) in extended Lamber II system can be calculated by the following steps:

1) Transform (𝛼 𝑔0 , 𝛽 𝑔0 ) from decimal degree format to degree/radians format (𝛼 𝑔 , 𝛽 𝑔 ).

2) Convert (𝛼 𝑔 , 𝛽 𝑔 ) to cartesian coordinate (𝑥 𝑤 , 𝑦 𝑤 , 𝑧 𝑤 )

we have the following equations:

3) Translate (𝑥 𝑤 , 𝑦 𝑤 , 𝑧 𝑤 ) to Cartesian coordinate NTF (Novelle Triangulation de la 123 France)(𝑥 𝑛 , 𝑦 𝑛 , 𝑧 𝑛 )

Given:

where 𝑎 𝑛 = 6378249.2, 𝑏 𝑛 = 6356515 and

(𝛼 𝑛 , 𝛽 𝑛 ) can be derived by the following algortihm:

Algorithm for calculating (𝛼 𝑛 , 𝛽 𝑛 )

Calculate 𝜆 1 with Eq.A.5.

End

Then, 𝛼 𝑛 = 𝜆 1 and 𝛽 𝑛 = 𝑎𝑡𝑎𝑛( 𝑦𝑛 𝑥𝑛 )

5) Calculate (𝑥 𝑙 , 𝑦 𝑙 ) through (𝛼 𝑛 , 𝛽 𝑛 )

Appendix B Abstract

Road scene understanding is one of key research topics of intelligent vehicles. This thesis focuses on detection and tracking of obstacles by multisensors data fusion and analysis. The considered system is composed of a lidar, a fisheye camera and a global positioning system (GPS). Several steps of the perception scheme are studied: extrinsic calibration between fisheye camera and lidar, road detection and obstacles detection and tracking.

Firstly, a new method for extinsic calibration between fisheye camera and lidar is proposed. For intrinsic modeling of the fisheye camera, three models of the literature are studied and compared. For extrinsic calibration between the two sensors, the normal to the lidar plane is firstly estimated based on the determination of ń known ż points. The extrinsic parameters are then computed using a least square approach based on geometrical constraints, the lidar plane normal and the lidar measurements.

The second part of this thesis is dedicated to road detection exploiting both fisheye camera and lidar data. The road is firstly coarse detected considering the illumination invariant image. Then the normalised histogram based classification is validated using the lidar data. The road segmentation is finally refined exploiting two successive road detection results and distance map computed in HSI color space.

The third step focuses on obstacles detection, especially in case of motion blur.

The proposed method combines previously detected road, map, GPS and lidar information. Regions of interest are extracted from previously road detection. Then road central lines are extracted from the image and matched with road shape model extracted from 2DŋSIG map. Lidar measurements are used to validated the results.

The final step is object tracking still using fisheye camera and lidar. The proposed method is based on previously detected obstacles and a region growth approach.All the methods proposed in this thesis are tested, evaluated and compared to stateŋofŋtheŋart approaches using real data acquired with the IRTESŋSET laboratory experimental platform.

Keywords: fisheye camera, laser ranger finder, GPS, extrinsic calibrationn, road detection, obstacle detection, object tracking.