Keywords: Intelligent transportation systems, Public dataset, Multi-modality, Fusion, Object detection, UWB radar, Entropy, Segmentation, Deep learning, LSTM ADAS . . . . . . . . . . . . . . . . . . . detection . . . . . . . . . . . . . . . 2.3.3.2 Lidar sensor-based object detection . . . . . . . . . . 2.3.3.3 Radar sensor-based object detection . . . . . . . . . . 2.3.4 Multi-modal based systems for road obstacle detection . . . . . 2.3.4.1 Data fusion methods for automotive applications . . . 2.3.4.2 Fusion using camera and lidar data . . . . . . . . . . 2.3.4.3 Fusion using camera and thermal camera data . . . . . 2.3.4.4 Fusion using radar and camera data . . . . . . . . . . 2.4 Performance evaluation metrics for intelligent transportation systems . . 2.5 Conclusion . . . . . . . . . . . . .

Building reliable environment perception systems is a crucial task for autonomous driving, especially in dense traffic areas. Researching in this field is evolving increasingly. However, we are at the beginning of a research pathway towards a future generation of intelligent transportation systems. In fact, challenging conditions in real-world driving circumstances, infrastructure monitoring, and accurate real-time system response, are the predominant concerns when developing such systems. Recent improvements and breakthroughs in scene understanding for intelligent transportation systems have been mainly based on deep learning and the fusion of different modalities. In this context, firstly, we introduce OLIMP 1 : A heterOgeneous MuLtimodal Dataset for Advanced EnvIronMent Perception . This is the first public, multimodal and synchronized dataset that includes Ultra Wide-Band (UWB) radar data, acoustic data, narrowband radar data and images. OLIMP comprises 407 scenes and 47,354 synchronized frames, including four categories: pedestrians, cyclists, cars and trams. The dataset presents various challenges related to dense urban traffic such as cluttered environments and different weather conditions. To demonstrate the usefulness of the introduced dataset, we propose, afterwards, a fusion framework that combines the four modalities for multi object detection. The obtained results are promising and spur for future research. In short range settings, UWB radars represent a promising technology for building reliable obstacle detection systems as they are robust to environmental conditions. However, UWB radars suffer from a segmentation challenge: localizing relevant Regions Of Interests (ROIs) within its signals. Therefore, we put froward a segmentation approach to detect ROIs in an environment perception-dedicated UWB radar as a third contribution. Specifically, we implement a differential entropy analysis to detect ROIs. The obtained results show higher performance in terms of obstacle detection compared to state-of-theart techniques, as well as stable robustness even with low amplitude signals. Subsequently, we propose a novel framework that exploits Recurrent Neural Networks (RNNs) with UWB signals for multiple road obstacle detection as a deep learning-based approach. Features are extracted from the time-frequency domain using the discrete wavelet transform and are forwarded to the Long short-term memory (LSTM) network.

1 https://sites.google.com/view/ihsen-alouani/datasets

The obtained results show that the LSTM-based system outperforms the other implemented related techniques in terms of obstacle detection.

R E S U M É

Une perception fiable de l'environnement est une tâche cruciale pour la conduite autonome, en particulier dans les zones de trafic dense. La recherche dans ce domaine évolue de plus en plus. Cependant, nous sommes au début d'une voie de recherche vers une future génération de systèmes de transport intelligents. En effet, les principales préoccupations lors du développement de tels systèmes sont les conditions de la conduite, la surveillance des infrastructures et la réponse précise du système en temps réel. Les récentes améliorations et percées dans la compréhension de l'environnement pour les systèmes de transport intelligents reposent principalement sur l'apprentissage profond et la fusion de différentes modalités. Dans ce contexte, tout d'abord, nous introduisons OLIMP: A heterOgeneous MuLtimodal Dataset for Advanced EnvIronMent Perception 1 . C'est la première base de données public, multimodale et synchronisée qui comprend des données radar ultra large bande (ULB), des données acoustiques, des données radar à bande étroite et des images. OLIMP comprend 407 scènes et 47 354 données synchronisées, dont quatre catégories: piétons, cyclistes, voitures et tramways. L'ensemble de données présente divers défis liés au trafic urbain dense, tels que des environnements encombrés et des conditions météorologiques différentes. Pour démontrer l'utilité de la base introduite, nous proposons, par la suite, un framework de fusion qui combine les quatre modalités pour la détection multi-objets. Les résultats obtenus sont prometteurs et incitent à de futures recherches. Dans les applications à courte portée, les radars ULB représentent une technologie prometteuse pour la construction de systèmes de détection d'obstacles fiables car ils sont robustes aux conditions environnementales. Cependant, ces radars souffrent d'un défi de segmentation: localiser les régions d'intérêt (ROIs) pertinentes dans ses signaux. Par conséquent, nous mettons en avant une approche de segmentation pour détecter les ROIs dans un environnement dédié à la perception de l'environnement c'est la troisième contribution. Plus précisément, nous mettons en oeuvre une analyse d'entropie différentielle pour détecter les ROIs. Les résultats obtenus montrent des performances supérieures en termes de détection d'obstacles par rapport aux techniques de l'état de l'art, et une robustesse même avec des signaux de faible amplitude. Par la suite, nous proposons un nouveau framework basée sur l'apprentissage profond qui exploite le réseau de neurones récurrents avec les signaux ULB pour la détection multiple d'obstacles routiers. Les caractéristiques sont extraites du domaine temps-fréquence à l'aide de la transformée en ondelettes discrète et sont transmises au réseau récurrent à mémoire courte et long terme. Les résultats obtenus montrent que le système basé sur l'LSTM surpasse les autres techniques implémentées en termes de détection d'obstacles. Threshold definition based on exhaustive space exploration.

L I S T O F F I G U R E S

The threshold corresponds to the highest F1-score. . . . . . . . Figure 4. [START_REF] Behnam Behroozpour | Lidar system architectures and circuits[END_REF] Experimental results: Precision, Recall and F1-score using HOS, CA-CFAR, [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . . . . . . . . . . . . . . . Figure 4. [START_REF] Beltrán | Birdnet: a 3d object detection framework from lidar information[END_REF] Experimental results for pedestrian detection using HOS, CA-CFAR, [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . . . . . . . . . . . . . . . . . Figure 4. [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF] Illustration of multiple-target detection limitations using HOS, CA-CFAR and Entropy detectors . . . . . . . . . . . . . . . . Figure 4. [START_REF] Bijelic | Seeing through fog without seeing fog: Deep sensor fusion in the absence of labeled training data[END_REF] Precision results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR, [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure 4. [START_REF] Blake | OS-CFAR theory for multiple targets and nonuniform clutter[END_REF] Recall results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR, [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . . . Figure 4.13 F1-score results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR, [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Characteristics of different object detection methods . . . . . . Table 2. [START_REF] Moeness | Understanding deep neural networks performance for radar-based human motion recognition[END_REF] Summary of fusion approaches for obstacle detection . . . . . Table 3. [START_REF] Ahmed | Hand Gesture Recognition Using an IR-UWB Radar with an Inception Module-Based Classifier[END_REF] Categorization of some autonomous driving datasets by task . Table 3. 2 Overview of some autonomous driving datasets . . . . . . . . Table 3. [START_REF] Moeness | Understanding deep neural networks performance for radar-based human motion recognition[END_REF] Sensor specifications and properties. Measure latency is the time necessary to collect one complete data stream from the sensor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Table 4. [START_REF] Ahmed | Hand Gesture Recognition Using an IR-UWB Radar with an Inception Module-Based Classifier[END_REF] Comparison between CW-radars and IR-UB radars [START_REF] Hussein | Signal Processing of UWB Radar Signals for Human Detection Behind Walls[END_REF] . . . . The main human capacities are the power and adaptability of our brain to process and interpret data acquired from our senses. While the human visual cortex is precise in processing the received information, humans can classify and locate complex objects that surround them by characterizing the objects' shapes, colors and orientations. Furthermore, humans are able to continuously analyze their environment and scan for new objects in their surroundings. With the speedy ascension of Artificial Intelligence (AI), researchers aim to develop intelligent systems that efficiently imitate the human perception. To be accurate and reliable, these systems must consider several constraints that are spontaneously handled by the human brain. Autonomous vehicles (AVs) are one of the most important developed Intelligent Transportation Systems (ITSs) for the recent years. These vehicles require to perceive their environments like humans. Thus, the environment perception task is considered as one of the prominent fields of research.

1.1

Traffic congestion has been increasing worldwide, which is caused mainly by the expansion of number of road users. This leads to a rising number of road accidents and fatalities, which definitely threats the quality of urban life.

Currently road injuries is estimated to be the seventh leading cause of death for all age groups globally, and it is ranked first for people aged between 10 and 24 years old [START_REF] Vos | Global burden of 369 diseases and injuries in 204 countries and territories, 1990-2019: a systematic analysis for the Global Burden of Disease Study 2019[END_REF]. According to the World Health Organization (WHO) [139], road traffic crashes generate a loss of over than 1.35 million lives per year, where 54% are vulnerable road users. Moreover, they cause non-fatal injuries for almost 50 million people all over the world each year. From another point of view, these road traffic causalities account for 23% of all injury deaths worldwide in accordance to the distribution related to the global injury mortality via cause, as illustrated in figure 1.1 [START_REF] Peden | World report on road traffic injury prevention[END_REF]. Furthermore, economically, road injuries engender considerable economic losses since they cost most countries around 3% of their gross domestic product [139].

Based on the statistics reported by the National Highway Traffic Safety Administration (NTHSA) [138] in 2018, 94% of the road accidents are principally linked to human errors. These errors are caused mainly by fatigue, heedlessness and immature behavior (as immature driving, distraction when using the phone, etc.) according to information about crashes reported by the police [138]. For these reasons, AVs have been introduced in order to import an eloquent change for drivers, road users, infrastructure and pollution to contribute to fuel savings. Despite the fact that AVs aim to avoid accidents and provide economic, comfortable and intelligent driving, the development of such systems is highly complex. This is due to the permanent changes of the vehicle's environment that includes weather and illumination conditions, as well as the type of the obstacles and the behavior of road users. However, with the recent improvements and breakthroughs of AI, which relies on the appearance of machine and deep learning algorithms, as well as the availability of new sensor modalities, AVs achieve huge progress. Thus, sophisticated driver assistance technologies have been developed to ensure road safety [START_REF] Supriya | Review on autonomous vehicle challenges[END_REF] such as collision avoidance, driver assistance and driver behavior monitoring systems [START_REF] Jegham | Safe Driving : Driver Action Recognition using SURF Keypoints[END_REF] [75] [START_REF] Ben Khalifa | Pedestrian detection using a moving camera: A novel framework for foreground detection[END_REF].

In fact, most of the proposed solutions require robust environment perception, especially in urban circumstances, and rely on the object detection process to prevent accidents and to protect, inter alia, vulnerable road users such as pedestrians and cyclists. To achieve such requirements, object detection exploring sensors' fusion remains essential because the AV perception capability can exceed that of human drivers particularly in degraded conditions.

As the employed sensors offer complementary data and their collaboration can guarantee a better scene understanding and since training deep learning algorithms requires huge datasets, developing accurate and robust object detectors is a primary challenge in the AV field. A long list of companies are interested in developing AVs and aim to launch highly automated vehicles by improving software capabilities and safety, like Audi, Google, Bosch, Nvidia, etc. Despite the fact that AVs have shown immense progress during the last few years, there have been 13 serious car crashes using the autonomous mode or the autopilot mode [35]. Some of these crashes are shown in figure 1.2. On February 14th, 2016, the Google's AV commits its first crash with a bus while lane changing [START_REF]Crash Google[END_REF]. One of the most critical of these serious accidents took place when the Tesla's Model X car driving in an autopilot mode had a collision with a highway barrier in California in 2018 [START_REF]Crash Tesla[END_REF]. This collision resulted in killing the driver. Accordingly, with the implication of Tesla with a number of other fatal crashes, the National Transportation Safety Board (NTSB) declared that both autopilot and driver errors were factors in these car accidents. Also, the Hyundai self-driving car crashed because of rain [START_REF] Hyundai | [END_REF].

Google's car

Tesla's car Uber's car Based on these reported fatal crashes that involve self-driving cars, the autonomous driving in urban roads remains an open and a challenging problem. In fact, the environmental variables, which vary from weather conditions to the surrounding human behaviors, are extremely indeterministic and difficult to predict. For these reasons and since system failures lead to catastrophic accidents and fatalities, the improvements of object detectors and the development of new algorithms are still an inevitable process.

For the detection stage, most of the companies use either a unique sensor or a combination of modalities such as the camera, the lidar, the radar, etc. While cameras are limited to bad weather conditions, the radar is robust and provides additional information about 1.3 5 the objects' characteristics (as the distance and the velocity) which can reduce fatal errors. Thus, the combination of various sensor types can make a significant enhancement.

Datasets are decisive for researchers and developers as most of the tools and algorithms have to be tested and then trained before functioning on the road. A typical practice consists in testing and validating the developed algorithms on annotated datasets.

In this context, various autonomous driving datasets have been published in order to enhance reasarch for environment perception such as Kitti [START_REF] Geiger | Are we ready for autonomous driving? the kitti vision benchmark suite[END_REF], Kaist Multi-Spectral [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF], nuScene [START_REF] Caesar | nuscenes: A multimodal dataset for autonomous driving[END_REF], etc. Most of these datasets are multimodal, combining different heterogeneous modalities. While some of the existing datasets use the narrow-band radar, UWB carries richer information. The UWB radar provides a signal that results from the reflection of a transmitted UWB pulse on the object. The deformation of the initial wave represents the signature of the object. This signature contains information about the distance, the material and the shape of the object. In fact, the UWB radar offers a huge interest in shortrange applications, but its employment addresses a main challenge which distinguishes the target over noise and static clutter.

From another point of view, different objects have distinguishable acoustic signatures that may help recognize each of them. In spite of the usefulness of the acoustic data, we notice that none of the-state-of-the-art ITSs benchmarks use the acoustic modality. Moreover, nowadays, the employment of deep learning remains unavoidable when developing object detection systems due to the achieved success in this area of research. Therefore, in order to increase accuracy and provide tangible improvements, we have to cope with their complexity. Further, training such algorithms needs huge amounts of data. For these reasons and motivated by the fact that current databases lack in radar and acoustic sensors adoption, the development of new ones is required.

1.3

In this thesis, we propose a new multi-object detection framework by exploring either uni-modality or multi-modality via developing a multi-modal dataset for advanced urban environment perception.

In order to achieve the thesis aim, the contributions of this work are as follows:

1.4

1. The first contribution consists in developing OLIMP (A HeterOgeneous MuLtimodal Dataset for Advanced EnvIronMent Perception) with over than 47, 000 samples. It is a new heterogeneous dataset collected using a camera, a UWB radar, a narrow-band radar and a microphone. To the best of our knowledge, this is the first dataset that includes UWB signals and acoustic data including several challenges related to dense urban traffic. Thus, potential advancement could be accomplished in the environment perception area using our dataset.

2. Our second contribution consists in proposing a new fusion framework that combines data acquired from the different sensors used in the introduced dataset in order to achieve better performances for the obstacle detection task. This fusion framework demonstrates the usefulness of the introduced dataset and it mainly highlights the importance of multimodality in environment perception.

3. A segmentation approach to detect ROIs in an environment perception-dedicated UWB radar is our third contribution. Specifically, we implement a differential analysis of the entropy of UWB signals to detect ROIs. We evaluate our technique on OLIMP. The obtained results show higher performance in terms of obstacle detection compared to the implemented state-of-the-art techniques, as well as stable robustness even with low amplitude signals.

4. The fourth contribution is a novel framework that exploits RNNs with UWB signals for the detection of multiple road obstacles. We evaluate our approach on the OLIMP dataset. The obtained results show that the system outperforms the other implemented related techniques in terms of obstacle detection by learning the temporal relationship between the data sequences.

1.4

This document is organized as follows.

In chapter 2, the state of the art is presented by firstly putting the topic of object detection and environment perception into context with the autonomous driving. In addition, the exploited sensors and the relative main challenges will be detailed. Afterwards, we review road object detectors using uni-modal and multi-modal systems by explaining the possible fusion levels.

1. 4 7 Chapter 3 presents the developed dataset. This chapter goes further into the implementation details in terms of employed sensors, synchronisation setup, annotation process and relative challenges. In addition to this, a fusion framework exploiting different fusion levels is presented to highlight the potential enhancement that can be achieved using the introduced dataset.

The following chapter goes deeper into investigating the UWB radar. In chapter 4, two UWB-based detectors are proposed. The first detector aims to segment obstacles within UWB signals via an entropy-based approach. Regarding the second framework, it is a deep-based detector that takes advantage of the LSTM network to distinguish noise from real targets. A comparative study with the state-of-the-art techniques is conducted, and the obtained results are discussed in this chapter.

Finally, chapter 5 presents the conclusion of this thesis and some perspectives.

2.1

In this chapter, we firstly present the state of the art relative to environment perception systems including road object detection for intelligent and AVs. The relevant sensors and the fusion challenges are described. Afterwards, a detailed overview of object detection methods is exposed. This chapter is organized as follows. Section 2 presents an environment perception background that includes the Advanced Driver Assistance System (ADAS) applications, the employed sensors and the challenges related to the vehicle's surroundings perception task. Section 3 exhibits a review of object detectors from one uni-modal system to multimodal systems for road obstacles. The performance evaluation metrics of the detectors are detailed in Section 4. Finally, section 5 refers to the conclusion of this chapter.

2.2

According to the WHO, every day, around 3,700 people are killed in road traffic crashes and over than half of those are vulnerable road users: pedestrians, cyclists, motorcyclists, drivers [139]. Existing surveys point out that the human errors are one of the principal causes of road accidents. These errors can be distinguished in fatigue, heedlessness and immature behaviors. For this reason, it becomes critical to equip vehicles with safety systems to provide security to drivers and vulnerable road users.

Aside from these mentioned consequences, road accidents have additional undesirable side effects. They are identified as a considerable cause of energy consumption and air pollution [START_REF] Lowson | Energy use and sustainability of transport systems[END_REF]. Furthermore, people waste countless hours in the urban traffic environment.

Addressing the aforementioned issues of safety, efficiency and pollution remains a primordial concern to guarantee a better life quality. Therefore, the development of intelligent vehicles is a viable solution to the mentioned problems in order to ensure security by avoiding accidents and providing an economic, comfortable and intelligent driving.

Recently, AVs receive worldwide attention thanks to the considerable advancement and progress that have been achieved in this field of research. These improvements are made on account of the prompt advances constructed in information, electronics and 2.2 10 communications technologies, and by employing essentially AI. In fact, an AV is a car that moves safely and takes decisions by sensing its surroundings with little or no human intervention in real traffic conditions. Some basic characterizations for automation levels are currently set according to the Society of Automotive Engineers (SAE) [START_REF]Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles[END_REF] for better understanding of selfdriving cars. Different levels of automation are defined and they cover degrees from no automation to complete control. Otherwise, the higher level is the more the vehicle's monitoring responsibilities. According to the SAE international's J3016, there exist 6 levels describing the state of automation for a vehicle [START_REF]Level of automation[END_REF]. These levels are presented as below and described in figure 2.1.

ADAS Higher levels of driving systems

• Level 0: It is labeled as no automation and the driver entirely controls the vehicle continually.

• Level 1: It is known as 'feet off'. The automated system and the driver all together control the car. Automatic parking and the automatic breaking are given as examples. Besides, the driver must be always prepared to retake total control any time.

• Level 2: It is entitled 'hands off' and the automated system is responsible for accelerating, breaking and steering the car. The driver monitors the driving and must be ready to rapidly intervene if the system fails to respond correctly.

11

• Level 3: It is called 'eyes off'. At this stage, the driver can ignore safe driving task, like texting, working and even watching a movie. The car handles situations that require an instantaneous response such as the emergency breaking. However, the driver still must be willing to intervene in limited time fixed by the manufacturer.

• Level 4: It is known as 'minds off'. At this level, the driver cedes the full control to the vehicle and is not expected to control it at any time. For example, the driver can safely sleep.

• Level 5: It is entitled 'steering wheel optional', and the human intervention is not required (e.g. a robotic taxi). A typical autonomous driving framework can be categorized into three main stages: perception, planning and control. The mentioned stages and the car's interactions with the environment are depicted in Figure 2.2.

Environment

The perception process is responsible of accurately perceiving the vehicle's surroundings through suitable sensors. Environmental perception aims to understand the environment, by positioning the obstacles, detecting the road signs, and categorizing the acquired data. Localization refers to the determination of the vehicle's position compared to the environment. The planning process makes decisions in order to achieve autonomous driving goals, such as avoiding obstacles.

Finally, the control stage refers to the execution of the planned actions generated by the planning process.

Advanced driver-assistance systems: ADAS

The supporting and assessment of the driver in critical conditions are provided by ADAS applications. Literally, these applications present extra traffic information, an overview of the driver's behavior, and environment information to guarantee an efficient performance.

In fact, the advancement of driver assistance systems was set out at the end of the 70's through an Anti-lock Braking System (ABS) embedded into a serial production [START_REF] Ziebinski | A survey of ADAS technologies for the future perspective of sensor fusion[END_REF]. Afterwards, the improvement steps in this area can be distributed to three classes: network sensors, proprioceptive and exteroceptive sensors. As regards to the proprioceptive sensors, they are responsible for detecting danger situations and then responding through analyzing the vehicle's behavior. Concerning exteroceptive sensors, like the radar, ultrasonic, lidar and vision sensors, they are capable of responding on a prior stage and predicting dangers [START_REF] Kuutti | A survey of the state-of-the-art localization techniques and their potentials for autonomous vehicle applications[END_REF]. The employment of these sensors ensures ADAS applications. Some of these ADAS applications are detailed below and illustrated in figure 2.3.

• Blind spot detection: It observes the vehicle's adjacent area. Thus, it warns the driver of the existence of obstacles in the blind spot, via a visual sign in the side view mirror or through an audible signal.

• Road cross traffic alert : It helps avoiding accidents when the driver reverses out of parking. This functionality provides a visible signal and an audible warning if an object is detected in the driving direction's reverse.

• Traffic sign assist : It automatically recognizes traffic signs, even the signs of other countries. Therefore, safer and relaxed driving is provided.

• Lane Departure Warning : It is in charge of scanning the edges of the road and detecting when the car is deviating the lane of the road. Thus, the driver will be warned via an hyptical signal as a steering wheel vibration or through a visual warning.

• Emergency Brake Assist : It is an active braking support that automatically brakes on critical situations, so, rear-end collisions will be refrained absolutely and pedestrians will be protected.

• Adaptive Cruise Control : It it is able to control the distance from the car ahead.

Moreover, it alerts the driver or slows the vehicle's speed if the respective distance becomes inadequate (too small).

Sensing Modalities for intelligent transportation systems

Sensors collect information about road conditions, and vehicle's surrounding can be categorized into two classes: active and passive. Active sensors diffuse signals, and based on the reflected signal it identifies targets, such as the radar and the lidar. Passive sensors acquire data without diffusing, and cameras are the most widely used example.

• Radar

Nowadays, the radar is well exploited in many fields such as mapping, meteorology and especially in the area of automation [START_REF] Sujeet Milind Patole | Automotive radars: A review of signal processing techniques[END_REF]. In fact, the principal goal of a radar system is to detect the existence of one or more targets in the area of interest. A radar simultaneously transmits and receives electromagnetic waves in frequency bands between 3 MHz and 300 GHz, and it extracts information (range, position, velocity) using the reflected EM waves from the targets [START_REF] Sujeet Milind Patole | Automotive radars: A review of signal processing techniques[END_REF]. It is robust against fog, rain, bad weather and lightning conditions (day and night). Furthermore, the automotive radar systems can be divided into three classes: short range radars that are mostly employed for parking assist, medium range radars used for rear collision avoidance, and long range radar utilized for adaptive cruise control [START_REF] Holger | Radar sensors in cars[END_REF].

• Lidar

Light detection and ranging known as lidar, is considered as one of the dominant technologies in the field of AVs. It is fixed on the roof of the car and it spins regularly. Lidar is a laser scanner that provides a 360-degrees of visibility of the vehicle's environment and measures the range from 1m to 60m depending on the sensor. Besides, it illuminates the target with a pulsed laser light and measures the reflected pulses through a sensor [START_REF] Behnam Behroozpour | Lidar system architectures and circuits[END_REF]. The sensor outputs point cloud data that include the position (x, y, z coordinates) of the objects and their intensity information which indicates the object reflectively. There are three major lidar types used in the autonomous driving field: 2D lidar, 3D lidar and solid state lidar. This sensor is expensive and sensible to weather conditions such as fog and snow owing to the diffraction of light within these circumstances.

• Camera

Cameras are considered as the eyes of the vehicle through outfitting the car with several cameras at all angles. In fact, two versions of visible cameras are used in this field which are mono and stereo vision [START_REF] Ziebinski | A survey of ADAS technologies for the future perspective of sensor fusion[END_REF]. Fora mono vision camera, it is usually employed to understand the essential surroundings as detecting the speed limit signs or lane marking. Concerning stereo vision cameras, they are analogous to human eyes and provide two video sources. In fact, the utilization of such a technology helps the vehicle to understand the traffic flow and obstacles' positions. In addition to mono and stereo vision, there exists a night vision system, which adopts infrared cameras. Although cameras are sensible to lighting variations and weather conditions, they are the only sensors capable of detecting color, texture and contrast information [START_REF] Rosique | A systematic review of perception system and simulators for autonomous vehicles research[END_REF].

• Ultrasonic

It is a low-cost sensor that sends sound waves in high frequency in order to determinate the object's distance. Actually, it is widely used in this field to detect near obstacles; however, it is affected by noise and interference [START_REF] Hyo | Low-cost curb detection and localization system using multiple ultrasonic sensors[END_REF].

The aforementioned sensors are highlighted in figure 2.4 to highlight the benefits and limits of each sensor, Table 2.1 summarized the mentioned sensors' characteristics.

Multi-modal environment perception challenges

The environment perception stage is the first and most primordial process for automated driving. It provides the vehicle with decisive information on the driving environment over time. Thus, the vehicle should determine its position in order to correctly interrupt the data acquired from the perceptional sensors. This task is known as localization. Afterwards, the vehicle senses its surroundings through the employed sensor. Robust detection and classification of stationary and moving obstacles is essential to correctly perceive the environment. Furthermore, the surrounding objects' positions and velocities are determined and tracked during this stage, and even future states can be predicted.

Thereby, the development of a reliable perception system remains a challenging task, because the car must perceive its surroundings in real world situations that include uncontrolled and complex scenarios such as urban environment. Figure 2.5 illustrates a complex urban scenario using multi-modal sensors, and including multiple relevant traffic participants.

Accordingly, the current challenges related to the environment perception are caused by the complex outdoor environments that include different road agents, as well as the presented requirement to develop efficient algorithms for real-time perception. In addition, variable lighting and adverse weather conditions, uncontrolled backgrounds Figure 2.5: Complex urban environment for autonomous driving [START_REF] Feng | Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges[END_REF] and the occlusion of multiple objects remain critical challenges for the perception process for intelligent vehicles.

On account of the aforementioned challenges and as a small error can engender fatal accidents, the environment perception system should be:

(i) accurate: It must to provide precise information about the driving environment.

(ii) robust: It should work accurately in adverse weather, under various situations and circumstances, and even when the sensor degraded.

(iii) real-time: It should have a real-time response particularly when driving at a high speed.

To achieve achieving these goals, the environment perception system takes advantage of multi-modal sensors to completely perceive the vehicle surroundings. As already indicated, the most utilized sensors in the automation field are: cameras and lidar and radar sensors. In fact, every sensor has its typical advantages and disadvantages, as mentioned in section 2.2.2 so each of these sensors can be employed in different situations. Thus, a comparative study of the most used sensors according to the autonomous requirements is presented in figure 2.6. As depicted from the figure, it becomes obvious that all the requirements for autonomous driving can not be ensured by an individual sensor type. However, a combination of two sensors or more can achieve good results for the environment perception task. Hence, the fusion of various sensing modalities permits exploiting their complementary properties.

2.3

As mentioned in Chapter 1, in our work we focus particularly on one of the fundamental environment perception dilemmas which is object detection. Object detection points out the identification of the objects of interest locations and determines their sizes. This section discusses the state of the art of object detection and especially the work related to object detection for ADAS applications.

Recent developments in object detection

The state of the art of object detection can be practically divided to pre and postdeep learning appearance. In fact, preceding the progress of deep learning approaches, the literature were principally based on hand crafted features such as Histogram of Oriented Gradients (HOG) or the scale-invariant feature transform (SIFT), which are employed with traditional classification methods like the Support Vector Machine (SVM), Adaboost, etc. Some major pre-deep learning contributions made in the object detection field are listed in the following.

• Cascade of weak classifiers : It is one of the basic approaches that was proposed by Viola and Jones [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF]. Haar features are extracted, and Adaboost with cascade classifiers are used for object detection. The proposed technique is based mainly on the sliding-window principle.

• HOG : In [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF], HOG features were introduced by Dalal and Triggs, and they are based on the edge directions within the image. SVM classifier was utilized for object classification.

• Deformable Parts Model (DPM) : It is a graphical model proposed by Felzenszwalb et al. Actually, the DPM introduced to face objects' deformation in the image [START_REF] Felzenszwalb | Object detection with discriminatively trained part-based models[END_REF]. DPM is based on the fact that each object is composed of its parts.

• Selective Search (SS) : This method was put forward by Uijlings et al. [START_REF] Jasper Rr Uijlings | Selective search for object recognition[END_REF].

The SS technique generates independent object proposals. In fact, the input image is segmented into different scales, which generates several ROIs. A similarity comparison based on the color, size, texture, etc. is made to merge the redundant regions and set will be suggested afterwards.

Nowadays, with the significant advancement of employing deep learning, a relevant impact on the filed of object detection is made. It is considered a powerful tool as it can learn hierarchical features for large amounts of data. Correspondingly, several methods have been proposed to tackle the object detection dilemma. The state of the art related to deep object detection can be divided in two categories: the two-stage or one-stage pipelines. The two-stage network is also known as the region-based technique and the one-stage is called the single shot object detector.

• Region-based object detection network:

In the primary stage, several ROIs are extracted and are considered as object candidates. Afterwards, these region candidates are confirmed and classified. Then, classification scores and locations are refined. The pioneering work that utilized deep learning for object detection was OverFeat [START_REF] Sermanet | Overfeat: Integrated recognition, localization and detection using convolutional networks[END_REF] and R-CNNs (Regionbased ConvNetworks) [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF]. Indeed, ROIs are firstly generated using the sliding window technique for OverFeat and selective search for R-CNNs. Subsequently, the suggested ROIs are processed via CNNs to extract relevant features for the classification and the regression of the bounding boxes. For the SPPnet (Spatial Pyramid Pooling networks) [START_REF] He | Spatial pyramid pooling in deep convolutional networks for visual recognition[END_REF] and Fast-RCNN [START_REF] Girshick | Fast r-cnn[END_REF] networks, the features are directly generated from feature maps by employing a larger CNNs on the entire image (e.g. Resnet [START_REF] He | Deep residual learning for image recognition[END_REF], VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], GoogleLeNet [START_REF] Szegedy | Going deeper with convolutions[END_REF]). In Faster R-CNNs [START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], the object detection pipeline is unified and a region proposal network is introduced for generating region proposals. Following this line, in [START_REF] Dai | R-fcn: Object detection via region-based fully convolutional networks[END_REF] the R-FCN (Region-based Fully Convolutional Networks) was put forward, which was a fullyconvolutional network. In fact, the fully-connected layers of the RPN are replaced 2.3 20 with convolutional layers. Run time and accuracy are increased for object detection performances.

• Single shot object detection network:

This type of model proposes to straightly match the feature maps with the bounding boxes and classification scores via a unified CNN network. For example, YOLO (You Only Look Once) [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF] models the object detection task as a regression dilemma where the object proposal generation process is eliminated. The bounding boxes are directly regressed from the CNN. The SSD (Single Shot Detection) [START_REF] Liu | Ssd: Single shot multibox detector[END_REF] treats objects with different sizes via regressing numerous feature maps of various resolutions. Small convolutional filters are used for predicting multi-scale bounding boxes.

Generally, the region-based object detectors like the Faster-RCNN tend to reach higher detection accuracy owing to the generation of various region proposals and the refinement model. This comes with a higher execution time, high power computation and a more complicated training process. In contrast, the one-stage networks are faster and can be optimized easier, which makes them well-suited for real-time applications. Yet, in terms of accuracy, they are under-performed in comparison to the two-stage object detectors [START_REF] Huang | Speed/accuracy trade-offs for modern convolutional object detectors[END_REF].

Object detection in ADAS

Dynamic environments contain various static and moving obstacles that interact with each other. Therefore, distinguishing these objects is an essential task of the perception environment stage. In fact, the development of object detection systems for autonomous driving is specifically challenging as it has high requirements in terms of real-time, accuracy, and robustness performances. The results obtained from object detection are commonly transferred to the other units such as decision making. Thus, a reliable object detection system is a prerequisite for safe driving under complex and uncontrolled driving environments [START_REF] Wei | Enhanced object detection with deep convolutional neural networks for advanced driving assistance[END_REF].

Object detection in ADAS faces various challenges that are related to the complex driving environment, which includes a cluttered background and various road-agents. These agents have different types (traffic signs, pedestrians, cars, motorcyclists, etc) with various sizes. Moreover, many obstacles can be occluded, so occlusion adds additional challenges to the object detection task. Likewise, bad lighting and weather conditions still affect the detection performances significantly. Furthermore, when developing object detection systems, it is essential to consider some critical aspects. For the input data, the questions that remain are: Are there any available multi-modal or uni modal datasets? Are the data of high quality and labeled?. Furthermore, several important questions should be answered: Which modalities should be used or combined? How are required data represented and how can we process them correctly?. Accordingly, which fusion methods can be applied and at which stage can we have an accurate and reliable object detection system? Correspondingly, various challenges related to object detection when developing ADAS should be considered .

Uni-modal based systems for road obstacle detection

Complex driving situations often present various obstacles. Some works has focused on 2D detection, while some others deal with 3D object detection, which includes more challenges thanks to the development of complex datasets. To address this challenge, the use of a single modality or a combination of various ones has been adopted in the literature. In this section, we summarize various existing techniques for uni-modal based systems for obstacle detection.

Vision sensor-based object detection

One of the essential sensors used for observing the vehicle's surroundings is the camera, so, computer-vision based approaches have been widely employed for obstacle detection. Some research has focused on using only one camera while others have employed several cameras to obtain multi views of the vehicle surroundings and accurately detect the obstacles.

Existing vision-based methods can be distinguished into three categories: stereovision, classical-based and deep learning based methods. The stereo-based methods use two cameras that generate a depth map due to their capacity of 3D scene reconsecration. Accordingly, the objects are segmented within the depth map relative to their spatial locations. These approaches are able to detect various obstacles with different shapes and various motion statuses. Moreover, they can accurately determine the distance and the 3D geometric size [START_REF] Yang | Multiclass obstacles detection and classification using stereovision and Bayesian network for intelligent vehicles[END_REF].

Classic visual obstacle detection approaches employ hand-crafted descriptors such as the HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF], Aggregated Channel Features (ACF) [START_REF] Dollár | Fast feature pyramids for object detection[END_REF] or the Integral Channel Features (ICF) [START_REF] Dollár | Integral channel features[END_REF]. A pedestrian detector was proposed in [START_REF] Rakesh Nattoji Rajaram | Looking at pedestrians at different scales: A multiresolution approach and evaluations[END_REF], and features were extracted from numerous image scales using the ACF. The approach consisted in training several models using multiple resolution. Finally, the bounding boxes generated by each model were concatenated. In fact, an improvement of 6 % was achieved in terms of average precision over the employment of single resolution. A vehicle proposal location framework was introduced in [START_REF] Yuan | A graph-based vehicle proposal location and detection algorithm[END_REF]. The suggested algorithm was a graph-based method that generates accurate region vehicle proposals compared with traditional approaches. The experiments were carried out on the PASCAL VOC2007 and the Kitti dataset. In [START_REF] Liu | Reversing obstacle detection based on binocular vision image[END_REF], obstacle regions were segmented via a specific threshold to obtain binary images. Yet, this technique depended on the defined threshold, so some obstacles were not well segmented and noise also affected the obstacle detection process. The V-disparity algorithm has been used in various studies for obstacle detection [START_REF] Chen | Monocular 3d object detection for autonomous driving[END_REF] [54] [START_REF] Musleh | Uv disparity analysis in urban environments[END_REF]. Nevertheless, this approach is sensitive to large objects that will influence the detection of small objects. The latest classical object detector is the DPM which has achieved significant improvements in the object detection area. Yet, its detection accuracy is still limited for driving object detection and its computation complexity remains very high. In [START_REF] Yebes | Visual object recognition with 3D-aware features in KITTI urban scenes[END_REF], HOG features were incorporated with disparity maps via a modified DPM. The disparity maps were determined from the stereo images using the semi-global matching method. In [START_REF] Safat B Wali | An automatic traffic sign detection and recognition system based on colour segmentation, shape matching, and svm[END_REF], a traffic sign detection system was put forward, which included three processes: image preprocessing, detection and recognition. An RGB color segmentation was adopted and followed by a shape matching technique. Then, SVM was employed as a classifier. A vehicle detection system that relied on a stereo vision classifier was proposed in [START_REF] Neumann | Online vehicle detection using Haar-like, LBP and HOG feature based image classifiers with stereo vision preselection[END_REF]. The fusion Haar, the Local Binary Patterns (LBP) and the HOG features provided good detection results. In [START_REF] Hariyono | Moving object localization using optical flow for pedestrian detection from a moving vehicle[END_REF], an ROI obstacle was defined based on the distinctive characteristics of the optical flow of the background over the optical flow of the obstacle region. Despite the fact that promising results have been found using the classical techniques for object detection, these methods tend to fail in numerous complex driving environments that comprise different objects' sizes and types, and critical challenges.

While classical object detectors get stuck in the bottleneck, there are a wide deep learning models that boost research for visual object detection. In [START_REF] Xiang | Subcategoryaware convolutional neural networks for object proposals and detection[END_REF], a novel RPN was suggested which exploited subcategory information to lead the region proposal process. The input of the RPN was a pyramid image where it was processed via various convolution and pooling layers to generate for every scale one feature map. Afterwards, a convolutional layer was added for the subcategory object detection, where every filter would fit with a specific subcategory. Correspondingly, the ROI generation layer would use a thresholding technique on the heat maps to provide the object candidates. Cai et al. [START_REF] Cai | A unified multi-scale deep convolutional neural network for fast object detection[END_REF] proposed a multi-scale CNN (MS-CNN) in order to handle the wide variation of the objects' size. The MS-CNN performed the object detection over various scales of feature layers, that lead to an improved detection rates with a speed of 15 frames per second. The authors in [START_REF] Yang | Exploit all the layers: Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers[END_REF] employed scale dependent pooling to exploit suitable convolutional features related to the sacle of the object candidate in order to improve the detection accuracy. Furthermore, cascaded rejection classifiers utilized the convolutional features and excluded the negative object candidates using the cascaded manner to speed up the detection. The combination of the two contributions achieved better detection accuracy on Kitti, PASCAL and Inner-city datasets. A Recurrent Rolling Convolution (RRC) network was proposed in [START_REF] Ren | Accurate single stage detector using recurrent rolling convolution[END_REF], inspired by the SSD architecture. The pedestrian and cyclist detection accuracy achieved higher performances than the state of the art on the Kitti dataset. Nevertheless, the RRC was a complex model with a high computation time. In [START_REF] Wei | Enhanced object detection with deep convolutional neural networks for advanced driving assistance[END_REF], visual object detection was guaranteed using three enhancements for the CNN. They include a deconvolution and a fusion of the CNN feature maps to obtain deeper features. Moreover, a soft non maximal suppression was adopted to address the occlusion challenge. The Kitti dataset was exploited in this work. Zhang et al. proposed in [START_REF] Zhang | A cascaded R-CNN with multiscale attention and imbalanced samples for traffic sign detection[END_REF] a cascaded R-CNN to extract pyramids that represent the weighted multi-scale features using the dot product and the softmax to enhance the traffic sign accuracy detection. The authors in [START_REF] Rateke | Road obstacles positional and dynamic features extraction combining object detection, stereo disparity maps and optical flow data[END_REF] used the mask R-CNN to detect the object and the optical flow method was utilized to analyze the detected object movements.

Lidar sensor-based object detection

Compared to vision based detection, reliable depth information is provided by a lidar point cloud. The lidar point cloud is a set of points represented in space and corresponds to a 3D object or shape. Moreover, 3D voxels can be generated via grouping the neighbor points onto sets, so the computational cost is reduced. These data can be exploited to accurately detect 3D objects and distinguish their shapes. For this reason, lidar-based object detection has a deceive role for the visual perception system of ADAS to tackle the camera-related limitations.

The object detection using lidar point clouds faces some challenges such as the sparsity of the lidar points, the high variability of the point density, the occlusion, the corresponding pose variation and the non-uniform sampling related to the 3D space [START_REF] Jiao | A survey of deep learning-based object detection[END_REF].

The existing works on lidar-aided object detection methods can be distinguished into three categories [START_REF] Weng | Monocular 3d object detection with pseudolidar point cloud[END_REF]:

(i) Projection-based methods: Several studies have focused on the projection of the lidar points onto the bird's view images. Then, 3D bounding boxes are regressed according to the extracted features from these images [START_REF] Beltrán | Birdnet: a 3d object detection framework from lidar information[END_REF] [176]

(ii) Point-voxel methods: Lidar data are represented as 3D voxels, and 3D CNNs are applied for the prediction of 3D bounding boxes [START_REF] Luo | Fast and furious: Real time endto-end 3d detection, tracking and motion forecasting with a single convolutional net[END_REF] [181]

(iii) Pointnets-based methods: Lidar points are processed directly through neural models without a pre-processing step [87] [158].

In the following, we briefly review some existing work that has focused on object detection from the point cloud. Classical methods using lidar data for object detection have used clustering algorithms in order to segment the point clouds and assign the obtained groups to multiple classes [START_REF] Vaquero | Low cost, robust and real time system for detecting and tracking moving objects to automate cargo handling in port terminals[END_REF]. The Vote3Deep modal was introduced by Engelcke et al. [START_REF] Engelcke | Vote3deep: Fast object detection in 3d point clouds using efficient convolutional neural networks[END_REF] based on sparse convolutional layers. L1 regularisation was adopted in this work for adequate processing of the 3D lidar data. Other work has exploited some handcrafted features associated to the spatial relations between the segmented points as fast-point-feature histogram [START_REF] Tao | Automatic apple recognition based on the fusion of color and 3D feature for robotic fruit picking[END_REF] and the pin images [START_REF] Michael S Foster | Spin-image target detection algorithm applied to low density 3D point clouds[END_REF] were extracted to identify the categories of clusters. PointNet [START_REF] Charles R Qi | Pointnet: Deep learning on point sets for 3d classification and segmentation[END_REF] and PointNet++ [START_REF] Ruizhongtai | Pointnet++: Deep hierarchical feature learning on point sets in a metric space[END_REF] were introduced for processing the point sets and achieved good results in indoor environments. Nonetheless, with the expanding amount of point cloud data involved in developing 3D object detectors, computational power and huge memory requirements increased. In [START_REF] Minemura | LM-Net: Real-time multiclass object detection on CPU using 3D LiDAR[END_REF], the point cloud was represented under five views: the distance, the size, the reflection, the range and the height. These views were the CNN inputs. The projection of the 3D data onto 2D views generated the loss of important information that was critical for the detection task, particularly in congested scenes. An efficient deep-learning-based network, entitled VoxelNet, was put forward in [START_REF] Zhou | Voxelnet: End-to-end learning for point cloud based 3d object detection[END_REF]. Features were directly extracted from sparse points within the 3D voxel grid, so, remarkable results were achieved on the Kitti dataset. However, the detection accuracy rate decreased slightly as the distance between the voxels in the grid and was smaller than the distance of 3D lidar data. In addition, various voxel-based methods have been proposed such as SECOND [START_REF] Yan | Second: Sparsely embedded convolutional detection[END_REF] and PointPillar [START_REF] Lang | Pointpillars: Fast encoders for object detection from point clouds[END_REF].

Although lidar-based approaches can achieve remarkable object detection performances, these methods require high resolution and a precise and available lidar point cloud. Furthermore, the lidar has typically blind areas when detecting objects closer to the vehicle.

Radar sensor-based object detection

Radar sensors provide rich information about the vehicle's surroundings on account of the received signals and the characteristics extracted from these signals such as: the amplitude, the range, the velocity, the Doppler spectrum, etc. The acquired radar data depend on the sensor type and its characteristics. In fact, it can be depicted by 2D feature maps. Next, it can be processed via a CNN or a point cloud, or it can be represented as 1 dimensional (1D) signals that can be treated, afterwards, to detect the obstacles within these signals.

Various object detection methods firstly cluster the radar targets according to their characteristics (e.g. range, velocity, azimuth) to a group of object proposals. Next, these clusters are classified. In [START_REF] Prophet | Pedestrian classification with a 79 GHz automotive radar sensor[END_REF], DBSCAN [START_REF] Ester | Density-based clustering algorithms for discovering clusters[END_REF] was used to cluster the radar targets. Subsequently, various cluster-wise features were extracted as the variance of the velocity. The performance of numerous classifiers was compared for pedestrian detection as the SVM and the Random Forest (RF). The authors in [START_REF] Schumann | Comparison of random forest and long short-term memory network performances in classification tasks using radar[END_REF] also used DBSCAN to cluster different targets. Then the LSTM and the RF were compared for the multi-class (pedestrian, car, truck, etc.) detection task.

Wile clustering-based approaches are widely employed, it is often observed that objects can be wrongly merged or can be split apart. The performance depends essentially on the initial clustering step that relies on the definition of the suitable parameters that must be utilized for all classes (as the radius of the cluster using DBSCAN). Furthermore, small objects may be ignored using such methods.

To tackle these challenges, some researchers have suggested to classify each target separately rather than in clusters. In [START_REF] Schumann | Semantic segmentation on radar point clouds[END_REF], inspired by the results achieved using then lidar point cloud, the authors proposed to process radar data via PointNet++ [START_REF] Ruizhongtai | Pointnet++: Deep hierarchical feature learning on point sets in a metric space[END_REF] to detect dynamic objects. Yet, a single radar frame datum is too sparse.

On the other hand, others have chosen to extract features from the acquired radar data for classification. The Radar Cross-Section (RCS) values were obtained using different frequencies, and the angles of simulated and measured obstacles have constituted the feature vectors. Consequently, SVM was used for the classification of indoor targets [START_REF] Travis | Radar classification of indoor targets using support vector machines[END_REF]. The received amplitude, the Doppler spectrum and the range characteristics were extracted from the radar received signals. An SVM was employed to classify pedestrians and cars [START_REF] Heuel | Pedestrian recognition based on 24 GHz radar sensors[END_REF]. A similar approach was suggested in [START_REF] Heuel | Two-stage pedestrian classification in automotive radar systems[END_REF].

Despite the fact that deep learning techniques have achieved remarkable advancement in various fields of research, only few studies have employed radar data with deep learning methods. In fact, to apply such networks, radar data require a pre-prossecing step where the radar reflections are represented as an image-like data [START_REF] Schumann | Semantic segmentation on radar point clouds[END_REF]. For static object detection, occupancy grids were used as inputs for CNNs in [START_REF] Lombacher | Potential of radar for static object classification using deep learning methods[END_REF] [START_REF] Lombacher | Object classification in radar using ensemble methods[END_REF]. Kim et al. [START_REF] Kim | Moving target classification in automotive radar systems using convolutional recurrent neural networks[END_REF] opted for utilizing convolutional Recurrent Neural Networks (RNNs) for moving object classification via range-velocity maps. The range-velocity images are obtained by transforming the time-series radar data by adopting the 2D discrete fourier transform. The obtained results show that the LSTM-based network is able to learn the dynamics of the lateral movements related to the vulnerable road users in the time-series radar images. In [START_REF] Moeness | Understanding deep neural networks performance for radar-based human motion recognition[END_REF], the spectrogram extracted from the time-frequency signals was represented as 2D images. These images fed the stacked auto-encoders for extracting high-level radar features. The authors in [START_REF] Kim | YOLO-Based Simultaneous Target Detection and Classification in Automotive FMCW Radar Systems[END_REF] transformed the acquired radar data to range-angle representation. Following this, YOlO was trained on the transformed data. The system accuracy reached 90% on a self-recorded dataset. Table 2.2 summarizes the shortcomings of the aforementioned methods of object detection according to their different characteristics.

Generally, it is challenging to utilize only data acquired from a single sensor in a complex environment, specially for critical applications. The reasons can be caused by the sensor's shortages, the sensed environment, or both. Perceptional sensors suffer from various limitations and inadequacies, which can degrade the detection performance. Apart from the imperfections of sensors, the challenging environment conditions have a huge impact on their outputs as weather circumstances (eg. rain, fog) or illumination conditions (e.g. nighttime, low light). Besides, each sensor is suitable for a specific use case, scenario and application. For instance, the radar provides certain information about

Detection mode Detection method Detection characteristics

Computer-vision Binary threshold

Cannot completely segment obstacles the obstacle's distance, but its sparse data require a constructive processing step, otherwise objects cannot be detected. Concerning the vision-based system, it can accurately detect objects as pedestrians, but it is time-consuming. Additionally, using lidar data can guarantee impressive performance, but this sensor is still easily affected by rain, fog and dust.

UV-disparity

Analyzing the advantages and limitations of employing each sensor separately, we observe that the sensors mentioned above are complementary for the object detection task. In other words, each sensor compensates the limits of the other sensor. Therefore, based on these reasons, the need for employing multiple sensors and merging the acquired data remain essential to enhance the efficiency of environment perception tasks.

Multi-modal based systems for road obstacle detection

Since incomplete and unreliable information can result in fatal driving situations due to the challenging driving environment, combining data collected from disparate sensors remains a good solution to upgrade the system's overall detection performance. Even though merging information obtained from numerous sensing modalities is a challenging task, thanks to advanced sensor technologies and the progress in data processing algorithms, with the improvements in hardware, the fusion is becoming realizable.

In this section, an overview of the adopted fusion strategies in the autonomous driving field is presented and some multi-modal perception systems for the obstacle detection task are reviewed.

Data fusion methods for automotive applications

The combination of data acquired from various modalities is of great interest. In this regard, most existing work fuses RGB images with lidar point clouds. In addition, some further work couples RGB images with thermal ones. However, we highlight that there is a lack of research on combining radar data with images.

In fact, for object detection, there exist two predominant categories when fusing various modalities: hand-crafted feature methods and deep-neural-network approaches. Furthermore, the fusion of sensing modalities can be achieved at three possible stages: early, intermediate or late. These levels are detailed in figure 2.8. For simplicity, the sensing modalities are restricted to two.

• Low level

The low-level fusion is also known as an early or signal level. During this level, the raw data acquired from multiple sensors are directly combined in order to obtain merged data that can be used for successive tasks. For instance, the lidar depth map is combined with the color camera data to define an RGB-D format that is processed afterwards. This level has a low memory budget and low computation requirements. However, it is sensitive to data misalignment among the employed sensors, which can be caused by different sampling rates or calibration errors.

• Intermediate level

The intermediate level or the medium-level fusion intends to extract features from several data collected via multiple sensors. These features are, then, combined into a feature vector that serves as the input for the subsequent process. An example is the extraction of features from RGB images as HOG features and from the depth map independently. These features are concatenated to a single feature vector. Although this fusion type enables the system to learn different feature representations at several depths, it is not straightforward to identify an optimal way to couple them for a specific architecture.

• Late level

The late or high-level fusion is as known as decision level. The obtained decisions that process each datum from different sources separately are combined to define the final decision. For instance, combining the detected bounding boxes obtained from the object detection algorithm processed on the RGB and depth map separately determines the final detections using the voting method, for example. The late fusion level is highly flexible; e.g. when an additional sensing modality is introduced, it does not affect the overall architecture. Nevertheless, it suffers from high memory and computation cost requirements.

In addition, there exists multi-level fusion (known as the hybrid-level). This level focuses on the integration of various data at different levels of abstraction.

Moreover, based on the literature, five fusion operations are mainly used to fuse multiple modalities based on a deep architecture [START_REF] Feng | Deep multimodal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges[END_REF]: 1) Addition, 2) Average mean, 3) Concatenation, 4) Ensemble: It is used to combine the ROIs for object detection, 5) Mixture of Experts: This operation tends to modal explicitly the weights of the feature maps.

It should be pointed out that there is no evidence that confirms that one fusion level is better than the others. The performances are extremely dependent on the data, the employed modalities and the network.

In the following, some multi-modal perception systems that exploit fusion are reviewed. A summary of these approaches is presented in Table 2.3.

Fusion using camera and lidar data

Multiple studies have proved that fusing images with lidar data improves the accuracy of the object detection process, particularly for far ranges and small obstacles [START_REF] Arnold | A survey on 3d object detection methods for autonomous driving applications[END_REF]. There are three techniques to combine lidar point clouds with camera images. Firstly, the results obtained from training images and lidar points separately are merged. Secondly, the targets are detected using camera images. Afterwards, the confirmation of the results is provided using the lidar point clouds. Finally, the third method consists in defining ROIs utilizing lidar data, and the camera is used to detect the objects.

Conzalez et al. [START_REF] González | On-board object detection: Multicue, multimodal, and multiview random forest of local experts[END_REF] used transformed depth maps and RGB images as inputs to detect pedestrians. In this work, the objects' poses in multi view were taken into account, and In [START_REF] Vishwanath | MVX-Net: Multimodal voxelnet for 3D object detection[END_REF], a point fusion method was proposed where lidar points were mapped onto the image plane and features were extracted from the image using a pre-trained 2D detector. Afterwards, features were concatenated via a VoxelNet architecture. The authors in [START_REF] Bijelic | Seeing through fog without seeing fog: Deep sensor fusion in the absence of labeled training data[END_REF] suggested an architecture based on two single-stage detectors. The information provided by lidar data (height, distance, intensity) was transformed into images. These latter with RGB images, were the inputs. These data were processed by VGG16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] to provide features. Afterwards, an SSD [START_REF] Liu | Ssd: Single shot multibox detector[END_REF] network was adapted to generate the bounding boxes of 2D cars in foggy weather based on a deep feature exchange that relied principally on feature concatenation. In the work of Xu et al. [START_REF] Xu | Pointfusion: Deep sensor fusion for 3d bounding box estimation[END_REF], the raw data acquired from lidar were proceeded by a PointNet architecture, and images features were extracted via the CNN. The obtained results were then pooled in order to locate the coordinates of the 3D bounding boxes. Qi et al. [START_REF] Charles R Qi | Frustum pointnets for 3d object detection from rgb-d data[END_REF] adapted a similar approach in their work. In the work of [START_REF] Oh | Object detection and classification by decisionlevel fusion for intelligent vehicle systems[END_REF], object proposals were generated using a segmentation method applied on the data of lidar point clouds and RGB images. After that, the candidates generated from lidar data and images trained two separate CNNs in order to classify the proposals. The output decisions were combined using a basic belief assignment to associate the bounding boxes. Then a CNN model was implemented to determine the final decision along with the SVM.

Fusion using camera and thermal camera data

Even though visual cameras are affected by weather and lighting conditions, thermal cameras are robust to nighttime and daytime circumstances because they detect the object's heat reflected by the infrared radiation. For this reason, the combination of the provided data can ensure detailed scene understanding as they are correlated in terms of illumination conditions. Hwang et al. [START_REF] Hwang | Multispectral pedestrian detection: Benchmark dataset and baseline[END_REF] introduced an extension of the ACF dedicated for pedestrian detection.

The extended modal consisted of a multispectral ACF obtained from the augmentation of the thermal intensity and the employment of HOG features as additional ones. In [START_REF] Khalid | Person detection by fusion of visible and thermal images using convolutional neural network[END_REF], visible and thermal images were fused according to two approaches in the intention to detect people. The first consisted in encoding the two types of images using independent encoders, and the encoded features are merged and then decoded back in order to generate a single fused image that would be the input of a Residential Network (ResNet) architecture. This technique was called DenseFuse. The second proposed method was an intermediate level fusion technique. Indeed, ResNet-152 was employed separately for infrared and visual images. Thereafter, the extracted features were concatenated into a single array that would serve as the input of the fully connected layer.

Early fusion and late fusion based on the CNN architecture to couple infrared and visible images were investigated in [START_REF] Wagner | Multispectral Pedestrian Detection using Deep Fusion Convolutional Neural Networks[END_REF]. The early fusion method consisted in combining the pixels captured from the two modalities, in contrast to late fusion where two sub-networks would provide a feature representation for both modalities. These representations were fused on a supplementary fully connected layer. Besides, the proposals were generated using the ACF+THOG detector. According to the obtained results, a pre-trained late fusion method evaluated on KAIST multispectral dataset guaranteed better performances. In [START_REF] Li | Illumination-aware faster R-CNN for robust multispectral pedestrian detection[END_REF], an illumination-aware architecture was proposed based on the Faster R-CNN [START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF].

Infrared and visible images were respectively the inputs of two separate sub-networks. Meanwhile, an illumination aware network was developed to estimate an illumination value from color images. Thereafter, an illumination weight layer is integrated in order to determine the fusion weights for both modalities. Consequently, the final decision was achieved by weighting the final results obtained from the two sub-networks due to the estimated fusion weights.

2.3 32

Fusion using radar and camera data

For obstacle detection, the radar and the camera are two complementary sensors, but only a few studies have addressed this challenge. Similar to the other kinds of sensing combinations, the three types of fusion can be applied to couple these modalities.

In [START_REF] Wang | Onroad vehicle detection and tracking using MMW radar and monovision fusion[END_REF], radar tracks generated the ROIs in the images. Afterwards, for the vision module, a symmetry algorithm and a contour detection technique were applied to the ROIs to identify vehicles. The goal of the work presented in [START_REF] Bouain | Multi-sensor fusion for obstacle detection and recognition: A belief-based approach[END_REF] was to detect pedestrians. The radar sensor provided a list of tracks and the ACF object detector was adopted to generate a list of identified pedestrians in the images. Subsequently, the fusion of the obtained descions was ensured using the Dempster Shafter method. Wang et al. [START_REF] Wang | Vehicle detection and width estimation in rain by fusing radar and vision[END_REF] proposed a decision approach to fuse radar data and images. The YOLO [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF] network is employed in this work to detect vehicles from visible images. The radar sensor detects the centroid of the obstacles. Afterwards, these detections were projected on the image plane. Finally, the results obtained from the two modalities are combined. A real-time Radar Region Proposals network (RPNP) was developed in [START_REF] Nabati | RRPN: Radar Region Proposal Network for Object Detection in Autonomous Vehicles[END_REF]. The suggested network consisted in generating ROIs based only on radar detection. In fact, the tracks are mapped into images so that anchor boxes are proposed, which are inspired by fast R-CNN architecture.

Then, these boxes are scaled according to the distance of the objects to have accurate detection. Radar data are transformed into images in [START_REF] Chadwick | Distant vehicle detection using radar and vision[END_REF] in order to be combined with RGB images. Actually, these data will be proceeded via the ResNet network separately. Accordingly, features are concatenated after the second block of ResNet.

To fuse different modalities for understanding the vehicle surroundings, many approaches have employed deep neural network architectures, while others are based on hand crafted features. From the aforementioned reviewed studies, we observe that the fusion performance depends mainly on the sensing modalities, the quality of data and the selected architecture. For fusion operations, feature concatenation is a widely exploited method, specifically in early and intermediate levels. Likewise, the addition and mixture of experts are mainly employed for intermediate and high stages.

-

Commonly, object detector outcomes include a list of detected bounding boxes, confidence levels and affected classes. In fact, the bounding box is mostly presented by its bottom-right and top-left coordinates (x initial ,y initial ,x end ,y end ), with an exception for the YOlO algorithm.

The most common performance metrics used to evaluate the object detector are the Average Precision (AP) and and its derivatives as the mean Average Precision (mAP) over all the objects' classes. Previously, some concepts should be reviewed which are shared among these metrics. The basic ones are presented below :

• True Positive (TP): It is the accurate detection of a ground truth bounding box.

• False Positive (FP): It is the incorrect detection related to a misplaced bounding box of an existing object or a nonexistant object.

• False Negative (FN): It is a miss-detected ground-truth bounding box.

It is important to point that a True Negative (TN) result is not considered in the scope of object detection, since there are limitless bounding boxes that should not be detected among any image.

The aforementioned definitions require distinguishing correct detection from incorrect detection. For this reason, the Intersection Over Union (IOU) measure is used. This metric is based on the Jaccard index, which represents a measurement of similarity between two sets of data [START_REF] Jaccard | Étude comparative de la distribution florale dans une portion des Alpes et des Jura[END_REF]. In the object detection context, the IOU is employed to measure the overlapping area among the ground-truth bounding box (B Gt presented by the orange color) and the predicted bounding box (B P presented by the green color) divided by the area of the union between them [START_REF] Padilla | A survey on performance metrics for object-detection algorithms[END_REF]. The IOU is presented by Equation 2.1 and illustrated in figure 2.9.

IOU = J(B P , B Gt ) = B P B Gt B P B Gt (2.1)
Subsequently, the obtained IOU value is compared to a defined threshold (thr) to decide if the detection is accurate or inaccurate. If the IOU ≥ thr , then the detection is correct. Otherwise, if the IOU < thr, then the detection is incorrect. After determining the TP, FP and FN through the IOU calculation, the assessment of the detectors rely mostly on the Precision (P), the Recall (R) and the F1-score metrics. The P presents the ability of the detector to identify just the relevant objects. This metric introduces the percentage of the correct positive predictions. Concerning the R, it presents the ability of the detector to identify all the relevant samples, i.e. all the groundtruth bounding boxes. It exhibits the percentage of the accurate positive predictions within all the considered ground truths. The F1-score is the metric that balances P and R. The metrics mentioned above are presented respectively by Equation 2

F1 -score = 2 × P . R P+R (2.4)
Besides, an object detector achieves a good performance when it identifies all the ground-truth objects, so FN is equal to 0 (high recall) while finding only relevant obstacles so FP is equal 0 (high precision). Hence, the detector is considered good when its precision rate is high while its recall rate increases, which means if the confidence threshold is changed, the P and R will remain high. Thus, the AP metric is calculated via Equation 2.5. It uses the all-point interpolation concept where the precision is interpolate 2. 5 36 at each level and it takes the maximum precision value whose recall rate is greater than R n .

AP = ∑ n R n -R n-1 P n (2.5)
The mAP is the metric that measures the accuracy of the object detector over all the classes considered in a specific dataset. The mAP is the average of th AP over all classes and it is presented by Equation 2.6.

mAP = 1 N N ∑ i AP i (2.6)
where AP i is the AP of the ith class and N is the total number of the classes that are being evaluated.

2.5

In this chapter, we have presented the state-of-the-art of object detection dilemma for environment perception for intelligent and autonomous systems. By analysing the advantages and the disadvantages of using each sensor separately, we have concluded that these sensors are complementary to perceive the vehicle's surroundings. Following, we reviewed the fusion strategies and object detectors based on combing different sensors at various levels.

In the next chapter, we will review the datasets used in this field of research and we will present our multimodal developed dataset for advanced environment perception. After that, a fusion framework will be presented. As we mentioned in the previous chapter, almost of the multi-modal environment perception approaches are based on supervised learning. Accordingly, multi-modal datasets including labeled ground-truth is needed in order to train such methods, so, to develop road object detectors. This chapter summarizes numerous published real-world datasets regarding employed sensors, the recording conditions, their size and labels. Consequently, we detail our proposed multi-modal dataset and a new fusion framework that combines data acquired from the different sensors used in our dataset to achieve better performances for obstacle detection task. Section 3.2 refers to an exhaustive overview of the available public environment perception databases. The proposed dataset is introduced in section 3.3 which includes the sensors setup, the scenarios, the challenges and the dataset details. The section 3.4 exhibits the proposed fusion framework and the obtained results.

O L I M P : A H E T E R O G E N E O U S M U LT I M O DA L DA T A S E T F O R A DVA N C E D E N V I R O N M E N T P E R

3.2

Public multimodal datasets are indispensable for autonomous driving's advancement. In the last decade, several datasets have been released for this purpose, Kitti [START_REF] Geiger | Are we ready for autonomous driving? the kitti vision benchmark suite[END_REF] dataset and Cityscapes [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF] dataset are considered the first datasets that have addressed real-world challenges. Until few years ago, datasets that contain only sparsely annotated data were satisfactory to treat several problems. But nowadays, with the evolution of deep learning techniques the exploitation of such datasets is insufficient [START_REF] Kang | Test your self-driving algorithm: An overview of publicly available driving datasets and virtual testing environments[END_REF].

In fact, the training of deep models requires datasets with a huge number of labeled data though collecting such amount of data is not an obvious task. Hence, this requirement has led to the development of several new sophisticated autonomous driving datasets [START_REF] Guo | Is It Safe to Drive? An Overview of Factors, Metrics, and Datasets for Driveability Assessment in Autonomous Driving[END_REF]. In this section, we review various existing public monomodal and multimodal environment perception databases by detailing and observing the characteristics of each one. Table 3.2 exhibit an overview of various environment perception datasets.

• Kitti : It is a vision benchmark dataset that was released in 2012 and comprises stereo camera, Velodyne lidar and inertial sensors [START_REF] Geiger | Are we ready for autonomous driving? the kitti vision benchmark suite[END_REF]. Within the introduction of this database, various vision tasks were launched as pedestrian detection, road detection, etc. It was recorded in six different emplacement with cluttered scenes and it provides over 200k boxes that was manually labeled. Nevertheless, only 3D objects that exist in frontal view are annotated and it covers only daytime conditions. Moreover, the preeminent limitation of Kitti database is the small amount of data that is not suitable for deep learning algorithms.

• CamVid : The University of Cambridge has introduced a new driving dataset named CamVid. It was the first that contains videos with semantic segmentation labels related to 32 classes. However, the size of this dataset is small; it contains a limited number of sequences: only four scenes [START_REF] Gabriel | Segmentation and recognition using structure from motion point clouds[END_REF].

• Cityscapes : It is a dataset that was published in 2016 [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF]. It covers urban traffic scenarios in 50 cities, in only spring and summer and it includes 30 categories. Cityscapes consists of a pixel-level and instance-level segmentation labeling. Indeed, it contains mainly images and few videos with 5000 images which have fine-annotation over 20000 images along coarse annotations.

• BBD100k: It was recorded in 2016 in four different regions in the US [START_REF] Yu | Bdd100k: A diverse driving video database with scalable annotation tooling[END_REF]. It is considered as the largest driving video dataset due to its diversity in terms of data and driving conditions. The database comprises 100k videos containing almost 1000 hours recorded under different weather conditions. Indeed, only one image is selected from each video sequence for labelling likewise Cityscapes dataset. Ten thousands images are labeled in pixel level and bounding box labels are provided for 100k images.

• Kaist Multi-Spectral: It is a multimodal database that was repeatedly collected in urban, residential and campus environments [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF]. Several sensors were fixed on the vehicle, namely: stereo camera, thermal camera, GNSS, 3D lidar and inertial sensors. Moreover, it covers a diverse time slots ( day, night, morning, sunset, etc.) and the annotation is provided in 2D. Compared to the newest released datasets, the size of the Kaist dataset is limited.

• ApolloScape : Compared to Kitti and Cityscape databases, the ApolloScape dataset [START_REF] Huang | The apolloscape open dataset for autonomous driving and its application[END_REF] contains an extensive amount of data and has many properties that will be detailed in the following. In fact, it includes stereo driving sequences that reach over one hundred hours of recording under diverse day times and about 144k images. It covers also 2D and 3D pixel-level segmentation, instance segmentation, lane marking and depth. Further, in the intention to label such a database, the authors developed several tools customized mainly for the annotation process. However, data acquired from lidar is used to provide only static depth maps.

• H3D : It was introduced in 2019 , and it considers various complex and congested scenes over 160 [START_REF] Patil | The h3d dataset for full-surround 3d multi-object detection and tracking in crowded urban scenes[END_REF]. Three cameras, a lidar, a GPS and inertial sensors were used to collect this dataset. The main challenge addressed in this dataset is 3D multi-object detection and tracking. In fact, it consists of 1.1M 3D boxes annotated data, which includes over 27 k frames. Plus, objects are labeled in 360 • view. Eight classes was taken into consideration when recording this dataset : car, pedestrian, cyclist, truck, misc, animals, motorcyclist and bus. It is true that the dataset comprises rich scenes and annotation with a particular size, nevertheless, the data was registered under daytime conditions.

• BLVD: The dataset introduced in [START_REF] Xue | BLVD: Building a large-scale 5d semantics benchmark for autonomous driving[END_REF] and entitled BLVD does not focus on static obstacle detection only, but especially on dynamic object detection. Indeed, this dataset proposes a platform that involves 4D tracking (3D+temporal), 5D interactive recognition events and 5D intention prediction. It includes 3 categories : vehicle, pedestrian and rider, the data was recorded in daytime and nighttime conditions. It provides 120k frames with a 5D semantic annotation and beyond 249 3D annotation.

• nuScene (nuTonomy scenes): It is the first dataset that involves the three preeminent sensors exploited to ensure an autonomous driving which are a lidar, 5 radars and 6 cameras [START_REF] Caesar | nuscenes: A multimodal dataset for autonomous driving[END_REF]. This database consists of 1000 scenes where the duration of each scene is 20s. The annotation provided is a 3D bounding boxes specified for 23 categories. The data were gathered under several lighting and weather conditions : daytime, nighttime and rain . This new released dataset is rich in terms of utilised sensors, size, acquisition conditions diversity, amount of data with 1.4M frames and annotation numbers. Yet, the main issue of this dataset is the class imbalance represented by the inequality number of examples of infrequent and ordinary object classes.

• A2D2: The A2D2 (Audi Autonomous Driving Dataset) is recorded via six cameras and five Lidar in order to provide a full 360 • coverage. It includes 41,277 frames along semantic segmentation images and point cloud labels. In addition, this dataset is the only dataset that contains vehicle bus as the steering wheel angle, the throttle, and the braking. The A2D2 data were recorded on cities, highways and country roads in the south of Germany under sunny, cloudy and rainy weather conditions.

Other than the autonomous driving databases mentioned previously, it exists additional datasets that are released for the same purpose, such as the Oxford Robotcar [START_REF] Maddern | 1 year, 1000 km: The Oxford RobotCar dataset[END_REF], Udacity [164] and DBNet Dataset [START_REF] Chen | Lidar-video driving dataset: Learning driving policies effectively[END_REF]. Cityscapes [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF] x x BDD100k [START_REF] Yu | Bdd100k: A diverse driving video database with scalable annotation tooling[END_REF] x x x x

Kaist Multispectral [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF] x x x x ApolloScape [START_REF] Huang | The apolloscape open dataset for autonomous driving and its application[END_REF] x H3D [START_REF] Patil | The h3d dataset for full-surround 3d multi-object detection and tracking in crowded urban scenes[END_REF] x x BLVD [START_REF] Xue | BLVD: Building a large-scale 5d semantics benchmark for autonomous driving[END_REF] x nuScenes [START_REF] Caesar | nuscenes: A multimodal dataset for autonomous driving[END_REF] x Table 3.1: Categorization of some autonomous driving datasets by task Furthermore, some multi-modal virtual datasets or virtual simulators have been developed in order to generate variable driving situations, specifically the dangerous scenarios that can't be or hard to be collected in real-world. For instance, Gaidon et al. [START_REF] Gaidon | Virtual worlds as proxy for multi-object tracking analysis[END_REF] develop a virtual Kitti database, in [START_REF] Stephan R Richter | Playing for data: Ground truth from computer games[END_REF] and in [START_REF] Wrenninge | Synscapes: A photorealistic synthetic dataset for street scene parsing[END_REF] virtual dataset have been generated based on game engines as GTA-V. Dosovitskiy et al. [START_REF] Dosovitskiy | CARLA: An open urban driving simulator[END_REF] build an open-source simulator for simulating multiple sensors. Even though various virtual dataset are available now, the question that remains is how accurate the simulator can represent real-world circumstances.

As the databases are mainly released to enhance the scenes understanding and environment perception, we provide in Table 3.1 a categorization of the most important autonomous driving datasets according to a particular tasks.

From table 3.1, we can notice that most of the reviewed datasets were dedicated to multi-object detection as it is an inevitable process in the autonomous world. Likewise, there are favorable number of datasets dedicated to object tracking, lane recognition and semantic recognition, but in return, just a few ones can be used to optical flow exploitation and SLAM (Simultaneous localization and mapping) process.

Following the exhibition of the most datsets, we provide above a comparative study in terms of the recording conditions, the employed sensors, the dataset size, etc.

-Sensing Modalities : In terms of sensing modalities, all the examined datasets contain RGB images acquired from one or more cameras or video in HEVC (high efficiency video coding) standard or in recent coding [START_REF] Ferroukhi | Medical video coding based on 2nd-generation wavelets: Performance evaluation[END_REF]. Lidar sensor also have been well exploited. For radar data, it is only presented in nuScenes dataset [START_REF] Caesar | nuscenes: A multimodal dataset for autonomous driving[END_REF] and the newly released Oxford Radar RobotCar Dataset [START_REF] Barnes | The oxford radar robotcar dataset: A radar extension to the oxford robotcar dataset[END_REF] and Astyx HiRes2019 [START_REF] Meyer | Automotive radar dataset for deep learning based 3d object detection[END_REF]. It is a very limited number despite that this sensor provide rich information and helps in the environment perception and taking the right decisions. For that reason, nowadays, it becomes essential to exploit radar sensor in developing autonomous driving datasets.

-Recording Conditions : The majority of the collected data is specialized in urban driving, and was recorded in different locations: Europe, the United States, Asian cities, etc. this variance in locations allows us to have a global view of roads conditions, environments, etc.

One of the important criteria to have a complete dataset, is that it is collected under different lighting and weather conditions in order to cover various scenarios [START_REF] Jegham | Vision-based human action recognition: An overview and real world challenges[END_REF]. Nonetheless Kitti dataset is broadly used in this field of research, the variety of its recording environmental conditions is reduced: it is gathered just under daytime and sunlit days, similar to CamVid, CityScapes and H3D datasets. In order to enrich light recording conditions [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF], [START_REF] Caesar | nuscenes: A multimodal dataset for autonomous driving[END_REF], [START_REF] Yu | Bdd100k: A diverse driving video database with scalable annotation tooling[END_REF], [START_REF] Huang | The apolloscape open dataset for autonomous driving and its application[END_REF] and [START_REF] Xue | BLVD: Building a large-scale 5d semantics benchmark for autonomous driving[END_REF] collected data considering both daytime and nighttime all day long. Concerning the diversity of weather conditions, only BDD100k and nuScenes covers rain and snow situations. Actually, seasonal changes are not well covered as the majority of the databases were recorded in short periods.

-Dataset Size : We notice that since 2016, the number of the published datasets becomes extensive because of this importance in the development of selfdriving cars. As the dataset size plays a key role in this field, it varies from 1,569 frames to above than 11 million frames, and it has grown over the years. nuScences is considered as the largest dataset with 1,4M frames. Yet, compared to the size of the image datasets related to the computer vision community, the environment perception datasets remain relatively small.

-Labels : Depending on the principal aim of the published dataset, objects are labeled into various categories. Comparing the object classes existing in each dataset, we can observe that the number of examples attributed to each class is imbalanced. For example, we compare the samples related to two different classes: car and pedestrian for nuScenes, Kitti and Kaist Multispectral databases. We can observe that there are much more car labels than pedestrian labels, as shown in figure 3.1.

3.3

As mentioned in the previous section, the importance of multimodal perception techniques for ITS and the extent of research efforts in this direction emphasize the need for multimodal datasets that explore complementary sensors. Therefore, in this section we present our proposed dataset and we detail its implementation, challenges and opportunities.

Background

Various autonomous driving datasets have been published in order to enhance research for environment perception. Most of these datasets are multimodal, combining different heterogeneous modalities. While some of the existing datasets use narrow-band radar, the UWB radar carries richer information. The UWB radar provides a signal that results from the reflection of a transmitted UWB pulse on the object. The deformation of the initial wave represents the signature of the object. This signature contains information about the distance, the material and the shape of the object. Moreover, different objects have distinguishable acoustic signatures that may help recognize each of them. In spite of the usefulness of the acoustic data, we notice that none of the state of the art ITS benchmarks uses acoustic modality.

Thus, we introduce OLIMP ( A heterOgeneous MuLtimodal Dataset for Advanced EnvIronMent Perception ) (https://sites.google.com/view/ihsen-alouani), a new public dataset for road environment perception. The introduced OLIMP dataset is a multimodal synchronized dataset that was collected using four heterogeneous sensors to better understand the vehicle's environment.

Our benchmark contains four complementary modalities namely: UWB radar data, narrow-band streams, images and acoustic data. In fact, camera is affected by degraded condition such as foggy weather, while, UWB radar is not influenced by neither luminosity nor weather conditions. The acoustic data is orthogonal to the vision field. Concerning the narrow-band radar, it provides position and velocity. To the best of our knowledge, OLIMP is the first benchmark that contains UWB radar data and acoustic data.

The data was collected in the campus of the Polytechnic University Hauts-de-France in Valenciennes -France (Valenciennes is known for its foggy weather). Data was captured during 3 months and consists of 47354 synchronized frames.

Hardware and data acquisition

On the one hand, we used four heterogeneous sensors: a monocular camera, an UWB radar which is a short range radar, a narrow-band radar that is a long range radar and a microphone. On the other hand, we exploited the EFFIBOX platform to acquire data from the different sensors simultaneously [START_REF]The Effidence Organization[END_REF]. In table 3.3, we highlight the sensors' characteristics and their technical details.

UMAIN radar: it is an UWB radar. The exploited kit is called HST-D3 developed by the UMAIN corporation [163]. The kit comprises a UWB short radar with a Rasperby Pi 3 for the acquisition. Following, the received radar raw data are transmitted to the computer through the Raspberry Pi that is connected via TCP/IP protocol. More details about the UMAIN radar are provided in the next chapter.

Narrow-band radar (ARS 404-X): This Premium sensor from Continental is a long range radar that is able to detect multiple obstacles up to 250 meters. It genertaes raw data that include: distance, velocity and radar cross section RCS [29]. Data are transmitted to the EFFIBOX platform via CAN bus.

The EFFIBOX platform : is a software developed in (C/C++) dedicated to the design of multi-sensor embedded applications. In addition, diverse adequate development functionalities are available such as : acquiring and saving sensor streams, processing/post-processing, visualization, etc.

It should be pointed that, the EFFOBOX platform has its own API(Application Programming Interface) to communicate with the ARS radar, the network camera and the microphone in order to acquire and record data. For the UMAIN radar, we developed our API so that the EFFIBOX can communicate with the radar. Then, the acquired data has been decoded following a particular protocol provided by the company. Besides, the frame acquired from the ARS radar are decoded also according to the protocol provided by the Continental datasheet.

Sensors embedding

With regard to the sensors configuration, we designed a structure where all the sensors are placed in the front view. To simplify the data fusion, the narrow-band and the UWB radars and the camera were mounted on the same vertical axis. Figure 3.2 shows the proposed data acquisition architecture, and figure 3.3 highlights the structure setup.

Sensor synchronisation

To develop an efficient autonomous driving dataset, sensor synchronisation is a challenging and inevitable task. In fact, sensor fusion often requires that collected data from all the sensors have to be captured on the same time as each sensor has its own latency. To illustrate this phenomena, figure 3.4 (a) shows a samples of a simultaneous acquisition from different data streams.

Therefore, we developed our method to achieve an accurate alignment between the modalities' data streams. In the simultaneous data recording process, we register times- tamps relative to each sensor separately. We first start with synchronizing the radars and the camera. Since these sensors have different frequencies and time responses, we choose the narrow-band radar as a primary sensor. This is explained by the fact that the narrow-band radar is the slowest among these sensors; it has the highest latency of a complete measure compared to the other modalities as shown in Table 3.3. In fact, the narrow-band radar raw data is represented in the form of a stream of discrete measures. Each one of these measures comprises a main data frame including the obstacle's number followed by successive information about each obstacle (distance, velocity, dynamic property, etc.). Once a narrow-band measure is taken, we capture its timestamp and look for the camera frame as well as the UWB frame that have the closest timestamp to the synchronization narrow-band timestamp.

49

Regarding the acoustic modality, the frame corresponds to an analog signal (sound). The challenge is to find the most suitable time window size that: (i) corresponds to the exact scene recorded at given timestamp, and (ii) is long enough to hold meaningful information about the scene. After thorough explorations, we empirically choose an optimal window size of 5 seconds for acoustic signal frame. This frame is recorded according to the narrow-band synchronization timestamp mentioned above.

Overall, the proposed algorithm consists of selecting the timestamp acquisition of every narrow-band measure and find the corresponding frames of the other sensors which have the closest timestamp. The frames synchronization step is illustrated in figure 3.4 (b).

Labeling process

In addition to the background, we consider four classes: pedestrian, cyclist, vehicle and tram since these are the most probably encountered possibilities in an urban transport environment. The vehicle class contains cars, tracks, etc. For the labeling process, we manually annotate data consecutively one image per three as this task is time consuming and the changes between two successive images are practically negligible. We avoided automatic annotation to have a high quality labeled ground truth. Thus, we used the Matlab Image Labeler toolbox whom we have the license as semi-automatic labeler tool.

50

Data annotation includes 2D bounding boxes that present respectively x, y, the width W and the height H of the object in pixels.

Scenario selection and data formats

In order to collect raw sensor data, we carefully choose diverse driving situations. The duration of scenes differs and depends mainly on the situation complexity. While recording our dataset, we consider diverse challenges that will be detailed in the following subsection. Accordingly, we emphasize the data variety through employing different locations (8 emplacements) that vary in terms of structures, environment, road markings, traffic signs, etc. Some of these recording emplacements are illustrated in figure 3.5. Driving situations are carefully selected and collected under different lighting conditions, we covered also sunny, cloudy and snowy weather.

For data format, the dataset provides synchronized frames of each scenario, the data are stored as : RGB images, .txt files presenting UWB radar signals, .txt files of narrow-band radar data stream and .wav microphone files.

Dataset challenges

With the intention of developing a complete dataset, we cover realistic conditions for environment perception (such as: cluttered environment, occlusions, lighting conditions, etc.). Thus, we employed several sensors to obtain redundant information or complementary data that may compensate the challenges evoked by each sensor. Figure 3.6 highlights the most introduced challenges in our dataset.

The object's types exhibit an immense variability since they vary in terms of appearance, movement and differ from the point of view of the class: pedestrian, vehicle, etc. When recording our data, we take into consideration this camera-radar challenge as we consider 4 categories of obstacles. Furthermore, our dataset was performed by several pedestrians and cyclists of different ages, looks, body sizes, etc. Moreover vehicles are varied: multiple cars, vans and trucks. We can see this differentiation through UWB radar signatures shown in figure 3.7 that correspond to each of the considered categories. Moreover, the considered objects can be static or dynamic.

Distance is one of the fundamental challenges presented for autonomous driving either for camera, the two exploited radars or even the microphone. According to this, we consider two representations when capturing our dataset, depending on the range: near and far obstacles.

A further challenge is presented : the cluttered environment since generally dense urban driving involves many traffic agents with a complex background. For UWB radar, multiple reflections can influence the quality of the signal in the presence of many objects. Concerning narrow-band radar, it generates many detections when various obstacles exist, thus a selection process is required to identify the relevant ones. So, we attempt to introduce several complex scenes during recording.

Furthermore, we consider diverse lighting conditions as we record data throughout the day (morning, afternoon and sunset). We collect our dataset under sunny, foggy and snowy weather to increase the diversity and cover the possible real driving situations. In fact, the camera is highly sensitive to the last mentioned challenges whereas the radar is robust against them.

Besides, the object detection task is extremely delicate to occlusions that occur between several classes which is frequently presented in diverse cluttered scenes. OLIMP includes severe occlusions situations combining the four classes as pedestrians that are often occluded by each other or by a cyclist, a vehicle or a tram, or the opposite.

Furthermore, the inter and intra class challenges are depicted in figure 3.8 by presenting the synchronized data acquired from the camera, the UWB radar and the microphone.

Statistics and dataset organisation

OLIMP is organized in 6 subsets from C0 to C5. C0 contains background only, C1 includes either one, two or a group of pedestrians. C2 comprises cyclists, C3 and C4 include respectively vehicles and trams. The final subset C5 contains the different possible combinations of the aforementioned classes introduced in OLIMP dataset considering various scenarios. In fact, we only focus on the main moving road objects that can be presented in an urban traffic scene.

The dataset consists of 407 scenes, and the number of scenarios in each subset vary as follows C0: 12 scenarios, C1 :144 scenarios, C2:31 scenarios, C3: 51 scenarios, C4: 18 scenarios and C5: 151 scenarios.

Our dataset was performed by 93 pedestrians, 14 cyclist and using 90 vehicles and 2 trams. Precisely, the dataset presents 47354 data for each sensor. For the evaluation protocol, 2 3 of the dataset is used for training, and 1 3 for test. In order to propose an accurate multi-modal system for obstacle detection, in this section, we aim to evaluate each modality individually in order to propose, afterwards, a fusionbased system that takes advantage of each modality contribution.

3.4 54

Image-based system

The multiple obstacle detection task can be divided into two steps: the recognition that is insured via a probability estimation and the localization that defines the bounding boxes. Thus, deep learning techniques have been widely adopted in image-based object detection as it has been mentioned in chapter 2.

Among the known deep architectures used in the literature, we used the pretrained MobileNet-v2 [START_REF] Sandler | Mobilenetv2: Inverted residuals and linear bottlenecks[END_REF] model on a subset of the ImageNet dataset for detecting objects on RGB images. Inspired by the constraint of computations for mobile devices, the the light-weighted MobileNet networks was introduced in 2017 by presenting the MobileNet-v1 [START_REF] Qin | Fd-mobilenet: Improved mobilenet with a fast downsampling strategy[END_REF]. The next version is MobileNet-v2 which further improves the previous released version by employing a mobile inverted depth-wise convolution together with a residual connection. The MobileNetV2 is based mainly on depthwise separable convolutions and it contains two blocks. The first block is a residual block with a stride equal to 1 and the second block is a downsizing block with a stride equal to 2. Its architecture contains three convolution layers for both mentioned blocks: 1x1 convolution layer with RelU6 named pointwise conv, a depthwise convolution and a 1x1 pointwise convolution. The MobileNet-v2 building block is illustared in figure 3.9. The overall MobileNetV2 architecture contains 17 of these blocks. These blocks are followed by a regular convolution layer, an average pooling layer and a fully connected classification layer.The network consists of 54 layers deep and uses 3.5 millions of parameters [START_REF] Sandler | Mobilenetv2: Inverted residuals and linear bottlenecks[END_REF]. In fact, the presiding model was chosen due to its compromise between performance and execution time [START_REF] Qin | Fd-mobilenet: Improved mobilenet with a fast downsampling strategy[END_REF].

The results relating to the training of this network are presented in figure 3.10, and the metrics that are chosen to evaluate the performance are P, R and AP. The mAP reaches 60,5 %.

As shown in figure 3.10, MobileNet-v2 achieved a significantly higher results on the four categories in terms of precision. However, the image-based system provides high rates of recall for all the classes which explains that the system generates too many false negative samples.

Radar-based system

To demonstrate the importance of using the UWB radar, we proposed a radar-based system to discriminate the four classes for short distances. First of all, we classified the whole signals using SVM in the intention of distinguish the classes, yet, the results were not promising as the signals present rich information with a significant leakage in the beginning. For this reason, we decide to exploit the narrow-band radar data to achieve better performance. Though, the proposed approach consists of selecting ROIs 3.4 56 in the signals acquired from the UWB radar in order to localize radar signatures that characterise the obstacle. Afterwards, these ROIs will be classified using SVM.

In fact, narrow-band radar generates a list of targets with their position and velocity. Thus, we injected the distances taken from narrow-band radar data to define the ROIs in UWB signals. In this state, we focus our attention to obstacles which are located less than 6 meters, while after various experiments the UWB radar is less efficient for a range that exceeds this margin.

We can observe that we obtain multiple ROIs when matching the narrow-band points with the signatures as the acquired radar are is too sparse. The detected distances are too close and may refer to a same object. Thus, in the aim to select the relevant distance and minimize the ROIs number, we proposed to exploit the velocity of each obstacle with the distance. This leads to a better localising of the signature. For that, two objects that are side by side and have the same velocity are considered as one target (represented in green color) as shown in figure 3.11.

Narrow-band

Radar ARSX In addition to this, we set an amplitude threshold to validate the ROIs. Figure 3.12 illustrates this process. The selected ROIs are classified using an SVM classifier with an Radial Basis Function (RBF) kernel. The results of the UWB radar-based system are shown in figure 3.13.

According to our experiments and obtained results, we assume that the proposed radarbased system can better distinguish pedestrians and cyclists. Aside from the fact that the UWB radar provides a unique signature for each class, it is not able to classify tram and vehicle. Since the results in table 6 include the overall dataset testing, the accuracy results for those two classes are remarkably low. For experiments safety,the tram and the vehicle are generally located a far from the radar, in a range greater than 6 meters. Thus, reflections' magnitude from these two classes are low compared to reflections acquired from a cyclist or a pedestrian that are usually closer to the field of view of the radar. This explains the difference of accuracy between the two latter classes and the first classes.

3UHFLVLRQ 5HFDOO

(YDOXDWLRQ0HWULFV

3HGHVWULDQ &\FOLVW 9HKLFOH 7UDP

Acoustic-based system

According to the state of art, the MFCC (Mel-Frequency Cepstral Coefficients) are widly used in sound processing and analysis as it provides a better representation of the sound [START_REF] Serizel | Acoustic features for environmental sound analysis[END_REF]. Hence, for acoustic data, we extracted temporal features and spectral features using MFCC based on several experiments. These features are concatenated and classified using SVM with RBF kernel.

As shown in the results presented in figure 3.14, using acoustic data leads to better performance for the two categories tram and vehicle. This is due to the relevant sound generated by these two classes. In other words, a walking pedestrian sound is narrow compared to the tram sound that presents more information. For this reason, precision and recall rates related to the tram and the vehicle classes are higher than the two others.

3UHFLVLRQ 5HFDOO

(YDOXDWLRQ0HWULFV

3HGHVWULDQ &\FOLVW 9HKLFOH 7UDP
Figure 3.14: Acoustic-based system results (%)

Multi-modality fusion system

To prove the significance of our dataset and the importance of multi-modality aspect, we take advantage of the different sensors by proposing a fusion framework system. This framework is built in the lights of the results obtained from the aforementioned The first step of the framework consists of extracting the labels from the outputs of the MobileNet. If the extracted label is a car or a tram, we use the acoustic-based system to verify the attributed label, and, all the labels are updated accordingly. Subsequently, if the CNN-extracted label is neither a tram neither a car, the distance of the object will be calculated. The process of calculating the object distance is explained in the following. Thus, if it is a far obstacle we will keep the same labels of the CNN model. Nonetheless, if it is a near obstacle then it will be either a pedestrian or a cyclist. Along with, we will adopt the radar-based system to confirm the attributed label, since it can particularly discriminate the aforementioned categories in a range less than 6 meters. Thus, the results related to the fusion framework are illustrated in figure 3.17. Distance calculation using camera images : In the aim to find a way to select suitable object's distance, We adopt to use the bounding boxes obtained after the testing process using MobileNet-v2 network.

In fact, to perceive the depth information from images, stereo-vision techniques are generally used. However, they require an intensive computation and in our case we deal with a monocular camera. For this reason, we propose to exploit an area-based approach to determine the object distance from the available bounding 3.4 60 boxes. In other words, we look to find a relation between the object's area in the image and its real distance. Therefore, the area of the bounding boxes is estimated at several locations for each obstacle type and are stored in a training dataset. By using a curve fitting and optimization techniques, the data give a non-linear relationship between the area of the bounding boxes of the images and the real distance of the objects.

The relationship between the area of the bounding box and the object's distance can be modulated via Equation 3.1.

Object distance = α(Area) b + c (3.1)
where the relative parameters are : α = 741.3 , b = -0.507 and c = -0.3258. This process is illustrated in figure 3.16.

It should be pointed that the obtained relationship between the area and the distance using images is only valid for our employed camera.

Object's Distance estimated Figure 3.16: Estimation of object distance using area of detected bounding boxes.

Discussion

On the one hand, we conducted various experiments using mono-modality and multimodalities to validate our dataset and to open perspectives the way for future research.

On the other hand, these experiments show the significant impact of exploring multimodality and data fusion for an ITS to improve the obstacle detection task.

In fact, the fusion levels exploited in our work are the following: low level, intermediate level and late level. We can recognize the low level fusion when projecting narrow band From analysing the fusion results presented in figure 3.17, we notice that the performance has been clearly improved in terms of precision. Some detection results that include bounding boxes and scores are presented in figure 3.18 using just image-based system and with using fusion framework. The enhancement brought along with the acoustic system has a higher importance compared with the contribution of the radar-based system. This is mainly because of the range and power limitations of the UWB radar, and also owing to the sparse and noisy data provided by the narrow-band radar. Despite this fact, the UWB radar provides a unique signature for each type of object with a low price compared to the new sophisticated radars for short range applications. For this reason, this radar is more explored in the next chapter. Thus, we will focus especially on improving this single-sensor performance because it carries rich information, and due to its important short-range settings aspect. For the acoustic system, the distance between the obstacle and the sensor presents an important challenge. Moreover, obstacles like pedestrians and cyclists have low magnitude acoustic signals and could not easily detected through acoustic based systems. In addition, The considered environments in OLIMP are challenging and present various confusing categories such as metal infrastructure, trafic signs, glass-surface buildings, etc. In this chapter, we have presented a state-of-the-art of available datasets that were introduced for environment perception. By analysing the characteristics of each dataset, we have concluded that there is no dataset that employs UWB radar and acoustic data. Following, we represented our multi-modal dataset by detailing its hardware, its scenarios, employed sensors, etc. To validate our dataset, we conducted various experiments using mono-modality and multi-modalities, then, we proposed a fusion framework that enhances the detection performance.

In the next chapter, we will focus on detecting objects using just the UWB radar. Thus, two detectors will be proposed one based on entropy and the other a deep-based detector. The development of high performance obstacle detection systems is a safety critical task for intelligent and autonomous systems. Therefore, various sensors are employed to efficiently detect and recognize objects. Due to its robustness to weather conditions, radar is a promising sensor for environment perception systems. The main radar technologies that have been exploited in this domain are wide-band radars for short range applications and narrow-band radars for long range settings. In this chapter, we focus on the UWB sensor as it carries rich information.

In section 4.2, the UWB radar specifications are depicted. An overview of existing hand-crafted and deep learning-based methods using UWB signals are presented in section 4.3. In section 4.4, a novel entropy-based method is proposed for UWB radar-based multi-target detection and we discuss our results and their potential impact. Following, we propose the first framework that exploits LSTM with UWB signals for multi obstacle detection in an outdoor complex environment. Afterwards, we exhibit a comparison of our results with the state of the-art-techniques in section 4.5.2. We conclude the chapter in section 4.7.

4.2

In 2002, the US Federal Communication Commission (FCC) allows the unlicensed of UWB operations and commercial use of UWB-based devices [START_REF] Chong | Potential of UWB technology for the next generation wireless communications[END_REF]. Firstly, intended for military purpose applications, the UWB radar has been exploited in various applications. In fact, three main applications are defined (i) communications and measurement systems [START_REF] Mark | Ultra Wide Band Surveillance Radar[END_REF], (ii) imaging systems as through-wall imaging systems [START_REF] Shao | A Phase Shift and Sum Method for UWB Radar Imaging in Dispersive Media[END_REF], and medical systems [START_REF] Ahmed | Hand Gesture Recognition Using an IR-UWB Radar with an Inception Module-Based Classifier[END_REF], (iii) vehicular radar systems [START_REF] Lim | Deep Neural Network-Based In-Vehicle People Localization Using Ultra-Wideband Radar[END_REF]. The FCC permits a frequency range of UWB of 3.1 to 10.6 GHz in order to avoid the interference along the existing communication systems [START_REF] Chong | Potential of UWB technology for the next generation wireless communications[END_REF]. In addition, it defines that the UWB signal should has a fractional bandwidth more than 500 MHz or characterized by 20 % of the center frequency.

The UWB radar transmits a very short electromagnetic pulses with low energy in the order of sub-nanoseconds. These short pulses provide a huge interest in short range radar applications [START_REF] Sakkila | Short range automotive radar based on UWB pseudo-random coding[END_REF]. Various types of waveforms are used to generate the UWB pulse as the Gaussian and its derivatives. Besides, the Gaussian monocycle is the commonly type that is utilized as UWB impulses.

65

Basically, UWB radars can be categorized into two groups: pulse-based radars (e.g. time domain) that are typically referred to impulse-radio UWB (IR-UWB) and frequency modulations-based radars refereed as CW UWB radars (e.g. frequency domain) such as the frequency modulated continuous wave (FMCW). A comparison of the two technologies is provided in Table 4.1. In our work, we are interested in IR-UWB radar. In fact, the UWB radar has interesting characteristics for ITS applications as it has a better resolution than existing narrow-band radar devices. It consumes low power, it has a simple implementation and has a high data rate commutation. The UWB radar enables the penetration in dielectric materials. Therefore, the major property of the UWB radar consists of the distortion of the initial pulses. This signal deformation is impacted by the obstacle properties and thereby represents the object signature. This signature contains information that goes beyond the distance and the velocity; it is shaped by the object material, its shape and size [START_REF] Sakkila | A Real Time Signal Processing for an Anticollision Road Radar System[END_REF]. The UWB is able to detect stationary and moving obstacles on the vehicle's surroundings (on and nearby the road). The target's distance R is measured based on the delay between the emission and reception (τ), c is the speed of light. This relation is presented by Equation 4.1. The target's range calculation is illustrated in figure 4

.1. R = τc 2 (4.1)
These aforementioned characteristics show that the use of such radars is promising in detecting and recognizing objects, especially in short range applications, contrary to narrow-band radars that detect range targets with low accuracy and generate sparse data that contain numerous false alarms [START_REF] Hussein | Signal Processing of UWB Radar Signals for Human Detection Behind Walls[END_REF].

While the UWB radar offers rich information that is complementary to other sensors, its deployment presents serious challenges. One of the main challenges is differentiating targets from noise, which can practically be formulated as a segmentation problem.

UMAIN radar

The radar considered to record the OLIMP dataset, is an UWB radar developed by the UMAIN Inc company [163] entitled HST-D3. It has an efficient range of 6 meters and a frequency range in [3GHZ, 4GHZ] with a bandwidth of 0.45-1Ghz. The HST-D3 radar is a combination of HST-S1 Pi module radar and a Raspberry Pi 3 for the acquisition. It is a high-resolution radar transmitting and receiving UWB impulse on a single chip. Moreover, it has a Baudrate of 921600 and the UWB radar signal comprises 660 samples per frame. Total frame time is renewed every 22.5 ms and the interval per value in a single frame is about 2.0303 cm. This radar provides implemented algorithms as respiration detection, human and animal detection. In addition, the user can develop and implement its own algorithms.

The UMAIN radar contains an UWB monopole and UWB directional antennas. We exploit the available directional antennas since they guarantee better target echo-to-clutter and noise ratio. The radar is presented in figure 4.2 and table 4.2 exposes the radar specifications.

In fact, as we mentioned in section 4.2 UWB signals have excellent multipath immunity, and less susceptibility to interference acquired from other radios, due to its wide bandwidth property [START_REF] Gresham | Ultra-wideband radar sensors for short-range vehicular applications[END_REF]. However, the limitation of the UMAIN frequency range is that interference can appear due to WiMax(Worldwide Interoperability for Microwave Access) technology-based applications. Concerning the atmospheric attenuation, it is negligible for short ranges fields using the S-band [START_REF] Curry | Radar essentials: a concise handbook for radar design and performance[END_REF].

Hand crafted-based detectors using UWB radar

Most of the studies conducted on UWB-based obstacle detection systems have taken advantage of the well-known of three mainly approaches that have been proposed in the literature: correlation-based method as the matched filter [START_REF] Liang | UWB radar for target detection: DCT versus matched filter approaches[END_REF], higher order statistics (HOS) [START_REF] Mendel | Tutorial on higher-order statistics (spectra) in signal processing and system theory: theoretical results and some applications[END_REF] and constant false alarm rate (CFAR) [START_REF] Gamba | Radar Target Detection[END_REF]. These techniques are known as hand-crafted based detectors. HOS technique relies on higher-order moment spectra to analyse random process characteristics. This technique is commonly used to suppress Gaussian noise [START_REF] Mendel | Tutorial on higher-order statistics (spectra) in signal processing and system theory: theoretical results and some applications[END_REF].

The HST-D3 UWB Radar

CFAR techniques [START_REF] Gamba | Radar Target Detection[END_REF] [116] detect objects using an adaptive threshold that is defined based on the background noise and local information. Several algorithms have been extended from CFAR including the cell averging CFAR (CA-CFAR) [START_REF] Conte | Performance analysis of CA-CFAR in the presence of compound Gaussian clutter[END_REF], the order static CFAR (OS-CFAR) [START_REF] Blake | OS-CFAR theory for multiple targets and nonuniform clutter[END_REF], the Smallest Of CFAR (SO-CFAR) [START_REF] Cai | Some analysis of fuzzy CAGO/SO CFAR detector in non-Gaussian background[END_REF], etc.

In [START_REF] Sakkila | UWB Radar system for road anti-collision application[END_REF] the correlation is used to detect either a car, a metal plate, a motorway barrier or a pedestrian. A recent adaptive clutter suppression algorithm is proposed in [START_REF] Yoo | Adaptive Clutter Suppression Algorithm for Detection and Positioning using IR-UWB Radar[END_REF] based on CFAR technique, for human detection and positioning. An hybrid method was introduced in [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF], where a radar target detector based on the combination of HOS and CA-CFAR techniques was deployed. Nevertheless, experiments were performed under controlled scenarios. A new thresholding method based on CFAR technique for UWBbased detection application is proposed in [START_REF] Quan | A New Thresholding Method for IR-UWB Radar-Based Detection Applications[END_REF]. The approach takes into consideration false alarm and miss-detection criteria. New parameter entitled constant miss-detection rate (CMDR) is defined. The final threshold is calculated by adding the CFAR and CMDR rates. Experiments show good performances, however, they are just carried out in indoor environment using self-recorded dataset. A recent work in [START_REF] Kim | Lane Detection Method with Impulse Radio Ultra-Wideband Radar and Metal Lane Reflectors[END_REF] proposes UWB radar-based system to detect metal lane. The energy of the echo is calculated and if it exceeds a fixed threshold, the lane is detected. The use of this method requires a specific infrastructure. Moreover, energy values are dissimilar for different object types compared to metal obstacles energy.

In fact, almost of the aforementioned related work are a threshold-based methods. Otherwise, these techniques depend essentially on the amplitude of the object's signature. Moreover, it should be pointed that the considered environment and scenarios are controlled using restrained datasets. The aforementioned algorithms have been exploited ever since the development of the conventional radar.

Deep learning-based detectors using UWB radar

To the best of our knowledge, UWB-based systems that rely on deep learning techniques are employed only for indoor applications as: activities recognition [START_REF] Sadreazami | On the use of ultra wideband radar and stacked LSTM-RNN for at home fall detection[END_REF], indoor people localization [START_REF] Poulose | UWB Indoor Localization Using Deep Learning LSTM Networks[END_REF], recognizing movements during sleep [START_REF] Piriyajitakonkij | SleepPoseNet: Multi-view learning for sleep postural transition recognition using UWB[END_REF], etc.

Deep learning methods have been used with 2D UWB data, thus, this dilemma is considered as an image processing-based challenge. In [START_REF] Chen | Low PRF Low Frequency Radar Sensor for Fall Detection by Using Deep Learning[END_REF], an SFCW-UWB radar is used for fall detection which generates time-frequency spectrum as UWB data. Hence, the employed deep learning architecture is the fine-tuned Alexnet model. The authors in [START_REF] Ko | Object classification of UWB responses using S T-CNN[END_REF] convert the time-series UWB data to a time-frequency representation by Stockwell transform. Afterwards, the reshaped images serve as inputs of the CNN. The exploited CNN model is LeNet. In [START_REF] Ahmed | Hand Gesture Recognition Using an IR-UWB Radar with an Inception Module-Based Classifier[END_REF], UWB data are stored as 2D matrix including the slow-time and the fast-time properties. Subsequently, it is converted to a grayscale image, later to an RGB one. GoogLeNet was adopted as deep learning architecture for hand gesture recognition. In [START_REF] Piriyajitakonkij | SleepPoseNet: Multi-view learning for sleep postural transition recognition using UWB[END_REF], weighted range-time-frequency images of UWB radar are utilized as input for CNN to classify human sleep postures.

In point of fact, the 2D radar imaging based systems are no longer considered a signal processing dilemma as much as an image processing challenge.

Otherwise, 1D UWB signals are likewise employed with deep learning for indoor applications. To enhance transportation safety, an UWB radar is installed in the rear view mirror to estimate the number and the location of the in-vehicle people. Multi-layer perception is employed where the time-sampled radar signal data are the input of the network. To define the suitable parameters, the number of hidden layers are adjusted. Compared to machine learning techniques, the proposed network achieves better results [START_REF] Lim | Deep Neural Network-Based In-Vehicle People Localization Using Ultra-Wideband Radar[END_REF]. For activities recognition, a CNN-LSTM network using three UWB radars is implemented in [START_REF] Maitre | Recognizing activities of daily living from UWB radars and deep learning[END_REF]. Features are extracted using CNN architecture that includes: two 1D convolution layers where 64(1x3) filters are used, Relu activation function, a 1D max pooling layer. Subsequently, the output is flattened into 1D vector to feed the LSTM network that The aforementioned applications consider only indoor environments, using either a 2D radar imaging or a 1D UWB data. Nevertheless, ITS environment includes complex driving situations with various type of targets. Moreover, in our case the choice of adopting 1D radar signal rather than using a 2D radar data representation is justified by the fact that we deal with an ITS application where response time is a crucial criteria.

-

Background

While UWB radar has been widely used for obstacle detection, most of the reviewed works are evaluated on restrained datasets that have been recorded under controlled indoor environment. Furthermore, these techniques rely on the signature amplitude, which depends on the distance between the radar and the object location. This represents a fundamental challenge for UWB signals segmentation using amplitude-based techniques. This challenge is even more critical for human obstacles because of their low reflection compared to metallic obstacles.

Thus, the fundamental challenge of the detector is to distinguish a target from a received additive noise. To tackle this challenge, we suggest to use the signal entropy as an indicator of the existence of useful portions of the signal that can be differentially extracted out of the channel noise. So, in our case, we aim at segmenting 1D UWB signal used for an outdoor setting with a complex environment.

Recently, entropy-based information using UWB technology has been exploited in various fields of research as telecommunication [START_REF] Yin | Entropy-based TOA estimation and SVM-based ranging error mitigation in UWB ranging systems[END_REF], medical [START_REF] Borowska | Entropy-based algorithms in the analysis of biomedical signals[END_REF], etc. In [START_REF] Maherin | A mutual information based approach for target detection through foliage using UWB radar[END_REF], mutual information (MI)-based methods are applied to detect targets through foliage based on the calculation of entropy and conditional entropy. The defined threshold is log2(level of quantization of the signal). In fact, if the MI of the received echo exceeds the threshold, the target is detected. The proposed approach in [START_REF] Yang | Vital sign signal extraction method based on permutation entropy and EEMD algorithm for ultra-wideband radar[END_REF] adopts the permutation entropy (PE) for detecting human vital signs. As the PE detects dynamics changes in time series signals, it is employed to determine the range between the radar and the human target. For people-localization-based on UWB technology, the Shannon's entropy is used to accurately estimate the time of arrival of the first path in indoor environments. It is detected by identifying a great decrease in the entropy curve, and that exceeds a threshold value which has been defined via numerical simulation [START_REF] Yin | Entropy-based TOA estimation and SVM-based ranging error mitigation in UWB ranging systems[END_REF].

While the entropy is a measure of the signal's complexity, different types are employed in the literature as: PE, conditional entropy, Shannon entropy, etc. In fact, every type of entropy is suitable for a specific application and data. For our application, the entropy is applied as a segmentation tool for UWB signals in outdoor environment.

From an information theory perspective and as defined by Shannon [START_REF] Shannon | A mathematical theory of communication[END_REF], the Shannon's entropy is a theoretical metric of information. It is attributed to an information source or a given signal, and models the amount of information contained in this source or vehiculated by the signal. The Shannon's entropy may be used globally or locally: taking into account the whole data or a subset of data [START_REF] Borowska | Entropy-based algorithms in the analysis of biomedical signals[END_REF]. Therefore, to localize objects within an UWB radar signal, the use of the Shannon's entropy should be helpful in distinguishing useful ROIs from noise. This is based on the assumption that compared to noise, reflections from road obstacles present richer information. An illustration of this assumption is given in figure 4.3. The figure represents a received signals from two different objects (a cyclist and a vehicle) and its corresponding entropy curves. The bounded region (in purple) within the received signal represents the correct ROI which is depicted by the highest entropy value. Thus, our hypothesis is that the signal corresponding to an obstacle has potential information that is different from a random Gaussian noise. Thereby, the entropy distribution within each part of the signal should be different. Therefore, we suggest to exploit a differential entropy analysis of the received echo to localize ROIs within the signal.

In the following, we first provide a theoretical basis that backs the use of entropy for segmentation. Then, we detail our segmentation method.

Theoretical basis

In this work, our hypothesis is to use the signal's entropy as an indicator of the existence of useful parts of the signal (i.e., parts corresponding to obstacles in the radar's neighborhood) that are distinguishable from noise. Accordingly, we attempt to show that statistically, a useful signal should contain higher entropy than a noise signal. Notice that this observation will practically depends on different real-life parameters. Hence, in this study we make the following assumptions:

Since UWB radar is mainly used for urban traffic situations because of its short range, we assume a propagation model of an UWB radar that follows Rayleigh distribution, as shown in [START_REF] Molisch | A Comprehensive Standardized Model for Ultrawideband Propagation Channels[END_REF].

We assume an Additive White Gaussian Noise (AWGN) in the propagation channel of the UWB radar.

The received UWB signal r(t) can be modeled following Equation 4.12:

r(t) = s(t) + n(t) (4.2)
where s(t) is the received echo and n(t) is the noise of the transmission channel of the UWB radar. In fact, the process of emitting and receiving an impulse by the UWB radar is labeled as a radar scan, and the received echo of the received signal of the i th radar scan r i (t) can be modeled by Equation 4.3.

r i (t) = M i ∑ k=1 a ik x(t -t ik ) + n(t). (4.3)
where: x(t) is the transmitted pulse that is received as M i reflected signals, a ik is the amplitude, t is the reflected back time of the pulse signal after being transmitted from the radar, t ik is the delay of the k th received signal in the i th radar scan, n(t) an additive Gaussian noise of the transmission channel. Therefore, r k can be observed with corresponding probabilities p k ∀k ∈ [1, L] and the entropy H is expressed by Equation 4.4.

H = - L ∑ k=1 p k log 2 (p k ) (4.4)
The Rayleigh and the normal distributions have been proven log-concave in [START_REF] Bagnoli | Log-Concave Probability and Its Applications[END_REF]. Hence, based on the work on [START_REF] Bobkov | The Entropy Per Coordinate of a Random Vector is Highly Constrained Under Convexity Conditions[END_REF], the entropy could be approximated as: H(X) ≈ log(σ), where X is a random variable following a log-concave probability distribution and σ is its standard deviation.

Based on a statistical analysis of the distribution of both noise and useful signals (i.e. those in the presence of an obstacle) from a real-world dataset (OLIMP) [START_REF] Mimouna | OLIMP: A Heterogeneous Multimodal Dataset for Advanced Environment Perception[END_REF]. We found that σ noise = 0.0056 and σ obstacle = 0.0416. Hence, log(σ noise ) < log(σ obstacle ), as shown in figure 4.4.

Therefore, we can conclude that for a given UWB radar signal propagated in a Rayleigh fading channel with AWGN, we statistically have: H(N) < H(U), where N and U are random variables that represent the noise signal and the useful signal, respectively and H is the Shannon entropy.

In the following subsection, we detail our proposed segmentation approach. The received echo is partitioned into fixed-size overlapping sliding windows. The sliding window's size (W slide ) is empirically defined based on the objects' signature length. As the overlapping sliding windows should not miss the signal's valuable parts, we slide the window by one sample at a time. Subsequently, the Shannon's entropy is calculated for each window. The variation of the entropy values is then obtained by sliding the window through the whole signal. Since the entropy's value increases relatively with the rise of the signal complexity (for example: presence of different objects or noise), we localize the local maximums presented in the entropy curve in a range of W slide length.

Local maximums within the same W slide are ignored to prevent false detections that can be recurrent for the same object. The obtained maximums represent potential candidates for ROIs. Some of these candidates may be false positives generated by the noise entropy. Therefore, to detect only relevant entropy peaks, and by consequence limit false positives, a selection process is designed based on the study in Subsection 4.4.2. In fact, an empirical threshold is defined to withdraw noise-generated entropy peaks. Therewith, OLIMP dataset contains four object classes: pedestrian, cyclist, vehicle and tramway, and it includes various urban driving scenarios. In the following experiments, we compare our approach to HOS technique, CFAR technique and the work in [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF]. A brief description of these techniques is provided in the following.

(a) HOS

The HOS algorithm relies on the higher order moment spectra in the intention of interpreting and analyzing the characteristics of a random signal. One of the main advantages of employing this technique is its ability to reduce the Gaussian noise and the secondary lobes. Moreover, it characterizes and detects 4.4

-76

the non linearity in the data [START_REF] Sakkila | High order statistic receiver applied to UWB radar[END_REF]. Thus, Hos is used to detect various obstacles by applying a simple threshold. In this work, the 4 th order cumulant based on Tuganit4 algorithm is implemented for HOS method. The Tugnait4 Algorithm was introduced by Jitendra K.Tugnait [START_REF] Jitendra | Time delay estimation with unknown spatially correlated Gaussian noise[END_REF] in 1989 and it is expressed by Equation 4.5.

J 4 (i 0 ) = cum 4 (c(i -i 0 ), c(i -i 0 ), r(i), r(i)) |cum 4 (c(i), c(i))| |cum 4 (r(i), r(i))| (4.5)
where: c is the reference signal, r is the received signal, i 0 is the decision time index and cum 4 is expressed by Equation 4.6.

cum 4 (c(i -i 0 ), c(i -i 0 ), r(i), r(i)) = 1 N N-1 ∑ i=1 c 2 (i -i 0 )r 2 (i)) -2 1 N N-1 ∑ i=1 c(i -i 0 )r(i)) 2 - 1 N N-1 ∑ i=1 c 2 (i -i 0 ) 1 N N-1 ∑ i=1 r 2 (i) (4.6)

(b) CA-CFAR

For CFAR, the automatic threshold CA-CFAR detector is considered. The basic principle of this technique is shown in figure 4.6. In fact, the CA-CFAR is based on local information to adaptively define a threshold to detect the targets [52] [136].

The parts on the two sides of the Cell Under Test (CUT) are named guard cells, these cells do not participate when estimating the ground clutter power in order to avoid missing detection. The detection threshold T is expressed in Equation 4.7 [START_REF] Ni | Small Foreign Object Debris Detection for Millimeter-Wave Radar Based on Power Spectrum Features[END_REF]. The CA-CFAR detector decides whether there is an object or not by comparing the power of the CUT to the threshold.

T = γZ = 1 2n γ(P + Q) = 1 2n γ n ∑ i p i + n ∑ j q j (4.7)
where: p i and q i , (i, j = 1, ..., n) are the samples of the reference cells on both positions of the CUT, P and Q are the power summation related to the front and the back edge reference cells, Z is the average of all the reference cells and γ is the threshold coefficient that depends on the desired probability of false alarm rate P f a . (c) The work in [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] The work in [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] is a combination of the HOS technique and the automatic CA-CFAR detector. In fact HOS is employed first to suppress the noise. Afterwards, the CA-CFAR is applied on the received signal following the elimination of the noise.

In our work, all the experimental results are evaluated in terms of P, R and F1-score that balances P and R, these metrics were presented in chapter 2 in section 2.4.

Moreover, to evaluate the performance we used the Multiple Object Detection Precision (MODP) metric. This measure evaluates the positions' precision of the accurately detected objects. The overlap information between the system's detection and the ground truth is used to calculate a Mapped Overlap Ratio (MOR) for each frame as defined in Equation 4.8.

MOR = Nm ∑ i=1 Gi Di Gi Di (4.8) 4.4 - 78
Where Gi is the bounding box of the i th ground truth and Di is the bounding box of the i th detection and Nm is the number of the mapped outputs. For a single frame, MODP is obtained by normalizing MOR as shown in Equation 4.9.

MODP(t) = MOR Nm (4.9)
Hence for a multiple frames, the score MODP is determined by Equation 4.10.

MODP = ∑

N f rames i=1 MODP(t)

N f rames (4.10) where: N f rames is the total number of frames.

All the experiments were performed on a PC with an Intel (R) core (TM) i7-8565U CPU @ 1.8 GHz, 16 GB of RAM, using Matlab 2020a.

Threshold definition

Noise-induced entropy peaks need to be distinguished from actual objects-related entropy ones. A threshold-based decision is set in order to minimize the false positive rate of the proposed detector. Practically, we attempt to empirically identify an entropy threshold that allows the distinction of useful signal from noise.

To guarantee the most possible generalization, the OLIMP dataset has been randomly partitioned into two different subsets: reference and testing dataset. On that account, the threshold is determined related to the reference set, and the method is evaluated on the testing samples using the defined parameter. This evaluation methodology is coherent with state-of-the-art obstacle detection evaluation methodology [START_REF] Dairi | Unsupervised obstacle detection in driving environments using deep-learning-based stereovision[END_REF]. To define the suitable value that ensures an accurate and reliable radar-based obstacle detection, the threshold has been explored based P, R and F1-score metrics. figure 4.7 shows the exploration results.

We selected the threshold that maximizes the F1-score. This choice is justified since F1-score is a measure that achieves a trade-off between P and R and yields to an accurate reliable system. The highest F1 occurs at the threshold value of 0.0153, which is thereby selected as the empirical threshold.

Results

To validate the efficiency of the proposed method, a comparative benchmarking is elaborated. The considered targets in this comparative study are: Pedestrian, Cyclist, Car and Tram. The obtained results are illustrated in figure 4.8 and figure 4.9. It can be seen from the figure that the proposed approach provides higher detection performances compared to HOS, CA-CFAR and [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] in terms of P, R and F1 score.

Since we are using real-world conditions benchmark, the CA-CFAR achieves the lowest performance. Its performance is degraded when challenged with the presence of multiple targets. HOS-based detector enhances the accuracy detection since among its characteristics it is able to suppress noise in the received signal. Nevertheless, HOS fails when a target is located further than another since its signature amplitude will be lower and the obstacle is consequently ignored.

To visualize this behavior, figure 4.10 gives an illustration of the aforementioned limitations for multiple targets detection. The figure shows that while HOS and CA-CFAR fail to accurately detect all the real targets presented in the UWB signal by generating either false positives or missdetections (false negatives), our method can determine the right targets' positions. Due to physical characteristics, pedestrian's wave reflection amplitude is generally lower than other objects' reflections (especially metallic), and it gets further attenuated when moving away from the radar. This makes pedestrian detection challenging for HOS, CA-CFAR and [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF]. Thus, we reported the performance related to pedestrian detection using the implemented detectors in figure 4.9. As depicted from results, even though the reflection intensity of a human body is low, the proposed method reports the best overall performance for pedestrian detection, and with even higher relative improvement. For example, while for all objects (figure 4.8), we achieve 6.11% higher precision than HOS, this improvement is up to 12.75% for pedestrian. In terms of recall, our approach provides an improvement of 49.7% for all objects and an increase of 65.63% for pedestrian detection compared with the same technique. These results show the robustness of our technique and can be explained by the fact that it is more concerned with entropy within the signal than the signal magnitude.

To further explore the effectiveness of entropy-based obstacle detection, we purpose to present results amplitude-wise. This experiment allows us to evaluate the robustness of the proposed technique comparatively both in terms of signal amplitude, but also in terms of the obstacle distance. In fact, the amplitude of a given reflected signal on an object decreases accordingly with the corresponding object distance from the radar. As reported in figure 4.11, HOS precision degrades proportionally with the target amplitude. Objects with low amplitude are not detected as this technique is sensible to high magnitude. CA-CFAR and [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] have practically balanced performance in terms of precision for all the amplitude ranges. Most of the state-of-the-art techniques show higher detection performance in high amplitude cases, i.e., close obstacles. Surprisingly, for our method, we noticed a counter-intuitive precision increase as the amplitude decreases. This observation will be explained later.

According to the presented results in figures 4.12 and 4.13, the performances of HOS, CA-CFAR and [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] degrade proportionally with decreasing the object amplitude, which is correlated with increasing the object distance. In fact, as the object is moving away from the radar field, its amplitude reflection attenuate and its signal amplitude becomes close to noise signals. Since these techniques depend particularly on the signal amplitude, several objects will not be detected, thereby resulting in false negatives, which explains the detection performance degradation (R). Nevertheless, the entropybased method results increase remarkably as the object's amplitude attenuates. In low amplitude cases, our technique succeeds in limiting false negatives, thereby increasing recall. This observation is important since the proposed detector is robust even with challenging low amplitude targets. In the following we attempt to explain the observation of increasing detection performance with challenging signal situations.

We believe that this property is due to the increasing number of multipath components of the reflected signal, which increases the entropy within the corresponding signal. In fact, as illustrated in figure 4.14, the number of reflected signals goes higher with distant objects. While in the case of further objects the overall reflected signal amplitude is lower, magnitude-based techniques either loose in sensitivity or keep the same levels, our technique takes advantage from the differential entropy enhanced by the reflected multipath components. This observation is obviously limited by the radar range. Furthermore, the MODP results related to the implemented detectors are reported in Table 4.3. The presented performances show that our detector can correctly detect targets position more than the other techniques with a MODP that reaches 0.42.

Finally, in terms of complexity, the execution time to detect obstacles in a received signal using the implemented detectors are reported in Table 4.3. From the results, we can observe that our proposed detector has the lowest execution time.

Discussion

An ITS is a safety-critical domain where achieving robust automatic environment perception is a challenging keystone. While most of the environment perception systems are based on camera and lidar sensors, UWB radars show interesting characteristics such as robustness to weather and luminosity challenges. This aspect could be complementary to other modalities and promising to enhance computer vision reliability.

In this work, an entropy-based ROI identification approach for UWB radar detector is proposed. The proposed technique is designed to identify relevant information in the received signal, hence, to differentiate a real object from noise based on their respective vehiculated entropy.

In fact, the entropy-based segmentation method for UWB radar signals technique exploits signal entropy instead of amplitude to localize useful parts of the signal, thereby detecting obstacles. A threshold based on the maximization of the F1-score is determined.

Our results show an overall improvement of detection performance using our technique compared with related work. In a detailed look at the results we made a surprising property of our technique that consists of an increase in detection robustness with lower amplitudes, and consequently further obstacles from the sensor. Although above the considered objects' distance, the method will be limited as Signal-to-noise ratio (SNR) will be very low, in our case we are interested in detecting near-range obstacles. For this reason, even with low amplitude, the entropy-based method can differentiate between noise and real target. This aspect is due to the wireless signal propagation multipath components on the signal entropy. On the other hand, there is room for improvement. In this section, we propose the first framework that exploits UWB signals with LSTM network in an outdoor environment involving complex urban driving situations for multitarget detection. As the UWB received signal is a time-series data, the exploitation of a RNNs is suitable to exhibit the temporal dependencies. Therefore, the main intention of the developed network is to discriminate the real target from noise within an UWB received signal. A comparison between our proposed approach and the state-of-the-art techniques based on expanded experiments using OLIMP is detailed in the following.

Proposed LSTM-based detector

In this section, we present firstly the background of the RNNs and LSTM networks, followed by a description of the proposed approach.

RNNs for sequential data

The independence among the data samples is one of the fundamental assumption for neural networks. Nevertheless, this assumption does not deal with sequential data as speech, video, time series, etc. The individual elements of this data exhibit dependency across time. Besides, the neural networks treat every data sample independently and thereby suffer the loss of benefiting from the exploitation of the sequential information.

In addition, another disadvantage of the employment of neural networks is that they are not able to handle variable length sequences. For various domains like language translation or speech modeling, the sequences vary in length.

For these reasons, RNNs are introduced as a type of neural networks that are suitable for processing sequential data. In fact, this type of network processes the input sequence one element at a time and maintains a hidden state vector that serves as a memory for past information. RNNs learn to selectively conserve relevant information to capture dependencies within several time steps.

The architecture of RNN is shown in figure 4.15. The RNN has a feedback connection that connects the hidden neurons across time. At time t, it receives as input the current sequence's element x t as input and the s t-1 the hidden state extracted from the previous time step. Afterwards, the hidden state is updated to s t and h t is calculated (the output of the network). U is the weight matrix that links the input and the hidden layers similar to a conventional neural network. The weight matrix related to the recurrent transition is presented by W, it links a hidden state to the next. V presents the weight matrix for hidden to the output transition. The RNN is trained using the back-propagation through time (BPTT) [START_REF] Paul | Backpropagation through time: what it does and how to do it[END_REF] in order to learn long-range dependencies across long intervals. However, in practice training RNNs is a difficult process [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF]. In fact, training the RNN with BPTT requires back-propagating the error gradients through several steps. According to the RNN shown in figure 4.15, the recurrent edge has the same weight in each time step. Hence, back-propagating the error implicates multiplying the error gradient together with the same value repeatedly. This causes that the gradient to either decay to zero or become too large with respect to the layers' number [START_REF] Hochreiter | Gradient flow in recurrent nets: the difficulty of learning long-term dependencies[END_REF]. These aforementioned problems are referred to vanishing gradients and exploding gradients respectively.

Consequently, numerous methods have been proposed to deal with the problems of learning long-term dependencies when training RNNs as the gradient clipping method that has proven to be effective to the exploding gradient problem [START_REF] Pascanu | On the difficulty of training recurrent neural networks[END_REF]. Subsequently, the LSTM network has been introduced to overcome the vanishing problem due to the replacement of an ordinary neuron by a complex architecture entitled the LSTM unit [START_REF] Hochreiter | Long short-term memory[END_REF]. The LSTM network has become the popular variant of the RNNs.

LSTM background

The LSTM network is a special architecture of the artificial RNNs developed in 1997 by Hochreiter and Schmidhuber [START_REF] Hochreiter | Long short-term memory[END_REF]. It has been introduced to avoid the issues that occur when modeling long-term dependencies with RNN such as the vanishing or the exploding gradient problems. Therefore, the LSTM network is able to learn short-time as well as long-term dependencies. It is especially known by its effectiveness to treat time-series data [2]. In other words, the LSTM network is able to model the temporal changes in a series of data owing to its memory units and recurrent architecture. The LSTM units are connected sequentially. Each LSTM cell includes its own memory with three gates: the input, the output and the forget gates. These gates are responsible of protecting and controlling the flow of information through the cell. Otherwise, they decide which information has to be forgetting or reminded. These gates are detailed in the following:

• The input gate: It can allow the incoming signal to block the state of the memory cell or to change it.

• The output gate: It can authorize the state of the memory cell modify the other neurons or prevent it.

• The forget gate: It can let the cell to forget or remember its previous state, as required.

4.5

-89

An LSTM cell can be expressed by Equation 4.11 and illustrated in figure 4. [START_REF] Borowska | Entropy-based algorithms in the analysis of biomedical signals[END_REF].

           I t = σ(W I [H t-1 , X t ] + b I ) F t = σ(W F [H t-1 , X t ] + b F ) O t = σ(W O [H t-1 , X t ] + b O ) C t = F t * C t-1 + I t * tanh(W C [H t-1 , X t ] + b C ) H t = O t * tanh(C t ) Y t = so f tmax(W Y H t + b Y )            (4.11)
where:

X={X(1),X(2),..,X(N)} is a an input sequence, where N is the length of the time series sequence.

I t , F t , O t and C t are respectively the input, forget gate, the output gates and the memory cell state.

H t is the cell output and Y t is the final output.

tanh and σ are respectively the hyperbolic tangent and the logistic sigmoid activation functions.

W and b are respectively the input weights associating the LSTM cell to the inputs, and the bias vectors.

In fact, if a new input comes and the input gate I t is activated, a new information will be added to the cell. Moreover, if the forget gate F t was activated, the past cell status C t-1 could be forgotten. The output gate O t controls either the last cell output C t propagated into the final state H t or not. The Nonlinear sigmoid σ = (1e -1 ) -1 outputs values between zero and one, zero indicates that "let nothing through," while one means "let everything through!". Thus, the LSTM architecture utilizes the memory cells to use and store information, to identify the long-range temporal relations [START_REF] Neil | Phased lstm: Accelerating recurrent network training for long or event-based sequences[END_REF].

Proposed UWB-based system for obstacle detection

From the one side, despite the fact that UWB reflected signal incorporates rich information, the discrimination between the object's signature from noise is a fundamental challenge. In fact, as we already mentioned that the UWB received signal r(t) can be modulated following Equation 4.12.

r(t) = s(t) + n(t) (4.12)
where n(t) is the noise of the transmission channel of UWB radar and s(t) is the received echo.

Therefore, based on the UWB property that indicates that each obstacle has its own signature, noise also should be different.

From the other side, the target's wave reflection amplitude is generally changing over time due to physical characteristics (material, shape, size, etc.). Otherwise, the received echo is a function of time, and also a function of the obstacle distance from the sensor. This time-distance relation is expressed by Equation 4.1.

Hence, on account of the temporal changes presented in the UWB signal we adopt the idea of employing LSTM network. This choice is made on account of the fact that this type of RNN is able to recognize and synthesize the dynamics variations within the UWB received echo. Thereby, in this work we propose the first framework that exploits LSTM network with UWB signals for distinguishing obstacles from noise in a vehicle environment perception context.

The proposed framework is explained in Algorithm 2 and illustrated in figure 4.17. Primarily, the received echo is split into time series sequences [x t ]. These sequences present the useful regions which contain the real targets (illustrated by the orange windows in figure 4.17) and noise partitions (depicted by green rectangles in figure 4.17). For data variety, the noise parts are randomly selected from the UWB signal. In fact, the window's size (Wsig) is empirically set according to the objects' signature length. Afterwards, features from the time-frequency domain are extracted from the defined regions. Thus, the discrete wavelet transform (DWT) is utilized. We extract four features from the approximation coefficients (Ca) and the detail ones (Cd) for each sequence. Subsequently, the fattened 1D descriptor vector feeds the LSTM network. Finally, the output of the LSTM is linked to the fully connected layer of size two followed by a Softmax layer, and a classification layer.

To conclude, the proposed detector analyzes the temporal changes within the UWB signal via learning the extracted time-frequency features that highly present the signal characteristics.

Experimental setup

To highlight the efficiency of the proposed architecture, we compare it to the implemented state-of-the art techniques: CFAR, HOS and the work in [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF]. The experimental results are evaluated using the P, R and the F1-score metrics.

The details of the implementation are illustrated below:

• Network architecture : We exploit the unidirectional network with 100 LSTM hidden units. The number of epochs and mini batch size during experiments are set to 100 and 64 instances respectively

• Feature extraction : To extract features using the discrete wavelet transform, the Daubechies wavelet is employed to extract the features from the time series sequences.

• Dataset : We conducted our experiments on a variety of the OLIMP urban driving scenarios.

• Training : For our experiments, we use 2/3 of the data for training step and 1/3 for the test process following the OLIMP protocol. Moreover, we utilized the Adam Optimizer [START_REF] Bock | An improvement of the convergence proof of the ADAM-Optimizer[END_REF]. The initial learning rate is set to 0.001.

• Comparative techniques : For HOS we take advantage of the 4 th order cumulant that relies on Tuganit4 algorithm. Concerning CFAR, CA-CFAR detector is considered with automatic threshold. These techniques are already explained in Section 4.4.4.

It shall be mentioned that for training process, The optimal set of parameters has been selected based on preliminary tests.

Results

The experimental results concerning the comparative study are summarized in figure 4. [START_REF] Gabriel | Segmentation and recognition using structure from motion point clouds[END_REF]. In fact, it can be seen from the figure that the obtained results show that our deep learning-based method achieves the highest performance. Our proposed method The performance of CA-CFAR, HOS and their combination in the work of [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] depend essentially on the definition of the threshold parameter. A higher threshold generates more false negatives, however, the opposite case produces additional false positives. Furthermore, the object detection rate using the aforementioned techniques rely on the object's amplitude. In low magnitude cases, which means that the object is moving away from the radar, the target can't be detected and it is considered as noise.

The LSTM-based method can distinguish noise from real targets thanks to the relevant features that are extracted from the time-frequency domain, and by learning the temporal relationship between the data sequences. In fact, the time-frequency features lead to a high-performing as they can represent well the signal's characteristics. It shall be mentioned that, some missclassifications are still present due the challenges related to the interclass similarity of the obstacle's signature and the noise signal in cases where the object's signature has a low amplitude.

In terms of complexity, table 4.4 depicts a comparison of the execution time of our method with the state-of-the-art techniques. As it can be seen from the results, our architecture has the highest execution time, though, advanced hardware resources are to be deployed to acquire faster detection time. In this chapter, we tackled the problem of differentiating real objects from noise for the environment perception purpose using UWB radar. For this purpose, we proposed to explore an entropy-based method and a deep-based framework. We compared the proposed approaches to the well-known techniques HOS, CA-CFAR and [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and experiments were conducted on our previously developed dataset OLIMP.

Even though the main aim is to discriminate the useful portions within the signal from noise and to detect the obstacle, the two proposed detectors treat the dilemma differently. On the one hand, the entropy-based method exploits the signal's complexity instead of the amplitude which the state-of-the art techniques rely on it. This leads us to prove that our entropy-detector remains robust even with lower amplitude cases. As a future work, it should be pointed out that an adaptive thresholding could be considered in our case.

On the other hand, the LSTM-based detector aims to localise the real obstacles by learning and synthesizing the temporal variations within the UWB received data sequences. This detector achieves higher results in terms of recall, but not in terms of precision compared to our proposed entropy-based detector. In terms of complexity, the execution time of the entropy-based method is faster to detect the obstacles compared with the LSTM-based as it is a time consuming (deep-based method) and all the implemented methods.

While the LSTM-based network has proven to be particularly powerful to solve noise and real target distinguishing problems, the entropy characterises the UWB signal, therefore, a hybrid approach that combines these methods could be interesting to compensate their limitations and obtain a better detector.

4.7

In this chapter, we focused on the segmentation of 1D UWB signals. Thus, we reviewed the existing methods to detect obstacle using UWB radar. Afterwards, we proposed an entropy-based detector based on a theoretical study related to the Shannon entropy. A comparative study has been conducted to evaluate our entropy method by employing HOS, CA-CFAR and [START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] techniques. In addition, as a deep-based method, we proposed the first framework that exploits LSTM network to distinguish real target from noise using UWB radar. According to the obtained results that surpass the state-of-the-art techniques, the proposed systems constitute an important step towards distinguishing real obstacles from noise. As future work, we believe that an adaptive hybrid segmentation technique could be interesting and may achieve high detection performance.

Conclusions and contributions

Environment perception is one of the key challenges in automated driving applications especially in the quest for higher degrees of automation. These systems represent the the forthcoming of transportation. In fact, early ADASs have been developed and improved over many generations thanks to the better performance of the sensors and their enhancement of processing algorithms. In the recent years, sensor data fusion becomes a key aspect in developing such systems to describe the complete vehicle environment optimally and efficiently. Therefore, since the level of automation increases and ADASs become complicated, reliable environment perception is required. Moreover, developing accurate and reliable systems will always be addressed particularly for next-generation automated systems as they guarantee safety.

To address this challenge, in this thesis, we have focused on environment perception. Particularly, we have tackled the problem of detecting multiple objects using data fusion or a unique sensor. This thesis integrates four principal contributions.

Firstly, we have reviewed the environment perception state-of-the-art for intelligent and autonomous vehicles, specifically the object detection task which includes the relevant sensors, the data fusion methods and the challenges. Afterwards, by reviewing the existing public multi-modal environment perception databases, we have introduced OLIMP dataset. It is the first synchronized dataset that includes these four modalities: images, UWB radar signals, narrow band data streams and acoustic data.

We have also presented a new fusion framework that combines data acquired from different sensors used in our dataset to achieve better performances for the obstacle detection task. Various levels of fusion are exploited and promising results are found. These results undoubtedly allow us to conclude that multimodality is indeed to guarantee an accurate environment perception.

Afterwards, we have tackled the problem of multi-obstacle detection in short-range settings using the UWB radar as it provides rich information about the vehicle's surroundings. Thus, two detectors are proposed.

The first detector is an entropy-based segmentation approach. It exploits the signal entropy instead of the amplitude to localize the useful UWB parts. Based on exhaustive experiments and a comparative study with the state-of-the-art techniques our method remains robust even with challenging low amplitude signals. Furthermore, our detector's performance is improved when the obstacle is far from the sensor. This observation overcomes the fundamental challenge in related techniques.

The second detector is a deep learning-based framework. It is the first framework that exploits LSTM with UWB signals for multi obstacle detection in an outdoor complex environment. By learning the temporal dependencies within UWB series of data, the LSTM-based detector outperforms the conventional techniques.

Finally, based on the aforementioned contributions made in this thesis dealing with the problem of object detection, we conclude that for sensor data fusion architectures there is not a dominant level of fusion. In fact, it depends particularly on the targeted ADAS applications, the employed sensors and the environment. In addition, in urban environments, the environment perception task is more challenging as they are very dense and it is difficult to separate objects near each other.

Perspectives

Despite these advances made through our contributions, the complete environment perception remains an open topic of research and several improvements can be envisaged. principal limitations to be considered and the perspectives are presented as follows.

• Sensor fusion

The proposed fusion framework have shown that data fusion at different levels offer a higher performance than employing a unique sensor. The proposed fusion framework is limited because of its sequential aspect. We believe that this could be improved using advanced parallel fusion systems. This will be investigated in future work.

Despite the fact that the suggested fusion detection method deals with the limitations of each modality, only the detection of false positives is considered. Incorporating the cases of false negatives in the fusion framework can be explored as future work to obtain a more reliable object detection system. Furthermore, the development of a new deep learning-based architecture that fuses narrow-band data streams and images is our current field of research.

• Entropy-based detector

The entropy-based segmentation method for UWB radar signals exploits the signal entropy. It relies on the detection of a threshold that allows the distinction of useful signals from noise. It should be pointed out that adaptive thresholding could be considered in our case.

Even with a low amplitude, the entropy-based method can differentiate between noise and real targets. However, we believe that an adaptive hybrid segmentation technique could be interesting and may achieve a high detection performance.

• LSTM-based detector

We have shown that analyzing the temporal data changes among UWB signals for the object detection process provide higher performances by exploiting RNNs. Promising results are found using the LSTM-based detector which outperforms the performances of the related techniques. It shall be mentioned that some missclassifications are still present. Accordingly, further research can be expanded to extract deep features to tackle this dilemma. For this reason, the employment of CNNs to extract features from the UWB signals can be investigated.

From another perspective, an hybrid approach that combines the entropy-based detector and the LSTM one could be interesting to compensate limitations and obtain a better detector.

Part I B I B L I O G R A P H Y

humaines qui sont principalement la conséquence soit de la fatigue, l'accélération, la perte de concentration ou aussi la conduite immature, etc. Pour toutes ces raisons, la conception des voitures autonomes devient actuellement une nécessité incontournable. En effet, un véhicule autonome est une voiture intelligente qui peut circuler et prendre la décision sans aucune intervention humaine. De nos jours, plusieurs travaux de recherche s'orientent vers cette nouvelle thématique qui devient un objectif principal de l'industrie automobile. Il existe plusieurs technologies qui sont déjà implémentées dans les véhicules tels que : l'avertissement de sortie de la voie, la reconnaissance des panneaux de signalisation, la régulation de la vitesse adaptative, etc. Ces fonctionnalités donnent une perception complète sur l'environnement de la voiture, pour cela, elles ont été employées dans le but de garantir la sécurité des usagers de la route d'une part, et des occupants des voitures d'autre part. De plus, elles assurent la conduite écologique tout en mettant au point la gestion du trafic et donc moins de stress et plus de confort et de productivité. Plusieurs types de capteurs tels que : le LIDAR, le radar, la caméra, le capteur ultrason, etc., sont embarqués, aujourd'hui, dans les systèmes de transport intelligents en vu d'assurer diverses fonctionnalités.

2-Problématique et objectifs de la thèse

De l'évolution de la première voiture à essence au 19 ème siècle à l'invention des voitures autonomes, plusieurs solutions technologiques ont été développées pour rendre les véhicules de demain plus intelligents, assurer la sécurité routière, le confort, la gestion du trafic, etc. Conséquemment, la détection multi-objets est considérée comme une tâche cruciale pour la conception et le développement des voitures autonomes, en faveur d'avoir plus de précision et de prendre la meilleure décision. Etant donné que l'aspect de ce domaine d'application est crucial, nous nous sommes orientés vers l'exploitation du deep learning, puisque ces techniques ont révolutionné l'intelligence artificielle dans divers domaines de recherche. Il existe plusieurs algorithmes d'apprentissage profond employés, citons : les réseaux de neurones convolutionnels (CNN-Convolutional neural network), Fast R-CNN, Faster R-CNN, Yolo, etc. Parmi ces réseaux, les CNNs sont souvent employés dans le secteur des systèmes de transport intelligent vu leur meilleure performance pour la détection des obstacles. En effet, un CNN utilise directement une image en entrée, et extrait hiérarchiquement les caractéristiques pour la classification. L'objectif principal est d'exploiter l'image brute en appliquant différents filtres pour aboutir à une représentation hiérarchique de haut niveau d'une couche à une autre.

Le transport a évolué grâce à l'implémentation de plusieurs systèmes ADAS (Advanced driverassistance systems) dans les voitures. Pour ce faire, divers capteurs sont employés et peuvent être classés en deux catégories : actifs (radar, lidar, etc.) et passifs (caméra, etc.). La fusion de données issues de divers capteurs est fréquemment exploitée dans ce domaine de recherche pour une perception complète de l'environnement. Cependant, les travaux traitant le couplage des ondes radar et des images sont restreints. Dans nos travaux de thèse, nous nous intéressons à la fusion des données issues du radar et de la caméra vu leur complémentarité. Plusieurs challenges sont liés à cette tâche, citons : les conditions dégradées, les occultions, la variété interclasse (piétons, cyclistes, voitures, etc.) et intra classe, etc. De plus, les bases de données reliées à ce domaine sont limitées puisque ce sujet attire plus l'attention des industriels dans la filière de l'automobile comme Tesla, BMW, Google, Apple, etc.

L'objectif principal de notre thèse est de doter une voiture par une capacité de perception assez complète lui permettant d'assurer la compréhension de son environnement. De ce fait, dans nos travaux de recherche, les tâches principales sont les suivantes :

-La reconnaissance des obstacles et de l'environnement.

-La détection des piétons et la reconnaissance des situations dangereuses.

Travaux réalisés et résultats obtenus (pendant la 3ème année)

Durant la troisième année de thèse, les travaux réalisés et les résultats obtenus sont les suivants :

▪

Figure 4 . 14

 414 Figure 3.12 ROIs selection using UWB and narrow-band data. . . . . . . . Figure3.13 Radar-based system results . . . . . . . . . . . . . . . . . . . Figure 3.14 Acoustic-based system results (%) . . . . . . . . . . . . . . . Figure 3.15 Proposed fusion framework architecture . . . . . . . . . . . . Figure 3.16 Estimation of object distance using area of detected bounding boxes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.17 Fusion framework results (%) . . . . . . . . . . . . . . . . . . Figure 3.18 Detection results before and after fusion . . . . . . . . . . . . Figure 4.1 Target range calculation using automotive radar. . . . . . . . . Figure 4.2 Used HST-D3 UWB radar hardware specifications. . . . . . . Figure 4.3 Illustration of the entropy variation of two different received UWB signals. . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.4 Comparative distribution of noise signals, and signals in presence of an obstacle. . . . . . . . . . . . . . . . . . . . . . . . Figure 4.5 Entropy-based detector architecture . . . . . . . . . . . . . . . Figure 4.6 CA-CFAR algorithm architecture [113] . . . . . . . . . . . . . Figure 4.7 Threshold definition based on exhaustive space exploration. The threshold corresponds to the highest F1-score. . . . . . . . Figure 4.8 Experimental results: Precision, Recall and F1-score using HOS, CA-CFAR, [141] and our method. . . . . . . . . . . . . . . . . Figure 4.9 Experimental results for pedestrian detection using HOS, CA-CFAR, [141] and our method. . . . . . . . . . . . . . . . . . . Figure 4.10 Illustration of multiple-target detection limitations using HOS, CA-CFAR and Entropy detectors . . . . . . . . . . . . . . . . Figure 4.11Precision results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure4.[START_REF] Blake | OS-CFAR theory for multiple targets and nonuniform clutter[END_REF] Recall results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . . . Figure4.13 F1-score results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure 4.14 Illustration of multipath component vs distance: (a) close object with high amplitude, (b) object in further location with low amplitude. . . . . . . . . . . . . . . . . . . . . . . . . . . . .Figure 4.15 A standard RNN architecture. The left side of the figure represents a standard RNN [157]. . . . . . . . . . . . . . . . . . . .Figure 4.16 Architecture of LSTM unit . . . . . . . . . . . . . . . . . . .Figure 4.17 Proposed LSTM-based framework using UWB signals . . . . .Figure 4.18 Experimental results using HOS, CFAR, the work in [141] and our method . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 15 A

 415 Figure 3.12 ROIs selection using UWB and narrow-band data. . . . . . . . Figure3.13 Radar-based system results . . . . . . . . . . . . . . . . . . . Figure 3.14 Acoustic-based system results (%) . . . . . . . . . . . . . . . Figure 3.15 Proposed fusion framework architecture . . . . . . . . . . . . Figure 3.16 Estimation of object distance using area of detected bounding boxes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.17 Fusion framework results (%) . . . . . . . . . . . . . . . . . . Figure 3.18 Detection results before and after fusion . . . . . . . . . . . . Figure 4.1 Target range calculation using automotive radar. . . . . . . . . Figure 4.2 Used HST-D3 UWB radar hardware specifications. . . . . . . Figure 4.3 Illustration of the entropy variation of two different received UWB signals. . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.4 Comparative distribution of noise signals, and signals in presence of an obstacle. . . . . . . . . . . . . . . . . . . . . . . . Figure 4.5 Entropy-based detector architecture . . . . . . . . . . . . . . . Figure 4.6 CA-CFAR algorithm architecture [113] . . . . . . . . . . . . . Figure 4.7 Threshold definition based on exhaustive space exploration. The threshold corresponds to the highest F1-score. . . . . . . . Figure 4.8 Experimental results: Precision, Recall and F1-score using HOS, CA-CFAR, [141] and our method. . . . . . . . . . . . . . . . . Figure 4.9 Experimental results for pedestrian detection using HOS, CA-CFAR, [141] and our method. . . . . . . . . . . . . . . . . . . Figure 4.10 Illustration of multiple-target detection limitations using HOS, CA-CFAR and Entropy detectors . . . . . . . . . . . . . . . . Figure 4.11Precision results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure4.[START_REF] Blake | OS-CFAR theory for multiple targets and nonuniform clutter[END_REF] Recall results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . . . Figure4.13 F1-score results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure 4.14 Illustration of multipath component vs distance: (a) close object with high amplitude, (b) object in further location with low amplitude. . . . . . . . . . . . . . . . . . . . . . . . . . . . .Figure 4.15 A standard RNN architecture. The left side of the figure represents a standard RNN [157]. . . . . . . . . . . . . . . . . . . .Figure 4.16 Architecture of LSTM unit . . . . . . . . . . . . . . . . . . .Figure 4.17 Proposed LSTM-based framework using UWB signals . . . . .Figure 4.18 Experimental results using HOS, CFAR, the work in [141] and our method . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 16

 416 Figure 3.12 ROIs selection using UWB and narrow-band data. . . . . . . . Figure3.13 Radar-based system results . . . . . . . . . . . . . . . . . . . Figure 3.14 Acoustic-based system results (%) . . . . . . . . . . . . . . . Figure 3.15 Proposed fusion framework architecture . . . . . . . . . . . . Figure 3.16 Estimation of object distance using area of detected bounding boxes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.17 Fusion framework results (%) . . . . . . . . . . . . . . . . . . Figure 3.18 Detection results before and after fusion . . . . . . . . . . . . Figure 4.1 Target range calculation using automotive radar. . . . . . . . . Figure 4.2 Used HST-D3 UWB radar hardware specifications. . . . . . . Figure 4.3 Illustration of the entropy variation of two different received UWB signals. . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.4 Comparative distribution of noise signals, and signals in presence of an obstacle. . . . . . . . . . . . . . . . . . . . . . . . Figure 4.5 Entropy-based detector architecture . . . . . . . . . . . . . . . Figure 4.6 CA-CFAR algorithm architecture [113] . . . . . . . . . . . . . Figure 4.7 Threshold definition based on exhaustive space exploration. The threshold corresponds to the highest F1-score. . . . . . . . Figure 4.8 Experimental results: Precision, Recall and F1-score using HOS, CA-CFAR, [141] and our method. . . . . . . . . . . . . . . . . Figure 4.9 Experimental results for pedestrian detection using HOS, CA-CFAR, [141] and our method. . . . . . . . . . . . . . . . . . . Figure 4.10 Illustration of multiple-target detection limitations using HOS, CA-CFAR and Entropy detectors . . . . . . . . . . . . . . . . Figure 4.11Precision results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure4.[START_REF] Blake | OS-CFAR theory for multiple targets and nonuniform clutter[END_REF] Recall results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . . . Figure4.13 F1-score results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure 4.14 Illustration of multipath component vs distance: (a) close object with high amplitude, (b) object in further location with low amplitude. . . . . . . . . . . . . . . . . . . . . . . . . . . . .Figure 4.15 A standard RNN architecture. The left side of the figure represents a standard RNN [157]. . . . . . . . . . . . . . . . . . . .Figure 4.16 Architecture of LSTM unit . . . . . . . . . . . . . . . . . . .Figure 4.17 Proposed LSTM-based framework using UWB signals . . . . .Figure 4.18 Experimental results using HOS, CFAR, the work in [141] and our method . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 .

 4 Figure 3.12 ROIs selection using UWB and narrow-band data. . . . . . . . Figure3.13 Radar-based system results . . . . . . . . . . . . . . . . . . . Figure 3.14 Acoustic-based system results (%) . . . . . . . . . . . . . . . Figure 3.15 Proposed fusion framework architecture . . . . . . . . . . . . Figure 3.16 Estimation of object distance using area of detected bounding boxes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.17 Fusion framework results (%) . . . . . . . . . . . . . . . . . . Figure 3.18 Detection results before and after fusion . . . . . . . . . . . . Figure 4.1 Target range calculation using automotive radar. . . . . . . . . Figure 4.2 Used HST-D3 UWB radar hardware specifications. . . . . . . Figure 4.3 Illustration of the entropy variation of two different received UWB signals. . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.4 Comparative distribution of noise signals, and signals in presence of an obstacle. . . . . . . . . . . . . . . . . . . . . . . . Figure 4.5 Entropy-based detector architecture . . . . . . . . . . . . . . . Figure 4.6 CA-CFAR algorithm architecture [113] . . . . . . . . . . . . . Figure 4.7 Threshold definition based on exhaustive space exploration. The threshold corresponds to the highest F1-score. . . . . . . . Figure 4.8 Experimental results: Precision, Recall and F1-score using HOS, CA-CFAR, [141] and our method. . . . . . . . . . . . . . . . . Figure 4.9 Experimental results for pedestrian detection using HOS, CA-CFAR, [141] and our method. . . . . . . . . . . . . . . . . . . Figure 4.10 Illustration of multiple-target detection limitations using HOS, CA-CFAR and Entropy detectors . . . . . . . . . . . . . . . . Figure 4.11Precision results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure4.[START_REF] Blake | OS-CFAR theory for multiple targets and nonuniform clutter[END_REF] Recall results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . . . Figure4.13 F1-score results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure 4.14 Illustration of multipath component vs distance: (a) close object with high amplitude, (b) object in further location with low amplitude. . . . . . . . . . . . . . . . . . . . . . . . . . . . .Figure 4.15 A standard RNN architecture. The left side of the figure represents a standard RNN [157]. . . . . . . . . . . . . . . . . . . .Figure 4.16 Architecture of LSTM unit . . . . . . . . . . . . . . . . . . .Figure 4.17 Proposed LSTM-based framework using UWB signals . . . . .Figure 4.18 Experimental results using HOS, CFAR, the work in [141] and our method . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 18

 418 Figure 3.12 ROIs selection using UWB and narrow-band data. . . . . . . . Figure3.13 Radar-based system results . . . . . . . . . . . . . . . . . . . Figure 3.14 Acoustic-based system results (%) . . . . . . . . . . . . . . . Figure 3.15 Proposed fusion framework architecture . . . . . . . . . . . . Figure 3.16 Estimation of object distance using area of detected bounding boxes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.17 Fusion framework results (%) . . . . . . . . . . . . . . . . . . Figure 3.18 Detection results before and after fusion . . . . . . . . . . . . Figure 4.1 Target range calculation using automotive radar. . . . . . . . . Figure 4.2 Used HST-D3 UWB radar hardware specifications. . . . . . . Figure 4.3 Illustration of the entropy variation of two different received UWB signals. . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.4 Comparative distribution of noise signals, and signals in presence of an obstacle. . . . . . . . . . . . . . . . . . . . . . . . Figure 4.5 Entropy-based detector architecture . . . . . . . . . . . . . . . Figure 4.6 CA-CFAR algorithm architecture [113] . . . . . . . . . . . . . Figure 4.7 Threshold definition based on exhaustive space exploration. The threshold corresponds to the highest F1-score. . . . . . . . Figure 4.8 Experimental results: Precision, Recall and F1-score using HOS, CA-CFAR, [141] and our method. . . . . . . . . . . . . . . . . Figure 4.9 Experimental results for pedestrian detection using HOS, CA-CFAR, [141] and our method. . . . . . . . . . . . . . . . . . . Figure 4.10 Illustration of multiple-target detection limitations using HOS, CA-CFAR and Entropy detectors . . . . . . . . . . . . . . . . Figure 4.11Precision results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure4.[START_REF] Blake | OS-CFAR theory for multiple targets and nonuniform clutter[END_REF] Recall results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . . . Figure4.13 F1-score results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method. . . Figure 4.14 Illustration of multipath component vs distance: (a) close object with high amplitude, (b) object in further location with low amplitude. . . . . . . . . . . . . . . . . . . . . . . . . . . . .Figure 4.15 A standard RNN architecture. The left side of the figure represents a standard RNN [157]. . . . . . . . . . . . . . . . . . . .Figure 4.16 Architecture of LSTM unit . . . . . . . . . . . . . . . . . . .Figure 4.17 Proposed LSTM-based framework using UWB signals . . . . .Figure 4.18 Experimental results using HOS, CFAR, the work in [141] and our method . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Contents 1 . 1

 11 Thesis context . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Motivations and objectives . . . . . . . . . . . . . . . . . . . . . . 1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 . 1 3Figure 1 . 1 :

 1111 Figure 1.1: Distribution of global injury mortality by cause according to WHO Global Burden of Disease project [121].

Figure 1 . 2 :

 12 Figure 1.2: Serious crashes caused by AV.

Figure 2 . 1 :

 21 Figure 2.1: Levels of driving automation according to SAE J3016 [89]

Figure 2 . 2 :

 22 Figure 2.2: Typical autonomous vehicle framework

Figure 2

 2 Figure 2.3: ADAS

-

  Provide distance, velocity without any computing resources -Robust in all weather conditions (fog, rain, dust) and during nighttime -Generate false alarms easily -

Figure 2 . 4 :

 24 Figure 2.4: Sensors employed for ITSs

Figure 2 .

 2 7 shows the performance evolution for basic CNN-based architectures from 2012 to 2019 evaluated on the ImageNet 2012 dataset. This comparison is based on the achieved accuracy and the number of parameters illustrated with a varied circle. The mixed colors in the figure indicate that two architecture kinds are combined [69].

Figure 2 . 7 :

 27 Figure 2.7: Performance evaluation for basic CNN-based architecture from 2012 to 2019 carried on ImageNet 2012 dataset [69]

Figure 2 . 8 :

 28 Figure 2.8: Data fusion levels: (a) early fusion, (b) intermediate fusion, and (c) late fusion

Figure 2 . 9 :

 29 Figure 2.9: Intersection-over-union (IOU) metric samples of single object.

  C E P T I O N Contents 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Existing public multimodal environment perception databases . . . 3.3 Proposed dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Hardware and data acquisition . . . . . . . . . . . . . . . 3.3.3 Sensors embedding . . . . . . . . . . . . . . . . . . . . . . 3.3.4 Sensor synchronisation . . . . . . . . . . . . . . . . . . . . 3.3.5 Labeling process . . . . . . . . . . . . . . . . . . . . . . 3.3.6 Scenario selection and data formats . . . . . . . . . . . . . 3.3.7 Dataset challenges . . . . . . . . . . . . . . . . . . . . . . 3.3.8 Statistics and dataset organisation . . . . . . . . . . . . . . 3.4 Fusion framework . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Image-based system . . . . . . . . . . . . . . . . . . . . . 3.4.2 Radar-based system . . . . . . . . . . . . . . . . . . . . . 3.4.3 Acoustic-based system . . . . . . . . . . . . . . . . . . . 3.4.4 Multi-modality fusion system . . . . . . . . . . . . . . . . 3.4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  detection Object tracking Optical flow Lane detection Semantic segmentation SLAM 3D

Figure 3 . 2 :

 32 Figure 3.2: Data acquisition architecture

  camera, RGB images, 25 FPS, 640x480 resolution, angle of view: 65 • -25 • 250 range, provides distance; velocity and RCS, 77GHz, ±0.40 m accuracy for far range, ±0.10 m accuracy for near range 72 ms Microphone CM-520 -≤ to 20 range, +10dB sensitivity, it fits well with video cameras, 50Hz-16Khz for frequency response Not ApplicableTable 3.3: Sensor specifications and properties. Measure latency is the time necessary to collect one complete data stream from the sensor.

Figure 3 . 3 :

 33 Figure 3.3: Structure setup

Figure 3 . 4 :

 34 Figure 3.4: Frames synchronisation (ST: object stream, n: number of obstacles in the scene).

Figure 3 . 5 :

 35 Figure 3.5: Recording emplacements at the University Polytechnic Hauts-de-France

Figure 3 . 6 :

 36 Figure 3.6: Challenges presented in our dataset: a) weather conditions, b) lighting variation, c) occlusions, and d) object types

Figure 3 . 7 :Figure 3 . 8 :

 3738 Figure 3.7: Object signatures extracted from UWB signals (near obstacles)

Figure 3 . 9 :

 39 Figure 3.9: MobileNet-v2 building block

Figure 3 . 10 :

 310 Figure 3.10: MobileNet results (%)

Figure 3 . 11 :

 311 Figure 3.11: ERadar point cloud representation with optimisation using velocity.

Figure 3 . 12 :

 312 Figure 3.12: ROIs selection using UWB and narrow-band data.

Figure 3

 3 Figure 3.13: Radar-based system results

  fact, we identify the effectiveness of each sensor individually and its ability to differentiate one class of another according to the results presented in Section 3.4.1-3.4.3. The architecture of the proposed fusion framework is represented in figure3.15.

Figure 3 . 15 :

 315 Figure 3.15: Proposed fusion framework architecture

Figure 3 . 17 :

 317 Figure 3.17: Fusion framework results (%)

3 . 5 62Finally,Figure 3 . 18 :

 35318 Figure 3.18: Detection results before and after fusion

Contents 4 . 1

 41 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 UWB radar specifications . . . . . . . . . . . . . . . . . . . . . . 4.2.1 UMAIN radar . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Hand crafted-based detectors using UWB radar . . . . . . . 4.3.2 Deep learning-based detectors using UWB radar . . . . . . 4.4 Entropy-based detector . . . . . . . . . . . . . . . . . . . . . . . . 4.4.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.2 Theoretical basis . . . . . . . . . . . . . . . . . . . . . . . 4.4.3 Entropy-based segmentation . . . . . . . . . . . . . . . . 4.4.4 Experimental setup . . . . . . . . . . . . . . . . . . . . . 4.4.5 Threshold definition . . . . . . . . . . . . . . . . . . . . . 4.4.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 LSTM-based detector . . . . . . . . . . . . . . . . . . . . . . . . 4.5.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.2 Proposed LSTM-based detector . . . . . . . . . . . . . . . 4.5.3 RNNs for sequential data . . . . . . . . . . . . . . . . . . 4.5.4 LSTM background . . . . . . . . . . . . . . . . . . . . . 4.5.5 Proposed UWB-based system for obstacle detection . . . . 4.5.6 Experimental setup . . . . . . . . . . . . . . . . . . . . . 4.5.7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4 . 2 66Figure 4 . 1 :

 4241 Figure 4.1: Target range calculation using automotive radar.

Figure 4 . 2 :

 42 Figure 4.2: Used HST-D3 UWB radar hardware specifications.

  layers. Promising results are obtained. Jiang et al.[START_REF] Jiang | UWB NLOS/LOS Classification Using Deep Learning Method[END_REF] trained a CNN-LSTM to classify Line-Of-Sight (LOS) and None-Line-Of-Sight (NLOS) signals in the context of indoor positioning applications. The UWB channel impulse response is used as input of the CNN that deploys two conventional layers. Afterwards, the CNN outputs are linked to the LSTM network. Bi-direction LSTM and a stacked LSTM are used. The achieved accuracy is equal to 81%, but, the training dataset was limited.

Figure 4 . 3 :

 43 Figure 4.3: Illustration of the entropy variation of two different received UWB signals.

Figure 4 . 4 :Figure 4 . 5 :

 4445 Figure 4.4: Comparative distribution of noise signals, and signals in presence of an obstacle.

Figure 4 . 6 :

 46 Figure 4.6: CA-CFAR algorithm architecture[START_REF] Ni | Small Foreign Object Debris Detection for Millimeter-Wave Radar Based on Power Spectrum Features[END_REF]

Figure 4 . 7 :

 47 Figure 4.7: Threshold definition based on exhaustive space exploration. The threshold corresponds to the highest F1-score.

Figure 4 . 9 :Figure 4 . 10 :

 49410 Figure 4.9: Experimental results for pedestrian detection using HOS, CA-CFAR, [141] and our method.

Figures 4. 11

 11 Figures 4.11, 4.12 and 4.13 show respectively the precision, recall and F1-score of the different obstacle detection techniques as a function of the obstacle's amplitude (in the ground truth).

  Figures 4.11, 4.12 and 4.13 show respectively the precision, recall and F1-score of the different obstacle detection techniques as a function of the obstacle's amplitude (in the ground truth).

Figure 4 . 11 :

 411 Figure 4.11: Precision results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR, [141] and our method.

Figure 4 . 12 :

 412 Figure 4.12: Recall results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR,[START_REF] Sadli | UWB radar recognition system based on HOS and SVMs[END_REF] and our method.

Figure 4 .

 4 Figure 4.13: F1-score results over varying target amplitude in terms of P, R and F1-score using HOS, CA-CFAR, [141] and our method.

Figure 4 . 14 :

 414 Figure 4.14: Illustration of multipath component vs distance: (a) close object with high amplitude, (b) object in further location with low amplitude.

Figure 4 .

 4 Figure 4.15: A standard RNN architecture. The left side of the figure represents a standard RNN [157].

Figure 4 . 16 :

 416 Figure 4.16: Architecture of LSTM unit

4 . 5 - 92 Algorithmus 2 :Figure 4 . 17 : 5 - 94

 45922417594 Figure 4.17: Proposed LSTM-based framework using UWB signals

Figure 4 . 18 :

 418 Figure 4.18: Experimental results using HOS, CFAR, the work in [141] and our method

▪

  Proposition des systèmes uni-modaux exploitant chaque capteur seul. ▪ Exploitation du modèle pré-entraîné MobileNet-v2 sur la base de données ImageNet pour détecter les objets dans les images RGB. ▪ Extraction des caractéristiques temporelles des données acoustiques en utilisant MFCC (Mel-Frequency Coefficients Cepstral). Ces descripteurs sont classés en utilisant SVM avec le noyau RBF. Les résultats obtenus montrent que le système acoustique peut mieux différencier entre les deux catégories tram et véhicule. ▪ Identification de l'efficacité de l'utilisation de chaque capteur individuellement (radar, caméra et microphone) et sa capacité à différencier une classe d'une autre. ▪ Proposition et développement d'un Framework de fusion multimodal exploitant les données issues des différents capteurs utilisés. Ce Framework est développé en se basant sur les résultats obtenus à partir des systèmes uni-modaux proposés. Les performances ont été améliorées en termes de précision. ▪ Publication d'un papier présentant la base de données développée « A Heterogeneous Multimodal Dataset for Advanced Environment Perception : OLIMP » et intégrant le Framework proposé dans le journal Electronics. ▪ Publication en ligne de la base de données OLIMP. « https://sites.google.com/view/ihsen-alouani » ▪ Développement d'une approche basée sur l'entropie pour segmenter (localiser les régions d'intérêts) les signaux issus du radar ultra-large bande pour la détection multi-objet. La technique proposée est évaluée sur la base de données OLIMP. Les résultats obtenus montrent des performances supérieures par rapport aux techniques de l'état de l'art (Higher Order Statistics (HOS) and Constant false alarm rate (CFAR)). Le détecteur proposé est robuste même avec des signaux de faible amplitude. ▪ Soumission d'un article pour présenter la nouvelle approche dans le journal IEEE Sensors. ▪ Proposition d'un nouveau Framework qui exploite les réseaux de neurones récurrent (RNN) pour la détection de multiples obstacles routiers dans les signaux ULB. Veille et stratégie de recherche documentaire SESSION SPI (13 février 2020) Crédit : 2, UPHF ▪ Ethique de la recherche (15 juin 2020 -15 juin 2020) En visioconférence enregistrés par : Ecole doctorale Sciences Pour l'Ingénieur Lille Nord-de-France. Crédit : 5, Intervenant : Mr Jean-Dominique Polack ▪ Dans la peau d'un DRH-Session 2 (29 septembre 2020 -30 septembre 2020) Crédit : 7, Université de Lille, Intervenant : Jean-François Bart ▪ J'ai participé à l'organisation du Workshop : C-ITS France et démonstrateurs INTERCOR le 13 Mars 2020 à l'Institut des Mobilités et des Transports Durables UPHF.

  1.1 Distribution of global injury mortality by cause according to WHO Global Burden of Disease project [121]. . . . . . . . . . Figure 1.2 Serious crashes caused by AV. . . . . . . . . . . . . . . . . . . Figure 2.1 Levels of driving automation according to SAE J3016 [89] . . Figure 2.2 Typical autonomous vehicle framework . . . . . . . . . . . . . Figure 2.3 ADAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.4 Sensors employed for ITSs . . . . . . . . . . . . . . . . . . . Figure 2.5 Complex urban environment for autonomous driving [47] . . . Figure 2.6 Comparison of perception environment sensors . . . . . . . . Figure 2.7 Performance evaluation for basic CNN-based architecture from 2012 to 2019 carried on ImageNet 2012 dataset [69] . . . . . . Inter and intra class challenges . . . . . . . . . . . . . . . . . Figure 3.9 MobileNet-v2 building block . . . . . . . . . . . . . . . . . . Figure 3.10 MobileNet results (%) . . . . . . . . . . . . . . . . . . . . . . Figure 3.11 ERadar point cloud representation with optimisation using velocity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ROIs selection using UWB and narrow-band data. . . . . . . . Figure 3.13 Radar-based system results . . . . . . . . . . . . . . . . . . . Figure 3.14 Acoustic-based system results (%) . . . . . . . . . . . . . . . Figure 3.15 Proposed fusion framework architecture . . . . . . . . . . . . Figure 3.16 Estimation of object distance using area of detected bounding boxes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.17 Fusion framework results (%) . . . . . . . . . . . . . . . . . . Figure 3.18 Detection results before and after fusion . . . . . . . . . . . . Figure 4.1 Target range calculation using automotive radar. . . . . . . . . Figure 4.2 Used HST-D3 UWB radar hardware specifications. . . . . . . Figure 4.3 Illustration of the entropy variation of two different received UWB signals. . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.4 Comparative distribution of noise signals, and signals in presence of an obstacle. . . . . . . . . . . . . . . . . . . . . . . . Figure 4.5 Entropy-based detector architecture . . . . . . . . . . . . . . . Figure 4.6 CA-CFAR algorithm architecture [113] . . . . . . . . . . . . . Figure 4.7

	Figure 3.6	Challenges presented in our dataset: a) weather conditions, b)
		lighting variation, c) occlusions, and d) object types . . . . . .
	Figure 3.7	Object signatures extracted from UWB signals (near obstacles)
	Figure 3.8

Figure 2.8 Data fusion levels: (a) early fusion, (b) intermediate fusion, and (c) late fusion . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 2.9 Intersection-over-union (IOU) metric samples of single object.

Figure 3.1 Comparison a normalized percentage of objects' samples: car, person and cyclist related to Kitti, KAIST Multispectral, Apolloscape (E: easy, M: moderate and H: hard) and nuScene dataset [48]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.2 Data acquisition architecture . . . . . . . . . . . . . . . . . . Figure 3.3 Structure setup . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.4 Frames synchronisation (ST: object stream, n: number of obstacles in the scene). . . . . . . . . . . . . . . . . . . . . . . Figure 3.5 Recording emplacements at the University Polytechnic Hautsde-France . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Figure 3.12

Table 2 .

 2 1 ADAS sensors' characteristics . . . . . . . . . . . . . . . . . .

Table 2 . 2

 22

Table 4 .

 4 2 Umain radar specifications . . . . . . . . . . . . . . . . . . .

Table 4 .

 4

	A C R O N Y M S
	ACF	Aggregated Channel Features
	ADAS Advanced Driver Assistance Systems
	AI	Artificial Intelligence
	AV	Autonomous Vehicle
	AP	Average Precision
	AWGN Additive White Gaussian Noise
	BPTT Back-Propagation Through Time
	Ca	Coefficients of Approximation
	Cd	Coefficients of detail
	CMDR Constant Miss-Detection Rate
	CNN	Convolutional Neural Network
	DPM	Deformable Parts Model
	DWT	Discrete Wavelet Transform
	CA-CFAR Cell Averging CFAR
	CFAR Constant False Alarm Rate
	FCC	Federal Communication Commission
	FN	False Negative
	FP	False Positive
	IOU	Intersection Over Union

3 MODP and execution time results . . . . . . . . . . . . . . . . Table 4.4 Execution time results . . . . . . . . . . . . . . . . . . . . . . MFCC Mel-Frequency Cepstral Coefficients MOR Mapped Overlap Ratio MFCC Mel-Frequency Cepstral Coefficients MODP Multiple Object Detection Precision MS-CNN Multi-Scale CNN OLIMP A heterOgeneous muLtimodal dataset for advanced envIronMent Perception PE Permutation Entropy UWB Ultra Wide-Band P Precision R Recall YOLO You Only Look Once

Table 2 .

 2

	1: ADAS sensors' characteristics

Light detection and ranging Rotating Lidar Radar Short range radars Long range radar Ultrasonic Solid State Lidar

Table 2 .

 2

			Sensitive to large obstacles and
			smal obsacles can be ignored
		CNN	Large amount of data required
			Time-consuming
		Optical flow	Complex implementation and time-consuming
		Multi-camera	Suffer from fisheye distortion
			Camera calbiration required
	Lidar	-	Sparse data
	Radar	-	Sparse data

2: Characteristics of different object detection methods

  .2, Equation 2.3 and Equation 2.4.

	P =	TP TP + FP	=	TP All detection	(2.2)
	R =	TP TP+ FP	=	TP All ground truths	(2.3)

Table 4 .

 4 1: Comparison between CW-radars and IR-UB radars[START_REF] Hussein | Signal Processing of UWB Radar Signals for Human Detection Behind Walls[END_REF]

	UWB radar Technology	CW UWB radars	IR-UWB radars
	Time stablility	Stable	Stable
	Availability	Require highly linear	Available and a widely
		signal generator	used technology
	Cost	High cost	Inexpensive technology
	Data acquisition Time Slow data acquisition (ms)	Moderately fast data acquisition (µs)
	Interference Immunity	Relatively immune towards	Relatively immune towards
		narrow-band interfernce	narrow-band interfernce
	Dynamic range	They possess very good dynamic range	Might have a problem maintaining linear
			dynamic range
	Power	Good power budget	Low average transmitted power
	Range gating	No possiblity of range gating	Allow range gating

Table 4 .

 4 2: Umain radar specifications

	Parameter	Value and comments
	Frequency range	3∼4 Ghz
	Bandwidth	0.45∼1Ghz
	Output Power	Typ. -25dBm
		UWB Directional Antenna : Gain = Avg.7 dBi
	Antenna Specification	Antenna angle (@-3dB) = 56.0°(X-Z plane)
		77.5°(Y-Z plane)
		Size= 76mm x 58.5mm x 17mm
	Number of samples	660 samples per frame
	Sampling frequency 7,69Ghz
	4.3

  maximum of the entropy is greater than the fixed threshold, a ROI is detected. The candidate is considered as noise otherwise. The threshold definition methodology is detailed in Section 4.4.5. Sw = r(t + 1, t + Wslide); IndxLmaxEn] = Findpeaks(En t ); 8 // Compare LmaxEn with the threshold 9 ROI = ∅ ∀i ∈ IndxLmaxEn if LmaxEn i > Thr then // Define the ROIs ROI = Append(ROI, [i : i + Wslide]);

		4.4	-	75
	if a local Algorithmus 1 : Entropy-based segmentation technique
		Data : Received signal r = [r t ], Sliding window length: Wslide ; Threshold:
		Thr;
		Output : ROI
	1 for t ← 0 to length(r) -Wslide do
	2
	3	// Calculate the Shannon's entropy
	4	En t = Entropy(Sw)
	5 end for
	6 // (peaks, indexes) of local max.
		end if
		return (ROI)
	4.4.4 Experimental setup

7

[LmaxEn,

19.64 30.45 71.43 30.7 42.94 73.19 44.3 55.19 77.58 57.14 65.81 HOS CFAR [18] Our_method

									4.4	-		80
	40 60 80 100	70.74	61.58	67.46	75.34	31.72	29.03	39.27	63.06	43.8	39.46	49.64 HOS 68.65 CFAR [18] Our_method
	20
	0		Precision			Recall			F1-score
	Figure 4.8: Experimental results: Precision, Recall and F1-score using HOS, CA-CFAR, [141]
						and our method.
	100
	80	67.69
	60
	40
	20
	0		Precision		Recall			F1-score

Table 4 .

 4

		3: MODP and execution time results
		HOS CFAR [17] Our Method
	MODP	0.22	0.17 0.24	0.42
	Execution time(s) 1.34	1.41 1.52	1.26

  As we mentioned before, one of the fundamental challenges that could occur is distinguishing the real target's signature from noise within UWB signals. For this purpose, various studies have been conducted by proposing UWB-based detectors. Even though the interest of processing UWB data via deep learning techniques is growing, there is no work that treats deep with UWB signals for outdoor environment perception to the best of our knowledge.

		4.5	-	86
	4.5	-
	4.5.1 Background

Table 4 .

 4

	4.6	96

https://sites.google.com/view/ihsen-alouani/datasets

Résumé du manuscrit :

Mes travaux de thèse de rentre dans le cadre d'une convention de cotutelle entre l'Université Polytechnique Hauts-de-France et l'Université de Sousse. Le sujet de thèse a porté sur « L'exploration de la fusion de données pour la détection multi-objets pour les systèmes de transport intelligents utilisant l'apprentissage profond » L'absence de base de données publiques pour la perception de l'environnement contenant des données radar Ultra large bande (ULB) et des données acoustiques, nous a amené à concevoir notre propre base de données. Sur ce jeu de données, on a proposé un ensemble de contributions pour la détection multi-objets. Dans une première phase de la thèse, une étude bibliographique exhaustive a été réalisée sur la perception de l'environnement, la détection d'obstacles et les bases de données multimodales publiques existantes dédiées pour la perception de l'environnement. Suite à l'étude bibliographique, notre travail se décompose principalement en quatre contributions :

• La première contribution proposée est le développement de la base de données OLIMP (A HeterOgeneous MuLtimodal Dataset for Advanced EnvIronMent Perception), qui contient plus de 47 000 échantillons. C'est la première base de données hétérogène collecté à l'aide d'une caméra, un radar ULB, un radar à bande étroite et un microphone. Cette base comprend plusieurs challenges liés au trafic urbain dense.

• Par la suite, on s'est intéressée à développer des systèmes uni-modaux pour détecter les obstacles en utilisant chaque capteur seul utilisé dans la base de données introduite. Alors, la deuxième contribution proposée est un nouveau Framework de fusion qui combine les résultats obtenus des systèmes uni-modaux. L'apprentissage profond est exploité dans cette partie. Ce Framework de fusion démontre l'utilité de l'ensemble de données introduit et il souligne l'importance de la multimodalité pour la perception de l'environnement.

• La troisième contribution est une approche de segmentation qui sert à détecter les régions d'intérêts (ROIs) dans les signaux issus du radar ULB. Plus précisément, la candidate a implémenté une analyse différentielle de l'entropie des signaux ULB pour détecter les ROIs. Le travail a été évalué sur la base de données OLIMP.

Une étude comparative avec les techniques de l'état de l'art a été menée. Les résultats obtenus montrent que le système proposé présente des performances supérieures en termes de détection d'obstacles par rapport aux techniques de l'état de l'art. De plus, le détecteur proposé est robuste même avec des signaux de faible amplitude.

• Comme quatrième contribution, on a proposé un nouveau Framework qui exploite les réseaux de neurones récurrent (RNN) pour la détection de multiples obstacles routiers dans les signaux ULB. Plus précisément, le réseau récurrent à mémoire court et long terme (Long short-term memory (LSTM)) a été exploité. L'approche proposée a été évaluée sur la base de données OLIMP. Les résultats obtenus sont meilleurs que ceux de la littérature en termes de détection d'obstacles.

1-Contexte scientifique

Travaux réalisés et résultats obtenus (pendant la 2 ème année)

Durant la deuxième année de thèse, les travaux réalisés et les résultats obtenus sont les suivants :

4-Perspectives

▪ Développent d'une architecture deep learning pour la détection multi-objets en fusionnant les données radar longue-portée et les images.

5-Production scientifique

Les travaux cités dans cette liste s'insèrent dans le cadre de mes activités de recherche depuis mon inscription en thèse. Ces travaux de recherche ont fait l'objet de 2 papiers acceptés avec facteur d'impact (ISI-Thomson :