You Li

Julien Moreau

Javier Ibanez-Guzman

Emergent Visual Sensors for Autonomous Vehicles

Keywords: etc Mie Scattering Rayleigh Scattering RGB Image Data Stream Perception System Object 2D/3D detection, Semantic segmentation, etc IR

For vehicles to navigate autonomously, they need to perceive and understand their immediate surroundings. Currently, cameras are the preferred sensors, due to their high performance and relatively low-cost compared with other sensors like LiDARs and Radars. However, their performance is limited by inherent imaging constraints, a standard RGB camera may perform poorly in extreme conditions, including low illumination, high contrast, bad weather (e.g. fog, rain, snow, etc.), glare, etc. Further, when using monocular cameras, it is more challenging to determine spatial distances than when using active range sensors such as LiDARs or Radars. Over the past years, novel image sensors, namely, infrared cameras, range-gated cameras, polarization cameras, and event cameras, have demonstrated strong potential. Some of them could be game-changers for future autonomous vehicles, they are the result of progress in sensor technology and the development of the accompanying perception algorithms. This paper presents in a systematic manner their principles, comparative advantages, data processing algorithms, and related applications. The purpose is to provide practitioners with an in-depth overview of novel sensing technologies that can contribute to the safe deployment of autonomous vehicles. Diffuse Reflection Specular Reflection Solar Irradiance Fog/Rain/

I. INTRODUCTION

Since the dawn of the automotive industry, the dream of building autonomous cars never ends. The 2004 and 2007 DARPA Grand Challenges have demonstrated that vehicles could be driven unmanned in challenging conditions [START_REF] Urmson | Autonomous driving in urban environments: Boss and the Urban Challenge[END_REF]. Since then, progresses in sensors, processors, and algorithms, have pushed autonomous vehicles to be considered strategic in industry and research. In parallel, to improve safety and comfort, advanced driver assistance systems (ADAS) such as Lane Keeping Assistance (LKA), adaptive cruise control (ACC), automatic emergency braking (AEB), are being integrated into modern vehicles. Currently, the automation level of vehicles is defined and classified by the SAE (Society of Automotive Engineers) [START_REF]Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles[END_REF]: From level 0 to level 3, the autonomous driving functions need the driver to be part of the control loop that they are usually named as ADAS. Level 4 and 5 allow for fully autonomous driving (AD) in restricted areas and anywhere. Under such context, the SIVALab [5], a joint research laboratory collaborated between Renault, UTC and CNRS, has been established in 2017 for the purpose of investigating the integrity and safety of autonomous vehicles.

Perception sensors, like human eyes, are critical for all levels of autonomous vehicles. Perception sensors include visual sensors like the cameras, and range sensors such as LiDARs [START_REF] Li | Lidar for autonomous driving: The principles, challenges, and trends for automotive lidar and perception systems[END_REF], microwave and ultrasonic radars [START_REF] Patole | Automotive radars: A review of signal processing techniques[END_REF]. By using 1 You Li and Javier Ibanez-Guzman are with research department of Renault S.A.S, 1 Avenue du Golf, 78280 Guyancourt, France {you.li, javier.ibanez-guzman}@renault.com Compiègne Cedex, France julien.moreau@hds.utc.fr a focal plane array (FPA), a typical camera passively senses the intensities of ambient light at certain wavelengths within its optical field-of-view (FOV). Such information is saved as an image, with ambient light intensities sampled as millions of pixel values. A standard video camera operates within the visible spectrum that each pixel value is represented by a combination of three basic colors, i.e. red, green, and blue. LiDARs and radars are sparse active range sensors that measure distance along the directions of the transmitted lasers or microwaves. A LiDAR usually has higher accuracy and angular resolution than a radar, whereas a microwave radar can measure velocity using the Doppler effect. In general, cameras mimic human vision and provide rich and dense contextual information. By using range measurements, LiDARs and radars are more accurate than cameras at modeling the 3D world.

The sensor data are then processed by a perception system to provide useful information for vehicle navigation. A typical perception system outputs two layers of information, as shown by C. Eising et al. [START_REF] Eising | Near-field perception for lowspeed vehicle automation using surround-view fisheye cameras[END_REF]: 1) Semantic and 2) Physical. The semantic layer recognizes the objects of interest (e.g. pedestrians, vehicles, lane markings, traffic lights, etc), while the physical layer provides attributes to the classified objects: 3D positions, velocities, sizes, etc. In general, cameras are superior in the semantic layer, while LiDARs/radars are more reliable in locating objects. Accelerated by the breakthrough of deep neural networks (DNN), RGB cameras have been widely applied in all levels of autonomous vehicles as indispensable components in perception systems. For instance, usually combined with radars, monocular or binocular vision systems are used to detect pedestrians and vehicles for ACC and AEB functions, and detect lane markings for LKA functions. Nevertheless, Teslas autopilot system only utilizes several monocular cameras to realize ADAS without radars or LiDARs. To get rid of cost constraints, level 4 autonomous vehicles equipped with many cameras combined with multiple LiDARs and radars to create a 360 perception cacoon without blind zones. The applications of cameras in AD systems are similar to ADAS functions, but the applied scenarios are more complex and challenging.

Despite major successes, the limitations of RGB cameras in challenging situations have been recognized when used for safety-critical functions. Low illumination, glare, fog, rain, or other adverse conditions, can degrade their performance. For example, the glare generated by oncoming headlamps, and mirror-like reflections could blind the camera imager. Such image defects would lead to missed detections or unknown behaviors for a perception system, which might result in hazardous conditions. To enlarge ODDs (operational design domain [START_REF] Gyllenhammar | Towards an operational design domain that supports the safety argumentation of an automated driving system[END_REF]) and hence to improve safety, several emerging imaging technologies, e.g. the infrared (IR) cameras, dynamic vision sensors (event cameras), polarization cameras, gated cameras, etc, start to get spotlights. Addressing one or more weaknesses of a conventional RGB camera, those novel image sensors bring extra benefits to complement the common cameras for a better perception system. In the literature, there are plenty of review papers on various aspects of autonomous vehicles. Yurtsever et. al. [START_REF] Yurtsever | A survey of autonomous driving: Common practices and emerging technologies[END_REF] give an overall introduction of all the modules in autonomous vehicles and envisaged future trends. Brummelen et. al. [START_REF] Van Brummelen | Autonomous vehicle perception: The technology of today and tomorrow[END_REF] focus on the current and future technologies for the perception of AD systems. The sensor fusion for both perception and localization is reviewed by Velasco-Hernandez el. al. [START_REF] Velasco-Hernandez | Autonomous driving architectures, perception and data fusion: A review[END_REF] and Wang et.al. [START_REF] Wang | Multi-sensor fusion in automated driving: A survey[END_REF]. Marti el. al. [START_REF] Marti | A review of sensor technologies for perception in automated driving[END_REF] presents a review of the perception sensor technologies for automated driving systems. The included sensors are conventional such as visual spectrum (RGB) cameras, stereo cameras, millimeter-wave radars, LiDARs, and ultrasonic radars. Some reviews are around specific sensor technologies, such as the polarization camera [START_REF] Judd | Review of spectral and polarization imaging systems[END_REF], wide-angle camera [START_REF] Hughes | Wide-angle camera technology for automotive applications: a review[END_REF], and the event camera [START_REF] Gallego | Event-based vision: A survey[END_REF]. Some focus on certain ADAS or AD functions, such as the perception of valet parking [START_REF] Heimberger | Computer vision in automated parking systems: Design, implementation and challenges[END_REF]. Nevertheless, other than the reviews on the conventional sensors and well-known perception systems, in this paper, we present a comprehensive survey of the emergent visual sensors that are designed to address the deficiency of the widely applied visual spectrum cameras. The introduced sensors and sensor data processing methods could broaden the horizons of the practitioners in ADAS/AD fields.

In this paper, a review of RGB cameras is firstly presented (Sec. II) to provide a baseline for comparative purposes. An analysis of the principles, applications, and associated algorithms for each of the selected cameras are included in the following sections: Section III for the infrared cameras, Section IV for the range-gated cameras, Section V for the polarization cameras and Section VI for the event cameras.

Finally, Section VII concludes the paper and summarizes our findings as well as trends on the perceived technologies.

II. PRINCIPLE OF CONVENTIONAL RGB CAMERA

A. The Light

Light is a type of electromagnetic (EM) waves that is formed through the interaction between electric and magnetic fields. As shown in Fig. 1 (a), an EM wave is a transverse wave composed of oscillating magnetic and electric fields that are perpendicular to each other, and to the wave's propagation direction as well. Any type of EM wave has three fundamental properties: amplitude, wavelength, and polarization. The wavelength λ of visible light (λ ∈ [400nm, 700nm]) is only a small portion of the EM spectrum ranging from Gamma rays (λ < 1nm) to radio waves (λ > 1m), as shown in Fig. 1 (b). A common RGB camera detects only the intensities i.e. amplitudes of the captured visible light through its lens and is unable to measure polarization information. In a typical road scene, the light is primarily issued from complex interactions between the emitted light from luminous objects (e.g. sun, streetlamp, headlamp, etc), the reflected light from illuminated objects (e.g. vehicle, pedestrian, building, etc.) and the scattered light from transmission medium (e.g. foggy air).

During the daytime, the sun is the most common source of light. However, human-perceivable sunshine is just a part of the whole solar irradiance on the ground. As shown by Fig. 1 (c), the spectrum of solar irradiance approximately contains 5% ultraviolet wavelengths, 43% visible wavelengths, and 52% infrared wavelengths (values from [START_REF]Standard tables for reference solar spectral irradiances: Direct normal and hemispherical on 37 tilted surface[END_REF]). At night, vehicle headlamps and streetlamps are the primary sources of light [START_REF] Boullough | An investigation of headlamp glare: Intensity, spectrum and size[END_REF]. However, the lighting pattern of the car headlamps is strictly regulated for safety reasons: the maximum range of low beams can only reach around 60m [START_REF]Regulation of no 112 of the economic commission for europe of the united nations (un/ece)[END_REF] (as shown in Fig. 1 (d)), the high beams can reach over 150m but are not allowed to be used continuously.

The targets of interest (e.g. vehicles, pedestrians, etc) are visible in the images due to the light reflection from their surfaces. Two types of reflection contribute to the imaging results: (1) diffuse reflection and (2) specular reflection. Rough surfaces, such as asphalt roads and clothing, typically produce diffuse reflections that scatter incident light in various directions. Smooth surfaces, such as metallic material or wet roads, would generate specular reflections (a mirror-like reflection) in which the reflected light is concentrated in specific directions determined by the incident angle and the surface property.

In many cases, the light transmission medium (e.g. air) is assumed to be transparent. However, in adverse conditions, such as fog, rain, snow, or smoke, the floating particles would cause light scattering that results in image blur. Light scattering can be roughly classified as Mie scattering or Rayleigh scattering based on the particle size to light wavelength ratio. Rayleigh scattering occurs when the particle size is very tiny w.r.t the light wavelength: the blue color of the sky is primarily caused by the Rayleigh scattering of solar irradiance at short wavelengths (e.g. blue at the end of the visible spectrum). For particle sizes similar to or larger than a wavelength, such as the water droplet in fog, Mie scattering predominates [START_REF] Duthon | Light transmission in fog: The influence of wavelength on the extinction coefficient[END_REF].

B. From Light to Digital Images

The captured light from the various sources is focused by the lens of a camera to its focal plane, where an FPA (i.e. image sensor) is placed to generate images. An image sensor is indeed a 2D array of photosites that can convert light intensities into electrical signals, which are then converted into digits. Each photosite gives a pixel of the image. A photosite is a circuit made up of a photodetector and other electronic components. Based on the photoelectric effect of semiconductor material, Photodiodes are the most commonly used components. A photodiode [START_REF] Murari | Which photodiode to use: A comparison of cmos-compatible structures[END_REF] is a semiconductor that converts light into an electrical signal. When the incident photon energy absorbed by a photodiode exceeds the bandgap of its material, electron-hole pairs (EHPs) are generated. Then, a photocurrent I p is generated that is approximately linearly proportional to illuminance intensity. A photodiode only responds to specific wavelengths depending on the semiconductor material, which includes, but is not limited to silicon (Si), germanium (Ge), indium gallium arsenide (InGaAs).

Two important metrics represent a photodiode's sensitivity, quantum efficiency (QE) and responsivity. QE η represents the conversion efficiency of photons to electrons. For a specific wavelength λ, QE η(λ) is defined as the percentage of photons hitting the photoreactive surface that produces EHPs:

η(λ) = r e r p = Electrons Out Photons Input (1)
The responsivity R measures the electrical output per optical input. It is defined as the ratio of photocurrent output I p (in amperes) to the optical power (in watts) P :

R(λ) = I p P = η(λ) q hf ≈ η(λ) λ 1.24 [A/W ] (2)
where q is electron charge, h is Planck's constant and f is the frequency of the optical signal. Fig. 1 (e) shows an example of responsivity curves for three common semiconductor materials, Si, Ge, and InGaAs. Silicon is sensitive to light in the visible and near-infrared spectrum. InGaAs photodiodes can detect wavelengths ranging from 800nm to 2600nm. Connecting a photodiode with resistors and amplifiers creates a photosite that converts the photocurrent into a voltage for further signal processing. An image sensor is created by assembling millions of photosites, together with other components into a 2D array. By default, image sensors output grayscale values that represent light intensity. To enable color information, a color filter array (CFA) is placed just above the image sensor, so that each pixel is sensitive to a specific color wavelength. The Bayer filter array is the most common CFA, consisting of repeated 2×2 RGGB (red-green-green-blue) filter kernels because the human eye is more sensitive to green light. Only one of the three primary colors is recorded in the raw output from a Bayer-filter integrated image sensor. In image signal processor (ISP), a demosaicing algorithm is implemented to interpolate full color (e.g. RGB vector) for every pixel. Other types of CFA, such as RCCC or RCCB [START_REF] Weikl | Optimization of automotive color filter arrays for traffic light color separation[END_REF] (C stands for the wideband clear filter, i.e. no color filtering), are specifically designed for automotive applications. By only keeping red information for 1/4 pixels and not filtering light for all the other pixels, RCCC could improve the imaging sensitivity for both traffic signals and dark scenarios. However, full color is hard to be recovered by demosaicking algorithms. RCCB replaces the green pixels with clear ones to achieve low light sensitivity and hence lower noise. Unlike RCCC, the color information could be restored from RCCB as in [START_REF] Park | G-channel restoration for rwb cfa with double-exposed w channel[END_REF]. As shown in Fig. 1 (e), silicon-based imagers have sensitivities extending into the near-infrared. An infrared cut-off filter (IRCF) is designed to block near-infrared wavelengths for better color quality.

C. Limitations and Latest Advancements

Conventional RGB cameras offer a good general purpose vision with a dense representation of the scene and its textures. Their limitations can be roughly classified as [START_REF] Cleveland | Handbook of Energy[END_REF] Image degradation in adverse conditions, (2) Motion blur in case of fast dynamics, and (3) Lack of depth information. As described in Sec. II-A, a camera is a passive sensor that relies on captured light through a complex interaction between external luminous objects, illuminated targets, and transmission medium. When the received light exceeds the imaging capability, the image quality degrades, affecting the perception results for ADAS/AD. For example, at night, the external illuminations may be insufficient to produce a clear image. During sunny days, specular reflections may appear on the surfaces of the vehicles or the road [START_REF] Roser | Camera-based bidirectional reflectance measurement for road surface reflectivity classification[END_REF] that leads to over-saturation. Under adverse weather conditions (e.g. fog, rain, or snow), the strong scattering inside the transmission medium would reduce the image's visibility [START_REF] Duthon | Methodology used to evaluate computer vision algorithms in adverse weather conditions[END_REF]. Fig. 2 (b) demonstrates such challenging scenarios for RGB cameras.

In the automotive industry, CMOS Image sensor (CIS) [30] [31] has dominated the markets because of lower cost and better imaging quality, compared with CCD image sensors. To accomplish the autonomous driving functions, the cameras are required to achieve high resolution for far object detection, wide dynamic range for high contrast lighting [START_REF] Akahane | Wide dynamic range cmos image sensors for high quality digital camera, security, automotive and medical applications[END_REF], high sensitivity for low light conditions [START_REF] Gilroy | Characterisation of cmos image sensor performance in low light automotive applications[END_REF], high frame rate for high-speed applications [START_REF] Stevanovic | Low-Cost High Speed CMOS Camera for Automotive Applications[END_REF], and LED flickering mitigation [START_REF] Behmann | Real-time led flicker detection and mitigation: Architecture and fpga-implementation[END_REF] for stable traffic light recognition. Automotive CMOS cameras have made tremendous progress in the past decades. For instance, the image resolution and frame rate have reached 8MP pixels at 40fps for OmniVision's OX08B24C1 , or Sony's IMX3242 . ONSemi's AR0821CS [START_REF] Innocent | Automotive 8.3 mp cmos image sensor with 150 db dynamic range and light flicker mitigation[END_REF] has achieved a 150db dynamic range, which is very close to human eyes. For the nights, SONY's IMX390 [START_REF][END_REF] can stable output colorful images at 0.1lux, which is equivalent to moonlight.

In parallel, computer vision algorithms, especially the deep neural network (DNN) based methods, gained unbelievable progress in the past ten years that the commercial ADAS prevail in current passenger vehicles and various L4 autonomous driving start-ups appeared across the world [START_REF] Huang | Survey of state-of-art autonomous driving technologies with deep learning[END_REF]. Among various DNN structures, convolutional neural networks (CNNs) show superior performance in computer vision because the convolutional operations can efficiently capture spatial features from images. In 2015, CNN-based image classification in ImageNet surpassed human performance for the first time in history [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF]. In ADAS/AD, the CNNs are principally used for object detection, with classic algorithms such as YOLO [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF], [START_REF] Wang | YOLOv7: Trainable bag-of-freebies sets new stateof-the-art for real-time object detectors[END_REF], fasterRCNN [START_REF] Ren | Faster r-cnn: towards real-time object detection with region proposal networks[END_REF] and SSD [START_REF] Liu | SSD: Single Shot MultiBox Detector[END_REF], etc. However, in recent years, transformers [START_REF] Varcheie | Attention is all you need[END_REF] have surpassed CNNs and they almost dominated state-of-the-art performance in most computer vision fields. For instance, object detection [START_REF] Carion | End-to-end object detection with transformers[END_REF], lane marking detection [START_REF] Liu | End-to-end lane shape prediction with transformers[END_REF], and sensor fusion [START_REF] Prakash | Multi-modal fusion transformer for end-to-end autonomous driving[END_REF]. Its believed that in the future, more and more powerful algorithms will be coming to squeeze the information of each pixel.

III. INFRARED (IR) CAMERA

Conventional RGB cameras only "see" the visible spectrum, as highlighted in Fig. 1 (a). When the light wavelength exceeds 700nm, it enters the "infrared (IR)" spectrum, which is invisible for humans and is often divided as follows: (1), Nearinfrared (NIR): wavelength ranging from 0.7µm to 1.4µm. (2), Short-wavelength infrared (SWIR): wavelength ranging from 1.4µm to 3µm. (3), Long-wavelength infrared (LWIR, or Farinfrared (FIR) ): wavelength ranging from 8µm to 14µm. The Mid-wavelength infrared is too rare in automotive applications to be included in this paper. The researches and developments of IR cameras for automotive usages mainly focus on NIR, LWIR, and SWIR wavelengths [START_REF] Thakur | Infrared Sensors for Autonomous Vehicles[END_REF]. NIR and SWIR are "reflected infrared" wavelengths that rely on external light sources such as the sun or other infrared illuminators. NIR and SWIR imagers work similarly to RGB imagers in that they directly transform photons into electrical signals. LWIR is usually referred to as "thermal infrared", a typical LWIR imager converts the thermal radiation to heat, which is then converted to electrical signals. LWIR cameras can image the world solely through thermal emissions and thus do not require any external sources.

A. NIR camera

NIR imagery shares many properties with RGB imagery: as shown in Fig. 1 (e), a silicon-based imager can still exhibit NIR sensitivity until around 1100nm. As a result, with proper modifications, an RGB camera can be converted into a NIR camera. Because the CFA still has transmission spectra that bleed into NIR wavelengths, removing the IRCF or replacing it with a NIR bandpass filter converts a consumer-grade RGB camera to a NIR camera, as demonstrated in [START_REF] Fredembach | Colouring the near-infrared[END_REF] and [START_REF] Dean | Using near-infrared photography to better study snow microstructure and its variability over time and space[END_REF]. Fig. 3 (b) shows an RGB image and a NIR image of the same road scene. NIR-dedicated pixels are developed to increase the NIR sensitivity. For instance, the Nyxel technology [START_REF] Omnivision | Nyxel technology generation2[END_REF] achieves 50% QE at 940nm, and 70% QE at 850nm NIR wavelength. Although other types of materials, such as InGaAs [START_REF] De Borniol | High-performance 640 x 512 pixel hybrid ingaas image sensor for night vision[END_REF] may have higher sensitivity in the NIR spectrum, siliconbased image sensors are more popular due to their lower cost.

In recent years, simultaneously capturing RGB-NIR images has become popular. To achieve that purpose, the CFA, e.g. the Bayer filter, is modified to pass NIR light for specific NIR pixels. Chen et al. [START_REF] Chen | Rgb-nir multispectral camera[END_REF] present a four-bandpass filter array to acquire RGB-NIR images. In Lu et al.'s work [START_REF] Lu | Designing color filter arrays for the joint capture of visible and near-infrared images[END_REF], a 4×4 pattern containing 15 visible/NIR filters and 1 NIRonly filter, is made. Park et al. [START_REF] Park | Color restoration of rgbn multispectral filter array sensor images based on spectral decomposition[END_REF] and Skorka et al. [START_REF] Skorka | Color correction for rgb sensors with dual-band filters for incabin imaging applications[END_REF] further discuss the color distortion and correction problems caused by the RGB-IR filter array. On the industry side, Omnivision has commercialized RGB-NIR imaging systems [START_REF] Omnivision | Rgb-ir technology[END_REF] for automotive applications, as shown in Fig. 3 (a). A more comprehensive study on RGB-IR camera design could be found in Geelen et al. [START_REF] Geelen | System-level analysis and design for rgb-nir cmos camera[END_REF].

By simply adding external NIR illuminators, usually NIR LEDs (light-emitting diodes) or VCSELs (vertical-cavity surface-emitting lasers), a passive NIR camera can be converted to an active night vision system. A NIR LED produces a very broad diffused light distribution, whereas a laser produces a narrow beam. For acquiring 2D images, LEDs are more affordable and thus more popular. While VCSELs enable 3D perception applications [START_REF] Dummer | The role of vcsels in 3d sensing and lidar[END_REF], e.g. structured light-based 3D reconstruction. Two popular wavelengths are 850nm and 940nm. In the early days, 850nm NIR emitters were used because of their higher sensitivity than 940nm. However, human eyes can still see a deep red glow from the 850nm emitter in dark conditions. This can be uncomfortable and/or confusing. Currently, 940nm is preferred due to its complete invisibility, and fewer interferences from the natural environment, as solar IR levels at 940nm are less than half compared to 850nm (see Fig. 1 (c)) due to atmospheric absorption.

B. SWIR camera

Covering the wavelengths ranging from 1.4µm to 3µm, the SWIR images are generated by reflected SWIR light like the NIR and RGB cameras. The longer wavelengths of the SWIR spectrum would reduce the scattering effects caused by the small particles existing in the transmission medium. In theory, the SWIR wavelengths can better penetrate fog, smoke, and other adverse weather conditions. At night, the "nightglow" (a night sky radiance emitted from the relaxation of hydroxyl molecules in the atmosphere) comprising mainly SWIR wavelengths ranging from 1.4µm to 1.8µm can provide illumination for SWIR cameras [START_REF] Vollmerhausen | Night illumination in the near-and short-wave infrared spectral bands and the potential for silicon and indium-gallium-arsenide imagers to perform night targeting[END_REF] as well.

Though silicon-based image sensors have excellent responsivity from visible to NIR spectrum, the bandgap properties of silicon prevent them from having sufficient sensitivity above 1.1µm. The Indium gallium arsenide (InGaAs) has a lower bandgap, making it the preferred technology for SWIR imaging [START_REF] Rutz | Ingaas infrared detector development for swir imaging applications[END_REF], as shown in Fig. 1. In comparison to other semiconductor materials used in the SWIR spectrum e.g. Ge or HgCdTe (Mercury Cadmium Telluride), InGaAs detectors are cost-effective and high-sensitive while being operated at room temperature [START_REF] Hansen | Overview of SWIR detectors, cameras, and applications[END_REF]. However, compared to silicon-based sensors, InGaAs detectors suffer issues of the higher fabrication cost and pixel detects. Here, we mainly review the characteristics and applications of NIR and LWIR cameras applied in the automotive industry.

C. LWIR (Thermal) camera

As a phenomenon of converting thermal energy into electromagnetic energy, all matter with a temperature greater than absolute zero emits thermal radiation. This thermal radiation does not consist of a single wavelength, yet comprises a continuous spectrum. Suppose the radiating matter is ideal, i.e. the black-body, its thermal radiation B for wavelength λ is a function of temperature given by Planck's law [START_REF] Blundell | Concepts in Thermal Physics[END_REF]: where λ, h, c, K B are the wavelength, Planck's constant, light speed, and Boltzmann's constant. The standard unit of

B(λ, T ) = 2hc 2 λ 5 1 e hc/λk B T -1 (3)
B(λ, T ) is W • sr -1 • m -3 .
Most of the radiation emitted by the human body is mainly at the wavelength of 12µm, which is located in the LWIR spectrum. That's the reason for using an LWIR camera for pedestrian/animal detection at night. Photon detectors are excellent in thermal imaging because they directly convert the absorbed thermal radiation into electronic changes. However, due to the prohibitively expensive cryogenic cooling systems, their applications in ordinary scenarios are severely limited. Instead, detecting radiant heat is more popular in LWIR imaging technologies. Without the need for cooling systems, a bolometer [64] is an instrument that measures heat radiation and converts it into certain measurable quantities. Fig. 4 (a) depicts a block diagram of a bolometer. An absorber and an attached thermometer are deposited above a read-out integrated circuit (ROIC) and substrate for the reason of heat insulation. The incident LWIR radiation heats the absorber material, which is typically measured by a thermometer via resistance changes. Historically, the Salisbury screen absorber has been used for bolometers in the LWIR spectrum [START_REF] Jung | Infrared broadband metasurface absorber for reducing the thermal mass of a microbolometer[END_REF]. The vanadium oxide (VOx) or the amorphous silicon (a-Si) are the common materials for the thermometer layer because they are compatible with standard semiconductor processing technologies, as Tissot et al. [START_REF] Tissot | Uncooled microbolometer detector: recent developments at ulis[END_REF], Yon et al. [START_REF] Yon | Latest amorphous silicon microbolometer developments at leti-lir[END_REF]. The thermal measures are then transferred to the ROIC for further processing.

A 2D microbolometer array [START_REF] Niklaus | Mems-based uncooled infrared bolometer arrays: a review[END_REF] can capture thermal images with a much more affordable price and compact size that it is particularly well-suited for mobile applications such as the automobile. Driven by the rapid progress of semiconductor technologies and MEMS technologies, modern microbolometer arrays can capture images at 60Hz speed with 1024×768 pixels that each pixel is fabricated in 12µm size. For the uncooled microbolometer imaging systems, pixel non-uniformity and temperature fluctuations from the ambient conditions result in thermal drift and spatial nonuniformity of each pixel. To overcome those disturbing influences for better image quality, a series of compensation procedures such as non-uniformity correction (NUC, or referred as a flat-field correction) [START_REF] Oranowski | Nonuniformity correction algorithm with efficient pixel offset estimation for infrared focal plane arrays[END_REF], gain correction, offset correction, and radiometric calibration are carried out. In traditional thermal vision systems, an optical shutter is utilized for a run-time re-calibration during its closure time. However, such a shutter-based mechanism would interrupt the imaging processing and impede more compact sensor sizes. For automotive applications, shutter-less thermal vision systems [START_REF] Tempellhahn | Shutter-less calibration of uncooled infrared cameras[END_REF] [71] being capable of self-calibration have more advantages. From the industry side, commercially available shutterless thermal cameras are off-the-shelf, such as the BOSON camera from Teledyne [START_REF] Flir | FLIR thermal sensing for ADAS[END_REF], Adasky camera [START_REF] Velzquez | Analysis of thermal imaging performance under extreme foggy conditions: Applications to autonomous driving[END_REF], and Lynred camera [START_REF] Farooq | Evaluation of thermal imaging on embedded gpu platforms for application in vehicular assistance systems[END_REF].

D. Advantages

The principal advantages of IR cameras are the capabilities of addressing adverse weather conditions and low illumination conditions. Operating without the need of external light sources, LWIR cameras are particularly suitable for detecting hot-blooded creatures (humans, animals, etc) and other objects with heat signatures (e.g. the engine of a moving vehicle) at night. Furthermore, the LWIR imagery does not suffer from the glare effects caused by the facing headlamps [START_REF] Pinchon | All-weather vision for automotive safety: Which spectral band?[END_REF]. Fig. 4 (b) -(c) show a comparison of thermal imagery and visible imagery in several harsh conditions. Coupled with one or more invisible NIR transmitters, an active NIR camera could be a cost-effective substitute for a thermal camera. Because an NIR camera behaves similarly to the visible spectrum as "reflective infrared", the NIR imagery provides more contextual information, e.g. driver's lane markings, texts in traffic signs, etc. Such information enables NIR cameras to offer more functions (e.g. lane-keeping assistance, driver monitoring systems) at night.

E. Applications in Autonomous Vehicles

Automotive night vision system (NVS) is one of the key areas utilizing NIR or LWIR cameras. In 2000, General Motors launched the first automotive NVS on the Cadillac DeVille using an LWIR sensor supplied by Raytheon [START_REF] Martinelli | Cadillac deville thermal imaging night vision system[END_REF]. In 2004, Honda [START_REF] Tsuji | Development of night-vision system[END_REF] introduced a thermal camera-based Intelligent NVS on Honda Legend. In 2005, BMW began to use LWIR cameras in its 7 Series. Peugeot incorporated a thermal camera into its flagship sedan Peugeot 508 in 2018. In 2002, Toyota presented an active NIR NVS in Toyota LandCruiser and Lexus 470, but in 2014 Lexus decided to discard the NVS in the subsequent generations. In 2022, Kyocera [START_REF] Kyocera | Kyocera develops world's first automotive night vision system with white and near-infrared light diodes integrated into a single gan laser device[END_REF] announced a night vision system consisting of a vehicle headlight that can emit both white and NIR light on the same optical axis, and a vehicle-mounted RGB-NIR camera to detect objects.

Many comparisons and discussions have taken place between the active NIR cameras and passive LWIR night vision systems as in Kallhammer [START_REF] Kallhammer | Night vision: requirements and possible roadmap for fir and nir systems[END_REF]. In general, it has been demonstrated in Tsimhoni et al. [START_REF] Tsimhoni | Pedestrian detection with near and far infrared night vision enhancement[END_REF] and [START_REF] Tsimhoni | Pedestrian detection with night vision systems enhanced by automatic warnings[END_REF] that at night, the pedestrian detection range of a LWIR camera (165m) is significantly greater than an active NIR camera (59m). Under other adverse conditions, thermal imaging systems are found to be more stable than NIR cameras. The tests conducted in Judd et al. [START_REF] Judd | Automotive sensing: Assessing the impact of fog on lwir, mwir, swir, visible and lidar imaging performance[END_REF] show that LWIR imaging is significantly less affected by fog than NIR cameras. The experiments conducted by Pinchon et al. [START_REF] Pinchon | All-weather vision for automotive safety: Which spectral band?[END_REF] confirm the advantage of LWIR imagery over NIR imagery in pedestrian detection and demonstrate that the glare caused by oncoming headlamps under fog would not occur in thermal imagery. A recent evaluation (Velazquez el al. [START_REF] Velzquez | Analysis of thermal imaging performance under extreme foggy conditions: Applications to autonomous driving[END_REF]) of thermal image based pedestrian detection under fog shows that, when the fog's visibility is less than 20m, a thermal camera (VGA resolution, 60HFoV) can still reliably detect a pedestrian at 25m. On the other hand, the tests in Pinchon et al. [START_REF] Pinchon | All-weather vision for automotive safety: Which spectral band?[END_REF] show that thermal cameras are unable to detect lane markings or recognize traffic signs, whereas NIR imaging systems can. Keeping contexture information let NIR cameras dominate the market of driver monitoring systems (DMS). For instance, in the DMS named SuperCruised launched by Cadillac in 2018, one NIR camera is mounted in the instrument panel to monitor whether or not the driver is watching the road.

1) NIR Cameras in Driver Monitoring Systems: According to the NHTSA (National Highway Traffic Safety Administration), approximately 25% of reported crashes in the U.S.A. involve a certain form of driver inattention [START_REF] Eskandarian | Advanced driver fatigue research[END_REF]. Distraction and fatigue are the two principal causes of driver inattention. A visual distraction, such as looking away from the front road, is the most common type of distraction. Fatigue can be defined as a subjective feeling of drowsiness caused by physical or mental factors. A driver monitoring system (DMS) utilizes sensors (e.g. image sensor, pressure sensor, etc) to ensure a driver keeps attention on the road, as shown in Fig. 5 (a). A typical DMS usually contains gaze detection and drowsiness detection to warn the driver when an inattention event is detected. Researches (Ahlstrom et al. [START_REF] Ahlstrom | A gaze-based driver distraction warning system and its effect on visual behavior[END_REF], Schwarz et al. [START_REF] Schwarz | The detection of drowsiness using a driver monitoring system[END_REF]) have proved that the DMS could effectively improve safety. In Europe, a general safety regulation 3 Following the localization of the face and eye regions, additional processing is required to detect drowsiness or distraction. PERCLOS (percentage of eye closure over time) proposed by Dinges et al. [START_REF] Dinges | Evaluation of techniques for ocular measurement as an index of fatigue and the basis for alertness management[END_REF] is a valid metric for detecting drowsiness that has been used in many studies, e.g. Ji et al. [START_REF] Ji | Real-time eye, gaze, and face pose tracking for monitoring driver vigilance[END_REF], Flores et al. [START_REF] Flores | Driver drowsiness detection system under infrared illumination for an intelligent vehicle[END_REF], Garcia et al. [START_REF] Garcia | Vision-based drowsiness detector for real driving conditions[END_REF] and Dasgupta et al. [START_REF] Dasgupta | A smartphone-based drowsiness detection and warning system for automotive drivers[END_REF]. Gaze detection based distraction warning is more complex than drowsiness detection. In the literature, two types of solutions were proposed: 1) geometric approaches and 2) machine learning approaches. The geometric approaches rely on the 3D gaze estimation via 3D modeling of face/eyes. As in the AttenD algorithm proposed by Ahlstrom et al. [START_REF] Ahlstrom | A gaze-based driver distraction warning system and its effect on visual behavior[END_REF], the estimated 3D gaze direction is compared with a predefined 3D safe region to detect distraction events. Vicente et al. [START_REF] Vicente | Driver gaze tracking and eyes off the road detection system[END_REF] compute the intersection of the driver's 3D gaze line and the car windshield plane. An EOR (eyes off the road) event would be triggered when the intersection point lies out of the safe region. Machine learning based methods directly predict a gaze zone from face and eyes image detections, avoiding 3D gaze direction estimation, which can be disrupted by scenario changes. Fig. 5 (c) shows an example of 15 divided gaze zones for gaze classification. Fridman et al. [START_REF] Fridman | Owl and lizard patterns of head pose and eye pose in driver gaze classification[END_REF] and Naqv et al. [START_REF] Naqv | Deep learning-based gaze detection system for automobile drivers using a nir camera sensor[END_REF] utilize respectively a random forest algorithm on facial landmarks vector, and directly a VGG neural network, to classify the gaze zones, i.e. which zone the driver is looking at. Yoon et al. [START_REF] Yoon | Driver gaze detection based on deep residual networks using the combined single image of dual near-infrared cameras[END_REF] upgrade this method by using two NIR cameras and residual DNN to improve the accuracy and the robustness. More detailed reviews on gaze detection and DMS could be found in Dong et al. [START_REF] Dong | Driver inattention monitoring system for intelligent vehicles: A review[END_REF] and Akinyelu et al. [START_REF] Akinyelu | Convolutional neural network-based methods for eye gaze estimation: A survey[END_REF].

2) LWIR cameras in Night Vision Systems: Before the era of deep learning, object detection followed a traditional pipeline as: candidate region proposal, feature extraction and machine learning based classification, such as Haar featurebased cascade AdaBoost classifier [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF], SVM classifier [START_REF] Burges | A tutorial on support vector machines for pattern recognition[END_REF]. Fang et al. [START_REF] Fang | A shape-independent method for pedestrian detection with far-infrared images[END_REF] manually design features from hotspots in a thermal image to train a SVM classifier to recognize pedestrians. Forslund et al. [START_REF] Forslund | Night vision animal detection[END_REF] present a large animal thermal image dataset gathered over an 8-year driving period and a cascade AdaBoost classifier for animal detection.

Entering into the deep learning era when the CNNs sweep all the computer vision benchmarks, there is no exception in thermal image processing. Kristo et al. [START_REF] Kristo | Thermal object detection in difficult weather conditions using yolo[END_REF] benchmark several popular object detectors, including Faster R-CNN [START_REF] Ren | Faster r-cnn: towards real-time object detection with region proposal networks[END_REF], SSD [START_REF] Liu | SSD: Single Shot MultiBox Detector[END_REF] and YOLOv3 [START_REF] Redmon | YOLOv3: An Incremental Improvement[END_REF], that are retrained on a thermal image dataset for a surveillance system. YOLOv3 has been found to be significantly faster than other methods while still achieving comparable performance to the best. Dai et al. [START_REF] Dai | Tirnet: Object detection in thermal infrared images for autonomous driving[END_REF] propose a TIRNet for pedestrian detection (as shown in Fig. 7) by modifying the SSD detector. The performance of TIRNet is reported better than YOLOv3 based on their annotated dataset and the KAIST dataset [START_REF] Choi | Kaist multi-spectral day/night data set for autonomous and assisted driving[END_REF]. A large-scale thermal pedestrian dataset SCUT is presented in Xu et al. [START_REF] Xu | Benchmarking a large-scale fir dataset for on-road pedestrian detection[END_REF]. Based on this dataset, the authors provide a detailed comparison between widely used detectors. Tumas et al. [START_REF] Tumas | Pedestrian detection in severe weather conditions[END_REF] present a ZUT dataset containing vehicle odometry and weather measures. Launched in 2019, an EU project HELIAUS 4 aims to promote a thermal perception system for both in-cabin passengers monitoring and exterior object detection for ADAS in all light conditions. Within this project, Farooq et al. [107] [74] elaborately evaluate the performance of a YOLOv5 [108] detector based on a shutterless thermal camera. Meanwhile, an annotated thermal automotive dataset C3I-ADAS [START_REF] Farooq | C3i thermal automotive dataset[END_REF] (36K 640 × 480 thermal images with annotations) is released under the HELIAUS project for research purposes.

F. Fusion of multiple spectrum images

Infrared images and visual spectrum images are complementary to each other. Fusion of the images from multiple spectrum enable a perception system have robust performance both in daytime and at night, as well as harsh weathers. Due to the length limitations, we take the fusion between LWIR and visual spectrum as an example to introduce the principles and mainstream methods of fusion.

1) Fusion in CNN framework: fusion between thermal and RGB images is still indispensable in practice to address perception issues in all lighting conditions [110]. Under the framework of CNN, the fusion can be performed in various stages within a typical CNN architecture, and accordingly be roughly divided as input fusion, early fusion, halfway fusion, late fusion and score fusion, as illustrated in Fig. 6. Late fusion offers the flexibility to directly fuse existing detectors inferring in parallel. Choi et al. [START_REF] Choi | Multi-spectral pedestrian detection based on accumulated object proposal with fully convolutional networks[END_REF] and Park et al. [START_REF] Park | Unified multi-spectral pedestrian detection based on probabilistic fusion networks[END_REF] fuse this way two CNNs for proposal generation on color and thermal streams. With more modalities, Humblot-Renaux et al. [START_REF] Humblot-Renaux | Thermal imaging on smart vehicles for person and road detection: Can a lazy approach work?[END_REF] investigate the late fusion for multispectral people detection from YOLO detectors, as well as Takumi et al. [START_REF] Takumi | Multispectral object detection for autonomous vehicles[END_REF] from RGB, NIR, MIR and LWIR images. Other authors, e.g. Wagner et al. [START_REF] Wagner | Multispectral pedestrian detection using deep fusion convolutional neural networks[END_REF], Liu et al. [START_REF] Liu | Multispectral deep neural networks for pedestrian detection[END_REF], Li et al. [START_REF] Li | Illumination-aware faster r-cnn for robust multispectral pedestrian detection[END_REF], compare fusion schemes for pedestrian detection. The findings show that halfway fusion is superior to other approaches. As a result, the halfway fusion has become the default fusion strategy in CNN based multispectral image understanding, as demonstrated by Li et al. [START_REF]Multispectral pedestrian detection via simultaneous detection and segmentation[END_REF], Guan et al. [START_REF] Guan | Fusion of multispectral data through illuminationaware deep neural networks for pedestrian detection[END_REF] and Yadav et al. [START_REF] Yadav | Cnn based color and thermal image fusion for object detection in automated driving[END_REF].

Another trend is looking for new neural network modules as fusion operator. Zhang et al. [START_REF] Zhang | Multispectral fusion for object detection with cyclic fuse-and-refine blocks[END_REF] propose a 'Cyclic Fuse-and-Refine" module to optimize the complementary and consistency of multispectral features. In Li et al. [START_REF] Li | Illumination-aware faster r-cnn for robust multispectral pedestrian detection[END_REF] and Guan et al. [START_REF] Guan | Fusion of multispectral data through illuminationaware deep neural networks for pedestrian detection[END_REF], illumination detection modules are proposed to dynamically assign the weights of multispectral features under a halfway fusion architecture. Dasgupta et al. [START_REF] Dasgupta | Spatio-contextual deep network based multimodal pedestrian detection for autonomous driving[END_REF] extend the halfway fusion architecture with a multimodal feature embedding module (MuFEm) and a CRF-based Spatial-Contextual feature aggregation module.

2) Fusion based on attention model and Transformer:

The above fusion methods within CNN architecture are restrained by the CNNs' limited expressive capability. The convolution operator is regarded as a non-fully connected graph with a local receptive field that only integrates local information. In contrast, an attention operator acts as a fully connected graph to have a global receptive field to learn long-range dependencies for more complex feature expressiveness. In recent years, attention models, in particular, the visual transformers are regarded to be more suitable in multi-modality fusion than tradi-tional CNN methods. A well-designed "cross-modality fusion" (CMF) transformer proposed in [START_REF] Fang | Cross-modality fusion transformer for multispectral object detection[END_REF] is used to s used to fuse the intra-modality and inter-modality features simultaneously. The experiments show that, after the enhancement of CMF, a modified YOLOv5 detector demonstrates better performance than the original version. Shared with similar ideas, Y. Zhang et al. [START_REF] Zhang | Attention based multi-layer fusion of multispectral images for pedestrian detection[END_REF] propose a channel-wise attention module (CAM) and a spatial-wise attention module (SAM) for multi-spectral feature fusion applied in pedestrian detection. Q. Fang et al. [START_REF] Jiang | Attention-based cross-modality feature complementation for multispectral pedestrian detection[END_REF] propose a cross-modality feature complementary module working on channel-wise feature fusion and an attention-based feature enhancement fusion module working on spatial feature aggregation before the detection head. K. Dasgupta et al. [START_REF] Dasgupta | Spatio-contextual deep network-based multimodal pedestrian detection for autonomous driving[END_REF] use a graph attention module to extract multi-spectrum features and a feature fusion unit to address the modality imbalance problem. In general, the proposed attention models are mainly used for enhanced feature extraction and fusion, the fused features are then sent to certain typical detection heads for final object detection. Some popular open-sourced multi-spectrum datasets are listed in Tab. II.

G. Remaining Challenges

Both the NIR and LWIR cameras have been successfully applied in mass-produced cars for night vision systems or driver monitoring systems. For LWIR cameras, the main challenge is to increase the resolution while still maintaining affordable prices. The ordinary resolution for an LWIR camera is still VGA (640 × 480 = 0.3M pixels), which is much smaller than a standard RGB camera used for ADAS/AD (2M pixels to 8M pixels). Another issue is the integration position. Because regular glass is opaque to thermal radiation, a thermal camera has to be mounted outside the windshield, typically placed in the front grille. Such a position makes the thermal camera vulnerable to being damaged by road debris or covered by snow/ice/dust. As for NIR cameras, they are more mature in production due to the shared fabrication technology with visible spectrum imaging sensors. However, to be performance comparable with LWIR camera at night, more advanced algorithms or hardware improvements are expected to increase NIR's imaging stability in challenging lighting conditions, such as the oncoming vehicle's headlamp at night. According to the experiments in Boullough [START_REF] Boullough | An investigation of headlamp glare: Intensity, spectrum and size[END_REF], the spectrum of an ordinary halogen headlamp contains a large portion of NIR wavelengths ranging from 700nm to 1000nm. Those emitted NIR lights would cause image clutters and thus reduce the performance of a perception system.

IV. RANGE-GATED CAMERA

To enhance the imaging quality under harsh conditions, range-gated imaging was first proposed in the 1960s [START_REF] Gillespie | Apparent illuminance as a function of range in gated, laser night-viewing systems[END_REF] and has been applied in night vision systems [START_REF] Baker | A low-noise laser-gated imaging system for longrange target identification[END_REF], submarine vision [START_REF] Pinto | Maresye: A hybrid imaging system for underwater robotic applications[END_REF]. In recent years, range-gated cameras have gained popularity for their resistance to adverse conditions [START_REF] Bijelic | Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather[END_REF].

A. Principles

A range-gated camera is an active imaging system in which an illuminator transmits pulsed light, and an image sensor is precisely synchronized to image the reflected lights within certain defined "gates". A general principle of a range-gated imaging system is shown in Fig. 8. In the illuminator module, light pulses are emitted to illuminate the environment within the lens's field-of-view. Parts of the transmitted lights will be reflected by the surfaces of the objects and then be captured partially by the receiving optics. Because the objects are at different ranges, the reflected photons are captured at different times. Unlike conventional cameras' exposure methods (global shutter or rolling shutter), a gated camera employs several gate functions to expose the photons arriving at different times. Therefore, only the light arriving within the right timing window contributes to the final image. Usually, the exposure gates are very short: in the order of 0.01 -2µs. As the example in Fig. 8, three programmed gated functions generate three image slices containing objects at different ranges. The final image is obtained by merging those image slices.

The main components are introduced as follows: Illuminator is triggered by the gating signals from a controller. Owing to narrow spectral width and high peak power, the laser is preferred over other kinds of lights. Different laser wavelengths ranging from visible, NIR to SWIR wavelengths could be applied. The NIR laser is popular because of its maturity and cost. For instance, 808nm laser is used in David et al. [START_REF] David | Range gated active night vision system for automobiles[END_REF] and Spooren et al. [START_REF] Spooren | RGB-NIR active gated imaging[END_REF]. When considering better penetration in long-distance through fog or smoke, the SWIR laser is preferred because it can achieve much higher transmission power while still meeting eye safety standards. In [START_REF] Baker | A low-noise laser-gated imaging system for longrange target identification[END_REF], a range-gated imaging system based on an Nd YAG laser at 1571nm reaches a 10km detection range. Similarly, in Baker et al. [START_REF] Willitsford | Range-gated active short-wave infrared imaging for rain penetration[END_REF], a range-gated SWIR (1527nm) camera successfully penetrates heavy rains and detects obstacles 10km away. Gated image sensor: The gated image sensors can perform multiintegration to generate a merged image by using gated signals. Due to the extremely short integration time, the gated image sensor has to be highly efficient. In Spooren et al. [START_REF] Spooren | RGB-NIR active gated imaging[END_REF], a gated RGB-NIR image sensor with high NIR quantum efficiency (≈40%) is built. In Rutz et al. [START_REF] Rutz | Ingaas apd matrix sensors for swir gated viewing[END_REF], a high-gain avalanche photodetector (APD) array containing 640 × 512 InGaAs pixels is coupled with a SWIR laser transmitter. When operated in Geiger mode, the APDs become single-photon avalanche diodes (SPADs), meaning that even a single photon could trigger the avalanche effect. Burri et al. [START_REF] Burri | Architecture and applications of a high resolution gated spad image sensor[END_REF] present a 512 × 128 pixel CMOS SPAD sensor capable of operating within an exposure window as small as 4ns. In Morimoto et al. [START_REF] Morimoto | Megapixel time-gated spad image sensor for 2d and 3d imaging applications[END_REF], a 1M pixel CMOS SPAD image sensor is built for 3.8ns gating time.

B. Advantages

The range-gated cameras, as active sensors, are better suited to low-light conditions, such as a country road at night. Furthermore, owing to the slicing mechanism, only photons received at appropriate times are utilized for imaging. Such an attribute has two advantages: [START_REF] Cleveland | Handbook of Energy[END_REF] No blooming effect when the photons from highly reflective objects do not fall within the sampling range. For example, the oncoming vehicles' headlamps have almost no impact on range-gated images. (2) Resistance to backscattering environments, such as fog/rain/smoke. According to [START_REF] David | Range gated active night vision system for automobiles[END_REF], a key parameter deciding the quality of a range-gated image is the modulation contrast:

Contrast ≃ I target -I background I target + I background + 2I bsc ( 4
)
Where I target , I background , I bsc are the luminance of the target, background, and due to the backscattering effect. Hence, the image quality in backscattering condition is defined by the strength of I bsc , which can be calculated as:

I bsc = ∫ 2γR of f 2γRon P Ge -X γ 2 2F 2 n θ 2 X 2 dX (5)
where R on , R of f define a range interval during one exposure. G is the backscatter gain, γ is the atmospheric attenuation coefficient, θ is the laser beam divergence, P is the laser power, F n is the speed of the lens, and X is the integration variable. Compared with a conventional camera counting all the backscattering photons, a range-gated camera only performs the photon integration during a very short opening time, i.e. between R on and R of f , so that a higher contrast defined in Eq. 4 can be achieved.

C. Applications in Autonomous Vehicles

Owing to its excellent performance in harsh conditions, the range-gated camera has the potential to be a strong competitor to infrared cameras, and has gained recognition in recent years. A comparison between an RGB camera and a range-gated camera in a fog environment is shown in Fig. 9. Walz et al. [START_REF] Walz | Uncertainty depth estimation with gated images for 3d reconstruction[END_REF] benchmark multi-model sensors in a well-controlled artificial fog chamber. Both the quantitative and qualitative results show the superiority of range-gated cameras. On the industry side, [START_REF] Christnacher | Bistatic range-gated active imaging in vehicles with leds or headlights illumination[END_REF] first applies a NIR range-gated camera to aid driving at night. Grauer et al. [START_REF] Grauer | Active gated imaging in driver assistance system[END_REF] and [START_REF] Grauer | Active gated imaging for automotive safety applications[END_REF] present a high resolution (1.2M pixel) range-gated camera based on NIR VCSEL laser (808nm) and a gated CMOS image sensor. This sensor is suitable for use in active safety systems such as vulnerable object detection, forward collision warning, lane departure warning, traffic sign detection, etc.

From 2017, a series of works around range-gated camera images were developed within the EU-founded DENSE project5 . Supported by this project, the DENSE dataset6 containing multi-model sensors (a range-gated camera, an RGB stereo camera, an LWIR camera, and a LiDAR) is released to the public. The dataset covers snow, rain, and urban and suburban scenarios. The DENSE dataset is further annotated in Julca-Aguilar et al. [START_REF] Julca-Aguilar | Gated3d: Monocular 3d object detection from temporal illumination cues[END_REF] as Gated3D dataset, in which more than 100K objects in 4 classes are manually annotated over 12997 image frames. Based on these datasets, Tobias et al. [START_REF] Tobias | Gated2depth: Real-time dense lidar from gated images[END_REF] present a deep neural network (DNN) named "gated2depth", which can estimate the depth of each pixel in the range-gated camera. The proposed DNN architecture utilizes all three slice images. Walz et al. [START_REF] Walz | Uncertainty depth estimation with gated images for 3d reconstruction[END_REF] extend gated2depth by incorporating aleatoric uncertainties into the pixel-wise depth estimation. Bijelic et al. [START_REF] Bijelic | Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather[END_REF] propose a fusion neural network for adaptively fusing LiDAR, RGB camera, gated camera, and radar features in an entropy estimation framework (higher entropy indicates more confidence). A delicate feature exchange network is designed to dynamically allocate the best features for each sensor. To explore the implied range information in the slice images, Julca-Aguilar et al. [START_REF] Julca-Aguilar | Gated3d: Monocular 3d object detection from temporal illumination cues[END_REF] propose a DNN for 3D object detection. The proposed DNN is tailored to the temporal illumination cues from the three image slices. Based on the Gated3D dataset, they demonstrated that using temporal cures from a range-gated camera, the 3D object detection results outperform a pure RGB-based detection method.

D. Remaining Challenges

As an active sensor, a range-gated camera needs an illuminator precisely synchronized with the image sensor which Fig. 9. Imaging results of a conventional RGB camera (left) and a range-gated camera (right) in an artificial fog (from [START_REF] Bijelic | Benchmarking image sensors under adverse weather conditions for autonomous driving[END_REF]). Due to the backscattering effect caused by the fog, many objects are obscured in the RGB camera image, while the range-gated camera image is almost immune to the fog. Meanwhile, the headlamp of the target car causes a strong blooming effect in the RGB camera image but has no impact on the range-gated camera image. is more complicated than passive cameras in practice. The illuminator may potentially cause interferences with other sensors such as visual spectrum cameras or LiDARs. In addition, because they are novel, range-gated cameras are rare in usage compared to other cameras. Therefore, there is still a lack of large open-sourced datasets to boost relevant studies to explore the capabilities of the range-gated camera.

V. POLARIZATION CAMERA

A. Principle

According to Sec. II-A, light passes through a medium as a transverse wave, i.e. oscillating perpendicularly to the direction of propagation, that consists of an oscillating electric field and a magnetic field. For computer vision applications, only the electric field is considered. Polarization is a fundamental and distinct property that describes the orientation of the light oscillation [146]. There are in general three kinds of polarized light: totally polarized (linear, circular or elliptic), partially polarized and unpolarized. The majority of the light sources, e.g. the sun, streetlamps, emit unpolarized light, i.e. it vibrates randomly in all directions.

Although most natural light is unpolarized, it can be converted to polarized light through the reflection from certain surfaces. In an ideal situation when the incident angle of unpolarized light is the angle of Brewster, according to Fresnel equations [START_REF] Kliger | Polarized Light in Optics and Spectroscopy[END_REF], the reflected light is linear polarized (as shown in Fig. 10 (a)). Otherwise, it would be partially polarized. Reflections from most flat surfaces are partially polarized as a function of incident angle. A more controllable way to obtain polarized light is to use a polarizer, which is an optical filter that passes only specific polarized light while blocking light from other polarizations, as shown in Fig. 10 (b).

A concise representation of polarized light is the Stokes vector S [START_REF] Bass | Handbook of Optics[END_REF], consisting of 4 parameters: S = [S 0 , S 1 , S 2 , S 3 ]. S 0 (> 0) is the total light intensity, S 1 and S 2 roughly represent the degree of linearly polarized light (S 1 stands for horizontal or vertical linear polarization, S 2 stands for 45 or 135 linear polarization). S 3 stands for ellipticity, which is usually ignored in applications. Other important physical properties, e.g. angle of polarization (AoP) and the degree of polarization (DoP) can be inferred from the Stokes vector as:

AoP = 1 2 × arctan( S 2 S 1 ), DoP = √ S 2 1 + S 2 2 S 0 (7)
Varying between 0 and 180, AoP represents the predominant axis of the light vibration. DoP is the ratio of the intensity of the polarized portion to the total intensity. For instance, linearly polarized light has a DoP of 1, and natural light usually has a DoP between 0 to 0.5. Creating a practical and convenient polarimetric imaging system is not easy work. In early research, Morel et al. [START_REF] Morel | Polarization imaging applied to 3d reconstruction of specular metallic surfaces[END_REF] make a polarization camera by manually rotating a polarizer in front of a normal camera. Three images are taken at different rotating angles of the polarizer to determine the Stokes vector for each pixel. In Wolff et al. [START_REF] Wolff | Polarization camera for computer vision with a beam splitter[END_REF], a polarizing beam splitter is placed in front of 2 cameras so that the reflected and the transmitted beams are utilized to compute the polarization of each pixel. However, those methods either require a special environment for imaging or are too expensive.

Powered by on-chip polarizer technology, modern image sensors can simultaneously acquire polarization and color information through a single shot. For instance, inside SONY's Pregius IMX250 CMOS sensor (as shown in Fig. 10 (c)), a Polarization Filter Array (PFA) composed of four various angled micro-polarizers (0, [START_REF] Carion | End-to-end object detection with transformers[END_REF][START_REF] Dinges | Evaluation of techniques for ocular measurement as an index of fatigue and the basis for alertness management[END_REF][START_REF] Burri | Architecture and applications of a high resolution gated spad image sensor[END_REF] is placed on top of the CFA and photodiodes. The Stokes vector (as in Eq. 6) and RGB vector for each pixel can be interpolated by using a special demosaicing process afterward. Such snapshot technology has a price advantage that has been employed in many computer vision studies.

B. Advantages

As discussed in Sec. II-C, the specular reflection, high contrast regions, and adverse weather would degrade the image quality. Although many intensity-based solutions (e.g. Li et al. [START_REF] Li | Multiframe-based high dynamic range monocular vision system for advanced driver assistance systems[END_REF], and Wang et al. [START_REF] Wang | Efficient road specular reflection removal based on gradient properties[END_REF]) could alleviate these issues, polarization cameras offer a new perspective. One of the biggest advantages is that the polarization camera can detect transparent objects like windows or glasses [START_REF] Yamaguchi | Glass detection using polarization camera and lrf for slam in environment with glass[END_REF], which are hard to be detected by ordinary cameras or LiDARs. The other man-made objects, like the vehicles, are more distinct in polarization images [START_REF] Wang | Polarizationbased car detection[END_REF], [START_REF] Sheeny | POL-LWIR vehicle detection: Convolutional neural networks meet polarised infrared sensors[END_REF].

C. Applications in Autonomous Vehicles

Polarization cameras are not yet commercialized for automotive usage. Nevertheless, as the snapshot P-RGB image sensors (e.g. SONY IMX250, IMX253) become more popular, more researchers are beginning to investigate the potential benefits of light polarization. Current research focuses on image enhancement, object detection and semantic segmentation. Wang et al. [START_REF] Wang | Specularity removal: A global energy minimization approach based on polarization imaging[END_REF] utilize a polarization camera to remove specular reflection because the DoP of the specular reflection part is much larger than the part of diffuse reflection when an unpolarized light beam is reflected. Polarization cameras can also achieve high dynamic range (HDR) imaging to solve the over/under-saturation in high contrast conditions. As proposed by Wu et al. [START_REF] Wu | Hdr reconstruction based on the polarization camera[END_REF], the 4 micro-polarizer patterns have similar effects as 4 different exposure times. Therefore, by using multiple polarization images at known pixel-specific exposure times, the irradiance maps can be estimated and hence construct an HDR image.

Polarimetric images provide physical properties of the object, such as surface material and roughness, which can be utilized as a complement to traditional RGB image based object detection and segmentation. Wang et al. [START_REF] Wang | Polarizationbased car detection[END_REF] implement a feature selection process in polarimetric images and discover that the AoP is the most informative polarization feature. Then, for car detection, the AoP features are incorporated with deformable-part based models (DPM). The experimental results demonstrate that polarization features significantly reduce the false detection rate. Adding the polarization features to an object detection DNN, Blin et al. [START_REF] Blin | Road scenes analysis in adverse weather conditions by polarization-encoded images and adapted deep learning[END_REF] and [START_REF]A new multimodal rgb and polarimetric image dataset for road scenes analysis[END_REF] show that car detection results under adverse weather conditions could be improved by 20% -50%. In addition, a new dataset PolarLITIS [162] containing RGB and polarimetric images under fog conditions was released to evaluate the performance gain of object detection from polarization information. The experiments in Blanchon et al. [START_REF] Blanchon | Outdoor scenes pixel-wise semantic segmentation using polarimetry and fully convolutional network[END_REF] and Xiang et al. [START_REF] Xiang | Polarization-driven semantic segmentation via efficient attention-bridged fusion[END_REF] both find that the semantic segmentation for car and windows is largely improved thanks to the polarization features.

D. Remaining Challenges

Polarization measures as a new data dimension would benefit the perception system. While, since a polarization camera needs a micro-polarizer array, it sacrifices its imaging performance in dark conditions, which is a critical issue for autonomous vehicles. Another problem is similar to the range-gated camera, polarization cameras are pretty young for researchers in robotics/autonomous vehicles fields, and more open-sourced datasets and related studies are needed to show the unique advantages brought by polarization.

VI. EVENT CAMERA

In dynamic and unpredictable environments, traditional cameras would give blurry images or under/over-exposed images. The neuromorphic vision sensor is a good choice for a robust perception system. A general survey on event cameras is given in [START_REF] Gallego | Event-based vision: A survey[END_REF], and a tutorial aiming at some common processing methods applied for autonomous driving is given in [START_REF] Chen | Event-based neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception[END_REF]. This section is complementary to these papers in providing a review of event vision for driving applications.

A. Principles

The event camera is also called address-event representation silicon retina, neuromorphic, or retinomorphic camera, because it is inspired by eye retina, as described by [START_REF] Posch | Retinomorphic event-based vision sensors: Bioinspired cameras with spiking output[END_REF]. In the retina, the fundus of the eye, are located the cones and rods, which are sensitive to light, followed by layers of neurons. Photosensitive cells convert light into electric signal transmitted to nerve cells. Some signal exchanges occur from each photosensitive cell up to two bipolar ganglion cells: when activated, the first one represents ON pulse whereas the second one represents OFF pulse. In summary, ON cell activates when a spatiotemporal brighter change in contrast occurs, OFF cell activates when a spatiotemporal darker contrast change occurs. The brain is able to interpret these voltage spikes to give to us our sight sense. This process leads to the following advantages: Independence from absolute light level: It can be seen as an automatic gain control from the retina and allows vision capabilities for a very wide range of brightness. Lightweight data encoding for fast transmission: Spikes are emitted continuously to the brain, avoiding the need to encode absolute intensities, and giving a high temporal resolution.

An event camera is designed to imitate the retina by bionomic pixel circuits (as shown in Fig. 11 (a)), and hence inherit these advantages. Pixel outputs of an event camera are independent, they represent signed spikes as long as the photosensor observes a log-intensity difference above a threshold. The rate of following spikes of the same sign is an indication of the brightness change speed. Then, a stream of events is a sequence of timestamped signals, where each signal represents a positive or negative pulse (that is respectively, a state change to be more or less bright) for one or several points of the matrix sensor. An event camera does not stream full image frames in the way a conventional camera does at a given framerate. It acts in an asynchronous way with a very high temporal resolution and low latency, in an order of microseconds. The output difference between both sensors is shown in Fig. 11 (b). Similarly to conventional image sensors, event sensors are made of Silicon and are sensitive to visible and NIR light. On the contrary, event cameras are often made without IR cut filters in order to gather more light. However, the use of specific wavelength filters may be necessary for certain applications. Modern neuromorphic cameras reach HD resolution, such as Prophesee Gen4 CD (1280 × 720 pixels) [START_REF] Finateu | 5.10 a 1280720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86m pixels, 1.066geps readout, programmable event-rate controller and[END_REF], Samsung DVS-Gen4 (1280 × 960 pixels) [START_REF] Suh | A 1280960 dynamic vision sensor with a 4.95-m pixel pitch and motion artifact minimization[END_REF], CelePixel CeleX-V (1280 × 800 pixels) [START_REF] Chen | Live demonstration: Celex-v: A 1m pixel multimode event-based sensor[END_REF]. Some event cameras (e.g. iniVation DAVIS346 7 , CelePixel CeleX-V) incorporate additional circuits in order to simultaneously output conventional images (monochrome in most cases) and sensed events. Such design gives the advantage of data fusion at exact superimposition, while at the expense of increasing noises caused by residual currents brought by those additional circuits. Rare event cameras are able to output both RGB events and frames, as iniVation DAVIS346B-Color8 , which includes a Bayer filter array to estimate RGB channels. Sample data of RGB event camera is available through the Color Event Camera Dataset (CED) [START_REF] Scheerlinck | Ced: Color event camera dataset[END_REF].

B. Advantages

Event cameras are bio-inspired passive sensors that try to imitate millions of years of evolution of sight sense. General advantages of event camera are stated by Gallego et al. [START_REF] Gallego | Event-based vision: A survey[END_REF]: Microsecond temporal resolution for detection and timestamp. A direct consequence is the ability to always avoid motion blur as it exists for conventional cameras. Furthermore, the event camera outputs at sub-millisecond latency, which is approximately equivalent to a virtual > 1000F P S framebased camera. Low power consumption, in the order of 10mW to 100mW for typical event cameras, while usually between 1W and 3.5W for industrial RGB cameras. Broad dynamic range: an event camera's dynamic range can easily reach > 120dB without a special design. In contrast, a normal RGB camera needs a dedicated pixel design to boost its dynamic range from typical 60 -70dB to 110dB.

All these advantages are desirable for intelligent vehicles: very high temporal resolution allows to detect fast-moving entities; very low latency is important for safety-critical applications; very high dynamic range allows to perceive in challenging lighting conditions. Event camera capabilities in driving scenes are illustrated in Fig. 12 (a),(b).

C. Data representation and processing

Unlike frame cameras, neuromorphic cameras output stream of events E = {e∀(x, y, t, p)} that each event e(x, y, t, p) encodes pixel position (x, y) ∈ N 2 , timestamp t, polarity of the brightness change p ∈ {-1, +1}. In signal processing point of view, an event e(x, y, t, p) can be considered as a continuous function using diracs e = p • δ(ξ -x, υ -y) • δ(τ -t) where (ξ, υ) ∈ R + 2 represents 2D spatial positioning in pixel array and τ ∈ R + represents continuous running time. There are two ways of processing event flows: asynchronous processing when an event arises, that is event-by-event processing or accumulation of events within a temporal window, i.e. process them as an array or as a tensor.

Event-by-event processing: it is a natural way to keep the raw asynchronous and sparse event(spike) flow, whereas current computers are not designed for spikes processing. Standard processor architectures (CPUs and GPUs) are good to process dense arrays of data but are not able to process irregular flows of independent events at a very high rate. Specific biologically inspired hardware is designed to efficiently process event-by-event, such as ROLLS processor [START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses[END_REF], IBM TrueNorth chip [START_REF] Akopyan | TrueNorth: Design and tool flow of a 65 mW 1 million neuron programmable neurosynaptic chip[END_REF], and Intel Loihi chip [START_REF] Davies | Advancing neuromorphic computing with loihi: A survey of results and outlook[END_REF]. These spike processors are particularly interesting since they open the door for hardware SNN (spiking neural networks) with low power consumption. SNNs are designed to imitate brain neurons and are the most popular and direct way to process event-by-event flows. With or without specific hardware, some early investigations about SNNs have been done. However, as a single event gives insufficient information for understanding, new events are used iteratively to update a system's state. While some methods apply standard optimization techniques or filters (such as [START_REF] Gehrig | EKLT: Asynchronous photometric feature tracking using events and frames[END_REF]- [START_REF] Nunes | Robust event-based vision model estimation by dispersion minimisation[END_REF]), most of them integrate asynchronous events in artificial neural networks. SNNs have already been proposed for many applications. For example, [START_REF] Osswald | A spiking neural network model of 3d perception for event-based neuromorphic stereo vision systems[END_REF], [START_REF] Barbier | Spike timing-based unsupervised learning of orientation, disparity, and motion representations in a spiking neural network[END_REF] for stereo depth estimation, [START_REF] Viale | CarSNN: An efficient spiking neural network for event-based autonomous cars on the loihi neuromorphic research processor[END_REF] for classification, [START_REF] Paredes-Valls | Unsupervised learning of a hierarchical spiking neural network for optical flow estimation: From events to global motion perception[END_REF] for optical flow, [START_REF] Parameshwara | SpikeMS: Deep spiking neural network for motion segmentation[END_REF] for background motion separation, [START_REF] Kreiser | A neuromorphic approach to path integration: A headdirection spiking neural network with vision-driven reset[END_REF], [START_REF] Kreiser | Error estimation and correction in a spiking neural network for map formation in neuromorphic hardware[END_REF] for heading estimation and loop closure detection, [START_REF] Stagsted | Towards neuromorphic control: A spiking neural network based PID controller for UAV[END_REF], [START_REF] Vitale | Event-driven vision and control for UAVs on a neuromorphic chip[END_REF] for robotic control, [START_REF] Rckauer | Closing the accuracy gap in an event-based visual recognition task[END_REF] for target following, [START_REF] Salvatore | A neuroinspired approach to intelligent collision avoidance and navigation[END_REF] for collision avoidance (for drone).

Events binning: It is generally more practical to process batches of incoming events rather than processing individual events. Usually, successive events are gathered and compressed into a dense array or a tensor that is similar to an image frame. Both arrays and tensors can be efficiently processed by standard computer hardware. Binning events cause additional latency. However, this drawback is moderate and acceptable as the general advantages are still kept, allowing for example Brebion et al. [START_REF] Brebion | Real-time optical flow for vehicular perception with low-and high-resolution event cameras[END_REF] to optimize a pipeline for real-time optical flow from HD events stream. There are two strategies of events binning: via a time window, or via a queue of a fixed number of events. The usage of time windows is easy and common but can lead to accumulated arrays without or too many events.

Hence, the sampling time should be tuned accordingly and gives a synchronous process. Zou et al. [START_REF] Zou | Robust dense depth maps generations from sparse DVS stereos[END_REF] use adaptive accumulation time (making the method asynchronous), Rebecq et al. [START_REF] Rebecq | Real-time visualinertial odometry for event cameras using keyframe-based nonlinear optimization[END_REF] use overlapping pairs of time windows, and Joubert et al. [START_REF] Joubert | Characterization setup for event-based imagers applied to modulated light signal detection[END_REF] combine time windows of different lengths. The study [START_REF] Akolkar | What can neuromorphic event-driven precise timing add to spike-based pattern recognition?[END_REF] contains a performance analysis of various time window durations for a classification task. The strategy of binning by a fixed number of events is used in [START_REF] Moeys | Steering a predator robot using a mixed frame/event-driven convolutional neural network[END_REF]- [START_REF] Paredes-Valls | Self-supervised learning of event-based optical flow with spiking neural networks[END_REF]. It allows for keeping accumulated representations with similar appearances (same density of events) and for asynchronous processes. Nevertheless, the following operations should be fast enough when huge flows of events arrive in a short time. At last, event accumulation can be motion compensated with a fast algorithm, typically using a joint IMU inside the event camera [START_REF] Rebecq | Real-time visualinertial odometry for event cameras using keyframe-based nonlinear optimization[END_REF]. This guarantees event binning with no blur effect in case of a long accumulation time.

After accumulating enough events, the next issue is encoding, i.e. extracting effective event attributes. Several handcrafted encoding methods have been proposed in the literature (for interested readers, a categorisation is suggested in [START_REF] Liu | Motion robust high-speed light-weighted object detection with event camera[END_REF]). For example, leveraging "frequency encoding" representation from Chen et al. [START_REF] Chen | Pseudo-labels for supervised learning on dynamic vision sensor data, applied to object detection under ego-motion[END_REF], where a standard YOLOv3 CNN architecture [START_REF] Redmon | YOLOv3: An Incremental Improvement[END_REF] is utilized for pedestrian detection. Chen et al. [START_REF] Chen | NeuroIV: Neuromorphic vision meets intelligent vehicle towards safe driving with a new database and baseline evaluations[END_REF] also get the best results with the "frequency encoding" among other encoding schemes for driver monitoring applications. Perot et al. [START_REF] Perot | Learning to detect objects with a 1 megapixel event camera[END_REF] test different accumulation and encoding strategies for object detection, with the best results using the "discretized event volume" representation from Zhu et al. [START_REF] Zhu | Unsupervised event-based learning of optical flow, depth, and egomotion[END_REF]. As a step above, [START_REF] Liu | Motion robust high-speed light-weighted object detection with event camera[END_REF] propose a "temporal active focus" representation, allowing them to surpass the results from other representations. Besides human-designed features, generalized expression can also be learned automatically in an end-to-end manner. Tulyakov et al. [START_REF] Tulyakov | Learning an event sequence embedding for dense event-based deep stereo[END_REF] model events as a stream of sparse 3D data points, and then apply a MLP (Multi-Layer Perceptrons) to learn an optimal encoding for a stereo-matching problem. Experimental results show that the learning-based encoding is better than the best hand-crafted approach. Cannici et al. [START_REF] Cannici | A differentiable recurrent surface for asynchronous event-based data[END_REF] propose specific LSTM (Long Short-Term Memory) recurrent modules as a flexible way to learn task-dependent event-surfaces, and show better performance in optical flow estimation. Li et al. [START_REF] Li | Event-based vision enhanced: A joint detection framework in autonomous driving[END_REF] apply a SNN to encode events and generate attention maps for further fusion with frame images, in an object detection framework. Or, as events are sparse, GNN (graph neural networks) can efficiently process events as spatio-temporal graphs with no information loss [START_REF] Schaefer | Aegnn: Asynchronous event-based graph neural networks[END_REF].

D. Applications in Autonomous Vehicles

The event camera is especially useful for systems running with real-time interactions, non-controlled enlightenment conditions, and low latency. In this paper, we focus on their application in the autonomous driving field.

1) Dataset in driving scenes: To apply event cameras in autonomous vehicles, large and well-annotated datasets are indispensable. The neuromorphic vision community is very active in it. Because event cameras are still in the early stages, many published datasets (e.g. MVSEC [START_REF] Zhu | The multivehicle stereo event camera dataset: An event camera dataset for 3d perception[END_REF], DDD17 [START_REF] Binas | DDD17: End-toend DAVIS driving dataset[END_REF], etc.) in recent years are still in low image resolution (e.g. less than 640×480) due to hardware limits. The first HD event camera released in public is the CelePixel CeleX-V in 2019 [START_REF] Chen | Live demonstration: Celex-v: A 1m pixel multimode event-based sensor[END_REF]. Larger resolution benefits further object detection range and better recognition for small objects, while posing challenges for computation capability because of huge event flows. We expect to see more and more HD event camera datasets as Perot et al. [START_REF] Perot | Learning to detect objects with a 1 megapixel event camera[END_REF] appear in public. The published datasets are for various purposes, such as target detection [START_REF] Binas | DDD17: End-toend DAVIS driving dataset[END_REF], lane detection [START_REF] Cheng | DET: A high-resolution DVS dataset for lane extraction[END_REF], drowsiness detection [START_REF] Chen | NeuroIV: Neuromorphic vision meets intelligent vehicle towards safe driving with a new database and baseline evaluations[END_REF] etc. A comprehensive summary of the current open datasets is demonstrated in Table III.

2) Object detection: Object detection is a traditional but critical topic for autonomous driving systems. Since the event camera is a new sensor, labeled event datasets are scarce. Leveraged by the pseudo-labels warped from frame images in the DDD17 dataset, Chen et al. [START_REF] Chen | Pseudo-labels for supervised learning on dynamic vision sensor data, applied to object detection under ego-motion[END_REF] apply several popular CNN object detectors in event camera images, and achieved good results in motion-blurring scenarios. The PKU-DDD17-CAR dataset, i.e. the annotation of DDD17 by Li et al. [START_REF] Li | Event-based vision enhanced: A joint detection framework in autonomous driving[END_REF], is used by Cao et al. [START_REF] Cao | Fusion-based feature attention gate component for vehicle detection based on event camera[END_REF] to detect vehicles. Hu et al. [START_REF] Hu | Learning to exploit multiple vision modalities by using grafted networks[END_REF], [START_REF] Hu | v2e: From video frames to realistic dvs events[END_REF] augment respectively parts of a day and a night sequence of MVSEC dataset [START_REF] Zhu | The multivehicle stereo event camera dataset: An event camera dataset for 3d perception[END_REF] with car annotations for DNN training. Except for events-only object detection, another direction is to fuse frame images for better performance. Li et al. [START_REF] Li | Event-based vision enhanced: A joint detection framework in autonomous driving[END_REF] apply a SNN for the event stream to generate attention maps that feed to a CNN concatenated to standard frames, as in an early fusion scheme. Cao et al. [START_REF] Cao | Fusion-based feature attention gate component for vehicle detection based on event camera[END_REF] fuse events and frames at different encoding levels from parallel heads using feature attention gate components. Hu et al. [START_REF] Hu | Learning to exploit multiple vision modalities by using grafted networks[END_REF], [START_REF] Hu | v2e: From video frames to realistic dvs events[END_REF] illustrate proposed grafted networks and events synthesis from video frames with a car detection use case. Pedestrian detection is also important and is explored in [START_REF] Chen | Multi-cue event information fusion for pedestrian detection with neuromorphic vision sensors[END_REF]- [START_REF] Wan | Event-based pedestrian detection using dynamic vision sensors[END_REF], in which individual datasets are utilized according to specific cameras. Chen et al. [START_REF] Chen | Multi-cue event information fusion for pedestrian detection with neuromorphic vision sensors[END_REF] compare different accumulation methods coupled with early fusion and late fusion schemes. In Jiang et al. [START_REF] Jiang | Mixed frame-/event-driven fast pedestrian detection[END_REF], events and frames data channels are both fed into different CNNs and then fuse multiple confidence maps to achieve good pedestrian detection. Cladera et al. [START_REF] Ojeda | On-device event filtering with binary neural networks for pedestrian detection using neuromorphic vision sensors[END_REF] implement a BNN (binary neural networks) on FPGA for fast detection. Wan et al. [START_REF] Wan | Event-based pedestrian detection using dynamic vision sensors[END_REF] propose a Pedestrian-SARI dataset and alternative events representations for asynchronous CNN detection. The lane extraction problem is investigated in Cheng et al. [START_REF] Cheng | DET: A high-resolution DVS dataset for lane extraction[END_REF], in which a DET dataset, labeled lane markings in HD event camera, is released to be public. Meanwhile, several popular CNN-based lane extraction algorithms are benchmarked and the results show good performances. A more general object detection method and an annotated dataset are proposed in Perot et al. [START_REF] Perot | Learning to detect objects with a 1 megapixel event camera[END_REF], where a CNN combined with a LSTM is used to keep detections when movements stop. The proposed method is evaluated for HD events road scenes, released in their 1 Megapixel Automotive Detection Dataset. GNN architecture of Schaefer et al. [START_REF] Schaefer | Aegnn: Asynchronous event-based graph neural networks[END_REF] is evaluated on this dataset and shows state-of-the-art results, while being low computationally intensive.

3) Motion segmentation: motion segmentation or moving object detection by an event camera is more convenient than a conventional camera. This topic is addressed in [START_REF] Mitrokhin | Event-based moving object detection and tracking[END_REF]- [START_REF] Parameshwara | 0-MMS: Zero-shot multi-motion segmentation with a monocular event camera[END_REF]. In general, those approaches compensate first camera motion as background movement, as it is likely to cause the most prominent number of events. Then, moving objects are segmented through different clustering strategies. For instance, Mitrokhin et al. [START_REF] Mitrokhin | Event-based moving object detection and tracking[END_REF] group events into clusters via morphological operators, then track the multiple moving objects. Stoffregen et al. [START_REF] Stoffregen | Event-based motion segmentation by motion compensation[END_REF] warp the events several times to cluster moving objects. Zhou et al. [START_REF] Zhou | Event-based motion segmentation with spatio-temporal graph cuts[END_REF] cluster the objects via graph cut on linked space-time event graph. [START_REF] Parameshwara | 0-MMS: Zero-shot multi-motion segmentation with a monocular event camera[END_REF] cluster the objects with split and merge strategy and track grouped events. Monda et al. [START_REF]Moving object detection for event-based vision using graph spectral clustering[END_REF] don't consider background motion. Instead, moving objects are segmented from a fixed event camera flow and then grouped by a k-NN graph method.

4) Driver monitoring system: Currently, few investigations have been done with event cameras to monitor driver status. Chen et al. [START_REF] Chen | EDDD: Event-based drowsiness driving detection through facial motion analysis with neuromorphic vision sensor[END_REF] focuses on drowsiness detection and compares some classification algorithms on their event dataset. Provided with a new dataset, Chen et al. [START_REF] Chen | NeuroIV: Neuromorphic vision meets intelligent vehicle towards safe driving with a new database and baseline evaluations[END_REF] compares different CNN architectures and events accumulation schemes for driver drowsiness detection, gaze-zone, and hand-gesture recognition.

Finally, for all applications, a new trend will be to apply transformers networks on event data also, with both temporal and spatial attentions. It can even be used conjointly with SNN, such as Zhang et al. [START_REF] Zhang | Spiking transformers for event-based single object tracking[END_REF] in the case of object tracking.

E. Remaining Challenges

Although the attributes of event cameras are attractive, they are still quite young, and hence suffer several restrictions for wide applications. The first restriction involves the optimal performance in complex enlightenment scenarios, for example, the camera biases and noise. Biases need to be carefully tuned to achieve optimal perception according to the conditions (scene brightness and dynamics, ambient temperature, admissible noise, etc). Tuning the event camera's parameters is not a straightforward task, as there are too many correlated parameters to adjust. Usually, some general tries are required at first to correct the parameters. Details on how to control an event camera are given in Delbruck et al. [START_REF] Delbruck | Feedback control of event cameras[END_REF]. Other constraints concern the sensing characteristic. The most typical issue is the relative static object, as illustrated in Fig. 12 (c), because the front car and ego-vehicle have the same dynamics, the event camera barely perceives the front car. Fortunately, such a problem could be overcome by applying RNN in object detection Perot et al. [START_REF] Perot | Learning to detect objects with a 1 megapixel event camera[END_REF]. Other issues are the disturbances of shadows and of streetlights. The first is shown in Fig. 12 (d), where the shadow of a traffic sign on the ground may generate a false alarm. The second occurs when the scene is mostly illuminated by artificial lights (at night): most of them present a fast flicker effect invisible to the naked eye, but well visible to the event camera over enlightened scene. Irregular data bandwidth is another constraint caused by a huge amount of events generated at instants. Unlike a fixed bandwidth for a frame camera, the bandwidth of an event camera could reach the limits of a vehicle's onboard network capabilities, causing network jamming or package losses. To deal with this problem, Khan et al. [START_REF] Khan | Time-aggregation-based lossless video encoding for neuromorphic vision sensor data[END_REF] propose an efficient compression algorithm.

VII. CONCLUSION AND FUTURE WORKS

Although RGB cameras have cost advantages and are extensively applied in current vehicles, their inherent limitations impede the deployment of autonomous driving systems beyond constrained ODDs. To overcome these drawbacks, other types of sensing modalities are emerging. In this paper, several of these sensors have been reviewed as complements to conventional RGB cameras, these include, infrared, range-gated, polarization, and event cameras. Among them, NIR cameras are being mass-produced and integrated into production cars. Some are still in the early stages, such as the polarization, event, and range-gated cameras. A concise summary of those sensors' characteristics and typical scenarios is given in Tab. I. These additional sensing modalities should enable the extension of the operating conditions of autonomous vehicles. The review has shown that most of the perception algorithms used for these sensors are similar to those used to process RGB images. That is, the RGB channels are replaced by infrared or polarization channels. For range-gated and event cameras, since their imaging principles are different, algorithms have been specifically designed successfully to leverage their unique imaging properties. Experiments in public roads have shown their advantages as well as their weaknesses. The need to ensure reliable and resilient perception systems for safetycritical vehicle maneuvers has led to the use of these sensors in conjunction with conventional sensors. The fusion between these emergent sensors and RGB cameras has resulted in several common fusion strategies. Field trials have demonstrated that these novel sensors can be part of different perception systems contributing to their performance. It is envisaged that lower prices, high reliability, and more powerful algorithms will be attained in the future, with some of these sensors playing critical roles in future ADAS or AD systems or for specific applications involving autonomous navigation in land, air, or sea.

Fig. 1 .

 1 Fig. 1. (a) An example of electromagnetic (EM) waves. The mutually perpendicular electric field (in purple) and magnetic field (in purple) are periodically vibrating. (b) Electromagnetic spectrum. Visible light is a special kind of EM wave. (c) Solar irradiance spectrum as a function of wavelength (source from [1]). (d) At night, the area in front of the vehicle is illuminated by the low-beam headlamps. The maximum range is regulated to be around 60m. (e) Typical responsivity w.r.t wavelength for Si, InGaAs, and Ge-based image sensors (source from [2])

  While

Fig. 3 .

 3 Fig. 3. (a) A RGB-IR imaging system: the CFA is replaced by an RGB-IR filter array to capture RGB and infrared intensities (from [26]) (b) RGB (left)and NIR (right) images for the same scene. Vegetation in the NIR spectrum is "brighter" than in RGB image (from[START_REF] Brown | Multispectral sift for scene category recognition[END_REF]).

Fig. 4 .

 4 Fig. 4. (a) The architecture of a microbolometer. When exposed to LWIR radiance, the absorber generates heat, which is measured by a thermometer and converted into electrical signals. (b) and (c) RGB images (left) and thermal images (right) of the same scenes (from[START_REF] Dai | Tirnet: Object detection in thermal infrared images for autonomous driving[END_REF]). The RGB image quality is severely degraded by low illumination, glare, and fog, whereas the thermal camera is unaffected and provides clear images for object detection.

  Fig.5. (a) An example of a DMS consisting of a camera and an IR illuminator facing the driver to detect whether the driver's eyes are on the road or not (from[START_REF] Vicente | Driver gaze tracking and eyes off the road detection system[END_REF]). (b) Face landmark detection results (from[START_REF] Fridman | Owl and lizard patterns of head pose and eye pose in driver gaze classification[END_REF]). (c) 15 zones for gaze classification (from[START_REF] Yoon | Driver gaze detection based on deep residual networks using the combined single image of dual near-infrared cameras[END_REF]).

  has been passed in 2019 to mandate automakers to install advanced safety systems including DMS in new cars in the EU market from 2022. Because an active NIR night vision system is barely perceptible by human eyes and conserves abundant contextual details, it plays a critical role in modern DMS. Face and eye detection and tracking via image processing are usually required as a preliminary step before detecting gaze and drowsiness. Fig. 5 (b) shows an example of detected facial landmarks. In recent years, DNNs dominate this domain. For example, Yoon et al. [78] utilize a VGG network for face detection and Park et al. [89] develop a Faster-RCNN based eye detection method.

Fig. 6 .

 6 Fig. 6. Different strategies of fusing RGB and thermal images according to specific stages.

Fig. 7 .

 7 Fig. 7. Object detection in thermal images. Pedestrians, vehicles, etc are localized in 2D bounding boxes with different colors. (from [62]).

Fig. 8 .

 8 Fig. 8. The imaging principle of a range-gated camera: merging several image slices into a final image. A very short exposure defined by a gating function yields an image slice. The backscattering interference outside the range defined by a gating function has almost no effect on the outcome.

S 0 =

 0 I 0 + I 90 = I 45 + I 135 S 1 = I 0 -I 90 , S 2 = I 45 -I 135 (6) where I 0 , I 45 , I 90 and I 135 are the optical intensities at the corresponding polarization direction, i.e. 0, 45, 90 and 135.

Fig. 10 .

 10 Fig. 10. (a) Unpolarized light beams could be converted to polarized light after reflection. (from [144]). (b) A polarizer's function is to convert an unpolarized beam into a (linear) polarized beam (from [145]). (c) A schematic diagram of SONY IMX250 CMOS sensor. To acquire color and polarization information, a micro-polarizer array and a color filter array are placed on top of pixels.

Fig. 11 .

 11 Fig. 11. (a) Retinomorphic event vision with spiking output (from[START_REF] Posch | Retinomorphic event-based vision sensors: Bioinspired cameras with spiking output[END_REF]). Human retina layers and corresponding dynamic vision sensor circuit for one pixel (left). Sample of the signal waveform (top right), and the response of an array of such pixels accumulated for a short time, for a sample scene showing a moving person (bottom right). (b) RGB frames from a standard camera (on top), compared to events flow (bottom), from the same time-synced scene issued from DSEC dataset[START_REF] Gehrig | DSEC: A stereo event camera dataset for driving scenarios[END_REF]. Red points depict positive events, while blue points depict negative events.

  (a) There is a bike on right side. (b) What comes after the tunnel? (c) Front car's speed equals ours so it is not visible in events. (d) With events, shadows can look like noise or even obstacles.

Fig. 12 .

 12 Fig. 12. Advantages (a), (b), and limits (c), (d), of event camera images compared to normal images: (a) Night scene, (b) Glare light, (c) Stationary car, (d) Shadow on the ground. Scenes taken from DSEC dataset [159].

TABLE I SUMMARY

 I OF VISION SENSOR FEATURES.

					Characteristic
	Visual Sensor	Resolution	Frame Rate	Night Vision	Adverse Weather	Specular Reflection	Dynamic Range	Cost	Technology Maturity	Applied Scenarios
	RGB Camera	* * *	* *	*	*	*	* *	*	* * *	Object detection in good conditions General purpose
	Near Infrared Camera *	* * *	* *	* *	* *	* *	* *	*	* * *	Driver monitoring system Object detection at night
	Thermal(LWIR) Camera	*	*	* * *	* * *	* * *	-	* *	* * *	Object detection at night and adverse weather, living things detection
	Range Gated Camera *	* *	* *	* * *	* * *	* * *	* *	* * * *	Object detection at night and adverse weather
	Polarization Camera	* * *	* *	*	* *	* * *	* * *	* *	*	Mitigate specular reflection, HDR imaging, object detection in adverse weather
	Event Camera	* *	* * *	* *	*	*	* * *	* *	* *	Super fast object detection driver monitoring system
	* Active sensor, used with a joint illuminator.

TABLE II SEVERAL

 II TYPICAL OPEN DATASETS FOR MULTIPLE SENSING MODALITIES (INFRARED, GATED INFRARED AND POLARIZATION CAMERAS).

	Name				Modality	Size	Annotation	Location	Year
		RGB NIR	LWIR Gated Polar Event
	FLIR-ADAS [72]	✓		✓		13K	Person, car, bicycle dog, other vehicles	US	2020
	C3I-ADAS [109]			✓		39K	Person, car, pole, bike, bicycle, bus	EU	2022
	KAIST [104]	✓		✓		95K	Person, pedestrian and cyclist	Korea	2015
	LLVIP [223]	✓		✓		18K	Pedestrian	China	2021
	SCUT [105]			✓		211K	Walk/ride/squat people.	China	2019
	ZUT [106]			✓		110K	9 classes including pedestrian, cyclist, animal	EU	2020
	RANUS [224]	✓	✓			4K	10 classes including vehicle, road, pedestrian, vegetation	Korea	2018
	SparsePPG [225]	✓	✓			19 seq.	With ground truth driver PPG waveform	US	2018
	DENSE [226]	✓		✓	✓	13K	4 classes including pedestrian and car	EU	2020
	PolarLITIS [162]	✓			✓	2.5K	Car, Person, bike, motorbike	EU	2021
	ZJU-RGB-P [164]	✓			✓	394	Pixel-wise semantic segmentation building, glass, car, pedestrian, road, etc	China	2021
	ViVID++ [227]	✓		✓	✓	22 driving seq.	Positions	Korea	2022

TABLE III EVENT

 III CAMERA DATASETS WITH DRIVING SCENES. TOP PART LISTS LOW RESOLUTION DATASETS (< 1280 × 720PX), BOTTOM PART LISTS HIGH RESOLUTION DATASETS (≥ 1280 × 720PX).

	Name	Pixel resolution	Other modalities	Aimed problems	Size	Annotations	Location	Year
	PRED18 [194] (includes previous PRED16 [193])	240 × 180	Grey frames *	Mobile target following	1.25h	prey size, prey position	Northern Ireland	2018, 2016
	DDD20 [228] (includes previous DDD17 [207])	346 × 260	Grey frames * IMU * Car data GNSS	Vehicle control	39h + 12h	-	USA, Swiss, Germany	2020, 2017
	PKU-DDD17-CAR † [204]			Detection		"Car"		2019
	Ev-Seg † [229]			Segmentation	20 intervals	Semantic seg		2019
	N-Cars [230]	304 × 240	-	Classification	24K samples	"Car" "Background"	unknown	2018
			Grey frames *
			Grey stereo camera
	MVSEC [206]	346 × 260 2 cameras	IMU LiDAR	Depth Localisation	1h	Depth	USA	2018
			GPS
			Motion capture
	MVSEC-OF ‡ [231]			Optical flow		Optical flow		2018
	MVSEC-DAY20 ‡ [210]			Detection	partial seq. "outdoor day2"	"Car"		2020
	MVSEC-NIGHTL21 ‡ [211]			Detection	partial seq. "outdoor night1"	"Car"		2021
			Grey frames *
	Slasher dataset [232]	346 × 260	Steering	Vehicle control	2 sequences	-	Swiss	2019
			Radio localisation
	Event Camera Driving Sequences [233]	640 × 480	RGB camera	Frames reconstruction	40 sequences	-	Swiss	2019
	CED [170]	346 × 260 RGB event cam	RGB frames *	Color frames reconstruction	50min	-	unknown	2019
	Pedestrian Detection Dataset [234]	346 × 260 RGB event cam	-	Detection	12 recordings	"Pedestrian"	China	2019
	EDDD § [221]	346 × 260	-	Driver monitoring	260 sequences	Drowsiness	China	2020
	NeuroIV § [199]	346 × 260 RGB event cam	RGB frames * NIR frames Depth maps	Driver monitoring	27K samples	Drowsiness Hand-gestures Gaze-zones	China	2020
	GAD Dataset [235]	304 × 240	-	Detection	39h	"Car" "Pedestrian"	France	2020
			RGB frames *
	Brisbane Event VPR [236]	346 × 260 RGB event cam	RGB camera IMU *	Visual place recognition	8km	Landmarks	Australia	2020
			GPS
	DENSE ¶,|| [237]	346 × 260	RGB frames Depth maps	Depth Segmentation	8K samples	Depth Semantic seg	-	2020
	DSEC [159]	640 × 480 2 cameras	2x RGB cameras LiDAR § RTK GPS §	Depth Localisation	53min	Depth	Swiss	2021
	DSEC-OF ** [238]			Optical flow		Optical flow		2021
	EventScape ¶ [239]	512 × 256	RGB frames Depth maps Car data	Depth Segmentation	2h	Depth Semantic seg	-	2021
	Pedestrian-SARI § [215]	346 × 260	Grey frames *	Detection	141 sequences	"Person"	China	2021
			RGB camera
	ViVID++ [227] (driving scenes part)	640 × 480	Thermal camera LiDAR (for 8 seq.)	Localisation VSLAM	22 sequences	Positions	South Korea	2022
			RTK GPS
	DET [208]	1280 × 800	-	Lane extraction	5h	Road lanes	China	2019
						"Car"
	1Mp Detection [200]	1280 × 720	-	Detection	14h	"Pedestrian"	France	2020
						"Two-wheeler"

* Available from the event camera itself. † Extension of DDD17, providing ground truth to other problem. ‡ Extension of MVSEC, providing ground truth to other problem. § Not available for download, might be available upon request to the authors. ¶ Simulated data. || Distinct from DENSE dataset for LWIR and range-gated cameras [226] presented in Table II and in Section IV-C. ** Extension of DSEC, providing ground truth to other problem.

Julien Moreau is with Université de technologie de Compiègne, CNRS, Heudiasyc (Heuristics and Diagnosis of Complex Systems), CS 60319 -60203

https://www.ovt.com/sensors/OX08B4C

https://www.sony-semicon.co.jp/products/common/pdf/IMX324 424.pdf

https://ec.europa.eu/commission/presscorner/detail/en/IP 19 1793

https://www.heliaus.eu/news-events/

https://www.dense247.eu/

https://www.uni-ulm.de/en/in/driveu/projects/dense-datasets/

http://inivation.com/wp-content/uploads/2020/09/DAVIS346.pdf

http://inivation.github.io/inivation-docs/Hardwareuserguides/User guide -DAVIS USB3 development kit.html

ACKNOWLEDGMENT

This work has been carried out within SIVALab, joint laboratory between Renault and Heudiasyc UMR UTC/CNRS.