Jean-Philippe Tarel
email: jean-philippe.tarel@ifsttar.fr

Nicolas Hautière
email: nicolas.hautiere@ifsttar.fr

Laurent Caraffa
email: lau-rent.caraffa@ifsttar.fr.

Aurélien Cord
email: aurelien.cord@ifsttar.fr

Houssam Halmaoui
email: houssam.halmaoui@ifsttar.fr

Dominique Gruyer
email: minique.gruyer@ifsttar.fr.

J.-P Tarel

Vision Enhancement in Homogeneous and Heterogeneous Fog

One source of accidents when driving a vehicle is the presence of fog. Fog fades the colors and reduces the contrasts in the scene with respect to their distances from the driver. Various camera-based Advanced Driver Assistance Systems (ADAS) can be improved if efficient algorithms are designed for visibility enhancement in road images. The visibility enhancement algorithm proposed in [1] is not optimized for road images. In this paper, we reformulate the problem as the inference of the local atmospheric veil from constraints. The algorithm in [1] thus becomes a particular case. From this new derivation, we propose to better handle road images by introducing an extra constraint taking into account that a large part of the image can be assumed to be a planar road. The advantages of the proposed local algorithm are the speed, the possibility to handle both color and gray-level images, and the small number of parameters. A new scheme is proposed for rating visibility enhancement algorithms based on the addition of several types of generated fog on synthetic and camera images. A comparative study and quantitative evaluation with other state-of-the-art algorithms is thus proposed. This evaluation demonstrates that the new algorithm produces better results with homogeneous fog and that it is able to deal better with the presence of heterogeneous fog. Finally, we also propose a model allowing to evaluate the potential safety benefit of an ADAS based on the display of defogged images.

I. INTRODUCTION

A cause of vehicle accidents is reduced visibility due to bad weather conditions such as fog. This suggests that an algorithm able to improve visibility and contrast in foggy images will be useful for various camera-based Advanced Driver Assistance Systems (ADAS). In [START_REF] Hautière | Mitigation of visibility loss for advanced camera based driver assistances[END_REF], it is shown for several types of detection algorithms, that a visibility enhancement pre-processing allows to improve detection performance in presence of fog. This is due to a better respect after pre-processing of the assumption that objects to be detected have a minimal contrast which is set to be uniform over the whole image. Two kinds of ADAS can be considered. The first possibility is to display the image from a frontal camera after visibility enhancement. We call this kind of ADAS a Fog Vision Enhancement System (FVES). The second possibility is to combine visibility enhancement pre-processing with detection of stopped cars/moving cars/pedestrians/twowheeled vehicles, to deliver adequate warning. An example is a warning when the distance to the previous moving vehicle is too short with respect to the driver's speed.

For ADAS based on the use of a single camera in the vehicle, the contrast enhancement algorithm must be able to robustly process each image in a sequence in real time. The key problem is that, from a single foggy image, contrast enhancement is an ill-posed problem. Indeed, due to the physics of fog, visibility restoration requires to estimate both the scene luminance without fog and the scene depth-map. This implies estimating two unknown parameters per pixel from a single image.

The first approach proposed to tackle the visibility restoration problem from a single image is described in [START_REF] Narashiman | Interactive deweathering of an image using physical model[END_REF]. The main idea is to provide interactively an approximate depthmap of the scene geometry allowing to deduce an approximate luminance map without fog. The drawback of this approach for camera-based ADAS is clear: it is not easy to provide the approximate depth-map of the scene geometry from the point of view of the driver all along its road path. In [START_REF] Hautière | Towards fog-free in-vehicle vision systems through contrast restoration[END_REF], this idea of approximate depth-map was refined by proposing several simple parametric geometric models dedicated to road scenes seen in front of a vehicle. For each type of model, the parameters are fit on each view by maximizing the scene depths globally without producing black pixels in the enhanced image. The limit of this approach is the lack of flexibility of the proposed geometric models.

During the same period of time, another approach was proposed in [START_REF] Tan | Vvisibility enhancement for roads with foggy or hazy scenes[END_REF] based on the use of color images with pixels having a hue different from gray. A difficulty with this approach, for the applications we focus on, is that a large part of the image corresponds to the road which is gray and white. Moreover, in many intelligent vehicle applications, only graylevel images are processed.

More recently and for the first time in [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF], [START_REF] Tan | Visibility in bad weather from a single image[END_REF], [START_REF] He | Single image haze removal using dark channel prior[END_REF], three visibility enhancement algorithms were proposed working from a single gray-level or color image without using any other extra source of information. These three algorithms rely on a local spatial regularization to solve the problem. Being local, these algorithms can cope with homogeneous and heterogeneous fog. The main drawback of the algorithms in [START_REF] Tan | Visibility in bad weather from a single image[END_REF] and [START_REF] He | Single image haze removal using dark channel prior[END_REF] is their processing time: 5 to 7 minutes and 10 to 20 seconds on a 600 × 400 image, respectively. The algorithm proposed in [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF] is much faster with a processing time of 0.2 second on a Dual-Core PC on similar image size. A fast variant of [START_REF] He | Single image haze removal using dark channel prior[END_REF] was very recently proposed in [START_REF]Guided image filtering[END_REF]. The disadvantage of these three visibility enhancement methods, and of the other variants or improved algorithms more recently proposed, is that they are not dedicated to road images and thus the road part of the image which is gray may be over-enhanced. This is due to the ambiguity between light colored objects and the presence of fog, and leads to the apparition of unwelcome structures in the enhanced image, as it is illustrated on three images in Fig. 2.

The important property of a road image is that a large part of the image corresponds to the road way which can reasonably be assumed to be planar. Visibility enhancement dedicated to planar surface was first proposed in [START_REF] Hautière | Contrast restoration of foggy images through use of an onboard camera[END_REF], but this algorithm is not able to correctly enhance visibility of objects out of the road plane. Recently, a visibility enhancement algorithm [START_REF] Hautière | Mitigation of visibility loss for advanced camera based driver assistances[END_REF] dedicated to road images was proposed which was also able to enhance contrast for objects out of the road plane. This algorithm makes good use of the planar road assumption but relies on an homogeneous fog assumption.

In this work, we formulate the restoration problem as the inference of the atmospheric veil from to three constraints. The first constraint relies on photometrical properties of the foggy scene. The second constraint, named the no-black-pixel constraint, was not used in [START_REF] Tan | Visibility in bad weather from a single image[END_REF], [START_REF] He | Single image haze removal using dark channel prior[END_REF] and [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF]. It involves filtering the image. The algorithm described in [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF] corresponds to the particular case where two constraints are used with the median filter. To take into account that a large part of the image is a planar road, as introduced first in [2], a third constraint based on the planar road assumption is added. The new algorithm can thus be seen as the extension of the local visibility enhancement algorithm [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF] combined with the roadspecific enhancement algorithm [START_REF] Hautière | Contrast restoration of foggy images through use of an onboard camera[END_REF]. The proposed algorithm is suitable for FVES since it is able to process gray-level as well as color images and runs close to real time.

To compare the proposed algorithm to previously presented algorithms, we propose an evaluation scheme and we build up a set of synthetic and camera images with and without homogeneous and heterogeneous fog. The algorithms are applied on foggy images and results are compared with the images without fog. For FVES in which the image after visibility enhancement is displayed to the driver, we also propose an accident scenario and a model of the probability of fatal injury as a function of the setting of the visibility enhancement algorithm.

The article is structured as follows. Section II presents the fog model we use. In section III, the multiscale retinex algorithm (MSR) [START_REF] Jobson | A multiscale retinex for bridging the gap between color images and the human observation of scenes[END_REF] and the contrast-limited adaptive histogram equalization (CLAHE) are summarized. In section IV, different approaches of visibility enhancement are described: based on the planar assumption (PA) [START_REF] Hautière | Contrast restoration of foggy images through use of an onboard camera[END_REF], on a free-space segmentation (FSS) [START_REF] Hautière | Mitigation of visibility loss for advanced camera based driver assistances[END_REF], on our new derivation using the no-black-pixel constraint (NBPC) [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF], on Dark Chanel Prior (DCP) [START_REF] He | Single image haze removal using dark channel prior[END_REF], and finally the new combined algorithm named NBPC+PA is proposed. In section V, a comparison is provided between MSR, CLAHE, DCP, FSS, NBPC and NBPC+PA algorithms based on a quantitative evaluation on two sets of 66 × 4 and 10 × 4 foggy images, illustrating the properties of each algorithm. Finally, in section VI an accident scenario is proposed with a model allowing to estimate potential safety benefits of FVES.

II. EFFECTS OF FOG

Assuming an object of intrinsic luminance L 0 (u, v), its apparent luminance L(u, v) in presence of a fog of extinction coefficient k is modeled by Koschmieder's law [START_REF] Hautière | Automatic fog detection and estimation of visibility distance through use of an onboard camera[END_REF]:

L(u, v) = L 0 (u, v)e -kd(u,v) + L s (1 -e -kd(u,v) ) (1)
where d(u, v) is the distance of the object at pixel (u, v) and L s is the luminance of the sky. As described by ( 1), fog has two effects: first an exponential decay e -kd(u,v) of the intrinsic luminance L 0 (u, v), and second the addition of the luminance of the atmospheric veil L s (1 -e -kd(u,v) ) which is an increasing function of the object distance d(u, v). These two effects can be seen on the same scene in Fig. 1 for different values of k. The meteorological visibility distance is defined as d m = -ln(0.05) k , see [START_REF] Hautière | Automatic fog detection and estimation of visibility distance through use of an onboard camera[END_REF].

From now on, we assume that the camera response is linear, and thus image intensity I is substituted to luminance L.

III. COLOR AND CONTRAST ENHANCEMENT

We now recall the Multiscale Retinex (MSR) and Contrastlimited adaptive histogram equalization (CLAHE) algorithms. These two algorithms are not based on Koschmieder's law [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF] and thus are only able to remove a fog of constant thickness on an image. They are not visibility enhancement algorithms. However, we found it interesting to include these two algorithms in our comparison in order to verify that visibility enhancement algorithms achieve better results.

A. Multiscale Retinex (MSR)

The multiscale retinex (MSR) is a non-linear image enhancement algorithm proposed by [START_REF] Jobson | A multiscale retinex for bridging the gap between color images and the human observation of scenes[END_REF]. The overall impact is to brighten up areas of poor contrast/brightness but not at the expense of saturating areas of good contrast/brightness. The MSR output is simply the weighted sum of the outputs of several single scale retinex (SSR) at different scales.

Each color component being processed independently, the basic form of the SSR for on input image I(u, v) is:

R k (u, v) = log I(u, v) -log[F k (u, v) * I(u, v)] ( 2
)
where R k (u, v) is the SSR output, F k represents the k th surround function, and * is the convolution operator. The surround functions, F k are given as normalized Gaussians:

F k (u, v) = κ k e -(u 2 +v 2 )/σ 2 k (3)
where σ k is the scale controlling the extent of the surround and κ k is for unit normalization. Finally the MSR output is:

R(u, v) = k=K ∑ k=1 W k R k (u, v) (4)
where W k is the weight associated to F k . The number of scales used for the MSR is, of course, application dependent. We have tested different sets of parameters, and we did not find a better parametrization than the one proposed by [START_REF] Jobson | A multiscale retinex for bridging the gap between color images and the human observation of scenes[END_REF]. It consists of three scales representing narrow, medium, and wide surrounds that are sufficient to provide both dynamic range compression and tonal rendition:

K = 3, σ 1 = 15 , σ 2 = 80, σ 3 = 250, and W k = 1/3 for k = 1, 2, 3.
Results obtained using the multiple retinex on three foggy images are presented in column two of Fig. 2.

B. Contrast-Limited Adaptive Histogram Equalization (CLAHE)

Contrast-limited adaptive histogram equalization (CLAHE) locally enhances the image contrast. As proposed in [START_REF] Zuiderveld | Contrast limited adaptive histogram equalization[END_REF], CLAHE operates on 8 × 8 regions in the image, called tiles, rather than the entire image. Each tile's contrast is enhanced, so that the histogram of the output region approximately matches a flat histogram. The neighboring tiles are then combined using bilinear interpolation to eliminate artificially induced boundaries. The enhanced contrast, especially in homogeneous areas, is limited to avoid amplifying noise or unwelcome structures, such as object textures, that might be present in the image. The parameter controlling this limitation was optimized on 40 images, varying both the scene and the fog properties.

Results obtained using the CLAHE algorithm are presented in column three of Fig. 2.

IV. ENHANCEMENT BASED ON KOSCHMIEDER'S LAW

Four visibility enhancement algorithms are now presented: enhancement assuming a planar scene assumption (PA), enhancement with free-space segmentation (FSS), enhancement with the no-black-pixel constraint (NBPC) and enhancement with the dark channel prior (DCP). The advantages and limits of these algorithms are discussed. A new algorithm, named NBPC+PA, which combines the advantages of PA and NBPC algorithms, is proposed. The results obtained by the five algorithms are presented in Fig. 2 on three images.

A. With the planar assumption (PA)

Dedicated to in-vehicle applications, the algorithm proposed in [START_REF] Lavenant | Procédé de détermination de la distance de visibilité et procédé de détermination de la présence d'un brouillard[END_REF], [START_REF] Hautière | Automatic fog detection and estimation of visibility distance through use of an onboard camera[END_REF] is able to detect the presence of fog and to estimate the visibility distance which is directly related to the k in Koschmieder's law [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF]. This algorithm, also known as the inflection point algorithm, mainly relies on three assumptions: fog is homogeneous, the main part of the image displays the road surface which is assumed to be planar and homogeneous surface. From the estimated fog parameters, the contrast in the road part of the image can be restored as explained in [START_REF] Hautière | Contrast restoration of foggy images through use of an onboard camera[END_REF].

Using the planar road surface assumption and knowing the approximate camera calibration with respect to the road, it is possible to associate a distance d with each line v of the image:

d = λ v -v h if v > v h (5)
where v h is the vertical position of the horizon line in the image and λ depends on intrinsic and extrinsic parameters of the camera, see [START_REF] Hautière | Automatic fog detection and estimation of visibility distance through use of an onboard camera[END_REF] for details.

Using the assumption of a road with homogeneous photometric properties (I 0 is constant), fog can be detected and the extinction coefficient of the atmosphere k can be estimated using Koschmieder's law [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF]. After substitution of d given by ( 5), (1) becomes:

I(v) = I 0 e -k λ v-v h + I s (1 -e -k λ v-v h ) (6)
By taking twice the derivative of I with respect to v, the following is obtained:

d 2 I dv 2 (v) = k λ (I 0 -I s ) (v -v h ) 3 e -k λ v-v h kλ v -v h -2 (7)
The equation

d 2 I dv 2 = 0 has two solutions. The solution k = 0 is of no interest. The only useful solution is given by k = 2(v i -v h ) λ ,
where v i denotes the position of the inflection point of I(v). An illustration of this method is presented in Figure 3(b). The value of I s is obtained as the intensity of the sky. Most of the time, it corresponds to the maximum intensity in the image. Having estimated the value of k and I s , the pixels on the road plane can be restored as R(u, v) by reversing Koschmieder's law [START_REF] Hautière | Contrast restoration of foggy images through use of an onboard camera[END_REF]:

R(u, v) = I(u, v)e k λ v-v h + I s (1 -e k λ v-v h ) (8)
As in [START_REF] Hautière | Towards fog-free in-vehicle vision systems through contrast restoration[END_REF], the introduction of a clipping plane in equation ( 5) allows to apply the reverse of Koschmieder's law in the whole image. More precisely, the used geometrical model consists in the road plane [START_REF] Tan | Vvisibility enhancement for roads with foggy or hazy scenes[END_REF] in the bottom part of the image, and in a vertical plane in front of the camera in the top part of the image. The height of the line which separates the road model and the clipping plane is denoted c. As a consequence only large distances are clipped. In summary, the geometrical model d c (u, v) of a pixel at position (u, v) is expressed as:

d c (u, v) =        λ (v -v h ) if v > c λ (c -v h ) if v ≤ c (9)
Results obtained with the previous model where the clipping plane is set at the meteorological visibility distance d m are shown on three foggy images in column four of Fig. 2. From these results, it appears that only the road part of the image is correctly restored.

B. With free-space segmentation (FSS)

To be able to enhance the visibility in the rest of the scene, an estimate of the depth d(u, v) of each pixel is needed. In [START_REF] Hautière | Towards fog-free in-vehicle vision systems through contrast restoration[END_REF], a parameterized 3D model of the road scene was proposed with a reduced number of geometric parameters. Even if these models are relevant for most road scenes and even if the parameters of the selected model are optimized to achieve best enhancement without black pixel in the resulting image, the proposed model is not generic enough to handle all traffic configurations.

In [START_REF] Hautière | Mitigation of visibility loss for advanced camera based driver assistances[END_REF], a different scheme is proposed. Once again, the road is assumed to be planar with a clipping plane, see [START_REF] Hautière | Contrast restoration of foggy images through use of an onboard camera[END_REF]. When a geometric model ( 9) is assumed, the contrast of objects belonging to the road plane is correctly restored, as seen in previous section. Conversely, the contrast of vertical objects of the scene (vehicles, trees,...) is incorrectly restored since their depth in the scene is largely overestimated. Consequently, their restored intensity using (8) are negative and thus set to zero in the enhanced image. These are named black pixels. The set of all black pixels gives a segmentation of the image in two regions, one inside the road plane in 3D and the other outside. This allows to deduce the free-space region, as illustrated in green and red in Fig. 3, see [START_REF] Hautière | Free space detection for autonomous navigation in daytime foggy weather[END_REF] for details.

For each pixel in the free-space region, the road plane model ( 5) is correct. For pixels out of the road plane (red region in third image of Fig. 3), it is proposed in [START_REF] Hautière | Mitigation of visibility loss for advanced camera based driver assistances[END_REF] to use the geometric model ( 9) and, for each pixel, to search for the smallest value of c which leads to a positive intensity in the restored image. The obtained values are denoted c min (u, v). Indeed, when c is close to the v h , the clipping plane is far from the camera and the visibility is only slightly enhanced. The larger the value of c, the closer the clipping plane is to the camera, and thus the stronger the enhancement. The enhancement in ( 8) can be so strong that enhanced intensity becomes negative.

Every c min (u, v) value can be associated with a distance d min (u, v) using ( 9). The resulting depthmap on the foggy image is displayed in Fig. 3. Then, a rough estimate of the depthmap d(u, v) is obtained as a fixed percentage p of depth map d min (u, v). Percentage p specifies the strength of the enhancement and is usually set to 95% for this method. The depthmap is used to enhance the contrast on the whole image using the reversed Koschmieder's law as illustrated in the fourth image of Fig. 3. The algorithm is detailed in [START_REF] Hautière | Mitigation of visibility loss for advanced camera based driver assistances[END_REF], [START_REF] Hautière | Free space detection for autonomous navigation in daytime foggy weather[END_REF] and more results are shown in the sixth column of Fig. 2.

C. With no-black-pixel constraint (NBPC)

In [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF], an algorithm which relies on a local regularization is proposed. The distance d(u, v) being unknown, the goal of the visibility enhancement in a single image can be set as inferring the intensity of the atmospheric veil V (u, v) = I s (1 -e -kd(u,v) ). Most of the time, the intensity of the sky I s corresponds to the maximum intensity in the image, and thus I s can be set to one without loss of generality, assuming the input image is normalized. After substitution of V in (1) and with I s = 1, Koschmieder's law is rewritten as:

I(u, v) = I 0 (u, v)(1 -V (u, v)) +V (u, v) (10)
The foggy image I(u, v) is enhanced as R, the estimate of I 0 , simply by the reversing of (10):

R(u, v) = I(u, v) -V (u, v) 1 -V (u, v) (11)
The enhancement equation provided by Koschmieder's law is a linear transformation. Interestingly, it gives the exact link between its intercept and its slope.

The atmospheric veil V (u, v) being unknown, let us enumerate the constraints which apply to V (u, v). V (u, v) must be higher or equal to zero and V (u, v) is lower than I(u, v):

0 ≤ V (u, v) ≤ I(u, v) (12)
These are the photometric constraints as named in [START_REF] Tan | Visibility in bad weather from a single image[END_REF].

We now introduce a new constraint, not used in [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF], which focuses on the reduction of the number of black pixels in the enhanced image R. This constraint is named no-black-pixel constraint and states that the local standard deviation of the enhanced pixels around a given pixel position must be lower than its local average:

f std(R) ≤ R ( 13
)
where f is a factor usually set to 1. In case of a Gaussian distribution of the intensities and f = 1, this criterion implies 15.8% of the intensities becoming black. Using f = 2 leads to a stronger criterion where only 2.2% of the intensities become black.

The difficulty with this last constraint is that it is set as a function of the unknown result R. Thanks to the linearity of ( 11), the no-black-pixel constraint can be turned into a constraint involving V and I only. For this purpose, we now enforce local spatial regularization by assuming that locally around pixel position (u, v), the scene depth is constant and the fog is homogeneous, i.e equivalently, the atmospheric veil locally equals V (u, v) at the central position. Under this assumption, we derive using [START_REF] Hautière | Automatic fog detection and estimation of visibility distance through use of an onboard camera[END_REF] that the local averages Ī and R are related by R = Ī-V (u,v) 1-V (u,v) and that the standard deviations std(I) and std(R) are related by std(R) = std(I)

1-V (u,v) . We therefore obtain, after substitution of the two previous results in [START_REF] Lavenant | Procédé de détermination de la distance de visibilité et procédé de détermination de la présence d'un brouillard[END_REF], the no-black-pixel constraint rewritten as a function of V (u, v) and I:

V (u, v) ≤ Ī -f std(I) (14)
The atmospheric veil V (u, v) is set as a percentage p of the minimum over the two previous upper bounds ( 12) and ( 14):

V (u, v) = p min(I(u, v), Ī -f std(I)) ( 15
)
Percentage p specifies the strength of the enhancement and is usually set to 95% for this method. The enhanced image is obtained by applying (11) using the previous V . V may be thresholded to zero in case of negative values. The algorithm derived from the photometric and no-black-pixel constraints turns out to be the one described in [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF] where Ī is obtained as the median of the local intensities in a window of size s v and the standard deviation as the median of the absolute differences between the intensities and Ī using same window size. Other edge-preserving filters can be also used, such as the median of median along lines [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF] or bilateral filtering. Due to edge smoothing of complex borders, small artifacts are produced in the restored image around complex depth discontinuities such as tree silhouettes. A post-processing with the cross/joint bilateral filter on V using I as a guide can be used to clean these artifacts as proposed in [START_REF] Yu | Physics-based fast single image fog removal[END_REF]. This enhancement algorithm is presented with a gray-level input image but can be extended easily to color images (r(u, v), g(u, v), b(u, v)) by applying the photometric constraint to substitute I in the previous equation by the gray-level image I(u, v) = min(r(u, v), g(u, v), b(u, v)) after adequate white balance. The obtained V gives the amount of white that must be subtracted to the three color channels. The algorithm is available 1 in Matlab T M . Fig. 2 shows the visibility enhancement obtained by the NBPC algorithm in the seventh column. One can notice that the contrast on the texture of the road part of the resulting image is over-enhanced. This is due to the fact that the atmospheric veil V (u, v) in the road part of the image is overestimated. This is a consequence of the locality property of the NBPC algorithm. As detailed in [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF], a final gamma mapping can be used to attenuate this problem.

D. Dark Channel Prior (DCP)

An algorithm for local visibility enhancement named Dark Channel Prior was proposed in [START_REF] He | Single image haze removal using dark channel prior[END_REF]. For gray level images, the DCP algorithm consists first in applying a morphological erosion or opening with a structuring element of size s v , which removes all white objects with a size smaller than s v . Then, the atmospheric veil V (u, v) is set as a percentage p of the opening result. This first step can thus be seen as a particular case of the NBPC algorithm using a morphological operator as filter and with f = 0. Similarly to what was explained in the previous section, an erosion or an opening does not preserve accurate complex borders along depth discontinuities. In [START_REF] He | Single image haze removal using dark channel prior[END_REF], a matting algorithm is used to restore complex borders in V . A faster alternative consists in using iterations of the guided-filter, as proposed in [START_REF]Guided image filtering[END_REF]. The cross/joint bilateral filter is another alternative. The implementation used in our experiments is based on the guided filter. The enhanced image is obtained by applying the inverse of Koschmieder's law [START_REF] Hautière | Automatic fog detection and estimation of visibility distance through use of an onboard camera[END_REF] using the previous V . Fig. 2 shows the visibility enhancement obtained by the DCP algorithm in the fifth column. A final fixed gamma mapping is used to attenuate the darkening of the road region.

As in the NBPC algorithm, color images are handled by using r(u,v),g(u,v),b(u,v)) as the input graylevel image.

I(u, v) = min(

E. Combining the no-black-pixel constraint and the planar assumption (NBPC+PA)

On the one hand, the visibility enhancement with FSS, as explained in section IV-B, performs a segmentation to split the image into three regions: the sky, the objects out of the road plane, and the free-space in the road plane. Various enhancement processes are performed depending on the region. The difficulty with an approach based on segmentation is to manage correctly the transition between regions. On the other hand, the visibility enhancement with NBPC and DCP are local methods which are not dedicated to road images and which are in difficulties in presence of a large uniform gray region such as a road, as underlined in [START_REF] Tarel | Improved visibility of road scene images under heterogeneous fog[END_REF]. Indeed, the atmospheric veil in the bottom part of the image is overestimated.

To combine the advantages of the two approaches, we introduce in the NBPC a third constraint, during the inference of the atmospheric veil V (u, v), which prevents over-estimation in the bottom part of the image by taking into account the reduced distance between the camera and the road.

In practice, it is very rare to observe fog with a meteorological visibility distance d m lower than 60m. Assuming that the minimum meteorological visibility distance is sixty meters, i.e d m ≥ 60, we deduce k ≤ -ln(0.05) 60 . We also assume that the road is a plane up to a certain distance, and that the camera calibration is known with respect to the road, so that λ and v h are known. Thus, using the last term of equation ( 6), the atmospheric veil is subject to the following third constraint :

V (u, v) ≤ I s (1 -e ln(0.05)λ d min (v-v h ) ) (16)
where d min can be set for instance to the minimum distance 60m. We named ( 16) the planar assumption constraint. As in the NBPC algorithm, the atmospheric veil V (u, v) is set to a percentage p of the minimum over the three upper bounds:

V (u, v) = p min(I(u, v), Ī -f std(I), I s (1 -e ln(0.05)λ d min (v-v h ) )) (17)
The enhanced image results of the application of [START_REF] Hautière | Automatic fog detection and estimation of visibility distance through use of an onboard camera[END_REF]. We named it visibility enhancement with NBPC+PA.

In the presence of fog with a meteorological visibility distance lower than d min = 60m, this third constraint limits the possibilities of enhancement which will be partial at short distances even with p = 100%. An interesting consequence of introducing the third constraint is that the final gamma mapping used in NBPC and DCP algorithms is no longer needed to attenuate the image darkening, as illustrated in the eighth column of Fig. 2.

Rather than fixing d min = 60m, an alternate approach, not tested here, would be to run a fog detection algorithm with the k estimation as explained in section IV-A and to use the estimated k in ( 16) instead ofln(0.05) 60 . This estimation of k assumes an homogeneous fog. Therefore, this refinement should lead to more accurate results compared to the NBPC+PA when the fog is uniform, but may lead to a bias when the fog is not homogeneous.

V. EXPERIMENTS

To evaluate visibility enhancement algorithms, we need images of the same scene with and without fog. However, obtaining such pairs of images is extremely difficult in practice since it requires to check that the illumination conditions are the same into the scene with and without fog. As a consequence, for the evaluation of the proposed visibility enhancement algorithm and its comparison with existing algorithms, we build up two sets of images without fog and with synthetic fog, from 66 synthetic and 10 camera scenes.

A. Synthetic fog

We generated 66 synthetic images using the SiVIC T M software which allows to build physically-based road environments, to generate a moving vehicle with a physicallydriven model of dynamic behavior [START_REF] Gruyer | Sivic and rtmaps, interconnected platforms for the conception and the evaluation of driving assistance systems[END_REF], and virtual embedded sensors. From three realistic and complex models (urban, highway and mounts), we produced images from a virtual camera inboard a simulated vehicle moving on a road path. We have computed 66 images without fog from various viewpoints trying to sample as many scene aspects as possible. Each image is of size 640 × 480. A subset of 4 images is shown in the first column of Fig. 4. For each point of view, the true depthmap is also computed, as shown in the second column of Fig. 4. Indeed, the depthmap is required to add fog in the images.

Synthetic images were computed from the image database, using 4 different types of fog:

• Uniform fog: Koschmieder's law ( 1) is applied with a meteorological visibility distance of 80m. • Heterogeneous k fog: as fog is not always homogeneous, we introduced a variability in Koschmieder's law by weighting k differently with respect to the pixel position. These spatial weights are obtained by means of a Perlin's noise between 0 and 1, i.e a noise spatially correlated at different scales (2, 4, 8, up to the size of the image in pixels) [START_REF] Perlin | An image synthesizer[END_REF]. Perlin's noise is obtained as a linear combination over the spatially correlated noise generated at different scales with weight log 2 (s)2 for scale s. The average meteorological visibility distance is set to 80m. • Heterogeneous L s fog: rather than having k heterogeneous and L s constant, we also tested the case where L s is heterogeneous thanks again to Perlin's noise and where k is constant. The meteorological visibility distance corresponding to k is 80m. This method produces fog with a cloudy sky. • Heterogeneous k and L s fog: in order to challenge the algorithms, we also generated a fog based on Koschmieder's law [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF] where k and L s are both heterogeneous thanks to two independent Perlin's noises. The average k is set to enforce a 80m visibility distance. Finally the synthetic image database contains 4 sets of 66 foggy images, i.e a total of 264 foggy images, associated with the 66 original images. Examples of foggy images are displayed in the last four columns of Fig. 4. Notice the differences in aspect between the different types of generated fog. The set of 330 synthetic images and 66 depthmaps used for the ground truth is available 2 for research purpose and in particular to allow other researchers to rate their own visibility enhancement algorithms.

We applied the same process to add the 4 types of fog to 10 camera images. The 10 camera images were selected as the left image of stereo sequences from Karlsruhe dataset 3 . The point in using these images is that the disparity maps obtained using the stereo reconstruction algorithm Libelas [START_REF] Geiger | Efficient large-scale stereo matching[END_REF] are also available from Karlsruhe dataset. These disparity maps being of sufficient quality, we performed a cross/joint bilateral filtering to fill in the remaining holes using the original image as an interpolation guide [START_REF] Yang | Spatial-depth super resolution for range images[END_REF]. The depthmaps are then deduced from the disparity maps using the cameras calibration. Finally the camera image database contains 4 sets of 10 foggy images associated with the 10 original images, i.e a total of 40 foggy images. Examples of original and foggy images with a uniform fog are displayed in the first two columns of Fig. 6.

B. Comparison on synthetic images

We apply each algorithm on the 4 types of fog. The tested algorithms are: multiscale retinex (MSR), adaptive histogram equalization (CLAHE), dark channel prior (DCP), enhancement with free-space segmentation (FSS), enhancement with no-black-pixel constraint (NBPC) and enhancement with noblack-pixel constraint combined with planar scene assumption (NBPC+PA). The results on 11 images with uniform and hetereogeneous fog are presented in Figure 5. Notice the contrast increase for the farther objects: some objects which were barely visible in foggy image appear clearly in enhanced images. A first visual analysis confirms that: first, MSR and CLAHE are not suited for foggy images; second, far away objects are more foggy after DCP than after NBPC+PA; third, vertical objects appear too dark with FSS; fourth, the roadway looks over-corrected by NBPC; fifth, NBPC+PA comes as a nice trade-off.

The same quantitative comparison consists in computing the absolute difference between the image without fog and the image obtained after enhancement. The results, averaged over the 66 images, the number of image pixels and the number of image color components, are shown in Tab. I. In this average, pixels in the sky, in the original image, are discarded not to bias results. Indeed, the sky intensity cannot be restored as constant white when L s is heterogeneous. By computing the average enhancement on the whole image, the proposed metric is global and is not very sensitive to errors around edges. We think that this metric is appropriate for intelligent vehicle applications but it is probably not in other domains such as computational photography.

In order to easily rate the improvement brought by the tested algorithms, the average absolute difference between the foggy image and the image without fog is also computed and shown in row two of the table. One can notice that the proposed algorithms are able, in the best case, to divide the average difference by a factor slightly higher than two.

The multiscale retinex (MSR) is not a visibility enhancement algorithm dedicated to scene with various object depths. The average difference is decreased for the uniform fog and for fog with heterogeneous L s , compared to doing nothing. Interestingly, when k is heterogeneous, the multiscale retinex is worse than doing nothing. This is due to the fact that MSR increases some contrasts corresponding to fog and not to the scene.

Compared to doing nothing, the average difference is always improved when using the adaptive histogram equalization. Nevertheless, it is not a visibility enhancement algorithm based on Koschmieder's law (1) and thus the improvement is small. As an illustration, CLAHE obtains worse results than the multiscale retinex for uniform fog and for fog with heterogeneous L s .

The dark channel prior (DCP) and no-black-pixel constraint (NBPC) algorithms achieve similar performance in average on the whole database. Nevertheless, we notice that with the NBPC algorithm the visibility improvement is slightly superior at long range distances. With uniform fog, enhancement with free-space segmentation (FSS) and with no-black-pixel constraint combined with planar scene assumption (NBPC+PA) gives the best results. A second group of algorithms with similar performance for uniform fog images contains: dark channel prior (DCP), noblack-pixel constraint (NBPC) and multiscale retinex (MSR). These last three algorithms are less efficient than the first two due to the difficulty to restore the correct average intensity on the road part of the image. NBPC+PA brings the performance of NBPC at long range distances without contrast distortions on the road part of the image thanks to the combination with the planar assumption.

For the three types of heterogeneous fog, enhancement with NBPC+PA leads to better results compared to FSS. This can be explained by the fact that the FSS enhancement algorithm relies strongly on the assumption that k and L s are constant over the whole image while NBPC+PA algorithm does not. Indeed, the NBPC+PA algorithm only assumes that k and L s are locally constant in the image and thus, most of the time, it performs better with heterogeneous fog compared to others.

C. Comparison on camera images

We applied the same algorithms as in the previous section on the 10 images of the Karlsruhe database with the 4 types of fog. The results on 5 images with a uniform fog are presented in Figure 6. Notice how the contrast is restored for the farther objects. The quantitative comparison is shown in Tab. II. The results are quite consistent with previous results despite the fact that images are in gray levels and not in colors. The two visibility enhancement algorithms which perform best are NBPC+PA and FSS.

VI. DRIVER ASSISTANCE IN FOG

A. Principle

In [START_REF] Hautière | Mitigation of visibility loss for advanced camera based driver assistances[END_REF], it is shown that applying a visibility enhancement pre-processing improves detection performances for sign and road markings, by restoring the uniformity of the detection processing over the whole image, this assumption not being possible in case of fog. This is a key point of the deployment since it allows to extend the field of application of many camera based ADAS to foggy weather. According to accident surveys, fog accidents are few, but they are more severe and often involve several vehicles. Indeed, dramatic pile-ups often occur due to hard braking implied by reduced visibility in fog. In particular, elderly people are likely to have more accidents in fog than young people due, among other factors, to reduced contrast perception [START_REF] Bian | Aging and the detection of collision events in fog[END_REF]. One ADAS partially dedicated to fog consists in adapting the speed of the vehicle with respect to the prevailing weather conditions as proposed in [START_REF] Gallen | Advisory speed for intelligent speed adaptation in adverse conditions[END_REF] so as to increase the safety margin of the driver. We believe that visibility enhancement algorithms may also be used to develop what we call a Fog Vision Enhancement System (FVES), as it is already the case for night driving assistance (NVES). The principle of a NVES is to display warm objects, like pedestrians, using NIR or FIR cameras and to warn the driver in case of danger [START_REF] Bi | Using Image-Based Metrics to Model Pedestrian Detection Performance With Night-Vision Systems[END_REF]. NVES are shown in [START_REF] Mahlke | Evaluation of Six Night Vision Enhancement Systems : Qualitative and Quantitative Support for Intelligent Image Processing[END_REF] to have positive effects on safety and are thus being introduced into vehicles. In the future, they will benefit from the use of Head-Up Displays (HUD). It is thus a good opportunity to propose a new use of the HUD. Following the principles described in [START_REF] Livingston | A Visibility Improvement System for Low Vision Drivers by Nonlinear Enhancement of Fused Visible and Infrared Video[END_REF], images with restored contrast of the road scene can be shown to drivers on the HUD, see for instance Fig. 7.

B. Safety Benefits

To illustrate the potential safety benefits of a FVES system, we introduce a scenario of accident in fog. A car is stopped on the road in presence of fog. Another car is moving in the direction of the stopped car and the driver performs an emergency braking when he detects, at time t = 0, the brake or fog lights of the stopped car. During an emergency braking, the speed of the car w.r.t the vehicle position x is expressed as the following model, which is sufficient in our case:

s(x) =        s 0 if x ∈ [0, s 0 t R [ s 2 0 -2g(µ + ν)(x -s 0 t R ) if x ∈ [s 0 t R , x s [ 0 if x ∈ [x s , +∞[ (18)
where t R denotes the perception-reaction time of the driver, s 0 the initial speed of the vehicle, g the standard gravity, µ and ν the friction and the slope of the road and

x s = s 0 t R + s 2 0 2g (µ+ν)
the stopping distance of the vehicle. Our point is that the FVES allows the driver to reduce his or her reaction time by acquiring a stronger confidence in the presence of a vehicle ahead. This is explained by Piéron's law [START_REF] Hsu | A generalization of Piéron's law to include background intensity and latency distribution[END_REF] which relates the reaction time to the visual stimulus intensity. This law is expressed as:

t R = t 0 + αI -β (19)
where t 0 , α and β are positive parameters. t 0 is the so-called "irreducible" reaction time, I is the intensity of the visual stimulus and α and β are related with the object setup and with the involved subject. Whatever the setup and whoever the person, the reaction time varies as an hyperbola w.r.t the stimulus intensity, as shown in Fig. 8. In our case, the stimulus intensity I is the intensity of the brake or fog lights of the stopped vehicle. With the FVES, these lights are seen in the HUD with a restored intensity I 1-V using enhancement factor in [START_REF] Hautière | Automatic fog detection and estimation of visibility distance through use of an onboard camera[END_REF], assuming a proper ergonomic design. Also assuming that the enhancement algorithm does its best, we have V = p(1 -exp(-kd)) where d is the distance to the stopped vehicle and k is always the fog extinction coefficient. Consequently, the restored intensity is close to I 1-p for just noticeable brake or fog lights. Therefore, using the FVES, the reaction time of the driver is reduced by ∆t R given by:

∆t R = αI -β 1 -(1 -p) β (20)
When Piéron's law is applied in the visual domain, the exponent parameter β is generally between 0.30 and 0.35. In our scenario, two parameters cannot be set without extra experiments with drivers: the α of Piéron's law and the "irreducible" reaction time t 0 . Indeed, they depend strongly on the driver, in particular on age and attention which can affect the reaction time by a factor of 2. As an illustration, in figure 9, ∆t R in seconds is shown for α = 3, β = 1/3, p ∈ [0.84, 0.98] and I ∈ [50, 400]cd. The smallest gain in reaction time is ∆t R = 0.2s and the largest gain is ∆t R = 0.6s, for these values.

From [START_REF] Geiger | Efficient large-scale stereo matching[END_REF] and [START_REF] Perlin | An image synthesizer[END_REF], the vehicle speed at collision using or not the FVES can be computed. As proposed in [START_REF] Gallen | Advisory speed for intelligent speed adaptation in adverse conditions[END_REF], the speed at collision can be related to the probability of fatal injury. Thus, from the vehicle speeds at collision, the safety benefit of the FVES can be estimated in term of the ratio of probabilities of fatal injury.

To illustrate the proposed scenario, we show in Fig. 10 the speeds and the corresponding probabilities of fatal injury for two reaction times with a difference of 0.2s. The blue curve is when the FVES is used, and the pink one is without the system. To obtain these curves, we set s 0 = 36m.s -1 , µ = 0.7, ν = 0, α = 3, β = 1/3, p = 0.95, I = 100cd and t 0 = 0.5s. Without the system, the collision would occur at a speed of 15.2m.s -1 for d = 120m. With the system, the collision would occur at a reduced speed of 11.0m.s -1 . Even if ∆t R = 0.2s is small, this nevertheless induces a probability of fatal injury divided by more than two. This illustrates how non-linear the relation is between the reaction time and the probability of fatal injury.

By introducing an obstacle detection algorithm in the FVES, a bounding box can be added around detected obstacles. Displaying this bounding box would certainly draw the driver's attention. Therefore, when this obstacle detection is performed early enough, the use of obstacle detection in the FVES may lead to an important reduction of the time where emergency brake is initiated by the driver.

VII. CONCLUSION

Thanks to the new derivation of the local visibility enhancement algorithm [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF] in terms of two constraints on the inference of the atmospheric veil, we introduce a third constraint to take into account the fact that road images contain a large part of planar roadway, assuming a minimum meteorological visibility distance. The obtained visibility enhancement algorithm performs better than the original algorithm on road images as demonstrated on a set of 66 synthetic images and on a set of 10 camera images, where a uniform fog is added following Koschmieder's law. We also generated three different types of heterogeneous fog, a situation never considered previously in our domain. The proposed algorithm also demonstrates its ability to improve visibility in such difficult heterogeneous situations. Our results are successfully compared to state-of-the-art algorithms: free-space segmentation (FSS) [START_REF] Hautière | Mitigation of visibility loss for advanced camera based driver assistances[END_REF], Dark Channel Prior [START_REF] He | Single image haze removal using dark channel prior[END_REF], [START_REF]Guided image filtering[END_REF] and no-black-pixel constraint (NBPC) [START_REF] Tarel | Fast visibility restoration from a single color or gray level image[END_REF].

Finally, potential safety benefits of a Fog Vision Enhancement System, based on the proposed visibility enhancement algorithm, are evaluated on a scenario of accident in fog using Piéron's law.

From this work, several improvements are possible. First, new constraints can be added easily as a function of our prior knowledge about the vehicle environment or coming from other sensors such as a lidar. Second, the metric used to compute the distance between the restored image and the image without fog can be refined by focusing only on the roadway and on the objects on the road, i.e the important objects for intelligent vehicles applications [START_REF] Bossu | Validation and benchmarking for pedestrian video detection based on a sensors simulation platform[END_REF], or by using a model of human vision as proposed in [START_REF] Brémond | Vision models for image quality assessment: One is not enough[END_REF]. Third, the image rendering as well as the visibility enhancement algorithms presented are all based on Koschmieder's law. As explained in [START_REF] Gazzi | Distant contrast measurements through fog and thick haze[END_REF], stray light and shadowing effect can be introduced to improve the fog model at the cost of an increased number of parameters. This opens new perspectives of research.

Fig. 1 .

 1 Fig. 1. Contrast fading on the same scene due to various values of the extinction coefficient k.

Fig. 2 .

 2 Fig.2. From left to right, the original image with fog, the images enhanced using algorithms: multiscale retinex (MSR), contrast-limited adaptive histogram equalization (CLAHE), planar assumption with clipping (PA), dark channel prior (DCP), free-space segmentation (FFS), no-black-pixel constraint (NBPC) and no-black-pixel constraint combined with planar assumption (NBPC+PA).

Fig. 3 .

 3 Fig.3. From left to right, steps of visibility enhancement with the FSS algorithm: original image, fog detection using the vertical inflection point, segmentation of vertical objects (in red) and free-space region (in green), rough estimate of the scene depthmap, obtained visibility enhancement.

Fig. 4 .

 4 Fig. 4. Synthetic road images database. First column is the original synthetic image. Second column is the depth map. Third to sixth columns are the original image with different types of synthetic fog added, from left to right: uniform fog, heterogeneous k fog, heterogeneous L s fog, and heterogeneous k and L s fog.

Fig. 5 .

 5 Fig.5. Visibility enhancement results on synthetic images. From left to right, the original synthetic image without fog, the image with fog, the images enhanced using multiscale retinex, adaptive histogram equalization, dark channel prior, free-space segmentation, no-black-pixel constraint and no-black-pixel constraint combined with planar scene assumption.

Fig. 7 .

 7 Fig. 7. Principle of a Fog Vision Enhancement System (FVES): the restored image is displayed to the driver by means of a HUD which allows the driver to better see potential obstacles and thus decreases his reaction time.

Fig. 8 .

 8 Fig. 8.Illustration of the psychophysics Piéron's law, which links the intensity of a visual stimulus to the reaction time.

Fig. 6 .

 6 Fig.6. Visibility enhancement results on camera images. From left to right, the original camera image without fog, the image with a uniform fog added, the images enhanced using multiscale retinex, adaptive histogram equalization, dark channel prior, free-space segmentation, no-black-pixel constraint and no-black-pixel constraint combined with planar scene assumption.

Fig. 9 .

 9 Fig. 9. Gain in reaction time ∆t R in seconds as a function of the percentage p of enhancement and of the intensity I of the brake or fog lights.

Fig. 10 .

 10 Fig. 10. Safety benefit of a decrease of 0.2s of the perception-reaction time: speed of the vehicle versus the distance to obstacle detection position and then decrease of the probability of fatal injury with respect to distance.

TABLE I AVERAGE

 I ABSOLUTE DIFFERENCE BETWEEN ENHANCED IMAGES AND TARGET IMAGES WITHOUT FOG, FOR THE 6 COMPARED ALGORITHMS, ON THE 4 TYPES OF SYNTHETIC FOG (66 IMAGES FOR EACH TYPE) AND FOR THE WHOLE DATABASE (264 IMAGES) IN THE LAST COLUMN.

TABLE II AVERAGE

 II ABSOLUTE DIFFERENCE BETWEEN ENHANCED IMAGES AND CAMERA IMAGES WITHOUT FOG, FOR THE 6 COMPARED ALGORITHMS, ON THE 4 TYPES OF SYNTHETIC FOG (10 IMAGES FOR EACH TYPE) AND FOR THE WHOLE DATABASE (40 IMAGES) IN THE LAST COLUMN.

perso.lcpc.fr/tarel.jean-philippe/visibility/

www.lcpc.fr/english/products/image-databases/ article/frida-foggy-road-image-database

http://www.rainsoft.de/software/datasets.html

ACKNOWLEDGMENTS

Thanks to Robby T. Tan for providing the first image of Fig. 2, to Kaiming He for providing insight in the implementation and the tuning of the Dark Chanel Prior algorithm, and to Fabrice Neyret for his suggestions about fog synthesis. This work is partly funded by the ANR (French National Research Agency) within the ICADAC project (6866C0210).