Maxime Ferrera

Alexandre Boulch

Julien Moras

Fast Stereo Disparity Maps Refinement By Fusion of Data-Based And Model-Based Estimations

The estimation of disparity maps from stereo pairs has many applications in robotics and autonomous driving. Stereo matching has first been solved using model-based approaches, with real-time considerations for some, but today's most recent works rely on deep convolutional neural networks and mainly focus on accuracy at the expense of computing time. In this paper, we present a new method for disparity maps estimation getting the best of both worlds: the accuracy of data-based methods and the speed of fast model-based ones. The proposed approach fuses prior disparity maps to estimate a refined version. The core of this fusion pipeline is a convolutional neural network that leverages dilated convolutions for fast context aggregation without spatial resolution loss. The resulting architecture is both very effective for the task of refining and fusing prior disparity maps and very light, allowing our fusion pipeline to produce disparity maps at rates up to 125 Hz. We obtain state-of-the-art results in terms of speed and accuracy on the KITTI benchmarks. Code and pre-trained models are available on our github:

Introduction

In mobile robotics and autonomous driving, knowledge about the 3D structure of the environment is required for safe navigation. This 3D information has to be inferred from embedded sensors such as LiDAR, RGB-D cameras or stereo cameras. Stereo setups offer the great advantage of working both indoors and outdoors, in opposition to RGB-D cameras which are limited to indoor environments. Besides, they are cheaper and lighter than LiDAR systems, making them easier to use and embed on weightconstrained robots such as unmanned aerial vehicles (UAV).

As the 3D information is used for decision-making in autonomous navigation, highly accurate 3D measurements should be produced at high-rates. Indeed, safety in such context depends on the accuracy of the 3D measurements and on the frequency of these measurements, both directly impacting the pertinence of the given measurements. These speed constraints have led to the development of several fast algorithms for disparity maps estimation from stereo images and, today, hardware-specific systems output disparity map at cameras' acquisition rate (60 Hz for Mynt-Eye cameras 1 , 90 Hz for Intel RealSense2 ). However, such maps are estimated from model-based algorithms and lack the accuracy of current state-of-the-art methods relying on Convolutional Neural Networks (CNN).

In this paper, we tackle the challenge of producing accurate disparity maps from stereo images while respecting the timing constraints required for safe autonomous navigation. We propose a fusion pipeline that combines disparity maps produced by model-based and data-based stereo matching methods, acting as a fast disparity refinement filter. The proposed pipeline is built on-top of a light CNN that leverages dilated convolutions [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF] (also referred to as atrous convolutions) in order to quickly increase the receptive field of the CNN without giving up on spatial resolution. This allows to aggregate context information in a fast way to correct the disparity estimated at each pixel.

The developed CNN architecture is highly versatile as it can be used for disparity maps prediction from stereo images only, refine prior disparity maps and fuse modelbased and learning-based stereo matching methods outputs, highly increasing the accuracy of the inferred maps. It is also extremely fast as a single-pass through the network takes an average of 2.5 ms on GPU. This CNN being the core of the proposed fusion pipeline, we hence propose a new method able to predict accurate disparity maps with a minimal computation time. We list the paper contributions as follows:

• Dil-Net: a light CNN architecture, based on dilated convolutions, able to combine and refine prior disparity maps and predict disparity maps from stereo images • FD-Fusion, for Fast-Disparity-Fusion: a new pipeline for fusing model-based and data-based disparity maps and produce a refined result • We show that combining model-based and data-based disparity maps largely increase the accuracy of the final results, highlighting the fact that complementary features exist between both kind of approaches The paper is organized as follow: section 2 presents the works related to our approach, section 3 details our pipeline for fast disparity maps refinement and fusion and presents the detail of Dil-Net's architecture. Finally, section 4 exposes the implementation details of our method, an ablation study of the main components of the pipeline and compare the proposed approach to state-of-the-art methods on the KITTI online benchmarks.

Related Work

Model-based Stereo Matching. In the past decades, the stereo matching problem has been widely studied and mainly solved using model-based methods [START_REF] Hamzah | Literature survey on stereo vision disparity map algorithms[END_REF]. These methods mainly rely on semi-local [START_REF] Hirschmuller | Stereo processing by semiglobal matching and mutual information[END_REF][START_REF] Geiger | Efficient large-scale stereo matching[END_REF] or global matching [START_REF] Kolmogorov | Computing visual correspondence with occlusions using graph cuts[END_REF] to infer disparity maps based on stereo images, usually rectified to satisfy epipolar geometry constraints. The great advantage of these methods is that they have been developed on-top of photometric and geometric heuristics, making them quite reliable for most kind of scenes.

Moreover, many have been developed for robotic applications and are hence computationally quite fast, even on CPUs. However, as matching texture-less areas is quite problematic in computer vision, these methods tend to produce false results in such areas. Fine details, such as small objects and objects boundaries, are also difficult to recover with such methods. By using the products from these methods as an input for our refinement approach, we intend to benefit from their robustness and computation speed.

CNN-based Stereo Matching. Since [START_REF] Zbontar | Computing the stereo matching cost with a convolutional neural network[END_REF], the use of convolutional neural networks has been highly investigated to solve stereo matching. First works relied on CNNs to assess the matching of stereo image patches. While giving more accurate results than model-based methods, the matching process was highly ineffective in terms of computational load (67s per image on GPU). The authors of [START_REF] Mayer | A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[END_REF] designed DispNet, the first end-to-end architecture able to estimate disparity maps from stereo pairs of images. Following this seminal work, [START_REF] Kendall | End-to-end learning of geometry and context for deep stereo regression[END_REF] proposed an end-to-end architecture inspired by the classical stereo matching pipeline. Many works followed this trend [START_REF] Chang | Pyramid stereo matching network[END_REF][START_REF] Yang | Segstereo: Exploiting semantic information for disparity estimation[END_REF][START_REF] Song | Edgestereo: A context integrated residual pyramid network for stereo matching[END_REF][START_REF] Nie | Multi-level context ultra-aggregation for stereo matching[END_REF][START_REF] Yin | Hierarchical discrete distribution decomposition for match density estimation[END_REF][START_REF] Guo | Group-wise correlation stereo network[END_REF][START_REF] Du | Amnet: Deep atrous multiscale stereo disparity estimation networks[END_REF] and completely outmatched previous methods in terms of accuracy. However, this gain in accuracy is at the cost of high computation requirement. For real-time applications such as robotics or autonomous driving, disparity maps are required at a higher-rate in order to detect as quickly as possible potential obstacles or dangers and make decision upon it.

A few works take this run-time constraint into account and proposed CNN-based methods able to process stereo images at high-rate. First, [START_REF] Khamis | Stereonet: Guided hierarchical refinement for real-time edge-aware depth prediction[END_REF] proposed StereoNet, the first CNN architecture with a high frame-rate suitable for real-time applications. Following this work, [START_REF] Tonioni | Real-time self-adaptive deep stereo[END_REF] presented a CNN not only able to produce disparity maps at highrate but also able to learn online to refine its predictions. Nonetheless, there is a trade-off between speed and accuracy. With respect to these methods, we intend to lower this trade-off by keeping a very high frame-rate while increasing the accuracy.

Disparity Maps Enhancement. Some works also tackled the challenge of refining disparity maps. The authors of [START_REF] Gidaris | Detect, replace, refine: Deep structured prediction for pixel wise labeling[END_REF], proposed to refine the outputs of a given baseline for dense pixel-wise labelling tasks such as semantic segmentation or disparity maps regression. Focusing on the refinement of disparity maps, [START_REF] Batsos | RecResNet: A recurrent residual cnn architecture for disparity map enhancement[END_REF] designed a deep architecture relying on recurrent neural networks and [START_REF] Cheng | Depth estimation via affinity learned with convolutional spatial propagation network[END_REF][START_REF] Cheng | Learning depth with convolutional spatial propagation network[END_REF] proposed a Figure 2. The FD-Fusion pipeline. The top of the pipeline is composed of a data-based branch and a model-based one, each taking a stereo pair of images as input and estimating a disparity map from it. The next stage is the fusion pipeline, an instance of Dil-Net, which takes as inputs the resulting disparity maps as well as the original stereo images to produce a refined disparity map by fusing the prior ones. spatial propagation network. In opposition to the proposed approach, these methods rely only on data-based disparity maps refinement and do not take into account any timing constraint. Closer to our work, [START_REF] Poggi | Deep stereo fusion: combining multiple disparity hypotheses with deep-learning[END_REF] investigated the fusion of model-based disparity maps produced by different methods. Their method take as input the disparity hypothesis of eight different methods and choose the best candidate for each pixel. In our approach, we also operate a fusion of multiple disparity maps. However, we use both modelbased and data-based outputs to perform the fusion. Furthermore, their method is only meant for fusing disparity maps whereas in our approach a refinement of individual disparity maps is also possible.

The FD-Fusion Pipeline

The proposed pipeline for disparity maps estimation is presented on Figure 2. It is a two-stage pipeline. First, initial disparity maps are computed on two parallel branches: one using a model-based approach, the other using a data based approach, typically a neural network. Second, the generated maps are then stacked with the stereo pairs and given to the fusion module that will merge the input information to generate an enhanced disparity map. The general assumption is that, first both model-based and data-based estimations are sensible to different features of the input image, and second, their failure cases are also different. In this case, the objective is for the fusion module to benefit from both model-based and learned disparity maps while refining the borders according to the original stereo pair.

Fusion module

The fusion module (in green in Figure 2) aims at exploiting the input maps and images to produce a disparity map. In this work, we learn this fusion step using a convolution neural network.

The proposed method is based on the use of dilated convolutions [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF] inside a small CNN in order to refine disparity maps. The idea is to learn how a disparity map estimated by a given method should be corrected to be closer to reality. By using dilated convolutions, we quickly increase the receptive field of the deep network without decreasing too much the input resolutions. This allows the aggregation of context information around every pixel. This information will be used as a clue to predict whether a disparity value is good or not and what value is most likely to be the correct one given the surrounding of the pixel. Hence, by feeding the network with a stereo pair along with the disparity map estimated from it, we want to infer a new disparity map which takes into account the disparity values and the visual context. By concatenating the inputs along one dimension, we avoid the need of several input branches in the network and hence limit the number of parameters and reduce the processing time.

Disparity maps predicted by learning-based methods are going to use very different heuristics than model-based methods. By concatenating the disparity maps estimated from learning-based and model-based methods, along with their respective stereo pairs, the network can then learn how to combine them by analyzing the strength and weakness of each methods given a visual context. By doing so, the network can infer refined disparity maps that surpass the ones that would have been estimated separately (i.e. the one estimated by refining only the disparity map computed by method 1 or 2).

The strength of the proposed network is that it can perform either of these tasks in an extremely fast way. Indeed, a single pass through Dil-Net takes an average of 2.5 ms, making it compatible with real-time requirements such as in robotics.

Name Layer Settings Output Dimension

Input H × W × N in CNN conv 1 k=3, s=1, d=1 H × W × 64 conv 2 k=3, s=2, d=1 1 2 H × 1 2 W × 128 Dilated Part conv 3 k=3, s=1, d=2 1 2 H × 1 2 W × 128 conv 4 k=3, s=1, d=4 1 2 H × 1 2 W × 128 conv 5 k=3, s=1, d=8 1 2 H × 1 2 W × 128 conv 6 k=3, s=1, d=12 1 2 H × 1 2 W × 128 conv 7 k=3, s=1, d=8 1 2 H × 1 2 W × 128 conv 8 k=3, s=1, d=4 1 2 H × 1 2 W × 128 conv 9
k=3, s=1, d=2

Network Description

The network structure, referred to as Dil-Net, is presented in table 1. First the input is preprocessed through two convolutional layers for extracting low-level features. The second layer also reduces the resolution of the input by a factor of 2 to start increasing the receptive field of the network. Then the feature maps are processed through a total of seven convolution layers composed of filters with first increasing and then decreasing dilatation rates. This dilated part acts has a context aggregation pipeline with a bigger and bigger receptive field. The output of this section is then up-sampled and concatenated to its input feature maps by means of a skip-connection. In the next layer, we further add the input of the network to the current features map. The following four layers are then used for merging step by step the feature maps into a 32-D feature maps which is finally used to infer a disparity value for each pixel through a convolution kernel of size 1.

Model-based branch

The model-based branch (in blue in Figure 2) produce disparity image using classical handcrafted stereo matching algorithms. We have experimented three different algorithms: SGBM [START_REF] Hirschmuller | Stereo processing by semiglobal matching and mutual information[END_REF], SGM [START_REF] Hernandez-Juarez | Embedded real-time stereo estimation via semi-global matching on the GPU[END_REF] and ELAS [START_REF] Geiger | Efficient large-scale stereo matching[END_REF].

The first kind of method tested is SGBM / SGM which are semi-global approaches that compute a matching score for each disparity hypotheses (computed on a single pixel for SGM and on a patch for SGBM). This score is aggregated to compute cost that considers the cost of neighboring points (approximating this using a finite set of directions) and disparities. One main issue with SGBM is that the block matching is not possible on the extreme side of the image leaving a band where the disparity is not computed.

The second method considered is ELAS. It is a semilocal approach that performs a two-stage estimation. First, it generates a prior disparity map by computing a robust disparity over a sparse grid and next extend the disparity to the whole image by using a triangulation over this set of support points. Then a bayesian inference scheme is used to fuse this prior and image similarity score and to compute final disparity map.

Data-based branch

The data-based branch (in yellow in Figure 2) uses a learning algorithm to infer a disparity map from a stereo image. The major difference with the model-based approach is that these models are trained using supervised learning, i.e. giving the stereo image and the groundtruth disparity map during training phase. Once the network have been trained, it can be used to infer disparity maps from new stereo images.

Training

The loss used to train the network is a 1 -loss:

r(x) = 1 N N n |x n -y n | (1)
where r(x) is the residual, N is the number of samples, y n is the groundtruth and x n is the output of the network.

This architecture is based on only 16 layers and simply relies on 3×3 convolution kernels (except for the last layer), making it extremely efficient in terms of computation.

The FD-Fusion pipeline is a multi-stage structure. However, the training of the CNN-based parts of the pipeline is not done end-to-end. First the data-based branch is trained for its own task, i.e. stereo-only predictions, and then the fusion module is trained in a second step.

Modularity of the Pipeline

Despite the speed and lightness of Dil-Net, the proposed network is highly versatile in its use and can be used to perform the following tasks:

• Predict disparity maps from stereo images only • Refine single disparity map • Combine multiple disparity maps to produce a refined result Furthermore, the proposed FD-Fusion pipeline is very modular in that it can take any method for the data-based and model-based branches. The number of branches can even be adjusted as needed, by adding more model-based methods and removing the data-based branch for example. The final stage of the pipeline, an instance of Dil-Net, is then responsible of fusing all the prior disparity maps.

In this paper, as we focus on fast disparity map estimations, we stick to the Dil-Net architecture for the data-based branch of the pipeline and to model-based methods relying on local matching. In the following, when we refer to FD-Fusion, we are assuming that a stereo-only instance of Dil-Net is used as the data-based module.

Experiments

We have evaluated the proposed pipeline on three stereo datasets: SceneFlow [START_REF] Mayer | A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[END_REF], KITTI 2012 [START_REF] Geiger | Are we ready for autonomous driving? the KITTI vision benchmark suite[END_REF] and KITTI 2015 [START_REF] Menze | Joint 3d estimation of vehicles and scene flow[END_REF]. We first describe the datasets used and the implementation details of our method. An ablation study is presented on SceneFlow and KITTI 2015 and then follow the evaluation of our best combinations on the official KITTI 2012 and 2015 benchmarks.

Datasets description

The stereo datasets used for the evaluation are: 1. Scene Flow: a large synthetic dataset providing stereo RGB pairs along with their dense disparity map groundtruth. The dataset contains 35454 training and 4370 testing images of size 960 × 540. 2. KITTI 2015: a real-world dataset focusing on autonomous driving for cars. It provides stereo RGB pairs as well as their associated disparity map in the form of a sparse groundtruth extracted from an embedded LiDAR. The training and testing split are of 200 images each with an average size of 1240 × 376. The groundtruth is only available for the training split. In order to validate our models, we have further divided the training split into a training set of 160 images (80%) and a validation set of 40 images (20%). 3. KITTI 2012: another dataset focusing on autonomous driving for cars, similar to KITTI 2015.

Implementation Details

The architecture details of Dil-Net are given in Table 1, that is the size of the convolution layers, the dilatation rate, the activation policy and the resolution of the features map.

The network has been implemented using Pytorch. The different Dil-Net instances tested are the following: a stereo-only model, a refinement model (not using the databased branch) per model-based method and fusion models that combine model-based outputs with the stereo-only Dil-Net predictions. Moreover, we have trained Dil-Net instances taking as input the model-based methods estimations altogether. All these models have been trained using the Adam optimizer (β 1 = 0.9, β 2 = 0.999). The RGB images provided as inputs are normalized by dividing them by 255. The prior disparity maps and the groundtruth maps are converted into approximate z-scores maps (disp = disp-µ σ ) -taking µ and σ as the mean and standard deviation of the datasets disparity maps groundtruth. The inputs fed to the network are resized to 512 × 256. The outputs of the network are also of size 512 × 256 and are hence up-sampled afterwards through bilinear interpolation to match the original image size before computing the error metrics.

All the models are first trained on SceneFlow for 30 epochs, halving the learning rate every 5 epochs on the first 20 epochs and then every 2 epochs. The pre-trained models are then separately fine-tuned on each one of the KITTI set for 500 epochs, halving the learning rate every 50 epochs. On an NVIDIA GTX 1080-Ti, the training of one model on SceneFlow takes an average of 30 hours and around 3 hours on each KITTI sets. The inference time is around 2.5 ms for a one-pass through the network and around 4.5 ms when we combine two instances of this network (when using one instance to predict a disparity map given a stereo pair of images and another instance to fuse this output with the output of a model-based method for example).

When training on SceneFlow, for the models trained for the tasks of disparity map prediction from stereo images and of model-based disparity maps refinement, the starting learning rate is 0.001. For the ones used for model-based and data-based outputs fusion, the starting learning rate is 0.002. When finetuning the different models on the KITTI sets, the starting learning rates are respectively divided by 2 given the ones used on SceneFlow. In all the different trainings, the batch size is set by the maximum size allowed by the GPU (typically between 4 and 6 depending on the models on a GTX 1080-Ti).

On every dataset, the inputs are randomly flipped horizontally and, on the KITTI sets, the input RGB images are further modified by randomly modifying the level of brightness, contrast, hue and saturation.

The tested model-based methods are the following: SGBM [START_REF] Hirschmuller | Stereo processing by semiglobal matching and mutual information[END_REF], ELAS [START_REF] Geiger | Efficient large-scale stereo matching[END_REF] and a CUDA-version of SGM [START_REF] Hernandez-Juarez | Embedded real-time stereo estimation via semi-global matching on the GPU[END_REF] (referred to as Cuda-SGM). For SGBM, we use the OpenCV implementation of the algorithm 3 . For ELAS, we use the python wrapper of the algorithm 4 . For Cuda-SGM, we use the code available on their github 5 . The different methods' parameters used in this work are detailed on our github repository.

Ablation Study

The objective of this section is to motivate each block of the proposed pipeline and to validate the efficiency of Dil-Net. Results are presented in table 2 and table 5.

Methods

No refinement Refinement FD-Fusion

Cuda-SGM [START_REF] Hernandez-Juarez | Embedded real-time stereo estimation via semi-global matching on the GPU[END_REF] 7.56 px 2.09 px 1.57 px ELAS [START_REF] Geiger | Efficient large-scale stereo matching[END_REF] 4.41 px 2.37 px 1.54 px SGBM [START_REF] Hirschmuller | Stereo processing by semiglobal matching and mutual information[END_REF] 3.04 px

Results on SceneFlow

We now compare the performance of each module of the proposed FD-Fusion pipeline. We first evaluate on the SceneFlow test set. Following [START_REF] Kendall | End-to-end learning of geometry and context for deep stereo regression[END_REF][START_REF] Chang | Pyramid stereo matching network[END_REF], we do not take into account pixels with disparity value higher that 192 when computing the loss at training time and when computing the error metrics at testing time. The metric used for evaluation is the End-Point-Error (EPE), that is the mean of the absolute error at each pixel. The results obtained are listed in Table 2. We give the results of the raw model-based methods and of Dil-Net trained with stereo images only in the no refinement column. Then we compare the accuracy obtained after refinement of each model-based methods taken individually and taken altogether (Multi-models raw in the table). We finally compare the final results obtained with our FD-Fusion scheme, using the output of the stereo-only Dil-Net as the data-based module. First, the refinement column results assess the refining abilities of the Dil-Net architecture, increasing the accuracy of all the methods by a large rate (72.5 % with Cuda-SGM, 42 % with ELAS, 37 % with SGBM). We also note that, while the results of stereo-only Dil-Net are finer on this dataset than the ones of the model-based methods, we get more accurate results after refinement of the prior disparity maps. Besides, the network produces even better results when taking the multiple models outputs as inputs, highlighting its efficiency in fusing disparity maps estimated by different means. Secondly, when evaluating the full FD-Fusion pipeline outputs we see an even bigger gain in terms of accuracy (80 % with Cuda-SGM, 62 % with ELAS, 50 % with SGBM and 16 % over the fused and refined multi-models). Given these last results, we demonstrate that data-based produced disparity maps and model-based ones have complementary features that can be efficiently combined to produce more accurate results. We also assess the efficiency of the Dil-Net architecture for this task in that it fully leverages all the given inputs to estimate a refined results (see Figure 1).

Results on KITTI

We further compare the performance of the proposed pipeline on the KITTI split, created by taking 160 images for training and 40 for testing from the training split of the KITTI 2015 official set. The metric used for evaluation on this set is the percentage of pixels misclassified

Methods

No refinement Refinement FD-Fusion Cuda-SGM [START_REF] Hernandez-Juarez | Embedded real-time stereo estimation via semi-global matching on the GPU[END_REF] 8.11 % 4.13 % 3.07 % ELAS [START_REF] Geiger | Efficient large-scale stereo matching[END_REF] 10.28 % 5.84 % 3.96 % SGBM [START_REF] Hirschmuller | Stereo processing by semiglobal matching and mutual information[END_REF] 5. [START_REF] Kolmogorov | Computing visual correspondence with occlusions using graph cuts[END_REF] 3.

Results Analysis

Given all these results, we show that the proposed approach offers a big boost in terms of accuracy with respect to its processing time (2.5 ms for a single-pass through one instance of Dil-Net). We compare the run-time performance of the different methods in Table 4 on their respective hardware implementations. As it can be seen, the main bottleneck of the FD-Fusion pipeline in terms of speed comes from the on-top branches (data-based and modelbased modules). If we take Cuda-SGM as the model-based module and an instance of Dil-Net as the data-based one, we can reach rates of around 125 Hz for the estimation of refined disparity maps (3.5 ms for the prior estimation of a disparity map with Cuda-SGM and 4.5 ms for the inference of a disparity map from stereo images with Dil-Net and the fusion and refinement of both through another instance of Dil-Net). Note that, this calculus is done by taking the different step sequentially. However, one could assume that the model-based and data-based outputs are computed in parallel and then concatenated and fed to the fusion module, in which case we could reach rates of approximately 165 Hz (3.5 ms + 2.5 ms per disparity maps). We also stress out the fact that while we have only used Dil-Net as the data-based module in these experiments to fulfill the run-time constraints of real-time applications, any other method could be used as a replacement. One should note that we do not process images at full resolution to speed-up the inference time here. Hence, adding a very accurate method as the data-based module might require to adapt the fusion module in order to see improvements as the Dil-Net architecture has been designed for low-resolution inputs. These considerations are left for future work.

KITTI Stereo 2015 / 2012

We evaluate our pipeline against state-of-the-art methods on the KITTI Stereo 2012 and 2015 official benchmarks. On the KITTI Stereo 2015 benchmark, the evaluation metric is the percentage of misclassified pixels (error > 3 px) and is given for the foreground (D 1 -fg) and background (D 1 -bg) part of the images as well as for the whole image (D 1 -all). On the KITTI Stereo 2012 benchmark, the evaluation metrics are the percentage of misclassified pixels given different threshold (we give the results for error > 2 and > 3 px here) and the average disparity error. Furthermore, these results are computed for the non-occluded part of the images and the whole images (Noc / All).

The disparity maps submitted to the online platform are the ones resulting from the use of Cuda-SGM as the modelbased module within the FD-Fusion pipeline, i.e. the fastest combination. The obtained results are given in Table 5. We give both the results of precision-oriented and run-timeoriented methods in this table to give a good overview of the current state-of-the-art methods in terms of speed and accuracy on the KITTI Stereo benchmarks. In the category of fast methods we find -ranked by speed -DispNet [START_REF] Mayer | A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[END_REF], MADnet [START_REF] Tonioni | Real-time self-adaptive deep stereo[END_REF] and StereoNet [START_REF] Khamis | Stereonet: Guided hierarchical refinement for real-time edge-aware depth prediction[END_REF]. Compared to these methods, our proposed pipeline is the fastest, being almost two times faster than StereoNet. Besides, we also obtain the best performance in terms of accuracy, surpassing all the three methods on every metric but the one considering the foreground part of the image in the KITTI 2015 benchmark, on which only DispNet performs better.

Conclusion

In this paper we have presented FD-Fusion, a new pipeline for fast disparity maps refinement through the fusion of model-based and data-based outputs. The core of this pipeline is Dil-Net, a CNN based on dilated convolutions that is able to predict, refine and fuse disparity maps to produce accurate results at very high rates. The full pipeline can be used to produce state-of-the-art disparity maps at rates up to 125 Hz. We have shown that one of the strength of the proposed pipeline lies in the fusion of data-based and model-based methods which exhibits complementary features. Future works will include investigating the use of different data-based methods at the top of the pipeline. We will also investigate the potential effectiveness of the proposed pipeline for other dense regression tasks such as monocular depth estimation or dense optical flow.

Figure 1 .

 1 Figure 1. FD-Fusion pipeline results on a sample image from SceneFlow. Top row: left stereo image (left), groundtruth disparity map (middle), SGBM estimate (right). Middle rows: zoom-in on the disparity maps. Last row: Dil-Net stereo-only estimate (left), FD-Fusion output (middle), SGBM refined with Dil-Net (right).

Figure 3 .

 3 Figure 3. Result samples from our KITTI 2015 validation set. (a) Left stereo image, (b) groundtruth disparity, outputs of (c) stereo-only Dil-Net, (d) Cuda-SGM, (e) Cuda-SGM refined , (f) FD-Fusion with Cuda-SGM, (g) ELAS, (h) ELAS refined, (i) FD-Fusion with ELAS, (j) SGBM, (k) SGBM refined, (l) FD-Fusion with SGBM, (m) the multi-models refined, (n) FD-Fusion with the multi-models.

Table 2 .

 2 Absolute mean error (End-Point-Error) on the SceneFlow test set.

			1.92 px	1.51 px
	Multi-models	-	1.63 px	1.37 px
	Dil-Net	2.83 px	-	-

Table 3 .

 3 Percentage of pixels misclassified (error > 3 px) on the KITTI 2015 validation set (40 images).(error > 3 px). The evaluation of each method is done after finetuning the different instances of Dil-Net pre-trained on SceneFlow. The obtained results are listed in Table3. First, we can note that on this dataset the stereo-only Dil-Net is outperformed by all the model-based methods but Cuda-SGM. Second, we see that the pattern of the results follows the one obtained on the SceneFlow dataset. The refinement of each model-based outputs largely surpasses the prior estimations (49 % with Cuda-SGM, 43 % with ELAS, 11 % with SGBM). Once again, the results obtained with the model-based estimations taken altogether are better than their indivual refinement. Furthermore, we see that the FD-Fusion scheme also produces finer results (62 % with Cuda-SGM, 61 % with ELAS, 38 % with SGBM and 21 % over the fused and refined multi-models). Given the poorer accuracy of the stereo-only Dil-Net estimations on this dataset, we highlight the fact that data-based on model-based outputs contain very different features that can be reliably extracted and combined by CNN architectures. Qualitative results are given in Figure

		%	4.58 %	3.19 %
	Multi-models	-	3.84 %	3.03 %
	Dil-Net	8.69 %	-	-

Table 4 .

 4 Hardware Intel Xeon 3.0GHz Intel Xeon 3.0GHz GTX-1070 / GTX 1080-Ti GTX 1070 / GTX 1080-Ti GTX 1070 / GTX 1080-Ti Run-time analysis of the different stereo matching methods given the tested harware. Dil-Net two-passes denotes the total processing time when loading two instances of Dil-Net and feeding the second one with the ouput of the first one.

	Methods	SGBM	ELAS		Cuda-SGM	Dil-Net (single-pass)	Dil-Net (two-passes)
	Runtime	310 ms	210 ms		7.5 / 3.5 ms	2.5 / 2.5 ms	4.5 / 4.5 ms
			KITTI Stereo 2015	KITTI Stereo 2012 (< 2 px / < 3 px)
		Method	D1-bg	D1-fg	D1-all	Out-Noc	Out-All	Avg-Noc Avg-All Runtime
	Precision-oriented methods
		M2S CSPN [4]	1.51 % 2.88 % 1.74 % 1.79 / 1.19 % 2.27 / 1.53 %	0.4 px	0.5 px	500 ms
		AMNet [5]	1.53 % 3.43 % 1.84 %	2.12 / 1.32 % 2.71 / 1.73 %	0.5 px	0.5 px	900 ms
		MS CSPN [4]	1.56 % 3.78 % 1.93 %	-	-	-	-	500 ms
		GANet-15 [27]	1.55 % 3.82 % 1.93 %	2.18 / 1.36 % 2.79 / 1.80 %	0.5 px	0.5 px	360 ms
		HD 3 -Stereo [24]	1.70 % 3.63 % 2.02 %	2.00 / 1.40 % 2.56 / 1.80 %	0.5 px	0.5 px	140 ms
		EdgeStereo-V2 [21]	1.84 % 3.30 % 2.08 %	2.32 / 1.46 % 2.88 / 1.83 %	0.4 px	0.5 px	320 ms
		ECMUA [18]	1.66 % 4.27 % 2.09 %	2.02 / 1.26 % 2.56 / 1.64 %	0.4 px	0.5 px	900 ms
		GwcNet-g [9]	1.74 % 3.93 % 2.11 %	2.16 / 1.32 % 2.71 / 1.70 %	0.5 px	0.5 px	320 ms
		EdgeStereo [20]	1.87 % 3.61 % 2.16 %	-	-	-	-	700 ms
		SegStereo [23]	1.88 % 4.07 % 2.25 %	2.66 / 1.68 % 3.19 / 2.03 %	0.5 px	0.6 px	600 ms
		PSMNet [2]	1.86 % 4.62 % 2.32 %	2.44 / 1.49 % 3.01 / 1.89 %	0.5 px	0.6 px	410 ms
	Running time oriented methods
		DispNetC [16]	4.32 % 4.41 % 4.34 %	7.38 / 4.11 % 8.11 / 4.65 %	0.9 px	1.0 px	60 ms
		MADnet [22]	3.75 % 9.20 % 4.66 %	-	-	-	-	20 ms
		StereoNet [14]	4.30 % 7.45 % 4.83 %	4.91 / -%	6.02 / -%	0.8 px	0.9 px	15 ms
	FD-Fusion with Cuda-SGM 3.22 % 7.44 % 3.92 %	4.8 / 3.16 % 5.73 / 3.85 %	0.7 px	0.8 px	8 ms

Table 5 .

 5 KITTI Stereo 2015 -2012 -Official benchmark results.

https://www.mynteye.com/products/ mynt-eye-stereo-camera

https://www.intelrealsense.com/

https://docs.opencv.org/3.4.2/d2/d85/classcv_ 1_1StereoSGBM.html

https://github.com/jlowenz/pyELAS

https://github.com/dhernandez0/sgm

https://github.com/ ferreram/


Jean-Claude Bermond

Anne Germa

Dominique Sotteau

est destinée au dépôt


I propose a reading of Iris Murdoch's A Word Child and show how her novel resonates with important themes in Ludwig Wittgenstein's Philosophical Investigations, as read by Stanley Cavell, such as the idealization of language. I highlight the ethical significance of such an idealization.

INTRODUCTION

I would like to propose here a few thoughts about the novel A Word Child, published by Iris [START_REF] Murdoch | A Word Child[END_REF].

A Word Child is a first-person narrative about Hilary Burde, a poor orphaned child, who grew up lonely and unloved. He was subjected to violence, and was becoming violent himself, when a schoolteacher spotted him as a gifted child, and made him discover languages. Hilary learned how to write his own language, English, then proceeded to Latin, French and Italian. Discovering the order of grammar gave him his first vision of order and beauty: this vision allowed him to rise above the violence and disorder around him. Eventually his love for grammar led him up to a famous university, where he became a Fellow. (( I discovered words)), writes Hilary, (( and words were my salvation)).

But A Word Child is not a fairy tale, and after a catastrophic love affair which ended in a scandal, Hilary resigned from his University position and took a job indifferent to his talents: at the beginning of the novel, the reader finds Hilary stuck in this job, and presenting himself as a failure. It seems that words did not save him afterall, or that he could have been saved but for something that was still lacking. Why salvation did not happen and what was lacking in Hilary's first love for words and their order, is a problem the reader has to solve while reading the novel.

A few years before the publication of A Word Child, a very similar problem was given a philosophical expression by Iris Murdoch, in a conference she delivered in New York, on this very subject of salvation by words. This latter expression is the title of the conference, then reprinted in Existentialists and Mystics. The conference has a political tone: it underlines how tyranny degrades language, and it states that, with no doubt, the art which is (( the most practically important for our survival and our salvation)) is literature. Murdoch writes that (( ( . . . ) words are the ultimate texture and stuff of our moral being, since they are the most refined and delicate and detailed ( . . . ) of the symbolisms whereby we express ourselves into existence)) [7, p. 241].

The novel, A Word Child, could be seen as a literary exploration of this idea of salvation by words. This novel is one way to explore what it could mean. Let us note that Murdoch uses two different expressions in the words I just quoted from her article: (( survival)) and (( salvation )). One belongs to the empiricist way in philosophy about which Murdoch declares that it is not sufficient for ethics, but she still considers it contains something valuable enough to be conserved; the other word belongs to the christian and theological vocabulary, but it is stripped of the social background that gave it its sense 1 . So, in the profane context of modern life, the use of the word (( salvation)) has to be explored and rebuilt: the novel can be read as such an exploration, where the fictitious character of Hilary Burde provides a secularized setting for the word. And, interestingly, a setting where he deems himself not saved. At least at the beginning of the novel.

Since a kind of salvation is seen as essential to the work of literature by Murdoch, it would be interesting to find other works in modern literature that could be described in the terms of a salvation by words. Each of these works would depict different kinds of relations to words. It would be interesting too to see which philosophical works would answer to this description. I would like to show here how A Word Child resonates with ordinary language philosophy works, especially with Wittgenstein's philosophical works. We could see Wittgenstein as another word child, a word child of a certain kind, who may have been an inspiration for the character of Hilary Burde. But I will not make any historical guess; I will propose rather to bring together the novel and Wittgenstein's Philosophical Investigations, with the idea that each work sheds light on the other.

In the first part of my talk, I will present some elements of Hilary Burde's story: how words were special for him, how he admired the order of language, and what could have been missing here.

In the second part, I will come to Wittgenstein and to the path he followed from the Tractatus to the Philosophical Investigations. In the Philosophical Investigations, 1 Cf. Camille Fort, (( Le Dieu caché d'Iris Murdoch)) [START_REF] Fort | Le dieu caché d'Iris Murdoch))[END_REF]: the article analyses the tensions produced in Murdoch's work by her desire to retain christian and theological vocabulary in her novels and philosophical texts (in particular to retain the word (( God )) itself), in spite of her agnosticism.

Wittgenstein says of the Tractatus that it is a work which sublimes language: it explains its apparent disorder by a logical order which is supposed to be underneath the apparent form of proposition. Logical analysis was supposed to unveil this hidden order. But the distinction between the apparent form and the real form of language is abandoned in the Philosophical Investigations as mythological: Wittgenstein asks the philosopher to come back to the rough ground of language, to real words, as they are used in their actual contexts, in an innumerable variety of ways. In the Philosophical Investigations, Wittgenstein tells his own story of how he came back from the temptation to see language as something pure and orderly to a sober examination of the innumerable, unclassifiable, different language-games in which we talk.

This would show us a way to answer my question about what was missing in Hilary's first love for words, and why he looked for salvation in a place where he could not find it: our words are not logical; our language-games are innumerable; our language is first of all ordinary, evolving in a relative disorder and a relative mess. Language is like an old medieval town, it is nothing similar to neatly designed suburbs. These do exist, but they are not to be confused with the whole town. Hilary's first love consisted in this confusion, as the Tractatus vision of language was. Hilary's first love was more precisely an illusion, necessary for his survival, but to survive is not to live, and to live he had to give up his old vision.

The salvation by words Iris Murdoch refers to in her article is one which accepts the words in their relative disorder, and in their endless differences.

WHO IS HILARY BURDE ?

Let us come back to Hilary Burde and to his attempt to save himself through the study of words and their grammar.

Hilary Burde is an anti-hero, an underground man, as he describes himself. After his mother's death, he lived in an orphanage, separated from his beloved sister Crystal. He did not know who his father was: I was informed, before I knew what the word meant, that my mother was a 'tart'. [8, p. 17] This is the harsh way to understand that our moral being is made of words. The sentence sums up in a few words the small child's status in language: before mastering their own language, they are the impotent objects of gossip. Children are talked about in every way before being able to say anything for themselves: they are the object of description, and become authors of descriptions concerning themselves much later, if at all. Salvation was needed here, and it seems that it was not only salvation by words that was needed, but salvation from words. And the first because of the second: violent words must be fought with other words. Hilary found these other words at school, under the protection of Mr Osmand.

Mr Osmand, a schoolteacher, was the first adult to remark Hilary's talents, and the first adult to care about him:

The realization that people had simply given up trying to teach me anything enlightened me at last, more than the lectures from magistrates, about how utterly shipwrecked I was; and increased my anger and my sense of injustice. For with the dawning despair came also the tormenting idea that in spite of everything I was clever, I had a mind though I had never wanted to use it. I could learn things, only now it was too late and nobody would let me. Mr Osmand looked at me quietly. He had grey eyes. He gave me his full attention. [8, p. 21] Hilary describes their relation as nourishing and creative. The child was saved by his master's attention and their common and shared attention for words: he was Mr Osmand's spiritual son. This is expressed in his being a (( word child)). I learnt French. I learnt Latin. Mr Osmand promised me Greek. An ability to write fluent correct Latin prose began to offer me an escape from (perhaps literally) the prison house, began to show me vistas headier and more glorious than any I had ever before known how to dream of. In the beginning was the word. Amo, amas, amat was my open sesame, 'Learn these verbs by Friday' the essence of my education; perhaps it was mutatis mutandis the essence of any education. ( . . . ) I discovered words and words were my salvation. I was not, except in some very broken-down sense of that ambiguous term, a love child. I was a word child. [8, p. 21] Words gave him a vision of something higher. More precisely, Hilary's admiration went to grammar: words appeared to him as a certain order, as belonging to grammatical systems. So one aspect only of words were given to him: order ! And this order appeared to him as worthy of his respect.

Moreover this order could be consistently obeyed: it was reliable. And this reliability offered him something he never had before. Something which allowed him to counteract the effect of violence, to resist violence.

Violence is a kind of magic, the sense that the world will always yield. When I understood grammatical structure I understood something which I respected and which did not yield. The exhilaration of this discovery, though it did not 'cure' me, informed my studies and cast on them a light which was not purely academic. [8, p. 22] Murdoch uses here the vocabulary she uses to describe moral vision in the Sovereignty of Good. An idea of the Good was given to Hilary through the study of grammar. Reality was given to him through the experience of obedience to the master and, at the same time, through the experience of obedience to their common object of attention: words.

But Hilary the narrator is conscious that something was missing in this relation of admiration and respect, precisely because what he admired and respected was the order that was missing from his own life. He did not use the languages that he learnt. After playing on words with the expression (( a word child)), the narrator crafts another expression, which signifies this partial, disengaged relation to words: a word-watcher. He admires words, but at a distance.

I loved words, but I was not a word-user, rather a word-watcher, in the way that some people are bird-watchers. I loved languages but I knew by now that I would never speak the languages that I read. I was one for whom the spoken and the written words are themselves different languages. [8, p. 28] In his study of languages, Hilary was not interested in people, and their linguistic interactions. He was not interested in the fact that their lives were made of words, but he was interested in words abstracted from theses lives, and built into systems.

WITTGENSTEIN AS A WORD-WATCHER

I propose to see Wittgenstein as another word child, one who similarly came back from a failed salvation by words in the Tractatus to the recognition of what salvation could really be in the Philosophical Investigations: one can see Wittgenstein's philosophy as going from word-watching to realism.

In the Philosophical Investigations, Wittgenstein reassesses his first work by showing how it was under the illusion that language was a perfect order, behind the appearences. He recalls how language appeared to him as something pure and superior, as having the purity of crystal -Hilary's terms are of the same kind. We find in the salvation sought by Hilary the same temptation of subliming language that Wittgenstein sees at work in his former treatise on logic: 94. (( A proposition is a queer thing!)) Here we have in germ the subliming of our whole account of logic. The tendency to assume a pure intermediary between the propositional signs and the facts. Or even to purify, to sublime, the signs themselves.

-For our forms of expression prevent us in all sorts of ways from seeing that nothing out of the ordinary is involved, by sending us in pursuit of chimeras. [9, §94] Ordinariness was precisely not what Hilary needed to escape the ugliness and violence of the orphanage; he would return to ordinariness, but at the end of a long journey, which the novel recounts. In his school years, and his studies afterwards, his salvation depended on the greatness of the order he was discovering. The price to pay was that this sublimity left real words, in their real use, outside his attention. The use of words was not a subject of reflection. Words were treated as magical creatures. As such, they were bound to lose the reality that first allowed Hilary to escape from violence and disorder: because it was sublimed, their order was bound to lose its reality, and its appeal.

In the Philosophical Investigations, Wittgenstein compares words with crystals, when he describes his first relation to words in the Tractatus: 97. Thought is surrounded by a halo. -Its essence, logic, presents an order, in fact the a priori order of the world: that is, the order of possibilities, which must be common to both world and thought. But this order, it seems, must be utterly simple. It is prior to all experience, must run through all experience; no empirical cloudiness or uncertainty can be allowed to affect it -It must rather be of the purest crystal. ( . . . ) This order is a super-order between -so to speak -super-concepts.

Whereas, of course, if the words (( language )), (( experience)), (( world )), have a use, it must be as humble a one as that of the words (( table)), (( lamp)), (( door)). [9, § 97] As Wittgenstein describes the conflict created by this first relation, a conflict between the requirement of crystalline purity and actual language, he calls the philosopher back to the ordinary uses of words:

107. The more narrowly we examine actual language, the sharper becomes the conflict between it and our requirement. (For the crystalline purity of logic was, of course, not a result of investigation: it was a requirement.) The conflict becomes intolerable; the requirement is now in danger of becoming empty. -We have got on to slippery ice where there is no friction and so in a certain sense the conditions are ideal, but also, just because of that, we are unable to walk. We want to walk: so we need friction. Back to the rough ground! [9, §107] For Hilary, it was vital and life-saving that grammar should be a reliable order. But what saved him from prison (or worse) was at the end an illusion. The novel depicts the difficulty of the way back to reality. Now Crystal is Hilary's sister's name. She completely exists for him, and him for her. Her name shines for him. But she is not an illusion, she is plainly real. And Murdoch depicts her as the apparent contrary of a crystal; she gives her a heavy plain body. Crystal embodies the ordinariness Hilary has to come back to, another image of love:

Let me try to describe Crystal. She cannot be said to be beautiful. She was short and dumpy, she had no perceptible waist. She had pretty small well-worn capable hands ( . . . ) She had a large mouth with a prominent moist lower lip, very mobile. Rather bad teeth. ( . . . ) Her eyes were hazel, of the kind which are pure golden without a hint of green, but they were usually hidden behind thick round spectacles which made them look like gleaming stones. None of this really describes Crystal however. How is it possible to describe someone to whom you are oned in love ? [8, p. 14] In §111 Wittgenstein underlines the depth of the word-watcher's illusion. He suggests this depth to his reader, but passes then to another question. This is the kind of place where a novelist could begin a story, and the kind of place where Murdoch would have considered perhaps that philosophical writing is not powerful or expressive enough, and that it should give way to literature: 110. (( Language (or thought) is something unique)) -this proves to be a superstition (not a mistake!), itself produced by grammatical illusions.

And now the impressiveness retreats to these illusions, to the problems.

111. The problems arising through a misinterpretation of our forms of language have the character of depth. They are deep disquietudes; their roots are as deep in us as the forms of our language and their significance is as great as the importance of our language. -Let us ask ourselves: why do we feel a grammatical joke to be deep ? (And that is what the depth of philosophy is.) [9, §110-111] The ideas of a word child and of a word-watcher are such grammatical jokes, which have philosophical depth.

THE ORDINARY LANGUAGE MAN

I come now to some elements in Murdoch's philosophy that may seem to go contrary to this reading of Wittgenstein as a former word child, trying to find his salvation in words, the real words of ordinary language, where crystals should be searched for and found.

In her 1959 article, (( The Sublime and the Beautiful Revisited)), Iris Murdoch describes two strands of contemporary philosophy she connects with (( symptoms of decline in contemporary literature)) [7, p. 267]: existentialism and linguistic empiricism. She describes the latter as associated with (( the tradition of Moore and Wittgenstein)) [7, p. 267]. One of Murdoch's objection to this inheritance of Wittgenstein's philosophy is that its picture of the moral agent, being conventional and behaviouristic in spirit, is incapable of approaching the complexity of moral life.

The (( Ordinary Language Man )), as she calls this construal of the moral agent, is (( a being subject to rules )), surrounded by (( the network of ordinary language )), understood as (( the network of moral conceptual activity at its more common and universally accepted level)) [7, p. 268]. With this picture we turn (( away from the chaos of empirical inwardness )) to (( the clarity of overt action)) [7, p. 268]. For the Ordinary Language Man, (( [even] the presence of others is felt, if at all, simply as the presence of rational critics)) [7, p. 268]. As a consequence, concludes Murdoch, this version of the moral agent is incapable of understanding virtue, which is essentially a concern not with actions or rules or reason, but with (( really apprehending that other people exist )) [7, p. 284].

Murdoch sees this appeal to convention as a surrender when faced with the difficulty of understanding how each of us makes place for the other in our life. The orderliness and rationality of convention plays the role of an intellectual shield against the disruptive character and turmoil of human encounters. This disruption and turmoil are at the center of both Murdoch's philosophical works and novels.

In The Sublime and the Beautiful revisited, she describes convention as the enemy of love and understanding:

One might say that whereas the Ordinary Language Man represents the surrender to convention, the Totalitarian Man of Sartre represents the surrender to neurosis: convention and neurosis, the two enemies of understanding, one might say the enemies of love; and how difficult it is in the modern world to escape from one without invoking the help of the other. [7, p. 268] How does this link to the word child? Hilary Burde shares one thing with this creature of convention: an idealization of language, an irrealistic recoil from its actual variety and messiness. There are different ways to fantasize the order of ordinary language: the Ordinary Language Man is one of them. It is the fantasy that human beings could have convened to build something common to resolve the most important disagreements in advance -as if we could have come to some general agreement beforehand to answer Socrates' question to Eutyphro: (( But what kind of disagreement, my friend, causes hatred and anger ? )). (Stanley Cavell puts this question as an epigraph to his essay on the perlocutory aspect of language, (( Passionate and Performative Utterances)) [3, p. 155]: the perlocutory is notoriously disorderly and messy !) Now, this Ordinary Language Man is not Wittgenstein's language-user. As Cavell showed in the Claim of Reason [START_REF] Cavell | The Claim of Reason. Wittgenstein, Scepticism, Morality, and Tragedy[END_REF], the image of human beings as creatures of convention that Murdoch finds in some works inspired by Wittgenstein is precisely what Wittgenstein (and Austin) tried in fact to avoid, while feeling at the same time the temptation to picture human beings in such a way.

The linguistic empiricism Murdoch opposes is linked with an early reception of the Philosophical Investigations, one that Cavell opposes too, stating that it is a misundertanding of Wittgenstein's vision of language [START_REF] Cavell | The Claim of Reason. Wittgenstein, Scepticism, Morality, and Tragedy[END_REF]. Interestingly, Murdoch's criticism of conventionalism is consonant with the interpretation of Wittgenstein defended by Cavell. The critique Murdoch opposes to the first Ordinary Language Man leads her to the moral themes of this second reception, that developed well after linguistic empiricism (roughly, from the 1970s on).

Thus, in criticising the (( Ordinary Language Man)), Murdoch comes closer in fact to Wittgensteinian conceptions 2 . And A Word Child is a novel where Murdoch comes closer to Wittgensteinian themes.

CONCLUSION

As a conclusion, I would like to point to another theme in the novel which can be linked fruitfully to Wittgenstein's philosophical method.

All along the Word Child, another novel is woven into its narrative: Peter Pan. In his introduction to the Vintage edition of A Word Child, Ray Monk, who is the author of an important biography of Wittgenstein [START_REF] Monk | The Duty of Genius[END_REF], proposes an interpretation of the role of Peter Pan in the novel. Peter Pan is the master of make-believe, and his main tool is language. Let me quote, for example, the passage where Peter Pan and the Lost Boys are playing doctor. (( Please, sir)), said Peter, going to Slightly, one of the Lost Boys, (( Are you a doctor?)):

The difference between [Peter Pan] and the other boys at such a time was that they knew it was make-believe, while to him make-believe and true were exactly the same thing. This sometimes troubled them, as when they had to make-believe they had had their dinners. [1, p. 79] Monk shows how the risk of a salvation by words is that the word-watcher is entangled in his or her own linguistic imagination. Words can be veils that create illusions, as well as tools to pursue the truth. What was missing in Latin was the language-games in which Latin words were used: but that was all the point for young Hilary.

Monk does not refer to Wittgenstein in his introduction. But it is an interesting result of Murdoch's novel that Peter Pan and Wittgenstein's Philosophical Investigations, two works that seem to belong to very different worlds, do illuminate each other in the end.

2 See

 2 Murdoch's (( Wittgenstein and Inner Life )) in Metaphysics as a Guide to Morals[START_REF] Murdoch | Metaphysics as a Guide to Morals[END_REF]: the last pages of this article acknowledge the presence of a voice in Wittgenstein which is at variance with the voice of convention. Murdoch sees it particularly present in the second part of Philosophical Investigations.


Frederique Crete

Thierry Dolmiere

Patricia Ladret

Marina Nicolas

The Blur Effect: Perception and Estimation with a New No-Reference Perceptual Blur Metric

Keywords: Blur, Perception, No-Reference, Objective metric, Subjective test

To achieve the best image quality, noise and artifacts are generally removed at the cost of a loss of details generating the blur effect. To control and quantify the emergence of the blur effect, blur metrics have already been proposed in the literature. By associating the blur effect with the edge spreading, these metrics are sensitive not only to the threshold choice to classify the edge, but also to the presence of noise which can mislead the edge detection. Based on the observation that we have difficulties to perceive differences between a blurred image and the same reblurred image, we propose a new approach which is not based on transient characteristics but on the discrimination between different levels of blur perceptible on the same picture. Using subjective tests and psychophysics functions, we validate our blur perception theory for a set of pictures which are naturally unsharp or more or less blurred through one or two-dimensional low-pass filters. Those tests show the robustness and the ability of the metric to evaluate not only the blur introduced by a restoration processing but also focal blur or motion blur. Requiring no reference and a low cost implementation, this new perceptual blur metric is applicable in a large domain from a simple metric to a means to fine-tune artifacts corrections.

INTRODUCTION

New technologies of displays, panels, cameras or mobiles aim at obtaining the best image quality. This phenomenon plays an important role in the evolution of the image correction algorithms to remove noise or compression artifacts. Unfortunately, most of the corrections using low-pass filters not only smooth artifacts, but also lose a part of the high frequency content and generate a blur effect. Others methods such as rescaling algorithms can also generate this artifact.

To control and quantify the emergence of the blur effect, blur metrics have been proposed in the literature. One of the first blur estimation [START_REF] Boult | Local Blur Estimation and Super-Resolution[END_REF] was established in 1997 to improve the super resolution algorithms. Then, the blur estimation was used as a quality metric. In 1999, Marichal et al [START_REF] Marichal | Blur Determination in the Compressed Domain using DCT information[END_REF] estimated the blur by using an histogram of the DCT coefficients, but this method was limited to the frames in the compressed domain. Then, several metrics based on an edge detection were proposed. Among them, Marziliano et al [START_REF] Marziliano | A No-Reference Perceptual Blur Metric[END_REF] took into account the edge spreading using the inflection points delimiting an edge. Caviedes et al [START_REF] Caviedes | No-reference sharpness metric based on local edge kurtosis[END_REF] made an evaluation of the sharpness (the inverse of the blur) by computing the Kurtosis of the DCT on blocks containing edges. Ong et al [START_REF] Ong | A no-reference quality metric for measuring image blur[END_REF] characterized the amount of blur by computing the average extent of the edges. Based on a given edge detector, these metrics are sensitive not only to the threshold choice to classify the edge, but also to the presence of noise which can mislead the edge detection. Hu et al [START_REF] Hu | Low Cost Robust Blur Estimator[END_REF] proposed a Gaussian blur estimation algorithm which is not based on an edge detection. Their method model the focal blur with the normalized Gaussian function and is well adapted for the out of focus blur detection in images or videos.

To be independent from any edge detector and to be able to predict any type of blur annoyance, we propose a new approach which is not based on transient characteristics but on the discrimination between different levels of blur perceptible on the same picture. In fact, we observe that we have difficulties to perceive differences between a blurred image and the same re-blurred image. Consequently, we use this phenomenon to estimate the blur annoyance.

The major interest of a no-reference metric is to be able to replace subjective tests requiring time and means. Nevertheless, these tests are necessary to validate the metric by correlating them with the human judgment. Following the ITU recommendations [START_REF]Methodology for the subjective assessment of the quality of television pictures[END_REF] to make subjective tests, we take also advantage of this step to analyse the behaviour of the viewers and their abilities to evaluate the blur effect on a picture.

In this paper, we present the description and the validation of our no-reference perceptual blur metric. Section 2 presents the method used to evaluate without reference the blur perception, section 3 is the description of the experimental protocol used to establish subjective tests, and section 4 proposes an analysis of the behaviour of the viewers and shows the correlation between the objective measurements of the quality metric and the subjective tests. Then, section 5 lists the possible applications of the metric and section 6 provides the conclusion of the paper

THE NO-REFERENCE BLUR ESTIMATION 2.1 The blur discrimination of the human perception

Because the blur effect is caused by a loss of the high frequency content, it can be reproduced with a low-pass filter. By studying naturally blurred pictures and using different low-pass filters to cover the largest possible range of blur levels, we observe that we have difficulties to perceive differences between a blurred image and the same re-blurred image. In Figure 1, we present from left to right the original sharp picture, the original picture blurred with a low-pass filter and the blurred picture re-blurred with the same low-pass filter. We observe a high difference in term of loss of details between the first and the second picture and a slight difference between the second and the third picture. Actually, the more we blur, the more the neighboring pixels converge to the same gray level. If we blur a sharp picture, gray levels of neighboring pixels will change with a major variation. On the contrary, if we blur an already blurred picture, gray levels of neighboring pixels will still change, but only to a weak extent. We can explain this phenomenon by the fact that the second blur effect reduces a difference between pixels that has already been reduced by the first blurring effect. To illustrate this phenomenon, the Figure 2 represents the three histograms of the absolute difference between neighbouring pixels for a same picture sharp, blurred and re-blurred. We notice that the high differences significantly decrease after the first blurring step and slightly decrease after the second blurring step. On the contrary, the number of slight variations increases with the two blurring steps.

The key idea of our blur estimation principle is to blur the initial image and to analyse the behaviour of the neighbouring pixels variation. The blurring step should be done with a strong low-pass filter in order to be sure to compare the initial image with an image which seems blurred for the human perception. The choice of the type of filter is not exhaustive if it is a strong filter.

Figure 2: histograms of the variations (the absolute differences) between neighboring pixels for a sharp, blurred and reblurred picture

General observations on the blur perception

In addition to this previous principle, we note others phenomena on the blur perception which lead us to a specific principle of the blur estimation.

The first observation depends on the local content of the picture. If we look at a picture containing a small blurred part over a homogeneous area, we perceive it as an non-sharp picture even if only a minor part of this picture is blurred (Figure 3). For this reason, in the analysis of the neighboring pixels variation, we only take into account the pixels which have changed after the blurring step. Moreover, by blurring a sharp edge on a flat area, we observe that we create slight variations between neighboring pixels around the edge which were not existing on the sharp picture. To avoid to take into account these variations, we consider only the neighboring pixels variations which have decreased after the blurring step. This method takes advantage of the possibility to access to specific local variations representatives of the blur effect.

Finally, by using our principle, we are robust to an eventual presence of noise. In fact, more or less additive noise makes blurred pictures appear more or less sharp. By taking into account all variations of the image which have decrease after the blurring step, we analyze also the pixels containing the noise information which can be located on the edges, on the textured areas or on the homogeneous areas.

The last observation is done for pictures with a motion blur. For example, we observe this phenomenon for a picture extract from a traveling sequence. On a picture blurred in one direction, the eye perceives the blur effect on this direction but does not perceive the sharpness on the others directions. For this reason, we estimate the blur in the horizontal and the vertical directions and select the blur the more annoying as the final blur value.

Description of the blur estimation principle

Based on the phenomena explained in the previous section, we are able to quantify the blur annoyance on a picture by blurring it and comparing the variations between neighboring pixels before and after the low-pass filtering step. Consequently, the first step consists in the computation of the intensity variations between neighboring pixels of the input image. On this same image, we apply a low-pass filter and compute also the variations between the neighboring pixels. Then, the comparison between these intensity variations allows us to evaluate the blur annoyance. Thus, an high variation between the original and the blurred image means that the original image was sharp whereas a slight variation between the original and the blurred image means that the original image was already blurred. This description is summarized in Figure 4. Based on this principle, we describe in the following subsection the algorithm of this no-reference blur metric.

Algorithm description of the no-reference blur metric

It is known that the sharpness of an image is contained in its gray component. This assumption, which is verified with subjective tests, justify that we estimate the blur annoyance only on the luminance component. The flow chart in Figure 5 describes the steps of the algorithm description and refers to the following equations.

Let F be the luminance component of an image or a video frame of size of m × n pixels. To estimate the blur annoyance of F the first step consists in blurred it in order to obtain a blurred image B. We choose an horizontal and a vertical strong low-pass filter (1) to model the blur effect and to create B Ver and B Hor .

[ ]

1 1 1 1 1 1 1 1 1 9 1 × = v h ' ) ( v v h h h transpose h = = F h B v Ver * = F h B 1 , ) , ( _ _ n m j i Ver Ver j i F D F s ∑ - - = = 1 , 1 1 , ) , ( _ _ n m j i Hor Hor j i F D F s ∑ - - = = 1 , 1 1 , ) , ( _ _ n m j i Ver Ver j i V D V s ∑ - - = = 1 , 1 1 , ) , ( _ _ n m j i Hor Hor j i V D V s
Finally, we have to normalize the result in a defined range from 0 to 1:

Ver Ver Ver Ver F s V s F s F b _ _ _ _ - = Hor Hor Hor Hor F s V s F s F b _ _ _ _ - =
We note that the variations between the two differences images D_F and D_B are always slighter than the values of the initial difference image D_F.

Then, we select the blur the more annoying among the vertical one and the horizontal one as the final blur value. To summarize, we obtain a no-reference perceptual blur estimation blur_ F ranging from 0 to 1 which are respectively the best and the worst quality in term of blur perception.

EXPERIMENTS

Following the ITU-R recommendations [START_REF]Methodology for the subjective assessment of the quality of television pictures[END_REF], we made one main subjective experiment in order to assess the global quality of some blurred pictures.

Subjective test procedure

The aim of the procedure is to collect an absolute subjective opinion of each picture. In this way, we develop a program for psychophysics experiments with the psychtoolbox of Matlab. Before this experiment, a training session calibrates the opinion of each viewer with ten pretest pictures. These results are not taken into account in the final data. The experiment is a test-retest experiment, containing two equivalent parts separated by a pause of 3 minutes minimum. Each part contains a whole set of one hundred pictures. For the experiment, we set several timing parameters. Each picture is shown during a duration of 5 seconds. This time has been chosen after many tests to collect only the first spontaneous opinion of the viewers. Nevertheless, the viewers can assess the pictures faster by hitting a button. This allows to obtain results about visualization time. The picture is then replaced by a graph showing a five level graduation. No time limit has been set for the assessment. The viewer can give five scores between 1 and 5 corresponding to several quality levels as shown on the Table 1. After the assessment, 0.5 seconds of a neutral gray is presented before the next picture in order to suppress any retinal effect.

The room

The assessment of the quality of a picture depends on the environment. Thus, we make the main experiment in a total neutral gray environment in order to minimize parasite reflections on screen, walls, ceiling and table. During the experiment, the light was turned off contrary to the pause where the light was on in order to make the viewer's eyes rest (Figure 6).

The screen

The screen used is a 21" View Sonic G220f which has been previously calibrated with the GretagMacBeth Eye-One in order to have a 6500K color temperature. The resolution of the display is set to a resolution of 1400x1050 in order to fit with the picture's original size. The viewing distance is about 4 times the height of the screen, compliant with the ITU recommendations (Figure 6).

The pictures

The content of the pretest is chosen to represent the different levels of degradation. Then, several filters are randomly applied to 12 pictures: 2 types of Gaussian filters (3x3 and 5x5), 4 types of averaging (3x3, 5x5, 7x7 and 9x9) and motion filters (3x1, 5x1, 7x1 and 9x1) and one average disk filter with a diameter of 11 pixels. The 12 initial pictures represent natural scenes more or less textured. The Figure 7 illustrates a sample of the data set. In each part of the experiment, all pictures are randomly put on the screen and assessed. The random order allows to suppress any influence from the order of the presentation. Finally, 132 pictures are assessed twice per viewer.

Figure 7: Example of pictures from the data base

The paddle

We only used a paddle to make the experiment. Thus, the viewers are almost twice faster than using a keyboard. Finally, they take about 25 minutes to assess the 264 pictures. The experiment is all the more easy to pass that there are only 3 buttons (validate-confirm-cancel) and 2 arrows to change a score given to a picture.

The viewers

At the beginning of the experiment, information about the viewers are written in the database (age, sex, activity, ...). An Ishihara test is made in order to detect possible color blindness. Finally, the panel of observers is made up with 15 nonexpert viewers. Each one has a perfect visual acuity.

RESULTS

A similar experiment with gray scaled pictures and colored pictures has been made before this main one in order to prove that the luminance component study was sufficient to estimate the blur artifact. We notice that the scores corresponding to the gray scaled pictures were different only by 0.21 with a standard deviation of 0.19 on the five-grade scale. That confirms the fact that the analysis of the chrominance is not very important when we evaluate the blur effect.

Analysis of the viewers behavior

Following the ITU recommendations, the computation of the Mean Opinion Score (MOS) and statistical analysis allowed us to reject 3 viewers out of 15. Then, we examine the different times taken by the viewers to assess a picture, and we obtain some interesting results represented in Figure 8:

• There are no hesitation when the pictures are too blurred.

• On average, the highest scores have been given after a longer time than the lowest scores. Indeed, the viewers wanted to find some defaults before to give the score.

• There is a psychological jump in the subjective evaluation of the blur. Indeed there are less assessments with scores between 2 and 3.5. That implies that the blur is generally perceived or not.

Correlation between the subjective tests and the no-reference perceptual blur metric

In order to evaluate the proposed algorithm, we compared the subjective and the objective assessments. First, we observe a correlation between the objective and the subjective measurements. The Figure 9, which represents the MOS with the 95 percent interval versus the objective metric illustrates this correlation: the no-reference metric follows the human estimation. In order to evaluate more precisely these results, we try to find a relation between these two assessments.

Referring to psychophysics functions, if A Obj is the objective assessment and A Sub the subjective assessment, we tried to approximate the function ASub = f(Aobj) by the following function [START_REF]Methodology for the subjective assessment of the quality of television pictures[END_REF]. This interpolation allows to compare the subjective and the objective assessments relatively to the different filters. The Figure 10 illustrates for each filter, the subjective and the objective assessments. First, we note that the largest the filter is, the worst the quality is for both objective and subjective assessments. Then, by introducing the 95 percent confidence interval computed with the subjective results, we observe that the no-reference blur metric fits into this interval except for the nine pixels averaging filter that falls slightly outside. Note: In order to be a low-cost metric, we choose to compute the blur annoyance only in two directions (horizontal and vertical) but our principle is adapted to each directions. This choice depends not only to the wished precision but also to the cost implementation. In our case, we consider that the two directions give a result sufficiently accurate and well correlated with the human perception.

To conclude this subsection, we summarize in Figure 11 the several steps used to allow the validation of our noreference blur quality metric as a no-reference perceptual blur quality metric.

Figure 11: Flowchart of the no-reference blur quality metric validation step.

APPLICATIONS

This low-cost and robust no-reference perceptual blur metric can be used in different applications:

• A simple quality metric for pictures or video frames.

• A means to compare the quality of restoration methods or scaling methods. For example, this perceptual blur metric is well correlated with a subjective test aiming at estimate the quality of several scaling algorithms. • A means to fine-tune artifacts correction algorithms which have a tendency to smooth details. For example, we use this algorithm to control the emergence of blur of a deblocking algorithm.

• An other application consists in the sharpness improvement by using an inverse method to evaluate the best appropriate sharpness filter from the blur coefficient

CONCLUSION

Based on the discrimination between different levels of blur perceptible on the same image, we propose a new method to estimate the blur annoyance. We justify each step of our method with a general observation on the blur perception and we develop a no-reference perceptual blur metric, ranging from 0 to 1 which are respectively the best and the worst quality in term of blur perception. To evaluate its performance, we compared this metric with subjective tests done following the ITU recommendations as good as possible. We validated this metric for static images by showing the high correlation between this no-reference blur metric and the human estimation.

This new measure can now be used as a blur quality metric but also as a means to improve a picture sharpness or to finetune artifact corrections avoiding the emergence of blur. About the subjective tests, it is interesting to notice that there was only one default introduced in the tested pictures. If one more default was introduced, the subjective measures could be less well aligned to the objective ones. Further experiments are currently in progress to assess several artifacts at the same time and to understand how to combine objective metrics of different artifacts to produce only one final quality factor.

Figure 1 :

 1 Figure 1: from left to right: original picture, original picture blurred with an averaging filter, blurred picture re-blurred with the same averaging filter.

Figure 3 :

 3 Figure 3: A small blurred text over a large flat area.

Figure 4 :

 4 Figure 4: Simplified flow-chart of the blur estimation principle

Figure 5 :

 5 Figure 5: Flow chart of the algorithm with the equations references.

Figure 6 :

 6 Figure 6: Subjective tests room

Figure 8 :

 8 Figure 8: Average response time function of the average assessment

  numbers A and B are found with the boundary values of the function, and G, D and D M by computing the coefficients of a linear approximation of the following function:

Figure 9 :

 9 Figure 9: The no-reference perceptual blur metric versus the Mean Opinion Score (with the confidence interval)

ACKNOWLEDGMENTS

We would like to thank Nathalie Guyader who helped us to prepare the subjective test and led us for the results analysis. We also thanks the viewers who participated to our tests.

h Hor * = Then, in order to study the variations of the neighboring pixels, we compute the absolute difference images D_F Ver , D_F Hor , D_B Ver and D_B Hor as followed:

As we explain in the previous subsection, we need to analyze the variation of the neighboring pixels after the blurring step. If this variation is high, the initial image or frame was sharp whereas if the variation is slight, the initial image or frame was already blur. This variation is evaluated only on the absolute differences which have decreased:

Then, in order to compare the variations from the initial picture, we compute the sum of the coefficients of D_F Ver , D_F Hor , D_V Ver , D_V Hor as followed: