Keywords: Real-time multimodal semantic scene understanding for autonomous navigation Semantic segmentation, Image fusion, Multi-modal, Deep learning

ACKNOWLEDGEMENTS

First of all, I would like to express my sincere gratitude to my supervisors D ésir é Sidib é, Olivier Morel, and Fabrice M ériaudeau for their constant encouragement and guidance on my thesis. Without their patient instruction, insightful criticism, and expert guidance, the completion of experiments and papers would not have been possible. In preparing the doctoral thesis, they have spent much time reading through each draft and provided me with inspiring advice. I am very thankful to D ésir é for his continuous support and valuable advice in all our discussions. Thank him for choosing me and giving me the chance to pursue my dream of scientific research. I am very grateful to Olivier for his insightful comments, which enable my thesis work further improved, and thanks for his encouragement, which makes me more confident in my presentation. I would also like to convey my sincere thanks to Fabrice for his patience, motivation, enthusiasm, and trust.

Then, my faithful appreciation also goes to the jury members for their kindness in participating in my Ph.D. defense committee. Many thanks to Prof. Vincent Fremont and Prof. Samia Ainouz for their precious time reading the thesis and their constructive remarks, and to Prof.Christophe Ducottet for his support as the president of the defense and valuable suggestions for the thesis work.

I also owe my sincere gratitude to Samia B. and Frederick for their friendly reception.

During my visit to the IBISC laboratory, they have provided a lot of convenience for my life and work. During the epidemic, they provided me with the necessary conditions to prevent epidemics and tried their best to ensure my health and research progress. Since the 1960s, scientists are dedicated to creating machines that can see and understand the world like humans, which led to the emergence of computer vision. It has now become an active subfield of artificial intelligence and computer science for processing visual data. This thesis tackles the challenge of semantic segmentation in scene understanding, particularly multimodal image segmentation of the outdoor road scenes. Semantic segmentation, as a high-level task in the computer vision field, paves the way towards complete scene understanding. From a more technical perspective, semantic image segmentation refers to the task of assigning a semantic label to each pixel in the image [START_REF] Ohta | An analysis system for scenes containing objects with substructures[END_REF][START_REF] Garcia-Garcia | A Review on Deep Learning Techniques Applied to Semantic Segmentation[END_REF][START_REF] Yu | Methods and datasets on semantic segmentation: A review[END_REF]. This terminology was further distinguished from instancelevel segmentation [START_REF] Edelman | Integrating visual cues for object segmentation and recognition[END_REF] that devotes to produce per-instance mask and class label.

Recently, panoptic segmentation [START_REF] Kirillov | Panoptic Segmentation[END_REF][START_REF] Cheng | Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation[END_REF] is getting popular which combines pixel-level and instance-level semantic segmentation. Although there are many traditional machine learning algorithms available to tackle these challenges, the rise of deep learning techniques [START_REF] Lecun | Deep learning[END_REF][START_REF] Goodfellow | Deep learning[END_REF] gains unprecedented success and tops other approaches by a large margin. The various milestones in the evolution of deep learning significantly promote the advancement of semantic segmentation research.

Especially in recent years, deep multimodal fusion methods benefit from the massive amount of data and increased computing power. These fusion methods fully exploit hierarchical feature representations in an end-to-end manner. Multimodal information sources provide rich but redundant scene information, which also accompanied by uncertainty. Researchers engage in designing compact neural networks to extract valuable features, thus enhancing the perception of intelligent systems. The underlying motivation for deep multimodal image segmentation is to learn the optimal joint representation from rich and complementary features of the same scene. Moreover, the availability of multiple sensing modalities has encouraged the development of multimodal fusion, such as 3D LiDARs, RGB-D cameras, thermal cameras, etc. These modalities are usually used as complementary sensors in complex scenarios, reducing the uncertainty of scene information. For example, visual cameras perform advanced information processing in lighting conditions, while LiDARs are robust to challenging weather conditions such as rain, snow, or fog. Thermal cameras work well in the nighttime as they are more sensitive to infrared radiation emitted by all objects with a temperature above absolute zero [START_REF] Gade | Thermal cameras and applications: a survey[END_REF]. Arguably, the captured multimodal data provide more spatial and contextual information for robust and accurate scene understanding. Compared to using a single modality, multi-modalities significantly improve the performance of learning models [START_REF] Eitel | Multimodal deep learning for robust RGB-D object recognition[END_REF][START_REF] Liu | Multispectral deep neural networks for pedestrian detection[END_REF][START_REF] Xu | Multi-modal deep feature learning for rgb-d object detection[END_REF][START_REF] Asvadi | Multimodal vehicle detection: fusing 3d-lidar and color camera data[END_REF][START_REF] Yang | Multimodal Scene Understanding: Algorithms, Applications and Deep Learning[END_REF]. However, the generalization and discrimination abilities of existing unimodal few-shot segmentation methods still remain to be improved, especially for complex scenes. The semantic understanding of outdoor road scenes is usually affected by environmental changes, such as occlusion of objects and variable lighting conditions, which makes learning and prediction of the few-shot network difficult. In order to obtain complete scene information, we are also committed to extending the RGB-centric approach to take advantage of complementary depth information. The original intention of our work is to incorporate supplementary multimodal image information into a few-shot segmentation model. These multimodal data provide rich color and geometric information of scenes, leading to more accurate segmentation performance. voice, each form of information can be called a modality. We adopt the definition of modality from [START_REF] Lahat | Multimodal data fusion: an overview of methods, challenges, and prospects[END_REF], which refers to each detector acquiring information about the same scene.

The range of modalities is wider than our perception ability. In addition to the information obtained by vision, we can also collect multimodal information by various sensors such as radar and infrared cameras. Multimodal fusion systems work like the human brain, which synthesizes multiple sources of information for semantic perception and further decision making. Ideally, we would like to have an all-in-one sensor to capture all the information, but for most complex scenarios, it is hard for a single modality to provide complete knowledge. Consequently, the primary motivation for multimodal scene understanding is to obtain rich characteristics of the scenes by integrating multiple sensory modalities.

Our work focuses on deep learning-based multimodal fusion technology, which also involves multimodal collaborative learning, multimodal feature representation, multimodal alignment, etc.

As a multi-disciplinary research, the meaning of multi-modality varies in different fields.

For example, in medical image analysis, the principal modalities involve Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), Single-Photon Emission Computed Tomography (SPECT) [START_REF] Litjens | A survey on deep learning in medical image analysis[END_REF], to name a few.

Benefiting from the complementary and functional information about a target (e.g., organ), multimodal fusion models can achieve a precise diagnosis and treatment [START_REF] Moeskops | Deep learning for multi-task medical image segmentation in multiple modalities[END_REF][START_REF] Liu | A medical image fusion method based on convolutional neural networks[END_REF][START_REF] Zhou | A review: Deep learning for medical image segmentation using multi-modality fusion[END_REF].

In multimedia analysis, multimodal data collected from audio, video as well as text modalities [START_REF] Pradeep | Multimodal fusion for multimedia analysis: A survey[END_REF][START_REF] Mroueh | Deep multimodal learning for Audio-Visual Speech Recognition[END_REF][START_REF] Baltrušaitis | Multimodal machine learning: A survey and taxonomy[END_REF] are used to tackle semantic concept detection, including human-vehicle interaction [START_REF] Erzin | Multimodal person recognition for human-vehicle interaction[END_REF], biometric identification [START_REF] Robert | Biold: a multimodal biometric identification system[END_REF][START_REF] Soleymani | Multi-level feature abstraction from convolutional neural networks for multimodal biometric identification[END_REF][START_REF] Connor | Biometric recognition by gait: A survey of modalities and features[END_REF]. In remote sensing applications, multimodal fusion leverages the high-resolution optical data, synthetic aperture radar, and 3D

point cloud [START_REF] Audebert | Beyond RGB: Very high resolution urban remote sensing with multimodal deep networks[END_REF][START_REF] Zhang | Effective fusion of multi-modal remote sensing data in a Fully convolutional network for semantic labeling[END_REF]. In this thesis, we clarify the definition of modality for semantic segmentation tasks as a single image sensor. Relevant sensory modalities reviewed and experimented include RGB-D cameras, Near-InfraRed cameras, thermal cameras, and

2 0 1 0 2 0 1 1 2 0 1 2 2 0 1 3 2 0 1 4 2 0 1 5 2 0 1 6 2 0 1 7 2

1.2.2/ SEMANTIC IMAGE SEGMENTATION

In recent years, there have been many studies addressing semantic image segmentation with deep learning techniques [START_REF] Garcia-Garcia | A Review on Deep Learning Techniques Applied to Semantic Segmentation[END_REF][START_REF] Ghosh | Understanding Deep Learning Techniques for Image Segmentation[END_REF]. As a core problem of computer vision, scene understanding relies heavily on semantic segmentation technology to obtain semantic information and infer knowledge from imagery. Looking back at the history of semantic image segmentation, Fully Convolutional Network (FCN) [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] was first proposed for effective pixel-level classification. In FCN, the last fully connected layer is substituted by convolutional layers. DeconvNet [START_REF] Noh | Learning deconvolution network for semantic segmentation[END_REF], which is composed of deconvolution and unpooling layers, was proposed in the same year. Badrinarayanan et al. [START_REF] Badrinarayanan | SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation[END_REF] introduced a typical encoder-decoder architecture with forwarding pooling indices, mentioned as SegNet. Another typical segmentation network with multi-scale features concatenation, U-Net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF],

was initially proposed for biomedical image segmentation. In particular, U-Net employs skip connections to combine deep semantic features from the decoder with low-level finegrained feature maps of the encoder. Then a compact network called ENet [START_REF] Paszke | ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation[END_REF] was presented for real-time segmentation. In the work of PixelNet, Bansal et al. [START_REF] Bansal | Pixelnet: Towards a general pixel-level architecture[END_REF] explore the spatial correlations between pixels to improve the efficiency and performance of segmentation models. It is worth noting that Dilated Convolution was introduced in DeepLab [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF] and DilatedNet [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF], which helps to keep the resolution of output feature maps learn with large receptive fields. Besides, a series of Deeplab models also achieves excellent success on semantic image segmentation [START_REF] Liang Chieh Chen | DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs[END_REF][START_REF] Chen | Rethinking Atrous Convolution for Semantic Image Segmentation[END_REF][START_REF] Liang Chieh Chen | Encoder-decoder with atrous separable convolution for semantic image segmentation[END_REF].

Furthermore, Peng et al. [START_REF] Peng | Large kernel matters -Improve semantic segmentation by global convolutional network[END_REF] dedicated to employing larger kernels to address both the classification and localization issues for semantic segmentation. RefineNet [START_REF] Lin | Refinenet: Multi-path refinement networks for high-resolution semantic segmentation[END_REF] ex-RGB Depth NIR Polar ...

Multimodal data

Pixel-wise prediction

Fusion network plicitly exploits multi-level features for high-resolution prediction using long-range residual connections. Zhao et al. [START_REF] Zhao | IC-Net for Real-Time Semantic Segmentation on High-Resolution Images[END_REF] presented an image cascade network, known as ICNet, that incorporates multi-resolution branches under proper label guidance. In more recent years, semantic segmentation for adverse weather conditions [START_REF] Sakaridis | Model adaptation with synthetic and real data for semantic dense foggy scene understanding[END_REF][START_REF] Pfeuffer | Robust Semantic Segmentation in Adverse Weather Conditions by means of Sensor Data Fusion[END_REF] and nighttime [START_REF] Guan | Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection[END_REF][START_REF] Sun | See clearer at night: Towards robust nighttime semantic segmentation through day-night image conversion[END_REF] has also been addressed to perform the generalization capacity and robustness of deep learning models. Figure 1.2 shows the number of papers about segmentation published in the past decade. The statistical results include well-known computer vision conferences such as CVPR, ECCV, ICCV, BMVC, etc., as well as top journals such as IJCV, PAMI, etc.

In addition to the aforementioned networks, many practical deep learning techniques (e.g., Spatial Pyramid Pooling [START_REF] He | Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition[END_REF], CRF-RNN [START_REF] Zheng | Conditional random fields as recurrent neural networks[END_REF], Batch Normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], Dropout [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF]) were proposed for improving the effectiveness of learning models. Notably, multiscale feature aggregation was frequently used in semantic segmentation [START_REF] Zhang | Exfuse: Enhancing feature fusion for semantic segmentation[END_REF][START_REF] Li | DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation[END_REF][START_REF] Li | GFF: Gated Fully Fusion for Semantic Segmentation[END_REF][START_REF] Tian | Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation[END_REF][START_REF] Li | Global Aggregation then Local Distribution in Fully Convolutional Networks[END_REF]. These learning models experimentally achieve significant performance improvement. Lin et al. [START_REF] Lin | Network in network. 2nd International Conference on Learning Representations, ICLR 2014 -Conference Track Proceedings[END_REF] introduced the Global Average Pooling (GAP) that replaces the traditional fully connected layers in CNN models. GAP computes the mean value for each feature map without additional training of model parameters. Thus it minimizes overfitting and makes the network more robust. Related applications in multimodal fusion networks can be found in [START_REF] Zhu | FANet: Quality-Aware Feature Aggregation Network for RGB-T Tracking[END_REF][START_REF] Valada | Self-supervised model adaptation for multimodal semantic segmentation[END_REF][START_REF] Hu | ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation[END_REF]. Also, the 1 × 1 convolution layer is commonly used to allow complex and learnable interaction across modalities and channels [START_REF] Hung | Incorporating Luminance, Depth and Color Information by a Fusion-based Network for Semantic Segmentation[END_REF][START_REF] Valada | Self-supervised model adaptation for multimodal semantic segmentation[END_REF]. Besides, attention mechanism has become a powerful tool for image recognition [START_REF] Chen | Attention to scale: Scale-aware semantic image segmentation[END_REF][START_REF] Ren | End-to-end instance segmentation and counting with recurrent attention[END_REF][START_REF] Li | Attention-guided unified network for panoptic segmentation[END_REF].

The attention distribution enables the model to selectively pick valuable information [START_REF] Vaswani | Attention is all you need[END_REF],

achieving more robust feature representation and more accurate prediction.

1.2.3/ DEEP MULTIMODAL SEMANTIC SEGMENTATION

Before the tremendous success of deep learning, researchers expressed an interest in combining data captured from multiple information sources into a low-dimensional space, known as early fusion or data fusion [START_REF] Khaleghi | Multisensor data fusion: A review of the state-of-the-art[END_REF]. Machine learning techniques used for such fusion include Principal Component Analysis (PCA), Independent Components Analy-sis (ICA), and Canonical Correlation Analysis (CCA) [START_REF] Ramachandram | Deep multimodal learning: A survey on recent advances and trends[END_REF]. As the discriminative classifiers [START_REF] Ludmila | Combining pattern classifiers: methods and algorithms[END_REF] become increasingly popular (e.g. SVM [START_REF] Fi Érrez-Aguilar | Fusion strategies in multimodal biometric verification[END_REF] and Random Forest [START_REF] Gonz Ález | Onboard object detection: Multicue, multimodal, and multiview random forest of local experts[END_REF]), a growing body of research focus on integrating multimodal features at the late stage, such fusion strategy was called late fusion or decision fusion. These fusion strategies had become mainstream for a long time until the popularity of convolutional neural networks.

Compared to conventional machine learning algorithms, deep learning-based methods have competitive advantages in high-level performance and learning ability. In many cases, deep multimodal fusion methods extend the unimodal algorithms with an effective fusion strategy. Namely, these fusion methods do not exist independently but derive from existing unimodal methods. The representative unimodal neural networks, such as

VGGNet [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] and ResNet [START_REF] He | Deep residual learning for image recognition[END_REF], are chosen as the backbone network for processing data in a holistic or separated manner. The initial attempt of deep multimodal fusion for image segmentation is to train the concatenated multimodal data on a single neural network [START_REF] Couprie | Indoor Semantic Segmentation using depth information[END_REF]. We will present a detailed literature review of recent achievements in Chapter 3, covering various existing fusion methodologies and multimodal image datasets. Next, we point out three core challenges of deep multimodal fusion:

Accuracy As one of the most critical metrics, accuracy is commonly used to evaluate the performance of a learning system. Arguably, the architectural design and the quality of multimodal data have a significant influence on accuracy. How to optimally explore the complementary and mutually enriching information from multiple modalities is the first fundamental challenge.

Robustness Generally, we assume that deep multimodal models are trained under the premise of extensive and high-quality multimodal data input. However, multimodal data not only brings sufficient information but also brings redundancy and uncertainty. Ensuring network convergence can be a significant challenge with the use of redundant multimodal data. Moreover, sensors may behave differently or even in reverse during information collection. The poor performance of individual modality and the absence of modalities should be seriously considered.

Real-time

In practical applications, multimodal fusion models need to satisfy specific requirements, including the simplicity of implementation, scalability, etc. These factors have a vital impact on the efficiency of the autonomous navigation system.

1.2.4/ SCENARIOS AND APPLICATIONS

As one of the major challenges in scene understanding, deep multimodal fusion for semantic segmentation task cover a wider variety of scenarios. For instance, Hazirbas et al. [START_REF] Hazirbas | FuseNet: Incorporating depth into semantic segmentation via fusion-based CNN architecture[END_REF] address the problem of pixel-level prediction of indoor scenes using color and depth data. Schneider et al. [START_REF] Schneider | Multimodal neural networks: RGB-D for semantic segmentation and object detection[END_REF] present a mid-level fusion architecture for urban scene segmentation. Similar works in both indoor and outdoor scene segmentation can be found in [START_REF] Valada | Self-supervised model adaptation for multimodal semantic segmentation[END_REF]. Furthermore, the work by Valada et al. [START_REF] Valada | Deep Multispectral Semantic Scene Understanding of Forested Environments Using Multimodal Fusion[END_REF] led to a new research topic in scene understanding of unstructured forested environments. Considering non-optimal weather conditions, Pfeuffer and Dietmayer [START_REF] Pfeuffer | Robust Semantic Segmentation in Adverse Weather Conditions by means of Sensor Data Fusion[END_REF] investigated a robust fusion approach for foggy scene segmentation. As an illustration, Figure 1.3 shows the pipeline of deep multimodal image segmentation.

Besides the image segmentation task mentioned above, there are many other scene understanding tasks that benefit from multimodal fusion, such as object detection [START_REF] Eitel | Multimodal deep learning for robust RGB-D object recognition[END_REF][START_REF] Karasawa | Multispectral object detection for autonomous vehicles[END_REF][START_REF] Bijelic | Seeing Through Fog Without Seeing Fog: Deep Sensor Fusion in the Absence of Labeled Training Data[END_REF], human detection [START_REF] Mees | Choosing smartly: Adaptive multimodal fusion for object detection in changing environments[END_REF][START_REF] Liu | Multispectral deep neural networks for pedestrian detection[END_REF][START_REF] Wagner | Multispectral pedestrian detection using deep fusion convolutional neural networks[END_REF][START_REF] Guerry | Look at this one" detection sharing between modality-independent classifiers for robotic discovery of people[END_REF], salient object detection [START_REF] Chen | Progressively complementarity-aware fusion network for RGB-D salient object detection[END_REF][START_REF] Wang | Adaptive Fusion for RGB-D Salient Object Detection[END_REF], trip hazard detection [START_REF] Mcmahon | Multimodal Trip Hazard Affordance Detection on Construction Sites[END_REF] and object tracking [START_REF] Zhu | FANet: Quality-Aware Feature Aggregation Network for RGB-T Tracking[END_REF]. Especially for autonomous systems, LiDAR is always employed to provide highly accurate three-dimensional point cloud information [START_REF] Zhang | Sensor fusion for semantic segmentation of urban scenes[END_REF][START_REF] Janai | Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art[END_REF]. Patel et al. [START_REF] Patel | Sensor modality fusion with CNNs for UGV autonomous driving in indoor environments[END_REF] demonstrated the utility of fusing RGB and 3D LiDAR data for autonomous navigation in the indoor environment. Moreover, many works adopting point cloud maps reported in recent years have focused on 3D object detection (e.g., [START_REF] Chen | Multi-view 3D object detection network for autonomous driving[END_REF][START_REF] Pfeuffer | Optimal sensor data fusion architecture for object detection in adverse weather conditions[END_REF][START_REF] Xu | PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation[END_REF]). It is reasonably foreseeable that deep multimodal fusion of homogeneous and heterogeneous information sources can be a strong emphasis for intelligent mobility [START_REF] Feng | Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges[END_REF][START_REF] Xiao | Multimodal End-to-End Autonomous Driving[END_REF] in the near future.

1.2.5/ FEW-SHOT SEMANTIC SEGMENTATION

Few-shot segmentation presents a significant challenge for semantic scene understanding under limited supervision. Namely, this task targets at generalizing the segmentation ability of the model to new categories given a few samples. Existing methods generally address this problem by learning a set of parameters or prototypes from support images and guiding the pixel-wise segmentation on the query image. However, few-shot learning comes with the problem of data imbalance. Small-scale data may lead to overfitting and insufficient model expression ability. Therefore in this thesis, we are committed to exploring effective few-shot segmentation models with advanced deep learning techniques such as multiscale feature aggregation and attention mechanism. We expect that the discriminative power, the generalizability, and the training efficiency of the segmentation model can be significantly improved. In addition, distinguishing objects with similar characteristics, especially when they are placed in an overlapping manner, is one of the most challenging tasks for few-shot segmentation. The model's ability to recognize irregular objects and their boundary contours is also crucial.

Moreover, multimodal data such as depth maps are frequently used to provide rich geometric information of the scenes in fully-supervised semantic segmentation. Deep neural networks usually exploit the depth maps as an additional image channel or point cloud in 3D space. Arguably, the integration of supplementary scene information leads to significant performance improvement. However, existing methods focus on the unimodal few-shot segmentation. In our work, we take inspiration from existing RGB-centric methods for few-shot semantic segmentation and explore the effective use of depth information and fusion architecture for few-shot segmentation.

1.3/ CONTRIBUTIONS

Our contribution mainly consists of two parts, deep multimodal fusion for fully-supervised semantic image segmentation and semi-supervised semantic scene understanding. Regarding the former case, our contributions can be summarized as follows:

I We propose a late fusion-based neural network for outdoor scene understanding.

In particular, we introduce the first-of-its-kind dataset for multimodal image segmentation, which contains aligned raw RGB images and polarimetric images. II We present a two-stream deep neural network based on metric learning, which incorporates depth information into few-shot semantic segmentation. We also build a novel benchmark dataset, known as Cityscapes-3 i , to evaluate the multimodal few-shot semantic image segmentation.

II

ORGANIZATION

The different contributions have been published in the following papers:

• Journal papers In general, convolution is a mathematical operator that generates a third function s from two functions f and g. Formally,

s(t) = ( f * g)(t) (2.1)
In continuous space, convolution can be defined as:

( f * g)(t) = ∞ -∞ f (τ)g(t -τ)dτ (2.2)
In discrete space, it can be written as:

( f * g)(n) = ∞ k=-∞ f (k)g(n -k) (2.3)
According to Equations (2.2) and (2.3), we can see that convolution is the accumulation of the persistent consequences of instantaneous action. Therefore convolution is used as an effective method of mixing information, which is frequently applied in various fields such as signal analysis and image processing (e.g., image blur, edge enhancement).

In deep learning-based image processing, the function f is usually called Input, while the function g is called Kernel function. The output s is sometimes referred to as Feature map. The inputs are usually multidimensional arrays of data, and kernel functions are the parameters of multidimensional arrays optimized by learning algorithms. Suppose that we take a two-dimensional image I as input, and the two-dimensional kernel is G, then

S (i, j) = (I * G)(i, j) = m n I(m, n)G(i -m, j -n) (2.4)
Because of the commutativity of convolution, Equation (2.4) can also be written as:

S (i, j) = (G * I)(i, j) = m n I(i -m, j -n)G(m, n) (2.5)
At this point, G is equivalent to the learned filter, while the feature map S can be computed CNNs are composed of multiple neural units, which can be generally divided into three types, namely, the input layer, the hidden layer and the output layer. The input layer of a convolutional neural network is mainly used to obtain input information, which can process multidimensional data. For example, one-dimensional data is usually time or sampled spectrum, while two-dimensional data may contain multiple channels, such as a three-channel color image, and three-dimensional data may be 3D images such as CR and MRI image. In particular, we focus on 2D/2.5D multimodal images in this thesis.

The output layer of a convolutional neural network usually employs a logical function or softmax function to output the classification labels. The practical application varies according to the type of task. For instance, the output layer can be the central coordinates, size, and classification of objects in object detection. In semantic image segmentation, the output layer directly outputs the classification label of each pixel.

In general, each neural unit in the input layer directly connects to the original data, and provides feature information to the hidden layer. Each neural unit in the hidden layer represents different weights for different neural units in the input layer, so it tends to be sensitive to a certain recognition pattern. The values in the output layer vary according to the activation degree of hidden layers, which is the final recognition result of the model.

Compared with the input layer and output layer, the hidden layer is more complex because it is designed for abstract feature extraction. It usually includes the convolutional layer, pooling layer and fully connected layer. In Section 2.2, we introduce the common layers in CNNs.

2.1.3/ ENCODER-DECODER ARCHITECTURE

A convolutional encoder-decoder network is a standard architecture used for tasks requiring dense pixel-wise predictions. Such neural network design pattern is frequently used in semantic image segmentation, which is partitioned into two parts, the encoder and the decoder. Figure 2.2 shows a typical example of the encoder-decoder architecture.

In general, an encoder takes the image input and progressively computes higher-level abstract features. The role of the encoder is to encode the low-level image features into a high-dimensional feature vector. The spatial resolution of the feature maps is reduced progressively via the down-sampling operation. Multiple common backbone networks such as VGG, Inception, and ResNet, can be employed for abstract feature extraction in the encoder. The extracted semantic information is then passed into the decoder to compute feature maps of progressively increasing resolution via un-pooling or up-sampling.

The decoder restores the learned valuable feature representation into a pixel-level segmentation mask, which is one of the reasons why the encoder-decoder architecture can be effectively used for image segmentation tasks.

Besides, different variations of the encoder-decoder architecture have been explored to improve the segmentation performance. To name a few, skip connections [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] have been used to recover the fine spatial details during decoding which get lost due to successive down-sampling operations involved in the encoder. Moreover, larger context information using image-level features [START_REF] Liu | ParseNet: Looking Wider to See Better[END_REF], recurrent connections [START_REF] Pinheiro | Recurrent convolutional neural networks for scene labeling[END_REF][START_REF] Zheng | Conditional random fields as recurrent neural networks[END_REF], and larger convolutional kernels [START_REF] Peng | Large kernel matters -Improve semantic segmentation by global convolutional network[END_REF] has also significantly improved the accuracy of semantic segmentation. the filter applications to the input image is referred to as the stride, and it is almost always symmetrical in height and width dimensions. For example, the stride (2, 2) means moving the filter two pixels right for each horizontal movement of the filter and two pixels down for each vertical movement of the filter when creating the feature map. The stride of the filter on the input image can be seen as the downsampling of the output feature map.

Next, we introduce three essential properties of the convolutional layers: sparse interactions, parameter sharing, and equivariant representations.

Sparse interactions Convolutional neural networks have sparse interactions by making

the kernel smaller than the input. When processing an image with thousands of pixels, we can detect small valuable features such as the edges of the image by taking only tens to hundreds of pixels. This not only reduces the storage requirements of the model but also improves its overall computation efficiency.

Parameter sharing Parameter sharing is the sharing of weights by all neurons in a particular feature map. Each neuron is connected only to a subset of the input image, which is also called local connectivity. This property helps to reduce the number of parameters in the whole system and makes the computation more efficient.

Equivariant representations For convolution operation, parameter sharing makes the neural network layers have an equivariant representation. Namely, if the input is slightly shifted, the result of the convolution operation is the same. Note that the convolution is not naturally equivalent to some other transformations such as image scaling or rotation transformation.

2.2.2/ POOLING LAYER

Pooling operation plays a vital role in the structure of convolutional neural networks. First of all, pooling layers improve the spatial invariance to some extent, such as translation invariance, scale invariance, and deformation invariance. Namely, even if the image input is transformed slightly, the pooling layer can still produce similar pooling features, making the learning system more robust. Secondly, pooling operation is equivalent to feature downsampling, which increases the receptive field size. For some visual tasks, a large receptive field helps learn long-range spatial relationships and implicit spatial models. In addition, pooling operation greatly reduces the model parameters, which leads to a lower risk of overfitting. Suppose that the dimension of the image input is c × w × h, where c is the number of channels, and w and h are the width and height, respectively. If the stride of the pooling layer is set to 2, the dimension of the output image will be c × w/2 × h/2.

In this case, both the computational cost and memory consumption will be reduced by a factor of 4.

Common pooling methods include average pooling and maximum pooling. Maximum pooling calculates the maximum value of the target patch, which retains more texture information of the image input, whereas average pooling keeps more background information and tends to transfer the comprehensive information in the architecture of convo-

2.2.3/ RELU LAYER

In neural networks, the activation function performs the nonlinear transformation to the input, making it capable of learning and performing more complex tasks. In order to increase the nonlinearity of neural networks, some nonlinear functions are introduced.

Obviously, the accumulation of multiple linear functions is still linear, while linear functions have limited expression. The use of nonlinear functions makes the network more expressive and thus better fits the target function. Two common nolinear function used in convolutional neural networks are the sigmoid function and the rectified linear unit (ReLU).

As shown in Figure 2.4, we can observe that the ReLU activation function has more advantages than the sigmoid function. ReLU can carry out negative suppression so as to be more sparsely active. More importantly, ReLU activation functions suffer less from the vanishing gradient problem. The derivative of the sigmoid function has good activation only when it is near zero. The gradient in the positive and negative saturation region is close to zero. Also, the derivative of the ReLU function is easy to calculate, which can accelerate the model training to some extent.

2.2.4/ FULLY CONNECTED LAYER

The fully connected layer is usually used as a classifier to connect the hidden layer and the final output. In the architecture of convolutional neural networks, adding several fully connected layers after the convolution layers can map the generated feature map into a fixed-length feature vector. The final output represents the numerical description of the input image. This structural property is conducive to the realization of image-level classification and regression tasks.

Although multiple fully connected layers can significantly improve the nonlinear expression ability of learning models, a large number of neurons increase the model complexity.

Plenty of model parameters will reduce the efficiency of the learning algorithm and even lead to overfitting. Therefore, the trade-off between accuracy and efficiency has been deeply explored in deep learning technology research. For the segmentation task, however, spatial information should be stored to make a pixel-wise classification. Hence, the fully connected layer is usually substituted by another convolution layer with a large receptive field.

2.3/ OPTIMIZATION

2.3.2/ DROPOUT

Deep neural networks are likely to get overfitting while training with few examples. As early as 2012, Hinton et al. [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF] has proposed the concept of dropout, which is now widely used in advanced neural networks. Probabilistically dropping out nodes in the network is a simple and effective regularization method [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF]. In each iteration, some nodes are randomly deleted, and only the remaining nodes are trained. This optimization method reduces the correlation between nodes and the complexity of the model to achieve the effect of regularization.

Generally, dropout only needs to set a hyper-parameter that is the proportion of nodes randomly preserved in each layer. Namely, the parameter matrix of this layer is calculated with the binary matrix generated by the hyper-parameter via the point by point product.

Suppose that the hyper-parameter is set to 0.7, then 30% of the nodes will be randomly ignored and produce no outputs from the layer. A good value for dropout in a hidden layer is between 0.5 and 0.8. Input layers use a larger dropout rate, such as of 0.8.

2.4/ MODEL TRAINING

The process of training neural networks is the most challenging part of using deep learn- Besides, data augmentation [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF] is frequently used in model training, which increases the amount of input data by adding slightly modified copies of already existing data or newly created synthetic data from existing data. In the real world scenarios, we may have a small-scale dataset of images taken in a limited set of conditions. In the case of limited data, data augmentation can increase the diversity of training samples, so as to improve the robustness of the model and avoid overfitting. Typical operations include flipping, rotation, shift, resize, random scale, random crop, color jittering, contrast, noise, fancy PCA, GAN, etc.

2.4.2/ WEIGHT INITIALIZATION

Training a deep learning model means learning good values for all the weights and the bias from labeled examples. In particular, the bias allows to shift the activation function by adding a constant. In order to consistently update the weights, the models require each parameter to have the corresponding initial value. For convolutional neural networks, the nonlinear function is superimposed by multiple layers, and how to select the initial value of parameters becomes a problem worthy of discussion.

In general, the purpose of weight initialization [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF][START_REF] Mishkin | All you need is a good init[END_REF] is to prevent the layer activation output from exploding or disappearing in the forward transfer process of deep neural networks. In either case, the loss gradient is either too large or too small to flow backward advantageously. The learning model will then take a longer time to converge. Also, it is notable that initializing all the weights with zeros leads the neurons to learn the same features during training. The model can not get the update of parameters correctly. For example, assume we initialize all the biases to zero and the weights with some constant α. If we forward propagate an input (x 1 , x 2 ) in the network, the output of hidden layers will be relu(αx 1 + αx 2 ). Namely, the hidden layers will have an identical influence on the cost, which will result in identical gradients.

In practice, researchers usually employ the Xavier initialization [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF] to keep the variance the same across every layer. Another common initialization is He initialization [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF], in which the weights are initialized by multiplying by two the variance of the Xavier initialization.

2.4.3/ LOSS FUNCTION

The neural networks are usually trained using the gradient descent optimization algorithm. Generally, the optimization problem involves an objective function that indicates the direction of optimization. During the optimization process, the network tries to find a candidate solution to maximize or minimize the objective function. Under constraint conditions, we calculate and minimize the model error via a loss function or cost function, which evaluates the fit of the learning model. This series of constraints is a regularization term that helps to prevent overfitting. The weights are updated using the backpropagation of error algorithm. Therefore we need to choose a suitable loss function when designing and configuring the model.

Suppose that there is a series of training samples {(x i , y i )} i=1,...N in supervised learning.

The model learns the mapping relation of x → y, so that given a x, even if the x is not in the training samples, it can get the output ŷ as close to the real y as possible. The loss function is a key component to indicate the direction of model optimization, which is used to measure the difference between the output ŷ of the model and the real output y, namely, L = f (y i , ŷi ).

In the following, we introduce several loss functions commonly used for classification and regression in deep learning, including mean squared error loss, mean absolute error loss and cross-entropy loss.

* Mean Squared Error Loss

Mean Squared Error Loss (MSE), also known as Quadratic Loss or L 2 Loss, is the most commonly used loss function in machine learning and deep learning regression tasks. Formally,

J MS E = 1 N N i=1 (y i -ŷi ) 2 (2.6)
Under the assumption that the error between the output of the model and the real value follows a Gaussian distribution, the minimum mean square error loss function and the maximum likelihood estimate are essentially consistent. Therefore, in the scenario where the assumption can be satisfied (such as regression), the mean square error loss is a good choice for the loss function. In scenarios where this assumption is not satisfied (such as classification), other losses have to be considered.

* Mean Absolute Error Loss

Mean Absolute Error Loss (MAE), also known as L 1 Loss, is another common loss function. MSE loss generally converges faster than MAE loss, however the latter is more robust to outlier, which can be defined as:

J MAE = 1 N N i=1 |y i -ŷi | (2.7)
*

Cross-Entropy Loss

Previous loss functions introduced above are applicable to regression problems.

For classification tasks, especially for semantic image segmentation, the most commonly used loss function is the cross-entropy loss that increases as the predicted probability diverges from the actual label. For the multi-class cross-entropy loss, we can obtain:

J CE = - N i=1 y c i i log( ŷi c i ) (2.8)
Here, y c i i can be 0 or 1, indicating whether class label c i is the correct classification.

2.4.4/ GRADIENT DESCENT

A neural network is merely a complicated function, consisting of millions of parameters, that represents a mathematical solution to a problem. By optimizing various parameters of the network, the trained model is finally matched to the learning task under a certain parameter set. The gradient descent method is one of the most commonly used methods to achieve such learning process. In brief, the gradient descent method, also referred to as steepest descent method, is a method to find the minimum of the objective function.

From a mathematical point of view, the gradient indicates the direction of the fastest growth of the function. In other words, the opposite direction of the gradient is the direction of the fastest function decrease. Once we have the direction to move in, we need to decide the size of the step we take. The size of this step is called the learning rate. It is worth noting that if the value of the learning rate is too large, we might overshoot the minima, and keep bouncing along the ridges of the "valley" without ever reaching the minima. And small learning rate might lead to painfully slow convergence, even get stuck in a minima. The closer the gradient descent method is to the target value, the smaller the step size is and the slower the progress is.

There are three variants of gradient descent, including batch gradient descent, stochastic gradient descent, and mini-batch gradient descent. These variants differ in how much data we use to compute the gradient of the objective function. In detail, batch gradient descent computes the gradient of the cost function for the entire training dataset. Stochastic gradient descent in contrast performs a weight update for each training example and the corresponding label. As for mini-batch gradient descent, it performs an update for every mini-batch of n training examples. Taking batch gradient descent as an example, the weight update process can be defined as:

θ := θ -α∇ θ J(θ) (2.9)
where J(θ) is the objective function parameterized by the weights θ of the learning model.

The parameters is updated in the opposite direction of the gradient of the objective function ∇ θ J(θ). The learning rate α determines the size of the steps we take to reach a minimum.

Moreover, several gradient descent optimizers have been proposed, including Momentum SGD, AdaGrad, RMSProp, Adam, etc. For example, Momentum SGD [START_REF] Ning Qian | On the momentum term in gradient descent learning algorithms[END_REF] is a method that helps accelerate stochastic gradient descent in the relevant direction and dampens oscillations. It adds a fraction γ of the update vector of the past time step to the current update vector.

v t = γv t-1 + α∇ θ J(θ) (2.10)
Then the weights can be updated by θ := θv t . The momentum term γ increases for dimensions whose gradients point in the same directions and reduces updates for dimensions whose gradients change directions. As a result, we gain faster convergence and reduced oscillation. Besides, AdaGrad [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF] is an alternative algorithm for gradient-based optimization. It adapts the learning rate to the parameters, performs smaller updates for parameters associated with frequently occurring features, and larger updates for infrequent features. Adaptive Moment Estimation (Adam) [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] is a method that computes adaptive learning rates for each parameter. In addition to storing an exponentially decaying average of past squared gradients, Adam also keeps an exponentially decaying average of past gradients, similar to momentum. Each algorithm has its advantages and disadvantages, we can choose the appropriate optimizer according to the input data and training requirements.

2.5/ EVALUATION METRICS

The segmentation performance is generally affected by many factors, such as the preprocessing of data, fusion strategy, the choice of the backbone network, the practice of state-of-the-art deep learning technologies, etc. Therefore how to evaluate and compare the performance of segmentation algorithms is a critical issue. The validity and usefulness of a learning system can be measured in many aspects, such as execution time, memory footprint, and accuracy. It is notable that existing large-scale benchmark datasets promote the standardization of comparison metrics, providing a fair comparison of the state-of-the-art methods. In Chapter 3.2.3, we provide a comparative analysis of existing deep multimodal segmentation methods in terms of common metrics.

In general, accuracy is the most common evaluation criteria to measure the performance of pixel-level prediction [START_REF] Fernandez-Moral | A new metric for evaluating semantic segmentation: leveraging global and contour accuracy[END_REF]. For multimodal image segmentation, the most popular metrics are not different from those used in unimodal approaches, including Pixel Accuracy (PA), Mean Accuracy (MA), Mean Intersection over Union (MIoU), and Frequency Weighted Intersection over Union (FWIoU), which are first employed in [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF].

In our work, we mainly report a series of segmentation results in Intersection over Union (IoU), also known as the Jaccard Index. The IoU for each category is defined as IoU = T P/(T P + FP + FN), where TP, FP and FN denote true positives, false positives and false negatives, respectively.

For the sake of explanation, we denote n i j as the number of pixels belonging to class i which are classified into class j, and we consider that there are n cl classes, and t i = j n i j is the numbers of pixel in class i. Therefore we can define these accuracy metrics as follows:

• Pixel Accuracy

i n ii / i t i • Mean Accuracy (1/n cl ) i n ii /t i • Mean Intersection over Union (1/n cl ) i n ii /(t i + j n ji -n ii )
• Frequency Weighted Intersection over Union

( k t k ) -1 i t i n ii /(t i + j n ji -n ii )
Besides, researchers usually evaluate the real-time performance of autonomous navigation systems by measuring the execution time and memory consumption. Although these indicators are usually closely related to hardware settings, they are also essential for model optimization.

2.6/ SUMMARY

3.1/ FULLY-SUPERVISED SEMANTIC IMAGE SEGMENTATION

In this subsection, we provide a comprehensive review of deep multimodal fusion methods according to our taxonomy. We highlight their benefits and drawbacks, providing interested readers with a complete overview of deep fusion strategies.

3.1.1/ TAXONOMY OF DEEP MULTIMODAL FUSION

In the early works [START_REF] Pradeep | Multimodal fusion for multimedia analysis: A survey[END_REF][START_REF] Zhang | Sensor fusion for semantic segmentation of urban scenes[END_REF][START_REF] Liu | Learn to combine modalities in multimodal deep learning[END_REF], the classification of multimodal fusion strategies involves various taxonomic methods, including data fusion, early fusion, late fusion, intermediate fusion, and hybrid fusion. In this review, we explicitly divide the deep multimodal fusion methods into early fusion, late fusion, and hybrid fusion, according to the fusion stage and motivation (see Figure 3.1).

Early fusion methods involve raw data-level fusion and feature-level fusion. The initial attempt of early fusion is to concatenate the raw data from different modalities into multiple channels. The learning model can be trained end-to-end using an individual segmentation network. Almost all the state-of-the-art segmentation networks are adaptable for such fusion strategy. Moreover, cross-modal interactions throughout the encoding stage, namely feature-level fusion, is also a distinctive manifestation of early fusion. For the sake of explanation, we denote the single segmentation network as I, (x 1 , x 2 , ..., x n ) is a set of n modalities as input, then the final prediction y can be defined as:

y = I(x 1 , x 2 , ..., x n ). (3.1)
On the contrary, late fusion methods aim to integrate multimodal feature maps at decisionlevel. More precisely, late fusion separately processes the multimodal data in different branches. During the decoding stage, all the feature maps computed by branches are mapped into a common feature space via fusion operations (e.g., concatenation, addition, averaging, weighted voting, etc.) [START_REF] Feng | Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges[END_REF], followed by a series of convolutional layers. Besides, we consider the common feature learned by the transformation network as a further refinement of decoding and prediction, some conventional intermediate fusion approaches (e.g., [START_REF] Wang | Learning common and specific features for rgb-d semantic segmentation with deconvolutional networks[END_REF]) are therefore categorized into late fusion strategy in this review.

Suppose that the segmentation networks (I 1 , . . . , I n ) are used to process the multimodal data (x 1 , x 2 , ..., x n ) from different modalities, and P is the fusion operation as well as the following convolutional layers, the final output y can be formulated as:

y = P(I 1 (x 1 ), I 2 (x 2 ), ..., I n (x n )). (3.2)
Hybrid fusion methods are elaborately designed to combine the strengths of both early and late fusion strategies. Generally, the segmentation network accesses the data through the corresponding branch. Then more than one extra module is employed to compute the class-wise or modality-wise weights and bridge the encoder and decoder with skip connections. Therefore the hybrid fusion networks can adaptively generate a joint feature representation over multiple modalities, yielding a better performance in terms of accuracy and robustness.

Based on such common taxonomy of fusion strategy, we systematically review the existing deep multimodal fusion networks for semantic image segmentation in the following sections.

3.1.1.1/ EARLY FUSION

The first attempt at deep multimodal fusion was made by Couprie et al. [START_REF] Couprie | Indoor Semantic Segmentation using depth information[END_REF] in 2013.

This work presents an early fusion strategy via a simple concatenation of RGB and depth channels before feeding into a segmentation network. In the case of similar depth appearance and location, this method shows positive results for indoor scene recognition.

However, the simple concatenation of images provides limited help in multimodal feature extraction. The high variability of depth maps, to a certain extent, increase the uncertainty of feature learning.

To further explore semantic labeling on RGB-D data, FuseNet [START_REF] Hazirbas | FuseNet: Incorporating depth into semantic segmentation via fusion-based CNN architecture[END_REF] was proposed in 2016 (see Figure 3.2). FuseNet is a clear example of incorporating the auxiliary depth information into an encoder-decoder segmentation framework. The abstract features obtained from the depth encoder are simultaneously fused to the RGB branch as the network goes deeper. Motivated by FuseNet, Ma et al. [START_REF] Ma | Multi-view deep learning for consistent semantic mapping with rgb-d cameras[END_REF] proposed MVCNet to predict multi-view consistent semantics. Then Hung et al. [START_REF] Hung | Incorporating Luminance, Depth and Color Information by a Fusion-based Network for Semantic Segmentation[END_REF] presented LDFNet that contains a welldesigned encoder for the non-RGB branch, aiming to fully make use of luminance, depth, and color information. Recently, RFBNet [START_REF] Deng | RFB-Net: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation[END_REF] was proposed with an efficient fusion mechanism that explores the interdependence between the encoders. The Residual Fusion Block (RFB), which consists of two modality-specific residual units (RUs) and one gated fusion unit (GFU), was employed as the basic module to achieve the interactive fusion in a bottom-up way. Hu et al. [START_REF] Hu | ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation[END_REF] proposed a novel early fusion architecture based on attention mechanism, known as ACNet, which selectively gathers valuable features from RGB and depth branches. Besides, RTFNet [START_REF] Sun | RTFNet: RGB-Thermal Fusion Network for Semantic Segmentation of Urban Scenes[END_REF] was particularly designed to fuse both RGB and thermal images by element-wise summation. Notably, average pooling and the fully connected layers in the backbone network was removed to avoid the excessive loss of spatial information. Besides, Wang et al. [START_REF] Wang | Learning common and specific features for rgb-d semantic segmentation with deconvolutional networks[END_REF] proposed a feature transformation network for learning the followed by a series of convolutional layers. Afterward, the authors extended the LFC method with a convoluted mixture of deep expert units, referred to as CMoDE [START_REF] Valada | Adapnet: Adaptive semantic segmentation in adverse environmental conditions[END_REF]. This deep fusion framework was inspired by the work [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF][START_REF] Eigen | Learning factored representations in a deep mixture of experts[END_REF], in which multimodal features are mapped to a particular subspace. An adaptive gating subnetwork is employed to produce class-wise probability distribution over the experts (see Figure 3.3). In the work of LSD-GF, Cheng et al. [START_REF] Cheng | Locality-sensitive deconvolution networks with gated fusion for rgb-d indoor semantic segmentation[END_REF] proposed a gated fusion module to adaptively merge RGB and depth score maps according to their weighted contributions.

3.1.1.3/ HYBRID FUSION

Previous studies have shown that simply concatenating multimodal features or fusing weighted feature maps at decision level may not be sufficient to meet the requirements of highly accurate and robust segmentation. The hybrid fusion strategy is proposed to combine the strengths of early fusion and late fusion as an alternative method.

In the early stages of hybrid fusion, Park et al. [START_REF] Park | RDFNet: RGB-D multi-level residual feature fusion for indoor semantic segmentation[END_REF] extended the core idea of resid- Additionally, semantics-guided multi-level fusion [START_REF] Li | Semanticsguided multi-level RGB-D feature fusion for indoor semantic segmentation[END_REF], referred to as S-M Fusion, was proposed to learn the feature representation in a bottom-up manner. This fusion strategy employed the cascaded Semantics-guided Fusion Block (SFB) to fuse lower-level features across modalities sequentially.

Moreover, Jiang et al. [START_REF] Jiang | RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation[END_REF] described a residual encoder-decoder network for RGB-D semantic segmentation, named RedNet. The complementary features are fused into the RGB branch before upsampling. The skip-connection was used to bypass the spatial feature between the encoder and decoder. Instead of VGG, the residual module was applied as the basic building block. A more recent method addressed the issue of deep multimodal fusion using a Self-Supervised Model Adaptation module (SSMA) [START_REF] Valada | Self-supervised model adaptation for multimodal semantic segmentation[END_REF]. This Considering Bayes categorical fusion as an example, the unimodal classification output is used as index to the confusion matrix in order to produce the conditional class likelihoods.

The final prediction is therefore the class with the highest joint likelihood.

Combining multiple classifiers in a statistical way is not a new concept [START_REF] Xu | Methods of combining multiple classifiers and their applications to handwriting recognition[END_REF], but this work leads to an interesting research direction in the combination of deep learning and statistics.

3.1.2/ DISCUSSION

Deep multimodal fusion for scene understanding is an extremely complex issue that involves several factors, including the spatial location of objects, the semantic context of the scenes, the effectiveness of fusion models, the physical properties of the modalities, etc. The fusion strategies mentioned above follow different design concepts to tackle this challenges. Early fusion methods make an effort to optimally integrate information from multimodal sources during feature extraction. Namely, the representative features from complementary modalities are automatically fused to the RGB branch or a gated branch at the early stage, while features are reconstructed via a common decoder. These works emphasize the importance of cross-modal information interaction. Late fusion methods generally map multimodal features into a common space at the decision level. In other words, the fusion model is trained to learn unimodal features separately. Thus, late fusion may offer more flexibility and scalability but lacks sufficient cross-modal correlation.

Regarding hybrid fusion, such fusion strategy is elaborated to combine the strengths of early fusion and late fusion, achieving a more robust performance. However, the tradeoff between accuracy and execution time should be carefully considered in architectural design.

This brings us to two main questions:

• Multi-level fusion is helpful for segmentation model to learn representative features.

Fusing multimodal contextual information in multi-level stages represents the current trend. Moreover, semantic guiding across layers, such as skip connections, can be effectively used to bridge early feature extraction and late decision making.

The state-of-the-art method SSMA shows a typical example. Such fusion methods usually contain specific gating units that assign class-wise or modality-wise weights. One extreme case that should be noted is modality missing. Most of the existing deep fusion models can not work effectively when the supplementary modality is unavailable. Piasco et al. [START_REF] Piasco | Learning scene geometry for visual localization in challenging conditions[END_REF] offers some ideas based on learning with the privileged information paradigm to tackle this challenge. Otherwise, the trade-offs between accuracy/speed [START_REF] Huang | Speed/accuracy trade-offs for modern convolutional object detectors[END_REF][START_REF] Holliday | Speedup of deep learning ensembles for semantic segmentation using a model compression technique[END_REF][START_REF] Feng | Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges[END_REF] or memory/robustness should be carefully considered in the architectural design. In order to provide readers a more intuitive understanding, we show more detailed evaluations in Subsection 3.2.3.

3.2/ DATASETS

Over the last decade, a large number of datasets have been proposed to meet the needs of deep learning-based methods. The quantity and quality of training data significantly affect the performance of learning models. For this reason, many academic and research institutions have released several large-scale benchmark datasets for different scenarios. The creation of these well-annotated datasets actively promotes semantic scene understanding, which also facilitates the performance evaluation and inspires innovative approaches.

With the advent of multiple sensory modalities, numerous multimodal benchmark datasets have been released to the public successively. These datasets provide complementary properties of the same scene, such as geometric information, toward learning an improved feature representation. Figure 3.6 shows the accumulated dataset importance for image segmentation task since 2010. We observe that several large-scale datasets have emerged from 2015. Notably, PASCAL VOC 2012 [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] and Cityscapes [START_REF] Cordts | The Cityscapes Dataset for Semantic Urban Scene Understanding[END_REF] are two of the most popular datasets for semantic segmentation. As the representative RGB-D dataset, NYU-D [START_REF] Silberman | Indoor segmentation and support inference from rgbd images[END_REF] and SUN RGB-D [START_REF] Song | SUN RGB-D: A RGB-D scene understanding benchmark suite[END_REF] are frequently used for indoor scene understanding.

In the following parts, we provide a summary of current unimodal and multimodal datasets for semantic image segmentation. The aim is to grab the reader's interest in multimodal scene understanding and facilitate the preliminary experiments on deep multimodal segmentation.

3.2.1/ POPULAR DATASETS FOR IMAGE SEGMENTATION TASK

As one of the earliest pixel-wise labeled image databases, MSRC dataset [START_REF] Shotton | Textonboost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation[END_REF] was Furthermore, several outdoor road scene datasets are constantly emerging during the last decade, e.g. CamVid [START_REF] Gabriel | Segmentation and recognition using structure from motion point clouds[END_REF], KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF], Cityscapes [START_REF] Cordts | The Cityscapes Dataset for Semantic Urban Scene Understanding[END_REF], Mapillary Vistas [START_REF] Neuhold | The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes[END_REF], toward promoting the commercialization and advancement of autonomous driving technology [START_REF] Yin | When to use what data set for your self-driving car algorithm: An overview of publicly available driving datasets[END_REF]. To be specific, CamVid database is the first collection of fully segmented videos, captured from a moving vehicle. It provides over 700 manually labeled images of naturally complex driving scenes sampling from the video sequences. After that, KITTI Vision Benchmark was published to tackle various real-world computer vision problems, such as stereo, optical flow, visual odometry/SLAM, and 3D object detection. It consists of around 400 semantically annotated images recorded by RGB cameras, grayscale stereo cameras, and a 3D laser scanner.

released
During the past few years, Cityscapes dataset has been a strong performer in outdoor scene semantic segmentation. This high-quality dataset contains around five thousand high-resolution images with pixel-level annotations, recording the street scenes from 50 different cities. Also, Cityscapes is a superior multimodal segmentation dataset, containing precomputed depth maps of the same scenes. Besides, Mapillary Vistas dataset [START_REF] Neuhold | The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes[END_REF] provides 25,000 high-resolution images of street scenes captured from all over the world at various conditions regarding weather, season, and daytime. The images were annotated into 66 object categories, aiming to support the development of state-of-the-art methods for road scene understanding. More recently, for the sake of robustness and performance evaluation, WildDash [START_REF] Zendel | WildDash-creating hazard-aware benchmarks[END_REF] was released to the research community. This new benchmark provides standard data of driving scenarios under real-world conditions for a fair comparison of semantic segmentation algorithms. It is worth noting that RailSem19 [START_REF] Zendel | Railsem19: A dataset for semantic rail scene understanding[END_REF] is the first public outdoor scene dataset for semantic segmentation targeting the rail domain, which is useful for rail applications and road applications alike.

We present a summary of the reviewed segmentation datasets in

3.2.2/ MULTIMODAL DATASETS

Throughout the years, multimodal data are gaining the attention of researchers in various domains. The primary motivation for using multiple sensory modalities is to improve learning models' performance by enriching the feature representation. Table 3.5 lists numerous multimodal datasets reviewed in this survey, providing valuable information such as their application scenarios and data information. Next, we describe the potential multimodal datasets for image segmentation in detail, covering RGB-D datasets, Near

InfraRed datasets, thermal datasets, and polarization datasets. Multiple samples can be found in Table 3.6.

3.2.2.1/ RGB-D DATASETS

RGB-D cameras are widely used to augment the conventional color images with a depth map, which provides supplementary depth information about the distance of the object surface. Gupta et al. [START_REF] Gupta | Learning rich features from rgb-d images for object detection and segmentation[END_REF] proposed a method to encode horizontal disparity, height above ground, and the angle of the local surface normal into more efficient HHA images using raw depth images. Apart from semantic segmentation, depth information also makes significant contributions to other scene understanding tasks, such as object detection [START_REF] Gupta | Learning rich features from rgb-d images for object detection and segmentation[END_REF][START_REF] Eitel | Multimodal deep learning for robust RGB-D object recognition[END_REF] and pose estimation [START_REF] Schwarz | Rgb-d object recognition and pose estimation based on pre-trained convolutional neural network features[END_REF]. The first row in Table 3. In order to simulate different seasons, weather, and illumination conditions, several synthetic RGB-D datasets (e.g., SYNTHIA [START_REF] Ros | The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes[END_REF]) are generated for driving scenes semantic segmentation.

3.2.2.2/ NEAR-INFRARED DATASETS

Infrared imaging captured from multi-spectral cameras shows high contrast of natural and artificial objects [START_REF] Velte | Semantic image segmentation combining visible and nearinfrared channels with depth information[END_REF][START_REF] Jin | A survey of infrared and visual image fusion methods[END_REF]. In the computer vision field, multi-spectral images make up the data in the non-visible light spectrum and help better understand the scene characteristics. For example, Freiburg Forest dataset [START_REF] Valada | Deep Multispectral Semantic Scene Understanding of Forested Environments Using Multimodal Fusion[END_REF] Apart from semantic segmentation, multi-spectral images are also used in other computer vision tasks, including pedestrian detection [START_REF] Choi | KAIST Multi-Spectral Day/Night Data Set for Autonomous and Assisted Driving[END_REF][START_REF] Li | Illumination-aware faster rcnn for robust multispectral pedestrian detection[END_REF], face recognition [START_REF] Farokhi | Near infrared face recognition: A literature survey[END_REF], image dehazing [START_REF] Jang | Colour image dehazing using near-infrared fusion[END_REF][START_REF] Frederike D Ümbgen | Near-infrared fusion for photorealistic image dehazing[END_REF], video surveillance [START_REF] Benezeth | Background subtraction with multispectral video sequences[END_REF], to name a few.

3.2.2.3/ THERMAL DATASETS

Different from NIR images, thermal images are captured to recognize visible and invisible objects under various lighting conditions. The thermal imaging cameras are sensitive to all the objects that constantly emit thermal radiations [START_REF] Gade | Thermal cameras and applications: A survey[END_REF]. The wavelength is generally detected up to 14µm. In the early years, thermal imaging cameras were invented for military uses. With the cost of sensors decreasing, many scene understanding tasks can now benefit from thermal information [START_REF] Sun | RTFNet: RGB-Thermal Fusion Network for Semantic Segmentation of Urban Scenes[END_REF].

Tokyo Multi-Spectral [START_REF] Ha | MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[END_REF] is the first large-scale color-thermal dataset for urban scene seg-

3.2.2.4/ POLARIZATION DATASETS

As a universal phenomenon existing in natural scenes, polarimetric imaging is highly sensitive to the vibration pattern of the light [START_REF] Wolff | Polarization vision: A new sensory approach to image understanding[END_REF]. In the natural environment, the polarization of light is generally obtained by reflection or scattering. The polarization images carry crucial information of reflection surface related to object shape and surface material properties. To tackle practical problems in computer vision, polarization images have been widely applied to object detection [START_REF] Wolff | Constraining object features using a polarization reflectance model[END_REF], image dehazing [START_REF] Yoav | Instant dehazing of images using polarization[END_REF], depth estimation [START_REF] William | Linear depth estimation from an uncalibrated, monocular polarisation image[END_REF][START_REF] Zhu | Depth from a polarisation + rgb stereo pair[END_REF]. dangerous for autonomous systems. Currently, the use of polarimetric data leads to new directions for deep multimodal fusion research. The polarimetric imaging offers great potential [START_REF] Qiu | Polarization demosaicking for monochrome and color polarization focal plane arrays[END_REF][START_REF] Noah A Rubin | Matrix fourier optics enables a compact full-stokes polarization camera[END_REF] in scene understanding. For future perspectives, polarization cameras may be extremely valuable in autonomous driving [START_REF] Blanchon | Outdoor Scenes Pixel-wise Semantic Segmentation using Polarimetry and Fully Convolutional Network[END_REF][START_REF] Sun | A multimodal vision sensor for autonomous driving[END_REF] and robotics [START_REF] Yang | Predicting Polarization beyond Semantics for Wearable Robotics[END_REF][START_REF] Rastgoo | Attitude estimation from polarimetric cameras[END_REF].

3.2.2.5/ CRITICAL CHALLENGES FOR MULTIMODAL DATA

Based on the review of multimodal image datasets, we summarized four critical challenges for multimodal data:

• Data diversity: different image sensors offer different representative features of the scene according to their physical properties. The accuracy and robustness of deep fusion models are closely related to the amount and variety of multimodal data. In addition to the multimodal types mentioned above, more data types are expected for complex tasks in computer vision.

• Quantity and quality: in order to meet the needs of deep learning model training, what kind of multimodal data is easier to collect in practice?

3.2.3/ COMPARATIVE ANALYSIS

In this part, we report the evaluations of existing deep multimodal fusion methods on four benchmark datasets: SUN RGB-D [START_REF] Song | SUN RGB-D: A RGB-D scene understanding benchmark suite[END_REF], NYU Dv2 [START_REF] Silberman | Indoor segmentation and support inference from rgbd images[END_REF], Cityscapes [START_REF] Cordts | The Cityscapes Dataset for Semantic Urban Scene Understanding[END_REF], and Tokyo Multi-Spetral dataset [START_REF] Ha | MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[END_REF]. We also conduct a direct comparison of different unimodal and multimodal methods, aiming to demonstrate the necessity and importance of multimodal fusion approaches. All the results reported in this survey are collected from the original publications to ensure fairness.

3.2.3.1/ ACCURACY

We gathered quantitative results of the aforementioned fusion approaches from the corresponding papers and grouped them according to the benchmark datasets. The mean accuracy (%) and mean IoU (%) are the most reported metrics for a fair comparison. In

• NYU Depth v2 dataset

Regarding the NYU Depth v2 dataset, which is also a typical indoor scene dataset with high-quality depth information, we select six methods to make a detailed comparison. Table 3.8 demonstrates the experimental results with 13 and 40 classes.

ACNet is again the best performing method with a mean IoU score of 48.3% for 40 classes. Note that when the methods are evaluated on 13 classes only, the performances are higher because most challenging classes are not taken into account.

• Cityscapes dataset

Apart from the indoor scene datasets, we also show the segmentation results on a more challenging urban scene dataset, Cityscapes in Table 3.9. For this outdoor dataset, SSMA, as a typical hybrid fusion architecture, achieves the best performance with a mean IoU score of 83.94%. Moreover, we have observed that HHA representation provides more valuable properties than the original depth map. The multimodal fusion methods generally outperform the performance of the unimodal methods.

• Tokyo Multi-Spectral dataset

As shown in Table 3.10, we report the evaluation results on Tokyo Multi-Spectral dataset. Both visible spectral images and thermal images were used in the fusion experiments. We also collect 4-channel early fusion methods for comparative study.

The winner, RTFNet, achieves a maximum accuracy of 53.2% mean IoU. Notably, the segmentation accuracy is significantly increased by adding thermal infrared information. These results clearly show the effectiveness of multimodal data and the advancement of deep multimodal methods.

Based on the analysis of these results, we can draw some conclusions. First, depth information is the most commonly used supplementary information for multimodal image fusion. Most deep fusion methods report their results on the large-scale RGB-D datasets for both outdoor and indoor scene understanding. However, other types of multimodal datasets are of varying quality and lack further evaluation. The establishment of standard benchmark datasets is the premise of multimodal fusion study. Also, reported fusion methods employed various backbone networks, input size, and setups for the experiment, making fair performance comparisons difficult. Although many deep learning frameworks and libraries already exist, more multimodal toolkits are expected to facilitate multimodal fusion study. In light of the reported results, we have observed that ACNet and SSMA achieved remarkable results on the RGB-D datasets. A major reason is that these methods adopt many advanced deep learning techniques, such as attention mechanism, multiscale feature aggregation, and skip connection. It can be seen that the development of deep learning technology is of great benefit to multimodal fusion. Moreover, it is worth noting that most methods focus on accuracy, which does not allow for a comprehensive evaluation of fusion models. Multiple metrics can also reflect the effectiveness of multimodal data, which is instructive to the construction of the multimodal data collection platform. In general, deep multimodal fusion methods require higher memory footprint and execution time. We report more detailed results in the following subsections.

3.2.3.2/ EXECUTION TIME

In order to evaluate the real-time performance of deep multimodal fusion networks, we Most existing datasets and methods for outdoor scene semantic segmentation are mainly based on RGB camera. They are only well acceptable in general conditions excluding complex environment and small amount of samples. To develop additional practical solutions, one of the main challenges is data fusion from multi-modalities. Therefore, considering the RGB modality as a kind of imperfect sensor, we attempt to fuse the complementary feature information of the same scene from other modalities. Actually, several modalities are ubiquitous in robotic systems, such as RGB-D, LIDAR, near infrared sensor, etc.

In this work, we use a polarimetric camera, as a complementary modality, to provide a richer description of a scene. Polarization of light radiation has more general physical characteristic than intensity and color [START_REF] Wolff | Polarization vision: A new sensory approach to image understanding[END_REF]. We can figure out that windows of a building, the asphalt road, and the puddle of water have reflected polarizations [START_REF] Walraven | Polarization Imagery. Optical Polarimetry $[END_REF]. Plenty of research have demonstrated that the use of polarization camera can significantly enhance the capabilities of scene understanding, especially for reflective areas [START_REF] Harchanko | Water-surface object detection and classification using imaging polarimetry[END_REF].

Over the past few years, a variety of deep learning-based end-to-end approaches have been proposed. One factor that increased the popularity of deep learning is the availability of massive data. In the case without large amount of samples, we attempt to acquire more features of the same scene using several modalities. To some degree, an effective encoding of complementary information enables learning without the need for massive data, therefore the use of small-scale dataset can also lead to good performances. Recent works have shown promising results in extracting and fusing features from complementary modalities at pixel-level. The idea is to separately or jointly train the model using This fusion architecture, combining features before training, seems simple and light. However, it is also more likely to overfit. To see why, let consider the model's complexity. Let H be a family of functions taking values in {-1, +1} with VC-dimensions d vc [START_REF] Vapnik | Statistical Learning Theory[END_REF]. Then, for any δ > 0 and all h ∈ H, the VC-dimension bound [START_REF] Mohri | Foundations of machine learning[END_REF] can be derived with a high probability:

E out (h) ≤ E in (h) + 8 N ln( 4(2N) d vc δ ), (4.1)
where E out denotes out-of-sample error, E in denotes in-sample error, and N denotes the data points that the hypothesis space can shatter the set. As the amount of input's dimensions increases, so does the VC-dimensions. Then the model complexity Ω(N, H, δ)

rises along with the increase of VC-dimensions. As a result, larger data samples should be fed to fit the deep neural model for less in-sample error. In other words, in the case that samples are not huge enough, the model may be easier to overfit. This approach has the advantages that each network computes weights separately while encoding. Compared with early fusion, to some extent, it may reduce the difficulty of model fitting and yield a better outcomes. Furthermore, thanks to the scalability and flexibility of this architecture, the model can be designed in accordance with requirements and easily extend to multi-inputs without a large dimension increase. fusion module can be written as:

4.1.2.2/ LATE FUSION

ŷn = f (X n ) = softmax[W * (M 1 (x a ) + M 2 (x b ))], (4.2)
where, W is a series of convolution kernels for upsampling. The softmax function is introduced to represent the categorical distribution, and is defined as:

softmax(z) j = e z j K k=1 e z k , (4.3)
where z = [z 1 , . . . , z K ] T .

Moreover, the process of model training is to minimize the error while regularizing the parameters. It can be framed as an optimization one, which can be formulated as:

θ * = arg min θ 1 N N i=1 L(y n , f (x n ; θ)), (4.4)
where the loss is computed as:

L(u, y) = - k y k logŷ. (4.5)
Then we can use gradient descent algorithm to find local minimum. On the one hand, the branch for RGB modality incorporates a SegNet-like encoder. By copying the indices from max-pooling, it can capture and store boundary information in the encoder feature maps before sub-sampling. We keep this strength to make the network more memory efficient and improve boundary delineation. On the other hand, we focus on the feature quality of the extra modality. Other modalities can provide rich complementary information on low level appearance features.

However, how to captures rich contextual information from extra modality is a challenging task. We refer to the state-of-the-art segmentation network Deeplab v3+ , which uses a new pooling method named ASPP (Atrous Spatial Pyramid Pooling) to incorporate the multi-scale contextual information. We experimentally apply this network structure as the other branch's encoder for the complementary modality and achieved improved results.

The first upsampling stage is subsequently applied to each branch to recover the feature representation to the same fusion size, then we fuse these three feature maps, which contains high-level and low-level multimodal features information simultaneously. The second upsampling stage and softmax are applied to the fused feature map, which produces the final results. We can find the nature of the object's roughness, orientation and the reflection in each pixel of polarimetric images [START_REF] Lawrence | Polarization camera sensors[END_REF].

The electrical field of a progressive transverse wave in its propagation plan [START_REF] Blin | A new multimodal rgb and polarimetric image dataset for road scenes analysis[END_REF] can be defined as:

E(t) = E x (t)cos(-k z + ωt) u x + E y (t)sin(-k z + ωt + φ(t)) u y , (4.6)
where k is the wave number, z denotes the direction of propagation. ω and φ are a pulsation and a phase in the orthonormal basis B = {u x , u y }. E x and E y are the amplitudes of E(t) according to u x and u y , respectively.

In nature, light determined by scattering or reflection is generally unpolarized or partially linearly polarized and can be described by the linear Stokes vector, S = [S 0 S 1 S 2 ] .

Using the parameters in Equation 4.6, it can be rewritten as: images [START_REF] Blanchon | Outdoor Scenes Pixel-wise Semantic Segmentation using Polarimetry and Fully Convolutional Network[END_REF]. The benchmark dataset contains 175 images with pixel-wise ground truth annotation.

S =               S 0 S 1 S 2               =               E 2 x + E 2 y E 2 x -E 2 y 2 E x E y cos(φ)               . ( 4
Moreover, the images have been semantically dispatched into 8 classes: unlabeled, sky, water, windows, road, car, buildings and others. Benefiting from the use of a polarimetric camera, our mobile robot platform is more capable of discerning on windows, water and other reflective areas. That facilitates exploratory research on the use of polarimetric cameras in semantic scene understanding domain. In this thesis, we mainly employ aligned RGB and polarimetric images as inputs to train and evaluate the deep fusion models. For integrating the acquired images, we apply an automatic homographic method to image alignment [START_REF] Moisan | Automatic homographic registration of a pair of images, with a contrario elimination of outliers[END_REF]. This method allows to transform the RGB images with respect to the polarimetric images, and crop to the intersecting regions of interest. Moreover, was proposed to improve sensitivity to high biomass regions and vegetation monitoring.

It shows stronger capacities on feature representation than NIR in the previous work.

To extract more accurate information, here in our case, we select EVI images as the second modality input besides the visible input. We crop the RGB and EVI images as size 3×256×256, and use them as inputs correspondingly. We report several metrics to assess segmentation models: pixel accuracy (PA), mean accuracy (MA), mean intersection over union (MIoU), frequency weighted IoU (FWIoU). These metrics are defined in Section 2.5.

The results shown in Table 4.1 show that segmentation using RGB images yields better results than EVI images on the whole. This shows that RGB images provide better highlevel features while training. For fusion architectures, late fusion methods outperform channel fusion method as we analyzed in the previous section. Our network yields around 1% ∼ 2% comprehensive improvements comparing with other methods.

Furthermore, the results in Table 4.2 demonstrate the evaluations by class. We report the main four classes as Road, Grass, Veg/Tree and Sky. For uni-modality network, we can find that EVI shows good performance on Road and Grass classes, and RGB modality has a significant advantage on Sky class, which is susceptible to lighting changes. Moreover, the fusion architecture outperforms uni-modality scheme by integrating complemen- We process the RGB and polarimetric images with size 3×448×448. While training the networks, we experimentally found that stochastic gradient descent (batch size=1) does not work well. It is reasonable that online learning adds too much instability to the learning process as the weights widely vary with each batch, especially for small scale dataset with

4.2/ A CENTRAL MULTIMODAL FUSION FRAMEWORK 4.2.1/ INTRODUCTION

Robust and reliable scene understanding of robotic systems in the real world has been among the most challenging tasks in computer vision. A wealth of research on deep learning focuses on image segmentation of outdoor road scenes, especially the perception of autonomous vehicles. However, robotic systems still exhibit a limited ability to the semantic understanding of complex environments because they may be negatively affected in challenging situations such as varying illumination conditions or seasons changes. Specific sensors display much better performance than standard ones in some scenarios. For instance, numerous studies have shown that the use of a polarization camera delivers outstanding performance on semantic scene understanding in reflective areas [START_REF] Harchanko | Water-surface object detection and classification using imaging polarimetry[END_REF][START_REF] Blanchon | Outdoor Scenes Pixel-wise Semantic Segmentation using Polarimetry and Fully Convolutional Network[END_REF]. Over the past few years, academia and industry have expressed a significant interest in multimodal data acquisition systems and analysis methods. Previous works [START_REF] Gupta | Learning rich features from rgb-d images for object detection and segmentation[END_REF][START_REF] Hazirbas | Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture[END_REF][START_REF] Valada | Deep multispectral semantic scene understanding of forested environments using multimodal fusion[END_REF][START_REF] Zhang | Exploration of Deep Learning-based Multimodal Fusion for Semantic Road Scene Segmentation[END_REF] experimentally demonstrate that qualified multimodal fusion models yield better results than the unimodal ones, due to richer information representation of scenes provided by complementary modalities.

State-of-the-art segmentation models have demonstrated remarkable pixel-level classification results. However, most of the existing methods focus on segmentation with RGB or 3D data. With the advent of low-cost RGB-depth and multi-spectral cameras, there is

4.2.2.2/ ADAPTIVE CENTRAL FUSION NETWORK

The main idea of CMFnet is to project the multimodal information into a common feature space. By integrating the feature maps in a succession of layers into the central branch, the fusion network can sequentially learn the joint feature representation. Contrary to the typical structure of early and late fusion, such central fusion can automatically identify which are the best levels for feature integration and how these feature maps should be combined [START_REF] Vielzeuf | Central-Net: a Multilayer Approach for Multimodal Fusion[END_REF]. The overall network architecture is illustrated in Figure 4.9.

Suppose that we have two modalities x i and x j , each modality goes through a series of convolutional layers to produce different feature maps. Let l k i and l k j be the feature maps obtaining at layer k from each modality. To evaluate the value of feature maps and obtain a joint representation, we employ an adaptive gating unit (AGU) to assign the weights. This unit is the stack of a concatenating layer, a convolutional layer with the ReLu activation function, and a global average pooling layer followed by a softmax layer. The global average pooling layer (GAP) [START_REF] Lin | Network in network. 2nd International Conference on Learning Representations, ICLR 2014 -Conference Track Proceedings[END_REF] is applied to compute the spatial average of feature maps from the previous convolutional layer. It is used to prevent overfitting and enforce the correspondence between feature maps and weights [START_REF] Lin | Network in network. 2nd International Conference on Learning Representations, ICLR 2014 -Conference Track Proceedings[END_REF]. The output of AGU is a set of two weights w k i and w k j , therefore the first central layer can be defined as

F 1 = [ l1 i ∨ l1 j ] (4.10)
where l1 i = w 1 i l 1 i and ∨ denotes the tensor concatenation. This operation is repeated for the different layers of the central fusion network, each time concatenating the weighted feature maps of layer k with the output of the previous layer:

F k = [F k-1 ∨ lk i ∨ lk j ] (4.11)
In particular, considering the size of the last central layer is N × W × H, we compute a 1 × W × H feature map for each central layer in the encoder part. The feature maps containing multi-scale spatial details are integrated into the last central layer for further refinement. Finally, we add a softmax layer for loss computation.

4.2.2.3/ STATISTICAL PRIOR FUSION

To further exploit the multimodal prior probability distribution, we aim to statistically combine the output of the central fusion network with unimodal baselines. Similar work can be found in [START_REF] Blum | Modular sensor fusion for semantic segmentation[END_REF]. In a series of experiments, we have seen significant improvements in the accuracy of certain categories. The statistical methods are proved to be effective in dealing with model uncertainty. Especially for deep learning-based multimodal fusion, multiple modalities bring more model uncertainty in decision making. Different from the previous work, we employ Bayesian fusion as post-processing to statistically integrate the outputs of the optimal unimodal baseline and CMFnet. This fine-tuning process, to some extent, can optimize the pixel-wise prediction and achieve robust performance. with the output of CMFnet, which referred to as O c . The prediction probability for each possible class k ∈ 1, ..., K can be defined as p(k|{O c , O s }). For every feature, the class with the highest probability is chosen as the final prediction. By maximizing the probability p(k|{O c , O s }), we can obtain optimized segmentation results. The statistical fusion process can be described as:

k * = arg max k p(k|{O c , O s }) = arg max k p(k)p(O c |k)p(O s |k) = arg max k [log p(k) + log p(O c |k) + log p(O s |k)] (4.12)
where p(O c |k) denotes the categorical distribution over the output of CMFnet, p(O s |k) is the categorical distribution over the optimal baseline output, and p(k) is the prior probability of class k.

Note that given the class k, we consider O c and O s are conditional independent because they were trained separately. Every term in the Equation (4.12) can be obtained from the confusion matrix of O c and O s , which referred to as M c and M s . In more detail, if the first dimension of the confusion matrix denotes the actual output (i.e., O c or O s ) and the second dimension is the ground truth class k, the conditional probabilities can be computed as:

p(O c |k) = M c [O c ][k] K j=1 M c [O j ][k] (4.13) p(O s |k) = M s [O s ][k] K j=1 M s [O j ][k] (4.14)
Based on the Equation (4.13)(4.14), we can obtain the final segmentation prediction.

4.2.3/ EXPERIMENTS

4.2.3.1/ IMPLEMENTATION DETAILS

We conduct experiments on two public outdoor road scene datasets: POLABOT [START_REF] Blanchon | Outdoor Scenes Pixel-wise Semantic Segmentation using Polarimetry and Fully Convolutional Network[END_REF][START_REF] Zhang | Exploration of Deep Learning-based Multimodal Fusion for Semantic Road Scene Segmentation[END_REF] and Cityscapes [START_REF] Cordts | The Cityscapes Dataset for Semantic Urban Scene Understanding[END_REF]. In order to fully evaluate the effectiveness and robustness of multimodal fusion models, we apply different types of image inputs for training and testing, including RGB, depth, and polarimetric images. We use the VGG-16 model [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] pretrained on ImageNet [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF] as backbone. All the methods are implemented using Pytorch [START_REF] Paszke | Automatic differentiation in PyTorch[END_REF] deep learning library on dual Nvidia TITAN Xp GPU (12GB memory). We choose the batch size according to the computing power of hardware and the learning rate is initialized to 1e-4.

In this work, we employ SegNet [START_REF] Badrinarayanan | SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation[END_REF] and ENet [START_REF] Paszke | ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation[END_REF] as unimodal baseline network. Additionally, we compare the proposed framework with several fusion approaches based on the corresponding baseline network, including average, LFC [START_REF] Valada | Deep multispectral semantic scene understanding of forested environments using multimodal fusion[END_REF], and CMoDE [START_REF] Valada | Adapnet: Adaptive semantic segmentation in adverse environmental conditions[END_REF].

The simple averaging fuses the RGB and depth baseline by taking the mean over the outputs of the last convolutional layer, followed by a shared softmax layer, while LFC and CMoDE are two competitive late fusion approaches. Specifically, we perform the ablation studies on the proposed central fusion method:

• CMFnet: the central multimodal fusion without post-processing, i.e., the output of the central fusion network is the final segmentation result.

• CMFnet+BF2: the central multimodal fusion with Bayesian fusion, i.e., the output of CMFnet is statistically fused with the output of the optimal unimodal baseline.

• CMFnet+BF3: an extension of CMFnet+BF2, where the CMFnet is fused with all available unimodal baselines. We present a detailed comparison of different fusion approaches on the POLABOT [START_REF] Zhang | Exploration of Deep Learning-based Multimodal Fusion for Semantic Road Scene Segmentation[END_REF] dataset. This dataset dedicates to polarimetric imaging, containing synchronized RGB and polarimetric images. It provides 175 aligned image pairs with ground truth annotation and focuses on the segmentation of reflective areas such as windows and water. In order to enhance the generalization ability of models, we randomly apply a series of data augmentation while training [START_REF] Blanchon | Outdoor Scenes Pixel-wise Semantic Segmentation using Polarimetry and Fully Convolutional Network[END_REF], including rotation and flipping. In this experiment, we adopt SegNet as the unimodal baseline and the backbone of fusion methods. The segmentation models were trained with batch size 4 at a resolution of 448 × 448.

The experimental results are shown in Table 4.4. We report the mean IoU as the primary metric to evaluate the overall performance of different segmentation methods. Firstly, we observe that the segmentation results of the polarimetric baseline are better than the RGB ones. This is particularly the case for the categories Cars, Windows, and Water.

Therefore in post-processing, we adopt the polarimetric baseline to provide complementary statistical information for CMFnet+BF2. Regarding these fusion methods, we can observe that the central fusion method without statistical fusion, CMFnet, is slightly better than other fusion approaches, such as simple average, LFC and CMoDE, but the improvement is marginal. With statistical fusion, the proposed framework CMFnet+BF2 achieves excellent performance with a mean IoU of 86.62%, a relative improvement of +3.93% and +2.97% over the RGB and polarimetric baseline, respectively. Compared with CMFnet, we find that the statistical fusion module leads to a relative improvement of +2.47% overall. Furthermore, CMFnet+BF2 performs reliably for most of the categories, especially for the category Water, Windows, Cars and Buildings. The results demonstrate that our fusion framework effectively exploits RGB and polarimetric information, yielding better segmentation accuracy than the previous fusion methods.

Furthermore, our central fusion strategy speeds up the convergence while training (see Figure 4.10), which also reduces the time cost of model training to some extent. Figure 4.12 shows the experimental results for two pairs of RGB and polarimetric images from the POLABOT dataset.

4.2.3.3/ EVALUATION ON CITYSCAPES DATASET

In this subsection, we adopt ENet as the baseline network and further evaluate the segmentation performance of the proposed fusion framework on the Cityscapes dataset.

Cityscapes is a standard urban street scenes benchmark dataset that contains RGB, disparity images, and pixel-wise semantic segmentation annotation. Due to the limitation of computing memory, we resize the full input images to the resolution of 768 × 384 and use the stochastic gradient descent method with one batch size for training. We group Table 4.5 shows the overall performance of the unimodal and multimodal fusion approaches. Our proposed fusion framework, CMFnet+BF2, yields a mean IoU of 58.97%, which constitutes an improvement of +4.80% over the best unimodal baselines, i.e., RGB.

It also outperforms other fusion approaches by around +2%. We can also observe that fusing all the unimodal networks, i.e., CMFnet+BF3, does not exhibit superior performance due to the poor performance of the depth baseline.

4.3/ SUMMARY

In this chapter, we first explored the typical early fusion and late fusion architectures that extract features from multi-modalities, and extensively evaluated theirs merits and deficiencies. We also proposed an extensible late fusion scheme for outdoor road scene semantic segmentation. It provides design choices for future research directions. We presented comprehensive quantitative evaluations of multimodal fusion on the two smallscale benchmark datasets. In addition, we introduced a first-of-a-kind outdoor scene segmentation dataset for road scene navigation, which contains high-quality aligned polarimetric images. We empirically demonstrate that the use of polarization camera enhance

5.1/ INTRODUCTION ON FEW-SHOT SEGMENTATION

This section summarizes the existing methods for few-shot semantic segmentation. Many approaches for few-shot learning are proposed to generalize prior knowledge to new tasks using only a few examples. Some research [START_REF] Vinyals | Matching networks for one shot learning[END_REF][START_REF] Li | Revisiting metric learning for few-shot image classification[END_REF] introduced the metric learningbased matching network for the few-shot classification task. The non-parametric structure facilitates the generalization of models to new training sets. Snell et al. [START_REF] Snell | Prototypical networks for few-shot learning[END_REF] presented a method to represent the prototypes per class in a representation space, known as Prototypical Networks. Moreover, several studies such as [START_REF] Garcia | Few-shot learning with graph neural networks[END_REF] have focused on the graphbased methods for few-shot learning.

For segmentation tasks, few-shot semantic segmentation refers to the pixel-level prediction of new categories on the query set, given only a few labeled support images. For example, Shaban et al. [START_REF] Shaban | One-shot learning for semantic segmentation[END_REF] first presents a dual branch parallel network for one-shot segmentation, known as OSLSM, including a conditioning branch and a segmentation branch. The conditioning branch extracts representative high-level features from the supporting image-label pair, whilst the segmentation branch integrates the parameters learned from the conditioning branch and performs a segmentation mask on the query image. Other variants of OSLSM include Co-FCN [START_REF] Rakelly | Conditional networks for few-shot semantic segmentation[END_REF], PL+SEG [START_REF] Dong | Few-shot semantic segmentation with prototype learning[END_REF] and MDL [START_REF] Dong | Multi-scale discriminative location-aware network for few-shot semantic segmentation[END_REF]. All of which extend such dual branch structure to achieve a substantial performance improvement. In the AMP model, Siam et al. [START_REF] Siam | Adaptive masked proxies for few-shot segmentation[END_REF] replaces the guidance branch with a multiresolution weight. Moreover, SG-One [START_REF] Zhang | Sg-one: Similarity guidance network for one-shot semantic segmentation[END_REF] proposed a Masked Average Pooling block (MAP) to extract the representative vectors of support objects. Then the segmentation mask was predicted via a similarity guidance network. More recently, Wang et al. [START_REF] Wang | Panet: Few-shot image semantic segmentation with prototype alignment[END_REF] presents a novel prototype alignment network, called PANet, based on non-parametric metric learning. and challenging scenarios, such as robotics [START_REF] Duan | One-shot imitation learning[END_REF], natural language processing [START_REF] Vinyals | Matching networks for one shot learning[END_REF][START_REF] Kaiser | Learning to remember rare events[END_REF],

5.2/ MAPNET

and drug discovery in medical applications [START_REF] Altae-Tran | Low data drug discovery with one-shot learning[END_REF]. Recently, semi-supervised learning [START_REF] Fei-Fei | One-shot learning of object categories[END_REF][START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF][START_REF] Ravi | Optimization as a model for few-shot learning[END_REF] has emerged as a hot topic in the computer vision community. Contrary to leveraging a large amount of data, few-shot learning aims to recognize new categories under limited supervision. Especially for few-shot segmentation task, the trained model Existing methods generally address this problem by learning a set of parameters or prototypes from support images and guiding the pixel-wise segmentation on the query image.

However, most of the previous studies do not explore the support information sufficiently, which is not taking enough advantage of potential semantic information of support images. Usually, they only consider a simple connection between the support set and query set (e.g., cosine similarity), which is adverse to the generalizability.

For the above reasons, we propose a novel few-shot segmentation network called multiscale attention-based prototypical network (MAPnet). To fully exploit the representative features from labeled support images, our method extracts rich contextual information via a multiscale feature enhancement module. This module consists of three elaborated branches that aggregate multiscale features of target classes. Multiple learned prototypes provide further similarity-based guidance on the query feature, containing multiscale feature attention. Then we employ the attention mechanism to adaptively weight the probability maps for the final mask prediction. We find that this method effectively strengthens the segmentation model's generalizability, especially for the 5-shot setting.

Moreover, the use of attention-based gating accelerates the convergence to a lower loss.

The network was trained in an end-to-end manner without any post-processing steps. Both the training and test sets contain several episodes that consist of a set of labeled support images S = (x i s , y(l) i s ) K i=1 and a set of query images Q = (x q , y q (l)) where l is the semantic class. The support set comprises K labeled examples for each of the N classes, which defines a N-way-K-shot segmentation. During training, episodes = (S , Q) randomly sampled from D train are used to perform segmentation on the query set, namely ŷq = (S , x q ). The performance is measured by a loss function loss( ŷq , y q ) where y q is the corresponding segmentation mask. Therefore, the optimal parameters of few-shot segmentation model are θ * = arg min θ loss( ŷq , y q ). While testing, the model is given a set of labeled support images sampled from D test . Then the few-shot segmentation model is expected to predict the segmentation mask on the query image for the relative class. By taking advantage of prior knowledge, the model can rapidly generalize for a new class of limited supervised information.

5.2.3/ METHOD

The proposed MAPnet, as shown in Figure 5.2, is based on the prototypical network.

The prototype learning-based methods enable the network to learn a set of feature vectors with adequate discriminative information. In the early work of [START_REF] Snell | Prototypical networks for few-shot learning[END_REF], researchers proposed a prototypical network that learns a common metric space. Few-shot classification can be achieved by computing distances to prototype representations of each class. However, such methods do not explore potential semantic information of support images in sufficient depth. The learned prototypes provide limited semantic guidance on the query feature, constraining the segmentation model's generalizability. Therefore we adopt the idea of prototype learning and introduce a novel few-shot segmentation method with multiscale feature attention.

In general, our method contains a feature extractor, a multiscale feature enhancement module, and an attention-based gating (see Figure 5.2). The support images and query images are embedded into a high-level feature space via a shared feature extractor. Concerning the practical implementation, we employ the first five convolutional blocks of VGG-16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] as the backbone network. The convolutions in the fifth convolutional blocks are replaced by atrous convolutions with a rate of 2. Besides, we retain the third convolutional block's output as intermediate features for further multiscale feature enhancement.

* Multiscale Feature Enhancement Module

Generally, each category that appeared in the input images differs in shape and size. The uniscale filters learned by the neural network may lead to many restrictions on the similarity-guided semantic guidance. Thus, we define a multiscale feature enhancement module (MFE module) to provide multiscale feature supervision.

In consideration of the trade-off between high performance and computational cost, In order to enhance the discriminative power of the model, we leverage both foreground and background information of support images, known as y(l)(+, -), to extract the representative prototypes of target classes l. The background information provides complementary clues for semantic understanding. Masked Average Pool-ing [START_REF] Zhang | Sg-one: Similarity guidance network for one-shot semantic segmentation[END_REF] is used in the process to compute these feature vectors. The set of feature vectors can be defined as V = {v 0 (+, -), v 1 (+, -), ..., v n (+, -)} where v n (+, -) is the n th pair of support feature vectors. Each pair of prototypes is used to generate the corresponding probability map with multiscale feature attention.

* Attention-Based Gating

Different from existing few-shot segmentation methods, our model produces a set of similarity-guided probability maps by estimating the distance between a series of representative prototypes and the high-level query features. Following the work in [START_REF] Zhang | Sg-one: Similarity guidance network for one-shot semantic segmentation[END_REF][START_REF] Wang | Panet: Few-shot image semantic segmentation with prototype alignment[END_REF], we employ the cosine similarity as the non-parametric nearest neighbor classifier. Thus we employ the attention-based gating block as a combination strategy to generate an optimal mask prediction. Suppose that the concatenation of probability maps is P, the attention score g can be defined as g = so f tmax(w * P),

where w denotes a convolutional layer and global average pooling layer (GAP) [START_REF] Lin | Network in network. 2nd International Conference on Learning Representations, ICLR 2014 -Conference Track Proceedings[END_REF].

Then our network learns the convolutional kernels ρ over the fused probability map.

More formally,

ŷq = so f tmax[ρ * I i (g i • p i )] (5.1)
where g i and p i denote the i th attention score and probability map, respectively.

5.2.4/ EXPERIMENTS

5.2.4.1/ SETUP

Dataset We evaluate the proposed method on the PASCAL-5 i dataset, which derives from PASCAL VOC 2012 [START_REF] Everingham | The PAS-CAL Visual Object Classes Challenge 2012[END_REF] with SBD [START_REF] Hariharan | Semantic contours from inverse detectors[END_REF] augmentation. This dataset was firstly created by Shaban et al. [START_REF] Shaban | One-shot learning for semantic segmentation[END_REF], then widely used in the few-shot segmentation task. Similar to the setup of OSLSM [START_REF] Shaban | One-shot learning for semantic segmentation[END_REF], we sample 5 classes out of all 20 categories as test labelset L test = {5i + 1, ..., 5i + 5} with i being the folder number. The remaining 15 classes form the train label-set L train . As shown in Table 5.1, our model is trained on three splits, then tested on the rest one in a cross-validation manner. In this work, we evaluate the performance of our model on 1,000 randomly sampled episodes for each folder.

Implementation details

We conduct the experiments with implementations in Py-Torch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF]. The backbone network (i.e., VGG-16) was initialized with pre-trained weights on ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. We resized the input images to 320 × 320 with random horizontal flipping. All the few-shot segmentation models were trained on a single Nvidia TITAN Xp GPU with 12GB memory, using stochastic gradient descent (SGD) with a batch size of 1, a momentum of 0.9, and weight decay of 0.0005 for a maximum of 40,000 iterations. The

MAPNET: A MULTISCALE ATTENTION-BASED PROTOTYPICAL NETWORK 75

Table 5.1: Training and evaluation on PASCAL-5 i dataset using 4-fold cross-validation, where i denotes the number of subsets.

Dataset

Test classes Pascal-5 0 aeroplane, bicycle, bird, boat, bottle Pascal-5 1 bus, car, cat, chair, cow Pascal-5 2 diningtable, dog, horse, motorbike, person Pascal-5 3 potted plant, sheep, sofa, train, tv/monitor initial learning rate was set to 1e-3 and reduced by 0.1 every 10,000 iterations.

Evaluation metrics Following the previous works on the few-shot segmentation [START_REF] Shaban | One-shot learning for semantic segmentation[END_REF][START_REF] Zhang | Sg-one: Similarity guidance network for one-shot semantic segmentation[END_REF][START_REF] Wang | Panet: Few-shot image semantic segmentation with prototype alignment[END_REF], we apply two standard metrics to evaluate the performance of learning models: mean-IoU and binary-IoU. Generally, the mean Intersection-over-Union (mean-IoU) is used to measure each foreground class's accuracy and average over all the categories. Binary-IoU deals uniformly with all object categories as one foreground class and averages the IoU of both foreground and background. Based on these two metrics, we can fairly compare the accuracy in terms of 1-way N-shot semantic segmentation.

5.2.4.2/ EVALUATION

In this part, we report the experimental evaluations on the benchmark dataset. Table 5. 2 shows the comparison result of our method MAPnet and other previous methods in terms of 1-way 1-shot and 1-way 5-shot segmentation. We observe that our model achieves 48.2% on the whole for the 1-way 1-shot setting, which substantially outperforms the baseline network OSLSM by +7.4%. Also, the performance of MAPnet is competitive to the state-of-the-art method PANet. Our model earns the largest gain of +1.4% on PASCAL-5 3 compared to PANet, where the test classes are potted plant, sheep, sofa, train and tv/monitor. Our method yields a mean IoU of 56.0% overall with five support images, which achieves significant improvement over other baseline networks. Compared to SG-One, which has a simple but effective prototypical structure, we can see that the proposed method leads to a relative improvement of +9.7% on PASCAL-5 0 , +6.5% on PASCAL-5 1 , +9.8% on PASCAL-5 2 , and +9.4% on PASCAL-5 3 .

Besides, we report the averaged binary-IoU on the four-fold cross-validation in Table 5.3.

Our method shows remarkable improvement in 1-way 5-shot, which gains an increment of 5.1% comparing to 1-way 1-shot. The main reason behind the increase of accuracy is that our multiscale feature enhancement module provides richer contextual information for the semantic guidance of target classes. Namely, the attention-based multiscale feature aggregation becomes more prominent as the number of support images increases. Moreover, we replace the attention-based gating with element-wise addition, aiming to Table 5.3: Results of 1-way 1-shot and 1-way 5-shot segmentation on PASCAL-5 i using binary-IoU(%) metric. ∆ denotes the difference between 1-shot and 5-shot. compare the convergence speed of our method trained with and without multiscale attention. As shown in Figure 5.4, we observe that the attention-based gating speeds up the convergence and reaches a lower loss, which also earns a 3.5% gain in accuracy.

Furthermore, we demonstrate qualitative results on PASCAL-5 i in Figure 5.3. We present different cases involving outdoor scenes and indoor scenes. These examples show the high discriminatory power and generalizability of our method. It is capable of extracting sufficient contextual information of the target class, and a challenging case is shown in Figure 5.8 row 4. We also present two typical failure cases in our experiments. Based on the observation of the first failure case, we find that it is not easy to distinguish the objects with similar characteristics, especially when these objects are placed in an overlapping manner. Another failure case shows that the model has a limited capacity to recognize irregular objects and their boundary delineation. These failure cases may be challenging issues in future work.

5.2.4.3/ TEST WITH WEAK ANNOTATIONS

To further validate the generalization ability of our model, we report the experimental results on different weak annotations in Table 5.4, including scribbles and bounding box annotations. Different from tedious and inefficient per-pixel annotating, these weak annotations are frequently used in interactive image segmentation [START_REF] Lin | Scribblesup: Scribble-supervised convolutional networks for semantic segmentation[END_REF]. In our experiments, This work also promotes the use of multimodal data in the few-shot learning field.

5.3.2/ PROBLEM SETTING

In this work, few-shot semantic segmentation involves three datasets: a training set D train , a support set D s , and a query set D q . The segmentation model is trained on D train , and evaluated on D s and D q . Moreover, we adopt the training and testing protocols in Suppose the set of semantic classes in D train is C seen . We assume that the set of classes at test time, C unseen , does not overlap with C seen , i.e. C seen ∩ C unseen = ∅. We formally define these datasets in the following lines:

• D train = (x R i , x D i , y(l) i ) N i=1
, where x R i is a color image, x D i is a depth image of the same scene, y(l) i denotes the corresponding segmentation mask of class l (l ∈ C seen ), and

• D s = (x R j , x D j , y(l) j ) M j=1
, where x R j and x D j denote the corresponding RGB and depth image, y(l) j is the mask for the semantic class l (l ∈ C unseen ), and M indicates the number of labeled samples given in the test phase.

• D q = (x R j , x D j ) n j=1
is the query set of n pairs of RGB and depth images. Evaluations on D q show the relative performance of the models.

The goal of few-shot segmentation is to train a model f with high discriminative power and generalizability from D train , then produces a segmentation prediction ŷq on D q given a support set D s . The performance is measured by a loss function l( ŷq , y q ), where y q is the corresponding annotation. The optimal parameters of few-shot segmentation model are θ * = arg min θ l( ŷq , y q ). Usually, if the the support set consists of K labeled samples for each of C semantic classes, we consider such few-shot learning problem as C-way K-shot segmentation task.

5.3.3/ METHOD

The main motivation of our work is to facilitate the few-shot segmentation task by incorporating complementary depth information. Existing supervised semantic segmentation approaches for RGB-D data do not offer a satisfactory solution to learn new categories rapidly from limited data. For this reason, we employ ideas from previous work of nonparametric metric learning and propose a two-stream deep neural network (RDNet). The main novelty of this study is to separately learn the RGB and depth prototype representations in different embedding spaces. The learned prototypes are applied to the corresponding query features as semantic guidance. Then we integrate the results from these two streams for an improved segmentation performance.

• RGB-D input: As shown in Figure 5.7, the proposed RDNet consists of two mirrored prototypical networks, which process RGB and depth input separately. Note that the support and query set through the depth stream provide the same scene information as the RGB Stream. Then the support images are embedded into highlevel abstract features via a base network. For efficient implementation, we adopt a VGG-16 as the backbone network following the setup in [START_REF] Wang | Panet: Few-shot image semantic segmentation with prototype alignment[END_REF]. In this way, we can map RGB and depth data into different embedding spaces.

• Prototype learning for RGB-D data: Snell et al. [START_REF] Snell | Prototypical networks for few-shot learning[END_REF] proposed a prototypical network that learns a common metric space. Few-shot classification can be achieved by computing distances to prototype representations of each class. We employ the Masked Average Pooling [START_REF] Zhang | Sg-one: Similarity guidance network for one-shot semantic segmentation[END_REF] to build pre-class prototypes from both foreground and background information of the support images. Given a support set

D s = (x R j , x D j , y(l) j ) M j=1
, let F(l) j be the output feature maps of the base network with support RGB or depth input. Then F(l) j denotes the resized feature maps, which have the same width w and height h as the semantic mask y(l) j ∈ {0, 1} W×H . The prototype of target class l can then defined via Masked Average Pooling by the following equation: • Similarity guidance and feature fusion: We compare the abstract query feature with expressive prototypes using distance metric learning method. To be specific, we map the query feature vector into the corresponding embedding space. The computed cosine distance indicates the similarity of target class. Besides, according to the previous work in fully-supervised semantic segmentation with RGB-D data [START_REF] Hazirbas | Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture[END_REF][START_REF] Deng | RFB-Net: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation[END_REF] , there are two main fusion stategies, i.e., early fusion and late fusion [START_REF] Baltrušaitis | Multimodal machine learning: A survey and taxonomy[END_REF][START_REF] Ramachandram | Deep multimodal learning: A survey on recent advances and trends[END_REF] . In our work, we employ the late fusion strategy, and concatenate all the probability maps generated from RGB and depth steams for a joint prediction.

p c = 1 M j (w,h) w=0,h=0 F(l) (w,h) j 1[y(l) (w,h) j = l] w,h w=0,h=0 1[y(l) (w,h) j = l] (5.

More formally,

ŷq = so f tmax[w * (p R ∨ p D )] (5.3)
where w denotes the convolution kernels for upsampling.

5.3.4/ CITYSCAPES-3I DATASET

To fully exploit few-shot semantic segmentation with additional depth information, we create a new dataset, named Cityscapes-3 i . We adopt the annotated RGB images and the depth images of the same scene from the Cityscapes dataset [START_REF] Cordts | The Cityscapes Dataset for Semantic Urban Scene Understanding[END_REF]. Cityscapes is a popular benchmark dataset for semantic understanding of outdoor scenes, which consists of thousands of precise depth images and pixel-wise semantic segmentation. Compared with object segmentation datasets such as PASCAL VOC [START_REF] Everingham | The PAS-CAL Visual Object Classes Challenge 2012[END_REF] and COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] , it is more challenging to predict a pixel-wise mask for semantic classes in the image of Cityscapes.

First, Cityscapes contains more complex urban street scenes. Images provide a broader perspective from the ground to the sky, involving a variety of categories. Then, most of the categories in the image have irregular shapes and lack distinct boundaries. Objects may overlap and be arranged randomly. Therefore it is a difficult task for segmentation models to learn characteristic features from only a few labeled samples and generalize to unseen classes.

We adopt all the RGB-D image pairs as well as the corresponding segmentation masks where i ∈ [START_REF] Altae-Tran | Low data drug discovery with one-shot learning[END_REF][START_REF] Pradeep | Multimodal fusion for multimedia analysis: A survey[END_REF] denotes the number of subsets, and the remaining 6 classes form the train label-set L train (see Table 5.5). Namely, L train ∩ L test = ∅. The images in D train and D test contain at least one pixel in the semantic mask from the label-set L train and L test , respectively. Moreover, we reset the pixels in segmentation masks that not belong to the corresponding label-sets as the background. In our experiments, we train and evaluate the proposed model on 3 folders in a cross-validation manner. For each folder, we take a random 500 samples and average the results from 5 runs to evaluate the performance of the models. Torch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF] . The backbone network (i.e., VGG-16) was initialized with pre-trained weights on ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. We resized the input images to 768×384 and trained on a single Nvidia TITAN Xp GPU with 12GB memory. All the few-shot segmentation models were trained using stochastic gradient descent (SGD) with a batch size of 1, a momentum of 0.9, and weight decay of 0.0005 for a maximum of 30,000 iterations. The initial learning rate was set to 0.0001 and reduced by 0.1 every 10,000 iterations.

Evaluation metrics Following the previous works on few-shot segmentation [START_REF] Shaban | One-shot learning for semantic segmentation[END_REF][START_REF] Zhang | Sg-one: Similarity guidance network for one-shot semantic segmentation[END_REF][START_REF] Wang | Panet: Few-shot image semantic segmentation with prototype alignment[END_REF] , we apply two standard metrics to evaluate the performance of learning models: mean-IoU and binary-IoU. Generally, the mean Intersection-over-Union (mean-IoU) is used to measure the accuracy of each foreground class and average over all the classes.

Binary-IoU deals uniformly with all object categories as one foreground class and averages the IoU of both foreground and background. Based on these two metrics, we can fairly compare the accuracy and efficiency of baselines in terms of 1-way N-shot semantic segmentation.

5.3.5.2/ EXPERIMENTAL RESULTS

In Table 5.6, we illustrate the performance of our proposed RDNet and other baseline methods on Cityscapes-3 i , including 1-way 1-shot and 1-way 2-shot semantic segmentation. First, we observe that using RGB data provides better segmentation results than using depth data as input. Moreover, one can also notice that a simple concatenation of for 1-way 1-shot and 33.7% for 1-way 2-shot, which represents an increase of +7% compared to RDNet-concat.

We further conduct ablation studies to investigate the validity of RDNet. The results are shown in Table 5.7 . We can observe a satisfactory performance enhancement of our method for most of the classes. In particular, the vegetation, sidewalk and sky classes.

These experimental results illustrate the effectiveness of our method and the potential of depth information in scene understanding with limited supervision.

Compared with RDNet-concat, our proposed method provides an improvement of +7.8% and +7.3% in terms of mIoU and binary IoU for 1-way 1-shot segmentation (see Table 5.8) . The results also show that simple concatenation has no significant improvement in the segmenatation prediction. Besides, Figure 5.8 shows the qualitative results of our method, including multimodal input and the segmentation prediction. Our model yields promising segmentation results in 1-shot settings. However, it is still challenging to distinguish the irregular objects and categories with similar characteristics in the complex scenes, such as car and bus.

5.3.5.3/ FEATURE VISUALIZATION

To clearly demonstrate the generalization and discrimination of the proposed model, we visualize the prototype representations of target classes in the RGB and depth embedding space using t-SNE (see Figure 5.9). In our work, each figure was generated using 500 samples of test classes in Cityscapes-3 i . On the whole, the prototypes generated from support RGB input can be well separated, especially for vegetation, terrain, sky in Figure 5.9c. Although it is challenging to produce distinctive prototypes in the depth embedding space, these prototype representations provide complementary cues regarding depth information. For example, the depth embeddings in Cityscapes-3 0 clearly show the discrimination on the classes vegetation, terrain and sky (see Figure 5.9b). Consequently, the generalizability of our few-shot segmentation network gets improved by incorporating supplementary depth information, leading to more promising prediction results.

5.4/ SUMMARY

Our work has presented MAPnet, a novel few-shot segmentation method based on the prototypical network. The proposed method provides effective semantic guidance on the query feature by a multiscale feature enhancement module. We elaborate the branches in this module to fully exploit the support information. Moreover, we employ the attention mechanism on the similarity-guided probability maps to produce an optimal pixelwise prediction, which also speeds up the convergence. Extensive experiments demonstrate the improved generalizability and discriminating ability of the proposed method.

Our model achieves a comparable accuracy with the state-of-the-art, outperforming most of the previous methods. In more detail, we have presented an extensible multi-level fusion network for fullysupervised multimodal semantic segmentation, known as CMnet, to integrate RGB and polarimetric images [START_REF] Zhang | Exploration of Deep Learning-based Multimodal Fusion for Semantic Road Scene Segmentation[END_REF]. Moreover, a central multimodal fusion framework was introduced to adaptively learn the joint feature representations of low-level and high-level modality-specific features. We also employed statistical methods as post-processing.

With regard to few-shot semantic segmentation, we first developed a novel segmentation method based on the prototypical network. The proposed MAPnet contains a multiscale feature enhancement module to fully exploit the support features, and attention mechanism to fuse multiple probability maps for an optimal pixel-wise prediction [START_REF] Zhang | Multiscale attention-based prototypical network for few-shot semantic segmentation[END_REF]. Then CHAPTER 6. CONCLUSION AND FUTURE WORK we extended the RGB-centric methods to take advantage of complementary depth information [START_REF] Zhang | Incorporating depth information into few-shot semantic segmentation[END_REF]. Our experiments have demonstrated the effectiveness of the proposed RDNet to integrate complementary depth cues in few-shot semantic segmentation. The rich color and geometric information of scenes can provide valuable semantic features.

All the proposed deep multimodal fusion methods show excellent performance in segmentation tasks with the support of a large number of labeled image data and computing power. We employed a variety of multimodal image inputs in the experiments, such as depth maps, near-infrared images, polarization images, etc. These multimodal image data provide valuable supplementary information of the same scene, making the segmentation model more robust and efficient. Our work illustrates the effectiveness and necessity of multi-modality in outdoor scene understanding.

6.2/ FUTURE PERSPECTIVES

Based on the work presented in this thesis, we give some recommendations for future research.

In Chapter 4, we have introduced two multimodal image fusion methods for semantic seg- integrating depth information, how to fuse multimodal data in high-level few-shot learning tasks for optimal performance is still in the preliminary stage. As a future perspective, we will focus on the impact of multimodal data in semi-supervised semantic understand-ing as well as the optimal network architecture for image fusion. As we know that deep learning-based artificial intelligence is gradually evolving from perception to cognitive intelligence, we expect deep multimodal fusion to facilitate this evolution and offer a host of innovations in the following years.

Robust semantic scene understanding is challenging due to complex object types, as well as environmental changes caused by varying illumination and weather conditions. This thesis studies the problem of deep semantic segmentation with multimodal image inputs. Multimodal images captured from various sensory modalities provide complementary information for complete scene understanding.

We provided effective solutions for fully-supervised multimodal image segmentation and few-shot semantic segmentation of the outdoor road scene.

Regarding the former case, we proposed a multi-level fusion network to integrate RGB and polarimetric images. A central fusion framework was also introduced to adaptively learn the joint representations of modality-specific features and reduce model uncertainty via statistical postprocessing.

In the case of semi-supervised semantic scene understanding, we first proposed a novel few-shot segmentation method based on the prototypical network, which employs multiscale feature enhancement and the attention mechanism. Then we extended the RGB-centric algorithms to take advantage of supplementary depth cues. Comprehensive empirical evaluations on different benchmark datasets demonstrate that all the proposed algorithms achieve superior performance in terms of accuracy as well as demonstrating the effectiveness of complementary modalities for outdoor scene understanding for autonomous navigation. Nous proposons des solutions efficaces pour la segmentation supervis ée d'images multimodales, de m ême que pour la segmentation semi-supervis ée de sc ènes routi ères en ext érieur. Concernant le premier cas, nous avons propos é un r éseau de fusion multi-niveaux pour int égrer des images couleur et polarim étriques.

Une m éthode de fusion centrale a également ét é introduite pour apprendre de mani ère adaptative les repr ésentations conjointes des caract éristiques sp écifiques aux modalit és et r éduire l'incertitude du mod èle via un post-traitement statistique. Dans le cas de la segmentation semi-supervis ée, nous avons d'abord propos é une nouvelle m éthode de segmentation bas ée sur un r éseau prototypique, qui utilise l'am élioration des fonctionnalit és multiéchelles et un m écanisme d'attention. Ensuite, nous avons étendu les algorithmes centr és sur les images RGB, pour tirer parti des informations de profondeur suppl émentaires fournies par les cam éras RGBD. Des évaluations empiriques compl ètes sur diff érentes bases de donn ées de r éf érence montrent que les algorithmes propos és atteignent des performances sup érieures en termes de pr écision et d émontrent le b én éfice de l'emploi de modalit és compl émentaires pour l'analyse de sc ènes ext érieures dans le cadre de la navigation autonome.

I

  would also like to thank all my colleagues in the VIBOT team. During the two years in Le Creusot, they have given me generous support and kindness in my research work and teaching. To name a few: David F., C édric, Ralph, Nathalie, Christophe S., Christophe L., Zawawi, Nathan P., Thomas, Thibault, Daniel... I am incredibly very thankful to Mojdeh and Cansen for their help in my early work of the thesis. They spent a lot of time teaching me how to do academic research and developing good research habits. Besides, I enjoyed the time working with Marc. I have fond memories of the days when we went to Prague to attend the conference. I also miss having dinner with friends and playing games at Mojdeh and Guillaume's house, climbing with David S., eating Girls' lunch with Ashvaany and Abir, training with Raphael and Nathan C. in Toulouse, with Eric and Fauvet taught in Nanjing together... Thank you very much for giving me many precious memories. Although I am far away from home, I still feel warm because of them. iii LIST OF FIGURES 1.1 Example of multimodal semantic segmentation from Cityscapes dataset. The prediction was generated by our CMFnet+BF2 framework. . . . . . . . 1.2 Number of papers published per year. Statistical analysis is based on the work by Caesar [15]. Segmentation includes image/instance/panoptic segmentation and joint depth estimation. . . . . . . . . . . . . . . . . . . . . . . 1.3 An illustration of deep multimodal segmentation pipeline. . . . . . . . . . . 2.1 Example of a convolutional neural network with four inputs. . . . . . . . . . 2.2 Example of the encoder-decoder architecture. . . . . . . . . . . . . . . . . . 2.3 Example of max pooling operation. . . . . . . . . . . . . . . . . . . . . . . . 2.4 Examples of popular activation functions. . . . . . . . . . . . . . . . . . . . 3.1 An illustration of different fusion strategies for deep multimodal learning. . . 3.2 FuseNet architecture with RGB-D input. Figure reproduced from [69]. . . . 3.3 Convoluted Mixture of Deep Experts framework. Figure extracted from [177]. 3.4 Fusion architecture with self-supervised model sdaptation modules. Figure extracted from [177]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Individual semantic segmentation experts are combined modularly using different statistical methods. Figure extracted from [13] . . . . . . . . . . . . 3.6 Accumulated dataset importance. Statistical analysis is based on the work by [15]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Real-time and accuracy performance. Performance of SSMA fusion method using different real-time backbones on the Cityscapes validation set (input image size: 768 × 384, GPU: NVIDIA TITAN X). . . . . . . . . . . 3.8 Real-time and accuracy performance. Performance of different fusion methods on the Tokyo Multi-Spectral dataset.(input image size: 640 × 480, GPU: NVIDIA 1080 Ti graphics card). . . . . . . . . . . . . . . . . . . . . . 4.1 Typical early fusion and Late fusion architectures comparison. . . . . . . . . ix x LIST OF FIGURES 4.2 Our proposed fusion architecture: CMnet for multimodal fusion based on late fusion architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.3 The electric and magnetic field of light as well as their continuous selfpropagating. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.4 Reflection influence on polarimetry. (a) and (b) represent a zoom on the non-polarized and polarized area, respectively. Figure extracted from [11]. . 54 4.5 (a) Mobile robot platform used for the acquisition of the POLABOT dataset. It is equipped with the IDS Ucam, PolarCam, Kinect 2 and a NIR camera. (b)(c)(d) Multimodal images in the POLABOT dataset. . . . . . . . . . . . . 55 4.6 Two segmented examples from Freiburg Forest dataset. RGB and/or EVI images were given as inputs. . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.7 Two segmented examples from POLABOT dataset. RGB and/or POLA images were given as inputs. . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.8 Typical fusion strategies with RGB and depth input. (a) Early fusion. (b) Late fusion. (c) Central fusion. As a comparison, the proposed fusion structure integrates the feature maps in a succession of layers into a central branch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.9 Overview of the central fusion network (CMFnet). The adaptive gating unit automatically produces the weights of modality-specific branch in each layer. GAP denotes to Global Average Pooling, "x" means multiplication, "C" means concatenation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.10 Training loss of models on the POLABOT dataset. . . . . . . . . . . . . . . 65 4.11 Improvement/error maps of the proposed CMFnet and CMFnet+BF2 in comparison to the RGB baseline. Note that the improved pixels and the misclassified pixels are recorded in green and red, respectively. . . . . . . . 66 4.12 Segmentation results on the POLABOT dataset. The first row of examples contains the RGB image and the corresponding polarimetric image. The second row of examples shows, from left to right, the ground truth image, the segmentation outputs of the individual experts (RGB and Polar), and the fusion results of CMFnet-BF2 and CMFnet-BF3. . . . . . . . . . . . . . 67 4.13 Two sets of segmentation results on Cityscapes dataset. The first row of examples contains the RGB image and the corresponding depth image. From left to right of the second row of examples: ground truth, the prediction of RGB input, average and LFC. From left to right of the second row of examples: the fusion results of CMoDE, CMFnet, CMFnet-BF2 and CMFnet-BF3, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 LIST OF FIGURES xi 5.1 An overview of the proposed method (MAPnet). Given a query image of a new category, e.g., aeroplane, the goal of few-shot segmentation is to predict a mask of this category regarding only a few labeled samples. . . . 5.2 Illustration of the proposed method (MAPnet) for few-shot semantic segmentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Qualitative results of our method for 1-way 1-shot segmentation on the PASCAL-5 i dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Training loss of models with and without attention-based gating (ABG) for 1-way 1-shot segmentation on PASCAL-5 0 . . . . . . . . . . . . . . . . . . . 5.5 Qualitative results of our model using scribble and bounding box annotations for 1-way 5-shot setting. The chosen example in support images shows the annotation types. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Overview of the proposed RDNet approach. R and D indicate the RGB and depth image input, respectively. The abstract features of labeled support images are mapped into the corresponding embedding space (circles). Multiple prototypes (blue and yellow solid circles) are generated to perform semantic guidance (dashed lines) on the corresponding query features (rhombus). RDNet further produces the final prediction by combining the probability maps from RGB and depth stream. . . . . . . . . . . . . . . 5.7 Details of the proposed RDNet architecture. It includes two mirrored streams: an RGB stream and a depth stream. Each stream processes the corresponding input data, including a support set and a query set. The prototypes of support images are obtained by masked average pooling. Then the semantic guidance is performed on the query feature by computing the relative cosine distance. The results from these two streams are combined at the late stage. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8 Qualitative results of our method for 1-way 1-shot semantic segmentation on Cityscapes-3 i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9 Visualization using t-SNE [179] for RGB and depth prototype representations in our RDNet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LIST OF TABLES 3.1 Typical early fusion methods reviewed in this chapter. . . . . . . . . . . . . 3.2 Typical late fusion methods reviewed in this chapter. . . . . . . . . . . . . . 3.3 Typical hybrid fusion methods reviewed in this chapter. . . . . . . . . . . . . 3.4 Summary of popular datasets for image segmentation task. . . . . . . . . . 3.5 Summary of popular 2D/2.5D multimodal datasets for scene understanding.

3. 6

 6 Examples of multimodal image datasets mentioned in Subsection 3.2.2.For each dataset, the top image shows two modal representations of the same scene. The bottom image is the corresponding groundtruth. . . . . . 3.7 Performance results of deep multimodal fusion methods on SUN RGB-D dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Performance results of deep multimodal fusion methods on NYU Depth v2 dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Experimental results of deep multimodal fusion methods on Cityscapes dataset. Input images are uniformly resized to 768 × 384. . . . . . . . . . . . 3.10 Experimental results of deep multimodal fusion methods on Tokyo Multi-Spectral dataset. The image resolution in the dataset is 640 × 480. . . . . . 3.11 Parameters and inference time performance. The reported results on the Cityscapes dataset are collected from [178]. . . . . . . . . . . . . . . . . . . 4.1 Performance of segmentation models on Freiburg Multispectral Forest dataset. EF, LF refer to early fusion and late fusion respectively. We report pixel accuracy (PA), mean accuracy (MA), mean intersection over union (MIoU), frequency weighted IoU (FWIoU) as metric to evaluate the performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Comparison of deep unimodal and multimodal fusion approaches by class. We report MIoU as metric to evaluate the performance. . . . . . . . . . . . 4.3 Segmentation performance on POLABOT dataset . . . . . . . . . . . . . . xiii xiv LIST OF TABLES 4.4 Performance comparison of our method with baseline models on the PO-LABOT dataset. Note that SegNet is used as the unimodal baseline and the backbone of multimodal fusion methods. . . . . . . . . . . . . . . . . . . 65 4.5 Ablation study of our method. Per class performance of our proposed framework in comparison to individual modalities with ENet baseline on Cityscapes dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.6 Performance of fusion models with ENet backbone on Cityscapes dataset. 66 5.1 Training and evaluation on PASCAL-5 i dataset using 4-fold crossvalidation, where i denotes the number of subsets. . . . . . . . . . . . . . . 75 5.2 Results of 1-way 1-shot and 1-way 5-shot semantic segmentation on PASCAL-5 i using mean-IoU(%) metric. The results of 1-NN and LogReg are reported by [158]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5.3 Results of 1-way 1-shot and 1-way 5-shot segmentation on PASCAL-5 i using binary-IoU(%) metric. ∆ denotes the difference between 1-shot and 5-shot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5.4 Evaluation results of using different types of annotations in mean-IoU(%) metric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 5.5 Training and evaluation on Cityscapes-3 i dataset using 3-fold crossvalidation, where i denotes the number of subsets. . . . . . . . . . . . . . . 83 5.6 Results of 1-way 1-shot and 1-way 2-shot semantic segmentation on Cityscapes-5 i using mean-IoU(%) metric. . . . . . . . . . . . . . . . . . . . 85 5.7 Per-class mean-IoU(%) comparison of ablation studies for 1-way 1-shot semantic segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 5.8 Results of 1-way 1-shot semantic segmentation using binary IoU and the runtime. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 INTRODUCTION 1.1/ CONTEXT AND MOTIVATION

Figure 1 . 1 :

 11 Figure 1.1: Example of multimodal semantic segmentation from Cityscapes dataset. The prediction was generated by our CMFnet+BF2 framework.

Figure 1 .

 1 1 illustrates an example of semantic image segmentation with RGB and depth input. Besides, much research dedicates to exploring advanced technologies under limited supervision. Recently, few-shot learning has emerged as a hot topic in the computer vision community. Deep learning-based image understanding techniques require a large number of labeled images for training. Few-shot semantic segmentation, on the contrary, aims at generalizing the segmentation ability of the model to new categories given a few samples. Namely, the trained neural network predicts pixel-level mask of new categories on the query image, given only a few labeled support images. In this thesis, we explore the attention mechanism-based method for few-shot segmentation task, aiming to improve the semantic feature representation and generalization capabilities of the models.

Figure 1 . 2 :

 12 Figure 1.2: Number of papers published per year. Statistical analysis is based on the work by Caesar [15]. Segmentation includes image/instance/panoptic segmentation and joint depth estimation.

Figure 1 . 3 :

 13 Figure 1.3: An illustration of deep multimodal segmentation pipeline.

I

  We present a novel multimodal fusion framework for semantic segmentation. The fusion model adaptively learns the joint feature representations of both low-level and high-level modality-specific feature via a central neural network and statistical post-processing.IIIWe provide a systematic review of 2D/2.5D deep multimodal image segmentation in fusion methodology and dataset. We gather quantitative experimental results of multimodal fusion methods on different benchmark datasets, including their accuracy, runtime, and memory footprint.In the case of semi-supervised semantic scene understanding, our contributions are: We propose a novel few-shot segmentation method based on the prototypical network. The proposed network provides effective semantic guidance on the query feature by a multiscale feature enhancement module. The attention mechanism is employed to fuse the similarity-guided probability maps.

Figure 2 .

 2 Figure 2.1 illustrates a typical architecture of convolutional neural networks. In detail,

Figure 2 . 2 :

 22 Figure 2.2: Example of the encoder-decoder architecture.

2 Figure 2 . 3 :

 223 Figure 2.3: Example of max pooling operation.

Figure 2 . 4 :

 24 Figure 2.4: Examples of popular activation functions.

2. 3

 3 .1/ BATCH NORMALIZATION Training deep neural networks with multiple hidden layers is quite challenging. One reason is that the model is updated layer-by-layer backward from the output to the input using an error estimate that assumes the weights in the layers prior to the current layer are fixed. This slows down the training by requiring lower learning rates and careful parameter initialization and makes it notoriously hard to train models with saturating nonlinearities [79]. Therefore batch normalization, as an effective optimization technique, is proposed to standardize the inputs to a layer for each mini-batch while training very deep neural networks. It stabilizes the learning process and dramatically reduces the number of training epochs required to train deep networks. Batch normalization can be implemented during training by calculating the mean and standard deviation of each input variable to a layer per mini-batch and using these statistics to perform the standardization. Alternatively, a running average of mean and standard deviation can be maintained across mini-batches but may result in unstable training. For example, He et al. [72] used batch normalization after the convolutional layers in their very deep model, referred to as ResNet. The reported results achieved state-of-the-art in the image classification task. In our work of model design, we usually add batch normalization transformation before nonlinearity.

  ing techniques and is by far the most time consuming, both in terms of effort required for configuration and computational complexity required for execution. In the following, we summarize the commonly used techniques in model training, including data preprocessing, weight initialization, loss function and gradient descent optimization.2.4.1/ DATA PREPROCESSINGGenerally, training deep learning models requires a lot of data because of the huge number of parameters needed to be tuned by the learning algorithm. Data preprocessing is a prerequisite guarantee for effectively model training, and we need to be careful to prepare the training data to achieve the best prediction results. For example, many deep learning models have normalized input processing, namely whitening operation, which changes the average pixel value of the image to zero and the variance of the image to unit variance. In detail, the mean and variance of the original image are first calculated, then each pixel value of the original image is transformed. This operation enables the convergence of the neural network faster. Common data preprocessing methods also include data quality assessment, feature aggregation, feature sampling, dimensionality reduction, and feature encoding.

Figure 3 . 2 :

 32 Figure 3.2: FuseNet architecture with RGB-D input. Figure reproduced from [69].

3. 1

 1 .1.2/ LATE FUSION As early as 2014, Gupta et al. [63] proposed a geocentric embedding for object detection and segmentation. The authors employed two convolutional neural network streams to extract RGB and depth features, respectively. The feature maps obtained from these two streams are combined by SVM classifier at the late stage. Then the work by Li et al. [104] addresses semantic labeling of RGB-D scenes by developing a Long Short-Term Memorized Context Fusion (LSTM-CF) model. This network captures photometric and depth information in parallel, facilitating deep integration of contextual information. The global contexts and the last convolutional features of the RGB stream are fused by simple tensor concatenation.

Figure 3 . 3 :

 33 Figure 3.3: Convoluted Mixture of Deep Experts framework. Figure extracted from [177].

Figure 3 . 4 :

 34 Figure 3.4: Fusion architecture with self-supervised model sdaptation modules. Figure extracted from [177].

Figure 3 . 5 :

 35 Figure 3.5: Individual semantic segmentation experts are combined modularly using different statistical methods. Figure extracted from[START_REF] Blum | Modular sensor fusion for semantic segmentation[END_REF]

  When to Fuse: Many deep multimodal fusion methods are extended from existing unimodal methods, or derived from other typical neural networks. In the former case, multiple unimodal segmentation networks are integrated into a composite end-to-end training model in early, late, or multi-level stages. Early fusion strategy allows stronger cross-modal information interaction, while late fusion shows more flexibility and scalability for implementation. Extensive experiments demonstrate that both low-level and high-level features are valuable to the final prediction.

Figure 3 . 6 :•

 36 Figure 3.6: Accumulated dataset importance. Statistical analysis is based on the work by [15].

  for full scene segmentation. It consists of 591 images and 23 object classes.However, along with the development of deep learning techniques, small-scale datasets can not meet the demands of model training. PASCAL VOC dataset[START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] is one of the most popular object segmentation datasets, which derived from the early stage competition: PASCAL Visual Object Classes (VOC) challenge. It provides thousands of images with pixel-level labeling. Up to now, it has been augmented to several additional datasets with a set of extra annotations, such as PASCAL-Context[START_REF] Mottaghi | The role of context for object detec-tion and semantic segmentation in the wild[END_REF], PASCAL-Part[START_REF] Chen | Detect what you can: Detecting and representing objects using holistic models and body parts[END_REF], SBDB[START_REF] Hariharan | Semantic contours from inverse detectors[END_REF]. Another similar large-scale dataset is Microsoft COCO dataset[START_REF] Lin | Microsoft coco: Common objects in context[END_REF], which contains 81 categories of objects, including 21 categories of PASCAL VOC. It covers complex everyday scenes and their contextual information. PASCAL VOC and COCO dataset are not only the most popular benchmarks for fully supervised segmentation but also frequently-used in weakly supervised learning for object segmentation.

  6 illustrates RGB-D image examples sampling from the datasets reviewed in this part.• Indoor scenes: One of the main difficulties for indoor scene segmentation is that object classes always come in various positions, shapes, and sizes. By taking ad-vantage of RGB-D data, we can encode the pixel-level color and depth information of the same scene into a high-level feature representation. Such information fusion, to a certain extent, reduces the difficulty of indoor object recognition. NYUDv2[START_REF] Silberman | Indoor segmentation and support inference from rgbd images[END_REF] is an early RGB-D database containing 795 training images and 654 testing images with pixel-wise labels for 40 semantic categories. A Microsoft Kinect camera captured all the RGB and depth image pairs with favorable frame synchronization. This dataset aims to inform a structured 3D interpretation of indoor scenes, having become one of the most popular multimodal benchmarks so far. Another standard benchmark for indoor scene recognition is SUN RGB-D[START_REF] Song | SUN RGB-D: A RGB-D scene understanding benchmark suite[END_REF]. It consists of around 10K RGB-D images with 37 indoor object classes. This dataset advances the stateof-the-art in all major scene understanding tasks and provides a fair comparison of deep multimodal fusion methods.• Outdoor scenes: Unlike indoor scenes, the depth information of outdoor scenes is generally captured by stereo vision cameras or LiDAR due to Kinect's poor performance in sunlight. As one of the segmentation benchmark datasets, Cityscapes consists of thousands of high-quality depth images of the same scene. These depth maps overcome the lack of depth information of objects for road scene recognition.

  was created to tackle the semantic segmentation problem in forested environments. It consists of 366 aligned color, depth, and near-infrared images with six classes pixel-wise annotation. Due to the abundant presence of vegetation in the unstructured forest environment, this dataset provides enhanced NIR images (e.g., Normalized Difference Vegetation Index images, Enhanced Vegetation Index images) to ensure border accuracy. Besides, RANUS dataset[START_REF] Choe | RANUS: RGB and NIR urban scene dataset for deep scene parsing[END_REF] has been released to the public in 2018. It consists of 40k spatially-aligned RGB-NIR pairs for real-world road scenes, and thousands of keyframes are annotated with ground truth masks for ten classes: sky, ground, water, mountain, road, construction, vegetation, object, vehicle, and pedestrian.

  mentation. It contains both visible and thermal infrared images captured in daily and night conditions. There are 1569 images manually labeled to eight classes: car, person, bike, curve, car stop, guardrail, color cone, and bump. Then Shivakumar et al.[START_REF] Shreyas S Shivakumar | PST900: RGB-Thermal Calibration, Dataset and Segmentation Network[END_REF] presented PST900, a dataset of 894 synchronized and calibrated RGB and thermal image pairs with pixel-level annotations across four distinct classes from the DARPA Subterranean Challenge. The long-wave infrared (LWIR) imagery was used as a supporting modality for semantic segmentation of subterranean scenes. These large-scale RGB-Thermal datasets broaden the research field of deep learning-based scene understanding, allowing for more in-depth exploration in poor visibility and adverse weather conditions.

Figure 3 . 7 :

 37 Figure 3.7: Real-time and accuracy performance. Performance of SSMA fusion method using different real-time backbones on the Cityscapes validation set (input image size: 768 × 384, GPU: NVIDIA TITAN X).

  summarized and provided the researchers with two sets of execution time comparisons, as shown in Figure3.73.8. Execution time or runtime, as an essential metric, obviously shows the learning model's execution efficiency. Although this metric is easily ignored in the accuracy-centric algorithm optimization, it should be carefully considered in industriallevel applications, such as self-driving cars. The inference time is usually dependant on the hardware and backend implementation.

Figure 3 . 8 :

 38 Figure 3.8: Real-time and accuracy performance. Performance of different fusion methods on the Tokyo Multi-Spectral dataset.(input image size: 640 × 480, GPU: NVIDIA 1080 Ti graphics card).

4 DEEPR

 4 obust multimodal fusion is one of the challenging research problems in semantic scene understanding. In real-world applications, the fusion system can overcome the drawbacks of individual sensors by taking different feature representations and statistical properties of multiple modalities. This chapter is dedicated to fully-supervised semantic segmentation with multimodal image input. We seek robust solutions that can effectively learn feature representations from multi-modalities and optimally produce the pixel-wise classification. As detailed in Chapter 3, we conclude the architectural design and input data for multimodal image segmentation from the comprehensive review of the literature. Based on this background knowledge, we explore different fusion architectures with various imaging modalities, including late fusion and central fusion. The former can obtain richer semantic information by using different feature extractors. The latter sequentially maps both low-level and high-level multimodal features into a central branch. Statistical post-processing is employed to reduce model uncertainty, which leads to significant performance improvement.In the following, we describe the proposed deep multimodal fusion methods for the outdoor scene semantic segmentation task in detail. Multiple fusion architectures are also compared and analyzed in this chapter. Moreover, we introduce a novel multimodal benchmark dataset dedicated to the polarimetric imaging of outdoor road scenes. The performances of the proposed algorithms are evaluated using extensive experiments on the Freiburg Forest dataset, POLABOT dataset, and Cityscapes dataset.

4. 1 /

 1 CMNET: DEEP MULTIMODAL FUSION FOR ROAD SCENE SEGMENTATION 4.1.1/ INTRODUCTION Semantic segmentation is one of the main challenges in computer vision. Along with the appearance and development of Deep Convolutional Neural Network (DCNN) [93], the trained model can predict which class each pixel in the input images belongs to. By learning from massive data sets of diverse samples, this method achieves a good performance on end-to-end image recognition. Robust and accurate scene parsing of outdoor environments paves the way towards autonomous navigation and relationship inference. Compared with indoor scenes, off-road perception is more challenging due to dynamic and complex situations. The outdoor environment may easily change in different time slots with light or color variations. Even in structured environments, for instance on urban roads, there are still several challenges such as the detection of glass and muddy puddles.

4. 1 .

 1 CMNET: DEEP MULTIMODAL FUSION FOR ROAD SCENE SEGMENTATION 49 (a) early fusion (b) late fusion

Figure 4 . 1 :

 41 Figure 4.1: Typical early fusion and Late fusion architectures comparison.

Figure 4 .

 4 Figure 4.1b shows a typical late fusion architecture. It has two separated branches of network, with each branch trained to extract features from a special modality. Fusion takes place after a series of downsampling. Assuming that the two feature maps have size 1×H×W, after concatenation, the resulting feature will be 2×H×W. Then a 1×1 convolution is applied to reduce the number of channels.

4. 1 . 3 /

 13 PROPOSED METHOD We propose a new approach for multimodal data fusion, Complex Modality Neural Network (CMnet), based on late fusion architecture since it has aforementioned merits. Let S = {(X n , y n )|n = 1, 2, ..., N} denotes the training set, and X n = {x a , x b } is the training example, where x a and x b are the vector of input images from modality a and b, respectively. Also let M 1 , and M 2 , denote the map between the input and output of the first, and second branch of the encoder-decoder network, respectively. Then the output of the

Figure 4 . 2 :

 42 Figure 4.2: Our proposed fusion architecture: CMnet for multimodal fusion based on late fusion architecture.

Figure 4 .

 4 Figure 4.2 presents the whole architecture of CMnet. It has an Encoder-Decoder structure and two separated branches. The encoder is used for mapping raw inputs to feature

4. 1 .

 1 4/ DATASET 4.1.4.1/ POLARIZATION FORMALISM Polarization is a common property of light waves that specifies the geometrical orientation of the oscillations. Figure 4.3 indicates the electric and magnetic field component of a light which oscillates in phase perpendicular to each other and to the direction in which the radiation propagates. In nature scenes, many light sources such as sunlight, LED spotlights, and incandescent bulbs produce unpolarized light because their electric field's direction fluctuates randomly in time. When the direction of the electric field of light is restricted to a single plane by filtration, then it is called polarized light. If an unpolarized light is reflected by an object, it becomes partially linearly polarized. In general, we can classify the states of polarization of the light into unpolarized, partially polarized, and fully polarized. Moreover, polarimetric imaging carries not only the color and shape information of objects, but also characterizes the special physical information for reflecting surface.

Figure 4 . 3 :

 43 Figure 4.3: The electric and magnetic field of light as well as their continuous selfpropagating.

Figure 4 . 4 :

 44 Figure 4.4: Reflection influence on polarimetry. (a) and (b) represent a zoom on the non-polarized and polarized area, respectively. Figure extracted from [11].

Figure 4 .

 4 [START_REF] Audebert | Beyond RGB: Very high resolution urban remote sensing with multimodal deep networks[END_REF] shows the detected polarization performance with a polarimetric camera.4.1.4.2/ POLABOT DATASETTo fully explore the multimodal image fusion for semantic segmentation task, we built a novel outdoor road scene dataset called POLABOT. As shown in Figure4.5, we collected multimodal images using a mobile robot platform equipped with four cameras: the RGB camera (IDS Ucam), a polarimetric camera (PolarCam), a depth camera (Kinect 2.0), and a near-infrared camera. The raw dataset contains over 700 multimodal images of the same scene. All the images were acquired, synchronized, and calibrated using the Robot Operating System (ROS) framework. In particular, the three gray-scale description images of the raw polarimetric data can be obtained by calculating the AOP, DOP, and the intensity, which enables further generate the HSL (Hue Saturation Luminance)

4. 1 .Figure 4 . 6 :Figure 4 . 7 :

 14647 Figure 4.6: Two segmented examples from Freiburg Forest dataset. RGB and/or EVI images were given as inputs.

60CHAPTER 4 .

 4 DEEP MULTIMODAL FUSION FOR SEMANTIC IMAGE SEGMENTATION an increasing study on 2D/2.5D multimodal fusion. Existing deep fusion methods generally incorporate the features at an early or late stage, which may cause a loss of semantic information. As shown in Figure 4.8, we employ a centralized fusion strategy to continuously correlate multimodal information. Besides, traditional neural networks lack probabilistic considerations, while the representation of uncertainty should be taken into account to tackle this problem.

Figure 4 . 8 :

 48 Figure 4.8: Typical fusion strategies with RGB and depth input. (a) Early fusion. (b) Late fusion. (c) Central fusion. As a comparison, the proposed fusion structure integrates the feature maps in a succession of layers into a central branch.

Figure 4 . 9 :

 49 Figure 4.9: Overview of the central fusion network (CMFnet). The adaptive gating unit automatically produces the weights of modality-specific branch in each layer. GAP denotes to Global Average Pooling, "x" means multiplication, "C" means concatenation.

64CHAPTER 4 .

 4 DEEP MULTIMODAL FUSION FOR SEMANTIC IMAGE SEGMENTATION 4.2.3.2/ EVALUATION ON POLABOT DATASET

Figure 4 . 10 :

 410 Figure 4.10: Training loss of models on the POLABOT dataset.

Figure 4 . 11 :

 411 Figure 4.11: Improvement/error maps of the proposed CMFnet and CMFnet+BF2 in comparison to the RGB baseline. Note that the improved pixels and the misclassified pixels are recorded in green and red, respectively.

Figure 4 .

 4 Figure 4.11 illustrates the improvement/error maps of CMFnet and CMFnet+BF2, which denote the improvement over the output of the RGB baseline in green and the misclassifications in red. From the visualized improvement/error maps, we can clearly see our fusion network gains in certain classes, such as vegetation, traffic sign, and road. After statistical post-processing, the segmentation prediction achieves a significant improvement due to the reduction in model uncertainty. We show more segmentation results on

Figure 4 . 12 :

 412 Figure 4.12: Segmentation results on the POLABOT dataset. The first row of examples contains the RGB image and the corresponding polarimetric image. The second row of examples shows, from left to right, the ground truth image, the segmentation outputs of the individual experts (RGB and Polar), and the fusion results of CMFnet-BF2 and CMFnet-BF3.

Figure 4 . 13 :

 413 Figure 4.13: Two sets of segmentation results on Cityscapes dataset. The first row of examples contains the RGB image and the corresponding depth image. From left to right of the second row of examples: ground truth, the prediction of RGB input, average and LFC. From left to right of the second row of examples: the fusion results of CMoDE, CMFnet, CMFnet-BF2 and CMFnet-BF3, respectively.

5. 2 .

 2 MAPNET: A MULTISCALE ATTENTION-BASED PROTOTYPICAL NETWORK 71 predicts pixel-level mask of new categories on the query image, given only a few labeled support images. The semantic guidance ability of support images and the generalizability to unseen class may significantly affect the segmentation performance.

Figure 5 .

 5 Figure 5.1 illustrates the overall workflow of our MAPnet.

5. 2 . 16 MultiscaleFigure 5 . 2 :

 21652 Figure 5.2: Illustration of the proposed method (MAPnet) for few-shot semantic segmentation.

  we elaborate three branches in MFE module. Each branch takes the intermediate support features and high-level support features as input. The support features are resized and concatenated for providing effective feature aggregation. This module enables the few-shot segmentation network more expressive as the model becomes deeper and wider. Empirically, we employ multiscale atrous convolution with rates r = 2, 4, 8 to preserve more spatial and contextual information.

Figure 5 . 4 :

 54 Figure 5.4: Training loss of models with and without attention-based gating (ABG) for 1-way 1-shot segmentation on PASCAL-5 0 .

Figure 5 .Figure 5 . 6 :

 556 Figure 5.6: Overview of the proposed RDNet approach. R and D indicate the RGB and depth image input, respectively. The abstract features of labeled support images are mapped into the corresponding embedding space (circles). Multiple prototypes (blue and yellow solid circles) are generated to perform semantic guidance (dashed lines) on the corresponding query features (rhombus). RDNet further produces the final prediction by combining the probability maps from RGB and depth stream.

5. 3 .

 3 RDNET: INCORPORATING DEPTH INFORMATION INTO FEW-SHOT SEGMENTATION81 N indicates the number of training examples.

Figure 5 . 7 :

 57 Figure 5.7: Details of the proposed RDNet architecture. It includes two mirrored streams: an RGB stream and a depth stream. Each stream processes the corresponding input data, including a support set and a query set. The prototypes of support images are obtained by masked average pooling. Then the semantic guidance is performed on the query feature by computing the relative cosine distance. The results from these two streams are combined at the late stage.

2 ) where 1 (

 21 •) is the indicator function that equals to 1 if the argument is true or 0 otherwise. Similarly, the prototypes for the background can be computed with 1[y(l) (w,h) j l]. It is notable that both foreground and background information of RGD and depth images should be considered in this work. These representative prototypes are the premise of reliable semantic guidance. To take an example, Figure shows the visualization of RGB and depth prototype representations in our experi-5.3. RDNET: INCORPORATING DEPTH INFORMATION INTO FEW-SHOT SEGMENTATION83

5. 3

 3 .5/ EXPERIMENTS 5.3.5.1/ SETUP Implementation details We conduct the experiments with implementations in Py-

ClassFigure 5 . 8 :

 58 Figure 5.8: Qualitative results of our method for 1-way 1-shot semantic segmentation on Cityscapes-3 i .

  (a) RGB embeddings in Cityscapes-3 0 (b) Depth embeddings in Cityscapes-3 0 (c) RGB embeddings in Cityscapes-3 1 (d) Depth embeddings in Cityscapes-3 1 (e) RGB embeddings in Cityscapes-3 2 (f) Depth embeddings in Cityscapes-3 2

Figure 5 . 9 :

 59 Figure5.9: Visualization using t-SNE[START_REF] Van Der Maaten | Visualizing data using t-SNE[END_REF] for RGB and depth prototype representations in our RDNet.

6 CONCLUSION

 6 Moreover, in order to explore the multimodal image fusion in few-shot segmentation, we proposed a novel segmentation network to incorporate complementary depth information, which consists of two mirrored streams based on metric learning. To fully take advantage of color and geometric information of the scenes, we mapped the representative features of target classes into different embedding spaces. The learned prototype represenations provide effective semantic guidance on the corresponding query feature. Then we integrated the generated probability maps at a late stage. Comprehensive experiments and ablation studies on Cityscapes-3 i dataset demonstrate the improved generalizability and discriminating ability of our method. The proposed RDNet is simple yet effective, and explore the use of depth information in few-shot segmentation task. AND FUTURE WORK "As a technologist, I see how AI and the fourth industrial revolution will impact every aspect of people's lives."-Fei-Fei Li, Professor of Computer Science at Stanford University6.1/ GENERAL CONCLUSIONThe general aim of this thesis was to improve the semantic scene understanding for outdoor road scene using complementary multimodal data. In Chapter 2, we first introduced the fundamental background knowledge of deep neural networks. Chapter 3 explored existing multimodal image fusion methods using state-of-the-art deep learning techniques.Compared with traditional machine learning methods, deep multimodal fusion performs better in terms of accuracy when trained with huge amounts of data. Then a comprehensive review and analysis of the related work for multimodal images was made to provide the reader with a broad overview of input data[START_REF] Zhang | Deep multimodal fusion for semantic image segmentation: A survey[END_REF]. Moreover, we have studied the problem of multimodal semantic segmentation from two aspects, including data and algorithms. Chapter 4 presented different semantic segmentation methods and related ablation studies with multiple data input. Chapter 5 further explored the unimodal and multimodal semantic segmentation under limited supervision.

  mentation tasks. Although the proposed CMnet and CMFnet+BF2 achieve satisfactory results, there still remain inherent drawbacks of such methods: (i) The trade-off between prediction accuracy and cost-effectiveness should be considered in practical applications. The simplicity of implementation, real-time, and scalability are very challenging problems for autonomous UGV navigation. (ii) The deep multimodal fusion requires a large number of high-quality labeled images. The training data greatly influences the stability of the segmentation model, and it is especially necessary to be wary of noisy data and missing data. Thus, it would be interesting to investigate more robust multimodal image segmentation algorithms. Furthermore, we have shown the critical role of multimodal image data in the outdoor scene understanding and indicated that different sensory modalities could affect the segmentation performance. Hence, we expect to employ multimodal data in other high-level image segmentation tasks such as panoptic segmentation. Such segmentation unifies the typically distinct tasks of semantic segmentation and instance segmentation, which represents an important step toward real-world vision systems. Besides, we have presented advanced few-shot semantic segmentation methods in Chapter 5. Although extensive experiments and ablation studies demonstrate the improved generalizability and discriminating ability of few-shot segmentation algorithms by

Titre:

  Analyse et fusion d'images multimodales pour la navigation autonome Mots-cl és : Segmentation s émantique, Fusion d'images, Multimodalit é, Apprentissage profond R ésum é : Une analyse s émantique robuste des sc ènes ext érieures est difficile en raison des changements environnementaux caus és par l' éclairage et les conditions m ét éorologiques variables, ainsi que par la variation des types d'objets rencontr és. Cette th èse étudie le probl ème de la segmentation s émantique à l'aide de l'apprentissage profond et avec des d'images de diff érentes modalit és. Les images captur ées à partir de diverses modalit és d'acquisition fournissent des informations compl émentaires pour une compr éhension compl ète de la sc ène.

Table 3 .

 3 1: Typical early fusion methods reviewed in this chapter.

	Ref.	Method	Backbone	Contribution(s)	Year Source Code
	[30]	Couprie' -	Initial attempt	2013 Available
	[69]	FuseNet VGG-16	Dense fusion/Sparse fusion	2016 Available
	[118] MVCNet VGG-16	Multi-view consistency	2017 -
	[78]	LDFNet VGG-16	D&Y Encoder	2018 Available
	[32]	RFBNet AdapNet++ Bottom-up interactive fusion structure	2019 -
	[76]	ACNet	ResNet-50	Multi-branch attention based network	2019 Available
	[172] RTFNet ResNet-152 RGB-Thermal fusion with Upcepton blocks 2019 Available

Table 3 .

 3 2: Typical late fusion methods reviewed in this chapter.

	Ref.	Method	Backbone Contribution(s)	Year Source Code
	[63]	Gupta'	-	CNN+SVM	2014 Available
	[104] LSTM-CF Deeplab	LSTM-based context fusion	2016 Available
	[176] LFC	VGG-16	Late-fused convolution	2016 Available
	[187] Wang'	VGG-16	Feature transformation network	2016 -
	[177] CMoDE	AdapNet	Class-wise adaptive gating network	2017 Available
	[25]	LSD-GF	VGG-16	Locality-sensitive DeconvNet with gated fusion 2017 -

Table 3 .

 3 3: Typical hybrid fusion methods reviewed in this chapter.

	Ref.	Method	Backbone	Contribution(s)	Year Source Code
	[130] RDFNet	ResNet-152	Extension of residual learning	2017 Available
	[84]	DFCN-DCRF VGG-16	Dense-sensitive FCN/ Dense-sensitive CRF	2017 Available
	[102] S-M Fusion	VGG-16	Semantics-guided Multi-level feature fusion	2017 -
	[107] CFN	RefineNet-152 Context-aware receptive field/ Cascaded structure	2017 -
	[85]	RedNet	ResNet-50	Residual Encoder-Decoder structure	2018 Available
	[178] SSMA	AdapNet++	self-supervised model adaptation fusion mechanism 2019 Available

Table 3 .

 3 4: Summary of popular datasets for image segmentation task.

	Ref.	Dataset	Classes Resolution Images Scene	Data	Year
	[161] MSRC	23	320x213	591	Outdoor	2D	2006
	[60]	Stanford background 8	320x240	715	Outdoor	2D	2009
	[14]	CamVid	32	960x720	701	Outdoor	2D	2009
	[43]	PASCAL VOC	20	Variable	11K	Variable	2D	2012
	[163] NYU Depth v2	40	480x640	1449	Indoor	2.5D	2012
	[111] Microsoft COCO	80	Variable	330K	Variable	2D	2014
	[55]	KITTI	11	Variable	400	Outdoor	2D/3D 2015
	[29]	Cityscapes	30	2048x1024 5K	Outdoor	2.5D	2015
	[151] SYNTHIA	13	960x720	13K	Outdoor(synthetic) 2.5D	2016
	[149] GTA5	19	1914x1052 13K	Outdoor(synthetic) 2D	2016
	[168] SUN RGB-D	37	Variable	10K	Indoor	2.5D	2015
	[216] ADE20K	150	Variable	22K	Variable	2D	2017
	[127] Mapillary Vistas	66	1920x1080 25K	Outdoor	2D	2017
	[203] WildDash	28	Variable	1.8K	Outdoor	2D	2018

Table 3

 3

.4. Further information are provided, including numbers of classes, size of the database, and the

Table 3 .

 3 5: Summary of popular 2D/2.5D multimodal datasets for scene understanding.

	Ref.	Dataset	Images Scene	Multi-modal data	Year
	[163] NYUDv2	1449	Indoor	RGB/Depth	2012
	[168] SUN RGB-D	10K	Indoor	RGB/Depth	2015
	[29]	Cityscapes	5K	Urban street	RGB/Depth	2015
	[151] SYNTHIA	13K	Urban street	RGB/Depth	2016
	[176] Freiburg Forest	5K	Forest	RGB/Depth/NIR	2016
	[31]	ScanNet	19K	Indoor	RGB/Depth	2017
	[64]	Tokyo Multi-Spectral 1569	Urban street	RGB/Thermal	2017
	[174] CATS 2	686	Variable	RGB/Depth/Thermal	2018
	[26]	RANUS	40k	Urban street	RGB/NIR	2018
	[210] POLABOT	175	Outdoor	RGB/NIR/Polarization 2019
	[159] PST900	894	Subterranean RGB/Thermal	2019
	[92]	DISCOMAN	600K	Indoor	RGB/Depth	2019
	type of scenes.

Table 3 . 6

 36

	Zhang et al. [210] released a small-scale segmentation dataset, known as POLABOT,
	that dedicates to the polarimetric imaging of outdoor scenes. Synchronized cameras
	collect hundreds of raw color, Near-InfraRed, and polarimetric images. All the images
	are manually labeled into eight classes according to the polarimetric characteristic of the

scenes. For example, reflective areas such as windows and water are typically considered. More recently, Sun et al.

[START_REF] Sun | A multimodal vision sensor for autonomous driving[END_REF]

developed a multimodal vision system that integrates a stereo camera, a polarization camera, and a panoramic camera. The polarization camera is mainly used to detect specular materials such as glass and puddles, potentially : Examples of multimodal image datasets mentioned in Subsection 3.2.2. For each dataset, the top image shows two modal representations of the same scene. The bottom image is the corresponding groundtruth.

Table 3 .

 3 7: Performance results of deep multimodal fusion methods on SUN RGB-D dataset.

	Method	Backbone	Input size Modality Fusion strategy Mean Acc Mean IoU
	Bayesian SegNet [88] VGG-16	-		-	45.9	30.7
	Context [109]	VGG-16	-	RGB	-	53.4	42.3
	RefineNet [108]	ResNet-152	-		-	58.5	45.9
	LSTM-CF [104]	VGG-16	426x426		Late	48.1	-
	FuseNet [69]	VGG-16	224x224		Early	48.30	37.3
	DFCN-DCRF [84]	VGG-16	480x480		Early	50.6	39.3
	S-M Fusion [102]	VGG-16	449x449		Hybrid	53.93	40.98
	LSD-GF [25]	VGG-16	417x417	RGB-D	Late	58.0	-
	SSMA [178]	ResNet-50	768x384		Hybrid	-	44.52
	RDFNet [130]	ResNet-152	-		Early	60.1	47.7
	RedNet [85]	ResNet-50	640x480		Hybrid	60.3	47.8
	CFN [107]	RefineNet-152 -		Hybrid	-	48.1
	ACNet [76]	ResNet-50	640x480		Early	-	48.1
	high-quality and large-scale multimodal image datasets are expected to cover var-
	ious scenarios. Meanwhile, inaccuracy and noise should be considered in image
	processing.

• Data alignment: data collected by image sensors should be well aligned before training. Such alignment is often referred to as multi-modality calibration, and is an essential prerequisite for effective multimodal fusion.

• Dataset construction: in the construction of multimodal datasets, we should think about 1) what kind of multimodal data do we need for the target scenarios? 2) what kind of multimodal data can provide more efficient information for specific tasks? 3)

Table 3 .

 3 8: Performance results of deep multimodal fusion methods on NYU Depth v2 dataset.

	# of classes Method	Backbone	Input size Modality Fusion strategy Mean Acc Mean IoU
		FuseNet [69]	VGG-16	320x240		Early	67.46	56.01
	13	Wang'[187]	VGG-16	-	RGB-D	Late	52.7	-
		MVCNet [118] VGG-16	320x240		Early	70.59	59.07
		Gupta' [63]	-	-		Late	35.1	-
		FuseNet [69]	VGG-16	320x240		Early	44.92	35.36
	40	Wang'[187]	VGG-16	-	RGB-D	Late	47.3	-
		MVCNet [118] VGG-16	320x240		Early	51.78	40.07
		LSD-GF [25]	VGG-16	417x417		Late	60.7	45.9
		CFN [107]	RefineNet-152 -		Hybrid	-	47.7
		ACNet [76]	ResNet-50	640x480		Early	-	48.3

Table 3 .

 3 9: Experimental results of deep multimodal fusion methods on Cityscapes dataset. Input images are uniformly resized to 768 × 384.

	Method	Backbone	Modality	Fusion strategy Mean IoU
	ERFnet [9]	-		-	62.71
	AdapNet [177]	ResNet-50 RGB	-	69.39
	AdapNet++ [178] ResNet-50		-	80.80
	AdapNet [177]	ResNet-50 Depth	-	59.25
	AdapNet++ [178] ResNet-50		-	66.36
	AdapNet++ [178] ResNet-50 HHA	-	67.66
	LFC [176]	VGG-16	RGB-D	Late	69.25
	CMoDE [177]	AdapNet		Late	71.72
	SSMA [178]	AdapNet++		Hybrid	83.44
	SSMA [178]	AdapNet++ RGB-HHA Hybrid	83.94

the comparison tables, deep multimodal fusion methods are differentiated based on the used backbone network, the type of multimodal input, and the fusion strategy.

• SUN RGB-D dataset

Firstly, we report the experimental results on the indoor scene dataset, SUN RGB-D (see Table

3

.7). Ten fusion methods and three unimodal methods are compared on this benchmark dataset. We observe that ACNet and CFN are the two top scorers with a mean IoU score of 48.1%. RedNet and RDFNet are not far behind with a score of 47.8% and 47.7%, respectively. In general, multimodal fusion methods are superior to unimodal methods, which have a similar backbone network.

Table 3 .

 3 10: Experimental results of deep multimodal fusion methods on Tokyo Multi-Spectral dataset. The image resolution in the dataset is 640 × 480.

	Method	Backbone	Modality	Mean Acc Mean IoU
	SegNet [5]	VGG-16		35.4	31.7
	PSPNet [213]	ResNet-50	RGB	44.9	39.0
	DUC-HDC [190]	ResNet-101		58.9	47.7
	MFNet [64]	VGG-16		45.1	39.7
	SegNet-4c [5]	VGG-16		49.1	42.3
	FuseNet [69]	VGG-16		52.4	45.6
	PSPNet-4c [213]	ResNet-50		51.3	46.1
	DUC-HDC-4c [190] ResNet-101 RGB-Thermal 59.3	50.1
	RTFNet [172]	ResNet-152		63.1	53.2

Table 3 .

 3

	Network	Backbone	mIoU (%) Parms. (M) Time (ms)
	PSPNet [213]	ResNet-101	81.19	56.27	172.42
	DeepLab v3 [19]	ResNet-101	81.34	58.16	79.90
	DeepLab v3+ [21] Modified Xception 82.14	43.48	127.97
	AdapNet++ [178] ResNet-50	81.34	30.20	72.92
	SSMA [178]	ResNet-50	82.31	56.44	101.95
	3.2.3.3/ MEMORY USAGE
	Another performance indicator in the implementation aspect is memory usage. Large
	memory usage may increase computation time during training and testing. In this regard,
	proper use of deep learning frameworks, GPU acceleration, appropriate batch size, and
	compressed input may be beneficial for the model training. Table 3.11 demonstrates
	the comparisons on the number of parameters and inference time for various network
	architectures. It is worth noting that the elaborated models have higher accuracy but
	may require higher memory and inference time, which leads to a greater challenge to the
	real-time performance of the autonomous navigation system.
	3.3/ SUMMARY

11: Parameters and inference time performance. The reported results on the Cityscapes dataset are collected from

[START_REF] Valada | Self-supervised model adaptation for multimodal semantic segmentation[END_REF]

.

In this chapter, we reviewed deep multimodal image segmentation from two aspects: fusion methodology and dataset. Various existing multimodal fusion methods are cate-multimodal image fusion. The state-of-the-art methods make efficient use of multimodal data, yielding an improved performance on semantic scene understanding. However, the optimal fusion strategy remains an open question in need of further exploration.

Table 4 .

 4 1: Performance of segmentation models on Freiburg Multispectral Forest dataset. EF, LF refer to early fusion and late fusion respectively. We report pixel accuracy (PA), mean accuracy (MA), mean intersection over union (MIoU), frequency weighted IoU (FWIoU) as metric to evaluate the performance.

		PA	MA	MIoU FWIoU
	RGB	92.07 89.56 79.87 86.19
	EVI	92.05 88.76 79.66 85.82
	EF	91.80 88.02 78.95 85.67
	LF	92.26 89.52 80.36 86.34
	CMnet 93.

02 90.06 81.64 87.68Table 4 .

 4 2: Comparison of deep unimodal and multimodal fusion approaches by class. We report MIoU as metric to evaluate the performance.

		Road Grass Veg/Tree	Sky
	RGB	77.18 73.47	89.78	80.66
	EVI	81.55 73.50	88.08	76.39
	EF	80.78 74.07	86.90	78.68
	LF	82.27 75.66	88.54	77.68
	CMnet 81.01 76.55	90.64	83.25
	ground truth annotations for 6 classes, which are the sky, trail, grass, vegetation, obstacle
	and others. In this unstructured forest environment, Enhanced Vegetation Index (EVI)

Table 4 .

 4

	3: Segmentation performance on POLABOT dataset
	Input Methods	PA	MA	F1	MIoU
	RGB	SegNet 87.76 81.44 87.67 64.79
	POLA SegNet 90.51 84.15 90.77 68.58
	RGB E-Fusion 90.25 85.06 90.64 69.48
	+	L-Fusion 90.02 84.28 90.11 68.81
	POLA	CMnet	90.

70 85.90 90.92 72.59 multi

  -classes. As a complement of previous analysis of training on small scale dataset, a novel data augmentation technology[START_REF] Blanchon | Outdoor Scenes Pixel-wise Semantic Segmentation using Polarimetry and Fully Convolutional Network[END_REF] applied to POLABOT dataset gives the additional guarantee for weights learning. As a result, we can find that polarimetric images in our dataset provide high quality feature information, it is a beneficial premise for further data fusion. The overall best performance in this dataset was obtained with CMnet integrating RGB and polarimetric inputs, achieving a mean IoU of 72.59%. It yields around 3% comprehensive improvements comparing with the second best methods.

Some segmentation results on the POLABOT dataset are shown in Figure

4

.7.

Table 4 .

 4 4: Performance comparison of our method with baseline models on the PO-LABOT dataset. Note that SegNet is used as the unimodal baseline and the backbone of multimodal fusion methods.

	Class	RGB	Pola	Average	LFC	CMoDE	CMFnet	CMFnet+BF2 CMFnet+BF3
	Mean IoU (%)	82.69	83.65	84.07	83.92	84.15	84.49	86.62	85.89
	Sky	94.08	94.91	94.67	93.92	95.30	94.22	94.61	95.43
	Water	83.42	86.61	84.21	82.83	83.21	86.11	88.03	85.31
	Windows	73.28	77.71	73.21	73.14	72.72	77.49	79.80	79.54
	Road	80.61	75.32	83.71	85.16	84.94	83.76	84.50	83.80
	Cars	83.89	87.29	86.41	85.64	86.78	85.65	90.42	86.83
	Building	80.53	81.82	82.26	81.83	81.69	82.39	84.05	83.92
	None	83.01	81.91	84.02	84.94	84.41	85.53	84.96	86.41

Table 4 .

 4 The ablation studies indicate that the advanced capabilities of central fusion framework are based on the premise that modalities can extract high-quality semantic features. As shown in Table4.6, we further 5: Ablation study of our method. Per class performance of our proposed framework in comparison to individual modalities with ENet baseline on Cityscapes dataset.

	Input	Mean IoU	Background	Bicycle	Person	Traffic Sign	Vehicle	Pole	Vegetation	Fence	Sidewalk	Road	Building	Sky
	RGB	54.17 67.17 33.89 25.71 20.97 80.09 22.03 83.33 23.46 54.90 95.00 62.39 83.09
	DEPTH	34.64 54.42 2.10	18.63 5.80	42.19 16.63 54.39 0.00	30.00 88.74 34.95 67.73
	CMFnet	56.58 67.16 38.48 26.57 26.01 70.71 35.62 84.56 24.54 61.37 95.35 65.32 83.20
	CMFnet+BF2	58.97 66.19 42.89 28.81 21.70 83.49 26.50 85.15 34.99 64.72 96.38 68.78 87.99
	CMFnet+BF3	50.18 64.89 25.92 3.87	15.45 83.01 19.78 83.76 14.21 41.98 93.78 67.34 88.24

provide the qualitative comparisons of segmentation per class, including the unimodal methods and central multimodal fusion methods. The proposed framework achieves significant improvements in most of the classes. Notably, CMFnet+BF2 earns a large gain of +11.53% in fence class, +9.00% in bicycle class, +9.82% in sidewalk class, compared

Table 4 .

 4 6: Performance of fusion models with ENet backbone on Cityscapes dataset.

	Input	Method	MIoU
	RGB	Unimodal	54.17
	DEPTH	Unimodal	34.64
	RGB-D	Average	55.72
		LFC	55.19
		CMoDE	56.42
		CMFnet	56.58
		CMFnet+BF2	58.97
		CMFnet+BF3	50.18

  In this work, we adopt the problem definition proposed in[START_REF] Shaban | One-shot learning for semantic segmentation[END_REF][START_REF] Siam | Adaptive masked proxies for few-shot segmentation[END_REF][START_REF] Wang | Panet: Few-shot image semantic segmentation with prototype alignment[END_REF]. Suppose there are two semantic class sets L train and L test , the few-shot segmentation model deals with a dataset D = {D train , D test } where D train and D test are composed of image samples including at least one pixel belonging to L train and L test , respectively. The training set, which contains N train image samples, can be defined as D train = (X i , Y(l) i ) N train i where X i denotes the i th training image and Y(l) i is the corresponding segmentation mask of class l. The test set is given as D test = (X i , Y(l) i ) N test i . It is important to note that the model is tested on new semantic classes that do not belong to the training set, i.e. L train ∩ L test = ∅.

	5.2.2/ PROBLEM SETTING
	The primary motivation of the few-shot segmentation is to develop a segmentation model
	with high generalizability. Given only one or a few examples, the model can produce pixel-
	level prediction with sufficient accuracy on a new category. Usually, few-shot learning is
	considered as a N-way-K-shot classification task that discriminates between N classes
	with K examples per category.
	Mask	Support Image Query Image	Prediction
		Feature Extractor
	Masked Average Pooling	Support Feature	Attention-Based Gating Query Feature
	Multiscale Feature Enhancement Module
	Figure 5.1: An overview of the proposed method (MAPnet). Given a query image of a
	new category, e.g., aeroplane, the goal of few-shot segmentation is to predict a mask of
	this category regarding only a few labeled samples.

Table 5 . 4

 54

	Methods		1-shot			5-shot
		Dense Scribble Bbox Dense Scribble Bbox
	PANet [188]	48.1	44.8	45.1	55.7	54.6	52.8
	MAPnet	48.2	44.1	45.7	56.0	53.5	53.7
	using bounding box annotations achieves higher accuracy than using scribble annota-
	tions. A potential reason could be that our method learns more representative prototypes
	within the valid region of the bounding box.

: Evaluation results of using different types of annotations in mean-IoU(%) metric.

Table 5 .

 5 5: Training and evaluation on Cityscapes-3 i dataset using 3-fold cross-validation, where i denotes the number of subsets.

	Dataset	Test classes
	Cityscapes-3 0 road, sidewalk, bus
	Cityscapes-3 1 vegetation, terrain, sky
	Cityscapes-3 2 human, car, building
	ments.

  from Cityscapes training set for training, referred to as D train . The test set D test is formed by including all the samples in Cityscapes validation set. Then we choose 9 typical cate-gories out of 30 as our target classes, containing road, sidewalk, bus, vegetation, terrain, sky, human, car, building. Following the setup of few-shot segmentation dataset PASCAL-5 i , we sample 3 classes out of all 9 categories as test label-set L test = {3i + 1, 3i + 2, 3i + 3}

Table 5 .

 5 7: Per-class mean-IoU(%) comparison of ablation studies for 1-way 1-shot semantic segmentation RGB and depth features, RDNet-concat, does not provide satisfactory results. Indeed, RDNet-concat achieves a mIoU score of 26.4%, which is higher than the score obtained with RDNet-D (25.4%) but much lower than the score obtained by RDNet-R (32.8%) for 1-way 2-shot semantic segmentation. Our method, RDNet, outperforms other unimodal networks and concatenated approach overall. RDNet achieves a mIoU score of 31.2%

	Class	RDNet RDNet-R RDNet-D
	Mean	31.2	30.2	24.0
	Road	83.0	80.9	84.4
	Sidewalk	17.8	15.7	15.7
	Bus	9.5	10.6	5.3
	Vegetation	43.1	40.2	26.9
	Terrain	8.3	10.1	6.8
	Sky	19.1	16.7	13.7
	Human	47.8	46.6	36.9
	Car	12.1	12.1	5.0
	Building	39.9	39.2	21.1

4.1.5/ EXPERIMENTS

In this subsection, we evaluate the different fusion models, and report a series of experimental results on two benchmark datasets. One is the publicly available Freiburg Multispectral Forest dataset [START_REF] Valada | Deep multispectral semantic scene understanding of forested environments using multimodal fusion[END_REF] In general, our model's performance using weak annotations is comparable to the result with pixel-wise annotations, indicating the robustness of MAPnet. We also observe that for each target class in the support images, then guide the pixel-level prediction on the query image. However, the generalization and discrimination abilities of these methods still remain to be improved, especially for complex scenes.

For the above reasons, we take inspiration from existing RGB-centric methods for few-