Dr Jean-Christophe Burie

Prof. Dr Hedi Tabia

Prof. Dr Samia Aniouz

INTRODUCTION 1.1/ AUTONOMOUS DRIVING

The concept of autonomous driving has been around for nearly a hundred years, but the first self-sufficient and truly autonomous cars appeared in the 1980s Fenton (1970) Dickmanns (2002). Since then, various automakers, including General Motors, Mercedes-Benz, Tesla, Toyota, Ford, Audi, Nissan, have developed working autonomous vehicles. Research institutions and tech giants such as Google, Waymo, NVIDIA, Uber, Autonomous Vision Group of the University of Tuebingen, Daimler AG, Max Planck Institute for Informatics, VisLab of the University of Parma, Visual Inference Lab TU Darmstadt, and many others are seriously engaged in autonomous driving. The Society of Automotive Engineers (SAE) has published the international standard J3016 international (2016) "Levels of Driving Automation" for consumers. This sets out six levels of driving automation, from SAE Level Zero (no automation) to SAE Level 5 (full autonomy), summarized in Figure 1.1.

• Level 0 (No Automation): The human driver is fully in control of the vehicle all the time.

• Level 1 (Driver Assistance): The human driver is still in control with few functions carried out by the vehicle, either lane-centering or adaptive cruise control.

• Level 2 (Partial Automation): The driver is still committed to the driving, though adaptive cruise control and lane-centering functions are taken simultaneously by the vehicle.

• Level 3 (Conditional Automation): All driving functions can be fully undertaken by the vehicle under limited conditions and will not operate unless all the required conditions are met.

Human intervention is still required when requested. Traffic jams, chauffeur are example features.

• Level 4 (High Automation): The vehicle is full in-charge to perform all driving functions without human intervention but under limited conditions.

• Level 5 (Full Automation): The vehicle perform all driving functions in any conditions. The benefits of automated vehicles are mainly in the areas of safety (comfortable driving, potential to eliminate human error, fewer accidents, lives saved, fewer injuries), efficiency, and convenience (smoother traffic flow, less congestion, stable speed profiles, and smoother driving). Also, environmental improvements (energy consumption), mobility (beneficial for the disabled, elderly, underage travelers, and those who cannot afford to own a car), increased capacity (better coordination between road users and reduced safety gaps), reduced transportation costs (in terms of time and stress), and many social and economic benefits.

1.1.1/ CHALLENGES

There are several difficulties towards full autonomy for future autonomous vehicles, and to figure them out we need to understand the autonomous driving system. The system consists of three main parts (perception, planning, and control) shown in Figure 1.2, and each part includes different tasks that are expected to be fully understood by the system. In the perception part, knowledge about the vehicle environment (including road, traffic, vehicle location, and obstacle information) is perceived by various sensors such as camera, lidar, radar, GPS, and inertial sensors. The field of computer vision includes methods for analyzing raw sensor data and processing them into meaningful structured information for understanding the environment. These methods work with the various input data from different sensors: for example, object detection and tracking might input data from a camera, LIDAR, and RADAR; traffic light detection, traffic sign classification, and lane detection input data from a camera; localization and mapping system might input data from a camera, LIDAR, and GPS. The planning part uses the results of perception and predicts the intentions of other road elements, i.e., future trajectories: based on the appropriately chosen ego trajectory, driving behavior is created and planned, deciding what explicit action the vehicle needs to take next, what is helpful in high-level route planning for the vehicle, etc. The control part is deeply coupled with the perception and The companies and researchers have been working very hard to achieve the ultimate goal of Level 5 in autonomous vehicle operation. Today, autonomous technology has reached Level 4 automation, where a vehicle can handle the majority of driving situations independently. However, they still struggle to handle complex traffic situations due to their inability to accurately perceive their surroundings. Perception errors are sure to lead to erratic behavior -and accidents: False or missed object detection, classification or tracking errors, incorrect prediction of movement, unreliable assessment of collision risk, incorrect interpretation of the scene. Therefore, perceptual errors can have potentially catastrophic consequences. Human behavioral variations and unpredictability are major challenges in planning, especially when traffic rules are not followed. According to Rasmussen (1983), there are three types of human behavior: skill-based behavior (activities that occur without conscious attention or control), rule-based behavior (activities that follow a memorized rule or procedure, often based on instructions or preparation), and knowledge-based behavior (activities during an unfamiliar situation that are achieved through previous similar experiences and the combination of rule-based or skill-based behavior). Nevertheless, predicting the behavior of other road elements is also essential for decision-making (control part) in autonomous driving.

Considering that today's challenges in autonomous driving are mainly in the perception part (scene understanding), this requires immense robustness to handle highly complex driving environments. The perception part relies heavily on an extensive infrastructure of active and passive sensors. Active sensors such as LIDAR (create 3D representations of the environment), RADAR (dynamic object detection ), GPS, and IMUs for accurate positioning provide an amorphous 3D (geometry) route for the planning part. Besides, the planning part needs semantic information such as the type of objects (e.g., vehicles, pedestrians) to consider their typical dynamics (e.g., speed, direction, position), the state of regulatory traffic elements (e.g., traffic signs and signals) to comply with traffic rules (e.g., speed limits or stopping), etc. Cameras and computer vision algorithms extract all this semantic information by performing various auxiliary tasks (explained in section 1.1.2) to interpret traffic scenes. Considerable progress has been made in improving perception to achieve scene understanding using Deep Learning-based technology (Goodfellow et al. (2016b)), i.e., equipping machines with a semantic understanding of the world to reliably identify objects and make predictions and actions. However, deep learning models bring with them the well-known shortcomings associated with these trained architectures. Also, interpreting traffic scenarios using computer vision algorithms is far more challenging and complex. Mainly, in urban areas where different road users, static and moving objects may be present, the geometric layout of roads and intersections is variable. Lighting conditions such as cast shadows from vegetation or infrastructure easily confound these image processing algorithms. Also, the limited aperture angle of on-board cameras, their low mounting point, and the limited depth perception of stereo complicate the inference problem, resulting in reliable localization of only nearby objects.

Many factors, summarized in Figure 1.3, are stocking to understandability problems for autonomous driving systems from a deep learning perspective. The researchers in the field do not fully understand the dataset, the trained model, and the learning phase. A finite training dataset cannot exhaustively cover all possible driving situations, and it is likely to under-and over-represent some specific situations. The trained model (aspects of generalizability and robustness) and the mapping function it represents are poorly understood and considered as a black box. The model is highly nonlinear and offers no guarantee of robustness, as small input changes can dramatically alter the output behavior. The learning phase is not perfectly understood. Among other things, there are no guarantees that the model will settle at a minimal point that generalizes well to new situ- ations and that the model will not be under-fitted in some situations and overfit in others. Also, during training, the model may learn to base its decisions on spurious correlations rather than using causal signals [START_REF] Zablocki | Explainability of vision-based autonomous driving systems: Review and challenges[END_REF].

1.1.2/ SEMANTIC ENVIRONMENT UNDERSTANDING Scene Understanding can be viewed as the process of adding and extracting semantic information from the sensor data characterizing a scene, or scene understanding is the analysis of a scene, taking into account the semantic and geometric context of its contents and the internal relations between them. We humans can understand a complex dynamic scene only from its projection into our eye by classifying, locating, segmenting, and identifying objects and features at one look. These tasks are performed sequentially to form a consistent process that provides an output of valuable information for semantic understanding of the projected image. Figure 1.4 (a) Raw images can be defined as images of outdoor scenes, images of urban driving scenes, or scene images with multiple dynamic (vehicles, pedestrians, cyclists, and tram) and static (buildings, sky, road, and trees) objects. Humans have the ability to classify (object type and status moving/static), specify (spatial position), identify (motion, position, direction, and velocity), and track these objects in the driving scene. In addition, humans focus their visual attention more on important or purposeful elements and ignore unnecessary ones in their field of view. These properties are usually interrelated, and humans can easily associate them with the scene at different levels. Conferring these phenomenal abilities into machine-learning systems has been a long-standing goal in the field of computer vision.

Numerous approaches and methods have been proposed to improve scene understanding and extract semantic information about the driving environment from images and videos. Deep learning (DL) Goodfellow et al. (2016b) is now ubiquitous in computer vision, which has adopted deep convolutional neural networks to understand highdimensional data, such as images and videos. Representations are learned by encoding inputs through multiple nonlinear layers and sub-sampling operations, resulting in strong image-level understanding and recognition capabilities. Thanks to significant technological advances at both the hardware (computational speed) and software (strong and robust, using multiple neural networks) levels, the methods of DL have achieved amazing results. They have been mainly used in computer vision for image recognition tasks [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] [START_REF] He | Deep residual learning for image recognition[END_REF]. Since then, one has observed the adaptation of DL methods in various computer vision and image-based scene understanding tasks, such as object detection [START_REF] Mottaghi | The role of context for object detection and semantic segmentation in the wild[END_REF], semantic segmentation [START_REF] Kemker | Algorithms for semantic segmentation of multispectral remote sensing imagery using deep learning[END_REF], motion estimation/compensation Yu et al. (2019b), depth estimation [START_REF] Jiang | Incorporating depth into both CNN and CRF for indoor semantic segmentation[END_REF], saliency prediction Borji (2019), etc. These tasks can typically be formulated as image labeling problems, where labels are assigned to a set of locations corresponding to image pixels. They differ in the requirements for human supervision and the amount of work required to generate the labels.

In recent years, much research has been done in the field of " Object Detection ", where the goal is to localize objects with a bounding box and object types or classes in an image.

Object detection algorithms restrict the semantic information to different categories, e.g., building, road, sky, trees, and sidewalk are considered in one background category, while the rest of the objects are in different categories. Object detection approaches are very efficient, especially for frequent occurrences of objects such as cars Li et al. (2020c), persons Dollar et al. (2011) due to a great number of training samples and comparatively low intra-class variance. A much stronger representation is achieved by the task " Semantic Segmentation ", which assigns a semantic category or class such as car, pedestrian, building, road, sky, etc. to each pixel in an image. It estimates the probability that the pixel belongs to a set of the defined object class. Several methods for semantic segmentation have been developed in the community and have made important contributions to the field [START_REF] Feng | Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges[END_REF], [START_REF] Lateef | Survey on semantic segmentation using deep learning techniques[END_REF]. Other commonly studied tasks for scene understanding problems include " Optical-Flow & Ego-Motion " and "Depth & Shape Estimation ", which represent different aspects of objects in a scene, i.e. object motion and geometry. Optical flow encodes temporal-visual information from image sequences and is often used to relate scene changes over time. Ego-motion is defined as the three-dimensional movement of a camera within an environment. Depth estimation refers to algorithms that aim to obtain a representation of the spatial structure of a scene.

Each of these tasks provides different cues to understanding the scene and could be correlated. The motion of the object (flow and semantic) provides specific cues to its motion pattern, and the geometry of the object provides cues to depth and shape. Several works, outlined in the literature, estimate optical flow and depth information from stereo image pairs or video sequences Ilg et al. (2017), [START_REF] Rateke | Road obstacles positional and dynamic features extraction combining object detection, stereo disparity maps and optical flow data[END_REF].

Another important task that has been explored for scene understanding is "Saliency Prediction for Visual Attention," where the goal is to detect salient regions that correspond to important objects and events in a scene and their mutual relationships. This ability is fundamental to the way humans perceive and interpret a scene. Their visual system selectively focuses its attention on salient parts and performs a detailed understanding of the most salient regions.

This work aims to emulate some of the utilities of human behavior and build image representations that can efficiently facilitate semantic information associated with the image, given some training examples of previously seen semantic concepts. Our goal is to obtain a representation that can be effectively used in applications such as autonomous driving [START_REF] Geiger | Are we ready for autonomous driving? the kitti vision benchmark suite[END_REF] and robot navigation K ümmerle et al. (2015).

OBJECTIVES AND CONTRIBUTIONS

1.2/ OBJECTIVES AND CONTRIBUTIONS

This thesis investigates, designs, implements, and evaluates classical and deep learningbased solutions for semantic analysis of the driving environment in urban scenarios, where we mainly deal with image and video processing. We restrict ourselves to the area of scene perception (understanding scene semantics and visual attention) and ways to improve it using semantic segmentation, motion estimation, depth estimation, saliency prediction for visual attention, as well as other available cues. We must consider the difficulty of having real ground truth data available to train supervised models for these tasks.

In this work, cameras are intended to be the primary control and do not cover additional sensory information (LIDAR, radar, IMU, GPS...). We seek to provide an autonomous vehicle moving in an urban environment to adequately perceive, analyze, and interpret traffic as humans do.

Objectives of this thesis are as follows:

1. Give a comprehensive overview of deep learning techniques used for semantic segmentation, which is the most studied topic in the literature for understanding urban scenes.

2.

Understand the geometric structure of the urban driving scene and the Spatiotemporal evolution of the participants (vehicle, pedestrian, cyclist, etc.). The ultimate goal is to semantically reason about the scene's evolution to provide clues that can aid in decision making and autonomous vehicle control.

3.

Understanding the processes that determine where one looks in scenes (Visual Attention) is one of the fundamental goals in the study of scene perception. The third objective is to investigate known saliency algorithms (classical and deep learning approaches) for their applicability in visual saliency for multiple objects in driving scenes.

4.

Propose a DL-based solution for visual attention that highlights road context objects as salient in driving scenes. Furthermore, we seek to ensure consistent robustness (generalization performance) and high accuracy of the solution under various adverse conditions.

We would like to point out that all of the above objectives were not necessarily implemented in the order in which they were described, but we present them in the following order for the sake of comprehensibility. The work formulates classical and deep learning methods for several vision tasks' strengths to achieve better semantics and visual cues for understanding driving scenes in cities. These tasks include many processing possibilities, e.g., semantic segmentation, instance segmentation, moving object detection, motion compensation and estimation, stereo disparity/depth estimation, and saliency prediction for visual attention. The main contributions are We begin by studying the advances and innovations in Deep Learning and semantic segmentation. There is a dearth of state-of-the-art reviews on these topics. Deep Learning is a new sub-field of machine learning that is growing at a rate that makes it very difficult to stay up to date, even following the work that is being done in semantic segmentation. This includes developing new methods, improving existing methods, and using them in new application domains. Therefore, we first created a taxonomy to classify these methods and approaches into ten different classes based on the common concepts of their architectures. We review the state-of-theart techniques and analyze their architectures to find out how they achieve their stated performances. We provide a detailed survey of publicly available datasets on which these methods have been evaluated. We also point out some open problems in semantic segmentation and their possible solutions.

Next, we developed a new framework for visual attention in driving scenarios highlighting objects in the road context as salient based on Generative Adversarial Network. We started with a review of well-known saliency algorithms, including classical and deep learning approaches, used for visual attention and tested these algorithms for their applicability to visual saliency for multiple objects in driving scenes. We add a new scheme to generate data for a model of visual attention in autonomous driving. An extensive Visual Attention Driving Database (VADD) of heatmap labels is created from publicly available driving nature datasets that contain various driving activities and environments, including rain, night, snow, highways, and urban scenes.

In the next step, we seek to extend our visual scene understanding solution by incorporating motion and distance information about the various components of the driving scene. We have developed a framework that can detect objects and extract their behavioral characteristics in terms of motion, position, velocity, and distance to better understand the driving scene. We design a moving object detection model within the framework by integrating an encoder-decoder network with a segmentation model. The approach involves two mutual tasks: Object segmentation of specific classes and binary pixel classification to predict whether the detected object is moving or static based on temporal information. We propose to use image registration as a tool to compensate for ego-motion due to the moving camera and then compute optical flow to extract the actual motion information of the moving objects. The work contributes a novel dataset for moving object detection that covers all kinds of dynamic objects.

THESIS OUTLINE

The work advances state-of-the-art tasks with effective and efficient models, and outperforms previously published approaches on some of the problems mentioned before.

Examples of the variety of methods developed and used in this thesis are shown in Figure

1.3/ THESIS OUTLINE

The main body of this thesis is divided into three chapters, each containing one or more contributions. The chapters address core computer vision tasks for scene perception: a taxonomy of deep neural network-based semantic segmentation approaches is given in Chapter 2, visual attention for urban driving in Chapter 3, and disparity estimation, moving object detection, and motion compensation/estimation for urban driving scenes in Chapter 4. For each chapter topic, the state of the art is discussed. We present formulations for deep learning architectures and discuss how they can be used to improve results for all the tasks considered. In Chapter 5, we draw general conclusions, and suggest directions for future research. In this chapter we give a comprehensive overview of semantic segmentation using the methods of Deep Learning. We have classified these methods into ten classes, according to the common concepts underlying their architectures. The categories are presented in tabular form, with each method, its main idea, the origin of its architecture, test benchmarks, and code availability. This categorization provides a complete summary of the methods, which both inspire and diverge from each other. The chapter also gives an overview of the publicly available datasets on which the studied methods have been evaluated. It also presents the evaluation matrices that were used to measure their accuracy.

A detailed analysis of the known methods and their architectures is presented to find out how they achieve their stated performances. Later, the open problems and their possible solutions are discussed.

2.2/ SEMANTIC SEGMENTATION

Semantic segmentation is the most studied research topic and core task in scene understanding. This task relates to the labeling of each pixel in an image with its corresponding (2018); in cognitive and computer sciences -saliency object detection [START_REF] Luo | Weakly-supervised semantic segmentation with saliency and incremental supervision updating[END_REF]; in agricultural sciences [START_REF] Milioto | Realtime semantic segmentation of crop and weed for precision agriculture robots leveraging background knowledge in CNNs[END_REF]; fashion -clothing categorization [START_REF] Martinsson | Semantic segmentation of fashion images using feature pyramid networks[END_REF]; in medical sciences -medical image analysis Taghanaki et al. (2021) etc. The earlier approaches used for semantic segmentation were texton forest [START_REF] Shotton | Semantic texton forests for image categorization and segmentation[END_REF], random forest based classifiers Shotton et al. (2011a), while deep learning techniques provide accurate and much faster segmentation.

2.2.1/ REVIEW -DEEP LEARNING ARCHITECTURES

The first successful application of convolutional neural networks was developed by [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. They presented an architecture called LeNet5 to read zip codes and digits and extract features at multiple locations in the image. Later, [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] published a large Deep Convolutional Neural Network (AlexNet) , which is considered one of the most influential publications in the field. AlexNet is a deeper and wider version of LeNet used for learning complex objects and object hierarchies. [START_REF] Zeiler | Visualizing and understanding convolutional networks[END_REF] It is difficult to train a deep neural network with a large number of layers. As the depth increases, the performance becomes saturated or even starts to degrade due to the vanishing gradient problem. Several solutions were proposed by [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF]) Hinton (2009) Byeon et al. (2015), but none of them seemed to really tackle the problem. He

F R A M E W O R K S E N C O D E R D E C O D E R Full-resolution Residual Networks (FRRN)

YES

The focus on VGG and ResNet approaches in recent work led to remarkable results in semantic segmentation. The residual learning frameworks follow the core idea of "skip connection", which is the main intuition behind their success. However, using them on a large scale can lead to memory problems. This pioneering work makes it possible to train deeper networks with good performance. The ability of RNN to learn long-term dependencies from sequential data and the ability to remember along the sequence makes it applicable in many computer vision tasks, including semantic segmentation. A recurrent neural network (RNN) can be very beneficial in semantic segmentation; it has recurrent connections (ability to retain previous information) and the ability to capture context in an image by modeling long-range semantic dependencies for the image.

2.2.1.4/ UPSAMPLING / DECONVOLUTION BASED METHODS

Convolutional neural network models have the ability to automatically learn high-level features via layer-by-layer propagation, while losing spatial information. One deep understanding is that spatial information lost in the down-sampling operation can be recovered by upsampling and deconvolution. Secondly, a reconstruction technique is developed to increase spatial accuracy and a refinement technique is developed to merge low level and high level features. Table 2.5 shows upsampling / deconvolution based methods. [START_REF] Noh | Learning deconvolution network for semantic segmentation[END_REF] used this idea and developed a network model by learning a deconvolution network. The convolutional network reduces the size of the activation's by feed forwarding, and the deconvolution network increases the activation's by combining unpooling and deconvolution operations. Wang et al. (2016) proposed an object-based semantic segmentation (OA-Seg) method using two networks: an object proposal network (OPN) for predicting object bounding boxes and their objectness scores, and a lightweight deconvolution neural network (Light-DCNN) for up-sampling feature maps to higher resolution. [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] [START_REF] Mamalet | Simplifying convnets for fast learning[END_REF] into a fully convolutional network and generates a probability map for an input of arbitrary size. FCN recovers spatial information from downsampling layers by adding upsampling layers to the standard convolutional network. They defined a skip architecture (shallow fine layer) that combines semantic information from a deep coarse layer with appearance information to produce a precise and deep segmentation. The basic idea was to re-architect and fine-tune the classification model (image classification) to efficiently learn from whole image inputs and whole image ground truths (semantic segmentation prediction). This leads to extending these classification models to segmentation and improving the architecture with combinations of multiple resolution layers. Figure 2.7 shows the FCN architecture. [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF] present an encoder-decoder structure for deep fully convolutional neural network called SegNet. The encoder network has the same topology as VGG without fully connected layers, followed by a decoder network (from [START_REF] Ranzato | Unsupervised learning of invariant feature hierarchies with applications to object recognition[END_REF]) for pixel-wise classification. SegNet achieves higher resolution than in FCN by using a set of decoders, each corresponding to an encoder. One key feature of Seg- Net is that it transfers information directly, rather than convolving it. SegNet has been one of the best models for handling image segmentation problems, especially for scene segmentation tasks. Lin et al. (2017b) proposed a multi-path neural network named refinement network (RefineNet). RefineNet is an encoder-decoder architecture inspired by the residual connection design [START_REF] He | Deep residual learning for image recognition[END_REF] and consists of three components: residual convolution unit (RCU), multi-resolution fusion, and chained residual pooling. The multi-path network uses features at multiple levels, it refines low-resolution features with low-level concentrated features in a recursive manner to produce high-resolution feature maps for semantic segmentation. [START_REF] Vertens | Smsnet: Semantic motion segmentation using deep convolutional neural networks[END_REF] developed an architecture for a semantic motion segmentation network (SMS-Net) consisting of three components: a section that learns motion features from generated optical flow maps, a parallel section that generates features for semantic segmentation, and a fusion section that combines both the motion and semantic features and also learns deep representations for pixelwise semantic motion segmentation. Islam et al. (2017) presented a refinement structure architecture called Label Refinement Network (LRN). LRN learns the prediction of segmentation labels at multiple levels in the network and gradually refines the results at finer scale. LRN is an encoder-decoder architecture and has monitoring at multiple levels (at each stage of the decoder). [START_REF] Zhao | Icnet for real-time semantic segmentation on high-resolution images[END_REF] proposed an image cascade network (IC-Net) that efficiently uses low resolution semantic information along with details from high resolution images. The network focuses on fusion of features from multiple layers. They proposed a cascade feature fusion (CFF) which fuses the low feature maps with the high feature maps. J égou et al. ( 2017) builds a Fully Convolutional DenseNet FC -DenseNet, extending [START_REF] Huang | Densely connected convolutional networks[END_REF] by adding an upsampling path and skipping connections to restore full input resolution. [START_REF] Bilinski | Dense decoder shortcut connections for single-pass semantic segmentation[END_REF] designed an architecture that follows an encoder-decoder strategy. The encoder is based on the ResNeXt architecture and the decoder consists of blocks (dense decoder shortcut connections ) that generate semantic feature maps and allow multi-level fusion in a single pass. [START_REF] Wu | Fully combined convolutional network with soft cost function for traffic scene parsing[END_REF] proposed a fully combined convolutional network (FCCN) to improve the upsampling operation of FCN. The network follows a layer-wise upsampling strategy, and after each upsampling operation, the size of the input feature map is doubled. They also proposed a soft cost function to further improve the training. The FCCN was ex- which is a binary classification network that selects patches that contain object edges or details that need refinement, while patches contain only background or flat regions that are more likely to be ignored. They also embed the PPN in a global-local network that contains a global branch and a refinement branch, called GRNet. GRNet consists of the global branch (generates the preliminary global-level segmentation feature of downsampling), the PPN (patch selection) and the refinement branch (feature extraction and refinement). The global-level feature and the refined local feature are fused to produce the final segmentation.

2.2.1.5/ INCREASE RESOLUTION OF FEATURE BASED METHODS

Another type of method is to restore spatial resolution by using atrous convolution [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF] and dilated convolution [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF], which can produce highresolution feature maps for dense prediction. The dilated convolution accommodates another parameter "dilation rate" (which describes the space between values in a kernel) in the convolutional layer and has the ability to expand the receptive field without losing resolution. Table 2.6 shows the increase in resolution of feature-based network models. [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF] of Google proposed a deep convolutional neural network model called DeepLab. Instead of using deconvolution, they proposed Atrous ("holes") convolution.

The Atrous algorithm was originally developed by [START_REF] Holschneider | A real-time algorithm for signal analysis with the help of the wavelet transform[END_REF] for computing the undecimated wavelet transform (UWT). The DeepLab architecture is similar to that of [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] with some modifications, converting fully-connected layers to convolutional layers, using a stride of 8 pixels, skipping sub-sampling after the last two pooling layers, and modifying the convolutional filters in the layers (increasing the length of the last three convolutional layers by twice and the first fully connected layer by four times) by introducing zeros. The proposed method is combined with fully connected conditional random fields (CRF) and is able to efficiently generate semantically accurate predictions and detailed segmentation maps. [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF] developed a convolutional network module for dense prediction that uses dilated convolutions to combine multi-scale contextual information without losing resolution, and to analyze re-scaled images for semantic segmentation. This module can be plugged into existing architectures at any resolution. (2017) and applied the depth-wise separable convolution (to reduce the computational complexity) to both Atrous Spatial Pyramid Pooling (ASPP) and the decoder modules.

A detailed network structure is presented by [START_REF] Gaihua | A Serial-Parallel Self-Attention Network Joint With Multi-Scale Dilated Convolution[END_REF]. They introduced a self-attention module based on the serial-parallel structure combined with dilated convolution instead of downsampling. The module improves the receptive field of the network, Compared to regular convolution with larger filters, atrous convolution allows to effectively enlarging the field of view of the filters without increasing the number of parameters or computational complexity. Dilated convolution is a simple but powerful alternative to deconvolution for dense prediction tasks.

2.2.1.6/ ENHANCEMENT OF FEATURES BASED METHODS

Enhancement of feature based methods include extracting features at multiple scales or from a sequence of nested regions. In deep networks for semantic segmentation, CNNs are applied to square image patches, often referred to as fixed-size kernels, centered on each pixel, where each pixel is labeled by observing a small region around it. The network covering a large and wide context (size of the receptive field) is essential for better performance, which can be achieved but increases the computational complexity. Feature extraction at multiple scales or extraction from a sequence of nested regions can be considered while ensuring computational efficiency. Table 2.7 shows the enhancement of feature-based network models. 2017) developed a semi-supervised semantic segmentation method using adversarial learning inspired by Generative Adversarial Networks (GANs) Goodfellow et al. (2014a). Later, Emre [START_REF] Yurdakul | Semantic segmentation of rgbd videos with recurrent fully convolutional neural networks[END_REF] proposed a similar approach consisting of two sub-networks; the segmentation network (for generating class probability maps) and the discriminator network (for generating spatial probability maps with both labeled and unlabeled data). A mechanism for self-attention is introduced by the Zhang et al. ( 2020), the network is based on adversarial learning and effectively considers relationships between distant spatial regions of the input image with supervision based on pixel-level ground truth data. [START_REF] Wei | Revisiting Dilated Convolution: A Simple Approach for Weakly-and Semi-Supervised Semantic Segmentation[END_REF] presented a weakly and semi-supervised approach using multiple dilated convolutions. They proposed an augmented classification network with multiple dilated convolutional (MDC) blocks that produce dense object localization maps used for semantic segmentation in both weakly and semi-supervised ways. Huang et al. (2018c) proposed a weakly supervised network that generates labels using the contextual information within an image. They proposed a seeded region growing module to find small and tiny discriminative regions of the object of interest by using image labels to generate complete and precise pixel-level labels that are used to train the semantic segmentation network.

Semi-supervised and weakly supervised learning aim to reduce the effort required for full annotation. These methods improve learning performance using weak annotations in the form of image-level labels (information about which object classes are present) and bounding boxes (coarse object locations).

2.2.1.8/ SPATIO-TEMPORAL BASED METHODS

This subsection will study the deep convolutional networks that use spatial information and temporal information for semantic segmentation. In a video, frames are associated with each other and have temporal information (i.e., features of continuous sequences of frames) that can be useful for semantic interpretation of a video. Spatio-temporal structured prediction can prove useful in both supervised and semi-supervised ways. Table 2.9 shows Spatio-Temporal based network models for semantic segmentation.

Several methods are proposed in the combination of Recurrent Neural Networks (RNN) and Convolutional Neural Network (CNN) for video segmentation. [START_REF] Fayyaz | STFCN: spatio-temporal FCN for semantic video segmentation[END_REF] presented a full convolutional network Spatio-Temporal Fully Convolutional Network (STFCN) employing spatial and temporal features. They proposed a spatio-temporal module that takes advantage of LSTM to define temporal features. The spatial feature maps of the region in a single image fed into the LSTM establish a relationship with the spatial features of equivalent regions in the images before it. Furthermore, the spatial and temporal information is fed into an dilated convolution network [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF] Integrating the conditional random field into the original architecture is a difficult task due to the additional parameters and the high computational complexity of training. Moreover, the majority of CRFs use hand-constructed color-based affinities, which may lead to spatial false predictions. Several methods have been proposed to overcome these problems and can be used as an alternative to CRFs. Table 2.12 shows network models that are an alternative to CRFs. Creating datasets is both time consuming and labor intensive, so for researchers and developers the most practical and viable approach is to use existing standard datasets that are representative enough of the domain of the problem. Some datasets have become standard and are often used by researchers to compare their work with others using stan-dard metrics for evaluation. Selecting a dataset at the beginning of research is a difficult task, so providing a comprehensive description of the dataset can help.

In Table 2.13, the datasets used by deep learning networks that are publicly available are listed. Various information is provided, such as the type of environment, the number of classes, the training/test patterns, the image resolution, the year of construction, and the best performance obtained so far (to the best of our knowledge) by the semantic segmentation models. Shotton et al. (2011b); [START_REF] Koppula | Semantic labeling of 3d point clouds for indoor scenes[END_REF]; [START_REF] Lai | A large-scale hierarchical multi-view rgb-d object dataset[END_REF] Datasets are not used for semantics, but they can be used for semantic segmentation.

2.2.3/ EVALUATION METRICS

We describe commonly used evaluation metrics for semantic segmentation. The overall performance of semantic segmentation systems can be evaluated in terms of accuracy, time, memory, and power consumption.

Accuracy:

The accuracy of the semantic segmentation system is a measure of the correctness of the segmentation, or is the ratio of the correctly segmented area to the ground truth.

Pixel wise Accuracy:

The ratio between the amount of correctly classified pixels and the total number of pixels. Confusion matrix terminology is used to describe the performance of a classification model.

Let N cls be the number of classes, N xy the number of pixels belonging to class x and labeled as class y. The confusion matrix gives the number of false positives (N xy ), false negatives (N yx ), true positives (N xx ) and true negatives (N yy ).

PixelAccuracy = Σ N cls x=1 N xx Σ N cls x=1 Σ N cls y=1 N xy (2.1)
Pixel-wise classification accuracy is not reliable for the actual performance of a classifier, as it gives misleading results if the dataset is unbalanced (i.e., large regions that have a class or labeled images might have coarser labeling).

Mean Accuracy:

The ratio of correct pixels is calculated per class and then averaged over the total number of classes N cls .

MeanAccuracy = 1 N cls Σ N cls x=1 N xx Σ N cls y=1 N xy (2.2)

Mean Intersection over Union (MIoU):

The ratio between the number of true positives N xx , (Intersection) over the sum of true positives N xx , false negatives N yx , false positives N xy (Union). Intersection over union is calculated for each class and then averaged.

MIoU = 1 N cls Σ N cls x=1 N xx Σ N cls y=1 N xy + Σ N cls y=1 N yx -N xx (2.
3)

The most widely used accuracy measuring strategy is MIoU, due to its easiness and simplicity.

Frequency Weighted Intersection over Union (FWIoU)

FWIoU = 1 Σ N cls x=1 Σ N cls y=1 N yx Σ N cls x=1 Σ N cls y=1 N xy N xx Σ N cls y=1 N xy + Σ N cls y=1 N yx -N xx

Time, Memory and Power:

The memory and processing time of the system is highly dependent on the hardware and backend implementation. The use of hardware accelerator GPUs makes the processing time of these systems very fast, but consumes a lot of memory and power. Most of the methods do not provide information related to time, memory and hardware, which is very important because these network models can be used in areas (mobile systems, robotics, autonomous driving, etc.) where extremely accurate image segmentation is required with limited power and memory. Moreover, this information can help researchers to estimate, compare or select methods depending on the application and requirement.

2.2.4/ ANALYSIS

We analyze some of the network models based on their performance on datasets and their design structure to find out the reasons for their performances. It is difficult to compare these methods because most of them were evaluated on very few datasets. Some methods used different metrics and also lack information about the experimental setup (hardware, time, memory). units and an efficient atrous spatial pyramid pooling that has a larger effective receptive field with more than 10x fewer parameters, complemented by a strong decoder with a multi-resolution supervision scheme that recovers high-resolution details. is a portable deep learning framework from Facebook that is capable of training large models, and allows machine learning applications to be built for mobile systems. DNN compression and acceleration has made a lot of progress. However, there are some potential problems such as: Compression may lead to accuracy loss; Decomposition process; Transfer of information to convolutional filters is not suitable for some networks.

PSPNet

Apply to Adverse Conditions:

There are a few network models that are used in real-world, challenging environments or deal with adverse conditions such as direct lighting, reflections from reflective surfaces, changing seasons, fog, or rain. Although some CNN models have used synthetic data along with real data to enhance the performance of stateof-the-art methods for semantic segmentation under challenging environmental conditions. However, the use of large amounts of high-quality real-world data is still indispensable so far. One possible solution is to use synthetic data together with real-world data. Obviously, there are significant visual differences between the two data domains and to reduce this gap, a domain adaptation technique can be used.

Need large and high quality labeled data:

The classification performance of DNNs and dataset size are positively correlated.

Current state-of-the-art methods require high quality labeled data, which is not available on large-scale as they are time consuming and labour exhaustive. The effective solution to this problem would be to build large and high quality datasets, which seems hard to achieve. Therefore, the researchers rely on semi and weakly supervised methods making DNNs less reliant on the labeling of large datasets.

These methods has considerably improved the semantic segmentation performance by using additional weak annotations either alone or in combination with a small number of strong annotations. However, they are far from fully supervised learning methods in terms of accuracy. Thus, this opens new challenges for improvement.

Overfitting:

As mentioned earlier, DNNs are data hungry and do not perform well unless fed with large datasets. The majority of available datasets are relatively small, so DNN models become very complex to capture all the useful information needed to solve a problem. With a limited amount of data, there is a risk of "overfitting" the model. Overfitting occurs when the gap between the training error and the test error is too large. Regularization techniques help to overcome this problem.

Regularization is any modification we make to a learning algorithm that aims to reduce its generalization error but not its training error Goodfellow et al. (2016a).

Several of these methods are applied in DNNs to prevent overfitting, e.g., L1 and L2 regularization, Lp norm, dropout, DropConnect. Data Augmentation is also used to reduce overfitting (e.g., increase training data size -rotate, flip, scale, move images). However, regularization can increase training time (e.g., using dropout increases training time by 2 or 3 times compared to a standard neural network of the same architecture) and there is no standard for regularizing CNNs. Introducing a better or improved regularization method would be an interesting direction for future work.

Segmentation in Real-time:

Real-time semantic segmentation without losing too much accuracy is of great importance as it can be useful in autonomous driving, robot interaction, and mobile computing where runtime is crucial to evaluate system performance. DNN methods for semantic segmentation are more focused on accuracy than speed. 2014), etc.) to reduce the size of the network. However, real-time semantic segmentation still lacks higher accuracy, and new methods and approaches need to be developed to find a trade-off between runtime and accuracy.

Video / 3D Segmentation:

DNNs have been successfully used for semantic segmentation of 2D images, while they are hardly used for 3D images and on videos despite their importance. Several video and 3D network models for semantic segmentation have been proposed

over the years and progress has been made, but there are still some challenges.

The lack of large datasets of 3D images and sequence images (videos) makes it difficult to make progress in semantic segmentation of 3D and video images. 3D networks are computationally expensive when dealing with high resolution and complex scenes (large number of classes). In the task of 3D semantic segmentation, the use of 3D point cloud information is very effective. have been proposed that use temporal information with spatial information to increase the accuracy of pixel labeling.

2.3/ CONCLUSION

In this chapter, a comprehensive overview of deep learning techniques used for semantic segmentation has been given. The methods reviewed have been categorized into ten classes according to the common concept underlying their architectures. A summary of these methods was also provided, indicating for each method the main idea, the origin of its architecture, test benchmarks, code availability (Table 2.14 provides links of available source codes) and year of publication. Thirty-five datasets to which these methods were applied were reported and described in detail, indicating the type of environment, number of classes, resolution, number of images, and the method that, to the best of our knowledge, achieved the best performance on each dataset. We mainly analyzed the design and performance of some of these methods that were reported to have achieved high scores. The goal was to find out how they do this. We also discussed some of the open problems and tried to suggest some of the possible solutions. The study showed that there is a lot of room for improvement in terms of accuracy, speed and complexity. the type of object is identified, i.e. vehicle, bicycle, cyclist, etc. It is also important to determine the location and movement of these objects and to be able to estimate the distance and direction of movement, i.e. whether each object detected has the potential to become a hazard to the vehicle. In this work, we will try to solve the first part of the question to detect important objects. We first review well-known saliency algorithms, both classical and deep learning, used for visual attention and evaluate their applicability to the driving environment. Followed by our new approach to visual attention for driving based on conditional Generative Adversarial Network. Then, We present our new strategy for obtaining data saliency heatmaps from existing publicly available datasets. [START_REF] Pylyshyn | Tracking multiple independent targets: Evidence for a parallel tracking mechanism[END_REF]. The theory is that each object in the visual field has a priority value for attention that is assigned in a goal-directed manner. Objects are indexed to this value and quickly attended to before other objects. [START_REF] Lee | Real-time tracking of visually attended objects in virtual environments and its application to LOD[END_REF] proposes a visual attention model that finds out an object or set of objects that could possibly receive more attention from the user without considering the position of the viewpoint. Therefore, different methods with different assumptions and predictions have been developed for modeling attention. (2017) presented a fast Bayes-based object proposal generator for night driving scenes.

3.2/ RELATED WORKS

3.2.2/ VISUAL ATTENTION USING DEEP LEARNING

3.3/ OUR APPROACH FOR VISUAL ATTENTION

After reviewing the literature, we wanted to test some of the saliency algorithms, both classical and deep, for their applicability in visual saliency for multiple objects in driving scenes. These algorithms are based on different mechanisms and use different views of saliency. Our goal is to detect important salient objects in the road context (i.e., car, pedestrians, and traffic lights/signs) that should receive more attention than other objects in the driving scene. We propose a new visual attention framework that can detect road context objects as salient and neglect other objects in a driving scene. We focus on exploring the advantages of using conditional generative adversarial network (cGAN) in our visual attention framework to generate the saliency maps from the real scene images.

3.3.1/ GENERATIVE ADVERSARIAL NETWORK

GAN is originally proposed by Goodfellow et al. (2014b). It consists of two competing convolutional neural networks: a generator (G) and a discriminator (D). The generator tries to generate random synthetic outputs (new data similar to the expected ones), while the discriminator tries to recognize if an input data is real (belongs to the original dataset)

or fake (generated). GAN can generate good quality images from a random vector similar to the real ones. Conditional GAN (cGAN) is one of the most important extensions of the original GAN, proposed by [START_REF] Mirza | Conditional generative adversarial nets[END_REF]. They add a parameter to the generator as a label that allows to condition the data generation process. The objective function is summarized as follows:

L cGAN (G, D) = E x,y [logD(x, y)]+ E z,x [log(1 -D(G(z, x), x))] (3.1)
The generator tries to minimize log(1 -D(G(z, x), x)) while discriminator tries to maximize logD(x, y), following the min-max optimization rule:

min G max D E x,y∼p data(x,y) [logD(x, y)] +E z∼p z ,x∼p x [log(1 -D(G(z, x), x))] (3.2) thus G * = arg min G max D L cGAN (G, D)
The L1 loss [START_REF] Bloomfield | Least absolute deviations: theory, applications, and algorithms[END_REF] (L L1 ) is combined with the conditional adversarial loss(L cGAN ) which encourages less blurring:

L L1 (G) = E x,y,z [ y -G(z, x) 1 ] (3.3)
The final objective is then:

G * = arg min G max D L cGAN (G, D) + λL L1 (G) (3.4)
where λ is a regularization constant which is set to 100 as reported in Isola et al. (2017).

Table 3.1 shows the summary of the notations.

3.4.1/ OBJECT/CLASS SELECTION

The objects in the driving scene can be ranked or prioritized according to their importance or relevance to safe driving. Depending on the driving situation, human drivers make decisions and prioritize more relevant objects over less relevant ones (e.g., people over animals, pedestrians over cars), but how would a machine make such decisions in advance? A good article [START_REF] Awad | The moral machine experiment[END_REF] from MIT probes public opinion on this question. Several things affect driving situations, such as each road user and object in the scene, the driver's state and experience, and also the vehicle being driven. According to the somatic marker hypothesis Fuller (2011), the attention priority given to objects in the driving scene is a function of the strength of the driver's sense of risk. The objects that receive higher ratings of sense of risk and attention are vehicles, pedestrians, traffic The shape property is so important in our application that the driver can easily recognize any highlighted object in the scene when we integrate this framework into a Advanced Driver Assistance System (ADAS) or 3D driving simulator. Moreover, the full object shape is useful for semantic segmentation because the computer can quickly process the object shapes from the heat map to segment the important classes.

Finally, we created heatmaps from the grayscale masks and overlaid them on the original images to obtain a saliency heatmap that highlights the selected class objects as the most prominent and salient regions in the images.

3.5.2/ BENCHMARKS

The framework is evaluated with three driving datasets, Berkeley Deep Drive (BDD), Cityscapes and CamVid. We used a cross-validation protocol that resulted in 3 training sessions and 9 evaluation experiments. The datasets used for each training are: 7000

for T rain BDD , 2975 for T rain Cityscapes , and 367 for T rain CamVid . For validation, we considered 1000, 2975, and 367 number of images for Val BDD , Val Cityscapes , and Val CamVid , respectively.

We also trained the model with data combining all three datasets (10342 images), and evaluated each dataset.

3.5.3/ EVALUATION METRICS

We evaluated our results using several quantitative metrics. Saliency algorithms are evaluated using MSE and PSNR.

1) Mean Squared Error (MSE), representing average of the squares of the errors between clean image and degraded noisy image.

MS E = 1 mn m-1 i=0 n-1 j=0 ||a(i, j) -b(i, j)|| 2 (3.5)
where a is the matrix data of the original clean image, b is the matrix data of the degraded noisy image. m represents the number of rows and n serves as the number of columns of the images. i and j are indexes for these rows and columns.

2) Peak Signal to Noise Ratio (PSNR) is a ratio between the maximum and minimum possible values of a changeable quantity.

PS NR = 20log 10 ( MAX a √ MS E ) (3.6)
where MAX a is the maximum value that exists in original clean image.

The results of our framework are evaluated using the SSIM, FID and WD metrics. For saliency map evaluation (comparison with SOTA methods), we used four evaluation metrics that are commonly used for saliency attention map prediction: Kullback-Leibler divergence (KL -Div), Correlation Coefficient (CC), Area under Curve -Judd (AUC-Judd) and Normalized Scanpath Saliency (NSS).

1) Structure Similarity Index (SSIM) [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF], is a widely used metric that measures the structural or perceptual difference between two images. SSIM includes important structural information (luminance and contrast), which means that nearby pixels have strong dependencies on each other and carry information about the structure of objects in the visual scene. Luminance tends to be less visible in bright regions, while contrast tends to be less visible where there is significant activity in the image. SSIM ranges from 0 to 1, the higher the better.

The SSIM metric is calculated on multiple windows of an image. The SSIM is expressed as follow:

SSIM(d, d) = (2µ d µ d + c 1 )(2σ d d + c 2 ) (µ 2 d + µ 2 d + c 1 )(σ 2 d + σ 2 d + c 2 ) (3.7)
where σ is variance, σ d d is covariance, c 1 and c 2 are two variables used to stabilize the division.

c 1 = (k 1 L) 2 and c 2 = (k 2 L) 2 .
L is the dynamic range of the pixel-values (i.e.

2 Bits/Pixel -1 ) and k 1 = 0.01 and k 2 = 0.03 by default.

FID = ||µ d -µ d|| 2 2 + Tr(Σ d + Σ d -2(Σ d Σ d) 1
2 ) (3.9)

where Tr refers to trace of matrix. Lower FID score indicates less diversity between real and generated images.

3) Wasserstein distance (WD) Huang et al. (2018a) is the measure of distance between two probability distributions P d and P d. 2018) is an asymmetric dissimilarity metric, which measures the difference between two probability distributions P d and

WD(P d , P d) = inf γ∈Γ(P d ,P d ) E (s d ,s d )∼γ [D(s d , s d)] ( 3
P d. KL(P d , P d) = i P di log( + P di + P di ) (3.11)
where is a regularization constant. Lower the KL score, better the approximation. NS S (P d , P d) = 1 N i P di × P di (3.13) where i indexes the i th pixel, and N is the total number of fixated pixels; N = i P di and

P d = P d -µ(P d )
σ(P d )

3.5.4/ RESULTS AND DISCUSSION

This section is divided into two subsections. In the first subsection, we summarize the quantitative and qualitative performance of our proposed framework. The second subsection presents the comparison of the proposed framework with the SOTA saliency and eye fixation network models.

3.5.4.1/ PROPOSED FRAMEWORK EXPERIMENTS

We trained the model on each of the three datasets and performed cross-validation to compare performance (scores in black). Table 3. 4 shows the quantitative performance on the validation datasets. We also trained the model on the combined dataset (which combines all three datasets and is called VADD) and evaluated performance (scores in blue). The model trained with the combined dataset scored low on the cityscapes validation set compared to the model trained with just cityscapes. This fact is due to that BDD constitutes around 2/3 of the combined dataset and the conditions that occur in BDD are larger than those in cityscapes (e.g., night, snow, rain, etc.). Therefore, the additional conditions may affect the GAN training ("over tuned"). The results show that the model performs better when trained with the combined training dataset. In the metrics, the SSIM score ranges from 0 -1, meaning that closer to 1 is better, and for WD / FID, the lower the better, ranges from 0 -∞.

SSIM WD FID SSIM WD FID SSIM WD FID SSIM WD FID 0 -1 0 -∞ 0 -∞ 0 -1 0 -∞ 0 -∞ 0 -1 0 -∞ 0 -∞ 0 -1 0 -∞ 0 -∞
Figure 3.9 shows the visual results of our framework on the validation set (2 samples from each subset in rows 1 2 and 3). Several objects such as vehicles ahead, cyclists, traffic lights/signs and pedestrians nearby require attention consistently, and it can be seen that our framework accurately highlights these objects as salient, similar to the targets (ground-truth). In Figure 3.10, we tested our framework on images from the validation set with different environments and adverse driving conditions, such as rain, fog, snow, night, city traffic, highways, bridges, and tunnels. We evaluated the performance, as shown in Table 3.5. The model scores slightly low on night images, which is due to the smaller number of night images in the dataset compared to the other images. The overall performance of the framework is really good compared to the targets as it detects or pays attention to important objects like vehicles, pedestrians and traffic lights while ignoring irrelevant objects like buildings, trees etc.

The performance of deep learning models decreases when they are evaluated on datasets that were not used for training. We wanted to test our framework on datasets other than those used in training. We considered two datasets; first, the EU long-term dataset developed by [START_REF] Yan | EU Long-term Dataset with Multiple Sensors for Autonomous Driving[END_REF], and second, the Synthia dataset developed by Ros et al. (2016b). The EU long term dataset is a public dataset for autonomous driving covering different environments, seasons, weather and lighting conditions. Synthia is an extensive public dataset with synthetic images for driving scenes. From Figure 3.11, we can see the promising results of our framework. We also tested our model on unseen random images from the Internet and obtained good results, as shown in Figure 3.12. In the second experiment, we trained and tested the proposed framework on the wellknown SALICON dataset [START_REF] Jiang | SALICON: Saliency in Context[END_REF]. We compared our results both quantitatively and qualitatively with SOTA methods. We calculated the mean prediction errors by applying three different metrics (AUC-Judd, NSS, and CC) on the SALICON test set. The results are shown in Table 3.7 where the scores of the compared methods are taken from the original papers. The first three methods are traditional methods and the last three are deep learning based models. As can be seen from Table 3.7, the proposed framework achieves high scores in the AUC and NSS metrics compared to the other methods. For CC metric, the proposed method provided the second better score, with 0.003 difference from SAM -ResNet, which had the best score. Our framework outperforms the BDDA model on the VADD dataset in all evaluation metrics. In Figure 3.15, we can clearly see that the fixation-based approach BDDA is not able to clearly highlight objects (with their outlines and boundaries), moreover, multiple salient objects are connected even if they are far away from each other. We also find that the BDDA model leads to a lot of wrong predictions almost in every saliency map output.

Moreover, the output saliency maps are very low resolution (80 ×60) and are scaled up to the size of the input image, which drastically reduces the accuracy of the prediction. In our framework, the saliency map is exactly the same size as the input image. Our intention in this comparison was to test how well the BDDA model performs on the proposed dataset.

We also attempted to train or fine-tune other eye fixation models for driving on the VADD dataset, but were unable to prevail. The authors only provided the demonstration code and not the training source code.

Comparison on BDDA:

The proposed framework was trained on the BDDA dataset and compared both qualitatively and quantitatively with SOTA visual attention methods, as shown in Figure 3.16 and Table 3.9, respectively. We used the metric scores and outcome-attention maps from the original papers. In Table 3.9, we can see that the proposed framework outperforms all We trained the proposed framework on the VADD dataset and compared the results vi- It can be seen that the predictions of BDD-A and Dr(eye)VE models mainly focus on the middle of the road and ignore the important objects and elements in the scene. For example, in Figure 3.17, for the raw images in row 1, the Dr(eye)VE model missed traffic signs and hardly focused on the motorcyclist, and for the first image of row 2, it missed the important pedestrian. Also, the BDD-A model missed the pedestrians, traffic lights and traffic signs, as can be for the image 1 of row 1 in Figure 3.18. Compared with the results of BDD-A and Dr(eye)VE models, our proposed system successfully detected several important objects in the scenes simultaneously.

3.6/ CONCLUSION

An autonomous driving system with the ability to pay attention to the most important objects/regions of the driving environment is very important to make safe driving decisions.

In this chapter, we presented a new visual attention framework that highlights objects in the road context as salient based on Generative Adversarial Network. We reviewed the well-known saliency algorithms, including classical and deep learning approaches, used for visual attention. We tested these algorithms for their applicability to visual saliency for multiple objects in driving scenes. We concluded that none of these algorithms can work in complex and diverse environments such as driving. We presented a new strategy of data generation and visual saliency prediction. We investigated the noise robustness of various computational saliency algorithms on images corrupted by white Gaussian noise. The VSF algorithm was found to perform better, both quantitatively and qualitatively, for constructing data ground truth. The data are obtained from publicly available driving datasets [START_REF] Yu | Bdd100k: A diverse driving video database with scalable annotation tooling[END_REF][START_REF] Fauqueur | Assisted video object labeling by joint tracking of regions and keypoints[END_REF][START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF], which contain various driving activities and environments, including rain, night, snow, highways, and urban scenes. We evaluated our results using various metrics for quantitative performance evaluation. Experimental results, quantitative and qualitative comparisons with SOTA saliency and eye fixation attention models demonstrated the ability of our framework to predict several important objects in interactive, complex, and dynamic driving environments.

3.6.1/ LIMITATIONS

Overall, the proposed framework performs effectively in all cases by predicting several important objects as salient in driving scenes. However, there are some limitations. First, the model predicts the target objects class-wise in the image. We assume that not all of these objects are demanding all the time, e.g., static cars parked on the roadside, distant cars, pedestrians walking on the sidewalk, some irrelevant advertising signs, etc. Second, in a few cases, the framework predict false regions as salient. For example, for raw image

4.2/ RELATED WORKS

In this section, we present the contributions of works that are most related to ours, i.e., scene understanding for driving by combining motion and geometry related information.

These works mainly adopt moving object detection, motion segmentation, motion compensation for ego-motion, optical flow estimation, and depth estimation of stereo visionbased systems. A few works focused on object recognition or identification in a driving scene by combining various tasks among those mentioned above.

Some recent works have focused on hybridizing learning-based and geometry-based approaches. Chen et al. (2017c) proposed an approach to detect moving objects and estimate their motion states using sequential stereo images. The proposed system is a combination of several tasks; semi-global matching algorithm to compute disparity maps, and image segmentation is performed using simple linear iterative clustering (SLIC). The relationship between superpixels is sorted into coplanar, hinge and occlusion by applying slanted plane method. The motion of each superpixel is estimated based on the extracted feature points using RANSAC algorithm. Finally, superpixels with large possibilities of forming a single target and similar in motion are merged to extract moving objects. In [START_REF] Rateke | Road obstacles positional and dynamic features extraction combining object detection, stereo disparity maps and optical flow data[END_REF], the authors presented an approach to A real-time end-to-end CNN architecture for moving object detection (RST-MODNet) is presented in [START_REF] Ramzy | RST-MODNet: Real-time Spatio-temporal Moving Object Detection for Autonomous Driving[END_REF]. The proposed architecture benefits from temporal motion information embedded in sequential images and motion color maps using optical flow images.

By [START_REF] Yoo | A multi-scale fully convolutional network for semantic labeling of 3D point clouds[END_REF], a moving object detection algorithm is developed using an object motion reflection model of motion vectors. The proposed method first generates the disparity map using stereo images and estimates the road by applying the v-disparity method of the disparity map. The motion vectors of symmetric pixels between adjacent frames are detected using optical flow (in which/where the road has been removed).

They designed a probability model for how much local motion is reflected in the motion However, their method is very computationally intensive and cannot be applied to realtime. Moreover, their dataset primarily focused on vehicles only. Yu et al. (2019b) proposed an effective method for detecting moving objects using background subtraction.

Global motion is estimated by tracking the grid-based key-points using optical flow. The authors [START_REF] Zhou | Moving object detection and segmentation in urban environments from a moving platform[END_REF] presented an approach for detecting moving objects from two consecutive stereo images. Their approach estimates the ego-motion uncertainty using the first-order error propagation model, which preserves the motion probability of each pixel. Pixels with similar depth and high motion probability are detected as moving pixels based on a graph-cut motion segmentation approach. However, the method is not robust to noise and unsuitable for real-world applications.

Our work is different in several aspects:

1. Most of the existing methods focused on detecting and tracking vehicles while ignoring the behavioral features related to movement, position, distance, and velocity.

2.

Our proposed FOI is based on vision techniques that require data only from camera sensors, unlike existing methods, which used the combination of different sensors, i.e., camera, LIDAR, radar, inertial measurement unit, and other active sensors. The usage of multiple sensors is expensive, and it adds complexity and more challenges like multiple-sensor calibration, signal synchronization, and information association.

3.

We proposed using image registration as a tool for ego-motion compensation due to the moving camera, then compute the optical flow to extract the moving objects' actual motion information in the driving scene.

4.

Our approach is more generic than SOTA as we do not assume the object to be any specific type. The proposed FOI targeted objects covering vehicles, pedestrians, cyclists, motorcyclists, and others. The available moving object detection (MOD) datasets for the driving environment only consider moving vehicles while ignoring other critical dynamic objects mentioned above. Consequently, we developed an entirely new MOD dataset containing all of these dynamic elements.

The proposed framework extract all these features in order to allow better understanding of the driving scene.

4.3/ PROPOSED FRAMEWORK

In this section, we present the components of our object identification framework that extract accurate information about each object within the driving scene. These include depth estimation using the semi-global matching algorithm, motion estimation using image registration and the optical flow method, and the moving object detection model (MOD).

We also present the constructed motion-relevant annotations used to train the proposed MOD model. Finally, we will discuss how all the information are extracted and fused to understand the scene.

4.3.1/ DISPARITY/DEPTH ESTIMATION

Disparity is the distance between two pixel values or corresponding points in stereo pair images. The distance is calculated or estimated by comparing each pixel in the left image with the corresponding pixel in the right image.

Disparity(D) = X le f t -X right (4.1)
where X le f t and X right are the same specific pixel coordinates in left, and right images, respectively, and D is the disparity value between these points. The depth (z-axis location point) can be calculated by using the disparity of the corresponding point Jain et al.

( 1995).

Depth(Z) = b * f /Disparity(D) (4.2)
f is the focal length, and b is the baseline distance between the two cameras. The disparity map is a simple image representing pixel disparity values as an intensity image, the greater the intensity values, the higher the disparities or vice versa. The depth map image can be obtained by getting the depth of every pixel.

In this work, we adopt a well known Semi-Global Matching (SGM) algorithm Hirschmuller (2008), which calculates the matching cost (pixelwise), and aggregate these matching cost (from 2, 4, 8, or 16 paths) using equations 4.3 and 4.4:

L r (p, d) = C(p, d) + min                    L r (p -r, d) L r (p -r, d + 1) + p 1 -min k L r (p -r, k) min i L r (p -r, i) + p 2 (4.3)
where p is location of interest pixel, d is disparity value, L r (p, d) is cost path toward the actual pixel of path, C(p, d) is pixel-wise matching cost, r is actual path and k is pixels in each path, p 1 and p 2 are the small and large values penalizing disparity changes between neighboring pixels of one pixel respectively.

4.3.2/ MOTION ESTIMATION

A vehicle may be driven in a driving scenario on different roads, at different speeds, daylight, conditions, seasons, and environments (e.g., urban, highway, and rural). Therefore, the situation is unpredictable while driving and is made more complex by the presence of moving objects in the scene, e.g., moving vehicles and pedestrians. The motion information of these moving objects is of great importance for safe driving in such scenarios.

Numerous methods and techniques for extracting motion information have been studied

and proposed. One of the most commonly used methods is optical flow estimation. It is expected that the accuracy of optical flow in the above scenarios or situations is good enough to ensure the reliability of the driving system. Optical flow based methods give satisfactory results when the camera is fixed or carefully displaced. However, the optical flow of image sequences captured by a moving camera encodes two pieces of information. The motion of the surrounding objects and secondly the motion of the ego vehicle result in significant motion vectors associated with the static objects, leading to a misperception of the static objects as moving objects. In this case, compensation of the camera motion is required. Therefore, we first compensate the ego-motion and later proceed with traditional optical flow method. where k 0 is an arbitrary scaling constant, p = [ x, y, 1] T , p = [x, y, 1] T , and H ∈ R 3×3

with H 33 = 1 is the unknown projective transformation matrix. Given degree of freedom

d > 3 correspondences (x i , y i ) → ( x i , y i ) d i=1 ,
H can be estimated in a least squares sense [START_REF] Forsyth | Computer vision: a modern approach[END_REF],

min h Ah 2 s.t.h 9 = 1 (4.7)
where h = vec(H) is the vectorized version of H formed by stacking its columns into a vector,

A T = [A T 1 , ..., A T d ],
with

A i =          0 p T i -y i p T i p T i 0 -x i p T i          ∈ R 2×9 (4.8)
The solution of the equation 4.7 is the smallest right singular vector of A, scaled so that the last element is 1.The method adopts the Binary Robust Invariant Scalable Keypoints (BRISK) algorithm [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF] to compute features (multi-scale corner features) from the reference and sensed images. Then, the Random Sample Consensus (RANSAC) algorithm [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF] is used to find a robust subset of the correspondences that yield a solution H of the equation 4.7. RANSAC is designed to remove the outliers from the matching features and keep only the correct matches, which The more accurate and practical way to detect a moving object in vision tasks is to understand the motion over two or more successive images. We used a simple moving object detection idea to first detect the interesting objects and then classify the moving ones.

Our approach involves two mutual tasks: Object segmentation of certain classes such as

SEGMENTATION NETWORK

We used an instance segmentation network in the object segmentation task, which provides the segmentation mask, bounding boxes, and category probabilities for each object of interest. We incorporate segmentation network Mask R- CNN He et al. (2017a) with different backbone architectures into our framework for the object segmentation task. We However, there are about 7k labels that do not contain moving objects. Many labels are ambiguous, i.e., objects are labeled in a square area, incorrect labeling of moving objects, etc. For this reason, we manually selected only the image labels where the moving objects are correctly labeled (4800 for training and 1927 for testing).

Our goal is to detect all types of moving objects in the scene. Therefore, we developed a large moving object detection dataset that covers all dynamic objects such as all types of vehicles, pedestrians, cyclists, motorcyclists, buses, trains, and trucks. Our idea for moving object detection is to first detect the objects of interest that may exhibit motion, and then identify the moving ones among them. We have adopted the mask R-CNN segmentation model which generates the segmentation masks for each instance of an object in the image. We build our new dataset using these generated segmentation masks. The object segmentation step is considered as data pre-processing/preparation for the temporal processing step. An overview of the dataset preparation flow can be found in the 4.7) generates the segmentation masks of the objects of interest for each frame (t), which is combined with the previous frame (t i-1 ), and both are fed to the EDNet, which helps the EDNet to learn the temporal relationships between the pixels and use the relationships to predict the motion class. The first step in the EDNet is a depth concatenation layer, which takes as input the binarized image masks (t i ) and (t i-1 ) and concatenates them along the third dimension before entering the first downsampling block, which consists of a convolution with a kernel size of 7 and a feature map size of 64, followed by an element-wise batch normalization layer and a ReLU operation. Next, the two remaining downsampling blocks are executed with a kernel size of 3 and a stride of 2. Then, the ED-Net extracts more learnable features through ResNet blocks (3, 6 and 9), each containing 5 operations. In the decoder, the first two blocks use a transposed convolution with a kernel size of 3, batch normalization, and ReLU layers for upsampling feature maps before running through the final transposed convolution with a kernel of 7 and softmax layers.

We chose the cross-entropy loss function to fit the predicted probability distribution (q) to the ground truth (true distribution p).

H(p, q) = -Nc i=1 p(i) log q(i) (4.9) where Nc is the number of classes, which is in our case equal to two since we want to classify moving and static objects. We trained our MOD model on the proposed moving object dataset and evaluated its performance. We split the annotated images into 85% (8550) and 15% (1509) for the training set and validation set, respectively. Out of the total 10059 mask images, 6249 masks have moving objects, and the remaining 3810 masks have no moving object (black image). We also use masks without moving objects during training, which helps the model to understand the appearance of static objects as well and also reduces the over-fitting. The evaluation is performed on images with resolution of 1242 × 375 on Nvidia GTX-2080Ti GPU. We obtain promising results (in terms of accuracy) for all segmentation models within our MOD, trained on the proposed dataset.

DIRECTION

The optical flow map gives polar coordinates of motion direction and intensity for each pixel of the detected object. We compute the average motion values by finding their mean values for the exact direction and motion intensity. The direction values can be calculated from the motion vector or color map (angle to direction and magnitude to velocity). For example, from the motion vector on the x axis for labeling if the object is motionless/static (-1 = 1). Similarly on the y axis for labeling if the object is moving away (y <= 1) or approaching (y <= -1), or is motionless (-1 < y < 1).

POSITION

The direction was discretized from the viewpoint of the target vehicle (see Figure 4.14).

The relative position of each object can be defined by the object "front left", "front" or "front right".

LABELLING ANS SCALING

The extracted information of each object is labeled and scaled in different colors, using the mapping in Figure 4.14 for visual representation. Also, the extracted information details of each detected object in the image are stored in a json file (later used for evaluation). .17, e.g., a car, a pedestrian, two cyclists. Zoom (A) shows that there are two moving cyclists, and both of them approach the ego vehicle. FOI successfully identifies their direction of movement, position with respect to the ego-vehicle, velocity, and distance from the ego-vehicle.

OBJECT-WISE SEMANTIC INFORMATION (OSI) ANNOTATIONS

In Zoom (C), a pedestrian is detected in f ront having a distance of 5.5m, who is movingaway from the ego-vehicle with medium velocity. Zoom (B) shows a car and a bus (actually a van) as static and stationary objects, respectively. The detected car is motionless and very far away from the ego-vehicle, while the bus is parked on the side of the road. The exact identification of these objects by the FOI can be seen in the json file output "Object is car status: motionless position: front direction from motionless having a distance of 34.5m or very far from ego vehicle, velocity stationery" can be seen. 4.18, seven objects are detected. Among them, three are moving objects (car, person, bicycle), and four (two cars, two bicycles) are static. The OSI of each object is labeled and highlighted. E.g., object type "car" with status "moving away" from the ego-vehicle, position "front," and the direction of movement is "left to right" with a distance of "7.2m" from the ego-vehicle, which is scaled as "close" to the ego-vehicle. The two bicycles standing on the sidewalk are correctly detected by FOI, with status "stationary," direction "motionless," position "front right," and distance "close" to the ego-vehicle. The person on the bike "in front" is "moving away," "fast" from "right to left," and "very close" to the egovehicle. Image three of the figure 4.18 shows a vehicle "in front" moving "away" from ego-vehicle at a "slow" velocity. Person "far" moving "slowly" from "left to right" and three static is robust in terms of camera movement and correct object identification. This information would help to plan the ego-trajectories based on the future states of the identified objects, thus avoiding collision risks and assisting ADAS in decision making. Except for the camera sensor, our approach does not rely on data from active sensors. Overall, FOI provides a high accuracy of 81.27% and an acceptable processing rate of 8.02 fps in multiple sequences. An important issue is the computational complexity of the proposed framework. This is mainly due to the computation of image registration and optical flow, as their estimation is often difficult and time-consuming in a complex dynamic environment.

GPU-based techniques could be used to overcome this weakness.

GENERAL CONCLUSION

This chapter summarizes the contributions of the thesis with the main conclusions and recommendations for future research. The second part of the thesis deals with an autonomous driving system that is able to pay attention to the most important objects/regions in the driving environment. We proposed a novel idea for visual attention for driving images that highlight objects in the road context as salient using a Generative Adversarial Network. We first investigated wellknown saliency algorithms, including classical and deep learning approaches, and their applicability to visual saliency for multiple objects in driving scenes. We concluded that none of these tested algorithms could work in complex and diverse environments such as driving. We developed a new strategy for data generation and visual saliency prediction.

We investigated the robustness of different algorithms for computing visual saliency for images corrupted by white Gaussian noise, and concluded that the VSF algorithm is best suited to construct the ground truth for our proposed attention system. Data are collected from publicly available driving datasets [START_REF] Yu | Bdd100k: A diverse driving video database with scalable annotation tooling[END_REF], [START_REF] Fauqueur | Assisted video object labeling by joint tracking of regions and keypoints[END_REF][START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF] that include various driving activities and environments, including rain, night, snow, highways, and urban scenes. Experimental results and quantitative and qualitative comparisons with SOTA saliency and eye fixation attention models demonstrate the ability of our system to predict several important objects in interactive, complex, and dynamic driving environments.

The environment in which an autonomous vehicle moves evolves regularly, and this evolution is closely related to its motion and semantic characteristics. In the third part of the thesis, we focused on the motion characteristics of objects in urban driving scenarios. To this end, we have proposed a framework for object identification that focuses on detecting objects from a moving camera and extracting their characteristics such as object type/class, status (moving/static), direction, distance and position to the ego-vehicle, and object velocity. The proposed work is the first approach that uses image registration The result information from the proposed framework can help ADAS or autonomous vehicles in situation interpretation (with prior knowledge such as traffic rules and knowledge from previous experiences), identify potential threats, provide more accurate warnings to a human driver, and data to an intelligent agent module responsible for decision making.

5.2/ FUTURE PERSPECTIVES

In the field of autonomous driving, there are many technical challenges that still leave much room for development. These challenges are related to sensors, computer hardware, mapping and localization, planning, decision making, and control. Our work does not address all challenges, but is limited to the part of perception in driving, including semantic segmentation, visual attention, moving object detection, motion compensation/estimation, and disparity estimation, which plays a crucial role in planning and decision making.

The perspectives of this work include, first, the development of a new network model for semantic segmentation to improve the accuracy and computational efficiency of segmentation networks for autonomous driving applications. Our study (survey) has shown that CNN-based semantic segmentation approaches suffer from higher-order inconsistencies between the ground-truth labels and the labels predicted by the segmentation model. In our approach, adversarial learning (Generative Adversarial Network) is used as a postprocessing method to make the semantic segmentation network output more realistic, refined and better structure-preserving (closer to ground-truth).

Second, the perceptual data (object identification) can be used to plan safe and smooth trajectories for the objects of interest, taking into account their dynamic limits, navigational convenience and safety, and traffic rules. One can further extend our work on object identification and add other sources of information, such as object tracking, line detection, traffic signs, and live traffic light detection, to determine the relevance of objects depending on the driving situation. Adding this additional information to the proposed framework could help prioritize the detected objects and help in various tasks such as lane change, obstacle avoidance, highlighting detected objects in different priority levels (critical, high, medium, and low), and handling critical driving situations.

Third, the deployment of the our object identification framework on a autonomous vehicles to validate this work and define the limitations and challenges in a real scenario. The biggest challenge in validating autonomous vehicles is safety (protection of road users).

Such utility is very important and therefore requires the system to be robust and reliable.

Our validation process must test whether the system can detect object attributes precisely, and whether it can function successfully in bad weather or adverse environmental conditions.

From this study, it is inferred that the existing optical flow dataset for driving scenarios, where the images are captured by a moving camera does not reflect the true distribution of the apparent motion velocities of the brightness pattern. We believe that an accurate optical flow dataset or a compensated optical flow dataset is needed that assigns an accurate and precise color to each vector based on its orientation. In the future, we will

Abstract:

Understanding urban scenes require recognizing the semantic constituents of a scene and the complex interactions between them. In this work, we explore and provide effective representations for understanding urban scenes based on in situ perception, which can be helpful for planning and decision-making in various complex urban environments and under a variety of environmental conditions.

We first present a taxonomy of deep learning methods in the area of semantic segmentation, the most studied topic in the literature for understanding urban driving scenes.

The methods are categorized based on their architectural structure and further elaborated with a discussion of their advantages, possible limitations, and future directions. Then, we proposed a new approach to visual attention for driving based on a conditional generative adversarial network. We have presented the well-known salience algorithms, both classical and Deep Learning approaches, used for visual attention. We built a large visual attention database on a new strategy for mining saliency heatmaps from existing driving datasets. We then proposed a novel object identification framework that combines motion and geometry cues to understand the urban driving environment. A new moving object detection model is developed by integrating an encoder-decoder network with semantic segmentation and a disparity estimator.

An image registration algorithm is proposed along with the optical flow to compensate for ego-motion. Extensive empirical evaluations on various driving datasets show that all the proposed methods achieve remarkable performance in terms of accuracy and demonstrate the effectiveness of the essential techniques for scene understanding in autonomous driving.

R ésum é :

La t âche de compr éhension des sc ènes urbaines n écessite la reconnaissance des constituants s émantiques de la sc ène et les interactions complexes entre eux. Par le biais de cette th èse, nous explorons et fournissons des repr ésentations efficaces pour comprendre les sc ènes urbaines bas ées sur la perception, qui peuvent être utiles pour la planification et la prise de d écision dans divers environnements urbains complexes et conditions environnementales vari ées. Nous pr ésentons d'abord une taxonomie des m éthodes d'apprentissage profond dans le domaine de la segmentation s émantique, en vue de l'int éret que porte la communaut é scientifique à ce sujet pour la compr éhension des sc ènes de conduite urbaine. Ainsi, nous avons d'abord classifi é ces m éthodes en fonction de leur structure architecturale afin d' élaborer ensuite une discussion sur leurs avantages, limites possibles et orientations futures. En suite, nous avons propos é une nouvelle approche de l'attention visuelle pour la conduite bas ée sur un r éseau g én ératif conditionnel (GAN). Pr ésentation des algorithmes de saillance bien connus, à la fois des approches classiques et De nombreuses évaluations approfondies sur divers ensembles de donn ées de conduite montrent que toutes les m éthodes propos ées atteignent des performances remarquables en termes de pr écision et d émontrent l'efficacit é des techniques essentielles pour la compr éhension de la sc ène en conduite autonome.

Figure 1 . 1 :

 11 Figure 1.1: The levels of autonomous driving.

Figure 1 . 2 :

 12 Figure 1.2: Full Autonomous Driving System

Figure 1 . 3 :

 13 Figure 1.3: Challenges and Questions in Deep Leaning

Figure 1 . 4 :

 14 Figure 1.4: Examples of the variety of methods developed and used in this thesis. (a) Semantic Segmentation (b) Visual attention for urban driving and (c) Understanding the semantics and geometry of a scene.

  semantically meaningful category. Recent work in Deep Learning dealing with semantic segmentation has been greatly enhanced by the use of neural networks. Neutral networks have made tremendous strides with the availability of large amounts of data thanks to the advent of digital cameras, cell phone cameras, and the ever-faster processing power of GPUs. Semantic segmentation has several applications in computer vision & artificial intelligence -autonomous driving Feng et al. (2020), robot navigation Zhang et al. (2018b), industrial inspection Tao et al. (2018); remote sensing Kemker et al.CHAPTER 2. DEEP SEMANTIC SEGMENTATION TAXONOMY

Figure 2 . 1 ,

 21 Figure 2.1, which is a combination of 1×1, 3×3 and 5×5 convolutional filters and a pooling layer. It reduces the number of features and operations on each layer, save time and computational cost. Later, Ioffe and Szegedy (2015) proposed an algorithm called BN-Inception for constructing, training and performing inference using Batch Normalization method. Szegedy et al. (2016) introduced two new modules, Inception V2 and Inception V3, making some significant changes (e.g., factorization of convolutions and use of grid reduction methods) to their previous module. In Szegedy et al. (2017), they replaced the filter concatenation stage of the Inception architecture with residual connections to

Figure 2 . 2 :

 22 Figure 2.2: VGG-16 Layer Structure

Figure 2 . 3 :

 23 Figure 2.3: Residual Learning: A building block

Figure 2 . 4 : 3 .

 243 Figure 2.4: The architecture of R-CNN Girshick et al. (2014)

Figure 2 .

 2 Figure 2.6: ReNet Network Visin et al. (2016)

  introduced the first Fully Convolutional Network (FCN) and achieved a breakthrough in Deep Learning based semantic segmentation. FCN architectures have become the standard in semantic segmentation; most methods use the FCN architecture. FCN covers the classification network Krizhevsky et al. (2012)Szegedy et al. (2015)

Figure 2 .

 2 Figure 2.7: FCN: Segmentation Network Long et al. (2015)

  et al. (2019a) to include a highly fused network. The proposed network consists of three main parts: Feature Down-sampling, Combined Feature Upsampling, and Multiple Predictions. The fused network uses the information of the multiple scaled features in the lower layers. Multiple soft cost functions are used to train the proposed model. Inspired by RefineNet, Park et al. (2017) presented an RGB-D fusion network (RDFNet) for semantic segmentation. The proposed architecture consists of two feature fusion blocks: the multi-modal feature fusion (MMF) to fuse features (RGB and depth) in different modalities, and the multi-level feature refinement block to further refine features for semantic segmentation. Amirul Islam et al. (2017) developed Gated Feedback Refinement Network (G-FRNet), an encoder-decoder style architecture. The proposed gated mechanism (Gate Unit) takes two feature maps in sequence, i.e., low-resolution features with larger receptive fields and high-resolution features with smaller receptive fields, and combines them to generate contextual information. The feature maps with different spatial dimensions generated by the encoder network pass through the gate unit before being fed to the decoder (feedback refinement network). The refinement network gradually refines the feature label maps. Nanfack et al. (2018) introduced an encoderdecoder architecture based on Squeeze-SegNet. The encoder module is a SqueezeNet architecture Iandola et al. (2016) (using the Fire module and removing the Average Pooling layer) inspired by SegNet and removing all fully connected layers of the VGG. The Squeeze decoder module is the inversion of the Fire module and the convolutional layers of SqueezeNet. Recently, Wu et al. (2020a) design a Patch Proposal Network (PPN),

  Figure 2.8 shows an example of a dilation convolution with different dilation rates defining the distance between values in a kernel. Treml et al. (2016) proposed an encoder-decoder structured architecture (SQNet). The encoder is a modified SqueezeNet architecture Iandola et al. (2016), called "Fire", consisting of convolutional and pooling layers. The decoder is based on a parallel dilated convolution layer. Wu et al. (2016b) present a Fully Convolutional Residual Network (FCRN), a new network for generating feature maps of arbitrary higher resolution without changing the weights. They proposed a method to simulate a high-resolution network with a low-resolution network, and an online bootstrapping method for training. Chen et al. (2017a) proposed the Atrous Spatial Pyramid Pooling (ASPP) module, which consists of multiple parallel Atrous convolutional layers with different sampling rates to strongly segment objects at multiple scales.

  Figure 2.9 shows an example of ASPP. The proposed network is based on the state-of-the-art ResNet-101 image classification DCNN. They combine the network with a fully connected Conditional Random Field (CRF) to improve object boundary localization.

Figure 2 .

 2 Figure 2.8: Dilated convolution with size of 3 × 3 with different dilation rates. (a) dilation rate = 1, receptive field = 3 × 3 (b) dilation rate = 2, receptive field = 7 × 7.

Figure 2 .

 2 Figure 2.9: Atrous Spatial Pyramid Pooling (ASPP) Chen et al. (2017a)

Farabet

  Figure 2.11: Multiscale CNN for scene parsing Farabet et al. (2013)

  Figure2.12: Pyramid Scene Parsing Network (PSPNet)[START_REF] Zhao | Pyramid scene parsing network[END_REF]

FCN

  ) for upsampling and fused for semantic predictions (summation operation).He et al. (2017b) proposed the Spatio-temporal data-driven pooling model (STD2P), a method for integrating multi-view information using superpixels and optical flow. The goal of semantic segmentation from multiple views is to exploit the potentially richer information from multiple views with better segmentation than from a single view. Qiu et al. (2018) introduced an architectural model based on 2D/3D FCNs called Deep Spatio-Temporal Fully Convolutional Networks (DST-FCN), which exploits the spatial and temporal dependencies between pixels and voxels. The proposed architecture is a network with two streams, a sequential frame stream (2DFCN for spatial and ConvLSTM for temporal information) and a clip stream (3DFCN based on C3D Tran et al. (2015) developed at voxel level). The authors Cheng et al. (2021c) propose a single-shot segmenta-tion strategy named S3-Net that locates and segments the target scene into sub-scenes (optimized object regions without background) instead of segmenting all pixels or each candidate object in a frame. The proposed model is an LSTM-based spatio-temporal model based on the structured semantic time series features extracted from the previous segmentation model for activity detection in the video stream. Some architectures are based on Gated Recurrent Architectures, to overcome the gradient problem. Siam et al. (2017) presented a fully convolutional network based on a gated-recurrent architecture (RFCN). Three different architectures were used following two approaches, conventional recurrent units (RFCLeNet) and convolutional recurrent units (RFC VGG, RFC Dilated), which learn spatio-temporal features with a smaller number of parameters. Nilsson and Sminchisescu (2016) proposed Gated Recurrent Flow Propagation network. They proposed Spatio Temporal Transformer Gated Recurrent Unit (STGRU), which combines the strength of spatial transformer (for optical flow warping) with convolution gated architecture (for adaptive propagation and fusion of estimates). Shelhamer et al. (2016) proposed a network called Clockworks, which is a combination of FCN and clockwork recurrent network Koutnik et al. (2014), where the layers of the network are grouped into stages with different clock rates (either fixed clock rate or adaptive clock) and then fused via skip connections. Saleh et al. (2017) introduced a weakly supervised framework for semantic segmentation of videos that treats both foreground and background classes equally. The basic idea is to treat multiple foreground objects and multiple background objects equally. They propose an approach to extract class-specific heat maps from the classifier that locates the different classes for both without pixel-level or bounding-box annotations.[START_REF] Kundu | Feature space optimization for semantic video segmentation[END_REF] proposed a model to optimize the feature space used by the fully connected conditional random field for Spatio-temporal regularization. Recently,Hu et al. (2021a) proposed an adaptive aggregation approach called Auto-Path Aggregation Network (APANet), in which the spatio-temporal contextual information contained in the features of each layer is selectively aggregated using the developed "auto-path". The "auto-path" links each pair of features extracted at different pyramid levels for task-specific hierarchical aggregation of contextual information, which enables selective and adaptive aggregation of pyramid features in accordance with different frames. The APANet can be further optimized together with the mask R-CNN head as a feature decoder and a Feature Pyramid Network (FPN) feature encoder, forming a joint learning system for future instance segmentation predictions. 2.2.1.9/ TRANSFORMER BASED METHODS The Transformer Vaswani et al. (2017) is encoder decoder structured network that uses multi-head attention mechanisms (MHAM) and point-wise feed-forward (PFF) networks to eliminate recurrence and convolutions, illustrated in Figure 2.13. A stack of six identical

Figure 2 .

 2 Figure 2.13: The Transformer -model structure. Vaswani et al. (2017)

  Semantic segmentation is formulated in the authors' Strudel et al. (2021) words as a problem of sequencing from one sequence to another. They propose the Segmenter transformer architecture, which uses contextual information at every stage of the model. ViT encoder is used to extract image features from the image after it has been divided into patches. The model then treats linear patch embeddings as input tokens through the ViT encoder. Later, the contextualized sequence of tokens is decoded using a pointwise linear mapping of patch embeddings to classification space, which results in the generation of class masks. Wang et al. (2021b) proposed VisTR, an end-to-end parallel sequence decoding/prediction framework based on Transformers for video instance segmentation. VisTR uses a bipartite matching loss based on instance sequence level to maintain output order, forcing one-to-one predictions. An encoder-decoder Transformer with 3D position encoding is used to model the similarity of pixel-level and instance-level features. VisTR approaches VIS from a new similarity learning angle. Instance segmentation learns pixel-level similarity while instance tracking learns inter-instance similarity. Based on ViT, Zheng et al. (2021) presented the SEgmentation TRansformer (SETR), an extension of the visual Transformer to semantic segmentation tasks. Only the class token is missing from the input-output structure of ViT's transformer encoder, which is based on CNN. More than that, it makes use of multiple decoder styles to accurately classify pixels based on progressive upsampling and multilevel feature aggregation (MLA) decoder styles. SETR shows that the Transformer encoder is a viable option for segmentation, but it requires expensive GPU clusters and additional RAMs due to the number of stack layers and quadratic computational costs associated with the task. Cheng et al. (2021a) developed MaskFormer, a parallel Transformer-CNN decoder that uses the set prediction mechanism proposed in DETR to separate mask embeddings and per-pixel features.

Figure 2 .

 2 Figure 2.14: An overview of mask head in panoptic DETR Carion et al. (2020).

Figure 2 .

 2 14 shows an overview of mask head in panoptic DETR. The model performs well on the COCO panoptic benchmark. A SegFormer transformer model is presented by Xie et al. (2021), which consists of a hierarchical pyramid Transformer as an encoder that outputs multiscale features (without position encoding) and a lightweight decoder with multiple MLP layers that combines local and global attention to produce the segmentation mask. Chen et al. (2021a) proposed TransUNet, the first visual Transformer for medical image segmentation. The structure was designed as a combination of U-Net [128] and Transformer to improve finer details by restoring localized spatial information. It encodes the tokenized image patches before directly upsampling the hidden feature representations to produce a dense output. Because of the low efficiency, SegFormer, DETR, and TransUNet Transformer-based methods cannot be used in real-time applications.2.2.1.10/ METHODS REFINING PIXEL PREDICTIONS METHODS USING CRF / MRF Semantic segmentation involves pixel-by-pixel classification, and such pixel-by-pixel classification often produces unsatisfactory results (poor, incorrect, and noisy predictions) that are inconsistent with the actual visual features of the image Arnab et al. (2018). Markov Random Field (MRF) and its variant Conditional Random Fields are classical frameworks widely used to overcome these problems. They express both unary terms (per-pixel label assignment confidence) and pairwise terms (constraints between adjacent pixels).CNNs can be trained to model unary and pairwise terms to capture contextual information. Context provides important information for scene understanding tasks, such as spatial context, which provides the semantic compatibility/incompatibility relationship between objects, scenes, and situations. CRFs can be a post-processing or end-to-end to smooth and refine pixel prediction in semantic segmentation. They combine class scores from classifiers with the information captured by the local interactions of pixels and edges or superpixels. Table2.11 shows network models with CRF. Kr ähenb ühl and Koltun (2011) proposed a fully connected CRF (DenseCRF) model in which the pairwise edge potentials are defined by a linear combination of Gaussian kernels. The method is based on the mean-field approximation, and message passing is performed using Gaussian filtering techniques Adams et al. (2010a). Methods Noh et al. (2015); Chen et al. (2014); Papandreou et al. (2015); Dai et al. (2015); Saleh et al. (2016); Khoreva et al. (2017); Wei et al. (2018); Saleh et al. (2017) coupled fully connected CRF with their proposed DCNNs to produce accurate predictions and detailed segmentation maps to improve performance. Zheng et al. (2015) formulate a mean-field inference algorithm for dense CRF with Gaussian filtering technique as a recurrent neural network (CRF-RNN) that performs CRF-based probabilistic graphical modeling for structured predictions. Figure 2.15 shows CRF as an RNN. Vemulapalli et al. (2016) proposed a model called Gaussian Mean Field (GMF) network that models unary potentials, pairwise potentials and Gaussian CRF inference for the task of semantic segmentation. In the proposed network, the output of each layer is closer to the maximum a posteriori probability (MAP) estimated for the input. Chandra and Kokkinos (2016) presented a Gaussian Conditional Random Field (G-CRF) module using a quadratic energy function that captures unary and pairwise interactions. Lin et al. (2016) introduced a model Context CNN CRF that learns CNNs and CRFs jointly. They formulate a CRF with a pairwise CNN potential to capture the contextual relationship between neighboring patches, and a sliding pyramid pooling (multiscale image network input) to capture the patch background context, which can be combined to improve segmentation.Instead of learning the potentials,Lin et al. (2015) proposed a method that learns CNN

Figure 2 .

 2 Figure 2.15: CRF as a recurrent Neural Network Zheng et al. (2015)

  (2011) (by replacing Gaussian potentials with bilateral convolution) to learn pairwise potentials from fully connected CRF.[START_REF] Barron | The fast bilateral solver[END_REF] proposed an edge-aware smoothing algorithm using a bilateral filtering technique called the bilateral solver.[START_REF] Peng | Large kernel matters improve semantic segmentation by global convolutional network[END_REF] proposed a residual based boundary refinement model, Global Convolutional network (GCN), for semantic segmentation. They proposed a Boundary Refinement Block (FCN structure without fully connected and global pooling layers) to model boundary alignment as a residual structure. Chen et al. (2016a) introduced a model with Domain Transform (DT) module as a replacement for CRF, an edge-preserving filtering method. The model consists of three modules. The first module generates a prediction of semantic segmentation results based on DeepLab. The second module named Edge Net predicts edge features from middle layers and the third module is an edge-preserving filter named Domain Transform (recursive filtering) proposed in Gastal and Oliveira (2011). The authors Chen et al. (2021b) introduced a Semantic Boundary Enhancement and position network (SBEPNet) that can detect semantic boundaries in a semantic segmentation task to improve high-level feature maps. The semantic boundaries can be efficiently obtained by explicitly exploiting the continuity of connected regions and overlaid with the original feature maps to improve the features. The Boundary Enhancement Attention Module (BEAM) is proposed to learn the longrange spatial dependencies along semantic boundaries to capture discriminative context information. Dong et al. (2021) present a lightweight boundary refinement module with point supervision named BRPS to improve the edge quality for the segmentation result produced by various existing segmentation models. Several methods have been proposed that can be used as an alternative to CRF with the advantage of speed and fewer parameters. Bilateral filtering techniques can be a useful tool in the construction of deep learning frameworks. The Figure 2.16 gives the readers an overview of the categorization of the different semantic segmentation methods.

Figure 2 .

 2 Figure 2.16: Illustration of the ten categories into which we have classified the reviewed semantic segmentation methods

Figure 2 .

 2 Figure 2.17: Getting Label Data

( 2 . 4 )

 24 Precision: The relation between true positives N xx , and all elements classified as positives Precision = N xx N xx + N xy (2.5) Recall: measures how good all the positives are found. Mean precision at a set of eleven equal space recall levels (0Mean of all the Average Precision values across all classes.

FCCN

  Zhao et al. (2017): • Competitive results are obtained on Cityscapes and Pascal VOC with 80.2% IoU and 85.4% IoU respectively. PSPNet has developed an effective optimization strategy for Deep ResNet-101 He et al. (2016) based on deeply supervised loss; two loss functions: Main softmax loss to train the final classifier and auxiliary loss applied after the fourth stage, this helps in optimizing the learning process. PSPNet applies multi-scale tests, experiments with different depths of the pre-trained ResNet and performs data augmentation. SETR Zheng et al. (2021): • Achieves the best results on ADE20K and Pascal Context with 50.28% IoU and 55.83% IoU respectively. Promising results are obtained on cityscapes with 80.2% IoU. SEgmentation TRansformer (SETR) is an encoder-decoder based network model. In SETR encoder, the stacked convolution layers with gradually reduced spatial resolution are replaced by a pure transformer Vaswani et al. (2017). This pure transformer encoder treats an input image as a sequence of image patches represented by learned patch embedding, and transforms the sequence with global self-attention modelling for discriminative feature representation learning. The authors also proposed three different designs for the decoder: Naive Upsampling (Naive), Progressive Upsampling (PUP) and Multi-Level feature Aggregation (MLA). The model achieves the best results with the MLA assumption. Yang et al. (2019a): • Achieves a scores of 69.94% IoU on CamVid and score of 44.23% IoU on ADE20K dataset. FCCN proposed a cost function that significantly improves segmentation performance. Very few researchers attempted to modify the cost function when training their models. FCCN computes the cost function on each pre-output layer including the final output layer. VPLR Zhu et al. (2019): • Achieves a top score of 82.9% IoU on CamVid, and a score of 83.5% IoU on Cityscapes dataset. A joint propagation strategy is proposed to mitigate misalignment's in synthesized patterns. The training segmentation models on datasets augmented with the synthesized samples leads to significant improvements in accuracy. The novel boundary-label relaxation technique makes training robust to annotation noise and propagation artifacts along object boundaries. DeepLab V3 Chen et al. (2017b): • Achieves score of 81.3% IoU on cityscapes. The improvement comes mainly from changing the hyper-perimeter: fine-tuning batch normalization, varying batch size, larger clipping size, changing the output stride, multiscale inputs during inference, adding left-right flipped inputs, trained on 3475 finely and additional 20000 coarsely annotated images of the Cityscapes dataset. Furthermore, using the ResNet-101 model pre-trained on ImageNet and the JFT dataset yields the second best score of 86.90 IoU on Pascal VOC. DeepLab V3+ Chen et al. (2018): • Achieves 89.0% IoU on Pascal VOC and 82.1% IoU on cityscapes. DeepLab V3+ is a modified version of DeepLab V3, adapted to output stride = 16 or 8 instead of 32. It is also adapted to the Xception module, further increasing performance. DSSPN Liang et al. (2018):

  The model is an adapted version of[START_REF] Shuai | Dagrecurrent neural networks for scene labeling[END_REF], where the pooling and convolutional layers of conv4/conv5 are replaced by two dilated convolutional layers with dilation factors of 2 and 4, respectively. This leads to a reduction in the size of the network and its runtime for real-time applications.ResNet DUC+HDCWang et al. (2018): • Achieves a score of 80.10% IoU on Cityscapes, 83.10% IoU on PASCAL VOC, 39.40% IoU on ADE20K dataset. DUC provides the dense pixel-wise predictions, HDC uses arbitrary dilation rates that increase the receptive fields of the network. Experiments are performed using ResNet at different depths, and data augmentation is applied (for cityscapes, each image in the training set is partitioned into twelve 800 × 800 patches, yielding 35700 images). The model is trained using the combination of the MS -COCO dataset, augmented PASCAL VOC 2012 training set, and the valid training set. ResNet DUC +HDC is also evaluated on the KITTI dataset and achieves an average precision of 92.88% for road segmentation using the ResNet 101-DUC model pre-trained from ImageNet during training. HMSA Tao et al. (2020): • Achieves top scores of 61.1% IoU on Mapillary Vistas, and 85.1% IoU on Cityscapes dataset. Hierarchical multiscale attention mechanism by which the network learns to predict the relative weights between adjacent scales. This requires only the addition of one additional scale to the training pipeline, whereas SOTA methods require each additional inference scale to be explicitly added during the training phase. A hard threshold based autolabelling strategy that uses unlabeled images and improves IOU. ST-Dilation Fayyaz et al. (2016): • Achieves the score of 65.90% IoU on CamVid dataset. Model ST-FCN32s scores 50.60% IoU on Camvid dataset and Model ST-FCN8s scores 30.90% IoU on NYUDv2 dataset.No post-processing is required in the STFCN model, the spatio-temporal module is embedded on the last convolutional layer. LSTM blocks are used to derive the relationships between spatial features, which provide valuable information and improve the accuracy of segmentation. Moreover, the application of dilated convolutions for contextual information at multiple layers leads to better results.STGRU (GRFP + Dilation) Nilsson and Sminchisescu (2016): • Achieves the score of 66.10 IoU on CamVid dataset. Model GRFP + Dilation scores 67.80% IoU and model GRFP + LRR-4x achieves the score of 72.80% IoU on Cityscapes dataset. The model combines the power of both convolutional-gated architecture and spatial transformers (CNN). The model GRFP is trained with Dilation 10 [88] and LRR [70] network which improve the performance for video. The model improves semantic video segmentation and labeling accuracy by propagating information from labeled video frames to nearby unlabeled frames with low computational overhead.It can be noted that the methods that achieve the high performance results do so because of the availability of a large amount of labelled data. Additional training data is beneficial to increase the accuracy of the model; several models used large datasets (merging two or three datasets) when training.2.2.5/ OPEN PROBLEMS AND POSSIBLE SOLUTIONS1. Reducing Complexity & Computation:Deep neural networks are not very suitable for use on mobile platforms (e.g., embedded devices), which have limited resources, because DNNs are memoryintensive, time-consuming, and energy-consuming. There is also a problem with computational complexity due to a large number of operations required for inference. It is important to investigate how to reduce the complexity of the model to achieve high efficiency without loss of accuracy. Some CNN compression approaches have been proposed to reduce the complexity and computational cost.Wang et al.Wang et al. (2017) proposed a method to remove and reduce the redundancy in feature maps extracted from a large number of filters in each layer of the network.[START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF] proposed a one-shot approach to compress the entire network consisting of three steps: rank selection, low-rank tensor decomposition, and fine-tuning. Andrew et al.[START_REF] Holliday | Speedup of deep learning ensembles for semantic segmentation using a model compression technique[END_REF] applied model compression techniques to the problem of semantic segmentation. Caffe2

  [START_REF] Hoffman | Fcns in the wild: Pixel-level adversarial and constraint-based adaptation[END_REF] proposed an unsupervised domain adaptation method to transfer semantic segmentation FCNs across image domains. Yang et al.[START_REF] Zhang | Curriculum domain adaptation for semantic segmentation of urban scenes[END_REF] proposed a curriculum-like learning approach to minimise the domain gap. The authors in[START_REF] Sankaranarayanan | Learning from synthetic data: Addressing domain shift for semantic segmentation[END_REF] proposed a domain shift approach based on Generative Adversarial Network (GAN), which transfers the target distribution information to the learned embedding using a generator-discriminator pair.

  The majority of methods are far from real-time segmentation. One possible solution to the problem could be to perform convolutions in an efficient way. Several works aim to develop efficient architectures that can run in real time and are based on convolution factorization (decomposition of the convolution operation into several steps). Some computationally efficient modules for convolution have been presented. For example, Inception[START_REF] Szegedy | Going deeper with convolutions[END_REF], Xception Chollet (2017), ResNet He et al. (2016), ASP Chen et al. (2017a), ESP Mehta et al. (2018); ShuffleNet Ma et al. (2018) and MobileNet Howard et al. (2017), use grouped and depthwise convolutions. Another possible solution would be to apply network compression using various techniques (e.g., parameter pruning and sharing Li et al. (2016a), low-rank factorization and sparsity Jaderberg et al. (

3 VISUALFigure 3 . 1 :

 331 Figure 3.1: Example of Visual Attention for Driving

A

  new wave of developments and improvements in saliency or attention prediction has been observed through the use of deep learning architectures. Provided with enough training data, these architectures have performed well.[START_REF] Vig | Large-scale optimization of hierarchical features for saliency prediction in natural images[END_REF] proposed the eDN (ensembles of deep networks) saliency prediction model, learns complex and plausible salient features from gaze-labeled natural images.The eDN model performs better than the DeepGaze K ümmerer et al. (2014), the first model that used transfer learning for saliency prediction. The DeepGaze model was first an end-to-end deep convolutional neural network for the saliency prediction task using Alexnet. Later, they built DeepGaze II saliency model[START_REF] Kummerer | Understanding low-and high-level contributions to fixation prediction[END_REF] based on VGGNet, which uses a pointwise nonlinear combination of deep features. Another deep learning framework SalDet, which combines global and local context in a multi-context system for saliency detection, was proposed in Zhao et al. (2015). Saliency in Context (SALICON) Huang et al. (2015) is a selective visual attention model that incorporates information at multiple scales to predict human fixations. Models such as ML -Net Cornia et al. (2016) learn hierarchies of visual features extracted by CNN to predict saliency. The saliency detection framework in Jia et al. (2016) is based on two models, a generative model that measures saliency through sparse residuals based on the background dictionary, and a discriminative model that distinguishes objects from the background using neighbourhood information. DeepFix Kruthiventi et al. (2017) network architecture was developed to capture object-level semantics at different scales and extract local/global features for predicting eye fixations and salient objects in the image. Tavakoli et al. (2017) presents the saliency prediction algorithm iSEEL based on similarities between images and an ensemble architecture (deep convolutional neural networks) that constructs saliency maps. Wang et al. (2019) presents a pyramid attentive and salient edge-aware saliency model called PAGE -Net. The authors proposed a salient edge detection module that emphasises the importance of salient edge information as it provides a strong hint for better segmentation of salient objects and refinement of object boundaries. Hsu et al. (2019) proposed a weakly supervised method for top-down saliency detection, where the idea is to focus on the regions of specific objects that indicate the presence or absence of a target object in an image. Some saliency models use the recurrent neural network as an attentional mechanism. Kuen et al. (2016) proposed a recurrent attentional convolutional-deconvolutional network (RACDNN) that continuously selects local regions and progressively refines the saliency prediction of these regions. Recurrent Mixture Density Network (RMDN) Bazzani et al. (2016) is a visual attention model that learns from human fixation data. Cornia et al. (2018) proposed a recurrent attention model called Saliency Attentive Model ( SAM ), which combines the power of a recurrent convolutional network and a fully convolutional network. The Deep Spatial Contextual Long Term Recurrent Convolutional Network (DSCLRCN) proposed by Liu and Han (2018) incorporates global and scene contexts to determine image saliency. In recent years, researchers have shown the potential application of a generative adversarial network (GAN) Goodfellow et al. (2014b) for saliency detection of images. Several GAN based saliency detection methods have been proposed to generate synthetic saliency maps. Pan et al. (2017) proposed a method called SalGAN based on convolutional encoder-decoder architecture. It consists of two networks, a generator network trained with binary cross entropy (BCE) on existing saliency maps, and a discriminator network that identifies whether the given saliency map was created from actual fixations or by the generator. A fully supervised saliency detection model Supervised Adversarial Network ( SAN ) is proposed by Pan and Jiang (2017). Zhu et al. (2018) proposed a multi-scale adversarial feature learning model (MAFL) for image saliency detection. DSAL-GAN Mukherjee et al. (2019) was developed for salient object detection in noisy images. The model uses cycle consistency loss to refine saliency. Recently, Che et al. (2019) proposed the saliency model GazeGAN, which incorporates skip connections (deep encoder/decoder layered architecture for precise salient-object localization) and center-surround connections to exploit multi-level features.3.2.3/ VISUAL ATTENTION FOR DRIVING ENVIRONMENTVisual saliency detection while driving has become an important topic for research in intelligent vehicle systems. The driving environment, especially in an urban scenario, is extremely complex and the driver should pay more attention to various objects and regions while driving. The visual saliency detected/predicted by the saliency model may not be viable for the real driving scene. There is a lack of experimental research in this area, as well as a lack of saliency datasets for driving.Currently, visual attention models for the driving environment refer to the actual attention and gaze of the human driver, as well as fixations of the region based on eye-position cues or traffic light/sign detection. Over the years, several saliency datasets for driving have been published to improve and advance these models. Work byDeng et al. (2014) Deng et al. (2016) exploited the top-down saliency mechanism and built a traffic saliency model that uses eye-tracking for saliency detection. They built a database of saliency maps for driving by recording the eye movements of some experienced and less experienced drivers. LaterDeng et al. (2017), proposed an attention model that predicts driver fixation positions using the Random Forest learning method.[START_REF] John | Saliency map generation by the convolutional neural network for real-time traffic light detection using template matching[END_REF] developed a method that identifies regions of interest in the image containing the traffic light using generated saliency maps.Yu et al. (2019a) presents a different approach for traffic sign detection, based on visual co-saliency that integrates bottom-up and top-down visual processing in an unsupervised manner. The model in[START_REF] Kim | Adaptive driver assistance system based on traffic information saliency map[END_REF] estimates driver attention based on facial features and head direction information. Tawari et al. (2018) proposed a fully convolutional RNN model to replicate the driver's gaze fixations in the driving scene videos. Kuang et al. (2017) presented a fast Bayes saliency-based object suggestion generator for night driving scenes. The model computes saliency maps based on prior estimation (via edge detection), feature extraction (luminance, local contrast, and vehicle taillight map), weight estimation (using the variance of the feature of each class), and Bayes rule. Palazzi et al. (2018) proposed a multi-branched deep architecture called DR (eye)VE Model for predicting the attentional focus of drivers. The proposed model combines raw visual scene data, motion information about optic flow, and semantic segmentation probability to predict driver attentional focus. They created a large dataset with more than 500K frames combining egocentric views (eye-tracking information) and vehicle-centric views (roof camera information). Xia et al. (2018) presented an attention model that uses driver eye movement to predict attention while driving. They developed the method Human Weighted Sampling (HWS) that identifies frames that are more critical driving moments and weights them according to their importance during training. Another huge contribution is that they created a large dataset that contains various driving scenes including driving at night, in rain, lane changing and following, turning, braking in crowded & congested situations, etc. Recently, a traffic saliency detection model was presented byDeng et al. (2020) to predict drivers' eye fixations in driving videos. They proposed a new dataset for traffic driving videos based on eye-tracking data collected from 28 experienced drivers watching driving videos. Several researchers have proposed visual attention models that examine driver attention without using eye-tracking or gaze data. These models are based on facial feature extraction Fridman et al. (2016) and head pose estimation Borghi et al. (2017). Tawari et al. (2018) proposed a fully convolutional RNN model to replicate the driver's gaze fixations in the driving scene videos. Kuang et al.

Figure 3 . 2

 32 provides the result of the tested algorithms (GBVS Harel et al. (2007), Itti Itti et al. (1998), SR Hou and Zhang (2007), SignatureSal Hou et al. (2011), ML -Net Cornia et al. (2016), BMS Zhang and Sclaroff (2013), iSEEL Tavakoli et al. (2017), VSF Montabone and Soto (2010)). All tested algorithms resulted in different saliency maps and cannot estimate the actual saliency we aim for by considering only objects in the road context.

Figure 3 . 3

 33 illustrates the schematic overview of the proposed visual attention framework (training and testing phases) in this work. First, we train GAN on a set of image pairs (input, target), where the input is an image from the real driving scene, while the target image is a saliency heatmap (built with the VSF Montabone and Soto (2010) saliency algorithm) of the same scene, highlighting the most salient objects as salient'. We then used the trained GAN to generate target heat-maps of unseen images. Subsections 3.3.1 and 3.4 provide details of the used GAN model Isola et al. (2017) used and the constructed heatmap dataset used for training and evaluation.

CHAPTER 3 .Figure 3 . 2 :Figure 3 . 3 :

 33233 Figure 3.2: Comparison of the different saliency algorithms results

Figure 3 . 4 :

 34 Figure 3.4: GAN Architecture used in our framework

3. 4 /

 4 Figure 3.7: Data gathering through different processes

Figure 3 . 8 : 2 values

 382 Figure 3.8: The results of the saliency algorithms given a noisy image. The added noise is a white Gaussian noise with different variance σ 2 values

  3.5/ EXPERIMENTAL ANALYSISWe first describe the configurations of the model GAN (training/ Testing Protocols). Then we present the data used for training & testing. Next, we present the metrics used for performance evaluation.

  As previously defined, the generator of GAN is an encoder-decoder architecture using a U-network, and the discriminator design is based on the PatchGAN model. The generator network consists of 2D convolutional blocks, batch normalization, dropout and activation layers. In the last layer of the generator, the activation function tanh is used (produces image pixel values in the range[-1,1]). In the discriminator model, we tested discriminator with two different patch sizes, 70 × 70 PatchGAN and 1 × 1 PixelGAN. These models take two concatenated images as input and classify whether the patch output is real or fake.The discriminator model is trained with real and generated images, and the generator model is trained by the discriminator model. The generator is updated to minimize the L1 loss between the target and generated images. The discriminator uses a sigmoid function in the last layer. The model is optimized with binary cross entropy, and the momentum is set to 0.5. The batch size is set to 1. The learning rate is initially set to 0.0002 and linearly decaying close to zero after 150 epochs. The learning process stops after 300 epochs. To reduce the training time, the images are resized to 512 × 512. The model is trained using an NVIDIA GTX 1080 Ti 12GB GPU, and the GAN model implementation is based on PyTorch.

2 )

 2 Frechet Inception Distance (FID)[START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF], is a well-known metric used to evaluate GANs. FID is an improved version of Inception Score[START_REF] Salimans | Improved techniques for training gans[END_REF], which uses a pre-trained inception model[START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF] (trained on ImageNet) to measure the objectiveness and diversity of generated images. FID compares the statistics of real target and generated target samples using the Frechet distance between two multivariate Gaussians.InceptionS core = E g∼P g D KL (p(y|g)||p(y))(3.8)Equation 3.8 compares the real target distribution (p(y|g) low entropy with the generated target distribution p(g) = g p(y|g)p g (g) high entropy, and KL-divergence between them.

4 )

 4 .10) where Γ(P d , P d) denotes the set of all joint distributions (i.e. probabilistic couplings), and D (s d , s d) denotes the base distance between the two sample images. The smaller the Wasserstein distance, the more similar two distributions are. Kullback-Leibler divergence (KL-Div) Bylinskii et al. (

5)

  Area Under Curve (AUC) proposed by Judd Bylinskii et al. (2016), measures the trade off between true and false positives distinguished by different thresholds using the saliency map as a binary classifier. The true positives are saliency map values above the threshold of fixed pixels, and their ratio to the total number of fixations is called the true positive rate (T P rate). The false positives are saliency map values above the threshold at non-fixed pixels, and their ratio to the total number of saliency map pixels at a given threshold is called the false positive rate (F P -rate) Bylinskii et al. (2018).6) Linear Correlation Coefficient(CC)Bylinskii et al. (2018), is a measure of the linear relationship between saliency map (P d ) and fixation map (P d).CC(P d , Pd) = σ(P d , P d) σ(P d ) × σ(P d)(3.12)where σ(P d , P d) is the covariance of (P d ) and (P d). 7) Normalized Scanpath Saliency (NSS) Bylinskii et al. (2018), is measured by taking the average of the values in a saliency map (P d ) normalized to have a mean of zero and a standard deviation of one unit at a binary map of fixation locations (P d).

Figure 3 .

 3 Figure 3.9: Visual results on VADD validation set

Figure 3 .

 3 Figure 3.11: Test on EU long-term and Synthia Datasets

Figure 3 .

 3 Figure 3.13: Comparison of the proposed framework with the saliency networks (ML-Net, SAM-Vgg and SAM-ResNet) on VADD validation set. It can be seen that, our proposed framework captures better results, more detailed and closer to the ground truth (GT) targets.

Figure 3 .

 3 14 shows a qualitative comparison of the methods on the SALICON test dataset. The predicted saliency maps of our proposed framework are much closer to the ground truth fixation maps compared to the others.

2 )

 2 Two experiments were conducted to compare the results of the proposed framework with visual attention models that use eye fixation data for prediction in driving scenes: 1) Train and test the SOTA visual attention modelBDDA Xia et al. (2018) directly on the proposed VADD dataset. Train the proposed framework on the BDDA dataset and perform comparison with SOTA methods.Comparison on VADD:We trained the Berkeley DeepDrive Attention (BDDA) model[START_REF] Xia | Predicting driver attention in critical situations[END_REF] on our proposed VADD benchmark. The authors provided the code and all the details to train and test their model, we trained the model from scratch. Table 3.8 and Figure 3.15 present the quantitative and qualitative evaluation of the BDDA model against the proposed framework on the VADD validation set.

Figure 3 .

 3 Figure 3.14: Comparison with different saliency algorithms both classic and deep leaning ones on the SALICON testing set.

Figure 3 .

 3 Figure 3.15: Comparison of the BDDA Network with our proposed framework on the VADD validation set. Our framework outcomes are better with clear object boundaries and outlines (close to GT) compared to the BDDA model.

  other methods with the lowest KL -Div and the highest AUC, NSS and CC -values. The qualitative comparison in Figure 3.16 shows that the results of our framework are very close to the ground truths compared to the others. The aim of this comparison was to test the performance and capability of the proposed framework (generative model) trained and tested on an eye fixation based driving attention dataset.

Figure 3 .

 3 Figure 3.16: Comparison of results from our proposed framework and eye fixation attention networks on the BDDA testing Dataset

  sually against Dr(eye)VE's visual attention model Palazzi et al. (2018) and the Berkeley DeepDrive Attention (BDD-A) model Xia et al. (2018) on their datasets, as shown in Figures 3.17 and 3.18.

Figure 3 .

 3 Figure 3.17: Visual Comparison with Dr(eye)VE Project results

1 in

 1 Figure 3.19, the approaching vehicle and traffic light are not detected and trees and billboards are classified as salient. Our framework also triggers false predictions due to direct sunlight or light reflections, as shown in image 2 in Figure 3.19.

Figure 3

 3 Figure 3.19: Examples of false prediction

Figure 4 . 1 ,

 41 shows specific blocks that highlight the main steps of our framework for object identification (FOI).

Figure 4 . 1 :

 41 Figure 4.1: Structure and Work-flow of the proposed Framework for Object Identification (FOI)

  detect and recognize driving obstacles by combining multiple tasks, including image segmentation, depth estimation, and motion pattern extraction using optical flow. The method achieves good results by using depth and motion patterns. However, it cannot obtain the actual motion information of obstacles due to ego-motion.Li et al. (2020c) developed an image processing based system to identify various objects and predict the intention of pedestrians in the driving scene. The proposed model integrates multiple tasks including object detection, pose estimation, intent detection, dangerous vehicle detection, and traffic light detection. The proposed system uses the YOLOv4 model for object detection, skeleton-based intent detection for pedestrian pose estimation, and explainable artificial intelligence (XAI) technology is added for risk assessment (dangerous vehicle detection and traffic light detection). Most of the existing work focused on the detection and tracking of moving objects in the driving environment. The authors in Cho et al. (2014) presented a multi-sensor fusion system for moving object detection and tracking in autonomous driving in urban environments. The proposed system has two parts; 1) sensor part composed of six radars, six LIDARs and three cameras. 2) fusion part where the measurements from different sensors are fused and presented according to their detection modalities (class, bounding box, distance, position, velocity and shape information of the objects). The system achieves promising results but requires multiple sensors which are quite expensive.[START_REF] Menze | Object scene flow for autonomous vehicles[END_REF] proposed a model that estimates 3D scene flow using geometry and motion information of a small number of objects in the scene. This information (disparity and optical flow) is extracted directly from active sensors. The authors introduced a new scene flow dataset with ground truth annotations for all static and dynamic objects in the scene. However, their approach is computationally expensive, and the proposed dataset is not large enough. InSiam et al. (2018b), the authors presented a Moving Object Detection Network (MOD -Net) model for autonomous driving that merges appearance and motion cues. They proposed from the existing KITTI dataset a new moving object detection dataset with weakly annotated segmentation masks (KITTI-MoSeg). Furthermore, Rashed et al. (2019) proposed a CNN (Fuse-MODNet) architecture for moving object detection by fusing RGB and LiDAR information. They provided an extended version of the KITTI-MoSeg dataset, the Dark-KITTI dataset, to simulate low-light driving environments.

  vector to determine if the object is moving. The authorsWu et al. (2020b) proposed a separate-predict-composite model for predicting future frames. Within the model, an encoder-decoder-based architecture for dynamic object detection is presented to identify objects between two classes, moving or static. The model takes multiple inputs (image sequences, semantic map, instance map and optical flow) and generates a binary mask to indicate the region of each moving object. Jung et al. (2020) proposed a foreground/background extraction based method for detecting moving objects from a moving camera using an inertial measurement sensor (IMU). The method used the Harris detector to extract points of interest, and epipolar geometry to classify the foreground (through the extracted map from image registration) and background feature points from successive images. Lee et al. (2020) developed a moving object detection and tracking method based on the interaction between Static Obstacle Map, which represents static obstacles, and Geometric Model-Free Approach for tracking moving objects, using point cloud information. A few methods deal with the simultaneous estimation of the ego-motion of the vehicle and the motion of multiple moving objects in the scene. The authors in Vertens et al. (2017) propose an architecture for a semantic motion segmentation network (SMS-Net) that learns to predict both the semantic category and the motion state of each pixel from a pair of consecutive monocular images. They created their motion dataset (Cityscape-100CHAPTER 4. SEMANTIC-AWARE OBJECT IDENTIFICATION IN URBAN DRIVING SCENARIOS KITTI-Motion), which contains over 3,155 manually annotated semantic motion labels.

102CHAPTER 4 .

 4 SEMANTIC-AWARE OBJECT IDENTIFICATION IN URBAN DRIVING SCENARIOS S (p, d) minimizing the aggregated cost values (equation 4.5), disparity for each pixel is calculated. S = min d S (p, d) (4.5)The SGM algorithm is faster than global matching algorithms and efficient compared to other methodsHirschmuller and Scharstein (2008).

Figure 4 . 2

 42 show the disparity map example using SGM method. More details about SGM are given in the literatureHirschmuller (2008).

Figure 4 . 2 :

 42 Figure 4.2: Disparity Map example on KITTI

4. 3 . 2 . 1 /

 321 APPROACH TO MOTION COMPENSATION Inspired by recent trends in aerial Zhang and Zhu (2020)and medical imaging Li et al. (2020a), we suggest a method called image registration for motion compensation. Image registration involves superimposing two or more images taken at different times, from different vantage points, and at different angles to obtain a 2D or 3D perspective. Various techniques are used for image registration such as wavelets, Fourier transform, correlation methods and feature based approaches. Image registration is done in four steps namely feature detection, feature matching, transformation model estimation, resampling of image and transformation. We used the image registration method presented in Forsyth and Ponce (2002) to compensate for the ego-motion of two consecutive images. The method relates different views of a scene via homographic transformations, finds and extracts features on one image (reference image), and matches them with the corresponding image (sensed image). Each considering pixel point (x, y) in the reference image and its corresponding pixel point ( x, y) in the sensed image can be related through projective transformation k p = H T p (4.6)

104CHAPTER 4 .

 4 SEMANTIC-AWARE OBJECT IDENTIFICATION IN URBAN DRIVING SCENARIOS are used to estimate the registration parameters. The nature of the transformation is projective. We finally obtain a registered image in which the background becomes stable, and we call it compensated background image.

Figure 4 . 3

 43 shows the workflow of image registration.

Figure 4 . 3 :

 43 Figure 4.3: Work Flow of Image Registration

Figure 4 . 4 : 4 . 4 ,

 4444 Figure 4.4: Example of optical flow with flow-vectors

Figure 4 . 5 :

 45 Figure 4.5: Left to right: (a) First image from a pair. (b) Flow vectors without image registration. (c) Flow vectors with image registration. (d) Flow Velocity difference

106CHAPTER 4 .

 4 SEMANTIC-AWARE OBJECT IDENTIFICATION IN URBAN DRIVING SCENARIOS

  used the model implemented byWu et al. (2019a) based on Feature Pyramid Network (FPN), ResNet-50 and ResNet-101 backbone trained onMSCOCO Lin et al. (2014b) dataset. We also explore another more effective and faster instance segmentation network with two different backbone architectures (ResNet-101-FPN, VoVNetV2) recently proposed by[START_REF] Lee | Centermask: Real-time anchor-free instance segmentation[END_REF], called CenterMask. We used these networks for the following reasons: They are state-of-the-art instance segmentation networks with the highest classification accuracy and high speed. Table4.1 shows their detection performance on the COCO dataset reported inWu et al. (2019a)[START_REF] Lee | Centermask: Real-time anchor-free instance segmentation[END_REF].

Figure 4 .

 4 Figure 4.7: Structure and Flow of two mutual tasks for moving object detection (MOD).

Figure 4 .

 4 7 illustrates the structure and flow of two mutual tasks for moving object detection. The input of EDNet is the concatenation of two consecutive masks (temporal information) of objects of interest generated by segmentation network. The masks contain both moving and static objects of the scene. The EDNet then further classifies them and extracts only the moving objects using back-propagation training according to the ground truths. 4.3.3.1/ MOD DATASETS There are few datasets for detecting moving objects in a driving environment. The existing publicly available MOD datasets focused only on vehicles with object categories of cars, trucks and vans (summarized and compared in Table 4.2). KITTI-Motion contains 273 training and 230 test images, while 1300 training and 349 test images are provided for KITTI-MoSeg. The extended KittiMoSeg dataset offers more than 12k binary mask labels (10222 training and 2697 test images) for different sequence runs from the KITTI dataset.

108CHAPTER 4 .

 4 SEMANTIC-AWARE OBJECT IDENTIFICATION IN URBAN DRIVING SCENARIOS PROPOSED MOD DATASET

Figure 4 . 8 .

 48 From the segmentation masks, we quickly obtain the masks of the objects of interest (c). Next, we manually annotated the masks of objects of interest for relevance to object motion from sequence frames (manually identifying objects from multiple frames and keeping moving). We used different sequences from the KITTI raw dataset[START_REF] Geiger | The KITTI vision benchmark suite[END_REF] and EU long term dataset[START_REF] Yan | EU Long-term Dataset with Multiple Sensors for Autonomous Driving[END_REF] to create a total of 10059 semantic segmentation mask images (with static/moving objects of interest) with corresponding annotated binary mask labels (with moving objects only). Each binary mask label for moving objects is created from the corresponding sequence pair images. Table4.2 shows a summary comparison of our dataset with existing available MOD datasets.

Figure 4 . 8 :

 48 Figure 4.8: Flow for generating motion relevant annotations. (a) input image (b) model generate bounding boxes and segmentation masks for each instance of an object in the frame (c) objects of interest mask (d) manually annotated moving objects mask

110CHAPTER 4 .F1

 4 SEMANTIC-AWARE OBJECT IDENTIFICATION IN URBAN DRIVING SCENARIOS 4.3.3.3/ MOD EXPERIMENTS We first start with the metrics used for the evaluation. Then, we present the proposed Moving Object Detection model, training and testing parameter, proposed MOD dataset, evaluation and comparison with state-of-the-art methods for moving object detection on existing MOD datasets.EVALUATION METRICSWe evaluate our MOD model using different metrics; a standard mean intersection over union (mIoU) metric, Precision/Recall, and F1 score.IoU: Intersection over union can be computed for class as followsIoU =T P (T P + FP + FN) (4.10)where T P, FP and FN correspond to true positives, false positive and false negative respectively. Then, the mIoU is the average of the computed IoU s regarding the is the number of classes and IoU i is the Intersection over union calculated for i th class.Precision: Describes the purity of positive detections relative to the ground truth It is the harmonic mean of the precision and recall

Figure 4 .Figure 4 .

 44 Figure 4.9 presents the qualitative results of our MOD model on the proposed dataset with complex scenes. It can be seen that the model accurately segments the moving objects including vehicles, pedestrians, cyclists and motorcyclists in different sequence passes.

4. 3 . 4 /

 34 MOTION COMPENSATION -FULLY COMPENSATED OPTICAL FLOWWe integrate the optical flow results with the moving object detector to obtain the flow map for moving objects only, which we call the Fully Compensated Optical Flow (FCOF) color map, as shown in Figure4.13. So we can say that we fully compensate the moving camera's ego-motion from these resulting flow maps. The sought motion information such as direction, position and velocity are extracted from the specific pixel values of each object, which allows us to create a detailed motion analysis for each object in the driving scene.

Figure 4 .

 4 Figure 4.11: Qualitative comparison against MODNet Siam et al. (2018b) on KITTI-MoSeg.

4. 3

 3 .5/ FUSION OF MOD, FCOF AND DISPARITY The results of each stage of the proposed framework, such as disparity, moving object detection, and motion estimation, are fused to extract information such as object ID, static or moving, distance, direction, position, and velocity. The pseudo-code for the information extraction is given in Algorithm 2.

Figure 4 .

 4 Figure 4.12: Qualitative comparison against Fuse-MODNet Rashed et al. (2019), RST-MODNet Ramzy et al. (2019), and U 2 -ONet Wang et al. (2021a) on KITTI-MoSeg Extended.

Figure 4 .

 4 Figure 4.13: Left to right: (a) Frames (b) Detected moving objects masks by (MOD) Model (c) registered Optical flow maps after image registration (d) Fully compensated optical flow color maps (combining (b) and (c))

Figure 4 .

 4 Figure 4.14: Qualitative Mapping

Figure 4 .

 4 Figure 4.15: Manually annotated Object-wise Semantic Information (OSI) from two consecutive images (t and t + 1).

Figure 4 .

 4 Figure 4.16 defines the structure of GT and Pred OSIs.

Figure 4 .

 4 Figure 4.17: Example predicted object identification f OI P (Highlighting the labeling and colorized scaling) of sample frame 30. The frame

Figure 4 .

 4 Figure 4.17 shows an example of the predicted output f OI P and json file json f ile generated by the proposed FOI for an input frame f 30 . A total of seven objects are detected, five of which are moving and two of which are static. The moving objects are labeled/segmented and colored green, while the static objects are only labeled with a bounding box. Each object is shown with the labels ID /class and the distance in meters above the bounding box. The black arrow in the center of the bounding box indicates the direction of the object and marks it in the generated output json file json f ile as approaching, moving away, right to left, and left to right. The distance, position, and velocity of each object are color-coded according to scale and displayed in the bounding box of the corresponding object in the upper right corner. In the city scenario example f OI P , different object types occur 4.17, e.g., a car, a pedestrian, two cyclists.

  The white arrow in the lower left corner shows the direction of movement of the egovehicle and the frame number (shown in zoom (D)). The qualitative results of FOI on different sequence runs are shown in Figures 4.18, 4.19, and 4.20, respectively.

128CHAPTER 4 .

 4 Figure 4.18: FOI Results on different sequence runs.

Figure 4 .

 4 Figure 4.19: FOI Results on different sequence runs.

130CHAPTER 4 .

 4 Figure 4.20: FOI Results on different sequence runs.

  vehicles, one in front and two in front right of ego-vehicle. A motorcycle approaching fast can be seen in the first image of the figure 4.19, along with static cars parked on the side of the road very far from the ego-vehicle. Image three of figure 4.19 shows several pedestrians moving in different directions and taking different positions. In figure 4.20 image two, the three cars in front are moving very fast. Detailed results containing consecutive frames of our proposed FOI could be found at https://youtube/.... The results show that our proposed FOI achieves promising results in identifying the objects within the driving scene. The objects include vehicles, buses, trucks, trains, pedestrians, cyclists and motorcyclists. The performance of the proposed FOI could be affected by several typical factors, including object motion speed, ego-vehicle speed, overlapping or very close objects, object reflectance, and object size. It was found that when the velocities of the moving object and the ego-vehicle are the same (for the same direction) or cancel (for the opposite direction), the system detects the object as static. e.g., in column two image 3 of Figure 4.21, the blue vehicle approaches the ego-vehicle and is detected as static. A similar thing happens in column two image 4 of Figure 4.21, the red car moving in front of the ego vehicle is detected as static. The problem also occurs when the object is far away from the ego-vehicle and moving very slowly. The images in column one of Figure 4.21 Q Objects overlap or very close to each other Q Object reflection Q Object motion speed is same as ego-vehicle speed

Figure 4 .

 4 Figure 4.21: FOI False Detections, Affected by object motion speed/ego-vehicle speed, objects overlap, object reflection etc.

5. 1 /

 1 SUMMARY OF THE PHD THESISIn this thesis, we focused on the problem of visual scene understanding by recognizing the semantic constituents of a driving scene. The underlying theme of this thesis was to investigate, design, implement and evaluate the Deep Learning based solutions for semantic analysis of the driving environment in urban scenarios. We have proposed several novel Deep Learning methods for visual scene understanding using only image data. We described several theoretical contributions for the proposed methods, reported qualitative and quantitative results through extensive experimental evaluation on standard benchmarks and in different real-world environments, discussed related work, and demonstrated that our proposed architectures substantially exceed the state-of-the-art.At the beginning, we gave a comprehensive overview of deep-learning-based methods for semantic segmentation, a very well-studied topic and one of the fundamental problems in scene understanding. Existing deep methods are grouped according to a common taxonomy: Concept (fully convolutional, encoder-decoder architectures, multiscale and pyramid-based approaches, Atrous/Dilated convolutional models, recurrent networks, regional proposal-based methods, Transformers, generative models in adversarial setting, context-aware models, semi-supervised and weakly supervised methods), network architecture (highlighting their contributions to model design), architecture origin (inspire or deviate from previous SOTA methods), test benchmarks, and code availability. A detailed review of publicly available benchmark datasets is presented, including data type/nature, number of classes, image resolution, year of publication, and peak performance achieved by the network model (up to the submission of this thesis). Furthermore, we described the common evaluation metrics for semantic segmentation. In addition, we explored the similarity, strengths, and challenges of the deep learning models based on their design strategies and evaluated performance. Moreover, we discussed some open problems and their possible solutions for deep learning-based semantic segmentation. The aim of this study was to provide the reader with a comprehensive and heuristic overview of deep learning based semantic segmentation techniques. The comprehensive description of network architecture design and datasets can help new researchers to strengthen their understanding, make comparisons or select methods and datasets according to their application and requirements.

  along with optical flow estimation to extract the actual motion of moving objects from the moving camera. We introduced a deep neural network (MOD) moving object detection model based on the combination of a segmentation network with an encoder-decoder network, which can detect the pixel-wise motion state of the object (moving/static) from two consecutive images. The recognition of objects in the framework is not object type specific. We created a new dataset for moving object detection that includes all vehicles, pedestrians, cyclists, and motorcyclists. We reported the superior performance in terms of accuracy of our proposed MOD technique compared to state-of-the-art methods on a publicly available MOD dataset (KITTI-Motion, KITTI-MoSeg, and KITTI-MoSeg Extended). We incorporated the well-known SGM algorithm for disparity estimation. We advanced by combining the outcomes of actual motion estimation and moving object detection network to fully compensate the camera motion. The information related to object class, status, motion, position, velocity and distance are extracted from MOD, compensated optical flow maps and disparity maps. All these pieces of information or object-wise semantic information (OSI) are highlighted as colored labels on the bounding boxes of each object, and also the OSI of each object is stored in a json file. The final evaluation is based on matching the predicted OSIs (from json) against manually annotated ground truth OSIs (from json). The experimental results in several different sequences show that the proposed framework is robust in terms of camera motion and correct object identification. This work aimed to combine the motion and semantic characteristics of objects in the urban driving environment using image processing-based techniques.

[[[["[

  attempt to use image registration to create new ground truths for optical flow and create a new dataset. BIBLIOGRAPHY [Achanta et al. 2012] ACHANTA, Radhakrishna ; SHAJI, Appu ; SMITH, Kevin ; LUCCHI, Aurelien ; FUA, Pascal ; S ÜSSTRUNK, Sabine ; OTHERS: "SLIC superpixels compared to state-of-the-art superpixel methods". In IEEE transactions on pattern analysis and machine intelligence 34 (2012), number 11, pages 2274-2282 [Adams et al. 2010a] ADAMS, Andrew ; BAEK, Jongmin ; DAVIS, Myers A.: "Fast highdimensional filtering using the permutohedral lattice". In Computer Graphics Forum Volume 29 Wiley Online Library (event), 2010, pages 753-762 [Adams et al. 2010b] ADAMS, Andrew ; BAEK, Jongmin ; DAVIS, Myers A.: "Fast high-dimensional filtering using the permutohedral lattice". In Computer Graphics Forum Volume 29 Wiley Online Library (event), 2010, pages 753-762 [Alletto et al. 2016] ALLETTO, Stefano ; PALAZZI, Andrea ; SOLERA, Francesco ; CALDERARA, Simone ; CUCCHIARA, Rita: "Dr (eye) ve: a dataset for attention-based tasks with applications to autonomous and assisted driving". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2016, pages 54-60 [Alvarez et al. 2012] ALVAREZ, Jose M. ; LECUN, Yann ; GEVERS, Theo ; LOPEZ, Antonio M.: "Semantic road segmentation via multi-scale ensembles of learned features". In European Conference on Computer Vision Springer (event), 2012, pages 586-595 [Amirul Islam et al. 2017] AMIRUL ISLAM, Md ; ROCHAN, Mrigank ; BRUCE, Neil D. ; WANG, Yang: "Gated feedback refinement network for dense image labeling". In Proceedings of the IEEE conference on computer vision and pattern recognition, Arnab et al. 2016] ARNAB, Anurag ; JAYASUMANA, Sadeep ; ZHENG, Shuai ; TORR, Philip H.: "Higher order conditional random fields in deep neural networks". In European Conference on Computer Vision Springer (event), 2016, pages 524-540 [Borghi et al. 2017] BORGHI, Guido ; VENTURELLI, Marco ; VEZZANI, Roberto ; CUC-CHIARA, Rita: "Poseidon: Face-from-depth for driver pose estimation". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pages 4661-4670 [Borji 2019] BORJI, Ali: "Saliency Prediction in the Deep Learning Era: Successes and Limitations". In IEEE transactions on pattern analysis and machine intelligence Brostow et al. 2009] BROSTOW, Gabriel J. ; FAUQUEUR, Julien ; CIPOLLA, Roberto: "Semantic object classes in video: A high-definition ground truth database". In Pattern Recognition Letters 30 (2009), number 2, pages 88-97 [Bulo et al. 2018] BULO, Samuel R. ; PORZI, Lorenzo ; KONTSCHIEDER, Peter: "Inplace activated batchnorm for memory-optimized training of dnns". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pages 5639-5647 [Byeon et al. 2015] BYEON, Wonmin ; BREUEL, Thomas M. ; RAUE, Federico ; LIWICKI, Marcus: "Scene labeling with lstm recurrent neural networks". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pages 3547-3555 Bylinskii et al. 2016] BYLINSKII, Zoya ; JUDD, Tilke ; OLIVA, Aude ; TORRALBA, Antonio ; DURAND, Fr édo: "What do different evaluation metrics tell us about saliency models?". In arXiv preprint arXiv:1604.03605 (2016) [Bylinskii et al. 2018] BYLINSKII, Zoya ; JUDD, Tilke ; OLIVA, Aude ; TORRALBA, Antonio ; DURAND, Fr édo: "What do different evaluation metrics tell us about saliency models?". In IEEE transactions on pattern analysis and machine intelligence 41 (2018), number 3, pages 740-757 [Caesar et al. 2016] CAESAR, Holger ; UIJLINGS, Jasper ; FERRARI, Vittorio: "COCO-Stuff: Thing and stuff classes in context". In CoRR, abs/1612.03716 5 (2016), pages 8 Carion et al. 2020] CARION, Nicolas ; MASSA, Francisco ; SYNNAEVE, Gabriel ; USUNIER, Nicolas ; KIRILLOV, Alexander ; ZAGORUYKO, Sergey: "End-to-end object detection with transformers". In European Conference on Computer Vision Springer (event), 2020, pages 213-229 [Hirschmuller and Scharstein 2008] HIRSCHMULLER, Heiko ; SCHARSTEIN, Daniel: "Evaluation of stereo matching costs on images with radiometric differences". In IEEE transactions on pattern analysis and machine intelligence 31 (2008), num-Long short-term memory". In Neural computation 9 (1997), number 8, pages 1735-accuracy with 50x fewer parameters and¡ 0.5 MB model size". In arXiv preprint arXiv:1602.07360 (2016) [Ilg et al. 2017] ILG, Eddy ; MAYER, Nikolaus ; SAIKIA, Tonmoy ; KEUPER, Margret ; DOSOVITSKIY, Alexey ; BROX, Thomas: "Flownet 2.0: Evolution of optical flow estimation with deep networks". In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pages 2462-2470 [international 2016] INTERNATIONAL, SAE: "Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles". In SAE International,(J3016) (2016) [Ioffe and Szegedy 2015] IOFFE, Sergey ; SZEGEDY, Christian: "Batch normalization: Accelerating deep network training by reducing internal covariate shift". In International conference on machine learning PMLR (event), 2015, pages 448-456 [Islam et al. 2017] ISLAM, Md A. ; NAHA, Shujon ; ROCHAN, Mrigank ; BRUCE, Neil ; WANG, Yang: "Label refinement network for coarse-to-fine semantic segmentation". In arXiv preprint arXiv:1703.00551 (2017) [Isola et al. 2017] ISOLA, Phillip ; ZHU, Jun-Yan ; ZHOU, Tinghui ; EFROS, Alexei A.: "Image-to-image translation with conditional adversarial networks". In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pages 1125-1134 [Itti et al. 1998] ITTI, Laurent ; KOCH, Christof ; NIEBUR, Ernst: "A model of saliencybased visual attention for rapid scene analysis". In IEEE Transactions on Pattern Analysis & Machine Intelligence (1998), number 11, pages 1254-1259 [Jaderberg et al. 2014] JADERBERG, Max ; VEDALDI, Andrea ; ZISSERMAN, Andrew: "Speeding up convolutional neural networks with low rank expansions". In arXiv preprint arXiv:1405.3866 (2014) [Jain et al. 1995] JAIN, Ramesh ; KASTURI, Rangachar ; SCHUNCK, Brian G.: Machine vision. Volume 5. McGraw-hill New York, 1995 in video". In European Conference on Computer Vision Springer (event), 2014, pages 656-671 [Jampani et al. 2016] JAMPANI, Varun ; KIEFEL, Martin ; GEHLER, Peter V.: "Learning sparse high dimensional filters: Image filtering, dense crfs and bilateral neural networks". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pages 4452-4461 [Visin et al. 2015] VISIN, F ; KASTNER, K ; CHO, K ; MATTEUCCI, M ; COURVILLE, A ; BENGIO, Y: "A recurrent neural network based alternative to convolutional networks". In arXiv preprint arXiv:1505.00393 (2015) [Visin et al. 2016] VISIN, Francesco ; CICCONE, Marco ; ROMERO, Adriana ; KASTNER, Kyle ; CHO, Kyunghyun ; BENGIO, Yoshua ; MATTEUCCI, Matteo ; COURVILLE, Aaron: "Reseg: A recurrent neural network-based model for semantic segmentation". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2016, pages 41-48 [Vo and Lee 2018] VO, Duc M. ; LEE, Sang-Woong: "Semantic image segmentation using fully convolutional neural networks with multi-scale images and multiscale dilated convolutions". In Multimedia Tools and Applications (2018), pages 1-[Wang et al. 2021a] WANG, Chenjie ; LI, Chengyuan ; LIU, Jun ; LUO, Bin ; SU, Xin ; WANG, Yajun ; GAO, Yan: "U2-ONet: A Two-Level Nested Octave U-Structure Network with a Multi-Scale Attention Mechanism for Moving Object Segmentation". In Remote Sensing 13 (2021), number 1, pages 60 [Wang et al. 2018] WANG, Panqu ; CHEN, Pengfei ; YUAN, Ye ; LIU, Ding ; HUANG, Zehua ; HOU, Xiaodi ; COTTRELL, Garrison: "Understanding convolution for semantic segmentation". In 2018 IEEE winter conference on applications of computer vision (WACV) IEEE (event), 2018, pages 1451-1460 [Wang and Neumann 2018] WANG, Weiyue ; NEUMANN, Ulrich: "Depth-aware CNN for RGB-D Segmentation". In arXiv preprint arXiv:1803.06791 (2018) [Wang et al. 2019] WANG, Wenguan ; ZHAO, Shuyang ; SHEN, Jianbing ; HOI, Steven C. ; BORJI, Ali: "Salient Object Detection With Pyramid Attention and Salient Edges". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pages 1448-1457 [Wang et al. 2016] WANG, Yuhang ; LIU, Jing ; LI, Yong ; YAN, Junjie ; LU, Hanqing: "Objectness-aware semantic segmentation". In Proceedings of the 24th ACM international conference on Multimedia, 2016, pages 307-311 [Wang et al. 2017] WANG, Yunhe ; XU, Chang ; XU, Chao ; TAO, Dacheng: "Beyond filters: Compact feature map for portable deep model". In International Conference on Machine Learning, 2017, pages 3703-3711 [Wang et al. 2021b] WANG, Yuqing ; XU, Zhaoliang ; WANG, Xinlong ; SHEN, Chunhua ; CHENG, Baoshan ; SHEN, Hao ; XIA, Huaxia: "End-to-end video instance segmentation with transformers". In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pages 8741-8750 Document generated with L AT E X and: the L AT E X style for PhD Thesis created by S. Galland -https://github.com/gallandarakhneorg/tex-templates the tex-upmethodology package suite -http://www.arakhne.org/tex-upmethodology/ écol e doct oral e sci ences pour l 'i ngéni eur et mi crotechni ques Universit é Bourgogne Franche-Comt é 32, avenue de l'Observatoire 25000 Besanc ¸on, France Title: Semantic Analysis of the Driving Environment in Urban Scenarios Keywords: Deep Learning, Autonomous Driving, Semantic Segmentation, Visual Attention, Generative Adversarial Network, Moving Object Detection, Motion Estimation, Motion Compensation.

Titre:

  Analyse S émantique de l'Environnement de Conduite dans les Sc énarios Urbains Mots-cl és : Apprentissage profond, Segmentation S émantique, Attention Visuelle, Conduite autonome, R éseaux g én ératifs conditionnels (GAN), D étection d'objets en mouvement, Estimation de mouvement, Compensation de mouvement.

  d'apprentissage profond utilis ées pour l'attention visuelle. Dans ce contexte, nous avons mis en place une large base de donn ées d'attention visuelle bas ée sur une nouvelle strat égie d'extraction de cartes de saillance à partir d'un ensemble de donn ées de conduite existant. Nous avons ensuite propos é un nouveau cadre d'identification d'objets qui combine des indices de mouvement et de g éom étrie pour comprendre l'environnement de conduite urbain. Par ailleurs, un nouveau mod èle de d étection d'objets en mouvement a ét é d évelopp é en int égrant un r éseau codeur-d écodeur coupl é avec la segmentation s émantique et un r éseau d'estimation de disparit é. Un algorithme d'enregistrement d'image est propos é avec le flux optique pour compenser l'ego-mouvement.

Table 2 . 1 :

 21 GoogLeNet Modules

	Model	Corpus	Original Architecture Testing Benchmark Code Available
		Inception module:
		Bottleneck	NIN	ImageNet	YES
		Szegedy et al. (2015)
		Batch Normalization Modified BN-Inception	Inception	ImageNet	YES
		Ioffe and Szegedy (2015)
		Inception V2, V3	BN-Inception	ImageNet	YES
		Szegedy et al. (2016)
		Inception V4 and
		Inception-ResNet-v1, 2	Inception V3
	GoogLeNet	Combining the Inception		ImageNet	YES
		architecture with Residual	ResNet
		connections
		Szegedy et al. (2017)
		Xception Chollet (2017)	Inception V3	ImageNet
		Depthwise Separable		JFT (Google's)
		Convolutions	ResNet	FastEval14k
		Mamalet and Garcia (2012)

YES

increase efficiency and performance. They proposed Inception-ResNet-v1, Inception-ResNet-v2, and an Inception-only variant called Inception V4.

[START_REF] Chollet | [END_REF]

proposed a module called Xception, which means extreme Inception. They replaced the Inception modules with depth-wise separable convolutions proposed in

[START_REF] Mamalet | Simplifying convnets for fast learning[END_REF]

.

Table 2 .

 2

	Unlike LeNet and AlexNet, VGGNet uses multiple 3×3 convolutions in the sequence,
	which can mimic the effect of larger receptive fields, e.g., 5×5 and 7×7. However, it
	requires a large number of parameters and high learning power since it uses large clas-
	sifiers. Figure 2.2 shows a VGGNet with 16 convolutional layers. Residual Network
	(ResNet) is the most popular and widely used neural network for semantic segmentation.

1 shows the GoogLeNet modules.

2.2.1.1/ FEATURE ENCODER BASED METHODS

The dominant approaches to feature extraction method in the literature are Visual Geometry Group (VGG)

Simonyan and Zisserman (2014)

network and Residual Learning

Frameworks (methods that use residual block

[START_REF] He | Deep residual learning for image recognition[END_REF]

as a fundamental building block in their architecture). In this category, we present these methods and their invariants presented in Table

2

.2. The idea behind the concept is to extract feature maps based on stacked convolutional layers, ReLu layers and pooling layers.

The VGG network was introduced by the prestigious Visual Geometry Group at Oxford.

Table 2 .2: Feature Encoder based Methods

 2

	Category	Strategy / Structure	Main Contribution	Architecture Origin Testing Benchmark Code Available
				Convolutional Networks (ConvNets)
	Visual Geometry Group Network	(VGGNet) Simonyan and Zisserman (2014)	Used much smaller 3×3 filters in each convolutional layers which match	AlexNet	ImageNet, PASCAL VOC	YES
				the effect of larger receptive fields,
				e.g. 5×5 and 7×7
						ImageNet,
			Residual Network	Bottelneck Approach		Cityscapes,
			(ResNet)	Shortcut Connections are added	VGG	CIFAR-10,	YES
			He et al. (2016)	(MLPs -Multi Layer Perceptrons)		COCO,
						PASCAL VOC
			ResNet-38 Wu et al. (2019b)	(Shallow Network) ReNet for Image classification FCN for semantic image	ResNet + FCN	Cityscapesss, PASCAL VOC ADE20K,	YES
				segmentation
				Combining the strength of
	R E S I		Fully Convolutional Dense ResNet (FC-DRN) Casanova et al. (2018)	FC-ResNet: gradient flow and feature representation and deep iterative refinement and FC-DenseNet: Multi-Scale	ResNet	CamVid	-
	D			supervision).
	U
	A			High-resolution representations
	L L		High-Resolution Network (HRNet) Sun et al. (2019)	by connecting high-to-low and repeatedly conducting multi-scale resolution convolutions in parallel	ResNet	Cityscapes, LIP PASCAL Context,	YES
	E			fusions across parallel
	A			convolutions.
	R
	N			Combine multi-scale predictions
	I N		Hierarchical Multi-scale Attention Network	together at pixel level. Network learns to predict	HRNet	Cityscapes, Mapillary	YES
	G		Tao et al. (2020)	a relative weighting between
				adjacent scales.
				Label Propagation (LP):
			Video Propagation Li et al. (2020b) and Label Relaxation	Pairing a propagated label Pairing a propagated label with Joint image-label Propagation (JP): with the original future frame.	ResNet	Cityscapes, Camvid KITTI,	YES
				the corresponding propagated image.
	Feature Encoder Concept		Adaptive Network (AdapNet) Valada et al. (2017)	Convoluted Mixture of Deep scheme Experts (CMoDE) fusion	ResNet	Cityscapes, Freiburg forest Synthia,	-
			AdapNet++ Valada et al. (2019)	Self-Supervised Model Adaptation (SSMA): Fuses modality-specific the scene context. class, its spatial location and feature maps based on object	AdapNet	Cityscapes, Synthia, ScanNet Freiburg forest, SUN RGB-D,	YES

Table 2 .

 2

3: Region Proposal based Methods

Category Strategy / Structure Corpus Original Architecture Testing Benchmark Code Available

	and segmentation can be achieved simultaneously. The proposals are generated by al-
	gorithms (Hosang et al. (2015) provide deep analysis) that are semantically meaningful
	and related to objects. They may contain an object class or several other classes that can
	help in determining the semantic labels. Moreover, feeding the wrapped region proposals
	into a convolutional neural network for classification can reduce the computational cost.
	2.2.1.3/ RECURRENT NEURAL NETWORK BASED METHODS
	Recurrent neural networks (RNNs) have actually been introduced for sequence process-
	ing Goodfellow et al. (2016b) Graves et al. (2013) Gao et al. (2018). In addition to their
			Regional proposal generator:
		Regional	Selective Search Method
		Convolutional Neural Network (R-CNN)	CNN: for extracting features from each region	AlexNet VGG-16	PASCAL VOC	YES
		Girshick et al. (2014)	Set of class specific linear
			SVMs to score features.
		Fast R-CNN Girshick (2015)	Improvement in R-CNN pooling layer. Region of Interest (RoI)	VGG-16	PASCAL VOC	YES
		Faster R-CNN Ren et al. (2015)	Region Proposal Network (RPN) Merge of RPN and Fast R-CNN.	VGG-16 FCN as RPN ZFNet	PASCAL VOC COCO	YES
	Regional Proposals	Mask R-CNN He et al. (2017a)	Region of Interest Alignment (RoIAlign): for pixel-to-pixel alignment	VGG-16 FCN as RPN ZFNet	Cityscapes, COCO	YES
			Create feature pyramids
		Feature Pyramid Network (FPN) Lin et al. (2017c)	having semantics at all levels, that can be used to replace	Fast/Faster R-CNN	COCO	YES
			featured image pyramids.
		Path Aggregation Network (PANet) Liu et al. (2018)	Bottom up Path Augmentation Adaptive Feature Pooling: Fully connected Fusion:	Mask R-CNN / FPN	COCO, Mapillary vistas Cityscapes,	-
		(Mask-R-FCN) Zhang and Chi (2020)	Combining the pixel-based FCN and object based Mask-RCNN	FCN / Mask R-CNN	Zurich GID	-

success in handwriting and speech recognition, RNNs have been very successful in computer vision (image processing). We have only studied network models that use RNNs in 2D images (integrating convolutional layers with RNNs). The recurrent neural network consists of long Short-Term Memory (LSTM) Hochreiter and Schmidhuber (1997) blocks.

Table 2

 2

.

4

shows RNN-based methods.

Table 2 .

 2

	Category	Strategy / Structure	Corpus	Original Architecture	Testing Benchmark	Code Available
				Feed-Forward Approach:
		Recurrent Convolution Neural Network ( R CNN) Pinheiro and Collobert (2014)	Models non-local class dependencies in a scene from the raw image (Extract	LeNet	Stanford Background SIFT Flow	-
				contextual information).
				Model the contextual
		Directed Acyclic Graph RNNs	DAG-RNNs Shuai et al. (2016)	dependencies of local features. Class Weighting Function that attends to rare classes. Model long-range semantic	VGGNet + RNN	SiftFlow, CamVid, Barcelona	-
			DAG-RNNs Shuai et al. (2017)	dependencies for graphical Class Weighting Function that structured images.	VGGNet + RNN	Sift Flow, COCO Stuff Pascal Context	-
				attends to rare classes.
				Modified ReNet
	Recurrent Neural Network	ReSeg: Recurrent Segmentation Visin et al. (2016)	Recurrent Layer: Composed by multiple RNNs. Gated Recurrent Unit	ReNet + RNN	CamVid, Weizmann Horse Oxford Flower,	YES
				(GRU) or LSTM
		Multi-level Contextual Recurrent Neural Networks (MCRNNs) Fan et al. (2018)	CRNNs encode three contextual cues (local, global and GIST). Attention model is effectiveness. adopted to improve	VGGNet + RNN	CamVid, Cityscapes KITTI, Stanford-background, SiftFlow,	-
				Formulates graph neural network
		Multi-level Graph Convolutional Recurrent Neural Network (MGCRNN) Jiang et al. (2021)	(GNN) as a RNN to reconstruct pairwise relationships between pixels and aggregate multi-level	VGGNet GNN	Pascal VOC, Cityscapes	-
				contextual information.
				Encoder/Decoder based
		Recurrent model for semantic instance segmentation Salvador et al. (2017)	Recurrent Neural Network LSTM, predicting Encoder: Feature extractor Decoder: Convolutional	ResNet LSTM + Convolutional	Pascal VOC 2012, Leaf Segmentation Cityscapes, CVPPP Plant	YES
				one instance at a time

4: Recurrent Neural Network based Methods

Table 2 .5: Upsampling / Deconvolution based Methods Category Strategy / Structure Corpus Original Architecture Testing Benchmark Code Available

 2

				Object Proposal Network
			Ojectness-Aware	(OPN) generate object
	.		Segmentation (OA-Seg)	proposals Lightweight deconvolutional	VGGNet	PASCAL VOC	-
			Wang et al. (2016)	neural network (Light-DCNN)
				for upsampling
				Built from a down-sampling
		Fully Convolutional DenseNet (FC-DenseNet) J égou et al. (2017)	path, an upsampling path and skip connections. The main goal is to exploit	DenseNet	CamVid Gatech	YES
				the feature reuses
				Convolution Network:
	Unpooling of Low Level Features or Score Maps	Encoder Decoder	ConvDeconvNet Noh et al. (2015) SegNet Badrinarayanan et al. (2017)	Feature extractor Deconvolution Network: Shape Generartor from the feature extractor Obtain higher resolution encoder. by using a set of decoders one corresponding to each	VGGNet VGGNet, DeconvNet	PASCAL VOC Cityscapes, CamVid KITTI, SUN RGB-D,	YES YES
			Squeeze-SegNet Nanfack et al. (2018)	DFire Module: Series of concatenation of expand module of SqueezeNet.	SqueezeNet SegNet	CamVid, Cityscapes	-
				Deep filter consisting
				(convolution, pooling,
		Fully Convolutional Network (FCN) Long et al. (2015) Skip Layer Architecture	activation functions, deconvolution) layers. Upsampling: end-to-end learning by backpropagation from the pixel-wise loss. Skip (Shallow fine layer) that appearance information to improve from a deep, coarse layer with the combines semantic information	Finetuning of AlexNet, GoogLeNet VGGNet,	Cityscapes, CIFAR10, Freiburg Forest SYNTHIA, KITTI, PASCAL VOC, PASCAL Context, ADE20K, CamVid,	YES
				segmentation.
				FCN32s FCN16s FCN8s
			Fully Combine
			Convolutional Network (FCCN)	Fusing and reusing feature maps Layer by Layer	FCN-VGG	CamVid, ADE20K PASCAL VOC,	-
			Wu et al. (2017)
				Motion feature component:
			Semantic Motion	FlowNet2 architectureIlg et al. (2017)
	Upsampling / Deconvolution		Segmentation Network (SMSNet)	Semantic Segmentation component: AdapNet architecture Fusion component: combines both	FlowNet, AdapNet	Cityscapes, KITTI	YES
		F	Vertens et al. (2017)	the motion and
		e		semantic features
		a		Encoder: ResNeXt architecture
		t	Dense Decoder	A decoder is made up of blocks		Pascal VOC,
		u r	Shortcut Connections	which generate semantic features maps.	ResNeXt	Pascal-Context, Pascal Person-Part,	-
		e	Bilinski and Prisacariu (2018)	Multi-level fusion in single-pass		NYUD, CamVid
				inference
		F	Image Cascade
		u s	Network (ICNet)	Proposed a cascade feature fusion (CFF) unit	Modified PSPNet	Cityscapes	YES
		i	Zhao et al. (2018)
		o n	Refine Network (RefineNet) Lin et al. (2017b)	Three Components 1. Residual convolution unit (RCU) 3. Chained residual pooling 2. Multi-resolution fusion	ResNet	Cityscapes, ADE20K, & Context PASCAL VOC NYUDv2, SUN-RGBD,	YES
				GRNet, consisting
				1. Global branch (generates
			Patch Proposal Network (PPN) Wu et al. (2020a)	( the preliminary global-level 2. PPN (patch selection) segmentation feature of downsampling)	Faster RCNN GRNet	Cityscapes	-
				3. Refinement branch (feature
				extraction and refinement)
	Reconstruction and Refinement		RGB-D Multi-level Residual Feature Fusion Network Park et al. (2017) (RDFNET)	Multi-modal feature fusion (MMF): Refining feature the fusion of features (RGB and depth) Multi-level feature refinement:	RefineNet	NYUDv2, SUN RGB-D	YES
		Encoder Decoder	Gated Feedback Refinement Network (G-FRNet) Amirul Islam et al. (2017)	Gate Unit: Combines low-resolution features and high-resolution features to produce contextual information. maps with larger spatial dimensions. Refinement unit: Generate new label	VGGNet	CamVid, Parsing PASCAL VOC, Horse-Cow	YES
			Label Refinement Network (LRN) Islam et al. (2017)	Predicts semantic labels at several different resolutions in a coarse-to-fine fashion.

Table 2 .6: Increase Resolution of Features based Methods Category Strategy / Structure Corpus Original Architecture Testing Benchmark Code Available .

 2

		.	DeepLab Chen et al. (2014)	Atrous ('Holes') Convolution	FCN-VGG	Cityscapes, PASCAL VOC	YES
				Atrous Spatial Pyramid
		Atrous Convolution	DeepLabV2 Chen et al. (2017a)	Pooling (ASPP). the field of view of Method effectively enlarge	FCN-ResNet	Cityscapes, COCO PASCAL VOC,	YES
				filters to incorporate
				multi-scale context.
			DeepLabV3 Chen et al. (2017b)	Rethink Atrous Convolution Augment the Atrous Spatial Pyramid Pooling (ASPP).	DeepLabV2	Cityscapes, PASCAL VOC	-
			DeepLabV3+ Chen et al. (2018)	Xception Encoder Decoder Approach	DeepLabV3	PASCAL VOC	YES
			Dilated	Rectangular Prism
		.	Convolutions Module	convolutional layers, with no pooling or	VGGNet	Cityscapes, PASCAL VOC	YES
			Yu and Koltun (2015)	subsampling for multi-scale
				context aggregation .
	Increase of Features Resolution	Dilated Convolution	SQ Network Treml et al. (2016)	SharpMask approach Refinement module: Fire module: modified SqueezeNet convolution layer. Parallel dilated	SqueezeNet	Cityscapes	-
			Hybrid Dilated Convolution (HDC) Wang et al. (2018)	Dense Upsampling Convolution (DUC) by TuSimple.	ResNet + DUC	KITTI, PASCAL VOC	YES
			Series-parallel Structure Self-attention Gaihua et al. (2021) Network	Self-Attention Module: dilated convolution Based on the serial-parallel structure combined with	ResNet	Cityscapes, PASCAL VOC	-
	.		Dilated Residual Network (DRN) Yu et al. (2017)	into ResNet model. Replacing dilated convolutions layers	ResNet	Cityscapes	YES
				Method to simulate a
		Fully Convolutional Residual Network (FCRN) Wu et al. (2016b)	high resolution network (FoV) of features. with a low resolution network. Enlarge the field-of-view	ResNet + FCN DeepLab	Cityscapes, PASCAL VOC	-
				Online bootstrapping
				method for training.
		Efficient semantic segmentation with pyramidal Oršić and Šegvić (2021) fusion (SwiftNet)	Multi-scale architecture for boundary pixels. Increasing the penalty with pyramidal fusion. Spatial Pyramid Pooling (SPP).	ResNet MobileNet V2	Cityscapes, Mapillary Vistas ADE20k, CamVid,	YES

Table 2 .7: Enhancement of Features based Methods Category Strategy / Structure Corpus Original Architecture Testing Benchmark Code Available

 2

		Multi-Scale Network	Multi-scale Convolutional Network extract dense feature vectors that encode regions of multiple sizes Multiple post-processing methods centered on each pixel.	LeNet	Sift Flow, Background Barcelona, Stanford	-
		Farabet et al. (2013)	for labeling.
			Learn multi-scale features using the image depth information.	LeNet	NYUDv2	-
			Multi-scale Patch Generator:
			Cropping corresponding feature
			grids from Image, and aligning
		Multi-scale Patch Aggregation (MPA)	these grids to improve the generalization ability.	VGG-16	PASCAL VOC, COCO	-
	Multi-scale	Liu et al. (2016)	A strategy is proposed to assign
	Features		the classification and segmentation
	Extraction		labels to the patches.
		DeepLab Attention Model Chen et al. (2016c)	Learns to weight the multi-scale for each scale outputs a weight map features according to the object scales presented in the image, then	DeepLab	PASCAL VOC, COCO	-
			which weights feature pixel by pixel.
		Pyramid Scene Parsing Network (PSPNet) Zhao et al. (2017)	Pyramid pooling module consists of the large kernel pooling layers for global scene prior construction	ResNet Dilated FCN	ImageNet, PASCAL VOC Cityscapes, ADE20K,	YES
			Cascading dilated convolutions
	Enhancement of	Cascade Dilated Convolutions Network	(consecutive layers connection) to extract dense features. Feature fusion through Maxout Layer	Dialted-ResNet FCN-VGG	PASCAL VOC	-
	Features	Vo and Lee (2018)	(Maxout Network
			Goodfellow et al. (2013))
			Reformulating global aggregation
		Context Aggregation Network Yang et al. (2021)	normalization and selection for and local distribution (GALD) blocks. Fusion block (FFM) to assists in feature	MobileNetV3 Dialted-ResNet	Cityscapes, UAVid	-
			optimal scene segmentation.
		Multiply Spatial Fusion Network	Multi-features Fusion Module (MFM): Obtain spatial information and	ResNet	Cityscapes, Camvid	-
		(MSFNet) Si et al. (2019)	enlarge receptive field.
			CCL: Consists of several chained
		Context Contrasted	context-local blocks to make multi-		Pascal Context,
		Local (CCL)	level context contrasted local features.	ResNet	SUN-RGBD,	-
		Model	Gate Sum: Fusion strategy to		COCO Stuff
		Ding et al. (2018)	aggregate appropriate score maps.
		Cascaded Feature Network (CFN) Lin et al. (2017a)	features. Context-aware Receptive Field (CaRF): to aggregate convolutional features of local context into strong	FCN + RefineNet	NYUDv2, SUN-RGBD	-
			Self-attention based encoder:
	Feature Extraction from sequence	SEgmentation TRansformer (SETR) Zheng et al. (2021)	Fully attentive feature representation encoder by sequentializing images. 1. Naive upsampling Three different decoder designs;	FCN	Cityscapes, Pascal Context ADE20K,	YES
	of nested regions		2. Progressive UPsampling (PUP)
			3. Multi-Level feature Aggregation (MLA).
			Zoom out features construction
			using superpixels (SLIC Method)
			from different levels of spatial context
		Zoom Out Mostajabi et al. (2015)	Local Level: Superpixel itself Distant Level: Regions large enough to cover fractions of an object or
			entire object.
			Scene Level: Entire scene
			Combining features across levels rather
			than predicting.

Table 2 .8: Semi and Weakly Supervised based Methods

 2

	Category	Strategy / Structure	Corpus	Original Architecture	Testing Benchmark	Code Available

-

Souly et al. (

Table 2 .

 2 9: Spatio-Temporal based Methods

	Category	Strategy / Structure	Corpus	Original Architecture	Testing Benchmark	Code Available
	.	Clockwork FCN Shelhamer et al. (2016)	Clockworks: clock signals that control the learning of different layers with different rates	FCN Clockwork RN	Youtube-Objects, Cityscapes NYUD,	YES
			APANet: predicting multi-level
			pyramid features that selectively
		Auto-Path Aggregation (APANet) Hu et al. (2021a)	and adaptively aggregate the task-specific hierarchical spatio-temporal contextual information obtained	Mask R-CNN FPN	Camvid NYUDv2	YES
			on the features of each
			individual level.
		Spatio-Temporal FCN Fayyaz et al. (2016)	Spatial-Temporal Module image frames embedding into FCN LSTM to define relationships between	FCN	Camvid NYUDv2	YES
	Spatio-	Spatio-Temporal Data-Driven Pooling (STD2P) He et al. (2017b)	Incorporate superpixels and multi-view information into convolutional networks	FCN	NYUDv2 SUN 3D	-
	Temporal	Feature Space Optimization Kundu et al. (2016) (FSO)	Optimize the mapping of pixels regularization to a Euclidean feature space used by DenseCRF for spatio-temporal	VGG Dilation	CityScapes, Camvid	YES
		Deep Spatio-Temporal FCN (DST-FCN) Qiu et al. (2018)	Learn spatial-temporal dependencies through 2D FCN on pixels and 3D FCN on voxels	VGG C3D	A2D, CamVid	-
			Implementation of three gated
			recurrent architectures
		Gated Recurrent	RFC-LeNet: Conventional Recurrent
		FCN Siam et al. (2017)	Units.
			RFC-VGG and RFC-Dilated:
			Convolutional Recurrent Units.

Table 2 .

 2 10: Transformer based Methods

	Category	Strategy / Structure	Corpus	Original Architecture	Testing Benchmark	Code Available
		Patch-Encoding	SETR Zheng et al. (2021)	Progressive upsampling Multilevel feature Aggregation (MLA)	ViT	Cityscapes Pascal Context ADE20K,	Yes
			TransUNet Chen et al. (2021a)	Hybrid CNN-Transformer	ViT U-Net	Synapse multi-organ CT	Yes
			SegFormer Xie et al. (2021)	Positional-encoding-free hierarchical Lightweight All-MLP decoder Transformer.	ViT	Cityscapes COCO Stuff ADE20K,	Yes
	Transformer	Mask Encoding	Segmenter Strudel et al. (2021)	Point-wise linear mapping. Mask transformer	ViT DETR	Cityscapes Pascal Context ADE20K,	Yes
						ADE20K,
			MaskFormer Cheng et al. (2021a)	Mask Classification	ResNet DETR	COCO Stuff, Mapillary Vistas,	Yes
						Cityscapes
			ISTR Hu et al. (2021b)	Low-dimensional Mask embedding. Recurrent Refinement strategy.	R-101-FPN	COCO	Yes
		Object Encoding	Panoptic DETR Carion et al. (2020)	Predictions via Bipartite matching. Non-autoregressive parallel decoding.	ViT FPN	COCO panoptic Yes
			VisTR Wang et al. (2021b)	Similarity Learning	ResNet-50 DETR	YouTube-VIS	Yes
	ding, object embedding, and mask embedding.

Table 2

 2 Arnab et al. (2016) proposed a method in which CRF models unary and pairwise potentials together with higher-order object detector potentials (to provide semantic cues for segmentation) and superpixels (with label con-

				.11: Methods using CRF/MRF
	Category		Strategy / Structure	Corpus	Original Architecture	Testing Benchmark	Code Available
					Based on mean field
			Fully Connected-CRF	approximation, message
				(DenseCRF)	passing performed using	ResNet	PASCAL VOC	Yes
			Kr ähenb ühl and Koltun (2011)	Gaussian filtering
					techniques.
					Multiple Mean-field
					Iterations.
				CRF-RNN Zheng et al. (2015)	Interpretation of dense CRFs as Recurrent Neural	FCN	PASCAL VOC Cityscapes	-
					Networks (CRF-RNN)
					combined with CNN.
		Gaussian Conditional Random Field (GCRF)	Gaussian Mean Field (GMF) Network Vemulapalli et al. (2016)	GMF Network: Performing Gaussian mean field inference.	DeepLab	PASCAL VOC ImageNet	Yes
			Quadratic Optimization (QO) Chandra and Kokkinos (2016)	(QO) module Quadratic Optimization	FCN	PASCAL VOC	-
			Convolutional-CRF
				(ConvCRF)	Inference in terms of	ResNet	PASCAL VOC	Yes
			Teichmann and Cipolla (2018)	convolutions.
					Object-detection based
	CRFs / MRFs	Incorporating Higher potentials Order		Higher-order CRF Arnab et al. (2016)	over regions. potentials: Provide Semantic Encourage label consistency cues for segmentation. Superpixel-based potentials:	CRF-RNN	PASCAL VOC, Context	-
				Structured Patch	Integrate segmentation		PASCAL VOC
				Prediction	specified features,	FCN	Cityscapes	-
			(SegModel) Shen et al. (2017)	high order context and		ADE20K
					boundary guidance.
			Deep Parsing Network (DPN) Liu et al. (2015b)	Models Unary term and Pairwise terms	VGG	PASCAL VOC	-
					in single CNN.
			Deep Parsing Network (DPN) Liu et al. (2015b)	Models Unary term and Pairwise terms	VGG	PASCAL VOC	-
					in single CNN.
		.		Learning Messages Lin et al. (2015)	CNN message estimators for the message passing	VGG-16	PASCAL VOC	-
					inference.
		Adelaide	Bounding -box Detection	Adelaide Very Deep FCN Wu et al. (2016a)	Hough transform based method for training. approach Online bootstrapping	FCRN	PASCAL VOC	-
					Patch-patch context:
					Formulate CRFs to capture		PASCAL VOC
				Context CNN CRF Lin et al. (2016)	contextual relationship between neighboring patches	VGG-16	NYUDv2 Pascal Context	-
					Patch-background context:		Siftflow
					Sliding Pyramid Pooling.
				Depth-sensitive
		incorporate		fully-connected	Fully-connected CRFs with
		the depth	Conditional Random Field	RGB information and	FCN	SUN-RGBD	-
		information		(DFCN-DCRF)	depth information.
				Jiang et al. (2017)
				.
	message estimators for message passing inference for structured Conditional Random
	Field (CRFs) predictions. Teichmann and Cipolla (2018) developed a convolutional CRFs

method (ConvCRFs) that reformulates message passing inference in terms of convolutions. Some methods used higher-order potentials (based on object detection or superpixels) modeled as CNN layers when they used mean-field inference and effectively improved semantic segmentation performance. posed a depth-sensitive fully connected conditional random field combined with a fully convolutional network (DFCN-DCRF). The basic idea is to integrate depth information in Dilated-FCN and Fully Connected CRF to improve the accuracy of semantic segmentation.

CRF inference with deep convolutional neural networks improves pixel-level label prediction by producing sharp boundaries and dense segmentation. Several methods learn arbitrary potentials in CRFs. It has been used as post-processing, end-to-end mode, formulated as RNN and integrated as a module into existing neural networks.

Table 2 .

 2 12: Alternative to CRF based Methods The network model predicts semantic segmentation potentials and affinities at the pixel level and combines them through the proposed random walk layer that applies spatial smooth-

	Category	Strategy / Structure	Corpus	Original Architecture	Testing Benchmark	Code Available
		Bilateral Neural Network (BNN) Jampani et al. (2016)	Bilateral filter inference in DenseCRF pairwise potentials . Replacing Gaussian potentials with bilateral convolution to learn	DeepLab	Pascal VOC	Yes
		Fast Bilateral Solver (BS) Barron and Poole (2016)	Edge-aware smoothness algorithm using bilateral filtering technique.	CRF-RNN	Pascal VOC MS COCO	-
		Boundary Neural Field Bertasius et al. (2016) (BNF)	Build unary and pairwise potentials combine them in global manner. from input RGB image, then	FCN	Semantic Dataset Boundaries	-
	Alternative to CRF Approaches	DT-EdgeNet Chen et al. (2016a)	Domain transform (DT) Module: from midway layers. Edge-preserving filter. Edge Net: Predicts edge features	DeepLab	Pascal VOC	-
			Large kernels used for
		Global Convolutional Network (GCN) Peng et al. (2017)	classification and localization. Boundary Refinement Block: Model the boundary alignment as a	FCN ResNet	Cityscapes PASCAL VOC COCO	-
			residual structure.
			Boundary refinement module
			adopts the learned direction field
		Boundary Refinement	to guide the object edge		Cityscapes
		with Point Supervision BRPS	points rectification. Uncertainty estimation, key points	UNet	PASCAL VOC, NYUDv2,	-
		Dong et al. (2021)	detection and offset		BDD100K
			relaxation based on point
			supervised learning.
			Boundary Enhancement
			Attention Module (BEAM)
		Semantic Boundary Enhancement and Position Chen et al. (2021b) network (SBEPNet)	and Position Attention Module (PAM). semantic boundaries to capture inter dependencies along Learn the long-range spatial	ResNet	Cityscapes, PASCAL VOC CamVid,	-
			discriminative contextual
			information.
		Random Walk Network (RWN) Bertasius et al. (2017)	Random Walk Network Pixel labeling framework	DeepLab-largeFOV	Pascal, Sift Flow SBD-Stanford Background,	-
	Bertasius et al. (2016) proposed an FCN architecture called Boundary Neural Field
	(BNF) for predicting semantic boundaries and building semantic segmentation maps us-
	ing global optimization. The BNF combines the unary potentials (prediction by FCN)
	and the pairwise potentials (boundary-based pixel affinities) from the input RGB image

in a global way. The basic idea is to assign pixels to foreground and background labels for each of the different object classes and apply constraint relaxation. Later in Bertasius et al. (2017) they proposed Convolutional Random Walk Network (RWN) which addresses the same problem, a model based on the random walk method Lov ász et al. (1993).

  AdapNet[START_REF] Valada | Adapnet: Adaptive semantic segmentation in adverse environmental conditions[END_REF]:• Achieves top score of 88.25% IoU on Freiburg Forest. The network reached Mean IoU of 69.39% on cityscapes and 72.91% on Synthia dataset.The improvement is due to the highly representative multiscale features learned by the model, which allow segmentation of very distant objects present in Synthia and Cityscapes. AdapNet's modeling approach is based on a mixture of convolutional neural network (CNN) experts (Convoluted Mixture of Deep Experts -CMoDE) and considers multiple modalities such as appearance, depth and motion.

	AdapNet++ Valada et al. (2019):

• Achieves top score of 92.1% IoU on Synthia and 57.7% IoU on ScanNetv2 dataset. The network achieves the score of 83.94% IoU on Cityscapes, 45.75% IoU on SUN RGB-D, and 84.18% IoU on Freiburg Forest dataset. Self-Supervised Model Adaptation which includes a new encoder with multiscale residual

•

  Achieves score of 38.9% IoU on COCO, 43.6% IoU on ADE20K, 58.6% IoU on Pascal Context and 45.01% IoU on Mapillary dataset. DSSPN constructs a semantic neuron graph in which each neuron segments regions of a parent concept in a semantic concept hierarchy (by combining labels from four datasets) and aims to recognize between its child concepts. Instead of using a completely large semantic neural graph, DSSPN only activates a relatively small neural graph for each image during training, making DSSPN memory and computationally efficient. Within SwiftNet, two approaches to increasing the size of the receptive field are considered. First, Spatial Pyramid Pooling (generates feature maps with varying levels of detail by enriching features from the encoder output with their pools over coarse spatial grids The segmentation network uses a pre-trained CNN with DAG -RNN that fuses low-level features with DAG -RNN. A new class-weighted loss function is proposed to control the class-wise loss during training. The performance of the segmentation network increases with increase of DAGs with DAG -RNN. A fully connected CRF is used to further improve the performance of the network.

	1×1, 2×2, 4×4, and 8×8.) Second, Pyramidal Fusion (true multiscale representations,
	train with the boundary-aware loss to avoid overfitting). This shows significant improve-
	ments on all tested datasets. RefineNet Lin et al. (2017b):
	Residual framework ResNet-38 Wu et al. (2019b):
	• Achieves a score of 45.90% IoU on SUN-RGB, 46.50% IoU on NYUDv2 and 47.30%
	RFCNet Siam et al. (2017): • Achieves the highest score of 48.1% IoU on Pascal Context, 80.6% IoU on IoU on Pascal Context datasets. The results on Pascal VOC, cityscapes, and
	• Achieves scores of 81.20% IoU on SYNTHIA, 80.12% IoU on SegTrack and com-cityscapes and 43.43% IoU on ADE20K. ADE20K datasets are 83.40% IoU, 73.60% IoU, and 40.70 % IoU respectively.
	petitive score of 69.84% IoU on DAVIS dataset.
	Clockwork-FCN Shelhamer et al. (2016): FCN-8s Long et al. (2015):
	• Achieves 68.50% IoU on Youtube Object, 68.40% IoU on Cityscapes, 28.90% IoU • Achieves the score of 77.46% IoU on Freiburg Forest, 67.20% IoU on PASCAL
	on NYUDv2 dataset. VOC, 65.30% IoU on CIFAR-10, 65.30% IoU on Cityscapes, 56.10% IoU on KITTI,
	29.39% IoU on ADE20K, 35.10% IoU on PASCAL CONTEXT, 65.24% IoU on SYN-
	Clockwork-FCN uses different clocking schemes; fixed-rate clock reduces computational THIA, and 57.00% IoU on CamVid dataset.
	overhead by assigning different clock rates to each stage, so that later stages execute
	less often. Adaptive clockwork updates when the output score maps are expected to Performance is enhanced by transferring pre-trained classifier weights, fusing different
	change, reducing computation while maintaining accuracy. layer representations, and learning on whole images throughout.

The model uses different FCN architectures such as a recurrent node to use temporal information, a deconvolution layer for upsampling, and a support skip architecture for finer segmentation. The use of temporal data is the reason for the performance improvement and not the simple addition of extra convolutional filters.

Adelaide

Context CNN-CRF Lin et al. (2016): • Achieves score of 40.6% IoU on NYUDv2, 42.30% IoU on SUN-RGB, 78.00% IoU on Pascal VOC, 66.40% IoU on CIFAR-100, 71.60% IoU on Cityscapes, and 43.30% IoU on Pascal Context dataset. The model uses CNN-based pairwise potential functions to capture semantic correlations between neighboring patches that improve coarse-level prediction. The model uses FCN with sliding pyramid pooling, CNN contextual pairwise, boundary refinement (dense CRF method) and trained the model with additional images from the COCO dataset to improve the overall performance of the model. SwiftNet Oršić and Šegvić (2021): • Achieves top score of 69.77% IoU on KITTI, and reach 76.4% mIoU on Cityscapes. Further, 55.0% IoU on Camvid and 44.8% IoU on Mapillary dataset. The model introduces residual units into ResNet (17 residual units for 101 layers of ResNet) and extends it into a sufficiently large number of subnets. Each connection in the ResNet unit shares same kernel size and number of channels, which improves model accuracy. ResNet-38 does not apply multi-scale testing, model averaging, or CRF-based post-processing, except for the ADE20K test set. ESPNet: Mehta et al. (2018): • Efficient real-time segmentation network, achieves 60.2% IoU on cityscape, 40.0% IoU on Mapillary dataset with 0.364M parameters, 63.01% IoU on Pascal VOC test set with 0.364M parameters. Efficient Spatial Pyramid (ESP) network is an efficient neural network in terms of speed and memory. ESP , based on factorized form of convolutions (pointwise convolution and spatial pyramid of dilated convolutions), reduces the number of parameters, memory, with large receptive field. DAG-RNN Shuai et al. (2017): • Achieves 44.8% IoU on Sift-flow, 31.2% IoU on COCO (171 classes) and 43.7% IoU on PASCAL Context dataset. RefineNet applies data augmentation during training (random scaling, cropping, and horizontal flipping of the image) and multiscale evaluation (averaging predictions for the same image over different scales for the final prediction). The Dense CRF method is only used for Pascal VOC. Dilation10 Yu and Koltun (2015): • Achieves 67.60% IoU on PASCAL VOC, 67.10% IoU on Cityscapes, 32.31% IoU on ADE20K and 65.29% IoU on CamVid dataset.

  Zhang et al. Zhang et al.

	(2018a) proposed an efficient large-scale point cloud segmentation method by fus-
	ing 2D images with 3D point clouds to CNN to segment complex 3D urban scenes.
	The authors in Yousefhussien et al. (2018); Charles et al. (2017) proposed methods
	for direct semantic labeling of 3D point clouds with spectral information. However,
	3D segmentation methods face many challenges compared to 2D segmentation,
	i.e., high complexity, computational cost, slow processing, and most importantly, a
	lack of 3D datasets. In semantic video segmentation, two approaches can be use-
	ful, one to improve the computational cost (by reducing the latency); The authors in
	Shelhamer et al. (2016); Li et al. (2018) proposed designed scheduling frameworks
	that reduce the overall cost and maximum latency of semantic video segmenta-

tion. However, these approaches are far from meeting the latency requirements in real-time applications. The second approach is to improve accuracy (by exploiting temporal continuity -temporal features and temporal correlations between video frames). Several methods

[START_REF] Fayyaz | STFCN: spatio-temporal FCN for semantic video segmentation[END_REF]

;

He et al. (2017b)

;

[START_REF] Qiu | Learning deep spatio-temporal dependence for semantic video segmentation[END_REF]

  Our review focuses on saliency detection models, both classical and deep-learning based, used for visual attention in general and in the driving environment in particular.

	3.2.1/ VISUAL ATTENTION USING CLASSICAL APPROACH
	The term visual attention was used early in "Feature Integration Theory" by Treisman and
	Gelade (1980) to define human visual search strategies. According to this theory, salient
	areas in the visual scene are identified by the combination or relationship of visual feature
	information such as color, orientation, spatial frequency, brightness, direction of motion
	that direct human attention. The concept of saliency map was first proposed by Koch
	proposed by Zhang and Sclaroff (2013) is based on a set of random thresholded Boolean
	maps. A new form of VOCUS saliency method called VOCUS 2 is proposed by Frintrop
	et al. (2015). The idea is to measure the center-surround contrast at different scales
	(Gaussian difference), and the model provides pixel-precise saliency maps. A similar
	center-surround difference logic is used in Montabone and Soto (2010), which proposed
	a "fine-grained" saliency algorithm called Visual Saliency Feature (VSF). The algorithm

Modeling visual attention is an active research topic in image processing and computer vision, and is closely related to topics such as object saliency detection and gaze fixa-tion. and Ullman (1987) to achieve attentional selection according to Treisman theory Treisman and Gelade (1980). The visual attention methods that use saliency are divided into two categories: bottom-up (biologically inspired methods; image color and intensity are common examples) and top-down (true computational methods; prior knowledge, memories, goals are common factors). Itti and Koch proposed a visual attention mechanism Itti et al. (1998) inspired by Treisman and Gelade (1980) and Koch and Ullman (1987). Their saliency detection model extracts multi-scale image features by covering different size ratios between the center and surrounding regions and combining them into a single saliency map. This classical model is considered one of the successful and widely used methods for selective attention in the human visual system. Based on its success, Harel et al. (2007) proposed a model called Graph-based visual saliency (GBVS), which applies the graph algorithms to achieve efficient saliency computations. Hou and Zhang (2007) makes use of the spectral residuals approach. The model, called Spectral Residual Model ( SR ), is based on the logarithmic spectral representation of images.

[START_REF] Frintrop | VOCUS: A Visual Attention System for Object Detection and Goal-directed Search[END_REF]

introduced a new attention system called Visual Object detection with Computational Attention System -VOCUS that detects regions that are more likely to contain relevant information in the image (region of interest).

[START_REF] Hou | Image signature: Highlighting sparse salient regions[END_REF]

proposed an algorithm called SignatureSal, which is a comprehensive image descriptor that detects salient regions in the image. An efficient saliency detection algorithm called BMS provides fine-grained feature maps and much better defined boundaries.

[START_REF] Zhang | Salient object detection via proposal selection[END_REF]

presents a saliency detection framework that uses object proposals in an unsupervised manner. Few attention methods are based on Multiple Object Tracking Theory (MOT)

Table 3 . 1 :

 31 Notation Overview

	Term Meaning	Distribution Meaning
	x	Real Image	p data	Real and target training data distribution
	y	Target Image	p z	Noise distribution (e.g. N(0; 1)), z ∼ p z
	g	Generated Target Image	p y	Known target distribution, y ∼ p y
	z	Noise	p x	Real data distribution, x ∼ p x
	µ	Average	p g	Generated Target data distribution

Table 3 . 2 :

 32 Summary of Datasets

	Dataset	Samples Training Validation
	Berkeley Deep Drive Yu et al. (2018)	7000	1000
	CamVid Fauqueur et al. (2007)	367	101
	Cityscapes Cordts et al. (2016)	2975	500
	VADD	10342	1601
	lights, and traffic signs. Similar risk sense responses are obtained by tracking the se-
	quence of the driver's eye fixations on road objects while viewing the driving scene. Our
	object class selection for saliency heatmap data generation is also based on these road
	objects, i.e. persons (pedestrians, cyclists), vehicles (cars, motorcycles, trucks, trams),
	and others (traffic lights/traffic signs). The attending driving-specific salient features are
	to be used as input for decision making and/or planning or monitoring. We incorporate
	three driving datasets BDD, Cityscape, and CamVid (Table 3.2) that provide semantic
	labels (annotation of each object in images).
	3.4.2/ SALIENCY ALGORITHM SELECTION

Numerous saliency works have models on various metrics, noise robustness, and sideby-side comparison of computed saliency maps (visualization) Bylinskii et al. (

2018

)

Kim and Milanfar (2013)

compared. We study the robustness to noise of saliency algorithms, Itti, GBVS, SR, ML -Net, BMS, iSEEL, and VSF (presented in subsections 3.2.1 and 3.2.2). Our goal is to choose the better saliency detection algorithm for constructing ground truth for our desired application of visual attention. The white Gaussian noise is added to 500 test images with a mean of zero and three different variance values σ 2 (0.04, 0.12 and 0.19) as shown in Figure

3

.8. We fed clean and noisy images into the saliency detection algorithms and used the matrices Peak Signal to Noise Ratio (PSNR) and Mean Squared Error (MSE)

(3.5.3)

for evaluation. The (VSF)

[START_REF] Montabone | Human detection using a mobile platform and novel features derived from a visual saliency mechanism[END_REF]

algorithm shows more stable results (with low MSE and high PSNR) as shown in

Table 3 .

 3 3, and provides the complete shape of the highlighted objects.

Table 3 .3: Noise robustness based saliency algorithm evaluation

 3

	Saliency Methods	σ 2 =0.04	MSE (Low is good) σ 2 =0.12 σ 2 =0.19	PSNR (High is good) σ 2 =0.04 σ 2 =0.12	σ 2 =0.19
	Itti/Koch	1127.56	1109.98	1109.98	18.0269	18.0985	18.1531
	BMS	1228.03	1399.94	1399.94	18.1355	17.3990	17.1982
	ML-Net	693.624	755.848	755.848	19.9791	19.6315	18.8982
	SR	1366.642 2306.73	2306.73	19.6558	16.9105	14.7811
	GBVS	1153.76	1338.19	1338.19	18.2219	17.5340	17.2169
	iSEEL	1011.23	1209.01	1209.01	19.7487	18.6684	17.9921
	VSF	55.5966	180.129	180.129	31.4493	25.7475	23.3944

Table 3 . 4 :

 34 Quantitative Performance

				Validation
			BDD	CamVid	Cityscapes	VADD
	Database	Noo f Images	1000	101	500	1601

Table 3 .

 3

	Diverse Conditions	SSIM 0 -1	WD 0 -∞	FID 0 -∞
	Weather	Rain Fog Snow	0.7711 0.7601 0.8084	1.98 2.08 1.70	40.79 59.80 31.14
	Environment	Tunnel Night Bridge Highway Urban	0.8051 0.5725 0.8225 0.7588 0.8567	1.90 9.91 1.62 2.58 0.97	39.58 101.12 30.47 82.22 22.61

5: Quantitative Performance in Different Environment Conditions

Table 3 .

 3

	Network Models	KL-Div AUC ↓ ↑	NSS ↑	CC ↑
	ML-Net Cornia et al. (2016)	2.737 0.595 1.690 0.620
	SAM-VGG Cornia et al. (2018)	1.915 0.691 1.703 0.685
	SAM-ResNet Cornia et al. (2018) 1.757 0.718 1.720 0.697
	Proposed	1.450 0.754 1.969 0.736

6: Quantitative Performance Vs Saliency Network Models on VADD validation set

Table 3 .

 3 7: Quantitative Performance Vs Saliency Network Models on SALICON test Dataset

	Network Models	AUC ↑	NSS ↑	CC ↑
	Itti Itti et al. (1998)	0.667	-	0.205
	GBVS Harel et al. (2007)	0.789	-	0.421
	BMS Zhang and Sclaroff (2013)	0.789	-	0.427
	ML-Net Cornia et al. (2016)	0.866	2.789	0.743
	SAM-VGG Cornia et al. (2018)	0.881	3.143	0.825
	SAM-ResNet Cornia et al. (2018)	0.883	3.204	0.842
	Proposed	0.889	3.231	0.839
	AGAINST EYE FIXATION ATTENTION MODELS

Table 3 .

 3

	Network Models	SSIM 1 -0	WD 0 -∞	FID 0 -∞
	BDDA Xia et al. (2018)	0.7081	5.54	29.68
	Proposed	0.8022	2.65	18.77

8: Quantitative Performance Vs BDDA -Driving Attention Network Model on VADD validation set

Table 3 .

 3

	Network Models	KL-Div ↓	AUC ↑	NSS ↑	CC ↑
	SALICON Huang et al. (2015)	1.41	0.915	3.14	0.53
	Dr(eye)Ve Palazzi et al. (2018)	1.95	0.866	2.90	0.50
	BDDA Xia et al. (2018)	1.24	0.931	3.51	0.59
	Proposed	1.15	0.947	3.68	0.60
	VADD Vs BDDA & Dr(eye)Ve Datasets:

9: Quantitative Performance Vs Eye Fixation Network Models on BDD-A testing Dataset

Table 4 .

 4

	1: SOTA Instance segmentation networks detection performance on COCO
	dataset test-dev2017
	Method	Backbone Mask AP Box AP fps GPU
	Mask R-CNN Wu et al. (2019a)	R-50-FPN	35.2	38.6	23.2 V100
	Mask R-CNN Wu et al. (2019a)	R-101-FPN	38.6	42.9	17.8 V100
	CenterMask Lee and Park (2020)	R-101-FPN	39.8	44.0	15.2 V100
	CenterMask-Lite Lee and Park (2020) V-39-FPN	36.3	40.7	35.7	Xp
	vehicles, pedestrians, bicyclists, and motorcycles. The second task is binary pixel classi-
	fication, which uses temporal information to predict whether the detected object is moving
	or static.

Table 4 . 2 : Comparison with existing available Moving Object datasets

 42 -batch of m two consecutive frames [{ f x , f x-1 }, ..... ,{ f y , f y-1 } ] (x, y) ∈ [1, N f ] ; calculate the corresponding segmentation masks [{M x , M x-1 }, ..... ,{M y , M y-1 } ] using the SEG() : {M x , M x-1 }=SEG({ f x , f x-1 }); convert M x andMx-1 to binary BW x and BW x-1 ; concatenate BW x and BW x-1 on the channel dimension BW x x-1 ; feed BW x x-1 to the ED-Net to calculate the moving objects mask M OM

	Datasets	No of Images Object Classes Course	Image
				Resolution
	KITTI-Motion Vertens et al. (2017)	455	Vehicles Only	384 × 1048
	Cityscapes Motion Vertens et al. (2017)	3475	Vehicles Only	384 × 768
	KITTI-MoSeg Siam et al. (2018b)	1300	Vehicles Only	384 × 1280
	KITTI-MoSeg Extended Rashed et al. (2019)	12919	Vehicles Only	1024 × 2048
			All type Vehicles
	Ours	10059	Pedestrians Cyclists
			Motor bike

Table 4 .

 4 3: Quantitative evaluation of our MOD model on the validation set of the proposed MOD dataset. Comparison of different design variants for segmentation (Mask-RCNN and CenterMask) and Encoder-decoder network (with 3, 6 and 9 residual blocks). The evaluation is in the form of intersection over union, precision, Recall, F-score and frame per second, using the respective image resolutions.

	Approach	Segmentation Method used	Backbone	N Image Validation	mIoU	Moving IoU	Static IoU	Precision Recall Fscore	fps
		Mask R-CNN	R-50-FPN		82.48	66.07	98.91	72.54	77.43	73.07	9.309
	ResNet 3-Block	Mask R-CNN CenterMask	R-101-FPN R-101-FPN	1509	83.16 83.60	67.37 68.24	98.95 98.97	73.70 74.79	77.32 77.42	75.52 76.79	8.199 8.152
		CenterMask-Lite V-39-FPN		82.98	66.09	98.91	73.28	77.39	73.99	9.990
		Mask R-CNN	R-50-FPN		84.36	69.99	98.74	76.90	78.01	76.39	9.265
	ResNet 6-Block	Mask R-CNN CenterMask	R-101-FPN R-101-FPN	1509	84.46 85.58	71.99 73.18	98.65 98.99	77.41 77.95	78.6 78.71	76.48 76.68	8.174 8.092
		CenterMask-Lite V-39-FPN		84.41	71.06	98.75	76.03	78.03	76.4	10.006
		Mask R-CNN	R-50-FPN		83.15	67.47	97.94	74.42	77.20	75.37	9.201
	ResNet 9-Block	Mask R-CNN CenterMask	R-101-FPN R-101-FPN	1509	84.04 84.84	70.9 71.83	98.77 97.51	75.13 75.49	78.01 78.27	75.91 75.15	8.098 8.003
		CenterMask-Lite V-39-FPN		83.33	69.91	98.95	73.78	77.18	75.59	9.998
	SEGMENTATION NETWORK AND EDNET ADOPTION
	We performed an ablation study with the different numbers of ResNet residual blocks,
	i.e., 3, 6, and 9 blocks, together with four segmentation model choices (Mask-RCNN
	R50/101, CenterMask-R101/V39) to observe the trade-off between accuracy and speed.

Table 4 .

 4

	3 show the metric scores

Table 4 .

 4 6. The authors in Rashed114CHAPTER 4. SEMANTIC-AWARE OBJECT IDENTIFICATION IN URBAN DRIVING SCENARIOS et al. (2019) and Ramzy et al. (2019) proposed architectures that support different inputs from different sensors. We compared our results with all their input configurations. The evaluation is performed with input image resolution 384 × 1280. We have significantly outperformed all Fuse-MODNet and RST-MODNet sensors fusion methodologies and U 2 -ONet in terms of mIoU and Moving IoU.Figure 4.12 shows the qualitative assessment between the proposed model, Fuse-MODNet Rashed et al. (2019), RST-MODNet Ramzy et al. (2019), and U 2 -ONet Wang et al. (2021a). The extended KittiMoSeg dataset provides more than 12k binary mask labels for different sequence runs from the Kitti dataset.However, there are approximately 7k labels that do not have moving objects. Many labels are ambiguous, i.e., objects are labeled in a square area, incorrect labeling of moving objects, etc. For this reason, we manually selected only those image labels where the moving objects are labeled accurately (4800 for training and 1927 for testing).

Table 4 .

 4

5: Quantitative comparison on KITTI-MoSeg

Siam et al. (2018b)

dataset Approach Moving IoU Precision Recall Fscore fps GPU

	MODNet Siam et al. (2018b)	45.41	56.18	70.32	62.46	8	Titan Xp
	U 2 -ONet Wang et al. (2021a)	55.47	68.08	72.36	64.23	-	Tesla V100
	Ours	58.82	70.83	76.87	70.23	10	RTX-2080Ti

Table 4 .

 4 6: Quantitative comparison on KITTI-MoSeg Extended Rashed et al. (2019) dataset

	Approach	mIoU	Moving IoU	fps	GPU
	Fuse-MODNet Rashed et al. (2019) (RGB)	65.6	32.7	40	Titan Xp
	Fuse-MODNet Rashed et al. (2019) (RGB+rgbFlow)	74.24	49.36	25	Titan Xp
	Fuse-MODNet Rashed et al. (2019) (RGB+lidarFlow)	70.27	41.64	25	Titan Xp
	Fuse-MODNet Rashed et al. (2019) (RGB+rgbFlow+lidarFlow)	75.3	51.46	18	Titan Xp
	RST-MODNet Ramzy et al. (2019) (LSTM-Multistage)	76.3	53.3	23	Titan Xp
	U 2 -ONet Wang et al. (2021a)	-	62.5	-	Tesla V100
	Ours	80.15	64.11	10 RTX-2080Ti

  Pseudocode for extraction of Motion Direction, Position, Distance, and Velocity information. Vy avg < 1 then label Vdir = Motionless VC() ← V x avg & Vy avg Calculate vector length (V Len HV ) V Length HV = V x avg × Vy avg if V Len HV ≥ 90 then label velocity = Very Fast elif 9 ≤ V Len HV < 90 then label velocity = Fast elif 0.9 ≤ V Len HV < 9 then label velocity = Medium elif 0.09 ≤ V Len HV < 0.9 then label velocity = Slow elif V Len HV < 0.09 then label velocity = Stationary PC() ← f rame width if w < f rame width /3 then label pos =

	Inputs: M d prty Disparity Mask M OM P Moving Object Mask BC OI Bounding P box, class information of objects of	if Vy avg ≥ 1 then Approaching elif Vy avg ≤ -1 then label Vdir = label Vdir =
	interest from MRC V OF uv	Fully	Moving Away
	Compensated Optical flow vectors V x and V y Outputs: f OI P Frame pred -Objects Identified json f ile json file -Information of each object in each frame Functions: DC(): returns average intensity from M d prty ; DRC(): returns direction from V OF uv ; VC(): returns velocity ; PC(): returns position ; DC() ← M d prty Step 1: Find Contours on the M d prty Step 2: Calculate the distances to the contour Step 3: Calculate average intensiites (DI avg ) Step 4: Calculate distance in meters S dis	elif -1 < Front Left
	using equation 4.2			elif w < f rame width × 0.66 then
	if S dis ≥ 0.18 then	label dis = Very	label pos = Front
	close			else label pos = Front Right
	elif 0.12 ≤ S dis ≤ 0.179 then label dis = Close	In the predicted f OI P frame output, show the object ID/class on top of binding box
	elif 0.05 ≤ S dis ≤ 0.119 then label dis =	← BC OI P , direction arrow ← (label Hdir &
	Far			label Vdir ) in the center, while distance ←
	elif S dis ≤ 0.049 then label dis = Very	(label dis ), position ← (label pos ) and velocity
	far			← (label velocity ) on top-right of the binding
	DRC() ← V OF uv Calculate averages (V x avg & Vy avg ) if V x avg ≥ 1 then label Hdir = Left to	box of each object. Label moving objects ← M OM P the frame f OI P .	in green on
	Right			Generate json file json f ile , containing each
	elif V x avg ≤ -1 then label Hdir = Right	detected object information for
	to Left			identification.
	elif -1 < V x avg < 1 then label Hdir =
	Motionless

VELOCITY

Velocity can be calculated from vector values representing the displacement of a pixel between two frames. The displacement values or intensity values from each object are collected by multiplying the mean values of the x axis and y axis from each object. These intensity values represent the speed of movement and are labeled as very fast, fast, medium, slow, very slow, and stationary.

Algorithm 2:

DISTANCE

We compute the disparity and depth values (intensities) using the (SGM) algorithm and calibrate it to roughly visualize the distance of each segmented object according to its average intensity value. We define four labels for depth: very far, far, close, and very 118CHAPTER 4. SEMANTIC-AWARE OBJECT IDENTIFICATION IN URBAN DRIVING SCENARIOS close.

Table 4 .

 4 Pred LINE = [Ob ject ID , Ob ject Class , Ob ject S tatus , Ob ject Position , Ob ject Direction , Ob ject Distance , Ob ject Velocity ] GT LINE = [Ob ject ID , Ob ject Class , Ob ject S tatus , Ob ject Position , Ob ject Direction , Ob ject Distance , Pred LINE [4] = "moving away" & GT LINE [4]= "moving away" then Ob ject Direction = Away Correct if Pred LINE [4] = "approaching" or "motionless" or "L2R" or "R2L" & GT LINE [4]= "moving Ob ject Velocity = S low False Ob ject Velocity = Fast Correct if Pred LINE [6] == "slow" or "medium" or "very fast" & GT LINE [6]== "fast" then Ob ject Velocity = Fast False == "very fast" & GT LINE [6]== "very fast" then Ob ject Velocity = V Fast Correct if Pred LINE [6] == "slow" or "medium" or "fast" & GT LINE [6]== "very fast" then Ob ject Velocity = V Fast False Ob ject Distance = VClose False Ob ject Distance = Close False Ob ject Distance = V Far Correct 124CHAPTER 4. SEMANTIC-AWARE OBJECT IDENTIFICATION IN URBAN DRIVING SCENARIOS if Pred LINE [5] = "very close" or "close" or "far" & GT LINE [5]== "very far" then Ob ject Distance = V Far False We computed the overall accuracy of the moving/static, motion, velocity, and distance of 8: Accuracies for Movement, Distance, Velocity and Position Ob ject Class , Ob ject Moving , Ob ject S tatic , Ob ject Position , Ob ject distance , Ob ject motion , and Ob ject velocity must be predicted CORRECTLY. If any of the predictions

	4.4.1/ EVALUATION -PART I

We calculate the accuracy of each attribute individually i.e., object class, status (moving/static), object motion

(approaching, away, left-to-right, right-to-left)

, position (front,front-left,-right), and distance (very close, close, far, very far) using precision metric. if Pred LINE

[1]

== GT LINE

[1]

then

Ob ject Class = Correct else Ob ject Class = False Figure 4.16: Example OSI Tree (Ground Truth and Predicted)

if if Pred LINE [4] = "R2L" & GT LINE [4]= "R2L" then Ob ject Direction = R2L Correct if Pred LINE

[4]

= "moving away" or "motionless" or "approaching" or "L2R" & GT LINE

[4]

= if Pred LINE [6] == "slow" & GT LINE [6]== "slow" then Ob ject Velocity = S low Correct if Pred LINE [6] == "medium" or "fast" or "very fast" & GT LINE [6]== "slow" then if Pred LINE [6] == "fast" & GT LINE [6]== "fast" then if Pred LINE [5] = "very close" & GT LINE [5]== "very close" then Ob ject Distance = VClose Correct if Pred LINE [5] = "close" or "far" or "very far" & GT LINE [5]== "very close" then if Pred LINE [5] = "close" & GT LINE [5]== "close" then Ob ject Distance = Close Correct if Pred LINE [5] = "very close" or "far" or "very far" & GT LINE [5]== "close" then if Pred LINE [5] = "very far" & GT LINE [5]== "very far" then are wrong, the system should consider it a FALSE or incorrect identification. We also calculate the computation time of the FOI. Table 4.9 illustrates the overall object identification accuracy of the FOI, and Table 4.10 shows the performance (processing time) of each task within the FOI and the overall speed.

Table 4 .

 4 9: Over-All Accuracy of FOI The moving object detection model takes about 0.1045 seconds, and the disparity map calculation step takes about 0.0438 seconds per frame. About 0.0094 seconds per image is required for fusion and information extraction. The total inference time of the proposed FOI is about 0.1247 seconds per frame, which is equivalent to 8.02 fps.

	No of Frames	No of Objects Moving Static	FOI Over-All Accuracy (%)
	309	723	1809		81.27
	COMPUTATIONAL TIME
	All experiments were performed on a standard desktop (Intel core i9, two RTX 2080Ti
	GPUs, using 375×1242 input images) with the Python processing environment. The aver-
	age computation time for motion estimation/compensation is about 0.1153 seconds (im-
	age registration takes 0.0652 seconds and optical flow takes 0.0501 seconds) for each
	frame. Table 4.10: Performance (processing time) within FOI and overall speed using
	375 × 1242 input images
		Task			Inference Time [s/f]
	Motion Estimation / Compensation	0.1153
	MOD				0.1045
	Disparity				0.0438
	Information Fusion			0.0094
		FOI			0.1247

68.5% MIoUClockwork-FCN[START_REF] Shelhamer | Clockwork convnets for video semantic segmentation[END_REF]

ACKNOWLEDGMENTS

Table 2.14: Links to the Source Codes Network Model Code Link Network Model Code Link InceptionSzegedy et al. (2015) RSIS Salvador et al. (2017) https://github.com/imatge-upc/rsis

OBJECT STATUS

The accuracy of each moving and static object is given by Accuracy if Pred LINE [2] == "moving" & GT LINE [2]== "moving" then Ob ject S tatus = Moving Correct if Pred LINE [2] == "static" & GT LINE [2]== "moving" then Ob ject S tatus = Moving False if Pred LINE [2] == "moving" & GT LINE [2]== "static" then Ob ject S tatus = S tatic False if Pred LINE [2] == "static" & GT LINE [2]== "static" then Ob ject S tatus = S tatic Correct

OBJECT MOTION

The motion accuracy includes position, direction,and velocity are given by Position :

Accuracy