Keywords: visual localization, place recognition, image retrieval, feature combination, deep learning

Chapter 1 Introduction

The overall aim of this thesis is to improve place recognition for mobile robots or vehicles visual localization in changing environments. This thesis presents methods allowing to increase the robustness of visual localization system by improving the place recognition performance using appearance rather than metric information. Our approach allows robot or vehicle to globally re-localize itself by recognizing places it has previously visited under variations in appearance and illumination. This chapter introduces the main topic of this thesis that is place recognition based visual localization, and the effect of environmental change on place recognition reliability. It begins by outlining the background of the research topic (Section 1.1). Section 1.2 introduces the advantages of place recognition based visual localization using camera. Section 1.3 illustrates the problems caused by changing environments as well as the research objectives and contributions of this thesis. Section 1.4 describes the experimental platform used to acquire datasets for evaluating the proposed approaches. The thesis organization is outlined in Section 1.5.

Background

During the past decades, autonomous mobile robots and intelligent vehicles (IV) have obtained increasingly attention and developments from research society and industry community [START_REF] Bishop | Intelligent Vehicle Technology and Trends[END_REF][START_REF] Chen | Multi-sensor information fusion algorithm with central level architecture for intelligent vehicle environmental perception system[END_REF]. In order to promote the development of autonomous vehicles, American Department of Defense has hold autonomous vehicle competition called DARPA (Defense Advanced Research Projects Agency) Grand Challenge. This challenge attracted many toplevel research institutes (See Fig. 1-1). In 2009, Google started self-driving car project, it has self-driven for more than 1.5 million miles and is currently out on the streets of Mountain View and so on. In 2014, google released its new version of self-driving car. Fig. 12shows the new prototype of googles driverless cars. In recent years, some companies like Baidu, BMW and Uber also launched their self-driving cars.

(a) MIT Land Rover LR3 1(b) The Stanford Racing Team2 Google prototype driverless car. 3Some experimental projects in indoor environments have shown that robots can run autonomously [START_REF] Milford | Persistent navigation and mapping using a biologically inspired slam system[END_REF]. However, long-term autonomous navigation in changing outdoor environments is still an ongoing challenge.

Autonomous cars or robots comprise fundamental systems, such as surrounding perception, navigation, driving control and localization, which make sure vehicle to be driven safely in complex environment. Among these tasks, localization is a prerequisite for accomplishing other tasks, as it determines the vehicle position in the environment [START_REF] Lidberg | Automated driving and autonomous functions on road vehicles[END_REF].

Based on localization information, we can compute the position of an obstacle and perform path planning. Therefore, the correctness and accuracy of the localization system impact the functionality of an autonomous vehicle.

Visual localization can be achieved in particular through place recognition, which is the process of identifying a previously-seen location in an environment. This thesis focuses on place recognition based visual localization task. Robust visual localization system should benefit from improved performance of place recognition.

Place Recognition Based Visual Localization

In general, Global Positioning System (GPS) is commonly used for outdoor localization.

GPS seems offer a simple and low-cost solution, but it requires line-of-sight satellites.

GPS-denied environments, accuracy decreasing or intermittent available conditions occur frequently in areas where there are tall buildings and trees or half-outdoor space [START_REF] Wei | Camera/laser/gps fusion method for vehicle positioning under extended nis-based sensor validation[END_REF].

At the same time, visual sensors recently have become the primary components of many state-of-the-art place recognition and Simultaneous Localisation And Mapping (SLAM)

systems [START_REF] Eustice | Visually navigating the rms titanic with slam information filters[END_REF][START_REF] Milford | Long exposure localization in darkness using consumer cameras[END_REF][START_REF] Neubert | Appearance change prediction for longterm navigation across seasons[END_REF][START_REF] Ranganathan | Towards illumination invariance for visual localization[END_REF][START_REF] Schindler | City-scale location recognition[END_REF].

There are several advantages of using camera sensors for visual localization: (1) Firstly, digital cameras are light-weight and cheap, benefit from small form factor, have modest power requirements and their size expands their applicability to smaller hardware and mass-production; (2) Furthermore, the rich image data received from cameras offers rich appearance and texture information, as well as high potential for semantic interpretation;

(3) Finally, a camera can provide information about far away landmarks. Camera such as bumblebee2 can see large landmarks (such as mountains) hundreds of meters away. In contrast, LiDAR such as the Velodyne HDL-64E 4 has maximum range of 120m and costs higher than camera.

Vision localization using camera is often based on place (image) recognition to locate position within the world, which typically adopts image or sequence matching (retrieval) method [START_REF] Cummins | FAB-MAP: Probabilistic localization and mapping in the space of appearance[END_REF][START_REF] Santos | A visual place recognition procedure with a markov chain based filter[END_REF][START_REF] Johns | Dynamic scene models for incremental, long-term, appearance-based localisation[END_REF] to recognize a previously-seen place in an environment. Here, the place may either be considered as a precise position-"a place describes part of the environment as a zero-dimensional point" , or as a larger area-"a place may also be defined as the abstraction of a region" where a region represents a two-dimensional subset of the environment [START_REF] Kuipers | The spatial semantic hierarchy[END_REF]. According to this, each place can be represented by an image or sequence, thus visual localization can be achieved through place recognition based on image or sequence matching (retrieval) technique [START_REF] Garcia | Vision-based topological mapping and localization methods: A survey[END_REF].

In place recognition based localization system, as Fig. 1-3 shows, representative images are captured from the environment and stored in a database, with their corresponding location (GPS information). During on-line localization, each observation (image) is compared to the images of database. The location whose corresponding image (from the database) best matches to the current observation is then considered to be the currently visible location (process of place recognition). Then, visual localization is realized through the GPS information from the matched image. Recently, visual localization has been largely facilitated thanks to the progress of image features. Effectively, feature extraction enables efficient describing of environments. Thus, matching current visual input with a set of images of known places can be conveniently conducted based on extracted features.

Problem Statements and Objectives

Visual localization based on place recognition is a well-defined but extremely challenging problem to solve especially in complex outdoor environments. Most of existing place recognition based localization systems can perform successfully in static environments but fail in highly dynamic environments. In particular, visual localization systems are sus- ceptible in large-scale changing environments where drastic appearance and illumination changes, caused by weather conditions or seasonal changing occur. This thesis addresses the challenge of improving place recognition techniques so that global perceptual changes in the environment do not cause complete failure of the robot or vehicle localization system.

Given two observations (typically images), judging whether these observations are collected from the same location will meet several troubles. Firstly, the biggest challenge is how to describe a place without being affected by environment changing. Two images from the same place may look different due to variances in appearance or illumination. In order to perform robust localization task using vision, it is necessary to describe the acquired images (or sequences) and to be able to compare these descriptions. Consequently, the recognition performance and the localization results will directly rely on the method used for visually describing the different environment locations. In addition, among local or global features, how to combine advantageous of these features for place describing is also a challenge. Secondly, recognizing or matching methods still can be improved. The appropriate recognizing or matching strategy can improve the recognition accuracy. There are two main place recognizing methods-based on image matching and -based on sequence matching. In general, sequence matching is more robust. The last challenge is computation time. The robots or intelligent vehicles need to localize themselves in high speed driving situation, therefore the judging method for visual localization system should performed fast enough to satisfy the real-time requirement.

Our research is part of the project CPER "Intelligence du Véhicule Terrestre" (Intelligence of ground vehicle), conducted within IRTES-SET in UTBM. The goal of this thesis is to improve place recognition performance for visual localization in changing environment. The proposed approaches are tested on extensive outdoor environment datasets (some datasets are acquired by our own vehicle platform and others are public datasets)

and the final outcome of this thesis is an all-environment visual localization system that is capable of running in real-time.

The three main objectives and contributions are explained as follows:

(1) The first objective is to explore feature combination for vehicle localization. Since different types of feature have their own advantages, combining some powerful features will be helpful for place recognition. A new multi-feature (D-CSLBP++HOG) is proposed for visual localization. D-CSLBP++HOG feature combine HOG (Histogram of Oriented Gradients) and CSLBP (Center-symmetric local binary patterns) features that are built from both gray-scale image and disparity map. The integration of disparity information, permits to improve the performance of place recognition, especially in complex environment situation. In addition, for real-time visual localization, local sensitive hashing method (LSH) is used to compress the high dimension of the multi-feature into binary vector. It can thus speed up the process of image retrieval. To show its effectiveness, the proposed method is tested and evaluated using real datasets acquired in outdoor environments. As we will show, our approach allows more effective visual localization compared with the state-of-the-art FAB-MAP (Fast Appearance Based Mapping) method.

(2) When single image matching is used for visual localization, it is easy to perceive two different places as the same due to seasonal changing. This is known as "perceptual aliasing". In order to decrease the perceptual aliasing influence, retrieval based on sequence of images rather than on single image can be used to improve place recognition results. Therefore, another task is to develop a visual localization system based on sequence matching to improve the recognition performance and localization results. An approach of visual localization across seasons is proposed using sequence matching based on the combination of GIST and CSLBP features. Studies of the relationship between image sequence length and sequence matching performance is conducted. To show its effectiveness, the proposed method is tested and evaluated in four seasons outdoor environments. The results have shown the improved precision-recall performance against the state-of-the-art SeqSLAM (Sequence Simultaneous Localization and Mapping) algorithm.

(3) Most of the used features are hand-crafted features and they have demonstrated good performance in place recognition and visual localization. However, for database with specific surroundings (i.e. trees, buildings or mountains), it is difficult to decide what kind of features should be taken to describe places. Suitable features can achieve good place recognition results while unreliable features could lead to false recognizing.

With the rapidly development of deep learning networks, it is becoming apparent in place recognition tasks that hand-crafted features are being outperformed by learnt features. Our contribution is to use the automatic learned Convolutional Network (ConvNet) features to accomplish all-environment visual localization task under appearance and illumination changing situations. A comprehensive performance comparison of different ConvNet layers is conducted on four real world datasets. To speed up the computational efficiency, locality sensitive hashing method is taken to achieve real-time performance with minimal accuracy degradation.

Experimental Platform

As illustrated in Fig. 1234, our experimental GEM vehicle is equipped with many sensors (stereo vision system, camera, RTK-GPS, etc). A Bumblebee XB3 stereo vision system is installed on the top, and oriented to the front. It is composed of three collinear cameras with a maximum baseline of 240 mm. Images are captured in a format of 1280×960 pixels.

In our application, only the left and right cameras are used. A RTK-GPS receiver (10 Hz) is used to collect position (GPS) information. All the installed sensors are fixed rigidly.

Thesis Organization

The rest of the thesis is divided into five chapters:

In chapter 2, existing approaches for place recognition based visual localization are reviewed. According to the features used in place describing, the visual localization methods are classified as: approaches based on global descriptors, approaches based on local features, approaches based on multi-feature combination, approaches based on 3D information, approaches using deep learning features. According to the place recognizing method, two main approaches are introduced: image matching/retrieval approach and sequence matching/retrieval approach.

In chapter 3, a visual localization method based on multi-feature combination and disparity information using stereo camera is proposed. Disparity information is integrated into complete center-symmetric local binary pattern (CSLBP) to obtain robust global im-age description (D-CSLBP). In order to describe the scene more accurately, multi-feature combining D-CSLBP and HOG features is adopted to provide valuable information and to decrease the effect of some typical problems in place recognition such as perceptual aliasing. It improves visual recognition performance by taking the advantage of depth, texture and shape information. In addition, for real-time visual localization, local sensitive hashing method (LSH) is used to compress the high dimensional multi-feature into low dimensional binary vector. The proposed method is tested and evaluated using real datasets acquired in outdoor environments.

In chapter 4, visual localization across seasons using sequence matching is presented.

Matching places by considering sequences instead of single images denotes higher robustness to extreme perceptual changes. The recognition results of different sequence lengths is compared. The results obtained from Nordland dataset shows that the proposed method can achieve better performance than SeqSLAM method.

In chapter 5, all-environment visual localization system based on ConvNet features and localized sequence matching is proposed. The pre-trained network provided by MatCon-vNet is used to extract features and then a localized sequence matching technique is applied for visual recognition. Compared with the traditional approaches based on hand-craft features and single image matching, the proposed method shows better performances even in presence of appearance and illumination changes. A comprehensive performance comparison of different ConvNet layers (each defining a level of features) is conducted considering both appearance and illumination changes.

In chapter 6, a summary of the achieved outcomes and discussions of their relevance to the current research in place recognition based visual localization are provided. In addition, some research perspectives for future work are also indicated.

Overview

With the low cost of cameras and the richness of provided sensor data , place recognition for visual localization is attracting more and more attention [START_REF] Zetao Chen | Multiscale bio-inspired place recognition[END_REF][START_REF] Collier | Multi-sensor appearance-based place recognition[END_REF][START_REF] Newman | Outdoor slam using visual appearance and laser ranging[END_REF][START_REF] Ulrich | Appearance-based place recognition for topological localization[END_REF]. In this context, each place (location) can be represented by an image or sequence of images, and a robot or vehicle localizes itself by identifying a previously-visited location through image or sequence retrieval. Thus, place recognition based approaches have to be robust even in situations in which the robot's metric position estimation is largely erroneous.

Solving the all-environment visual localization problem for identifying where a robot is over time has become one of the main challenging research areas [START_REF] Johns | Feature co-occurrence maps: Appearance-based localisation throughout the day[END_REF]. Unfortunately, this is not an easy task, because place appearance strongly changes at different times of day, along months and especially along seasons [START_REF] Arroyo | Fast and effective visual place recognition using binary codes and disparity information[END_REF][START_REF] Sünderhauf | Are we there yet? Challenging SeqSLAM on a 3000 km journey across all four seasons[END_REF]. tem. This localization system has the following essential components:

(1) Visual information inputs: Images or videos are the data source for the whole system. It also includes data preprocessing, which transforms observations into a suitable form for description or storage, e.g. collection of feature descriptors or whole images.

(2) Place describing: Places must be described in a way that enables them to be efficiently stored and recognized when they are revisited. Visual place description techniques mainly fall into two broad categories: those that selectively extract parts of the image that are in some way interesting or notable; and those that describe the whole scene, without a selection phase.

(3) Place remembering: A place recognition system needs to refer to a map, where the extracted descriptors are stored and organized for comparison and retrieval. The state-ofthe-art approaches can be divided into three main categories: those using place databases, those using topological maps, and those using metric maps. The appropriate type of place remembering depends on the purpose of the place recognition system.

• A place database is the simplest mean to represent a particular environment where only appearance information is stored. In this approach, place recognition is based solely on appearance similarity and applies image retrieval techniques. Although valuable information is lost due to not including of relative pose information, there are computationally efficient indexing techniques that can be exploited.

• Topological maps contain relative information about places in an environment. It can simply consists of an ordered collection of images: a linear database that reflects the order in which places are consistently encountered when an environment is visited.

In these cases, localization simply identifies the most likely location.

• Metric maps depict the absolute scale of environments more accurately, maintaining a lot of information about environment details, such as distances, driving direction or landmark position, and they are usually referenced according to a global coordinate system. This representation is most appropriate for vehicle localization and guidance. However, metric maps are more difficult to build and maintain, and are computationally demanding.

(4) Place recognizing: It refers to the mean of comparing current observation to the stored ones. Ultimately, the purpose of place recognition is to determine whether a place has been seen before. There is a general understanding that if two place descriptions appear similar there is a greater likelihood that they have been captured at the same physical location. Thus, the central goal of any place recognition system is reconciling visual input with the stored data to generate a belief distribution. There are two main approaches for place recognizing: based on single image or based on sequence of images. Sequence-based approach is more robust by removing some false positive recognition.

(5) Visual localization: Throughout the place recognition process, robot localizes itself based on a previously-visited location, which are tagged with GPS information. The vehicle or robot can then achieve its localization by assimilating its position to that of the retrieved image through the recognized place. Using place recognition based visual localization, accumulation error that often occur in odometry-like approach can be avoided. It also should be noted that, the system simply identifies the most likely places and then get a rough position. This is a topological level localization rather than a very accurate metric localization.

Among the module of a place recognition based system, place describing and remembering are the two cores. In order to perform visual localization using vision, it is necessary to describe the acquired images and to be able to compare their descriptions. Consequently, the quality of place remembering and the posterior localization will directly rely on the method used for visually describing the different environment locations.

Due to the above reasons, we classify the different visual localization approaches according to the description method employed as: approaches based on local features, approaches based on global features, approaches based on multi-feature combination, approaches using three-dimensional information and approaches using deep learning features.

On the other hand, from the perspective of recognizing method, visual localization methods can also be divided into two mainly categories: methods based on single image matching; and methods based on sequence matching.

In this chapter, we review the main approaches with regard to place recognition based visual localization and SLAM. The rest of this chapter is organized as follows: Section 2.

Approaches Based on Different Place Describing Features

Each place is a unique location and it must be described in a way that enables it to be efficiently stored and recognized when revisited. Many issues relevant to place recognitionbased approaches have been proposed. The different methods can be classified according to the way of place description employed as: methods that describe places by local features; methods that describe places by global features; methods that describe places by local and global features combination; methods that describe places using three-dimensional information and methods that describe places using deep learning features.

Methods based on local features

Local feature methods make use of distinct features (or feature keypoints) within images, which capture the essence of the image [START_REF] Brahnam | Introduction to local binary patterns: New variants and applications[END_REF][START_REF] Yang | Local difference binary for ultrafast and distinctive feature description[END_REF]. During extraction step, a set of distinct features (or feature keypoints, i.e. corners or edges) is first detected by analyzing images and searching for distinctive pixel patterns (see Fig. 2-2). Then, a description step is performed, where some measurements (i.e. comparison or concatenation) are taken from the vicinity of each local descriptor to form the final feature [START_REF] Garcia | Vision-based topological mapping and localization methods: A survey[END_REF]. In general, features are formed as a multi-dimensional floating-point vectors or bit strings.

A good local feature typically needs to be invariant to one or more affine transformationssuch as image scale and camera rotations. Thus, the same local features can be identified in the similar images, which enables recognition of familiar places. Many works have used local features in the field of place recognition and visual detection tasks [START_REF] Gil | Improving data association in vision-based SLAM[END_REF], especially since the development of Scale-Invariant Feature Transform (SIFT) algorithm [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]. Recently, a number of local binary features have been proposed in the literature, providing an interesting research line to explore for place description and recognition [START_REF] Sun | Application of image retrieval based on the improved local binary pattern[END_REF].

Their advantages are that they are invariant to monotonic changes in gray-scale and fast to calculate. One typical binary feature LBP (Local Binary Pattern) [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] is used in paper [START_REF] Qiao | Place recognition based visual localization using LBP feature and SVM[END_REF],

where SVM (support vector machine) recognition models are built based on the extracted LBP features and each place can be recognized using these SVM models.

Other local features based on keypoints detection like BRIEF (Binary Robust Independent Elementary Features) [START_REF] Calonder | BRIEF: Binary robust independent elementary features[END_REF], ORB(oriented BRIEF) [START_REF] Rublee | ORB: An efficient alternative to sift or surf[END_REF], BRISK (Binary Robust Invariant Scalable Keypoints) [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF], Local Difference Binary (LDB) [START_REF] Arroyo | Fast and effective visual place recognition using binary codes and disparity information[END_REF], FREAK (Fast Retina Keypoints) [START_REF] Alahi | FREAK: Fast retina keypoint[END_REF], BRIEF (Binary Robust Independent Elementary Features) [START_REF] Calonder | BRIEF: Binary robust independent elementary features[END_REF] and KAZE [3] are also used in place recognition.

Local features present high discrimination capacity, resulting into higher recognition rates and less detection errors. However, the total local features dimension of each image could be very high, and directly matching image features can be inefficient. The bagof-words model [START_REF] Sivic | Video Google: A text retrieval approach to object matching in videos[END_REF] increases efficiency by quantizing local features into a vocabulary, where every feature is assigned to a particular word and each image can be described by low dimensional vector or binary string. As the bag-of-words model ignores the geometric structure of the place that is described, the resulting place description is pose invariant; that is, the place can be recognized regardless of the position of the robot with respect to the place.

The biggest disadvantage of local image features is that they perform poorly-or fail entirely-in the presence of extreme condition variance [START_REF] Glover | Repeatable condition-invariant visual odometry for sequence-based place recognition[END_REF]. In such cases, either feature detection and matching process fail because the object of interest being described by local descriptors is less distinctive in different conditions.

Methods based on global features

In the previous section, solutions based on local features were reviewed, where the description pays more attention to parts of image (sub-regions). Such descriptions work well for partial occlusions or camera rotations, but are not able to deal with general structure or framework of the whole scene.

Global features describe the image in a holistic manner, using the whole-image (or global descriptors) rather than (sub-regions) for place recognition. Global features are normally very fast to extract and are more robust to the environment changing conditions where particular features are unrecognizable [START_REF] Garcia | Vision-based topological mapping and localization methods: A survey[END_REF]. Some of the main global features used in place recognition and scene classification approaches are shown in ples of some global features used in early visual localization systems include: color histograms [START_REF] Ulrich | Appearance-based place recognition for topological localization[END_REF], features based on principal component analysis [START_REF] Gaspar | Vision-based navigation and environmental representations with an omnidirectional camera[END_REF] and Gradient Orientation Histograms [START_REF] Kosecka | Qualitative image based localization in indoors environments[END_REF]. In paper [START_REF] Ulrich | Appearance-based place recognition for topological localization[END_REF], the authors describe place using six one-dimensional color histograms from HLS and RGB color spaces. The reference images are retrieved using a nearest neighbor learning scheme in their topological map and they obtains at least 87% of correctly classified images in their whole appearance-based place recognition system.

Kosecka et al. [START_REF] Kosecka | Qualitative image based localization in indoors environments[END_REF] propose a vision-based navigation strategy using gradient orientation histograms as image feature. The similar places are determined through comparison of these gradient orientation histograms.

Winters et al. [START_REF] Winters | Omni-directional vision for robot navigation[END_REF] utilize an omni-directional camera to create a topological map.

The large image set is compressed using PCA to form a low dimensional eigenspace, then robot could determine its global topological position using an appearance based matching method.

Besides that, one of the popular global features-GIST [START_REF] Oliva | Modeling the shape of the scene: A holistic representation of the spatial envelope[END_REF], which was initially developed for scene recognition, has already been used for place recognition on a number of research works [START_REF] Sünderhauf | BRIEF-Gist-Closing the loop by simple means[END_REF]. GIST uses Gabor filters at different orientations and different frequencies to extract information from the image. The results are averaged to generate a compact vector that represents the "GIST" descriptor of the scene.

Global features can be also constructed by detecting and concatenating local features.

For example, Lamon et al. [START_REF] Lamon | Deriving and matching image fingerprint sequences for mobile robot localization[END_REF] propose fingerprints, composed of a variety of image features -such as color patches, edges and corners. By ordering these features in a sequence between 0 • and 360 • , place recognition could be reduced to string-matching. These systems used omni-directional cameras which allow rotation-invariant matching at each place.

Alternatively, Badino et al. [START_REF] Badino | Real-time topometric localization[END_REF] apply SURF descriptor to entire images, constructing a single feature Whole-Image SURF (WI-SURF) for each place. The successful results

for long-term localization experiments are reported, concluding its validity for solving the global localization problem. Similarly, motivated by the success of GIST and BRIEF binary descriptors, BRIEF-GIST [START_REF] Sünderhauf | BRIEF-Gist-Closing the loop by simple means[END_REF] is proposed. BRIEF feature is computed in a similar whole-image fashion [START_REF] Calonder | BRIEF: Binary robust independent elementary features[END_REF]. BRIEF-GIST demonstrated high computational efficiency and no requirement for vocabulary training is needed.

Other possible implementation consists in partitioning the image into a grid, computing descriptor for each patch and concatenating the obtained descriptors to form the final feature. In paper [START_REF] Lategahn | How to learn an illumination robust image feature for place recognition[END_REF], In general, global image features are easier to compute and save storage space, they are better suited to varying conditions and can be modified into more robust patch-normalized or shadow-invariant forms, they are more sensitive to camera viewpoint. Some practical viewpoint problems-such as those due to vehicle rotation-can be ameliorated with panoramic imagery and the modified GIST descriptor.

Methods based on local and global features combination

Global and local features demonstrate useful interest for place recognition based visual localization. However, each has its own advantages and disadvantages. In order to maximize the benefits of each feature type, several authors have proposed solutions based on combination of different image features for place recognition based visual localization [START_REF] Johns | Feature co-occurrence maps: Appearance-based localisation throughout the day[END_REF].

Murillo et al. [START_REF] Murillo | From omnidirectional images to hierarchical localization[END_REF] propose a three-step hierarchical localization method using omnidirectional images. A global color descriptor is applied to obtain a set of susceptible loop candidates, and then line features described by their line support regions are matched using pyramidal matching in order to find the most similar image given a predefined visual memory.

Another approach is to use a global descriptor to perform a fast selection of similar images during image searching and then use a more accurate process to confirm the association, such as matching local features. Goedemé, et al. [START_REF] Goedemé | Markerless computer vision based localization using automatically generated topological maps[END_REF] present a localization system using omni-directional cameras where, for each acquired image, vertical column segments are extracted and described with ten different descriptors. After a clustering process, these local descriptors are inserted into a kd-tree structure that is used by the localization process.

When a query image arrives, the same local descriptors applied to the vertical structures are computed over the entire image and used to rapidly retrieve possible loop candidates.

Next, a matching distance based on the column segments is applied between the image and each of the candidates in order to ensure a correct image matching.

In work [START_REF] Lu | Robust place recognition based on omnidirectional vision and real-time local visual features for mobile robots[END_REF], a robust and real-time visual place recognition algorithm is proposed by combining the local visual features FAST (Features from Accelerated Segment Test)

and CSLBP (Complete Center-symmetric Local Binary Patterns). Based on the proposed features, bag-of-features and support vector machines are used to realize place recognition based on omni-directional vision for mobile robots. The experimental results show that the robot can achieve robust place recognition with high classification rate in real-time.

Wang and Lin present a combined local and global descriptor for omni-directional images called Hull Census Transform (HCT) [START_REF] Wang | A hull census transform for scene change detection and recognition towards topological map building[END_REF], which consists of repeatedly generating the convex hull from the extracted SURF features and computing the relative magnitude between these features that compose the convex hull, resulting into a set of binary vectors.

This representation is then used for detecting scene changes.

A location recognition system which combined edges, local features and color histograms was proposed by Wang and Yagi [START_REF] Wang | Robust location recognition based on efficient feature integration[END_REF]. In this system, image description process is computed in an integrated way: the Harris detector is used to obtain both edges and interests points, while SIFT is used for describing interest points.

Methods using three-dimensional information

In addition to description of the places with 2D model directly in the visual domain (instead of making a geometric-model), they can also be extended with metric information [START_REF] Neira | Guest editorial special issue on visual SLAM[END_REF].

Thus, 2D image with metric information can be regarded as three-dimensional (3D) information. Metric range information can be inferred using stereo cameras [START_REF] Davison | Simultaneous localization and map-building using active vision[END_REF]. Monocular cameras can also infer metric information using structure-from-motion algorithms as in the following methods: MonoSLAM [START_REF] Davison | Monoslam: Real-time single camera SLAM[END_REF], PTAM [START_REF] Klein | Parallel tracking and mapping for small ar workspaces[END_REF], LSD-SLAM [START_REF] Caruso | Large-scale direct slam for omnidirectional cameras[END_REF], and ORB-SLAM [START_REF] Mur-Artal | ORB-SLAM: A versatile and accurate monocular slam system[END_REF].

Several works in the literature use 3D (three-Dimensional) information to improve performance of place recognition based localization methods. In [START_REF] Cummins | FAB-MAP: Probabilistic localization and mapping in the space of appearance[END_REF], FAB-MAP (Fast Appearance Based Mapping) is extended to incorporate the spatial distribution of visual words in 3D. Similarly, a combination of visual words with 3D information from stereo sequences is used in [START_REF] Cadena | Robust place recognition with stereo sequences[END_REF] to perform robust place recognition.

Morioka et al. [START_REF] Morioka | Vision-based mobile robot's slam and navigation in crowded environments[END_REF] propose a SLAM navigation method that is effective even in crowded environments by extracting robust 3D PIRF (Position Invariant Robust Feature) points from sequential images and odometry.

In paper [START_REF] Fiolka | Distinctive 3D surface entropy features for place recognition[END_REF], the authors present a variant of SURE, an interest point detector and descriptor for 3D point clouds and depth images, and use them for recognizing semantically distinct places in indoor environments. They also demonstrated that SURE features are well suited for place recognition using a bag-of-words approach.

Paper [START_REF] Maddern | CAT-SLAM: Probabilistic localisation and mapping using a continuous appearance-based trajectory[END_REF] Paper [START_REF] Cadena | Robust place recognition with stereo sequences[END_REF] proposes a place recognition algorithm for SLAM system using cameras.

It considers both appearance and geometric information of interest points in the images.

Hypotheses about loop closings are generated using a fast appearance-only technique based on the bag-of-words method. According to the indoor and outdoor data experiments, it

shows that the proposed system can attain at least full precision (no false positives) for high recall (fewer false negatives).

Figure 23: An example of detecting SURE features in depth images at locations with locally prominent surface curvature (from [START_REF] Fiolka | Distinctive 3D surface entropy features for place recognition[END_REF]). SURE feature captures local shape and colored texture at interest points. Based on SURE features, places are recognized using a bag-of-words approach.

Many other systems use data from additional sensors such as RGB-D cameras [START_REF] Endres | An evaluation of the RGB-D SLAM system[END_REF].

These sensors provide dense depth information as well as image data and then can exploit 3D object information to improve place recognition and localization accuracy [START_REF] Finman | Toward object-based place recognition in dense RGB-D maps[END_REF]. As illustrated in Fig. 2

Methods using deep learning features

Place recognition methods based on the hand-crafted features are prone to be affected by the changing of illumination or appearance. Their performance in challenging environments strongly depends on the invariance of those descriptors to perceptual changes. Nowadays, it is rapidly becoming apparent that in recognition tasks hand-crafted features are being outperformed by deep learning features [START_REF] Gomez-Ojeda | Training a convolutional neural network for appearanceinvariant place recognition[END_REF]. Deep learning features obtained from deep neural networks show strong power for place describing [START_REF] Sharif Razavian | Cnn features off-the-shelf: An astounding baseline for recognition[END_REF]. Thanks to the deep neural networks, place recognition based visual localization using automatic learned features is interesting and promising.

Convolutional Neural Network (ConvNet) as one of the popular deep neural networks, was firstly proposed by LeCun et al. [START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF] in 1989. ConvNet features are learned automatically from datasets through multi-layer supervised networks. ConvNets permit to achieve significant performance improvement on object classification or recognition, and outperform traditional hand-crafted features based approaches [5].

In paper [START_REF] Chen | Convolutional Neural Network-based place recognition[END_REF], a place recognition technique based on ConvNets model is presented by combining the powerful features learned by ConvNets with a spatial and sequential filter.

Applying the system to a 70 km benchmark place recognition dataset (Eynsham dataset), 85.7% recall is achieved at 100% precision.

Sünderhauf et al. [START_REF] Sunderhauf | Place recognition with ConvNet landmarks: Viewpoint-robust, condition-robust, training-free[END_REF] present a novel place recognition system that is built on state-ofthe-art object detection methods and convolutional visual features. As illustrated in Fig. 234, the astonishing power of convolutional neural network features is used to identify matching landmark proposals between images to perform place recognition over extreme appearance and viewpoint variations. The experiment results have also revealed further insights: midlevel ConvNet features appear to be highly suitable as descriptors for landmarks of various sizes in a place recognition context.

Paper [START_REF] Sünderhauf | On the performance of ConvNet features for place recognition[END_REF] presents a thorough investigation on the utility of ConvNet features for the important task of visual place recognition in robotics. Then, a novel method is proposed by combining the individual strengths of the high-level and mid-level feature layers to partition the search space and to recognize places under severe appearance changes. In addition, locality-sensitive hashing and novel (semantic search space partitioning) optimization techniques are used for real-time place recognition. Comprehensive study on four real world datasets highlighted that the proposed method performed better for place recognition when faced with appearance and viewpoint changes.

In paper [5], the authors develop a convolutional neural network architecture that is trainable in an end-to-end manner. The main component of this architecture is a new generalized VLAD layer (NetVLAD). The layer is readily pluggable into any convolutional neural network architecture and amenable to training via back-propagation. The proposed architecture significantly outperforms non-learnt image representations on Tokyo dataset, as well as on the Oxford and Paris image retrieval benchmarks.

In paper [START_REF] Gomez-Ojeda | Training a convolutional neural network for appearanceinvariant place recognition[END_REF], a convolutional neural network is trained for the first time with the purpose of recognizing revisited locations under severe appearance changes. It maps images to a low dimensional space where euclidean distance represent place dissimilarity. In order to help the network to deal with weather or illumination variations, the authors train the network with triplets of images selected from datasets which present a challenging variability in visual appearance.

In paper [START_REF] Chen | 3D object tracking via image sets and depth-based occlusion detection[END_REF], the authors conduct a comprehensive performance comparison of the utility of features from all the ConvNet 21 layers for place recognition. In work [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF],

AlexNet ConvNet model was trained on the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) for object recognition.

In addition, the availability of pre-trained network models makes ConvNets easy to experiment for place recognition. The software packages Overfeat [START_REF] Sermanet | OverFeat: Integrated recognition, localization and detection using convolutional networks[END_REF], Caffe [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF] and MatConvNet [START_REF] Vedaldi | Matconvnet: Convolutional neural networks for MATLAB[END_REF] provide network architectures pre-trained for a variety of recognition tasks. Especially, MatConvNet, an important ConvNet MATLAB toolbox designed with an emphasis on simplicity and flexibility, allows fast prototyping of new ConvNet architectures and supports efficient computation on CPU and GPU [START_REF] László | Analysis of a GPU based CNN implementation[END_REF].

Approaches Based on Place Recognizing Methods

Place recognizing methods for visual localization can be divided into two categories: 1) methods based on single image matching; 2) methods based on sequence matching.

Methods based on single image matching

Traditionally, visual localization has been performed by considering place as single image.

The basic technique consists of building a database of images collected off-line by a robot or a vehicle. Then the most similar to the currently acquired one can be retrieved. If two places are similar enough, they can be regarded as taken from the same location. As Fig. 2345shows, each testing image has to retrieve its similar one from the training database. Once place is recognized based on retrieved images, the robot or vehicle can localize itself, by assimilating its position to the one of the retrieved image from the training database.

Many place recognition based visual localization approaches are realized through matching appearance of the current scene image to the training images from database [START_REF] Lowry | Visual place recognition: A survey[END_REF]. FAB- Chow Liu tree, computed from a set of training data as the maximum-weight spanning tree of a directed graph of co-occurrences between visual words. FAB-MAP handles the perceptual aliasing problem by considering not only whether two locations were similar in the sense that they have many visual words in common, but also whether the words in common were sufficiently rare so that the locations could be considered distinctive.

Knopp et al. [START_REF] Knopp | Avoiding Confusing Features in Place Recognition[END_REF] perform large-scale appearance-based localization using bag-of-feature representation, but consider only matches to individual images in the database without considering linear combination of bag-of-feature vectors.

Using single image matching for place recognition is an easy and simple way. However, when robotic systems operate in larger uncontrolled environments and for longer time periods, place recognition using single image is prone to be affected by the changing of illumination and moving objects (e.g. cars or pedestrians).

Methods based on sequence matching

Early place recognition systems often implicitly used the simplifying assumption that the visual appearance of each place would not change over the course of the experiment. However, as robotic systems operate in ever-larger uncontrolled environments and for longer time periods, it has rapidly become apparent that this assumption is no longer valid. When the appearance of an environment is changing, appearance-based place matching becomes less reliable and the relative topological structure of an environment becomes more important. Instead of calculating the single location similarity between a current image, sequences of images can be used to match places despite changes in lighting and weather conditions, or poor visibility [START_REF] Milford | SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights[END_REF][START_REF] Pepperell | All-environment visual place recognition with SMART[END_REF]. As Fig. 2-6 illustrates, sequences A and A' are taken in a time interval of two weeks, although illumination and objects (cars and trees) are changed, matching using sequence can still recognize the place successfully.

More recently, SeqSLAM (Sequence Simultaneous Localization and Mapping) [START_REF] Milford | SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights[END_REF] introduces the idea of matching places by considering sequences instead of single images like previous proposals such as FAB-MAP. In SeqSLAM method, the matrix of image similarities between local query (testing) image sequence and training image sequence is constructed firstly. Image similarity is evaluated using the sum of absolute differences between contrast enhanced, low-resolution images without the need of image keypoint extraction.

The place recognition score is the maximum sum of normalized similarity scores over predefined constant velocity paths (i.e. alignments between the query sequence and database sequence images) through the matrix. Using this sequence matching approach significantly improves place recognition reliability.

The sequence-based approach can operate reliably in these conditions because it does not require the image comparison step to achieve 100% correctness -so long as the correct location is more similar than an incorrect location sufficiently frequently the sequence filter can identify the path [START_REF] Milford | Long exposure localization in darkness using consumer cameras[END_REF].

Conclusions

This chapter presented a survey of relevant works in the field of place recognition based visual localization. Approaches of this issue have been studied extensively, however, place recognition based visual localization in changing environments can still be improved. The contributions of this thesis are motivated by the reviewed research, in particular, the presented works are inspired by multi-feature combination and sequence-based methods:

(1) As presented in section 2. This multi-feature takes the advantage of image texture, depth and shape information at the same time, it hence permits to achieve better place recognition performance than image single feature. To evaluate the proposed method, experiments are conducted on several real outdoor datasets. Furthermore, to speed up the process of place recognition, Locality Sensitive Hashing (LSH) is used to compress the high dimensional feature data and accelerate the process of similar images search.

Introduction

One of the prerequisites of navigation issue is to make the vehicle or robot able to reliably determine its position within its environment. With the wide use of cameras, varieties of approaches were proposed to address the challenges of place recognition based visual localization [115] [35].

As already mentioned in the literature review (Chapter 2), FAB-MAP method can be considered as the milestone in the field of visual localization. FAB-MAP approach consists in matching the appearance of current scene to a same (similar) past visited place by converting the images into bag-of-words representations built on local features such as SIFT or SURF.

In local feature based place recognition approaches, image representation is defined as collection of local features which contribute with their robustness when faced with local image variations as well as from discriminative power of their descriptors. Nevertheless, most of these works exhibit a high computation cost or complex feature extraction for image matching. Also, few works pay attention to the depth information for visual place recognition.

Recently, binary image descriptors that encode patch appearance using compact binary string with low memory requirements, are widely used in image description and visual recognition [START_REF] Sun | Application of image retrieval based on the improved local binary pattern[END_REF]. Their advantages are that they are invariant to monotonic changes in gray-scale and fast to calculate. One typical binary descriptor is LBP (Local Binary Pattern). Since it was firstly proposed in 1996, several new variants of binary descriptors have been proposed [START_REF] Bekele | Evaluation of binary keypoint descriptors[END_REF]. In this chapter, the most relevant binary descriptors for visual place recognition that will be tested and compared in our approach are: LBP, CLBP (Complete Local Binary Pattern) [START_REF] Guo | A completed modeling of local binary pattern operator for texture classification[END_REF], CSLBP (Center-symmetric local binary patterns) [START_REF] Heikkilä | Description of interest regions with center-symmetric local binary patterns[END_REF], CSLDP (center-symmetric local derivative pattern) [START_REF] Xue | Hybrid center-symmetric local pattern for dynamic background subtraction[END_REF] and XCSLBP (extended CSLBP) [START_REF] Silva | An extended centersymmetric local binary pattern for background modeling and subtraction in videos[END_REF].

These different local binary descriptors are noted as λ LBP.

Despite local binary features efficiency, histograms of oriented gradients (HOG) features have also been successfully used in various vision tasks such as object classification, image search and scene classification [START_REF] Yamauchi | Relational HOG feature with wild-card for object detection[END_REF]. Xiaoyu Wang et al. [START_REF] Wang | An HOG-LBP human detector with partial occlusion handling[END_REF] combine histograms of oriented gradients (HOG) and local binary pattern (LBP), and propose a novel human detection approach capable of handling partial occlusion. For such applications, HOG is one of the best features to capture edge or local shape information which provides a rough description (shape information) of the scene.

Considering the robust and strong image representation ability of binary descriptors and HOG feature, we expect that their combination would provide more useful information and We extend the application of λ LBP descriptor to disparity map in order to incorporate disparity information in image representation by simply concatenating the two descriptors (λ LBP from gray-scale image and λ LBP from disparity map). This produces a new descriptor is named D-λ LBP. The integration of disparity information in image representation provides depth information which should be helpful for place recognition, especially in complex environment situation. Indeed, image description using features λ LBP and HOG and the depth information will permit to reduce perceptual aliasing problems related to visual place recognition. As it will be shown in our experiments, features combination permits to achieve better recognition performance than single feature. Also the performance of place recognition is compared with the state-of-the-art FAB-MAP algorithm: the achieved F 1 scores on four tested datasets using our approach are better than those resulted from FAB-MAP method. Furthermore, considering high dimensional multi-features comparison is time-consuming, locality sensitive hashing is applied on multi-features to speed up the process of features comparison and image matching.

The most important contributions introduced in this chapter are the following:

• An innovative method for place recognition based visual localization using multifeature descriptor (D-λ LBP++HOG) extracted from gray-scale image and disparity map. The proposed multi-feature descriptor takes advantage of texture, depth and shape information and hence performs better than single feature (see Section 3.5.2).

• The impact image block size for the binary descriptors is studied. Binary descriptor extracted from small block has better discriminative ability in local details of different locations, while considering large block size for image representation may cause loss of some discriminative information (see Section 3.5.1).

• A speeding-up of the place recognition method is achieved by approximating the euclidean distance between features with hamming distance over bit-vectors obtained by Locality Sensitive Hashing (see Section 3.5.3).

The rest of this chapter is organized as follows. Firstly, the LBP descriptor and several of its variants as well as HOG feature are introduced in Section 3.2. Then, in Section 3.3, the proposed approach is described in detail. Section 3.4 deals with the presentation of the tested database and the used performance evaluation parameters. The obtained results are presented and discussed in Section 3.5. Finally, conclusions and future works close this chapter (Section 3.6).

Overview of used Image Descriptors

In this part, some of the state-of-the-art image descriptors used and compared in the proposed approach are described.

LBP (Local Binary Pattern)

LBP is a texture descriptor that codifies local primitives (such as curved edges, spots, flat areas) into a feature histogram. The original LBP operator labels the pixels of an image with decimal numbers, called Local Binary Patterns or LBP codes, which encode the local structure around each pixel [START_REF] Kylberg | Evaluation of noise robustness for local binary pattern descriptors in texture classification[END_REF].

As illustrated in Fig. 3 The local binary code for the position (x c , y c ) can be computed by comparing the gray-scale value g c of this center pixel located at (x c , y c ) and the gray-scale values g p of its neighbor pixels located at (x p , y p ) where p ∈ {0, ..., P -1}. The value of the LBP code of the center pixel at position (x c , y c ) is given by:

LBP P,R (x c , y c ) = P-1 ∑ p=0 s(g p -g c )2 p (3.2)
where s is the Heaviside function:

s(x) =    1, x ≥ 0 0, otherwize (3.3)
The operator LBP P,R produces 2 P different output values, corresponding to 2 P different binary patterns formed by the P pixels in the neighborhood. Although this method can capture the relations of nearby and adjacent pixels, it leads to a large data dimension.

Ojala et al. [START_REF] Ojala | A comparative study of texture measures with classification based on featured distributions[END_REF] further propose an "uniform patterns" to reduce the dimension of LBP feature while keeping its discrimination power. For this, an uniformity measure of a pattern is used: U ("pattern") is the number of bitwise transitions from 0 to 1 or vice versa when the bit pattern is considered circular. The U value of an LBP pattern can be computed by:

U(LBP P,R ) = |s(g P-1 -g c ) -s(g 0 -g c )| + P-1 ∑ p=1 |s(g p -g c ) -s(g p-1 -g c )| (3.4)
Uniform LBP patterns refer to the patterns which have limited transitions or discontinuities (U ≤ 2) in the circular binary representation. For instance, 11111111 (0 transitions) and 01110000 (2 transitions) are both uniform whereas 11001001 (4 transitions) and 01010010 (6 transitions) are not. Thus, for P neighborhood pixels, a uniform LBP P,R operator produces P(P -1) + 3 possible distinct uniform LBP patterns. After the uniform LBP patterns are identified, for an image with size N × M, a histogram is built which can be used as the image feature to represent the image texture :

h(l) = N ∑ i=1 M ∑ j=1 f (LBP P,R (i, j), l) , l ∈ [0, L], (3.5) f (x, y) =    1, x = y 0, otherwize (3.6)
where L is the maximal LBP pattern value. The length of the histogram is a P(P -1) + 3.

CLBP (Complete Local Binary Pattern)

LBP feature considers only signs of local differences (i.e. difference of each pixel with its neighbors) whereas CLBP feature [START_REF] Guo | A completed modeling of local binary pattern operator for texture classification[END_REF] considers both magnitude (M) and sign (S) of local differences as well as original center gray level value (C) . Consequently, three operators, namely CLBP_M, CLBP_S and CLBP_C, are used to code the magnitude, sign and center gray level.

Given the gray-scale value g c of the center pixel (x c , y c ) and its P circularly and evenly spaced neighbors with gray-scale value g p , p ∈ {0, ..., P -1}, the difference between g c and g p can be simply calculated using

d p = g p -g c . The local difference vector [d 0 , d 1 , • • • , d P-1 ] characterizes the image local structure at (x c , y c ). Because the central gray level g c is removed in local difference vector, [d 0 , d 1 , • • • , d P-1
] is robust to illumination changes and is more efficient in pattern matching. d p can be further decomposed into two components:

d p = s p * m p (3.7) m p = |d p |, s p =    1, d p ≥ 0 -1, d p < 0 (3.8)
where s p is the sign component of d p and m p is the magnitude component of d p .

CLBP_M is used to code the magnitude information of local differences:

CLBP_M P,R (x c , y c ) = P-1 ∑ p=0 t(m p , c)2 p , t(x, T ) =    1, x ≥ T 0, x < T (3.9)
where T is a threshold which is set to the mean value of the m p values from the whole image.

CLBP_S is the same as the original LBP and is used to code the sign information of local differences:

CLBP_S P,R (x c , y c ) = P-1 ∑ p=0 t(s p , 0)2 p , t(x, T ) =    1, x ≥ T 0, x < T (3.10)
CLBP_C is used to code the information of original center gray level value:

CLBP_C P,R (x c , y c ) = t(g c , c I ), t(x, T ) =    1, x ≥ T 0, x < T (3.11)
where the threshold c I is set to the average gray level of the input image.

The dimension of the histograms corresponding to CLBP_S and CLBP_M is 2 P , while the dimension of CLBP_C is 2. The CLBP_C only uses the center gray level value which can be easily affected by the changing of viewpoints or illumination. Therefore, in our work, only the histograms of CLBP_S and CLBP_M codes are computed and then concatenated together to construct CLBP feature. Thus, the final dimension of CLBP feature is 2 P+1 .

CSLBP (Center-symmetric local binary patterns)

CSLBP [START_REF] Heikkilä | Description of interest regions with center-symmetric local binary patterns[END_REF] is another modified version of LBP. CSLBP produces shorter feature set than LBP, but it is also a first order local pattern in center symmetric direction and it ignores the central pixel information. CSLBP is closely related to the gradient operator, because it compares the gray levels of pairs of pixels in centered symmetric directions instead of comparing the central pixel to its neighbors. In this way, CSLBP feature takes advantage of the properties of both LBP and gradient based features.

For an even number P of neighboring pixels distributed on radius R, CSLBP operator produces 2 P/2 possible distinct patterns. The operator is given by:

CSLBP P,R (x c , y c ) = (P/2)-1 ∑ i=0 s(|g i -g i+(P/2) |)2 i (3.12) s(x)    1, x ≥ T 0, otherwise (3.13)
where g i and g i+(P/2) are the gray values of center-symmetric pairs of pixels. T is used to threshold the gray-level difference so as to increase the robustness of CSLBP feature on flat image regions. Since the gray levels are normalized in [0,1], the authors of paper [START_REF] Heikkilä | Description of interest regions with center-symmetric local binary patterns[END_REF] recommend to use small value for T .

It should be noticed that CSLBP is closely related to gradient operator, because like some gradient operators it considers gray level difference between opposite pixels in a neighborhood.

Given an image of size N × M, after the computation of CSLBP patterns, a histogram is built to represent the texture image:

h(l) = N ∑ i=1 M ∑ j=1 f (CSLBP P,R (i, j), l) , l = 0, 1, 2, 3, • • • , 2 P/2 -1 (3.14) f (x, y) =    1, x = y 0, otherwize (3.15)
By construction, the length of the histogram resulting from CSLBP feature is 2 P/2 .

CSLDP (Center-Symmetric Local Derivative Pattern)

CSLDP operator [START_REF] Xue | Hybrid center-symmetric local pattern for dynamic background subtraction[END_REF] is a second order derivative pattern in center symmetric direction.

CSLDP captures more information by encoding the relationship between central pixel and center symmetric neighbors. Moreover, CSLDP has shorter length than LBP.

For an even number P of neighboring pixels distributed on radius R, CSLDP operator produces 2 P/2 possible distinct patterns and is defined as:

CSLDP P,R (x c , y c ) = P/2-1 ∑ i=0 t[(g i -g c ), (g c -g (i+(P/2) )]2 i (3.16)
where g i , g (i+(P/2) are gray-scale values of neighborhood pixels in center symmetric direction. g c corresponds to the gray value of central pixel located at (x c , y c ). The threshold function t(•, •) is used to determine the type of local pattern transition and is defined as:

t(x 1 , x 2 ) =    1, x 1 • x 2 ≤ 0 0, x 1 • x 2 > 0 (3.17)
A CSLDP pattern encodes the second order center symmetric derivatives at pixel (x c , y c ) along 0 • , 45 • , 90 • and 135 • directions. They can be represented as:

                 CSLDP 0 • (x c , y c ) = t[(g 0 -g c ), (g c -g 4 )] CSLDP 45 • (x c , y c ) = t[(g 1 -g c ), (g c -g 5 )] CSLDP 90 • (x c , y c ) = t[(g 2 -g c ), (g c -g 6 )] CSLDP 135 • (x c , y c ) = t[(g 3 -g c ), (g c -g 7 )]
(3.18)

The CSLDP histogram construction method is the same as for CSLBP, and its histogram length is also 2 P/2 .

XCSLBP (extended CSLBP)

The work in [START_REF] Silva | An extended centersymmetric local binary pattern for background modeling and subtraction in videos[END_REF] proposes a new LBP variant called XCSLBP (eXtended CSLBP), which compares the gray values of pairs of center symmetric pixels considering the central pixel, without increasing histogram length. This combination makes the resulting descriptor robust to illumination changes and noise. For an even number P of neighboring pixels distributed on radius R, XCSLBP is expressed as:

XCSLBP P,R (x c , y c ) = (P/2)-1 ∑ i=0 s(g 2 c + g i+(P/2) (g i -2g c ))2 i , ( 3.19)
where the threshold function s, which is used to determine the types of local pattern transition, is defined as:

s(x) =    1, (x ≥ 0) 0, otherwise (3.20)
where g i and g i+(P/2) are the gray values of center symmetric pixels. XCSLBP operator produces histograms with a length of 2 P/2 .

HOG (Histograms of Oriented Gradients)

Besides LBP and its variants, another histogram feature named HOG has also been widely accepted as one of the best features to capture the edge or local shape information. HOG feature is proposed by Dalal et al. [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] and widely used to detect objects in computer vision.

The essential idea of HOG feature is that the shape or appearance of local object can be described by the distribution of intensity gradients and edge directions [START_REF] Ren | Object detection using edge histogram of oriented gradient[END_REF]. HOG descriptor is a one-dimensional histogram of gradient orientations of intensity in local regions that can represent object shape.

As shown in Fig. 34, HOG divides the image into small connected blocks, and for each block, a histogram of gradient directions for the pixels within the block is computed.

The combination of these cell histograms represents the feature vector. At each pixel, the gradient is a 2D vector with a real-valued magnitude and a discretized direction (9 possible directions uniformly distributed in [0, π]). During the construction of the integral image of HOG, the feature value at each pixel is treated as a 9D vector, and the value at each dimension is the interpolated magnitude value at the corresponding direction. Since HOG takes adjacent pixel gradients information as basis to extract features, it is robust to changes in geometry and is not easily affected by local lighting conditions.

Overview of Proposed Approach

In this section, a robust visual localization based on multi-feature combination is developed.

The general principle is to find the image that best matches the current acquired one, among a set of previously acquired and GPS-tagged training images.

The whole system includes an off-line phase and an on-line phase. In the off-line phase, i , and then compared with each multi-feature of V train based on euclidean distance. The computed distances are then used to select the best candidate (see Section 3.3.4), smaller the distance is, higher similarity between the images will be. A distance ratio SS between the two best candidates (i.e. corresponding to the two minimum computed distances) is considered for matching validation (see Section 3.3.5). If the ratio SS is lower than or equal to a threshold T h, the first best image candidate (with the lower As illustrated in Fig. 345, the overall approach comprises six stages:

a

Image preprocessing

Image preprocessing is composed of two parts: down-sampling and contrast-limited adaptive histogram equalization (CLAHE). Down-sampling permits to reduce the original image size, which makes feature extraction faster. In fact, it has been already proved in [START_REF] Roy | Visual Route Recognition with a Handful of Bits[END_REF] that high resolution images are not more helpful than lower resolution ones. Therefore, down-sampling is the first step before feature extraction. As it is well known, illumination has significant influence on outdoor image appearance. Therefore, another applied image preprocessing is contrastlimited adaptive histogram equalization (CLAHE), which permits to enhance the contrast of the gray-scale image by transforming the values using contrast-limited adaptive histogram equalization [START_REF] Etta D Pisano | Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms[END_REF]. Through this adjustment, the intensities can be better distributed on the histogram. This allows for areas of lower local contrast to gain higher contrast. This contrast, especially in homogeneous areas, can be limited to avoid amplifying any noise that might be present in the image. On the same time, it also decreases the shadow influence. An image example after applying contrast-limited adaptive histogram equalization can be seen in Fig. 3456. It is obvious that CLAHE prepossessing improves the image contrast and makes the image more brighten (especially in some dark parts).

Block based feature extraction

After image preprocessing, features are extracted, as illustrated in Fig. 345678. λ LBP feature is extracted from gray-scale image and disparity map independently. While HOG feature v HOG is extracted from gray-scale image. For both λ LBP or HOG, the features are extracted based on image blocks. In order to facilitate the process of block based feature extraction, image blocks in the full image have the same size. The influence of different block sizes will be studied in the Section 3.5.1. Image parts that cannot satisfy a whole block will be ignored.

• λ LBP feature extraction λ LBP feature from gray-scale image and disparity map are obtained using the following equations:

   v gray =
(i ∈ [1, 2, • • • , m]) is
the λ LBP histogram of the i th block of the grayscale image and

h dis i (i ∈ [1, 2, • • • , n]
) is the λ LBP histogram of the i th block of the disparity map. In our work, the disparity map is calculated using the SGBM (Semi-Global Block Matching) algorithm [START_REF] Hirschmuller | Stereo processing by semiglobal matching and mutual information[END_REF].

Using this SGBM method, there are some useless parts (" black areas"), for which no depth information is computed, especially on the left and right sides of the disparity map. In these " black areas", λ LBP operator is not applied, therefore these useless parts are simply removed. Thus, due to the removing of the " black areas" in the disparity map, m and n are not identical. By using the block based approach, the features v gray and v dis are extracted from grayscale image and disparity map respectively. Then, D-λ LBP feature can be computed by concatenating v gray and v dis :

v D-λ LBP = v gray + +v dis (3.22)
• HOG feature extraction HOG feature is also computed for each image block of the gray-scale image. The obtained HOG features from all the image blocks are then concatenated:

v HOG = h hog 1 + +h hog 2 + +... + +h hog m (3.23)
Here,

h hog i (i ∈ [1, 2, • • • , m]
) is the HOG feature extracted from the i th image block. It should be noted that HOG feature adopts the same image block size as the λ LBP feature extraction from gray-scale image, therefore the number of image blocks is same.

Multi-feature concatenation

In order to take advantage of the different features, D-λ LBP and HOG are combined together to represent the image. Since the D-λ LBP and HOG are two independent features, we simply consider that they have the same weight in the role of place recognition. The final multi-feature can be obtained easily through concatenation using the following equation:

v = v D-λ LBP + +v HOG (3.24)
Using this method, a multi-feature set V train = {v train

Feature comparison and image matching

Feature comparison is performed based on the euclidean distance between features. Each testing image multi-feature v test i is compared with all the training images multi-features

v train j ( j = [1, 2, • • • , N train ]) of the training database.
The distance D i, j between the multi-feature

v test i (i = [1, 2, • • • , N test ]
) of the testing image and multi-feature vector v train j

( j = [1, 2, • • • , N train ]) of a training image is computed as follows: D i, j = ∥v test i -v train j ∥ 2 (3.25)
where ∥•∥ denotes the euclidean norm.

In fact, small distance means high similarity. Based on euclidean distance, image matching candidates are searched. After distance computation, for the testing image, the two minimum distances (D i,m1 and D i,m2 ) and their corresponding training images (the two best candidates) are conserved.

Final matching validation

For a given current image pair I test i , the validation of matching candidate from the training database is based on the ratio SS i , calculated as follows:

SS i = D i,m1 D i,m2 (3.26)
where D i,m1 and D i,m2 are respectively the first and second minimum distances between the current image multi-feature v test i and the multi-features {v train j } N train j=1 of all the training images:

       D i,m1 = min j {D i, j } D i,m2 = min j( j̸ =m1) {D i, j } (3.27)
As said before, lower the distance is, more similar the images are. The potential matching candidate is the image m i (the one giving the lower distance with the testing image). However, if the second best matching candidate provides a distance very close to the first one, this means that the matching algorithm provides two confused solutions. In this case, we propose to ignore the matching result and consider that the testing image has no matching image. For that, a threshold T h is applied to the ratio SS i , which takes its values in the

range [0 1].
The last decision is as follows: if SS i is lower than or equal to the threshold T h, then the pair (i, m1) is considered as positive, and the pair is matched. Otherwise, the pair is considered as negative and the pair is ignored.

Visual localization

After image matching result is successfully validated, the vehicle can localize itself through the matched training image position. Since the training images are tagged with the GPS or pose information, the vehicle can get its position information by assimilating its position to the GPS position of the training image matched with the current testing image. This is a topological level localization, that is, the system simply identifies the most likely location.

Therefore, this is not a very accurate metric localization, because the training and testing trajectories are not exactly same.

It should be noted that, some places can not be localized at the situation of validation failure (negative matching case).

Algorithm of multi-feature based visual localization

The approach for place recognition based visual localization proposed in this chapter is summed up in Algorithm 3.1. The multi-features are constructed firstly from training database, this is off-line processing. Then multi-feature built from the current image pair is compared with the whole multi-features from training database to obtain the euclidean distance vector. Based on the this, image matching result is validated for the final visual localization.

Experimental Setup

Datasets and ground-truth

The proposed method is tested on four different datasets (UTBM-1, UTBM-2, KITTI 05 and KITTI 06).

The taken route for UTBM-1 dataset is shown in Fig. 3 The UTBM-2 dataset (Fig. 3 For UTBM-1 and UTBM-2 datasets, ground-truth was constructed by manually finding pairs of frame correspondences according to the GPS data. While the KITTI dataset ground-truth was built according to the pose information [START_REF] Arroyo | Fast and effective visual place recognition using binary codes and disparity information[END_REF].

Image preprocessing and feature extraction

In our work, for faster feature extraction, the original color images were down-sampled into half scale size grayscale image. That means images in dataset UTBM-1 and UTBM-2

were resized to 640 × 480 and the images in dataset KITTI 05 and KITTI 06 were resized to 613×235.

In order to reduce the illumination influence on the outdoor image appearance, contrastlimited adaptive histogram equalization (CLAHE) method was used (see Section 3.3.1).

Moreover, as a pair of images is acquired at each instant, a disparity map can be computed easily using the SGBM (Semi-Global Block Matching) algorithm [START_REF] Hirschmuller | Stereo processing by semiglobal matching and mutual information[END_REF].

After image preprocessing, binary descriptors (LBP, CLBP, CSLBP, CSLDP and XC-SLBP) are extracted with the following parameters: 8 sampling points and 3 pixels radius.

HOG descriptor is extracted from the grayscale images. To capture large-scale spatial information, the cell size of HOG is 32×32. The number of cells in each block is specified as a 2-element vector. An example of extracted image features can be seen in Fig. 3-10. It can be seen that the local binary features pay more attention to texture information. It can also be noted that CSLBP and XCSLBP perform better than LBP. HOG feature depicts object shape information in the image. Therefore, combining LBP and HOG features could bring more information and make place(scene) better described. The final curve is computed by varying the threshold T h (applied to the ratio SS) in a linear distribution between 0 and 1, with the calculation of the corresponding values of precision and recall. 100 values of threshold T h are considered to obtain well-defined curves. When the threshold is set to 1, the candidates whose ratio is below or equal 1 are positives. In this case, the number of retrieved images is identical to the number of testing images. While when the threshold is 0, it means that the candidates whose ratio is below or equal 0 are regarded as positives. In this case, there is no retrieved image.

Performance evaluation

Precision relates the number of correct matches to the number of false matches, whereas recall relates the number of correct matches to the number of missed matches. A perfect system would return a result where both precision and recall have a value of one. The F 1 score value is a single value that indicates the overall effectiveness of image retrieval method. Based on the precision and recall, F 1 score is defined as:

F 1 = 2 × Precision × Recall Precision + Recall (3.28)

Experiments and Results

Different aspects of our proposal are evaluated in the following sections. In Section 3.5.1, the performance of binary features (LBP and its variants) with and without disparity information is studied. In addition, the image block size influence for the binary feature D-CSLBP is investigated in Section 3.5.1. In Section 3.5.2, the effect of the multi-feature fusion proposed in our approach is analyzed. It is to note that the experiment results obtained in Sections 3.5.1 and 3.5.2, are based on euclidean distance. In Section 3.5.3, the efficiency of our LSH based visual recognition is checked: the execution time and recognition performance of our complete system are evaluated. Finally, visual localization at 100% recognition level is discussed in Section 3.5.4.

Comparison of the different binary features and image block sizes Performance of different binary features

In this section, we compare binary features performance in two situations: with or without disparity map. Here the features are compared based on the euclidean distance.

Table 3.1 gives the F 1 scores of the binary descriptors in two cases (without and with disparity information). It can be seen that, LBP, CLBP, CSLBP and CSLDP with disparity information improve the image retrieval ability as F 1 scores are higher with disparity information than without disparity information. Among them, D-CSLBP is the best one, it achieves the highest F 1 score.

Comparison of different image block sizes

In this section, the influence of block size of block-based D-CSLBP feature is studied.

Small block size permits to discriminate local details, while large block size makes the representation more robust. Each image block is a square block in our experiment (block size 32×32 is short for 32 It is obvious that, the binary descriptor D-CSLBP extracted from small block size may benefit from discriminative local details. While feature extraction using larger block size makes image representation easy to drop some discriminative information.

However, when the block size is too small, the abundant information can not bring more improvement to the image matching process. At the same time, smaller image block size may lead to computation time increase during feature extraction. So, on our following experiments, the image block size for D-CSLBP is set to 32.

Performance of multi-feature combination

In this section, we compare the performance of multi-feature descriptor (D-CSLBP ++ HOG) with single independent feature descriptor. than single feature. The F 1 score of D-CSLBP++HOG provides the highest value for all the four datasets. Furthermore, the proposed method outperforms the FAB-MAP method.

For a better comprehension of the proposed multi-feature, an example of distance matrices for UTBM-1 dataset is presented in Fig. 3-14. Here, for clearly demonstrates the feature performance, the distance matrix D is normalized into 0-1 range. The distances of same or similar images are close to 0 (red color), while for the larger distances, the corresponding color is close to yellow. As plotted in Fig. 3-14 (b), the ground-truth line is a red. When perceptual aliasing occurs, there will be some red points (noisy) appeared which is outside the ground-truth line. In the distance matrix provided by our method using the D-CSLBP++HOG feature (see Fig. 3-14 (c)), it can be seen that the noisy which appear around the diagonal (ground-truth line) due to perceptual aliasing are clearly reduced with respect to other feature approaches (CSLBP, D-CSLBP and HOG). All the previous affirmations are supported by the precision-recall curves depicted in in Fig. 3-13 (a) and results in Table 3.2.

We can thus conclude that integrating HOG and disparity information permits to improve the image matching results. The reason why the D-CSLBP++HOG achieves better performance than the other features is mainly because the feature combination takes the ad- vantage of texture, shape and depth information, which makes image representation more robust than considering each single feature independently.

LSH based visual recognition

Since the block based feature dimension is huge in our approach, computing the euclidean distance between high dimensional feature vectors is an expensive operation. Therefore, in order to speed up image matching significantly, Locality Sensitive Hashing (LSH) method that preserves the euclidean similarity [START_REF] Moses S Charikar | Similarity estimation techniques from rounding algorithms[END_REF], is used for visual recognition. LSH is arguably the most popular unsupervised hashing method and has been applied to many problems, including information retrieval and computer vision [START_REF] Ravichandran | Randomized algorithms and NLP: Using locality sensitive hash function for high speed noun clustering[END_REF]. The paper [START_REF] Ravichandran | Randomized algorithms and NLP: Using locality sensitive hash function for high speed noun clustering[END_REF] demonstrates that euclidean distance between two high-dimensional vectors can be closely approximated by the hamming distance between the respective hashed bit vectors. More hash bits that hash method contains, the better approximation is.

The LSH method simply uses a random projection matrix to project the high dimensional data into a low-dimensional binary (Hamming) space; that is, each data point is mapped to a K-bit vector, called the hash key. Thus approximate nearest neighbors in sublinear time can be found. A key ingredient of locality sensitive hashing is mapping similar features to the same bucket with high probability.

More precisely, for multi-features V test obtained from testing image and V train obtained from training image, the hashing functions H(•) from LSH family satisfy the following elegant locality preserving property:

P{H(V test ) = H(V train )} = sim(V test ,V train ) (3.29)
where the similarity measure sim is directly linked to the euclidean distance function. Hash keys are constructed by applying K binary-valued hash functions to each image feature.

The K binary-valued LSH functions consists of random projections and thresholds as:

H test (K) = sign(w ⊤ V test + b) H train (K) = sign(w ⊤ V train + b) (3.30)
where w is a K dimensional data-independent random hyperplane, which is usually constructed from a standard Gaussian distribution [START_REF] Datar | Localitysensitive hashing scheme based on p-stable distributions[END_REF]. b is a random intercept. For a normalized data set with zero mean, the approximately balanced partition is obtained with b = 0.

By applying K binary-valued hash functions to each image feature, high dimension multi-features V test and V train are converted into a low K dimension bits H test and H train .

Since H test and H train are binary bits, they can be more efficiently compared in low dimension space than original feature.

In our experiment, we compare the place recognition performance achieved with hashed As Table 3.3 shows, the average matching time using 4096 bits is almost half of the one using the euclidean distance over the original full features. Compared with the full multi-feature matching, hashing the original multi-feature into 4096 bits makes the dis-tance computation and comparison easier and faster. There is no doubt that, for large scale datasets, the speed up advantages can be more significant.

Visual localization results

In the previous section, 4096 bits obtained by hashing the original feature shows its good performance in place recognition. Therefore, in this section, we describe visual localization results achieved by 4096 hash bits. Fig. 3 Visual localization results obtained by our system on four datasets. The trajectory of the vehicle is depicted with black lines, the loop closure zone is plotted by blue lines. Red points are correctly recognized locations at 100% precision by using our proposed approach. There are no false positives in any case. It is noted that the loop closure zone of datasets UTBM-1 and UTBM-2 is the whole trajectory, while the loop closure zone of KITTI 05 and KITTI 06 is only parts of the trajectory in blue.

UTBM-1 and UTBM-2 datasets respectively. It can be seen that, the visual localization error is below 5m at 100% precision level. While with threshold T h increased to 0.9, the precision level is decreasing and localization error of some locations exceeds 5m . When the threshold is set to 1, which means every image matching result is positive, in this case, the precision level is lowest and there are many false matching for place recognition, which lead to huge localization error. In general, if small threshold is used, there are few false recognition cases.

In addition, for visual recognition precision level below 100% , meaning that recog- Table 3.4 gives the average localization error and recall ratio at different precision levels. For all these datasets, at 100% precision, the minimum localization error is 0 while the maximum error is not larger than 5 m. It should be noted that, at 100% precision level, some places can not be recognized and no localization results are obtained at these places. Here, a 2D LiDAR is used, and the transformation between two successive LiDAR point sets can be computed using ICP (Iterative Closest Point) method. Thus, based on the previous position and the transformation information, the positions of unrecognized places can be computed.

This problem can be easily solved by visual odometry technique or extra sensors (as

Conclusion and Future Works

In this chapter, we presented a visual vehicle localization approach that uses multi-feature built from gray-scale image and disparity map. The multi-feature concatenates the D-CSLBP and HOG features together to take the advantage of texture, depth and shape information. Also, block based feature extraction was used to consider the spatial information.

Image matching using the proposed multi-feature D-CSLBP++HOG based on local sensitive hashing makes the visual recognition more efficient. The results of our experiment demonstrated that this approach provides an available place recognition based visual localization in outdoor environment compared with the state of art FAB-MAP method.

However, in the long-term visual localization, place recognition is prone to be influenced by appearance or seasonal changing. The future objective of our research is to achieve a robust long-life localization at different times and seasons. Sequence matching will be considered for place recognition in the following research.

Introduction

Visual localization is an ongoing challenge in robotics, especially in seasonal changing situation. A single can look extremely different depending on the current season and weather conditions. Visual localization system across different seasons must be robust without being influenced by seasonal and weather variations that lead to vast variations in image appearance.

As already cited in the previous chapter, FAB-MAP is a single image based matching algorithm, which employs Bag-of-Words (BOW) image retrieval technique and a Bayesian frame-work [START_REF] Cummins | FAB-MAP: Probabilistic localization and mapping in the space of appearance[END_REF] to achieve robust place recognition. Recently, sequence SLAM (SeqS-LAM) [START_REF] Milford | SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights[END_REF] adopting sequence matching rather than single image matching for place recognition achieves significant performance improvements with respect to FAB-MAP. The usage of sequences allows higher robustness to lighting or extreme perceptual changes. In SeqSLAM, image similarity is evaluated using the sum of absolute differences between contrast enhanced and low-resolution images without the need of image keypoint extraction. However, in [START_REF] Sünderhauf | Are we there yet? Challenging SeqSLAM on a 3000 km journey across all four seasons[END_REF] some weaknesses of SeqSLAM were reported, such as the field of view dependence and the complexity of parameters configuration. For these reasons, the community continues searching for new methods which can satisfy the high requirements needed to achieve robust life-long visual localization.

Besides that, CSLBP is one of the widely used binary descriptors, which is invariant to monotonic changes in gray-scale and fast to calculate. As proved in Chapter 3, CSLBP has strong place describing ability. Considering object shape information is changing (e.g. leaves falling down, snow covering) at different seasons, GIST feature rather than HOG is used in this work. GIST focuses more on the whole scene itself and on the relationship between the outlines of the surfaces and their properties [START_REF] Douze | Evaluation of gist descriptors for web-scale image search[END_REF]. GIST and CSLBP can be seen complementary for image representation in the sense that GIST focuses more on global information and CSLBP emphasizes local texture information. Inspired by the advantages of feature combination, CSLBP and GIST are combined together for place describing in this chapter.

In this chapter, we present a visual localization method using sequence matching based on feature combination that is robust to extreme perceptual changes.

Proposed Visual Localization Approach

The proposed visual localization method using CSLBP and GIST features to represent image sequence, is realized based on sequence matching. As illustrated in Fig. 4- transitions is no more than 2) is a P(P -1) + 3 dimension histogram (details can be found in Section 3.2 of Chapter 3). While CSLBP operator produces 2 P/2 patterns as follows:

CSLBP P,R (x c , y c ) = (P/2)-1 ∑ i=0 s(|g i -g i+(P/2) |)2 i (4.1) s(x)    1, x > T 0, otherwise (4.2)
where g i and g i+(P/2) correspond to the gray values of center-symmetric pairs of pixels (P in total) equally spaced around central pixel (x c , y c ). T is used to threshold the gray-level difference so as to increase robustness of CSLBP feature on flat image regions. Since the gray levels are normalized in [0,1], the authors of paper [START_REF] Heikkilä | Description of interest regions with center-symmetric local binary patterns[END_REF] recommend to use small value for T .

CSLBP is closely related to the gradient operator, because it compares the gray levels of pairs of pixels in centered symmetric directions instead of comparing the central pixel to its neighbors. In this way, CSLBP feature takes advantage of the properties of both LBP and gradient based features. For an image of size 32 × 32, after CSLBP pattern of each pixel is computed, a histogram is built to represent the image texture:

f CSLBP = 32 ∑ i=1 32 ∑ j=1 f (CSLBP P,R (i, j), l) , l = 0, 1, 2, 3, • • • , 2 P/2 -1, (4.3) f (x, y) =    1, x = y 0, othersize (4.4)
By construction, the length of the histogram resulting from the CSLBP feature is 2 P/2 . It is obvious that CSLBP produces shorter feature set than LBP. Also, it is a first order local pattern in center symmetric direction and it ignores the central pixel information.

In this work, 8 sampling points and 3 pixels radius around the center pixel are set, thus 16-dimensional CSLBP features are obtained.

2) GIST feature: It is a global image feature, which characterizes several important statistic information about a scene [START_REF] Sikirić | Classifying traffic scenes using the GIST image descriptor[END_REF]. A variety of experimental studies have demon-strated that humans perform rapid categorization of a scene by integrating only coarse global information that can be extracted using " GIST " [START_REF] Song | Biologically inspired feature manifold for scene classification[END_REF]. Using the model proposed by

Oliva [START_REF] Oliva | Modeling the shape of the scene: A holistic representation of the spatial envelope[END_REF], GIST feature is computed by convolving an oriented filter with down-sampled images (32×32) at several different orientations and scales. The scores for the filter convolution at each orientation and scale are stored in an array and resulting in a 512-dimensional feature.

3) Feature combination: After getting CSLBP feature f LBP and GIST feature f GIST from an image, they are combined into a new CSLBP++GIST feature f . The combination consists simply in concatenating (++) the two features:

f = f CSLBP + + f GIST (4.
F = f i + + f i+1 + + f i+2 + +... + + f m-2 + + f m-1 + + f m (4.6)
where, i, i + 1, • • • m are the indexes of the consecutive images composing the sequence, and L length = mi + 1 is the length of the sequence. The total feature dimension is 528×

L length . 528 is the sum of 16 (dimension of CSLBP feature) and 512 (dimension of GIST feature).

Here, each sequence is composed of consecutive images. For that, original image database is simply divided into sequences with the same length. Since training and testing route is traveled with a similar speed, the same sequence length is also used for testing sequence. Thus, each sequence can be represented using sequence feature (F) with the same dimension.

Image sequence matching

To perform sequence matching, similarity between sequence features is evaluated through

Chi-squared distance. The Chi-squared distance is a nonlinear metric which can be cal- The similarity value between the two sequences Q test i and Q train j is measured using the Chi-squared distance D i j , computed as follows:

D i, j = χ 2 (F test i , F train j ) = ∑ k ((F test i ) k -(F train j ) k ) 2 |(F test i ) k + (F train j ) k | (4.7)
Where F test i is the feature vector of the current testing sequence

Q test i ,

Matching validation

In order to reduce false matching cases, the ratio SS i computed from the first minimum distance and second minimum distance is used to validate the matching result (as for single image matching):

SS i = D i,m1 D i,m2
       D i,m1 = min j {D i, j } D i,m2 = min j( j̸ =m1) {D i, j } (4.9)
The values of SS i are between 0 and 1. A threshold is then applied to the score SS i to determine if the sequence pair (i, m1) is matched or not. The matching is considered as positive when the distance ratio SS i is lower than or equal to the threshold T h, otherwise it is considered as negative and the sequence pair is ignored.

Visual localization

After one sequence matching candidate is successfully validated, the vehicle can localize itself through the GPS information attached to the matched training sequence. Effectively, since the training images are tagged with GPS or pose information, the vehicle can get its position through the training images that matched with the current testing sequence.

This is a topological level localization, that is, the system simply identifies the most likely location. Therefore, this is not a very accurate localization, because the training and testing trajectories are not exactly same.

Algorithm of proposed visual localization

D i, j ← ∑ k ((F test i ) k -(F train j ) k ) 2 |(F test i ) k +(F train j ) k | ; //Chi-

Experimental Setup

In this section, the used dataset is described as well as ground-truth and image pre-processing.

Dataset and ground-truth

The dataset used in our work is an open dataset called Nordland1 . It is composed of footage videos of a 728 km long train ride between two cities in north Norway [START_REF] Sünderhauf | Are we there yet? Challenging SeqSLAM on a 3000 km journey across all four seasons[END_REF] (see Fig. 4-4).

The complete 10 hours journey has been recorded in four seasons. Thus, the dataset can be considered as a single 728 km long loop that is traversed four times. As illustrated in Fig. 45, there is an immense variation in the appearance of the landscape, reaching from green vegetation in spring and summer to colored foliage in autumn and complete snow-cover in winter over fresh. In addition to the seasonal changes, different local weather conditions For the experiments described in the following, image frames are extracted from the original videos at 1 fps, there are then 35768 image frames for each season. Each image is then down-sampled these images to 32×32 pixels and converted into gray-level images. 100% precision, which outperforms a little LBP++GIST and significantly the SeqSLAM method. It should be noted that the image size used in SeqSLAM is also 32×32 and the other parameters of SeqSLAM method correspond to default situation as used in [START_REF] Sünderhauf | Are we there yet? Challenging SeqSLAM on a 3000 km journey across all four seasons[END_REF].

Recall (%)

Sequence length selection

Traditionally visual localization has been performed by considering places represented by single images. Recently, several approaches such as SeqSLAM, have proved that recognizing places through sequences of images is more robust and effective [START_REF] Milford | SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights[END_REF]. In this chapter, we also follow the idea of using sequences of images instead of single image for identifying places. This approach allows to achieve better results for visual localization in different seasons, as it can be seen in Fig. 4567. Precision (%) Precision (%)

L length =1 L length =10 L length =50 L length =100 L length =150 L length =200 L length =250 L length =300 (
L length =1 L length =10 L length =50 L length =100 L length =150 L length =200 L length =250 L length =300 (

Visual localization under different season couples

After feature performance evaluation and sequence length selection, visual localization using sequence matching based on multi-feature combination (CSLBP++GIST) was compared under different season couples. The matching results under different season couples are depicted in Fig. 456789. As our objective is to correctly identify the place as much as possible (along the diagonal), it can be seen that the result of "summer vs fall" is the best among the others. It can be also noticed that when the winter sequence is evaluated (Fig. 4-9 (c), (e) and (f)), the number of unrecognized places increase, that is because the snow in winter leads to featureless scenes.

Fig. 4-10 shows precision-recall curves of matching results under different season couples. It can be easily found that visual recognition performance of our method is better under (spring vs summer), (spring vs fall) and (summer vs fall), where we can reach above 85% of recall at 100% precision level. It can be seen also that our proposed multi-feature combination method can achieve recall rate above 60% at 100% precision under all the season couples. The overall performance of CSLBP++GIST is better than that of LBP++GIST.

As expected, when winter sequence is evaluated, the effectiveness of our method decreases due to the extreme changes that this season causes in place appearance because of environmental conditions such as presence of snow, illumination and vegetation changes, etc.

Fig. 4-11 shows an example of frame matches using the proposed method. Despite the large variations in appearance (many vegetations in fall while snow covering the ground in winter), place recognition using the multi-feature attained good matching performance.

For the visual localization based on place recognition, we are primarily interested in the recognition rate high precision level. The recall scores for high selected precision values of SeqSLAM method and our proposed approach are given in proposed place recognition based visual localization algorithm achieves the better recall rate. Moreover, in " spring vs summer " and " spring vs fall " situations, the recall rates of our approach is higher than 85% for all the high precision values recorded in Table 4.1. For both SeqSLAM method and our approach, recall rate increases when precision is decreasing. The recall rate of the two methods increases drastically at 90% precision. Besides that, the recall rate of SeqSLAM method is lower than the recall rate of the proposed method, and worst for all the high precision values. This is probably due to the fact that the SeqSLAM method has a certain dependence of the field of view and the image size, as demonstrated in [START_REF] Sünderhauf | Are we there yet? Challenging SeqSLAM on a 3000 km journey across all four seasons[END_REF].

Fig. 4-12 shows visual localization results of different season couples at 100% precision level. It can be seen that most places can be successfully localized, and at least 60% of the places (red points) can be localized in the worse matching case (spring vs winter).

In addition, the computational time of sequence matching using the combined feature (CSLBP++GIST), is illustrated in Table 4.2. The computational time is increasing when the size of training database is large. Since CSLBP++GIST feature dimension is lower that of LBP++GIST, so the processing time of CSLBP++GIST based matching is shorter.

X(m) ×10 5
Compared with SeqSLAM, the proposed method using CSLBP++GIST feature is faster.

The experiment were conducted on a Intel Core i7, 2.40 GHz laptop.

Conclusion and Future Works

In this chapter, we proposed a feature combination based sequence matching method to perform robust localization even under substantial seasonal changes. After feature extraction, Chi-square distance is used to measure similarity between a testing sequence and the training sequences of a training database. A distance ratio is then calculated before applying a thresholding procedure to validate the good matching candidates.

Thanks to precision-recall based evaluation, experimental results showed that the proposed sequence matching method is more robust and effective for long-term visual localization in challenging environments. The proposed method takes advantages of local and global image information, which can reduce aliasing problem. Sequence length analysis demonstrated that sequences as long as 200 frames could provide viable recognition results. Shorter sequences cannot achieve acceptable results, while longer ones cannot bring significant improvement. Compared to the state of the art SeqSLAM method, the proposed approach provides better recognition performances. In addition, according to the localization results, at least 60% of the places can be localized using the appearance through the proposed method.

However, using feature combination increases feature vector dimension and thus increases time computation. To overcome this limitation, we envision to deal with dimension Chapter 5

All-environment Visual Localization

Based on ConvNet Features and

Localized Sequence Matching

In Chapter 4, the used hand-craft feature CSLBP++GIST and sequence matching technique bring some benefits for visual localization in some simple driving traffic environment. In this chapter, for further study, deep learning features (ConvNet features) instead of handcraft features are used to strengthen place describing ability, and localized sequence matching is developed to improve sequence matching performance. Therefore, the advantages of ConvNet features obtained from convolutional neural networks and localized sequence matching technique are investigated in this chapter. There are particular issues in the domain of environments and appearance changes (such as illumination changes, across seasons, structural changes in the environment, etc.) that affect place recognition accuracy, which need to be addressed to achieve all-environment visual localization. Therefore, there is a need to have robust place recognition systems which have strong place describing and recognizing abilities that can deal with these changes (appearance and illumination invariants). The biggest advantage of a such system would be for all-environment localization across months or years, as there will not be a need to update the map with multiple copies of the same location under different conditions.

Introduction

Currently, deep learning applied in computer vision areas can help to robustly solve the previously mentioned dilemmas associated with all-environment visual localization. Deep learning methods aim at learning feature hierarchies with features from higher levels of the hierarchy formed by the composition of lower level features. Automatically learning features at multiple levels of abstraction allow a system to learn complex functions mapping the input to the output directly from image data, without depending completely on humancrafted features [START_REF] Bengio | Learning deep architectures for AI[END_REF]. Convolutional neural network (ConvNet) as one of promising deep learning which removes complicated and problematic hand-crafted feature engineering, are sensitive to small sub-regions of visual field which are well-suited to exploit the strong spatially local correlation present in natural images [START_REF] Lu | Learning FRAME models using CNN filters for knowledge visualization[END_REF]. ConvNet features extracted from convolutional neural network have been demonstrated to be versatile and transferable that is, even though they have be trained to solve a particular task, they can be used to solve different problems [START_REF] Sünderhauf | On the performance of ConvNet features for place recognition[END_REF].

The supervised deep convolutional neural networks permit to deliver high level performance on most of challenging classification tasks [START_REF] Gomez-Ojeda | Training a convolutional neural network for appearanceinvariant place recognition[END_REF]. Through training on large amounts of labeled data, millions of network parameters can be optimized which makes the deep networks robust and powerful. Once trained, ConvNets obtain discriminative and humaninterpretable feature representations [START_REF] Milford | Sequence searching with deep-learnt depth for condition-and viewpoint-invariant route-based place recognition[END_REF]. Therefore, it is practicable to develop powerful and robust visual localization systems by taking advantage of ConvNet features.

In this chapter, the problem of all-environment visual localization is addressed by de- In our work, we exploit the hierarchical nature of ConvNet features and compare different ConvNet layers for place recognition under severe appearance and illumination variations. Furthermore, a comparison with state-of-the-art place recognition methods is performed on four datasets. The F 1 scores attained with the conv4 layer of ConvNet for the four different datasets are higher than 0.85, which is significant better than those of FAB-MAP and SeqSLAM. At last, for real-time visual localization consideration, a speed-up method is achieved by approximating the cosine distance between features with hamming distance over bit vectors obtained by Locality Sensitive Hashing (LSH). Using 4096 hash bits instead of the original feature permits to accelerate by 12 times the computation time, retaining 95% of original place recognition performance.

The chapter structure is as follows. In Section 5.2, the components of the proposed place recognition based visual localization system are described. Experimental setup is described in Section 5.3, and results are presented in Section 5.4. Finally, the chapter is concluded and future works are discussed in Section 5.5.

Proposed Approach

The proposed visual localization approach can be divided into off-line and on-line parts. In For each possible training sequence candidate, a score S is calculated by summing all the cosine distances along the sequence (Section 5.2.3). The sequence candidate that provides the minimum score can be considered as the most similar one to the testing sequence.

In fact, the two best sequences (according to matching score) are conserved for further validating the final matching result.

Following, the best matching candidate will be validated through a distance ratio SS (see Section 5.2.4). This distance ratio SS of the two minimum computed distances (corresponding to the two best candidates) is considered to validate the training sequence that finally best matches to the current testing sequence. If the ratio SS is below or equal to a threshold T h, the first best sequence candidate (with the lower matching score) is confirmed and regarded as positive, otherwise it is considered as negative one (in this case, no matching result is conserved). When a sequence candidate is confirmed as positive, the position can be obtained from the matched GPS-tagged training images (see Section 5.2.5).

As illustrated in Fig. 5-2, there are four important components in our visual localization approach:

• ConvNet features extraction (detailed in Section 5. Based on this, localized sequence matching will be conducted in the next step.

• Localized sequence matching (detailed in Section 5. Several advantages of our approach can be highlighted:

1) The system uses an off-the-shelf pre-trained convolutional network to extract features which makes feature extraction more conveniently.

2) ConvNet features as auto-learned features are more stable and powerful. By using these robust features as descriptors for place representation, we inherit their robustness against appearance and weather changing.

3) Using a localized sequence matching allows us to search in a small range rather than in the whole training database. This makes place recognition more robust and efficient.

ConvNet features extraction

In this work, caffe-alex [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF] ConvNet pre-trained model (provided by MatConvNet) and

MatConvNet toolbox [START_REF] Vedaldi | Matconvnet: Convolutional neural networks for MATLAB[END_REF] are deployed to extract features. The caffe-alex ConvNet model is a 21 layers network (see Fig. 5-3) which is mainly constructed by five different layer Norm layers can be used to get some kind of inhibition scheme. The fc layers, as fullyconnected layers, compute the class scores.

Let consider an image x and ConvNet learned parameter w. From the input layer (taking the image x) of the network, a sequence of layered outputs is produced. Each layer output is the input of the next layer. Thus, the original input image is transformed layer by layer to the final class scores: Considering that middle layers take the advantage of both low-level and semantic information, our approach exploits feature information of these middle layers to handle large appearance changes and then alleviate false recognition. The used layers and their dimensionality are listed in Table 5.1. The corresponding ConvNet features generated by convolutional Networks from an example of input image are illustrated in Fig. 5-4. It can be seen that conv4, conv5 and relu5 layers provide more image spatial information while pool5, fc6 and fc7 layers bring more semantic information.

f (x) = f 21 (. . . f 2 ( f 1 (x; w 1 ); w 2 ) . . . ), w
d T,i = cos⟨ f test T , f train i ⟩ = f test T • f train i ∥ f test T ∥∥ f train i ∥ ; i = 1, 2, • • • , N train (5.2)
Then, these N train distances are concatenated to form a vector D T ∈ R N train ×1 :

D T = [cos⟨ f test T , f train 1 ⟩, cos⟨ f test T , f train 2 ⟩, • • • , cos⟨ f test T , f train N train ⟩] (5.3)
where N train is the total number of images in training database. D T is the vector that contains the cosine distance between the testing image I test T and all the training images. It is represented as a column in a matrix as shown in Fig. 5-5. where k(t) is the index of the column vector D t by which the path (line) passes through:

Localized sequence matching

k(t) = s +V (t -(T -d s + 1)) (5.6)
where s is the training image index from which the path is originated. The initial value of s is 0, then increased by 1 at each step. V is the vehicle speed varying between V min and V max with a step value V step . The score S (sum of distance values along path (line)) is used to identify the best matching candidate (who has the lowest score) for each testing sequence.

Final matching validation

Given the current testing image number T , the corresponding testing sequence T (images indexed from Td s + 1 to T ) can be constructed. Using the localized sequence matching method, the best two sequence candidates who have smaller scores are conserved for further validation. Suppose S m1 and S m2 are respectively the first and second minimum score of the top two training sequence candidates to the testing sequence, where

       S T,m1 = min j {S T, j } S T,m2 = min j( j̸ =m1)
{S T, j }

(5.7)

In order to verify the best sequence matching, a ratio SS T is calculated as follows:

SS T = S T,m1 S T,m2 (5.8)
The value of ratio SS is between 0 and 1. A threshold T h is then applied to the ratio SS T to determine if the sequence pair (T, m1) is matched or not. If the ratio SS T is not larger than the threshold T h, which means the training sequence corresponding to m1 is matched to the current testing sequence, this is also called positive matching. Otherwise no matching is considered (negative matching).

Visual localization

After a matching result is successfully validated, the vehicle can localize itself through the matched training image position. Since the training images are tagged with the GPS information, the vehicle can get its position information through the training image matched with the current testing image. As for the approaches presented in the two previous chapters, this is also a topological level localization-simply identifies the most likely location.

Algorithm of proposed visual localization

Datasets and ground-truth

Four datasets with different characteristics (as described in Table 5.2) will be used to evaluate our method.

Experiments and Results

Performance comparison between single image and sequence based approach

Traditionally, visual localization has been performed by considering places as single images. However, other more recent proposals, such as SeqSLAM, changed this concept and introduced the idea of recognizing places as sequences of images.

In this section, the place recognition performances based on sequences of images and single images are compared. In Fig. 5-10, results obtained for UTBM-1 and Nordland datasets are presented. Attending to the precision-recall curves depicted in Fig. 5

ConvNet features layer-by-layer study

This section provides a thorough investigation of the utility of different layers in the Con-vNet hierarchy for place recognition and evaluates their individual robustness against the two main challenges in visual place recognition: appearance and illumination changes. (2) City Center dataset: This dataset was collected along public roads near the oxford city center with many dynamic objects such as traffic and pedestrians. In addition, it was collected on a windy day with bright sunshine, which makes the abundant foliage and shadow features unstable. The precision-recall curves are shown in Fig. 5-13. Except that the recall at 100% precision of the layer fc7 is less than 70%, the performance of the other layers (conv4, conv5, relu5, pool5 and fc6) reaches above than 75% recall at totally correct level. The conv4 layer is the best one achieving the highest recall level (above 80%).

Appearance change robustness

City Center dataset: twice traveling be seen that in summer vs fall case, the performances obtained from the six layers (conv4, conv5, relu5, pool5, fc6 and fc7) are excellent (around 80% recall at 100% precision level).

For the other cases, conv4, conv5, relu5 and pool5 are more robust considering appearance changes than the higher layers fc6 and fc7.

UTBM-2 dataset: morning vs afternoon

Illumination change robustness

Illumination is another important factor for visual recognition. We investigate the ConvNet features performance on UTBM-2 dataset that considers morning versus afternoon situa-

LSH based visual recognition

In contrast to typical computer vision benchmarks where the recognition accuracy is the most important performance metric [START_REF] Milford | Sequence searching with deep-learnt depth for condition-and viewpoint-invariant route-based place recognition[END_REF], visual localization for vehicles or robots always needs agile algorithms for real-time application. As introduced in Chapter 3, Locality Sensitive Hashing (LSH) is arguably the most popular feature compression method and has been applied to many problems, including information retrieval and computer vision [START_REF] Xia | Sparse projections for highdimensional binary codes[END_REF].

Therefore, in order to speed up image matching significantly, LSH method that preserves the euclidean similarity [START_REF] Moses S Charikar | Similarity estimation techniques from rounding algorithms[END_REF] is considered again for faster visual recognition.

According to the study results of Section 5.4.2, conv4 has shown its strong ability in place recognition. However, computing the cosine distance between many 64,896 dimensional conv4 feature is an expensive operation. For real-time place recognition, Locality-Sensitive Hashing (LSH) method is used, mapping the conv4 feature f conv4 to a lowdimensional binary vector:

H(K) = sign(w ⊤ f conv4 + b) (5.10)
where w is a K dimension data-independent random matrix, which is satisfy a standard

Gaussian distribution [START_REF] Datar | Localitysensitive hashing scheme based on p-stable distributions[END_REF]. And b is a random intercept. In our experiment, conv4 feature f conv4 is normalized with zero mean, then approximately balanced partition is obtained with b = 0. Thus, high dimensional feature is converted into a low K dimension binary bits. The binary bit vectors can then be compared using hamming distance more efficiently.

We implement this method and compare the place recognition performance achieved with the hashed conv4 feature vectors of different lengths (2 7 . . .2 12 bits) on the four datasets (Shown in Fig. 5678910111213141516). Hashing the original 64,896 dimensional vectors into 4096 bits corresponds to a data compression of 63.1%. In addition, the 4096 hash bits representation retains approximately 95% of the original place recognition performance. It can be seen from Fig. 5-16 that, when the length of hash bits is decreasing, the place recognition performance is also descending. As shown in Table 5.4, the average time per matching using 4096 hash bits is 0.0291s which corresponds almost to a speed-up factor of 12 compared to using the cosine distance over the original conv4 feature requiring 0.3259s per matching. There is no doubt that for larger scale datasets, the speed up advantages can be more significant. Figure 5-17: Visual localization results obtained by our system in the four datasets. The feature used here is 4096 hash bits of conv4 layer. Two images coming from the same location (on the basis of appearance alone) are marked with red point and joined with a blue line.

Visual localization results

X(m)

For the visual localization based on place recognition, the recognition rate at high pre-125 cision level is a key indicator to reflect whether the system is enough robust to determine position under changing environment. A correct place recognition means a successful visual localization while an incorrect place recognition could cause huge localization error.

Therefore, the higher the recognition rate at 100% precision is, the more robust visual localization system is. Red "o" is localization error for the 100% precision level. Green "+" is the localization error for 99% precision level.

Through changing the threshold value (T h), different recognition precision levels can be recorded. At 100% precision level, all the recognized places are true positives, the localiza-tion error is small. For achieving 100% recognition precision level, the threshold value is set to 0.75 and 0.51 for UTBM-1 and UTBM-2 datasets respectively. While for City Center and Nordland (spring vs winter) datasets, the threshold is set to 0.95 and 0.92 respectively.

By increasing the threshold value, the precision is decreasing. When the threshold is set to 1, the precision level is lowest and there are many false matching for place recognition. Some false matching will lead to huge localization error, because the places can be wrongly matched to any place in the whole trajectory. In general, smaller the threshold is, fewer false recognition cases occur.

In Table 5.5, the average localization errors and recall ratio at different precision levels are given. Using 4096 hash bits, at 100% precision, the proposed approach achieves above 75% recall on the City Center dataset, and above 72.88% on the more challenging Nordland dataset (Spring vs winter). While on the UTBM-1 and UTBM-2 datasets, the recall are 32.88% and 11.54% respectively. For all these datasets, the average visual localization errors at 100% recognition precision are below than 4m. It also can be noted that, the average visual localization error is increasing with the recognition precision decreasing.

Conclusion and Future Works

Along this chapter a visual vehicle localization approach based on ConvNet features and localized sequence matching is presented. The approach takes the strengths of deep conventional network and localized sequence matching, which make place recognition fast and accurate. The proposed visual localization approach allows the vehicle to localize itself in the most places in changing environments.

We conduct experiments on four typical datasets that consider big challenges in visual place recognition: appearance, viewpoint and illumination changes. The experimental results show that the use of ConvNet conv4 feature can obtain F 1 score above 0.85 in these four datasets, which outperforms the methods of D-CSLBP++HOG (approach in Chapter 3) and CSLBP++GIST (approach in Chapter 4), FAB-MAP and SeqSLAM. In addition, for satisfying real-time constraints, speed-up approach based on LSH method was used to compress the high dimension ConvNet feature. By using the 4096 hashing bits representation instead of the original conv4 feature, the average time per matching is almost 12 times faster.

Although ConvNet features can improve visual localization, using pre-trained network still can be improved for visual localization because the original network is trained for object classification rather than place recognition. In future work, how to train visual localization based networks and features optimizing method specifically for life-long visual localization under changing conditions will be investigated.

Chapter 6

Conclusion and Future Works Furthermore, a comprehensive performance comparison of different ConvNet layers (each defining a level of features) is conducted on four real world datasets. The F 1 scores attained with the conv4 of ConvNet feature on the different datasets are higher than 0.85, which are significant better than those of FAB-MAP and SeqSLAM in presence of appearance and illumination changes. To speed up the computational efficiency, locality sensitive hashing method is applied to achieve real-time visual localization with minimal accuracy degradation.

Future Works

In the author's point of view, there are many ways in which the research presented in this thesis could be developed in the future.

• More metric information can be used for visual localization in further research. In this thesis, metric information was not used just because we concentrate on feature comparison and image matching under changing environment. In fact, visual localization performance could be improved considerably by integrating real vehicle motion model. Operating on relative geometric relations between poses along the trajectory during data acquisition, metric map representations (feature maps or occupancy grid maps) of the environment can be geometric and clearly link with the real world. The localization can be done continuously and highly accurate locally.

• In terms of visual place recognition, how to describe a place appropriately is the key point. Therefore, some powerful and robust image descriptors can be developed for place recognition. Most of visual place description techniques can be classified into two broad categories: local feature descriptors and global or whole-image descriptors. Global descriptors are normally pose dependent and very fast to compute but less robustness to occlusion and perceptual aliasing effect. While local features hold a strong discriminative power but with high cost of computation time and complex match processing. How to describe a place with "locally " and "globally" information and take the advantage of the these information can be further studied.

• Three-Dimensional (3D) Information can be considered in visual localization. Unlike the 2D geometry of interest points in an image, 3D geometry information in space is an invariant property of a location. It should be possible to integrate this relative 3D distance information to reduce false place recognition between some similar locations.

• Research in place recognition can also benefit from the ongoing research in object detection and scene classification. Based on object detection, moving objects such as pedestrians or cars can be ignored while other landmark objects such as buildings can be used for long-term place recognition. Semantic scene context can furthermore limit the search space for place recognition to ensure scalability towards long-term autonomy. Semantic context can support learning and predicting the changes in a scene and help to increase robustness against environmental condition changes. Exploiting knowledge about which objects are dynamic or static and object semantic information can increase the robustness to appearance changing. Therefore, using object detection and scene classification for the task of place recognition is a worthwhile direction for future research.

Figure 1 - 1 :

 11 Figure 1-1: Example of participants in 2007 Grand Challenge

Figure 1 - 3 :

 13 Figure 1-3: Synopsis of place recognition based visual localization system.

Figure 1 - 4 :

 14 Figure 1-4: Experimental vehicle equipped with sensors (especially camera and RTK-GPS).

Chapter 2 Related

 2 Works on Place Recognition Based Visual Localization This chapter presents an overview of relevant works in the field of place recognition based visual localization. It begins by defining the core aspects of place recognition based visual localization, namely: place describing, place remembering and place recognizing. Then, relevant works about place describing and place remembering are summarized respectively.

Fig. 2 -Figure 2 - 1 :

 221 Fig.2-1 presents a general scheme of a place recognition based visual localization sys-

Figure 2 - 2 :

 22 Figure 2-2: SIFT extracts interest points in an image for description. The circles are interest points selected by SIFT within the image. The number of possible features may vary depending on the number of interest points detected in the image.

  an illumination robust feature (DIRD) is proposed based on normalized filter responses of small images regions. The experiments showed that DIRD achieved good performance for loop closure detection. Arroyo et al. [7] divide each panorama into sub-panoramas and extract LDB (Local Difference Binary) binary descriptor for each subpanorama. The final image feature is created by concatenating the different LDB descriptors. The proposed LDB feature also achieved good performance in life-long visual localization.

Figure 2 - 4 :

 24 Figure 2-4: Place recognition system utilizing convolutional network features as robust landmark descriptors to recognize places despite severe viewpoint and condition changes, without requiring any environment-specific training. The colored boxes in the images above show ConvNet landmarks that have been correctly matched between two significantly different viewpoints of a scene. This enabling place recognition under challenging conditions (from [102]).

Figure 2 - 5 :

 25 Figure 2-5: Example of place recognition based on single image. Each testing image has to retrieve its similar one from the training database.

Figure 2 - 6 :

 26 Figure 2-6: An example of place recognition based on image sequence. Sequences A and A' are taken in a time interval of two weeks.

2 . 5 ,( 2 )

 252 feature combination and three-dimensional information can improve the performance of place recognition. Inspired by this, a new multifeature (D-CSLBP++HOG) based visual localization will be proposed in Chapter 3. D-CSLBP++HOG combines HOG and CSLBP features that are built from both gray-scale image and disparity map. By taking advantage of texture, shape and depth information, the proposed multi-feature is more robust for place describing, thus it can improve performance of place recognition. Considering single image matching for place recognition is fragile in drastic changing environment (i.e. different seasons), the second task is to develop sequence matching based visual localization system without being affected by seasons changing. In Chapter 4, visual localization across seasons is then proposed based on sequence matching and feature combination. Here, global feature GIST and local binary feature CSLBP are combined together for place describing. The proposed method is tested and evaluated in four seasons outdoor environments.(3) Motivated by the success of deep learning features in visual recognition, a visual localization technique based on ConvNet networks and localized sequence matching is proposed in Chapter 5. Compared with the traditional approaches based on hand-craft features and single image matching, the proposed method can achieve good performances even in presence of appearance and illumination changes.

Chapter 3 Visual

 3 Localization by Place RecognitionBased on Multi-feature (D-λ LBP++HOG)In this chapter, a multi-feature based method for vehicle visual localization in urban environments is proposed. The considered multi-feature combines HOG descriptor and Dλ LBP descriptor (λ LBP which is extracted from both gray-scale image and disparity map).

Figure 3 - 1 :

 31 Figure 3-1: Multi-feature built from gray-scale image and disparity map. Features are firstly extracted from each image block and then concatenated together. The symbol "++" means concatenation.

Figure 3 - 2 :

 32 Figure 3-2: Illustration of the basic LBP operator

Figure 3 - 3 :

 33 Figure 3-3: Examples of (P,R) neighborhood used to compute LBP: (8,1), (16,2) and (8,2) .

Figure 3 - 4 :

 34 Figure 3-4: Example of HOG feature.

  set of GPS tagged training image pairs (left and right images) I train = {I train j } N train j=1 are firstly acquired, where N train is the number of training image pairs. After image preprocessing (see Section 3.3.1), multi-feature set V train = {v train j } N train j=1 is extracted from the training database (see Sections 3.3.2 and 3.3.3) , where v train j is the multi-feature extracted from the training image pair I train j . In on-line phase, multi-feature v test i is extracted from current image pair I test

Figure 3 - 5 :

 35 Figure 3-5: The process of the proposed place recognition based visual localization.

1 . 2 . 6 .

 126 Image preprocessing: This step consist of down-sampling and contrast-limited adaptive histogram equalization (detailed in Section 3.3.1). Block based feature extraction: λ LBP feature is extracted from grayscale image and disparity map; HOG feature is extracted from grayscale image (detailed in Section 3.3.2). 3. Multi-feature concatenation: The final multi-feature D-λ LBP++HOG is obtained by concatenating λ LBP and HOG feature. (detailed in Section 3.3.3) 4. Feature comparison and image matching: Based on the extracted multi-feature descriptors, image matching is conducted through multi-feature comparison using euclidean distance (detailed in Section 3.3.4). 5. Final Matching validation: According to the distance ratio of the top two best candidates, image matching result is validated (detailed in Section 3.3.5). Visual localization: The vehicle current position can be obtained through the matched GPS-tagged training image (detailed in Section 3.3.6).

Figure 3 - 6 :

 36 Figure 3-6: Image preprocessing using contrast-limited adaptive histogram equalization(CLAHE). The left image is processed using CLAHE and the prepossessing result is the right image.

3. 3 . 2

 32 Fig.3-7. Block based approach (that relies on image blocks) can address spatial properties of images. It can be used for any histogram descriptors.

Figure 3 - 7 :

 37 Figure 3-7: An example of block based local binary descriptor extraction. Features are extracted from each image block firstly and then concatenated together. Here, image blocks are non-overlapped and do not need any image segmentation.

Figure 3 - 8 :

 38 Figure 3-8: Block based feature extraction procedure (applied to each images pair of the training database for the off-line phase and to current images pair for the testing phase)

  j } N train j=1 of all training image pairs {I train j } N train j=1 is obtained. For a current testing image pair I test i , a multi-feature v test i is also obtained. Then the image matching is conducted based on the euclidean distance comparison between the multi-feature v test i of the current testing image and all training image multi-features v train j ( j = [1, 2, • • • , N train ]) from the training images dataset.

  -9 (a): the experimental vehicle traversed about 4 km in a typical outdoor environment. Three typical areas were traversed: urban city road (area A), lots of factories building (area B) and a nature scene surrounding a lake (area C). The training and testing data were collected at different times respectively in 2014/9/11 and 2014/9/5. The training database is composed of 849 images while the testing database is composed of 819 images. The average distance between two successive frames was around 3.5 m. To tag the training images, GPS position of each image is obtained by a RTK-GPS receiver.

Figure 3 - 9 :

 39 Figure 3-9: Vehicle paths for the UTBM-1 and UTBM-2 datasets. Source: Google Maps

Figure 3 - 10 :

 310 Figure 3-10: Example of gray-scale image and its corresponding local binary images (LBP, CLBP, CSLBP, CSLDP and XCSLBP) and HOG feature.

Precision-recall characteristics

  and F 1 score are widely used to show the effectiveness of image retrieval method. Therefore, our evaluation methodology is based on precision-recall curves and F 1 score. In our experiments, the training image number is larger than or equal to the testing image number, thus each testing image has a ground-truth matching. Therefore, among the positives, there are true positives (correct results among successfully validated images matching candidates) and false positives (wrong results among successfully validated images matching candidates). The sum of the true positives and false positives is the total retrieved images number. More specifically, precision is the ratio of true positives over the retrieved images number (number of all the successfully validated image matching candidates), and recall is the ratio of true positives over the total testing images: Precision = Number of true positives Number of retrieved images × 100% Recall = Number of true positives Number of total testing images × 100%

Fig. 3 -

 3 Fig.3-11 depicts the precision-recall curves obtained by the different binary features in two typical datasets UTBM-1 and KITTI 06. It can be seen that, the performance of D-CSLBP is better than the performance with the features D-LBP, D-CLBP, D-CSLDP and D-XCSLBP. Also, it can be seen that the maximum recall at 100% precision for D-CSLBP is higher than the one of the other features.

Figure 3 - 11 :

 311 Figure 3-11: Image retrieval performance (precision-recall curve) comparison considering different block based binary features on UTBM-1 and KITTI 06 datasets. Here the image block size is 32.

Figure 3 - 12 :

 312 Figure 3-12: Image retrieval performance (precision-recall curve) comparison considering D-CSLBP feature extracted with different image block sizes, on four datasets.

Figure 3 - 13 :Fig. 3 -

 3133 Figure 3-13: Image retrieval performance (precision-recall curve) comparison of HOG, D-CSLBP, D-CSLBP++HOG based approaches, on four datasets.

Figure 3 - 14 :

 314 Figure 3-14: Example of distance matrices for UTBM-1 dataset. Here, the distance matrix D is normalized into 0-1 range. The distances of same or similar images are close to 0 (red color), while for the larger distances, the corresponding color is close to yellow. In (a), two images from a same place are taken at different times (difference of two weeks). From figure (b) to figure (f), the distance matrix D is plotted. The distance matrices show that multi-feature combination (c) reduces the noise appeared around the diagonal (groundtruth line). Besides, compared with (d), after adding disparity information in (e), perceptual aliasing decreases, as confirmed by the precision-recall curves in Fig.3-13 (a).

Fig. 3 -Figure 3 -

 33 Fig.3-17 shows the place recognition based visual localization errors at different precision levels. When adjusting the threshold value T h, the recognition precision is also changing. At 100% precision level, each recognized place is true positive and its localization error is small (depending on the ground-truth criteria, in our case it is 5m). For achieving the 100% recognition precision level, threshold value is set to 0.88 and 0.58 for

  (a) Example of two images and their corresponding LiDAR data. (b) LiDAR based localization results. It can be seen that using LiDAR only for long-term localization, will lead to accumulated errror.

Figure 3 - 18 :

 318 Figure 3-18: An example of LiDAR localization results.

  Li-DAR or Inertial Measurement Unit (IMU)). Fig.3-18 shows the example of LiDAR based localization in case of unrecognized positions with our place recognition based method.

Fig. 4

 4 -1 illustrates the general diagram of our approach. Based on features extracted from images, sequences are efficiently matched using Chi-square distance and the best candidate to place matching is recognized according to coherent sequence matching. Thus, visual localization is realized through the recognized places. Image feature used in this chapter is a combination of CSLBP and GIST, which should improve image distinguishing ability by capturing local and global image information. We will demonstrate the algorithm performance using multi-season videos of 30000 km long train ride in the northern Norway. For this, an extensive experimental study is conducted according to sequence matching length, as well as a comparison of the proposed approach with the state of the art SeqSLAM[START_REF] Milford | SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights[END_REF] method.

Figure 4 - 1 :

 41 Figure 4-1: General diagram of visual localization system using sequence matching.

1 andFig. 4 - 2 ,Figure 4 - 2 :

 14242 Fig.4-2, there are five main components in our approach: image preprocessing, feature extraction, sequence matching, matching validation and the last is visual localization based on the matching result. To detail, a set of GPS tagged training images is firstly acquired. After image preprocessing (Section 4.2.1), CSLBP and GIST features are extracted independently from the images of the training database and then concatenated together to form multi-feature CSLBP++GIST (Section 4.2.2). Then, multi-feature CSLBP++GIST obtained from images of a sequence are concatenated (++) to form the final sequence feature (F) representing the sequence. Here, sequence consists of consecutive images and each sequence is independent. A current place (represented by a testing sequence) is then recognized through

5 )Fig. 4 -

 54 Fig.4-3 illustrates an example of extracted features. It can be seen that CSLBP and LBP features pay more attention to the image detail (local information), and CSLBP represents better the image than LBP. While GIST feature pays more attention to the whole scene. Therefore, combination of the local feature CSLBP and global feature GIST will describe the place better and should improve place recognition performance. 4) Sequence feature: Finally, the CSLBP++GIST features extracted from images of a sequence are concatenated (++) to form the final sequence feature (F) representing the sequence of images:

Figure 4 - 3 :

 43 Figure 4-3: Example of extracted features. The first row shows the original images. The second and third rows show the images of LBP and CSLBP features respectively. The fourth row gives the images of GIST features.

  culated easily. Suppose the numbers of training and testing images are N train and N test respectively, and sequence length is L length . Therefore, the training sequence number is M = N train /L length and the testing sequence number is N = N test /L length . Given a testing sequence Q test i (composed of the L length last consecutive testing images), it will be compared with each training sequence Q train j ( j = 1, 2, • • • , M) from the training database. Since the sequence lengths are same, therefore the sequence matching can be conveniently conducted using the Chi-squared distance.

  training dataset). k is index of the components of feature vector: k = 1, 2, • • • , 528 × L length . Then, all the computed D i, j form a distance matrix D. For sequence matching, feature vector of the current sequence Q test i is compared with feature vector of each training sequence Q train j ( j = 1, 2, • • • , M). Based on the distances D i, j ( j = 1, 2, • • • , M), the two best training sequence candidates (which have the first minimum distance and second minimum distance) that best match the current testing sequence are conserved.

(4. 8 )

 8 where D i,m1 and D i,m2 are respectively the first and second minimum distances between the feature vector of current testing sequence Q test i and the feature vectors of all the training sequences Q train j ( j = 1, 2, • • • , M), as follows:

Figure 4 - 4 :

 44 Figure 4-4: Nordland route. Source: Google map. The trajectory is recorded four times, once in every season. Video sequences are synchronized and the camera position and field of view are always the same. GPS readings are available.

Figure 4 - 5 :

 45 Figure 4-5: A typical four seasons images representing the same scene in spring, summer, fall and winter. It can be seen that huge differences appear in the images with season changing.

Figure 4 - 6 :

 46 Figure 4-6: Performance of the proposed method according to different used features and comparison with SeqSLAM method (summer vs winter, sequence length L length =200).

Fig. 4 -

 4 Fig.4-7shows the performance achieved when varying sequence length between 1 and 300 frames for two different feature combination: LBP++GIST and CSLBP++GIST. Sig-

Figure 4 - 7 :Figure 4 - 8 :Figure 4 - 9 :

 474849 Figure 4-7: Performance comparison of our proposed method with different feature combination, according to image sequence length L length (spring vs winter).

Fig. 4 -

 4 Fig.4-8 illustrates the ground-truth of image matching (for every possible couple of seasons). It should be noted that the position of the train in an arbitrary frame in one season corresponds to the same frame in any of the other three seasons thanks to timesynchronization.

Figure 4 - 10 :

 410 Figure 4-10: Precision-recall curves comparing the performance of different feature combination under different season couples.

Figure 4 - 11 :

 411 Figure 4-11: Corresponding frames from sequence matching between two seasons (fall and winter). The left column shows fall image frames queried from winter traversal, and the right column shows the winter image frames recalled by our approach.

Figure 4 - 12 :

 412 Figure 4-12: Visual localization results under different season couples at 100% precision level. Successful matched images that come from the same location (on the basis of appearance alone) are marked in red points.

  As illustrated in the previous chapters, place recognition based visual localization can be achieved by sophisticated hand-crafted features. However, as robots or vehicles operate for longer periods of time in real-world environments, the huge variations on the visual perception of a place caused by factors such as different days, varying weather conditions and seasonal changes, remain a significant challenge for place recognition based visual localization. Though many advances have been made in the recent past[START_REF] Milford | SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights[END_REF][START_REF] Upcroft | Lighting invariant urban street classification[END_REF], improv-ing place recognition accuracy and reliability for visual localization, still remains an open problem.

Figure 5 - 1 :

 51 Figure 5-1: Schematic illustration of visual localization using ConvNet features and localized sequence matching. ConvNet features are extracted from testing images and then compared to those extracted from all images of the training database. After feature comparison, localized sequence matching is conducted to find the best image matching.

  the off-line part, a set of GPS tagged training images I train = {I train i } N train i=1 is firstly acquired, where N train is the number of training images. Then, the pre-trained caffe-alex network (trained using ILSVRC2012 dataset) is used to extract features from training images [54]. The extracted ConvNet features from training database are noted F train ={ f train i } N train i=1 , where f train i is the feature extracted from the training image I train i . For the on-line phase, the current testing image I test T is input into caffe-alex network and the ConvNet feature f test T of the current testing image is computed. Then, f test T is compared with the training image feature set { f train i } N train i=1 using cosine distance (Section 5.2.2).

Figure 5 - 2 :

 52 Figure 5-2: Detailed block diagram of the proposed visual localization method. Feature extraction uses pre-trained network, feature comparison uses cosine distance and localized sequence searching is based on potential paths.

2 . 1 )•

 21 : ConvNet features F train are extracted from all training database images by off-line processing and f test T is extracted from current testing image by on-line processing, using the pre-trained caffe-alex network. These learned features are robust to both appearance and illumination changes and allow representing each location (place) very well. The extracted ConvNet features will be compared in the next step. Feature comparison (detailed in Section 5.2.2): The cosine distances are computed between the feature f test T of the current testing image and the features { f train i } N train i=1 of all the images of the training database. All these distances formed a vector D T .

  2.3): To achieve efficient place recognition, localized sequence matching is used instead of single image match-ing. Considering the testing sequence composed of the last d s testing images (indexed from Td s + 1 to T ), sequence matching is conducted in the matrix M T = [D T -d s +1 , D T -d s +2 , • • • , D T ]. According to the ratio between the testing and training trajectory speeds, some possible training sequence candidates in the training database can be firstly determined. A score S is calculated by summing all the testing image to training image cosine distances along each possible sequence candidate. The sequence that provides the minimum score can be considered as the most similar one to the testing sequence. The two best sequence matching scores are conserved for further matching validation.

Figure 5 - 3 :

 53 Figure 5-3: Architecture of the caffe-alex network. ConvNet transforms the original image, layer by layer, from the original pixel values to the final class scores.

f 1 ,

 1 • • • , f 21 are the corresponding layer functions as illustrated in Fig.5-3. Each layer output is a deep learnt representation of the image (ConvNet feature). The low layers retain high spatial resolution with low-level visual information. While high layers capture more semantic information and less fine-grained spatial details. The network is able to process images of any size equal to or greater than 227×227 pixels (the original caffe-alex net-work was trained on 227×227 images). Place recognition is then performed by comparing the ConvNet features extracted from current testing image I test T with the ConvNet features extracted from all the images {I train T } N train i=1 of the training database.

Figure 5 - 4 :

 54 Figure 5-4: An example of a scene and extracted features from different layers of the caffealex network. Features obtained from different ConvNet layers can serve as holistic image descriptors for place describing.

  Assume that the vehicle travels in repeated route with negligible relative acceleration. For a given testing sequence, composed of d s images, indexed from Td s + 1 to T , where T is the index of the current testing image, we search the corresponding sequence (from the training database) in a local matrix rather than in the whole training database. This searching procedure is performed by considering possible training sequence candidates, that are determined by the speed ratio between the training and testing trajectories. This

Figure 5 - 5 :

 55 Figure 5-5: Procedure for ConvNet features comparison. Features are extracted from each testing image and then compared with features extracted from all training images.

Figure 5 - 6 :

 56 Figure 5-6: The search algorithm finds the lowest-cost straight line within the searching matrix M T . These lines are the set of potential paths through the matrix. The red line is the lowest-cost path which aligns the testing sequence and training sequence. Each element indicates the cosine distance between two images (test and train).

1) UTBM- 1 datasetFigure 5 - 7 :

 157 Figure 5-7: UTBM-1 dataset: Example of training and testing images (interval time of one week). Left column is training images and right column is testing images.

Figure 5 - 8 :Figure 5 - 9 :

 5859 Figure 5-8: UTBM-2 dataset: morning vs afternoon. Left column is training images and right column is testing images.

Figure 5 - 11 :

 511 Figure 5-11: Examples of frame matches from the Nordland dataset (fall vs winter). The top row shows testing frames, and the middle and third rows show the training frames recalled by d s = 1 (single image) and d s = 6, respectively. Visual recognition based on sequence matching achieves better performance than those obtained using single image.

Fig. 5 -

 5 Fig.5-11 shows examples of frame matches on Nordland dataset (fall vs winter). Despite the large appearance variations between different seasons, the proposed ConvNet based visual localization using sequence matching (d s =6) attained better recognition results than those obtained using single image (d s =1).

( 1 )Figure 5 - 12 :

 1512 Fig.5-12 (bottom). The recall obtained for the conv4 layer at totally correct level is around

Figure 5 - 13 :Figure 5 - 14 :

 513514 Figure 5-13: Precision-recall curves for City Center dataset (the trajectory acrosses city and parking areas)(d s = 3).

Figure 5 - 15 :

 515 Figure 5-15: Precision-recall curves for UTBM-2 dataset considering different layers (d s = 8).

Fig. 5 -Figure 5 - 18 :

 5518 Figure 5-18: Localization error at different recognition precision levels. Dark line is the localization error for all recognition results (including the true positives and false positives).Red "o" is localization error for the 100% precision level. Green "+" is the localization error for 99% precision level.

6. 1 Conclusion

 1 This thesis focuses on improving place recognition for visual localization in changing outdoor environment. For many applications, it is crucial that a robot localizes itself within the world for autonomous navigation and driving. After a review of some state-of-art place recognition and visual localization technologies in Chapter 2, new methods to perform visual localization in changing environments were proposed in this thesis.Firstly, multi-feature combination for vehicle localization is explored in Chapter 3.Since different types of features have their own advantages, combining some powerful features will be helpful in place recognition. We firstly integrate disparity information into complete center-symmetric local binary patterns (CSLBP) to obtain a robust global image description (D-CSLBP). Furthermore, D-CSLBP and HOG features are combined together to strengthen the place describing ability by taking the advantage of depth, texture and shape information. The multi-feature (D-CSLBP++HOG) improves visual recognition performance, it thus allows decreasing the effect of some typical problems in place recognition such as perceptual aliasing. In addition, for real-time visual localization, local sensitive hashing method (LSH) is used to speed up the process of image matching. Our approach allows more effective visual localization compared with the state-of-the-art FAB-MAP (Fast Appearance Based Mapping) method. Secondly, visual localization across seasons based on sequence matching and feature combination of GIST and CSLBP is developed in Chapter 4. Matching places by considering sequences instead of single images denotes high robustness to extreme perceptual changes. The proposed method is tested and evaluated in four seasons outdoor environments. Studies of the relationship between image sequence length and sequences matching based place recognition performance is conducted. The achieved results have shown improved precision-recall performance and the proposed approach outperformed the state-ofthe-art SeqSLAM (Sequence Simultaneous Localization and Mapping) algorithm. Thirdly, all-environment visual localization system based on ConvNet features and localized sequence matching is proposed in Chapter 5. We use the automatic learned Convolutional Network (ConvNet) features and localized matching technique to accomplish all-environment visual localization task under appearance and illumination changing situations. The pre-trained networks provided by MatConvNet are used to extract ConvNet features and then a localized sequence search technique is applied for visual localization.

2 - 5 3 - 2 3 - 3 3 - 7 4 - 1 4 - 2 4 - 4 4 - 7

 2532333741424447 tations tool for various category-level recognition tasks such as scene recognition, object detection and classification. However, direct use of ConvNet representation trained for object classification as "black-box" feature extractor can not achieve significant improvements in performance on instance-level recognition tasks. Another problem is that, ConvNet parameters learning is complex and hard, which restrict the real-time place recognizing for visual localization. Therefore, some flexible and easy-training deep learning networks should be developed for place recognition based visual localization. 2-4 Place recognition system utilizing convolutional network features as robust landmark descriptors to recognize places despite severe viewpoint and condition changes, without requiring any environment-specific training. The colored boxes in the images above show ConvNet landmarks that have been correctly matched between two significantly different viewpoints of a scene. This enabling place recognition under challenging conditions (from [102]). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example of place recognition based on single image. Each testing image has to retrieve its similar one from the training database. . . . . . . . . . . 2-6 An example of place recognition based on image sequence. Sequences A and A' are taken in a time interval of two weeks. . . . . . . . . . . . . . . . 3-1 Multi-feature built from gray-scale image and disparity map. Features are firstly extracted from each image block and then concatenated together. The symbol "++" means concatenation. . . . . . . . . . . . . . . . . . . . Illustration of the basic LBP operator . . . . . . . . . . . . . . . . . . . . . Examples of (P,R) neighborhood used to compute LBP: (8,1), (16,2) and (8,2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4 Example of HOG feature. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5 The process of the proposed place recognition based visual localization. . . 3-6 Image preprocessing using contrast-limited adaptive histogram equalization(CLAHE). The left image is processed using CLAHE and the prepossessing result is the right image. . . . . . . . . . . . . . . . . . . . . . . . An example of block based local binary descriptor extraction. Features are extracted from each image block firstly and then concatenated together. Here, image blocks are non-overlapped and do not need any image segmentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8 Block based feature extraction procedure (applied to each images pair of the training database for the off-line phase and to current images pair for the testing phase) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-17 Place recognition based localization errors at different precision levels. The dark line is the localization error for all recognition results (including the true positives and false positives). Red "o" is localization error at 100% precision level. Green "+" is localization error at 99% precision level. Blue "." is localization error at 90% precision level. . . . . . . . . . . . . . . . 3-18 An example of LiDAR localization results. . . . . . . . . . . . . . . . . . General diagram of visual localization system using sequence matching. . . Flow chart of proposed visual localization using sequence matching. . . . . 4-3 Example of extracted features. The first row shows the original images. The second and third row show the images of LBP and CSLBP features respectively. The fourth row gives the images of GIST features. . . . . . . . Nordland route. Source: Google map. The trajectory is recorded four times, once in every season. Video sequences are synchronized and the camera position and field of view are always the same. GPS readings are available. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5 A typical four seasons images representing the same scene in spring, summer, fall and winter. It can be seen that show huge differences appear in the images with season changing. . . . . . . . . . . . . . . . . . . . . . . 4-6 Performance of the proposed method according to different used features and comparison with SeqSLAM method (summer vs winter, sequence length L length =200). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Performance comparison of our proposed method with different feature combination, according to image sequence length L length (spring vs winter). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8 Ground-truth. Since frame from one season corresponds to the same frame in any of the other three seasons, the ground-truth is diagonal. . . . . . . . 4-9 Matching results under different season couples at 100% recall situation. The expected result is along the diagonal. . . . . . . . . . . . . . . . . . .

Table 2 .

 2

	1: Summary of main local features
	Name	Dimensions type References
	SIFT	128	Float	[63]
	SURF	32, 64, 128 Float	[9]
	PCA-SIFT	36	Float	[49]
	KAZE	64	Float	[3]
	LBP	59	Bit	[80]
	CLBP	514	Bit	[42]
	CSLBP	16	Bit	[43]
	CSLDP	16	Bit	[112]
	XCSLBP	16	Bit	[95]
	BRISK	512	Bit	[61]
	BRIEF	512	Bit	[16]
	ORB	256	Bit	[90]
	FREAK	512	Bit	[1]
	AKAZE	488	Bit	[2]
	LDB	256, 512	Bit	[6]

Table 2 .

 2

	each place is described by a dictionary of these representative PIRFs, whose appearance
	variation is assumed relatively small with regard to robot motion. Andreasson and Duck-
	ett [4] present a simplified version of the SIFT algorithm-M-SIFT (Modified SIFT fea-
	tures) which selects interest points from omni-directional images. The results show that the
	method based on local features M-SIFT obtain a significant high level of performance and
	robustness to environmental variations.

1 collects some of the main local features used in place recognition. For ex-ample, Murillo et al.

[START_REF] Bay | SURF: Speeded Up Robust Features[END_REF]

use SURF while FrameSLAM

[START_REF] Konolige | Frameslam: From bundle adjustment to real-time visual mapping[END_REF]

adopts center-surround feature (CenSurE). Kawewong et al.

[START_REF] Kawewong | Position-invariant robust features for long-term recognition of dynamic outdoor scenes[END_REF]

use Position-Invariant Robust Features (PIRFs):

Table 2

 2

	.2. Exam-

  Algorithm 3.1 Place recognition based visual localization using multi-feature.

	Inputs:
	{I train j	} N train j=1 {training image pairs database (left and right images)};
	{I test i } N test i=1 {testing images pairs (left and right images)}; N train , N test {training and testing images number};
	Outputs:
	SS{distance ratio}; Vehicle position
	Proposed Algorithm:
	/ * OFF-LINE PHASE * /
	/ * Training images multi-feature extraction and concatenation; {Section 3.3.2 and 3.3.3}
	* /
	for j ← 1 to N train do
	v train gray, j , v train HOG, j ← Block based λ LBP and HOG features extraction from gray-scale
	images;
	v train dis, j ← Block based λ LBP feature extraction from disparity map;
	v train j	← v train gray, j + +v train dis, j + +v train HOG, j ; //Multi-feature concatenation.
	end for
	/ * ON-LINE PHASE * /
	/ * Feature comparison * /
	for i ← 1 to N test do
	v test i	← v test gray,i + +v test dis,i + +v test HOG,i ; //Multi-feature computation for the current testing
	image pair.
	for j ← 1 to N train do
	D i, j = ∥v test i -v train j	∥ 2 ; Euclidean distance computation between the multi-feature
	v test i	of the testing image and the multi-feature v train j	in the training database.
	end for
	/ * Matching validation and localization * /
	SS i =	min {D i, j } j min j( j̸ =m1) D i, j ; m1 is the index of the first best candidate.
	if SS i <= T h
	Matching validation is positive;
	Vehicle position ← the matched training image position
	if SS i > T h
	Matching validation is negative;
	Vehicle position ← NaN (no position result)
	end for

Table 3 .

 3 1: F 1 score comparison for different tested binary features on four datasets. Here the block size is set to 32×32.

	Feature	UTBM-1 UTBM-2 KITTI 05 KITTI 06
	LBP	0.5171	0.9058	0.7361	0.8261
	D-LBP	0.7665	0.9441	0.7663	0.8639
	CLBP	0.5111	0.9194	0.7437	0.8279
	D-CLBP	0.6672	0.9221	0.7735	0.8813
	CSLBP	0.6292	0.9337	0.7569	0.8461
	D-CSLBP	0.8043	0.9457	0.7763	0.8850
	CSLDP	0.6093	0.9474	0.7536	0.8335
	D-CSLDP	0.8062	0.9490	0.7709	0.8709
	XCSLBP	0.4796	0.8986	0.7107	0.7814
	D-XCSLBP	0.7190	0.8350	0.7401	0.7775

  ). The performance of D-CSLBP feature with different block sizes[START_REF] Eustice | Visually navigating the rms titanic with slam information filters[END_REF][START_REF] Lowry | Visual place recognition: A survey[END_REF] 128, and 32+64+128 (multi-block sizes, there different block sizes used together)) in place recognition is evaluated.

			1													1
																0.99
			0.95													0.98
			0.9													0.97
	Precision	0.85												Precision	0.94 0.95 0.96
			0.8													0.93
			0.75			D-CSLBP(Block size 32) D-CSLBP(Block size 64)							0.92		D-CSLBP(Block size 32) D-CSLBP(Block size 64)
						D-CSLBP(Block size 128)							0.91		D-CSLBP(Block size 128)
						D-CSLBP(Block size 32+64+128)								D-CSLBP(Block size 32+64+128)
			0.7	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1	0.9	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1
									Recall											Recall
						(a) UTBM-1 dataset								(b) UTBM-2 dataset
			1												1
		0.995												0.98
			0.99												0.96
		0.985												0.94
	Precision	0.97 0.975 0.98											Precision	0.9 0.92
		0.965												0.88
			0.96		D-CSLBP(Block size 32)							0.86		D-CSLBP(Block size 32)
		0.955		D-CSLBP(Block size 64) D-CSLBP(Block size 128)							0.84		D-CSLBP(Block size 64) D-CSLBP(Block size 128)
						D-CSLBP(Block size 32+64+128)								D-CSLBP(Block size 32+64+128)
			0.95	0	0.1		0.2	0.3	0.4	0.5		0.6	0.7	0.1 0.82	0.2	0.3	0.4	0.5	0.6		0.7	0.8	0.9	1
									Recall											Recall
								(c) KITTI 05										(d) KITTI 06

Table 3 .

 3 2: Comparison of F 1 scores for different features and the state-of-the-art FAB-MAP method, on four datasets.

	Dataset	F 1 score D-CSLBP HOG D-CSLBP++HOG FAB-MAP
	UTBM-1	0.8043	0.8752	0.8869	0.2356
	UTBM-2	0.9299	0.9440	0.9532	0.4813
	KITTI 05	0.7763	0.7782	0.7873	0.7417
	KITTI 06	0.8850	0.8648	0.8973	0.3519

Table 3 .

 3 2 compares the F 1 scores of different features with the state-of-the-art FAB-

MAP method. It confirms that the multi-feature D-CSLBP++HOG achieves better results

Table 3 .

 3 3 shows the F 1 score obtained form different hash bit lengths applied on the

		1												1
		0.95
													0.98
		0.9
													0.96
	Precision	0.75 0.8 0.85										Precision	0.94		D-CSLBP++HOG(Full feature) D-CSLBP++HOG(4096 bits) D-CSLBP++HOG(2048 bits) D-CSLBP++HOG(1024 bits) D-CSLBP++HOG(512 bits) D-CSLBP++HOG(256 bits)
													0.92
		0.7		D-CSLBP++HOG(Full feature) D-CSLBP++HOG(4096 bits)
		0.65		D-CSLBP++HOG(512 bits) D-CSLBP++HOG(2048 bits) D-CSLBP++HOG(1024 bits)							0.9
				D-CSLBP++HOG(256 bits)
		0.6	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	0.88	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1
							Recall											Recall
					(a) UTBM-1 dataset									(b) UTBM-2 dataset
		1													1
		0.99
		0.98												0.95
		0.97
	Precision	0.95 0.96		D-CSLBP++HOG(Full feature) D-CSLBP++HOG(4096 bits) D-CSLBP++HOG(2048 bits) D-CSLBP++HOG(1024 bits) D-CSLBP++HOG(512 bits) D-CSLBP++HOG(256 bits)					Precision	0.9 0.85	D-CSLBP++HOG(Full feature) D-CSLBP++HOG(4096 bits) D-CSLBP++HOG(2048 bits) D-CSLBP++HOG(1024 bits) D-CSLBP++HOG(512 bits) D-CSLBP++HOG(256 bits)
		0.94
		0.93												0.8
		0.92
		0.91	0	0.1		0.2	0.3	0.4	0.5		0.6	0.7		0.1 0.75	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1
							Recall											Recall
						(c) KITTI 05										(d) KITTI 06
	Figure 3-15: Image retrieval performance (precision-recall curve) comparison of different
	hash bit lengths.
	multi-feature (D-CSLBP++HOG) of our place recognition method. The average matching
	time is also presented. Here average matching time does not include the feature extraction
	time. The experiments were conducted on a laptop machine with intel i7-4700MQ CPU
	and 32G RAM.

multi-feature of different binary lengths (2 8 . . . 2 12 bits) on four datasets in Fig.

3

-15. Since the image size is different, multi-feature dimension in datasets UTBM-1 and UTBM-2 is 18696 while the multi-feature dimension in KITTI 05 and KITTI 06 is 6432. It can be seen that, using 4096 and 2048 bits retain above 86% total place recognition performance.

Table 3 .

 3 3: F 1 score and matching times comparison of different hash bit lengths for our approach and the state-of-the-art FAB-MAP method.

	Method	F 1 score UTBM-1 UTBM-2 KITTI 05 KITTI 06	Average time per matching (All datasets)
	256 hash bits	0.6571	0.8989	0.7425	0.8411	0.11×10 -2 s
	512 hash bits	0.8097	0.9409	0.7862	0.8786	0.38×10 -2 s
	1024 hash bits	0.8488	0.9562	0.7794	0.8936	0.88×10 -2 s
	2048 hash bits	0.8771	0.9532	0.7865	0.8973	1.78×10 -2 s
	4096 hash bits	0.8869	0.9537	0.7873	0.9006	3.61×10 -2 s
	Multi-feature	0.9258	0.9110	0.9166	0.9203	8.82× 10 -2 s
	FAB-MAP	0.2356	0.4813	0.7417	0.3519	2.83× 10 -2 s

  places are not totally correct, some false recognition places appear. For these false recognized places, the localization error can be very large, because the testing image can be wrongly matched to anyone in the training image database. That is also the reason why some locations have huge localization error.

									Th=1
									Th=0.84 (precision=100/%)
									Th=0.88 (precision=99/%)
									Th=0.99 (precision=90/%)
	Table 3.4: Recall results and average localization error at three precision levels (4096 hash bits). Localization error(m) 100% precision 99% precision 90% precision Dateset Recall (%) Error (/m) Recall (%) Error (/m) Recall(%) Error (/m)
	UTBM-1	23.81		2.08		89.62	2.34		95.00	3.53
	UTBM-2	11.35		2.11		51.89	2.41		89.50	2.49
	KITTI 05	17.38		3.50		61.59	13.08		65.92	13.08
	KITTI 06	32.39	100	200 2.56	300	400 59.12	500 3.18	600	700 77.26	800 4.20	900
					Image number(frame)
					(a) UTBM-1 dataset
									Th=1
									Th=0.58 (precision=100/%)
									Th=0.88 (precision=99/%)
									Th=0.93 (precision=90/%)
	Localization error (m)
				100	200	300		400	500		600
					Image number(frame)
					(b) UTBM-2 dataset

Place recognition based localization errors at different precision levels. The dark line is the localization error for all recognition results (including the true positives and false positives). Red "o" is localization error at 100% precision level. Green "+" is localization error at 99% precision level. Blue "." is localization error at 90% precision level. nized

  Algorithm 4.1 Sequence matching based visual localization using feature combination. Sequence matching based on feature sequences; {Section 4.2.3} * / for i ← 1 to N test /L length do for j ← 1 to N train /L length do

	Inputs:
	{I train j	} N train j=1 {training images}; {I test i } N test i=1 {testing images };
	N train , N test {training and testing images numbers};
	Outputs:
	SS{distance ratio}; Vehicle position
	Algorithm:
	/ * Feature extraction and combination; {Section 4.2.2} * /
	for j j ← 1 to N train do
	f CSLBP , f GIST ← CSLBP and GIST features extraction for training images;
	f train j j	← f CSLBP + + f GIST ; //Feature combination.
	end for
	for j ← 1 to N train /L length do
	j j ← ( j -1)×L length ;
	F train j j j+L end for ← f train j j+1 + + f train j j+2 + + • • • f train
	for i ← 1 to N test /L length do
	ii ← (i -1)×L length ;
	F test i	← f test ii+1 + + f test ii+2 + + • • • f test ii+L length ; //Feature of testing sequence.
	end for
	/ *

Algorithm 4.1 illustrates the proposed method for sequence matching based visual localization. It includes feature extraction and combination, image sequence matching, matching validation and visual localization steps. length ; //Feature of training sequence. end for for ii ← 1 to N test do f CSLBP , f GIST ← CSLBP and GIST features extraction for testing images; f test ii ← f CSLBP + + f GIST ; //Feature combination.

  square distance computation, k is the index of the compents of feature vector.

	end for
	/ * Matching validation and visual localization; {Section 4.2.4 and 4.2.5} * /
	SS i =	min {D i, j } j min j( j̸ =m1) {D i, j } ; m1 is the index of the first minimum distance.
	if SS i <= T h
	Matching validation is positive;
	Vehicle position ← the matched training image position
	if SS

i > T h Matching validation is negative; Vehicle position ← NaN (no position result) end for

Table 4

 4

	Precision (%)
	spring vs summer
	spring vs fall
	spring vs winter
	summer vs fall
	summer vs winter
	fall vs winter
	10	20	30	40	50	60	70	80	90	100
					Recall (%)

.1. For all the cases, our

Table 4 .

 4

	1: Recall scores at selected high precision levels (100%, 99%, 90%)
	Different season	Method	100%	99%	90%
	couples		precision	precision	precision
		SeqSLAM	20.45	27.73	66.11
	Spring vs Summer	LBP++GIST	87.64	87.64	92.70
		CSLBP++GIST	87.64	87.64	93.26
		SeqSLAM	15.41	27.45	63.87
	Spring vs Fall	LBP++GIST	88.20	89.89	93.82
		CSLBP++GIST	89.33	91.01	93.28
		SeqSLAM	14.29	17.37	62.18
	Spring vs Winter	LBP++GIST	63.48	66.58	82.58
		CSLBP++GIST	60.67	62.92	82.58
		SeqSLAM	9.80	23.81	65.27
	Summer vs Fall	LBP++GIST	71.35	76.97	87.64
		CSLBP++GIST	93.26	95.51	97.19
		SeqSLAM	14.01	27.45	53.50
	Summer vs Winter	LBP++GIST	62.92	67.42	79.21
		CSLBP++GIST	64.04	67.98	80.90
		SeqSLAM	2.24	2.35	44.82
	Fall vs Winter	LBP++GIST	64.40	66.29	77.53
		CSLBP++GIST	66.85	67.42	76.97

Table 4 .

 4

	2: Comparison of average processing times for image matching(/s)
	No. images in	SeqSLAM LBP++GIST CSLBP++GIST
	database
	200	3.5335	3.1280	2.9463
	2000	54.6982	20.4664	18.3309
	20000	87.6982	33.4664	37.3309

•

  Final Matching Validation (detailed in Section 5.2.4): For each testing sequence, the best training sequence candidate will be validate in this step to reduce some false recognition. The ratio SS between the two best sequence matching scores is used to verify the best sequence candidate. If the ratio SS is below or equal to a threshold T h, the first candidate (with the lower matching score) is confirmed and regarded as positive matching, otherwise it is considered as negative one (in this case, no matching is conserved).

Table 5 .

 5 1: The layers from the caffe-alex ConvNet model used in our evaluation and their output dimensionality (height×width×feature map number).

		Layer Dimensions Layer Dimensions
		conv4 13 × 13 × 384 pool5 6 × 6 × 256
		conv5 13 × 13 × 256 fc6 1 × 1 × 4096
		relu5 13 × 13 × 256 fc7 1 × 1 × 4096
	5.2.2 Feature comparison
	Feature comparison is performed based on the cosine distance between ConvNet features.
	Each test image feature is compared with the image features of all the training images. For
	that, the cosine distances between the feature f test T of the current testing image I test T and the
	features { f train i	} N train i=1 of all the images of the training database are computed as follows :

  N train , N test {training and testing images numbers}; V max ,V min {maximum and minimum vehicle speeds}; V step {Vehicle speed step-size}d s {Sequence length} ; ←[d i,1 , d i,2 , • • • , d i,N train ]; Column vector D i ∈ R N train×1 that contains the cosine distance between the testing image I test Localized sequence matching and validation * / for T ← d s to N test doM T ← [D T -d s +1 , D T -d s +2 , • • • , D T ]; // Construction of the local searching matrix. j ← 1; // Initialization; for s ← 0 to (N train -V max × d s ) do for V ← V min :V step : V max do S T, j ← 0; for t ← (Td s + 1) to T do k(t) ← s +V (t -(Td s +1)); // k is a line index in the column vector D t ; s is the training image number from which the path originated in (Section 5.2.3).

	Algorithm 5.1 Localized sequence matching based visual localization. 5.3 Experimental Setup
	Inputs:
	{I train i	} N train i=1 {training images database}; {I test i } N test i=1 {testing images database};
	Outputs:
	S {Path-line (sequence candidate) score };
	Vehicle position;
	Algorithm:
	/ * ConvNet features extraction; {section 5.2.1} * /
	for i ← 1 to N train do F train = { f train } ← Feature extraction for training images; i end for for i ← 1 to N test do F test = { f test i } ← Feature extraction for testing images; end for
	/ * Feature comparison * /
	for i ← 1 to N test do for j ← 1 to N train do
	d i, j ← cos⟨ f test i , f train j	⟩; // Computation of the cosine distance between the test
	image I test i end for	and the training image I train i	{Section 5.2.2}.
	D i i end for	and all the training images.
	/ * S T, j ←S T, j + D t k(t) ;// Score S is calculated for each possible path.
		end for
		j ← j + 1; Path-line number (sequence candidates) update;
	end for
	end for SS T = j( j̸ =m1) min {S T, j } j min {S T, j } ; m1 is the index of minimum score.
	if SS T <= T h Vehicle position ← The matched training images position Algorithm 5.1 illustrates the ConvNet features and localized sequence matching based vi-if SS T > T h Vehicle position ← NaN (no position results) sual localization. end for
				111

Table 5 . 2 :

 52 Description of the main characteristics of the datasets employed in the experiments.

	Dataset	Length	No.images	Description
	UTBM-1 2×4.0 KM	training:848 testing:819	minor variations in appearance and illumination
	UTBM-2 2×2.3 KM	training:540 testing:520	medium variations in appearance and illumination
	Nordland 4×728 KM 4×3577	severe variations in appearance
	City Center 2×2.0 KM	training:1352 testing:1122	medium variations in appearance and illumination

  -10, the influence of the sequence length (d s ) is decisive to improve the performance of visual localization in life-long conditions. It can be clearly found that the approach using sequence allows to achieve better results than that of single image (almost no recall at 100% precision) in long-term visual localization.

		1											1
		0.9
													0.95
		0.8
		0.7											0.9
		0.6
	Precision	0.5										Precision	0.8 0.85
		0.4
		0.3											0.75
		0.2	d s =1 d s =4											d s =1 d s =4
			d s =6										0.7	d s =6
		0.1	d s =8											d s =8
			d s =10											d s =10
		0											0.65
		0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1
							Recall											Recall
					(a) UTBM-1						(b) Nordland (spring vs summer)
	Figure 5-10: Two examples of performance comparison of our proposal depending on
	the image sequence length (d s ) in the challenging UTBM-1 and Nordland (fall vs winter)
	datasets. The feature used here is conv4 layer.
	Furthermore, there is a limit near to a length of 8 for UTBM-1 dataset and a length
	of 6 for Nordland dataset form which the results are not greatly enhanced. Based on this

sequence length comparison and the driving speed, a sequence length of d s =8 was chosen for datasets UTBM-1 and UTBM-2 in the rest of the experiments and results. For City Center dataset, sequence length was set to 3 and for Nordland dataset is 6. For all datasets,

Table 5 .

 5 4 shows the F 1 scores of different hash bit lengths achieved in four datasets.

	The average times per matching are also presented. The experiments are conducted on a
	123

Table 5 .

 5 4: F 1 scores and matching time comparison of different lengths of hash bits.

	Method	F 1 score UTBM-1 UTBM-2 City Center	Nordland (spring vs Winter)	Average time per matching(All datasets)
	256 hash bits 0.9276 0.9574	0.9094	0.8817	0.0135 s
	512 hash bits 0.9219 0.9554	0.9084	0.8944	0.0147s
	1024 hash bits 0.9460 0.9612	0.9162	0.9046	0.0170s
	2048 hash bits 0.9497 0.9632	0.9246	0.9064	0.0209s
	4096 hash bits 0.9478 0.9641	0.9166	0.9099	0.0291s
	Full feature 0.9607 0.9622	0.9228	0.9255	0.3259 s

Table 5 .

 5 5: Recall results and average localization error at two precision levels (4096 hash bits).

	Dateset	100% precision Recall (%) Error (/m) Recall (%) Error (/m) 99% precision
	UTBM-1	32.88	2.32	89.62	2.67
	UTBM-2	11.54	2.39	51.89	2.62
	City Center	76.29	3.36	82.89	3.88
	Nordland (spring vs winter)	72.88	2.56	82.95	3.18

http://grandchallenge.mit.edu/images/all

http://cs.stanford.edu/group/roadrunner/photos.html

https://www.google.com/selfdrivingcar

http://velodyneLIDAR.com/hdl-64e.html

https://nrkbeta.no/2013/01/15/nordlandsbanen-minute-by-minute-season-by-season/

Acknowledgements I would like to thank my supervisors Prof. Yassine RUICHEK and associate Prof. Cindy CAPPELLE, for the continuous support and guidance on improving my research. Working with them, I learned a lot both scientific paper writing and critical thinking, both of which are important to my future projects. I am grateful to them for the help they have given me. With their research experience, they teach me how to become an independent researcher. I would like to give my gratitude to the financial support from the program of China Scholarship Council (CSC). The CSC program is really helpful which gave me a chance to study abroad. I would like to express my gratitude to Professor Louahdi KHOUDOUR and Professor Abdelmalik TALEB-AHMED, for reviewing this thesis and posing insightful questions. I would like to extend my appreciation to Mr. Cyril MEURIE and Mr. Pierre GOUTON, for examining this thesis. I would like to thank Ms. Lijun WEI, thank you very much for your encourage and help in the early stage of PhD study. Also Thanks very much for Mr. You LI, Mr. Hongjian WANG and Mr. Julian MURGIA, for their help in my study and research. I would also like to thank my colleagues who helped me in all these years of my thesis: Mr. Yong FANG, Mr. Tao YANG, Ms. Citlalli GAMEZ and all the others who worked with me. It is really happy and helpful to communicate and worked with them. I also express my gratitude to Prof. Fadi DORNAIKA, Professor at the University of the Basque Country (Spain), whose advice is always valuable.

Lastly, I would like to express a special thanks to my family and friends, thanks for their emotional supports and research help.

Chapter 4

Visual Localization Across Seasons Using Sequence Matching Based on

Feature Combination

In Chapter 3, visual localization is achieved by single image matching based on multifeature D-CSLBP++HOG. The multi-feature obtained from gray-scale image and disparity map showed its advantages in place describing. However, using stereo camera to get disparity map, increases the hardware costs and processing time. In addition, HOG feature is easy to be influenced by huge appearance variation, different illumination and seasonal changing in long-term localization.

In this chapter, the problem of visual localization across seasons is addressed using feature combination and sequence matching. Feature combination of CSLBP and GIST is used to make place describing more robust, and sequence matching rather than single image matching is used to improve the place recognition ability. Based on the above consideration, a more robust visual localization system based on feature combination and sequence matching is proposed. Experimental evaluation will show that our method is an effective tool to perform visual localization across seasons. The results will also show an improved precision-recall performance against state-of-the-art SeqSLAM algorithm. sequence matching step based on Chi-square distance (Section 4.2.3).

In the step of sequence matching, for each testing sequence, the sequence candidate from the training database that provides the minimum distance can be considered as the most similar one to the testing sequence. In fact, the two best sequence matching candidates are conserved for further verifying the final matching result.

Effectively, the best matching candidate will be validated through a distance ratio SS (Section 4.2.4), computed from the two minimum scores (the first minimum distance divided by the second minimum distance). If the ratio SS is lower than or equal to a threshold T h, the first best sequence candidate (with the lower distance) is confirmed and regarded as positive matching, otherwise it is considered as negative one (in this case, no matching result is conserved). When a sequence candidate is confirmed as positive, the position can be obtained from the GPS information that correspond to the matched training images (Section 4.2.5).

Image preprocessing

As mentioned in paper [START_REF] Roy | Visual Route Recognition with a Handful of Bits[END_REF] and already considered in the previous chapter, high resolution images are not needed to perform an effective visual recognition along time. Indeed, high resolution images increase computational cost without bringing significant visual recognition improvement. For image storage and efficient matching, in this work, the original images are down-sampled into 32×32 pixels before feature extraction.

Feature extraction

Feature extraction consists of three steps: (1) CSLBP and GIST are firstly extracted from image independently; (2) Then they are concatenated (++) together to form multi-feature;

(3) Finally, the CSLBP++GIST multi-features obtained from images of a sequence are concatenated (++) to form the final sequence feature (F) representing the sequence. 1) CSLBP feature: As already described, CSLBP is a modified version of LBP. We recall that, for an even number P of neighboring pixels distributed on radius R, the feature of uniform LBP pattern (the property of the "uniform patterns" is that the number of 0-1

Evaluation method

Precision-recall characteristics are widely used to evaluate the effectiveness of image retrieval. Therefore, as used in the previous chapter, our evaluation methodology is based on precision-recall curves. These curves are determined by varying the threshold T h between 0 and 1, applied to the ratio SS and calculating precision and recall (see section 4.3.2).

Precision relates to the number of correct matches to the number of false matches, whereas recall relates to the number of correct matches to the number of missed matches. Positives are considered when ratio is lower than or equal to the threshold T h. Here 100 threshold values are considered to obtain well-defined precision-recall curves.

In this experiment, the training image number is equal to the testing image number, and each testing image has a ground truth matching. Therefore, there are only true positives (correct results among successfully validated image matching candidates) and false positives (wrong results among successfully validated image matching candidates). The sum of the true positives and false positives is the total retrieved image numbers.

Experiments and Results

Feature combination analysis

In a first set of experiments, we evaluate how well do feature combinations perform for place recognition and also compare the results with those obtained by the state-of-art Se-qSLAM method. The experiments were conducted using the videos presenting extreme situation in terms of appearance changes (Spring vs Winter). The length of each sequence is 200 images. As shown in Fig. 456, the CSLBP and GIST features perform very well when they are used independently. Indeed, our method with CSLBP performs relatively well at high precision level, while GIST outperforms SeqSLAM method. When using the multi-feature (CSLBP++GIST), the retrieval ability is increased significantly. The reason is that CSLBP++GIST takes advantage of local and global information can distinguish the similar images more accurately.

It can be seen that our method with CSLBP++GIST can reach around 65% of recall at reduction using space projection techniques or searching methods like local sensitive hashing (as done in the approach presented in Chapter 3). Another drawback is that, in the performed experiments, testing and training sequence lengths are same as that twice driving speeds of train are very close. In the future, more flexible sequence length selection and matching strategy should be considered.

The City Center dataset was collected by Mobile Robotics Group of the University of Oxford [START_REF] Cummins | FAB-MAP: Probabilistic localization and mapping in the space of appearance[END_REF]. The robot traveled twice around a loop with total path length of 2 km, and 2,474 images were collected by two (left and right) cameras mounted on the robot while traveling. This dataset was collected on a windy day with bright sunshine, which makes the abundant foliage and shadow features unstable, as can be observed in Fig. 56789.

For all the four datasets, ground-truth was constructed by manually finding pairs of frame correspondences based on GPS position.

Performance evaluation

As already described and used in the previous chapter, precision-recall curves and F 1 scores are still used in this chapter to evaluate the proposed approach. The final curve is computed by varying the threshold T h in a linear distribution between 0 and 1 and calculating the corresponding values of precision and recall. 100 threshold value are processed to obtain well-defined precision-recall curves.

In our experiments, the training images number is larger than or equal to the testing images number, thus each testing image has a ground-truth match. Therefore, among the positives, there are only true positives (correct results among successfully validated image matching candidates) and false positives (wrong results among successfully validated image matching candidates). The sum of the true positives and false positives is the total retrieved images number.

More specifically, precision is the ratio of true-positives over the retrieved images number (number of all the successfully validated images matching candidates), and recall is the ratio of true-positives over the total testing images. A perfect system would return a result where both precision and recall have a value of one. Based on the precision and recall, F 1 score can be defined as:

tions. The precision-recall curves are presented in Fig. 5-15. According to this figure, the features from mild-layers (conv4, conv5, relu5, pool5) can slightly deal with the illumination changing problem, while the other layer features (fc6 and fc7) can not deal with the severe illumination changes. This is maybe because the pre-trained network build the model under good quality images (normal illumination situation), and then it does not show strong robust ability in illumination variance situation.

According to the precision-recall curves for appearance and illumination changing situation on these four different tested datasets. It can be found that, the mid-level features from layers conv4 and relu5 are more robust against appearance and illumination changes than the other layer features. While higher layers (fc6 and fc7) in the feature hierarchy lack robustness and exhibit inferior place recognition performance.

Table 5.3 shows F 1 scores obtained for the proposed method using different layers, the previous proposed methods (D-CSLBP++HOG and CSLBP++GIST) and state-of-the-art methods like FAB-MAP and SeqSALM. For SeqSLAM comparison, the OpenSeqSLAM code [START_REF] Milford | SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights[END_REF] was used and the same sequence lengths were taken as settled above. While the other parameters are set to default values as reported in [START_REF] Milford | SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights[END_REF]. For FAB-MAP comparison, the OpenFABMAP code [START_REF] Glover | Open-FABMAP: An open source toolbox for appearance-based loop closure detection[END_REF] was used. It can be found that localized sequence matching using features extracted by layer conv4 matches or exceeds the performance of the other methods.

Abstract :

In many applications, it is crucial that a robot or vehicle localizes itself within the world especially for autonomous navigation and driving. The goal of this thesis is to improve place recognition performance for visual localization in changing environment. The approach is as follows : in off-line phase, geo-referenced images of each location are acquired, features are extracted and saved. Then, in the on-line phase, the vehicle localizes itself by identifying a previously-visited location through image or sequence retrieving. However, visual localization is challenging due to drastic appearance and illumination changes caused by weather conditions or seasonal changing. This thesis addresses the challenge of improving place recognition techniques through strengthen the ability of place describing and recognizing. Several approaches are proposed in this thesis : 1) Multi-feature combination of CSLBP (extracted from gray-scale image and disparity map) and HOG features is used for visual localization. By taking the advantages of depth, texture and shape information, visual recognition performance can be improved. In addition, local sensitive hashing method (LSH) is used to speed up the process of place recognition ; 2) Visual localization across seasons is proposed based on sequence matching and feature combination of GIST and CSLBP. Matching places by considering sequences and feature combination denotes high robustness to extreme perceptual changes ; 3) All-environment visual localization is proposed based on automatic learned Convolutional Network (ConvNet) features and localized sequence matching. To speed up the computational efficiency, LSH is taken to achieve real-time visual localization with minimal accuracy degradation.