Xiaolong Wu

Stéphanie Aravecchia

Cédric Pradalier

Design and Implementation of Computer Vision based In-Row Weeding System

Autonomous robotic weeding systems in precision farming have demonstrated their full potential to alleviate the current dependency on herbicides or pesticides by introducing selective spraying or mechanical weed removal modules, thus reducing the environmental pollution and improving the sustainability. However, most previous works require fast weed detection system to achieve real-time treatment. In this paper, a novel computer vision based weeding control system is presented, where a non-overlapping multi-camera system is introduced to compensate the indeterminate classification delays, thus allowing for more complicated and advanced detection algorithms, e.g. deep learning based methods. The suitable tracking and control strategies are developed to achieve accurate and robust in-row weed treatment, and the performance of the proposed system is evaluated in different terrain conditions in the presence of various delays.

I. INTRODUCTION

Feeding a growing population and protecting the environment are two of the main challenges for the next few decades. Precision agriculture, also known as smart farming, addresses both challenges notably by targeting treatment only to plants that need it. Indeed, today's crop cultivation practices rely heavily on the uniform application of herbicides for weed control. In this scope, developing flexible robotic systems is essential to widespread precision agriculture. Within the EUproject Flourish [START_REF]The flourish project[END_REF], an autonomous weed control system has been designed, which can manage weeds either chemically with a precise spot-spraying of herbicide, or mechanically with a stamper tool destroying the weeds.

To precisely target the weeds, the first step is actually to separate the weeds from the value crops. Vision-based classification approaches have proven effective in this domain, is even capable enough to detect the stem of the weeds [START_REF] Lottes | Joint stem detection and crop-weed classification for plant-specific treatment in precision farming[END_REF]. Such methods, using Convolutional Neural Network (CNN), introduce a new challenge which is the indeterminate delay between the time the image is captured to perform the classification and the time the results of the classification are available to trigger the actuators.

Our proposed system, in Fig. 1, addresses this by introducing a non-overlapping multi-camera tracking system that could reliably retrace the 3D locations of weeds using intra-and inter-camera template-based tracking. The adverse effect of the delays induced by the CNN-based classifier can be efficiently alleviated, and the predictive control layer is designed to provide an accurate prediction of trigger timing and locations for weed treatment tools. A mechanical stamp tool, as well as a selective sprayer, are implemented in *All authors are with the UMI2958 GeorgiaTech-CNRS, Metz, France.

firstname.lastname@georgiatech-metz.fr Fig. 1: Left: the Flourish BoniRob with our proposed weed control hardware module mounted at the bottom. Upper-Right: a CAD view of the module including the actuators and sensors. Bottom-Right: the example pictures of the selective sprayer and the mechanical stamping tool. our weed control system to achieve fast and accurate weed treatment.

This paper makes the following contributions: 1) A novel self-sustained weed control system is designed, implemented, and extensively evaluated in the field for in-row weed removal; 2) a 3D multi-camera multi-object tracking algorithm using direct or indirect template matching is developed to provide fast and accurate weeds tracking across nonoverlapping cameras.

II. RELATED WORK

Robotic system for weed control has been in the field of research for a few decades [START_REF] Bechar | Agricultural robots for field operations. part 2: Operations and systems[END_REF]. Several methods to remove weeds adapted to robotics have been developed, either chemically or mechanically, for in-row weeding. Spraying of herbicides is a common weed-control method in traditional agriculture. Therefore, many robotics platforms performing selective spraying have been developed [START_REF]Ecorobotix[END_REF]. Significant effort has been made towards the in-row weeding [START_REF]Naio dino large-scale vegetable weeding robot[END_REF] [6], however, the mechanical weed removal in the close-to-crop area is still challenging. One solution is to stamp the weeds with a metal rod with a diameter of 1 cm, proven effective in [START_REF] Michaels | Vision-based high-speed manipulation for robotic ultra-precise weed control[END_REF].

In recent years, significant progress has been made towards machine vision based real-time weed detection using learning-based methods. In [START_REF] Mccool | Mixtures of lightweight deep convolutional neural networks: applied to agricultural robotics[END_REF], a mixture of small components of a fine-tuned deep conventional neural network (DCNN) is utilized to achieve accurate weed-crop classification, which could achieve a 90% precision with a 1.07-1.83Hz processing time. In [START_REF] Sa | weednet: Dense semantic weed classification using multispectral images and mav for smart farming[END_REF], an encoder-decoder cascaded conventional neural network (CNN) is utilized to perform the dense semantic weed-crop classification, which achieves Fig. 2: Our proposed weed control system utilizes a multicamera system to track the detected weeds using template matching techniques with uncertain detection delays, where the tracking can be naturally divided into intra-and intercamera tracking layers. The tracked targets will be fed into a predictive control module to estimate the timing and position of treatment when approaching the weeding app. a high classification precision but cannot guarantee a 2-5Hz real-time performance. In [START_REF] Lottes | Fully convolutional networks with sequential information for robust crop and weed detection in precision farming[END_REF], a fully convolutional network (FCN) is formulated to encode the spatial information of plants in a row over sequentially acquired image sequence to perform the classification task, resulting in a stable 5Hz runtime performance . Other learning-based methods have been specialized on stem detection [START_REF] Haug | Plant stem detection and position estimation using machine vision[END_REF], [START_REF] Kraemer | From plants to landmarks: Time-invariant plant localization that uses deep pose regression in agricultural fields[END_REF], [START_REF] Lottes | Joint stem detection and crop-weed classification for plant-specific treatment in precision farming[END_REF].

Concerning the tracking module, our proposed system implements a fixed-viewpoint down-looking camera and an entirely controlled lighting system, so that most of the state-of-the-art non-overlapping camera tracking methods are excessively complex for our application. For this reason, only applicable works are surveyed in this paper. In [START_REF] Chilgunde | Multi-camera target tracking in blind regions of cameras with non-overlapping fields of view[END_REF] [14] [START_REF] Chu | Tracking across multiple cameras with overlapping views based on brightness and tangent transfer functions[END_REF], Kalman filter based multi-camera tracking systems are developed to track multi-object in 2D image space. In [START_REF] Choi | Robust 3d visual tracking using particle filtering on the special euclidean group: A combined approach of keypoint and edge features[END_REF], the particle filter based 3D model-based visual tracking algorithm utilizing lie group components is developed, where the both indirect and direct formulation are implemented to improve its robustness.

III. SYSTEM DESIGN

In this section, the hardware description in Sec.III-A and software designs: the classification module in Sec. III-B, the multi-camera tracking module in Sec. III-C, and finally the control module in Sec. III-D, are briefly described.

A. Hardware Description 1) Mechanical design:

The weeding-unit, in Fig. 1, is the self-sustained module responsible for detecting and treating the weeds. The weeds are treated either mechanically by stampers or chemically by sprayers: 18 individually controlled stamps with 10mm-diameter bolts provide the mechanical weeiding, while 9 individually controlled nozzles deliver the chemical weeding. The actuators are controlled with a scalable Programmable Logic Controller (PLC using absolute timestamps, and the clocks (PC, PLC) are synchronized to fulfill the timing requirement for precision treatment.

2) Perception system and calibration: Three downlooking 4-channels JAI AD-130 GE cameras are mounting on our proposed system with the distance to the ground being approximatively 80cm. The ground resolution is approximatively 3 px/mm, and the field of view is 24 x 31 cm. To keep the same light conditions under the cameras the whole unit is covered by 3 mm acetal copolymer sheets, and artificial lights are mounted under to control the lighting. The RGB+NIR images are used for classification, and the RGB images only are used for tracking. Both the intrinsic and Inter-camera extrinsic parameters are estimated using OpenCV.

Three narrow-beam sonars (SRF235 Ultrasonic range finder), are mounted next to each camera to avoid doing the calibration every time the unit is pulled up and then down. The accuracy of the range measurement of these sonars is 1cm.

B. Classification

The first step toward weed treatment is weed detection. In this project, the weed detection, published in [START_REF] Lottes | Semi-supervised online visual crop and weed classification in precision farming exploiting plant arrangement[END_REF], is a vision-based online classification of crops and weeds using RGB+NIR data. The runtime on the UGV computer (Intel i7 CPU, GeForce GTX-1080 GPU) goes up to 191ms from a mean of 124ms±14ms.

C. Multi-Camera Tracking

The primary objective of our proposed multi-camera tracking algorithm is to keep updating the positions of all the detected weeds across multiple cameras with the existence of indeterminate classification delays to send the most possible weeds to the treatment layer.

1) Multi-camera weed tracking using EKFs: The proposed multi-camera multi-weed tracking algorithm is essentially a collection of EKFs with a cascade structure, which allows for the sequential update of object locations in Cartesian space. Each EKF in this framework could be seen as a single-camera model-based 3D visual tracking algorithm using edges or keypoints, which is so-called intracamera tracking module. The inter-camera tracking module, in contrast, tries to project out-of-view objects on other camera planes, performing inter-camera template-based matching and eventually passes retraced object from one EKF to another.

Our system employs a pre-calibrated multi-camera system, so that we can have the camera intrinsic matrices K 1 , K 2 , . . . , K N , lens distortion parameter L 1 , L 2 , . . . , L N , and camera extrinsic matrix T 1 2 , T 2 3 , . . . , T N-1 N as the prior. The acquired images are rectified, normalized, and converted into gray-scale before feeding into our tracking layer.

Our proposed EKF algorithm for intra-camera tracking is defined in terms of its states x x x k,i , control inputs u u u k,i , measurements z z z k,i , where i represents the object index, k indicates the discrete time. The state vector and control input for the i th object can be described as

x x x k,i = [P P P k,i , s s s k,i ] T u u u k,i = [ξ ξ ξ k k+1 , ∆s s s k,i ] T ( 1
)
where

P P P k,i = [x k,i , y k,i , z k,i ]
T represents the 3D coordinates of the centroid of i th object in camera frame, ξ ξ ξ k k+1 ∈ se(3) represents the 6DOF camera motion vector from k to k + 1 camera frame in Lie Group, and s s s

k,i = [h k,i , w k,i ] T ∆s s s k,i = [∆h k,i , ∆w k,i
] represent the size and the predicted change of search scope of i th object in template matching procedure, respectively.

The measurement vector z z z k,i of each object can be written as

z z z k,i = [p p p k,i , d k,i ] T (2)
where p p p k,i = [u k,i , v k,i ] T represents the measured 2D coordinates of the centroid of the i th object in image space after template matching, and d k,i represent the measured depth of i th object from sonar system. Following the process and measurement equation in EKF framework, our proposed system can be formulated as following:

x x x k+1,i = f (x x x k,i , u u u k,i ) + w w w k,i = R R R(ξ ξ ξ k k+1 )P P P k,i + t t t(ξ ξ ξ k k+1 ) s s s k,i + ∆s s s k,i + w w w k,i (3)
z z z k,i = h(x x x k,i ) + v v v k,i = π(K K KP P P k,i ) d k,i + v v v k,i (4)
where R R R(ξ ξ ξ ) ∈ SO(3) t t t(ξ ξ ξ ) ∈ R 3 are the rotation matrix and the translation vector, π(•) represents a projection function that p p p k,i = π(P

P P k,i ) = [x k,i /z k,i , y k,i /z k,i ] T , and w w w k,i v v v k,i
are Gaussian process and measurement noise with normal probability distribution, respectively.

For tracking applications, the established object motion model serves as a coarse initialization for later templatebased matching, thus reducing search scope and processing time to achieve fast implementation. For our application, we utilize a simplified VO algorithm, assuming a flat ground surface, to generate ξ ξ ξ k k+1 , so that we can achieve high computational efficiency without compromising tracking accuracy. After propagation, template-based matching is performed for every object in the field of view with a search region defined by a projected center p p p k,i = π(K K KP P P k,i ) and a window size s s s k,i in the EKF state vector. If successful, the corrected template center p p p k,i combined with average depth d k,i measured from the sonar system, are used as new measurements, feeding into EKF correction process; otherwise, the control input set ∆s s s k,i > 0 to be positive to increase the search scope for the next iteration.

If the object is already out of scope from the current camera, the inter-camera tracking passes the disappeared object to next EKF. The inter-camera tracking algorithm is, in essence, a simple template-based matching algorithm. Firstly, the out-of-view object is projected from current camera m to next camera n, which can be described as

p p p n = π(K K K n T T T m n (R(ξ ξ ξ )P P P m + t t t(ξ ξ ξ ))) (5)
where ξ ξ ξ represents the estimated camera motion between two image time-stamp from unsynchronized camera m and n. If the predicted center of object p p p n enters the visible area of camera n, an inter-camera template matching is carried out. If successfully matched, the state vector of this object is removed from the EKFs in camera m and inserted into those from camera n ; otherwise, we increase the search region of this object in the EKF in camera m and try to match it at the next iteration.

It should be noted that we implement both the indirect (keypoints) and direct (edges) method -denomination from [START_REF] Choi | Robust 3d visual tracking using particle filtering on the special euclidean group: A combined approach of keypoint and edge features[END_REF] -in motion propagation and correction for the intraand inter-camera tracking modules. A detailed evaluation and comparison of their performances are presented in Sec. IV-B.

2) Delayed initialization and pose propagation: To deal with the unpredictable delays, a time-stamp indexed frame buffer is preserved in the memory to ease the later image searching and the pose propagation. Each frame in the buffer, indexed by the image capture time, contains the received gray-scale image, the estimated camera pose from the VO, and the average pixel depth from the sonar system. When a classification result arrives at the tracking layer, the corresponding frame is extracted from the buffer given its time-stamp. The 2D appearance-based templates of detected plants are extracted, and the centroid of the object is backprojected on Cartesian space to initialize the state vectors in the first EKF. Then their positions are propagated through the intra-and inter-camera tracking modules given interframe poses in the buffer. When the object has transformed to the latest camera frame and successfully matched to current image template, the distances between the new object to all known ones are calculated. If the shortest distance falls below a certain threshold, these two objects will be considered to be the same, and the state vectors are combined.

3) Biased naive Bayesian classifier: To protect the valuable plants among the weeds, an incremental naive Bayesian classifier with biased probabilistic model is utilized to filter out the false positives from weed detector. Consider conditional probability model P(C i |l l l i ), with sequentially arrived classified label l l l i = l i,1 , l i,2 , . . . , l i,n from detector and C i as the output label from our proposed classifier, the naive Bayesian filter formula can be written as P(C i |l l l i ) ∝ P(l l l i |C i )P(C i ), where P(C i ) is a class prior probability of an object belonging to weed or plant, and P(C i |l l l i ) is conditional probability that is pre-defined with a probability model defined as follows:

P(l l l i , |C i ) l l l i = weed l l l i = plant C i = weed 0.8 0.2 C i = plant 0.1 0.9
It should be noted that the probability model is biased because destroying a valuable plant is always considered to be a much more serious problem than just failing to remove a weed from the user's point of view. We follow a standard incremental implementation of this filter described in [START_REF] Murphy | Naive bayes classifiers[END_REF], and the classification results is finalized when all the classification results have been delivered to our tracking algorithm. At this time, the objects classified as plants are deleted from the corresponding EKF and are not tracked anymore.

D. Control Design

Considering an extrinsic calibrated camera-tool system, we already have the homogeneous transformation matrix from last camera frame to the first row of the stamping tool T T T cam r1 . The i th object position P P P cam,i in camera frame and the estimated velocity v v v cam from VO system can be transformed to the coordinate system of the first-row stamping tool, so that the triggering timing t pred and predicted horizontal displacement x prime r1,i can be calculated by solving the first two rows of the formulated linear system equation:

P P P ′ r1,i = T T T (R R R cam r1 v v v cam t pred + R R R cam r1 v v v cam t delay )T T T cam r1 P P P cam,i (6)
where R R R cam r1 is the rotation matrix extracted from T T T cam r1 , T (•) is the transformation function using 6-DOF motion prediction vector as input, t delay is the measured processing delay, and the P P P ′ r1,i = [x prime r1,i , 0, z prime r1,i , 1] T is the predicted position of object.

It should be noted that the accuracy achieved by our proposed open-loop predictive control strategy can be easily affected by the whole treatment process delay, introduced by PLC latency and execution tool dynamics. With our highperformance PLC, the accumulated processing delay can be simplified as a constant delay parameter t delay , which is measured and validated by experiments in the laboratory.

IV. EXPERIMENTS

The experimental evaluation section is designed to illustrate the performance of our proposed weed control system in various real field conditions with indeterminate delays.

A. Field Robot

The BoniRob is a multi-propose field robot designed by BOSCH DeepField Robotics, which provides a slot to install various tools for specific tasks. In recent years, a variety of agriculture applications have been successfully developed and validated using this robot, such as selective spraying, mechanical weed control, and weed monitoring. In this work, our proposed weed control system is mounted on BoniRob, operating in various real field condition to evaluate our proposed method.

B. Evaluation of Multi-Camera Tracking

The proposed multi-camera tracking performance is extensively evaluated regarding accuracy, robustness, and runtime performance in the real field. We both compare the indirect and direct methods in our proposed framework, as well as evaluate the effect of terrain conditions, the density of plants, and various delay introduced by classification. A simple vegetation detection classifier provides the ground truth centroid positions of the plants.

1) Different terrain condition:

The experiments presented here are designed to analyze the intra-and inter-camera tracking accuracy and robustness quantitatively in various field conditions, where a 0.2m/s vehicle speed and a simple vegetation detection algorithm with a 0.138s average classification delay are chosen. In Fig. 3, a detailed comparison between the direct and indirect tracking method in four typical terrain conditions is presented.

From the comparison of the two methods in our proposed tracking framework, we can generalize that (1) the direct method shows marginally better tracking precision in motion propagation phase than that of indirect method in all terrain conditions, (2) the indirect method holds a larger convergence radius, with a similar tracking accuracy, compared with direct approach in the context of templatebased matching, (3) both methods works slightly better in intra-camera tracking than that in inter-camera tracking even though the lighting condition is fully controlled, and (4) the overall tracking accuracy is mainly limited by the template matching precision rather than that of motion propagation, which can better explain why all VO algorithms showed similar tracking precision in the end in Sec. ??. It should be clarified that keeping motion propagation error small is still critical for system robustness since it is tightly related with the tracking error after a blind area between cameras, which define the initial displacement of template matching.

Analyzing the effect of the terrain roughness and the density of plants on our tracking framework, we can observe that the motion propagation precision and convergence radius of template matching is correlated with the number of good features or edge pixels in image, where both the rough terrain and the plants can provide such features and edges for motion estimation and template matching. In contrast, the degree of ground flatness plays a minor role in tracking with the downlooking camera setup.

2) Different classification delay: In this paper, one of our major claims is that our system could work with a much higher range of classification delays without compromising overall tracking accuracy and efficiency compared with previously proposed weed control systems. To support our claim, the effect of classification delays on our proposed tracking system are studied in term of the tracking accuracy and runtime performance. In this evaluation, a simple vegetation detection algorithm with a fake classification delay generator is utilized to examine our tracking performance. The vehicle speed for this test is set to be 0.1m/s, thus allowing to test classification with more significant delays. In Fig. 4, we can find that the higher classification delay would request higher computational effort, which is caused by the template matching for every three propagation to ensure the convergence, but wouldn't compromise the overall tracking precision.

3) Different vehicle speed: To examine the system capabilities, the tracking accuracy, both before and after template matching, are evaluated in different vehicle velocities in the real field, with the simple vegetation detection algorithm. It should be noted that the maximum allowed speed could only reach up to 0.4m/s for safety reasons, where the results for higher velocities are simulated by sampling recorded images to study the limitation of our proposed system. In Fig. 4, the motion propagation error increases exponentially after a certain speed, which is attributed to the reduced overlap area between sequential images. However, the tracking error after

C. Evaluation of Predictive Control

The proposed predictive control module is evaluated on both the mechanical weeding tool and the selective sprayer in a real-world field with flat and rough terrain areas. The simple vegetation detection algorithm utilized to generate classification results.

For tests of the mechanical weed removal, real leaves with an average radius of 10mm are chosen as targets, and we manually count the successful stamping rate after execution as the performance metric. The stamping evaluation is performed in short paths to control the test condition in an outdoor environment better. For short-path tests, we use 5-10 For experiments of selective spraying, we set up a webcam to monitor targets after spraying, taking advantage of the wet area is visible on the dry field surface. Due to this simple monitoring approach, the evaluation of selective spraying can be done with a full-row operation by counting the successful execution rate manually afterward from the recorded video. The example pictures of the test field and targets after successful treatment are presented in Fig. 5, and the quantitative experimental results are provided in Table I. The successful treatment rates regarding different vehicle speeds and terrain roughness are presented in the table, where the leaves with an average 10mm radius are chosen as targets. We don't have the experimental evaluation of mechanical weeding with higher speed than 0.2 m/s because the higher driving speed of the vehicle may introduce a huge tangential force on the stamper, which may lead to the malfunction of the stamping tool. From the table, we can observe that the successful treatment rate is almost invariant with speed in both flat and

D. Evaluation of In-Row Weed Control

The purpose of the in-row weed control evaluation is to analyze the whole system performance, especially the classification, and final treatment rates. The overall performance evaluation, named in-row weed control, is performed in the real field and simulated environment. The reason why we introduce the extra experiment in a simulated environment is that (1) the extensive test on the real field with mechanical tool is not very welcome by farmers due to the potential failure to destroy of the plants, [START_REF] Lottes | Joint stem detection and crop-weed classification for plant-specific treatment in precision farming[END_REF] after some trials we found that the stamping holes after real field weeding are not really visible on leaves and cannot easily be distinguished with other holes on ground, and (3) this extra-test on flat terrain at the same time tests our classification results and provides us with a baseline performance, which can help us to better understand the limitation of our proposed system.

The example pictures for in-row weeding test condition, both in the real field and simulated environment, with representative results, are shown in Fig. 6. The quantitative evaluation of the classification rate from the detector, Bayesian classifier, and tracker, as well as the successful treatment rate, from each layer of the proposed system, is recorded in Table II. The classification rate from the detector, Bayesian classifier, and multi-camera tracker, as well as the final treatment rates, regarding two typically set vehicle speed, are presented in the table, where the real weeds serve as targets in the real field, and the leaves serve as fake targets in simulated tests. From the table, we can find that our proposed Bayesian filter could effectively remove the false positives from the detector, and the proposed weed control system that integrates classifier, tracker and weeding machine could perform reliable treatment for in-row weeds with good precision. From the real field tests, we observe that the stamping solution for weed removal is, in general, less efficient than using the selective sprayer, which can be attributed to their hard-to-locate stem positions and the hardto-compensate uncertainty induced by the vibration while driving on the unstructured terrain. V. CONCLUSION A novel computer vision based weeding control system is designed, implemented and evaluated. A non-overlapping multi-camera system is introduced to compensate the indeterminate classification delays induced by modern CNNbased detection algorithms, and a 3D multi-camera multiobject 3D tracking algorithm using EKF is developed to provide high-precision tracking results across cameras. In our proposed framework, the direct or indirect template matching methods are implemented and compared to justify our system design. A biased naive Bayesian filter is designed to remove the false positives from the detector. To adopt an operation-while-driving strategy, both low-and high-level control strategies are deliberately designed for high-precision weed removal. The tracking and control performance of the proposed system is extensively evaluated in different terrain conditions regarding various classification delays and vehicle speeds, and the final in-row weed removal performance is also assessed to validate our claim that our system can provide accurate and reliable in-row weed removal service in the real field.

Fig. 3 :

 3 Fig. 3: Left: Sample images acquired in four typical real field condition and their keypoints and edges extraction results are presented. Middle: the image space tracking error with and without template matching correction, using direct and indirect methods, are plotted versus the tracked frame number. Right: the intra-and inter-camera tracking error using direct and indirect approaches are plotted versus initial displacement of the reference template to study the convergence radius of template matching.

Fig. 4 :

 4 Fig. 4: Left: the MSE [pixel] and processing time [ms] versus various simulated classification delays [s] in the delayed initialization phase are plotted, where the template matching is performed after every three propagation to maintain the realtime performance. Right: the overall tracking MSE before and after template matching versus vehicle speeds are plotted to study the effect of vehicle speed on our proposed tracking system.

Fig. 5 :

 5 Fig. 5: Left: sample pictures of leaves after mechanical weed removal. Right: sample pictures after selective spraying.

Fig. 6 :

 6 Fig. 6: Left: an in-row mechanical weeding and spraying test is conducted in the real-world field in Ancona, IT. The weeds after spraying can be seen in the picture, and the mechanical treatment results are not visible on the ground. Right: the performance of in-row mechanical weeding is presented in a simulated test environment in Bonn, GE. The faked targets made of leaves attached to modeling clay is utilized to visualize the execution results.

TABLE I :

 I Successful Treatment Rates

	vel. [m/s]	0.05	0.1	0.2	0.3	0.4
				stamping
	flat	112/113	99/102	90/91	-	-
	rough	120/121	84/86	93/96	-	-
				spraying
	flat	125/125	124/124	129/129	132/134	112/113
	rough	121/121	111/112	118/119	131/133	129/129

TABLE II :

 II The Weed Classification and Treatment Rate

	module	detection b. classifier	tracker	treatment
			spraying in real field
	0.1m/s	199/182	182/182	182/182	182/182
	0.2m/s	193/182	181/182	181/182	181/182
			stamping in simulated test
	0.1m/s	92/58	57/58	57/58	57/58
	0.2m/s	88/58	57/58	57/58	57/58
	rough field ground.

ACKNOWLEDGMENTS

This work has partly been supported by the European Commission under the grant number H2020-ICT-644227-FLOURISH. We furthermore thank the people from BOSCH Corporate Research and DeepField Robotics for their support during joint experiments.


Xiaolong Wu

Cédric Pradalier

Illumination Robust Monocular Direct Visual Odometry for Outdoor Environment Mapping

Vision-based localization and mapping in outdoor environments is still a challenging issue, which requests significant robustness against various unpredictable illumination changes. In this paper, an illumination-robust direct monocular SLAM system that focuses on modeling outdoor scenery is presented. To deal with global and local lighting changes, such as solar flares, the state-of-art illumination invariant photometric costs for RGB-D and stereo SLAM systems are revisited in the context of their monocular counterpart, where the camera motion and scene structure are jointly optimized with a reasonably poor initialization. Based on our analysis, a combined cost is proposed to achieve a high-precision motion estimation with an improved convergence radius. The proposed system is extensively evaluated on the synthetic and real-world datasets regarding accuracy, robustness, and processing time, where our approach outperforms systems with other costs and state-of-art DSO and ORBSLAM2 systems.

I. INTRODUCTION

In recent decades, real-time Visual Odometry (VO) and Simultaneous Localization and Mapping (SLAM) systems have shown their full potentials to assist various robotic applications in outdoor operation -from autonomous driving in the urban scene to environmental monitoring in the natural environment. In large-scale mapping applications, the monocular VO and SLAM systems have gained significant popularity due to the simple calibration procedure and their flexibility introduced by the scale ambiguity, which allows for the seamlessly switching between differently scaled scenes.

Among monocular VO and SLAM systems, the indirect method [START_REF] Klein | Parallel tracking and mapping for small ar workspaces[END_REF] [2] [START_REF] Mur-Artal | Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras[END_REF] has dominated the research field for a long time since the self-recognized features can provide considerable robustness to both the photometric noise and geometric distortion in images. However, recent works have shown that the direct method [START_REF] Newcombe | Dtam: Dense tracking and mapping in real-time[END_REF] [START_REF] Pizzoli | Remode: Probabilistic, monocular dense reconstruction in real time[END_REF] [6] [START_REF] Engel | Direct sparse odometry[END_REF] could provide more accurate and robust motion estimation due to their high flexibility of image information usage in comparison of the indirect approaches using certain types of features. In general, the direct VO and SLAM algorithms compare the intensity value of pixels over a local patch across images by making a brightness consistency assumption. However, it is impossible to control the illumination perfectly in realworld applications, especially in outdoor environments, thus resulting in performance degradation.

In our previous works [START_REF] Chahine | Survey of monocular slam algorithms in natural environments[END_REF], we have observed that one of the primary sources of the tracking failures in outdoor environment stems from the solar glares in acquired images. The *All authors are with the UMI2958 GeorgiaTech-CNRS, Metz, France.

firstname.lastname@georgiatech-metz.fr affected pixels would either dominate the motion estimation resulting in false trajectory or be treated as outliers resulting in a lack of points in optimization, either of which could devastate the system robustness and enforce a restart. In this work, we propose to treat the solar glares as local and/or global lighting changes and utilizing the illumination-robust method to overcome the described difficulty. Many illumination invariant algorithms have been developed to model the global and local lighting changes or alleviate their adverse effect in motion estimation. However, most of them are merely evaluated for RGB-D or Stereo systems, where the scene structure is usually well initialized. In this paper, we revisit the illumination-invariant costs that have shown excellent performance in indoor evaluations in [START_REF] Park | Illumination change robustness in direct visual slam[END_REF], and we evaluate their performance in the monocular joint optimization framework [START_REF] Engel | Direct sparse odometry[END_REF], typically with a poor environmental structure initialization, using the vKITTI dataset [START_REF] Gaidon | Virtual worlds as proxy for multi-object tracking analysis[END_REF] with simulated global and/or local lighting changes.

Based on the analysis, we propose a combined cost that adaptively weights a global affine model-based cost and a gradient-based cost in different optimization phases to achieve high-precision motion estimation, while preserving a large convergence basin. The proposed system is extensively evaluated regarding accuracy, robustness, and runtime performance using synthetic vKITTI dataset, real-world DEVON Island dataset, and Symphony Lake dataset against various global and local lighting changes, in Fig. 1. Besides, a qualitative analysis of the robustness of our proposed method against solar glares is studied to support our claim further.

This paper makes the following contributions: 1) To our knowledge, this is the first paper evaluating illuminationrobust costs in the context of a monocular joint optimization framework, considering a large initial inverse depth error; 2) a novel illumination-robust monocular direct VO system is proposed and evaluated against the real-world outdoor lighting changes; 3) the solar glares in the outdoor images are treated as local illumination change, and its adverse effects on motion estimation are proven to be alleviated by implementing global and local illumination invariant costs.

II. RELATED WORK

The so-called direct monocular VO and SLAM has shown exceptional performance in ego-motion estimation and scene structure mapping. As the state-of-art direct VO system, the DSO [START_REF] Engel | Direct sparse odometry[END_REF] utilizes a fully direct formulation that jointly optimizes the camera pose and scene structure in a sliding window, while taking advantage of explicit photometric calibration [START_REF] Engel | A photometrically calibrated benchmark for monocular visual odometry[END_REF] and exposure compensation strategy to further improve the tracking accuracy and robustness. However, this approach makes the static scene lighting assumption, so that is still sensitive to the external lighting changes.

In recent years, a number of illumination-robust VO and SLAM systems have been proposed, implementing various models or descriptors to alleviate the adverse effect of external lighting changes, so as to achieve robust motion estimation in challenging indoor and outdoor environments. To gain robustness against the global illumination changes, either the median value of pixel residuals [START_REF] Meilland | Realtime dense visual tracking under large lighting variations[END_REF] [14] [START_REF] Bloesch | Robust visual inertial odometry using a direct ekf-based approach[END_REF] [16] or an affine brightness transfer function [17] [18] is estimated to compensate the induced adverse effect in the optimization. For local lighting changes, [START_REF] Dai | Bundlefusion: Real-time globally consistent 3d reconstruction using on-thefly surface reintegration[END_REF] propose to use image gradients, rather than pixel intensities, to formulate the direct energy function, thus gaining local lighting invariance; [START_REF] Crivellaro | Robust 3d tracking with descriptor fields[END_REF] relies on dense computation of a deliberately designed local descriptor to obtain a clear global minimum in energy function while preserving convergence basin by convolving with a low-pass filter; the methods based on the census transform [START_REF] Alismail | Direct visual odometry using bit-planes[END_REF] [22] use a binary descriptor to achieve local illumination invariance during the motion estimation.

A thorough evaluation regarding accuracy and robustness in the context of visual odometry is conducted for all described methods above in [START_REF] Park | Illumination change robustness in direct visual slam[END_REF]. The analysis results suggest that the gradient-based method [START_REF] Dai | Bundlefusion: Real-time globally consistent 3d reconstruction using on-thefly surface reintegration[END_REF] and census-transformbased method [START_REF] Park | Illumination change robustness in direct visual slam[END_REF] show state-of-the-art performances. However, all the methods and their evaluations are conducted with RGB-D and stereo setup, where the depth information is provided with some precision.

Mutual information (MI) is a global metric which can be used to register images with huge appearance differences. NID-SLAM [START_REF] Pascoe | Nidslam: Robust monocular slam using normalised information distance[END_REF] incorporates a whole-image MI metric, named normalized information distance, into a monocular SLAM framework to achieve illumination-robust motion estimation in real-time on a high-performance GPU.

Illumination invariant imaging provides an alternative solution to our problem. In [START_REF] Maddern | Illumination invariant imaging: Applications in robust vision-based localisation, mapping and classification for autonomous vehicles[END_REF], the illumination invariant image transform is developed to remove the variation from lighting changes in the acquired RGB images. This improves the performance and robustness of localization and mapping.

III. ILLUMINATION ROBUST MONOCULAR DIRECT VISUAL ODOMETRY

In this section, the mathematical representation of the direct formulation is described in Sec. III-A, and the joint optimization algorithm utilized in this paper is presented in Sec. III-B. Most importantly, the state-of-art illumination invariant costs are described as plug-ins of the basic cost function in Sec. III-C, and our proposed cost is then derived in Sec. III-D.

A. Direct Formulation

Consider a reference frame, a gray-scale reference image I r : Ω → R and an inverse depth map D r : Ω → R + are included, where Ω ⊂ R 2 is the image domain. A 3D scene point x x x = (x, y, z) T of a pixel is parameterized by its inverse depth d = z -1 in the reference frame instead of the conventional 3 unknowns. Defining a 3D projective warp function π(x x x) = (x/z, y/z) T , a pixel u u u = (u, v) T ∈ Ω can be back-projected into 3D world as x x x = π -1 (p p p, d) = K K K -1 p p p/d, where p p p = (u, v, 1) T is the homogeneous coordinate of such pixel and K K K is the pre-calibrated camera intrinsic matrix.

Given a 3D rigid body transformation G G G ∈ SE(3) from the reference frame to frame i can be written as:

G G G ir = R R R ir t t t ir 0 0 0 1 (1)
where R R R ir ∈ SO(3) and t t t ir ∈ R 3 are the 3D rigid body rotation and translation from reference frame to frame i, respectively.

To better explain all costs involved in this paper, the pixelwise direct error E k of the k th pixel between reference frame and the i th frame can be generally written as:

E k := ∑ p p p∈S p p p w p p p F i (π(p p p ′ ) -F r (π(p p p)) γ ( 2
)
p p p ′ = R R R ir π -1 (p p p, D r (p p p)) + t t t ir (3)
where F(•) represents some representation calculated from original Image I, such as intensity, gradient or some descriptors. The set S p p p is the set of pixels in a pre-defined local patch in the reference frame, w p p p is the weight assigned for each pixel, • γ is the Huber norm, and the subscript k ∈ S pixel is the index of a sampled pixel selected from a selection algorithm.

B. Joint Optimization

A sliding window optimization framework using the Gauss-Newton algorithm, described in [START_REF] Engel | Direct sparse odometry[END_REF], is utilized to achieve the real-time motion estimation and 3D structure mapping. The optimization problem, in Eqn. 2, is furthered reduced to solve a nonlinear least-square minimization problem on Lie-manifolds. The corresponding Lie Group component ξ ξ ξ ∈ se(3) is introduced to represent the 6-DoF camera pose, where this element can be mapped to G G G ∈ SE(3) through the exponential mapping as:

G G G = exp se(3) (ξ ξ ξ ) (4)
and the update rule in Lie Manifold can be performed through logarithm and exponential mapping as:

ξ ξ ξ ik = log se(3) (exp se(3) (ξ ξ ξ i j ) • exp se(3) (ξ ξ ξ jk )) (5)
For monocular VO algorithms, there are typically two phases performing the optimization: tracking phase and reconstruction phase. In the tracking phase, the inter-frame camera pose G G G ir is estimated given the depth map D r as a prior. In the reconstruction phase, both the depth map D r and the camera pose G G G ir are jointly optimized to improve the overall performance. To boost the tracking robustness over large camera motion, a minimization over an image pyramid, named coarse-to-fine approach in [START_REF] Engel | Lsd-slam: Large-scale direct monocular slam[END_REF], is utilized to achieve a good trade-off between precision and speed.

C. Illumination Robust Cost

Instead of presenting all real-time capable illuminationinvariant formulations suitable for the joint optimization framework, we merely describe the intensity-based method as a baseline approach and the ones that achieved excellent performances in [START_REF] Park | Illumination change robustness in direct visual slam[END_REF]. In Sec. III-C.1, the intensity values of the grayscale image is utilized to serve as a baseline method; in Sec. III-C.2, a global affine model is estimated to compensate the global lighting changes; in Sec. III-C.3, the gradient magnitude is used to formulate energy function thus achieving invariance to local illumination change; In Sec. III-C.4, the local descriptor from [START_REF] Alismail | Direct visual odometry in low light using binary descriptors[END_REF], named Census Transform or bit-plane, is presented to achieve local illumination invariance. Ultimately, an adaptively combined cost, using both an intensity-based energy with a global affine model and gradient-based energy, is proposed to achieve the invariance against global and local lighting changes without compromising convergence robustness.

1) Intensity: For standard implementation [START_REF] Engel | Direct sparse odometry[END_REF], we model the unknown function F(•) as an image function mapping 2D pixel coordinate to pixel intensity values obtained by bilinear interpolation, where a mathematical expression can be written as:

F Int (•) := I(•) (6)
2) Global Affine Model: The global affine model in [17] [18] can be plugged into the basic direct formulation, in Eqn. 2, to compensate for the additive and multiplicative global lighting or exposure changes, where the pixel-wise energy function can be modified as:

E k,GA f f := ∑ p p p∈S p p p w p p p I i (π(p p p ′ )) -β i - e α i e α r I r (π(p p p)) -β r γ (7)
where α i,r and β i,r are global illumination affine model parameters, which are jointly optimized at each iteration. Combined with Huber norm, this affine method can work well in an environment without substantial local illumination changes.

3) Gradient Magnitude: In [START_REF] Dai | Bundlefusion: Real-time globally consistent 3d reconstruction using on-thefly surface reintegration[END_REF], the gradient magnitudes are utilized, instead of intensities, to formulate unknown function F(•) with a mathematical expression as:

F Grad (•) := ∇I(•) 2 (8)
where ∇I(•) calculates the gradient vector from the image intensities around a given pixel. The gradient method has proven to be robust with the local lighting changes and could achieve state-of-art tracking precision for RGB-D and Stereo applications.

4) Census Transform:

The Census Transform [START_REF] Alismail | Direct visual odometry using bit-planes[END_REF] [22] is a local binary descriptor that compares a pixel intensity with its neighborhoods and results in one-bit results indicating the neighbors are lighter or darker than this given pixel. The results are then packed to formulate the bit-plane descriptor, which is the F(•) in this work. Considering a N × N local patch is utilized to calculate the descriptor, where the i th descriptor can be written as:

F i,CT (•) := 1, if I(•) > I(N i (•)) 0, otherwise (9)
where N i represent the i th neighborhood of the given pixel. It should be noted that the choice of the comparison operator is flexible, which could be >, ≤, <, or ≥, but it must be consistent for all pixels of interest. The gradient of the descriptor is approximated by using its image gradient for Jacobian computation in optimization.

D. Our Proposed Method

From the analysis of tracking accuracy and the convergence radius in Sec. IV-B, we found that the gradient method showed maximum robustness against various simulated lighting changes over other methods. However, it has a smaller convergence radius compared with the affine approach.

Motivated by descriptor field method [START_REF] Crivellaro | Robust 3d tracking with descriptor fields[END_REF], we propose to combine the energy functions of affine, and gradient approaches to obtain a clear global minimum while preserving large convergence basin through an adaptive weighting strategy. Specifically, we adaptively assign weights to affine and gradient costs at the different stage of the optimization: more weights put on the affine component at the beginning of optimization and more on the gradient component at the end. Mathematically, the proposed pixel-wise energy can be expressed as: where the 2 √ 2 is the factor that scales the gradient magnitude to be the same range with intensity, and the w k is the adaptive weight that starts from 0 and ends up with 1, which can be defined differently in the tracking phase w k,1 and reconstruction phase w k,2 as:

E k := (1 -w k )E k,GA f f + 1 2 √ 2 w k E k,Grad (10)
w k,1 := 1 tr(Σ Σ Σ ξ ) w k,2 := 1 σ 2 d -1 k (11)
where tr(•) represents the trace operator, Σ Σ Σ ξ is the covariance matrix of pose estimates, and σ 2 d -1 k is the variance of inverse depth of the pixel from joint optimization.

In the tracking phase, the inverse depth of pixel is set constant, the camera pose is the only parameter to be optimized. In the reconstruction phase, the camera poses are already well-initialized, but the inverse depth estimates could still have large errors. As a result, the uncertainty based weights, w k,1 w k,2 , could force the estimate with a huge amount of uncertainty to have a large convergence basin and guide the determinate estimate to arrive at the global minimum, thus improving the tracking precision and robustness. The formulated pixel-wise energy function is analyzed in four typical lighting conditions in Fig. 2. In the comparison of all other methods, our proposed method holds nice cost functions in all conditions: a clear global minimum and large convergence basin.

IV. EVALUATION

In this section, the dataset utilized in our evaluation is first introduced in Sec. IV-A. The tracking accuracy and convergence radius during tracking phase are quantitatively evaluated and compared for each described methods in Sec. IV-B, and their overall system performance analysis regarding tracking accuracy, robustness, and runtime properties are presented in Sec. IV-C. Besides, a qualitative evaluation is performed to study the effect of solar glare on our proposed system to further support our claims in Sec. IV-D.

A. Dataset

All the evaluations, described in this paper, are performed on both publicly available urban-scene dataset: Virtual KITTI Dataset [START_REF] Gaidon | Virtual worlds as proxy for multi-object tracking analysis[END_REF], as well as two challenging natural environment datasets: Devon Island Rover Navigation Dataset [START_REF] Furgale | The devon island rover navigation dataset[END_REF] and Symphony Lake Dataset [START_REF] Griffith | Symphony lake dataset[END_REF].

The vKITTI Dataset is a photo-realistic synthetic video dataset, where both the ground truth depth maps and poses are available. We perform the tracking accuracy and convergence radius test using such dataset with the simulated global or/and local illumination changes in Sec. IV-B, and quantitatively compare all described methods concerning tracking accuracy and robustness in Sec. IV-C.

The Devon Island Rover Navigation Dataset, where a rover platform travels ten kilometers at Devon Island, is famous for rocky canyons and sandy terrains, with a forwardlooking narrow field of view camera mounted on-board. Unlike vKITTI Dataset collecting data at the high frame rate (10 Hz), Devon Island dataset record images at 2 Hz, which is essentially more difficult for tracking tasks due to the potential large motion variation and the unexpected natural lighting condition changes. The solar glares slightly contaminate this dataset, and our evaluation is mainly conducted using these sequences regarding tracking accuracy and robustness. A pair of Magellan ProMark3 GPS units were utilized to produce differential GPS data for ground truth position. The quantitative comparison of all methods described in this paper, regarding tracking accuracy and robustness, is presented in Sec. IV-C.

The Symphony Lake Dataset consists of millions of natural lakeshore images at a resolution of 704 x 480 pixels captured by an Axis pan-tilt-zoom camera mounted on a boat with an auto-exposure strategy. The images captured in this dataset are abundant with sun glares in Fig. 4, which makes it suitable for our evaluation. However, the ground truth poses are not provided, so that merely the robustness properties can be evaluated in Sec. IV-C. Due to its abundance of solar glares, the qualitative evaluation of the VO performance under different degrees of solar glares are studied to support our claims in Sec. IV-D.

B. Tracking Accuracy and Convergence Radius

The tracking accuracy and the convergence radius, in the tracking phase, are quantitatively evaluated using synthetic vKITTI dataset, where both ground truth pose and depth map are provided. The images are pre-processed to add the global illumination bias or/and solar-glare patterned local illumination changes with the value ranging from -20 to +20. To prepare the inverse depth map as initialization, the ground truth depths are first mapped to inverse depths and normalized to average 1.0, and the corresponding scale is calculated by comparison of average inverse depth before and after normalization. The ground truth pose is then scaled base on this computed scale. A generated Gaussian noise is added to their ground truth values, and the pose initialization is set by moving the camera with specific displacement in a random direction. The absolute value of this metric is meaningless, but it enables the relative comparisons between methods. As the result, there is no absolute metric is defined in Fig. 3. We examine the tracking accuracy and robustness by merely analyzing the translational drift of pose estimates. A gradient-based pixel selection strategy with 3000 pixels of interest, 5 minimum and 10 maximum iterations are set to all tested systems to facilitate a fair comparison. In Fig. 3, we can observe that (1) the intensity method is only working correctly in const setting with the largest convergence radius; (2) the affine method can provide reasonable tracking precision up to global illumination changes with good convergence basin; (3) both the gradient and census methods can perform reliable tracking in all kinds of conditions but with a considerably smaller convergence radius, and gradient method shows better precision than that of census; (4) Our proposed method outperforms all other methods, which exhibits a similar convergence radius to the affine method and works correctly for all simulated lighting conditions.

It should be noted that this test is merely optimizing the inter-frame poses rather than the joint optimization of pose and scene structure in the reconstruction phase. This test aims to study and compare the performances of all described methods concerning poor initialization in the tracking phase. A more thorough evaluation of the whole system performance including the joint optimization will be provided in the next subsection.

C. Whole System Performance Evaluation

The whole system performance is assessed regarding the overall tracking accuracy, robustness, and runtime performance. For vKITTI and Devon Island dataset, the ground truth positions are provided, so we choose the translational drift and the average failure rates as metric for our evaluations. For Symphony Lake dataset, where no ground truth poses are available, we therefore merely count the average failure rate. It should be noted that the failure rate is either self-detected as tracking lost event or automatically detected as a substantial abnormal movement -five times larger than average displacement.

To facilitate a fair comparison, all critical parameters are set to be the same: a gradient-based pixel selection strategy, 2000 pixels of interest, 2 minimum and 6 maximum iterations. Since the precision of pose estimation in the initialization phase is lower in average, all pose estimates during initialization are not involved in our final evaluation. We run every test 10 times to calculate the average, and we do not consider loop-closure events. The scales drifts are recovered by comparing estimated camera translations with ground truth translations for every 100 frames.

In Fig. 4, the qualitative results generated from our proposed system and the challenging images in the datasets are presented. A quantitative evaluation concerning translatonal drift, failure rate and average runtime between each described methods, as well as state-of-art ORBSLAM2 [START_REF] Mur-Artal | Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras[END_REF], are presented and compared in Table . I.

Overall, the joint optimization faces a much server robustness issue than that of visual tracking in Sec. IV-B, which is suspected to be the side effect of the joint optimization. The overall robustness of side-looking camera sequence is better than that of forward-looking cameras, however, without a comparison of tracking accuracy we cannot generate more conclusion. Specifically for each method, the intensity method achieves a zero failure rate for the original synthetic dataset, but soon faces severe divergence under lighting changes. The affine methods show good robustness but less accurate motion estimation in most dataset sequences, while the census and gradient present attractive tracking accuracy but diverges easily. Our proposed method, combining the sound characteristics of affine and gradient methods, provides the most reliable tracking estimates in all tested datasets. However, the computational load of our system is higher than that of other methods except for census, which is primarily attributed to the doubled residual and Jacobian computational load induced by our combined cost.

It should be noted that the missing entries for the vKITTI dataset indicate that the corresponding method continuously fails at the initialization phase, and the missing run-time performance of orbslam2 is because it is a GPU implemented algorithm, but all other methods are CPU implemented..

D. Resistance to Solar Glare

In this paper, one of our major claims is that the solar glares can be modeled as global+local illumination changes, and our proposed system can operate in this situation without losing the track. To further support our claim, a qualitative analysis of the effect of solar flares on our motion estimation task is conducted. In Fig. 5, the reconstructed inverse depth map can be seen as an indicator about the tracking performance. Our proposed methods can generate well-spread inverse depth maps even facing strong solar glares, while the affine method tends to produce polarized depth map with sun glares nearby (red), which can be seen

Fig. 1 .

 1 Fig. 1. Local and global lighting changes are abundant in Symphony Lake Dataset [9]. The example tracking trajectory and 3D reconstruction using our proposed method is presented in (a), and two pairs of example images with global illumination changes in (b) and two pairs with local+global illumination changes in (c) are illustrated.

Fig. 2 .

 2 Fig. 2. The pixel-wise energy function of each described method is plotted against the pixel depth in four lighting conditions: (a) no illumination changes, (b) local illumination change, (c) global illumination change, and (d) local and global illumination change.

Fig. 3 .

 3 Fig. 3. Left: the example images with simulated illumination changes are provided in four lighting conditions: no lighting changes, global lighting changes, local lighting changes, and global+local lighting changes. Right: the tracking errors are plotted versus initial pose displacement (x-axis) and maximum inverse depth error (y-axis) for each described methods in different conditions.

  (j) Example images from Devan Island Dataset (k) Example images from Symphony Lake Dataset (a) vKITTI seq0001 with simulated illumination changes (

Fig. 4 .

 4 Fig. 4. The qualitative results of tracking and 3D scene reconstruction are presented utilizing synthetic vKITTI Dataset in (a), real-world Devon Island Dataset in (b) (c), and our Symphony Lake Dataset in (d) -(i). The example images from Devan Island Dataset in (j) and Symphony Lake Dataset in (k) are provided, all of which suffer from the global+local illumination changes induced by solar glare.

ACKNOWLEDGMENT

This work has partly been supported by the European Commission under the grant number H2020-ICT-644227-FLOURISH.

A collection of images with gradually stronger (left to right) sun glare are chosen to test our system robustness. The reconstructed inverse depth maps using global affine model and our proposed method are presented, where the failure ones are marked with a big 'X'.

as a strong sign of tracking failure when the solar glare gets strong. Our proposed algorithm is tested to be able to reliably reconstructed observed scene even for strong solar glare cases. The only failure case we observed is that the sun glare appears at the same time with a swan close to the camera, which is a significantly more difficult issue.

V. CONCLUSIONS

State-of-art illumination-robust costs are evaluated in the context of a monocular joint optimization framework using a synthetic dataset with simulated light changes. Based on our analysis, a robust monocular VO approach is developed by combining intensity-and gradient-based costs with an adaptive weight. The proposed algorithm is extensively evaluated using real-world datasets regarding tracking accuracy, robustness, and runtime performance, which has shown the superiority of our proposed formulation relative to other methods. We further present a brief evaluation about the mapping performance using images affected by solar glare, which illustrates that the sun glare can be modeled as local illumination changes and its adverse effect on motion estimation can be alleviated by introducing an algorithm robust to such local changes. However, the computational cost of our system is higher than that of other methods, which is primarily attributed to the doubled residual and Jacobian computational load induced by our combined cost.