J.-R Chardonnet

A De Carvalho

J.-C Léon

M.-P Cani

Hand Navigator: Experimenting hand navigation in desktop virtual reality

Keywords: (according to ACM CCS): B, 4, 2 [Input/Output and Data Communications]: Input/Output Devices-Channels and controllers

This demonstration provides experimentation on several versions of the Hand Navigator, a peripheral device allowing to control a virtual hand with fine dexterity. Our devices, as easy to hold and to release as a mouse, integrate various small sensors enabling the simultaneous control of a large number of degrees of freedom. Our demonstration invites users to experience new ways of interaction with virtual environments through relevant examples of manipulation tasks such as playing with deformable objects.

Introduction

Manipulating virtual objects with our hands is a great challenge for the virtual reality community. Several solutions have been proposed, such as data gloves or vision capture systems. Their main drawbacks are: (i) they need a calibration, which is generally time-consuming, (ii) they do not integrate tactile feedback.

One solution for the second point is to implement systems with haptics perception (also referred as active feedback). However, the complexity of the system and its cost increase.

Another solution is to implement passive feedback using a proxy (e.g. a sponge) to fool the proprioceptive senses of the user [LCK * 00]. This solution is cheap with an improvement compared to the device without feedback [START_REF] Insko | Passive Haptics Significantly Enhances Virtual Environments[END_REF].

Our solution relies on passive feedback to provide a device especially fitted to hands-on interaction.

Objectives

Our goal is to develop a device that allows a user to control a virtual hand in a virtual space with his real hand, using a large number of degrees of freedom to achieve fine dexterity motions, while sensing passive feedback. Our device should also be cheap, calibration free and ready to use.

We also want to address ergonomics issues on the device to allow complex movements without generating tiredness for the user. Well chosen sensors must be integrated on the device to achieve a good control of the virtual fingers.

Finally, to integrate our device easily in various applications, a good interface must be implemented, with a well modeled hand and interactions for visual feedback.

Existing version and related issues

A first version of the Hand Navigator, called V2, has been described in [START_REF] Kry | Handnavigator: Hands-on interaction for desktop virtual reality[END_REF] and patented. The motion of the virtual hand was controlled using a SpaceNavigator (http://www.3dconnexion.com ), enhanced with petals and two force sensors for each finger to get passive feedback (see figure 1). This device was cheap and very easy to integrate in a desktop environment. It was however difficult to use and raised two main issues: (i) ergonomics of the physical device and (ii) realization of tasks.

Ergonomics of the physical device

With the V2 prototype, the user faced tiredness as he had to put his hand in an uncomfortable posture. Other issues included a bad interaction of the sensors with the user's nails, a bad force distribution on the device and the presence of a threshold in the force sensors, leading to an interference between the force the user applies to move the fingers and the one the user applies on the SpaceNavigator to move the hand. Thus, the shape of the device and sensor's technologies, especially their mechanical properties, are important elements to be considered.

Realization of tasks

To achieve fine dexterity, specific controls must be implemented that can be velocity-based or position-based. Velocity control was implemented for the V2 prototype, as a task can then be interrupted and re-started anytime. However, there was no limitation on the virtual hand and the virtual camera was not fixed in the virtual hand's frame, thus the user quickly lost the position reference between the virtual and the real hands. Visual feedback and types of control are thus important elements to improve.

Designing new devices

Several prototypes were designed considering the issues described above. We conducted a deep analysis of different sensors technologies. Many different types of sensors were tested with different types of control. We succeeded in integrating various sensors in our devices. In this demonstration, we show two prototypes called respectively V3c and V3b (see figure 1). These two prototypes are coupled with the SpaceNavigator. The V3c integrates a touchpad for the fingers and a trackball for the thumb, whereas the V3b integrates only trackballs. These devices do not require large forces that could interfere with the motion of the hand.

In order to improve dexterity in manipulation tasks, we analysed different possible shapes for the devices, leading to a mouse-shaped design. This shape allows modularity with different types of sensors, and an easy integration in different applications.

We also implemented a C++ API, allowing a user to quickly interface the Hand Navigator with any application.

Demonstration

We propose in our demonstration to perform several manipulation tasks using the Hand Navigator. Users are invited to experience new ways of interaction with virtual worlds, through manipulations of deformable objects such as playing with gum animals or making virtual sculpture with clay (see figure 2). Users will sense passive feedback when contacting objects, allowing them to achieve complex manipulation tasks with fine dexterity.

Extensions

We expect our device to be used in several applications, such as physical simulations, interactive shape manipulation and telemanipulation. In physical simulations, our device could increase the sensation of immersion in virtual worlds. Shape modeling could be made very naturally thanks to hands-on interaction. Lastly, handicaped persons could use our device to control a robotic arm helping them to grasp an object.

Figure 1 :

 1 Figure 1: From left to right: the V2 prototype with petals and force sensors, the V3c prototype with trackballs and the V3b prototype with a touchpad and trackballs.

Figure 2 :

 2 Figure 2: Virtual sculpture using the V3c prototype.

submitted to EUROGRAPHICS 200x.


Adrian Bojko
email: adrian.bojko@cea.fr

Romain Dupont
email: romain.dupont@cea.fr

Mohamed Tamaazousti
email: mohamed.tamaazousti@cea.fr

Hervé Le Borgne
email: herve.le-borgne@cea.fr

-IMPROVING SLAM IN DYNAMIC ENVIRONMENTS

Visual SLAM -Simultaneous Localization and Mapping -in dynamic environments typically relies on identifying and masking image features on moving objects to prevent them from negatively affecting performance. Current approaches are suboptimal: they either fail to mask objects when needed or, on the contrary, mask objects needlessly. Thus, we propose a novel SLAM that learns when masking objects improves its performance in dynamic scenarios. Given a method to segment objects and a SLAM, we give the latter the ability of Temporal Masking, i.e., to infer when certain classes of objects should be masked to maximize any given SLAM metric. We do not make any priors on motion: our method learns to mask moving objects by itself. To prevent high annotations costs, we created an automatic annotation method for self-supervised training. We constructed a new dataset, named ConsInv, which includes challenging real-world dynamic sequences respectively indoors and outdoors. Our method reaches the state of the art on the TUM RGB-D dataset and outperforms it on KITTI and ConsInv datasets.

Introduction

Visual Dynamic SLAM algorithms are camera-based Simultaneous Localization and Mapping algorithms that filter the visual features, based on direct motion detection [START_REF] Cheng | An accurate localization scheme for mobile robots using optical flow in dynamic environments[END_REF][START_REF] Lenz | Sparse scene flow segmentation for moving object detection in urban environments[END_REF] or a combination of semantic segmentation and motion detection [START_REF] Ballester | DOT: Dynamic object tracking for visual SLAM[END_REF][START_REF] Bescos | DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes[END_REF][START_REF] Schorghuber | SLAMANTIC -leveraging semantics to improve VSLAM in dynamic environments[END_REF][START_REF] Xiao | Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment[END_REF][START_REF] Xiao | Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment[END_REF][START_REF] Zhang | A front-end for dense monocular SLAM using a learned outlier mask prior[END_REF]. However, when the motion of a dynamic object is dominant in the image, these approaches may fail due to motion consensus inversion [START_REF] Bojko | Learning to segment dynamic objects using slam outliers[END_REF]. Moreover, approaches that rely on semantic segmentation also tend to mask objects even if they are not moving (e.g., parked vehicles) -this excessive masking causes failures due to lack of features. To address these issues, [START_REF] Schorghuber | SLAMANTIC -leveraging semantics to improve VSLAM in dynamic environments[END_REF] introduces a "dynamic factor" for each 3D point to identify those that should be used for pose estimation. It nevertheless fails under motion consensus inversion, as the instantaneous motion of an object is too difficult to detect geometrically, or when an object moves after being considered static. [START_REF] Ballester | DOT: Dynamic object tracking for visual SLAM[END_REF] computes a camera motion using features that are static according to semantic segmentation but fails if too few features are located on static objects.

Figure 1: Illustration of our results on the TUM RGB-D dataset. a) A Basic SLAM does not mask dynamic objects and consequently drifts. b) A Dynamic SLAM masks all supposedly dynamic objects, even when they are static, and may consequently fail if there are not enough unmasked features. c) We add Temporal Masking to Dynamic SLAM: we mask dynamic objects when appropriate, i.e., when we predict that masking them improves SLAM performance. Our model learns by itself that masking objects in motion is beneficial for the SLAM, while other approaches must make this assumption.

We propose a novel SLAM algorithm that learns when to mask object classes based on its own performance in dynamic environments, in a self-supervised fashion. Unlike existing approaches, our decision process is learned. We infer masking decisions that maximize SLAM performance separately from the SLAM itself: we use only past frames to compute which classes to mask in the current image while not using any SLAM-dependent data -keypoints, reconstructed map, camera pose, etc. This results in an improved robustness in dynamic scenarios, especially in challenging cases. We do not assume that certain classes (e.g., cars) or moving objects have to be masked as our models learn these facts by themselves.

This paper shows the feasibility and interest of this new masking decision paradigm that we name Temporal Masking. Our technical contribution is an automatic annotation method for self-supervised training and the corresponding neural architecture, which relies on recurrent neural networks to take into account possible dependencies with past frames. Given a SLAM algorithm and a method to generate semantic masks, we train a model that decides for every supported semantic class if objects of this class should be masked in the current frame, without any underlying geometrical considerations nor assumptions on what object classes are dynamic. Our approach is agnostic with respect to the choice of SLAM algorithm and semantic segmentation method. Since most effective learning-based approaches are supervised, we prefer such an approach although it requires annotations. To annotate a sequence, we must make a binary decision mask/no_mask for every frame and every semantic class; we call such a set a temporal mask. Beyond the tediousness of the task, it is difficult to make such a choice a priori with regard to the final SLAM performance, even for an expert, and it may be algorithm-dependent. Thus, we propose to create temporal masks automatically. As evaluating all possible combinations of masking decisions is computationally intractable, we uniformly sample a constrained space of all possible temporal masks, then evaluate and aggregate the samples according to their SLAM performance (e.g., accuracy). The complexity of the annotation process decreases from exponential to constant time w.r.t. the number of benchmarked labels per sequence. Our annotation method can use any performance metric, including the classical trajectory accuracy (ATE RMSE) and robustness (Tracking Rate). We propose a new metric that combines both and is consistent with a human user expectations.

We improved a SLAM algorithm to make it robust to the two points of failure of existing methods in difficult dynamic scenarios, by making it decide whether applying a given spatial mask is relevant or not with regard to the final SLAM performance. It is on par or beyond the state of the art on TUM RGB-D and KITTI datasets. It is also beyond it on a new dataset we built, ConsInv, which includes motion consensus inversions and the risk of failure due to excessive masking, rarely present in usual benchmarks. We also report the limits of our approach when the test context is different from the training one.

Related Work

Saputra et al. [START_REF] Risqi | Visual SLAM and structure from motion in dynamic environments: A survey[END_REF] gives an overview of Dynamic SLAM: the general principle is to remove image features on dynamic objects so that the SLAM only uses static features. Current methods detect motion, semantically segment dynamic objects, or do both. Motion-based masking approaches filter features in dynamic regions [START_REF] Cheng | An accurate localization scheme for mobile robots using optical flow in dynamic environments[END_REF][START_REF] Cheng | Improving monocular visual SLAM in dynamic environments: an optical-flow-based approach[END_REF][START_REF] Lenz | Sparse scene flow segmentation for moving object detection in urban environments[END_REF][START_REF] Li | RGB-d SLAM in dynamic environments using static point weighting[END_REF][START_REF] Scona | StaticFusion: Background reconstruction for dense RGB-d SLAM in dynamic environments[END_REF][START_REF] Sun | Improving RGB-d SLAM in dynamic environments: A motion removal approach[END_REF][START_REF] Sun | Motion removal for reliable RGB-d SLAM in dynamic environments[END_REF][START_REF] Xu | MID-fusion: Octree-based object-level multi-instance dynamic SLAM[END_REF], typically with optical flow or clustering approaches. A flaw of such approaches is the underlying prior that most of the image corresponds to static objects. In case of motion consensus inversion, static objects are considered dynamic and vice-versa, making the SLAM map only dynamic objects, thus drifting when the objects move. Such inversions occur when most of the features are on dynamic objects, which may occur when they are too close to the camera. Semantic masking approaches [START_REF] Kaneko | Mask-SLAM: Robust feature-based monocular SLAM by masking using semantic segmentation[END_REF] typically use networks as Mask R-CNN [START_REF] He | Mask r-CNN[END_REF] to filter features on objects of specific classes (e.g., cars). They are unlikely to cause SLAM drifts as semantic segmentation does not rely on motion consensus, unless there are unknown dynamic object classes. However, they often mask objects even if they are not moving (e.g., parked cars) and cause SLAM failure when there are not enough remaining features.

Combined approaches combine semantic and motion masking. [START_REF] Bescos | DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes[END_REF] combines multiview geometry and semantic segmentation to remove features on dynamic objects of known and unknown classes. However, it may remove too many features as it masks all detected objects and fail. Some combined approaches apply semantic masks only when they consider the masked object dynamic. [START_REF] Schorghuber | SLAMANTIC -leveraging semantics to improve VSLAM in dynamic environments[END_REF] classifies features as static/dynamic according to the semantic class of the object they are on and how stable the positions of the corresponding map points are. However, features on a dynamic object may be mistakenly considered static under motion consensus inversion, making the method fail. Moreover, each semantic class is manually assigned a probability of moving, while we learn when an object is likely to move. [START_REF] Ballester | DOT: Dynamic object tracking for visual SLAM[END_REF] computes a camera motion using features that are static according to the semantic segmentation, selects the dynamic features on motionless objects using photometric error and inputs both static and selected dynamic features into the SLAM backend. However, if there are features only on objects of dynamic classes, it will be unable to compute the camera pose and fail. [START_REF] Xiao | Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment[END_REF] is similar to [START_REF] Ballester | DOT: Dynamic object tracking for visual SLAM[END_REF] but uses reprojection errors instead of photometric errors. [START_REF] Barnes | Driven to distraction: Selfsupervised distractor learning for robust monocular visual odometry in urban environments[END_REF][START_REF] Bojko | Learning to segment dynamic objects using slam outliers[END_REF][START_REF] Zhang | A front-end for dense monocular SLAM using a learned outlier mask prior[END_REF] are two-step approaches: they learn to segment dynamic objects at training time and do semantic masking at runtime, so they may suffer from failures due to excessive masking.

Current Visual Dynamic SLAM algorithms are unable to handle both motion consensus inversions and failures due to excessive masking. A key cause is the dependency on instantaneous motion detection, unreliable in difficult scenarios. More generally, the dependence on ad hoc conditions limits generalization to challenging scenarios or unknown contexts. Our approach is different as we infer the effect of masking objects on SLAM performance with temporal masking. We consider the SLAM as a black box with respect to masking decisions; so Dynamic SLAM algorithms that use internal SLAM data for object masking [START_REF] Ballester | DOT: Dynamic object tracking for visual SLAM[END_REF][START_REF] Bescos | DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes[END_REF][START_REF] Schorghuber | SLAMANTIC -leveraging semantics to improve VSLAM in dynamic environments[END_REF][START_REF] Xiao | Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment[END_REF] are incompatible.

We tackle difficult scenarios where existing methods [START_REF] Klein | Parallel tracking and mapping for small AR workspaces[END_REF][START_REF] Mouragnon | Real time localization and 3d reconstruction[END_REF][START_REF] Tamaazousti | Nonlinear refinement of structure from motion reconstruction by taking advantage of a partial knowledge of the environment[END_REF][START_REF] Tamaazousti | Real-time accurate localization in a partially known environment: Application to augmented reality on textureless 3d objects[END_REF][START_REF] Tamaazousti | The constrained slam framework for noninstrumented augmented reality[END_REF] do not work well, including Visual-Inertial SLAM as [START_REF] Campos | Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam[END_REF][START_REF] Coulin | Tightly-coupled magneto-visual-inertial fusion for long term localization in indoor environment[END_REF]. IMUs cannot be used for SLAM or odometry in non-inertial frames of references: airplanes, ships, trains, submarines, or space. Filtering dynamic objects in LiDAR point clouds is not trivial without robust vision -moreover, they cannot be used everywhere (e.g., medical / military contexts). We propose a method that makes camera-only visual SLAM robust in difficult scenarios that are poorly handled today. Our aim is to prevent critical failures (i.e., accidents) without the convenience of IMUs or other sensors. Finally, a better and cleaner vision signal benefits multi-sensor SLAM fusion.

3 Temporal Masking

System Overview

Our approach can be applied to any Dynamic SLAM approach based on keypoint detection then filtering (Fig. 2, top row). This last step usually consists in applying a spatial semantic mask to determine which keypoints to ignore when estimating the camera trajectory. Our approach consists in adding a Temporal Masking Network that decides whether the semantic mask should be applied or not to the current frame, with regard to the past sequence. This network is trained through self-supervision, as detailed in the next section, such that it memorizes for every class the circumstances when masking is beneficial for the SLAM. At runtime, it infers masking decisions for unknown sequences. If it infers that masking a certain class is beneficial to the SLAM, we filter keypoints on the corresponding masked area. We focus on feature-based SLAM as image masking is straightforward, but since our method is SLAM-agnostic, it can be applied to direct SLAM algorithms as [START_REF] Engel | Lsd-slam: large-scale direct monocular slam[END_REF]. The pseudocode of our method is in the supplementary materials. The proposed architecture for the Temporal Masking Network is presented in Fig. 3. Our rationale is that relying on memory instead of geometry makes it possible to overcome the points of failure of current Dynamic SLAMs, unlike geometrical approaches that cannot understand the long-term context of a scene. We designed the network for multi-label classification, in which every output corresponds to a semantic class we may need to mask. It is composed of 1) a spatial representation module that encodes input frames and 2) an LSTM Encoder-Decoder that makes memory-based decisions on which semantic classes to mask. The network takes as input a video sample, computes a spatial representation of every frame of the sample, inputs the computed features into the LSTM Encoder-Decoder, and finally applies a threshold to the result to obtain a per-class binary masking decision for the last frame of the sample. A possible way to compute features is to use a CNN encoder. We train only the LSTM Encoder-Decoder; we describe its architecture in supplementary materials.

Temporal Annotation with Self-Supervision

Training a network for temporal masking requires annotating video sequences accordingly. Let us consider a video sequence of length l and a generator of semantic masks (e.g., Mask R-CNN) segmenting p classes: a temporal mask is a binary matrix of size l × p storing masking decisions (i.e., which classes to mask using the given generator) for every frame of the sequence. We first present the annotation method for a single class to mask. Note that iterating through the full temporal mask space is computationally intractable (up to 2 seq. length masks large), so we cannot exhaustively evaluate all temporal masks. Therefore, we compute temporal masks in three steps for every sequence: 1) Uniform random sampling of q samples from a subset of all possible temporal masks. Sampling from a subset instead of the full space makes the problem computationally tractable. 2) Benchmarking of the sampled temporal masks using a unified metric. 3) Performance-weighted aggregation of sampled temporal masks into a unique mask. Samples that perform well tend to mask objects more appropriately, so the result from the aggregation masks objects precisely when appropriate.

Step 2 is straightforward: for every sample, we run the SLAM applying semantic masks according to the masking decisions in the sample and measure its performance with a unified metric. We detail steps 1) and 3) in the single-class case, then generalize to multiclass.

Random Sampling of Temporal Masks Subspace. We generate a set of basic temporal masks that we will benchmark to know their impact on SLAM performance. Let l the sequence length, k 0 and k 1 the min. required length of resp. contiguous blocks of zeros and blocks of ones in a temporal mask (0 = do not mask, 1 = mask). For instance, if l = 7, k 0 = 2 and k 1 = 3, then the masks 1110000 and 0011100 respect k 0 and k 1 , but 1110111 and 0011000 do not. We uniformly sample masks from the space E(l, k 0 , k 1 ) of all temporal masks of length l that respect the min. lengths k 0 , k 1 . The rationale is that the states of the objects in the scene do not change too quickly: by skipping high-frequency changes (low k 0 , k 1 ), we focus on masks more suited to the scene. If k 0 = k 1 = 1 the sampling is trivial. We explore E with a binary tree where node value/depth ⇐⇒ frame label/index (see Fig. 4).

To keep the problem computationally tractable we cannot compute the whole tree of possible masks. The key is to use a closed-form expression of the number of possible paths (i.e., masks) C(N) under any bifurcating node N. Starting at the root, whenever we reach a node that has two children, we randomly pick a child with the odds of each child proportional to the number of possible masks under it. To compute a closed-form solution for C, we define u n and v n as the number of masks under resp. bifurcating nodes of value 0 and 1 at depth n, reindexed from the end of the tree. Let k := k 0 + k 1 . Then:

u n = 2u n-1 -u n-2 + v n-k-1 v n = 2v n-1 -v n-2 + v n-k-1 (1)
(1) shows that (u n ), (v n ) are linear recurrence relations with constant coefficients and the same characteristic equation: x k+1 -2x k + x k-1 -1 = 0. Thus, we can numerically compute a closed-form solution for (u n ), (v n ), and C(N). Full proof and details in supp. mats.

Sampled Temporal Mask Aggregation. After measuring the performance (i.e., score) of all temporal masks previously sampled, we aggregate them. We sum the differences between all pairs of sampled masks (sampled masks are binary vectors made of 0/1, so their differences are vectors made of -1/0/1) weighted by their score difference. The idea is that a difference in performance is explained by the difference in masking decisions, so the scoreweighted sum is a real vector where high values indicate frames that must be masked and low values frames that must not be masked [START_REF] Smeaton | Coherent segmentation of video into syntactic regions[END_REF]. Let x, y be two sampled temporal masks and s x , s y their respective scores. To consider only significant score differences, let σ a be the absolute noise (below which score differences are meaningless) and σ r the relative noise (the score difference must be at least σ r times the first score of the pair). Then we compute the result vector R:

R = ∑ x,y∈Samples max(0, sgn( s y -s x -max(σ r |s x | , σ a ))) ×(s y -s x )(y -x) (2)
R is a real-valued vector that we normalize in [0, 1] then binarize with thresholds in [0, 1], generating an arbitrary number of masks. We test them and select the best one score-wise. If there are equivalent masks within σ a and σ r , we select the one that masks the most frames.

Generalization to multiple classes. Let a sequence of length l and p classes to mask. In the single-class case, we sample q vectors of length l. In the multiclass case, we sample pq vectors that we join in matrices of size l × p. The benchmark step is unchanged. The aggregation step is the same up to the computation of R, which is now a real-valued matrix. In the single-class case, R is a vector that we binarize by applying thresholds. In the multiclass case, every class (i.e., column) may have a different optimal threshold. Hence, for every class i: 1) We generate a matrix R i by zeroing out all columns other than column i. 2) As in the single-class case, we apply thresholds, generating an arbitrary number of temporal masks, evaluate them and select the best one T i . 3) We select the best temporal mask score-wise. 3) We concatenate columns 1, . . . , p of resp. T 1 , . . . , T p , resulting in a temporal mask where all classes are appropriately masked.

USM: Unified SLAM Metric

ATE RMSE (Absolute Trajectory Error) and sometimes Tracking Rate (% of tracked frames) are two SLAM metrics of interest for our method since they resp. reflect SLAM accuracy and robustness. As our method maximizes a single scalar value, we propose a metric to unify both ATE RMSE and Tracking Rate (TR), USM λ :

USM λ (ATE, TR) = TR × e -λ ×ATE (3)
λ balances ATE RMSE and Tracking Rate while ensuring dimensional consistency. If TR=100% and ATE ≪ 1 λ , then USM λ ∼ 1λ × ATE, which is consistent with the usual ATE RMSE metric. If the system fails early (low TR), the score is penalized proportionally. Thus, the general behavior of our metric corresponds to user expectations.

A practical way to ensure ATE ≪ 1 λ while balancing ATE RMSE/TR is to set λ = 0.1

Avg. dataset ATE RMSE . Another benefit of the USM is to make comparisons between SLAM algorithms easier to interpret even in very difficult scenarios, e.g., comparing (ATE RMSE=1mm, TR=10%) to (ATE RMSE=1cm, TR=100%). Existing approaches that take failures into account like the AUC [START_REF] Joyce | Asynchronous multi-view slam[END_REF] do not balance both metrics. We developed the USM for automatic data annotation. It is, however, practical for comparing SLAM algorithms: while multi-dimensional comparison is difficult, USM is a scalar whose balance between ATE RMSE and Tracking Rate is adjusted with the parameter λ . Moreover, using ATE RMSE / Tracking Rate alone may be misleading: consider a full sequence run with 10cm accuracy and a run that crashes after 10% of the sequence: the accuracy is likely better in the latter case as there is less time for error accumulation. However, the USM takes such failures into account. More explanations in supplementary materials.

Experiments

Experimental setup

Datasets. We evaluate the 8 dynamic sequences of TUM RGB-D dataset [START_REF] Sturm | A benchmark for the evaluation of RGB-d SLAM systems[END_REF] in RGB-D, a reference benchmark for Dynamic SLAM, the first 11 sequences of KITTI odometry dataset [START_REF] Geiger | Are we ready for autonomous driving? the kitti vision benchmark suite[END_REF] in stereo, the ConsInv-Indoors dataset in monocular and the ConsInv-Outdoors dataset in stereo. For TUM RGB-D and KITTI, given the small number of sequences, we run our tests in a leave-one-out approach: we run the annotation/train pipeline on all sequences but one and infer/test temporal masks for the remaining one in a round-robin fashion. We propose the ConsInv dataset to evaluate Dynamic SLAM methods in very challenging conditions, including consensus inversions, risk of excessive masking and false starts. Difficulty levels and failure conditions are controlled (details and illustrations in supp. mats).

Spatial representation. We use ResNet-50 [START_REF] He | Deep residual learning for image recognition[END_REF] (TensorFlow 2, pretrained on ImageNet) to compute frame features. For every frame, we collect the output of each of the 17 convolutional blocks of ResNet-50. For each block, we apply a 2D average pooling on its output (into size (4, 4, .)), flatten and apply Principal Component Analysis, computed per block on the training sequences. It results in a feature vector of length about 100 to 1000 per convolutional block, which we concatenate, resulting in a vector of size 15k to 20k.

LSTM Encoder-Decoder and Training. Given a frame index k, we generate video samples by randomly sampling 49 frames from frames 0 to k -1 and adding frame k. We sample frames from the whole past to avoid depending on instantaneous motion detection (see Sec. 1). We mask the first 49 frames as we cannot sample them without repetition. We train the LSTM Encoder-Decoder on train+val with early stopping. Training lasted 40 to 60 epochs in our experiments. We augment training sequences with horizontal flipping and process video samples in a random order.

SLAM and Semantic Masking. We use ORB-SLAM 2 [START_REF] Mur-Artal | ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-d cameras[END_REF] (3000 features). As ORB-SLAM 2 is non-deterministic, we run it five times on KITTI dataset and ten times on other datasets for every sequence/temporal mask to evaluate and report the median score. We generate semantic masks for KITTI/ConsInv-Outdoors datasets using Mask R-CNN [START_REF] He | Mask r-CNN[END_REF], trained on the COCO dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], and for ConsInv-Indoors/TUM RGB-D datasets we used the DeepLabv3+ [START_REF] Chen | Encoder-decoder with atrous separable convolution for semantic image segmentation[END_REF] models from [START_REF] Bojko | Learning to segment dynamic objects using slam outliers[END_REF]. We segment people/vehicles (7 classes) for KITTI, people and cars for ConsInv-Outdoors (2 classes), people for TUM RGB-D (1 class), and drom / lambo / dragon for ConsInv-Indoors (1 class1 ).

Dataset Annotation and Metrics. For automatic annotations (i.e., self-supervision), we generate 200 temporal masks for each sequence with min. block sizes k 0 = k 1 = 25 and use the USM (Sec. 3.3) to benchmark/aggregate temporal masks. As the ATE RMSE of sequences in the datasets TUM RGB-D/ConsInv and KITTI is resp ≈1cm and ≈1m with manual expert labels, we set λ = 10m -1 for TUM RGB-D/ConsInv and λ = 0.1m -1 for KITTI, which makes USM ≈ T R(1λ ATE) and comparisons between SLAM algorithms match user expectations. We compare annotation methods in supp. materials (full/weak supervision), where we show that automatic annotations are easier to learn that human ones.

Comparison with the State of the Art

We compare our method to DynaSLAM2 [START_REF] Bescos | DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes[END_REF], Slamantic [START_REF] Schorghuber | SLAMANTIC -leveraging semantics to improve VSLAM in dynamic environments[END_REF], Lucas-Kanade optical flow [START_REF] Cheng | Improving monocular visual SLAM in dynamic environments: an optical-flow-based approach[END_REF] (only in monocular), all based on ORB-SLAM 2. We also evaluate StaticFusion [START_REF] Scona | StaticFusion: Background reconstruction for dense RGB-d SLAM in dynamic environments[END_REF], that supports only RGB-D. No masks refers to the original ORB-SLAM2, Full masks refers to ORB-SLAM 2 with semantic masks always applied. All methods use the same semantic masks. Note that median USM is not computed from median ATE RMSE and Tracking Rate.

TUM RGB-D dataset. Tab. 1 shows that the major obstacles for the SLAM for this dataset are motion consensus inversions, and our self-supervised model learned to mask objects most of the time to prevent them. All methods performed well, close or equal to the manual annotations, except DynaSLAM that performed barely above the No masks baseline and StaticFusion below it. DynaSLAM removes even more features than the Full masks approach, which delays SLAM initialization and causes temporary tracking loss. The Tracking Rate of DynaSLAM is in fact exceedingly low despite a good accuracy (i.e., low ATE RMSE), which proves the importance of measuring the Tracking Rate and the relevance of our metric, the USM. StaticFusion fails in dynamic sequences with fast rotations, as it is unable to filter dynamic objects quickly enough. KITTI dataset. Tab. 1 shows that our method slightly outperforms others as it makes slightly better masking decisions, and that the KITTI dataset is unsuitable for Dynamic SLAM testing since no matter when objects are masked, performance hardly changes. In particular, the Tracking Rate is always 100%, so tracking failures are absent from this dataset. ConsInv-Indoors-Dynamic subset. Tab. 1 shows that our approach outperforms all current methods. DynaSLAM and Slamantic perform worse than the Full masks baseline: thus, masking as much as possible (DynaSLAM does semantic + motion-based masking) or masking with prior geometric criteria (Slamantic) is suboptimal in difficult scenarios; masking appropriately is challenging. Lucas-Kanade optical flow has a low score as it does not avoid all consensus inversions and removes features during fast motions, lowering accuracy.

ConsInv-Indoors-Static subset. We also evaluate the ability to prevent false starts on the ConsInv-Indoors-Static subset. Tab. 1 shows that most methods, including ours, prevent all false starts. The underperformance of Slamantic shows that motion-based geometric criteria are vulnerable to motion consensus inversions.

ConsInv-Outdoors dataset. Tab. 1 shows that our multiclass self-supervised method outperforms the state of the art (USM = 0.88). Surprisingly, almost all other approaches have the same score (USM ≈ 0.80). This means that the masking strategy has to be learned from the environment, or performance may be suboptimal.

Ours vs Full Masks. The high performance of Full Masks shows that it is a good baseline, better than most Dynamic SLAM algorithms in difficult situations. Still, it often has a lowered Tracking Rate (i.e., it loses tracking), which is unacceptable in critical applications. Our approach has both a high tracking rate and high accuracy.

USM vs. ATE RMSE/TR. Results on ConsInv show that ATE RMSE and TR alone do not reflect SLAM perf. meaningfully in difficult scenarios: i.e., low TR + low ATE RMSE and high TR + high ATE RMSE. The USM balances those meaningfully for all datasets. This also shows that ATE RMSE is not always more important than TR / USM.

Qualitative results and performance. The objects of a frame are masked when they are likely to move, e.g., when they were seen in the past of the same sequence. The masking inference can be done in real-time. Details are in supplementary materials. Global conclusion. Our method outperforms the state of the art in all datasets. A remarkable result is that when object motion is difficult to detect as in the ConsInv dataset, the simple semantic approach Full Masks is safer than motion-based, geometry-dependent approaches. Our method does not depend on geometric priors and outperforms Full Masks, unlike current methods. And the USM makes comparisons more meaningful.

Limitations: tests in out-of-context

While we designed our method to adapt a SLAM to any context (same object classes/similar environment) at a low cost with self-supervision, we evaluated how the model trained on ConsInv-Indoors performs in the contexts of the ConsInv-Extra dataset -a living room and a meeting room. Tab. 1 shows that we perform slightly better than other SLAM algorithms. Thus, our model works in new contexts similar to the training one and with the same classes i.e., it supports domain shifts to a certain extent. If the change is more drastic, performance would likely decrease and require adaptations that are out of the scope of this paper. Although we mask all objects of the same class at once, with no instance separation, we still achieve state-of-the-art results in difficult scenarios, but performance may not be optimal if multiple objects of the same class cover the majority of an image with incoherent motions (e.g., a busy car parking).

Conclusion

In this paper, we introduced the paradigm of Temporal Masking to overcome the points of failures of current Dynamic SLAM algorithms. Given a source of semantic masks, the general idea is to apply semantic masks to filter objects of certain classes, when appropriate, to maximize a chosen SLAM metric. In detail, we proposed a memory-based neural network architecture that we train in a self-supervised way thanks to the proposed automatic annotation method. We compared the two obtained models with the state of the art [START_REF] Bescos | DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes[END_REF][START_REF] Cheng | Improving monocular visual SLAM in dynamic environments: an optical-flow-based approach[END_REF][START_REF] Schorghuber | SLAMANTIC -leveraging semantics to improve VSLAM in dynamic environments[END_REF][START_REF] Scona | StaticFusion: Background reconstruction for dense RGB-d SLAM in dynamic environments[END_REF] and outperformed it on real data: on the TUM RGB-D, KITTI and the proposed ConsInv dataset, that include challenging indoors/outdoors scenarios designed to test the limits of Dynamic SLAM algorithms. For this comparison, we designed a unified metric named USM that balances trajectory accuracy and robustness to failures due to excessive masking.

We are considering several improvements in future works. The proposed approach makes masking decisions at the class level, as a refinement, it seems valuable to explore a decision strategy that considers class instances separately. We are also considering developing a method merging semantic and Temporal Masking. A method that outputs semantic masks only when it improves a given metric would make our Temporal Masking method end-to-end with potentially superior semantic segmentation and masking decisions.

Finally, the Temporal Masking paradigm could be used outside SLAM applications, as it is a method to maximize the performance of an algorithm that takes as input a time series by triggering a specific action when appropriate. For instance, in path planning algorithms, it could prevent the exploration of probable dead ends based on the previous frames.

Acknowledgment this work relied on the use of the FactoryIA cluster, financially supported by the Ile-de-France Regional Council.

Figure 2 :

 2 Figure 2: Dynamic SLAM with Temporal Masking: to compute which semantic classes should be masked to maximize SLAM performance even in challenging dynamic scenarios. The resulting trajectory is as accurate as possible given the base SLAM and semantic segmentation network.

Figure 3 :

 3 Figure 3: Temporal Masking Network. It takes as input a video sample and outputs, for the last sampled frame, per-class binary masking decisions on which classes should be masked. Spatial Representation outputs features for every input frame.

Figure 4 :

 4 Figure 4: Temporal mask space E(l = 7, k 0 = 2, k 1 = 3) as a masking binary tree. A temporal mask is equivalent to a root-to-leaf path.

Table 1 :

 1 Comparison with the state of the art on various datasets in their preferred mode. Best in bold. Best ATE/TR results that are contradictory are underlined (USM combines ATE RMSE/TR to solve this contradiction). '-' indicates that the mode is not supported by the SLAM algorithm. Scores are medians over the dataset (every sequence is evaluated at least 5 times). For ATE RMSE, lower is better (↓). For Tracking Rate / USM / Prevented false starts, higher is better (↑). We use our model trained on ConsInv-Indoors when evaluating our method on ConsInv-Extra to measure how it performs in new domains. *Difficult sequences in ConsInv dataset makes the lone interpretations of ATE RMSE/TR misleading (Sec. 3.3).

					Baselines			State of the Art
	Mode	Dataset	Metric		No masks Full masks	Optical flow [8]	DynaSLAM [3] Slamantic [22] StaticFusion [23]	Ours
			ATE RMSE (m) ↓	0.105	0.019	-	0.019	0.028	0.099	0.019
	RGB-D	TUM RGB-D	Tracking Rate ↑	96%	96%	-	69%	96%	96%	96%
			USM ↑		0.55	0.80	-	0.57	0.76	0.54	0.80
			ATE RMSE (m) ↓	2.59	2.67	-	2.74	2.70	-	2.51
	Stereo	KITTI	Tracking Rate ↑	100%	100%	-	100%	100%	-	100%
			USM ↑		0.80	0.80	-	0.79	0.80	-	0.81
			ATE RMSE * ↓		0.084	0.019	-	0.025	0.032	-	0.024
	Stereo	ConsInv-Outdoors	Tracking Rate * ↑	100%	75%	-	74%	85%	-	88%
			USM ↑		0.61	0.81	-	0.80	0.82	-	0.88
			ATE RMSE * ↓		0.074	0.003	0.050	0.010	0.032	-	0.014
	Mono	ConsInv-Indoors-Dynamic	Tracking Rate * ↑	94%	74%	74%	70%	84%	-	84%
			USM ↑		0.57	0.71	0.49	0.63	0.68	-	0.75
			ATE RMSE * ↓		0.170	0.012	0.077	0.011	0.077	-	0.012
	Mono	ConsInv-Extra-MeetingRoom (domain shift)	Tracking Rate * ↑	96%	73%	62%	66%	86%	-	76%
			USM ↑		0.33	0.65	0.34	0.60	0.54	-	0.67
			ATE RMSE * ↓		0.091	0.012	0.012	0.020	0.016	-	0.013
	Mono	ConsInv-Extra-LivingRoom (domain shift)	Tracking Rate * ↑	96%	82%	62%	71%	84%	-	85%
			USM ↑		0.51	0.73	0.55	0.60	0.69	-	0.74
	Mono	ConsInv-Indoors-Static	Prevented false starts	↑	56%	100%	67%	100%	78%	-	100%

The DeepLabv3+ model we trained segments dromedary / lambo / dragon in a single dynamic class.

DynaSLAM crashes on TUM RGB-D seq. fr3_w_xyz. We compute the avg. on the 7 others in this case.