Ignacio Rocco

Relja Arandjelović

Josef Sivic

D I Ens

End-to-end weakly-supervised semantic alignment

We tackle the task of semantic alignment where the goal is to compute dense semantic correspondence aligning two images depicting objects of the same category. This is a challenging task due to large intra-class variation, changes in viewpoint and background clutter. We present the following three principal contributions. First, we develop a convolutional neural network architecture for semantic alignment that is trainable in an end-to-end manner from weak imagelevel supervision in the form of matching image pairs. The outcome is that parameters are learnt from rich appearance variation present in different but semantically related images without the need for tedious manual annotation of correspondences at training time. Second, the main component of this architecture is a differentiable soft inlier scoring module, inspired by the RANSAC inlier scoring procedure, that computes the quality of the alignment based on only geometrically consistent correspondences thereby reducing the effect of background clutter. Third, we demonstrate that the proposed approach achieves state-of-the-art performance on multiple standard benchmarks for semantic alignment.

Introduction

Finding correspondence is one of the fundamental problems in computer vision. Initial work has focused on finding correspondence between images depicting the same object or scene with applications in image stitching [START_REF] Szeliski | Image alignment and stitching: A tutorial[END_REF], multiview 3D reconstruction [START_REF] Hartley | Multiple view geometry in computer vision[END_REF], motion estimation [START_REF] Fischer | FlowNet: Learning optical flow with convolutional networks[END_REF][START_REF] Weinzaepfel | DeepFlow: Large displacement optical flow with deep matching[END_REF] or tracking [START_REF] Engel | LSD-SLAM: Largescale direct monocular SLAM[END_REF][START_REF] Newcombe | DTAM: Dense tracking and mapping in real-time[END_REF]. In this work we study the problem of finding category-level correspondence, or semantic alignment [START_REF] Berg | Shape matching and object recognition using low distortion correspondence[END_REF][START_REF] Liu | SIFT flow: Dense correspondence across different scenes[END_REF], where the goal is to establish dense correspondence between different objects belonging to the same category, such as the two different motorcycles illustrated in Fig. 1. This is an important problem with applications in object recognition [START_REF] Liu | SIFT flow: Dense correspondence across scenes and its applications[END_REF], image editing [START_REF] Dale | Image restoration using online photo collections[END_REF], or robotics [START_REF] Nikandrova | Category-based task specific grasping[END_REF]. 1 Département d'informatique de l'ENS, École normale supérieure, CNRS, PSL Research University, 75005 Paris, France. 4 Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague. This is also an extremely challenging task because of the large intra-class variation, changes in viewpoint and presence of background clutter.

Input pair

The current best semantic alignment methods [START_REF] Han | SCNet: Learning semantic correspondence[END_REF][START_REF] Kim | DCTM: Discretecontinuous transformation matching for semantic flow[END_REF][START_REF] Novotny | AnchorNet: A weakly supervised network to learn geometry-sensitive features for semantic matching[END_REF] employ powerful image representations based on convolutional neural networks coupled with a geometric deformation model. However, these methods suffer from one of the following two major limitations. First, the image representation and the geometric alignment model are not trained together in an end-to-end manner. Typically, the image representation is trained on some auxiliary task such as image classification and then employed in an often ad-hoc geometric alignment model. Second, while trainable geometric alignment models exist [START_REF] Brachmann | DSAC -Differentiable RANSAC for camera localization[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF], they require strong supervision in the form of ground truth correspondences, which is hard to obtain for a diverse set of real images on a large scale.

In this paper, we address both these limitations and develop a semantic alignment model that is trainable end-toend from weakly supervised data in the form of matching image pairs without the need for ground truth correspondences. To achieve that we design a novel convolutional neural network architecture for semantic alignment with a differentiable soft inlier scoring module inspired by the RANSAC inlier scoring procedure. The resulting architecture is end-to-end trainable with only image-level supervision. The outcome is that the image representation can be trained from rich appearance variations present in different but semantically related image pairs, rather than synthetically deformed imagery [START_REF] Kanazawa | WarpNet: Weakly supervised matching for single-view reconstruction[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF]. We show that our approach allows to significantly improve the performance of the baseline deep CNN alignment model, achieving stateof-the-art performance on multiple standard benchmarks for semantic alignment. Our code and trained models are available online [START_REF] Rocco | Webpage: End-to-end weakly-supervised semantic alignment[END_REF].

Related work

The problem of semantic alignment has received significant attention in the last few years with progress in both (i) image descriptors and (ii) geometric models. The key innovation has been making the two components trainable from data. We summarize the recent progress in Table 1 where we indicate for each method whether the descriptor (D) or the alignment model (A) are trainable, whether the entire architecture is trainable end-to-end (E-E), and whether the required supervision is strong (s) or weak (w).

Early methods, such as [START_REF] Berg | Shape matching and object recognition using low distortion correspondence[END_REF][START_REF] Kim | Deformable spatial pyramid matching for fast dense correspondences[END_REF][START_REF] Liu | SIFT flow: Dense correspondence across scenes and its applications[END_REF], employed handengineered descriptors like SIFT or HOG together with hand-engineered alignment models based on minimizing a given matching energy. This approach has been quite successful [START_REF] Ham | Proposal flow: Semantic correspondences from object proposals[END_REF][START_REF] Taniai | Joint recovery of dense correspondence and cosegmentation in two images[END_REF][START_REF] Ufer | Deep semantic feature matching[END_REF][START_REF] Yang | Object-aware dense semantic correspondence[END_REF] using in some cases [START_REF] Ufer | Deep semantic feature matching[END_REF] pre-trained (but fixed) convolutional neural network (CNN) descriptors. However, none of these methods train the image descriptor or the geometric model directly for semantic alignment.

Others [START_REF] Kim | FCSS: Fully convolutional self-similarity for dense semantic correspondence[END_REF][START_REF] Kim | DCTM: Discretecontinuous transformation matching for semantic flow[END_REF][START_REF] Novotny | AnchorNet: A weakly supervised network to learn geometry-sensitive features for semantic matching[END_REF] have investigated trainable image descriptors for semantic matching but have combined them with hand-engineered alignment models still rendering the alignment pipeline not trainable end-to-end.

Finally, recent work [START_REF] Han | SCNet: Learning semantic correspondence[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF] has employed trainable CNN descriptors together with trainable geometric alignment methods. However, in [START_REF] Han | SCNet: Learning semantic correspondence[END_REF] the matching is learned at the object-proposal level and a non-trainable fusion step is necessary to output the final alignment making the method non end-to-end trainable. On the contrary, [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF] ing the method end-to-end trainable. However, the method is trained with strong supervision in the form of ground truth correspondences obtained from synthetically warped images, which significantly limits the appearance variation in the training data.

Contributions. We develop a network architecture where both the descriptor and the alignment model are trainable in an end-to-end manner from weakly supervised data. This enables training from real images with rich appearance variation and without the need for manual ground-truth correspondence. We demonstrate that the proposed approach significantly improves alignment results achieving state-ofthe-art performance on several datasets for semantic alignment.

Weakly-supervised semantic alignment

This section presents a method for training a semantic alignment model in an end-to-end fashion using only weak supervision -the information that two images should match -but without access to the underlying geometric transformation at training time. The approach is outlined in Fig. 2. Source and target images (I s , I t ) are passed through an alignment network used to estimate the geometric transformation g. Then, the soft-inlier count is computed (in green) by first finding the inlier region m in agreement with g, and then adding up the pairwise matching scores inside this area. The soft-inlier count is differentiable, which allows the whole model to be trained using back-propagation. Functions are represented in blue and tensors in pink.

Namely, given a pair of images, an alignment network estimates the geometric transformation that aligns them. The quality of the estimated transformation is assessed using the proposed soft-inlier count which aggregates the observed evidence in the form of feature matches. The training objective then is to maximize the alignment quality for pairs of images which should match.

The key idea is that, instead of requiring strongly supervised training data in the form of known pairwise alignments and training the alignment network with these, the network is "forced" into learning to estimate good alignments in order to achieve high alignment scores (soft-inlier counts) for matching image pairs. The details of the alignment network and the soft-inlier count are presented next.

Semantic alignment network

In order to make use of the error signal coming from the soft-inlier count, our framework requires an alignment network which is trainable end-to-end. We build on the Siamese CNN architecture described in [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF], illustrated in the left section of Fig. 2. The architecture is composed of three main stages -feature extraction, followed by feature matching and geometric transformation estimation -which we review below.

Feature extraction. The input source and target images, (I s , I t ), are passed through two fully-convolutional feature extraction CNN branches, F , with shared weights. The resulting feature maps (f s , f t ) are h × w × d tensors which can be interpreted as dense h × w grids of d-dimensional

local features f ij: ∈ R d . These individual d-dimensional features are L2 normalized.
Pairwise feature matching. This stage computes all pairwise similarities, or match scores, between local features in the two images. This is done with the normalized correlation function, defined as:

S : R h×w×d × R h×w×d → R h×w×h×w (1)
s ijkl = S(f s , f t ) ijkl = f s ij: , f t kl: a,b f s ab: , f t kl: 2 , ( 2
)
where the numerator in (2) computes the raw pairwise match scores by computing the dot product between features pairs. The denominator performs a normalization operation with the effect of down-weighing ambiguous matches, by penalizing features from one image which have multiple highly-rated matches in the other image. This is in line with the classical second nearest neighbour test of Lowe [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF]. The resulting tensor s contains all normalized match scores between the source and target features.

Geometric transformation estimation. The final stage of the alignment network consists of estimating the parameters of a geometric transformation g given the match scores s. This is done by a transformation regression CNN, represented by the function G:

G : R h×w×h×w → R K , g = G(s) (3)
where K is the number of degrees of freedom, or parameters, of the geometric model; e.g. K = 6 for an affine model. The estimated transformation parameters g are used to define the 2-D warping T g :

T g : R 2 → R 2 , (u s , v s ) = T g (u t , v t ) (4)
where (u t , v t ) are the spatial coordinates of the target image, and (u s , v s ) the corresponding sampling coordinates in the source image. Using T g , it is possible to warp the source to the target image. Note that all parts of the geometric alignment network are differentiable and therefore amenable to end-to-end training [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF], including the feature extractor F which can learn better features for the task of semantic alignment.

Soft-inlier count

We propose the soft-inlier count used to automatically evaluate the estimated geometric transformation g. Making an effort to maximize this count provides the weaksupervisory signal required to train the alignment network, avoiding the need for expensive manual annotations for g. The soft-inlier count is inspired by the inlier count used in the robust RANSAC method [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF], which is reviewed first.

RANSAC inlier count. For simplicity, let us consider the problem of fitting a line to a set of observed points p i , with i = 1, . . . N , as illustrated in Fig. 3a. RANSAC proceeds by sampling random pairs of points used to propose line hypotheses, each of which is then scored using the inlier count, and the highest scoring line is chosen; here we only focus on the inlier count aspect of RANSAC used to score a hypothesis. Given a hypothesized line , the RANSAC inlier scoring function counts the number of observed points which are in agreement with this hypothesis, called the inliers. A point p is typically deemed to be an inlier iff its distance to the line is smaller than a chosen distance threshold t, i.e. d(p, ) < t.

The RANSAC inlier count, c R , can be formulated by means of an auxiliary indicator function illustrated in Fig. 3b, which we call the inlier mask function m:

c R = i m(p i ), where m(p) = 1, if d(p, ) < t 0, otherwise. (5)
Soft-inlier count. The RANSAC inlier count cannot be used directly in a neural network as it is not differentiable. Furthermore, in our setting there is no sparse set of matching points, but rather a match score for every match in a discretized match space. Therefore, we propose a direct extension, the soft-inlier count, which, instead of counting over a sparse set of matches, sums the match scores over all possible matches. The running line-fitting example can now be revisited under the discrete-space conditions, as illustrated in Fig- ure 3c. The proposed soft-inlier count for this case is:

c = i,j s ij m ij , (6)
where s ij is the match score at each grid point (i,j), and m ij is the discretized inlier mask:

m ij = 1 if d (i, j), < t 0 otherwise (7)
Translating the discrete-space line-fitting example to our semantic alignment problem, s is a 4-D tensor containing scores for all pairwise feature matches between the two images (Section 3.1), and matches are deemed to be inliers if they fit the estimated geometric transformation g. More formally, the inlier mask m is now also a 4-D tensor, constructed by thresholding the transfer error:

m ijkl = 1 if d (i, j), T g (k, l) < t 0 otherwise, (8)
where T g (k, l) are the estimated coordinates of target image's point (k, l) in the source image according to the geometric transformation g; d (i, j), T g (k, l) is the transfer error as it measures how aligned is the point (i, j) in the source image, with the projection of the target image point (k, l) into the source image. The soft-inlier count c is then computed by summing the masked matching scores over the entire space of matches:

c = i,j,k,l s ijkl m ijkl . (9)
Differentiability. The proposed soft-inlier count c is differentiable with respect to the transformation parameters g as long as the geometric transformation T g is differentiable [START_REF] Jaderberg | Spatial transformer networks[END_REF], which is the case for a range of standard geometric transformations such as 2D affine, homography or thinplate spline transformations. Furthermore, it is also differentiable w.r.t. the match scores, which facilitates training of the feature extractor.

Implementation as a CNN layer. The inlier mask m can be computed by warping an identity mask m Id with the estimated transformation T g , where m Id is constructed by thresholding the transfer error of the identity transformation:

m Id ijkl = 1 d (i, j), (k, l) < t 0 otherwise. ( 10
)
The warping is implemented using a spatial transformer layer [START_REF] Jaderberg | Spatial transformer networks[END_REF], which consists of a grid generation layer and a bilinear sampling layer. Both of these functions are readily available in most deep learning frameworks.

Optimization objective. For a given training pair of images that should match, the goal is to maximize the soft-inlier count c, or, equivalently, to minimize the loss L = -c. Analogy to RANSAC. Please also note that our method is similar in spirit to RANSAC [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF], where (i) transformations are proposed (by random sampling) and then (ii) scored by their support (number of inliers). In our case, during training (i) the transformations are proposed (estimated) by the regressor network G and (ii) scored using the proposed softinlier score. The gradient of this score is used to improve both the regressor G and feature extractor F (see Fig. 2). In turn, the regressor produces better transformations and the feature extractor better feature matches that maximize the soft-inlier score on training images.

Evaluation and results

In this section we provide implementation details, benchmarks used to evaluate our approach, and quantitative and qualitative results.

Implementation details

Semantic alignment network. For the underlying semantic alignment network, we use the best-performing architecture from [START_REF] Rocco | Webpage: Convolutional neural network architecture for geometric matching[END_REF] which employs a ResNet-101 [START_REF] He | Deep residual learning for image recognition[END_REF], cropped after conv4-23, as the feature extraction CNN F . Note that this is a better performing model than the one described in [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF], mainly due to use of ResNet versus VGG-16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. Given an image pair, the model produces a thin-plate spline geometric transformation T g which aligns the two images; T g has 18 degrees of freedom. The network is initialized with the pre-trained weights from [START_REF] Rocco | Webpage: Convolutional neural network architecture for geometric matching[END_REF], and we finetune it with our weakly supervised method. Note that the initial model has been trained in a self-supervised way from synthetic data, not requiring human supervision [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF], therefore not affecting our claim of weakly supervised training 1 .

Training details.

Training and validation image pairs are obtained from the training set of PF-PASCAL, described in Section 4.2. All input images are resized to 240 × 240, and the value t = L/30 (where L = h = w is the size of the extracted feature maps) was used for the transfer error threshold. The whole model is trained end-to-end, including the affine parameters in the batch normalization layers. However, the running averages of the batch normalization layers are kept fixed, in order to be less dependent on the particular statistics of the training dataset. The network is implemented in PyTorch [START_REF] Paszke | PyTorch[END_REF] and trained using the Adam optimizer [START_REF] Kingma | A method for stochastic optimization[END_REF] with learning rate 5 • 10 -8 , no weight decay and batch size of 16. The training dataset is augmented by horizontal flipping, swapping the source and target images, and random cropping. Early stopping is required to avoid overfitting, given the small size of the training set. This results in 13 training epochs, taking about an hour on a modern GPU.

Evaluation benchmarks

Evaluation is performed on three standard image alignment benchmarks: PF-PASCAL, Caltech-101 and TSS. [START_REF] Ham | Proposal flow: Semantic correspondences from object proposals[END_REF]. This dataset contains 1351 semantically related image pairs from 20 object categories, which present challenging appearance differences and background clutter. We use the split proposed in [START_REF] Han | SCNet: Learning semantic correspondence[END_REF], which divides the dataset into roughly 700 pairs for training, 300 pairs for validation, and 300 pairs for testing. Keypoint annotations are provided for each image pair, which are used only for evaluation purposes. Alignment quality is evaluated in terms of the percentage of correct keypoints (PCK) metric [START_REF] Yang | Articulated human detection with flexible mixtures of parts[END_REF], which counts the number of keypoints which have a transfer error below a given threshold. We follow the procedure employed in [START_REF] Han | SCNet: Learning semantic correspondence[END_REF], where keypoint (x, y) coordinates are nor- malized in the [0, 1] range by dividing with the image width and height respectively, and the value α = 0.1 is employed as the distance threshold.

PF-PASCAL

Caltech-101 [START_REF] Fei-Fei | One-shot learning of object categories[END_REF]. Although originally introduced for the image classification task, the dataset was adopted in [START_REF] Kim | Deformable spatial pyramid matching for fast dense correspondences[END_REF] for assessing semantic alignment, and has been then extensively used for this purpose [START_REF] Ham | Proposal flow: Semantic correspondences from object proposals[END_REF][START_REF] Han | SCNet: Learning semantic correspondence[END_REF][START_REF] Kim | FCSS: Fully convolutional self-similarity for dense semantic correspondence[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF]. The evaluation is performed on 1515 semantically related image pairs, 15 pairs for each of the 101 object categories of the dataset. The semantic alignment is evaluated using three different metrics: (i) the label transfer accuracy (LT-ACC); (ii) the intersection-over-union (IoU), and; (iii) the object localization error (LOC-ERR). The label transfer accuracy and the intersection-over-union both measure the overlap between the annotated foreground object segmentation masks, with former putting more emphasis on the background class and the latter on the foreground object. The localization error computes a dense displacement error. However, given the lack of dense displacement annotations, the metric computes the ground-truth transformation from the source and target bounding boxes, thus assuming that the transformation is a simple translation with axis-aligned anisotropic scaling. This assumption is unrealistic as, amongst others, it does not cover rotations, affine or deformable transformations. Therefore, we believe that LOC-ERR should not be reported any more, but report it here for completeness and in order to adhere to the currently adopted evaluation protocol.

TSS [START_REF] Taniai | Joint recovery of dense correspondence and cosegmentation in two images[END_REF]. The recently introduced TSS dataset contains 400 semantically related image pairs, which are split into three different subsets: FG3DCar, JODS and PASCAL, according to the origin of the images. Ground-truth flow is provided for each pair, which was obtained by manual annotation of sparse keypoints, followed by automatic densification using an interpolation algorithm. The evaluation metric is the PCK computed densely over the foreground object. The distance threshold is defined as α max(w s , h s ) with (w s , h s ) being the dimensions of the source image, and α = 0.05.

Assessing generalization. We train a single semantic alignment network with the 700 training pairs from PF-PASCAL without using the keypoint annotations, and stress that our weakly-supervised training objective only uses the information that the image pair should match. The same model is then used for all experiments -evaluation on the test sets of PF-PASCAL, Caltech-101 and TSS datasets. This poses an additional difficulty as these datasets contain images of different object categories or of different nature. While PF-PASCAL contains images of common objects such as car, bicycle, boat, etc., Caltech-101 contains images of much less common categories such as accordion, buddha or octopus. On the other hand, while the classes of TSS do appear in PF-PASCAL, the pose differences in TSS are usually smaller than in PF-PASCAL, which modifies the challenge into obtaining a very precise alignment.

Results

In the following, our alignment network trained with weak supervision is compared to the state-of-the-art alignment methods, many of which require manual annotations or strong supervision (c.f . Table 1). 2 it is clear that our method sets the new state-of-the-art, achieving an overall PCK of 75.8%, which is a 3.6% improvement over the best competitor [START_REF] Han | SCNet: Learning semantic correspondence[END_REF]. This result is impressive as the two methods are trained on the same image pairs, with ours being weakly supervised while [START_REF] Han | SCNet: Learning semantic correspondence[END_REF] make use of bounding box annotations.

PF-PASCAL. From Table

The benefits of weakly supervised training can be seen by comparing our method with ResNet-101+CNNGeo [START_REF] Rocco | Webpage: Convolutional neural network architecture for geometric matching[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF]. The two use the same base alignment network (c.f . Section 4.1), but ResNet-101+CNNGeo was trained only on synthetically deformed image pairs, while ours employs the proposed weakly supervised fine-tuning. The 3.9% boost clearly demonstrates the advantage obtained by training on real image pairs and thus encountering rich appearance variations, as opposed to using synthetically transformed pairs in ResNet-101+CNNGeo [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF].

Caltech-101. Table 3 presents the quantitative results for this dataset. The proposed method beats state-ofthe-art results in terms of the label-transfer accuracy and intersection-over-union metrics. Weakly supervised training again improves the results, by 2%, over the synthetically trained ResNet-101+CNNGeo. In terms of the localizationerror metric, our model does not attain state-of-the-art per-Method LT-ACC IoU LOC-ERR HOG+PF-LOM [START_REF] Ham | Proposal flow: Semantic correspondences from object proposals[END_REF] 0.78 0.50 0.26 FCSS+SIFT Flow [START_REF] Kim | FCSS: Fully convolutional self-similarity for dense semantic correspondence[END_REF] 0.80 0.50 0.21 FCSS+PF-LOM [START_REF] Kim | FCSS: Fully convolutional self-similarity for dense semantic correspondence[END_REF] 0.83 0.52 0.22 VGG-16+SCNet-A [START_REF] Han | SCNet: Learning semantic correspondence[END_REF] 0.78 0.50 0.28 VGG-16+SCNet-AG [START_REF] Han | SCNet: Learning semantic correspondence[END_REF] 0.78 0.50 0.27 VGG-16+SCNet-AG+ [START_REF] Han | SCNet: Learning semantic correspondence[END_REF] 0.79 0.51 0.25 HOG+OADSC [START_REF] Yang | Object-aware dense semantic correspondence[END_REF] 0.81 0.55 0.19 VGG-16+CNNGeo [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF] 0.80 0.55 0.26 ResNet-101+CNNGeo [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF] 0.83 0.61 0.25 Proposed 0.85 0.63 0.24 HOG+PF-LOM [START_REF] Ham | Proposal flow: Semantic correspondences from object proposals[END_REF] 0.786 0.653 0.531 0.657 HOG+TSS [START_REF] Taniai | Joint recovery of dense correspondence and cosegmentation in two images[END_REF] 0.830 0.595 0.483 0.636 FCSS+SIFT Flow [START_REF] Kim | FCSS: Fully convolutional self-similarity for dense semantic correspondence[END_REF] 0.830 0.656 0.494 0.660 FCSS+PF-LOM [START_REF] Kim | FCSS: Fully convolutional self-similarity for dense semantic correspondence[END_REF] 0.839 0.635 0.582 0.685 HOG+OADSC [START_REF] Yang | Object-aware dense semantic correspondence[END_REF] 0.875 0.708 0.729 0.771 FCSS+DCTM [START_REF] Kim | DCTM: Discretecontinuous transformation matching for semantic flow[END_REF] 0.891 0.721 0.610 0.740 VGG-16+CNNGeo [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF] 0.839 0.658 0.528 0.675 ResNet-101+CNNGeo [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF] formance, but we argue that this metric is not a good indication of the alignment quality, as explained in section 4.2. This claim is further backed up by noticing that the relative ordering of various methods based on this metric is in direct opposition with the other two metrics.

TSS. The quantitative results for the TSS dataset are presented in Table 4. We set the state-of-the-art for two of the three subsets of the TSS dataset: FG3DCar and JODS. Although our weakly supervised training provides an improvement over the base alignment network, ResNet-101+CNNGeo, the gain is modest. We believe the reason is a very different balancing of classes in this dataset compared to our training. Recall our model is trained only once on the PF-PASCAL dataset, and is then applied without any further training on TSS and Caltech-101.

Qualitative results. Figures 4a, 4b and 5 show qualitative results on the Caltech-101, TSS and PF-PASCAL datasets, respectively. Our method is able to align images across prominent viewpoint changes, in the presence of significant clutter, while simultaneously tolerating large intraclass variations. For additional qualitative examples, please refer to [START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF].

Conclusions

We have designed a network architecture and training procedure for semantic image alignment inspired by the ro-

Figure 1 :

 1 Figure 1: We describe a CNN architecture that, given an input image pair (top), outputs dense semantic correspondence between the two images together with the aligning geometric transformation (middle) and discards geometrically inconsistent matches (bottom). The alignment model is learnt from weak supervision in the form of matching image pairs without correspondences.

Figure 2 :

 2 Figure2: End-to-end weakly-supervised alignment. Source and target images (I s , I t ) are passed through an alignment network used to estimate the geometric transformation g. Then, the soft-inlier count is computed (in green) by first finding the inlier region m in agreement with g, and then adding up the pairwise matching scores inside this area. The soft-inlier count is differentiable, which allows the whole model to be trained using back-propagation. Functions are represented in blue and tensors in pink.

Figure 3 :

 3 Figure 3: Line-fitting example. (a) The line hypothesis can be evaluated in terms of the number of inliers. (b) The inlier mask m specifies the region where the inlier distance threshold is satisfied. (c) In the discretized space setting, where the match score sij exists for every point (i, j), the soft-inlier count is computed by summing up match scores masked by the inlier mask m from (b).

Table 4 :

 4 0.901 0.764 0.563 0.743 Proposed 0.903 0.764 0.565 0.744 Evaluation results on the TSS dataset.

Figure 4 :

 4 Figure 4: Alignment examples on the Caltech-101 and TSS datasets. Each row shows the (left) source and (middle) target images, and (right) the automatic semantic alignment. bust inlier scoring used in the widely successful RANSAC fitting algorithm [7]. The architecture requires supervision only in the form of matching image pairs and sets the new state-of-the-art on multiple standard semantic alignment benchmarks, even beating alignment methods that require geometric supervision at training time. However, handling multiple objects and non-matching image pairs still remains an open challenge. These results open-up the possibility of learning powerful correspondence networks from large-scale datasets such as ImageNet.

Table 1 :

 1 Comparison of recent related work.

	estimate a
	parametric geometric model, which can be converted into
	dense pixel correspondences in a differentiable way, mak-

Table 2 :

 2 Method aero bike bird boat bottle bus car cat chair cow d.table dog horse moto person plant sheep sofa train tv all HOG+PF-LOM [8] 73.3 74.4 54.4 50.9 49.6 73.8 72.9 63.6 46.1 79.8 42.5 48.0 68.3 66.3 42.1 62.1 65.2 57.1 64.4 58.0 62.5 VGG-16+SCNet-A [10] 67.6 72.9 69.3 59.7 74.5 72.7 73.2 59.5 51.4 78.2 39.4 50.1 67.0 62.1 69.3 68.5 78.2 63.3 57.7 59.8 66.3 VGG-16+SCNet-AG [10] 83.9 81.4 70.6 62.5 60.6 81.3 81.2 59.5 53.1 81.2 62.0 58.7 65.5 73.3 51.2 58.3 60.0 69.3 61.5 80.0 69.7 VGG-16+SCNet-AG+ [10] 85.5 84.4 66.3 70.8 57.4 82.7 82.3 71.6 54.3 95.8 55.2 59.5 68.6 75.0 56.3 60.4 60.0 73.7 66.5 76.7 72.2 Per-class PCK on the PF-PASCAL dataset.

	VGG-16+CNNGeo [29]	75.2 80.1 73.4 59.7 43.8 77.9 84.0 67.7 44.3 89.6 33.9 67.1 60.5 72.6 54.0 41.0 60.0 45.1 58.3 37.2 65.0
	ResNet-101+CNNGeo [29] 82.4 80.9 85.9 47.2 57.8 83.1 92.8 86.9 43.8 91.7 28.1 76.4 70.2 76.6 68.9 65.7 80.0 50.1 46.3 60.6 71.9
	Proposed	83.7 88.0 83.4 58.3 68.8 90.3 92.3 83.7 47.4 91.7 28.1 76.3 77.0 76.0 71.4 76.2 80.0 59.5 62.3 63.9 75.8

Table 3 :

 3 Evaluation results on the Caltech-101 dataset.

	Method	FG3D. JODS PASC. avg.

The initial model is trained with a supervised loss, but the "supervision" is automatic due to the use of synthetic data.

Acknowledgements. This work has been partly supported by ERC grant LEAP (no. 336845), the Inria CityLab IPL, CIFAR Learning in Machines & Brains program and the European Regional Development Fund under the project IMPACT (reg. no. CZ.02.1.01/0.0/0.0/15 003/0000468).


Hajime Taira

Masatoshi Okutomi

Torsten Sattler

Mircea Cimpoi

Marc Pollefeys

Josef Sivic

Tomas Pajdla

Akihiko Torii

InLoc: Indoor Visual Localization with Dense Matching and View Synthesis

We seek to predict the 6 degree-of-freedom (6DoF) pose of a query photograph with respect to a large indoor 3D map. The contributions of this work are three-fold. First, we develop a new large-scale visual localization method targeted for indoor environments. The method proceeds along three steps: (i) efficient retrieval of candidate poses that ensures scalability to large-scale environments, (ii) pose estimation using dense matching rather than local features to deal with textureless indoor scenes, and (iii) pose verification by virtual view synthesis to cope with significant changes in viewpoint, scene layout, and occluders. Second, we collect a new dataset with reference 6DoF poses for large-scale indoor localization. Query photographs are captured by mobile phones at a different time than the reference 3D map, thus presenting a realistic indoor localization scenario. Third, we demonstrate that our method significantly outperforms current state-of-the-art indoor localization approaches on this new challenging data.

Introduction

Autonomous navigation inside buildings is a key ability of robotic intelligent systems [START_REF] Debski | Open-source localization device for indoor mobile robots[END_REF][START_REF] Lim | Realtime image-based 6-dof localization in large-scale environments[END_REF]. Successful navigation requires both to localize a robot and to determine a path to its goal. One approach to solving the localization problem is to build a 3D map of the building and then use a camera 1 to estimate the current position and orientation of the robot (Figure 1). Imagine also the benefit of an intelligent indoor navigation system that helps you find your way, for exam-

Database of RGBD images

Query image

Large-scale indoor 3D maps ? 6DoF camera pose Figure 1. Large-scale indoor visual localization. Given a database of geometrically-registered RGBD images, we predict the 6DoF camera pose of a query RGB image by retrieving candidate images, estimating candicate camera poses, and selecting the best matching camera pose. To address inherent difficulties in indoor visual localization, we introduce the "InLoc" approach that performs a sequence of progressively stricter verification steps. ple, at Chicago airport, Tokyo Metropolitan station or the CVPR conference center. Besides intelligent systems, the visual localization problem is also highly relevant for any type of Mixed Reality application, including Augmented Reality [START_REF] Castle | Video-rate localization in multiple maps for wearable augmented reality[END_REF][START_REF] Middelberg | Scalable 6-dof localization on mobile devices[END_REF][START_REF] Wagner | Real-time detection and tracking for augmented reality on mobile phones[END_REF].

Due to the availability of datasets, e.g., obtained from Flickr [START_REF] Li | Worldwide pose estimation using 3d point clouds[END_REF] or captured from autonomous vehicles [START_REF] Chen | City-scale landmark identification on mobile devices[END_REF][START_REF] Maddern | 1 Year, 1000km: The Oxford RobotCar Dataset[END_REF], large-scale localization in urban environments has been an active field of research [6, 9, 14, 15, 19, 20, 27, 29, 34, 38, 44, 53-57, 65-67, 75, 79, 80]. In contrast, indoor localization [START_REF] Brachmann | DSAC -Differentiable RANSAC for Camera Localization[END_REF][START_REF] Brachmann | Uncertainty-Driven 6D Pose Estimation of Objects and Scenes from a Single RGB Image[END_REF][START_REF] Lim | Realtime image-based 6-dof localization in large-scale environments[END_REF][START_REF] Schmidt | Self-Supervised Visual Descriptor Learning for Dense Correspondence[END_REF][START_REF] Shotton | Scene coordinate regression forests for camera relocalization in RGB-D images[END_REF][START_REF] Sun | A Dataset for Benchmarking Image-based Localization[END_REF][START_REF] Valentin | Learning to Navigate the Energy Landscape[END_REF][START_REF] Wang | Lost shopping! monocular localization in large indoor spaces[END_REF] has received less atten-tion in the last years. At the same time, indoor localization is, in many ways, a harder problem than urban localization: 1) Due to the short distance to the scene geometry, even small changes in viewpoint lead to large changes in image appearance. For the same reason, ocluders such as humans or chairs often have a stronger impact compared to urban scenes. Thus, indoor localization approaches have to handle significantly larger changes in appearance between a query and reference images. 2) Large parts of indoor scenes are textureless and textured areas are typically rather small. As a result, feature matches are often clustered in small regions of the images, resulting in unstable pose estimates [START_REF] Irschara | From structure-from-motion point clouds to fast location recognition[END_REF]. 3) To make matters worse, buildings are often highly symmetric with many repetitive elements, both on large (similar corridors, rooms, etc.) and small (similar chairs, tables, doors etc.) scale. While structural ambiguities also cause problems in urban environments, they often only occur in larger scenes [START_REF] Arandjelović | Dislocation: Scalable descriptor distinctiveness for location recognition[END_REF][START_REF] Sattler | Large-scale location recognition and the geometric burstiness problem[END_REF][START_REF] Torii | Visual place recognition with repetitive structures[END_REF]. 4) The appearance of indoor scenes changes considerably over the course of a day due to the complex illumination conditions (indirect light through windows and active illumination from lamps). 5) Indoor scenes are often highly dynamic over time as furniture and personal effects are moved through the environment. In contrast, the overall appearance of building facades does not change too much over time. This paper addresses these difficulties inherent to indoor visual localization by proposing a new localization method. Our approach starts with an image retrieval step, using a compact image representation [6] that scales to large scenes. Given a shortlist of potentially relevant database images, we apply two progressively more discriminative geometric verification steps: (i) We use dense matching of CNN descriptors that capture spatial configurations of higher-level structures (rather than individual local features) to obtain the correspondences required for camera pose estimation. (ii) We then apply a novel pose verification step based on virtual view synthesis that can accurately verify whether the query image depicts the same place by dense pixel-level matching, again not relying on sparse local features.

Historically, the datasets used to evaluate indoor visual localization were restricted to small, often room-scale, scenes. Driven by the interest in semantic scene understanding [START_REF] Armeni | 3D Semantic Parsing of Large-Scale Indoor Spaces[END_REF][START_REF] Dai | ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes[END_REF][START_REF] Xiao | SUN3D: A database of big spaces reconstructed using sfm and object labels[END_REF] and enabled by scalable reconstruction techniques [START_REF] Halber | Fine-To-Coarse Global Registration of RGB-D Scans[END_REF][START_REF] Newcombe | Kinectfusion: Real-time dense surface mapping and tracking[END_REF]48], large-scale indoor datasets covering multiple rooms or even whole buildings are becoming available [START_REF] Armeni | 3D Semantic Parsing of Large-Scale Indoor Spaces[END_REF][START_REF] Chang | Matterport3D: Learning from RGB-D Data in Indoor Environments[END_REF][START_REF] Dai | ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes[END_REF][START_REF] Sun | A Dataset for Benchmarking Image-based Localization[END_REF][START_REF] Wang | Lost shopping! monocular localization in large indoor spaces[END_REF][START_REF] Wijmans | Exploiting 2D floorplan for building-scale panorama RGBD alignment[END_REF][START_REF] Xiao | Reconstructing the worlds museums[END_REF][START_REF] Xiao | SUN3D: A database of big spaces reconstructed using sfm and object labels[END_REF]. However, most of these datasets focus on reconstruction [START_REF] Wijmans | Exploiting 2D floorplan for building-scale panorama RGBD alignment[END_REF][START_REF] Xiao | Reconstructing the worlds museums[END_REF] and semantic scene understanding [START_REF] Armeni | 3D Semantic Parsing of Large-Scale Indoor Spaces[END_REF][START_REF] Chang | Matterport3D: Learning from RGB-D Data in Indoor Environments[END_REF][START_REF] Dai | ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes[END_REF][START_REF] Xiao | SUN3D: A database of big spaces reconstructed using sfm and object labels[END_REF] and are not suitable for localization. To address this issue, we create a new dataset for indoor localization that, in contrast to other existing indoor localization datasets [START_REF] Armeni | 3D Semantic Parsing of Large-Scale Indoor Spaces[END_REF][START_REF] Glocker | Real-time RGB-D camera relocalization[END_REF][START_REF] Sun | A Dataset for Benchmarking Image-based Localization[END_REF], has two important properties. First, the dataset is large-scale, capturing two university buildings. Second, the query images are acquired using a smartphone at a time months apart from the date of capture of the reference 3D model. As a result, the query images and the reference 3D model often contain large changes in scene appearance due to the different layout of furniture, occluders (people), and illumination, representing a realistic and challenging indoor localization scenario.

Contributions. Our contributions are three-fold. First, we develop a novel visual localization approach suitable for large-scale indoor environments. The key novelty of our approach lies in carefully introducing dense feature extraction and matching in a sequence of progressively stricter verification steps. To the best of our knowledge, the present work is the first to clearly demonstrate the benefit of dense data association for indoor localization. Second, we create a new dataset suitably designed for large-scale indoor localization that contains large variation in appearance between queries and the 3D database due to large viewpoint changes, moving furniture, occluders or changing illumination. The query images are taken at a different time from the reference database, using a handheld device, and at different moments of the day, to capture enough variability, bridging the gap to realistic usage scenarios. The code and data are publicly available on the project page [START_REF]Project webpage[END_REF]. Third, the proposed method shows a solid improvement over existing state-ofthe-art results, showing an absolute improvement of 17-20% in the percent of correctly localized queries within a 0.25 -0.5 m error, which is of high importance for indoor localization.

Related work

We next review previous work on visual localization.

Image retrieval based localization. Visual localization in large-scale urban environments is often approached as an image retrieval problem. The location of a given query image is predicted by transferring the geotag of the most similar image retrieved from a geotagged database [6,[START_REF] Arandjelović | Dislocation: Scalable descriptor distinctiveness for location recognition[END_REF][START_REF] Chen | Residual enhanced visual vectors for on-device image matching[END_REF][START_REF] Kim | Learned contextual feature reweighting for image geo-localization[END_REF][START_REF] Sattler | Large-scale location recognition and the geometric burstiness problem[END_REF][START_REF] Torii | Pajdla. 24/7 place recognition by view synthesis[END_REF][START_REF] Torii | Visual place recognition with repetitive structures[END_REF]. This approach scales to entire cities thanks to compact image descriptors and efficient indexing techniques [START_REF] Arandjelović | Three things everyone should know to improve object retrieval[END_REF][START_REF] Arandjelovic | All about vlad[END_REF][START_REF] Chum | Total recall: Automatic query expansion with a generative feature model for object retrieval[END_REF][START_REF] Jegou | Hamming embedding and weak geometric consistency for large scale image search[END_REF][START_REF] Jegou | Aggregating local image descriptors into compact codes[END_REF][START_REF] Nister | Scalable recognition with a vocabulary tree[END_REF][START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF][START_REF] Van Gemert | Visual word ambiguity[END_REF] and can be further improved by spatial re-ranking [START_REF] Philbin | Object retrieval with large vocabularies and fast spatial matching[END_REF], informative feature selection [START_REF] Chum | Total recall ii: Query expansion revisited[END_REF][START_REF] Chum | Total recall: Automatic query expansion with a generative feature model for object retrieval[END_REF] or feature weighting [START_REF] Gronat | Learning and calibrating per-location classifiers for visual place recognition[END_REF][START_REF] Jégou | On the burstiness of visual elements[END_REF][START_REF] Sattler | Large-scale location recognition and the geometric burstiness problem[END_REF][START_REF] Torii | Visual place recognition with repetitive structures[END_REF]. Most of the above methods are based on image representations using sparsely sampled local invariant features. While these representations have been very successful, outdoor image-based localization has recently also been approached using densely sampled local descriptors [START_REF] Torii | Pajdla. 24/7 place recognition by view synthesis[END_REF] or (densely extracted) descriptors based on convolutional neural networks [6,[START_REF] Kim | Learned contextual feature reweighting for image geo-localization[END_REF][START_REF] Lin | Bilinear cnn models for fine-grained visual recognition[END_REF][START_REF] Weyand | Planet-photo geolocation with convolutional neural networks[END_REF]. However, the main shortcoming of all the above methods is that they output only an approximate location of the query, not an exact 6DoF pose.

Visual localization using 3D maps. Another approach is to directly obtain 6DoF camera pose with respect to a pre-built 3D map. The map is usually composed of a 3D point cloud constructed via Structure-from-Motion (SfM) [START_REF] Agarwal | Building rome in a day[END_REF] where each 3D point is associated with one or more local feature descriptors. The query pose is then obtained by feature matching and solving a Perspective-n-Point problem (PnP) [START_REF] Camposeco | Toroidal constraints for two-point localization under high outlier ratios[END_REF][START_REF] Cao | Minimal scene descriptions from structure from motion models[END_REF][START_REF] Choudhary | Visibility probability structure from sfm datasets and applications[END_REF][START_REF] Irschara | From structure-from-motion point clouds to fast location recognition[END_REF][START_REF] Kendall | Geometric loss functions for camera pose regression with deep learning[END_REF][START_REF] Li | Worldwide pose estimation using 3d point clouds[END_REF][START_REF] Sattler | Hyperpoints and fine vocabularies for largescale location recognition[END_REF][START_REF] Sattler | Efficient & effective prioritized matching for large-scale image-based localization[END_REF]. Alternatively, pose estimation can be formulated as a learning problem, where the goal is to train a regressor from the input RGB(D) space to camera pose parameters [START_REF] Brachmann | DSAC -Differentiable RANSAC for Camera Localization[END_REF][START_REF] Kendall | Geometric loss functions for camera pose regression with deep learning[END_REF][START_REF] Shotton | Scene coordinate regression forests for camera relocalization in RGB-D images[END_REF][START_REF] Walch | Image-Based Localization Using LSTMs for Structured Feature Correlation[END_REF]. While promising, scaling these methods to large-scale datasets is still an open challenge.

Indoor 3D maps. Indoor scene datasets [START_REF] Pandey | Scene recognition and weakly supervised object localization with deformable part-based models[END_REF][START_REF] Quattoni | Recognizing indoor scenes[END_REF][START_REF] Singh | Unsupervised discovery of mid-level discriminative patches[END_REF][START_REF] Torralba | Context-based vision systems for place and object recognition[END_REF] have been introduced for tasks such scene recognition, classification, and object retrieval. With the increased availability of laser range scanners and time-of-flight (ToF) sensors, several datasets include depth data besides RGB images [START_REF] Anand | Contextually guided semantic labeling and search for threedimensional point clouds[END_REF][START_REF] Armeni | 3D Semantic Parsing of Large-Scale Indoor Spaces[END_REF][START_REF] Dai | ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes[END_REF][START_REF] Glocker | Real-time RGB-D camera relocalization[END_REF][START_REF] Lai | Unsupervised feature learning for 3D scene labeling[END_REF][START_REF] Silberman | Indoor segmentation and support inference from RGBD images[END_REF][START_REF] Xiao | SUN3D: A database of big spaces reconstructed using sfm and object labels[END_REF] and some of these datasets also provide reference camera poses registered into the 3D point cloud [START_REF] Armeni | 3D Semantic Parsing of Large-Scale Indoor Spaces[END_REF][START_REF] Glocker | Real-time RGB-D camera relocalization[END_REF][START_REF] Xiao | SUN3D: A database of big spaces reconstructed using sfm and object labels[END_REF], though their focus is not on localization. Datasets focused specifically on indoor localization [START_REF] Shotton | Scene coordinate regression forests for camera relocalization in RGB-D images[END_REF][START_REF] Sun | A Dataset for Benchmarking Image-based Localization[END_REF][START_REF] Valentin | Learning to Navigate the Energy Landscape[END_REF] have so far captured fairly small spaces such as a single room (or a single floor at largest) and have been constructed from densely-captured sequences of RGBD images. More recent datasets [START_REF] Chang | Matterport3D: Learning from RGB-D Data in Indoor Environments[END_REF][START_REF] Wijmans | Exploiting 2D floorplan for building-scale panorama RGBD alignment[END_REF] provide larger scale (multi-floor) indoor 3D maps containing RGBD images registered to a global floor map. However, they are designed for object retrieval, 3D reconstruction, or training deep-learning architectures. Most importantly, they do not contain query images taken from viewpoints far from database images, which are necessary for evaluating visual localization.

To address the shortcomings of the above datasets for large-scale indoor visual localization, we introduce a new dataset that includes query images captured at a different time from the database, taken from a wide range of viewpoints, with a considerably larger 3D database distributed across multiple floors of multiple buildings. Furthermore, our dataset contains various difficult situations for visual localization, e.g., textureless and highly symmetric office scenes, repetitive tiles, and repetitive objects that confuse the existing visual localization methods designed for outdoor scenes. The newly collected dataset is described next.

The InLoc dataset for visual localization

Our dataset is composed of a database of RGBD images geometrically registered to the floor maps augmented with a separate set of RGB query images taken by hand-held devices to make it suitable for the task of indoor localization (Figure 2). The provided query images are annotated with manually verified ground-truth 6DoF camera poses (reference poses) in the global coordinate system of the 3D map.

Database. The base indoor RGBD dataset [START_REF] Wijmans | Exploiting 2D floorplan for building-scale panorama RGBD alignment[END_REF] 277 RGBD panoramic images obtained from scanning two buildings at the Washington University in St. Louis with a Faro 3D scanner. Each RGBD panorama has about 40M 3D points in color. The base images are divided into five scenes: DUC1, DUC2, CSE3, CSE4, and CSE5, representing five floors of the mentioned buildings, and are geometrically registered to a known floor plan [START_REF] Wijmans | Exploiting 2D floorplan for building-scale panorama RGBD alignment[END_REF]. The scenes are scanned sparsely on purpose, to cover a larger area with a small number of scans to reduce the required manual work, as well as due to the long operating times of the high-end scanner used. The area per scan varies between 23.5 and 185.8 m2 . This inherently leads to critical view changes between query and database images when compared with other existing datasets [START_REF] Sun | A Dataset for Benchmarking Image-based Localization[END_REF][START_REF] Valentin | Learning to Navigate the Energy Landscape[END_REF][START_REF] Wang | Lost shopping! monocular localization in large indoor spaces[END_REF] 2 .

For creating an image database suitable for indoor visual localization evaluation, a set of perspective images is generated by following the best practices from outdoor visual localization [START_REF] Chen | City-scale landmark identification on mobile devices[END_REF][START_REF] Torii | Pajdla. 24/7 place recognition by view synthesis[END_REF][START_REF] Zamir | Accurate image localization based on google maps street view[END_REF]. We obtain 36 perspective RGBD images from each panorama by extracting standard perspective views (60 • FoV) with a sampling stride of 30 • in yaw and ±30 • in pitch directions, resulting in 10K perspective images in total (Table 1). Our database contains significant challenges, such as repetitive patterns (stairs, pillars), frequently appearing building structures (doors, windows), furniture changing position, people moving across the scene, and textureless and highly symmetric areas (walls, floors, corridors, classrooms, open spaces). Query images. We captured 356 photos using a smartphone camera (iPhone 7), distributed only across two floors, DUC1 and DUC2. The other three floors in the database are not represented in the query images, and play the role We evaluated the quality of the reference camera poses both visually and quantitatively, as described in section 3. Red dots are the database 3D points projected onto a query image using its estimated pose.

of confusers at search time, contributing to the buildingscale localization scenario. Note that these query photos are taken at different times of the day, to capture the variety of occluders and layouts (e.g., people, furniture) as well as illumination changes.

Reference pose generation. For all query photos, we estimate 6DoF reference camera poses w.r.t. the 3D map. Each query camera reference pose is computed as follows:

(i) Selection of the visually most similar database images.

For each query, we manually select one panorama location which is visually most similar to the query image using the perspective images generated from the panorama.

(ii) Automatic matching of query images to selected database images. We match the query and perspective images by using affine covariant features [START_REF] Mikolajczyk | Scale & affine invariant interest point detectors[END_REF] and nearestneighbor search followed by Lowe's ratio test [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF].

(iii) Computing the query camera pose and visually verifying the reprojection. All the panoramas (and perspective images) are already registered to the floor plan and have pixel-wise depth information. Therefore, we compute query pose via P3P-RANSAC [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF], followed by bundle adjustment [START_REF] Agarwal | Ceres solver[END_REF], using correspondences between query image points and scene 3D points obtained by feature matching. We evaluate the obtained poses visually by inspecting the reprojection of edges detected in the corresponding RGB panorama into the query image (see examples in figure 3).

(iv) Manual matching of difficult queries to selected database images. Pose estimation from automatic matches often gives inaccurate poses for difficult queries which are, e.g., far from any database image. Hence, for queries with significant misalignment in reprojected edges, we manually annotate 5 to 20 correspondences between image pixels and 3D points and apply step (iii) on the manual matches.

(v) Quantitative and visual inspection. For all estimated poses, we measure the median reprojection error, computed as the distance of the reprojected 3D database point to the nearest edge pixel detected in the query image, after removing correspondences with gross errors (with distance over 20 pixels) due to, e.g., occlusions. For query images that have under 5 pixels median reprojection error, we manually inspect the reprojected edges in the query image and finally accept 329 reference poses out of the 356 query images.

Indoor visual localization with dense matching and view synthesis

We propose a new method for large-scale indoor visual localization. We address the three main challenges of indoor environments:

(1) Lack of sparse local features. Indoor environments are full of large textureless areas, e.g., walls, ceilings, floors and windows, where sparse feature extraction methods detect very few features. To overcome this problem, we use multi-scale dense CNN features for both image description and feature matching. Our features are generic enough to be pre-trained beforehand on (outdoor) scenes, avoiding costly re-training, e.g., as in [START_REF] Brachmann | DSAC -Differentiable RANSAC for Camera Localization[END_REF][START_REF] Kendall | Geometric loss functions for camera pose regression with deep learning[END_REF][START_REF] Walch | Image-Based Localization Using LSTMs for Structured Feature Correlation[END_REF], of the localization machine for each particular environment.

(2) Large image changes. Indoor environments are cluttered with movable objects, e.g., furniture and people, and 3D structures, e.g., pillars add concave bays, causing severe occlusions when viewed from a close distance. The most similar images obtained by retrieval may therefore be visually very different from a query image. To overcome this problem, we rely on dense feature matches to collect as much positive evidence as possible. We employ image descriptors extracted from a convolutional neural network that can match higher-level structures of the scene rather than relying on matching individual local features. In detail, our pose estimation step performs coarse-to-fine dense feature matching, followed by geometric verification and estimation of the camera pose using P3P-RANSAC.

(3) Self-similarity. Indoor environments are often very self-similar, e.g., due to many symmetric and repetitive elements on a large and small scale (corridors, rooms, tiles, windows, chairs, doors, etc.). Existing matching strategies count the positive evidence, i.e., how much of the image (or how many inliers) have been matched, to decide whether two images match. This is, however, problematic as large textureless areas can be matched well, hence providing strong (incorrect) positive evidence. To overcome this problem, we propose to count also the negative evidence, i.e., what portion of the image does not match, to decide whether two views are taken from the same location.

To achieve this, we perform explicit pose estimate verification based on view synthesis. In detail, we compare the query image with a virtual view of the 3D model rendered from the estimated camera pose of the query. This novel approach takes advantage of the high quality of the RGBD image database and incorporates both the positive and negative evidence by counting matching and non-matching pixels across the entire query image. As shown by our experiments, this approach is orthogonal to the choice of local descriptors. The proposed verification by view synthesis is consistently showing a significant improvement regardless of the choice of features used for estimating the pose. The pipeline of InLoc has the following three steps. Given a query image, (1) we obtain a set of candidate images by finding the N best matching images from the reference image database registered to the map. (2) For these N retrieved candidate images, we compute the query poses using the associated 3D information that is stored together with the database images. (3) Finally, we re-rank the computed camera poses based on verification by view synthesis. The three steps are detailed next.

Candidate pose retrieval

As demonstrated by existing work [6,[START_REF] Kim | Learned contextual feature reweighting for image geo-localization[END_REF][START_REF] Torii | Pajdla. 24/7 place recognition by view synthesis[END_REF], aggregating feature descriptors computed densely on a regular grid mitigates issues such as a lack of repeatability of local features detected on textureless scenes, large-illumination changes, and a lack of discriminability of image description, dominated by features from repetitive structures (burstiness). As already mentioned in section 1, these problems are also occurring in large-scale indoor localization, which motivates our choice of using an image descriptor based on dense feature aggregation. Both query and database images are described by NetVLAD [6] (but other variants could also be used), normalized L2 distances of the descriptors are computed, and the poses of the N best matching images from the database are chosen as candidate poses. In section 5, we compare our approach with the state-of-the-art image descriptors based on local feature detection and show benefits of our approach for indoor localization.

Pose estimation using dense matching

A severe problem in indoor localization is that standard geometric verification based on local feature detection [START_REF] Philbin | Object retrieval with large vocabularies and fast spatial matching[END_REF][START_REF] Sattler | Large-scale location recognition and the geometric burstiness problem[END_REF] does not work on textureless or self-repetitive scenes, such as corridors, where robots (and also humans) often get lost. Motivated by the improvements in candidate pose retrieval with dense feature aggregation (Section 4.1), we use features densely extracted on a regular grid for verifying and re-ranking the candidate images by feature matching and pose estimation. A possible approach would be to match DenseSIFT [START_REF] Liu | Sift flow: Dense correspondence across different scenes[END_REF] followed by RANSAC-based verification. Instead of tailoring DenseSIFT description parameters (patch sizes, strides, scales) to match across images with significant viewpoint changes, we use an image representation extracted by a convolutional neural network (VGG-16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]) as a set of multi-scale features extracted on a regular grid that describes more higher-level information with a larger receptive field (patch size).

We first find geometrically consistent sets of correspondences using the coarser conv5 layer containing high-level information. Then we refine the correspondence by search-ing for additional matches on the conv3 layer. Examples in figure 4 demonstrate that our dense CNN matching (4th column) obtains better matches in indoor environments when compared to matching standard local features (3rd column), even for less-textured areas. Notice that dense-feature extraction and description requires no additional computation at query time as the intermediate convolutional layers are already computed when extracting the NetVLAD descriptors as described in section 4.1. As will also be demonstrated in section 5, memory requirements and computational speed of feature matching can be addressed by binarizing the convolutional features without loss in matching performance.

As perspective images in our database have depth values, and hence associated 3D points, the query camera pose can be estimated by finding pixel-to-pixel correspondences between the query and the matching database image followed by P3P-RANSAC [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF].

Pose verification with view synthesis

We propose here to collect both positive and negative evidence to determine what is and is not matched 3 . This is achieved by harnessing the power of the high-quality RGBD image database that provides a dense and accurate 3D structure of the indoor environment. This structure is used to render a virtual view that shows how the scene would look like from the estimated query pose. The rendered image enables us to count, in a pixel-wise manner, both positive and negative evidence by counting which regions are and are not consistent between the query image and the underlying 3D structure. To gain invariance to illumination changes and small misalignments, we evaluate image similarity by comparing local patch descriptors (DenseRootSIFT [START_REF] Arandjelović | Three things everyone should know to improve object retrieval[END_REF][START_REF] Liu | Sift flow: Dense correspondence across different scenes[END_REF]) at corresponding pixel locations. The final similarity is computed as the median of descriptor distances across the entire image while ignoring areas with missing 3D structure.

Experiments

We first describe the experimental setup for evaluating visual localization performance using our dataset (Section 5.1). The proposed method, termed "InLoc", is compared with state-of-the-art methods (Section 5.2) and we show the benefits of each component in detail (Section 5.3).

Implementation details

In the candidate pose retrieval step, we retrieve 100 candidate database images using NetVLAD. We use the implementation provided by the authors and the pre-trained Pitts30K [6] VGG-16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] In the second pose estimation step, we obtain tentative correspondences by matching densely extracted convolutional features in a coarse-to-fine manner: we first find mutually nearest matches among the conv5 features and then find matches in the finer conv3 features restricted by the coarse conv5 correspondences. The tentative matches are geometrically verified by estimating up to two homographies using RANSAC [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF]. We re-rank the 100 candidates using the number of RANSAC inliers and keep the top-10 database images. For each of the 10 images, the 6DoF query pose is computed by P3P-LO-RANSAC [START_REF] Lebeda | Fixing the locally optimized ransac-full experimental evaluation[END_REF] (referred to as DensePE), assuming a known focal length, e.g., from EXIF data, using the inlier matches and depth (i.e. the 3D structure) associated to each database image.

In the final pose verification step, we generate synthesized views by rendering colored 3D points while taking care of self-occlusions. For computing the scores that measure the similarities of the query image and the image rendered from the estimated pose, we use the Dens-eSIFT extractor and its RootSIFT descriptor [START_REF] Arandjelović | Three things everyone should know to improve object retrieval[END_REF][START_REF] Liu | Sift flow: Dense correspondence across different scenes[END_REF] from VLFeat [START_REF] Vedaldi | VLFeat -an open and portable library of computer vision algorithms[END_REF] 4 . Finally, we localize the query image by the 4 When computing the descriptors, the blank pixels induced by missing 3D points are filled by linear inter(/extra)-polation using the values of nonblank pixels on the boundary. best pose among its top-10 candidates.

Evaluation metrics. We evaluate the localization accuracy as the consistency of the estimated poses with our reference poses. We measure positional and angular differences in meters and degrees between the estimated poses and the manually verified reference poses.

Comparison with the state-of-the-art methods

Direct 2D-3D matching [START_REF] Sattler | Hyperpoints and fine vocabularies for largescale location recognition[END_REF][START_REF] Sattler | Efficient & effective prioritized matching for large-scale image-based localization[END_REF]. We first compare with a variation5 of a state-of-the-art 3D structure-based image localization approach [START_REF] Sattler | Hyperpoints and fine vocabularies for largescale location recognition[END_REF]. We compute affine covariant RootSIFT features for all the database images and associate them with 3D coordinates via the known scene geometry. Features extracted from a query image are then matched to the database 3D descriptors [START_REF] Muja | Fast approximate nearest neighbors with automatic algorithmic configuration[END_REF]. We select at most five database images receiving the largest numbers of matches and use all these matches together for pose estimation. Similar to [START_REF] Sattler | Hyperpoints and fine vocabularies for largescale location recognition[END_REF], we did not apply Lowe's ratio test [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] as it lowered the performance. The 6DoF query pose is finally computed by P3P-LO-RANSAC [START_REF] Lebeda | Fixing the locally optimized ransac-full experimental evaluation[END_REF]. As shown in table Loc outperforms direct 2D-3D matching by a large margin (40.7% at the localization accuracy of 0.5m). We believe that this is because our large-scale indoor dataset involves many distractors and large viewpoint changes that present a major challenge for 3D structure-based methods.

Disloc [START_REF] Arandjelović | Dislocation: Scalable descriptor distinctiveness for location recognition[END_REF] + sparse pose estimation (SparsePE) [START_REF] Philbin | Object retrieval with large vocabularies and fast spatial matching[END_REF]. We next compare with the state-of-the-art image retrieval-based localization method. Disloc represents images using bagof-visual-words with Hamming-Embedding [START_REF] Jegou | Hamming embedding and weak geometric consistency for large scale image search[END_REF] while also taking local descriptor space density into account. We use a publicly available implementation [START_REF] Sattler | Large-scale location recognition and the geometric burstiness problem[END_REF] of Disloc with a 200K vocabulary trained on affine covariant features [START_REF] Mikolajczyk | Scale & affine invariant interest point detectors[END_REF], described by RootSIFT [START_REF] Arandjelović | Three things everyone should know to improve object retrieval[END_REF], extracted from the database images of our indoor dataset. The top-100 candidate images shortlisted by Disloc are re-ranked by spatial verification [START_REF] Philbin | Object retrieval with large vocabularies and fast spatial matching[END_REF] using (sparse) affine covariant features [START_REF] Mikolajczyk | Scale & affine invariant interest point detectors[END_REF].

The ratio test [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] was not applied here as it was removing too many features that need to be retained in the indoor scenario. Using the inliers, the 6DoF query pose is computed with P3P-LO-RANSAC [START_REF] Lebeda | Fixing the locally optimized ransac-full experimental evaluation[END_REF]. To make a fair comparison, we use exactly the same features and P3P-LO-RANSAC for pose estimation as the direct 2D-3D matching method described above. As shown in table 2, Disloc [START_REF] Arandjelović | Dislocation: Scalable descriptor distinctiveness for location recognition[END_REF]+SparsePE [START_REF] Philbin | Object retrieval with large vocabularies and fast spatial matching[END_REF] results in a 13.7% performance gain compared to Direct 2D-3D matching [START_REF] Sattler | Efficient & effective prioritized matching for large-scale image-based localization[END_REF]. This can be attributed to the image retrieval step that discounts burst of repetitive features. However, the results are still significantly worse compared to our InLoc approach.

NetVLAD [6] + sparse pose estimation (SparsePE) [START_REF] Philbin | Object retrieval with large vocabularies and fast spatial matching[END_REF].

We also evaluate a variation of the above image retrievalbased localization method. Here the candidate shortlist is obtained by NetVLAD [6], which is then re-ranked using SparsePE [START_REF] Philbin | Object retrieval with large vocabularies and fast spatial matching[END_REF], followed by pose estimation using P3P-LO-RANSAC [START_REF] Lebeda | Fixing the locally optimized ransac-full experimental evaluation[END_REF]. This is a strong baseline building on the state-of-the-art place recognition results obtained by [6]. Interestingly, as shown in table 2, there is no significant difference between NetVLAD+SparsePE and Dis-Loc+SparsePE, which is in line with results reported in outdoor settings [START_REF] Sattler | Are large-scale 3D models really necessary for accurate visual localization?[END_REF]. Yet, NetVLAD outperforms Dis-Loc (5.8% at the localization accuracy of 0.5m) before reranking via SparsePE (c.f . figure 5) in this indoor setting (see also figure 4). Overall, both methods, even though they represent the state-of-the-art in outdoor localization, still perform significantly worse than our proposed approach based on dense feature matching and view synthesis.

Evaluation of each component

Next, we demonstrate the benefits of the individual components of our approach.

Benefits of pose estimation using dense matching. Using the NetVLAD retrieval as the base retrieval method (Figure 5 (a)), our pose estimation with dense matching (NetVLAD [6]+DensePE (blue line)) constantly improves the localization rate by about 15% when compared to the state-of-the-art sparse local feature matching (NetVLAD [6]+SparsePE (green line)). This result supports our conclusion that dense feature matching and verification is superior to sparse feature matching for often weakly textured indoor scenes. This effect is also clearly demonstrated in qualitative results in figure 4 (cf. columns 3 and 4).

Benefits of pose verification with view synthesis. We apply our pose verification step (DensePV) to the top-10 pose estimates obtained by different spatial re-reranking methods. Results are shown in figure 5 and demonstrate significant and consistent improvements obtained by our pose verification approach (compare "-•-" to "-" in figure 5). Improvements are most pronounced for the position accuracy within 1.5 meters (13% or more).

Binarized representation. A binary representation (instead of floats) of features in the intermediate CNN layers significantly reduces memory requirements. We use feature binarization that follows the standard Hamming embedding approach [START_REF] Jegou | Hamming embedding and weak geometric consistency for large scale image search[END_REF] but without dimensionality reduction. Matching is then performed by computing Hamming distances. This simple binarization scheme results in a negligible performance loss (less than 1% at 0.5 meters) compared to the original descriptors, which is in line with results reported for object recognition [START_REF] Agrawal | Analyzing the performance of multilayer neural networks for object recognition[END_REF]. At the same time, binarization reduces the memory requirements by a factor of 32, compressing 428GB of original descriptors to just 13.4GB.

Comparison with learning based localization methods.

We have attempted a comparison with DSAC [START_REF] Brachmann | DSAC -Differentiable RANSAC for Camera Localization[END_REF], which is a state-of-the-art pose estimator for indoor scenes. Despite our best efforts, training DSAC on our indoor dataset failed to converge. We believe this is because the RGBD scans in our database are sparsely distributed [START_REF] Wijmans | Exploiting 2D floorplan for building-scale panorama RGBD alignment[END_REF] and each scan has only a small overlap with neighboring scans. Training on such a dataset is challenging for methods designed for densely captured RGBD sequences [START_REF] Glocker | Real-time RGB-D camera relocalization[END_REF]. We believe this would also be the case for PoseNet [START_REF] Kendall | Geometric loss functions for camera pose regression with deep learning[END_REF], another method for CNN-based pose regression. We do provide the comparison with DSAC and PoseNet on much smaller datasets next.

Direct2D-3D DisLoc [START_REF] Arandjelović | Dislocation: Scalable descriptor distinctiveness for location recognition[END_REF] DisLoc [START_REF] Arandjelović | Dislocation: Scalable descriptor distinctiveness for location recognition[END_REF] + DensePV DisLoc [START_REF] Arandjelović | Dislocation: Scalable descriptor distinctiveness for location recognition[END_REF] DisLoc [START_REF] Arandjelović | Dislocation: Scalable descriptor distinctiveness for location recognition[END_REF] + DensePV + SparsePE + SparsePE NetVLAD [6] NetVLAD [6] + SparsePE + SparsePE

NetVLAD [6] NetVLAD [6] NetVLAD [START_REF] Glocker | Real-time RGB-D camera relocalization[END_REF][START_REF] Shotton | Scene coordinate regression forests for camera relocalization in RGB-D images[END_REF]. Numbers show the median positional (cm) and angular errors (degrees).

Evaluation on other datasets

We also evaluate InLoc on two existing indoor datasets [START_REF] Chang | Matterport3D: Learning from RGB-D Data in Indoor Environments[END_REF][START_REF] Shotton | Scene coordinate regression forests for camera relocalization in RGB-D images[END_REF] to confirm the relevance of our results. The Matter-port3D [START_REF] Chang | Matterport3D: Learning from RGB-D Data in Indoor Environments[END_REF] dataset consists of RGBD scans of 90 buildings. Each RGBD scan contains 18 images that capture the scene around the scan position with known camera poses. We created a test set by randomly choosing 10% of the scan positions and selected their horizontal views. This resulted in 58,074 database images and a query set of 6,726 images. Results are shown in table 3. Our approach (InLoc) outperforms the baselines, which is in line with results on the InLoc dataset. We also tested PoseNet [START_REF] Kendall | Geometric loss functions for camera pose regression with deep learning[END_REF] and DSAC [START_REF] Brachmann | DSAC -Differentiable RANSAC for Camera Localization[END_REF] on a single (the largest) building. The test set is created in the same manner as above and contains 1,884 database images and 210 query images. Even in this much easier case, DSAC fails to converge. PoseNet produces large localization errors (24.8 meters and 80.0 degrees) in comparison with InLoc (0.26 meters and 2.78 degrees). We also report results on the 7 Scenes dataset [START_REF] Glocker | Real-time RGB-D camera relocalization[END_REF][START_REF] Shotton | Scene coordinate regression forests for camera relocalization in RGB-D images[END_REF]] which is, while relatively small, a standard benchmark for indoor localization. The 7 Scenes dataset [START_REF] Shotton | Scene coordinate regression forests for camera relocalization in RGB-D images[END_REF] consists of geometrically-registered video frames representing seven scenes, together with associated depth images and camera poses. Table 4 shows localization results for our approach (NetVLAD+DensePE) compared with state-of-theart methods [START_REF] Brachmann | DSAC -Differentiable RANSAC for Camera Localization[END_REF][START_REF] Kendall | Geometric loss functions for camera pose regression with deep learning[END_REF][START_REF] Sattler | Efficient & effective prioritized matching for large-scale image-based localization[END_REF]. Note that our approach performs comparably to these methods on this relatively small and densely captured data, while it does not need any scene specific training (which is needed by [START_REF] Brachmann | DSAC -Differentiable RANSAC for Camera Localization[END_REF][START_REF] Kendall | Geometric loss functions for camera pose regression with deep learning[END_REF]).

Conclusion

We have presented InLoc -a new approach for large-scale indoor visual localization that estimates the 6DoF camera pose of a query image with respect to a large indoor 3D map. To overcome the difficulties of indoor camera pose estimation, we have developed new pose estimation and verification methods that use dense feature extraction and matching in a sequence of progressively stricter verification steps. The localization performance is evaluated on a new large indoor dataset with realistic and challenging query images captured by mobile phones. Our results demonstrate significant improvements compared to state-of-theart localization methods. To encourage further progress on high-accuracy large-scale indoor localization, we make our dataset publicly available [START_REF]Project webpage[END_REF].

Figure A shows the 3D maps (grey dots), the 329 reference poses of the query images (blue dots), and the 129 database scan positions (red circles) in our InLoc dataset. The query images are distributed across two floors (DUC1 and DUC2) that cover an area of ≈ 100,000 f t 2 (9,290 m 2 ) each [START_REF] Wijmans | Exploiting 2D floorplan for building-scale panorama RGBD alignment[END_REF], and are taken from significantly distant positions from database scans.

Figure B illustrates the verification process for the reference poses. We show example query images on the 1st and 3rd row. The edges extracted on the best matching database image were reprojected on the query image (2nd and 4th row) to verify the quality of the reference poses. The manually and visually verified reference poses, in total 329, have at most 5 pixels median re-projection error, out of which, 101 reference poses have median re-projection error below 1 pixel.

B. Qualitative results

In what follows, we will consider the query image correctly localized, if the error for the estimated pose is within 1 meter and 5 • with respect to the reference pose.

We first consider situations in which InLoc successfully localizes the query images, while the state-of-theart NetVLAD+SparsePE fails. Figure C shows qualitative examples of the results obtained by NetVLAD+SparsePE (a,c,e) versus our InLoc (b,d,f). As shown in (a) and (c), sparse features are often detected on highly repetitive structures e.g., fonts (text), textured surfaces (fabric pattern on the sofa). As shown in (a) for the baseline, matching features found on such objects can result in matches with unrelated parts of the scene, leading to incorrect camera pose estimates. The fact that sparse features are predominantly found in few textured regions leads to problems in the largely untextured indoor scenes. This is shown in (e), where matches are found only in a small part of the query image, which leads to an unstable configuration for camera pose estimation. This, in turn, leads to more stable pose estimates in (b), (d), and (f). Our pose verification, DensePV (section 4.3), allows us to identify incorrect poses, resulting from features found on repetitive structures, since most parts of the image rendered from a false pose are not consistent with the query image. Thus, InLoc is better suited to handle highly repetitive indoor scenes with rich feature correspondences.

The next set of qualitative results demonstrates the benefits of dense pose verification. For this, figure D compares results obtained by InLoc (b,d,f) with results obtained by baseline NetVLAD+DensePE (a,c,e). In this case, the baseline NetVLAD+DensePE uses our dense matching (DensePE) but selects the best pose based only on the number of inlier matches and not using our pose verification by virtual view synthesis (DensePV). For scenes dominated by symmetries and repetitive structures (a,c) or largely textureless regions (a, e), there can be a large amount of geometrically consistent matches even for unrelated database images. This still holds true even if matches are obtained by dense features and geometrically verified. Our dense pose verification strategy using synthesized images (b,d,f) effectively provides "negative" evidence in such situations. The error maps (bottom row) clearly show that it detects (in)consistent areas between the query and its synthesized image. Limitations. Our pose verification (section 4.3) evaluates the estimated camera pose by dense pixel-level matching between the query image and the synthesized view. This verification is robust up to a certain level of scene changes, e.g., illumination changes and some amount of misalignment, but cannot deal with extreme changes in the scene such as very large occlusions or when the view is dominated by moving objects.

Figure E shows typical failure cases of InLoc, due to our pose verification not being able to identify the correct pose in highly dynamic scenes. In both cases, the query images capture many moving objects, e.g., people (a) or chairs (b), and highly dynamic scenes, e.g., opened/closed shutters (a) or pictures on the wall/removed (b). These moving objects cover a large part of the image.

Those are the remaining open-issues that can be potentially addressed by adopting further semantic information [START_REF] Anand | Contextually guided semantic labeling and search for threedimensional point clouds[END_REF][START_REF] Kim | Learned contextual feature reweighting for image geo-localization[END_REF].

Figure 2 .

 2 Figure 2. Example images from InLoc dataset. (Top) Database images. (Bottom) Query images. The selected images show the challenges encountered in indoor environments: even small changes in viewpoint lead to large differences in appearance; large textureless surfaces (e.g. walls); self-repetitive structures (e.g. corridors); significant variation throughout the day due to different illumination sources (e.g., active vs. indirect illumination).

Figure 3 .

 3 Figure 3. Examples of verified query poses. We evaluated the quality of the reference camera poses both visually and quantitatively, as described in section 3. Red dots are the database 3D points projected onto a query image using its estimated pose.

Figure 4 .

 4 Figure 4. Qualitative comparison of different localization methods (columns). From top to bottom: query image, the best matching database image, synthesized view at the estimated pose (without inter/extra-polation), error map between the query image and the synthesized view, localization error (meters, degrees). Green dots are the inlier matches obtained by P3P-LO-RANSAC. Methods using the proposed dense pose estimation (DensePE) and dense pose verification (DensePV) are shown in bold. The query images in the 2nd, 4th and 6th column are well localized within 1.0 meters and 5.0 degrees whereas localization results in the 1st, 3rd and 5th column are incorrect.

Figure 5 .

 5 Figure 5. Impact of different components. The graphs show impact of dense matching (DensePE) and dense pose verification (DensePV) on pose estimation quality for (a) the pose candidates retrieved by NetVLAD and (b) state-of-the-art baselines. Plots show the fraction of correctly localized queries (y-axis) within a certain distance (x-axis) whose rotation error is at most 10 • .

Figure C .

 C Figure C. Qualitative comparison between InLoc and NetVLAD+SparsePE. In these examples, InLoc successfully localizes a query image within 1 meter distance error and 5 • angular error with respect to the reference pose whereas the state-of-the-art NetVLAD+SparsePE fails. From top to bottom: query image, the best matching database image, synthesized view rendered from the estimated pose, error map between the query image and the synthesized view, localization error (meters, degrees). Warm colors correspond to large errors. Green dots are the inlier matches obtained by P3P-LO-RANSAC.

Table 1 .

 1 consists of Number Image size [pixel] FoV [degree] Statistics of the InLoc dataset.

	Query	356	4,032×3,024	65.57
	Database	9,972	1,600×1,200	60

  model to generate 4, 096dimensional NetVLAD descriptor vectors.

8.39, 152.74 • 0.43, 2.05 • 0.27, 17.43 • 0.20, 0.72 • 7.97, 2.04 • 0.13, 1.95 • Disloc [9] NetVLAD [6] NetVLAD [6]+ NetVLAD [6]+ NetVLAD [6]+ InLoc: NetVLAD [6]+ SparsePE DensePE DensePE DensePE+DensePV

Table 2 .

 2 Comparison with the state-of-the-art localization methods on the InLoc dataset. We show the rate (%) of correctly localized queries within a given distance (m) threshold and within a 10 • angular error threshold.

	2, In-

Table 3 .

 3 Comparison on Matterport3D[START_REF] Chang | Matterport3D: Learning from RGB-D Data in Indoor Environments[END_REF]. Numbers show the median positional (m) and angular (degrees) errors.

		Disloc [9] NetVLAD [6] NetVLAD [6]	InLoc
		+SparsePE +SparsePE	+DensePE	(Ours)
	90 bldgs. 0.42, 4.58 • 0.44, 4.70 •	0.23, 2.53 • 0.17, 2.15 •
		PoseNet ActiveSearch	DSAC	NetVLAD [6] NetVLAD [6]
	Scene	[34]	[55]	[11, 13] +SparsePE [51] +DensePE
	Chess 13, 4.48 •	4, 1.96 •	2, 1.2 •	4, 1.83	3, 1.05 •
	Fire	27, 11.3 •	3, 1.53 •	4, 1.5 •	4, 1.55	3, 1.07 •
	Heads 17, 13.0 •	2, 1.45 •	3, 2.7 •	2, 1.65	2, 1.16 •
	Office 19, 5.55 •	9, 3.61 •	4, 1.6 •	5, 1.49	3, 1.05 •
	Pumpkin 26, 4.75 •	8, 3.10 •	5, 2.0 •	7, 1.87	5, 1.55 •
	Red kit. 23, 5.35 •	7, 3.37 •	5, 2.0 •	5, 1.61	4, 1.31 •
	Stairs 35, 12.4 •	3, 2.22 •	117, 33.1 •	12, 3.41	9, 2.47 •

• .

Table 4 .

 4 Evaluation on the 7 Scenes dataset

  10.51, 178.81 • 0.06, 1.00 • 16.59, 133.63 • 0.19, 2.46 • 2.27, 17.39 • 0.46, 2.89

• NetVLAD [6]+ InLoc: NetVLAD [6]+ NetVLAD [6]+ InLoc: NetVLAD [6]+ NetVLAD [6]+ InLoc: NetVLAD [6]+ SparsePE DensePE+DensePV SparsePE DensePE+DensePV SparsePE DensePE+DensePV

For example, in the database of[START_REF] Sun | A Dataset for Benchmarking Image-based Localization[END_REF], the scans are distributed on one single floor, and the area per each database image is less than 45 m 2 .

The impact of negative evidence in feature aggregation is demonstrated in[START_REF] Jégou | Negative evidences and cooccurrences in image retrieval: the benefit of PCA and whitening[END_REF].

Due to the sparse sampling of viewpoints in our indoor dataset, we cannot establish feature tracks between database images. This prevents us from applying algorithms relying on co-visibility[START_REF] Choudhary | Visibility probability structure from sfm datasets and applications[END_REF][START_REF] Li | Worldwide pose estimation using 3d point clouds[END_REF][START_REF] Sattler | Hyperpoints and fine vocabularies for largescale location recognition[END_REF][START_REF] Sattler | Efficient & effective prioritized matching for large-scale image-based localization[END_REF][START_REF] Zeisl | Camera Pose Voting for Large-Scale Image-Based Localization[END_REF].

Acknowledgements. This work was partially supported by JSPS KAKENHI Grant Numbers 15H05313, 17H00744, 17J05908, EU-H2020 project LADIO No. 731970, ERC grant LEAP No. 336845, CIFAR Learning in Machines & Brains program and the European Regional Development Fund under the project IMPACT (reg. no. CZ.02.1.01/0.0/0.0/15 003/0000468). The authors would like to express the deepest appreciation to Yasutaka Furukawa for his arrangement to capture query photographs at Washington University in St. Louis.

Appendix

This appendix first provides additional examples of query images and their reference poses in our InLoc dataset (section A). We also present additional qualitative results, illustrating situations in which the proposed InLoc method succeeds while the investigated baseline methods fail (section B).

A. Additional examples of query images and

reference poses in the InLoc dataset