We introduce a novel object proposals method specific to building facades. We define new image cues that measure typical facade characteristics such as semantic, symmetry and repetitions. They are combined to generate a few facade candidates in urban environments fast. We show that our method outperforms state-of-the-art object proposals techniques for this task on the 1000 images of the Zurich Building Database. We demonstrate the interest of this procedure for augmented reality through facade recognition and camera pose initialization. In a very time-efficient pipeline we classify the candidates and match them to a facade references database using CNN-based descriptors. We prove that this approach is more robust to severe changes of viewpoint and occlusions than standard object recognition methods.

Facade Proposals for Urban Augmented Reality

Antoine Fond1 , Marie-Odile Berger2 , and Gilles Simon Figure 1: Facade proposals for Urban AR. A facade of the Nantes Event Center building (a) is automatically detected and recognized in two views of the building (b, red polygons). From these results, any planar virtual object added to the facade (here the ISMAR logo) can be warped according to the transformation of the facade. (c) When some geometric information about the facade is available, any 3D virtual object expressed in the same reference frame as the facade (here the ISMAR building) can be added to the view.

INTRODUCTION

The purpose of Augmented Reality (AR) is to enhance the user's view of the real world with context specific information in such a way they appear to naturally belong to that world. In urban environments AR, buildings play a twofold role : they are the main object of interest for many AR applications [START_REF] Liu | Detection of Facade Regions in Street View Images from Split-and-Merge of Perspective Patches[END_REF] and they are semantically meaningful city-scale landmark to rely on for localization in GPSdenied areas [START_REF] Arth | Instant Outdoor Localization and SLAM Initialization from 2.5D Maps[END_REF][START_REF] Chen | City-scale Landmark Identification on Mobile Devices[END_REF][START_REF] Košecká | Extraction, Matching, and Pose Recovery Based on Dominant Rectangular Structures[END_REF][START_REF] Li | A Two-view based Multilayer Feature Graph for Robot Navigation[END_REF].

To overlay annotations on building facades such as anecdotes for tourism, advertisements for shopping, or addresses for city navigation aid in GPS-denied areas it is necessary to detect and recognize the building of interest in the camera image, followed by an estimation of the camera pose to correctly project virtual objects into the image.

Building detection from monocular images is a challenging task due to perspective deformations, repetitive structures and partial occlusions. Two categories of methods have been proposed in the past for the detection of building facades. Geometry-based methods attempt to identify rectangle facades in images rectified thanks to orthogonal vanishing points [START_REF] Fond | Prior-based Facade Rectification for AR in Urban Environment[END_REF][START_REF] Košecká | Extraction, Matching, and Pose Recovery Based on Dominant Rectangular Structures[END_REF]. Various geometric and photometric criteria are then used to characterize facades. However, they are generally too strict to take into account the variability of the facades encountered in urban areas.

On the other hand, with the advent of learning based techniques for classification, several methods have been designed with the aim to classify pixels or superpixels into categories. Among them, [START_REF] Farabet | Learning Hierarchical Features for Scene Labeling[END_REF][START_REF] Hoiem | Automatic Photo Pop-up[END_REF] show examples of classification, "building" being one of the categories. Some are even tailored for classifying building subparts [START_REF] Gadde | Efficient 2D and 3D Facade Segmentation using Auto-Context[END_REF], [START_REF] Martinovic | A Three-Layered Approach to Facade Parsing[END_REF]. Though promising, these methods do not allow to distinguish between adjacent facades. Recently a novel deep convolutional encoder-decoder architecture has been proposed in [START_REF] Badrinarayanan | SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling[END_REF] for semantic pixel-wise labeling. By learning decoders to map the deepest layer features to full image dimensions, smoother predictions are obtained compared to [START_REF] Farabet | Learning Hierarchical Features for Scene Labeling[END_REF]. The inference is also faster than [START_REF] Gadde | Efficient 2D and 3D Facade Segmentation using Auto-Context[END_REF], [START_REF] Martinovic | A Three-Layered Approach to Facade Parsing[END_REF].

Object detection is traditionally formulated as a classification problem in the sliding windows paradigm in which classification is performed at each image location and scale in the image. Object detection has made great strides in recent years with the emergence of object proposals techniques. Their goal is to generate at high speed a reduced set of object bounding box proposals. A more complex classifier is then used in a second step for scoring.

The idea of defining an "objectness measure" for the preselection step, designed to produce a small number of regions such that top-ranked regions are likely to contain some categories, is developed in [START_REF] Alexe | Measuring the Objectness of Image Windows[END_REF][START_REF] Endres | Category Independent Object Proposals[END_REF][START_REF] Zitnick | Edge Boxes: Locating Object Proposals from Edges[END_REF]. Though some methods integrate that objects have well defined closed boundaries [START_REF] Zitnick | Edge Boxes: Locating Object Proposals from Edges[END_REF], these methods are too general to be applied to the pre-selection of facades.

We thus propose in this paper a "facadeness" measure of image windows that can be evaluated rapidly and integrates geometric, photometric and semantic constraints. With respect to existing object proposal techniques, we especially introduce symmetry and repetitions constraints which are specific to building objects. We also propose to use semantic constraints based on the labelling of a SegNet-like network.

The aim of this work is to detect and identify facades for overlay purpose. Our framework for facade recognition does not require a city-scale 3D model but only a collection of fronto-parallel facade images and their associated virtual objects. However, as our method is partially geometry-based, we can also estimate the camera pose relatively to the detected facade as a by-product. Thus if this facade is part of a larger georeferenced 3D model our method can propose an approximate camera pose for geo-localization in the sense of [START_REF] Arth | Instant Outdoor Localization and SLAM Initialization from 2.5D Maps[END_REF][START_REF] Chu | HouseCraft: Building Houses from Rental Ads and Street Views[END_REF].

The paper is organized as follows: related work is described in section 2. Our method for facade proposals is explained in section 3. Application of our method for facade recognition is described in section 4. Results about facade proposals and facade recognition are presented in section 5.

STATE OF THE ART

Urban Augmented Reality

Many approaches have been proposed for augmented-reality in outdoor urban environnements. These methods can either be geometry-based or image-based. The former ones assume that a 3D model of the scene is available and seek to estimate the camera pose in order to project virtual objects into the image. In [START_REF] Reitmayr | Going out: Robust Model-based Tracking for Outdoor Augmented Reality[END_REF] the camera pose is tracked using measurements from prominent edges in the image and inertial-sensors data fused together with a Kalman filter. This method needs a fully textured 3D model. To relax this constraint, in [START_REF] Karlekar | Positioning, tracking and mapping for outdoor augmentation[END_REF] the authors propose to rely on building silhouettes rather than internal edges. Initial camera pose is refined using silhouettes matching through shape context descriptors and then tracked in a very similar way to [START_REF] Reitmayr | Going out: Robust Model-based Tracking for Outdoor Augmented Reality[END_REF]. The 3D model has to be sufficiently detailed so that building shapes can be matched. On the contrary in [START_REF] Arth | Instant Outdoor Localization and SLAM Initialization from 2.5D Maps[END_REF] the 3D model is very coarse using only building footprint and height. After finding vanishing points of the scene the method relies on vertical edges to generate translational hypotheses. Semantic segmentation is used to design a facade likelihood maximized relatively to the camera pose. In a similar way in [START_REF] Chu | HouseCraft: Building Houses from Rental Ads and Street Views[END_REF] edges, semantic segmentation and windows detection are combined in a cost function that is minimized to find the pose. The main drawbacks of all these geometry-based approaches is that they strongly rely on GPS and inertial sensors to initialize the camera pose and they require an accurate 3D model. On the other hand, image-based methods do not require any scene geometry. Buildings are detected in the image and then matched to an image references database.

Facade detection

Several methods have been proposed in the past to detect rectangular structures in Manhattan worlds. In [START_REF] Košecká | Extraction, Matching, and Pose Recovery Based on Dominant Rectangular Structures[END_REF], line segments are automatically detected and intersected to generate hypotheses of rectangles in agreement with the vanishing points. For each hypothesis, the input image is orthorectified and a histogram of gradient (HOG) is computed inside the warped rectangle. Hypotheses whose HOG contains more than two dominant horizontal and vertical directions are discarded. This method is computationally expensive generating many superfluous hypotheses. To keep the problem tractable and efficient, Micusik et al. formulate the detection of the rectangles on a restricted neighborhood structure given by Delaunay triangulation [START_REF] Micusík | Detection and Matching of Rectilinear Structures[END_REF]. The problem is then expressed as a search for the maximum aposteriori probability solution of a Markov random field. In [START_REF] Fond | Prior-based Facade Rectification for AR in Urban Environment[END_REF], right-angle corners are detected in the orthorectified image using a Support Vector Machine. A Delaunay triangulation is performed from the right-angle corners and a min-cut-like algorithm is used to generate windows in which a high density of right corners is observed.

All these methods allow to detect rectangular structures appearing on facades, like windows or rows of windows, but not, in general, entire facades. In [START_REF] Liu | Local Regularity-Driven City-Scale Facade Detection from Aerial Images[END_REF], the Gini Index is used to form an edge-based regularity metric relating regularity and distribution sparsity. The facade region detection is treated as a regional regularity/sparsity maximization problem, which is solved using greedy adaptive expansion over a down-sampled grid. Integer Quadratic Programming is then used to select a subset of facades that have maximum regularity score and facade coverage, with minimum overlap. However, the method still suffers from the use of a grid, and the regularity assumption makes it more suitable to large building facades with many regularly spaced windows than to the various kind of facades we consider in this work.

Object proposals

Landmark or building identification is of high interest in place recognition or pose computation with the goal to extract only roughly the buildings. As recognition from full images suffers from changes in viewpoints changes, the focus of research in place recognition has recently moved towards the identification of prominent landmarks in the scene through the use of object proposal methods thereby avoiding exhaustive sliding window search across images. [START_REF] Alexe | Measuring the Objectness of Image Windows[END_REF] was the first to define the concept of "objectness", that is expressed as a score based on the combination of multiple cues (color contrast, edge density,...). Since then, several detection proposals have been proposed. For example, segmentation achieved at different scales is used as a selective search strategy in [START_REF] Uijlings | Selective Search for Object Recognition[END_REF] whereas EdgeBox [START_REF] Zitnick | Edge Boxes: Locating Object Proposals from Edges[END_REF] uses object boundaries estimates for scoring. Edge-Box appears as the best compromise in speed versus quality in a comparative study conducted in [START_REF] Hosang | How Good are Detection Proposals, really?[END_REF] on then object proposal methods.

These techniques have been recently applied to place recognition [START_REF] Suenderhauf | Place Recognition with ConvNet Landmarks: Viewpoint-Robust, Condition-Robust, Training-Free[END_REF]. Indeed comparing images on the basis of the whole image is sensitive to viewpoint changes. Comparing regions between images as proposed by EdgeBox [START_REF] Zitnick | Edge Boxes: Locating Object Proposals from Edges[END_REF] has proven to be more robust to viewpoints changes [START_REF] Suenderhauf | Place Recognition with ConvNet Landmarks: Viewpoint-Robust, Condition-Robust, Training-Free[END_REF]. However, the cost of the method can be prohibitive if the fraction of buildings recovered by the object detector is small with respect to the number effective buildings present in the image. This leads to use a large number of proposal regions to be sure that a sufficient number of buildings can be matched between images.

We thus propose in this paper an efficient method for facade proposals which outperforms existing objectness methods. On the tested datasets, we show that only 100 candidates are required to obtain 84% of recall. This thus opens the way towards efficient methods for relocalization, place recognition and pose initialization.

FACADE PROPOSALS

Our algorithm for facade proposals consists in a two-stage procedure. A first set of facade candidates relying on contours is initialized. Ad hoc facade features (facadeness cues) are then evaluated on that set, and the best facade candidates are selected by combining the obtained values in a machine-learning framework. A database of 1500 images from Google Street View and ImageNet was used for learning purpose whereas testing was done on the 1000 images of Zurich Building Database (ZuBuD)0 . Each image is orthorectified and the bounding boxes of the facades are provided manually. These bounding boxes are referred as ground truth (GT) in the following.

Geometry of the scene and rectification

Manhattan hypotheses are well suited for modeling urban environments. All the buildings of the scene are considered to be 3D boxes and shall be parallel one to another. Each of the box faces is a facade except for top and down faces. Thus the geometric shape of a facade is a rectangle and its texture is defined by different visual characteristics that will be described below.

The method of [START_REF] Simon | A Simple and Effective Method to Detect Orthogonal Vanishing Points in Uncalibrated Images of Man-Made Environments[END_REF] is used in this paper to find the Manhattan vanishing points. The intrinsic parameters of the camera as well as the homographies that warp the image of the scene to orthorectified images are automatically computed. Doing so, all the vertical facades of the scene appear in either of these orthorectified images as in a frontal view.

Joint semantic parsing and contours detection

Semantic parsing does not solve the problem of facade detection by itself as it cannot distinguish between two nearby facades. How-ever we exploit this pixel-wise labeling in ours facadeness cues. We trained a modified version of SegNet to infer 7 semantic classes (background, facade, window, balcony, door, sky and road) as well as contours (see Fig. 2). Jointly solving these two problems enables semantic parsing to be more aware of edges and contours to be more located on meaningful edges (Fig. 3). The training/testing database is a label-consistent merge of CMPfacadeDB 1 , eTrims 2 , ECP 3 , INRIA 4 and LabelMeFacade [START_REF] Fröhlich | A Fast Approach for Pixelwise Labeling of Facade Images[END_REF]. It contains a variety of classic and modern European style buildings from different cities (Paris, Prague, Berlin,...). Ground truth contours of that database are contours between different semantic areas. The network architecture is the same as the regular SegNet but the final deconvolutional layer has 9 outputs (7 for semantics and 2 for contours) that are sliced into 2 different layers. The final loss function used for training is a weighted sum of the logistic loss functions of these layers.

Semantic SegNet

Contour

SoftMax Layer 1x360x480x9 Input Image 1x360x480x3

Output GT Output GT

Rectangular candidate sampling

The main hypothesis we made on facades is that they are rectangular-shaped. As we work with orthorectified images we are explicitly looking for rectangles. We choose to rely on the contours of the image to generate a first set of candidates. Indeed, the border of a facade should create high gradient values on the image. Edge map E is one of the two outputs of our modified version of SegNet. These edges are then accumulated in both a histogram of verticalprojected edges H x and a histogram of horizontal-projected edges facade candidates are retained. For instance, for the 1500 images in our learning database, the average number of facade candidates per image is 16288 at this very first step.

Facadeness cues

Facades share several common visual characteristics. They are usually composed of rectangular features such as floors, windows, balconies, doors. These features repeat themselves along the facade in both vertical and horizontal directions. Facades are also roughly symmetrical. Eventually facades are homogeneous in color at least compared to their background. For each of these facade candidates we evaluate 6 different had hoc features (cues). We here reuse the color contrast and shape cues defined in [START_REF] Alexe | Measuring the Objectness of Image Windows[END_REF]. With respect to [START_REF] Alexe | Measuring the Objectness of Image Windows[END_REF] a stronger edge cue is defined which favors vertical and horizontal segments. Three new criteria are introduced aiming at characterizing semantic contrast, symmetry and repetitive patterns on facades. Subsequently the combination of all these cues enables us to discard the candidates that do not match our facade hypotheses and keep only the best ones. For each cue presented below, Fig. 5 shows the best rectangle obtained among all candidates, in an example image of our database (right column) and the probabilities of the cue values to be obtained on a facade (in green) or on a non facade (in red).

Shape cue

Facades are rectangular-shaped, but all rectangles are not as likely to be observed. Indeed, architectural rules allow just a few values of the facade aspect-ratio. Extremely thin facades are almost impossible for example. We have learned the probability distribution of two rectangular parameters (height and width) on our 1500 learning images (Fig. 5, top-right image) in the same way as in [START_REF] Alexe | Measuring the Objectness of Image Windows[END_REF]. We use a 24 × 24 discretized version H of that distribution for efficiency s shape (r) = H(h, w).

Color cue

Color itself is a poorly informative feature to describe facades as they can have many different colors. However the color homogeneity of a facade compared to its local context is a much more interesting feature as it is described in [START_REF] Alexe | Measuring the Objectness of Image Windows[END_REF]. The difference of color distribution between the inside of the rectangle and the surrounding region can distinguish facades as in Fig. 5, last row:

s color (r) = 1 -exp -d χ 2 H b(r,β ) c , H r c /σ c (1)
where H r

Oriented contour cue

As facades are rectangular-shaped, we can expect high gradient values along their border. More precisely we can expect vertical (re-spectively horizontal) contours along the vertical (respectively horizontal) edges of their bounding box :

s cont (r) = 1 2α(l + h) ∑ b t (r,α)∪b b (r,α) E x + ∑ b l (r,α)∪b r (r,α) E y (2)
where α is the thickness of the band b x (r, α) positioned on the x ∈ {top, bottom, right, le f t} of the rectangle r. E x and E y are the binary images of the horizontal and vertical SegNet contour. (h, w) are the height and width of r, respectively.

Structure cue

Windows and balconies repeat themselves along both vertical and horizontal directions on facades. The "window" or "balcony" labels projected on the horizontal axis are binned in an histogram H r x so are the same labels projected on the vertical axis in H r y . The autocorrelation of both these two signals is sparse if there are strong repetitions. We thus define the structure cue :

s struc (r) = W (r) ∑ peaks R(H r x ) ∑ R(H r x ) + ∑ peaks R(H r y ) ∑ R(H r y ) (3)
where H r x and H r y are the 32 bins-normalized-histograms defined above for the rectangle r. R( f ) = F -1 |F ( f )| 2 is the autocorrelation of f . F and F -1 are respectively the Fourier transform and inverse Fourier transform. Peaks are local maxima of the signal. W (r) is the sum of the labels "facade", "window", "balcony", "door" inside r normalized by the area of r.

Symmetry cue

Facades have a non-perfect axial symmetry. What we want is a metric that evolves continuously with the symmetrical aspect of the facade. For example the cross-correlation between the left and the right half of the image would be very high for even a small asymmetry. We propose to subdivise the rectangle into 16 patches. For each of these patches we compute the HOG descriptor with 8 bins on SegNet contour. Then we evaluate the distance between each of the 8 patches on the left with their symmetrical patch on the right :

s sym (r) = exp - 4 ∑ i=1 2 ∑ j=1 d χ 2 H sym e (s(i, j)), H e (i, j) 8σ s (4)
where H e (i, j) is the HOG descriptor with 8 bins of the patch (i, j). H sym e (i, j) is the flipped version of vector H e (i, j). s is the axial symmetry of vertical axis and d χ 2 the χ 2 distance.

Semantic contrast cue

Facades are composed of semantic features such as windows, balcony, door, walls in specific proportions. Their proportions in one facade differ from the proportions of the surrounding region that can include other semantic features like portions of sky or roads.

s sem (r) = 1 -exp -d χ 2 H b(r,γ) s , H r s /σ sc ( 5
)
where H r s and H b(r,γ) s

are respectively the histogram of the semantic labels inside r and inside the band of thickness γ surrounding r.

Computation of all the cues is in constant time for one rectangle thanks to the use of integral images. This trick is detailed in [START_REF] Viola | Robust Real-Time Face Detection[END_REF] for the computation of sums in regions as well as local histograms. The quantification of the histograms and the number of patches have been chosen to keep that constant time reasonable (below 10 3 operations). The parameters α, β , γ have been learned on our training set (see section 3.5) so as to maximize the separability between positive and negative probability distributions of the cues values (Fig. 5, left column). The optimal values for these parameters are 5%, 30% and 20% of the dimensions of the rectangles, respectively. σ c , σ s , σ sc are the standard deviation of the distances used respectively in the color, symmetry and semantic contrast cues.

Cues combination

Intersections between facade and non-facade probability distributions of cues values (Fig. 5, left column) mean that one cue alone cannot separate between facades and non-facades. To combine all these features into a single metric we use a multi-layer perceptron. It is composed of two hidden layers of 8 neurons. This neural network has been trained on positive and negative examples, taken from the rectangle sets generated by the sampling procedure presented in section 3.3, applied to all images of the learning database. To decide if a rectangle is a positive or a negative example, we used the commonly used metric "Intersection over Union" (IoU score s IoU ) [START_REF] Zitnick | Edge Boxes: Locating Object Proposals from Edges[END_REF]. An IoU threshold of 0.5 is often used in the literature to decide whether or not two image regions coincide. Moreover, an illustration in [START_REF] Zitnick | Edge Boxes: Locating Object Proposals from Edges[END_REF] shows that an IoU score of 0.5 already indicates a relatively high overlap. For these reasons, we took as positive examples rectangles that overlap the GT with s IoU ≥ 0.5, whereas negative examples are candidates with s IoU < 0.5. This set of examples will be referred as our training set.

The final output of the perceptron can be seen as a probability score of being a facade. All candidates are sorted using this metric and a greedy algorithm keeps the best candidates that do not overlap strongly a higher-ranked rectangle (according to s IoU ≥ 0.5) as in [START_REF] Alexe | Measuring the Objectness of Image Windows[END_REF]. Figure 6 shows the recall rate obtained on ZuBuD database as a function of the number of proposals. We compare the combination of our problem specific cues (contour, structure, symmetry and semantic contrast) to the combination of already-defined cues (contour, shape, color contrast) and the combination of all cues. A GT rectangle is counted retrieved when at least one of the selected candidates overlaps it with a s IoU ≥ 0.5. In practice, we use the 100 first proposals which corresponds to 85 % of recall. We can use much fewer proposals than state of the art methods for the same performance and about the same computational time (0.42s). (in red) all the cues, (in green) only the problem-specific cues (contour, structure, symmetry, semantic contrast), (in blue) only the objectness-inspired cues (contour, shape, color contrast). We also compare our method to other object proposals methods.

APPLICATION : FACADE RECOGNITION

Our facade proposals method could be used to improve [START_REF] Arth | Instant Outdoor Localization and SLAM Initialization from 2.5D Maps[END_REF]. Indeed it could help to generate translational hypotheses and disambiguate situations with many adjacent facades lying on a single plane in the alignment step. However we choose to highlight another application of our method : facade recognition.

The problem we refer to as facade recognition and that we are trying to solve in this part is the following: taking a new single picture of a known urban place, we want to recognize the most prominent facades of this image. By recognition we mean not only detecting the facades but also match them to the correct reference in the facade database that represents our place. This problem is different from place recognition as we additionally intend to roughly locate the facades existing in the database in the considered image. Solving this problem opens the way towards AR applications in urban context such as practical or cultural information overlay on the building or may help to guide the users towards a desired building.

In the context of urban augmented reality the lack of GPS accuracy does not allow a seamless overlay information about the buildings in the image. However the GPS may help reduce the numbers of buildings to be matched to a dozen.

Our facade recognition method is threefold (Fig. 7):

1. Generate a few number of facade candidates through our facade proposal method.

2. Classify these candidates into "facade" and "non-facade" through a neural network using SPP descriptors [START_REF] He | Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition[END_REF] with surrounding context as inputs. Resorting to spatial pyramid pooling allows us to compute the feature maps from the entire image only once, avoiding repeatedly computing the convolutional features for each box proposal.

3. Match the remaining facades with the facade database of the current place using a semantic metric learned through a siamese neural network taking SPP descriptors as inputs.

Facade classification

The main difference between facade classification and general object classification is that facades cannot be described only from the inside of their bounding box. Indeed, a part of a facade may be visually a facade too (the entire facade cropped so that one floor is missing for example). The only way to avoid this multiple parts problem is to consider the surrounding visual context. A true facade shall look like a facade inside but its context may differ. We propose to build a descriptor by concatenating the SPP descriptor inside the rectangle with SPP descriptors of the surrounding band.

The SPP descriptor of the inside is computed from the 5th convolution (Conv5) layer of our modified SegNet. We use a 3-levels spatial pyramid

( 1 × 1, 2 × 2, 4 ×

Facade matching

At this step, the descriptor associated with each candidate is their inside SPP descriptor already computed during facade classification. However distinguishing between two different facades and still be robust to few appearance changes of the same facade can be challenging. This problem is similar to fine-grained classification and is generally solved by learning a similarity metric using a siamese network [START_REF] Chopra | Learning a Similarity Metric Discriminatively, with Application to Face Verification[END_REF]. The idea is to map images in a low dimensional space so that the distance between points of the same class is small and is large for points from different classes.

Here, the inside SPP descriptor is the input of such a neural network with 2 layers of size 2048 trained in the siamese way on a third of the ZuBuD database. The positive pairs are generated from positive samples (s IoU ≥ 0.5) of different views of the same facade whereas negative pairs are generated from positive samples of different facades. The embedded space induced by this siamese network is then much smaller and tuned to distinguish facades that can be visually similar. We then compute the euclidian distance between the siamese network outputs of both the candidates and the facade references from the database. For each candidate we choose its closest neighbors in the facade reference database. To assure that the match is correct we apply crosschecking i.e. we intend the closest neighbors to be both-ways.

EXPERIMENTAL RESULTS

Facade proposals

For testing we use the Zurich Buildings Database (ZuBuD). This database is composed of 1000 pedestrian street views of Zurich. It is divided into 200 scenes each one focusing on one building. The changes of viewpoint are severe and it is not rare for the facades to be occluded by trees, street lamps or electric lines. There is also a good diversity of buildings with different architectures from classic European style to more modern ones. For all of these reasons we evaluate our facade proposals method on that dataset. We compare our method to other object proposals methods through 3 different points : recall, precision and time (Tab. 1).

Recall is one of the most important measurement for an object proposals method. It is computed as the rate of object recovered among the n first proposals. If most general object proposals methods perform well on ZuBuD over n = 500 proposals with more than 90 % recall our method shows much better results below. Indeed with n ≤ 100 proposals our method has a constant gain of more than 10 % compared to state of the art object proposals methods (Fig. 6).

This improvement can be explained by the use of cues that are more discriminative for facades than general objects. More precisely some assumptions that are made by general object proposals method are violated in urban environments. Selective Search is based on super-pixel merging. As the local context is not considered there is no way to distinguish between subparts and plain facades which leads to a drop in recall for few proposals. The main assumption of EdgeBox is that the edges are wholly enclosed inside proposals. In the case of adjacent facades, there are usually mutual edges between facades that overlap the proposal. In Objectness the way to sample the proposals in the first place is based on changes in the frequency distribution of natural images. But in urban environments the frequency distribution is already biased by verticals edges and strong repetitions. Eventually none of these methods take into account the rectification of the image whereas the symmetry and the structure cues use this information in our method. These poorly suited assumptions for urban environments affect the ranking of proposals. More specifically bounding boxes merging adjacent facades are usually ranked up top by general object proposals methods causing the true facade to be ranked further down. Thus more proposals are needed for a further task (e.g. detection or recognition) while n = 100 proposals are enough in our method.

The precision is computed here as the average value s IoU of all the proposals that overlap the GT with s IoU ≥ 0.5 for n = 100. This measurement is not critical for object proposals but for facade recognition it is important to accurately locate the facade. Our better results in precision compared to others can be explained by the use of cues based on contours along the facade boundaries and surrounding context in both color and semantic.

The second measurement that actually matters for object proposals is time. All our cues are computed in constant time thanks to integral images and spatial pooling. That allows us to get a computational time which is better or in the same order of magnitude than the others. This time is compatible with augmented reality applications. All the codes are written in Matlab with critical parts executed in C. The computational times shown in Tab. 1 are the average times for n = 100 proposals on I7-3520M CPU with an Nvidia TITAN X GPU for the forward pass in SegNet.

Facade Recognition

The test set of our facade recognition application is the 2/3 of ZuBuD that have not been used for the metric learning. There are 937 GT facades tagged on this part of ZuBuD. These facades are grouped into 171 classes, each one representing a unique facade. Each class gathers images of the same facade but observed through different viewpoints. Consequently images from the same class have different rectification artifacts, different resolutions (Fig. 8). Due to occlusions they can be cropped versions of each other and some parts can be hidden or not. In addition to all this diversity intraclass different classes can be visually similar from one another. To represent the whole class we choose the facade reference the closest to the frontal view with the fewest occlusions.

In such challenging conditions classic matching approaches fail to match a facade to its correct reference (Tab. 2). We have evaluated the matching performance of our facade matching method compared to others in our facade recognition pipeline. For every approach we first execute our facade proposals method with n = 100 candidates. For each ground truth facade of the ZuBuD test set we select the candidate that overlaps it the most in the sense of s IoU . Thus these facade candidates are the best we can expect using our facade proposals method no matter how well our detection step performs. For each of these facades we compute their descriptor and we search their closest neighbor in the 171 reference database for Euclidean distance. A match is correct if the class of its closest neighbors is the same as its ground truth class. Tab. 2 shows the results of different descriptors used for place recognition. Bag of Words (BoW) descriptor is the histogram of visual words (SIFT) inside the facade candidate. VGG is the 4096 output vector from VGG CNN [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] applied to the subimage of the facade candidate resized to fit the input of the network. SPP is the SPP descriptor from Conv5 of our modified SegNet and SPP (siamese) the output of the siamese network for that same vector. the similarity of their descriptors in repetitive structures can explain the poor results of BoW. Moreover without any spatial information BoW can only discriminate between facades through their proportion of visual words that is clearly not sufficient. The difference between VGG and SPP is mostly imputable to the fine tuning of our modified SegNet for buildings segmentation. Indeed the architecture of both networks is pretty similar and SPP is essentially a speed-up approximation of a true CNN descriptor. The Conv5 layer of our network yet contains more meaningful information about facades rather than an ImageNet trained VGG. Eventually metric learning through siamese networks helps for fined-grained classification.

We then evaluate the whole pipeline including the detection step. Example results are shown in Fig. 10. At the end of the facade proposals we have 100 facade candidates. As the matching is based on the closest neighbors we cannot apply it directly to each of the candidates. Indeed each candidate will always find a closest reference in the database even if no facade of that class is present in the image. We need to select the facades that are actually visible in the image and only them. The facade classification based on the inside and surrounding regions remove most of the candidates and the cross-checking assures that the match is correct. Thus they are few false detections 16.14 % with still a high recall of 71.13 % (SPP siamese) for the whole test set. These results outperforms BoW and are comparable to VGG even though the facade selection was biased to the best possible solution in their case. False detections do not necessarily mean the detection has completely failed. It can usually be explained by one of the following situations: the detected facade is too small but included in the GT facades (image 1 in Fig. 11), the detected facade groups several GT facades into one (image 3 in Fig. 11), the detected facade has been missed in GT tagging (image 1 in Fig. 11). However we have noticed one regular failure case that happened when small detected facade with poor photometric information match with a quasi-homogenous reference facade (image 1 in Fig. 11). The matching also can fail when there are two very similar facades in the database. For example, this situation can occur for facades coming from the same building (image 4 in Fig. 11).

We also evaluate the whole pipeline on a smaller set that comes from the Street part of Cambridge Relocalisation Dataset5 . This set is composed of 80 images divided into 20 classes. That situation is more suitable for a real AR application using GPS localisation to prune the database. Unlike ZuBud there are no occlusions but the changes of viewpoint are more extreme. The statistics are pretty similar to ZuBuD with 73.38 % of recall and 19.34 % of false detections. Thus our method proves to be robust to severe changes of viewpoints as well as partial observations and occlusions (Fig. 12).

Another benefit of our pipeline for facade recognition is speed. As we use SPP-based descriptors we only need one single forward pass in our modified SegNet for the whole method. We reuse the outputs of the latter in different parts of the algorithm (rectangles sampling, cues computation,...) as well as its Conv5 feature maps for descriptors. Our method can be seen as an initialization for further more accurate pose estimation [START_REF] Karlekar | Positioning, tracking and mapping for outdoor augmentation[END_REF][START_REF] Reitmayr | Going out: Robust Model-based Tracking for Outdoor Augmented Reality[END_REF]. As an initialization process it does not need to be executed at each frame but only at the beginning and when the tracking process fails. The time efficiency of the pipeline (0.45s) makes it compatible with AR applications as a new facade recognition step can be computed every 11 frames. We could imagine that detected facades could be tracked the rest of the time in such application. Thus the initialization could be processed server-side whereas the tracking could be done on the mobile device. Although the lack the GPU power is still a limitation for our method to perform in real-time on regular mobile-device, we believe that future generations of mobile devices will overcome this limitation (the SegNet inferences on an Nvidia TX1 embedded GPU requires 0.7s, contrary to 0.1s in our desktop setup). As most mobile-device manufacturers tend to develop embedded GPU in their devices hardware, we expect that in the future real-time CNN inference will become possible with commodity devices. This trend is also supported by major software corporations, e.g., Facebook (Caffe2Go) and Google (Tensorflow Lite).

AR applications

Once facades in the image have been detected and identified it is already possible to overlay planar virtual objects onto them. It is sufficient to project the boundaries of the detected facades back into the original image through the inverse homography used to rectify the image. For example in Fig. 10 each colored area represents a facade that has been successfully detected and matched to its reference. These areas could be easily replaced by any meaningful information specific to their building (Fig. 1,b). However, if in addition to our facade references database each of these references are augmented with geometric information (such as a georeferenced frame and their real world dimensions) a complete 6DOF camera pose can be estimated (Fig. 1,c) for each recognized facade. Let us assume such geometrically-enriched model is available for a facade that has been detected and identified by our method. We already have the rotation R that transforms the camera frame into the local frame of the facade from the rectification step so only the translation T needs to be computed. That can be done by associating facade corners in the image with corresponding real world facade corners from the geometric information [START_REF] Simon | Markerless Tracking using Planar Structures in the Scene[END_REF]. The real world dimensions are important to have a shared scale between detected facades and associated virtual objects (such as a 3D model of buildings). The georeferenced frame enables to transform the camera pose relative to the facade to an absolute pose for geolocalization purpose. As an example, we used a public 3D model 6 of the Nantes Event Center building where ISMAR 2017 will take place and we tried to recognize one of its facade on GoogleStreetView images showing the building from different viewpoints. The facade reference has been added to the ZuBuD references database used in the previous test. In all test images the facade has been successfully detected (always in the top-10 proposals) and recognized as part of the Nantes Event Center building. We choose to describe both applications depending on what information we suppose about the database. If the database is only made of facade images with associated planar virtual information (the ISMAR logo for Nantes Event Center building), we overlay it onto the detected facade (Fig. 9,1st row and Fig. 1,b). If we assume geometric information about the facades is available in the references database, we project the wireframe of the 3D model of the building associated to the recognized facade back into the image (Fig. 9,2nd row and Fig. 1,c).

CONCLUSION

We presented a fast facade proposals method that can be applied in a very efficient way to facades recognition and camera pose initial- ization in urban environments. We demonstrated the relevance of combining ad-hoc features and deep learning framework together for object-specific localization and recognition. Though the invariance of CNN descriptors to small translations makes our method not currently suited for accurate pose estimation, it proposes a good initialization that could be refined with a further gradient-based image registration method based on deep learned-metric.

1

 1

  (a) Image from the database (b) Facade detection and recognition + warping of a 2D virtual object (c) Pose computation and projection of a 3D virtual object

Figure 2 :

 2 Figure 2: Architecture of our modified version of SegNet with two outputs : a semantic labeling map and a contour map.

Figure 3 :

 3 Figure 3: Result of the joint training on semantic inference map. First row shows an image of facade (left) and its ground truth semantic labeling (right). Second row shows the inference map of a standard SegNet (left) and our modified SegNet (right). The shape of the windows are much more rectangular in our version due to the joint training of semantic and contours.

  H y . The product H x H T y can be seen as a corners likelihood map (see Fig.4). The n local maxima of that map are used to generate n(n-1) 2 rectangles. Actually, as both (top-left,bottom-right) and (top-right,bottom-left) pair of corners define the same rectangle, only a set of n(n-1) 4

Figure 4 :

 4 Figure 4: Example image from our database (left), with the contour map (middle) and the corners likelihood (right).

  c and H b(r,β ) c are respectively the color histogram of the inside of r and the color histogram of the band of thickness β surrounding r. We use LAB color space quantized into 256 = 4 × 8 × 8 bins. of facade (green) or non-facade (red) of facade (green) or non-facade (red) of facade (green) or non-facade (red) of facade (green) or non-facade (red) of facade (green) or non-facade (red) of facade (green) or non-facade (red)

Figure 5 :

 5 Figure 5: Left column: probabilities for each cue values to be obtained on a facade (in green) or on a non facade (in red). Right column: best rectangle obtained among all candidates for each cue, in example images of our database (for the shape cue, the whole heatmap of the histogram H is shown).

Figure 6 :

 6 Figure6: Recall rate as a function of the number of candidates chosen on ZuBuD test database. 3 versions of our facade proposal method with different cues combinations are shown : (in red) all the cues, (in green) only the problem-specific cues (contour, structure, symmetry, semantic contrast), (in blue) only the objectness-inspired cues (contour, shape, color contrast). We also compare our method to other object proposals methods.

  [START_REF] Chen | City-scale Landmark Identification on Mobile Devices[END_REF] to pool the values of the 512 feature maps of Conv5. As the resolution of Conv5 is pretty low (23 × 30) and the surrounding band too thin we use a different scheme for the spatial pooling of the band. The surrounding band can be divided into 4 bands (top, down, left, right). For each band we use a 2-levels spatial pyramid ( 1 × 1, 1 × 4 ). Consequently the SPP descriptor of the inside has 10752 dimensions whereas the SPP descriptor of the band has 10240. Their 20992 concatenated vector is the input of a neural network classifier. The classifier is composed of 2 more hidden linear layers of size 4096. They have been trained on an augmented version of our training set. Indeed we have added synthetic data to our training set. These synthetic samples are plain facade images from ImageNet and GoogleStreetView pasted in urban context images. The positive and negative samples are defined in the same way as for the training of the cue combination.

Figure 7 :

 7 Figure 7: Overview of the whole facade recognition method.

Figure 8 :

 8 Figure 8: Illustration of the diversity of facades that belong to the same class with rectification artifacts, occlusions and partial observation.

Figure 9 :

 9 Figure 9: Examples of facade recognition for AR applications. 1st row shows the case of an image-only database with a facade reference and its associated feature overlaid in the images. 2nd row shows the case of a geometrically-enriched database with a 3D model and its wireframe version projected in the images.

Figure 10 :

 10 Figure 10: Example results of the whole facade recognition method obtained on the Zubud database. The green polygons are the ground truth facades. All the facades shown in this figure were correctly recognized.

Figure 11 :Figure 12 :

 1112 Figure 11: Example of failures with different cases of false detections (images 1, 2 and 3) and matching failure with two different facades matched to the same reference (image 4)

Table 1 :

 1 Statistics of facade proposals

		Selective Search	Objectness	EdgeBox	Our
	Recall (n = 100)	74.89	74.18	70.81	85.16
	Precision (s IoU )	0.59	0.63	0.61	0.68
	Time (seconds)	0.28	1.42	0.35	0.42

Table 2 :

 2 Statistics of facade matching

		BoW	VGG	SPP (SegNet)	SPP (siamese)
	Correct match (%)	44.06	72.80	78.91	82.93
	Time (seconds)	0.29	2.19	0.07	0.05
	The very few number of SIFT features extracted in facades,

Antoine Fond and Gilles Simon are with the Université de Lorraine, Loria, Vandoeuvre-lès-Nancy, 54506, France antoine.fond@loria.fr, gilles.simon@loria.fr

Marie-Odile Berger is with the Inria Nancy Grand Est, Villers-lès-Nancy, 54600 France marie-odile.berger@inria.fr

http://www.vision.ee.ethz.ch/showroom/zubud/

http://mi.eng.cam.ac.uk/projects/relocalisation/#dataset