Antoine Fond

Marie-Odile Berger

Gilles Simon

Model-image registration of a building's facade based on dense semantic segmentation

Keywords: Image Registration, Semantic Segmentation, Camera Pose Estimation

This article presents an efficient approach for accurate registration of a building facade model "dressed" with dense semantic information. Localization sensors such as the GPS as well as vision-based methods are able to provide a camera pose in an efficient and stable way, but at the expense of low accuracy. We propose here to rely on semantic maps to improve the accuracy of a rough camera pose. Simultaneously we aim to iteratively improve the quality of the semantic map through the registration. Registration and semantic segmentation are jointly refined in an Expectation-Maximization framework. We especially introduce a Bayesian model that uses prior semantic segmentation as well as geometric structure of the facade reference modeled by Generalized Gaussian Mixtures. We show the advantages of our method in terms of robustness to clutter and change of illumination on urban images from various databases.

I. INTRODUCTION

Urban localization plays a major role in many applications including navigation aid [START_REF] Krolewski | The mobile personal augmented reality navigation system[END_REF], labeling of local touristic landmarks [START_REF] Chen | City-scale landmark identification on mobile devices[END_REF], [START_REF] Xu | Visual registration for geographical labeling in wearable computing[END_REF], and robot localization [START_REF] Wendel | Natural landmark-based monocular localization for mavs[END_REF]. The outdoor accuracy of mobile phone GPS is only 12.5 meters [START_REF] Zandbergen | Positional accuracy of assisted gps data from high-sensitivity gps-enabled mobile phones[END_REF] and can be easily worse in urban areas where the street is flanked by buildings on both sides. Vision-based solutions are prone to be more accurate. However, a recent benchmark [START_REF] Sattler | Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions[END_REF] shows that the long-term visual localization problem is far from solved.

Most of the image-based solutions rely on correspondences between features in the image and features from a city model build using Structure from Motion (SfM) algorithm [START_REF] Irschara | From structurefrom-motion point clouds to fast location recognition[END_REF], [START_REF] Li | Worldwide pose estimation using 3d point clouds[END_REF], [START_REF] Sattler | Improving image-based localization by active correspondence search[END_REF], [START_REF]Efficient & effective prioritized matching for large-scale imagebased localization[END_REF]. Though often successful and quite accurate, these methods can fail if the newly acquired image is taken in very different conditions compared to the SfM data. The main reason is that both hand-crafted [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF] and learned [START_REF] Yi | Lift: Learned invariant feature transform[END_REF] features are non-invariant e.g. to image blur, day-night and large viewpoint changes [START_REF] Schönberger | Comparative evaluation of hand-crafted and learned local features[END_REF]. Furthermore, these features are based on local information, which leads to multiple hypotheses in the presence of similar or repeated patterns. In large-scale environments, disambiguation between all hypotheses can be difficult if not impossible, and computationally expensive.

By contrast, global (image-level) descriptors and particularly ConvNet features exhibit strong robustness against appearance changes induced by the time of day, seasons, or weather conditions [START_REF] Sünderhauf | On the performance of convnet features for place recognition[END_REF] and are weakly impacted by the presence of similar or repeated patterns. However, those features only allow to approximate the pose of a query image by the one related to the closest image in a database, which is obviously not accurate unless a very dense view sampling is used [START_REF] Sattler | Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions[END_REF].

An intermediate approach was proposed in [START_REF] Fond | Facade Proposals for Urban Augmented Reality[END_REF]. In this approach, semi-global ConvNet features are used to match facade proposals generated in a query image with reference images of facades. However, the detected boundaries of a facade in the query image rarely fit exactly the reference boundaries, which usually yields coarse pose estimation.

Finally, some convolutional neural networks have been designed to regress the 6-DOF camera pose from a single RGB image in an end-to-end manner. The most emblematic of these is PoseNet [START_REF] Kendall | Posenet: A convolutional net-information[END_REF]. However, the main drawback of the PoseNet approach is that it is inaccurate (see e.g. [START_REF] Brachmann | Uncertainty-driven 6d pose estimation of objects and scenes from a single rgb image[END_REF]) unless an excessively large and well sampled training set is used.

In short, it seems that solving urban localization is faced with the choice between accurate but unstable versus stable but inaccurate methods. Several suggestions have been formulated in [START_REF] Sattler | Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions[END_REF] to increase the robustness of (local) featurebased methods, such as designing novel features, e.g., based on scene semantics [START_REF] Schönberger | Semantic visual localization[END_REF], or using multiple images for pose estimation. However, whatever improvements made to feature-based methods, they will be confronted with the local nature of the features, which, in particular, does not allow the case of repeated patterns to be handled correctly. For instance, augmenting the features with pixel-wise semantic labels such as "windows", "facade" etc. would not help to distinguish between two features in the center of similar windows on the same or a different facade. Moreover, the complexity of the SfM models (millions of points for Dubrovnik [START_REF] Li | Location recognition using prioritized feature matching[END_REF] i.e. several Go of memory) can make the method unsuitable to run on mobile devices.

On the other hand, with city-scale data available (Google Street View/Maps), interests grew in 3D textured model as an alternative to SfM models. Their richer geometric information (i.e. facade planes) in combination with vanishing points [START_REF] Fond | Facade Proposals for Urban Augmented Reality[END_REF] or coarse sensor pose prior [START_REF] Arth | Instant outdoor localization and SLAM initialization from 2.5d maps[END_REF], [START_REF] Chu | Housecraft: Building houses from rental ads and street views[END_REF], [START_REF] Reitmayr | Going out: Robust model-based tracking for outdoor augmented reality[END_REF] enables more reliable model-image registration in strong viewpoint changes. Furthermore, the texture information can be condensed into low dimension descriptors [START_REF] Fond | Facade Proposals for Urban Augmented Reality[END_REF], semantic parts [START_REF] Chu | Housecraft: Building houses from rental ads and street views[END_REF], edges only [START_REF] Reitmayr | Going out: Robust model-based tracking for outdoor augmented reality[END_REF] or even not considered at all [START_REF] Arth | Instant outdoor localization and SLAM initialization from 2.5d maps[END_REF] leading to much lighter models.

Relying on a semantic segmentation to register a reference image of a facade in a target image [START_REF] Arth | Instant outdoor localization and SLAM initialization from 2.5d maps[END_REF], [START_REF] Chu | Housecraft: Building houses from rental ads and street views[END_REF] has several advantages. First, there is no need for a complex similarity metric as semantic segmentation already manages appearance changes between the two images [START_REF] Castaldo | Semantic cross-view matching[END_REF]. Second, the registration focuses on meaningful components on both images reducing possible local minima. In the methods proposed by [START_REF] Arth | Instant outdoor localization and SLAM initialization from 2.5d maps[END_REF] and [START_REF] Chu | Housecraft: Building houses from rental ads and street views[END_REF], the highest classification score for each pixel is used for 3D-2D registration. Unfortunately, segmentation can include many misclassified pixels. For instance, a door can be easily misclassified as a window at the output of the neural network with a slightly higher score than the one assigned to the correct class.

Our method for model-image registration is summarized in Fig. 1 and is composed of three main blocks. In a first initializing step, a rough pose estimation is computed thanks to vanishing point detection, image rectification and rough facade detection. In a second step, a semantic segmentation is realized in the target rectified image. In a third step, registration is performed through an expectation-maximization (EM) algorithm based on the semantic labeling of the target and the reference image. This algorithm is the major contribution of our approach. The main idea is to consider all (and not only the highest) pixel-wise classification scores as segmentation priors in a Bayesian framework. The EM algorithm is used to iteratively compute the pose that best assigns the image pixels to their related structural elements (window, door, . . . ) extracted from a ground-truth, orthorectified semantic map. The final assignments can be seen as a posterior segmentation of the target image. Thus, if semantic segmentation makes it possible to guide the registration, registration in turn makes it possible to guide the semantic segmentation (Fig. 2). This idea is similar, in spirit, to older works estimating simultaneoulsy the pose and point correspondences in an iterative process [START_REF] David | Softposit: Simultaneous pose and correspondence determination[END_REF], [START_REF] Moreno-Noguer | Pose priors for simultaneously solving alignment and correspondence[END_REF], [START_REF] Serradell | Combining geometric and appearance priors for robust homography estimation[END_REF]. Of course, as we assume the ground-truth semantic segmentation of the reference to be known, the main goal of our method is registration. The online improvement of the target segmentation through the joint approach is rather a means to achieve that goal. The weights of the GGMM being allowed to vary during the iterations of the EM algorithm, our method is robust to facade occlusions. Last but not least, because a GGMM has an infinite support, our method is also robust to poorly initialized localization.

Section II provides a state of the art of 3D-2D registration methods based on a camera pose prior. The initialization steps of our method are described in section III.

The proposed bayesian model and expectation-maximization algorithm are presented in section IV. Finally, extensive experimental results are provided in section V.

II. RELATED WORK

In addition to extensive benchmarks presented in [START_REF] Sattler | Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions[END_REF], a recent survey on visual-based localization can be found in [START_REF] Piasco | A survey on Visual-Based Localization: On the benefit of heterogeneous data[END_REF]. This section focuses on 3D-2D registration methods based on a prior camera pose and a 2.5D or 3D model of a building or a facade, "dressed" with texture and/or semantic information. The pose prior is usually obtained through localization sensors (e.g. the GPS and a magnetic compass), though any robust, even not accurate visual-based method such as those presented above may be used. In these methods, the pose is iteratively refined starting from the prior, so that the projection of the model gets accurately aligned with its image counterpart. The different approaches differ essentially with respect to which information is used to measure this alignment.

A. Feature-based methods

Feature-based methods rely on textured 3D models of buildings [START_REF] Reitmayr | Going out: Robust model-based tracking for outdoor augmented reality[END_REF], [START_REF] Simon | Tracking-by-Synthesis Using Point Features and Pyramidal Blurring[END_REF] or reference images of facades in frontal view [START_REF] Robertson | An image-based system for urban navigation[END_REF]. In the case of a building, the 3D model is rendered using the prior pose. In the case of a facade, the target video image is orthorectified based on the prior pose. In both cases we obtain two close images between which feature matching is facilitated. Features can be edgels [START_REF] Reitmayr | Going out: Robust model-based tracking for outdoor augmented reality[END_REF] or points [START_REF] Robertson | An image-based system for urban navigation[END_REF], [START_REF] Simon | Tracking-by-Synthesis Using Point Features and Pyramidal Blurring[END_REF] and the matching procedure is based on 1D [START_REF] Reitmayr | Going out: Robust model-based tracking for outdoor augmented reality[END_REF] or 2D [START_REF] Robertson | An image-based system for urban navigation[END_REF], [START_REF] Simon | Tracking-by-Synthesis Using Point Features and Pyramidal Blurring[END_REF] cross-correlation, or comparison between descriptors [START_REF] Simon | Tracking-by-Synthesis Using Point Features and Pyramidal Blurring[END_REF]. Unfortunately, edge detection is sensitive to illumination, shadows and occlusions, as well as scene-to-camera distances. On the other hand, [START_REF] Simon | Tracking-by-Synthesis Using Point Features and Pyramidal Blurring[END_REF] has shown that fast-to-compute corner features such as Harris [START_REF] Harris | A combined corner and edge detector[END_REF] or FAST [START_REF] Rosten | Machine learning for high-speed corner detection[END_REF] are weakly repeatable between a rendered frame and a video frame, due to depth blur changes. Unless an adaptive depth blur is applied to the synthetic image, which requires a tedious calibration of the camera response curve, feature points such as SIFT [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF] or SURF [START_REF] Bay | Surf: Speeded up robust features[END_REF], both based on approximations of Laplacian of Gaussian, prove more repeatable. However, as already mentioned, SIFT features have several drawbacks, which is confirmed by our experiments (see section V).

B. Template-based methods

Template based-methods usually aim at tracking a planar patch between two image I 0 and I i . More exactly, homographies H i 0 are computed so that I 0 • H i 0 is "similar" to I i inside the patch. The similarity measure is very important with these methods.

The first measure used was the sum of the squares of the differences between pixel gray levels (L 2 -norm) [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF]. Optimization is done very quickly by gradient descent using the Gauss-Newton algorithm. While the transformation was initially limited to a simple 2D translation, the geometric models were then enriched to cover affine transformations [START_REF] Hager | Efficient region tracking with parametric models of geometry and illumination[END_REF] and homographies [START_REF] Baker | Equivalence and efficiency of image alignment algorithms[END_REF]. The computational efficiency of the minimization has been improved in [START_REF] Benhimane | Real-time image-based tracking of planes using efficient second-order minimization[END_REF] by a secondorder approximation without calculating the Hessian. The similarity measure based on the L 2 -norm remains sensitive to changes in illumination and occlusions. In [START_REF] Jurie | Real time robust template matching[END_REF], the reference image is decomposed into a pyramid of sub-images that are registered independently according to the L 2 -norm. The global solution of the registration is searched recursively in the parameter space in such a way that it maximizes the number of sub-registrations. If decomposition makes it possible to effectively treat occlusions, it can be sensitive to frequent repetitions on facades.

Kim et al. [START_REF] Kim | Intensity-based image registration using robust correlation coefficients[END_REF] use a M-estimator for a more robust similarity measure. Mutual information between images is also a measure of similarity that is less sensitive to changes in illumination and occlusions [START_REF] Viola | Alignment by maximization of mutual[END_REF] and has long been used for multimodal registration in medical imaging [START_REF] Pluim | Mutual-informationbased registration of medical images: a survey[END_REF]. While these measures significantly increase the complexity of optimization, progress has since been made that allows for effective resolution [START_REF] Dame | Accurate real-time tracking using mutual information[END_REF]. Nevertheless, the convergence of all these methods depends strongly on the accuracy of the initialization.

In the case of translational shift only, the global solution can quickly be found by phase shift in the frequency domain. This method can be generalized to similarities [START_REF] Reddy | An fft-based technique for translation, rotation, and scale-invariant image registration[END_REF] and homographies [START_REF] Zokai | Image registration using log-polar mappings for recovery of large-scale similarity and projective transformations[END_REF]. However, areas of the current image that do not correspond to the reference image can disturb the Fourier transform and cause the method to fail. This happens regularly in urban images where a building can be observed at very different scales.

C. Semantic-based methods

Semantic segmentation has been used in at least two previous works. In [START_REF] Arth | Instant outdoor localization and SLAM initialization from 2.5d maps[END_REF], an initial pose provided by a GPS is refined by fitting a coarse 2.5D (building's footprints and height) city model to the image. The rotation is computed from vanishing points and translational hypothesis are generated by matching vertical ridges of the model with vertical lines detected in the image. A semantic segmentation using a SVM classifier on local image descriptors allows pixels that belong to facades and pixels from the background to be distinguished. The log-likelihood between that probability of classification and the projected facades of the model is then maximized over all the pose hypotheses. Though this method is interesting, the accuracy of the registration relies on the pixel-wise segmentation, which is noisy and does not separate adjacent facades. Moreover, structural elements on the facades (windows, doors, etc.) are not detected by the classifier (they are simply classified as facade), though these elements would be useful to get a more accurate registration.

Chu et al. [START_REF] Chu | Housecraft: Building houses from rental ads and street views[END_REF] exploit this structural information to better estimate the camera location as well as some geometric parameters of the building's model (height of each floor, vertical positions of windows and doors, etc.). As in [START_REF] Arth | Instant outdoor localization and SLAM initialization from 2.5d maps[END_REF], the method assumes the camera pose to be initialized by GPS and requires geo-referenced footprint of buildings as a base for creating the 3D models. The problem is formulated as inference in a Markov random field, which encourages the projection of the 3D model to match the image edges, semantics (based on SegNet [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF]) and location of doors and windows (based on Edgeboxes [START_REF] Zitnick | Edge boxes: Locating object proposals from edges[END_REF] and AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]) and to differ from the background in all GoogleStreetView images around the building. Nevertheless the complexity of the inference that uses a discretized parameters search space and multiple views are disadvantages for real time applications to urban localization.

III. INITIALIZATION

Initialization of the Expectation-Maximization procedure is based on four steps (see Fig. 3): (i) vanishing points as well as the camera intrinsic parameters are computed from the image content. (ii) the image is rectified so that the facades of the buildings appear as if they where fronto-parallel to the camera (several rectified images can be obtained), (iii) facades in the rectified images are detected (approximate bounding boxes are obtained) and recognized among facades of the model, (iv) semantic segmentation and registration are initialized from the bounding boxes of the recognized facades. In the following, this initialization step will be referred to as t = t 0 . We now detail each of its subtasks.

A. Autocalibration and plane rectification

Steps (i) and (ii) of the initialization process are performed using the method described in [START_REF] Simon | A Simple and Effective Method to Detect Orthogonal Vanishing Points in Uncalibrated Images of Man-Made Environments[END_REF]. Horizontal vanishing points of the image are detected by exploiting accumulations of oriented segments around the horizon line. The principal point is assumed to be at the center of the image and the focal length is computed from a detected pair of orthogonal vanishing points. For each detected vanishing point a homography is computed, that transforms all vertical planes in the direction of the vanishing point to a fronto-parallel view of the planes.

B. Facade detection and recognition

Facades are detected and recognized in the rectified images using the method presented in [START_REF] Fond | Facade Proposals for Urban Augmented Reality[END_REF]. This method relies on image cues that measure facade characteristics such as shape, color, contours, semantic structure (windows and balconies are detected using SegNet [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF]) and symmetry. These cues are combined to generate a few facade candidates quickly. The candidates are then classified into "facade" and "non facade" through a neural network using SPP descriptors [START_REF] He | Spatial pyramid pooling in deep convolutional networks for visual recognition[END_REF]. The remaining facades are matched with the facade database using a metric learned through a siamese neural network [START_REF] Chopra | Learning a similarity metric discriminatively, with application to face verification[END_REF] taking the SPP descriptors as inputs.

C. Registration and segmentation initialization

In this method we aim to jointly solve the registration of the recognized reference to the detected facade in the target image and the segmentation of the latter into semantic parts. As the image has been previously rectified using calibrated camera intrinsics the only remaining parameters to register the reference image onto the target image are one scale parameter s (the aspect ratio is preserved) and two translational parameters (t x ,t y ). Facade recognition enables (ii) The target image is rectified so that the facades of the buildings appear as if they where fronto-parallel to the camera (top-right, here the yellow and blue vanishing points were used). (iii) A facade is detected (red dashed rectangle at topright) and recognized by using the method presented in [START_REF] Fond | Facade Proposals for Urban Augmented Reality[END_REF] (the selected reference facade is shown in Fig. 6, right). (iv) The semantic segmentation is computed inside the enlarged detected region (bottom-left). For sake of comparison, the semantic segmentation obtained in the whole rectified image is shown at bottom-right.

to select the correct facade reference to be registered in a larger facades database. Moreover thanks to facade detection we can estimate a first initialization of the registration parameters by solving the least-square problem that maps the four transformed corners of the reference to the four corners of the detection.

As the facade detection [START_REF] Fond | Facade Proposals for Urban Augmented Reality[END_REF] step relies on semantic segmentation, it also provides a first initialization of the latter. However the SegNet [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF] inference is sensitive to scale (Fig. 3, bottom row). To improve the initial segmentation we zoom into the target image and we perform another inference. Rather than restricting I to the detected facade which would mean to put too much confidence on the detection, we restrict I to an enlarged detection scaled by a constant value of 40 %.

IV. JOINT REGISTRATION AND SEMANTIC SEGMENTATION

A. Bayesian model

We wish to register the recognized image reference I re f onto the target image I in which the facade has been detected through a transformation T and simultaneously improve the quality of the semantic segmentation in the target image.

We denote L = l j 1≤ j≤K the different labels from the semantic segmentation that are characteristic of a facade architecture such as "window", "door" and "balcony". The target image is considered as sets of 2D labeled points. Let X = {X i } 1≤i≤N be a set of N data points X i = (x i , y i ) from the target image I. These points are the coordinates of the pixels i from the target image I that have a fair probability of being one of the labels P(l j |i, I) ≥ 0.01 (Fig. 4). This probability P(l j |i, I) is the score of the last layer of the CNN for semantic segmentation at X i . The ground-truth semantic segmentation corresponding to the image reference I re f is assumed known (Fig. 6, left) and is modeled in a compact way as follows. For each label l j , we extract the connected components of the reference segmentation, and a generalized Gaussian N p with shape parameter p is fitted to each of them. Generalized Gaussians [START_REF] Yu | A review of estimating the shape parameter of generalized gaussian distribution[END_REF] are well suited for facade architectural components as the

L p norm M p p,Σ = m p x Σ xx + m p y
Σ yy unit ball is roughly rectangular with a high value of p (Fig. 5). To properly model the rectangular shape of facade components and keep the computation tractable we choose p = 4 (Fig. 6, right). As the image is rectified and the shape of the connected components is typically rectangular, the axes of the generalized Gaussians are aligned with the image axes. Let m j be the number of the connected component labelled with l j . The centers of the generalized Gaussians µ k j 1≤k j ≤m j are initialized to the mean of the pixels coordinates of the connected components and the covariances

Σ k j = diag σ p/2
x , σ p/2 y are initialized from their vertical and horizontal variances (respectively σ x and σ y ). They are then refined by minimizing the error between the connected component and the true generalized Gaussian form using Gauss-Newton.

Finally, the ground-truth semantic segmentation is modeled by a mixture of generalized Gaussian distributions for each label l j : π k j , µ k j , Σ k j 1≤k j ≤m j . The mixture priors π k j 1≤ j≤K ,1≤k j ≤m j are initialized such as π k j is the ratio of the number of points from the connected component k j over the total number of points from the image reference.

Then they are normalized so that ∑ j,k j π k j = 1. The goal is to estimate the geometric transformation T (Θ) of parameters Θ = (t x ,t y , s) that registers these generalized Gaussians to the set of observed data points X from the target image I. In addition, the assignment of a data point X i to a transformed generalized Gaussian can be seen as a posterior segmentation. Assuming that the observed data X are independent and taking the logarithm, the a posteriori distribution is maximized to find Θ :

Θ = arg max Θ ln P(X|Θ, I)P(Θ) = arg max Θ N ∑ i=1 ln P(X i |Θ, I) + ln P(Θ), (1)
with N the number of data points. Using the law of total probability, P(X i |Θ, I) can be expressed as follows:

P(X i |Θ, I) = K ∑ j=1 P(X i |l j , Θ, I)P(l j |i, Θ, I) + P(X i |o, Θ, I)P(o|i, Θ, I), (2)
where K is the number of labels and:

• P(X i |l j , Θ, I) is the likelihood of an observation X i given its assignment to label j through the transformation T , and is modeled by a mixture of transformed generalized Gaussians:

P(X i |l j , Θ, I) = m j ∑ k j =1 π k j N p X i |T µ k j , s p Σ k j = m j ∑ k j =1 π k j exp -X i -T µ k j p p,s p Σ k j 4/p 2 Γ(1/p) 2 |s p Σ k j | , (3)
• P(l j |i, Θ, I) is the segmentation prior probability. Thanks to the scale reestimation and the invariance of CNN to small translations, the semantic segmentation inference is pretty stable. Thus we can assume that P(l j |i, Θ, I) = P(l j |i, Θ (t 0 ) , I); • the likelihood of an observation given its assignment to the outlier class P(X i |o, Θ, I) = P(X i |o, I) is modeled as a uniform distribution P(X i |o, I) = 1 HW , with H,W the dimensions of the target image;

• ν = P(o|i, Θ, I) is the outliers rate. To be more robust to clutter we let the mixture weights free to vary during the inference but, as a tradeoff, we assume a prior distribution over them. We can actually add the mixture weights to Θ = t x ,t y , s, {π k j } 1≤ j≤K ,1≤k j ≤m j , α without changing Eq. 1. We don't assume any prior for the transformation parameters (t x ,t y , s) but we choose a Dirichlet distribution as a prior for the mixture weights π k j :

P(Θ) = Dir π k j |α k j 1≤ j≤K ,1≤k j ≤m j ∝ ∏ j,k j π α k j -1 k j (4)
Gauvain et al. [START_REF] Gauvain | Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains[END_REF] show that Dirichlet distribution is a practical prior candidate for mixture distributions that enables closed-form optimizations in the EM framework. α k j 1≤ j≤K ,1≤k j ≤m j are set to the same values as the initialized mixture priors α k j = π

(t 0 ) k j .

B. Expectation-Maximization

This Maximum A Posteriori (MAP) problem can be solved in the framework of expectation-maximization. We define the latent variables Z = z i, j,k j ∈ {0, 1}, z i,o ∈ {0, 1} 1≤i≤N ,1≤ j≤K ,1≤k j ≤m j such that z i, j,k j = 1 means that X i is assigned to a generalized Gaussian T µ k j , s p Σ k j from the label l j and z i,o = 1 means that X i is assigned to the outlier extra class o. The Expectation-Maximization algorithm seeks to find the solution iteratively by alternating between calculating the expected complete-data log-likelihood Q(Θ|Θ (t) ) with respect to Z given X and the current parameters Θ (t) and finding the parameters Θ that maximizes this quantity :

Q Θ|Θ (t) = E Z|X,Θ (t) ln P(X, Z|Θ) = ∑ Z P(Z|X, Θ (t) ) ln P(X, Z|Θ) = ∑ i, j ∑ k j β i, j,k j ln π k j + ln P(l j |i, I) + ∑ i, j ∑ k j β i, j,k j ln N p X i |T µ k j , s p Σ k j + ∑ i γ i ln ν HW (5)
with β i, j,k j = E z i, j,k j |X, Θ (t) and

γ i = E z i,o |X, Θ (t)
Thus the Expectation-Maximization framework iterates between the two steps :

• E-Step: compute β i, j,k j and γ i • M-Step: Θ (t+1) = arg max Θ Q Θ|Θ (t) + ln P(Θ)
The E-Step can be seen as the computation of an assignment probability of each data point X i to a generalized Gaussian T µ k j , s p Σ k j from the label l j knowing the current pa-rameters

Θ (t) = {π k j } (t) 1≤ j≤K ,1≤k j ≤m j , α (t) ,t (t)
x ,t (t) y , s (t) . Using Bayes rule and by denoting λ = ν HW , we can write :

β i, j,k j = E z i, j,k j |X, Θ (t) = π k j N p X i |T µ k j , s p Σ k j P(l j |i, I) ∑ j ,k j π k j N p X i |T µ k j , s p Σ k j P(l j |i, I) + λ (6)
γ i = E z i,o |X, Θ (t) = λ ∑ j ,k j π k j N p X i |T µ k j , s p Σ k j P(l j |i, I) + λ (7)
In the M-Step we aim to maximize R = Q Θ|Θ (t) + ln P(Θ) knowing the assignments β i, j,k and γ i . By replacing the expressions of the distribution from equations 3 and 4 and by ignoring the constant terms, R can be re-written as R:

R = -∑ i, j,k j β i, j,k j 2 ln |s p Σ j,k j | + X i -T µ k j p p,s p Σ j,k j + ∑ i, j,k j β i, j,k j ln π k j + ∑ i γ i ln λ + ∑ j,k j α k j -1 ln π k j (8)
The form of R permits independent maximization of each of the following parameters sets (t x ,t y , s), {π k j } 1≤ j≤K ,1≤k j ≤m j and α. From the partial derivatives ∂

R ∂t x = ∂ R ∂t y = ∂ R
∂ s = 0 we can derive a polynomial system which cannot be solved in closed-form for p = 4. Our solving strategy is similar to the one we used in the initialization of the mixture from the reference. First, we solve the polynomial system in closed-form with p = 2, setting the partial derivatives of R to zero. This leads to solving a polynomial system of one quadratic equation in s and two linear equations in t x and t y . The closed-form solution is the following:

     s = -2a 2 a 7 a 8 +a 3 a 5 a 8 +a 4 a 6 a 7 ± √ ∆ 4a 9 a 7 a 8 t x = -a 3 -2a 5 s 2a 7 t y = -a 4 -2a
a 1 = -∑ i, j,k j β i, j,k j x 2 i σ k j ,x + y 2 i σ k j ,y a 2 = ∑ i, j,k j β i, j,k j x i µ k j ,x σ k j ,x + y i µ k j ,y σ k j ,y a 3 = 2 ∑ i, j,k j β i, j,k j x i σ k j ,x a 4 = 2 ∑ i, j,k j β i, j,k j y i σ k j ,y a 5 = -∑ i, j,k j β i, j,k j µ k j ,x σ k j ,x a 6 = -∑ i, j,k j β i, j,k j µ k j ,y σ k j ,y a 7 = -∑ i, j,k j β i, j,k j σ k j ,x a 8 = -∑ i, j,k j β i, j,k j σ k j ,y a 9 = 2 ∑ i, j,k j β i, j,k j
Then, we refine the result by minimizing

J = ∂ R ∂t x 2 + ∂ R ∂t y 2 + ∂ R ∂ s 2
for p = 4 using gradient descent. As J is polynomial both the gradient and the hessian can be computed using their polynomial expression in the Gauss-Newton algorithm. The convergence is reached after a few iterations and we can update the transformation parameters t , s (t+1) . The update for the mixtures weights π k j and the outliers rate α follows the formula from [START_REF] Gauvain | Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains[END_REF]:

π (t+1) k j = ∑ i β i, j,k j + α k j -1 ∑ i,k j β i, j,k j + ∑ k j α k j -1 (9) α (t+1) = ∑ i γ i ∑ i,k j β i, j,k j + ∑ k j α k j -1 (10)
V. RESULTS

A. Implementation and efficiency

Unlike most EM approaches, in our method the generalized Gaussian parameters are fixed except for the mixture prior weights. Indeed here the generalized Gaussians model the semantic components of the reference facade. This compact representation of a facade enables our method to be efficient. The number of generalized Gaussians is in the order of the number of windows (typically between 2 and 30). If we assume that the image is full of adjacent facades and the empty space between windows is as large as the window itself we can approximate the number of data points N ≈ 0.25HW . In our testing data, this approximation is valid with an average N = 31000. Actually registration does not request the points to be sampled at each pixel. In our implementation we use a multi-resolution scheme with 2 levels. The EM algorithm is executed on a downsampled version of the set of points X until convergence Θ (t+1) (1 : 2) -Θ (t) (1 : 2) ≤ ε t and Θ (t+1) (3) -Θ (t) (3) ≤ ε s and then executed again on the full set X from the last estimated Θ (t) .

The complexity for one iteration t of the EM algorithm is O (NK max j m j ) and parallelization is easy for the E-Step as β i, j,k j computations are independent. This efficient complexity is also a consequence of the partial solvability of the M-Step in closed-form with negligible Gauss-Newton inner-iterations. The code of our implementation is in Matlab with the EM in C. The average computation time for one iteration t is 0.023 second on an I7-3520M CPU. The number of steps for the EM to converge strongly depends on the initialization. In our testing data, only 6 iterations are needed to converge for the downsampled level and 2 more for the upper level (Fig. 7). Our M-Step optimization scheme is also faster and more accurate on this problem than homotopy continuation methods. Thus the average computation time of the whole EM is 0.121 second. To avoid the problem of the EM converging to a local maximum, we use several initializations in practice. We apply our method not only to the detected facade but also to the top-20 facade proposals [START_REF] Fond | Facade Proposals for Urban Augmented Reality[END_REF] that overlap the detected facade. The final solution is the one with the highest R values.

B. Validation with ground-truth semantic references

We test our method on 3 different datasets. The first one is VarCity 3D1 . It consists of 401 street-view images of buildings along the same street. Images are also semantically labelled and a SfM reconstruction of the scene is available as well as the camera parameters. The image viewpoints are roughly fronto-parallel and facades cover most of the image (see e.g. Fig. 9). Therefore the change of scale from the reference is minor but the translation value can be high with large image parts not visible.

The second one is the first 100 buildings from Zurich Buildings Database (ZuBuD) with 5 different viewpoints per building. Among those scenes we keep only the ones that have been correctly reconstructed by SfM2 . The diversity of viewpoints in this dataset enables a wider range of scale as well as occlusions.

The last dataset NancyLights3 aims to show the robustness of the proposed method to change in illumination. It consists of 2 time-lapses of the same facade taken from the same viewpoint at sunrise and sunset for a total of 56 images.

For each building in all 3 databases we select the facade reference from the most fronto-parallel viewpoint where the facade is fully visible with the least occlusions possible. The reference is manually segmented into the 3 semantic labels "window", "door" and "balcony" (Fig. 6). The ground truth boundaries of the reference are transferred to all the images where this facade is visible using the geometric information from the SfM model.Examples of registration results are shown in Fig. 8. More examples can be found in the supplementary material.

We compare our method to both template-based and feature-based registration between the rectified target image and the reference image. In the first category we are competing against raw detection [START_REF] Fond | Facade Proposals for Urban Augmented Reality[END_REF], L 2 norm minimization between images by gradient descent [START_REF] Benhimane | Real-time image-based tracking of planes using efficient second-order minimization[END_REF], Mutual Informa-tion maximization [START_REF] Mattes | Nonrigid multimodality image registration[END_REF], [START_REF] Smriti | Image registration using rigid registration and maximization of mutual information[END_REF], and phase correlation [START_REF] Reddy | An fft-based technique for translation, rotation, and scale-invariant image registration[END_REF]. For the optimization methods the same initializations as for our method are chosen. For the feature-based method we extract SIFT descriptors in the rectified image with fixed orientation. 2 pairs of matched SIFT descriptors using Lowe's criteria [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] are used to generate transformation samples in a RANSAC framework. The comparison is done in the image itself computing the cumulative normalized histogram of the error in translation and scale. For ZuBuD and VarCity 3D the SfM models enable us to also show the error on the camera pose translation deducted from the registration (Table I and Fig. 9). Fig. 9.

An example image from VarCity 3D (left) and the related SfM model (right). The ground-truth pose is shown in light green. Poses obtained by using SIFT+RANSAC, phase correlation, least squares, mutual information and our method are shown in blue, red, cyan, dark green and purple, respectively. Our result is superimposed on the ground truth.

The good results on VarCity 3D (Fig. 10) show that our method can handle large translations thanks to the infinite generalized Gaussian support. Even when this phenomenon concurs with very repetitive patterns, the multiple initializations that exploit those repetitions and symmetries as well as the MAP regularization globally provide a correct registration. On the contrary these conditions are a major weakness for template-based methods that get easily stuck in a local minima (Fig. 11, top). Still, in our method, the lack of discriminative architectural components like doors can cause the same shift in registration aligning the wrong floor or windows when SIFT can handle it using other features.

On the other hand, our approach gives the best results on this dataset and benefits from a decent initial detection (Fig. 10, black, middle). Occlusions are another consequence of the diversity in viewpoints. Updating the mixture weights during the EM enables our method to be robust to them (Fig. 11, bottom) as well as hidden parts (Fig. 11, middle) as π k j values can decrease if a component is not visible. Acting as a regularizer, the Dirichlet prior on mixture weights avoids complete ignorance of data by keeping the mixture close to their original values α k j as shown in Fig. 12. These results were obtained with p = 2 but this behavior is independent of the value of p.

The visual appearance of facades can change a lot : windows can change according to sun reflexions and to the presence of closed shutters, balconies orientation are dependent on viewpoints. If it can be noticed on ZuBuD it is clearer for the last database where the robustness to illumination changes is evaluated (Fig. 10, bottom). Relying on semantic segmentation enables our method to focus on the geometric structure of the facade whereas the changes in appearance are encoded in the network. The illumination invariance of the network handles extreme changes in lighting that make other methods fail (Fig. 11).

Though the semantic segmentation prior P(l j |i, I) is not updated during the EM, label assignments can change from one iteration to another as shown in Fig. 2 where the prior and the posterior semantic segmentation are shown: some points not classified as windows in the prior are correctly classified in the posterior segmentation. The doors on the ground-floor wrongly classified in the prior segmentation are progressively correctly classified as well as they are guiding the registration process. Globally, if misclassification is common for visually similar labels like "door" and "window", the prior probability of the expected label can be increased

C. Method analysis and discussion

Our approach is well suited for images with sparse structures as facades but cannot be generalized to all kind of images because of spatial distributions chosen to model them (generalized Gaussians and uniform distribution for outliers). Moreover, in cases where data points are close to a uniform distribution densely sampled (e.g. a facade densely covered with windows), the method tends to label all points as outliers or as belonging to one Gaussian if the initialization is not close enough (see e.g. Fig. 13).

Using ground-truth semantic references can be seen as a limitation as this kind of information is not easily available for augmented reality or robotics applications. However, ground-truth segmentation may be carried out by precise and efficient automatic methods, although costly in terms of computing time. We are thinking of methods based on shape grammars [START_REF] Teboul | Parsing facades with shape grammars and reinforcement learning[END_REF]. The structural organization of facades, which originates from architectural rules, make them good candidates for this kind of model. The fact that these methods are expensive in computing time is not an issue since these operations can be done offline, and must be done only once. Another possible way to automate segmentation in the reference images would be to simply use our segmentation network. An example is shown in Fig. 14. Registration from a manually labeled reference image (left) is compared with registration from an automatically segmented one (right). Au- The unique presence of the "window" semantic label in conjunction with non respect of the planar geometry hypothesis led our method to end in a local minimum where all points are assigned to a single large Gaussian.

tomatically calculated labels are much more noisy than those set manually, but the noise is partially neutralized during the registration process thanks to the overall coherence of the labels, leading to a less precise but not outlier result. Moreover, automatic segmentations may, again, be offline post-processed by introducing regularizing information based on architectural rules [START_REF] Kozinski | A mrf shape prior for facade parsing with occlusions[END_REF], [START_REF] Cohen | Efficient structured parsing of facades using dynamic programming[END_REF].

Our method depends on the accuracy of the facade rectification resulting from the detection of Manhattan vanishing points. Figure 15 sheds light on the sensitivity of segmentation and registration to the accuracy of this stage. A Gaussian noise of standard deviation varying from 0.2 to 1.2 was applied to the ends of the line segments used for the vanishing point calculation (ten times for each noise level), which results in more imprecise rectifications than what we normally obtain. Segmentation and registration are performed on the warped images and the registration results are transferred back-again into the unwarped image. This experiment thus contributes to evaluate how rectification and segmentation errors affect the whole process. Figure 15(top) shows the resulting variability of the facade boundaries, which is increasingly greater as the noise increases. The graphics plot the mean orientation errors and the mean position relative errors versus the noise level. These errors are lower than 2 and, respectively, 3% as long as the faade deformation induced by Gaussian noise remains reasonable. Beyond that (Fig. 15(bottom-left), it is no longer a question of deformations obtained due to inaccuracies in the line segment detection, but rather to a notable failure of the vanishing point extraction procedure, against which nothing can be done anyway. It can be noticed that the semantic segmentation itself is very little affected by these geometric deformations, even when they are relatively large as in the example in Fig. 15.

Finally, a mean to improve the accuracy of the framework and to reduce the sensitivity of the semantic segmentation to scale could be to perform semantic segmentation at each step of the EM algorithm. Indeed, the semantic segmentation network was mainly trained with close-up facades, which explains that the segmentation is all the better the closer we are to the boundaries of the facade. In order to allow for convergence, segmentation is done on an enlarged window (40%) around the predicted position of the facade, in order to be as close as possible to the facade but to be sure that the targeted facade is inside the window. Performing semantic segmentation at each step of the EM in conjunction with a progressive decrease of the enlargement factor could thus improve the quality of segmentation as the transformation becomes close to the actual one. We do not adopt this strategy in our experiments in order to meet real time.

VI. CONCLUSION

We have presented a Bayesian model to solve jointly facade registration and semantic segmentation. The method is efficient and handles registration issues like occlusions, repetitions and changes in illumination.

Registration is currently done from one facade in the presented work. However, if a full model of a building is available we could also extend this work to perform coregistration of the visible facades. Also in our tests, the initialization was close enough to the solution to assume that the semantic segmentation inference was stable enough and does not need to be reestimated online. In future work, this assumption could be relaxed in the model to improve accuracy and dependence on initialization.

Declaration of competing interest: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Fig. 1 .

 1 Fig. 1. Overview of the method. Starting from an initial estimate of the pose, a Generalized Gaussian Mixture Model of the identified facade is registered by Expectation-Maximization into the semantic map of the rectified image. All-not only the highest-pixel-wise classification scores are considered over iterations so that both pose and segmentation are refined during this process.

Fig. 2 .

 2 Fig. 2. Evolution of the semantic segmentation during the EM on the first 3 iterations. Top: (from left to right) the target image I with the orange building as reference, the prior semantic segmentation P(l j |i, I), and the posterior semantic segmentation after registration. Bottom: The doors on the ground-floor wrongly classified in the prior are progressively correctly classified as well as they are guiding the registration.Furthermore, operating in a Bayesian framework makes it possible to use a very compact model, namely the parameters of a generalized Gaussian mixture model (GGMM), with one generalized Gaussian per structural element of the facade. The weights of the GGMM being allowed to vary during the iterations of the EM algorithm, our method is robust to facade occlusions. Last but not least, because a GGMM has an infinite support, our method is also robust to poorly initialized localization.Section II provides a state of the art of 3D-2D registration methods based on a camera pose prior. The initialization steps of our method are described in section III.

Fig. 3 .

 3 Fig. 3. Initialization steps. (i) Vanishing points are detected using the method described in [47] (top-left, vanishing points are shown by their supporting line segments, with one color per vanishing point).(ii) The target image is rectified so that the facades of the buildings appear as if they where fronto-parallel to the camera (top-right, here the yellow and blue vanishing points were used). (iii) A facade is detected (red dashed rectangle at topright) and recognized by using the method presented in[START_REF] Fond | Facade Proposals for Urban Augmented Reality[END_REF] (the selected reference facade is shown in Fig.6, right). (iv) The semantic segmentation is computed inside the enlarged detected region (bottom-left). For sake of comparison, the semantic segmentation obtained in the whole rectified image is shown at bottom-right.

Fig. 4 .

 4 Fig. 4. Data points X from the target image I. Only the points which are likely (P(l j |i, I) ≥ 0.01) to be a characteristic facade architecture components are considered.

Fig. 5 .

 5 Fig. 5. L p norm unit ball for different values of p.

Fig. 6 .

 6 Fig. 6. Ground-truth of the semantic segmentation from the reference image I re f (left) and the associated generalized Gaussian mixtures (right).

Fig. 7 .

 7 Fig. 7. The registered reference boundaries of the image reference for each iteration of the algorithm are drawn in color according to the jet colormap. From dark blue for the initial iteration to red for the final one.

Fig. 8 .

 8 Fig. 8. Examples of registration results on VarCity 3D (first line), ZuBuD (second line) and NancyLights (third line). Left: reference facade; middle: registration result; right: A posteriori segmentation.

Fig. 10 .

 10 Fig. 10. Registration errors in Varcity (top), ZuBuD (middle) and NancyLights (bottom)

Fig. 11 .

 11 Fig. 11. Examples where other methods (red) fail to estimate the registration whereas our method (green) succeeds. The initial (dashed line) and final (plain line) registered reference boundaries overlay the target image. Top: intensity-based registration fall into local minima. Middle: SIFT+RANSAC registration fails due to facade symmetry. Bottom: strong change of illumination makes phase-correlation registration to fail.

Fig. 12 .

 12 Fig. 12. Evolution of the mixture weights of label "window" over the iterations (left) until the registration converges in the target image (middle). The weight of the occluded bottom-left window drops (light blue). The Dirichlet distribution keeps the other weights close to their initial values despite the bad prior segmentation (right).

Fig. 13 .

 13 Fig. 13.The unique presence of the "window" semantic label in conjunction with non respect of the planar geometry hypothesis led our method to end in a local minimum where all points are assigned to a single large Gaussian.

Fig. 14 .

 14 Fig. 14. On the left, registration is performed from a handmade ground truth, on the right, from a semantic segmentation inferred by the network.

Fig. 15 .

 15 Fig. 15. Registration versus vanishing points accuracy. Top: Gaussian noise of standard deviation 0.2, 0.4, 0.8 and 1.2 pixels (resp. from left to right) is applied to the ends of the line segments used to compute the vanishing points. Registration results are shown for ten trials per noise level. The reference semantic segmentation is shown in Fig. 14 left. The initial box is the same for all trials and is shown in dashed green in the bottom left image, obtained with 1.2 pixel noise. It can be seen (next image) that the targeted semantic segmentation is little affected by the geometric deformation, which is quite large in this example. Bottom right : orientation and relative position errors are plotted in function of the noise level.

TABLE I MEDIAN

 I ERRORS FOR THE 3D CAMERA TRANSLATION (RELATIVE TO THE FACADE DISTANCE)

Antoine Fond is with Synthesia Technologies, London and with Université de Lorraine & Inria

[START_REF] Chen | City-scale landmark identification on mobile devices[END_REF] Marie-Odile Berger and Gilles Simon are with Inria and Université de Lorraine, France marie-odile.berger@inria.fr, gilles.simon@loria.fr

iterations. Top: (from left to right) the target image I with the orange building as reference, the prior semantic segmentation P(l j |i, I), and the posterior semantic segmentation after registration. Bottom: The doors on the ground-floor wrongly classified in the prior are progressively correctly classified as well as they are guiding the registration.

https://varcity.ethz.ch/3dchallenge

http://ccwu.me/vsfm

This dataset is freely available on our team website https://magrit.loria.fr/dataset.html

Acknowledgments: The work of Antoine Fond was funded by the French Ministre de lEnseignement suprieur de la Recherche et de lInnovation and by Inria.


Tulika Bose
email: tulika.bose@loria.fr

Irina Illina
email: illina@loria.fr

Dominique Fohr
email: dominique.fohr@loria.fr

Unsupervised Domain Adaptation in Cross-corpora Abusive Language Detection

The state-of-the-art abusive language detection models report great in-corpus performance, but underperform when evaluated on abusive comments that differ from the training scenario. As human annotation involves substantial time and effort, models that can adapt to newly collected comments can prove to be useful. In this paper, we investigate the effectiveness of several Unsupervised Domain Adaptation (UDA) approaches for the task of cross-corpora abusive language detection. In comparison, we adapt a variant of the BERT model, trained on large-scale abusive comments, using Masked Language Model (MLM) fine-tuning. Our evaluation shows that the UDA approaches result in sub-optimal performance, while the MLM fine-tuning does better in the cross-corpora setting. Detailed analysis reveals the limitations of the UDA approaches and emphasizes the need to build efficient adaptation methods for this task.

Introduction

Social networking platforms have been used as a medium for expressing opinions, ideas, and feelings. This has resulted in a serious concern of abusive language, which is commonly described as hurtful, obscene, or toxic towards an individual or a group sharing common societal characteristics such as race, religion, gender, etc. The huge amount of comments generated every day on these platforms make it increasingly infeasible for manual moderators to review every comment for its abusive content. As such, automated abuse detection mechanisms are employed to assist moderators. We consider the variations of online abuse, toxicity, hate speech, and offensive language as abusive language and this work addresses the detection of abusive versus non-abusive comments.

Supervised classification approaches for abuse detection require a large amount of expensive annotated data [START_REF] Lee | Comparative studies of detecting abusive language on twitter[END_REF]. Moreover, models already trained on the available annotated corpus report degraded performance on new content [START_REF] Yin | Towards generalisable hate speech detection: a review on obstacles and solutions[END_REF][START_REF] Durairaj Swamy | Studying generalisability across abusive language detection datasets[END_REF][START_REF] Wiegand | Detection of Abusive Language: the Problem of Biased Datasets[END_REF]. This is due to phenomena like change of topics discussed in social media, and differences across corpora, such as varying sampling strategies, targets of abuse, abusive language forms, etc. These call for approaches that can adapt to newly seen content out of the original training corpus. Annotating such content is non-trivial and may require substantial time and effort [START_REF] Poletto | Annotating hate speech: Three schemes at comparison[END_REF][START_REF] Ombui | Annotation framework for hate speech identification in tweets: Case study of tweets during kenyan elections[END_REF]. Thus, Unsupervised Domain Adaptation (UDA) methods that can adapt without the target domain labels [START_REF] Ramponi | Neural unsupervised domain adaptation in NLP-A survey[END_REF], turn out to be attractive in this task. Given an automatic text classification or tagging task, such as abusive language detection, a corpus with coherence can be considered a domain [START_REF] Ramponi | Neural unsupervised domain adaptation in NLP-A survey[END_REF][START_REF] Plank | Domain adaptation for parsing[END_REF]. Under this condition, domain adaptation approaches can be applied in cross-corpora evaluation setups. This motivates us to explore UDA for cross-corpora abusive language detection.

A task related to abuse detection is sentiment classification [START_REF] Bauwelinck | Measuring the impact sentiment for hate speech detection on twitter[END_REF][START_REF] Rajamanickam | Joint modelling of emotion and abusive language detection[END_REF], and it involves an extensive body of work on domain adaptation. In this work, we analyze if the problem of cross-corpora abusive language detection can be addressed by the existing advancements in domain adaptation. Alongside different UDA approaches, we also evaluate the effectiveness of recently proposed Hate-BERT model [START_REF] Caselli | Hatebert: Retraining bert for abusive language detection in english[END_REF] that has finetuned BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] on a large corpus of abusive language from Reddit using the Masked Language Model (MLM) objective. Furthermore, we perform the MLM fine-tuning of HateBERT on target corpus, which can be considered a form of unsupervised adaptation. Our contribution is summarised below:

• We investigate some of the best perform-ing UDA approaches, originally proposed for cross-domain sentiment classification, and analyze their performance on the task of crosscorpora abusive language detection. We provide some insights on the sub-optimal performance of these approaches. To the best of our knowledge, this is the first work that analyzes UDA approaches for cross-corpora abuse detection.

• We analyze the performance of HateBERT in our cross-corpora evaluation set-up. In particular, we use the Masked Language Model (MLM) objective to further fine-tune Hate-BERT over the unlabeled target corpus, and subsequently perform supervised fine-tuning over the source corpus.

The remaining of this paper is structured as follows: Section 2 discusses the shifts across different abusive corpora. Section 3 surveys some recently proposed UDA models for sentiment classification and discusses the main differences in the approaches. Section 4 presents the experimental settings used in our evaluation. The results of our evaluation and a discussion on performances of different approaches are present in Section 5. Finally, Section 6 concludes the paper and highlights some future work.

2 Shifts in Abusive Language Corpora [START_REF] Saha | Learning evolving and emerging topics in social media: A dynamic nmf approach with temporal regularization[END_REF] have detailed the problem of changing topics in social media with time. Hence, temporal or contextual shifts are commonly witnessed across different abusive corpora. For example, the datasets by [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on Twitter[END_REF]; Basile et al. (2019) were collected in or before 2016, and during 2018, respectively, and also involve different contexts of discussion.

Moreover, sampling strategies across datasets also introduce bias in the data [START_REF] Wiegand | Detection of Abusive Language: the Problem of Biased Datasets[END_REF], and could be a cause for differences across datasets. For instance, [START_REF] Davidson | Automated hate speech detection and the problem of offensive language[END_REF] sample tweets containing keywords from a hate speech lexicon, which has resulted in the corpus having a major proportion (83%) of abusive content. As mentioned by [START_REF] Waseem | Bridging the gaps: Multi task learning for domain transfer of hate speech detection[END_REF], tweets in [START_REF] Davidson | Automated hate speech detection and the problem of offensive language[END_REF] originate from the United States, whereas [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on Twitter[END_REF] sample them without such a demographic constraint.

Apart from sampling differences, the targets and types of abuse may vary across datasets. For instance, even though women are targeted both in [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on Twitter[END_REF] and [START_REF] Davidson | Automated hate speech detection and the problem of offensive language[END_REF], the former involves more subtle and implicit forms of abuse, while the the latter involves explicit abuse involving profane words. Besides, religious minorities are the other targeted groups in [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on Twitter[END_REF], while African Americans are targeted in [START_REF] Davidson | Automated hate speech detection and the problem of offensive language[END_REF]. Owing to these differences across corpora, abusive language detection in a cross-corpora setting remains a challenge. This has been empirically validated by [START_REF] Wiegand | Detection of Abusive Language: the Problem of Biased Datasets[END_REF]; [START_REF] Arango | Hate speech detection is not as easy as you may think: A closer look at model validation[END_REF][START_REF] Durairaj Swamy | Studying generalisability across abusive language detection datasets[END_REF]; [START_REF] Karan | Cross-domain detection of abusive language online[END_REF] with performance degradation across the cross-corpora evaluation settings. Thus, it can be concluded that the different collection time frames, sampling strategies, and targets of abuse would induce a shift in the data.

Unsupervised Domain Adaptation

As discussed by [START_REF] Ramponi | Neural unsupervised domain adaptation in NLP-A survey[END_REF]; [START_REF] Plank | Domain adaptation for parsing[END_REF], a coherent type of corpus can typically be considered a domain for tasks such as automatic text classification. We, therefore, decide to apply domain adaptation methods for our task of crosscorpora abuse detection. Besides, UDA methods aim to adapt a classifier learned on the source domain D S to the target domain D T , where only the unlabeled target domain samples X T and the labeled source domain samples X S are assumed to be available. We denote the source labels by Y S . In this work, we use the unlabeled samples X T for adaptation and evaluate the performance over the remaining unseen target samples from D T .

Survey of UDA Approaches

There is a vast body of research on UDA for the related task of cross-domain sentiment classification. Amongst them, the feature-centric approaches typically construct an aligned feature space either using pivot features [START_REF] Blitzer | Domain adaptation with structural correspondence learning[END_REF] or using Autoencoders [START_REF] Glorot | Domain adaptation for large-scale sentiment classification: A deep learning approach[END_REF][START_REF] Chen | Marginalized denoising autoencoders for domain adaptation[END_REF]. Besides these, domain adversarial training is used widely as a loss-centric approach to maximize the confusion in domain identification and align the source and target representations [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF][START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF]. Owing to their success in cross-domain sentiment classification, we decide to apply the following pivot-based and domain-adversarial UDA approaches to the task of cross-corpora abusive language detection.

Pivot-based approaches: Following [START_REF] Blitzer | Domain adaptation with structural correspondence learning[END_REF], pivot-based approaches extract a set of common shared features, called pivots, across domains that are (i) frequent in X S and X T ; and (ii) highly correlated with Y S . Pivot Based Language Modeling (PBLM) [START_REF] Ziser | Pivot based language modeling for improved neural domain adaptation[END_REF] has outperformed the Autoencoder based pivot prediction [START_REF] Ziser | Neural structural correspondence learning for domain adaptation[END_REF]. It performs representation learning by employing a Long Short-Term Memory (LSTM) based language model to predict the pivots using other non-pivots features in the input samples from both X S and X T . Convolutional Neural Networks (CNN) and LSTM based classifiers are subsequently employed for the final supervised training with X S and Y S . Pivot-based Encoder Representation of Language (PERL) [START_REF] Ben-David | Perl: Pivot-based domain adaptation for pre-trained deep contextualized embedding models[END_REF], a recently proposed UDA model, integrates BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] with pivot-based fine-tuning using the MLM objective. It involves prediction of the masked unigram/ bigram pivots from the non-pivots of the input samples from both X S and X T . This is followed by supervised task training with a convolution, average pooling and a linear layer over the encoded representations of the input samples from X S . During the supervised task training, the encoder weights are kept frozen. Both PBLM and PERL use unigrams and bi-grams as pivots, although higher order n-grams can also be used.

Domain adversarial approaches:

Hierarchical Attention Transfer Network (HATN) [START_REF] Li | End-to-end adversarial memory network for cross-domain sentiment classification[END_REF][START_REF] Li | Hierarchical attention transfer network for crossdomain sentiment classification[END_REF] employs the domain classification based adversarial training using X S and X T , along with an attention mechanism using X S and Y S to automate the pivot construction. The Gradient Reversal Layer (GRL) [START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF] is used in the adversarial training to ensure that the learned pivots are domain-shared, and the attention mechanism ensures that they are useful for the end task. During training, the pivots are predicted using the non-pivots while jointly performing the domain adversarial training, and the supervised endtask training. Recently BERT-based approaches for UDA are proposed by [START_REF] Du | Adversarial and domain-aware BERT for cross-domain sentiment analysis[END_REF]; [START_REF] Ryu | Knowledge distillation for bert unsupervised domain adaptation[END_REF] that also apply the domain adversarial training. Adversarial Adaptation with Distillation (AAD) [START_REF] Ryu | Knowledge distillation for bert unsupervised domain adaptation[END_REF] is such a domain adversarial approach that is applied over BERT. Unlike HATN, in AAD, the domain adversarial training is done with the framework of the Adversarial Discriminative Domain Adaptation (ADDA) [START_REF] Tzeng | Adversarial discriminative domain adaptation[END_REF], using X S and X T . This aims to make the source and target representations similar. Moreover, it leverages knowledge distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] as an additional loss function during adaptation.

Adaptation through Masked Language

Model Fine-tuning with HateBERT

Rietzler et al. ( 2020); [START_REF] Xu | BERT post-training for review reading comprehension and aspect-based sentiment analysis[END_REF] show that the language model fine-tuning of BERT (using the MLM and the Next Sentence Prediction task) results in incorporating domain-specific knowledge into the model and is useful for cross-domain adaptation. This step does not require task-specific labels. The recently proposed HateBERT model [START_REF] Caselli | Hatebert: Retraining bert for abusive language detection in english[END_REF] extends the pre-trained BERT model using the MLM objective over a large corpus of unlabeled abusive comments from Reddit. This is expected to shift the pre-trained BERT model towards abusive language. It is shown by [START_REF] Caselli | Hatebert: Retraining bert for abusive language detection in english[END_REF] that HateBERT is more portable across abusive language datasets, as compared to BERT. We, thus, decide to perform further analysis over HateBERT for our task.

In particular, we begin with the HateBERT model and perform MLM fine-tuning incorporating the unlabeled train set from the target corpus. We hypothesize that performing this step should incorporate the variations in the abusive language present in the target corpus into the model. For the classification task, supervised fine-tuning is performed over the MLM fine-tuned model obtained from the previous step, using X S and Y S .

4 Experimental Setup We present experiments over three different publicly available abusive language corpora from Twitter as they cover different forms of abuse, namely [START_REF] Davidson | Automated hate speech detection and the problem of offensive language[END_REF],Waseem [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on Twitter[END_REF] and HatEval (Basile et al., 2019). Following the precedent of other works on crosscorpora abuse detection [START_REF] Wiegand | Detection of Abusive Language: the Problem of Biased Datasets[END_REF][START_REF] Durairaj Swamy | Studying generalisability across abusive language detection datasets[END_REF][START_REF] Karan | Cross-domain detection of abusive language online[END_REF], we target a binary classification task with classes: abusive and non-abusive. We randomly split Davidson and Waseem into train (80%), development (10%), and test (10%), whereas in the case of HatEval, we use the standard partition of the shared task. Statistics of the train-test splits of these datasets are listed in Table 1.

During pre-processing, we remove the URLs and retain the frequently occurring Twitter handles (user names) present in the datasets, as they could provide important information. 1 The words contained in hashtags are split using the tool Crazy-Tokenizer 2 and the words are converted into lowercase.

Evaluation Setup

Given the three corpora listed above, we experiment with all the six pairs of X S and X T for our cross-corpora analysis. The UDA approaches leverage the respective unlabeled train sets in D T for adaptation, along with the train sets in D S . The abusive language classifier is subsequently trained on the labeled train set in D S and evaluated on the test set in D T . In the "no adaptation" case, the HateBERT model is fine-tuned in a supervised manner on the labeled source corpus train set, and evaluated on the target test set. Unsupervised adaptation using HateBERT involves training of the HateBERT model on the target corpus train set using the MLM objective. This is followed by a supervised fine-tuning on the source corpus train set.

We use the original implementations of the UDA models 3 and the pre-trained HateBERT 4 model for our experiments. We select the best model checkpoints by performing early-stopping of the training while evaluating the performance on the respective development sets in D S . FastText 5 word vectors, pre-trained over Wikipedia, are used for word embedding initialization for both HATN and PBLM. PERL and AAD are initialized with the BERT baseuncased model. 6 In PBLM, we employ the LSTM based classifier. 7 For both PERL and PBLM, words with the highest mutual information with respect to the source labels and occurring at least 10 times in both the source and target corpora are considered as pivots [START_REF] Ziser | Pivot based language modeling for improved neural domain adaptation[END_REF]. Our evaluation reports the mean and standard deviation of macro averaged F1 scores, obtained by an approach, over five runs with different random initializations. We first present the in-corpus performance of the HateBERT model in Table 2, obtained after supervised fine-tuning on the respective datasets, along with the frequent abuse-related words. As shown in Table 2, the in-corpus performance is high for Davidson and Waseem, but not for HatEval. HatEval shared task presents a challenging test set and similar performance have been reported in prior work [START_REF] Caselli | Hatebert: Retraining bert for abusive language detection in english[END_REF]. Crosscorpora performance of HateBERT and the UDA models discussed in Section 3.1, is presented in Table 3. Comparing Table 2 andTable 3, substantial degradation of performance is observed across the datasets in the cross-corpora setting. This highlights the challenge of cross-corpora performance in abusive language detection.

Results and Analysis

Cross-corpora evaluation in Table 3 shows that all the UDA methods experience drop in average performance when compared to the no-adaptation case of supervised fine-tuning of HateBERT. However, the additional step of MLM fine-tuning of HateBERT on the unlabeled train set from target corpus results in an improved performance in most of the cases. In the following sub-sections, we perform a detailed analysis to get further insights into the sub-optimal performance of the UDA approaches for our task.

Pivot Characteristics in Pivot-based Approaches

To understand the performance of the pivot-based models, we probe the characteristics of the pivots used by these models as they control the transfer of information across source and target corpora. As mentioned in Section 3.1, one of the criteria for pivot selection is their affinity to the available labels. Accordingly, if the adaptation results in better performance, a higher proportion of pivots would have more affinity to one of the two classes. In the following, we aim to study this particular characteristic across the source train set and the target test set. To compute class affinities, we obtain a ratio of the class membership of every pivot p i :

r i = #abusive comments with p i #non-abusive comments with p i (1)
The ratios obtained for the train set of the source and the test set of the target, for the pivot p i , are denoted as r i s and r i t , respectively. A pivot p i with similar class affinities in both the source train and target test should satisfy:

(r i s , r i t ) < 1 -th or (r i s , r i t ) > 1 + th (2)
Here, th denotes the threshold. Ratios less than (1 -th) indicate affinity towards non-abusive class, while those greater than (1 + th) indicate affinity towards the abusive class. For every source →target pair, we select the pivots that satisfy Equation (2) with threshold th = 0.3, and calculate the percentage of the selected pivots as: perc s→t = #pivots satisfying Equation (2) #Total pivots × 100

(3) This indicates the percentage of pivots having similar affinity towards one of the two classes. We now analyze this percentage in the best and the worst case scenarios of PBLM.8

Worst cases: For the worst case of Waseem →Davidson, Equation (3) yields a low perc s→t of 18.8%. This indicates that the percentage of pivots having similar class affinities, across the source and the target, remains low in the worst performing pair.

Best case: The best case in PBLM corresponds to HatEval →Davidson. In this case, Equation (3) yields a relatively higher perc s→t of 51.4%. This is because the pivots extracted in this case involve a lot of profane words. Since in Davidson, the majority of abusive content involves the use of profane words (as also reflected in Table 2), the pivots extracted by PBLM can represent the target corpus well in this case.

Domain Adversarial Approaches

On an average, the adversarial approach of HATN performs slightly better than AAD. In order to analyze the difference, we investigate the representation spaces of the two approaches for the best case of HATN i. One of the limitations of these domain adversarial approaches is the class-agnostic alignment of the common source-target representation space. As discussed in [START_REF] Saito | Maximum classifier discrepancy for unsupervised domain adaptation[END_REF], methods that do not consider the class boundary information while aligning the source and target distributions, often result in having ambiguous and non-discriminative target domain features near class boundaries. Besides, such an alignment can be achieved without having access to the target domain class labels [START_REF] Saito | Maximum classifier discrepancy for unsupervised domain adaptation[END_REF]. As such, an effective alignment should also attempt to minimize the intra-class, and maximize the inter-class domain discrepancy [START_REF] Kang | Contrastive adaptation network for unsupervised domain adaptation[END_REF].

MLM Fine-tuning of HateBERT

It is evident from Table 3 that the MLM fine-tuning of HateBERT, before the subsequent supervised fine-tuning over the source corpus, results in improved performance in majority of the cases. We investigated the MLM fine-tuning over different combinations of the source and target corpora, in order to identify the best configuration. These include: a combination of the train sets from all the three corpora, combining the source and target train sets, and using only the target train set. Table 4 shows that MLM fine-tuning over only the unlabeled target corpus results in the best overall performance. This is in agreement to [START_REF] Rietzler | Adapt or get left behind: Domain adaptation through BERT language model finetuning for aspect-target sentiment classification[END_REF] who observe a better capture of domain-specific knowledge with fine-tuning only on the target domain.

Bridging the Gap between PERL and

HateBERT MLM Fine-tuning

Since PERL originally incorporates BERT, Table 3 reports the performance of PERL initialized with the pre-trained BERT model. As discussed in Section 3. [START_REF] Caselli | Hatebert: Retraining bert for abusive language detection in english[END_REF]. 9 This results in improved performance from PERL. However, it stills remains behind the best performing HateBERT model with MLM fine-tuning on target.

Source Corpora Specific Behaviour

In general, when models are trained over HatEval, they are found to be more robust towards addressing the shifts across corpora. One of the primary reasons is that HatEval captures wider forms of abuse directed towards both immigrants and women. The most frequent words in in this dataset rarely involve abuse directed towards target groups other than women (99.3% of the abusive comments are sexist and 0.6% racist). This is because majority of these comments have been removed before crawling. Besides, Waseem mostly involves subtle and implicit abuse, and less use of profane words.

Conclusion and Future Work

This work analyzed the efficacy of some successful Unsupervised Domain Adaptation approaches of cross-domain sentiment classification in crosscorpora abuse detection. Our experiments highlighted some of the problems with these approaches that render them sub-optimal in the cross-corpora abuse detection task. While the extraction of pivots, in the pivot-based models, is not optimal enough to capture the shared space across domains, the domain adversarial methods underperform substantially. The analysis of the Masked Language Model fine-tuning of HateBERT on the target corpus displayed improvements in general as compared to only fine-tuning HateBERT over the source corpus, suggesting that it helps in adapting the model towards target-specific language variations. The overall performance of all the approaches, however, indicates that building robust and portable abuse detection models is a challenging problem, far from being solved. Future work along the lines of domain adversarial training should explore methods which learn class boundaries that generalize well to the target corpora while performing alignment of the source and target representation spaces. Such an alignment can be performed without target class labels by minimizing the intra-class domain discrepancy [START_REF] Kang | Contrastive adaptation network for unsupervised domain adaptation[END_REF]. Pivot-based approaches should explore pivot extraction methods that account for higher-level semantics of abusive language across source and target corpora.

Figure 1 :

 1 Figure 1: (Best viewed in color) PCA based visualization of HatEval →Davidson in the adversarial approaches.

  e. HatEval →Davidson. To this end, we apply the Principal Component Analysis (PCA) to obtain the two-dimensional visualization of the feature spaces from the train set of the source corpus HatEval and the test set of the target corpus Davidson. The PCA plots are shown in Figure 1. Adversarial training in both the HATN and AAD models tends to bring the representation regions of the source and target corpora close to each other. At the same time, separation of abusive and non-abusive classes in source train set seems to be happening in both the models. However, in the representation space of AAD, samples corresponding to abusive and non-abusive classes in the target test set do not follow the class separation seen in the source train set. But in the representation space of HATN, samples in the target test set appear to follow the class separation exhibited by its source train set. Considering the abusive class as positive, this is reflected in the higher number of True Positives in HATN as compared to that of AAD for this pair (#TP for HATN: 1393, #TP for AAD: 1105), while the True Negatives remain almost the same (#TN for HATN: 370, #TN for AAD: 373).

Table 1 :

 1 Statistics of the datasets used (average comment length is reported in terms of word numbers).

	4.1 Data Description and Pre-processing
	Datasets Number of	Average	Abuse
		comments	comment	%
			length
		Train Test
	Davidson 19817 2477	14.1	83.2
	Waseem 8720 1090	14.7	26.8
	HatEval	9000 3000	21.3	42.1
	.

Table 2 :

 2

F1 macro-average (mean ± std-dev) for incorpus classification using supervised fine-tuning of HateBERT.

Table 3 :

 3 Macro average F1 scores (mean±std-dev) on different source and target pairs for cross-corpora abuse detection (Hat : HatEval, Was : Waseem, Dav : Davidson). The best in each row is marked in bold.

	Source	No-		Unsupervised Domain Adaptation
	→Target	adaptation
		HateBERT	HateBERT	PBLM	PERL-	HATN	AAD-
		supervised	MLM		BERT		BERT
		fine-tune	fine-tune
		only	on Target
	Hat →Was	66.4±1.1	68.0±1.0	57.5±3.4	57.1±1.8	57.3±1.7	60.4±7.8
	Was →Hat	57.8±0.6	56.5±1.1	51.0±5.2	55.3±0.7	53.5±0.4	55.7±1.3
	Dav →Was	67.5±0.5	66.7±0.8	57.2±4.8	67.4±1.0	57.5±6.7	41.5±2.8
	Was →Dav	60.1±4.4	67.1±2.9	46.5±1.3	48.3±1.5	28.0±2.3	35.6±3.7
	Hat →Dav	63.8±2.3	67.8±1.6	61.8±5.7	62.6±3.8	61.5±5.8	55.2±0.7
	Dav →Hat	51.3±0.2	51.4±0.4	49.9±0.2	50.3±0.9	50.3±0.5	50.4±3.0
	Average	61.2	62.9	54.0	56.8	51.4	49.8

  1, PERL applies MLM fine-tuning over the pre-trained BERT model, where only the pivots are predicted rather than all the masked tokens. Following[START_REF] Ben-David | Perl: Pivot-based domain adaptation for pre-trained deep contextualized embedding models[END_REF], after the encoder weights are learned during the MLM fine-tuning

	Source	HBERT	HBERT	HBERT
	→Target	MLM	MLM on	MLM
		on all 3	Source +	on
		corpora	Target	Target
	Hat →Was 69.7±0.8 68.9±0.6 68.0±1.0
	Was →Hat 57.2±1.4 56.8±1.1 56.5±1.1
	Dav →Was 60.2±0.7 58.8±0.8 66.7±0.8
	Was →Dav 63.4±3.9 63.4±3.9 67.1±2.9
	Hat →Dav 66.6±1.1 66.7±2.1 67.8±1.6
	Dav →Hat 51.4±0.2 51.5±0.1 51.4±0.4
	Average	61.4	61.0	62.9
	Table 4: Macro average F1 scores (mean ± std-dev)
	for Masked Language Model fine-tuning of HateBERT
	(HBERT MLM) over different corpora combinations,
	before supervised fine-tuning on source; Hat : HatEval,
	Was : Waseem, Dav : Davidson. The best in each row
	is marked in bold.
	step of PERL, they are kept frozen during super-
	vised training for the classification task. As an
	additional verification, we try to leverage the Hate-
	BERT model for initializing PERL in the same way
	as BERT is used in the original PERL model, with
	frozen encoder layers. As shown in Table 5, this
	does not result in substantial performance gains
	over PERL-BERT on average. As a further exten-
	sion, we update all the layers in PERL during the
	supervised training step and use the same hyper-
	parameters as those used for HateBERT

Table 5 :

 5 Table 2 also highlight the same. The corpus involves a mix of implicit as well as explicit abusive language. On the contrary, models trained over Waseem are generally unable to adapt well in cross-corpora settings. Since only tweet IDs were made available in Waseem, we observe that our crawled comments Dav →Was 67.4±1.0 65.9±1.3 57.3±3.1 Was →Dav 48.3±1.5 48.1±3.7 64.4±2.1 Hat →Dav 62.6±3.8 63.6±0.9 66.1±1.8 Dav →Hat 50.3±0.9 50.4±0.6 51.1±0.3 Macro average F1 scores (mean ± std-dev) of PERL initialized with BERT and HateBERT (HBERT) with frozen encoder layers, and PERL initialized with HateBERT with updates across all layers, for all the pairs (Hat : HatEval, Was : Waseem, Dav : Davidson). The best in each row is marked in bold.

	Source	PERL-	PERL-	PERL-
	→Target	BERT	HBERT	HBERT
		(frozen	(frozen	(with
		encoder	encoder	layer up-
		layers)	layers)	dates)
	Hat →Was 57.1±1.8 63.2±1.7 68.3±0.8
	Was →Hat 55.3±0.7 55.0±0.9 57.8±0.8
	Average	56.8	57.7	60.8

https://github.com/huggingface/ transformers

CNN classifier obtained similar performance.

Pivot extraction criteria are same for PBLM and PERL and similar percentages are expected with PERL.

Note that the ablation study in[START_REF] Ben-David | Perl: Pivot-based domain adaptation for pre-trained deep contextualized embedding models[END_REF] discusses the effect of the number of unfrozen encoder layers only in the MLM fine-tuning step, but not in the supervised training step for the end task.

Acknowledgements

This work was supported partly by the french PIA project "Lorraine Université d'Excellence", reference ANR-15-IDEX-04-LUE.