Yassine Guermazi
email: yassine.guermazi@lis-lab.fr

Sana Sellami
email: sana.sellami@lis-lab.fr

Omar Boucelma
email: omar.boucelma@lis-lab.fr

GeoRoBERTa: A Transformer-based Approach for Semantic Address Matching

In this paper, we describe a solution for a specific Entity Matching problem, where entities contain (postal) address information. The matching process is very challenging as addresses are often prone to (data) quality issues such as typos, missing or redundant information. Besides, they do not always comply with a standardized (address) schema and may contain polysemous elements. Recent address matching approaches combine static word embedding models with machine learning algorithms. While the solutions provided in this setting partially solve data quality issues, neither they handle polysemy, nor they leverage of geolocation information. In this paper, we propose GeoRoBERTa, a semantic address matching approach based on RoBERTa, a Transformer-based model, enhanced by geographical knowledge. We validate the approach in conducting experiments on two different real datasets and demonstrate its effectiveness in comparison to baseline methods.

Introduction

Entity Matching (EM) is the problem of identifying data instances that refer to the same real-world entities [START_REF] Elmagarmid | Duplicate record detection: A survey[END_REF][START_REF] Christen | Data Matching -Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection[END_REF]. In this paper, we address a specific EM problem where entities consist of postal addresses. More precisely, Given two postal addresses 𝐴 and 𝐵, do those addresses refer to the same real world (address) entity ? We coin this problem as Address Matching although that terminology may also refer to either work based on Geocoding [START_REF] Lin | A deep learning architecture for semantic address matching[END_REF], or to software tools such as PlaceKey 1 . Address matching is a crucial task for various location-based businesses as one may lose clients or prospects in case of delivery failure. It is a challenging one, especially in absence of a standard address model.

Formally, the address matching task may be considered as a binary classification problem [START_REF] Shan | Geographical address representation learning for address matching[END_REF][START_REF] Lin | A deep learning architecture for semantic address matching[END_REF][START_REF] Chen | Deep contrast learning approach for address semantic matching[END_REF][START_REF] Matci | Address standardization using the natural language process for improving geocoding results[END_REF][START_REF] Gschwind | Fast record linkage for company entities[END_REF] where the predicted class is either Match or No Match. However, given two companies with the same name, it is important to identify addresses that are partially similar, such as those having the same city and the same road but differ in the house number or in the case where both addresses are correct but one of them corresponds to a former address company, in order to complete addresses with up-to-date information. As a result, we consider the problem as a multiclass classification one in adding a 𝑃 𝑎𝑟𝑡𝑖𝑎𝑙𝑀 𝑎𝑡𝑐ℎ class. Table 1 shows examples of address matching. Given two Senegalese addresses 𝐴 and 𝐵, the first pair illustrates the case where there is no similarity between address elements apart from the City Dakar (NoMatch label). The second address pair has a PartialMatch label as there is a similarity between at least one of its elements (Road: Avenue Lamine Gueye), apart from the similarity between the City Dakar. The last row represents an example of a Match between addresses as all their elements are similar (except the missing PoBox in address A).

Former address matching approaches [START_REF] Matci | Address standardization using the natural language process for improving geocoding results[END_REF][START_REF] Gschwind | Fast record linkage for company entities[END_REF] are based on similarity measures and matching rules. However, these methods perform a structural comparison of addresses and are unable to identify some relationship between two addresses when they have few literal overlaps [START_REF] Lin | A deep learning architecture for semantic address matching[END_REF]. In such cases, semantic address matching is required for identifying/exhibiting semantic similarities between addresses that have the same location with different representations [START_REF] Xu | Deep transfer learning model for semantic address matching[END_REF].

Recently, semantic address matching solutions have been proposed [START_REF] Shan | Geographical address representation learning for address matching[END_REF][START_REF] Lin | A deep learning architecture for semantic address matching[END_REF][START_REF] Chen | Deep contrast learning approach for address semantic matching[END_REF], based mainly on word embedding models combined with classical Machine Learning (ML) or Deep Learning (DL). Nevertheless, these solutions may be impacted by the presence of polysemous words since they are based on static word embedding models. Polysemy cases may occur in an address when it contains a place name that refers to different places in a country or worldwide as illustrated in Table 2. Identifying and resolving polysemic situations is mandatory to avoid matching distortion. This has led to the advent of transformer-based solutions [START_REF] Vaswani | Attention is all you need[END_REF] which have shown promising results on general Entity Matching [START_REF] Li | Deep entity matching with pre-trained language models[END_REF][START_REF] Brunner | Entity matching with transformer architectures -A step forward in data integration[END_REF] thanks to their highly contextualized embedding.

This motivated us to explore the effectiveness of Transformers in address matching by proposing an approach based on RoBERTa [START_REF] Liu | Roberta: A robustly optimized BERT pretraining approach[END_REF], a pre-trained Transformer language model, for address matching in the context of French-speaking countries. Nevertheless, since these models produce address embedding mainly from linguistic contexts, they may miss some (domain) knowledge, There is no match between A and B although they contain the same place name "Garennes" which is a polysemous word as it refers to two different places: a Road and an industrial zone (in two different cities).

which is difficult to learn from raw texts. Therefore, we propose to enhance the contextual address embedding of RoBERTa by two types of geographical knowledge, obtained from address tag embedding and address geographic coordinates.

The contributions of this paper can be summarized as follows:

• We defined GeoRoBERTa, a semantic address matching approach, which relies on RoBERTa, a transformer-based model. • We injected two types of geographical knowledge into RoBERTa: address tag embedding and geographic coordinates encoding. This enables better handling of polysemy and better identification of semantic similarity between addresses.

• We conducted an extensive experimental study

where GeoRoBERTa is compared to baseline methods. Real (unstructured and structured) data, consisting of French postal addresses, has been used.

The rest of the paper is organized as follows: Related work on address matching is reviewed in Section 2. Section 3 presents a formalization of the problem. We describe our solution in Section 4, and present experimental results in Section 5. Finally, Section 6 concludes the paper.

Related Work

Address matching pipeline [START_REF] Comber | Machine learning innovations in address matching: A practical comparison of word2vec and crfs[END_REF][START_REF] Guermazi | Address validation in transportation and logistics: A machine learning based entity matching approach[END_REF] is generally composed by three steps: (1) address parsing, i.e., decomposition of an address into its different components (e.g. street name, zip code), (2) generation of an embedded address vector by means of word embedding models and (3) application of a ML or a DL model resulting in a binary address classification (Match, No Match). Word embedding techniques have gained momentum for solving the semantic address matching problem. They are integrated in address matching pipeline. For example, several studies [START_REF] Comber | Machine learning innovations in address matching: A practical comparison of word2vec and crfs[END_REF][START_REF] Guermazi | Address validation in transportation and logistics: A machine learning based entity matching approach[END_REF][START_REF] Lin | A deep learning architecture for semantic address matching[END_REF][START_REF] Chen | Deep contrast learning approach for address semantic matching[END_REF] adopted the same pipeline with different used techniques in the three steps: CRF model, Trie syntax tree algorithm, jieba library2 or rule based method [START_REF] Guermazi | Address validation in transportation and logistics: A machine learning based entity matching approach[END_REF] as address parser, Word2vec [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] or fastText [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] as word embedding models and several ML (e.g. SVM, XGBoost) and DL models (e.g. enhanced sequential inference model, Bi-LSTM, CNN) as classifiers. These works have shown the effectiveness of their proposed approaches compared to baseline methods (non word embedding-based methods) thanks to their capacity to detect semantic similarity between address attributes.

However, these approaches may present two weaknesses. The first one is related to the management of polysemous cases. In fact, these approaches are based on static word embedding models, which cannot handle polysemy as they generate static vector representations of words. Contrariwise, contextual word embedding models, among which the transformer-based ones, resolve this problem thanks to their highly contextualized embedding as demonstrated in entity matching works [START_REF] Brunner | Entity matching with transformer architectures -A step forward in data integration[END_REF][START_REF] Li | Deep entity matching with pre-trained language models[END_REF]. The second weakness is related to the leveraging of geographic information. Indeed, these approaches are designed without geographic location information, which ignores the geographic features when performing address matching. And yet, addresses that belong to the same geographic area should have intuitively similar geospatial characteristics. However, these assumptions may fail as existing methods rely only on address text which can contain vernacular content or place synonyms and does not follow a standard structure making them inherently ambiguous. Thus, modelling the problem from linguistic perspective alone is not enough.

In this context, former approaches have specifically used geocoding in the address standardization process to obtain the geolocation followed by a reverse geocoding, which generates a complete and proper address before performing the matching. This strategy has been applied for example in [START_REF] Koumarelas | Experience: Enhancing address matching with geocoding and similarity measure selection[END_REF]. Recently, some works [START_REF] Jin | A geohash based place2vec model[END_REF][START_REF] Zhang | Poi-transformers: Poi entity matching through poi embeddings by incorporating semantic and geographic information[END_REF] focused on the enrichment of Point Of Interest (POI) embedding using geographic information. The most popular form of this information is the encoding string of the geographic coordinates, obtained by the Geohash geocode system 3 . In [START_REF] Zhang | Poi-transformers: Poi entity matching through poi embeddings by incorporating semantic and geographic information[END_REF], authors proposed a POI-Transformers framework to generate POI embeddings in order to perform POI Matching. A POI is defined as an entity composed by four attributes: name, category, address and geographic coordinates. The proposed matching approach consists firstly in generating an embedding vector for each POI by fusing the text embedding of the first three attributes using BERT [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF], a Transformerbased model, and the geographic location embedding of the last attribute. Then, the similarity between each pair of POI's embedding is computed using two techniques: cosine similarity and SentEval toolkit [START_REF] Conneau | Senteval: An evaluation toolkit for universal sentence representations[END_REF]. The proposed approach achieves results comparable in terms of performance with those of existing DL-based methods (e.g. DeepER [START_REF] Ebraheem | Distributed representations of tuples for entity resolution[END_REF], DeepMatcher [START_REF] Mudgal | Deep learning for entity matching: A design space exploration[END_REF]) on general Entity Matching benchmark datasets but it outperforms them on POI Entity Matching datasets. In summary, Transformer-based models have proven their effectiveness in general entity matching but they are less explored in the address matching task. Their application on these domain-specific data should also take into account geographic information in addition to the linguistic context. From this perspective, some works start introducing geographical knowledge (geohash encoding) in Transformer-based model to perform, especially, POI matching, but they may miss additional domain information to effectively deal with polysemy. Therefore, in this work, we propose GeoRoBERTa a semantic address matching approach based on a pre-trained transformerbased language model (RoBERTa) which incorporates two types of geographical knowledge: address tag embedding and geohash encoding in order to better deal with polysemous cases and to improve the identification of semantic similarity between addresses.

Problem Statement

As discussed in Section 1, due to the heterogeneity in address representations, we need to extract some « intuitive/hidden » semantic relationships between addresses. Prior to that, we first present the address model that we adopted in this paper. Then, we provide a definition of semantic address matching, along the lines of the one provided in Xu et al [START_REF] Xu | Deep transfer learning model for semantic address matching[END_REF].

3 http://geohash.org/

Address model

Definition 1 (Address Schema). Given a set of (entity) attributes {𝑎1, .., 𝑎𝑁 }, an address 𝐴 = 𝑙𝑖𝑠𝑡 {𝑎1, .., 𝑎𝑛} where 𝑎𝑖 is the i-th address token (word) and 𝑛 is the address length, with 𝑛 ≤ 𝑁 , and 𝑙𝑖𝑠𝑡() is a "list" constructor.

More formally, to cope with different address representations (e.g., France and Senegal in this paper), we distinguish between two types of addresses:

1. A Simple Address is a sequence of attributes (Table 3) which are defined by the address model proposed in [START_REF] Guermazi | A roberta based approach for address validation[END_REF]. 2. A Complex Address is a composition of (at least) two simple addresses by means of a spatial operator. Table 4 below illustrates the proximity operator 𝑜𝑝𝑝𝑟𝑜𝑥 and the intersection operator 𝑜𝑝𝑖𝑛𝑡, while Table 5 shows two complex Senegalese addresses. respectively. The addresses on either side of the equality operator refer to the same real-world object with the same geographic location (coincide with relationship). Whereas, the addresses on either side of the approximation operator are semantically related: there is a specific relationship located in between their attributes (i.e. an address 𝐴 is located in an address 𝐵 or vice versa). In this work, the address pairs labels are defined as follows:

• Match: it is attributed to an address pair, between which there is the relationship coincide with • PartialMatch: it is attributed in two scenarios: [START_REF] Elmagarmid | Duplicate record detection: A survey[END_REF] there is a relationship located in between an address pair or [START_REF] Christen | Data Matching -Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection[END_REF] there is a relationship coincide with between a partial part of an address pair. • NoMatch: otherwise

Proposed Approach

In this section we describe GeoRoBERTa (Figure 1), a RoBERTa-based approach and model for semantic address matching.

GeoRoBERTa consists of three main tasks: (1) Data Preprocessing in order to clean data, (2) Geographical Knowledge Generation and (3) Address Matching which is based on a pre-trained RoBERTa model enhanced by the geographical knowledge in order to classify each address pair as either Match, PartialMatch or NoMatch.

Data Preprocessing

The purpose of this step is to normalize and clean addresses with removing special characters and expanding abbreviations. For that, we adopt a dictionary-based approach which provides the keywords that may be used to define the components of addresses as well as common abbreviations of these words. As we are interested in addresses belonging to French-speaking countries, we extract French keywords from official sources, in France, such as the Post Office, the INSEE4 service and unofficial sources which generally have common abbreviations, such as the list of abbreviations recognized by the Open-StreetMap5 query tools. In addition, all addresses are normalized with expanding abbreviations to their corresponding words in the created dictionary which contains a set of keywords that are likely to be used to define address's components (avenue, road, building, etc.) and their abbreviations.

Geographical Knowledge Generation

Geographic Coordinates Encoding

We augment each address by a geographical knowledge derived from the encoding of geographical location represented as a latitude (lat) and a longitude (long) pair. First, we used Google Geocoding API 6 to convert each address into geographic coordinates (lat and long). Then, we translate the two-dimensional location into geographically meaningful embeddings using Geohash [START_REF] Balkic | Geohash and UUID identifier for multi-agent systems[END_REF] which is a geocoding system that encodes the geographic location of a place into a short string of letters and digits. An important property of geohash is that two places with a long common geohash prefix are close to each other [START_REF] Lee | Efficient spatial query processing for big data[END_REF].

We append address texts with geohashes to provide the geospatial context to the RoBERTa model. Figure 2 shows an example of geographic coordinates encoding of a French address.

Generation of Address Tag Embedding

It consists of two steps: address parsing and address tag embedding.

(1) Address Parsing: The parsing of an address 𝐴 = {𝑎1, .., 𝑎𝑛} aims to assign a label 𝑙 to each word 𝑎𝑖 of 𝐴 among the corresponding list of address tags 𝑌 = {𝐼𝐵, 𝐸𝐵, 𝑃, 𝑍, 𝐻𝑁, 𝑅𝑁, 𝐷, 𝐼𝑁, 𝑃 𝑅, 𝑃 𝐵, 𝑍𝐶, 𝐶, 𝑆, 𝐶𝑂} These tags (Table 3 and4) are defined following the address model described in section 3.1.

We applied the address parsing method (Figure 3) proposed in [START_REF] Guermazi | A roberta based approach for address validation[END_REF], thanks to its effectiveness compared to several baseline methods, especially in identifying polysemous address elements. The parsing is based on the use of a RoBERTa model, which generates firstly a contextual representation of an input address 𝐴, following these two sub-steps:

• RoBERTa calculates the input representations of 𝐴 by summing over the token, position, and segment embedding. • Input address representation goes through 12 transformer encoders which capture the contextual information for each token by self-attention and produces a sequence of contextual embeddings.

The resulted representation is then provided to a tagging layer (a Fully Connected Layer) to obtain address tags, using the IOB (Inside-outside-beginning) tagging scheme [START_REF] Ramshaw | Text chunking using transformation-based learning[END_REF], where a token is labeled as B-tag if it is at the beginning of the address element, or I-tag if inside the address element but not first, otherwise O-tag. The tagging layer takes as input the last hidden state of the obtained sequence of contextual embeddings and provides as result the prediction of the tags.

(2) Address Tag Embedding: The output of the parsing step of the address 𝐴 (respectively address 𝐵) is 𝑛 tags (respectively 𝑚 tags). Since these tags are at the word level, their length is equal to the length 𝑛 of 𝐴 (respectively the length 𝑚 of 𝐵). We augment these tags by another tag (B-GC) which represents the corresponding geohash of each address. Then, we use a look-up table to map these tags to identifiers and feed a linear layer to obtain the representations of the tags of the address pair.

Address Matching

It consists of two steps (Figure 1): (1) generating a fusion of two vector representations which are the contextual vector representation of the address pair and the vector representation of the address pair tags, and (2) a classification of each pair according to resulted vectors.

Vectors Fusion

We fuse two embedding vectors as follows:

1. Contextual embedding of address pair: The byte pair encoding (BPE) tokenizer 7 of RoBERTa was used to encode the input addresses into tokens. These tokens and the two geohashes, representing the address pair (A, B), form the input to the pre-trained RoBERTa model. Then, this model generates the contextual vectors representations of the address pair (A, B). 2. Tags embedding of address pair: They are generated from the previous step (as described in Section 4.2.2).

The fusion of vectors is performed by a concatenation function which is the most popular feature-level fusion methodology [START_REF] Peters | Deep contextualized word representations[END_REF][START_REF] Zheng | Feature concatenation multi-view subspace clustering[END_REF].

Address Pair Classification

It is performed using a fully connected layer (a linear layer), which is the classifier layer by default in RoBERTa packages. This layer takes as input the resulting embedding fusion vector and generates as output the class logits (probabilities), knowing that the objective of the training is the CrossEntropy. Then, the Argmax function is applied to these probabilities to get the predicted class.

Evaluation

In this section, we describe the experiments carried out in order to evaluate our address matching approach. Source code is available at the following Git repository: https: //github.com/MatchSystem/GeoRoBERTa. The frequency of the classes (labels) of address pairs for the two datasets (French dataset denoted 𝐽𝐹 and Senegalese dataset denoted 𝐽𝑆) is given by the Table 7. Besides, 𝐽𝐹 and 𝐽𝑆 are split into the training, validation, and test sets using the ratio of 3:1:1. Table 6 shows a sample of the training set of 𝐽𝑆, on which RoBERTa is trained.

Experimental Settings

Compared Methods

We compare GeoRoBERTa with baseline methods used in some address matching related works [START_REF] Comber | Machine learning innovations in address matching: A practical comparison of word2vec and crfs[END_REF][START_REF] Guermazi | Address validation in transportation and logistics: A machine learning based entity matching approach[END_REF]. We also compare with variants of GeoRoBERTa without the Geographic Tag embedding (GT) and/or the Geohash encoding (GH) Knowledge to evaluate the effectiveness of the model after the injection of each knowledge type. We summarize these approaches below.

• Word2vec + XGBoost [START_REF] Comber | Machine learning innovations in address matching: A practical comparison of word2vec and crfs[END_REF]: in adopting this approach, we trained a Word2vec model over an address corpus (section 5.2.2) using Gensim 10 library with vectors of dimension 100, a window size of 15. Then, the model is used to generate word embedding of each address of the training dataset. We obtain the embedding of each address attribute by averaging all their words embedding. The cosine similarity between the embedding of the same type of address attributes is used as features in a XGBoost classifier implemented using scikit-learn 11 . • fastText + SVM [START_REF] Guermazi | Address validation in transportation and logistics: A machine learning based entity matching approach[END_REF]: fastText model is firstly used to obtain address embedding. It is trained over an address corpus (section 5.2.2) using Gensim library with vectors of dimension 100, a window size of 15. Then, features are obtained by applying cosine similarity between embedding of the same and of the different type of address attributes. These features serve as input to a SVM classifer. • RoBERTa: This base form of GeoRoBERTa corresponds to fine-tuning the pre-trained RoBERTa on address matching. We did not inject any geographical knowledge. This variant is similar to the entity matching approach proposed in [START_REF] Brunner | Entity matching with transformer architectures -A step forward in data integration[END_REF]. • GeoRoBERTa(GT): In this version, only geographic tags embedding knowledge has been added to 10 https://pypi.org/project/gensim/ 11 https://scikit-learn.org/stable/ RoBERTa, i.e., we removed the Geographic Coordinated Encoding block in Figure 1. • GeoRoBERTa(GH): This version includes the geohash encoding knowledge only, i.e., we removed the Generation of address tag embedding block in Figure 1.

As illustrated in Section 4, GeoRoBERTa takes as input the whole address pairs augmented with the corresponding geohash, to the contrary of the two baseline approaches [START_REF] Comber | Machine learning innovations in address matching: A practical comparison of word2vec and crfs[END_REF][START_REF] Guermazi | Address validation in transportation and logistics: A machine learning based entity matching approach[END_REF] where the input is the set of attributes of each address pair. For a fair comparison, we added two attributes to each address pair corresponding to its geohash strings.

Evaluation Setup

Hardware

The experiments were carried out on a Dell PC with the following characteristics: The compared approaches are executed on "NVIDIA Tesla K80" GPU using Google Colab (with 12 GB of RAM).

RoBERTa pre-training and fine-tuning

RoBERTa-base architecture (12-layer, 768-hidden, 12heads, 125M parameters) is used for pre-training and fine-tuning. The model is pre-trained to optimize the Masked Language Modeling objective. RoBERTa pretraining was performed with the Pytorch framework 12and Transformers library 13 with a vocabulary size of 30000 tokens. We generated two pre-trained RoBERTa models corresponding to each of the following corpora:

(1) French corpora composed of 1,048,575 addresses 14 and (2) Senegalese corpora composed of 31893 addresses collected from Web business directories 15 / 16 / 17 / 18 . These datasets have been processed according to the steps described in Section 4.1.

Hyperparameters Tuning

GeLU activation is used in RoBERTa with the ADAM Optimizer. For both tasks (parsing and matching), the dropout and learning rates are set respectively to 0.1 and 3e-5 in such a way as to maximize the accuracy in the validation set. To avoid overfitting, we use the early stop technique based on loss validation by setting a maximum number of training epochs (=12) and a batch size of 32.

Evaluation Metric

To evaluate the performance of our model and all the baselines, we use the F-measure, which is the harmonic mean of the precision, the rate of correct predictions, and the recall, the fraction of correct classes being predicted.

𝐹 -𝑚𝑒𝑎𝑠𝑢𝑟𝑒 = 2 × Precision × Recall Precision + Recall (1)

Results

Comparison with baselines

First, GeoRoBERTa is compared to baseline approaches, using the same address parsing method (RoBERTa). The evaluation results, illustrated in Table 8, show that GeoRoBERTa outperforms the other approaches on the two datasets thanks to the highly contextualized vector representations of RoBERTa compared to fastText and Word2vec. Besides, the fastText-based approach outperforms the Word2vec-based one due to the richness of the extracted features in the former approach compared to the second one. These features represent the cosine similarity between attributes from different types (e.g., Road vs District) and those from the same type (e.g., Road vs Road). Overall, the performance of the different models, in terms of F-measure, is higher in the case of French addresses (vs. Senegalese ones) due to their structured nature.

Impact of the parsing method

To evaluate the impact of the parsing on matching results, we consider three address baseline parsing methods: rules-based [START_REF] Guermazi | Address validation in transportation and logistics: A machine learning based entity matching approach[END_REF], CRFbased, and RoBERTa. Parsing evaluation results (Table 9) show that RoBERTa is more accurate than the other methods because it handles polysemous words. Table 10 illustrates the impact of the parsing method on the performance of the address-matching approaches. We note that all the matching approaches combined with a parsing method based on RoBERTa perform better than the approaches combined with CRF or those based on Rules. Besides, the impact of the parsing method is more important with Senegalese data since it contains more polysemous cases.

Ablation Study

We analyze the contribution of each type of geographic knowledge by comparing GeoRoBERTa with its variants (described in section 5.1.2). The experimental results are shown in Table 12. We first focus on comparing GeoRoBERTa(GT) and GeoRoBERTa(GH) to RoBERTa. The obtained results show that the injection of geographical knowledge (regardless of their types) slightly improves the performance as we note an increase of F-measure in GeoRoBERTa(GT) and GeoRoBERTa(GH) compared to RoBERTa on the two datasets. In fact, these models are more robust when dealing with semantic similarities and polysemy cases.

Next, we note that the precision results of GeoRoBERTa(GT) and GeoRoBERTa(GH) are close to each other. Moreover, unlike GeoRoBERTa(GT), GeoRoBERTa(GH) can detect semantic similarities between unseen addresses during the pre-training or the training steps, thanks to the geohash, as illustrated in Table 13 (first row's example): There is a Semantic similarity between Zone Industrielle Les Blanchisseries and Rue Louis Leprince Ringuet: The road exists in the zone area (Similar geohash between the two addresses).

On the other hand, GeoRoBERTa(GT) is more efficient when dealing with polysemy cases thanks to the semantic labels embedding. Indeed, polysemy cases can represent examples of ambiguous addresses that are difficult to geocode as illustrated in Table 13 (second row's example): Rufisque is a polysemous element which may refer to a Road or a District in Senegal and can be found in different geographical areas. GeoRoBERTa(GH) did not consider this polysemy case as the two generated geohash are similar, while GeoRoBERTa(GT) captures the polysemy and predicts the correct label of the address pair. Furthermore, the quality of geographic coordinates can influence the performance of GeoRoBERTa(GH). In such cases, we note that this model is almost competitive with RoBERTa for the Senegalese dataset due to the low accuracy of Google Geocoding API, which is 64 % (Table 14). On the other side, GeoRoBERTa(GH) outperforms GeoRoBERTa(GT) when dealing with the French dataset for which the geocoding accuracy is better (89%).

Overall, we can note that GeoRoBERTa outperforms all its variants against the two datasets as it leverages the two types of incorporated knowledge. The incorporation of geohash encoding allowed us to have a more efficient model able to improve the identification of semantically similar address pairs, mainly when they are not used in the training of RoBERTa. Incorporating address tag embeddings allowed GeoRoBERTa to better deal with polysemous cases, (e.g., Senegal).

Conclusion

In this paper, we described GeoRoBERTa, a transformerbased address-matching solution that relies on RoBERTa, a pre-trained transformer language model, leveraging two types of geographical knowledge during the matching phase. Extensive experimental evaluations on two real-world datasets show that our solution is effective and outperforms baseline models. Besides, the ablation study demonstrated the positive impact of geographical knowledge injection in improving the matching phase, especially in semantic similarities and polysemy cases.

In the future, we intend to extend this work in two directions: (1) evaluating the impact of the geocoding in the matching result by testing other geocoding solutions, and (2) studying the performance of GeoRoBERTa on dirty address datasets (by injecting spelling errors).

Figure 1 :

 1 Figure 1: GeoRoBERTa Architecture

Figure 2 :

 2 Figure 2: Geographic Coordinates Encoding

Figure 3 :

 3 Figure 3: Address Parsing Method

•

  Processor: Intel® Core 8th (4 core), HT, 1.9Ghz, 8Mo, 15W / UHD 620 • Hard disk: SSD 512Go M.2 SATA • RAM: 16Go 2400MHz DDR4 (2x8Go) • Operating system: Microsoft Windows 10 Pro, 64 bits

Table 1

 1 Examples of address pairs with their corresponding matching label

		Address A		Address B	Label
	Medina 39 X 18 Dakar	16 Rue Parchappe Dakar	NoMatch
	25 Avenue Lamine Gueye Dakar	59 Avenue Lamine Gueye X Galandou Diouf Dakar	PartialMatch
	4373 Sicap Amitie 3 Dakar	Sicap Amitié 3 Numero 4373 BP 3110 Dakar	Match
	Table 2
	Examples of Address Matching Challenges
	Challenge	Address A	Address B	Description
	Semantic	Immeuble Azur Dakar	12 Boulevard Djily	Immeuble Azur and 12 Boulevard Djily Mbaye
	similarity	Senegal	Mbaye Dakar Senegal	refer to the same geographic location, in Dakar
	Polysemy	Les Dunes 9002 Rue Des Garennes France	Les Garennes 78130 Les Mureaux France

Table 3

 3 Simple Address Attributes

	Address Attribute	Tag
	Country	CO
	State	S	Table 4
	City	C	Spatial Operators
	District	D
	Zone	Z	Spatial Operator	Tag
	Road	RN	𝑜𝑝 𝑖𝑛𝑡	IN
	HouseNum	HN	𝑜𝑝𝑝𝑟𝑜𝑥	PR
	POI	P
	ExtBuilding	EB
	InBuilding	IB
	Zipcode	ZC
	PoBox	PB

Table 5

 5 Spatial operators in Senegalese addresses

	Examples of Spatial	Examples of
	Operators	Addresses
	Intersection operator X	2 Avenue Ballaz X Avenue De
	(Intersect)	L'Administration Dakar Senegal
	Proximity operator Face	Route De La Gare Face Pharmacie
	(In front of)	Baol Dakar
	3.2. Semantic Address Matching

Definition 2 (Semantic Address Matching).

Given two address datasets: 𝐷1 = {𝐴1, .., 𝐴 𝑙 } and 𝐷2 = {𝐵1, .., 𝐵 𝑙 ′ }, where 𝑙 and 𝑙 ′ are the size of 𝐷1 and 𝐷2 (respectively), the Semantic Address Matching aims to find each address pair (𝐴𝑖, 𝐵𝑗), satisfying 𝐴𝑖 = 𝐵𝑗 or 𝐴𝑖 ≈ 𝐵𝑗, where 𝐴𝑖 and 𝐵𝑗 are simple or complex addresses such as 𝐴𝑖 ∈ 𝐷1 and 𝐵𝑗 ∈ 𝐷2, = and ≈ represent the equality and the approximation operator,

Table 6

 6 𝐽 𝑆 training set (extract)

	𝐴𝑑𝑑𝑟𝑒𝑠𝑠(𝐴)+𝐺𝑒𝑜ℎ𝑎𝑠ℎ 𝐴	𝑇 𝑎𝑔𝑠 𝐴	𝐴𝑑𝑑𝑟𝑒𝑠𝑠(𝐵)+𝐺𝑒𝑜ℎ𝑎𝑠ℎ 𝐵	𝑇 𝑎𝑔𝑠 𝐵	Class
	['Medina', '39', 'X', '18',
	'Dakar', 'edeedbud527v']

Table 7

 7 Frequency of labels in French and Senegalese datasets

	Label	𝐽 𝐹	𝐽 𝑆
	NoMatch	20000	2500
	PartialMatch	10000	1250
	Match	10000	1250

Table 8 F

 8 -measure of Address Matching Approaches.

	Approach Name	𝐽 𝐹	𝐽 𝑆
	Word2vec + XGBoost	0.917	0.9
	fastText + SVM	0.931	0.916
	GeoRoBERTa	0.949	0.94

Table 9 F

 9 -measure of Address Parsing Methods.

	Method Name	𝐽 𝐹	𝐽 𝑆
	Rule-based	0.962	0.905
	CRF	0.981	0.943
	RoBERTa	0.989	0.959

Table 10

 10 Impact of the Address Parsing Method on the Matching Results.

	Matching Approach	Parsing Method	𝐽 𝐹	𝐽 𝑆
		Rule-based	0.894	0.87
	Word2vec + XGBoost	CRF	0.908	0.888
		RoBERTa	0.917	0.9
		Rule-based	0.913	0.89
	fastText + SVM	CRF	0.924	0.906
		RoBERTa	0.931	0.916
		Rule-based	0.938	0.925
	GeoRoBERTa	CRF	0.944	0.934
		RoBERTa	0.949	0.94

Table 11

 11 Computation time (sec.) of Address Matching Approaches.

	Approach		𝐽 𝐹		𝐽 𝑆
	Name	Training	Evaluation	Training	Evaluation
	Word2vec + XGBoost	1381	32	96	10
	fastText + SVM	1870	38	114	12
	GeoRoBERTa	7843	127	971	36
	Runtime We evaluate the different address match-
	ing models on their training and evaluation in the test
	set. Results (

Table 11 )

 11 show that the training time of GeoRoBERTa is costly due to the deep transformer based architecture of RoBERTa. On the other hand, evaluation time of GeoRoBERTa takes just few seconds (127s for 𝐽𝐹 and 36s for 𝐽𝑆).

Table 12 F

 12 -measure of Ablation Analysis.

	Approach Name	𝐽 𝐹	𝐽 𝑆
	RoBERTa	0.935	0.926
	GeoRoBERTa(GT)	0.94	0.937
	GeoRoBERTa(GH)	0.946	0.93
	GeoRoBERTa	0.949	0.94

https://github.com/fxsjy/jieba

https://www.sirene.fr/sirene/public/variable/typvoie

https://wiki.openstreetmap.org/wiki/Name_finder:Abbreviations

https://developers.google.com/maps/documentation/geocoding

https://www.gleif.org/en/lei-data/gleif-golden-copy/ download-the-golden-copy#/

https://www.goafricaonline.com/sn

https://pytorch.org/

https://huggingface.co/docs/transformers/index

Table 14

Accuracy of the Geocoding System on the French and Senegalese Datasets.

Geocoding system

𝐽 𝐹 𝐽 𝑆 Google Geocoding API 89% 64%