A Ferrara
email: alfio.ferrara@unimi.it

A Nikolov
email: andriy.nikolov@fluidops.com

J Noessner

F Scharffe

Evaluation of Instance Matching Tools: The Experience of OAEI

Keywords: instance matching evaluation, data linking, semantic web

Nowadays, the availability of large collections of data requires techniques and tools capable of linking data together, by retrieving potentially useful relations among them and helping in associating together data representing same or similar real objects. One of the main problems in developing data linking techniques and tools is to understand the quality of the results produced by the matching process. In this paper, we describe the experience of instance matching and data linking evaluation in the context of the Ontology Alignment Evaluation Initiative (IM@OAEI). Our goal is to be able to validate different proposed methods, identify most promising techniques and directions for improvement, and, subsequently, guide further research in the area as well as development of robust tools for real-world tasks.

Introduction

Problems concerning the automatic matching of data and ontology instances, such as data linking and instance matching, are becoming crucial for the future directions of the semantic web and the web in general. The availability of large collections of data requires techniques and tools capable of linking data together, by retrieving potentially useful relations among them and helping in associating together data representing same or similar real objects. One of the main problems in developing these kind of techniques and tools is to have a methodology and a set of benchmarks for understanding the quality of the results produced by the matching process. Moreover, the developers of matching tools need a framework in which their tools can be compared with other similar tools on the same data in order to understand where improvements and new solutions are possible and needed. We addressed this needs by organizing the Instance Matching track of the Ontology Alignments Evaluation Initiative (IM@OAEI).

In the remainder of this section we introduce the instance matching problem and present the requirement for evaluating instance matching and data linking approaches. We then introduce the context of OAEI in which we conducted this evaluation.

The instance matching problem

The instance matching problem can be informally defined as a special case of the relation discovery task which takes two collections of data as input and produces a set of mappings between entities of the two collections as output. In case of instance matching task, mappings denote binary relations between entities which are considered equivalent one to another. The following can serve as a highlevel formal definition: Definition: Let D 1 and D 2 represent two datasets, each one containing a set of data individuals I i and structured according to a schema O i . Each individual I ij ∈ I i describes some entity ω j . Two individuals are said to be equivalent I j ≡ I k if they describe the same entity ω j = ω k according to a chosen identity criterion. The goal of the entity resolution task is to discover all pairs of individuals {(I 1i , I 2j )|I 1i ∈ I 1 , I 2j ∈ I 2 } such that ω 1i = ω 2j .

Actual format of data individuals depends on the format of the datasets D i . If D i represent relational databases, then database records serve as individuals I i and are identified by the primary key values. In the context of semantic web data, datasets D i are represented by RDF graphs. Individuals I i ∈ I i are identified by URIs and described using the classification schema and properties defined in the corresponding ontology O i .

While there are many instantiations of the generic relation discovery task depending on the type of relation (e.g., identifying subsumption between ontological concepts, or arbitrary relation extraction from text), instance matching has emerged as a special problem on its own with the majority of methods targeting this particular problem rather than being specifications of generic relation discovery techniques.

The task can be approached using different types of available evidence. Based on this, existing techniques for instance matching can be classified into three main categories [START_REF] Ferrara | Data linking for the semantic web[END_REF]:

• Value matching. These techniques usually serve as basic building blocks of data linking tools and focus on identifying equivalence between property values of instances. Typical examples of these techniques are string similarity metrics like edit distance or Jaro-Winkler.

• Individual matching. These techniques decide whether two individuals represent the same real-world object. They operate with descriptions of a pair of individuals, which may contain multiple attributes. They utilize aggregation of similarities between corresponding property values.

• Dataset matching. These techniques take into account all individuals in two datasets and try to construct an optimal alignment between these whole sets of individuals. They rely on the results of the individual matching and can further refine them. These techniques utilize different methods such as similarity propagation, optimization algorithms, logical reasoning, etc.

Existing tools normally combine different techniques from several categories and aim to specify an instance matching workflow which would yield high quality results. Detailed surveys of the instance matching techniques and tools created in the database community can be found in [START_REF] Elmagarmid | Duplicate Record Detection: A Survey[END_REF] and [START_REF] Köpcke | Frameworks for entity matching: A comparison[END_REF], while [START_REF] Ferrara | Data linking for the semantic web[END_REF] lists the approaches applied to the semantic web domain and linked data.

Requirements for the evaluation of instance matching and data linking approaches

The main goal of developing common evaluation approaches for data linking is to be able to validate different proposed methods, identify most promising techniques and directions for improvement, and, subsequently, guide further research in the area as well as development of robust tools for real-world tasks. In order to achieve this, the evaluation procedure must satisfy several requirements, which present non-trivial challenges.

The first subset of these requirements relates to the representative capabilities of the evaluation approach. Evaluation results must provide useful information about expected performance of evaluated tools and techniques if they are applied to other real-world tasks as well as compare different methods and choose the best suited ones. The research performed in the area of data linking primarily builds on top of two research areas: database record linkage [START_REF] Fellegi | A theory for record linkage[END_REF] and ontology matching [START_REF] Euzenat | Ontology Matching[END_REF]. In both these areas, the primary evaluation approach, which fits these requirements, is benchmarking, where a set of pre-defined tests are used on which the results produced by methods can be measured with respect to a well-defined scale. However, real-world tasks can differ with respect to many parameters: for example, domain of processed datasets, richness of information represented in these datasets, dataset size, chosen data format, availability of background knowledge which can be utilized. These parameters present different challenges to the data linking tools. Given the multitude of different possible combinations of these parameters, it is impossible to devise a single benchmarking test which would approximate all the variety of different real-world tasks. The set of benchmarking tests used for evaluation of data linking tools must aim at achieving two diverse goals:

2

This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d

• being comprehensive: including as many challenges occurring in real-world matching tasks, as possible (e.g., diversity of data formats, attributes, schemas)

• being illustrative: reflect the distribution of different data features similar to the most likely parameters of real-world tasks, i.e., a data feature which rarely occurs in real-world tasks should not dominate test data.

Besides the parameters of test data, the evaluation procedure must utilize appropriate evaluation criteria. In existing research, the most important criterion corresponds to the quality of data linking results. This is normally measured in terms of precision (proportion of correct mappings among the method results) and recall (proportion of correct mappings identified by the tool among all actual mappings). However, other evaluation criteria can be important as well: for example, given the large volumes of information which must be processed on the web of data, the computation time required by the linking tool becomes an orthogonal evaluation dimension.

The second kind of requirements are pragmatic and related to the evaluation procedure itself. In particular, the evaluation procedure should not require extensive effort, as this would discourage both tool developers and users from adopting it. Moreover, the chosen approach should ensure that the evaluation results are correctly measured with respect to the chosen criteria. This is not always trivial to achieve, especially with real-world data: e.g., constructing the gold standard alignments is problematic for large-scale datasets.

Since these requirements to some extent are mutually contradictory, it is hardly possible to satisfy them all to the full extent. However, while devising a reusable evaluation method, it is necessary to consider all of them and aim at a reasonable compromise. Although large amount of relevant work on the evaluation of both record linkage and ontology matching tools has been performed in the corresponding domains, substantial differences existing between these tasks and the problems of data linking motivated the need to develop special instance matching benchmarks.

In Section 1.3, we present the context in which we conducted a series of evaluations.

Instance Matching at the Ontology Alignment Evaluation Initiative

The Ontology Alignment Evaluation Initiative (OAEI) [START_REF] Euzenat | Ontology alignment evaluation initiative: Six years of experience[END_REF] aims to evaluate the performance of ontology matching systems on various problems related to ontology matching. The evaluation is performed as follows:

preparation phase Datasets to be matched and reference alignments are provided in advance. This gives potential participants the occasion to send observations, bug corrections, remarks and other test cases to the organizers. The goal of this preparatory period is to ensure that the delivered tests make sense to the participants. The final test base is then released after a month. The data sets do not evolve after this period.

execution phase During the execution phase, participants use their systems to automatically match the instance data from the test cases.

Participants are asked to use one algorithm and the same set of parameters for all tests in all tracks. It is fair to select the set of parameters that provide the best results. Beside parameters, the input of the algorithms must be the two datasets to be matched and any general purpose resource available to everyone, i.e., no resource especially designed for the test. In all cases datasets are serialized in RDF and contain the ontology declarations in some cases.

The expected alignments are provided in the Alignment format expressed in RDF/XML [START_REF] David | The alignment api 4.0[END_REF]. Participants also provided the papers that are published in the Ontology Matching Workshop proceedings and a link to their systems and their configuration parameters.

evaluation phase The organizers evaluate the alignments provided by the participants and return comparisons of these results. In order to ensure that it is possible to process automatically the provided results, the participants are requested to provide (preliminary) results after two months. The standard evaluation measures are precision and recall computed against the reference alignments. For the matter of aggregation of the measures we use weighted harmonic means (weights being the size of the true positives). This clearly helps in the case of empty alignments. Another technique that 3

This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d has been used is the computation of precision/recall graphs so it was advised that participants provide their results with a weight to each correspondence they found. New measures addressing some limitations of precision and recall have also been used for testing purposes as well as measures compensating for the lack of complete reference alignments.

The instance matching track was organized over the last three editions of the OAEI. 1 In this paper we describe the benchmarks we developed for evaluating instance matching tools in this context, which include both real-world (Section 3) and automatically generated (Section 4) datasets, provide an overview of our experience with using these datasets to evaluate proposed tools and discuss the lessons learnt from this experience (Section 5). Before in Section 2, we survey existing evaluations from the record-linkage and ontology matching communities.

State of the art

The task of data linking is closely related both to the record linkage problem studied in the database community and to the ontology matching task. Given that both these tasks require approximate methods, their evaluation requires estimating the output quality by performing experiments with realistic benchmarks. The evaluation initiatives performed to evaluate tools developed in these areas are relevant to the data linking domain both because they have to deal with similar requirements and because they can be partially reused.

Evaluation initiatives in the database community

In parallel with the development of record linkage algorithms, work on their evaluation has been conducted in the database community for a long time. Evaluation test sets used to validate these methods can generally be classified into two types:

• Real-world data sources. Usually, such a benchmark includes two or more publicly available datasets which originate from different sources but describe the same domain. Gold standard mappings between records in these 1 Results of every OAEI campains are available on http: //oaei.ontologymatching.org.

datasets are either created manually or validated manually after an initial automatic generation of candidates.

• Artificially generated datasets. An artificially generated benchmark is normally created by taking one reference dataset in advance and introducing artificial distortions into it in a controlled way: e.g., by removing/adding the attributes and changing their values randomly.

In this way, creation of the gold standard set of mappings is straightforward: it includes all mappings between an original record and its distorted version.

Both approaches have their advantages and disadvantages. Taking real-world data sources allows the matching techniques to be evaluated in realistic conditions. In particular, this concerns the presence of heterogeneity problems, such as specific format differences, missed and incorrect attribute values, as well as the distribution of these problems in the dataset. However, this approach also has its disadvantages: given that the parameters of the datasets depend on the domain, it is difficult to generalize the results obtained on these datasets to other application cases. Moreover, creating the set of gold standard mappings for such datasets is problematic. Ideally, all mappings have to be checked manually, which is difficult to achieve for large-scale datasets. On the other hand, artificially generated datasets provide fully controlled test conditions, in which matching challenges can be added at will. However, matching problems in the artificially generated datasets usually not cover domain-specific features, and the distribution of these problems can be unrealistic.

During the earlier stages of research public databases describing the domain of scientific publications were particularly popular as the sources of evaluation data. This was caused both by the fact that citation matching is a particularly important application domain in the academic environment as well as by the public availability of these datasets. In particular, Cora2 is a commonly used evaluation datasets in the database domain (e.g., used in [START_REF] Rendle | Object identification with constraints[END_REF], [START_REF] Domingos | Multi-relational record linkage[END_REF], [START_REF] Dong | Reference reconciliation in complex information spaces[END_REF]). It includes citations collected from the web, which referenced a set of preselected academic papers. Another dataset, ACM-DBLP was used in [START_REF] Köpcke | Comparative evaluation of entity resolution approaches with fever[END_REF], [START_REF] Thor | MOMA -a mapping-based object matching system[END_REF] and created by matching references to the same publications mentioned in the ACM 3 and DBLP4 web repositories. Several other datasets used by different researchers to evaluate their tools separately were collected within the RIDDLE repository 5 .

These datasets have also been adapted to the semantic web standards and used to evaluate the instance matching algorithms in the semantic web domain: e.g., Cora was used in [START_REF] Dong | Reference reconciliation in complex information spaces[END_REF], [START_REF] Nikolov | Refining instance coreferencing results using belief propagation[END_REF], and [START_REF] Isele | Learning linkage rules using genetic programming[END_REF], while the Restaurants dataset from the RIDDLE repository was used by [START_REF] Stoermer | Featurebased entity matching: The FBEM model, implementation, evaluation[END_REF]. The advantage of reusing these is the possibility to compare with the techniques developed in the database community, despite the differences in the format of processed data. However, these benchmark datasets are not fully representative of the challenges of the linked data environment. In particular, the data do not utilize the specific semantic web features such as, e.g., class and property hierarchy, heterogeneous schema ontologies. The second problem commonly occurring with the datasets is the lack of version consistency. Sometimes, different versions of the same dataset exist, as researchers can introduce minor modifications into a dataset in order to conduct a specific experiments and report them. Then, a modified version is re-used for other experiments.

In the process, it becomes difficult for the users to track provenance of the datasets reported in different publications, which leads to their results being compared with each other. For instance, the popular Cora dataset exists in at least four different versions 6 .

In order to avoid these problems when evaluating data linking techniques, there is a need to create benchmarks which represent realistic matching challenges occurring in linked data sources and to maintain "canonical" versions of benchmark datasets. These requirements were the primary motivations for the instance matching evaluation initiative within the OAEI evaluation campaign.

In addition to the choice of the test datasets, the evaluation methodology must include the choice of the valid quantitative evaluation measures. A variety of evaluation measures has been used to validate record linkage algorithms:

• Maximum F-Measure: harmonic mean between pairwise precision and recall achieved with the optimal settings of an algorithm [3] [7].

• Pairwise accuracy for the optimal number of pairs [START_REF] Tejada | Learning object identification rules for information integration[END_REF].

• Percentage of the correct equivalence classes (sets of equivalent instances obtained by computing the transitive closure over obtained mappings) [START_REF] Lawrence | Autonomous citation matching[END_REF].

• Proportions of true matching pairs at different error rate levels [START_REF] Winkler | Methods for record linkage and bayesian networks[END_REF].

• Precision-recall curves which visualize the algorithm's performance over the whole range of possible threshold values [START_REF] Bilenko | On evaluation and training set construction for duplicate detection[END_REF].

These similarity measures highlight different performance aspects and also have their advantages and disadvantages. F-Measure combines the precision and recall in one balanced metric, but it does not take account of true negative matches. Pairwise accuracy, on the contrary, counts both correctly identified positive and negative matches. However, the disadvantage of this is that in case of predominance of negative examples in the gold standard (which is normal for an instance matching task) the metric becomes non-discriminative: in this case, a simple matching algorithm which would reject all possible mappings between two datasets would already obtain high accuracy. Counting equivalence classes instead of atomic mappings does not consider errors within each cluster. Error rate thresholds assume a specific matching methodology [START_REF] Fellegi | A theory for record linkage[END_REF]. Finally, precision-recall curves can visualize complex behavior patterns of algorithms, but assume that these algorithms use a threshold-based cut-off, which is not always the case. Given these pros and cons, for the evaluation of semantic web data linking tools we found the precision and recall measures the most informative: given the large volumes of data containing largely distinct individuals, considering correctly identified non-matches measuring the performance of the tools does not add valuable information. To show the balance between these metrics, we used the maximum F-Measure as a single quantitative indicator and precision-recall curves as a more finegrained illustration means.

Evaluation of ontology matching tools

In the area of ontological schema matching, evaluation efforts have been joined within the Ontol-

5

This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d ogy Alignment Evaluation Initiative (OAEI) [START_REF] Euzenat | Ontology alignment evaluation initiative: Six years of experience[END_REF] to produce a comprehensive set of benchmark tests which test different aspects of existing matching tools. Since 2005, several datasets were included into the evaluation campaign in order to evaluate various aspects of the ontology matching task.

Originally, the OAEI evaluation campaign featured a single artificial benchmark (known as the benchmark dataset) that includes various tests illustrating different challenges occurring in real-world matching tasks: controlled modifications concern both the level of atomic elements (e.g., random modifications of elements/labels) and the structure level (e.g., deleting/inserting classes in the hierarchy). This serves well the purpose of checking the capabilities of schema matching tools to deal with the presence or absence of different features occurring in the ontologies. However, as evaluation experience has shown [START_REF] Euzenat | Ontology alignment evaluation initiative: Six years of experience[END_REF], this artificial benchmark is less suited for comparing the overall performance of tools: each test focuses on a specific type of situation while not providing a realistic test as a whole. To deal with this problem, the evaluation campaign was extended to include several realistic benchmarks involving real-world ontologies covering the same topics. These formed the basis of other benchmarks, in particular:

• Conference, which consists of a set of 16 ontologies dedicated to the topic of conference organisation and developed within the Onto-Farm project 7 .

• Anatomy, which includes detailed ontologies describing human and mouse anatomy 8 .

Thus, experiences of the ontology matching evaluation led to a conclusion that to achieve effective evaluation of the tools, benchmark tests have to utilise both artificial and real-world datasets.

While some ontologies used in the ontology matching benchmarks contain instance data as well, they are not fully suitable for reuse to evaluate data linking methods due to important differences between these tasks. These differences involve the following aspects.

Larger datasets. One of the main problems in matching instances with respect to the problem of matching ontology concepts is that instance 7 http://nb.vse.cz/ svatek/ontofarm.html 8 http://oaei.ontologymatching.org/2011/anatomy/index.html datasets are usually much larger than ontologies in terms of number of entities, properties and values. This puts more emphasis on the problem of matching tool performance and complexity.

Large number of literal data values. Even if literals are often used also in ontologies as labels, comments, and values of property restrictions, in instance datasets the number of properties which have literals as values is usually larger. This puts more emphasis on the problem of having specific data matching functions for strings, dates, numbers.

Identity and similarity: The same-as problem.

In ontologies, the semantics of equivalence and subsumption is well defined and can be checked through standard reasoning techniques. On the contrary, in instance datasets the semantics of same-as relations is ambiguous and, in fact, sameas relations are often used as links with different meanings. This puts emphasis on the problem of providing a formal interpretation of links resulting from the matching process.

Different role of names and property values. In ontologies, names and property values are often used as labels for concepts but the relation between these labels and the concept meaning is questionable. In fact, the semantics of ontology concepts depends on the mutual relations between concepts and the kind of constraints involving concepts. In instance datasets, property values determine the meaning of the instance at hand and they are the main objects to look at for matching purposes. Moreover, some properties are used to determine the identity of each instance, in that they contain unique values. This makes instance datasets more similar to relational database records and puts emphasis on the use of record linkage techniques for matching instances.

Different kinds of data heterogeneities. When matching labels and strings, the main problem of ontology matching is due to the fact that labels are used to describe the concept meaning in human understandable terms. Thus, the heterogeneity is mainly due to the fact that often the same label is used to denote different concepts or, on the contrary, that the same concept is labeled with different strings. This happens also in instance matching, but, in this case, there are other problems. In fact, data heterogeneity is often due to errors in 6

This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d data or different conventions about name abbreviations, acronyms and other string/date/number format heterogeneities. This requires more sophisticated techniques for syntactic data value matching. Moreover, in the generic ontology matching, diversity between descriptions of matching concepts and properties can be caused by different factors [START_REF] Klein | Change Management for Distributed Ontologies[END_REF]: in one case, usage of synonyms, in another, rephrasing of descriptions, in the third one, slight semantic difference, etc. In the instance matching tasks, it is more common that the same factor occurs for many pairs of instances and is specific for a particular pair of datasets. Thus, while for an ontology matching tool it is important to be able to apply a wide range of techniques to recognise a mapping between a pair of entities, for an instance matching method the focus is more on discovering the most appropriate set of techniques and applying them in a consistent way. The capability to adapt to specific types of data at hand is thus particularly valuable.

Structural differences between ontology and instances as graphs. Graph-based matching techniques are crucial and much used both in ontology and in instance matching. However, when seen as graphs, there are some differences between ontologies and instance datasets. Usually, the density of instance graphs is higher than ontologies and the average number of other entities connected to a given instance is also higher. This leads to more complex graph matching problems, where the identity of an instance often depends on the meaning of other instances connected to it. In addition, while the ontology schema usually represents a single graph, at the data level, information is commonly described using a set of homogeneous subgraphs. This makes it important to be able to extract and analyse relevant subgraphs which carry the instance identity.

Relations between datasets and the real-world. Typically, ontologies are used to represent a shared vocabulary of concepts. On the contrary, instances are usually intended to represent real world objects or digital documents. This implies that thesauri and top ontologies that are useful resources for ontology matching are often less useful for instance matching. On the other side, instance matching may be based on external data sources of interest so that is possible to use existing collection of objects as a reference for the validation of matching results.

Mutual relations between ontology and instance matching. Ontology matching may use instance matching as part of the matching process, in that instances characterize the meaning of concepts: they serve as the main type of evidence for instancebased ontology matching techniques. On the other hand, ontological heterogeneity presents an additional challenge for instance matching, as corresponding sets of instances and properties relevant to the instance identity can be represented in different way. In these cases, ability to perform partial ontology matching is important for an instance matching tool.

These differences have motivated the need to develop a different set of benchmarks specifically for the task of instance matching in the semantic web domain. For this reason, it was decided to establish the instance matching evaluation as a separate subtrack within the OAEI evaluation campaign, which by now has been performed 3 times (in 2009, 2010, and 2011). In the following sections, we describe the proposed benchmarks as well as the results of the tests.

The real-data benchmark

In this section we describe three benchmarks proposed to OAEI instance matching participants. These benchmarks are based on datasets actually available as web data and describing data used in applications. We also give the evaluation results on each benchmark.

OAEI 2009: interlinking scientific publications data

Starting from 2009 we proposed a track to OAEI participants focusing on instance matching.

Benchmark

In the 2009 edition the test bed was made of three datasets in the domain of scientific publications.

• AKT EPrints archive, containing information about papers produced within the AKT research project.9

• Rexa dataset, extracted from the Rexa search server10 , which was constructed at the University of Massachusetts using automatic information extraction algorithms.

• SWETO-DBLP dataset 11 , a publicly available dataset listing publications from the computer science domain.

The SWETO-DBLP dataset was originally represented in RDF. Two other datasets (AKT EPrints and Rexa) were extracted from the HTML sources using specially constructed wrappers and structured according to the SWETO-DBLP ontology. This heterogeneity resulted in many non-trivial cases of data mismatches. Sometimes, the sources contained misrepresentation of data fields, missing values, and even incorrect values (e.g., incorrect publication dates and missing authors).

The ontology describes information about scientific publications and their authors and extends the commonly used FOAF ontology. Authors are represented as individuals of the foaf:Person class, and a special class sweto:Publication is defined for publications, with two subclasses sweto:Article and sweto:Article in Proceedings for journal and conference publications respectively. The participants were invited to produce alignments for each pair of datasets (AKTRexa, AKTDBLP, and RexaDBLP).

Results

5 systems participated to the evaluation: DSSim [START_REF] Nagy | Dssim results for oaei[END_REF], RiMOM [START_REF] Li | RiMOM: A dynamic multistrategy ontology alignment framework[END_REF], OKKAM [START_REF] Stoermer | Featurebased entity matching: The FBEM model, implementation, evaluation[END_REF], HMatch [START_REF] Castano | Mapping validation by probabilistic reasoning[END_REF], and ASMOV [START_REF] Jean-Mary | Ontology matching with semantic verification[END_REF].

In this first instance matching track, 4 systems out of 5 represented generic ontology matching tools, which included instance matching as a part of their functionality, while only one (OKKAM) was specifically aimed at resolving data level coreferences.

Table 1 shows the results. The AKT/Rexa test scenario was the only one for which the results for ASMOV were available and the only one for which all the systems provided alignments for both foaf:Person and sweto:Publication classes.

OKKAM for the AKT/DBLP test case and RiMOM for the Rexa/DBLP test case only produced alignments for Publication instances, which reduced their overall recall. For the class Publication the best F-measure in all three cases was achieved by HMatch with Ri-MOM being the second. OKKAM, which specifically focused on precision, achieved the highest precision in all three cases at the expense of recall. It is interesting to see the difference between systems in 11 http://lsdis.cs.uga.edu/projects/semdis/swetodblp/ the Rexa/DBLP scenario where many distinct individuals had identical titles (e.g., "Editorial.", or "Minitrack Introduction."): this primarily affected the precision in the case of HMatch and RiMOM, but reduced recall for OKKAM.

The performance of all systems was lower for the class Person where ambiguous personal names and different label formats reduced the performance of string similarity techniques. The highest F-measure was achieved by RiMOM for the AKT/Rexa scenario and by HMatch for the AKT/DBLP and Rexa/DBLP cases. Again, it is interesting to note the difference between HMatch and OKKAM in the Rexa/DBLP case where the first system focused on F-measure and the second one on precision. This distinction of approaches can be an important criterion when a tool has to be selected for a real world use case: in some cases the cost of an erroneous correspondence is much higher than than the cost of a missed one (e.g., the large-scale entity naming service such as OKKAM) while in other scenarios this might not be true (e.g., assisting the user who performs manual alignment of datasets). In contrast, in the AKT/Rexa scenario the performance of OKKAM was lower than the performance of other systems both in terms of precision and recall. This was caused by different label formats used by AKT and Rexa datasets ("FirstName Last-Name" vs "LastName, FirstName"), which affected OKKAM most.

In this first evaluation track, the benchmark provided a representative set of realistic challenges occurring in real-world data, which gave interesting insights about the capabilities of different algorithms. However, in itself this benchmark was insufficient, as it did not cover some important aspects. Most notably, these included the large scale and ontological heterogeneity. Two out of three datasets were relatively small scale: this meant that in each pairwise matching task one of the datasets was small, which reduced the complexity of the matching task. Moreover, all three datasets were structured using the same ontology (SWETO) which also simplified the challenges, which the tools had to tackle.

OAEI 2010: interlinking health-care data

In OAEI 2010, participants were asked to interlink together four datasets, selected for their potential to be interlinked, for the availability of curated interlinks between them, and for their size.

8

This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d

Benchmark

All datasets are on the health-care domain and all of them contain information about drugs (see [START_REF] Jentzsch | Linking open drug data[END_REF] for more details on the datasets): dailymed is published by the US National Library of Medecine and has for topic marketed drugs. Dailymed contains information on the chemical structure, mechanism of action, indication, usage, contraindications and adverse reactions for the drugs.

diseasome contains information about 4300 disorders and genes.

drugbank is a repository of more than 5000 drugs approved by the US Federal Drugs Agency. It contains information about chemical, pharmaceutical and pharmacological data along with the drugs data.

sider contains information on marketed drugs and their recorded adverse reactions. It was originally published on flat files before being converted as linked-data through a relational database.

These datasets were semi-automatically interlinked using Silk [START_REF] Bizer | Silk -a link discovery framework for the web of data[END_REF] and ODD Linker [START_REF] Hassanzadeh | A declarative framework for semantic link discovery over relational data[END_REF] providing the reference alignments for this task and participants were asked to retrieve these links using an automatic method.

Results

Two systems participated in the data interlinking task: ObjectCoref [START_REF] Hassanzadeh | A declarative framework for semantic link discovery over relational data[END_REF] and RiMOM [START_REF] Li | RiMOM: A dynamic multistrategy ontology alignment framework[END_REF]. Table 3 shows the results.

The results are very different for the two systems, with ObjectCoref being better in precision and RiMOM being better in recall. A difficult task with interlinking real data is to understand if the results are due to a weakness of the matching system or because links can be not very reliable. In any case, we could conclude from this experiment with linked data that a lot of work was still required in three directions: i) providing a reliable mechanism for systems evaluation; ii) improving the performances of matching systems in terms of both precision and recall; iii) work on the scalability of matching techniques in order to make affordable the task of matching large collections of real data. We have thus built the OAEI 2011 instance matching track with these challenges in mind.

This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d

OAEI 2011: interlinking the New York Times data

While using subsets of publicly available linked data repositories has shown its value, in particular, to illustrate matching challenges occurring in realworld domains, the evaluation procedure in OAEI 2010 also identified some issues. In particular, the topic of datasets was restricted to the medical domain, which biased the overall benchmark towards the specific features of this domain. Second, due to the need to support large-scale matching tasks, the set of gold standard mappings could not be constructed manually, and evaluation had to rely on pre-existing mappings between datasets. These pre-existing mappings were constructed by semiautomated tools, which themselves did not always produce results with 100% quality.

Benchmark

To deal with these issues, the instance matching track in the OAEI 2011 included the of tests involving the New York Times (NYT) linked data 12 . The NYT repository includes three subdescribing different types of entities mentioned in the New York Times articles: people, organisations, and places. These three subsets were linked to three commonly used semantic web data repos-itories: DBpedia 13 , Freebase 14 , and Geonames 15 . These links were provided by the data publishers, which improved the gold standard quality.

The data in the NYT datasets is structured using the commonly used SKOS vocabulary 16 : individuals are modelled as instances of the class skos:Concept, and instance labels use the property skos:label rather than generic rdfs:label. Other vocabularies are used to represent domain-specific properties, such as number of relevant NYT articles and geo-coordinates (for locations).

Results

In the OAEI 2011 evaluation initiative, test results with the NYT benchmark dataset were produced using three instance matching tools: Agree-mentMaker, SERIMI, and Zhishi.links [START_REF] Cruz | Using agreementmaker to align ontologies for OAEI[END_REF][START_REF] Araujo | Serimi results for OAEI[END_REF][START_REF] Niu | Zhishi.links results for OAEI[END_REF]. Table 5 shows an overview of the Precision, Recall and F 1 -measure results per dataset for these tools, while Figure 1 shows the Precision-Recall graph for their results. In the experiments, Zhishi.links managed to produce high quality mappings consistently over all datasets: it obtained the highest scores on 4 tests out of 7, and the highest average scores. SERIMI performed particularly well on Freebase datasets (it outperformed other tools on 2 tests), while AgreementMaker was particularly successful on linking people from NYT and Freebase. However, the results also highlighted some important issues. The first one is still the quality of This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d the gold standard mappings: despite the fact the links were checked by the data publisher at the time of their generation, errors in the gold standard still occurred. These errors were caused by several factors: evolution of datasets since the time of interlinking, omitted mappings (false negatives) which are particularly difficult to discover manually, as well as the presence of ambiguous and duplicate instances in target repositories. The second issue concerns the use of specific techniques by the matching systems, in particular, the use of domainspecific knowledge such as common abbreviations.

There are different points of view on the use of domain knowledge: for example, it is explicitly forbidden in the OAEI schema matching tests, as the schema matching tools must be able to deal with generic knowledge models. Although this can be seen as too restrictive for instance matching tools, as many cases of heterogeneity in instance matching tasks cannot be resolved without possessing domain knowledge, the cases where some tool is specifically targeted at solving the benchmark tasks have to be prevented to ensure that evaluation results can be generalized. Establishing the rules concerning the use of domain knowledge constitutes an important challenges for the OAEI instance matching track and future instance matching initiatives in general as it influences the quality of comparative evaluation.

The automatically generated benchmark

The automatically generated benchmark (called IIMB) is based on the idea of automatically acquiring a potentially large set of data from an existing datasource and to represent data in form of an OWL Abox, serialized in RDF (Either in 2010 and in 2011, the dataset was extracted from Freebase17 ). Then, starting from the initial set of data, we programmatically introduce several kinds of data transformations, with the goal of producing a final set of Aboxes in a controlled way. Participants are then required to match each of the transformed Aboxes against the initial one, trying to find the correct mappings between the original entities and the transformed ones. The main advantage in such an approach is that we have a control over the type and strength of each transformation, which means that it is possible to analytically evaluate the results produced by each tool, by highlighting potential points of strength and weakness of each tool.

Creation of the benchmark

The benchmark is created using the SWING approach (Semantic Web INstance Generation) [START_REF] Ferrara | Benchmarking matching applications on the semantic web[END_REF] a disciplined approach to the semi-automatic generation of benchmarks to be used for the evaluation of matching applications. The SWING approach has been implemented as a Java application and it is available at http://code.google.com/p/swing.

The SWING approach is articulated in three phases as shown in Figure 2. These phases will be briefly described in the Sections 4.2 -4.4. These sections are a comprehensive summary of [START_REF] Ferrara | Benchmarking matching applications on the semantic web[END_REF].

Data acquisition techniques

The SWING data acquisition phase is based on the idea of acquiring data from a linked data repository in a controlled way using a set of predefined queries and then, to enrich the structural and semantic complexity of acquired data from the description logic ALE(D) up to ALCHI(D). The main reason of the enrichment step is that linked data are typically featured by a limited level of semantic complexity, while we are interested in providing a benchmark suitable for the evaluation also of logical and reasoning capabilities of the matching tools at hand. In order to briefly summarize the extensions added during the enrichment step, we report the main operations supported by SWING:

• Add super classes and super properties.

• Convert attributes to class assertions.

• Determine disjointness restrictions.

• Enrich with inverse properties.

• Specify domain and range restrictions.

All these operations are semi-automatically performed by focusing on data features like property values and class relations. The benchmark designer can choose which operations have to be applied to data in order to control the semantic complexity of the final Abox.

Data transformation techniques

In the subsequent data transformation activity the initial ABox is modified in several ways by generating a set of new ABoxes, called test cases. Each test case, is produced by transforming the individual descriptions in the reference ABox in new individual descriptions that are inserted in the test case at hand. In particular, the SWING approach supports the following automatic transformation techniques. In our implementation the evaluation designer has control over these techniques with an easy understandable parameter file. Deletion/Addition of Individuals. The SWING approach allows the evaluation designer to select a portion of individuals that must be deleted and/or duplicated in the new ontology. The reason behind this functionality is to obtain a new ontology where each original individual can have none, one, or more matching counterparts. The goal is to add some noise in the expected mappings in such a way that the resulting benchmark contains both test cases where each original instance has only one matching counterpart (i.e., one-to-one mappings) and test cases where each original instance may have more than one matching counterpart (i.e., one-to-many mappings).

Data value transformation. operations work on the concrete values of data properties and their datatypes when available. The output is a new concrete value. In the standard transformation typos are simulated as well as special value transformations for dates, names, gender attributes, and numbers like integers and float. Furthermore, in our synonym transformation we extract synonyms from WordNet (e.g. Jackson has won multiple awards is transformed to Jackson has gained several prizes). Data structure transformation. operations change the way data values are connected to individuals in the original ontology graph and change the type and number of properties associated with a given individual. A comprehensive example of data structure transformation is shown in Table 7, where an initial set of assertions A is transformed in the corresponding set of assertions A by applying the property type transformation, property assertion deletion/addition, and property assertion splitting.

12

This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d Data semantics transformation. operations are based on the idea of changing the way individuals are classified and described in the original ontology.

For the sake of brevity, we illustrate the main semantic transformation operations by means of the following example, by taking into account the portion of T O and the assertions sets A and A shown in Table 8.

Data evaluation techniques.

Finally, in the data evaluation activity, we automatically create a ground-truth as a reference alignment for each test case. A reference alignment contains the mappings between the reference ABox individuals and the corresponding transformed individuals in the test case. These mappings are what an instance matching application is expected to find between the original ABox and the test case. creates(b, k) acted by(k, r) f eaturing(k, r) name(k, "Luke Skywalker") name(k, "Luke Skywalker") name(b, "George Lucas") name(b, "George Lucas") name(r, "Mark Hamill") name(r, "Mark Hamill")

The Benchmarks for the OAEI 2010 and 2011

For the OAEI 2010 campaign two datasets of different size have been used. We provided one small dataset containing about 400 individuals and one larger dataset with about 1400 individuals. In the OAEI 2011 campaign we increased the size of the dataset to 12.333 individuals since we were interested in being more realistic with the size of the benchmark. Table 9 summarizes the different characteristics of the datasets.

There exist 80 test cases for each of the datasets, divided into 4 sets of 20 test cases each. The first three sets are different implementations of data value, data structure, and data semantic transformations, respectively, while the fourth set is obtained by combining together the three kinds of [START_REF] Euzenat | Ontology Matching[END_REF] This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d transformations.

Results

The IIMB benchmark has been used for evaluation of matching systems in the campains of 2010 and 2011. In 2010 the three Systems ASMOV [START_REF] Jean-Mary | Asmov: Results for oaei 2010[END_REF], CODI [START_REF] Noessner | Codi: Combinatorial optimization for data integration-results for oaei[END_REF], and RiMOM [START_REF] Wang | Rimom results for oaei[END_REF] participated in both datasets, the small and the large version of IIMB.

Figure 3 shows the results of the large version. All the systems obtained very good results when dealing with data value transformations and logical transformations, both in terms of precision and in terms of recall. Instead, in case of structural transformations (e.g., property value deletion of addition, property hierarchy modification) and of the combination of different kinds of transformations we have worse results, especially concerning recall. Looking at the results, it seems that the combination of different kinds of heterogeneity in data descriptions is still an open problem for instance matching systems. When comparing the overall F-measures of the participating systems, all systems are comparable. CODI reached the highest Fmeasure score of 0.87, RiMOM had 0.84 F-measure, and ASMOV's F-measure was 0.82.

The systems produced almost similar results in the small IIMB benchmark. However, the average F-measure was slightly better for all participating systems in the small benchmark compared to the large one. CODI's F-measure was 0.89, RiMOM reached 0.88, and ASMOV had 0.84 F-measure.

In the 2011 campain we increased the size of the benchmark in order to create a more realistic testing scenario. In the IIMB 2011 dataset it was not suitable any more to calculate the similarity values of every possible indivdual correspondence because this would result in approximative 12, 333 • 12, 333 = 152, 102, 889 similarity comparisons. Unfortunately, only CODI could cope with this large dataset. CODI gained an average Fmeasure of 0.60. This score is not comparable to the IIMB 2010 benchmarks since the transformation complexity was heavily increased in the 2011 benchmark.

From these two years we can conclude that actual matching systems perform quite well on small to medium sized benchmarks, but have difficulties with large datasets. This, however, is an important requirement in linked open data where a vast amount of instances exist.

Current issues and open problems

The benchmarks proposed for instance matching at OAEI so far have been shown to be adequate for the evaluation of instance matching algorithms and tools. This approach has been derived from the work done at OAEI about ontology matching [START_REF] Euzenat | Ontology alignment evaluation initiative: Six years of experience[END_REF]. The main effort of the instance matching track has been devoted in providing specific datasets for instance matching and in introducing the idea of using artificially generated data with the goal of analytically controlling the performances of instance matching tools in different situations. However, there are several open issues about the evaluation of instance matching technologies with respect to ontology matching tools that are still open and that we have learned through our experience at OAEI.

Future directons for IM@OAEI

The experience of the first editions of the instance matching track in the context of OAEI has provided materials, suggestions, and tools for improving the contest and trying to address the main issues still open in future editions. In particular, some of the open issues discussed in the previous section have been already partially addressed, while others require new actions or new tools. In Table 10, we summarize what we already have and what is still required in the future.

More in detail, possible future actions for IM@OAEI are the following: Larger datasets. We already have large datasets, especially for what concerns the real-data benchmark. However, the contest has been focused in the previous editions only on precision and recall. In future editions, we will focus more on time performance of matching tools, trying to understand the scalability of matching techniques and the capability of matching tools to incrementally evaluate 14 This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d

Artificially generated benchmark

Improve the transformation functions including more standard transformations (e.g., acronyms) Structural differences between ontology and instances as graphs

Artificially generated benchmark

Add the graph density as a parameter for transformation functions

Relations between datasets and the real-world

Real-data benchmark Include the usage of external sources as a reference for matching tools Mutual relations between ontology and instance matching

Artificially generated benchmark

Improve transformations based on the logical structure of the reference Tbox matching when the number of entities in a dataset grows. To this end, we will provide datasets with a different number of entities, relations, and properties, in order to put in relation tools time efficiency with the number of elements involved in the matching process.

Identity and similarity. One of the main problems in evaluating the capability of matching tools to identify different kind of "same-as" relations between instances is that a reference set of expected relations among data is usually missing. In order to address this problem, one of the possible solutions is to implement in the artificially generated benchmark tool a functionality for generating an expected set of mappings including several kinds of possible relations, ranging from strict identity to simple similarity. As an example, we could think to an instance I featured by two properties: the first property is sufficient to identify the object represented by the instance (i.e., an ID code), while the second property as a more generic value (e.g., a date or the gender). In such a case, we will generate two different instances, I and I . The first instance I is featured by both the properties with values transformed by means of one of the string/date transformation functions already available in our transformation tools. In generating the second instance I we apply transformations functions but we also delete the ID property. Then, we generate to mappings for I. The first mapping I ↔ I denotes the identity between I and I , since both the instance refer to the same real-world object. The second mapping I ↔ I denotes a generic similarity between the two instances, since we do not have enough information to conclude the identity of I . Finally, participants will be required to discover both the mappings with the correct meaning.

Different role of names and property values. This issue is already partially addressed in the artificially generated benchmark. In fact, we are already able to generate new property values by applying arbitrary transformation functions. However, the transformation process does not take into account the value distribution of each property. Our plan is to include such a parameter in order to make it possible for the benchmark designer to choose if she wants to apply transformations only to properties featured by a limited range of possible values (i.e., no identifying properties) and/or also to properties with many possible values (i.e., highly identifying properties).

Different kinds of data heterogeneities. This issue is also already partially addressed in the artificially generated benchmark. However, the majority of string transformation functions used in the transformation tool are based on random string transformations. In order to make transformations more realistic, we will implement more transformation functions for some kinds of standard data formats, such as for examples acronyms.

Structural differences between ontology and instance as graphs. Actually, in the automatically generated benchmark it is possible to transform a property which has another instance as value into a property with a concrete value, and viceversa. As an example, suppose to have an instance representing a movie, featured by a property director having as value another instance D representing the director John Smith. D is featured by a property name, having the string "John Smith" as value. In our benchmark, it is possible to transform the movie instance by deleting the instance D and changing the property director which will have the string "John Smith" as value. On the contrary, it is also possible to transform a string into a new individual. This benchmark generation capability helps the designer in controlling the density of the resulting graph of the test cases. However, this procedure is not iterable to obtain large graphs with a very high number of individuals and properties. In order to increase the complexity of the graphs resulting from transformations, we will add a parameter which will make it possible to control the number of individuals and new properties added to the graph by adopting this specific transformation.

Relations between datasets and the real-world. Our real-data benchmark already contains real-world data. However, till now, we have not explored the role played by external data sources of referent on the instance matching process. A possible approach to this end is to provide a specific subset of data which includes additional data to be used as a support in correctly determining the mappings between elements. Then, we will require participants to execute a first run of matching using only the dataset without the additional data, which will instead be available only for a second run. The idea is to compare evaluation results after the first run agains evaluation results after the second run, in order to observe the impact of using additional data in the matching process.

Mutual relations between ontology and instance matching. One section of test-cases produced in the artificially generated benchmark is devoted to transformations which are based on the Tbox structure that is transformed as well. About this issue, we plan to improve the transformations based on the logical structure of the original dataset, by including new ontological transformations by reducing at the same time the information provided by property values, especially for what concerns concrete values. In such a way, the information about instances derived from the Tbox constraints will be-

16

This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d come more crucial to the end of finding the correct mappings. This approach has the goal of determining if and how much matching tools are capable of exploiting the ontology during the instance matching process.

Concluding remarks

In this paper, we have presented the experience of IM@OAEI, an initiative to promote the evaluation of instance matching and data linking techniques and tools, in the context of OAEI, the Ontology Alignment Evaluation Initiative. In particular, we have presented our approach that is based on the idea of combining real-data and automatically generated data for the evaluation in order to provide on one side a realistic context for instance matching tools and, on the other side, a framework where we can reproduce different causes of data heterogeneity in order to analytically and programmatically verify the points of strength and weakness of each evaluated tool. At the time of writing this paper, we concluded the 2012 edition of IM@OAEI [START_REF] Aguirre | Results of the Ontology Alignment Evaluation Initiative[END_REF]. The 2012 results actually confirm the considerations done for the previous editions of the initiative. Our future work in the next editions of IM@OAEI will be devoted to the study of new measures for the evaluation besides the classical precision and recall as well as on the improvement of our benchmarks with the goal of evaluating the behavior of the instance matching tools with respect to some crucial open problems in the field, such as the semantics of instance mappings and the efficiency of matching tools when dealing with large collections of data.

Figure 1 :

 1 Figure 1: Precision/recall of tools participating in the DI subtrack.

(Figure 2 :

 2 Figure 2: The SWING approach

Figure 3 :

 3 Figure 3: Results of IIMB 2010, large version

Table 1 :

 1 Results of the real-data benchmark subtrack.

	Concept	sweto:Publication		foaf:Person			Overall
	System	Prec.	Rec.	FMeas. Prec. Rec. FMeas. Prec. Rec. FMeas.
					AKT/REXA
	DSSim	0.15	0.16 0.16		0.81	0.43	0.30	0.60	0.38	0.28
	RiMOM	0.96	0.96	0.97	0.93	0.79	0.70	0.94	0.80	0.59
	OKKAM	0.99	0.76	0.61	0.73	0.03	0.02	0.94	0.18	0.10
	HMatch	0.97	0.93	0.89	0.94	0.56	0.39	0.95	0.62	0.46
	ASMOV	0.32	0.46	0.79	0.76	0.37	0.24	0.52	0.39	0.32
					AKT/DBLP
	DSSim	0	0	0	0.15	0.19	0.11	0.17	0.13	0.15
	RiMOM	1.0	0.84	0.72	0.92	0.79	0.70	0.93	0.73	0.70
	OKKAM	0.98	0.88	0.80	0	0	0	0.98	0.28	0.16
	HMatch	0.93	0.95	0.97	0.58	0.57	0.57	0.65	0.65	0.65
					REXA/DBLP
	DSSim	0	0	0	0	0	0	0	0	0
	RiMOM	0.94	0.94	0.95	0.76	0.71	0.66	0.80	0.76	0.72
	OKKAM	0.98	0.26	0.15	1.00	0.20	0.11	0.99	0.21	0.12
	HMatch	0.45	0.61	0.96	0.40	0.37	0.34	0.42	0.45	0.48

Table 2 :

 2 Health-care benchmark composition

	Dataset	Dailymed	Diseasome	Drugbank	Sider
	Dailymed	0	0	0	1,592
	Diseasome	0	0	0	238
	Drugbank	0	0	0	283
	Sider	1,986	0	1,140	0

Table 3 :

 3 Results of the real-data benchmark subtrack.

			ObjectCoref
	Dataset	Prec. FMeas. Rec.
	dailymed	0,55	0,09	0,05
	diseasome 0,84	0,10	0,05
	drugbank	0,30	0,05	0,03
	sider	0,00	NaN	0,00
	H-mean	0,50	0,08	0,04
			RiMOM
	Dataset	Prec. FMeas. Rec.
	dailymed	0,08	0,13	0,30
	diseasome 0,08	0,13	0,47
	drugbank	0,05	0,08	0,42
	sider	0,62	0,53	0,47
	H-mean	0,08	0,13	0,35

  13 http://dbpedia.org 14 http://www.freebase.com/ 15 http://www.geonames.org/ 16 http://www.w3.org/TR/skos-reference/

	10

Table 4 :

 4 The New York Times benchmark composition

	Facet	# Concepts Links to Freebase Links to DBPedia Links to Geonames
	People	4,979	4,979	4,977	0
	Organizations	3,044	3,044	1,965	0
	Locations	1,920	1,920	1,920	1,920

Table 5 :

 5 Results of the real-data benchmark subtrack.

		AgreementMaker		SERIMI			Zhishi.links
	Dataset	Prec. FMeas. Rec. Prec. FMeas. Rec. Prec. FMeas. Rec.
	DI-nyt-dbpedia-loc.	0,79	0,69	0,61	0,69	0,68	0,67	0,92	0,92	0,91
	DI-nyt-dbpedia-org.	0,84	0,74	0,67	0,89	0,88	0,87	0,90	0,91	0,93
	DI-nyt-dbpedia-peo.	0,98	0,88	0,80	0,94	0,94	0,94	0,97	0,97	0,97
	DI-nyt-freebase-loc.	0,88	0,85	0,81	0,92	0,91	0,90	0,90	0,88	0,86
	DI-nyt-freebase-org.	0,87	0,80	0,74	0,92	0,91	0,89	0,89	0,87	0,85
	DI-nyt-freebase-peo.	0,97	0,96	0,95	0,93	0,92	0,91	0,93	0,93	0,92
	DI-nyt-geonames.	0,90	0,85	0,80	0,79	0,80	0,81	0,94	0,91	0,88
	H-mean.	0,92	0,85	0,80	0,89	0,89	0,88	0,93	0,92	0,92

Table 6 :

 6 Examples of data transformation operations

	Operation	Original value	Transformed value
	Standard trans-	Luke Skywalker	L4kd Skiwaldek
	formation
	Date format	1948-12-21	December 21, 1948
	Name format	Samuel L. Jack-	Jackson, S.L.
		son
	Gender format	Male	M
	Synonyms	Jackson has won	Jackson has gained
		multiple awards	several prizes [...]
		[...]
	Integer	10	110
	Float	1.3	1.30

Table 7 :

 7 Example of data structure transformations

	Original Abox	Transformed Abox
	name(n, "Natalie Portman")	name(n, "Natalie")
	born in(n, m)	name(n, "Portman")
	name(m, "Jerusalem")	born in(n, m)
	gender(n, "Female")	name(m, "Jerusalem")
	date of birth(n, "1981-06-09")	name(m, "Auckland")
		obj gender(n, y)
		has value(y, "Female")

Table 8 :

 8 Example of data semantic transformations

	Tbox
	Character		Creature, created by	≡	creates -,
	acted by	f eaturing, Creature Country	⊥
	Original Abox	Transformed Abox
	Character(k)	Creature(k)
	Creature(b)	Country(b)
	Creature(r)	(r)
	created by(k, b)

Table 9 :

 9 Characteristics of the automatically generated benchmarks.

		2010 small	2010 large	2011
	Individuals	363	1416	12,333
	Classes	29	81	163
	Object-Properties	32	32	45
	Data-Properties	13	13	13
	DL-Expressivity	ALCHI(D)	ALCHI(D)	ALCHI(D)

Table 10 :

 10 Current and future actions with respect to the open issues in the evaluation of instance matching tools

	Issue	Current actions /	Future directions
		material
	Larger datasets	Real-data benchmark More focus on time performances of matching
			tools
	Identity and similarity	Artificially generated	Generate different kinds of possible mappings
		benchmark	according to different definitions of "same-as".
			Participants should be able to discriminate
			among the different mappings
	Different role of names and	Artificially generated	Generate new transformations on the basis of the
	property values	benchmark	property values distribution
	Different kinds of data het-
	erogeneities

http://www.cs.umass.edu/ mccallum/data.html

http://dl.acm.org

http://www.informatik.uni-trier.de/ ley/db/

http://www.cs.utexas.edu/users/ml/riddle/

Experiments with different Cora versions are reported in[START_REF] Dong | Reference reconciliation in complex information spaces[END_REF],[START_REF] Domingos | Multi-relational record linkage[END_REF],[START_REF] Rendle | Object identification with constraints[END_REF], and[START_REF] Weis | Relationship-based duplicate detection[END_REF]

http://www.aktors.org/

http://rexa.info/

This preprint research paper has not been peer reviewed. Electronic copy available at: https://ssrn.com/abstract=3199068 P r e p r i n t n o t p e e r r e v i e w e d

http://data.nytimes.com/

http://www.freebase.com